Batch processing with IBM App Connect provides you with the capability to migrate large volumes of data from one system to another. However, depending on the volume of data the whole process can take some time and some data might not migrate successfully due to various errors. It’s critical to know which records/entries in your data have failed and where the errors are so that you can take further action. This blog will show you how to specify unique log IDs for each record in the batch process to enable you to debug each record individually.
Let’s take the flow that we created in a previous blog, Introducing batch processing in IBM App Connect, or you can access the template “Sync leads between Salesforce accounts and send email with the sync result” from the template gallery. This flow starts a batch process using the scheduler node to retrieve leads from Salesforce Account 1 and then create or update leads in Salesforce Account 2.
Defining a batch record ID
To specify a unique log ID for each record, in the Salesforce Retrieve leads action, select Specify a log ID for each record of the batch. An input box is displayed for you to define the ID. In this example, we are going to use the Lead ID from the retrieved Salesforce lead node as the batch record ID.
Using the batch record ID
Now, let’s take a look at how to utilise the batch record ID to identify failed records.Â From the dashboard, find the tile for your flow. In its three-dots context menu, you will find the options to view errors. In this demonstration, in the Error messages dialog box, there are some errors messages which indicate that not all of the records have been updated successfully. To gain access to more detailed information about the errors, we can access the Kibana logging tool by opening the options menu in the Actions column and selecting View logs:
To display the batch record ID in the Kibana log table, in the side panel of the Kibana logging tool, find batch-record-id_str in the Available Fields section and add it to the Selected Fields section by clicking Add:Â
We can now easily identify the failed records. To view the logs for a specific record, you can modify the search query by adding a batch-record-id_str filter. For example, as displayed in the following image, append the following string to the search query,