Business Transaction Monitoring is a new capability that has been provided in IBM Integration Bus 10.0.0.3. This new capability has been introduced in Business Transaction Monitoring – Why, what, how and the steps to set up and use BTM have been described in Business Transaction Monitoring in IIB.
The following areas are covered in this article:
MQ administration for BTM: You will see what aspects of MQ are used by BTM and what further configuration or tuning you may need to do.
Database administration for BTM: You will see the database tables used by BTM and their inter-dependencies. You will see how to configure BTM for your database.
IIB administration for BTM: You will see how to configure the recording and viewing capability provided by BTM. You will also see how to transfer your configuration for BTM from a test system to a production system.
Planning your use of BTM: You will see how you should consider configuring monitoring events in your message flows and using multiple Business Transaction Definitions across a set of message flows that are used by your business.
MQ administration for BTM
Monitoring events are configured on message flow nodes using the IBM Integration Toolkit or by creating a monitoring profile. When a message is processed by a node which has an event configured on it, an MQ message is published to an MQ topic. The MQ topic name has this structure:
When a Business Transaction Definition is created, a selection of monitoring events are flagged as business events. They are deemed crucial to understanding the state of a business transaction. An MQ subscription is created for each topic that is used by these monitoring events that have been flagged.
After creating, each BTD, an MQ subscription is created for the relevant topics. In MQ Explorer, I have these subscriptions that were created automatically:
When a transaction is processed, the events are emitted to the MQ topic. The relevant subscription picks up the event and puts it to a destination queue. The destination queue is defined in the Data Capture policy supplied with IIB v10.0.0.3. It is defined in this policy attribute:
Messages are read from this queue and recorded into the database. If a database error occurs while recording the date, then the messages are put to a backout queue. The backout queue is also defined in the Data Capture policy and is defined in this policy attribute:
These queues are defined when you run the script iib_createqueues.[cmd|sh]. If you wish to use alternative queues, then you can modify the Data Capture policy provided by IIB. To, do this run this command:
mqsireportpolicy integrationNodeName -t DataCapture -l default -f default.xml
Edit the file default.xml that has been created and replace the queue names with the alternative ones that you want to use. Now, update the Data Capture policy, by running this command:
mqsichangepolicy integrationNodeName -t DataCapture -l default -f default.xml
The Data Capture policy will now be using the queues that you have defined. The depth of the RECORD queue should be monitored so that it stays at a constant depth. Performance of business transaction monitoring can be impacted if business events are written to the queue faster than they can be written to the database. For this reason, you should be careful to flag only those events that you deem critical for monitoring your business transaction.
Database administration for BTM
A prerequisite for using BTM is to have a database created and tables defined in it. The steps to configure the database are described in Business Transaction Monitoring in IIB.
The tables that are used by Business Transaction are:
After you have configured your message flow nodes to emit events and created a business transaction definition containing those flows and events, entries are written to these database tables when business transactions are processed by those flows.
The set of events that are correlated by BTM to ascertain the state of a business transaction are recorded in the WMB_BUSTRANS table. This table contains all the events that have been flagged across all BTDs and have been emitted by the message flow nodes.
Entries that are in the WMB_BUSTRANS table for business transactions must have corresponding entries in the WMB_MSGS table.
The state of a business transaction is calculated depending on the events that have been received for that business transaction. There are 4 states for a business transaction:
- In progress – indicates that the transaction is still running. The Start and any Progress events have been recorded, but the End or Failure event has not been recorded. If only a Start event has been recorded, the business transaction is still considered to be In Progress.
- Ended – indicates that the transaction has completed successfully. The Start, Progress (if any) and End events have been recorded.
- Failed – indicates that the transaction completed in a way that the user has designed as a failure path when defining the BTD. The Start, Progress (if any) and Failure events have been recorded.
- Inconsistent – indicates that the transaction’s state could not be resolved from the set of events that have been recorded.
A business transaction may be marked as inconsistent for different reasons. Here are some examples:
- Only a Progress event has been received but no Start
- Only a Progress and End event has been received but no Start
- Only an End event has been received but no Start
- Two or more Start events or End events were received for the same transaction. This could have happened when 2 or more transactions used the same global transaction ID. A global transaction ID must be unique per business transaction.
- A transaction was received, an error thrown during the transaction causing a rollback and then the transaction was re-processed using the same global transaction ID.
The Data Capture profile supplied with IIB can be tailored for settings that affect the recording of data into the database. These fields are used in the Data Capture policy which can be configured:
- This is the name of a datasource defined for the database using mqsisetdbparms. This can be altered using the web user interface in the Business Transactions section.
- This is the schema used for the database.
- This indicates the number of transactions that are processed on each thread before they are committed. This value must be 1 if the value of threadPoolSize is greater than 1. Otherwise, this value is ignored.
- This indicates the time interval at which a commit is taken when the commitCount value is greater than 1 but the number of transactions that have been processed has not reached the value of the commitCount property.
- This indicates the number of threads that are used by the integration server used for recording the business data.
- This indicates whether transactions are globally coordinated across queue manager and database resources.
If you wish to modify any of these values, run mqsireportpolicy to get a file containing the Data Capture policy, update the properties and then run mqsichangepolicy to set the Data Capture policy.
You can run mqsireportpolicy again to check your new values are in action.
IIB administration for BTM
Read how to configure the recording and viewing capability provided by BTM, and how to transfer your configuration for BTM from a test system to a production system.
How to specify servers for recorder and viewing
To make it easier to get going with Business Transaction Monitoring, an Integration Server is chosen for recording and viewing automatically after you have created your business transaction definition.
You might want to change this to designate a specific server for recording and another for viewing to achieve better utilisation or distribution of work in your deployment scenario.
If you want to configure a specific Integration Server to use for recording and viewing, you can configure this in the Data Capture policy.
Run mqsireportpolicy as described above to get a file containing the Data Capture policy. Update these properties for the Integration Server to be used for recording or viewing:
After you have configured the Integration Server, run mqsichangepolicy as described above for the changes to take effect.
You will see the following message in the syslog to indicate which Integration Server is being used for recording:
BIP2159I: ( btm_node.main ) Integration server ''main'' is now recording data for the following data capture stores: ''/apiv1/policy/DataCapture/default#default''.
The Data Capture policy provided by IIB has no specific Integration Server configured for recording and viewing. It uses the first Integration Server that has been started for recording and viewing.
Moving BTDs between different Integration Nodes
If you want to move your BTDs between systems, you can use a REST client using an HTTP GET to get your BTD from one system and then use a HTTP PUT put it to another system.
Specify this URL to get your BTD with an HTTP GET:
using these HTTP Headers:
This will return a JSON response.
You can now put it to the target system using your REST client, specifying an HTTP PUT.
Specify this URL to put your BTD with an HTTP PUT:
You must specify these HTTP Headers when issuing the request:
You must copy the JSON output from the HTTP GET request that you just did and place it in the Request Body for the HTTP PUT request.
You may need to create the same Integration Servers on your target machine as you did on your source machine, or you may need to tailor what you put in the Request Body according to the Integration Servers that you are using on your target machine.
Planning your use of BTM
In order to get the maximum benefit from using Business Transaction Monitoring, you should take the following approach:
- Be very clear on what events are most important to you for understanding the state of your business transaction.
- Consider what message rates you are expecting your system to be handling and based on that consider how many events you want recorded per transaction.
- Think about what aspects of your business that you want to monitor using Business Transaction Monitoring. If you have a very heavily loaded system, then you may want to consider just tracking transactions that have failed, rather than tracking all transactions.
Here is a simple example which illustrates what can happen if you do not plan your BTDs and events.
I have a simple Retail system which receives customer requests and forwards them to three different business areas: Stock Checking, Ordering and Purchasing. The responses are received from the difference business areas and sent back to the customer.
There is a Request flow which handles the customer requests and a Response flow which sends the responses back to the customer:
I have separate applications containing message flows for the different business areas.
Lets create an orderItem BTD and a purchaseItem BTD. I will specify the start event in the request flow, a progress event in each business area and the end event in the response flow.
These events have been flagged as business events for the orderItem BTD:
The purchaseItem BTD has these flows with events configured on 3 flows as well. Although the Progress event for orderitem is from the Order_Item_Despatch_flow, while the Progress event for purchaseItem is from the Process_Payment_Flow.
These events have been flagged as business events for the purchaseItem BTD:
You can see that both orderItem and purchaseItem are using the same flows: Request_Flow and Response_Flow for their Start and End events.
Let’s send in 2 orderItem and 2 purchaseItem transactions.
When I view the transactions for orderItem, I would want to see that 2 order item transactions were processed. However, I see 4 transactions!!
This has happened as the start and end events for both business transactions were defined in a common flow. The Start and End event were captured for all events by both BTDs. The business transaction definitions should have been split up in a better way so that we only see the transactions for orderItem in its BTD and the same for purchaseItem.
The recommended practice for this scenario is to define a general BTD to capture the events by the main processing flows Request_Flow and Response_Flow. The orderItem BTD should have only events that are specific to that business area, so only flag the events the message flows for ordering an item. The same goes for purchaseItem.
Here is a better structure for the BTDs.
- I have defined a general BTD called customerRequests which has Start, Progress, End and Failure events on the Request_Flow and Response_Flow flows.
- I have removed the Request_Flow and Response_Flow flows from the orderItem and purchaseItem flows.
- For orderItem, I have defined the Start event in the Order_Item_Request_Flow and the End event in the Order_Item_Response_Flow.
- For purchaseItem, I have defined the Start event in the Purchase_Item_Request_Flow and the End event in the Purchase_Item_Response_Flow.
customerRequests – define tab
orderItem – define tab
purchaseItem – define tab
Now when we send in the events again, we see 4 events shown for customerRequests, 2 events for orderItem and 2 events for purchaseItem which is exactly as expected.
customerRequests – view tab
4 events are shown as expected.
orderItem – view tab
2 events are shown as expected.
purchaseItem – view tab
2 events are shown as expected.
In this article, we have discussed different areas of administration for the new capability provided in 10.0.0.3, Business Transaction Monitoring. We have shown you how to configure BTM for aspects of MQ, your Database and IIB. You should consider where you define monitoring events and which business transaction definitions should flag those events as business events.