Christmas bow and hollyLearn about different areas of administration for Business Transaction Monitoring (BTM); see how to configure BTM for aspects of MQ, your database, and IBM Integration Bus.

Business Transaction Monitoring is a new capability that has been provided in IBM Integration Bus 10.0.0.3. This new capability has been introduced in Business Transaction Monitoring – Why, what, how and the steps to set up and use BTM have been described in Business Transaction Monitoring in IIB.

The following areas are covered in this article:

MQ administration for BTM: You will see what aspects of MQ are used by BTM and what further configuration or tuning you may need to do.

Database administration for BTM: You will see the database tables used by BTM and their inter-dependencies. You will see how to configure BTM for your database.

IIB administration for BTM: You will see how to configure the recording and viewing capability provided by BTM. You will also see how to transfer your configuration for BTM from a test system to a production system.

Planning your use of BTM: You will see how you should consider configuring monitoring events in your message flows and using multiple Business Transaction Definitions across a set of message flows that are used by your business.

MQ administration for BTM

Monitoring events are configured on message flow nodes using the IBM Integration Toolkit or by creating a monitoring profile. When a message is processed by a node which has an event configured on it, an MQ message is published to an MQ topic. The MQ topic name has this structure:

$SYS/Broker/integrationNodeName/Monitoring/integrationServerName/flow_name

When a Business Transaction Definition is created, a selection of monitoring events are flagged as business events. They are deemed crucial to understanding the state of a business transaction. An MQ subscription is created for each topic that is used by these monitoring events that have been flagged.

Read more...
For example, I have created three separate business transaction definitions (BTDs), as shown below:

fig01

After creating, each BTD, an MQ subscription is created for the relevant topics. In MQ Explorer, I have these subscriptions that were created automatically:

fig02

When a transaction is processed, the events are emitted to the MQ topic. The relevant subscription picks up the event and puts it to a destination queue. The destination queue is defined in the Data Capture policy supplied with IIB v10.0.0.3. It is defined in this policy attribute:

<queueName>SYSTEM.BROKER.DC.RECORD</queueName>

Messages are read from this queue and recorded into the database. If a database error occurs while recording the date, then the messages are put to a backout queue. The backout queue is also defined in the Data Capture policy and is defined in this policy attribute:

<backoutQueue>SYSTEM.BROKER.DC.BACKOUT</backoutQueue>

These queues are defined when you run the script iib_createqueues.[cmd|sh]. If you wish to use alternative queues, then you can modify the Data Capture policy provided by IIB. To, do this run this command:

mqsireportpolicy integrationNodeName -t DataCapture -l default -f default.xml

Edit the file default.xml that has been created and replace the queue names with the alternative ones that you want to use. Now, update the Data Capture policy, by running this command:

mqsichangepolicy integrationNodeName -t DataCapture -l default -f default.xml

The Data Capture policy will now be using the queues that you have defined. The depth of the RECORD queue should be monitored so that it stays at a constant depth. Performance of business transaction monitoring can be impacted if business events are written to the queue faster than they can be written to the database. For this reason, you should be careful to flag only those events that you deem critical for monitoring your business transaction.

Database administration for BTM

A prerequisite for using BTM is to have a database created and tables defined in it. The steps to configure the database are described in Business Transaction Monitoring in IIB.

The tables that are used by Business Transaction are:

  • WMB_MSGS
  • WMB_BINARY_DATA
  • WMB_BUSTRANS

After you have configured your message flow nodes to emit events and created a business transaction definition containing those flows and events, entries are written to these database tables when business transactions are processed by those flows.

Read more...
All monitoring events that were defined on the message flow nodes are recorded in the WMB_MSGS table. If you have specified that you want the payload data to be included in the recorded event, then the payload data is recorded in the WMB_BINARY_DATA table.

The set of events that are correlated by BTM to ascertain the state of a business transaction are recorded in the WMB_BUSTRANS table. This table contains all the events that have been flagged across all BTDs and have been emitted by the message flow nodes.

Entries that are in the WMB_BUSTRANS table for business transactions must have corresponding entries in the WMB_MSGS table.

The state of a business transaction is calculated depending on the events that have been received for that business transaction. There are 4 states for a business transaction:

  • In progress – indicates that the transaction is still running. The Start and any Progress events have been recorded, but the End or Failure event has not been recorded. If only a Start event has been recorded, the business transaction is still considered to be In Progress.
  • Ended – indicates that the transaction has completed successfully. The Start, Progress (if any) and End events have been recorded.
  • Failed – indicates that the transaction completed in a way that the user has designed as a failure path when defining the BTD. The Start, Progress (if any) and Failure events have been recorded.
  • Inconsistent – indicates that the transaction’s state could not be resolved from the set of events that have been recorded.

A business transaction may be marked as inconsistent for different reasons. Here are some examples:

  1. Only a Progress event has been received but no Start
  2. Only a Progress and End event has been received but no Start
  3. Only an End event has been received but no Start
  4. Two or more Start events or End events were received for the same transaction. This could have happened when 2 or more transactions used the same global transaction ID. A global transaction ID must be unique per business transaction.
  5. A transaction was received, an error thrown during the transaction causing a rollback and then the transaction was re-processed using the same global transaction ID.

The Data Capture profile supplied with IIB can be tailored for settings that affect the recording of data into the database. These fields are used in the Data Capture policy which can be configured:

<dataSourceName>MBRECORD</dataSourceName>

  • This is the name of a datasource defined for the database using mqsisetdbparms. This can be altered using the web user interface in the Business Transactions section.

<schema></schema>

  • This is the schema used for the database.

<commitCount>1</commitCount>

  • This indicates the number of transactions that are processed on each thread before they are committed. This value must be 1 if the value of threadPoolSize is greater than 1. Otherwise, this value is ignored.

<commitIntervalSecs>3</commitIntervalSecs>

  • This indicates the time interval at which a commit is taken when the commitCount value is greater than 1 but the number of transactions that have been processed has not reached the value of the commitCount property.

<threadPoolSize>10</threadPoolSize>

  • This indicates the number of threads that are used by the integration server used for recording the business data.

<useCoordinatedTransaction>false</useCoordinatedTransaction>

  • This indicates whether transactions are globally coordinated across queue manager and database resources.

If you wish to modify any of these values, run mqsireportpolicy to get a file containing the Data Capture policy, update the properties and then run mqsichangepolicy to set the Data Capture policy.

You can run mqsireportpolicy again to check your new values are in action.

IIB administration for BTM

Read how to configure the recording and viewing capability provided by BTM, and how to transfer your configuration for BTM from a test system to a production system.

Read more...

How to specify servers for recorder and viewing

To make it easier to get going with Business Transaction Monitoring, an Integration Server is chosen for recording and viewing automatically after you have created your business transaction definition.

You might want to change this to designate a specific server for recording and another for viewing to achieve better utilisation or distribution of work in your deployment scenario.

If you want to configure a specific Integration Server to use for recording and viewing, you can configure this in the Data Capture policy.

Run mqsireportpolicy as described above to get a file containing the Data Capture policy. Update these properties for the Integration Server to be used for recording or viewing:

<recordingServer></recordingServer>

<viewingServer></viewingServer>

After you have configured the Integration Server, run mqsichangepolicy as described above for the changes to take effect.

You will see the following message in the syslog to indicate which Integration Server is being used for recording:

BIP2159I: ( btm_node.main ) Integration server ''main'' is now recording data for the following data capture stores: ''/apiv1/policy/DataCapture/default#default''.

The Data Capture policy provided by IIB has no specific Integration Server configured for recording and viewing. It uses the first Integration Server that has been started for recording and viewing.

Moving BTDs between different Integration Nodes

If you want to move your BTDs between systems, you can use a REST client using an HTTP GET to get your BTD from one system and then use a HTTP PUT put it to another system.

Specify this URL to get your BTD with an HTTP GET:

http://hostname1:4414/apiv1/business/businesstransactions/BTDname

using these HTTP Headers:

  • Content-Type: application/x-www-form-urlencoded
  • Accept: application/javascript, application/json

This will return a JSON response.

You can now put it to the target system using your REST client, specifying an HTTP PUT.

Specify this URL to put your BTD with an HTTP PUT:

http://hostname2:4414/apiv1/business/businesstransactions/BTDname

You must specify these HTTP Headers when issuing the request:

  • Content-Type:application/json
  • Accept:application/json

You must copy the JSON output from the HTTP GET request that you just did and place it in the Request Body for the HTTP PUT request.

You may need to create the same Integration Servers on your target machine as you did on your source machine, or you may need to tailor what you put in the Request Body according to the Integration Servers that you are using on your target machine.

Planning your use of BTM

In order to get the maximum benefit from using Business Transaction Monitoring, you should take the following approach:

  • Be very clear on what events are most important to you for understanding the state of your business transaction.
  • Consider what message rates you are expecting your system to be handling and based on that consider how many events you want recorded per transaction.
  • Think about what aspects of your business that you want to monitor using Business Transaction Monitoring. If you have a very heavily loaded system, then you may want to consider just tracking transactions that have failed, rather than tracking all transactions.
Read more...
You are likely to be using multiple applications with many message flows across multiple Integration Servers. Business Transaction Monitoring is quite capable of supporting that topology, but in order to understand the business transaction results, it is encouraged that you separate your BTDs rather than them overlapping across flows.

Here is a simple example which illustrates what can happen if you do not plan your BTDs and events.

I have a simple Retail system which receives customer requests and forwards them to three different business areas: Stock Checking, Ordering and Purchasing. The responses are received from the difference business areas and sent back to the customer.

There is a Request flow which handles the customer requests and a Response flow which sends the responses back to the customer:

I have separate applications containing message flows for the different business areas.

Lets create an orderItem BTD and a purchaseItem BTD. I will specify the start event in the request flow, a progress event in each business area and the end event in the response flow.

The orderItem BTD has these flows with events configured on 3 of the flows. You can see which flows has events configured on it as it has afig03 icon above the flow.

fig04

These events have been flagged as business events for the orderItem BTD:

fig05

The purchaseItem BTD has these flows with events configured on 3 flows as well. Although the Progress event for orderitem is from the Order_Item_Despatch_flow, while the Progress event for purchaseItem is from the Process_Payment_Flow.

fig06

These events have been flagged as business events for the purchaseItem BTD:

fig07

You can see that both orderItem and purchaseItem are using the same flows: Request_Flow and Response_Flow for their Start and End events.

Let’s send in 2 orderItem and 2 purchaseItem transactions.

When I view the transactions for orderItem, I would want to see that 2 order item transactions were processed. However, I see 4 transactions!!

fig08

This has happened as the start and end events for both business transactions were defined in a common flow. The Start and End event were captured for all events by both BTDs. The business transaction definitions should have been split up in a better way so that we only see the transactions for orderItem in its BTD and the same for purchaseItem.

The recommended practice for this scenario is to define a general BTD to capture the events by the main processing flows Request_Flow and Response_Flow. The orderItem BTD should have only events that are specific to that business area, so only flag the events the message flows for ordering an item. The same goes for purchaseItem.

Here is a better structure for the BTDs.

  1. I have defined a general BTD called customerRequests which has Start, Progress, End and Failure events on the Request_Flow and Response_Flow flows.
  2. I have removed the Request_Flow and Response_Flow flows from the orderItem and purchaseItem flows.
  3. For orderItem, I have defined the Start event in the Order_Item_Request_Flow and the End event in the Order_Item_Response_Flow.
  4. For purchaseItem, I have defined the Start event in the Purchase_Item_Request_Flow and the End event in the Purchase_Item_Response_Flow.

customerRequests – define tab

fig09

orderItem – define tab

fig10

purchaseItem – define tab

fig11

Now when we send in the events again, we see 4 events shown for customerRequests, 2 events for orderItem and 2 events for purchaseItem which is exactly as expected.

customerRequests – view tab

4 events are shown as expected.

fig12

orderItem – view tab

2 events are shown as expected.

fig13

purchaseItem – view tab

2 events are shown as expected.

fig14

Summary

In this article, we have discussed different areas of administration for the new capability provided in 10.0.0.3, Business Transaction Monitoring. We have shown you how to configure BTM for aspects of MQ, your Database and IIB. You should consider where you define monitoring events and which business transaction definitions should flag those events as business events.

PS: Do you like the festive IIB posts? You can easily find them all by having a look at our festive calendar or click on the festive2015 tag.

3 comments on"Advanced usages of Business Transaction Monitoring"

  1. “If you wish to modify any of these values, run mqsireportpolicy to get a file containing the Data Capture policy, update the properties and then run mqsichangepolicy to set the Data Capture policy.”

    Simple when you know how…

  2. Our client has an existing IIB 10 installation utilizing JMS .bindings files to connect to a remotely located queue manager for performing business as usual publish and subscribe operations. Does BTM require a local MQ installation with IIB or is there an alternative approach where the queues listed here are added to a .bindings file for them to use?
    The preference is to not create a new MQ co-existence dependency if possible for this BPM implementation.

    • Sanjay Nagchowdhury January 30, 2018

      Hi Roy,
      BTM will only work using an Integration Node which has an associated Queue Manager. You cannot use JMS, nor can you use a remote Queue Manager.

      Thanks

      Sanjay

Join The Discussion

Your email address will not be published. Required fields are marked *