With the release of IBM Integration Bus version 10.0.0.8 you can now simply exploit Product Insights with IBM Integration Bus to view a centralized inventory of your integration servers, and access usage metrics and operational logs to quickly gain operational status insight and help optimize environments.
New functional content is provided in this fixpack release that enables you to configure integration servers to register startup and usage to IBM Cloud Product Insights enabling you to manage your inventory and identify servers that may needed to be optimized for better usage efficiency, view available version upgrades, and access information on relevant hybrid cloud services. Additionally integration servers can be configured to publish BIP log events to Bluemix Logmet beta service enabling them to be viewed centrally using a Kibana dashboard to highlight issues and anomalies.
IBM Integration Bus is one of a number of IBM products supporting Product Insights, enabling you to implement your unique Bluemix service instances in which centralized product registration and usage information is securely stored and can be viewed with the provided dashboards.
In this post I will introduce these IBM Cloud hybrid services and provide the steps for exploiting IBM Cloud Product Insights with IBM Integration Bus 10.0.0.8.
Product Insights extends new cloud capability to IBM software products so that IT administrators or capacity planners can view, track, and report usage metrics in a single dashboard. Through this dashboard it is possible to track products across the enterprise, get usage reports to help optimize environments, and gain advice on relevant hybrid cloud services or Cloud patterns for those product.
IBM Integration Bus version 10.0.0.8 provides inbuilt support for exploiting IBM Cloud Product Insights with IBM Integration Bus so that you can simply set-up the reporting to the service.
You can visit the IBM Cloud Product Insights Technical Community to obtain details of the full set of IBM products offering support, the minimum product support levels, and getting started guides on how to configure and use the service and as well as provide your feedback.
Using Product Insights with Integration Bus means you can configure servers to report their startup and usage data to a service in the IBM cloud. You can then view through the Product Insights dashboard:
- Cross-product registration and inventory, including host and installation location and product version. Integration Bus also includes the fixpack level and reports any service APARs applied to each instance
- Usage Analysis, view product usage metrics graphically (displayed over time) to help analyze environments. Integration Bus provides both CPU and memory usage so that you can discover over/under-used instances to help improve ROI
- Advisor, Product Insights can recommend other Bluemix cloud services that can be connected to the specific registered IBM product, to improve value and ROI.
The product instance and usage information is securely stored and can only be viewed within the scope of your unique Bluemix service.
When you register Integration Bus instances you can optionally provide a group name enabling you to group instances into business or operational groups. The Product Insights UI will then enable you to view aggregated usage metrics graphically for these groupings, for example totaling CPU usage in your production and development/test systems
Bluemix Logmet beta service and Kibana dashboards
Using the Bluemix Logmet beta service, you can configure enabled IBM software products to send logging data to a centralized service in the IBM Cloud. You can then view these through a Kibana dashboard in Bluemix to gain insight into the operation status of systems over time.
The Logmet service in Bluemix is an IBM hosted implementation of the ELK stack. This is formed from a stack of three open source products; Elasticsearch, a clusterable search server; Logstash, a data pipeline technology used to ingest data; and Kibana, a flexible rapid development web browser based dashboard technology. Each enabled IBM product will provide a pre-built Kibana dashboard.
In this section I will detail how to enable IBM Integration Bus instances to report into a service instance of a Product Insights and then view all instances, a group of instances or a single instance in the service’s dashboard.
You can choose to send startup and usage information for a specified integration server, or for all integration servers on a specified integration node.
Complete the following steps to enable your IBM Integration Bus integration servers to report startup and usage information to the Product Insights service:
- If you don’t already have one, create an account in IBM Bluemix.
- If you don’t already have one, create your instance of the Product Insights service. Product Insights is located in Integration Services the Bluemix catalogue and your service instance list
- Within your Product Insights service dashboard you can access “Getting Started” information and on the “Manage” tab you can simply click the “Register a product” button to detail the two steps to configure Integration Bus.
The command templates can be copy-paste and edited to your instance values.
- Within your Product Insights service dashboard, move to the “Service Credentials” tab so that you can view and copy the “apiHost” and “apiKey” values which will be used to configure your Integration Bus to send data to this specific service instance
- Open an IBM Integration Bus console and use the mqsichangebluemixreporting command to configure a specific integration server, or all integration servers, to send startup and usage information to your Product Insights service instance.
For example, the following mqsichangebluemixreporting command specifies an integration node called TESTNODE, sets the reporting state to active, and sets the group name to “Test” for all integration servers
mqsichangebluemixreporting TESTNODE -g -c active -j https://productinsights-api.ng.bluemix.net -k e5fd5b7e-e52c-4227-9e58-cc5e2292e22b -n Test
Ensure that you use the apiHost and apikey value for your service instance in the “-j” and “-k” parameter value.
- Restart the integration server(s) for the changes to take effect, issue
You can use the mqsireportbluemixreporting command to view the current startup and reporting configuration for the integration server.
Once you have configured your integration servers to enable reporting and restarted them they will post information about the instance on each startup and at the configured reporting interval, which defaults to 15 minutes, the usage data. Note that to do this the integration servers require connectivity to the internet. If the server cannot send data, it will issue a BIP6502W, however the data be stored locally and added to the data send at the next interval or following a restart.
Now that your integration servers are configured to send data you instance of Product Insights service you can login to IBM Bluemix and select the relevant integration service instance.
The Product Insights “Manage” tab allows you to view all of the registered product instances. You can use the left hand column to select all instances of a product, or search / select a specific instance in the middle column.
While viewing all, or a group of registered product instances the right hand column “Details” tab is not populated, however the “Advisor” tab can be used to view information applicable to all selected instances. This includes viewing “Updates” which informs you of any instances which have a pending version or fixpack that can be applied.
When you select Integration Bus in the left column the right column provides you with a visualization of the aggregated usage data for all registered integration servers. (Note: Remember that the default reporting interval is 15 minutes, hence data will only become available after this period). In the “Usage” section you can view either:
- CPU time
This metric reports the CPU milliseconds consumed by the selected instances in the reported interval
- Resident Set Size
This metric reports the maximum physical memory consumed by the highest of the samples from of the instances during the reported interval
You can hover or select a point on the charts to highlight a particular point, for example in the above figure we can see that there was a peek in CPU and memory usage occurring at the samples taken at midday.
In order to drill into the usage you can either use the center column to select a specific integration server and view just it’s usage metrics, or if you grouped your integration servers using the “
-n” flag of
mqsichangebluemixreporting command select that group in the right hand column to view an aggregation of that group’s usage data.
Integration servers are automatically grouped under their owning integration node.
The Product Insights manage tab displays a breadcrumb banner to show the selection, for example:
All Products > IBM Integration Bus > DEV > IIBNODE_one
Would be displayed if you selected Integration Bus, then the registered group “DEV”, then the integration node “IIBNODE_one”.
With a single integration sever selected you can view:
- CPU time
- Number of active CPUs
The number of CPUs available to the process at the sample interval, CPU time may be consumed on one or more of these during the interval
- Resident Set Size
The default time duration for the visualization of the usage metrics is 24 hours, as highlighted in the following figure this can be changed to gain a usage pattern over longer periods.
Once an instance is selected the right hand panel switches to initially show the “Usage”. You can switch the right hand panel to “Details”, to view the details of the IBM product software and the host operating system environment. Integration bus provides the product version and details of any APARs applied, plus the install location and node and server names. Additionally it sends the URL of the web administration console enabling you to use the “Launch Console” action in to open a new browser tab to drill into the particular instance. For example you might enable message flow statistics or view activity logging data to understand a high or abnormal usage metric.
You can switch the right hand panel to “Advisor”, to view the recommendations for value-add services under “Services” or check for applicable updates under “Updates”. Note that the “Services” section is populated using Bluemix catalogue APIs to be dynamic adding newly available applicable services.
However also note that some Cloud offerings you might expect to see for Integration Bus are not identified as they do not currently fit this Bluemix catalogue API discovery model. For example
- IBM Integration Bus on Cloud
- IBM Integration Bus for Developers edition “ibm-integration-bus” image for IBM Bluemix Container Service.
- IBM App Connect, and the Integration Bus trigger pattern
Integration Bus Toolkit v10.0.0.8 provides a new tutorial “Using Bluemix Product Insights to view IBM Integration Bus Startup and Usage” that walks you though the setup and provides a sample application to drive a CPU load to enable you to observe the effect on the Product Insights CPU usage metric chart.
In this section I will detail how to enable IBM Integration Bus instances to publish their BIP message events into an instance of a Bluemix Logmet beta service and then view all instances in the provided IBM Integration bus Kibana dashboard. I will also provide references to information for customizing your own Kibana dashboards to enable you to tailor the display to your needs.
Integration Bus reports start, stop, configuration change informational and warning and error events know as “BIP messages“. Viewing these enables you to understand the operational status of your system and is essential in diagnosing any issues. These BIP messages are written to the operating system event log on each separate host running integration bus.
In Integration Bus version 10.0.0.8 each integration server provides an internal feature to enable it to publish it’s BIP messages to a centralized instance of the Logmet service in IBM Bluemix. The event data in this centralized Logmet instance can then be queried and the data visualized in the provided Integration Bus Kibana dashboard.
For each Integration Bus server, or all servers owned by a single integration node, that you want to publish BIP message events to the Logmet service in IBM Bluemix, perform the following steps to configure the server and load the IBM Integration Bus Kibana dashboard into your specific Bluemix space.
Note: When configuring an Integration Bus to use Logmet service in IBM Bluemix you need to provide Bluemix service credentials. Integration Bus will not store your Bluemix password, and it is only used within the the
mqsichangebluemixreporting command. If you do not want to provide the Bluemix service credentials directly see alternative manual configuration steps.
Open an Integration Bus console window and run the mqsichangebluemixreporting command from the host machine that is running your integration node to configure and activate the Logmet logging feature. For example, the following command specifies that all integration servers (as specified by the -g parameter) on integration node TESTNODE will report to the Logmet service in IBM Bluemix us-south region, for the user organization and space specified.
mqsichangebluemixreporting TESTNODE -g -l active -r us-south -o email@example.com -a dev -u firstname.lastname@example.org -p pass -d
You can set the logging status to active only if all of the following required parameters are set:
‘-r’ sets the Bluemix region for logging services. Valid values are ‘us-south’ or ‘eu-gb’.
‘-o’ sets the Bluemix organisation for logging services.
‘-a’ sets the Bluemix space for logging services.
‘-u’ sets the Bluemix username for logging services.
‘-p’ sets the Bluemix password for logging services.
‘-d’ indicates publish current IIB Kibana dashboard for logging services.
The successful command response will be similar to that show below:
Request Bluemix logging tokens: https://logmet.ng.bluemix.net/login
Successful HTTPStatus: 200
Publishing Kibana dashboard
Successful HTTPStatus: 200
Response: Your Kibana 4 dashboards can be accessed at the following location: https://logmet.ng.bluemix.net/app/#/kibana4
The Kibana 4 dashboards that have been uploaded include: IBM Integration Bus Dashboard
BIP8377I: Current dashboard has been loaded into Kibana for Bluemix logging space 'dev'. Dashboard files are also prepared for manual import into Kibana from 'C:\ProgramData\IBM\MQSI\BluemixReporting\TESTNODE'.
- Restart the integration node for the changes to take effect.
You can use the mqsireportbluemixreporting command to view the current logging service configuration for the integration server.
You are now able to view the configured Integration Server BIP message events in the Logmet service in IBM Bluemix by logging in and switching to the same region, organization and space as configured above using the following URLs:
- For the us_south region: https://logmet.ng.bluemix.net/app/#/kibana4/
- For the eu_gb region: https://logmet.eu-gb.bluemix.net/app/#/kibana4/
To open the Integration Bus Kibana 4 Dashboard, select the Kibana 4 tab, then the Dashboard tab and then click the “load saved dashboard” open folder icon and select the “IBM Integration Bus Dashboard”.
The IBM Integration Bus dashboard provides a set of charts that enable you to quickly identify the operation status of your integration servers by enabling you to view:
- The machines in your network which are producing the most BIP log events
- The integration servers in your network which are producing the most BIP log events
- The top 5 BIP Log event types
- Number of BIP log events charted against time and categorized by their severity
- Chart plotting stop, start and configure indication BIP events
- A tabular view of the detailed BIP events in the selected time period is also provided.
Note, if you get errors in a loaded Kibana dashboard, it could be that the Logmet index has not been rebuilt since event data from the integration server has been delivered. The events from IBM integration bus use a set of fields that are designed to be consistent with other enabled IBM products, for example
ibm_... fields need to be indexed. To reload the dashboard index pattern, ensure you have sent data events, then in the Kibana4 dashboard, click “Settings –> Indices” and select the existing index pattern. Click the reload icon to refresh the index pattern.
Optionally if you wish to use the Integration bus Kibana 3 dashboard.
Select the Kibana 3 tab. Click the “load” button then open the “advanced” link and browse to the folder containing your prepared IIB dashboard files, as reported when you ran the mqsichangebluemixreporting command response, and select the file “IIB_K3_dashboard_< Bluemix space>.json” file and import. Once the import completes the dashboard is opened. However you must then save it, by clicking the “save” disk icon button, to be able to reopen the next time you login. To re-open the Integration Bus Kibana 3 Dashboard click the “Load” open folder icon button and select “IBM Integration Bus Dashboard”.
Integration Bus Toolkit v10.0.0.8 provides a new tutorial “Using Bluemix Kibana dashboards to view IBM Integration Bus Logs” that walks you though the setup and provides a sample application to drive BIP event log messages which can then be viewed in the IIB Kibana dashboard.
Customizing Kibana dashboards
In addition to the IBM Integration Bus Dashboards a number of other IBM Products are providing support for publishing logs to Logmet service in IBM Bluemix and viewing though sample Kibana dashboards. For more details see the IBM Product Insights Beta Community .
You could also publish your own events into the service using technology such as Filebeat.
You can then use the simple facilities of Kibana to create customized dashboards that meet your specific needs to quickly visualize the status of your environments by combining log event data events using the provides sample dashboards as working references.
For a guide to developing Kibana dashboards in IBM Bluemix see here.
In this article, I have described how you can simply configure your inventory of integration servers and quickly gain insight into their usage and operational status by exploiting IBM Cloud Product Insights with IBM Integration Bus 10.0.0.8.