With the release of IBM Integration Bus version 10.0.0.8 you can now simply exploit Product Insights with IBM Integration Bus to view a centralized inventory of your integration servers, and access usage metrics and operational logs to quickly gain operational status insight and help optimize environments.

New functional content is provided in this fixpack release that enables you to configure integration servers to register startup and usage to IBM Cloud Product Insights enabling you to manage your inventory and identify servers that may needed to be optimized for better usage efficiency, view available version upgrades, and access information on relevant hybrid cloud services. Additionally integration servers can be configured to publish BIP log events to Bluemix Logmet beta service enabling them to be viewed centrally using a Kibana dashboard to highlight issues and anomalies.

Product Insights with IBM Integration Bus and Logmet Kibana dashboard
IBM Integration Bus in IBM Cloud Product Insights and Bluemix Logmet dashboards

IBM Integration Bus is one of a number of IBM products supporting Product Insights, enabling you to implement your unique Bluemix service instances in which centralized product registration and usage information is securely stored and can be viewed with the provided dashboards.

In this post I will introduce these IBM Cloud hybrid services and provide the steps for exploiting IBM Cloud Product Insights with IBM Integration Bus 10.0.0.8.

IBM Cloud Product Insights

Product Insights

Product Insights extends new cloud capability to IBM software products so that IT administrators or capacity planners can view, track, and report usage metrics in a single dashboard. Through this dashboard it is possible to track products across the enterprise, get usage reports to help optimize environments, and gain advice on relevant hybrid cloud services or Cloud patterns for those product.

IBM Cloud Product Insights Service
IBM Cloud Product Insights Service tile

IBM Integration Bus version 10.0.0.8 provides inbuilt support for exploiting IBM Cloud Product Insights with IBM Integration Bus so that you can simply set-up the reporting to the service.

You can visit the IBM Cloud Product Insights Technical Community to obtain details of the full set of IBM products offering support, the minimum product support levels, and getting started guides on how to configure and use the service and as well as provide your feedback.

Using Product Insights with Integration Bus means you can configure servers to report their startup and usage data to a service in the IBM cloud. You can then view through the Product Insights dashboard:

  • Cross-product registration and inventory, including host and installation location and product version. Integration Bus also includes the fixpack level and reports any service APARs applied to each instance
  • Usage Analysis, view product usage metrics graphically (displayed over time) to help analyze environments. Integration Bus provides both CPU and memory usage so that you can discover over/under-used instances to help improve ROI
  • Advisor, Product Insights can recommend other Bluemix cloud services that can be connected to the specific registered IBM product, to improve value and ROI.

The product instance and usage information is securely stored and can only be viewed within the scope of your unique Bluemix service.

When you register Integration Bus instances you can optionally provide a group name enabling you to group instances into business or operational groups. The Product Insights UI will then enable you to view aggregated usage metrics graphically for these groupings, for example totaling CPU usage in your production and development/test systems

Bluemix Logmet beta service and Kibana dashboards

Using the Bluemix Logmet beta service, you can configure enabled IBM software products to send logging data to a centralized service in the IBM Cloud. You can then view these through a Kibana dashboard in Bluemix to gain insight into the operation status of systems over time.
The Logmet service in Bluemix is an IBM hosted implementation of the ELK stack. This is formed from a stack of three open source products; Elasticsearch, a clusterable search server; Logstash, a data pipeline technology used to ingest data; and Kibana, a flexible rapid development web browser based dashboard technology. Each enabled IBM product will provide a pre-built Kibana dashboard.

Bluemix Logment service and Kibana dashboard

Exploiting Product Insights with IBM Integration Bus

In this section I will detail how to enable IBM Integration Bus instances to report into a service instance of a Product Insights and then view all instances, a group of instances or a single instance in the service’s dashboard.

Enabling


You can choose to send startup and usage information for a specified integration server, or for all integration servers on a specified integration node.

Complete the following steps to enable your IBM Integration Bus integration servers to report startup and usage information to the Product Insights service:

  • If you don’t already have one, create an account in IBM Bluemix.
  • If you don’t already have one, create your instance of the Product Insights service. Product Insights is located in Integration Services the Bluemix catalogue and your service instance list
  • Within your Product Insights service dashboard you can access “Getting Started” information and on the “Manage” tab you can simply click the “Register a product” button to detail the two steps to configure Integration Bus.
    Product Insights Integration Bus Registration
    Product Insights Integration Bus registration
    The command templates can be copy-paste and edited to your instance values.
  • Within your Product Insights service dashboard, move to the “Service Credentials” tab so that you can view and copy the “apiHost” and “apiKey” values which will be used to configure your Integration Bus to send data to this specific service instance
    Product Insights service credentials
    Product Insights service credentials
  1. Open an IBM Integration Bus console and use the mqsichangebluemixreporting command to configure a specific integration server, or all integration servers, to send startup and usage information to your Product Insights service instance.
    For example, the following mqsichangebluemixreporting command specifies an integration node called TESTNODE, sets the reporting state to active, and sets the group name to “Test” for all integration servers
    mqsichangebluemixreporting TESTNODE -g -c active -j https://productinsights-api.ng.bluemix.net -k e5fd5b7e-e52c-4227-9e58-cc5e2292e22b -n Test
    Ensure that you use the apiHost and apikey value for your service instance in the “-j” and “-k” parameter value.
  2. Restart the integration server(s) for the changes to take effect, issue
    mqsistop TESTNODE
    mqsistart TESTNODE

You can use the mqsireportbluemixreporting command to view the current startup and reporting configuration for the integration server.

Once you have configured your integration servers to enable reporting and restarted them they will post information about the instance on each startup and at the configured reporting interval, which defaults to 15 minutes, the usage data. Note that to do this the integration servers require connectivity to the internet. If the server cannot send data, it will issue a BIP6502W, however the data be stored locally and added to the data send at the next interval or following a restart.

Viewing

Now that your integration servers are configured to send data you instance of Product Insights service you can login to IBM Bluemix and select the relevant integration service instance.

The Product Insights “Manage” tab allows you to view all of the registered product instances. You can use the left hand column to select all instances of a product, or search / select a specific instance in the middle column.

Product Insights all products
Product Insights all products

While viewing all, or a group of registered product instances the right hand column “Details” tab is not populated, however the “Advisor” tab can be used to view information applicable to all selected instances. This includes viewing “Updates” which informs you of any instances which have a pending version or fixpack that can be applied.

Advisor showing pending updates for Integration Bus
Product Insights Advisor pending updates for Integration Bus

When you select Integration Bus in the left column the right column provides you with a visualization of the aggregated usage data for all registered integration servers. (Note: Remember that the default reporting interval is 15 minutes, hence data will only become available after this period). In the “Usage” section you can view either:

  • CPU time
    This metric reports the CPU milliseconds consumed by the selected instances in the reported interval
  • Resident Set Size
    This metric reports the maximum physical memory consumed by the highest of the samples from of the instances during the reported interval
Product Insights all Integration Bus usage
Product Insights all Integration Bus Usage

You can hover or select a point on the charts to highlight a particular point, for example in the above figure we can see that there was a peek in CPU and memory usage occurring at the samples taken at midday.

In order to drill into the usage you can either use the center column to select a specific integration server and view just it’s usage metrics, or if you grouped your integration servers using the “-n” flag of mqsichangebluemixreporting command select that group in the right hand column to view an aggregation of that group’s usage data.

Integration servers are automatically grouped under their owning integration node.

The Product Insights manage tab displays a breadcrumb banner to show the selection, for example:
All Products > IBM Integration Bus > DEV > IIBNODE_one
Would be displayed if you selected Integration Bus, then the registered group “DEV”, then the integration node “IIBNODE_one”.

With a single integration sever selected you can view:

  • CPU time
  • Number of active CPUs
    The number of CPUs available to the process at the sample interval, CPU time may be consumed on one or more of these during the interval
  • Resident Set Size

The default time duration for the visualization of the usage metrics is 24 hours, as highlighted in the following figure this can be changed to gain a usage pattern over longer periods.

Product Insights Integration Bus integration server selected
Product Insights Integration Bus integration server

Once an instance is selected the right hand panel switches to initially show the “Usage”. You can switch the right hand panel to “Details”, to view the details of the IBM product software and the host operating system environment. Integration bus provides the product version and details of any APARs applied, plus the install location and node and server names. Additionally it sends the URL of the web administration console enabling you to use the “Launch Console” action in to open a new browser tab to drill into the particular instance. For example you might enable message flow statistics or view activity logging data to understand a high or abnormal usage metric.

ProductInsights Integration Bus details
ProductInsights Integration Bus details

You can switch the right hand panel to “Advisor”, to view the recommendations for value-add services under “Services” or check for applicable updates under “Updates”. Note that the “Services” section is populated using Bluemix catalogue APIs to be dynamic adding newly available applicable services.

Product Insights Integration Bus Advisor
Product Insights Integration Bus Advisor

However also note that some Cloud offerings you might expect to see for Integration Bus are not identified as they do not currently fit this Bluemix catalogue API discovery model. For example

Tutorial

Integration Bus Toolkit v10.0.0.8 provides a new tutorial “Using Bluemix Product Insights to view IBM Integration Bus Startup and Usage” that walks you though the setup and provides a sample application to drive a CPU load to enable you to observe the effect on the Product Insights CPU usage metric chart.

Product Insights tutorial
Product Insights tutorial

Exploiting Bluemix Logmet service and Kibana dashboard with IBM Integration Bus

In this section I will detail how to enable IBM Integration Bus instances to publish their BIP message events into an instance of a Bluemix Logmet beta service and then view all instances in the provided IBM Integration bus Kibana dashboard. I will also provide references to information for customizing your own Kibana dashboards to enable you to tailor the display to your needs.

Integration Bus reports start, stop, configuration change informational and warning and error events know as “BIP messages“. Viewing these enables you to understand the operational status of your system and is essential in diagnosing any issues. These BIP messages are written to the operating system event log on each separate host running integration bus.

In Integration Bus version 10.0.0.8 each integration server provides an internal feature to enable it to publish it’s BIP messages to a centralized instance of the Logmet service in IBM Bluemix. The event data in this centralized Logmet instance can then be queried and the data visualized in the provided Integration Bus Kibana dashboard.

Enabling

For each Integration Bus server, or all servers owned by a single integration node, that you want to publish BIP message events to the Logmet service in IBM Bluemix, perform the following steps to configure the server and load the IBM Integration Bus Kibana dashboard into your specific Bluemix space.

Note: When configuring an Integration Bus to use Logmet service in IBM Bluemix you need to provide Bluemix service credentials. Integration Bus will not store your Bluemix password, and it is only used within the the mqsichangebluemixreporting command. If you do not want to provide the Bluemix service credentials directly see alternative manual configuration steps.

  1. Open an Integration Bus console window and run the mqsichangebluemixreporting command from the host machine that is running your integration node to configure and activate the Logmet logging feature. For example, the following command specifies that all integration servers (as specified by the -g parameter) on integration node TESTNODE will report to the Logmet service in IBM Bluemix us-south region, for the user organization and space specified.

    mqsichangebluemixreporting TESTNODE -g -l active -r us-south -o a.user@comp.com -a dev -u a.user@comp.com -p pass -d

    You can set the logging status to active only if all of the following required parameters are set:
    ‘-r’ sets the Bluemix region for logging services. Valid values are ‘us-south’ or ‘eu-gb’.
    ‘-o’ sets the Bluemix organisation for logging services.
    ‘-a’ sets the Bluemix space for logging services.
    ‘-u’ sets the Bluemix username for logging services.
    ‘-p’ sets the Bluemix password for logging services.
    ‘-d’ indicates publish current IIB Kibana dashboard for logging services.

    The successful command response will be similar to that show below:

    Request Bluemix logging tokens: https://logmet.ng.bluemix.net/login
    Successful HTTPStatus: 200
    Publishing Kibana dashboard
    Successful HTTPStatus: 200
    Response: Your Kibana 4 dashboards can be accessed at the following location: https://logmet.ng.bluemix.net/app/#/kibana4
    The Kibana 4 dashboards that have been uploaded include: IBM Integration Bus Dashboard
    BIP8377I: Current dashboard has been loaded into Kibana for Bluemix logging space 'dev'. Dashboard files are also prepared for manual import into Kibana from 'C:\ProgramData\IBM\MQSI\BluemixReporting\TESTNODE'.

  2. Restart the integration node for the changes to take effect.

You can use the mqsireportbluemixreporting command to view the current logging service configuration for the integration server.

Viewing

You are now able to view the configured Integration Server BIP message events in the Logmet service in IBM Bluemix by logging in and switching to the same region, organization and space as configured above using the following URLs:

To open the Integration Bus Kibana 4 Dashboard, select the Kibana 4 tab, then the Dashboard tab and then click the “load saved dashboard” open folder icon and select the “IBM Integration Bus Dashboard”.

The IBM Integration Bus dashboard provides a set of charts that enable you to quickly identify the operation status of your integration servers by enabling you to view:

  • The machines in your network which are producing the most BIP log events
  • The integration servers in your network which are producing the most BIP log events
  • The top 5 BIP Log event types
  • Number of BIP log events charted against time and categorized by their severity
  • Chart plotting stop, start and configure indication BIP events
  • A tabular view of the detailed BIP events in the selected time period is also provided.

IBM Integration Bus Kibana Dashboard

Note, if you get errors in a loaded Kibana dashboard, it could be that the Logmet index has not been rebuilt since event data from the integration server has been delivered. The events from IBM integration bus use a set of fields that are designed to be consistent with other enabled IBM products, for example ibm_serverName. These ibm_... fields need to be indexed. To reload the dashboard index pattern, ensure you have sent data events, then in the Kibana4 dashboard, click “Settings –> Indices” and select the existing index pattern. Click the reload icon to refresh the index pattern.

Optionally if you wish to use the Integration bus Kibana 3 dashboard.
Select the Kibana 3 tab. Click the “load” button then open the “advanced” link and browse to the folder containing your prepared IIB dashboard files, as reported when you ran the mqsichangebluemixreporting command response, and select the file “IIB_K3_dashboard_< Bluemix space>.json” file and import. Once the import completes the dashboard is opened. However you must then save it, by clicking the “save” disk icon button, to be able to reopen the next time you login. To re-open the Integration Bus Kibana 3 Dashboard click the “Load” open folder icon button and select “IBM Integration Bus Dashboard”.

Tutorial

Integration Bus Toolkit v10.0.0.8 provides a new tutorial “Using Bluemix Kibana dashboards to view IBM Integration Bus Logs” that walks you though the setup and provides a sample application to drive BIP event log messages which can then be viewed in the IIB Kibana dashboard.

Bluemix Kibana dashboards tutorial
Bluemix Kibana dashboards tutorial

Customizing Kibana dashboards

In addition to the IBM Integration Bus Dashboards a number of other IBM Products are providing support for publishing logs to Logmet service in IBM Bluemix and viewing though sample Kibana dashboards. For more details see the IBM Product Insights Beta Community .

You could also publish your own events into the service using technology such as Filebeat.

You can then use the simple facilities of Kibana to create customized dashboards that meet your specific needs to quickly visualize the status of your environments by combining log event data events using the provides sample dashboards as working references.

For a guide to developing Kibana dashboards in IBM Bluemix see here.

Conclusion - Exploiting IBM Cloud Product Insights and Logmet Kibana

IBM Cloud is providing Product Insights GA service and Bluemix Logmet beta service to use with your typically on-prem IBM software products so that IT administrators or capacity planners can view, track, and report usage metrics and operational logs from centralized dashboards. Through these dashboards it is possible to track products across the enterprise, get usage and operational logging reports to help optimize environments, and find information on relevant hybrid cloud services.

In this article, I have described how you can simply configure your inventory of integration servers and quickly gain insight into their usage and operational status by exploiting IBM Cloud Product Insights with IBM Integration Bus 10.0.0.8.

10 comments on"Exploiting IBM Cloud Product Insights with IBM Integration Bus 10.0.0.8"

  1. Hi Martin,

    I am using IIB10.0.0.9 and trying to do this from a RHEL6 box. I see some errors(json parsing errors) with my Product Insights logging and having an issue bypassing proxies for the logmet service. Can you help?

    BIP6501E: The integration server failed to perform Bluemix startup and usage reporting. Diagnostic information: ‘ParserException BIP5705E: ImbJSONParser::parseLastChild /build/S1000_slot1/S1000_P/src/DataFlowEngine/JSON/ImbJSONParser.cpp:257 “JSON parsing errors have occurred” [] ParserException BIP5701E: ImbJSONDocHandler::onInvalidCharacter /build/S1000_slot1/S1000_P/src/DataFlowEngine/JSON/ImbJSONDocHandler.cpp:550 “A JSON parsing error has occurred whilst parsing the JSON document” [1, 33565, 0x00000075, 0x00000018, “}”, “key”]’.

    Please let me know if you want me to send the logs and the errors I am seeing.

    • Hi
      Sorry to hear you are facing issues getting IIB setup and running with the Bluemix Product Insights services.
      I would suggest you consider raising a PMR though your IBM support channel as they will be able to guide you though the steps to gather required diagnostics etc.

      With regard to the JSON parse error you show, perhaps check if somehow your system has a corrupted file at $MQSI_WORKPATH/components///C2C_dialhomedata/[registration.json | usage.json].

      HTH
      Martin

      • Yes. I have openned a PMR for the logmet issue. Also, i found the usage.json in the location you mentioned. Do i delete it and rerun the mqsichangebluemixreporting command?

        • OK hopefully with the diagnostics from the service team we can get you up and running.
          Data which could not be delivered to Product Insights is held in that file for resend on the next interval. If the content is valid JSON it should be fine. Perhaps backup the file, and remove it to see if it clears the ParserException BIP5701E:. If it does also report that in the PMR.

  2. Hi Timm,

    Absolutely you can have all of your instances of IIB registered to report logging events into a single central Bluemix Logmet service.

    That said I suspect what you are currently hitting is a problem in the mqsichangebluemixreporting command which has recently arisen due to higher SSL encryption requirements when connecting to the Bluemix service to obtain the Logging token that the command needs to obtain for the IIB runtime to use.

    I have prepared a fix which should be available in the next IIB fixpack, and can be requested as APAR IT20533.

    In the meanwhile to get past this setup issue there are a couple options

    – Contact IBM support so we can provide a interim fix for IIB 10.0.0.8
    – Bypass the issue by using the alternative setup see “Configuring an integration server and loading the IBM Integration Bus Kibana dashboard manually” (https://www.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/bz91188_.htm). Note: Since you have one server already configured, you can use mqsireportbluemixreporting command to view the Logging token instead of having to run the manual “curl” command.

    • Timm Bryant May 09, 2017

      Thanks for the response. The manual process won’t work due to the following error: BIP8001E: Unknownn flag ‘f’ selected.

      I’ll open a PMR and request the interim fix.

      • Hi Timm

        The usage you need is

        mqsichangebluemixreporting {node} -g -l active -x
        -i {spaceIdentifier} -t {loggingToken}

        There is a typo on bz91188_ that will also be addressed in the next KC update.

        • Timm Bryant May 10, 2017

          Thank you. Your recommendation resolved the issue, and I am now successfully capturing data from two IIB nodes.

  3. Timm Bryant May 08, 2017

    I’m trying to register a second IIB Node to the Bluemix Logmet service and I keep getting a ‘java.net.SocketException: Connection reset’ error when I run the mqsichangebluemixreporting command. My first instance is working fine, I just cannot register a second instance from the same server.

    Is the Bluemix Logmet service limited to a single IIB Node?

Join The Discussion

Your email address will not be published. Required fields are marked *