DevSecOps Solutions - Group home

IBM Z Decision Support has recently released an update that contains two significant enhancements.
1) The first is the ability to stream the data it gathers to user-selected subscribers through the IBM Common Data Provider for z Systems. This enhancement is discussed in this blog
2) The second is a set of customization for Splunk to accept IBM Z Decision Support data streamed through the IBM Common Data Provider and a set of Dashboards that replicate some of the Key Performance Metric reports.
This blog covers the second enhancement.

KEY PERFORMANCE METRICS FOR Z

One of the concerns customers face when using IBM Z Decision Support is that it can gather an awful lot of data about the performance and behavior of their Z Systems. To realize the true value of IBM Z Decision Support you must understand this data and work to extract what is truly significant from this vast sea of data.
Key performance Metrics (KPMs) were introduced in IBM Tivoli Decision Support 1.8.2 to assist in improving the time to value for this data.
To produce the KPMs, the development team worked with subject matter experts to determine which bits of SMF data would be of the most value to customers who were trying to understand how well their systems were running.
This expert knowledge was encoded in the form of some additional aggregation tables and reports. The aggregation tables gathered a focused set of data from SMF and the reports presented a subset of that data in a way that highlighted significant information. Configuring IBM Z Decision Support to gather the data required for the Key Performance Metrics will therefore let you gather a small set of meaningful data and more easily extract the true value of that data.

KPM SPLUNK DASHBOARDS

The Key Performance Metrics reports that have been ported to Spunk include the following.
KPMZLP02 – LPAR CPU UTILIZATION, HOURLY
For a given CPC, this report shows the percentage of it’s total CPU that was consumed by each LPAR as well as the total CPU utilization of the CPC.



KPMZLP02: LPAR CPU utilization hourly

The data is also presented in a tabular format below the chart



KPMZLP02 – LPAR CPU Utilization, hourly table

KPMZLP05 – LPAR 4 HOUR MSU UTILIZATION, HOURLY

This dashboard shows the 4 Hour Rolling Average value for MSU Utilization, as reported by RMF. It should be noted that while these values are part of the input into SCRT, the results may not be consistent with a SCRT report as they can be subject to mathematical manipulation, based on pricing schemes before the final SCRT values are arrived at.
 

KPMZLP05 – LPAR 4 Hour MSU utilization, hourly

KPMZAS02 – JOB MAXIMUM CPU TASK, DAILY

This report shows the programs, jobs and jobnames that have consumed the most CPU during the day.
 

KPMZAS02 – JOB MAXIMUM CPU TASK, DAILY

KPMZWK02 – WORKLOAD SERVICE CLASS UTILIZATION, HOURLY

This shows the CPU consumed by each service class within a given workload definition for each hour during the day.
 

KPMZWK02 – WORKLOAD SERVICE CLASS UTILIZATION, HOURLY

There are several other dashboards that are provided for Splunk and it is expected that this number will increase with time.

SO, HOW DO YOU GET TO SEE THEM?

The first step is to follow the instructions in this blog and set up IBM Z Decision Support to stream data through the IBM Common Data Provider for z (CDPz).
The second step is to configure Splunk.
 

Figure 6 Splunk configuration

The basic flow is to set the CDPz Data Receiver up to receive the files and write them out to a directory. The IZDS ingestion app will then direct Splunk to read them in and store the data in the izds_data index. Data model definitions then map the data into tables, and the dashboards draw on the Data models for the data they present.
INSTALLING THE IBM COMMON DATA PROVIDER FOR Z DATARECEIVER
Follow the instructions in the CDPz User Guide to install and configure the DataReceiver. Be sure to set the CDPDR_PATH environment variable to point to the directory where the DataReceiver will write the data it receives from the host systems. This variable will be used by the IZDS ingestion App to locate the data.
INSTALL SPLUNK
If you don’t have it already installed, download a copy of Splunk Enterprise and install it. Note that the trial license is only good for 500MB a day.
CREATE AN INDEX TO HOLD THE IBM Z DECISION SUPPORT DATA.
You need to create an index to hold the streamed data. This index is called izds_data in the ingestion app and the data model definitions. If you wish to change it, you will need to modify the apps after you have installed them.
For simple Splunk set up, just go to Settings -> index and create a new index using the defaults.
The most useful way to distinguish data in the index is by the TABLE variable – it is set to izds_ followed by the Db2 table name in lower case. For example, the KPMZ_LPAR_H data records all have a TABLE variable set to izds_kpmz_lpar_h. The variable name is TABLE, all in upper case.
DOWNLOAD THE IBM Z DECISION SUPPORT APPLICATIONS FOR SPLUNK
The apps are SMP/e installed as files into the SDRLIDM target library. The default location is /usr/lpp/IBM/IDSz/v1r9m0/IBM.


The apps need to be installed through Splunk’s browser based UI as follows:
1. Do a binary download of the required parts to a location accessible to the Splunk browser UI and rename each part to the App File Fame as listed in the table above.
2. Start the Splunk browser UI and click the setting cog wheel near Apps on the left side of the UI.
3. On the top right click Install app from file.
4. Pick the Data Owner app first from the browse button and install it.
5. Click Restart Later and repeat step 3 to install the other required apps.
When you have installed all of the apps, click Restart Now and wait for Splunk to restart.
When you sign into Splunk the Applications and their dashboards will be available. There won’t be any data available in the dashboards yet.

IBM Z DECISION SUPPORT APPLICATIONS

THE IBM Z DECISION SUPPORT KEY PERFORMANCE METRICS FOR Z APPLICATION
This contains most of the dashboards that are provided – Address Space, CEC, LPAR, Workload and Storage.
THE IBM Z DECISION SUPPORT KEY PERFORMANCE METRICS FOR CICS APPLICATION
This contains a single Key Performance Metrics for CICS dashboard – KPMCTM01 – CICS Transaction performance – which can be used to evaluate the performance of your CICS transactions.
THE IBM Z DECISION SUPPORT DATA OWNER APPLICATION
This doesn’t contain any dashboards, but it does contain the Data Models for the IBM Z Decision Support data that has been loaded into Splunk. Using these Data Models, you can write your own ad hoc reports to further explore and refine the data.

GET THE DATA FLOWING

Assuming you have followed the instructions in the first blog and have your collected IBM Z Decision Support data already flowing to the IBM Common Data provider for z Data Streamer, the final step is to define the DataReceiver you installed for Splunk as a subscriber and assign it some data.
The IBM Common Data provider for z User Guide covers defining the subscriber, but it is, essentially, just the host name of the machine it is running on, it’s port number and the CDP Data Receiver communications protocol.
The next step is to pick the tables you need to stream to Splunk. To make all the Dashboards work you need:
• KPMC_TRAN_MON_H (for CICS)
• KPMZ_LPAR_H (For the CEC and LPAR dashboards)
• KPMZ_STORAGE_H (For the storage usage dashboard)
• KPMZ_WORKLOAD_H (For the workload and service class dashboards)
• KPMZ_JOB_INT_T (For the address space dashboards)

Once you have them defined, transcribed to UTF-8 and linked to your Splunk receiver you can rebuild your CDPz Policy and then Activate it on the CDPz instance running on your IBM Z Decision Support Hub system.

The data will start to flow.