The Common Data Provider for z Systems (CDPz) provides a common infrastructure for accessing operational data from z/OS systems in a consumable format to various analytics platforms. Operational data can be gathered in near-real time or in batch. Various SMF and log data can be read once (for example SMF 30 records or SYSLOG data), and then sent to multiple consumers such as IBM Operations Analytics for z Systems and other on- or off-platform analytics solutions. The Common Data Provider for z Systems uses a simple web-based configuration tool for configuration, which is supplied as a plug-in to IBM z/OS Management Facility (z/OSMF).
CDPz recently released PTFs in the service stream that provide a number of major enhancements and new capabilities that broaden the scenarios in which CDPz can be used
- Exposed parts of System Data Engine (SDE) collector language
- Streaming IMS user log records to your analytics platform
- Filtering a System Data Engine (SDE) data stream
Following are a bit level deeper introduction on these topics:
Exposing part of the System Data Engine (SDE) collector language
The central part of SDE is the data collector that reads and processes SMF records from various sources including SMF user exit, SMF in-memory resource or SMF logstream, and streams the resulting data to analytics platforms. The data collector is controlled by the instructions written in the SDE collector language. SDE exposes key parts of the collector language that customers can use to stream their own data or customize the data streams delivered by CDPz.
- Use the DEFINE RECORD statement to define the layout of the SMF record or IMS user log record.
- Use the DEFINE UPDATE statement to instruct SDE to process a specific record type and storing the resulting data into the target.
- Optionally, use the DEFINE TEMPLATE statement to select a subset of the fields in the original record.
Refer to the Knowledge Center for the syntax and description of each of the collector language statements.
As the complement to the SDE collector language, the Configuration Tool is enhanced to support the creation of user data streams.
Streaming IMS user log records to your analytics platform
By using the SDE collector language, you can extend the CDP to stream IMS user log records to analytics platforms.
SDE delivers data streams for a number of IMS log record types including program accounting records (type xâ€™07â€™ and xâ€™08â€™), transaction level statistics (type xâ€™56FAâ€™), security violation records (type xâ€™10â€™), etc. To steam your own IMS user log records, you must use the SDE collector language statements to define your own record definition, update definition and optionally template definition. In addition, you use the Configuration Tool to create the corresponding data stream for your update definition. You can then create or update the policy in the Configuration Tool to include your user data streams in the same way as for the data streams delivered by CDPz.
To configure CDPz to stream an IMS user log record you use the DEFINE RECORD statement to map the layout of the new record:
When the IMS user log record is written to SMF in the IMS Logger user exit, the CDPz will pick up and process the IMS user log record and stream it through CDPz.
CDPz provides detailed step-by-step guidance on how to stream IMS user log records into your analytics platform. The detailed procedure can be found here:
Filtering a System Data Engine (SDE) data stream
You can use the SDE collector language to filter SDE data streams to decrease the volume of data sent to the subscribers. This is extremely useful for analytic platforms such as Splunk which charge based on the amount of data ingested. Ingesting only the data which you need to process/report on in Splunk will result in significant savings.
To filter you create your own update definition based on the existing one for the data stream you want to filter. Optionally, you can define a template definition for your update definition.
To do filtering at the record level to specify which records to collect, code WHERE conditions in your update definition to limit the update to process only those source records for which the condition is true.
The following is an example of record level filtering which performs filtering on the SMF type 101 subtype 1 record (DB2):
Based on the record filter, the following table indicates which of the records will actually be collected and processed by CDPz:
To do filtering at the field level, you can define a template definition for your update definition to stream only selected fields in the source record which is useful to support use cases such as:
- Creating additional output files with the same data, or a subset of the data
- Changing the order of the fields in the output file
The following template definition only sends 5 fields from the original record to the subscriber:
Which results in the filtered record streamed through CDPz to contain only the fields which were identified in the template definition.
Additionally you create the corresponding user data stream for your new definition in the CDPz Configuration Tool, then you create or update the policy in the Configuration Tool to include your user data stream in the same way as data streams delivered by CDPz.
CDPz provides detailed step-by-step guidance on how to filter a SDE data stream. The detailed procedure can be found here:
For additional information on whatâ€™s new in the CDPz 2Q 2018 PTF see https://www.ibm.com/support/knowledgecenter/SSGE3R_1.1.0/cdp_v110.html#cdp_v110
For additional information on IBM Common Data Provider for z Systems see http://ibm.biz/CDPzInfo