In addition to the data streams that IBM Common Data Provider for z Systems (CDPz) provides to process and stream the SMF records produced by IBM products and several third-party products, you can extend the CDPz functionality to stream your own SMF records and application data.
This ability provides you the option to stream your applicationâ€™s structured data to analytics platforms (Logstash or Splunk) via CDPz. You implement the code to write your applicationâ€™s structured data to SMF with the application structured data as the payload in the SMF record and define the CDPz artifacts to support your application structured data definition. CDPz then picks up the special SMF record and processes your application structured data to stream it to the appropriate subscribers.
To stream your applicationâ€™s structured data via CDPz, the main tasks you will need to do include:
- If your applicationâ€™s structured data is non-SMF data, you need implement the code to write your applicationâ€™s structured data via a special SMF record.
- On System Data Engine (SDE), create record definitions, update definitions and optionally template definitions for your special SMF record.
- On Configuration Tool, create the corresponding user data streams for your update definitions. Once the user data streams are defined, you can create or update the policy for System Data Engine (SDE) and Data Streamer to include the user data streams in the same way as data streams delivered by CDPz.
- On analytics platforms (Logstash or Splunk), update the configuration files to support the new data source types.
CDPz support team issued a technote that provide the step by step guide to stream your applicationâ€™s structured data to analytics platforms. See this technote for more information.