Introducing direct Splunk ingestion with IBM Common Data Provider for z SystemsThe IBM Common Data Provider for z Systems team provides a new method for getting your business-critical data into the Splunk platform. Normally, to make Z data easily ingestible by the Splunk platform, customers must configure additional components, such as the IBM Common Data Provider for z Systems Data Receiver and Buffered Splunk Ingestion App. With the new ingestion method, these additional components are no longer necessary. Although the new method provides quicker configuration, it is not meant to supersede previous methods. It is meant only to provide another option for data ingestion, depending on the needs and requirements of your IT environment. This paper describes more about the new method, and when you might want to choose this option.
HTTP Event Collector (HEC): the Splunk component that makes direct ingestion possibleFirst, let’s talk about the component that makes this new method possible. Splunk 6.3.0 includes a new feature, known as the HTTP Event Collector (HEC). This feature enables an application to communicate with Splunk over HTTP or HTTPS. The HEC uses a token-based authentication model, which allows users to create an Event Collector token for an application, such as IBM Common Data Provider for z Systems. This 32-character globally unique identifier (GUID) token (or tokens) enables IBM Common Data Provider for z Systems to securely communicate with your Splunk instance over HTTP or HTTPS.
Why send data by using the HEC?Here are some primary advantages of sending your data directly to Splunk by using the HEC:
- Your Mainframe and Splunk teams can work together and quickly get IBM Common Data Provider for z Systems deployed end-to-end, without the need for a distributed team to configure the Data Receiver or update the Buffered Splunk Ingestion App. The deployment time is streamlined by reducing the teams involved. Simply configure your host-based components, create an Event Collector token, and send your data directly to Splunk.
- The HEC provides an easy interface to manage and control the Event Collector tokens that you create. You can set the indexers that you want to be associated with each token. You can also enable or disable any token that you create to control what data is sent by using HEC. Token-based inputs also enable you to segregate, and search on, a specific input token.
Important if you have custom dashboards: If you currently have any custom dashboards and want to display both CSV and key-value data, you must update the underlying queries. The new data format appends _KV at the end of the source type field, which enables both CSV and key-value data to coexist. For any search query that uses sourcetype in the query stanza, you must add an OR condition. For example, if the query code includes "sourcetype = SMF_030", you now must now have the following query code: "sourcetype = SMF_030 OR sourcetype = SMF_030_KV”. If you are just collecting key-value data then all you need to have in your custom queries is