Sending IZDS or IZPCA data to ELK - Part 2

Installing ELK on Linux

Download ELK

Download the supported version of Elasticsearch, Logstash and Kibana:

https://www.elastic.co/downloads

At the moment IZPCA supports ELK version 7.8.1, so I downloaded these files in folder /opt/ELK-7.8.1

• elasticsearch-7.8.1-linux-x86_64.tar.gz
• kibana-7.8.1-linux-x86_64.tar.gz
• logstash-7.8.1.tar.gz

Requirements

Elastic Stack is running with bundled version of OpenJDK (Java) which is recommended way. If you require your own Java, you have to set JAVA_HOME variable. If you already have JAVA_HOME set and it is not supported (not working), then for the user that will run ELK, set:

# export JAVA_HOME=

Install Elasticsearch 

Unzip Elasticsearch

# cd /opt/ELK-7.8.1/
# tar -xzf elasticsearch-7.8.1.tar.gz
# cd /opt/ELK-7.8.1/elasticsearch-7.8.1

 This directory is known as $ES_HOME

Configure Elasticsearch

This step is optional if you will have all components on the same system (Elasticsearch, Logstash, Kibana and you are suing localhost.

Edit $ES_HOME/config/elasticsearch.yml

cluster.name: elk-rhel72
network.host: rhel72.ibm.com
discovery.seed_hosts: ["rhel72.ibm.com"]cluster.initial_master_nodes: ["node-rhel72"] 

That is also the place where you can change the port for Elasticsearch. The default is: 9200.

The default network.host or hostname is localhost or 127.0.0.1 for all Elastic Stack component, so if you are not exposing them to external machine, it is not required to change network setup.

Installing Kibana 

Unzip Kibana

# cd /opt/ELK-7.8.1/
# tar -xzf kibana-7.8.1-darwin-x86_64.tar.gz
# cd /opt/ELK-7.8.1/kibana-6.6.0-linux-x86_64

This directory is known as $KIBANA_HOME

Configuring Kibana

Open config/kibana.yml in an editor

Set elasticsearch.hosts to point at your Elasticsearch instance, if on different host or if any of the components will be exposed to external machine, such as Problem Inisght server

#server.port: 5601
server.host: "rhel72.ibm.com"
elasticsearch.hosts: ["http://rhel72.ibm.com:9200"] 

Installing Logstash 

 Unzip the files

# tar -xzf logstash-7.8.1.tar.gz
# cd /opt/ELK-7.8.1/logstash-7.8.1/

Configuration

We will configure it later in Configuring Logstash step

Downloading and installing the IZPCA dashboard files

The files tha you have to download are part of the SMP/E installation IZPCA filesystem: /usr/lpp/IBM/IZPCA/v3r1m0/IBM

These files are tar files, When you extract the content of them you get:

DRLWING (Ingestions)

• B_IZDS_Input.lsh
• E_IZDS_Filter.lsh
• Q_IZDS_Output.lsh

DRLWECPA (IZPCA reporting package)

• IZDS_Indexes.ndjson
• IZDS_Visualizations.ndjson
• IZDS_Dashboards.ndjson

You can extract this file with tar -xvfo source -C destination on z/OS and ftp in binary to target platform or you can ftp tar files in binary and extract them on the target platform

Let’s assume the files are extracted in /opt/ELK-7.8.1/IZPCA directory.

Configuring Logstash

Place Ingestions files in logastash configuration directory, for example izpca-config

# su – virtuser
# mkdir /opt/ELK-7.8.1/izpca-config
# cp /opt/ELK-7.8.1/IZPCA/*.lsh /opt/ELK-7.8.1/izpca-config

Edit Pipelines.yml to add izpca pipeline

- pipeline.id: izoa
#   queue.type: persisted
  path.config: "/opt/ELK-7.8.1/zoa-config/*.conf"
- pipeline.id: izpca
#   queue.type: persisted
  path.config: "/opt/ELK-7.8.1/izpca-config/*.lsh"
 

Edit B_IZDS_Input.lsh and Q_IZDS_Output.lsh

Set the correct port in B_IZDS_Input.lsh:

input {
   tcp {
    port => 8082
    codec => json
    tags => [ "izds_cdpz_tcpip" ]  }
} 

Set the correct name in Q_IZDS_Output.lsh:

output {
  if "izds_cdpz_tcpip" in [tags] or "izds_cdpz_file" in [tags] or "izds_raw_file" in [tags] {
    elasticsearch {
      hosts => ["rhel72.ibm.com:9200"]      document_id => "%{id}"
      index => "%{TABLE}-%{+YYYY.MM}"
    }
  }

Starting Elasticsearch and Kibana

ELK has to run under the user different than root, for example virtuser.

 So, with root, change the ownership of the folder where ELK is installed:

# chown -R virtuser /opt/ELK-7.8.1

Logon with virtuser

Start Elasticserach:

$ /opt/ELK-7.8.1/elasticsearch-7.8.1/bin/elasticsearch

Start Kibana:

$ /opt/ELK-7.8.1/kibana-7.8.1-darwin-x86_64/bin/kibana

To test Kibana, point your browser at http://hostname:5601

Configuring Kibana

Sign into Kibana if authentication is enabled.

http://hostname:5601

Navigate to Management, then select Saved Objects.

Import the provided IBM Z Performance and Capacity Analytics newline-delimited JSON files in the following sequence:

1. IZDS_Indexes.ndjson
2. IZDS_Visualizations.ndjson
3. IZDS_Dashboards.ndjson

Configuring Kibana Homepage

This is IZPCA Kibana Homepage:

http://hostname:5601/app/kibana#/dashboard/d4d23d00-f0bb-11ea-a417-2f876e2492bd

Starting Logstash

Logon with the user different from root, for example virtuser:

Start Logstash:

$ /opt/ELK-7.8.1/logstash-7.8.1/bin/logstash

Sample reports

LPAR 4 HOUR MSU Utilization by Day

LPAR MSU Hourly Comparison by Day of Week

Next steps

Now, you can check Part 1 where I described how to implement the Shadower to send IZDS or IZPCA data from Db2 database to IBM Z Common Data Provider.