Accelerate the value of multicloud with collaborative DevSecOps Learn more

Use Prometheus metrics to monitor your IBM Blockchain Platform network

Introduction

As a blockchain network operator, you probably have an IBM Blockchain Platform network up and running in IBM Cloud, and you might have even played around with a Sysdig dashboard to monitor your nodes. Hyperledger Fabric v1.4 introduced the Operations Service, an API that exposes a Prometheus target for a set of operational metrics that you can use to monitor and track the status of your network components.

Your IBM Blockchain Platform for IBM Cloud network has already exposed the operational metrics for the peer and ordering nodes. How do you leverage this capability in your Sysdig dashboard? What are some useful insights to get started? This tutorial walks you through the steps required to configure a Sysdig dashboard to include the Prometheus metrics to monitor your IBM Blockchain network. It focuses on the peer metrics, but you can extend the same process to ordering nodes as well. It is meant to be an introduction to using the Prometheus metrics so that you can get started exploring how you can use them to fit your own scenario.

Prerequisites

  • An account in IBM Cloud with an instance of the IBM Blockchain Platform and an instance of IBM Cloud Monitoring with Sysdig.

  • If you haven’t used Sysdig with IBM Blockchain Platform before, check out this tutorial on IBM Cloud Monitoring with Sysdig to set one up.

  • Because the tutorial uses the IBM Cloud CLI to configure the Prometheus metrics in Sysdig, you need to log in to your IBM Cloud Kubernetes cluster from the command line.

  • You should be familiar with the Hyperledger Fabric Operations Service and the Prometheus metrics that are available for monitoring.

  • You should have already deployed peers and ordering service nodes on your IBM Blockchain Platform. Check out the “Build a network” tutorial series if you need help getting set up.

Estimated time

The estimated time to complete this tutorial is about one hour.

Steps

  1. Create an identity that Sysdig can use to access the nodes
  2. Configure the TLS secret
  3. Configure the Prometheus settings
  4. View the Prometheus metrics in the Sysdig dashboard
  5. Use the Prometheus metrics in your Sysdig dashboard
  6. Use the Prometheus metrics to monitor an ordering node

After you complete the following five steps, you will have a working Sysdig dashboard configured with two Prometheus metrics for monitoring peers. The sixth step describes what you would do differently if you wanted to monitor ordering nodes instead of peers.

1

Create an identity that Sysdig can use to access the nodes

Sysdig uses mutual Transport Layer Security (TLS) to access the nodes in your cluster, so you need to use your organization’s TLS certificate authority (CA) to generate a private and public key. It’s the same process you followed when you registered and enrolled identities for your peer nodes, but this time you need to use the TLS CA instead.

When you open your peer’s CA in the console, click Register user to create a new user that will be used to access your blockchain network from Sysdig. Specify the enroll ID and secret for this user and set the type to client. After you add the user, click Enroll identity on the user’s action menu to generate the certificate and private key for that user.

Figure 1. Register and enroll a user for Sysdig

Important: Be sure to select the TLS Certificate Authority from the Certificate Authority drop-down list.

Figure 2. How to select the TLS CA

Enter the enroll ID and secret that you specified when you created the user in the previous step. When you click Next, the subsequent panel contains the generated certs that are required for Sysdig.

Figure 3. Download the certificate and private key

You’ll need the certificate and private keys in the next step, so download them to your file system and then click Add identity to Wallet to store these keys in your console wallet for safekeeping.

Tip: The IBM Blockchain Platform console does not store these certificates; you are responsible for managing them. When you close your browser, these keys are not saved. If you need to use them again (from a different browser, for example), you need to import the identity into your console wallet on that browser. Therefore, we recommend that you click Export identity to save the identity to your file system so that you can import it later as needed.

2

Configure the TLS secret

When you configured Sysdig in your Kubernetes cluster, the ibm-observe namespace was created in your Kubernetes cluster. You need to edit the associated daemonset and configmap to configure them for Prometheus, but first, you need to store the TLS certs that you downloaded in a Kubernetes secret. (Run these commands from the same IBM Cloud CLI that you used when you configured Sysdig.)

  1. Create a TLS secret in the ibm-observe namespace by running the following, where <cert-path> and <key-path> is the path to the certs you downloaded to your local file system. Make a note of the <secret-name> that you use.

    kubectl create secret tls <secret-name> --cert=<cert-path> --key=<key-path> -n ibm-observe
    
  2. When Sysdig is deployed to your cluster, it creates a daemonset within the ibm-observe namespace. Edit this daemonset by running:

    kubectl edit daemonset -n ibm-observe sysdig-agent
    
  3. Scroll down to spec.template.spec.containers.volumeMounts and add a volume mount for your secret:

    - mountPath: /<mount-path>
      name: <volume-name>
    

    For example:

    - mountPath: /ibp
      name: ibp-volume
    
  4. Then find spec.template.spec.containers.volumes and add your secret to the volume, specify any value for the <volume-name>, and enter the name of the secret you created above in the ibm-observe namespace.

    - name: <volume-name>
      secret:
        defaultMode: 420
        secretName: <secret-name>
    

    For example:

    - name: ibp-volume
      secret:
        defaultMode: 420
        secretName: sysdigtls
    
3

Configure the Prometheus settings

Almost done! Now you can edit the ibm-observe namespace configmap by running:

kubectl edit configmap -n ibm-observe sysdig-agent
  1. Find items.data.dragent.yaml.prometheus.
  2. Change it to the following, paying close attention to indentation:

    prometheus:
      enabled: true
      # Helpful for determining why the metrics aren't being detected
      log_errors: true
      process_filter:
      # Exclusions avoid any Prometheus metrics other than the ones included from being detected
      - exclude:
          process.name: docker-proxy
      - exclude:
          container.image: sysdig/agent
      - exclude:
          appcheck.match: prometheus
      # Includes all Kubernetes pods that match the specified label (name in this case)
      - include:
          kubernetes.pod.label.orgname: <peer-organization-msp-id>
          conf:
              use_https: true
              auth_cert_path: "<mount-path where secret is located>/tls.crt"
              auth_key_path: "<mount-path where secret is located>/tls.key"
    

    Replace <peer-organization-msp-id> with the msp ID of your peer’s organization, and the “<mount-path where secret is located>/tls.key” with the value you specified in the daemonset above, for example, /ibp.

Configuration is complete. Now, let’s look at the metrics in the Sysdig dashboard.

4

View the Prometheus metrics in the Sysdig dashboard

After you edit the configmap, it can take several minutes for the Sysdig dashboard to reflect the metrics. You’ll know the configuration worked by clicking the Explore tab in the dashboard. From Hosts & Containers, click Entire Infrastructure. Click the Overview by Host twistie and scroll down to see all the Prometheus metrics.

Figure 4. Prometheus metrics visible in the Sysdig dashboard

Once you can see the Prometheus metrics as shown here, Sysdig and the Prometheus metrics are successfully configured.

5

Use the Prometheus metrics in your Sysdig dashboard

You can always check out the Hyperledger Fabric Metrics Reference guide for more information about what each Prometheus metric captures. To get started, there are a few metrics that are useful for assessing the state of your peers. For this tutorial, we examine ledger_blockchain_height and couchdb_processing_time.

ledger_blockchain_height

We’ll start by assessing the ledger height of the peers in the organization. From the list of Prometheus metrics, choose ledger_blockchain_height. Click the AVG drop-down menu next to the metric and ensure that the metric aggregations are set to Maximum instead of Average. In the Segment by drop-down list, select kubernetes.pod.label.name. When you click on the line in the graph, you get a breakdown of ledger height by peer. This is a useful view when you want to ensure that all peers in the organization are keeping up with the transactions, a potential indicator of peer health. Use this chart to identify any peers whose ledgers are falling behind.

Figure 5. Metric: ledger_blockchain_height by peer

This chart was created by adding the metric to a new Blockchain – Prometheus Metrics dashboard. (If you followed the “IBM Cloud Monitoring with Sysdig” tutorial in the Prerequisites, the process for creating a new Dashboard is described there.) Dashboards are a useful place to save any customizations you make to a chart and easily see the data on-demand later. Simply create a new Sysdig dashboard and click the + sign to add a new panel where you can select the metric and segmentation type.

Tip: If your data is not showing up on the graph, there is a gray timeline at the bottom of the page that you can use to adjust the time range and even set a custom range:

Figure 6. Sysdig dashboard timeline

couchdb_processing_time

Let’s explore the couchdb_processing_time metric. Repeat the same process that we used for adding the ledger_blockchain_height metric, but this time select the metric couchdb_processing_time. Again, we segment the data by kubernetes.pod.label.name. The resulting graph is useful for tracking the average amount of time in seconds that CouchDB takes to respond to ledger requests across your peers, exposing untuned queries.

Figure 7. Tracking time in seconds

6

Use the Prometheus metrics to monitor an ordering node

If you are more interested in monitoring your ordering nodes, the process for configuring the monitoring of them is largely the same as monitoring peer nodes:

  • In Step 1, when you register and enroll the user, you need to use the CA for the ordering node instead of the CA for the peer node.

  • In Step 2, create a Kubernetes secret for the ordering node TLS CA certificates.

  • In Step 3, edit the daemonset and add the ordering node TLS CA certificate and key as described. When you edit the configmap, you need to specify the msp ID of the orderer organization for the kubernetes.pod.label.orgname parameter instead of the msp ID of the peer organization.

You are now ready to monitor your ordering node using the Prometheus metrics.

Summary

In this tutorial, you configured your Sysdig dashboard to work with the Prometheus metrics to monitor peers or ordering nodes in your IBM Blockchain Platform network. With the configuration complete, you can now experiment with the other Prometheus metrics to gain insights and debug problem areas in your network. If you have not already completed the IBM Cloud Monitoring with Sysdig tutorial for monitoring your node resource allocation, you should do that now.

When you combine the insights from that dashboard with your metric dashboard you begin to form a more holistic picture of how your network is responding to the transaction requests. You can then check out the Hyperledger Fabric Metrics Reference guide to find additional metrics for your dashboard to delve deeper into the node activity on your blockchain network.