As a blockchain network operator, you probably have an IBM Blockchain Platform network up and running in IBM Cloud, and you might have even played around with a Sysdig dashboard to monitor your nodes. Hyperledger Fabric v1.4 introduced the Operations Service, an API that exposes a Prometheus target for a set of operational metrics that you can use to monitor and track the status of your network components.
Your IBM Blockchain Platform for IBM Cloud network has already exposed the operational metrics for the peer and ordering nodes. How do you leverage this capability in your Sysdig dashboard? What are some useful insights to get started? This tutorial walks you through the steps required to configure a Sysdig dashboard to include the Prometheus metrics to monitor your IBM Blockchain network. It focuses on the peer metrics, but you can extend the same process to ordering nodes as well. It is meant to be an introduction to using the Prometheus metrics so that you can get started exploring how you can use them to fit your own scenario.
If you haven’t used Sysdig with IBM Blockchain Platform before, check out this tutorial on IBM Cloud Monitoring with Sysdig to set one up.
Because the tutorial uses the IBM Cloud CLI to configure the Prometheus metrics in Sysdig, you need to log in to your IBM Cloud Kubernetes cluster from the command line.
You should have already deployed peers and ordering service nodes on your IBM Blockchain Platform. Check out the “Build a network” tutorial series if you need help getting set up.
The estimated time to complete this tutorial is about one hour.
- Create an identity that Sysdig can use to access the nodes
- Configure the TLS secret
- Configure the Prometheus settings
- View the Prometheus metrics in the Sysdig dashboard
- Use the Prometheus metrics in your Sysdig dashboard
- Use the Prometheus metrics to monitor an ordering node
After you complete the following five steps, you will have a working Sysdig dashboard configured with two Prometheus metrics for monitoring peers. The sixth step describes what you would do differently if you wanted to monitor ordering nodes instead of peers.
Create an identity that Sysdig can use to access the nodes
Sysdig uses mutual Transport Layer Security (TLS) to access the nodes in your cluster, so you need to use your organization’s TLS certificate authority (CA) to generate a private and public key. It’s the same process you followed when you registered and enrolled identities for your peer nodes, but this time you need to use the TLS CA instead.
When you open your peer’s CA in the console, click Register user to create a new user that will be used to access your blockchain network from Sysdig. Specify the enroll ID and secret for this user and set the type to client. After you add the user, click Enroll identity on the user’s action menu to generate the certificate and private key for that user.
Important: Be sure to select the TLS Certificate Authority from the Certificate Authority drop-down list.
Enter the enroll ID and secret that you specified when you created the user in the previous step. When you click Next, the subsequent panel contains the generated certs that are required for Sysdig.
You’ll need the certificate and private keys in the next step, so download them to your file system and then click Add identity to Wallet to store these keys in your console wallet for safekeeping.
Tip: The IBM Blockchain Platform console does not store these certificates; you are responsible for managing them. When you close your browser, these keys are not saved. If you need to use them again (from a different browser, for example), you need to import the identity into your console wallet on that browser. Therefore, we recommend that you click Export identity to save the identity to your file system so that you can import it later as needed.
Configure the TLS secret
When you configured Sysdig in your Kubernetes cluster, the
ibm-observe namespace was created in your Kubernetes cluster. You need to edit the associated
configmap to configure them for Prometheus, but first, you need to store the TLS certs that you downloaded in a Kubernetes secret. (Run these commands from the same IBM Cloud CLI that you used when you configured Sysdig.)
Create a TLS secret in the
ibm-observenamespace by running the following, where
<key-path>is the path to the certs you downloaded to your local file system. Make a note of the
<secret-name>that you use.
kubectl create secret tls <secret-name> --cert=<cert-path> --key=<key-path> -n ibm-observe
When Sysdig is deployed to your cluster, it creates a daemonset within the
ibm-observenamespace. Edit this daemonset by running:
kubectl edit daemonset -n ibm-observe sysdig-agent
Scroll down to
spec.template.spec.containers.volumeMountsand add a volume mount for your secret:
- mountPath: /<mount-path> name: <volume-name>
- mountPath: /ibp name: ibp-volume
spec.template.spec.containers.volumesand add your secret to the volume, specify any value for the
<volume-name>, and enter the name of the secret you created above in the
- name: <volume-name> secret: defaultMode: 420 secretName: <secret-name>
- name: ibp-volume secret: defaultMode: 420 secretName: sysdigtls
Configure the Prometheus settings
Almost done! Now you can edit the
ibm-observe namespace configmap by running:
kubectl edit configmap -n ibm-observe sysdig-agent
Change it to the following, paying close attention to indentation:
prometheus: enabled: true # Helpful for determining why the metrics aren't being detected log_errors: true process_filter: # Exclusions avoid any Prometheus metrics other than the ones included from being detected - exclude: process.name: docker-proxy - exclude: container.image: sysdig/agent - exclude: appcheck.match: prometheus # Includes all Kubernetes pods that match the specified label (name in this case) - include: kubernetes.pod.label.orgname: <peer-organization-msp-id> conf: use_https: true auth_cert_path: "<mount-path where secret is located>/tls.crt" auth_key_path: "<mount-path where secret is located>/tls.key"
<peer-organization-msp-id>with the msp ID of your peer’s organization, and the “
<mount-path where secret is located>/tls.key” with the value you specified in the daemonset above, for example,
Configuration is complete. Now, let’s look at the metrics in the Sysdig dashboard.
View the Prometheus metrics in the Sysdig dashboard
After you edit the configmap, it can take several minutes for the Sysdig dashboard to reflect the metrics. You’ll know the configuration worked by clicking the Explore tab in the dashboard. From Hosts & Containers, click Entire Infrastructure. Click the Overview by Host twistie and scroll down to see all the Prometheus metrics.
Once you can see the Prometheus metrics as shown here, Sysdig and the Prometheus metrics are successfully configured.
Use the Prometheus metrics in your Sysdig dashboard
You can always check out the Hyperledger Fabric Metrics Reference guide for more information about what each Prometheus metric captures. To get started, there are a few metrics that are useful for assessing the state of your peers. For this tutorial, we examine
We’ll start by assessing the ledger height of the peers in the organization. From the list of Prometheus metrics, choose ledger_blockchain_height. Click the AVG drop-down menu next to the metric and ensure that the metric aggregations are set to Maximum instead of Average. In the Segment by drop-down list, select kubernetes.pod.label.name. When you click on the line in the graph, you get a breakdown of ledger height by peer. This is a useful view when you want to ensure that all peers in the organization are keeping up with the transactions, a potential indicator of peer health. Use this chart to identify any peers whose ledgers are falling behind.
This chart was created by adding the metric to a new Blockchain – Prometheus Metrics dashboard. (If you followed the “IBM Cloud Monitoring with Sysdig” tutorial in the Prerequisites, the process for creating a new Dashboard is described there.) Dashboards are a useful place to save any customizations you make to a chart and easily see the data on-demand later. Simply create a new Sysdig dashboard and click the + sign to add a new panel where you can select the metric and segmentation type.
Tip: If your data is not showing up on the graph, there is a gray timeline at the bottom of the page that you can use to adjust the time range and even set a custom range:
Let’s explore the
couchdb_processing_time metric. Repeat the same process that we used for adding the
ledger_blockchain_height metric, but this time select the metric
couchdb_processing_time. Again, we segment the data by
kubernetes.pod.label.name. The resulting graph is useful for tracking the average amount of time in seconds that CouchDB takes to respond to ledger requests across your peers, exposing untuned queries.
Use the Prometheus metrics to monitor an ordering node
If you are more interested in monitoring your ordering nodes, the process for configuring the monitoring of them is largely the same as monitoring peer nodes:
In Step 1, when you register and enroll the user, you need to use the CA for the ordering node instead of the CA for the peer node.
In Step 2, create a Kubernetes secret for the ordering node TLS CA certificates.
In Step 3, edit the daemonset and add the ordering node TLS CA certificate and key as described. When you edit the configmap, you need to specify the msp ID of the orderer organization for the
kubernetes.pod.label.orgnameparameter instead of the msp ID of the peer organization.
You are now ready to monitor your ordering node using the Prometheus metrics.
In this tutorial, you configured your Sysdig dashboard to work with the Prometheus metrics to monitor peers or ordering nodes in your IBM Blockchain Platform network. With the configuration complete, you can now experiment with the other Prometheus metrics to gain insights and debug problem areas in your network. If you have not already completed the IBM Cloud Monitoring with Sysdig tutorial for monitoring your node resource allocation, you should do that now.
When you combine the insights from that dashboard with your metric dashboard you begin to form a more holistic picture of how your network is responding to the transaction requests. You can then check out the Hyperledger Fabric Metrics Reference guide to find additional metrics for your dashboard to delve deeper into the node activity on your blockchain network.