Build efficient Grafana dashboards from the built-in Prometheus of OpenShift Container Platform 4.3

Introduction

The existing built-in Grafana dashboards provided by Red Hat OpenShift Container Platform clusters have a fixed template with many metric details. Whereas with a customized dashboard, a system administrator can focus only on required monitoring parameters. However, it is not easy to accomplish because of constraints with writing custom queries. This step-by-step tutorial explains how to deploy the community edition of the Grafana Operator and leverage an existing Prometheus as a Grafana data source to create customizable Grafana dashboards.

A bit of background

OpenShift Container Platform includes a Prometheus-based monitoring stack by default. However, this built-in monitoring capability provides read-only cluster monitoring and does not allow monitoring any additional target. The built-in monitoring feature monitors the cluster components such as pods, workspaces, nodes, and provides a set of Grafana dashboards that are non-customisable.

Prerequisites

• Red Hat OpenShift on IBM Cloud is a managed service that simplifies deployment and configuration of the OpenShift Container Platform.
• Red Hat CodeReady Containers are preconfigured OpenShift 4.1 (or newer) clusters. They offer a minimal OpenShift cluster environment for test and development purposes on local computers.

Deployment

Install Grafana Operator community edition

1. Log in to the OpenShift Container Platform cluster console with the cluster-admin role.

2. Create a new project named Grafana.

    $ oc adm new-project Grafana
    Created project Grafana
   

   Show more

3. On the web console, click Operators, and then click OperatorHub.

4. Search for Grafana Operator and install the community edition of the Grafana Operator.

5. On Create Operator Subscription, under Installation Mode, click A specific namespace on the cluster; under Update Channel, click alpha; and under Approval Strategy, click Automatic. Then click Subscribe.

6. Check the pod status to see if the install is complete.

    $ oc get pods -n grafana -o name
    pod/grafana-operator-655f76684-7jsz5
   

   Show more

Create a Prometheus user

Before creating Grafana and Grafana data source instances, you need to create a special user in the existing Prometheus on openshift-monitoring project.

1. Navigate to the openshift-monitoring namespace:

    $ oc project openshift-monitoring
    Now using project "openshift-monitoring" on server
   

   Show more

2. Load prometheus-k8s-htpassword data in the tmp file:

    $ oc get secret prometheus-k8s-htpasswd -o jsonpath='{.data.auth}' | base64 -d > /tmp/htpasswd-tmp
   

   Show more

3. Create a special user to the existing Prometheus secret:

    $ htpasswd -s -b  /tmp/htpasswd-tmp grafana-user mysupersecretpasswd
    Adding password for user grafana-user
   

   Show more

4. Check the content of /tmp/htpasswd-tmp for grafana-user:

    $ cat /tmp/htpasswd-tmp | tail -1
    grafana-user:{SHA}xxxxxxSwuJxNmjPI6vdZEyyyyy=
   

   Show more

5. Replace the prometheus-k8s-secret data with our /tmp/htpasswd-tmp:

    $ oc patch secret prometheus-k8s-htpasswd -p "{\"data\":{\"auth\":\"$(base64 -w0 /tmp/htpasswd-tmp)\"}}"
    secret/prometheus-k8s-htpasswd patched
   

   Show more

6. Delete the Prometheus pods to restart the pods with new data:

    $ oc delete pods -l app=prometheus
    pod "prometheus-k8s-0" deleted
    pod "prometheus-k8s-1" deleted
   
    $ oc get pods -l app=prometheus -o name
    pod/prometheus-k8s-0
    pod/prometheus-k8s-1
   

   Show more

Create Grafana instances

1. On the Grafana namespace, click Installed Operators > Grafana Operator > Create Instance on the Grafana card (shown below):

2. On Create Grafana YAML file, edit the name and spec.config.security.admin_user and spec.config.security.admin_password as required, and then click Create. The following image shows the YAML file used for creating the Grafana instance.

3. Make sure the Grafana pods are created and running:

    $ oc get pods -n grafana -o name
    pods/grafana-deployment-689d864797-n4lpl
    pods/grafana-operator-655f76684-7jsz5
   

   Show more

Create Grafana data source instances

1. Click Installed Operators > Grafana Operator > Create Grafana DataSource instance.

2. Modify the metadata.name, spec.name, basicAuthUser, and basicAuthPassword in the Create GrafanaDataSource YAML file.

   Note: Make sure to add spec.datasources.jsonData.tlsSkipVerify and spec.datasources.basicAuth set to true. Also, verify the preconfigured Prometheus url and add the right one under spec.datasources.url in the Create GrafanaDataSource YAML file.

   The Operator automatically replaces the grafana-deployment-xxx-xxx pods to reflect the new configuration.

3. Now, you are ready to access the Grafana route:

    $ oc get route
    NAME    HOST/PORT
    grafana-route    grafana-route-grafana.xxxx.appdomain.cloud
   

   Show more

Import dashboards

Now, you export an existing dashboard from the built-in Grafana and import it into the new Grafana instance created using an operator to check if the Prometheus data source is integrated.

On the openshift-monitoring stack:

1. Log in to the openshift-monitoring stack Grafana.
2. Select any of the dashboard (for example, Kubernetes / Compute Resources / Workload).

3. Click the Share dashboard icon export tab and copy the .json file.

4. Now open the newly created Grafana instance route.

5. Go to Dashboard > Manage > Import. Paste the .json file and click Load.

6. Modify the Name as required and click Import to import the dashboard.

7. Review the dashboard.

8. Once you import the dashboard onto the Grafana UI, create an instance of the Grafana dashboard to preserve the dashboards when Grafana instances are restarted.

Create Grafana DataSource instances

1. On the Grafana namespace, create an instance of the Grafana dashboard (Operators > Grafana Operator > Grafana Dashboard).
2. Copy the .json file of the dashboard imported into Grafana (as mentioned in Steps 3 and 4 under Import Dashboards).
3. Paste the copied .json file under spec.json in the Create Grafana Dashboard YAML space.
4. Modify the metadata.name as required and click Save.

With the custom Grafana dashboard, you can create a custom dashboard as required for individual pods, workspaces, namespaces, and more.

Summary

These custom Grafana dashboards with a preconfigured data source saves the computing resources on the cluster and provides the option to create your own dashboard views giving high level on compute resource usage for each application and making the monitoring easy for infrastructure administrators.