Overview

Skill Level: Intermediate

This recipe is co-authored with Felix Wong (fmhwong@ca.ibm.com) of IBM WebSphere Application Server Development. The recipe outlines how to integrate Prometheus to monitor WebSphere Liberty in IBM Cloud Private.

Ingredients

IBM Cloud Private.

kubectl command line.

The reader is assumed to be famiiar with IBM Cloud Private, Kubernetes, WebSphere Liberty and Docker.

Step-by-step

  1. Introduction

    WebSphere Liberty is a popular, reliable JEE compliant middleware available on IBM Cloud Private (ICP) for shifting traditional WebSphere loads to ICP in containerized forms as well as for developing cloud-native Java-based microservice applications. For running applications deployed in WebSphere Liberty containers in ICP, you may want to monitor and/or set alerts on various Java Virtual Machine (JVM) and system level statistics. Prometheus, the popular open-source monitoring tool, is part of the ICP platform and can be used to monitor various metrics and set alerts when necessary.

    Presently, ICP does not provide an out of the box integration for containerized Liberty instances with Prometheus. In this short note, we will provide a simple method for wiring a Prometheus deployment with your Liberty pods to scrape JVM and system level metrics from the running Liberty containers.

  2. The Monitoring Scheme

    The wiring technique essentially consists of three simple steps:

    1. Enable the mpMetrics-1.0 feature for the Liberty instances that you want to monitor
    2. Provide annotation metadata in the service description YAML file of your deployment
    3. In the Prometheus configuration file, introduce a job with the same name as that of the annotation metadata of step 2, along with a few other required attributes for the job

     

    It should be noted that the metrics provided by the mpMetrics-1.0 feature of Liberty are secured and must be accessed using a secure protocol. This automatically implies that for the Liberty-based web applications hosted in ICP, you must use the https protocol to access your application if you want to monitor the JVM and system level metrics of Liberty.

    A simple illustration of the proposed technique for monitoring the JVM and system level statistics for a Liberty-based deployment in ICP follows.

  3. Add the mpMetrics-1.0 Feature

    Presently the Liberty docker image does not contain the mpMetrics-1.0 feature by default. For real-life production usage, it is recommended that you create your own docker images for Liberty and while creating the relevant images, add the mpMetrics-1.0 feature.

    Figure 1 shows relevant portions of the server xml file for our sample Liberty application.

    <server description="Test Server">

        <!-- Enable features -->
    <featureManager>
    <!-- All the required Liberty features for your application -->

            <feature>mpMetrics-1.0</feature>

        </featureManager>

         <!-- To access this server from a remote client add a host attribute to the following element, e.g. host="*" -->
        <httpEndpoint host="*" httpPort="9080" httpsPort="9443" id="defaultHttpEndpoint"/>

        <quickStartSecurity userName="admin" userPassword="admin"/>

    <!— JEE application archives and required resources -->

    </server> 

    Figure 1

    Notes:

    1. mpMetrics-1.0 feature (shown in bold in Figure 1) is added in the list of features.
    2. The metrics need to be accessed securely. In this sample for the sake of simplicity and illustration, we used a simple keystore based security, and also provided the userid and password in the server.xml file in clear text form. In a real-life production usage, these credentials can be accessed from a Kubernetes Secret or preferably through LDAP or other secure repositories as per your organization’s security guidelines.

     

    Since mpMetrics-1.0 feature is not presently available in the docker image of Liberty, you have to include the following bold line in your dockerfile for building the desired image, as shown in Figure 2.

    FROM websphere-liberty
    COPY server.xml /opt/ibm/wlp/usr/servers/defaultServer/
    RUN installUtility install --acceptLicense defaultServer
    # application packaging specific copy commands
    # other commands

    Figure 2

  4. Annotate the Service Deployment

    In order for Prometheus to scrape the metrics data from your Liberty pods, you need to provide an annotation in your service YAML file as shown in Figure 3.

    # Deploy the service
    apiVersion: v1
    kind: Service
    metadata:
      annotations:
        debliberty: "true"
      # name, labels and all other components of the Service
      # …

    Figure 3

    Note that the service deployment has the annotation of ‘debliberty’ — an arbitrary string. This annotation will be used to pair Prometheus with the annotated service — the same annotation string need to be present in Prometheus configuration for the intended pairing. In our sample, we deployed Liberty manually. If you use a Helm chart to deploy Liberty, you have to manually edit the corresponding service YAML file to add this annotation metadata.  

  5. Install and Configure Prometheus

    You may deploy Prometheus from the ICP-Console or from outside using Helm CLI. The following example shows how to deploy Prometheus from the ICP-Console. Following the Catalog –> Helm Charts path, locate and click on the ibm-icpmonitoring chart and then click on the Configure button. In the resulting panel, provide a name (deb-prometheus in this instance), select a namespace (default is selected), accept the license and click Install as shown in Figure 4.

     Picture1

    Figure 4

    The Helm chart will get installed in your ICP.

     Issuing the following command

    kubectl get svc | grep prometheus

    from a properly sourced terminal (CLI having the contextual configuraion information of the ICP cluster) you will see a number of services will get deployed by the ibm-icpmonitoring chart. In this exercise, we will focus on and use the service named <name used for the helm release>-promethuesdeb-prometheus-prometheus here as shown in Figure 5.

    Fig5

    Figure 5

    Each of these services has an associated configmap object. To scrape the metrics data of the Liberty pods, you need to modify the configmap associated with the deb-prometheus-prometheus service by adding the ‘debliberty’ annotation of the Service deployment along with a few other attributes.

    The simplest to way to achieve this is to edit the intended configmap object from a sourced terminal using the 

    kubectl edit configmap deb-prometheus-prometheus

    command.

    The command will bring up the requested YAML file in the vi editor.  Scroll down to the end of the file and insert the yellow highlighted portion just before the line kind: ConfigMap  as shown in Figure 6.

    Fig6

    Figure 6

    Save the configmap YAML file from the vi editor.

    Note the ‘value’ of the ‘source_labels’. The debliberty end portion of the self-explanatory name ties the Prometheus instance to the service whose metadata has the debliberty: “true” annotation — refer to Figure 3.

    The saved configmap file needs to be reloaded into Prometheus. Use the following curl command from any terminal that has connectivity to the Proxy node of the ICP cluster.

    curl -s -XPOST http://<ip address of the ICp Proxy>:30178/-/reload

    Note, for deb-prometheus-prometheus service, 30178 is the port of its endpoint, as shown in Figure 5. Also since, the deb-prometheus-prometheus is of type NodePort, you can use the ip address of any node of the ICP cluster to reload the Prometheus changed configuration.

    Once the reloading of the configmap is complete, you have completed all the configurations and wiring needed to monitor the JVM and system level statistics of Liberty using Prometheus.  

  6. Monitoring Liberty Statistics

    Browse to the Prometheus GUI from a browser using the URL: http://<ip address of the Proxy Node>:30178.

    In the Prometheus console, first click on Status and then select Targets from the drop-down as shown in Figure 7.

    Fig7

    Figure 7

    You should see all the Targets whose statistics are being scraped by Prometheus.

    Fig8

    Figure 8

    Figure 8 clearly shows the ‘deb-liberty’ Target – refer to the value of ‘job_name’ attribute of the configmap YAML file of Figure 6.

    We scaled up our web deployment sample to two replicas, and that’s why Prometheus shows two endpoints getting scraped.

    If you click on Graph (Figure 9),

    Fig8

    Figure 9

    and in the resulting panel, click on the insert metric at cursor as shown in Figure 10

    Fig10

    Figure 10

    you will see a number of metrics that can be monitored by the present Prometheus configuration. Of all the metrics in the long list, the metric names starting with base: are from the Liberty containers contributed by the mpMetrics-1.0 feature – refer to Figure 11.

    Fig11

    Figure 11

     Selecting any Liberty metric (say base:thread_count), you can see the values from both the Liberty pods in the Prometheus graph, as shown in Figure 12. 

    Fig12

    Figure 12

    You can explore other relevant metrics from Liberty in Prometheus in graphical as well as in numeric form by clicking on Console.

    You can also scale your deployments. Within a short period of time the number of endpoints in the Prometheus console (see Figure 8) will match to the number of replicas.  

    Note: Though we have used clear text for password in the configmap file of Prometheus, Prometheus will not display the password when its configuration is viewed in the Prometheus panel as shown in Figure 13.

    Fig13

    Figure 13

  7. Conclusions

    The technique provided here is simple and easy to follow. You should be able to apply the wiring strategy sketched out in this note to monitor the JVM and system level statistics of your Liberty containers deployed in ICP. It should be noted that the mpMetrics-1.0 feature can also be used to monitor appication level metrics — refer to the details here.

  8. Acknowledgements

    Don Bourne, STSM, WebSphere Development and Alan Little, IBM DE, WebSphere Cloud Architect, reviewed the article for technical accuracy. Monica Tamboli of WebSphere Development provided sample deployment files. Peter Van Sickel of the ICP CTE team reviewed the draft and provided suggestions for improvement. 

Join The Discussion