Monitor Java applications on OpenShift with Cryostat

Cryostat helps you monitor Java applications on Red Hat OpenShift by capturing and analyzing JDK Flight Recorder (JFR) data from containerized JVMs.

In this article, you learn how to set up Cryostat, connect it to a sample application, capture recordings, review automated analysis, and run diagnostic commands.

Cryostat is an open source, container-native tool for securely managing JFR data on OpenShift. You can start, stop, retrieve, and archive recordings through the web console or HTTP API. You can also use Cryostat to diagnose performance issues, optimize resource consumption, and investigate JVM behavior.

For more background, see the Cryostat architecture documentation.

Set up Cryostat to monitor a Java application

To use Cryostat to monitor an application, first set up a Kubernetes cluster with the Cryostat Operator and deploy a sample application.

You can install Cryostat from OpenShift OperatorHub or Helm. When you use the Helm chart, you need to manually set up the agent.

Then, you need to create a Cryostat custom resource in a cryostat-sample.yaml definition file.

apiVersion: operator.cryostat.io/v1beta2

kind: Cryostat

metadata:

  name: cryostat-sample

spec:

  targetNamespaces:

    - my-app-namespace

    - my-other-app-namespace

Finally, you install the custom resource into the cluster using the following command:

kubectl apply -f cryostat-sample.yaml

For more advanced or production-ready deployments of Cryostat, you should use an S3-compatible external storage provider to store recordings, thread dumps, and heap dumps instead of relying on the simple object storage included with Cryostat.

For a sample application, use the Quarkus Petclinic application. Petclinic is a simple web application with a backend database and a frontend for tracking pets, owners, vets, and visits.

To use Cryostat with this application in an OpenShift environment, use agent autoconfiguration to streamline setup. To automatically attach the Cryostat Agent and begin monitoring, add the following labels to the deployment pod template labels, or directly to the deployment labels in Cryostat 4.2 and later:

cryostat.io/name: cryostat-sample
cryostat.io/namespace: cryostat

View JVM details and application topology

To get a general overview of the application, use the JVM Details and Topology view. The topology view shows all discovered JVMs, known as targets, and their associated resources. Targets appear within related groups, such as Pod or Deployment.

Starting with Cryostat 4.1, the Topology view also shows Automated Analysis scores on discovered targets. A status badge displays the highest severity score. You can configure which automated analysis results contribute to that score when a reported issue is not relevant to your application.

Capture JFR recordings with Cryostat

Cryostat’s main feature is to capture and archive recordings for JVMs running in a containerized environment.

You can capture recordings for JVMs in one of these ways:

• Manually on a target
• Automated rules
• Smart triggers
• Automatically collected from the Agent Harvester. See Cryostat Agent Harvester configuration for more information.

Manually capturing a recording on a target

To capture a recording manually, in the Cryostat UI, navigate to the Recordings page, and select the target and then select the Create button. Cryostat uses custom recording templates to capture Flight Recorder events.

Because the Petclinic application is a Quarkus application, you can use the Flight Recorder events implemented by Quarkus using the Quarkus template. There are also several other built in templates available to capture other sets of events, for instance Continuous and Profiling. The Quarkus events require the quarkus-jfr extension.

Once complete, the recording with the selected settings is created in the target. The recording can then be stopped or archived through the recordings table for continuous recordings or automatically for recordings with fixed durations.

Capturing a recording with automated rules

Use automated rules to capture recordings when you cannot retrieve them manually, such as when an application, in this case the Petclinic sample application, is crashing or restarting very frequently and is too short-lived.

Automated rules are configurations that instruct Cryostat to create JDK Flight Recordings on matching target JVM applications. Each automated rule specifies parameters for which event template to use, how much data should be kept in the application recording buffer, and how frequently Cryostat should copy the application recording buffer into Cryostat’s own archived storage.

You can create a custom automated rule for the Petclinic sample application with the following match expression to start a recording using the Quarkus template:

target.alias.contains(“quarkus”)

Now, if the application crashes and is restarted, when Cryostat detects the new target and detects that it is a Quarkus target, it will immediately start a recording using the custom template.

For step-by-step instructions, see the Cryostat guide for creating an automated rule.

Capture a recording with smart triggers

For targets configured with the Cryostat Agent, smart triggers can be used to automatically capture recordings when specific conditions are met. The Cryostat Agent supports custom triggers that are based on MBean metric values. You can configure the Cryostat Agent to start JFR recordings dynamically when these custom trigger conditions are met.

The trigger condition is based on MBean counters, covering various runtime, memory, thread, and OS metrics. The condition includes MBean counter types and allows specifying a duration for the trigger to activate only if the specified values persist for that duration.

For more information, see the Cryostat smart triggers guide.

Analyze recordings automatically

Cryostat automatically analyzes recordings. This automated analysis for a target looks for anti-patterns in Flight Recorder data and identifies potential application problems.

To view the automated analysis results for captured recordings in more detail, navigate to the Recording Archives and expand the row for a given recording.

In the Petclinic example, the application is consuming a very large amount of physical memory available to the container.

You can also view an overview of all recent automated analysis reports per target with the Automated Reports view. You can sort and filter the reports to help identify which JVMs in your deployment require attention.

Run diagnostic commands for Java performance issues

When application performance becomes degraded, the analysis of the flight recordings suggests that there are issues with concurrency (such as application threads are becoming deadlocked) or with memory (such as a memory leak is likely). Analyzing the flight recording alone might not be enough to properly diagnose these situations, but you can use Cryostat to provide more diagnostic information:

• Perform garbage collection
• Capture a thread dump
• Capture a heap dump

To perform these diagnostic commands, navigate to the Diagnostics page.

Also, once a thread dump has been collected, you can use the Analyze Thread Dumps view, which displays aggregated information for thread states, method calls, and lock instances, and a detailed table for deadlocks detected and thread information.

Summary

Cryostat simplifies Java application monitoring in hybrid cloud environments such as Kubernetes and Red Hat OpenShift. With Cryostat, developers and site reliability engineers can capture JFR data, analyze JVM behavior, and run diagnostics to resolve performance issues faster.