Application monitoring in cloud deployments for proactive observability

To build a resilient application, developers must implement application monitoring tools that capture failures, potential dependency issues, and potential performance issues. Good application monitoring provides proactive alerting and observability with real-time insights into the health of your application so that you can take timely corrective actions.

In cloud-based microservices deployments that involve multiple integration points, transient failures might occur. You must plan for and handle these transient failures. Good application monitoring must capture how the application performs under various constraints and provide insights into the performance of each application component and integration point. Baseline measurement of the performance of an application is needed to assess if the addition of new features or new dependencies might cause a degradation of system performance.

A popular open source tool for monitoring an application is Prometheus. And once your metrics data is available in Prometheus, you can use Grafana to visualize the application monitoring data.

In this observability article, we explain how to identify and capture the right metrics in applications that run in a distributed manner across multiple containers in cloud deployments and that have integrations with several dependent services, such as a typical blockchain-based application.

Capturing metrics for application monitoring and observability

To enable metrics collection using Prometheus in any application, you must add the following dependencies:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId> io.micrometer</groupId>
  <artifactId> micrometer-registry-prometheus</artifactId>
</dependency>

Then in the application properties file, you can specify these parameters to configure a custom endpoint for the metrics:

#Metrics related configurations
management.endpoint.metrics.enabled=true
management.endpoints.web.exposure.include=prometheus
management.endpoint.prometheus.enabled=true
management.metrics.export.prometheus.enabled=true
management.endpoints.web.base-path=/
management.endpoints.web.path-mapping.prometheus=/management/plt/admin/v1/metrics

Now we are ready to identify the point of integration -- the functional part in the application -- that needs to be monitored for performance. Also, we must define the custom metrics in the application code to capture the performance of the different application components. We will use tags when defining the custom metrics so that the metrics data can be more easily analyzed.

The following code snippet of a util class implements capturing custom metrics for performance data:

public class PerformanceCaptureUtil {

  long startTime;

    /**
     * Start timing. Log and stop can be called once start has been called.
     *
     */
  public void start() {
    this.startTime = Utils.getGMTTimeStamp();
  }

  /**
     * Returns the time spent in milliseconds from the previous call of start function.
     * If stop is called before start, it returns -1.
     *
     * 
     * @return Time spent since start
     */
  public long stop() {
    if (this.startTime == 0) {
      return -1;
    } else {
      long measuredTime = Utils.getGMTTimeStamp() – 
               this.startTime;
      this.startTime = 0;
      return measuredTime;
    }
  }


/**
     * Same functionality as Stop but also adds logs and record Timer metric.
     *
     * @param logEvent - The String which should be added in the logger
     * @param timerMetric
   * @param stats
     */
  public void stopAndRecordTimerMetric(String 
     logEvent,CustomTimerMetric timerMetric, Map<String, 
     String> stats) {

  long duration = this.stop();

  logger.info("Performance Capture(ms): { {}: {}}", 
     logEvent, duration);

  metricHelper.recordTimerMetric(timerMetric, stats, 
     duration);

}

}

The following code snippet of a helper class implements how to capture the timing of the custom metrics in Prometheus:

@Component
public class PerformanceMetricHelper {

private Logger logger = LoggerFactory.getLogger(PerformanceMetricHelper.class);

private MeterRegistry meterRegistry;

public static enum CustomTimerMetric {

    METRIC_AF_PERFORMACTION_PREPARE_LATENCY("af_performaction_prepare_latency"),
    METRIC_AF_PERFORMACTION_SUBMIT_LATENCY("af_performaction_prepare_latency"),
    METRIC_AF_WORKER_QUEUE_LATENCY("af_worker_queue_latency"),
    METRIC_AF_REQUEST_FLOW_LATENCY("af_request_flow_latency");


  private final String timeMetricName;

  CustomTimerMetric(String metricName) {
    this.timeMetricName = metricName;
  }

  public String value() {
    return this.timeMetricName;
  }

  @Override
  public String toString() {
    return this.timeMetricName;
  }

}




private static Map<CustomTimerMetric, Set<String>> metricTagMap = ImmutableMap.<CustomTimerMetric, Set<String>>builder()
.put(CustomTimerMetric.METRIC_AF_PERFORMACTION_PREPARE_LATENCY,ImmutableSet.of("actionType","peerOrgs"))
    .put(CustomTimerMetric.METRIC_AF_PERFORMACTION_SUBMIT_LATENCY, ImmutableSet.of("orderer","channel"))

.put(CustomTimerMetric.METRIC_AF_WORKER_QUEUE_LATENCY, ImmutableSet.of("type"))

.put(CustomTimerMetric.METRIC_AF_REQUEST_FLOW_LATENCY, ImmutableSet.of("status","errorCode","uri","no_of_actions","actionType")).build();


 @Autowired
public PerformanceMetricHelper(MeterRegistry meterRegistry) {
    this.meterRegistry = meterRegistry;
  }



public void recordTimerMetric(CustomTimerMetric timerMetric, Map<String, String> stats, long duration) {

logger.info(String.format("Logging timer metric for %s.",timerMetric));

  try {
    List<Tag> tags = new ArrayList<>();
    Set<String> allowedStatsForCurrentMetric =         
          metricTagMap.get(timerMetric);

    allowedStatsForCurrentMetric.forEach(key -> 
          tags.add(Tag.of(key, stats.get(key))));

          Timer.builder(timerMetric.value()).tags(tags).
register(this.meterRegistry).record(duration,TimeUnit.MILLISECONDS);

    }catch(Exception e) {
      logger.error(String.format("Error while 
            recording %s metric. Ignoring it !!!",  
            timerMetric));
    }

  }

}

Capturing metrics for monitoring a blockchain-based application

In typical blockchain-based applications, most invocations have a significantly high response time. Therefore, the recommended design pattern is to have an asynchronous transaction submission model for any fairly complex backend API with a high response time. See the following architecture diagram.

The following operations are performed in our example blockchain application for any transaction:

1. The client application makes an initial API asynchronous request.
2. The REST API layer supports asynchronous request execution. If the initial request validation goes through, the API accepts the request and returns a 202 to the client application with the transaction/request information.
3. The client application polls the transaction endpoint for the status. If the status is complete, the request is considered successful, meaning all the processing on the server side has completed. An InProgress status indicates the client application should continue to poll, whereas a FAILED status means the client application is expected to retry any transaction with a 5xx error code. The client can be notified via a webhook or message with the status of the transaction.
4. As a part of the asynchronous request processing, the blockchain application interacts with several other services and subsystems. All asynchronous processing is achieved using IBM Event Streams (or Apache Kafka).
5. The API layer will leverage the events service to be notified of new blocks being added to the blockchain.

We will explore the different custom metrics required at each step and how these metrics can be used for proactive alerting and real-time insights. In this example, we explore capturing performance metrics for a blockchain transaction, Kafka message, and overall API request flow.

Blockchain transaction custom metrics

Standard blockchain transaction flow involves the following steps:

1. The client application submits a transaction proposal.
2. Endorsing peers (E0, E1, and E2 in our example) each execute the proposed transaction. None of these executions update the ledger. Each execution captures the set of read/write data (called an RW-set), which now flows in the blockchain network.
3. The application receives responses. The RW-sets are signed by each endorser and include each record version number.
4. The application submits proposal responses as transactions for ordering.
5. Orderer sends blocks to committing peers.
6. Committing peers validate transactions. Validated transactions are applied to the world state and retained on the ledger.
7. The application is notified when a block is committed to the ledger of a peer.

In the above steps, a very important insight from an application integration context is the performance (avg/max) of blockchain proposal/submit time (latency of steps 1-3 and latency of steps 4-7). In a production environment, this metric helps to get the performance at different peers and orderer levels, and take proactive action before a customer reports an anomaly in the behavior.

The following sample code snippet captures blockchain transaction metrics.

private ObjectFactory<PerformanceCaptureUtil> perfCaptureFactory;

PerformanceCaptureUtil perfCapture = perfCaptureFactory.getObject();

perfCapture.start();

TransactionProposalResponse<String> proposalResponse = client.sendTransactionProposal(afChaincodeName, afChaincodeVersion, afChaincodePerformActionMethod, args, handler);

logger.debug("performAction: Got proposal for Audit action. TxnId:{}, Identifiers:{}",proposalResponse.getTransactionID(), identifiers);

Map<String, String> perfStats = new HashMap<String, String>();

List<ProposalResponseModel> proposalResponses = proposalResponse.getProposalResponses();

String peerOrgs = proposalResponses.stream().sorted((o1,o2)->o1.getPeerName().compareTo(o2.getPeerName())) .map(o->o.getPeerName()).collect(Collectors.joining(","));

perfStats.put("actionType", actionType);

perfStats.put("peerOrgs", peerOrgs);

perfCapture.stopAndRecordTimerMetric(String.format("%s Proposal request time", actionType),

CustomTimerMetric.METRIC_AF_PERFORMACTION_PREPARE_LATENCY, perfStats);

return proposalResponse;

The custom metrics in this code capture blockchain proposal response times, along with additional details of peers on which the transaction was performed. Metrics captured like this can be translated into multiple visual representations in Grafana, which can provide detailed insights into performance bottlenecks or failures at the lowest transaction components.

The following visualization of the blockchain transaction metrics shows the average time taken for a blockchain proposal submission based on action type and peers.

The following visualization shows the maximum time taken for a blockchain proposal submission based on action type and peers.

Now let’s take an example from a real-world scenario. A user reports that the transaction is taking longer to complete than usual. Reviewing our monitoring dashboard in Grafana, during the timespan that the user reported the issue, we identify that the average blockchain query time is relatively high. Upon further analysis, it indicates that it is higher on specific peers. We can then move on to analyze why the peer is non-responsive or slow and take the recommended steps for resolution, which helps to avoid larger impacts to users.

Based on our metric tag containing peers information, we get the details of the peer causing latency in Production environment: grpcs://xxxxx-peer1.prod-yyyy-0000.us-south.containers.appdomain.cloud:7051.

Good monitoring practice is to have the threshold defined and alerts in place so that such issues are captured and resolved proactively.

Kafka message metrics

In our example application, all asynchronous processing is implemented using IBM Event Streams, a managed instance of Apache Kafka. This is another critical integration area for capturing producer or consumer latency, which impacts the performance of the blockchain application. For example, the application cannot process messages at the rate at which requests are coming, which typically indicates the production load is higher than baselined. Consumers can receive different types of messages based on application requirements, and, as a first step, we need to have identifiers for each type of message.

Below is an example in Grafana where we capture the latency for different types of messages being consumed for request progression and resiliency. The visualization shows the average time taken to receive the Kafka message based on record type.

Request flow metrics

The Spring Boot default metric http_server_requests_seconds helps capture success and failed API request rates synchronously. The following visualization shows the success rate for synchronous requests.

The following visualization shows the failure rate for synchronous requests.

The following visualization shows the average time taken to complete the transaction for an asynchronous request.

For asynchronous request flows, we can further customize metrics with tags capturing uri, status, and errorCode for success and failure insights.

The following visualization shows the failure rate for asynchronous requests.

Summary and next steps

In this article, we introduced some recommended custom metrics that can be used for blockchain-based distributed applications. As a developer, understanding and monitoring the points of failure enables you to build a resilient system and minimize customer impact if there are any issues.