Recently, I have been asked by customers for a way to monitor memory consumption and alert if it exceeds a certain threshold. The primary reason for such a requirement was Hiveserver2 aborting due to out of memory while running large queries, and it was hard for customer to tell the cause for query failure unless the process terminated and they reviewed the logs.

Ambari provides robust alerting and monitoring capabilities, but currently there is limited out of box support in monitoring and notifying memory consumption of Hive. Although there are several monitoring and notification applications available in the market, in this article I have made an attempt to highlight how to build our own monitoring and notification application with the ability to customize it.

Technologies used

1. Java attach API
2. Apache Commons Email
3. Maven
4. OpenJDK 8

The project is uploaded to my git repo and can be accessed using the following link https://github.com/bharathdcs/attach_api_notification

Java Attach API’s allows user program to attach to running JVM’s. The JVM could be hosting NodeManagers , HiveServer2 or any other process. One of the advantages of attach API is you can retrieve the JMX url from JVM and fetch various statistics, one of them is memory consumption as shown in following code snippet

   VirtualMachine virtualMachine = VirtualMachine.attach(virtualMachineDescriptor);  String propertyValue = agentProperties.getProperty("com.sun.management.jmxremote.localConnectorAddress");  c.getMBeanServerConnection().getAttribute(new ObjectName("java.lang:type=Memory"), "HeapMemoryUsage");  

Using the memory object (bean) returned we can retrieve stats like used, allocated and max memory. More details about MemoryMXBean can be found in the java documentation

Properties File for Parameters

Users specify the memory usage thresholds (in percentage) for triggering alerts and SMTP server details to send the notifications in a property file.
Following is a snippet from a sample properties file

  procId=3124  smtp_host=smtp.gmail.com  smtp_port=465  smpt_user=test@gmail.com  stmp_password=testing123  threshold=10  email=test@gmail.com  

The procId is the process Id of JVM to be attached and monitored**.
provide your SMTP server details using smtp_* variables.
Threshold specifies the threshold for triggering alert when memory usage crosses (threshold/100)*totalHeap.

Steps to run the Appliation

The application uses maven build framework.
1. Compile using “mvn clean compile assembly:single”
2. Run the application using following syntax

  java -cp ${JAVA_HOME}/lib/tools.jar:heap-0.0.1-SNAPSHOT-jar-with-dependencies.jar monitor.heap.App sample.properties   
  

The application monitors memory usage continuously and when it exceeds a certain threshold , a one time alert is triggered and following email is sent to your configured email Id.

Possible Future Enhancements

1. Integrating with mobile notification APIs
2. Publish the memory units as JSON enabling java script widgets to chart them.

** – Users can only attach to JVMs owned by them.For example, Hive user can only attach to hive JVM’s .