Introduction

Dumps are a vital tool for application problems in production environments, allowing system administrators to capture comprehensive diagnostics information for later analysis by application developers and maintainers. For Node.js applications there are three main dump types available:

  1. node-report, a human-readable diagnostic summary, written to file. The report includes JavaScript and native stack traces, heap statistics, platform information and resource usage data.
  2. heap dump, a snapshot of the V8 heap in JSON format, providing a summary of the JavaScript objects in the heap, their types, sizes and references. You can use Chrome DevTools to inspect the heap dump.
  3. core dump, an operating system snapshot of the application’s process memory, written as a binary file. You can use the lldb debugger with the llnode plugin to inspect the core dump.

When running Node.js applications in the Bluemix Container Service it can be difficult to capture dumps because direct access to the application process is not readily available and because files written to the local application disk are lost when application containers are restarted. A new Node.js module, cloud-diagnostics makes capturing dumps on demand from live Node.js applications in Bluemix Containers easier. The module provides a consistent set of command-line and JavaScript APIs for generating the three Node.js dump types above, with support for dumps to be written to NFS file storage (persistent volumes) for later collection and analysis. It uses the node-report, heap dump and gencore modules to provide the underlying dump support.

This is one of three articles on how to capture dumps for Node.js applications running in Bluemix environments. Other articles in this series are:

Note 1: The cloud-diagnostics module provides support for capturing dumps from running applications. Automatic capture of dumps on failure (for example on uncaught exception, fatal error or native code crash) is not currently supported.

Note 2: Some Linux images in the Bluemix Container Service currently have a /proc/sys/kernel/core_pattern setting which prevents core dumps from being created. When triggering core dumps you may see the following error message on the Bluemix application log: “cloud-diagnostics: gencore.collectCore() failedError: Unable to locate core file“. This issue is currently under investigation.

Configuration

Add the cloud-diagnostics module as a dependency to your application by editing your application’s package.json file:

  "dependencies": {
    ...
    "cloud-diagnostics": "^1.0.0"
  },

Add the cloud-diagnostics module to the application start CMD line in your container’s Dockerfile:

  CMD node -r cloud-diagnostics app.js

Using the Bluemix Command Line Interface (CLI) to trigger dumps

The cloud-diagnostics module supports dump triggering via external signals sent to the Node.js process. The module uses handlers for Linux real-time signals SIGRTMIN, SIGRTMIN+1, SIGRTMIN+2 to avoid interference with signals used by Node.js. These signals are handled by the cloud-diagnostics module and trigger a node report, heap dump or core dump respectively. The application will continue to run when the dump has completed. You can use the Bluemix CLI commands directly, or use the Linux script and Windows command files provided in the cloud-diagnostics module scripts directory. By default the cloud-diagnostics module will write dumps to the application container’s file system. You can configure the module to use persistent storage for the dumps so that they are preserved across container restarts, see “Using Bluemix NFS file storage (persistent volumes) for storing dumps” below.

Log in to the Bluemix Container Service CLI (using the ‘bx login‘ and ‘bx cs init‘ commands) then use the ‘bx ic exec‘ command to access the application container and send the required signal to the Node.js process. To trigger a node report use the RTMIN signal, to trigger a heap dump use the RTMIN+1 signal, to trigger a core dump use the RTMIN+2 signal:

> bx ic exec CONTAINER pkill -RTMIN node
> bx ic exec CONTAINER pkill -RTMIN+1 node
> bx ic exec CONTAINER pkill -RTMIN+2 node

Alternatively use the ‘icdump’ command provided in the cloud-diagnostics module scripts directory. The command takes two parameters. The first parameter allows you to specify whether you want a node report, heap dump or core dump, the second parameter is the name of your Bluemix Container Service container:

> icdump node|heap|core CONTAINER

You will see messages on the Bluemix console log for your application indicating that the requested dump has been written:


By default, dumps will be written to the application’s local disk. You can list and retrieve them using bx ic exec commands:

> bx ic exec CONTAINER ls -l
> bx ic cp CONTAINER:DUMP_FILE_NAME .


Using JavaScript APIs to trigger dumps

The cloud-diagnostics module provides JavaScript APIs for writing node reports, heap dumps and core dumps. You can call these APIs to trigger dumps, for example on error paths within your application where detailed diagnostic data is required. By default these APIs will write dumps to the application container’s file system. You can configure the module to use persistent storage for the dumps so that they are preserved across container restarts, see “Using Bluemix persistent storage for storing dumps” below.

  // require the cloud-diagnostics module in the application
  const diagnostics = require('cloud-diagnostics');
 
  // trigger a node report
  diagnostics.storeNodeReport((err, filename) => {
    console.log('node report written to: ' + filename);
  });

  // trigger a heap dump
  diagnostics.storeHeapDump((err, filename) => {
    console.log('heap dump written to: ' + filename);
  });

  // trigger a core dump
  diagnostics.storeCoreDump((err, filename) => {
    console.log('core dump written to: ' + filename);
  });

Using Bluemix NFS file storage (persistent volumes) for storing dumps

By default, the cloud-diagnostics module writes dumps to the local disk in the Bluemix container, i.e. the Node.js application’s current working directory. Support is also provided in the module for dumps to be written to NFS file storage (persistent volumes) so that they are preserved across container restarts and are available for later collection and analysis. See Storing persistent data in a volume for single and scalable containers for instructions on creating disk volumes and connecting them to your container. You can create a disk volume by using the ‘Advanced Options’ selection on the Bluemix UI when you create your application container:


When you have created a volume, select it and set the volume mount path, again using the ‘Advanced Options’ selection on the Bluemix UI when you create your container. If you use “/var/dumps” as your volume mount path, no further configuration is required. If you use a different name you will need to supply the name in a cloud-diagnostics.json options file, see below.


The following Bluemix CLI commands will trigger a dump for a Node.js application, list dumps stored on a persistent volume mounted as /var/dumps, and download one of the dumps:

> bx ic exec CONTAINER pkill -RTMIN node
> bx ic exec CONTAINER ls -l /var/dumps
> bx ic cp CONTAINER:/var/dumps/DUMP_FILE_NAME .


Configuration options for the cloud-diagnostics module

You can supply a cloud-diagnostics.json options file in your application’s home directory to control the behaviour of the cloud-diagnostics module. A sample options file is provided, which contains the following default settings:

  {
    "name": "cloud-diagnostics",
    "nodereport": "api+signal",
    "heapdump": "api+signal",
    "coredump": "api+signal",
    "volume": "var/dumps"
  }
  • The nodereport, heapdump and coredump properties control which triggers are enabled for each dump type. The default triggers for all three dump types are “api” and “signal”.
  • The volume property sets the name of the mount path of the persistent storage volume that will be used by the module when writing dump files to NFS storage. The default is “/var/dumps“.

1 comment on"Capturing diagnostic dumps for Node.js applications in Bluemix Container Service"

  1. […] Capturing diagnostic dumps for Node.js applications in Bluemix Container Service […]

Join The Discussion

Your email address will not be published. Required fields are marked *