Introduction

For application problems in production environments, dumps are a vital tool, allowing diagnostic information to be captured when failure events occur, and problem analysis to continue off-line. There are three types of dump available to developers, system administrators or maintainers of Node.js applications in Bluemix:

  1. node-report, a human-readable diagnostic summary, written to file. The report includes JavaScript and native stack traces, heap statistics, platform information and resource usage etc. With the report enabled, reports can be triggered on unhandled exceptions, fatal errors, signals and calls to a JavaScript API.
  2. heapdump, a snapshot of the V8 heap in JSON format, providing a summary of the JavaScript objects in the heap, their types, sizes and references.
  3. core dump, an operating system snapshot of the process memory, as a binary file, including program libraries and data areas.

This article will take you through the steps needed to configure your Bluemix application to produce these dumps for a range of problem scenarios, and how to download and analyze them.

Note [18 July 2017]: In recent Bluemix Cloud Foundry deployments, the ‘proxy’ facility using the BLUEMIX_APP_MGMT_ENABLE environment variable described below has been seen to fail, usually with a console message:
STG/0 WARN: App Management cannot be installed because the start script cannot be found.
This issue is currently under investigation.

Configuring your Bluemix application for dump support

To enable node-report or heapdump in your application, first add the module names and versions into the dependencies section in your application’s package.json file, for example:

"dependencies": {
        .....
        "node-report": "2.1.x",
        "heapdump": "0.3.x"
},

Note: if you are running the IBM SDK for Node.js in your container, the node-report line in the package.json file is not required, as the node-report module is bundled in the SDK.

Next either require the modules explicitly in your JavaScript application, or add them using -r option to your node.js start command. For example, to enable node-report:

"scripts": {
        "start": "node -r node-report app.js"
},

or to enable heapdump:

"scripts": {
        "start": "node -r heapdump app.js"
},

To obtain application core dumps in Bluemix, you need to set a Linux resource limit in your application’s container. The default setting for the size of core files (the ulimit -c setting) is zero. You can set it to a specific size or to ‘unlimited’ by placing a shell script containing the following commands into your Bluemix app/.profile.d directory, for example as file app/.profile.d/ulimit.sh:

echo "Setting core dump size limit to unlimited"
ulimit -c unlimited

Push or restage the application to upload the script. Scripts in the app/.profile.d directory are run by Bluemix before the application is started. The echo command will write a message to the Bluemix console log, which you can check to ensure that the script ran during start-up. For more information see “Configuring the startup environment” in Bluemix: Managing Apps


By default, Bluemix restarts your application container if the application terminates or crashes, deleting any dumps. You can keep the container alive by using the Bluemix application management ‘proxy’ facility. The proxy facility provides application management between your application and Bluemix. When enabled, the buildpack starts a proxy agent that is located between your application’s runtime and container. By enabling proxy, your application container continues to live even if the application crashes. For more information see Bluemix CLI and Dev Tools. The proxy facility is enabled by setting an environment variable, you can do this using the Cloud Foundry command line interface (CLI) as follows:

cf set-env Application_Name BLUEMIX_APP_MGMT_ENABLE proxy
cf restage Application_Name

alternatively, you can set the environment variable using the Bluemix application management console:


You can check that the ulimit and proxy settings have been enabled for your application by viewing the Bluemix application log when your application starts:


Capturing reports and heapdumps on request – using the node-report and heapdump APIs

You may want to capture dumps from Node.js applications while they are running, for example to obtain stack traces or investigate application cpu or memory usage. You can use the node-report and heapdump JavaScript APIs in your application to trigger dumps, while allowing the application to continue running:

var nodereport = require('node-report');
nodereport.triggerReport();

var heapdump = require('heapdump');
heapdump.writeSnapshot();

Log messages on the Bluemix application console show when a dump has been produced:

Bluemix supports the use of the Cloud Foundry SSH facility to securely access the application container. The 'cf ssh' command can be used to set up an interactive SSH session with your application. You then have a Linux command-line shell that you can use to interact with your application. You can use this facility to download reports and heapdumps from your application, for example using the Cloud Foundry command 'cf ssh Application_Name -c "cat ./app/node-report.20170322.120753.70.001.txt" > node-report.20170322.120753.70.001.txt' to download the report shown in the Bluemix log above.

Capturing reports and heapdumps on request – using appmetrics-dash

The appmetrics-dash module lets you connect your Node.js application to a monitoring dashboard providing a comprehensive set of live statistics about the application. It also provides ‘Generate Node Report’ and ‘Generate Heap Snapshot’ buttons allowing you to trigger reports and heap dumps. For more information see appmetrics-dash in Bluemix in 3 easy steps.

Capturing reports and heapdumps on request – using the OS SIGUSR2 signal from a Bluemix SSH session

Bluemix supports the use of the Cloud Foundry SSH facility to securely access the application container. The 'cf ssh' command can be used to set up an interactive SSH session with your application. You then have a Linux command-line shell that you can use to interact with your application.

In the example below, the 'ps -ef' command is used to find out what processes are running in the container, then using the process ID of Node.js application, a SIGUSR2 signal is sent to the application to trigger a heapdump. The SSH session is then closed, and a Cloud Foundry command 'cf ssh Application_Name -c "cat ./app/heapdump..." > heapdump...' is used to download the heapdump.

Capturing core dumps on request – using the OS SIGABRT signal from a Bluemix SSH session

Using the Cloud Foundry SSH facility described above, you can obtain a core dump by terminating your application with a SIGABRT signal. Note that you will need to set the core file ulimit for the application process as described in ‘Configuring your Bluemix application for dump support’.

The following screenshot shows a terminal session logging into the SSH facility, obtaining the process ID of the node application using the 'ps -ef' command, sending the SIGABRT signal to the process, then listing the files in the /app directory to show that the core dump has been created. The SSH session is then closed, and a Cloud Foundry command 'cf ssh Application_Name -c "cat ./app/core" > core' is used to download the core dump.


Note: Sending the SIGABRT signal to the process will terminate it. In some Linux environments you can use the ‘gcore’ command to obtain a core dump from a process while allowing the process to continue execution. This is not supported in the Bluemix environment (the Linux /proc/sys/kernel/yama/ptrace_scope setting prohibits it).

Once the core dump has been downloaded, you can open it with a debugger. The LLDB debugger with the LLNODE plugin allows you to examine JavaScript stacks, code and heap objects in a core dump. For information on how to read and analyze a core dump from a Node.js application see advances in core dump debugging for node.js


Capturing dumps automatically on failure

Node.js applications can be configured in Bluemix to produce dumps automatically on crashes caused by out-of-memory conditions or uncaught exceptions. You will need to configure the Bluemix application management ‘proxy’ facility described in ‘Configuring your Bluemix application for dump support’ above, in order to keep the container alive after a crash.To enable reports on out-of-memory conditions or uncaught exceptions, use the following option to require the node-report module in your application package.json file:

"scripts": {
        "start": "node -r node-report app.js"
},

To enable core dumps on out-of-memory conditions or uncaught exceptions you will need to configure the process core file ulimit setting, as described in ‘Configuring your Bluemix application for dump support’ above. Node.js will then produce a core dump automatically on an out-of-memory error. To produce a core dump on an uncaught exception, you will need to add an additional command-line option to the Node.js application, in the application package.json file:

"scripts": {
        "start": "node --abort-on-uncaught-exception app.js"
},

You can combine both the options above, if you want reports as well as core dumps:

"scripts": {
        "start": "node -r node-report --abort-on-uncaught-exception app.js"
},

Note: currently it is not possible to produce heapdumps on out-of-memory conditions or uncaught exceptions.


The following screenshot of the Bluemix application log shows the messages output when a Node.js application crashes due to a JavaScript heap out-of-memory condition. The messages indicate that both a report and a core dump have been produced. The dumps can be downloaded using the Cloud Foundry SSH facility described above.


For information on how to read and analyze a core dump from a Node.js application see advances in core dump debugging for node.js

Join The Discussion

Your email address will not be published. Required fields are marked *