page-brochureware.php

Monitoring & Diagnostics – Scenarios


 

Problems occurring in Node.js production deployments have a range of symptoms, but can generally be categorized into the following scenarios:

The approach you take to diagnose the problem depends on the scenario, but also on the requirements of the production deployment. The priority is to maintain the availability of the application, which usually involves fail-over to a separate application instance or restart of the failing application.

Techniques used during development, such as attaching a debugger or adding instrumentation to the application, are not usually available in production deployments. The use of tracing or monitoring tools that are not invasive and have minimal impact on the running application might be possible, but the capture of diagnostic information such as logs and dumps at the point of failure is often the most practical approach.

Uncaught exceptions or error events in JavaScript code


Uncaught exceptions or error events in JavaScript code usually result in termination of the Node.js application, with an error message and stack trace written to the console: that is, to the stdout or stderr stream.

For example, the following output is written if a Node.js application cannot find a JavaScript module that it is required to load:


  Cannot find module 'C:\test\unknown'
     at Function.Module._resolveFilename (module.js:469:15)
     at Function.Module._load (module.js:417:25)
     at Module.runMain (module.js:604:10)
     at run (bootstrap_node.js:394:7)
     at startup (bootstrap_node.js:149:9)
     at bootstrap_node.js:509:3
  

 


Tooling suggestions:

Excessive memory usage (memory leak), which might result in an out-of-memory (OOM) error


Excessive memory usage by a Node.js application is often detected by scripts that use operating system facilities, or by production monitoring tools. Sometimes the application fails as it reaches a limit configured in Node.js or in the operating system, producing error output, such as the following example:


  <--- Last few GCs --->
     7299 ms: Mark-sweep 33.0 (64.2) -> 33.0 (64.2) MB, 72.2 / 0.0 ms (+ 14.6 ms in 13 steps since start of marking, biggest ms) [allocation failure] [GC in old space requested].
     7391 ms: Mark-sweep 33.0 (64.2) -> 33.0 (64.2) MB, 92.8 / 0.0 ms [allocation failure] [GC in old space requested].
     7485 ms: Mark-sweep 33.0 (64.2) -> 33.0 (46.2) MB, 93.5 / 0.0 ms [last resort gc].
     7580 ms: Mark-sweep 33.0 (46.2) -> 33.0 (46.2) MB, 95.4 / 0.0 ms [last resort gc].

  <--- JS stacktrace --->
  Security context: 0000037D843CFB61 <JS Object>
     2: my_listener [C:\test\heap_oom.js:~9] [pc=00000255FA5B3A8D] (this=000001240487E6E9 <a Server with map 000002302D633F89>,request=000001240487E5F9 <an IncomingMessage with map 000002302D636EF9>,response=000001240487E581 <a ServerResponse with map 00000F79>)
     3: emitTwo(aka emitTwo) [events.js:106] [pc=00000255FA522153] (this=0000037D84304381 <undefined>,handler=000003F40AEE39F

  FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
  

In the case of excessive memory usage, the error output is not usually sufficient for you to diagnose the problem, as it cannot provide details of the complex memory use by the application. You must take additional steps to capture more information, and use appropriate tooling to analyse it.


Tooling suggestions:

Unresponsive application, which might indicate a looping or hanging application


An unresponsive Node.js application can be detected if you use watchdog facilities in the production environment, or by the application users themselves.

You can initially use operating-system commands such as ps (on Linux systems) to find out if the application is looping (indicated by high CPU usage) or waiting (indicated by low CPU usage).


Tooling suggestions:

Poor performance


Performance issues are detected in the same way as for an unresponsive Node.js application: by watchdog facilities in the production environment or by the application users themselves. There might be a response-time issue, or excessive use of CPU or memory by the application.


Tooling suggestions:

Crash or abort in JavaScript or native code


If a Node.js application crashes or aborts in JavaScript or native code, the symptoms are minimal. The application stops immediately, but usually produces at least a simple message on the stdout or stderr stream. The key diagnostic technique is to capture a core-dump. Operating-system configuration settings such as ulimit (on Linux systems) might be needed for a core-dump to be written.


Tooling suggestions:

  • Use the LLDB debugger with the llnode v8 plugin. If a core-dump can be captured, the LLDB debugger with the llnode v8 plugin can be used to obtain a native stack trace showing the point of failure, and allowing other data such as register and memory values to be obtained. See Exploring Node.js core dumps using the llnode plugin for lldb.

Unexpected application behavior or functional issue


Symptoms in this case depend on the application, but generally results in incorrect output, observed during routine testing or later by application users.


Tooling suggestions:

  • Use the built-in Node.js --trace*, --log*, and --print* options to obtain more information about the application.
  • Use the LLDB debugger with the llnode v8 plugin. Capture a core-dump by using an operating-system command such as gcore (on Linux systems). You can use the LLDB debugger with the llnode v8 plugin to investigate data inside the application (Javascript object properties) and to examine JavaScript application code. See Exploring Node.js core dumps using the llnode plugin for lldb and Advances in core-dump debugging for Node.js.

Trademarks and affiliations

Node.js is an official trademark of Joyent. IBM SDK for Node.js is not formally related to or endorsed by the official Joyent Node.js open source or commercial project. Java, JavaScript and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Apple, Macintosh, and Mac OS are trademarks of Apple Inc., registered in the United States and other countries. Other company, product and service names may be trademarks or service marks of others.