Update 28th Feb 2017: The LLNODE plugin for LLDB is now available for easy installation as an npm module https://www.npmjs.org/package/llnode.

Traditional core dumps continue to be a vital diagnostic in production environments, and their value is equally applicable to Node.js. They provide rapid and reliable capture by the operating system of the memory image of a failing application. The application can be re-started immediately and problem analysis using the core dump can proceed off-line. As introduced in Future directions for diagnostics in Node.js production environments, development work in the IBM Runtimes team is focused on improving the support for analysis of core dumps on Linux, Mac OSX and Windows platforms by contributing to the LLDB debugger and Fedor Indutny’s LLNODE debugger plugin open-source projects. The LLNODE project provides a plugin to the LLDB debugger allowing JavaScript stacks and heap objects to be analyzed from a core dump.

Howard Hellyer has provided an introduction to the use of the LLDB debugger in his post Exploring Node.js core dumps using the llnode plugin for lldb and describes new commands available in the LLNODE plugin findrefs command for llnode and nodeinfo command for llnode. He also provided a key enhancement to the underlying LLDB debugger API to allow all areas of memory in a Node.js process to be accessed from the core dump. This means that with LLDB version 3.9 and later, JavaScript objects in the V8 heap can be examined using the new ‘v8 findjsobjects’, ‘v8 findjsinstances’ and ‘v8 findrefs’ plugin commands, see the example later in this blog.

Debugger commands available for Node.js core-dump analysis

Support for Node.js core-dump analysis has been available for some years on illumos-based operating systems (e.g. SmartOS) via the mdb_v8 plugin for MDB. The aim is to bring the LLNODE plugin commands up to the same level of functionality. A current comparison between the two is as follows:

 

MDB V8

 

LLNODE

 

Display the JavaScript/C++ stack trace

 

::jsstack

-v option provides verbose output, including source file, line
number, object addresses and source code

::jsframe displays
an individual stackframe

::walk jsframes
displays just the frame pointers

v8 bt [<number of frames>]

default output includes source file, line number, object addresses

source code can be displayed using ‘frame select <frame number>’
followed by ‘v8 source list’

List objects in the JavaScript heap

::findjsobjects

listed by object with same properties

provides address of a sample object, number of objects, number of
properties and constructor name

v8 findjsobjects

listed by object type name*

provides number of objects, total size and type name

Display a JavaScript object

<address>::jsprint
[<property name>]

provides source file name, property names and values

-d option provides deeper analysis

<property name> displays just that property

<address>::walk jsprop
lists property values for an object

v8 print <object address>

v8 inspect <object address>

provides constructor name, property names and values

-m option provides the map address

–full-string and –string-length output options

Find JavaScript objects by type name

::findjsobjects -c <type name>

 

v8 findjsinstances <type name>

-m option provides the map address

–full-string and –string-length output options

Find JavaScript objects by property name

::findjsobjects -p <property name>

 

v8 findrefs -n <property name>

Find references to a specified JavaScript object

::findjsobjects -r <object address>

 

v8 findrefs <object address>

Display the constructor name for an object

::jsconstructor
<object address>

-v option provides constructor’s function object address

v8 inspect (object address>

provides the constructor name but not the function object address

List JavaScript functions

[<address>]::jsfunctions

-n, -s and -x options filter by function name, script filename or
instruction address

n/a

Display JavaScript source code

<function address>::jssource

-n options controls number of source lines displayed

v8 inspect -s <function address>, or

v8 source list for current stackframe, see ‘v8 bt’ above

Display closure variables for a function

::jsclosure
<function address>

n/a

Display address of buffer memory

<buffer object address>::nodebuffer

v8 inspect <buffer object address>

Display summary Node.js information

n/a

v8 nodeinfo

Using LLNODE plugin commands with the LLDB debugger for Node.js heap memory analysis

LLDB versions 3.9 and 4.0 are recommended for use with core dumps from Node.js applications. The LLDB installation packages are available from http://apt.llvm.org/ The following steps show how LLDB and the LLNODE plugin can be used to investigate heap usage and examine JavaScript objects on the heap.

  1. Install LLDB 3.9
    The following commands show installation on Ubuntu 14.04.

    wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
    sudo apt-add-repository "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main"
    sudo apt-add-repository "deb-src http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9 main"
    sudo apt-get update
    sudo apt-get install clang-3.9 lldb-3.9
    sudo apt-get install liblldb-3.9-dev
    
  2. alternatively, install LLDB 4.0
    The following commands show installation on Ubuntu 16.04.

    wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
    sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-4.0 main"
    sudo apt-get update
    sudo apt-get install python-lldb-4.0
    sudo apt-get install clang-4.0 lldb-4.0
    
  3. Install the LLNODE debugger plugin for LLDB see https://www.npmjs.org/package/llnode.
    npm install llnode
    
  4. Open the core dump in the LLDB debugger and load the LLNODE plugin

    lldb node -c core
    (lldb) target create --core "core"
    Core file '/home/rchamberlain/core' (x86_64) was loaded.
    (lldb) plugin load ./node_modules/llnode/llnode.so
    (lldb) v8 help
     Node.js helpers, the following subcommands are supported:
      bt              -- Show a backtrace with node.js JavaScript functions and their args. An optional argument is accepted; if
                         that argument is a number, it specifies the number of frames to display.
      findjsinstances -- List all objects which share the specified map.
                         Accepts the same options as `v8 inspect`
      findjsobjects   -- List all object types and instance counts grouped by map and sorted by instance count.
      findrefs        -- Finds all the object properties that reference the specified value.
                         Flags:
                          * -v, --value expr     - all properties that refer to the specified JavaScript object (default)
                          * -n, --name  name     - all properties with the specified name
                          * -s, --string string  - all properties that refer to the specified JavaScript string value
      inspect         -- Print detailed description and contents of the JavaScript value.
                         Flags:
                          * -F, --full-string    - print whole string without adding ellipsis
                          * -m, --print-map      - print object's map address
                          * --string-length num  - print maximum of `num` characters in string
      .... etc
    
    For more help on any particular subcommand, type 'help <command> '.</command>
    
  5. Use the 'v8 findjsobjects' command to obtain a table of JavaScript object types in the V8 heap showing memory usage. In this example a relatively large number of 'MyRecord' objects are consuming 18Mb of heap space, which may indicate a memory leak and is worth further investigation:

    (lldb) v8 findjsobjects
     Instances  Total Size Name
     ---------- ---------- ----
              1         24 FastBuffer
              1         24 JSON
              1         24 Math
              1         48 TickObject
              1         96 Console
              1        112 Server
              1        120 ServerResponse
              1        224 Socket
              1        240 IncomingMessage
              2         48 process
              2         64 HTTPParser
              2         64 Signal
              2        208 EventEmitter
              2        208 WriteStream
              2        256 TimersList
              2        272 Module
              3         96 TCP
              3         96 TTY
              3         96 Timer
              3        864 WritableState
              4        448 BufferList
              4        448 CorkedRequest
              4       1152 ReadableState
              7        720 (ArrayBufferView)
              9        360 EventHandlers
             16        608 (anonymous)
             35        904 (Object)
             46       2944 NativeModule
            107       3424 (Array)
            607      32912 Object
         386396   18547008 MyRecord
    
  6. To examine instances of a JavaScript object, use the 'v8 findjsinstances' and 'v8 inspect' commands. For example to find the memory addresses of the 'MyRecord' objects, and display details of their contents, use the 'v8 findjsinstances' and 'v8 inspect' commands as follows:

    (lldb) v8 findjsinstances MyRecord
    0x00003f8278bffd81:<Object: MyRecord>
    0x00003f8278bffdb1:<Object: MyRecord>
    0x00003f8278bffde1:<Object: MyRecord>
    0x00003f8278bffe11:<Object: MyRecord>
    0x00003f8278bffe41:<Object: MyRecord>
    0x00003f8278bffe71:<Object: MyRecord>
    0x00003f8278bffea1:<Object: MyRecord>
    0x00003f8278bffed1:<Object: MyRecord>
    0x00003f8278bfff01:<Object: MyRecord>
    0x00003f8278bfff31:<Object: MyRecord>
    0x00003f8278bfff61:<Object: MyRecord>
    0x00003f8278bfff91:<Object: MyRecord>
    0x00003f8278bfffc1:<Object: MyRecord>
    ....etc
    0x00003f8278c10b81:<Object: MyRecord>
    0x00003f8278c10bb1:<Object: MyRecord>
    
    (lldb) v8 inspect 0x00003f8278bfffc1
    0x00003f8278bfffc1:<Object: MyRecord properties {
        .name=0x00003f8278b5c571:<String: "foo">,
        .id=<Smi: 128>,
        .account=<Smi: 98454324>}>
    
  7. The types and contents of the MyRecord objects may give a good clue to the JavaScript application code which is causing the problem. Alternatively, the 'v8 findrefs' command can be used to find out what is keeping the MyRecord objects alive:
    (lldb) v8 findrefs 0x00003f8278c10bb1
    0x1142b037fdf9: (Array)[24083]=0x00003f8278c10bb1
    

    So this shows that the MyRecord objects are referenced from a large array, which has 24,083 elements. By searching the application code for occurances of 'MyRecord', we can find the offending application code. In this example we can conclude that the value of 'batch_size' is very large:

    var list = [];
    for (var i = 0; i < batch_size; i++) {
        list.push(new MyRecord());
    }
    

1 comment on"Advances in core-dump debugging for Node.js"

  1. […] Once the core dump has been downloaded, you can open it with a debugger. The LLDB debugger with the LLNODE plugin allows you to examine JavaScript stacks, code and heap objects in a core dump. For information on how to read and analyze a core dump from a Node.js application see advances in core dump debugging for node.js […]

Join The Discussion

Your email address will not be published. Required fields are marked *