Edit June 2017:
Prebuilding node-rdkafka is no longer necessary. See https://developer.ibm.com/messaging/2017/06/23/messagehub-node-apps-no-prebuilt/

node-rdkafka and librdkafka

node-rdkafka is an interesting Node.js client for Apache Kafka that works well with IBM Message Hub.

 

Node-rdkafka is a wrapper of the C library librdkafka
that supports well the SASL protocol over SSL that client applications need to use to authenticate to Message Hub.

 

Due to the SASL and SSL requirements, there are a few considerations that users need to be aware of:

 

  1. Microsoft Windows is not a supported platform, as librdkafka can’t be built with SASL support on Windows.
  2. MacOS users need to install Homebrew and its openssl package.
  3. Bluemix users need to package a prebuilt node-rdkafka binary.
  4. Code that runs on both Bluemix and local workstations should use ‘require’ in an “if/else” conditional (see sample at the bottom).

On MacOS, build using Homebrew openssl

To be built with SSL support, librdkafka needs to be compiled and linked against Homebrew’s version of openssl.
An easy way to do this is to export these environment variables in your ~/.bash_profile file:

export CPPFLAGS=-I/usr/local/opt/openssl/include
export LDFLAGS=-L/usr/local/opt/openssl/lib

as Homebrew packages get symlinked by default in /usr/local/opt .
To compile the code, you will need Apple’s XCode command line tools installed.

Package a prebuilt node-rdkafka binary for Bluemix

Bluemix currently does not yet include the libsasl2-dev package that is needed to build librdkafka with SASL support.
Pushing in Bluemix a Node application that requires node-rdkafka causes the node-rdkafka package from npm to be built in Bluemix. The build will fail because of the missing libsasl2-dev library.

 

The workaround documented in this post consists of using a prebuilt binary node module:

 

  1. Build node-rdkafka on a machine running Ubuntu 14.04 x86_64 LTS (a VM will do, of course).
    The machine must have libsasl2-dev and libssl-dev installed, and Node.js installed at the level targeted by your Bluemix Node application in its package.json file.
  2. The node installation must have the node-gyp module installed.
  3. To build node-rdkafka just run npm install node-rdkafka
  4. Copy the following files from the Ubuntu build machine to your application’s node_modules directory:

    node_modules/node-rdkafka/build/Release/node-librdkafka.node
    node_modules/node-rdkafka/build/Release/rdkafka_cpp.a
    node_modules/node-rdkafka/build/Release/rdkafka.a
    node_modules/node-rdkafka/lib/*
    node_modules/node-rdkafka/librdkafka.js
    node_modules/node-rdkafka/package.json

  5. In your application, rename the directory node_modules/node-rdkafka to node_modules/node-rdkafka-prebuilt
  6. Edit the file node_modules/node-rdkafka-prebuilt/package.json changing the name and removing everything in the scripts block:


    “name”: “node-rdkafka-prebuilt”,

    “scripts”: {},

  7. Update your application’s package.json file to identify node-rdkafka-prebuilt as a local dependency:

    “node-rdkafka-prebuilt”: “file:./node_modules/node-rdkafka-prebuilt/”

  8. Make sure that your .cfignore file, if you are using one, excludes /node_modules/node-rdkafka-prebuilt from being ignored. E.g.:

    node_modules/*
    !node_modules/node-rdkafka-prebuilt

  9. In your application’s code, use var Kafka = require('node-rdkafka-prebuilt')

 

The application code using the local node-rdkafka module can now be pushed to Bluemix, which will not try to build node-rdkafka, resolving instead the dependency against the local binaries.

Enable the application with a prebuilt node-rdkafka to also run locally

Using the workaround shown above, the Node application will resolve node-rdkafka against the local binary also outside of the Bluemix environment.

 

This might fail on a different Linux platform and will definitely fail on MacOS.

 

To use the prebuilt binary only in Bluemix and use instead the correct node module when running on a developer’s machine, you can use the following workaround.

 

Edit your application’s package.json file, adding node-rdkafka in the devDependencies section:

“devDependencies”: {
….,
“node-rdkafka”: “0.6.x”
},

Then, in your application code, require the dependency in a conditional branch, for example checking for the presence of Bluemix’s VCAP_SERVICES environment variable, e.g. :


var Kafka = {};
if (process.env.VCAP_SERVICES) {
// Running in Bluemix
Kafka = require('node-rdkafka-prebuilt');
....
} else {
//Running locally on development machine
Kafka = require('node-rdkafka');
...
}
...
//to authenticate with MessageHub some connections options must be specified ...
var options = {
'metadata.broker.list': ... // e.g from VCAP_SERVICES or local environment,
'security.protocol': 'sasl_ssl',
'ssl.ca.location': /etc/ssl/certs, //or export from Keychain Access on MacOS
'sasl.mechanisms': 'PLAIN',
'sasl.username': ..., // e.g from VCAP_SERVICES or local environment
'sasl.password': ..., // e.g from VCAP_SERVICES or local environment
.... //other client options
};
var consumer = new Kafka.KafkaConsumer(options);
...

With this approach, your code will use the appropriate node-rdkafka module depending on where it’s running.
Hopefully this is all you need to get started writing Message Hub applications with node-rdkafka.

Example code

To see it all in action, please see this sample project:
IBM Message Hub kafka-nodejs-console-sample

 

 

Please try it out and feel free to leave comments on this blog or on github.com

 

 

Edoardo Comar, IBM Message Hub development (why doesn’t my display name get updated on this blog 🙂 ) ?

 

Thanks to Mickael Maison for his work on using node-rdkafka with Message Hub and Bluemix.

13 comments on"Node.js applications for Message Hub with node-rdkafka"

  1. Edoardo Comar June 28, 2017

    Kyungmo, thanks for noting this – which is a recent development :
    https://developer.ibm.com/messaging/2017/06/23/messagehub-node-apps-no-prebuilt/

  2. Edoardo Comar May 30, 2017

    You can’t change the Bluemix runtime.
    You need to prebuild your binary on a system that matches the Runtime.
    You may want to try using a VM with a clean Ubuntu 14.04 installation – and please make sure that the version of Node in the VM matches the one that you specified in your package.json

    Hopefully all these gimmicks will soon be not needed as libsasl2-dev is finding its way in the runtime.

    • Kyungmo Jung June 28, 2017

      Now, Bluemix may not require prebuilt the module as it contains the libs. I could push successfully without prebuilt modules. Please see installed packages in node.js app container;
      vcap@c6b4d55a-e6b3-40d2-6874-f19189e4c8f4:~$ apt list –installed | grep libsasl

      WARNING: apt does not have a stable CLI interface yet. Use with caution in scripts.

      libsasl2-2/trusty,now 2.1.25.dfsg1-17build1 amd64 [installed,automatic]
      libsasl2-dev/trusty,now 2.1.25.dfsg1-17build1 amd64 [installed]

  3. Edoardo Comar May 30, 2017

    Hi Harit,
    it looks that the binary you built for some reason depends on a library not available in the Bluemix runtime.

    • Harit Patel May 30, 2017

      Is it possible for me to somehow install the library in the runtime?
      Or perhaps remove this dependency all together?

  4. Harit Patel May 30, 2017

    This doesn’t seem to work for me. I am using an Ubuntu machine and Node 4.2.6. I appreciate the help!

    Here are the error logs:

    2017-05-30T09:45:43.01-0400 [APP/0] ERR Debugger listening on port 9229.
    2017-05-30T09:45:43.01-0400 [APP/0] ERR Warning: This is an experimental feature and could change at any time.
    2017-05-30T09:45:43.01-0400 [APP/0] ERR To start debugging, open the following URL in Chrome:
    2017-05-30T09:45:43.01-0400 [APP/0] ERR chrome-devtools://devtools/remote/serve_file/@60cd6e859b9f557d2312f5bf532f6aec5f284980/inspector.html?experiments=true&v8only=true&ws=
    2017-05-30T09:45:43.05-0400 [APP/0] ERR Warning: You will need a SSH tunnel for port 9229 to be able to use the Chrome DevTools to remotely debug your app, then use the chrome-devtools URL which can also be found in this output.
    2017-05-30T09:45:43.10-0400 [APP/0] ERR [Tue May 30 13:45:43 2017] com.ibm.diagnostics.healthcenter.loader INFO: Node Application Metrics 1.2.0.201611300818 (Agent Core 3.1.0)
    2017-05-30T09:45:43.17-0400 [APP/0] ERR (node:141) DeprecationWarning: process.EventEmitter is deprecated. Use require(‘events’) instead.
    2017-05-30T09:45:43.39-0400 [APP/0] ERR /home/vcap/app/node_modules/bindings/bindings.js:83
    2017-05-30T09:45:43.39-0400 [APP/0] ERR throw e
    2017-05-30T09:45:43.39-0400 [APP/0] ERR ^
    2017-05-30T09:45:43.39-0400 [APP/0] ERR Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.21′ not found (required by /home/vcap/app/node_modules/node-rdkafka-prebuilt/build/Release/node-librdkafka.node)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at Error (native)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at Object.Module._extensions..node (module.js:597:18)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at Module.load (module.js:487:32)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at Function.Module._load (module.js:438:3)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at Module.require (module.js:497:17)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at Module.newFunc (/home/vcap/app/vendor/node/lib/node/appmetrics/lib/aspect.js:100:24)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at require (internal/module.js:20:19)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at bindings (/home/vcap/app/node_modules/bindings/bindings.js:76:44)
    2017-05-30T09:45:43.39-0400 [APP/0] ERR at Object. (/home/vcap/app/node_modules/node-rdkafka-prebuilt/librdkafka.js:10:32)

  5. Jeremy Roy April 24, 2017

    Mickael,

    I had been using Eclipse in the DevOps toolchain to deploy my changes with success before. I tried “cf push” this time, and it seems to be working. I saw “invalid” next to the prebuilt package during staging but I was able to start the app and I’m sending and receiving from message hub successfully.

    Thanks!
    Jeremy

  6. Jeremy Roy April 21, 2017

    This worked perfectly for me back in February. Bluemix was just updated though, and this doesn’t seem to work anymore. I can see that Node.js was updated to 6.10.0; not sure if anything else changed. I tried re-building node-rdkafka-prebuilt under node 6.10 and the same Ubuntu version, but it’s not working. I get errors that the package can’t be found, which is what happened in the past when I pre-built using the wrong OS.

    Was anyone able to get this working in the new BM containers?

    Thanks.

    • Mickael Maison April 24, 2017

      Hi Jeremy,
      I just cloned again https://github.com/ibm-messaging/message-hub-samples, updated the manifest under kafka-nodejs-console-sample and did ‘cf push’ and it worked for me.

      Can you share the error message you see ?

      • Jeremy Roy April 24, 2017

        The logs are below. I’m using node-rdkafka in a Node-Red application. The error message is the same as what I was getting back in February when I made the node-rdkafka-prebuilt package using the wrong Ubuntu version. I assume the “no such file or directory” message indicates the package wasn’t recognized.

        2017-04-24T10:30:31.66-0400 [STG/0] OUT npm ERR! addLocal Could not install /tmp/app/node_modules/node-rdkafka-prebuilt
        2017-04-24T10:30:31.66-0400 [STG/0] OUT npm ERR! Linux 4.4.0-45-generic
        2017-04-24T10:30:31.66-0400 [STG/0] OUT npm ERR! argv “/tmp/app/vendor/node/bin/node” “/tmp/app/vendor/node/bin/npm” “install” “–unsafe-perm” “–userconfig” “/tmp/app/.npmrc”
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! node v6.10.0
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! path /tmp/app/node_modules/node-rdkafka-prebuilt
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! code ENOENT
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! errno -2
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! syscall open
        2017-04-24T10:30:31.67-0400 [STG/0] OUT
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! enoent ENOENT: no such file or directory, open ‘/tmp/app/node_modules/node-rdkafka-prebuilt’
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! enoent This is most likely not a problem with npm itself
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! enoent
        2017-04-24T10:30:31.68-0400 [STG/0] OUT npm ERR! Please include the following file with any support request:
        2017-04-24T10:30:31.68-0400 [STG/0] OUT npm ERR! /tmp/app/npm-debug.log
        2017-04-24T10:30:31.71-0400 [STG/0] OUT —–> Build failed
        2017-04-24T10:30:31.71-0400 [STG/0] OUT
        2017-04-24T10:30:31.71-0400 [STG/0] OUT Check our support community to get help on common issues:
        2017-04-24T10:30:31.71-0400 [STG/0] OUT http://ibm.biz/bluemixcommunitysupport
        2017-04-24T10:30:31.71-0400 [STG/0] OUT http://ibm.biz/bluemixsupport
        2017-04-24T10:30:31.71-0400 [STG/0] OUT
        2017-04-24T10:30:31.75-0400 [STG/0] ERR Staging failed: Exited with status 223
        2017-04-24T10:30:31.85-0400 [API/10] ERR Failed to stage application: staging failed
        2017-04-24T10:30:31.67-0400 [STG/0] OUT npm ERR! enoent ENOENT: no such file or directory, open ‘/tmp/app/node_modules/node-rdkafka-prebuilt’
        2017-04-24T10:30:31.72-0400 [STG/0] OUT Exit status 223
        2017-04-24T10:30:31.72-0400 [STG/0] ERR Failed to compile droplet

  7. Sarah Bienenstock January 24, 2017

    Also, in step 4 I needed to copy over the packages from node_modules/node-rdkafka/node_modules/, otherwise it hit errors trying to find bindings library

  8. Sarah Bienenstock January 24, 2017

    Please note that at step 4, when copying the files over to node-rdkafka. It should not be copied into an existing node-rdkafka project, but rather node-rdkafka should be an empty directory. Otherwise, the cf push will end up rebuilding this project leading to SASL build errors

Join The Discussion

Your email address will not be published.