This post is part of a series of posts created by the two newest members of our Developer Advocate team here at IBM Cloud Data Services. In honour of the book Seven Databases in Seven Weeks by Eric Redmond and Jim R. Wilson, we challenged Lorna and Matt to take a new database from our portfolio every day, get it set up and working, and write a blog post about their experiences. Each post reflects the story of their day with a new database. We’ll update our seven-days GitHub repo with example code as the series progresses. —The Editors

etcd logo
A distributed key-value store with some neat consistency features: it’s etcd!
  • Database type: Fast and reliable central key-value store
  • Best tool for: Persisting configuration settings that need to be updated across distributed systems

A [Config] State of Mind

etcd is a durable key-value store that provides consistency guarantees, making it good for coordinating configuration state across many distributed services.

The database usually runs as a cluster to improve scalability. When you create your deployment on Compose, they set up the cluster for you: three database nodes and two proxies. etcd has a HTTP interface that speaks JSON, so it works well cross-platform, and it also comes with a command-line tool, etcdctl.

In this article we’ll look at using etcd running on Compose and how to store and change values using the HTTP interface and command-line tool. We’ll also show a simple example of an application getting its config settings and config changes from etcd.

Getting Started

We’ll begin by setting up an etcd cluster on Bluemix. You can absolutely configure your own cluster to use with this tutorial, but since we recommend running the cluster with an odd-numbered set of servers, it’s not trivial. (The Raft consensus algorithm shows why odd-numbered is a good idea.)

We’re taking a shortcut by using Bluemix, which configures a cluster with three database nodes: one will always be the leader, and in the event of server failure, the rest of the cluster will renegotiate and reconfigure. This design is entirely intentional. It gives our applications good reliability and high availability. A system built this way can handle a lot of load, and the platform is designed to scale as you grow.

Set Up an etcd Cluster

Log into your Bluemix account and add a new service from the Catalog; you’ll find “Compose for etcd” as an option under the “Data and Analytics” section. Once the database has initialized, use the “Service Credentials” tab to get the details we’ll use to access it:

  1. First we’ll need the uri value so that we can access the service
  2. Nex†, get the value of the ca_certificate_base64 field. This is a base64-encoded version of the SSL certificate required to connect to your etcd
  3. Base64-decode the certificate and you should get a string with BEGIN CERTIFICATE and END CERTIFICATE near the start and end. Save this string into a file called cert.

We’re all set to start connecting to the etcd installation.

Access etcd over HTTP

With the details you have acquired, we can use curl to talk to etcd: this command refers to the cert file you created in the previous section, and uses the uri value from the service credentials tab as well:

$ curl --cacert cert https://root:password@bluemix-sandbox-dal-9-portal.4.dblayer.com:18929/v2/keys
{"action":"get","node":{"dir":true}}

We don’t have any keys or values stored yet, so let’s add some using etcd’s HTTP API. We’ll be using this to store credentials that our app will need. Start by setting a key config/api/username using a request like this:

$ curl -X PUT --cacert cert \
https://root:password@bluemix-sandbox-dal-9-portal.4.dblayer.com:18929/v2/keys/config/api/username \
-d  value="apiuser"
{"action":"set","node":{"key":"/config/api/username","value":"apiuser","modifiedIndex":17,"createdIndex":17}}

We can also check this specific key using curl by making a GET request to its endpoint:

$ https://root:password@bluemix-sandbox-dal-9-portal.4.dblayer.com:18929/v2/keys/config/api/username

And we’ll see:

{"action":"get","node":{"key":"/config/api/username","value":"apiuser","modifiedIndex":38,"createdIndex":38}}

Using this RESTful interface over HTTP with JSON means that this API can be easily managed from any technology stack. In fact, quite a few of them will offer wrappers or libraries for easier syntax (such as the Node.js example later on in this article).

Use etcd From The Command Line with etcdctl

There is an excellent command-line tool for etcd called etcdctl, which you may prefer over making the curl requests as we have done so far. It is available as part of the etcd package itself. To install it, get the newest release of etcd from GitHub and follow the installation instructions for your platform. (You don’t need to start the etcd command.)

Before using the tool, we need to indicate which version of the API we speak. Set the version number to an environment variable:

ETCDCTL_API=3

Now we can start adding more keys. How about a password to go with the username we set in the previous section? The command is already created for you in the Service Credentials tab we accessed earlier, in a field called uri_cli.

$ etcdctl --ca-file [CA_CERT_FILE] --no-sync --peers https://bluemix-sandbox-dal-9-portal.4.dblayer.com:18929,https://bluemix-sandbox-dal-9-portal.5.dblayer.com:18929 -u root:NIFCXDROLSQFLHFJ ls /

etcdctl creates and parses the JSON for us. Instead of outputting the detailed response, as we get when we make the raw HTTP commands ourselves, etcdctl gives us a more humanely-formatted response. The etcdctl utility lets us perform many useful tasks. For a comprehensive set of instructions, run etcdctl with no arguments and you’ll see the usage instructions. For now, here’s a quick cheatsheet:

  • ls / – see all the top level keys
  • ls /config/api see all the keys in the /config/api directory
  • get /config/api/username see the value of /config/api/username
  • set /config/api/username write to this key, creating it if it doesn’t exist and overwriting it if it does
  • rm /config/api/username remove this key

We can use this tool to update config as we wish, and then use it in our applications.

Building a Simple Config Manager in Node.js

To demonstrate how you might use etcd, we are going to build a simple config manager in Node.js. The aim is to have our app load the necessary config on start-up, and then wait for changes to happen and apply them.

Let’s get started!

Connecting

There is currently no official library for etcd in Node.js; however, there are plenty of good unofficial libraries out there. We are going to be using node-etcd.

First up, we’ll get ourselves connected to etcd. Thankfully, this is very simple. Here’s our connect.js:

# connect.js

// Load in the ETCD module
var Etcd = require('node-etcd');

// SSL cert
var fs = require('fs');
var options = {
  ca: fs.readFileSync('cert')
};

// Connect!
var etcd = new Etcd("https://root:password@bluemix-sandbox-dal-9-portal.4.dblayer.com:18929", options);

// Export
module.exports = etcd;

We need to reference the SSL certificate we mentioned earlier and pass that in when we make the connection, which we achieve using the options parameter in the example above. Once that’s done, we are connected and ready to go! We have exported the connection at the bottom of the file so we can reuse this bit of code across our whole application.

If you are following along, make sure that you enter your own credentials in place of our old ones above. Also, don’t forget to download your SSL certificate!

Creating Our Config

All of our config data, for all of our services, is going to reside in etcd — which means we need to create more config data to make this example interesting.

We’ll have two services — an api and a website — and each of these services needs config for username, password and hostname. So using the examples above, let’s set some keys as shown below (values are shown in parentheses):

/config/website/username (webuser)
/config/website/password (webpass)
/config/website/hostname (example.com)
/config/api/username (apiuser)
/config/api/password (apipass)
/config/api/hostname (api.example.com)

Now that we have our config, it’s time to build something that will use it.

Loading our config

First things first: We need to decide which service we want to load config for and connect to etcd.

# etcd.js
// get our connection
const etcd = require('./connect.js');

// helper functions to parse the responses
const parseConfig = require('./parsing.js').parseConfig;
const changeEvent = require('./parsing.js').changeEvent;

// use optimist to read command line arguments
const argv = require('optimist').argv;

// define our service
const service = argv.service || null;
if (service === null) {
    console.error("Please provide a --service parameter");
    process.exit(0);
}

// define our service key
const serviceKey = "/config/"+service;

Here we are using the connect.js we created earlier to create a connection to etcd. Then, we use the excellent optimist module to listen for command line arguments. We’re particularly interested in the --service argument to tell us which service we should load config for so that we can create our serviceKey. This is the key we will be loading from etcd later on.

You may also notice that we have a couple of helper functions: parseConfig and changeEvent. We have created these functions to help keep this example tidy and easy to follow. Feel free to check out parsing.js to see what’s going on there.

Now, let’s load our config!

# etcd.js continued..

// get our config on wakeup
etcd.get(serviceKey, function(err, data) {

    // handle the error
    if (err) throw (err);

    // parse the initial config load using helper function
    var config = parseConfig(data);

    // log out what the config is
    console.log("========= CONFIG LOADED ==========");
    console.log(config);

});

All we’re doing here is asking etcd to load the config for our service. Using the .get() method, we are asking for the data belonging to either the /config/api or /config/website key, depending on which service was specified at run time via the --service argument. Then we use the parseConfig() function mentioned earlier to create a simple JSON object and log it out to the console window.

Lets see what happens!

$ node etcd.js --service api
========= CONFIG LOADED ==========
{ username: 'apiuser',
  password: 'apipass',
  hostname: 'api.example.com' }

Perfect! We’ve successfully loaded our config. If we specified the website service, we would see the website-specific config instead. Why not try it out?

Listening for config changes

Configs change over time, which means that at some point we are going to want to update the data we have stored in etcd. But what then? Luckily for us, etcd allows you to listen out for changes to your data and then act upon them!

If we were to extend our example:

# etcd.js continued..

// get our config on wakeup
etcd.get(serviceKey, function(err, data) {

    // handle the error
    if (err) throw (err);

    // parse the initial config load using helper function
    var config = parseConfig(data);

    // log out what the config is
    console.log("========= CONFIG LOADED ==========");
    console.log(config);

    // watch for changes
    const changes = etcd.watcher(serviceKey+"/username", null, { recursive: true });

    // when something changes
    changes.on("change", function(change) {

        // parse the changes using a helper function
        var changeData = changeEvent(change);

        // make our changes
        config[changeData.key] = changeData.value;

        // log out the changes
        console.log("========= CONFIG CHANGED ==========");
        console.log(`${changeData.key} changed from ${changeData.previousValue} to ${changeData.value}`);
        console.log(config);

    });

});

Here we are using etcd’s ability to listen for changes using the .watcher() method of our etcd connection, creating a Node.js EventEmitter in the process. We specify the key we want to listen to via our serviceKey, and flag that we want to monitor all keys in this group by using the recursive: true option.

Whenever a change affects our config, changes will emit a “change” event, and we can act upon it. In this example, we are parsing the change data to give us a value, previousValue and the key that has changed before logging this out along with our updated config.

So, what does this look like? If we start our app up again, we should still see the original output:

$ node etcd.js --service api
========= CONFIG LOADED ==========
{ username: 'apiuser',
  password: 'apipass',
  hostname: 'api.example.com' }

But then if we change one of the values — for instance, /config/api/username — we should see this change reflected in the output from our app:

========= CONFIG CHANGED ==========
username changed from apiuser to newuser
{ username: 'newuser',
  password: 'apipass',
  hostname: 'api.example.com' }

So there we have it!: using etcd as a distributed data store to update configuration values in a Node.js app. Try running multiple instances of your app to see how this could really be helpful when dealing with large distributed systems. You’ll never have out-of-date config again!

Conclusion

On the face of it, etcd looks like quite a simple tool — a key-value store that you can read/write to via HTTP — but it really is so much more:

  • It’s fast, benchmarked at over 10k reads/second
  • It’s reliable, using the Raft consensus algorithm to provide a highly-available distributed setup along with network partition tolerence
  • It’s secure, with TLS and SSL support

Add in the ability to monitor changes and react to them (as opposed to polling for changes), and you can really start to see the benefits of using etcd to create distributed management systems for config state on bigger platforms. The tools for the command line are written in Go, so they work well across platforms.

Will etcd be a good fit for your system? Well, your mileage should and will vary, but given the free trial period on Compose, and this tutorial showing how you to begin, you can now find out!

Head to https://github.com/ibm-cds-labs/seven-days for example code and other database tutorials in this series. Cheers.

Join The Discussion

Your email address will not be published. Required fields are marked *