Kubernetes with OpenShift World Tour: Get hands-on experience and build applications fast! Find a workshop!

Learn Node.js, Unit 14: Node.js with Cloudant DBaaS

In previous units of the Node.js Learning Path, you set up the Shopping List application with two local database installations, SQLite3 and MongoDB. In today’s technology landscape, it’s also important that you know how to scale your Node.js applications for the cloud. Most enterprises are moving to cloud storage because server hosting is expensive and can lead to availability issues. With cloud computing, you pay for the cloud services you need, so you’re not stuck with expensive hardware you can’t afford.

In this final unit of the Node.js Learning Path, learn how to move your Shopping List application’s data to Cloudant, a NoSQL database-as-a-service (DBaaS) offering from IBM Cloud. The Shopping List application will continue to run locally on your computer, but you’ll use an open source Cloudant driver (@cloudant/cloudant) to access and manipulate the data.

What is Cloudant?

Cloudant is a distributed, document-oriented NoSQL database that is part of IBM Cloud. It is derived from an MIT project started in 2008 and based on Apache CouchDB 2.0. While it’s available as a standalone installation, Cloudant has evolved into a database-as-a-service offering.

Unlike SQLite3 (introduced in Unit 6) or MondoDB (introduced in Unit 12), Cloudant does not use tables or collections for storage. Instead, it stores all of your documents together in the database.

As the developer, you determine how to partition your data for optimized document search and retrieval. As an example, in this unit, we’ll use a type property to distinguish between two different types of documents — item and shoppingList.

Be sure to check out the Cloudant documents to learn more about Cloudant.

Using Cloudant with Node.js

As you work through the example application, you will need to refer to documentation for the following tools:

Requirements for this unit

You will need an IBM Cloud account in order to follow along with this unit. If you already have an IBM Cloud account, you’re all set. If not, you can sign up for a free account. (Click on the Get IBM Cloud Lite button.)

You’ll also need the source code for this unit, which is available on GitHub.

Set up Cloudant

In this section you will set up Cloudant as the database service for the Shopping List application. Just follow the steps below.

1. Log in to the IBM Cloud

Go to the IBM Cloud page to log in.

If you don’t have an IBM Cloud account, go to the IBM Cloud Lite signup page to get your free Lite account. The process is quick, and you’ll be logged in and ready to code in no time.

Lite accounts are limited, but your login will let you explore what’s available from IBM Cloud and run all the code for this unit.

2. Create a Cloudant database service

Once you’ve signed into your IBM Cloud account, you’ll be taken to your IBM Cloud dashboard.

From the hamburger menu in the top left corner, choose Infrastructure. On the next page, click the storage icon, and on the page after that, click Databases from the list of categories. Choose Cloudant from the list of databases on the Databases page.

Next you’ll create your Cloudant service. Give your service a name, such as Node-LP-Cloudant, and choose Use both legacy credentials and IAM as the authentication method, then click the Create button, as shown in Figure 1.

Create a Cloudant service Figure 1. Create a Cloudant service

After you create the Cloudant service, you’ll be taken to your dashboard, where you’ll see the service:

IBM Cloud dashboard Figure 2. IBM Cloud dashboard showing the newly created Cloudant service

3. Create credentials

You will need credentials in order to connect remotely to your Cloudant service. To create your credentials, select your Cloudant service, then select Service credentials from the left menu on the next screen. Select the New credential (+) button, as Figure 3 shows.

Launch Cloudant Dashboard Figure 3. The Service credentials screen in Cloudant dashboard

In the Add new credential dialog, give the new credential a name, select Auto Generate as the Service ID, then click Add.

New Credential dialog Figure 4. The Add new credential dialog

Once the credential has been created, it shows up in the list of service credentials. Choose View credentials, then copy the credentials to a clipboard, as shown in Figure 5.

Copy credential to the clipboard Figure 5. Copied credentials

You will need the copied credentials to create a local configuration file, which will allow your Node.js application to talk to your Cloudant service.

4. Create the service credentials config file (vcap-local.json)

In your editor (mine is VSCode), create a new file and paste in the credentials you just copied from the service credentials screen. You will notice that it is a JSON object, complete with surrounding braces.

Next, copy the following code block and paste it into your editor.

Listing 1.vcap-local.json template

{
    "services": {
        "cloudantNoSQLDB": {
            "credentials":
            PASTE CREDENTIAL INFO HERE
            ,
            "label": "cloudantNoSQLDB"
        }
    }
}

Replace the PASTE CREDENTIAL INFO HERE with the credentials you’ve just pasted into the editor. Include the braces so that you have a well-formed JSON object.

Save the file as vcap-local.json in the Unit-14/config directory. The file should look something like Listing 2. (I have obfuscated my credentials.)

Listing 2. vcap-local.json example

{
    "services": {
        "cloudantNoSQLDB": {
            "credentials": {
                "apikey": "YOUR_API_KEY_HERE",
                "host": "YOUR_USERNAME_HERE.cloudant.com",
                "iam_apikey_description": "Auto generated apikey during resource-key operation for Instance - crn:v1:bluemix:public:cloudantnosqldb:us-south:a/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx::",
                "iam_apikey_name": "auto-generated-apikey-xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
                "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Manager",
                "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
                "password": "YOUR_PASSWORD_HERE",
                "port": 443,
                "url": "https://YOUR_USERNAME_HERE:YOUR_PASSWORD_HERE@YOUR_HOST_HERE.cloudant.com",
                "username": "YOUR_USERNAME_HERE"
            },
            "label": "cloudantNoSQLDB"
        }
    }
}

5. Create a database

Use the hamburger menu to navigate to your IBM Cloud dashboard, then select your Cloudant service. Choose Launch Cloudant Dashboard and you’ll see the dashboard shown in Figure 6.

The Cloudant Dashboard Figure 6. The Cloudant database dashboard

Click on the database icon from the left menu in Figure 6, then select Create Database. Call your database “shopping_list” and click Create. You’ll see a screen like Figure 7.

The shopping_list database Figure 7. The newly created shopping_list database

Now you’re ready to load your application data into the database.

6. Load application data into Cloudant

I’ve provided a data loader program called load-db-cloudant.js in my GitHub repo, which you can run using the npm run load-db script. The output looks like this:

Ix:~/src/projects/IBM-Developer/Node.js/Course/Unit-14 sperry$ npm run load-db

> unit-14@1.0.0 load-db /Users/sperry/home/development/projects/IBM-Developer/Node.js/Course/Unit-14
> node ./utils/load-cloudant

[2018-08-17T23:20:07.351] [INFO] default - 1534566007350:INFO: Connect success! Connected to DB: shopping_list
[2018-08-17T23:20:07.355] [INFO] default - 1534566007355:INFO: mainline(): Initializing Cloudant... Done.
[2018-08-17T23:20:07.359] [INFO] default - 1534566007359:INFO: mainline(): Script start at: 8/17/2018, 11:20:07 PM
[2018-08-17T23:20:07.359] [INFO] default - 1534566007359:INFO: mainline(): Loading data for item...
[2018-08-17T23:20:07.359] [INFO] default - 1534566007359:INFO: loadData(): Loading data files...
[2018-08-17T23:20:07.360] [INFO] default - 1534566007360:INFO: loadData():readableStream.on(open): Opened file: ../data/Grocery_UPC_Database.csv
[2018-08-17T23:20:07.389] [DEBUG] default - 1534566007389:DEBUG: Creating documents: 1-10000
[2018-08-17T23:20:10.506] [DEBUG] default - 1534566010506:DEBUG: Creating documents: 10001-20000
[2018-08-17T23:20:13.591] [DEBUG] default - 1534566013591:DEBUG: Creating documents: 20001-30000
[2018-08-17T23:20:16.693] [DEBUG] default - 1534566016693:DEBUG: Creating documents: 30001-40000
[2018-08-17T23:20:19.804] [DEBUG] default - 1534566019804:DEBUG: Creating documents: 40001-50000
[2018-08-17T23:20:22.902] [DEBUG] default - 1534566022902:DEBUG: Creating documents: 50001-60000
[2018-08-17T23:20:26.009] [DEBUG] default - 1534566026009:DEBUG: Creating documents: 60001-70000
[2018-08-17T23:20:29.136] [DEBUG] default - 1534566029136:DEBUG: Creating documents: 70001-80000
[2018-08-17T23:20:32.253] [DEBUG] default - 1534566032253:DEBUG: Creating documents: 80001-90000
[2018-08-17T23:20:35.335] [DEBUG] default - 1534566035335:DEBUG: Creating documents: 90001-100000
[2018-08-17T23:20:38.433] [DEBUG] default - 1534566038433:DEBUG: Creating documents: 100001-110000
[2018-08-17T23:20:41.511] [INFO] default - 1534566041511:INFO: loadData():readableStream.on(close): Closed file: ../data/Grocery_UPC_Database.csv
[2018-08-17T23:20:41.511] [DEBUG] default - 1534566041511:DEBUG: Creating documents: 110001-110436
[2018-08-17T23:20:44.515] [INFO] default - 1534566044515:INFO: mainline(): Loading item data, done.
[2018-08-17T23:20:44.515] [INFO] default - 1534566044515:INFO: Total item documents loaded: 110436
[2018-08-17T23:20:44.515] [INFO] default - 1534566044515:INFO: mainline(): Script finished at: 8/17/2018, 11:20:44 PM
Ix:~/src/projects/IBM-Developer/Node.js/Course/Unit-14 sperry$

7. Verify the data has loaded

The shopping_list database consists of 110,436 items. You should see this number reflected in the # of Docs in your shopping_list dashboard, as Figure 8 shows.

shopping_list database Dashboard Figure 8. The shopping_list database after the load-db-cloudant data loader has run successfully

To query the data, select the shopping_list database, then click Query from the left menu. Replace the contents of the query window with the following query.

Listing 3. Cloudant querying JSON for a regular expression search

{
   "selector": {
      "itemDescription": {
         "$regex": ".*Free Range.*"
      }
   },
   "fields": [
      "_id",
      "_rev",
      "itemDescription"
   ]
}

This query is seeking documents whose itemDescription property contains the phrase “Free Range.” The query should produce 15 results, as shown in Figure 9.

JSON Query results to test the data load Figure 9. Cloudant query results

You have set up the Cloudant database, verified your data, and everything is working as it should. Next you’ll run the Shopping List application.

8. Run the Shopping List application

Other than being hosted in the cloud, the only difference between the Shopping List application you’ve run in previous units and this one is the system of record. The UI’s look and feel will be the same.

To run the application, go to a terminal window and run npm start, or run npm run start-dev to use nodemon.

Point your browser at http://localhost:3000 to access the main page of the application.

Cloudant Node API

The official Cloudant library for Node.js, @cloudant/cloudant, is a wrapper around a fork of Apache CouchDB. That fork, called cloudant-nano, is maintained by the IBM Cloudant team.

In this section, I show you how to use Cloudant’s Node API calls to interact with the DAO modules in the Shopping List application. We start by connecting to your Cloudant service in the IBM Cloud.

Connect to Cloudant

To connect your Shopping List application with Cloudant, you first need to create a new Cloudant object and pass it the credentials from your vcap-local.json file in Listing 1. You then call the use() function to tell the driver which database you want to use:

Listing 4. Connect to Cloudant using the @cloudant/cloudant Node driver (from ./utils/utils.js)

const Cloudant = require('@cloudant/cloudant');

const vcap = require('../config/vcap-local.json');

function dbCloudantConnect() {
    return new Promise((resolve, reject) => {
        Cloudant({  // eslint-disable-line
            url: vcap.services.cloudantNoSQLDB.credentials.url
        }, ((err, cloudant) => {
            if (err) {
                logger.error('Connect failure: ' + err.message + ' for Cloudant DB: ' +
                    appSettings.cloudant_db_name);
                reject(err);
            } else {
                let db = cloudant.use(appSettings.cloudant_db_name);
                logger.info('Connect success! Connected to DB: ' + appSettings.cloudant_db_name);
                resolve(db);
            }
        }));
    });
}

The dbCloudantConnect() function returns a promise. When that promise is fulfilled, it will contain the database object you need in order to use the API. If the promise is rejected, the Error object is non-null and is logged and used to reject the promise.

I’ve written the DAO implementations to use an IIFE (Immediately Invoked Function Expression) to call dbCloudantConnect(). The call is executed at module-load time:

Listing 5. Call dbCloudantConnect() to connect to Cloudant (from ./models/lists-dao-cloudant.js)

let db;

// Initialize the DB when this module is loaded
(function getDbConnection() {
    logger.info('Initializing Cloudant connection...', 'items-dao-cloudant.getDbConnection()');
    utils.dbCloudantConnect().then((database) => {
        logger.info('Cloudant connection initialized.', 'items-dao-cloudant.getDbConnection()');
        db = database;
    }).catch((err) => {
        logger.error('Error while initializing DB: ' + err.message, 'items-dao-cloudant.getDbConnection()');
        throw err;
    });
})();

When the promise is fulfilled, the database object is passed to then(), where it is saved. The DAO functions in the module will use the db object to make Cloudant Node API function calls.

If getDbConnection() succeeds (that is, it doesn’t throw an Error), the connection is ready to use.

Create a new shopping list

To create a new shopping list, you call the create function in lists-dao-cloudant.js and pass the shopping list’s description, as shown in Listing 6.

Listing 6. Call create() to create a new shopping list (from ./models/lists-dao-cloudant.js)

function create(description) {
    return new Promise((resolve, reject) => {
        let listId = uuidv4();
        let whenCreated = Date.now();
        let list = {
            _id: listId,
            id: listId,
            type: 'shoppingList',
            items: [],
            description: description,
            whenCreated: whenCreated,
            whenUpdated: null
        };
        db.insert(list, (err, result) => {
            if (err) {
                logger.error('Error occurred: ' + err.message, 'create()');
                reject(err);
            } else {
                resolve({ data: { createdId: result.id, createdRevId: result.rev }, statusCode: 201 });
            }
        });
    });
}

This function first creates a new JSON document object called list, which represents the shopping list. The function then populates the list object, including the type, which is set to shoppingList to distinguish it from items (recall that there is no concept of collections in Cloudant), then calls db.insert() to create the document.

When the document has been created, the caller receives a JSON response containing the new document _id (as createdId) and _rev (as createdRevId).

If the document could not be created, the promise is rejected.

Read a shopping list

To fetch a single shopping list by its _id property, call the findById() function in lists-dao-cloudant.js, and pass the value of the document _id you want to retrieve:

Listing 7. Call findById() to retrieve a single shopping list by its _id property (from ./models/lists-dao-cloudant.js)

function findById(id) {
    return new Promise((resolve, reject) => {
        db.get(id, (err, document) => {
            if (err) {
                if (err.message == 'missing') {
                    logger.warn(`Document id ${id} does not exist.`, 'findById()');
                    resolve({ data: {}, statusCode: 404 });
                } else {
                    logger.error('Error occurred: ' + err.message, 'findById()');
                    reject(err);
                }
            } else {
                resolve({ data: JSON.stringify(document), statusCode: 200 });
            }
        });
    });
}

This function calls db.get(), passing the document _id to fetch. If the document _id does not exist, then an empty JSON object is returned, along with HTTP status code 404 (NOT FOUND). Otherwise, the document is returned to the caller.

Update a shopping list

To update a shopping list, you call the update() function in lists-dao-cloudant.js, passing the shopping list’s document _id and the new description value for the update.

Listing 8. Call update() to update a single shopping list (from ./models/lists-dao-cloudant.js)

function update(id, description) {
    return new Promise((resolve, reject) => {
        // Retrieve the list (need the rev)
        findById(id).then((response) => {
            // Parse the stringified JSON
            let list = JSON.parse(response.data);
            // Update the description
            list.description = description;
            list.whenModified = Date.now();
            // Update the document in Cloudant
            db.insert(list, (err, response) => {
                if (err) {
                    logger.error('Error occurred: ' + err.message, 'update()');
                    reject(err);
                } else {
                    resolve({ data: { updatedId: response.id, updatedRevId: response.rev }, statusCode: 200 });
                }
            });
        }).catch((err) => {
            logger.error('Error occurred: ' + err.message, 'update()');
            reject(err);
        });
    });
}

The _rev value of the existing document is required in order to update it, so this function’s first task is to retrieve the document from Cloudant. It then sets the new description and sets whenModified to the current date and time. Finally, it calls db.insert() to update the document. (Cloudant does not have an “update” function because it inserts a new revision of the document and deletes the original.)

When the document has been updated, the caller receives a JSON response containing the updated document ID (updatedId) and the new revision (updatedRevId).

If an error occurs during any of this, the error is logged and the promise is rejected.

Search items by description

To search items by partial description, call the findByDescription() function in items-dao-cloudant.js and pass the partial description value, as shown in Listing 9.

Listing 9. Call findByDescription() to fetch all documents whose itemDescription property contains the specified partialDescription parameter*

function findByDescription(partialDescription) {
    return new Promise((resolve, reject) => {
        let search = `.*${partialDescription}.*`;
        db.find({
            'selector': {
                'itemDescription': {
                    '$regex': search
                }
            }
        }, (err, documents) => {
            if (err) {
                reject(err);
            } else {
                resolve({ data: JSON.stringify(documents.docs), statusCode: (documents.docs.length > 0) ? 200 : 404 });
            }
        });
    });
}

This function first creates a regular expression that contains the specified partialDescription parameter as its search variable. It then calls db.find() to locate all matching documents.

Returned documents are sent to the caller. If an error occurs, the promise is rejected.

That concludes this quick introduction to Node client API calls to connect to the Cloudant database. Check out the Cloudant Query documentation to learn more about querying your Cloudant database.

Conclusion to Unit 14

This unit introduced Cloudant, a document-oriented NoSQL database that is part of the IBM Cloud. I showed you how to set up a Cloudant service in the IBM Cloud, connect Cloudant to the Shopping List example application, and load and verify data from the shopping_list database.

I also introduced you to a handful of API calls in the Cloudant Node.js driver, which you used to interact with DAO modules in the Shopping List application.

To learn more about Cloudant, I encourage you to play around with the code from this unit: run the code in the debugger (from Unit 13 and use it to learn how Node and Cloudant work together. You can also experiment with the code and add new features. As an example, you could try adding different types of partialDescription searches on top of the contains search I’ve provided.

Also check out the video below, which shows you how to set up your Cloudant service, familiarizes you with the Cloudant dashboard, and shows you how to work with the database directly through the dashboard.

Video

Conclusion to the Node.js Learning Path

This unit is the final installment in the Node.js Learning Path, which has been a complete introduction to Node.js development. If you’ve followed the course from start to finish, you should have a firm foundation for your continuing education and practice as a professional Node.js developer.

If you want to continue learning from this course, start by playing with the Shopping List application. Use the source code to deepen your understanding of the code and patterns in Node.js applications; I guarantee you’ll notice things about the code now that you didn’t see when you first started this course! You can also use npm to integrate new tools and packages from the Node ecosystem. And, of course, you should always log, debug, and profile your code for optimal functioning and performance.

J Steven Perry