Learn how to use MongoDB Client-Side Field Level Encryption

Introduction

Protecting sensitive data from unauthorized use and keeping it away from bad actors is a common requirement of most enterprises, and there are multiple ways of achieving data security, including:

• User-/role-based permissions that restrict access to certain fields to subsets of users
• Encryption of data in transit that protects data while it travels across networks
• Encryption of data at rest, which protects data while it sits on storage disks

You can also encrypt data even before saving it to the database to guarantee that no one -- even the operators of the database, like employees of IBM or any other cloud service provider -- can see the data without the decryption keys. When encrypting data in this way, you make your data secure but also difficult to work with because it becomes impossible to search for specific documents when everything is garbled by the encryption.

In this tutorial, I will show you how to work with a useful feature of MongoDB: Client-Side Field Level Encryption (CSFLE). It allows you to encrypt specific fields within documents and at the same time search for documents that match these encrypted fields.

CSFLE works by encrypting data before it gets transmitted over the network (on the client) and stored in the database. Similarly, data retrieved from the database arrives encrypted at the client and only then is it decrypted. Encryption/decryption is done by the client using an encryption key. In a production situation, this key will be held in a key management service like IBM Key Protect for IBM Cloud. For the purposes of this tutorial and for simplicity, we will use a key held locally in a text file.

Any MongoDB client with access to the database will be able to see the encrypted fields as cypher text. Only MongoDB clients with access to the encryption key will be able to encrypt/decrypt these sensitive fields. Crucially, fields can be encrypted using deterministic algorithms that allow the fields to be indexed and searched. As an application developer, you can create indexes against the encrypted fields and when you add documents, these will be indexed in the usual way. When retrieving documents, you can simply issue the usual find() commands, and the client takes care of the rest, encrypting the query fields as needed before sending to the database, where the correct documents are then retrieved, sent back and decrypted for the user to see.

We leaned heavily on the MongoDB tutorial called How to use MongoDB Client-Side Field Level Encryption (CSFLE) with Node.js in the writing of this tutorial, but we tried to simplify it and provide some IBM-specific help, like how to set up the infrastructure using Terraform.

Following along, you will:

• Create the MongoDB instance in the IBM Cloud using Terraform. CSFLE is only available to MongoDB Enterprise users, so we will create an Enterprise instance.
• Install the client-side Mongo libraries needed for encryption/decryption.
• Create an encryption key from a key held in a local file.
• Write some medical records with some fields encrypted.
• Fetch one of those records using the encrypted client to see all the data.
• Fetch the same record using the unencrypted client to see how the encrypted data appears to an unauthorized user.

Prerequisites

You will need the following:

• An IBM Cloud account
• Access to a Mac or Linux terminal and some basic command-line knowledge
• Git
• Terraform -- We will use Terraform to deploy all the required infrastructure
• Node.js and npm, as well as basic knowledge of Node.js scripting

Estimated time

This tutorial will take a few hours to complete. It is not cost-free because there is no free tier for MongoDB Enterprise on the IBM Cloud, but if you de-provision your resources after completing it, it will not cost more than a few dollars.

Steps

Step 1. Obtain an API key to deploy infrastructure to your account

Follow the steps in on IBM Cloud for Creating an API key to create a key and make a note of it.

Step 2. Clone the repo and access the Terraform directory

git clone https://github.com/IBM-Cloud/mongo-csfle.git
cd mongo-csfle/terraform

Create a document called terraform.tfvars with the following fields:

ibmcloud_api_key = "<your_api_key_from_step_1>"
region = "eu-gb"
admin_password  = "<make_up_a_password>"

The terraform.tfvars document contains variables you may want to keep secret so that it is ignored by the GitHub repo.

Step 3. Create the infrastructure

Create the MongoDB instance in the IBM Cloud by running the Terraform script:

terraform init 
terraform apply --auto-approve

The Terraform folder contains a number of simple scripts:

• main.tf tells Terraform to use IBM Cloud.
• variables.tf contains the variable definitions whose values will be populated from terraform.tfvars.
• mongo.tf creates the MongoDB Enterprise Edition instance.

It will take up to an hour for the resources to be ready, but you should now you have an IBM Cloud Databases for MongoDB instance. You can check by visiting the Resources section of your IBM Cloud account.

The Terraform script will output several bits of information that you will need for the next steps.

Step 4. Install client libraries

While the infrastructure is provisioning, you can open new terminal windows to install a couple of MongoDB libraries needed for your client encryption.

libmongocrypt

Follow the instructions for your operating system in the MongoDB documentation.

mongocryptd

This is a binary that comes inside the installation download of MongoDB Enterprise. Installing it requires downloading the TAR file, finding the mongocryptd binary in the /bin folder and moving it into your machine's path (normally /usr/local/bin, but you should check that with echo $PATH).

cp /path/to/mongocryptd  /usr/local/bin

Step 5. Install dependencies and obtain database configuration parameters

At this point, you should have a MongoDB instance ready to use and all the required libraries in your local machine. Now we are going to use the database parameters generated by the Terraform script. In your terminal, make sure you are in the terraform folder of the project and type:

terraform output -json > ../code/config.json

This will create a file in the code folder with parameters that our code will use later.

Now navigate to the code folder in in the project and install all the Node.js dependencies by typing:

npm install

Step 6. Create an encryption key for the Mongo client

The next step is to create an encryption key. MongoDB CSFLE uses an encryption strategy called envelope encryption, in which keys used to encrypt/decrypt data called data encryption keys are encrypted with another key called the master key. In a real-life production environment, a master key would be generated and stored by a key management service, such as IBM Key Protect. For simplicity in this tutorial, we are going to use a master key generated and stored locally.

In the code folder, create a file called master-key.txt and put a 96-character string in it:

echo "86oestfjSAC5UZNFfj6xwsxxRjVt1iL8U9xRUl6nk4lteLzbVlYqpZwmIvpNhMPCBKCtdKxpQ8R41tzqat9WWck3WYBI0HVO" > master-key.txt

Now run the make-data-key.js script:

node make-data-key.js

This script reads in the master key from the file and, using the Mongo encryption libraries, creates a data key, which will be used to encrypt all data from then on. This key is saved in a "vault" database created for that purpose inside MongoDB. To make life easier for us, we are saving the ID of that key locally in a file called keyid.txt.

Step 7. Writing encrypted data

Now we are going to write 1,000 medical records to the database. The fictitious medical data was generated using the Datamaker tool. The records are all of the form:

{
        "name": "Suzi Haynes",
        "ssn": 423287316,
        "bloodType": "AB+",
        "medicalRecords": [{
                "weight": 140,
                "bloodPressure": "165/76"
        }],
        "insurance": {
                "provider": "Medicare",
                "policyNumber": 307319
        }
}

What we want to do is encrypt sensitive data like the social security number, blood type, and medical records objects, as well as the insurance policy number. We do this by running the encrypt.js script:

node encrypt.js

There are a lot of things going on here, so let's examine them.

1. The script first reads in 1,000 records from the records.txt file and turns them into valid JSON that can be uploaded to the database.
2. It then creates an encryption-enabled connection to the database (handled by the mongoencryptedclient.js script). The important thing to look at in this client is the schema map. This is the fundamental component of the encryption model. Every time we instantiate a MongoDB client that can handle encrypted data, we pass it a schema map, which is a description of what we want to do with every field in the document that we want to encrypt. We also give it the data key we created in the previous step to perform the encryption/decryption operations on these fields.
3. This schema map is created by the schemamap.js script. In our case, you can see that we are telling it to encrypt the SSN, bloodType, and insurance.policyNumber fields, as well as the entire medicalRecords object. You will notice that some fields are encrypted using a deterministic algorithm and others using a random algorithm. To index and retrieve by a particular field you want to encrypt, it has to be encrypted deterministically. Randomly encrypted fields cannot be queried or indexed.
4. Finally, the encrypt.js script creates an index on two of the fields (SSN and policyNumber) and uploads the data. The indexing step is not strictly necessary as the data set is relatively small and you wouldn't notice if all the docs were simply scanned, but all real-life uses of MongoDB require an index, so we put it in and used one of them.

So the data is in, but how do you know it's been encrypted?

Step 8. Read data

There are two reading scripts. The first one, encryptedread.js, contains an encryption-enabled client. When you run it, it will try to retrieve a single document by its social security number, which is an encrypted field:

node encryptedread.js

You will get the full record in plain text. The client took the SSN, encrypted it using the right key, then sent the read command to the database. The database retrieved the record and sent it, encrypted, back to the client, which then unencrypted the data and displayed it.

{
  _id: new ObjectId("62da80d199622ef999361a0e"),
  name: 'Genevie Rains',
  ssn: 374007263,
  bloodType: 'O',
  medicalRecords: [ { weight: 195, bloodPressure: '152/93' } ],
  insurance: { provider: 'MaestCare', policyNumber: 319024 }
}

Now try the unencryptedread.js script. This one has a client with access to the database, but no access to the encryption key or the schema map. We can't retrieve the record by the SSN because it is encrypted, but we can get the same record using the patient's name, Genevie Rains.

node unencryptedread.js

You will get a record back, and you will be able to see the unencrypted fields, like name and insurance provider. But the rest is a lot of garbled junk that cannot be deciphered. Only clients with access to the key and schema map can see the encrypted field data.

{
  _id: new ObjectId("62da80d199622ef999361a0e"),
  name: 'Genevie Rains',
  ssn: new Binary(Buffer.from("011bffb4b73a11401888b2b4b9c3aff34110b2d20e2f5aa0fce1fa11886bb87249e9d776231a2968e6485ec027ca0786676e47e0362f7b30890f7e88f4b66155b931c9fbf037f83ce5caa6bef19ca029a3ee", "hex"), 6),
  bloodType: new Binary(Buffer.from("021bffb4b73a11401888b2b4b9c3aff341028af7ea44675fc6588df8eab54dc124deb37bc0e26612c693fff847f0a5448c7a029621eec249ed24047ae66cfab5cb58a17ff9330c71d2a29e7c946a8f1b5fe1", "hex"), 6),
  medicalRecords: new Binary(Buffer.from("021bffb4b73a11401888b2b4b9c3aff341041853cd146ac17e7c2502fce467388947a045b7a0bc9d48850daada234e0022a794507a4d0e7a76cea05f1f01ba7c9dd909a1fad52a379eb14e81f16b182c75c18280f6cd73c5686872d41e9b120fb18535904987729d7b12a7bc00106cbad8c644a1dce7d1143587250ff695aa76b30c", "hex"), 6),
  insurance: {
    provider: 'MaestCare',
    policyNumber: new Binary(Buffer.from("011bffb4b73a11401888b2b4b9c3aff34110a39a85de0be2cbf872a649908dd699ae6cf3748a645f673349b9ba2f7130ba742e5c0c9696dce0bb9b5c968ac80b48957836f59e812aebc3cb3833abebd7e560", "hex"), 6)
  }
}

Summary

Client-Side Field Level Encryption (CSFLE) is a powerful tool that MongoDB provides to encrypt specific field data while allowing that data to be accessed/indexed/queried) as if it was unencrypted. Application developers can code the database access commands in their usual way, and the Mongo drivers take care of the encryption/decryption under the hood. You can take advantage of this feature by provisioning an instance of MongoDB Enterprise Edition on the IBM Cloud.

And finally, remember to de-provision your resources to stop incurring charges by going into the terraform folder of the project and typing:

terraform destroy