One of my first tasks here at IBM Cloud Data Services was to blog about my first impressions of RethinkDB. One thing that article briefly touched upon was RethinkDB’s unique ability of being able to push updates to your app as and when the data changes — making it a strong contender to be your database of choice when building real-time apps.

This was something that got my attention back then, and it’s about time we revisited RethinkDB’s push functionality to see just how easy it is to build a real-time app.

The Challenge

As a Developer Advocate, a large part of my job is going out into the community and delivering talks on a range of topics. These talks invariably end with a Q&A session where I can clear up anything that was confusing or misunderstood.

Live Q&As seem like a good use-case for building a real-time app: allow attendees to ask questions from their smartphone during the talk, and vote on which questions they want answered. We can then update the list of questions on-screen in real time, showing the most popular questions and any answers that surfaced during the talk.

Screenshot of the live Q&A voting app we're going to build using RethinkDB
The live Q&A app we’ll build using Node.js and RethinkDB.

Tools

Node.js logo

Node.js is what I am going to use to build this app. Node’s ability to deal with a large amount of concurrent connections is something that stands it in good stead when building real-time apps. Although it’s not something we need to consider in this post, it’s good to design for future scale.

RethinkDB logo

RethinkDB is our database. As mentioned above, we are going to be making use of the changefeeds functionality to push any changes to our app as and when they happen. We will also be looking at ReQL and how that works with Node.js. You can start up a free RethinkDB instance with Compose to get you going.

For the front end we will be using Vue.js and Bootstrap. Vue is one of many Javascript Frameworks for building front ends. I prefer it to something like Angular, as I think Vue is easier to get up and running. Bootstrap, of course, is the popular HTML/CSS framework from Twitter.

We still need to be able to get this data from our app to the front end, and to do that we will be using Socket.IO. This is a simple way to implement WebSockets in your Node.js app, but it will also take care of any cross-browser/platform issues for you. We will also use this Socket.IO component for Vue, so that we can easily incorporate Socket.IO into our Vue app.

Set Up

All the code below can be found in the rethinkdb-questions GitHub repository for this article.

Clone the repo and npm install to get all of the dependencies and follow along below!

app.js is the brains of our whole app. We are using Express to help us get up and running quickly. The first portion of the file is just including dependencies and so on.

Once we get to line 25 or so, we have some configuration to do. We need to define a connection object for RethinkDB. There are two connection objects defined in the code at the moment: one for if you are using a local RethinkDB instance, and one if you are using a hosted instance via Compose. Uncomment the one you wish to use, and if you are using Compose, make sure you enter your connection details!

On the topic of Compose connection details for RethinkDB, note that Compose’s Deployment Overview gives you a proxy connection string that is similar — yet subtly different — than the URL for the RethinkDB admin UI. It’s the difference of a single ., so make sure you’re using the right string for your host.

We will use this connection object every time we create a RethinkDB connection.

RethinkDB and ReQL

Before we dive into the code we should take a bit of time to talk about ReQL, the query language for RethinkDB.

When you create a connection to RethinkDB, this is a permanent, socket-like connection that will stay open until it is closed by the application. This is useful for a couple of reasons:

  • It allows RethinkDB to return a cursor instead of a dataset to allow us to iterate through the data in an efficient manner
  • We can push updates down this connection using changefeeds

We get this cursor by querying the database using ReQL, which is RethinkDB’s native query language. It is designed to embed itself into your code — i.e., if you’re building your app in JavaScript, your ReQL looks like JavaScript — so that it feels familiar and comfortable to the developer. It also fits into the standard coding patterns of whichever language you are using.

It is important to know that even though the query looks like JavaScript (or Ruby, Python, etc.), none of the heavy lifting is being performed in JavaScript, or even by your app. What is happening in the background is that your ReQL expression is being compiled down into a query that RethinkDB understands, sent to the server, and then executed in a distributed fashion across the whole cluster — allowing for performant querying of large datasets.

That being said, lets look at how we can use ReQL in our app.

API Endpoints

We will start with the API, or the back end of the app. The API is going to provide the front end with the ability to get our questions data from the database, as well as add new questions and update existing questions.

The API consists of a collection of Express routes that we will define, which will in turn query our RethinkDB instance. We won’t cover how Express routing works today, however there is a very simple guide on the Express website that should help if you’re unfamiliar.

Adding new questions

Having a real-time Q&A app is no good if we have no questions, so the first thing we need to do is create a way to add some. This is done using the POST /question endpoint.

// Create a new question
app.post("/question", bpJSON, bpUrlencoded, (req, res) => {

    var question = {
        question: req.body.question,
        score: 1,
        answer: ""
    }

    r.connect(connection, function(err, conn) {

        r.table("questions").insert(question).run(conn, (err, cursor) => {

            conn.close();

            if (err) {
                return res.status(404).send({ success: false })
            }

            return res.send({ success: true })

        });

    })

})

We create our question object using the question parameter provided as part of the request. Then we connect to the database, build up our query in ReQL, and run the query. The ReQL portion of this code is here:

r.table("questions").insert(question)

Lets take a minute to examine what this query is doing:

  • r is the RethinkDB namespace
  • We can tag on the table("questions") method (just like JavaScript, remember) to select our desired table
  • And then, we can use the insert(question) method to say we wish to insert a new document, passing in our question object.

Simple, huh?

Once the query completes, we return a simple JSON response, just to signify whether this request was successful or not, and close our connection to the database.

It’s important to close your RethinkDB connection, as you don’t want unused, open connections consuming resources.

Getting question data

Now that we have some questions, we probably want to be able to get them back, right?

The GET /questions endpoint is designed to do just that – return all existing questions from the RethinkDB database in one go.

// Get all questions
app.get("/questions", (req, res) => {

    r.connect(connection, (err, conn) => {

        r.table("questions").run(conn, (err, cursor) => {

            cursor.toArray((err, results) => {

                conn.close();
                return res.send(results);

            })

        });

    })

})

All we are doing here is connecting to RethinkDB, using ReQL to ask for everything from the questions table, and transforming the full dataset into an array which is then sent back to the client in the response. In the meantime, we close our connection to RethinkDB. Again, the ReQL portion of this code is here:

// equivalent to SELECT * FROM questions
r.table("questions")

We previously touched upon the fact that we don’t receive a dataset back from RethinkDB; instead, we receive a cursor. In this instance, we don’t want a cursor. So in the Get all questions snippet we use the toArray() method to return our full dataset.

Updating our questions

We mentioned before that we wanted our users to be able to vote on questions. We actually have two endpoints for voting: POST /upvote/:id and POST /downvote/:id, which will either add or subtract 1 from the score of a question.

Here’s what the ReQL looks like:

// get by ID
// update score to score+1
// default of 1 if no score set
r.table("questions").get(req.params.id).update({
    score: r.row("score").add(1).default(1)
})

In English, we are saying:

  • Get a question by a specified ID
  • Update this question
  • Set the value of score to be score+1
  • If score does not currently exist, then set it to 1.

This is how we are upvoting a question. Similarly, we use .sub(1) in the downvote endpoint.

Finally, a question is no use without an answer. We add answers using POST /answer/:id.

This process is similar to changing the score of a question. All we need to do is find our question by its unique ID and update it to include an answer that is provided via this request:

// get by ID
// update answer
r.table("questions").get(req.params.id).update({
    answer: req.body.answer
})

Front End

Next, we need to define some routes for Express to allow us to access our app via a web front end.

  • GET / is the homepage and will be used to display our questions & answers
  • GET /answer is identical to the homepage, but will allow an administrator to answer questions

Both of these endpoints will return index.html, which is where we will create our front end.

Once jQuery has told us that our document is ready to go, we define our Vue app with the app variable. This is where we can define our data model on the client side, along with a bunch of methods we can call and handlers for our Socket.IO events.

app = new Vue({
  el: '#app',  // the HTML element that this Vue app relates to
  data: {
    questions: [] // our data model, an array of questions
  },
  methods: {  
    ... // a bunch of methods we have defined that can interact with our data model
  },
  computed: {
    ... // some computed values that we can use
  },
  sockets:{ 
    ... // socket event handlers
  }
})

After defining our app, we call the app.getQuestions() method, which will hit the GET /questions endpoint, retrieving all of our questions data. We can then store this data in our data model at app.questions.

It is easy to define how we want our data to be displayed with Vue. The app that we defined above relates to everything that is defined inside div#app.

<div class="container" id="app">
....
</div>

In here we have a button, which toggles a form that we can use to ask a new question. Below the form, we have another div that will house all of our questions.

We can do some powerful things like iterate through our questions array to render a new div for each question. The example below is saying “iterate through the sortedQuestions computed array, and create a new div for each element, exposing this element as question“.

<div v-for="question in sortedQuestions" id="{{ question.id }}" class="col-lg-12">
    ...
</div>

We can then use the special Vue handlebars notation to define the ID of this div like so id="{{ question.id }}". At any point in this div we can refer to the question variable within the handlebars notation to refer to the current question in the array. This gives us access to other properties such as score, answer, and question to help us create our question template in HTML.

We can also call on the methods we defined in our app with the following notation:

<button type="button" v-on:click="upvote(question.id)" href="#"></button>

When this button is clicked, we will call the upvote() method that is defined in our Vue app, passing the unique ID of this question as the only parameter.

Real-Time Updates

We have a number of methods defined in general.js and questions.js on the front end that make requests back to the API endpoints we discussed earlier in the article. We pass these requests using the jQuery ajax APIs.

  • getQuestions() calls GET /questions
  • askQuestion() handles the submission of the question form to POST /question
  • doUpvote() calls POST /upvote/:id
  • doDownvote calls POST /downvote/:id
  • answerQuestion() handles the submission of the answer form to POST /amswer/:id

On closer inspection of these functions, you might notice that when we add or update a question, there is no code to react to the response from the API and update the front end. The reason is that we want to handle any updates to the data in real time, and we want all of our clients to respond to the same stimulus — i.e., an event from our app to tell us that the data has changed in the database. This approach helps us to manage state within our real-time app.

How else could it work? Well, if we have multiple ways of updating the front end — i.e., after making an API call, or after receiving a WebSocket event — then there is a greater chance of introducing bugs or inconsistencies in our code, meaning that our clients could get out of sync. If every client is reacting to data changes from a single source, then there is a much greater chance of consistent results across the whole user base.

Historically, computer systems have been built with a single source of data (a database!), but this doesn’t necessarily work well with data-driven, real-time apps. Traditional databases are not designed for this use-case. To get such a database to work in this context, the developer has to use an antiquated approach such as polling (repeatedly asking the database for an update). Developers could also use message queues and additional infrastructure to help manage the flow of data. Neither of these solutions, however, scales well.

Scaling data-driven web apps is the problem we’re trying to solve by using RethinkDB and changefeeds. Speaking of changefeeds…

Changefeeds & Socket.IO

As mentioned previously, we’ll use of the RethinkDB changefeed feature to get updates from our database whenever a new question is added or updated. Towards the bottom of app.js you should see some code that creates our changefeed.

r.connect(connection, (err, conn) => {

    r.table("questions").changes().run(conn, (err, cursor) => {

        // for each update emit the data via Socket.IO
        cursor.each((err, item) => {

            if (err) return;

            // new
            if (item.old_val === null && item.new_val !== null) {
                io.emit('new', item.new_val)
            }

            // deleted
            else if (item.old_val !== null && item.new_val === null) {
                io.emit('deleted', item.old_val)
            }

            // updated
            else if (item.old_val !== null && item.new_val !== null) {
                io.emit('updated', item.new_val)
            }

        });

    });

});

The first thing to note here is that we are not closing the connection to RethinkDB — this is because we want to keep the connection open so that we can continue receiving updates.

Using changefeeds is similar to doing a normal query in ReQL: you just attach the .changes() method to the end of your query. We still receive a cursor, and we can use that cursor to iterate over any incoming update events.

Update events look like this:

{
    new_val: { ... },
    old_val: { ... }
}

The new_val is what is currently being stored in the database, whilst the old_val is what was previously there. If the old_val is null, then that indicates there was no previous value and this event represents an insert of a new document. If new_val is null, then that indicates a deletion. If both new_val and old_val have data, then this signifies an update has taken place.

In the code example above, you can see that we are determining which event has occurred, and that we are using Socket.IO to emit() the relevant data to the front end via WebSockets. When emitting events via Socket.IO, the first parameter is the name of the event (new, deleted, updated) and the second parameter is the data you wish to send.

Again, we will not examine Socket.IO here, but there is an easy-to-follow article on the official website that shows how to get started with Socket.IO and Express.

In our front end HTML, we have included the Socket.IO client library and a Socket.IO component for Vue:

<script src="/socket.io/socket.io.js"></script>
<script src="/js/vue-socketio.min.js"></script>

We then configure Vue to use Socket.IO and define the location of the server:

// Tell Vue to use Socket.io
var socketUrl = `${location.protocol}//${location.hostname}${(location.port ? ':'+location.port: '')}`;
Vue.use(VueSocketio, socketUrl);

We can then define event handlers for Socket.IO that will listen for events. We have three events:

  • new – a question has been inserted into the database
  • updated – a question has been updated (either answered or voted on)
  • deleted – a question has been deleted from the database

The handlers are defined in the sockets object of the Vue app we created at the beginning of the article. The handlers are simply functions that update the data model of our app to reflect the change in our questions data.

Because of Vue’s data bindings, we don’t have to do anything else — the front end will automatically update to reflect the changed data! Now, whenever the data changes within the database these updates will be reflected, in real time, within our app. Pretty cool, huh?

Conclusion

So there we have it: from nothing to a real-time Q&A app in relatively little time! What have we learnt?

Simply put, RethinkDB is a great place to start building your real-time apps. It provides a single source of data to power your apps that isn’t available from other database offerings without resorting to clumsy polling or complicated architecture to manage the flow of data within your app.

RethinkDB doesn’t give you everything you need to create a real-time app front-to-back. If you want to update users in real time, you still need a way to get your updates to the front end, but RethinkDB does a lot of the heavy lifting for you in a clear and succinct way.

Join The Discussion

Your email address will not be published. Required fields are marked *