Digital Developer Conference: Hybrid Cloud 2021. On Sep 21, gain free hybrid cloud skills from experts and partners. Register now

Composing promises in Node.js

My last article on promises focused on the Promises/A+ specification and the ES6 implementation of promises. I showed you the benefits of using promises over raw callbacks and also covered some terminology to help wrap our minds around the concepts. This article builds on the terminology and concepts we talked about there.

Now that you know the basics of promises, let’s take it a step further and focus on composing promises in Node.js, which I define as functional programming and asynchronous control flow using promises.

Promise chaining

The then method allows us to chain promises (see 3.2.6 in the Promises/A+ spec). The value returned from a chain of promises is itself a promise. This returned promise resolves to the value returned from the last onFulfilled or onRejected handler in the chain.

Let’s look at some examples:

let chainPromise = getActiveUser()
  .then(JSON.parse)
  .then(getSubscriptionsForUser)
  .then(JSON.parse)
chainPromise.then(console.log, console.error)

Here, chainPromise could either be:

  1. Fulfilled with a list of parsed subscriptions for the active user since that is the last onFulfilled handler in the chain
  2. Rejected at any point of the chain since there are no onRejected handlers

The same is true for nesting; this accomplishes the same task as above:

let nestedPromise = getActiveUser().then(userJson => {
  let user = JSON.parse(userJson)
  return getSubscriptionsForUser(user).then(subJson => {
    return JSON.parse(subJson)
  })
})
nestedPromise.then(console.log, console.error)

Armed with this knowledge, you can create a recursive chain that calls a function forever until an error occurs.

function forever(fn) {
  return fn().then(function() {
    return forever(fn) // Re-execute if successful
  })
}
// If an error occurs, log and done
forever(doThis).catch(console.error)

You might wonder: Won’t this blow the stack? No, because Promises/A+ requires the onFulfilled and onRejected handlers to trigger on a future turn in the event loop after the stack unwinds (3.2.4 in Promises/A+).

Starting chains and grouping

Promises provide some tools out-of-the-box to aid in composition. Two of these out-the-box tools are Promise.resolve and Promise.all.

Promise.resolve(value) converts a value into a promise that is automatically fulfilled with that value. This is useful for a couple reasons:

  • First, if you have a function that could return a value or promise, you can make your API consistent by turning the value into a promise, so the caller will always expect that.
  • Second, Promise.resolve helps you start promise chains.

Here is an example:

Promise.resolve('monkeys').then(console.log) // Will log 'monkeys'

The second tool is Promise.all. It takes an array of promises and returns a new promise, which I refer to as a group promise. The group promise is either:

  1. Resolved when all the promises resolve
  2. Rejected when any reject.

Promise.all is helpful for grouping the fulfillment values from promises, regardless of if the execution is in a series or parallel:

let groupPromise = Promise.all([doThis(), doThat()])
groupPromise.then(([resultOfThis, resultOfThat]) => {}, console.error)

Promise.all maintains the ordering of the results array, so the result of doThis() would be index 0 and so on. If either promise rejects, then the groupPromise also rejects and you could log it with console.error.

Working with collections

Concurrent maps

Let’s look at iterating through collections of data that require an asynchronous action per element. Like Array.prototype.map, which synchronously deals with each element in an array, you can write a function that performs an asynchronous action on each element using promises.

function promiseMap(xs, fn) {
  // Execute the function for each element in the array and collect the results
  let promises = xs.map(function(x) {
    return fn(x)
  })
  return Promise.all(promises) // Return the group promise
}

You can shorten this useful utility down to one line:

let promiseMap = (xs, fn) => Promise.all(xs.map(x => fn(x)))

You then can use this function as such:

const fs = require('fs')
let stat = util.promisify(fs.stat)
promiseMap(['list', 'of', 'files'], stat).then(console.log, console.error)

The beauty of this approach is that even non-promise-returning functions will work. For example, let’s say you wanted to stat the files you do not already have in your cache:

let cache = Object.create(null)
function statCache(file) {
  if (cache[file]) return cache[file] // If exists, return cache.
  let promise = stat(file)

  // Here we introduce a promise side-effect, we cache the stat.
  promise.then(stat => (cache[file] = stat))
  return promise
}
promiseMap(['list', 'of', 'files'], statCache).then(console.log, console.error)

Here, statCache returns a value or a promise. Regardless of what’s returned, you can group it and provide the results with Promise.all. Sweet!

How does this work? Promise.all internally transforms all values returned into promises.

Serial maps

I talked about iterating through a collection with promises concurrently, but what about iterations in series? To do this, we programmatically create a promise chain:

let promiseMapSeries = (xs, fn) => {
  // Create a empty promise to start our chain.
  let chain = Promise.resolve()

  let promises = xs.map(
    // Execute the next function after the previous has resolved.
    x => (chain = chain.then(() => fn(x)))
  )
  // Group the results and return the group promise.
  return Promise.all(promises)
}

The example above uses Promise.all to group operations done in series and Promise.resolve to start a promise chain. Each time through xs.map, I built a larger chain and returned a promise for that point in the chain until I reached the end of the array.

If this code is unraveled, it looks something like this:

let promises = []
let series1 = Promise.resolve().then(first)
promises.push(series1)
let series2 = series1.then(second)
promises.push(series2)
let series3 = series2.then(third)
promises.push(series3)
// ... etc
Promise.all(promises)

When structured this way, we maintain the order in our results and the serial execution.

Combining recursion and composition

Now that you understand some of the basics, let’s write a program that uses some of the skills acquired earlier to recursively traverse a directory tree while asynchronously providing the caller with a list of file paths.

The program executes as so:

node traverse.js [STARTING DIRECTORY]

Here is the implementation:

const fs = require('fs')
const {promisify} = require('util')

let readdir = promisify(fs.readdir)
let stat = promisify(fs.stat)

// Create our helper function.
let promiseMap = (xs, fn) => Promise.all(xs.map(x => fn(x)))

function readTree(path) {
  return (
    readdir(path) // Read directory at path.
      .then(files =>
        promiseMap(files, file => {
          let fpath = path + '/' + file

          // Get file stats.
          return stat(fpath).then(stat => {
            // If we have a directory, recurse into it.
            if (stat.isDirectory()) return readTree(fpath)
            return fpath
          })
        })
      )
      // Flatten any nested arrays and sort.
      .then(paths => paths.flat().sort())
  )
}

readTree(process.argv[2]).then(console.log, console.error)

If you want to execute this serially, you would swap out promiseMap with promiseMapSeries. Nice!

Going further with promises

Your best grasp of these concepts will come by playing around with them. Here are some ideas to get you started:

  1. Try implementing promiseMap where it limits the number of items it handles concurrently. Then, try implementing it for promiseMapSeries.
  2. Turn setTimeout into a promise-returning function called delay. Then, use delay to create a new promiseMapSeries function that adds a delay between calls.

For more examples of chaining, grouping, and using recursion with promises, this gist implements other functional concepts.