This is part 2 of the Serverless approach to Weather Underground series, describing our team’s steps over the past year to modernize our Weather Underground website.

Previously, I wrote in Part 1: A serverless approach to Weather Underground about the process of creating cloud functions to modernize our Radar Map products in order to move away from legacy backend code and APIs. The next phase of this migration effort is to generate and write all resulting assets to IBM Cloud Object Storage instead of AWS S3.

Prerequisites

This article assumes that you have working experience with JavaScript and have some familiarity with cloud terminology. To understand the context, and the serverless architecture that object storage works with, read Part 1 of this series.

Estimated time

Take 20 minutes to read the sections of this article and consider some of the tools and approaches that we used at Weather Underground. Invest more time if you want to try to work with cloud object storage in your own environment.

Using IBM Cloud Object Storage with IBM Cloud Functions

To be able to read and write files to IBM Cloud Object Storage, you need to provision Object Storage as a service at cloud.ibm.com/resources. Here are the steps to create the resource:

  1. At cloud.ibm.com/resources, click Create resource.
  2. Click Storage and select Object Storage.
  3. Give service an unique name, and click Create.

Here’s an example of the “Weather Labs” service that I created:

screen capture of the object storage service in resource list

Then, after you have Object Storage provisioned, you need to create a bucket. A bucket is where files are stored. In the following example, you can see that I created a bucket for each of our environments (production, development, and staging):

screen capture of buckets for the object storage service

So you have the proper permissions to read and write to the Object Storage service, you need to go to Service credential and click New credential. Here’s an example of what I did:

screen capture of credentials for the object storage service

You also need to update your /.aws/ credential file with your newly created access_key_id and secret_access_key like I did in the following example:

screen capture of credential file

Now that your Cloud Object Storage is set up, you can use the ibm-cos-sdk library to refactor Cloud Functions to work with it. This library is a fork of the AWS S3 SDK, and its API is fairly compatible and similar.

Run the following command to install ibm-cos-sdk to your project:

npm install --save ibm-cos-sdk

To initialize the Cloud Object Storage client, you need to provide additional configuration info (the endpoint, the ibmAuthEndpoint, and the serviceInstanceId) like the following example:

object storage configuration

Moving vanilla JavaScript serverless functions to TypeScript

While I was porting the cloud functions code to work with cloud object storage, I found it tedious that each action was a separate project and that I couldn’t share code between them. I have been itching to move the existing code to TypeScript, and this issue gives me a very good reason to do so.

I decided to refactor the ES6 code to use TypeScript instead, for better code reuse and smaller bundle file sizes. I was inspired by Van Staub’s post: Building IBM Cloud Functions with TypeScript. This section describes my experience learning how to use TypeScript for Cloud Functions and the benefits.

Because node-canvas, one of the dependent libraries, does not have TypeScript support yet, I am in a holding pattern for converting one of the actions to TypeScript until TypeScript support is added for that dependent library. Hopefully, that won’t be long – it looks like that PR for node-canvas is about to be merged.

First, consider the reason for moving to TypeScript. Currently, I have each of the cloud functions written in ES6, and they are housed in a separate project, which makes reusing code difficult. To deploy the function, I basically zip up the entire project and push it up to the cloud. I use the following commands:

zip -r intellicast-animate-action.zip * && bx wsk action update ${package}/intellicast-animate-action - kind nodejs:8 - memory 2048 - timeout 280000 intellicast-animate-action.zip

The project structure looks something like the following example:

project structure

You can see this is a standard node application structure, and the code and packages are repeated in each project. I want to move to TypeScript for the following reasons:

  • To take advantage of TypeScript features
  • To catch issues early in development and build time
  • To leverage tools and editors (such as autocomplete and intellisense)
  • To write safer, more efficient, and reusable code

Porting cloud functions to TypeScript

By porting the existing cloud functions to TypeScript, I was able to house all of the actions and share the same cos-client.ts library file among all the actions in a single project, as you can see in the following screen capture:

project structure with TypeScript

With the following webpack.config.js file, the number of entries depends on the number actions compiled and bundled:

const path = require('path');
const CopyPlugin = require('copy-webpack-plugin');
const dist = 'dist';  // be aware 'dist' folder is also used for tsconfig output

module.exports = {
  entry: {
    'intellicast-action': `./src/intellicast-action.ts`,
    'intellicast-purge-action': `./src/intellicast-purge-action.ts`
  },
  output: {
    path: path.resolve(__dirname, dist),
    filename: '[name].js',
  },
  module: {
    rules: [{
      test: /\.tsx?$/,
      use: 'ts-loader',
      exclude: /node_modules/
    },
    {
      test: /\.node$/,
      loader: "native-ext-loader"
    }],
  },
  plugins: [
    new CopyPlugin([
      {
        from: 'dist/cos-client.js',
        to: '../intellicast-animate-action/cos-client.js',
        toType: 'file',
      },
      {
        from: 'dist/params.js',
        to: '../intellicast-animate-action/params.js',
        toType: 'file',
      },
      {
        from: 'dist/messenger.js',
        to: '../intellicast-animate-action/messenger.js',
        toType: 'file',
      },
      {
        from: 'dist/intellicast-animate-action.js',
        to: '../intellicast-animate-action/index.js',
        toType: 'file',
      }
    ]),
  ],
  resolve: {
    extensions: ['.js', '.ts', '.tsx', '.json']
  },
  mode: 'production',
  target: 'node',
  node: {
    __dirname: true
  },
  externals: [{
    // pre-defined modules computed below
  }]
}

TypeScript not only helps produce a smaller codebase – it also generates a much smaller output bundle as a result of code reuse and tree shaking.

Here is the comparison between the two output file sizes:

output file sizes

Consider the TypeScript version of intellicast-action. After transpiling with tsc and bundling with Webpack magic, the output of intellicast-action.js is about 368k, as opposed to 2.8mb for the ES6 compressed version intellicast-action.zip. It’s quite a difference in size.

Ease of deployment with manifest.yaml

I also adopted manifest.yaml for deploying the actions rather than using command lines as I have previously employed. Here is an example of my manifest.yaml file:

manifest.yaml file

Within manifest.yaml, there are references to environment variables (denoted by ${...}) for any sensitive data that the development team wants to hide from the public. I created an npm script to set up the necessary environment variables before running the deployment process with manifest.yaml. See the following example of the deploy.js script:

deploy.js script

The package.json scripts section contains the following code:

package.json scripts

If you run the following command, you see a result like the screen capture of weather-stage.

npm deploy

weather-stage actions screen capture

Benefits of typescript

If you’re like me, after so many years of coding in JavaScript, sometimes it’s hard to move away from it because it just feels so natural. That’s why I wrote the cloud functions in JavaScript when I started working on this migration project for our Intellicast Radar Maps. As the code base grows in size, I started to feel the pain, and now I find myself repeating code because JavaScript doesn’t naturally come with concept of classes, interfaces, and abstractions.

Of course, I can keep coding my cloud functions in JavaScript, but it requires a certain level of discipline as a developer to keep the code well structured and clean. More importantly, to ensure our code base remains pristine, other team members need to share the same discipline and adhere to the design pattern. Ultimately, someone needs to lay down some rules and be the gatekeeper. Otherwise the code base quickly gets out of control. I’m pretty sure we all have been there more than once throughout our programming careers!

TypeScript is a strictly typed language that serves as a guard rail by providing encapsulation and abstraction to reduce the number of common errors or bugs that get introduced by developers. We all should use unit testing, but it usually falls by the wayside due to our busy schedules. Since TypeScript is strictly typed, tools like editors can help us catch those errors (such as syntax and misspelling) early – either at development time or at build time – to avoid runtime bugs. Editors can also provide autocomplete and intellisense to help us to be more productive and efficient.

Summary

Cloud Object Storage APIs are compatible with the AWS SDK. I found that after I made the configuration and code changes that I described in this section, my Cloud Functions were able to read and write to Cloud Object Storage, and they worked as expected.

Overall, the transition from AWS S3 to IBM Cloud Object Storage was pretty smooth. In the ibm-cos-sdk, I couldn’t find an equivalent S3 deleteDir API to remove a directory and all its children, so I implemented my own.

Another thing to note is that the UI doesn’t provide a directory tree view like we are accustomed to. Instead, everything is in one long list, and there are not features for deleting nested folders and files. Here is what it looks like:

cloud object storage screen capture

If you want to delete everything under staticmaps/test2, you need to select each file to delete it. I developed a custom web app to help me get things done faster and easier, such as deleting folders and files and making them public and private:

cloud object storage screen capture

Now you have seen what we did at Weather Underground, you can try out IBM Cloud Object Storage and IBM Cloud Functions for yourself.