IBM Developer Advocacy

Building Offline-First, Progressive Web Apps



Glynn Bird
11/8/16

I’ve been creating websites for many years and I’ve watched the definition of “best practice” evolve over time. Web technology is a movable feast driven by:

  • Web users who consume the websites being built
  • Web developers who are tasked with building websites using the tools available
  • Browser developers who introduce new features into their products that developers can utilise
  • Standards committees who attempt to gain consensus between all the interested parties so that innovation happens in a way that is mutually beneficial

Inevitably there are casualties along the way: standards or browser innovations that show promise but are little-used, fail to gain cross-platform consensus or are superseded by another round of innovation.

Message in a Bottle video, Sting and Andy Summers

“Best Practice in a Bottle, Yeah”

In this blog post I aim to summarise Progressive Web Apps (PWAs), which seem to me to form a manifesto of best practices for the websites of today (November 2016.) The recommendations herein stem from my experience refactoring one of my apps in the summer of 2016. I hope they help you get started with your own PWA implementation.

It’s important to note that this blog will only have a limited shelf life. In a year or two, the advice set out here will be out-of-date, perhaps laughably so, but that’s the nature of the beast. It’s OK to feel lost in the web development landscape as frameworks rise and fall; “best practice” rolls onwards and the very programming language we use to pin it all together changes radically.

What Are PWAs?

The term “Progressive Web App” refers to a website that aims to provide a user experience akin to a native app. The “Progressive” bit refers to the web app selecting which technologies it engages depending on the capabilities of the platform the website is running on. On an older browser, a PWA may not have any special features, but on the latest Firefox or Chrome builds they may silently enable the modern APIs that those platforms afford.

Simply put, a PWA aims to provide:

  • Responsive design – that displays well on mobile, tablet and desktop form factors
  • Offline rendering – where the web page can be viewed and used with no network connection
  • Offline-First storage – where data is stored locally on the device and synced to the cloud later
  • App-Like install – where a mobile user can save the web app to their desktop
Pokedex.org main page, progressive web app by Nolan Lawson

Pokedex.org by Nolan Lawson runs offline and can install to your phone like a normal app. See this write-up or the source on GitHub.

Some of these aims are not new, but the PWA manifesto brings together this shopping list of best practice and offers APIs that web developers can use today. Many of the aims are technology-neutral and can be solved using a variety of tools.

Responsive Design

There are any number of CSS frameworks that dictate the markup you can use to achieve a fluid, collapsible web interface that looks good on all devices. I have used Bootstrap for years but for this blog post, and in the interest of variety, I chose the Materialize library instead. Incorporating Google’s Material Design principles, Materialize makes it very simple to create a good-looking, responsive website that works well on mobile devices.

Offline Rendering

A standard website won’t function at all if there’s no network connection. Even if the network connection is patchy, such as when browsing on a mobile device, a site may struggle to deliver a satisfactory user-experience. The AppCache API allows websites to be aggressively cached on the device to the point where they can render with no network connectivity (as long as they were visited at least once on a previous occasion!). The AppCache API is an example of a solution that was designed by committee, received widespread browser adoption but was not widely loved by developers. It has been superseded by the Service Worker API.

Service Workers are JavaScript tasks (a bit like server-side daemons but running on the client side) that are instantiated by web pages and from that point, can intercept and route traffic emanating from that page. The Service Worker API is much more flexible than AppCache as it allows the developer to decide in minute detail what happens to each client-side web request — but with flexibility comes complexity.

Offline-First Storage

Offline-First storage allows data to be stored in an in-browser database, giving your web application the opportunity to read and write data to and from its local database, even when offline. There are several solutions to this problem. IndexedDB, DexieJS, and SQLite are supported by a range of browsers, but my favourite in-browser database is PouchDB, which works on a wide variety of browsers and devices and provides the same API to you (the developer) while choosing the best in-browser storage technology at runtime. Making a website work on a range of browsers and platforms is hard enough, but in-browser storage varies greatly from browser to browser, and PouchDB smooths the path immeasurably.

PouchDB also allows the in-browser database to be synced to a remote Apache CouchDB™, IBM Cloudant or PouchDB database when there is network connectivity using the CouchDB replication protocol. The ability of CouchDB-like databases to allow the same data to be replicated, modified in different ways and re-synced without data-loss makes this an ideal solution for offline-first storage.

App-Like Install

Progressive Web Apps are not installed from an app-store like native apps; they are shared using URLs as the Web’s design intends. Once loaded on a phone’s browser, the URL can be added to the phone’s home screen, but implementations of this functionality vary between browsers and platforms. Google Chrome supports a manifest.json file that lists the application’s name, colours, icons and other metadata.

Building a PWA

While I can’t share all the source code, I will include some snippets from my work refactoring my app earlier this summer. Here’s the toolkit I chose to produce the PWA features I was after:

  • Cloudant Envoy – to allow my one-database-per-user model to result in a single database on the server side
  • MaterializeCSS – for responsive CSS and markup. Other frameworks are available, of course.
  • jQuery – I’m not a full-time front-end developer. I understand jQuery, and I haven’t the time to learn one of the formal frameworks like Angular or ReactJS.
  • PouchDB – for in-browser storage and sync
  • LeafletJS – for maps and HTML5 geolocation
  • Mustache – for HTML templating
  • Simple Data Vis – absurdly simple visualisation library based on d3

The range of choices is bewildering. This list doesn’t represent the only way to build a PWA by any means, but it’s the tooling I was comfortable using.

Office space meme: Yeaaah, I'm gonna need you to refactor ALL the JavaScript https://jordankasper.com/js-testing/images/meme-refactor.jpg

Don’t be overwhelmed! Here’s where I started with my PWA.

I found it easiest to start with the front end in my app. I wrote my front end code assuming that the user in my application was authenticated and by hard-coding a few settings. Then I wrote my front end app to read and write its data from its local PouchDB database. I knew that with a few more lines of code I could get it to sync correctly, so that “solved problem” wasn’t one I needed to waste time on. If I could get an app to allow data to be added, edited and deleted on the client side, then the rest should fall into place.

I also ignored the offline caching code until the last minute too. I assumed (correctly) that if I got my app working then I could add the Service Worker to provide a caching service at a later date.

Getting Started with Cloudant Envoy

In your blank directory create a new “package.json” file with:

> npm init

We can then add Cloudant Envoy:

> npm install --save cloudant-envoy

We are going to put our static website (index.html, JavaScript, CSS, images, etc.) in a “public” sub-directory and our Node.js app in “app.js”:

> mkdir public
> touch public/index.html
> mkdir public/js
> mkdir public/css
> touch app.js

Create your app in app.js:

var path = require('path'),
  express = require('express'),
  router = express.Router();

// my custom API call
router.post('/myapicall', function(req, res) {
  res.send({ok: true});
});

// setup Envoy to 
//     - log incoming requests
//     - switch off demo app
//     - serve out our static files
//     - add our routes
var opts = {
  logFormat: 'dev',
  production: true,
  static: path.join(__dirname, './public'),
  router: router
};

// start up the web server
var envoy = require('cloudant-envoy')(opts);
envoy.events.on('listening', function() {
  console.log('[OK]  Server is up');
});

The above code uses Envoy to start the web server and adds in:

  • Our “public” directory to be served out
  • Our custom API calls to be incorporated

This design allows us to build a website that is static web server, handles API calls, and is a CouchDB-compatible replication target all in one go.

In the client-side code, the app then uses PouchDB to create a database:

var db = new PouchDB('mylocaldatabase');

That PouchDB database can then be used to store data:

var mydata = { a:1, b:2, c: 'three'};
db.post(mydata).then(function(d) {
  console.log('Data saved to', d.id);
  });

When you need to sync the data, simply use the PouchDB replicate or sync tools:

var remotedb = new PouchDB('https://username:password@mywebserver.myhost.com/envoy');
  db.sync(remotedb);

The URL you sync to depends on where your app is running. It could be https://username:password@myapp.mybluemix.net/envoy or http://localhost:8000/envoy. The database name (after the last slash) has to match the one that your app is using (envoy is the default db name).

Creating Users with Envoy

By default, Cloudant Envoy looks for users in its envoyusers database. Here’s what a user object looks like:

{
  "_id": "user123",
  "_rev": "1-89de8ebc2b1ad4385ced1f0ed29fa708",
  "type": "user",
  "name": "user123",
  "roles": [],
  "username": "user123",
  "password_scheme": "simple",
  "salt": "1d5d80c9-d925-4f1e-8114-ed44501c38a5",
  "password": "4809dcd4f8dd1cf16f592d90d518875d3c5916f8",
  "seq": null,
  "meta": {
    "user_name": "johnsmith",
    "facebook_id": "johnsmith88",
    "premium": true
  }
}

Envoy can create users for you. In your code, simply call:

var username = 'user123';
var password = 'mysecretpassword';
var meta = {
  "user_name": "johnsmith",
  "facebook_id": "johnsmith88",
  "premium": true
};

envoy.auth.newUser(username, mysecretpassword, meta, function (err, data) {
    // your code goes here
});

Once added, the username-password combination should work for replication too.

Local documents

If you need to store state locally that you don’t want to be replicated to the remote replica, then simply store data to a document whose _id begins with _local/, e.g.:

var localstate = { _id: '_local/mystate', a:1, b:2};
db.put(localstate);

Local documents are only stored on the device and are not included in the list of documents to be copied during replication.

Offline Maps

The Leaflet JavaScript library is easy enough to cache so that it works offline, but the map tiles themselves are pretty tricky: there’s lots of them at lots of resolutions. The solution I developed was to use an empty map and add a GeoJSON layer that contained a rough outline of the world. For my application, I only need to geo-locate users approximately, and I didn’t need every road, river and hill to be rendered on the map.

To render the map, I created a Leaflet map:

var mymap = L.map('mapid').setView([20, 0], 1);

Then, I fetched the 250k GeoJSON file and rendered it on top:

$.ajax({url: '/js/world.json',
  success: function(data) {
    var style = {
        color: "#666",
        fillColor: "#66bb66",
        fillOpacity: 1.0,
        weight: 1,
        opacity: 1
    };
    var l = L.geoJson(data, {style: style}).addTo(mymap);
  }
});

If we cache the Leaflet CSS & JavaScript files together with the world.json file referenced in the snippet, then we have offline-first maps!

Conclusion

Progressive Web Apps give users a vastly improved experience when used with modern browsers:

  • The same app can be used on desktop and mobile browsers
  • Data is stored and retrieved from a local data set, so performance and battery life are excellent
  • Site assets can be cached locally, making the app available despite the network connection status
  • Apps can be distributed through URLs without app store submission and installation with much smaller application size

Compliments? Complaints? Mild salutations? Direct them to @glynn_bird, and don’t forget to have a look at PouchDB, Cloudant Envoy and the other tools here for your next Progressive Web App.

blog comments powered by Disqus