There is so much time that a full-stack developer unfortunately spends dealing with just the serialization and deserialization of his or her data model. This is why I am really excited for any tools that helps break up that monotony, get me beyond that, and allow me to focus on working on the parts of my application that makes it really unique. Fortunately, we are beyond the days of parsing XML’s from DTD and now mostly using simpler object notation, namely JSON. However, there are some notable problems with JSON:

  1. It’s text. So it’s not as compact and efficient as it could be. This sometimes is not an issue, but when a response contains a large payload, this becomes much more important.
  2. It’s not typed. Makes no guarantees about the existence of fields, and also the data type of the fields inside of them. You can’t guarantee that the JSON that the client sends to the server matches the JSON that the server knows how to parse.

Protocol Buffers

What we want is to be able to specify a unified way of representing your application’s data model so that it can be shared regardless of language or platform it’s used on. This is what Protocol buffers are designed to achieve. It was designed at Google, and it is a language-neutral, platform-neutral language for serializing structured data. You write your model in the Protobuf language, run the code generation tool called protoc, and an automatically generated struct (or class) that matches that structured data type will be produced for you. Protobufs have been out for a while, but only for C++, C#, Go, Python, Java languages.

Fortunately, Apple has recently released an open source plugin for protoc that enables you to build Swift structs from the Protobuf language. Once these structs are generated, it allows these structs to be serialized and deserialized both as binary Protobuf objects or as JSON. Currently, the Swift Package Manager has limited integrated support for Protocol Buffers, but in the future, we expect it to be more closely integrated. For now, you must run some of these tools manually.

Let’s direct our focus on how to integrate Protocol Buffers with a Kitura REST web service.

Install protoc and the swift-protobuf plugin

The getting started section on the swift-protobuf repository will step you through the details for installing Google’s protoc tool and downloading the swift-protobuf plugin and setting up your system’s path to find that plugin.

Create a Kitura project

Creating and structuring a Kitura application is outside of the scope of this article, but there are some good starters online. For instance, there is the IBM Cloud Kitura Starter that has everything you need to get started. You can also download a finished version of this work from my Kitura-Protobufs repository.

The basic Kitura application contains routes for getting tasks and running the web server. It is important to add the BodyParser to the routes so that the HTTP body can be read from the requests.

import Kitura
import Web

let router = Router()

var myLibrary = MyLibrary()

router.all("/", middleware: BodyParser())

// Handle HTTP GET requests to /
router.get("/v1/book") { }

router.post("/v1/book") { }

Kitura.addHTTPServer(onPort: 8090, with: router)
Kitura.run()

If you are using XCode to build you Kitura application, it needs to know how to find the Protobuf.framework file. First download the Swift Protobuf runtime from the repository, open the SwiftProtoRuntime.xcodeproj and build the project in XCode. Locate the Protobuf.framework file probably in your Derived Data directory, and drop it in your Kitura App’s XCode project.

protoc-framework

Create a Proto file

The following model example came directly from the swift-protobuf guide. It has a BookInfo and a collection of books, called MyLibrary. Save it as BookLibrary.proto.


syntax = "proto3";

message BookInfo {
   int64 id = 1;
   string title = 2;
   string author = 3;
}

message MyLibrary {
   int64 id = 1;
   string name = 2;
   repeated BookInfo books = 3;
   map<string,string> keys = 4;
}

Build Swift structs

After you run the code generator, it will produce a Swift struct that has some nice properties. For instance, it will conform to Equatable, so that it can be compared against other structs based on the properties inside of the struct, and also Hashable so that it can be inserted in to a Dictionary and looked up quickly. Additionally, it has some mutating properties that make using the structure efficient. In particular, Copy-on-Write (CoW) semantics. Struct’s are value types in the Swift language, and properties inside of them are by default immutable. Therefore, if you make a copy of a struct, the Swift language often doesn’t have to actually make a copy of the information on the stack, and instead just use a reference instead. This makes using structs in many times very efficient. However, if you end up mutating values, a new structure has to be deeply copied. There are some interesting optimizations that the protoc generator can do to add CoW semantics on these structs so that they only have to be deeply copied if written to.

The following command will create BookLibrary.pb.swift in the present working directory. You can now import this Swift file into your Swift project by placing the Swift in Sources next to your other code in your application.

protoc --swift_out=. BookLibrary.proto

Get a book with Protocol Buffers

In this example, we look at the Accept header to either return JSON or a binary protobuf back to the client based on its setting and the MIME type for each. There currently does not exist a special MIME type for Protocol Buffers except for octet-binary. There have been proposals made, but nothing officially accepted.

router.get("/v1/book") {
    request, response, next in
    
    let b = BookInfo(id: 303, title: "The Odyssey", author: "Homer")
    
    guard let acceptType = request.headers["Accept"] else {
        let jsonBook = try b.serializeJSON()
        response.send(jsonBook)
        next()
        return
    }
    
    switch acceptType {
        case "application/json":
            let jsonBook = try b.serializeJSON()
            response.send(jsonBook)
        case "application/octet-stream":
            let data = try b.serializeProtobuf()
            response.send(data: data)
        default:
            let jsonBook = try b.serializeJSON()
            response.send(jsonBook)
    }

    next()
}

Add a book

With this route, we read the Content-type. If the Content-Type is application/json, then try to create a BookInfo from JSON with the rawString. If the body contains binary information, the BookInfo is created with the Foundation struct, Data directly.

router.post("/v1/book") {
    request, response, next in
    
    guard let contentType = request.headers["Content-Type"] else {
        response.status(.badRequest)
        next()
        return
    }
    
    guard let body = request.body else {
        next()
        return
    }
    
    let book: BookInfo
    
    switch body {
        case .raw(let raw):       book = try BookInfo.init(protobuf: raw)
        case .json(let json):     book = try BookInfo.init(json: json.rawString()!)
        default: return
    }
    
    myLibrary.books.append(book)
    response.status(.OK).send("Added book \(book.id)")
    next()
    
}

Test your Protobufs

Run your server and test it out. Use a HTTP request tool like Postman or curl to try your GET and POST requests.

Getting back a binary protobuf for the book

Get a Protocol Buffer back for a book

Getting back JSON for the book

Get JSON back for a book using Protocol Buffers

Add a book by sending JSON

Add a Book by using JSON and Protocol Buffers

If you want the finished example, you can clone the repository: https://github.com/IBM-Swift/Kitura-Protobuf-Sample

Next Steps

Now that you have a Kitura web service that can consume Protobufs or JSON using a data model that has been automatically generated for you, you can now proceed to develop many interesting end-to-end applications like the following:

  • Use your generated BookLibrary.pb.swift file in an iOS project, and make calls to the Kitura server.
  • Build a BookLibrary.java file and import it into your Android project, and make calls to Kitura server.

You are now set to create cross-platform applications that leverages Kitura and Swift. For a basic tutorial about how to write a ToDo application in Kitura, check out my other tutorial Building End-to-End Cloud Applications Using Swift Kitura or other tutorials at our Kitura.io website.

Stay tuned for the next article in the series where we discuss how to use Protocol buffers in your iOS application.

5 comments on"Protocol Buffers in your Kitura Apps"

  1. Milan Stevanovic October 09, 2016

    Hi Robert,

    What would be the best way to serialize an array of protobuf objects(in this case myLibrary.books) and send it as a response, using Apple’s protobuf library?

    • I think you have to create another model, like this example’s “Library” that can contain Books. But maybe there is something easier?

      • Milan Stevanovic October 09, 2016

        Well I hoped you could tell me, I’m doing it like that now, sending MyLibrary model from the example, but it just doesn’t feel right. Or maybe I’m too acustomed to JSON. 🙂
        Also, what content type do you set in header when sending protobuff serialized into Data? I’m sending from an iOS app, I’m using Apple’s lib for protobuf, the same one like on the server side, but I can’t get the server to receive the body of the request. Communication in other direction is working fine, I’m getting protobuf from the server, and using it in my app. I’m doing requests via URLRequest and setting httpBody with ‘try? book.serializeProtobuf()’ and headers with ‘setValue(“Content-Type”, forHTTPHeaderField:”application/octet-stream”)’, but it just doesn’t work, the request always has empty body when it comes to server.

        • Milan,

          The issue is that BodyParser doesn’t handle application/octet-stream and instead of making the request’s body raw, it drops it entirely. I made a pull request to the example application here: https://github.com/rfdickerson/Kitura-Protobufs/pull/2

          You can read the request for more information. Hope that helps!

          • Milan Stevanovic December 04, 2016

            Thanks Jacob,

            Your pull request was very informative, and it’s always nice to see that the community is actively contributing, keep up the good work, server side Swift needs love! 🙂

Join The Discussion

Your email address will not be published. Required fields are marked *