In the quickly expanding field of AI technologies the idea of fast and easy to use machine learning is closer to becoming a reality. Machine learning engines like TensorFlow, Keras, PyTorch, and Caffe2 are making it easier for data scientists and developers to do scalable machine learning. This has brought about the rise of various open source and proprietary “model zoos” for accessing the ever-growing number of machine learning models.

We believe this growing community deserves an access point for finding models that are open and free. The Model Asset eXchange (MAX) is a one-stop exchange for data scientists and developers to find and use machine learning models. MAX offers both pretrained and trainable models that are created with open source ML engines like TensorFlow, Keras, PyTorch, and Caffe2. MAX models include detailed metadata on how they were trained and on what data.

Creating a web app wrapper around a model

Continuing from my previous work, I wanted to create another interesting application based on a MAX model. While researching and testing different models available on MAX, there was one model that frustrated me. The Object Detector model seemed like it might be interesting but the JSON output was difficult to visualize in my head. My frustration prompted me to start creating a small visual wrapper for the model, which eventually grew into a fully functioning web application.

When starting on my wrapper web app, I made the decision to expand my knowledge base while creating it. Given the recent popularity of Node.js I decided to use it for my web app server. To create my UI, I pulled the bare-bones HTML, Javascript, and CSS code from my previous web app as both a starting point and to keep a consistent look and feel. I then used Express, a Node.js web app framework, to host my static UI content on my server. When deciding how to upload images and send them to the model for prediction, I was influenced by my web app research. During my research on interesting application ideas, I was looking at potential security use-cases. This caused me to make my wrapper web app transient, doing all the work client-side. To handle image upload I used the HTML5 File API, which let me display the image in my UI without sending it to the server. This brought me to my first roadblock when creating my wrapper: how to send an image to the model API for prediction. I originally wanted to send it directly to the API from the web UI, but because the model API isn’t in the same domain there’s no simple and secure way to do so. To send calls to the model API, I had to go through my Node.js server, but this ended up being simpler than expected because Node has the functionality to pipe requests straight through to the model API without touching them.

With my basic web server, web UI, and model API access working it was time for me to get back to the issue at hand: how to visualize the output of the Object Detector model. When given an image, the model API returns JSON output containing each detected object. Each object has a label, a probability, and a bounding box, with the bounding box represented by normalized coordinates. Normalized coordinates are numbers between 0 and 1 representing a percent of the image’s width or height. My wrapper’s goal was to make understanding these coordinates easier by visually drawing them over the image in my UI. To accomplish this, I used an HTML canvas to display the bounding boxes on top of the image in the UI. With the ability to see the objects detected in an image in a UI, my wrapper web app was complete.

object-detector 1

After sharing my wrapper web app with my team and other MAX users, I received a lot of positive feedback and interest. This surprising interest convinced me to continue working on my wrapper and flesh it out into a mini web app that I could share externally. Based on the feedback I received, I began adding new features while addressing various usability issues in my initial work. I made my web app code public and added the label and probability of each object to the canvas. I then worked with the model developers to add control of the probability threshold to the model API. Using this new feature, I added a threshold control tool to the UI. The threshold slider allows users to control what bounding boxes are displayed on the imaged based on the prediction probability. With a cleaner user experience and new features, I released the Object Detector Mini Web App. I decided to refer to it as a mini web app because it was entirely transient and to add no new functionality to the model beyond visualizing the output. The MAX developer team and I quickly realized that the mini web app would work well directly embedded into the model, rather than deployed separately. Because the web app consisted entirely of static files and making calls to the model, it was simple to include in the model API server at a new /app endpoint. With the mini web app included in the model, my Node.js server implementation was relegated to be a development environment.

Object detector 2

The new visibility from embedding the Mini Web App in the model brought about new user feedback and insights. These insights inspired me to create another new feature, interactive label highlighting and filtering. I added a new toolbar to the UI that displayed label icons for any labels currently displayed. I used the label icons from the COCO website because the Object Detector model uses the COCO data set labels for its prediction. When mousing over an icon, all the boxes with the matching label are highlighted in the canvas. Then when an icon is clicked boxes with the label are hidden, allowing filtering of displayed objects based on the label. With this new feature, the Object Detector Web App was released and the model was updated to embed the latest version. What began as a simple visual wrapper evolved over time into a fully functioning web application that not only visually wrapped a model but added interactive functionality.

Object detector 3

Try it out

The web app is available on GitHub, and we would love it if you took it for a test run. As you saw, we are also open to improving it, so if you have suggestions feel free to open an issue on the repo. You can follow along with this code pattern to learn more about it.

Learn more