Join us for Code @ Think 2019 | San Francisco | February 12 – 15 Register now Limited availability
By Alex Bozarth | Published September 5, 2018
In the quickly expanding field of AI technologies the idea of fast and easy to use machine learning is closer to becoming a reality. Machine learning engines like TensorFlow, Keras, PyTorch, and Caffe2 are making it easier for data scientists and developers to do scalable machine learning. This has brought about the rise of various open source and proprietary “model zoos” for accessing the ever-growing number of machine learning models.
We believe this growing community deserves an access point for finding models that are open and free. The Model Asset eXchange (MAX) is a one-stop exchange for data scientists and developers to find and use machine learning models. MAX offers both pretrained and trainable models that are created with open source ML engines like TensorFlow, Keras, PyTorch, and Caffe2. MAX models include detailed metadata on how they were trained and on what data.
Continuing from my previous work, I wanted to create another interesting application based on a MAX model. While researching and testing different models available on MAX, there was one model that frustrated me. The Object Detector model seemed like it might be interesting but the JSON output was difficult to visualize in my head. My frustration prompted me to start creating a small visual wrapper for the model, which eventually grew into a fully functioning web application.
With my basic web server, web UI, and model API access working it was time for me to get back to the issue at hand: how to visualize the output of the Object Detector model. When given an image, the model API returns JSON output containing each detected object. Each object has a label, a probability, and a bounding box, with the bounding box represented by normalized coordinates. Normalized coordinates are numbers between 0 and 1 representing a percent of the image’s width or height. My wrapper’s goal was to make understanding these coordinates easier by visually drawing them over the image in my UI. To accomplish this, I used an HTML canvas to display the bounding boxes on top of the image in the UI. With the ability to see the objects detected in an image in a UI, my wrapper web app was complete.
After sharing my wrapper web app with my team and other MAX users, I received a lot of positive feedback and interest. This surprising interest convinced me to continue working on my wrapper and flesh it out into a mini web app that I could share externally. Based on the feedback I received, I began adding new features while addressing various usability issues in my initial work. I made my web app code public and added the label and probability of each object to the canvas. I then worked with the model developers to add control of the probability threshold to the model API. Using this new feature, I added a threshold control tool to the UI. The threshold slider allows users to control what bounding boxes are displayed on the imaged based on the prediction probability. With a cleaner user experience and new features, I released the Object Detector Mini Web App. I decided to refer to it as a mini web app because it was entirely transient and to add no new functionality to the model beyond visualizing the output. The MAX developer team and I quickly realized that the mini web app would work well directly embedded into the model, rather than deployed separately. Because the web app consisted entirely of static files and making calls to the model, it was simple to include in the model API server at a new /app endpoint. With the mini web app included in the model, my Node.js server implementation was relegated to be a development environment.
The new visibility from embedding the Mini Web App in the model brought about new user feedback and insights. These insights inspired me to create another new feature, interactive label highlighting and filtering. I added a new toolbar to the UI that displayed label icons for any labels currently displayed. I used the label icons from the COCO website because the Object Detector model uses the COCO data set labels for its prediction. When mousing over an icon, all the boxes with the matching label are highlighted in the canvas. Then when an icon is clicked boxes with the label are hidden, allowing filtering of displayed objects based on the label. With this new feature, the Object Detector Web App was released and the model was updated to embed the latest version. What began as a simple visual wrapper evolved over time into a fully functioning web application that not only visually wrapped a model but added interactive functionality.
Try it out
The web app is available on GitHub, and we would love it if you took it for a test run. As you saw, we are also open to improving it, so if you have suggestions feel free to open an issue on the repo. You can follow along with this code pattern to learn more about it.
Back to top