This post is co-authored by Animesh Singh and Clive Cox.

As I outlined in my previous post, training deep neural network models requires a highly tuned system with the right combination of software, drivers, compute, memory, network, and storage resources. To address these challenges, we launched Fabric for Deep Learning (FfDL) as an open source project. FfDL offers a stack that removes these concerns so that data scientists can execute training jobs with their choice of deep learning frameworks, such as Caffe, Torch, and TensorFlow at scale in the cloud.

Enterprise AI lifecycle

But another part of the enterprise AI lifecycle is deploying and serving these models. Watson Studio, which uses FfDL core for training part of the enterprise AI lifecycle, gives you an option to deploy and serve your models, in addition to services for gathering and analyzing data. Watson Studio also gives you tools to design your neural network models using Neural Network Modeler (NNM), which is a visual drag-and-drop tool for creating layers of your neural architecture and assigning hyperparameters.

Complement model training with model serving

On the open source side, to complete the FfDL trained model deployment story we’ve worked together with the Seldon team to provide a FfDL integration with the Seldon machine learning deployment platform. Beyond deployment, Seldon enables machine learning models to be exposed through REST or gRPC endpoints. Runtime graphs of models, routers (for example, AB-tests and multi-armed bandits), transformers (for example, feature normalization) and combiners (for example, ensemblers) can be described using a custom Kubernetes resource JSON/YAML file and then deployed, scaled, and managed.

This means that you can deploy both FfDL and Seldon on the same Kubernetes infrastructure using Helm charts and then push the models that have been trained using FfDL to the Seldon machine learning deployment platform. To deploy your FfDL trained models on Seldon you simply need to:

  1. Wrap your model runtime inference components as Docker containers
  2. Describe the runtime graph as a custom Kubernetes SeldonDeployment resource
  3. Apply the graph using the Kubernetes API

You can find the details of how to deploy a FfDL trained model using Seldon in the Deploy FfDL Trained Models with Seldon readme file.

Get started today!

We’re working to extend the examples to add PyTorch and Caffe models, in addition to the examples we’ve already added with respect to TensorFlow. We’ve also created an end-to-end demonstration pipeline using the Fashion MNIST dataset.

Get started with FfDL and Seldon today. Deploy, use, and extend them with capabilities that you find helpful. We’re waiting for your feedback and pull requests!

Join The Discussion

Your email address will not be published. Required fields are marked *