Maintain the most suitable model with the Model Management API

It’s common today to find many pretrained machine learning models from model zoo, such as TensorFlow Hub, Caffe model zoo, modelzoo.co, and ModelDepot.io. These models differ in the training data set, model hyperparameters, or the task being solved. For a given distributed environment, it is not a straightforward process to choose the model that might perform the best, with or without customization to the local environment. Additionally, there might not be sufficient training data available on the target distributed site to train a model from scratch in a collaborative environment. Each party might have a different environment.

In essence, how well a model discriminates the target data set is a good indicator of the model’s accuracy, regardless of the label set, data set, or hyperparameters.

Using our earlier example of Meera, she is now tasked with managing thousands, if not hundreds of thousands, of devices with varying types and multiple locations around the globe, with each geography governed by a different set of rules for data requirements. To accelerate her innovation, she opted to use the Model Management API to satisfy some of her challenges in terms of model drifting over time and context changes. She wants to automatically tune the model, retrain, reselect, and redeploy as needed.

In this tutorial, get a step-by-step guide on using the Distributed AI Model Management API to solve challenges associated with selecting the right model at distributed and edge environments.

Following is a complete Python notebook to help you get started using the API. This tutorial also covers a few basic steps such as getting access to a trial subscription of Distributed AI APIs on the IBM API Hub platform.

The tutorial uses a Python notebook, which you can run in your preferred IDE.

Prerequisites

To complete this tutorial, you need:

  • An IBM ID
  • Python 3.8
  • Python notebook IDE

Estimated time

It should take you approximately 30 minutes to complete this tutorial.

Steps

Step 1. Environment setup

To set up your environment:

  1. Navigate to the Distributed AI APIs documentation page, and click Get trial subscription.

    Trial subscription

  2. Log in on the registration page if you already have an IBM ID. Otherwise, create a new IBM ID for yourself.

  3. After you log in, the system entitles you with a trial subscription and takes you to the My IBM page. Locate the Trial for Distributed AI APIs tile, and click Launch.

  4. On the My APIs page, click the Distributed AI APIs tile. When the page opens, locate the Key management section, expand the row to see both the Client ID and Client secret, and click the visibility (eye) icon to reveal the actual values. Make a note of these values because they are the API keys you use throughout this tutorial.

    Key management

  5. Create a config.json file with the API key values that you received.

     {
      "x-ibm-client-id":  "REPLACE_THIS_WITH_YOUR_CLIENT_ID",
      "x-ibm-client-secret": "REPLACE_WITH_YOUR_CLIENT_SECRET"
     }
    
  6. Install a requests Python package using pip.

    pip install keras
    

Step 2. Invoke Distributed AI Model Managements API

On the API documentation page, look for the Model Management API.

Model Management API

Step 3. Python notebook example

The example Python notebook creates a model fingerprint and identifies model outliers. Step through the Python notebook code in your preferred Python notebook IDE.

Model Management API

Model fingerprint is a technique for defining a fingerprint of a model, which can be used to determine if any given input belongs to a data set of reasonable input to the model or if it is an outlier to the data set. Fingerprint is calculated with a trained model.

Two methods for layered or non-layered models are:

  1. Layered. The first component takes the activations of the training data for each of the layers and trains a unique autoencoder for each layer. The second component consists of the distribution of reconstruction errors that are calculated by running the activations of the training data through the autoencoder and calculating the root mean squared difference between the input activations and the outputted values.

  2. Non-layered. The first component consists of an autoencoder that is calculated directly from the training data. The second component consists of the distribution of reconstruction errors that are calculated by running the training data through the autoencoder and calculating the root mean square difference between the input training data and the outputted values.

Installation and REST service startup

  1. To install from Artifactory, run pip install edgeai_model_mgmt.
  2. After successful installation, run the python -m modelmgmt.apis.rest_services command.
  3. A REST server should start and report the IP Address and port number used to accept requests.

Import libraries for calling APIs and data serialization

import requests
import pickle
import json

Fingerprint service requirements

The Distributed AI Model Management Fingerprinting service expects a pretrained model and the original training data set to generate a fingerprint. However, if a layered model is not available, you can provide only the training data set to generate a fingerprint. We train a Keras MNIST model to use for this example.

Prepare MNIST data set

from keras import utils
from keras.datasets import mnist, fashion_mnist

img_rows, img_cols = 28, 28
(mnist_x_train, mnist_y_train), _ = mnist.load_data()

x = mnist_x_train.astype('float32').reshape(mnist_x_train.shape[0], img_rows, img_cols, 1) / 255.
y = utils.to_categorical(mnist_y_train, 10)

Build the model

from keras import models
from keras import layers
from keras.models import Sequential

model = models.Sequential()
model.add(layers.Conv2D(32, (5, 5), activation='relu', input_shape=(28, 28, 1), padding='same'))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Conv2D(64, (5, 5), activation='relu'))
model.add(layers.MaxPool2D(2, 2))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dropout(0.4))
model.add(layers.Dense(10))
model.add(layers.Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
-    Train Model
model.fit(x, y, batch_size=128, epochs=3)
-    Save newly trained model and dataset to desired path
import tempfile
import os

mfilename = 'model.h5'
dsfilename = 'mnist.pickle'


model.save(mfilename)
with open(dsfilename, 'wb') as file:
    pickle.dump(x, file)

Invoke REST service endpoint

Retrieve the REST endpoint (IP and Port) as reported when the REST server was started and invoke the generate_fingerprint service. The generate_fingeprint service accepts multipart/formdata requests with the following arguments:

  • dataset(required): Pickled file of a numpy array data set that is used during model training. Type=file
  • model: Path to the saved model that is used to fingerprint. Type=File
  • model_type: Type of model to fingerprint: ‘keras’ or ‘pytorch’. Type=string
  • model_def: Required if model_type is pytorch and should be the path to the model definition file. Type=file
  • partial_activations: Percentage of partial activations to retain when fingerprinting. Type=float
url = 'URL/generate_fingerprint' # your url may differ. change as neccessary

data = {'model_type': 'keras', 'partial_activations': 0.2}

multiple_files = [
    ('dataset', ('mnist.pickle', open(dsfilename, 'rb'), 'file/pickle')),
    ('model', ('mnist.h5', open(mfilename, 'rb'), 'file/h5'))]

r = requests.post(url,  data=data, files=multiple_files)

Check response code and save compressed file returned from server (generated fingerprint)

The saved fingerprint could be used for outlier detection in determining if a given input is an outlier to the trained model.

r

saved_fingerprint = 'keras_fingerprint.zip'
with open(saved_fingerprint, 'wb') as fd:
        for chunk in r.iter_content(chunk_size=128):
            fd.write(chunk)

Utilize generated fingerprint for outlier detection

After a fingerprint is successfully generated, this fingerprint can be used to determine if any given input belongs to a data set of reasonable input to the model or if it is an outlier to the data set. We can use the outlier_detection service to make this determination on a provided data set for a generated fingerprint.

Import fashion MNIST model to use as input for outlier detection

from keras.datasets import mnist, fashion_mnist

(fm_x, fm_y), _ = fashion_mnist.load_data()
fm_x = fm_x.astype('float32').reshape(fm_x.shape[0], 28, 28, 1) / 255.
fm_x = fm_x[:500]

with open('fmnist.pickle', 'wb') as file:
    pickle.dump(fm_x, file)

Invoke REST service endpoint

Retrieve the REST endpoint (IP and Port) as reported when the REST server was started, and invoke the outlier_detection service. The outlier_detection service accepts multipart and formdata requests with the following arguments:

  • fingerprint (required): Compressed directory containing the fingerprint. Type=file
  • num_layers (required): Number of layers in the model fingerprint. Type=int
  • dataset (required): Pickled file of a numpy array data set to determine if outlier. Type=file
  • model: Path to the saved model that is used to fingerprint. Type=file
  • model_type: Type of model to fingerprint: ‘keras’ or ‘pytorch’. Type=string
  • model_def: Required if model_type is pytorch, and should be the path to the model definition file. Type=file
  • activation_mappings: Acknowledges whether activation mapping was used for fingerprint. Type=bool
num_layers = 8
activation_mapping = True

url = 'URL/outlier_detection'

payload = {'num_layers': num_layers, 'model_type': 'keras', 'activation_mapping': activation_mapping}

multiple_files = [
    ('dataset', ('fmnist.pickle', open('fmnist.pickle', 'rb'), 'file/pickle')),
    ('model', ('mnist.h5', open('mnist.h5', 'rb'), 'file/h5')),
    ('fingerprint', (saved_fingerprint, open(saved_fingerprint, 'rb'), 'file/zip'))]


r = requests.post(url,  data=payload, files=multiple_files)
-    Retrive outlier score returned from server; should be < 10%
score = r.json()['score']
print(score)

Summary

This tutorial explained how to secure API keys and easily invoke the Distributed AI APIs hosted in IBM Cloud. The APIs in the suite help with invoking different algorithms to work with your application needs. If you have any questions or queries after the trial subscription, email us at resai@us.ibm.com.