Serving AI Models from Kubernetes Persistent Volumes with KServe ModelMesh

ModelMesh Serving is a mature, general-purpose model serving framework designed to serve high volumes of inference requests for thousands of frequently changing models. ModelMesh intelligently loads and unloads models to and from memory to strike a balance between responsiveness and efficient resource utilization.

IBM has used ModelMesh in production for several years before it was contributed to open source as part of KServe. ModelMesh is an integral part of multiple Watson services, such as Watson NLU and watsonx Assistant, and the upcoming enterprise-ready AI and data platform watsonx.

You can learn more about ModelMesh's features, how to install ModelMesh Serving, and how to deploy and make inference requests to your first model, by following our Getting Started with KServe ModelMesh tutorial.

The model files to be served by ModelMesh typically reside on cloud object storage like Amazon S3, Google Cloud Storage (GCS), or Azure Blob Storage. When a new predictor, or inference service, is deployed on ModelMesh, the model files have to be downloaded by a storage adapter which can delay the availability of the predictor.

The latest release of ModelMesh introduces the capability to store model files on Kubernetes persistent volumes. The persistent volumes can then be mounted directly onto the serving runtime pods. Depending on the selected storage solution, this approach can significantly reduce latency when deploying new predictors and potentially remove the need for additional cloud object storage altogether.

In this tutorial, we will take a closer look at how to configure ModelMesh Serving to leverage Kubernetes' "built-in" storage by using persistent volumes.

Prerequisites

• A Kubernetes cluster with admin privileges (or Minikube) with 4 CPU and 8 GB memory
• kubectl
• kustomize (v4.0.0+)
• A "Quickstart" installation of ModelMesh Serving

Steps

• Step 1: Create a persistent volume claim (PVC)
• Step 2: Create a pod to access the PVC
• Step 3: Store the model on the persistent volume
• Step 4: Configure ModelMesh Serving to use the persistent volume claim
• Step 5: Deploy a new inference service
• Step 6: Run an inference request

Step 1: Create a persistent volume claim (PVC)

Once you have ModelMesh Serving installed and running, you need to create a Kubernetes persistent volume Cclaim (PVC). The easiest way to achieve that is by relying on Kubernetes dynamic volume provisioning to provision a persistent volume with 1 GB of storage and the ReadWriteMany access mode. Since Persistent Volumes are namespace-scoped, the persistent volume claim has to be created in the same namespace as the ModelMesh Serving deployment.

If you installed ModelMesh Serving at an earlier time, make sure to set the current context for this tutorial:

kubectl config set-context --current --namespace=modelmesh-serving

Let's create the Persistent Volume Claim my-models-pvc:

kubectl apply -f - <<EOF
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "my-models-pvc"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
EOF

After a short while, the PVC should be "bound" to a persistent volume:

kubectl get pvc

NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
my-models-pvc   Bound    pvc-5836269d-6be6-49f6-8aba-fe812c29842b   20Gi       RWX            retain-file-gold   1m

Step 2: Create a pod to access the PVC

In order to make use of our new PVC, we need to mount it as a volume to a Kubernetes pod. We can then use this pod to upload our model files to the persistent volume.

Let's deploy a pvc-access pod and ask the Kubernetes controller to claim the persistent volume we requested earlier by specifying the claimName "my-models-pvc":

kubectl apply -f - <<EOF
---
apiVersion: v1
kind: Pod
metadata:
  name: "pvc-access"
spec:
  containers:
    - name: main
      image: ubuntu
      command: ["/bin/sh", "-ec", "sleep 10000"]
      volumeMounts:
        - name: "my-pvc"
          mountPath: "/mnt/models"
  volumes:
    - name: "my-pvc"
      persistentVolumeClaim:
        claimName: "my-models-pvc"
EOF

The persistent volume will be mounted to the specified mountPath "/mnt/models".

After a short while, our pvc-access pod should be running:

kubectl get pods | grep pvc\|STATUS

NAME                 READY   STATUS    RESTARTS   AGE
pvc-access           1/1     Running   0          1m24s

Step 3: Store the model on the persistent volume

Now we need to add our AI model to the persistent volume. For this tutorial, we will use the MNIST handwritten digit character recognition model trained with scikit-learn. A copy of the model file mnist-svm.joblib can be downloaded from the kserve/modelmesh-minio-examples repo.

curl -sOL https://github.com/kserve/modelmesh-minio-examples/raw/main/sklearn/mnist-svm.joblib

Copy the mnist-svm.joblib model file onto the pvc-access pod into the folder /mnt/models to which we bound our my-models-pvc persistent volume claim.

kubectl cp mnist-svm.joblib pvc-access:/mnt/models/

Then, verify the model exists on the persistent volume.

kubectl exec -it pvc-access -- ls -alr /mnt/models/

# total 352
# -rw-r--r-- 1    501 staff      344817 Mar 16 08:55 mnist-svm.joblib
# drwxr-xr-x 3 nobody 4294967294   4096 Mar 16 08:55 ..
# drwxr-xr-x 2 nobody 4294967294   4096 Mar 16 08:55 .

Step 4: Configure ModelMesh Serving to use the persistent volume claim

There are two ways to configure ModelMesh Serving to be able to access persistent volumes:

• Persistent volume claims can be added in the storage-config secret so that the PVCs will be mounted to all serving runtime pods
• ModelMesh Serving can be configured to dynamically bind PVCs to a given serving runtime pod when a new Predictor or Inference Service that references it is being deployed.

We will use the dynamic binding method by setting the allowAnyPVC configuration flag to true.

Let's create the model-serving-config ConfigMap with the setting allowAnyPVC: true:

kubectl apply -f - <<EOF
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: model-serving-config
data:
  config.yaml: |
    allowAnyPVC: true
EOF

If you already have a model-serving-config ConfigMap, you might want to retain the existing config overrides. You can check your current configuration flags by running:

kubectl get cm "model-serving-config" -o jsonpath="{.data['config\.yaml']}"`

After applying the new configuration, the modelmesh-serving deployment will get updated and the serving runtime pods should get restarted.

Step 5: Deploy a new inference service

Next we need to deploy a new inference service sklearn-mnist with storage type pvc and the name of our persistent volume claim my-models-pvc:

kubectl apply -f - <<EOF
---
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-mnist
  annotations:
    serving.kserve.io/deploymentMode: ModelMesh
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storage:
        parameters:
          type: pvc
          name: my-models-pvc
        path: mnist-svm.joblib
EOF

After a few seconds, the new inference service sklearn-mnist should be ready:

kubectl get isvc

NAME            URL                                               READY   PREV   LATEST   AGE
sklearn-mnist   grpc://modelmesh-serving.modelmesh-serving:8033   True                    23s

Step 6: Run an inference request

Before we can run inference requests to our sklearn-mnist digit recognition model, we need to set up a port-forward:

kubectl port-forward --address 0.0.0.0 service/modelmesh-serving 8008 &

[1] running kubectl port-forward in the background
Forwarding from 0.0.0.0:8008 -> 8008

Now we can use curl to perform inference requests to our sklearn-mnist model. The data array represents the grayscale values of the 64 pixels in the image scan of the digit to be classified.

MODEL_NAME="sklearn-mnist"

curl -X POST -k "http://localhost:8008/v2/models/${MODEL_NAME}/infer" -d '{"inputs": [{ "name": "predict", "shape": [1, 64], "datatype": "FP32", "data": [0.0, 0.0, 1.0, 11.0, 14.0, 15.0, 3.0, 0.0, 0.0, 1.0, 13.0, 16.0, 12.0, 16.0, 8.0, 0.0, 0.0, 8.0, 16.0, 4.0, 6.0, 16.0, 5.0, 0.0, 0.0, 5.0, 15.0, 11.0, 13.0, 14.0, 0.0, 0.0, 0.0, 0.0, 2.0, 12.0, 16.0, 13.0, 0.0, 0.0, 0.0, 0.0, 0.0, 13.0, 16.0, 16.0, 6.0, 0.0, 0.0, 0.0, 0.0, 16.0, 16.0, 16.0, 7.0, 0.0, 0.0, 0.0, 0.0, 11.0, 13.0, 12.0, 1.0, 0.0]}]}'

The JSON response should look like the following, inferring that the scanned digit was an "8":

{
  "model_name": "sklearn-mnist__isvc-3d2daa3370",
  "outputs": [
    {"name": "predict", "datatype": "INT64", "shape": [1], "data": [8]}
  ]
}

Summary and next steps

Now that you know more about the different storage options for ModelMesh, take a look at our tutorial on creating custom serving runtimes in KServe ModelMesh to learn how ModelMesh can easily be extended to support any AI model format or how to enhance the functionality of an existing model serving runtime.

If you want an enterprise-grade platform for your AI workloads built on top of open source software like ModelMesh, try watsonx. Explore more articles and tutorials about watsonx on IBM Developer.