Deploy stand-alone Kubeflow Pipelines on a local Kubernetes cluster using kind

In the era of rapid AI advancement, the ability to expedite the machine learning (ML) development cycle and deliver products or solutions on schedule is crucial for achieving true success. However, the complexity of ML development necessitates robust solutions that simplify workflows, foster collaboration, and ensure reproducibility. Adopting an ML platform that facilitates the composition and execution of ML workflows can significantly enhance ML adoption.

Kubeflow Pipelines is an end-to-end platform designed for building and deploying portable, scalable ML workflows using Docker containers. Kubeflow Pipelines, which is an open source solution built on Kubernetes, empowers ML practitioners to streamline and automate their development processes with ease. By programming ML operations and tasks into an ML pipeline, Kubeflow Pipelines enables automated ML development while minimizing the risk of human errors.

IBM uses Kubeflow Pipelines as the orchestration engine to build the enterprise-grade ML platform IBM Watson Pipelines which is part of IBM Watson Studio. Learn more about using Watson Pipelines to orchestrate end-to-end ML workflows in this other tutorial.

In this tutorial, you learn how to set up a stand-alone Kubeflow Pipelines on a local Kubernetes cluster using kind. Then, you can use Kubeflow Pipelines as a local development environment to compose and run ML pipelines.

Prerequisites

For optimal performance of stand-alone Kubeflow Pipelines on a local machine, it is recommended to have a minimum of 8 CPUs and 16 GB of memory.

You need the following tools to set up the development environment and deploy Kubeflow Pipelines:

• kind: Use it to set up a local Kubernetes cluster. Follow these installation instructions.
• Docker: kind uses a Docker container to run a local Kubernetes cluster. Follow these Docker installation instructions. Alternatively, you can use Podman, and follow these Podman installation instructions.
• kubectl: kubectl is a CLI tool that you use to interact with a Kubernetes cluster. Follow these installation instructions. Because this tutorial uses Kubernetes v1.26.4, you should install the same version for kubectl.

Steps

At a high level, these are the steps you must complete:

1. Spin up a local Kubernetes cluster using kind.
2. Deploy stand-alone Kubeflow Pipelines using Tekton.
3. Run the sample pipeline.
4. Clean up the local development environment.

Spin up a local Kubernetes cluster using kind

First, let's create a local Kubernetes cluster that allows us to deploy Kubeflow Pipelines on top of it.

1. Make sure you can access the Docker engine by issuing a simple container query command:

    docker ps
   

   This command should return the running containers on your system or an empty list without errors.

2. Create a kind cluster named kfp and specify the specific image to run Kubernetes (v1.26.4) by using this command:

    kind create cluster --name kfp --image "kindest/node:v1.26.4"
   

   You should see the following outputs which indicate a local Kubernetes is created successfully:

    Creating cluster "kfp" ...
      Ensuring node image (kindest/node:v1.26.4)
      Preparing nodes
      Writing configuration
      Starting control-plane
      Installing CNI
      Installing StorageClass
   
    Set kubectl context to "kind-kfp"
    You can now use your cluster with:
   
    kubectl cluster-info --context kind-kfp
   
    Have a nice day!
   

3. Verify the newly created Kubernetes cluster with the kubectl version command:

    kubectl version --short
   

   The output should be something similar to this output:

    Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
    Client Version: v1.26.4
    Kustomize Version: v4.5.7
    Server Version: v1.26.4
   

   The output indicates the version of kubectl (Client Version: v1.26.4) matches the version of the local Kubernetes cluster (Server Version: v1.26.4). At this point, you have created a local Kubernetes cluster.

Deploy stand-alone Kubeflow Pipelines with Tekton

Once your local Kubernetes is up and running, we can start to deploy the standalone Kubeflow Pipelines with Tekton.

1. Download the deployment files from the GitHub repository.

   You can download the whole repository as a tarball with this link, extract the files from it, and get into the kubeflow-kfp-tekton-64bf7db directory. Or, you can run the following command if you are using a Linux-like operation system that provides wget and tar commands:

    wget -O - https://github.com/kubeflow/kfp-tekton/tarball/64bf7dba3601a8e8f2f16c4fef169d796ab1428d | tar zxf - && cd kubeflow-kfp-tekton-64bf7db
   

   The deployment files are under manifest/kustomize directory after you get into the kubeflow-kfp-tekton-64bf7db directory.

2. Install Kubeflow Pipelines with Tekton:

    kubectl apply -k "manifests/kustomize/cluster-scoped-resources"
    kubectl wait "crd/applications.app.k8s.io" --for "condition=established" --timeout "60s"
    kubectl apply -k "manifests/kustomize/env/platform-agnostic-kind"
    kubectl wait pods -l "application-crd-id=kubeflow-pipelines" -n kubeflow --for "condition=Ready" --timeout "1800s"
   

   The last command is to wait for the pods that run Kubeflow Pipelines components to be in the ready state (and it would take several minutes). Once all Kubeflow Pipelines components are available, you will receive messages similar to the following:

   pod/cache-deployer-deployment-55dc4678fb-8t9lm condition met
   pod/cache-server-5d8f47cb68-kj4nt condition met
   pod/metadata-envoy-deployment-55df78bdc7-sh6xj condition met
   pod/metadata-grpc-deployment-6d744c66bb-shz6q condition met
   pod/metadata-writer-857c6c974-jscpd condition met
   pod/minio-549846c488-7flqz condition met
   pod/ml-pipeline-6884f97df7-lkjrm condition met
   pod/ml-pipeline-persistenceagent-d48ddccb7-8d6l7 condition met
   pod/ml-pipeline-scheduledworkflow-5745f8947d-794k6 condition met
   pod/ml-pipeline-ui-5f9945f75f-6s2dx condition met
   pod/ml-pipeline-viewer-crd-5996f65649-dxcwk condition met
   pod/ml-pipeline-visualizationserver-59c764c5fb-4kdbg condition met
   pod/mysql-5f968d4688-xtchc condition met
   

3. Use the port-forward command to make the dashboard of Kubeflow Pipelines available locally on port 8080:

    kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
   

   To view the Kubeflow Pipelines dashboard, simply enter this URL in your browser: http://localhost:8080:

   

Run the Sample Pipeline

On the Kubeflow Pipelines dashboard, you'll find important features listed on the left-hand side. These include Pipelines, Experiments, Runs, Recurring Runs, Artifacts, and Executions. In this tutorial, our focus will be on Pipelines. Specifically, we'll examine the flip coin sample pipeline that is available on the Pipelines page.

1. Click the [Demo] flip-coin pipeline. You will see the pipeline details:

   

2. Click the Simplify Graph toggle to make the graph easier to read:

   

   A pipeline is a definition of a workflow that composes one or more components together to form a computational directed acyclic graph (DAG). At runtime, each component execution corresponds to a single container execution, which can create ML artifacts. If you click on a single vertex node, you see its details. In the flip-coin pipeline, it starts with the flip-coin node which generates either heads or tails as its output. If the output is "heads," the "condition-1" path is executed. If the output is "tails," the "condition-4" path is chosen.

3. Create a run for the flip-coin pipeline by clicking the Create run button on the top:

   

   Use default values for all the input fields, and then click the Start button at the bottom of the page to start a run:

   

4. You will automatically switch to the Runs page. Click the run you just created:

   

5. Check the run progress and see the results of the node marked with a green checkmark:

   

   Same as on the Pipelines page, you can toggle the Simplify Graph option to have a simpler view:

   

   Your results may be different from this screen capture. If your flip-coin node generates tails as its output, the green checkmarks will appear on the nodes of the condition-4 path.

Congratulations! You successfully ran a pipeline in your local development environment.

Clean Up

When you are done with the local development environment, you can tear down the local Kubernetes cluster by using this command:

kind delete cluster --name kfp

Conclusion

Kubeflow Pipelines provides an efficient solution for ML development, addressing the challenges faced by data scientists and ML engineers. By ensuring reproducibility, scalability, collaboration, and flexibility, Kubeflow Pipelines empowers practitioners to accelerate their workflows, enhance collaboration, and drive innovation.

By following the steps provided in this tutorial, you can easily create a local development environment and run simple and lightweight pipelines. However, if you require a robust platform to run your ML workflows, you should check out watsonx.ai. Watsonx.ai brings together new generative AI capabilities, powered by foundation models, and traditional machine learning into a powerful platform spanning the AI lifecycle. With watsonx.ai, you can train, validate, tune, and deploy models with ease and build AI applications in a fraction of the time with a fraction of the data.

Try watsonx.ai -- next-generation studio for AI builders. Explore more articles and tutorials about watsonx on IBM Developer.