Set up a Kubeflow development environment
Set up a Kubeflow development environment for compilation, then test a Kubeflow Pipeline application using Kubeflow Dashboard
This tutorial is part of the Get started with Kubeflow learning path.
|200||Kubeflow in IBM Cloud Private-CE||Tutorial|
|201||Kubeflow in Minikube||Tutorial|
|300||Set up the development environment||Tutorial|
|301||Train and serve a machine learning model using Kubeflow in Minikube||Tutorial|
|400||Leverage Kubeflow for enterprise data in Db2||Tutorial|
|401||Train and serve a machine learning model using Kubeflow in IBM Cloud Private-CE||Tutorial|
After you have installed Kubeflow, you want to set up a development environment to compile and test a Kubeflow Pipeline application. In this tutorial, you learn:
- Set up a Kubeflow development for compilation
- Test a Kubeflow Pipeline application using Kubeflow Dashboard
To run this tutorial, you need an Ubuntu 18 machine with a minimum 8 cores, 16 GB RAM, and 250 GB storage. You also need root privileges to run the tutorial steps. Kubeflow is installed in a cloud or on-premises environment. If you have not already installed it, then install it using one of the following tutorials:
It should take you approximately 15 minutes to complete the tutorial.
Compilation of a sample Pipeline application
This tutorial covers the steps required to compile a sample Pipeline application. The Full instructions to set up the development environment can be found at the official Kubeflow site.
Set up the Python environment. Python 3.5 or later is required for setup.
apt-get update; apt-get install -y wget bzip2 wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh Create a Python 3.7 environment name of mlpipeline(or any name you preferred): conda create --name mlpipeline python=3.7 conda init
Reopen your current shell.
conda activate mlpipeline
Install the Kubeflow SDK.
pip install https://storage.googleapis.com/ml-pipeline/release/latest/kfp.tar.gz --upgrade
Clone or download the Kubeflow samples and compile.
mkdir /root/kubeflow/src cd /root/kubeflow/src git clone https://github.com/kubeflow/pipelines.git dsl-compile --py /root/kubeflow/src/pipelines/samples/core/sequential/sequential.py --output /tmp/mysequential.tar.gz
The compilation output is stored in /tmp/mysequential.tar.gz.
Upload the compiled Pipeline to Kubeflow Dashboard
To upload the compiled Pipeline to Kubeflow Dashboard:
In the main page, under “Quick shortcuts”, click Upload a pipeline.
In the Pipelines GUI, click Upload pipeline.
In the Upload and name your pipeline window, select Upload a file, then choose the file you have compiled. Click Upload.
Execution of compiled Pipeline in Kubeflow Dashboard
In the Pipelines window, click mysequential (the one we just uploaded).
In the mysequential pipeline, you can view the graph and source of this pipeline. As you can see in the following image, there are two rectangles that represent two containers. Click Create experiment.
In the Experiment details window, complete the Experiment name, and then click Next.
In the Run details window, complete the Run name, select One-off under Run Type, and click Start.
This brings you back to the Experiments window where you can check the run status and other information.
Under Runs, click mySeqRun1 for more details.
Under Graph, you can click individual containers to view their input and output as well as other details.
In this tutorial, I explained how to set up a development environment to compile a Pipeline application and test it using Kubeflow dashboard.
This tutorial is part of the Get started with Kubeflow learning path. To continue with the learning path, look at the next tutorial in the series, Train and serve a machine learning model using Kubeflow in Minikube.