IBM Cloud Satellite: Run and manage services anywhere Learn more

Set up a Kubeflow development environment

This tutorial is part of the Get started with Kubeflow learning path.

Introduction

After you have installed Kubeflow, you want to set up a development environment to compile and test a Kubeflow Pipeline application. In this tutorial, you learn:

  • Set up a Kubeflow development for compilation
  • Test a Kubeflow Pipeline application using Kubeflow Dashboard

Prerequisites

To run this tutorial, you need an Ubuntu 18 machine with a minimum 8 cores, 16 GB RAM, and 250 GB storage. You also need root privileges to run the tutorial steps. Kubeflow is installed in a cloud or on-premises environment. If you have not already installed it, then install it using one of the following tutorials:

Estimated time

It should take you approximately 15 minutes to complete the tutorial.

Compilation of a sample Pipeline application

This tutorial covers the steps required to compile a sample Pipeline application. The Full instructions to set up the development environment can be found at the official Kubeflow site.

  1. Set up the Python environment. Python 3.5 or later is required for setup.

     apt-get update; apt-get install -y wget bzip2
     wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
     bash Miniconda3-latest-Linux-x86_64.sh
     Create a Python 3.7 environment name of mlpipeline(or any name you preferred):
     conda create --name mlpipeline python=3.7
     conda init
    
  2. Reopen your current shell.

     conda activate mlpipeline
    
  3. Install the Kubeflow SDK.

     pip install https://storage.googleapis.com/ml-pipeline/release/latest/kfp.tar.gz --upgrade
    
  4. Clone or download the Kubeflow samples and compile.

     mkdir /root/kubeflow/src
     cd /root/kubeflow/src
    
     git clone https://github.com/kubeflow/pipelines.git
     dsl-compile --py /root/kubeflow/src/pipelines/samples/core/sequential/sequential.py --output /tmp/mysequential.tar.gz
    

    The compilation output is stored in /tmp/mysequential.tar.gz.

Upload the compiled Pipeline to Kubeflow Dashboard

To upload the compiled Pipeline to Kubeflow Dashboard:

  1. In the main page, under “Quick shortcuts”, click Upload a pipeline.

    Uploading a pipeline option

  2. In the Pipelines GUI, click Upload pipeline.

    Upload pipeline option

  3. In the Upload and name your pipeline window, select Upload a file, then choose the file you have compiled. Click Upload.

    Upload and name your pipeline

Execution of compiled Pipeline in Kubeflow Dashboard

  1. In the Pipelines window, click mysequential (the one we just uploaded).

    mysequential pipeline

  2. In the mysequential pipeline, you can view the graph and source of this pipeline. As you can see in the following image, there are two rectangles that represent two containers. Click Create experiment.

    Mysequential pipeline window

  3. In the Experiment details window, complete the Experiment name, and then click Next.

    Experiment details

  4. In the Run details window, complete the Run name, select One-off under Run Type, and click Start.

    Run details window

This brings you back to the Experiments window where you can check the run status and other information.

Experiments window

Under Runs, click mySeqRun1 for more details.

Details on myseqrun1

Under Graph, you can click individual containers to view their input and output as well as other details.

Summary

In this tutorial, I explained how to set up a development environment to compile a Pipeline application and test it using Kubeflow dashboard.

This tutorial is part of the Get started with Kubeflow learning path. To continue with the learning path, look at the next tutorial in the series, Train and serve a machine learning model using Kubeflow in Minikube.