Kubeflow in IBM Cloud

This tutorial is part of the Get started with Kubeflow in IBM Cloud learning path.

Level Topic Type
200 Kubeflow in IBM Cloud Tutorial
201 Train and serve a machine learning model using Kubeflow in IBM Cloud Tutorial

Introduction

Kubeflow is known as a machine learning toolkit for Kubernetes. It is an open source project dedicated to making deployments of machine learning workflows on Kubernetes simple, portable, and scalable. In this tutorial, you learn:

  1. Provisioning of a Kubernetes cluster in IBM Cloud and installing the required tools
  2. Installing Kubeflow
  3. Launching the Kubeflow central dashboard

Prerequistes

To run this tutorial, you need:

  • An Ubuntu 18 machine with a minimum 8 cores, 16 GB RAM, and 250 GB storage
  • Root privilege on the ubuntu machine
  • An IBM Cloud account.

Estimated time

It should take you approximately 30 minutes to complete the tutorial.

Provisioning of a Kubernetes cluster in IBM Cloud and installing required tools

Create a Kubernetes cluster

  1. Using a web browser, visit https://cloud.ibm.com/kubernetes/overview.

  2. After logging in to IBM Cloud, click Create a cluster from the overview page.

  3. In the Kubernetes Cluster page, you can select a plan, an environment, a location, cluster metadata, and default worker pools. I recommend that you use a meaningful cluster name instead of using the default cluster name (mycluster). After reviewing the cost estimate in the right pane, click Create in the lower right corner. Currently, there is a free cluster plan (30-day cluster with 1 worker node and limited capabilities) that is perfect for testing and learning purposes. The following figure shows cluster setup and access details that you’ll see after you create your cluster.

Creating a cluster page

Install required tools

Before your cluster provisions, install the IBM Cloud CLI tool. In your Ubuntu command line, enter:

curl -sL https://ibm.biz/idt-installer | bash

You can find more details in the IBM Cloud documentation.

Access your cluster

After the cluster provisions, you can access your cluster from your Ubuntu machine using the access details when you create your cluster (see the previous image). Remember to use your own access information.

ibmcloud login -a cloud.ibm.com -r us-south -g default -sso

You might be given a url to get your one-time passcode for authentication. After authentication, if you have multiple accounts, you are asked which account you want to use.

ibmcloud ks cluster config --cluster bptoesud0ijlu9ktvcn0

Note: In case you forget your cluster setting, you can retrieve this information by visiting https://cloud.ibm.com/kubernetes/clusters and clicking Access under your cluster.

Access information

To verify your setting, run the following code.

kubectl version --short

The following code shows sample output.

Client Version: v1.16.8
Server Version: v1.16.8+IKS

Set up IBM Cloud block storage

  1. Install the Helm version 3 client on your local machine. Full instruction can be found here.

     wget https://get.helm.sh/helm-v3.0.0-linux-amd64.tar.gz
     tar -zxvf helm-v3.0.0-linux-amd64.tar.gz
     mv linux-amd64/helm /usr/local/bin/helm
    
  2. Add the IBM Cloud Helm chart repository.

     helm repo add iks-charts https://icr.io/helm/iks-charts
     helm repo update
     helm install 1.6.0 iks-charts/ibmcloud-block-storage-plugin -n kube-system
    

    The following code shows sample output.

     NAME: 1.6.0
     LAST DEPLOYED: Fri Mar 20 11:38:21 2020
     NAMESPACE: kube-system
     STATUS: deployed
     REVISION: 1
     NOTES:
     Thank you for installing: ibmcloud-block-storage-plugin.   Your release is named: 1.6.0
    
  3. Set the block storage as default.

     kubectl patch storageclass ibmc-block-gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
     kubectl patch storageclass ibmc-file-bronze -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
    

Installing Kubeflow v.1.0.1

I have created a directory called kubeflow under /root, although you can use whatever directory you like. The following instructions are for installing Kubeflow 1.0.1 in the kubeflow directory.

mkdir -p /root/kubeflow/v1.0.1
cd /root/kubeflow/v1.0.1
wget https://github.com/kubeflow/kfctl/releases/download/v1.0.1/kfctl_v1.0.1-0-gf3edb9b_linux.tar.gz
tar -xvf kfctl_v1.0.1-0-gf3edb9b_linux.tar.gz
export PATH=$PATH:/root/kubeflow/v1.0.1
export KF_NAME=my-kubeflow
export BASE_DIR=/root/kubeflow/v1.0.1
export KF_DIR=${BASE_DIR}/${KF_NAME}
# config file is IBM specific
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/master/kfdef/kfctl_ibm.yaml"

mkdir -p ${KF_DIR}
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_URI}

When the installation is complete, run the following command to see whether all of the pods are in a running status.

kubectl get pod -n kubeflow

The following figure shows sample output.

sample output

Note that you must wait until all of the pods are running before moving to next step. If the installation fails, you can delete kubeflow.

cd ${KFAPP}
kfctl delete all --delete_storage

Launching the Kubeflow central dashboard

Kubeflow Dashboard can be accessed through the istio-ingressgateway service. To see your setting of istio-ingressgateway service, execute the following code.

kubectl get -n istio-system svc/istio-ingressgateway

The following code shows sample output.

NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                                                                                                      AGE
istio-ingressgateway   LoadBalancer   172.21.178.223   169.62.98.13   15020:31759/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:30421/TCP,15030:31917/TCP,15031:30450/TCP,15032:32528/TCP,15443:30312/TCP   4d19h

If your output has a NodePort TYPE (not LoadBalancer) and EXTERNAL-IP is blank, run the following command to change the type from NodePort to LoadBalancer, then save.

kubectl edit -n istio-system svc/istio-ingressgateway

Run the following command again to get the EXTERNAL-IP.

kubectl get -n istio-system svc/istio-ingressgateway

Now you can access the Kubeflow dashboard in a web browser: http://<EXTERNAL-IP>.

Kubeflow dashboard

Summary

In this tutorial, I explained how to install Kubeflow in IBM Cloud, and how to launch the Kubeflow dashboard.

This tutorial is part of the Get started with Kubeflow in IBM Cloud learning path. To continue with the learning path, look at the next tutorial in the series, Train and Serve a machine learning model using Kubeflow in IBM Cloud.