IBM Developer Blog

Follow the latest happenings with IBM Developer and stay in the know.

Use IBM Cloud Schematics to set up a multi-user Kubeflow with a few clicks


Kubeflow is an end-to-end machine learning platform on Kubernetes that provides components and rich features to compose machine learning pipelines. The new release, v1.3, came out recently, and you can deploy the new release on the IBM Kubernetes Service. The deployment involves Kubernetes cluster creation, local environment setup, deployment, and configuration, and if you are not familiar with these operations, they can look intimidating. One mistake in these procedures can lead to a painful debugging, reconfiguration, and even scrap and redo. Don’t worry, we’ve got you. Resources to lower the barrier for you: IBM Cloud Schematics to deploy multi-user Kubeflow v1.3, which integrates with App ID as a login mechanism, and an Auto-Kubeflow repository.

The Git repository provides a convenient approach for deploying Kubeflow v1.3 on IBM Cloud. It leverages IBM Cloud Schematics to do resource provisioning, including the Kubernetes cluster service and App ID. It finishes the deployment and configuration with an Ansible playbook. After clicking the Apply button on the Schematics service, you just sit back, relax, and wait for the deployment to complete automatically. When it finishes, a fully functional multi-user Kubeflow environment will be read for you. Currently, you can find two configurations for the Kubeflow deployment inside the repo:

  • Classic: Run your cluster with native subnet and VLAN networking on a classic infrastructure
  • VPC Gen 2: Create a fully customizable, software-defined virtual network with superior isolation using IBM Cloud VPC

Choose the one that fits your needs because the end results are the same. However, for VPC Gen 2, you gain more control of the IBM Kubernetes Service cluster in networking, public gateways, access control lists, and security groups. After deployment, you can modify these configurations.

IBM Cloud Schematics

IBM Cloud Schematics enables Infrastructure as Code (IaC) by using a high-level scripting language to model and manage resources on IBM Cloud. Furthermore, combining with Ansible helps you provision use cases, configuration, and deploy applications. In this case, Ansible is used to deploy multi-user Kubeflow, set up configuration, and integrate with App ID. As for App ID, it secures the Kubeflow with authentication without changing the application. The authentication supports public social profiles and email signup. You can further customize the login, if needed. From the IBM Cloud Schematics workplace, you can find the links to all of the resources created for the Kubeflow deployment, including the IBM Kubernetes Service cluster, App ID, and Gateway for VPC Gen 2. You can find the resources and modify configurations for each resource. By using the Destroy action in IBM Schematics, all related resources would be freed up with a single click.

For more information, you can check out a tutorial that guides you through the deployment process. Ideally, you would be able to have a Kubeflow cluster up and running with just a few clicks. Both the classic and VPC Gen 2 approaches give you the same Kubeflow configuration, but with different underlying IBM Kubernetes Service infrastructures. For the Kubeflow perspective, the functionality is the same. After you finish the deployment and access the public endpoint, you are redirected to the App ID authentication page to sign in with your social profiles or sign up with your email. When you complete the authentication, you see the Kubeflow dashboard and can try out the machine learning pipeline experiments. The provisioned Kubeflow is using Kubeflow Pipeline with Tekton. You can check out useful pipeline samples. From there, you can learn how to compose your pipeline by using Kubeflow domain-specific language in Python. Then, upload and run your machine learning pipelines on the Kubeflow dashboard.

If you are interested in DevOps, IBM DevOps Toolchain lets you integrate IBM Cloud Schematics into your IBM Continuous Delivery pipeline. From the IBM Cloud Schematics workspace, you can find the Enable continuous delivery option in Settings. You can create a DevOps Toolchain that uses IBM Cloud Schematics to provision Kubeflow as one of the operations in the IBM Continuous Delivery pipeline. Detailed Toolchain information is also available.

I hope you can benefit by using IBM Cloud Schematics templates to provision multi-user Kubeflow deployment. If you have any questions, feel free to open an issue there. We, as IBM developer advocates, are happy to provide comments, suggestions, help troubleshoot any problems, or even add features you might need.