Kubernetes with OpenShift World Tour: Get hands-on experience and build applications fast! Find a workshop!

Build models using Jupyter Notebooks in IBM Watson Studio

This tutorial is part of the Getting started with Watson Studio learning path.

Introduction

This tutorial explains how to set up and run Jupyter Notebooks from within IBM® Watson™ Studio. We start with a data set for customer churn that is available on Kaggle. The data set has a corresponding Customer Churn Analysis Jupyter Notebook (originally developed by Sandip Datta), which shows the archetypical steps in developing a machine learning model by going through the following essential steps:

  1. Import the data set.

  2. Analyze the data by creating visualizations and inspecting basic statistic parameters (for example, mean or standard variation).

  3. Prepare the data for machine model building (for example, by transforming categorical features into numeric features and by normalizing the data).

  4. Split the data into training and test data to be used for model training and model validation.

  5. Train the model by using various machine learning algorithms for binary classification.

  6. Evaluate the various models for accuracy and precision using a confusion matrix.

  7. Select the model that’s the best fit for the given data set, and analyze which features have low and significant impact on the outcome of the prediction.

  8. Use Watson Machine Learning to save and deploy the model so that it can be accessed outside of the notebook.

The notebook is defined in terms of 35 Python cells and requires familiarity with the main libraries used: Python scikit-learn for machine learning, Python numpy for scientific computing, Python pandas for managing and analyzing data structures, and matplotlib and seaborn for visualization of the data.

Prerequisites

To complete the tutorials in this learning path, you need an IBM Cloud account. You can obtain a free trial account, which gives you access to IBM Cloud, IBM Watson Studio, and the IBM Watson Machine Learning Service.

Estimated time

It should take you approximately 30 minutes to complete this tutorial.

Steps

The steps to set up your environment for the learning path are explained in the Data visualization, preparation, and transformation using IBM Watson Studio tutorial. These steps show how to:

  1. Create an IBM Cloud Object Storage service.
  2. Create an IBM Watson Studio project.
  3. Provision IBM Cloud services.
  4. Upload the data set.

You must complete these steps before continuing with the learning path. If you have finished setting up your environment, continue with the next step, creating the notebook.

NOTE: The Watson Machine Learning service is required to run the notebook.

Create the notebook

Create a Jupyter Notebook for predicting customer churn and change it to use the data set that you have uploaded to the project.

  1. In the Asset tab, click Add to Project.

    add-to-project

  2. Select the Notebook asset type.

  3. On the New Notebook page, configure the notebook as follows:

    1. Select the From URL tab:

      create-notebook

    2. Enter the name for the notebook (for example, ‘customer-churn-kaggle’).

    3. Select the Python 3.6 runtime system

    4. Enter the following URL for the notebook:

      https://github.com/IBM/watson-studio-learning-path-assets/blob/master/notebooks/customer-churn-kaggle.ipynb
      
    5. Click Create Notebook. This initiates the loading and running of the notebook within IBM Watson Studio.

Run the notebook

The notebook page should be displayed.

If the notebook is not currently open, you can start it by clicking the Edit icon displayed next to the notebook in the Asset page for the project:

start-notebook

NOTE: If you run into any issues completing the steps to execute the notebook, a completed notebook with output is available for reference at the following URL: https://github.com/IBM/watson-studio-learning-path-assets/blob/master/examples/example-customer-churn.ipynb.

From the notebook page, make the following changes:

  1. Scroll down to the third cell, and select the empty line in the middle of the cell. If not already open, click the 1001 data icon at the upper part of the page to open the Files subpanel.

    insert-pandas-dataframe

  2. In the right part of the page, select the Customer Churn data set. Click insert to code, and select Insert pandas DataFrame. This adds code to the data cell for reading the data set into a pandas DataFrame.

    notebook-with-dataframe

  3. Change the generated variable name df_data_1 for the data frame to df, which is used in the rest of the notebook. When displayed in the notebook, the data frame appears as the following:

    data-set

  4. Select File > Save to save the notebook.

Run the cells of the notebook one by one, and observe the effect and how the notebook is defined.

Background on running notebooks

When a notebook is run, each code cell in the notebook is executed, in order, from top to bottom.

Each code cell is selectable and is preceded by a tag in the left margin. The tag format is In [x]:. Depending on the state of the notebook, the x can be:

  • A blank, which indicates that the cell has never been run
  • A number, which represents the relative order that this code step was run
  • An *, which indicates that the cell is running

There are several ways to run the code cells in your notebook:

  • One cell at a time. Select the cell, and then press Play in the toolbar.
  • Batch mode, in sequential order. From the Cell menu, there are several options available. For example, you can Run All cells in your notebook, or you can Run All Below, which starts running from the first cell under the currently selected cell, and then continues running all of the cells that follow.
  • At a scheduled time. Press the Schedule button that is located in the upper-right section of your notebook page. Here, you can schedule your notebook to be run once at some future time or repeatedly at your specified interval.

Data understanding and visualization

During the data understanding phase, the initial set of data is collected. The phase then proceeds with activities that enable you to become familiar with the data, identify data quality problems, and discover first insights into the data. In the Jupyter Notebook, these activities are done using pandas and the embodied matplotlib functions of pandas. The describe function of pandas is used to generate descriptive statistics for the features, and the plot function is used to generate diagrams showing the distribution of the data.

data-understanding

Data preparation

The data preparation phase covers all activities that are needed to construct the final data set that will be fed into the machine learning service. Data preparation tasks are likely to be performed multiple times and not in any prescribed order. Tasks include table, record, and attribute selection as well as transformation and cleansing of data for the modeling tools. In the Jupyter Notebook, this involves turning categorical features into numerical ones, normalizing the features, and removing columns that are not relevant for prediction (such as the phone number of the client). The following image shows a subset of the operations.

data-preparation

Modeling and evaluation

In the modeling phase, various modeling techniques are selected and applied and their parameters are calibrated to achieve an optimal prediction. Typically, there are several techniques that can be applied, and some techniques have specific requirements on the form of the data. Therefore, going back to the data preparation phase is often necessary. However, in the model evaluation phase, the goal is to build a model that has high quality from a data analysis perspective. Before proceeding to final deployment of the model, it’s important to thoroughly evaluate it and review the steps that are executed to create it to be certain that the model properly achieves the business objectives.

In the Jupyter Notebook, this involved splitting the data set into training and testing data sets (using stratified cross-validation) and then training several models using distinct classification algorithms such as GradientBoostingClassifier, support vector machines, random forest, and K-Nearest Neighbors.

model-training

Following this step, we continue with printing the confusion matrix for each algorithm to get a more in-depth view of the accuracy and precision offered by the models.

model-evaluation

Deploying your model to Watson Machine Learning

In the last section of the notebook, we save and deploy the model to the Watson Machine Learning service. To access the service, we need to cut and paste the Machine learning credentials into this notebook cell:

wml-creds

After the model is saved and deployed to Watson Machine Learning, we can access it in a number of ways.

In the Jupypter Notebook, we can pass data to the model scoring endpoint to test it.

score-model

If we go back to the Watson Studio console, we can see in the Assets tab that the new model is listed in the Models section.

assets-model

If we click on the Deployments tab, we can see that the model has been successfully deployed.

deploy-list

Click on the deployment to get more details. If you click the Implementation tab, you will see the scoring endpoint. In the Code Snippets section, you can see examples of how to access the scoring endpoint programmatically.

deploy-implementation

On the Test tab, we can pass in a scoring payload JSON object to score the model (similar to what we did in the notebook). After supplying the data, press Predict to score the model.

deploy-test

SPSS model notebook

Now that you have learned how to create and run a Jupyter Notebook in Watson Studio, you can revisit the Scoring machine learning models using the API section in the SPSS Modeler Flow tutorial. It has instructions for running a notebook that accesses and scores your SPSS model that you deployed in Watson Studio.

Conclusion

This tutorial covered the basics for running a Jupyter Notebook in Watson Studio, which includes:

  • Creating a project
  • Provisioning and assigning services to the project
  • Adding assets such as data sets to the project
  • Importing Jupyter Notebooks into the project
  • Loading and running the notebook

The purpose of the notebook is to build a machine learning model to predict customer churn using a Jupyter Notebook. Other tutorials in this learning path discuss alternative, non-programatic ways to accomplish the same objective, using tools and features built into Watson Studio.

Rich Hagarty
Einar Karlsen