Getting started with IBM Watson OpenScale (automated setup)

This tutorial is part of the Getting started with Watson OpenScale learning path.

Introduction

In this tutorial, you’ll see how IBM® Watson™ OpenScale can be used to monitor your artificial intelligence (AI) models for fairness and accuracy. You’ll get a hands-on look at how Watson OpenScale will automatically generate a debiased model endpoint to mitigate your fairness issues and provides an explainability view to help you understand how your model makes its predictions. In addition, you’ll see how Watson OpenScale uses drift detection. Drift detection will tell you when runtime data is inconsistent with your training data or if there is an increase the data that is likely to lead to lower accuracy.

This tutorial works with IBM Cloud Pak for Data or on IBM Cloud . The automated setup is used to get you started quickly with an example model.

The fairness use case

The model used in this tutorial is a credit risk predictor. The data set contains loan applicant data and is used to predict “Risk” or “No Risk”. The data includes two attributes that are considered sensitive: sex (gender) and age. Using Watson OpenScale with this model, you will be able to detect, explain and fix gender discrimination in the credit risk predictor.

Automated setup

The automated setup guides you through the process by performing tasks for you in the background. The automated setup tour is designed to work with the least possible user interaction. It automatically makes the following decisions for you:

  • If you have multiple IBM Watson Machine Learning instances set up, the installation process runs an API call to list the instances and chooses the Watson Machine Learning instance that appears first in the resulting list.
  • To create a new lite version of a Watson Machine Learning instance, the Watson OpenScale installer uses the default resource group for your IBM Cloud account.

The automated setup ends with a guided tour, which highlights key features of Watson OpenScale as you move through the scenario by clicking Next. When you exit the tour (you can exit at any point), you can explore the UI on your own. The credit risk model was automatically deployed so that you have something to explore. This tutorial uses the credit risk model to help you explore the features of Watson OpenScale.

Estimated time

It should take you approximately 45 minutes to complete this tutorial.

Steps

In this tutorial, you learn how to:

  1. Provision a Watson OpenScale service
  2. Use the Insights Dashboard
  3. Using the Analytics tools
  4. Conclusion

Provision a Watson OpenScale service

Note: Watson OpenScale now allows for provisioning multiple instances of the service. The steps below assume we will configure the default service instance (which is automatically provisioned after installation).

On IBM Cloud Pak for Data platform In the Cloud Pak for Data instance, go the (☰) menu and under Services section, click on the Instances menu option. Service Find the OpenScale-default instance from the instances table and click the three vertical dots to open the action menu, then click on the Open option. Openscale Tile Since this is the first time we are launching OpenScale, you will be presented with a welcome message, where we can launch the auto setup process. Click on the Auto setup button. Openscale Auto Setup Launch In the ‘Connect to Watson Machine Learning’ panel, leave the defaults since we are using the WML instance deployed in the same cluster. Click on the Next button. Openscale Auto Setup WML In the ‘Connect to your database’ panel, enter the connection details for your local DB2 database (this is the database you provisioned in a previous section of the admin guide). Click on the Next button. Openscale Auto Setup DB >Note: If you used a DB2 Warehouse on Cloud, you will need to select the ‘Use SSL’ checkbox but dont need to provide a certificate. The auto setup of a model will take some time to run. Openscale Auto Setup Running Once it completes, you will see a message if it succeded. Openscale Auto Setup Completed Click through the insights dashboard for the deployed models to make sure the pages load. If you need to give other users access to the OpenScale instance, go the (☰) menu and under Services section, click on the Instances menu option. Service Find the OpenScale-default instance from the instances table and click the three vertical dots to open the action menu, then click on the Manage access option. Openscale Tile To add users to the service instance, click the Add users button. Openscale Tile For all of the user accounts, select the Editor role for each user and then click the Add button. Openscale Tile
On IBM Cloud

  1. If you do not have an IBM Cloud account, register for an account here.
  2. Create a Watson OpenScale instance from the catalog.
  3. Select the Lite (Free) plan, enter a Service name, and click Create.
  4. Click Launch Application to start Watson OpenScale.
  5. Click Auto setup to automatically set up your Watson OpenScale instance with sample data.
    Demo welcome
  6. Click Start tour to tour the Watson OpenScale dashboard.

Use the Insights Dashboard

  • To launch the OpenScale service, go the (☰) navigation menu and click Services -> Instances.

(☰) Services -> Instances”/></p>
<ul>
<li>Click the 3 horizontal dots next to the OpenScale instance that your Administrator has provisioned and click <code style=Open.

OpenScale instance

Now lets interact with the tools.

  • OpenScale will load the Insights Dashboard. This will contain tiles for any models being monitored. The tile for GermanCreditRiskModelICP will be the one we will use for this tutorial, which was configured using the Auto setup script.

  • Click on the left-hand menu icon for Insights, make sure that you are on the Model monitors tab, and then open the tile for the GermanCreditRiskModelICP model (click the 3-dot menu on the tile and then View Details):

OpenScale Insight Dashboard Tile Open

  • Notice the red alert indicators on the various monitors (Fairness, Quality, Drift). You should see a red indicator under Fairness. Click on the Fairness score.

Model Overview

  • Click on the triangle with ! under Fairness -> Sex. This indicates that there has been an alert for this attribute in the Fairness monitor. Alerts are configurable, based on thresholds for fairness outcomes which can be set and altered as desired.

  • By moving your mouse pointer over the trend chart, you can see the values change, and which contains bias. Find and click on a spot in the graph that is below the red threshold line to view details.

OpenScale Fairness Monitor

  • Once you click on one of the time periods, you will see details of the Fairness monitor, including a bar chart that shows how many females received the “No Risk” outcome vs. males. You can click view calculation to see how the fairness score is calculated. Click on View payload transactions.

OpenScale Fairness Monitor chart

  • You will see a list of Transactions. Look for one of the Monitored Group – Female with a “Group Bias” check mark and Prediction of “Risk”. Click Explain prediction. If the time period on the graph for Fairness Monitoring doesn’t contain such an element, go back and choose another time period until you can find one. This will make the explanation more interesting.

OpenScale Fairness Detail

Note: Each of the individual transactions can be examined to see them in detail. Doing so will cache that transaction, as we will see later. Be aware of the fact that the Explainability feature requires 1000’s of REST calls to the endpoint using variations of the data that are slightly perturbed, which can require several seconds to complete.

  • On the Explain tab for this individual transaction, you can see the relative weights of the most important features for this prediction. Examine the data, then click the Inspect tab.

OpenScale View Transaction

  • In the Inspect view of this transaction you can see the original features that led to this prediction as well as a series of drop downs and input boxes that offer the ability to change each feature. We can find which features will change the outcome (in this case, from “Risk” to “No Risk”) by clicking the Analysis button. Note that this requires 1000’s of REST calls to the endpoint with slight perturbations in the data, so it can take a few minutes. Click the Analysis tab now.

OpenScale Inspect Transaction

  • In this particular transaction, we see that the presence of a “guarantor” on the loan is the only thing required to flip the outcome from “Risk” to “No Risk”. Other transactions might show a different analysis, so please be aware that your results might vary from this. In the case in this example, you can click the drop down for Others on Loan and change to guarantor.

OpenScale choose new value

  • Choosing this new value for guarantor will expose a button for Score new values. Click this button.

OpenScale Score new value

  • In this example, we can see that the outcome has now been flipped from “Risk” to “No Risk”.

OpenScale flip outcome

  • Now, go back to the Insights Dashboard page by clicking on the left-hand menu icon for Insights, make sure that you are on the Model monitors tab. This time open the monitor configuration for the GermanCreditRiskModelICP model by clicking the 3-dot menu on the tile and then Configure monitors.

OpenScale Insight Dashboard Tile Open

  • Click the Endpoints menu on the left, then the Endpoints tab. Use the Endpoint pulldown to select Debiased transactions. This is the REST endpoint that offers a debiased version of the credit risk ML model, based on the features that were configured (i.e. Sex and Age). It will present an inference that attempts to remove the bias that has been detected.

OpenScale Monitors Endpoints

  • You can see code snippets using cURL, Java, and Python, which can be used in your scripts or applications.

  • Similarly, you can choose the Feedback logging endpoint to get code for Feedback Logging. This provides an endpoint for sending fresh test data for ongoing quality evaluation. You can upload feedback data here or work with your developer to integrate the code snippet provided to publish feedback data to your Watson OpenScale database.

Using the Analytics tools

  • Click on the left-hand menu icon for Insights, make sure that you are on the Model monitors tab, and then open the tile for the GermanCreditRiskModelICP model (click the 3-dot menu on the tile and then View Details):

OpenScale Insight Dashboard Tile Open

  • Notice the red alert indicators on the various monitors (Fairness, Quality, Drift). You should see a red indicator under Fairness. Click on the Fairness score.

Model Overview

  • Click on Analytics -> Predictions by Confidence. It may take a minute or more to create the chart. Here you can see a bar chart that indicates confidence levels and predictions of “Risk” and “No Risk”.

Analytics Predictions by Confidence

  • From this dashboard click on Analytics -> Chart Builder. Here you can create charts using various Measurements, Features, and Dimensions of your machine learning model. You can see a chart that breaks down Predictions by Confidence

    Note: You may need to click the date range for ‘Past Week’ or ‘Yesterday’ to load the data.

  • You can experiment with changing the values and examine the charts that are created.

Analytics Chart builder

Conclusion

This tutorial provides a walkthrough of many of the GUI features using the Watson OpenScale tools. The Auto setup deployment creates a machine learning model, deploys it, and inserts historical data to simulate a model that has been used in production over time. The OpenScale monitors are configured, and the user can then explore the various metrics and data. Please continue to explore on your own. The tutorial is part of the Getting started with Watson OpenScale learning path. To continue, look at the next step Monitoring the model with Watson OpenScale.