K-means clustering using Python on IBM watsonx.ai

K-means clustering is one of the most popular unsupervised learning algorithms, a class of machine learning algorithms that can be used to find patterns in unlabeled data. Clustering algorithms work by classifying data into similar groups or clusters and labeling them so that they can be used for supervised learning. IBM watsonx.ai is an enterprise studio offering for AI builders that integrates all parts of the AI and Data Science lifecycle into one Hybrid Cloud platform for developers. This tutorial explains the fundamentals of performing K-means clustering in Python by using IBM Watson Studio Jupyter Notebooks on watsonx.ai.

The tutorial does not assume previous experience with unsupervised learning or the K-means model. More information on the different types of machine learning algorithms can be found in this IBM Developer article, "Models for machine learning."

What is K-means clustering?

The K-means clustering algorithm (or K-means algorithm) is one of the most popular clustering techniques and machine learning algorithms. The algorithm works by assigning data points to clusters based on a mathematical distance metric (like Euclidean distance) to the center of the cluster. The K-means algorithm’s main goal is to minimize the sum of distances between data points and their respective clusters.

In K-means clustering, the letter "K" represents the number of clusters, or distinct groups, that the algorithm will identify in the data. For example, if K = 2, there will be two clusters. Essentially, it is an iterative process of using a distance metric to measure the similarity of the data within the cluster and to keep adjusting the cluster boundaries until the optimal ones are found.

K-means clustering offers some unique benefits over other clustering techniques, which has led to its widescale adoption. It is relatively simple to implement, scales well to large data sets, guarantees convergence to the centroids, easily adapts to new examples, and generalizes quite robustly to clusters of varying shapes and sizes. Some of the popular real-world use cases for K-means clustering include, but are not limited to, insurance fraud detection, customer segmentation, document classification, delivery store optimization, and cybercrime identification.

K-means clustering is often used when you don’t have a specific outcome variable to predict. Instead, the algorithm automatically discovers features in data to find collections of observations that share similar characteristics. For example, within the context of an image data set, a K-means clustering algorithm could automatically classify different animals into separate groups based on common characteristics such as mammals, reptiles, and amphibians.

Additionally, K-means clustering should not be confused with K-nearest neighbors (KNN), a similarly named supervised learning algorithm that can be used for regression and classification.

Implementing K-means clustering with Watson Studio and watsonx.ai

IBM watsonx.ai is a fully integrated data and AI platform that enables organizations to accelerate AI-powered transformation by enhancing productivity and reducing complexity. It is a cloud-native solution that enables you to put your data to work quickly and efficiently. To perform K-means clustering, watsonx users can seamlessly integrate their data into Watson Studio Jupyter Notebooks and run the algorithm with Python.

The main features of IBM watsonx.ai include:

• Simplifying and automating access to data across multicloud and on-premises data sources without moving data

• Universal safeguarding the use of all data, regardless of source, with applicable governance

• Providing business users with a self-service experience for finding and using data

• Building and deploying trustworthy machine learning models using AutoAI or customizing the models in Python and R notebooks

• Using AI-powered capabilities to automate and orchestrate the data lifecycle

Prerequisites

This tutorial contains general instructions and excerpts of code. To follow along or modify the code for your own use, see the full code on GitHub.

To follow this tutorial, you need an IBM Cloud Account. If you do not have an IBM Cloud account, you can sign up for one to try watsonx.ai for free. If you are creating a free account, Watson Studio, Watson Machine Learning, and Cloud Object Storage instances on the Lite plan are all provisioned for you.

Steps

In this tutorial, you use K-means clustering to show how a real estate agency might identify groups of similar properties to show customers personalized property recommendations based on their budget.

These are the steps you will follow:

1. Create a watsonx.ai project and open a Jupyter Notebook.
2. Generate a data set.
3. Perform exploratory data analysis.
4. Choose K clusters.
5. Interpret K-means clustering.

Step 1. Create a watsonx.ai project

1. Log in to watsonx.ai by using your IBM Cloud account.

2. Create a watsonx.ai project by clicking the + sign in the upper right of the Projects box.

   

3. Create and open a Jupyter Notebook.

   1. On the Assets tab of the project home page, click New task +.

      

   2. Select Work with data models in Python or R notebooks.

      

   3. Name your notebook and, optionally, give it a description.

   4. In the Select runtime drop-down menu, make sure that a Python runtime is selected. The Spark runtimes and NLP runtimes are applicable to big data and NLP use cases specifically. If you are unsure which runtime to pick, select the latest version of the standard Python runtime. Spark and NLP are not applicable to this tutorial.

      

Step 2. Generate a data set

You can choose from multiple open source libraries to perform K-means clustering. This tutorial uses the popular scikit-learn (sklearn) library to generate a test data set and perform the clustering. We chose Scikit-learn because it is an industry standard, and it contains a useful function for clustering called make_blobs. This function creates similar groups of data or "blobs" that you could use to test the efficacy of your clustering algorithms.

To generate your data:

1. Import the required libraries.

    import pandas as pd
    import sklearn
    import matplotlib.pyplot as plt
    import seaborn as sns
    import numpy
   
    from sklearn.cluster import KMeans
    from sklearn.datasets import make_blobs
    from sklearn.decomposition import PCA
    from sklearn.preprocessing import StandardScaler
   

2. Call make_blobs(), and pass in the n_samples, n_features, centers, cluster_std, and random_state arguments. For more information about make_blobs, see the scikit-learn docs.

    X, y = make_blobs(n_samples=5000, n_features=5, centers=5, cluster_std = 0.5, random_state=42)
   

3. Convert your data to a Pandas DataFrame and, optionally, save it as a .csv file for later.

    X, y = make_blobs(n_samples=5000, n_features=5, centers=5, cluster_std = 0.5, random_state=42)
   
    # Convert to DataFrame
    df = pd.DataFrame(X, columns=[f"feature_{i}" for i in range(X.shape[1])])
   
    #add a final column explaining which blob each row should be in
    df['label'] = y
   
    #save df as csv
    #df.to_csv('df.csv')
   
    #to open a csv, use read_csv()
   

4. Rename columns to correspond to the real estate agency scenario by using the following syntax.

    df.rename(columns={
        'feature_0': 'distance_center',
        'feature_1': 'num_rooms',
        'feature_2': 'property_size',
        'feature_3': 'age',
        'feature_4': 'utility_cost',
    }, inplace=True)
   
    df.head() 
   
      feature_0    feature_1    feature_2    feature_3    feature_4    label
    0    -7.125259    -8.744736    7.188210    1.819037    4.104709    1
    1    -7.217549    -8.930194    6.774529    0.740323    4.112526    1
    2    -9.278846    8.544481    6.165782    -5.859933    -5.758941    2
    3    -9.278868    9.960691    6.785733    -5.763617    -6.156320    2
    4    -6.450608    -3.684872    -0.295743    -0.239884    -4.289207    3
   

Step 3. Perform exploratory data analysis

After you've successfully imported your data set, the next step is to perform exploratory data analysis (EDA) to better understand your data. EDA can consist of making visualizations of the data, cleaning and preprocessing it, embedding any text data to convert it into a numerical form, and choosing how to deal with missing or imbalanced data. The exact details on how to perform EDA are outside the scope of this tutorial, but the code here will provide a simplified example.

Following are some typical steps involved in EDA.

1. Visualize the data. If your data has more than two features, you must visualize it in a two-dimensional space. For example, you can plot two features against each other.

    # Visualize the blobs
    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='viridis')
   
    # Annotate each point with its label
    unique_labels = set(y)
    for label in unique_labels:
        plt.scatter(X[y == label][:, 0], X[y == label][:, 1], s=50, label=f'Cluster {label}')
   
    plt.legend(loc='upper right')
   
    plt.show()
   

   

2. Ensure that the columns were imported with the correct data types.

    <div><br class="Apple-interchange-newline">print(df.dtypes)</div>
   
    print(df.dtypes)
   
    distance_center    float64
    num_rooms          float64
    property_size      float64
    age                float64
    utility_cost       float64
    label                int64
    dtype: object
   

3. Check for missing values. Consider imputation or removing missing values if there are many.

    df.isnull().sum()
   
    distance_center    0
    num_rooms          0
    property_size      0
    age                0
    utility_cost       0
    label              0
    dtype: int64
   

4. Check for outliers. If appropriate, consider removing outliers so that the final cluster shapes are not skewed.

5. Standardize scales.

6. Identify correlated features. Consider using a pairwise plot to identify strong correlations and remove features that are highly correlated.

   

Step 4. Choose K

When EDA is complete, you must choose how many clusters to separate your data into. If you already know how many groups you want to make, you can set K manually in scikit-learn. If you are not sure how many groups are appropriate, you can use the elbow method to test different K values.

The elbow method works by computing the Within Cluster Sum of Squares (WCSS), which quantifies the variance (or spread) of data within a cluster. If WCSS decreases, the data within the clusters are more similar. As K increases, there should be a natural decrease in WCSS because the distance between the data in its cluster to its center will be smaller. The ideal K is the "elbow" point of the graph, where WCSS stops decreasing or starts to marginally decrease as K increases. The elbow method is particularly useful for big data sets that have lots of potential clusters because there is a trade-off between computational power required to run the algorithm and the number of clusters generated.

Use the following steps to implement the elbow method.

1. Create an empty list to hold WCSS values at different K values.
2. Fit the clusters with different K values.
3. At each K value, append the WCSS value to the list.

4. Graph the WCSS values versus the number of clusters.

    # Use the elbow method to find a good number of clusters using WCSS (within-cluster sums of squares)
    wcss = []
   
    # Let's check for up to 10 clusters
    for i in range(1, 11):
        kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=42)
        kmeans.fit(df)
        wcss.append(kmeans.inertia_)
   
    plt.plot(range(1, 11), wcss)
    plt.title('Elbow Method')
    plt.xlabel('Number of clusters')
    plt.ylabel('WCSS')
    plt.show()
   

5. Select the "elbow point" of the graph as your K value.

   

6. Rerun K-means with your chosen K value. Use the n_clusters parameter to set the K value. Be sure to set the random_state parameter for reproducibility.

The graph indicates that 5 is the ideal number of clusters, which makes sense because we set the make-blobs number of centers parameter equal to 5.

Step 5. Interpret K-means clustering

Using a dimensionality reduction technique like Principal Component Analysis (PCA), you can graph your clusters in a two-dimensional space and visualize the groupings. This tutorial does not cover the exact details of PCA, but our data has five dimensions, and thus dimensionality reduction is needed to visualize clusters of data with more than two dimensions in a 2-D space. It's likely that most data sets that you encounter when performing K-means clustering have data with more than two dimensions.

The following steps are the high-level steps to interpret K-means clustering.

1. Visualize your clusters.

   

2. Notice the distance between clusters. Clusters that are closer together are likely to be more similar than clusters located further apart.

3. Notice any outliers. Certain data might not fit neatly into a cluster. These points can be considered as outliers or anomalies.

4. Consider a pairwise plot to further visualize how similar the data within the clusters are to each other with scatter plots.

    # Assuming `df` is your DataFrame and `predicted_labels` is the column with cluster assignments
    sns.pairplot(df, hue='predicted_labels', palette='Set1')
    plt.show()
   

   

Now that you've found your clusters, it's time to understand them. If you already have labeled data, you can validate your clustering model by measuring how many of your data points the algorithm was able to label correctly. For unlabeled data, cluster groupings can be used as labels to perform supervised learning. This can be especially applicable for multiclass classification problems. By grouping similar data together into a smaller number of categories, you can decrease the amount of computational power that is required to train a classification model without sacrificing too much model accuracy.

In anomaly detection use cases, any data points that do not belong to a specific cluster or are far from cluster centers can be considered outliers. After clusters are identified, you have a starting point to try and identify why your data might fall into certain groups. When visualized, if there is a small distance between two clusters (like the red and blue clusters in the previous graph), that means that the difference between those two groupings is likely less pronounced than between other ones.

Summary

To summarize, K-means clustering is a powerful unsupervised learning algorithm with many applicable real-world business use cases for automatically grouping data. With Watson Studio Python Notebooks on watsonx.ai, you can deploy this powerful algorithm on data wherever it lives in a hybrid cloud environment.

The real estate agency use case in this tutorial identified five distinct property groupings, which allows the agency to advise customers accordingly. With this data, the agency can determine whether an individual property is competitively priced relative to similar properties by comparing prices within clusters. Additionally, the agency can show personalized recommendations to customers who are interested in specific property profiles or price points.

By categorizing each property into a cluster, the real estate agency now has a labeled data set to perform supervised learning with. Using the computed cluster label as an outcome variable, the agency could use a supervised method like Linear Regression to train a model to predict which category future properties would fall into. This could be used to recommend new properties to customers as they are identified. Additionally, with this labeled data, the agency could consider using this data to train an artificial neural network (also commonly known as deep learning).

Try watsonx for free

Build an AI strategy for your business on IBM watsonx, which brings together new generative AI capabilities, powered by foundation models, and traditional machine learning into a powerful portfolio spanning the AI lifecycle. With watsonx.ai, you can train, validate, tune, and deploy models with ease and build AI applications in a fraction of the time with a fraction of the data.

Try watsonx.ai, the next-generation studio for AI builders.

Next steps

Explore more articles and tutorials about watsonx on IBM Developer.

If you want to learn more, take a look at the following resources:

Machine learning (both unsupervised and supervised learning)

• Machine learning for developers
• Models for machine learning
• What is supervised learning?
• Unsupervised learning for data classification

Neural networks (and deep learning)

• AI vs. Machine Learning vs. Deep Learning vs. Neural Networks: What’s the difference?
• What are neural networks?
• A neural networks deep dive
• Neural networks from scratch

Linear regression algorithms

• Linear regression from scratch
• Learn regression algorithms using Python and scikit-learn