Classifying data using the SVM algorithm using Python

Support vector machine (SVM) is a supervised machine learning algorithm that classifies data by finding an optimal line or hyperplane that maximizes the distance between each class in an N-dimensional space.

The Support Vector Machine algorithm is commonly used within classification problems. They distinguish between classes by finding the maximum margin between the closest data points of opposite classes, creating the optimal hyperplane. The number of features in the input data determine if the hyperplane is a line in a 2D space or a plane in an N-dimensional space.

Because multiple hyperplanes can be found to differentiate classes, maximizing the margin between points enables the algorithm to find the best decision boundary between classes. This differentiation, in turn, enables the SVM algorithm to generalize well to new data and make accurate classification predictions. The lines that are adjacent to the optimal hyperplane are known as support vectors as these vectors run through the data points that determine the maximal margin.

The SVM algorithm is widely used in machine learning as it can handle both linear and nonlinear classification tasks. However, when the data is not linearly separable, kernel functions are used to transform the data higher dimension feature space to enable linear separation. This application of kernel functions can be known as the “kernel trick,” and the choice of kernel function, such as linear kernels, polynomial kernels, radial basis function (RBF) kernels, or sigmoid kernels, depends on data characteristics and the specific use case.

In this tutorial, learn how to apply support vector classification to a credit card clients data set to predict default payments for the following month. The tutorial provides a step-by-step guide for how to implement this classification in Python using scikit-learn. You also gain insights into how to reduce dimensionality within the data set using principal component analysis (PCA), enhancing the efficiency of the model.

For more information, look at the Reducing dimensionality with principal component analysis with Python tutorial.

Prerequisites

• Create an IBM Cloud account
• Install scikit-learn

Steps

Step 1. Set up your environment

While you can choose from several tools, this tutorial walks you through how to set up an IBM account to use a Jupyter Notebook. Jupyter Notebooks are widely used within data science to combine code, text, images, and data visualizations to formulate a well-formed analysis.

1. Log in to watsonx.ai using your IBM Cloud account.

2. Create a watsonx.ai project.

3. Create a Jupyter Notebook.

From here, a notebook environment opens for you to load your data set and copy code from this beginner tutorial to tackle a simple classification problem.

Step 2. Import libraries and load the data set

You must import the necessary Python libraries so that you can work with the default of the credit card clients data set, perform data preprocessing, and build and evaluate your SVM model. These libraries are crucial for data manipulation, visualization, and machine learning tasks. If they're not installed, you can resolve this with a quick pip install.

The study associated with this data set focused on customers' default payments and compared the predictive accuracy of the probability that a client will default on payment across six data mining methods. It used a binary variable, "default payment" (Yes = 1; No = 0), as the response variable and used 23 variables as explanatory variables. To explore the data definitions, refer to UCI’s data repository.

#Load the required libraries
import pandas as pd
import numpy as np
import seaborn as sns
from sklearn.utils import resample
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.decomposition import PCA
import matplotlib.colors as colors
import matplotlib.pyplot as plt
!pip3 install xlrd

# Import the data set
df = pd.read_excel('https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls', header=1)

Step 3. Explore the data set

Prior to initiating data preprocessing, you should conduct an exploratory data analysis to understand the data's structure and format, including the types of variables, their distributions, and the overall organization of information. This exploration direct the modeling approach.

In this step, you explore the first ten rows of the pandas DataFrame.

#Explore the first ten rows of the data set
df.head(10)

In the output provided, each row represents an individual entry. The columns represent specific features like the identification number, credit limit, sex, education, marriage status, age, payment status across several months, bill statement amounts, payment amounts, and a target variable indicating default in the following month. The numerical data in each column provides information about the respective feature or attribute. The "default payment next month" column represents the class label or target variable for classification tasks in an SVM.

# Rename the columns
df.rename({'default payment next month': 'DEFAULT'}, axis='columns', inplace=True)

#Remove the ID column as it is not informative
df.drop('ID', axis=1, inplace=True)
df.head()

To clean up the data set, rename and simplify the "default payment next month" column to "DEFAULT", making it convenient for further analysis or modeling tasks. The DataFrame also no longer contains the ID column because it is considered non-informative for analysis or modeling tasks.

Step 4. Analyze missing data

In this step, you proceed with data preprocessing to refine the data for subsequent analysis and modeling tasks. You ensure that each column only contains acceptable values by checking for null or invalid inputs based on the data definitions.

# check dimensions for invalid values
df['SEX'].unique()
df['MARRIAGE'].unique()
df['EDUCATION'].unique()
df['AGE'].unique()

# count missing or null values
print(len(df[pd.isnull(df.SEX)]))
print(len(df[pd.isnull(df.MARRIAGE)]))
print(len(df[pd.isnull(df.EDUCATION)]))
print(len(df[pd.isnull(df.AGE)]))

#count of missing data
len(df.loc[(df['EDUCATION'] == 0) | (df['MARRIAGE'] == 0)]) #output: 68

The output indicates that some of the data does not align with the data definitions, specifically EDUCATION and MARRIAGE columns. EDUCATION includes three types of invalid values, which are 0, 5, and 6, and the MARRIAGE column includes 0 as an invalid value. Assume that a 0 encoding is supposed to represent missing data and that a value of 5 or 6 within EDUCATION is representative of other unspecified education levels (for example, Ph.D. or a master's degree), which is not represented within the data definition.

68 rows exist in the DataFrame where either the EDUCATION or the MARRIAGE column is zero. Next, filter the rows where the EDUCATION and MARRIAGE columns have non-zero values.

#Filter the DataFrame
df_no_missing_data = df.loc[(df['EDUCATION'] != 0) & (df['MARRIAGE'] != 0)]

The new DataFrame contains rows where the EDUCATION column and the MARRIAGE column do not have values equal to 0; this filtering yields 29,932 rows. Now that you have removed these missing values, check whether the number of defaults in this data set is balanced.

# Explore distribution of data set
# count plot on ouput variable
ax = sns.countplot(x = df_no_missing_data['DEFAULT'], palette = 'rocket')

#add data labels
ax.bar_label(ax.containers[0])

# add plot title
plt.title("Observations by Classification Type")

# show plot
plt.show()

The output displays the count of accounts with and without a credit card default, indicating an unbalanced data set. To address this, downsample the data to balance it out.

Step 5. Downsample the data set

In this step, you balance the data set by splitting the data set into two categories: individuals regularly paying their credit card debt (default) and individuals whose debt remains unpaid (no default). You then downsample the data set by obtaining 1,000 samples for each category, representing the default payment as Yes = 1 and No = 0. From there, you merge the two data sets to create your balanced data set.

from sklearn.utils import resample

# split data
df_no_default = df_no_missing_data.loc[(df_no_missing_data['DEFAULT']==0)]
df_default = df_no_missing_data.loc[(df_no_missing_data['DEFAULT']==1)]

# downsample the data set
df_no_default_downsampled = resample(df_no_default, replace=False, n_samples=1000, random_state=42 )
df_default_downsampled = resample(df_default, replace=False, n_samples=1000, random_state=42 )

#check ouput
len(df_no_default_downsampled)
len(df_default_downsampled)

# merge the data sets
df_downsample = pd.concat([df_no_default_downsampled, df_default_downsampled ])
len(df_downsample)

Step 6. Hot-encode independent variables

Scikit-learn does not natively support categorical data, and as a result, you must transform this data using hot-encoding, which codes each category in a given column as a binary variable using 0 or 1. This data transformation prevents SVM from treating the data as continuous, and ensures that each categorical variable has an equal likelihood of clustering together. You use OneHotEncoder for this task. To learn more, explore the scikit documentation.

from sklearn.preprocessing import OneHotEncoder
# isolate independent variables
X = df_downsample.drop('DEFAULT', axis=1).copy()

ohe = OneHotEncoder(sparse_output=False, dtype="int")
ohe.fit(X[['SEX', 'EDUCATION', 'MARRIAGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']])
X_ohe_train = ohe.transform(X[['SEX', 'EDUCATION', 'MARRIAGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']])

X_ohe_train

transformed_ohe = pd.DataFrame(
    data=X_ohe_train,
    columns=ohe.get_feature_names_out(['SEX', 'EDUCATION', 'MARRIAGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']),
    index=X.index,
)
transformed_ohe.head()

# merge dataframes
X_encoded = pd.concat([X, transformed_ohe], axis=1)
X_encoded

Step 7. Split the data set

You must split the data set into two distinct sets: the training set and the test set. The training set trains the SVM model to differentiate between different classes based on the features provided. The test set helps you to evaluate model performance.

You should also scale your data before applying your SVM model to the data. This means that your independent variables (that is, X_encoded) have a mean of zero and a standard deviation of 1. As noted in this paper, you scale the data "to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges," enabling SVMs to find an optimal solution more quickly.

from sklearn.preprocessing import scale
y = df_downsample['DEFAULT'].copy()
X_train, X_test, y_train, y_test = train_test_split(X_encoded, y, test_size=0.3, random_state=42)

#scale the data
X_train_scaled = scale(X_train)
X_test_scaled = scale(X_test)

Step 8. Classify accounts and evaluate the model

Now, you build an initial SVM classifier, fitting your training data on the model and evaluating it with the test data set. You then plot the results using a confusion matrix to evaluate model performance.

from sklearn.metrics import confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay

clf_svm = SVC(random_state = 42)
clf_svm.fit(X_train_scaled, y_train)

#calculate overall accuracy
y_pred = clf_svm.predict(X_test_scaled)
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy:.2%}')

class_names = ['Did Not Default', 'Defaulted']
disp = ConfusionMatrixDisplay.from_estimator(
        clf_svm,
        X_test_scaled,
        y_test,
        display_labels=class_names,
        cmap=plt.cm.Blues)

Here, you can see that your classifier did not perform as well as expected. Of the 302 accounts that did not default, and of the 298 accounts that did default, only 79% (239 accounts) and 58% (172 accounts) were correctly classified in their respective categories.

Step 9. Optimize model with hyperparameter tuning

To improve model performance, you can use both cross validation and GridSearchCV() to find the optimal hyperparameters within your model. More specifically, you're looking to identify the best values for your regularization parameter (C), gamma, and kernel.

You might recall from your SVM explainer that your C parameter applies a penalty for misclassifications and your gamma parameter applies weights for different data points. It's important to be aware that higher values of gamma have a tendency to increase the risk of overfitting the training set, which in turn leads to poor generalization onto new data. For the sake of data processing time, you only use the radial basis function kernel (RBF kernel) for this hyperparameter optimization, but note that this could include other kernels, such as poly or sigmoid.

param_grid = {'C':[0.5,0.1,1,10,100,1000],
              'gamma':['scale', 1,0.1, 0.01,0.001,0.0001],
              'kernel':['rbf']}

optimal_params = GridSearchCV(SVC(), param_grid, cv = 5, scoring='accuracy', verbose=3)
optimal_params.fit(X_train_scaled, y_train)

# see "best" parameters
optimal_params.best_params_

# refit model with optimal hyperparameters
grid_predictions = optimal_params.predict(X_test.values)
clf_svm = SVC(random_state = 42, C=.5, gamma=0.01)
clf_svm.fit(X_train_scaled, y_train)

#calculate overall accuracy
y_pred = clf_svm.predict(X_test_scaled)
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy:.2%}')

# plot confusion matrix
class_names = ['Did Not Default', 'Defaulted']
disp = ConfusionMatrixDisplay.from_estimator(
        clf_svm,
        X_test_scaled,
        y_test,
        display_labels=class_names,
        cmap=plt.cm.Blues)

The new confusion matrix demonstrates that hyperparameter tuning didn't help improve the model much. You've seen minor improvements to the classification of accounts that did not default. Now, the model correctly classifies 81% of the data correctly versus 79%. Overall, you've improved the accuracy score of the model by less than 1 percentage point.

Step 10. Reducing dimensionality within the data

Because it is difficult to visualize data in high dimensional spaces, such as this 24-column data set, you can use a dimensionality reduction approach, such as PCA, to help you project the data in a lower dimensional space.

pca = PCA()
X_train_pca = pca.fit_transform(X_train_scaled)

per_var = np.round(pca.explained_variance_ratio_*100, decimals=1)
labels = [str(x) for x in range(1, len(per_var)+1)]

#plot scree plot
plt.bar(x=range(1, len(per_var)+1), height=per_var)
plt.tick_params(axis='x', which = 'both', bottom=False, top=False, labelbottom=False)
plt.ylabel("Explained variance (%)")
plt.xlabel('Principal Components')
plt.title('Scree Plot')
plt.show()

Looking at a scree plot, the data "elbows" around the 10th component; that said, it's better than 24 dimensions. You'll project the data using two principal components to visualize the data. However, acknowledge that it will not do a great job at capturing the majority of the variation in the data.

train_pc1_coords = X_train_pca[:, 0]
train_pc2_coords = X_train_pca[:, 1]

pca_train_scaled = scale(np.column_stack((train_pc1_coords, train_pc2_coords)))

param_grid = {'C':[0.01, 0.1, 0.5, 1, 10, 100],
              'gamma':[1, 0.75, 0.5, 0.25, 0.1, 0.01, 0.001],
              'kernel':['rbf']}
optimal_params = GridSearchCV(SVC(), param_grid, cv = 5, scoring='accuracy', verbose=3)

optimal_params.fit(pca_train_scaled, y_train)

Now, you plot the data and the decision boundaries.

clf_svm = SVC(random_state=42, C=1000, gamma=0.001)
clf_svm.fit(pca_train_scaled, y_train)

X_test_pca = pca.transform(X_train_scaled)
test_pc1_coords = X_test_pca[:, 0]
test_pc2_coords = X_test_pca[:, 1]

x_min = test_pc1_coords.min()-1
x_max = test_pc1_coords.max()+1
y_min = test_pc2_coords.min()-1
y_max = test_pc2_coords.max()+1

xx, yy = np.meshgrid(np.arange(start=x_min, stop=x_max, step=0.1),np.arange(start=y_min, stop=y_max, step=0.1) )

Z = clf_svm.predict(np.column_stack((xx.ravel(), yy.ravel())))
Z = Z.reshape(xx.shape)

# visualizing the data
fig, ax = plt.subplots(figsize=(10,10))
ax.contourf(xx,yy, Z, alpha=0.1)
cmap = colors.ListedColormap(['#e41a1c', '#4daf4a'])
scatter = ax.scatter(test_pc1_coords, test_pc2_coords, c=y_train, cmap=cmap, s=100, edgecolors='k', alpha=0.7)
legend = ax.legend(scatter.legend_elements()[0], scatter.legend_elements()[1], loc='upper right')
legend.get_texts()[0].set_text('Did Not Default')
legend.get_texts()[1].set_text('Defaulted')
ax.set_ylabel('PC2')
ax.set_xlabel('PC1')
ax.set_title('Visualizing the Decision Boundary Using Principal Components')
plt.show()

The yellow area represents "defaulted" predictions, whereas the pink area represents data where accounts are not expected to default. All of the dots are part of the training set, indicating their respective classification as denoted in the legend.

Summary and next steps

In this tutorial, you learned how to apply SVM classification to classify data around credit card defaults. You also fine-tuned your classifier by optimizing the hyperparameters, allowing for minor improvements in accuracy. Finally, you applied PCA to visualize the decision boundaries for any new data, gaining insight into class separability and the limitations of this particular classifier.

Try watsonx for free

Build an artificial intelligence (AI) strategy for your business on one collaborative AI and data platform called IBM watsonx, which combines new generative AI capabilities powered by foundation models and traditional machine learning into a powerful platform spanning the AI lifecycle. With watsonx.ai, you can train, validate, tune, and deploy models with ease and build AI applications in a fraction of the time with a fraction of the data.

Try watsonx.ai, the next-generation studio for AI builders.

Next steps

Explore more articles and tutorials about watsonx on IBM Developer.

Also, to learn more about other supervised learning algorithms that you can apply to classification and regression problems, see these tutorials in the Getting started with machine learning learning path:

• Tutorial: Learn classification algorithms using Python and scikit-learn
• Tutorial: Learn regression algorithms using Python and scikit-learn