Apply lasso regression to automate feature selection

Lasso regression, also known as L1 regularization, is a form of regularization for linear regression models. Regularization is a statistical method to reduce errors caused by overfitting on training data.

Lasso stands for Least Absolute Shrinkage and Selection Operator. It is frequently used in machine learning to handle high dimensional data as it facilitates automatic feature selection. It does this by adding a penalty term to the residual sum of squares (RSS), which is then multiplied by the regularization parameter (lambda or λ). This regularization parameter controls the amount of regularization applied. Larger values of lambda increase the penalty, shrinking more of the coefficients towards zero, which subsequently reduces the importance of (or altogether eliminates) some of the features from the model, which results in automatic feature selection. Conversely, smaller values of lambda reduce the effect of the penalty, retaining more features within the model.

This penalty promotes sparsity within the model, which can help avoid issues of multicollinearity and overfitting issues within data sets. Multicollinearity occurs when two or more independent variables are highly correlated with one another, which can be problematic for causal modeling. Overfit models will generalize poorly to new data, diminishing their value altogether. By reducing regression coefficients to zero, lasso regression can effectively eliminate independent variables from the model, sidestepping these potential issues within the modeling process. Model sparsity can also improve the interpretability of the model compared to other regularization techniques such as ridge regression (also known as L2 regularization).

As a note, this tutorial focuses on regularization of linear regression models, but it’s worth noting that lasso regression can also be applied in logistic regression.

Prerequisites

To follow this tutorial, you must:

• Create an IBM Cloud account so that you can create a watsonx.ai project.

• Install scikit-learn

Steps

Step 1: Set up your environment

While you can choose from several tools, this tutorial walks you through how to set up an IBM account to use a Jupyter Notebook. Jupyter Notebooks are widely used within data science to combine code, text, images, and data visualizations to formulate a well-formed analysis.

1. Log in to watsonx.ai using your IBM Cloud account.

2. Create a watsonx.ai project.

3. Create a Jupyter Notebook.

This step will open a Notebook environment where you can load your dataset and copy the code from this tutorial to use lasso regression for feature selection.

Step 2. Import libraries and load the data set

In this step, we’ll import the necessary Python libraries for implementing lasso regression. If the necessary libraries are not installed, you can resolve this using the pip install command.

#Load the libraries
import numpy as np
import pandas as pd from sklearn.model_selection
import train_test_split from sklearn.linear_model
import Lasso from sklearn.metrics
import mean_squared_error, r2_score import statsmodels.api as sm
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Lasso
from sklearn.model_selection import GridSearchCV
import seaborn as sns
import matplotlib.pyplot as plt

#Load the dataset
mtcars = sm.datasets.get_rdataset("mtcars", "datasets", cache=True).data
df = pd.DataFrame(mtcars)

This data set is originally from the 1974 Motor Trend US magazine, highlighting different attributes of car models from 1973-1974. To explore the different definitions in this data set, please check out this link.

Step 3. Explore the data set

Before initiating data preprocessing, you should conduct an exploratory data analysis to understand the data's structure and format, including the types of variables, their distributions, and the overall organization of information.

#check shape of the data
df.shape

#Check for missing values in each column of the dataset
print(df.isnull().sum())

There are no missing values in any of the columns of this 11-dimensional data set, which means that we won’t have to drop any rows from our data set or impute any values.

To understand whether any of the features are correlated to one another, different data visualizations, such as pair plots and correlation heatmaps, can show potential signs of multicollinearity; this can subsequently indicate the need for a dimensionality reduction.

#Create a heatmap
corr = df.corr()
plt.figure(figsize=(10,6))
sns.heatmap(corr,
            xticklabels=corr.columns.values,
            yticklabels=corr.columns.values,
            cmap="BuPu",
            vmin=-1,
            vmax=1,
            annot=True)
plt.title("Correlation Heatmap of mtcars dataset")
plt.show()

The heatmap computes and displays the correlation between two variables. The values range from -1 to 1, where -1 indicates perfect negative correlation, 0 indicates no correlation, and 1 denotes perfect positive correlation. In this data set, there is a strong positive correlation of 0.9 between cylinders (cyl) and displacement (disp), meaning that as one increases, the other also increases. There is also a strong negative correlation between mpg and cyl (-.85), mpg and disp (-.85), and mpg and wt (-.87). This means that as displacement, weight, and the number of cylinders increase, the fuel efficiency (mpg) tends to decrease.

Step 4. Split the data set

Given the exploratory analysis, you can conclude that there are some correlated features within the data set, and as a result, you can expect that the lasso regression model will automatically drop features to reduce the redundancy within the data set. That said, it is important to keep in mind that lasso regression has its limitations, and it will arbitrarily drop one of the correlated features from the model.

From here, you split the data set into two sets, a training set and the test set. Setting the random state ensures that the splits you generate are reproducible.

features = df.columns[1:]
target = df.columns[0]
X = df[features].values
y = df[target].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Step 5. Standardize data points through feature scaling

Next, you’ll normalize the data to ensure that the scale of our predictors does not negatively impact variable selection as the scale of our variable does affect the size of our coefficients. Scaling our variables with a mean of zero and a standard deviation of one is a common feature scaling technique; this will allow the lasso model to select the most important features more accurately.

#scaling and centering the data
sc = StandardScaler()
X_train_scaled = sc.fit_transform(X_train)
X_test_scaled = sc.transform(X_test)

Step 6: Implement and evaluate the model

Now, we’ll apply this algorithm to our data set and return the predicted values. It’s worth noting that we’re using GridSearchCV with the Lasso model for this tutorial, but you can also use LassoCV, which automatically incorporates cross-validation into the model.

Important: While you can use LassoCV or GridSearchCV with the Lasso model in scikit-learn, they won’t necessarily yield the same results. According to the documentation, LassoCV is warm started using the coefficients of a previous iteration of the model on the regularization path. This tends to speed up the hyperparameter search for alpha value. This is further explained in a scikit-learn bug ticket.

#Initialize lasso regression model
model = Lasso(max_iter=10000) #default alpha is 1
model.fit(X_train_scaled,y_train)

y_pred = model.predict(X_test_scaled)

#Calculate R-squared
rsquared = r2_score(y_test, y_pred)
print(f"R-squared: {rsquared}")

#Calculate the mean squared error
mse = mean_squared_error(y_test, y_pred)
print(f"Mean Squared Error: {mse}")

The R-squared and mean squared error (MSE) for this model is 0.73 and 9.06, respectively. To see which coefficients shrunk down to zero, we will plot our coefficients in both a dataframe and a graph.

coeff = pd.Series(model.coef_, index=features)
coeff

alphas = np.linspace(0.01, 1000, 100)
coefs = []

for a in alphas:
    model.set_params(alpha=a)
    model.fit(X_train_scaled, y_train)
    coefs.append(model.coef_)

ax = plt.gca()
ax.plot(alphas*2, coefs)
ax.set_xscale('log')
ax.legend(features)
ax.grid(False)
plt.axis('tight')
plt.xlabel('alpha')
plt.ylabel('coefficients')
plt.title("Coefficient values with increasing values of alpha")

In our dataframe, we can see that 7 of the 10 variables have shrunk down to zero. According to this model, there are three features that are key predictors of mpg, which are the number of cylinders, horsepower, and the weight of the vehicle. However, when we observe the line graph, it looks like the model may have dropped some variables, like disp, due to high collinearity, which may have also had good predictive power. As you can see, lasso regression is helpful in reducing the number of features in a model to some of the most important ones, but it's not without its limitations.

Step 7: Optimize model with hyperparameter tuning

We want the lowest possible value of MSE for the optimal lasso model, and we find this by trying different values of alpha via grid search. GridSearchCV will help us to conduct this optimization via cross-validation, allowing us to find or confirm the best value for the alpha hyperparameter.

alphas = {"alpha": 10.0 ** np.arange(-5, 6)}
grid_search = GridSearchCV(model, alphas, scoring='neg_mean_squared_error', cv=5)
grid_search.fit(X_train_scaled,y_train)

print(f"Best value for lambda : ", grid_search.best_params_)
print("Best score for cost function: ", grid_search.best_score_)

This grid search confirms that the default alpha value of 1 is, in fact, the optimal value for this hyperparameter.

Summary and next steps

In this tutorial, you learned how to apply lasso regression to conduct automatic feature selection, which identified the subset of features in the data set that have the most predictive value to our target variable. Want to try this out with a few exercises? Then, take a look at the accompanying guided project.

Try watsonx for free

Build an artificial intelligence (AI) strategy for your business on one collaborative AI and data platform called IBM watsonx, which combines new generative AI capabilities powered by foundation models and traditional machine learning into a powerful platform spanning the AI lifecycle. With watsonx.ai, you can train, validate, tune, and deploy models with ease and build AI applications in a fraction of the time with a fraction of the data.

Try watsonx.ai, the next-generation studio for AI builders.

Next steps

Explore more articles and tutorials about watsonx on IBM Developer.

Also, to learn more about other supervised machine learning models that you can apply to classification and regression problems, see these tutorials in the Getting started with machine learning learning path:

• Tutorial: Learn classification algorithms using Python and scikit-learn
• Tutorial: Learn regression algorithms using Python and scikit-learn