Prediction intervals explained: A LightGBM tutorial

When you are performing regression tasks, you have the option of generating prediction intervals by using quantile regression, which is a fancy way of estimating the median value for a regression value in a specific quantile. Simply put, a prediction interval is just about generating a lower and upper bound on the final regression value. This is incredibly important for some tasks, which I explain in this article.

Light Gradient Boosting Machine (LightGBM) helps to increase the efficiency of a model, reduce memory usage, and is one of the fastest and most accurate libraries for regression tasks. To add even more utility to the model, LightGBM implemented prediction intervals for the community to be able to give a range of possible values.

This article describes some theoretical and practical approaches to making a regression model that includes the upper and lower bounds to form prediction intervals.

Why use them?

You can never be 100 percent certain about one prediction from one model. Instead, the idea is to give an interval back to a person who ends up controlling the final decision based on the range that is given by the model. For example, if you are trying to set the price for a house, it's common knowledge that the price depends on how well-maintained and renovated the house is. Therefore, you want to give a range. If the house is poorly maintained, perhaps the price would land in the lower end of the price interval, and if it's well-maintained, the price might be at the upper end.

Quantile regression

In the typical linear regression model, you track the mean difference from the ground truth to optimize the model. However, in quantile regression, as the name suggests, you track a specific quantile (also known as a percentile) against the median of the ground truth.

Quantiles and assumptions

Using the median approach lets you specify the quantiles. For example, you might specify that you want the 5-percent quantile (covering 5 percent of the data) and the 95-percent quantile (covering 95 percent of the data). This gives you a lower and upper boundary that you can use as the smallest and highest estimates in a regression task.

You might wonder why it's important to estimate these quantiles by using the median rather than the mean. The problem with just using the mean comes when outliers are present, as this can sometimes result in poor predictions because the mean of a group of values heavily emphasizes the outlier values. Therefore, when using the median, you avoid being prone to outliers and end up producing better lower and upper boundaries.

Additionally, unlike linear regression, you do not make any assumptions about the distribution of the data, which makes it even more useful and more accurate in certain scenarios.

Regression

For linear regression, the following image shows the equation that you often see used to make predictions (see objective function).

However, you must understand that this is incredibly "linear" and works well only for linear problems. The following image shows the loss function for optimizing the linear regression.

That changes in quantile regression because you must be able to account for the different quantiles.

• When the alpha is high (for example, 0.95), the errors that are less than zero receive a lower error value than if they are greater than zero.

• When the alpha is low (for example, 0.05), the errors that are less than zero receive a higher error value than if they are greater than zero, where they receive a smaller error value.

Now that you've seen how the loss function is calculated for a quantile regression model, I'll dive into the Python example prepared for this article.

Python example

To make prediction intervals, you need a lower bound and upper bound for the prediction that you generate using your model. To generate these bounds, you use the following method.

1. Choose a prediction interval. Typically, you set it to 95 percent or 0.95. I call this the alpha parameter ($\alpha$) when making prediction intervals.

2. Train your model for making predictions on your data set.

3. Train two models, one for the lower bound and another for the upper bound. You must set two parameters for this to work: objective and alpha.

However, to train any model, you first need to find a suitable data set for your use case.

Data loading and preparation

For this article, I use the California Housing data set. The following table shows four out of the eight available features in the data set along with a sample row of data. Upon inspection of the data, you see that there are built-in features to help you predict the price of a house, for example, the median income (in thousands of dollars) of the family.

You only need a few packages to load and prepare the data for use in a LightGBM model. For loading and preprocessing the data, you only need the Pandas and scikit-learn package.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_california_housing

To load the data, you define a generic function that takes the data_loader object from the scikit-learn data sets module. This function loads the data and splits it into input (x) and output (y) in Pandas DataFrames.

def sklearn_to_df(data_loader):
    X_data = data_loader.data
    X_columns = data_loader.feature_names
    x = pd.DataFrame(X_data, columns=X_columns)

    y_data = data_loader.target
    y = pd.Series(y_data, name='target')

    return x, y

After defining the data_loader function, you can call it and get the input and output data. You must split this into training and testing data sets, so you can use the train_test_split function that the scikit-learn package comes with.

x, y = sklearn_to_df(fetch_california_housing())

x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.2, random_state=42)

Now that your training and testing data is ready for modeling, I'll proceed to the practical example for the prediction intervals using LightGBM.

Prediction intervals

Moving into modeling, you need a few packages to start with. You need the LightGBM package because that is your model, and then you need scikit-learn and Matplotlib for utilities along the way. Additionally, you load the data from the data_loader explained in the previous section.

import lightgbm as lgb
import matplotlib.pyplot as plt
from sklearn.metrics import r2_score
from data_loader import x_train, x_test, y_train, y_test

You start by training your regular regression model for predicting the price of the house. This is the standard procedure in which you define the model and fit it to the training data (x_train and y_train). Additionally, you also let the fitted model make predictions on the test set x_test.

regressor = lgb.LGBMRegressor()
regressor.fit(x_train, y_train)
regressor_pred = regressor.predict(x_test)

To train the lower-bound model, you specify the quantile and alpha parameter, so the procedure is the same as when you are training any other LightGBM model.

lower = lgb.LGBMRegressor(objective = 'quantile', alpha = 1 - 0.95)
lower.fit(x_train, y_train)
lower_pred = lower.predict(x_test)

The same approach goes for the upper-bound model. You specify the quantile and alpha parameter, train the model, and then make predictions.

upper = lgb.LGBMRegressor(objective = 'quantile', alpha = 0.95)
upper.fit(x_train, y_train)
upper_pred = upper.predict(x_test)

After this, you still want to check how well your model performs. You only measure your normal model and not the upper- or lower-bound models. This is because the score is unreliable for these quantile regression models because it tells you the most probable lower or upper predictions. Essentially, that would mean that you could get an r-squared score of 0.3 without being able to use this as your final score of the model.

score = r2_score(y_test, regressor_pred)
print(score)

Finally, you get the r-squared score of 0.836, which is decent for a regression model.

Next, I describe how to plot the final values of the model.

The first few lines (1-5) create the plot and "scatter" your data points by the test feature "MedInc" and the house price together on your coordinate system. You mark the lower bound with a green color, the regression model's predictions with the aqua/teal color, and your upper bound with a blue color. Additionally, you want to plot a line through the middle of all of the points to distinguish where you would normally cut off the lower and upper bounds when making a final decision on the price of a house. Therefore, you use the plot function and make sure to sort both the MedInc test feature and the predictions so they fit into the plot.

plt.figure(figsize=(10, 6))

plt.scatter(x_test.MedInc, lower_pred, color='limegreen', marker='o', label='lower', lw=0.5, alpha=0.5)
plt.scatter(x_test.MedInc, regressor_pred, color='aqua', marker='x', label='pred', alpha=0.7)
plt.scatter(x_test.MedInc, upper_pred, color='dodgerblue', marker='o', label='upper', lw=0.5, alpha=0.5)
plt.plot(sorted(x_test.MedInc), sorted(lower_pred), color='black')
plt.plot(sorted(x_test.MedInc), sorted(regressor_pred), color='red')
plt.plot(sorted(x_test.MedInc), sorted(upper_pred), color='black')
plt.legend()

plt.show()

Finally, I produce the actual chart, which is shown in the following image. Everything that I just described is something that you can see. The x-axis is the MedInc feature, and the y-axis is the house price prediction.

Conclusion

In this article, you learned theoretical and practical approaches to making a regression model that includes the upper and lower bounds to form prediction intervals. This will now be in your toolbox when you need to perform regression tasks in the future on other data sets.