Implementing KNN in R

KNN is a machine learning algorithm that can be used for classification or regression. KNN is often used for exploratory data mining technique or as a first step in a more complex data pipeline.

The KNN algorithm is a robust and very versatile classifier that is often used as a benchmark for more complex classifiers like the support vector machines (SVM) or for more complex neural networks. Even though it's simple and easy to understand, KNN can outperform more powerful classifiers and is used in a wide variety of applications.

In this tutorial we'll review how the KNN algorithm works and how you can implement it in R. We'll review how to tune hyperparameters of KNN models and then evaluate the performance of models with different data sets. Let's get started!

What is KNN?

The KNN algorithm is a classic non-parametric supervised machine learning model. Unlike unsupervised machine learning algorithms like K-Means, KNN requires labeled data. The abbreviation stands for "K Nearest Neighbors" and the algorithm predicts the labels of the test data set by looking at the labels of its closest neighbors in the feature space of the training data set. Typically KNN uses Euclidean distance, though other distance metrics like the Manhattan distance can be used as well. The “K” is the most important hyperparameter that can be tuned to optimize the performance of the model.

KNN is a comparatively simple algorithm which provides good results for a wide range of classification problems. KNN can be applied to both small and large data sets. It does have some drawbacks though, such as it can be very computationally expensive for large data sets or when a data set has a feature space with a high number of dimensions.

The KNN algorithm is non-parametric, which means it makes no explicit assumptions about the underlying distribution of the data. If your data does not at all fit a specific distribution but you choose a learning model that assumes a linear distribution, for instance a Naive Bayes model, then the algorithm would make extremely poor predictions. Because KNN doesn't require specific distributions for the features of the data, it requires less assumption checking.

Another feature of KNN is that it uses memory-based learning so it doesn’t explicitly learn a model (this is also called lazy learning). The algorithm memorizes training instances which are subsequently used as knowledge for the prediction phase, which means that only when a query to KNN is made the algorithm will simply use the training instances to return an answer.

When we use KNN as a classification algorithm, it follows these steps to obtain a new data point:

1. We choose a value for K, which is the number of nearest neighbors that will be used to make the prediction.
2. We then calculate the distance between that point and all the points in the training set.
3. Select the K nearest neighbors (KNN) based on the distances calculated.
4. Assign the label of the majority class to the new data point.

Training KNN uses the above steps for all the data points in the training set and then evaluates the accuracy of the algorithm.

Prerequisites

You need an IBM Cloud account to create a watsonx.ai project.

Steps

We'll use the KNN algorithm as implemented in the caret library to create labels for a data set containing data about potentially diabetic patients. The data consist of 19 variables on 403 subjects from 1046 subjects who were interviewed in a study to understand the prevalence of obesity, diabetes, and other cardiovascular risk factors.

The original data came from the Biostatistics program at Vanderbilt. More information can be found here.

Step 1: Set up your environment

While you can choose from several tools, this tutorial walks you through how to set up an IBM account to use a Jupyter Notebook. Jupyter Notebooks are widely used within data science to combine code, text, images, and data visualizations to formulate a well-formed analysis.

1. Log in to watsonx.ai using your IBM Cloud account.

2. Create a watsonx.ai project.

3. Create a Jupyter Notebook using the Python Kernel.

This step will open a Notebook environment where you can load your data set and copy the code from this tutorial to perform the association rule mining process.

Step 2: Install and load libraries

In this step, we’ll install the following libraries:

• httr is a library that provides utilities for reading data from the web
• caret used for simplifying the process of model training, tuning, and evaluation
• dplyr is a Swiss army knife of a library for data manipulation
• readxl is used to turn Excel files into R-compatible formats

Before you move forward, check in the console to be sure that all the required libraries have been installed correctly.

install.packages("caret")
install.packages("httr")
install.packages("readxl")
install.packages("dplyr")
install.packages('mt')

library(mt)
library(caret)
library(httr)
library(readxl)
library(dplyr)

Step 3: Load our data

We'll get the .xlsx file from Data.World and turn it into an R data.frame. This is very similar to the way that you read in a CSV file but has one additional step. The data file is real patient data to help predict diabetes (yes or no) using demographic and lab variables.

GET("https://query.data.world/s/qmc5py3qjodoyglqcny7ihdowxwepf?dws=00000", write_disk(tf <- tempfile(fileext = ".xlsx")))
df <- data.frame(read_excel(tf))

Now that we've loaded our data, we can see the first few rows:

head(df)

We'll be building a KNN classifier to predict diabetes, so we'll want to look at the Diabetes field, so we should see how well balanced it is:

# Checking how balance is with the dependent variable
prop.table(table(df$Diabetes))

This isn't quite balanced, but it's very common for medical data sets like this to be unbalanced. It can be important to correctly account for the actual frequency of the condition in the population. We'll select the non-demographic columns to begin and create a reduced data set:

df_reduced <- df[c("Diabetes", "Cholesterol", "Glucose", "BMI", "Waist.hip.ratio", "HDL.Chol", "Chol.HDL.ratio", "Systolic.BP", "Diastolic.BP", "Weight")]

KNN is quite sensitve to the scale of the variables that it's using so it's often helpful to first standardize the variables. We can do this using the preProcess function of the caret package:

preproc_reduced <- preProcess(df_reduced[,2:10], method = c("scale", "center"))
df_standardized <- predict(preproc_reduced, df_reduced[, 2:10])

Now we have scaled and centered values for our data.frame:

summary(df_standardized)

We can apply those scaled and centered values to our data using the bind_cols method from dpylr to normalize all of our data.

df_stdize <- bind_cols(Diabetes = df_reduced$Diabetes, df_standardized)

Now we have a data set which will work well for KNN.

Step 4: Training KNN

We've done some initial EDA and then scaled and centered all the variables that we're using, so we're ready to create a test/train split so that we can train a KNN model. We can use the createDataPartition from caret to randomly select 80% of the data to use for the training set and 20% of the data to use for the test set. Then, we create two different data.frames using partition point returned by createDataPartition.

set.seed(42)
# p indicates how much of the data should go into the first created partition
param_split<- createDataPartition(df_stdize$Diabetes, times = 1, p = 0.8, list = FALSE)
train_df <- df_stdize[param_split, ]
test_df <- df_stdize[-param_split, ]

Now we're ready to train the KNN. The trainControl method is a helper function for all the nuances of the KNN train method. It allows you to easily set up the KNN training parameters. In this case we're using the following two hyperparameters:

• method: This is the resampling method used. There are many options, among them: "boot", "adaptive_boot", "LOOCV" (Leave One Out Cross Validation). In our case, we want to use "cv" or cross validation.
• number: This is either the number of folds or number of resampling iterations. Because we're using "cv," it will be the number of folds.

After we've created our training parameter object, we can pass it to the train method. The train method uses the same formula notation as many R methods and libraries, with the form: y ~ x, where y is the dependent variable and x is the independent variable.

In our case, we'll use all variables to predict Diabetes. In R, we can simply pass Diabetes ~ ., with the . indictating that we want to use all of the variables.

We then specify the training data, that we're training a KNN, and pass the training hyperparameters that we built with trainControl.

trnctrl_df <- trainControl(method = "cv", number = 10)
model_knn_df <- train(Diabetes ~ ., data = train_df, method = "knn", 
                       trControl = trnctrl_df, 
                       tuneLength = 10)

model_knn_df

This returns a series of KNN models and their associated metrics and tells us that the optimal number of nearest neighbors to be considered is 11 and the model accuracy for that K is 0.9198387. That's good but the Kappa score is 0.6138024, which can indicate that our model is not predicting the two classes with an even accuracy. Remember that most of the records in our data set are not patients with diabetes. Our model might simply guess that no patients have diabetes and achieve fairly high accuracy that way.

Sometimes we can find that simpler models have better accuracy. To see which explanatory variables are having the most impact on our outcome variable, we can use fs.anova to check the analysis of variance for each variable and use that in a second KNN.

df_stdize$Diabetes = ifelse(df_stdize$Diabetes == "Diabetes", 1, 0)

fs <- fs.anova(df_stdize, df_stdize$Diabetes)
print(fs$fs.order)

We can see that "Glucose" is the most important feature in our data set for predicting diabetes. We'll do two things here to improve our performance. First, we'll balance our data set so that we have equal numbers of records with and without diabetes. Second, we'll see how we do when we use just the Glucose readings to calculate the distance to our 'neighbors' in KNN:

positive_cases <- nrow(train_df[train_df$Diabetes == 'Diabetes',])

balanced = rbind(train_df[sample(which(train_df$Diabetes == 'No diabetes'), positive_cases),], train_df[train_df$Diabetes == 'Diabetes',])

trnctrl_df <- trainControl(method = "cv", number = 10)
model_knn_df_simple <- train(Diabetes ~ Glucose, data = balanced, method = "knn", 
                       trControl = trnctrl_df, 
                       tuneLength = 10)

model_knn_df_simple

This simpler model seems to have better performance. With a K value of 15, we see an accuracy of 0.93 and a Kappa of 0.86. Both of these scores are higher than we had seen with the larger model and the Kappa of 0.86 indicates that we are predicting both classes better although still not perfectly.

Step 5: Evaluating KNN

Since we held out a testing set of 20% of our training data, we have an easy way to evaluate both of our models. We pass the model to predict and check the results against the actual test labels. We can use a confusion matrix to assess how well our model predicts the test data:

predict_knn <- predict(model_knn_df, test_df)
confusionMatrix(as.factor(predict_knn), as.factor(test_df$Diabetes), positive = "Diabetes")

You can read more about confusion matrices in this tutorial, "Create a confusion matrix with R".

As we suspected, our everything model does a good job of identifying patients without diabetes but a poor job of identifying patients with diabetes. As we can see in the confusion matrix, the model only accurately identifies 25% of the records labeled 'Diabetes'.

predict_knn_simple <- predict(model_knn_df_simple, test_df)
confusionMatrix(as.factor(predict_knn_simple), as.factor(test_df$Diabetes), positive = "Diabetes")

As we can see, our overall accuracy rate has fallen slightly, but our ability to predict the diabetes label is much much better. This model correctly predicts 10 of the 12 true 'diabetes' records. On the negative side though, the model mis-classifies 9 of the 66 'no diabetes' records as 'diabetes'. This is a downside of training our KNN on a balanced data set: when we introduce new data points from our test set, our model is a little too over-eager to label them 'diabetes' when they aren't.

Summary

In this tutorial, you learned how the KNN algorithm works for classification tasks and which hyperparameters can be tuned to improve its performance. We also learned how you can prepare data for use with KNN by scaling and centering a data set and the different techniques you can use to tune the algorithms. We ran KNN on a data set with two different settings and compared the performance using a confusion matrix.

Try watsonx for free

Build an AI strategy for your business on one collaborative AI and data platform called IBM watsonx, which brings together new generative AI capabilities, powered by foundation models, and traditional machine learning into a powerful platform spanning the AI lifecycle. With watsonx.ai, you can train, validate, tune, and deploy models with ease and build AI applications in a fraction of the time with a fraction of the data.

Try watsonx.ai, the next-generation studio for AI builders.

Next Steps

Explore more articles and tutorials about watsonx on IBM Developer.