Disclaimer: I am not a medical, radiology, or epidemiology professional. This article was an experiment from an engineering and data scientist perspective, and should be regarded as such.

To apply deep learning for COVID-19, you need a good data set, one with lots of samples, edge cases, metadata, and different images. You want your model to generalize to the data so that it can make accurate predictions on new, unseen data. Unfortunately, not much data is available. There are posts on sites such as LinkedIn and Medium that claim >90%, or in some cases 100%, accuracy on detecting COVID-19 cases. However, these posts usually contain mistakes, which are not always acknowledged. This article tries to mitigate some of the usual mistakes by exploring solutions to the following problems:

Testing your model's performance with the same data you used to train it with. The first rule of machine learning is to never test your model's performance with the same data you used to train it with. Not using a testing data set, but instead testing and measuring the accuracy of your model on the training data set does not give an accurate representation of how well the model generalizes to new, unseen data. You might not believe this is an issue, but it is a typical fallacy for people new to machine learning and data science.
Not using computer vision techniques to achieve better generalization. Augmentation is an absolute necessity, especially in the case where there are very few samples for the model to learn from.
Not thinking about what the model learns, that is, will your model actually learn the pattern of what COVID-19 looks like on an X-Ray image, or is it likely that there is another noisy pattern in your data set that it learns instead? One amusing story of early machine learning is called Detecting Tanks. In this story, photos of camouflaged tanks had been taken on cloudy days, while photos of plain forest had been taken on sunny days.
Not using the correct metrics. If you use the accuracy metric on a heavily imbalanced data set, then your model might look like it's performing well in the general case, even though it's performing poorly on COVID-19 cases. 95% accuracy might not be that impressive if your accuracy on COVID-19 cases is 30%.

All of the work for this article is available on GitHub.

Acquiring data

The following list is different data sources that I accumulated over the period of the coronavirus spreading. Currently, this is the best I can get from the internet because the data being collected in many countries and hospitals is classified information. Even if it was not classified information, you still need consent from each patient.

GitHub: COVID-19 Chest X-Ray and CT images. This data set is continually updated as more cases appear.
Kaggle 1: Chest X-Ray images with both positive and negative cases of pneumonia.
Kaggle 2: Chest X-Ray images with positive and negative cases of pneumonia.

Problem with mixed data sources

All of the data available on the internet has not been subjected to the same preprocessing, and all of the images are, as explained below, clearly different in the amount of black bars. Most of the positive COVID-19 data has the entire X-ray take up most of the screen, with little to no black bars on the sides (except for a few). However, the data set with negative COVID-19 cases has mostly black bars on the side of each image.

Images comparison

This becomes a problem because the model might later learn that it just needs to look at the black bars on the side to know whether a new sample is a positive or negative case of COVID-19.

After manual inspection of the data set, it becomes apparent that almost all of the negative cases have these black bars, while approximately 10-20% of the positive cases have black bars. Wouldn't it be quite a coincidence if the accuracy of predicting positive or negative later turns out to be around the 80-90% mark?

Images markers cropped

In the previous image, the left side shows an image with a negative COVID-19 case (black side and top bars). The image on the right shows the same image, but is cropped toward the center and scaled up to match sizes.

Preprocessing - improving the data

The data problem from earlier is clearly one we need to solve. Can we somehow detect when there are black top, black bottom, or black side bars? And, can we somehow automatically correct this?

The following steps are the key to solving this:

A simple metric: Is there a number x or more black pixels in a given row or column?
Yes? Remove that row or column. Repeat for all rows and columns.
Done checking for black pixels? Then, crop the image.
Next image and repeat.

Ok, not that fast. Here's a new problem: We need to keep the same aspect ratio. And, we can't upscale or downscale if the aspect ratio is not the same. This is due to the existing architectures needing an input size of 224x224, which is an aspect ratio of 1x1.

In other words, can we check whether we are removing as many columns as we are removing rows? Can we overcompensate by removing specific rows if there are too many columns with black pixels but little to no rows of black pixels?

Removing the black bars and fixing aspect ratio

The following image demonstrates the input and output of removing the black bars, with one example where it was particularly successful. This is quite powerful, but we are still not adhering to the aspect ratio of 1x1.

Images black bars removed

The most effective and easiest solution that I found to fixing the aspect ratio was calculating the difference between the number of rows removed versus the number of columns removed. Divide the difference by two, and remove rows from top and bottom, or remove columns from left and right.

Images adjusted aspect ratio

The last image is saved as a 224x224 image because we can scale it down now that the aspect ratio is 1x1. This saves space and some time when loading the image again in another script.

Running the script

It takes approximately 14 seconds per image to find the rows and columns for removal. I use a naive approach, which could probably be done more efficiently, nevertheless, it works for this experiment. I had 8851 images in the Kaggle 1 data set, which turned out to be 39 hours of preprocessing. All of the data is released and available in the GitHub repository for this article.

I ran the code on both the Kaggle 1 and GitHub data sets, and the result is that I have all of these images saved in a 224x224 image format. The following code is only for running it on the Kaggle 1 data set, but you can substitute the first part of the code to load any data set of images.

import pandas as pd
import matplotlib.pyplot as plt
import cv2, time, os
import numpy as np

df = pd.read_csv('kaggle-pneumonia-jpg/stage_2_detailed_class_info.csv')
image_folder = 'kaggle-pneumonia-jpg/stage_2_train_images_jpg/'

df = df.loc[df['class'] == 'Normal']

The following code shows how I loaded the COVID-19 data set. Note that the rest of the code is the same. This is just loading a different data set.

columns = ['finding', 'modality', 'filename', 'view']

# only use COVID-19 samples done with X-ray and PA view
df = pd.read_csv('metadata.csv', usecols=columns)
df = df.loc[df['finding'] == 'COVID-19']
df = df.loc[df['modality'] == 'X-ray']
df = df.loc[df['view'] == 'PA']

After loading the dataframe, I load the images, labels, and file names into arrays.

images = []
labels = []
filenames = []

for index, row in df.iterrows():
    image = cv2.imread(image_folder + row.patientId + '.jpg')
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    images.append(image)

    label = row['class']
    labels.append(label)

    filename = row.patientId
    filenames.append(filename)

images = np.array(images)

Next, I have a get_black_pixel_indices() function that loops through rows and columns of pixels. You can imagine an image as a table that has rows and columns of pixels that has pixel values.

I count the number of black pixels, defined by the number of pixel values that are less than 40. A pixel value consists of one value for each color channel, that is, red, green, or blue (RGB). I could have an array [39, 67, 45], and the average pixel value would be the average of those numbers, which is 50.33.

When I am finished counting the number of black pixels for a given row and column, I check to see whether there are more black pixels than non-black pixels. If that is the case, I choose to remove that row or column.

def get_black_pixel_indices(img):
    rows_remove = []
    cols_remove = []

    for j, counter in enumerate(range(img.shape[0])):
        row = img[counter]
        # shift column dim to first dim, row dim to second dim
        # acts like a 90 degree counterclock-wise rotation of image
        col = img.transpose(1, 0, 2)[counter]

        row_black_pixel_count = 0
        col_black_pixel_count = 0
        for selector in range(img.shape[1]):
            row_pixel = row[selector]
            col_pixel = col[selector]

            if np.average(row_pixel) < 40:
                row_black_pixel_count += 1
            if np.average(col_pixel) < 40:
                col_black_pixel_count += 1

        if row_black_pixel_count > len(row)/2:
            rows_remove.append(j)
        if col_black_pixel_count > len(col)/2:
            cols_remove.append(j)

    return rows_remove, cols_remove

Before I remove the rows and columns with a higher black pixel count than non black pixel count, I must check for rows and columns that if removed would obstruct the image. The simplest way that I found was that no rows and columns should be removed between column and row 200 and 800. After filtering these obstructing indices, I remove the rows and columns.

def remove_obstructing_indices(rows_remove, cols_remove):
    for i, value in enumerate(rows_remove):
        if 200 <= value <= 800:
            del rows_remove[i]
    for i, value in enumerate(cols_remove):
        if 200 <= value <= 800:
            del cols_remove[i]

    return rows_remove, cols_remove

def remove_black_pixels(img, rows_remove, cols_remove):
    img = np.delete(img, rows_remove, axis=0)
    img = np.delete(img, cols_remove, axis=1)

    return img

The last thing for preprocessing is adjusting the aspect ratio. I know that in most cases a human torso is more tall than wide, so I choose to arbitrarily cut the image from the top and bottom. There are also cases where the torso is wider, but those were usually rare when inspecting the data set.

I start by calculating the column-to-row difference and row-to-column difference, and based on that, I find how many rows or columns that I need to remove to fix the aspect ratio. I found that I can divide the difference by two, then remove half of the difference from either the top and bottom, or left and right. After that, I might have one leftover row or column if the difference is an odd number, so I compensate by either removing a row from the bottom or a column from the right.

def adjust_aspect_ratio(img, rows_remove, cols_remove):
    col_row_diff = len(cols_remove) - len(rows_remove)
    row_col_diff = len(rows_remove) - len(cols_remove)

    if col_row_diff > 0:
        slice_size = int(col_row_diff/2)

        img = img[:-slice_size]
        img = img[slice_size:]

        if img.shape[0] != img.shape[1]:
            img = img[:-1]

    elif row_col_diff > 0:
        slice_size = int(row_col_diff/2)

        img = img[:,:-slice_size,:]
        img = img[:,slice_size:,:]

        if img.shape[0] != img.shape[1]:
            img = img[:,:-1,:]

    if img.shape[0] == img.shape[1]:
        return img, True
    else:
        return img, False

Finally, I run the loop that calls the four functions previously described. This results in images that have removed black bars from the left, right, top, and bottom. I downscale that image to a 224x224 image, then I save the image along with the label and file name in a .csv file that cumulates all of the paths and file names of all the preprocessed images.

save_folder = 'normal_dataset/'

start = time.time()
for i, img in enumerate(images):
    rows_remove, cols_remove = get_black_pixel_indices(img)
    rows_remove, cols_remove = remove_obstructing_indices(rows_remove, cols_remove)

    new_img = remove_black_pixels(img, rows_remove, cols_remove)
    adj_img, shape_match = adjust_aspect_ratio(new_img, rows_remove, cols_remove)

    if shape_match:
        resized_image = cv2.resize(adj_img, (224, 224))

        label = labels[i]
        name = filenames[i] + '.jpg'

        cv2.imwrite(save_folder + name, resized_image)
        new_df = pd.DataFrame({'filename': [name], 'finding': [label]})

        if i == 0:
            new_df.to_csv('normal_xray_dataset.csv')
        else:
            new_df.to_csv('normal_xray_dataset.csv', mode='a', header=False)

    print('image number {0}, time spent {1:2f}s'.format(i+1, time.time() - start))

For a final overview of another 40 images of negative and positive cases, I can clearly see that the preprocessing approach helped.

Image comparison 2

Augmentation of COVID-19 samples

To make the most of what I have (99 images), I need to use augmentation of the training data. Augmentation is a manipulation of the image, which will look new to CNN. Some common examples of augmentation are rotations, shears, and flips of images.

Keep in mind that I use augmentation to generalize the model better and prevent overfitting. Essentially, each time I use one form of augmentation, I add another similar, but slightly different, image to the training data set. Note that this is also a demonstration of how you could augment the images, and not the augmentation we will use later.