Bias is a prejudice in favor or against a person, group, or thing that is considered to be unfair. But as machine learning becomes more of an integral part of our lives, the question becomes will it include bias? In this article, I’ll dig into this question, its impact, and look at ways of eliminating bias from machine learning models.

Machine learning has shown great promise in powering self-driving cars, accurately recognizing cancer in radiographs, and predicting our interests based upon past behavior (to name just a few). But with the benefits from machine learning, there are also challenges. One key challenge is the presence of bias in the classifications and predictions of machine learning. These biases are not benign. They have consequences based upon the decisions resulting from a machine learning model. Therefore, it’s important to understand how bias is introduced into machine learning models, how to test for it, and how then how to remove it.

The question of bias

One example of bias in machine learning comes from a tool used to assess the sentencing and parole of convicted criminals (COMPAS). Because of overcrowding in many prisons, assessments are sought to identify prisoners who have a low likelihood of re-offending. These prisoners are then scrutinized for potential release as a way to make room for incoming criminals. A large set of questions about the prisoner defines a risk score, which includes questions like whether one of the prisoner’s parents were ever in prison, or if friends or acquaintances have been in prison (but does not include the question of race).

This tool has been found to successfully predict recidivism for convicted criminals, but when it has been wrong in its prediction, it brought race into the equation. Northpointe, the company that developed COMPAS, has subsequently presented data that supports its algorithm’s findings, so the jury is still out on this, but it indicates that whether bias exists or not. These approaches will be challenged and require subsequent data to demonstrate fairness.

Machine learning is viewed as a critical tool in human resources in a variety of use cases — from training recommendations to recruiting and other tactical activities. In 2014, Amazon began developing a system to screen job applicants as a way to automate the process of identifying key applicants to pursue based on the text on their resumes. But what Amazon found was that the algorithm appeared to favor men over women in engineering roles. Amazon abandoned the system after discovering that it wasn’t fair after multiple attempts to instill fairness into the algorithm.

The Google Photos application can be used to categorize pictures by identifying objects within them. But some racial bias was found when it was used. Amazon’s Rekognition, a commercial system for facial analysis, was also discovered to include gender and racial bias.

A final example was Microsoft’s Tay Twitter bot. Tay was a conversational AI (chatbot) that learned through engaging with people on Twitter. The algorithm mined public data to build a model for conversation, but also learned from interactions over time on Twitter. Unfortunately, not all of the interactions that Tay experienced were positive, and Tay learned the prejudices of modern society, indicating that even with machine models, you get out what you put into it.

The impact of bias

No matter what the bias is, the recommendations of machine learning algorithms have a real impact on individuals and groups. Machine learning models that include bias can help to perpetuate bias in a way that’s self-fulfilling. Therefore, it’s important to detect bias in these models and eliminate it as much as possible.

The source of bias

A simple answer to the presence of bias is that it’s a result of the data, but the origins are more subtle and related to the source of data, the contents of the data (does it include elements that the model should be ignorant of?), and the training of the model itself (for example, how do we define good and bad in the context of a model’s classification).

If a machine learning algorithm was trained solely on video of daytime driving, it would have tragic results if the model were permitted to drive at night. This is different from human bias, but demonstrates the issue of lacking a representative data set for the problem at hand.

Bias can also show up where we don’t expect it. In the case of Amazon’s recruitment tool, the model penalized wording used by some candidates and rewarded words by others. In this case, the penalized words were gendered words used commonly by women (who were also underrepresented in the data set) So Amazon’s tool was trained with 10 years of resumes coming primarily from men and developed a bias toward male resumes based upon the language that was used within them.

Even humans can unintentionally amplify bias in machine learning models. Bias in humans can be unconscious (also called implicit bias), which indicates humans can introduce bias without even knowing they are doing so.

Let’s explore how we can detect bias in machine learning models and how it can be eliminated.

Types of bias

Bias in machine learning data sets and models is such a problem that you’ll find tools from many of the leaders in machine learning development.

Detecting bias starts with the data set. A data set might not represent the problem space (such as training an autonomous vehicle with only daytime data). A data set can also incorporate data that might not be valid to consider (for example, a person’s race or gender). These are called sample bias and prejudicial bias, respectively.

Because data is commonly cleansed before being used in training or testing a machine learning model, there’s also exclusion bias. This occurs when we remove features that we think are not relevant. Measurement bias occurs when the data collected for training differs from the data collected during production. This can occur when your data set is collected with a specific type of camera, but your production data comes from a camera with different characteristics.

Finally, there’s algorithmic bias, which stems not from the data that a model was trained from but from the machine learning model itself. This includes how the model was developed or how the model was trained that results in unfair outcomes.

Dealing with bias

Now that I’ve given you examples of bias and the sources of bias, let’s explore how you can detect and prevent bias in your machine learning models. We’ll explore solutions from Google, Microsoft, IBM, and other open source solutions.

Google’s What-If tool

Google’s What-If Tool (WIT) is an interactive tool that allows a user to visually investigate machine learning models. WIT is now part of the open source TensorBoard web application and provides a way to analyze data sets in addition to trained TensorFlow models. One example of WIT is the ability to manually edit examples from a data set and see the effect of those changes through the associated model. It can also generate partial dependence plots to illustrate how predictions change when a feature is changed. WIT can apply various fairness criteria to analyze the performance of the model (optimizing for group unawareness or equal opportunity). WIT is straightforward to use and includes a number of demonstrations to get users up to speed quickly.

IBM AI Fairness 360

One of the most comprehensive toolkits for detecting and removing bias from machine learning models is the AI Fairness 360 from IBM. AI Fairness 360 is an open source toolkit and includes more than 70 fairness metrics and 10 bias mitigation algorithms that can help you detect bias and remove it. Bias mitigation algorithms include optimized preprocessing, re-weighting, prejudice remover regularizer, and others. Metrics include Euclidean and Manhattan distance, statistical parity difference, and many others.

AI Fairness 360 includes a number of tutorials and a wealth of documentation. You can also use an online interactive demonstration over three data sets (including the COMPAS recidivism data set) that allows you to explore bias metrics, then apply a bias mitigation algorithm and view the results as compared to the original model. The toolkit is designed to be open to permit researchers to add their own fairness metrics and migration algorithms.

IBM researchers have also proposed a bias rating system for machine learning models in “Towards Composable Bias Rating of AI Services.” This envisions a third-party rating system for the validation of machine learning models for bias.

Microsoft bias discovery in word embeddings

Similar to Microsoft’s experience learning in the wild, data sets can incorporate bias. A recent paper from the University of Maryland and Microsoft Research titled “What are the biases in my word embedding?” developed an approach using crowdsourcing to identify bias within word encodings (natural language). Word embedding represents words by feature vectors in a highly dimensional space. These feature vectors then support vector arithmetic operations. This permits analogy puzzles, such as “man is to king as woman is to x.” Computing x results in queen, which is a reasonable answer. But looking at other analogies identifies some potential areas of bias. For example, “man is to computer-programmer as woman is to homemaker” reflects a gender bias. Equally problematic is “father is to doctor as mother is to nurse.”

Microsoft demonstrated the ability to discover bias within word encodings in an automated way using association tests. This was inspired by the Implicit Association Test (IAT), which is widely used to measure human bias. The results of this discovery were then validated using crowdsourcing to confirm the bias.

Through this process, users of word embeddings benefit from a reduction in bias of this data set.

Other frameworks

Bias has become one of the most studied aspects of machine learning in the past few years, and other frameworks have appeared to detect and mitigate bias in models.

Local Interpretable Model-Agnostic Explanations (Lime) can be used to understand why a model provides a particular prediction. Lime works on any model and provides a human-understandable interpretation for a given prediction.

FairML is a toolbox that can be used to audit predictive models by quantifying the relative significance of the model’s inputs. This relative significance can then be used to assess the fairness of the model. FairML also works on any black-box model.

Human in the loop

In many cases, machine learning models are black box. You can feed them inputs and look at their outputs, but how they map those inputs to outputs is concealed within the trained model. Explainable models can help to bring to light how machine learning models come to their conclusions, but until these are commonplace, an alternative is human in the loop.

Human in the loop is hybrid model that couples traditional machine learning with humans monitoring the results of the machine learning model. This allows them to observe when algorithmic or other data set biases come into play. Recall that Microsoft used crowdsourcing to validate their word embedding bias discoveries, which indicates that it is a useful hybrid model to employ.

Going further

Fairness is a double-edged sword, and there is no consensus over a mathematical definition of fairness. We could employ the four-fifths rule or other measures of parity, but flaws are quick to appear.

Until we can build fully transparent and explainable models, we’ll need to rely on toolkits to measure and mitigate bias in our machine learning models. Thankfully, these toolkits are feature rich with a plethora of fairness metrics and algorithms for bias mitigation.