Use balancing to produce more relevant models and data results

To achieve reliable data results, start by balancing data correctly based on a specific business objective before training a predictive model. Once you determine the initial balancing of the data, be sure to regularly monitor the balance of the incoming data, because the original balance might shift over time. You might need to make adjustments to keep your data balance accurate.

This article describes how to evaluate different methods and levels of balancing to achieve different business objectives. It describes how to implement balancing using IBM SPSS Modeler.

Understanding the differences between balancing and weighting

"Balancing the data" in data mining is similar to the concept of weighting the data in traditional statistics, but there are a number of important differences.

Traditional statistics was primarily developed in order to extrapolate conclusions from a sample of a population to a complete population. For example, political pollsters can ask 1,024 people who they would vote for in a presidential election and then use these responses to predict the winner of a presidential election. In data mining, we often have the advantage of working with data that covers the entire population of interest. For example, data can contain all of the customers who bought a given product, all of the machines that performed a given task, all of the students in the district, and so on. So, in data mining, we can work with all the data, not just a representative subset.

In traditional statistics, weighting is used to ensure that the sample being used for the analysis has the same makeup as the population that is being predicted. For example, if the sample has a ratio of 100 males to 80 females, the weighting applied to the sample would be designed to adjust the ratio back to 49 males to 51 females as it is in the population. Weighting schemes for such scenarios can be extremely complex and take dozens of different dimensions into account, depending on the sample and the analysis to be performed. For a political poll such as the one used in the example above, one would need to calibrate the weights for gender, age, income, geography, political affiliation, home ownership, occupation, and many more factors. This is a highly complex and specialized task that requires you to know the true distribution in the population to complete. In data mining, we rarely have to make these types of corrections because we are most often working with the entire population.

If the data used in the data mining project is derived from a sample that is not a random sample from the entire population of interest, then you might need to consider making similar adjustments to those described for weighting. Balancing the data in a data mining context is more often done to distort the relative proportions in the data to better aid model building and to train models that are better capable of addressing the underlying business problem. Technically, the balancing is done in a way that is very similar (if not identical) to weighting, but the objective and reason for doing so are entirely different, as is the way that the factors for balancing are selected.

Learning when to use balancing

Balancing is most often used to enable the data scientist to train a model that will better address the business problem that is the objective of the project – note that this objective might be different from building a more accurate model from a purely technical perspective. There are many examples of these situations, many of which include data sets where the distribution of the target variable is skewed or has at least one category that is underrepresented relative to the other categories.

Example 1: Insurance fraud

While fraudulent insurance claims are considered a big issue for property insurance companies, the number of identified fraudulent claims is very small compared to the total number of claims handled by any individual insurance company. This poses a challenging situation when tasked with executing a data mining project to identify potentially fraudulent claims for further investigation.

Consider the case of a fictitious insurance company that handles 800,000 auto insurance claims each year. As many as 20% of all auto insurance claims might contain an element of fraud, but it is only a very small percentage of claims where fraud is actually proven—often as few as 2% of the claims (16,000 claims). The other claims suspected to be fraudulent are either withdrawn by the claimant or settled at a lower amount than originally claimed.

If you were to use the data set as-is to train a predictive model to identify the fraudulent claims, it is very likely that one would end up with a model that is 98% accurate. This would be achieved merely by predicting all claims to be non-fraudulent and while this technically is a very accurate model, it is a model that does not address the underlying business problem of identifying potentially fraudulent claims. There might be some algorithms such as the "Decision List" that are better equipped to handle this type of business problem, but it will take you only so far.

Balancing the data might be the solution. Balancing, in this case, could be used to increase the proportion of fraudulent claims by duplicating the 16,000 known fraudulent claims in the data set to a level where the algorithms are able to create a model where a suitable number of claims are predicted as fraudulent.

Balancing might also be used to put additional emphasis on a certain subset of data, such as more recent observations if there is a suspicion that patterns might be changing slowly over time. In this case, balancing might be an alternative to only using more recent observations.

Deploying balancing in IBM SPSS Modeler

Depending on the exact use case, there are several methods to deploy balancing in IBM SPSS Modeler. This article describes a subset that includes the most commonly used methods, but they can all be expanded upon or combined as needed.

The simplest way to balance a data set is to use the built-in balance node. In the majority of cases, this method is sufficient.

The balance node works by duplicating or discarding records in the data set based on the balancing directives specified in the node. A factor greater than 1 results in records being duplicated in the data set, while a factor lower than 1 results in records being discarded. For any non-integer factors, the records for duplication and discarding are selected at random. There is no option to set the random seed, which means that the results may be slightly different from one execution to the next.

The node does have an option to Only balance training data, which is selected by default and should remain so unless there is a compelling reason to balance the testing and validation data.

Consider a data set with six observations that only include the gender and the number of unique web pages viewed by each individual. The data set contains 4 males and 2 females. We will examine the effects of the balance node as we create a data set with an even distribution between the two genders.

Boosting using the balance node

When the factors used in balancing causes the observations from the less frequent category to be duplicated, it's called boosting. Example 2 shows boosting.

Example 2: Boosting

Figure 1. The balancing example stream with boosting

The original data set has 4 males and 2 females.

The data has only 1 female for every 2 males, but we would like the distribution of males and females to be even. Configure the balance node to use a factor of 2.0 when GENDER = "Female".

Figure 2. The balance node with the directive to boost the number of females

Any records for which no condition holds are always passed through as-is.

In the balanced data set, records with IDs E and F each appear twice in the data set.

The data set now has an even distribution of males and females with 4 males and 4 females.

Reducing using the balance node

Reduction is an alternative to boosting, which is the term used when the factors used in the balancing causes observations from the more frequent category to be discarded.

Example 3: Reduction

The downloadable stream used in this example is the same stream used in Example 2.

Figure 3. The simple balancing example stream with reduction

The original data set again has 4 males and 2 females.

The data still has 2 males for every 1 female. We would still like the distribution of males and females to be even. This time the balance node is configured to use a factor of 0.5 when GENDER = "Male".

Figure 4. The balance node with the directive to reduce the number of males

In the balanced data set, the records with IDs B and D have been discarded from the data set.

The data set now has an even distribution with 2 males and 2 females.

Because the factor is a non-integer, the records that are discarded will change with each execution.

Combining boosting and reduction

You can also combine the two methods to create a balanced data set that has 3 males and 3 females. Set the factors to discard 1 in 4 males and to duplicate 1 in 2 females.

Generating balance nodes from graphs

Simple balance nodes that create a uniform distribution between two or more categories can be generated from a distribution graph for categorical fields or from the histogram graph for a continuous field. This is done by generating a graph that shows the true distribution of the field in question from the data set and then clicking Generate on the toolbar.

Figure 5. Create a balance node from a graph

The downloadable stream from Example 2 also includes a distribution chart and a histogram that you can use to generate balancing nodes this way.

There are several different ways to generate nodes from graphs in IBM SPSS Modeler.

These options are available only in graphs created using the distribution and histogram nodes and not in graphs created using the graph board node.

Handling repeatable assignments

Remember that when using non-integer factors, this duplicating or discarding of records is random and is done each time the data passes through the node. This means that the resulting dataset will be different every time a node is executed.

Some of the nodes that include an element of random assignment in IBM SPSS Modeler, such as the sample node and the partition node, have an option to enforce repeatable partition assignment. There is no such option for the balance node. If the balancing needs to be done in the exact same manner each time, use a cache or an export.

Using a cache

Enabling a cache on the balance node is the simplest option. This ensures that the balancing, along with the rest of the data set, is stored in a temporary file or database table that will be used for the remainder of the session once it has been filled. A small icon, which is added to the top-right corner of the node, turns green when the cache is filled.

The assignment, however, will not persist across sessions, and the records will be reassigned each time the stream is closed and reopened or if the cache is flushed for some other reason. For data sets with a large number of fields, this may also result in an unnecessarily large temporary file, because it essentially creates a copy of the complete data set. You can find more details in the Caching options for nodes article.

Using an export

The other repeatable-assignment option is to create the balanced dataset once and then to export it to a file or database table along with one or more fields that uniquely identify each record. The exported balancing can then be merged with the original source data for a balancing that will persist across sessions.

Consider, however, that if a slight change in balancing causes significant changes in the estimated model, then this should be interpreted as an indication that the model is unstable and the issue might be something more substantial than the data set being unbalanced.

Dynamic balancing

The balance node allows the data scientist to input the conditions and the factors used for balancing, but the factors often depend on the balance in the data set being analyzed. Therefore, in an environment where the data set changes and the models are retrained on a regular basis, the factors used for balancing will also change. If the models are retrained manually, or if the balance in the underlying data does not change significantly, then it might not be much additional work for the data scientist to change the factors after examining the data and before retraining the models. If, however, the model building and retraining is done in an automated fashion where the same stream is used to build models for different customer segments or offers, the balancing will need to be dynamic to adjust to data sets with different proportions.

A stream script can be used to manipulate the balance node and reset the factors and even the conditions of the node.

Example 4: Dynamic balancing using a script

Download the stream and the stream script used in this example.

Example 4 shows how to dynamically create the factors and conditions required to balance the data set used in earlier examples so that there is a uniform distribution of males and females by reducing the number of males in the data set.

Figure 6. The stream used for dynamic balancing by a stream script

The longest branch of the stream computes the factors and conditions. The count_by_gender aggregate node merely creates a data set with the number of males and females respectively.

The aggregate node named Target Count determines the method of the balancing (in this example, reduction) by taking the minimum value of the gender count. This is the targeted number of cases for each category, and it can easily be changed from Min (reducing) to Max (boosting) or Mean (a combination of reducing and boosting). The node can also be replaced by one or more nodes to create the desired target number in a different way.

The merge node merges two aggregates into a data set with one record for each gender.

The factor node is a derive node that derives the factor to be used for each gender in the balancing using the formula target_count/count_by_gender. The formula computes the ratio between the targeted number of cases for each category and the actual number of cases for the category.

The condition node derives the condition to be used for each gender in the balancing node using the formula "GENDER = \"" >< GENDER >< "\""

The table node named Directives displays the factors and conditions in a table and makes the values in the table available to the script.

A select node could be added before the directives node to discard any records where the factor is equal to 1, because these will have no effect in the balance node.

The following code shows the stream script that executes the directives table node to display the computed directives in a table.

 #Execute the "Directives" table node to compute the directives 
 directivesResults = [] 
 diagram.findByType('tablenode', "Directives:).run(directivesResults) 
 directivesRowSet = directivesResults[0].getRowSet() 
 directivesRowCount = directivesRowSet.getRowCount() 

That code also captures the total number of directives as the number of rows in the table.

The directivesAll = [ ] variable is declared to hold each directive in a list as they are captured from the table output.

As the script iterates through each row of the table output, the following code assembles the factor and the condition into a directive, which is stored as a list with two elements.

 #Iterate through the table output to read each directive 
 for ii in range(directivesRowCount): 
   directiveCurrent = [] 
   #Get the factor 
   directiveFactor = directivesRowSet.getValueAt(ii, 0) 
   directiveCurrent.append(directiveFactor) 
   #Get the condition 
   directiveCondition = directivesRowSet.getValueAt(ii, 1) 
   directiveCurrent.append(directiveCondition) 
   #Append the directive into a list of lists 
   directivesAll.append(directiveCurrent) 

Once the directives are assembled using the following code, they are used as the directives in the balance node.

 #Set the directives in the Balance node 
 balanceNode = diagram.findByType('balancenode', "Balance") 
 balanceNode.setPropertyValue('directives', directivesAll) 

Whatever the settings in the balance node were before executing the script, the settings will reflect the content of the directives table node output after executing the script, as shown below in Figure 7.

Figure 7. The scripted balance node directives

The data set now has a uniform distribution of males and females, and it will always have a uniform distribution even when the underlying distribution in the source data changes.

This method works regardless of whether you prefer to boost the less frequent categories or to reduce the more frequent categories, because this depends on your calculation of the Target_count field.

You can also replicate the output of a balance node with dynamically allocated factors using only nodes on the canvas without having to resort to scripting. You might prefer this option if the stream script is already complex enough or if there is a need for a more complex balancing across multiple attributes.

Dynamic balancing is best used when reducing the number of cases in the more frequent categories, because this enables you to reach the same results as illustrated in Example 4 using many of the same basic nodes but without having to execute the script.

Example 5: Dynamic balancing using complex sampling

Download the stream used in this example.

The objective for Example 5 is the same as in Example 4: to create a data set with a uniform distribution of males and females.

Figure 8. The stream used for dynamic balancing by sampling

The upper branch of the stream is almost identical to the one used in Example 4 except that the derive nodes named Factor and Condition are no longer needed and have been left out.

The second merge node merges the source data with the directives that were computed in the upper branch, including both the target count for each category and the factor that will be used for sampling the more frequent categories.

The sample node named Complex is the node that will do the balancing in place of the balance node.

Set the sampling method to Complex, as shown below in Figure 9. The GENDER field is selected as the field by which to stratify the sampling.

Figure 9. The settings of the sample node

The computed field Factor is then selected as the variable for the random sampling by proportions.

Again, the resulting data set has a uniform distribution of males and females.

One advantage of the dynamic balancing method is that the sample node includes the option to set the random seed to a constant value and thus have the random sampling return the exact same records with each execution. The regular balance node does not have this function, as described in Handling repeatable assignments.

Selecting the best balancing factors

Determining whether balancing is merited or whether it might be beneficial for the modeling is usually dependent on the business objectives of the project. Start by examining the confusion matrix for the model to see whether the model fulfills the business objective. In the situation outlined in Example 1, the confusion matrix might reveal that the model based on an unbalanced data set is able to correctly identify almost all of the non-fraudulent insurance claims, but none (or very few) of the fraudulent claims. In other words, the model has:

• A large number of false negatives
• A high type II error rate
• A specificity close to 1.0

If the objective is to avoid the investigation of a valid claim, then this might be considered a good model. But if the objective is to detect and deter fraud (which is more often the case), then this model does not meet the objective, and balancing the data might be a way to improve the model.

The choice to use boosting, reduction, or a combination depends on the relative proportions of the data set and the business objectives. The best way to discover the right levels of balancing is to build models with different methods and factors and to evaluate these against each other to determine which ones best meet the business objectives.

There are some rules-of-thumb and some guiding principles:

• When balancing a data set, you do not need to arrive at a uniform distribution.
• Avoid over-sampling the more frequent categories, because this can cause you to discard enough data from the less frequent categories that important correlations or patterns are sampled out of the data set entirely.
• Avoid replicating individual records too heavily, because this might cause random and rare occurrences to be recognized as meaningful patterns by the modeling algorithms.

In the insurance fraud example, you would not want to create a uniform distribution by replicating the fraudulent claims. Doing so would probably have the modeling algorithms recognize the exact combination of properties from each of the original fraudulent claims as a pattern that indicates fraud. You would also not want to create a uniform distribution by reducing the non-fraudulent claims. This would discard a large portion of the relevant data that would have patterns that are characteristic of non‑fraudulent claims and would make it more difficult to identify these. The solution is somewhere in‑between these two extremes: replicate the fraudulent claims and reduce the non-fraudulent claims to create a balance that is reasonable and that allows the model to meet the business objective.

The most important defence against over-balancing the data set is to ensure that the data set is partitioned into training and testing (and possibly validation) partitions where the testing and validation partitions are not balanced, but retains the original proportions. The model performance is then evaluated based on the testing or validation partition to determine the true effectiveness of the model.

Example 6: Credit card offer

In Example 6, a credit card company wants to present different offers and marketing campaigns to existing cardholders when the customers contact the credit card company. One of these offers is for the cardholder to sign up for a supplemental credit card for the same account to give to a spouse or other family member.

From 5,904 calls into the call center, this offer was presented to just fewer than 2,000 cardholders, and the acceptance rate for the offer was 13.8%.

First, train a model based on the true, unbalanced distribution of the data set.

Figure 10. The basic stream used for training the credit card offer models

In the figure, Credit cards > Reclassify > Offer made > Partition > Balance > Type and then branches off to either Training, to Offer response, or to Offer response that goes through additional testing and analysis to get to the Confusion matrix.

The model correctly classifies 87.5% of the respondents, but the confusion matrix predicts that a full 94.9% of the respondents will reject the offer. The model also identifies only 23.9% of the card holders who accepted the offer. This gives a false negative rate of 76.1%.

Then train a model where a uniform distribution of accepted and rejected offers is achieved using boosting. This is done by adding a balance node with a factor of 6.2409 for records where the offer was accepted. The data set includes 274 respondents, who accepted the offer, and 1,710 respondents, who rejected the offer. The factor to create a uniform distribution between the two groups can be found by dividing the number of respondents in the larger group (1,710) by the number of respondents in the smaller group (274). This results in the factor of 1,710/274=6.2409.

The resulting model correctly classifies 95.1% of the responses in the training partition, but only 81.4% of the responses in the testing partition, which indicates that the model has been over-trained; the level of boosting is too high and needs to be adjusted.

On the other hand, you did manage to reduce the false negative rate to 43.7%, which is a significant improvement over the original model.

Building models with different levels of balancing will yield slightly different results. The following table shows the same model trained on the same data set but uses different levels of balancing. The false negative and false positive rates are based on the testing partition.

The results from Table 3 are charted in Example 6 to better illustrate the trade-off between the two error rates when using different levels of balancing. The dotted line shows a highly subjective border of where one might expect other levels of balancing to fall in the same space.

Figure 11. Plot of the relationship between the false negative and false positive rates of models based on data sets with different levels of balancing

There are many ways to evaluate these models, but for the purpose of balancing, it comes back to the business case to decide what the right balance is. If you are offering a high revenue product through a low cost channel, then you might be willing to accept a higher false positive rate than if you are selling a low revenue product through a high cost channel.

The offer in Example 6 is a supplementary credit card for an existing account. Adding a card to the account means that there is one additional person using the credit card account and thus generating revenue for the credit card provider each time the card is swiped. The expected revenue for each account depends on a number of different factors and most likely requires that we build a model to estimate the expected revenue per year and the expected lifetime. For this example, we will assume that expected revenue by adding a supplemental credit card is $10.

The cost of presenting this offer is also made up of several components, including the time spent by the call center agent and the revenue lost by presenting this offer to the account holder over another one that might have been accepted, but for simplicity, we will set the cost at $2. This would mean that the company can accept a false positive rate as high as 80% and still break even.

Determining the correct balancing directives to use in any use case is difficult and inevitably relies on some trial and error to determine the balance that best suits the business objective. The key is to have a good understanding of the business objective and how the predictive model is going to be implemented and used.

Conclusion

This article described how to evaluate different methods and levels of balancing and how balancing can be deployed in IBM SPSS Modeler. Using the provided test data sets, you can use balance nodes to boost or reduce the model as needed to get the desired results. By implementing dynamic balancing, balance nodes can change as analyzed data changes. For example, when a predictive model is used to select customers that are offered a supplemental credit card, the ratio of customers who accept the offer will increase, which then shifts the balance in the data. Data is often dynamic, and the balancing factors may also need to be adjusted manually on a regular basis. Use IBM SPSS Modeler to determine what those balancing factors should be for your data based on your own business objectives.