Training

Before creating a predictive model, you should always examine the distribution of the output field to avoid problems.

If you are using a symbolic output field, check to see if the distribution of the field is heavily skewed in favor of one of the categories. If that is the case, you may encounter difficulties when generating a predictive model. For example, suppose you are trying to predict which customers are most likely to defect or “churn.” If only five percent of your customers leave in a given year, it is quite possible that the model using this data will classify every individual as a non-churner, simply because these individuals make up the larger group. So while the overall accuracy rate will be 95 percent, this result is not very useful if the goal is to predict who is likely to churn.

One solution to overcome this problem is to use the Balance Node to ensure that all categories have roughly the same number of cases. You can either reduce the number of records in the more frequent categories, or boost the number of records in categories with fewer cases. Usually it is better to reduce the number of records in the more frequent categories because boosting duplicate records magnifies problems and irregularities in the data (only a few cases can be weighted). However, when you are working with smaller data sets, data reducing may not be feasible and data boosting may be the only sensible solution.

You can automatically generate the Balance Node from the Distribution Node output. Figure 1 shows the distribution of a field called RISK. The proportion of bad risk but profitable customers (bad profit) greatly exceeds the number of bad risk but unprofitable customers (bad loss) and good risk customers. Consequently, the model is likely to do a better job of predicting bad risk but profitable customers (bad profit) than the other two groups:

Figure 1: Distribution of the RISK field

Since the proportions are skewed, you’ll need to balance the data. Boosting the data will duplicate both the bad loss and good risk customers to more than 2400 each, while reducing the data will drop the bad loss and bad profit customers to around 800 cases, which is the size of the good risk category. Neither method is ideal, but you should reduce the data to eliminate the magnification of errors. To do so:

Click Generate…Balance Node (reduce)

Close the Distribution plot window

A Balance Node will appear in the Stream Canvas. Add the Balance Node to the stream and attach a Distribution Node to it. Then, request a distribution of the RISK field. The results are shown in Figure 2:

Figure 2: Distribution of the RISK field after balancing the distribution

Notice that now the number of cases in each category are nearly equal. To determine whether balancing the data actually improved your predictions, create classification tables from two different C5.0 models. As shown in Figure 3, one classification table uses balanced data and the other uses unbalanced data.

Figure 3: Classification tables

In both tables, the rows represent the actual categories and the columns represent the predicted categories. As expected, the unbalanced model did extremely well predicting the bad profit category (92.9 percent), simply because it was so much larger than the other two categories. In contrast, the model only correctly predicted 36.8 percent of the bad loss category. However, after balancing the data, the accuracy rate for the bad loss category rose to nearly 70 percent, an increase of well over 30 percentage points! Unfortunately, as a result of reducing the number of cases, the balanced model did not do nearly as well in predicting bad profit customers.

While balancing the data definitely improved your ability to accurately predict who would be a bad risk and unprofitable customer, it is important to note that you still need to test the balanced model against unbalanced data to see if it the balanced model provides better predictions. If it does, then that should be your model of choice for predicting new cases in the future.