Training

Within Clementine, several of the modeling techniques are based on traditional statistical theory, which functions best when the data are normal as opposed to skewed. These techniques include regression, logistic regression, and discriminant analysis. Even the accuracy of Neural Net models can be adversely affected when the predictor fields are severely skewed. Unfortunately, the assumption of normality is not usually true for real-world data.

However, you can easily modify a field's values so that the overall distribution is more normal. The Transform Node, located in the output palette, provides the capability to perform a visual assessment of the best transformation to use for a field. After selecting the preferred transformation, you can generate Derive or Filler nodes that perform the transformations. Then you can attach these nodes to the stream and use them for modeling.

This example describes how you apply the Transform node to a field, Local, which represents the number of minutes a telephone company customer spends on local calling in a month. This field is heavily skewed because very few customers spend a substantial amount of time on the phone making local calls.

Figure 1: Histogram of Local, a heavily skewed field. Click image to enlarge.

To apply the Transform node:

Figure 2: The Options tab includes five formulas to modify a field. Click image to enlarge.

Whenever a field has any values set at 0, you need to enter an offset value for the inverse and the two logs because they are undefined for values of 0. So, enter the value "1" as the offset:

5. Click “Select formulas”
6. Select all five check boxes
7. Enter “1” as the offset for “Inverse,” “LogN,” and “Log10”
8. Click “Execute

After applying the selected distributions, the transform output displays histograms for the original distribution and for the selected distributions. You'll notice that both the LogN and Log10 distributions appear to be fairly normal.

Figure 3: The Transform output for the Local field. Click image to enlarge.

To examine the LogN distribution in detail, double-click on the LogN histogram.

Clementine adds a normal curve based on the mean and the standard distribution of the transformed field:

Figure 4: A histogram of the natural log of the Local field. Click image to enlarge.

If you use this field when building the model, the predictions will be based on the transformed values, not the original ones. This can make the interpretation of the coefficients a little tricky. However, if predictive accuracy is your ultimate goal, then it may be a good strategy to use the transformations.