Detecting outliers
Given a variable, outliers are values that are very distant from other values of that variable. Outliers are quite common, and often caused by human or measurement errors. Outliers can strongly derail a model.
To demonstrate, let's look at two simple datasets and see how their mean is influenced by the presence of an outlier.
Consider the two datasets with few samples each: A = [1,2,3,4] and B = [1,2,3,4, 100]. The 5th value in the B dataset, 100, is obviously an outlier: mean(A) = 2.5, while mean(B) = 22. An outlier can have a large impact on a metric. Since most machine learning algorithms are based on distance or variance measurements, outliers can have a high impact on the performance of a model.
Multiple linear regression is sensitive to outlier effects, as shown in the following graph where adding a single outlier point derails the solid regression line into the dashed one:
Removing the samples associated with the outliers is the simplest solution.
Another solution can be to apply quantile binning to the predictor by splitting the values into N ordered intervals or bins, each approximately containing an equal number of samples. This will transform a numeric (continuous) predictor into a categorical one. For example, [1,2,3,4,5,6,7,8,9,10,11,100] split into three equally sized bins becomes [1,1,1,1,2,2,2,2,3,3,3,3]; the outlier value 100 has been included in the third bin and hidden.
The downside of quantile binning is that some granularity of information is lost in the process, which may degrade the performance of the model.
Quantile binning is available as a data transformation process in Amazon ML and is also used to quantify non-linearities in the original dataset.
In fact, Quantile Binning (QB) is applied by default by Amazon ML to all continuous variables that do not exhibit a straightforward linear relation to the outcome. In all our trials, and contrary to our prior assumptions, we have found that QB is a very efficient data transformation in the Amazon ML context.