  Because most machine learning (ML) algorithms are designed for numerical
inputs, efficiently encoding categorical variables is a crucial aspect during
data analysis. An often encountered problem are high cardinality features, i.e.
unordered categorical predictor variables with a high number of levels. We
study techniques that yield numeric representations of categorical variables
which can then be used in subsequent ML applications. We focus on the impact of
those techniques on a subsequent algorithm's predictive performance, and -- if
possible -- derive best practices on when to use which technique. We conducted
a large-scale benchmark experiment, where we compared different encoding
strategies together with five ML algorithms (lasso, random forest, gradient
boosting, k-nearest neighbours, support vector machine) using datasets from
regression, binary- and multiclass- classification settings. Throughout our
study, regularized versions of target encoding (i.e. using target predictions
based on the feature levels in the training set as a new numerical feature)
consistently provided the best results. Traditional encodings that make
unreasonable assumptions to map levels to integers (e.g. integer encoding) or
to reduce the number of levels (possibly based on target information, e.g. leaf
encoding) before creating binary indicator variables (one-hot or dummy
encoding) were not as effective.
