Experiments with Random Projections Ensembles: Linear Versus Quadratic Discriminants Gaussian classifiers make the modelling assumption that each class follows a multivariate Gaussian distribution, and include the Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA). In high dimensional low sample size settings, a general drawback of these methods is that the sample covariance matrix may be singular, so their inverse is not available. Among many regularisation methods that attempt to get around this problem, the random projection ensemble approach for LDA has been both practically successful and theoretically well justified. The aim of this work is to investigate random projection ensembles for QDA, and to experimentally determine the relative strengths of LDA vs QDA in this context. We identify two different ways to construct averaging ensembles for QDA. The first is to aggregate low dimensional matrices to construct a smoothed estimate of the inverse covariance, which is then plugged into QDA. The second approach is to aggregate the predictions of low dimensional QDAs instead. By implementing experiments in eight different kinds of data sets, we conduct a thorough evaluation. From it we conclude that both QDA ensembles are able to improve on vanilla QDA, however the LDA ensemble tends to be more stable with respect to the choice of target dimension of the random projections.

Random rotation ensembles In machine learning, ensemble methods combine the predictions of multiple base learners to construct more accurate aggregate predictions. Established supervised learning algorithms inject randomness into the construction of the individual base learners in an effort to promote diversity within the resulting ensembles. An undesirable side effect of this approach is that it generally also reduces the accuracy of the base learners. In this paper, we introduce a method that is simple to implement yet general and effective in improving ensemble diversity with only modest impact on the accuracy of the individual base learners. By randomly rotating the feature space prior to inducing the base learners, we achieve favorable aggregate predictions on standard data sets compared to state of the art ensemble methods, most notably for treebased ensembles, which are particularly sensitive to rotation.