# **3.3 Application using Random Forest**

The prediction and validation framework is applicable to any binary classification algorithm that generates predicted probabilities for the positive class. In this paper we use a Random Forest classifier.<sup>14</sup> We focus on this particular approach in light of the additional results provided in supplementary Appendix B which show that in this same data set, the Random Forest offers a substantial improvement in predictive performance over simple linear models, but that little further is gained by moving to more complex learners such as neural networks.

Implementing the algorithm requires specifying several tuning parameters that control the structure of the forest or its trees. We select these to optimize predictive performance using the cross-validation procedures described above. This is implemented by searching over a practical grid of values.

First, since we require predicted probabilities, we implement a probability forest in which trees are grown as regression trees following Malley et al. (2012). An important tuning parameter is minimum node size that regulates tree depth by setting the minimum number of cases at the terminal nodes of the decision trees. We use the original value of 1, as well as the default value of 10 in the implementation of Wright and Ziegler (2017).<sup>15</sup> The out-of-sample performance is heavily dependent on the predictive power of individual trees and the correlation between them.<sup>16</sup> The relationship between the two is mainly one of trade-offs and can to a certain degree be regulated by optimizing over the number of variables that are used to grow individual decision trees.<sup>17</sup> This number is controlled by _mtry_ , and its default value is ~~�~~ ( _D_ ) where _D_ is the number of predictors in the data set. Performing a grid search for _mtry_ is computationally challenging and in this paper is made feasible by focusing on particular combinations of tuning-parameters.<sup>18</sup> In particular, the grid search is performed only when the minimum node size is 10, and makes use of a

> 14The algorithm was introduced by Breiman (2001) as a generalization of the tree bagging by combining it with the random subspace method of Ho (1998). It has quickly become a popular non-parametric classification tool due to its ability to generate good results using standard tuning parameter values and its robustness to noise. It constructs prediction rules without imposing strong prior assumptions on the functional form of the relationship between predictors and the target variable. Good review articles include those of Biau (2012); Biau and Scornet (2016). Random Forest generates results that are accurate while relatively straightforward to interpret (Banerjee et al., 2012; Petkovic et al., 2018).

> 15Breiman’s original algorithm grows trees to purity, that is, until each terminal node contains only observations from one class. When nodes are pure, the probability estimate at each terminal node is thus always either 0 or 1. This means that, apart from a low correlation between trees, a large number of them is needed to obtain precise estimates of probabilities by the forest. Increasing minimum node size allows the frequency of class occurrence at the terminal nodes itself to be a reasonable probability estimate.

> 16The generalization error converges _almost surely_ to a limit as the number of trees in the forest grows. We use 500 trees, corresponding to the default value in the implementation by Wright and Ziegler (2017). The magnitude of expected out-of-sample error in the limit in turn depends on the predictive strength of the individual trees and the correlation between them. See also (Scornet, 2017).

> 17The predictive strength of individual trees can to a degree be improved by allowing more variables to enter individual base learners. However, while larger values for _mtry_ may result in stronger base learners, it also leads to a higher correlation between them. In particular, as _mtry → D_ , the probability that two base trees contain the same variables approaches 1. If many individual trees contain a highly similar set of variables, then their predictions become more correlated. Smaller values for _mtry_ may instead increase diversity of base learners, at the cost of lower individual predictive power.

> 18Larger values of _mtry_ significantly increase computational cost as it becomes more complex to process the individual trees. A grid search for _mtry_ is more manageable in combination with tuning values that lead to simplifications elsewhere in the calculations; higher minimum node size that leads to faster individual tree-building, and randomized splitting does not require calculating Gini coefficients.

11