To this end, consider splitting a node in a given tree into “left” and “right” child nodes based on some candidate splitting rule. Let _GL_ and _GR_ represent _Gj_ for the left and right child nodes, respectively. Similarly, let _HL_ and _HR_ represent _Hj_ for the child nodes. We can then find the reduction in the objective function as follows: where the first two terms in parentheses capture the loss associated with the child nodes and the third term captures the loss associated with the parent node being split. When splitting a node in any given tree, XGBoost then evaluates Eq. (12) for every possible split rule for every available explanatory variable and chooses the split associated with the maximal reduction in the objective function.<sup>12</sup> XGBoost then continues to split nodes in a given tree until a stopping criterion is met (e.g., a user-specified maximum tree depth). Once a given tree is built, XGBoost then moves to building the next tree in the ensemble and continues to do so until reaching the user-specified number of trees to build.

The above references several hyperparameters that must be chosen by the user, including the hyperparameters in the regularization term (i.e., _γ_ and _λ_ ), the maximum tree depth, and the number of trees in the ensemble. There are two additional hyperparameters that can be used to further prevent overfitting. First, one can use column or feature subsampling where each tree in the ensemble is built based on a (uniformly) random subset of all explanatory variables. This procedure can also speed up the algorithm because it reduces the number of covariates that must be searched across when creating new splits. Second, one can use shrinkage by scaling the predictions associated with each tree by a factor _η_ , which is often called the learning rate. The predicted value of the outcome at iteration _t_ is then given by _yi,t_<sup>_p_=</sup><sup>_y_</sup> _i,t_<sup>_p_</sup> _−_ 1<sup>+</sup><sup>_ηft_(</sup><sup>_xi_).Shrinkage reduces the influence</sup> of each tree so that no individual tree has too much influence on the ensemble prediction.

One possible approach to hyperparameter selection is to use a grid search where the user (1) specifies the domain of the hyperparameters, (2) applies the model to every combination of hyperparameters, and (3) selects the hyperparameters that perform the best in terms of some metric (e.g., mean squared prediction error in the validation dataset). Grid search, however, is computationally inefficient because hyperparameter proposals are uninformed by previous evaluations. We instead use a Bayesian hyperparameter optimization procedure, which sequentially updates a probability model that relates the hyperparameter space to the performance metric and then uses the model to intelligently explore the hyperparameter space (Bergstra et al., 2011). Specifically, our implementation of gradient boosting relies on the open-source library XGBoost, which we combine with the Hyperopt library that conducts Bayesian hyperparameter optimization using the tree-structured Parzen estimator.

Finally, regarding our application to the Mexican data, recall that mapping procedures entail

> 12This is known as the “exact greedy algorithm.” XGBoost also has an approximate algorithm that can be used for large datasets. See Chen and Guestrin (2016) for details.

10