more accurate predictions in sample than out of sample. EBP shows the largest drop in performance out of sample, as shown in subsection 3.1. The more flexible nature of the machine learning models appears to be better suited for predicting into out-of-sample areas than EBP. In addition, out-of-sample EBP predictions suffer from the unavailability of sample data to condition on when estimating the random area effect.

There are also noticeable differences in precision across in-sample and out-of-sample areas. Since the machine learning models do not include a random effect conditioned on the sample, the bootstrap procedure estimates confidence intervals that are roughly the same size for in-sample and out-of-sample areas. Out-of-sample accuracy is lower than in-sample accuracy, meaning that out-of-sample coverage rates tend to be lower as well; for example, average coverage rates for wealth estimates using XGBoost fall from 91.8 percent to 83.3 percent. This does not result from overfitting the model to the sample, given that LASSO is used for model selection in the EBP model and that regularization methods are built into the machine learning algorithms. Instead, because villages are sampled proportional to their population size, out-of-sample areas tend to be less populated and may therefore systematically different from in-sample areas, even after taking into account sampling probabilities. Although sample weights are included, out-of-sample predictions suffer from the relative paucity of training data from rural, less populated areas.

To better understand which geospatial predictors are important in the models, we examine the prevalence of different types of variables selected by LASSO for the EBP models. In addition, we calculate Shapely decompositions of the impact of predictors on the predictors generated by extreme gradient boosting models (Lundberg et al., 2019). Overall, land cover classification variables, including vegetation indices, consistently emerge as important predictors. Population estimates, pollution variables, and night time lights also make important contributions in different contexts. This is consistent with geospatial predictors acting as proxies for population density and urbanity, which are in turn systematically correlated with wealth and poverty. (Newhouse, 2024) Finally, we examine the impact of altering the model specification by expanding the set of candidate geospatial variables and including additional interactions. In particular, we include MOSAIKS variables (Rolf et al., 2021) and separately experiment with interacting all predictors (including the MOSAIKS variables) with a measure of urbanity. Neither adding geospatial predictors nor adding these interactions leads to a meaningful improvement in prediction accuracy on average. However, both adding additional predictors and interactions moderately increases estimated uncertainty and improves coverage rates with the machine learning methods. Interestingly, the inclusion of more candidate predictors leads to worse results for EBP. These results suggest limited benefits from supplementing a relatively small set of publicly available features with MOSAIKS indicators. This result is useful because MOSAIKS variables, which are derived from a 5