after running gradient boosting with all covariates on each sample and averaging each covariate’s importance across the 500 samples. We find that all of the top 10 covariates are census-based, with only five geo-referenced covariates entering the top 20.

Figure 4 presents MSE and bias estimates for various models across different “poverty quantiles.” That is, we order all municipalities by their true poverty rates, divide them into 50 quantiles, and calculate the average MSE and bias for each quantile. For the sake of simplicity, we only present results for two machine-learning models – the municipality-level models using only geo-referenced and census-based covariates – and select traditional methods. Panel (a) presents the MSE results and panel (b) presents the bias results. In both plots, gradient boosting with census-based covariates performs quite similarly to the “gold standard” unit-level model across all quantiles. However, gradient boosting with geo-referenced covariates again departs considerably from this benchmark, particularly at the lowest and highest quantiles. The bias results are particularly notable: the model with geo-referenced covariates systematically underestimates poverty at the lowest quantiles and overestimates poverty at the highest quantiles. The standard implementation thus understates the spatial distribution of poverty.

An additional consideration is how the methods perform for predictions which are out of sample. Because the experiment consists of 500 samples taken from the census data, a given municipality may be present in one of the samples and may be absent in others. Consequently, to evaluate out of sample properties we rank municipalities by their likelihood of selection across the 500 samples. The likelihood of being present ranges from 0.09 to 1, with a median of 0.48 and a mean of 0.53. Among the 20 percent least likely to be sampled municipalities, the gradient boosting model with GIS-based covariates is the worst performing in terms of MSE followed closely by unit-context models (Figure 5). In terms of bias among the 20 percent of municipalities that are least likely to be sampled, the gradient boosting model with the geo-referenced covariates shows very wide variability, while unit context models show downward bias. The other gradient boosting models perform similarly to CensusEB models in terms of bias and MSE. The performance, in terms of MSE, of the different methods seems to improve with the likelihood of being selected. Among municipalities most likely to be selected, the gradient boosting model with the geo-referenced covariates is still the worst performing (Figure 6). However, in terms of bias, among the most likely to be selected municipalities, the bias of Fay-Herriot methods at the municipality level is the best performing illustrating the importance of the methods EBLUP.

While we have focused on the performance of gradient boosting up to this point, there are nevertheless several other machine-learning models that can be used for poverty mapping. Leading alternatives include the least absolute shrinkage and selection operator (lasso) (Tibshirani, 1996), the random forest (Breiman, 2001), and Bayesian additive regression trees (BART) (Chipman et al., 2010). While the lasso is a simple way to conduct variable selection and regularization in the context of linear regression, the random forest and BART models are similar to gradient boosting in that they all rely on regression tree frameworks. We ask whether these alternative models can perform better than gradient boosting within the confines of the standard implementation, which 24