The results of the targeting simulation are relevant to cases where targeting is based solely on the ranking of municipalities – all methods appear to do a solid job. Nevertheless, there are many instances where the targeting mechanism may rely on the method’s poverty estimates. For example, Senegal’s Unique National Registry (RNU - in French) of vulnerable households relies on actual estimates from poverty maps to determine the quotas for sampling by municipality (Ferre 2018). It is in those cases where a method that yields unbiased and precise estimates should be employed.

# **6 Conclusion**

This paper sets out to validate traditional small area estimation and machine learning approaches for poverty mapping. Machine learning approaches have become popular across the globe for producing highly disaggregated estimates of poverty which often feed into policy making. The methods are particularly popular in the most data deprived scenarios where census data are not frequently collected, and even when collected may not be contemporaneous to a household survey. Hence, the methods are also a popular alternative for cases where the census data is too old for use under a traditional unit-level small area estimation exercise. To the best of our knowledge, a rigorous validation of machine learning methods like the one done here, where machine learning methods are compared to traditional small area estimates and to a true welfare measure, has not been done before.

We first showed analytically that that the _R_<sup>2</sup> can be a misleading measure of model performance: it is biased downward when direct estimates are used as the reference measure of model performance, it is insensitive to differences in location and scale between the reference measure and the model predictions, and it is influenced by the variance of the reference measure. One practical issue this raises is that it can be misleading to compare the performance of poverty mapping methods across different datasets, as is done in Yeh et al. (2020), Chi et al. (2022), and Aiken et al. (2022), among others. Most notably, survey design elements can affect measurement error in the direct estimates (e.g., through different sample sizes), which in turn influences the magnitude of bias in the _R_<sup>2</sup> . All else equal, models applied to low-bias settings will appear to perform better than those applied to high-bias settings.

Our simulations built on these analytical results. First, we showed empirically that the direct _R_<sup>2</sup> exhibits considerable downward bias, with the true _R_<sup>2</sup> being roughly 35 to 50 percent higher than the direct _R_<sup>2</sup> . We also showed that this bias has implications for model selection, as the direct _R_<sup>2</sup> commonly identified the incorrect level of estimation in our data. Second, we assessed the performance of machine learning methods and found that they may outperform or achieve similar bias and MSE as the best performing traditional small area estimation approaches, particularly in out-of-sample predictions. Third, we present a comparison on the potential benefits of publicly available GIS covariates. We show how GIS covariates under the data used here do not outperform census aggregate data. This result suggests that there may be considerable gains in linking census 30