Bangladesh (Steele et al., 2017), and Belize (Hersh et al., 2021), among others.<sup>3</sup> While often subsumed under the term “machine learning,” this modern approach to poverty mapping actually consists of several practices that differ from more traditional approaches. Most obviously, the modern approach substitutes non-parametric statistical methods for the parametric approaches traditionally used for poverty mapping. In addition, the modern approach relies heavily on remotely-sensed covariates rather than covariates derived from census data. Finally, poverty maps produced with machine-learning methods generally share a common validation procedure. The key feature of this procedure is that model performance is assessed by calculating the _R_<sup>2</sup> from a regression of the observed poverty measures on the estimated poverty measures for the same geographical units.<sup>4</sup> The poverty measures used in this procedure are most often direct, sample-based estimates rather than the true poverty measures for the regions of interest. While direct estimates are unbiased, they can be imprecise estimates of true poverty measures, meaning that it is unclear whether the validation procedure is informative of actual model performance (Corral et al., 2021b). In this paper, we provide a more rigorous assessment of the performance of the modern approach to poverty mapping. Our approach consists of constructing a pseudo-census (hereafter “census”) that we use to conduct design-based simulation experiments. The census provides the true values we wish to estimate. Our experiments entail repeatedly drawing survey samples from the census where each sample produces direct poverty estimates for the sampled geographic areas. For each direct estimate, the presence of the census means that we observe the true poverty measure corresponding to the direct estimate and we use these true poverty measures to gain insight not only into the credibility of model validation based on direct estimates, but also the performance of machinelearning methods when validated against the true poverty measures. Given that population censuses rarely collect detailed data on income or expenditures, we construct our census from a large-scale household survey conducted in Mexico: the Mexican Intercensal Survey of 2015.

Prior to presenting our simulation results, we discuss three ways in which the _R_<sup>2</sup> can be misleading in terms of model assessment. Specifically, we show analytically that the _R_<sup>2</sup> based on direct estimates is biased downward, that it is insensitive to differences in location and scale between the poverty estimates and the reference measures, and that it is context-dependent in the sense that it is influenced by the variance of the true poverty rates. We argue that the mean squared error is a better measure for understanding the strength of the association between the poverty estimates and reference measures. In addition, we argue that the ultimate concern is to understand how different methodological choices affect poverty by influencing the efficiency of targeting. We therefore propose a targeting experiment through which we examine the relative ability of alternative

> 3See Jean et al. (2016), Yeh et al. (2020), Lee and Braithwaite (2020), and Aiken et al. (2022) for examples of other recent applications.

> 4For examples of the above procedure, see Jean et al. (2016), Pokhriyal and Jacques (2017), Steele et al. (2017), Lee and Braithwaite (2020), Yeh et al. (2020), or Chi et al. (2022). Note that a few studies use a similar procedure, but fit the machine-learning model to household-level data and then aggregate the model’s predictions to the geographic unit of interest (Blumenstock et al., 2015; Hersh et al., 2021; Aiken et al., 2022). Others rely on reporting correlation coefficients between survey based estimates and model predictions, which is similar (see Smythe and Blumenstock 2022).

3