4. Repeat steps 2 and 3 for each estimation method for all 500 sets of small area estimates corresponding to each sample.

The benchmark poverty alleviation from this scheme corresponds to the results of applying the above to the true municipal poverty rankings. The results for each method are compared to that result. In our implementation, we fix the poverty line at the 25th percentile of per capita income prior to transfers. The benchmark poverty rate, after delivering transfers to people in municipalities ranked by their true poverty rate, is 19.9 percent.

# **5 Results**

Recall from Section 4 that the _R_<sup>2</sup> measure often used to assess the Recall that the _R_<sup>2</sup> measure often used to assess the performance of machine-learning models is biased downward.<sup>27</sup> To understand the extent of this bias in practice, we have calculated the _R_<sup>2</sup> associated with gradient boosting when using both the direct estimates and true poverty rates as the basis for performance assessment. We conduct this comparison at two different levels of estimation, namely the PSU and municipality levels. All _R_<sup>2</sup> calculations are nevertheless made at the municipality level, meaning that the PSU-level model is estimated at that level and then the poverty estimates are aggregated to the municipality level. For each of our 500 samples, we thus calculate four _R_<sup>2</sup> measures: (1) PSUlevel estimates evaluated against the direct poverty estimates, (2) PSU-level estimates evaluated against the true poverty rates, (3) municipality-level estimates evaluated against the direct poverty estimates, and (4) municipality-level estimates evaluated against the true poverty rates. All models use only census-based covariates.<sup>28</sup> Panel (a) of Figure 1 presents these results and we find evidence of considerable downward bias for both the PSU- and municipality-level models. For example, for the PSU-level model, we find that the median “true” _R_<sup>2</sup> is 0.88 whereas the median “direct” _R_<sup>2</sup> is 0.59. Interestingly, the true _R_<sup>2</sup> tends to be higher for the PSU-level model than the municipality-level model, but the opposite is the case for the direct _R_<sup>2</sup> . This suggests that model selection based on the direct _R_<sup>2</sup> can be misleading in that the direct _R_<sup>2</sup> incorrectly suggests that the municipality is the preferred level of estimation. More specifically, we find that the true _R_<sup>2</sup> identifies the PSU as the preferred level of estimation for 437 of the 500 samples, but the direct _R_<sup>2</sup> only selects the PSU level for four of the 500 samples. The direct _R_<sup>2</sup> then identifies the incorrect level of estimation for 433 of the 500 samples.

In Section 4, we discussed two other ways that the _R_<sup>2</sup> can be misleading for assessing model performance: it is invariant to locational shifts and influenced by the variance of the true poverty rates. Using the Mexican data, we can assess the relevance of the former issue by comparing the

> 27For examples, see Jean et al. (2016), Pokhriyal and Jacques (2017), Steele et al. (2017), Lee and Braithwaite (2020), Yeh et al. (2020), or Chi et al. (2022).

> 28To ensure that our results are not affected by the composition of the sample, we calculate all _R_ 2 measures for each sample using only those municipalities with direct poverty estimates.

18