16 households per enumeration area in the survey sample. Besides the disappointing performance of the standard geospatial poverty estimation and the Meta wealth index estimates in this challenging context, this exercise also illustrates how much the inclusion of additional data from the partial registry improves the performance of the prediction. The partial registry effectively adds valuable information to the training data when using geospatial data for small area estimation. This enables the development of a much more accurate prediction model, using proxy welfare variables that are cheaper and easier to collect.

The importance of training data raises the question of whether estimating different models for different geographies, such as urban and rural areas, improves prediction accuracy. Estimating separate models may improve the accuracy of the estimates by better accounting for heterogeneity across regions. However, these models also utilize less training data, which reduces the richness of the prediction model in the typical case when the sample is used to select or tune models. Newhouse et al (2022) provide some evidence on this question, comparing monetary poverty estimates in Mexico derived from a national model, separate models for urban and rural areas, and separate models for each of six state groupings. Compared to a baseline national model, estimating models separately for urban and rural areas leads to a minor improvement in accuracy, raising the correlation with census-based estimates from 0.86 to 0.87 in sampled areas and from 0.70 to 0.71 in non-sampled areas. Estimating separate models for six groups of states led to a similar minor improvement. These findings for Mexico. however, do not necessarily generalize to other contexts, and additional evidence comparing the accuracy of models specified at different levels would be useful. Tree-based machine learning methods, such as those discussed below, provide more flexible alternatives that explicitly model interactions between predictors. Use of these models should therefore mitigate any benefit from estimating multiple models in disaggregated geographic regions or areas.

e. Out-of-sample predictions are significantly less accurate than in-sample predictions Informative sampling occurs when sampling probabilities for Primary Sampling Units vary in a way that is correlated with the outcome of interest, such as welfare. This is typically the case in two-stage samples in which primary sampling units are sampled with probability proportional to size, since population size is correlated with most outcomes of interest. Informative samples, if not appropriately adjusted, produce biased estimates. The standard approach to adjust for informative sampling is to weight observations by the inverse probability of selection. This is usually straightforward to do when estimating EBP models with household surveys containing sample weights, for example when using the R EMDI and Stata SAE software packages. This protects against bias from informative sampling within sampled areas, but not against bias in predictions for areas that are not included in the sample (Pfefferman and Sverchkov, 2009). The bias in estimates for non-sampled areas can be severe. Table 2 shows evidence on the extent to which informative sampling leads to less accurate predictions in nonsampled areas. The extent of this bias appears to depend on the country context and the nature of the sample. In the examples reported in Table 2, the estimates in Burkina Faso appear as a particular outlier, with an out-of-sample R<sup>2</sup> of 0.21. It is difficult to know exactly why the out-of-sample estimates for Burkina Faso are so inaccurate, but this likely reflects large differences in the relationship between covariates and predictors between sampled and non-sampled areas. In addition, the source of validation 10