When it comes to census data, how old is too old? Or, put another way, at what age do census-based predictions become less accurate than current geospatial predictions? It is difficult to know, but Newhouse et al. (2022) offer one small piece of evidence on this point. When evaluated against Mexican 2015 small area poverty estimated based on the intercensus, 2010 census-based estimates are more accurate than 2015 estimates based on geospatial indicators (correlation of 0.91 vs 0.86). However, this is only representative of one context, and regional patterns of poverty in Mexico may have been more static during this time than in other contexts.

# d. Prediction accuracy is very sensitive to the training data

Many of the papers discussed above predict poverty rates or average asset indices estimated at the village level. Since these are often derived from surveys with a limited number of observations per village, this raises the issue of noise in the dependent variable. In fact, correlations between predicted values and census-based estimates depend critically on the extent of noise in the training data, which will reduce measured accuracy. For example, Engstrom et al. (2022) considered how the accuracy of predictions depends on the size and nature of the sample used to estimate average per capita consumption at the GN division level. That analysis correlated interpretable geospatial features with predicted per capita consumption imputed into a census. Model R<sup>2</sup> fell from 0.61 when using the mean over all census households, to 0.55 when using the mean over thirty households, and further to 0.40 when taking the mean over 8 households per enumeration area. Differences in the extent of noise present in the training data, as well as the reference evaluation measure, will not necessarily affect the ranking of different types of models within the same context. But it explains much of the wide variation in R<sup>2</sup> observed in Table 1 across different studies, which underscores the benefits of evaluation studies that compare different methods in the same context, using the same reference measure.

Gualavisi and Newhouse (2022) offer another stark example of how sensitive predictive accuracy is to the source of training data. Using a census extract from 10 districts in Malawi, the analysis compared estimates of average village welfare imputed into a household census with estimates derived from combining a survey with publicly available geospatial indicators. However, it also considers a third option, which involves hypothetically supplementing the survey with a partial registry, a “microcensus” that interviews all households in a randomly selected 450 of the 4,500 villages with geolocated data. This involves a two-step approach, where welfare is first predicted into the partial registry and then a geospatial model is trained against the partial registry predictions.

Using a partial registry in this way yields an R<sup>2</sup> of 0.35, as opposed to 0.01 for the geospatial poverty map based on survey data and 0.02 for the wealth estimates from Chi et al. (2021). These R<sup>2</sup> values are much lower than those cited in the previous section. The weak correlation between the Chi et al (2021) estimates and these census-based predictions reflects the challenge of distinguishing between village welfare levels in this context. In particular, the sample consists of 4,500 villages, in 10 poor Malawian districts, for which names could be matched between the census and the Unified Beneficiary Registry data containing household geocoordinates. In this context, the Chi et al. (2021) estimates may perform poorly because they come from a model trained to wealth while the benchmark reference indicator is per capita consumption; these are conceptually different measures of welfare that tend to diverge more in rural areas and among the extreme poor (Ngo and Christiaensen, 2019). For the standard geospatial poverty map based on the survey, the low R<sup>2</sup> is also due to the paucity of survey data, as there are only 9