We assess the model’s in-sample performance using five-fold cross-validation to determine the subset of variables that best perform across folds.<sup>9</sup> In-sample performance tests show that both urban and rural models display a high R-square, low Mean Square Error (MSE), and can predict welfare ranking well (Spearman rank correlation). Overall, in-sample tests show very small differences between observed and predicted poverty levels of between 1.6 and 2 percent in urban and rural models, respectively. The higher predictive capacity in the urban samples is likely the result of the lower variability in income levels and a limited number of variables to fully model rural households' income-generating processes, particularly assets used in agricultural production (Table 4).

**Table 4: SWIFT-Plus model validation tests**

|||**R2**|**Mean**<br>**MSE**|**Spearman**<br>**correlation**|**In-s**<br>**I**<br>**Observed**<br>**poverty**|**ample predic**<br>**E-LFS 2019/2**<br>**Predicted**<br>**Poverty**|**tion**<br>**0**<br>**Difference**<br>**(%)**|**Out-o**<br>**Observed**<br>**poverty**|**f-sample pre**<br>**IE-LFS 2021**<br>**Predicted**<br>**Poverty**|**diction**<br> <br>**Difference**<br>**(%)**|
|---|---|---|---|---|---|---|---|---|---|---|
|IE-LFS<br>|Urban|66.9|0.083|84.2|55|54.1|1.6%|51.1|52.8|3.3%|
|2020/19<br>Quarter 3|Rural|58.2|0.083|72.4|51|52.0|2.0%|52.3|54.4|4.0%|

**Source** : World Bank estimations using 2019/20 IE-LFS and 2019/20 and AWMS R3 surveys.

# Validity of the results

The validity of the survey-to-survey imputation model relies on two main assumptions **.** First, sampling must be comparable across the surveys. Second, the predictors for poverty imputations should be key determinants of the intertemporal variations in household expenditure and be stable over time. This section discusses these assumptions in detail.

**Assumption 1** : The survey sampling process must be comparable across surveys. To estimate comparable poverty measures, survey-to-survey imputation techniques require that information in both surveys was collected following similar protocols. Doing this requires that both surveys follow the same sampling process. If that is not the case, poverty imputations can be biased and over/underestimate the true poverty rate, even if the model correctly predicts the relation between household characteristics and expenditure in training data. This is a particularly valid concern in the case of phone surveys since phone ownership positively correlates with higher expenditure.

The comparability of the AWMS sample with the IE-LFS 2019-20 sample relies on the reweighting process that corrects a possible sampling bias induced by phone ownership and differences in survey response. As stated in the data section, the AWMS R3 listing comes from households that provided a cellphone number

> 9 To avoid over-fitting, Cross-validation (CV) is a standard empirical test in machine learning literature. The Q3 urban and rural samples of the IE-LFS 2019-20 are randomly split into five subsamples each; then, the model is estimated on four folds (training dataset) and evaluated on the remaining one (testing dataset). Since the remaining fold is not included when the model is trained, all performance indicators in the remaining fold are not subject to the over-fitting problem. This exercise is repeated iteratively for different levels of statistical significance to maximize model performance.

10