## <mark>4.1.4 Overall bias</mark>

The results so far have examined the separate sources of bias introduced at each stage of the phone survey selection process. Column 3, 6, 9, and 12 in Table 4.4a present the cumulative bias for the Ethiopia, Malawi, Nigeria, and Uganda HFPS interviewed samples. It is clear that the successfully interviewed HFPS samples in all four countries are biased across many different household characteristics. Overall, the bias is skewed towards urban households (except in Uganda) as well as households that are relatively better off in terms of material well-being. Household heads that are successfully interviewed are better educated and more likely to working in formal (wage) employment compared to the overall LSMS-ISA sample. These results are mimicked in Table 4.4b when disaggregating the Ethiopia sample into the urban and rural components.

# <mark>4.2 Testing Effectiveness of Weighting Adjustments</mark>

The results above have clearly demonstrated the bias present in the sample of successfully interviewed households in the HFPS in the four countries. This bias would compromise the external validity of the results obtained from the HFPS which would not be fully representative of the general population. Furthermore, the bias in all four countries skews towards households that are generally better off and therefore would not accurately reflect the situation for poorer households, a segment of the population that is likely most vulnerable to the COVID-19 crisis.

However, since the samples for these two surveys are derived from representative samples in the ESS, IHPS, GHS-Panel and UNPS, there is potential to reduce the bias by harnessing the detailed information available in the F2F surveys to conduct the weighting adjustments described in Section 3.1.2. Implementing these weighting adjustments will likely counteract the bias but will not fully eliminate it. This section presents results from an assessment of the effectiveness of the adjustments performed on the HFPS in each country.

Table 4.5a presents three different characteristic profiles representing: (i) the full representative LSMSISA samples applying usual survey weights (columns 1, 6, 11, and 16) , (ii) the HFPS samples applying weights that **_include_** the bias reweighting adjustment (columns 2, 7, 12, and 17), and (iii) the HFPS samples applying weights that **_do not include_** the bias reweighting adjustment (columns 4, 9, 14, and 18). Comparing the relative gaps between the representative profile in (i) with the profile of characteristics obtained in (ii) and (iii) will provide an indication both of the effectiveness of the bias adjustment as well as the extent of bias that remains after the adjustment. The difference between the profile of characteristics in the representative sample and the HFPS sample using the adjusted and unadjusted weights was estimated and presented in Table 4.5a.

Examining first the difference between the representative sample and the HFPS sample using the unadjusted weights (columns 5, 10, 15, and 20), in Ethiopia and Malawi there is substantial bias across the majority of characteristics. For Nigeria and Uganda, the bias is still present, though across fewer different indicators than in Ethiopia and Malawi. The results here using the HFPS unadjusted weights largely mimic the results from above (using the LSMS-ISA weights) indicating that there is substantial bias.

27