conflict event, where estimates using the alternative weights are higher and therefore farther away from the PPCS estimates.<sup>12</sup>

## An unweighted approach

As an alternative way to try to control the distribution of observable characteristics, we sequentially restrict both survey samples to three groups that are overrepresented in the full Facebook sample: people with secondary or higher education, men, and people who report working in the past week. We then reestimate the main indicators from Table 2 using each pair of restricted samples, now without weights. We compare these estimates to those generated using the full (unweighted) samples for each survey. None of the three sample restrictions significantly alter the between-survey differences estimated for the main indicators using the full (unweighted) samples. Estimates of the between-survey differences in the main indicators for all four specifications are included in Appendix C.

# Conclusion

Given the opportunity to compare a Facebook survey to an in-person survey covering conflict exposure and potentially associated socio-economic and mental health outcomes in the West Bank and Gaza over a similar time period, we found that the Facebook survey consistently estimated more severe outcomes for a variety of indicators. Multiple mechanisms may have contributed to this difference in estimates, including overrepresentation in the Facebook sample of respondents who were interested in the survey topics, reduced sensitivity bias in the context of a self-administered online questionnaire, and reporting more severe outcomes than personally experienced to encourage resource flows to perceived needs.

Estimates were more similar between the surveys in Gaza, where the population is more homogeneous in relevant characteristics and more likely to be interested in responding to a survey on the effects of the May 2021 violence. The overall pattern of more severe outcomes estimated from the Facebook survey is robust to different ways of controlling the distribution of observable characteristics.

### **To what extent might these results generalize to other internet surveys?**

Most internet surveys are likely to produce results that differ meaningfully from those estimated by an inperson survey. Most are also likely to find that their achieved samples are biased on important sociodemographic characteristics relative to the general population, but these observable differences may or may not explain the differences in estimated outcomes.

The three mechanisms that we explore to explain the differences we estimate are good candidates to affect other internet surveys, but it should not be assumed that they will always move internet estimates in the same direction or to the same extent. For example, many internet samples will overrepresent people who are interested in the survey topics. In the case we discuss, we suggest that selection into the survey based on topic interest was correlated with more severe outcomes – that is, people were more

> 12 The change in the Facebook estimates that results from switching to the more minimalist set of weights is closely associated with differences between outcomes for more and less educated respondents, as the strongest correction that the initial calibration constraints make is to the distribution of educational attainment.

14