# Introduction

Internet surveys generally cannot match the coverage and quality of their more traditional in-person counterparts, but they remain important tools for rapid, remote, and low-cost data collection. This is particularly useful to track changes in response to a shock, as during the COVID-19 pandemic, or to provide an additional way to reach people when security or other circumstances preclude a face-to-face strategy, as in some fragile and violence-affected contexts. The self-administered online format may also encourage more honest responses to sensitive questions than are usually expected of an interviewer-administered survey. One case in which these attributes may be particularly valuable is in tracking exposure to violence and the various demographic, socio-economic, and mental health variables with which that exposure may be associated.

The West Bank and Gaza Poverty and Equity team had a unique opportunity to compare a Facebook survey with an in-person survey on these topics in 2022. Following the spike in violence in the West Bank and the attacks on Gaza in mid-2021, the World Bank co-led the production of a Rapid Damage Needs Assessment (RDNA). During this process it became clear that there was a large gap in the understanding of the mental health impacts on a population that is subject to continuous exposure to violence and conflict. To fill this gap, the West Bank and Gaza Poverty team together with local and international partners<sup>3</sup> designed and implemented the first nationally representative survey of mental health outcomes among Palestinians. The survey was conducted between March and May 2022. At the same time, the World Bank team ran a web-based survey on Facebook which matched, to the extent possible, the same questions contained in the in-person survey questionnaire. This Facebook survey built on the experience and expertise of the MENA Data Lab, which supported 15 online surveys advertised through Facebook in 2022/2023.

The main report on the in-person survey<sup>4</sup> outlines the relevant context of the West Bank and Gaza as well as theoretical frameworks linking mental health outcomes to socio-economic outcomes, experience of adversity, and exposure to violence and trauma. Taking this context and theoretical relationships as given, the analysis that follows explores differences between the in-person and Facebook surveys.

We can reasonably expect estimates from internet surveys and in-person surveys to differ, even if they aim to measure the same things. In the West Bank and Gaza example, the Facebook survey estimates more severe outcomes (i.e., higher exposure to conflict, worse mental health) then its in-person counterpart for most topics and populations. In this report we explore three possible mechanisms that may have contributed to this pattern of differences: (1) nonresponse bias, (2) sensitivity bias, and (3) misreporting to influence resource allocation.

We find that estimated outcomes tend to be more similar for people in Gaza, possibly because of greater homogeneity in socioeconomic experiences and exposure to violent conflict and broader interest in a survey on the effects of the May 2021 violence. We also assess the extent to which our results are robust to different ways of controlling for observable characteristics, and find that neither alternative weights nor sample restrictions erase the systematic differences between surveys.

> 3 These included the Palestinian Central Bureau of Statistics (PCBS), Zentrum Überleben, and the International Security and Development Center (ISDC).

> 4 World Bank Group et al (2022), available here. The PPCS data [World Bank (2023)] can also be accessed directly <u>here.</u>

2