metadata: authors: 0: author: Cullen,Claire Alexis
count: Nigeria,Rwanda
docna: 0: docna: Method Matters : Underreporting of Intimate
            Partner Violence in Nigeria and Rwanda
docty: Policy Research Working Paper
subtopic: Gender and Development,Social Cohesion,Educational Sciences,Services & Transfers to Poor,Access of Poor to Social Services,Economic Assistance,Disability,Social Conflict and Violence
teratopic: Gender,Education,Poverty Reduction,Macroeconomics and Economic Growth,Social Protections and Labor,Social Development
abstracts: cdata!: This paper analyzes the magnitude and
            predictors of misreporting on intimate partner and sexual
            violence in Nigeria and Rwanda. Respondents were randomly
            assigned to answer questions using one of three survey
            methods: an indirect method (list experiment) that gives
            respondents anonymity; a direct, self-administered method
            that increases privacy; and the standard, direct
            face-to-face method. In Rwanda, intimate partner violence
            rates increase by 100 percent, and in Nigeria, they increase
            by up to 39 percent when measured using the list method,
            compared with direct methods. Misreporting was associated
            with indicators often targeted in women's empowerment
            programs, such as gender norms and female employment and
            education. These results suggest that standard survey
            methods may generate significant underestimates of the
            prevalence of intimate partner violence and biased
            correlations and treatment effect estimates.
geo_reg_key: 517196,517191,1927320
repnme: repnme: Method Matters : Underreporting of Intimate
            Partner Violence in Nigeria and Rwanda
display_title: Method Matters : Underreporting of
            Intimate Partner Violence in Nigeria and Rwanda
listing_relative_url: /research/2020/06/32110536/method-matters-underreporting-intimate-partner-violence-nigeria-rwanda
topicv3: Rape and Sexual Violence m1326262 2443,Education m1327048 470,Cash Transfers m1327437 1608,Gender m1327621 615,Gender Based Violence m1504507 2901
publishtoextweb_dt: 2020-06-09T00:00:00Z
historic_topic: Gender,Education,Poverty Reduction,Macroeconomics and Economic Growth,Social Protections and Labor,Social Development
origu: Office of the Chief Economist (AFECE)
no_of_pages: 43
author_keys: 0: author_key: 1481172
model_extractions: Policy Research Working Paper 9274

# Method Matters

Underreporting of Intimate Partner Violence in Nigeria and Rwanda _Claire Cullen_ Africa Region Gender Innovation Lab June 2020
Policy Research Working Paper 9274

# **Abstract**

This paper analyzes the magnitude and predictors of misreporting on intimate partner and sexual violence in Nigeria and Rwanda. Respondents were randomly assigned to answer questions using one of three survey methods: an indirect method (list experiment) that gives respondents anonymity; a direct, self-administered method that increases privacy; and the standard, direct face-to-face method. In Rwanda, intimate partner violence rates increase by 100 percent, and in Nigeria, they increase by up to 39 percent when measured using the list method, compared with direct methods. Misreporting was associated with indicators often targeted in women’s empowerment programs, such as gender norms and female employment and education. These results suggest that standard survey methods may generate significant underestimates of the prevalence of intimate partner violence and biased correlations and treatment effect estimates.

This paper is a product of the Gender Innovation Lab, Africa Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The author may be contacted at claire.cullen@bsg.ox.ac.uk.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Method Matters:

Underreporting of Intimate Partner Violence in Nigeria and Rwanda Claire Cullen<sup>_∗_</sup>

## JEL: J12, J16, K42, O15

Keywords: Gender, Domestic violence, Measurement, Norms

> _∗_ Cullen: World Bank Africa Gender Innovation Lab & Blavatnik School of Government, University of Oxford, claire.cullen@bsg.ox.ac.uk. Acknowledgements: I thank Julien Labonne, Stefan Dercon, Arthur Alik-Lagrange, Noam Angrist, Dara Kay Cohen, Cheryl Doss, Markus Goldstein, Summer Lindsey, Anandi Mani, M ̃uthoni Ngatia, Sreelakshmi Papineni, Rachael Pierotti, Simon Quinn, Diana Contreras Suarez, Alex Twahirwa, Julia Vaillant, and seminar participants at the World Bank Africa Region Gender Innovation Lab, Oxford CSAE, IFPRI, UC Berkeley, Yale EGEN, Australian Gender Economics Workshop, and SVRI for helpful comments and suggestions. I would also like to thank the respondents, and the staff at Laterite Ltd in Rwanda and TNS Global in Nigeria, and Victoria Isika, Jean Paul Ngabonziza, Maritetou Sanogo and Oluwatoyin Zakariya for their superb research assistance and data collection work. The impact evaluations through which this data was collected were conducted by the World Bank Africa Region Gender Innovation Lab and were funded by the World Bank’s Nordic Trust Fund, the Swiss Development Cooperation and the Government of Rwanda, and in Nigeria, by USAID and the Umbrella Facility for Gender Equality, a World Bank Group multi-donor trust fund. The views presented in this paper are those of the author and do not represent those of the World Bank or its member countries. All remaining errors are my own. This study was approved by the Oxford University Central University Research Ethics Committee, HML IRB, and the Rwandan National Ethics Committee. This paper was previously circulated under the title “Truth be told: underreporting of intimate partner violence in Rwanda and Nigeria.”
# **1 Introduction**

It is challenging to accurately measure sensitive attitudes, behaviors, and experiences such as political preferences, prejudice, risky sex, and intimate partner violence (IPV). Often, researchers and policy makers must rely on self-reported survey data, yet such data may not be accurate. For sensitive topics in particular, there are many reasons why survey respondents may be reluctant to tell the truth, including a fear of being judged, endangered, or legally penalized (Blair, Coppock and Moor, 2018). If misreporting bias is significant and systematic, researchers and policy makers risk over or under-estimating the prevalence and welfare costs of sensitive issues, and drawing inaccurate conclusions about their causes and potential solutions.

survey methods. We assess whether survey methods that increase respondent privacy and anonymity affect IPV reporting compared to the status quo of direct face-to-face interviews.<sup>1</sup> In Rwanda and Nigeria, we randomly assigned women to answer questions about their experience of emotional, physical and sexual violence using one of three methods: an indirect method (list experiment) and two direct survey methods: face-to-face questions asked by an enumerator, or audio computer-assisted self-interview (ACASI) on an electronic tablet. Compared to the direct face-to-face method, the list experiment method provides additional confidentiality as respondents do not directly disclose their experience of violence. Instead, they give a response that allows researchers to calculate average prevalence across the whole sample, essentially making respondents’ answers anonymous (Blair and Imai, 2012). The ACASI method affords more privacy than face-to-face methods as respondents answer questions themselves by listening to pre-recorded questions in a headset and then selecting their answer on a tablet.

Results suggest that estimated IPV prevalence varies depending on the survey method used to measure it. The most widely used method, face-to-face interview, resulted in the lowest IPV rates, followed by the more private ACASI method, and finally, the anonymous list method with the highest. In Rwanda, IPV rates were double, and in Nigeria, up to 39 percent greater when measured using the list method compared to the direct methods. There was no statistically significant difference between the list and direct methods for rates of non-partner sexual violence in Rwanda or sexual IPV in Nigeria. This lack of difference for sexual violence could be explained by a lack of statistical power in Rwanda, and a Nigerian context where sexual IPV was not significantly stigmatized.

We also that misreporting is systematic and correlated with characteristics often theorized as causes of IPV and targeted in women’s empowerment programs. The characteristics that correlate differently with IPV depending on the measurement method used include indicators of women’s education and employment, community gender norms, vulnerability, and marital relationship quality. For example, in Rwanda, when IPV was measured using the direct methods, there was no relationship between IPV and conservative community gender norms. However, when IPV was measured using the list method,

> 1In more than 60 developing countries, Demographic and Health Surveys (DHS) have included a standardized domestic violence module where enumerators directly ask women about their experience of a set of specific emotional, physical, and sexual abuse episodes by an intimate partner. Using the face-to-face method, the WHO estimates that approximately one in three women worldwide has experienced IPV in her lifetime (García-Moreno et al., 2013). IPV prevalence varies widely, though rates are generally highest in Africa (Devries et al., 2013; Global Burden of Disease, 2017). Researchers collecting IPV survey data take steps to minimize the risk of underreporting, for example by instituting ethics protocols about privacy during interviews, asking multiple questions to give respondents several opportunities to disclose violence, and using specially-trained female enumerators to conduct women’s interviews (Heise and Hossain, 2017; WHO, 2016, 2001). However, it is still assumed that face-to-face data collection may yield underestimates given the sensitivity.

1
we a large, positive relationship between IPV and conservative norms. This suggests that women living in the most conservative villages underreport IPV when asked directly and that the list method’s anonymity (but not ACASI’s privacy) makes them comfortable enough to report it. Also, in both Rwanda and Nigeria, the education-IPV relationship switches depending on how IPV is measured. There is suggestive evidence of a positive relationship between IPV and women’s education when IPV is measured directly, but a negative relationship when using the list method. These results suggest that the standard, direct methods currently used in most surveys risk generating significant underestimates of IPV prevalence. Further, social scientists risk estimating biased correlations and, if treatment is correlated with misreporting, estimating biased treatment effects and drawing inaccurate conclusions about IPV’s causes and the effectiveness of interventions to prevent it.<sup>2</sup> This paper contributes to the growing social science literature studying alternative survey methods that may reduce misreporting (Karlan and Zinman, 2012; Blattman et al., 2016; Chuang et al., 2018; Blair, Coppock and Moor, 2018; Lépine et al., 2020), including on IPV measurement (Agüero and Frisancho, 2017; Peterman et al., 2017; Joseph et al., 2017; Bulte and Lensink, 2019; Agüero et al., 2020).

Recent studies on IPV measurement have compared respondents’ experience of violence across the face-to-face and list experiment methods. They found mixed results, with one study in Peru finding the same rates of IPV between face-to-face and list methods (Agüero and Frisancho, 2017) and others in Vietnam, India, and Nigeria finding a difference (Bulte and Lensink, 2019; Joseph et al., 2017; Banerjee, La Ferrara, and Orozco as cited in World Bank, 2017).<sup>3</sup> However, in several of these studies, respondents were asked about their IPV experience using two methods within the same survey about similar, but slightly different, violence acts. This potentially limits comparability across methods. To avoid this issue, we conducted one of the first IPV measurement experiments that randomly assigns respondents to answer questions using only one method rather than asking respondents two overlapping questions using different methods. Also, this paper aims to contribute to the limited evidence on who misreports on sensitive questions and why, the implications of misreporting for bias, and the trade-offs between different survey methods that seek to address misreporting.

This is the paper to compare the prevalence of sexual and intimate partner violence using three survey methods that offer respondents differing levels of privacy and anonymity, and to compare across two countries, and for women and men.<sup>4</sup> It is also one of the first papers to study IPV misreporting in Africa and to demonstrate that misreporting is substantial in some contexts, varies by the type of violence, and is correlated with characteristics commonly targeted in women’s empowerment interventions.

The rest of this paper is organized as follows. Section 2 reviews the literature on the measurement of sensitive topics and IPV. Section 3 outlines the experimental design and describes the sample and data. Section 4 presents the results, and 5 concludes.

> 2A further motivation for assessing IPV measurement methods is that the status quo of asking direct face-to-face questions places a high emotional burden on both respondents and enumerators. WHO guidelines thus recommend particular protocols for IPV research, for example, ensuring respondents have access to referral services (WHO, 2016). If an alternative measurement method was identified that was less biased, logistically easier and cheaper, and more ethical than the status quo, then it would likely be adopted by more researchers and contribute to new research on IPV. This would improve our understanding of the causes of IPV and may broaden the range of interventions that could be used to reduce it (Peterman et al., 2017).

> 3A related study by Agüero et al. (2020) in rural Peru compares women’s direct self-reported experience of IPV with a listing of IPV survivors by female community leaders. They found that community leaders significantly underreport IPV rates compared to women’s own self-reports.

> 4In Rwanda, we asked men about their perpetration of emotional violence using one of the three methods. The ACASI method was only tested in Rwanda.

2
# **2 Survey methods to measure sensitive topics**

There is robust evidence that people underreport the truth about sensitive topics when they are directly asked in a survey. Such research compares self-reported survey answers to observed or independently verifiable outcomes such as tax records, receipt of welfare benefits, drug prescriptions for mental illness, and voting in an abortion law referendum (Bharadwaj, Pai and Suziedelyte, 2017; Meyer and Mittag, 2019; Gottschalk and Huynh, 2010; Murray-Close and Heggeness, 2018; Rosenfeld, Imai and Shapiro, 2016).

However, reliable, unbiased observational data are rare for many sensitive topics, including IPV.<sup>5</sup> This is particularly true in developing countries, where there is limited available administrative data from the medical or legal services that IPV survivors may use. Further, where these services are available, the people able or choosing to access them are likely not representative of the broader population of those experiencing IPV.<sup>6</sup> As a result of these administrative data limitations and concerns about underreporting on face-toface surveys, researchers are considering alternative survey methods to minimize misreporting bias. Two popular alternative methods, ACASI and the list experiment, address slightly different motivations for misreporting. The ACASI method is primarily designed to overcome misreporting driven by shame when responding face-to-face to an enumerator, while the list experiment addresses misreporting driven by a preference for anonymity or fear of repercussions if one’s answer becomes known, as well as shame. Table 1 shows these methods by their approach to questioning and administration.

Table 1: Method comparisons by <u>questioning</u> method and administration

||Question & answer through enumerator|Question & answer through tablet|
|---|---|---|
|Direct questioning method|Face-to-face (status quo)|ACASI|
|Indirect questioning method|List experiment||

## **2.1 More Privacy: Audio Computer-Assisted Self-Interview**

ACASI is a method of data collection where respondents listen to pre-recorded questions through headphones and respond by selecting their answers (for example, the corresponding colored square) on a touchscreen or keypad.<sup>7</sup> After demonstrating how to use the technology, enumerators stand a small distance from the respondent and ensure that privacy is maintained.

ACASI eliminates the embarrassment that respondents may feel about disclosing a sensitive answer face-to-face with an enumerator, potentially minimizing underreporting (Falb et al., 2016; Jarlais et al., 1999; Jaya, Hindin and Ahmed, 2008; Langhaug, Sherr and Cowan, 2010). If shame is a key driver of underreporting, then reporting under ACASI should be greater than face-to-face. ACASI may also reduce enumerator-associated variance by providing a more uniform approach to data collection.

> 5Tourangeau and Yan (2007) define sensitive questions as those a respondent deems intrusive; where they fear the threat of disclosure and the possible consequences of giving a truthful response; or where there are social norms that imply there are socially desirable and undesirable responses.

> 6Palermo, Bleck and Peterman (2014) analyzed data from 24 developing countries DHSs and found that only 7 percent of women who report experiencing IPV made a report to a formal source that would then be captured in administrative data.

> 7Figure A1 in the appendix shows ACASI being used.

3
Several experimental studies have compared the prevalence of sensitive health behaviors between ACASI and face-to-face survey methods. Most found that respondents reported sensitive health behaviors and experiences, including IPV, at higher rates under ACASI than the face-to-face method, though the evidence is mixed (Fay et al., 1998; Jaya, Hindin and Ahmed, 2008; Langhaug, Sherr and Cowan, 2010; Mensch et al., 2008; Phillips et al., 2010; Fincher et al., 2015). In one of the few method comparisons with an objective indicator, in Brazil, Mensch et al. (2008) compared sexually transmitted infection bio-markers with face-to-face and ACASI self-reports of risky sexual behavior. They found more underreporting when respondents were interviewed face-to-face than with ACASI.

# **2.2 More Confidentiality and Anonymity: List Experiment**

Researchers have also used the list experiment to improve data accuracy when measuring sensitive behaviors and attitudes.<sup>8</sup> The standard list experiment works as follows: respondents are randomly assigned to one of two groups to answer the list question. In the ‘non-sensitive’ or ‘control’ group, enumerators read respondents a set of three non-sensitive statements and ask the respondent to tell them how many but not which statements are true for the respondent, with possible answers between zero and three. Meanwhile, the ‘sensitive’ or ‘treatment’ group is read the same list of statements, with the addition of a fourth sensitive statement. They are given the same instructions, to say how many, between zero and four statements, are true. Given the sensitive and non-sensitive list groups are randomly assigned, the two groups are, in expectation, equivalent and the only difference is the inclusion of the sensitive statement. The prevalence of the sensitive item across the whole sample is thus the difference in the mean number of items reported by the two groups.

Unlike with the ACASI and face-to-face methods, anonymity is assured because the lists are designed in such a way that very few people in the list sensitive group have all four statements as either true or not true.<sup>9</sup> This way, a respondent is not revealing their true answer to the sensitive item- to the enumerator or anyone with data access. This could mitigate underreporting by respondents concerned about consequences if someone were to find out their answers, or those who doubt the confidentiality of their data, or who find it easier to truthfully admit sensitive answers when asked indirectly.

Ex-ante, it is not clear whether shame or fear would be the biggest driver of misreporting on IPV survey questions. Assuming that responding in the affirmative is stigmatized, if the shame from answering a question face-to-face dominates other motivations for underreporting, then reporting is likely to be greatest under ACASI. If concerns about being identified or fear of potential repercussions dominate, then reporting would likely be greatest under the list experiment.

There are potential drawbacks to the list experiment. One challenge is that, by design, it cannot be used to identify individuals who respond affirmatively to the sensitive question of interest, potentially limiting its use where individuals need to be referred to support services, for example. Also, with list experiment answers ranging from zero to four, it typically has greater variance than direct questioning

> 8For example, Miller (1984) developed the list method to study illicit drug use. Others have used it to study sensitive topics including prejudice in the United States (Kuklinski, Cobb and Gilens, 1997), political preferences in Lebanon (Corstange, 2009), support for militant groups in Afghanistan (Blair, Imai and Lyall, 2014), risky sexual behavior (Chuang et al., 2018; Jamison, Karlan and Raffler, 2013; Lépine et al., 2020), support for female genital cutting (De Cao and Lutz, 2018), vote buying (Gonzalez-Ocantos et al., 2012), and use of microfinance loans (Karlan and Zinman, 2012).

> 9This can be done through careful piloting and selecting a set of statements that are all plausible but where the cooccurrence of all being true or untrue for any one person is unlikely. See Glynn (2013) and Corstange (2009) for a discussion and suggestions.

4
methods (but also likely lower misreporting bias because of respondent anonymity). Thus list experiments may reduce bias at the cost of efficiency (Blair and Imai, 2012).<sup>10</sup> Recently, researchers have begun using the list method to measure IPV and other forms of genderbased violence. In several studies, IPV rates are compared between the face-to-face and list methods. The study most closely related to this paper, conducted in Lima, Peru, found no difference in overall IPV prevalence by survey method (Agüero and Frisancho, 2017); studies in other contexts did find a difference. In Sri Lanka, Traunmüller, Kijewski and Freitag (2019) found the self-reported experience of wartime sexual violence (not IPV) to be higher under the list method than face-to-face, and in India, Joseph et al. (2017) found higher rates of IPV under the list method than face-to-face.<sup>11</sup> In these studies, respondents were either a) randomly assigned to answer questions about the same experience of IPV using the direct face-to-face method or the list method (Agüero and Frisancho, 2017; World Bank, 2017), or b) all respondents were asked about their IPV experience using both methods within the same survey- either with similar, but slightly different, question wording (Bulte and Lensink, 2019) or the same question wording (Joseph et al., 2017).<sup>12</sup> One issue with approach b) is that being asked the same question in two different ways may bias responses to the second question. We take approach a), asking individuals each question only once to minimize this risk, and we also test three methods with varying degrees of privacy and anonymity to provide additional insights into the reasons why people misreport.

# **2.3 Misreporting and truth**

Tourangeau and Yan (2007) argue that for sensitive questions where responding in the is taboo, the survey method that finds the highest prevalence is closest to truth. In this paper, particularly given the stigma surrounding the issue of IPV in the study contexts, we assume that in most cases, the method with the highest prevalence is closest to truth and that most misreporting is underreporting. In rare cases, people may have incentives to overreport, for example, if a respondent believed she would have a greater chance of accessing a program if she reports experiencing IPV. However, it seems likely that for most sensitive, stigmatized topics such as the perpetration and experience of IPV, people are more likely to under than overreport.

Without objective, non-self-reported data, we cannot, ultimately, which method is closest to truth. However, the two alternative survey methods tested in this paper are designed to address several reasons why people might underreport when interviewed with the face-to-face method- shame and fear.

> 10Blair, Coppock and Moor (2018) illustrate this trade-off in their review of the literature comparing the list method versus direct questioning experiments in social psychology and political science.

> 11Aside from the Nigeria study (Banerjee, La Ferrara and Orozco, 2019; World Bank, 2017), one other study in Africa has used the list method to measure IPV. Peterman et al. (2017) measured the prevalence of physical violence using the list method in Zambia. They found rates of IPV similar to those reported in the Zambian DHS, which used the face-to-face method, though the samples are not directly comparable.

> 12For example, in Bulte and Lensink’s (2019) Vietnam study, the sensitive list method group responded to a list that included the sensitive statement “I am regularly hit by my spouse,” then all respondents later in the survey were also asked “how often did your husband push, slap, beat or hit you during the last 6 months” with response options never, rarely, sometimes, often, very often, or refuse. With this design, the questions may not be directly comparable given potential differences in interpretation and biased responses to the second question. Similarly, Joseph et al. (2017) first asked all respondents using the list method, with the sensitive statement being: “At least one woman member of my household has faced physical aggression from her husband anytime during her life,” followed by all respondents directly being asked: “Has at least one woman member of your household faced physical aggression from her husband anytime during her life?” Further, this latter study departs from IPV research guidance, which advises asking about specific, unambiguous episodes of violence that minimize the risk of subjective interpretation (Heise and Hossain, 2017). The survey was also administered at the household level implying that respondents could be men or women, potentially including IPV perpetrators.

5
Further, the alternative methods are unlikely to introduce new reasons for people to report IPV that they have not experienced. This is particularly true for the list experiment, where answers are anonymous by design, hence individuals have no incentive to overreport given they cannot be followed-up with to provide aid or program services. Therefore, it seems likely that the method with the highest reporting indicates the method that best addresses respondents’ reasons for misreporting and is closest to truth.

This assumption is supported by empirical evidence from the only study we that compares an observed sensitive behavior with self-reported behavior measured using direct and indirect methods including the list experiment. Rosenfeld, Imai and Shapiro (2016) found that when they were asked directly, people underreported their past voting behavior on a sensitive abortion referendum in Mississippi. However, rates were higher and closer to truth when they were asked using the list experiment. No method yielded rates higher than the true ‘socially undesirable’ outcome of voting against tightened abortion restrictions. This study suggests that indirect methods such as the list experiment, which mask a respondent’s true answer, can reduce misreporting bias.<sup>13</sup> Overall, evidence on the measurement of sensitive topics suggests that misreporting appears to be highly issue and context dependent (Blair, Coppock and Moor, 2018; Langhaug, Sherr and Cowan, 2010; Tourangeau and Yan, 2007), presumably affected by factors such as the interview setting, cultural norms, security and safety, and confidence in data privacy. Therefore, this paper primarily relates to misreporting about intimate partner and sexual violence, and in contexts similar to those studied here.

# **3 Experimental design**

## **3.1 Context, data & design**

Rwanda and Nigeria have high rates of IPV compared to the global average. According to data gathered using the direct face-to-face method, 45 percent of Rwandan women and 31 percent of Nigerian women report having experienced physical or sexual IPV in the past 12 months (Global Burden of Disease, 2017). This compares to 13 percent of women in the United States and United Kingdom, 42 percent in Kenya, 39 percent in India, and 31 percent in South Africa. In terms of IPV predictors, poorer, less educated women report IPV at higher rates in Rwanda, while richer, more educated women report more IPV in Nigeria (DHS, 2016, 2014).

Both experiments described below were conducted in accordance with WHO guidelines on the safe and ethical collection of data on violence against women (WHO, 2001, 2016).

### **3.1.1 Rwanda**

In Rwanda, the measurement experiment was conducted on a sample of 2,728 heterosexual couples who were participating in the baseline survey of an IPV prevention study. To be eligible for the intervention, couples had to be aged 18 years or over and married or cohabiting for at least 12 months. The study took place in Eastern Province, in eight sectors selected by the Government of Rwanda for high rates of IPV and a high concentration of Village Savings and Loan Associations (VSLAs) (which the intervention was

> 13It also demonstrates that experiments that compare reporting rates across methods can put a sign and bounds on the likely misreporting bias of standard survey methods.

6
to be delivered through).<sup>14</sup> The sample comprised mostly poor rural households, with women’s mean age of 37 and a median of 4 years of schooling.

For the measurement experiment, individual respondents were randomly assigned to answer two experimental questions using one of the three survey methods: face-to-face, ACASI, or list experiment. The sample was equally split across the three groups.

The two experimental questions were asked approximately one hour into the survey, immediately after ACASI-administered questions from the modified Conflict Tactics Scale about all respondents’ experiences of IPV in the past 12 months (for women), and the likelihood of becoming violent in a set of hypothetical circumstances (for men).<sup>15</sup> Respondents were then asked the experimental questions according to the method they were assigned. The questions were: Women:

1. In the past 12 months, has your husband pushed or thrown you against something like a wall or piece of furniture?

2. Has someone other than your current husband ever forced you to have sex with them when you did not want to?

Men:

1. In the past 12 months, have you tried to limit your wife’s contact with her family?

2. In the past 12 months, have you threatened to hurt your wife or someone close to her?

The surveys were administered by enumerators the same sex as the respondent, in private settings where interviews could not be overheard. Most surveys were conducted in empty school classrooms, with one enumerator and one respondent per classroom.<sup>16</sup> The empty classroom interview sites afforded a more private setting than is achieved in many DHS and other household survey settings. Therefore, if a lack of privacy causes women to underreport, then the prevalence differences seen in this more private study setting between face-to-face and the two more private methods are likely to be smaller than the differences that might be observed in standard interview settings.

> 14Sectors are administrative units two levels below province, and 2 levels above village. Within the selected sectors, existing VSLAs were identified in 128 villages. The researchers ruled out 30 villages with fewer than three VSLAs each, and villages that were very close. This was done to minimize the risk of cross-village contamination for the separate randomized controlled trial, detailed in the RCT pre-analysis plan (Alik-Lagrange et al., 2017). Within the 98 study villages, the final sample included 2,042 intervention-eligible couples where at least one partner was a member of a VSLA, and 686 couples who were community members selected at random from a village household roster and who were ineligible for the intervention. VSLA couples’ treatment status was assigned by the research team only after baseline surveys were completed. Assignment was based on their ranking in a public lottery process that they applied to. The intervention, called ‘Indashyikirwa 2’, is a weekly couples group discussion that encourages couples to question traditional gender norms and build communication skills. Data were collected from December 2017 to February 2018.

> 15When all respondents reached the main section on IPV (which was to be asked through ACASI), they were shown how to use the ACASI module on the tablet and then instructed through practice questions. After the enumerator was satisfied that the respondent could use ACASI, respondents were left to answer the module while the enumerator stood approximately 5 meters away. Over 95% of female respondents and 98% of male respondents got the first ACASI practice question correct. After being assisted again, 99% of male and female respondents were able to answer the ACASI module in a way that maintained their privacy but with some support, for example, if they could not touch the tablet screens, they listened in the headphones and told the enumerator which button to select for them. The IPV ACASI questions asked of all respondents will be used in the randomized controlled trial. For men, the two experimental questions were the only IPV perpetration questions asked in the survey. The survey themes and order of questions are listed in Table A1 and Table A2.

> 16When classrooms were not available, surveys were conducted in empty offices, churches, or outside but strict privacy protocols were adhered to, including the requirement that interviews pause if someone else was within earshot or eyesight.

7
## **3.1.2 Nigeria**

In Nigeria, the measurement experiment was conducted during the endline survey wave of a cluster randomized control trial studying the effects of the Feed the Future Nigeria Livelihoods Project, a multicomponent cash transfer, nutrition, and agriculture program.<sup>17</sup> The survey took place in Northwest Nigeria on the border region with Niger. The sample consisted of ultra-poor households, with approximately 30 percent in polygamous marriages, 84 percent identifying as Muslim, and 15 percent of women self-reporting as literate.

The eligibility criteria for the measurement experiment were that the respondent was female, over 17 years of age, and married and living in the same household as a male. In total, 2,817 women met these criteria and were randomly assigned to answer three experimental questions about their experience of IPV using either the face-to-face or list method. Assignment to the method groups was stratified on the same geographic and poverty-related strata used to assign respondents to treatment arms for the randomized control trial.

The three IPV questions used for the measurement experiment were either directly from or combined several violence questions from the DHS.<sup>18</sup> The experimental questions were:

1. In the past 12 months, has your husband said or done something to humiliate you in front of others?

2. In the past 12 months, has your husband slapped you, pushed you, shaken you, or thrown something at you?

3. In the past 12 months, has your husband physically forced you to have sexual intercourse with him when you did not want to?

The respondents were interviewed one-on-one by a female enumerator in a private setting, usually in the household compound. Enumerators were instructed to pause if anyone came within earshot of the interview, not proceeding until privacy was assured.<sup>19</sup>

# **3.2 The measurement experiment**

We assess misreporting bias by comparing rates of IPV and sexual violence across three survey methods: face-to-face, list experiment, and ACASI (with the latter only used in Rwanda). Given that assignment to the survey method was random, we assume that the only difference in reported violence between the groups is due to the method used.

## **3.2.1 Estimation: Calculating prevalence under each survey method**

Prevalence under the ACASI and face-to-face methods, where the sensitive item is asked directly, is simply the mean proportion of respondents who said they had experienced violence.

> 17The study evaluated the impacts of a package of livelihoods interventions, unconditional cash transfers, and a mentoring program, and was implemented by Catholic Relief Services (Bastian, Goldstein and Papineni, 2017). The cash transfers and most livelihoods programming ended in 2017, approximately 12 months before these data were collected.

> 18Female respondents were first asked about their experience of IPV using the list method (asked either the sensitive or non-sensitive list, depending on her randomly assigned group), followed by direct face-to-face IPV questions. The direct questions included the three measurement experiment questions (asked only to those in the face-to-face assigned group), and five face-to-face questions asked of all respondents (to be used as outcome measures for the main randomized trial).

> 19Occasionally this was difficult to achieve in busy household compounds, for example, when household members came to the house to collect items, causing enumerators to pause interviews. Data were collected from May-July 2018.

8
Under the list method, prevalence of the sensitive statement is calculated by taking the in the mean number of statements reported as true by the two groups (Blair and Imai, 2012). There are two approaches to administering the list to the non-sensitive group. Under the original approach, which we used in Nigeria, respondents are read the same list, bar the fourth sensitive statement, and asked to report how many of the three statements apply to them. Under the alternative design proposed by Corstange (2009), which we used in Rwanda, the non-sensitive group is read out each non-sensitive statement and asked directly whether or not each was true for them, yes or no. Individuals’ responses were then summed to create an equivalent count of three non-sensitive statements per respondent, from which prevalence can be estimated in the same way, by calculating the difference in means.<sup>20</sup> To help respondents keep track of their answers, they were advised to put their behind their back or under their hijab out of sight of the enumerator and to straighten a finger each time a true statement was read out. Once all statements had been read out, respondents showed the enumerator their hand to count the number of straightened fingers.<sup>21</sup> Prevalence under the list method was calculated using a multivariate regression model, following the approach of Agüero and Frisancho (2017). The OLS regression estimated was as follows,

# _Yi_ = _α_ + _ρTi_ + _εi_

where _ρ_ is the of interest, the in the mean number of items reported across the two groups, or prevalence. _Yi_ is the number of statements on the list that are true for individual _i_ , _α_ is the mean number of items for the non-sensitive group, and _Ti_ is assignment to the list method treatment group.<sup>22</sup> The list non-sensitive group comprised the share of the sample that answered the experimental questions using ACASI or face-to-face. Table 2 shows the method assignment and group size.<sup>23</sup> Table 2: Method assignment group size

||List experiment|Face-to-face|ACASI|
|---|---|---|---|
|Answer type|Indirect|Direct|Direct|
|Rwanda||||
|_N_: women|873|918|937|
|_N_: men|877|909|942|
|Nigeria||||
|_N:_women|1,405|1,412||

Overall, both samples were well balanced on key observable characteristics. Results in Tables A5 to

> 20We used Corstange’s approach in Rwanda because of early concerns about statistical power for conducting multivariate analysis, and this alternative approach allows an associated estimator that can be more efficient than OLS. Given the different construction of the lists, this alternative design potentially poses the risk that respondents do not answer in the same way across the sensitive and non-sensitive lists.

> 21In Rwanda, respondents also put their hands behind their backs and transferred a stone from one hand to the other each time a true statement was read out.

> 22Power calculations indicated that the sample in Nigeria was large enough to detect a difference of approximately 6 percentage points between the groups, and in Rwanda, approximately 5 percentage points, with a significance level of 0.05, power of 0.8, and base prevalence of 20 percent.

> 23Table A3 shows the wording of the questions by method that were asked in Rwanda. The non-sensitive questions that were asked in the list experiments are shown in Table A4. These non-sensitive questions were selected from a larger pool of piloted questions following design advice discussed in Glynn (2013), and informed by questions used by Peterman et al. (2017) and Agüero and Frisancho (2017).

9
A8 show balance across the groups assigned to the list sensitive and list non-sensitive groups, and in Rwanda, also across the three groups (ACASI versus face-to-face versus list sensitive). While several of the raw differences are statistically significant at conventional levels, the differences do not appear to be meaningfully significant, they are within the bounds of what might be expected given the number of variables tested for balance, and no normalized differences are greater than 0.25 standard deviations, a suggested rule of thumb indicating good balance (Imbens and Rubin, 2015). In additional specifications, we estimate prevalence with regressions that include controls selected using post-double selection LASSO (Belloni, Chernozhukov and Hansen, 2014). This does not meaningfully change results.

# **4 Results**

## **4.1 Method matters: IPV reporting differs by the survey method used**

between the three survey methods. A difference in prevalence implies misreporting under at least one of the methods.<sup>24</sup> In Table 3, columns 1-3 show the prevalence of sexual and intimate partner violence by survey method in Rwanda and Nigeria. Columns 4-6 show the difference in prevalence between pairs of methods.<sup>25</sup> The p-values for these prevalence difference tests were adjusted for multiple hypothesis testing. Sharpened q-values were computed using the approach proposed by Benjamini and Hochberg (1995) and Anderson (2008).<sup>26</sup> Results are also shown graphically in Figure 1.

Results demonstrate that the survey method used to measure sexual and intimate partner violence can substantially affect reporting. Overall, the most widely used survey method, face-to-face interviewing, resulted in the lowest reported IPV rates, followed by ACASI, and finally the list method with the highest. Women’s reports of experiencing physical IPV in Rwanda were double when measured using the indirect, anonymous list method (20.6 percent) compared to the two direct methods (9.3 percent with face-to-face and 10.3 percent with ACASI). The differences between the list method and each of the direct methods were statistically significant, but the difference between ACASI and face-to-face prevalence was not.

In Nigeria, emotional and physical IPV rates were 33 and 39 percent greater respectively when measured using the list method (emotional violence: 39.7 percent; physical violence: 26.6 percent) compared to face-to-face (emotional violence: 29.9 percent; physical violence: 19.2 percent). The difference between list and face-to-face emotional violence prevalence was statistically significant at the 5 percent level. For physical violence, the difference was statistically significant at the 10 percent level. There was no statistically significant difference between direct and indirect rates of non-partner sexual violence in Rwanda or sexual IPV in Nigeria. Reported prevalence for these sexual violence questions was between 5.1 to 8.8 percent in Rwanda and 26 to 28.1 percent in Nigeria.

For men in Rwanda, the same pattern across methods was observed, with 13.3, 32.6, and 45.1 percent

> 24As discussed above, we assume that in most cases, the method with the highest reported prevalence is closest to truth. This assumption is based on the results of several studies that compare observed truth with self-reported sensitive behavior, and an assumed lack of incentive to overreport IPV, particularly under the list experiment.

> 25Non-response rates were extremely low in both contexts. For all methods, refusal rates were no greater than 0.84% for women and 2.4% for men. Non-response rates are shown in Table A16.

> 26The q-values control for the false discovery rate for the 12 tests for difference in means that we calculate in Rwanda and for the three in Nigeria. All list method estimates were calculated with heteroskedastic-consistent robust standard errors because the error variance necessarily depends on the list group assignment (Blair and Imai, 2012).

10
of men admitting they had tried to limit their wife’s contact with her family, with the all significant at the 1 percent level. These large differences between the methods suggest that, for one reason or another, men believe that they should report that they do not limit their wife’s contact with her family. The added privacy of ACASI addresses some of this underreporting, but even more is addressed by the list method’s anonymity. For the second men’s question about threatening behavior, prevalence was much lower and although the pattern held, the differences were not statistically significant.

Table 3: Prevalence of sexual and intimate partner violence by survey method

|||||Differe|nce between<br>methods|pairs of|
|---|---|---|---|---|---|---|
|_Rwanda_|(1)<br>Face-to-<br>Face<br>mean|(2)<br>ACASI<br>mean|(3)<br>List method<br>prevalence<br>(difference in<br>means)|(4)<br>List-<br>F2F<br>(3)-(1)|(5)<br>List-<br>ACASI<br>(3)-(2)|(6)<br>ACASI-<br>F2F<br>(2)-(1)|
|Women|||||||
|Q1: husband pushed/ thrown you against|0.093***|0.103***|0.206***|0.113***|0.103**|0.01|
|something in past 12 months|(0.010)|(0.010)|(0.035)|[0.007]|[0.014]|[0.552]|
|_N_|918|914|2,728||||
|Q2: someone other than current husband ever|0.051***|0.082***|0.088***|0.037|0.006|0.031**|
|forced you to have sex when didn't want to|(0.007)|(0.009)|(0.034)|[0.443]|[0.872]|[0.017]|
|_N_|917|914|2,727||||
|Men|||||||
|Q1: tried to limit wife's contact with her|0.133***|0.326***|0.451***|0.318***|0.125***|0.193***|
|family in past 12 months|(0.011)|(0.016)|(0.038)|[0.001]|[0.007]|[0.001]|
|_N_|909|877|2,728||||
|Q2: threatened to hurt wife or someone close|0.041***|0.059***|0.071**|0.03|0.012|0.018|
|in past 12 months|(0.007)|(0.008)|(0.036)|[0.542]|[0.816]|[0.113]|
|_N_|909|926|2,728||||
|_Nigeria_|||||||
|In the past 12 months, has your husband:|||||||
|Q1: said or done something to humiliate you|0.299***||0.397***|0.098**|||
|in front of others|(0.012)||(0.032)|[0.015]|||
|_N_|1,394||2,817||||
|Q2: slapped you, pushed you, shaken you, or|0.192***||0.266***|0.074*|||
|thrown something at you|(0.011)||(0.043)|[0.083]|||
|_N_|1,394||2,817||||
|Q3: physically forced you to have sex even|0.281***||0.260***|-0.017|||
|when you did not want to|(0.012)||(0.031)|[0.551]|||
|_N_|1,381||2,817||||

_Note:_ The left column shows the violence variables. Columns 1, 2 and 3 show prevalence by method. Standard errors are shown in parentheses. Columns 4, 5 and 6 shows differences in prevalence between pairs of methods. For the test of difference in means, Benjamini Hochberg (1995) adjusted q-values are displayed in square brackets, from a chi-square test of the differences in means across groups. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. Nigeria regressions are adjusted for sampling stratification. List estimates show robust standard errors.

Overall, these results suggest that women’s experience and men’s perpetration of emotional and physical IPV may be substantially underreported when measured using face-to-face or ACASI methods. In Rwanda, where we tested three methods, the smaller differences between ACASI and face-to-face prevalence suggest that it is primarily the list method’s anonymity rather than ACASI’s privacy that makes people more likely to report IPV. If shame were the biggest driver of misreporting, it seems likely 11
<!-- Start of picture text --> Physical | Method<br>a Bi List<br>-g = ACASI<br>= Non-partner a @ FOF<br>< sexual a<br>;—<br>Smotional -1<br>=|<br>E<br>a |<br>Emotional - Q2 YF<br>m7<br>|<br>Emotional a<br>< . ee<br>= £ Physical |<br>“25, 2= Sexual laone<br>PS<br>0 5 10 15 20 25 30 35 40 45<br>Violence prevalence (%)<br><!-- End of picture text -->
stakes for women who having experienced IPV- which is illegal and husbands could be and jailed if found guilty- versus confirming experience of non-current partner sexual violence - which may feel shameful to admit face-to-face but does not carry the perceived risk that one’s husband might be jailed. This would imply that answer anonymity improves truthful reporting for sensitive questions where the perceived legal repercussions may be large, but methods like ACASI that improve privacy do not help address such misreporting. Instead, ACASI’s added privacy may primarily increase truthful reporting on sensitive questions when there is no risk of negative legal repercussions if one’s answer is discovered.

# **4.2 Characteristics of misreporting**

In this section, we consider whether misreporting is random, and if not, identify the characteristics that predict it. To do this, we assess correlations between respondent characteristics and IPV, as reported under each of the survey methods. When the IPV-characteristic coefficient differs across methods, this implies systematic misreporting under at least one of the methods that is associated with the characteristic. This analysis provides suggestive evidence on the type of person that might misreport, potential reasons for misreporting, and the risks of drawing biased descriptive and causal inferences from such data.

In this section, we limit analysis to women’s experience of IPV given that there may be systematically different reasons for women to misreport on non-partner sexual violence questions and given that men’s self-reported perpetration is generally considered less reliable.

## **4.2.1 Estimation: Characteristics associated with misreporting**

For those assigned to the face-to-face or ACASI groups, we assess the characteristics of those reporting IPV by regressing women’s self-reported experience of IPV against the variable of interest, such as years of education, as below.

For the list method, we regress the number of list items reported by an individual against an indicator variable for whether they were in the list sensitive group, as well as the variable of interest and the interaction term. This regression model is shown below, where _Yi_ is the number of true statements for individual _i_ , _Ti_ is assignment to the sensitive list method group, and _δ_ is the parameter of interest.<sup>27</sup> If there is no (or consistent) misreporting across the methods, the relevant coefficient for each method should be equal. If there are differences, this implies that the relationship between IPV and the characteristic is sensitive to the measurement method, and we may be estimating biased descriptive and causal relationships. For example, if we find that IPV rates are lower for educated women when we use face-to-face data, but higher for educated women when we use list method data, then we can conclude that misreporting bias is correlated with women’s education. To

> 27This approach follows that of Holbrook and Krosnick (2010), Coutts and Jann (2011), and Agüero and Frisancho (2017). While linear regression may be less efficient than other multivariate statistical estimators proposed for list experiments by Corstange (2009), Imai (2011), Blair and Imai (2012), and Imai, Park and Greene (2015), it is simpler to interpret and compare relevant coefficients across methods and specifications for a descriptive analysis.

13
better understand the characteristics of those who misreport, we also compare mean IPV prevalence for respondents above and below the characteristic median (for numeric variable characteristics) or by their indicator variable characteristics.

# **4.2.2 Results: Respondent characteristics predict misreporting**

We analyze misreporting by respondent characteristics that fall into three overlapping categories. These characteristics include: 1) those most likely to motivate respondents to misreport (to understand why people misreport IPV); 2) variables typically targeted in IPV prevention interventions (to evaluate risks of estimating biased treatment effects); and 3) characteristics frequently cited as IPV risk factors (to assess potential challenges to IPV-related theory that has been informed by analysis of face-to-face survey data). The first two categories of characteristics include perceptions of village gender norms; poverty and vulnerability; marital relationship quality; and personal gender attitudes. Additional IPV risk factors considered include women’s education and literacy, age, labor force participation and other indicators of women’s bargaining power, where theory and empirical evidence are mixed on the relationship with IPV (Bobonis, González-Brenes and Castro, 2013; Buller et al., 2018; Erten and Keskin, 2018; Heise and Kotsadam, 2015; Haushofer et al., 2019; Eswaran and Malhotra, 2011).

Tables 4 and 5 show the relationships between IPV and respondent characteristics by survey method. The first 3 (6) columns show the relevant regression coefficient of interest for Rwanda (Nigeria), and the last 3 columns show the results of chi-squared tests of the coefficient differences between pairs of methods. Each coefficient shown in the tables was estimated in a separate regression. These tables show several significant differences in coefficients across the methods, implying that a number of characteristics are associated with systematic misreporting of IPV. These results are also shown graphically in Figures A2 and A3. We also show the differences in mean IPV prevalence for respondents above and below the median of the characteristic in Tables A9 and A10 and graphically in Figures A4 and A5.

This subsection presents the results of descriptive analysis and as such, usual caution should be applied when interpreting results given that omitted variables or other sources of bias may be driving observed correlations. Given this is exploratory analysis only, we do not adjust the p-values of tests of equivalence for multiple hypothesis testing.<sup>28</sup> These results show that in both countries, vulnerable women report IPV at lower rates when they are asked about their experience using direct methods than when interviewed using the list method. Specifically, in Rwanda, a one standard deviation increase in a woman’s vulnerability index was associated with a 6 percentage point increase in reported experience of IPV when violence was measured face-toface, a 5 percentage point increase when measured using ACASI, and a 23 percentage point increase when measured using the list method. In Nigeria, vulnerable women were also more likely to report physical IPV under the list method than face-to-face. For emotional and sexual IPV, the same pattern existed but the differences were not statistically significant. Table A9 shows that, in Rwanda, as we might expect, the difference between IPV reporting when it was measured using the list and direct methods was driven

> 28For conceptually linked variables (for example, questions about relationship quality such as trust and empathy in the marriage), we generated weighted standardized summary indices (Anderson, 2008) to minimize the false discovery rate. For other indicators of interest where there is less theoretical coherence between the variables, we report coefficients on the raw variable. For instance, it is not clear that the following indicators of women’s bargaining power are measures of the same construct, nor whether they would each induce a respondent to underreport IPV more or less: education, asset ownership, age, marital status, and engagement in paid employment.

14
Table 4: Rwanda- correlations between IPV and respondent characteristics across methods

||Q: husband|pushed or t|hrown you|Method<br>t|differences: P-<br>ests of differen|values from<br>ces|
|---|---|---|---|---|---|---|
|Variable|(1)<br>List|(2)<br>F2F|(3)<br>ACASI|(1)-(2)<br>List-F2F|(1)-(3)<br>List-ACASI|(3)-(2)<br>ACASI-F2F|
|Vulnerability index|0.23***<br>(0.08)|0.06***<br>(0.02)|0.05*<br>(0.03)|0.07|0.05|0.65|
|Unequal village gender norms index|0.37***<br>(0.10)|0.03<br>(0.03)|0.04<br>(0.03)|0.00|0.00|0.75|
|Relationship quality index|0.03<br>(0.05)|-0.07***<br>(0.01)|-0.04***<br>(0.01)|0.06|0.17|0.18|
|Gender engagement index|-0.10**<br>(0.04)|-0.02*<br>(0.01)|-0.01<br>(0.01)|0.11|0.07|0.48|
|Progressive gender attitudes index|0.01<br>(0.06)|-0.01<br>(0.02)|-0.03<br>(0.02)|0.75|0.56|0.47|
|B**argaining power measures**|||||||
|Legally married|-0.05<br>(0.07)|0.01<br>(0.02)|-0.03<br>(0.02)|0.49|0.83|0.21|
|Wife's age|0.00<br>(0.00)|0.00<br>(0.00)|0.00<br>(0.00)|0.43|0.34|0.68|
|Wife owns land|0.03<br>(0.08)|0.00<br>(0.02)|-0.03<br>(0.02)|0.72|0.49|0.36|
|Wife's years of education|0.02<br>(0.01)|-0.00<br>(0.00)|-0.01*<br>(0.00)|0.13|0.06|0.27|
|Wife in paid work last week|-0.15**|-0.00|0.03|0.04|0.02|0.33|
||(0.07)|(0.02)|(0.02)||||

**Note:** Standard errors are shown in parentheses (List method estimates show robust standard errors).  *** p<0.01, ** p<0.05, * p<0.1. Columns 1-3 show the relevant covariate coefficients from separate regressions. Columns 4-6 show the p-values from a chi-squared test of the differences between these coefficients. The vulnerability index includes household economic shock experience, husband's trauma score, husband's alcohol consumption, and whether any household member has skipped a meal in the past week. The unequal gender norms index includes incentivized and unincentivized questions about perceived unequal gender attitudes held by others in their village. The relationship quality index includes measures of marital trust, empathy and communication. The gender engagement index includes questions on the wife's previous participation in rallies and workshops about family violence and gender equality. The gender attitudes index includes questions about respondent beliefs about gender roles and the acceptability of IPV.

by women above median vulnerability. There was no IPV reporting across the list and direct methods for the least vulnerable women.

More vulnerable women may face greater personal safety or economic security risks if they report that their husband perpetrates IPV and, for example, he discovered her answer or was jailed as a result. For such women, the list experiment’s anonymity may provide them with additional reassurance and safety, making them more willing to report IPV. These results demonstrate that the relationship between vulnerability and IPV can be very different if estimated using anonymous list experiment IPV data. In Rwanda, it also demonstrates that ACASI’s added privacy does not address the reasons vulnerable women are misreporting.

In Rwanda, we also observe a similar misreporting pattern based on a respondent’s perception of unequal village gender norms, which could proxy for shame about IPV or fear of potential repercussions if a woman’s survey answer were to be discovered. When a woman perceived that village gender norms were unequal, she was more likely to report IPV under the list method, where her answer is anonymous, than she was under the face-to-face or ACASI methods. Specifically, in Rwanda, a one standard deviation 15
Table 5: Nigeria- correlations between IPV and respondent characteristics across methods

||Q1: emoti|onal violence|Q2: physic|al violence|Q3: sexu|al violence|Method<br>from|difference<br>tests of di|s: P-values<br>fferences|
|---|---|---|---|---|---|---|---|---|---|
|Variable|(1)<br>List|(2)<br>F2F|(3)<br>List|(4)<br>F2F|(5)<br>List|(6)<br>F2F|(1)-(2) <br>Q1|(3)-(4)<br>Q2|(5)-(6)<br>Q3|
|Vulnerability index|0.02|-0.08***|0.09|-0.05***|0.03|-0.05**|0.14|0.05|0.25|
||(0.06)|(0.02)|(0.07)|(0.02)|(0.07)|(0.02)||||
|Unequal village gender norms index|0.06|0.08***|0.09*|0.05***|0.08*|0.05***|0.61|0.43|0.50|
||(0.04)|(0.02)|(0.05)|(0.02)|(0.04)|(0.02)||||
|Relationship quality index|-0.07|-0.21***|-0.02|-0.21***|-0.04|-0.27***|0.11|0.04|0.01|
||(0.08)|(0.03)|(0.09)|(0.02)|(0.08)|(0.03)||||
|Public speaking confidence index|-0.02|-0.08***|0.01|-0.03***|0.00|-0.01|0.11|0.34|0.68|
||(0.04)|(0.01)|(0.04)|(0.01)|(0.04)|(0.01)||||
|Progressive gender attitudes index|-0.12*|-0.13***|-0.23***|-0.09***|-0.16**|-0.16***|0.91|0.11|0.95|
||(0.07)|(0.03)|(0.08)|(0.03)|(0.07)|(0.03)||||
|**Bargaining power measures**||||||||||
|Wife's age|0.00|0.00**|0.00|0.00|0.00|-0.00|0.57|0.49|0.59|
||(0.00)|(0.00)|(0.00)|(0.00)|(0.00)|(0.00)||||
|Wife reads and writes|0.12|-0.08**|0.00|-0.12***|0.11|-0.04|0.03|0.20|0.12|
||(0.08)|(0.03)|(0.10)|(0.02)|(0.09)|(0.03)||||
|Polygamous: wife number (1-4)|-0.03|0.02|0.06|0.03|0.13|-0.02|0.56|0.74|0.13|
||(0.08)|(0.04)|(0.10)|(0.03)|(0.09)|(0.03)||||
|Wife brought assets to marriage|-0.04|-0.18***|0.03|-0.13***|-0.02|-0.09***|0.04|0.03|0.34|
||(0.06)|(0.02)|(0.07)|(0.02)|(0.07)|(0.02)||||

**Note:** Standard errors are shown in parentheses (List method estimates show robust standard errors). All regressions are adjusted for sampling stratification. *** p<0.01, ** p<0.05, * p<0.1. Columns 1-6 show the relevant covariate coefficients from separate regressions. Columns 7-9 show the p-values from a chi-squared test of the differences between these coefficients. The vulnerability index includes measures of food poverty and household economic insecurity, and wife's stress. The unequal village gender norms index includes beliefs about unequal gender attitudes held by others in the village. The relationship quality index includes measures of  trust, satisfaction, empathy and communication in the marriage. The public speaking confidence index includes measures of willingness to speak up in public. The progressive gender attitudes index includes measures of personal views on the acceptability of IPV and on gender roles.

increase in a woman’s perception of unequal village gender norms was associated with a 3 percentage point increase in the probability of experiencing IPV when violence was measured face-to-face, a 4 percentage point increase when measured using ACASI, and a 37 percentage point increase when using the list method. However, the positive conservative norms-IPV relationship was only statistically significant when IPV was measured using the list method. In Rwanda, the differences between the list and direct methods were statistically significant; in Nigeria, they were not.<sup>29</sup> Table A9 shows that the gender norms-IPV relationship difference between the methods was driven by the respondents who believed their village had the most conservative norms. There was no IPV reporting difference across the list and direct methods for respondents who believed their village had progressive norms. As with vulnerability, these results demonstrate that the relationship between community gender norms and IPV can be very different if estimated using anonymous list experiment data.

> 29In Nigeria, the lack of IPV reporting difference across the methods by perceived norms may be due to the different gender norms questions that comprise the index in the two countries. The Nigerian norms index included only unincentivized questions about perceptions of other villagers’ attitudes regarding the acceptability of women working or traveling outside the home. In Rwanda, the norms index focused on IPV-specific norms and included both unincentivized questions about perceptions of village IPV prevalence and others’ attitudes, as well as four incentivized norms vignettes based on an adapted version of the Krupka-Weber norms coordination game (Krupka and Weber, 2013). In this game, respondents were paid if they selected the modal answer in their village about whether described behaviors about IPV and women leaving violent husbands were considered socially acceptable or unacceptable in their village. This incentive-compatible measure was intended to overcome social desirability bias.

16
The Rwandan result challenges an alternative hypothesis about norms and IPV misreporting which posits that in villages with highly unequal gender norms, where IPV is normalized, women may feel more able to openly report IPV. If that were the case, we should have observed no difference between the coefficients for face-to-face-estimated IPV and the two more private or anonymous methods. Instead, in Rwanda at least, it appears that only the anonymous method, but not ACASI, made vulnerable women and women living in conservative communities comfortable enough to report IPV.

Greater marital relationship quality, measured using questions about satisfaction, empathy, and trust in marriage, was also associated with underreporting. In both countries, women who reported greater relationship quality were more likely to report IPV when interviewed using the list method than face-toface; the method differences were statistically significant on three out of the four IPV questions considered. A potential reason for women who report being in a more trusting, supportive relationship to underreport IPV when asked directly (but not indirectly) is because the direct question forces her to confront an uncomfortable truth about her marriage, or she may feel disloyal by directly admitting her husband has been violent. As Tables 4 and 5, and the associated figures A2 and A3 show, when IPV is measured using the list experiment, there is no relationship between IPV and relationship quality; it is only when using direct methods that we find a negative relationship between IPV and marital quality.

Several of these results highlight a potential measurement bias issue for the growing number of policies and interventions being assessed on their IPV impact. For example, an intervention that improves relationship quality- as several IPV prevention programs seek to do- may have no impact on actual IPV rates. However, it may lead to a reduction in a wife’s willingness to directly report IPV for the reasons discussed above. This would bias treatment effect estimates, suggesting a reduction in IPV that was, in fact, an artefact of the measurement method used. Alternatively, if a policy led to a worsening in community gender norms, IPV outcomes that were measured using direct methods may not detect a resulting actual increase in IPV if women are less willing to report it. In both of these cases, list experiment-measured IPV may be less prone to reporting bias and more likely to reflect true treatment effects.

We also examined whether characteristics that are proposed as causes or common correlates of IPV remain stable or at least with the same sign on the relationship, regardless of the method used, or whether the nature of the relationship depends on the measurement method. Across different indicators of women’s intrahousehold bargaining power, we find some evidence of systematic misreporting of IPV, though no clear pattern across different indicators. For example, in Rwanda, we find a strong relationship between employment and IPV misreporting. Results in Table A9 show that this reporting effect is driven by a large difference in reporting for women who are not engaged in paid employment. Similar to the norms and vulnerability results, this may suggest that women with the weakest bargaining power or outside options underreport the most when asked directly about their IPV experience. Again, the list experiment appears to make these women more willing to report IPV.

Potentially most concerning, we observe some opposite-signed relationships, where the sign on the relationship between IPV and the respondent’s characteristic is positive for one measurement method and negative for another method. In Rwanda, these variables include wife’s years of education, relationship quality, and engagement in paid work, and in Nigeria, wife’s vulnerability, literacy, and assets brought into the marriage. For example, in both Rwanda and Nigeria, there is suggestive evidence of a positive relationship between IPV and women’s education when IPV is measured directly, but a negative 17
relationship when using the list method. literate Nigerian women are 12 percentage points more likely to experience emotional IPV if it is measured using the list method, and 8 percentage points less likely if IPV is measured face-to-face. This is similar to results in Peru and India, where researchers found women’s education to be positively correlated with IPV when measured using the list method but negatively when measured face-to-face (Agüero and Frisancho, 2017; Joseph et al., 2017). The oppositesigned IPV-education associations imply opposing theories about women’s bargaining power as a cause of IPV, highlighting the importance of minimizing measurement bias to better understand IPV causes, theory, and prevention.

Overall, the sensitivity of the relationships between IPV and key bargaining power and other indicators to the survey method used is concerning. It suggests that caution should be applied when using direct IPV measures to understand descriptive and causal relationships as misreporting may bias correlations and treatment effect estimates. The anonymity of the list experiment may minimize this reporting bias.

# **4.3 Robustness checks**

In the section below, we report the results of robustness checks on the list method, followed by checks on the overall analysis.

## **4.3.1 Checks on list method assumptions**

There are two assumptions that underpin the validity of statistical analyses of list experiments, and which, if they hold, should result in an unbiased difference-in-means estimator (Blair and Imai, 2012). The first assumption is that of ‘no design effect’, meaning that the inclusion of the sensitive item in the list does not affect answers to non-sensitive items. The second assumption is that of ‘no liars’, or that the answers given for the sensitive item are truthful.

is violated. The test is based on the assumption that the inclusion of a sensitive statement should not change the sum of affirmative answers to the non-sensitive statements, and that cumulative proportions of different respondent types should be non-negative (Bulte and Lensink, 2019). If one of the proportions is negative, then the no design effect assumption has likely been violated. The rejection of the null hypothesis of no design effect using Blair and Imai’s statistical test, made available on the open-source software R (Blair and Imai, 2011), would suggest there was a design effect. The test returns a Bonferronicorrected minimum p-value of the test.

In Nigeria, the test returned a minimum p-value of 1, suggesting there was no design for the three list questions. In Rwanda, where the list non-sensitive group was directly asked each non-sensitive question, there is statistically significant evidence of a design effect for three out of the four questions, suggesting that group assignment affected answers to the non-sensitive questions. There was no design effect for the first men’s question.

What then is the potential direction of bias introduced by this assumption violation in Rwanda? In the three cases in Rwanda where this assumption was violated, the design effect is likely to have resulted in downwards-biased prevalence estimates for the list method. This is because the tests that it failed, assessing whether the cumulative proportion of responses in the non-sensitive group was greater than the proportion in the sensitive list group, were for the share of respondents reporting that zero items were true 18
for them. More respondents in the list sensitive group reported zero than in the non-sensitive group. This implies that the mean in the list sensitive group is likely to be downwards biased, implying the same for the difference-in-means estimate and the resulting IPV prevalence.<sup>30</sup> Further, the one Rwanda question that did not fail the no design effects test showed the greatest difference in prevalence between the list and other methods. It therefore seems plausible that, if biased, the three Rwandan list experiments that failed the no design effects test produced downwards biased prevalence estimates. The associated test p-values and proportions are shown in Tables A11 and A12.

There is no explicit test for the second ‘no liars’ assumption underpinning the list experiment. However, the most obvious threat to this assumption is when a respondent’s anonymity for their answer to the sensitive statement is compromised. This ‘ceiling effect’ exists when a respondent’s true answer is affirmative for all of the statements, thus their affirmative response to the sensitive statement will no longer be hidden. However, the proportion of respondents answering with the maximum number of items (three and four for the non-sensitive and sensitive list groups, respectively) is within the range of other studies.<sup>31</sup> Further, the presence of a ceiling effect likely indicates downwards biased estimates because other respondents would be likely to have understated their true number of 4 affirmative answers in order to conceal their affirmative sensitive answer (Blair and Imai, 2012).

# **4.3.2 Checks on overall approach**

The two key assumptions underpinning the unbiasedness of the list experiment estimator have parallels for the experiment as a whole. While there is no direct test for these assumptions holding for the ACASI and face-to-face methods, there are some indicators of whether or not they were violated.

First, we checked balance on survey questions asked of all respondents after the experimental violence questions. If respondents were differentially affected by one survey method, perhaps indicating a design effect, we may observe them answering subsequent survey questions differently. We found very few statistically significant differences in respondent answers despite the many tests for significance. None of these differences are materially significant.

Second, we analyzed the main list method prevalence regressions with enumerator and controls selected using post-double-selection LASSO (Belloni, Chernozhukov and Hansen, 2014). We found no notable differences in coefficients or significance. These results are shown in Tables A13 to A15.

Third, although a general ‘no liars’ assumption is untestable, we compared refusal rates across methods. This could indicate the relative risk of misreporting for each method. The refusal rate with ACASI, where all one has to do to skip a question is touch a button, may be a lower-bound on the share of people who might have lied if they were interviewed face-to-face and would be uncomfortable asking an

> 30One study by Flavin and Keane (2009) evaluates whether the design of the non-sensitive list affects responses. This study found that asking the non-sensitive group directly about the non-sensitive items resulted in a higher mean number of items than when the non-sensitive questions were asked using the traditional ‘how many of the following items’ approach. As a result, the direct approach to asking the non-sensitive questions resulted in downwards-biased prevalence estimates, though it should be noted that their ‘non-sensitive’ items were somewhat sensitive. Further, a recent paper by Chuang et al. (2018) found that having relatively sensitive ‘non-sensitive’ statements on the list appears to mask the sensitive item of interest, and results in lower variance and higher prevalence estimates. In Rwanda and Nigeria, we used relatively innocuous non-sensitive list items, which, if results from the Chuang et al. measurement experiment on risky sexual behavior hold, then it again seems plausible that these list experiment results may be downwards biased.

> 31The proportion of respondents in the non-sensitive list group reporting three items as true is never greater than 15.3 percent (for the women’s physical violence question in Rwanda), and the proportion of respondents in the sensitive list group reporting the full number of items is never greater than 5.4 percent (for the women’s physical violence question in Nigeria).

19
enumerator to skip the question. Indeed, we that ACASI, has the highest refusal rate out of the methods, though it is still low, at 0.84 percent for women. Table A16 shows refusal rates by question and method.

Additional indications that these results are robust are supported by the broadly similar patterns of misreporting by survey method that were observed over different countries, genders, violence types, perpetration and victimization questions, and list method designs.

# **5 Conclusion**

This paper provided some of the evidence that the prevalence of intimate partner violence varies substantially depending on the survey method used to measure it. Our results suggest that in some contexts, standard survey methods are likely to result in significant underestimates of the prevalence and welfare costs of IPV. The most widely used method, face-to-face interview, resulted in the lowest prevalence estimates, followed by the more private ACASI method. Finally, the list method, which allows respondents to report anonymously, resulted in the highest prevalence. In Rwanda, physical IPV rates were double, and in Nigeria, 39 percent greater when measured using the list method compared to the direct methods.

This paper also found that misreporting was systematic. The women who underreported when interviewed directly (compared to when interviewed with the list method) tended to be more vulnerable, not engaged in paid work, living in conservative communities, and more educated. These results demonstrate that misreporting was correlated with indicators that theory suggests are risk factors for IPV and with variables often targeted in women’s empowerment programs. Our results show non-trivial patterns of misreporting, such that gradients on some IPV-predictor relationships changed substantially, and some signs flipped, depending on the method used to measure IPV.

As other studies have found, direct measures of IPV can produce biased treatment estimates, for example, finding programs caused a reduction in IPV if measured face-to-face, but an increase if measured with the list experiment (Bulte and Lensink, 2019). Given the growing number of studies measuring program impacts on IPV, our results further highlight the risks of collecting measurement error-prone data for generating unbiased correlations and treatment effect estimates. Given that most IPV-related theory and prevention interventions are informed by face-to-face survey data, misreporting bias could have significant implications for our understanding of the prevalence of IPV, what causes it, and how to prevent it.

Our three-method comparison in Rwanda provides additional insights into potential reasons for misreporting. There were strikingly few differences between ACASI-measured and face-to-face-measured prevalence and IPV-predictor relationships. This suggests that women in Rwanda were no more comfortable reporting IPV with ACASI’s added privacy, yet the list method’s anonymity made many more women willing to report. This may suggest that on questions about illegal behaviors where respondents fear a risk of legal sanctions or repercussions from their answer, for example, in places where physical and sexual IPV are criminalized, then only anonymous survey methods may induce some respondents to report.<sup>32</sup>

> 32The similar face-to-face and ACASI reporting patterns may also be affected by the strict private interview setting in Rwanda. In a busier, more typical survey setting where privacy from curious onlookers and other household members is harder to enforce, we may have found that women would be even less willing to report IPV face-to-face, and ACASI may

20
Our results also have implications for the growing literature that uses face-to-face IPV data from multiple countries to draw inferences about IPV prevalence, predictors, and causes (Devries et al., 2013; García-Moreno et al., 2013; Cools and Kotsadam, 2017; Jewkes et al., 2017; Alesina, Brioschi and La Ferrara, 2020; Heise and Kotsadam, 2015). Our finding of significant and systematic underreporting for some types of IPV in our two rural African contexts contrasts with findings from Lima, Peru, where researchers found no overall difference between face-to-face and list method prevalence (Agüero and Frisancho, 2017). These divergent results indicate that reporting bias is, unsurprisingly, strongly affected by contextual factors. Given significant, systematic face-to-face misreporting in some contexts but not others, our results imply that caution should be used when aggregating and analyzing face-to-face IPV data from diverse settings, lest context-specific misreporting bias results.

Policy makers and social scientists measuring sensitive topics could use similar randomized survey experiments to compare IPV reporting rates between the list experiment and direct methods to identify levels of stigma, the types of people most likely to misreport, and the risk of generating biased treatment effects. For example, social scientists researching the impact of interventions on IPV might use such an experiment to sign potential reporting bias in their treatment effect estimates. Also, given Demographic and Health Surveys’ large sample sizes and use in multi-country analysis, perhaps such surveys should measure several IPV questions using the list method as well as the face-to-face method. This would allow data users to assess the likely extent of reporting bias by context, allowing them to make adjustments to inferences about true levels and correlates of IPV.<sup>33</sup> Future research could be conducted in other contexts and on other types of violence and sensitive topics to better understand what drives misreporting. Other potential areas for additional research include studying how to best design and administer the list experiment to minimize bias, and developing alternative measurement methods for efficient causal research on IPV which, according to face-to-face survey data, is estimated to affect one in three women globally (García-Moreno et al., 2013).

> have made more of a difference. See Appendix B for a further discussion on IPV measurement method trade-offs.

> 33DHS data on IPV cover more than 60 developing countries and are cited in more than 5,000 papers on google scholar. If DHS IPV data are measured with substantial, systematic, and context-specific bias as this paper suggests is likely, this has significant implications for the use of such data.

21