metadata: authors: 0: author: Kaplan,Lennart Christian
1: author: Pape,Utz Johann
2: author: Walsh,James Sonam
count: South Sudan
docna: 0: docna: A Light-Touch Method to Improve Accurate
            Reporting of IDP’s Food Consumption
docty: Policy Research Working Paper
subtopic: Inequality,Gender and Development,Law and Justice Institutions,Energy Demand,Energy and Mining,Energy and Environment,Global Environment
teratopic: Poverty Reduction,Gender,Law and Development,Energy
abstracts: cdata!: Misreporting is a well-known challenge
            for researchers in social sciences and even more so for
            policymakers who rely on accurate data to design effective
            relief strategies. This issue is especially prevalent if
            incentives for misreporting exist, for example, if there are
            perceived social or material implications to the answer
            reported. Due to the vulnerabilities that internally
            displaced people face and their dependency on aid support,
            this challenge is particularly acute when trying to measure
            economic welfare. In contexts of emergency, the need for
            accurate information is acute, but many approaches to
            improve data are intrusive or impractical. To address this
            challenge, this paper proposes a light touch method to
            improve the accuracy of consumption data: the inclusion of
            checks and primes to emphasize the importance of accurate
            reporting. The study assesses the effectiveness of this
            method, by randomly distributing the bundle of primes across
            a survey of internally displaced persons in South Sudan. In
            line with the main hypothesis, positive and significant
            effects arise for low consumption quantiles, especially
            consumption quantities that are more susceptible to
            manipulation. The findings suggest that light touch
            approaches to improving survey design can act as a
            cost-effective tool to induce more accurate reporting.
            Further research is needed to validate the measure against a
            gold standard approach and to understand what mechanisms are
            at play.
repnme: repnme: A Light-Touch Method to Improve Accurate
            Reporting of IDP’s Food Consumption
display_title: A Light-Touch Method to Improve
            Accurate Reporting of IDP’s Food Consumption
listing_relative_url: /research/2018/08/30329121/light-touch-method-improve-accurate-reporting-idp’s-food-consumption
topicv3: Energy and Extractives m1326595 507,Poverty m1326687 695,Gender m1327621 615,Mortality m1504365 3661
publishtoextweb_dt: 2018-08-06T00:00:00Z
historic_topic: Poverty Reduction,Gender,Law and Development,Energy
no_of_pages: 36
author_keys: 0: author_key: 1611636
1: author_key: 1154467
2: author_key: 1276394
model_extractions: WPS8539 Policy Research Working Paper 8539 A Light-Touch Method to Improve Accurate Reporting of IDP’s Food Consumption _Lennart Kaplan Utz Johann Pape James Walsh_ Poverty and Equity Global Practice August 2018
Policy Research Working Paper 8539

# **Abstract**

Misreporting is a well-known challenge for researchers in social sciences and even more so for policymakers who rely on accurate data to design effective relief strategies. This issue is especially prevalent if incentives for misreporting exist, for example, if there are perceived social or material implications to the answer reported. Due to the vulnerabilities that internally displaced people face and their dependency on aid support, this challenge is particularly acute when trying to measure economic welfare. In contexts of emergency, the need for accurate information is acute, but many approaches to improve data are intrusive or impractical. To address this challenge, this paper proposes a light touch method to improve the accuracy of consumption data: the inclusion of checks and primes to emphasize the importance of accurate reporting. The study assesses the effectiveness of this method, by randomly distributing the bundle of primes across a survey of internally displaced persons in South Sudan. In line with the main hypothesis, positive and significant effects arise for low consumption quantiles, especially consumption quantities that are more susceptible to manipulation. The findings suggest that light touch approaches to improving survey design can act as a cost-effective tool to induce more accurate reporting. Further research is needed to validate the measure against a gold standard approach and to understand what mechanisms are at play.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at upape@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **A Light‐Touch Method to Improve Accurate Reporting of IDP’s Food Consumption**

## **Lennart Kaplan, Utz Johann Pape, James Walsh**<sup>**1**</sup>

Keywords: Consumption Measurement, Poverty, Questionnaire Design, Behavioral Intervention JEL: C83, D63, D90, I32

> 1 Authors in alphabetically order. Corresponding author: Utz Pape (upape@worldbank.org). The findings, interpretations and conclusions expressed in this paper are entirely those of the authors, and do not necessarily represent the views of the World Bank, its Executive Directors, or the governments of the countries they represent. This paper benefited largely from the valuable input provided by Julie Perng and Luca Parisotto as well as discussions with Quy‐Toan Do.
# 1. Introduction

Accurate data on the key economic variables affecting people who have been forcibly displaced, such as consumption and assets, are essential to understanding their situation and to developing evidence based policies to support them. Poor information can lead to flawed diagnostics or incorrect assessments of impact. Data inaccuracies may lead policy makers to allocate funds to the wrong people or to the wrong programs. The standard way in which the World Bank and other policy organizations develop statistics is through individuals’ responses to questions in economic surveys. Self‐reported information is vulnerable to myriad reporting inaccuracies when social scientists ask personal or intrusive questions or when respondents anticipate social or material implications to the answers they provide.<sup>2</sup> This is of particular concern when respondents believe that misreporting may provide relief, both because of the of the sensitivity and the gravity of the policy challenge. In situations where it has been possible to compare survey responses to revealed economic behavior, striking disparities are sometimes found. In one investigation for example, Poterba and Summers (1986) report that misstatements regarding employment status in the Current Population Survey led to an underestimation of the duration of unemployment by up to 80 percent and even greater overestimates of the frequency of labor market entries and exits. In another study, Rosenfeld, Imai, and Shapiro (2016) look at voting behaviors in a sensitive anti‐abortion referendum held in Mississippi in 2011. They compare actual county level vote shares against survey results from a sample frame of individuals who voted during the election (based on public records). Surveys that used direct questioning led to an underestimation of casting a “no” vote by more than 20 percentage points in the majority of counties.

There are a number of mechanisms through which the validity of self‐reported information in surveys can be compromised. Some inaccuracies result from cognitive biases – for example, acquiescence or “yea‐saying” (Bachman and O’Malley 1984; Hurd 1999), extreme responding (Cronbach 1946; Hamilton 1968), and question order bias (Sigelman 1981). One solution to problems such as question order bias is to randomize the order of questions (Warner 1965). Other inaccuracies emerge from conscious but not calculated behavior. Respondents may deliberately misreport information on sensitive subjects not to distort statistics but to maintain their reputation or to abide by political norms (Gilens, Sniderman, and Kuklinski 1998). A common solution to this is to enable participants to cloak their behaviors or beliefs. List experiments, endorsement experiments, and randomized experiments are commonly used techniques for this purpose (Rosenfeld, Imai, and Shapiro 2016).

> 2 This is of particular concern, for example when asking about race (Kuklinski et al. 1997) or corruption (Gingerich 2010).

2
The explanations above assume that people intend to report accurately but are prevented from doing so due to aspects of the situation. In some contexts, individuals may misreport due to expectations about the implications of the results of the study. For example, individuals may misreport to increase earnings in a study context (Mazar, Amir, and Ariely 2008) or to shape the results of the study if they believe that it will inform policy. In situations where individuals wish to influence a particular research outcome, a guise of anonymity will not shift their behavior. It is important to note that our concern is not with the ethics of individual misreporting – this is a reasonable response to contexts of extreme vulnerability – but rather to ensure that policymakers have access to data that enables them to adequately serve the vulnerable population as a whole.

Behavioral science is increasingly being used as a policy tool to help policymakers create better policy and solve collective action problems more effectively (World Bank 2015). This is based on research illustrating that people make decisions on the basis of both external and internal reward mechanisms (Mazar and Ariely 2006). Even in cases where people have an extrinsic incentive to misreport, this may be overridden by a preference for remaining consistent with their values. One example of this is when individuals’ beliefs regarding the consequences of misreporting affects their behavior. In an two‐person experiment where one participant can increase her payoff by misreporting but at the expense to her counterpart, Gneezy (2005) finds that individuals’ propensity to misreporting is sensitive to the costs it imposes on the other person. Contextual cues affect the salience of internal incentives (or intrinsic motivations) and thus the accuracy of responses. This psychological mechanism has been put to practical use in policy. In multiple contexts, normative messaging has been used to increase tax payments (Hallsworth et al. 2017; Hernandez et al. 2017) or reduce littering and environment theft (Cialdini 2003).

In this paper, we apply the tools of behavioral science to investigate the veracity of consumption reports by internally displaced persons (IDPs). In numerous rounds of data collection in Somalia and South Sudan, IDPs report significantly lower levels of consumption than non‐IDP households. In previous survey rounds 45 percent of Somali IDP households report food consumption below subsistence levels and approximately 80 percent below recommended levels (refer Figure 5). While the data may be accurate, there are two reasons to suspect that it is not. First, such high levels of non consumption would be associated with high rates of mortality due to starvation. Although being high, the mortality rates among IDPs suggest that this is not happening systematically across the country at such a scale (FEWS NET, 2018).<sup>3</sup> Second, non‐IDP households that are statistically similar on observable characteristics report higher levels of consumption than IDP households. While IDPs and non‐IDPs may

> 3 Although data from the USAID led Famine Early Warning Systems Network (FEWS NET) suggest high level of malnutrition, evidence on mortality across the counties is mixed (FEWS NET, 2018).

3
have different opportunities to generate income, it is unlikely that IDPs choose not to smooth their resources to balance between food and non‐food consumption in a way that endangers their life.<sup>4</sup> If it is the case that survey respondents misreport, the inaccuracies it generates in the data are highly problematic. At best, it makes the data spurious and unusable. At worst, it could lead to misallocations of aid, from more vulnerable areas to less vulnerable areas, or from solutions emphasizing sustainability to immediate relief where immediate relief is unnecessary. Due to the dangerous environment in South Sudan and Somalia, it is not currently possible to do use alternative data collection methods, for example ethnographic research, to investigate this puzzle in the data. The validity of alternative investigative methods such as food diaries is vulnerable to the same incentive to game as surveys.

One way to investigate whether people misreport is to test whether consumption rates change in response to nudges. If these primes are effective, they would be expected to particularly affect potentially underreporting, hence, poor households. Moreover, as vulnerable populations would have higher incentives to underreport, priming should be stronger for IDPs than for comparable non‐IDP populations. We find the primes induce higher reporting in lower quintiles of reported consumption. This treatment pattern is driven by aid reliant IDPs and vanishes when considering the comparison group of non‐IDPs. The results are especially strong for consumption quantities (items and kilograms), which are most easily subject to intentional misreporting. This suggests that IDPs are indeed misreporting. The paper has two main limitations. First, it can only compare the treated group against an estimate of the “true” consumption rates. Second, the intervention is bundled. For this reason, it is impossible to isolate the causal mechanism affecting the observed changes in reporting. Further work is needed to identify an estimate of the true level of consumption against which to compare the primed individuals and to isolate the causal mechanisms by which people are changing their behavior.

The paper proceeds as follows. Section 2 provides an overview about the underlying context and the compiled data. Section 3 provides an overview about the underlying methods, while Section 4 introduces the empirical approach, which builds the foundation for the results in Section 5. This is complemented by an assessment of robustness and potential channels in Section 6. Finally, findings are discussed and summarized in Section 7.

# 2. Context and data

On July 9 2011 South Sudan became the 55th African independent state after seceding peacefully from the Republic of Sudan. Facing a history of a 50 year lasting conflict South Sudan slid back to instability

> 4 The underlying survey data of this study discussed at a later stage actually indicates that IDPs have a more calorie intensive food consumption profile (refer

Figure 8).

4
enumerators nor the respondents see the structure of the sub‐modules, but the assigned items are asked in a categorically meaningful way (like cereals, fruits, etc).<sup>5</sup> The data is used to construct four outcome measures. The surveys collect information on quantities in terms of (i) number of consumption items and (ii) kilograms. The quantities can be used to construct measures of (iii) monetary and (iv) caloric food consumption scaling the quantities with data on average prices and energy levels.<sup>6</sup> Though we are mainly interested in evaluating the impact of the nudges on the total consumption value ‐ both in terms of money and food intake ‐ these variables are difficult for respondents to falsify because these are _second‐order_ values that are calculated as a function of other variables, including consumption quantities and calories or prices that are in turn deflated. All of this adds noise to the answer provided by the respondent, and they depend in part on variables over which the respondent has no control. The consumption quantity in kilograms is a more direct measure of the quantity consumed as expressed by the respondent, and may lead to more accurate estimation of the impact of the nudges. Finally, counting the number of items may lead to an even more accurate measure, since the variable does not undergo any cleaning at all and is taken at face value. Furthermore, omitting an item is likely to be the easiest and quickest way for respondents to reduce the true value of the household’s consumption.<sup>7</sup> Poverty amongst IDP households is high, and 9 in 10 IDP households across South Sudan live under $1.90 USD PPP (2011) per capita per day in 2017. IDP households in the sample interviewed for the experiment consume on average 333 SSP (2017) per capita per day. IDP households reported on average 6.63 core consumption items. These figures represent about 20 percent of core items asked to the households. Figure 2 visualizes that 39 percent of households report consumption below the recommended daily intake of 2,100 kcal (R) and 16 percent below the subsistence level of 1,200 kcal (S) (Ravallion & Bidani, 1994).

> 5 Due to the survey method applied CRS surveys contain the core consumption module and one additional consumption module. The share in imputed consumption is on average 99.9 percent. IDP surveys contain due to the previously outlined time constraints only core consumption items. However, by design these items capture the lion’s share of consumption (on average approx. 94 percent of total consumption in more comprehensive CRS surveys).

> 6 For a description of the caloric intake measure, please consult the appendix.

> 7 Note that the number of consumption items is not reported per capita as it does not increase proportionally with household size.

6
<!-- Start of picture text --> Calory Consumption p.a.<br>2/\<br>/\<br>: \<br>@\<br>2<br>a / \<br>/ \<br>2 / N\<br>/<br>o oo<br>ossoaow.e<br>Calory Consumption<br><!-- End of picture text -->
- **Appeal to Honesty** The importance of accurate answers and the purpose of the survey will be reiterated.

- • **"** Thank you for taking the time to speak to us. We really appreciate the time you are giving to participate in the survey. We encourage you to provide honest information. By participating in the survey and by providing accurate information, you are playing an

- **1** important role in helping us understand the situation in South Sudan."

- • **Prime to encourage more accurate reporting** At the beginning of the survey module concerning food consumption, the respondent will be given a short fictional scenario which will require passing judgement.

- • **"** John asks his good friend Deng if he has some money that he can lend him to help him pay for medicine for his sick son. Deng has money but was planning to buy cigarettes

- **2** with it. He lies and tells John that he has none. Is it okay for Deng to lie to John?"

- • **Investigative Probing:** At the start of the survey module concerning food consumption, the respondent will be asked to tell when was the last time their household had a meal. This question will then also be asked for each of four major food categories: ‘Bread and Cereals’, ‘Meat’, ‘Fruits’, ‘Pulses and vegetables’.

- **3** •E.g., "When was the last time that any of the household members had Bread and Cereals?"

<!-- Start of picture text --> • Prime to encourage more accurate reporting<br>At the beginning of the survey module concerning food consumption, the respondent<br>will be given a short fictional scenario which will require passing judgement.<br>• " John asks his good friend Deng if he has some money that he can lend him to help him<br>pay for medicine for his sick son. Deng has money but was planning to buy cigarettes<br>2<br>with it. He lies and tells John that he has none. Is it okay for Deng to lie to John?"<br><!-- End of picture text --> <!-- Start of picture text --> • Investigative Probing:<br>At the start of the survey module concerning food consumption, the respondent will be<br>asked to tell when was the last time their household had a meal. This question will then<br>also be asked for each of four major food categories: ‘Bread and Cereals’, ‘Meat’,<br>‘Fruits’, ‘Pulses and vegetables’.<br>3 •E.g., "When was the last time that any of the household members had Bread and<br>Cereals?"<br><!-- End of picture text --> Households are randomly exposed to behavioral treatments, in the form of a prime for more accurate reporting and investigative probing, to try and elicit more truthful answers from respondents. This way they do not constrain the choice frame, but rather alter the anchoring towards more truthful reporting(The World Bank, 2015).

The bundle of primes addresses different behavioral processes. (1) Appeals to honesty are a standard tool in surveys to increase data accuracy by relying on social approval (Talwar, Arruda, & Yachison, 2015). (2) Primes to encourage more accurate reporting induce unconscious cognitions, which are intended to affect subsequent behavior. When facing incentives to misreport, respondents would answer more accurately to sustain self‐consistency. (3) Investigative probing puts a higher salience on the question. By asking for broader categories first, subsequent sub categories are put under more scrutiny. Self‐consistency is reinforced by relating to a longer recall period of seven days.<sup>10</sup> While the appeal to honesty and the prime target intentional misreporting, investigative probing is addressing classical measurement error.

The sample was randomly selected into each treatment arms in two groups of approximately 50 percent, with 2,467 households in the control group and 2,459 in the treatment group. The 10 The methodological appendix provides an overview of the relevant questions in the food consumption module. 8
randomization process was built into the CAPI (Computer Assisted Personal Interview) questionnaires administered in the surveys. As our research hypothesis suggests stronger effects of nudges for more vulnerable populations, we focus on IDPs for the main analysis. The availability of the HFS sample provides a comparison group of non‐IDP households for the experiment, which will be used for robustness checks. The treatment and control groups are relatively balanced. There is a higher share of male headed households in the treatment group, which have also more members, though in practical terms these differences are relatively small. As gender of the household head and household size are potentially correlated with poverty, these variables are included in the regression models and interacted with the treatment to control for potential impacts (Lanjouw & Ravallion, 1995).

_Table 1: Balance across treatment and control arms (IDP sample)._

||Control|Treatment|Difference, p‐value|
|---|---|---|---|
|Household size|4.835|5.098|0.003***|
||(0.060)|(0.064)||
|Gender of household head|0.492|0.448|0.005***|
||(0.011)|(0.011)||
|Literacy of household head|0.507|0.529|0.155|
||(0.011)|(0.011)||
|Household head completed some primary school|0.540|0.563|0.133|
||(0.011)|(0.011)||
|Is the household head employed|0.328|0.319|0.555|
||(0.010)|(0.010)||
|Share of children in household|0.364|0.373|0.309|
||(0.006)|(0.006)||
|Share of elderly in household|0.011|0.010|0.582|
||(0.002)|(0.001)||
|First Component of Asset Principal Component Analysis|‐0.126|‐0.194|0.162|
||(0.037)|(0.032)||
|N|2079|2066||
|Proportion|0.502|0.498||
|_Standard errors in parentheses; **p<0.05, ***p<0.01_||||

# 4. Empirical Strategy

To assess the effect of our prime on reporting behavior, we can formulate following simple regression equation.

Y� �β� �β�T� �Ti ∗Xiβ� �X�β� �γ� �α� �ε�, (3) 9
where Y� is the log of the outcome variable. Across different models we estimate the effect for (i) the number of consumption items consumed [referred to in the regression equation as Cons. Num.], (ii) consumption quantity per adult equivalent (in kilograms) [Cons. Quant.], (iii) monetary consumption value per adult equivalent [Cons. Val.] and (iv) daily caloric intake per adult equivalent [Cons. Cal.].

Our main treatment variable T� is a dummy variable which takes the value of 1 if the household i was assigned to the treatment group. γ� indicates a set of camp fixed effects, α� are month fixed effects, and ε� is the idiosyncratic error term. X� denotes a vector of control variables generally associated with consumption, including household size, the gender of the household head, and the proportion of children (under 18) in the household. Moreover, we add an asset index based on the first component of a principal component analysis (Filmer & Pritchett, 2001; McKenzie, 2005).<sup>11</sup> The model will be estimated with and without controls to check the impact they may have. As the treatment might interact with the unbalanced covariates, it makes sense to add to the regression T� ∗X� , the interaction of the unbalanced controls with the treatment variable (Lin & Green, 2016; Baranov, Bhalotra, Biroli, & Maselko, 2017).

It is expected that the respondents who will be affected by the treatment are respondents that would otherwise misreport and, hence, a more likely to be at the extremes of the distribution.<sup>12</sup> Therefore, we complement our analysis with a quantile regression approach. The idea of the quantile regression framework, which was introduced by Koenker & Bassett Jr (1978), is to take the entire distribution of the dependent variable into account by estimating several regressions, which put more weight to the quantile of interest. The underlying minimization problem can be stated as follows: where θ i is the quantile of interest and the weighted sums of deviations |y� �τ| of the outcome per quantile. Minimizing the latter, differential effects conditional on the quantile of the dependent variable are obtained. Further, it has the advantage of being less prone to outliers and non‐normality of the error term. For our purpose, quantile regressions offer the advantage that they are more flexible than simple interactions with poverty lines, which would be endogenous to consumption levels.

> 11 As assets (bikes, fans, rickshaws etc.) can be more easily surveyed by enumerators, those are likely to capture parts of the household wealth.

> 12 Although these hypotheses were not pre‐registered, they are based on theoretical considerations about the mechanisms of the underlying behavioral primes.

10
# 5 Results

There is a slight indication that the treatment may have worked, based on consumption distributions across treatment and control group. The consumption distribution shown in The median of calorie consumption is well above the recommended daily intake. However, still a substantial part of the distribution of 16 percent reports below the subsistence level and 40 percent below the recommended daily intake. Hence, the prime would also be relevant in the adult equivalent setting to achieve more precise reporting, which is analyzed in our regression framework subsequently. Taking into account the finding that consumption levels are lower than to be expected, the most relevant treatment effects can be expected at the left tail of the distribution.

Figure 3 _shows a slight difference in caloric consumption between IDP households in the treatment group and the control group, though this is apparent only at lower levels of consumption, i.e. below the subsistence level of 1,200 kcal._ The median of calorie consumption is well above the recommended daily intake. However, still a substantial part of the distribution of 16 percent reports below the subsistence level and 40 percent below the recommended daily intake.<sup>13</sup> Hence, the prime would also be relevant in the adult equivalent setting to achieve more precise reporting, which is analyzed in our regression framework subsequently. Taking into account the finding that consumption levels are lower than to be expected, the most relevant treatment effects can be expected at the left tail of the distribution.

_Figure 3: Caloric consumption p.c. (adult equivalents)._

> 13 Compared to the monetary consumption levels, the calory consumption p.a. seems rather high. This is partly attributable to the fact that IDP’s consumption focuses on energy intensive products, where cooking oil and sorghum constitute 45 percent of food expenditure. If we contrast the consumption shares with non‐IDPs, we find that although the diet of non‐IDPs is less energy intensive, it comprises a higher variety (see

Figure 8).

11
<!-- Start of picture text --> Calory Consumption p.a.<br>88<br>8<br>283 LV’ va INtN. .<br>a) 74 \\<br>°<br>weoSS atWSQs<br>Calory Consumption<br><!-- End of picture text -->
|Treatment|0.035**<br>(0.016)|0.061*<br>(0.033)|0.028<br>(0.018)|0.137***<br>(0.042)|‐0.018<br>(0.018)|0.081*<br>(0.039)|0.019<br>(0.028)|0.001<br>(0.067)|
|---|---|---|---|---|---|---|---|---|
|Observations|3,955|3,955|3,955|3,955|3,955|3,955|3,955|3,955|
|R‐squared|0.001|0.273|0.001|0.070|0.000|0.078|0.000|0.123|
|State FE|NO|YES|NO|YES|NO|YES|NO|YES|
|Month FE|NO|YES|NO|YES|NO|YES|NO|YES|
|Controls|NO|YES|NO|YES|NO|YES|NO|YES|
|Controls<br>Interacted|NO|YES|NO|YES|NO|YES|NO|YES|

**Note:** Robust standard errors in parentheses (White, 1980): *** p<0.01, ** p<0.05, * p<0.1. Columns (1‐2) are measured on the household level. Columns (3‐8) refer to per capita OECD adult equivalents. A full set of coefficients for control variables can be found in

Table 15 the appendix.

To capture this heterogeneity across consumption levels, quantile regressions are applied. Results are shown in Figure 4 and Table 3.

_Table 3: Results from quantile regressions of different outcome variables._

||(1)|(2)|(3)|(4)|
|---|---|---|---|---|
|Outcome Variables|ln(Cons. Num.)|ln(Cons. Quant.)|ln(Cons. Val.)|ln(Cons. Cal.)|
|Q0.1|0.165**|0.342***|0.079|0.235*|
||(0.064)|(0.079)|(0.068)|(0.127)|
|Q0.25|0.058**|0.201***|0.198***|0.140*|
||(0.028)|(0.067)|(0.053)|(0.080)|
|Q0.5|0.018|0.136**|0.119**|0.042|
||(0.032)|(0.056)|(0.050)|(0.062)|
|Q0.75|0.047|0.114**|0.071|0.032|
||(0.034)|(0.050)|(0.051)|(0.067)|
|Q0.9|‐0.016|0.049|‐0.015|0.013|
||(0.028)|(0.050)|(0.054)|(0.064)|
|Observations|3,955|3,955|3,955|3,955|
|Month FE|YES|YES|YES|YES|
|State FE|YES|YES|YES|YES|
|Controls|YES|YES|YES|YES|
|Interacted Controls|YES|YES|YES|YES|

Robust standard errors in parentheses (White, 1980): *** p<0.01, ** p<0.05, * p<0.1. Column (1) is measured on the household level. Columns (2‐4) refer to per capita OECD adult equivalents.

13
<!-- Start of picture text --> In(Number Consumption Items) In(Consumption Quantity)<br>oO o<br>Eooo LZ EeO°<br>Fe Feo<br>9 &<br>e<br>0 2 4 6 8 1 0 2 A 6 8 1<br>Consumption Segment (Quantile) Consumption Segment (Quantile)<br>In(Consumption Value) In(Consumption Calories)<br>8 8<br>-o ~~?<br>88 8S<br>is ie<br>Es Ea<br>28 es<br>we eS 4 ——<br>2<= ge<br>Fes FR<br>N3);<br>0 2 4 6 8 1 0 2 4 6 8 1<br>Consumption Segment (Quantile) Consumption Segment (Quantile)<br><!-- End of picture text -->
Column (2) and (3). Therefore, the results stress the nuanced effect of the prime, which only affects certain strata of the population.

_Table 4: Results using poverty thresholds, model (2) and (3)._

||(1)|(2)|(3)|(4)|
|---|---|---|---|---|
|VARIABLES|**>1200kcal**|**>2100kcal**|**>**�<br>2<br>3<br>~~�~~Poverty Line|**>Poverty Line**|
|Treatment|0.010|0.069*|0.063*|0.029|
||(0.027)|(0.037)|(0.037)|(0.036)|
|Observations|3,955|3,955|3,955|3,955|
|R‐squared|0.067|0.098|0.118|0.135|
|State FE|YES|YES|YES|YES|
|Month FE|YES|YES|YES|YES|
|Controls|YES|YES|YES|YES|
|Controls Interacted|YES|YES|YES|YES|

# 5. Treatment heterogeneity and robustness <u>Heterogenous effects:</u>

If the primes would reduce misreporting, stronger effects are to be expected among subpopulations that have higher incentives to misreport, e.g., aid‐reliant IDPs. In order to assess this channel more thoroughly, (i) heterogenous effects are estimated contingent on aid reliance and (ii) the sample is compared to a non‐IDP comparison group.

Parts of the respondents from the CRS and HFS were also interviewed with regard to their previous support through UN agencies. This dummy indicator can be used for an assessment of heterogenous effects.<sup>17</sup> The model is analogous to equation (3), where we add UN assistance as a further control variable as well as an interaction term of UN assistance with the behavioral treatment. The results are depicted in Table 5 and indicate no clear pattern. Only for the number of consumption items a positive significant coefficient is found. The significant positive interaction of the treatment and previous aid exposure could be treated as some weak evidence that the prime is more effective for aid exposed IDPs, but should not be overstated due to the non‐significance for the other three outcomes of interest.

_Table 5: Channel – UN assistance._

> 17 The results can only be interpreted as an explorative analysis as UN assistance was not balanced across treatment and control groups, where treatment households have a higher probability of being previously exposed to aid.

15
||(1)<br>ln(Cons. Num.)|(2)<br>ln(Cons. Quant.)|(3)<br>ln(Cons. Val)|(4)<br>ln(Cons. Cal.)|
|---|---|---|---|---|
|Treatment|0.100|0.195**|0.171*|0.105|
||(0.066)|(0.080)|(0.081)|(0.087)|
|UN Assistance|‐0.028|‐0.065|‐0.152***|‐0.143***|
|**Treatment*UN Assistance**|(0.038)<br>**0.104****|(0.045)<br>**‐0.059**|(0.043)<br>**0.016**|(0.046)<br>**0.011**|
||**(0.051)**|**(0.060)**|**(0.061)**|**(0.064)**|
|Observations|2,204|2,204|2,204|2,204|
|R‐squared|0.38|0.086|0.098|0.108|
|State FE|YES|YES|YES|YES|
|Month FE|YES|YES|YES|YES|
|Controls|YES|YES|YES|YES|
|Interacted Controls|YES|YES|YES|YES|

Robust standard errors in parentheses (White, 1980): *** p<0.01, ** p<0.05, * p<0.1. Column (1) is measured on the household level. Columns (2‐4) refer to per capita OECD adult equivalents.

The non‐IDP subsample offers an interesting opportunity to assess the robustness of the results. Constraining the sample only on non‐IDPs, we can estimate our results analogous to Table 3. The pattern of positive and significant treatment coefficients in the lower quantiles vanishes, except for Column (1). This could be interpreted as evidence that the light‐touch method applied are more efficient for the vulnerable IDP population, which has higher incentives to indicate need than the non‐ IDPs. This would be in line with previous studies (e.g., Cilliers, Dube, & Siddiqi, 2015) suggesting a high degree of social desirability bias in the setting of foreign assistance. Specifically, the populations exposed to development aid, in our setting the IDPs, would be more likely to provide socially desirable answers to signal their “worthiness” for assistance. This corresponds to Table 5, providing some weak evidence that the primes are more effective for respondents relying on UN aid. It would be of particular interest to examine those heterogenous effects based on more fine‐grained data on neediness and degree of aid reliance of recipients. For this purpose, however, a “true” benchmark would be needed. As administrative data is non‐existent or of poor quality, an alternative for future research might be to build on measures from qualitative work as suggested by Blattman, Jamison, Koroknay‐Palicz, Rodrigues, & Sheridan (2016). Moreover, one should be careful to draw too strong conclusions from these results as the number of observations is limited in this comparatively small sub‐sample.

_Table 6: Quantile Regressions – reduced sample (only non‐IDPs)._

||(1)|(2)|(3)|(4)|
|---|---|---|---|---|
|Outcome Variables|ln(Cons. Num.)|ln(Cons. Quant.)|ln(Cons. Val.)|ln(Cons. Cal.)|
|Q0.1|‐0.027|‐0.069|‐0.026|0.032|

16
||(0.079)|(0.102)|(0.110)|(0.113)||
|---|---|---|---|---|---|
|Q0.25|0.148**|‐0.052|0.012|‐0.057||
||(0.073)|(0.095)|(0.107)|(0.122)||
|Q0.5|0.067|‐0.041|‐0.032|0.044||
||(0.067)|(0.081)|(0.100)|(0.100)||
|Q0.75|‐0.071|‐0.072|‐0.015|‐0.052||
||(0.054)|(0.080)|(0.092)|(0.080)||
|Q0.9|‐0.041|0.157|0.074|0.119||
||(0.047)|(0.105)|(0.144)|(0.127)||
|Observations|780|780|780|770||
|Month FE|YES|YES|YES|YES||
|State FE|YES|YES|YES|YES||
|Controls|YES|YES|YES|YES||
|Interacted Controls|YES|YES|YES|YES||
|Robust<br>standard<br>errors<br>Column (1) is measured on the ho|in<br>parentheses<br>usehold level. Columns (|(White,<br>1980):<br>***<br>2‐4) refer to per capita OECD ad|p<0.01,<br>**<br>ult equivalents.|p<0.05,<br>*|p<0.1.|

# <u>Robustness:</u>

In line with hardly credible low consumption levels, misreporting could be considered to be more prevalent at the tails of the distribution, hence, among the extreme values. On the one hand, it makes, thus, sense to consider those outliers. On the other hand, it is problematic to base the inference mainly on those extreme values. Ideally, one would know how to distinguish the intentionally misreported outliers and the ones that are caused by errors in reporting or data entry. The log normalization in the main analysis is chosen as a compromise of keeping most data possible, but making estimates less susceptible to outliers. This suggests two natural robustness checks: (i) in a more liberal setting, the outcomes in levels are used and (ii) in a more conservative setting, the outliers at the 5<sup>th</sup> and 95<sup>th</sup> percentile are discarded. Regression results using the levels are depicted in Table 7.<sup>18</sup> _Table 7: Quantile Regressions – outcomes in levels._

||(1)|(2)|(3)|(4)|
|---|---|---|---|---|
|Outcome Variables|Cons. Num.|Cons. Quant.|Cons. Val.|Cons. Cal.|
|Q0.1|0.544**|0.741***|9.440|229.126*|
||(0.254)|(0.173)|(13.305)|(136.013)|
|Q0.25|0.298*|0.675**|60.585***|179.447|

18 As scaling of the outcome variables is different – e.g., the outliers with regard to consumption quantity in kilograms might not correspond to the consumption quantity in calories – the outliers for one measure do not always correspond to outliers in the other measure. In order to guarantee that we still base the inference on the same observations, outliers from all corresponding variables are dropped, which explains that the resulting sample is smaller than 90 percent of the full sample.

17
||(0.156)|(0.224)<br>(16.261)|(157.584)|
|---|---|---|---|
|Q0.5|0.151|0.638*<br>49.404**|197.589|
||(0.194)|(0.280)<br>(20.477)|(192.043)|
|Q0.75|0.341|0.700**<br>19.499|281.317|
||(0.246)|(0.339)<br>(30.762)|(286.250)|
|Q0.9|‐0.077|0.609<br>‐30.117|‐279.159|
||(0.333)|(0.540)<br>(41.581)|(284.569)|
|Observations|3,955|3,955<br>3,955|3,955|
|Month FE|YES|YES<br>YES|YES|
|State FE|YES|YES<br>YES|YES|
|Controls|YES|YES<br>YES|YES|
|Interacted Controls|YES|YES<br>YES|YES|
|Robust<br>standard<br>errors<br>Column (1) is measured on the ho|in<br>parentheses<br>usehold level. Columns (|(White,<br>1980):<br>***<br>p<0.01,<br>**<br>2‐4) refer to per capita OECD adult equivalents.|p<0.05,<br>*<br>p<0.1.|

Table 8 depicts the results without outliers and indicates a slightly less nuanced pattern. In line with our hypothesis of stronger misreporting tendencies on the extremes, Column (1) indicates significant treatment effects at the 10<sup>th</sup> and 25<sup>th</sup> percentile. Although significant treatment effects among higher quintiles can be found in Column (2) and (3), the coefficients for the 25<sup>th</sup> percentile are quantitatively larger. Finally, with regard to caloric consumption in Column (4) statistical significance vanishes, but the largest coefficient is to be found in the 10<sup>th</sup> percentile. Hence, although the pattern gets weakened when excluding outliers, the prime still significantly affects the reported consumption quantities with stronger effects in the lower quantiles.

_Table 8: Quantile Regressions – without outliers._

||(1)|(2)|(3)|(4)|
|---|---|---|---|---|
|Outcome Variables|ln(Cons. Num.)|ln(Cons. Quant.)|ln(Cons. Val.)|ln(Cons. Cal.)|
|Q0.1|0.124**|0.106|0.085|0.058|
||(0.049)|(0.067)|(0.064)|(0.091)|
|Q0.25|0.045*|0.139**|0.162***|0.042|
||(0.027)|(0.055)|(0.044)|(0.077)|
|Q0.5|0.000|0.065|0.119**|0.037|
||(0.032)|(0.050)|(0.046)|(0.059)|
|Q0.75|0.028|0.077*|0.086*|0.049|
||(0.032)|(0.043)|(0.048)|(0.063)|
|Q0.9|‐0.027|0.064|0.027|0.039|
||(0.023)|(0.039)|(0.049)|(0.051)|
|Observations|3,711|3,605|3,576|3,500|

18
Month FE YES YES YES YES State FE YES YES YES YES Controls YES YES YES YES Interacted Controls YES YES YES YES Robust standard errors in parentheses (White, 1980): *** p<0.01, ** p<0.05, * p<0.1. Column (1) is measured on the household level. Columns (2‐4) refer to per capita OECD adult equivalents.

Regression techniques, which are based on assumptions for large samples drawn from finite populations, are often not suitable in the context of randomized experiments (Heß, 2017). The uncertainty is in this case not coming from the sampled units observed, but from the fact that we can only observe one of the potential outcomes, which is due to the treatment applied to the different units (Athey & Imbens, 2017). One approach would be to take the randomization explicitly into account and follow R.A. Fisher’s idea of statistical inference via permutation tests of treatment allocation (Young, 2016). The idea is to assume uncertainty about the treatment allocation and compare the actual treatment allocation to re‐randomizations. The results of this exercise are depicted in Table 11, underscoring the robustness of the main results.

_Table 9: Results from baseline estimation, model (2), with random‐inference based p‐values._

||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|
|---|---|---|---|---|---|---|---|---|
|VARIABLES|ln(Cons.<br>Num.)|ln(Cons.<br>Num.)|ln(Cons.<br>Quant.)|ln(Cons.<br>Quant.)|ln(Cons.<br>Val.)|ln(Cons.<br>Val.)|ln(Cons.<br>Cal.)|ln(Cons.<br>cal.)|
|Treatment|0.0348**<br>(0.0200)|0.0614**<br>(0.0560)|0.0281<br>(0.1300)|0.1374***<br>(0.0020)|‐0.0178<br>(0.2820)|0.0812**<br>(0.0340)|0.0189<br>(0.4980)|0.0007<br>(0.9940)|
|Observations|3,995|3,995|3,995|3,995|3,995|3,995|3,995|3,995|
|R‐squared|0.0012|0.2744|0.0003|0.0805|0.0006|0.0725|0.0001|0.1232|
|State FE|NO|YES|NO|YES|NO|YES|NO|YES|
|Month FE|NO|YES|NO|YES|NO|YES|NO|YES|
|Controls|NO|YES|NO|YES|NO|YES|NO|YES|
|Controls<br>Interacted|YES|YES|YES|YES|YES|YES|YES|YES|

Robust standard errors in parentheses (White, 1980): *** p<0.01, ** p<0.05, * p<0.1.

# 6. Discussion and Conclusion

The conflict in South Sudan displaced circa two million persons, constituting more than 15 percent of the country’s population (UN OCHA, 2017). Moreover, the majority of population is living in extreme poverty (The World Bank, 2016). Humanitarian crises like the one in South Sudan ask for well targeted policy responses, which address the population strata with the highest need first. This, however, is no arbitrary task as aid allocation mechanisms might set adverse incentives to underreport. Even given 19
the extreme context, surveyed consumption levels indicate an unusually high share below subsistence levels.

For this purpose, this study assesses the effectiveness of a bundle of light‐touch measures. In line with our hypothesis we find significant treatment effects, which cluster in lower (potentially underreported) consumption quintiles. Moreover, effects are stronger for the number of consumption items than for monetary consumption quantities, where former are more susceptible to deliberate misreporting. Furthermore, the significant treatment effects are driven mainly by the vulnerable IDP subpopulation, which are more likely to be in need for foreign aid. Primes can, hence, help to improve data accuracy and inform policy to develop durable solutions. However, results should be taken with a grain of salt as it is not possible to compare the reported consumption outcomes to more objective consumption data. Although the mortality rates among IDPs suggest that starvation is not happening systematically across the country, the precarious situation calls for further scrutiny.<sup>19</sup> Before adjusting poverty estimates a thorough comparison with more “objective” data from administrative, anthropometric or observational sources is needed. While this type of data was not available in IDP camps due to the fragile context, future research could validate this finding in other settings.

Moreover, unbundling the primes in different treatment arms could help to shed light on the underlying causal mechanisms. The underlying design of one treatment and control arm does not allow for further disentangling the results. However, if classical measurement error would be affected only, treatment effects of the primes should be uniform. In contrast, heterogenous effects across quantiles suggest that the targeting of intentional misreporting via the appeal to honesty and prime to report more accurately would be the driver of our results. In order to design more effective primes, disentangling the pathways and trying different combinations could be a beneficial way forward. Our research can be considered as an early step to employ priming for better targeted policy responses in challenging contexts, which might not only be applicable in South Sudan, but also in other contexts facing humanitarian crises.

19 FEWS NET (2018).

20