metadata: authors: 0: author: Tandon,Sharad Alan
count: Yemen,Republic of,Yemen, Republic of
docna: 0: docna: Are Subjective Questions on Welfare
            Appropriate during Crises ? Evidence from the Onset of
            Conflict in the Republic of Yemen
docty: Policy Research Working Paper
abstracts: cdata!: Subjective questions on welfare, such as
            satisfaction with particular welfare dimensions, are
            increasingly being used to measure changes in well-being
            during crises. Although subjective questions on welfare have
            well-known limitations, it is possible that analyzing
            changes in response to large events can address some of
            these concerns. However, this paper illustrates a new
            difficulty in interpreting changes in such measures in
            tumultuous circumstances. Specifically, crises can impact
            both the scale with which households report their subjective
            assessments and the behaviors from which deprivations are
            traditionally inferred, and it is unclear how well
            subjective measures align with traditional welfare metrics
            in such a setting. This paper demonstrates the importance of
            this issue following the onset of the conflict in the
            Republic of Yemen, which caused large declines in nearly all
            traditional measures of well-being. However, the findings
            show that households reported a large increase in
            satisfaction in the same welfare dimensions where
            deprivations increased.
display_title: Are Subjective Questions on Welfare
            Appropriate during Crises ? Evidence from the Onset of
            Conflict in the Republic of Yemen
listing_relative_url: /research/2022/09/33898138/subjective-questions-welfare-appropriate-during-crises-evidence-onset-conflict-republic-yemen
topicv3: Conflict and Violence m1327549 2433
publishtoextweb_dt: 2022-09-14T00:00:00Z
dois: 10.1596/1813-9450-10178
author_keys: 0: author_key: 1542515
model_extractions: Policy Research Working Paper 10178 Are Subjective Questions on Welfare Appropriate during Crises?

Evidence from the Onset of Conflict in the Republic of Yemen _Sharad Tandon_ Poverty and Equity Global Practice September 2022
Policy Research Working Paper 10178

# **Abstract**

Subjective questions on welfare, such as satisfaction with particular welfare dimensions, are increasingly being used to measure changes in well-being during crises. Although subjective questions on welfare have well-known limitations, it is possible that analyzing changes in response to large events can address some of these concerns. However, this paper illustrates a new difficulty in interpreting changes in such measures in tumultuous circumstances. Specifically, crises can impact both the scale with which households report their subjective assessments and the behaviors from which deprivations are traditionally inferred, and it is unclear how well subjective measures align with traditional welfare metrics in such a setting. This paper demonstrates the importance of this issue following the onset of the conflict in the Republic of Yemen, which caused large declines in nearly all traditional measures of well-being. However, the findings show that households reported a large increase in satisfaction in the same welfare dimensions where deprivations increased.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The author may be contacted at standon3@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
**Are Subjective Questions on Welfare Appropriate during Crises? Evidence from the Onset of Conflict in the Republic of Yemen**<sup>_∗_</sup>

Sharad Tandon<sup>_†_</sup> JEL classification: D12, E24, O12 Keywords: Subjective Welfare; Measurement; Conflict; Yemen

> _∗_ The author would like to thank Benu Bidani, Anna D‘Souza, Ruth Hill, Sailesh Tiwari, and Tara Vishwanath for comments on earlier drafts. The author would also like to thank audience members at the World Bank. The views expressed here are those of the author and may not be attributed to the World Bank. All remaining errors are attributable to the author.

> _†_ Poverty and Equity Global Practice of the World Bank, 1818 H St. NW, Washington, D.C. 20420, USA, standon3@worldbank.org
# **1. Introduction**

Subjective welfare measures, such as satisfaction with particular welfare dimensions, have become increasingly prevalent in the past decade.<sup>1</sup> Importantly, subjective welfare metrics have advantages over more traditional measures of welfare such as monetary poverty (e.g., Ravallion 2013), and sharp changes in peoples’ subjective perceptions of their welfare have been demonstrated to be particularly important to many instances of political change that have occurred across the world (e.g., Ianchovichina 2017; etc.). Furthermore, given their simplicity compared to more traditional consumption and employment-based metrics of deprivations, subjective metrics are often used to track changes in consumption and employment during times of crises.<sup>2</sup> Despite the importance of subjective welfare metrics and the greater ease in fielding subjective welfare modules during crises, there are well-known problems with using such measures. Most importantly, answers to subjective welfare questions depend on respondent-specific scales that are potentially not comparable across individuals, are potentially subject to frame-of-reference effects, and suffer from potential measurement error in ways that could differ from more traditional welfare metrics (e.g., Ravallion 2013).<sup>3</sup> However, even with these potential sources of biases that make the levels of subjective welfare metrics difficult to interpret, one common applied microeconomic technique to help better interpret subjective welfare metrics is to focus on sharp changes in response to arguably exogenous shocks.

This paper focuses on potential difficulties in interpreting changes in subjective mea-

> 1For example, a number of Sustainable Development Goals can be tracked using subjective welfare metrics.

> 2For example, traditional consumption modules typically capture the quantity and value of consumption of potentially hundreds of food and non-food items over reference periods varying from the past week to the past year. Such expansive modules are unable to be implemented via mobile phone. However, the Food Insecurity Experience Scale (FIES), which is a subjective measure of food access, is composed of an 8-question module that is amenable to phone surveys (e.g., Ballard 2013).

> 3A separate literature investigates the degree to which food security indicators, including subjective food security indicators, overlap with each other. There is growing evidence of a lack of alignment between subjective measures of food security and more traditional measures, such as undernourishment, obtained from the same respondents. However, there is also a lack of alignment between some traditional measures and other non-subjective food security indicators (e.g., Headey and Ecker 2013; Maxwell et al. 2014; Broussard and Tandon 2016; etc.). Similarly, Beegle et al. (2012) report subjective welfare assessments by expenditure per capita, and demonstrate that the two measures do not perfectly align. However, these differences could be driven by imperfections in either the subjective welfare question or in the measurement of expenditure.

2
sures of well-being. First, we develop a simple conceptual framework of whether an individual is satisfied with their consumption where we incorporate the salience of nonconsumption factors. The framework incorporates the well-known issues affecting subjective indicators of well-being- the possibility that scales used to answer such questions are individual-specific and can be impacted by frame-of-reference effects. However, this framework adds a new way in which subjective welfare metrics might be difficult to interpret- the possibility that the respondent-specific scales on which subjective welfare questions depend might not be stable over time. Specifically, we demonstrate how crises might cause the scales used by individuals to determine satisfaction to change at the same time that consumption might be changing, and it becomes an empirical issue as to whether subjective welfare metrics move in the same direction as consumption.

Second, we empirically investigate the ability of subjective measures of welfare to capture changes in traditional measures of deprivation in a crisis using the mostly non-violent capture of the Republic of Yemen’s capital. The capital was captured by Houthi forces in September 2014 while a nationally representative household survey was being conducted. The survey included both traditional consumption modules and a full subjective welfare module. Importantly, previous research has demonstrated that the capture led to immediate and large increases in consumption-based and employment-based measures of deprivations, and there is little evidence that the sampling of the survey was affected by the capture (e.g., Tandon 2019).

However, the change in subjective welfare metrics did not match the increase in deprivations captured by more traditional welfare metrics. The survey asked the household head to rate their satisfaction with a number of welfare dimensions individually (e.g., satisfaction with food consumption, income, etc.). Despite the large and immediate increase in traditional measures of deprivation following the capture, the subjective metrics of welfare moved in the _opposite_ direction. There was an increase of between 11 and 15 percentage points of household heads that reported to be satisfied with food consumption, income, and work, all of which were dimensions in which traditional metrics significantly worsened.

3
There were also similar increases in satisfaction with dimensions that were likely unaffected by the capture, and the increases in satisfaction were the strongest for households that had the largest and most immediate increase in deprivations. Overall, these patterns are consistent with household heads re-evaluating what it meant to be satisfied following the onset of conflict despite significantly lower expenditure and greater underemployment.

In addition to demonstrating that subjective questions on welfare can move in the opposite direction of more traditional measures of deprivations during a crisis, these results identify further problems with such measures. In particular, the results are sensitive to whom and to how the questions are asked. The spouse of the head was also asked about their satisfaction with each welfare dimension. Although the spouse also had an increase in satisfaction following the capture, the increase was smaller in magnitude and limited to a smaller set of welfare dimensions. Furthermore, a separate food security module asked a subjective question similar to the question on satisfaction with food consumption analyzed here. However, the change following the capture differed substantially between the two different questions, potentially due either to the slightly different wording or the order of the modules.

There are two important implications of these findings. First, although there are a number of well-known problems with subjective welfare metrics (e.g., Ravallion and Lokshin 2002; Beegle et al. 2012; Ravallion 2013; Ravallion et al. 2016), the results here highlight a new difficulty in interpreting subjective welfare indicators- instability of individual-specific scales over time. This particular difficulty with subjective welfare metrics can be very important to empirical researchers and the policy dialogue. As mentioned above, one of the typical empirical strategies used by applied researchers to sidestep some of the traditional difficulties in interpreting subjective welfare metrics is to analyze large changes in subjective welfare, sometimes in response to arguably exogenous events. For example many researchers have highlighted a significant change in subjective well-being leading up to the start of the Arab Spring (e.g., Ianchovichina 2017); or since it is difficult and costly to fully measure micro- and macro-nutrient consumption, food security researchers often ask about food coping strategies in rapid surveys during crises 4
(e.g., financial crisis, COVID-19 pandemic, etc.), some of which rely on individual-specific scales (e.g., FAO 2020; United Nations 2020; World Bank 2020; Adjognon et al. 2021; Amare et al. 2021; etc.). However, given that both well-being and individual specific scales could be moving at the same time, it might be difficult to precisely interpret these types of empirical strategies.

Second, although many difficulties in interpreting subjective welfare metrics have been highlighted before, it is an empirical issue to understand how difficult such measures are to interpret. Previous results have found substantial differences between subjective and traditional welfare metrics (e.g., Beegle et al. 2012; etc.). However, rather than simply making it more difficult to detect changes in consumption and employment, these results demonstrate that subjective welfare metrics can possibly produce estimates that move in the _opposite_ direction of traditional metrics that are needed to target critical social protection programs during crises.<sup>4</sup> Although subjective welfare metrics can still be informative to policy decisions, these results demonstrate that such metrics cannot always be substitutes for traditional metrics of deprivations, and it is important to utilize subjective welfare indicators alongside more traditional welfare metrics during especially tumultuous times.<sup>5</sup> The rest of the paper is structured as follows. Section 2 presents a conceptual frame-

> 4This finding is also related to Headey (2012), which found in a cross-country comparison that subjective assessments of food security improved between 2005 and 2007/2008 using two simple subjective questions about food security from the Gallup World Poll. This time period was a time of both rapid income growth and rapid increases in food prices. However, unlike the case described here, subjective assessments of food security and more traditional metrics are aligned in Headey (2012), where higher food prices are associated with worse subjective assessments of food security in the data and larger income growth is associated with better subjective assessments. The overall result is driven by the large income growth outweighing any negative impact of price increases; and the results are dependent on the inclusion of China and India, where price controls limited the increases in food prices and income growth was substantial.

> 5The problem of changing scales is not only a problem of subjective welfare questions. For example, the Reduced Coping Strategy Index (rCSI) is based on a number of questions about whether households engaged in particular coping strategies, and the frequency they used such a strategy if one was used. Some questions have individual-specific scales, such as whether households relied on less expensive foods; and other questions have standardized scales, such as whether households borrowed money to purchase food. However, during the conflict in the Republic of Yemen, both types of questions have been affected and the interpretation of the questions has likely changed relative to less tumultuous settings. For example, there might not be less expensive food available given poorly functioning food markets and supply chains, or being able to borrow money might be interpreted as a positive development because it indicates that lending markets work. Because of these difficulties in interpretation, these questions in particular have diverged from other food security measures (e.g., WFP 2019; Tandon and Vishwanath 2020; Tandon and Vishwanath 2021).

5
work; Section 3 presents a brief background on the context in the Republic of Yemen as the capital is captured by Houthi forces; Section 4 presents the data; Section 5 presents the empirical results; Section 6 demonstrates the sensitivity of the results to the identity of the respondent and the wording of the subjective questions; and Section 7 concludes.

# **2. Conceptual Framework**

Households choose a consumption bundle ( _C_ ) based on prices _p_ and wealth _w_ . Following the choice of consumption bundle, it is assumed that individuals are asked a simple ”yes” or ”no” question regarding whether they are satisfied with their consumption.

However, it is further assumed that non-consumption factors can affect this satisfaction and that the salience of these factors is allowed to vary relative to consumption factors. In this framework, salience refers to the degree to which an individual might weight the non-consumption factor relative to consumption in their decision. Specifically, an individual responds that she is satisfied with consumption if the following decision rule is positive: where _gi_ denotes a function by which individuals compare their consumption to their ideal consumption bundle, and it is assumed that higher consumption leads to individuals being closer to their ideal consumption at a diminishing rate (e.g., _g_<sup>_′_</sup> _>_ 0, _g_<sup>_′′_</sup> _<_ 0 for consumption below their ideal bundle).

However, it is assumed that consumption factors are explicitly compared to nonconsumption factors _Si_ , which capture all factors that would make it more likely an individual would report being satisfied for any level of consumption (e.g., _S ≥_ 0).<sup>6</sup> Importantly, the salience of non-consumption factors _S_ are allowed to vary relative to consumption factors through weights denoted by _σ_ (bounded between zero and one), where higher values of _σ_ imply a higher salience for _S_ .

> 6One example of information that could be contained in _S_ is perceptions of safety of individual _i_ , which might make somebody care less about consuming below their ideal consumption.

6
Lastly, _θ_ represents unexpected events that might affect an individual‘s reporting of satisfaction (e.g., current events, individuals‘ moods at the time of the decision, etc.). It is further assumed that _θ_ is drawn from a common distribution function _Fθ_ , where the unexpected component has a mean of zero and larger unexpected swings affecting reporting are less likely than smaller ones.<sup>7</sup> Given the unexpected component _θi_ , the probability that individual i will respond affirmatively that they are satisfied with their consumption and how that probability changes as consumption increases can be expressed as: (2) _φ_ = 1 _− Fθ_ [ _−E_ ( _Di_ )] This simple framework nests a number of different cases in how consumption is related to the likelihood of being satisfied with one’s consumption bundle. First, it is possible that consumption is positively correlated with the likelihood of being satisfied. In the case where consumption does impact satisfaction (0 _≤ σ <_ 1), then it follows that increasing consumption does lead to a higher likelihood of being satisfied (e.g., _δCδφi_<sup>_>_0).8</sup> However, three factors can attenuate the relationship and make it more difficult for subjective welfare metrics to detect changes in underlying consumption- a higher salience of non-consumption factors (e.g., larger _σ_ ), a higher variance of the unexpected swing in satisfaction not related to consumption, and lower sensitivity of satisfaction to being away from the ideal consumption bundle.<sup>9</sup> Second, it is possible that there is no relationship between actual consumption and

> 7Specifically, the density _fθ_ is increasing over the interval ( _−∞,_ 0) and decreasing over the interval (0 _, ∞_ ).

> 8The inequality follows from the assumption that _σ ∈_ [0 _,_ 1] and the assumption that _g′ >_ 0 for consumption below one‘s ideal consumption bundle.

> 9See Appendix 1a.

7
the likelihood of being satisfied with consumption. In the limiting case of only nonconsumption factors affecting reporting ( _σ_ = 1), the link between consumption and the probability of reporting satisfaction is obscured entirely, and increasing consumption does not change the probability of reporting satisfaction.

And lastly, it is possible that consumption and the likelihood of consumption can be negatively correlated following the onset of a shock. This overly simple framework has assumed that changes in _σ_ and changes in consumption are independent of each other. But there exists the possibility that the onset of a large event could cause both _σ_ and _C_ to co-move in opposite directions. In such a case, non-consumption factors would be weighted more at exactly the time that consumption and welfare might be changing, and one could find that subjective welfare measures move in exactly the _opposite_ direction of traditional consumption-based measures.

One potential example of a shock where both consumption and the weights that individuals place on non-consumption factors could simultaneously change is the onset of violence. Numerous settings demonstrate that consumption declines following the onset of violence;<sup>10</sup> and in such a case, households might suddenly place a higher emphasis on safety in their overall satisfaction with consumption. The key factor in whether satisfaction might actually increase despite consumption following is how large the increased emphasis on non-consumption factors might be relative to the size of the consumption decline.<sup>11</sup><sup>_,_12</sup> Importantly, the increased difficulty for subjective welfare metrics to detect changes in actual consumption is an issue highlighted by previous work (e.g., Ravallion 2013; etc.). However, to our knowledge, the possibility that the individual-specific scales on which subjective welfare questions rely might not be stable over time or in response to strong shocks has not been previously explored.

> 10See Blattman and Miguel (2010) for a survey of these findings.

> 11See appendix 1b for more detail.

> 12Another possibility is that the shock causes consumption and non-consumption factors to change in opposite directions. In such a case, even if the weight of non-consumption factors does not change, the shock could result in an increase in the likelihood an individual reports to be satisfied even if consumption declined. One example of this could be the toppling of an unpopular government. Consumption could decline, but that decline could be overwhelmed by an increase in satisfaction at the change in the political situation (captured by a change in _Si_ ).

8
# **3. The Capture of the Capital in the Republic of Yemen**

We investigate the ability of subjective questions on welfare to track more traditional measures of deprivation during a crisis using the capture of the Republic of Yemen’s capital at the end of September 2014 by Houthi forces. The assault began on September 21 and ended on September 24, with Houthi forces in charge of key government ministries (e.g., BBC 2018). The capture was a surprise, and was in part driven by Houthi forces being upset with a new constitution that was being finalized by the transition government following the Arab Spring (e.g., Baron 2015). The significant capture of territory by Houthi forces, that continued in early 2015 after the president fled from Sana’a, led to a coalition of nine countries in the region to join the conflict and support the internationally recognized government (e.g., World Bank 2017a).

The capture of Sana’a resulted in a number of shocks that affected households. The political uncertainty negatively impacted economic activity and investment (e.g., BBC 2015), and also negatively impacted safety (e.g., Human Rights Watch 2016). Additionally, the capture disrupted supply chains and led to immediate increases in prices of particular types of food items (e.g., Tandon 2019). Overall, the increase in prices and the decline in economic activity had unambiguously negative impacts on the population of Sana’a. Almost immediately after the capture, there were large declines in expenditure, food consumption, and stable employment, and a worsening in other welfare dimensions, all of which were not evident at the same time elsewhere in the Republic of Yemen (Tandon 2019).

Importantly, the government that was effectively overthrown was not popular among people in the Republic of Yemen, both in Sana’a and in the rest of the country. The president was not elected, but rather temporarily assumed the position following the Arab Spring and the resignation of the elected president (e.g., Baron 2015). Furthermore, there were large demonstrations against the government shortly after it took power.<sup>13</sup> Thus, although there was evidence of a decline in safety following the capture by Houthi forces (e.g., Human Rights Watch 2016), some portion of the population might have not been

> 13One example was protests over an elimination of a fuel subsidy earlier in the year that the government had to reinstate (e.g., World Bank 2017a).

9
opposed to seeing the government toppled even if they were not necessarily supporting the Houthi forces. These factors are also potentially important when investigating changes in subjective questions on welfare, particularly in dimensions aside from access to food and basic services.

# **4. Data**

We use the 2014 Household Budget Survey (HBS) to analyze whether changes in subjective welfare metrics were similar to the nearly universal worsening of traditional welfare metrics following the capture of Sana’a in 2014. The survey was conducted by the Central Statistical Organization of the Republic of Yemen, and was in progress as Houthi forces captured the capital. Given the capture was accomplished with relatively little violence, the survey was able to continue uninterrupted across the entire country. The survey was conducted over the entire calendar year with roughly equal numbers of households interviewed every month. The total sample size was 9391 households. But only 8772 households had precise information on the dates of the interviews, which is needed to identify whether the interviews were conducted prior to or following the capture of Sana’a.

In Sana’a, there were 314 households surveyed in September through December for which the survey was not completed before the capture of the city; and there were 1905 households where the entire survey was completed before the capture of the city. Importantly, given the sampling design, the survey can identify how households were affected by the capture. The survey was representative at the governorate/capital city level. Additionally, respondents were randomized over space and time, and there were no observable differences in difficult-to-adjust characteristics immediately after the capture of the capital, which might have happened if the onset of the conflict affected the sampling of the survey (Tandon 2019).

The 2014 HBS was a multi-purpose household survey that captured standard information on household consumption, employment, education, health, etc. However, the survey also included a thorough module on subjective welfare. For each of 13 dimen10
sions, the survey separately asked the household head and the head’s spouse about their satisfaction. Each respondent could choose “very satisfied,” “fairly satisfied,” “not very satisfied,” or “not at all satisfied,” with each welfare dimensions. Out of the 13 dimensions reported, we focus on the five dimensions that correspond to the consumption and employment-based metrics that significantly declined following the capture of the citysatisfaction with food consumption, income, work, life, and health.<sup>14</sup> Additionally, there was another subjective question on welfare in a separate food security module that was very similar to the question on satisfaction with food consumption. Specifically, the most senior woman in the household was asked whether there was a time in the past week that the household “did not have enough food or money to have food.”<sup>15</sup> If the respondent answered affirmatively, then the respondent was asked if the household engaged in each of five common food coping strategies in the past week, and the number of days in which those coping strategies were employed.<sup>16</sup> Table 1 presents summary statistics of the share of households that reported each level of satisfaction separately for the head and spouse. The vast majority of respondents had low levels of satisfaction- this was especially true for satisfaction with food, life, and health. These satisfaction levels likely reflect the decline in living standards over the decade prior to the survey. Poverty significantly increased in the Republic of Yemen since 2005, in part driven by the political uncertainty of 2012 and the sharp increase in food prices beginning in 2011 (e.g., WFP 2012; Dang and Ianchovichina 2016; World Bank 2017a).

Somewhat surprisingly, subjective welfare was similar in the capital city to the rest of the Republic ofYemen despite the fact that the capital city had significantly lower poverty

> 14Although the focus is on the five dimensions described in the text, the appendix further reports the change in all remaining dimensions captured by the survey. Other dimensions include satisfaction with one’s education level, dwelling, friends, family, freedom in controlling your life, neighborhood/city/community, ability to help others, and security. All results are qualitatively identical for changes in these other dimensions aside from security.

> 15In Sana’a, 13.4 percent of households answered affirmatively to not having enough food in the past week; and in the rest of the Republic of Yemen, 7.8 percent of households answered affirmatively to not having enough food in the past week.

> 16The coping strategies included relying on less preferred and less expensive food, borrowing food or relying on help, limiting portion sizes of meals, giving priority to small children to eat, and to reduce the number of meals eaten in a day.

11
rates, underemployment, and food insecurity (e.g., WFP 2012; World Bank 2017a; etc.). Furthermore, there was a broad agreement between household heads and spouses in the average levels of satisfaction, which is also slightly surprising given significant evidence of intra-household inequities (e.g., World Bank 2012).

However, there is an important data issue to highlight that might affect estimating the impact of capturing Sana‘a. Ramadan occurred primarily in July in 2014, and this might have biased peoples’ subjective assessment of their welfare given different consumption patterns and household choices during that month (e.g., Jolliffe and Serajuddin 2012; World Bank 2017a). Thus, baseline specifications exclude households surveyed during Ramadan.<sup>17</sup><sup>_,_18</sup>

# **5. The Impacts of the Capture on Subjective Assessments of Welfare**

Using the 2014 HBS, we estimate how much more the share of household heads reporting to be either “very satisfied” or “fairly satisfied” increased in Sana’a relative to the rest of the Republic of Yemen following the capture of the city. Specifically, for each welfare dimension, we estimate the following specification: where _Satisfaction_<sup>_j_</sup> _i_<sup>isanindicatorequalingoneiftheheadofhousehold</sup><sup>_i_iseither</sup> very or fairly satisfied with dimension _j_ ; _Sana_<sup>_′_</sup> _ai_ is an indicator if the household lives in the capital city; _Posti_ is an indicator if the household completed their final interview before Sana’a was captured by Houthi forces; and _Xi_ is a set of control variables.<sup>19</sup> The estimate of _α_ is by how much the share of households satisfied with welfare dimension

> 17However, all results are qualitatively identical when the excluded households are included, and these estimates are included in the Appendix.

> 18This reduces the total sample size from 8772 households to 8006.

> 19Controls include household size, and indicators equaling one if the household was urban, the head was married, whether the head ever attended primary school, whether the head smoked, whether the head chewed qat, whether there were children in the household, whether the house was built with stone, whether the roof of the house was reinforced concrete, whether the roof of the house was made of wood, whether the roof of the house was made with mud, whether the floor of the house was made with concrete, whether the floor of the house was made with tile, and whether the house had a flush toilet.

12
_j_ increased in regions other than Sana’a; and _γ_ denotes by how much more the share of households satisfied with dimension _j_ increased in Sana’a relative to the rest of the Republic of Yemen.

The results of specification (5) are reported in Table 2. Despite the significant declines in expenditure, employment, and food consumption immediately following the capture of Sana’a (e.g., Tandon 2019), there was a large increase in the share of households with heads that were satisfied with nearly all welfare dimensions in Sana’a relative to the rest of the Republic of Yemen. In particular, the share of households satisfied with food consumption increased by 13 percentage points in Sana’a relative to the rest of the Republic of Yemen; the share satisfied with work and income increased by 11 and 15 percentage points respectively; the share of households satisfied with their health increased by 7 percentage points; and the share of households satisfied with their life increased by 11 percentage points. Importantly, based on estimates of _α_ , there was little change in satisfaction in the rest of the Republic of Yemen in nearly all specifications and these patterns were primarily driven by large changes in Sana’a.

However, Table 2 also analyzes satisfaction with dimensions of welfare that were likely not affected by the capture. Each respondent was asked about their satisfaction with their education level, which was fixed long before the capture; and each respondent was asked about their satisfaction with their dwelling, which was largely unaffected given the quick and mostly non-violent capture of the city. Satisfaction with both dimensions increased in Sana’a relative to the rest of the country despite likely being unaffected by the capture. The increase in heads satisfied with their dwelling was 12 percentage points higher in Sana’a than the rest of the Republic of Yemen, and the increase in heads satisfied with their education level was 10 percentage points higher (although less precisely estimated).<sup>20</sup><sup>_,_21</sup> Combined, these results suggest that the scale with which household heads used to determine satisfaction significantly changed following the capture. For example, household

> 20See Appendix 2 for the changes for all additional welfare dimensions captured by the survey.

> 21See Appendix 3 for estimates of the baseline specification that include households surveyed during Ramadan.

13
heads might have re-evaluated what it meant to be satisfied when the city had been captured and safety was more of an issue; or heads might have had an increase in satisfaction irrespective of the dimension after the fall of an unpopular government. Regardless of the exact reason, these results demonstrate that it might be difficult to interpret changes to subjective welfare questions during times of crises.

Importantly, the results are robust to a number of concerns. First, it is possible that the results are sensitive to the choice of the pre-period, and that there were other points in the year where satisfaction is significantly higher than following the capture of Sana’a. However, Table 3 demonstrates that the results do not depend on the choice of the months to include in the pre-period. Specifically, Table 3 re-estimates specification (5) for the satisfaction with food consumption (the dimension with one of the largest and most precisely estimated changes), and varies the length of the pre-period. Column (1) includes the entire sample, while each subsequent column restricts the pre-period by an additional month (e.g., the second column includes all months after February, etc.). In all cases, the magnitude and the precision of the estimates are similar.<sup>22</sup> Second, we demonstrate that satisfaction was not trending differently leading up to the capture of the city and the results are likely driven by the capture of Sana’a. We re-estimate specification (5), but we break up the changes in Sana’a relative to the rest of the country by month. Figure 1 reports by how much larger the change in satisfaction was in Sana’a relative to the rest of the country for each month separately (relative to the base month of May), and further reports the bounds of the 95 percent confidence interval of each estimate.

In all specifications, there was little change in the share satisfied leading up to the capture in June and August.<sup>23</sup> None of the pre-capture estimates were statistically significant at the 95 percent significance level, and in no specification can the hypothesis that there were no pre-trends be ruled out at conventional significance levels (p-values

> 22The results are qualitatively identical for the other welfare dimensions as well.

> 23Households interviewed in July are excluded from the specification because of Ramadan, but results are qualitatively identical if these households are included and the change in satisfaction is estimated separately for July.

14
between 0.12 and 0.72).<sup>24</sup> The estimates began to diverge between Sana’a and the rest of the country in September, when the vast majority of households completed their final interview following the capture. The estimates continued to be different for the rest of the survey period between October and December, and were more precisely estimated than the change in September.<sup>25</sup> Third, it is possible that the changes in the subjective questions on welfare and the changes in deprivations were being driven by different segments of the population. For example, the least educated household heads had the most immediate and the largest declines in expenditure and stable employment while other households were better able to cope for the initial months (e.g., Tandon 2019). Thus, it is possible that the increase in satisfaction was driven primarily by households that were able to initially maintain their consumption while others were significantly suffering.

However, Table 4 demonstrates that those that had the largest and most immediate increases in deprivations were those that had the highest increases in satisfaction. Specifically, we re-estimate the baseline specification separately for households where the head did and did not finish primary school. The estimates demonstrate that both types of households had an increase in satisfaction. However, households where the head did not finish primary school had larger increases in satisfaction for all dimensions.<sup>26</sup>

# **6. Potential Sensitivity to the Identity of the Respondent and the Wording of Subjective Questions**

In addition to analyzing changes in satisfaction of the household head, we also investigate differences in responses between household members to the same subjective questions. As discussed in the data section, both the household head and the head’s spouse were asked about their satisfaction with each welfare dimension. Table 5 re-estimates

> 24P-values are from hypothesis tests of the joint significance of all pre-capture coefficients.

> 25Figure 1 reports estimates for the five dimensions that were likely strongly affected by the capture. See Appendix 4 for the estimates for the rest of dimensions that are reported in Table 4.

> 26See Appendix 5 for specifications that statistically test the difference in the impact the capture had by the household head’s years of schooling. For each additional year of schooling, the impact the capture had on satisfaction was attenuated. The coefficient on the triple interaction term of interest is statistically significant at the one percent level in three of the seven baseline specifications and statistically significant at the five percent level in one of the seven specifications.

15
the baseline specification but uses the responses of the household head’s spouse as the dependent variable.

Similar to the changes of the household reported in Table 2, all estimates for the head’s spouse in Table 5 continue to be positive. However, the estimates of the five dimensions that were likely most affected by the capture were smaller in magnitude and less precisely estimated, and one cannot reject the hypothesis that there was no change in satisfaction at conventional significance levels.<sup>27</sup> Interestingly, the increase in satisfaction for spouses was much stronger for welfare dimensions that were most likely to be unaffected by the capture (e.g., dwelling and education), where the change for spouses was similar to that of household heads.<sup>28</sup> There are many potential reasons why different respondents could have had different responses regarding subjective levels of satisfaction in this setting. The differences could have reflected differences in the scales used by respondents, or they also could have been the result of differences in intra-household outcomes.<sup>29</sup> Additionally, different respondents might have interpreted the questions differently- some might have considered the household head’s work or income in the question, while others might have only interpreted their own individual work either inside or outside of the household. Regardless of the exact reason, the strong differences in the responses between household members in this turbulent setting further raises questions as to who should be asked subjective welfare questions during crises and also raises questions on how to interpret the responses.

However, in addition to comparing responses of different household members to the same subjective questions on welfare, we further investigate changes to other subjective questions where heterogeneous scales might also change following the capture of Sana’a.

> 27See Appendix 6 for specifications that statistically evaluate the difference between the responses of household heads and their spouses. Specifically, the specification pools the responses, and compares how much larger the relative increase in Sana’a was for head than the relative increase in Sana’a for spouses. Out of the five separate specifications, the triple interaction term was statistically different at the one percent level in one specification, statistically different at the five percent level in one specification, and statistically different at the 10 percent level in two specifications.

> 28Appendix 6 demonstrates that one cannot reject the hypothesis that the change was identical for the head and spouse for dwelling and education.

> 29The subjective question on satisfaction with welfare dimensions inquires about the individual respondent’s satisfaction, while the question to the most senior woman about having enough food refers to the household.

16
In particular, we compare the change in the satisfaction with food consumption of both the head and the spouse to the change in the share of households where the most senior woman (most often the head’s spouse) claimed to “not have enough food or money to have food.” Similar to the question on satisfaction, the interpretation of “enough” might vary from household to household, and a respondent’s interpretation of “enough” might have changed following the capture of Sana’a. Also, there could have been a difference in opinion between household heads and the most senior woman in the house about the adequacy of food consumption.

Table 6 re-estimates the baseline specification, but uses the indicator for whether the household had enough food. Column (1) demonstrates that there was an increase in the share of households where the most senior woman reported not having enough to eat following the capture of the city of 8.5 percentage points. Columns (2) and (3) respectively limit the sample to households where the head did and did not finish primary school. The results were stronger for less educated households that were more strongly and immediately impacted by the capture. These patterns are consistent with previously estimated changes in food security following the capture of Sana’a (e.g., Tandon 2019).<sup>30</sup> Furthermore, given that each household answered both the question on satisfaction with food consumption and whether the household had enough to eat, we are able to identify households where the two metrics did not align well. Specifically, we use an indicator equal to one if the household did not have enough food but did report being satisfied with food consumption as the dependent variable and re-estimate the baseline specification. We use separate indicators for alignment between the head’s satisfaction with food consumption and whether the most senior woman in the house claimed there was enough food, and for alignment between the spouse’s satisfaction with food consumption and whether there was enough food.

The estimates in columns (4)-(9) of Table 6 demonstrate that there was a significant increase in the share of households where the two indicators did not align. Column (4) demonstrates that the misalignment with the household head’s satisfaction with food

> 30See Appendix 5 for specifications that allow the impacts to vary by years of schooling of the head.

17
consumption increased by 6.8 percentage points in Sana’a relative to the rest of the Republic of Yemen following the capture of the city. Importantly, there were only 0.5 percent of households in Sana’a before the capture where these two metrics did not align when using the head’s satisfaction with food consumption. However, following the capture, this share in Sana’a increased nearly 10 times to nearly five percent.<sup>31</sup> Consistent with the empirical patterns described above, columns (5) and (6) demonstrate that the impact is being driven by the least educated households. The least educated households were the ones with the largest increase in satisfaction, and also the households with the largest increase in the most senior woman reported that they did not have enough to eat in the past week.

Interestingly, columns (7)-(9) of demonstrate a qualitatively identical pattern when analyzing the alignment between the spouse’s satisfaction with food consumption and the most senior woman’s perception of whether there was enough food in the past week. Thus, even in the case when the identity of the respondent was identical in the vast majority of cases (spouse and most senior woman), the answers to the two similar but differently worded subjective questions on food consumption diverged following the capture.

Combined, these results suggest that in addition to the sensitivity to the identity of the respondent, the impact of crises on the scales used by respondents could be sensitive to the wording of the question and the ordering of the survey modules. When asked about satisfaction with food consumption, households seemed to adjust their scale and the answers were in contrast to traditional metrics capturing deprivations. However, when the same households were asked whether they had “enough” food to eat, the question gave a more similar answer to traditional metrics of deprivation.<sup>32</sup> These findings further illustrate the importance of standardizing scales across households to aide in the interpretation of the results.

# **7. Conclusion**

> 31The rest of the 6.8 percentage point difference is driven by a small change in the rest of the country and the inclusion of control variables in the empirical specification.

> 32But the magnitude of the estimate was lower and the estimate was less precise when compared to the changes in traditional metrics.

18
These results demonstrate that subjective questions on welfare that rely on heterogeneous scales across households might be ill-suited to precisely track how consumption and employment are changing during and after a crisis. These results suggest that in the case where a reference scale is not explicitly provided to a household, the scale itself could change following a crisis making it difficult to even identify the sign of changes in actual consumption, employment, and access to basic services.

Furthermore, the results demonstrate that the responses are sensitive to the specific wording of the subjective welfare question, where similar but different wording of scales (“satisfied” versus “enough”) can result in different estimates of the impact of a crisis. And lastly, the results are also sensitive to who in the household responds to the subjective questions on welfare. Overall, the results demonstrate the importance of standardizing scales in questions across households, or to focus on questions where the scales are identical across households.

The results are particularly relevant for the current high-frequency surveys being used to track welfare through the COVID-19 pandemic (e.g., World Bank 2020), and will also likely become more relevant in other settings as the need for high-frequency monitoring in conflict and fragile environments continues to increase (e.g., World Bank 2017b). Subjective metrics are often easier to collect than entire consumption modules; and including subjective welfare metrics that are official indicators tracking progress in achieving the Sustainable Development Goals in the high-frequency surveys can allow a comparison to the last household survey and to international benchmarks. Because of these advantages, many of the high-frequency surveys currently being conducted to track the COVID-19 pandemic include questions in which the scale is not explicitly introduced, and it might be difficult to interpret these changes to these questions as the pandemic evolves.

However, there are a number of questions that this analysis is unable to address. First, these are all short-term effects of the capture. The survey ended three months after the capture of the city, and it is unclear how long the impacts might last. It is possible that satisfaction levels only improved in the short run in Sana’a immediately 19
following the capture, and that over time the satisfaction levels might more closely track more traditional measures of deprivations.

Second, it is difficult to extrapolate to the current-day. Although satisfaction increased immediately following the capture of Sana’a, the escalation of the conflict in March 2015 created what the UN identified as the worst humanitarian crisis in the world (e.g., OCHA 2019). Thus, it is highly unlikely that satisfaction continued to increase or remain stable in Sana’a, or anywhere else in the country, following the repeated and devastating shocks caused by the conflict.

20