metadata: authors: 0: author: Beegle, Kathleen
1: author: De Weerdt, Joachim
2: author: Friedman, Jed
3: author: Gibson, John
count: Tanzania
docna: 0: docna: Methods of household consumption measurement
            through surveys : experimental results from Tanzania
docty: Policy Research Working Paper
subtopic: Economic Theory & Research,Urban Housing and Land Settlements,Industrial Economics,Urban Governance and Management,Urban Housing,Educational Sciences,Poverty Impact Evaluation,Poverty Monitoring & Analysis,Economic Growth,Municipal Management and Reform,Inequality,Poverty Assessment,Poverty Diagnostics,Poverty Lines,Small Area Estimation Poverty Mapping
teratopic: Macroeconomics and Economic Growth,Poverty Reduction
abstracts: cdata!: Consumption expenditure has long been
            the preferred measure of household living standards.
            However, accurate measurement is a challenge and household
            expenditure surveys vary widely across many dimensions,
            including the level of reporting, the length of the
            reference period, and the degree of commodity detail. These
            variations occur both across countries and also over time
            within countries. There is little current understanding of
            the implications of such changes for spatially and
            temporally consistent measurement of household consumption
            and poverty. A field experiment in Tanzania tests eight
            alternative methods to measure household consumption on a
            sample of 4,000 households. There are significant
            differences between consumption reported by the benchmark
            personal diary and other diary and recall formats.
            Under-reporting is particularly relevant in illiterate
            households and for urban respondents completing household
            diaries; recall modules measure lower consumption than a
            personal diary, with larger gaps among poorer households and
            households with more adult members. Variations in reporting
            accuracy by household characteristics are also discussed and
            differences in measured poverty as a result of survey design
            are explored. The study concludes with recommendations for
            methods of survey based consumption measurement in
            low-income countries.
geo_reg_key: 80475,80458,80473
repnme: repnme: Methods of household consumption
            measurement through surveys : experimental results from Tanzania
display_title: Methods of household consumption
            measurement through surveys : experimental results from Tanzania
listing_relative_url: /research/2010/12/13233228/methods-household-consumption-measurement-through-surveys-experimental-results-tanzania
topicv3: Economic Growth m1326241 471,Poverty m1326687 695,Poverty Measurement and Analysis m1326941 707,Education m1327048 470,Land and Housing m1327648 817
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Macroeconomics and Economic Growth,Poverty Reduction
no_of_pages: 49
model_extractions: WPS5501 Policy Research Working Paper 5501 Methods of Household Consumption Measurement through Surveys Experimental Results from Tanzania _Kathleen Beegle Joachim De Weerdt Jed Friedman John Gibson_ The World Bank Development Research Group Poverty and Inequality Team December 2010
Policy Research Working Paper 5501

# **Abstract**

Consumption expenditure has long been the preferred measure of household living standards. However, accurate measurement is a challenge and household expenditure surveys vary widely across many dimensions, including the level of reporting, the length of the reference period, and the degree of commodity detail. These variations occur both across countries and also over time within countries. There is little current understanding of the implications of such changes for spatially and temporally consistent measurement of household consumption and poverty. A field experiment in Tanzania tests eight alternative methods to measure household consumption on a sample of 4,000 households. There are significant differences between consumption reported by the benchmark personal diary and other diary and recall formats. Under-reporting is particularly relevant in illiterate households and for urban respondents completing household diaries; recall modules measure lower consumption than a personal diary, with larger gaps among poorer households and households with more adult members. Variations in reporting accuracy by household characteristics are also discussed and differences in measured poverty as a result of survey design are explored. The study concludes with recommendations for methods of survey based consumption measurement in low-income countries.

This paper is a product of the Poverty and Inequality Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at kbeegle@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Methods of Household Consumption Measurement through Surveys: Experimental Results from Tanzania**

Kathleen Beegle, World Bank Joachim De Weerdt, EDI Tanzania Jed Friedman, World Bank John Gibson, University of Waikato

**Keywords:** consumption, expenditure, survey design **JEL** : C83, D12

* This work was supported by the World Bank Research Committee. We would like to thank two anonymous referees, seminar participants at IFPRI, the ASSA annual meetings, and the Survey Design and Measurement in Development Economics conference for very useful comments. All views are those of the authors and do not reflect the views of the World Bank or its member countries.
# **1. Introduction**

Household consumption is typically the core concept at the center of any attempt to measure living standards, inequality and poverty in the developing world.<sup>1</sup> Measures of consumption are derived almost exclusively through survey, however a review of survey practice will quickly reveal extensive variation over several dimensions, such as the method of data capture (diary versus recall), the level of respondent (individual versus household), the reference period for which consumption is reported (anywhere from 3 days to one year) and the degree of commodity detail (from less than 20 items to over 400 items). Variations occur both across countries and over time as statistical offices alter survey designs, with little understanding of the implications of such changes for accurate and consistent measurement of consumption. Such variation hampers both crosscountry studies of poverty and inequality measures as well as the measurement of welfare trends within a country (Lanjouw and Lanjouw, 2001).<sup>2</sup> For example, the national living standards survey in South Africa found a decline in real food expenditure from 2000 to 2005 even though the same data sources indicated increases in overall consumption and declines in self-reported hunger/food insecurity. This contradictory finding is largely attributed to a change in survey design from recall to diary for food consumption measurement in 2005 (Yu, 2008).

Changes in survey design, if they matter, need to be considered in analysis of trends in consumption or poverty over time. By extension, analysis that pools surveys of different types to study economic growth and inequality should take these differences seriously. Consider, for example, the widely used UNU/WIDER World Income Inequality Database (WIID) on income distribution and inequality. Concerns about consistency in such cross-country databases focus on whether the source surveys are income or consumption surveys (such as Atkinson and Brandolini, 2001, and Pinkovskiy and Sala-i-Martin, 2009), and the comprehensiveness of the consumption measure (Chen and Ravallion, 2008), but not on the implications of variation in consumption survey design. Although we do not attempt a complete inventory of the consumption surveys in this database, for the data of six African countries (chosen on the basis of our past experience or access to survey materials: Côte d`Ivoire, Ghana, Malawi, South Africa, Tanzania, and Zambia)

> 1 Conceptual and practical reasons favoring consumption expenditures over income as a welfare measure are discussed in Deaton (1997). Empirically, statistics offices in a majority of the developing countries, with metadata in the United Nations Handbook of Poverty Statistics (UN 2005), use either consumption expenditures solely or in combination with income as their welfare measure. The only region with a high reliance on income surveys is Latin America, although even there increased use is made of expenditure surveys for poverty measurement.

> 2 In addition, the large and growing gaps between micro and aggregate estimates of household consumption hinder assessments of global progress in poverty reduction and the effect of economic growth on that process (Ravallion, 2003; Deaton, 2005). The extent to which possible measurement flaws in survey design format may contribute to this gap is unknown.

2
five of the six countries had made a major change in the design of the module collecting consumption data in their surveys.<sup>3</sup> Only South Africa had not changed the consumption survey design across the two years in the database, although they did in 2005. As we will show in this paper, these design changes can result in large changes in both mean consumption and distributional measures.

Our findings come from a study specifically intended to provide more comprehensive evidence on the sensitivity of consumption estimates to varying survey designs. We developed eight alternative consumption questionnaires that were randomly distributed across 4,000 households surveyed in Tanzania. These eight designs all represent common approaches to consumption measurement that have been implemented in numerous settings. We consider the various strengths and weaknesses of each approach to survey-based consumption measurement, including the types of errors that are likely to be more prominent. While in this experiment there are no validation data available for the study households, one resource intensive variant – a personal consumption diary with intensive and frequent supervision – is believed the most accurate and in many ways represents a “gold standard” for field based consumption measurement in survey format. We investigate how alternative survey designs compare with our benchmark, with a focus on the patterns of over or under-reporting of consumption expenditure with respect to household characteristics, and poverty and inequality measures.

The paper is organized as follows. A review of related literature is presented in Section 2. Section 3 describes the experiment, the Survey of Household Welfare and Labour in Tanzania, and construction of the consumption aggregates. Section 4 discusses the results. We compare estimates of total and component consumption for each of the modules, explore how household characteristics affect reporting, and consider the impact of alternative survey designs on inequality and poverty measures. Section 5 discusses the resource requirements for each survey design. The final section reviews the lessons learned herein.

> 3 The variation in the type of consumption survey for these countries included: bounded recall and usual consumption (5 surveys Côte d`Ivoire), bounded recall and usual consumption (3 surveys in Ghana), bounded recall and diary (2 surveys in Malawi), usual consumption and diary (3 surveys in Tanzania), and bounded recall and diary (2 surveys in Zambia). The terms bounded recall, usual, and diary are explained in the next section.
# **2. Issues in the Measurement of Consumption through Surveys**

There is a large literature on methods of collecting consumption data through household surveys.<sup>4</sup> Much of the evidence from survey experiments was generated in developed countries, while studies from lowincome settings typically draw on surveys not specifically designed to systematically compare contrasting methods.<sup>5</sup> The specific experimental design described in the next section was motivated by the desire to carefully investigate the divergences across the main methods of consumption data collection in common use today in a developing country setting. The primary dimensions in which these methods vary are three: the use of diary vs. recall, the level of aggregation or detail in the commodity list, and the reference period. Some existing evidence of the measurement consequences of the three aspects is reviewed in turn; lengthier discussions of this literature can be found in Gibson (2006), Deaton and Grosh (2000), and Scott and Amenuvegbe (1991).

There are many potential sources of reporting error that lead survey estimates of consumption to deviate from actual consumption. Perhaps the most commonly discussed in the literature is recall error, where a household under-reports true consumption over the period of recall due to faulty memory. Presumably the longer the period of recall, the greater the cognitive demand on the respondent and the greater the divergence between reported and actual consumption. Several studies have documented that, all else equal, the longer the period of recall, the lower the reported consumption per standardized unit of time. Closely related to recall error is telescoping, where a household compresses consumption that occurred over a longer period of time into the reference period asked and thus reports consumption greater than the actual value.

A third important source of error is the inability to accurately capture individual consumption by household members that occurs outside the purview of the survey respondent. Clearly this inability may be more significant for certain types of consumption goods such as transport, telecommunication, or meals outside the home. The degree of inaccuracy likely also increases with the number of adult household members and the diversity of their activities outside the home.

Other sources of error with no obvious direction of bias include rounding error and cognitive errors that result from consideration of hypothetical consumption constructs such as consumption in a “usual” month which may present additional cognitive demands compared to a definitive recall period in

> 4 This section does not review the literature from cognitive psychology on the effects of question framing and other issues related to survey design generally; we focus instead on issues and evidence specific to household consumption expenditure.

> 5 The developed country studies often compare recall to diary methods (see, for example, Neter, 1970; Neter and Waksburg, 1964; McWhinney and Champion, 1974; Kemsley and Nicholson, 1960; Gieseman, 1987). More recent work examines other dimensions such as bracketing and question wording/prompting, as in Comerford et al. (2009).
the immediate past. All of these sources of error can be considered non-deliberative or unintentional. On the other hand, intentional misreporting can arise if there is perceived social pressure to appear either “wealthier” or “poorer” to the interviewer and thus the social context of the interview may also be relevant. Finally the interviewer and respondent could suffer from fatigue in the latter part of the interview when surveys are long, or lack integrity when supervision is limited.

The evidence on the implications of consumption survey design which we review is fragmentary and typically sheds light on only one aspect of survey design while ignoring resource implications of the variants considered. Thus previous work at best either explicitly or implicitly investigates how only one source of reporting error varies with design. For example, systematic variation in the length of recall can explore the net relative effect of recall and telescoping errors at different reference periods. A challenge to this previous research is the lack of a consumption benchmark, making it difficult to conclude one design is more accurate than another since there are no data on actual consumption to validate the survey estimates. Scanner data may allow validation in certain contexts in developed countries but will be unavailable for many years in developing countries where many goods are either home produced or bought in informal markets. Prior attempts to form a benchmark for consumption surveys in developing countries have not been fully successful. For example, in India’s NSS experiments enumerators visited households every day and gave volumetric containers for measuring food consumption but for some foods less than two-thirds of respondents used the containers and many respondents did not use the daily diary given to them (NSSO 2003).<sup>6</sup>

# _Diary or Recall_

In a non-negligible number of developing countries, including Brazil, China, and many countries of Central Europe and Central Asia (where literacy rates are high), there exists a tradition of diary-based collection of consumption data – at least for certain consumption items such as food. This contrasts with the more common practice in Living Standards Measurement Study (LSMS) surveys and other multitopic survey instruments to base data collection on recall. It is far from clear to what extent diary and recall surveys are substitutes, or if there are biases introduced by switching from one to the other. In theory, entries are recorded in the diary by the respondent household at the moment of consumption or soon thereafter. However, in practice it can often be the case that interviewers assist in completing the

> 6 National Accounts (NA) estimates of household food consumption are also not a plausible source of validation data, at least in developing countries where NA data may also suffer from various sources of inaccuracy. For example, in comparisons between survey and NA estimates of food consumption in India, both survey and national account statisticians have concluded that discrepancies more likely reflect errors in the national accounts (Minhas 1988; Kulshreshtha and Kar 2005).
diary, effectively blurring the line between consumption data collected by diary and by recall interview. This mediation by interviewers is especially likely since most diary surveys instruct interviewers to return every few days to update the diary in cases where it may not be completed by household members, such as in households with no literate adults.

While well-implemented diary surveys might be expected to yield higher (and presumably closer to actual) levels of consumption, experimental evidence for this from developing countries is fragmentary. In urban Papua New Guinea, Gibson (1999) found that mean total expenditure per capita was 14 percent higher (and food consumption 26 percent higher) using a diary rather than a recall survey. However, preliminary comparisons of consumption from the Bosnia and Herzegovina LSMS (recall) survey with the household budget survey (diary) in 2004 show levels to be similar (World Bank, 2006).

These studies pertain to household-level diaries and thus do not address an important source of mis-measurement – the difficulty of a sole respondent to perfectly capture total household consumption. Personal diaries are generally considered better for obtaining complete household expenditure or consumption data because it is unusual, in most societies, for any one household member to know the expenditure/consumption of every other member, especially on items such as alcohol, tobacco, daily travel, personal toiletries, daily newspapers/magazines, and meals (especially snacks and lunches) eaten outside the home. In Russia, as part of an effort to study the reliability of the Household Budget Survey, a random sample of households in the 3<sup>rd</sup> quarter of 2003 were assigned personal diaries rather than the household diary. The personal diary yielded expenditure levels which were 6-11 percent higher than a household diary (World Bank, 2005). However, the personal diary was plagued with non-respondent problems in this experiment; about 54 percent of households assigned the personal diary did not complete it. Personal diaries also create the possibility of inadvertent double-counting of items consumed in common but reported separately by two or more individuals, and so care must be taken with this method. We discuss this further in the next section.

While the perfect diary experiment is, essentially, the benchmark for collecting consumption data through survey, it is difficult to extrapolate from experiments between diary and recall consumption modules if the implementation of the diary in the experiment does not reflect the reality of fielding a diary in developing or transition countries. It may be that in the case of illiterate (or simply unmotivated) respondents, a 7-day diary becomes a 7-day recall survey if the information has to be obtained orally by the interviewer at the end of the period. The implications of variation in literacy, motivation, and other factors, although not well-documented, implies that it can be quite difficult to conduct a high-quality diary survey, regardless of issues related to respondent recall bias.

In their study of measurement errors in food consumption, Ahmed et al. (2006) observe problems in the Canadian Food Expenditure Survey diary related to infrequency and diary exhaustion and conclude
that the “superiority of the diary data may not be as obvious as the literature suggests.” In Tanzania, the mean number of transactions in the official Household Budget Survey diary declined by 10 percent from the 1<sup>st</sup> to 3<sup>rd</sup> survey month, and by 27 percent from the 1<sup>st</sup> to 13<sup>th</sup> (last) survey month (NBS, 2002), suggestive of inconsistent supervision of interviewers and/or diary fatigue on the part of respondents. In Malawi, nearly 40 percent of 10,698 households in the 1997/98 national household survey were judged to have incomplete or unreliable expenditure data due to poor maintenance of the diary (National Economic Council, 2000); this drastic waste of survey resources prompted the switch to recall consumption methods in the subsequent national household surveys in 2004 and 2010. In the case of the Russian Household Budget Survey, the pattern of non-response to the household diary, especially among wealthier households, generates extreme sample weights which Gibson and Poduzov (2003) conclude are inefficient from both a statistical and a budget perspective.

# _Level of Detail in Consumption Questionnaires_

The number of items about which data are collected is one of the central issues in designing a consumption module. National consumer price index baskets may have in excess of 300 items. To reduce the burden on respondents and survey activities in general, shorter lists of items are created by focusing on those items that represent the greatest share of consumption and aggregating other, less important, items into categories. At present, the number of consumption items (or categories) for which data are collected from households in LSMS surveys ranges from 37 to 305, with the mean being 137 and the median 130: the mean number of food items alone is 75.<sup>7</sup> This level of detail on consumption is thought to be important as it prompts respondents to remember more completely and accurately their consumption. But, the costs may be high: longer interview time, greater respondent fatigue and higher non-response.

While there are surprisingly few studies from developing countries evaluating the accuracy of shorter versus longer lists — none were identified from any African countries — it appears that more disaggregation and longer lists of items result in higher levels of recorded consumption. In El Salvador, data from 1994 showed that the longer, more detailed questions on consumption resulted in an estimate of mean household consumption that was 31 percent higher than that from the condensed version of the questionnaire (Jolliffe, 2001). This resulted in much higher poverty estimates from the short

> 7 Figures are calculated from the Comparative Living Standards Project data base. Note that although 100+ consumption items seems a large number, many Household Budget Surveys (HBS) will cover even more food items in the diary. HBSs usually do not collect much additional information on household socioeconomic conditions, whereas the LSMS surveys do, thereby necessitating both recall and food lists that are not exhaustive of all foods. The objective of an LSMS survey is consistent measurement of total consumption for each household in the sample, rather than accurate measures of commodity-specific consumption for the population.
questionnaire, and the geographic distribution of relatively poor persons was significantly different, suggesting some re-ranking of households based on survey design. Pradhan (2001) found that the shorter consumption module in the _Susenas_ questionnaire in Indonesia results in average consumption that is 1220 percent below the levels from the longer consumption module, although the ranking of households based on observable characteristics was not any different. Earlier experimental work, which led to the introduction of the short-form consumption module in the _Susenas_ , also found that a reduction in the number of items from more than 100 to 15 decreased consumption but did so proportionally, thus not reranking households (World Bank, 1993). In Jamaica, the shortened consumption modules produced mean per capita consumption results that were about 20 percent lower than the standard modules (Statistical Institute and Planning Institute of Jamaica, 1994). In Ecuador, where two versions of the food module were piloted, the ratio of food expenditures in the long module to the short module was 1.67 (Steele, 1998). Again there appear to be plenty of examples where aggregated commodity lists result in a lower mean and presumably less accurate consumption measure. What is not clear is the relative savings in survey resources, and the effects on distributionally sensitive measures, such as inequality, from such a design.

# _Reference Period_

In recall modules there are also important issues associated with the choice of reference period to be used. The direction of bias introduced by lengthening or shortening the reference period is ambiguous _a priori_ . Respondents may have difficulty recalling consumption expenditure with longer reference periods due to diminished capacity to remember (memory lapse). On the other hand, short recall periods may produce over-estimates (telescoping errors) if respondents include consumption/expenditures just outside the reference period. The extent to which expenditures are misreported in these ways is presumed to depend on the item in question and the characteristics of the household.

In India, there has been debate in recent years as to the impact on measured poverty and inequality of having altered the reference period over which (recall-based) consumption data has been collected in two key rounds of the National Sample Survey (see Deaton and Kozel, 2005). Lanjouw (2005) argues that, in Brazil, the application of an inappropriately short reference period over which purchases of foods are recorded in the 2002 _Pesquisa de Orçamientos Familiares_ consumption survey diary might account for the surprisingly high frequency of households who report zero consumption of certain key food items.

In Ghana, Scott and Amenuvegbe (1991) ran a well-documented experiment for 144 households on the impact of recall duration on reported expenditure (not consumption value) of the 13 most
frequently purchased items. They find that each additional day of recall results in about 3 percentage points decline in reported daily expenditure.<sup>8</sup> Other experiments on recall period that were identified are all based on surveys in which the same household reports expenditures across two different recall periods _within_ the same survey. This is potentially problematic if there is conditioning bias due to households or interviewers imposing consistency between their responses.<sup>9</sup> Grosh et al _._ (1995) compare consumption estimates from the Ghana LSMS reported by short (2-week) recall periods and longer recall. The latter were “explicitly normative” questions in which households report the number of months purchases were made in the last year, how often they were made, and how much was usually spent each time (referred to here as “usual-month” questions). While food expenditures were not different, non-food expenditures were 72 percent higher based on short recall, contrary to expectations. Similar analysis is presented for Jamaica (Grosh et al _._ , 1995), Côte d’Ivoire, Vietnam, and Pakistan (Deaton and Grosh, 2000). Deaton and Grosh (2000) conclude that there is only a slight sensitivity to the choice of these recall periods. In much earlier work from Malawi in 1970, Plewis (1972) presents admittedly weak evidence that 7-day food recall is consistent with 24-hours recall (in some sense, this is like a diary). However, it is not clear to what extent conditioning bias is driving these results. The Indian National Sample Survey (NSS) experimented with using a “last week” versus a “last month” recall and found that for the all-food aggregate the estimates based on weekly recall were 21 percent higher (NSSO 2003).

In short there are numerous features of survey design that can result in different reported consumption for the same household over the same period of time. Usually based on ad hoc or opportunistic studies, clear divergences arise in measured consumption when recall periods are varied, commodity lists are shortened or lengthened, and consumption is recorded through diary or recall. Underlying these divergences are different magnitudes of various reporting errors, thus making it unclear (i) which variant of consumption measurement more closely hews to the truth; (ii) the implications of survey design for subsequent analysis, especially when consumption measures derived from different variants are combined in the analysis; and (iii) generalizable lessons for future consumption measurement in developing country surveys. The current study was designed to shed light on all three questions.

> 8 Dutta Roy and Mabey (1968) present results of studying alternative recall periods in Ghana for experiments circa 1966; they found significantly larger declines in reported expenditure than Scott and Amenuvegbe (1991). They attribute this to a combination of memory lapse for longer recall periods and over stated expenditures for short recall periods, although it is not clear how this conclusion is drawn.

> 9 A related research design would be to collect both diary and recall consumption data from the same household. Ahmed et al. (2006) study food expenditure among Canadian households reported by recall (past one month) and then collected for the following 2 weeks by diary. However, this design also risks cross-module spillovers within the household.
# **3. Tanzanian Consumption Survey Experiment and Consumption Aggregate**

# _Survey Experiment_

Our survey experiment entailed fielding eight alternative consumption questionnaires randomly assigned to 4,000 households in Tanzania. The eight designs vary by method of data capture, level of respondent, length of reference period, number of items in the recall list, and nature of the cognitive task required of the respondent. They are summarized in Table 1. Modules 1-5 are recall designs and modules 6-8 are diaries. These eight designs were strategically selected to reflect the most common methods utilized. The alternative designs focus on variation in the measurement of food consumption and expenditure on select frequently purchased non-food items, and not on general non-food expenditure where, due to the infrequency of purchase, there is a much greater degree of design harmonization in practice. Food consumption includes the quantity consumed from three sources (purchases, home production, and gifts/payments) and, for purchases only, the corresponding value of the quantity (in Tanzania shillings). Modules 1 and 2 seek the consumption values for a long list of 58 commodities. In module 3, the subset list consists of the 17 most important food items that constitute, on average, 77 percent of food consumption expenditure in Tanzania based on the previous Household Budget Survey. When comparing consumption expenditure in module 3 with other modules, we scale up food expenditures for that module as is commonly done in practice. Module 4 is a collapsed list where the 58 food items are aggregated into 11 comprehensive categories.<sup>10</sup> Among the recall modules, module 5 deviates from a reporting of actual expenditure over a specified time period. Instead it asks for “usual” consumption, following a recommendation in Deaton and Grosh (2000), where households report the number of months in which the food item is usually consumed by the household, the quantity usually consumed, and usual expenditure in those months. These questions aim to measure permanent rather than transitory living standards, without interviewing the same households repeatedly throughout the year. Hence, module 5 introduces two key differences from the other recall modules: a longer time frame and a different cognitive task required of respondents. The longer time frame should allow averaging over short-term fluctuations in living standards, which are caused not just by seasonality but by any short-term idiosyncratic shocks occurring in the survey reference period. Therefore, in comparison with a series of short-period surveys, staggered evenly over

> 10 Since respondents often use local units for reporting (Capéau and Dercon, 2006), the survey teams also established conversion factors for these local units and surveyed local prices so that quantity and value information in diaries could be checked for outliers.
the year for an entire sample, there may be less inequality in the “usual” month measure than in formats with shorter recall periods.<sup>11</sup> Second, the “usual” month format forces respondents to undertake a more difficult cognitive task of _estimation_ , rather than just _memory recall and counting_ required by other modules. There are many estimation strategies that respondents might try when estimating “usual” month consumption, each with their errors. For example, they could estimate monthly quantity from a rule-of-thumb daily consumption rate (“one mango a day during mango season”) and then apply either a lagged price or the current price to convert into consumption values. Or they might estimate monthly spending from mental accounts for shorter periods (“my weekly beer budget is 10,000 shillings, so 43,000 shillings per ‘usual’ month”). What is most unlikely, however, is that they try to remember all occurrences, count them, and then average them.<sup>12</sup> The three diary modules are of the “acquisition type.” Specifically, they add everything that came into the household through harvests, purchases, gifts, and stock reductions and subtract everything that went out of the household through sales, gifts, and stock increases. For example, items that are purchased to be resold, given away, or kept in stock are not counted as consumption. Two modules are household diaries in which a single diary is used to record all household consumption activities. For the third diary module, each adult member keeps their own diary while children were placed on the diaries of the adults who knew most about their daily activities.<sup>13</sup> Following the literature, we label this as a “personal diary.” The personal diary was carefully designed to avoid double counting. Diary entries are specific to an individual and should leave no scope for double-counting purchases or self-produced goods. It is possible that a “gift” could be given to the household and accidentally recorded by two individuals. However, interviewers were trained to cross-check individual diaries for similar items purchased, produced, or gifted that occur on the same day and to query these during the checks. In many cases, one person will 11 For example, Gibson, Rozelle, and Huang (2003) find the Gini coefficient in urban China is 64 percent higher if respondents kept a diary for one month, with the sample spread evenly across the year, compared with the Gini that results from all respondents keeping the diary for one year. In contrast, the annual mean from extrapolating staggered monthly expenditures almost exactly equals the mean from the annual diaries. Since this experiment was carried out in an urban area, the difference is not primarily due to seasonality but instead to short-term fluctuations within the survey period that are subsequently reversed over the year.

12 While the literature has findings about errors in memory and counting (more error the longer the period and the more transactions to count), almost nothing is known about errors in estimation strategies. For example, if respondents apply current market prices to usual month quantities, it reintroduces seasonality into the “usual” month estimate and so will tend to raise inequality toward the level found with short reference periods. In the Vietnam Living Standards Survey, the unit value for rice, formed from the ratio of “usual” month expenditure to “usual” month quantity, has considerable seasonality, with the same pattern across months that is observed in community market price surveys (Gibson, 2007). The community market price surveys reflect current seasonal conditions because they are carried out only once in each cluster but the unit values should not because they are meant to refer to purchases made in a “usual” (that is, an a-seasonal) month.

13 Just over 52 percent of individuals in the respondent households maintained a personal diary, with the remaining members (typically children) allocated to a specific adult personal diary.
acquire food for the household (such as buying 5 kilograms of rice), which is entered in the diary of the person acquiring the food. So the personal diary is a not an individual’s record of food consumption. Rather, it records the food brought into the household by each member even if for several members to consume (as well as food consumed outside the household). This intensive supervision of the personal diary sample would be impractical for most surveys but these investments were made in order to establish a benchmark for analytic comparisons.

Each of the eight designs varies how food expenditures (including value of home production and consumption) are collected along the lines specified in Table 1. Non-food items are divided into two groups based on frequency of purchase. Frequently purchased items (charcoal, firewood, kerosene/paraffin, matches, candles, lighters, laundry soap, toilet soap, cigarettes, tobacco, cell phone and internet, and transport) were collected by 14-day recall for modules 1-5 and in the 14-day diary for modules 6-8. Non-frequent non-food items (utilities, durables, clothing, health, education, contributions, and other; housing is excluded) are collected by recall identically across all modules at the end of the interview (and at the end of the two-week period for the diaries) and over the identical one or 12-month reference period, depending on the item in question. Any cross-module differences in measured nonfrequent non-food consumption we take as due to spillovers from the different amount of memory training or conditioning of respondents that may be induced by the different food consumption modules.

The survey experiment conducted was termed the Survey of Household Welfare and Labour in Tanzania (SHWALITA), and was implemented by a well-established data collection enterprise, Economic Development Initiatives (EDI). This survey was designed and fielded to study the implications of the alternative survey designs for consumption expenditure measures and labor market indicators (see Bardasi et al., 2010 for a description of the labor survey experiment). Here we focus on the component that applies to consumption expenditure measures.

The field work was conducted from September 2007 to August 2008 in villages and urban areas from seven districts across Tanzania: one district in the regions of Dodoma, Pwani, Dar es Salaam, Manyara, and Shinyanga and two districts in the Kagera Region. The districts were purposively selected to capture variations between urban and rural areas as well as across other socio-economic dimensions to inform survey design related to labor statistics and consumption expenditure for low-income settings. The sample was constructed to be representative at the district level, but not at the national level. Data from the 2002 Census were used to enumerate all villages in the district. In the first stage of the sampling process, a probability-proportional-to-size (PPS) sample of 24 villages was selected per district. In a second stage, a random sub-village (or
enumeration area, EA) was chosen within the village through simple random sampling.<sup>14</sup> In the selected EA, all households were listed. From these lists three households were randomly assigned to each of the eight modules. This was done through simple random sampling starting with module 1 and moving to module 8. The five alternative recall questionnaires were conducted in the span of the 14 days the survey team was in the EA to conduct the household diaries. Fortunately refusal and attrition after starting the survey are not an issue. Among the original households selected for the survey and assigned to a module, there were 13 replacements due to refusals. Three households that started a diary were dropped because they did not complete their final interview. This yields a final sample size of 4,029 households.<sup>15,16,17</sup> The basic characteristics of the sampled households generally match the nationally representative data from the Household Budget Survey (2006/07) (results not presented here but available from the authors upon request). Household interviews were conducted over a 12-month period, but because of relatively small samples within the period, we do not explore the survey assignment effects across seasons. Instead, the analysis reports the mean effect of questionnaire design across all seasons in Tanzania since each season is equally weighted in the data.

The randomized assignment of households to different questionnaire variants appears to have been successful when examining a set of core household characteristics, presented in Appendix Table 1, in terms of balance across characteristics relevant for consumption and consumption measurement. The table presents these mean characteristics by each module. At any point where there is a significantly different pairwise comparison, those pairs are indicated. Out of a possible 420 pairwise comparisons in Appendix Table 1, only 13 pairs are significantly different at the 5 percent level.

14 The equivalent of a village in urban areas is the _mtaa_ (Swahili for “street”). As there is no official subdivision of an _mtaa_ , our listing teams carefully defined and delineated Enumeration Areas within the _mtaa_ in cooperation with local informants.

15 Interviewers were trained extensively and were under continuous quality control. Regular supervisor re-interviews of households as well as supervisory direct observations were made to prevent any interviewer idiosyncrasies from developing. Each interviewer implemented all eight modules in equal proportion in order not to confound module effects with interviewer effects. There is the possibility of spillover effects in that an interviewer may influence how one module is completed based on experience with another module. Such learning and spillovers may also exist in other surveys, where experienced interviewers have exposure to different consumption modules. To the extent that such an effect may exist, it would be expected to result is an underestimation of any real gap between consumption collected by alternatively designed modules. However, this design was selected by investigators to minimize the presumed more serious risk of interviewer effects confounding the estimated module effects that can occur if individual interviewers were exclusively assigned to one module type.

16 The experiment used double blind data entry and high levels of quality control to reduce any effect of data entry on estimated cross-module differences. The data entry protocol was the same for all versions of the questionnaire and hence should not be a source of systematic error biasing the comparison of module performance. 17 We have almost no item non-response (in that the respondent does not report that the household consumed the specific item in the specified period) for food in the recall modules or for non-food in any of the modules. We do not observe any distinct patterns in non-response across our survey designs, or within a recall design by the location of the item on the list.
# _Consumption Aggregate_

We construct annual consumption aggregates (total household consumption expenditure) consistent with standard practices (see Deaton and Zaidi, 2002) but adapted to module specifications. For modules with the quantity of own-produced consumption but not shilling values (modules 1-3), monetary values were computed from unit values constructed from household information on purchases.<sup>18</sup> In each of the three modules, about 50 percent of quantities were able to be transformed by household-specific unit values (i.e., households also purchased the item), while in 25 percent of the cases the unit in which the household reported own-produced consumption was Tanzanian shillings. In the diaries, for each item we take the quantities reported minus what was recorded as sold or given away (deducted from the value as a percentage). Non-item specific (lump sum) corrections were made for changes in stocks (added consumption from stocks and adjusted for recorded acquisitions that were stocked instead of consumed), for food given to animals, and for items that were recorded as “forgotten.”<sup>19</sup> Module 3 stands out since it explicitly does not cover the same range of food as the other modules. Rather it is a subset list of the most frequently consumed foods based on the Household Budget Survey data. We scale food totals measured with this module up by 29.87 percent in order to compare with totals from the other modules, which are all covering a broader universe of foods (as stated earlier, we base this scaling up on the Household Budget Survey, where this subset of goods accounts for 77 percent [=1/(1+0.2987)] of total food consumption). For every module, we exclude expenditures on taxes and ceremonies. These consumption categories are not typically included in consumption aggregates due to their idiosyncratic nature and infrequency.

The direct comparison of results from the different survey modules sheds light on the relative influences of the various reporting errors discussed in Section 2. Since the frequently supervised personal diary is believed the most accurate in that it minimizes the magnitude of recall lapse, telescoping, and missed individual consumption, comparisons with this benchmark will be of particular use in understanding the net impact of the different reporting errors in each of the

> 18 For non-purchased products, we used the first available unit price in the following order: (i) Household-specific unit value, only available if household also purchased item (values more than five standard deviations from the district median price were replaced with district medians); (ii) EA median unit value; (iii) District median unit value; (iv) Overall median unit value; (v) EA median price from price questionnaire; (vi) District median from price questionnaire; (vii) Overall median from price questionnaire; and (viii) A price assigned by EDI staff, using best guesses based on the prices of similar items or units in the EA or district - only a handful of consumed items were valued in this way.

> 19 Forgotten items refer to the following. At the end of each of the 2 weeks of the three diary variants, the household is asked about the three main foods eaten daily and then the diary is reviewed to ensure that acquisition of these foods (including from stocks) are recorded. If these foods were omitted in the diary by mistake, the information is then collected by the enumerator (effectively in the form of recall).
different modules. However, comparisons across variants other than the benchmark will also be informative for the performance of each module. As discussed, every attempt was made to ensure the personal diary is close to ‘true’ consumption and as a result there appears to be very little double-counting or consequences of respondent fatigue. We find no differences in the number of diary entries and the total consumption expenditure between the first and second week, either overall or by key household characteristics.

# **4. Results**

# _Differences in Mean Consumption Estimates_

We begin our examination of the alterative consumption modules by investigating differences in reported per capita expenditure across modules. Table 2 presents the means and medians of per capita consumption by module for total, food, frequent non-food (either collected by recall or diary), and non-frequent nonfood (all recall) consumption. The other seven modules generally report lower consumption, especially food consumption, than the personal diary (module 8) benchmark. The module that comes closest in both mean and median to the personal diary is module 2 – the 7-day long list module. Module 4 (7-day collapsed list) has the lowest mean and median levels of food consumption.

The simple comparison of means already reveals some patterns that will persist in the analysis to follow. For example, the inability of household-based diaries to fully account for individual consumption, including that outside the household, contributes to 19 percent lower mean consumption and 15 percent lower median per capita consumption in the study sample. This is from a direct comparison of module 8 (personal diary) with module 6 (household diary supervised at the same frequency). In the same comparison, food consumption is 22 percent lower while total non-food is 16 percent lower (at the mean). Perhaps somewhat surprisingly, in the less-frequently supervised household diary the reported food consumption is 8 percent higher than the more regularly supervised household diary.

When looking at the recall modules, the 7-day long list recall (module 2) has the highest mean food consumption, even slightly higher than the personal diary, followed by the 14-day recall (module 1) and then the 7-day subset list recall (module 3) after it is re-scaled. As expected, a full listing of commodities results in a greater total value of consumption, in fact a 21 percent increase when comparing module 2 with the collapsed 7-day recall (module 4). This substantial difference indicates that the decision to adopt aggregate (shorter) consumption lists should not only consider the extent of savings in survey cost or reduced respondent fatigue, but also the potential downward bias in measured consumption. In terms of recall period, the 7 percent decline in reported food expenditures as we move from a 7 to a 14-day recall period is consistent with other studies, but we should not take this to mean that
7-day recall is necessarily the more accurate method since the higher values may simply reflect differing magnitudes of recall and telescoping error.

The recall module that asks about the “usual” month consumption for the last 12 months tends to report food consumption that is lower than the 14-day recall.<sup>20</sup> This is not surprising since with the lengthened recall period a greater degree of recall error is expected. However, we cannot separately identify this cause of divergence from the other deviation in design, which is to ask about the “usual” month rather than a specific recall period. One indication that the “usual” period approach does result in response differences is found when looking at non-food expenditures for which there is no variation in questions across recall modules. Frequent non-food expenditures are much higher for the “usual month” respondents, on average 25 percent higher than with the full list 7-day recall (module 2) and 32 percent higher than the aggregated 7-day recall (module 4), suggesting that the cognitive demands of food recall in module 5 affect subsequent responses.

For a formal test of difference across module designs we regress the natural logarithm of food, non-food, and total consumption on dummies indicating the module assignment (with the personal diary as the left-out category unless otherwise mentioned). The log-specification allows us to interpret the coefficients as percent deviations in mean value from the excluded category. Because the survey experiment was randomized, we do not include any controls.<sup>21</sup> Formally, the framework is as follows: where _Cik_ is (log) consumption of household _i_ assessed with questionnaire _k_ ( _k_ = 1... 7). _Mk_ is a vector of dummy variables for module type. Randomization of modules assures that the residual error term, _eik_ , is orthogonal to _Mk ._ Consumption refers to total consumption or its three subcomponents (food, frequent non-food, non-frequent non-food). The expenditure level from questionnaire _k_ in relation to the excluded category, module 8, will be given by _βk_ , which can be compared with the relative cost of administering that questionnaire type. Additional analysis that explores how consumption reporting varies by particular household characteristics such as household size or education of household head adopts the related specification: where Xik is the household characteristic and γk reports the coefficient on the interaction term.

> 20 As stated earlier, the survey was conducted over 12 months in a balanced fashion, so any differences between Module 5 and other recall formats are not due to seasonal effects.

> 21 The addition of controls may introduce bias into the experimental estimator (Freedman, 2008). Nevertheless, if we also control for the demographic composition (detailed age-sex categories), EA fixed effects, interviewer fixedeffects, and other household characteristics, the results are almost identical to those reported.
The regressions in Table 3 compare each of the seven modules against the personal diary and confirm the results from the first descriptive table.<sup>22</sup> The results in column (1) show that, with the exception of 7-day recall with the long list, other modules record between 7 and 28 percent less consumption compared with the personal diary. The impact on food consumption is of a similar magnitude (column 2). Having just one respondent complete the diary for an entire household is associated with significantly lower consumption of 14-17 percent, potentially because unobservable personal consumption is not explicitly captured in the design. Differences in frequent non-food expenditures are also observed. The largest deviation, that for the collapsed list (module 4), indicates that this shortcut in food consumption measurement results in significantly less measured consumption even for non-food (where the format is identical across modules). Among non-frequent non-food (collected by recall in every module), we generally see much less difference across modules (column 4). This is expected since the questionnaire wording and structure are identical. Differences can still result from three sources: respondent fatigue, since these recalled items come after lengthy food recall sections in modules 1-5 or a two-week diary; cognitive framing; and differing ability to capture personal nonfrequent non-food consumption outside the purview of the main respondent. Contrary to concerns of respondent fatigue, module 4 with the collapsed food categories and shorter interview time yielded significantly less (14 percent less) non-frequent non-food consumption.

To focus more sharply on the consequences of design differences among recall modules, in Table 4 we compare modules with different recall periods but the same detail of the food list (modules 1, 2, and 5 in panel A) and modules that have varying detail of food lists but the same recall period (modules 2, 3, and 4 in panel B). For all five of these modules, the non-food expenditure questions are identical (in detail and recall period). In panel A, columns 1-4, we see that lower total consumption associated with the longer recall periods is mainly driven by lower food expenditures. In the case of the “usual” food module, frequent non-food is statistically significantly higher (16 percent) than the 7-day module, and the non-frequent non-food is also higher (8 percent) but this difference is not precisely estimated. Given that the non-food questions are identical, we attribute this difference to the adaptation of respondents to the cognitive demands of the “usual” food questions, which immediately preceded the non-food questions.

Comparing the differing length of food lists (panel B columns 1-4), the collapsed list results in lower total consumption as a result of lower measured food consumption. Reported food consumption is almost 32 percent lower while the subset list, when scaled, results in only 6 percent lower consumption. There are no statistically significant differences in either of the non-food

> 22 If, instead of OLS, we estimate quantile regressions at the 35th percentile, a common cut-off for poverty analysis, the results are largely similar to Table 3 (available upon request).
categories, indicating no observable effect of respondent fatigue from the detailed food list on subsequent non-food categories. Clearly, at least in the context of Tanzania, the 17 most important food items (a subset of the full list of 58) performs only marginally worse than the full detailed listing.

Table 5 compares results among the three diaries. In Panel A, columns 1 and 2, we find that household diaries record significantly less total and food consumption than the personal diary, from 13 to 20 percent lower. Between the two types of household diaries, the frequency of interviewer visits makes little difference in total consumption, but food consumption is 7 percent higher in the infrequent diaries than the frequent household diary (and this difference is marginally significant at the 10 percent level; results available upon request), suggesting that the net impact of telescoping and recall error in the infrequently supervised households is positive but not especially large. Interestingly, the frequent non-food consumption exhibits the largest deviations in the household diaries from the individual diary, suggesting that a relatively large amount of these goods are consumed outside the purview of the household respondent.

In panel B of Table 5, we subdivide the household diaries on the basis of the literacy status of the household. Approximately 16 percent of households had no literate adults to fill in diaries. Households assigned to the diary with infrequent field visits had more visits if the household was deemed illiterate (see Table 1), although not as many as for households in the frequent visits subsample (module 6). The presence of an illiterate member does not affect reported consumption when households are regularly supervised (panel B columns 1-4, module 6). However, infrequent supervision results in significantly lower reported consumption among illiterate households, even for non-food non-frequent items, which are asked only in recall format. Clearly, leaving a diary with a household lacking a literate member will yield greater mis-measurement unless this difficulty can be overcome through frequent visits, thus converting the survey format to a de facto high frequency recall.

# _Consumption Measurement and Household Characteristics_

The literacy of household members is one characteristic that may affect the relative performance across modules. Table 6 explores the influence of additional characteristics by estimating equation (2) for the following: total household size, the number of adult household members (age 15 and above), urban location, education of the household head, and an asset index as an alternative measure of household resources (asset wealth) derived from housing conditions and household durable goods (Filmer and Pritchett, 2001). All of these characteristics are strongly
related to true household consumption, but also can affect the accuracy of consumption reporting. For example, respondents with more years of education may be more able to accurately calculate household consumption over the recent past, or more able to estimate consumption in a “usual” month abstracted from consumption in any actual month. Respondents in urban areas or assetwealthy households may experience a greater variety of consumption in any given reference period and so accurate recall may present additional challenges.

The results in Table 6 contain both the main module effect as well as the interaction term – due to space constraints the standard errors are suppressed (results available upon request) but the significance of coefficients is indicated by asterisks.

Household size is a significant determinant of overall household consumption (larger households typically have lower per capita consumption), but Table 6 also reveals it to be a determinant of increasing divergence from benchmark consumption in select recall modules. For three out of five recall modules (modules 1, 3, and 5) the discordancy in measured consumption increases with the number of household members. The interaction terms for the other 2 recall modules are also negative but not precisely estimated. The same pattern holds when we restrict to food consumption, and grows in magnitude when looking only at frequent non-food expenditure (and for this category the interaction terms for all modules are now significant). This may be due to increased cognitive demands of one respondent asked to recall the consumption of an entire household as the number of members increase. Alternatively the relative importance of out-ofhousehold consumption may increase with household size, particularly for frequent non-food expenditure, which is often privately consumed (e.g., cigarettes, cell-phone top-ups, bus and taxi fares) and this consumption is systematically missed by reliance on a single household respondent. However, if the lone respondent misses some amount of personal consumption, it appears to be a factor only for recall modules. The household size interaction terms for either household diary module are not significantly different from zero.

The second column of each panel in Table 6 refines the above discussion by looking at the influence of the number of adult (age 15 and above) household members. The effect of household size on reporting accuracy is strengthened when looking only at the adults in the house. With regard to overall consumption, the interaction terms for four recall modules are now significantly negative (all but module 4) and are at least twice the magnitude of the coefficients on total household size. The same general results hold for food consumption and, especially, frequent nonfood purchases, suggesting perhaps that it is largely adult consumption that is missed by sole respondents, either because this consumption takes place beyond the purview of the respondent or adult consumption is more complex than child consumption (more of which is food from a
common pot), and more difficult to recall. The interaction coefficients for the household diaries are also negative but not significant. These coefficients are also smaller in magnitude, suggesting again that performance with respect to household size differs dramatically by recall or diary format.

Urban households with recall modules do not perform appreciably worse than their rural counterparts vis-à-vis the benchmark. No interaction term for urban location is significant for any recall module except module 5, the “usual” month recall. For this module, the difference in total and food consumption from the benchmark is attenuated in urban areas, suggesting that the cognitive demands of the usual month construct present particular challenges for rural households. This may be because urban households rely more on the market for their consumption, with potentially smoother prices and availabilities over time so that rule-of-thumb extrapolations from daily consumption to a “usual” month are more accurate. Alternatively, this may reflect the higher education of urban respondents since in the results that control for the education of the household head, the only significant interaction term with years of education is again for module 5.

Education of the household head does not affect the performance of the household diaries relative to the individual diary but urbanity does, with household diaries performing significantly closer to the benchmark in rural areas. In fact the lower mean consumption of the infrequent household diary (module 7) summarized in panel A of Table 5 is entirely due to urban households – there is no significant consumption difference among rural households that were administered module 7 or module 8. However, urban households with module 7 report 29 percent lower total consumption, 26 percent lower food consumption, 45 percent lower frequent non-food, and 31 percent lower non-frequent non-food. The frequently supervised household diary (module 6) also reports significantly lower non-food consumption (of either category) when administered to urban households. These results are consistent with the supposition that urban settings provide more diverse personal consumption choices (such as outside meals and public transport) than rural areas, and much of this consumption out of what Deaton and Grosh (2000) term “walking around money” is missed by the sole diary keepers of modules 6 and 7.<sup>23</sup> The difficulty in capturing this “walking around money” may be somewhat mitigated by the frequency of supervision since the magnitude of the interaction terms are less for the frequently supervised module 6 and indeed the interaction term for food consumption for that module is not significantly different from zero.

> 23 This supposition is supported by the differential number of line item entries in urban and rural diaries. While the mean total number of unique commodities consumed by the household over the 2-week period is roughly equal across diary formats in rural areas (ranging from 108 to 111 items depending on the module), the personal diary in urban areas records significantly more commodity types consumed than either household diary. In urban areas, the infrequent household diarists record consumption of 139 unique commodity types while the frequent household diarists record 140 commodities. In contrast, households with personal diaries report consumption of 158 separate commodities over the same 2-week period.
The final characteristic in Table 6 is a measure of the asset wealth of the household. We see that consumption reporting with respect to the benchmark diverges dramatically depending on whether we look at the recall or diary format. All recall formats have a significant positive interaction term with the household asset index, implying that asset-poor households significantly under-report recall consumption vis-à-vis asset-poor households with the benchmark module, while asset-rich households report roughly equal or even greater amounts of consumption than asset-rich benchmark households. The asset index is a normalized mean-zero standard deviation-one random variable, which implies that rich households with an index score of 1 to 2 achieve consumption parity with rich benchmark households, depending on the particular module administered. This finding suggests that net recall error may be particularly acute among poorer households. If this is the case, then recall modules may actually overstate the degree of consumption inequality in a population relative to actual consumption. This possibility is explored further below.

Household diary performance interacted with the household asset index yields qualitatively different findings from that for recall, although this conclusion is more tentative given the relative lack of estimate precision. For both household diary types, the interaction terms are negative, and statistically significant for the infrequently supervised diary for select consumption subcomponents. Household diaries diverge from actual consumption as the wealth of the household increases, likely due to the increasing importance of private information about consumption as households grow wealthier. Where recall surveys may overstate the degree of inequality in the population, household diaries may actually understate it.

# _Distributional Measurement_

The results so far have focused on mean differences in measured consumption but it is possible that module type also affects distributional measures, as suggested in the sub-section above. Table 7 reports the Gini coefficient and its jackknife standard error for each module over the four consumption categories. When looking at total per capita consumption, inequality for the diary-based modules is lower than for the recall modules, although the differences are not always statistically significant. The same general pattern persists when looking at food consumption and the differences between recall and diaries are especially large for frequent non-food consumption. Inequality differences for non-frequent non-food consumption items are relatively slight as we
would expect, although the recall modules still yield somewhat more inequality for this type of consumption.<sup>24</sup> One surprising result from Table 7 is that the highest measured inequality is for the sample given the “usual” month recall. The aim of the “usual” month format is to measure consumption over a longer time frame, averaging over short-term fluctuations in living standards to yield lower inequality. Hence finding higher inequality with this design suggests that another effect is operating and a likely candidate is the heavy cognitive burden of this design. As noted below, the interviews for the “usual” month module took far longer than for any other interview, and the understatement by this module is most apparent for the least educated households. So it may be that some degree of inequality in education (the Gini for years of schooling of household heads is 0.43) is combined with the actual consumption inequality when a cognitively burdensome module like the “usual” month format is used.

Among the other recall modules, the subset and collapsed list (modules 3 and 4) generally yield a more equal distribution. For both module types, this may not be surprising since the relative lack of prompts among the aggregate categories is expected to yield compressed consumption measures due to omissions. If the diversity of consumption goods is greater among wealthier households, this error type will lead to truncated distributions at the upper end. Looking within the diary formats, the personal diary (module 8) yields slightly higher inequality, suggesting that the extent of omission of individual consumption in household diaries is not constant across wealth levels (as we have seen in Table 6 with respect to urban/rural differences and asset-index values) and this variation results in more compressed distributions.

Should these inter-module differences in Gini coefficients in Table 7 be considered large? The difference between the Gini coefficient calculated with the benchmark diary and with module 1 is 14 percent and with module 5 it is 19 percent. These are equivalent to the average gap between consumption-based and income-based Gini coefficients reported by Deininger and Squire (1996), the basis for the current UNU/WIDER WIID database. Thus, just as those authors were careful to identify which inequality estimates came from consumption surveys and which from income surveys, so that investigators might adjust them to a consistent basis, so too would it be worth identifying which Gini coefficients are computed from surveys that use diaries and which recall, and over what reference period and level of respondent. Yet such meta-data are hardly ever reported, forcing investigators who combine inequality measures from various surveys to introduce considerable measurement error into their analyses.

> 24 The pattern of results also holds for Generalized Inequality measures GE(0), GE(1), and GE(2) (not presented but available upon request).
# _Implications for Poverty Analysis_

The fact that both mean consumption and distributional measures vary substantially across the modules implies that there will be complex comparability problems when poverty estimates from different modes of consumption measurement are combined. For example, consider a poverty monitoring survey that switched from 14-day to 7-day recall. This switch would raise reported mean consumption (see Table 3) and lower measured inequality (see Table 7). In a setting where the poverty line is set below median consumption, these effects will work in concert to make poverty look lower than it would if the method of consumption measurement had not changed (as occurred with changes in India’s NSS). Conversely, if a switch was made from a disaggregated recall list to a collapsed list (as the Indonesian _Susenas_ does every 3<sup>rd</sup> year), it would lower the mean (implying higher measured poverty) and lower inequality (lower measured poverty) with ambiguous implications for poverty measures. Hence there are unlikely to be simple correction factors that can be used to enforce comparability on different consumption modules when attempting to measure poverty consistently over either time or space. Similarly, the literature that reacts to possible survey understatement by combining survey measures of inequality with national accounts measures of mean consumption (such as Bhalla, 2000, and Sala-i-Martin and Pinkovskiy, 2010) is likely to be misguided since survey design changes will also affect estimated inequality.

We attempt to quantify the implications of survey design for poverty analysis in a setting like Tanzania by presenting aggregate poverty numbers based on an international poverty line, as well as the characteristics of those households deemed poor by each module. Figure 1a presents the poverty headcount rate based on a common international poverty line of $1.25/person/day converted into 2008 Tanzanian shillings on a PPP basis.<sup>25</sup> With this absolute poverty line fixed at the same value regardless of module, we see dramatically different headcount rates depending on how consumption is measured. The benchmark personal diary records the lowest level of poverty at 47.5 percent of the population, followed by the 7-day long list (module 2) at 54.9 percent. Even though module 2 yielded mean consumption levels closest to the benchmark (Table 3 lists consumption at 3.9 percent lower and not significantly different from benchmark consumption), the fact that module 2 inequality was also higher (the estimated Gini is three points higher – Table 7) results in substantially higher poverty numbers. The 7-day subset design (module 3) reports

> 25 An alternative to the international poverty line would be to use a nationally defined poverty line (for example, the line defined by the HBS 2006/07). In either case, we avoid defining eight separate poverty lines, each on the basis of each consumption aggregate measure, since this would introduce problems as these lines would then not be comparable, as discussed in Lanjouw and Lanjouw (2001).
poverty almost identical to module 2 – 55.1 percent – while the other recall modules report poverty headcounts all over 60 percent, with module 4, the collapsed list, yielding a poverty estimate of 66.8 percent. Poverty numbers derived from household diary figures also record substantially higher poverty levels than the benchmark – the infrequent household diary records the poverty rate at 55.6 percent and the frequent diary at 59.5 percent.

The poverty headcount is attractive for its ease of understanding, but is also notorious for its implied welfare discontinuity at the poverty line as well as the inability to measure the extent or depth of poverty (Ravallion, 1996). Figure 1b presents two additional poverty measures, the poverty gap and the squared gap measures, that give greater weight to the poorest households among those below the poverty line. The overall message in Figure 1b is the same as Figure 1a. The lowest poverty figures are for the benchmark module 8 while the highest poverty measures are observed in recall modules, especially modules 1, 4, and 5. The ranking of poverty scores varies somewhat by poverty measure. While the 7-day full recall module 2 had the second lowest headcount estimate (54.9%) followed by module 3 (55.1%) and then module 7 (55.6%), module 7 has the lowest poverty gap measure (18.9%) followed by module 2 (19.1%) and then module 3 (21.4%). The largest proportional differences across module are apparent in the squared gap measure where the highest estimated squared poverty gaps, those for modules 4 and 5, are more than twice as large as the benchmark squared gap measure. The changes in module rankings across the different poverty measures, as well as the greater proportional differences in the squared gap measures compared with the poverty headcount, reflect the differing degrees of inequality below the poverty line measured by the various modules.

Clearly poverty measures are highly sensitive to the mode of consumption measurement and care must be taken when poverty studies combine measures of consumption poverty, either across countries or within a country but over time, derived from differing survey designs. Another important goal of poverty measurement, however, is to identify the characteristics of poor households in order to inform the targeting of social policy or to better understand poverty determinants. Hence an equally important consideration, other than whether survey design influences estimated poverty levels, is whether the mode of survey design influences the identified characteristics of the poor. We approach this question through a simple regression framework similar to equation (2) where, using a linear probability model, we regress the household specific poverty indicator on module type interacted with household characteristics. The results, in Panel A of Table 8, again adopt the $1.25/person/day line used in Figures 1a and 1b. A complementary approach to the same general question of the relative rankings of households is to utilize the full information of the continuous consumption distribution, as opposed to the binary poverty indicator,
and regress the percentile of the household in the consumption distribution on module type and select characteristics. These rank regressions are presented in Panel B of Table 8 (here again the standard errors are suppressed due to space constraints but available from the authors).

The results from the analysis in Panel A show virtually no difference in the characteristics of poor households identified by any module. The number of adults in the household, the rural/urban location, and the education of the household do not predict greater or lesser likelihood of household poverty in any module with respect to the benchmark measure – while all three characteristics are important predictors of poverty in their own right, none of them affect the relative probability of being deemed poor across the modules. Total household size is a significant relative predictor for two modules – larger households are significantly more likely to be deemed poor when administered the long 7-day recall (module 2) while larger households are significantly less likely to be deemed poor when administered the frequent household diary (module 6). These estimated effects are not especially large in magnitude – the addition of a household member affects the poverty indicator probability by 2 percentage points in either direction depending on the module. The only other characteristic that is significant at standard levels is the asset-index interacted with the usual month method (module 5). An increment in the asset-index of one standard deviation reduces the relative likelihood of poverty indication by 6 percentage points for module 5 households. Thus a relatively large change in household assets changes the relative likelihood of poverty inclusion only slightly.<sup>26</sup> Turning to the rankings of household consumption in Panel B, we observe a very similar pattern to Panel A. Few observable characteristics result in different within module rankings. Neither the education of the household head or the household asset index affects the relative rankings of households for any module vis-à-vis the benchmark. Overall household size does have a moderate affect on household percentile score for module 3 households, where an increase in household size of one member reduces the ranking of the household by one percentile point. The number of adult household members affects the ranking of 7-day and 14-day full list recall, where an additional adult in modules 1 and 2 households lowers the relative household rankings by 3 percentiles. Finally urban households are ranked an average of 8 percentiles lower in module 7

> 26 An alternative relative approach to poverty measurement sets identical 35 percent poverty headcounts across modules. Even though the poverty rate for each module is fixed at the same level, the characteristics of households below that cut-off can, in principle, differ. Using the same analytic framework as Panel A, however, reveals virtually no interaction term to be significant when the indicator is relative poverty. Hence whether poverty is defined in absolute or relative terms, poverty assignation is largely consistent across modules with respect to observable household characteristics. The same conclusions hold when either the absolute or relative poverty indicator is regressed on module and characteristic interactions with a binary dependent variable model  (probit or logit) rather than a linear probability model.
households than the benchmark. All other interaction terms are not significantly different from zero. As we have seen in this sub-section, while absolute poverty measures vary dramatically across the different modules, the partial correlations of poverty status with selected characteristics do not differ significantly and the relative rankings of households with those same characteristics are largely stable between modules.

# **5. Resource Implications**

The willingness to accept some degree of inaccuracy in consumption estimates from using a recall survey or a household diary compared with the personal diary is driven by the practical need to reduce the study costs and the burden to respondents of survey work. Our benchmark of frequently supervised individual diaries is often not a feasible option for field researchers, which leads to the question of recommendations for alternative approaches. Here we review the cost implications of different survey designs, focusing first on the cost in terms of time to administer a recall module and then on dollar expenditures for survey implementation in order to contrast recall with diary approaches.

Among the recall modules, the shorter lists in the subset and collapsed recall modules are assumed to significantly reduce the length of time to complete the module and this consideration in our experiences is often used as a justification to adopt a shorter consumption module. Figure 2 presents the average time to complete the recall modules. Respondents required an average of 50 minutes to complete the recall module that listed 58 foods for a 14-day reference period; for the 7- day full list module, the length of time to complete is just 1 minute shorter. The mean time savings is only 8 minutes (42 minutes total) when using the 11 broad food group headings (module 4) and 9 minutes (41 minutes total) when using the module 3 subset recall list with the 17 most important food items. While the interview time is shorter on average, the 8 or 9 minute savings in interview time of the shortened modules must be weighed against the overall lower mean consumption (especially for module 4) and truncated distribution relative to the longer list recall that these two survey modules generate. These results, of course, pertain to the study setting and in Tanzania households do not have a great deal of dietary diversity. In other settings with greater diversity, these estimated relative time costs may be less applicable.

The most demanding recall module, in terms of the average time taken by respondents, was by far the “usual” month consumption module. The module requires respondents to think about the number of months they consume the food and the typical consumption in those months. This is evidently a time-consuming computational task – taking 76 minutes on average. The previous
sections have demonstrated that this method results in consumption measures quite divergent from the “benchmark” and may also significantly over-estimate the degree of inequality in the distribution. Given both its poor relative performance and onerous time imposition, this method is not a recommended choice for this study setting.

Turning to dollar costs as a way to assess the resource utilization of the different diary modules with respect to a recall survey, the exact costs involved to administer each of the questionnaires and diaries depends on a number of context-specific factors, such as the price of labor, literacy levels, costs of travel between the sample EAs, printing costs, overheads, and so on. The main drivers of cost differences, however, are relatively easy to pinpoint. They are (i) differential interviewer-days needed to complete a household, and (ii) differences in time needed to enter the data into electronic format.

Table 9 summarizes these differences and their cost implications under a set of assumptions about time to walk between households, review diaries, and implement recall modules. We are not considering a large multi-topic survey, but a short questionnaire with a complete consumption module (either by diary or recall). In the most labor-intensive set-up, for an interviewer doing personal diaries with daily visits, six households can be interviewed in 17 days. The alternatives to the personal diary are household diaries with varying visits in combination with a second enumeration area (to avoid days with no work in the enumeration area, given that the length of the diary is 14 days). For the analysis in Table 9, no distinction is made between the different types of recall questionnaires as they all yielded around four interviews per interviewer per day, which is three times less intensive than in the ‘light’ fieldwork set-up of household diaries with infrequent visits.

Once the diaries and questionnaires come back from the field, they go through several steps before being available in electronic format: administration and filing of forms, coding of items, double blind entry, reconciliation of the two entries, and so forth. We use information from the survey experiment to compute the time for these activities. On a given day, a data entry operator can process 2.86 personal diary households, 4.31 household diary households, or 8.51 recall questionnaire households.

Based on the person days for field work and data processing and with data entry staff in the office at half the cost of an interviewer in the field, we then compute the total costs of field work (Table 9, column 6). With the average cost to administer a recall questionnaire set at US$100 per household (the numeraire to scale the other field costs), the personal diary will fall between US$597 and US$974, depending on whether the household is visited every day or every two days. For a frequently visited household diary, the estimated costs lie between US$442 and US$726; for
the infrequently visited household diary, it will lie between US$280 and US$334. Clearly the diary approach, especially the benchmark of a personal diary, is a much more resource intensive form of consumption measurement, roughly six to ten times the cost of administering a recall survey to the same household. The cost-savings from a frequently supervised household diary is a relatively modest 25 percent of total personal diary costs. Given the performance of the household diary, especially in urban areas, these resource savings may not be worth the trade-off in terms of downward biases in consumption measurement.

# **6. Conclusions**

In most developing countries, consumption expenditures measured by household survey is the empirical basis for studying poverty and inequality. Yet there is substantial variation both within and across countries in the design of consumption modules. These differences result in challenges to the cross-population comparisons of poverty and living standards as well as efforts to study changes in poverty over time. This study explores the implications of alternative consumption modules design in Tanzania. We designed and fielded eight alternative surveys, selected to capture the most common designs in practice. We benchmark our comparisons to frequently-supervised personal diaries, which we assume come closest to measuring actual consumption by accounting for personal outside-the-house consumption as well as minimizing recall and telescoping error due to the frequent supervision. This method is, however, impractical for large-scale survey work due to high costs – we estimate that it will cost roughly 6 to 10 times as much as a recall format, and roughly twice as much as infrequently supervised household diaries.

With regard to diaries, the quality of reporting in household diaries did not vary a great deal with the frequency of field staff visits designed to minimize recall and telescoping error. This is true for all consumption categories (although food consumption was marginally higher (6 percent higher) in the infrequent diary). One notable exception is the literacy of the household. The diary method will dramatically underestimate consumption if the household is illiterate and receives infrequent supervision. For other households, the relative similarities in measured consumption between the frequently and infrequently supervised diary modules suggests that resources can be saved through the infrequent supervision schedule. The gap between both household diary types and the personal diary likely reflects the omission of personal and out-of-household consumption in the household diary, and this gap, while not noticeable in rural areas, is substantial among urban households. The design decision is then whether to invest in the full personal diary approach, which costs roughly three times as much as the infrequent household diary, or cover many more
households with the infrequent household diary. Clearly the characteristics of the study population will be a critical factor in this decision – if the population contains a large proportion of urban households, the need for extra resources to implement a personal diary may well be justified.

From among the recall surveys, all far less expensive than diaries, certain recommendations are clear. For one, the savings in survey time from a reduced number of consumption categories in the recall list through aggregation (the collapsed list module 4) is minimal compared with a substantial cost in terms of loss in accuracy. We do not recommend this module type. On the other hand, the other short commodity list design (the subset list module 3), when scaled up based on reference data, performs very close to the longer list form and may be a suitable substitute to longer list recall modules. The gain from such a reduction in list length, however, is slight – less than 10 minutes of interview time – and researchers would have to decide whether the loss of additional detail in consumption information is worth the moderate reductions in interview time. The hypothetical “usual” month recall almost doubles the interview time while reducing the accuracy of measured consumption, leaving no practical basis to recommend this approach.

These considerations would lead to a recommendation of a long-list recall module with a reference period of 1 or 2 weeks. Both modules take the same amount of time to implement. The 7- day recall results in mean consumption closest to the benchmark as well as summary inequality measures only slightly higher and not significantly different from the benchmark. The 14-day recall yields significantly lower mean consumption and higher inequality (thus much higher poverty estimates as in Figure 1). By process of elimination, if one recall module must be chosen, the 7-day full list (module 1) presents the fewest complications. Nevertheless, researchers need be aware of the underlying fact that this module is still likely subject to recall and telescoping errors of varying degrees as well as presumably the inability to capture personal out-of-household consumption. For example, the 7-day recall reports significantly less consumption for households with more adult members than does the benchmark, and has a slight tendency to identify larger households as poor. Like other recall modules, it appears subject to either net telescoping or deliberate misreporting that increases in magnitude with the asset wealth of the respondent (Table 6). Nevertheless, this module, as virtually all other modules, yields household consumption rankings that are remarkably stable with respect to key household characteristics, indicating that analysis that focuses on the determinants of household ranking within the consumption distribution will be largely consistent regardless of which module is used. There are long-held perceptions of the inadequacy of using a recall module compared with a household diary to measure consumption expenditures, despite the
mixed evidence discussed in Section 2. Our findings call into question these perceptions, particularly when resource implications are considered.

As we have said throughout, the conclusions drawn from this study are dependent on the specific setting. The results presented are likely a result of traits such as the share of consumption from home production, dietary diversity, and the fraction of meals eaten in the dwelling (as opposed to private food consumption at restaurants). These traits observed in Tanzania are mostly associated with low-income countries and not middle-income settings. Rural Tanzania is similar to much of rural Africa and indeed other rural regions in South and South East Asia, and so the general lessons from this study may be broadly applicable to these areas. Urban Tanzania is also not appreciably different from other urban settings in low-income African countries, but may diverge from urban middle-income settings in terms of diversity of consumption choices and household structure. This setting may also diverge from results for parts of West Africa where, as noted by Boozer, Goldstein, and Suri (2009), wives and husbands manage separate budgets for some expenditure items, described as “a decentralized structure,” resulting in a higher level of incomplete information about expenditure than observed in the household modules in Tanzania. Further work will be necessary, including the implementation of similar experiments in other settings, to better understand the relative performance of differing approaches to consumption measurement as well as their analytic implications as developing countries continue to raise living standards, increase consumption diversity, and urbanize.