metadata: authors: 0: author: Pape,Utz Johann
count: World
docna: 0: docna: Measuring Poverty Rapidly Using
            Within-Survey Imputations
docty: Policy Research Working Paper
subtopic: Inequality,Small Area Estimation Poverty Mapping,Poverty Impact Evaluation,Poverty Assessment,Poverty Monitoring & Analysis,Poverty Lines,Poverty Diagnostics,International Trade and Trade Rules,Health Care Services Industry
teratopic: Poverty Reduction,International Economics and Trade,Industry
abstracts: cdata!: Poverty is an indicator of paramount
            importance for gauging the socioeconomic well-being of a
            population. Especially during or after a shock, poverty
            estimates are invaluable for assessing the severity of the
            impact and for identifying which parts of the population
            were most affected. The measurement of consumption-based
            monetary poverty, however, has traditionally been very time
            consuming. A household consumption questionnaire usually
            includes more than 200 items, including food and nonfood
            items, often requiring more than two hours to administer.
            This paper proposes a new methodology that combines an
            innovative questionnaire design with standard imputation
            techniques. It substantially shortens the time required to
            administer a household consumption questionnaire to less
            than 60 minutes by imputing deliberately absent consumption
            values for items that are not explicitly asked. The proposed
            methodology makes it possible to derive poverty estimates
            without compromising the credibility of the resulting
            estimate, and it performs considerably better than
            alternative approaches based on reduced consumption
            aggregates and cross-survey imputations. This new
            methodology is particularly useful in fragile states given
            the significant risks associated with lengthy interviews, as
            well as to rapidly assess the impact of a shock or of a
            project. It can also be useful to reduce enumerator and
            respondent fatigue, or to mitigate the problem of high
            nonresponse rates.
repnme: repnme: Measuring Poverty Rapidly Using
            Within-Survey Imputations
display_title: Measuring Poverty Rapidly Using
            Within-Survey Imputations
listing_relative_url: /research/2021/01/32786122/measuring-poverty-rapidly-using-within-survey-imputations
topicv3: Poverty m1326687 695,Trade m1326924 698,Poverty Measurement and Analysis m1326941 707,Health m1504661 3480,Psychiatry, Psychology m1504686 3613
publishtoextweb_dt: 2021-01-28T00:00:00Z
historic_topic: Poverty Reduction,International Economics and Trade,Industry
origu: EFI-AFR1-POV-Poverty and Equity (EAEPV)
no_of_pages: 28
author_keys: 0: author_key: 1154467
model_extractions: Policy Research Working Paper 9530 Measuring Poverty Rapidly Using Within-Survey Imputations _Utz Pape_ Poverty and Equity Global Practice January 2021
Policy Research Working Paper 9530

# **Abstract**

Poverty is an indicator of paramount importance for gauging the socioeconomic well-being of a population. Especially during or after a shock, poverty estimates are invaluable for assessing the severity of the impact and for identifying which parts of the population were most affected. The measurement of consumption-based monetary poverty, however, has traditionally been very time consuming. A household consumption questionnaire usually includes more than 200 items, including food and nonfood items, often requiring more than two hours to administer. This paper proposes a new methodology that combines an innovative questionnaire design with standard imputation techniques. It substantially shortens the time required to administer a household consumption questionnaire to less than 60 minutes by imputing deliberately absent consumption values for items that are not explicitly asked. The proposed methodology makes it possible to derive poverty estimates without compromising the credibility of the resulting estimate, and it performs considerably better than alternative approaches based on reduced consumption aggregates and cross-survey imputations. This new methodology is particularly useful in fragile states given the significant risks associated with lengthy interviews, as well as to rapidly assess the impact of a shock or of a project. It can also be useful to reduce enumerator and respondent fatigue, or to mitigate the problem of high nonresponse rates.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The author may be contacted at upape@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Measuring Poverty Rapidly Using Within-Survey Imputations Utz Pape<sup>1</sup> Keywords: Poverty and inequality measurement, survey methods JEL: C83, D63, I32 1 Corresponding author: Utz Pape (upape@worldbank.org). The findings, interpretations and conclusions expressed in this paper are entirely those of the author, and do not necessarily represent the views of the World Bank, its Executive Directors, or the governments of the countries they represent. The author would like to thank Johan Mistiaen for discussions and support in pursuing the idea and Chris Elbers for help with the statistical properties of the new methodology, as well as Kathleen Beegle, Tomoki Fujii, Kristen Himelein, Dean Jolliffe, Peter Lanjouw, Emmanuel Skoufias, Shinya Takamatsu, Roy Van der Weide and Nobuo Yoshida for discussions. The author is also grateful to the Kenyan National Bureau of Statistics for implementing the methodology in a pilot survey.
# **1. Introduction**

Poverty is an indicator of paramount importance for gauging the socioeconomic well-being of a population. Especially during or after a shock, poverty estimates are invaluable for understanding the situation, as well as for assessing the severity of the impact and for identifying which parts of the population were most affected. Especially in the developing world, consumption-based monetary poverty measures are used, defining the poor as those households with consumption levels that fall below a set poverty line (Angus Deaton and Zaidi 2002). The poverty line is usually set at a consumption level adequate for sustaining the minimum level of welfare required for healthy living (Ravallion 1998). Consumptionbased poverty measures are widely used in development contexts and play a critical role in policy decisions (e.g. Beegle et al. 2016).

The measurement of consumption, however, has traditionally been very time consuming. A typical household consumption questionnaire contains a series of questions about the price and quantity consumed for each item, and whether it has been purchased, self-produced, or bartered. Usually encompassing more than 200 food and nonfood items, the time required to administer such a questionnaire can often substantially exceed two hours. In addition to high administration costs due to long interview times, measurement errors may become significant towards the end of the questionnaire as enumerators and respondents become fatigued. Respondents might also cancel the interview before it is completed, thus contributing to a higher non-response rate.

Enumerator and respondent fatigue are well documented in the literature (Tourangeau, Rips, and Rasinski 2000; Krosnick 1991) and become more pronounced for longer questionnaires (Rolstad, Adler, and Rydén 2011; Diehr et al. 2005; Snyder et al. 2007). Enumerator fatigue increases measurement errors often over the course of a day as well as over the time the survey progresses (Baird, Hamory, and Miguel 2008). Especially in consumption surveys, a long list of items can lead to enumerators cutting corners and fabricating data (Fiedler and Mwangi 2016; Finn and Ranchhod 2015) as well as prematurely ending interviews (A. Deaton and Grosh 2000). Respondents also become fatigued and, for example, learn to say no to consumption of items to evade more detailed follow-up questions (Kreuter et al. 2011; Eckman et al. 2014).

To overcome the challenges inherent to measuring consumption poverty, we propose a new methodology that combines an innovative questionnaire design with standard imputation techniques.<sup>2</sup> This new methodology allows us to substantially shorten the consumption questionnaire and reduce the interview time (less than 60 minutes for a standard questionnaire) by imputing deliberately absent consumption values for those items that are not explicitly asked about. Poverty estimates can be derived in this way without compromising the credibility of the resulting estimate. This new methodology is particularly useful in fragile states given the significant risks associated with lengthy interviews. It can also be useful to reduce enumerator and respondent fatigue, or to mitigate the problem of high non-response rates.

The most straightforward way to reduce the expected interview time is to skip rarely-consumed items. Another simple strategy is to ask the respondent about an aggregate amount of spending on an entire

> 2 A precursor methodology based on the same principle was previously published in Pape and Mistiaen 2015.

2
category of consumption (e.g., total expenditure on flour) instead of individual items (e.g., expenditure on corn flour, wheat flour, etc.). However, altering the set of items in the questionnaire can result in a nontrivial change in the reported consumption amount (Olson-Lanjouw and Lanjouw 2001). Both approaches are likely to lead to an underestimation of consumption and overestimation of poverty, as was demonstrated in a study in Tanzania that directly compared various methods of measuring consumption (Beegle et al. 2012).

An alternative approach is to apply methods of cross-survey imputation. In situations where full household expenditure surveys are too costly or impractical to administer, data gaps can be filled using other surveys that have common covariates that are correlated with household expenditure. For example, data from a full consumption survey can be combined with data from shorter and more frequent labor force surveys to generate poverty estimates (Douidich et al. 2013). While such methods may work well even when there is a rapid economic change (Christiaensen et al. 2011), the assumption of a stable structural parameter typically cannot be tested and may not be valid, especially in the context of large and systemic shocks, after implementation of projects, or if a substantial amount of time has passed since the baseline survey was implemented. It is also possible to design a survey such that one sample has a full consumption module and another sample has only the covariates of consumption. Consumption can thus be imputed and poverty estimates can be derived at a reduced cost, even though the magnitude of potential cost reduction may be modest (Fujii and van der Weide 2016). In such a setup, however, the sample for the full consumption module must be chosen randomly to avoid biased estimates of the model parameters. Thus, this approach is only of limited usability in the case of fragile countries as it might not be feasible to administer the full consumption module in particular insecure areas, creating a downward bias in poverty estimates for those areas.

The proposed methodology uses statistical imputations to obtain estimates for deliberately absent consumption values. Statistical imputation techniques are widely used to replace missing values in surveys (Little and Rubin 2019; Ambler, Omar, and Royston 2007; Van Buuren 2007). Straight-forward methods simply replace the missing values with aggregate statistics like a mean. However, this makes the strong and often violated assumption that data are missing at random (Carpenter, Kenward, and White 2007). Model-based approaches can take into account covariates and often use a regression framework to estimate missing values but distort the variance if based on point-estimates. Multiple imputations help to mitigate this by drawing multiple estimates from the posterior distribution using a Bayesian approach (Rubin 2004).

This paper is organized as follows. We first present the proposed methodology with its statistical properties as well as the data in Section 2. In Section 3, we apply the methodology to different scenarios showing the trade-offs between the performance and parameters of the approach, and then compare it to a reduced consumption approach as well as a more sophisticated reduced consumption approach adjusting the poverty line. The section ends with a real-world example based on a pilot survey in Kenya, assessing the performance of the new methodology and comparing it to a cross-survey imputation approach. The paper finishes with Section 4 concluding the findings and discussing some of the limitations of the new approach.

3
# **2. Methodology**

## **Overview**

The rapid approach being proposed here applies a split-questionnaire design to the consumption module of a household survey, thereby generating systematically absent data that can be conveniently imputed.<sup>3</sup> Instead of having all households report on all consumption items, important items are assigned to a core module and the remaining items are split into two or more optional modules.<sup>4</sup> Each household then answers the questions in the core module and in only one of the optional modules. This approach reduces average interview time considerably, down to 45 to 60 minutes per household for a standard household consumption survey. The cost of this efficiency gain is that data are deliberately absent for those optional modules that were not administered to certain households. We can however offset this cost by estimating the deliberately absent data for each household based on the data collected from other households for that module. While this approach utilizes a structural model for the imputation of the deliberately absent data, the model is estimated within the survey rather than between two surveys, thereby circumventing the problem of biased structural parameters due to having different sample populations or considering the same population at different points in time.

The rapid approach starts by defining the number of core items and the number of optional modules for the non-core consumption items. The smaller the number of core items and the greater the number of optional modules used, the faster the questionnaire can be administered, as fewer items need to be asked for each household. However, having fewer core items and more optional modules also increases the uncertainty in the estimation as less consumption information is available. Thus, the choice of these two parameters can be informed by simulations on a previous or similar survey to gauge the performance of the estimation vis-à-vis the time savings in administering the questionnaire. Another consideration is that it is beneficial to balance the number of households for each optional module, ideally at the cluster level of the survey.

The next step is to select core consumption items. Although consumption in any given country will exhibit some variability, data on a few dozen key items will usually be sufficient to capture the majority of consumption. Important consumption items can be identified using average consumption share per household or across households, as estimated by previous consumption surveys in the same context or recorded consumption shares in neighboring and/or similar countries. While a good choice of core items will improve the performance of the estimation, the methodology still works if no core items are used, e.g. in a context without any prior information. The identified key items are then assigned to the core module that will be administered to all households.

Finally, non-core items are randomly partitioned into optional modules. It is important to note that the conceptual distinction between core and optional items should not be reflected in the layout of the questionnaire. Instead, all items per household need to be grouped into categories of consumption items

> 3 While the split-questionnaire design is more popular in other disciplines such as psychology (e.g. Graham, Hofer, and MacKinnon 1996), the approach has not yet been applied to large-scale household-based surveys, nor with the goal of reducing the time required to estimate consumption or poverty.

> 4 As is shown below, the core module is not strictly necessary, further reducing the interview time.

4
“Sy DD =E)y ~~—=~~ ) Ey )=EQy ~~—~~ )=)£0 =-E.)y =E)y +28 y =y ~~—~~ +2 ~~——~~
y y =) ky EY = ES )+ >. Vary +2 >. Cov(y + Cov(y )+) var + 2) Covly iy Var(p
log(y ) = BX

|For

lE <u>(FG~</u> )

|
# **Data**

We applied this method to recent household consumption data from Kenya. Kenya’s source for official poverty estimates is the Kenyan Integrated Household Budget Survey (KIHBS). The two last rounds were implemented in 2005/6 and 2015/16. The 2005/6 round used a representative sample of households in Kenya stratified by 7 provinces and 69 districts split into urban and rural. The sample size of the cleaned data set includes 12,695 households in 1,338 clusters. The 2015/16 round used a representative sample of households in Kenya stratified by 47 counties split into urban and rural. The sample size of the cleaned data set includes 21,585 households in 2,387 clusters. For both surveys, data collection was carried out over a period of 12 months.

The 2015/16 round was accompanied by a CAPI pilot implementing the rapid approach. The pilot used the same sampling frame as the 2015/16 round and interviewed up to an additional 6 households in the same clusters, resulting in a sample size of 12,662 households (as not all clusters and all intended households were interviewed due to non-response).<sup>8</sup> The questionnaire was derived from the KIHBS 2015/16 questionnaire but was considerably simplified across all modules. Specifically, the consumption module was administered according to the rapid approach. We are using the data set that was constructed with 5 food and 5 non-food items in the core (selected based on the highest consumption share in KIHBS 2005/6), with the remaining 128 food and 76 non-food items partitioned into 3 optional modules.<sup>9</sup> Thus, the expected time saving was about 30 percent.

In the remainder of the paper, we use KIHBS 2015/16 with the full consumption module as a benchmark. The previous round of 2005/6 is used to define the core items, and for two of the alternative approaches it is used to adjust the poverty line and to build the consumption model for the cross-survey imputation. The 2015/16 pilot is used as a real-world example for the implementation of the rapid approach. We harmonize the data sets to ensure comparability across the different surveys. The harmonized data sets include 133 food and 81 non-food items. The consumption aggregate is exclusively based on these 214 items. In addition, we use harmonized location and household characteristics for the various models, including a binary and a categorical location variable as well as 6 additional binary, 9 additional categorical and 9 continuous variables for household characteristics (Table 5 in the Appendix).

Consumption shares differ markedly between the 2005/6 and 2015/16 surveys, which is unsurprising given the 10-year gap (Kenya National Bureau of Statistics 2018).<sup>10</sup> For example, the top 10 food items in 2005/6 capture 58 percent of the food consumption share, but only 39 percent of the consumption share in 2015/16. Non-food consumption changed to a lesser degree. The top 10 items in 2005/6 represent 64 percent of non-food consumption shrinking to 59 percent in 2015/16. These differences will impact the

> 8 Balance mean tests are shown in Table 6 indicating similar households in both surveys with similar although not always statistically indistinguishable characteristics.

> 9 The assignment of optional modules to households was balanced with 4,222 households assigned to module 1, 4,192 to optional module 2, and 4,248 to optional module 3.

> 10 See Table 4 for the shares of the items with the top 20 consumption shares for KIHBS 2005/6 and KIHBS 2015/16.

8
performance of those approaches that strongly rely on data from previous surveys, e.g. adjusting poverty lines and cross-survey imputations.

# **3. Results**

We assess the performance of the rapid approach vis-à-vis alternative approaches (Table 1). The long questionnaire of the full-consumption approach can increase unit and item non-response but is nevertheless used as benchmark due to its _de facto_ standard for consumption surveys. The rapid approach compromises on the long list of items by introducing a subset of core items and distributing remaining items in optional modules, reducing the questionnaire length with beneficial impacts on unit and item non-response but at the cost of additional estimation error. The reduced approach further decreases questionnaire length by simply dropping items altogether, creating substantial bias in the resulting estimates. The bias can be minimized by adjusting the poverty line based on data from a previous consumption survey, called the adjusted reduced approach below. Finally, the largest time savings are generated by completely abandoning consumption data and using a structural model to impute consumption from a baseline survey using common co-variates. The time gap with the baseline survey as well as shocks and other structural changes, for example from the implementation of a project to reduce poverty, invalidate the structural model leading to substantial bias in the estimates.

**Table 1: Comparison of consumption methodologies and sources of error.**

||**Unit non-response**|**Item non-response**|**Implementation issues**|**Error inherent to method**|
|---|---|---|---|---|
|**Full**<br>**consumption**|Elevated levels of<br>non-response,<br>particularly<br>among<br>urban and wealthy<br>households (Korinek,<br>Mistiaen,<br>and<br>Ravallion 2006; Osier<br>2016)|Long list of items<br>increase item non-<br>response<br>and<br>measurement<br>error<br>(Fiedler and Mwangi<br>2016;<br>Finn<br>and<br>Ranchhod 2015) as<br>well as unfinished<br>interviews (A. Deaton<br>and Grosh 2000)|Relatively limited issue<br>as<br>questionnaire<br>is<br>straightforward though<br>length may be issue|Theoretically<br>unbiased<br>if<br>implemented correctly with<br>full response|
|**Rapid**<br>**approach**|Non-response is an<br>issue but not as<br>much<br>as<br>in<br>full<br>consumption due to<br>the<br>shorter<br>questionnaire|Less of an issue than<br>full consumption due<br>to<br>shorter<br>questionnaire|Could be substantial<br>issue<br>with<br>paper<br>questionnaires<br>but<br>almost<br>completely<br>mitigated with CAPI|Trade-off between length of<br>questionnaire and accuracy of<br>poverty<br>and<br>inequality<br>estimates<br>due<br>to<br>underestimation of variance<br>attenuated by less core items<br>and more optional modules<br>(Table 2; Figure 1; Figure 2<br>and Figure 3).|
|**Reduced**<br>**consumption**|Non-response is an<br>issue but not as<br>much<br>as<br>in<br>full<br>consumption due to|Less of an issue than<br>full consumption due|Limited<br>issue<br>as<br>questionnaire<br>is<br>straightforward|Substantial<br>bias<br>in<br>total<br>consumption<br>attenuating|

9
||the<br>shorter<br>questionnaire|to<br>shorter<br>questionnaire||inequality (Figure 1 and Figure<br>2) (Beegle et al. 2012)|
|---|---|---|---|---|
|**Cross-Survey**<br>**(X-Survey)**<br>**imputations**|Non-response is an<br>issue but not as<br>much<br>as<br>in<br>full<br>consumption due to<br>the<br>shorter<br>questionnaire|Very small issue as no<br>consumption section|Very small issue as<br>questions are simple|Reliance<br>on<br>old<br>data<br>introducing bias in structural<br>model<br>leading<br>to<br>biased<br>poverty estimates (Figure 3)|

Using the full consumption data from KIHBS 2015/16 as benchmark, in this section, first, we investigate the empirical trade-off between the number of core items and the number of optional modules for the rapid approach with respect to the performance of poverty indicators. Second, we compare the rapid approach with a traditional reduced consumption aggregate, without any adjustment of the poverty line. Third, we again use a reduced consumption aggregate, but adjust the poverty line based on previously observed consumption from 2005/6. Fourth, we compare the application of the rapid approach in the pilot in Kenya with a cross-survey imputation approach also using the 2005/6 data as baseline.

# **Trade-off with Number of Core Items and Optional Modules**

As discussed in the methodology section, the rapid approach creates a trade-off between the number of core items and the number of optional modules, as a larger number of core items (smaller number of optional modules) will improve the performance of the estimation. A larger number of core items will capture a larger fraction of the variance in the core module, minimizing the estimation error for the variance. Similarly, a smaller number of optional modules reduces the estimation error of the covariance between modules. However, the time savings by the rapid approach are larger for a smaller number of items in the core module and a larger number of optional modules as fewer items are asked about for each household.

**Table 2: Trade-off for FGT measures by number of optional modules and number of core items measured by bias and cv.**

||||Bia|s|||||CV||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Opt.|||Core It|ems|||||Core It|ems|||
|Mod.|0|1|3|5|10|20|0|1|3|5|10|20|
|2|0.021|0.019|0.015|0.012|0.009|0.006|0.081|0.075|0.051|0.043|0.029|0.017|
|4<br>t0|0.033|0.032|0.024|0.019|0.014|0.007|0.125|0.122|0.085|0.07|0.051|0.026|
|6<br>fg|0.039|0.036|0.027|0.021|0.015|0.01|0.139|0.14|0.099|0.081|0.06|0.037|
|8|0.042|0.042|0.031|0.026|0.021|0.009|0.151|0.155|0.105|0.096|0.08|0.035|
|2|0.015|0.014|0.01|0.008|0.005|0.003|0.128|0.12|0.081|0.071|0.051|0.028|
|4<br>t1|0.024|0.023|0.016|0.013|0.009|0.005|0.19|0.189|0.129|0.114|0.087|0.045|
|6<br>fg|0.027|0.026|0.019|0.015|0.011|0.007|0.207|0.214|0.15|0.131|0.103|0.062|
|8|0.029|0.029|0.02|0.018|0.015|0.006|0.224|0.232|0.156|0.151|0.132|0.063|
|2|0.011|0.01|0.007|0.006|0.004|0.002|0.166|0.157|0.108|0.098|0.075|0.04|
|4<br>t2|0.017|0.017|0.012|0.01|0.007|0.004|0.241|0.241|0.166|0.154|0.123|0.064|
|6<br>fg|0.02|0.019|0.014|0.011|0.008|0.005|0.259|0.272|0.192|0.174|0.144|0.085|
|8|0.021|0.022|0.015|0.013|0.011|0.005|0.279|0.29|0.198|0.198|0.179|0.09|
|2|0.005|0.001|0.001|0|0.003|0.002|0.014|0.004|0.004|0.004|0.009|0.005|
|4<br>ini|0.004|0.001|0.003|0.001|0|0.003|0.013|0.009|0.01|0.004|0.004|0.01|
|6<br>g|0.009|0.005|0.005|0.008|0.006|0.008|0.025|0.014|0.014|0.021|0.015|0.021|
|8|0.001|0.008|0.011|0.003|0.006|0.007|0.011|0.022|0.03|0.01|0.015|0.018|

Based on simulations using KIHBS 2015/16, we estimate the bias and coefficient of variation to estimate FGT0, FGT1 and FGT2 as well as the Gini using as reference the full consumption aggregate from the 10
survey.<sup>11</sup> As expected, all performance measures deteriorate for a smaller number of core items, as well as for a larger number of optional modules (Table 1). Using the minimum number of optional modules (2), we obtain an average bias of 0.021 for FGT0 using no core items, but a considerably smaller bias of only 0.006 (a more than 70 percent reduction) if using 20 core items. An increase of the number of optional modules from 2 to 8 almost doubles the bias for FGT0 from 0.021 to 0.042 using 0 core items. We observe similar trade-offs for FGT1 and FGT2. The trade-offs for the Gini coefficient are less clear as independent of the number of core items and modules the bias is consistently extremely low, almost always below 0.01.

# **Rapid vs. Reduced Estimation**

Traditionally, time savings in administering consumption modules are achieved by reducing the number of consumption items included in the questionnaire. Here we compare the rapid approach with a reduced approach based on the effective time savings achieved. Assuming that only consumed items require substantial interview time, we estimate the average number of items that were administered and consumed by households relative to the total number of items in the full consumption module. The smaller the number of items consumed and administered in the questionnaire, the larger the time savings. This measure takes into account that effective time savings are smaller if fewer but often-consumed items are administered to a household, compared to a larger number of items, which are rarely consumed. The results show simulations with varying numbers of core items and optional modules for the rapid approach, and varying numbers of items included in the reduced module.

> 11 If not noted otherwise, each core and optional module configuration is run 20 times, each using 50 multiple imputations. Note that the same survey data are used to define the core module items (based on highest consumption shares).

11
<!-- Start of picture text --> fgtO fgtO<br>8 6 os ge, .<br>8 6 ‘4 FS 4 . ee eee hee,° a re<br>0 0 ee .<br>0 2 4 6 8 1 0 2 4 6 8<br>Proportion of effective questions Proportion of effective questions<br>fgt1 fgt1<br>5 8<br>a4 . 6 eSe Soee e<br>g 3 34 = i<br>00<br>02 4 6 8 1 0 2 4 6 8<br>Proportion of effective questions Proportion of effective questions<br>fgt2 fgt2<br>4 : 8 set<br>0 0<br>ie) 2 4 6 8 1 0 2 4 6 8<br>Proportion of effective questions Proportion of effective questions<br>gini gini<br>.04 oh oe .<br>an 03 e oor 08 .<br>& 0 12 oo eeeUwe er eee- 3 02‘01 nerof eeeee *<br>0 e? eve ry toa ry 0 ee te e<br>0 2 A 6 8 1 0 2 4 6 8<br>Proportion of effective questions Proportion of effective questions<br>e Rapid (avg) ———._ Rapid (avg; fitted) e Rapid (max) —= Rapid (max; fitted)<br>e Reduced (avg) ——————_ Reduced (avg; fitted) Reduced (max) Reduced (max; fitted)<br><!-- End of picture text -->
<!-- Start of picture text --> fgtO fgtO<br>15 6 Pers °<br>0 o's 0 ott en ge ee, ee<br>0 2 4 6 8 1 0 2 4 6 8<br>Proportion of effective questions Proportion of effective questions<br>fgt1 fgt1<br>8 .06.04 site 5 * > 8¢ tea ee<br>) . ° — : ) we tet*_s epee as ee<br>0 2 4 6 8 1 0 2 4 6 8<br>Proportion of effective questions Proportion of effective questions<br>fgt2 fgt2<br>P 6 eS<br>0 Py A —" “ 0 a*__* ee re ee Ye<br>0 2 4 6 8 1 0 2 4 6 8<br>Proportion of effective questions Proportion of effective questions<br>gini gini<br>n ou ween tSs at 06.08 . °<br>g2 || ° e ° ° 3 .04 e oe @ . s<br>02 ° ° ee ee * oe ows 2 we, 02 I<br>0 oor fe ° we ® ge 0 ce e ee .<br>0 2 4 6 8 1 0) 2 4 6 8<br>Proportion of effective questions Proportion of effective questions<br>e Rapid (avg) ———_ Rapid (avg; fitted) e Rapid (max) ——= Rapid (max; fitted)<br>e Adj. reduced (avg) ——————_ Adj. reduced (avg; fitted) Adj. reduced (max) Adj. reduced (max; fitted)<br><!-- End of picture text -->
a few core items with highest consumption shares, even if they are selected from a rather outdated survey as in Kenya with a 10-year gap.

The approach of using an adjusted poverty line performs significantly better than the simple reduced approach presented in the previous section, but at the cost of the distributional shape captured in the Gini coefficient. For the FGT measures, it is, thus, generally advisable to adjust the poverty line if a reduced approach must be used, even if the adjustment is based on outdated consumption shares from an old survey. For FGT0, the average bias of the adjusted reduced approach is usually around or above the maximum bias of the rapid approach. Furthermore, the maximum bias is considerably larger than the average bias for the adjusted, reduced approach compared to the rapid approach. For FGT1, the average bias of the adjusted, reduced approach becomes more comparable with the rapid approach, but the maximum bias is still significantly larger than the maximum bias of the rapid approach. For FGT2, the performance of the adjusted, reduced approach becomes more difficult to interpret. The Gini coefficient is not well conserved approaching a bias of 0.05 for highest time savings.

As the results show, the adjusted, reduced approach has larger variation across poverty lines (as shown by the larger difference between average and maximum bias) as well as across the different number of items (implying different time savings). The definition of the estimator explains this. The estimator depends strongly on the items selected for the reduced approach, as the accuracy relies on the adjustment factor of the poverty line, which is the share captured by those items. The error can be decomposed into two components. The first component is the change of the distributional shape between the survey used for the adjustment and the application of the adjusted poverty line. The second component depends on the difference in the share captured by those items between the two surveys. Both errors become zero if the approach is implemented for the same population at the same time. In the usual case though, neither the population nor the time point is the same. In these cases, especially the second component of the error leads to large variation of the performance. Even though the consumption shares of the items can change, the changes might cancel out leading to a good performance of the approach. However, adding one more item to the consumption module can void the cancellation, leading to a worse performance. Without knowing the share of the items from total consumption (which is not measured), it is impossible to predict how many items should be selected for a good performance of the approach.

# **Application to Kenya: Rapid vs. Cross-Survey Estimation**

In 2015/16, a CAPI pilot was implemented alongside KIHBS 2015/16 using the rapid approach. While the configuration is conservative with only 30 percent time savings, the results show impressive performance for all FGT measures (Figure 3) in comparison with the full consumption as estimated for KIHBS 2015/16. Across all potential poverty lines, the rapid approach has a bias of below 1.3 percentage points for FGT0, 0.7 percentage point for FGT1 and 0.6 percentage point for FGT2. The Gini has a bias of only 0.012.

We compare the performance with a cross-survey imputation of consumption using a structural model built based on the KIHBS 2005/6 data set and applied to KIHBS 2015/16, ignoring the collected consumption data in 2015/16. The performance of the structural model is then assessed against the KIHBS 14
<!-- Start of picture text --> fgtO<br>.08<br>.06<br>8 .04<br>2 02<br>0<br>-.02<br>0 20 40 60 80 100<br>Poverty Percentile<br>fgt1<br>.06<br>.04<br>S .02<br>2<br>0<br>-.02<br>0 20 40 60 80 100<br>Poverty Percentile<br>fgt2<br>.04<br>.03<br>g 02<br>2 01<br>0<br>-.01<br>0 20 40 60 80 100<br>Poverty Percentile<br>————._ Rapid -_—_—X-Survey<br><!-- End of picture text -->
is helpful to utilize a previous survey from the same or a similar population to assign a few key items to the core module. In the best case, the selected items are still highly consumed and will improve the estimates of poverty. In the worst case, the selected items are no longer important in which case they will hardly affect the time savings compared to a design with zero core items and also equal its performance. In countries with large variation in diet across regions, subnational core modules can potentially improve estimates even further.

This paper also demonstrates the difficulty of achieving convincing results using alternative methods. Simply removing items from the consumption aggregate to create time savings, but without adjusting the poverty line, can lead to considerable bias in the poverty estimates. Better results can be achieved by adjusting the poverty line based on a previous survey in order to accommodate the reduced number of items, but without conserving the shape of the consumption distribution measured by the Gini. Furthermore, the potential for large changes in consumption patterns, which cannot be determined under this approach, creates considerable uncertainty in the resulting estimates. Similarly, a survey based only on covariates and their structural relationship with poverty estimates from a previous survey introduces a large bias in the estimates, at least in the studied case with a 10-year gap between surveys. The proposed rapid approach outperforms all these approaches, only at the cost of increased complexity.

The rapid approach introduces additional complexity into both the questionnaire design and the estimation of poverty. The capacity of enumerators is often low in developing countries. While the rapid approach increases the complexity of the questionnaire, CAPI technology easily solves this problem. Survey software can automatically compile a single consumption module based on the core and optional modules for each household, without making the partition explicit to the enumerator or demanding the execution of complex skip patterns. Furthermore, advanced CAPI technology can be used to generate the questionnaire automatically based on the assignment of the household to an optional module. While enumerators should be made aware that different households will be asked for different items, administering a rapid questionnaire does not require any additional training of enumerators beyond the standard skills for consumption questionnaires.

Conversely, the analysis of data using the rapid approach requires high analytical capacity, something that is usually lacking in developing countries. While the general concept of the assignment of optional consumption modules to households can usually be explained to local partners, poverty analysis based on a bootstrapped sample of the consumption distribution can potentially be too demanding for local capacity. However, even standard poverty analysis often surpasses the limits of local capacity, especially in conflict or post-conflict settings. Therefore, capacity building tends to focus on data collection skills with the long-term perspective of creating data analysis capacity. In addition, the rapid approach might be the only possibility to create poverty estimates in certain areas. For example in the case of Somalia, the rapid approach limited overall questionnaire administering time to less than 60 minutes for more than 90 percent of households as required by security considerations for enumerators (Pape and Mistiaen 2018; Pape and Wollburg 2019).

The rapid approach administers different consumption modules to different households. In theory, this can create a response bias if households report differently on a consumed item depending on the type 16
and number of items previously asked. Unfortunately, we cannot estimate such a response bias in the available data. However, implementation of the rapid approach with an enhanced design with different optional modules varying in their comprehensiveness of items can in general shed light on this bias. Comparison between responses for the same item in a comprehensive and a non-comprehensive list can also indicate a lower bound for response bias. Assuming that the context of a comprehensive list is a better estimate, the response bias could be corrected for. However, it is expected that this type of response bias is very small in comparison to general measurement and estimation errors.

The main source of bias for the rapid approach is created by the assumption of zero co-variance between optional modules. Further research can help to estimate co-variance between modules within the survey and adjust the consumption estimates accordingly. Using a random assignment of items to optional modules, the co-variance between groups of items within an optional module can potentially be used to estimate the co-variance between optional modules. Or administering optional modules that share items might also be helpful to estimate the co-variance between modules.

The rapid approach reduces administering time considerably. While this creates opportunities to include additional questionnaire modules on different topics (e.g. remittances or health), it also has the potential to reduce the non-response rate (A. Deaton and Grosh 2000). The KIHBS 2015/16 survey suffered from a high non-response rate specifically in wealthier areas. For example, the capital city Nairobi had a response rate of 77 percent compared to 92 percent in the rest of the country. The highest non-response rates were specifically observed in clusters in wealthier areas within Nairobi. A high correlation of non-response with welfare status can lead to biased poverty estimates. The considerably shorter pilot survey – which not only used the rapid approach for consumption but generally shortened the questionnaire across modules and was carried out using tablets rather than paper – did not show the same pattern of lower response rates in wealthier areas. The pilot response rate of 99 percent in Nairobi was considerably higher than the standard KIHBS and was the same as in the rest of the country.<sup>14</sup> Thus, the rapid approach can help to contribute to shorter questionnaires mitigating concerns around low response rates, especially if correlated with welfare status. In addition, the rapid approach is likely to reduce enumerator and respondent fatigue based on the documented impact of fatigue on (consumption) estimates in the literature (Rolstad, Adler, and Rydén 2011; Diehr et al. 2005; Snyder et al. 2007; Fiedler and Mwangi 2016; Finn and Ranchhod 2015).<sup>15</sup> The rapid approach might also be particularly useful in the context of evaluating shock or project impacts on poverty. In these cases, reliance on structural models estimated between surveys is dangerous. Shocks are likely to distort structural relationships between household characteristics and poverty. For example, a light shock is often mitigated by the household reducing its consumption, rather than selling assets or moving into another dwelling. A structural model estimated before the shock will not be able to capture the reduced consumption, thereby underestimating the impact of the shock. Similarly, project impacts cannot be adequately estimated using a structural model from before the project. For example, the

> 14 The change in the response rate is unlikely to be explained by the transition from PAPI to CAPI (Banks and Laurie 2000; Schräpler, Schupp, and Wagner 2010).

> 15 Estimating the reduction of fatigue on measurement error would require a specifically designed survey.

17
distribution of metal sheets as rooftop materials is unlikely to change consumption patterns, but a structural model might use the absence of metal roofs to help predict poverty. While administering a full consumption module is often not feasible, especially in the case of shocks or in fragile settings, the rapid approach can readily be applied without relying on the assumptions of a structural model that would likely be violated.

18