metadata: authors: 0: author: Demombynes,Gabriel
1: author: Elbers,Chris T.M.
2: author: Lanjouw,Jean O.
3: author: Lanjouw,Peter F.
count: Mexico
docna: 0: docna: How good a map ? Putting small area
            estimation to the test
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Nutrition,Poverty Impact Evaluation,Poverty Monitoring & Analysis,Employment and Unemployment,Inequality,Poverty Assessment,Poverty Diagnostics,Poverty Lines,Small Area Estimation Poverty Mapping
teratopic: Health, Nutrition and Population,Science and Technology Development,Poverty Reduction,Education
abstracts: cdata!: The authors examine the performance of
            small area welfare estimation. The method combines census
            and survey data to produce spatially disaggregated poverty
            and inequality estimates. To test the method, they compare
            predicted welfare indicators for a set of target populations
            with their true values. They construct target populations
            using actual data from a census of households in a set of
            rural Mexican communities. They examine estimates along
            three criteria: accuracy of confidence intervals, bias, and
            correlation with true values. The authors find that while
            point estimates are very stable, the precision of the
            estimates varies with alternative simulation methods. While
            the original approach of numerical gradient estimation
            yields standard errors that seem appropriate, some
            computationally less-intensive simulation procedures yield
            confidence intervals that are slightly too narrow. The
            precision of estimates is shown to diminish markedly if
            unobserved location effects at the village level are not
            well captured in underlying consumption models. With well
            specified models there is only slight evidence of bias, but
            the authors show that bias increases if underlying models
            fail to capture latent location effects. Correlations
            between estimated and true welfare at the local level are
            highest for mean expenditure and poverty measures and lower
            for inequality measures.
geo_reg_key: 517191,119228
repnme: repnme: How good a map ? Putting small area
            estimation to the test
display_title: How good a map Putting small area
            estimation to the test
listing_relative_url: /research/2007/03/7488024/good-map-putting-small-area-estimation-test
topicv3: Poverty m1326687 695,Nutrition m1326779 644,Poverty Measurement and Analysis m1326941 707,Census Data m1504073 2763,Health m1504661 3480
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Health, Nutrition and Population,Science and Technology Development,Poverty Reduction,Education
no_of_pages: 41
author_keys: 0: author_key: 145125
1: author_key: 606516
2: author_key: 166048
3: author_key: 95180
model_extractions: WPS4155

# **How Good a Map? Putting Small Area Estimation to the Test**

Gabriel Demombynes, Chris Elbers, Jean O. Lanjouw, and Peter Lanjouw<sup>1</sup>

# <u>Abstract</u>

This paper examines the performance of small area welfare estimation. The method combines census and survey data to produce spatially disaggregated poverty and inequality estimates. To test the method, predicted welfare indicators for a set of target populations are compared with their true values. The target populations are constructed using actual data from a census of households in a set of rural Mexican communities. Estimates are examined along three criteria: accuracy of confidence intervals, bias and correlation with true values. We find that while point estimates are very stable, the precision of the estimates varies with alternative simulation methods. While the original Elbers et al (2002, 2003) approach of numerical gradient estimation yields standard errors that seem appropriate, some computationally less-intensive simulation procedures yield confidence intervals that are slightly too narrow. Precision of estimates is shown to diminish markedly if unobserved location effects at the village level are not well captured in underlying consumption models. With well specified models there is only slight evidence of bias, but we show that bias increases if underlying models fail to capture latent location effects. Correlations between estimated and true welfare at the local level are highest for mean expenditure and poverty measures and lower for inequality measures.

**Keywords:** Poverty, Inequality, Small Area Estimation **JEL Classification:** C13, C88, D31, I32, O15, R13

# World Bank Policy Research Working Paper 4155, March 2007

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent. Policy Research Working Papers are available online at http://econ.worldbank.org._

> 1 World Bank, Free University of Amsterdam, UC Berkeley and World Bank.  We are grateful to Martin Ravallion and Danny Pfeffermann for comments and suggestions.  The views in this paper are the authors’ and should not be interpreted to reflect those of the World Bank or affiliated institutions.
# **1 Introduction**

This paper examines the performance of a method for producing small area estimates of the spatial description of economic welfare. The methodology is described in Elbers, Lanjouw and Lanjouw (2002, 2003), henceforth referred to as ELL (2002). These “poverty maps” offer the promise of generating useful data about poverty and inequality at the local level, information which has potential applications in both the policy and research spheres. In this paper, an unusual data set is used to compare community-level welfare measures estimated using the small area estimation method against measures created from direct observations of household expenditure collected over the entire population within those communities.

Poverty maps have two sets of uses. They can be used as tools for geographical targeting of social spending. In a number of countries they have been used by governments and non-governmental organizations to identify those areas where the poor are concentrated as a first step towards directing resources to the poor. While policymakers in wealthy nations are accustomed to having information about local level conditions and welfare readily at hand, in the typical less developed country, information compiled at the local level is scarce and only available through specialized surveys. In such environments poverty maps are a potentially valuable resource.

On the research front, poverty maps have a variety of applications. With the resurgent interest in economic growth theory, and in particular the focus on inequality’s role, spatial profiles of welfare within a country can be useful. Poverty maps can also be used to investigate the spatial relationship between poverty and a variety of outcomes, including health and crime. The research applications for poverty maps are particularly strong when poverty maps can be produced for multiple years in a single country. In such cases poverty maps can be employed for policy evaluation.

2
The method examined here has been employed for a number of countries, and the resulting poverty maps have been utilized by both policymakers and researchers.<sup>2</sup> The growing popularity of the methodology adds to the need for a validation exercise.

The analysis in this paper compares the _predicted_ poverty and inequality rates produced by the methodology for groups of rural Mexican communities to the _actual_ poverty and inequality rates in those communities. One strength of the small area estimation approach is that it produces confidence intervals for its estimated welfare measures. An important objective in this paper is to assess to what degree the confidence intervals produced by the ELL method capture the distribution of error in the point estimates. Bias in the point estimates is also examined. The paper is organized as follows. Section 2 details the poverty mapping methodology. Section 3 describes the data employed, Section 4 sketches the validation exercise, and Section 5 presents the results. Section 6 concludes with a discussion of results and their implications.

# **2.  Methodology**

This section reviews the poverty mapping methodology, which is explained in more detail in ELL (2002).<sup>3</sup> The basic approach is straightforward and typically involves a household survey and a population census as data sources. First, the survey data are used to estimate a prediction model for either consumption or incomes. The selection of explanatory variables is restricted to those variables that can also be found in the census (or some other large dataset) or in a tertiary dataset that can be linked to both the census and survey. The parameter estimates are then applied to the census data, expenditures are predicted, and poverty (and other welfare) statistics are derived. The key assumption is that the models estimated from the survey data apply to census observations. The first stage begins with an association model of per capita household expenditure for a household _h_ in location _c_ , where the explanatory variables are a set of observable characteristics:

> 2  Poverty Maps based on this method are now underway or completed in more than 30 developing countries. Early examples include Alderman et al. (2002), and Mistiaen , Ozler, Razafimanantena and Razafindravonona (2002).  See also Demombynes et al (2002).

3
The locations correspond to the survey clusters as they are defined in a typical two-stage sampling scheme. The observable characteristics must be found as variables in both the survey and the census or in a tertiary data source that can be linked to both data sets.<sup>4</sup> Using a linear approximation to the conditional expectation, the household’s logarithmic per capita expenditure is modeled as The vector of disturbances, **_u_ ,** is distributed _F_ (0 **,Σ** ). The model in (2) is estimated by Generalized Least Squares using the household survey data. In order to estimate the GLS model, **Σ** , the associated error variance-covariance matrix, is estimated. Individual disturbances are modeled as where η _c_ is a location component and ε _ch_ is a household component. This error structure allows for both spatial autocorrelation, i.e. a “location effect” for households in the same area to the extent that it is not already covered by location-level explanatory variables, and heteroskedasticity in the household component of the disturbance. The two components are uncorrelated and (by construction) uncorrelated with observable characteristics in the regression equation.

The model in (2) is first estimated by simple OLS. The residuals from this ˆ regression serve as estimates of overall disturbances, given by _uch_ . These residuals are decomposed into uncorrelated household and location components:

> 3 Early variants of the methodology were presented in Hentschel et al (2000) and Elbers, Lanjouw and Lanjouw (2000).  These earlier versions differ in important ways with the approach outlined in ELL(2002).

4
The estimated location components, given by η ˆ _c_ , are the within-cluster means of the overall residuals. The household component estimates, _ech_ , are the overall residuals net ˆ 2 of location components. Additional parameters are estimated: ση , the variance of η _c_ ˆ 2 2 and _V_ σ , the variance of σ .<sup>5</sup> ( η) η To allow for heteroskedasticity in the household component, a logistic model of the variance of ε _ch_ conditional on a set of variables, **z** _ch_ , is estimated, bounding the prediction between zero and a maximum, _A_ , set equal to .1(05) * max{ _ech_ 2 } : ˆ Letting exp{ _z ch_<sup>_T_</sup> α } = _B_ and using the delta method, the model implies a household specific variance estimator for ε _ch_ of ˆ This heteroskedasticity model generates a vector of coefficient estimates, α , and the variance-covariance matrix, _V_<sup>ˆ</sup> ( α )ˆ . The coefficient estimates are used to predict ˆ 2 σε , _ch_ , the household-specific term for the variance of ε _ch_ .

These error calculations are used to produce two square matrices of dimension _n_ , where _n_ is the number of survey households. The first is a block matrix, where each block corresponds to a cluster, and the cell entries within each block are σ ˆ η 2 . The second

> 4   Note that these variables need not be exogenous.

> 5 See Appendix 1 of Elbers et al (2002) for details.

5
is a diagonal matrix, with household-specific entries given by σ ˆ ε 2, _ch_ . The sum of these two matrices is Σ<sup>ˆ</sup> , the estimated variance-covariance matrix for the original model given by equation (2). Once this matrix has been calculated, the original model is estimated by GLS.

In the second stage predicted log expenditures and subsequently local-level estimates of poverty and their accompanying standard errors can be generated via several routes. Elbers et al (2002) describe a method based on numerical gradient estimation. An alternative approach known as parametric bootstrapping (Pfeffermann and Tiller, 2005) has been found to yield closely similar results and proceeds as follows.<sup>6</sup> A series of simulations are conducted, where for each simulation _r_ a set of first stage parameters are drawn from their corresponding distributions estimated in the first stage. A set of beta and alpha coefficients, **_β_**<sup>~</sup> _r_ and **_α_**<sup>~</sup> _r_ , are drawn from the multivariate normal distributions described by the first stage point estimates and their associated variance-covariance ~ 2 _r_ matrices. Additionally, ( ση ) , a simulated value of the variance of the location error component is drawn.<sup>7</sup> Combining the alpha coefficients with census data, for each census household ( σ ~ ε 2, _ch_<sup>)</sup> _r_ , the household-specific variance of the household error component, is estimated. Then, for each household simulated disturbance terms, η<sup>~</sup> _cr_ and ε<sup>~</sup> _chr_ , are drawn from their corresponding distributions.<sup>8</sup> A value of expenditure for each ˆ _r_ ′ ~ _r_ household, _ych_ , is simulated based on both predicted log expenditure, **_x_** _ch_<sup>**_β_**</sup> , and the disturbance terms:

> 6 We will see below that while the methods yield very similar point estimates, the approach employed in ELL (2002) produces slightly wider (and possibly more plausible) confidence intervals.  In Appendix 1 we outline yet a third approach that yields confidence intervals that also more closely track those obtained with the method outlined in ELL (2002).

> 7 The ( σ ~ η 2 ) _r_ value is drawn from a gamma distribution defined so as to have mean σ ˆ η 2 and variance ˆ 2 _V_ σ . ( η)

> 8 Non-normality is allowed for in the distribution of both η _c_ and ε _ch_ .  For example, for each distribution, a Student's t-distribution can be chosen with degrees of freedom such that its kurtosis most closely matches ˆ

> that of the first stage residual components, η _c_ or _ech_ .  An alternative, semi-parametric, approach can also be adopted in which stardardized residuals are drawn from the first-stage survey residuals.

6
> Finally, the full set of simulated per capita expenditures, _y_ ˆ _chr_ , are used to calculate estimates of the welfare measures for each target population.<sup>9</sup>

> This procedure is repeated _R_ times drawing a new **_α_**<sup>~</sup> _r_ , **_β_**<sup>~</sup> _r_ , ( σ ~ η 2 ) _r_ and disturbance terms for each simulation.  For each subgroup, the mean and standard

> deviation of each welfare measure are calculated over all _r_ =1,..., _R_ simulations.  For any given location, these means constitute our point estimates of the welfare measure, while the standard deviations are the standard errors of these estimates.

There are two principal sources of error in the welfare measure estimates produced by this method.<sup>10</sup> The first component, referred to as _model error_ in ELL (2002) _,_ is due to the fact that the parameters from the first-stage model in equation (2) are estimated. The second component, termed _idiosyncratic error_ , is associated with the disturbance term in the same model, which implies that households’ actual expenditures deviate from their expected values. While population size in a location does not affect the model error, the idiosyncratic error increases as the number of households in a target subgroup decreases.

# **3. Data**

The analysis in this paper uses data collected as part of the targeting and evaluation program of PROGRESA, a health, education, and nutrition program of the Mexican government. Assignment to PROGRESA for households in these communities was randomized by community; a census of all households in 506 communities was conducted in November 1997, 320 were integrated into PROGRESA in late spring of 1998, and three follow up surveys (complete censuses) of households in all 506 communities were conducted in 1998 and 1999. Additionally, a survey was conducted in

> 9 These calculations are performed using household size as weights, implicitly assuming that expenditure is distributed uniformly within households. The same methodology could be applied using equivalence scales to capture alternative intrahousehold distributional assumptions.

7
March 1998, before PROGRESA was introduced to treatment communities. The March survey included a fairly detailed expenditure survey.

This paper employs household characteristic data from the November 1997 survey and an expenditure aggregate constructed using the March 1998 survey.<sup>11</sup> While it would be possible to undertake the analysis using income data from the November survey, the expenditure data is preferred for two reasons. First, the income data is very noisy. A substantial fraction of households report no income at all, and the income data shows no correlation with the March expenditure aggregate. The March expenditure aggregate, in contrast, is highly correlated with an expenditure aggregate from the June 1999 survey (for control group households), suggesting that it is a fairly consistent measure of household welfare. Second, the applications of the ELL methodology thus far have most commonly used household expenditure or consumption as the basis for welfare analysis, following the consensus that given the potential for consumption smoothing, consumption is likely to be a better indicator of long-term welfare than income. While it would be preferable to have expenditure data collected at the same time as household characteristics data, the household variables used here are unlikely to change substantially over time. Consequently the time gap between the November and March surveys should not distort the analysis.

While detailed, the expenditure aggregate is less comprehensive than typical consumption aggregates developed from some surveys carried out in developing countries. It covers only cash expenditures and does not include figures for rent. The expenditure survey was not carried out in 14% of households interviewed in November 1997. These households, which are concentrated in a small number of communities, are not included in the analysis. The ten communities with fewer than 10 households with expenditure information are also not included, leaving 20544 households in 496 communities.

> 10 A third potential source of error is associated with computation methods.  Elbers et al (2002) show that this can be set arbitrarily small by selecting a sufficiently large number of simulations.

> 11 Most questions in the November 1997 survey were similar to those in the 2000 national Mexican census. They concerned household characteristics and recent income of the household.

8
# **4. Analysis**

The approach used for the validation exercise is to estimate a first-stage model using a “pseudo-survey” drawn from the PROGRESA households, using a two-stage sampling procedure. Welfare measures are then predicted with target populations composed of groups of PROGRESA households. The PROGRESA communities themselves have too few households to produce meaningful confidence intervals for the estimates using the methodology. Previous experience, e.g. ELL (2002), has shown that standard errors are very large for target populations with less than a few hundred households. In order to generate a group of more suitably sized target populations, the communities were grouped at random into 20 target populations. Both the pseudo-survey and the target populations were drawn repeatedly, in order to generate estimates for a large number of target populations.

Specifically, the steps in the analysis were as follows:

- 1) A random sample of 50 localities was drawn from the 496 localities, with probability of selection proportional to the size of the locality. From each of the 50 localities, 10 households were selected at random. The data from these households (a total of 500) serve as a pseudo-survey.

- 2) The first-stage methodology described above was applied using the pseudo-survey. A set of explanatory variables for log per capita expenditure was selected from a candidate list.  An additional set of explanatory variables which best explained estimated location effects were selected from a set of community-level averages.<sup>12</sup>

- 3) The 496 localities were grouped into 4 groups of 24 communities and 16 groups of 25 communities. These serve as the 20 target populations for the poverty mapping

> 12 From equation (2) and (3) it is clear that the variance of the location effect η _c_ must be small if acceptable standard errors on welfare predictions are to be obtained. We have found that the inclusion of means of 2 explanatory variables, calculated from the census for the relevant enumerationa areas, reduces ση considerably. See ELL(2002) for details and see also below.

9
analysis, and the location effect is modeled at the level of the localities. The target populations each cover an average of 1042 households.

- 4) True poverty and inequality rates were calculated for the 20 target populations based on actual per capita expenditure.<sup>13</sup>

- 5) The poverty mapping methodology was applied to predict poverty and inequality rates for the 20 target populations, using first-stage models estimated with the pseudosurvey.

- 6) The entire procedure was repeated 10 times, drawing a new pseudo-survey for each round of analysis.

The output of this procedure is a set of poverty and inequality estimates and associated standard errors for 200 target populations. To examine the sensitivity of the estimates to the error specification, two different specifications are used for the secondstage analysis. In the first, both the location component and the household component of the error are modeled as Student’s t-distributions. For the second specification, a semiparametric approach is used for both the location and the household components. In this semi-parametric approach, instead of drawing from a t-distribution, the standardized residuals are drawn from the first-stage survey residuals. For both specifications, the household component of the error is modeled as heteroskedastic, with the predicted log per capita expenditure as the sole explanatory variables.<sup>14</sup>

> 13 The poverty line was set to 159 pesos, the per capita expenditure of the median household in the full set of households. This corresponds roughly to PROGRESA’s poverty-classification scheme; using discriminant analysis techniques based on household income, approximately 50% of households were initially classified as “poor” and thus qualified for PROGRESA.

> 14 Note that for the semi-parametric approach, it is the _standardized_ residuals that are drawn from the observed distributions in the survey. These standardized residuals, with mean zero and variance equal one, are drawn and multiplied by the square root of the relevant simulated variance (of the location or household effect) to produce simulated residual values.

10
# **5. Results**

# _First-Stage Results_

OLS Regression results from the first-stage models are given in Appendix 2 Tables A1-A10. Across the ten pseudo-surveys used here, the R<sup>2</sup> ranges from 0.415-0.53 (see Table 1). The explanatory power of the models in this analysis is in the general range of models from past applications. The R<sup>2</sup> for models for particular strata ranged from 0.45 to 0.77 in Ecuador (Hentschel et al, 2000), 0.29 to 0.63 in Madagascar (Mistiaen et al, 2002), and 0.47 to 0.72 in South Africa (Alderman et al, 2002). The explanatory power achieved with the PROGRESA models is rather good given that the households in the PROGRESA communities are more homogenous than those within a stratum in a typical application. All the communities in the PROGRESA sample were selected for the program because they were poor and rural, based on indicators in the 1990 and 1995 censuses. Consequently, the households are more similar to one than another than the households in an entire stratum of a country.

Household size was used in all models, and some variables were selected in models for several pseudo-surveys, but there was generally little consistency in models chosen across pseudo-surveys. The estimated location effects were generally small, with variances ranging from 0.9% to 3.1% of the overall variance of the disturbance term after the addition of cluster-level means. This can also be seen in that the models achieved levels of explanatory power very close to what would be achievable with models that employed, instead, a cluster-level fixed-effects specification (see Table 1).

# _Second-Stage Results_

# 5.1 <u>Point Estimates and Precision</u>

Tables 2 and 3 present illustrative results for the headcount rate based on two pseudo surveys: 2 and 3.<sup>15</sup> These tables present for each of the 20 target populations a measure of the true headcount rate as well as the estimated headcount rate based on a variety of procedures. Column 1 presents estimates and standard errors based on the

> 15 These two pseudo-surveys have been chosen arbitrarily in order to avoid unnecessary repetition. Qualitative conclusions are unchanged if other, or all, pseudo-surveys are examined.

11
numerical gradient simulation procedure sketched out in Elbers et al (2002). Columns 2- 4 present estimates based on the “parametric bootstrapping” (Pfeffermann and Tiller, 2005) procedure outlined in section 3 and are computed using the POVMAP2 software that has been purpose-written by Qinghua Zhao in the Research Department of the World Bank.<sup>16</sup> The parametric bootstrapping results vary depending on whether disturbances are drawn from the empirical distribution (Column 2) or from parametric distributions (Column 3). The estimates in column 4 are based on a program written in SAS, based also on application of the procedure outlined in section 3 (with disturbances drawn from a parametric distribution), and are presented to illustrate that simulation based results do vary depending on different random number generating algorithms as well as seeds. Finally the results presented in Column 5 are based on an alternative, non-parametric, scheme outlined in Appendix 1.<sup>17</sup> Point estimates differ only slightly across different simulation approaches. In Table 2, while the true headcount rate for target population 1 is 60.5% the estimated rate for this target population varies between 60.9% and 61.6% across the different estimation approaches. The approaches are more clearly at odds in terms of the estimated standard errors. In particular, standard errors deriving from the “parametric bootstrapping” procedure described in Section 3 and summarized in Columns 2-4, tend to be somewhat smaller than those based on the numerical gradient method described in ELL(2002) – Column 1 - and the non-parametric approach of Appendix 1 (Column 5). In the case of pseudosurvey 2 the distinction is not of great significance: irrespective of methodology, the 95% confidence interval around each target population’s estimated headcount rate encompasses the true poverty rate in 19 out of 20 cases. However, with other pseudo surveys the distinction does matter. In Table 3, results are presented based on a model of consumption estimated from pseudosurvey 3. With this survey, the “classical” approach (Elbers et al, 2002) and the alternative approach outlined in the appendix yield three

> 16 POVMAP2 can be freely downloaded at <u>http://iresearch.worldbank.org.</u>

> 17  Note that these estimates do not show significant differences in poverty between target populations. This reflects both the relative homogeneity of the group of PROGRESA households, the random composition of target populations, and  the small sizes of the target populations, about 1000 households. On the other hand, discriminating between poverty of the target populations is not the subject of the current paper and all standard errors are about the same size as one would get from survey-based estimates at the aggregate level..

12
cases where a target population’s true poverty rate falls outside the 95% confidence interval around the estimated poverty rate. But with the parametric bootstrap approach underpinning estimates in Columns 2-4 the failure rate is higher (7 cases). For this pseudosurvey the parametric bootstrapping approach appears to produce standard errors that are too “optimistic” - suggesting greater precision of estimates than is warranted.

Given this evidence of a tendency for the parametric bootstrapping procedure to produce confidence intervals that are somewhat too narrow, we employ from now on, unless noted explicitly otherwise, the non-parametric approach outlined in Appendix 1. Additional comparisons, not reported here, confirm that conclusions derived with this simulation procedure hold also for estimates based on the considerably more computationally-intensive numerical gradient approach outlined in ELL(2002). The important point to take away here is that simulation methods do seem to matter (with respect to standard errors, if not point estimates). Further research is underway to understand better why the different simulation methods do not always agree.<sup>18</sup> Table 4 looks more closely at the confidence intervals estimated around welfare estimates produced with our non-parametric simulation scheme. If the confidence intervals accurately reflect the true uncertainty in the estimates, the fraction of cases of the “truth” falling within a confidence interval around an estimate should be approximately equal to the corresponding confidence level. Note however that twenty ‘target populations’ are drawn for each of the ten ‘surveys’ and so the experiments are not entirely independent.

For each welfare measure and each of the ten pseudosurveys the number of instances is counted when true welfare in each of the 20 target populations falls within two standard deviations around the target population’s estimated welfare level. For example, in the case of pseudo survey 1, the true welfare estimate (mean, headcount, squared poverty gap, and General Entropy Class inequality measure with parameter 0) always fall within the confidence interval around the estimated welfare measure. In Table 2 we saw that for pseudosurvey 2 this occurred 95% of the time (19 out of 20 cases) for the headcount, and Table 4 shows the same was observed for the mean, while

> 18 The most recent version of POVMAP2 now offers the user the choice of the  “classical” numerical gradient or  the parametric bootstrapping procedures outlined in Section 3.  .

13
for the squared poverty gap and inequality calculated on the basis of the GE0 the truth always falls within the confidence intervals calculated around the estimates. On average, across all pseudo surveys the success rate is just under 95% for the mean consumption, headcount, and squared poverty gap measures, and just below 90% for the GE0 measure.

In Table 5 we consider how sensitive are our estimated standard errors to the presence of unobserved location effects. We saw in Table 1 that our preferred specifications for the different pseudosurveys were quite successful in proxying ˆ 2 σ η unobserved location effects ( ˆ 2 ranges between 0.9% and 3%). How much larger σ _u_ would standard errors be if our underlying models had not been so successful in this respect? Table 5 compares estimates and standard errors on small area estimates of the headcount rate from pseudosurvey 2 based on two models: one with our preferred specification; and the other with a specification in which no census-mean variables were included.<sup>19</sup> In the latter model the share of the variance of overall disturbance term that is attributable to the variance of the cluster component is now 11.9%, a four-fold increase over the 2.7% in the preferred model (Table 5). At the all-census level, the two models predict headcount rates of 61.9% and 61.5%, respectively, both virtually indistinguishable from the 61.1% actual headcount rate in the population. However, the standard error on the model with no location variables is now 0.024, up by more than two fifths from the standard error of 0.017 obtained with the preferred model. Part of the increase in the standard error is due to the fact that the explanatory power of the model with no location variables is lower than that of the preferred model. As a result, idiosyncratic error would be expected to be higher – see Section 2 and ELL(2002). However, at the level of the total population most of the idiosyncratic error will have cancelled out (poverty is being estimated over a population of more than 20,000 households). Thus the increase in the standard error from 0.017 to 0.024 is likely due mainly to the consequence of our failure to adequately capture unobserved location effects. At the target population level, standard errors are higher than at the level of the total population, irrespective of underlying models. Moving from the preferred specification to the model with no location variables, standard errors rise considerably,

> 19  Our calculations here are based on the numerical gradient “classical” simulation procedure.

14
and in some cases the percentage change is even greater than at the level of the total population. For example, standard errors across the two models rise by as much as 43% for target population 2 (0.030*1.43=0.043). However, here, the changes in standard errors are reflecting both the influence of idiosyncratic error and our failure to capture location effects.

# 5.2 <u>The Level of Location Effects</u>

Note that the location effect η _c_ may include group effects at levels higher than the survey cluster. To see this consider the following model with group random effects at a ‘district’ level ( _v_ ), as well a the cluster level ( _c_ ).: As before, the error components are uncorrelated. If clusters are the primary sampling unit, a district is sampled only indirectly, _viz._ if one of the sampled clusters happens to be located in that district. In a typical living standards survey there will only rarely be districts that have been sampled more than once in this way, making it impossible to separate the location effect in the sample into a ‘district effect’ ω and a ‘cluster effect’ η . Assume accordingly that a district is sampled at most once, and write _v_ ( _c_ ) for the unique district sampled along with the cluster. The model now becomes Or, with obvious relabelling:

* where η _c_ = ω _v_ ( _c_ ) + η _v_ ( _c_ ) _c_ .  Consequently, the estimated variance of the location effect in a model with only cluster-level random effects is in fact an estimate of σω 2 + ση 2, the combined group effects operating at the sample’s cluster level.

15
In the simulation phase the analyst has to choose whether the location effect estimated from the pseudosurvey should be applied at the cluster or the ‘district’ level. When there is no way of separating the location effect into a cluster and ‘district’ effect the best that one can do is to assume either that the effect is entirely a cluster-level effect, or that it occurs entirely at the district-level. The latter will be quite a conservative assumption as it will rule out that any part of the estimated location effect applies only at the cluster level. This approach might be considered as yielding an “upper-bound” on the standard error. The former will be “optimistic” in the sense that it will yield standard errors that could be under-estimates of the true-standard error – particularly if the location effect is big. In our setting, it does not make sense to apply the location effect at a level higher than the cluster, as the latter correspond to villages and these have been assembled _randomly_ into 20 target populations. ELL (2002) illustrate in the more plausible setting of rural Ecuador, however, that when it is assumed that the location effect estimated at the cluster level applies entirely at a higher level (in Ecuador, at the _parroquia_ level), then the idiosyncratic component of the standard error does rise appreciably. However, they also show that the impact on overall standard errors is negligible because in their setting – as in the present study – the size of the estimated location effect is small. If the introduction of cluster-means or other cluster-level variables is not successful in capturing group effects then the choice of level of aggregation at which to apply the location effect in the simulations can affect final results more substantially. In such a case there would be a larger range between the “optimistic” standard errors and the upper-bound estimates obtained by assuming that the location effect occurs entirely at the ‘district’ level.

# <u>5.3 Bias</u>

Another way in which to gauge the reliability of small-area estimates of welfare is to consider whether there is evidence of bias - a systematic tendency for estimates to deviate from the truth in any way. Figures 1-4 show, for each target population and for four different welfare measures, the relationship between true welfare and the difference between true and estimated welfare. In Figure 1 we can see that there is some tendency for the estimation procedure to overestimate mean per-capita consumption for those 16
target populations with a true mean consumption level that is low, and to underestimate the mean consumption level of rich target populations. To see this note that when true consumption is low, the bias - defined here as “truth” minus estimated consumption, is negative – while it is positive when true average consumption is high. However this relationship is not strong. Overall, the average difference between the estimated mean consumption and true consumption is about 1.5 pesos: about 1% of the mean consumption level of the poorest target population. The bias is similarly modest for the headcount (Figure 2), squared poverty gap (Figure 3) and mean log deviation (General Entropy class measure with parameter 0) inequality measure (Figure 4).

The extent of bias in these estimates is related to the degree to which the model specification fails to capture location effects on the basis of census-mean variables or other variables intended to capture locality-level characteristics. As we saw in the preceding section and in Table 1, our model specifications are quite successful in removing the effect of latent community level characteristics, and as a result the bias in our estimates is quite modest. If we produce estimates that omit village-level census means, then the bias is accentuated. Figure 5 illustrates how the slope of the line capturing the extent to which headcount is overestimated in truly non-poor communities and the headcount is underestimated in truly poor communities becomes steeper when estimates are based on a consumption model that fails to capture unobserved location effects. The intuition behind this bias is quite straightforward: if there is a sizeable location effect, and our model fails to capture it, then there will be a tendency for poverty to be over-estimated in communities that are relatively well-off, given the explanatory variables in the model, i.e. that have large positive location effects. Part of the reason that the communities are well-off is likely attributable to community-wide characteristics of the community, and this will not be reflected in estimates based on a model that fails to capture the effect of those characteristics. As a result estimates will tend to overstate poverty of such communities. Conversely, in truly poor communities, part of the reason they are so poor will be due to the broader characteristics of the community. Again, if the consumption model does not capture the impact of those broader characteristics, there will be a tendency for estimated poverty to be an understatement of true poverty in the community. We see, therefore, that not only is there a strong incentive to proxy location 17
characteristics in order to improve the precision of estimates (Section 5.1), but also in order to minimize a systematic tendency to overstate poverty in truly non-poor communities and understate poverty in truly poor communities.

# <u>5.4 Correlation</u>

A further way to consider the reliability of the small area estimates is to examine the correlation between the predictions and the true values. Table 6 shows simple pearson and spearman rank correlations between true and predicted values. Each cell shows the correlation between predicted welfare and true welfare across the 20 target populations. Rows represent alternative pseudosurveys and columns indicate alternative welfare measures. Correlations (both pearson and rank) are positive and reasonably high for mean consumption and the two poverty measures (headcount rate and squared poverty gap). In the case of inequality the correlations are much lower – presumably because the target populations vary very little in terms of true inequality. Indeed, households in the PROGRESA communities are more homogeneous than those within a stratum in a typical poverty mapping application. All the communities in the PROGRESA sample were selected for the program because they were poor and rural, based on indicators in the 1990 and 1995 censuses. Consequently, the households are more similar to one another than the households in an entire stratum of a country. This high level of homogeneity across households (and target populations) is a somewhat unusual feature of this empirical application. However, it might be expected to present a particularly difficult setting in which to implement the small-area estimation methodology and therefore does provide a useful (conservative) setting in which to gauge the methodology’s performance.

# **6.  Discussion**

The results presented here offer a rough test of the ELL methodology and point to some tentative conclusions that may inform future applications of the ELL welfare mapping method. In terms of the predictive power of the method, the results provide strong evidence that ELL estimates have important information content. Bias is low, the correlations between actual and predicted values of poverty indices and the mean are 18
generally positive and not insubstantial. For inequality figures, the results are generally weaker. Because the signal-to-noise ratio is lower in these inequality estimates, it is particularly important to take into account error in the estimates when applying them to research or policy applications.

The ability to provide confidence intervals is a crucial advantage to the ELL method as compared with alternative approaches to welfare mapping. In the analysis presented here, it was found that alternative simulation methods do influence the size of the estimated standard errors on welfare estimates. The numerical gradient approach, originally proposed in ELL(2002) was found to produce satisfactory standard errors, and similarly for the non-parametric simulation procedure outlined in Appendix 1. However, the parametric bootstrapping procedure described in Section 3 was found to yield standard errors that are somewhat understated. It is not entirely clear why this latter procedure should suffer from this propensity, and further research is needed to resolve this concern. An important objective of this analysis has been to document how important it is, when applying small-area estimation methods, to think hard about possible unobserved, community-level, factors that may influence welfare outcomes. Experience with “poverty mapping” in a large number of countries indicates that inclusion of censusmeans as regressors in the underlying consumption model (and/or the inclusion of household variables that capture “network” effects, or of additional community-level variables from tertiary datasets such as administrative and GIS data) can go a long way towards helping to secure specifications in which unobserved location effects are kept small. The analysis here has shown that failure to capture such location effects in this way can lead to markedly higher standard errors and also an increase in bias.

It is important to recognize the limitations of the analysis in this paper. The data used here are less well-suited to poverty mapping than those usually employed. First, the expenditure aggregate used is less comprehensive than that found in a typical developing country survey, and the general quality of the data may be worse than, for example, data collected in a World Bank LSMS survey. This reduces the potential for variation in expenditure to be explained by observed variables. Second, the data all come from poor households in rural Mexico. Consequently, there is relatively little variation in 19
expenditure across households, and a relatively large fraction of the variation is due to measurement error or short-term fluctuations and cannot be explained by observable characteristics.

The problem associated with the small range of expenditures is compounded in this exercise by the fact that it was necessary to construct target populations by randomly assembling groups of communities. This resulted in a narrow spread of welfare measure values across the target populations. The ELL method is likely to produce estimates with a higher signal-to-noise ratio when the underlying population has greater variation in consumption.

All in all, the analysis presented here suggests that the details of poverty mapping matter. But the evidence does also suggest that the small area estimation procedure can provide useful, and reliable, estimates of welfare at fine levels of aggregation that survey data themselves would not be able to accommodate.

20