metadata: authors: 0: author: Corral Rodas,Paul Andres
1: author: Molina,Isabel
2: author: Nguyen,Minh Cong
count: World
docna: 0: docna: Pull Your Small Area Estimates up by the Bootstraps
docty: Policy Research Working Paper
subtopic: Inequality,De Facto Governments,Public Sector Administrative & Civil
            Service Reform,Public Sector Administrative and Civil
            Service Reform,Economics and Finance of Public
            Institution Development,State Owned Enterprise Reform,Democratic Government,Employment and Unemployment,Labor & Employment Law,Adaptation to Climate Change
teratopic: Poverty Reduction,Public Sector Development,Governance,Social Protections and Labor,Law and Development,Environment
abstracts: cdata!: After almost two decades of poverty maps
            produced by the World Bank and multiple advances in the
            literature, this paper presents a methodological update to
            the World Bank's toolkit for small area estimation. The
            paper reviews the computational procedures of the current
            methods used by the World Bank: the traditional approach by
            Elbers, Lanjouw and Lanjouw (2003) and the Empirical
            Best/Bayes (EB) addition introduced by Van der Weide (2014).
            The addition extends the EB procedure of Molina and Rao
            (2010) by considering heteroscedasticity and includes survey
            weights, but uses a different bootstrap approach, here
            referred to as clustered bootstrap. Simulation experiments
            comparing these methods to the original EB approach of
            Molina and Rao (2010) provide empirical evidence of the
            shortcomings of the clustered bootstrap approach, which
            yields biased point estimates. The main contributions of
            this paper are then two: 1) to adapt the original Monte
            Carlo simulation procedure of Molina and Rao (2010) for the
            approximation of the extended EB estimators that include
            heteroscedasticity and survey weights as in Van der Weide
            (2014); and 2) to adapt the parametric bootstrap approach
            for mean squared error (MSE) estimation considered by Molina
            and Rao (2010), and proposed originally by González-Manteiga
            et al. (2008), to these extended EB estimators. Simulation
            experiments illustrate that the revised Monte Carlo
            simulation method yields estimators that are considerably
            less biased and more efficient in terms of MSE than those
            obtained from the clustered bootstrap approach, and that the
            parametric bootstrap MSE estimators are in line with the
            true MSEs under realistic scenarios.
repnme: repnme: Pull Your Small Area Estimates up by the Bootstraps
display_title: Pull Your Small Area Estimates up by
            the Bootstraps
listing_relative_url: /research/2020/05/32057711/pull-your-small-area-estimates-up-bootstraps
topicv3: State Owned Enterprises m1326323 721,Poverty m1326687 695,Climate Change Adaptation m1327767 574,Household Surveys m1504071 2761,Census Data m1504073 2763
publishtoextweb_dt: 2020-05-21T00:00:00Z
historic_topic: Poverty Reduction,Public Sector Development,Governance,Social Protections and Labor,Law and Development,Environment
origu: VP, HD Practice Group (GGHVP)
no_of_pages: 52
author_keys: 0: author_key: 1204461
1: author_key: 1234915
2: author_key: 954025
model_extractions: Policy Research Working Paper 9256 Pull Your Small Area Estimates up by the Bootstraps _Paul Corral Isabel Molina Minh Nguyen_ Poverty and Equity Global Practice May 2020
Policy Research Working Paper 9256

# **Abstract**

After almost two decades of poverty maps produced by the World Bank and multiple advances in the literature, this paper presents a methodological update to the World Bank’s toolkit for small area estimation. The paper reviews the computational procedures of the current methods used by the World Bank: the traditional approach by Elbers, Lanjouw and Lanjouw (2003) and the Empirical Best/Bayes (EB) addition introduced by Van der Weide (2014). The addition extends the EB procedure of Molina and Rao (2010) by considering heteroscedasticity and includes survey weights, but uses a different bootstrap approach, here referred to as clustered bootstrap. Simulation experiments comparing these methods to the original EB approach of Molina and Rao (2010) provide empirical evidence of the shortcomings of the clustered bootstrap approach, which yields biased point estimates. The main contributions of this paper are then two: 1) to adapt the original Monte Carlo simulation procedure of Molina and Rao (2010) for the approximation of the extended EB estimators that include heteroscedasticity and survey weights as in Van der Weide (2014); and 2) to adapt the parametric bootstrap approach for mean squared error (MSE) estimation considered by Molina and Rao (2010), and proposed originally by González-Manteiga et al. (2008), to these extended EB estimators. Simulation experiments illustrate that the revised Monte Carlo simulation method yields estimators that are considerably less biased and more efficient in terms of MSE than those obtained from the clustered bootstrap approach, and that the parametric bootstrap MSE estimators are in line with the true MSEs under realistic scenarios.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at pcorralrodas@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Pull Your Small Area Estimates the up by Bootstraps

Paul Corral,<sup>_∗_</sup> Isabel Molina<sup>_†_</sup> and Minh Nguyen<sup>_‡_§</sup>

**Key words:** Small area estimation, ELL, Poverty mapping, Poverty map, Empirical best, Parametric bootstrap

**JEL classification:** C55, C87, C15

> _∗_ The World Bank Group - Human Development Group Chief Economist’s Office

> _†_ Universidad Carlos III de Madrid - Department of Statistics

> _‡_ The World Bank Group - Poverty and Inequality Global Practice

> §The authors acknowledge financial support from the World Bank. We thank Samuel Freije-Rodriguez, Roy van der Weide, Alexandru Cojocaru, and David Newhouse for comments on an earlier draft. We also thank Kristen Himelein, and Carlos Rodriguez for comments, suggestions and overall guidance. Additionally we thank Carolina Sanchez for providing support and space to work on this. Finally, we thank the Global Solutions Group on Welfare Measurement and Statistical Capacity, as well as all attendants of the Summer University courses on Small Area Estimation. Any error or omission is the authors’ responsiblility alone.
# **1 Introduction**

It has been almost two decades since the publication of Elbers, Lanjouw and Lanjouw’s (2003 - ELL henceforth) seminal paper on small area estimation (SAE). The methodology proposed by these authors has been the de facto methodology used by the World Bank to obtain small area estimates of poverty and inequality indicators and perhaps constitutes the most applied SAE method across the globe. The World Bank, in an effort to make the implementation of the method as simple as possible, created a free software package with a simple point and click interface that could be easily used by any practitioner. The PovMap software (Zhao, 2006) is of great value to World Bank staff and to many statistical agencies, and its ease of use has allowed the ELL approach to be adopted worldwide. It has also permitted the World Bank to provide training and spread this knowledge to many statistical agencies.

After the work from Van der Weide (2014), several important modifications and additions were made to the PovMap software, and thus to the toolkit used by the World Bank. The first change was to add a new method for estimating the variances of the model errors, called Henderson’s Method III (H3 - Henderson, 1953), which does not require normality because it is based on the method of moments. The second improvement involved adjusting the original Generalized Least Squares estimators of the regression parameters considered in Elbers, Lanjouw and Lanjouw (2003) by including the survey weights following Huang and Hidiroglou (2003). Another important modification involved the addition of Empirical Best/Bayes (EB) prediction assuming normality. The modifications came about due to advances in the SAE literature (Molina and Rao, 2010 - MR) and criticism of the ELL methodology (Haslett et al., 2010 and MR, 2010). The final important change was to consider a different bootstrap approach for obtaining the EB estimators and their corresponding error measures.<sup>1</sup> To make the application of the World Bank’s method friendlier to more advanced practitioners and to those who seek more flexibility, a World Bank team produced a Stata version of PovMap (Nguyen et al., 2018) which is available under the name “sae”. The development of the tool has made running multiple simulations and tests of the software much more straightforward. Since the creation of the “sae” Stata suite of commands, the World Bank has shifted towards training statistical agencies using this approach, which can facilitate replicability. Additionally, the code in the Stata package is open, so that curious practitioners can see in detail how methods have been implemented.

Small area estimation is a broad branch of statistics that focuses on improving estimates’ precision when household surveys are not large enough to achieve a desired level of precision. Within small area estimation methods, model-based techniques “borrow strength” from a larger data set or auxiliary data across areas through models linking the areas (regression-type techniques), which yield indirect estimators (MR, 2010). Most of the model-based techniques fall into two groups, methods based on unit-level models and those based on area-level ones. The former are commonly used when data on units (e.g. households) are available; when only area-level data are available (e.g. area means), the latter are used, see Fay and Herriot (1979).

In the context of poverty, methods based on unit-level models typically rely on estimating the distribution of the household’s welfare given a set of auxiliary variables or correlates. The model parameters are then used to simulate multiple welfare vectors from the fitted distribution for every single household in the census which commonly lacks a welfare measure for poverty measurement (Elbers et al., 2007). Using the simulated census vectors, it is possible to obtain poverty rates or any other welfare indicator, for every area (including the non-sampled ones). Perhaps the two most popular approaches for unit-level small area estimation of poverty indicators are the traditional ELL (2003) method used by the World

> 1The bootstrap is not discussed in Van der Weide’s paper.

1
Bank, and the EB approach by MR (2010).<sup>2</sup> This paper focuses on unit-level models for small area estimation; in particular, on the traditional ELL (2003) approach, the latest additions to the World Bank toolkit by Van der Weide (2014), and the original EB approach introduced by MR (2010). It sheds light on the nuances of the traditional approach by ELL (2003) and the EB addition by Van der Weide (2014), and it updates these methods in line with the original EB approach by MR (2010). The paper goes in depth into the differences between the estimation of the noise in the ELL approach (as implemented in PovMap and the sae Stata command), and the estimation of the MSE according to MR (2010). Then, the paper proposes: 1) an adaptation of the Monte Carlo simulation procedure by MR (2010) for calculation of the extended EB estimators that incorporate heteroscedasticity and survey weights as proposed by Van der Weide (2014); and 2) an adaptation of the parametric bootstrap method considered by MR (2010), which comes from González-Manteiga et al. (2008), for estimation of the corresponding MSE of the extended EB estimators. According to our simulation results, the adapted procedures for calculation of extended EB point estimates and corresponding MSEs represent a substantial improvement over the current approaches.

The paper reviews the current methods in chronological order. It begins describing the traditional ELL method in Section 2, then moves on to the EB method proposed by MR (2010) in Section 3 and finishes with the additions by Van der Weide (2014) in Section 4. It discusses these methods in Section 5. Then, in Section 6 it describes the proposed computational approaches for calculation of the extended EB estimators and for estimation of their MSEs. In Section 7, several simulation exercises, imitating those performed by MR (2010) but extending them to more realistic scenarios, are conducted to compare the different methods. Finally, conclusions are presented in Section 8.

# **2 Traditional ELL approach**

The Elbers, Lanjouw and Lanjouw (2002, 2003) methodology is the de facto small area estimation approach utilized by the World Bank. The methodology has been widely applied across the globe to produce poverty maps conducted by the institution.<sup>3</sup> The popularity of the methodology can be attributed, to some degree, to the availability of the PovMap software (Zhao, 2006),<sup>4</sup> which was programmed in C and offers users a simple point and click interface. The PovMap software is also incredibly computationally efficient and fast, allowing users, even with limited computing power, to work with census data without facing memory limitations. In 2018, a Stata version of the PovMap software was released (Nguyen et al., 2018). The Stata command “sae” replicates most of the procedures and methods of the original PovMap software, and has become popular because it allows for expansion of the methods available to users.<sup>5</sup> The ELL method assumes that the welfare _ych_ for each household _h_ within each location _c_ in the population is linearly related to a 1 _× K_ vector of characteristics (or correlates) _xch_ for that household, according to the nested error model: where _ηc_ and _ech_ are respectively location and household-specific idiosyncratic errors, assumed to be

> 2A factor that likely contributes to these being well known is the availability of software, that can be easily used to apply these approaches; PovMap (Zhao, 2006) for ELL and R’s sae package (Molina and Marhuenda, 2015)

> 3Poverty mapping is the common name within the World Bank for SAE methodology, where the obtained estimates are mapped for illustrative purposes.

> 4Downloadable from: http://iresearch.worldbank.org/PovMap/PovMap2/setup.zip

> 5Users should be aware that results from Stata and PovMap differ slightly due to the use of different random number generators

2
independent from each other, following Here, _C_ is the number of locations in which the population is divided and _Nc_ is the number of households in location _c_ , for _c_ = 1 _, . . . , C_ . Finally, _β_ is the _K ×_ 1 vector of coefficients.

Under the original ELL methodology, the locations indexed with _c_ are supposed to be the clusters, or primary sampling units (PSUs), of the sampling design and do not necessarily correspond to the level at which the estimates need to be produced. In fact, clusters are typically nested within the areas of interest (e.g. census tracks). Presenting estimates at a higher aggregation level than the clusters (for which random effects are included in the model) may not be appropriate in cases of considerable betweenarea variability, and may underestimate the estimator’s standard errors (Das and Chambers, 2017). A way to alleviate this is to include covariates that explain sufficiently well the between-area heterogeneity in the model ( _ibid_ ). In this regard, ELL (2002) suggests the inclusion of area-level covariates as a way to improve precision, because they can explain the between-area variation in welfare and hence might reduce the contribution of area-specific residuals. Marhuenda et al. (2017) recommend putting the location effects at the same aggregation level where estimation is desired. According to this, in this paper we consider that the clusters _c_ are equal to the areas where estimation is desired.

A key feature of the implemented ELL approach is that it considers heteroscedasticity and fits a preliminary model for that, called the alpha model. Very little research has gone into this aspect of the method. In most applications, the alpha model usually has a small adjusted _R_<sup>2</sup> (which may not reach 0.05) yet tends to play a considerable role in the obtained point estimates, especially for parameters beyond the poverty rate such as the Gini index, Theil index, Poverty Gap, etc. Heteroscedasticity is included in the description of the EB approach given in Rao and Molina (2015), but this is not implemented in the sae R package (Molina and Marhuenda, 2015). For simplicity, below we spell the steps of the traditional ELL approach only in the case of homoscedasticity.

The nested error model in (1) was originally fitted by Generalized Least Squares (GLS) accounting for heteroscedasticity, but Van der Weide (2014) updated this original GLS method to properly account for survey weights.<sup>6</sup> Currently, the implemented method fits the model by Feasible Generalized Least Squares (FGLS). In this procedure, a simple linear regression obtained by considering as model errors _uch_ = _ηh_ + _ech_ is fit using ordinary least squares (OLS), and afterwards the appropriate covariance matrix of regression parameter estimators is estimated.<sup>7</sup> The first stage of the process being an OLS is an important aspect, because most of the model testing and validation done for the ELL approach is in practice undertaken with the OLS fit. Nevertheless, the parameter estimates used for generating welfare in the census (see the steps below) come from the GLS fit accounting for heteroscedasticity and including the survey weights.

The variances of the model parameter estimators are required in the considered ELL bootstrap approach. To estimate these variances, ELL (2002) proposed to use the delta method. The actual implementation of the delta method is through a computationally intensive simulated numerical gradient. An estimate of the gradient vector is obtained by making perturbations of the parameters, and this is used to estimate the variances of the estimators via the delta method. ELL (2002) mentions also the possibility of drawing from the sampling distribution (parametric) as a way to incorporate the estimation error into the total prediction error. This later method has become the de facto approach used under the ELL methodology.<sup>8</sup>

> 6Haslett et al. (2010) presents the problems in the original GLS implementation of ELL.

> 7For a detailed look into the ELL approach, interested readers should refer to the original ELL papers (ELL, 2003; 2002) and Section 3 of Nguyen et al. (2018), which presents the current GLS estimator from Van der Weide (2014)

> 8In a comparison of the simulation methods proposed by ELL (2002), Demombynes et al. (2008) shows that the delta

3
The first implementation of the World Bank poverty mapping software was actually done in SAS (Demombynes, 2002). Poverty indicators and their corresponding standard errors were obtained by a bootstrap procedure relying on simulation. This simulation approach is very reminiscent of multiple imputation methods, where every single relevant parameter necessary for simulating vectors of welfare is drawn from its corresponding estimated asymptotic distribution (or an approximation to it). This approach is the one currently used in the World Bank’s tools for implementing the ELL (2002) approach: Stata’s sae package and PovMap. Specifically, the steps of the implemented bootstrap procedure designed to obtain the traditional ELL point estimates and their estimated noise are:

1. Fit the nested error model (1) to the survey data. This yields the vector of initial parameter estimates

- _θ_ ˆ0 = (ˆ _β_ 0 _,_ ˆ _ση_<sup>2</sup> 0<sup>_,_ˆ</sup><sup>_σ_</sup> _e_<sup>2</sup> 0<sup>)</sup><sup>_,_</sup>

where the 0 subscript is used hereafter to indicate that the estimates come from the original household survey. In the implemented version, the model is fit via FGLS as specified in Nguyen et al. (2018).

2. Draw new model parameters as follows. First, regression coefficients are drawn from The variance of the household-level errors is drawn, according to Gelman et al. (2004, pp. 364-365), from where _χ_<sup>2</sup> _n_<sup>_∗_</sup> _−K_<sup>denotesarandomnumberfromachi-squareddistributionwith</sup><sup>_n−K_degreesof</sup> freedom. Here, _n_ is the number of observations in the survey data used to fit the model and _K_ is the number of correlates used in the model. The variance of the cluster effects _ση_<sup>2</sup><sup>_∗_</sup> is drawn, according to Demombynes (2008) and Demombynes et al. (2002), from 3. Using the simulated model parameters in step 2, calculate the welfare for every household in the census _ych_<sup>_∗_fromthemodelas</sup> where the household-specific errors are generated as and the location effects are generated as Then construct the vector containing the _Nc_ simulated welfares for the households in location _c_ , denoted _yc_<sup>_∗_= (</sup><sup>_y_</sup> _c_<sup>_∗_</sup> 1<sup>_, . . . , y_</sup> _cN_<sup>_∗_</sup> _c_<sup>)</sup><sup>_T_,where</sup><sup>_Nc_isthenumberofcensushouseholdsinlocation</sup><sup>_c_.</sup> 4. With the vectors _yc_<sup>_∗_,</sup><sup>_c_= 1</sup><sup>_, . . . , C_,ofsimulatedcensuswelfares,indicatorscanbeproducedforall</sup> the locations. Let _τc_<sup>_∗_=</sup><sup>_f_(</sup><sup>_y_</sup> _c_<sup>_∗_)betheindicatorofinterestcalculatedbasedonthesimulatedvector</sup>

> method from ELL and the parametric drawing of the parameters provide similar results. In tests with pseudo surveys, the delta method seems to provide wider standard errors than the parametric approach (Demombynes et al. 2008), suggesting that perhaps the parametric estimates are too optimistic when compared to the delta method

4
for location _c_ ; for example, for the Foster, Greer and Thorbecke (1984 - FGT) class of decomposable poverty measures, the function _f_ ( _yc_<sup>_∗_)isdefinedas</sup> where _z_ is the poverty line, _pch_ is the size of household _h_ from location _c_ in the census and _I_ ( _ych_<sup>_∗< z_) = 1if</sup><sup>_y_</sup> _ch_<sup>_∗< z_andisequalto0otherwise.</sup> 5. Repeat steps 2 to 4 _M_ times. Although traditionally _M_ = 100, a larger number of replicates _M_ is recommended. Let _τc_<sup>_∗_(</sup><sup>_m_)</sup> be the indicator of interest obtained in mth replicate of the bootstrap. The ELL estimator is then given by the average across the _M_ bootstrap replicates, and the ELL estimated variance of the ELL estimator is given by As opposed to the EB method of MR (2010), which uses a Monte Carlo simulation procedure and a separate bootstrap procedure for MSE (see Section 3), in the above ELL procedure, a single computational algorithm tries to capture the noise of the initial model parameter estimates in the ELL standard error by varying the model parameters across simulations, which is aligned to Rubin’s rules (Rubin, 2004). This means that, in each replicate of the bootstrap, different values of the model parameters _β_<sup>_∗_</sup> , _ση_<sup>2</sup><sup>_∗_and</sup> _σe_<sup>2</sup><sup>_∗_</sup> are used to generate the welfare data. However, this step might entail an increase of noise in the ELL estimators.

Note that, if the model parameters were kept fixed to the initial survey estimates from step 1 (skipping step 2), the above ELL estimate _τ_ ˆ _c_<sup>_ELL_</sup> would be a basic Monte Carlo approximation to the marginal mean of the indicator, _E_ ( _τc_ ; _θ_<sup>ˆ</sup> 0), with respect to the distribution of the welfare (given the correlates) induced by the nested error model (1), with _θ_ replaced by the initial estimate _θ_<sup>ˆ</sup> 0. Note that _E_ ( _τc_ ; _θ_ ), evaluated at the true _θ_ = ( _β, ση_<sup>2</sup><sup>_, σ_</sup> _e_<sup>2),isunbiasedbydefinitionsinceitequalstheexpectationofthe</sup> indicator under the true model.<sup>9</sup> According to this, the prediction error of the ELL estimator can be decomposed as described in ELL (2003), where these sources of error are described as follows:

1. The **idiosyncratic error** is related to how the actual value of the expenditure for a given location, _c_ , deviates from its expected value due to unobserved aspects in the expenditure for the location (ELL, 2002). For locations with smaller population sizes, the underlying distribution is not likely approximated when errors are drawn. This, however, is related to the explanatory power of the independent variables in the model, the smaller the unexplained portion of the model, the smaller the idiosyncratic error. With poorly fitting models, estimates for locations with smaller populations are likely to suffer from more variability across simulations. Therefore, in order to minimize this error, one of the goals of the modeling stage is to obtain the highest possible _R_<sup>2</sup> .

> 9Note that under a model-based approach, the true value of the indicator _τc_ is random. Under this setup, an estimator/predictor _τ_ ˆ _c_ of _τc_ is said to be unbiased when _E_ (ˆ _τc − τc_ ) = 0 or, accordingly, when _E_ (ˆ _τc_ ) = _E_ ( _τc_ ).

5
2. The **model error** is related to the properties of the model parameters and is unrelated to the size of the target population (ELL, 2002). The magnitude of the error is dependent on the precision of the _β_ coefficients of the welfare model and the sensitivity of the indicator to deviations in welfare (ibid). Consequently, in order to minimize this source of error in the final estimates, it is recommended to remove all non-significant independent variables in the modeling stage.

3. The final source of error is the **computation error** , which is not related to the other two sources of error. This is related to the simulation and can be made as small as possible, as computational resources allow, by running a larger number of simulations.

Consider the case that model parameter estimates were kept fixed across simulations and equal to the initial survey estimates from step 1, _θ_<sup>ˆ</sup> 0 = ( _β_<sup>ˆ</sup> 0 _,_ ˆ _ση_<sup>2</sup> 0<sup>_,_ˆ</sup><sup>_σ_</sup> _e_<sup>2</sup> 0<sup>).In that case, consistency of these estimators as the</sup> total sample size _n_ tends to infinity,<sup>10</sup> and of Monte Carlo averages to the actual expectations under the model as the number of bootstrap replicates _M_ tends to infinity, would ensure that _τ_ ˆ _c_<sup>_ELL_</sup> approximates correctly the “theoretical” ELL estimator, _E_ ( _τc_ ; _θ_ ), which is unbiased by definition. However, there is an additional source of error due to generation of welfare in each simulation from a different set of parameter values in step 3, instead of using the initial estimates from step 1.

Moreover, if our target indicator is the mean welfare in location _c_ , _τc_ = _Nc_<sup>_−_1</sup> � _Nh_ =1 _d_<sup>_ych_,andthewelfare</sup> is not transformed with log, then _E_ ( _τc_ ; _θ_ ) = _E_ ( ̄ _xcβ_ + _ηc_ + ̄ _ec_ ) = _x_ ̄ _cβ_ , where _x_ ̄ _c_ = _Nc_<sup>_−_1</sup> � _Nh_ =1 _d_<sup>_xch_isthe</sup> area mean of the correlates and _e_ ̄ _c_ = _Nc_<sup>_−_1</sup> � _Nh_ =1 _d_<sup>_ech_istheareameanofthehousehold-levelerrors.This</sup> means that the area effect _ηc_ vanishes in the final ELL estimator, which becomes basically the regression ̄ estimator _xcβ_ . This estimator is called “synthetic” in the SAE literature because it does not account for between-area heterogeneity (or idiosyncrasy of the areas) beyond that explained by the correlates. Note that, if the regression parameter _β_ was actually known (best case), _x_ ̄ _cβ_ would not be using at all the actual survey data, only the census auxiliary information. Synthetic estimators rely very strongly on the assumed regression model and might be misleading if the model is incorrectly specified. In fact, standard errors are obtained assuming that the model actually holds and do not account for model misspecification. Hence, under model failure, they might still be small, giving a lot of credibility to the (misleading) estimates. As will be shown, even if the model is correctly specified, the MSE of the ELL estimators can be substantial, especially if the prediction power of the model is weak.

# **3 Molina and Rao’s EB and Census EB estimators**

Molina and Rao (2010) consider the nested error model (1) as in the ELL procedure, but the location effects are originally defined for the areas of interest. A more recent paper by Guadarrama, Molina and Rao (2018) further extends the EB method of MR (2010) to complex sampling designs, but so far it has not been implemented in any software package as a library or command, unlike the traditional ELL (2002) with the subsequent updates by Van der Weide (2014). For simplicity, this section describes the original EB approach proposed by MR (2010).

The main difference with the ELL approach is that the EB method of MR (2010) conditions on the survey sample data and thus makes a more efficient use of the survey data, which contains the only available (and hence precious) information on actual welfare. Nevertheless, conditioning on the survey data requires areas and households across survey and census to be matched. Thus, if _Pc_ denotes the area’s population of size _Nc_ , then the survey sample, _sc_ , is a sample of size _nc ≤ Nc_ drawn from _Pc_ . The complement to the sampled population is referred to as _rc_ , which is just _Pc − sc_ . Thus, the census

> 10Note that the total survey sample size _n_ is typically large.

6
welfare vector for any area _c_ is defined as _yc_ = ( _yc,s_<sup>_T, y_</sup> _c,r_<sup>_T_)</sup><sup>_T_,whichcontainsthewelfareforsampled</sup> and non-sampled observations in area _c_ , denoted _yc,s_ and _yc,r_ respectively. Typically, the non-sampled population (of size _Nc − nc_ ) is much larger than the sampled population (of size _nc_ ). It may also happen that _nc_ = 0 for some areas, meaning that the area is not sampled in the survey. The empirical best (EB) predictor of _τc_ for an area _c_ that is sampled ( _nc >_ 0) is defined as the conditional expectation _E_ ( _τc|yc,s_ ; _θ_<sup>ˆ</sup> ). For an out-of-sample area ( _nc_ = 0), the empirical best (EB) predictor of _τc_ is _E_ ( _τc_ ; _θ_<sup>ˆ</sup> ), which is similar to the ELL estimator. Here, _θ_<sup>ˆ</sup> is a consistent estimator of _θ_ as the overall sample size _n_ tends to infinity. In contrast with this procedure, the traditional ELL approach does not require to link the census and survey observations.

When the expectation _E_ ( _τc|yc,s_ ; _θ_<sup>ˆ</sup> ) has no explicit form, the empirical best (EB) predictor of _τc_ can be approximated using a Monte Carlo (MC) simulation method. The MC approximation to the EB ˆ predictor _τc_<sup>_EB_</sup> = _E_ ( _τc|yc,s_ ; _θ_<sup>ˆ</sup> ), as described in Rao and Molina (2015), is obtained as follows:

1. Using the survey data, fit model (1) via any method providing consistent estimators. This yields the vector of parameter estimates:

Usual fitting methods under this approach are maximum likelihood (ML) or restricted maximum likelihood (REML), both based on the normal likelihood, and H3 method, which does not require to specify a distribution. In the implemented sae R package, REML is used.

2. Use the parameter estimates obtained in step 1 as true values to simulate a vector of welfare in the census. First, the welfare from each out-of-sample household is generated as follows, ln( _ych,r_<sup>_∗_) =</sup><sup>_xch,rβ_ˆ0+</sup><sup>_η_</sup> _c_<sup>_∗_+</sup><sup>_e∗_</sup> _ch,r_<sup>_,_</sup> where here we add the subscript _r_ to indicate that the household is not sampled. Here, for an area _c_ that is included in the sample, the area effect _ηc_<sup>_∗_isgeneratedas</sup> where _η_ ˆ _c_ 0 and _γ_ ˆ _c_ are respectively given by Here, _y_ ̄ _c,s_ and _x_ ̄ _c,s_ are respectively the sample means of the welfare variable and the correlates in area _c_ . If area _c_ is not sampled, the area effect is generated as _ηc_<sup>_∗∼N_</sup> �0 _,_ ˆ _ση_<sup>2</sup> 0�. Finally, the household error _e_<sup>_∗_</sup> _ch,r_<sup>isgeneratedas:</sup> For an area _c_ that is sampled in the survey, the generated non-sample vector _yc,r_<sup>_∗_is then augmented</sup> by the survey data _yc,s_ . Therefore, the final vector for the whole census in area _c_ is _yc_<sup>_∗_= (</sup><sup>_y_</sup> _c,s_<sup>_T, y_</sup> _c,r_<sup>_∗T_)</sup><sup>_T_,</sup> which is made up of the generated out-of-sample welfares and the survey ones.

3. The previous step yields a census vector of welfare _yc_<sup>_∗_, simulated using the fitted model parameters.</sup> This census vector is then used to calculate the indicator for each area _c_ = 1 _, . . . , C_ , as where _f_ ( _·_ ) can be any indicator function such as the FGT poverty indicators.

7
4. Repeat steps 1 to 3 a large number of times _M_ . If _yc_<sup>_∗_(</sup><sup>_m_)</sup> denotes the _mth_ replicate of the census vector _yc_<sup>_∗_= (</sup><sup>_y_</sup> _c,s_<sup>_T, y_</sup> _c,r_<sup>_∗T_)</sup><sup>_T_,then</sup><sup>_τ ∗_</sup> _c_<sup>(</sup><sup>_m_)</sup> = _f_ ( _yc_<sup>_∗_(</sup><sup>_m_)</sup> ) is the corresponding indicator, _m_ = 1 _, . . . , M_ . The final MC approximation to the EB estimator of _τc_ is just the average across the _M_ simulations of these indicators Molina (2019) comments that the effect of adding the survey data is negligible when the sample is small relative to the census population of the area. Thus, a slight variation of the EB estimator, called Census EB (see e.g. Molina 2019), avoids appending the survey data in step 2 by generating all the census welfares (including the sample ones), similarly as the non-sampled ones are generated. Thus, the welfare for each household _h_ in the census is generated from the model with _ηc_<sup>_∗_and</sup><sup>_e∗_</sup> _ch_<sup>obtained exactly the same as in step 2.The result is the full census vector for area</sup><sup>_c_given</sup> by _yc_<sup>_∗_= (</sup><sup>_y_</sup> _c_<sup>_∗_</sup> 1<sup>_, . . . , y_</sup> _c,N_<sup>_∗_</sup> _c_<sup>)</sup><sup>_T_,whichisusedtocomputethetargetindicator</sup><sup>_τ ∗_</sup> _c_<sup>=</sup><sup>_f_(</sup><sup>_y_</sup> _c_<sup>_∗_).Unfortunately,the</sup> Census EB estimator is not implemented in the sae R package. One possible approach for applying this software when survey units cannot be identified in the census would be to append the survey data to the available census data, which would lead to a different approximation of the EB estimator. A drawback of this approach is that, in this case, the total size of the area is then _Nc_ + _nc ≥ Nc_ .

The EB estimator _τ_ ˆ _c_<sup>_EB_</sup> approximates the so called “best” predictor of _τc_ , which is the optimal predictor in the sense of minimizing the MSE under the model. This MSE of an estimator _τ_ ˆ _c_ is defined as the expectation of the squared prediction error under the considered model, MSE(ˆ _τc_ ) = _E_ (ˆ _τc − τc_ )<sup>2</sup> . ˆ The best predictor is given by the conditional expectation _τc_<sup>_B_</sup> = _E_ ( _τc|yc,s_ ; _θ_ ), where the expectation is with respect to the conditional distribution of the non-sampled data _yc,r_ given the sample data _yc,s_ , determined by the nested error model. Under normality of the random terms in the model, the conditional distribution is also normal. In step 2 of the above procedure, welfare values for non-sampled households are generated from that conditional distribution, but with estimated model parameters. Hence, the above ˆ MC simulation procedure actually approximates the EB estimator _τc_<sup>_EB_</sup> = _E_ ( _τc|yc,s_ ; _θ_<sup>ˆ</sup> ), which is the best predictor with model parameters replaced by the corresponding estimates based on the original sample survey data _θ_<sup>ˆ</sup> 0 = ( _β_<sup>ˆ</sup> 0 _,_ ˆ _ση_<sup>2</sup> 0<sup>_,_ˆ</sup><sup>_σ_</sup> _e_<sup>2</sup> 0<sup>).Notethat,unliketheELLbootstrapprocedure,heretheseestimates</sup> are kept constant across simulations because there is a unique DGP. Hence, this simulation procedure delivers a simple MC approximation for the EB predictor ˆ _τc_<sup>_EB_</sup> = _E_ ( _τc|yc,s_ ; _θ_<sup>ˆ</sup> ) of _τc_ , where the expectation is replaced by the corresponding MC average.

If the target parameter is just the area mean _τc_ = _Nc_<sup>_−_1</sup> � _Nh_ =1 _d_<sup>_ych_,andthereisnolog-transformation</sup> of welfare, then the best predictor reduces to _E_ ( _τc|yc,s_ ; _θ_ ) = _x_ ̄ _cβ_ + _γc_ ( ̄ _yc,s − x_ ̄ _c,sβ_ ), which corrects the regression synthetic estimator _x_ ̄ _cβ_ by accounting for the area effect using the survey data. Moreover, the size of the correction depends on _γc_ , which measures the share of between-area heterogeneity _ση_<sup>2from the</sup> total variance in the area, _ση_<sup>2+</sup><sup>_σ_</sup> _e_<sup>2</sup><sup>_/nc_.Thus, the correction is stronger in the case of highly heterogeneous</sup> areas and it is weak otherwise, becoming null only when all the existing area heterogeneity is completely explained by the available auxiliary variables. In that case, the EB estimator reduces to the “theoretical” ELL estimator _E_ ( _τc_ ; _θ_ ) = _x_ ̄ _cβ_ . Hence, the EB procedure makes more efficient use of the survey data for areas that are sampled and, for those areas, it does not rely so strongly on the assumed linear regression (which could be misspecified).

Concerning error measures of the estimators, the ELL and EB approaches consider different error measures. In the EB approach of MR (2010), the noise is measured by the actual MSE of the EB estimator 8
under the nested error model, MSE(ˆ _τc_<sup>_EB_</sup> ) = _E_ (ˆ _τc_<sup>_EB_</sup> _− τc_ )<sup>2</sup> . Under the model-based framework, the target indicators _τc_ are regarded as random quantities (they are generated from the model) and hence incorporate uncertainty as well. Even if an estimator (or predictor) _τ_ ˆ _c_ is unbiased with respect to the model in the sense that the expectation of its prediction error is zero _E_ (ˆ _τc − τc_ ) = 0, it would still leave the variance of the prediction error var(ˆ _τc −τc_ ), which is not exactly equal to the variance of the predictor var(ˆ _τc_ ) because of the randomness of the target indicator _τc_ .

The MSE of the EB estimator is estimated using a parametric bootstrap procedure for finite populations introduced by González-Manteiga et al. (2008), which is computationally more intensive than the ELL bootstrap approach. This is because, in every bootstrap replicate, all the steps for obtaining the EB estimator are reproduced, including model fitting and the MC procedure for approximation of the conditional expectation defining the EB estimator. This means that the number of simulated census vectors is the product of the MC simulations and the number of bootstrap replicates, except when the target indicator has an explicit EB estimator (such as the poverty rate). In this latter case, the MC simulation procedure can be replaced by an explicit formula calculated without simulation, and then the number of simulated censuses will be only the number of bootstrap replicates.

The steps for the parametric bootstrap estimation of the MSE are as follows (MR, 2015):

1. Using the survey data, fit model (1) using a method providing consistent estimators. This yields the set of parameter estimates from the observed sample:

In the implemented sae R package, the REML method is used.

2. Use the estimates in _θ_<sup>ˆ</sup> 0 to simulate census welfares<sup>11</sup> from the fitted model as follows: where the area effects _ηc_<sup>_∗_aregeneratedas</sup> and the household-specific errors are generated as 3. From the simulated census vector _yc_<sup>_∗_,wecalculatethetruevalueoftheindicatorofinterestfor</sup> each area _c_ : _τc_<sup>_∗_=</sup><sup>_f_(</sup><sup>_y_</sup> _c_<sup>_∗_)</sup><sup>_,_</sup> where _f_ ( _yc_<sup>_∗_)canbeanyindicatorfunctionsuchastheFGTindicators.</sup> 4. Since the survey is regarded as a subset of the census, the survey sample _sc_ is now extracted and, using the corresponding sample welfares _yc,s_<sup>_∗_(newlygeneratedinstep2),onefitsthemodel(1).</sup> This yields bootstrap estimates of the model parameters,

- ˆ ˆ _θ_<sup>_∗_</sup> = � _β_<sup>_∗_</sup> _,_ ˆ _ση_<sup>2</sup><sup>_∗,_ˆ</sup><sup>_σ_</sup> _e_<sup>2</sup><sup>_∗_</sup> � _._

5. With the bootstrap vectors of sample welfare _yc,s_<sup>_∗_,</sup><sup>_c_=1</sup><sup>_, . . . , C_,obtaintheEBestimators</sup><sup>_τ_ˆ</sup><sup>_EB_</sup> _c_<sup>_∗_</sup> , using MC simulation if needed. The estimation procedure is exactly the same EB procedure based on the original sample, but using the bootstrap sample data _yc,s_<sup>_∗_,</sup><sup>_c_= 1</sup><sup>_, . . . , C_and the corresponding</sup> bootstrap model parameter estimates _θ_<sup>ˆ</sup><sup>_∗_</sup> from step 4.

> 11Note that here welfares are generated for all households in the census, sampled and non-sampled.

9
( ) ~~-~~ y( ) <!-- Start of picture text --> ( =) =<br>(E— E —)EE E F<br>f s- )= —- ( E-) E-<br>{ = (-) |<br><!-- End of picture text -->
As mentioned before, for sampled clusters, EB prediction makes use of the estimated cluster effects from the survey data to improve the point estimates of the poverty indicators and their errors. A crucial assumption under EB prediction is that model errors _ech_ and _ηc_ are normally distributed. This yields ˆ normality for _ηc_ given _ec,s_ , which leads to the proposed predicted cluster effect _ηc_ .

This EB addition implemented in the PovMap software uses an extended version of Henderson’s method III (Henderson, 1953) to obtain estimators _σ_ ˆ _η_<sup>2and</sup><sup>_σ_ˆ</sup> _e_<sup>2ofthevariancecomponents</sup><sup>_σ_</sup> _η_<sup>2and</sup><sup>_σ_</sup> _e_<sup>2thatin-</sup> corporate the sampling weights.<sup>12</sup> In step 2 of section 2 for the ELL, _ση_<sup>2</sup><sup>_∗_</sup> is drawn from a _Gamma_ distribution, however to avoid this assumption and because var [ˆ _η_ c] is unkonwn,<sup>13</sup> it is necessary to rely on bootstrap re-sampling to obtain the noise of the point estimates in similar fashion to ELL.<sup>14</sup> Even if the estimator is based on the EB method of MR (2010), the bootstrap procedure used to generate the census data is completely different. This new bootstrap algorithm tries to avoid the distributional assumptions by generating in the first step samples of clusters from the survey data set. The full procedure for obtaining the estimator of _τc_ , called here clustered bootstrap EB (CB-EB), is as follows:

1. Take a bootstrap sample of clusters (PSUs) with replacement from the survey data.<sup>15</sup>

2. Fit model (1) to the bootstrapped survey data from step 1. This yields the set of parameter estimates from the bootstrap sampled data ( _β_<sup>ˆ</sup><sup>_∗_</sup> _,_ ˆ _ση_<sup>2</sup><sup>_∗,_ˆ</sup><sup>_σ_</sup> _e_<sup>2</sup><sup>_∗_),alongwithpredictedeffectsforthe</sup> clusters that were sampled in step 1, _η_ ˆ _c_<sup>_∗_,andtheircorrespondingestimatedvariance</sup> var<sup>�</sup> _∗_ [ˆ _ηc_<sup>_∗_].In</sup> the implemented version, the model is fit by FGLS as specified in Nguyen et al. (2018).

3. Use the model parameter estimates from step 2 to simulate welfares _ych_<sup>_∗_inthecensusas16</sup> where, for a cluster _c_ that is sampled in step 1, the cluster effect is generated as where _η_ ˆ _c_<sup>_∗_and</sup> var<sup>�</sup> _∗_ [ˆ _ηc_<sup>_∗_]arethebootstrapversionsof</sup><sup>_η_ˆ</sup><sup>_c_anditscorrespondingvariance,givenin(2)</sup> and (3) respectively. For a cluster that is not sampled in step 1, generate Under homoscedasticity, the household specific errors are generated as and, in the case of heteroscedasticity, as Construct the vector _yc_<sup>_∗_= (</sup><sup>_yc_1</sup><sup>_, . . . , ycN_</sup> _c_<sup>)</sup><sup>_T_of simulated welfare for every household within location</sup> _c_ of size _Nc_ , for all the locations _c_ = 1 _, . . . , C_ in the census.

> 12Interested readers should refer to Van der Weide (2014) and/or Nguyen et al. (2018) for an in-depth look at how these are obtained under ELL and Henderson’s method III.

> 13In the traditional ELL method of Section 2, _σ_ ˆ _η_ 2<sup>isassumedtofollowaGammadistribution,whichdoesnotholdin</sup> this case.

> 14Actually, Van der Weide (2014) does not offer a method for the estimation of the standard errors, but the approach described below was the one implemented on PovMap and consequently also in the Stata sae package from Nguyen et al. (2018).

> 15Note that not all clusters are expected to appear

> 16Note that, in each bootstrap replicate, the census is generated from a different model.

11
4. With the simulated welfare vector _yc_<sup>_∗_,calculatetheindicatorofinterestas</sup><sup>_τ ∗_</sup> _c_<sup>=</sup><sup>_f_(</sup><sup>_y_</sup> _c_<sup>_∗_)suchasthe</sup> FGT indicator of order _α ≥_ 0, which is calculated as follows: 5. Repeat steps 1 to 3 _M_ times. Even if traditionally _M_ = 100, a larger number of replicates is recommended. Let _τc_<sup>_∗_(</sup><sup>_m_)</sup> be the indicator obtained in mth replicate, _m_ = 1 _, . . . , M_ . The CB-EB estimator _τ_ ˆ _c_<sup>_CB−EB_</sup> is then obtained as the average across the _M_ bootstrap replicates, and the variance of the estimator is approximated as Unlike the traditional ELL approach, this procedure requires to match the location codes in the survey and the census. However, in the application of the traditional ELL approach, the authors recommended the inclusion of contextual variables at the location level and thus the linking between the two data sources was still necessary.

As in the previous computational procedures, the final estimator of the poverty indicator for a location is obtained as an average across bootstrap replicates. By the Monte Carlo principle, this average is approximating an expectation. However, here it is not clear with respect to which DGP is the expectation taken, since the bootstrap procedure involves generation of several measures: In step 1, the sample of clusters varies, and this would define an expectation with respect to the sampling design, but considering only the first stage of the design. In step 2, model parameters are newly generated in each simulation. Finally, in step 3, location effects are generated from their conditional distribution given the sample residuals (determined by the nested error model under normality), and household-specific errors are generated from their distribution under the assumed nested error model with normality.

Even if the bootstrap procedure is not the same, standard errors are obtained similarly as in the traditional ELL approach and are thus different from MSEs. Note that the traditional ELL approach was initially based on the multiple imputation literature. According to Rubin (1996, and noted in StataCorp, 2019), the objective of multiple imputation is not to re-create the missing data as closely as possible, but to handle it in a manner that allows for statistical inference.

# **5 Comparison of CB-EB, traditional ELL and EB methods**

Because EB prediction uses the survey data to estimate the location effect, its benefits are evident only in locations present both in the census and the survey. In the traditional ELL approach, the location effect was originally at the cluster level and, in most surveys from developing countries where the World Bank focuses its efforts, the sampled clusters represent a small percentage of all the clusters in the country. This may also be the case for locations above clusters. For example, in a recently completed exercise in Moldova, the survey only contained 129 comunas out of a total of 901. Consequently, only a small share of these comunas benefit from EB prediction (Corral and Cojocaru, 2019). Nevertheless, differences also arise due to the computational procedures used to obtain the point estimates and their corresponding measures of noise.

12
MR (2010) presented evidence that their EB approach is superior to ELL in terms of a smaller MSE. This finding has met some criticism, mostly because the simulation experiment in MR (2010) was conducted under scenarios where ELL is seldom applied. First, the population size in MR (2010) was 20,000 uniformly spread across 80 areas. Second, all the areas were sampled and the sample within each area represented 20 percent of the population, something hardly seen in real-world applications. ELL (2002) illustrates how the noise of the estimator falls as the size of the population increases, and advises against estimating below populations of 100 households. In practice, the level of disaggregation of ELL estimates depends on the model quality, and even with an _R_<sup>2</sup> not smaller than 0.6, it is seldom advised to go below 1,000 households (Elbers et al., 2007).

Another point of departure is that the model considered in the simulation experiments of MR (2010) was really poor, yielding an adjusted _R_<sup>2</sup> of less than 0.01. Under the usual applications of the ELL approach, the number of auxiliary variables is much greater, reaching most of the times an explanatory power, as measured by adjusted _R_<sup>2</sup> , of over 0.5. Moreover, to improve precision, ELL (2002; 2003) also advocated for the use of contextual area-level variables<sup>17</sup> on the right-hand side, since these explain (and thus minimize) the variation across locations. These were not used in the simulation experiment conducted in MR (2010). In such a setup, the much noisier ELL estimates obtained in their simulation experiments are not surprising.

A final important difference is the fact that the error measures of estimators and the bootstrap procedures for obtaining them are considerably different in the three procedures. Currently, the error measures of the traditional ELL and of the more recent CB-EB procedure of Van der Weide (2014) are obtained under a very similar framework to that of multiple imputation. As an error measure, these consider the standard deviation of all the simulated point estimates of the indicators in the _M_ bootstrap replicates, and so every simulated indicator is compared to the average of the simulated indicators. Moreover, in the bootstrap approaches used for that average of the simulated indicators, the initial model parameter estimates _β_<sup>ˆ</sup> 0, _σ_ ˆ _η_<sup>2</sup> 0<sup>and</sup><sup>_σ_ˆ</sup> _e_<sup>2</sup> 0<sup>obtainedusingtheoriginalsurveysamplearenotthoseusedtogeneratethe</sup> welfare vectors in the population. In fact, in each bootstrap replicate, model parameters � _β, ση_<sup>2</sup><sup>_, σ_</sup> _e_<sup>2</sup> � are simulated from the estimated distribution of the initial estimators (or an approximation to it). In contrast, in MR’s (2010) parametric bootstrap procedure, a simple MC simulation procedure is used to approximate the actual MSE under the nested error model with the average across bootstrap simulations of the squared prediction errors. In this bootstrap procedure, the parameter estimates used to generate the bootstrap censuses of welfare are kept fixed to the original survey estimates. Consistency of these initial survey estimators to their corresponding true values as the total sample size _n_ tends to infinity ensures that the bootstrap MSE approximates the true MSE (under the model with the true values of the parameters).

# **6 Extended Census EB estimators**

The EB estimator in MR (2010) requires linking the survey and census households, which may not be possible in real applications. In fact, the survey sample is likely not a subset of the available census. Hence, here we consider the Census EB (CEB) estimator, which avoids this step. In fact, in most cases, the number of sample households for a given area is much smaller than the number of census households and, in such cases, the CEB estimator is expected to perform very much like the original EB.

CEB small area estimators are obtained with a similar MC simulation approach as the one used by MR (2010) and described in Section 3, but simulating the vectors of welfare for all the census households

> 17These may come from the census or administrative records (ELL, 2003 p356).

13
( ) ( ~~=) —~~ = (5 5 ~~—)~~ { = ~~(—)~~ } ( ) ( ) ~~US~~ ( ~~—)~~
4. Repeat steps 1 to 3 a large number of times _M_ . Let _τc_<sup>_∗_(</sup><sup>_m_)</sup> be indicator obtained in mth replicate, for _m_ = 1 _, . . . , M_ . The extended Census EB estimate is just the average of the _M_ indicators, In contrast with the traditional ELL and CB-EB approaches, in the above MC procedure, the model parameter estimates _β_<sup>ˆ</sup> 0, _σ_ ˆ _η_<sup>2</sup> 0<sup>and ˆ</sup><sup>_σ_</sup> _e_<sup>2</sup> 0<sup>, as well as</sup><sup>_η_ˆ</sup><sup>_c_0and</sup> var [ˆ<sup>�</sup> _ηc_ 0], are kept fixed across the whole procedure; that is, there is a single DGP.

As a measure of noise of the extended CEB estimators, here the actual MSE under the nested error model is considered. To estimate this MSE, we propose to apply a parametric bootstrap procedure similar to the one in MR (2010), but adapted to the case where the survey is not necessarily a subset of the census. This procedure is also described in Molina (2019) for the CEB predictor, but here we apply it to the extended CEB predictor that includes heteroscedasticity and sampling weights:

1. Fit model (1) to the survey data. This yields the set of parameter estimates:

In the implemented version, the model is fit either via FGLS and decomposing residuals as in the traditional ELL fitting approach, or by H3.

2. Using the estimates obtained in step 1 as true values of the model parameters, simulate a vector of census welfare _yc_<sup>_∗_= (</sup><sup>_yc_1</sup><sup>_, . . . , ycN_</sup> _c_<sup>)</sup><sup>_T_asfollows:</sup> ln( _ych_<sup>_∗_) =</sup><sup>_xchβ_ˆ0+</sup><sup>_η_</sup> _c_<sup>_∗_+</sup><sup>_e∗_</sup> _ch_<sup>_,h_= 1</sup><sup>_, . . . , Nc,_</sup> where the location effect is generated as In the homoscedastic case, household-specific errors are generated as and, in the case of heteroscedasticity, as 3. With the simulated census of welfare _yc_<sup>_∗_=(</sup><sup>_yc_1</sup><sup>_, . . . , ycN_</sup> _c_<sup>)</sup><sup>_T_foreacharea</sup><sup>_c_=1</sup><sup>_, . . . , C_,produce</sup> indicators of interest: _τc_<sup>_∗_=</sup><sup>_f_(</sup><sup>_y_</sup> _c_<sup>_∗_)</sup><sup>_._</sup> where _f_ ( _yc_<sup>_∗_)canbeanyindicatorfunction,suchasoneoftheFGTpovertyindicators.</sup> 4. Use the model parameter estimates obtained in step 1 to obtain new sample welfares, _ych,s_<sup>_∗_,for</sup> every location _c_ as follows: ln( _ych,s_<sup>_∗_) =</sup><sup>_xch,sβ_ˆ0+</sup><sup>_η_</sup> _c_<sup>_∗_+</sup><sup>_e∗_</sup> _ch,s_<sup>_._</sup>

- (a) The estimates _β_<sup>ˆ</sup> 0 come from step 1.

- (b) _ηc_<sup>_∗_isthesameonegeneratedinstep2.Specifically,alllocationspresentinthesurveyare</sup> matched to the census locations and the same value of _ηc_<sup>_∗_that was simulated for that location</sup> in step 2 is applied to the survey households within the same location.

15
- (c) The household-specific errors are simulated as

in the homoscedastic case. When there is heteroscedasticity, they are simulated as Unlike the original parametric bootstrap procedure in MR (2010) described in Section 3, here the sample errors _e_<sup>_∗_</sup> _ch,s_<sup>are not a subset of the census errors simulated in step 2, although they</sup> are generated from exactly the same model (with the same variance).

5. Fit the model (1) to the newly simulated survey data from step 4. This yields a bootstrap vector of estimated model parameters: 6. Obtain the bootstrap extended CEB estimator _τ_ ˆ _c,b_<sup>_CEB∗_</sup> through MC simulation using the bootstrap parameter estimates _θ_<sup>ˆ</sup><sup>_∗_</sup> from step 5, by the approach described in the previous section.

7. Repeat steps 2 to 6 a sufficiently large number of times, _B_ . Let _τc_<sup>_∗_(</sup><sup>_b_)</sup> be the true value and _τ_ ˆ _c,b_<sup>_CEB∗_(</sup><sup>_b_)</sup> be the CEB estimator obtained in the bth replicate of the bootstrap procedure. A parametric bootstrap estimator of the MSE is given by:

# **7 Simulation experiments for comparison of methods**

## **7.1 Simulation experiment with poor model**

Here we compare the procedures described in the previous sections for the estimation of poverty rates and gaps (FGT indicators with _α_ = 0 _,_ 1 respectively) and their corresponding error measures. To this aim, we perform simulation experiments under the same setup as in MR (2010), but extended to consider also more realistic scenarios.

Thus, like in MR (2010), a census data set of _N_ = 20 _,_ 000 observations is created, where all observations are uniformly spread among _C_ = 80 areas, labeled from 1 to 80. This means that every area consists of _iid Nc_ = 250 observations. Location effects are simulated as _ηc ∼ N_ �0 _,_ 0 _._ 15<sup>2�</sup> ; note that every observation within a given area will have the same simulated effect. Then, values of two right-hand side binary variables are simulated. The first one, _x_ 1, takes the value 1 if a generated random uniform value between 0 and 1 is less than or equal to 0 _._ 3 + 0 _._ 5 80<sup>_<u>c</u>_.Thismeansthatobservationsinareaswithahigherlabel</sup> are more likely to get value 1. The next one, _x_ 2, is not tied to the area’s label. This variable takes the value 1 if a simulated random uniform value between 0 and 1 is less than or equal to 0.2. The census welfare vectors _yc_ = ( _yc,_ 1 _, . . . , yc,Nc_ )<sup>_T_</sup> for each area _c_ = 1 _, . . . , C_ , are then created as follows: ln( _ych_ ) = 3 + 0 _._ 03 _x_ 1 _,ch −_ 0 _._ 04 _x_ 2 _,ch_ + _ηc_ + _ech,_ _iid_ where household-level errors are generated under the homoscedastic setup, as _ech ∼ N_ �0 _,_ 0 _._ 5<sup>2�</sup> . The poverty line is fixed at _z_ = 12, which is roughly 60 percent of the median welfare of a generated population.

16
From the created “census” in each of the areas, 20 percent of the observations are sampled using simple random sampling without replacement;<sup>18</sup> this yields our “survey” data. Thus, in this experiment, survey weights are all equal. This generation process is repeated _L_ = 10 _,_ 000 times. In each simulation, the following quantities are computed for the poverty rates and gaps in each area:

1. True poverty indicators _τc_ , using the “census”.

2. Direct estimators _τ_ ˆ _c_<sup>_DIR_</sup> using the “survey”, defined as the sample versions of _τc_ .

3. Original EB estimators ˆ _τc_<sup>_EB_</sup> by MR (2010) with REML fitting as implemented in the sae R package, with _M_ = 50 MC replicates.

4. Traditional ELL estimators _τ_ ˆ _c_<sup>_ELL_</sup> and their estimated standard errors, without location means and also including the location means of the considered auxiliary variables, where _M_ = 50.

5. Census EB (CEB) estimators ˆ _τc_<sup>_CEB_</sup> , obtained as in the MC procedure of MR (2010), but generating full census vectors from the conditional distribution and without appending the survey welfares, using REML fitting and with _M_ = 50.

6. Extended CEB estimators using the MC and parametric bootstrap methods described in Section 6, using either H3 or ELL fitting methods with _M_ = 50. Since in this simulation experiment model errors are homoscedastic and sampling weights are constant, here the extended CEB estimators only differ in the above CEB ones in the model fitting method.

7. Estimators based on the CB-EB approach from Section 4, _τ_ ˆ _c_<sup>_CB−EB_</sup> using either H3 or ELL fitting methods.<sup>19</sup> Since clusters are originally supposed to be nested within the areas and the clustered bootstrap method is supposed to sample clusters (PSUs) instead of areas in step 1, we computed additional CB-EB estimators, where every area is divided into 25 PSUs (with 10 households each) and selecting 5 PSUs within every area (instead of areas). Under this scenario _M_ = 50 as well.

Model bias and MSE are approximated empirically as in MR (2010), as the averages across the _L_ = 10 _,_ 000 simulations of the prediction errors _τ_ ˆ _c_<sup>_j−τc_andofthesquaredpredictionerrorsrespectively,where</sup><sup>_j_</sup> stands for one of the methods: DIR, ELL, CEB or CB-EB.<sup>20</sup> Before getting into the results of the full simulation experiment, the methods are initially compared using just a single generated population along with its corresponding sample. For this one population, the different methods are run by setting _M_ = 1000 to get point estimates and _B_ = 1000 for the MSE.<sup>21</sup> For expediency, results that do not conform to the main message of the paper are relegated to the appendix. However, from the figures in the appendix, we can draw the following conclusions: 1) The differences in the EB and Census EB point estimates are negligible even if the area sampling fractions _fc_ = _nc/Nc_ are not so small in this experiment (20 percent), see Figure 19. Although differences appear in their estimated MSEs (see Figure 20), these differences become closer to zero as the sampling fractions _fc_ decrease. 2) Census EB estimates using H3 method as implemented in the sae Stata package are practically identical to the Census EB ones with REML fitting (Figure 21). Similarly, when comparing the Census EB estimators obtained with the proposed MC procedure using either H3 or ELL estimation methods, we can see that they are also aligned (Figure 22). This indicates that the actual method used

> 18The same units are sampled in every simulation and the values of _x_ 1 and _x_ 2 for all the census units are also kept fixed; this means that the _x_ 1 and _x_ 2 values for the sample units are always the same across simulations.

> 19For a detailed description of the difference between the methods readers should refer to Nguyen et al. (2018).

> 20 _E_<sup>�</sup> _τ_ ˆ _c_<sup>_j−τ_</sup> _c_ � for the bias, and _E_<sup>�</sup> _τ_ ˆ _c_<sup>_j−τ_</sup> _c_ �2 for the MSE.

> 21For ELL from section 2 and H3 CBEB from section 4 since the process relies on a single computational algorithm _M_ = 1 _,_ 000

17
to fit the model is not an important factor. In fact, in real applications, the overall survey sample is typically very large so any fitting method, as long as it provides consistent estimators, is expected to deliver estimates that will be very close to the corresponding true values.

The above conclusions support the use of the Census EB, using either REML, H3 or ELL estimation methods, instead of the original EB procedure in real-world applications, where it is seldom possible to link the survey and census data. Moreover, since the large sampling fractions of this experiment favor the original EB method, in the following we show only results for the Census EB instead of the original EB, where welfares are generated in the MC procedure for all the census units and the survey welfares are not appended.

Figure 1 compares, for the poverty rates, Census EB estimators obtained from the proposed MC procedure of Section 6 using H3 estimation method (labeled “H3-CensusEB”) with the analogues obtained using the CB-EB procedure of Section 4 (labeled “H3-CBEB (old bootstrap)”). We include also the results from the CB-EB method with selection of PSUs within the areas instead of selection of areas in step 1 (labeled “H3-CBEB-old with PSU”). Note that the CB-EB estimates presented in Section 4 differ considerably from the true poverty rates whereas the Census EB ones track approximately the true values. In fact, the sum of absolute differences of the CB-EB estimates to the true values is almost three times that of the Census EB estimates.

The main reason for this difference of behavior, even if both estimates are supposed to implement EB prediction is that, under the CB-EB approach in Section 4 with areas equal to clusters, a sample of areas is drawn in each simulation. This leads to a given area benefiting from EB prediction of the area effect only in a portion of the simulations, since it is likely that an area is not selected in every single bootstrap sample. This leads to CB-EB point estimates that are far off from the Census EB estimates obtained with the proposed MC procedure of Section 6. Another reason might be that the parameters used to simulate the census vectors in CB-EB approach are not kept fixed across simulations, making difficult the convergence to the conditional expectation defining the estimator. When using the CB-EB approach with PSUs, there is still a clear difference with the Census EB estimates obtained with the new bootstrap procedure of Section 6.

Turning now to the traditional ELL approach, Figure 2 shows ELL estimates for the poverty rates, based on the model with only _x_ 1 and _x_ 2 (labeled “ELL”) and the same, but including the area means of _x_ 1 and _x_ 2 in the model (labeled “ELL with context”), together with true poverty rates and Census EB estimates obtained using the original MC procedure of MR (2010), without appending the survey data. In this plot, one finds that the traditional ELL estimates face the same troubles as the CB-EB ones. This is in line with what MR (2010) find in their simulations. Given the present simulation set-up with a poor model fit, ELL performs much worse than MR’s Census EB method in terms of point estimates. Also note the rather flat nature of ELL, which is mostly due to the model providing very little information and, given the limited explanatory power of the correlates, the ELL estimates fall close to the average poverty rate of 16 for all the areas.<sup>22</sup> What is somewhat concerning is that the addition of contextual variables to the model for ELL, in the form of the area means, does little to improve the point estimates. Thus, even if the survey data are not preserved in the Census EB estimation method, the benefit of EB is quite considerable and caution should be taken when doing out of sample predictions in case of little explanatory power being brought in by the auxiliary variables.

The complete simulation experiment with _L_ = 10 _,_ 000 simulated populations are now discussed. Results in this case focus on the poverty gap, although the observed tendencies are the same for the poverty rates. Figures 3 and 4 show respectively the empirical bias and empirical MSE across simulated populations

> 22The ELL estimates are not really flat if sorted from smallest to largest.

18
<!-- Start of picture text --> as<br>&<br>t<br>of. ©. © tet o*a oe ee> *: eue + tj 0e “+2uaeara~*<br>oo] soos © 3,,oteg "iat + +<br>Si TM 0te cee«© tetggrsergee. «® eo tye rus: ee 4 “= 6a taae++t +7+ 4<br>—-le ee,+7 eot wae”+ + cp + + +<br>+gaathafaaseasl<br>YY<br>oS<br>0 20 40 60 80<br>Sorted areas by true FGT0O<br>+ FGTO H3-CensusEB<br>« FGTO H3-CBEB (old bootstrap)<br>® FGTO H3-CBEB-old with PSU<br>4 True value from simulated Census<br>+<br>+ ,af<br>9 . Saat<br>. Wo<br>+ + aa* + st<br>Oo + kagh + *<br>= + kaa + *<br>wo) “ + 4h + A +t + +<br>u. seaegrgteattonesttiojeetisotetaretteenatennbdntettoerirerstoseitestectectestite<br>+ + et +<br>.<br>- +<br>: + +4+ +4 wae + 7<br>+a a a!lba areal”<br>, YY<br>oS<br>0 20 40 60 80<br>Sorted areas by "true" FGTO<br>+ FGTO M&R (2010)<br>« FGTO ELL with context<br>* FGTO ELL<br>4 True value from simulated Census<br><!-- End of picture text -->
<!-- Start of picture text --> t+<br>POLDELOETODGSEEFAET STS EOTHRESETLOF ESF OSFESERUTOE TOSSESOODHOETEFONSEOEESORLEFONES<br>oO<br>Ss<br>oO<br>xa<br>ha<br>oO<br>uwn -<br>&<br>iva) x x a<br>Oe saeaddeehdey cis iret SSEIAAL 1 chee SanaMnd MeL AES Sekar ad tM Dat oa<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB ¢ H3-CBEB (old bootstrap)<br>« ELL = H3-CBEB (old bootstrap with PSU)<br>« Direct<br><!-- End of picture text --> <!-- Start of picture text --> re) srtarnenbetscgsestegesreseguetzeeteet gorge gttenstUEssoonenste* tes tugbeeul Shy<br>Ss<br>So<br>Se<br>x<br>ha<br>Oo<br>Ww<br>a”S 903g 9 6 96 50 96 “Xxxa Ge 36 I I RKDe ae Hy FH56 IE FHM3 9 1 a 96 x gg HHI xFH ae 5 I ae 5g CHEE GTxx<br>mabe ne bbs sb ash dass bbbbba ys ghhagebaghhbag<br>oO<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB ¢ H3-CBEB (old bootstrap)<br>« ELL = H3-CBEB (old bootstrap with PSU)<br>« Direct<br><!-- End of picture text -->
of Census EB from Section 6 using H3 estimation method (labeled “H3-CensusEB”), traditional ELL estimators (labeled “ELL”), estimators based on the CB-EB approach with H3 method (labeled “H3CBEB (old bootstrap)”), the same with PSUs within areas selected in step 1 (labeled “H3-CBEB (old bootstrap with PSU)”) and direct estimators (labeled “Direct”). These full results are quite sobering. While the bias for the traditional ELL estimators is not remarkable, the bias of the estimators obtained by the CB-EB method from Section 4 is substantial, showing upward bias for all areas. In contrast, the bias of the Census-EB estimators obtained from Section 6 seems negligible, ranging between -0.025 and 0.027, similar to the bias range of the original EB estimators from MR (2010).<sup>23</sup> This figure shows a considerable bias reduction of the proposed MC procedure for obtaining the Census EB estimators with respect to the CB-EB method.

In terms of MSE, results are aligned to those in MR (2010), see Figure 4. The traditional ELL yields an empirical MSE above the one of the direct estimator, which uses no model. Even if ELL estimators do not show a large bias when averaging across all the simulated populations (the ELL bootstrap procedure is supposed to approximate the marginal mean _E_ ( _τc_ ; _θ_ ), which is exactly unbiased), as shown in Figure 2, it does not capture the area effect of the true poverty rates for a single population. Note that, under the model-based setup, welfares are generated from the model and in each simulation, new location effects are generated. This means that the true poverty rates vary for one population to another. In fact, a given area might have a positive location effect in one generated population and a negative effect in another. Since the location effects have an expected value of zero, the ELL method is tracking the average zero effect, which is the same for all the areas, but it does not capture the area effects in a given population. What is even more concerning is that the CB-EB method applied in Nguyen et al. (2018) and PovMap described in Section 4 yields an MSE that is more than 3 times greater that of the traditional ELL. This happens because a large amount of the MSE is the squared bias, which has been shown to be substantial for the CB-EB estimator. The proposed MC method for Census EB estimation presented here represents a massive improvement over the traditional ELL and the CB-EB approaches under the setup of this simulation experiment based on a poor model. The next section describes results when increasing the prediction power of the model.

# **7.2 Simulation experiment with improved model**

This experiment addresses one of the concerns of the previous simulation experiment: the poor explanatory power of the model. Hence, in this section a more informed model is considered, yielding an adjusted _R_<sup>2</sup> of about 0.42, which is much closer to that of real-world applications of ELL. The model includes six covariates. The first two, _x_ 1 and _x_ 2, are generated exactly as in the previous section. Four additional covariates are included, with values generated as follows:

1. _x_ 3 is a binary variable, taking value 1 when a random uniform number between 0 and 1 is less than or equal to 0 _._ 1 + 0 _._ 2 80<sup>_<u>c</u>_.</sup>

2. _x_ 4 is a binary variable, taking value 1 when a random uniform value between 0 and 1 is less than or equal to 0 _._ 5 + 0 _._ 3 80<sup>_<u>c</u>_.</sup> 3. _x_ 5 is a discrete variable, simulated as the rounded integer value of the maximum value between 1 and a random Poisson variable with mean _λ_ = 3 �1 _−_ 0 _._ 1 80<sup>_<u>c</u>_</sup> �.

4. _x_ 6 is a binary variable, taking value 1 when a random uniform value between 0 and 1 is less than or equal to 0.4. Note that the values of _x_ 6 are not related to the area’s label.

> 23Results for the Census EB estimators obtained from the original procedure of MR’s (2010) with REML estimation are not shown since they are aligned to the Census EB estimators from Section 6.

21
The welfare vector for each area is created from the model with these covariates as follows ln _ych_ = 3 + 0 _._ 09 _x_ 1 _,ch −_ 0 _._ 04 _x_ 2 _,ch −_ 0 _._ 09 _x_ 3 _,ch_ + 0 _._ 4 _x_ 4 _,ch −_ 0 _._ 25 _x_ 5 _,ch_ + 0 _._ 1 _x_ 6 _,ch_ + _ηc_ + _ech,_ _iid iid_ where _ech ∼ N_ �0 _,_ 0 _._ 5<sup>2�</sup> and _ηc ∼ N_ �0 _,_ 0 _._ 15<sup>2�</sup> . The poverty line in this scenario is fixed at _z_ = 10 _._ 2. All other steps are the same as in the previous simulation.<sup>24</sup> Even if the performance of CB-EB estimators improves when considering a better model, these are still clearly worse than the Census EB estimators obtained with the proposed MC approach of Section 6, see Figure 5. This conclusion remains true even after decreasing the location effect, by simulating the _iid_ populations with _ηc ∼ N_ �0 _,_ 0 _._ 07<sup>2�</sup> .<sup>25</sup> Similarly, despite the improved explanatory power of the model, Figure 6 still portrays a rather flat ELL hovering around the national average poverty rate. Under a single simulated population from the improved model, the traditional ELL method still performs poorly when compared with the Census EB according to MR (2010).

Results from the full simulation with _L_ = 10 _,_ 000 populations for the poverty gap also resemble those from the previous section, see Figures 7 and 8 showing empirical bias and empirical MSE respectively.<sup>26</sup> The CB-EB method implemented as described in Nguyen et al. (2018), which replicates the methods of PovMap, still presents considerable bias in this new simulation exercise. In comparison, the proposed Census EB procedure seems to be nearly unbiased, although the obtained empirical biases actually range between -0.04 and 0.05. Moreover, even under a better regression model, the MSE of the proposed Census EB estimator is still much smaller. What is concerning is that the MSEs of the direct estimators are smaller than those of the traditional ELL estimators and of the CB-EB estimators.

Another criticism of the EB approach of MR (2010), which extends to the Census EB, is its dependence on normality. In order to study the sensitivity of the procedures to the normality assumption, an additional simulation experiment is run under the same setup of this section, but generating the household-specific errors from a Student’s _t_ distribution with 5 degrees of freedom and scaled by 0.5. This distribution is symmetric but has heavier tails than the normal distribution, which may represent the existence of outlying welfares in the data, often encountered in real applications.

Empirical bias and MSE across the _L_ = 10 _,_ 000 simulations are shown in Figures 9 and 10 respectively, for the proposed Census EB, traditional ELL and direct estimators.<sup>27</sup> Note that the ELL method takes household-specific residuals from their empirical distribution, so its performance is not supposed to rely so strongly on normality. However, results suggest that, even under lack of normality, ELL’s performance is still worse than that of the Census EB update. In terms of bias, some areas under the traditional ELL show quite considerable bias, yet on average across areas the bias can be near zero. This suggests that, even if the bias across all areas of ELL can be small, it is not an accurate estimator in each area.<sup>28</sup> In fact, the empirical MSEs of the traditional ELL estimators for many areas are still much greater than those of the proposed Census EB ones in this experiment.

Even after the better performance of the Census EB procedure compared with the traditional ELL method under normality departure in the form of higher tails, model checking and diagnostics are extremely important, as in any model-based procedure. In fact, good performance of EB and Census EB

> 24Another slight modification made in separate simulations is that the location effect is simulated as _ηciid∼ N_<sup>�</sup> 0 _,_ 0 _._ 07<sup>2�</sup> .

> 25Results available upon request. Although it must be noted that under this scenario under many populations _σ_ ˆ _η_ <0, and thus many of the populations need to be re-simulated.

> 26For other FGT indicators see figures: 24, 25, 26, and 27.

> 27For other FGT indicators see figures: 28, 29, 30, and 31.

> 28Note that the purpose in SAE is not obtaining estimators performing well on average across areas since, in that case, the overall sample mean at the population level would be sufficient.

22
<!-- Start of picture text --> 2<br>A<br>nd ++ °. e e.<br>+, eo<br>ig 7ae et"<br>e<br>io) e? + t,2° e0%Fog, f hi + se hd +<br>re 0706 ote +e—e*, +, My + ° +<br>ie] % 90?" ,%0 % 00° ti Tea!<br>as eer? 4° hag + ++<br>9, oothaatingll +<br>N4e ia<br>A<br>+<br>A<br>0 20 40 60 80<br>Sorted areas by true FGTO<br>+ FGTO H3-CensusEB<br>« FGTO H3-CBEB (old bootstrap)<br>© FGTO H3-CBEB-old with PSU<br>4 True value from simulated Census<br><!-- End of picture text --> <!-- Start of picture text --> 2<br>A<br>ce} +, teamWes<br>+. steAY oo ty +<br>. . +<br>e ~ ry + *e eee tet, oe t+ eit! St be °; oe<br>© 2 0g + Set a ot ete oo ;-<br>ro) ¢ totee ¢tee?+ at +te+ m +At 1 % Fels °<br>+h<br>a + paatlatgaaat<br>nf fakes<br>A<br>+<br>A<br>0 20 40 60 80<br>Sorted areas by "true" FGTO<br>+ FGTO M&R (2010)<br>« FGTO ELL with context<br>» FGTO ELL<br>4 True value from simulated Census<br><!-- End of picture text -->
<!-- Start of picture text --> a<br>2q99t00%ee, 9%e, “egetenen* tenn **ee "sees,aatetyteenery”“ 4g%o%tegt* tePYtee,ef,,0% ag<br>aN - a<br>oS a * a“ x a<br>oO “ ry “<br>2 are x a . * x “- « “« a * x A<br>- aex . * x x x * a Be * Xx a *<br>rE ° a # + * x x x<br>rd * x ty at, xe” a ae x x Re<br>n a “4 .* “ aa a<br>&a a. ay * a 4 s a . « “<br>N . 7 - “ =<br>ef¢j<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB ¢ H3-CBEB (old bootstrap)<br>« ELL = H3-CBEB (old bootstrap with PSU)<br>« Direct<br>oO<br>N<br>~lo x * x x “* xx bad .<br>ouo a a eet ee a ¢ Ox x of fe<br>= o,.ost or’ Le* * oogteteoe. oXe* +,osebe, ant%fF xghto%ahah ge eet!xo SP axttataeaad ° sents<br>Xe — . . = os "tag "an x ae a . onan onyx “ gx .<br>Oo a a “ a<br>Wwe “ a<br>a ws Cotte tk a “as a<br>20 asia * * i. ae 4 “ a “aa a “ee aa * “s a we at, “ ‘<br>+pSht+ EER ER ct. chpeepee = gpot+. “HER eh bp bby.meeeaaa sty,‘<br>oO<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB ¢ H3-CBEB (old bootstrap)<br>« ELL = H3-CBEB (old bootstrap with PSU)<br>« Direct<br><!-- End of picture text -->
estimators in terms of bias and efficiency also relies on normality. Note that the sample size of household surveys is typically very large and therefore normality tests are all expected to reject the simple null hypothesis of normality due to the strong evidence against it (which is never exactly true in real applications). Hence, usual residual plots for checking the normality assumption using both predicted area effects and conditional unit-level residuals should be an important part of any application with real data.

In fact, log transformation of welfares is indeed conventionally taken because practically all welfare variables show strong right-skewness and heteroscedasticity. First of all, it is important to note that, even after log transformation, residuals in some applications still show a skewed distribution, where the skewness is transferred to the left tail. This occurs often in the presence of very small welfare values, which the log function actually shifts to minus infinity. A simple way to solve this left skewness problem caused by the log function is to add a shift _c >_ 0 to the original survey welfares, shifting them to values further apart from zero, where the slope of the log function is less pronounced. As Molina, Nandram and Rao (2014) recommends, this shift _c_ can be obtained by fitting the nested error model to a grid of values of _c_ in the range of the welfare values, and selecting the value for which the Fisher skewness of unit level residuals is closest to zero. Another possibility is to select a different transformation from the Box-Cox or power families of transformations (which contain the log), as implemented in the sae R package of Molina and Marhuenda (2015). In fact, Box and Cox (1964) included also the shift in their original family of transformations.

If one wishes to avoid finding the correct transformation to achieve normality<sup>29</sup> or if residuals still show skewness, even after selecting the best possible transformation of welfare, approaches for EB estimation have been recently developed for skewed distributions, see Graf, Marín and Molina (2019) for EB under general skewed distributions and Diallo and Rao (2018) considering skew-normal errors.

A simulation experiment was done by increasing the size of the population to _N_ = 100 _,_ 000, but the sample size is kept as _nc_ = 50 households per area. In this case, each area is made up of _Nc_ = 1 _,_ 250 households, which means that sampling fractions are now 4 percent (much smaller than the previous 20 percent). Everything else in the simulation experiment is kept the same as in the simulation experiment with the improved model and assuming normality. Under this scenario, ELL still performs worse than the proposed Census EB, see Figures 11 and 12. However, it now shows improved performance in certain areas, where it achieves a smaller MSE than the direct estimators (Figure 12). This bodes well for real world scenarios, where the sampling fractions hardly reach 0.5 percent and thus ELL will likely be preferable to direct estimators.<sup>30</sup> Under this same (more realistic) scenario, we analyze the performance of the MSE estimators obtained from each method. Figure 13 shows respectively the true MSEs of the traditional ELL estimators (approximated empirically with the _L_ = 10 _,_ 000 simulations), and the corresponding ELL estimated variances (section2). We can see that estimated ELL variances are severely understating the true MSEs. This severe underestimation also affects the CB-EB procedure (see appendix Figure 23). On the other hand, according to Figure 14, showing true MSEs of Census EB estimators and their corresponding parametric bootstrap MSEs, where Census EB and bootstrap MSE estimators are obtained from the algorithms of Section 6, the parametric bootstrap MSE estimators are tracking adequately the true MSEs.<sup>31</sup>

> 29Note that for a random variable _v_ , the transformation that leads to normality is given by Φ _−_ 1( _F_ ( _v_ )), where _F_ ( _·_ ) is the true cumulative distribution function (cdf) of _v_ and Φ( _·_ ) is the standard normal cdf. The problem is that the true cdf _F_ ( _·_ ) is typically unknown.

> 30For other FGT indicators see figures: 32, 33, 34, and 35

> 31Similar conclusions were obtained for other indicators, such as the poverty severity (FGT indicator with _α_ = 2), although results are not shown for brevity.

25
<!-- Start of picture text --> wo x x<br>x a bad<br>+ + x a a a a * *<br>+Xmaaterrx + x tHigath* Rox++ yHt+* pyaa+t t +4+t +<br>Sor, pee +,“a ha eae+ + +t. P Aex gt x~ +xf 4 + +t+, xx at *<br>= + “oe . an an + fay + .<br>— a ya ee t 7 ‘ Xe x +<br>5Bio “ 6 aA x ax +x .<br>ory*<br>oO<br>=F<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB ~*« ELLwithcontext + Direct<br>oO<br>oO<br>=<br>ay<br>So<br>So<br>oO<br>=<br>2-_- . x *<br>b©a oStd x x ad<br>Luan x :<br>= x<br>a“ “axy x * x x xa Me . * x ‘ * +, ‘ x a. *a * Xx<br>oo a+tet4at “ atm. Teteax Heyaegh pectthta Bex,OX AE eee* * AteaXatye +t,FaXA aggEtata waa<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB ~*« ELLwithcontext + Direct<br><!-- End of picture text -->
<!-- Start of picture text --> in nr<br>Onyee® patetgnnnneg®""e"euetgetteccggcngee"sgeg"eeneet es ateeaan, eeneneene”s<br>on a‘ a “ n<br>Ss “ a n a ra a a 4<br>bv: ax a aoa a a *~a4<br>~— »TM x * x ‘ a a x xy x x x a” x<br>Ww * . a ax x x, x a xTM x ee x « x x x .<br>iva] - “<br>N‘ Ae . é a a ‘ a ‘<br>a<br>. 4<br>s<br>+*<br>020 40 60 80<br>Area<br>+ H3-CensusEB » ELL with context<br>» H3-CBEB (old bootstrap with PSU) + Direct<br>a<br>o<br>N<br>a<br>AS ‘<br>SsSS a ‘ ‘ a<br>oOS“ 4 * a<br>—5 LLiat tar3 Lesial e ee eea x e ge eex nx” ee)Ay eex le bel |"5,xo"a eneex .<br>Lo a Oe ae<br>= a at at ee te at At a<br>wo ‘ = a “4 a “ at a “ ‘ a “ . a aaa a * a<br>FREER EH ERR HE ER ER EE EEE HERE CHEE EEE EEEE EEE EHR EEREEER EER CHER ht<br>oO<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB « ELL with context<br>* H3-CBEB (old bootstrap with PSU) + Direct<br><!-- End of picture text -->
<!-- Start of picture text --> o-Aa<br>= casasaaa<br>Sf<br>Ss mannan<br>co) Fons<br>vrx Wom x<br>~— wa x x<br>Wo abd x x<br>a a x x x<br>= x x x x xx x x<br>4 x x x xx x xx<br>x x x y<br>iad *x* *x xxxx * x x xx *x<br>xx pov Xx x xx x x x - x x 7 x x<br>.<br>o ma xX x yx x ra x * x<br>0 20 40 60 80<br>Areas sorted by MSE<br>4 TrueELLMSE- ~*~ Estimated ELL Variance<br>note: True MSE is the outcome of 10k populations (size 100k), estimate is from bootstrap<br>iQ<br>o<br>x<br>x<br>fon) x<br>x x x x a<br>Ss4——-w x x y x x * x x x* x x x*pannelx x:* ayanaiaaA®xx x x *<br>xN x aaah x x"<br>WW x * x x<br>n x* x x =x x x<br>x, x x<br>N annx x x<br>A<br>x<br>x<br>wo<br>0 20 40 60 80<br>Areas sorted by MSE<br>4 True MSE’ ~*~ Estimated H3-CensusEB MSE<br>note: True MSE is the outcome of 10k populations (size 100k), estimate is from bootstrap<br><!-- End of picture text -->
<!-- Start of picture text --> oO<br>=<br>o* g ° ~ ¢@ Pand a tee a<br>Sso *. x *.* ° * ne a Pos oar’Sa a oo 9, , °<br>S~x Fe°ot T+ oy+eet4O 6 x HO° .++ o, 0%°ofwoeox” ate.¢ te: x ye *<br>a + + + % o«% x % MF + x<br>ro et TH Pepe Pg xx tele<br>(0) +4, «4 x + to x ¥ y + ¢ + +)<br>re eet Ne et OF . * x +.<br>iPe x 2 + teh« ¢ x tta Pa nn$ey+,+ .<br>iva)©‘.++*4 * +t x<br>oOo<br>=1<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB * H3-CBEB (old bootstrap)<br>« ELL with context<br>oO<br>oO<br>co a x oa ol<br>So=} 4 Pad a4 °<br>= o ° o + 6<br>x * °<br>~— x * x a<br>5— ° ° xe +<br>Woan x + +x x oot x «?¢ ° o + . °<br>S . « ° x + ee x xe oe x<br>+ _* o ote + © ex :<br>+ +ete. a+Ba* | Te‘ , ++TO e+ag* Se+ te Hyat+<br>+ +4 t oy oy xt + +. x<br>o xNee+ FOO¢° + xe +.tee ot,t+ te AbePYe %ee Psd +7tye ep x + sot +<br>0 20 40 60 80<br>Area<br>+ H3-CensusEB * H3-CBEB (old bootstrap)<br>« ELL with context<br><!-- End of picture text -->
Table 1: Aggregate results across areas (FGT1)

||1|2|3|4|5|6|
|---|---|---|---|---|---|---|
|Model _R_<sup>2</sup>|<0.01|~0.42|~0.42|~0.42|~0.42|~0.46|
|_K_|2|6|6|6|6|6|
|_σηc_|_N_(0_,_0_._15<sup>2</sup>)<br>|_N_(0_,_0_._15<sup>2</sup>)<br>|_N_(0_,_0_._07<sup>2</sup>)<br>|_N_(0_,_0_._15<sup>2</sup>)<br>|_N_(0_,_0_._15<sup>2</sup>)<br>|_N_(0_,_0_._15<sup>2</sup>)<br>|
|_σech_|_N_(0_,_0_._5<sup>2</sup>)|_N_(0_,_0_._5<sup>2</sup>)|_N_(0_,_0_._5<sup>2</sup>)|Student’s t*|_N_(0_,_0_._5<sup>2</sup>)|_N_(0_,_0_._5<sup>2</sup>)|
|Pov. threshold|12|10.2|10.2|10.2|10.2|10.2|
|_nc_|50|50|50|50|50|10|
|_Nc_|20,000|20,000|20,000|20,000|100,000|100,000|
|**Direct estimates**|||||||
|AAB (x100)|0.031|1.180|1.183|1.721|1.556|3.744|
|AARB (x100)|33.000|17.446|17.212|16.538|20.101|43.347|
|ARMSE (x100)|1.269|2.417|2.404|2.386|2.835|5.143|
|ARRMSE (x100)|43.114|21.661|21.116|18.466|24.823|47.335|
|**ELL**|||||||
|AAB (x100)|0.155|0.450|0.175|2.363|Not run|Not run|
|AARB (x100)|76.528|27.755|13.663|25.288|Not run|Not run|
|ARMSE (x100)|2.042|3.627|1.906|3.608|Not run|Not run|
|ARRMSE (x100)|510.106|39.350|17.824|28.903|Not run|Not run|
|**ELL - with conte**|**xt**||||||
|AAB (x100)|0.169|0.413|0.169|2.264|0.408|1.936|
|AARB (x100)|77.412|27.496|13.723|25.077|26.040|25.372|
|ARMSE (x100)|2.036|3.591|2.098|3.566|3.529|2.929|
|ARRMSE (x100)|448.674|38.957|19.679|28.715|36.171|28.378|
|**H3-CBEB (with **|**PSU)**||||||
|AAB (x100)|3.742|3.125|3.205|Not run|3.120|Not run|
|AARB (x100)<br>|157.127|32.002|30.199|Not run|30.554|Not run|
|ARMSE (x100)|3.888|3.503|3.481|Not run|3.484|Not run|
|ARRMSE (x100)|363.459|38.417|34.072|Not run|36.025|Not run|
|**H3-CBEB**|||||||
|AAB (x100)|3.737|3.123|3.205|Not run|Not run|3.383|
|AARB (x100)|176.202|35.250|30.726|Not run|Not run|36.295|
|ARMSE (x100)|3.879|3.707|3.507|Not run|Not run|3.840|
|ARRMSE (x100)|577.523|45.157|35.216|Not run|Not run|40.608|
|**H3-CensusEB**|||||||
|AAB (x100)|0.007|0.014|0.012|1.272|0.013|1.590|
|AARB (x100)|26.387|11.170|9.755|12.959|10.550|19.432|
|ARMSE (x100)|0.932|1.560|1.372|1.739|1.542|2.319|
|ARRMSE (x100)|63.121|14.668|12.586|14.300|13.590|21.818|

AAB: Average absolute bias; AARB: Average absolute relative bias; ARMSE: Average root MSE; ARRMSE: Average relative root MSE. ***** Student’s _t_ distribution with 5 degrees of freedom and scaled by 0.5 30
One additional simulation is executed with the population of _N_ = 100 _,_ 000, but the sample size is kept as _nc_ = 10 households per area, which makes the total sample size _n_ = 800 . Each area is made up of _Nc_ = 1 _,_ 250 households, which means that sampling fractions are now 0.8 percent. The purpose of the simulation is to assess the performance of the estimators under a scenario more aligned to ELL. ELL (2002) proposed the location effect be at the cluster level, and in some applications this is a cluster of 10 observations. However, in this case the total survey sample size may not be so realistic, since in household surveys from many countries, the overall sample size is of several thousands. As expected, under this scenario, the differences in bias of the different methods is less stark. Note that with zero area sample size ( _nc_ = 0), Census EB and traditional ELL become approximately the same, so, as we approach this scenario, estimators come together (Figure 15). Because of the limited sample, all methods now show considerably more bias and a higher MSE. Concurrently, H3-CensusEB now appears to be more aligned to the ELL and H3-CBEB. However, H3-Census EB still outperforms the older methods in terms of bias and MSE. Note also, that usually ELL is done by setting the location effect at the cluster level but results are presented at higher levels.<sup>32</sup> However, aggregation to higher levels may underestimate the true MSE as noted by Das and Chambers (2017). The inclusion of contextual level variables may help ( _ibid_ ) although this requires being able to link clusters in the survey sample to the census which in practice is not always feasible.

Aggregate results across areas are presented in Table 1 for the simulations discussed.<sup>33</sup> Results from column 1 are those that replicate the simulation experiment of Molina and Rao (2010). Note that the large values of the average relative root mean squared error (ARRMSE) may arise from true values of FGT2 close to 0, but the true values are common to all estimators. Regardless, the better performance of the H3-Census EB update is quite clear in terms of all the performance measures and under all the simulations run. Moreover, the results in the tables corroborate what is observed in the figures.

# **7.3 Examining the bias of the CB-EB PovMap update**

The bias of the CB-EB estimator observed in our simulation experiments is a reason for concern. In the CB-EB method with clusters equal to the areas, one could argue that the bias comes from the fact that not all areas are equally likely to be included in the sample that is selected in step 1 of this procedure. However, in our simulation experiments, bias also appears for the CB-EB method with PSUs selected within the areas in step 1. Thus, the question remains: what is the source of this bias?

Investigating whether part of the bias comes from the regression coefficients used to draw the welfares _β_ ˆ<sup>_∗_</sup> , which vary across the bootstrap replicates, we have plotted a histogram of the simulated regression intercepts _β_<sup>ˆ</sup><sup>_∗_</sup> obtained in the _M_ replicates of the CB-EB bootstrap procedure, with _M_ = 50 _,_ 100 _,_ 300, for a single population. Figure 17 shows the results for _M_ = 50 and _M_ = 300 bootstrap replicates. This figure shows that the distribution of the simulated regression intercepts _β_ 1<sup>_∗_forsmall</sup><sup>_M_isnotactually</sup> centered around the true value _β_ 1 = 3; in fact, it is right-skewed with a mean clearly exceeding the true value. While increasing the bootstrap replicates _M_ yields better results (right plot), the regression intercepts are still not centered around the true intercept.

A second plausible source of bias is through the simulated location effects, _ηc_<sup>_∗_.Sincethemethodselects</sup> PSUs with replacement, some areas might be selected more than once and others might not be selected. As a consequence, var [ˆ� _ηc_ ] might be underestimated in an area selected more than once because there will artificially be more observations in that area. On the other hand, if an area is not selected, then

> 32For other FGT indicators see figures: 36, 37, 38, and 39.

> 33See tables 2 and 3 for FGT0 and FGT2, respectively.

31
<!-- Start of picture text --> wo~<br>S<br>2<br>@®QoCc<br>o<br>o<br>2.95 3 3.05 3.1<br>Intercept 50 simulations<br><!-- End of picture text --> <!-- Start of picture text --> 2<br>So<br>ze<br>QaCc@<br>wo<br>o<br>2.95 3 3.05 3.1<br>Intercept 300 simulations<br><!-- End of picture text --> <!-- Start of picture text --> N-<br>ere hat ah, oF ae<br>PN+ TTSe ght ates<br>se 4 +<br>x<br>2<br>8<br>ao ~<br>°<br>020 40 60 80<br>area<br>¢@ Bias 100 sims ~ Bias 50 sims<br><!-- End of picture text --> <!-- Start of picture text --> ©x<br>y te ts .<br>=. te ’ . xt + % -<br>=}S x* x % e «xd ot o% +<br>sg > a ee o ° } %<br>é” ° e ¢ oo? ox »% oo x . + x<br>x + } ex ° 3¢<br>&ag x+ ¢% x?x Pa x oex° ¢ *x x & ¢x<br>cys * * ¢ sfx<br>+ t s o. :<br>v<br>0 20 40 60 80<br>area<br>¢@ Bias 100 sims ~ Bias 50 sims<br><!-- End of picture text -->
# **8 Conclusions**

Since the turn of the 21st century, the World Bank has been obtaining small area estimates of poverty and inequality indicators using the ELL approach. One of the reasons for the method’s popularity was the existence of software that made applying the method to real data relatively easy. This began with an implementation in SAS by Demombynes (2002), which was followed by PovMap implemented by Zhao (2006), and finally a Stata version (“sae” package by Nguyen et al., 2018). Along those lines, the research effected in this paper also represents a considerable update to the “sae” package in Stata.<sup>36</sup> Countries often set out to obtain small area estimates because more precise poverty estimates at a more granular level allow for improved allocation of resources, see Elbers et al. (2007). This research comes just in time for the 2020 round of population census and should provide an improved tool for the operationalization of the SDGs at a sub-national level.

An important aspect of the small area estimation procedures is how parameter estimates from an assumed population model are applied to census data to obtain the small area estimates. As noted in this document, the traditional ELL procedure is aligned to the approach used in multiple imputation, where the main aim is not prediction. Under multiple imputation, the method that yields the lowest MSE yields invalid statistical inference (Van Buuren, 2018). On the other hand, the goal of small area estimation is to improve precision.

This paper presents a substantial revision to the traditional methods by ELL (2002, 2003) and the updates by Van der Weide (2014) for small area estimation.<sup>37</sup> Our results show substantial gains of the proposed Census EB method with respect to the previous methods in all the considered scenarios (varying area population sizes and sampling fractions, model prediction power and distributional assumptions) and also of the considered parametric bootstrap MSE estimators. Thus, the updated methodology provides a considerable improvement in the quality of the SAE estimates obtained under the World Bank’s agenda. We show that, for a single population, the original ELL method tends to align with the national poverty estimates and does not capture the area heterogeneity, although its performance improves when averaging across populations. Nevertheless, caution should be taken when doing out of sample prediction even under EB methods, since in that case estimates are very similar to the traditional ELL ones and can thus deviate considerably from the truth. This is particularly relevant in cases where the explanatory power of the chosen correlates is low.

An additional finding of this research is the considerable bias of the CB-EB update by Van der Weide (2014) and as implemented by Nguyen et al. (2018) to the World Bank’s toolkit. The revised Census EB method is aligned with MR’s (2010) method, which shows a level of bias that is several orders of magnitude smaller than the previous CB-EB method. Furthermore, the MSE for the Census EB method is also considerably smaller. This research and its accompanying software implementation represent a massive improvement to the World Bank’s current toolkit and overall poverty mapping agenda.

This note serves as a background piece for a guidance note on small area estimation for the World Bank. The simulations executed here are done under a controlled scenario, and thus a natural question is how the findings would differ under a real-world application. Now that the initial limitations of the methods have been identified, the revisions can be put to the test on real-world data.

> 36An update to Nguyen et al. (2018) is in progress, but all Stata codes and commands used in this document are available at https://github.com/pcorralrodas/SAE-Stata-Package

> 37The revision is also accompanied by an update to the 2018 Stata “sae” package by Nguyen et al. (2018)

33