metadata: authors: 0: author: Corral Rodas,Paul Andres
1: author: Kastelic,Kristen Himelein
2: author: Mcgee,Kevin Robert
3: author: Molina,Isabel
count: Mexico
docna: 0: docna: A Map of the Poor or a Poor Map ?
docty: Policy Research Working Paper
subtopic: Inequality,Employment and Unemployment,Democratic Government,De Facto Governments,Public Sector Administrative & Civil
            Service Reform,Public Sector Administrative and Civil
            Service Reform,Economics and Finance of Public
            Institution Development,State Owned Enterprise Reform,Labor & Employment Law,Information Technology
teratopic: Law and Development,Governance,Social Protections and Labor,Information and Communication Technologies,Public Sector Development,Poverty Reduction
abstracts: cdata!: This paper evaluates the performance of
            different small area estimation methods using model and
            design-based simulation experiments. Design-based simulation
            experiments are carried out using the Mexican Intra Censal
            survey as a census of roughly 3.9 million households from
            which 500 samples are drawn using a two-stage selection
            procedure similar to that of Living Standards Measurement
            Study surveys. Several unit-level methods are considered as
            well as a method that combines unit and area level
            information, which has been proposed as an alternative when
            the available census data is outdated. The findings show the
            importance of selecting a proper model and data
            transformation so that the model assumptions hold. A proper
            data transformation can lead to a considerable improvement
            in mean squared errors. The results from design-based
            validation show that all small area estimation methods
            represent an improvement, in terms of mean squared errors,
            over direct estimates. However, methods that model unit
            level welfare using only area level information suffer from
            considerable bias. Because the magnitude and direction of
            the bias are unknown ex ante, methods that rely only on
            aggregated covariates should be used with caution, but they
            may be an alternative to traditional area level models when
            these are not applicable.
geo_reg_key: 119228,517191
repnme: repnme: A Map of the Poor or a Poor Map ?
display_title: A Map of the Poor or a Poor Map
listing_relative_url: /research/2021/04/32989889/map-poor-or-poor-map
topicv3: State Owned Enterprises m1326323 721,Information and Communication Technologies
            m1326624 133,Poverty m1326687 695,Household Surveys m1504071 2761,Census Data m1504073 2763
publishtoextweb_dt: 2021-04-12T00:00:00Z
dois: 10.1596/1813-9450-9620
historic_topic: Law and Development,Governance,Social Protections and Labor,Information and Communication Technologies,Public Sector Development,Poverty Reduction
origu: VP, HD Practice Group (GGHVP)
no_of_pages: 57
author_keys: 0: author_key: 1204461
1: author_key: 903202
2: author_key: 1207565
3: author_key: 1234915
model_extractions: Policy Research Working Paper 9620 A Map of the Poor or a Poor Map?

_Paul Corral Kristen Himelein Kevin McGee Isabel Molina_ Poverty and Equity Global Practice April 2021
Policy Research Working Paper 9620

# **Abstract**

This paper evaluates the performance of different small area estimation methods using model and design-based simulation experiments. Design-based simulation experiments are carried out using the Mexican Intra Censal survey as a census of roughly 3.9 million households from which 500 samples are drawn using a two-stage selection procedure similar to that of Living Standards Measurement Study surveys. Several unit-level methods are considered as well as a method that combines unit and area level information, which has been proposed as an alternative when the available census data is outdated. The findings show the importance of selecting a proper model and data transformation so that the model assumptions hold. A proper data transformation can lead to a considerable improvement in mean squared errors. The results from design-based validation show that all small area estimation methods represent an improvement, in terms of mean squared errors, over direct estimates. However, methods that model unit level welfare using only area level information suffer from considerable bias. Because the magnitude and direction of the bias are unknown ex ante, methods that rely only on aggregated covariates should be used with caution, but they may be an alternative to traditional area level models when these are not applicable.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at pcorralrodas@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# A Map of the Poor or a Poor Map?

Paul Corral,<sup>∗</sup> Kristen Himelein,<sup>†</sup> Kevin McGee<sup>‡</sup> and Isabel Molina<sup>§¶</sup>

**Key words:** Small area estimation, ELL, Poverty mapping, Poverty map, Empirical best, Parametric bootstrap, nested error model, twofold nested error model

**JEL classification:** C55, C87, C15

> ∗The World Bank Group - Human Development Group Chief Economist’s Office (pcorralrodas@worldbank.org)

> †The World Bank Group - Poverty and Equity Global Practice

> ‡The World Bank Group - Development Economics Data Group

> §Universidad Carlos III de Madrid - Department of Statistics (imolina@est-econ.uc3m.es)

> ¶The authors acknowledge financial support from the World Bank. We thank Sandra Carolina Segovia for research assistance. We thank J.N.K. Rao for comments and suggestions. We also thank Roy van der Weide, Alexandru Cojocaru, David Newhouse, and Samuel Freije-Rodriguez for comments. Additionally, we thank Carolina Sanchez and Roberta Gatti for providing support and space to pursue this work. Finally, we thank the Global Solutions Group on Data for Policy, as well as all attendants of the Summer University courses on Small Area Estimation. Any error or omission is the authors’ responsibility alone.
# **1 Introduction**

The eradication of poverty was the first Millennium Development Goals (MDGs) established by the United Nations in 2000 and continues as Sustainable Development Goals (SDG) 1.1.1, but governments can only properly target poverty if they know where it is. Traditionally, for a given country, the best source for information on the living standards of its population are household surveys. These surveys are a powerful tool towards defining and addressing the needs of people. These surveys, however, usually offer reliable information only at highly aggregated levels of the population. In other words, _direct_ survey estimates tend to be adequate for very large populations but inadequate for smaller populations.

Small area estimation (SAE) is a branch of statistics focused on improving the reliability of estimates and the associated measures of uncertainty for populations where samples cannot produce sufficiently reliable estimates (Rao and Molina, 2015). Small areas can be any population subgroup and are not necessarily tied to geographical areas. According to Ghosh and Rao (1994), the use of small area statistics increased towards the end of the 20th century due to improved computing power and the advent of theoretically sound statistical methods. The main principle behind small area statistical methods is to use modeling to “borrow strength” from auxiliary data sources (e.g. census or administrative data) to produce more efficient estimators than direct survey data alone. Model-based techniques for small area estimation commonly fall within two groups: i) area based models; for examples see Fay and Herriot (1979) and Torabi and Rao (2014); ii) unit-level models; for examples see Molina and Rao (2010) and Elbers, Lanjouw and Lanjouw (2003). The focus of this paper is to conduct model and design-based evaluation of different unit-level SAE methods. Model-based methods assume a data generating process for the population that follows the model’s assumptions, and the target parameters are assumed to be random (Tzavidis et al., 2018). In this paper, model-based evaluation is done considering two-stage sampling strategies and thus assumes a model that includes variation at the domain level as well as at the cluster level, where clusters are the primary sampling units, nested within domains (see Marhuenda et al., 2017). Design-based simulation entails applying the chosen methods under realistic conditions (Tzavidis et al., 2018). Here, a design-based evaluation of several estimators is performed using the 2015 Mexican Intra Censal Survey, which is used here as a census, and from which we draw repeated samples. The estimation methods considered are those of Elbers, Lanjouw and Lanjouw (2003 - ELL henceforth), the empirical best predictor of Molina and Rao (2010 - MR henceforth),<sup>1</sup> the twofold nested error extension of MR (2010) presented by Marhuenda et al. (2017), and finally an adaptation, presented by Nguyen (2012), that combines unit and area level information, and which has been proposed as an alternative when the available census data is too outdated to be considered for use.

The paper’s contribution is twofold: i) it presents a thorough evaluation of the considered methods as well as their pros and cons, and ii) it employs the sampling strategies most often encountered

> 1In reality, we apply the Census EB variant introduced by Guadarrama, Molina and Rao (2016)

2
in real world applications to conduct design-based validations. The results are expected to provide a guideline to others undertaking small area estimation. The results from design-based validation show that all small area estimation methods represent an improvement, in terms of MSE, over direct estimates. However, given the inherent high level of bias of methods relying solely on aggregate information (see Nguyen, 2012; Lange et al., 2018; Masaki et al., 2020) these should be used with caution and only under certain circumstances.

The paper first presents an overview of the small area estimation methods considered, directing readers who are interested in a more in-depth description to the appropriate sources. Then the model-based simulation design is detailed and results of the simulation experiments are presented. This section is followed by a description of the 2015 Mexican Intra Censal Survey and how it is adapted to represent a census and how samples are drawn from the created census. Results from the design-based simulation experiments are then presented. Finally, conclusions and lessons learned are provided.

# **2 Unit-level models**

The nested error model used for small area estimation by ELL (2003) and Molina and Rao (2010) was originally proposed by Battese, Harter and Fuller (1988) to produce county-level corn and soybean crop area estimates for the U.S. state of Iowa. For the estimation of poverty and welfare, the ELL and MR methods assume the transformed welfare _ych_ for each household _h_ within each location _c_ in the population is linearly related to a 1 _× K_ vector of characteristics (or correlates) _xch_ for that household, according to the nested error model: where _ηc_ and _ech_ are respectively location and household-specific idiosyncratic errors, assumed to be independent from each other, following: where the variances _ση_<sup>2and</sup><sup>_σ_</sup> _e_<sup>2areunknown.Here,</sup><sup>_C_isthenumberoflocationsinwhichthe</sup> population is divided and _Nc_ is the number of households in location _c_ , for _c_ = 1 _, . . . , C_ . Finally, _β_ is the _K ×_ 1 vector of coefficients. Under the original ELL methodology, the locations indexed with _c_ are supposed to be the clusters, or primary sampling units (PSUs) of the sampling design and do not necessarily correspond to the level at which the estimates will be ultimately produced. In fact, clusters are typically nested within the areas of interest (e.g. census enumeration areas within large administrative areas). Presenting estimates at a higher aggregation level than the clusters (for which random effects are included in the model) may not be appropriate in cases of considerable between-area variability, and may underestimate the estimator’s standard errors (Das 3
and Chambers, 2017). The recommended approach to mitigate this issue is to include covariates that sufficiently explain the between-area heterogeneity in the model ( _ibid_ ). In this regard, ELL (2002) suggests the inclusion of cluster-level covariates as a way to explain location effects. Nevertheless, this approach is context specific and may not always suffice to ameliorate the issues with between-area heterogeneity. In this regard, Marhuenda et al. (2017) recommend and show that location effects should be at the same aggregation level at which estimation is desired.

If the location effect is specified at the same level where estimation is desired, then the difference between ELL (2003) and MR (2010) reduces to differences in how estimates are obtained and the addition of Empirical Best (EB) prediction by MR (2010). The EB method from MR (2010) conditions on the survey sample data and thus makes more efficient use of the information at hand, whereas ELL does not include this component. In essence, under ELL, for any given area present in the sample the ELL estimator of the census area mean _y_ ̄ _c_ is obtained by averaging across _M_ simulated censuses and is given by _y_ ̄ _c_<sup>_∗_(</sup><sup>_m_)</sup> ≈ _X_<sup> ̄</sup> _c_<sup>_′β_+</sup><sup>_η_</sup> _c_<sup>_∗_(</sup><sup>_m_)</sup> + ̄ _e_<sup>_∗_</sup> _c_<sup>(</sup><sup>_m_)</sup> , _m_ = 1 _, . . . , M_ , where _E_ [ _ηc_<sup>_∗_]=0</sup> <u>1</u> _M_ and _E_ [ _e_<sup>_∗_</sup> _ch_<sup>]=0.</sup> Thus, the ELL estimator _M_ � _m_ =1<sup>_y_ ̄</sup> _c_<sup>_∗_(</sup><sup>_m_)</sup> , which approximates _E_ ( ̄ _yc_ ), reduces to the regression synthetic estimator, _X_<sup> ̄</sup> _c_<sup>_′β_(MR,2010).Ontheotherhand,underMR(2010),</sup> conditioning on the survey sample ensures the estimator includes the random location effect, since _E_ [ ̄ _yc|ηc_ ] ≈ _X_<sup> ̄</sup> _c_<sup>_′β_+</sup><sup>_ηc_.Mechanically,however,conditioningonsurveydatarequiresthelinkingof</sup> areas across surveys and census, something that is not always straightforward.<sup>2</sup> Other differences between ELL (2003) and MR (2010) are the computational algorithms used to obtain point and noise estimates; see Corral, Molina and Nguyen (2020 - CMN henceforth) for further discussion.

ELL’s (2003) approach to obtain estimates builds upon the multiple imputation (MI) literature in that it uses a single algorithm that produces point and noise estimates by varying model parameters across simulations (see Tarozzi and Deaton, 2009 as well as CMN, 2020). The use of MI methods for obtaining point and noise estimates has shortcomings, however. Under multiple imputation, the method that yields the lowest MSE does not necessarily yield valid statistical inference (Van Buuren, 2018), which is contradictory to the goal of small area estimation in improving precision. Accordingly, CMN (2020) show using simulated populations that the noise estimate (referred to as variance by ELL) is not an appropriate estimate of the MSE. In contrast, MR’s (2010) point estimates are obtained through two separate procedures. Point estimates are obtained by a Monte Carlo simulation and noise estimates are obtained by a parametric bootstrap procedure originally proposed by González-Manteiga et al. (2008).<sup>3</sup> In light of the emerging literature, including MR (2010), the World Bank expanded the original ELL design by adding EB predictors and revising its fitting methodology to incorporate heteroskedasticity and survey weights to account for complex sampling designs (Van der Weide, 2014). The survey weights are incorporated in the estimates of the regression coefficients and in the variance of the components following Huang and Hidiroglou (2003) based on Henderson method III (Henderson,

> 2Under ELL, when including area level covariates, linking the survey and the census areas is also required. Note that it is not necessarily the case that enumeration areas for a census and survey will match.

> 3For more details, see Corral, Molina and Nguyen (2020).

4
1953) as well as in the predicted area effects as in the pseudo empirical best linear unbiased predictor (EBLUP) proposed by You and Rao (2002). In the absence of survey weights and heteroskedasticity, the fitting method yields parameter estimates that are quite similar to the restricted maximum likelihood (REML) fitting method used by MR (2010). The difference remains, however, in how point and noise estimates of the target indicators are obtained.

As further detailed in CMN (2020), however, even following these revisions, the bootstrap procedure, as implemented by `PovMap` software (Zhao, 2006) for EB and later detailed in Nguyen et al. (2018) differed from the original EB procedure from MR (2010). CMN (2020) note the remaining issues rested in the continued reliance on MI methods as the basis of the methodology. Because under the updated fitting methodology, Van der Weide (2014) does not offer an estimate of var � _ση_<sup>2</sup> � bootstrap samples of the data must be taken for each simulation to obtain the fitting parameters � _β_ ˆ _∗,_ ˆ _ση_ 2 _∗_<sup>_,_ˆ</sup><sup>_σ_</sup> _e_<sup>2</sup><sup>_∗_</sup> �.<sup>4</sup> This approach was taken to allow for an algorithm similar to the procedure used in the original implementation of ELL. However, if clusters were equal to areas, then it is unlikely that all areas are included in the sample and thus an area could benefit from EB only in a subset of the simulations, introducing bias into the resulting point and noise estimates. CMN (2020) also present evidence on the fact that, even if the location effect was modeled at the domain level and bootstrap samples of clusters are taken, some bias would still likely remain in the resulting estimates of the original implementation of EB in `PovMap` and the `sae` Stata package.

The Stata package for small area estimation developed by Nguyen et al. (2018) has been updated to integrate the fitting methods from Van der Weide (2014) with the prediction and bootstrap methods from MR (2010). The new method is referred to as the H3-CensusEB (Molina, 2019), and the previous method (i.e. original bootstrap using EB from Van der Weide (2014)) is called H3-CBEB.<sup>5</sup> CMN (2020) performs a model-based validation of the different methods: i) CensusEB, ii) EB, iii) H3-CBEB and iv) ELL. CMN (2020) extend the simulations done by MR (2010) by i) including the area means of the covariates as additional variables in the model; ii) considering a model that has considerably higher explanatory power by adding more covariates; iii) considering larger population sizes and smaller sampling fractions; and iv) generating errors from a Student’s _t_ 5 instead of a normal distribution. However, in all these simulations, population data are generated under model (1).

Currently, the software available for small area estimation of non-linear parameters, such as the `sae` Stata package by Nguyen et al. (2018) as well as the R package `sae` by Molina and Marhuenda (2015), only allow for estimation under the nested error model specified in (1). However, since household surveys often use two stage sampling, it seems appropriate to consider a twofold nested error model. Marhuenda et al. (2017) extend the EB method from MR (2010) to a twofold nested error model, given by:<sup>6</sup>

> 4ELL (2002) provides the derivation of the estimate for var<sup>�</sup> _ση_<sup>2</sup> � in their appendix.

> 5The method is called CensusEB because it does not link survey and census households. The old method is called the clustered bootstrap EB (CBEB). For the updated Stata package, see https://github.com/pcorralrodas/SAEStata-Package.

> 6For simplicity, we omit the heteroskedasticity weights considered by Marhuenda et al. (2017).

5
_yach_ = _xachβ_ + _ηa_ + _ηac_ + _each, h_ = 1 _, . . . , Nac, c_ = 1 _, . . . , Ca, a_ = 1 _, ..., A,_ (2) where _ηa_ is the random effect for area _a_ and _ηac_ is the random effect of cluster _c_ within area _a_ . These effects along with the individual model errors, _each_ , represent the unexplained variation of the transformed welfare, _yach_ , across areas, clusters, and households (Marhuenda et al., 2017).<sup>7</sup> All three components are assumed to be mutually independent, following:<sup>8</sup> Using the assumed twofold nested error model, Marhuenda et al. (2017) study the effect of a misspecified model (i.e. when assuming a cluster effect and presenting results at the area level) on the MSE estimator. The argument posited by Das and Chambers (2017) is that auxiliary variables should explain the between-area variation of the response variable. If this fails, there may be model misspecification, which can lead to an underestimation of the true MSE of the ELL estimator (Marhuenda et al., 2017). Through model-based simulation experiments under the assumed model (2), Marhuenda et al. (2017) reach three important conclusions under the model-based setup:

1. The relative values of _σac_<sup>2and</sup><sup>_σ_</sup> _a_<sup>2areofkeyimportance;thelargerthevalueof</sup><sup>_σ_</sup> _a_<sup>2relativeto</sup> _σac_<sup>2, the more problematic it is to apply models where effects are specified at the cluster level,</sup> including the original ELL and EB with locations specified at the cluster level. Additionally, EB with location effects specified at the cluster level, while performing better than ELL,<sup>9</sup> will also perform worse the larger the value of _σa_<sup>2isrelativeto</sup><sup>_σ_</sup> _ac_<sup>2.</sup>

2. Even if the true model contains random effects only at a single level, the assumption of a twofold model practically does not entail loss in efficiency.

3. EB estimates under model (1) with random effects specified at the area level will have similar performance to the twofold EB estimates. The recommendation is that if the estimation is done using model (1) because of software availability or simplicity, then the model’s random effects should be specified at the level at which results are desired.

As noted by Marhuenda et al. (2017), small area estimators based on unit level models often achieve very large reductions in MSE compared to direct estimators. EB estimators based on unit level models are also likely to achieve considerable gains in terms of MSE over Fay-Herriot (FH) area level models.<sup>10</sup> For example, Molina and Morales (2009) obtained mild gains over direct poverty

> 7Marhuenda et al. (2017) present these effects as domains and sub-domains.

> 8For full derivation of EB predictors under the twofold model, see Marhuenda et al. (2017).

> 9The performance is better than ELL because it includes the average of the cluster effects as opposed to ELL which only includes the linear fit.

> 10Fay-Herriot models were introduced by Fay and Herriot (1979) and are a popular area-level small area estimation model.

6
and poverty gap estimates when using a FH model. Molina and Rao (2010) though, using the same data sources as Molina and Morales (2009) but applying unit level models and EB prediction, obtain considerably larger gains. However, when the available census and survey are not from the same year, small area estimates based on unit level models may result in biased estimates. In such scenarios, FH models offer an alternative because these do not require census microdata. Alternatively, twofold models such as that of Torabi and Rao (2014) could also be considered to achieve larger gains as noted by Molina (2019).

Another potential solution is to use only aggregated covariates in the model for the household level welfare. This alternative is presented by Nguyen (2012) in an application for Vietnam. The author proposes a model where the dependent variable is household level logarithm of per capita expenditure from a recent survey, in this case the Vietnam Household Living Standard Survey from 2006, whereas all covariates are commune level means. These means are obtained from a dated (1999) census, although the author notes geographic information system data (GIS) could also be included into the set of covariates. Nguyen (2012) obtains ELL estimates for small areas under that model and compares the performance with that of typical ELL estimates obtained using unit level covariates from the Vietnam Household Living Standard Survey from 2006 and the 2006 Rural Agriculture and Fishery Census. The author finds provinces and districts hovering around the middle of the distribution suffer from considerable re-rankings across methods, but those at the top and the bottom are relatively stable.

Lange et al. (2018) present an approach similar to Nguyen’s (2012) which the authors suggest as an alternative in cases when census and survey data are not from similar periods, though the same issues noted above for the ELL method would likely persist in a model using only arealevel covariates. Masaki et al. (2020) use a similar modeling approach to Nguyen’s (2012), but take measures to address some of the shortcomings of a standard ELL approach and obtain EB estimators of Molina and Rao (2010) which appear to be more precise. The authors conduct a design-based validation study using a wealth index constructed with principal component analysis using census data for Sri Lanka and Tanzania. Their results show the approach may hold promise. In the sections that follow, the different procedures discussed here are tested under model-based and design-based simulation experiments.

# **3 Model-based simulation experiments**

The simulation experiment described here is based on those conducted by Marhuenda et al. (2017) in which the true data generating process is a twofold nested error model. Such a model will better accommodate the usual applications of poverty mapping, where household surveys use two-stage sampling. In fact, the traditional ELL method includes random effects only at the cluster level but estimates are given for a higher level.

7
In this simulation experiment a census data set of _N_ = 20 _,_ 000 observations is created, where observations are allocated among 40 areas ( _a_ = 1 _, . . . , A_ ). Within each area, observations are uniformly spread over 10 clusters ( _c_ = 1 _, . . . , Ca_ ). Each cluster, _c_ , consists of _Nac_ = 50 observations, and each cluster is labeled from 1 to 10. The assumed model contains both cluster and area effects. Cluster effects are simulated as _ηac iid∼ N_ �0 _,_ 0 _._ 12�, area effects as _ηa iid∼ N_ �0 _,_ 0 _._ 052� and household specific residuals as _eachiid∼N_ �0 _,_ 0 _._ 52�, where _h_ = 1 _, . . . , Nac_ ; _c_ = 1 _, . . . , Ca_ ; _a_ = 1 _, . . . , A._ Covariates are simulated as follows:<sup>11</sup>

1. _x_ 1 is a binary variable, taking value 1 when a random uniform number between 0 and 1, at the household level, is less than or equal to 0 _._ 3 + 0 _._ 5 40<sup>_<u>a</u>_+ 0</sup><sup>_._2</sup> 10<sup>_<u>c</u>_.</sup>

2. _x_ 2 is a binary variable, taking value 1 when a random uniform number between 0 and 1, at the household level, is less than or equal to 0 _._ 2.

3. _x_ 3 is a binary variable, taking value 1 when a random uniform number between 0 and 1, at the household level, is less than or equal to 0 _._ 1 + 0 _._ 2 40<sup>_<u>a</u>_.</sup> 4. _x_ 4 is a binary variable, taking value 1 when a random uniform number between 0 and 1, at the household level, is less than or equal to 0 _._ 5 + 0 _._ 3 40<sup>_<u>a</u>_+ 0</sup><sup>_._1</sup> 10<sup>_<u>c</u>_</sup> 5. _x_ 5 is a discrete variable, simulated as the rounded integer value of the maximum between 1 and a random Poisson variable with mean _λ_ = 3 �1 _−_ 0 _._ 1 80 _<u>a</u>_ �.

6. _x_ 6 is a binary variable, taking value 1 when a random uniform value between 0 and 1 is less than or equal to 0.4. Note that the values of _x_ 6 are not related to the area’s label.

7. _x_ 7 is a binary variable, taking value 1 when a random uniform number between 0 and 1 is greater than or equal to 0 _._ 2 + 0 _._ 4 40<sup>_<u>a</u>_+ 0</sup><sup>_._1</sup> 10<sup>_<u>c</u>_</sup> The welfare vector for each cluster within an area is created from the model with these covariates, as follows: The dependent variable, _yach_ , is in the log scale. The poverty line in this scenario is fixed at _z_ = 12. From the created “census,” 20% of the observations are sampled in each of the clusters using simple random sampling without replacement;<sup>12</sup> this yields our “survey” data. The generation and sampling processes are repeated _L_ = 10 _,_ 000 times. In each simulation replicate, the following quantities are computed for the poverty rates and gaps in each area:

> 11 Covariates are simulated following the approach from Molina and Rao (2010) and Marhuenda et al. (2017), with slight modifications.

> 12The same units are sampled in every simulation and the values of _x_ 1 to _x_ 7 for all the census units are also kept fixed; this implies that the values of these covariates for the sample units are always the same across simulations.

8
1. True poverty indicators _τa_ , using the “census”.

2. Direct estimators _τ_ ˆ _a_<sup>_DIR_</sup> using the “survey”, defined as the sample versions of _τc_ .

3. Census EB estimators _τ_ ˆ _a_<sup>_CEBac_</sup> presented in Marhuenda et al. (2017) based on a twofold nested error regression model, and obtained using a Monte Carlo approximation with _M_ = 50 replicates. Note that, for this estimator, the fitted model agrees with the true data generating process (3).

4. Census EB estimators _τ_ ˆ _a_<sup>_CEBa_</sup> presented in CMN (2020) based on a nested error model with only **area** random effects obtained using a Monte Carlo approximation with _M_ = 50 replicates.

5. Census EB estimators _τ_ ˆ _a_<sup>_CEBc_</sup> presented in CMN (2020) based on a nested error model with only **cluster** random effects and including the aggregate cluster and area means of the considered auxiliary variables, where _M_ = 50.

6. Traditional ELL estimators _τ_ ˆ _a_<sup>_ELLc_</sup> , based on a nested error model with only **cluster** random effects and including the aggregate cluster and area means of the considered auxiliary variables, where _M_ = 50.

7. Unit-context Census EB estimators _τ_ ˆ _a_<sup>_UC−CEBa_</sup> based on a nested error model with random effects at the **area level** . This estimator follows the approach from Masaaki et al. (2020) that is a modified version of that of Nguyen (2012), which uses only area means for some of the right hand side variables.<sup>13</sup> Nguyen (2012) proposes this solution for the case when only a dated census and a recent survey are available.

8. Unit-context **two-fold nested error** Census EB estimators _τ_ ˆ _a_<sup>_UC−CEBac_</sup> based on a twofold nested error model with random location effects at the **area and cluster level** . This estimator follows the approach from Masaaki et al. (2020) and Nguyen (2012), where only area means for some of the right hand side variables are used.<sup>14</sup> Model bias and MSE are approximated empirically as in MR (2010), as the averages across the _L_ = ˆ 10 _,_ 000 simulations of the prediction errors in each simulation ( _l_ ), _τa_<sup>_j_(</sup><sup>_l_)</sup> _− τa_<sup>(</sup><sup>_l_)</sup> and of the squared prediction errors respectively, where _j_ stands for one of the methods: _DIR, CEBac, CEBa, CEBc, ELLc, UC− CEBa, UC − CEBac_ .<sup>15</sup> Model bias and root MSE for a given area’s estimate are computed at the area level as follows:

> 13The covariates used in this model are: _x_  ̄1 _ac_ , _x_  ̄3 _a_ , _x_  ̄4 _ac_ , _x_  ̄5 _a_ , and _x_  ̄7 _ac_ .

> 14The covariates used in this model are: _x_  ̄1 _ac_ , _x_  ̄3 _a_ , _x_  ̄4 _ac_ , _x_  ̄5 _a_ , and _x_  ̄7 _ac_ .

> 15 _E_<sup>�</sup> _τ_ ˆ _c_<sup>_j−τ_</sup> _c_ � for the bias and _E_<sup>�</sup> _τ_ ˆ _c_<sup>_j−τ_</sup> _c_ �2 for the MSE, where _E_ ( _._ )denotes expectation under model 2.

9
# **3.1 Results**

The section presents the results from the model-based simulation experiments where the goal is to compare the performance of the different methods. Marhuenda et al. (2017) consider multiple scenarios where they simulate different values for _σa_<sup>2and</sup><sup>_σ_</sup> _ac_<sup>2.Theauthorsnotewhatmattersare</sup> the relative values. In this instance the interest is to assess how results differ when the random cluster effect is considerably smaller than the random area effect and when the random cluster effect is considerably larger than the random area effect. ELL would commonly specify its random location effect at the cluster level and then aggregate results to the area level. Consequently, we expect ELL to perform better when the random cluster effect is larger than the area random effect. We consider two scenarios: Simulation results under the two considered scenarios are presented respectively in Figures 1 and 3 for bias, and Figures 2 and 4 for MSE.

The results for bias and MSE are presented at the area level. With the exception of a few areas in which the ELL and unit-context methods demonstrate a slightly higher bias, all examined methods perform better than the direct estimators, in most cases by a substantial margin. For MSE, the result varies between the two simulation scenarios. Under scenario 1 with results shown in Figure 2, where the variance of cluster effects is double that of the area effects, the methods considered, including ELL and the unit-context methods, perform relatively well and in nearly all cases now do as well as or better than direct estimates, though again ELL and the unit-context methods perform worse than the other options.<sup>16</sup> However, in Figure 4, where the variance of the area effects is now much larger, ELL in particular performs poorly in terms of MSE likely due to the error misspecification and the contextual variables not explaining sufficiently the variability of the area effects. To a lesser extent a similar effect can be observed for the CensusEB estimator, based on a model with only cluster effects and contextual variables ( _CEBc_ with context), which performs well in terms of MSE under scenario 1 but under-performs in scenario 2.

The twofold model results are aligned to the results presented by Marhuenda et al. (2017); the bias and MSE of estimates obtained under twofold fitting and onefold CensusEB fitting at the area level

> 16Despite the result, other issues from the implementation of ELL (2003) noted by CMN (2020) like the underestimation of the MSE still remain, unless the method to estimate MSE is adjusted.

10
<!-- Start of picture text --> ©<br>++<br>+<br>+<br>S<br>oO :e +<br>x N bd e* °° ° + + +<br>oS eo 6 eo ° 4 e eo & efe °* 6<br>o : ary °° ° °° ee °<br>rea) x ~~ Oy + x + » O@txxt,*O © *<br>eGGuendguntatsegnteaeoas lg inagnneaeenaesae<br>a e@ + * x °* x otx +,°4<br>+ “<br>iN + + +<br>+ +<br>+<br>+<br>+++<br>010 20 30 40<br>Area<br>+ Direct © CEBac 4 CEB, o CEB.<br>« ELL. e UC-CEB. ¢ UC-CEBa<br><!-- End of picture text -->
<!-- Start of picture text --> oO<br>vt<br>++<br>SeoO *<br>oO<br>x + ++<br>ofO +<br>© i * -<br>ta at + * 4 ° + + +<br>g geet Hetty cette te ae tet terest<br>= Ge e @ ° e e e Ce) ee ¢<br>SaegtessSabeuehase oesacevetsyesesasess<br>oO<br>0 10 20 30 40<br>Area<br>+ Direct © CEBac 4 CEB, o CEB.<br>« ELL. e UC-CEB. ¢* UC-CEBa<br>(ee)<br>++<br>+<br>+ ry<br>= e ? ee + +<br>= ° e °<br>XN + $ e 6 be<br>e e e 9 + ° ‘ bd +<br>rd o .o ¢° x + e+* t+, ° 06 °<br>—) sab asetnete ita Scagechidgnataniuteagntane<br>oO + * * + x<br>a e e ° “y t+ +<br>N + ty e@ + ° e<br>+ +<br>+<br>+<br>= +<br>y<br>0 10 20 30 40<br>Area<br>+ Direct © CEBac 4 CEB, o CEB.<br>« ELL. e UC-CEB. ¢* UC-CEBa<br><!-- End of picture text -->
<!-- Start of picture text --> oO<br>+<br>++<br>So + .<br>6° x * x x * * x x" **<br>S * * xx" se a * ne x x” x x x * x *<br>x< * *<br>> x + + +<br>LeLu + e |<br>+og[,a oo79O, +090,500,579 5%, non Boot O797,0<br>++ + @ + +4 a +<br>+ + ++ $ + eo +t getetot+ *<br>S . 4-2 e? * et® ° o. ° e ° e oe ° @<br>PEPereer rr eer Pe rewe fier Coed Peer es Serre.<br>0 10 20 30 40<br>Area<br>+ Direct © CEBac 4 CEBa Oo CEB.<br>« ELL. @ UC-CEB.z ¢* UC-CEBac<br>UC-CEB,<br>e<br>N<br>ee<br>ee e<br>e<br>DH e e<br>3 e<br>_—o e e<br>©><br>ce} tb e<br>Or<br>xo ! e<br>2 ° e<br>a]<br>= e<br>~ @ ad . °° e ° e<br>@<br>e e<br>°?<br>-2 0 2 4 6 8<br>Xach - MEAN Xacn by cluster<br>@ hhres ——— Fitted values<br><!-- End of picture text -->
Table 1: Aggregate results across areas (FGT0)

||1|2|3|4|
|---|---|---|---|---|
|Model _R_<sup>2</sup>|~0.42|~0.42|~0.42|~0.42|
|_σa_|0.05|0.1|0.05|0.1|
|_σac_|0.1|0.05|0.1|0.05|
|_σe_|0.5|0.5|0.5|0.5|
|Sample per cluster|10 per all clusters|10 per all clusters|10 per 20% clusters|10 per 20% clusters|
|Population|20,000|20,000|100,000|100,000|
|**Direct**|||||
|AAB (_×_100)|1.780|1.783|1.802|1.795|
|ARMSE (_×_100)|4.039|4.041|4.396|4.563|
|**_CEBac_**|||||
|AAB (_×_100)|0.020|0.023|0.018|0.023|
|ARMSE (_×_100)|2.630|2.859|2.493|2.191|
|**_CEBa_**|||||
|AAB (_×_100)|0.020|0.024|0.021|0.017|
|ARMSE (_×_100)|2.642|2.876|2.515|2.237|
|**_CEBc_**|||||
|AAB (_×_100)|0.024|0.029|0.045|0.020|
|ARMSE (_×_100)|2.793|3.885|4.731|2.614|
|**_ELLc_**|||||
|AAB (_×_100)|0.238|0.236|0.144|0.139|
|ARMSE (_×_100)|3.433|5.269|5.048|2.768|
|**_UC −CEBa_**|||||
|AAB (_×_100)|1.115|1.362|1.296|0.986|
|ARMSE (_×_100)|3.074|3.404|3.068|2.584|
|**_UC −CEBac_**|||||
|AAB (_×_100)|1.106|1.370|1.272|0.929|
|ARMSE (_×_100)|3.059|3.383|3.046|2.591|

AAB: Average absolute bias; ARMSE: Average root MSE 14
bias.<sup>20</sup> This misspecification is manifested as failure of the linearity assumption and the magnitude of this bias depends on the form of the covariates (see Figure 5 and appendix). Additionally, it will vary for each type of indicator. For example, when estimating mean income in a model without log transformation, linearity is preserved when aggregating and thus the approach is expected to perform well.<sup>21</sup> However, the empirical MSEs for the unit-context models in most areas outperform those from ELL (see Table 1 for average results). In fact, under scenario 1 (Figure 2) in some areas the MSE for the unit-context models are only slightly worse than those of the _CEBa_ and _CEBac_ . Under scenario 2, where the area effects have a higher variance relative to that of the cluster effect, which is likely the case in real world scenarios, the empirical MSE for the unit-context models show considerable variability for both fitting models. The results suggest the method could outperform direct estimates as well as misspecified models with random effects for the area level only, in terms of MSE, and may be an alternative under scenarios where census and survey data are not aligned. However, bias in these unit-context models may be a considerable concern. As one can see in the results from Table 1 the unit context method yields FGT0 estimators with an average absolute bias that is 56 times larger than the bias of the CensusEB estimators, and almost 5 times larger than ELL’s average absolute bias.<sup>22</sup> In the sections that follow this is further explored.

# **4 Design-based simulation**

In this section we present a design-based simulation experiment based on the Mexican Intra Censal Survey of 2015 (Encuesta Intracensal). The purpose of the simulation is to observe performance of the different methods under more realistic scenarios. The survey is carried out by the Mexican National Institute of Statistics and Geography (Instituto Nacional de Estadistica y Geografia - INEGI). The survey has a sample of 5.9 million households and is representative at the national, state (32 states) and municipal or delegation level (2,457 municipalities), as well as for localities with a population of 50,000 or more inhabitants.

The 2015 Intra Censal Survey questionnaire includes the following topics related to the housing unit: dwelling features, size and use of the dwelling, conditions for cooking, ownership and access conditions, access to water, sanitation facilities and sanitation, electric power, solid waste, equipment, appliances and automobile; and information and communication technologies (ICT). It also includes the following demographic information: total population and structure, birth registration, marital status, health services, ethnicity, education, economic characteristics, nonpaid work, migration, daily mobility, fertility and mortality, household composition, non-labor household income,

> 20  ̄ The correlation is also present between _x_ 5 _ac_ and other omitted deviations, _zach_ .

> 21The empirical average relative root MSE of the mean for the UC approach is slightly higher than that of the CensusEB.

> 22The unit context method also yields mean welfare estimators that are 38 times more biased than Census EB variants (not shown).

15
food security, agricultural land use, relationship to the household head, indigenous language, occupation, economic activity, and accumulated education. The survey also includes indicators for states, municipalities, and counties.

One of the key features of the survey is its size and the fact that it includes a measure of income at the household level.<sup>23</sup> The inclusion of an income measure allows for a design-based validation of the different methods presented above. The next section describes how the survey is modified to create a census data set and how samples are then drawn from the created census data with the goal of obtaining small area estimates of poverty at the municipal level.

# **4.1 Creating a census and survey**

Because the goal of this exercise is to test how well the different methods perform under a real world scenario, the Intra Censal Survey is modified to mimic a Census in order to allow for a design-based simulation. The first step consists of randomly removing 90 percent of households that reported an income of 0. This is done to ensure that some households with an income of 0 are included in the population, but not as many as in the original data to make it more realistic.<sup>24</sup> In the second step, all municipalities with less than 500 households are removed, thus observations by municipality range from 501 to 23,513.<sup>25</sup> The final “Census” consists of 3.9 million households and 1,865 municipalities.

To draw survey samples, primary sampling units (PSU) need to be created.<sup>26</sup> Within each municipality, the original data’s PSUs are sorted<sup>27</sup> and joined until each created PSU has close to 300 households. Under the proposed approach, original PSUs are never split, just joined to others. Additionally, all original PSUs that were larger than 300 households are designated as a created PSU. The entire process yields 16,297 PSUs.

The resulting Census data is used to draw 500 survey samples to conduct a design-based simulation to establish how a method will perform over repeated sampling from a finite population (Tzavidis et al., 2018). The sampling approach reflects standard designs used in most face-toface household surveys conducted in the developing world, such as those conducted by the Living Standards Measurement Study (LSMS) program of the World Bank (Grosh and Muñoz, 1996), with some simplification for convenience. First, the 32 states that comprise Mexico are treated as the intended domains of the sample. The main indicator of interest for the survey is the welfare measure: household per capita income. The desired level of precision to be achieved in the sample is assumed to be a relative standard error ( _RSE_ ) of 7.5 percent for mean per capita income

> 23Income is defined as money received from work performed during the course of the reference week by individuals of age 12 or older within the household.

> 24Welfare values of 0 and/or missing are a common feature of household surveys.

> 25The median municipality has 1,613 households.

> 26The Intra Censal Survey has its own PSUs, however many of these have too few observations to properly work as PSUs in the created “Census” data.

> 27This assumes that the clusters’ numbering is tied to how proximate each cluster is to one and other.

16
in each state.<sup>28</sup> A two-stage clustered design is assumed with clusters (defined above) serving as primary sampling units (PSUs) selected in the first stage within each domain and then a sample of 10 households within each cluster is selected in the second stage. With these design features established, the trimmed census data was analyzed to identify the parameter estimates for per capita income (mean and standard deviation) for each state and the target standard error implied by the parameter estimates that corresponds to an _RSE_ of 7.5 percent. The minimum sample size required for state _s_ , given these parameters, under simple random sampling (SRS) design is then obtained as: where _σs_ is the standard deviation of per capita income in state _s_ , _RSE_<sup>_tgt_</sup> is the target standard error of 7.5 percent in state _s_ , and _y_ ̄ _c_ is the mean per capita income in state _s_ . The minimum sample size under SRS must then be adjusted to account for the clustered design. This design effect due to clustering is accounted for by estimating the intra-cluster correlation ( _ICC_ ) for per capita income within each state using the trimmed census data. The _ICC_ estimates can then be applied to the SRS size obtained above to arrive at the minimum sample size for state _s_ under the clustered design employed here, given by: where _DEFFs_ is the design effect in state _s_ , _npsu_ is the number of households selected within each cluster (10 in this case), and _ρs_ is the _ICC_ for per capita income in state _s_ . The minimum number of clusters to achieve _ns_ (assuming 10 households per cluster) was calculated and then the final (household) sample size established by multiplying the number of clusters by 10.<sup>29</sup> Taking the sample size requirements from above as fixed, the sample in each simulation is selected in accordance with the two-stage design. PSUs within each state, referred to here as clusters, are selected with probability proportional to size ( _PPS_ ) without replacement, where the measure of size is the number of households within the cluster. Then 10 households were selected within each cluster via simple random sampling. According to this design, the inclusion probability for household _h_ in cluster _c_ and in state _s_ is approximated<sup>30</sup> as:

> 28The desired precision of an RSE of 7.5 percent is somewhat arbitrary but corresponds to precision targets in similar surveys and yields a sample of reasonable size.

> 29Full estimates and results for the sample size determination are available upon request.

> 30The equation used to calculate inclusion probabilities assumes sampling with replacement but is used here as an approximation of inclusion probabilities under PPS selection without replacement. This should provide a reasonable approximation in this case since there are a relatively large number of PSUs present in the frame.

17
where _Ns_ is the total number of households in the census for state _s_ , _Nsc_ is the number of households in cluster _c_ from state _s_ , _Cs_ is the number of clusters selected in state _s_ , and _nsc_ is the number of households selected in cluster _c_ within state _s_ , which is fixed at 10.<sup>31</sup> The sample size across the 500 samples is roughly 23,540. Under the proposed sampling scenario, not all municipalities are included, and the number of municipalities included varies from sample to sample, ranging between 951 to 1,020 municipalities. The median municipality included in a given sample, is represented by a sole PSU and thus its sample size is of 10 households.

# **4.2 Model selection**

Model selection is conducted using the first sample drawn from the scenario detailed in the previous section. The target variable is household per capita income. However, this variable is highly skewed and to achieve an approximately normal distribution we test three transformations: i) natural logarithm,<sup>32</sup> ii) log-shift transformation, and iii) Box-Cox transformation of the natural logarithm.<sup>33</sup> As one can see in Figures 6, 7 and 8, the Box-Cox transformation as well as the log shift fix the skewness in the distribution of model residuals that appears after taking the natural logarithm of per capita income.<sup>34</sup> The goal of the model selection process is to arrive at a model that only includes stable covariates. Under each transformation, model selection is done using a least absolute shrinkage and selection operator, commonly known as lasso,<sup>35</sup> where the candidates for covariates include household characteristics and characteristics at the PSU, municipal and state level. Two models are selected: i) a model that includes household characteristics and characteristics at the PSU, municipal, and state levels and ii) another model that only includes characteristics at the PSU, municipal and state levels. The second model is used for the unit-context approach. All household level characteristics that are included also as aggregates at the PSU, municipality, and state levels have been previously standardized to ensure that these have mean 0 and standard deviation 1 for each PSU, municipality and state respectively. Note that aggregated covariates have been obtained from the “census,” thus, the aggregated covariates will be the same within the sample and the census. The lasso model selection process applied here ignores the error structure presented in (1) and (2) and does not ensure that selected covariates will be significant under the assumed models. Consequently, after the initial lasso model selection, model (1) is fit using Henderson’s Method III (with sampling weights) and random effects specified at the municipality level. Because the resulting model may

> 31The design weight for each household is simply the inverse of the inclusion probability. In a typical survey, the design weights would be further adjusted for nonresponse and calibrated to known population characteristics. However, since the sampling is only a simulation exercise, there is no nonresponse and thus no nonresponse adjustment is required. Calibration or post-stratification could be performed but was not implemented to simplify the process.

> 32In any given sample, roughly 11 observations have an income of 0, these are assigned an income of 1 prior to transformation.

> 33For more details on transformations, see Tzavidis et al. (2018).

> 34Figures make use of sample from a two-stage clustered design.

> 35Model is selected using 20 fold cross validation and shrinkage parameter ( _λ_ ) that is within 1 standard error of the one that minimizes the cross validated MSE.

18
<!-- Start of picture text --> ©<br>v.<br>ra<br>O<br>Cc<br>©<br>a<br>N<br>oO<br>-10 -5 0 5<br>Household level residuals<br><!-- End of picture text --> <!-- Start of picture text --> ce)<br>©<br>Pa<br>a<br>6TM.<br>a<br>N<br>oO<br>-4 -2 0 2 4<br>Household level residuals<br><!-- End of picture text -->
<!-- Start of picture text --> S<br>2<br>r=<br>g<br>oN<br>[m)<br>-10 -5 0 5 10<br>Household level residuals<br>o é vt<br>@<br>e<br>23<br>@<br>is=—= isoO=geoO<br>g5<br>2 &<br>5 Ke}<br>ow 8<br>55é<br>3501 @<br>e<br>e<br>2 +<br>-4 -2 0 2 4 -1 -.5 0 5 1<br>Theoretical quantiles Theoretical quantiles<br><!-- End of picture text -->
<!-- Start of picture text --> e e<br>+ e aid<br>@<br>2 an<br>n 2<br>88<br>@3 orae)<br>=Zo =3<br>[0 2<br>nana CcSa<br>2E<br>3 So<br>8 s<br>Ea Fe<br>fei ie}<br>©<br>e<br>+e<br>ee-1 e-<br>-4 -2 0 2 4 -1 -5 0 5 1<br>Theoretical quantiles Theoretical quantiles<br>=] o<br>e<br>e<br>e<br>N<br>LW 7)<br>© o<br>3 3 °<br>8 8<br>30 BH<br>xs}<br>g5<br>2 Eo<br>5 Ke}<br>8 s<br>Ze E<br>gs’ 5 _<br>SG Co!<br>é<br>2 %<br>TT 1e<br>ee<br>4-2 0 2 4 -2 -1 0 1 2<br>Theoretical quantiles Theoretical quantiles<br><!-- End of picture text -->
still include covariates that are not significant under the fitted model, all non-significant (at the 1% level) covariates are removed sequentially.<sup>36</sup> Finally, we check for multicollinearity and remove covariates with a variance inflation factor (VIF) above 3.

Fitted models using the first sample and the described model selection are presented in Tables 3 and 4 in the appendix. Since in small area estimation models are used for prediction, assessing the predictive power of the model is important. This may be measured using the coefficient of determination, _R_<sup>2</sup> (Tzavidis et al., 2018). For the household level model used in this exercise the _R_<sup>2</sup> is close to 0.45 while that of the unit-context model centers around 0.25.

Figures 9, 10, and 11 represent checks on the normality assumptions using normal _Q-Q_ -plots of household level residuals and estimated area effects for the onefold unit level model with random municipality location effects. The resulting plots for the unit-context models can be seen in the appendix figures 26, 27, and 28. The natural logarithm transformation (Fig. 9) presents evidence of deviation from normality. Marhuenda et al. (2017) notes in applications that use real data the exact fit to a distribution is barely met; however the Box-Cox and log-shift transformations provide considerably better normal approximations (see Figures 10 and 11).<sup>37</sup>

# **4.3 Results**

Once the models to be used have been selected and _L_ = 500 samples under two-stage sampling described in section 4.1 have been taken from the “census” created using the Mexican Intra Censal Survey of 2015 we obtain estimates using the different considered models. The target parameters for this simulation are mean welfare and the Foster et al. (1984) - FGT class of decomposable poverty measures for municipalities present in the census. The true values of the target parameters at the municipal level are based on the census data.

As a first step, we compare results using the three transformations discussed in the previous section. Figures 12 and 13 show box plots of empirical absolute relative bias and MSE of CensusEB estimates of poverty rates based on model (1) under each transformation. The Box-Cox and log-shift transformation yield estimates that are not only less biased, but also results with a lower empirical design MSE. As can be seen in the results from the first 3 columns in Table 2 the log-shift transformation yields aggregate results somewhat preferable to the Box-Cox. Additionally, in Table 2 it is quite clear that the natural logarithm may actually result in aggregate results that present considerably smaller gains over direct ones and thus model and residual checks should always be done to ensure adequate transformations are used to avoid such an outcome. The rest of the discussion will focus on estimates obtained from the log-shift transformation applied to the two-stage samples.<sup>38</sup>

> 36The removal is done sequentially starting with the least significant covariate, then the model is fit again without the covariate. The process is repeated until all covariates in the model are significant at the 1% level.

> 37 Box-Cox and log-shift transformations are available under R’s `sae` package as well as under Stata’s `sae` package. In the World Bank’s `PovMap` software, the only available transformation was natural logarithm. However, `PovMap` does allow for drawing residuals from the empirical distribution as well as from a Student’s _t_ distribution.

> 38Figures under other transformations are available upon request.

22
<!-- Start of picture text --> Box-Cox transformation (log) — ||| - e e<br>Nat log shift transformation — ||| - e e<br>Nat log transformation | @— || - e@ e0e e<br>-20 0 20 40 60<br>Design bias at municipal level<br><!-- End of picture text -->

<!-- Start of picture text --> Box-Cox transformation (log) |— e e<br>Nat log shift transformation |— e e<br>Nat log transformation {++ ee 8@ e<br>0 10 20 30 40<br>MSE at municipal level (x100)<br><!-- End of picture text -->
We consider estimators based on onefold and twofold nested error models, including unit-context (UC) models with only aggregated covariates, and direct estimators. Concretely, we consider the following:<sup>39</sup>

# **Direct Estimates**

1. Direct estimates from the survey samples for each municipality. These are calculated with weights from the considered design. Specifically, the calculation of FGT indicators under the two-stage design detailed in subsection 4.1 using the inclusion probabilities is obtained as:

- _τ_ ˆ _a_<sup>_direct_</sup> =<sup>�</sup> _h_ _<u>pach</u>_ <u>�</u> _pach_<sup>_Fach_;</sup>

where, _pach_ is the inclusion probability for household _h_ in cluster _c_ and in municipality _a_ , and _Fach_ is the FGT or welfare measure of interest for household _h_ in cluster _c_ and in municipality _a_ .<sup>40</sup>

# **Unit-level models:**

1. _CEBa_ : Model fit is done using Henderson’s Method III (with sampling weights) and estimates are obtained using CensusEB as noted in CMN (2020). The fitted model reads:

- _yach_ = _xachβ_ + _zacα_ + _taω_ + _gsλ_ + _ηa_ + _εach_ ;

- where _xach_ is a vector of household specific characteristics, _zac_ contains cluster level characteristics, _ta_ includes municipality level characteristics and _gs_ is composed of state level characteristics. The random effects, _ηa_ , are specified at the municipal level.

2. _CEBc_ : Model fit is done using Henderson’s Method III (with sampling weights) and estimates are obtained using Census EB as noted in CMN (2020). The difference with model (1) is that random effects are specified at the PSU level. That is, the fitted model is:

- _yach_ = _xachβ_ + _zacα_ + _taω_ + _gsλ_ + _ηac_ + _εach_ ;

where, _ηac_ , is a random effect for cluster _c_ within municipality _a_ .

3. _CEBac_ : Model fit is done using REML and estimates are obtained using CensusEB as noted in Marhuenda et al. (2017). Fitted model follows:<sup>41</sup>

- (a) _yach_ = _xachβ_ + _zacα_ + _taω_ + _gsλ_ + _ηa_ + _ηac_ + _εach_ ; random effects are specified at the municipality and PSU level.

4. _CEBsa_ : Similar to 4, however random effects are specified at the municipal and state level. The goal here is to borrow strength from the state for municipalities that are not in the sample which takes a cue from Marhuenda et al. (2017).

> 39Every model includes a constant term.

> 40Note that depending on the sampling strategy, it is likely that we do not have direct estimates for all areas.

> 41All twofold nested error methods are fit without the use of probability sampling weights and are thus not comparable to those that use survey weights.

24
5. _ELLc_ : under the same model as in 2. In cases where we use a transformation different from the natural logarithm, random location effects and household residuals are drawn from their empirical distribution.

# **Unit-context models:**

1. _UC − CEBa_ : Unit-context model originally proposed by Nguyen (2012) but with EB, similar to Masaki et al. (2020). Model fit is done using Henderson’s Method III (with sampling weights) and estimates are obtained using CensusEB as noted in CMN (2020). The fitted model follows:

- _yach_ = _zacα_ + _taω_ + _gsλ_ + _ηa_ + _εach_ .

2. _UC − CEBac_ : Unit-context model fit is done using REML and estimates are obtained using CensusEB as noted in Marhuenda et al. (2017). The fitted model is:

- _yach_ = _zacα_ + _taω_ + _gsλ_ + _ηa_ + _ηac_ + _εach_ ;

3. _UC − CEBsa_ : Similar to model 8, however random effects are specified at the municipal and state levels. Just like 5, the goal here is to borrow strength from the state for municipalities that are not in the sample.

4. _UC − ELLc_ : ELL estimates under model in _UC − CEBa_ but random effects specified at the PSU level, as was originally proposed by Nguyen (2012) and then by Lange et al. (2018). In cases where we use a transformation different from the natural logarithm, random location effects and household residuals are drawn from their empirical distribution.

- _yach_ = _zacα_ + _taω_ + _gsλ_ + _ηac_ + _εach_ .

The chosen measures to evaluate performance of the considered predictors are bias, MSE, and root MSE obtained as: where _j_ stands for one of the methods: _CEBa, CEBc, direct, CEBac, CEBsa, ELLc, UC − CEBa, UC − CEBac, UC − CEBsa,_ and _UC − ELLc_ ; _τa_ denotes the true population parameter 25
<!-- Start of picture text --> CEB..| HiiZj-—useesmese @me 0 ©@ e<br>CEB,| Hiiij-—amuuumeeeeen@ @oesmoem =o e<br>CEB,| Hiiiij-—<enummananmessnapememee e@ 0 © © e<br>CEB.,| Hiiiij-—sneuezpems@ec m m eo ee ee<br>Direct | +———-HE.--——tmmocemoce )<br>ELL.| Hiiiij-———caueummcemencmp @ comme © emcee co @ e<br>UC-CEB..| iij-——suGEEEETO @e0m coe © @ ©<br>UC-CEB, | Hijiii—ueeummcmm eee Cf ©<br>UC-CEB,,| HiZ=Zij——uEEEEGRONEDCe e@ 0 eo<br>UC-ELL.| (iijij-—amueeseemCGR coca 08 oe Oo e<br>° : MSE at municipal level (x100) ° 0<br>Values are trimmed at 10<br><!-- End of picture text -->
Table 2: Aggregate results for 1,865 municipalities in “Census” (FGT0) - Results from 500 samples

||1|2|3|4|
|---|---|---|---|---|
|Transformation|Box-Cox|Nat. log.|Log. Shift|Log. Shift|
|**Direct**|||||
|AAB (_×_100)|11.314|11.314|11.314|9.722|
|ARMSE (_×_100)|14.051|14.051|14.051|11.997|
|**_CEBac_**|||||
|AAB (_×_100)|6.277|8.642|6.273|5.393|
|ARMSE (_×_100)|6.580|9.169|6.574|5.964|
|**_CEBsa_**|||||
|AAB (_×_100)|6.382|8.953|6.380|5.740|
|ARMSE (_×_100)|6.695|9.414|6.687|5.997|
|**_CEBa_**|||||
|AAB (_×_100)|6.080|8.800|6.092|5.395|
|ARMSE (_×_100)|6.584|9.525|6.589|6.034|
|**_CEBc_**|||||
|AAB (_×_100)|6.253|8.690|6.277|6.054|
|ARMSE (_×_100)|6.363|8.847|6.384|6.181|
|**_ELLc_**|||||
|AAB (_×_100)|6.685|8.781|6.685|6.961|
|ARMSE (_×_100)|6.820|8.854|6.820|7.087|
|**_UC −CEBa_**|||||
|AAB (_×_100)|6.171|7.531|6.016|5.982|
|ARMSE (_×_100)|6.607|8.274|6.461|6.672|
|**_UC −CEBac_**|||||
|AAB (_×_100)|6.121|7.596|5.998|5.926|
|ARMSE (_×_100)|6.446|8.186|6.332|6.652|
|**_UC −ELLc_**|||||
|AAB (_×_100)|6.250|8.019|6.250|7.581|
|ARMSE (_×_100)|6.421|8.100|6.421|7.758|
|**_UC −CEBsa_**|||||
|AAB (_×_100)|6.002|7.539|5.875|5.937|
|ARMSE (_×_100)|6.414|8.284|6.294|6.570|

AAB: Average absolute bias; ARMSE: Average root MSE ELL under Box-Cox and log shift, model is still for Nat. log. but errors are drawn from the empirical distribution. Columns 1, 2, and 3 under two-stage samples (Section 4.1) Column 4 results are under 1% SRS by PSU, all PSUs are included 27
model-based estimators are achieved at the expense of design bias. As an additional check, we add simulations where _L_ = 500 samples are taken, each consisting of a 1% SRS without replacement in every PSU within the fixed census population. Aggregate results for this scenario are presented in column 4 of Table 2 and box plots for MSE and bias are presented in appendix Figures 29 and 30, respectively. The first thing to note in these figures is there are fewer extreme outliers when compared to results from the two-stage sample scenario in Figure 31 in the appendix. Despite fewer outliers, the results under the additional sampling scenario mimic those from two-stage samples. However, under the two-stage sampling used here, direct estimates for most municipalities are not available across all 500 samples (see subsection 4.1). Consequently, direct estimates are not included under the remaining figures that discuss results from the two-stage samples.

Under the simulation experiment with two-stage sampling, the empirical MSE of _ELLc_ estimates, where the location effect has been specified at the PSU level, appear to have a tighter spread than direct estimates (Fig. 14). Consequently, though ELL appears to perform relatively well, relative to direct estimates, the number of ELL outliers with a high MSE is considerable and _ELLc_ performs worse than all other small area approaches. This result was not expected, as the FGT0 results for _UC − ELLc_ appear to perform better than the traditional _ELLc_ which has a considerably better model fit.<sup>44</sup> Nevertheless, _UC − ELLc_ does very poorly on mean welfare where it has an MSE that is as large as that of direct estimates and is also considerably biased (see appendix Table 7). These results also hold under the simulation experiment with the 1% SRS by PSU samples.

As expected, _CEBa_ estimates show a considerably tighter MSE spread than direct estimates, but still with outliers. Another interesting finding is that under CensusEB but with random effects specified at the PSU level ( _CEBc_ ), the empirical MSE is tighter than that of the direct estimates and also displays better properties than _ELLc_ as seen in Table 2. However, given the discussion in the model-based validation and the results shown here, the results with the random effects at the municipality level are preferred over the ones where the effect is specified at the PSU level. Perhaps the most surprising result is the low MSE of the _UC − CEBa_ method with only contextual variables and the other _UC − CEB_ variants, as proposed by Masaki et al. (2020). The results display a tight spread and the results from Table 2 corroborate the finding. Nevertheless, beyond _UC_ models being an alternative if contemporaneous census data is not available, the method presents a couple of advantages over traditional Fay Herriot (FH) area level models under sampling scenarios that follow a two-stage design like the one considered here. First, as noted toward the end of section 4.1, the majority of municipalities are represented by 1 PSU and thus have quite small sample sizes which makes the likelihood of direct FGT0 estimates being equal to 0 or 1 much higher. Under these cases, the method from Nguyen (2012) is a valid alternative to FH because FH is not applicable in these municipalities with a sampling variance of 0. An additional advantage is that the model can be used for multiple indicators whereas the FH requires a model for each indicator considered.

> 44Note that under the MI inspired bootstrap of ELL, a very poor model fit will be heavily penalized and will likely yield noise estimates for _UC − ELLc_ that fare worse than direct estimates in most applications.

28
A possible explanation for the good performance of unit-context models under the considered experiments could be due to estimated random effects, where _σ_ ˆ _ac_<sup>2</sup><sup>_>σ_ˆ</sup> _a_<sup>2,whichcoincideswiththe</sup> better performing scenario in subsection 3.1. Although this observation is likely a coincidence since the performance of unit-context models depends on the particular covariates included in the model and on the shape of the target indicator. Under the Mexican data with the covariates used, the unit-context model performs rather well. However, the method considerably lags all others for estimation of the welfare mean by area. A similar result is observed in the results presented in subsection 3.1.<sup>45</sup> Under the simulations of column 3 of Table 2, the gains in MSE for the unit-context methods appear to come in municipalities with larger populations.<sup>46</sup> This is expected because in our sampling scenarios, municipalities with a larger population are more likely to be included in all the samples. For example, in the census there are 16,293 PSUs spread over 1,865 municipalities. The median municipality in the created census has 7 PSUs, and the median municipality in a given two-stage sample is only represented by one PSU. Consequently, it is not surprising the unit-context variants under-perform relative to unit models in terms of bias and MSE in municipalities with smaller populations, as can be observed in Figure 16. For municipalities with larger populations, and hence those likely to consist of more PSUs and more of these PSUs in the sample, direct estimates for FGT0 are more precise and unit-context models begin to catch up to unit models (see Fig. 32 in the appendix).

Under the simulations from a 1% SRS by PSU, shown in Table 2, column 4 and in the appendix Figures 29 and 30, we notice the unit models ( _CEBa_ ) present a slight upward bias that seems to become more pronounced in more populous municipalities (Fig. 17). Notice that some bias is acceptable since gains in MSE are achieved at the expense of bias, however in this case the presence of outliers seems to lead to increased upward bias that affects more municipalities as we move to more populous deciles (box-plots for _CEBa_ in Fig. 17). Unit-context ( _UC − CEBa_ ) models on the other hand, appear to have a downward bias. In the box-plots in Figure 17 for _UC − CEBa_ , the bias in lower deciles is downward and as we move to upper deciles the downward bias is considerably reduced.

As noted in section 3.1, the problem faced by unit-context models is omitted variable bias manifested as a lack of linearity (see Fig. 18).<sup>47</sup> Figure 19 further presents the issue. Under _CEBa_ residuals appear to follow a random pattern, however under _UC − CEBa_ households in municipalities represented by only one PSU in the sample will all have the same linear fit. This manifests itself in the figures for _UC − CEBa_ as a column of vertical dots. The problem can also be observed

> 45See Table 7 in the appendix.

> 46Similar findings are obtained under the simulations for column 1.

> 47The true model includes household size at the household level. Consequently, it is a determinant of the dependent  ̄ variable. Household size at the household level ( _xach_ ) can be broken down into _zach−xac_ , thus the omitted component, _zach_ , is also a determinant of the dependent variable. The unit-context model only includes the PSU average household size ( ̄ _xac_ ) obtained from the census as a covariate. When a given survey sample is taken, the non-included covariate

> ( _zach_ ) is correlated with the PSU average household size and the dependent variable.

29
<!-- Start of picture text --> CEBac coomm—___——(i]—‘a» °<br>CEBa conm—_ j}———e80 e r<br>CEB. ecoumn| |} — —emmmm_{ oe e<br>CEBsa ounm—| | }#——@eocoe fe)<br>Direct eump-||-me<br>ELL. © coemayp—_—#i] eum 000 0 0 e<br>UC-CEBac eocun___——_]——‘weewe e<br>UC-CEB.| ##@———/]i#—taece °<br>UC-CEBsa oom —__—_§ji]/_—w eee e<br>UC-ELL,.| @ ess] —tresmoce cc e<br>-20 0 20 40 60 80<br>Design bias at municipal level (x100)<br>9 +<br>ron : ° r<br>aga$=~<br>A 8 $ 3 g<br>©<br>te Ro<br>a $ 4 °*<br>i a<br>Zn -<br>C<br>0 2 4 6 8 10<br>Municipal population deciles<br>° CEBac © CEBsa 4 CEBa 5 CEB. + ELL.<br>4 UC-CEBa ° UC-CEBa ® UC-CEBsa ®" UC-ELLc<br><!-- End of picture text -->
<!-- Start of picture text --> CEB; Hetero UC-CEB.<br>4 +—_{_T. +“ 1 -—{_T }<br>2 +——{lJ*_ ewe e 2 +———_{ijiag}#———<br>3 t+—_{_T } 4H e 3 © [+#' —“<br>4 +—fil— “ HH —__] }#———4<br>5 ee—{ |} 5 e +—_{ijia#—__<br>6 e |—{jj} toe 6 ee |+—{f— e<br>7 e'—ii 7 -—_fiia—<br>8 +—{jjwee 8 eo ol e<br>9 e:_i Ve 9 eos—}—_<br>10 e |{j= 10 e eee He<br>-40 -20 0 20 40 -40 -20 0 20 40<br>Design bias at municipal level (x100)<br><!-- End of picture text -->

<!-- Start of picture text --> UC-CEBaz<br>@®<br>+ ee<br>% e<br>oN ee e? e<br>8 “S 9 ee<br>8 ee"<br>Lo ee “ee ; ;<br>ie) @<br>2 o*s °<br>B @ 08 °<br>Bey o €<br>r= eR ace °<br>“ee e<br>-, & “e<br>Yt ° e<br>-5 0 5 10 15 20<br>Household size - mean household size by PSU<br>@ hhres ———W Fitted values<br><!-- End of picture text -->
<!-- Start of picture text --> CEBa UC-CEBa<br>vt +<br>e@<br>ee<br>e e e° e . oeone > e<br>a |° % ee the e n e @ ee tps S o° ° e<br>»<br>35 e %%e %e eeyee ‘ 3 ec? pe:,$PR,% © 23 ° gee© °§$ aspee yt.“1S Oy8 ee<br>£® *ee cages . no yey $. ry3 - & =f Ooaaa4 .wor<br>$ e Yn, ( le £ fay: e<br>rr) e bd Cad re 4 oe e@ g e ~ -. ee<br>2 e -¢ 72 . e Bo . Y. %e<br>2© eee© S88 ? . rer. v 5 * 22 )6O 8 9 8k Se~ ms r e33<br>E3 0° e o® ayORDEREDNet 6 ere *. ° e o3 s ¥ +o.3°C.aea3 * ry<br>2 0 tS ee AE ena BS 8 3 e SU EAPUIES “cs %. 0 °<br>% ° og ote°oa aye5 ce @° e x= e 88@ &% a J aa °F ee68<br>e % * 000%oe5 ®0 e ceeOB ° ° 9 o (oe%,° eeee oseo? Pe 00, ° °<br>e . e<br>YT YT<br>5 6 7 8 9 5 6 7 8 9<br>XB linear fit XB linear fit<br>Only includes municipalities represented by one PSU<br>CEB. UC-CEBa<br>22<br>.ee ,<br>. a hoe ° i<br>ct te Letheetth, le b8§ uh Soh tae oma “<br>0 : ae . nite ee i i PERE at + sed Poitia tay . . y<br>ar RASS 3s % paeet: Tree: Wead aa ae 0 “3 3 as<br>= Fess ee SRS OE SAL & 5 5 = = . . - nage tel Fm. woty<br>3 BARS AE ag Wnt yt te ESL) chees Seats cf 3 . cer eg ah Bs Ss ee<br>fe Byles PRE Pe gee TY UE eS: “7.0 4, 'R oss ee its? ° oan %<br>4 eee ae + 5 ot .. wok o . : ae S<br>Ssfe oeKg te . . . ron :<br>. . o °<br>vt wo<br>Municipalities sorted by their average XB Municipalities sorted by their average XB<br><!-- End of picture text -->
<!-- Start of picture text --> CEB, UC-CEB.<br>1 +—_{iay}— 1 +——_{iaay}"—<br>2 +———_{ila-@#@— 2 +——_{ililay]}"_——_e e<br>3 e-—_iay-— 3 +———__iiy}#——<br>4 +—_{_[ #— e 4 — fT] | @<br>5 +——_{ile 5 +——_{ilay}-<br>6 —{I}— » 6 —LI}— e<br>—{[}—*# 7 ° +—{[}—+<br>7 oo os<br>8 eo |——_ij}_ 8 e |—_{ij#—e<br>9 oog—};i ee 9 e o|—_{j+_e<br>10 e @—|]}+=0 10 e e+e<br>-40 -20 0 20 40 -40 -20 0 20 40<br>Design bias at municipal level (x100)<br><!-- End of picture text -->

<!-- Start of picture text --> CEB, e e — (|| - Cr<br>Direct —|-<br>UC-CEB,|@ @ @ — ||) - ee<br>-30 -20 -10 0 10 20<br>Design bias at municipal level (x100)<br><!-- End of picture text -->
<!-- Start of picture text --> CEB: ee e °@ e<br>UC-CEB, renee eco @ e e e<br>0 2 4 6 8 10<br>MSE at municipal level (x100)<br><!-- End of picture text -->
<!-- Start of picture text --> CEB: UC-CEB:<br>' ° ——_ i i/o eo }+—_[yy<br>2 °'——_ia '——_ iad<br>3 e eo e +—{ijy+_{_ 3} e@ ee e+—iij}¥\4<br>4 «“—fi— «4 eT}<br>5 «¢———iilw7"——— e 5 e:—_—_{ii—H/|—— e<br>6 e +—_{il*"““| 6 +——_{jia]—1<br>7 *+—{[}—+ 7 ee +L}4<br>8 e }+— ij} 8 -— TW<br>9 ee |—{j} “10 9 oe:—_{}+—e<br>10 e e@ we 10 © enes—{||—0<br>-30 -20 -10 0 10 -30 -20 -10 0 10<br>Design bias at municipal level (x100)<br>CEB: UC-CEB:<br>1| Hii=3}——+ 1| i}<br>2| Hii=73¥——+ 2) a}—¥+_——__—VT<br>3| HiiZ— 3) H+ —_—<br>4) Hii'— ‘| }—¥+¥_\_i<br>5| Hiii-— 5| =3}———+<br>6| Hit .| i<br>7| H- 7| 43——<br>8| Hi | (ii<br>9} Hi 9| Hii——+<br>10} HH 10} HH<br>0 5 1 15 2 0 Re) 1 1.5 2<br>excludes outside values excludes outside values<br>MSE at municipal level (x100)<br><!-- End of picture text -->
census for all households. Then a unit-context model and a new unit model are selected, once again following the process described in section 4.2 using the first out of 500 samples where 1% SRS by PSU is selected. This is done to remove the issue of outliers from the data and to ensure that the data generating process follows the one assumed in Eq. (2). The simulation removes the potential misspecification due to deviations from normality in the data and allows us to isolate the problem present in the unit-context model ( _UC − CEBa_ ).

Results for the new hybrid simulations are presented in Figures 22 and 23. Note that in this simulation, where we have removed the normality issue, the upward bias that was present in the unit level model ( _CEBa_ ) is no longer evident. On the other hand, the previously suspected downward bias of the unit context models ( _UC − CEBa_ ) is salient, as can be seen in Figure 22 and by municipality deciles sorted by population in Figure 24. Note that under the _UC − CEBa_ model more than 75% of the municipalities present a downward bias (Fig. 22). This finding is aligned to what we observed in Figure 17. However, because there is no deviation from normality in the hybrid simulation, the downward bias of the unit-context models ( _UC − CEBa_ ) is never offset, and is quite considerable and leading to substantially larger empirical MSEs for the unit context models (Fig. 25). Simulations were repeated, where, instead of performing model selection, the selected model for CEB estimators contains exactly the same covariates as those used to generate the welfare, and considering only the area aggregates for U-C models. This was done just to check whether the observed biases could be due to model misspecification, in the sense that the selected covariates are different from those in the true model. Results were very similar to those observed in the previous hybrid simulation with a model selection step. Hence, the results suggest deviations from the assumed model are an issue and the countering of biases is what is driving the seemingly good results for unit-context models in the two-stage sampling scenario, highlighting the importance of proper data transformations and model selection to ensure that model assumptions hold when using Census EB methods.

Given the direction of the bias of unit-context models is not known _a priori_<sup>50</sup> - and that these might present high bias - unit-context models are unlikely to be preferred over traditional FH methods when the census auxiliary data are not aligned to survey microdata, unless the calculation of variances is not possible for various locations as noted before.<sup>51</sup> In this case benchmarking is not a recommended procedure and may not help in reducing the bias. EB estimators are approximately model unbiased and optimal in terms of minimizing the MSE for a given area, thus when adjusted afterwards for benchmarking, that is, so that these match usual estimates at higher aggregation levels, the optimal properties are lost and estimators usually become worse in terms of bias and MSE under the model. When benchmarking adjustments are large, as those likely required by _UC_ variants, it is an indication that the model does not really hold for the data. In the case of _UC_ models we have shown that the model will not hold due to omitted variable bias.

Also, note bias can lead to considerable re-ranking of locations and thus a limit on the acceptable

> 50Notice how under the simulations presented in Figures 1 and 3 the method appears to be upward biased.

> 51This holds for other measures of welfare, and particularly for ELL variants of the unit-context models.

36
bias should usually be determined according to need. This is of particular importance when determining priorities across areas based on small area estimates. If an area’s true poverty rate is 50% and the method yields an estimator of 10% due to a biased model, there is a real risk that this area may not be given assistance when needed. Molina (2019) suggests 5 or 10 percent of absolute relative bias as an acceptable threshold. An additional problem for unit-context models in many applications is it is not possible to match census and survey PSUs; in some cases it is due to confidentiality reasons and in others it is due to different sampling frames used for the survey. The latter is something that is likely to affect applications where census and surveys correspond to different years. Under these scenarios, unit-context models are unlikely to be superior to FH and alternative area models.

# **5 Conclusions**

In this paper we have illustrated that one of the most important aspects of SAE applications with Census EB methods under unit models is selecting a proper model; especially the issue of data transformation. Such a finding is not new, and has been quite often echoed by others in the area (see Marhuenda et al. 2017; Molina and Marhuenda 2015; Tzavidis et al. 2018). Here we show how data transformations can lead to improved results. For example, under onefold nested error models, the aggregate gains from moving to a log-shift transformation as opposed to just taking the natural logarithm are close to 30 percent in terms of MSE. Nevertheless, as Marhuenda et al. (2017) note, finding an appropriate transformation is not always straightforward and the resulting data could stray from normality which would lead to biased estimates as we also find here. Consequently, model checks and residual analysis are something that every SAE application should include in order to test if the model’s assumptions are not completely invalidated. In case of data deviating from model assumptions, the model should be changed accordingly. For instance, in case of area outliers, fixed effects could be included for those outlying areas in the model to reduce the design bias. If their sample sizes are not too small, the efficiency of the resulting model-based estimates might be acceptable in this case, even if specific model parameters are specific to these areas.

Second, we have validated SAE applications under model-based simulation and design-based simulation methods. Under model-based validation, where the data generating process follows a twofold nested error model, we note the ELL method still performs poorly in terms of MSE even with contextual level variables. The result is most evident in scenarios where area random effects are larger than cluster random effects, since contextual variables do not explain a sufficient amount of the area’s variability. Issues regarding the underestimation of this noise under ELL are not evaluated here, but should be considered by future practitioners, particularly when the noise is estimated under the MI inspired bootstrap method (see Corral et al. 2020; Das and Chambers 2017; Marhuenda et al. 2017 ). On the other hand, model-based simulations conducted here provide further evidence to the finding from Marhuenda et al. (2017) that misspecification of the model under the onefold CensusEB, i.e. modeling random effects at the area level only, when the true model has cluster and 37
area level random effects, with clusters nested within areas, entails virtually no loss of efficiency when estimating area level welfare-based indicators.

Under design-based validation, where the sampling strategy mimics real world scenarios such as those implemented under LSMS surveys, SAE methods present improvements over direct estimators. We have also investigated estimators based on unit-context models, originally proposed by Nguyen (2012), which can be applied when census auxiliary data at the household level is not valid. Given that under the two-stage samples used, many municipalities, which are the target areas, are represented by a small number of observations (or even zero), these data are not always suitable for FH area level models. Despite model-based simulations yielding poor results in terms of bias for unit-context models the CensusEB variant does considerably better than the ELL variant under the design-based simulations. The reason for the positive results is shown to be due to the bias that is inherent in the unit-context model, which was being offset by bias due to deviations from normality. This offset is something that is not guaranteed to occur in all scenarios because the direction of the bias of unit-context models is not known _a priori_ . In simulations where deviations from normality are not an issue, the bias of the unit-context method becomes quite clear. Additionally, the method’s performance is contingent upon the number of subdomains present in the domain. Moreover, PSUs or subdomains must be matched across census and survey for unit-context models, something that is not always feasible. Finally, since the direction and size of the bias of the unit-context model are not known beforehand, the models may be considered an alternative to FH and other area level models only when area level models are not applicable, like in our design-based experiment with the more realistic two-stage sampling.

38