metadata: authors: 0: author: Merfeld,Joshua David
1: author: Newhouse,David Locke
2: author: Weber,Michael
3: author: Lahiri,Partha
count: Mexico
docna: 0: docna: Combining Survey and Geospatial Data Can
            Significantly Improve Gender-Disaggregated Estimates of
            Labor Market Outcomes
docty: Policy Research Working Paper
abstracts: cdata!: Better understanding the geography of
            women&#x2019;s labor market outcomes within countries is
            important to inform targeted efforts to increase
            women&#x2019;s economic empowerment. This paper assesses
            the extent to which a method that combines simulated survey
            data from urban areas in Mexico with broadly available
            geospatial indicators from Google Earth Engine and
            OpenStreetMap can significantly improve estimates of labor
            force participation and unemployment rates. Incorporating
            geospatial information substantially increases the accuracy
            of male and female labor force participation and
            unemployment rates at the state level, reducing mean
            absolute deviation by 50 to 62 percent for labor force
            participation and 25 to 52 percent for unemployment. Small
            area estimation using a nested error conditional random
            effect model also greatly improves municipal estimates of
            labor force participation, as the mean absolute error falls
            by approximately half, while the mean squared error falls by
            almost 75 percent when holding coverage rates constant. In
            contrast, the results for municipal unemployment rate
            estimates are not reliable because values of unemployment
            rates are low and therefore poorly suited for linear models.
            The municipal results hold in repeated simulations of
            alternative samples. Models utilizing Basic Geo-Statistical
            Area (AGEB)&#x2013;level auxiliary information generate
            more accurate predictions than area-level models specified
            using the same auxiliary data. Overall, integrating survey
            data and publicly available geospatial indicators is
            feasible and can greatly improve state-level estimates of
            male and female labor force participation and unemployment
            rates, as well as municipal estimates of male and female
            labor force participation.
display_title: Combining Survey and Geospatial Data
            Can Significantly Improve Gender-Disaggregated Estimates of
            Labor Market Outcomes
listing_relative_url: /research/2022/06/33839458/combining-survey-geospatial-data-can-significantly-improve-gender-disaggregated-estimates-labor-market-outcomes
topicv3: Gender and Labor Market m1326905 923
publishtoextweb_dt: 2022-06-09T00:00:00Z
dois: 10.1596/1813-9450-10077
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
author_keys: 0: author_key: 1762622
1: author_key: 450079
2: author_key: 1008946
3: author_key: NA
model_extractions: Policy Research Working Paper 10077 Combining Survey and Geospatial Data Can Significantly Improve Gender-Disaggregated Estimates of Labor Market Outcomes _Joshua D. Merfeld David Newhouse Michael Weber Partha Lahiri_ Poverty and Equity Global Practice & Social Protection and Jobs Global Practice June 2022
Policy Research Working Paper 10077

# **Abstract**

Better understanding the geography of women’s labor market outcomes within countries is important to inform targeted efforts to increase women’s economic empowerment. This paper assesses the extent to which a method that combines simulated survey data from urban areas in Mexico with broadly available geospatial indicators from Google Earth Engine and OpenStreetMap can significantly improve estimates of labor force participation and unemployment rates. Incorporating geospatial information substantially increases the accuracy of male and female labor force participation and unemployment rates at the state level, reducing mean absolute deviation by 50 to 62 percent for labor force participation and 25 to 52 percent for unemployment. Small area estimation using a nested error conditional random effect model also greatly improves municipal estimates of labor force participation, as the mean absolute error falls by approximately half, while the mean squared error falls by almost 75 percent when holding coverage rates constant. In contrast, the results for municipal unemployment rate estimates are not reliable because values of unemployment rates are low and therefore poorly suited for linear models. The municipal results hold in repeated simulations of alternative samples. Models utilizing Basic Geo-Statistical Area (AGEB)–level auxiliary information generate more accurate predictions than area-level models specified using the same auxiliary data. Overall, integrating survey data and publicly available geospatial indicators is feasible and can greatly improve state-level estimates of male and female labor force participation and unemployment rates, as well as municipal estimates of male and female labor force participation.

This paper is a product of the Poverty and Equity Global Practice and the Social Protection and Jobs Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank. org/prwp. The authors may be contacted at dnewhouse@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Combining Survey and Geospatial Data Can Significantly Improve GenderDisaggregated Estimates of Labor Market Outcomes** Joshua D. Merfeld<sup>*</sup> , David Newhouse<sup>†</sup> , Michael Weber<sup>‡</sup> , and Partha Lahiri<sup>§</sup>

JEL codes: J21, C13 Keywords: Small Area Estimation, Data integration, Geospatial Data, Labor Force Participation, Unemployment, Mexico

* KDI School of Public Policy and Management and IZA

- The World Bank and IZA

- The World Bank and IZA

> § Department of Mathematics and Joint Program in Survey Methodology, University of Maryland
# **1. Introduction**

Low female labor force participation can be an important barrier to economic growth. In Mexico, for example, women’s labor force participation is considerably lower than that of men and is also low in comparison to other countries. The female labor force participation rate was just 45 percent in 2019, compared to 77 percent for men, a gap of 32 percentage points. In the OECD, only Turkey and Italy have a lower rate of female labor force participation, while in LAC only Guatemala has a lower value than Mexico. This has important economic impacts. A recent World Bank analysis suggests that if women participated at the same rate as men, per capita income could be as much as 22 percent higher (Inchauste et al., 2021). The same analysis suggests that if Mexico could implement policies to increase the female labor force participation rate by 0.6 percentage points a year – in line with that observed in Spain, Ireland, and Chile – this would help eliminate the gender gap in labor force participation and increase economic growth by 0.4 percent a year, highlighting how increases in female labor force participation are not just an output of the development process, but a driver of it, as well.

Trends in female labor force participation are determined by several key factors, including the relationship between households’ economic conditions and female participation, the supply and demand for jobs deemed appropriate for more educated women, national growth strategies, and occupational gender segregation (Klasen, 2018). To better understand from a spatial perspective how policy can address these factors, it is useful to understand where pockets of low rates of female labor participation exist. However, most labor force surveys can only provide adequate estimates for larger areas of aggregation. Mexico’s labor force survey is no exception, as it can only provide estimates for urban and rural areas within states, as well as for select larger municipalities. Since Mexico has 32 states and 2,450 municipalities, reliable estimates of labor force participation – for both women and men – at the level of the municipality could greatly inform the geographic targeting of education, labor market programs, and other measures designed to improve female labor force participation. This is also true of other developing countries.

Small area estimation is a branch of statistics that combines survey data with richer auxiliary data to generate more precise and accurate estimates of statistics. It has frequently been applied to measures of poverty and labor market outcomes. An extensive literature has documented that auxiliary data, typically taken from census or other administrative sources, can improve estimates of labor force participation and unemployment (Esteban et al., 2020, Chambers et al., 2016, LópezVizcaino et al., 2015, Ugarte et al., 2009, Molina et al., 2007, Datta et al, 1999). This literature employs a variety of methods, including hierarchical Bayes, empirical Best Predictors, and multinomial logit mixed models, as well as area level models. Intuitively, small area estimation allows surveys to “borrow strength” from auxiliary data that is more geographically comprehensive, and therefore covers unsampled areas. Combining the survey data with model predictions for unsampled areas, derived from the auxiliary data, makes the estimates more accurate and precise. In addition, by modeling the average relationship between the survey and auxiliary data across the entire sample, small area estimation can help correct sampling error even in sampled areas. This method can therefore produce more precise estimates for geographic levels, such as Mexican municipalities, where the labor force survey is too small to generate reliable estimates.

2
At the same time, the rapid proliferation of publicly available “big data” from satellite and crowdsourcing applications has made geographically comprehensive data freely available, while survey enumeration area geocoordinates or administrative shapefiles required to link surveys to geospatial data are also becoming increasingly available. These developments have sparked a burgeoning literature – ably reviewed in Burke (2021), McBride et al. (2021), and World Bank (2021) – on the use of satellite imagery for economic measurement. Most early work has focused on the ability of geospatial data to predict agricultural yields and crops (Erciulescu et al., 2019, Lobell et al., 2020), household asset wealth and poverty (Jean et al., 2016, Yeh et al., 2020 Steele et al., 2017, Engstrom et al., 2021), and population (Wardrop et al.., 2020, Engstrom et al., 2020).

This is the first paper to our knowledge that explores whether combining survey data with geospatial indicators can be used for the purposes of improving estimates of labor market outcomes, such as female and male labor force participation and unemployment, through small area estimation. Demonstrating that this methodology works is important, as many developing countries do not have the requisite data traditionally used for small area estimation, such as up-todate censuses or detailed administrative data. We show that the incorporation of geospatial data – which is available across the globe – is a feasible alternative when preferred sources of auxiliary data are not available.

This paper also contributes to the ongoing discussion over the proper methods for combining survey and geospatial data for the purpose of prediction. Some analysts take an explicitly Bayesian approach (Steele et al., 2017, Pokhriyal and Jacques, 2017, Erciulescu et al., 2019), while others employ an empirical best predictor model (Battese Harter and Fuller, 1988 Masaki et al., 2020).

We undertake this exercise in the context of Mexico, which has publicly available data that makes our approach feasible. We focus on three separate levels of aggregation: the state, the municipality, and the AGEB (Área Geoestadistica Básica, i.e. the Basic Geo-Statistical Area). Importantly, the census data has AGEB-level identifiers for all urban AGEBs in the country. This allows us to match geospatial data to AGEBs. In addition, we simulate a random sample from the census before implementing our preferred small area estimation approach. The use of census data allows us to compare the resulting estimates to the “truth,” derived from the census data itself. This type of data is rarely available in developing countries and, as such, Mexico is the perfect context in which to apply our approach.

It is important to note that our results on unemployment and labor force participation differ from official rates in two key ways. First, we include only urban AGEBs due to data limitations. Second, we are using census data, not a labor force survey, so the instruments differ. For example, the age range for which employment questions are asked differs across the instruments. However, despite these differences in the instruments, the underlying concepts of labor force participation and unemployment in the census and labor force survey are similar. There is therefore no reason to believe that the findings on the benefits of incorporating geospatial data would not apply to the official definitions as well.

This paper implements a sub-area model, which is essentially a unit-level model in which the unit is taken to be the AGEB. In particular, we specify a weighted empirical best predictor model, with conditional random effects specified at the municipality level in our main results. Municipal 3
predictions are then generated by taking the population-weighted mean of the AGEB-level predictions.<sup>1</sup> This approach leverages AGEB-level data, which leads to more accurate and efficient estimates than area-level models specified at the municipal level and requires minor modifications of publicly available software to generate point estimates and confidence intervals.<sup>2</sup> Six main findings emerge:

1. Combining survey and geospatial data substantially improves the precision and accuracy of state-level estimates of both labor force participation (LFP) and unemployment rates. Estimated mean absolute deviation falls by approximately 62 and 50 percent for female and male participation rates, respectively, and by approximately 52 and 25 percent for female and male unemployment rates.

2. When considering municipal estimates of labor force participation rates, incorporating geospatial data significantly improves accuracy and greatly improves precision. In estimates from repeated simulations, estimated mean absolute deviation falls by about 43 percent for women and 53 percent for men, and rank correlation increases by 0.13 for women and 0.11 for men. After adjusting direct survey estimates to equalize coverage rates, estimated mean squared error for male and female LFP falls by a factor of more than four.

3. Because unemployment rates are very low, municipal estimates of unemployment rates based on a linear model are not reliable. Although estimated mean absolute deviation falls for both man and women, relative bias is high, rank correlation falls significantly for women and the estimates exhibit little variation.

4. Accuracy is substantially lower in out-of-sample municipalities than sampled municipalities.

5. When using a sample that simulates fully enumerating sampled AGEBs, small area estimates offer very minor improvements on direct survey estimates.

6. A model specified at the AGEB level generates more accurate estimates than one specified at the municipal level. The improvement due to using AGEB rather than municipal variation is much larger for male LFP, which is harder to predict, than for female LFP.

In short, the results support the use of augmenting survey data with geospatial data when estimating state-level statistics and when estimating municipality-level female and male labor force participation rates. When estimating municipal unemployment rates, however, the estimates are not reliable because the sample contains insufficient information to train a linear model. In particular, the sample contains no unemployed men in 44 percent of municipalities and no employed women in 68 percent of municipalities. Modeling unemployment rates using a linear model is problematic in this setting, highlighting the usefulness of software that can conveniently estimate and apply a wider range of non-linear empirical best predictor models.

> 1 While this is in spirit similar to the sub-area model proposed in Torabi and Rao (2018), the approach outlined in that paper is difficult to implement using publicly available software. 2 Code is available from the authors upon request.

4
# **2. Data and Methodology**

# **a. Census and Survey Data**

The primary source of data on labor market outcomes is the 2020 Census of Population and Housing, carried out in March 2020 by the Mexican National Institute of Statistics, Geography, and Informatics (INEGI).<sup>3</sup> The census was collected the month before the COVID-19 pandemic led to widespread shutdowns. INEGI publishes census statistics publicly at different geographic levels, with the lowest level being the Área Geoestadistica Básica (AGEB). INEGI publishes aggregate statistics only for urban AGEBs. We use these to generate municipality statistics that are weighted by the population of urban AGEBs. Because only urban AGEBs are included in the analysis, the results in this paper pertain to urban state and municipality-level rates. Urban AGEBs total to around 96.4 million people, or more than 75% of Mexico’s estimated 127.6 million people.

We are primarily interested in examining the feasibility and effectiveness of combining geospatial data with survey data to improve municipal estimates of labor force participation rates and unemployment rates, separately for women and men. We also examine the ability of the procedure to improve state-level estimates of these four labor market outcomes. We restrict the analysis to urban AGEBs in order to utilize the census data as a credible benchmark against which to assess the performance of the small area estimates.

From the available urban census data, we construct a data set with six different variables: the total number of women in the labor force, the total number of women employed, the total number of women 12 years or older, plus the same three variables for men.<sup>4</sup> Since this is census data, we take these values to be the true values for each AGEB. We then randomly sample AGEBs from the full set of urban AGEBs to form a pseudo survey. We select AGEBs with probability proportional to size within states, which serve as the strata. We then choose the number of AGEBs per stratum (state) using sample sizes similar to INEGI’s labor force surveys.<sup>5</sup> Like INEGI, we assume an average of 2.7 people 12 years of age and older per household when selecting AGEBs.<sup>6</sup> Out of 50,942 AGEBs with non-missing geospatial auxiliary features, 7,642 – or 15.0 percent – are included in the pseudo sample.

After selecting AGEBs, we simulate sampling individuals within each selected AGEB. The census data indicate the number of men and women the number of people 12+ living in each AGEB, as well as the number of people 12+ who are active labor force participants as well as the number employed. Since being employed is conditional on being in the labor force, we can use this information to simulate a sample of men and women within each AGEB based on these variables. We essentially construct the total population and randomly select 40 individuals within each AGEB – or all individuals if there are fewer than 40 men and women 12+ in the AGEB. These are then be used to construct gender-specific labor force participation and unemployment rates in the sample.

> 3 The data are available online <u>here.</u>

> 4 The labor data are collected for those 12 years of age and older, hence why we use 12+ as our population. 5 More information is available here. One difference between our methodology and theirs – apart from the fact that we do not have access to actual enumeration areas – is that INEGI also includes some larger municipalities as strata. For simplicity, we only use states.

> 6 This number comes from INEGI’s labor force survey methodology manual in footnote 5.

5
In sample? ina] Yes , | No | | Not<sup>inresults</sup>

<!-- Start of picture text --> A. AGEBs<br><!-- End of picture text --> <!-- Start of picture text --> B. Municipalities<br>’3\N<br>’4<br><!-- End of picture text --> <!-- Start of picture text --> v<br><!-- End of picture text --> <!-- Start of picture text --> (a<br><!-- End of picture text -->
Figure 1 shows the AGEBs and municipalities that appear in the main sample. Panel A shows AGEBs, with individual AGEBs colored based on whether they are in the estimation sample (“Yes”), are not in our estimation sample but are urban (“No”), or are rural AGEBs that are not considered in the analysis (“Not in results”). Panel B shows the same, but for municipalities. Specifically, a municipality is coded as included in the sample if at least one AGEB within that municipality is contained in the sample. As such, a municipality in our sample could have a relatively small proportion of its total population actually included in our sample. There are 1,641 municipalities that contain at least one urban-AGEB with non-missing data. Of these, 1,034 – or 63.0 percent – have at least one AGEB in the sample.

# **b. Auxiliary geospatial data**

Auxiliary data is drawn from two sources: Google Earth Engine, and Open StreetMap, utilizing the 2020 shapefile of urban AGEBs provided by INEGI. These sources were largely selected because they are publicly available, cover a large portion of the world, and convenient to obtain. However, both contain a large number of candidate predictors that could be plausibly correlated with spatial patterns in labor force participation. From Google Earth engine, we extracted summary statistics by AGEB from six datasets: Nitrogen Dioxide from Sentinel 5P, Normalized Difference Vegetation Index (NDVI) from the Sentinel 2 Multi-spectral Instrument, Nighttime lights from VIIRS, estimated population from WorldPop, land cover classifications from the Copernicus dynamic land cover map, and the year of development, as proxied for by the change in pixels from pervious to impervious surfaces (Gong et al., 2020). All summary statistics were taken over March 2020, except for land cover, which pertains to the period between January and December 2019.

From Open StreetMap, indicators were obtained representing the total length and number of highways in each AGEB. Two measures of road length were calculated: The first is the total length of the portions of highways contained within the AGEB, and the second is the total length of all highways that intersect the AGEB, including the portions of the road outside the AGEBs. In addition, the total counts of open StreetMap amenities and points of interest, such as hospitals, offices, schools, churches, and so on, were calculated for each AGEB. A final auxiliary variable, besides the Open StreetMap indicators, is the total area of the AGEB of the AGEB in sq km, which was calculated directly from the AGEB level shapefile provided publicly by INEGI. For each auxiliary variable, municipal variables were constructed by taking either population-weighted averages or simple sums (for counts and area) of the AGEB values.

# **c. Model selection**

Variables for prediction were selected using the LASSO (Least Absolute Selection and Shrinkage Operator), with the penalty chosen to minimize the Bayesian Information Criteria (BIC) (Zhang et 7
al., 2010).<sup>7</sup> LASSO models were fitted separately to four dependent variables. These are the transformed versions of the four labor market outcomes, AGEB male or female labor force participation and unemployment rates, using the arcsine transformation. The candidate predictors included in the LASSO model include all of the AGEB and municipal level geospatial auxiliary data described in the last section, as well as dummy variables for states. The state dummies were included with no penalty, and were included to both improve the model and to prioritize the selection of variables that predict variation in the labor market outcomes within states.

# **d. Model estimation**

We begin by constructing two separate estimates of LFP and unemployment in Mexico:

1. Direct estimates using only the pseudo survey of sampled AGEBs, which can only be constructed for in-sample municipalities.

2. Small area estimates using the pseudo survey of sampled AGEBs and the auxiliary geospatial data, which can be constructed for all municipalities, both in and out of sample.

These estimates are compared with the full census results along several different dimensions to assess their accuracy, as discussed below.

To generate the small area estimates for municipalities, we utilize a standard nested error model at the AGEB level (Battese, Harter, and Fuller, 1988, Jiang and Lahiri, 2006, Molina and Rao, 2010). The transformed version of the labor market outcome at the AGEB level is modeled as a linear function of geospatial auxiliary variables. The dependent variable was transformed using an arcsine transformation, a common transformation for proportions bounded between zero and one. The model used to generate municipal estimates is specified as follows: (1) s s 1<sup>+XX</sup> s 2<sup>+ XX</sup> 3<sup>+ υυ</sup> s s ss<sup>) = XX</sup> ss<sup>ββββ</sup> ss<sup>ββ</sup> ss<sup>, s=1,...,S,m=1,...,Ms; a=1,...,Asm,</sup> where G(G(yys + uu female LFP, and male and female unemployment rate) of AGEB a within municipality m and state ss<sup>) denotes the arcsine of one of the four estimated labor market outcomes (male and</sup> s, derived from the sample. The model is assumed to hold for the full population of AGEBs yy although it is estimated using the sample. Xsma is a vector of geospatial auxiliary variables specified at the AGEB level, while Xsm is a vector of geospatial auxiliary variables aggregated to the municipal level. Xs is a vector of state dummy variables. β1 , β2 , and β3 are vectors of regression coefficients, s 2 , and is a set of municipality conditional random effects. s 2) is a residual error term. As noted above, the set of vv predictor variables was selected using the BIC-minimizing lasso procedure from the full set of υυ is distributed normally with variance σσ ss<sup>~NN(0, σσ</sup> uu candidate variables, with the full set of state dummies Xuu s included with no penalty.

For the model used to generate estimates for states, the specification is similar, except that the conditional random effect is omitted from the model:

> 7 The BIC was based on the unpenalized regression of selected variables rather than the penalized regression, and when estimating the lasso, standard errors were clustered at the municipality level.

8
s s 1<sup>+xx</sup> s 2<sup>+ XX</sup> 3<sup>+ uu</sup> s ss<sup>) = xx</sup> ss<sup>ββββ</sup> ss<sup>ββ</sup> ss<sup>, s=1,...,S,m=1,...,Ms; a=1,...,Asm.</sup> The state-level effect (2) G(yy is omitted because of the inclusion of a full set of state dummy variables Xs as independent variables. This mechanically causes the estimated variance of to be zero, due ss to the lack of variation across states in the residual after controlling for state dummies, which υυ ss effectively drops from the model. In other words, the state-level estimates are generated from υυ what is essentially a fixed effect estimator instead of a conditional random effect estimator. This ss is possible when generating state-level estimates, but not municipal level estimates, because the υυ survey is sufficiently large to estimate state level results precisely.<sup>8</sup> To generate the small area point estimates, the procedure simulates 100 draws from a normal distribution for both the random error term s s estimates) for each AGEB j. These estimates are then back-transformed, averaged across ss<sup>and the area effect υυ(or υυ</sup> ss<sup>for the state level</sup> simulations, and finally aggregated across AGEBs to generate municipal or state-level estimates. uu When aggregating across AGEBs, we weight using estimated population from WorldPop.<sup>9</sup> Mean squared error estimates are obtained from a parametric bootstrap procedure, as proposed by Gonzalez-Manteiga et al.(2007) and implemented in the R EMDI package (Kreutzmann et al., 2018), based on the theory developed by Butar (1997) and Butar and Lahiri (2003).

When estimating the parameters of models (1) and (2), observations are weighted using normalized inverse probability weights, using the weighting method implemented in the R nlme package (Pinheiro, et al., 2021). Because each municipality is given equal weight when evaluating the estimates, it is necessary to normalize the weights to avoid giving too much weight to more populous municipalities in the sample estimation.<sup>10</sup> We therefore normalize the weights by dividing the inverse probability of selection by the mean inverse probability of selection for the municipality, as follows: s 1 s where is the number of AGEBs contained in the sample for ∑ππ ss<sup>−1</sup> s municipality m and WW ss<sup>=</sup> NNss ss=1NNss ππ ss<sup>−1</sup> s NNss normalizes the sum of the weights in each municipality to equal the number of sample observations, ss<sup>is the probability of AGEB a being included in the sample.11 This</sup> which is one recommended method for normalizing weights when estimating conditional random ππ

> 8 The same software package was used to estimate both municipal and state level results. We experimented with omitting state-level dummies and the results were worse.

> 9 Using estimated population from WorldPop has the added benefit of being applicable to other contexts. While we have census information that would allow us to aggregate using arguably more “correct” data, this would not be possible in other contexts in which census data is missing or out of date. For this reason, we drop any AGEBs with an estimated population of zero. Only 0.6 percent of AGEBs are dropped.

> 10 For some policy contexts, it may be desirable to give greater weight to more populous municipalities when evaluating the accuracy of estimates, but in this analysis, we limit consideration to the case where each municipal estimate is given equal weight when evaluating the performance of the estimates as a whole.

11 We construct the sampling weight as the inverse of the probability of selection, using the formula for sampling with replacement, as an approximation for the actual probability of sampling based on our sampling strategy. In P s particular, the probability of selection is approximated as i = 1 −�1 − P PPt ss ~~�~~ NN , where popsma = population of AGEB a, poptotal = total national population and ππ s PP ttsstt 9 ss<sup>), the number of individuals in the sample.</sup> NN= m mm(40, P PP
effects models.<sup>12</sup> Below, as a robustness check, we show that giving equal weight to each municipality worsens the predictions.

Finally, we ignore heteroscedasticity in the dependent variable, which is the average AGEB labor market outcome taken from the sample. A sample of 40 individuals was taken from each AGEB, except for the few cases where AGEBs contain fewer than 40 working aged persons, which implies that heteroscedasticity is not a first order concern when estimating labor force participation rates. Different rates of labor force participation across sample AGEBs will affect the number of labor force participants used to calculate unemployment rates from the sample, but we do not adjust for this source of heteroscedasticity.<sup>13</sup> Population counts for each AGEB taken from the 2020 census were used as population weights when aggregating AGEBs to municipal estimates. We used a modified version 2.0 of the EMDI package in R, modified to incorporate sample and population weights, to estimate the model (Kreutzmann et al., 2018).<sup>14</sup>

# **e. Evaluating the estimates**

To evaluate the estimates, we compare each estimate to the population values from the 2020 census. We calculate the following statistics to compare these four separate estimation methods:

- Estimated mean squared error. For the direct estimates, this is assumed to be equal to the estimated variance of the mean, estimated using the Horwitz-Thompson approximation.<sup>15</sup> For the small area estimates, MSE estimates were generated using the parametric bootstrap procedure.

- The median estimated relative standard error across municipalities, where the relative standard error for each municipality is defined as the square root of the estimated mean squared error divided by the point estimate.

- Coverage rate: This is the share of municipalities for which the actual census value falls within the 95% confidence interval generated by the estimate.

> 12 See Rabe-Hesketh and Skondral (2006). In conditional random effect models, unlike in standard random effect models, multiplying the weights by a constant affects the parameter estimates.

> 13 This is partly because the high number of municipalities in the sample with zero unemployment is a more serious issue than heteroscedasticity when estimating unemployment rates.

> 14 The number of simulations and bootstraps, L and B, were each set to equal 100. The weighting strategy relies on passing through the specified weights to the lme function in the R nlme package that estimates the conditional random effect model (Pinheiro et al., 2020). In addition, we take a weighted mean instead of a simple mean when aggregating point estimates for AGEBs to generates estimates for municipalities. In the latter case, the aggregation weights are the population of the AGEBs.

15 We use the Horvitz-Thompson approximation of the variance described in Molina and Marhuenda (2015), in which the variance of the sample mean for municipality m is estimated as: �= ∑ 2 NNss Where m is a particular municipality, i is an individual within municipality m, N ss ii=1 ssii ssii ssii m is the number of relevant VV<sup>�</sup> �YY<sup>��</sup> ww (1 −ww )yy individuals in the municipality, w is the inverse probability sample weight, and ymi is the binary outcome variable (participation or unemployment) for individual i.

10
- Estimated relative bias: This is the deviation of the estimate from its true value, expressed in percentage terms relative to the true value.

- Estimated mean absolute error: This is the absolute value of the difference between the estimated rate and the actual census rate.

- Estimated correlations with census value: This is a simple correlation between a point estimate and the census value and is a measure of the accuracy of the prediction.

- Estimated rank correlations with census value: This is the spearman rank correlation between a point estimate and the census value, a measure of the accuracy of the predicted rank.

Estimated in this context refers to the fact that these statistics are estimated using one sample. In each case, statistics are calculated giving equal weight to each municipality. All comparisons between direct survey estimates and the small area estimates are restricted to in-sample municipalities. This makes for more accurate comparisons, since the direct estimates only pertain to these municipalities. Later, we also show how results for out-of-sample municipalities compare with results for sampled municipalities.

# **f. Repeated simulations**

A concern with drawing inferences about the performance from one sample is that any particular sample may not be representative. Since we construct this sample using a synthetic population, we can simulate the performance of the estimators across multiple samples. We therefore simulate 100 separate samples and calculate point estimates – for both direct estimates and small area estimates – for each of these samples. These are our preferred measure of the accuracy of small area estimates and direct estimates. However, due to computing time considerations, we do not perform a parametric bootstrap to calculate mean squared error in the simulation.<sup>16</sup> Therefore, our estimate of the efficiency gain due to small area estimation comes from only one sample.

# **g. Second-stage sample size**

As noted above, a key issue with respect to modeling unemployment rates is that the indicator is very close to zero in Urban Mexico, especially for women. In our sample, for instance, female unemployment is less than 1.5 percent. However, our sample size for each AGEB is only 40 and around half of those are women, for just 20 women per AGEB. Additionally, labor force participation is about 50 percent, meaning that unemployment is necessarily calculated from a sample of sometimes as small as just 10 women. With an average unemployment rate of less than 1.5 percent, the great majority of these AGEBs have an estimated unemployment rate of zero.<sup>17</sup> The same is true for men.

> 16 Drawing the sample, calculating the estimates, and performing the parametric bootstrap takes around two hours for a single sample. Since we simulate 100 different samples, this would take more than eight full days to estimate MSE across all simulations. Instead, we opt to calculate just point estimates – which takes only around 20 minutes for a single sample – and compare the performance of the point estimates.

> 17 For male unemployment, approximately 72 percent of AGEBs have a direct estimate of zero. For female unemployment, it is 87 percent.

11
The low unemployment rates in the population lead to two issues. First, the estimated R<sup>2</sup> of the model is very low, meaning that the linear model predicts little of the variation in unemployment rates in the sample. This is not surprising, given the large amount of sampling error in the AGEBlevel unemployment rates in the simple random sample. This does not rule out the possibility that the small area estimates improve on the direct estimates.<sup>18</sup> In this case, however, the large share of sample municipalities with zero unemployment also downwardly biases the variance estimate of the municipal conditional random effect. This in turn leads the predictions to give too much weight to inaccurate predictions from the model, relative to the sample, which in turn generates inaccurate estimates.

The most direct approach to this problem would be to implement a non-linear model such as a two-part model (Belotti et al., 2015), although doing so in an empirical best framework with existing software is not straightforward. However, increasing the size of the second stage of the sample may mitigate this issue by reducing the number of municipalities with zero unemployment in the sample.<sup>19</sup> As a robustness check, we calculate small area estimates using a hypothetical full enumeration of sampled AGEBs. In other words, we use the actual census value of the labor market outcomes in each selected AGEB to estimate the model. In many developing countries, it is not uncommon to perform a full listing of all households in selected enumeration areas before drawing the second stage sample. It may therefore be feasible to collect data on low-probability events like unemployment from all adults in selected enumeration areas, as part of a listing exercise.

# **3. Results**

# **A. Model diagnostics**

Before comparing results against the full census, we check the characteristics of the models themselves. Table 2 shows the number variables selected, the R<sup>2</sup> values of the regression, and a variety of other model diagnostics for the lasso and post-lasso results for all four outcome variables. The first panel presents the results for the state-level estimates, while the second and third panels present results for the municipality-level results. Marginal R<sup>2</sup> represents the variance explained by the auxiliary variables, while conditional R<sup>2</sup> represents the variance explained by both the auxiliary variables and the conditional random effect. The R<sup>2</sup> values for the simple random sample that generates municipal level estimates are generally lower than individual level models, particularly for modeling unemployment.<sup>20</sup> However, the conditional R<sup>2</sup> of the female LFP model is significantly higher, at approximately 0.3, and EBP models can perform well even in cases when the available covariates are not strong predictors (Marhuenda et al, 2017).

> 18 This is counterintuitive but follows from Stein’s paradox. Stein’s paradox implies that a shrinkage estimator that takes a weighted average of direct survey estimates for subpopulations and the grand mean, when there are three or more subpopulations, can substantially improve predictions for the subpopulations (Stein, 1964, Efron and Morris, 1977). This is also consistent with Marhuendra et al.(2017), who note that EB estimates can perform well even in cases where the available covariates are not strong.

> 19 The problem can also be fixed by using a non-linear estimator such as a two-part model (Belotti et al., 2015).

> 20 R2s may increase after adjusting for sampling error (Li and Lahiri, 2018).

12
Appendix Table A 1 shows the actual post-lasso model coefficients for the random sample results. In general, areas that are more urban on average are associated with higher female LFP, as mean vegetation intensity is negatively associated with participation, while the correlation is positive for the number of residential roads and street crossings. Consistent with this, areas that have been urbanized more recently, as indicated by the mean year of switching to impervious surfaces, have lower participation rates all else equal. Furthermore, smaller municipalities (by area), many of which are in Southern Mexico, are correlated with lower female labor force participation all else equal.

The relationship between other indicators of urban density and female labor force participation is more complex, however. Areas that are more heterogeneous in terms of vegetation intensity, such as the suburbs, are other things equal associated with higher labor force participation. The relationship between night-time lights and female LFP is another example. The median value of nighttime lights is positively associated with female LFP while the mean value is negatively associated, although the latter is not statistically significant. This nonetheless suggesting that AGEBs with a large number of very bright pixels, perhaps due to the presence of highways, have lower female labor force participation rates.

Some of the infrastructure variables are more easily interpretable, as well, but these are rarely significant. The number of schools is significantly positively correlated with higher male LFP but does not show up for any of the other three indicators.

Appendix Table A2 decomposes model R<sup>2</sup> across different categories of variables. The latter shows that the determinants of male and female LFP differ significantly. For example, the geographic size of the AGEB and municipality, nighttime light luminosity, the mean of the vegetation index, the number of different points of interest recorded in Open StreetMap, and the year that pixels changed to being an impervious surface are important predictors of female LFP. Municipal area is positively associated with female LFP, which makes sense in the Mexican context because many of the smaller municipalities are located in the poorer south of the country where female LFP tends to be lower. Median nighttime luminosity in the municipality is positively associated with female LFP, consistent with nighttime lights indicating economic activity. Mean nighttime lights in the municipality, which conditional on median is a measure of right-skewness or inequality in luminosity, is negatively associated with LFP. This is consistent with women in more developed areas being able to afford not to work. Areas with a greater concentration of vegetation, as proxied by mean NDVI, have lower female LFP, perhaps also reflecting greater wealth in suburban areas with more vegetation. The correlates of male LFP are more difficult to interpret. The share of bare land in the municipality is positively and significantly correlated with male LFP, as is the number of untagged highway points, and the latter may be positively correlated with economic activity.

<u>Table 2 - Model diagnostics by dependent variable and sample type</u>

||Female<br>LFP|Male<br>LFP|Female<br>Unemp|Male<br>Unemp|
|---|---|---|---|---|
|**State - Simple Random Sample within Selected A**|**GEBs**||||
|Number of (non-state) variables selected|41|22|19|10|
|<br>Marginal R<sup>2</sup>|0.127|0.061|0.025|0.028|

13
|Of which due to state dummies<br>|0.055|0.018|0.017|0.022|
|---|---|---|---|---|
|Conditional R<sup>2</sup>|0.127|0.061|0.025|0.028|
|Skewness of state effect|N/A|N/A|N/A|N/A|
|Kurtosis of state effect|N/A|N/A|N/A|N/A|
|Estimated variance of state effect|N/A|N/A|N/A|N/A|
|Estimated household residual|N/A|N/A|N/A|N/A|
|Ratio of estimated state effect variance to total<br>|N/A|N/A|N/A|N/A|

variance

# **Municipality - Simple Random Sample within Selected AGEBs**

|Number of (non-state) variables selected<br>|41|22|19|10|
|---|---|---|---|---|
|Marginal R<sup>2</sup>|0.132|0.071|0.038|0.032|
|Of which due to state dummies<br>|0.055|0.018|0.017|0.022|
|Conditional R<sup>2</sup>|0.293|0.170|0.132|0.077|
|Skewness of municipal effect|-0.102|0.222|4.047|1.011|
|Kurtosis of municipal effect|4.840|6.168|36.395|5.679|
|Estimated variance of municipal effect|0.004|0.002|0.0013|0.0008|
|Estimated household residual|0.017|0.017|0.012|0.017|
|Ratio of estimated municipal effect variance to<br>total variance|0.186|0.107|0.098|0.046|

# **Municipality - Full Enumeration of Selected AGEBs**

|Number of (non-state) variables selected<br>|33|26|28|39|
|---|---|---|---|---|
|Marginal R<sup>2</sup>|0.277|0.224|0.164|0.158|
|Of which due to state dummies|0.113|0.060|0.082|0.100|
|Conditional R<sup>2</sup>|0.597|0.451|0.467|0.525|
|Skewness of municipal effect|-0.610|-0.394|2.460|1.613|
|Kurtosis of municipal effect|6.128|4.779|24.136|12.472|
|Estimated variance of municipal effect|0.004|0.001|0.0011|0.0015|
|Estimated variance of household residual|0.005|0.003|0.0020|0.0020|
|Ratio of estimated municipal effect variance to<br>total variance|0.442|0.292|0.363|0.436|
|Number of AGEBs|7,733|7,733|7,733|7,733|
|Number of municipalities|1,072|1,072|1,072|1,072|

Dependent variable was transformed using arcsine transformation.

We focus here on the municipality-level diagnostics. The marginal R<sup>2</sup> values are generally low for the simple random sample, especially for unemployment rates. The low R<sup>2</sup> does not necessarily mean that the small area estimates do not improve on the direct estimates. The smaller the sample and the greater the heterogeneity in outcomes across AGEBs, the more likely a small area 14
estimation model is to improve upon sample estimates.<sup>21</sup> However, the low R<sup>2</sup> does raise concerns about biased estimates at the tails of the distribution, especially when estimating unemployment rates using the simple random sample. We expect that the combination of low marginal and conditional R<sup>2</sup> may lead to particularly inaccurate estimates for male and female unemployment rates when using the simple random sample. Model predictions for labor force participation using the simple random sample look significantly better for LFP than for unemployment, however, especially for women where the marginal and conditional R<sup>2</sup> values are 0.132 and 0.293 (as opposed to men where the same R<sup>2</sup> values are 0.071 and 0.17). The ratio of the estimated variance of the area effect to total variance is 0.19 for female participation and 0.11 for male participation, while the corresponding ratio for unemployment is about 0.1 for women and about 0.05 for men.

Using the full enumeration sample, not surprisingly, dramatically improves the prediction models. Marginal R<sup>2</sup> rises to about 0.27 for female LFP and 0.22 for men, and to approximately 0.16 for unemployment rates. The ratio of the variance of the estimated area effect to total variance ranges from 30 to 44 percent, depending on the indicator. Compared with the models developed using the simple random sample, more weight is given to the sample, which more accurately reflects municipal outcomes when sample AGEBs are fully enumerated. The higher R<sup>2</sup> s of the models reflects the elimination of the considerable noise in the sample data, which drove down the R<sup>2</sup> s of the models estimated using the simple random sample.

Finally, Appendix Figure A1 shows quantile-quantile plots for both components of the residuals, which is useful to check whether they appear to be normally distributed as assumed. The residuals are highly non-normal in most cases, except for the random area effect for male and female LFP. The non-normal random effects for unemployment, as well as non-normality in the household error terms may contribute to biased estimates of outcomes and uncertainty, which we will assess by comparing the small area estimates with the census. Future research can explore alternative transformations, such as the rank transformation used in Masaki et al.(2020), that might help achieve a more normal distribution.

# **B. State-Level Results**

We first consider estimates at the state level, the level at which the survey is considered to be representative. We present the seven statistics listed in section 2.e, separately for direct and small area estimates, in Table 3 .

The results are striking: Even though the direct estimates are considered to be reliable at the state level, we see large improvements in all seven statistics, across all four indicators, when incorporating geospatial data. The estimated mean squared error is smaller, the estimated correlation with the full-census benchmark increases, and the estimated absolute deviation from the benchmark decreases. Median relative bias declines for all outcomes. Coverage rates improve, sometimes dramatically. The estimates of unemployment rates improve on the direct estimates on most dimensions, even though the R<sup>2</sup> of some models is low. Because the outcomes measured in

> 21 For example, suppose as in this case that the sample covers 15 percent of the enumeration areas. Even a model that explains 10 percent of the variation in the remaining 85 percent of the population contributes at least the additional 8.5 percent of the total variation in the population not captured by the sample, in addition to any improvement in the estimates for sampled areas due to averaging samples with model predictions.

15
the sample contain a large amount of random measurement error, even models with low R<sup>2</sup> can contribute significant amounts of additional information and improve the estimates.

<u>Table 3 - State-level Results</u>

||(1)<br>Male LFP|(2)<br>Female<br>LFP|(3)<br>Male<br>Unemp|(4)<br>Female<br>Unemp|
|---|---|---|---|---|
|**Average estimated mean**<br>**squared error (x1,000)**|||||
|Direct|0.3787|0.1983|0.0067|0.0031|
|SAE|0.0899|0.1263|0.0055|0.0021|
|**Average estimated mean**|||||
|**squared error (x1,000)**<br>**with equalized coverage rates**|||||
|Direct|0.4048|1.2653|0.0555|0.0105|
|SAE|0.0898|0.1264|0.0055|0.0021|
|**Coverage rate**|||||
|Direct|0.8750|0.7188|0.5625|0.5000|
|SAE|0.9375|1.0000|0.9062|0.6875|
|**Estimated mean absolute error**|||||
|Direct|0.0179|0.0193|0.0048|0.0032|
|SAE|0.0088|0.0073|0.0023|0.0024|
|**Median estimated relative bias**|||||
|Direct|2.1164|-3.0573|-10.7269|-12.3980|
|SAE|-0.0757|-0.4322|1.5223|15.1544|
|**Estimated rank correlation**|||||
|Direct|0.5652|0.7933|0.7911|0.6635|
|SAE|0.7185|0.9498|0.8867|0.7529|
|**Estimated Pearson correlation**|||||
|Direct|0.6048|0.8141|0.7528|0.6506|
|SAE|0.7614|0.9583|0.8698|0.5467|
|States|32|32|32|32|

All figures represent unweighted averages or correlation across states. Equalized coverage rates refer to multiplying the estimated MSE of the direct estimates for each state by a constant to achieve the same coverage rate as the small area estimates. Rank or Pearson correlation is the unweighted rank or Pearson correlation between the estimated value and the actual state value. The absolute deviation is the absolute value of the difference between the estimate and the actual state value. Relative bias is 100 times the estimated minus the actual state value, divided by the actual state value.

Table A3 in the appendix presents a robustness check which calculates state-level results by aggregating municipality-level estimates up. The key difference between those results and the ones presented in this section are that the ones in Table 3 are estimated at the state level, without a random effect. The estimates in Table A3 are generally slightly worse than or similar to those in Table 3, with the possible exception of one accuracy indicator for male unemployment. The results in Table A3 give no empirical basis to prefer a method that aggregates the municipal estimates 16
with a random effect to one that estimates state results directly without a random effect. The latter has the advantage of being much simpler to estimate, particularly when it comes to measures of uncertainty.

# **C. Municipal-level estimates for sampled municipalities.**

We now turn to municipality-level statistics, which the survey is not designed to estimate reliably. All reported estimates in this section pertain to in-sample municipalities only, so that the direct and SAE estimators are compared over the same municipalities. We begin by comparing measures of uncertainty for direct estimates and small area estimates. We present these results for all four labor market statistics in Table 4 . The estimated mean squared error is always substantially smaller for the small area estimates than the direct estimates. This difference is largest – in percentage terms – for male labor force participation, where the estimated mean squared error is around a third as large as the direct estimate, roughly equivalent to effectively tripling the size of the sample.

The second row reports coverage rates, which is the share of municipalities for which the true census value is within 1.96 standard errors of the point estimate. Despite the smaller estimated mean squared error for the small area estimates, the coverage rates are also higher, again across all four indicators. For female labor force participation in particular, there is a significant increase in coverage rates from 87 to 98 percent in the small area estimates. This suggests that uncertainty for the direct estimates is underestimated and uncertainty for the small area estimates may be slightly overestimated. Therefore, for reach of the four indicators, we multiply the MSE of the municipal direct estimates by a constant greater than one to replicate the coverage rate of the small area estimates. When coverage rates are equalized in this way, the estimated MSE falls by a factor of four to five for labor force participation.<sup>22</sup> <u>Table 4 - Measures of Uncertainty and Coverage for Municipal Estimates</u>

||(1)<br>Male<br>LFP|(2)<br>Female<br>LFP|(3)<br>Male<br>Unemp|(4)<br>Female<br>Unemp|
|---|---|---|---|---|
|**Average estimated mean**<br>**squared error (x1,000)**|||||
|<br>Direct|9.329|5.494|0.288|0.161|
|SAE|3.331|4.991|0.179|0.093|
|**Average estimated mean**<br>**squared error (x1,000) with**<br>**equalized coverage rates**|||||
|<br>Direct|13.895|24.116|N/A|N/A|
|SAE|3.340|5.042|0.179|0.094|
|**Median estimated relative SE**|||||
|Direct|0.122|0.156|N/A|N/A|
|SAE|0.068|0.143|0.480|0.494|

> 22 Coverage rates cannot be equalized for unemployment rates because of the large share of direct estimates that are zero.

17
|**Coverage rate**<br>Direct|0.971|0.853|0.409|0.215|
|---|---|---|---|---|
|SAE|0.988|0.990|0.952|0.976|
|In-sample municipalities|1,072|1,072|1,072|1,072|

Only in-sample municipalities are included. The coverage rate is equal to one in a given municipality if the actual census value is within the calculated confidence interval (point estimates +/- 1.96 times the standard error). MSE with equalized coverage rates refer to MSEs after multiplying the estimated variance of the direct estimates by a constant to match the coverage rate of the small area estimates. Relative SE is missing for unemployment due to the large number of zeros in the direct estimates.

We note that both male and female unemployment rates are not normally distributed in our sample, even after transforming them, due to the mass at zero. The empirical best predictor method used here assumes a normal distribution. This may contribute to uncertainty if anything being slightly overestimated, as indicated by coverage rates that lie between 95 and 99 percent.

We next present statistics intended to measure the accuracy of the estimates, relative to results in the full census. We start with estimated relative bias and absolute deviation in Table 5 . These two measures differ along two key dimensions: The former is both relative, in the sense that it gives deviations from low true values greater weight than deviations from high true values, and directional, in the sense that negative and positive bias will cancel out. Estimated absolute deviation on the other hand is both measured in absolute terms and uses an absolute value and is therefore non-directional. It is also important in both cases to distinguish means from medians. Small (census) values of unemployment can lead to very large estimates of relative bias, since the bias formula contains the true value in the denominator.<sup>23</sup> For most policy applications relating to targeting, relative bias may be a misleading measure of accuracy for unemployment rates. This is because relative bias gives absolute discrepancies greater weight in municipalities with low true rates, which does not necessarily correspond to the objective of allocating resources to minimize unemployment.

For labor force participation, both estimated mean and median relative bias is modestly higher in the SAE estimates participation than in the direct estimates, though median relative bias is under 1 percent in both cases. For unemployment rates, estimated mean relative bias is much higher, reflecting the lack of explanatory power of the unemployment models, which in turn leads to large relative bias in areas where true unemployment rates are low. However, when it comes to absolute deviation the SAE estimates outperform the direct estimates by a substantial margin, for both LFP and unemployment. For female LFP, for example, the SAE procedure reduces estimated mean absolute error by about 45 percent, from 0.069 to 0.039, and for female unemployment rates the estimated absolute deviation falls by over half.

To get a further sense of the accuracy of the estimates, the bottom rows of the table report estimated rank and Pearson correlations. For the labor force participation estimates, the small area estimation increases rank correlation by 0.11 points for men, and by 0.13 points for women, from 0.59 to 0.73. For Pearson correlations the improvement is slightly smaller, 0.08 points for men and 0.11 points

> 23 For example, the smallest non-zero value for female unemployment is 0.0006349. An estimated unemployment rate of 0.005 would lead to a bias value of 688.

18
for women, but still sizeable. When it comes to the unemployment rate estimates, the estimated rank correlations also show improvements when comparing the SAE estimates to the direct estimates, increasing by 0.9 for men and 0.6 for women (from an admittedly low level). However, the estimated Pearson correlations show a marked decline, falling by 0.22 for men and 0.065 for women. This again reflects errors at the left and right tails of the true unemployment rate distribution, which the model does a very poor job of predicting.

|Table 5-Measure|s of Accurac<br>|y for Municip<br>|al Estimates<br>||
|---|---|---|---|---|
||(1)<br>Male<br>LFP|(2)<br>Female<br>LFP|(3)<br>Male<br>Unemp|(4)<br>Female<br>Unemp|
||Mean<br>(median)|Mean<br>(median)|Mean<br>(median)|Mean<br>(median)|
|**Estimated Relative Bias**|||||
|Direct|0.039|-1.017|-4.363|26.429|
||(0.306)|(-1.288)|(-59.069)|(-100.000)|
|SAE|-0.507|1.423|30.732|99.948|
||(-0.622)|(-0.726)|(9.651)|(40.522)|
|**Estimated mean absolute**|||||
|**error**|||||
|Direct|0.061|0.071|0.022|0.017|
||(0.046)|(0.054)|(0.015)|(0.009)|
|SAE|0.028|0.040|0.009|0.006|
||(0.022)|(0.028)|(0.006)|(0.005)|
|**Estimated Rank correlation**|||||
|Direct|0.471|0.623|0.363|0.324|
|SAE|0.541|0.750|0.497|0.264|
|**Estimated Pearson**<br>**correlation**|||||
|Direct|0.493|0.659|0.418|0.385|
|SAE|0.553|0.770|0.415|0.614|
|In-sample municipalities|1,072|1,072|1,072|1,072|

Only in-sample municipalities are included. Reported statistics are unweighted means, medians (in parentheses), or correlations across municipalities. Relative Bias is defined as 100 times the deviation of the estimated value and the true value divided by the true value. Rank or Pearson correlation is the unweighted rank or Pearson correlation between the estimated value and the actual municipal value.

The direct estimates of unemployment actually show negative estimated relative bias. This is likely driven by the fact that most AGEBs with a true unemployment rate of 1 percent are likely to have a sample with an unemployment rate of zero, leading to a consistent underestimate of the true rate. This is clear in Figure 2, which plots bias for male and female unemployment. Values of -100 indicate estimates of zero for unemployment. The direct estimates cluster disproportionately at this lower bound, indicating a large number of zeros. Although this averages out somewhat over municipalities (and different draws, which we return to below), the estimates for a single 19
SAE vs. direct estimates: Relative bias across municipalities Unemployment <!-- Start of picture text --> Male Female<br>Oo Oo<br>S on - oe on oo on = = « os S ©. =a =~ o-_ — =| =—s sae « A<br>lo]w lo)w<br>Qn nO<br>& Rss<br>a a<br>Zo Zor<br>&&<br>®<br>oc cc®<br>fo} fo}<br>ad<br>SAE SAE<br>8 co oe Direct 8) OO Direct<br>Municipalities (ordered by adult population) Municipalities (ordered by adult population)<br><!-- End of picture text -->

Note: Relative bias is defined as the estimate minus the actual census value, divided by the actual census value, times 100. Bias is topcoded at 100 for the sake of presentation.
<u>Table 6 - In-sample direct estimates vs. out-of-sample small area estimates</u>

||(1)<br>Male LFP|(2)<br>Female LFP|(3)<br>Male<br>Unemp|(4)<br>Female<br>Unemp|
|---|---|---|---|---|
|**Average estimated mean**<br>**squared error (x1,000)**|||||
|Direct (in sample only)|9.329|5.494|0.288|0.161|
|SAE (in sample only)|3.331|4.991|0.179|0.093|
|SAE (out of sample only)|6.736|10.458|0.379|0.202|
|**Coverage rate**|||||
|Direct (in sample only)|0.971|0.853|0.409|0.215|
|SAE (in sample only)|0.988|0.990|0.952|0.976|
|SAE (out of sample only)|0.974|0.956|0.940|0.979|
|**Estimated mean absolute error**|||||
|Direct (in sample only)|0.061|0.071|0.022|0.017|
|SAE (in sample only)|0.028|0.040|0.009|0.006|
|SAE (out of sample only)|0.046|0.073|0.013|0.009|
|**Estimated rank correlation**<br>|||||
|Direct (in sample only)|0.471|0.623|0.363|0.324|
|SAE (in sample only)|0.541|0.750|0.497|0.264|
|SAE (out of sample only)|0.225|0.312|0.309|0.207|
|**Estimated Pearson correlation**|||||
|Direct (in sample only)|0.493|0.659|0.418|0.385|
|SAE (in sample only)|0.553|0.770|0.415|0.614|
|SAE (out of sample only)|0.164|0.298|0.163|0.131|
|In-sample municipalities|1,072|1,072|1,072|1,072|
|Out-of-sample municipalities|569|569|569|569|

Coverage rate is equal to one if the true census value is within 1.96 standard errors of the point estimate.

There are several patterns to note. First, looking at estimated mean squared error, estimates for non-sampled municipalities are approximately half as precise as those from sampled municipalities, and - except for male LFPs - are less precise than the direct estimates for sampled municipalities. Second, for unemployment rates, the coverage rate is markedly better for the out-of-sample small area estimates relative to the direct estimates. Third, small area estimates are far less accurate out of sample than in-sample. For example, for female LFP estimated absolute deviation is approximately twice as large and correlations are nearly twice as low for out-of-sample municipalities compared with in-sample municipalities. Comparisons between survey-based insample municipalities and out-of-sample small area estimates are more mixed. With the exception of female LFP, out-of-sample estimates have slightly lower estimated mean absolute deviation than in-sample survey estimates, but far lower correlations.

21
<!-- Start of picture text --> Male LFP Female LFP<br>- 2<br>foe)<br>Oo<br>er eo<br>3 3<br>Ow ot<br>N<br>Oo Oo<br>-.2 -.1 0 1 2 -.2 -.1 0 | 2 3<br>Deviation from actual value Deviation from actual value<br>Male unemployment Female unemployment<br>8%<br><<br>8<br>Es € oO<br>8a 8*<br>° a<br>Oo Oo<br>-.1 -.05 0 .05 | 15 -.05 0 .05 | 15<br>Deviation from actual value Deviation from actual value<br><!-- End of picture text -->

# In sample (SAE) Out of sample (SAE) Direct estimate

All deviations are winsorized at the 1st and 99th percentile for the sake of presentation.
variation in unemployment rates, and also leads the small area estimation procedure to underestimate the variance of the municipal random effect, which in turn gives these inaccurate predictions more weight relative to the survey data in generating the estimates. Therefore, the small area estimates for unemployment suffer from high levels of relative bias and are more weakly correlated with the census values than the direct estimates.

While a non-linear estimation model might improve these estimates, a natural question is whether larger samples might improve the small area estimates for unemployment. This ties into a larger question about the potential benefits of expanding the second stage of samples when combining survey data with alternative data sources. To shed some light on this question, we repeat the small area estimation exercise after simulating a sample in which every household in selected AGEBs is included in the sample. This will reduce measurement error in the dependent variable, which substantially improves the quality of the predictions (Lobell et al., 2021, Engstrom et al., 2021). The exercise in intended to provide an upper bound estimate of the impact of expanding the second stage of the sample. But since listing exercises are common parts of household surveys in many developing countries, it may be possible to include selected questions as part of the listing exercise. The full enumeration results also shed light on the potential benefits of expanding the second stage of samples when combining survey data with alternative data sources.

<u>Table 7 - Sample of sampled AGEBs vs. full enumeration of sampled AGEBs</u>

||(1)<br>Male LFP|(2)<br>Female LFP|(3)<br>Male<br>Unemp|(4)<br>Female<br>Unemp|
|---|---|---|---|---|
||**Panel A: All m**|**unicipalities**|||
|**Average estimated mean**<br>**squared error (x1,000)**|||||
|Random sample SAE|4.512|6.887|0.248|0.131|
|Full enumeration SAE|1.306|3.450|0.147|0.049|
|**Coverage rate**|||||
|<br>Random sample SAE|0.983|0.978|0.948|0.977|
|<br>Full enumeration SAE|0.951|0.943|0.941|0.960|
|**Estimated relative bias**|||||
|**(median)**|||||
|Random sample SAE|-0.759|-0.816|9.858|56.367|
|<br>Full enumeration SAE|-0.510|-0.275|8.363|12.187|
|**Estimated mean absolute**|||||
|**error**|||||
|Random sample SAE|0.034|0.051|0.010|0.007|
|<br>Full enumeration SAE|0.026|0.042|0.008|0.004|
|**Estimated rank Correlation**|||||
|Random sample SAE|0.436|0.659|0.426|0.241|
|Full enumeration SAE|0.673|0.748|0.633|0.599|

# **<u>Panel B: In-sample municipalities</u>**

23
|**Average estimated mean**<br>**squared error (x1,000)**|||||
|---|---|---|---|---|
|Random sample SAE|3.331|4.991|0.179|0.093|
|Full enumeration SAE|0.869|2.014|0.084|0.033|
|Full enumeration direct|0.240|0.510|0.043|0.045|
|**Coverage rate**|||||
|Random sample SAE|0.988|0.990|0.952|0.976|
|Full enumeration SAE|0.970|0.966|0.953|0.965|
|Full enumeration direct|0.543|0.538|0.537|0.547|
|**Estimated median relative**|||||
|**bias**|||||
|Random sample SAE|-0.622|-0.726|9.651|40.522|
|Full enumeration SAE|-0.491|-0.325|7.281|10.281|
|Full enumeration direct|-0.041|0.000|0.000|-0.186|
|**Estimated mean absolute error**|||||
|Random sample SAE|0.028|0.040|0.009|0.006|
|Full enumeration SAE|0.017|0.026|0.006|0.003|
|Full enumeration direct|0.019|0.030|0.006|0.004|
|**Estimated rank correlation**|||||
|Random sample SAE|0.541|0.750|0.497|0.264|
|Full enumeration SAE|0.819|0.866|0.805|0.741|
|Full enumeration direct|0.830|0.852|0.808|0.722|

Table 7 presents summary statistics comparing the sample SAE and full enumeration SAE, as well as the direct estimates using the full enumeration of sampled AGEBs. Panel A compares across all municipalities, while Panel B compares only across in-sample municipalities. Not surprisingly, both the small area estimates and the direct estimates using the full enumeration sample perform much better than the small area estimates using the smaller, more typical, sampling strategy. The main finding from this exercise, however, is that the small area estimation only slightly improves on the direct estimates in this case, except for the coverage rate because uncertainty is estimated more accurately. For example, for female LFP, the small area estimates only reduces absolute deviation by 0.3 pp, from 2.9 to 2.6 pp, and only improves rank correlation from 0.86 to 0.875. For other indicators, the benefit of SAE when the second stage of the sample is fully enumerated is even smaller.

Figure 4 compares the accuracy of the random sample and full enumeration small area estimates, relative to the full census. The blue dots clearly demonstrate that the random sample SAE predictions for unemployment are close to the overall average, due to the poor predictive power of the unemployment models. In contrast, the orange dots representing the full enumeration unemployment results exhibit far more variation and better track the true rate, as do the LFP results for the random sample. This reflects the absence of variation available in the smaller sample to train an accurate model when predicting unemployment rates.

24
<!-- Start of picture text --> LFP<br>Male Female<br>Q «2<br>Random sample SAE Full enumeration SAE Random sample SAE Full enumeration SAE<br>© . @ 03, 82< a3¢ é © ae<br>S . 3 we op PE<br>ao gs n 0% > $ae<br>wo wa :<br>o°<br>56 7 8 9 0 2 4 6 8<br>Actual rate Actual rate<br>Unemployment<br>Male Female<br>8<br>“ Random sample SAE Full enumeration SAE * Random sample SAE Full enumeration SAE<br>2Sg‘°oeg88<br>g6 See 8 Ss ee fe<br>oo<br>002 04 .06 .08 | 0 .01 02 .03 .04 05<br>Actual rate Actual rate<br>All unemployment variables are winsorized at the 99th percentile for the sake of presentation.<br><!-- End of picture text -->
# **4. Robustness checks**

# **A. Design-based simulations**

The results in the previous section use just a single sample to compare results from two estimators: a direct estimator, using just survey results, and a small area estimator, which uses auxiliary information to improve the accuracy and precision of the survey results. However, it is difficult to come to a firm conclusion when using just a single sample. Therefore, we simulate the small area estimates 100 times, using 100 different samples and calculating 100 different point estimates for direct and small area estimates. We do not calculate MSE on each simulation due to computing limitations.

Table 8 - Simulation results for municipal estimates

||(1)<br>Male LFP|(2)<br>Female LFP|(3)<br>Male<br>Unemp|(4)<br>Female<br>Unemp|
|---|---|---|---|---|
||mean<br>(median)|mean<br>(median)|mean<br>(median)|mean<br>(median)|
|**Simulated relative bias**|||||
|Direct|0.186|-0.222|0.128|1.451|
||(0.534)|(-0.302)|(-58.406)|(-100.000)|
|SAE|-0.274|1.755|35.627|71.286|
||(-0.656)|(-0.561)|(11.734)|(24.950)|
|**Simulated mean absolute**|||||
|**error**|||||
|Direct|0.060|0.070|0.023|0.015|
|SAE|0.028|0.040|0.009|0.005|
|**Simulated rank correlation**|||||
|Direct|0.443|0.614|0.366|0.361|
|SAE|0.555|0.742|0.449|0.389|

Rank correlations are the average rank correlations across the 100 simulations.

Table 8 presents three separate statistics – bias, absolute deviation from the census value, and the correlation with the true value – for direct estimates and small area estimates. Simulated mean and median bias is quite small for both direct estimates and small area estimates for LFP, with the highest rate being 2 percent mean bias for female LFP. The means are always higher than the medians, especially for the small area estimates for unemployment, as very low base rates can lead to very high values in some simulations.

26
<!-- Start of picture text --> Male LFP Female LFP<br>w 2<br>foe)<br>~ S<br>ct ~~co<br>3 3<br>Ow ort<br>N<br>Oo oOo<br>-.2 -1 0 A 2 -.2 0 2<br>Deviation from actual value Deviation from actual value<br>Male unemployment Female unemployment<br>Oo fo]<br>wo fee)<br>< 8<br>3° ae<br>S& (Ss)<br>° a<br>Oo oO<br>-1 -.05 0 .05 A 15 -.05 0 .05 A 15<br>Deviation from actual value Deviation from actual value<br>In sample (SAE) Out of sample (SAE) Direct estimate<br>All deviations are winsorized by one percent in each tail for the sake of presentation.<br><!-- End of picture text -->
<!-- Start of picture text --> Male LFP Female LFP<br>°o<br>©<br>~co ~Cc vt<br>FS} 3<br>ot Og<br>N<br>Oo oO<br>-.3 -.2 -.1 0 A 2 -.4 -.2 0 2 4<br>Deviation from actual value Deviation from actual value<br>Male unemployment Female unemployment<br>Oo fo}<br>vt ive)<br>oOo oO©<br>€ €<br>3° 30°<br>(6)ON (6)ov<br>oO- oON<br>fo}A °| J<br>-1 -.05 0 .05 A 15 -1 -.05 0 .05 J 15<br>Deviation from actual value Deviation from actual value<br>In sample (SAE) Out of sample (SAE) Direct estimate<br>All deviations are winsorized by one percent in each tail for the sake of presentation.<br><!-- End of picture text -->
<!-- Start of picture text --> Male LFP Female LFP<br>xo"wo £& wo°<br>=}Bw 5Bw<br>=a<br>22<br><° <°<br>2 2<br>Ba BA<br>>r sor<br>®o<br>are) fa) wo<br>Municipalities Municipalities<br>Male unemp. Female unemp.<br>£ wo wo<br>5<br>=}Ew° £Ew°<br>eS eS<br>£ £<br>———) tan =~) TM :<br>S | rt iS) |<br>BA BA<br>>r Sor<br>o ®o<br>Qa fe) fa) wo<br>Municipalities Municipalities<br>SAE Direct<br><!-- End of picture text -->

The lines show the range of deviations within each municipality across simulations. Deviations are topcoded at 0.5 and bottomcoded at -0.5 for the sake of presentation.
One interesting area of future research will be to determine whether these smaller municipalities are simply more idiosyncratic and, thus, harder to predict, or whether they differ structurally in observables ways. If the latter is true, different types of data or methods might improve estimation for the smallest municipalities. In other words, is it possible to improve the estimation of these municipalities, or are there simply weaker systematic relationships between geospatial data and labor market statistics for these areas?

# **B. Area-level model**

An alternative method for combining survey data and geospatial data is to obtain municipal estimates is an area-level model (Fay and Herriot, 1979). This method is convenient and wellknown, but suffers from a few limitations in this context. The main shortcoming is that it requires aggregating the auxiliary data to the municipal level, which discards variation across AGEBs within municipalities. In addition, the method faces challenges to accurately generate estimates of municipal variance, which is an important input into the model. This is particularly true when there is a significant number of zero values, as is the case for unemployment rates in this context.

To better understand whether these potential limitations are important in practice, we examine the results of two area-level modeling exercises. Fay-Herriot models can either be assumed to be linear models or the dependent variable can be transformed prior to estimation. For ratios such as labor force participation and unemployment rates, it is common to transform the variable using an arcsine transformation (Halbmeier et al, 2019). Use of such a transformation adds additional complexity to the procedure, but ensures that predicted participation or unemployment rates lie between zero and one. In addition, using the arcsine transformation is more consistent with the AGEB-level model discussed above. For these reasons, we use the transformed model as our main results, but display the results of a linear model in Table A4.

We select a model for each outcome by applying LASSO to the full set of municipal candidate geospatial variables, again selecting lambda to minimize BIC. When using the arcsine transformation, the dependent variable is the survey-weighted estimate of the transformed mean outcome variable of interest in each municipality. When using the linear model, no transformation is applied prior to the LASSO. We then calculate the sampling variance of each outcome using the Horvitz-Thompson approximation, as recommended by Halbmeier et al (2019), and fit the model using the restricted information maximum likelihood option of Stata’s fayherriot package. The R<sup>2</sup> s of the transformed models are quite low: 0.06 and 0.19 for male and female LFP, and only 0.10 and 0.06 for male and female unemployment. Even after reducing the noise associated with the unemployment measure by aggregating to the municipality level, the geospatial variables are weak predictors of unemployment rates and male LFP, although they predict cross-municipal variation in female LFP somewhat better.

Table 9 displays the results from the transformed Fay-Herriot (FH) area-level model.<sup>25</sup> We estimate the FH using an arcsine transformation of the target variable, just like with the sub-area

> 25 The table omits MSE because the State fayherriot command used to estimate the FH model does not output standard errors. Instead, it outputs confidence intervals derived from a bootstrap procedure. Because the confidence intervals are asymmetric, calculation of the MSE is not straightforward.

30
(AGEB-level) model. Across all statistics and labor market indicators -- except one -- the arealevel model performs worse than the sub-area model. The exception is the estimated rank correlation for female unemployment, which the AGEB-level model predicts poorly due to the large number of zeroes in the unemployment indicator. The area-level model often performs better than the direct estimates (with absolute deviation, for example), especially for unemployment. Table A4 in the appendix shows the comparison between the transformed and linear Fay-Herriot model. The transformed area-level model generates much more accurate estimates of uncertainty for unemployment and male LFP than the linear area-level model, as the linear model greatly underestimates mean squared error for these indicators, leading to very low coverage rates. However, the results on accuracy are mixed. For example, when examining correlations, the arcsine estimates are more accurate for male LFP but less accurate than the linear model for female LFP. The results suggest that the linear F-H is preferred to the transformed F-H model in the case of female LFP, when the prediction model is better. However, when considering male and female LFP, the AGEB-level model generates more accurate estimates than both area-level model according to all criteria.

<u>Table 9 – Comparison of direct, AGEB-level, and municipal-level (FH)</u>

||(1)<br>Male<br>LFP|(2)<br>Female<br>LFP|(3)<br>Male<br>Unemp|(4)<br>Female<br>Unemp|
|---|---|---|---|---|
|**Coverage rate**|||||
|Direct|0.971|0.853|0.409|0.215|
|AGEB-level model|0.988|0.990|0.952|0.976|
|Transformed area-level model|0.887|0.823|0.804|0.842|
|**Estimated Relative Bias**|||||
|**(median)**|||||
|Direct|0.039|-1.017|-4.363|26.429|
||(0.306)|(-1.288)|(-59.069)|(-100.000)|
|AGEB-level model|-0.507|1.423|30.732|99.948|
||(-0.622)|(-0.726)|(9.651)|(40.522)|
|Transformed Area-level model|1.382|4.058|-37.108|-53.035|
||(0.717)|(-1.784)|(-51.308)|(-76.510)|
|**Estimated mean absolute error**|||||
|**(median)**|||||
|Direct|0.061|0.071|0.022|0.017|
||(0.046)|(0.054)|(0.015)|(0.009)|
|AGEB-level model|0.028|0.040|0.009|0.006|
||(0.022)|(0.028)|(0.006)|(0.005)|
|Transformed Area-level model|0.034|0.053|0.015|0.008|
||(0.025)|(0.040)|(0.010)|(0.006)|

**Estimated rank correlation**

31
|Direct|0.471|0.623|0.363|0.324|
|---|---|---|---|---|
|AGEB-level model|0.541|0.750|0.497|0.264|
|Transformed Area-level model|0.444|0.652|0.294|0.373|
|**Estimated Pearson correlation**|||||
|Direct|0.493|0.659|0.418|0.385|
|AGEB-level model|0.553|0.770|0.415|0.614|
|Transformed Area-level model|0.405|0.619|0.209|0.384|

Table A4 in the appendix also presents a comparison of an area-level model using the transformed target variable and an area-level model using a linear (untransformed) model. The transformed model generally performs better than the linear model, with the possible exception of some female LFP statistics (coverage rates, absolute deviation, and correlations). However, the AGEB-level model remains the best choice in this context.

# **5. Conclusion**

This paper considers the extent to which combining simulated sample data with publicly available geospatial data improves state and municipal estimates of male and female labor force participation and unemployment rates. Results are compared against the full 2020 census. The small area estimation procedure greatly improves the accuracy and precision of state level estimates for all four indicators, as well as municipal estimates of male and female labor force participation. This method can therefore be used to obtain more granular information on municipalities with low LFP, which can lead to a better understanding of the causes of low female LFP and potential policies to address it.

The method does not work nearly as well for estimating urban unemployment, because the values for unemployment rates are very low and zero for many clusters in the sample. In this setting, linear mixed models do not perform well. Although rank correlations and mean absolute deviations both improve, the simple correlation with the census value falls and mean relative bias is very high. In all cases except for one, estimates from a model specified at the AGEB level generate more accurate predictions than those obtained from an area-level model, with the lone exception being when considering the rank correlation for estimated female unemployment rates. Not surprisingly, using a hypothetical enumeration of all households in selected clusters dramatically improves the accuracy of the estimates. But in this case the small area estimates offer very minor improvements in accuracy and precision over direct estimates from the fully enumerated sample.

Non-linear models such as two-part models may be an appealing alternative for small area estimation of low-probability events in a sample. In the two-part model, the first part models the probability of a positive unemployment rate and the second part models the unemployment rate conditional on a positive rate. This is feasible to incorporate in a small area estimation framework, but significant effort would be required to implement it in existing software.

32
Another important area for further work is to continue to experiment with additional forms of geographically comprehensive auxiliary data to better predict labor market outcomes. This could include, for example, information on building footprints or the presence of different types of businesses. New types of geospatial data are being released each year that can be incorporated into this type of approach.

There is room for further work on model diagnostics. The results of this study demonstrate that R<sup>2</sup> can be a misleading metric for assessing the usefulness of the model, since even models with relatively low R<sup>2</sup> s can contribute important information to reduce bias in the predictions (Marhuenda et al., 2017). At the same time, model R<sup>2</sup> s are much higher in the full enumeration sample, but the small area estimates contribute little additional information. An important research agenda for further work is to better understand which diagnostic indicators, in cases where census data is not available, can provide a rough assessment of the gains from incorporating geospatial data.

Another open question is how optimal sample design changes in the presence of free, predictive geospatial data that can be linked to surveys. The evidence presented above indicates that estimates for sampled municipalities are substantially more accurate than non-sampled municipalities, suggesting that surveys should try to cover all target areas if possible. It is not clear, however, how the benefit of small area estimation relates to the sampling structure, and what this implies for optimally structuring household surveys.

Importantly, it remains to be seen how well the results here will generalize to other contexts. As such, it will be important to conduct studies to evaluate the methodology in other settings before applying it generally. This is easier said than done, however. In order to validate the methodology, it is best to use a census – or an unusually large survey –as a measure of ground truth. Furthermore, either the survey or ground truth must have either geolocated enumeration areas or, as in this case, highly disaggregated geographic identifiers that can be matched with a shapefile.

A final important issue pertains to the appropriate weighting strategy for the conditional random effect models relied upon here and in other small area estimation applications. We make three points regarding weights. The first relates to the choice of whether to give equal weight to each municipality or to weight by population when evaluating the estimates. We consider only the former, as is common in the literature on small area estimation. However, in some policy contexts, there is a strong argument that it is more important to generate accurate estimates for more populous areas than less populous areas. This is a decision best made based on political rather than technical grounds.

Second, properly weighing the estimates of a sub-area model such as this one is not trivial. In this case, the weights for each AGEB in the sample were normalized by dividing each weight by its municipal average. This ensured that each municipality was given weight proportional to its sample size, while appropriately giving more weight to more populous AGEBs within municipalities. The latter is important because population weights are used to aggregate AGEBlevel estimates to municipalities. Furthermore, in most sub-area models, it would be beneficial to adjust the weights further to correct for heteroscedasticity, to the extent that the number of 33
observations used to construct the sub-area average used as the dependent variable varies across observations.

Finally, there are different methods for accounting for weights in conditional random effect models. We used a particular method implemented by Pinheiro et al. (2021), although other methods have also been proposed and implemented.<sup>26</sup> Establishing the pros and cons of different approaches to incorporating weights in small area estimation models is an important topic for further research.

> 26 See, for example, Skarke and Kreutzmann (2021), who implement the correction for informative sampling proposed by Guadaramma et al.(2018).

34