metadata: authors: 0: author: Himelein,Kristen
1: author: Eckman,Stephanie
2: author: Murray,Siobhan
3: author: Bauer,Johannes
count: World
docna: 0: docna: Second-stage sampling for conflict areas :
            methods and implications
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,State Owned Enterprise Reform,Public Sector Administrative & Civil
            Service Reform,Public Health Promotion,Inequality,Democratic Government,Economics and Finance of Public
            Institution Development,Public Sector Administrative and Civil
            Service Reform,De Facto Governments,Armed Conflict
teratopic: Poverty Reduction,Conflict and Development
abstracts: cdata!: The collection of survey data from war
            zones or other unstable security situations is vulnerable to
            error because conflict often limits the implementation
            options. Although there are elevated risks throughout the
            process, this paper focuses specifically on challenges to
            frame construction and sample selection. The paper uses
            simulations based on data from the Mogadishu High Frequency
            Survey Pilot to examine the implications of the choice of
            second-stage selection methodology on bias and variance.
            Among the other findings, the simulations show the bias
            introduced by a random walk design leads to the
            underestimation of the poverty headcount by more than 10
            percent.  The paper also discusses the experience of the
            authors in the time required and technical complexity of the
            associated back-office preparation work and weight
            calculations for each method. Finally, as the simulations
            assume perfect implementation of the design, the paper also
            discusses practicality, including the ease of implementation
            and options for remote verification, and outlines areas for
            future research and pilot testing.
repnme: repnme: Second-stage sampling for conflict areas :
            methods and implications
display_title: Second-stage sampling for conflict
            areas : methods and implications
listing_relative_url: /research/2016/03/26144105/second-stage-sampling-conflict-areas-methods-implications
topicv3: State Owned Enterprises m1326323 721,Poverty m1326687 695,Political Violence and War m1327310 2462,Public Health m1327519 635,Health m1504661 3480
publishtoextweb_dt: 2016-03-29T00:00:00Z
historic_topic: Poverty Reduction,Conflict and Development
no_of_pages: 33
author_keys: 0: author_key: 1150711
1: author_key: 892365
2: author_key: 1369961
model_extractions: WPS7617 Policy Research Working Paper 7617 Second-Stage Sampling for Conflict Areas Methods and Implications _Kristen Himelein Stephanie Eckman Siobhan Murray Johannes Bauer_ Poverty and Equity Global Practice Group March 2016
Policy Research Working Paper 7617

# **Abstract**

The collection of survey data from war zones or other unstable security situations is vulnerable to error because conflict often limits the implementation options. Although there are elevated risks throughout the process, this paper focuses specifically on challenges to frame construction and sample selection. The paper uses simulations based on data from the Mogadishu High Frequency Survey Pilot to examine the implications of the choice of second-stage selection methodology on bias and variance. Among the other findings, the simulations show the bias introduced by a random walk design leads to the underestimation of the poverty headcount by more than 10 percent. The paper also discusses the experience of the authors in the time required and technical complexity of the associated back-office preparation work and weight calculations for each method. Finally, as the simulations assume perfect implementation of the design, the paper also discusses practicality, including the ease of implementation and options for remote verification, and outlines areas for future research and pilot testing.

This paper is a product of the Poverty and Equity Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at khimelein@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Second‐Stage Sampling for Conflict Areas: Methods and Implications**

Kristen Himelein, Stephanie Eckman, Siobhan Murray, and Johannes Bauer<sup>1</sup>

## **JEL classification:** C15, C83, F51, I32

> 1 Kristen Himelein is a senior economist / statistician in the Poverty Global Practice at the World Bank. Stephanie Eckman is a senior survey research methodologist at RTI International in Washington, DC. Siobhan Murray is a technical specialist in the Development Economics Data Group in the World Bank. Johannes Bauer is research fellow at the Institute for Sociology, Ludwigs-Maximilians University Munich and at the Institute for Employment Research (IAB).  All views are those of the authors and do not reflect the views of their employers including the World Bank or its member countries. The authors would like to thank Utz Pape and David McKenzie, from the World Bank, and Matthieu Dillais from Altai Consulting, for their comments on earlier drafts of this paper, and Hannah Mautner and Ruben Bach of IAB for their research assistance.
# **Introduction**

The collection of survey data from war zones or other unstable security situations provides important insights into the socioeconomic implications of conflict. Data collected during these periods, however, are vulnerable to error, because conflict often limits the options for survey implementation. For example, the traditional two‐stage sample design for face‐to‐face surveys in most developing countries first selects census enumeration areas as the primary sampling unit (PSU) with probability proportional to size and then conducts a listing operation to create a frame of households from which a sample is selected. Such an approach, however, may not be feasible in conflict areas. At the first stage, updated counts are often not available, making probability proportional to size selection inefficient. Also as the second stage requires that survey staff canvas the entire selected area, it may also be too dangerous in a conflict setting. As a result, many surveys of conflict areas are limited to qualitative work or resort to non‐probability designs.

This paper uses simulations to explore several alternative sampling approaches considered for the baseline of the Mogadishu High Frequency Survey Pilot (MHFS). The baseline was a face‐to‐face household survey in Mogadishu, Somalia, conducted from October to December 2014 by World Bank and Altai Consulting. A full listing (see Harter et al, 2010 for details) was deemed unsafe in Mogadishu because the additional time in the field and the predictable movements by interviewers would increase their exposure to robbery, kidnapping, and assault, and increase the likelihood that the local militias would object to their presence. The survey needed an alternative second‐stage sample design that would minimize the time spent in the field outside the households, but also could be implemented without expensive equipment or extensive technical training. In addition, international supervisors from the consulting firm could not go to the field, necessitating a sample design in which quality could be verified ex post.

The implementing partner originally proposed a random walk procedure. While this methodology has the benefits of fast implementation and unpredictability of movement, the method is non‐probability and literature has shown the procedure to give biased results, even if implemented under perfect conditions (Bauer, 2014). Intuitively, a random walk would only be unbiased if the paths taken during the selection crossed each household once and only once, which is extremely unlikely in the field. Therefore the team considered four alternatives for household selection. The first option was to use a satellite map (of which many high quality options exist, due the limited cloud cover and political importance of the region) to identify all structures in the PSU and select ten for the survey. The second option considered was to subdivide the selected PSUs into segments consisting of eight to ten households and ask enumerators to list and choose households from the segments.<sup>2</sup> The segments would be of roughly equal size in terms of number of households but are likely to have irregular outlines reflecting the irregular layout of structures in Mogadishu. The third option considered was to lay a uniform grid over the PSU and ask enumerators to list and choose households from selected grid boxes. The final option considered was to start at a random point in the cluster and walk in a set direction, in this case the Qibla, or direction in which Muslims pray, until the interviewer encountered a structure.

> 2 Interviewers had a selection application on their smart phones that they used whenever subsampling was needed.

2
The paper will make use of data from the completed Mogadishu pilot survey and geo‐referenced maps and three example EAs to explore the following questions: (1) How might a given method be implemented in the field, given the information available and the security constraints? (2) What information is necessary to generate sampling weights, necessary for representative estimates, and how should those weights be calculated? (3) What are the implications in terms of precision and bias for each of the methods described above? (4) What are the implementation concerns for each method, including the options for verification and the impact of non‐household structures?

The next section briefly describes the literature as it relates to the questions above, followed by section 2 describing the data. Section 3 addresses research questions 1 and 2 by giving further detail on the methods considered. Section 4 presents simulation results covering questions 3 and 4. Section 5 concludes by offering some discussion of the overall performance and potential future applications.

# **1. Literature Review**

The most common method for collecting household data in Sub‐Saharan Africa is to use a stratified two‐ stage sample, with census enumeration areas selected proportional to size in the first stage and a set number of households selected with simple random sampling in the second stage (Grosh and Munoz, 1996). Since administrative records are often incomplete and most structures do not have postal addresses, as is the case in Mogadishu, a household listing operation is usually necessary prior to the second‐stage selection. However, due to the security concerns cited above, listing was not feasible.

A number of alternatives for second stage selection can be used when household lists are not available. A common alternative used in both Europe (see Bauer, 2014, for recent examples) and in the developing world is a random‐walk. The Afrobarometer survey, which has been conducted in multiple rounds in 35 African countries since 1999, and the Gallup World Poll, which conducted surveys in 29 Sub‐Saharan African countries in 2012, use random walk methodologies. Although the random walk methods do not necessarily produce equal probability samples, they do not collect any information with which to calculate probabilities of selection. For this reason, weights are not calculable for random walk samples; instead, the samples are analyzed as if they were equal probability. Bauer (2014) shows that this assumption is not correct by simulating all possible random routes using standard procedures within a German city and finds substantial deviation from equal probability. These results apply even when interviewers perfectly implement the routing instructions, which is unlikely given the limited ability to conduct in‐field supervision of random walk selection and strong (though understandable) incentives for interviewers to select respondents who are willing to participate (Alt et al 1991). Several other studies have also shown that data collected via random walk do not match the population on basic demographics such as age, sex, education, household size, and marital status (Bien 1997, Hoffmeyer‐Zlotnick 2003, Blohm 2006, Eckman & Koch 2016).

In the context of Mogadishu, a household listing was too dangerous and costly, a random walk too biased, and no household or person register existed. Therefore the researchers explored several alternative methods using a combination of satellite maps and area‐based sampling. As satellite technology has improved in quality and become more readily available, it has been increasingly used for research in the developing world. Barry and Rüther (2001) and Turkstra and Raithelhuber (2004) use satellite imagery to study informal urban settlements in South Africa and Kenya, respectively. Aminipouri et al (2009) use 3
samples from high resolution satellite imagery to estimate slum populations in Dar‐es‐Salaam, Tanzania. Afzal et al (2015) incorporated satellite data into poverty prediction modeling for Pakistan and Sri Lanka, and concluded that its inclusion can lead to substantial improvements in modeled estimates of poverty.

Specifically related to sampling, the literature is more limited and mainly found in the public health literature. Dreiling et al (2009) tested the use of satellite images for household selection in rural counties of South Dakota, and found, while less time consuming, the method did a poor job of identifying the inhabited dwellings. Grais et al (2007) used a random point selection methodology in their study of vaccination rates in urban Niger, and compared the results to a random walk. They do not find statistically significant differences between the methods, though the sample size was limited, but conclude that interviewers found the random point selection methods more straightforward to implement than the random walk. Lowther et al (2009) used satellite imagery to map more than 16,000 households in urban Zambia to select young children for a measles prevalence survey. They find the method easy to implement, but do not do a formal comparison with alternatives. Kondo et al (2014) use a point selection mechanism in the city of Sanitiago Atitlán similar to our proposed Qibla method, but assume the method to be equal probability. A similar method was used by Kumar (2007), in which satellite maps overlaid with remote sensing data were used to create stratification for an air pollution study in India, and then selected random points. Kolbe and Hutson (2006) use a similar method to select households in Port‐au‐Prince, Haiti. Their study incorporates the probability of selection for randomly selecting from nearby structures when the selected points do not fall on the roof of a dwelling, but these probabilities would be distorted if the household density were unevenly distributed or for households close to the boundary. Other public health studies, such as the World Health Organization Expanded Program on Immunization studies, use the “spin the pen” method to choose a starting household and then interview a tight cluster of households, though this method has been shown to be nonprobability (Bennett et al 1994, Grais et al 2007).

Outside the field of public health, Himelein et al (2014) used circles generated around random points to survey pastoralist populations in eastern Ethiopia, with the stratification developed from satellite imagery. A variation of this method was considered for the High Frequency Survey Pilot, but the methodology is likely unsuited to a dense metropolitan area, because it involves surveying all households within the selected circles. The uncertainty over the final cluster size was also an issue in Mogadishu as clusters with too few households increased costs, while clusters with too many households increased the time in the field and raised security concerns.

This paper brings together alternatives developed from this literature and applies them to a conflict environment. We take a rigorous approach using simulations and careful estimation of weights to compare the methods across a variety of potential field conditions. The results offer general guidelines for practitioners developing implementation plans for conflict settings.

# **2. Data**

To explore the challenges of the random walk and the four proposed alternatives, we simulated the use of each method in three example PSUs from Mogadishu, Somalia. We purposefully chose three census enumeration areas as the PSUs for this exercise to illustrate the variation in physical layout present in 4
<!-- Start of picture text --> Nn<br>Ss<br>wo<br>Ss<br>S<br>w<br>So<br>Ss<br>Oo<br>) 100 200 300 400<br><!-- End of picture text -->
# **3. Sampling Methodologies**

For each of the methods (satellite mapping, segmentation, grid squares, Qibla method, and random walk) and assignment mechanisms (random assignment, spatial clustering, and extreme spatial clustering), 10,000 simulated samples were drawn and relevant probability weights calculated. Each sample consisted of ten structures per PSU. For the cases in which the household sample was selected in two stages, two segments of five households were chosen within each PSU for segmentation, and two grid boxes of up to five households were randomly chosen in the grid square selection. This section provides further detail on the selection methods and describes the weight calculations necessary to achieve unbiased results.

# _3.1 Satellite mapping_

A full mapping of the PSU entails using satellite maps to identify the outline of each structure (see appendix). In this case, we used maps publically available on Google Earth and maps of the EA boundaries provided by the Somali Directorate of Statistics. From these maps, the structures inside each PSU can be assigned numbers (either by hand or digitally) and selected easily in the office with simple random sampling. The coordinates of the selected households can be loaded onto GPS devices to assist interviewers’ locating efforts.

This approach is the closest of the proposed study methods to the gold standard of a well‐implemented full household listing. The main differences are that in a field listing, enumerators can exclude ineligible structures, such as uninhabited and commercial buildings, and include information not available from satellite maps, such as the identification of individual units within multi‐household structures. In addition, whereas listing is always done just before the data collection, the satellite maps may be out of date, leading to under‐coverage of newly constructed units and/or selection of units that no longer exist. As noted in the previous section, the maps used for this paper were about one year old at the time of the initial analysis. The historical imagery from Google Earth indicates these maps are generally updated at least once a year, but this may vary substantially depending on the specific location and year. Selection from a satellite mapping therefore requires an additional set of field protocols for addressing and documenting the above issues.

The calculation of the probability of selection, and by extension the survey weight, is straight‐forward. � The probability is simply ~~,~~ where _n_ is the number of structures selected and _N_ is the total number of � structures, plus any necessary adjustment for multi‐household structures (for example, one unit from a three unit building) encountered in the field.

# _3.2 Segmenting_

Segmenting is a standard field procedure of subdividing large PSUs into smaller units, approximately equal sized in terms of number of households, for listing and selection purposes. The individual segments are

> estimating the true mean in the population, it should not affect the means when compared between methods.  First, as this bias is associated with a selected household’s decision to participate, it would impact all methods equally. Secondly, as non-response attenuates any differences, they would appear smaller in magnitude in the simulations than in the true population.  To address this, we ran a large number of simulations to generate narrow confidence intervals on the results.

6
<!-- Start of picture text --> Baa<br><!-- End of picture text -->
The calculation of the probabilities of _Source: Authors’ diagram based on PSU boundaries and_ selection is also straightforward: it is the _Google Earth images_ product of the probability of selection of the segment and the probability of selection of the household within the segment. The additional clustering introduced by this method, however, could decrease the precision of estimates due to the design effects. The magnitude of the decrease in precision would depend on the number of segments selected, the number of households selected per segment, and the degree of homogeneity within PSUs for the study variables. The impact would largest if all ten households were selected from the same segment, and decrease as more segments were selected. At the other extreme, if one household were selected from ten segments, segmentation would produce more precise estimates than simple random sampling as the segmentation prevents a chance geographic concentration of selected households. As a balance between efficiency and practicality, two segments and five households per segment were selected for the simulations.

# _3.3 Grid Squares_

To implement the grid method, a uniform grid of squares (or other uniform shape) is overlaid on the PSU map. Figure 1 shows an example using 50 x 50 meter squares for the Dharkinley PSU. The area of a grid square includes all of the area that lies both within the grid square and within the PSU boundaries. For example, in grid square 17 in figure 1, the majority of the structures inside the square would not be eligible for the survey, as they lay outside of the PSU boundaries. Only the structures which lie in the bottom left corner are both within the grid and PSU boundaries.

One or more squares can then be selected with simple random sampling from the set of all squares that overlap the selected PSU. Depending on the survey protocols, a structure may be defined as eligible if all or part of it lies within the grid space. The more common protocol, including the structure if the majority lies within the grid square, has the benefit of simplifying the weight calculations, but the risk of subjective decisions made by interviewers in the field about where the majority of the building lies, which could lead to some buildings having no chance of selection. Since the options for supervision and field re‐verification were limited in this survey, it was decided to consider the structure as eligible if any portion of the structure lay within the grid boundaries, to ensure that all units had a positive probability of selection.

To select a sample of households within the selected squares, a common approach would be for interviews to be conducted with all eligible respondents within the grid square. This could lead, however, to issues with verification as well as decreasing control over the final total sample size. Therefore the protocol used in Mogadishu had interviewers list all households within the selected squares and use the application on their smart phones to select a fixed number of households for the survey.

This variation of the grid method has the advantage that it requires less preparation time compared to mapping or segmenting. There are considerable drawbacks, however, in the ease of implementation and additional work to accurately calculate the selection probabilities. Since the grid squares do not follow visible landmarks, the boundaries must therefore be programmed into the GPS and identified by the interviewers. As it is unlikely that they will be able to walk straight along the boundary, additional training may be required to correctly identify eligible structures.

8
MuPsU ~~nj RY~~ <u>'N- G</u> ~~s~~ Ss vnc = ~~(eS)~~ 4 ~~(Me S~~ \_ ~~(M Me S,Sot~~ “psu <u>\n Ss} \n, S) \N; Ny S S—1</u>
<!-- Start of picture text -->
|<br>x SSFy be ry 7 = —/<br>le<br>i~ /<br>a a -<br>12a/ ><br>4 523 ‘a<br>530<br>rs<br><!-- End of picture text -->

Tipsu = 1— (1 ~ <u>rotalAreaareaofof A;PSU)</u> .
<!-- Start of picture text -->
|<br>(490 ee<br>= |<br>NS —<br>| 530<br>a<br><!-- End of picture text -->
<!-- Start of picture text --> we,<br>ad ~<br>LIRS<br>ws :<br>Sk .<br>> Lf<br>Ty<br>4<br>2<br><!-- End of picture text --> <!-- Start of picture text --> ws :<br>Sk .<br>> Lf<br>Ty<br>4<br>2<br><!-- End of picture text -->
Fieldworker 3.... Walking in their designated direction away from the SSP, they will select the fifth household for their first interview, counting houses on both the right and the left (and starting with those on the right if they are opposite each other). Once they leave their first interview, they will continue on in the same direction, and select the tenth household (i.e., counting off an interval of ten more households), again counting houses on both the right and the left. If the settlement comes to an end and there are no more houses, the Fieldworker should turn at right angles to the right and keep walking, continuing to count until finding the tenth dwelling” (Afrobarometer, pg. 35).

To simulate the random walk in the Mogadishu context, we replicate the Afrobarometer protocols to the extent possible. First we selected a random starting point (since it is not possible to identify landmarks with the level of detail available on the maps, we simply use a random point as the path start). To simulate the direction of the sun, a random angle is chosen and the direction of the interviewer’s path assigned at 90 degree intervals. For example, if 13 degrees from due north was selected, then the four paths would be at 13 degrees, 103 degrees, 193 degrees, and 283 degrees. From these lines, it was assumed that every dwelling within five meters on either side of the direction of walking was within the interviewer’s line of sight. These dwellings were sequentially numbered and every fifth dwelling selected. If the interviewer reached the PSU boundary before selecting the requisite number of households, the path made a 90 degree turn and continued. If each of the four interviewers selected three households, the total cluster size would be twelve. In order to ensure comparability with the other methods, each of which aimed to select ten households, we dropped the last two selected households.<sup>4</sup>

**4. Results**

# _4.1 Simulations_

For each of the sampling methods discussed above and the three different methods of allocating consumption values to households (random, some spatial clustering, extreme clustering), we simulated 10,000 samples and calculated the mean for each one. We report in table A2 in the appendix the mean, standard deviation,<sup>5</sup> 5<sup>th</sup> percentile, and 95<sup>th</sup> percentile of the distribution across all 10,000 samples and evaluate the different sampling approaches in terms of their bias and variance. If a sampling method is unbiased, the expected value of the sample means should be 40, by design the true mean consumption in each simulated PSU.

While generally it was possible to implement all of the methods in our simulations, there were notable challenges with three of the designs. In simulating the Qibla method, certain selected points did not lead to a selection within the EA. The impact was largely negligible in Heliwa or Hodon, where only 0.4 percent and 1.4 percent, respectively, of the total area led to no selection, but in Dharkinley, the smallest and most regular of the PSUs tested, 13 percent of the area led to no selection within the PSU, substantially decreasing the efficiency of that method. Then in the implementation the grid selection method, there was little control over the number of households in each grid square. In some cases, grid squares were

> 4 The analogous action in the field would be for the supervisor to rotate the additional interviews between interviewers to assure an even workload, though most likely in the design stage the cluster size would have been set to be evenly divisible among interviewers.

> 5 The standard deviation of the distribution is the standard error of the estimate of the mean.

13
|28<br>||228<br>|20 <br>||p|<br>|28<br>|face<br><br>||<br>|<br>||<br>|<br>|<br>|<br>|<br>||<br>|<br>|<br>|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|<br><br>|<br>|2<br>|<br>ase<br>/|<br>|<br>fan<br><br>|<br>me<br>»<br>|<br><br>|<br>|<br>||<br>[pve <br><br>|<br>Lr<br>~~<br><br> <br>||<br>zs<br>|<br>a<br>|<br>zs<br>|<br>zz<br>|<br>2s<br>|<br> <br>|<br>a<br> <br>|<br>2<br>|<br>a|<br>|
|<br>10|<br>|54|<br><br>of|<br><br>|e|<br><br> t|<br><br>d<br>s||<br> 56|<br> <br>Lima<br>|va|<br> <br><br>f**e**<br>P<br>|<br>|F|<br><br>a|<br><br>[at|<br> <br>20|<br> <br> ae|<br><br>|24||<br>a<br>|<br><br>|2<br>||<br> <br>2|<br> <br>|a||
|<br>171 <br>|<br> |172<br>/|<br> <br>|<br>1173 <br> <br>|<br>|474 <br>~<br>|<br>| 1755<br>Pa<br>|<br>9176|<br>|<br> 177 <br>|<br> <br>|478 <br>|<br> <br><br>|179 <br>|<br> |480 <br><br>|<br>| 181 |<br><br>|<br> 182 <br><br>|<br> ie<br>|<br>184<br><br>|<br><br>|<br><br>186 <br>|<br><br>|187 <br>|<br>|188<br>|<br>|
|152 <br>||<br>7—<br>|154 <br>T<br>||155.<br><br>|7156 <br><br>|$157¢<br>|158<br><br>|:<br>|159<br>|1604|:<br>161<br>|—<br>162||f<br> 163<br>|7|'164 <br>|MN<br>195|[Te|167|<br>|168 <br><br>|169 <br><br>||170|<br>|
|<br>183)<br><br>|<br><br> 134°] <br><br>|<br><br>35 <br>|<br>w<br>| 486<br>><br><br>|<br>ae<br>|137<br>|<br>|138-1<br>2<br>|<br>an<br>139 <br>fe<br><br>|<br>7140||~|<br>7<br>|<br>144<br>ip<br><br>|<br>|145<br><br>|<br>|<br>|<br>148 <br>2<br><br>|<br>|<br>Ts<br><br>|<br><br>o<br>|<br>|151,<br><br> <br>|
|r<br>/|<br>Ly<br>ar|<br>"6|<br>“a<br>117|<br>|118|<br> | 119R|<br>@<br>ago||||i t<br>124|<br>y<br>|'125 ||<br>>.<br>426"||12|7.)128|<br><br> 129|<br>y<br>a|<br>|<br>><br> <br>j|
||<br>ol»<br>|<br>|<br>|<br>&<br>?<br>|<br>||n><br>|2<br>||> >,<br>|><br>|lay<br>||<br>|<br>|<br>a7<br>|<br>|<br><br>|
|B<br>||d<br>||g<br>|e<br>|<br>|o<br>|eo<br>|m<br>|ri<br>|c<br>|i<br>|cn<br>|c<br>|ic<br>|i<br>|e<br>|y.<br>|
|fo|[=|[|l|o|e|e|Le|o|||[o||S|e|||m|fay||
|<br>|<br><br>|<br>||<br>42<br>|<br>|<br>|<br>IS<br><br>|<br>NA"<br>PP<br>|<br>47 <br><br>|<br>| 48<br><br>|<br>|<br>|Be<br>|<br>||<br><br>|<br><br>||
|9 <br>||2<br>|<br>|2 <br>|a<br>|m<br>||2 <br>|2<br>||<br>|<br><br>|Se<br>or <br><br>|v<br>|e <br><br>|e<br>|<br><br>|pf<br><br><br>|r)<br>|<br><br><br>||<br><br>|w|<br><br>|m<br><br>|VA<br>[fs <br><br>|||<br><br>||
|<br>ot|<br> <br>t|<br><sup>d</sup>|<br>|<br><sup>e</sup>|<br><sup>t</sup>|<br><sup>e</sup>||<br><sup>t</sup><br>de|<br><sup>e</sup><br>le|<br><br>foi|<sup>t</sup><br>e]|~~<br><sup>s</sup><br> 2|<br><br>f|<br><sup>t</sup><br>e]|<br><sup>e</sup><br>ef|<br><sup>t</sup><br>6|<br><br>[e|<sup>r</sup><br>e||

<!-- Start of picture text --> detetetstetr dele foie] 2 fe] ef 6 [ee |<br><!-- End of picture text -->
as overall.<sup>6</sup> From this table, we can evaluate how well each method worked in terms of bias and variance. From a true mean of 40, it was unsurprising that the full listing / satellite mapping method showed the most consistently efficient and unbiased results. Segmentation also showed consistently unbiased results but had higher variance for higher degrees of clustering in the underlying distribution due to homogeneity within the segments. The Qibla method with the full weights yielded unbiased results but with wide confidence intervals, though these are likely artificially wide in the simulations. (See note in the technical appendix for more detail.) In addition, the wide confidence intervals are partially driven by a few outlier values. The values of the 5<sup>th</sup> and 95<sup>th</sup> percentiles of the distribution for this method are similar to those in the segmentation method when clustering is applied. The two methods of estimating the measure of size for the Qibla method showed a small amount of bias, ranging between 1.5 and 6.5 percent depending on the degree of clustering. There is also evidence of the trade‐off between bias and variance. The weights for the proxy methods are based on where the random point is selected, which is necessarily shorter than the full shadow width, truncating the values of the weights. While this introduces a bias into the measures, it also limits the possibility of having large weights for outlier values, which increase the variance. The width of the confidence interval showed almost no impact when clustering was introduced. There was also little difference in terms of bias and variance between proxy weights 1 and proxy weights 2, indicating there is little information lost if the categorical method is implemented correctly. The unweighted version consistently underestimated the true mean, though showed narrow confidence intervals, due to the weighting loss, or the increase in variance resulting from the application of weights (see Eckman and West, forthcoming, and Kish 11.2C, 1965 for further discussion). The final two sampling methods both over‐estimate the means with a bias up to ten percent for the clustered distributions. This is most likely due to grid squares which do not have the required number of dwellings, so that the final sample size did not reach 10. The random walk as noted above, is not theoretically unbiased and this is reflected in the simulation results.

Across the three PSUs, there are also some important differences in the methods, as shown in the violin graphs in figure A8 and A9 in the appendix. Dharkinley, despite being the most regular in terms of layout, was problematic for many of the methods. Satellite mapping, segmentation, and random walk all showed a second bulge in the distribution about 20 percent above the true mean, as compared to an expected smooth normal distribution. The full weighting scenarios for the Qibla method also had the most difficulties in Dharkinley, generating a small number of outlier estimates over 1000 compared to a true mean of 40. In contrast in the chaotic Hodon, there were no issues with satellite mapping and the full weight Qibla method estimates were on par in precision with segmentation for the clustered distributions. The Qibla proxy methods, however, showed substantial bias when clustering was introduced with only a slight decrease in variance. Random walk also had substantial difficulties in Hodon, showing both high levels of bias and variance. The bias is caused by the interviewer instructions, which predefine the path an interviewer has to take. Even though the starting location is randomly selected, interviewers tend to reach certain areas with a higher likelihood. Estimates for variables which are correlated with these unevenly distributed selection probabilities are biased.

> 6 “Overall” is defined as a constructed population with three equally weighted strata rather than randomly selected PSUs with their own probabilities of selection.

15
# ~~<u><mark>tted da in</mark> aes ol</u>~~
# ~~<mark>Abel</mark>~~
# _4.3 Ease of Implementation and Remote Supervision_

In conflict and capacity constrained environments, such as Mogadishu, the ease of implementation and options for remote supervision were also necessary considerations in the selection of the final methodology. Satellite mapping requires little specialized training beyond the use of a GPS device for navigation as target households were selected in the office. The Qibla and random walk methods similarly require the ability to navigate with a GPS to the selected point, but also require additional training for interviewers to correctly implement household selection protocols, which are substantially more complex with random walk. The proxy weights version of the Qibla methods also require interviewers to be training on using GPS for field measurements. Segmentation and grid methods are the most difficult to implement in the field as they require interviewers to identify the boundaries of sub‐sections, which in the case of the grid method may not follow landmarks and may cross through structures.

For the purposes of remote verification, the two main GPS‐based tools available to for supervision are waypoints and tracks. Waypoints record the latitude and longitude coordinates of a given location while track records the path of the GPS from the time it was activated. The satellite mapping can be effectively supervised remotely with waypoints. The point recorded by the interviewer when they arrive at the household can be compared to the coordinates of the selected household to ensure they correspond to the same structure. This would be most effective in sparsely populated spaces with little overhead obstruction. Verification would be more difficult in dense urban areas where the minimum of � 15 feet (5 meters) accuracy of the GPS could lead to multiple possible structures, or if heavy overhead cover of metal roofs blocks GPS signals. The Qibla and random walk methods would both use a way point to identify the starting point then the track to confirm the path taken. Grid and segmentation can both use waypoints to confirm points are within selected areas, and it may also be possible to use tracks to confirm the listing process if strict protocols are used (ie. Start in the NE corner and continue clockwise) though the intersection of the interviewers paths may make results less clear.

# _4.4 Replacements_

Due to high transportation costs, most surveys in the developing world use replacements for non‐ response due to refusals or out of sample selections. This is done either through selecting additional households from the PSU listing exercise, as is recommended in the World Bank’s Living Standards Measurement Study (Grosh and Munoz, 1996), or selecting a neighboring structure based on field protocols, such as selecting the dwelling immediately to the right (Lowther et al, 2009). While replacements for out of sample selections with new random points does not introduce bias, it is inefficient and increases costs. For non‐response due to refusal, it is likely to be non‐random, and therefore replacements will create at least some degree of bias in the data. The reason and method for the replacement may influence the degree. If refusals tend to come from households in the highest and lowest wealth quintiles, as the opportunity cost of their time is high, and replacements come from the main part of the distribution, the use of replacements will attenuate the variation in the sample. This may cause the results to underestimate measures such as inequality that depend on accurately capturing the extremes of the distribution. When using a replacement method that uses near neighbors, if structures
are abandoned or commercial buildings, those households living adjacent may be systematically different from the remainder of the PSU. In addition, those households near the boundary of the PSU would have a lower probability of selection since there are fewer households near them that would lead to them being selected as replacements.

Of the methods discussed above, segmenting and gridding require a short listing exercise at which time non‐eligible structures can be excluded. Satellite mapping and the Qibla method rely on maps that cannot differentiate based on eligibility, and are therefore more vulnerable to issues with out of sample selections. In addition, regardless of method, the survey protocols should address procedures for the inevitable refusals, which may be more likely in conflict areas.

# **5. Discussion**

Ultimately the most appropriate method for second stage sampling in any survey depends on a trade‐off between cost, necessary precision, and tolerable bias. In conflict zones, these decisions are further complicated by time pressures, available back office resources, and security concerns. Satellite mapping, segmentation, and the Qibla methods with full area weights are all probability methods for which it is possible (though necessarily not easy) to calculate weights, and thus all produce unbiased estimates of the population mean. Of these options, the simulations demonstrated that satellite mapping yielded the most consistently unbiased and efficient design, under the assumption that recent maps are available and potential issues with out‐of‐sample buildings can be adequately addressed. The Qibla method provides promising results in the simulations but has yet to be tried in the field. The proxy weight variations of the Qibla method also show promise as they remove the requirement of updated satellite maps and greatly reduce the calculation burden for the weights, but do show substantial bias in certain circumstances. The non‐probability methods, random walk and the unweighted Qibla method, do not produce unbiased results. Random walk, in particular, did not perform well in the simulations despite being common practice for many surveys.

The simulations also showed the implications of bias in the estimates can be substantial in terms of policy conclusions drawn from the data. In this study, the main indicator was household consumption, which underpins poverty calculations in much of the developing world. For a hypothetical poverty line set at the bottom 40 percent of the population, the bias resulting from using a random walk over satellite mapping would lead to an under‐estimation of a poverty rate by five percentage points. Given the expanding availability of satellite maps and decreasing costs of GPS technology, much of which is integrated into the phones and tablets used by interviewers, alternative methods based on probability sampling may reduce bias with little impact on cost or complexity of implementation.

Beyond the simulated results, a number of questions remain that can only be addressed by field testing. For example, it is not possible to discuss the cost considerations of the choice of method nor to comparatively discuss the implications on interviewer safety. Also, the simulations assume perfect implementation and further research is needed on the implications of human error or of outdated maps.

19
In the case specifically discussed here, the Mogadishu High Frequency Survey Pilot, the team opted to use segmentation as a compromise between preparation time, ease of implementation, and the time and complexity necessary for the weight calculations. The implementation was generally successful despite a number of difficulties in the field. Teams occasionally encountered high‐level security threats and exploitative rent‐seeking from local leadership. The complexity of the survey protocols, including the sampling design, slowed the implementation of the survey. Also a substantial number of observations had to be discarded because the interviewed points did not fall within the boundaries of the selected segments or because interviewed households did not appear on segment listing forms. Regardless of these challenges, however, it was possible implement a complex and yet rapid, high‐quality survey in one of the most challenging urban contexts known to date.

20