metadata: authors: 0: author: Demombynes,Gabriel
1: author: Gubbins,Paul Michael
2: author: Romeo,Alessandro
count: South Sudan
docna: 0: docna: Challenges and opportunities of mobile
            phone-based data collection : evidence from South Sudan
docty: Policy Research Working Paper
subtopic: Telecommunications Infrastructure,Educational Sciences,Gender and Development,Hydrology,Inequality
teratopic: Private Sector Development,Information and Communication Technologies,Social Development,Communities and Human Settlements
abstracts: cdata!: The proliferation of mobile phones in
            developing countries has generated a wave of interest in
            collecting high-frequency socioeconomic surveys using this
            technology. This paper considers lessons from one such
            survey effort in a difficult environment -- the South Sudan
            Experimental Phone Survey, which gathered data on living
            conditions, access to services, and citizen attitudes via
            monthly interviews by phones provided to respondents.
            Non-response, particularly in later rounds of the survey,
            was a substantial problem, largely due to erratic
            functioning of the mobile network. However, selection due to
            non-response does not appear to have markedly affected
            survey results. Response rates were much higher for
            respondents who owned their own phones. Both compensation
            provided to respondents in the form of airtime and the type
            of phone (solar-charged or traditional) were varied
            experimentally. The type of phone was uncorrelated with
            response rates and, contrary to expectation, attrition was
            slightly higher for those receiving the higher level of
            compensation. The South Sudan Experimental Phone Survey
            experience suggests that mobile phones can be a viable means
            of data collection for some purposes, that calling people on
            their own phones is preferred to handing out phones, and
            that careful attention should be given to the potential for
            selective non-response.
repnme: repnme: Challenges and opportunities of mobile
            phone-based data collection : evidence from South Sudan
display_title: Challenges and opportunities of
            mobile phone-based data collection : evidence from South Sudan
listing_relative_url: /research/2013/01/17186643/challenges-opportunities-mobile-phone-based-data-collection-evidence-south-sudan
topicv3: Water m1326522 137,Poverty m1326687 695,Education m1327048 470,Gender m1327621 615,Mobile Phone m1505675 3092
publishtoextweb_dt: 2013-01-01T00:00:00Z
historic_topic: Private Sector Development,Information and Communication Technologies,Social Development,Communities and Human Settlements
no_of_pages: 33
author_keys: 0: author_key: 145125
1: author_key: 1104757
2: author_key: 1075006
model_extractions: WPS6321 Policy Research Working Paper 6321 Challenges and Opportunities of Mobile Phone-Based Data Collection Evidence from South Sudan _Gabriel Demombynes Paul Gubbins Alessandro Romeo_ The World Bank Africa Region Poverty Reduction and Economic Management Unit January 2013
Policy Research Working Paper 6321

# **Abstract**

The proliferation of mobile phones in developing countries has generated a wave of interest in collecting high-frequency socioeconomic surveys using this technology. This paper considers lessons from one such survey effort in a difficult environment—the South Sudan Experimental Phone Survey, which gathered data on living conditions, access to services, and citizen attitudes via monthly interviews by phones provided to respondents. Non-response, particularly in later rounds of the survey, was a substantial problem, largely due to erratic functioning of the mobile network. However, selection due to non-response does not appear to have markedly affected survey results. Response rates were much higher for respondents who owned their own phones. Both compensation provided to respondents in the form of airtime and the type of phone (solarcharged or traditional) were varied experimentally. The type of phone was uncorrelated with response rates and, contrary to expectation, attrition was slightly higher for those receiving the higher level of compensation. The South Sudan Experimental Phone Survey experience suggests that mobile phones can be a viable means of data collection for some purposes, that calling people on their own phones is preferred to handing out phones, and that careful attention should be given to the potential for selective non-response.

This paper is a product of the Poverty Reduction and Economic Management Unit, Africa Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at gdemombynes@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Challenges and opportunities of mobile phone-based data collection: Evidence from South Sudan**

Gabriel Demombynes<sup>1</sup> Paul Gubbins<sup>1</sup> Alessandro Romeo<sup>2</sup> JEL Codes: C81, C83, D04 Keywords: South Sudan, mobile phone, survey methodology Sector Board: POV Affiliations:<sup>1</sup> World Bank<sup>2</sup> University of Rome “Tor Vergata” The authors greatly appreciate the collaboration of both the South Sudan National Bureau of Statistics and Horizon Contact Centers in Nairobi on the project described in this paper. The authors thank Andrew Dabalen, Johannes Hoogeveen, and Alvin Etan Ndip for helpful comments. The views expressed in this paper are the authors’ alone. They do not necessarily reflect the views of the World Bank or its Executive Directors. Demombynes and Gubbins are part of the Poverty Reduction staff of the Poverty Reduction and Economic Management (PREM) unit of the World Bank’s Kenya office. Corresponding email: gdemombynes@worldbank.org
# **1. Introduction**

In the past decade, mobile phones have become a ubiquitous feature of life in developing countries. Between 2005 and 2011 mobile cellular subscriptions nearly tripled in the developing world, increasing from 1.2 to 4.5 billion. In Africa, the region with the fastest mobile subscription growth rate, mobile cellular subscriptions increased from 87 million in 2005 to 433 million in 2011. A number of factors contributed to the luxury-to-commodity transformation of mobile phones. Technological advances along with economies of scale in network equipment and handsets in the developed world led to price declines. Additionally, the introduction of pre-paid billing systems lowered service costs by eliminating the need for operators to send bills and collect debts, thus making mobile phones accessible to lower income consumers. Finally, deregulation created a competitive marketplace, further reducing prices and bolstering mobile phone adoption (Economist 2009). Accompanying the growth in handset ownership has been growth in mobile applications, including digital money transfers and payments, citizen polling station surveillance, remote health care consultation and diagnosis, and transmission of timely market information to farmers.

One area in which mobile technology holds promise is for implementing surveys. Despite recent advances, in many developing countries there is a lack of reliable and timely socioeconomic data. Today, there is an opportunity to leverage mobile phones for routine data collection to help fill this gap and support policy decisions, strengthen civil society, and facilitate program management (Croke et al. 2012). This paper examines the opportunities and challenges of using mobile phones as a research tool, based on the experience with the South Sudan Experimental Phone Survey (SSEPS).

The paper is organized as follows. Section 2 of the paper describes options for data collection via mobile phones, Section 3 describes the South Sudan survey, Section 4 assesses patterns of non-response and survey results, and Section 5 concludes.

# **2. Mobiles for Data Collection**

For mobile phones to be useful as a survey device, they need to satisfy a number of criteria. A mobilebased survey should 1) yield a sample size large enough to draw statistically reliable inferences about a population of interest, 2) be economically viable compared to alternative data collection, and 3) generate data of adequate quality. The third criterion relates to sampling, attrition, and non-response as well as respondent behavior when interacting with a mobile phone interface.

2
<!-- Start of picture text --> Face-to-face interview, traditional paper-and-pencil<br>Respondent Enumerator Data entry<br>Paper Database<br>Face-to-face interview using phone, laptop or tablet computer with<br>electronic form<br>Respondent Enumerator ZA GPRS<br>Server/ Database<br>Direct responses to phone using SMS, USSD, IVR or electronic form<br>Server/ Database<br>Interview via phone with enumerator<br>Server/ Database<br><!-- End of picture text -->
**Table 1: Strengths and Limitations of Mobile Data Collection Client Options**

|Data Collection<br>Client / Interface|Data<br>Transfer<br>Method|Strengths|Limitations|
|---|---|---|---|
|Call Center|Voice|**-**Simple voice call (users will be<br>familiar with the interaction)<br>- Operators can troubleshoot during<br>the call, e.g. clarify questions, correct<br>misinterpretations)<br>- Opportunity to build relationship<br>with respondents over time<br>- Good for setting where literacy is<br>low|**-**Requires additional management<br>of call-center operations<br>- Requires a reliable network signal|
|Interactive Voice<br>Response (IVR)|Voice/<br>Keypad|- Simulates a voice call<br>- Good for settings where literacy is<br>low|- Cumbersome nature of keypad<br>based menu navigation<br>- Requires a reliable network signal|
|SMS|SMS|- Works on any GSM mobile phone<br>(Does not require installation)|- Requires a network signal<br>- Requires that users have a cue-card<br>to guide text input (changes to<br>questions on an ongoing basis may<br>be difficult)<br>- Possibility of lost or dropped<br>messages<br>- Limited to 160 characters of data<br>- Relatively high cost per byte of<br>data sent compared to GPRS<br>- Data collection efforts with high<br>expected data volumes will require a<br>commercial SMS gateway provider<br>(entailingadditional costs)|
|USSD|USSD|- Works on any GSM mobile phone<br>(Does not require installation)<br>- Widely used in East Africa for mobile<br>transactions<br>- Real time communication between<br>user and application|- Requires partnership with mobile<br>service operators<br>- Requires a consistent network<br>signal<br>- Relatively high cost compared to<br>SMS and GPRS|
|Electronic Forms<br>(Java or other<br>application)|GPRS|- Automated data upload from phone<br>to server<br>- Real-time visualization and<br>monitoring of results<br>- Branching, skip logic and enforced<br>validation at time of data entry<br>- Asynchronous: As data can be saved<br>to the phone, can be used in areas<br>without mobile coverage (data is sent<br>to server once in range of a signal)|- Requires smartphone and<br>additional installation of software|

4
Mobile phones have been used to collect data using three approaches: (1) _Face-to-face interview using phone, laptop, or tablet computer with electronic form._ The mobile phone essentially takes the place of paper in a traditional household survey, with enumerators conducting a “live” interview but capturing data provided by the respondent using an electronic device.<sup>2</sup> (2) _Direct response to phone using SMS, USSD, IVR, or electronic form._ Respondents interact with their own devices to answer questions digitally using the mobile’s keypad. The interface and data transfer technology employed could be SMS, USSD, or electronic forms. An alternative approach is to link the respondent to an automated interactive voice response system (IVR) in which the respondent dials a number and is prompted by a computer to record answers using the keypad.

(3) _Interview with enumerator via phone._ Respondents receive a call from an operator at a call-center, are asked questions and dictate their answers which are then recorded by the operator on a computer.

Few systematic studies are available that compare the data quality and data collection efficiency of these alternative approaches. In one India-based study evaluating the accuracy of data collection using electronic forms, voice, and SMS through mobile phones, MIT researchers found that a voice interface (in a setting where respondents dictated answers through the mobile phone to an operator in real time) provided the highest level of accuracy, leading them to favor the voice-operator combination for the implementation of a tuberculosis program (Patnaik et al. 2008). In Tanzania, a research team from Twaweza, a citizen-centered initiative in East Africa, favored a voice-operator interface to collect weekly data on provision of public services, due to implementation challenges associated with USSD, electronic forms, and IVR (MobileDataGathering 2011). For reasons explained below, SSEPS also employed a voice-operator approach.

> 2 The face-to-face, device assisted approach has also been used with devices other than mobile phones, such as tablet and notebook computers and is more broadly classified as Computer-Assisted Personal Interviewing (CAPI). See IRIS Center (2011) for a detailed review of CAPI software options and Caeyers et al. (2012) for an analysis of the performance of CAPI vs. traditional Pen-and-Paper Interviewing (PAPI).

5
# **3. The South Sudan Experimental Phone Survey**

In 2010 the World Bank financed a mobile phone-based survey to monitor conditions in South Sudan on a monthly basis and to contribute to knowledge regarding the feasibility and implementation of mobile phone surveys in low-income settings. The rationale for selecting a mobile data collection approach was rooted in context. In early 2010, South Sudan was a year away from the referendum for independence. Given a dynamic situation, setting up a platform to collect high frequency panel data on perceptions, living conditions, and access to services would help shed light on the evolving situation. Additionally, given the poor state of roads and the high costs of travel, mobile phones would offer the potential of a cost-effective solution to collecting high-frequency data by avoiding costly repeat visits to households.

While ownership of modern consumer durables (such as televisions, refrigerators and computers) is uncommon in South Sudan, reflecting low incomes and the lack of reliable access to electricity in homes, ownership of mobile phones has been steadily increasing in both urban and rural areas (Table 2). In 2006 the percentage of households that reported owning mobile phones in South Sudan was 2 percent (5 percent in urban areas and less than 1 percent in rural areas). By 2010 reported household ownership was up to 18 percent, with almost two in three households in urban areas owning at least one phone. (In urban areas, those households that have at least one phone own on average two phones.) The growth in household ownership of mobile phones in the span of four years is quite significant, especially in urban areas, suggesting a growing familiarity with the technology and the likelihood that a good proportion of respondents in urban areas would have used a mobile phone previously.

6
**Table 2: Household Ownership of Consumer Durables in South Sudan** _Percentage of Households Owning Item_

||20|06 SHH|S|200|8/2009 N|BHS|20|10 SHH|S|
|---|---|---|---|---|---|---|---|---|---|
|Consumer Durable|Overall|Rural|Urban|Overall|Rural|Urban|Overall|Rural|Urban|
|Radio|21.9|16.8|32.6|24.6|19.6|54.4|30|27.3|60.2|
|Bicycle|26.2|19.9|39.2|25.0|23.5|34.0|26.3|24.5|46.9|
|Mobile Phone|2|0.6|5|15.1|7.8|58.5|18.3|14.7|58.5|
|Motorcycle/ Scooter|1.9|1.1|3.6|3.3|2.0|11.1|4.3|3.5|13.6|
|Animal Drawn Cart|1|0.8|1.6|2.9|1.8|3.1|2.4|2.4|3.2|
|Television|1.1|0.4|2.4|3.3|0.6|19.4|2.2|0.8|17.3|
|Car or Truck|0.6|0.3|1.3|1.7|1.0|6.3|0.9|0.5|5.5|
|Refrigerator|0.9|0.6|1.7|0.9|0.2|5.0|0.6|0.2|5|
|Computer|0.5|0.2|1.3|0.7|0.3|3.3|0.5|0.3|3.4|
|Boat with motor|0.9|0.6|1.5|1.4|1.5|0.8|0.4|0.4|0.7|

Source: Authorss calculations using 2006 Sudan Household Health Survey, 2008 National Baseline Household Survey, and 2010 Sudan Household Health Survey.

Notes: The table is sorted by highest to lowest overall ownership in 2010.

# _Design, Implementation and Timeline_

Implementation of SSEPS involved two major phases: (1) Distributing mobile phones to individuals in households in South Sudan’s ten state capitals and training them on their use and (2) using a Nairobibased call center (Horizon Contact Center) to call respondents on a monthly basis and conduct a 15-20 minute survey. To provide an incentive for participation, individuals who successfully completed a survey were given pre-paid airtime credit for the mobile phone. Planning for the survey began in January 2010, a pilot survey was conducted in June and July of 2010, and the main implementation of the project was conducted between November 2010 and March of 2011 (Figure 2).<sup>3</sup> Specific design choices were influenced by context. South Sudan’s population is widely dispersed over rural areas without good connections to urban centers, less than 25 percent of the population has any formal education, half of households are poor, and very little infrastructure exists outside of state capitals. Given poor network coverage in rural areas, the sample was limited to households in urban settings. To avoid systematically omitting households without a phone, a phone was given to individuals in sampled households, regardless of whether they previously owned a phone or not. It was also hoped that distributing phones would help vest in participants a sense of ownership in the project.

> 3 After March, the project was handed off to Cordaid (an international NGO) which continued making calls on a monthly basis to participants through early 2012.

7
**Figure 2: SSEPS Timeline**

||_Year_|J F MA MJ J A S O N D J F MA MJ J A S O N D<br>2010<br>2011|
|---|---|---|
|**Project Component**|**Duration**||
|Project Planning/ Equipment Procurement<br>Pilot<br>Enumerator Training<br>Phone Delivery & Baseline Survey|5 months<br>1 month||
|Data collection(monthly)|4 months||

Households targeted for the SSEPS consisted of those that had been surveyed in round 4 of the Household Budget Survey (HBS), which were a representative sample drawn from the ten state capitals. These households were drawn from 10 enumeration areas in each of the state capitals, with 12 households per enumeration area for a total of approximately 1200 households. Of these households, 1007 were located and agreed to participate in the SSEPS. Given that the bulk of South Sudan’s urban population lives in the state capitals, results from the SSEPS are broadly representative of the conditions in urban areas. Selection of the individual respondent within each household to receive the mobile phone and answer calls was performed during an initial household visit made by enumerators from NBS.

To assess the effect of incentives on participation, sampled enumeration areas were randomly assigned to two incentive groups: one receiving pre-paid credit of 5 Sudanese Pounds (SDG)—approximately US$2.17—for each call leading to a completed survey and another receiving pre-paid credit of 10 SDG (US$4.35) for each call leading to a completed survey.<sup>4</sup> This credit was transferred on a weekly basis: the call center would send a list of participants that had completed the survey in the previous week to partners at the mobile phone operator (Zain), which would then transfer the credit to households. In the month of December, some of the transferred credits were delayed by two weeks as the purchased credit had not been activated. In addition, enumeration areas were randomly assigned to receive either a basic Nokia phone or a Safaricom solar-rechargeable phone. Given these combinations, four groups defined by phone type and incentive amount were formed: (1) Nokia, 5 SDG incentive group, (2) Nokia, 10 SDG incentive group, (3) Safaricom, 5 SDG incentive group and (4) Safaricom, 10 SDG group. A last incentive which was offered to all respondents came in the form of a lottery. All sampled households

> 4 US Dollar equivalent figures are based on annual exchange rate average for of 2.3 SDG per USD for 2010.

8
were told that they would be eligible for the chance to win a 230 SDG credit transfer in a monthly survey should they complete the survey.

Prior to visiting households, the mobile phones were fully charged, enabled with SIM cards, tested to ensure they could make and receive calls and pre-loaded with 2 SDG of credit.<sup>5</sup> A team of 20 enumerators from the National Bureau of Statistics were trained in the World Bank office in Juba on the procedure for the household visits. This involved an orientation about the purpose of the survey and the protocol for household visits.

Visits to sampled households were carried out in 10 teams of 3 beginning the first week in November 2010. Each team covered one state capital. Most visits occurred in the morning or late afternoon. Initial plans called for randomly selecting one respondent from the household roster and alternating between male and female respondents to ensure gender balance. The pilot revealed this would be logistically challenging since randomly selected individuals not present at the time of the visit would require coordinating additional visits to the household until the respondent was found. In addition, as the survey focused on gathering information on general household and community-level characteristics rather than very specific or personal characteristics, the risk of systematically biased responses was not deemed large. As such, when enumerators visited each household, rather than taking the list of household members and randomly selecting one member from the list, the first adult member of the household encountered at the time of the visit was selected for participation. Each selected participant was trained in the use of the phone and given a calendar displaying the specific day he or she would receive a call, with the first call occurring four weeks after the initial household visit. Enumerators collected an alternative phone contact number if available, the language preference of the participant (Arabic, English, Dinka, or Nuer) and the participant’s preferred time of day for calls.

Each month’s SSEPS questionnaire consisted of two modules: a set of core questions that were repeated each month and a set of “special” questions that were changed on a month-to-month basis. The core questions fall into three categories: past changes and outlook, material deprivation, and security. The wording for the core questions was taken from Afrobarometer surveys conducted in 2008 and 2009, in

> 5 Facilities for mobile charging through community charging stations, usually powered by generators or car batteries, are common in urban areas.

9
order to make it possible to compare results from Southern Sudan to other countries.<sup>6</sup> The core set of questions, as well as the special set of questions for each month are provided in Annex B.

# **4. Results**

## **<mark>Table 3: SSEPS Sample Characteristics</mark>**

## _Sample Characteristics_

|**Characteristic**|**N**<br>**1**|**Mean**<br>**2**|
|---|---|---|
|Gender: Female(%)|1004|62.5|
|Gender: Male(%)|1004|37.5|
|Age(years)|975|33.6|
|Age: Under-20(%)|975|7.2|
|Age: 20-29(%)|975|34.8|
|Age: 30-39(%)|975|30.9|
|Age: 40-49(%)|975|15.6|
|Age: 50plus(%)|975|8.9|
|Language: Arabic(%)|1007|44.4|
|Language: Dinka(%)|1007|37.4|
|Language: Nuer(%)|1007|8.8|
|Language: English(%)|1007|9.3|
|Top-upvalue: 5 SDG(%)|1007|51.9|
|Top-upvalue: 10 SDG(%)|1007|48.1|
|Phone Type: Nokia(%)|1007|52.5|
|Phone Type: Solar(%)|1007|47.5|
|Alternativephone available(%)|1007|36.6|
|Preferred call time: morning (%)|1007|57.9|
|Preferred call time: afternoon(%)|1007|17.3|
|Preferred call time: evening (%)|1007|24.8|
|Town of residence: AWEIL(%)|1007|11.3|
|Town of residence: BENTIU(%)|1007|7.5|
|Town of residence: BOR(%)|1007|11.6|
|Town of residence: JUBA(%)|1007|9.7|
|Town of residence: KWAJOK(%)|1007|8.1|
|Town of residence: MALAKAL(%)|1007|9.8|
|Town of residence: RUMBEK(%)|1007|10.0|
|Town of residence: TORIT(%)|1007|10.6|
|Town of residence: WAU(%)|1007|10.2|
|s resonse rates were hih: 84 ercen<br>Town of residence: YAMBIO(%)<br>_1N = Observations with non-missing values_<br>_2N = Unweighted_|in Dec<br>1007|mber<br>10.9|

Table 3 displays key characteristics of the sample of SSEPS participants. Sampled participants were disproportionately female, averaged 34 years of age at the time of the household visit (two-thirds of participants were between the age of 20 and 40), and primarily requested interviews in Arabic or Dinka. Thirty-six percent of respondents had access to another mobile phone.

## _Response Rates and Attrition_

At the end of the first round of data collection in December 2010, 68 percent of SSEPS participants completed the mobile-phone administered questionnaire. Overall response rates declined to 62 percent in January, 55 percent in February, and 52 percent in March (Table 4 and Figure 3). Both the level and trend of response rates varied substantially across town of residence (Figure 4). In Wau, for example, although response rates declined with each successive round, relative to other towns, response rates were high: 84 percent in December

> 6 The reference period for the Afrobarometer questions is one year. Because the survey was conducted on a monthly basis, a one-month reference period was of greater interest for the time series data. For this reason, in December 2010 the poverty and security questions were asked with _both_ a one-year and a one-month reference period. In subsequent months, they were asked with only a one-month reference period. The outlook questions are only asked with a one-year reference period.

10
and 64 percent in March. In Kwajok on the other hand, response rates were both declining and consistently low. After a drop in February, response rates in Malakal increased from their initial level of 73 percent in December to 77 percent in March.

# **Table 4: Attrition and Non-response**

|**Characteristic**|**N**<br>**1**|**Mean**<br>**2**|
|---|---|---|
|Overall response rate: Round 1(December) (%)|1007|68.8|
|Overall response rate: Round 2(January) (%)|1007|62.4|
|Overall response rate: Round 3(February) (%)|1007|55.2|
|Overall response rate: Round 4(March) (%)|1007|51.5|
|Total surveys completed(out of 4): None(%)|1007|16.9|
|Total surveys completed(out of 4): One(%)|1007|12.8|
|Total surveys completed(out of 4): Two(%)|1007|16.3|
|Total surveys completed(out of 4): Three(%)|1007|23.5|
|Total surveys completed(out of 4): Four(%)|1007|30.5|
|Attrition: Dropped out after round 1(%)|1007|7.2|
|Attrition: Dropped out after round 2(%)|1007|6.2|
|Attrition: Dropped out after round 3(%)<br>_1N = Observations with non-missing values_<br>_2N = Unweighted_|1007|10.4|

Participants can be categorized by response pattern: (1) “full-compliers,” who completed all four rounds of the survey, (2) “intermittent compliers,” who completed between one and three surveys intermittently over the four rounds, (3) “drop-outs,” who dropped out after one or more successfully completed surveys and (4) “non-compliers,” who did not complete a single survey. Approximately 17 percent of the sample did not complete a single survey, while 31 percent completed all four surveys between December and March. Approximately half of all households (52 percent) completed between one and three surveys over the data collection period. Of these households, close to half (24 percent of the sample) dropped out after at least one successfully completed interview, 7 percent after round 1, 6 percent after round 2, and 10 percent after round 3.

11
**Figure 3: Response Rates by Month, Type of Phone, Incentive and Participants by Response Pattern**

<!-- Start of picture text --> 100 Full Intermittent Non<br>Drop-outs<br>compliers compliers compliers<br>90<br>Full Sample<br>80 30.5 28.8 23.8 16.9<br>70 Nokia + 10 SDG<br>27.2 26.5 24 22.3<br>60<br>Nokia + 5 SDG<br>50<br>37 24.8 24 14.2<br>40<br>Solar + 10 SDG<br>30<br>31.3 28.8 19.5 20.4<br>Nokia + 5 SDG<br>20<br>Nokia + 10 SDG Solar + 5 SDG<br>Solar + 5 SDG<br>10 27.4 34.7 26.7 11.2<br>Solar + 10 SDG<br>0 [Month] Full Sample [Percent]<br>Dec Jan Feb Mar 0 20 40 60 80 100<br>[Response Rate (%)]<br><!-- End of picture text --> Among respondents receiving Nokia phones, those offered 10 SDG credit were about 10 percent less likely in each round to complete the survey compared to respondents offered the 5 SDG credit (Figure 3). There was not a similar pattern among respondents using solar-rechargeable phones. Both Nokia and Safaricom solar phone users completed surveys with the same frequency: among 4,028 possible completed surveys across all four survey rounds the overall response rate for users of both phones was 59 percent. This suggests that the constraint of having to charge the Nokia phone using an electrical outlet did not influence survey completion.

Three probit regression models were used to explore in more detail factors associated with successful completion of the survey. Model 1 in Table 5 assesses the likelihood that a respondent completed the questionnaire given personal characteristics and SSEPS design features. Models 2 and 3 in Table 5 assess the association of these same factors with the likelihood a participant completed all four surveys and with the likelihood a participant completed none.

The decline in response over the course of the four month implementation period is evident in the regression results. Controlling for other variables, respondents were 7, 16, and 20 percent less likely to have completed a questionnaire in January, February, and March relative to December. Operational data tracked by the Horizon Contact Center suggests that this result was in part due to declining network availability over the four month period.

12
Survey completion was most likely among participants over the age of 40, and female participants were close to 10 percent more likely to complete the questionnaire than male participants. Participant

**Figure 4: Response Rates by Town**

Malakal 72.7 75.8 56.6 77.8 Yambio 74.5 69.1 63.6 71.8 Juba 67.3 <u>67.3</u> 59.2 62.2 Bentiu 52.6 63.2 55.3 43.4 Rumbek 58.4 46.5 36.6 44.6 83.5 Wau 71.8 70.9 64.1 Aweil 80.7 72.8 68.4 57.9 Kwajok 58.5 45.1 37.8 28.0 Torit 66.4 51.4 44.9 35.5 Bor 65.8 57.3 53.8 26.5 language was not associated with response rate, suggesting that efforts to ensure interviewers at the call center could address participants in their preferred language were successful. As noted previously, the value of the incentive offered was negatively associated with survey completion: participants who were offered the 10 SDG top-up were 6.4 percent less likely to complete the survey, and the type of phone received did not affect response rates.<sup>7</sup> Participants who provided an alternative mobile phone contact number were 24 percent more likely to complete a survey. Monthly reports from the call center contracted to conduct the interviews suggest that this is partially explained by the fact that the respondents who previously owned a phone had given their phones to relatives, thus giving agents at the call center more opportunities to locate the respondent. Additionally, when the Zain network used on the phones distributed by the project was not functioning, the call center could use an alternative service provider (mainly MTN and Vivacel) on the alternative phone. Participants with access to another phone were also more likely to be familiar with the technology, and participants with their own phones probably choose service providers with the best local coverage.

> 7 Respondents who completed surveys in the first two weeks of December experienced a delay in the transfer of air-time credit. Analysis not presented here shows that controlling for regression covariates, these households were 9.4 percent less likely to complete surveys in rounds 2, 3 or 4 compared to households that did not experience the delay.

13
**Table 5: Marginal Effects from Probit Regression Assessing Relationship between Survey Participation** **<u>and Respondent Characteristics</u>**

||**Model 1**|**Model 2**|**Model 3**|
|---|---|---|---|
|**Explanatory Variables**|Dependent<br>variable:<br>Participant<br>completed<br>|Dependent<br>variable:<br>Participant<br>completed all 4|Dependent<br>variable:<br>participant<br>completed 0|
||surveyin round|surveys|surveys|
|Survey Round: December|[reference]|||
|Survey Round: January|-0.0741***|||
|Survey Round: February|-0.156***|||
|SurveyRound: March|-0.198***|||
|Age: Below 20||[reference]||
|Age: 20-29|0.0872**|0.0578|-0.0275|
|Age: 30-39|0.118***|0.127**|-0.0474|
|Age: 40-49|0.143***|0.108|-0.0807***|
|Age: 50plus|0.141***|0.139*|-0.0492|
|Gender: Male||[reference]||
|Gender: Female|0.0973***|0.0336|-0.0745***|
|Language: Arabic||||
|Language: Dinka|-0.0168|-0.00764|-0.00198|
|Language: Nuer|0.0834|0.0525|0.0170|
|Language: English|-0.0115|-0.0258|-0.0209|
|Top-up value: 5 SDG||[reference]||
|Top-upvalue: 10 SDG|-0.0637***|-0.0274|0.0700***|
|Phone Type: Nokia||[reference]||
|Phone Type: Solar|-0.0252|-0.0433|0.00230|
|Alternative phone not available||[reference]||
|Alternativephone available|0.237***|0.188***|-0.145***|
|Tow of residence: JUBA||[reference]||
|Town of residence: AWEIL|0.0805|-0.0511|-0.0923***|
|Town of residence: BENTIU|-0.175**|-0.248***|-0.0829*|
|Town of residence: BOR|-0.0799|-0.212***|-0.00464|
|Town of residence: KWAJOK|-0.146*|-0.191***|0.0346|
|Town of residence: MALAKAL|0.0386|-0.0453|-0.1000***|
|Town of residence: RUMBEK|-0.0594|-0.0434|0.00538|
|Town of residence: TORIT|-0.116**|-0.236***|-0.0332|
|Town of residence: WAU|0.101*|0.0577|-0.0732**|
|Town of residence: YAMBIO|0.0987*|0.0471|-0.0849***|
|Observations|3,900|975|975|
|Number of households|975|975|975|

*** p<0.01, ** p<0.05, * p<0.1

Participants who completed all four survey rounds were more likely to be above the age of 30 and more likely to have access to an alternative phone. Gender, the type of phone, and the incentive value were not associated with the probability a respondent would complete all four rounds of the survey. The very strong geographical pattern associated with completing all four survey rounds suggests that the differential reliability of network coverage across the ten urban centers was a strong determinant of 14
response rates. Participants who did not complete a single survey were less likely to be female, less likely to have access to an alternative phone, and more likely to have been offered a pre-paid credit of 10 SDG.

As additional socioeconomic characteristics of households were available from the NBHS for about half of the SSEPS households, an additional set of regressions was run to explore the association between survey participation and the level of educational attainment of the household head, poverty status, and television ownership. Results from this set of regressions (Annex A) show that respondents belonging to households in which the head member completed post-secondary education were more likely to complete at least one survey. The poverty status of the household and ownership of a television were not associated with the rate of survey completion.

# _Call Center Outcomes_

Horizon Contact Center kept records of the attempts to reach respondents (Figure 5). In December, among 3617 dials, 36 percent resulted in a contact with a respondent and 19 percent resulted in a completed survey. The efficiency of the calling process improved significantly in January—for every 100 dialed numbers, 34 resulted in a completed survey—but subsequently fell off in February and March where out of every 100 calls, only 17 and 15, respectively, resulted in a completed survey.

Major challenges related to reaching the correct respondent involved an unreachable number (37 percent of calls) and no network coverage (18 percent).<sup>8</sup> Problems with the network increased over the course of the four month period, peaking at 32 percent of all dialed numbers in March. Calls to Bentiu and Torit in December and January and Aweil and Bor in March were cited as being particularly and consistently problematic due to poor network connectivity. The fraction of unreachable numbers was particularly high in February, when 47 percent of all dialed numbers were unreachable.

> 8 Unreachable numbers exist when a number is out of service or a connection cannot be made to the number being called. No network refers to the situation where all numbers fail to go through because of a hitch on the network either the originating network or the destination network.

15
# **Figure 5: Calling Efficiency Indicators**

|Outcomes|3617<br>total calls<br>3.6<br>avg calls<br>per respondent<br>December|1861<br>total calls<br>1.8<br>avg calls<br>per respondent<br>January|3320<br>total calls<br>3.3<br>avg calls<br>per respondent<br>February|3394<br>total calls<br>3.4<br>avg calls<br>per respondent<br>March|
|---|---|---|---|---|
|Contacts per 100 calls|35.7||48.5<br>24.7|22.3|
|Completed surveys per 100 calls|19|34.1|17.1|15.3|
|Distribution of calls by outcome:|||||
|Unreachable number|39.2|26.2||47.6<br>33.4|
|Correct Respondent|21.1|34.1|17.6|15.9|
|Call Back|10.6|11.6|5.3|4.6|
|No Network|10|11.4|18.7|31.8|
|No Answer|8|8.5|7.1|8|
|Disconnected Number|6|2.7|1.6|3.7|
|Incorrect Respondent|1.7|1.1|.5|.3|
|Contact Hangup|1.2|.6|.5|.7|
|Number is busy|1.1|2.7|.2|.8|
|Language Barrier|.7|.4|.4|.2|
|Wrong number|.4|.7|.5|.5|
|Deceased|0|.1|0|.1|

Other challenges during survey implementation related to content. For example, in February the rotating set of questions focused on governance and leadership, and call agents asked respondents to share perspectives on leadership quality. Many were reluctant to do so, citing concern that the information would be used against them. In January, when respondents were asked to list all member of the household in order to collect information on educational attainment, many were cautious and wanted reassurance the answers would be kept confidential.

Call agents at Horizon also reported many respondents complained about faulty batteries (among solar phone users) and the costs of recharging (among Nokia phone users). There was also some evidence that respondents lost track of their call date and that some research phones were sold. In some cases when interviewers called the lines and made a contact, the person on the other end was not the listed respondent and mentioned having bought the phone on the market. Unfortunately, Horizon did not keep exact track of the extent of these issues so it is difficult to gauge how influential they were in determining overall response rates.

# _Survey Results_

16
**Figure 6: Summary of Responses to Core Questions, by Month**

<!-- Start of picture text --> 100 In last month, gone without Food: 100 In last month,<br>gone without Clean Water:<br>80 80<br>60 or Several timesOnce, Twice 60<br>40 40<br>Never<br>20 20<br>Many Times<br>or Always<br>0 0<br>Dec Jan Feb Mar Dec Jan Feb Mar<br>100 In last month, gone without Fuel: 100 Ability to access<br>medicine in last month:<br>80 80<br>Yes<br>60 60<br>40 40<br>No<br>20 20<br>0 0<br>Dec Jan Feb Mar Dec Jan Feb Mar<br>100 In last month, 100 In last month,<br>gone without Medicines: something was stolen:<br>80 80<br>60 60<br>40 40<br>20 20<br>0 0<br>Dec Jan Feb Mar Dec Jan Feb Mar<br><!-- End of picture text --> Figure 6 displays responses to the core questions by month. (The corresponding table is provided in Annex A). Overall, the response trends can be divided into two groups characterized by their level of volatility over the four survey rounds. Four questions yielded consistent response patterns: access to food, ability to access medicines or medical treatment, frequency of illness, and personal security. Several salient trends emerge. First, the percentage of households reporting food shortages “many times or always” declined from 15 percent to less than 2 percent between December and March. This downward trend most likely reflects the onset of the harvest season which spans midSeptember to mid-January. Second, the percentage of households reporting inability to access medicines doubled from 18 percent to 36 percent. Third, the frequency of reported illness in the household did not change significantly and, fourth, reported personal security improved somewhat as the percentage of households not having had something stolen from their homes in the last month increased from 67 to 82 percent.

Given that response rates gradually fell from December through March, some of the trends noted above could have plausibly resulted from a dynamic selection effect rather than changes in the economic, social, or political environment. However, the response patterns (both levels and trends) are very similar when comparing all responses, as presented above, with only responses from full compliers, i.e. the 17
subset of respondents that completed the survey in all four rounds. A plot of responses from full compliers is shown in Table A4 in Annex A. This suggests that election due to non-response does not appear to have markedly affected survey results.

The two questions with most volatility on a month-to-month basis were access to clean water and access to cooking fuel. Among these two questions, two trends are observed: (1) In February, there appears to have been a one-time shock that made it easier for households to get access to fuel, and (2) beginning in February, a shock lasting through March made it more difficult for households to access clean water. Given that access to fuel depends on market conditions that can fluctuate rapidly (and therefore change the availability and price of fuel commodities), and access to clean water depends on environmental conditions (such as the onset of the dry season) and public or private services that can be disrupted, the higher volatility of responses to these two questions on a month-to-month basis relative to others appears plausible.

# **5. Discussion**

The fact that response rates dropped by about 5 percentage points per round, starting at 68 percent in December and ending at 52 percent in March, is cause for concern that selective response may bias the survey results. This worry is compounded by the finding that those responding were disproportionately women, older participants, and those with alternative phone numbers. At the same time, the fact that trends for responses from full-compliers alone are similar to trends based on all responses suggests that selective attrition did not ultimately bias the survey results.<sup>9</sup> The largest contribution to non-response resulted from individuals who participated intermittently over the course of the survey period. Future studies using mobile data collection might mitigate non-response by sending simple reminders (possibly using SMS) to reduce the number of intermittent compliers. Targeting older, female household members could reduce the number of non-compliers, but such targeting would not be desirable for surveys seeking a representative sample of individuals, e.g. labor force surveys.

> 9 It is not possible to determine, however, whether responses from the 17 percent who _never_ responded would have differed from those of those who did response.

18
Notably, poverty status was not associated with survey completion. An important contributor to nonresponse over the course of the survey period seems to have been differential network coverage between geographic areas and disruption to network coverage within geographic areas over time. Having multiple options to call respondents by recording an alternative contact number helped significantly. This suggests that survey designs that use existing household mobile phones might yield higher response rates than survey designs that hand out mobile phones. The downside of this strategy is representativeness in areas with low household ownership of mobile phones.

Larger incentives in the form of pre-paid calling credit did not work to encourage survey completion. In fact, participation rates were slightly lower for those who received the greater incentive. The underlying mechanism of this result is not clear. Given that the design of the study did not incorporate a group without an incentive payment, it is not possible to say whether offering some incentive increased participation versus offering no incentive at all.

In terms of implementation, a critical feature of this study was partnering with the National Bureau of Statistics. The NBS provided the staff, knowledge and experience that enabled the sample selection and successful distribution of phones. Being able to tap the NBS’s experience in collecting data in South Sudan—given obvious logistical and operational challenges in the country—was a tremendous asset to the SSEPS project.

Overall, the SSEPS experience is encouraging as to the potential for mobile phone-based surveys while at the same time offering a cautionary tale about the potential hurdles to the approach. In particular, variability in the functioning of the mobile network, which was not anticipated by the project team beforehand, turned out to be a substantial problem. At the same time, selection due to non-response turned out to be less of a concern than expected, as it does not appear to have markedly affected survey results in this case. The overall success of the project in an extremely difficult environment suggests that further efforts at mobile data collection in developing countries are worth exploring.

19