metadata: authors: 0: author: Smets,Lode
count: World
docna: 0: docna: Impact of phone reminders on survey response
            rates : evidence from a web-based survey in an international organization
docty: Policy Research Working Paper
subtopic: Labor & Employment Law,Health Care Services Industry,Health Service Management and Delivery
teratopic: Private Sector Development,Governance,Science and Technology Development,Public Sector Development,Industry
abstracts: cdata!: This research note investigates the
            impact of phone reminders on response rates in the context
            of a web-based survey in an international organization, the
            World Bank. After randomly assigning treatment to 248 survey
            participants, the study finds an intention-to-treat effect
            of 19.86 percentage points. Given a relatively low treatment
            compliance rate (31 percent), the estimated average effect
            of treatment-on-the-treated is even larger, corresponding to
            an increase of 64 percentage points. Therefore, if ways can
            be found to increase treatment compliance, high response
            rates are attainable. This may lead World Bank surveyors to
            turn to sample surveys more often, reducing survey overload
            in the institution.
repnme: repnme: Impact of phone reminders on survey
            response rates : evidence from a web-based survey in an
            international organization
display_title: Impact of phone reminders on survey
            response rates : evidence from a web-based survey in an
            international organization
listing_relative_url: /research/2018/01/29514168/impact-phone-reminders-survey-response-rates-evidence-web-based-survey-international-organization
topicv3: Health Services Delivery m1327294 1305,Household Surveys m1504071 2761
publishtoextweb_dt: 2018-01-17T00:00:00Z
historic_topic: Private Sector Development,Governance,Science and Technology Development,Public Sector Development,Industry
no_of_pages: 12
model_extractions: WPS8305 Policy Research Working Paper 8305 Impact of Phone Reminders on Survey Response Rates Evidence from a Web-Based Survey in an International Organization _Lodewijk Smets_ Independent Evaluation Group January 2018
Policy Research Working Paper 8305

# **Abstract**

This research note investigates the impact of phone reminders on response rates in the context of a web-based survey in an international organization, the World Bank. After randomly assigning treatment to 248 survey participants, the study finds an intention-to-treat effect of 19.86 percentage points. Given a relatively low treatment compliance rate (31 percent), the estimated average effect of treatment-on-thetreated is even larger, corresponding to an increase of 64 percentage points. Therefore, if ways can be found to increase treatment compliance, high response rates are attainable. This may lead World Bank surveyors to turn to sample surveys more often, reducing survey overload in the institution.

This paper is a product of the Independent Evaluation Group . It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at lodewijk. smets@gmail.com _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Impact of Phone Reminders on Survey Response Rates:**

# **Evidence from a Web-Based Survey in an International Organization**

Lodewijk Smets<sup>1,2</sup> 1 The World Bank, Independent Evaluation Group 2 KU Leuven, Centre for Institutions and Economic Performance Keywords: Survey, international organizations, experiment, response rate JEL: C83, C81, C9.
# **1. Introduction**

As response rates are related to the risk of non-response bias,<sup>1</sup> several techniques have been developed to increase survey response rates, ranging from monetary incentives to personalized invitations.<sup>2</sup> Also telephone reminders are used as a way to boost response rates.

A number of studies have investigated the impact of phone reminders on survey response rates, at least dating back to Roscoe et al (1975). Somewhat surprisingly however, all studies are conducted in the context of postal surveys. This research note aims to complement this field of study by investigating the impact of phone reminders in the context of a web-survey, targeting professional staff in an international organization, i.e., the World Bank. The present paper is related to the work of Smets and Bogetić (2018) who provide a detailed discussion of the underlying survey and its design, methodology and implementation. To the best of our knowledge, this is the first study to examine survey response rates in such a setting.

We randomly assigned the reminder treatment to 248 survey participants. During the telephone call we briefly explained the purpose of the reminder, verified whether participants already filled out the survey and offered to resend the survey link immediately after the call in case they had not.

> 1 Response rates set bounds to the maximal non-response bias. However, it is important to note that also other factors such as measurement error, coverage error or editing error may bias survey estimates (e.g., see Davern, 2013; Meyer et al, 2015).

> 2 See Edwards et al (2009) or Fan and Yan (2010) for a recent systematic review of the methods to increase response rates in postal and web-based surveys.

2
This strategy led to an overall response rate in the treatment group of 54.03 percent. The response rate in the control group was 34.17 percent, resulting in an Intention-To-Treat (ITT) effect of 19.86 percentage points. Due to the fact that out of the 248 individuals assigned to treatment, only 77 staff answered the phone call, had not yet filled out the survey but were willing to receive another survey link, the local average treatment effect (LATE) is even larger. The LATE – which in this case measures the average effect of treatment-on-the-treated (TOT) – corresponds to a vast increase of 64 percentage points.

The remainder of the note is structured as follows. In the next section we highlight the specificities of conducting a web-based survey in an international organization, discuss the survey design and elaborate on the method used to evaluate the impact of phone reminders on survey response rates. In section 3 we discuss the results while section 4 concludes.

# **2. Background and Method**

As part of a larger evaluation, the World Bank’s Independent Evaluation Group (IEG) implemented a web-based survey among professional staff to elicit their views on one of the Bank’s corporate goals. The survey was sent out to a sample of 3,698 eligible staff members.<sup>3</sup> It was launched on September 7, 2016 and closed on October 18, 2016. Surveying professional staff in an international organization comes with a number of challenges.<sup>4</sup> First, the World Bank Group has more than 130 offices worldwide, which complicates the timing of the survey and the telephone reminders. Relatedly, given its

> 3 We originally selected a sample of 3,736 units. However, 32 individuals were excluded as they had left the institution by the time the survey was implemented. We also excluded 6 IEG staff members from the sample who were mistakenly included (IEG was not part of the target population), leading to a sample of 3,698 eligible units. 4 One important advantage of a web-based survey targeting staff of an international organization is the availability of a nearly complete sampling frame, thus minimizing coverage bias.

3
global scope, staff travel regularly. From July 2015 until June 2016, 56,090 World Bank flights were booked originating from Headquarters (Washington, D.C.), with top destinations all over the map, from Johannesburg to Kathmandu. Adding another 11,086 trips originating from country offices, the average number of World Bank flights per week amounts to 1,291. Third, web-based surveys are frequently used at the institution to collect data on a wide range of issues – from the quality of commuter services to workforce perspectives on managerial effectiveness. For instance, since 2009 IEG alone sent out 208 web-based surveys, corresponding to an average of 23 surveys per year.<sup>5</sup> As Porter et al (2004) note, overexposure to surveys may lead to survey fatigue and declining response rates.

Indeed, World Bank survey response rates are generally low and tend to be declining over time. For example, since 1999 IEG has surveyed its clients – WBG staff, the Board of Directors and external stakeholders – annually to gauge their perceptions of the quality and impact of IEG evaluations. In 2006, 22 percent of WBG staff responded to the client survey. In 2012 the response rate dropped to 11.3 percent while the 2015 client survey response rate plunged to a dramatic 4.7 percent.

Given these negative trends, it was found important to increase the response rate for this survey. Therefore, IEG’s Director General sent out the initial email invitation, including the expected duration of the survey, the survey deadline and a web link to the survey at the bottom of the invitation. Two reminder emails were sent out, one three weeks after launching the survey and another one the day before closing the survey.

> 5 This includes both internal surveys targeting IEG staff as well as external surveys targeting the World Bank Group and other stakeholders (e.g., client governments).

4
Between the two reminders – on October 12-13 – we called a random subset of the sample inviting them to take the survey in case they had not. During the telephone call we briefly explained the purpose of the reminder, verified whether participants already filled out the survey and offered to resend the survey link immediately after the call in case they had not.

Of the survey participants, 248 were randomly assigned to receive the treatment, while the remaining 3,450 participants served as the control group. As shows, there is no significant treatment-control imbalance on four observable characteristics – organizational mapping, pay grade, geographic location and managerial status – for which we have information. Furthermore, to verify joint orthogonality, we also regressed the treatment assignment on the four covariates. The F-test of this regression is 0.67 (p-value= 0.611), indicating that jointly the covariates do not explain treatment assignment.

Table 1: Balancing tests

|**covariate**|**mean treatment**|**mean control**|**p-value**|
|---|---|---|---|
|IBRD|0.725807|0.72058|0.859|
|Grade|1.895161|1.936232|0.423|
|Location|0.596774|0.557101|0.224|
|Manager|0.060484|0.074203|0.392|

<u>Note: The first column lists the four covariates on which we have information. IBRD is a dummy variable</u> that stands for the organizational mapping of the sample unit. It equals 1 if the sampled individual belongs to the International Bank for Reconstruction and Development (IBRD), 0 otherwise. Grade is a categorical variable indicating pay grade. Location is a dummy variable equal to 1 in case the sampled individual is located at Headquarters (Washington D.C.), 0 otherwise. And finally Manager is a dummy equal to 1 in case the sampled individual is a manager. The second and third column represent the mean values for the treatment and control group respectively. The forth column represents the p-value on treatment in an OLS regression of the covariate on assignment to treatment.

5
ITT = E[Y{|z; = 1] — E[Y;|z; = 0]

E[Ylz; <u>=</u> 1] — EL¥;|z; <u>=</u> 0]

P[D; = 1|z; = 1]
# **3. Results**

In the control group 1,179 individuals participated in the survey, corresponding to a response rate of 34.17 percent (=1,179/3,450). In the treatment group 134 individuals participated in the survey,<sup>10</sup> resulting in a response rate of 54.03 percent (=134/248). Thus, the ITT effect is 19.86 percentage points (p-value=.00001).<sup>11</sup> Given the relatively low treatment compliance rate (=77/248=0.31), the average effect of treatment-on-the-treated is more than three times as large: Therefore, if ways can be found to increase treatment compliance, (very) high response rates are within reach. In the context of this survey, we see three ways of increasing treatment compliance. First, due to time constraints, we were only able to make two telephone attempts. Increasing the number of calls is likely to improve treatment compliance (Munoz-Leiva et al, 2010). Second, knowing when staff are out-of-office can help in optimizing the timing of the telephone reminders. Finally, the survey was implemented with full anonymity, so that we did not know beforehand whether or not staff members completed the survey. Therefore, giving up on anonymity leads to better targeting of the phone reminders (even though it could potentially decrease survey participation).

> 10 Given full anonymity and imperfect compliance, we had to estimate the response rate of the 138 “non-compliers” in the treatment group. We used the overall response rate at the start of the experiment and the change in the number of responses from the start of the experiment until survey closure to do so.

> 11 P-value from employing a z-test for differences in proportions estimated from two independent samples.

7
# **4. Discussion**

Results from this experiment show that phone reminders can be a cost-effective way to increase survey response rates. That is, 55 out of the 110 people we were able to reach ended up filling out the survey. Given a labor cost of $200 per day – and two days of calling the treatment group – the cost per additional respondent is $7.27.<sup>12</sup> What is more, given the large TOT effect, (very) high response rates are within reach. This opens up the possibility of conducting sample surveys more regularly instead of population surveys. That is, because of low response rates for World Bank surveys, sampling does not guarantee representativeness. Furthermore, since the financial cost of sending out the survey to the whole population is low, such an approach is often taken. But this strategy does not internalize externalities such as survey fatigue and (high) opportunity costs of filling out questionnaires.<sup>13</sup> Knowing that high response rates are attainable, World Bank surveyors may turn to sample surveys more often, therefore reducing survey overload.

While these results have high internal validity, caution is required when generalizing the findings to other situations/surveys. Some important situational variables need to be taken into account. First, the survey topic was about one of the World Bank’s corporate goals. This may have raised additional interest from survey participants. Second, the timing of the survey was specifically chosen as to ensure staff availability. Third, the majority of World Bank staff are proud to work at the WBG and are motivated to contribute more than what is expected (World Bank, 2016). Such personal characteristics – which are

> 12 However, a more effective targeting strategy may further increase the cost-effectiveness of the phone reminders. 13 In an email conversation with an IEG Director, the World Bank’s Chief Economist raised the issue of high opportunity costs.

8
arguably related to the propensity to respond – may not necessarily translate to other institutions, hence limiting external validity. For these reasons we believe that findings from other international institutions as well as evidence from other surveys within the WBG may provide valuable information.

9