metadata: authors: 0: author: Soundararajan,Vidhya
1: author: Soubeiga,Sidiki
2: author: Newhouse,David Locke
3: author: Palacios-Lopez,Amparo
4: author: Pape,Utz Johann
5: author: Weber,Michael
count: World
docna: 0: docna: How Well Do Internet-Based Surveys Track
            Labor Market Indicators in Middle-Income Countries ?
docty: Policy Research Working Paper
abstracts: cdata!: Online surveys are convenient,
            cost-effective, speedy, and increasingly popular instruments
            for data collection. This study investigates whether online
            surveys that used Random Domain Intercept Technology to
            recruit respondents were accurately measured labor market
            outcomes in six middle-income countries in the aftermath of
            the COVID-19 pandemic. Compared with the national average,
            online surveys oversampled males, youth, those with higher
            levels of education, and those in smaller households.
            Reweighting using propensity score estimates fails to
            equalize the means of variables excluded from the model.
            When comparing the employment-to-population ratio from the
            internet surveys to the most recent relevant nationally
            representative surveys, the average deviation is 30 percent.
            Reweighting using propensity scores in that case worsened
            the bias. Internet survey estimates of informal and
            self-employment rates also tend to be inconsistent with
            benchmark data, although the latter are available for fewer
            countries. Overall, the results suggest that despite the
            advantages and convenience of recruiting internet survey
            participants through Random Domain Intercept Technology, the
            resulting sample is not representative and even after
            propensity score reweighting, it can yield estimates that
            are at odds with nationally representative surveys.
display_title: How Well Do Internet-Based Surveys
            Track Labor Market Indicators in Middle-Income Countries ?
listing_relative_url: /research/2023/03/34020263/well-internet-based-surveys-track-labor-market-indicators-middle-income-countries
topicv3: Employment Tools and Methods m1504070 2760,Coronavirus (COVID-19) m1939478 4828
publishtoextweb_dt: 2023-03-13T00:00:00Z
dois: 10.1596/1813-9450-10359
author_keys: 0: author_key: 1057141
1: author_key: 1189705
2: author_key: 450079
3: author_key: 1043558
4: author_key: 1154467
5: author_key: 1008946
model_extractions: Policy Research Working Paper 10359 How Well Do Internet-Based Surveys Track Labor Market Indicators in Middle-Income Countries?

_Vidhya Soundararajan Sidiki Soubeiga David Newhouse Amparo Palacios-Lopez Utz Johann Pape Michael Weber_ Poverty and Equity Global Practice March 2023
Policy Research Working Paper 10359

# **Abstract**

Online surveys are convenient, cost-effective, speedy, and increasingly popular instruments for data collection. This study investigates whether online surveys that used Random Domain Intercept Technology to recruit respondents were accurately measured labor market outcomes in six middle-income countries in the aftermath of the COVID-19 pandemic. Compared with the national average, online surveys oversampled males, youth, those with higher levels of education, and those in smaller households. Reweighting using propensity score estimates fails to equalize the means of variables excluded from the model. When comparing the employment-to-population ratio from the internet surveys to the most recent relevant nationally representative surveys, the average deviation is 30 percent. Reweighting using propensity scores in that case worsened the bias. Internet survey estimates of informal and self-employment rates also tend to be inconsistent with benchmark data, although the latter are available for fewer countries. Overall, the results suggest that despite the advantages and convenience of recruiting internet survey participants through Random Domain Intercept Technology, the resulting sample is not representative and even after propensity score reweighting, it can yield estimates that are at odds with nationally representative surveys.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at dnewhouse@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **<mark>How Well Do Internet-Based Surveys Track Labor Market Indicators in Middle-Income Countries?</mark>**<sup>**¥**</sup>

**<mark>Vidhya Soundararajan, Sidiki Soubeiga, David Newhouse, Amparo Palacios-Lopez, Utz Johann Pape, Michael Weber</mark>**

> ¥ We thank the RIWI corporation for sharing the data from their online surveys for this paper. The data were collected through the Random Domain Intercept Technology (RDIT), patented by RIWI corporation.
# **<mark>1. Introduction</mark>**

<mark>Internet-based surveys are a convenient option for policy makers and research organizations to collect data in a cost-effective way. Online surveys present many advantages. First, it is possible to achieve large sample sizes in a short amount of time, enabling the tracking of frequently changing events. This would enable policy makers to collect survey data more frequently on key policy indicators. Second, with the increased spread of the Internet, online surveys can be substantially more cost-effective than regular physical surveys in reaching out to more respondents. Third, the anonymity of respondents makes it possible to ask extremely sensitive questions, garner honest responses, and maintain the respondent’s safety, which could otherwise be difficult in face-to face or phone surveys (Braunsberger et al. 2007; Granello and Wheaton 2004).</mark> <mark>The COVID-19 pandemic curtailed economic activities worldwide and adversely affected labor market participation. These changes were abrupt and ever-evolving due to the unpredictable nature of the pandemic, which also in many countries prevented the implementation of face-toface surveys. In such a setting, internet surveys are a potentially attractive option for tracking the frequently changing economic situation.</mark> <mark>The main disadvantage of online surveys is that they are typically based on non-probabilitybased sampling techniques. Respondents are selected based on convenience and internet access rather than at random from a census-based sample frame.</mark><sup>1</sup> <mark>Online surveys are not nationally representative because some individuals in the target population may not have access to the internet, those with internet access may be more or less likely to be approached for a survey, and not all of those who are approached may respond to the survey. Since the probability of inclusion in the sample depends on respondent characteristics, the resulting sample is prone to selection bias. A large literature corroborates this evidence on the lack of representativeness of online surveys (e.g., Evans and Mathur 2005; Granello and Wheaton 2004; Steinmetz and Tijdens 2009; Steinmetz et al. 2014; Steinmetz, Tijdens, and de Pedraza 2009; Valliant and Dever 2011). Despite these issues, surprisingly, many surveys continue to be conducted online, for example, through river sampling (American Association for Public Opinion Research 2013), and many opinion polls continue to be conducted using nonprobability online panels</mark> <u><mark>(Callegaro, Villar, Yeager, and Krosnick 2014a; Callegaro, Baker, Bethlehem, Göritz, Krosnick, Lavrakas, 2014b).</mark></u> <mark>In comparison with Internet surveys, other modes of data collection have different strengths and weaknesses. Random Digit Dialing (RDD) phone surveys and traditional face-to-face surveys are common and more traditional alternatives. RDD is a method of probability sampling that provides a sample of households, families, or persons through a random selection of their telephone numbers. Since RDDs rely on the frame of those who own a phone, and not on the whole population, they may not be nationally representative. Biases can also arise based on the person who answers the phone call, because interviewers and respondents sometimes rush through conversations, and due to non-responses arising from network non-availability or other reasons (Holbrook et al. 2003).</mark><sup>2</sup> <mark>From the perspective of those requesting the survey, online</mark>

> 1 Increasingly, many companies are trying to conduct Internet surveys with a frame from which probability samples can be drawn (e.g., Kapteyn, Smith, and van Soest 2007; Malhotra and Kuo 2008; Moskalenko and McCauley 2009; Skitka and Bauman 2008), but these are rare because it is hard to develop a frame without knowledge of the universe of Internet users. 2 For example, Kohut et al. (2012) shows that RDD response rates decreased from 36% in 1997 to 9% in 2012.

2
<mark>surveys are cheaper to conduct compared to phone surveys.</mark><sup>3</sup> <mark>Traditional face-to-face surveys typically employ carefully designed probabilistic samples and are therefore representative. Another major advantage of in-person interviews is the ability to obtain high-quality and deeply insightful answers from these surveys, while Internet surveys are poorly suited for collecting answers to complex questions. However, face-to-face surveys are costly and are therefore difficult to implement frequently. They can also be difficult to field in conflict-affected environments due to security concerns.</mark> <mark>How important is the lack of representativeness of online surveys and to what extent can it be corrected using statistical reweighting techniques? To investigate these questions, we utilized data from online surveys to track labor market indicators during the COVID-19 pandemic. The surveys were administered over three data collection waves in Brazil, the Arab Republic of Egypt, Indonesia, Kenya, Sri Lanka, and Türkiye, between December 2020 and August 2021.</mark> <mark>Participants were recruited using Random Domain Intercept Technology (RDIT), a method that randomly selects internet users that mistype a web address or click on a broken link. The RDIT was developed by the RIWI corporation, and it works in the following way. RIWI takes ownership or temporary control of unused/abandoned websites or expired domains. Usually, users receive a “this page does not exist” message while they click on such websites while browsing the internet. However, because of RIWI’s ownership, users would now be led to a link to take up an RIWI survey. RIWI, by autodetecting the IP address of the survey participant, offers the survey in the local language. Participants are anonymized and do not get any incentives to take up the survey. The survey can reach countries where there could be potentially government restrictions because no entity has a list of RIWI-controlled domains, and RIWI often changes the list of domains it owns.</mark><sup>4</sup> <mark>RDIT has advantages over other methods of selecting Internet survey respondents including Internet panels or email-based solicitation. In panels, respondents might be conditioned to taking the survey and may even change their behavior if they have been repeatedly answering survey questions, or they may make more errors than fresh respondents, speeding through a survey or answering strategically to avoid follow-up questions (Toepoel et al. 2009). RDIT methodology, by the virtue of randomly targeting respondents and employing a repeated cross-section survey, avoids these issues. Email solicitation may also be problematic if the emails reach the respondents’ spam box, and because respondents are not anonymous (Schonlau and Couper, 2017; Roder-DeWan et al. 2019). Further, RDIT surveys are also consistent and reliable over time, as we see in our results in section 5. Another set of studies showed that an RDIT survey conducted every month for 21 months in India produced reliable estimates with low standard errors (Seeman et al. 2010; Seeman et al. 2016; Rizo et al, 2011). This is unlike online panel surveys where stability can be compromised due to high attrition among panel members (Toepoel et al. 2009).</mark>

> 3 However, responding to an online survey may not be cheap for responders living particularly in a Lower-Middle-Income (LMI) country, and having to pay a non-negligible amount of money to buy internet data. But data costs are increasingly cheaper in many developing countries.

> 4 RDIT has been used in several previous studies for examining attitudes related to mental illness, health care, and vaccination. For example, Sargent et al. (2022) tracked COVID-19 vaccination rates in real time, Seeman et al. (2016) studied attitudes on mental illness, and Roder-DeWan et al. (2019) studied the <mark>expectations of health care quality among low-income countries.</mark>

3
<mark>Despite these advantages, Internet surveys conducted with RDIT could be biased because of nonprobability-based sampling.</mark><sup>5</sup> <mark>To better understand these biases, we take two approaches to assess the quality of internet survey estimates. The first approach compares time-invariant measures, particularly demographic variables, calculated from the online surveys to nationally representative surveys collected prior to the crisis, contained in the World Bank’s Global Monitoring Database (GMD). The results indicate that online surveys represent a substantially different demographic group compared to the national picture, overrepresenting male, younger, and more educated members of the country’s population. Therefore, using the raw online survey results to track labor market indicators would be misleading.</mark> <mark>As a first quality check, we attempt to address the bias described above using a propensity score adjustment, based on the estimated probability of participation in the Internet survey. In particular, we estimate a logit model of selection into the online survey, as opposed to the GMD survey, in each country. The model is used to construct weights equal to the inverse of the predicted propensity scores. The results show that, for variables included in the propensity score calculation, the reweighting process produces a smaller deviation of the means of the internet survey from the GMD survey. The deviations between variables excluded from the propensity score model, however, remain relatively high. For example, the deviation for model variables was 32.0 percent in Brazil, but 56.8 percent for the non-model variables. In Indonesia, while the deviation for model variables was 15.3 percent, the deviation was 582.0 percent for non-model variables. This result highlights that it is problematic in this context to assume that selection bias becomes ignorable after conditioning on demographic characteristics such as age, gender, and education.</mark> <mark>The second quality check compares reweighted estimates of labor market indicators from the internet survey with other nationally representative data available in the same time period. This comparative data came from PNAD in Brazil, labor force surveys from Sri Lanka, Türkiye, and Indonesia, and the Continuous Household Survey from Kenya. Looking at the employment-topopulation ratio, the reweighted online survey captures the national picture reasonably well in Türkiye, but considerably overestimates the employment-to-population ratio in the other countries. On average, the internet surveys give employment-to-population ratios that are 30 percent above the rates from the benchmark data. We also examined the relationship between internet coverage in a country and how well the online survey in that country captures the employment-to-population ratio accurately. Notably, Türkiye has the lowest deviation and the highest Internet coverage, but other than that there is no systematic relationship between Internet coverage and prediction accuracy.</mark> <mark>We also examine two other labor outcomes, namely, the formal employment-to-total employment ratio, and the self-employment-to-total employment ratio. The sample size for these indicators is much smaller than that for employment as these questions were only administered for a subsample of those employed.</mark><sup>6</sup> <mark>The performance of the online survey in tracking formality</mark>

> 5 Besides non-probability-based sampling, another issue with standard Internet surveys is the potential duplication of responses. The same respondent can potentially answer the survey several times.  However, the RDIT can identify the device of the survey respondent using a combination of the respondent’s IP address, device details such as device type, internet browser, operating system, and other device details, and thus prevent the same device owner from participating in the survey multiple times. While the same person could potentially take the survey from multiple devices, the opportunity cost of time could prevent them from doing so.

> 6 The employment related questions were administered to all those taking up the survey. We define as employed those who mention that they were “employed at work” and “employed but not at work”.  The formality and self-employment related questions were administered only to those who are “employed at work”.

4
<mark>is poorer for these indicators, with the average relative absolute deviation of 39.1 percent, and for those in self-employment, it is even higher with a deviation of 166.9 percent.</mark> <mark>Previous studies from developed countries that attempt to reduce bias in online surveys find that the propensity score-based reweighting methods do not help in bringing the web survey</mark><sup>7</sup> <mark>estimates close to the probability-based reference survey/sample (Smyk et al. 2020; Schonlau et al. 2009; Yeager et al. 2011). We add to this literature by showing that these findings also tend to hold in developing countries. We compare the reduction in bias after adjusting the estimates based on propensity scores to estimates when we do not adjust at all. For the employment-topopulation ratio, the propensity score adjustment increased the average bias ratio across countries/waves by 18 percent, indicating that reweighting using propensity scores worsened the accuracy of the estimates. The employment measures that we are interested in are not included in propensity score matching models. To the best of our knowledge, other studies have only examined the accuracy of variables used in the propensity score model or in other calibration methods, and not the non-model variables. We are the first to attempt to study whether nonmodel variables can be estimated accurately using such adjustment techniques. We also contribute to the literature by testing our method across six countries, which gives a coarse sense of whether Internet surveys may be less biased in some settings over others.</mark> <mark>Overall, the results suggest considerable caution when interpreting labor market indicators collected through random intercept domain technology. Although the internet survey results do match benchmark data in some cases, the sample of internet responders is highly skewed, and in most cases, there are large discrepancies in outcomes even after reweighting. While there may be more room to experiment with the collection of additional demographic data to reweight the sample more effectively, achieving credibly representative estimates with internet surveys continues to be a major ongoing challenge.</mark> <mark>The next sections are laid out in the following format. Section 2 presents the previous literature on the methods and issues in utilizing online surveys to be representative of a larger target population. Section 3 presents the data sources. Section 4 describes the methodology. Section 5 presents the results, and section 6 concludes the study.</mark>

# **<mark>2. Literature and Methods Review</mark>**

## **<mark>2.1.Overview of Issues in Online Surveys</mark>**

There are two broad types of web surveys. The first one is a probability-based sample where there is a sampling frame with e-mail addresses available for all eligible persons and a random sample is asked to visit a website and complete a survey. The second type, in contrast, is a _nonprobability-based_ web survey, like entertainment surveys, self-selected web surveys, and surveys made up of volunteer panels of Internet users. In the latter type of surveys, the probability of selection differs for different members of the population, making the surveys less representative. For instance, in volunteer web surveys, open invitations on websites are used to select respondents.<sup>8</sup> The probability of receiving such an invitation and the probability of accepting it are unknown. Due to the lack of a sampling frame and volunteer-based data collection, it is difficult to ascertain the extent to which the results from these surveys can be generalized for the whole population (Valliant and Dever, 2011).

> 7 The terms web surveys and online surveys will be used interchangeably. 8 Many market research and survey companies use convenience sampling to track sentiment, behavior, consumer preferences, etc.

5
One can distinguish between three distinct populations. The first is the target population or the set of persons for which inferences are to be made. The second is the potential covered population, which in this case includes persons having access to the Internet and who use it. The third is the realized sample, which includes persons who are exposed to the offer to participate in the survey and accept the offer. The potential covered population may not cover the target population well because of those without Internet access or because of those who have access but do not use the Internet and are hence unreachable. The realized sample may not cover the target population or the potentially covered population. Due to self-selection, the reachable respondents among the potentially covered population may be a biased sample. Thus, the realized sample of volunteers will be expected to overrepresent or underrepresent some demographic groups of the target population, making inferences based on this sample biased.<sup>9</sup>

# **<mark>2.2.Methods for Correcting Bias</mark>**

<mark>Post-stratification, raking, propensity score adjustment (PSA), and matching are popular methods to address biases in online surveys. The efficacy of these different adjustment procedures in improving the representativeness of the web surveys is still debated (Bethlehem and Stoop 2007, Vehovar et al. 1999). Nonetheless, to apply these methods, we need a probability-based reference survey (or a subsample of the survey) that represents the distribution in the true population.</mark> <mark>Poststratification indicates a process of stratification after the sample has been selected and is often appropriate when the sample is not properly balanced by the representation. After the data is obtained, typically, weights are determined for each stratum by dividing the proportion of population in that stratum (from the reference-survey) by the proportion of the sample in that stratum.</mark> <mark>The most prevalent method for weighting is iterative proportional fitting, more commonly referred to as raking. With raking, a researcher chooses a set of variables where the population distribution is known, and the procedure iteratively adjusts the weight for each case in the internet sample until the sample distribution aligns with the population for those variables. Raking is popular because it is relatively simple to implement, and it only requires knowing the marginal proportions for each variable used in weighting.</mark> Propensity score adjustment aims to correct differences caused by the varying inclinations of individuals to participate in web surveys (Duffy et al. 2005). That is, it adjusts for selection-bias due to observed covariates. More technically, a propensity score ( _psi_ ) is the conditional probability that a person will be in one condition rather than in another (e.g., ‘being in the web or the reference survey’) given a set of observed covariates used to predict the person’s condition. The web surveys are then weighted by the inverse of the propensity score, that is 1/ _psi_ (see Lee 2006, Schonlau et al. 2004, 2007). Since the propensity score refers to both the web and reference survey respondents, the propensity score weight for the reference survey is the inverse of propensity of being in the reference survey (1/(1- _psi_ )).

> 9 This general criticism is not only applicable to web surveys. In phone surveys, too, for example, coverage may be incomplete due to non-response. However, it is possible to randomly sample respondents from the potential covered population because telephone users list is usually known. This contrasts with a Web survey where there is currently no way to identify all Internet users (Valliant and Dever, 2011).

6
While poststratification typically adjusts based on demographic variables, propensity score adjustments often use demographic and webographic (lifestyle/attitudinal) questions. Webographic variables capture general attitudes or behaviors that are hypothesized to differ between the web sample and the general population.<sup>10</sup> <mark>An alternative to propensity score adjustment is matching. This technique utilizes a sample of cases representative of the population. This serves as the “target” sample. The aim of the matching exercise is to pair each case in the target sample with the most similar case from the online opt-in sample. When the closest match has been found for all of the cases in the target sample, any unmatched cases from the online opt-in Internet survey sample are discarded. At the end of the procedure, the matched cases from the Internet survey should be a set that closely resembles the target population.</mark> <mark>There are several ways to conduct the matching procedure. To perform the matching, the target sample and the online survey data can be combined and a selection model for the Internet survey can be implemented as before. A propensity score can be constructed as in the above section, and matching can be based on those. If there are many variables for selection, machine learning algorithms, such as random forest, can be employed. Random forest algorithms can measure the probability that each case belongs to either the target sample or the survey and produce a measure of the similarity between each case and every other case. The similarity measure can also be used for matching. Another approach is to match based on Mahalanobis distance (Diamond and Sekhon, 2005).</mark>

# **<mark>2.3. Review of Studies Applying the above Methods</mark>**

<mark>The selection of covariates for constructing propensity scores has been widely studied.</mark><sup>11</sup> <mark>Covariate selection is usually done in a variety of ways, such as stepwise regression that drops variables that are not significant (Berk and Newton 1985, Lieberman et al. 1996), or conducting one-step covariate selection based on theoretical and logical significance. Some studies argue that there are no serious errors if the model for propensity score adjustment is mis-specified (Drake, 1993), other studies note that the choice of covariates is critical for propensity score weighting (Isaksson and Forsman, 2003). Covariates may be country specific (Stenbjerre, 2002). In countries with homogenous populations and high Internet penetration, there could be fewer covariates that discriminate web survey participation, unlike in countries that have low Internet penetration. There is also a general consensus in the literature that webographic questions, in addition to regular demographic variables, are needed to calculate the propensity scores more accurately.</mark> Several studies examine non-economic averages and how adjusting can potentially reduce bias between volunteer Internet surveys and non-Internet probability-based surveys. Most studies indicate that various weighting schemes do not adequately reduce the difference between weighted Internet survey data and the reference survey. Varedian and Forsman (2003) investigate

> 10 Many Internet survey companies, such as, Harris Interactive, increasingly use webographic questions that are thought to best capture the differences between the general population and those willing to answer a web survey.

> 11 There is a tendency to use only those covariates that are statistically significantly different between treatment and comparison groups. Rosenbaum (2002) offered three cautions against this approach because the relationship between the outcome variable and one covariate is not important, statistical significance is not relevant always especially because it depends on the sample size.

7
the efficacy of propensity score-based weights in a web marketing survey that asks questions on hygiene products and attitude towards local banks in a northern European country. They find that estimates from the web survey and the phone surveys (which they use as a comparison) are quite different, and various weighting schemes did not make a difference regarding this. Some studies find that the type of measure matters. Schonlau et al (2004) found among propensity score adjusted web-survey estimates, only 8 of 37 estimates were not significantly different from phone survey estimates. Web survey estimates were significantly more likely to agree with RDD phone survey estimates for found factual questions, and those regarding personal health, and when the questions involved two as opposed to multiple categories of responses. Lee and Valliant (2009) show that weighting adjustment that combines propensity score adjustment and calibration adjustment has the potential to reduce bias for volunteer panel web survey estimates that are contaminated by sample selection bias. However, their results are for model variables used for propensity score adjustment or in the calibration exercise, and not for non-model variables.

Malhotra and Krosnick (2007) compare the 2000 and 2004 American National Election Study (ANES) data collected via using probability sampling with simultaneous Internet surveys of volunteer samples. The non-probabilistic samples were weighted using the raking method using the US CPS so as to match sample proportions with the population proportions on age and education. A comparison of the results yielded many differences in the distributions of variables and in the associations between variables. Applying the weights did almost nothing to reduce the differences between modes/sampling methods in the distributions of the political variables. Chang and Krosnick (2009) simultaneously fielded a probabilistic telephone survey, a probabilistic Internet survey, and a non-probabilistic internet survey. The non-probabilistic Internet survey used a weighting procedure to adjust for variable propensity of individuals to have regular access to email and the Internet. The comparisons showed that th <mark>e probability samples were more representative than the nonprobability sample in terms of demographics and electoral participation, even after re-weighting.</mark> <mark>Brüggen et al. (2016) compare the accuracy of results obtained from 18 opt-in online “panels” participating in the Dutch Online Panel Comparison Study (NOPVO) with data from a probability sample collected from the Internet (the LISS panel; Scherpenzeel 2009), and two probability samples collected via computer assisted personal interviewing, and compared all of them with the Dutch government’s registry of all 16 million residents of the country, the Municipal Basic Administration (MBA). The non-probability sample surveys were weighted using General Regression (GREG) and Horvitz-Thompson Estimators. Results indicate that the nonprobability samples yielded less accurate estimates of proportions than the probability samples, and that weighting does not reduce selection bias in the level estimates. Dutwin and Buskirk (2017) compare RDD surveys and nonprobability Internet panel surveys to a highquality in-person survey. The surveys were weighted using propensity weighting, raking, and sample matching. Their results showed that nonprobability samples attained the greatest estimated bias, and the in-person sample, the lowest. The weighting techniques were not able to improve the measures. Maccinnis et al. (2018) compare data across a variety of probability and nonprobability sampling methods in the United States, using a set of 50 measures of 40 benchmark variables. The probability samples interviewed by telephone or the Internet were the most accurate. Internet surveys of a probability sample combined with an opt-in sample were less accurate; least accurate were internet surveys of opt-in panel samples. These results were not altered by implementing poststratification weights provided by survey companies.</mark><sup>12</sup>

> 12 Cornesse et al. 2020 provides an exhaustive literature review of studies comparing non-probability and probability samples.

8
Two studies have shown promising results in using non-probability-based surveys, and using statistical methods to make them representative. Goel et al. (2015) employed an online opt-in survey of social and political attitudes. The survey utilized 14 demographic questions and 49 attitudinal questions that were drawn from the 2012 General Social Survey (GSS) and recent Pew Research Center studies. To correct for the biases in the non-representative online data, the study generates population-level and subgroup-level estimates via model-based poststratification [Gelman and Little, 1997, Wang et al., 2015]. After statistical correction, the median absolute difference between non-probability and probability surveys is 7.4 percentage points. The authors find this to be comparable to the difference between various probability surveys themselves, and hence conclude that “non-representative surveys are a promising tool for fast, cheap, and (mostly) accurate measurement of public opinion.” Wang et al. (2015) conduct a series of daily voter intention polls in the Xbox gaming platform. After adjustments using regression and poststratification, the corrected estimates are comparable to estimates from a leading poll analyst that was based on aggregating traditional polls during the election cycle. From these results, the authors argue non-representative polling shows promise not only for election forecasting, but also for measuring public opinion on a broad range of social, economic and cultural issues.

Few studies look at economic variables. Smyk et al. (2020) study Wage Indicators (WI), an online survey in 17 countries and check if they provide wage distributions that match with benchmark data in each of these countries. They use covariate balancing propensity score to reweight the data. While these weights could match demography and human capital endowment of the representative samples, they are unable to match wage distributions from WI to the representative reference samples.

Propensity score adjustment seems to work better than post-stratification in settings where they have been compared directly. Isaksson and Forsman (2003) study political polls in Sweden and find that propensity score adjustment gives more accurate estimates than poststratification weighting. Yeager et al (2011) compared Internet surveys using non-probability samples of American adults with aggregate benchmark surveys. They tested the results for demographic and some non-demographic variables. Results indicate that post-stratification of non-probability samples did not consistently improve accuracy.

<mark>Szolnoki and Hoffmann (2013) compared a representative face-to-face survey with 2,000 respondents and a telephone survey with 1,000 respondents were compared with two online surveys, one based on quota sampling (2,000) and the other on snowball sampling (3,000) using identical questions. The objective was to assess consumer data for research in the wine business. The study did not conduct any reweighting for their online survey based on quota sampling. For the online survey with snowball sampling, they did not undertake reweighting because a weighting factor greater than five is seen as very problematic (Bandilla et al. 2003). Face-to-face data delivered the best results, followed by the telephone interviews and finally the online quota survey.</mark> <mark>Some studies do not directly study web surveys, but use data collected during regular survey interviews from households with and without Internet access. This helps avoid the possibility that collecting data via the Internet as opposed to face to face or over the phone may directly affect the answer to survey questions. Dever et al (2008) compare estimates collected during telephone interviews from households with and without Internet access using data from the 2003 Michigan Behavioral Risk Factor Surveillance System in the United States. Their analysis results</mark> 9
<mark>suggest that statistical adjustments can reduce, if not eliminate, coverage bias in the situation we study. Schonlau et al. (2009) study a survey of respondents that have Internet access in the Health and Retirement Study. They analyze whether propensity score and Genetic Matching algorithm can correct for imbalances between Internet and non-Internet samples for variables used for adjustment and for variables not used in the adjustment. Using these methods, they find that the estimated bias is almost always reduced, but significant differences in many cases remain.</mark> <mark>Mercer et al (2018) study public opinion polls and find that the choice of adjustment variables is more important than any reweighting method for the accuracy of estimates. R</mark> aking performs nearly as well as more elaborate techniques based on matching. In their application, b <mark>oth the demographic and political variables resulted in lower average bias than adjusting on demographics alone. V</mark> ery large sample sizes do not fix the shortcomings of online opt-in samples. While an online opt-in survey with 8,000 interviews may sound more impressive than one with 2,000, the study finds virtually no difference in accuracy. Ferri-Garcia (2020) uses Machine Learning methods as an <mark>alternative to propensity score adjustment (PSA) for reducing or eliminating selection bias in online surveys. The results indicate that the ML algorithms are more effective in removing selection bias than logistic regression when used for PSA, but this difference is not important when the dimensionality of the data is low, and the covariates are not very discriminant.</mark> <mark>In sum, the existing literature broadly suggests that</mark> various weighting schemes do not adequately reduce the difference between weighted Internet survey data and the reference survey. Propensity score adjustment seems to work better than post-stratification in settings where they have been compared directly, but other <mark>studies show that the choice of adjustment variables is more important than any reweighting method for the accuracy of estimates. Only one study examines economic indicators (wages) from web surveys. W</mark> hile this study was able to match demography and human capital variables with reference samples, they were unable to match wage distributions. Clearly, all studies are based on high income countries, with little research on developing countries. In this study, we try to address these gaps by studying the accuracy of Internet surveys in a set of six developing countries using the propensity score adjustment method.

# **<mark>3. Data and Descriptive Statistics</mark>**

# **<mark>3.1. Data Sources</mark>**

<mark>The study uses several sources of data. The primary data source is the web-survey data collected through the Random Domain Intercept Technology. The survey is offered to the online population at random, provided they incur a browsing data input error. Specifically, if an individual stumbles across a broken link, they will be invited to participate in the survey. After excluding bots, users are invited to participate in a survey in their country’s language. The targeting is based on the IP-address that also reveals information about different regions across the country. Respondents are identified by creating a unique non-personally identifiable respondent code which is a combination of the respondent’s IP address, device details such as device type, internet browser, operating system, and other device details. Using this code, respondents are filtered out from taking the survey more than once, as the various survey waves are designed as repeated cross-sections. While the same person could potentially take the survey from multiple devices, the opportunity cost of time could prevent them from doing so.</mark> <mark>The survey collected information on basic demographic details, such as age, gender, education, and household size. Beyond these basic details, the survey obtained information on respondents’</mark> 10
<mark>employment status and type, and how they coped with the COVID-19 crisis. Personally identifiable information, such as name, residential address, or identity card details, were not collected. Enticements were not offered to participate in the survey, and respondents could easily exit the survey if/when they wanted to.</mark>

|**Table**|**3.1: Survey timelines an**|**d sample siz**<br>|**es**<br>|||||
|---|---|---|---|---|---|---|---|
|||**Brazil**|**Egypt,**<br>**Arab**|**Indonesia**|**Kenya**|**Sri**<br>**Lanka**|**Türkiye**|
|**Panel A: Inter**|**net survey**<br>||**Rep.**|||||
|Wave-1|Dec 10<sup>th</sup>to 29<sup>th</sup>, 2020|1,098|1,071|1,198|1,024|1016|1,180|
|Wave-2|April 28<sup>th</sup>to May 9<sup>th</sup>, 2021|1,106|1,031|1,114|1,016|1,049|1,168|
|Wave-3|August 5<sup>th</sup> to23<sup>rd</sup>,2021|1,146|1,032|1,316|1,020|1,009|1,007|
|**Panel B: Glob**<br>|**al Monitoring Data base**|||||||
|Global<br>Monitoring<br>Database<br>**Panel C: Com**|**parative data from country**|Overall:<br>350,749<br>Internet:<br>272,598<br>Year: 2019<br>**labor force s**|Overall:<br>34,733<br>Internet:<br>11,043<br>Year:<br>2015<br>**urveys**|Overall:<br>804,698<br>Internet:<br>80,840<br>Year: 2018|Overall:<br>52,841<br>Internet:<br>12,355<br>Year: 2015|Overall:<br>61,768<br>Internet:<br>Not<br>availabl<br>e<br>Year:<br>2016|<br>Overall:<br>30,731<br>Internet:<br>20,256<br>Year: 2018|
|Sample size||193,000<br>HH/month||793,542<br>HHs and<br>203,464<br>individuals|11,997<br>individuals|25,750<br>HH|58,560 HH|
|Survey name<br>and<br>timeframe||PNAD:<br> Dec 2020;<br>May and,<br>June 2021||National<br>Labor<br>Force<br>Survey –<br>Indonesia;<br>February<br>2021.|KCBS-<br>Kenya;<br>28<sup>th</sup><br>Sep<br>2020 to<br>30<sup>th</sup><br>Nov<br>2020<br>(4<sup>th</sup>quarter<br>of<br>the<br>survey)|LFS<br>–<br>Sri<br>Lanka;<br>Oct<br>-<br>Dec<br>2021|Household<br>labor force<br>survey;<br>Jan - March<br>2021<br>Apr - June<br>2021|

<mark>Notes: Wave-1 data were collected between December 10</mark><sup>th</sup> <mark>and 29</mark><sup>th</sup> <mark>, 2020, wave-2 between April 28</mark><sup>th</sup> <mark>and May 9</mark><sup>th</sup> <mark>, 2021, and wave-3 between August 5</mark><sup>th</sup> <mark>and 23</mark><sup>rd</sup> <mark>, 2021. Each wave had over 1000 “completed” surveys per country. A “completed” survey is one where respondents answered until the eighteenth question, which included all questions related to employment status and characteristics. For the analysis, we use only completed surveys, and ignored partial-response surveys. The sample size for these surveys in each wave is presented in panel A in table 3.1.</mark> <mark>Second, we use the Global Monitoring Database, a probability-based survey, to compare demographic variables and individual attributes with our survey. The GMD is the repository of multitopic income and expenditure household surveys used by the World Bank to monitor global poverty and shared prosperity. The household survey data are typically collected by national</mark> 11
<mark>statistical offices in each country, and then compiled, processed, and harmonized.</mark><sup>13</sup> <mark>The data are nationally representative, and hence represent the demographics of the entire country. Since our survey targeted individuals and not households, we use the individual surveys from GMD for comparison. While the GMD survey years do not match our Internet survey years in all countries, they are not older than the online surveys by more than a few years. Since demographic characteristics, such as age and gender are unlikely to vary within a few years, the GMD is a valid comparison.</mark><sup>14</sup> <mark>Specifically, we used the GMD survey from Brazil in 2019, Egypt in 2015, Indonesia in 2018, Kenya in 2015, Sri Lanka in 2016, and Türkiye in 2018.</mark><sup>15</sup> <mark>The details are presented in panel B in table 3.1.</mark> <mark>Third, we compare labor market indicators from the online survey with probabilistic samplebased surveys through the pandemic depending on their availability. In Brazil, we use the National Household Sample Survey – PNAD, a nationally representative survey of 193,000 households per month, conducted during the pandemic in December 2020 and May/June 2021. In Indonesia, we use the labor force surveys conducted in August 2020 and February 2021, each comprising 793,542 and 203,464 households respectively. In Kenya, we use the Kenya Continuous Household Survey comprising 11,997 households conducted in the last quarter of 2020 (October to December 2020). We use labor force indicators from the fourth quarter of 2020’s Quarterly Report of the Sri Lanka Labor Force Survey covering up to 25,750 households, and from the Türkiye Household labor force survey covering 58,560 households for each quarter.</mark><sup>16</sup> <mark>The comparative data are drawn based on whether data are available in the same month as an online survey wave, or within one month before or after. The exception is Türkiye for which the comparative household labor force surveys are conducted two months prior to wave3 of the online survey. The details are presented in panel C in table 3.1.</mark><sup>17</sup>

# **<mark>3.2.  Data Description and Patterns</mark>**

Are the demographics represented in the online data nationally representative, or do they oversample or under-sample specific demographic groups? The answer to this question will indicate whether and how we can use the survey data to make broader conclusions, or if other adjustments will be required in order for them to be nationally representative. We compare the online survey’s wave-2 data with the GMD, and with the Internet-using population from the GMD if Internet usage data are available in that country.<sup>18</sup> The comparison with GMD relies on the assumption that the slightly dated GMD reflects the demographics at the time of the online survey accurately. Indeed, the pandemic itself could

> 13 <mark>The process of compiling the GMD is coordinated by the Data for Goals (D4G) team and supported by the six regional statistics teams in the Poverty and Equity Global Practice.</mark>

> 14 Admittedly, educational attainment might change drastically within a few years, and there have been instances of this in the past (for example, literacy rate changed from 53% in 2014 to 89% in 2019 in Côte d’Ivoire), but these types of drastic changes are minimal in the countries and period we study. For example, Brazil’s illiteracy rate among adults remained somewhat stable at 7.95% in 2015 and 6.8% in 2018; Indonesia’s adult illiteracy rate changed from 7.9% in 2015 to 8.55% in 2020; Sri Lanka’s illiteracy rate changed from 10.5% in 2016 to 7.9% in 2019 ( <mark>UNESCO Institute of Statistics,</mark> 2020).

15 The GMD was also available in Egypt in 2019, but the variable pertaining to Internet usage is only available in the 2015 survey. 16 For Sri Lanka, see http://www.statistics.gov.lk/LabourForce/StaticalInformation/QuarterlyReports/4thQuarter2020 (last accessed December 4, 2021); For <mark>Türkiye</mark> , see https://www.tuik.gov.tr/ (last accessed December 4, 2021).

> 17 Egypt is the only country for which we are unable to find labor force survey data in the same time frame as our online surveys were conducted.

> 18 <mark>We present all the demographic variables from the online data across waves in appendix figures A.4.1 to A.4.6. The figures indicate that the demographics captured by the online survey data are stable across different waves. Therefore, to make comparisons with GMD, we could potentially use any of the online survey waves. We use the median-wave, wave-2, for comparisons.</mark>

12
swiftly influence the distribution of demographic variables in countries. For example, the urban share of residents could have been heavily influenced by the internal migration of citizens returning to their home villages or to other rural areas or to take up remote work, or by people migrating from rural to urban areas in order to search for better livelihoods during the pandemic. These changes happened swiftly, in the aftermath of the onset of the pandemic in March 2020 in low- and middle-income countries (see, for example, evidence for Brazil (Garcia, 2021), SubSaharan Africa (FAO, 2020), and Peru (Dupraz-Dobias, 2020)). We take up such swift changes as a caveat while interpreting the data.

<mark>Data from figure 3.1 indicate that in Kenya and Indonesia, the urban share is well captured reasonably accurately in the online survey. For Brazil, the survey underrepresents urban areas. In Egypt and Sri Lanka, the survey overrepresents urban areas. Türkiye has a very high share of urban data points in the online survey, but location data are not available for Türkiye in the GMD. Overall, this mixed picture is consistent with the expectation that urbanization itself might be changing as a result of the COVID-19 pandemic, as noted above. Comparison with the Internet using population in GMD (figure 3.2) presents a different picture and shows that urban residents are underrepresented in the online survey. Female respondents are underrepresented in the online survey in all countries, which is consistent with many studies, particularly, Valliant and Dever (2011) for the United States. Younger people are overrepresented in online surveys in all countries and older people are underrepresented in the online survey, which is consistent with findings in Steinmetz et al. (2009) for the cases of Germany and the Netherlands, and by Valliant and Dever (2011) in the United States. These results are robust when we compare the online survey with the Internet-using population in GMD in figure 3.2.</mark> <mark>Primary and secondary education are underrepresented in the online survey across countries, except Sri Lanka for primary education and Indonesia for secondary education. This trend broadly holds true even if we compare the online survey with the Internet using population from GMD. Tertiary education is overrepresented in the online surveys in all countries. This holds true if we compare internet using population from GMD, except in the case of Indonesia. Consistent with this, the literature shows that those with higher education are overrepresented in online surveys in most countries, with one exception. For example, as per Steinmetz et al. (2009), in the Netherlands, low and medium educated persons are underrepresented, and in Germany highly educated persons are underrepresented. In the United States, Valliant and Dever (2011) find that individuals in web survey volunteer samples have better education and higher income.</mark><sup>19</sup> <mark>Households with one member are overrepresented in the online surveys, but households with three or more members are underrepresented in online surveys, and this is consistent in comparison with the Internet using population.</mark> <mark>Web surveys could be targeted to specific kinds of individuals, which adds to another layer of bias. Steinmetz et al. (2009) shows that low and medium education workers are more likely to fill the survey in Germany because the links are provided in the trade union website. In our case, the survey was posed to those who mistype web addresses or those who stumble on a broken link. While we are unable to quantify this bias, its direction is unclear. While those who access the Internet regularly are more likely to click on broken links, those who access the Internet sparsely are also likely to mistype addresses because of their unfamiliarity.</mark>

> 19 A possible reason why respondents in the Internet surveys have higher access to education than others is that those with access to Internet itself are more educated than those without access (Schonlau et al., 2009).

13
<mark>Overall, Internet surveys are biased in the demographic coverage of the population, in comparison to nationally representative data from the GMD. The sample is young, more educated, and more male compared to the national averages. The sample is not necessarily more urban than GMD, and the pattern of the bias depends on the country. This lack of a pattern in the urbanization bias could be because, as explained above, COVID-19 itself could have impacted urbanization that is not being reflected in the slightly dated GMD data.</mark>

**Figure 3.1. Absolute deviation of Online Surveys from GMD**

**<mark>Figure 3.2. Absolute deviation of Online Surveys from the Internet using sample in GMD</mark>**

14
# **<mark>4. Methodology</mark>**

<mark>Next, in an attempt to overcome the biases in the online survey, as observed in section 3.2, we construct weights based on the propensity scores to examine whether applying such weights would make online surveys nationally representative. For this, we estimated a logit model of the following type:</mark> <mark>Here, the dependent variable, y</mark> i <mark>, takes the value 1 for the online survey data (for a specific wave) and 0 for the Internet using population of the GMD. The exception is Sri Lanka where Internet usage is unavailable in the GMD, and where the dependent variable takes the value 0 for all GMD observations. X</mark> i <mark>is a vector that denotes the set of control variables that are associated with an individual’s participation in the online survey. We estimate this model for each crosssection of the online survey wave with the GMD. Equation (1) describes the possible factors that influence a person who has access to the internet and was exposed to the Internet survey, as to whether they take up the survey or not.</mark> <mark>Variables in X</mark> i <mark>include gender, age categories, education, rural versus urban residence, sector of work pre-COVID, household size categories, a variable set of assets specific to each country, and region fixed effects.</mark><sup>20</sup> <mark>We also include many interactions of these demographic variables between themselves, and with region.</mark><sup>21</sup> <mark>These demographic factors are highly correlated with an</mark>

> 20 <mark>The regional aggregations are different across countries, based on data availability. In some cases, the lowest administrative unit level where the data is available differs across Internet Surveys and GMD. In such cases, we harmonize at a higher level when running models or even in simply comparing indicators from the two datasets. Specifically, we define region at the state level in Brazil, province level in Indonesia and Sri Lanka, region level in Egypt, and county level in Kenya. For Türkiye, location data are missing in GMD, so region fixed effects are not used.</mark>

> 21 The specific interaction variables used in each country/wave model are available upon request.

15
<mark>individual’s inclination to take up the survey, as illustrated in section 3.2 (also observed in Dever (2011), and Valliant and Dever (2011)). Specifically, those who take up the Internet surveys are more likely to be younger individuals, more educated individuals, and males. Additionally, economic variables are important because higher income often correlates with whether an individual has the luxury of time to participate in the survey. Due to the unavailability of some economic variables in specific survey waves, models between GMD and wave-2 data do not include the asset variables, and models between GMD and wave-1 data do not include variables on assets and the pre-COVID sector of work. Moreover, in the education variable, the “no education” category was not collected in wave 1, and hence we use fewer categories on education in wave-1 regressions.</mark> <mark>Asset data in the online surveys varies across countries. In Brazil, the online survey collected information about the availability of land phones, cell phones, computer, and piped water. In Egypt, the online survey collected data on the availability of land phone, computer, and piped water. In Indonesia, the asset variables include the availability of land phone, computer, car, air conditioner, and fridge. In Türkiye, the data on assets include the availability of computer, car, air conditioner, cable connection, and stove. In Kenya, the data on assets include the availability of computer, piped water, modernized floor, and modernized wall. In Sri Lanka, the assets include toilet and piped water.</mark> <mark>In all regressions, standard errors were clustered at the region level, except in Türkiye where the regional indicators were not available.</mark><sup>22</sup> <mark>From the model in equation (1), we obtained the predicted probability of participation in the online survey (</mark> _<mark>p</mark>_ <mark>) and calculate weights for the online survey observations as</mark> _<mark>(1-p)/p</mark>_ <mark>. The GMD is weighted by its own survey weights. Theoretically, the weighted means of variables used in the model should match between the online surveys and the GMD (Li et al, 2018).</mark> <mark>We also test whether non-model variables match across the datasets. For this, we re-estimate the models after dropping the original model variables one by one. Using these restricted models, we predict the probability of participating in the online surveys (p), as before. Applying the weights as before, we compare the means of the dropped variable, and calculate the relative mean deviation of the Internet survey mean (for each wave) from the GMD mean.</mark>

# **<mark>5. Results</mark>**

# **<mark>5.1. Results from Reweighting</mark>**

<mark>This section presents the weighted averages of model variables based on propensity score weights obtained from the logit models in each country. Appendix tables A.4.1 to A.4.6 present the weighted averages from GMD (column 1), online survey wave-1 (column 2), online survey wave-2 (column 3), and online survey wave-3 (column 4), along with the confidence intervals. Results indicate that we do not reject the null that the GMD means are the same as the online</mark>

> 22 Clustering usually provides more conservative standard error estimates compared to robust standard errors. Still, we face the problem of the small number of clusters biasing our standard error estimates downwards because the number of regions in each country is low and this could lead to over-rejection of the null hypothesis (Cameron and Miller 2015). The ideal way to proceed is to make corrections by estimating the Wild cluster bootstrap- _t_ statistics which would provide more conservative standard errors. This downward bias will however not hurt us because, ideally, we are hoping that the null hypothesis of GMD and online survey mean equivalence is not rejected. But if it is not rejected with clustered standard errors, it will not be rejected with downward adjustments using the Wild method either.

16
<mark>surveys means in most instances across countries.</mark><sup>23</sup> <mark>Appendix tables A.4.7 to A.4.12 present the weighted averages along with confidence intervals for non-model variables. Notably, we reject the null hypothesis that the GMD means are same as the online survey means in more instances for the non-model variables compared to the model-variables. These details are summarized in table A.4.13.</mark> <mark>Compiling data from tables A.4.1 to A.4.12, tables A.4.14 and A.4.15 present the average of the relative deviations of online survey means with GMD means for model-variables and non-model variables, respectively. In both these tables, column (1) specifies the online survey waves that were considered in obtaining the deviation. For example, the variable age is available in all waves, and hence age was dropped to estimate models between wave-1 with GMD, wave-2 with GMD, and wave-3 with GMD. The average presented for age is the relative mean from all the three waves, and this is represented as ‘1-2-3’ in column 1. Since asset variables were only available in wave-3, the models that drop the asset variables were only estimated between wave3 and GMD. This is represented as ‘3’ in column 1 for all asset variables. Columns (3)-(8) present the deviation for each country.</mark> <mark>Results indicate that the average relative deviations for model variables between the Internet surveys and GMD range from 0 precent for cell phone usage in Egypt, to 87.1% percent for the “no education” category in Egypt. For non-model variables, the average varies widely, ranging from as low as 0.4 percent for computer usage in Brazil, to as high as 10,722.2 percent for working in “other sector” in Indonesia. Figure 5.1 presents a boxplot of the average deviations across all countries for model variables (panel A) and non-model variables (panel B). The spread of the deviation and the mean values of deviation are both lower for model variables compared to non-model variables. The mean values of these deviations for each country are reported in table 5.1.</mark> <mark>It is useful to check if the models perform better in countries that have higher Internet coverage data. For this, we present data on Internet coverage across the countries from various official sources (red dot markers in the figure) in the second y-axis and arrange the countries in the x- axis in ascending order of internet coverage. Internet coverage in 2019 is highest in Türkiye and Brazil, followed by Egypt, Indonesia, Sri Lanka, and Kenya in that order. For model variables, countries with higher Internet coverage seem to have lower median values of deviation on average. Türkiye, which has the lowest mean deviation for the model variables (as per table 5.1), has the highest Internet coverage. For non-model variables, evidently, there is no clear indication that the best performing models are in countries where Internet coverage is the highest. The median deviation for Kenya, the country that has the least Internet coverage, seems close to the median deviation for Türkiye, the country that has the highest Internet coverage. Collectively, these results indicate that even after reweighting using inverse probability weights, it is challenging to match the means of non-model variables of the online surveys with nationally representative surveys.</mark>

> 23 <mark>Online survey means statistically significantly different from GMD are indicated by yellow shaded cells in tables A.4.1 to A.4.6.</mark>

17
<!-- Start of picture text --> Q s = a o s<br>5 E<br>2. Le<br>esFa>o3sae8s 739 e74 8 no]ZY=2° @ra©3ae88 ° 9 74 g zz~~2°<br>2 38 2 22<br>= 0573 ef = 873 os<br>5 8 v 6 8 %<br>i)2 477 sg =°2 7 5g<br>6 8 28 £8 28<br>i o £ 3<br>Bo3 i)€ ao& o£<br>zsg | 6 g = Zi °a<br>2 eR 28<br>= =<br>$ 3<br>So 3 Zo s<br>Kenya Sri Lanka Indonesia Egypt Brazil Turkey Kenyayi Sri Lanka Indonesia EgyptgyPt Brazil Turke y<br><!-- End of picture text -->
<mark>comparative data, the definition of “employed” is the same as used in the online survey. That is, someone is defined as employed during the reference week if they performed some work for at least one hour or were temporarily not at work but had a job to go back to. Figure 5.2 presents a scatter plot of the employment to population ratio in online surveys and the comparative data.</mark><sup>24</sup> <mark>The 45-degree line is presented for reference. Evidently, the online survey estimates are mostly above the 45-degree line, indicating that the online survey perhaps overrepresents the employment to population ratio.</mark> We calculate the relative bias as the absolute value of the difference between the online estimates and the comparative data-based estimates divided by comparative data estimates. For the propensity score adjusted online estimates, the bias ranges from 9 percent to 55 percent, with an average of 30.0 percent. Without weight adjustments, the deviations ranged from 3 percent to 62 percent, with around the same average of 29.8 percent. Notably, the weight adjustment sometimes improved the accuracy of Internet sample surveys and sometimes reduced their accuracy. We also calculate bias reduction due to propensity score adjustment as (bias of unadjusted online survey estimates - bias of adjusted online survey estimates)/bias of unadjusted online survey estimates. We find the average of the bias reductions across countries/waves to be -18 percent, indicating that the weight adjustments overall worsened the accuracy of Internet sample surveys.<sup>25</sup>

**<mark>Figure 5.2: Employment to population ratio: Internet survey and comparative data</mark>**

<!-- Start of picture text --> Employment to Population Ratio comparisons<br>Avereage relative deviation of RIWI: 30.03%<br>90<br>45 deg line<br>80 Brazil<br>Indonesia<br>70<br>Kenya<br>Sri Lanka<br>60<br>Turkey<br>50<br>40<br>30<br>30 40 50 60 70 80 90<br>Comparative Data<br>RIWI<br><!-- End of picture text --> 24 The results for each country are presented as bar graphs in appendix figures A.4.6 to A.4.11. These figures also include comparative summary data from other nationally representative sources as line graphs whenever available. 25 We do not have enough information from the comparative data to test the statistical significance of employment measures between our online surveys and the comparative data.

19
<mark>Note: Each point represents the combination of employment to population ratio as (comparative data, Internet survey data). Average relative deviation is the average of relative values of absolute deviation of the Internet survey from the comparative data. Specifically, we calculate the average of abs(Onlinecomparative)/comparative.</mark> <mark>Data on other labor market indicators were collected by the survey, including the formal versus informal nature of employment, and type of employment. Respondents are classified to be working formally if they have a written contract, if they have any form of social insurance, and if they get annual paid or sick leave. Those who do not satisfy any of these three conditions are deemed to be working informally. The respondents are classified as self-employed if they own a business or business operated by the household or a family farm. They are instead classified as an employee if they work as an employee for another house, or another company, or as an apprentice, trainee, etc. The questions related to formality and self-employment are asked only to those who responded that they are</mark> _<mark>currently</mark>_ <mark>employed at work, and not to those employed but not</mark> _<mark>currently</mark>_ <mark>at work. Therefore, the survey sample size for these questions is lower than for the previous graph.</mark> <mark>The comparative data for formal and self-employment are available for fewer countries and fewer time periods. Figure 5.3 and figure 5.4 present a scatter plot of the formal share between online surveys and the comparative data, and a scatter plot of the self-employment share between online surveys and the comparative data, respectively. For the propensity score adjusted online estimates of formal employment, the deviations from the comparative data range from 12.7 percent to 55.04 precent, with an average of 39.1 percent. Without weight adjustments, the deviations ranged from 14 percent to 29 percent, with a lower average of 21 percent. The average of bias reductions due to propensity score weight-based adjustments across countries/waves is - 1.20 percent. Here again, the weight adjustment worsened the accuracy of Internet sample surveys. For the propensity score adjusted online estimates of self-employment, the deviations from comparative data range from 6 percent to 377 percent, with an average of 166.9 percent. Without weight adjustments, the deviations ranged from 38 percent to 402 percent, with a higher average of 210 percent. The average of bias reductions due to propensity score weight-based adjustments across countries/waves is 26 percent. In this case, the weight adjustment improved the accuracy of Internet sample surveys.</mark> 20
**<mark>Figure 5.3. Strictly Formal Employment Share comparisons</mark>**

<!-- Start of picture text --> 35<br>30<br>25<br>20 45 degree<br>Brazil<br>15 Kenya<br>10<br>Average relative deviation of RIWI: 39.12%<br>5<br>0<br>0 5 10 15 20 25 30 35<br>Comparison data<br>RIWI<br><!-- End of picture text --> <mark>Note: Each point represents the combination of formal employment share (comparative data, Internet survey data). Average relative absolute deviation is the average of relative values of absolute deviation of the Internet survey from the comparative data. Specifically, we calculate the average of abs(Internet-Comparative)/comparative.</mark>

**<mark>Figure 5.4. Self-employment share comparisons</mark>**

<!-- Start of picture text --> 80<br>70<br>60<br>50<br>45 degree<br>40<br>Brazil<br>30<br>Indonesia<br>20 Kenya<br>10 Avereage relative deviation of RIWI: 166.90%<br>0<br>0 10 20 30 40 50<br>Comparison data<br>RIWI<br><!-- End of picture text --> <mark>Note: Each point represents the combination of self-employment share (comparative data, Internet survey data). Average relative absolute deviation is the average of relative values of absolute deviation of the Internet survey from the comparative data. Specifically, we calculate the average of abs(Internet-Comparative)/comparative.</mark> <mark>Since we do not have comparative literature focusing on employment status or related indicators such as self-employment and formality rate, we are unable to compare the bias itself directly with the other estimates in the literature. However, we can compare our bias-reduction from propensity score weights-based adjustments with others’. For example, using propensity score</mark> 21
<mark>weights, and a combination of propensity score weighting and calibration, Lee and Valliant (2009) find a bias reduction in the range of 1 percent to 94 percent for health status, and 18 percent to 98 percent for health insurance coverage. However, both these variables themselves were used in the propensity score weight estimation, and health status was used in the calibration of weights. Specifically, the study finds that matching after adjustments is better in cases where the variable in question itself is used for the adjustment. In other words, the matching for model variables performs better than non-model variables, a result we also observe.</mark>

# **<mark>6. Conclusion</mark>**

<mark>Traditional surveys that track economic and social indicators use probability sampling techniques based on a complete sampling frame. While these surveys accurately represent the national population, they are expensive and difficult to conduct in national crisis situations, such as the COVID-19 pandemic or fragile or conflict affected environments. Quicker, less expensive, and more convenient methods of data collection have emerged, including phone surveys and online surveys. While Random Digit Dialing technique-based phone surveys are based on a representative sampling frame, they may be biased, since the population of phone owners who would answer the phone and respond to the survey may differ from the general population. Internet surveys are even more convenient and inexpensive to administer, but their scope in developing countries could be curtailed by lower levels of internet access and educational attainment. These surveys are based on non-probabilistic sampling frames because it is difficult to ascertain the universe of Internet users.</mark> <mark>In this study, we utilized three waves of online surveys conducted in six countries during the COVID-19 pandemic between December 2020 and August 2021. The objectives were to capture key economic variables during the pandemic that would have otherwise been harder to collect through physical door-to-door surveys. We compared economic indicators from these surveys with regular labor force surveys conducted in these countries around the same time in order to assess whether these surveys accurately captured and tracked indicators. We tracked three key indicators, namely, the employment-to-population ratio, the formal share of employment, and the share of self-employment.</mark> <mark>Examining the demographic representation of the online survey data, we find that the raw or unweighted online-survey data are not nationally representative. They overrepresent men, younger individuals, and the highly educated. To attempt to overcome these biases, we estimated a propensity score model to estimate weights to make the internet survey data more representative. For this, we estimated a model to predict the selection of observations in the online survey compared to a nationally representative survey from the Global Monitoring Database (GMD) using a variety of independent variables. From this model, we further predicted inverse probability weights for adjustments. The means of variables included in the propensity score model seem to be relatively close to the GMD means, in comparison to the non-model variables whose online survey means deviated widely from the GMD mean. For example, the deviation for model variables was 32.0 percent in Brazil, but 56.8 percent for the non-model variables. In Indonesia, while the deviation for model variables was 15.3 percent, the deviation was 582.0 percent for non-model variables.</mark> <mark>Applying these estimated weights, we compared the economic indicators from the online survey with those derived from nationally representative comparative labor force surveys available during the pandemic. The results indicate that there is substantial deviation of the online survey</mark> 22
<mark>indicators from the labor surveys. The average across countries of the relative values of the absolute deviation of the employment-to-population ratio from the online survey to that of the labor force survey data is 30.0 percent. The same figure for formal share of employment is 39.1 percent, and for self-employment it is 166.9 percent. The reweighting procedure worsens the accuracy of online estimates of the employment-to-population ratio by 18 percent on average, and the formal employment share by 1.2 percent. In the case of self-employment rate estimates, model-based re-weighting reduced the bias by 26.4 percent.</mark> <mark>We conclude that the bias from non-representative Internet survey data can be substantial. Corrections using propensity score-based weights based on a small set of demographic variables can actually worsen this bias, and therefore, the reliance on non-representative internet surveys could be misleading. However, Internet surveys may be useful to identify trends in times of emergency among the population that they represent (mostly men, younger individuals, and the highly educated) for specific purposes. On the other hand, phone surveys are much preferable to Internet surveys because they can be drawn from a nationally representative sample or utilize RDD when mobile phone coverage is high.</mark> 23
# **<mark>REFERENCES</mark>**

American Association for Public Opinion Research (2013). “Report of the AAPOR Task Force on Nonprobability Sampling,” available at <u>https://www.aapor.org/AAPOR_Main/media/MainSiteFiles/NPS_TF_Report_Final_7_revised _FNL_6_22_13.pdf</u> (accessed September 30, 2019).

<mark>Bethlehem, J., & Stoop, I. (2007). Online panels. A paradigm theft?</mark> _<mark>The challenges of a changing world</mark>_ <mark>, 113-131.</mark> <mark>Braunsberger, K., Wybenga, H., & Gates, R. (2007). A comparison of reliability between telephone and web-based surveys.</mark> _<mark>Journal of Business Research</mark>_ <mark>,</mark> _<mark>60</mark>_ <mark>(7), 758-764.</mark> <mark>Bruggen, E., J. A. van den Brakel, and J. Krosnick (2016), “Establishing the Accuracy of Online Panels for Survey Research,” Statistics Netherlands. Available at https://www.cbs.nl/en-gb/background/</mark> <mark>2016/15/establishing-the-accuracy-of-online-panels-for-survey-research (accessed September 30, 2019).</mark> <mark>Callegaro, M., Villar, A., Yeager, D. S., & Krosnick, J. A. (2014a). A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples. Online Panel Research: A Data Quality Perspective, Wiley (2014), pp. 23-53</mark> <mark>Callegaro, M., Baker, R. P., Bethlehem, J., Göritz, A. S., Krosnick, J. A., & Lavrakas, P. J. (Eds.). (2014b). Online panel research: A data quality perspective. John Wiley & Sons.</mark> <mark>Cameron, A. C., & Miller, D. L. (2015). A practitioner’s guide to cluster-robust inference. Journal of human resources, 50(2), 317-372.</mark> <mark>Chang, L., and J. A. Krosnick (2009), “National Surveys via RDD Telephone Interviewing versus the Internet,” Public Opinion Quarterly, 73, 641–678.</mark> <mark>Cornesse, C., Blom, A. G., Dutwin, D., Krosnick, J. A., De Leeuw, E. D., Legleye, S., ... & Wenz, A. (2020). A review of conceptual approaches and empirical evidence on probability and nonprobability sample survey research. Journal of Survey Statistics and Methodology, 8(1), 4-36.</mark> <mark>Dever, J. A., Rafferty, A., & Valliant, R. (2008, June). Internet surveys: Can statistical adjustments eliminate coverage bias? In Survey Research Methods (Vol. 2, No. 2, pp. 47-60).</mark> <mark>Diamond, A., & Sekhon, J. S. (2013). Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies.</mark> _<mark>Review of Economics and Statistics</mark>_ <mark>,</mark> _<mark>95</mark>_ <mark>(3), 932-945.</mark> <mark>Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005). Comparing data from online and face-to-face surveys.</mark> _<mark>International Journal of Market Research</mark>_ <mark>,</mark> _<mark>47</mark>_ <mark>(6), 615-639.</mark> 24
<mark>Dupraz-Dopias (2020). COVID-19 shifts Peru’s internal migration into reverse. The New Humanitarian. 8 July 2020.</mark> Dutwin, D., and T. D. Buskirk (2017), “Apples to Oranges or Gala versus Golden Delicious?,” Public Opinion Quarterly, 81, 213–239.

<mark>Evans, J. R., & Mathur, A. (2005). The value of online surveys.</mark> _<mark>Internet research</mark>_ <mark>.</mark> <mark>Fan Li, Kari Lock Morgan & Alan M. Zaslavsky (2018) Balancing Covariates via Propensity Score Weighting, Journal of the American Statistical Association, 113:521, 390-400, DOI: 10.1080/01621459.2016.1260466</mark> <mark>FAO (2020). Migrant workers and remittances in the context of COVID-19 in sub-Saharan Africa. Rome. Last accessed on March 29</mark><sup>th</sup> <mark>, 2022. Link: https://doi.org/10.4060/cb0184en</mark> <mark>Ferri-García, R., & Rueda, M. D. M. (2020). Propensity score adjustment using machine learning classification algorithms to control selection bias in online surveys. PloS one, 15(4), e0231500.</mark> <mark>Garcia, R</mark> aphael ( <mark>2021). Can Brazil’s remote workers thrive outside big cities? BBC, January 2021. Last access on March 29</mark><sup>th</sup> <mark>, 2022. Link: https://www.bbc.com/worklife/article/20210121-</mark> <u><mark>the-challenge-of-remote-work-in-developing-nations</mark></u> <mark>Goel, S., Obeng, A., & Rothschild, D. (2015). Non-representative surveys: Fast, cheap, and mostly accurate.</mark> _<mark>Work Pap</mark>_ <mark>.</mark> <mark>Granello, D. H., & Wheaton, J. E. (2004). Online data collection: Strategies for research.</mark> _<mark>Journal of Counseling & Development</mark>_ <mark>,</mark> _<mark>82</mark>_ <mark>(4), 387-393.</mark> <mark>Holbrook, A. L., Green, M. C., & Krosnick, J. A. (2003). Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent satisficing and social desirability response bias.</mark> _<mark>Public opinion quarterly</mark>_ <mark>, 67(1), 79125.</mark> <mark>Andrew Kohut, Scott Keeter, Carroll Doherty, Michael Dimock, and Leah Christian. 2012. Assessing the representativeness of public opinion surveys. Pew Research Center, Washington, DC, 2012.</mark> <mark>Kapteyn, A., Smith, J. P., & Van Soest, A. (2007). Vignettes and self-reports of work disability in the United States and the Netherlands.</mark> _<mark>American Economic Review</mark>_ <mark>,</mark> _<mark>97</mark>_ <mark>(1), 461-473.</mark> <mark>Lee, S. (2006). Propensity score adjustment as a weighting scheme for volunteer panel web surveys.</mark> _<mark>Journal of official statistics</mark>_ <mark>,</mark> _<mark>22</mark>_ <mark>(2), 329.</mark> <mark>Macinnis, Bo, Jon A. Krosnick, Annabell S. Ho, and Mu-Jung Cho. 2018. The accuracy of measurements with probability and nonprobability survey samples. Public Opinion Quarterly, Vol. 82, No. 4, Winter 2018, pp. 707–744.</mark> 25
<mark>Malhotra, N., and J. A. Krosnick (2007), “The Effect of Survey Mode and Sampling on Inferences about Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with Nonprobability Samples,” Political Analysis, 15, 286–323.</mark> <mark>Malhotra, N., & Kuo, A. G. (2008). Attributing blame: The public's response to Hurricane Katrina.</mark> _<mark>The Journal of Politics</mark>_ <mark>, 70(1), 120-135.</mark> <mark>Mercer, A., Lau, A., & Kennedy, C. (2018). For weighting online opt-in samples, what matters most? Pew research center. January 2018.</mark> <mark>Moskalenko, S., & McCauley, C. (2009). Measuring political mobilization: The distinction between activism and radicalism.</mark> _<mark>Terrorism and political violence</mark>_ <mark>,</mark> _<mark>21</mark>_ <mark>(2), 239-260.</mark> Pasek, J. (2016), “When Will Nonprobability Surveys Mirror Probability Surveys? Considering Types of Inference and Weighting Strategies as Criteria for Correspondence,” International Journal of Public Opinion Research, 28, 269–291 Rizo C, Deshpande A, Ing A, Seeman N. A rapid, Web-based method for obtaining patient views on effects and side-effects of antidepressants. Journal of Affective Disorders. 2011;130(1-2):290-3. doi: 10.1016/j.jad.2010.07.027.

Roder-DeWan, S., Gage, A. D., Hirschhorn, L. R., Twum-Danso, N. A., Liljestrand, J., AsanteShongwe, K., Rodríguez, V., Yahya, T., & Kruk, M. E. (2019). Expectations of healthcare quality: A cross-sectional study of internet users in 12 low-and middle-income countries. PLoS Medicine, 16(8), e1002879. https://doi.org/10.1371/journal.pmed.1002879 Rosenbaum, Paul R. 1987. ‘‘Model-Based Direct Adjustment.’’ Journal of the American Statistical Association 82:387-94.

Sargent RH, Laurie S, Weakland LF, Lavery JV, Salmon DA, Orenstein WA, Breiman RF (2022). Use of Random Domain Intercept Technology to Track COVID-19 Vaccination Rates in Real Time Across the United States: Survey Study. J Med Internet Res 2022;24(7):e37920. https://www.jmir.org/2022/7/e37920/ Scherpenzeel, A. C., and J. G. Bethlehem (2011), “How Representative Are Online Panels? Problems of Coverage and Selection and Possible Solutions,” in Social and Behavioral Research and the Internet: Advances in Applied Methods and Research Strategies, eds.M. Das, P. Ester, and L. Kaczmirek, pp. 105–132, New York: Routledge.

Seeman, N., Tang, S., Brown, A. D., & Ing, A. (2016). World survey of mental illness stigma. Journal of affective disorders, 190, 115-121.

Schonlau M, Couper MP. Options for Conducting Web Surveys. Statist Sci. 2017;32(2):27992. doi: 10.1214/16-STS597.

Schonlau, Matthias, Kinga Zapert, Lisa Payne Simon, Katherine Sanstad, Sue Marcus, John Adams, Mark Spranca, Hongjun Kan, Rachel Turner, and Sandra Berry. 2004. ‘‘A Comparison Between a Propensity Weighted Web Survey and an Identical RDD Survey.’’ Social Science Computer Review 22:128-38.

26
<mark>Schonlau, M., Van Soest, A., Kapteyn, A., & Couper, M. (2009). Selection bias in web surveys and the use of propensity scores.</mark> _<mark>Sociological Methods & Research</mark>_ <mark>,</mark> _<mark>37</mark>_ <mark>(3), 291-318.</mark> <mark>Skitka, L. J., & Bauman, C. W. (2008). Moral conviction and political engagement.</mark> _<mark>Political Psychology</mark>_ <mark>,</mark> _<mark>29</mark>_ <mark>(1), 29-54.</mark> <mark>Smyk, M., Tyrowicz, J., & Van der Velde, L. (2021). A cautionary note on the reliability of the online survey data: the case of wage indicator.</mark> _<mark>Sociological Methods & Research</mark>_ <mark>,</mark> _<mark>50</mark>_ <mark>(1), 429464.</mark> <mark>Steinmetz, S., & Tijdens, K. (2009). Can weighting improve the representativeness of volunteer online panels? Insights from the German Wage indicator data.</mark> _<mark>C&M Newsletter</mark>_ <mark>,</mark> _<mark>5</mark>_ <mark>.</mark> <mark>Steinmetz, S., Bianchi, A., Tijdens, K., & Biffignandi, S. Improving web survey quality– Potentials and constraints of propensity score weighting.</mark> _<mark>Online Panel Research: A Data Quality Perspective. Wiley</mark>_ <mark>.</mark> <mark>Szolnoki, G., and D. Hoffmann (2013), “Online, Face-to-Face and Telephone Surveys— Comparing Different Sampling Methods in Wine Consumer Research,” Wine Economics and Policy, 2, 57–66.</mark> <mark>UNESCO Institute of Statistics, 2022.</mark> Other Policy Relevant Indicators (OPRI) <mark>, 2022. Link:</mark> <u><mark>https://apiportal.uis.unesco.org/bdds. Last accessed on April 5, 2022.</mark></u> <mark>Valliant, R., & Dever, J. A. (2011). Estimating propensity adjustments for volunteer web surveys.</mark> _<mark>Sociological Methods & Research</mark>_ <mark>,</mark> _<mark>40</mark>_ <mark>(1), 105-137.</mark> <mark>Vehovar, V., Manfreda, K. L., & Batagelj, Z. (1999, May). Web surveys: Can the weighting solve the problem. In</mark> _<mark>Proceedings of the Section on Survey Research Methods, American Statistical Association</mark>_ <mark>(pp. 962-967).</mark> 27