metadata: authors: 0: author: Newhouse,David Locke
1: author: Swindle,Rachel Elizabeth
2: author: Wang,Shun
3: author: Merfeld,Joshua David
4: author: Pape,Utz Johann
5: author: Hirfrfot,Kibrom Tafere
6: author: Weber,Michael
count: World
docna: 0: docna: How Well Did Real-Time Indicators Track
            Household Welfare Changes in Developing Countries during the
            COVID-19 Crisis?
docty: Policy Research Working Paper
abstracts: cdata!: This paper investigates the extent to
            which real-time indicators derived from internet search,
            cell phones, and satellites predict changes in household
            socioeconomic indicators across approximately 300
            administrative level-1 regions in 20 countries during the
            COVID-19 crisis. Measures of changes in socioeconomic status
            in each region are taken from high-frequency phone surveys.
            When using the first wave of data, fielded between April and
            August 2020, models selected using the least absolute
            shrinkage and selection operator explain 37 percent of the
            cross-regional variation in the share of households
            reporting declines in total income and 34 percent of the
            share of respondents reporting work stoppages since the
            onset of the crisis. Real-time indicators explain a lower
            amount of the within-region variation in income losses and
            current employment over time, with an R2 of 15 percent for
            current employment and 22 to 26 percent for the prevalence
            of income declines. When limiting the sample to urban
            regions, real-time indicators are far more effective at
            explaining within-region variation in income losses and
            current employment, with R2 values of approximately 0.54 and
            0.38, respectively. Income gains, self-reported food
            insecurity, social distancing behavior, and child school
            engagement are more difficult to predict, with R2 values
            ranging from 0.06 to 0.17. Google search terms related to
            food, money, jobs, and religion were the most powerful
            predictors of work stoppage and income declines in the first
            survey wave, while those related to food, exercise, and
            religion better tracked changes in income declines and
            employment over time. Google mobility measures are also
            strong predictors of changes in employment and the
            prevalence of specific types of income declines. In general,
            satellite data on vegetation, pollution, and nighttime
            lights are far less predictive. Google mobility and search
            data, and to a lesser extent vegetation and pollution data,
            can provide a meaningful signal of regional economic
            distress and recovery, particularly during the early phases
            of a major crisis such as COVID-19.
display_title: How Well Did Real-Time Indicators
            Track Household Welfare Changes in Developing Countries
            during the COVID-19 Crisis?
listing_relative_url: /research/2024/09/34392677/well-real-time-indicators-track-household-welfare-changes-developing-countries-during-covid-19-crisis
topicv3: Geographic Information Systems m1326400 663,Shocks and Vulnerability to Poverty
            m1326691 1160,Poverty Measurement and Analysis m1326941 707
publishtoextweb_dt: 2024-09-18T00:00:00Z
dois: 10.1596/1813-9450-10916
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 46
author_keys: 0: author_key: 450079
1: author_key: 1770257
2: author_key: 1984203
3: author_key: 1762622
4: author_key: 1154467
5: author_key: 1579492
6: author_key: 1008946
model_extractions: Policy Research Working Paper 10916 How Well Did Real-Time Indicators Track Household Welfare Changes in Developing Countries during the COVID-19 Crisis?

_David Newhouse Rachel Swindle Shun Wang Joshua D. Merfeld Utz Pape Kibrom Tafere Michael Weber_ Development Economics Poverty and Equity Global Practice & Human Capital Project September 2024
Policy Research Working Paper 10916

# **Abstract**

This paper investigates the extent to which real-time indicators derived from internet search, cell phones, and satellites predict changes in household socioeconomic indicators across approximately 300 administrative level-1 regions in 20 countries during the COVID-19 crisis. Measures of changes in socioeconomic status in each region are taken from high-frequency phone surveys. When using the first wave of data, fielded between April and August 2020, models selected using the least absolute shrinkage and selection operator explain 37 percent of the cross-regional variation in the share of households reporting declines in total income and 34 percent of the share of respondents reporting work stoppages since the onset of the crisis. Realtime indicators explain a lower amount of the within-region variation in income losses and current employment over time, with an R2 of 15 percent for current employment and 22 to 26 percent for the prevalence of income declines. When limiting the sample to urban regions, real-time indicators are far more effective at explaining within-region variation in income losses and current employment, with R2 values of approximately 0.54 and 0.38, respectively. Income gains, self-reported food insecurity, social distancing behavior, and child school engagement are more difficult to predict, with R2 values ranging from 0.06 to 0.17. Google search terms related to food, money, jobs, and religion were the most powerful predictors of work stoppage and income declines in the first survey wave, while those related to food, exercise, and religion better tracked changes in income declines and employment over time. Google mobility measures are also strong predictors of changes in employment and the prevalence of specific types of income declines. In general, satellite data on vegetation, pollution, and nighttime lights are far less predictive. Google mobility and search data, and to a lesser extent vegetation and pollution data, can provide a meaningful signal of regional economic distress and recovery, particularly during the early phases of a major crisis such as COVID-19.

This paper is a product of the Development Economics, the Poverty and Equity Global Practice, and the Human Capital Project. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http:// www.worldbank.org/prwp. The authors may be contacted at dnewhouse@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
**How Well Did Real-Time Indicators Track Household Welfare Changes in Developing Countries during the COVID-19 Crisis?**<sup>**1**</sup>

**David Newhouse, Rachel Swindle, Shun Wang, Joshua D. Merfeld, Utz Pape, Kibrom Tafere, Michael Weber**

**JEL:** D31, D74, I31, O15

**Keywords** : COVID-19; Well-being; Work; Income; Google mobility; NO2 ; Google search

> 1 The team would like to thank the Korea Trust Fund for Economic and Peace-Building Transitions (KTF) for providing funding. The KTF, supported by the Ministry of Economy and Finance, Republic of Korea, is a global fund administered by the World Bank to finance critical development operations and analysis in situations of fragility, conflict, and violence.
# **1. Introduction**

COVID-19 triggered one of the most precipitous and dramatic shocks to households and regional economies since the end of World War II. Within weeks of the World Health Organization’s declaration of COVID-19 as a global pandemic on March 11, 2020, the virus had spread across most of the world.<sup>2</sup> As the crisis unfolded, there was an immediate need for fast, accurate data on the economic effects of the pandemic. The World Bank Group worked with national statistical offices to field phone surveys in dozens of member countries, with the goal of expeditiously measuring how the pandemic was impacting households. These surveys asked respondents about directional income changes, both for total and various sources of income. In addition, they asked about access to critical services and resources, food security, employment status, behavioral adjustments, and overall well-being. Insights from these survey data have proven to be invaluable for documenting the impact of the shocks on households in many developing countries and informing policy responses: they are enabling more precise targeting in assistance and recovery programs, whether in the form of direct cash transfers to residents or interventions to assist firms in the hardest-hit sectors. However, fielding phone surveys takes critical time, which is not always an available resource in the midst of a major crisis, and requires substantial resources to sustain over longer periods.

Real-time indicators – like satellite images, pollution data, and Google mobility data – are immediately available at low cost. Hence, it is not surprising to see that, for example, mobility data from Google has become popular for understanding community responses to the pandemic. When lockdowns were implemented, Google mobility data shows corresponding declines in activity in commercial areas and increases in time spent at residential spaces. These data are available in close to real time and can be used to document compliance with mobility restrictions. Similarly, near real-time data on nitrogen dioxide pollution showed less pollution in some areas in the aftermath of the shock (Masaki et al., 2020). The varied speed of the onset of the crisis and the progression of recovery in different countries, and regions within them, raises the natural question of how well these real-time indicators track the economic indicators collected in the phone surveys, and which real-time indicators are most informative.

> 2 Director-General’s opening remarks at the media briefing on COVID-19, 11 March 2020. <u>https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-atthe-media-briefing-on-COVID-19---11-march-2020.</u>

2
This study combines a large set of near real-time indicators with the World Bank’s highfrequency phone surveys to assess the ability of the indicators to predict changes in a variety of proxies for household economic well-being. In addition to Google Mobility data, Google search trends, night-time lights, vegetation and pollution data are used to develop predictive models of trends in a variety of outcomes measured in household surveys. First, the study examines the ability of these real-time indicators to predict variation in the severity of the initial shock, as measured by the prevalence of self-reported decreases in income and job losses since the start of the crisis, recorded in the first survey wave that each country fielded between April and August 2020. Second, the study examines the ability of the real-time-indicators to predict variation within regions – typically states or provinces – over time as the impact of the crisis evolved.

While the use of real-time data is relatively new in economic research, several studies provide insight into best practices for predictive modeling and how this data can contribute to poverty research. Many real-time measures, such as Google mobility, night lights from satellites, nitrogen dioxide (NO2 ), and Google search of certain keywords, have the potential to predict traditional well-being measures. Google mobility has been recently used to predict industrial production growth (Sampi & Jooste, 2020) and economic growth (Putra & Arini, 2020) during the pandemic. The Google mobility index shows how visits to places are changing over time in each geographic region. The index is measured for six categories of places, including workplaces, residential, grocery and pharmacy, transit stations, retail and recreation, as well as parks. The data are not representative, however, as they are only collected from smartphone users that have location services turned on.

Economists increasingly use night lights detected by satellites as a proxy for economic indicators (Chen & Nordhaus, 2011; Gibson et al., 2021; Henderson et al., 2012; Ishizawa et al., 2017; Mellander et al., 2015). In developing countries with low-quality economic and social accounting systems, or when satellite images are more frequently available than such data, the satellite-based nightlight data may provide valuable information on social and economic development. In addition to the widely used night lights, other types of remotely sensed data have recently been explored in economics. For example, Castellanos & Boersma (2012) find that NO2 is correlated with economic activities and thus can be used to measure the magnitude of social and economic activities (Keola & Hayakawa, 2021). The Normalized Difference 3
Vegetation Index (NDVI) explains about 4 percent of the variation in per capita consumption across Sri Lanka (Engstrom et al., 2021). Daytime satellite imagery has also been shown to predict 15 to 17 percent of changes in village-level asset indices extracted from Living Standards Measurement Surveys in Sub-Saharan Africa. However, that rises to 35 percent when predicting an index of changes in asset in ownership at the village level, and 50% when that index of changes is aggregated to the district level (Yeh et al, 2020). Khachiyan et al (2021) find that models trained on satellite imagery can explain between 32 and 46 percent of decadal changes in income and population in the United States.

Data from search engines have also been used to assess people’s real-time well-being without relying on survey questionnaires. _Google Trends_ and its local equivalents (such as Baidu Trends in China and Naver Trends in the Republic of Korea), based on search frequency, provide an index for search intensity (or relative popularity) by topic or term over time in a certain geographical area. A number of recent studies have used data from _Google Trends_ to explore the changes in psychological well-being during the COVID-19 pandemic _(_ Brodeur et al., 2021; Foa et al., 2020; Ma et al., 2021 _)_ . However, these studies focus on measuring the impact of lockdowns on search behavior and do not compare search data with large-scale phone survey data. In addition, data from search engines are also not nationally representative since they are only available for areas with access to the internet and collected from users of Google for online search, while data from satellites are affected by cloud cover, raising questions about their ability to track economic changes more broadly.

In this paper, we exploit a set of real-time indicators from different sources to predict changes in welfare in developing countries during the COVID-19 crisis. Six main findings emerge:

1. The selected real-time indicators predict a significant share of the cross-regional variation in the initial impacts of the crisis. Across about 300 regions in 20 countries, eight Google search terms explain a combined 36 percent of the variation in the share of households reporting total income declines since the start of the crisis between April and August 2020. Meanwhile, during the same period, 10 Google search indicators explain a combined 34 percent of the variation in the share of respondents reporting job loss since the start of the crisis.

2. Compared with initial impacts, selected real-time indicators explain less of the variation in the prevalence of income declines and employment between April 2020 and 4
February 2021. The R<sup>2</sup> values for these models are still substantial, however, and range from 22 to 26 percent for the incidence of income declines and are 15 percent for current employment.

3. When limiting the sample to urban regions only, real-time indicators are far more successful in explaining within-region variation in income declines and employment between April 2020 and February 2021. The R<sup>2</sup> values for these models range from 29 to 55 percent for the incidence of income declines and are 33 percent for current employment. Improved predictive performance in urban regions appears to be partly due to the greater predictive accuracy of Google search terms.

4. Google mobility measures are strongly associated with work stoppage and changes in current employment, as well as declines in farm, non-farm enterprise, and wage income, during both the initial phase and the subsequent period.

5. Google search terms related to food, money, jobs, and religion were also strong predictors of income declines and work stoppage during the initial stage of the crisis. Google searches related to food, assistance, religion, and in some cases exercise were strong predictors of changes in the prevalence of income declines and employment as the crisis evolved.

6. In general, Google search and mobility information are far more predictive of income and employment changes than satellite data from vegetation, pollution, and night-time lights. However, vegetation levels strongly predict a measure of food insecurity and the share of students that quit school, while pollution is a moderately strong predictor of work stoppage, current employment, and non-farm and wage changes. Night-time lights were not selected by LASSO in any of the predictive models.

Taken together, our findings suggest that these real-time indicators, particularly Google mobility and search terms, predict a respectable amount of the variation in changes in wellbeing. The signs – positive or negative correlation – are often intuitive, especially when explaining variation over time. Real-time indicators were better able to predict cross-sectional variation in initial crisis impacts, measured as retrospective changes, than intertemporal variation in outcomes over repeated waves of survey data. In addition, the real-time indicators were better suited for predicting income losses than gains. The importance of different indicators also varies widely by context, highlighting the benefits of training models that 5
combine information from several different sources. Such models may be a promising avenue for monitoring changes in a region’s economic circumstances, particularly in the midst of sudden, large shocks when more traditional survey data are not (yet) widely available.

The remainder of this paper is organized as follows: section 2 introduces the data and empirical methodology, section 3 describes trends in measures of well-being, section 4 presents the results of our analysis, and section 5 concludes this study.

# **2. Data and Empirical Methodology**

The analysis uses two types of data: (1) high-frequency phone surveys measuring household well-being during the COVID-19 crisis, and (2) real-time metrics of mobility, internet search activity, pollution, and night-time lights. We use the regionally aggregated high-frequency phone survey (HFPS) data as the dependent variable for the prediction based on real-time metrics.

# 2.1 Household Well-Being Measures from the HFPS

The HFPS data cover 59 developing countries.<sup>3</sup> The process of fielding the surveys for each country was done in conjunction with national statistics offices to ensure appropriate weighting and population representation. Enumerators contacted households by phone and followed up with individuals in subsequent waves. Surveys were either carried out by recontacting respondents from a previously fielded nationally representative survey or through random digit dialing (RDD). Lower-income countries primarily opted for the former, while middle-income countries with more widespread phone coverage were more likely to use RDD. In cases where previous survey respondents were re-contacted, weights were constructed to account for the selection bias due to only interviewing households with phones who responded to the survey. However, respondents tend to be heads and the weights did not

> 3 The HFPS includes 8 countries from East Asia and Pacific (Indonesia, Cambodia, the Lao People’s Democratic Republic, Myanmar, Mongolia, the Philippines, Solomon Islands, and Viet Nam), 8 from Europe and Central Asia (Armenia, Bulgaria, Croatia, Georgia, Poland, Romania, Tajikistan, and Uzbekistan), 14 from Latin America and the Caribbean (Argentina, Bolivia, Chile, Colombia, Costa Rica, Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Mexico, Paraguay, Peru, and Saint Lucia), 5 from the Middle East and North Africa (the Arab Republic of Egypt, Iraq, the West Bank and Gaza, Tunisia, and the Republic of Yemen), and 24 from Sub-Saharan Africa (Burkina Faso, the Central African Republic, Chad, the Democratic Republic of Congo, the Republic of Congo, Djibouti, Ethiopia, Gabon, Ghana, Kenya, Madagascar, Malawi, Mali, Mauritius, Mozambique, Nigeria, São Tomé and Príncipe, Senegal, Sierra Leone, Somalia, South Sudan, Uganda, Zambia, and Zimbabwe). Among them, three countries (Djibouti, Kenya, and Mauritius) do not have a subnational region code, and thus are omitted from the analysis.

6
correct for biases due to non-random selection of respondents within the household (Brubaker et al, 2021; Kugler et al., 2021). Therefore, while household level outcomes such as self-reported income declines are plausibly representative, individual level outcomes such as employment mainly apply to household heads. See World Bank (2023) for more information on the HFPS.

The outcome variables taken from the HFPS are listed in Table 1 and include income change, employment, food security, and prevention measures such as social distancing and quitting schools. Work stoppage is measured in the first wave of the phone survey in each country, and is constructed based on the questions “Was the respondent working at the time the survey was conducted?” and “Was the respondent working before the pandemic?”. In particular, work stoppage is set equal to 1 if a respondent reported that they were working prior to the pandemic but not currently working at the time of the survey, 0 if they were working prior to the pandemic and were currently working at the time of the survey, and missing if they were not working prior to the pandemic.

Income change is measured by asking respondents whether their household’s income has increased, decreased, or stayed the same since the start of the pandemic (first wave only) or since the last survey wave (all subsequent waves). When looking at the evolution of the crisis over time, we combined these two questions into a single measure, in order to include all waves. Income change is further asked separately for each income source including farm, nonfarm, wages, and pensions. The question for total income change is “Has your Total Household Income changed since the pandemic started?” There are 5 categories of answers: 1 = Increased, 2 = Stayed the same, 3 = Decreased, 4 = Not received, 9 = Do not know. We create a variable on income loss, where “Decreased” is recoded as 1, “Do not know” and “Not received” as missing, and “Stayed the same” or “Increased” as 0. We also create a variable on income gain, where “Increased” is recoded as 1, “Do not know” and “Not received” as missing, and “Stayed the same” or “Decreased” as 0. The creation of variables for different income sources, farm income, non-farm income and wage income follows the same rule. Not all income questions were asked in each survey, so the sample of available regions varies by indicator.

Food security is another key metric of household well-being, which is measured using a series of questions on household constraints accessing food. The analysis uses two questions: (1) “in 7
the last week did you or another adult in the household skip a meal due to lack of money or resources?”; and (2) “was there a time in the last week when you or another adult in the household went an entire day without food due to lack of money or other resources?”. The responses are binary, 1 for “Yes” and 0 for “No”.

We also consider measures related to social distancing and school attendance. For social distancing, the survey question is “Have you adopted social distancing/self-isolation?” We use two measures of educational disruption. When analyzing wave 1, we construct a measure of school dropout based on “Have children in the HH attended school (primary/secondary) before school closures?” and “Have children in the HH engaged in ANY learning or educational services since school closures?". Because the first of these two questions is generally not asked after the first wave, we only consider the latter of those questions when analyzing all waves of data.

We take the mean of each well-being measures at the regional (state/province) level by month to create a panel of regions.<sup>4</sup> Appendix Table 1 shows the number of regional observations by month in 2020. All variables have data beginning in April 2020. However, the panel data are highly unbalanced with most observations concentrated in June, July and August, implying that the HFPS surveys covered most countries in the three months.

The summary statistics of the regional-level HFPS well-being measures for wave 1 and all waves in 2020 are reported in panels A and B of Appendix Table 1.

# 2.2 Explanatory Variables

The main explanatory variables are a set of real-time indicators including night-time lights, air pollution, and EVI (a vegetation index) measured from satellites, Google mobility data, and the popularity of Google search of certain keywords. These are listed in Table 2. Summary statistics of real-time indicators for wave 1 and all waves in 2020 are reported in Appendix Table 2A and 2B, respectively.

The measures of night-time lights, air pollution, and EVI are all obtained from Google Earth Engine. The night-time lights data are taken from the VIIRS monthly composites produced

> 4 Each survey wave is assigned to a unique month.

8
by the Earth Observation Group at the Colorado School of Mines.<sup>5</sup> To measure pollution, we use Nitrogen Dioxide (NO2) concentrations measured by Sentinel 5-P satellites.<sup>6</sup> We expect that NO2 levels increase in response to human activity, namely burning fossil fuels and biomass, which may have fallen during the crisis. EVI was taken from the Landsat 8 Collection 1 Tier 1 composite.<sup>7</sup> The Google search index at subnational levels reflects the relative popularity of any given search term. The total search volume for each term is divided by total Google search volume for that week and subnational region. The search index for each term is rescaled such that the week-subnational region observation with the highest relative search score, for all weeks in that country, is assigned a value of 100 with all other weeks expressed relative to this peak. The baseline (weeks with a value of 100) itself is a mechanism for rescaling search term data such that values are comparable across regions and over time.<sup>8</sup> We aggregate data into monthly averages, taking the average of the weekly data points for each subnational district.

Google restricts the quantity of data that can be downloaded in one batch. Therefore, the datasets are extracted in multiple rounds to cover all subnational districts for all countries with HFPS data. This process requires indexing all downloads against a common administrative district which serves as a reference point. The reference district is included in all rounds of downloads such that the resulting search terms are comparable regardless of whether they are pulled in the same round. This approach is based on Abay et al. (2020).

The Google mobility index shows how visits to places are changing over time in each geographic region. Google mobility data is measured against a baseline established by Google

> 5 https://developers.google.com/earth-engine/datasets/catalog/NOAA_VIIRS_DNB_MONTHLY_V1_VCMSLCFG

> 6 “Sentinel-5P NRTI NO2 : Near Real-Time Nitrogen Dioxide”, Google Earth Engine. <u>https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S5P_NRTI_L3_NO2</u>

> 7 https://developers.google.com/earth-engine/datasets/catalog/LANDSAT_LC08_C01_T1_8DAY_EVI

a where G t t NNt at <u>,ww,tt,tt</u> represents the share of all search volume in a particular week w (in month m for region r) for that term, and ,tt,tt<sup>represents the subnatiuonal google search index for a term in month m and subnational region r.</sup> Max is the maximum, across regions and weeks, in country c. NN aa,ww,tt,tt 9
researchers and is consistent across countries. The baseline is for a “normal” day so, for example, Monday mobility data is measured as the deviation from the “normal” Monday baseline and similarly for other weekdays. The “normal” day reflects the median value from five weeks of pre-crisis data from January 3 through February 6, 2020. The data are aggregated to monthly values at the first subnational administrative division in each country. The index is measured for six places, including workplaces, residential, grocery and pharmacy, transit stations, retail and recreation, as well as parks.

The real-time indicators at the levels of subnational administrative division by month are merged with the aggregated HFPS panel data at the same administrative level, yielding the panel data used for the analysis.

# 2.3 Multiple Imputation

Because Google mobility data are missing for several regions, we employ multiple imputation techniques to fill them. Google does not disclose at least one mobility indicator in a total of 559 out of 1,096 regions, across the 6 indicators (Residential, Retail and Recreation, Workplace, Transit Stations, Groceries/Pharmacies, and Parks). Using multiple imputation techniques allows us to include these regions in the models, which generate more precise estimates of the relationship between outcomes and other non-mobility predictors. To fill in missing Google mobility indicators, we create 100 imputations using imputation by chained equations, based on the other mobility indicators, night-time lights, NO2, and vegetation index (EVI). The chained equations approach carries out imputation iteratively for each of the Google Mobility variables, each time substituting values drawn from the conditional distribution for missing values, until convergence is achieved. Because we carry out analysis on two different samples, one for initial impacts and one for the subsequent evolution of the crisis, we perform two imputation exercises: one only for data from the first wave, and one for data for all waves. In the first case, mobility values are imputed for between 65 and 135 of the 311 total regions, depending on the Google mobility indicator. When using the data for all waves, we demean the data using each region’s mean, to isolate the variation over time, and impute the demeaned variables. For this case, mobility values are imputed for between 315 and 540 of the 1,096 total region-waves.

10
A final issue is that of weights. The outcome variables are means of household indicators taken at the regional level, although these are demeaned when predicting the evolution of the shock. When constructing these regional averages, we use household weights. Because the number of observations varies by region, country, and survey weight, it is appropriate to correct for heteroscedasticity. We therefore weight each region-wave observation by the number of households in the original phone survey data used to construct the average for that region.<sup>9</sup> This gives greater weight to regions for which the outcomes are measured with greater precision. These weights are applied both in model selection and in model estimation.

2.4 LASSO Model Selection To construct predictive models, we use LASSO to select which variables among the real-time indicators best predict changes in income, work status, food insecurity, behavioral changes, and educational disruptions. When predicting initial impacts, the dependent variable is the regional average, and when predicting the subsequent evolution of the crisis the dependent variable and the predictors are demeaned using the regional mean over multiple waves. The full set of real-time indicators are included as candidate variables. We use a variant of the LASSO that selects the penalty parameter lambda to minimize the Bayesian Information Criteria, following Zhang et al. (2010). When selecting models to predict in the first wave, both the LASSO and the resulting post-lasso estimates are clustered on country, and when predicting changes over time, the LASSO and the post-lasso estimates are clustered on country-wave.<sup>10</sup> It is not straightforward to apply Lasso to multiply imputed data (Du et al., 2020) and different approaches have been proposed. For the purpose of maintaining simplicity and to avoid overfitting, we select the model using only data from one arbitrarily selected imputation, and report that R<sup>2</sup> . In addition, however, we report the mean R<sup>2</sup> from 100 models, one for each imputation, each selected independently using LASSO and then estimated using OLS. The latter is our preferred measure of R<sup>2</sup> .

> 9 We do not give greater weight to more populous regions, conditional on sample size.

> 10 The BIC is calculated based on the unpenalized variables, which is appropriate since we are using LASSO to select the model rather than for post-estimation. The calculation of the BIC takes into account clustering, as described in Stata.

11
After selecting variables by applying Lasso to a single imputation, we estimate the post-lasso OLS regressions across all 100 imputations, take the average of the point estimates, and estimate the variance using Rubin’s rule, as is standard when using multiple imputations.<sup>11</sup> To examine the predictions in the first wave, we estimate the following equation in the HFPS wave 1 data: wave 1 data: 0 1R , where indicates the subnational region. ii and R ii denote the lasso-selected set of HFPS cc WW = αα + αα RR + εε well-being and real-time indicators, respectively. ii is a random disturbance term, clustered on ii ii WW RR country. cc εε For the long-term analysis, we consider the following OLS model using the demeaned variables, 0 1R� , where and R� denote the set of demeaned HFPS well-being and real-time indicators iiww iiww ccww WW<sup>�</sup> = ββ + ββ RR + εε respectively for wave w in region i. iiww iiww is a random disturbance term, clustered on countryWW<sup>�</sup> RR wave. Using the demeaned variables is equivalent, at least in terms of the coefficient estimates, ccww εε as a model in levels that includes regional fixed effects in the estimation. The resulting R<sup>2</sup> measures provide a measure of how well the selected real-time indicators explain the variation in outcomes, within regions across time.

# **3. Trend Analysis**

# 3.1 Trend of HFPS Well-Being Measures

Figure 1 presents the trend of five well-being measures in 2020, averaging over countries in the sample. Not all measures are available for each country in each wave, so the composition of countries varies by measure and month. In addition, the first wave of the survey asks about changes since the pandemic began, while subsequent months ask about changes since the previous survey wave for that country. As the surveys start in April, we cannot report changes in the first quarter. The figure shows that the share of respondents reporting income loss is

> 11 We use the mi estimate command in Stata for this.

12
<!-- Start of picture text --> <<br>——®— Total inc. Farm inc. ——4&— Non-farm inc.<br>Wage inc. " Currently working?<br>yn<br><b) x<br>><br>D .<br>£—x<br>®<br>co —*<br>o.<br>cx<br>2<br>:<br>3° ;<br>Qa r <<br>fe)<br>=<br>Ox<br>N<br>Apr May Jun Jul Aug Sep Oct Nov Dec<br>Month<br>The income variables are based on whether the household reported an income loss.<br><!-- End of picture text -->
<!-- Start of picture text --> nw —*— NO2 Nightlights “te EVI |<br>~<br>a 4 ve ye 4<br>ae] *<br>® \<br>3 A<br>c<br>Bo<br>a fe ‘Ne<br>®<br>=} an<br>7 A- 4<br>A.<br>A<br>TM<br>Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov’ Dec<br>Month<br><!-- End of picture text -->

Values are standardized by subtracting the mean and dividing by the standard deviation.
<!-- Start of picture text --> Fe<br>° \<br>4<br>oO<br>| |<br>|<br>o ‘<br>Y . | ——*®— Grocery/pharmacy<br>Parks<br>° i ——+— Residential<br>y ¥ Retail and rec.<br>. Transit stations<br>} ——+—— Workplaces<br>'<br>Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov’ Dec<br>Month<br><!-- End of picture text -->

Values are deviations from a normal day and are calculated by Google, using Jan. 3rd to Feb. 6th as the baseline.
<!-- Start of picture text --> —7———————SS————ee—~ ® ——*— Bread<br>/\ —*— Milk<br>/ \ Groceries<br>/ \ =~ Wheat<br>¢ \<br>© / / a ~ NN yo<br>i 4<br>°<br>Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec<br>Month<br><!-- End of picture text --> <!-- Start of picture text --> a; ——*—a aFood bank Free food Cheap food<br>——*— Free meal - Food supply ———— Assistance<br>P<br>°<br>N44. rm \ a<br>-=4--—: \NE \ , 2>—ara “*<br>° \ eH<br>|<br>Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec<br>Month<br><!-- End of picture text -->
<!-- Start of picture text -->
|~*~+ MoneySalary<br>st a |<br>\<br>\ _—_*——*<br>N —\<br>®<br>°o Ne ya — a i<br>ee<br>Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec<br>Month<br><!-- End of picture text -->

<!-- Start of picture text --> ——————T—e—F[C—[—>——————<br>-_ ——*— SchoolChildcare<br>- \ Student<br>< / \ A = Online school<br>, \ —<br>° |<br>%<br>Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec<br>Month<br><!-- End of picture text -->

<!-- Start of picture text --> + aE——*— JobsEmploymentEE EE ~*~ JobUnemployment posting ——*——~ Temp.Linkedin job ]<br>N S tT<br>° _ —— = > —~ —s |<br>\—_— = ; ~<br>TM~<br><<br>Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec<br>Month<br><!-- End of picture text -->

<!-- Start of picture text --> EE ES EE LE CEE EE EE<br>ry ~~ —_—-— PayerQuran |<br>/A 1 Bible<br>/ \ \ > A,<br>\<br>ada —~<—<br>o<br>Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec<br>Month<br><!-- End of picture text -->
Table 3 reports the cross-sectional regression results of predicting income loss and work stoppage by real-time indicators using the first wave of data collected between April and August 2020.<sup>12</sup> Column 1 shows the results for the share of households reporting total income declines. Our preferred measure of overall model fit is the mean R<sup>2</sup> over the full set of one hundred imputations, which is 0.37. More frequent Google searches for “money” and “jobs”, and less frequent searches for “free food” are statistically significant correlates of the prevalence of total income declines. Columns 2-4 show the results for different types of income. The predictive power for a loss of farm income, nonfarm income, and wage income is significantly less, at 0.09, 0.13, and 0.24, respectively. The predictive power of real-time indicators for job loss is slightly lower than total income but substantially higher than those of the different types of income (average R<sup>2</sup> = 0.34).

The signs of the coefficients when analyzing the initial wave are not always intuitive. For example, searches for “bread” are negatively correlated with total income declines, searches for “Food Bank” are negatively correlated with the prevalence of wage income declines, and searches for “Free Food” are negatively correlated with the prevalence of work stoppage. This likely reflects correlations between Google search behavior and initial conditions of the regions, such as their overall level of development. For example, residents in poorer areas may be more likely to search for food banks or free food in response to an adverse shock, even if those shocks were more prevalent in more affluent areas. Similarly, searches for “jobs” and “unemployment” are negatively correlated with the prevalence of total income declines in the first wave, suggesting that areas hit less hard by the crisis, in terms of the prevalence of income declines, may have been more likely to search for unemployment insurance or new jobs. More generally, the strong correlations in the analysis of the first wave partly reflect the role of precrisis regional characteristics that are correlated with both the real-time indicators and the intensity of the shock.

The correlations with mobility data are more intuitive, as workplace mobility was negatively correlated with the prevalence of wage income declines. This reflects the fact that the mobility data is measured as a change from a pre-crisis baseline. Google searches for “job postings” are positively correlated with wage income declines, as one would expect. Google searches for

> 12 Each survey wave is associated with a month, and the real-time indicators for that month were used.

18
the “Quran” were also positively correlated with work stoppage, suggesting that increased religiosity may have played a role as a coping mechanism for those that stopped work in the early stages of the crisis.

# _4.2 Predicting Variation over Time during the Pandemic_

Table 4 reports the results of demeaned OLS regressions to predict income loss and working status by real-time indicators using all waves of HFPS data. Demeaning the data by region effectively controls for time-invariant characteristics of the region. The second-to-bottom row of column 1 shows that the predictive power for total income loss since the previous survey is 0.22. The explained variation by real-time indicators for the three subcategories of income loss is slightly larger than that for the total income loss, as shown in columns 2-4. The R<sup>2</sup> for farm income loss, nonfarm income loss, and wage income loss is 0.25, and 0.26, and 0.23 respectively. The last column presents the predicting power for whether the respondent was working when the HFPS survey was administered. The mean R<sup>2</sup> is 0.15. The results in Table 4 imply that the prevalence of income loss and working status during the survey period is meaningfully associated with real-time indicators during the pandemic. However, this association is weaker than the cross-sectional variation using wave 1, because it does not include any of the correlation between real-time indicators and time-invariant characteristics of the region such as the region’s level of development prior to the crisis.

The signs of the statistically significant variables generally make intuitive sense when predicting variation over time. Increased Google searches for “Milk” and “Bible” are a reaction to negative shocks and are therefore positively correlated with income loss and negatively correlated with current work. Google searches for “Gym” are positively associated with total income declines, suggesting that exercise may be a source of stress relief when income declines. Searches for “Jobs” are positively correlated with farm income declines and wage income declines. Google searches for “Free meal” are positively correlated with the prevalence of income declines and “Cheap food” is negatively correlated with employment. Searches for “online school” are negatively associated with the prevalence of non-farm and wage income declines, suggesting that households in harder hit areas are less able to invest in online schooling.

Since an individual’s income might rise and fall along with changes in infection risk as well as stringency measures over time during the pandemic, we also investigate how well real-time 19
indicators can predict income gains during the pandemic, in addition to income loss as shown in Table 4.<sup>13</sup> The three columns in Table 5 show the predictive power for farm income gain, nonfarm income gain, and wage income gain respectively. The predictive power for non-farm and farm income gain is highest (R<sup>2</sup> = 0.17 and 0.16), followed by wage income gain (R<sup>2</sup> = 0.12), and total income gain (R<sup>2</sup> = 0.06). The signs of the statistically significant correlations are generally opposite to those for income declines, as one would expect. For example, searches for the “Quran” and “Bible” are negatively correlated with gains in total and nonfarm incomes, while searches for “gym” are negatively correlated with non-farm income. Increases in pollution are also positively associated with wage income gains.

As shown in Table 6, predictive models for adults skipping meals as the dependent variable have similar explanatory power as those for income gain. Column 1 shows that a higher level of NO2 is associated with a smaller incidence of skipping meals, while additional searches for “bible” are positively associated with skipping meals. The average explained variation over the 100 imputations is 0.16. When evaluating the model using data from another question on food security (“adults going a whole day without food”), the explanatory power is slightly lower (R<sup>2</sup> = 0.13). In that model, higher vegetation is associated a greater tendency to skip meals for a full day, as are Google searches for “Milk”, which is consistent with the latter as an indicator of economic distress.

The real-time indicators can also predict people’s social distancing and schooling behaviors during the pandemic, as reported in Table 7. The predictive power for social distance is low, however, as the R<sup>2</sup> is only 0.13. The predicting power is slightly higher for quitting school, with four variables explaining 16 percent of the variation. However, none of the predictors is statistically significant in either model.

Tables 3a through 3d in the appendix reports results on models predicting the prevalence of reported income declines and current employment separately for subsamples of urban and rural regions. These results are therefore analogous to those reported in Tables 3 and 4, using models selected and estimated using only data from urban and rural regions. Urban regions are defined as those for which half or more of the weighted survey respondents in the region

> 13 There are three possible values for the income change variables: increase, decrease, and no change. As such, “increase” is not the complement of “decrease”.

20
are classified as an urban resident, while the rest are considered rural. In wave 1, 66 out of 146 (45 percent) of the regions in the sample are considered urban according to this definition. Across all waves, 223 out of 599 (37 percent) are.<sup>14</sup> The LASSO model selection and estimation in each case are carried out only for the relevant subsample of regions, drawing on multiple imputations conducted in the full sample.

Tables 3a-3d generally show much stronger predictive performance of the model in urban regions. In wave 1, the model explains 48 percent of the variation in reported income decreases on average across the imputations in urban regions, as opposed to only 14 percent in rural regions. However, predictions in rural areas are more accurate for the reported declines in farm income in wave 1, explaining 35 percent of the decline in variation, with google search data for assistance, school, and student being statistically significant predictors. When looking at variation within urban regions over time, the model explains 55 percent of the variation in reported declines in total income, with changes in google searches for employment and online school having significant predictive power. The models for predicting variation over time within rural regions are much less accurate, with R2 values ranging from 0.1 for total income to 0.25 for wage income loss. These results suggest that google search data in particular is better suited to predicting changes in welfare in urban areas than rural areas.

# _4.3 Decomposing the Explanatory Power of Predictors_

Turning back to the models estimated on the full sample of areas, we conduct Shapley R<sup>2</sup> decompositions to better understand the role of different types of variables in explaining variation in the model. Table 8 reports the results, which show the share of the R<sup>2</sup> that is accounted for by ten different sets of variables. The sets are the Google mobility data, Google searches related to food (“Milk”, “Bread”, “Cheap food”), Money (“Money”), Jobs (“Jobs”, “Salary”, “Unemployment”, “Vacancy”), assistance (“Assistance”, “Free Food”, “Free Meal”, and “Food Bank”), exercise (“Exercise” and “Gym”), Religion (“Bible”, “Quran”, “Prayer”), and School (“Student”, “Online School”, “Online Education”). The final two sets of variables are the EVI (vegetation), and No2 (a measure of pollution), taken from satellites.

> 14 These figures for both wave 1 and the full sample are obtained from each multiple imputation replication.

21
The results show that mobility variables, when selected by LASSO, are generally powerful predictors of income and employment changes. When examining the initial impact, they explain between 27 percent (for farm income declines) and 40 percent (for work stoppage) of the explained variation. For the subsequent evolution, they explain between 34 percent (Nonfarm income decrease) and 46 percent (current employment) of the explained variation.

The importance of different Google search terms varies considerably by indicator. When predicting initial impacts in the first wave, searches related to food, money, and jobs account for 80 percent of the explained variation. When examining the same variable (the prevalence of total income declines) over time, important variables include food, exercise, and religion, which combined account for over 90 percent of the explained variation. When modeling work stoppage in the initial wave, search related to food, money, and religion contribute 36 percent of the variation. When modeling variation over time in current employment, searches related to food and religion contribute 27 percent of the variation, while jobs and stopping school contribute 5 percent each.

In general, Google mobility and search terms explain a much larger share of the variation than the satellite-based measures. Night-time lights, although included in the candidate set of variables, was not selected by lasso for any of the 18 models listed in Table 8. There are a couple of cases however where EVI and NO2 contribute significant variation, particularly EVI in the case of stopping school and not eating all day. NO2 is a significant determinant in particular of non-farm enterprise income gains, which is suggestive of a particularly strong association between household enterprises and mobility. Combined, the Google mobility measures and NO2 explain 51 percent of the variation in the prevalence of non-farm income losses and 59 percent of the prevalence of non-farm income gains.

# **5. Conclusion and Discussion**

This paper assesses the ability of real-time indicators to predict household survey-based wellbeing measures. Reference outcomes are provided by high-frequency phone surveys administered from the second quarter of 2020 to February 2021. The real-time data are publicly available and derived from the internet (Google mobility and Google search) or satellites (night lights, air pollution, and EVI).

22
The survey data confirm that the pandemic had a profound impact on people’s well-being, particularly in the beginning of the pandemic in early 2020. Subsequent recovery is observed for many well-being indicators from mid-2020 onwards. While no single variable performs well in predicting changes in well-being, various combinations of indicators from Google search terms, mobility, and satellite data have considerable explanatory power. These predictive models explain the largest share of the variation when measuring the prevalence of income declines and work stoppage since the onset of the crisis in the initial wave of the survey. This is partly due to the large variation across different regions and countries in economic disruption caused by the initial impact of COVID-19. However, the real-time indicators are also correlated with predetermined regional characteristics, like the overall level of development, that were themselves correlated with the intensity of the shock. When predicting variation within regions across time, however, real-time indicators maintain substantial explanatory power. For example, the indicators can explain about a quarter of the variation in the prevalence of income declines across regions, and about 15 percent of the variation in current employment, income gain, food security, and the share of households with children that stopped school. The real-time indicators considered here are significantly stronger predictors of the prevalence of reported income declines across urban regions, where the indicators explain 55 percent of the prevalence of income declines across regions. The models suggest that Google search terms are more powerful at distinguishing differences in the prevalence of reported income declines across urban regions than rural regions. This is consistent with the fact that most Google search queries are originating from urban areas, due to greater connectivity.

Of the real-time indicators, Google mobility data and search data for food, jobs, assistance, exercise, and religion were generally most predictive, though relative importance varies from indicator to indicator. Google search variables accounted for all of the explained variation of the prevalence of total income decrease, both in the first wave and over time, and at least 40 percent of the variation of all indicators related to income and employment. Measures of vegetation (EVI) and pollution (NO2) were also predictive in some cases. However, most models contain a variety of different indicators, highlighting the benefits of training a model that can combine information from disparate sources.

23
Further research could expand on this work in several ways. First, indicators extracted from pre-crisis satellite imagery could be added to the model, to see if they improve predictive power. Second, google search terms are likely affected by seasonal patterns, and accounting for these could also improve predictive power. Third and more broadly, it would be worthwhile exploring additional real-time indicators, such as satellite imagery that measures the pace and characteristics of new building construction in urban areas, and agricultural outcomes in rural areas. Call-detail records from mobile phone operators may also be very useful for this type of nowcasting. Fourth, further work can better understand which indicators and methods perform better in different contexts, as well as how model accuracy depends on the use of different training data to estimate the models. Non-parametric machine learning methods such as extreme gradient boosting may also predict better, particularly with large datasets. Finally, it would also be useful to expand the scope of this type of exercise beyond the specific COVID-19 crisis.

More research is needed to better understand the potential biases associated with different real-time indicators in different contexts. For example, Google search terms and mobility measures may not accurately reflect changes in areas where internet connectivity is low and may be more accurate in reflecting sharp downturns during a crisis than gradual progress. Google mobility measures may also suffer from selection bias since they only capture movement for a subset of mobile phone owners. Because of these concerns, we believe that these types of methods require further testing, evaluation, validation, and verification before being implemented to generate experimental official statistics. After better understanding when, where, and why nowcasts work well, these types of methods have the potential to produce extremely useful estimates, particularly in informing policy responses to unanticipated shocks. The results are therefore sufficiently promising to warrant prioritizing further investment in piloting and testing different predictors and methods to generate nowcasts for these and other related socioeconomic outcomes.

24