metadata: authors: 0: author: Seitz,William Hutchins
1: author: Kudo,Yuya
2: author: Azevedo,Joao Pedro Wagner De
count: Tajikistan
docna: 0: docna: Blackout or Blanked Out ? Monitoring the
            Quality of Electricity Service in Developing Countries
docty: Policy Research Working Paper
abstracts: cdata!: Access to reliable electricity is a
            Sustainable Development Goal, and key for both economic
            growth and individual wellbeing. Yet, in the absence of
            sophisticated monitoring systems, policy makers in
            developing countries commonly rely on surveys to measure
            electricity reliability and prioritize investments. The
            accuracy of such survey-based methods is unclear. This study
            built a low-cost national electricity outage monitoring
            network, using off-the-shelf components in Tajikistan
            &#x2013; a country with severe electricity service
            constraints. The system was introduced alongside a monthly
            household survey called Listening to Tajikistan, which
            allowed benchmarking the survey summary statistics against
            unbiased measures. The results show that although the two
            measures were well correlated, the survey data suffered from
            significant and systematic bias. Survey respondents (i)
            systematically underreported the incidence and severity of
            electricity outages on average, but (ii) systematically
            overreported the incidence of outages during a period of
            abnormally widespread service disruption of long duration.
            These findings suggest that bias in survey-based measures is
            sensitive to the salience of outages to the respondent, and
            that, where feasible, automated electricity monitoring can
            provide more accurate quality measurement. For survey
            settings, the results also suggest that estimates are more
            accurate over short (daily) reference periods.
display_title: Blackout or Blanked Out ? Monitoring
            the Quality of Electricity Service in Developing Countries
listing_relative_url: /research/2023/04/34049471/blackout-or-blanked-out-monitoring-quality-electricity-service-developing-countries
topicv3: Electricity Access m1503452 3922
publishtoextweb_dt: 2023-04-25T00:00:00Z
dois: 10.1596/1813-9450-10423
author_keys: 0: author_key: 1298530
1: author_key: 1675174
2: author_key: 675142
model_extractions: Policy Research Working Paper 10423

# Blackout or Blanked Out?

Monitoring the Quality of Electricity Service in Developing Countries _William Seitz Yuya Kudo Joao Pedro Azevedo_ Poverty and Equity Global Practice April 2023
Policy Research Working Paper 10423

# **Abstract**

Access to reliable electricity is a Sustainable Development Goal, and key for both economic growth and individual wellbeing. Yet, in the absence of sophisticated monitoring systems, policy makers in developing countries commonly rely on surveys to measure electricity reliability and prioritize investments. The accuracy of such survey-based methods is unclear. This study built a low-cost national electricity outage monitoring network, using off-the-shelf components in Tajikistan—a country with severe electricity service constraints. The system was introduced alongside a monthly household survey called Listening to Tajikistan, which allowed benchmarking the survey summary statistics against unbiased measures. The results show that although the two measures were well correlated, the survey data suffered from significant and systematic bias. Survey respondents (i) systematically underreported the incidence and severity of electricity outages on average, but (ii) systematically overreported the incidence of outages during a period of abnormally widespread service disruption of long duration. These findings suggest that bias in survey-based measures is sensitive to the salience of outages to the respondent, and that, where feasible, automated electricity monitoring can provide more accurate quality measurement. For survey settings, the results also suggest that estimates are more accurate over short (daily) reference periods.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at wseitz@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
**Blackout or Blanked Out?** Monitoring the Quality of Electricity Service in Developing Countries

William Seitz<sup>1</sup> Yuya Kudo<sup>2</sup> Joao Pedro Azevedo<sup>3</sup>

**Keywords:** electricity outages, survey methods, Listening to Tajikistan, survey bias, automated monitoring

**JEL:** C42, L94, Q41

> 1 The World Bank. Email: wseitz@worldbank.org 2 Institute of Developing Economies (IDE-JETRO). Email: Yuya_Kudo@ide.go.jp 3 The World Bank. Email: jazevedo@worldbank.org
# I – Introduction

## I.I – Monitoring Electricity Reliability

Sustainable Development Goal 7 aims to ensure universal access to affordable, reliable, and modern energy services by 2030. Electricity outages have serious negative consequences on wellbeing, productivity, and livelihoods. Best-practice monitoring systems in upper-income countries rely on sophisticated monitoring equipment that relay detailed information on the provision of electricity service to users. However, such systems are absent in most developing countries, and even where the infrastructure is present, the data are only seldom reliable and publicly available.

To address data gaps, outages, and the constraints service disruptions impose on users, are often monitored using surveys. Data collection efforts as diverse as the World Bank Enterprise Surveys, the U.S. Consumer Expenditure Survey, Living Standards Measurement Surveys, and a wide range of theme-specific research surveys all collect data on electricity outages for the purposes of understanding the effects of electricity outages on economic and social outcomes. In most cases, these surveys collect information by means of traditional recall methods.

However, survey data that identify electricity outages are subject to imprecision arising from several sources and establishing the effects of outages is often complicated by shortcomings in measurement. Some of the better-known challenges relate to recency, recall, desirability, and rounding biases. The imprecision that arises from these errors can have practical consequences for policy in several ways. For instance, outage monitoring is often used as a measure of accountability, and imprecision or bias in reporting could lead to inefficient management of resources. Likewise, systematic reporting bias between consumer subgroups could generate inefficiencies in the allocation of scarce resources. Biases in consumer perceptions could also lead to invalid estimates of the impact or potential outcomes of policy reform.

There is empirical evidence in the consumer behavior literature that systematic reporting differences may be a concern when using surveys for accountability purposes. Levanon et al. (2015), for instance find that across 12 countries, consumers with high social status tended to register more service failures and to complain more frequently than customers of lower social status. In the literature on power sector reform, many analyses recommend taking political economy considerations into account (Sulistiyanto and Xun 2004; Victor and Heller 2007; Erdogdu 2013).

This study uses data from unbiased electricity outage monitoring devices that collected detailed information alongside survey-based measures. Comparing the automated measures to survey-based measures demonstrates the presence and extent of bias in survey-based reporting of electricity outages in a developing country setting.

The results suggest that bias in reporting electricity outages in Tajikistan primarily takes the form of underreporting outages of short duration. Respondents systematically report fewer instances of electricity outages, and when they do report an outage as having occurred, on average report that it lasted less long than was truly the case. However, the results also suggest that the direction of bias changes during periods of widespread outages of significant duration. During such episodes, 2
respondents are more likely to overstate the number of outages, though during widespread outage events, survey-based estimates of outage duration were more accurate than during more typical periods.

The results contribute to a growing literature investigating the sensitivity of survey results to design features and other potential sources of bias. For instance, Beegle et al. (2010) find that reference periods and mode (i.e., recall vs. diary-based data collection) can materially impact the results of consumption and welfare surveys, Bardasi et al. (2010) find similar relationships for labor force surveys, and Heath et al (2021) show large reference period and interview frequency effects in labor force surveys. In all cases, these studies establish the extent of qualitatively large and economically significant differences due entirely to survey design.

# I.II – Electricity Provision in Tajikistan

The Soviet Central Asian Power System (CAPS) was built in the 1970s and included what is now the independent country of Tajikistan. The system provided nearly universal electricity in the region, but production and distribution investments were optimized without regard to country borders. With independence, Tajikistan and other countries in Central Asia entered complex bilateral agreements for fuels, electricity, and water. Disagreements between the member countries led to a partial disintegration of the CAPS, and ultimately, Tajikistan’s exit from most cross-border arrangements including for electricity and gas. Official policy aimed for energy self-sufficiency with heavy reliance on hydroelectric power. However, low administered prices and a lack of alternative sources led to widespread use of electricity for heating in Tajikistan, leading to serious recurring electricity shortages in the winter months.

**Figure 1: Satisfaction with Electricity Provision in Europe and Central Asia**

<!-- Start of picture text --> 100%<br>90%<br>80%<br>70%<br>60%<br>50%<br>40%<br>30%<br>20%<br>10%<br>0%<br>Total Rural Urban<br><!-- End of picture text --> _Source: Life in Transition Survey, 2016_ When last measured, traditional survey measures indicated that Tajikistan was among the worst performers in the ECA region in terms of the reliability of electricity supply to households (figure 1). Tajikistan also suffered from a uniquely large divergence in satisfaction between urban and rural areas 3
(83 percent vs. 34 percent). Although during the period covered in this study almost all households were connected to the electricity grid, as much as 70 percent of the population suffered from extensive shortages of electricity, especially during winter months. Electricity shortages in 2013 were estimated at about 14 percent of annual electricity demand and imposed economic losses estimated at US$200 million per year, or 3 percent of GDP.

The next section introduces the survey and automated data collection methods employed in this study. Section III describes the analytical approaches used, focusing on comparisons between the surveybased reporting and the unbiased automated monitoring system. Section IV describes and the results and provides interpretation of their significance for monitoring initiatives and policy making. Section V concludes.

# II – Data

Data from two sources are used in this study: data gathered from a monthly phone-based panel survey of households, and from an automated system of Smart Survey Boxes (SSBs). The survey data begin in May 2015. However, SSB monitoring began in November 2016. Thus, for the direct comparisons that follow, only survey data covering comparable periods is used.

## II.I – Listening to Tajikistan

The analyses in the following section use novel data collected by the authors in the L2T survey. Fieldwork proceeded in two stages. The first “baseline” stage included an extensive face-to-face interview with household members. Households were selected using a stratified two-stage clustered sample design based on the 2010 national census of Tajikistan. A total of 150 clusters were selected, with a probability of selection proportional to size. From these clusters, 3,000 households were selected to participate in the baseline survey. During the baseline interview, detailed information was gathered on household composition, asset ownership, and migration. A full household roster was collected (including age, gender, etc.), as well as a comprehensive consumption module that allows analysis in terms of monetary welfare groups in the country.

The baseline survey was completed in March 2015. Following completion of the face-to-face fieldwork, interviewers began regularly calling a randomly selected panel of between 800-1,200 households over the phone to conduct short interviews, following a set monthly schedule agreed to by the participating household. The questionnaire for these phone interviews was designed to monitor trends in subjective wellbeing, alongside measures of income, employment, service disruptions, and related indicators. Phone-based interviews began in May 2015, and survey months 10-25 are used in the analysis that follows, covering the period from October 2016 to January 2018. After excluding missing values and inconsistent entries, a total of 30,350 unique observations are available for analysis.

4
<!-- Start of picture text --> daily average, SSB (all day)<br>44<br>i,<br>Maarjary(<br>o 80 80 120 00 20 240 = ©) 400 440 400 520 0<br><!-- End of picture text --> <!-- Start of picture text --> daily average, SSB (b/w 6pm and 10pm)<br>ws |<br>i-<br>tt<br>oo 8 WH tH 0 200 260 = m0 400 440 aso 20 "0<br><!-- End of picture text -->
The boxes sent four message types; two related to electricity events (defined as changing status to “the electricity is now on” and “the electricity is now off”), and two scheduled messages. The devices sent a message each time the device detected an electricity outage, and another when the electricity turned back on. The devices were also programmed to send messages twice daily at pre-specified times, confirming the presence/absence of electricity at that time. The messages were received by a tablet device located in country. This device automatically saved all messages and uploaded them to a cloud server for processing. The resulting data provided the timestamp of each message and its content, which were then processed to generate a data file detailing the state of each monitoring box minuteby-minute (extrapolating from the content of SMS messages received).

# II.III – Comparing Summary Statistics

**Figure 3: Survey reported hours of electricity outages vs. automated reported outages**

<!-- Start of picture text --> Unconditional Conditional on an outage<br>18 18<br>16 16<br>14 14<br>12 12<br>10 10<br>8 8<br>6 6<br>4 4<br>2 2<br>0 0<br>Overall SSB Overall Survey Conditional SSB Conditional Survey<br>2016w47 2016w51 2017w3 2017w7 2017w11 2017w15 2017w19 2017w23 2017w27 2017w31 2017w35 2017w39 2017w43 2017w47 2017w51 2018w3 2016w47 2016w51 2017w3 2017w7 2017w11 2017w15 2017w19 2017w23 2017w27 2017w31 2017w35 2017w39 2017w43 2017w47 2017w51 2018w3<br><!-- End of picture text --> Direct comparison of the survey vs. the box summary statistics reveals significant differences on average, due both to overreporting outages and underreporting at different times. Figure (4) reports the differences in the average hours of electricity outrages and the number of days (over the past 10) on which outages occurred. Respondents reported outages of slightly shorter duration than box-based estimates, but more noticeably underreported the number of days on which an outage occurred.

6
<!-- Start of picture text --> PA AverageNouts (afl dary) peat10 dayne<br>bi<br>i ”<br>a<br>oe :<br>~ - ar] a es Li] Pd “ ra<br>hours<br><!-- End of picture text --> <!-- Start of picture text --> “ outagedays, past 10 days<br>2<br>Is<br>°<br>1 4 4 4 2 ° 2 ‘ 4 8 0<br>Gays<br><!-- End of picture text -->
time, suggesting substantial within-PSU variation (see appendix A). Thus, the comparisons between box and household survey data are most accurately interpretable with summary statistics at a higher level of aggregation.

**Table 2: Descriptive Statistics from Survey and SSBs**

||Al|l day||Bet|ween 1|8:00 an|d 22:00|
|---|---|---|---|---|---|---|---|
|Mean|Med.|SD|Obs.|Mean|Med.|SD|Obs.|
|**(A) Unconditional**<br>**(A1) SSB Obs level:** _psu-by-month, averaged across b_<br> <br>|_oxes with_<br>|_in each_<br>|_psu_<br>|||||
|One if outage<br>0.88|1.00|0.31|1776|0.77|1.00|0.41|1776|
|Total hours/day<br>2.19|0.52|3.78|1776|0.32|0.04|0.62|1776|
|<br>**(A2) SSB Obs level:** _psu-by-date, averaged across bo_|_xes within_|_each ps_|_u_|||||
|One if any outage<br>0.28|0.00|0.45|96644|0.14|0.00|0.35|96644|
|Total hours/day<br>2.29|0.00|6.20|96644|0.34|0.00|1.05|96644|
|**(A3) Household Data Obs level:** _household-by-m_|_onthly-ro_|_und_||||||
|One if outage||||||||
|over the past 10 days<br>0.28|0.00|0.45|12954|-|-|-|-|
|yesterday<br>0.18|0.00|0.38|12928|-|-|-|-|
|day before yesterday<br>0.19|0.00|0.39|12940|-|-|-|-|
|Length of outage||||||||
|over the past 10 days (days)<br>0.66|0.00|1.44|12954|-|-|-|-|
|over the past 10 days (hours/day)<br>2.00|0.00|5.03|12953|-|-|-|-|
|yesterday (hours/day)<br>1.85|0.00|4.74|12928|0.07|0.00|0.45|12941|
|daybeforeyesterday (hours/day)<br>1.94|0.00|4.89|12939|0.08|0.00|0.48|12937|
|<br>**(B) Conditional on having any outage (B1 an**<br>**(B1) SSB Obs level:** _psu-by-month, averaged across b_|**d B2) o**<br>_oxes with_|**r any s**<br>_in each_|**elf-reporte**<br>_psu_|**d outage (**|**B3)**|||
|Total hours/day<br>2.48|0.68|3.93|1572|0.42|0.09|0.68|1383|
|<br>**(B2) SSB Obs level:** _psu-by-date, averaged across bo_|_xes within_|_each ps_|_u_|||||
|Total hours/day<br>7.97|3.32|9.41|27848|2.34|2.31|1.69|14117|
|<br> <br>**(B3) Household Data Obs level:** _household-by-m_<br>|<br>_onthly-ro_|<br>_und_||||||
|Length of outage||||||||
|over the past 10 days (days)<br>2.31|2.00|1.86|3726|-|-|-|-|
|over the past 10 days (hours/day)<br>6.96|4.00|7.32|3724|-|-|-|-|
|yesterday (hours/day)<br>10.04|12.00|6.31|2382|2.09|2.00|1.25|462|
|daybeforeyesterday (hours/day)<br>10.12|12.00|6.48|2480|2.29|2.00|1.31|455|
|All observations are in or after 2016, Nov 1.||||||||

These comparisons are drawn out in alternative formulation in table 3, which reports t-tests comparing the average difference between the two data collection methods from each source for the same location and period. Although the data are structured differently (SSB data provide continuous coverage), SSB data can be harmonized to cover the given reference period (for instance, “yesterday” or “the day before yesterday” calculated using the interview date) and compared. Survey-based methods report about 10 percent fewer outages “yesterday” on average, and a similar rate with respect to the “day before yesterday.” When asked about the preceding ten-day period, respondents reported less, reporting outages at about half the rate that boxes recorded them.

8
Survey-reported outage duration was also substantially lower than the box-based duration on average. For the previous day, estimates derived from survey responses were about 18 minutes less than automated monitoring of outage duration. Over prime hours during the previous day and the day before, respondents underreported by a smaller number of minutes. When asked over the previous ten days, estimates from survey data were lower by slightly more than two days, and average duration was underreported by an average of about 16 minutes. These differences are statistically significant at customary levels and suggest that data gathered using surveys in this context results in systematic underreporting of outages. Differences were relatively smaller over shorter and more concrete reference periods (past day/between 6 and 10PM), though systematic bias (in the form of underreporting) remained.

Another means of drawing these comparisons is provided in table 4, which focuses on measurements of “any outage yesterday” and “any outage the day before yesterday.” The possible outcomes were i) both the survey and the boxes report an outage, ii) both the survey and boxes report no outage, iii) the survey reports an outage, and the boxes report no outage, and iv) the boxes report an outage, and the survey reports no outage. The estimates of “survey underreporting” bias should be comparable in this approach to the estimates provided in table 3 and indeed, summary statistics reported in table 4 align as expected.

9
**Table 3: T-Test of Difference within the Same PSU and During the Same Reference Period.**

|**Mean**<br>**Std.**<br>**Err.**|**Std.**<br>**Dev.**|**[95**<br>**I**|**% Conf.**<br>**nterval]**|
|---|---|---|---|
|**1. Yesterday (Any Outage)**||||
|Survey<br>0.184<br>0.003|0.387|0.178|0.190|
|Boxes<br>0.288<br>0.003|0.453|0.281|0.296|
|_diff_<br>_-0.104 ***_<br>_0.004_|_0.537_|_-0.113_|_-0.095_|
|**2. Day Before Yesterday (Any Outage)**<br> <br> <br>||||
|Survey<br>0.191<br>0.003|0.393|0.184|0.198|
|Boxes<br>0.284<br>0.003|0.451|0.276|0.292|
|_diff_<br>_-0.092 ***_<br>_0.004_|_0.542_|_-0.101_|_-0.083_|
|**3. Past 10 Days (Any Outage)**||||
|Survey<br>0.287<br>0.003|0.452|0.279|0.295|
|Boxes<br>0.602<br>0.004|0.489|0.593|0.610|
|_diff_<br>_-0.314 ***_<br>_0.005_|_0.657_|_-0.325_|_-0.303_|
|**4. Yesterday Hours of Outage (Out of 24)**<br>||||
|Survey<br>1.851<br>0.041|4.745|1.769|1.933|
|Boxes<br>2.193<br>0.053|6.041|2.088|2.297|
|_diff_<br>_-0.341 ***_<br>_0.057_|_6.523_|_-0.453_|_-0.228_|
|<br>**5. Day Before Yesterday Hours of Outage (O**<br> <br> <br>|**ut of 24)**<br>|||
|Survey<br>1.940<br>0.043|4.893|1.856|2.025|
|Boxes<br>2.180<br>0.054|6.064|2.076|2.285|
|_diff_<br>_-0.239 ***_<br>_0.058_|_6.627_|_-0.353_|_-0.125_|
|<br>**6. Yesterday Hours 18:00-22:00 (Out of 4)**<br> <br> <br>||||
|Survey<br>0.074<br>0.003|0.454|0.066|0.082|
|Boxes<br>0.317<br>0.008|1.017|0.299|0.334|
|_diff_<br>_-0.242 ***_<br>_0.009_|_1.086_|_-0.261_|_-0.247_|
|<br>**7. Day Before Yesterday Hours 18:00-22:00 (O**<br> <br> <br>|**ut of 4)**<br>|||
|Survey<br>0.080<br>0.004|0.488|0.072|0.088|
|Boxes<br>0.319<br>0.008|1.022|0.302|0.337|
|_diff_<br>_-0.239 ***_<br>_0.009_|_1.106_|_-0.258_|_-0.220_|
|**8. Past 10 Days (Out of 10)**||||
|Survey<br>0.665<br>0.012|1.449|0.640|0.690|
|Boxes<br>2.921<br>0.029|3.365|2.863|2.979|
|_diff_<br>_-2.256 ***_<br>_0.031_|_3.605_|_-2.318_|_-2.194_|
|**9. Past 10 Days: Hours (Out of 24)**||||
|Survey<br>2.002<br>0.044|5.035|1.915|2.088|
|Boxes<br>2.280<br>0.049|5.664|2.182|2.377|
|_diff_<br>_-0.278 ***_<br>_0.064_|_7.318_|_-0.404_|_-0.152_|

All are observations in or after 2016, Nov 1.

10
**Table 4: Comparison of Status between Survey and SSBs**

1. Yesterday (Any Outage)

|||||**S**|**SBs**|||
|---|---|---|---|---|---|---|---|
||Obs.|on<br>o|ff|Total|Fraction|on|off|
||on|7993|2613|10606|on|0.615|0.201|
|**Survey**|off|1259|1123|2382|off|0.097|<br>0.086|
||Total|9252|3736|12988|Svy Unde|r report|-10.4%|
|. Day Be|fore Yest|erday (Any|Outage)|**S**|**SBs**|||
||Obs.|on<br>o|ff|Total|Fraction|on|off|
||on|7900|2560|10460|on|0.611|0.198|
|**Survey**|off|1362|1118|2480|off|0.105|<br>0.086|
||Total|9262|3678|12940|Svy Unde|r report|-9.3%|
|. Past 10|Days (A|ny Outage)||**S**|**SBs**|||
||Obs.|on<br>o|ff|Total|Fraction|on|off|
||on|3749|5479|9228|on|0.289|0.423|
|**Survey**|off|1405|2321|3726|off|0.108|<br>0.179|
||Total|5154|7800|12954|Svy Unde|r report|-31.4%|

2. Day Before Yesterday (Any Outage) 3. Past 10 Days (Any Outage)

# III – Accounting for fixed effects and time trends

A complimentary approach to document systematic differences between survey-based data and SSBs uses a regression framework to establish average cluster-level differences between survey-based reported outages and actual outages, while accounting for temporal fixed effects: where Ygt is the outcome variable of interest (such as average outage duration, or number of outages) aggregated in PSU _g_ for time _t_ . The variable Survg is a dummy variable indicating whether the observation was obtained using data collected from the survey (==1) or from the automated system (==0). The term τt is a vector of calendar date dummies and εgt is a random error term.

The first set of results provided in Table 5 demonstrate statistically significant differences between the data collection methods. The significantly negative differences between survey responses and SSB results confirm the expectations given the summary statistics in Tables 3 and 4. Survey-based estimates of outages and outage duration were systematically lower than automated measures. In terms of magnitudes, the largest errors were present in comparison 8, where, over a 10-day reference period of having “any outage”, survey estimates are off by about 4 days out of ten, on average.

11
**Table 5: Regression-based Estimates Accounting for Temporal and PSU Fixed Effects**

|Dependent var. (Survey):|One if<br>have<br>outage<br>1 day|One if<br>have<br>outage<br>2 days|Per-day<br>hours<br>of<br>outage<br>1 day|Per-day<br>hours<br>of<br>outage<br>2 days|Per-day<br>hours<br>of<br>outage<br>1 day|Per-day<br>hours<br>of<br>outage<br>2 days|One if<br>have<br>outage<br>over|No. of<br>days<br>having<br>outage<br>10|Per-day<br>hours<br>of outage<br>on|
|---|---|---|---|---|---|---|---|---|---|
||ago|ago|ago|ago|(6pm &|(6pm &|10 days|over<br>days|average|
||||||10pm)|10pm)||||
||(a)|(b)|(c)|(d)|(e)|(f)|(g)|(h)|(i)|
|(A) With no control of frequenci|es of messages|sent in the r|eference da|y & time||||||
|Boxes||||||||||
|one if have 1 day ago|0.070***<br>(0.008)|-|-|-|-|-|-|-|-|
|one if have 2 days ago|<br>-|0.067***<br>(0.008)|-|-|-|-|-|-|-|
|per-day hours 1 day ago|-|<br>-|0.057***<br>(0.007)|-|-|-|-|-|-|
|per-day hours 2 day ago|-|-|-|0.060***<br>(0.008)|-|-|-|-|-|
|per-day hours 1 day ago|-|-|-|-|0.012**|-|-|-|-|
|b/w 6pm and 10pm|||||(0.005)|||||
|per-day hours 2 days ago|-|-|-|-|-|0.008|-|-|-|
|b/w 6pm and 10pm||||||(0.006)||||
|One if have|-|-|-|-|-|-|0.020**|-|-|
|over the last 10 days|||||||(0.010)|||
|No. of days|-|-|-|-|-|-|-|0.013**|-|
|over the last 10 days||||||||(0.006)||
|per-day hours,|-|-|-|-|-|-|-|-|0.021|
|<br>average over 10 days|||||||||(0.013)|
|R-squared|0.412|0.387|0.563|0.534|0.072|0.074|0.071|0.066|0.081|
|<br>No. of obs.|12928|12940|12928|12939|12941|12937|12954|12954|12953|
|**Interview date FE**|YES|YES|YES|YES|YES|YES|YES|YES|YES|

12
# IV – What determines accuracy?

To understand the household and/or individual characteristics that correlate with reporting error, a new variable is constructed as the squared difference between reported and observed outages over specific periods (coinciding with the reference periods in the survey), such that: where ΔY̅it is the difference in the outcome variable of interest between the survey and the observed outcome; for instance, whether household _i_ had any outage, or the length of the outage, for individual _i_ in week _t_ . The variable Charsi is a set of household characteristics of interest, such as per capita income and demographic indicators. The term τt is a vector of calendar week dummies and εit is a random error term.

The results reported in tables 6 and 7 do not find that richer people are systematically more (or less) likely to mis-report outages. However, households with self-reported sick members were much more likely to report outages, bringing them closer to agreement with the automated measures. This may have been due to greater sensitivity to the effect of outages on wellbeing; however, the study did not have a direct means to observe this potential explanation. As suggested by the summary statistics reported in tables 3 and 4, reporting was more accurate when short (daily) reference periods were used, with similar accuracy of responses for “yesterday: and the “day before yesterday.” Rural households were more accurate than urban households.

13
**Table 6: Regression Framework including Correlates of Reporting Accuracy**

|Dependent var.|Differenc|e(= survey-|boxes)||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
||One if|One if|Per-day|Per-day|Per-day|Per-day|One if|No. of|Per-day|One if|Per-day|Per-day|
||have|have|hours|hours|hours<br>|hours<br>|have|days|hours|have|hours|hours|
||outage|outage|of outage|of outage|of<br>outage|of<br>outage|outage|having|of outage|outage||of outage|
||1 day|2 days|1 day|2 days|1 day|2 days|over|outage<br>|on|||(6pm &|
||ago|ago|ago|ago|(6pm &|(6pm &|10 days|over 10<br>days|average|||10pm)|
||||||10pm)|10pm)|||||||
||(a)|(b)|(c)|(d)|(e)|(f)|(g)|(h)|(i)|(j)|(k)|(l)|
|Work age|-0.005|-0.028|0.004|-0.036|-0.026|-0.004|-0.037|-0.084|-0.046|-0.016|-0.016|-0.015|
||(0.024)|(0.023)|(0.329)|(0.343)|(0.055)|(0.056)|(0.036)|(0.195)|(0.371)|(0.021)|(0.323)|(0.053)|
|No tertiary education|0.001|0.002|0.302|0.175|0.011|0.007|-0.035|-0.206|-0.127|0.002|0.238|0.009|
||(0.019)|(0.020)|(0.255)|(0.253)|(0.043)|(0.043)|(0.031)|(0.166)|(0.332)|(0.017)|(0.242)|(0.041)|
|Sick|0.049***|0.028*|0.446**|0.490**|0.075**|0.063*|0.128***|0.309***|0.872***|0.039***|0.469**|0.069**|
||(0.015)|(0.015)|(0.195)|(0.195)|(0.033)|(0.034)|(0.022)|(0.117)|(0.221)|(0.013)|(0.185)|(0.031)|
|Head\spouse unemployed|-0.012|-0.006|-0.140|-0.047|0.002|-0.026|-0.018|-0.067|-0.314|-0.009|-0.094|-0.012|
||(0.021)|(0.021)|(0.284)|(0.290)|(0.049)|(0.049)|(0.030)|(0.174)|(0.324)|(0.019)|(0.276)|(0.047)|
|Household size|-0.004|-0.001|-0.016|-0.004|-0.008|-0.003|-0.002|-0.030|-0.037|-0.002|-0.010|-0.006|
||(0.003)|(0.003)|(0.035)|(0.040)|(0.006)|(0.007)|(0.004)|(0.024)|(0.045)|(0.003)|(0.036)|(0.006)|
|Per capita income|0.004|0.008|0.003|0.056|0.003|0.003|0.003|0.008|-0.075|0.006|0.030|0.003|
||(0.007)|(0.007)|(0.097)|(0.096)|(0.016)|(0.016)|(0.010)|(0.050)|(0.107)|(0.006)|(0.091)|(0.015)|
|Have migrant|0.037|0.036|0.201|0.324|0.047|0.094*|0.106***|0.559***|0.738*|0.037|0.262|0.071|
||(0.025)|(0.028)|(0.321)|(0.346)|(0.052)|(0.057)|(0.032)|(0.177)|(0.389)|(0.023)|(0.317)|(0.051)|
|Urban|-0.095***|-0.065***|-0.881***|-0.735***|0.028|0.016|-0.016|0.258*|0.095|-0.080***|<br>-0.808***|0.022|
||(0.018)|(0.019)|(0.229)|(0.238)|(0.038)|(0.038)|(0.030)|(0.150)|(0.276)|(0.016)|(0.223)|(0.036)|
|2 day recall|-|-|-|-|-|-|-|-|-|0.017**|0.119**|0.007|
|(refgp: 1 dayrecall)||||||||||(0.007)|(0.048)|(0.009)|
|Interview date FE|YES|YES|YES|YES|YES|YES|YES|YES|YES|YES|YES|YES|
|R-squared|0.186|0.176|0.164|0.155|0.138|0.130|0.112|0.140|0.133|0.158|0.152|0.123|
|No. of obs.|6447|6453|6447|6453|6453|6450|6457|6457|6456|12900|12900|12903|

14
# **Table 7: Capturing Seasonality and Salience**

|Dependent var. (Self-report):|||One if h|ave outage 1|dayago|||
|---|---|---|---|---|---|---|---|
||All|Oct toJan|(dummy)|= 1|Oct toJan|(dummy|)= 0|
||season|Total|Urban|Rural|Total|Urban|Rural|
||(a)|(b)|(c)|(d)|(e)|(f)|(g)|
|Panel (A)||||||||
|<br>SSB||||||||
|One if have 1 day ago|0.169***|0.259***|0.069**|0.324***|0.059***|0.021|0.074***|
||(0.010)|(0.017)|(0.027)|(0.020)|(0.010)|(0.013)|(0.012)|
|R-squared|<br>0.084|<br>0.135|<br>0.116|<br>0.128|<br>0.086|<br>0.031|<br>0.091|
|<br>No. of obs.|12928|5322|1388|3934|7606|2001|5605|
||(h)|(i)|(j)|(k)|(l)|(m)|(n)|
|Panel (B)||||||||
|SSB||||||||
|One if have 1 day ago|0.105***|0.138***|0.047|0.170***|0.052***|0.013|0.067***|
|<br>but not b/w 6pm and 10pm|(0.011)|(0.020)|(0.035)|(0.023)|(0.012)|(0.017)|(0.015)|
|<br>One if have 1 day ago|<br>0.246***|<br>0.390***|<br>0.094***|<br>0.486***|<br>0.069***|<br>0.031|<br>0.085***|
|including b/w 6pm and<br>10pm|(0.013)|(0.021)|(0.036)|(0.024)|(0.013)|(0.020)|(0.017)|
|<br>R-squared|0.093|0.158|0.117|0.161|0.086|0.031|0.091|
|No. of obs.|12928|5322|1388|3934|7606|2001|5605|
|**Interview date/week FE**|NO|NO|NO|NO<br>|NO|NO|NO|
|PSU FE|YES|YES|YES|YES<br>|YES|YES|YES|
|Dependent var. (Self-report):|||Per-dayho|urs of outage|1 dayago|||
||All|Oct toJa|n(dummy)|= 1|Oct toJ|an(dumm|y)= 0|
||season|Total|Urban|Rural|Total|Urban|Rural|
||(a)|(b)|(c)|(d)|(e)|(f)|(g)|
|Panel (A)||||||||
|<br>SSB||||||||
|per-day hours 1 day ago|0.253***|<br>0.391***|<br>0.119***|<br>0.468***|0.013**|0.002|0.016**|
||(0.013)|(0.017)|(0.026)|(0.018)|(0.006)|(0.005)|(0.008)|
|R-squared|<br>0.133|<br>0.238|<br>0.194|<br>0.239|<br>0.110|<br>0.032|<br>0.109|
|<br>No. of obs.|12928|5322|1388|3934|7606|2001|5605|
|**Interview date/week FE**|NO|NO|NO|NO|NO|NO|NO|
|PSU FE|YES|YES|YES|YES|YES|YES|YES|

15
# V – Conclusion

Access to reliable electricity is a key determinant of both economic growth and individual wellbeing. However, when aggregated to summary statistics, traditional survey-based methods of collecting service reliability data are of questionable validity due to several potential sources of bias. A low-cost national electricity outage monitoring network of Smart Survey Boxes (SSBs) was built for this study, alongside the rollout of a high-frequency phone-based household survey called Listening to Tajikistan. The integration of the two allows benchmarking the survey responses against the unbiased automated system.

The results show that although the two measures were correlated, the survey data suffered from significant and systematic bias. Survey respondents i) systematically underreported the incidence and duration of electricity outages on average, but ii) systematically overreported the incidence and severity of outages during periods of abnormally widespread outages of long duration. These findings indicate that, where feasible, automated monitoring can provide more accurate measurement of the quality of electricity service provision. For survey settings, the results also suggest that estimates are much more accurate when short (daily) reference periods are used, but that care should be taken to account for i) time trends and ii) the salience of electricity outages for respondent subgroups.

16
# VI – References

<mark>Bardasi, Elena, et al. "Do labor statistics depend on how and to whom the questions are asked? Results from a survey experiment in Tanzania."</mark> _<mark>The World Bank Economic Review</mark>_ <mark>25.3 (2011): 418-447.</mark> <mark>Beegle, Kathleen, et al. "Methods of household consumption measurement through surveys: Experimental results from Tanzania."</mark> _<mark>Journal of Development Economics</mark>_ <mark>98.1 (2012): 3-18.</mark> <mark>Beenstock, Michael, Ephraim Goldin, and Yoel Haitovsky. "Response bias in a conjoint analysis of power outages."</mark> _<mark>Energy Economics</mark>_ <mark>20.2 (1998): 135-156.</mark> <mark>Watson, N. R., A. Miller, and M. Campbell. "Smart meters to monitor power quality at consumer premises." (2015).</mark> <mark>Fricker, Scott, et al. "A Review of Measurement Error Assessment in a US Household Consumer Expenditure Survey."</mark> _<mark>Journal of Survey Statistics and Methodology</mark>_ <mark>(2015)</mark> <mark>Hartman, Raymond S., Michael J. Doane, and Chi-Keung Woo. "Consumer rationality and the status quo."</mark> _<mark>The Quarterly Journal of Economics</mark>_ <mark>(1991): 141-162.</mark> <mark>Heath, Rachel, et al. "Measuring Employment: Experimental Evidence from Urban Ghana."</mark> _<mark>The World Bank Economic Review</mark>_ <mark>35.3 (2021): 635-651.</mark> <mark>Jimenez, Raul, Tomás Serebrisky, and Jorge Enrique Mercado Díaz.</mark> _<mark>What does better mean? Perceptions of Electricity and Water Services in Santo Domingo</mark>_ <mark>. Inter-American Development Bank, 2015.</mark> <mark>Kahneman, Daniel, Jack L. Knetsch, and Richard H. Thaler. "Anomalies: The endowment effect, loss aversion, and status quo bias."</mark> _<mark>The journal of economic perspectives</mark>_ <mark>5.1 (1991): 193-206.</mark> MacMillan, Douglas <mark>"Actual and Hypothetical Willingness to Pay for Environmental Outputs: Why are they Different?” (2004).</mark> <mark>Woo, Chi-Keung, et al. "Residential outage cost estimation: Hong Kong.”</mark> _<mark>Energy policy</mark>_ <mark>72 (2014): 204-210.</mark> 17
# Appendix A – Within PSU variation in outage reporting

Comparing SSBs within the same PSU provides an estimate of how often residents living in the same PSU will experience an outage. The results find a relatively high variance, with only 35 percent of outages reported by one box coinciding with the other in terms of timing and duration. Table 8 reports these metrics for 3 version of “agreement” between boxes: overall agreement, agreement on the timing and duration of outages, and agreement on electricity being on during a given 60-minute interval.

**Table 8: Within PSU Box Agreement**

||Overall|Agree<br>electricity<br>outage|Agree<br>electricity<br>on|
|---|---|---|---|
|Oct-16|98%|69%|94%|
|Nov-16|90%|68%|88%|
|Dec-16|76%|61%|77%|
|Jan-17|86%|41%|88%|
|Feb-17|90%|35%|90%|
|Mar-17|89%|26%|89%|
|Apr-17|86%|22%|88%|
|May-17|85%|22%|87%|
|Jun-17|86%|21%|87%|
|Jul-17|83%|19%|86%|
|Aug-17|83%|15%|87%|
|Sep-17|86%|16%|89%|
|Oct-17|87%|29%|89%|
|Nov-17|84%|25%|87%|
|Dec-17|88%|27%|89%|
|Jan-18|92%|30%|91%|
|Total|86%|35%|88%|

18
# Appendix B – Survey-based measures of electricity service

**Table 9: Survey-based Questions**

|Didyou have anyelectricityoutage at home over thepast 10 days?|
|---|
|On how manydays were there electricityoutages at home over thepast 10 days?|
|On the days with electricitycuts,for how manyhours was electricityout at home,on average?|
|For how manyhours was electricityavailable foryour householdyesterday?|
|For how manyhours was electricityavailable between 6PM and 10PMyesterday?|
|For how manyhours was electricityavailable foryour household the daybeforeyesterday?|
|For how manyhours was electricityavailable between 6PM and 10PM the daybeforeyesterday?|
|Over thepast 10 days,have anyofyour electrical appliances been damaged byapower surge?|
|What is source of electricitysupplyused most inyour household?|
|Haveyou changed the electricitysource used most often in thepast 6 months?|
|If so,what was source of electricitysupply previouslyused most inyour household?|
|Who doyou currently payfor electricity?|
|In the last 12 months, did any household members die or have permanent limb (bodily injury) damage<br>because of electricitysystem?|
|Doesyour household have a voltage stabilizer?|
|Isyour household advised not topower certain appliances at the same time?|

19