metadata: authors: 0: author: Jolliffe,Dean Mitchell
1: author: Mahler,Daniel Gerszon
2: author: Veerappan,Malarvizhi
3: author: Kilic,Talip
4: author: Wollburg,Philip Randolph
count: World
docna: 0: docna: Under What Conditions Are Data Valuable for
            Development ?
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Inequality,Hydrology,Gender and Development,Water Supply and Sanitation Economics,Sanitary Environmental Engineering,Small Private Water Supply Providers,Health and Sanitation,Water and Human Health,Town Water Supply and Sanitation,Sanitation and Sewerage,Engineering,Environmental Engineering
teratopic: Industry,Water Supply and Sanitation,Environment,Science and Technology Development,Health, Nutrition and Population,Poverty Reduction,Gender,Water Resources
abstracts: cdata!: Data produced by the public sector can
            have transformational impacts on development outcomes
            through better targeting of resources, improved service
            delivery, cost savings in policy implementation, increased
            accountability, and more. Around the world, the amount of
            data produced by the public sector is increasing at a rapid
            pace, yet their transformational impacts have not been
            realized fully. Why has the full value of these data not
            been realized yet? This paper outlines 12 conditions needed
            for the production and use of public sector data to generate
            value for development and presents case studies
            substantiating these conditions. The conditions are that
            data need to have adequate spatial and temporal coverage
            (are complete, frequent, and timely), are of high quality
            (are accurate, comparable, and granular), are easy to use
            (are accessible, understandable, and interoperable), and are
            safe to use (are impartial, confidential, and appropriate).
repnme: repnme: Under What Conditions Are Data Valuable for
            Development ?
display_title: Under What Conditions Are Data
            Valuable for Development
listing_relative_url: /research/2021/10/33497843/under-conditions-data-valuable-development
topicv3: Sanitation m1326453 2000,Water m1326522 137,Poverty m1326687 695,Gender m1327621 615,Health m1504661 3480
publishtoextweb_dt: 2021-10-19T00:00:00Z
dois: 10.1596/1813-9450-9811
historic_topic: Industry,Water Supply and Sanitation,Environment,Science and Technology Development,Health, Nutrition and Population,Poverty Reduction,Gender,Water Resources
origu: Indicators and Data Services (DECIS)
no_of_pages: 30
author_keys: 0: author_key: 773551
1: author_key: 1539953
2: author_key: 665420
3: author_key: 886702
4: author_key: 1464872
model_extractions: Policy Research Working Paper 9811

# Under What Conditions Are Data Valuable for Development?

_Dean Jolliffe Daniel Gerszon Mahler Malarvizhi Veerappan Talip Kilic Philip Wollburg_ Development Economics Development Data Group October 2021
Policy Research Working Paper 9811

# **Abstract**

Data produced by the public sector can have transformational impacts on development outcomes through better targeting of resources, improved service delivery, cost savings in policy implementation, increased accountability, and more. Around the world, the amount of data produced by the public sector is increasing at a rapid pace, yet their transformational impacts have not been realized fully. Why has the full value of these data not been realized yet? This paper outlines 12 conditions needed for the production and use of public sector data to generate value for development and presents case studies substantiating these conditions. The conditions are that data need to have adequate spatial and temporal coverage (are complete, frequent, and timely), are of high quality (are accurate, comparable, and granular), are easy to use (are accessible, understandable, and interoperable), and are safe to use (are impartial, confidential, and appropriate).

This paper is a product of the Development Data Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at djolliffe@worldbank.org, dmahler@worldbank.org, mveerappan@worldbank.org, tkilic@worldbank.org, and pwollburg@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Under What Conditions Are Data Valuable for Development?**<sup>**1**</sup>

Dean Jolliffe Daniel Gerszon Mahler Malarvizhi Veerappan Talip Kilic Philip Wollburg

**Keywords** : Data, development, statistics **JEL codes** : C81, C83, O10, O20

> 1 The author ordering was constructed through American Economic Association’s randomization tool (confirmation code: qhPJIjX8dDgr). All authors are with the Development Data Group of the World Bank. Jolliffe (djolliffe@worldbank.org) and Veerappan  (mveerappan@worldbank.org) are the co-director and the manager, respectively for the _World Development Report 2021: Data for Better Lives_ , while Kilic (tkilic@worldbank.org), Mahler (dmahler@worldbank.org), and Wollburg (pwollburg@worldbank.org)  are among the core team members. Jolliffe is also a Research Fellow with the Institute for the Study (IZA) of Labor and a Fellow with the Global Labor Organization (GLO). The authors are thankful for comments and feedback received from Andrew Dabalen, Gero Carletto, Kathleen Beegle, Lucas Kitzmuller, Luis Alberto Andres, Paolo Verme, Robert Cull, Tim Kelly, Umar Serajuddin, and Vivien Foster. In addition, they are thankful for the many colleagues who provided and reviewed case studies including AnnSofie Jespersen, Aparajita Goyal, Audrey Ariss, Benjamin David Roseth, Emilia Galiano, Elizabeth Goldman, Florence Kondylis, Frederic Meunier, Hana Brixi, Isis Gaddis, Joao Pedro Wagner De Azevedo, Madalina Papahagi, Marc Tobias Schiffbauer, Marelize Gorgens, Maria Poli, Natalia Baal, Natasha Rovo, Megumi Kubota, Paul Andres Corral Rodas, Sabina Alkire, Sonali Vyas, Stephane Hallegatte, Stephanie Jamilla, Tea Trumbic, Theresa Beltramo, Utz Johann Pape, and Zelalem Yilma Debebe. The authors gratefully acknowledge financial support from the UK Government through the Knowledge for Change Program.
# **1. Introduction**

Around the world, the public sector is producing more and more data. Since 2005, the number of countries without a population and housing census conducted over the preceding ten years has fallen by nearly 80 percent (from 36 to 8), the number of countries without a labor force survey conducted over the same period has fallen by 50 percent (from 98 to 49), and the number of countries without nationally representative administrative data on learning assessments within a five-year range has fallen by 83 percent since 2008 (from 36 to 6) (World Bank 2021a).

As highlighted by the World Development Report (WDR) 2021: Data for Better Lives, the abundance of data produced by the public sector can have a transformational role in development and efforts to reduce poverty. Among many fruitful uses, the data can be used to increase access to government services, prepare for and respond to emergencies, target resources and foster the inclusion of marginalized groups, save money and resources in policy implementation and service delivery, monitor progress and track performance, increase accountability, and empower individuals (World Bank 2021b).

Yet are we really seeing the returns from the data that one could hope for given their abundance? This paper identifies the conditions under which data produced by the public sector are valuable for development, and expands on the assertions and examples highlighted by the WDR 2021. The clear identification of these conditions can help governments understand how to improve the supply of and demand for data. To this end, we first develop a theoretical framework detailing 12 conditions needed for data to generate value. Next, we use a collection of case studies to provide support for these conditions and showcase how they matter in practice. Our examples and analysis distill that for data to maximize value, the data should have adequate coverage (be complete, frequent, and timely), be of high quality (be accurate, comparable, and granular), be easy to use (be accessible, understandable, and interoperable), and be safe to use (be impartial, confidential, and appropriate). Too often, we find, the data produced by governments do not satisfy these conditions and thus are not conducive to transforming development outcomes. The data may be of poor quality, siloed in various administrative systems, not shared with the public, not readable by computers, and so forth.

We restrict our analysis to data collected by government agencies, such as surveys, censuses, and administrative data, with a focus on low and middle-income countries. This means that we will neglect private sector data, citizen generated data, and data from high-income countries. We believe that the case for improving the stock of high-quality data and the safe use of data is particularly pertinent for governments in low and middle-income countries. Our objective is to provide a series of examples that illustrate the conditions under which development data can generate value. The realized social value in these examples is large and typically occurs in nonmonetary dimensions, such as improved health and safety.

Several other frameworks exist that list features conducive for data to be valuable. Most of these frameworks have been developed by national statistical offices or international organizations to guide data producers (see for example Statistics Canada (2017), OECD (2011), 2
UN (2019), and Biemer (2010)). We make sure to compare our framework with such other relevant frameworks.

Other collections of case studies of the impact of data on development exist as well, and we will draw examples from some of these other collections. Among these are the Data Impact Case Studies gathered by Open Data Watch (2021), the Value of Data Case Studies gathered by the Global Partnership for Sustainable Development Data (2021), Open Data Impact Map’s collection of use cases of open data (Center for Open Data Enterprise 2021), GovLab’s collection on the same topic (Verhulst and Young, 2016), a World Bank collection on the same topic (World Bank 2015), data2x’s collection of examples related to gender data (data2x 2018 and data2x 2019a), and the World Bank’s collection of examples related to digital identification in the health sector (World Bank 2018a), the public sector (World Bank 2018b), and the private sector (2018c).

We aim to contribute to the literature by directly tying the theoretical framework of data attributes to case studies, that is, to show how certain attributes of the data can help generate value for development. We make no attempt to estimate the net social value of government data, in part because many of the benefits occur in dimensions for which prices do not exist (e.g. improved health from drinking clean water), but this also reflects our view that there is no satisfactory way to assign a monetary value to an inexhaustible, or nonrival, good. The limitless scope for data to be used and reused to address new and unexpected problems, as well as the potential for data to be misused for public harm, suggests to us that any estimate of the expected net social value of government data is one that will be very imprecise. Rather by reviewing a series of examples where public data has been used to improve well-being, we provide evidence of the many ways in which data can be effectively used for the social good. Further by connecting the examples to our framework, we aim to bring structure to the series of examples which helps support our inferences on how best to scale up public data systems to realize greater value. Our hope is that this connection can arm data producers and users with evidence needed for decision makers to prioritize data.

# **2. Theory**

For data to generate value the data need to fulfill certain requirements related to their supply and demand. For the supply of data to generate value the data need to have adequate spatial and temporal coverage and be of high quality. With respect to coverage, the data need to cover all relevant units and time periods. Additionally, in the case of sample data, there needs to be proper sampling design, including adequate sample size, to support sufficiently precise inferences. We will argue that this means that the data need to be complete, frequent, and timely (these features, and the ones to follow will be defined more clearly in what follows). With respect to quality, the data need to accurately measure the concepts of interest to the users and be comparable and granular. To ensure a proper demand for data, the data should be easy to use and safe to use. Easy to use means that the data can be accessed, that they are understandable, and that they are 3
<!-- Start of picture text --> Value for<br>development<br>Supply Demand<br>Adequate High quality Easy to use Safe to use<br>coverage<br>Complete Granular Accessible Impartial<br>Frequent Accurate Understandable Confidential<br>Timely Comparable Interoperable Appropriate<br><!-- End of picture text -->
outcomes, such as macroeconomic data, the desired frequency may be monthly or quarterly. In each case, the desired frequency of data capture depends ultimately on the temporal dynamics of the outcome of interest, and this frequency may change as other conditions change (e.g. ideal frequency of economic data is most likely high during volatile economic times and might be low during stagnant periods).

- _Timely_ : Timely data means that the time from data collection to data release is short, or that the time between an event of interest occurring and data collection on the event is short. This is particularly relevant for emergencies where policy response does not have the luxury to wait for data to arrive, such as environmental disasters, health crises, conflicts and more. The timeliness of data depends on the mode of data collection. Some machine-generated data can be made available nearly instantaneously, while other data, such as survey data, often takes place over a longer period of time and needs careful analysis and cleaning before it can be released, leaving a time lag of several months. A related name for this feature is that the data need to be punctual.

# **High quality**

- _Accurate_ : For data to be valuable, they need to measure their concept of interest with minimal error. This means that data collection and processing should use methods and tools that are designed to capture the signal of some phenomena while filtering out noise or measurement error that can bias conclusions and evidence-based policies.

- _Comparable_ : For data to be used for cross-country analysis or for tracking patterns over time, they need to be comparable across space and time. This means, for example, that the data should conform to certain standards and data collection instruments should be relatively stable over time. Examples of conforming to standards include the near universal acceptance of governments in using the System of National Accounts for measuring the size of their economies, and similarly the widely accepted standards for measuring geolocation and units of time. There are fewer clear examples of how to maintain comparability over time, but one common view is to avoid making unnecessary changes to the data collection processes which may induce a break in comparability of the data series. A caveat to this view is that data collection instruments may require change to maintain comparability if the behavior of the enumerated entity changes significantly (e.g., list of consumption items may need to change if new items are introduced into society)<sup>2</sup> or if international standards of data collection are updated over time, in part to facilitate the adoption methods to increase accuracy.

- _Granular_ : Often for data to maximize their value they need to be granular, which, loosely defined, means that they can be broken down by subgroups, such as geographical areas, time, gender, disability status, and more. This requires these subgroups to be captured appropriately in the data. The level of detail available with granular data means that deidentifying subjects becomes instrumental to protect the privacy of the subjects.

> 2 Gibson et al. (2017) offer an example where a stable, unchanged consumption questionnaire during a period of rapidly changing consumption patterns in Vietnam led to concerns about comparability of the measures of household well-being over time.

5
# **Easy to use**

- _Accessible_ : For data to maximize their value for development they need to be accessible to a wide audience of users. Too often this is not the case, preventing high-quality data from being used. Making data accessible entails that the data are machine-readable, meaning that content can be processed by computers, and, where relevant, that it is made available free of charge. Not all data should be made available to everyone, and to ensure the safe use of data, accessibility may be conditioned on a terms of use agreement and on other restrictions, such as how and where data can be accessed.

- _Understandable_ : Making data accessible is not enough for them to be useful. For that, that data need to be easy to understand as well. This entails that the data come with metadata describing how they were generated/collected and how to process them. In certain cases, the value and use of data can also be maximized when they are summarized or visualized in figures, tables and more. This ensures that data literacy and data processability are not the constraining factors for data use.

- _Interoperable_ : For data to maximize their use, it is often necessary to link and combine different data sources through common identifiers for persons, facilities, or firms, geospatial coordinates, time stamps, and common classification standards. This ensures that information from multiple data sets can be combined, maximizing the potential uses. Interoperability is increased when data are confirming to specific standards. Interoperability increases the risk of data breaches and misuse, implying that terms of use agreements ought to be in place for users wishing to merge different data sets.

# **Safe to use**

- _Impartial_ : For data to be used safely, they need to be impartial, that is, immune to the influence of any stakeholders in the lifecycle of the data, such as funders, producers, or users. If certain stakeholders can influence the data, the data lose credibility and prevent objective data analysis. Lack of impartiality can have externalities on other data products; if users know that one data source has been meddled with, they may lose trust in all data products from the same institution, severely limiting the value of data for development.

- _Confidential_ : For data to be confidential, data should be de-identified and access should occur in a safe and secure manner. The former implies that individual persons, establishments or whichever is the subject of the data cannot be identified in the data. The latter could imply that data can only be accessed from certain virtual or physical enclaves or through systems preventing users from storing the data locally. Both aspects seek to minimize the possibility of data misuse, which can jeopardize the value data bring for development.

- _Appropriate_ : One way to restrict misuse of data is to ensure that only data appropriate for measuring the concepts of interest or a clear policy purpose are produced and used, without attempts to collect excessive information or conduct surveillance. Appropriateness implies a proportionality or adequacy principle, by which the amount of data is proportional or adequate to the data need.

6
# **Interactions between data features**

As a general rule, the more present a feature is, the greater value the data bring to development. But there are important caveats and nuances to this broad assertion, each linked to the point that the effect of one feature on the value of data is not independent of changes in the other data features. More specifically, there are three important ways in which value is determined from the interaction of features: (1) the feature least present is often the constraining factor on the value that can be derived (i.e., increasing other features may not increase value if the least present feature is unchanged), (2) at times there are positive spillovers between features, and (3) at times there are negative spillovers between features.

_(1) The importance of the features least present._ Often the derived value will be determined by the category of features (adequate coverage, high quality, easy to use, and safe to use) that is least present. Such a situation is comparable to Kremer’s O-Ring theory of economic development, where one malfunction in the value chain in production can dramatically reduce a product’s value (Kremer 1993). More colloquially, this is sometimes expressed as a chain being only as good as its weakest link. If nearly all features are present, but the data are not produced with adequate coverage, they may not be able to serve their intended policy objective at all, dramatically reducing their value. When data are not of high quality, they are much less valuable and could even misguide policy decisions. When the data are not easy to use, they might not be used at all. When data are not safe to use, they may end up causing harm due to data breaches, surveillance, or exclusion.

_(2) Positive spillovers between features._ The features that are present in a data set may foster the development or advancement of other features. For example, if data are made available and understandable to users, the use and demand for the data will likely increase. This, in turn, can have trickle-down effects on the supply of data. For example, government ministries’ reliance on and demand for high-quality data has been associated with a higher capacity of national statistical offices in Latin America (Lotta et al. 2020). In the same region, demand for and interest in accurate and high-quality statistics in civil society, academia, the media, and the private sector have been found to lead to better funding, autonomy, and capacity in national statistical systems (Lotta et al. 2020). In Figure 1, this is reflected by the double-ended arrow between supply and demand. The argument also goes the other way; the absence of a feature might cause the absence of another feature. For example, when data quality is weak, and especially when methodological foundations are vague, it becomes easier to manipulate statistics in one direction or another (Jerven, 2019).

_(3) Negative spillovers between features._ On other occasions, the presence of one feature has a negative impact on other features. For data producers, this can create trade-offs and conflicts between features. For example, data producers may need to find a balance between increasing completeness and the costs of doing so. They may also have to balance increased timeliness with having less time to assure the quality of the data and foster understandability of the data. And perhaps most importantly, the potential harm from data not being safe to use increases when certain other features are in place. For example, when data are granular, interoperable, and have wide coverage, they can be misused more easily and more effectively for illicit ends, magnifying the risk of confidentially violations. In essence, because these features make the data very useful and usable, they multiply the ways in which the data could be used to cause harm.

7
COVID-19 case data provide an interesting illustration of the importance of the interactions between features. Immense value has been derived from the presence of some of the 12 features but constrained by the absence of other features. This timeliness and frequency of the data has allowed countries to compare the possible trajectories they are likely to face and scale their emergency response, social distancing measures, and economic recovery packages accordingly. The case data are often tied to specific geographical locales and to an individual for whom other relevant data, such as age and medical history, are available. This granularity has made it possible to understand how clusters of cases evolved and how mortality rates differ by individual circumstances and characteristics. This has allowed governments to take extra measures for people who are suffering from or are vulnerable to the virus. The accessibility of the data has enabled quick research on the topic, providing the evidence policy makers need when deciding what measures to implement.

While these features have helped shape measures to contain and limit the spread and consequences of the virus, the measures could have improved had the quality of the data been better. In particular, questions about data accuracy and cross-country comparability have led to concerns about their relevance for policy making. As countries have increased their testing capabilities over time, more people have been reported as having contracted the virus. Though increased testing is critical, this has made data on confirmed cases less comparable over time within countries. Data on deaths suffer from cross-country comparability problems, given that the definition of deaths caused by COVID-19 differs across countries, challenging models using cross-country data.

COVID-19 case data also contain cases of positive and negative spillovers between features. On the positive side, the intensive data use during the pandemic will undoubtedly have an impact on the value attached to having timely and accurate data in other domains going forward. On the other hand, as we will argue in more detail later in the paper, the granularity and timeliness of the data have increased the risk of government surveillance for ends beyond just containing the virus.

# **Relation to other frameworks**

The 12 attributes relate closely to other frameworks that assess the quality of statistical data. Such frameworks have primarily been created by National Statistical Offices (see for example Statistics Canada (2017)), international organizations (see for example OECD (2011) and UN (2019)), and academia (see for example Biemer (2010)). Table 1 compares the framework derived in this paper with these other frameworks.

Unsurprisingly, there is a great overlap between the different frameworks, though the exact names and definitions differ from source to source. All 12 features we identify can be found in some shape or form in one of the other frameworks. Though many of the frameworks originate from work with survey data, they can be adapted to other data types. The framework of Biemer (2010), for example, which is based on the concept of a total survey error, has been adapted to big data (Amaya et al. 2020) and agricultural data (Carletto et al. 2021).

8
**Table 1: Comparison with other theoretical frameworks**

|||This paper|Statistics Canada<br>(2017)|OECD (2011)|UN (2019)|Biemer (2010)|
|---|---|---|---|---|---|---|
||erage|_Complete_|Coverage|||Completeness|
||uate cov|_Frequent_|Viability||||
|pply|Adeq|_Timely_|Timeliness &<br>punctuality|Timeliness &<br>punctuality|Timeliness &<br>punctuality|Timeliness/<br>punctuality|
|Su|ty|_Granular_|Granularity||||
||igh quali|_Accurate_|Accuracy &<br>reliability|Accuracy; reliability|<br>Accuracy & reliability;<br>Methodological<br>soundness|Accuracy|
||H|_Comparable_|Standardization or<br>conformance||Comparability|Comparability|
||e|_Accessible_|Accessibility|Openness/<br>transparency|Accessibility|Accessibility|
||Easy to us|_Understandable_|<br>Processability and<br>understandability||Clarity;<br>Transparency|Usability/<br>interpretability|
|and||_Interoperable_|Combinability or<br>linkability|Coherence|Coherence|Coherence|
|Dem|e|_Impartial_<br>|Perception of<br>authority, impartiality<br>& trustworthiness|<br>Credibility;<br>objectivity; integrity;<br>impartiality|Impartiality &<br>objectivity|Credibility|
||Safe to us|_Confidential_|Security;<br>Protection of sensitive<br>information|<br>Confidentiality<br>protected|Statistical<br>confidentiality &<br>data security||
|||_Appropriate_|Relevance &<br>usefulness|Relevance &<br>usefulness|Relevance|Relevance|
|||Other|Quality assurance;<br>reproducibility;<br>contactability|Effective stakeholder<br>engagement|Many, see table<br>note||

Note: This figure is inspired by, and takes some information from, Marker (2017). The UN framework has many other attributes most of which are related to managing the statistical system and hence do not relate to the framework of this paper: Coordinating the national statistical system, managing relationships, managing statistical standards, professional independence, adequacy of resources, commitment to quality, appropriate statistical procedures, managing the respondent burden, and cost-effectiveness.

9
# **3. Case studies**

When the 12 attributes of the data are present, data can yield vast value for development outcomes. This section relies on case studies illustrating how each attribute can bring about positive change. Most examples involve several features either directly or indirectly, and as outlined above, having a single feature in isolation is seldom enough to bring about value.

## **Completeness**

When data cover the entire population of interest it is possible to make inferences that one otherwise would not be able to. One of the most fundamental ways in which countries can make sure their data cover their entire population of interest is through foundational identification (ID) and population registration systems, and particularly by assuring that all individuals are covered in government databases.<sup>3</sup> In Thailand, at the turn of the century, only 71 percent of the population was covered by a public health insurance scheme that was intended to be universal. Yet the country had a near-universal population register in which citizens were issued a personal identification number when they were born or when their household was registered for the first time. Leveraging this register and the personal identification information from the existing public insurance scheme, the government was able to identify the population not covered and increased health insurance from 71 percent to 95 percent (World Bank 2020a and World Bank 2018a).

In the absence of nation-wide administrative data, representative survey data fulfill a similar role. A recent example in Nigeria illustrates the power of data that are representative of the entire population of interest. The Government of Nigeria commissioned the 2015 National Water Supply and Sanitation Survey to understand access to water and sanitation of all households in Nigeria. As part of the survey, 201,842 households, 89,721 improved water points, 5,100 water schemes, and 51,551 public facilities including schools and health care facilities were enumerated. Coupling these data with advanced mapping techniques to better understand geographic variation in access to water and sanitation services revealed that 130 million Nigerians did not meet the standard for sanitation as expressed in the United Nation’s Millennium Development Goals. More specifically, the data revealed that inadequate access to water is particularly an issue for poor households, and that public expenditure in water and sanitation is limited and of poor quality (Map 1) (World Bank 2017). In response to these findings, President Muhammadu Buhari declared a state of emergency in the sector and launched the National Action Plan for the Revitalization of Nigeria's Water, Sanitation and Hygiene (WASH) Sector (Nigeria Federal Ministry of Water Resources 2018). President Buhari pledged that his administration would prioritize WASH infrastructure development, long-term planning, and stakeholder coordination. In recognition of the sector-wide crisis, the government requested from the World Bank a 700 million USD lending operation to support the sector. The overall goal of

> 3 Identification systems that use common identifiers across data sets also promote interoperability which can be a powerful tool for improving granularity through the overlay of multiple data layers.

10
<!-- Start of picture text --> . ark > we<br>a y . rs<br>9 %{ 1 ad. 8© wy Sycs<br>Y ‘ ptABUJA xoefi<br>a<br>Me<br>obs‘ osaeé d<br>» ; italiano<br>Mm 0-199<br>Gsina J Ma 20-399 a<br>cn 40-59.9<br>—_ lm 60-799<br>80-100<br>* National capital<br>=———= International boundaries<br>=== State boundaries<br>IBRD NIRAS641| MARCH 2021<br><!-- End of picture text -->
allocating resources and monitoring and evaluating social programs. Through this Directive, the index has been used to modify the allocation of resources, which helped accelerate poverty reduction during austerity without an increase in the budget (Multidimensional Poverty Peer Network 2017). To effectively use indices like the Multidimensional Poverty Index as a management tool, it is vital that it is updated frequently and that the index use data sources that can be disaggregated to the level at which the budget is allocated (see, also, discussion of granularity). This allows the index to become a tool to inform policy decisions to reduce poverty and deprivation, and not just an occasional point of information.

Similarly, indices like Subnational Doing Business – an index created by the World Bank that measures business regulations at the subnational level -- can be used to track performance if they are reported frequently. The Subnational Doing Business in Nigeria series includes four rounds of studies analyzing business regulations and their implementation across all 36 states and the capital in 2008, 2010, 2014 and most recently 2018. This has inspired several reforms and improved the overall business climate in the country. Authorities in the state of Kaduna, for example, set in motion several reform initiatives after being ranked among one of the last states in 2014 in the area of registering property. The state responded by restructuring and digitizing the land registry, replacing percentage-based registration fees with flat fees, and making the issuance of the governor’s consent—previously the biggest bottleneck in the process—much more efficient. This overhaul resulted in one of the most significant improvements recorded in the Doing Business in Nigeria series since 2008. It made one-third of procedural requirements redundant, reduced the time to register property by more than two months and cut the cost by almost one-third (World Bank 2018d).

# **Timeliness**

When data are timely, they can lead to better emergency response when disasters hit, whether environmental, financial, health, or conflict related. For example, weather data, especially weather forecasts, can help people anticipate and prepare for extreme events. The value of such data can be illustrated by two intense cyclones in the Bay of Bengal that occurred 14 years apart. The 1999 cyclone caught the Indian state of Odisha by surprise, causing massive devastation, killing more than 10,000 people, and destroying housing and public infrastructure. Since then, the Odisha State Disaster Management Authority and the government of Odisha have invested in weather forecast data and disaster response measures. When another cyclone hit in 2013, nearly 1 million people were evacuated to cyclone shelters, safe houses, and inland locations, and only 38 people died during and immediately after the storm (Hallegatte et al. 2017). These impressive results would not have possible without the weather data that gave sufficient advance warning of the cyclone.

Mobile technologies have the potential to speed up emergency response. In Uganda, an SMS-based health reporting program that provides real-time health data to beneficiaries, health professionals, and the Ministry of Health was able to halve response time to disease outbreaks.

12
The technology was used following the 2012 Ebola outbreak to help implement quarantines and other protective measures (SDSN TReNDS 2018b).

By construction, machine-generated data also have the potential to bring about quick and relevant data. One example relates to water pumps, of which one-third in rural Africa has been found to be inoperable. In Kenya, sensors on water handpumps provide real-time data on their functionality, which helped reduce the average time to repair a broken pump from 27 to 3 days and the median time from 6 days to 1 day (Figure 2) (SDSN TReNDS 2018a and Oxford/RFL 2014).

**Figure 2: Real-time sensor data on broken handpumps greatly reduced repair times in Kenya**

<!-- Start of picture text --> Median<br>1<br>repair<br>6<br>time<br>Mean<br>2.6<br>repair<br>27<br>time<br>0 5 10 15 20 25 30<br>Days<br>With mobile-enabled data Baseline<br><!-- End of picture text --> _Source_ : Oxford/RFL 2014 Satellite data can likewise offer a timely response to emerging threats. In Sri Lanka, where more than two million hectares of natural forest are home to over 30 endangered mammals, satellite data have been used to safely and quickly respond to deforestation of protected areas (Jamilla and Ruiz, 2020). The Department of Forest Conservation used to rely mostly on routine patrolling, which can be a strain on resources and be a dangerous task due to the terrain and animals that may be encountered by patrollers. Propelled by COVID-19, which restricted the use of patrolling, the department now relies on a mobile app provided by the Global Forest Watch. The app relays real-time satellite imagery to alert users of possible deforestation and provide evidence when deforestation has occurred. In one case, a popular local TV channel reported that encroachments had taken place without being addressed by the department upon which the department used the satellite data and alerts to identify two locations of encroachment. As a result of this, legal actions were taken to stop the encroachments from expanding.

# **Granularity**

Data can be granular along a number of dimensions, for example granular by space, granular by time, or granular by demographic attribute, such as gender. A long-running and rich example of the value of granular data by gender is the Demographic and Health Surveys, which cover topics 13
<!-- Start of picture text --> SLOVENIA a saa HUNGARY<br>eaeatsie eetlg, P= “SseRBiA<br>ae .ae hac § Teles: Ra<br>S/ OF 6 RAO Acl. | As Sa os<br>a<br>Se “es TT Res “9<br>Fy 3<br>* SSSS NST BOSNIA AND<br>YS ey HERZEGOVINA<br>Nip Sia ~<br>~ % of population SSae<br>ivinginpoverty = ><br>mmmm 1-213-21 *eieSS<br>2-31 Rae ns<br>Ha 32-42 - ,~~ Sys yee— Ay Se’ ; MONTENEGRO<br>TALY, 43-71 ocigh+ ee<br>KK National capital .<br>>), InternationalMunicpality boundari eses 7 ti |<br>IBRDCROASAO]j MARCH 2071<br><!-- End of picture text -->
<!-- Start of picture text --> 100%<br>90%<br>80%<br>70% q<br>60%<br>50%<br>40%<br>30%<br>20%<br>10%<br>o% + =<br>04.00 06.00 08.00 10.00 12.00 14.00 16.00 18.00 20.00 22.00 24.00 02.00<br>Hours<br><!-- End of picture text --> <!-- Start of picture text --> 100%<br>90%<br>80% Freettimel<br>70%<br>60%<br>50%<br>40%<br>30%<br>20%<br>10%<br>0%<br>04.00 06.00 08.00 10.00 12.00 14.00 16.00 18.00 20.00 22.00 24.00 02.00<br>Hours<br><!-- End of picture text -->
# **Map 3: The use of citizen-collected data in Bogotá led to greater safety around bike paths**

_Source_ : Safetipin 2016. _Note:_ Ratings of poor to excellent are based on safety scores.

In the Middle East and North Africa, basic statistics such as unemployment rates are measured with imprecise definitions, blurring the line between unemployment and informality which distort the role of women and rural areas in assessments of national employment. Small changes in the definitions have large implications on the quantity of interest (Arezki et al. 2020).

In 2010, Ghana officially revised its GDP figures upwards by about 60 percent. This change suggested that Ghana’s economic performance had been significantly underestimated for years. The revision was the result of moving from the 1968 System of National Accounts guidelines to the 1993 vintage – as well as accounting more accurately for emerging sectors of the Ghanaian economy (Jerven and Duncan, 2012). Similar upwards revisions occurred in other African nations, including Nigeria, Senegal, Kenya, and Zimbabwe, among others (Koumane et al., 2019). While these revisions have no real direct effect on improving well-being, they do improve cross-country comparability which allows countries to better monitor relative performance of each country which can then inform macroeconomic policies which may indirectly improve well-being.

16
# **Comparability**

Cross-country comparable data sets that track certain indicators over time—often created by think tanks, research institutions, and international organizations— allow countries to benchmark their performance against peers, which they can use to evaluate national policies and assess national priorities. Countries often respond with reforms in areas where they are lagging.

As one example, the Democratic Republic of Congo (DRC) made gender equity reforms upon seeing data from the Women, Business and the Law Index – an index created by the World Bank to compare laws and regulations affecting women’s economic opportunity across economies. The reform effort, supported by DRC’s ministries of Gender and Justice, led to changes the DRC’s Family Code, which for decades contained legal provisions that prevented married women from carrying out economic activities. The adoption of a new Family Code in July 2016 allowed married female entrepreneurs in DRC to start formal businesses, open bank accounts, register a company, and perform a host of other economic activities without interference from their husbands. The law also allows them to have a greater voice in management of the marital property and raised the legal marriage age for girls from 15 to 18.

In Sri Lanka, policy proposals were made upon seeing a provincial breakdown of the Human Capital Index (HCI) – an index created by the World Bank to measure the amount of human capital a child born today can expect to attain by age 18. Though Sri Lanka performs better than its neighbors in terms of human capital, the provincial breakdown confirmed high levels of stunting in the less developed provinces. This contributed to the Manifesto of the President proposing a special child nutrition program to address child under-nutrition, a nutrition allowance for children from poor families, and a nutrition aid program for pregnant women in the Estate Sector.

# **Accessibility**

When governments make data widely available, they empower individuals to make better choices through more information and knowledge. The digital revolution has increased the potential and ease with which information can be shared. One example is providing agricultural information, often through text messages, to smallholder farmers to increase their productivity. Such data transmission can improve existing extension services, which rely on in-person agricultural advice and are more costly to sustain and whose quality is more difficult to assure. A meta-analysis suggests that providing agricultural information increases yields by 4 percent and also increases farmers’ probability of using productivity-enhancing agrochemical inputs (Fabregas et al. 2019). With more than 2 billion people living on smallholder farms, these numbers can have major consequences for global poverty and shared prosperity.

Likewise, providing real-time price data to farmers allows them to sell their products at the right time and place. In Ethiopia, small-scale farmers lack access to reliable price information, and thus often receive prices far below market value. In 2008 the Ethiopian Commodity Exchange opened, providing price information to farmers through text messages, hotlines, and online 17
information. As a result, the farmers have been able to cut trader margins by half and increase their revenue (Vaitla et al. 2015c).

When governments make their data available in a machine-readable format, the possible uses of the data expand, and correspondingly, so do their value. A recent policy evaluation looked at the impact of a Senegalese law reducing the length of pre-trials on the quality of decisions. The law was implemented to increase court efficiency, which has been shown to matter for economic development. One could fear that such an efficiency reform would deteriorate the quality of the judgments. Through a laborious effort of digitalizing case-level data from the court, the impact evaluation found this not to be the case and that firms benefitted from the faster process times, suggesting that the reform indeed worked as intended (Kondylis and Stein 2018). In this way, making data machine-readable could assure that past policy changes were appropriate.

Because of these benefits, the signaling value of being a country that makes data widely available can help countries reduce their sovereign bond spreads, decrease their external debt burden, and foster growth. If African countries could raise their level of data transparency to the level of the top quartile of upper middle-income countries, the region as a whole would save 4.87 basis points in sovereign bond spreads and decrease their external debt by more than 400 million dollars (Kubato and Zeufack 2020). Another study finds that <mark>data transparency has a positive and sizable effect on real gross domestic product per capita (Islam and Lederman 2020)</mark> . These findings and the examples above reveal that collecting and disseminating data should not be viewed as a drain on public finances.

# **Understandability**

Data accessibility is often not enough to ensure that data are used by governments or individuals when the data are difficult to understand or there is a lack of skills to understand how to use the data. One example where data only attained its value when it was understood properly relates to public procurement. Too often, public projects are not implemented adequately due to poor procurement, such as inflated costs, corruption, or ghost contracts. Since 12 percent of global GDP is spent on public procurement, this matters tremendously for development outcomes (Bosio and Djankov 2020). In Uganda, in an attempt to improve procurement outcomes, local government entities made administrative procurement data from the bidding process down to the execution of contracts available to certain Civil Society Organizations (CSOs). Such data are not easily comprehendible to a non-trained audience, so the CSOs trained community members to understand the information in the contracts and conduct site checks to verify it. The findings revealed mismanagement of resources by contractors and government officials and high dependence on noncompetitive contracts. Aside from the direct benefits of assuring that contracts complied with national procurement standards, the national public procurement agency upgraded its procurement portal in line with international open contracting data standards, making Uganda the first African country to do so (Africa Freedom of Information Centre 2018 and Global Partnership for Social Accountability 2020).

18
Other times, it is public officials that may lack the skills or time to summarize the data to a shape from which value can be generated. In Brazil, for instance, evidence from 2,150 municipalities found that informing municipal mayors of research findings on the effectiveness of a simple policy change increased the probability that their municipality implemented the policy by 10 percentage points (Hjort et al. 2021).

Another example, where making the data more understandable helped guide policy choices, comes from Pakistan. Prior to 2008, Pakistan’s Punjab province suffered from poor government service delivery due to inefficiencies, rent seeking and more. Lack of digitized service delivery processes made it impossible to track service delivery and monitor performance and satisfaction with services. In an attempt to take on these challenges, in 2008, officials in one district of Punjab put in place a pilot Citizen Feedback Monitoring Program (CFMP), in which data on service provision from citizens was crowd-sourced using simple text messaging and other information and communications technology (ICT) applications. Analytical reports were sent to government officials enabling them to identify patterns and take evidence-based corrective measures. In 2012, the initiative was scaled up to 36 districts of the province and across 25 different public services. To maximize feedback collection and minimize cost and data literacy issues, in 2016 the CFMP introduced a robocall method to gather the data. As of 2019, CFMP had contacted 29 million citizens to solicit their feedback, and the government of Punjab had taken 41,600 corrective measures in response to CFMP data, including warnings, penalties and suspension. One of many results of this is that the availability of medicine increased from 46 percent to 77 percent (Global Delivery Initiative 2019). In a 2015 evaluation of the program, 90 percent of respondents said it had helped build trust between citizens and authorities (Masud 2015).

# **Interoperability**

When data sources are interoperable the potential use and value of data expand. As one example, interoperability can help governments prioritize scarce resources by cutting costs such as by eliminating duplicate or ghost recipients of social transfers—beneficiaries, often of pension funds, who are no longer alive. This was the case in Argentina where the government identified noneligible beneficiaries across various social programs using the country’s system of unique taxpayer ID numbers to link data sets, generating an estimated savings of $143 million over eight years (World Bank 2020a and World Bank 2018b).

Interoperability can also help improve targeting. In Jordan, the Profile Global Registration System (ProGres), the main database used by the UNHCR to register refugees, has traditionally been used to target cash assistance to refugees outside camps. This database contains key socioeconomic characteristics on almost all refugees since most register with UNCHR at some point in time. ProGres does not contain information on expenditure of households, though, preventing transfers to be targeted explicitly to poor households. Combining ProGres with the Jordanian Home Visits database, which contains only a subset of the population of interest, but 19
enough information to create a targeting tool based on expenditures of households, the World Bank and UNCHR created a targeting algorithm that mapped directly to all individuals in the ProGres data (Verme et al. 2015). This algorithm revealed that UNHCR’s targeting efficiency could be improved markedly. As a result, this targeting model has been embedded into the UNHCR operational portal and used to rank refugee households for targeting cash assistance. For humanitarian organizations, this has been a paradigm shift that revolutionized targeted programs in North Africa and the Middle East and beyond, and the tool has been applied to displaced populations in Egypt, Lebanon by the UNCHR and the WFP.

Interoperability can also induce cost savings for the private sector, particularly data containing key national identifiers of companies, individuals, geographical units, and other entities, which allow for easy linking with the company’s own data. National ID systems, for example, can increase private sector efficiency through cost savings such as removing the need for companies to create their own ID systems, and increased revenues by expanding the potential customer base (World Bank 2020a and World Bank 2018c). In India, Aadhar, the unique 12-digit identity number that can be obtained by residents and citizens of India, has reduced the need for identity verification for firms. As a result, it is estimated that a firm’s typical onboarding costs for new employees could decrease from 1,500 rupees to 10 rupees – a reduction of 99.3% (World Bank 2020a and World Bank 2018c).

# **Impartiality**

When data are not impartial, their value falls apart due to potential misuse. Lack of data impartiality can seriously affect countries’ credibility and creditworthiness. For example, in 2007 the Argentine government began interfering with the consumer price index and other official statistics. This led to inaccurate and non-credible reported inflation figures (Todesca 2017). Similarly, Greece’s debt statistics appear to have deliberately misrepresented the country’s financial situation in the lead-up to the 2009-euro crisis (Katsimi and Moutos 2010). In both cases, the deliberate manipulation of statistics was the precursor to larger crises, revelations and fiscal austerity measures that triggered social unrest and had significant social welfare implications.

Assuring that data cannot be manipulated by producers can have positive consequences on government budgets. India saved 19 percent on a rural employment guarantee scheme—the world’s largest workfare program—after introducing e-governance to the payments system. Most of the savings were incurred because local officials were no longer tasked with distributing the funds and thus could not mismanage them and misreport the data. The change decreased officials’ personal wealth by 10 percent (Banerjee et al. forthcoming).

In some cases when data are not impartial, third-party sources may notice inconsistencies. In China, for instance, media and watchdog organizations in Beijing noted inconsistencies between official government data on air quality and data from independent air quality monitoring systems. Heightened concerns about air quality fueled a dramatic expansion in publicly available, real-time data from thousands of air quality monitoring locations (Yin et al.

20
2020). The central government launched a US$275 billion plan to improve air quality throughout the country, and the Beijing municipal government promised an additional US$160 billion toward that goal (Vaitla et al. 2015b).

# **Confidentiality**

When data are not confidential, subjects may be identifiable. This violates core principles of privacy and increases the likelihood that harm can be done to the identified subjects. Deidentifying individuals is not always enough to maintain confidentiality. In the 1990s, for example, the Governor of Massachusetts approved making de-identified medical records of state employees available for researchers. Though the data had key identifiers such as name and addresses removed, by triangulating the information available with other public information, a researcher was able to identify the medical records of the governor. Other individuals could likewise be identified (Heffetz and Ligett 2014).

Beyond re-identification, data breaches also pose a threat to confidentiality because they raise doubts as to whether personal information is safe. Aadhar, the unique 12-digit identity number of India mentioned before, has suffered from several data leaks. In one instance, more than 200 government websites accidentally made personal data including demographic data, names, phone numbers, religion, bank account numbers and more, available publicly on the internet.<sup>5</sup> Though the data were quickly removed, such leaks often have permanent implications on confidence in data systems.

# **Appropriateness**

For data to generate value, they need to be generated and used to serve a legitimate policy purpose. This means the data must contain information relevant to the policy problem at hand. Critically, this also means that the amount of information collected and used should not be excessive but appropriate for the particular needs. As such, appropriateness is key to preventing data misuse, such as surveillance or exclusion.

Governments’ tracking and contact tracing of infected and at-risk individuals through smartphone apps or phone location data during the COVID-19 pandemic shed light on the relevance of appropriateness. Some of these tracking efforts have been large-scale, data, and technology enabled extensions of manual contact tracing, with the aim to identify and isolate possibly contagious individuals to contain the spread of the virus. In principle, tracking and contact tracing measures thus serve an important policy function (though their effectiveness is still subject to debate (Almagor and Picascia (2020); Fraser et al. 2020)). Yet, the risk of surveillance and other kinds of overreach are high. To minimize the risk of surveillance beyond what is needed to minimize the spread of the virus, it is imperative that only data strictly necessary to carry out contact tracing are collected.

> 5 <u>https://www.firstpost.com/tech/news-analysis/aadhaar-security-breaches-here-are-the-major-untowardincidents-that-have-happened-with-aadhaar-and-what-was-actually-affected-4300349.html.</u>

21
One example of how to accomplish this, which has been taken up by some countries, is to rely on a protocol developed by Apple and Google for their contact tracing apps. With this protocol, phones in close proximity of each other will use low energy Bluetooth technology to exchange a temporary key code, which changes every 15 minutes. This information is stored on each phone, rather than in a centralized database. When a user reports a positive test result, the temporary key codes are used to notify other users of their potential exposure. (Zastrow, 2020). This solution serves the intended purpose of notifying users that may have been exposed to the virus, while being relatively privacy-preserving. It cannot be used to enforce quarantine, track, or identify individuals, limiting the scope for surveillance and other inappropriate uses (Privacy International 2020).

In contrast, some countries have taken a more expansive approach to contact tracing, in which more sensitive information was collected and used and, in some instances, shared with law enforcement agencies. In Israel, mobile phone location data was used by the domestic security agency to identify individuals exposed to the virus and by the police to enforce quarantine, a practice which was subsequently challenged as unconstitutional in the Supreme Court (Amit et al. 2020; Bradford et al. 2020). India’s contact tracing app stores location data alongside a set of demographic information, including age, gender, phone number, and travel history, on a centralized server. It was initially mandatory for air and rail travel as well as for government employees to return to work, raising fears this would effectively exclude those without a smartphone from these activities (Arun 2020; Bradford et al. 2020; Privacy International 2020).

# **4. Conclusion**

Around the world, governments are producing more and more data, yet it is highly doubtful that the full potential value of the data is being reaped. In this paper, we have presented a theoretical and empirical argument for why the returns to existing government data might not be maximized. We show that for data to yield returns for governments they must live up to certain conditions, these being that they are of adequate spatial and temporal coverage (complete, frequent, and timely), of high quality (accurate, comparable, and granular), easy to use (accessible, understandable, and interoperable) and safe to use (impartial, confidential, and appropriate). Through case studies covering a wide range of countries and topics, we illustrate how these 12 features matter in practice for generating value from data.

Too often governments are concerned with producing a particular data set, with less of an eye to how those data will generate value. It is imperative for governments not only to invest in more data, but also the right data, and working towards the safe use of data. Often the value derived from data is determined by the features least present, emphasizing the need for a comprehensive analysis of which of the 12 conditions elicited in this paper may be lacking. This can ensure that the whole chain from data production to data use is in place and the data can come closer to realizing its maximum value for development.

22