metadata: authors: 0: author: Stacy,Brian William
1: author: Kitzmüller,Lucas
2: author: Wang,Xiaoyu
3: author: Mahler,Daniel Gerszon
4: author: Serajuddin,Umar
count: World
docna: 0: docna: Missing Evidence : Tracking Academic Data
            Use around the World
docty: Policy Research Working Paper
abstracts: cdata!: Data-driven research on a country is key
            to producing evidence-based public policies. Yet little is
            known about where data-driven research is lacking and how it
            could be expanded. This paper proposes a method for tracking
            academic data use by country of subject, applying natural
            language processing to open-access research papers. The
            model&#x2019;s predictions produce country estimates of
            the number of articles using data that are highly correlated
            with a human-coded approach, with a correlation of 0.99.
            Analyzing more than 1 million academic articles, the paper
            finds that the number of articles on a country is strongly
            correlated with its gross domestic product per capita,
            population, and the quality of its national statistical
            system. The paper identifies data sources that are strongly
            associated with data-driven research and finds that
            availability of subnational data appears to be particularly
            important. Finally, the paper classifies countries into
            groups based on whether they could most benefit from
            increasing their supply of or demand for data. The findings
            show that the former applies to many low- and
            lower-middle-income countries, while the latter applies to
            many upper-middle- and high-income countries.
display_title: Missing Evidence : Tracking Academic
            Data Use around the World
listing_relative_url: /research/2024/01/34240433/missing-evidence-tracking-academic-data-use-around-world
topicv3: Public Funded Research and Development
            m1504101 2787,Statistical Surveys m1504545 3123
publishtoextweb_dt: 2024-01-17T00:00:00Z
dois: 10.1596/1813-9450-10673
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 39
author_keys: 0: author_key: 1312545
1: author_key: NA
2: author_key: 2172451
3: author_key: 1539953
4: author_key: 539942
model_extractions: Policy Research Working Paper 10673 Missing Evidence Tracking Academic Data Use around the World _Brian Stacy Lucas Kitzmüller Xiaoyu Wang Daniel Gerszon Mahler Umar Serajuddin_ Development Economics Development Data Group January 2024 A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.
Policy Research Working Paper 10673

# **Abstract**

Data-driven research on a country is key to producing evidence-based public policies. Yet little is known about where data-driven research is lacking and how it could be expanded. This paper proposes a method for tracking academic data use by country of subject, applying natural language processing to open-access research papers. The model’s predictions produce country estimates of the number of articles using data that are highly correlated with a human-coded approach, with a correlation of 0.99. Analyzing more than 1 million academic articles, the paper finds that the number of articles on a country is strongly correlated with its gross domestic product per capita, population, and the quality of its national statistical system. The paper identifies data sources that are strongly associated with data-driven research and finds that availability of subnational data appears to be particularly important. Finally, the paper classifies countries into groups based on whether they could most benefit from increasing their supply of or demand for data. The findings show that the former applies to many low- and lower-middle-income countries, while the latter applies to many upper-middle- and high-income countries.

This paper is a product of the Development Data Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at bstacy@worldbank.org. A verified reproducibility package for this paper is available at http://reproducibility. worldbank.org, click **here** for direct access.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Missing Evidence: Tracking Academic Data Use around the World**

Brian Stacy, Lucas Kitzmüller, Xiaoyu Wang, Daniel Gerszon Mahler, and Umar Serajuddin<sup>1</sup> Keywords: Data, academia, research, natural language processing JEL codes: C45, C52, O30

> 1 Stacy, Wang, Mahler, and Serajuddin are with the World Bank’s Development Data Group.  Lucas Kitzmüller completed the work while at the European Bank for Reconstruction and Development (EBRD).  Corresponding author: Brian Stacy (bstacy@worldbank.org). We are grateful for comments from Dean Jolliffe, Jishnu Das, Olivier Dupriez, and Patrick Brock. We acknowledge financial support from a World Bank Research Support Grant (P178728). The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Views presented are those of the authors and not necessarily of the EBRD.
# **1 Introduction**

In recent decades, the amount of data produced has exploded, generating boundless opportunities for policies to improve people’s lives (World Bank 2021). Though data can be valuable in their raw form, the full value of data is only realized when they are analyzed to create insights, and these insights are converted to public policies or increased accountability.

Researchers have a vital role to play in this regard. Many researchers spend countless hours digesting data, using data to create new knowledge, and communicating this knowledge with the intent of impacting public discourse and public policies. There are numerous examples of data-driven analyses having real and important impacts on people’s lives (Jolliffe et al. 2023). One example from Brazil explicitly looks at researchers’ ability to influence policy outcomes. There, evidence from 2,150 municipalities found that informing municipal mayors of research findings on the effectiveness of a simple policy change increased the probability that their municipality implemented the policy by 10 percentage points (Hjort et al. 2021).

Without research, there is a risk that the return of data to society will be reduced, and policies to improve lives unrealized. Yet very little is known about where there is missing data-driven evidence and how governments can best stimulate an evidence base for local decision makers. This paper attempts to fill these gaps by addressing two questions: (1) Which countries are the subject of research papers using data? (2) How can countries increase their national evidence base? We focus on data-driven research due to the increasing importance of data for policy making and the specific policies that are needed to increase the supply and demand of data, such as boosting statistical capacity and improving data literacy.

To answer the first question, we introduce a new method for measuring data use in research articles based on 1 million English-language articles spanning 216 countries and various academic fields. These articles are made available by Semantic Scholar Open Research Corpus (s2orc), which has digitized millions of research papers worldwide and made its raw text accessible via APIs (Lo et al. 2020). With the aid of Amazon Mturk workers, we manually code 900 of these articles as using data or not, upon which we train a natural language model to predict the coding of the Mturk workers (Devlin et al. 2018). The model achieved an 87% out-of-sample accuracy rate, and when the articles are aggregated to the country level, the model had a correlation of 0.99 with the number of articles classified by Mturk workers. The model was then applied to 1 million academic articles from the s2orc database from 2000-2020. The model estimates the amount of data-driven research on a country regardless of where the researchers may be located, _not_ the amount of data-driven research by the citizens of a country. We argue that the former is the relevant quantity to understand the evidence base available to national decision makers.

We find that data-driven research is strongly correlated with GDP per capita and population, which together account for around 75% of the variation across countries. High-income countries are the subject of nearly 50% of all papers using data despite representing only around 15% of the world's population, while low-income countries, comprising approximately 10% of the world population, account for only about 5% of articles using data.

2
To answer the second question – how countries can increase their national evidence base – we first establish that a country’s statistical capacity is predictive of data research even after controlling for population and GDP, and articles not using data (which we use as a proxy for the general research interest in the country). To understand which part of a country’s statistical capacity is most important for increasing data-driven research, we explore the data sources most related to academic data use. We find that the availability of geospatial data at the first administrative level is associated with a 1.1% increase in data use, that a population census in the past decade is associated with a 0.3% increase in data use, and that two or more labor force (agriculture) surveys over the past 10 years is associated with a 0.4% (0.2%) increase in data use. Though we are unable to establish these links causally, there are concrete data products that governments could supply to likely increase the evidence-base at its disposal.

Boosting the supply of data is one way for a country to increase the data-driven research it is subject to, another is to increase the demand for its data. This is particularly relevant for countries that have already invested in relevant data products but are nonetheless subject to relatively little data-driven research. These are cases where existing data are underutilized and where it may be relevant to make existing data more accessible to researchers and possibly boost data literacy in the country. To explore this distinction between boosting data supply and data demand, and building upon Porteous (2020), we classify countries into four groups: _Deserts_ have little data demand and little data supply, _swamps_ have high data supply but little data demand, _oases_ have high data demand but little data supply, and _lakes_ have high data demand and high data supply. Nearly two-thirds of low-income countries and countries in Sub-Saharan Africa are oases, suggesting that these countries are getting relatively large evidence from their data supply and that they have relatively little issue with lack of demand, but that they could benefit from increasing the data available to researchers. By contrast, nearly half of countries in Europe are data swamps, suggesting a priority on increasing the use of existing data.

Previous research has highlighted gaps between countries in economic research output and noted that richer countries are the subject of more economic research. For instance, Robinson, Hartley, and Schneider (2006), Das et al. (2013), and Porteous (2020) examine which countries are studied most by economists using the EconLit database. Cameron, Mishra, and Brown (2016) and Sabet and Brown (2018) extend this to note that impact evaluations are highly uneven across countries as well. Phillips and Greene (2022) show conflict research is biased towards Western countries, while Courtioux et al. (2022) show that academic research is highly related to public investments in scientific research.

We contribute to the literature by applying NLP to improve our understanding of what countries are under-researched. Using NLP allows us to go beyond the existing literature in three ways: (1) scale up the sample size and look at all fields of interest rather than only economics, (2) identify papers that use data, which is key to understanding whether data demand or data supply could be explaining a country’s lack of research, and (3) indicate measures countries can take to increase data research.

3
The remainder of the paper is structured as follows. Section 2 discusses our data sources, section 3 details our methodology, section 4 introduces a theoretical framework, section 5 presents our empirical results, section 6 conducts robustness checks, and section 7 concludes.

# **2 Data**

Our data source of academic articles is the Semantic Scholar Open Research Corpus (S2ORC) (Lo et al. 2020). The corpus contains more than 130 million English language academic papers across multiple disciplines. The papers included in the Semantic Scholar corpus are gathered directly from publishers, from open archives such as arXiv or PubMed, and crawled from the internet.

We placed some restrictions on the articles to make them usable and relevant for our purposes. First, only articles with an abstract and parsed PDF or latex file are included in the analysis. The full text of the abstract is necessary to classify the country of study and whether the article uses data. The parsed PDF and latex file are important for extracting important information like the date of publication and field of study. This restriction eliminated a large number of articles in the original corpus. Around 30 million articles remain after keeping only articles with a parsable (i.e., suitable for digital processing) PDF, and around 26% of those 30 million are eliminated when removing articles without an abstract. Second, only articles from the years 2000 to 2020 were considered. This restriction eliminated an additional 9% of the remaining articles. Finally, articles from the following fields of study were excluded, as we aim to focus on fields that are likely to use data produced by countries’ national statistical system: Biology, Chemistry, Engineering, Physics, Materials Science, Environmental Science, Geology, History, Philosophy, Math, Computer Science, and Art. Fields that _are_ included are: Economics, Political Science, Business, Sociology, Medicine, and Psychology. This third restriction eliminated around 34% of the remaining articles. From an initial corpus of 136 million articles, this resulted in a final corpus of around 10 million articles.

Due to the intensive computer resources required, a set of 1,037,748 articles were randomly selected from the 10 million articles in our restricted corpus as a convenience sample. Summary statistics of the final sample of 1 million articles are available in Table 1.

_Table 1. Summary Statistics of Article Corpus. 2000-2020_

|Field|Published in Journal<br>(1=yes)|Data Use<br>(1=yes)|Country Identified<br>(1=yes)|Articles|Share of<br>Articles|
|---|---|---|---|---|---|
|Business|0.56|0.64|0.30|28,571|2.8|
|Economics|0.79|0.68|0.28|62,241|6.0|
|Medicine|0.96|0.85|0.10|840,920|81.0|
|Political Science|0.42|0.33|0.34|26,185|2.5|
|Psychology|0.75|0.70|0.14|44,191|4.3|
|Sociology|0.90|0.33|0.25|35,640|3.4|

4
# **3 Empirical Strategy**

The empirical approach employed in this project utilizes text mining with natural language processing (NLP). The goal of NLP is to extract structured information from raw, unstructured text. In this project, NLP is used to extract the country of study and whether the paper makes use of data. We will discuss each of these in turn.

To determine the country or countries of study in each academic article, two approaches are employed based on information found in the title, abstract, or topic fields. The first approach uses regular expression searches based on the presence of ISO3166 country names. A defined set of country names is compiled, and the presence of these names is checked in the relevant fields. This approach is transparent, widely used in social science research, and easily extended to other languages. However, there is a potential for exclusion errors if a country’s name is spelled nonstandardly.

The second approach is based on Named Entity Recognition (NER), which uses machine learning to identify objects from text, utilizing the spaCy Python library. The Named Entity Recognition algorithm splits text into named entities, and NER is used in this project to identify countries of study in the academic articles. SpaCy supports multiple languages and has been trained on multiple spellings of countries, overcoming some of the limitations of the regular expression approach. If a country is identified by either the regular expression search or NER, it is linked to the article. Note that one article can be linked to more than one country.

The second task is to classify whether the paper uses data. A supervised machine learning approach is employed, where 3,500 publications were first randomly selected and manually labeled by human raters using the Mechanical Turk service (Paszke et al. 2019).<sup>2</sup> To make sure the human raters had a similar and appropriate definition of data in mind, they were given the following instructions before seeing their first paper: _<mark>Each of these documents is an academic article. The goal of this study is to measure whether a specific academic article is using data and from which country the data came.</mark>_ _<mark>There are two classification tasks in this exercise:</mark>_ _1._ _<mark>Identifying whether an academic article is using data from any country</mark>_ _2._ _<mark>Identifying from which country that data came.</mark>_ _<mark>For task 1, we are looking specifically at the use of data. Data is any information that has been collected, observed, generated or created to produce research findings. As an example, a study that reports findings or analysis using a survey data, uses data. Some clues to</mark>_

> 2 A stratified sample of articles for the training set was taken. Of the articles 15% were medical, 15% psychology, 25% economics, 25% political science, 10% were tagged as business, and 10% sociology articles.

5
_<mark>indicate that a study does use data includes whether a survey or census is described, a statistical model estimated, or a table or means or summary statistics is reported.</mark>_ _<mark>After an article is classified as using data, please note the type of data used. The options are population or business census, survey data, administrative data, geospatial data, private sector data, and other data. If no data is used, then mark "Not applicable". In cases where multiple data types are used, please click multiple options.</mark>_<sup>_3_</sup> _<mark>For task 2, we are looking at the country or countries that are studied in the article. In some cases, no country may be applicable. For instance, if the research is theoretical and has no specific country application. In some cases, the research article may involve multiple countries. In these cases, select all countries that are discussed in the paper</mark>_ <mark>.</mark>

# _<mark>We expect between 10 and 35 percent of all articles to use data.</mark>_

An image of the screen facing the MTurk workers when classifying an article is presented in Figure A.1 in the Appendix. The median amount of time that a worker spent on an article, measured as the time between when the article was accepted to be classified by the worker and when the classification was submitted was 25.4 minutes. If human raters were exclusively used rather than machine learning tools, then the corpus of 1,037,748 articles examined in this study would take around 50 years of human work time to review at a cost of $3,113,244, which assumes a cost of $3 per article as was paid to MTurk workers.

A model is next trained on the 3,500 labeled articles. We use a distilled version of the BERT (bidirectional Encoder Representations for transformers) model to encode raw text into a numeric format suitable for predictions (Devlin et al. (2018)). BERT is pre-trained on a large corpus comprising the Toronto Book Corpus and Wikipedia. The distilled version (DistilBERT) is a compressed model that is 60% the size of BERT and retains 97% of the language understanding capabilities and is 60% faster (Sanh, Debut, Chaumond, Wolf 2019). We use PyTorch to produce a model to classify articles based on the labeled data. Of the 3,500 articles that were hand coded by the MTurk workers, 900 are fed to the machine learning model. Nine hundred articles were selected because of computational limitations in training the NLP model. A classification of “uses data” was assigned if the model predicted an article used data with at least 90% confidence.

The performance of the models classifying articles to countries and as using data or not can be compared to the classification by the human raters. We consider the human raters as giving us the ground truth. This may underestimate the model performance if the workers at times got the allocation wrong in a way that would not apply to the model. For instance, a human rater could mistake the Republic of Korea for the Democratic People’s Republic of Korea. If both humans and the model perform the same kind of errors, then the performance reported here will be overestimated.

> 3 We ended up not using this in the paper.

6
<!-- Start of picture text --> United States<br>00 y=-0.3+ 13x, R2=0.996<br>(02) (00)<br>i<br>5 India United im<br>& Canada alia<br>2.<br>o8 XY 5<br>° Tance<br>5 10 be<br>= Mexico<br>Syrian Arab Republic Port- ort u 2egc a ll Pfe)* °¢ ° dan<br>United Arab Emirates-Libya . Korea, Rep.<br>Slovenia-Somalia Chile<br>Sri Lani ° °<br>Lebanon<br>4 » °Nigerig<br>1 10 100<br>Machine Learning Classification<br><!-- End of picture text -->
<!-- Start of picture text --> fGapetglue<br>em N\\ Au VAN) Ay WAAAY YY \ ltd 7 Hi wi 93 ___<br>This 6888). which is part of a sfmiposiumledition of the Akron Law Review, explores the practice of metadata<br>mining (i.e., the surfeptitious examination of an electronic documenta€TMs hidden a. The essay (EES<br>the (B@BBethics Spiniohs that have addressed the practice and contends that metadatalmining is simply a<br>variation ofbans the oft-QEB problem of inadvertently disclosed documents. The essay concludes thabifiat<br>inadvertent on meta@ata}miningdisclosures moreare misguided generally. andUnder that this the approach,issue shouldif a statebe treated permitsin thelawyers same to manner FEVIEW as<br>inadvertently disclosed privileged documents, the jurisdiction should also permit lawyers to review the<br>meta@iataicontained in electronic documents. In contrast, if a jurisdiction prohibits the feviewlof misdirected<br>privileged documents, the state should ban metadata mining, but only when recipients have reason to<br>believe that the metadata contains protected information.<br><!-- End of picture text --> <!-- Start of picture text --> ee ees inout " "<br>Objectivasifelestablish 5(ii@and reliable translated version of Smartphone Addiction Scale for adolescent<br>[SAS-SV) in Italian languages is a 10-item) monodimensional, self-administered questionnaire, Which Was<br>validated using 2 large Korean SSmple§Methods:We reported the linguistic validation process and the<br>youngmetric adults\frecruited validity ofthe Italian from version High Public of SAS-SV School in the ItalyTheiid)UniversityEBiapIBin Italy,consistedwith a Meanlageof 633 adolescentsof 18 years{iThe<br>validation process consisted in forward-backward translation {factorstructure testing)Results: Results<br>(BRIER: g000 FBliability (Cronbachs alpha = .75) Sian almost acceptable factor structuresConclusion: A<br>(lid)3nd) reliable Smartphone Addiction Scale in Sho Version Was developed which can be used for<br>fé5earch)and clinical assessment Ofadolescentsiandyoung adults with smartphone addiction|<br><!-- End of picture text -->

<!-- Start of picture text --> —04 yy sat — baseoot value 05 0.673803f(inputs) 28 it 14<br>Sy»)) inputsscram —_ ((<br>The article discusses theoretical aspects of integration approach to the {of the development of the<br>agrarian sphere of economy. The article presents authors who have made significant scientific contributions<br>to the theory of integration; identifies structural elements of the national agro-industrial complex:<br>determines the value of scientific and technological progress in the development and integration of<br>productive forces; identifies factors affecting the development of the agrarian sphere as a complex<br>integrated system. The article notes that integration as a system possesses characteristics of self-<br>organization, hierarchy, striving for equilibrium, and has the presence of a common goal. The article<br>particularly characterizes the socio-economic factors of integration, influencing the agricultural sphere<br>development and management. Particular emphasis is placed on the problem of integration of Russian<br>agricultural industries in the WTO.<br><!-- End of picture text -->
|J=BIN<br>)J-C<br>)Vij<br>()|
|---|
<!-- Start of picture text --> 4 0°". |<br>a aS<br>Papers \<br>= 10 ’ Ud PA<br><!-- End of picture text -->
<!-- Start of picture text --> High income 71,815<br>_— |“<br>Low income 5,517<br>0 25,000 50,000 75,000<br>Number of Papers using Data<br><!-- End of picture text -->

<!-- Start of picture text --> ed<br>Suope ScenaAS Ls<br>ee (iii)i<br>a Lda<br>‘ate Ares & careoean |TM<br>Sub-Saharan Africa | 143.1<br>sauA -<br>0 100 200 300 400 500<br>Number of Papers using Data per capita<br><!-- End of picture text -->
<!-- Start of picture text --> CHN ° USA<br>1,000 NGA, IND . IRN-2—$°IDN ,BRAZAF mys, ITASCAN®JPN ¢>-DEU Lp.AUS<br>g ETH PAK THAN MEX wTUR ee ee NOR<br>8 UGA KEN*® “3BGD VNM cot * RUS FOL. SAU a<br>2 RWA 1A" NPLeGHA JOR\PHL *~ £GY KA CHL RC * BEND a cp<br>53 MMR MAR PER, o.° * PRTQs %e<br>2 100 NER MW! E . ZMB eSDN YKHM NRQ—* . * IKR_GEOARG ROU CHE<br>goO con—+ yo, AFG SEN * ° gO gL AQUTUNSECU oee 5 SRN RV agree...HUN CZEt1St, OAT<br>& MOZ“Teq SNe BEN=GO ae eRiRiy=-0.49 + 0.50 x, R2= 0.12<br>3 LBR——*—BFarh HY) NIC SWZCIM*s poh s—LB (0.86) (0.09)<br>8 pp Som. SE 7 SLB PNG "asd. SLV BINN. *BLR, —TTO "BHR ux®<br>=— 10 Tcp *KIR® GMBLo" . TuKSq}gVUT vorSM UZBf Btz°°? MDAGuy’caB|sURARM, MOV—lMNE* “—~BHSsyc<br>FSM“—4/ WSM veT* mus<br>MRT “KM<br>LCA<br>| cnB° MHL° CPV®<br>1,000 10,000 100,000<br>GDP per capita, PPP (constant 2017 international $)<br><!-- End of picture text -->

<!-- Start of picture text --> USA CHN °<br>AUS<br>1,000 mys, CAN© DEUSIRN *JPN % BRA_ex ip) 3 IND<br>g& NZL *SWENOR NLD, Sau. eae3. 89 eM te PAK<br>8fo.) IRL tp A ca MeTza “itis 8D<br>23 FIN PR’te N PER NBLS COL RUS<br>5 100 HRV . LKA Q3SARG PHL ~EGY<br>g© . EST\. GEO,° BUNGE MAR «‘SoRENMR-—SDNEMIMR-UKR<br>¢ ISL 7 FJIXCYPPesVN ee Peesea 4”. Bly =-6.60A + 0.67anx, R= 0.59<br>3 vuTLUX*.,Sup2Gme” SLV, TEER cow ize<br>€ —. a s 0 “MD, f BDie .- e?<br>=5 10 SYC,FSM °° KIR % BLZ_-vsSUR 2,{tS3TTO,‘MUS’ GABcoz M Tép “civ<br>< M GUY ° MRT TJK<br>v pi =1SO<br>. TKM<br>MAL LCA cPv . GNB .<br>100,000 1,000,000 10,000,000 100,000,000 1,000,000,000<br>Population, Total<br><!-- End of picture text --> <!-- Start of picture text --> CHN® JPN USA<br>1,000 rawokPAK SAU mpmys_*ZAF tO *BRA GBR,ganitaAUS<br>© VNM eo ° THA TyRLOEY yor<br>® ETH e, BED \\usa ERY brus “Cie,<br>3 KEN ‘GHAeEOb cohSeP* ROW<br>& NPL TZA jon \PER Soy ge? *PRT<br>8 IRQ AFG wvR——e HRV AoePEL ze” AUT<br>2 100 . SDN MRS ‘SRB GEO<br>2 cop . NBN, 28% 2wetY Ut" eisy * HUN, SVK<br>S * .°PNG O08BEN » 2% 6 *RWA © MNG, GRe “Sj SVN<br>&633 ol BY cosHTy=y = 0.52+ 0.06 x,*120 R2=0.36- HReT e pses URpy .PSEsKD %.iatCYP,“ant*.~CR VA EST<br>M<br>€ som (0.37) (0.01) HND DOM aze ALB LuxS<br>3 ° KIR~SAPAD TCD VU TO e Nciv8CO,-BLZinetstv ° MLT af\ NDAARM<br>2 Guy MRT ‘SUR TUK? Ney MUS éz<br>FSM. Dil . vet * sYC_ Lso<br>TKM LCA<br>1 MHL . .GNB QoPV<br>20 40 60 80<br>SPI Overall Score<br><!-- End of picture text -->
_Note_ : Data on GDP, population and SPI are retrieved from the World Bank’s World Development Indicators For GDP per capita, the figure indicates an elasticity of 0.5, meaning that a 10% increase in GDP is associated with a 5% increase in academic articles using data. The elasticity with respect to population is 0.67, and population alone explains 59% of the variation in academic articles using data. The overall SPI score is also predictive of academic output using data. A 10-point increase in SPI scores, which is approximately the same as moving from the median SPI score to the 65th percentile, translates into around a 0.6% increase in the number of articles using data.

Table 2 shows coefficients from a regression of the log number of academic papers using data on log GDP per capita, log population, and the SPI scores. The first column shows a specification including just log GDP per capita and log population. These two indicators alone explain roughly 75% of the variation in the production of articles using data. The second column includes the SPI overall score as an additional control. The performance of a country’s statistical system is associated with greater academic output. Conditional on GDP per capita and population, a tenpoint increase in SPI overall scores (on a scale of 0-100) translates into a 0.2% increase in academic output.

It is possible that omitted variables are behind this result. For example, it could be the case that countries that are of general interest to the research community also happen to have high statistical performance. Das et al. (2013) and Porteous (2022) control for a range of variables which may influence the relationship sought after, such as whether English is a first language, international tourist arrivals, etc. We leverage that we have an estimate of the number of articles not using data – referred to as qualitative articles – to control for all factors that are likely to influence overall research output in a country unrelated to statistical performance. The number of qualitative articles is calculated as the number of total articles minus any articles using data, based on our NLP model.

Countries that tend to be over-represented in quantitative articles also tend to be overrepresented in qualitative articles, conditional on GDP per capita and population (Figure A.3). The elasticity is around 0.6, suggesting that a 1% increase in the number of qualitative articles associated with a 0.6% increase in the number of articles using data. We take this to imply that quantitative articles do not crowd out qualitative articles and vice versa. Returning to Equation (2), one prediction from the model is that countries with particularly high costs related to using data, (such as a lower statistical performance score), should see reductions in the amount of quantitative research, but it would not follow that it would see lower levels of qualitative ii XX research. This would predict the share of quantitative articles to total articles increasing in SPI score. Contrary to this, there is almost no relationship between SPI overall scores and the share of qualitative papers to total papers produced overall (Figure A.4). This suggests to us that some omitted variable may in part be driving the impact of statistical performance on academic articles using data, and that the number of qualitative articles can control for such possibly omitted factors.

13
The third column of Table 2 additionally controls for the log number of qualitative articles. The SPI overall scores are still statistically significant at the 5% level, as is country population. Column 4 examines the relationship with one of the pillars of the SPI, data sources, which covers the availability of recent censuses, surveys, academic data, and geospatial data. The data sources index from the SPI is also correlated with data use in academia, conditional on log GDP, population, and the number of qualitative papers. Similar regressions using the other pillars of the SPI, such as data services and data infrastructure are not associated with the number of articles using data in a similar regression, suggesting that it is the availability of particular data sources that is helpful in boosting academic data use.

_Table 2. Relationships between Number of Papers Using Data and Statistical Performance_

||(1)|(2)|(3)|(4)|
|---|---|---|---|---|
|(Intercept)|-12.41***|-10.66***|-3.96***|-3.57**|
||(0.82)|(0.95)|(1.14)|(1.17)|
|Log GDP per capita|0.58***|0.36***|0.12+|0.10|
||(0.05)|(0.07)|(0.07)|(0.08)|
|Log Population|0.70***|0.63***|0.29***|0.29***|
||(0.03)|(0.04)|(0.05)|(0.05)|
|SPI Overall Score||0.02***|0.01*||
|||(0.01)|(0.01)||
|Log Qualitative Papers|||0.64***|0.65***|
||||(0.08)|(0.08)|
|SPI Data Sources Score||||0.01*<br>(0.00)|
|Observations|168|168|168|168|
|R2|0.751|0.774|0.852|0.852|
|R2 Adj.|0.748|0.770|0.849|0.849|
|AIC|414.8|400.1|331.1|330.7|
|BIC|424.2|412.6|346.8|346.3|
|RMSE<br>_Note:_GDP, population and S|0.82<br>PI data are retr|0.78<br>ieved from th|0.63<br>e World Ba|0.63<br>nk's World|

_Note:_ GDP, population and SPI data are retrieved from the World Bank's World Development Indicators. Regressions include all papers using data from 20172019. Standard errors are robust to heteroskedasticity. ***=0.001 level, **=0.01 level, *=0.05 level, +=0.1 level.

Using more detailed data from the SPI makes it possible to assess how the availability of specific data sources relates to academic output. Figure 6 shows the linear regression coefficient for the availability of each of the ten data sources considered by the SPI, conditional on log GDP per capita, log population, and log of qualitative papers. The ten data sources include: population census, agriculture census, business/establishment census, household consumption/income survey, agriculture survey, labor force survey, health survey, business/establishment survey, civil registration and vital statistics system (CRVS), and geospatial data at the Admin 1 (usually province/state) level.

14
<!-- Start of picture text --> Population Census in past 10 years<br>Agriculture Census in past 10 years<br>Business/Establishment Census in past 10 years<br>Household surveys: 2 or more in 10 years<br>Agriculture Surveys: 2 or more in 10 years —_—<br>Labor Force Surveys: 2 or more in 10 years ad<br>Health Surveys: 2 or more in 10 years<br>Business/Establishments: 2 or more in 10 years ——_—_-<br>Complete Civil Registration and Vital Statistics System ‘2<br>Availability of Data at ist Admin Level (ODIN) Score — 7 > - ——<br>0 1 2<br>Coefficient estimates and 95% confidence intervals<br><!-- End of picture text -->
literacy among researchers, low trust in data from the national statistical system, and lack of infrastructure to access and use the data, which may be particularly an issue of low- and middleincome countries.

The policies needed to boost data supply differ from those needed to boost data demand. Boosting the data supply of the national statistical system requires more financing for national statistical offices, better technical capacity of its staff, and at times better statistical laws to ensure their independence from other government bodies. Policies to boost data demand, on the other hand, involve financially cheap (but not necessarily politically cheap) wins, such as making data more accessible for researchers. It also involves increasing the data literacy of academics. Given the non-overlapping policy responses, it is useful for countries to know whether data demand or data supply is their weakest link in the data value chain. Below, we classify countries by whether they are primarily constrained by data supply, data demand, neither or both.

Building on the semantics introduced by Porteous (2020) who classified countries into research deserts and research oases, we classify countries into four groups: _Data deserts_ have little data demand and little data supply, _data swamps_ have high data supply but little data demand, _data oases_ have high data demand but little data supply, and _data lakes_ have high data demand and high data supply. For assessing data supply, we again turn to the availability of various data sources, such as recent population and business censuses, household and health surveys, Civil Registration and Vital Statistics (CRVS), and sub-national geospatial data (for details see Dang et al., 2023). We measure data use as the residual from the regression of number of articles on the country using data on log population, log GDP per capita, and the average number of qualitative papers. The control for the number of qualitative papers once again serves as a proxy for general research interest and research environment unrelated to data.

We classify countries by whether they are below/above the median of our indicators of data supply and data use (Figure 7). Since we use median values in the two dimensions to create the categories, there are about as many countries belonging to each group. Yet there are large discrepancies by region and income group (Table 3). The poorest countries and poorest regions are more likely to be data deserts or data oases. These categories apply to more than 80% of lowand lower-middle income countries, 93% of countries in Sub-Saharan Africa, and only 7% of highincome countries. Thus, poorer countries and poorer countries tend to have lower data supply.

Low-income countries and countries in Sub-Saharan Africa are twice as likely to be oases than deserts. In fact, when looking at the map, it is evident that a large part of the world’s data lakes, are to be found in Sub-Saharan Africa. This suggests that, despite data scarcity, a significant volume of quantitative articles is already produced on these countries, when factoring in their population size, income, and attention in non-quantitative scholarship. These countries may benefit most from bolstering their data supply. This applies for example to Uganda, Ghana, Nepal, and Malawi. By contrast, nearly all upper-middle income countries and countries in Latin America who have low data supply are deserts rather than oasis. That is, they face both data 16
supply and data use constrains. This concerns countries such as Libya, Turkmenistan, and El Salvador.

High-income countries and countries in East Asia & the Pacific are more likely to be a lake than a swamp. By contrast, upper-middle income countries and countries in Europe and North America are more likely to be a swamp than a lake. Hence, the latter countries may have successfully improved data supply but are lacking in data use. This concerns countries such as the Russian Federation, the UK, France, Germany, and the U.S. The Scandinavian countries are all classified as data lakes, meaning they have high supply and high use of their data.

17
<!-- Start of picture text --> '<br>_a? Oases: ‘\ Lakes:<br>g low data supply, ' MNG_ high data supply,<br>-“£9" high data researchMDG LBR~_ BRASETH GMBche> *Mw LAO uePL RWA!&cd\OMNRN “ziKWT FINADNKhigh data research<br>g SWZ“GINOOS° tog 8 AL og |e THA op) 1%, NOR<br>bd> NIC~BENT\eQZ “SEN,zwe BGPwigan,ORT GEO SHEOR8 8+ |Soekor SHECITA<br>fo3 won ron noon pNg====SDN HT StEsalMARY AB - “paSC SKHM"“Tune:SSNS“EGY?oe~~ geecoySs CAN -------<br>g Gag—oOD=— MRT NGA COL gp RERs* » MUS» °s* POL MEX<br>iaz | AFGBOIBDI \TLS<CHNNise iigo ipn* 2 BIKSDZAARESALB°GBR® PHL”.<br>8 SOM TOD, Op.JI Lh SLV BOL“ aze\N\ KAZ IND USA<br>3 LBY acoUzB“7 MLI TJK. KGZT—-UKR BLR GTM, MDARUS<br>8'<br>3?©Doscrtel % '' Swamps:<br>2 low data supply, . ' high data supply,<br>& little data research GNB \ little data research<br>3 J<br>025 50 75 100<br>SPI Pillar 4 (avg. 2017-2019)<br>© Highincome © Lowincome © Lower middie income ¢ Notclassified © Upper middie income<br>Red lines indicate median values.<br><!-- End of picture text -->

<!-- Start of picture text --> = :EY 7, 4 Ss ”><br>;<br>ted ee =F<br>- = VY<br>kes - CAS t<br>Ranh poly x ’ ‘3<br>g@SS weae “aes<br>Classification ly, ¥ 1 :<br>BB vata Desert , ‘<br>| Data Lake ' 2<br>[0| vataData Swamp oasis \Ym,3 Pe 7a<br>Source: SPI<br><!-- End of picture text -->
_Table 3. Data Deserts, Oases, Lakes, and Swamps by Region and Income Group_

||Data<br>Deserts|Data<br>Oases|Data<br>Swamps|Data<br>Lakes|Number of<br>countries|
|---|---|---|---|---|---|
|Region||||||
|East Asia & Pacific|29%|12%|6%|53%|17|
|Europe & Central Asia|11%|0%|58%|31%|45|
|Latin America & Caribbean|37%|11%|21%|32%|19|
|Middle East & North Africa|33%|22%|11%|33%|18|
|North America|0%|0%|100%|0%|2|
|South Asia|17%|50%|17%|17%|6|
|Sub-Saharan Africa|31%|62%|5%|3%|39|
|Income group||||||
|Low-income|35%|65%|0%|0%|20|
|Lower-middle income|31%|49%|11%|9%|45|
|Upper-middle income|33%|0%|44%|22%|36|
|High-income|7%|0%|38%|56%|45|
|Total|25%|24%|26%|25%|146|

_Note_ : Share of countries within a region or income group that are classified as data deserts, oases, swamps, or lakes. For example, 29% of the 17 countries in East Asia & Pacific are classified as data deserts.

In Figure A.5 we show the classification when using number of quantitative articles per capita (unadjusted for GDP or number of qualitative articles) as the measure of data use. With this classification, high-income, often OECD, countries are doing well in generating articles using data. Many of the _data deserts_ are in low-income or conflict affected countries. While the per capita measure is useful for an overview of how countries are performing in producing articles using data, it is not always useful as a policy guide given that it is partially driven by GDP per capita and research interests, which are often outside the control of policy makers in the statistical system.

# 6 **Robustness Checks**

In this section we offer some robustness checks to our main results. First, we estimate our main regressions using country fixed effects and year dummy variables rather than controlling for qualitative articles. Second, we investigate our results if we exclude medical papers.

19
# **6.1 Panel Analysis**

Our analysis may be subject to some omitted variable bias if countries with high statistical capacity share commonalities unrelated to their number qualitative articles. Below, rather than controlling for the number of qualitative articles, we add country fixed effects and year dummy variables to our OLS regressions, so that only within-country variation over time is used to identify our main relationships. Standard errors are clustered at the country level. Note that this specification may remove relevant variation if some countries have a constant high output of papers using data due to a constant high statistical capacity and vice versa.

Using this specification, the relationship between academic output using data and log GDP per capita is similar as previously estimated, with an elasticity between around 0.4 and 0.6 depending on the specification (Table 4). The relationship with population is not statistically significant under this specification in most cases. The SPI overall scores are not statistically significant either under this specification, but because the SPI overall scores only are available starting in 2016, the time series is relatively short. To overcome this shorter time series, an “extended SPI” index is created that incorporates data from the older Statistical Capacity Indicator (SCI), which the World Bank has produced since around 2004.<sup>4</sup> Using the longer time series in the extended SPI, a 10 unit increase in the extended SPI overall score is associated with an increase in academic output using data of around 0.1%. The estimated coefficient is statistically significant at the 5% level and similar in magnitude to the cross-sectional results that control for the number of qualitative articles. The extended SPI data sources score is also statistically significant at the 5% level.

_Table 4. Longitudinal Relationships between Number of Papers Using Data and Statistical Performance_

||(1)|(2)|(3)|(4)|(5)|(6)|
|---|---|---|---|---|---|---|
|Log GDP per capita|0.58**|||0.51|0.58**|0.62**|
||(0.20)|||(0.33)|(0.22)|(0.22)|
|Log Population||0.18||1.09|0.40|0.38|
|||(0.35)||(0.69)|(0.37)|(0.37)|
|SPI Overall Score|||0.00|0.00|||
||||(0.00)|(0.00)|||
|SPI Overall Score<br>(Extended Series)|||||0.01+<br>(0.00)||
|SPI Data Sources Score<br>(Extended Series)||||||0.00+|

4 The SCI does not contain as many indicators as the SPI (25 indicators versus 51 in the SPI), and not all the indicators are similar. Eleven indicators in the SCI overlap with the SPI: national accounts base year, consumer price index, balance of payments manual in use, government finance accounting, population census, agriculture census, health surveys, household consumption/income survey, CRVS availability, and the IMF SDDS status of the country. Using these indicators, the extended SPI series contains 35 indicators with data back to 2004. These include 5 indicators covering data use, 1 covering data services, 16 covering data products, 8 covering data sources, and 4 covering data infrastructure. The correlation between the extended SPI and the official SPI overall score is around 0.90.

20
<!-- Start of picture text --> Partial regression coefficients conditional on<br>log GDP per capita, log population, and<br>country and year fixed effects.<br>Population Census in past 10 years<br>Agriculture Census in past 10 years<br>Household surveys: 2 or more in 10 years<br>Agriculture Surveys: 2 or more in 10 years<br>Labor Force Surveys: 2 or more in 10 years<br>Health Surveys: 2 or more in 10 years<br>Business/Establishments: 2 or more in 10 years - 7?<br>Complete Civil Registration and Vital Statistics System TT<br>-0.2 0.0 0.2 o4<br>Coefficient estimates and 95% confidence intervals<br>Data from the World Bank's World Development Indicators (WDI) and SPI.<br>Papers include all papers using data years 2000-2020.<br>Confidence Intervals at 95% significance level<br><!-- End of picture text -->
<!-- Start of picture text --> CH<br>°<br>MYS USA<br>1,000 IDN” oe AUS<br>2 286 UPN<br>g ROUz° one*. 29," BRA<br>a eee! ‘kor<br>D °3° ee<br>i= oe e<br>@ o ° e<br>> 100 &° °<br>2 ed a"<br><g °"35 4 e<br>3y = 18.34+ 0.29 x, R2=0.93<br>s tem (468) (0.01) G<br>=oD 10 ° ~~ ° ~~,CIV *,GIN,<br>5 DMA YCT v's .. CAP GAB TGO<br>2 DJl-*'* yop eve<br>TUV_STP.A 5 KIR'SSBMU<br>AND yw KNA FSMSGNQ<br>LiE*CywTM ASM “Nop<br>4 MNFATG: ME, MOQ.<br>N IMN 6NB\. GIB PYF<br>TCAZSMR COM PLW<br>M cpv cuw \MAE-FRO<br>10 100 1,000 10,000<br>All Papers Using Data<br><!-- End of picture text -->
<!-- Start of picture text --> Logged number of Articles<br>]<br>1.00 All Medicine Economics Psychology Sociology Poli Sci<br>0.30 Corr Corr Corr Corr Corr ><br>0.10 0.994*** 0.960*** 0.816*** 0.898*** 0.919*** i<br>1000 =<br>40 Corr Corr Corr Corr<br>0.937 0.776 0.880 0.896 ¥<br>10 ®<br>1000 3 3<br>fe g<br>10 e 0.735 0.819 0.862 2<br>°<br>° » wR * .<br>11° .ee, 2 eee<br>400 °. .. tone~~ . °.e e . 5<br>* Corr. Corr: ri<br>10 ¥ 0.844 0.838 &<br>é . a<br>4 2? See S<br>100 °.e °e° °ee . ~~°e ] g<br>ome ° e % @-e. Corr 8<br>e ° ° 3 98s e . ).973°** &fs]<br>z ls 4 2<br>4 we ee oes oe Lseeee°See be {__']<br>100 * .. fea. ° .el aera ° ~<br>° 2, e % Q<br>101 * le bd °° - ie ° *oe ° rybte 3 ° 3 ; e 2on<br>10 100 10001000011 bp aa10 100 1000100001a10 100 1000r 1 ons,10 100r tg1 288. ©10 1001 141 10r 1001<br><!-- End of picture text -->
accessible to researchers, for example through data sharing agreements or by boosting data literacy.

Our results can help countries better understand if their statistical system is supporting the needs of researchers. Given that research is at the heart of evidence-based policy making, a statistical system oriented around making data available and easily accessible for these purposes is important. Countries could use the results to facilitate an understanding of the maturity of their statistical systems in supporting researchers, point to other countries that a particular country could learn from as it seeks to improve, and track progress or assess return on investment for funding related to statistical capacity building.

Our results face some limitations. First, the s2orc database does not currently contain non-English language articles. While the dominance of the English language in academic publications has been well-documented (Altbach 2007), the database omits articles published in other languages. Second, because of computational limitations, only 1 million articles were analyzed out of a sampling frame of around 10 million articles. Future work could bring in these additional articles to improve precision. Third, while the natural language processing model was able to classify whether or not the articles made use of data with 87% accuracy, the model could be extended in ways to extract more information from the articles, including more details about the subject of study and which particular data sources may have been used (surveys, censuses, administrative data, geospatial information etc.).

# **References**

Altbach, Philip G. “The imperial tongue: English as the dominating academic language.” _Economic and political Weekly_ (2007): 3608-3611.

Cameron, Drew B, Anjini Mishra, and Annette N Brown. 2016. “The Growth of Impact Evaluation for International Development: How Much Have We Learned?” _Journal of Development Effectiveness_ 8 (1): 1–21.

Cameron, Grant J., Hai-Anh H. Dang, Mustafa Dinc, James Foster, and Michael M. Lokshin. 2021. “Measuring the Statistical Capacity of Nations.” _Oxford Bulletin of Economics and Statistics_ 83(4): 870-896.

Courtioux, Pierre, François Métivier, and Antoine Rebérioux. 2022. “Nations Ranking in Scientific Competition: Countries Get What They Paid For.” _Economic Modelling_ 116: 105976.

Dang, Hai-Anh H, John Pullinger, Umar Serajuddin, and Brian Stacy. 2023. “Statistical Performance Indicators and Index: A New Tool to Measure Country Statistical Capacity.” _Scientific Data_ 10(1): 146.

Das, Jishnu, Quy-Toan Do, Karen Shaines, and Sowmya Srikant. 2013. “US and Them: The Geography of Academic Research.” _Journal of Development Economics_ 105: 112–30.

24
Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” arXiv. . https://doi.org/10.48550/ARXIV.1810.04805 Hansen, Stephen, Michael McMahon, and Andrea Prat. 2018. “Transparency and Deliberation Within the FOMC: A Computational Linguistics Approach.” _The Quarterly Journal of Economics_ 133 (2): 801–70.

Hjort, Jonas, Diana Moreira, Gautam Rao, and Juan Francisco Santini. 2021. “How Research Affects Policy: Experimental Evidence from 2,150 Brazilian Municipalities.” _American Economic Review_ 111 (5): 1442-80.

<mark>Jolliffe, Dean, Daniel Gerszon Mahler, Malarvizhi Veerappan, Talip Kilic, Philip Wollburg. 2023. “What Makes Public Sector Data Relevant for Development?”</mark> _<mark>World Bank Research Observer</mark>_ <mark>38 (2): 325-346.</mark> Kleinberg, Bennett, Maximilian Mozes, Arnoud Arntz, and Bruno Verschuere. 2018. “Using Named Entities for Computer-Automated Verbal Deception Detection.” _Journal of Forensic Sciences_ 63 (3): 714–23.

Lo, Kyle, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. “S2ORC: The Semantic Scholar Open Research Corpus.” In _Proceedings of the 58_<sup>_th_</sup> _Annual Meeting of the Association for Computational Linguistics_ , 4969–83. Online: Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.447.

National Science Board, National Science Foundation. 2019. “Publications Output: US Trends and International Comparisons. Science & Engineering Indicators 2018.” _National Science Foundation_ .

Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, et al. 2019. “PyTorch: An Imperative Style, High-Performance Deep Learning . Library.” arXiv. https://doi.org/10.48550/ARXIV.1912.01703 Phillips, Brian J., and Kevin T. Greene. 2022. “Where is Conflict Research? Western Bias in the Literature on Armed Violence.” _International Studies Review_ 24 (3): viac038.

Porteous, Obie. 2020. “Research Deserts and Oases: Evidence from 27 Thousand Economics Journal Articles on Africa.” _Oxford Bulletin of Economics and Statistics_ .

Robinson, Michael D, James E Hartley, and Patricia Higino Schneider. 2006. “Which Countries Are Studied Most by Economists? An Examination of the Regional Distribution of Economic Research.” _Kyklos_ 59 (4): 611–26.

Sabet, Shayda Mae, and Annette N Brown. 2018. “Is Impact Evaluation Still on the Rise? The New Trends in 2010–2015.” _Journal of Development Effectiveness_ 10 (3): 291–304.

Sanh, Victor, Lysandre Debut, Julien Chaumond, and Thomas Wolf. “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.” _arXiv preprint arXiv:1910.01108_ (2019).

25
Shelar, Hemlata, Gagandeep Kaur, Neha Heda, and Poorva Agrawal. 2020. “Named Entity Recognition Approaches and Their Comparison for Custom Ner Model.” _Science & Technology Libraries_ 39 (3): 324–37.

Smirnov, Anatoly A., and Irina V. Stukova. “Determinants of integration approach in the agrarian sphere development in contexts of transformation.” _Review of European studies_ 7, no. 8 (2015): 8.

World Bank. 2021. “World Development Report 2021: Data for Better Lives.” Washington DC: World Bank. https://wdr2021.worldbank.org/ Yu Tian. 2020. “THE PARTNER REPORT ON SUPPORT TO STATISTICS PRESS 2020.” PARIS21. https://paris21.org/sites/default/files/inline-files/PRESS2020%20Final.pdf.

26