metadata: authors: 0: author: Patrinos,Harry Anthony
1: author: Angrist,Noam
count: World
docna: 0: docna: Global Dataset on Education Quality : A
            Review and Update (2000-2017)
docty: Policy Research Working Paper
subtopic: Education for Development (superceded),Education For All,Educational Populations,Educational Sciences,International Trade and Trade Rules,Economics of Education
teratopic: International Economics and Trade,Education
abstracts: cdata!: This paper presents the most expansive
            and current cross-country dataset on education quality. The
            database includes 164 countries/territories covering over 98
            percent of the global population from 2000-2017. Of these
            countries, 111 are low or middleincome economies. Harmonized
            learning outcomes are produced using a conversion factor to
            compare international and regional standardized achievement
            tests. These tests include PISA, TIMSS, PIRLS, SACMEQ, LLECE
            and PASEC. In addition, this paper includes the Early Grade
            Reading Assessment (EGRA) for the first time. This enables
            extension of the database substantially. This methodological
            update paves the way to include a series of assessments that
            are increasingly common in developing countries that are
            often excluded from large international assessments. The
            database includes mean scores and disaggregates the data by
            subject, level, and gender. This paper further presents a
            series of methodological improvements including measures of
            uncertainty, fixed conversion factors for greater
            comparability over time and year-by-year data.
repnme: repnme: Global Dataset on Education Quality : A
            Review and Update (2000-2017)
display_title: Global Dataset on Education Quality
            : A Review and Update (2000-2017)
listing_relative_url: /research/2018/09/30455261/global-dataset-education-quality-review-update-2000-2017
topicv3: Education, Skills Development and Labor
            Market m1326135 1484,Quality of Education m1326344 491,Trade m1326924 698,Education m1327048 470,Economics of Education m1327511 1540
publishtoextweb_dt: 2018-09-27T00:00:00Z
historic_topic: International Economics and Trade,Education
no_of_pages: 37
author_keys: 0: author_key: 95903
1: author_key: 1205875
model_extractions: WPS8592 Policy Research Working Paper 8592 Background Paper to the 2019 World Development Report Global Dataset on Education Quality A Review and Update (2000–2017) _Harry Anthony Patrinos Noam Angrist_ Education Global Practice September 2018
Policy Research Working Paper 8592

# **Abstract**

This paper presents the most expansive and current cross-country dataset on education quality. The database includes 164 countries/territories covering over 98 percent of the global population from 2000–2017. Of these countries, 111 are low or middleincome economies. Harmonized learning outcomes are produced using a conversion factor to compare international and regional standardized achievement tests. These tests include PISA, TIMSS, PIRLS, SACMEQ, LLECE and PASEC. In addition, this paper includes the Early Grade Reading Assessment (EGRA) for the first time. This enables extension of the database substantially. This methodological update paves the way to include a series of assessments that are increasingly common in developing countries that are often excluded from large international assessments. The database includes mean scores and disaggregates the data by subject, level, and gender. This paper further presents a series of methodological improvements including measures of uncertainty, fixed conversion factors for greater comparability over time and year-by-year data.

This paper—prepared as a background paper to the World Bank’s _World Development Report 2019: The Changing Nature of Work_ —is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at hpatrinos@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Global Dataset on Education Quality: A Review and Update (2000-2017)

## Harry Anthony Patrinos

Noam Angrist<sup>1</sup> Key words: Quality, Human Capital, Education, International, Achievement, Database JEL Classification: C8, I2, N3, J24, O15 1 We are grateful to Syedah Aroob Iqbal and Husein Abdul-Hamid who were essential contributors to this work. This work builds on co-authored work with Nadir Altinok. Particular thanks to Aart Kraay for detailed comments and contributions to the methodology and paper. Valuable comments were provided by Eva L. Baker, Eduardo Cascallar, Stefan Dercon, Deon Filmer, Roberta Gatti, Rachel Glennerster, Daniel Koretz, Silvia Montoya, George Psacharopoulos, Heiner Rindermann, Halsey Rogers, Jaime Saavedra, Shwetlena Sabarwal, Justin Sandefur, and Eduardo Velez. This paper benefited from multiple workshops held at the World Bank, World Congress of Cliometrics, American Economic Association, IDB, Oxford, USAID and FHI360. A great number of individuals and organizations supplied us with data. A special thanks to Rebecca Rhodes for access to many EGRAs. In the future, the database will be developed as part of a World Bank and UNESCO Institute for Statistics (UIS) partnership. Support from the World Bank’s Research Support Budget is gratefully acknowledged. The views expressed here are those of the authors and should not be attributed to the World Bank Group.
# 1. Introduction

A country’s human capital is critical for its economic success. A recent report found that human capital comprises the largest component of a nation’s wealth, at 64 percent (World Bank 2018). In this paper, we focus on one measure of human capital: learning outcomes. Fifteen years of literature demonstrates that learning outcomes, as well as years of schooling, matter substantially for growth (World Bank, 2018; Hanushek and Kimko, 2000; Pritchett, 2001; Hanushek and Woessmann, 2008, 2012). This insight comes at a time when the availability of international student achievement tests (ISATs) is growing. These tests are psychometrically designed, standardized assessments of cognitive skills and enable credible cross-country and over-time comparisons of learning. However, traditional ISATs often exclude developing countries – the countries that have the most potential to gain from human capital accumulation.

In this paper, we address this gap by including the largest number of developing countries in a globally comparable database of learning outcomes to date. We build on a literature producing comparable measures of learning across countries and over time (Barro and Lee, 1996; Hanushek and Kimko, 2000; Barro and Lee, 2001, 2015; Altinok and Murseli, 2007; Hanushek and Woessmann, 2012, 2016; Angrist, Patrinos and Schlotter, 2013; Altinok, Diebolt and de Meulemeester, 2014; UIS, 2017; Altinok, Angrist and Patrinos, 2018). Our database covers 164 countries/territories from 2000-2017.

The central intuition behind the methodology we use is the production of a conversion factor between international standardized achievement tests (ISATs) such as PISA, TIMSS, and PIRLS and their regional counterparts (RSATs) such as SACMEQ, LLECE, and PASEC. This conversion factor is derived by comparing scores for countries that participate in both an RSAT and an ISAT in a given time period, schooling level (primary and secondary), and subject. Following Altinok, Angrist and Patrinos (2018), we call these countries “doubloon countries.” A detailed description of each assessment is provided in Altinok, Angrist and Patrinos (2018). This approach puts all tests on a common scale enabling credible comparison across countries. These tests have been designed and scaled to be comparable over time since the late 1990s. Thus, there is no need for an intertemporal adjustment over the 2000-2017 interval.

This paper outlines a series of improvements over past databases. First, we extend our methodology to enable the first inclusion of the Early Grade Reading Assessment (EGRA). EGRA is a basic literacy assessment that has been conducted in over 65 countries since 2006. This enables us to extend our database substantially. We include 48 countries with data in the last ten years (2007-2017). Many of these countries are large developing economies previously excluded from globally comparable databases such as Ethiopia, Bangladesh, and Nepal. This methodological update paves the way to include a series of assessments that are increasingly common in developing countries and are often excluded from large international assessments.

2
We further expand our database with recent rounds of PASEC (2014), SACMEQ (2013) as well as PIRLS (2016). The size of our database has a few ramifications. Since the methodology we use to link assessments hinges on overlap in countries which take both an RSAT and an ISAT, the larger the database, the more overlap, and thus all scores are more robust. In addition, we include the largest number of developing countries to date. These countries have the most potential to benefit from educational progress. Thus, a more expansive database enables the inclusion of developing countries as well as enhances the robustness of the methodology used to include them, making each update significant.

In addition to the size of the database, this is the most current database with data as recent as 2017. Given a series of recent global initiatives which focus on education quality highlighted in the 2018 World Development Report, there is significant demand for current, credible and globally comparable measures of learning. This database provides the largest, most comparable, current learning data. Moreover, in this paper we provide a foundation for systematic and continual updates on a periodic basis going forward, enabling tracking of learning outcomes progress and longitudinal analysis.

We introduce a series of methodological improvements. First, we include single years, rather than 5-year intervals. Second, we use all available data to create conversion factors between assessments. Third, we construct a fixed conversion factor for data since 2000. This enables us to deduce changes in scores over time as a function of learning outcomes progress rather than changing conversion factors. Fourth, we provide measures of uncertainty in the form of plausible bounds for our estimates. Fifth, all data are derived directly from test score data. In contrast to black-box and complex imputation, this enables methodological transparency. Moreover, it produces a clear policy lever: when countries perform better on RSATs and ISATs they can be certain the HLO will improve.

# 2. Data

Each assessment included in prior databases is documented in depth in Altinok, Angrist and Patrinos (2018). The ISATs include old IEA assessments, PIRLS, TIMSS and PISA. The RSATs include SACMEQ, PASEC, LLECE and MLA. In this paper, we elaborate mostly on the inclusion of new data, including EGRA, as well as recent PASEC, SACMEQ and PIRLS data. Annex Table 1.0 shows all data included in this database.

# 2.1. EGRA

The Early Grade Reading Assessment (EGRA) is a basic literacy assessment. The assessment is conducted most often in grades 2-4. Since 2006, EGRA has been conducted in over 65 countries.

EGRA is focused on early grade proficiency on basic literacy skills. The EGRA testing framework is based on reading assessments such as DIBELS (Dynamic Measurement Group, 2008), CTOPP (Wagner et al. 1999), and a series of other assessments that measure literacy skills. The EGRA toolkit outlines the EGRA tool in depth (RTI 3
International, 2009). The assessment is a short oral assessment conducted with a child one-on-one. EGRA is designed to be flexible and adapted across countries and contexts, while maintaining core modules and similarities. EGRA is a timed test, enabling uniformity in how it is conducted. The tests often represent the most common features of the local language and align with the expectations of the grade level.

EGRA includes a series of sub-tasks that align with five phases of reading skills development: pre-alphabetic, partial alphabetic, alphabetic, consolidated-alphabetic and automatic. The subtasks in EGRA include:

- Orientation to print

- Letter name identification

- Letter-sound identification*

- Initial-sound identification

- Segmentation (phoneme or syllables)

- Familiar word reading

- Non-word reading*

- Oral reading fluency*

- Reading comprehension*

- Cloze

- Listening comprehension*

- Vocabulary

- Diction

Of these there is a sub-set of ‘core’ subtasks encouraged to be delivered across all countries and contexts marked with an asterisk above (Dubeck and Gove, 2015).

In our analysis, we analyze two specific EGRA sub-tasks: oral reading fluency (ORF) and reading comprehension. Both are core sub-tasks and therefore found across nearly all EGRA assessments. Moreover, both are conceptually linkable indicators supported by the theoretical and empirical literature.

The use of EGRA data hinges on using measures of comparable learning metrics, even if not perfectly equivalent. This approach is consistent with the approach taken by Dubeck and Gove (2015). Comparable measures must have a conceptual link to other reading assessments used in ISATs and RSATs. Moreover, indicators should be relatively invariant and robust to a myriad of changes that might occur across contexts such as language, timing, and details of implementation. For example, phonics is likely to differ across languages and is not conceptually linkable to other assessments. For this reason, we do not use the ‘letter-sound identification’ sub-task even though it is a core EGRA sub-task.

ORF scores have been used to make cross-country comparisons in the literature and have shown relatively high validity and reliability (Dubeck and Gove, 2015). Moreover, our two measures, comprehension and fluency, show high correlations with one another in 4
related analyses in the range of 0.8 and 0.9 (Abadzi, 2011; Piper, 2010). Abadzi (2011) compares various measures on EGRA and refers to the tight correlation between reading fluency and comprehension, consistent with the literature (Piper, 2010; Vagh, 2009; Gove, 2009; Kim et al., 2010). Our own analysis corroborates this. Across 81 country-year observations, we find a correlation of 0.86 (see Figure 1.0).

Figure 1.0: Correlation between Reading Comprehension and Oral Reading Fluency <!-- Start of picture text --> Pearson Correlation Coefficient: .8547<br>NIC<br>MKD<br>TJK IDNIDN<br>AFG KGZKGZ HND<br>IDNKGZKGZ PHL<br>LBR<br>MKD<br>JOR LAOTZARWA TJK PNG<br>GMB SDNRWATUVRWAMMRAFGJOR PNG<br>GMB VUTMMRBDIIRQBGDHTILBREGYEGYETH BDI BGDTONTLSAFGPNGLBRTONKENGUYKIR KEN<br>ZMBGHAMLIGHAMWINGASLEHTINGAGMBMLICODYEMLBRHTILBRCODMARNGASENBGD RWA<br>MWI UGA TLS<br>0 25 50 75<br>ORF score<br>100<br>75<br>50<br>Reading %<br>25<br>0<br><!-- End of picture text --> Despite the viability of both measures, the oral reading fluency sub-task does not have as sound a conceptual link to RSATs and ISATs as comprehension. Moreover, fluency and words per minute used in ORF scores are likely to be more susceptible to differences in implementation and language. Reading comprehension, on the other hand, is not as tightly timed and is less sensitive to language and implementation details. Moreover, it has the most plausibly direct and substantive proficiency link to RSATs and ISATs. For example, reading comprehension on EGRA maps directly onto one of the levels in Levels 1-8 on SACMEQ, the RSAT for East and Southern Africa. Analysis and work done by ACER for the UIS Reporting Scale – a global effort to compare underlying proficiencies across assessments – focuses on reading comprehension partially for this reason (Turner et al., 2018). Abadzi (2011) also cites reading comprehension as a likely proficiency link between EGRA and ISATs and RSATs. This is born out in the data. For example, when we use reading comprehension scores, Tanzania outperforms Kenya, consistent with SACMEQ. When using ORF scores, Kenya does not outperform Tanzania, which is no longer consistent with SACMEQ. Moreover, reading comprehension was the most available subtask across all EGRA datasets we collected.

To this end, we primarily analyze reading comprehension, which is available in nearly all EGRA datasets, is less sensitive to differences in context and implementation, and has a strong conceptual link to RSATs and ISATs. To ensure robustness to language effects, we only include data when students took the test in their language of instruction.

5
For each indicator, there is a threshold and continuous variable option. In theory, the threshold option ‘percent zero’ on either reading comprehension or oral reading fluency is most plausibly comparable. This measures the most basic foundational skills attained which should be relatively invariant to details which would affect performance higher up the learning margin. It is also easily interpretable and meaningful across contexts. However, since this variable is a threshold rather than a continuous measure of learning it is difficult to convert this to a score we can place on an international scale.

Instead we produce mean scores using a continuous variable that is plausibly comparable, the ‘percent of answers correct’ on reading comprehension. We use the percent correct since the number of test items varies slightly for each EGRA. If students did not attempt any question this is recorded as a zero. We further confirm that the threshold and continues variables are tightly correlated, with an inverse correlation of -0.9 (see Figure 2.0). We compute a standardized Z-score from the continuous reading comprehension metric and scale it with a mean of 500 and standard deviation of 100, akin to RSAT and ISAT scales.

EGRA data is designed for grades 2-4, although certain countries will participate out of this range. We restrict data used for our database to grades 2-4 to be consistent with the design of EGRA. This range further optimizes the coverage-bias tradeoff. If we include additional grade levels to expand coverage, the data starts to skew due to availability of data for certain countries coming from a particularly low or high grade (grades 1 or 6). If we restrict our interval to reduce grade bias, we start to lose countries. In the grade 2-4 range, we optimize this trade-off.

Figure 2.0: Correlation between Percent Zero Reading & Percent Reading Comprehension <!-- Start of picture text --> Pearson Correlation Coefficient: -.8991<br>NIC<br>MKD<br>IDN IDNTJKKGZAFGKGZKGZIDNKGZHNDPHL<br>MKD LBR<br>BDI BDIPNGTONTZAAFGTJKJORAFGPNGRWAJORLAOIRQGMBPNGEGYETHRWABGDTUVRWALBRKIRMMRSDNMMRKENGUYMMRSENTONKENTLSGMBTLSEGYLBRHTILBRVUTMARBGDCODRWAYEMGMBNGABGDHTINGAMWIHTICODSLE NGAMLIMLIZMBGHAGHAUGAMW<br>0 .2 .4 .6 .8 1<br>% Reading Zero<br>100<br>75<br>50<br>Reading %<br>25<br>0<br><!-- End of picture text --> Figure 3.0 demonstrates sensitivity to grade within this interval. We run a regression with and without grade-fixed effects comparing mean scores relative to a given country which participates across all three grade levels 2-4. This gives us an indication of sensitivity to 6
grade. We find small differences. Occasionally, we see slightly larger differences, however, in all instances the effect sizes fall within the confidence interval. This sensitivity analysis increases our confidence that the EGRA data is robust to data availability biases by grade.

Figure 3.0: <!-- Start of picture text --> EGRA Coefficients<br>Grade Effects, Nationally Representative<br>-100 -50 0 50 100<br>Coefficients<br>Effects, All Grades Effects, Grade-Fixed<br>Country-Year Dummies<br><!-- End of picture text --> Overall, our inclusion of EGRA enables us to include over 48 countries with at least one data point in the last ten years. Moreover, it paves the way for inclusion of non-traditional assessments through careful choice of testing regime, indicators, language considerations, grade-level participation, and score scaling. Below we show sensitivity to this exercise comparing HLO scores when derived using EGRA versus PIRLS, an ISAT with scores produced using IRT. Since both assessments are conducted in reading at primary level, we can compare scores and expect similar results. We do this for countries that participate in both EGRA and PIRLS from 2000 onwards and take the average. This will be biased by data availability by year, for example, if Egypt participated in EGRA in 2013 and PIRLS in 2016. However, it gives us an indication of whether HLOs are reasonable when derived from EGRA. Figure 4.0 reveals that scores are roughly similar, within 5-28 points of one another. Moreover, we see that ranks are preserved: Honduras outperforms Macedonia, followed by Indonesia, then by Egypt.

7
Figure 4.0: Average HLO derived from EGRA and PIRLS, 2000-2017 <!-- Start of picture text --> 330.5<br>Egypt<br>364.6<br>416.6<br>Indonesia<br>411.8<br>442.0<br>Macedonia<br>414.6<br>449.6<br>Honduras<br>432.2<br>0 100 200 300 400 500<br>HLO - PIRLS HLO - EGRA<br><!-- End of picture text -->

# 2.2. PASEC (2014)

The most recent PASEC in 2014 uses Item Response Theory (IRT) and is a high-quality RSAT. Ten countries participated, including Benin, Burkina Faso, Burundi, Cameroon, Chad, Congo, Cote d’Ivoire, Niger, Senegal and Togo. We include these countries using available microdata. Madagascar also participated in 2015 and was scaled to the PASEC 2014 round. We include Madagascar in our database using estimates from reports. To this end, inclusion of the recent PASEC data adds countries as well as enhances the quality of data from sub-Saharan Africa. This marks a significant improvement over past datasets. To provide a link to past PASEC rounds, which use classical test theory and have substantially different test items, we create an inter-temporal comparison using Togo, which participated in all rounds of PASEC. However, given that PASEC did not conduct an intertemporal scaling calibration as is typical for IRT ISATs and RSATs, priority should be given to analyzing the most recent PASEC 2014 data. Any intertemporal comparisons should be conducted with caution.

# 2.3. SACMEQ (2013)

SACMEQ collected its fourth round of data in fifteen countries in East and Southern Africa from 2012-2014. These include: Botswana, Kenya, Lesotho, Mauritius, Malawi, Mozambique, Namibia, Seychelles, South Africa, Tanzania, Uganda, Zambia, Zanzibar and Zimbabwe. SACMEQ was designed and scaled to be comparable to past rounds. We include estimates from reports for SACMEQ since the microdata is pending.

8
# 2.4. PIRLS (2016)

The newest round of PIRLS includes 50 countries. It was designed to be comparable to prior rounds. Inclusion of PIRLS is straightforward and we use the same parameters as used in prior databases using the microdata. Overall, through the inclusion of recent rounds of PASEC, SACMEQ, and PIRLS as well as the new inclusion of EGRA, we are able to improve the quality of, and include recent data for, over 75 countries since Altinok, Angrist and Patrinos (2018).

# 3. Methodology

# 3.1. Linking Methodology

We build on a literature linking learning assessments starting with Barro and Lee in 2001. Our methodological approach in this paper extends and improves on the one detailed in Altinok, Angrist and Patrinos (2018) to build the most current database covering the largest number of countries to date. Our Harmonized Learning Outcomes (HLO) database includes 164 countries/territories from 2000-2017.

Various methodologies can be used to make assessments comparable. We do not attempt to equate scores. Rather, we link scores to achieve comparability (Kolen and Brennan, 2014; AERA, APA, NCME, 1999; Holland and Dorans, 2006; Linn, 1993; Mislevy, 1992). This distinction is important, since we do not claim tests or test scores are equivalent across tests. Rather, we aim to compare scores on a similar scale.

We do not link using Item Response Theory (IRT) – the technique used to generate scores for each respective international and regional assessment. IRT models the probability a given pupil answers a given test item correctly as a function of pupil and item-specific characteristics. While this methodology is used within each of the international and regional tests we use, to use it across ISATs and RSATs would require overlap in test items. This is not true for a significant enough set of tests and time intervals to create a globally comparable panel dataset. Moreover, even when there is overlap, for IRT to be reliable there must be large item-specific overlap. When this overlap is small, standard maximum likelihood estimates will reflect both true variance and measurement error, overstating the variance in the test score distribution. Das and Zajonc (2010) elaborate on the various challenges of estimating IRT parameters with limited item-specific overlap.

It is possible to empirically test the conditions under which IRT produces reliable estimates by examining differential item functioning (DIF). Sandefur (2016) equates SACMEQ and TIMSS with IRT methods. Sandefur (2016) measures the DIF as the distance between the item-characteristic curve (ICC) for the reference population and actual responses for the focal group, an approach first proposed by Raju (1988). The resulting DIF is high, casting doubt on the IRT approach in a context with limited item overlap, which is the case for most RSATs we aim to include in our database.

9
While IRT might not be a reliable approach when there is limited item-by-item overlap, we conduct a few robustness tests where overlap is larger. We compare our results to the Linking International Comparative Student Assessment (LINCS) project which uses IRT methods and has significant overlap in items for a subset of international studies focused on reading at primary school from 1970 onwards (Strietholt, 2014; Strietholt and Rosén, 2016). If our expanded HLO database can produce similar results to original scores and IRT methods where there is overlap, we gain confidence in our results as well as an expanded dataset.

We note that while mean scores might vary by linking methods, and should be caveated appropriately, ranks and relative performance are relatively robust. While Sandefur (2016) finds large variation on mean scores depending on the equating method chosen, the Spearman rank correlations of the country averages are .97 or higher.

In building globally comparable education quality estimates, we rely on classical test theory (Holland and Hosken, 2003). Suppose that a population of pupils, sampled from the target population T, takes two different assessments X and Y. Here, we suppose that any differences in the score distributions on X and Y can be attributed entirely to the assessments themselves, since group ability is assumed to be constant.

The goal of linking is to summarize the difference in difficulty between two tests X and Y. We would like to link test X , the Anchored Test, to the scale of test Y, the Reference Test. For example, if we want to link PISA 2003 to TIMSS 2003, PISA 2003 is the Anchored Test X while TIMSS 2003 is the Reference Test Y.

Our main estimation method is ratio linking. To conduct ratio linking we index the difference in means in the Anchored Test X and Reference Test Y as follows: where 푙푖푛푘푖푛푔푌푝푙(푋) is the equation for converting observed scores on Anchored Test X to the scale of Reference Test Y. A limitation of this methodology is that it is sensitive to the scale of each test being similar. We address this potential limitation by using tests with a mean of 500 and standard deviation of 100.<sup>2</sup> This ensures that ratios capture differences in test difficulty rather than differences in scaling.

Below we specify each step in the process. The conversion rate is calculated as follows. Mean scores for tests X and Y, 휇(푋) and 휇(푌), are first calculated for a given round r, subject s and level l. We consider tests to be in the same round if they are five years apart and optimize to have the rounds as tight as possible. Most often this translates to within one to two years. In some cases, this extends to three to five years apart. In a few

> 2 In rare cases where tests do not have initial scores on this scale, we rescale the test using the microdata.

10
exceptions, we average adjacent years across one another as outlined in Annex Table 2. This minimizes the likelihood that test differences are a function of time, proficiency, schooling level, or data availability and are an accurate reflection of difficulty across tests. r is the number of testing rounds over the time period of the fixed conversion rate. We then calculate the fixed conversion rate 푑푠푙 from test X to test Y in a given subject and level such that: where i is a country in the set of 푛푟푠푙 doubloon countries that participate in both tests X and Y in a given round, subject, and schooling level (푌푟푠푙 ∩ 푋푟푠푙).

We then apply this conversion rate to a given country j that participates in test X but not test Y to produce a harmonized score: where y is the official year of anchored test X.

When producing aggregate scores across levels and subjects, we average across subjects and then across levels, weighting each equally.

A notable advantage of ratio linking is that it is easy to interpret and policy-friendly. This enables construction as well as consumption of comparable learning outcomes.

Linking methods include mean, linear linking and equipercentile linking, explained in detail in Altinok, Angrist and Patrinos (2018). A common linking method is linear linking – a shift in scores using the difference in means (rather than the ratio of means). This is an additive adjustment that accounts for standard deviations in a given testing round. Hanushek and Woessmann (2012) adopt this approach for OECD countries assuming standard deviations are constant across testing rounds. Hanushek and Woessmann 11
<!-- Start of picture text --> 800<br>Formula Bar<br>600<br>agg 400<br>4<br>& 200<br>g 5<br>a e<br>0<br>: 200 400 600 800<br>HLO Rank - Ratio<br><!-- End of picture text --> <!-- Start of picture text --> 0.05<br>0.04<br>2 0.03 agg<br>b=<br>a 0.02 4<br>: &<br>0.01 g<br>a<br>Oe<br>-50 0 50 100<br>Difference<br><!-- End of picture text -->
is based on data that is nationally representative and caveat the data appropriately. For EGRA, only a sub-set of countries have nationally representation data. Of 72 countryyear observations, 42 are nationally representative. We produce the linking function based on countries that took part in nationally representative tests to guarantee population overlap. We apply the linking function to all countries with nationally representative EGRA results producing internationally comparable results. We also apply this linking function to non-national EGRA assessments. While we have confidence in the former estimates, we caveat the latter estimates. We exclude TIMSS results for Yemen which TIMSS highlighted as having a flooring effect and has an unrealistically low score. Our HLO Yemen score is thus derived from EGRA data. For Sri Lanka, we use a score derived from the National Assessment which was linked to TIMSS (Dundar et al., 2017). In the case of China and India we validate or extrapolate nationally representative data, described in Annex 3 and 4.

Second, tests should measure similar proficiencies. We link across precise subjects (math, reading and science) to ensure proficiency overlap. For EGRA, since it often has multiple languages of instruction, we only include scores for tests where the language of instruction is the same as the language of the test. This ensures we capture underlying cognitive skills rather than language effects.

Third, test differences should capture difficulty differences rather than country-fixed effects. We address this assumption in a few ways. As noted above, we link tests across doubloon countries that participate in tests that are nationally representative. The reliability of this linking exercise increases the larger the set Y ∩ X, the number of countries that take both tests being linked. Each update increases this overlap, making the inclusion of new countries in this update significant.

A novel contribution of this update which increases the reliability of this last assumption is the fixing of our conversion rate across a 15-20-year interval for recent years from 20002017. This enables us to increase the sample size in the set Y ∩ X for each doubloon index. For example, if Botswana participates in SACMEQ I and II as well as TIMSS 2003 and TIMSS 2007, we now have four data points instead of two, and two rounds of conversion factors instead of one between SACMEQ and TIMSS. In fixing the conversion factor, we assume that the relationship between tests stays constant across rounds. This assumption is reasonable and is true by design for RSATs and ISATs since the mid 1990s. This is not the case prior to the 1990s, limiting our ability to apply a fixed conversion factor to earlier data. The fixing of conversion factors increases the likelihood that we capture test-specific rather than country-specific differences. It further guarantees that any changes in test scores over this interval are due to realized progress rather than changing conversion factors.

13
# 3.3. Additional Methodological Parameters

Over-time comparability. ISATs and RSATs have been designed to be comparable since the late 1990s and early 2000s. Thus, the use of these modern assessments enables comparability over time.

Time Intervals. While this is one of the largest and most comprehensive comparable learning outcomes database produced to date, it is still sparse given limited test frequency. In past databases we smoothed time intervals over 5-year periods. This produced continuously spaced intervals, was designed to reduce noise by averaging results within these intervals and was comparable to the Barro-Lee approach for years of schooling. In this update, we have moved away from this approach. We now provide the year of the Reference Test X as documented in official reports. Since the doubloon index is a fixed conversion rate over a given interval, we produce a score for Anchor Test Y in the same year as Reference Test X. This enables greater granularity and precision of the data and enables the users of the database to make trade-offs at their discretion.

Schooling Levels. We construct a score for each grade and pool across grades within each schooling level to produce primary and secondary school scores. We include the default grade for TIMSS as the anchor grades across levels: grade 4 for primary and grade 8 for secondary. If we have data for additional grades, we include the closest grade with a preference for older grades since learning trajectories are steepest in the early years of learning introducing more scope for grade bias at earlier grades. If we have multiple test scores, we include the closest grade, and exclude the rest, rather than averaging. For example, we include PASEC grade 5 scores and exclude grade 2 scores. We include SACMEQ grade 6 scores along with PIRLS grade 4 scores at primary. If the test is designed for an age group (for example, PISA) we code it at the schooling level (for example, secondary for PISA).

Conceptually, the broader categories of primary and secondary scores enable us to categorize learning at critical schooling levels across assessments which span multiple grades and age groups. At the same time, we specify an approach to including specific grade levels to ensure we have a tight grade interval to minimize scope for grade-specific differences. While the interval is relatively small, it still leaves room for grade-fixed effects rather than test-fixed effects when linking tests. For example, linking PIRLS 2001 grade 4 with SACMEQ 2000 grade 6 might capture a grade difference in PIRLS (which is likely to increase performance) in addition to difficulty. However, to enable greater coverage, we put up with the need to expand beyond single grade level intervals. Moreover, these differences are often small and since the linking functions are applied across the entire set of countries being linked, any bias will be applied equally such that cross-country bias is eliminated for each pair-wise link and regional ranks will be preserved.

We distinguish primary from secondary schooling since enrollment rates drop off quickly between levels in many developing countries. This introduces a potential selection term in secondary school scores, with the highest performing students progressing in the system, biasing scores up due to selection rather than actual learning. We give this methodological 14
parameter increased importance in this update. In the previous version of the database we linked SACMEQ scores in primary school to TIMSS scores in secondary school to increase the number of doubloon countries (Botswana and South Africa). In this version, we reduce the number of doubloon countries (just Botswana) and link at the primary school level only given the scope for selection between levels.

Subjects. We link tests within three specific subjects: mathematics, reading and science. While this is not granular at the test item level, this ensures that there is significant proficiency overlap in linked tests.

Data Availability. The availability of data is sparse. This introduces bias if data availability is correlated with education quality or progress. For example, if countries that perform worse only have available data in later years (since they were later to introduce assessments), this would mean the data that exists is more recent, likely biasing the average up due to testing later rather than stronger performance. Thus, countries that have historically performed poorly might appear to do better. Since we now provide yearby-year test scores, this can be accounted for rather than buried in 5-year intervals.

Relatedly, when aggregating data across subjects, levels and time, there is a trade-off between coverage and disaggregation. For example, while constructing averages across subjects would increase data coverage, it also makes it harder to interpret and introduces potential sources of bias. To this end, we construct disaggregated measures as well as aggregated ones. This enables analyses at each level, considering the trade-offs.

Subsamples. We construct a doubloon index over the full sample. When calculating Harmonized Learning Outcomes by gender we apply the overall doubloon index to raw scores for each sub-sample, rather than constructing sub-sample specific doubloon indices. While performance is likely to vary across sub-samples in a given test, the relationship between test X and Y is unlikely to vary across sub-samples or relative to the full sample.

Doubloon. In constructing the doubloon index, we use as much data as possible. This is an update over our previous methodology, where we choose one specific anchor year in which to construct the index. This enables us to verify if the doubloon index is stable over time, reduces noise, and increases the sample size of the doubloon index.

Steps in the Linking Function. When converting RSAT and EGRA results to an international scale, we have two options: (1) conversion first to regional assessments or (2) directly through an international assessment.

Approach (1) will best preserve underlying regional rankings. However, it will distort rankings at the margin of regional groupings. Approach (2) is direct, simple and uniform, however might distort underlying regional rankings. Each approach will produce more reliable estimates depending on the number of doubloon countries that is possible for each. For example, for EGRA the RSAT transformation for sub-Saharan Africa has three doubloons, while the direct PIRLS transformation also has three doubloons. In this case the doubloon advantage is ambiguous.

We go with approach (2) where possible since it is the simplest and most direct approach and leaves the least room for error, since each step in the linking function introduces 15
uncertainty. Exceptions include PASEC and MLA which only have overlap with SACMEQ. To this end, scores are first made comparable to SACMEQ scores, before they are converted to our final HLO score.

Hierarchy of Tests. We construct a hierarchy of tests to facilitate year-level-subject matching between anchored and reference tests. For math and science, scores are converted to TIMSS-comparable scores at primary and secondary level. For reading, scores are converted to PIRLS-comparable scores at primary level and PISA-comparable scores at secondary level.

This hierarchy of tests can be extended to adjudicate which HLO score to use when multiple scores are available across sources tests. (e.g. a country that took part in EGRA and SACMEQ). The first HLO choice is: PIRLS, PISA or TIMSS depending on the subject and schooling level. If a country took part in PISA and not in TIMSS for math and reading, we next convert PISA scores into subject-specific secondary HLO scores. If a country took part in LLECE, we convert these scores into a subject-specific primary HLO score. Next, we take SACMEQ HLO scores, followed by PASEC HLO scores, then EGRA HLO scores, and finally by MLA HLO scores.

Measures of Uncertainty. We include measures of uncertainty to quantify the degree of confidence around our estimates. We capture two sources of uncertainty: scores on the original test and uncertainty in the calculation of the conversion factor across tests. Simplifying equations (2) and (3), the HLO for a given country on test X is: We calculate the variance of the HLO by bootstrapping. We assume country-level average scores are asymptotically normally distributed . We take 1000 draws from the distribution of subject-level-grade average test scores for each testing regime. We create doubloon indices and HLOs from each bootstrapped sample. We take the 2.5<sup>th</sup> and 97.5<sup>th</sup> percentiles of the distribution and use this to construct lower and upper bounds of uncertainty for our HLO scores.

We find small uncertainty intervals overall, with an average of 3.5 points and ranging from 1 to 11 points (see Figure 6.0). This is consistent with original standard errors from each respective testing regime. We find larger uncertainty for HLO scores relative to original scores when testing regimes have fewer doubloon countries, such as SACMEQ and PASEC. By quantifying this uncertainty, we can reliably bound our estimates.

16
Figure 6.0: Standard Errors for Original and HLO Scores by testing regime <!-- Start of picture text --> EGRA<br>LLECE<br>PASEC<br>PIRLS<br>PISA<br>SACMEQ<br>TIMSS<br>0 5 10<br>Original Score HLO Score<br><!-- End of picture text --> Scale. Our methodology enables us to link countries participating in regional standardized achievement tests (RSATs) on an international scale. On this scale, 625 represents advanced attainment and 300 represents minimum attainment. This interpretation is derived by taking an ISAT benchmark for high performance on the upper end of the distribution and an RSAT benchmark (expressed in HLO units) on the lower end of the distribution. This approach enables us to capture performance across the entire distribution and accounts for floor and ceiling effects that would be introduced by just taking the ISAT or RSAT benchmarks alone on both ends of the spectrum.

# 4. Illustrative Analysis

We include a few key figures from our database as motivation for the utility of the database. We analyze disaggregated data by subject and schooling levels to explore the full texture of the data.

Figure 7.0 shows select countries on a global scale in reading in 2016. We see a few notable trends. Russia has taken the lead – a new development. Chile nearly outperforms France and has started to outperform Eastern European countries such as Georgia. Saudi Arabia places near the bottom outperforming only African, Pacific or war-affected countries. Rwanda outperforms the regional behemoth, South Africa. The performance gap between Egypt and Russia is nearly double.

Figure 8.0 shows the relationship between average harmonized learning outcomes in science from 2000-2017 and 2015 GDP per capita.

17
Figure 7.0: Select Countries, Primary, Reading, 2016 <!-- Start of picture text --> Primary, Reading, 2016<br>South Africa<br>Egypt<br>Afghanistan<br>Morocco<br>Rwanda<br>Kiribati<br>Saudi Arabia<br>Georgia<br>Chile<br>France<br>Israel<br>Germany<br>USA<br>Finland<br>Hong Kong, China<br>Singapore<br>Russian Federation<br>0 200 400 600 800<br>Score<br><!-- End of picture text --> In figure 8.0, we highlight countries in red that outperform relative to their incomes status, such as Singapore, Japan, Finland, Estonia, Cuba, and Vietnam. Countries that underperform on learning outcomes relative to their income status include Qatar, Saudi Arabia, Kuwait, South Africa, Panama, Botswana, and Ghana. We also highlight large developing countries on the same graph. These include India, China, Mexico, Brazil and South Africa. China outperforms its counterparts, while India and South Africa trail behind. These graphs demonstrate a tractable ‘distance to the frontier’ – large developing economies who might benefit their economies by focusing on human capital development.

Figure 8.0: Average HLO Science Score (2000-2017) vs. 2015 GDP per Capita <!-- Start of picture text --> Singapore<br>Finland<br>Vietnam Cuba Estonia Japan<br>China<br>Mexico<br>Brazil<br>Saudi Arabia<br>Panama<br>Qatar<br>Botswana Kuwait<br>India<br>South Africa<br>Ghana<br>7 8 9 10 11 12<br>log GDP per capita<br>High Performing vs. GDP per Cap Low Performing vs. GDP per Cap<br>Selected Large Developing Countries<br>600<br>500<br>HLO<br>400<br>300<br><!-- End of picture text --> 18
# 5. Robustness Checks

We compare Harmonized Learning Outcomes (HLOs) for primary reading scores with scores generated using IRT linking by the Linking International Comparative Student Assessment (LINCS) project. The LINCS project leverages overlap in items for a subset of ISATs focused on reading at primary school from 1970 onwards (Strietholt, 2014; Strietholt and Rosén, 2016).

Figures 9.0-9.6 below show our results. The HLO and IRT scores have large overlap with a correlation coefficient consistently above 0.95 for scores as well as ranks. This indicates we are able to produce similar results to an IRT methodology where there is overlap. This increases our confidence in the robustness of our estimates.

We further compare ranks derived from HLO scores versus raw scores and verify the original regional rankings are preserved. We are able to fully preserve underlying regional ranks within a year subject and schooling level. This is true by design due to a fixed exchange rate applied at this level of disaggregation. We do not show these results since they are true mechanically.

We conduct two additional robustness tests, shown in Annex 5. The first explores HLOs derived from multiple possible source tests and how this affects international and regional rankings. The second is a comparison of test scores by data availability of each test. These robustness tests confirm that even when HLO scores come from different tests they are broadly consistent.

A future robustness check could be the inclusion of psychometric adjustments. Jerrim et al. (2017) highlight a few features of ISATs that are important to consider when conducting aggregate test score comparisons. Jerrim et al. (2017) conclude that results of a paper they re-analyze by Lavy (2015) are robust to inclusion of these elements. This enhances our confidence that such adjustments are not strictly necessary, but we hope to include them as a robustness test in the future. For example, if we want a regional average we can weight countries to construct a reliable ‘regional average’ that is not distorted by artificially weighting each country equally since the true region consists of larger countries.

19
Figures 9.0-9.3: Robustness - HLO vs. IRT Primary Reading Scores by Year <!-- Start of picture text --> HLO vs IRT Scores HLO vs IRT Scores<br>2001, 33 countries, .9546 corr. 2006, 36 countries, .9879 corr.<br>350 400 450 500 550 350 400 450 500 550<br>IRT Primary Reading Score IRT Primary Reading Score<br>HLO vs IRT Scores<br>2011, 46 countries, .9879 corr.<br>350 400 450 500 550 600<br>IRT Primary Reading Score<br>600<br>550<br>500<br>500<br>450<br>400<br>400<br>HLO Primary Reading Score 350 HLO Primary Reading Score<br>300 300<br>600<br>500<br>400<br>HLO Primary Reading Score<br>300<br><!-- End of picture text --> Figures 9.4-9.6: Robustness - HLO vs. IRT Primary Reading Ranks by Year <!-- Start of picture text --> HLO vs IRT Ranks HLO vs IRT Ranks<br>2001, 33 countries, .9546 corr. 2006, 36 countries, .9879 corr.<br>0 10 20 30 40 0 10 20 30 40<br>IRT Primary Reading Rank IRT Primary Reading Rank<br>HLO vs IRT Ranks<br>2011, 46 countries, .9879 corr.<br>0 10 20 30 40 50<br>IRT Primary Reading Rank<br>40 40<br>30 30<br>20 20<br>10 10<br>HLO Primary Reading Rank HLO Primary Reading Rank<br>0 0<br>50<br>40<br>30<br>20<br>HLO Primary Reading Rank 10<br>0<br><!-- End of picture text --> 20
# Conclusion

There is policy and academic demand for a measure of education quality that is comparable across countries and over time. The growth of international standardized achievement tests is a huge step in this direction. However, the countries that participate in these tests are often high and middle-income countries. This limits the ability to track, compare, or understand education patterns in developing countries – the countries that often have the most to gain from quality education and human capital formation.

One option is to wait to make comparisons for low-income countries to participate in international assessments. Although this is a worthy aspiration and we hope it occurs, it will take time and render a rich array of retrospective data null, limiting longitudinal data analysis. Alternatively, we can use a rigorous approach – albeit with caveats – to harmonize available learning data across different types of international and regional assessments. This is the approach we take in this paper, creating a Harmonized Learning Outcomes (HLO) database. Until there is a global proficiency assessment for numeracy and literacy, and to enable rich longitudinal panel data analysis, the harmonization of existing learning assessments provides the next best alternative to compare education quality on a global scale. Moreover, as more countries join international and regional assessments, and do so for longer, the accuracy and robustness of the harmonization exercise will improve.

We build a globally comparable database of 164 countries/territories from 2000-2017, approximately two-thirds of which are in developing economies, representing more than 98 percent of the global population. While our methodology has limitations, our robustness tests indicate that this dataset produces similar results to each underlying assessment used, as well as Item Response Theory (IRT)-linking methodologies where there is overlap. We include double the number of countries of any individual assessment or IRT-linking methodology while demonstrating consistent estimates where there is overlap.

We contribute to the literature in several ways. This is the most current and expansive cross-country dataset on education quality, including the most developing countries. We include, for the first time, the Early Grade Reading Assessment (EGRA) in addition to TIMSS, PISA, PIRLS, SACMEQ, LLECE, PASEC and MLA. This enables us to extend our database substantially, including 48 countries with recent data in the last ten years, all of which are developing economies. This methodological update paves the way to include a series of assessments that are increasingly common in developing countries that are often excluded from large international assessments. We introduce a series of methodological updates, including measures of uncertainty, fixed conversion factors for greater comparability over time, and year-by-year data. All data are derived directly from test score data. In contrast to black-box and complex imputation, this enables methodological transparency. Moreover, it produces a clear policy lever: when countries perform better on RSATs and ISATs they can be certain the HLO will improve.

21
Our goal in this paper is not to provide a perfect measure of education quality. Rather, we provide a practical yet rigorous and globally comparable set of estimates with large and inclusive country coverage over time. We hope this dataset can be used to reveal important descriptive trends in human capital formation across developed and developing countries. We also hope to enable analysis of factors correlated with and that have plausible causal links to the formation of human capital and economic growth. Finally, we hope this dataset can be useful for monitoring and evaluation of important policy goals.

Future iterations of this dataset will continue to expand coverage across countries and time as countries join existing assessments and by including additional assessments such as early grade reading and mathematics assessments. Moreover, we aim to build a dataset that enables over-time isolation of value-added learning by including variables which can account for various selection effects. This includes linking quality of education data to quantity of education data, as well as including measures of enrollment and retention across schooling levels. We also aim to enable further identification of the link between education quality and economic growth by including variables such as comparable estimates on the returns to education. We hope this dataset, and future iterations, will enable a deeper understanding of mechanisms driving human capital formation, the link to development, and useful policy applications.

22