metadata: authors: 0: author: Jakubowski,Maciej Jan
1: author: Gajderowicz,Tomasz Janusz
2: author: Patrinos,Harry Anthony
count: World
docna: 0: docna: COVID-19, School Closures, and Student
            Learning Outcomes : New Global Evidence from PISA
docty: Policy Research Working Paper
abstracts: cdata!: The COVID-19 pandemic resulted in
            significant disruption in schooling worldwide. This paper
            uses global test score data to estimate learning losses. It
            models the effect of school closures on achievement by
            predicting the deviation of the most recent results from a
            linear trend using data from all rounds of the Programme for
            International Student Assessment. Scores declined by an
            average of 14 percent of a standard deviation, roughly equal
            to seven months of learning. Losses were greater for
            students in schools that faced relatively longer closures,
            boys, immigrants, and disadvantaged students. Educational
            losses may translate into significant national income losses
            over time.
display_title: COVID-19, School Closures, and
            Student Learning Outcomes : New Global Evidence from PISA
listing_relative_url: /research/2024/01/34236962/covid-19-school-closures-student-learning-outcomes-new-global-evidence-pisa
topicv3: Learning Assessment m1327283 1555,Economics of Education m1327511 1540,Economic Analysis of Education m1327512 2124
publishtoextweb_dt: 2024-01-11T00:00:00Z
dois: 10.1596/1813-9450-10666
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 32
author_keys: 0: author_key: 988487
1: author_key: 1538072
2: author_key: 95903
model_extractions: Policy Research Working Paper 10666 COVID-19, School Closures, and Student Learning Outcomes New Global Evidence from PISA _Maciej Jakubowski Tomasz Gajderowicz Harry Patrinos_ Education Global Practice January 2024
Policy Research Working Paper 10666

# **Abstract**

The COVID-19 pandemic resulted in significant disruption in schooling worldwide. This paper uses global test score data to estimate learning losses. It models the effect of school closures on achievement by predicting the deviation of the most recent results from a linear trend using data from all rounds of the Programme for International Student Assessment. Scores declined by an average of 14 percent of a standard deviation, roughly equal to seven months of learning. Losses were greater for students in schools that faced relatively longer closures, boys, immigrants, and disadvantaged students. Educational losses may translate into significant national income losses over time.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at hpatrinos@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# COVID-19, School Closures, and Student Learning Outcomes: New Global Evidence from PISA

**Maciej Jakubowski Tomasz Gajderowicz** University of Warsaw University of Warsaw

**Harry Patrinos**<sup>1</sup> World Bank

_JEL No_ : I19, I20 _Keywords_ : COVID-19; learning loss; student achievement; PISA; international large-scale assessments

> 1 Maciej Jakubowski, Faculty of Economic Sciences, University of Warsaw, Warsaw, Poland (mjakubowski@uw.edu.pl); Tomasz Gajderowicz, Faculty of Economic Sciences, University of Warsaw, Warsaw, Poland (tgajderowicz@wne.uw.edu.pl); Harry Anthony Patrinos, World Bank, Washington DC, USA (hpatrinos@worldbank.org).
# **1. Introduction**

Schools throughout the world closed in response to COVID-19 beginning in 2020. In some cases, they quickly reopened in 2021, but in some countries, they remained closed longer (Patrinos 2023). Globally, schools were closed for an average of 5.5 months (22 weeks) since the onset of the pandemic, equivalent to two-thirds of an academic year, when localized school closures are considered (UNESCO 2023a). The duration varies by region, from just one month in Oceania, to 2.5 months (10 weeks) in Europe, to as many as 5 months (20 weeks) in Latin America and the Caribbean. Early on, the school closures were expected to contribute to what was already described as a learning crisis (Angrist et al. 2021; World Bank 2017).

Country studies show large losses (Alasino et al. 2024; Betthäuser et al. 2023; Carlana and La Ferrara 2023; Donnelly and Patrinos 2021; Engzell et al. 2021; Hammerstein et al. 2021; Jack and Oster 2023; Jack et al 2022; Maldonado and De Witte 2022; Zierer 2021). On average, school closures induced by COVID-19 led to a learning loss of 0.17 of a standard deviation – that is, at least one-third of a year’s worth of learning (Patrinos et al. 2023). Distance learning during the school closures does not seem to have helped very much (Haelermans et al. 2022; Singh et al. 2022); only the duration of school closures led to variations. Most studies observe increases in inequality where certain demographics of students experienced learning losses that were more significant than others. These learning losses could translate to earnings losses and could cost this generation of students trillions of dollars (Psacharopoulos et al. 2021).

Also, global studies – that is, studies using data from international assessments where students in many countries take the same tests and under the same conditions, thus, making them comparable across country and time – document significant learning losses (Jakubowski et al 2023; Kennedy and Strietholt 2023). International reading scores declined an average of 2
33 percent of a standard deviation, equivalent to more than a year of schooling. Losses are greater for disadvantaged students, those struggling before the pandemic, and by duration. However, until now, there has only been one international student assessment. It covered only fourth grade students in just 55 countries.

In this paper we examine the impact of schooling disruptions on student reading, mathematics, and science scores using large-scale, internationally comparable student achievement tests from the OECD’s Programme for International Student Assessment (PISA) study. These assessments are conducted every three years since 2000, with the latest implementation in 2022 collecting student results after the school closures caused by the COVID-19 pandemic. Overall, the data explored in this study represent 175 million 15-yearold students in 72 countries. By combining data from previous rounds and the post-COVID round for countries that have over-time data, we can see the evolution of mathematics and science scores before and after COVID-19. We model the effect of closures on achievement by predicting the deviation of the most recent results from a linear trend in reading, mathematics, and science achievement.

Our analysis reveals that school closures led to student learning losses when de-trending the losses to account for school closures. Math scores declined due to pandemic-era school closures from 2018 to 2022 by an average of 12 points, or 14 percent of a standard deviation (SD), roughly equivalent to seven months of learning. Larger declines are recorded for students in schools that faced relatively longer closures and for lower-achieving students. Countries with the shortest closures experienced relatively small losses, 10 percent of a SD, or equivalent to 5 months of learning. Countries with average length of school closures experienced losses of around 13 percent of a SD, or 7 months of learning. Countries with the longest closures experienced losses of around 20 percent of a SD, or up to 12 months of learning.

3
Depending on the closure length measure, each week of school closures was associated with 0.15-0.17 points or around 0.2 percent of a SD of additional decline in achievement for boys. At around 40 weeks of full school closures, the overall learning loss for boys is the same as for girls and increases for longer closures. Students with an immigrant background lost 0.33 points or 0.4 percent of a SD with every week of full school closures and while in countries with shorter closures their learning loss was smaller, for countries with around 25 weeks of closures their learning loss is like the learning loss for natives, and it increases for longer closures.

Additional analyses using quantile regressions reveal significant differences in the learning losses between students at different achievement levels, but also across countries that vary in the length of school closures. For countries with the average length of school closures the learning loss is similar for low-, average-, and high-achieving students. However, in countries with relatively short closures, the best students lost very little in terms of achievement and the decline is mostly observed among the average- and low-achieving students. In countries with longer closures, the learning loss is larger for the best students (see Gambi and De Witte 2023 for similar findings in the case of Belgium). In countries with the longest closures, the low-achieving students lost around 16-17 points or 20% of a SD, while those at the top of achievement distribution lost 25 points or more (29 percent of a SD).

These differences in learning loss at different achievement levels in countries with short and long closures can be associated with differences in the overall achievement in these countries. Correlating average achievement with the length of school closures reveals that countries with the longest closures are also countries with the lowest achievement in PISA, while countries at the top of the PISA rankings closed schools for much shorter periods, on average. There are larger losses among the lowest achieving students in countries with short 4
closures and high achievement, and larger losses among the highest achieving students in countries with long closures and low achievement.

# **2. Data**

Internationally comparable achievement data in mathematics and science come from the OECD’s Programme for International Student Assessment (PISA). Since 2000, PISA was implemented every three years, with only a longer four-year break between 2018 and 2022 due to the pandemic. Databases include results of 15-year-olds coming from eight cycles of testing, including data from 100 countries and economies. We use data from 72 countries with results available at least from 2022 and one round of assessment before the pandemic. Our sample includes more than 3 million students participating in all rounds.

PISA data include plausible values measuring mathematics achievement comparably across all rounds since 2003. For reading, the results can be compared across all PISA cycles since 2000. For science, achievement scores are comparable since 2006. Every PISA cycle has its main domain, which in practice means that students answer many more questions in this domain and their scores are more reliable than in other domains. The most recent round in 2022 focused on mathematics. Mathematics was also the main domain in 2003 and in 2012, providing us with reliable data on long-term achievement trends in this domain. Thus, in our paper we focus on mathematics results, while we also discuss results for reading and science. Moreover, Goldhaber et al. (2023) demonstrate a much larger impact on math scores for students residing in high poverty areas.

Summary statistics for student achievement and sample size across 2000 and 2022 are presented in Annex Table A1. These data differ from those presented in official PISA reports as we included only countries for which trends could be estimated. While scores in all domains tend to decline, these are not directly comparable since different countries participated in each round. The sample sizes also vary across domains as in different years results for some 5
countries were withdrawn in single domains due to technical issues. Thus, the results cannot be simply compared across time to analyze achievement trends, and we propose a regression framework adjusting for time-invariant country characteristics and for country-specific time trends.

PISA data also include detailed characteristics of students and their schools. We use information on gender, socioeconomic status, and immigrant background to explain differences in achievement scores, but also to control for possible changes in student samples across time. The PISA measure of socio-economic status (ESCS) is an index measuring student family economic, social, and cultural status. We use scores that were equated to 2015 to make them comparable across cycles. Descriptive statistics for these variables are presented in Annex Table A1. In the regressions, we use student-level variables as typical control variables but also country averages for every cycle to adjust for sample and population changes over time.

Finally, we use information from students and principals to estimate length of school closures. In 2022, students and principals were asked for how long their schools were closed due to COVID-19. We recoded their responses to the number of weeks of school closures (see Annex Table A1). We also merge PISA data with information from UNESCO (2023b) on the total number of weeks schools were closed fully or partially. Since the pandemic began, UNESCO monitored school closures and modalities for delivery around the globe daily. Schools were considered fully closed in case of “government-mandated closures of educational institutions affecting most or all of the student population.” On the other hand, schools were considered partially open when: (a) open in certain regions and closed in others; and/or (b) open for some grades, levels, or age groups and closed for others; and/or (c) open with reduced in-person class time, combined with distance learning (UNESCO 2023c).

Descriptive statistics for the four measures of the length of school closures caused by COVID-19 are provided in Table 1, while Annex Table A2 provides exact values on each 6
measure, sample size, and years covered for every country included in the analysis. All four measures are reported in weeks. All measures are correlated (from 0.6 to 0.8), but as the definitions and data collection modes vary, the actual numbers are different. None of the measures is perfect and can be taken as preferred over others. The length of full closures as reported by UNESCO is probably the best objective measure, but it limits comparisons to closures of all schools in the whole country, while many countries relatively quickly decided to close schools only when necessary. Also, it does not differentiate between levels of education even if in some countries decisions were different for primary and secondary education. The second measure from UNESCO includes partial closures but does not differentiate between weeks when only one school was closed in a country and weeks when, for example, most schools were closed. PISA-provided data reflect the intensity of closures in every country as they are calculated as the average across students and schools. However, these data are self-reported relying on people’s memory and their understanding of what school closure means. Also, student-provided information must be recalculated into weeks from categorical responses (see Annex Table A2 for details). Finally, while principals report closures for their schools, many students taking PISA in 2022 where in different schools during the pandemic, for example, in primary schools that could experience different closures.

To make the results comparable across different measures of the length of school closures, we calculate the learning losses for the 10<sup>th</sup> , 25<sup>th</sup> , median, 75<sup>th</sup> , and 90<sup>th</sup> percentiles of each school closure measure. Thus, the results are reported for countries with very short, short, average, long, and very long school closures relatively in each of the four measures. Comparing estimates across school closure measures assures that our results are not driven by data sources or definitions.

7
**Table 1. UNESCO and PISA measures of school closures (in weeks)**

||UN|ESCO|PISA coun|try average|
|---|---|---|---|---|
||Full closures|Full and partial|Student-provided|Principal-provided|
|||closures|length of closures|length of closures|
|Shortest (p10)|5|15|10.6|8.0|
|Very short (p25)|9|29|16.1|13.7|
|Average (p50)|14|38|21.5|17.9|
|Long (p75)|20|54|29.5|29.0|
|Longest (p90)|33|77|38.8|40.4|

# **3. Empirical Strategy**

We model the effect of the pandemic on student achievement by predicting the deviation of the most recent 2022 results from an estimated linear trend in mathematics, reading, and science achievement using comparable data from all PISA rounds, using the model estimated in Jakubowski et al. (2023). We estimate each country’s linear trend separately and include country-level fixed effects to control for unobserved time-invariant country characteristics. The linear regression model is: where _i_ denotes students _, j_ schools, and _k_ countries with _n_ equal to the number of countries. In this case, _D_ equals 1 for 2022 data collected after the pandemic and zero for previous years; captures the impact of the pandemic on student achievement and is a vector of socio-ββ ii demographic variables at a student and country level (gender, socioeconomic status, immigrant XX background). In this case, is our estimate of interest capturing the average departure of achievement in 2022 from the long-term trends in different countries. ττ Identification of the impact of the pandemic on achievement depends on the estimation 8
of achievement trends before the pandemic and the dependency of results on regression specifications. Thus, we perform multiple robustness checks. We first estimate equation (1) on the pooled data from all rounds of PISA with comparable achievement data (8 rounds for reading, 7 for mathematics, and 6 for science). Next, we test for non-linear trends using quadratic terms for time. Then, we re-estimate it for shorter periods deleting data from 2000, then from 2003 and 2006, and so on, leaving at the end the 2018 to 2022 comparison only. Moreover, we estimate results for the OECD countries only, and we use different weighting methods: one estimating results for the whole population of 15-year-olds in participating countries and then using the so-called senate weights which weight each country equally, which is standard approach in OECD reports (see, for example, OECD 2009). Finally, we try different regression specifications, including additional control variables. We compare these results to see if the main estimates of the learning loss are robust to different regression and sample specifications.

The pandemic could affect students in different ways. Disruptions could affect health, economic, and social well-being. Thus, after estimating the overall effect of the pandemic on achievement, we use information on the number of weeks of school closures to show how achievement differs from the time trend depending on the country-average school closure duration. We estimate the following regression model: i = ∑ + ∑ ∗t t 2022 2022 w i i (2) nn nn ii ii=1 ii ii=1 ii ii ii ii YY αα ββ + ττDD + ππDD ∗wwt + γγXX + εε The model is specified as before, but the interaction term between a dummy denoting 2022 data and a measure of the length of school closure in weeks provides an estimate of how the departure in 2022 depends on the length of closures. In this case, captures the departure of achievement in 2022 from country-specific time trends assuming no school closures. In a sense, ττ it captures the impact of other factors than school closures that affected students during the 9
pandemic. We focus here on , which captures the difference in achievement change per week of closures. ππ As discussed in the data section, we use four measures of the length of school closures: (1) UNESCO full closures; (2) UNESCO full and partial closures; (3) PISA student reported closures; and (4) PISA principal reported closures. All are expressed in terms of weeks of school closures. We use and to calculate the learning loss for countries with shorter or longer school closures, comparing results at the percentiles of each school closure measure ττ ππ reported.

This model can be further expanded to test for differences in the impact of the pandemic on boys with girls as a baseline, immigrant students with natives as a baseline, or students with different socioeconomic background. To achieve this, we add interaction terms between all variables in the regression model above and individual student characteristics. In this case, the estimated coefficient for the interaction between 2022 and a dummy for boys, for example, shows the differential impact of the pandemic on boys, after controlling for separate time trends DD by gender in each country. Moreover, the interaction term between a dummy for boys and 2022 w shows how boys were differently affected by weeks of school closures. ii DD ∗wwtFinally, we investigate heterogeneity in the impact of school closures by achievement Finally, we investigate heterogeneity in the impact of school closures by achievement level. Some country studies reported more significant losses among low-achieving students, while others showed similar losses across achievement spectrum. To check how globally learning losses vary among students of different proficiency, we re-estimate the main models using quantile regressions, fitting regressions to explain achievement trends for students at different percentiles of PISA scores. To deal with many dummy variables and interactions, we use recent implementations of fast quantile regression algorithms (Chernozhukov et al. 2022).

PISA data are collected through a complex stratified survey with schools sampled as primary sampling units and students sampled at the second stage. We use the sets of replicate 10
balanced repeated replication (BRR) weights provided in the data to obtain sampling errors. In addition, we estimate measurement errors by estimating variation among point estimates obtained for different plausible values. As up to PISA 2018 databases contain only five plausible values and 10 since 2018, we use only the first five for 2018 and 2022 data. The final standard errors are obtained by the so-called Rubin’s formula, the same way as in the official OECD (2022) publication.

# **4. Results**

The overall learning loss estimates are presented in Table 2 (column 1), along with estimates depending on weeks of closures for four different measures of the length of school closures (columns 2-5). The overall decline in mathematics associated with pandemic-era school closures is 12 points. In 2022, the average within-country SD of mathematics is 86.2. Thus, the decline in achievement is equal to 14.2 percent of a SD. Considering that one year of learning is equivalent to roughly 20 points on the PISA scale (Avvisati and Givord 2023), then this means that on average across around 70 countries students lost an equivalent of more than 7 months of learning.

Results in columns (2) to (5) show that one week of school closures is associated with an additional decline in student achievement. The estimates per week of closures vary across the measures of the length of closures and are not directly comparable as they reflect different definitions of closures. The clearest definition is the full closure defined by UNESCO; thus, this parameter (-0.44) will be used further to estimate the global welfare losses. We compare results calculating the learning loss for different percentiles of each measure of the length of closures. Figure 1 compares the estimated learning loss for the countries with the shortest (10<sup>th</sup> percentile of the length of closures in each of the four measures), short (25<sup>th</sup> ), average (50<sup>th</sup> ), long (75<sup>th</sup> ) and the longest school closures (90<sup>th</sup> ).

11
**Table 2. Impact of Pandemic on Student Mathematics Achievement**

||(1)|(2)|(3)|(4)|(5)|
|---|---|---|---|---|---|
|Departure from linear trend|-12.20***|-5.27***|-8.70***|-4.74***|-7.50***|
|in 2022|(0.43)|(0.79)|(1.03)|(1.14)|(0.88)|
|UNESCO: number of weeks<br>with full closures||-0.44***<br>(0.04)||||
|UNESCO: number of weeks<br>with full or partial closures|||-0.08***<br>(0.02)|||
|PISA: country average<br>student-provided number of<br>weeks of closures||||-0.32***<br>(0.04)||
|PISA: country average|||||-0.21***|
|principal-provided number of<br>weeks of closures|||||(0.03)|
|_Additional controls_||||||
|Individual level: gender,<br>immigrant background, SES|Yes|Yes|Yes|Yes|Yes|
|Country average: gender,<br>immigrant background, SES|Yes|Yes|Yes|Yes|Yes|
|•<br>country fixed effects|Yes|Yes|Yes|Yes|Yes|
|•<br>country*time|Yes|Yes|Yes|Yes|Yes|
|N|2,896,916|2,896,916|2,896,916|2,791,234|2,791,234|

Standard errors in parentheses. * p<0.05; ** p<0.01; *** p<0.001. Full results available in the Annex.

Two things are worth noting when looking at the estimates in Figure 1. First, the learning losses increase with the length of school closures. This shows that the achievement decline is associated with school closures and not only the overall impact of the pandemic on students. Second, the four measures provide similar results when comparing relative standing of countries in terms of the length of school closures. Countries with the shortest closures experienced relatively small losses, 9%-12% of a SD, or to 4-6 months of learning. Countries with average length of school closures experienced losses of 13%-14% SD, or 7 months of learning. Countries with the longest closures, experienced losses of 17%-23% SD, or 9-12 months of learning.

12
**Figure 1. Learning loss depending on the length of school closures**

<!-- Start of picture text --> shortest (p10) short (p25) average (p50) long (p75) longest (p90)<br>0<br>-2<br>-4 full school closures (UNESCO)<br>-6<br>-8 full or partial school closures<br>(UNESCO)<br>-10<br>PISA country average closures<br>-12<br>reported by students<br>-14<br>PISA country average closures<br>-16 reported by principals<br>-18<br>-20<br>Length of school closures in weeks<br>Learning loss in mathematics<br><!-- End of picture text -->

# **Learning loss by gender, immigrant background and achievement level**

The pandemic and school closures could affect students of different backgrounds differently. For example, differences in self-regulation, motivation, home and school support, gender, immigrant background, socioeconomic status, and achievement level could lead to differences in how students learned during the pandemic. We re-estimate the main models for the overall impact of the pandemic and for the effect of the length of school closures using weeks of full closures from UNESCO. To check if students of different background were differently affected, we first interact their characteristics the indicators capturing the departure from the linear time trend in 2022 and with the measures of school closures. To exclude potential differences in pre-pandemic trends, we also estimate separate trends by these characteristics in each country, in addition to controlling for individual and country-average effects of these indicators on achievement as in the previous regressions.

While boys and immigrant students experienced a lower learning loss compared to girls and natives, nevertheless, longer school closures had a more negative effect on both groups (see Table 3). Depending on the closure length measure, each week of school closures was associated with 0.17 or 0.15 points of additional decline in achievement for boys. At around 40 weeks of full school closures, the overall learning loss for boys is the same as for girls and 13
increases for longer closures. Students with an immigrant background lost 0.33 points with every week of full school closures and while in countries with shorter closures their learning loss was smaller, for countries with around 25 weeks of closures their learning loss is equal to that for natives, but it increases for longer closures.

# **Table 3. Impact of Pandemic on Student Mathematics Achievement by gender and**

# **socioeconomic status**

||Overall impact|Impact by length<br>of full school<br>closures<br>(UNESCO)|
|---|---|---|
|Departure from linear trend in 2022|-14.20***|-10.00***|
|Departure from linear trend in 2022 for boys (girls=baseline)|4.18***|6.83***|
|Departure from linear trend in 2022 for immigrant students<br>(natives=baseline)|4.98***|8.00***|
|Departure from linear trend in 2022 for the slope of ESCS|2.62***|3.78***|
|Change in the departure from linear trend in 2022 by:|||
|- Length of closures (in weeks)||-0.29***|
|- Length of closures*Boys||-0.17**|
|- Length of closures*Immigrant||-0.33***|
|- Length of closures*ESCS slope||-0.12***|
|_Additional controls_|||
|Individual level: gender, immigrant background, SES|Yes|Yes|
|Country average: gender, immigrant background, SES|Yes|Yes|
|country fixed effects|Yes|Yes|
|country*time|Yes|Yes|
|Country*time*boys|Yes|Yes|
|Country*time*immigrant|Yes|Yes|
|Country*time*ESCS|Yes|Yes|
|N|2,896,916|2,896,916|

After the pandemic, the relationship between achievement and student socioeconomic background became slightly stronger. The slope of the ESCS index measuring socioeconomic status in PISA increased by 2.62 points. In PISA 2012, the last time mathematics was the main domain before the pandemic, the OECD average slope was around 39 points. Thus, the pandemic increased socioeconomic inequality as measured by the ESCS regression slope by around 7 percent. On the other hand, every week of full school closures diminished this increase 14
by 0.12 points. It means that socioeconomic disparities increased only in countries with relatively short closures, below 22 weeks, and for countries with longer closures the pandemic decreased the relationship between socioeconomic background and mathematics achievement.

Additional analyses using quantile regressions reveal significant differences in the learning losses between students at different achievement levels, but also across countries that vary in the length of school closures. As reported by the OECD, direct comparisons between PISA 2012 and 2022 results in mathematics show similar declines across the achievement spectrum (OECD 2023). Our quantile regression estimates presented in Figure 2 show that, indeed, for countries with the average length of school closures the learning loss is similar for low-, average-, and high-achieving students. However, in countries with relatively short closures, the best students lost very little in terms of achievement and the decline is mostly observed among the average- and low-achieving students. In countries with longer closures, the learning loss is larger for the best students. In countries with the longest closures the lowachieving students lost around 16-17 points, while those at the top of achievement distribution lost 25 points or more.

15
# **Figure 2. Learning loss estimates depending on student achievement quantiles and the**

# **length of closures**

<!-- Start of picture text --> length of closures:<br>very short<br>short<br>average<br>long<br>very long<br>0 2 4 6 8 10<br>achievement quantiles<br>-5<br>-10<br>-15<br>learning loss in mathematics -20<br>-25<br><!-- End of picture text --> These differences in learning loss at different achievement quantiles in countries with short and long closures can be associated with differences in the overall achievement in these countries. Correlating average achievement with the length of school closures reveals that countries with the longest closures are also countries with the lowest achievement in PISA, while countries at the top of the PISA rankings closed schools for much shorter periods, on average. Thus, quantile regression estimates can be interpreted as showing larger losses among the lowest achieving students in countries with short closures and high achievement, similar losses across the achievement distribution in countries with the average length of closures, and larger losses among the highest achieving students in countries with very long closures and low achievement. In general, however, losses are greater for the lowest achievers.

# **Results for reading and science**

In reading, the overall impact of the pandemic is similar to that for mathematics (Table 4). Also, the effects associated with school closures tend to be similar. In science, the overall 16
effect of the pandemic is insignificant. The results for reading and science should be interpreted with caution. The negative trends in student achievement started before the pandemic. Thus, our identification strategy, which relies on stable long-term trends, cannot be applied without doubts to reading and science. Our robustness checks for reading and science results confirm that indeed in these two domains the time trends are not linear, and one cannot distinguish between the effects of the pandemic and the long-term decline in achievement. What causes these long-term declines is beyond the scope of this paper. We can only note that both reading and science were minor domains in PISA 2022, meaning the measurement of student achievement was less precise than in mathematics.

**Table 4. Impact of Pandemic on Student Reading and Science Achievement**

||Readi<br>|ng (2000 to 2<br>|022)<br>|Scien<br>|ce (2006 to 2<br>|022)<br>|
|---|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|(6)|
|Departure from linear|-12.10***|<br>-6.96***|<br>-8.43***|<br>-0.78|6.16***|<br>8.13***|
|trend in 2022|(0.43)|(0.75)|(1.10)|(0.46)|(0.84)|(1.15)|
|UNESCO: number of||-0.33***|||-0.44***||
|weeks with full closures||(0.04)|||(0.04)||
|PISA: country average<br>student-provided number|||-0.14***|||-0.38***|
|of weeks of closures|||(0.04)|||(0.04)|
|_Additional controls_|||||||
|Individual level: gender,|||||||
|immigrant background,|Yes|Yes|Yes|Yes|Yes|Yes|
|SES|||||||
|Country average: gender,|||||||
|immigrant background,<br>SES|Yes|Yes|Yes|Yes|Yes|Yes|
|•<br>country fixed effects|Yes|Yes|Yes|Yes|Yes|Yes|
|•<br>country*time|Yes|Yes|Yes|Yes|Yes|Yes|
|N|3,080,007|3,080,007|2,966,140|2,663,094|2,663,094|2,565,541|

Standard errors in parentheses. * p<0.05; ** p<0.01; *** p<0.001. Full results available in the Annex.

17
# **Robustness checks**

We check the robustness of our results in several ways. First, we run placebo tests by re-estimating the main regression model with a dummy variable capturing the departure from linear trends for every PISA cycle. These placebo effects should be insignificantly different from zero if our assumption of stable trends over time is valid. Figure 3 shows that although estimates for different years vary, representing small variation in results of different PISA rounds, only the departure in 2022 is substantial. This confirms two things. First, PISA results in mathematics were relatively stable before the pandemic. Second, the results in 2022 are distinct as they do not follow the average results from previous rounds.

**Figure 3. Departures from the time trend separately for each year of PISA assessment**

**(95% confidence intervals)**

<!-- Start of picture text --> departure from the time trend assuming (placebo) treatment in various years<br>2003 2006 2009 2012 2015 2018 2022<br>5<br>0<br>-5<br>mathematics<br>-10<br>-15<br><!-- End of picture text --> To further test the robustness of our results, we use different regression specifications, controlling or not for individual and country-average effects. The results were nearly identical, 18
demonstrating that the control variables included help to explain achievement variation but are not driving our results. Moreover, we estimated models with quadratic time trends, allowing for non-linear trends in achievement, and with different years dropped from the analyses. In both cases the results for mathematics did not change substantially. Finally, we re-estimated results with original survey weights representing the target populations of 15-year-olds in every country. These results show learning loss estimates for the global population represented by the PISA sample rather than for the average across countries. Again, the results were almost identical to those presented above using senate weights that give equal weight to every country (see Annex Table A3). For brevity, we present the main estimates only and descriptions of the regression model applied. Full results are available upon request from the authors.

# **5. Global Economic Impact and the Need for Learning Recovery**

Our model predicts that every week of full school shutdown will result in a 0.44-point achievement decrease. Using the UNESCO database, we estimate that the average learning loss across all educational systems is approximately 9 points. The literature offers several estimates on how PISA scores translate to economic growth. Although the range of estimates available in the literature vary depending on the set of countries, estimation methods, and control variables, we used the coefficient of 1.74 (Hanushek and Woessmann 2010). The average loss of 9 points (9 percent of a SD) for 199 educations systems (recognized by the UNESCO school closure database) translates to an average of 0.15 percentage points of GDP growth losses, ranging from zero (if schools remained open) up to 0.57 points. This enormous GDP loss in nominal values using World Bank GDP 2021 data equals $17 trillion of economic loss. These estimates are similar to other studies (see, for example, Azevedo et al. 2021; Doty et al. 2022; Psacharopoulos et al. 2021).

To avert some of these losses, then learning recovery is needed. There is some evidence of recovery from some countries, but the evidence is mixed (Jack et al. 2023; Singh et al. 2022), 19
there are even cases of further losses (Jack et al. 2023; Gambi and de Witte 2023). During the school closures, online tutoring programs were shown to reduce learning loss significantly in several randomized controlled trials (see, for example, Carlana and La Ferrara 2021; Gortazar et al. 2022). A large-scale randomized trial testing low-technology interventions – SMS messages and phone calls – with parents to support their child in Botswana improved learning by 0.12 standard deviation (Angrist et al. 2022). Even low-cost, low- technology interventions can recover learning losses in some cases (Angrist et al. 2023), but more intense instruction will be needed in others (Crawfurd et al. 2023). High dosage tutoring was shown to be effective even before the pandemic; online tutoring variants are just as effective and much less costly (Guryan and Ludwig 2023).

# **Comparisons**

There are only two other international comparisons using global achievement data, both using PIRLS (Jakubowski et al. 2023; Kennedy and Strietholt 2023). For PIRLS, losses are about a year’s worth of learning. But using PISA, we find seven months’ worth of losses. This could be because more time has passed since the pandemic: PIRLS compared results over time to 2021, while PISA looked at results up to 2022. Also, PISA students are much older: 15 years compared to an average age of 10 for PIRLS. Studies investigating student self-regulation skills have shown that in general older students are better able to seek assistance, set goals, plan, monitor, and organize their learning. Hence, it may be that younger students and boys may be more strongly impacted by the school closures compared to older students (Schuurman et al. 2023).

In addition to the global studies, there are several national studies. Review of such studies find an average loss of 0.10 to 0.19 SD (Betthäuser et al. 2023; Di Pietro 2023; Donnelly and Patrinos 2021; Hammerstein et al. 2021; Patrinos et al. 2023; Storey and Zhang 2021; Zierer 2021). These losses are roughly to one-half school years’ worth of learning. A causal 20
estimate of the impact of duration of school closures finds that for every week that schools were closed, learning levels declined by almost 1% of a standard deviation (Patrinos 2023).

# **6. Conclusions**

We estimate the global impact of COVID-19 on student learning on standardized tests over time. We model the effect of closures on achievement by predicting the deviation of the most recent results from a linear trend in mathematics and science achievement using data from all rounds.

COVID-19-induced school closures led to significant student learning losses. Math scores declined from 2018 to 2022 by an average of 12 points, or 14 percent of a standard deviation (SD), roughly equivalent to seven months of learning. Larger declines are recorded for students in schools that faced relatively longer closures and for lower-achieving students. Countries with the shortest closures experienced relatively small losses, while countries with the longest closures experienced losses of around 20 percent of a SD, or up to 12 months of learning. Depending on the closure length measure, each week of school closures was associated with 0.15-0.17 points or around 0.2 percent of a SD of additional decline in achievement for boys. At around 40 weeks of full school closures, the overall learning loss for boys is the same as for girls and increases for longer closures. Students with an immigrant background lost 0.33 points or 0.4 percent of a SD with every week of full school closures and while in countries with shorter closures their learning loss was smaller, for countries with around 25 weeks of closures their learning loss is like the learning loss for natives, and it increases for longer closures.

Distributional analyses reveal significant differences in the learning losses between students at different achievement levels, but also across countries that vary in the length of school closures. For countries with the average length of school closures the learning loss is similar for low-, average-, and high-achieving students. However, in countries with relatively short closures, the best students lost very little in terms of achievement and the decline is mostly 21
observed among the average- and low-achieving students. In countries with longer closures, the learning loss is larger for the best students.

These differences in learning loss at different achievement quantiles in countries with short and long closures can be associated with differences in the overall achievement in these countries. Correlating average achievement with the length of school closures reveals that countries with the longest closures are also countries with the lowest achievement in PISA, while countries at the top of the PISA rankings closed schools for much shorter periods, on average. There are larger losses among the lowest achieving students in countries with short closures and high achievement, and larger losses among the highest achieving students in countries with long closures and low achievement.

22