metadata: authors: 0: author: Patrinos,Harry Anthony
1: author: Jakubowski,Maciej Jan
2: author: Gajderowicz,Tomasz Janusz
count: Europe and Central Asia
docna: 0: docna: Evaluation of Educational Loss in Europe and
            Central Asia
docty: Policy Research Working Paper
abstracts: cdata!: To what extent has the learning progress
            of school-aged children slowed during the COVID-19 pandemic?
            A pre-registered analysis of the first international
            assessment to be published since the pandemic is conducted
            to estimate the impact of COVID-19 on student reading. The
            effect of closures on achievement is modeled by predicting
            the deviation of the most recent results from a linear trend
            in reading achievement using data from all rounds using data
            from 28 countries in Europe and Central Asia. Reading scores
            declined by an average of 20 percent of a standard
            deviation, equivalent to just less than a year of schooling.
            Losses are significantly larger for students in schools that
            faced relatively longer closures. While there are no
            significant differences by sex, it is shown that
            lower-achieving students experienced much larger losses.
display_title: Evaluation of Educational Loss in
            Europe and Central Asia
listing_relative_url: /research/2023/08/34131766/evaluation-educational-loss-europe-central-asia
topicv3: Education m1327048 470,Economic Analysis of Education m1327512 2124
publishtoextweb_dt: 2023-08-08T00:00:00Z
dois: 10.1596/1813-9450-10542
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
author_keys: 0: author_key: 95903
1: author_key: 988487
2: author_key: 1538072
model_extractions: Policy Research Working Paper 10542

# Evaluation of Educational Loss in Europe and Central Asia

_Harry Anthony Patrinos Maciej Jakubowski Tomasz Gajderowicz_ Education Global Practice August 2023
Policy Research Working Paper 10542

# **Abstract**

To what extent has the learning progress of school-aged children slowed during the COVID-19 pandemic? A pre-registered analysis of the first international assessment to be published since the pandemic is conducted to estimate the impact of COVID-19 on student reading. The effect of closures on achievement is modeled by predicting the deviation of the most recent results from a linear trend in reading achievement using data from all rounds using data from 28 countries in Europe and Central Asia. Reading scores declined by an average of 20 percent of a standard deviation, equivalent to just less than a year of schooling. Losses are significantly larger for students in schools that faced relatively longer closures. While there are no significant differences by sex, it is shown that lower-achieving students experienced much larger losses.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at hpatrinos@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Evaluation of Educational Loss in Europe and Central Asia**

**Harry Anthony Patrinos, World Bank**

**Maciej Jakubowski, University of Warsaw**

**Tomasz Gajderowicz, University of Warsaw**

JEL No: I19, I20 Keywords: COVID-19; learning loss; student achievement
# **Introduction**

There is a learning crisis (World Bank 2017). It existed before COVID-19 (Angrist et al. 2021, 2022) and it is global. It has been exacerbated since the school closures began in March 2020 (McKinsey 2022).

COVID-19 caused significant disruption to the global education system. Early in the pandemic, the World Bank (Azevedo et al. 2021) presented simulations of the potential effect of school closures on learning outcomes. It forecast significant declines in the global level of schooling and learning outcomes. School closures could result in a loss of 0.3-1.1 years of schooling and students from the current cohort could, on average, face a reduction of $366 to $1,776 in yearly earnings. Globally, a school shutdown of 5 months could generate learning losses that have a present value of $10 trillion.

Early reviews of the first wave of lockdowns and school closures suggested significant learning loss in a few countries (see, for example, Engzell et al 2021; Haelermans et al. 2022; Jack et al. 2022; Maldonado and De Witte 2022). More recent and thorough analyses of recorded learning losses documented since the beginning of the school closures in March 2020 find even more evidence of learning loss (Donnelly and Patrinos 2021). Robust studies suggest learning losses on average amounting to 0.17 of a standard deviation, or at least one-third of a year's worth of learning (Betthäuser et al. 2023; Hammerstein et al. 2021; Patrinos et al. 2023; Zierer 2021). Most studies observe increases in inequality where certain demographics of students experienced learning losses that were more significant than others. But there are also outliers, countries that managed to limit the amount of loss.

The Progress in International Reading Literacy Study (PIRLS) is the largest international test of 4<sup>th</sup> -grade students’ reading achievement. It is the largest international assessment of reading skills at the primary school level. The 2021 edition was implemented when most countries reopened schools after closures related to the COVID-19 pandemic. Comparing the results from the previous editions conducted in 2001, 2006, 2011, and 2016, one can estimate the impact of the school closures and online teaching in the pandemic on student reading skills. Evidence from individual countries suggests that in many cases students in primary schools suffered larger education losses due to distant teaching in the pandemic. Moreover, the results suggest that the decline is larger among the low-achieving students. Finally, some research shows that the pandemic affected student attitudes, motivation, and well-being.

Negative consequences of the pandemic for students should be addressed through targeted policies aiming at groups of students with the larger losses and addressing their varying needs regarding attitudes and well-being. Changes in student performance might be driven by other factors than the pandemic and school closures. However, the COVID-19-related school closures are the only factor that is common to most countries.

Due to the emergency nature of the COVID-19 pandemic, education systems around the world are facing extreme disruption. At its peak, UNESCO (2020) reported that nearly 1.6 billion learners in more than 190 countries, or 94 percent of the world’s student population, were impacted by educational institution closures. Given the abruptness of the situation, teachers and administrations were unprepared for this transition and were forced to build emergency 2
remote-learning systems almost immediately. In response to this disruption, education researchers are beginning to analyze the impact of these school closures on student learning progress or lack thereof. The term _learning loss_ is commonly used in the literature to describe declines in student knowledge and skills (Pier et al. 2021). Historic data provides researchers with information regarding where student learning should be year over year and is often measured through regular testing. Learning loss occurs when educational progress does not occur at the same rate at which it has historically compared to previous years.

School closures have a large, persistent, and unequal effect on learning. Long-term, nationwide school closures result in learning losses (see, for example, Belot and Webbink 2010). In turn, learning losses can lead to decreased future employment potential and lower earnings (see, for example, Chetty et al. 2014; Currie and Thomas 2001; Psacharopoulos et al. 2021). Online education is an imperfect substitute for in-person learning, particularly for children from low-income families. Peer effects also change: schools allow children from different socioeconomic backgrounds to mix, and this effect is lost when schools are closed. Another factor is the response of parents, some of whom compensate for the changed environment through their own efforts, while others are unable to do so. All this leads to the finding that school closures have a large, persistent, and unequal effect on human capital accumulation (Agostinelli et al. 2022). The channels operating through schools, peers, and parents all contribute to growing educational inequality during the pandemic.

As a result of the first wave of lockdowns and school closures beginning in March 2020, on average, a student has lost about one-third to a half a year’s worth of learning. The magnitude of our findings appears to validate scenarios projected by the World Bank (Azevedo et al. 2021). This has consequences in terms of further education as the lost learning may impact opportunities for higher levels of schooling. There are also long-term future earnings losses associated with lost human capital, with students potentially losing trillions of dollars in future incomes.

# **Data**

The just-released PIRLS 2021 data include results for 65 countries and benchmarking participants. This is the first international picture of student achievement after the COVID-19 pandemic.

Our analysis uses data from 24 education systems of the European Union (22 countries and separate data for the Flemish and French parts of Belgium) and for an additional 7 countries when providing results for all ECA+ countries. We compare PIRLS results from assessment rounds in 2001, 2006, 2011, 2016 and 2021. Overall, our sample includes 582,112 students (459,039 in EU countries) participating in all rounds of PIRLS, representing data for nearly 30 million 4<sup>th</sup> grade students across 20 years. The sample from 2021 – the post-COVID-19 data – includes 157,167 students (120,895 in EU countries), representing the results of nearly 7 million 4<sup>th</sup> graders in ECA+ countries (4 million 4<sup>th</sup> graders in the European Union).

PIRLS data include the plausible values measuring reading achievement and information on student socio-demographic characteristics (e.g., age, grade, and gender). In addition, school principals surveyed by PIRLS provided details on how instruction was affected by COVID-193
related school closures. These data are available for 93% of the sample in ECA+ countries and 98% of the sample in EU countries. We also merge PIRLS data with information from UNESCO on the total number of weeks schools were closed fully or partially during the pandemic. These data are available for all EU countries, although they do not distinguish between primary and secondary levels. Thus, the UNESCO information might be misleading, as primary schools remained open in some countries while secondary schools were closed.

# **Empirical Strategy**

We model the effect of school closures on student achievement by predicting the deviation of the most recent 2021 results from an estimated linear trend in reading achievement using data from all PIRLS rounds. We estimated each country’s linear trend separately and included country-level fixed effects to control for unobserved time-invariant country characteristics. More formally, we estimate a linear regression model on pooled PIRLS data from different years, given by the following equation: In general, PIRLS surveys 4<sup>th</sup> -grade students who are around ten years old at the assessment time. However, students might take the test earlier or later. Some are repeating, and some are advancing a grade. Also, some countries decided to test students in 3<sup>rd</sup> , 5<sup>th,</sup> or even 6<sup>th</sup> grade. Moreover, while most countries sampled the same population over the years, some changed the target population. Finally, due to the pandemic, some countries postponed the assessment to the beginning of the fifth grade. To consider these differences, we took two approaches. In one, we additionally control for student age (10 dummies, one for each age decile, to allow for a non-linear age effect), student grade, and for the average age effect calculated as a mean age for each round and country. In the second approach, we re-estimated all regressions for countries from Wave 1 in PIRLS. These countries did not postpone the assessment due to the pandemic, and according to PIRLS organizers, their data are fully comparable with previous rounds without additional adjustments.

The pandemic could affect student results differently, not only through school closures. Disruptions could also affect health, economic and psychological situation, and general wellbeing. We distinguish these effects by comparing students in countries or schools closed for different periods. In this regard, we used information provided by principals on how their schools were affected by the pandemic. The baseline in the model is more than eight weeks of school closure, and we compare how they differed from a linear trend compared to schools that were closed for between 2 and 8 weeks and those affected for less than two weeks or not closed at all. We also used UNESCO-provided data on the number of weeks of school closures for 4
<!-- Start of picture text --> at ff + T<br>a +fPxtime+ T<br><!-- End of picture text -->
According to the adjusted regression (column 2), on average, across EU countries, reading achievement is lower than predicted by a linear trend by -14.86 points. For ECA+ countries, a similar departure from the linear trend is estimated at -16.43 points. Typically, effects on educational achievement are expressed as percentages of achievement standard deviation. The average within-country standard deviation of reading achievement in 2021 is 74.8 for ECA+ countries, and 73.8 points considering only EU countries. Thus, the average learning loss for EU countries equals 20% of SD, while for all ECA+ countries it is 22%. The unadjusted estimate from column (1) is -9.11 points for EU and -15.27 for ECA+ (12% and 20%, respectively, of within-countries SD of reading achievement). However, this estimate can be biased by within-country changes in student age and grade.

Results in columns (3), (4), and (5) show that the decline in achievement is significantly larger for students in schools that faced relatively more extended closures. In schools that were affected for more than eight weeks (declared by principals), actual achievement was lower than expected by 18.8 points in EU countries. Table 1 and 2 shows how school achievement changed for shorter periods than those affected for more than eight weeks. In schools affected for 2 to 8 weeks, in EU countries achievement declined by around 4 points less, but the difference is not statistically significant. However, in schools that were not closed or affected by two weeks maximum, the decline is lower by around 12 points in EU countries, and by 6 points in ECA+ countries. When comparing the departure from linear trend for this group, the decline is statistically significant at the 0.05 level and equal to 7.2 points. Thus, students in EU countries in schools affected by the pandemic for a relatively short time, two weeks or less, also experienced a decline in reading achievement, although it is half of the loss of students in the most affected schools.

Taking all ECA+ countries, the decline in schools affected for more than 8 weeks is 19.2 points. In schools affected for between 2 and 8 weeks the decline is similar – the estimate is smaller by 1.4 points, and it is not statistically different. However, in schools that were not affected or affected for less than 2 weeks the decline is lower by around 6 points and the difference is statistically significant. Estimating the same regression with these schools as a baseline, shows that students in schools that were not affected by the pandemic or affected for less than 2 weeks also experienced a substantial, statistically significant decline in achievement by around 13.2 points.

6
**Table 1: Student Reading Achievement Trends and Estimated Decline in Scores due to School Closures, EU Countries Only**

||(1)|(2)|(3)|(4)|(5)|
|---|---|---|---|---|---|
|Departure from linear trend in 2021|-9.11***|-14.86***||-0.40|-2.25|
|_Principal report on how many weeks sc_<br>>8 weeks|_hool was affected_||-18.75***|||
|0-2 weeks(difference from_>8 weeks_)|||11.59***|||
|2-8 weeks(difference from >_8 weeks_)|||4.25|||
|_UNESCO data on school closures_<br>weeks with full closures||||-1.14***||
|weeks full or partial closures|||||-0.41***|
|_Additional controls_||||||
|country fixed effects|Yes|Yes|Yes|Yes|Yes|
|country*year|Yes|Yes|Yes|Yes|Yes|
|student grade|No|Yes|Yes|Yes|Yes|
|student age|No|Yes|Yes|Yes|Yes|
|country average student age|No|Yes|Yes|Yes|Yes|
|N of students|459039|458869|449734|458869|458869|

* p<0.05; ** p<0.01; *** p<0.001

Another way of estimating the impact of school closures is to compare the departure from the learning achievement trend between countries with shorter and longer school closures, according to UNESCO country-level data. These results are presented in columns (4) and (5) using the number of weeks of full school closures and the sum of weeks of partial and full school closures. In both cases, for the EU, countries with no closures achieved 2021 results as predicted based on their previous achievement (with an insignificant overall decline by -0.48 points and -2.25 points, depending on the specification). However, the results decline by 1.14 points every week of full closures in EU countries. Thus, ten weeks of closure resulted in 11.9 points decline (or 16% of within-countries SD), while 25 weeks of closure resulted in 29 points decline or 39% of within-countries SD (the most prolonged full closures were in Poland and lasted for 26 weeks). When considering the total number of full or partial closures weeks, the score decline is 0.41 points per week, but in this case, the closures are longer, ranging from 12 weeks (France) to 49 weeks (Latvia). Taking all ECA+ countries, estimates are not significant for the weeks of full closures with the significant average decline of 15.7 points. For the total number of full and partial closures the results show a decline in achievement by 0.32 for every week with an overall decline by 7.9 points. Thus, for 25 weeks of full or partial closures, the estimated decline is 15.9 points, and 23.9 points for 50 weeks.

7
**Table 2. Student Reading Achievement Trends and Estimated Decline in Scores due to School Closures, All ECA+ Countries**

||(1)|(2)|(3)|(4)|(5)|
|---|---|---|---|---|---|
|Departure from linear trend in 2021|-15.27***|-16.43***||-15.69***|-7.90|
|_Principal report on how many weeks sc_|_hool was affected_|||||
|>8 weeks|||-19.16***|||
|0-2 weeks(difference from_>8 weeks_)|||5.97*|||
|2-8 weeks(difference from_>8 weeks_)|||1.43|||
|_UNESCO data on school closures_||||||
|weeks with full closures||||-0.08||
|weeks full or partial closures|||||-0.32*|
|_Additional controls_||||||
|country fixed effects|Yes|Yes|Yes|Yes|Yes|
|country*year|Yes|Yes|Yes|Yes|Yes|
|student grade|No|Yes|Yes|Yes|Yes|
|student age|No|Yes|Yes|Yes|Yes|
|country average student age|No|Yes|Yes|Yes|Yes|
|N of students|582112|581887|570955|581887|581887|

* p<0.05; ** p<0.01; *** p<0.001

# _Estimates for individual countries_

Under some assumptions, we can identify the learning loss for the EU student population as a whole. We take into account differences between countries that are stable over time using country fixed effects. We also control for time-variant effects of changes in the student sampling, namely for changes in individual student age and grade and country-average student age. We estimate the linear trend in student achievement, separately for each country. Assuming that these are valid controls for time variant and time invariant country characteristics, we identify the average departure from the achievement trend for the population of students in EU or ECA+ countries participating in PIRLS 2021. This departure can be interpreted as the learning loss caused by the COVID-19 pandemic as the pandemic was the only factor affecting achievement results in all countries. To separate the overall negative impact of the pandemic from the learning disruption caused by school closures, we also provide comparisons between schools and countries affected by shorter or longer closures.

Providing similar estimates for individual countries is possible with the model presented in the section discussing our empirical strategy. However, in this case, it is not possible to interpret achievement change in 2021 as a measure of learning loss as for individual countries other factors could change between 2016 and 2021 affecting student scores. Anyhow, we provide these results to allow comparisons between countries that are adjusted for the overall time trends and changes in the sampling (age and grade).

Table A1 in the annex compares results for ECA+ countries, using separate data for each country. The first three columns provide results for 2016, 2021, and the difference. These unadjusted results should not be interpreted as measures of the learning loss as they could reflect long-term trends in student achievement. Column 4 compares 2021 results to the 8
country-specific time trend in reading achievement, using data from all PIRLS rounds available for this country. Column 5 provides similar results, but additionally adjusting for student age and grade. However, these adjustments are made using country-specific data only and not using information on age or grade effects from other countries’ data.

Table A2 provides model-based estimates of the learning loss for all ECA+ countries. The first column compares 2021 results to those from the previous rounds. The second column shows the estimate of linear trend departure, with the linear trend fixed at the same value for all ECA+ countries. These estimates are based on strong assumptions that there is no trend in the data or that the trend is the same for all countries. Column 3 reports similar estimates to those reported in Table A1 in column 4 but testing whether the change is statistically different from zero. Finally, column 5 provides results controlling for the country-specific time trend but also for age and grade effects estimated for all ECA+ countries.

Table A3 provides calculations for EU countries that did not participate in PIRLS 2021. Learning loss is estimated for these countries considering the UNESCO data about the number of weeks with full closure and taking the average EU estimates from Table 1 for the overall decline and the decline associated with one week of closures.

_Within-countries differences in the learning decline_ Some research has shown that boys suffered larger learning losses during the pandemic. However, interactions of the COVID-19-related effects for boys show no difference in the decline of achievement by gender. In general, we found no evidence of gender differences in the impact of school closures on reading achievement of 4<sup>th</sup> grade students in EU or ECA+ countries.

However, as expected, quantile regression results show that lower-achieving students experienced much larger losses. Considering principal-provided data, presented for the EU countries on Figure 1A, students at the 1<sup>st</sup> achievement decile (lowest achieving students) in schools affected for more than eight weeks suffer a loss of nearly 25 points compared to less than 15 points for the top students (9<sup>th</sup> decile). Similarly, for students in schools affected between 2 and 8 weeks, the lowest achieving students lost more than 20 points, while those at the top lost less than 10 points. Figure 1A also shows that in schools not affected or affected by less than two weeks, the lowest achieving students lost only 10 points, while those at the top lost around 5 points. Overall, for EU countries, these results suggest that longer school closures affected the low-achieving students more, extending inequalities.

9
# **Figure 1A: Quantile Regression Estimates of the Differences between Schools More or Less Affected by the Pandemic According to their Principals, EU Countries Only**

<!-- Start of picture text --> 0<br>-5<br>-10<br>-15 0-2 weeks<br>2-8 weeks<br>-20 more than 8 weeks<br>-25<br>-30<br>1 2 3 4 5 6 7 8 9<br>Reading achievement decile<br>Departure from achievement trend<br><!-- End of picture text --> Results for the ECA+ countries suggest that in non-EU countries the decline in achievement affected students across the ability spectrum. The top-performing students lost between 10 and 20 points, with larger losses for students in schools more affected by the pandemic. For the lowest achieving students, the losses are larger, similarly to the estimates based solely on EU countries. Students at the 1<sup>st</sup> or 2<sup>nd</sup> decile lost around 25 points or more, both in schools affected for between 2 and 8 weeks, and those affected for more than 8 weeks. That can be compared to the lowest-achieving students in schools affected for two weeks or less. They lost between 15 and 20 points. In general, in ECA+ countries, the learning loss is larger for students in schools more affected by the pandemic, and this effect tend to be larger among the lowestachieving students.

10
**Figure 1B: Quantile Regression Estimates of the Differences between Schools More or Less Affected by the Pandemic according to their Principals, All ECA+ Countries**

<!-- Start of picture text --> 0<br>-5<br>-10<br>-15 0-2 weeks<br>2-8 weeks<br>more than 8 weeks<br>-20<br>-25<br>-30<br>1 2 3 4 5 6 7 8 9<br>Reading achievement decile<br>Departure from achievement trend<br><!-- End of picture text --> Figures 2A and 2B provide comparisons depending on the number of weeks of full and partial school closures according to UNESCO country-level data. In EU countries with around 10 weeks of full or partial closures, the best students have the achievement as expected – similar to the predictions before the pandemic. The weakest students in EU countries with around 10 weeks of closures lost around 10 points, which is considerable but also not a large loss. However, in EU countries with 50 weeks of full or partial closures, the best students lost around 20 points, while those at the 1<sup>st</sup> decile lost nearly 30 points. Overall, according to the estimates based on UNESCO data, the lowest-achieving students in EU countries lost more, but the differences to the best students are similar for countries with shorter and longer school closures.

Figure 2B show similar comparisons for all ECA+ countries. Overall, taking students in all ECA+ countries, the losses are larger. However, similarly to EU countries, the lowest achieving students lost more during the pandemic. The main difference is that in countries with the longest school closures, also students at the highest achievement levels tend to show large learning losses similar to those with lower achievement levels.

11
# **Figure 2A: Quantile Regression Estimates of the Differences between Countries Depending on the Total Number of Weeks of Full and Partial School Closure (UNESCO Data), EU Countries Only**

<!-- Start of picture text --> 5<br>0<br>-5<br>-10<br>-15<br>-20<br>-25<br>-30<br>1 2 3 4 5 6 7 8 9<br>Reading achievement decile<br>10 weeks closure 25 weeks closure 50 weeks closure<br>Departure from achievement trend<br><!-- End of picture text --> 12
# **Figure 2B. Quantile Regression Estimates of the Differences between Countries Depending on the Total Number of Weeks of Full and Partial School Closures (UNESCO Data), All ECA+ Countries**

<!-- Start of picture text --> 0<br>-5<br>-10<br>-15<br>-20<br>-25<br>-30<br>1 2 3 4 5 6 7 8 9<br>Reading achievement decile<br>10 weeks closure 25 weeks closure 50 weeks closure<br>Departure from achievement trend<br><!-- End of picture text --> How do these estimates compare to previous national estimates of learning loss? There are robust estimates of learning loss from 18 ECA countries published prior to PIRLS between 2020 and 2023 (Patrinos 2023). Learning losses range from 0 in Sweden (no school closures) to 0.37 SD in Turkey (40 weeks of closures, the longest in our sample). The average learning loss is 0.17 SDs for those studies; very close to the reading loss of 0.20 SDs recorded in our analysis of PIRLS over time.

Globally, using the same PIRLS data but for all countries – 55 countries and regions, representing more than 16 million 4<sup>th</sup> graders worldwide – the losses are estimated at 0.33 SDs, or just over a year’s worth of learning (Jakubowski et al. 2023). Again, losses are larger for students in schools that faced relatively longer closures and for lower-achieving students. The economic losses may translate into a 0.68 percentage point reduction of GDP growth totaling a global loss of more than $80 trillion.

# **Summary and conclusions**

We estimate the impact of COVID-19 on student reading on standardized tests over time for EU and ECA+ countries. We model the effect of closures on achievement by predicting the deviation of the most recent results from a linear trend in reading achievement using data from all rounds. Scores declined from 2016 to 2021 by an average of 20 percent of a standard deviation, just less than a year’s worth of schooling. This validates the scenarios projected by the World Bank (Azevedo et al. 2021) early in the pandemic. Losses are significantly larger for students in schools that faced relatively longer closures. While there are no significant differences by sex, it is shown that lower-achieving students experienced much larger losses.

13
In EU countries, losses were greater for weaker students and those experiencing longer school closures. This is in line with other studies (Goldhaber et al. 2022; Jack et al. 2022; Patrinos 2023). Losses are larger in all ECA+ countries, compared to EU countries only.

Our results are similar to national micro-studies conducted before the publication of PIRLS 2021 data (Patrinos 2023) and are in line with analyses of the same PIRLS data for the rest of the world (Jakubowski et al. 2023).

While the data go back to 2016, it is likely safe to say that most of the deviation from over time trends is due to the school closures induced by COVID-19. The analyses conducted correspond to the analysis plan set out in our pre-registered protocol (https://osf.io/nrbgz/).

14
# **<mark>Pre-registration</mark>**

<mark>We prospectively registered a protocol of our analysis in the Center for Open Science (OSF) on 10 February 2023 (https://osf.io/jcp2b).</mark>

# **<mark>References</mark>**

<mark>Agostinelli, F., Doepke, M., Sorrenti, G., Zilibotti, F. (2022). When the great equalizer shuts down: Schools, peers, and parents in pandemic times.</mark> _<mark>Journal of Public Economics</mark>_ <mark>206 104574.</mark> <mark>Angrist, N., Djankov, S., Goldberg, P.K., Patrinos, H.A. (2021). Measuring human capital using global learning data.</mark> _<mark>Nature</mark>_ <mark>592(7854): 403-408.</mark> <mark>Angrist, N., Djankov, S., Goldberg, P., Patrinos, H. (2022). Mapping the Global Learning Crisis: Despite record enrollments in school worldwide, learning is limited.</mark> _<mark>Education Next</mark>_ <mark>22(2): 50-56.</mark> Azevedo, J.P., Hasan, A., Goldemberg, D., Geven, K., Iqbal, S.A. (2021). Simulating the potential impacts of COVID-19 school closures on schooling and learning outcomes: A set of global estimates. _The World Bank Research Observer_ 36(1): 1-40.

Belot, M., Webbink, D. (2010). Do Teacher Strikes Harm Educational Attainment of Students? _Labour_ 24(4): 391–406.

Betthäuser, B.A., Bach-Mortensen, A.M., Engzell, P. (2023). A systematic review and metaanalysis of the evidence on learning during the COVID-19 pandemic. _Nature Human Behaviour_ 7(3): 375-385.

Chernozhukov, V., Fernández-Val, I., Melly. B. (2020). Quantile and distribution regression in Stata: algorithms, pointwise and functional inference. Working paper.

Chetty, R., Friedman, J.N., Rockoff, J.E. (2014). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. _American Economic Review_ 104: 2633–2679.

Currie, J., Thomas, D. (2001). Early test scores, school quality and SES: Longrun effects on wage and employment outcomes. _Research in Labor Economics_ 20: 103–132.

Donnelly, R., Patrinos, H.A. (2021). Learning loss during COVID-19: An early systematic review. _Prospects_ 51: 601-609.

Engzell, P, Frey, A, Verhagen, M. (2021). Learning loss due to school closures during the COVID-19 pandemic. _Proceedings of the National Academy of Sciences_ 118(17).

Goldhaber, D., Kane, T., McEachin, A., Morton, E., Patterson, T., Staiger, D. (2022). The Consequences of Remote and Hybrid Instruction during the Pandemic. NBER Working Paper No. w30010.

15
Haelermans, C., Jacobs, M., Van Der Velden, R., Van Vugt, L., Van Wetten, S. 2022, Inequality in the Effects of Primary School Closures due to the COVID-19 Pandemic. _American Economic Association Papers and Proceedings_ 112: 303-307.

Hammerstein, S., König, C., Dreisörner T., Frey, A. (2021). Effects of COVID-19-Related School Closures on Student Achievement-A Systematic Review. _Frontiers in Psychology_ .

Jack, R., Halloran, C, Okun, J, Oster, E. 2022. Pandemic Schooling Mode and Student Test Scores. _American Economic Review: Insights_ .

Jakubowski, M., T. Gajderowicz and H.A. Patrinos. (2023). Global Learning Loss in Student Achievement: First Estimates using Comparable Reading Scores. University of Warsaw Working Paper.

Maldonado, J.E., De Witte, K. (2022). The effect of school closures on standardised student test outcomes. _British Educational Research Journal_ 48(1): 49-94.

McKinsey (2022). How COVID-19 caused a global learning crisis. <u>https://www.mckinsey.com/industries/education/our-insights/how-covid-19-caused-a-globallearning-crisis#</u> Patrinos, H.A. (2023). The Longer Students Were Out of School, the Less They Learned. _Journal of School Choice_ 17(2): 161-175.

Patrinos, H., Vegas, E., Carter-Rau, R. (2023). An Analysis of COVID-19 Student Learning Loss. In Jonathan Hamilton (Ed.), _The Oxford Research Encyclopedia of Economics and Finance_ . Oxford: Oxford University Press.

Pier, L., Hough, H. J., Christian, M., Bookman, N., Wilkenfeld, B., Miller, R. (2021). COVID19 and the educational equity crisis: Evidence on learning loss from the CORE data collaborative. Policy Analysis for California Education.

Psacharopoulos, G., Collis, V., Patrinos, H.A., Vegas, E. (2021). The COVID-19 cost of school closures in earnings and income across the world. _Comparative Education Review_ 65(2): 271287.

UNESCO (2020). UN Secretary-General warns of education catastrophe, pointing to UNESCO - estimate of 24 million learners at risk of dropping out. <u>https://en.unesco.org/news/secretary general-warns-education-catastrophe-pointing-unesco-estimate-24-million-learners-0</u> World Bank (2017). _World Development Report 2018: Learning to Realize Education’s Promise_ . Washington DC: World Bank.

Zierer, K. (2021). Effects of Pandemic-Related School Closures on Pupils’ Performance and Learning in Selected Countries: A Rapid Review. _Education Sciences_ 11(6): 252.

16
# **Annex: Country-Specific Estimates of Achievement Change**

# **_Table A1. Raw Estimates of Achievement Change_**

||||Change from|Departu|re from time|
|---|---|---|---|---|---|
|Country|2016|2021|2016 to 2021||trend|
|||||No<br>controls|Control for<br>age & grade|
|Azerbaijan|472.3|440.3|-32.0|-42.0|-42.9|
|Austria|540.8|529.8|-11.0|-8.6|-7.7|
|Bulgaria|551.5|539.9|-11.6|-0.9|-0.2|
|Croatia|N/A|556.6|N/A|N/A|N/A|
|Cyprus|N/A|510.9|N/A|N/A|N/A|
|Czechia|543.3|539.7|-3.7|-7.9|-6.3|
|Denmark|547.5|539.0|-8.5|-11.7|-10.9|
|Finland|566.0|549.3|-16.7|-14.8|-14.9|
|France|511.2|513.7|2.5|5.1|7.3|
|Georgia|488.3|494.0|5.7|-5.7|-9.3|
|Germany|537.3|524.0|-13.4|-14.9|-12.0|
|Hungary|554.2|539.4|-14.7|-12.8|2.7|
|Ireland|566.6|577.3|10.7|-4.3|-5.1|
|Italy|548.0|537.2|-10.8|-11.3|-12.2|
|Kazakhstan|536.0|503.6|-32.5|N/A|N/A|
|Latvia|557.8|527.7|-30.1|-32.3|-28.3|
|Lithuania|548.3|552.2|3.9|13.2|13.9|
|Malta|452.0|514.6|62.6|87.6|84.0|
|Netherlands|544.9|527.2|-17.6|-13.5|-15.9|
|Norway|558.9|538.8|-20.2|-23.8|-28.4|
|Poland|564.6|549.1|-15.5|-31.8|-30.3|
|Portugal|527.8|519.8|-8.0|5.0|6.3|
|Russian Federation|580.8|567.1|-13.7|-35.1|-31.7|
|Slovak Republic|534.8|529.1|-5.7|-14.9|-15.6|
|Slovenia|542.5|519.7|-22.8|-37.3|-34.2|
|Spain|527.7|521.2|-6.5|-12.1|-12.1|
|Sweden|555.2|543.5|-11.6|-2.3|-1.9|
|England|558.7|557.6|-1.1|-0.4|-0.8|
|Northern Ireland|564.6|565.9|1.3|-4.9|-11.7|
|Belgium (Flemish)|525.1|510.7|-14.3|-3.3|-4.7|
|Belgium (French)|497.5|494.3|-3.2|-4.4|-5.5|

17
# **_Table A2. Model-based estimates of achievement change_**

## **_<u>Change in 2021 scores</u>_**

||No controls|Control for<br>general linear<br>time trend in<br>achievement|Control for general<br>linear time trend in<br>achievement, age<br>& grade effects|Control for<br>country-specific<br>linear time trend<br>in achievement|Control for<br>country-specific<br>linear time trend<br>in achievement,<br>age & grade<br>effects|
|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|
|Azerbaijan|-27.55***|-35.56***|-30.28***|-41.98***|-44.13***|
|Austria|-6.26**|-17.47***|-16.26***|-8.56*|-12.19**|
|Bulgaria|-5.78|-20.71***|-10.41**|-0.89|-2.08|
|Croatia|3.51|-7.57*|-37.78***|-16.43|-49.63***|
|Cyprus|16.89***|-5.27|-9.55|-23.00|-43.72|
|Czechia|-1.72|-15.45***|-13.75***|-7.91*|-16.36***|
|Denmark|-10.27***|-21.58***|-20.40***|-11.74**|-17.21***|
|Finland|-17.65***|-25.99***|-23.21***|-14.84**|-12.76**|
|France|-5.61*|-19.25***|-12.92***|5.12|-6.92|
|Georgia|12.14***|0.94|-45.43***|-5.71|-63.96***|
|Germany|-17.34***|-31.85***|-25.78***|-14.94***|-21.53***|
|Hungary|-7.35*|-21.89***|-53.35***|-12.82**|-42.78***|
|Ireland|18.13***|9.86***|-25.56***|-4.26|-26.22***|
|Italy|-8.07***|-22.11***|-17.80***|-11.26**|-15.37***|
|Kazakhstan|-32.46***|-38.00***|-71.48***|-42.43***|-74.69***|
|Latvia|-19.16***|-35.68***|-57.26***|-32.31***|-58.26***|
|Lithuania|12.64***|-2.58|-30.89***|13.18***|-16.91*|
|Malta|50.17***|41.86***|46.53***|87.55***|78.27***|
|Netherlands|-20.87***|-34.81***|-29.01***|-13.45***|-20.46***|
|Norway|23.52***|9.56***|-37.64***|-23.82***|-31.99***|
|Poland|13.90***|2.45|-34.64***|-31.76***|-28.94***|
|Portugal|-15.25***|-23.86***|-32.24***|5.01|-22.33**|
|Russian<br>Federation|9.94*|-4.71|-12.68**|-35.09***|-22.42***|
|Slovak<br>Republic|0.66|-14.20***|-13.67***|-14.90***|-14.91***|
|Slovenia|-3.71|-17.75***|-24.13***|-37.31***|-37.30***|
|Spain|3.06|-7.74**|-6.86**|-12.14**|-14.91***|
|Sweden|-8.95***|-23.14***|-17.90***|-2.34|-2.67|
|England|6.79**|-7.15*|-0.23|-0.39|-0.48|
|N Ireland|4.21|-3.93|-29.01***|-4.86|-29.57***|
|Belgium<br>(Flemish)|-25.04***|-35.97***|-33.39***|-3.34|0.61|
|Belgium<br>(French)|-6.77*|-17.65***|-13.94***|-4.38|-1.43|
|<br>N|582,112|582,112|581,887|582,112|581,887|

Note: * p<0.05; ** p<0.01; *** p<0.001 18
**Table A3. Calculations of the learning for EU countries not participating in PIRLS 2021**

|Country|UNESCO weeks with<br>full closures|Expected average<br>learning loss for<br>weeks with full<br>closures|Expected learning<br>loss for students at<br>the first achievement<br>decile|Expected learning<br>loss for students at<br>the ninth achievement<br>decile|
|---|---|---|---|---|
|Estonia|15|-17.50|-22.57|-12.62|
|Greece|18|-20.92|-25.51|-16.91|
|Romania|22|-25.48|-29.43|-22.63|
|Luxembourg|9|-10.66|-16.69|-4.04|

Note: Calculations using UNESCO data and estimates for the average loss in EU countries per week of school closures from Table 1, and quantile regression estimates.

19