metadata: authors: 0: author: Patrinos,Harry Anthony
1: author: Vegas,Emiliana
2: author: Carter-Rau,Rohan
count: World
docna: 0: docna: An Analysis of COVID-19 Student Learning Loss
docty: Policy Research Working Paper
abstracts: cdata!: COVID-19 caused significant disruption
            to the global education system. Early reviews of the first
            wave of lockdowns and school closures suggested significant
            learning loss in a few countries. A more recent and thorough
            analysis of recorded learning loss evidence documented since
            the beginning of the school closures between March 2020 and
            March 2022 finds even more evidence of learning loss. Most
            studies observed increases in inequality where certain
            demographics of students experienced more significant
            learning losses than others. But there are also outliers,
            countries that managed to limit the amount of loss. This
            review aims to consolidate all the available evidence and
            documents the empirical findings. Thirty-six robust studies
            were identified, the majority of which find learning losses
            on average amounting to 0.17 of a standard deviation,
            equivalent to roughly a one-half year’s worth of learning.
            These findings confirm that learning loss is real and
            significant, even compared to the first year of the
            pandemic. Further work is needed to increase the quantity of
            studies produced, and to ascertain the reasons for learning
            loss and in a few cases mitigation of loss.
display_title: An Analysis of COVID-19 Student
            Learning Loss
listing_relative_url: /research/2022/05/33810279/analysis-covid-19-student-learning-loss
topicv3: Economics of Education m1327511 1540
publishtoextweb_dt: 2022-05-04T00:00:00Z
dois: 10.1596/1813-9450-10033
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
author_keys: 0: author_key: 95903
1: author_key: 167386
2: author_key: NA
model_extractions: Policy Research Working Paper 10033 An Analysis of COVID-19 Student Learning Loss _Harry Anthony Patrinos Emiliana Vegas Rohan Carter-Rau_ Education Global Practice May 2022
Policy Research Working Paper 10033

# **Abstract**

COVID-19 caused significant disruption to the global education system. Early reviews of the first wave of lockdowns and school closures suggested significant learning loss in a few countries. A more recent and thorough analysis of recorded learning loss evidence documented since the beginning of the school closures between March 2020 and March 2022 finds even more evidence of learning loss. Most studies observed increases in inequality where certain demographics of students experienced more significant learning losses than others. But there are also outliers, countries that managed to limit the amount of loss. This review aims to consolidate all the available evidence and documents the empirical findings. Thirty-six robust studies were identified, the majority of which find learning losses on average amounting to 0.17 of a standard deviation, equivalent to roughly a one-half year’s worth of learning. These findings confirm that learning loss is real and significant, even compared to the first year of the pandemic. Further work is needed to increase the quantity of studies produced, and to ascertain the reasons for learning loss and in a few cases mitigation of loss.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at hpatrinos@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **An Analysis of COVID-19 Student Learning Loss**

**Harry Anthony Patrinos Emiliana Vegas Rohan Carter-Rau** World Bank Harvard University Brookings Institution

JEL codes: I14, I15, I20, I24, I25, O151 Keywords: COVID19, Education, Learning loss, Systematic review
# I. Introduction

The COVID-19 pandemic led to school closures around the world, affecting almost 1.6 billion students (UNESCO 2020). Even short disruptions in a child’s schooling have significant negative effects on their learning and can be long lasting. The capacities of education systems to respond to the crisis by delivering remote learning and support to children and families have been diverse yet uneven. In response to this disruption, education researchers are beginning to analyze the impact of these school closures on student learning loss. The term learning loss is commonly used in the literature to describe declines in student knowledge and skills. Outside the classroom, these losses may translate to greater long-term impacts (Azevedo et al 2021; Hanushek and Woessmann 2020; Psacharopoulos et al 2021). But there are also outliers, countries that managed to limit the amount of loss.

Research on the actual impact of COVID-19 on learning progress is beginning to emerge. School closures have a large, persistent, and unequal effect on learning. Online education is an imperfect substitute for in-person learning, particularly for children from low-income families. Peer effects also change; schools allow children from different socio-economic backgrounds to mix, and this effect is lost when schools are closed. Another factor is the response of parents, some of whom compensate for the changed environment through their own efforts, while others are unable to do so. All this leads to the finding that school closures have a large, persistent, and unequal effect on human capital accumulation. In the USA, high school students from low-income communities suffer a learning loss of 0.4 standard deviation after a one-year school closure, whereas children from high-income neighborhoods initially remain unscathed (Agostinelli et al. 2022). The channels operating through schools, peers, and parents all contribute to growing educational inequality during the pandemic. Early systematic reviews based on limited data from high-income countries 2
suggest large learning losses and increased learning inequality. Most recently, Donnelly and Patrinos (2021) find average learning losses of about 0.13 standard deviation (SD) in seven highincome countries. Hammerstein et al. (2021) find learning losses of 0.10 SD, Storey and Zhang (2021) estimate losses at 0.15 SD, and Zierer (2021) finds losses of 0.14 SD. Moscoviz and Evans (2022) also find that learning losses are mostly negative.

In this paper, we focus on the question, _to what extent did COVID-19 lead to student learning losses across the world?_ Our review covers learning loss evidence documented during the first two years of the pandemic. Previous reviews have documented learning loss, and we build on those efforts.

We make three contributions. First, we develop a comprehensive review that consolidates the research that has been completed related to the impact of COVID-19 on student learning progress and build a public use database. Second, we respond to the research question of the extent to which there is evidence of pandemic-related learning losses. Third, based on our review, we identify significant gaps in the literature and provide relevant guidance for further research.

We begin by describing the methods used in this review to identify and collect the articles analyzed. We then present an analytical review and our strategy for reporting effects, where each article was categorized by its geographical region studied, length of school closure, education level of students analyzed, sample size, subject(s) analyzed, documented learning impact, and presence of increased inequality. Next, we discuss the findings of our review of the emerging evidence on learning losses. Last, we briefly summarize the key findings and discuss areas for future research.

3
This paper aims to identify, review and analyze the existing evidence on the effect of the COVID19 pandemic and school closures on learning outcomes progress among school-age children and youth. The specific research questions are:

1. What is the impact of the COVID-19 pandemic and school closures on learning progress among school children and youth?

2. How does the impact differ across socioeconomic background, age, boys and girls, subjects, sub-national contexts, ethnicity, etc. (data permitting)?

# II. Search strategy and identification methods

We searched the following scholarly sources:

- ECONLIT, Google Scholar, PubMed, Education Resources Information Center, and Cochrane Library

- Coronavirus Research Database, Education Database (ProQuest), Web of Science

- Preprint and working paper repositories: SSRN, MPRA, IZA, NBER, OSF Preprints, PsyArXiv, SocArXiv, EdArXiv

- Reach out to researchers.

To conduct the search, the keywords “COVID-19”, “coronavirus”, “2019-ncov”, “sars-cov-2”, or “cov-19”, were used in combination with “learning loss”, “learning slide”, “education gap”, or “achievement gap”. We focused on learning loss among school-age children across all subjects, with special attention to the most reported subjects of math and reading. School-age children and youth are defined as being between ages 5 and 18, thus excluding both kindergarten and higher education.

4
We considered all observational research including, but not limited to, studies using quantitative data that is longitudinal or cross-sectional, individual or aggregate (e.g., at the level of schools) on the effect of the COVID-19 pandemic on learning loss among students.

To be included in our sample, the outcome measure must be a valid measure of learning progress. These include measures using scores from school-based tests or from assessments administered by researchers or international assessment programs, including EGRAs, and any study that can be scaled to the Harmonized Learning Outcomes (HLO) (Angrist et al 2021). In addition, studies must have measured learning loss with pre-COVID control data, and during or post-COVID treatment data. Studies using projections and simulations were excluded.

Secondary research such as systematic reviews were excluded from the analysis but used to identify further relevant studies. Given the focus on the COVID-19 pandemic, we restricted our search to papers published between March 1, 2020, and March 1, 2022.

Studies were screened according to the following selection criteria: Studies that conducted student analyses and reported impacts on learning progress (either positive, negative, or insignificant) because of COVID-19 school disruptions were included. Factors for rejecting studies from our review included the absence of student analyses or recorded impacts on learning progress, analyses occurring before the onset of COVID-19, or hypothesized results. We collected study details such as author name(s), sample size, information on the age of students in sample, length of closures, measures of social background recorded, outcomes measured, follow-up period(s), and the effect sizes. We recorded the data in an MS Excel spreadsheet. We assessed the methodological rigor of each study and subsequently rated the robustness of the overall body of evidence across all outcomes.

5
For studies that contained data on individual learning loss, we aggregated these into a composite measure of learning for the most common specific learning areas examined in all relevant studies (such as math, reading). If sufficient evidence was available, we also aggregated estimates of learning loss in other specific learning areas. If there were multiple model specifications including controls, we extracted the relevant coefficients from the models that allowed for comparison of effect sizes across studies.

Our initial search consisted of a google scholar query using the keywords (“COVID-19” OR “coronavirus” OR "pandemic)" AND ("learning loss" OR "learning slide"), restricted to publications between March 2020 and March 2022. This search revealed 3,840 results. Of these, 167 directly addressed learning loss during the pandemic. Of these studies, 32 attempted to directly measure learning loss caused by the COVID-19 pandemic. We then performed further searches of the other databases, and of the working paper repositories. This revealed an additional 13 studies. Finally, while evaluating these studies, we pulled a further 36 papers from reference lists and other review papers. This resulted in a total of 81 studies that examine learning loss due to the COVID19 pandemic. Five of these studies were duplicates published in different formats and were dropped from the analysis. Another five studies did not contain a direct measure of learning loss in the form of pre- and post-pandemic data and were also dropped from the analysis. Finally, we dropped from our analysis four studies that examined learning loss only at the higher education level.

III. Analytical review and effects reporting We carefully reviewed the remaining 67 studies and rated their methods scored on a scale of 1 (lowest quality methods) to 5 (highest quality methods). We rated 1 those papers that presented learning loss through a comparison of pre- and post-COVID means, without statistical analysis for 6
variation to account for potential confounding variables, or random sampling techniques. We rated 2 those papers that performed difference of means tests but used convenience sampling or had small sample sizes. We rated 3 papers that performed difference of means tests, had random or representative sampling, and a relatively large sample size. We rated 4 papers that performed additional statistical analysis beyond a comparison of means, including strong sampling and basic individual or school level controls. Finally, we rated 5 those papers that contained sophisticated statistical analysis, with strong controls and sampling. This systemic review focuses on the 35 studies that are scored 3, 4, and 5. The findings of studies scored 1 and 2 are included in Appendix 1.

When reporting learning loss effect size, we present findings in terms of standard deviation change of the treatment (post-COVID) group from the pre-COVID control group. As each study uses a different measure of learning loss, losses reported in percentiles and points are not easily comparable. Following Hanushek and Woessmann (2020), we convert results presented in months of loss to standard deviations, with 1 school year of learning equal to 0.33 standard deviation. Unfortunately, many papers do not present results in standard deviation or months of learning losses, and do not have sufficient information for us to calculate these. These results are excluded from Table 1 and Figure 1, but they are included in the overall analysis. For these papers, we primarily discuss the direction of their findings, as the quantitative findings are not comparable across papers.

Where possible, we also present heterogeneous effects in standard deviations. Many studies include only some of these effects, meaning the number of study results reported varies substantially by effect.

7
# IV. Learning losses due to COVID-19

Our final database consists of 35 robust studies and reports documenting learning loss, representing data from 20 countries (see Table 1). Most studies (32) find evidence of learning loss. Of the 35 studies reporting learning loss, 27 reported findings in a comparable effect size format. As shown in Figure 1, most studies found learning losses ranging from 0.25 to 0.12 SD. In five studies, learning losses were even greater. The average learning loss across these studies is 0.17 standard deviation – which equates to over half a school year of learning loss.

8
Table 1: Final database of robust studies of learning loss due to COVID-19

||**Source**|**Country**|**Closure**<br>**length**<br>**weeks**<sup>**1**</sup>|**Average**<br>**learning**<br>**losses (SD)**|**Equity**<br>**impact**|**Sample size**|
|---|---|---|---|---|---|---|
|1|Ardington et al 2021|South Africa|22|-0.22|Yes|5,810|
|2|Arenas and Gortazar 2022|Spain||-0.05|Yes|41,000|
|3|Asakawa and Ohtake 2021|Japan|11||Yes|5,400|
|4|Bielinski et al 2021|<br>United States||-0.14|N|not reported|
|5|Birkelund and Karlson 2021|Denmark|8|0|Yes|200,000|
|6|Borgonovi and Ferrara 2022|Italy|||Yes|850,000|
|7|Chen et al 2021|United Arab Emirates|||N/A|1,920|
|8|Clark et al. 2021|China|7|-0.22|N/A|<br>1,835|
|9|Contini et al 2021|Italy|15|-0.19|Yes|1,539|
|10|Domingue et al 2021|United States||-0.05|N|100,000|
|11|<br>Engzell et al 2021|Netherlands|8|-0.08|Yes|<br>350,000|
|12|EPI 2021|England||-0.09|Yes|180,000|
|13|Gambi and De Witte 2021|<br>Belgium||-0.23|Yes|213000|
|14|Gore et al. 2020|<br>Australia|8|0|N|4,800|
|15|Haelermans et al 2021|Netherlands|10|-0.17|Yes|201,819|
|16|Halloran et al 2021|United States|||Yes|<br>11,700|
|17|Hevia et al 2021|Mexico|48|-0.55|Yes|3,161|
|18|Hicks and Faulk 2022|United States|||Yes|not reported|
|19|Jakubowski 2022|Poland|29|-0.30|N/A|<br>4,581|
|20|Kogan and Lavertu 2021|United States||-0.23|Yes|124,700|
|21|<br>Korbel and Prokop 2021|Czech Republic||-0.11|N/A|2,234|
|22|Kuhfeld et al 2022|United States||-0.19|Yes|5,400,000|
|23|Lichand et al 2021|Brazil||-0.32|Yes|7,000,000|
|24|Locke et al 2021|United States||-0.15|No|<br>55,000|
|25|Ludewig et al 2022|Germany||-0.14|N/A|4,290|
|26|<br>Maldonado and Witte 2022|<br>Belgium|9|-0.18|Yes|1,300|
|27|Philadelphia 2021|United States|||N/A|65,000|
|28|<br>Pier et al 2021|United States||-0.08|N/A|100,000|
|29|Schult et al 2021|Germany|10|-0.08|N/A|<br>80,000|
|30|Schuurman et al 2021|<br>Netherlands|8|-0.09|Yes|886|
|31|Skar et al 2021|Norway|7|-0.24|N/A|2,453|
|32|Spitzer, Musslick|<br>Germany|8||N/A|2,500|
|33|<br>Tomasik et al 2020|<br>Switzerland|8|-0.20|N/A|<br>28,685|
|34|van der Velde|Netherlands|8||N/A|133,450|
|35|Chaban et al 2022|Russian Federation|8-20|-0.27|No|165,740|

> 1 We report the number of weeks of school closures as reported in each study.  When the study did not report the length of school closures, we also do not report it in Table _1_ or use it in our analyses.

9
<!-- Start of picture text --> 14<br>12<br>» 10<br>o<br>:<br>Bm 8<br>Ge<br>io)<br>3)<br>a6<br>|<br>Z<br>4<br>2<br>/ =z<br><-0.5 (-0.5, -0.375] (-0.375, -0.25] (-0.25, -0.125] —(-0.125,0]<br>Learning Loss (in SD)<br><!-- End of picture text -->
secondary schools. Some countries had good internet connectivity and students had access to devices; these systems were well positioned to provide quality distance education online. Other countries, with limited internet connectivity and access to devices, broadcast lessons and used radio or other mass media to provide remote education.

As Table 2 shows, the longer the schools remained closed, the greater were the learning losses. For the 19 countries for which we have robust learning loss data, average school closures were 15 weeks, leading to average learning losses of 0.18 standard deviation. Put another way, for every week that schools were closed, learning declined by 1.2 point, or 0.01 standard deviation, on average.

Most of our robust data come from 12 European countries. Given similar starting conditions overall (such as GDP per capita, access to connectivity and devices), these countries provide compelling evidence that the pandemic-related school closures led to learning losses. In these European countries, average school closures were 11 weeks in duration, and average learning losses were 0.16 standard deviation. Therefore, for every week of school closure, learning losses were about 1.5 points, or 0.015 standard deviation, per month.

11
Table 2: Length of school closures and average learning losses, by country

|**Country**|**Closure length weeks**|**Average learning losses**<br>**(SD)**|**Source**|
|---|---|---|---|
|Mexico|48|-0.55|Hevia et al 2021|
|Brazil|26|-0.32|Lichand et al 2021|
|Poland|20|-0.3|Jakubowski 2022|
|Russian<br>Federation|14|-0.27|Chaban et al 2022|
|Norway|7|-0.24|Skar et al 2021|
|China|7|-0.22|Clark et al. 2021|
|South Africa|22|-0.22|Ardington et al 2021|
|Switzerland|8|-0.2|Tomasik et al 2020|
|Italy|15|-0.19|Contini et al 2021|
|Belgium|9|-0.18|Maldonado and Witte 2022|
|Netherlands|10|-0.17|Haelermans et al 2021|
|Germany|10|-0.14|Ludewiget al 2022|
|United<br>States|23|-0.14|Average of: Bielinski et al<br>2021; Domingue et al 2021;<br>Kogan and Lavertu 2021;<br>Kuhfeld et al 2022; Locke et al<br>2021;Pier et al 2021|
|Czech<br>Republic|9|-0.11|Korbel and Prokop 2021|
|England|10|-0.09|EPI 2021|
|Spain|12|-0.05|Arenas and Gortazar 2022|
|Australia|8|0.00|Gore et al. 2020|
|Denmark|8|0.00|Birkelund and Karlson 2021|
|Japan|11|0.0|Asakawa and Ohtake 2021|

We also observed notable variation in learning loss by subject. The subjects most frequently analyzed in the reviewed studies are reading and mathematics. Of the 11 studies to include both, in 9 studies learning loss in mathematics exceeded that of reading, while reading losses exceeded mathematics in 2 studies.

Many studies have examined learning loss by gender, although the findings are not consistent. While some studies report greater learning loss among girls (Lichand et al 2021), others find
greater learning loss among boys (Birkelund and Karlson 2021; Schuurman et al. 2021). Seven other studies find no statistical difference in learning loss by gender.

Learning losses are significant in countries reporting losses through descriptive statistics. For example, adolescent girls’ literacy and numeracy scores declined by more than 6 percent in Bangladesh (Amin et al 2021). In India, the share of grade 3 children in government schools who can perform simple subtraction decreased from 24 percent in 2018 to only 16 percent in 2020 and the share who can read a grade 2 level text decreased from 19 percent in 2018 to 10 percent in 2020 (ASER 2021a). In Pakistan, the proportion of children in classes 1-5 who can read a story declined from 24 percent in 2019 to 22 percent in 2021 (ASER 2021b). There are reported learning losses in one college in Sri Lanka (Sayeejan and Nithlavarnan 2018). In Uganda, the percentage of learners rated proficient in literacy in English and numeracy in 2021 dropped by 5 percent and 13 percent from that of 2018 (NAPE 2021). In Canada, grade 2 and 3 students reading assessments declined by 4 to 5 points (Georgiou 2021). In the Republic of Korea, there was a significant decrease in scores in for medical school students (Kim et al. 2021).

# V. Discussion

As a result of the first wave of COVID-19-induced lockdowns and school closures beginning in March 2020, a student has lost on average about one-third to one-half year’s worth of learning. These learning losses may impact a student’s education trajectory, as the lost learning is likely to limit opportunities to advance to higher levels of schooling. There are also long-term future earnings losses associated with lost human capital, with students potentially losing trillions of dollars in future income (Psacharopoulos et al 2021).

Nevertheless, there are some causes for optimism from the existing research. In Nara City, Japan, math achievement scores had increased six months after COVID-related school closures. While 13
the initial closures had brought down scores, responsive policies largely overcame this decline (Asakawa and Ohtake 2021). During the school closures, students watched virtual lessons streamed by their teachers, carried out independent work, and submitted paperwork for revision by their teachers. Nara City also decreased its summer vacation in response to the school closures, from 34 days to 16 days. Although a broadly successful response, the school closures impacted lower income students significantly more, and these students also showed slower growth than their peers upon returning to school. Another positive finding comes from England, where the Educational Policy Institute found learning loss to decline substantially after schools had reopened. In autumn 2020, students were found to be behind by an average of 3.6 months in mathematics, but by the summer of 2021 the loss had decreased to an average of 2.2 months.

Learning loss evidence is mixed in France, with on average very low levels of learning loss experienced thus far, though there are still equity effects with more disadvantaged students suffering more (Thorn and Vincent-Lancrin 2021). However, no learning loss is evident in Denmark and no equity effect (Birkelund and Karlson 2021). It seems that Danish children received good home support and their reading behavior increased significantly because of the lockdown of schools (Reimer et al 2021). In Sweden, where primary schools did not close during the pandemic, there are no reported learning losses (Falth et al 2021). In the state of New South Wales, Australia, there are no significant differences between 2019 and 2020 in student achievement growth; however, learning did decrease in the least advantaged schools (Gore et al 2021).

As this emerging evidence suggests, the impact of COVID-19 on student learning worldwide has been substantial, and inequality in learning across more advantaged and disadvantaged groups is likely going to grow over time. We plan to continue to assess these trends over time and to 14
evaluate longer-term impacts on individuals’ education and labor market trajectories as well as productivity and economic development globally.

15