metadata: authors: 0: author: Patrinos,Harry Anthony
count: World
docna: 0: docna: The Longer Students Were Out of School, the
            Less They Learned
docty: Policy Research Working Paper
abstracts: cdata!: COVID-19 led to school closures and
            emergency remote learning systems. It is feared that
            students learned less when they were remote. This paper
            analyzes school closures during the pandemic using a unique
            data base. The determinants of the duration of school
            closures estimates were used to instrument school closures
            &#x2013; stringency of lockdown and vaccination
            &#x2013; and causally estimate the impact of duration on
            learning. It is estimated that for every week that schools
            were closed, learning levels declined by almost 1 percent of
            a standard deviation. This means that a 20 week closure, for
            example, would reduce learning outcomes by 0.20 standard
            deviation, almost one year of schooling.
display_title: The Longer Students Were Out of
            School, the Less They Learned
listing_relative_url: /research/2023/04/34048698/longer-students-out-school-less-learned
topicv3: Education m1327048 470,Learning Assessment m1327283 1555,Coronavirus (COVID-19) m1939478 4828
publishtoextweb_dt: 2023-04-24T00:00:00Z
dois: 10.1596/1813-9450-10420
author_keys: 0: author_key: 95903
model_extractions: Policy Research Working Paper 10420 The Longer Students Were Out of School, the Less They Learned _Harry Anthony Patrinos_ Europe and Central Asia Region Office of the Chief Economist April 2023
Policy Research Working Paper 10420

# **Abstract**

COVID-19 led to school closures and emergency remote learning systems. It is feared that students learned less when they were remote. This paper analyzes school closures during the pandemic using a unique data base. The determinants of the duration of school closures estimates were used to instrument school closures—stringency of lockdown and vaccination—and causally estimate the impact of duration on learning. It is estimated that for every week that schools were closed, learning levels declined by almost 1 percent of a standard deviation. This means that a 20 week closure, for example, would reduce learning outcomes by 0.20 standard deviation, almost one year of schooling.

This paper is a product of the Office of the Chief Economist, Europe and Central Asia Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The author may be contacted at hpatrinos@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **The Longer Students Were Out of School, the Less They Learned**

## <u>Harry Anthony Patrinos</u><sup>1</sup>

World Bank, 1818 H Street N.W., Washington DC, 20433, USA; hpatrinos@worldbank.org JEL classification: I10; I31; J13 Keywords: School closures; COVID-19 pandemic; learning loss

> 1 This is a working paper, and hence it represents research in progress. This paper represents the opinions of the author and should not be attributed to the World Bank Group. Any errors are the fault of the author. I benefited from discussions at seminars at Wesleyan University, University of Mississippi, University of Maryland, Carleton University, University of Arkansas, GLO conference, ICFES, Global Schools Forum and FHi360. I received excellent comments from George Psacharopoulos, Xiaoxue Zhao, Chris Sakellariou, Vicente Garcia Moreno, Kevin Macdonald, Gustavo Arcia, David Marshall, Karthika Radhakrishnan-Nair, and Noah Yarrow.
# **Introduction**

COVID-19 school closures led to emergency remote-learning systems almost immediately. At its peak, nearly 1.6 billion learners in more than 190 countries, or 94 percent of the world’s student population, were impacted by school closures (UNESCO 2020). Students learned less when they were remote and attending high-poverty schools, which were hit hardest (Engzell et al. 2021). Learning losses could cost this generation of students close to $15 trillion in lifetime earnings (Psacharopoulos et al. 2021). Worldwide, learning losses on average amount to 0.17 of a standard deviation (Patrinos et al. 2023), equivalent to roughly a one-half year’s worth of learning.

I estimate the impact of the duration of school closures on learning. The longer the duration of the closures, then the greater the losses. For countries with robust learning loss data, average school closures were 21 weeks, leading to average losses of 0.23 standard deviation, almost a whole year’s worth of learning. Each week of closures increases learning loss by 1 percent of a standard deviation. This will increase the education gaps between high and low socioeconomic status students (Agostinelli et al. 2022), especially in lower-income countries (Kaffenberger 2021). The main lesson learned is that if COVID continues as a low-intensity pandemic, or if a similar situation arises, keeping schools open should be a priority, as the evidence shows that the health benefits of school closures seemed to have been lower than the cost of learning losses (Allen 2022; Davies et al. 2021, 2020; Gandini et al. 2021; Lee and Raszka 2020; Ludvigsson 2020; Raffetti and Baldassarre 2022). The priority now should be to minimize the long-term impacts of school closures.

This paper contributes to several strands of the literature. First, it adds to the growing research on the impact of school closures on learning outcomes (Agostinelli et al. 2022; Belot and Webbink 2010; Haelermans et al 2022), especially on the impact of duration on learning loss (Goldhaber et al 2022; Jack et al 2022). It also adds to the literature on the determinants of school closure duration (Kurmann and Lalé 2021; Nitsche and Hudde 2022). This paper uses unique data to estimate the precise impact per week of closures on learning loss across a variety of countries. From a human capital perspective, it makes sense that school closures would lead to learning loss since schooling is a productive endeavor (Becker 1994); therefore, extending the time away from in-person instruction should lead to a reduction in learning outcomes. Programs to support learning loss mitigation have been explored.

# **1. Data**

We supplement the robust data on learning losses compiled by Patrinos et al. (2023) with a few more recent studies (see Annex 1 and https://microdata.worldbank.org/index.php/catalog/5367). Most refer to primary schooling. The number of weeks of school closure is compiled from the same studies. Not all surveys are nationally representative. In the United States, the learning loss surveys are representative, but school closure duration varied state to state, and our figure represents an average. In Nepal, the data come from a study that only includes adolescent girls from one disadvantaged district. Besides school closure and duration, other possible correlates of learning loss are included. One of those controls is the pandemic itself. We include the COVID19 death rate per 100,000 population; the lockdown stringency index; and the vaccination rate.

2
We also control for national income. Since there are some reports that students in private schools lost less learning than others (Arenas and Gortázar 2022; Jack et al 2022; Wolf et al 2021), we include the proportion of private schools as a control. To gauge to what extent connectivity helped, we include individuals using the internet (percent of the population). To measure pre-existing school quality, we include the score from the World Bank’s Harmonized Learning Outcomes (HLO). To measure the strength of trade unions, we include the trade union density rate. We use the Polity5 dataset as the measure of democracy. See Table 1 for means, standard deviations, and sources.

||**Table 1**<br>|**: Means and**<br>|**Standard Deviations**<br>|
|---|---|---|---|
|Variable|Mean|Std. Dev.|Source|
|Learning loss (SD)|0.23|0.16|see Annex 1 (Learning Loss COVID-19 2020-2022)|
|Weeks closed|20.78|18.21|see Annex 1|
|Death rate/100k|152.36|125.56|Johns Hopkins Coronavirus Resource Center|
|GDP p/c $US|21709|22717|World Bank|
|Private school (%)|14.84|14.10|World Bank|
|Internet (%)|69.44|25.12|International Telecommunication Union|
|School quality|462.08|89.97|Angrist et al. 2021|
|Vaccination rate|59.77|32.17|Oxford Coronavirus Government Response Tracker|
|Stringency index|56.29|8.90|Hale et al 2021|
|Union density|22.38|16.05|International Labour Organization|
|Democracy|7.02|4.76|Polity 5|

# **2. School Closures and Learning Losses**

The literature suggests that school closures are an efficient strategy to reduce the overall duration of a pandemic (Bin Nafisah et al 2018). The decision to close down schools was primarily aimed at mitigating the spread of COVID-19 during the beginning of the pandemic (Raffetti and Baldassarre 2022). Models from previous epidemics such as influenza suggested the role of schools as places that facilitate the spreading and the possible benefit from closure (Cauchemez et al. 2008; Jackson et al. 2014). It was estimated that since school holidays could lead to a 20-29 percent reduction in the rate at which influenza is transmitted to children, then this might work for COVID-19. School closures delay the epidemic peak if implemented earlier but do not eliminate it. Imperfect knowledge led to no guidance on how long the duration of closures should be. Democratic countries tended to implement closures more quickly from the start of the pandemic than those with a more authoritarian regime (Cronert 2020).

Researchers using national data – for example, the United States national achievement test, NAEP – have shown that there is a significant correlation between the length of school closures and learning loss, confirming earlier research (Barnum 2022; Goldhaber et al 2022; Jack et al 2022; Lehrer-Small 2022; West 2022). In the United States, public schools averaged less in-person teaching than private schools. These results are explained in large part by political preferences, vaccination rates, teacher unionization rates, and local labor conditions (Kurmann and Lalé 2021). (Teacher) unionization did not necessarily always favor school closures. For example, in Greece, teachers went on strike opposing the government’s distance education plan (Lambropoulos et al 3
2022). Nitsche and Hudde (2022) show that societal gender ideology has likely influenced school reopening policies – that is, countries with more supportive attitudes toward maternal employment reopened schools significantly sooner than those with less supportive attitudes toward maternal employment.

There is a clear link between weeks of school closures and learning loss (Figure 1). School closures were an exogenous event, brought about by the pandemic and the need to control the infection rates. Closures as part of national lockdowns were imposed. It turns out that an average learning loss of 0.23 standard deviation occurred on average with a school closure duration of 21 weeks in a sample of 41 countries (which represent 2/3 of the world’s population).

**Figure 1: Learning Loss and Duration**

<!-- Start of picture text --> 0.90<br>0.80 Nepal<br>0.70<br>0.60<br>India<br>0.50<br>Malawi<br>ArgentinaCambodia<br>0.40<br>Kenya Turkey Mexico<br>Brazil<br>0.30 Botswana Poland<br>Bangladesh<br>ChinaNorwayFinlandUSABelgiumGhanaGreeceChileSouth Africa Colombia<br>0.20 NetherlandsSwitzerlandGermanyEnglandHungaryRussiaItaly Canada<br>Uganda<br>0.10 IndonesiaEthiopia<br>Czech<br>SpainKorea<br>0.00 Sweden DenmarkFranceAustraliaJapan<br>0 10 20 30 40 50 60 70 80 90 100<br>Weeks closed<br>Learning loss (SD)<br><!-- End of picture text --> We estimate learning loss, _LL_ , in terms of standard deviations, as a result of weeks closed, _W_ , and several control variables, _Xn_ : _LL_ = β1 _W_ + β _nXn_ + ε (1) In terms of learning loss, it makes little difference if the controls are included; all are insignificant, and they do not change the coefficient on school closures much (see Table 2). With or without controls, every week of school closures increases learning loss by almost 0.01 of a standard deviation; or 1% of a standard deviation. This means that a 20-week closure will reduce learning outcomes by the equivalent of almost 3⁄4 of a year’s worth of schooling. No other observed variable has an impact, so it is merely duration that counts. Given that there is little evidence that school 4
closures reduced infection rates, then it is a high cost to pay, especially after it was deemed secure to open schools safely.

**<u>Table 2: Determinants of Learning Loss</u>**

|Variable|1|2|
|---|---|---|
|Weeks|0.006***|0.007***|
||(0.001)|(0.002)|
|Death rate||0.000|
|||(0.000)|
|Private||-0.0025|
|||(0.002)|
|Internet||0.0012|
|||(0.003)|
|School quality||-0.001|
|||(0.001)|
|Vaccination||0.001|
|||(0.001)|
|Stringency||0.000|
|||(0.003)|
|Union||0.002<br>(0.002)|
|Democracy||0.005|
|||(0.006)|
|Log GDP p/c||-0.0499|
|||(0.048)|
|Constant|0.110*|0.568*|
||(0.029)|(0.295)|
|Observations|41|41|
|R-squared|0.415|0.542|
|Standard errors in parentheses|||
|*** p<0.01, ** p<0.05, * p<0.1|||

# **3. The Determinants of Duration of School Closures**

While closures were exogenous, the decision to remain closed longer was a conscious choice, especially after the American Academy of Pediatrics (2020) released, on 24 June 2020, the first school guidance for safe in-person learning and the European Centre for Disease Prevention and Control (2020) released its recommendation on 6 August 2020. By the time of the 2020-21 school year, it had become clear that it was possible to safely open schools (de Hoyos and Saavedra 2021). By August 2020 evidence supported a marginal role of children as drivers of the first wave of infections (Munday 2021). School reopening did not play a causal role in the increased number of new cases and hospitalizations in Italy during the period of September–October 2020 (Gandini et al. 2021), and it was not the main driver of the increased spreading of the COVID-19 UK variant in England at the beginning of 2021 (Davies et al. 2021). Moreover, school closures have no detectable effect on the contact patterns of adults (Cauchemez et al. 2008). Other studies suggested that school closure during the 2009 H1N1 influenza epidemic in Pennsylvania would not have 5
been worth the cost (Brown et al. 2011). Costs such as productivity losses of education professionals and parental wages outweigh the benefits in terms of decreased infection. Simulations showed that closing schools resulted in substantially higher net costs than not closing schools (Brown et al. 2011). Therefore, during the 2009 H1N1 epidemic, school closures could have resulted in substantial costs to society as the potential costs of lost productivity and childcare could have far outweighed the cost savings in preventing influenza cases. In other words, the likely beneficial impact of school closures is limited. At most, studies suggested that school closure can be a useful control measure during an influenza pandemic, particularly for reducing peak demand on health services, but they were not precise on the quantifiable benefits (Jackson et al. 2014). It turned out that the actual benefits of school closure during the COVID-19 pandemic were less than expected (Raffetti and Baldassarre 2022). Children have been less contagious than adults (Ludvigsson 2020; Lee and Raszka 2020; Davies et al. 2020). Allen (2022) calculates a number of cost/benefit ratios of lockdowns in terms of life-years saved in Canada. Using a mid-point estimate for costs and benefits, the reasonable estimate for Canada is a cost/benefit ratio of just 141.

We estimate the determinants of weeks closed using the same control variables. In our analysis, the only significant variables are GDP, private schooling, school quality, the stringency of lockdowns, and the vaccination rate (Table 3). Higher-income countries have a shorter duration of school closure. A low vaccination rate is associated with longer school closures. Similarly, more stringent lockdowns also increase the length of closures. Higher test scores before COVID-19 are correlated with longer school closures. This might suggest that when authorities believe their education system and therefore its distance education version is of high quality, then they tend to prolong school closures. A higher proportion of private schools in a country leads to a longer duration of school closures. Interestingly, the death rate is not a factor, suggesting that school closures were not based on incidence of COVID-related fatalities.

6
**<u>Table 3: Determinants of Duration of School Closures</u>**

|Variable|Coefficient|
|---|---|
|Death rate<br>|-0.009<br>(0.021)<br>|
|Private|0.409**<br>(0.166)|
|Internet|-0.0243|
||(0.267)|
|School quality|0.139***<br>(0.039)|
|Vaccination|-0.255**<br>(0.098)|
|Stringency|0.486*<br>(0.280)|
|Union|0.112<br>(0.150)|
|Democracy|-0.355|
||(0.552)|
|Log GDP p/c|-8.742*<br>(4.566)|
|Constant|21.800<br>(29.15)|
|Observations|41|
|R-squared|0.639|
|Standard errors in parentheses<br>*** p<0.01, ** p<0.05, * p<0.1||

While the unplanned closures were imposed on schools due to an external event, the duration of the closures was a choice, to some extent at least. To properly estimate the determinants of the duration and its impact on learning outcomes, then one needs to address potential endogeneity issues. Table 3 suggests that national income, private schooling, school quality, the stringency of lockdowns, and the vaccination rate are associated with the duration of school closures. Since stringency is related to closures, and quality and private schooling are related to learning outcomes, then we experiment with the vaccination rate as a possible instrument. It could be argued that a high vaccination rate may lead to shorter school closures. A study shows that school closures can be minimized by regular testing and vaccination against COVID-19 (Colosi et al. 2022). The stringency index, a composite measure based on the strength of lockdown measures, can be used to measure how restrictive the measures were (Gros et al. 2021). We instrument the duration of school closures using the vaccination rate and the stringency index using 2SLS. The first stage and IV results are presented in Table 4.

7
**Table 4: Determinants of Learning Loss**

|Variable|First stage<br>weeks closed|IV learning<br>loss|
|---|---|---|
|Vaccination|-0.301**<br>(0.073)||
|Stringency|1.021*||
||(0.262)||
|Weeks||0.006***<br>(0.002)|
|Constant|-18.723|0.111|
||(14.567)|(0.034)|
|N|41|41|
|R-square|0.409|0.415|
|F-test|13.13|10.98|
|[_p_-value]|0.000|0.002|
|Sargan (overidentification) statistic||0.409|
|[_p_-value]||[0.5223]|

The vaccination rate and the stringency index turn out to be useful instruments. Since we do not have many degrees of freedom, we estimate a very parsimonious equation, explaining weeks closed with the vaccination rate. It turns out that a lower (higher) vaccination rate leads to a longer (shorter) duration of school closures, while a higher stringency index prolongs school closures. Therefore, we can tentatively conclude that the duration of school closures leads to a higher level of learning loss. In the IV estimates, a week of school closures leads to a 0.006 standard deviation increase in learning loss, just slightly higher than the OLS estimate.

# **4. Implications**

In terms of learning loss, it makes little difference if the controls are included; all are insignificant. This suggests that school closures themselves were responsible for learning loss and the severity of that loss was primarily due to how long schools were kept closed. What matters is time spent in school. While pre-COVID learning levels varied considerably across countries, depending on many factors, learning happens if children go to school. Keeping schools open reduces learning losses, even in the poorest countries and in countries with low pre-COVID learning levels. This is especially important because research has found little evidence showing that school closures reduced infection rates.

Students learn less when they are not being taught in school buildings. Duration played a key role. The long-term impact may be increased education gaps, especially between students from high and low socioeconomic status.

The main lesson learned here is that if COVID continues as a low-intensity pandemic, or if a new, similar situation arises, keeping schools open should be a priority, as the evidence shows that the benefits of school closures seemed to have been lower than the cost of learning losses. However, the task at hand is to figure out how to accelerate learning and make up for the lost time. Fortunately, there are several things that work in terms of mitigating learning losses. These include online tutoring programs, which were shown to reduce learning loss significantly in several 8
randomized controlled trials (see, for example, Carlana and La Ferrara 2021; Gortazar et al. 2022; Kraft et al. 2022). A large-scale randomized trial testing low-technology interventions—SMS messages and phone calls—with parents to support their child in Botswana improved learning by 0.12 standard deviation (Angrist et al. 2022). Compensatory education works as well. In Tamil Nadu, India, two-thirds of the school closure learning deficit was made up within 6 months after school reopening, with a government-run after-school remediation program contributing onequarter of the recovery (Singh et al. 2022). In Nigeria, a program designed to make up missed lessons during school closures led to a rebound within 2 months, and a recovery of all learning losses among students in low-cost schools (Adeniran et al. 2022).

# **5. Conclusion**

COVID-19 led to school closures and emergency remote-learning systems. The analysis investigated the impact of school closures using a unique data base. The determinants of the duration of school closures estimates were used to instrument school closures – stringency of lockdown and vaccination – and causally estimate the impact of duration on learning. It is concluded that for every week that schools were closed, learning levels declined by almost 1 percent of a standard deviation. This means that a 20 week closure, for example, would reduce learning outcomes by 0.20 standard deviation, almost one year of schooling.

The findings here could influence education policy decision-making. It would be wise for policy makers to be aware of the linkages between public health decisions – such as vaccinations – and the spillover effects on class time and learning. Such linkages have clear implications for the development of learning technologies (such as different applications for distance learning) and the resilience of learning outcomes.

9