metadata: authors: 0: author: Amorim,Vivian De Fatima
1: author: Piza,Caio
2: author: Lautharte Jr.,Ildo José
count: Brazil
docna: 0: docna: Learning When Schools Shutdown : Impacts of
            H1N1 Outbreak on Learning Loss and Learning Gaps
docty: Policy Research Working Paper
abstracts: cdata!: This paper contributes to the growing
            body of evidence on the effects of school closures on
            learning outcomes, focusing on a recent event in a
            developing country. During the 2009 H1N1 pandemic outbreak,
            a significant number of public schools in S&#xe3;o Paulo
            state, Brazil, extended the winter break by two to three
            weeks. By employing double- and triple-difference estimates,
            the study reveals that even such a relatively short period
            of school closure can result in a learning deficit
            equivalent to six to nine weeks of regular schooling.
            Furthermore, the findings indicate that the adverse impacts
            were more pronounced among students performing below the
            expected proficiency level in math, suggesting
            disproportionate effects on schools with a higher percentage
            of academically challenged students. Moreover, the research
            underscores the role of school decentralization, revealing
            that municipal schools exhibited greater resilience in
            mitigating the negative shock compared to state-run schools.
            This observation aligns with the broader evidence
            highlighting the advantages of decentralized governance
            structures in responding to crises within the education sector.
display_title: Learning When Schools Shutdown :
            Impacts of H1N1 Outbreak on Learning Loss and Learning Gaps
listing_relative_url: /research/2024/08/34381570/learning-schools-shutdown-impacts-h1n1-outbreak-learning-loss-learning-gaps
topicv3: Education m1327048 470,Health m1504661 3480
publishtoextweb_dt: 2024-08-26T00:00:00Z
dois: 10.1596/1813-9450-10879
no_of_pages: 53
author_keys: 0: author_key: 1435778
1: author_key: 1239892
2: author_key: NA
model_extractions: Policy Research Working Paper 10879

# Learning When Schools Shutdown

Impacts of H1N1 Outbreak on Learning Loss and Learning Gaps _Vivian Amorim Caio Piza Ildo José Lautharte Jr._ Development Economics Development Impact Group August 2024 A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.
Policy Research Working Paper 10879

# **Abstract**

This paper contributes to the growing body of evidence on the effects of school closures on learning outcomes, focusing on a recent event in a developing country. During the 2009 H1N1 pandemic outbreak, a significant number of public schools in São Paulo state, Brazil, extended the winter break by two to three weeks. By employing doubleand triple-difference estimates, the study reveals that even such a relatively short period of school closure can result in a learning deficit equivalent to six to nine weeks of regular schooling. Furthermore, the findings indicate that the adverse impacts were more pronounced among students performing below the expected proficiency level in math, suggesting disproportionate effects on schools with a higher percentage of academically challenged students. Moreover, the research underscores the role of school decentralization, revealing that municipal schools exhibited greater resilience in mitigating the negative shock compared to state-run schools. This observation aligns with the broader evidence highlighting the advantages of decentralized governance structures in responding to crises within the education sector.

This paper is a product of the Development Impact Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at viviamamorin5@gmail.com, caiopiza@worldbank.org, and ilautharthe@worldbank.org. A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.

<!-- Start of picture text --> TRANSPARENT<br>ANALYSIS<br>G<br>R E<br>N<br>S<br>I P<br>Y E<br>K A<br>C A<br>R P<br>LI R<br>O E<br>O C<br>W R<br>H<br>S<br>P<br><!-- End of picture text --> _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Learning When Schools Shutdown: Impacts of H1N1 Outbreak on Learning Loss and Learning Gaps Vivian Amorim<sup>_*_</sup> Caio Piza<sup>_†_</sup> Ildo José Lautharte Júnior<sup>_‡_§</sup>

**JEL Codes: C21, H12, I18, I21**

> _*_ Education Division at the Inter-American Development Bank. Contact: vamorim@iadb.org

> _†_ Development Impact Department (DIME) at the World Bank. Contact: caiopiza@worldbank.org

> _‡_ LAC Education Unit at the World Bank. Contact: ilautharte@worldbank.org

> §The authors wish to express their sincere gratitude to Christian Lehmann and Rafael Terra for their significant contributions, to Arianna Legovini and an anonymous referee. We thank the World Bank Brazil Country Office and the DEC Support Budget for funding support.
# **1 Introduction**

School closures are one of the most drastic measures to contain the spread of infectious diseases. Existing evidence shows that school shutdowns, even for a short period, can have large negative effects on students’ numeracy and literacy skills (Andrabi et al. (2020), Marcotte and Hemelt (2008)). Although most of the existing evidence applies to developed countries, and more recently the development community is eager to understand the consequences of the long period of school closure during the COVID-19 pandemic crisis on learning loss (Lichand et al., 2021; Hevia et al., 2022; Azevedo et al. (2020); Donnelly and Patrinos (2021)), our study focuses on the impacts of a relatively recent episode of school shutdown in a large developing country.

In July 2009, amid the H1N1 outbreak, the Health Department of São Paulo state, Brazil, recommended postponing the end of winter break due to the increasing number of H1N1 cases.<sup>1</sup> All state schools extended the winter break by two to three weeks. A few municipalities in the state followed the state Health Department guidelines, whereas the majority did not change the school schedule. The shutdowns affected 12,957 state and municipal schools and more than 5.5 million students (70% of the students). We leverage the variations within and across municipalities over time to estimate the impacts of school closures on students’ learning loss and learning gaps.

We first explore the policy variation across municipalities using a difference-in-differences (DiD) design to estimate the impacts of school closure on students’ learning in municipal schools. We then leverage a within-municipality variation across municipal and state schools using a triple difference-in-differences design to estimate the impact of the shutdowns on learning outcomes in state-managed schools. To assess whether this episode of school closure had distributional effects, we estimate quantile treatment effects using the changes-in-changes design. The results show that extending the winter break by two to three weeks reduced math scores by at least 0.18 of a standard deviation in municipal schools and 0.26 in state schools. These estimates are equivalent to at least six weeks of learning loss. We find evidence that the impact was slightly stronger in schools in the bottom deciles of the math test score distribution, suggesting that the effects of the school shutdown were more detrimental in schools with a higher share of students

> 1We focus on the state of São Paulo since the state has the largest public network in the country and we could find, by checking local newspapers, the names of the municipalities whose local authorities opted to extend the winter break due to the pandemic crisis.

1
already behind in math skills. The effects on reading skills reached at least 0.19 of a standard deviation, but they are restricted to state schools. This result suggests that the impacts of school closure vary across schools’ levels of administration - i.e., whether the education policy is centralized at the state level or decentralized to the municipality (local authority). These findings are consistent with the existing evidence on the impacts of school closures on learning, even though the point estimates are slightly larger than what has been documented in developed countries.

We investigate three potential mechanisms underlying these results: principals’ managerial skills, teacher absenteeism, and the shorter time frame to cover school curricula in closed schools. First, for the state-managed schools, we find suggestive evidence that the higher the teachers’ perception of principals’ skills, the more the negative impacts of the shutdowns are offset. Second, in state-managed schools where teacher absenteeism is seen as a big issue, the learning loss is at least 60% higher compared to state-managed schools where absenteeism is not a concern. The data suggest that state-managed schools did not extend the length of the school year to compensate for the period of shutdowns. Finally, the shorter length of an average school day seems to have made learning recovery more challenging for locally managed schools.

This paper has three contributions. First, it provides evidence of the effects of school closure in a developing country, where mitigating mechanisms are more limited than in developed contexts. Second, it shows that locally managed (municipal) schools coped better with the negative school shock. This result is consistent with the school decentralization literature that points out that local authorities, which are closer to where the policy is delivered, can better identify and act on population needs. Third, it suggests that some schools’ characteristics contributed to offsetting the adverse effects of school closures, giving rise to direct implications for policy recommendations.

Apart from this introduction, the paper is organized as follows: Section 2 provides some background information on the H1N1 outbreak in the Brazilian context. Section 3 describes the data used and presents the descriptive statistics. Section 4 discusses the empirical strategy, whereas Section 5 presents the main findings. We then conclude with a discussion and policy implications in Section 6.

2
# **2 Background: The H1N1 pandemic and school closures in São Paulo**

In June 2009, a new influenza outbreak, H1N1 (also known as swine flu), was declared a pandemic by the World Health Organization (WHO).<sup>2</sup> From April to December 2009, Brazil confirmed 54,171 cases and 2,196 deaths from H1N1.<sup>3</sup> However, the number is likely to be much higher as many people with flu symptoms do not seek help, not all the ones that look for health assistance are tested, and the under-reporting of hospitalizations and deaths are publicly known.

At the end of June 2009, amid the increasing number of cases, the Health Department of the State of São Paulo, where 8% of all national cases were registered, recommended that public schools extend the winter break of July 2009 for a few weeks. The recommendation was based on the fact that children can act as vectors of transmission as they are less likely to adopt behavioral changes, such as washing hands regularly and avoiding physical contact for prolonged times (Klaiman et al. (2011)).<sup>4</sup> The state authority postponed the return to school for two weeks in all the schools under their management across the 645 municipalities of São Paulo state. However, each local authority of the 645 municipalities had the autonomy to decide whether or not to follow the state guidelines. By checking local newspapers from 2009, we found that 13 local authorities followed the guidance of the state Health Department (Figure A.1)<sup>5</sup> . These 13 municipalities postponed the winter break of their locally-managed schools for two to three weeks. These municipalities are among the one-fifth most populous in the state. On average, they concentrated most confirmed cases of H1N1 per 100,000 inhabitants between April and July in the state (10 versus 2 cases, Figure A.2). Overall, more than half of the public schools were closed, affecting about 70% of the students (Table 1).

> 2By May 2010, 214 countries had reported cases and an estimated death toll of more than 200,000 people (The World Health Organization (2010), Dawood et al. (2012)).

> 3DATASUS, 2009. Available through: http://tabnet.datasus.gov.br/cgi/tabcgi.exe?sinannet/cnv/influbr.def.

> 4Check the official statement of the state government of São Paulo in this link: https://www.saopaulo.sp.gov.br/ultimas-noticias/nota-oficial-da-secretaria-da-saude-sobre-retorno-as-aulas/.

> 5São Paulo, the state capital, Campinas, Diadema, Embu das Artes, Indaiatuba, Mairiporã, Osasco, São Bernardo do Campo, Santo André, São Caetano do Sul, Sumaré, Ribeirão Preto and Taboão da Serra

3
Table 1: State and locally-managed schools in São Paulo (2009)

||||**Students**|||||**Schools**|||
|---|---|---|---|---|---|---|---|---|---|---|
||Total|Pre-K<br>**Sc**|Primary<br>Education<br>**hools shutd**|Lower<br>Secondary<br>**own, exten**|Upper<br>Secondary<br>**sion of the wi**|Total<br>**nter-break**|Pre-K|Primary<br>Education|Lower<br>Secondary|Upper<br>Secondary|
|State-managed schools|4,338,887|1,193<br>locat|852,116<br>ed in all the|1,870,373<br>645 municipa|1,615,205<br>lities in the sta|9,787<br>te of São Paulo|39|2,144|3,833|3,771|
|Locally-managed schools|1,202,386|454,499<br>located i|454,703<br>n 13 out of t|281,392<br>he 645 munic|11,792<br>ipalities sin the|3,170<br> state of São P|1,579<br>aulo|972|596|23|
|Total|5,541,273|455,692|1,306,819|2,151,765|1,626,997|12,957|1,618|3,116|4,429|3,794|
|||**No s**|**chool shutd**|**own, no ex**|**tension of the **|**winter break**|||||
|Locally-managed|2,458,858|727,745<br>located i|1,380,053<br>n 632 out of|326,645<br>the 645 muni|24,415<br>cipalities in the|12,192<br> state of São P|6,204<br>aulo|4,956|947|85|
||**% o**|**f students **|**afected by **|**the shutdo**|**wns**|**% of**|**school**|**s afected b**|**y the shutd**|**owns**|
||||69.2%|||||51.5%|||

Notes: Pre-K: Kindergarten and pre-school. Primary Education: first to fifth grades. Lower secondary: sixth to ninth grades. Upper secondary education: tenth to twelfth grades (high school). The locally-managed schools: schools managed by the local authorities of the 645 municipalities in São Paulo. The state-managed schools: schools managed by the state government of São Paulo. Source: Census of Education, 2009.
We use a linear probability model to uncover whether some observed characteristics at the municipal level help explain the 13 municipalities’ self-selection. We regress a dummy variable that takes the value of 1 if the municipality _m_ extended the winter break of municipal schools in 2009 and 0 otherwise on a vector of pre-determined covariates: the municipality GDP per capita in 2007, the municipality population size in 2007, the number of confirmed cases of H1N1 per 100,000 inhabitants (until July 25, 2009), and municipal averages of students’, teachers’ and principals’ characteristics. (Table A.2).<sup>6</sup> Although several coefficients are statistically significant, most are very small. The adjusted R<sup>2</sup> suggests that the municipalities’ observed characteristics do not explain much of the local authority’s decision to follow the state guidelines.

# **3 Data**

The Brazilian public education system is decentralized. The 26 states, the Federal district, and the 5,570 municipalities share the responsibilities of provisioning education. According to the Brazilian Constitution of 1988, the municipal governments should prioritize early childhood, primary (grades 1 to 5), and lower secondary education (grades 6 to 9), and the state authorities should prioritize primary, lower secondary, and upper secondary education (grades 10 to 12). Hence, the schools located in all the Brazilian municipalities can be managed either by the state government or by the municipal government. We will refer to the first group of schools as statemanaged and the second group of schools as locally-managed. In each of the 645 municipalities in São Paulo state, there is at least one locally-managed school and one state-managed school offering either preschool, primary, lower, or upper secondary education.

> 6All the descriptive statistics presented are relative to 2007 and based on the Brazilian Ministry of Education questionnaires that students, teachers, and principals fill out. Tables are available in the online appendix. The principal managerial skills is an index ranging from 0 to 1. It is calculated based on teachers’ answers of how frequently they believe that the principal pays attention to students’ learning, administrative norms, and school maintenance; motivates the teachers and encourages new ideas, and takes into consideration teachers’ inputs; and whether teachers trust the principal and can participate in decisions related to their work. All these variables have four possible answers: never (value 0), sometimes (0.33), often (0.66), and always (1). The principal managerial skills index at the teacher level is an average of these answers, and the index at the school level is an average of teachers’ answers. The student motivation index is similarly defined. It is calculated based on the teachers’ answers on whether students’ learning deficit is caused by low student motivation or bad behavior in the classroom. These variables have two possible answers: yes (0) and no (1). The index includes the variable ’students’ absenteeism’ which can take three possible values: a moderate/big issue (0), a small issue (0.5), and not a problem (1). The student motivation index is an average of these answers. Finally, the teacher motivation index also ranges from 0 to 1, and it is calculated based on students’ answers of how frequently the teacher corrects their reading and math homework. The variable has three possible answers: always (1), sometimes (0.5), and never (0). The teacher motivation at the school level averages students’ answers.

5
The Brazilian legislation for primary, lower, and upper secondary education determines a school year with a minimum of 200 days and 800 hours of instruction time. The state-managed schools are monitored by the State Department of Education, under the state government administration, and locally-managed schools are monitored by the Municipal Department of Education, under the municipal government administration. The respective Departments of Education are in charge of educational policies implemented at the school level, hiring teachers, providing textbooks, appointing principals, financing school infrastructure, and determining the length of the school day, and school breaks, as well as the beginning and the end of the school year in the schools under their responsibility. Usually, the academic year starts in February, and ends in November, summer break occurs in December and January, and winter breaks cover three to four weeks in July.

Data on students’ proficiency in reading and math come from the School Census, the Brazilian Institute of Geography and Statistics (IBGE), and from _Prova Brasil_ , which is the standardized national proficiency exam used to assess students’ learning in primary and lower secondary education.<sup>7</sup> Since 1995, all private and public schools offering primary and secondary education have participated in the annual School Census. The Census is implemented by the National Institute of Educational Studies and Research (INEP), a research agency under the Brazilian Ministry of Education.<sup>8</sup> The School Census collects information on (i) school facilities, such as libraries, sports courts, and science and computer labs; (ii) school infrastructure such as filtered water, electricity, and internet access; (iii) social services, for example, school transportation and provision of meals; (iv) students, such as gender, color of the skin, age, physical disabilities or mental illness, grade level, instruction time per day, class-size, subjects they are enrolled in, grade promotion, repetition and dropout rates; and (v) teachers, such as educational attainment, age, physical disabilities, subjects taught, and classes they are in charge of.

Every two years, the INEP applies a standardized national exam, _Prova Brasil_ , to assess students’ proficiency in reading and math. Since 2005, the test has been applied to 5<sup>th</sup> and 9<sup>th</sup> graders of all public schools.<sup>9</sup> _Prova Brasil_ is one of the proficiency tests within the scope of

> 7 IBGE stands for _Instituto Brasileiro de Geografia e Estatística_ . IBGE has information on GDP per capita and the population at the municipality level.

> 8INEP stands for _Instituto Nacional de Estudos e Pesquisas Educacionais_ .

> 9Schools with at least 20 students enrolled in fifth or ninth grade. Proficiency tests are also applied to students in the last grade of high school.

6
the Education Assessment System (SAEB).<sup>10</sup> Students take the test at the end of the school year (between October and November). In 2007, children took the test between November 5 and 20; and in 2009 between October 19 and 31 (approximately, two months after the school shutdowns). For fifth-graders, proficiency in reading has a scale ranging from 0 to 325, and math has a scale ranging from 0 to 350 (SAEB scale). The student’s proficiency can be classified as insufficient, basic, or advanced.

In addition to answering reading and math questions, students filled out a socioeconomic questionnaire with information on their parents’ educational attainment; incentive from the family to pursue an education; time watching TV, time spent on the internet, time dedicated to reading books, and to homework; if they already dropped out or repeated a grade; and if they attended kindergarten.

Data from the School Census and _Prova Brasil_ are used to calculate the National Education Development Index (IDEB), the most important educational indicator in Brazil, that monitors students’ grade promotion and learning levels.<sup>11</sup> Brazilian State and municipal governments use this indicator to monitor the quality of public education and compare the schools’ performance within and between municipalities.

# **4 Empirical strategy**

To estimate the treatment effects of the 2009 school closures due to the H1N1 outbreak on fifthgraders proficiency in reading and math, we explore the policy variation between and within the municipalities of the state of São Paulo. We focus the analysis on grade five because only five of the 13 municipalities that opted for extending the winter break had proficiency scores available for grade nine in 2007, our ’baseline’ year.

To estimate the impact of the shutdowns on learning, we first look at the impacts on locally-managed schools. To do so, we explore the policy variation between municipalities

> 10SAEB stands for _Sistema de Avaliação da Educação Básica._

> 11 IDEB stands for _Índice de Desenvolvimento da Educação Básica_ . To compute the index, the student’s reading and math performance is transformed into a standardized proficiency ranging from 0 to 10. The index is then multiplied by the student’s grade promotion rate (on a scale from 0 to 1) to obtain the IDEB at school, municipal, state, and country levels. For example, for primary education, the index is the product of the standardized performance of fifth-graders and the average grade promotion from first to fifth grade.

7
using a difference-in-differences design. We compare locally-managed schools in the municipalities whose local authorities extended children’s winter break (treatment group) with locally-managed schools in the municipalities that did not change the school calendar (comparison group). For locally-managed schools, student-level proficiency in _Prova Brasil_ has been available since 2005.<sup>12</sup> However, this proficiency assessment does not include all the primary education schools, mostly due to their size.<sup>13</sup> A key assumption of the DiD design is that in the absence of the policy, the trajectories in proficiency in treated and comparison groups would be the same. To test the plausibility of the parallel trends assumption, we compare the proficiency of both groups before the school closures (2005 and 2007). Due to data limitations, the test is restricted to a subset of treated (10 out of 13) and comparison municipalities (469 out of 629) (Table 2). Because the estimation takes the municipalities as clusters, we use wild bootstrap to estimate standard errors to account for the small number of treated clusters.

To estimate the impact of school closures on state-managed schools, we explore a within-municipality variation. For the same municipality, we can compare test scores in both state and locally-managed schools in which students experienced the same shocks at the municipality level. We restrict the sample to municipalities with at least one state and one locally-managed school offering primary education. This is the case of 112 municipalities where the local authority did not extend the winter break (a group that we classify as _G_ = 1) and for 10 out of the 13 municipalities that did ( _G_ = 0). For schools in _G_ = 1, we compare students’ outcomes of 929 state-managed schools that extended winter break with 1,334 locally-managed schools that remained open before and after the H1N1 shock. For schools in _G_ = 0, we compare students’ outcomes of 868 state-managed schools against 759 locally-managed schools, all closed before and after the shutdown. The triple difference leverages these two double differences in G0 and G1 respectively. In doing so, it accounts for any differences in the proficiency trajectories of state and locally-managed schools within and across municipalities (Table 2).

> 12Proficiency data at the school level for state-managed schools are available starting in 2007.

> 13As described in Section 3, to participate in the assessment, the class size has to have at least 20 students in grade 5.

8
Table 2: Sample of the study, São Paulo (2005-2009)

# **(A) Triple difference-in-differences. Locally and state managed schools**

||_M_|_unicipalities where_|_Other municipalities where_|
|---|---|---|---|
||_the local authoritie_|_s extended the winter break (G = 0)_|_the local authorities did not extend the winter break (G = 1)_|
||State|Locally|State<br>Locally|
||managed|managed|managed<br>managed|
||network|network|network<br>network|
|_Number of:_||||
|Municipalities|10|10|112<br>112|
|Schools|869|759|929<br>1,334|
|Students|144,655|110,606|99,172<br>139,017|

**(B) Difference-in-differences. Municipalities in São Paulo where there is at least one school managed by the local authority Locally-managed schools only**

||_Municipalities where_|_Other municipalities where_|
|---|---|---|
||_the local authorities extended the winter break (G = 0)_|_the local authorities did not extend the winter break (G = 1)_|
||Locally|Locally|
||managed|managed|
|_Number of:_|network|network|
|Municipalities|10|469|
|Schools|795|2,569|
|Students|118,804|239,778|

_Notes_ : To estimate the DiD, we consider all locally-managed schools with proficiency available since 2005 to be able to use more than one data point before the school closures. For the triple difference design, our sample includes the municipalities with at least one state and one locally-managed school available. The rows _students_ show the number of students enrolled in fifth grade. Source: Census of Education, 2009.
# **4.0.1 DiD on the sample of locally-managed schools**

To estimate the impacts of the school shutdowns on the locally managed schools we estimate the following regression equation: in which _ysmt_ is the school-average test score of fifth-graders in school _s_ , in municipality _m_ , in year _t_ ; _Tm_ is a dummy that takes the value of 1 if municipality _m_ extended the winter break of all municipal schools in 2009, and 0 otherwise; _H_ 1 _N_ 1 is a dummy that is equal to 1 in 2009, and 0 in 2007; _Xsmt_<sup>_′_is a vector of school characteristics, and school-average students’, teachers’,</sup> principals’ characteristics; and _δm_ are municipalities fixed effects. The parameter of interest, _γDD_ , is the average treatment effect on the treated (ATT), i.e., the causal impact of school closures on learning. Standard errors are clustered at the unit of intervention level, i.e., the municipality. Because the number of clusters in the treatment group is small (10), standard errors are estimated using _wild-bootstrap_ with 1,000 replications (Roodman et al. (2019)). All the regressions are weighted by the fifth-grade enrollment at the school level.

Intuitively, the effect of the winter break extension is estimated by comparing the differences in the average test scores between the treatment and comparison groups in 2009 (after the pandemic) and 2007 (before the pandemic). To claim the causal effects of the school closures on students’ proficiency, the comparison group is assumed to emulate what would have happened with the average test scores in the treatment group in the absence of the school shutdowns. As shown in Figure 1, we find suggestive evidence that the learning outcomes in both T and C groups were on the same time trend before the H1N1 outbreak.

10
Figure 1: Students’ proficiency in locally-managed schools, fifth-grade (2005-2009) <!-- Start of picture text --> Extended winter break Extended winter break<br>Other municipalities Other municipalities<br>Contrafactual Contrafactual<br>220 200<br>210<br>190<br>200<br>190<br>180<br>180<br>170<br>2005 2007 2009 2005 2007 2009<br>(a) Math (b) reading<br>Test score, SAEB scale Test score, SAEB scale<br><!-- End of picture text --> Note: For fifth-graders, the proficiency scale (SAEB scale) ranges from 0 to 350 in reading, and from 0 to 350 in Math. The proficiency for treatment and comparison groups is the average of the proficiency scores of a sample of locally-managed schools in Sao Paulo state, the ones that have the performance of fifth-graders available since 2005. These schools are located in 10 municipalities of the treatment group and 469 municipalities of the comparison group (Table 2). The contrafactual is calculated assuming that the average score of the treatment group would increase at the same pace as that of the comparison group in the absence of the school closures. Because the first year of _Prova Brasil_ assessment was 2005, we cannot test for parallel trends using a longer time frame. Figure A.3 shows the time-trend for grade promotion, repetition, and dropout rates. Source: Data from _Prova Brasil_ /INEP.

Aiming to increase precision and account for potential time-variant confounders, we include a vector of covariates at the school level, as these variables might be correlated with both the decision of the municipality to close its schools and students’ performance in the standardized tests. We use a Lasso ( _Least Absolute Shrinkage and Selection Operator_ ) regression to select the vector of covariates, _Xsmt_<sup>_′_,thatbestpredictthevariationintheschoolaverageproficiency</sup> score (Ahrens et al. (2019)).

The vector of covariates includes dummy variables indicating whether the school has a science and computer lab, sports court, library, and access to the internet; instruction hours per day; the number of students per class; GDP per capita of the municipality; and socioeconomic characteristics of fifth-graders. The vector of socioeconomic variables includes the percentage of mothers with a high school diploma; the percentage of students that have already been retained in one specific grade or dropped out of school; the percentage of white students; the percentage of female students; the percentage of students that work for pay; the percentage of students that previously studied in a private school; the percentage of students who have a computer at 11
home; the percentage of students whose parents incentivize them to study, to do the homework, to read, to not miss classes, and that talk about what happens in the school; the percentage of teachers with tenure; whether there is a lack of textbooks in the school; the quality of the textbooks; the number of enrollments in the school; and whether the students are allocated into their classes at the beginning of the school year based on their previous academic performance.<sup>14</sup> The sample used in this analysis consists of 795 schools with 118,804 fifth-grade students in the treatment group and 2,568 schools with 239,778 students in the comparison group (Table 2).

# **4.0.2 Triple DiD on the sample of state and locally-managed schools**

Because the DiD design considers only 13 treated municipalities and these are considerably larger than most municipalities in the comparison group, one may wonder whether some unobserved time-varying factors could bias the estimates through the selection mechanism. For instance, it could be that the municipalities decided to postpone school return because they had more resources and better infrastructure to put in place mitigating policies later on; or because these municipalities registered the higher incidence of H1N1 cases, it is more likely that parents, teachers and students of these municipalities were more concerned with the risks of contamination and this concern affect their both teachers and students’ attendance rates. As long as these factors could affect learning, the DiD estimates would be biased.

To overcome this identification threat, we leverage a within-municipality variation. We consider the municipalities with at least one state and one locally-managed school and split them into two groups. Let _G_ = 1 denote the 112 municipalities where state-managed schools were closed and locally-managed schools remained open, and _G_ = 0 the ten municipalities where state and municipals schools were closed (Table 2). We assume that idiosyncratic shocks at the municipal level affect students and schools (either state or locally managed) similarly.

Due to the differences in the learning trajectories between state and locally-managed schools preschool shutdown, a simple comparison of state and locally-managed schools in _G_ = 1 would likely

> 14Table A.3, Table A.4, Table A.5, Table A.6, Table A.7, Table A.8, Table A.9 present the descriptive statistics for these covariates. The model does not include variables that might have been affected by the shutdowns, such as absenteeism of students and teachers, principal managerial skills, student motivation, and teacher motivation. By including these variables, it would be more likely to underestimate the effects of the shutdowns.

12
lead to biased estimates of the impacts of the school shutdowns (Figure A.4).<sup>15</sup> Between 2007 and 2009, there was a higher increase in the students’ test scores in the state-managed schools that extended the winter break compared to test scores in municipal schools that remained open which could suggest that the state government was more successful than locally managed schools in counterbalancing the negative effects of school shutdowns.<sup>16</sup> To account for different learning trajectories across state and locally-managed schools, we compare state and locally-managed schools in _G_ = 1 and _G_ = 0 using a triple difference-in-differences design (TD). This design accounts for three sources of variations: time variation, across-municipality variation, and within-municipality school variation. The TD is computed as the difference between two double differences. The first double difference (Δ1) is given by the differences in average proficiency across the state (closed), and locally-managed schools (opened) in _G_ = 1, between 2007 (before the shutdowns) and 2009 (after the shutdowns). The second double difference (Δ2) consists of differences in average proficiency across state and locally-managed schools between 2007 and 2009 in _G_ = 0. The TD is given by (Δ1 _−_ Δ2) and it therefore measures the effect of the shutdowns on average proficiency in state-managed schools (Muralidharan and Prakash (2017)).

Given the full compliance with the policy mandate, we run the following regression equation to estimate the ATT in the TD setup: in which _Esm_ is equal to 1 if school _s_ in municipality _m_ is state-managed and 0, otherwise. _Xsmt_<sup>_′_isavectorconsistingofschoolaveragestudents’,teachers’,principals’characteristics,and</sup> schools’ characteristics.<sup>17</sup> The parameter of interest, _γDDD_ , is the average treatment effect on the treated (ATT), that is, the average effect of school closures on student learning in statemanaged schools. The standard error is clustered at the municipality level. All the regressions

> 15The first column of Table A.11 and Table A.12 shows that state and locally-managed schools did not follow the same time trend.

> 16In fact, in 2008 the state government implemented a program aimed at increasing managerial practices of schools in the bottom 5<sup>th</sup> percentile of the test score distribution. Despite the relatively small number of statemanaged schools targeted by this program (621 of 5,977), we excluded them from the evaluation sample to mitigate the risk of bias. In 2009, the state government had 5,977 state-managed schools offering first to ninth grades. These schools were the target population of the school management upgrading program.

> 17The controls were selected using a Lasso regression.

13
were weighted by the fifth-grade enrollment at the school level. Table A.11 shows that with this strategy, we can control for the differences in learning trajectories between state and locallymanaged schools across municipalities, as the coefficient of _E × G × D_ is insignificant before the shutdowns (there is no significant difference in the learning gap of state and locally-managed schools in _G_ = 1 and _G_ = 0).

To summarize:

- _G_ = 1 are the 112 municipalities where the municipal governments did not extend the winter break, and where at least one state and one locally-managed school offering first to fifth grade. These municipalities have 929 state-managed schools with 99,172 students and 1,334 locally-managed schools with 139,017 students.

- locally-managed schools opened.

- state-managed schools closed.

- 1<sup>_st_</sup> difference-in-differences:

- in which _YE,G_<sup>1</sup> =1<sup>and</sup><sup>_Y_</sup> _E,G_<sup>0</sup> =1<sup>aretheproficiencyofstate-managedschoolsin</sup><sup>_G_=1</sup> in _D_ = 1 (2009) and _D_ = 0 (2007), respectively; and _YM,G_<sup>1</sup> =1<sup>and</sup><sup>_Y_</sup> _M,G_<sup>0</sup> =1<sup>arethe</sup> proficiency of locally-managed schools in _G_ = 1 in _D_ = 1 and _D_ = 0, respectively.

- _G_ = 0 are the 10 municipalities in which the municipal authority extended the winter break, and where there is at least one state and one locally-managed school. These municipalities have 868 state-managed schools with 144,655 students and 759 locallymanaged schools with 110,606 students.

- locally-managed schools closed.

- state-managed schools closed.

- 2<sup>_nd_</sup> difference-in-differences:

- in which _YE,G_<sup>1</sup> =0<sup>and</sup><sup>_Y_</sup> _E,G_<sup>0</sup> =0<sup>aretheproficiencyofstate-managedschoolsin</sup><sup>_G_=0</sup>

14
in _D_ = 1 (2009) and _D_ = 0 (2007), respectively; and _YM,G_<sup>1</sup> =0<sup>and</sup><sup>_Y_</sup> _M,G_<sup>0</sup> =0<sup>arethe</sup> proficiency of locally-managed schools in _G_ = 0 in _D_ = 1 and _D_ = 0, respectively.

- The TD is given by: Δ1 - Δ2 = _γDDD_

# **5 Results**

The estimates show that the school shutdowns during the H1N1 outbreak had a negative impact on students’ learning, especially in math. The baseline estimates point to a decrease in math scores equivalent to -0.21 and -0.28 of a standard deviation in locally and state-managed schools, respectively (Columns 1 and 4 of Table 3). The decrease in reading scores is equivalent to -0.24 of a standard deviation and is restricted to the state-managed schools.

15
Table 3: Impact of the school shutdowns on students’ learning, fifth-grade (2007-2009)

|||||**M**|**Estimated **<br>**ath**|**decrease in**|**Math and **|**Portuguese**|**Profcien**|**cy, SAEB **|**scale**|**P**|**ortuguese**||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
||DiD|DiD|DiD|TD|TD|TD|TD|TD|DiD|DiD|DiD|TD|TD|TD|TD|TD|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|
|H1N1|-3.26**|-3.27**|-2.75**|-4.35***|-4.25***|-4.38***|-4.29***|-4.25***|-0.76|-0.78|-0.54|-3.49***|-3.40***|-3.51***|-4.29***|-2.73***|
|sd|(1.18)|(1.22)|(1.23)|(0.96)|(0.97)|(0.96)|(0.97)|(1.06)|(0.86)|(0.91)|(0.95)|(0.81)|(0.81)|(0.81)|(0.97)|(0.90)|
|Wild-bootstrap p-value|0.0400|0.0260|0.0410|0.0000|0.0000|0.0000|0.0000|0.0001|0.3920|0.4150|0.6060|0.0000|0.0000|0.0000|0.0000|0.0025|
|95% CI|[-5.6,-0.9]|[-5.7,-0.9]|[-5.2,-0.3]|[-6.2,-2.5]|[-6.1,-2.4]|[-6.3,-2.5]|[-6.2,-2.4]|[-6.3,-2.2]|[-2.5,0.9]|[-2.6,1.0]|[-2.4,1.3]|[-5.1,-1.9]|[-5.0,-1.8]|[-5.1,-1.9]|[-6.2,-2.4]|[-4.5,-1.0]|
|N. schools|3912|3912|3912|5329|5329|5329|5329|5329|3912|3912|3912|5329|5329|5329|5329|5329|
|Adj. R2|0.8|0.8|0.8|0.8|0.8|0.8|0.8|0.7|0.8|0.8|0.8|0.8|0.8|0.8|0.8|0.6|
|_Profciency - Treatment _|_Group before_|_the school s_|_hutdowns (_|_2007)_|||||||||||||
|Mean|193.38|193.38|193.38|195.3|195.3|195.3|195.3|195.3|175.44|175.44|175.44|177.88|177.88|177.88|195.3|177.88|
|Sd|15.44|15.44|15.44|16.09|16.09|16.09|16.09|16.09|15.4|15.4|15.4|15.46|15.46|15.46|16.09|15.46|
|ATT est (in sd)|-0.21|-0.21|-0.18|-0.27|-0.26|-0.27|-0.27|-0.26|-0.05|-0.05|-0.03|-0.23|-0.22|-0.23|-0.27|-0.18|
|**Estimated increase in **|**the perce**|**ntage of stu**|**dents belo**|**w the basic **<br>**M**|**level of lea**<br>**ath**|**rning in M**|**ath and Por**|**tuguese, in**|**%**|||**P**|**ortuguese**||||
||DiD|DiD|DiD|TD|TD|TD|TD|TD|DiD|DiD|DiD|TD|TD|TD|TD|TD|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|
|H1N1|2.59**|2.61**|2.20**|4.63***|4.59***|4.66***|4.62***|4.89***|1.19|1.2|0.97|3.12***|3.08***|3.13***|3.09***|2.62***|
|sd|(0.91)|(0.94)|(0.95)|(0.86)|(0.87)|(0.86)|(0.87)|(0.97)|(0.86)|(0.88)|(0.89)|(0.74)|(0.74)|(0.74)|(0.74)|(0.89)|
|Wild-bootstrap p-value|0.0310|0.0210|0.0340|0.0000|0.0000|0.0000|0.0000|0.0000|0.2170|0.2160|0.3140|0.0000|0.0000|0.0000|0.0000|0.0034|
|N. schools|3914|3914|3914|5330|5330|5330|5330|5330|3914|3914|3914|5330|5330|5330|5330|5330|
|Adj. R2|0.8|0.8|0.8|0.8|0.8|0.8|0.8|0.7|0.8|0.8|0.8|0.8|0.8|0.8|0.8|0.5|
|_Percentage of students be_|_low the basi_|_c level of pro_|_fciency - T_|_reatment Gro_|_up before the_|_school shutd_|_owns (2007)_||||||||||
|Mean|57.49|57.49|57.49|56.36|56.36|56.36|56.36|56.36|70.74|70.74|70.74|69.62|69.62|69.62|69.62|69.62|
|Increase, in %|5.12|5.17|4.36|8.78|8.69|8.83|8.76|9.26|1.82|1.82|1.47|4.67|4.6|4.69|4.62|3.92|
|(A) Municipal FE|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|
|(B) Controls|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|
|(C) % teachers|No|Yes|Yes|No|Yes|No|Yes|Yes|No|Yes|Yes|No|Yes|No|Yes|Yes|
|in both networks|||||||||||||||||
|(D) % teachers|No|No|Yes|No|No|Yes|Yes|Yes|No|No|Yes|No|No|Yes|Yes|Yes|
|management|||||||||||||||||
|(E) School FE|No|No|No|No|No|No|No|Yes|No|No|No|No|No|No|No|Yes|

_Notes_ : The authors’ estimate is based on data from _Prova Brasil_ , Census of Education, and IBGE. Wild bootstrap p-values. 95% CI in brackets. ***, **, and * indicate significance at the 1, 5, and 10% critical levels. Standard error in parentheses and clustered at the municipality level. All regressions are weighted by fifth-grade enrollment at the school level. Math performance on a scale from 0 to 350 (SAEB scale). reading performance on a scale from 0 to 325 (SAEB scale). The columns DiD refer to the differences-in-differences estimates and the columns TD show the triple difference-in-differences estimates. The sample of municipalities and schools included in the analysis are detailed in Table 2. (A) municipal fixed effects. (B) students’, teachers’, schools’, and principals’ characteristics. (C) the percentage of teachers that work in a state and a locally-managed school simultaneously. (D) the percentage of teachers working in a state-managed school that implemented the intervention of managerial practices. (E) schools’ fixed effects. All TD estimates exclude state-managed schools in which the state government implemented managerial practices intervention.
The stronger negative effects on math are in agreement with other estimates of school shutdowns on learning outcomes (Kuhfeld et al. (2020), Thum and Hauser (2015), Baker (2013), and Cooper et al. (1996)). The available evidence suggests that out-of-school enrichment activities during school breaks tend not to focus on math, which is associated with both math anxiety and new instructional methods that differ from what parents themselves learned. In this context, instead of math, families tend to focus more on the promotion of literacy skills at home (McCombs (2011), Murnane (1975), Bryk and Raudenbush (1989), Allinder et al. (1992), Harris and Sass (2009)). These factors help to explain why math skills depreciate faster than reading skills, why school shutdowns have larger negative effects on math than on reading, and inform policies aimed at mitigating learning losses caused by unexpected shocks that can result in school closures.

Because the interpretation of learning gains and losses in terms of standard deviation is not straightforward, we convert the estimates into expected years of schooling. To do so, we use the ideal learning gain of an average student between the fifth and ninth grades in the Brazilian school system as a benchmark. An average student who does not repeat a grade should experience an annual increase in proficiency of 20 points in the national standardized exam, on a learning scale that ranges from 0 to 325 for reading and 0 to 350 for math (Alves et al. (2016)). Therefore, the estimated drop in math performance of -3.3 and -4.6 points corresponds to at least six and nine weeks of learning loss in locally and state-managed schools, respectively.<sup>18</sup> Given the relatively large effects of the winter break extension on learning loss, we test whether the estimated drop in test scores increased the percentage of students below the basic learning level according to the SAEB scale. The baseline estimates shown in Table 3 point to an increase in the percentage of students below the basic level in math proficiency of 2.6 and 4.9 percentage points in locally and state-managed schools, respectively (a rise of 4.5% and 8.7%, compared to baseline levels). For reading, the point estimate is 3.4 percentage points in state-managed schools (an increase of 4.9% over the baseline levels).

The relatively small increase in the proportion of students below the basic learning level suggests that the adverse impacts of the school closures were disproportionately higher among those who were already behind even before the H1N1 pandemic. In fact, in 2007, almost 60% and 70% of

> 18200 school days _×_ 3.3/20 = 32 days, or 6.6 weeks (considering five school days a week). 200 school days _×_ 4.6/20 = 46 days, or 9.2 weeks (considering five school days a week).

17
fifth-graders in the affected schools did not achieve basic levels of learning in math and reading, respectively.<sup>19</sup> School shutdowns can increase learning gaps between low and top performers if their impacts are more detrimental for low performers.<sup>20</sup> To investigate whether the impacts of the school shutdowns are more pronounced at the lower tail of the learning distribution, we estimate quantile treatment effects using the Athey and Imbens (2006) changes-in-changes (CiC) estimator.<sup>21</sup> Figure 2 shows that the negative effects of the school’s shutdown were stronger in math, and potentially stronger in the lower tail of the test score distribution, as suggested by the inter-quantile difference between the lower and top deciles estimates in math.

Figure 2: Distributional impacts of the school shutdowns, fifth-grade (2007-2009) <!-- Start of picture text --> Extended winter break 95% CI Extended winter break 95% CI<br>2.0 2.0<br>0.0<br>0.0<br>-2.0<br>-2.0<br>-4.0<br>-4.0<br>-6.0<br>-6.0<br>-8.0<br>-10.0 -8.0<br>10 20 30 40 50 60 70 80 90 10 20 30 40 50 60 70 80 90<br>Percentiles of test score distribution Percentiles of test score distribution<br>(a) Math, TD (b) Reading, TD<br>, SAEB scale γ , SAEB scale γ<br><!-- End of picture text --> Note: The authors’ estimate is based on data from _Prova Brasil_ , Census of Education, and IBGE. 95% Confidence Interval. Math performance on a scale from 0 to 350 (SAEB scale). reading performance on a scale from 0 to 325 (SAEB scale). Figures (a) and (c) show the estimates for the TD. Figures (b) and (d) show the estimates for DiD. All regressions are weighted by fifth-grade enrollment at the school level. Standard errors clustered at the municipality level. The controls are municipal fixed effects, students’, teachers’, schools’, and principals’ characteristics, the percentage of teachers that work in a state and a locally-managed school at the same time, and the percentage of teachers that work in a state-managed school that implemented the managerial practices intervention. The triple difference specifications also include schools’ fixed effects. All triple DiD estimates exclude state-managed schools in which the state government implemented managerial practices intervention. We perform the estimates using the _cic_ (changes in changes) Stata command.

The more detrimental effects in the state-managed network might indicate that locally-managed

> 19The share of students with insufficient performance in 2007 considers all state and locally managed schools affected by the shutdowns.

> 20We also assess the effects of the school shutdowns on grade promotion, retention, and dropout. DiD estimates indicate a small increase in promotion rates and a decrease in school retention and dropout rates in locallymanaged schools. Triple difference estimates point to nil effects on those outcomes in state-managed schools (Table A.14).

> 21We use the command _cic_ in Stata to perform the estimation. This command allows the estimation of the confidence intervals with bootstrapped standard errors. However, it does not perform the _Wild Bootstrap_ to adjust the standard errors when the number of clusters is relatively small.

18
schools could better respond to the students’ needs when schools reopened. Whether the impact of school closures on learning varies with the school’s administration level is still an open question in the literature. Several papers identify benefits of decentralized management of education (Galiani et al. (2008), Faguet (1999), Jimenez and Sawada (1999), Filmer (2002), King and Ozler (2000)). Having a policymaker close to the population helps identify households’ needs. In the context of school shutdowns, parents of students of municipal schools have lower transaction costs to demand actions from the local authority to help students catch up with the school curriculum.

# **5.1 Robustness Checks**

## **_Effects on municipal schools_**

In the municipalities where the local authorities opted not to extend the winter break, 11% of the contracted teachers of their locally-managed schools also worked on at least one statemanaged school that adopted the school shutdown.<sup>22</sup> It could be that some of these teachers opted not to lecture, making students from the locally-managed network miss school content even though the municipal government opted not to extend their winter break. To deal with this potential spillover effect, we rerun the regressions controlling for the percentage of teachers working in a state and a locally-managed school. The estimates are shown in Column 2 of Table 3 and are very similar to the baseline specification.

As pointed out in Section 4, the DiD analysis is run on a sample of locally-managed schools. Therefore, none of these schools were included in the state government’s managerial practices program implemented in 2008 and 2009. However, some teachers of the locally-managed network who also worked in a state-managed school where this intervention was implemented. This is the case for 5.4% of teachers in locally-managed schools that extended the winter break, and for 1.3% of teachers in the comparison municipalities. Even though the percentages are small, DiD could be slightly biased if these teachers took better managerial practices to the locally-managed schools. We then add as control the percentage of teachers in each one of the locally-managed schools that also work in a state-managed school that implemented the

> 22School Census, 2009.

19
state-government intervention. As shown in Columns 3 of Table 3, the point estimate is lower in absolute terms, but not statistically different from the baseline specification.

# **_Effects on state schools_**

The triple difference design considers the sample of state and locally-managed schools in the municipalities that opted to extend the winter break ( _G_ = 0) and in the municipalities that followed the school calendar as previously planned ( _G_ = 1). To assess whether the managerial practices program implemented by the São Paulo state government in 2008 and 2009 could be confounding the results, we exclude from the analysis all the 621 schools in which this program was implemented. Because 18% of state school teachers also worked in a school that received the managerial practices program, we add the percentage of state school teachers in ‘contaminated’ schools as a control variable. The estimates are similar to the baseline estimate (columns 4 and 5 of Table 3).

It is interesting to notice that the first DiD of the triple differences design is run on the sample of state and locally-managed schools of the 112 municipalities ( _G_ = 1). The comparison here is among state schools that extended the winter break with the municipal schools that remained open (Equation 2 and Table 2). As pointed out earlier, a small percentage of teachers in these municipalities work in both networks and this could also affect the estimates. We then run a regression adding the percentage of teachers in this situation as a control variable. The point estimate is smaller, but not statistically different than the baseline one (Columns 4 and 6 of Table 3).

The second DiD of the triple difference model considers the schools in the 10 municipalities ( _G_ = 0) where both state and locally-managed schools were closed (Equation 3 and Table 2). This estimate will inform both the time trend gap between these two school networks ( _α_ 5), and indicate how each school network reacted to the shutdown. Although the educational indicators of locally-managed schools are, on average, better than the state-managed schools in _G_ = 1 (see Table A.4 and Table A.8), the descriptive statistics presented in Table A.5 and Table A.9 suggest that the state schools seem better in some dimensions, on average, than the municipal ones in _G_ = 0. For instance, the percentage of principals who perceive teacher absenteeism as a big issue is 11 percentage points lower in state-managed schools than in locally-managed schools.

20
On average, these state schools had longer school days than locally-managed ones (5.2 hours vs. 4.7 hours) and lower teacher absenteeism. In _G_ = 0, state-managed schools also seemed to be better prepared regarding the availability and quality of textbooks.<sup>23</sup> To deal with this heterogeneity between state and municipal schools in G0, we estimate the regressions controlling for schools, principals, teachers, and students’ characteristics, and schoolfixed effects. Adding school fixed effects allow us to control for time-invariant unobserved school characteristics associated with heterogeneity in schools’ ability to respond to the shock. With all the controls the point estimates become smaller, but not statistically significant different from the baseline ones (Columns 4 and 8 of Table 3.)

# **5.2 Potential mechanisms**

We find evidence that a relatively small period of school shutdowns (2 to 3 weeks) caused by the 2009 H1N1 outbreak led to a significant decrease in students’ proficiency in math, equivalent to at least 0.18 of a standard deviation in locally-managed schools and 0.26 in state-managed ones. For reading, the estimated decrease is restricted to the state-managed network and is equivalent to at least 0.19 of a standard deviation.

The magnitude of the effect, besides reflecting the short time frame to cover the school curriculum, might also reflect other factors associated with the pandemic context. It could be that teachers, parents, and students were not entirely comfortable with in-person classes soon after the schools reopened. More stressed teachers and students could negatively affect the quality of the classes, student-teacher, and peer-to-peer interactions at school. Teachers might also have been more condescending to the students after schools reopened, leading to children who would put less effort into learning. Also, there were between nine to ten weeks between the reopening of the schools and the application of _Prova Brasil_ .<sup>24</sup> Therefore, students and teachers did not have much time to cover the curriculum before the test. Although we do not have any data to test these hypotheses directly, we highlight a few differences on the school management side and in the availability of resources that help elucidate how state and

> 23The percentage of teachers that classify the textbooks as great is six percentage points higher, and the percentage of principals stating there is a lack of textbooks is 26 percentage points lower.

> 24There are between 44 and 53 business days between the reopening of the schools (August 17, 2009) and the proficiency test (which took place between October 19 and October 31, 2009).

21
municipal schools acted to counterbalance the effects of the school shutdown.

In the following analysis, we dig into the heterogeneous effects of the school closure by interacting the treatment status with the ratio of students per teacher, teachers’ perception of principals’ managerial skills, teacher absenteeism, percentage of teachers with adequate university degrees to teach reading and math, and whether the teachers always correct students’ homework.

Whenever possible we will also present descriptive statistics to highlight the differences between municipalities, as well as differences between state and municipal schools within municipalities. By doing this, we aim to get some understanding of how school staff and students dealt with the negative shock.

# **_School principals_**

We first assess whether schools in which principals have better managerial skills were better able to cope with the shutdowns. The index of principal managerial skills ranges from 0 to 1. It is calculated based on teachers’ answers of how frequently they believe that the principal pays attention to students’ learning, administrative norms, and school maintenance; motivates the teachers and encourages new ideas, and takes into consideration teachers’ inputs; and whether teachers trust the principal and can participate in the decisions related to their work.

For the state-managed network, we find suggestive evidence that better-prepared principals can attenuate the negative impacts of the shutdowns (Table 4). A 10% increase in principals’ managerial skills is associated with a negative impact on math proficiency 0.45 points smaller on a SAEB scale, equivalent to five school days, or an effect 10% smaller than the ATT estimate of our baseline specification of -4.6 points in SAEB scale (see Table 3). 25 For reading, the same increase in principals’ managerial skills is associated with an effect 8% smaller.<sup>26</sup>

> 25As the index of principal managerial skills ranges from 0 to 1, we compare a school with the principal managerial skills at 0.91 and with a school with the index of 1 (10% higher). All the same, the effect of the school shutdown in a school with the principals’ managerial skills at 0.91 is -3.3 points on a SAEB scale ( _−_ 8+5 _._ 15 _×_ 0 _._ 91), whereas the effect in a school with a principal scoring 1 in managerial skills is -2.85 ( _−_ 8 + 5 _._ 15 _×_ 1). Since in 200 school days students are supposed to increase their proficiency by 20 points, the 0.45 ( _−_ 2 _._ 85 _−_ ( _−_ 3 _._ 3) = 0 _._ 45) estimate is equivalent to 4.5 school days (200 school days _×_ 0.45/20 = 4.5 days).

> 26We perform the same exercise for reading, i.e., _−_ 5 _._ 2 + 3 _._ 3 _×_ 1 _−_ 5 _._ 2 + 3 _._ 3 _×_ 0 _._ 91 = 0 _._ 3. These coefficients are shown in Table 4. Since the baseline estimate for the decrease in reading proficiency is 3.7 on a SAEB scale, the 0.3 difference is equivalent to 8%.

22
Table 4: Principals heterogeneity on the impact of the school shutdowns on students’ learning, fifth-grade (2007-2009)

|**Efects of school shutdow**|**ns on the **|**locally-man**|**aged network**||||
|---|---|---|---|---|---|---|
||Math|||Portugues|e||
||DiD|DiD|DiD|DiD|DiD|DiD|
||(1)<br>|(2)|(3)|(1)|(2)|(3)|
|H1N1|-2.75**|-3.62|-2.69|-0.54|-0.34|-0.13|
||(1.23)|(2.04)|(1.62)|(0.95)|(1.77)|(1.38)|
|H1N1 versus||1.56|||0.01||
|principal managerial skills||(2.15)|||(2.30)||
|H1N1 versus|||0.1|||-0.33|
|program to reduce dropout|||(0.66)|||(0.58)|
|N. schools|3912|3838|3857|3912|3838|3857|
|Adj. R2|0.8|0.8|0.8|0.8|0.8|0.8|
|**Efects of school shutdow**|**ns on the **|**state-mana**|**ged network**||||
||Math|||Portugues|e||
||TD|TD|TD|TD|TD|TD|
||(1)|(2)|(3)|(1)|(2)|(3)|
|H1N1|-4.25***|-8.25***|-4.12***|-2.73***|-5.71***|-1.83*|
||(1.06)|(2.10)|(1.28)|(0.90)|(1.69)|(1.11)|
|H1N1 versus||5.34**|||4.12**||
|principal managerial skills||(2.47)|||(2.01)||
|H1N1 versus|||-0.04|||-0.75|
|program to reduce dropout|||(0.61)|||(0.53)|
|N. schools|5329|5227|5239|5329|5227|5239|
|Adj. R2|0.7|0.7|0.7|0.6|0.6|0.6|

_Notes_ : The authors’ estimate is based on data from _Prova Brasil_ , Census of Education, and IBGE. Wild bootstrap p-values. 95% CI in brackets. ***, **, and * indicate significance at the 1, 5, and 10% critical levels. Standard error in parenthesis and clustered at the municipality level. All regressions are weighted by fifth-grade enrollment at the school level. Math performance on a scale from 0 to 350 (SAEB scale). reading performance on a scale from 0 to 325 (SAEB scale). Columns DiD show the estimates of differences-in-differences as described in subsection 4.0.1 and Columns Triple D show the estimates for triple difference-in-differences as described in subsection 4.0.2. The sample of municipalities and schools included in the analysis are detailed in Table 2.

23
# **_Teachers_**

Teacher absenteeism is an important issue among schools impacted by the shutdowns.<sup>27</sup> School principals of municipal schools in the municipalities that extended the winter break are almost 20 percentage points more concerned with teacher absenteeism than school principals of municipal schools in the comparison group (Table A.7). In the 112 municipalities where the state schools closed and the municipal schools remained open, the percentage of principals that saw teacher absenteeism as a big concern was 11 percentage points higher in state schools (Table A.8). Therefore, one may wonder whether the higher absenteeism augmented the effects of the shutdowns.

We find evidence that teacher absenteeism exacerbated the impact of the shutdowns on math and reading proficiency by nearly 2 points on the SAEB scale in state schools (Table 5). The estimated learning loss in math and reading in a school where the principal sees teacher absenteeism as a big issue is 45% and 60% higher compared to schools in which absenteeism is not a concern, respectively.<sup>28</sup> Even with the shutdowns, the percentage of teachers in the state-managed network that covered at least 80% of the school curriculum is not statistically different from the municipal schools that remained open (Table A.8). Even with the shutdowns and higher teacher absenteeism, these schools found a way to cover most of the school curriculum on time.

> 27Teacher absenteeism is a dummy equal to 1 if the principal sees teacher absenteeism as a big issue and 0 if the principal sees it as a moderate or small issue.

> 28The estimated decrease in math and reading performance in a school where teacher absenteeism is not a concern is -3.8 and -2.6, respectively. If teacher absenteeism is a concern, the estimates are equal to -5.5 (-3.8 - 1.7) and -4.2 (-2.4 - 1.8), therefore, 45% and 60% higher, respectively.

24
Table 5: Teachers’ heterogeneity on the impact of the school shutdowns on students’ learning, fifth-grade (2007-2009)

|**Efects of school shutdowns on **|**the locally-ma**|**naged net**|**work**||||||
|---|---|---|---|---|---|---|---|---|
||Math||||Portugue|se|||
||DiD|DiD|DiD|DiD|DiD|DiD|DiD|DiD|
||(1)|(2)|(3)|(4)|(1)|(2)|(3)|(4)|
|H1N1|-2.75**|-3.04|-2.28|-6.24|-0.54|0.12|-0.24|2.74|
||(1.23)|(2.27)|(1.36)|(2.59)|(0.95)|(2.11)|(1.14)|(2.65)|
|H1N1 versus||0.01||||-0.03|||
|students per teacher||(0.11)||||(0.11)|||
|H1N1 versus|||-1.94||||-1.04||
|teacher absenteeism|||(1.17)||||(1.27)||
|H1N1 versus teachers that||||0.04|||||
|correct Math homework||||(0.03)|||||
|H1N1 versus teachers that||||||||-0.04|
|correct Portuguese homework||||||||(0.03)|
|N. schools|3912|3912|3886|3912|3912|3912|3886|3912|
|Adj. R2|0.8|0.8|0.8|0.8|0.8|0.8|0.8|0.8|
|**Efects of school shutdowns on **|**the state-man**|**aged netw**|**ork**||||||
||Math||||Portugue|se|||
||Triple|Triple|Triple|Triple|Triple|Triple|Triple|Triple|
||DiD|DiD|DiD|DiD|DiD|DiD|DiD|DiD|
||(1)|(2)|(3)|(4)|(1)|(2)|(3)|(4)|
|H1N1|-4.25***|-3.88***|-3.72***|-15.71***|-2.73***|-2.32*|-2.26**|-7.05**|
||(1.06)|(1.49)|(1.11)|(3.88)|(0.90)|(1.32)|(0.94)|(2.92)|
|H1N1 versus||-0.03||||-0.03|||
|students per teacher||(0.06)||||(0.05)|||
|H1N1 versus|||-2.08**||||-1.74**||
|teacher absenteeism|||(0.94)||||(0.80)||
|H1N1 versus teachers that||||0.14***|||||
|correct Math homework||||(0.04)|||||
|H1N1 versus teachers that||||||||0.05|
|correct Portuguese homework||||||||(0.03)|
|N. schools|5329|5329|5295|5329|5329|5329|5295|5329|
|Adj. R2|0.7|0.7|0.7|0.7|0.6|0.6|0.6|0.6|

_Notes_ : The authors’ estimate is based on data from _Prova Brasil_ , Census of Education, and IBGE. Wild bootstrap p-values. 95% CI in brackets. ***, **, and * indicate significance at the 1, 5, and 10% critical levels. Standard error in parenthesis and clustered at the municipality level. All regressions are weighted by fifth-grade enrollment at the school level. Math performance on a scale from 0 to 350 (SAEB scale). reading performance on a scale from 0 to 325 (SAEB scale). Columns DiD show the estimates of differences-in-differences as described in subsection 4.0.1 and Columns Triple D show the estimates for triple difference-in-differences as described in subsection 4.0.2. The sample of municipalities and schools included in the analysis are detailed in Table 2.

25
# **_Length of the school day and the quality of the textbooks_**

The fact that the state-managed network could cover as much school content as the locallymanaged network that remained open is associated with the longer school hours per day (5.3 hours versus 4.9 hours). This difference is equivalent to an addition of three to four school days in the whole school year compared to locally-managed schools (Table A.4).<sup>29</sup> However, the additional time in the classroom due to the longer duration of the classes in the state-managed schools is still inferior to the lost time in the classroom due to the period of the shutdowns. We did not find any source of information that could suggest that the state government extended the length of the school year to compensate for the days the schools were closed. The length of the school year was five days longer in the locally-managed network (Table A.4). These statistics also suggest that teachers of the state-managed network had to rush to cover the school curriculum, probably making it challenging for the students to keep up.

A glance at the descriptive statistics presented in Table A.3 also highlights the challenges faced by the municipal schools affected by the shutdowns. One of the issues that stands out is that these schools could not cover the school curriculum in a shorter time frame. While 55% of municipal schools in municipalities that did not extend the winter break covered more than 80% of the school curriculum, only 45% did so in municipalities that extended the winter break.

Data from _Prova Brasil_ questionnaire show that teachers faced many barriers to successfully covering the entire school curriculum. For instance, students in treated schools had fewer hours of classes per day (4.8 hours _×_ 5 hours). The 12-minute daily difference resulted in children from the affected schools having two days less of school content between the reopening of the schools and the proficiency test.<sup>30</sup> Also, we do not find any source of information of an extension of school days later in the year to account for the missing days during the winter break extension.

> 29There are between 44 and 53 business days between the reopening of the schools (August 17, 2009) and the proficiency test (which took place between October 19 and October 31, 2009). Therefore, the 24-minute difference would result in 1,056 to 1,272 additional minutes of class, equivalent to 17.6 to 21.2 hours. If the average number of class hours in the state-managed network in _G_ = 1 is 5.3, the estimate in terms of school days is between three and four.

> 30There are between 44 and 53 business days between the reopening of the schools (August 17, 2009) and the proficiency test (which took place between October 19 and October 31, 2009). Therefore, the 12-minute difference would result in 528 to 636 fewer minutes of class, equivalent to 8.8 to 10.6 hours. If the average number of class hours in the affected network is 4.8, the estimate in terms of school days is roughly two.

26
The length of the school year was a few days shorter in the affected network (Table A.3).

The availability and the quality of textbooks also seem to be a concerning point when the locally-managed network affected by the shutdowns is compared to the unaffected one. The percentage of teachers satisfied with the textbooks was 3.5 percentage points lower in treated schools, whereas the percentage of principals complaining about the lack of textbooks was 22.5 percentage points higher. The elevated teacher and student absenteeism (18 percentage points higher for teachers and 4 pp. higher for students compared to the unaffected schools) and the lack of proper high-quality textbook materials suggest that the treated schools faced more challenges in mitigating the learning loss caused by the school closure (see Table A.7).

# **6 Discussion and Conclusion**

The decision to close schools during a pandemic imposes a clear trade-off for policymakers: on the one hand, this policy helps reduce infection rates among students (Adda (2016)). On the other hand, its long-term consequences can be daunting to a whole generation of youth learners, particularly the most vulnerable.

In São Paulo, Brazil, amid the H1N1 outbreak in 2009, more than half of the public primary, lower, and upper secondary schools were closed for two to three weeks, affecting more than 5.5 million students. We used double-difference and triple-difference designs to estimate the impacts of school closures on fifth-graders’ reading and math skills. We find evidence that the school shutdowns led to a reduction in math scores of at least 0.18 of a standard deviation in locally-managed schools and 0.26 in state-managed ones, equivalent to more than six weeks of learning loss. For reading, the effects reached at least 0.19 of a standard deviation, but are restricted to state schools.

Quantile treatment effects estimates indicate that the effects of the school closure were higher at the lower tail of the math test score distribution. This result helps explain the relatively small increase in the percentage of students below the minimum level of math proficiency (4.5% and 8.7% in locally and state-managed schools, respectively). These results indicate that the shutdowns might have hit harder on students already behind before the pandemic struck.

27
Because a pandemic outbreak can also affect children’s socioemotional skills, the total impact of school closures on human capital accumulation is potentially larger than the effects captured by cognitive tests.<sup>31</sup> .

To give context to the school system’s challenges, we use a meta-analysis conducted by McEwan (2015) that summarizes the impacts of more than 70 randomized controlled trials of educational interventions in developing countries. According to the author, school interventions in primary education have, on average, positive effects that range from 0.05 to 0.15 of a standard deviation. The upper bound of the overall average effect of the education policies included in the metaanalysis (0.15 of an SD) is smaller than our lowest estimate. The expressive learning losses detected during a relatively small period of school shutdowns amid the H1N1 outbreak illustrate the challenges policymakers will face in designing sustainable learning acceleration policies to counterbalance the unprecedented skill losses caused by the long period of school shutdowns during the COVID-19 pandemic crisis, particularly in developing countries.

> 31For instance, the Global Education Monitoring Report (2019) points to the disruptive impact on learning as physical; emotional, with anxiety, fear, sadness, and lack of emotional control; and cognitive, expressed via difficulty paying attention, inability to process information and memory problems. As emotionally nurturing environments produce more capable learners, these factors will have more profound consequences for children’s development (Cunha and Heckman (2007))

28