implausible that improvements in students’ test scores could be the result of one day during the school year when teachers were observed and changed their practice.

# **5. Impacts on Student Learning**

# _a. Sample and balance_

Ceará administers an annual, state-wide, standardized learning assessment, the _Sistema de Avaliação Padronizada de Aprendizagem do estado do Ceará,_ SPAECE. It covers all students in 2<sup>nd</sup> , 5<sup>th</sup> and 9th grade of elementary school and until recently covered all three grades of secondary school. It tests Portuguese language and mathematics. The test also includes a student survey, and surveys of teachers and the school director.

Ceará’s secondary school students also take a national exam at the end of high school, called ENEM ( _Exame Nacional do Ensino Médio_ ). The ENEM was designed in 1998 as a low-stakes high-school leaving exam that could serve as a tool for monitoring education system quality. In 2009, however, Ministry of Education established ENEM as the official university entrance exam, and most universities now either require the ENEM alone or as a supplement. It has thus become a high-stakes exam for graduating students and a growing number of first and second year students also take the exam, to gain practice.

The ENEM is administered over two days, with the first session lasting four hours and 30 minutes and the second lasting five hours and 30 minutes. It covers all subjects: Natural Sciences (Biology, Physics, Chemistry), Human Sciences (History, Geography, Philosophy, Sociology), Languages and Codes (Portuguese Language, Literature, Foreign Languages, Information technology, and Communication), Math (Algebra, Geometry), and Text writing. The final score is a weighted average of the different tests. Almost 6 million Brazilian teenagers sat the exam in 2015.

In 2013 the Ceará Secretariat began applying the state assessment, SPAECE, on a sample basis<sup>16</sup> to second and third year secondary students, given that most now take the ENEM exam. In 2015, students in the 2<sup>nd</sup> year of secondary school were not tested at all and testing of 3<sup>rd</sup> year students was limited to schools participating in the _Jovem do Futuro_ program. Beginning in 2016, state policy is to apply SPAECE only to students in the first year of secondary school.

Table 20 - Test Participation Rates in Ceará secondary schools, 2013-2015

||Total|1<sup>st</sup> Grade|2<sup>nd</sup> Grade|3<sup>rd</sup> Grade|
|---|---|---|---|---|
|||2013|||
|Enrollment|214,822|82,850|70,228|61,744|
|Spaece|89767|60113|15559|14095|
|%|42%|73%|22%|23%|
|ENEM|92126|7921|40024|44181|
|%|43%|10%|57%|72%|
|Spaece & ENEM|28422|7158|10321|10943|
|%|13%|9%|15%|18%|
|||2014|||
|Enrollment|206,236|78,548|67,508|60,180|
|Spaece|92692|56433|18828|17431|
|%|45%|72%|28%|29%|
|ENEM|88601|1|41994|46606|
|%|43%|0%|62%|77%|
|Spaece & ENEM|28625|0|13562|15063|
|%|14%|0%|20%|25%|
|||2015|||
|Enrollment|191,968|73,143|62,917|55,908|
|Spaece|71103|59805|543|10755|
|%|37%|82%|1%|19%|
|ENEM|87027|7146|34043|45838|
|%|45%|10%|54%|82%|
|Spaece & ENEM|15920|6807|8|9105|
|%|8%|9%|0%|16%|

Table 20 shows test participation rates between 2013 and 2015 for our sample of 348 schools. In 2015, 82% of first year students and 19% of 3<sup>rd</sup> year students took the SPAECE exam. ENEM participation was almost the inverse, with 10% of first year students, 54% of 2<sup>nd</sup> year and 82% of 3<sup>rd</sup> year students taking the test in 2015. Because ENEM was designed to evaluate 3<sup>rd</sup> year students and since the participation of younger students is unrepresentative (biased towards stronger students) our analysis focuses on the ENEM results for 3<sup>rd</sup> year students. We also focus on the scores for Math and Portuguese, for two reasons. First, these are the

> 16 The sample of students tested was random and representative at the school level.

27