Table 25. Pre-treatment balance in student test scores, across schools, 2013-2014 (previous cohort)

||||201|3||201|4|Di|fference 2|014-2013|
|---|---|---|---|---|---|---|---|---|---|---|
||Grade|N|Control<br>Means<br>(T=0)|Difference<br> (T[1]-T[0])|N|Control<br>Means<br>(T=0)|Difference<br> (T[1]-T[0])|N|Control<br>Means<br>(T=0)|Difference<br> (T[1]-T[0])|
|Covariates|||||||||||
|%StdthtkS|1|350|0.72|0.03**|349|0.71|0.03*|699|-0.01|-0.00|
|uens wo oo paece|||[ 0.01]|[ 0.02]||[ 0.01]|[ 0.01]||[ 0.01]|[ 0.02]|
|S-Mthti|1|350|0.01|0.11**|349|-0.02|0.14**|699|-0.03|0.03|
|paece aemacs|||[ 0.04]|[ 0.05]||[ 0.04]|[ 0.06]||[ 0.05]|[ 0.08]|
||||0.01|0.10**||-0.02|0.10**||-0.03|0.00|
|Spaece - Portuguese|1|350|[ 0.03]|[ 0.05]|349|[ 0.03]|[ 0.05]|699|[ 0.05]|[ 0.07]|
|%StdthtkENEM|3|350|0.72|0.01|349|0.76|0.03**|699|0.04|0.02|
|uens wo oo|||[ 0.01]|[ 0.02]||[ 0.01]|[ 0.01]||[ 0.02]|[ 0.02]|
|ENEMPt|3|350|0.01|0.06|349|-0.00|0.01|699|-0.02|-0.04|
|- oruguese|||[ 0.03]|[ 0.04]||[ 0.02]|[ 0.03]||[ 0.04]|[ 0.05]|
|hi|||0.00|0.06||-0.01|0.03||-0.01|-0.04|
|ENEM - Matematcs|3|350|[ 0.03]|[ 0.04]|349|[ 0.03]|[ 0.04]|699|[ 0.04]|[ 0.06]|

Note: Standard errors in brakets. Statistical significance levels * p<0.1 ** p<0.05 *** p<0.01

# _b. Intention to treat effects_

Our estimates of the impact of the feedback and coaching intervention on student learning must account for the test score imbalance presented in the previous section. We perform several different analyses to address this.

First, we estimate intent-to-treat effects (ITT), using a parsimonious set of controls. The ITT effect is estimated from the equation below: �� = �� + �′��� + ���� + �� (6) where �� is achievement for student i; �� represents a vector of pre-intervention characteristics; �� is an indicator for whether the student was in a school that was offered participation in the intervention; and �� if the error term, clustered at the school level. The coefficient of interest is ��.

Ideally, we would control for all students’ previous achievement. However, prior year test scores are not available for all students, so we must use student achievement averaged at the school-grade level as one set of controls. We estimate (6) using five sets of control variables: the first specification does not include any controls, i.e., excludes �� variables; the second includes controls for students’ age, sex, race, the number of student per class and school average pass, failure and dropout rates for that grade in 2013 and 2014, which we denominate “controls 1”; the third specification includes “controls 1” plus the school-grade average for these outcome variables in 2013<sup>18</sup> ; the fourth includes “control 1” plus school-grade averages for these outcome variables in 2014<sup>19</sup> ; and the fifth includes all the controls.

Results are presented in Table 26. Outcome variables (��) are normalized relative to the distribution of the test score in the comparison group in each grade and year<sup>20</sup> and standard errors clustered at the school level are presented in brackets below each estimate. As we add controls for previous achievement averaged at the school-grade level, we see a drop in the magnitude of effect size but an increase in significance: results across specifications 3-5 are consistent and strongly statistically significant. Looking at specifications 3 to 5, we see positive and significant results in Math and Portuguese both for SPAECE 1<sup>st</sup> grade and ENEM 3<sup>rd</sup> grade. The magnitude of the effects is modest, however, ranging from 0.01 to 0.04 standard deviation (SD). To take advantage of our three years of data at the school-grade level, our second strategy is to analyze a difference-in-differences model, comparing student achievement trends in our treatment and control groups in 2014-2015 minus the 2013-2014 trends. As Table 25 showed, the trends on all variables were balanced over 2013-2014.

> 18 When analyzing results for SPAECE, we add the school-grade average of SPAECE on Math and Portuguese in 2013; and when analyzing results for ENEM, we add school-grade average for all areas of the test on 2013. 19 When analyzing results for SPAECE, we add the school-grade average of SPAECE on Math and Portuguese in 2014; and when analyzing results for ENEM, we add school-grade average for all areas of the test on 2014.

> 20 Scores are normalized for each grade and year such that the mean and standard deviation of the comparison group are zero and one, respectively. (We subtract the mean of the control group, and divide by the standard deviation.)

31