original control sample) could lead to a bias in unobservable characteristics. To test for this, we perform a simple exercise: instead of looking to the total of 58 schools not observed, we focus on the difference in the participation rate (the 20 school or 11% differential) and we make a series of assumptions about the possible distribution of our core variables if 20 additional control schools _had been_ part of the sample. This enables us to set some bounds on how our results might have been impacted.

First, we suppose that the 20 missing schools had perfect teaching and the variable for teacher time on instruction was at the 90% point of the distribution in all cases. In this case, the mean difference between treatment and control would have been 2.8% and with the standard deviation of 0.013—from specification (4) in table 5—we would still have found a significant effect at the 5% level. On the other hand, if we assume the opposite scenario, with time on instruction at the 10% point of the distribution in these 20 schools, we would have found a large effect of 7.9% at a 1% level of significance.

We perform this exercise for the 5 summary variables, playing with the different assumptions about where the average values for these 20 schools could have fallen in the distribution – 90%, 75%, 50%, 25% and 10%. Table 14 shows the results under four of our model specifications (OLS results with baseline and all covariates as controls).

Overall, the exercise confirms the robustness of our results: we would have still found sizeable and significant effects in most of the scenarios. For teacher time on instruction, coefficients range from 2.8% to 7.9% and are always significant at the 5% level. Results for teacher off-task range from -1.4% to -3.9% and are always significant. Classroom management results range from -2.7% to -5.3% and would be significant in all cases except the assumption of baseline performance at the lowest points in the distribution (25% and 10%).

The impacts of the program on the two student engagement measures – share of time with all students engaged and share of time with a large group of students off-task – would be affected if all the missing schools were at extremes of the distribution. For instructional activities with all students engaged, if the missing schools had been above the 75% point in the initial distribution (very high in comparison with the overall sample mean of 21%), the model predicts the intervention actually would have had a negative effect. For the share of time with a large group of students off-task, results would have remained significant only if the missing schools were above the very high 75% point in the distribution (compared with the sample mean of 20%).

This exercise provides additional assurance that our results are not affected by the loss of 20 schools from the control group. Given characteristics of the overall sample, it is highly unlikely that all 20 schools would rest at either extreme of the distribution on any of these variables, and under all other scenarios our impact estimates are not affected.

Table 14: Robustness check exercise

|Percentile assumption for<br>missing values on control<br>schools|A. Instructional<br>activities|B. Classroom<br>management<br>activities|C. Off-task<br>activities|D. Instructional<br>activities all<br>students engaged|E. Big group<br>(>6) of student<br>off-task|
|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|
|90%|0.028**|-0.053***|-0.039***|-0.071***|-0.081***|
|75%|0.028**|-0.04***|-0.027***|-0.039**|-0.04**|
|50%|0.041***|-0.027***|-0.014**|0.007|-0.008|
|25%|0.066***|-0.015|-0.014**|0.025|-0.008|
|10%<br>_Note_: 1 to 5 consider coeff<br>check exercise. * p<0.10|0.079***<br>icients from OLS r<br> ** p<0.05 *** p<|-0.002<br>esults with base<br>0.01|-0.014**<br>line and all co|0.025<br>variates as control fo|-0.008<br>r the robustness|

# _ii.Spillover_

Since the treatment was allocated at the school level, and the sample was drawn across different municipalities state-wide, teachers in the control schools were not likely to know about or participate in any part of the treatment. Only pedagogical coordinators from treatment schools were trained in the Stallings method and participated in the data collection. The online website for the coaching program could only be accessed with a school code.

Nevertheless, it is possible that some regional supervisors, who were aware of the intervention, conveyed information about the program to principals of control schools, even though they were asked to avoid this. If this happened, it could create spillover effects that reduced the quality of the counterfactual because outcomes in the control schools were also affected by the program.

23