At endline, the objective was insofar as possible to conduct observations in the same classrooms observed at baseline. Since individual teachers were guaranteed anonymity, the protocol was to observe classrooms with the same three characteristics: grade, subject and shift. As some schools changed their schedules between November 2014 and November 2015, only 75% of the classrooms were able to be “matched” on these criteria at endline.

The observers were state pedagogical coordinators who received a 40-hour training course in the Stallings method and scored 80% or higher on a final test, in which they coded videos and answered other questions. The observer teams were drawn only from schools identified for the treatment sub-sample, to avoid any contamination of control schools from having someone at the school familiar with the Stallings observation method and/or the training program. Observers were organized by district and assigned to districts other than their own, to avoid any familiarity with the schools they observed. Each team was coordinated by a supervisor with advanced expertise in the Stallings method. Supervisors conducted at least two observations side by side with each observer to check consistency, and reviewed the coding sheets submitted by observers for inconsistencies. In the cases of major inconsistencies, supervisors were responsible for making a repeat visit to the school to conduct new observations.

Out of the 348 schools of the randomization, with 174 each planned for treatment and controls, only 292 schools were able to be observed in November 2014 and in November 2015. The full initial sample could not be observed due to disruptions in the school calendar in November 2014 (standardized tests and holidays) and a shortage of observers in the Fortaleza district. The final sample of 292 schools included 156 schools in the treatment group and 136 in the control group. Because the loss of schools from the treatment and control groups was uneven, we conducted a series of balance checks to test the randomization. The 18 schools in the original treatment sample that were not observed could not receive the information treatment (benchmarked classroom observation feedback for the teachers in their school). But these schools were given access to the other three components of the program -- self-help materials, face to face training and coaching, and we could include these 18 schools (and the 38 schools in the control group that also were not observed) in the analysis of learning outcomes.

_c. Balance checks_ To ensure that our final 292-sample was balanced, we perform three sets of tests. First, we compare summary statistics for available outcome variables at baseline for the initially defined treatment and control groups in the 348-school sample. Second, we compare the same statistics for the final sample of 292 schools. Third, we check for balance in data from the baseline round of classroom observations collected in November 2014 for the 292 schools.

The randomization was based on 2013 data on school demographics and outcomes. When 2014 data became available, we performed a new balance check. All variables represent school averages.

Table 2 presents results for the first two sets of tests, along with the results of t tests of mean differences across the treatment and control groups for each variable, as well as joint significance tests. The first set of balancing tests (random sample) shows that the treatment and the control groups are well balanced, although the treatment schools present a higher average math proficiency. A joint test for the joint significance of the variables in predicting treatment fails to reject that they are jointly equal to zero, supporting the notion of baseline balance in these outcome variables.

7