The Ceará education secretariat agreed to randomize the implementation of the program across approximately half of its schools during the 2015 school year, to evaluate rigorously the following research questions:

**1.** Can providing schools with individualized teacher feedback based on classroom observations plus support materials and coaching stimulate measurable changes in teacher practice in a relatively short period (a single school year)?

**2.** Can providing classroom observation feedback and coaching reduce variation in teacher practices within a school?

**3.** Are positive changes in teachers' classroom practice, such as higher time on instruction and higher levels of student engagement, positively correlated with student learning results?

**4.** Is the combined program developed in Ceará (classroom observation feedback and school-level coaching) cost-effective in producing learning results when compared with alternative teacher training programs?

# **3. Instruments and Data**

_a. The Stallings Instrument_ Teachers’ classroom practice was measured using the Stallings “classroom snapshot” method, technically called the Stanford Research Institute Classroom Observation System, developed by Professor Jane Stallings for research on the efficiency and quality of basic education teachers in the United States in the 1970s. (Stallings, 1977; Stallings and Mohlman, 1990). The Stallings instrument generates quantitative data on the interaction of teachers and students in the classroom with a high degree of inter-rater reliability (0.8 or higher) among observers with relatively limited training.

This is a principal advantage of the Stallings instrument, in contrast to observation instruments such as CLASS, which capture more dimensions of teacher quality but which require a high degree of observer training and skill to apply reliably. The Stallings instrument’s relative simplicity makes it suitable for large scale samples in developing country settings (Jukes, 2006; Abadzi, 2009; DeStefano et al, 2010; SchuhMoore et al, 2010; World Bank 2014). The instrument is language and curriculum-neutral, so results are directly comparable across different types of schools and country contexts. A growing body of comparative country data –from more than 20,000 teachers in eight developing countries as of end-2016 –has been collected in collaboration with the World Bank. Once research studies are completed and data are suitably anonymized, data are hosted on the World Bank’s open data website for further research use and country benchmarking.

The strength of the Stallings method is that it is a way of converting the qualitative activities and interactions between a teacher and students that occur during a class into robust quantitative data on teachers’ instructional practice and students’ engagement. Observations are coded at ten different moments in every class, at exact intervals whose spacing depends on the length of the class; every 3 minutes in a 30-minute class, every 5 minutes in a 50-minute class, etc. It is essential that the observer be present in the classroom before the first official moment of class and stay through the official end time of the class, whether the teacher is present or not. Each observation consists of a 15 second scan of the classroom, starting with the teacher and proceeding clockwise around the room. Observers code what the teacher is doing, what materials s/he is using, and what the students are doing.

For the purposes of generating quantitative estimates of time on task, student engagement, teachers’ use of available materials and their core pedagogical practices, the coded activities are grouped into four categories:

**1. Instruction** : Reading Aloud; Demonstration/Lecture; Discussion/Debate/Question and Answer; Practice & Drill; Assignment/Class Work; Copying

**2. Classroom Management** : Verbal Instruction; Disciplining student(s); Classroom Management with Students; Classroom Management Alone

**3. Teacher Off-Task** : Teacher in Social Interaction with Students; Teacher in Social Interaction with Outsiders or Teacher Uninvolved; Teacher out of the classroom

**4. Students Off-Task:** Students in Social Interaction; Student(s) Uninvolved

For the purposes of generating quantitative estimates of the intensity of teachers’ use of available learning materials, the coding options are: No Materials; Textbooks; Workbooks; Blackboard or whiteboard; Learning aids (maps, blocks, science equipment, calculators); ITC (LCD projectors, computers, TV/radio). The original Stallings instrument is a one-page coding grid with classroom materials listed across the top and activities down the left side. Within each resulting cell, there is one row labeled “T”, for coding what the teacher is doing and what materials s/he is using at the moment of observation and one row labeled “P” for marking what the pupils are doing and what materials they are using. Each 15 second observation is coded on a single sheet, thus each class observed generates 10 coding sheets. As the paper-based version 5