metadata: authors: 0: author: Popova,Anna
1: author: Evans,David
2: author: Arancibia,Violeta
count: World
docna: 0: docna: Training teachers on the job : what works
            and how to measure it
docty: Policy Research Working Paper
subtopic: Skills Development and Labor Force Training,Educational Sciences,Educational Institutions & Facilities,Educational Populations,Education for Development (superceded),Educational Policy and Planning - Textbook,Effective Schools and Teachers,Education For All
abstracts: cdata!: A significant body of research
            demonstrates that teachers and the quality of their teaching
            are crucial components of student learning. Many teachers in
            resource-poor environments have limited knowledge, skills,
            or motivation. Some impact evaluations have shown promising
            results from interventions to improve the quality of
            teaching. This paper reviews the existing body of evidence
            on what kinds of in-service teacher training interventions
            are most effective, and highlights the knowledge gaps. It
            reveals the dearth of detail on the nature of teacher
            training interventions and proposes a standard set of
            indicators -- the In-Service Teacher Training Survey
            Instrument—for reporting on such programs as a prerequisite
            for understanding which interventions lead to improved
            student learning. Across a set of 26 programs with impact
            evaluations and student learning results, programs that
            provide complementary materials, focus on a specific
            subject, and include follow-up visits tend to show higher
            gains. Programs that use non-education professionals as
            trainers tend to have worse outcomes. Statistical power to
            identify these effects is limited, and use of these standard
            indicators in future impact evaluations will facilitate more
            precise inference.
repnme: repnme: Training teachers on the job : what works
            and how to measure it
display_title: Training teachers on the job : what
            works and how to measure it
listing_relative_url: /research/2016/09/26816796/training-teachers-job-works-measure
topicv3: Teachers Management m1326119 1523,Teacher Training m1326786 1532,Textbooks m1326802 2129,Education m1327048 470,Training Programs m1504140 2817
publishtoextweb_dt: 2016-09-26T00:00:00Z
no_of_pages: 62
author_keys: 0: author_key: 1190058
1: author_key: 1269570
model_extractions: Policy Research Working Paper 7834 Background Paper to the 2018 World Development Report Training Teachers on the Job What Works and How to Measure It _Anna Popova David K. Evans Violeta Arancibia_ Africa Region Office of the Chief Economist September 2016
Policy Research Working Paper 7834

# **Abstract**

A significant body of research demonstrates that teachers and the quality of their teaching are crucial components of student learning. Many teachers in resource-poor environments have limited knowledge, skills, or motivation. Some impact evaluations have shown promising results from interventions to improve the quality of teaching. This paper reviews the existing body of evidence on what kinds of in-service teacher training interventions are most effective, and highlights the knowledge gaps. It reveals the dearth of detail on the nature of teacher training interventions and proposes a standard set of indicators—the In-Service Teacher Training Survey Instrument—for reporting on such programs as a prerequisite for understanding which interventions lead to improved student learning. Across a set of 26 programs with impact evaluations and student learning results, programs that provide complementary materials, focus on a specific subject, and include follow-up visits tend to show higher gains. Programs that use non-education professionals as trainers tend to have worse outcomes. Statistical power to identify these effects is limited, and use of these standard indicators in future impact evaluations will facilitate more precise inference.

This paper—prepared as a background paper to the World Bank’s _World Development Report 2018: Realizing the Promise of Education for Development_ —is a product of Office of the Chief Economist, Africa Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at apopova@stanford.edu and devans2@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Training Teachers on the Job:**

# **What Works and How to Measure It**

Anna Popova David K. Evans Violeta Arancibia

**JEL Codes:** I20, J24, O10 **Keywords:** Education quality, teacher training, professional development, economic development

Acknowledgments: The authors are grateful for comments from Denise Bello, Luis Benveniste, Mary Breeding, Barbara Bruns, Joost de Laat, Margaret Dubeck, Deon Filmer, Andrew Ragatz, and Halsey Rogers, and for financial support from the Strategic Impact Evaluation Fund at the World Bank. They are also grateful for excellent research assistance by Fei Yuan, for the generous time provided by numerous teacher training implementers to fill in knowledge gaps, and for data on World Bank projects from Mary Breeding. Most work for this paper was completed when Popova and Evans worked in the Office of the Chief Economist for the Africa Region at the World Bank, and Arancibia is based at the Catholic University of Chile. The corresponding authors can be contacted at Popova (apopova@stanford.edu) and Evans (devans2@worldbank.org).
# **Introduction**

Increases in access to education across the developing world in recent years have stimulated a shift in policy focus towards improving education quality, often measured by student test score gains. Controlling for socioeconomic factors, teachers have been argued to be the most important determinant of student learning. The difference between a weak teacher and a great teacher has been measured – in the United States – at up to a full year of student learning (Hanushek and Rivkin, 2010). Similarly, in Chile, the impact of good teachers over multiple years accumulates to the equivalent of 0.3 standard deviation (SD) in secondary school (MINEDUC, 2009). Beyond these immediate improvements in student learning, teachers who raise student test scores also significantly improve students' long-term outcomes, such as their probability of graduating college and adult salaries, and decrease the likelihood of teenage pregnancy (Chetty, Friedman, & Rockoff, 2014).

Furthermore, in a recent review of the education literature, improving pedagogy so that it is more directed to individual student levels – an action that depends significantly on teachers either carrying out formative assessments or targeting instruction – was among the most recommended interventions for improving student learning (Evans and Popova, 2016; Kremer, Brannen, & Glennerster, 2013). For example, the full Early Grade Reading Assessment (EGRA) program in Liberia, which trained teachers to use an initial reading assessment and then continually assess student performance, increased students’ reading comprehension by 0.79 SD (Piper & Korda, 2011). Similarly, a program in Kenya that streamed students into classes based on ability so that teachers could tailor teaching to the appropriate level increased test scores by 0.17 SD in language and 0.16 SD in math (Kremer, Duflo, & Dupas, 2011).

In-service teacher training – or professional development – is important to evaluate even beyond the promising evidence from a collection of evaluations, which show that it can – when 2
designed correctly – improve student learning. Massive amounts of government resources are funneled into training programs. Of 171 World Bank projects with education components between 2000 and 2012, nearly two-thirds included professional development to support teachers. Despite the significant resources spent on in-service teacher training programs, rigorous evidence on the effectiveness of such programs remains limited. Overall, evidence for the small share of programs that have been evaluated is mixed, and it is often reported that most current teacher education programs are outdated and over-theoretical. At the same time, many evaluations fail to provide sufficient details on the actual content or delivery mechanisms of the trainings to inform the design of successful programs.

This paper focuses on in-service training, motivated partly by the fact that it faces fewer institutional constraints to change in many countries than do mechanisms that improve teacher practice at pre-service, although the latter are also imperative to teaching quality. In other words, it is often easier for a government to implement an innovative in-service teacher training program than to reform pre-service education, because the latter relies on more institutions outside the Ministry of Education. Furthermore, in-service teacher training policies tend to receive support from teachers’ unions, and provide an opportunity to improve the quality of both existing teachers (the stock) as well as future teachers (the flow).

Nonetheless, there is a sizeable evidence gap when it comes to whether resources spent on in-service teacher training are improving learning, partly owing to a lack of instruments designed to measure teacher professional development. Instruments exist to capture the design of teacher policy – SABER Teachers (World Bank, 2013) – on one end of the policy-practice spectrum, and how teachers behave in the classroom – Stallings (Stallings, 1977), the Classroom Assessment Scoring System or CLASS (La Paro & Pianta, 2003), and others – at the other end of the spectrum. However, to date, there exists no instrument to capture the step between teacher policy design and 3
teachers’ classroom practice; that is, how teachers are actually trained and which specific components of this training effectively improve teacher behavior and subsequently student learning. This is true despite the fact that the evidence we have suggests that there is much more variation in effectiveness across teacher training programs than across education programs more broadly (Evans and Popova, 2016; McEwan, 2015); in other words, teacher training programs vary enormously, both in their form and in their effectiveness.

This paper has two objectives. The first is to fill this gap in information on the essential characteristics of training programs by proposing a survey instrument – the In-Service Teacher Training Survey Instrument (ITTSI) – to document the design and implementation details of inservice teacher training programs. The instrument was piloted on a sample of programs from lowand middle-income countries whose impact has already been evaluated, and the resulting data analyzed using a combination of quantitative and qualitative methods. The second objective is to use those data to characterize the current evidence on in-service teacher training in low- and middle-income countries.

Findings suggest that characteristics positively associated with program impact on student learning include the provision of textbooks and other reading materials alongside the training, as well as linking participation to incentives such as promotion or salary implications, and the training having a specific subject focus, among others. Meanwhile program implementers themselves most commonly mention the provision of mentoring follow-up visits, engaging teachers for their opinions and ideas, and designing programs in response to local context as responsible for positive impacts on student learning.

This paper proceeds as follows. Section 2 summarizes a sample of the theoretical literature on in-service teacher training and provides insights from impact evaluations of programs in highincome countries. Section 3 describes the methodological approach, including the search strategy, 4
the instrument design, the data collection, and the analytical strategy. Section 4 presents the results of our quantitative and qualitative analyses, and Section 5 concludes.

# **Background**

# **Theory**

The education literature suggests a range of factors to consider in the design of teacher training programs, sometimes illustrated by empirical work. A brief discussion of this literature follows, organized around six questions.

_Who is learning?_ Because teachers are the students in in-service teacher training, principles of adult education are relevant. Adult education tends to work best with clear applications rather than a theoretical focus (Cardemil, 2001; Knowles, Holton, & Swanson, 2005). Teacher training will work best if it adjusts to different points in the teachers’ careers; in other words, one would not effectively teach a brand-new teacher in the same way as one would train a teacher with 20 years of experience (Huberman, 1989). Teachers see their greatest natural improvements in the first five years of teaching, so there may be a benefit from leveraging that time (TNTP, 2015).

_Who is teaching?_ Unsurprisingly, the quality of trainers is crucial to teacher learning, just as the quality of teachers is crucial to student learning (Knowles et al., 2005). This calls into question the standard cascade model of training in low-income environments, in which both information and pedagogical ability may be diluted as a master trainer trains a trainer, and so forth.

_How to train?_ At least two key characteristics emerge from the literature on how to train. First, teachers should learn how to carry out formative evaluation so that they can effectively evaluate their own progress towards their teaching goals (Bourgeois & Nizet, 1997); second, teacher training will work best with concrete, realistic goals (Baker & Smith, 1999).

_How long to train?_ There is no theoretical consensus on exactly how long training should last, although there is suggestive evidence in the literature, to be discussed in the next section. In 5
short, there is no reason to believe that one-off workshops will be effective, despite their being common in school systems.

_What to teach?_ Relative to theory or general pedagogy, subject-specific pedagogy is likely to be most effective, as different subjects require radically different pedagogies (Villegas-Reimers, 2003). For example, a more scripted approach may work for early grade reading, whereas later grade science will require higher-order thinking skills, and teachers need training in different pedagogies for these methods.

_Where to teach?_ In general, teacher training in the school (“embedded”) is likely to be most effective so that concrete problems faced in the local environment can be raised, and teachers can receive feedback on actual teaching (Wood & McQuarrie, 1999). However, this will depend on the environment. In very difficult teaching environments, some degree of training outside the school may facilitate focus on the part of the trainees (Kraft & Papay, 2014).

# **What works in high-income countries?**

A full review of the literature in high-income countries is beyond the scope of this study. However, it may be useful to highlight recent work on in-service teacher training from the United States – which spends almost $18,000 per teacher and 19 days of teacher time on training each year (TNTP, 2015) – and other high-income countries, in order to ensure that low- and middleincome countries are not ignoring well-established evidence. A recent meta-analysis of 196 randomized field experiments from the U.S. that measure student test scores as an outcome examined the impact of both “general” and “managed” professional development, relative to other interventions (Fryer, 2016). General professional development (PD) – as the name suggests – leaves a fair amount of flexibility, even as it may focus on classroom management or increasing the rigor of teachers’ knowledge. Managed professional development, on the other hand, is much more prescriptive; it prescribes a specific method, with detailed instructions on implementation 6
and follow-up support. On average, managed PD was found to increase student test scores by 2.5 times (0.052 standard deviations) as much as general PD, and was at least as effective as the combined average of all school-based interventions. However, the analysis is based on relatively few studies, with just seven general PD studies and two managed PD studies. Nonetheless, this finding in support of specific and practical teacher training aligns with that of Walter & Briggs (2012) who, in a review of 35 evidence-based studies of teacher PD, found that concrete and classroom-based programs make the most difference to teachers.

Another U.S.-focused review found that professional development programs with significant contact hours (between 30 and 100 in total) over the course of six to twelve months were more effective at raising student test scores (Yoon, Duncan, Lee, Scarloss, & Shapley, 2007). But this review also draws on few strong studies: of the 1,300 studies included in this review only nine were identified as having pre- and post-test data and some sort of control group. Similarly, a 2014 review of professional development in mathematics found more than 600 studies of math PD interventions, but only 32 used any research design to measure effectiveness, and only five of those were high-quality randomized trials. As the authors concluded, “The limited research on effectiveness means that schools and districts cannot use evidence of effectiveness alone to narrow their choice” (Gersten, Taylor, Keys, Rolfhus, & Newman-Gonchar, 2014). As such, we look also to a wider range of evidence. For example, one recent US review which includes qualitative as well as quantitative studies concludes that teacher training is most effective when it focuses on “concrete tasks of teaching, assessment, observation and reflection” instead of abstract teaching concepts. Characteristics of effective programs included that (a) they were not a “one-shot workshop” but rather embedded in the curriculum, and (b) they were also “sustained” and “intense” (Darling-Hammond, Wei, Andree, Richardson, & Orphanos, 2009).

Much evidence from other high-income countries is also more qualitative. Narrative 7
empirical analysis by Darling-Hammond, Wei, & Andree (2010) highlighted that in high-achieving countries, in-service support to teachers includes (a) mentoring for all beginners, coupled with a reduced teaching load and shared planning time for new and mentor teachers, (b) extensive opportunities for ongoing professional learning, embedded in substantial planning and collaboration time at school, and (c) teacher involvement in curriculum and assessment development and decision making. For example, teaching in Japan includes a practicum year for all beginning teachers during which teachers have a reduced teaching load, attend in-school training with guidance teachers twice a week, and receive weekly out-of-school training, including seminars and visits to other schools (Darling-Hammond, 2005). The Japanese system also includes a lesson study approach to professional development, in which teachers rotate in preparing and teaching lessons addressing a specific goal of their choosing, while others observe and record the lesson, and subsequently provide feedback and make suggestions for improvement (Darling-Hammond et al., 2010). Singapore’s Teachers Network learning circles – in which between four and ten teachers and a facilitator meet for eight two-hour sessions over a period of four to twelve months, to collaboratively identify and solve common problems using discussions and action research – similarly encourage teachers to be reflective practitioners (DarlingHammond et al., 2010).

Limited experimental or high-quality quasi-experimental evidence makes it difficult to draw detailed conclusions about what works within teacher training even in the rich country literature. However, from a combination of the more rigorous quantitative and qualitative studies above, there is suggestive evidence that in-service teacher training programs in high-income countries have been most effective at improving student learning where they have been embedded in the curriculum; prescribed a specific method, with detailed instructions on implementation; included significant and sustained in-person follow-up support for teachers; and involved teachers 8
in a co-learning model.

# **Method**

# **Search Strategy**

We searched the existing literature on in-service teacher training in developing countries to identify a sample of training programs which had been evaluated in terms of the impact they have on student learning. The resulting sample would serve first to inform a review of what we know to date about the effectiveness of different kinds of in-service teacher training in developing countries. Secondly, we would use this sample of studies to inform the design of our survey instrument – by including questions about relevant characteristics either reported or noticeably omitted by studies – and to pilot the instrument in interviews with the program implementers.

Our inclusion criteria for the search are impact evaluations of primary education interventions in low- and middle-income countries that either (a) focused on in-service teacher training, or included this as one component of a broader program, and (b) which reported impacts of the program on student test scores in math, language, or science. We include both published and unpublished papers and do not explicitly restrict by year of authorship.

In order to identify papers fulfilling the above criteria, we searched 10 meta-databases through EBSCOhost: the Education Resources Information Center (ERIC), Academic Search Complete, Business Source Complete, Econlit with Full Text, Education Full Text (H.W.Wilson), Education Index Retrospective:1929-1983, Education Source, Educational Administration Abstracts, Social Science Full Text (H.W.Wilson), Teacher Reference Center and EconLit. We looked for articles containing the terms _("teacher training" OR "teacher education" OR "professional development") AND (learning OR scores OR attainment) AND (“impact evaluation” OR effects) AND (“developing country 1” OR “developing country 2” OR ... “developing country N”),_ where “developing country” was replaced by country names _._ 9
The search yielded 6,049 results and automatically refined the results by removing exact duplicates from the original results, which reduced the number of results to 4,294. To this we added 20 impact evaluations which mention teacher training from the sample of a review of education reviews conducted last year and available online (Evans and Popova, 2016). We examined the 4,314 results from both sources to exclude articles that from their title and blurb, were clearly not impact evaluations of teacher training programs. This review process excluded 4,272 results and left us 42 full articles to assess their eligibility. After going through the full texts, another 18 papers were excluded as they did not meet the inclusion criteria. This yielded the final 23 papers, evaluating 26 programs between them. The search process is detailed in Figure 1. The 23 papers are listed in Appendix A.

# **The In-Service Teacher Training Survey Instrument (ITTSI)**

The ITTSI was designed based on (a) the descriptive, impact evaluation, and theoretical literatures characterized above, and (b) the authors’ previous experience studying in-service teacher-training. Drawing on these, we drafted a list of key indicators to capture details about a range of program characteristics falling into four categories: Overarching Aspects, Content, Delivery, and Perceptions. The ITTSI is summarized in Figure 2.

Taking each of these in turn, the Overarching Aspects section includes items such as the type of organization responsible for the design and implementation of a given teacher training program, to whom the program is targeted, what (if any) complementary materials it provides, the scale of the program, and its cost. Content includes indicators capturing the type of knowledge or skills that a given program aims to build among beneficiary teachers, for example, whether the program focuses on subject content (and if so, which), pedagogy, new technology, classroom management, counseling, assessment, or some combination of these.

Delivery focuses on indicators capturing program implementation details, such as whether 10
it is delivered through a cascade model (where the program trains trainers who in turn train the teachers), the profile of the trainers who directly train the teachers, the location of the training, the size of sessions, and the time division between lectures, practice, and other activities. Finally, the Perceptions section includes indicators capturing program implementers’ own perceptions of program effectiveness, such as which elements were responsible for any positive impacts, and which were popular or unpopular among teachers.

During the first phase of data collection, in which we coded the information reported in our sample of impact evaluations, as we learned more about the programs we added new indicators to our instrument and adjusted existing ones in an iterative process so as to accurately classify the full range of programs and the dimensions along which they differ. This resulted in a draft instrument consisting of a total of 51 indicators, for which we collected and analyzed data, the results of which are reported in this paper. Subsequent to this data collection and analysis, we shared our results with a series of experts and updated the indicators based on their feedback, including the addition of a series of questions specific to online programs. The resulting final version of the instrument, which includes 70 indicators plus 3 pieces of meta-data, is presented in Appendix B.

# **Data Collection**

Data collection and coding for our sample of 26 evaluated programs comprised two phases. The first of these phases consisted of carefully reviewing the impact evaluation studies and coding the information they provide. The draft version of the instrument for which we collected data included 51 indicators in total, and on average, information on 26 (50%) of these indicators was reported in the impact evaluations. Crucially, the amount of program information reported across the impact evaluations varies noticeably by topic (Table 1). As much as 60% of details concerning overarching aspects of teacher training programs – such as whether the program was designed by a government or non-governmental organization (NGO) – can be extracted from the evaluations. In 11
contrast, on average, only 43% and 36% of information concerning program content and delivery, respectively, is reported. This is of particular concern given that theory suggests certain aspects of delivery are crucial, such as how much practice the program involves and whether it is delivered through a cascade model.

The second phase of data collection sought to fill this gap in reported data by interviewing individuals involved in the actual implementation of each program. To do this, we emailed the authors of each of the impact evaluations in our sample, asking them to connect us with the program implementers. After three attempts to contact the implementers, we received responses from authors for 17 of the 26 programs. We contacted all of the individuals to whom the authors referred us – who in many cases directed us to more relevant counterparts – and were eventually able to hold interviews with program implementers for 12 of the 26 programs. In four cases, program implementers failed to schedule an interview after three attempts at contact, and in the case of one older program, the implementer had passed away. Interviews were held over the phone or in-person, and lasted between 45 and 90 minutes for each program. The interviews loosely followed the survey instrument, but included open-ended questions and space for program implementers to provide any additional program information which they perceived as important.

For the 12 programs for which we conducted interviews, we were able to collect information for an average of 50 out of the 51 (98%) indicators of interest. Consequently, conducting interviews decreased the differences in data availability across categories. The pooled average of indicators for which we had information after conducting interviews (for interviewed and not interviewed programs combined) increased to 80% for Overarching Aspects indicators, 67% of Content indicators, and 68% of Delivery indicators (Table 1).

# **Analytical Strategy**

For our sample of in-service teacher training programs, we analyze which characteristics of 12
teacher training programs are associated with the largest improvements in student learning, as measured by test score gains. We conduct both quantitative and qualitative analyses. The analytical strategy for the quantitative analysis consists of estimating a bivariate linear regression of the form: where _Y_ is the standardized impact on student test scores, _X_ an indicator of a given program characteristic, _α_ is the constant _, β_ is the coefficient on _X, ε_ is the estimation error, and _i_ represents each in-service teacher training program in the sample.

In preparation for this analysis, we standardize the impact estimates, _Y_ , for each of the programs. We convert the independent program variables, _X_ , to dummy variables wherever possible to facilitate comparability of coefficients.

First, although our sample of impact evaluations has a common outcome – impact on student test scores – these are reported on different scales across studies, based on different sample sizes. We standardize these effects and the associated standard errors in order to be able to compare them directly. Our unit of analysis for effect size is an experimental or quasiexperimental pair, where a group of students taught by teachers who participated in a given training program is compared to a control group taught by teachers who did not receive said training. Almost all the studies in our sample used difference-in-differences methods to estimate the effect of the teacher training programs – or the larger programs of which training is a subcomponent – on student learning and reported the effect size as a raw mean difference, _D_ , between treatment and control groups, before and after a given program. Following Borenstein, Hedges, Higgins, & Rothstein (2009), we calculate the standardized effect size or mean difference, _d_ , for each estimate, by dividing the raw mean difference, _D_ , by the pooled standard deviation, _Spooled_ , as follows: 13
D d= (Equation 1) Spooled _Spooled_ is the within-estimate standard deviation for the treatment and control groups combined. Where this is not directly reported in the studies we calculate it using the following equation derived from Borenstein et al. (2009): where _n1_ is the sample size for the treatment group, _n2_ is the sample size for the control group, and _SED_ is the standard error of the raw mean difference. For a complete derivation of Equation 2 please see the mathematical appendix in Appendix C.

Turning to the independent variables, as originally coded, the 51 indicators for which we collected information capturing various design and implementation characteristics of the training programs took a number of forms. These consisted of dummy variables (e.g., the intervention provides textbooks alongside training = 0 or 1), categorical variables (e.g., the primary focus of the training was subject content [=1], pedagogy [=2], new technology [=3]), continuous variables (e.g., the proportion of training hours spent practicing with students), and string variables capturing open-ended perceptions (e.g., which program elements do you think were most effective?). In order to maximize the comparability of output from our regression analysis we convert all categorical and continuous variables into dummy variables. For categorical variables, this is straightforward. For example, we convert the original categorical variable for the location of the initial teacher training – which includes response options of schools, a central location, a training center, or online – into four dummy variables. In order to convert the continuous variables to a comparable scale, we create a dummy for each continuous variable which, for a given program, takes a value of 1 if the continuous variable is greater than the median value of this variable across all programs, and a value of 0 if it is less than or equal to the value of this variable across all 14
programs. We apply this method to the conversion of all continuous variables except three – proportion of teachers that dropped out of the program, number of follow-up visits, and weeks of distance learning – which we convert directly to dummy variables that take a value of 1 if the original variable was greater than 0, and a value of 0 otherwise.

We then conduct bivariate regressions on this set of complete dummy variables with continuous impact estimates on test scores as our dependent variable for each regression. We supplement this regression analysis with a qualitative analysis of what works, relying on the selfreported perceptions of program implementers along three dimensions: (a) which program elements they identified as most responsible for any positive impacts on student learning, (b) which elements, if any, teachers particularly liked, and (c) which elements, if any, teachers particularly disliked.

# **Results**

This section contributes to the thus far limited knowledge base on the specific elements of teacher training programs that successfully improve student learning in low- and middle-income countries by presenting the results of our quantitative and qualitative analyses, in turn, for our sample of all programs whose impact has been evaluated. We first present descriptive statistics for all program characteristics for our sample of 26 programs, before proceeding to the results of our quantitative and qualitative analyses. It is worth remembering that these are characteristics of programs that have been rigorously evaluated, which are likely very different from the universe of teacher training programs. For example, one might imagine that the programs that are rigorously evaluated are those that researchers expected to have the highest probability of being effective, or those that are small-enough scale to permit a comparison group.

We then present associations between gains in student learning and characteristics of teacher training programs. With 26 observations, the statistical power to detect any significant 15
impacts is extremely low, so this exercise is principally useful for observing a pattern of suggestive results rather than drawing any definitive conclusions. If future impact evaluations gather a consistent set of indicators – using, for example, the ITTSI, then more conclusive analysis will be possible at a later date.

# **What do Evaluated Teacher Training Programs Look Like?**

For each of our quantifiable instrument categories - Overarching Aspects, Content, and Delivery – we calculate means and standard deviations for all indicators so as to describe, in detail, what programs in our sample look like, before analyzing which of these program characteristics are conducive to improving student learning. In what follows we summarize some of the more common program traits across the sample; full descriptive statistics can be found in Tables 2-4. Taking first the Overarching Aspects of the training programs in our sample (Table 2), these programs are most commonly designed by researchers (46%) and based on some kind of formal diagnostic or evaluation (41%). Consistent with the fact that most evaluated programs are pilot programs run by researchers or non-government organizations (73% combined), they tend to be small in scale with an average of 609 teachers receiving training across 57 schools, per year. The majority of programs target beneficiary teachers by the grade they teach (79%), and some target teachers by subject (25%), but none targets teachers based on their years of experience or specific skill gaps. Interestingly, for a large proportion of programs (41%), participation does have implications for salary or promotion.

Turning next to Content-related program characteristics (Table 3), the most common primary focus of the programs in our sample is on pedagogy (46%) - as opposed to technology or classroom management, etc. – and the most common secondary focus is on subject content (68%). Within subjects, the majority of programs focus on either literacy/language, or math, or both (90%). Most training programs are also linked with some sort of materials provision – for 16
example, textbooks, storybooks, teacher manuals, lesson plans – (82%), and involve lectures (93%), discussion (64%), and lesson enactment (57%), with fewer including the use of scripted lessons (33%) or training on how to conduct diagnostics of student performance (33%).

Finally, in terms of Delivery characteristics (Table 4), almost all of the programs in our sample have teachers meet face-to-face for some number of consecutive days of initial training (92%), and in most cases this training takes place at a central location such as a local government building or hotel conference room (82%), as opposed to in-school (6%). For half the programs this training is delivered using a cascade model. The majority of programs also include some kind of follow-up support visits (78%), with an average of 6 visits received per teacher. These visits are mixed between monitoring (47%) and in-class pedagogical support (41%), and are less often used for reviewing material (12%). Overall, programs provide on average 64 hours of training in total, with 48% of this time dedicated to lectures, 52% involving practice with other teachers, and only 6% geared towards practicing with students.

# **Quantitative Analysis**

We discuss, for each of our quantifiable instrument categories - Overarching Aspects, Content, and Delivery – those variables we observe to be most associated with student learning gains. Tables 5, 6, and 7 present the results of our bivariate regressions for each of these categories in turn, in each case reporting the results for the five variables displaying the largest association with program impact in absolute value, in descending order. The complete set of regression results for all variables included in our instrument, similarly presented by category, are included in Appendix D.

Among Overarching Aspects (Table 5), the largest impacts on student learning come from the provision of certain materials alongside the teacher training. In this sample, providing textbooks alongside training is associated with a test score gain of 0.36 standard deviation 17
(significant with 95% confidence), the provision of storybooks is associated with a gain of 0.13 standard deviation (not significant), and providing other reading materials increases the likelihood of improving student learning by 0.16 standard deviation (significant at 90%). The only other characteristic – given our small sample – with a statistically significant association relates to incentives. When participation in a teacher training programs has explicit implications for promotion, salary increases, or allows teachers to earn points that contribute to either of these, test scores are 0.14 standard deviation higher (significant at 90%). Targeting participant teachers by their years of experience, although insignificant, has the next largest association with student learning, also at 0.14 standard deviation. However, this is entirely driven by a single program, the Balsakhi program in rural India which trains women from the local community who have completed secondary school to provide remedial education to students falling behind (Banerjee, Cole, Duflo, & Linden, 2007). Indeed, that is the only program out of the 26 that explicitly targeted teachers based on their experience.

Among the Content variables (Table 6), a primary focus on classroom management is associated with the largest gain in test scores, with those programs showing 0.47 standard deviations greater impact on student learning. But this again is driven by a single program, the InService Teacher Education program in Thailand (Nitsaisook & Anderson, 1989), which is a cluster randomized-control trial but with few clusters. In contrast, in general, training programs that are not focused on a given subject – in other words, those that are focused on pedagogy, new technology, classroom management, assessment, or counseling, without direct application to math, language or another subject – are associated with 0.24 lower standard deviation in student learning. At the same time, programs which have a primary focus on pedagogy or training, and those that have a secondary focus on subject content are both associated with 0.18 standard deviation larger impacts on student learning. Likely due to limited statistical power, none of these associations are 18
# statistically significant.

Turning to Delivery characteristics (Table 7), the highest association with student learning gains is holding face-to-face training in a university or training center (as opposed to a central location such as a hotel or government administrative building, which was the omitted category), which is associated with 0.39 higher standard deviation in student learning (significant at 90%). The inclusion of follow-up visits to review material taught in the initial training – as opposed to visits for monitoring purposes alone or no follow-up visits – improves program impact on student learning by 0.26 standard deviation. The profile of teacher trainers seems important, with using researchers or local government officials – as opposed to education practitioners of some sort – as the trainers in direct contact with teachers being associated with 0.20 and 0.17 standard deviation lower program impacts on student test scores, respectively. Finally – and in alignment with recent literature highlighting the overly theoretical nature of many training programs as an explanation for their limited effects on student learning – the proportion of training time spent practicing with other teachers is highly correlated with learning impacts, with being above the median for the sample being associated with 0.17 standard deviation higher impacts on learning.

# **Qualitative Analysis**

We supplement the quantitative results with an analysis of self-reported perceptions by program implementers about the elements of their programs which they believe are most responsible for any positive effects on student learning, as well as those elements which were popular and unpopular among the beneficiary teachers. We elicited these perceptions using openended questions and then tallied up the number of program implementers that mentioned a given program element in their response, albeit not necessarily using the exact same language as other respondents. Table 8 presents the results for the program elements that implementers say work. Four of 13 interviewees – the most common response – mentioned that mentoring follow-up visits 19
were a crucial component in making their training work, casting doubt on the effectiveness of faceto-face initial training in the absence of follow-up. The next most commonly perceived elements responsible for positive effects on student learning were engaging teachers for their opinions and ideas – either through discussion or text messages – and designing the program in response to local context – building on what teachers already do and linking to everyday experiences – which were both mentioned by 3 of 13 interviewees. The other program element which received multiple mentions was the materials provided by the program, cited by 2 of 13 interviewees.

We asked similar questions of the program implementers about the program elements they perceived that teachers liked and disliked the most about their training programs, coding the responses in the same way and, interestingly, we only found one common response in each case. Tables 9 and 10 present those program elements which implementers reported were popular and unpopular among teachers, respectively. Three of the 13 interviewees reported that the part of their program that teachers most enjoyed was that it was fun and engaging (or some variation of that). In other words, teachers appreciated that certain programs were interactive and involved participation, discussion, and workshops rather than passive learning. Nine other program elements were mentioned by one implementer each as being popular among teachers, including the provision of concrete steps to follow, lesson plans, and materials, as well as a number of variables which are in a sense outcomes themselves, such as increased confidence and knowledge about content and technology.

Similarly, for unpopular program elements, 3 of the 13 program implementers we interviewed reported that teachers disliked the amount of time taken by participating in the training programs, which they perceived as excessive. In addition, one interviewee also mentioned that teachers complained about the high degree of control that the program exerts over how they exactly they use their time, rather than the amount of time committed to the program per se.

20
Beyond this, another 6 program elements were mentioned once each, with the most widely applicable of these including that training was too theoretical and that training was not linked to the curriculum.

# **Discussion**

Governments spend enormous amounts of time and money on in-service teacher training. Many countries have many different in-service teacher training programs running simultaneously, and it is likely that many are ineffective. This paper demonstrates that there are associations with some characteristics of teacher training programs and student test score gains, such as the inclusion of supplemental materials, follow-up visits, and focus on a specific subject. However, this review reveals broad weakness in reporting on these interventions. There are almost as many program types as there are programs, with variations in subject and pedagogical focus, hours spent, capacity of the trainers, and a host of other variables. Yet reporting on them often seeks to reduce them to a small handful of variables, and each scholar decides independently which variables are most relevant.

We propose a standard set of indicators – the ITTSI – that would encourage consistency and thoroughness in reporting. Academic journals may continue to pressure authors to report limited information about the interventions, wishing instead to reserve space for statistical analysis. However, authors could easily include the full set of indicators in an appendix – attached to the paper or on-line. It is only through developing a more thorough and consistent understanding of teacher training programs that practitioners can learn what to emulate from the best programs and what to avoid from the worst.

21
# **References**

Baker, S., & Smith, S. (1999). Starting off on the right foot: The influence of four principles of professional development in improving literacy instruction in two kindergarten programs. _Learning Disabilities Research and Practice_ , _14_ (4), 239-253. doi:10.1207/sldrp1404_5 Banerjee, A., Cole, S., Duflo, E., & Linden, L. L. (2007). Remedying education: Evidence from two randomized experiments in India. _The Quarterly Journal of Economics, 122_ , 12351264. doi:10.1162/qjec.122.3.1235 Borenstein, M., Hedges, L.V., Higgins, J.P.T., & Rothstein, H.R. (2009). _Introduction to metaanalysis._ Chichester, United Kingdom: John Wiley & Sons.

Bourgeois, E., & Nizet, J. (1997). _Aprendizaje y formación de personas adultas._ Paris, France: Presses Universite de France.

Cardemil, C. (2001). Procesos y condiciones en el aprendizaje de adultos. _Jornada Nacional de Supervisores. Supervisión para aprendizajes de calidad y oportunidades para todos. Educación Rural._ Santiago: Ministerio de Educación. Retrieved from http://biblioteca.uahurtado.cl/ujah/Reduc/pdf/pdf/mfn253.pdf. Accessed December 16<sup>th</sup> , 2016.

Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. _The American Economic Review_ , _104_ (9), 2633-2679. doi:10.1257/aer.104.9.2633 Darling-Hammond, L. (2005). Teaching as a profession: Lessons in teacher preparation and professional development. _Phi delta kappan_ , _87_ (3), 237.

doi:10.1177/003172170508700318 22
Darling-Hammond, L. Wei, R. C. & Andree, A. (2010). _How high-achieving countries develop great teachers_ . Stanford, CA: Stanford Center for Opportunity Policy in Education.

Retrieved from http://www.oup.hu/howhigh_doug.pdf. Accessed December 16<sup>th</sup> , 2016.

Darling-Hammond, L., Wei, R. C., Andree, A., Richardson, N., & Orphanos, S. (2009). _Professional learning in the learning profession._ Washington, DC: National Staff Development Council. Retrieved from http://www.ostrc.org/docs/document_library/ppd/Professionalism/Professional%20Learnin g%20in%20the%20Learning%20Profession.pdf. Accessed December 16<sup>th</sup> , 2016.

Evans, D. K. & Popova, A. 2016. What really works to improve learning in developing countries? An analysis of divergent findings in systematic reviews. _World Bank Research Observer_ . doi:10.1093/wbro/lkw004 Fryer Jr, R. G. (2016). _The production of human capital in developed countries: Evidence from 196 randomized field experiments_ (NBER Working Paper No. 22130). Cambridge, MA: National Bureau of Economic Research.

Gersten, R., Taylor, M. J., Keys, T.D., Rolfhus, E., and Newman-Gonchar, R. (2014). _Summary of research on the effectiveness of math professional development approaches._ Washington, DC: Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, U.S. Department of Education; and Regional Educational Laboratory Southeast at Florida State University. Retrieved from http://files.eric.ed.gov/fulltext/ED544681.pdf. Accessed December 16<sup>th</sup> , 2016.

Hanushek, E. A. & Rivkin, S.G. (2010). _Using value-added measures of teacher quality_ (CALDER Brief No. 9). National Center for Analysis of Longitudinal Data in Education 23
Research. Retrieved from http://files.eric.ed.gov/fulltext/ED509683.pdf. Accessed December 16<sup>th</sup> , 2016.

Huberman, M. 1989. The professional life cycle of teachers. _Teachers College Record_ , _91_ (1): 3157.

Knowles, M. S., Holton, E. F., & Swanson, R. A. (2005). _The adult learner_ (6<sup>th</sup> ed.). Burlington, MA: Elsevier.

Kraft, M. A., & Papay, J. P. (2014). Can professional environments in schools promote teacher development? Explaining heterogeneity in returns to teaching experience. _Educational Evaluation and Policy Analysis, 36_ (4), 476-500. doi:10.3102/0162373713519496 Kremer, M., Brannen, C., & Glennerster, R. (2013). The challenge of education and learning in the developing world. _Science_ , _340_ , 297-300. doi:10.1126/science.1235350 Kremer, M., Duflo, E. & Dupas, P. (2011). Peer effects, teacher incentives, and the impact of tracking. _American Economic Review, 101_ , 1739 -1774. doi:10.1257/aer.101.5.1739 La Paro, K. M., & Pianta, R. C. (2003). _CLASS: Classroom assessment scoring system_ . Charlottesville, VA: University of Virginia.

McEwan, P. (2015). Improving learning in primary schools of developing countries: A metaanalysis of randomized experiments. _Review of Educational Research, 85_ (3), 353-394. doi:10.3102/0034654314553127 MINEDUC. (2009). _Resultados nacionales SIMCE 2008_ . Santiago: Ministerio de Educación.

Nitsaisook, M., & Anderson, L.W. (1989). An experimental investigation of the effectiveness of inservice teacher education in Thailand. _Teaching & Teacher Education_ , _5_ (4), 287-302.

24
# doi:10.1016/0742-051X(89)90027-9

Piper, B., & Korda, M. (2011). _EGRA Plus: Liberia_ (Program evaluation report). Durham, NC: RTI International. doi:10.1016/0742-051X(89)90027-9 Stallings, J. (1977). _Learning to look: A handbook on classroom observation and teaching models_ . Belmont, CA: Wadsworth Publishing Company.

TNTP. (2015). _The mirage: Confronting the hard truth about our quest for teacher development_ . The New Teacher Project. Retrieved from http://files.eric.ed.gov/fulltext/ED558206.pdf. Accessed December 16<sup>th</sup> , 2016.

Villegas-Reimers, E. (2003). _Teacher professional development: An international review of the literature_ . Paris: UNESCO International Institute for Educational Planning. Retrieved from http://xa.yimg.com/kq/groups/23312027/1804288575/name/teacher+professional+develop ment+an+international+review+from+the+literature.pdf. Accessed December 16<sup>th</sup> , 2016.

Walter, C., & Briggs, J. (2012). _What professional development makes the most difference to teachers._ Oxford: University of Oxford Department of Education. Retrieved from http://clie.org.uk/wp-content/uploads/2011/10/Walter_Briggs_2012.pdf . Accessed December 16<sup>th</sup> , 2016.

Wood, F. H., & McQuarrie, F. Jr. (1999). On the job learning. New approaches will shape professional learning in the 21st century. _Journal of Staff Development_ , _20_ , 10-13.

World Bank. (2013). _What matters most for teacher policies: A framework paper_ (Systems Approach for Better Education Results Working Paper Series No. 4). Washington, DC: World Bank.

25
Yoon, K.S., Duncan, T., Lee, S.W.Y., Scarloss, B., & Shapley, K. (2007). _Reviewing the evidence on how teacher professional development affects student achievement_ (Issues & Answers Report No. 033). Washington, DC: Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, U.S. Department of Education; and Regional Educational Laboratory Southeast at Florida State University. Retrieved from http://files.eric.ed.gov/fulltext/ED498548.pdf. Accessed December 16<sup>th</sup> , 2016.

26
# Table 1

_Data collection from studies vs. interviews_

|From impac|t evalua<br>only|Data c<br>tion reports|ollected<br>After int<br>impl|erviews with<br>ementers||
|---|---|---|---|---|---|
|Number of<br>indicators|Per<br>ind|centage of<br>icators|Number of<br>indicators|Percentage of<br>indicators|Total<br>indicators|
|Overarching<br>aspects<br>15.6||60%|20.8|80%|27|
|Content<br>11.3||43%|17.4|67%|10|
|Delivery<br>9.4||36%|17.7|68%|14|
|TOTAL<br>25.50||50%|38|74%|51|
|For interviewed programs o|nly:||50|98%|51|

27
# Table 2

# _Overarching Aspects – Descriptive statistics_

|OverarchingAspects variable|Mean<br>Stan|dard deviation|
|---|---|---|
|Teachers pay some cost for the training (including their own transport) = 0|0|0|
|Number of teachers receiving training under this program in the last year|609.0714|1619.068|
|Number of schools in which the program was implemented|56.6667|44.098|
|Number of years the program been running|2.9605|3.087|
|Proportion of teachers who dropped out of the program in the last year|0.1739|0.2774|
|Targeting by geography|0.5417|0.509|
|Targeting by subject|0.25|0.4423|
|Targeting by grade|0.7917|0.4149|
|Targeting by years of experience|0.0417|0.2041|
|Targeting by skill gaps|0|0|
|Targeting by contract teachers|0.125|0.3378|
|Program provides materials|0.8182|0.3948|
|Program provides textbooks|0.0952|0.3008|
|Program provides storybooks|0.2857|0.4629|
|Program provides computers|0.1429|0.3586|
|Program provides teacher manuals|0.5714|0.5071|
|Program provides lesson plans/videos|0.381|0.4976|
|Program provides scripted lessons|0.0952|0.3008|
|Program provides craft materials|0.1429|0.3586|
|Program provides other reading materials (flashcards, word banks, primers)|0.3333|0.483|
|Program provides software|0.2727|0.4558|
|Program designed by Government|0.1538|0.3679|
|Program designed by NGO or social enterprise|0.3846|0.4961|
|Program designed by researchers|0.4615|0.5084|
|Program implemented by Government|0.2692|0.4523|
|Program implemented by NGO or social enterprise|0.3846|0.4961|
|Program implemented by researchers|0.3462|0.4852|
|Program design not based on any kind of diagnostics|0.2353|0.4372|
|Program design based on informal diagnostic|0.3529|0.4926|
|Program design based on formal diagnostic|0.4118|0.5073|
|Participation has no implications for salary or promotion|0.4706|0.5145|
|Participation has implications for status only|0.1176|0.3321|
|Participation has implications for promotion or points towards promotion or salary<br>implications|0.4118|0.5073|
|Teachers are not evaluated|0.3333|0.488|
|Positive consequence if teachers are well evaluated|0.2|0.414|
|No positive consequence if teachers are well evaluated|0.4667|0.5164|
|Negative consequence if teachers are well evaluated|0.1333|0.3519|
|No negative consequence if teachers are well evaluated|0.5333|0.5164|

28
# Table 3

# _Content – Descriptive statistics_

|Content variable|Mean|Standard<br>deviation|
|---|---|---|
|Primary focus is subject content|0.0833|0.2823|
|Primary focus is pedagogy|0.4583|0.509|
|Primary focus is technology|0.2917|0.4643|
|Primary focus is counseling|0.0833|0.2823|
|Primary focus is classroom management|0.0417|0.2041|
|Secondary focus is subject content|0.6842|0.4776|
|Secondary focus is pedagogy|0.2105|0.4189|
|Secondary focus is technology|0.1053|0.3153|
|No subject focus|0.087|0.2881|
|Subject focus is literacy/language|0.6957|0.4705|
|Subject focus is math|0.2174|0.4217|
|Subject focus is science|0.0435|0.2085|
|Subject focus is information technology|0.0435|0.2085|
|Training involves lectures|0.9286|0.2673|
|Training involves discussion|0.6429|0.4972|
|Training involves lesson enactment|0.5714|0.5136|
|Training involves materials development|0.2857|0.4688|
|Training involves training on how to conduct diagnostics|0.3333|0.488|
|Training involves lesson planning|0.5789|0.5073|
|Traininginvolves the use of scripted lessons|0.3333|0.4851|

29
# Table 4

# _Delivery – Descriptive statistics_

|Deliveryvariable|Mean|Standard<br>deviation|
|---|---|---|
|Total hours of face-to-face training|64.04|45.5764|
|Over how many weeks?|11.79|14.5273|
|Over how many months?|11.55|8.2874|
|Proportion of training spent in lectures|0.478|0.2917|
|Proportion of training spent practicing with other teachers|0.515|0.3192|
|Proportion of training spent practicing with students|0.062|0.0829|
|How many in-school follow-up support visits do teachers receive after the initial<br>training?|5.833|9.1684|
|How many weeks of distance learning does the program include (if any)?|1.438|4.3354|
|Is it a cascade training model (i.e. one where program trainers trainers who the|0.5|0.513|
|Is there a part of the training where teachers meet with trainers for several da|0.92|0.2769|
|Includes follow up visits|0.778|0.4278|
|Includes distance learning|0.188|0.4031|
|Trainers are primary or secondary teachers|0.222|0.4278|
|Trainers are experts - university professors or have graduate degrees in education|0.333|0.4851|
|Trainers are researchers|0.056|0.2357|
|Trainers are local government officials|0.333|0.4851|
|Training held at schools|0.063|0.25|
|Training held at central location including hotel conference room etc.|0.813|0.4031|
|Training held at university or training center|0.125|0.3416|
|Follow-up visits for in-class pedagogical support|0.412|0.5073|
|Follow-up visits for monitoring|0.471|0.5145|
|Follow-upvisits to review material|0.118|0.3321|

30
# Table 5

_Overarching Aspects – 5 largest regression coefficients_

|Overarching Aspects variable|Program impact<br>on student<br>learning|
|---|---|
|Program provides textbooks|0.355**<br>(0.128)|
|Program provides other reading materials (flashcards, word banks, primers)|0.159*<br>(0.087)|
|Participation has implications for promotion or points towards promotion or<br>salary implications|0.143**<br>(0.066)|
|Targeting by years of experience|0.136<br>(0.198)|
|Program provides storybooks|0.129<br>(0.094)|
|Observations|26|
|Standard errors inparentheses. ***p<0.01, **p<0.05, *p<0.1.||

31
Table 6 _Content – 5 largest regression coefficients_

|Content variable|Program impact<br>on student<br>learning|
|---|---|
|Primary focus of the training program is classroom management|0.471<br>(0.272)|
|No subject focus of training|-0.243<br>(0.204)|
|Secondary focus of the training program is subject content|0.182<br>(0.156)|
|Primary focus of the training program is new technology|0.180<br>(0.206)|
|Primary focus of the training program is pedagogy|0.177<br>(0.201)|
|Observations|26|
|Standard errors inparentheses. ***p<0.01, **p<0.05, *p<0.1.||

32
# Table 7

# _Delivery – 5 largest regression coefficients_

|Delivery variable|Program impact<br>on student<br>learning|
|---|---|
|Training takes place in university or training center|0.385**<br>(0.142)|
|Follow-up visits to review material|0.256<br>(0.156)|
|Most common profile of the direct trainers is researchers|-0.196<br>(0.336)|
|Most common profile of the direct trainers is local government officials|-0.170<br>(0.257)|
|Proportion of training spent practicing with other teachers|0.169<br>(0.134)|
|Observations|26|
|Standard errors inparentheses. ***p<0.01, **p<0.05, *p<0.1.||

33
# Table 8

_Trainers’ perceptions of most effective program elements_

|Program element|Tally|
|---|---|
|Mentoring follow-up visits|4|
|Engaging teachers for their opinions and ideas - either through<br>discussion or text messages|3|
|Programs designed in response to local context - building on what<br>teachers already do & linking to everyday experiences|3|
|Materials|2|
|Assessment|1|
|Giving status to teachers|1|
|Making learning fun for teachers and students|1|
|Quality of training|1|
|Support of headmasters and mayor|1|
|Cascade model|1|
|Making teachers aware that they should communicate with students|1|
|Establishing agreed upon characteristics for what makes a good lesson|1|
|Time between sessions for teachers to work independently|1|
|Use of open source math software|1|
|Tally reflects the number of program implementers (of the 13 interviewe<br>who mentioned in an open-ended question that they believed a give<br>element was one of the most important for the effectiveness of their<br>teacher training program.|d in total)<br>n program<br>in-service|

34
# Table 9

_Popular program elements among teachers_

|Program element|Tally|
|---|---|
|Fun - including participation, discussion, and workshops rather|3|
|than traditional lecture format with passive learning||
|Concrete steps to follow|1|
|Improved their content knowledge|1|
|Increased group work|1|
|Methodology|1|
|Lesson plans|1|
|Learning to use computers ahead of the curve|1|
|Motivational video|1|
|Gave confidence to teachers|1|
|Materials|1|

Tally reflects the number of program implementers (of the 13 interviewed in total) who mentioned in an open-ended question that they believed a given <u>program element was particularly liked by teachers.</u> 35
# Table 10

_Unpopular program elements among teachers_

|Program element|Tally|
|---|---|
|High demands on their time|3|
|Courses did not cover some newer material|1|
|Giving children information on the costs of schooling|1|
|High control of how they use their time|1|
|Not all schools and years received the program|1|
|Teaching in English|1|
|Too theoretical|1|
|Trainingnot linked to curriculum|1|

Tally reflects the number of program implementers (of the 13 interviewed in total) who mentioned in an open-ended question that they believed a given <u>program element was particularly disliked by teachers.</u> 36
<!-- Start of picture text --> Identification<br><!-- End of picture text --> <!-- Start of picture text --> Screening<br><!-- End of picture text --> <!-- Start of picture text --> Eligibility<br><!-- End of picture text --> <!-- Start of picture text --> Included<br><!-- End of picture text --> <!-- Start of picture text --> 4,294 records identified through<br>search of databases<br><!-- End of picture text --> <!-- Start of picture text --> 20 records identified through other<br>sources<br><!-- End of picture text --> <!-- Start of picture text --> 4,272 records excluded<br><!-- End of picture text --> <!-- Start of picture text --> All records screened<br><!-- End of picture text --> <!-- Start of picture text --> 42 full texts assessed for eligibility<br><!-- End of picture text --> <!-- Start of picture text --> 18 full texts excluded<br><!-- End of picture text --> <!-- Start of picture text --> 23 studies (26 programs) included<br><!-- End of picture text --> _Figure 1._ Search process and results 37
<!-- Start of picture text --> Overarching Content Delivery Perceptions<br>Focus?<br>Who What did<br>implemented? (Content, Core activities? teachers like?<br>pedagogy, etc.)<br>Professional What do you<br>Subject area? Cascade?<br>implications? think mattered?<br>Proportion in<br>Based on a<br>lecture?<br>diagnostic?<br>Practice?<br><!-- End of picture text --> _Figure 2._ Summary of the in-service teacher training survey instrument (ITTSI) 38
# **Appendix A: The List of Included Papers**

Abeberese, A. B., Kumler, T. J., & Linden, L. L. (2014). Improving reading skills by encouraging children to read in school: A randomized evaluation of the Sa Aklat Sisikat reading program in the Philippines. _Journal of Human Resources_ , _49_ (3), 611-633. doi:10.1353/jhr.2014.0020 Angrist, J., & Lavy, V. (2001). Does teacher training affect pupil learning? Evidence from matched comparisons in Jerusalem public schools. _Journal of Labor Economics, 19_ , 34369. doi:10.1086/319564 Banerjee, A. V., Banerji, R., Duflo, E., Glennerster, R., & Khemani, S. (2010). Pitfalls of participatory programs: Evidence from a randomized evaluation in education in India. _American Economic Journal: Economic Policy_ , _2_ (1), 1-30. doi:10.1257/pol.2.1.1 Banerjee, A., Cole, S., Duflo, E., & Linden, L. L. (2007). Remedying education: Evidence from two randomized experiments in India. _The Quarterly Journal of Economics, 122_ , 12351264. doi:10.1162/qjec.122.3.1235 Barrera-Osorio, F., & Linden, L. L. (2009). _The use and misuse of computers in education: Evidence from a randomized experiment in Colombia_ (World Bank Policy Research Working Paper No. 4836). Washington, DC: World Bank. doi:10.1596/1813-9450-4836 Beuermann, D. W., Naslund-Hadley, E., Ruprah, I. J., & Thompson, J. (2013). The pedagogy of science and environment: Experimental evidence from Peru. _The Journal of Development Studies_ , _49_ (5), 719-736. doi:10.1080/00220388.2012.754432 Brooker, S., Inyega, H., Estambale, B., Njagi, K., Juma, E., Jones, C., et al. (2013). _Impact of malaria control and enhanced literacy instruction on educational outcomes among Kenyan_ 39
_school children: A multi-sectoral, prospective, randomized evaluation_ . 3ie Draft Grantee Final Report. Washington, DC: International Initiative for Impact Evaluation. Retrieved from http://erepository.uonbi.ac.ke/bitstream/handle/11295/81003/Brooker_Impact of malaria control and enhanced literacy.pdf?sequence=1&isAllowed=y. Accessed December 16<sup>th</sup> , 2016.

Carillo, P., Onofa, M., & Ponce, J. (2010). _Information technology and student achievement: Evidence from a randomized experiment in Ecuador_ (IDB Working Paper No. 223). Washington, DC: Inter-American Development Bank.

He, F., Linden, L. L., & MacLeod, M. (2008). _How to teach English in India: Testing the relative productivity of instruction methods with Pratham English Language Education Program._ Unpublished manuscript, Columbia University, New York, NY.

Kerwin, J. T., & Thornton, R. (2015). _Making the grade: Understanding what works for teaching literacy in rural Uganda_ . Unpublished manuscript, University of Michigan, Ann Arbor, MI.

Lai, F., Luo, R., Zhang, L., Huang, X., & Rozelle, S. (2012). _Does computer-assisted learning improve learning outcomes? Evidence from a randomized experiment in migrant schools in Beijing_ (REAP Working Paper No. 228). Stanford, CA: Stanford University, Rural Education Action Project.

Lai, F., Zhang, L., Qu, Q., Hu, X., Shi, Y., Boswell, M., et al. (2012). _Does computer-assisted learning improve learning outcomes? Evidence from a randomized experiment in public schools in rural minority areas in Qinghai_ (REAP Working Paper No. 237). Stanford, CA: Stanford University, Rural Education Action Project.

Loyalka, P., Liu, C., Song, Y., Yi, H., Huang, X., Wei, J., et al. (2013). Can information and 40
counseling help students from poor rural areas go to high school? Evidence from China. _Journal of Comparative Economics_ , _41_ , 1012-1025. doi:10.1016/j.jce.2013.06.004 Lucas, A. M., & Mbiti, I. M. (2012). Access, sorting, and achievement: The short-run effects of free primary education in Kenya. _American Economic Journal: Applied Economics_ , _4_ (4), 226–225. doi:10.1257/app.4.4.226

- Mo, D., Zhang, L., Lui, R., Qu, Q., Huang, W., Wang, J., et al. (2013). _Integrating computerassisted learning into a regular curriculum: Evidence from a randomized experiment in rural schools in Shaanxi_ (REAP Working Paper No. 248). Stanford, CA: Stanford University, Rural Education Action Project.

Nitsaisook, M., & Anderson, L.W. (1989). An experimental investigation of the effectiveness of inservice teacher education in Thailand. _Teaching & Teacher Education_ , _5_ (4), 287-302. doi:10.1016/0742-051X(89)90027-9

- Piper, B., & Korda, M. (2011). _EGRA Plus: Liberia_ (Program evaluation report). Durham, NC: RTI International. Retrieved from http://files.eric.ed.gov/fulltext/ED516080.pdf. Accessed December 16<sup>th</sup> , 2016.

- Pournara, C., Hodgen, J., Adler, J., & Pillay, V. (2015). Can improving teachers' knowledge of mathematics lead to gains in learners' attainment in Mathematics?. _South African Journal of Education_ , _35_ (3), 1-10. doi:10.15700/saje.v35n3a1083

- Spratt, J., S. King, & Bulat, J. (2013). Independent Evaluation of the Effectiveness of Institut pour

- l’Education Populaire’s “Read-Learn-Lead” (RLL) Program in Mali. Menlo Park, CA: The William and Flora Hewlett Foundation. Retrieved from

http://www.educationinnovations.org/sites/default/files/Mali RLL evaluation Endline 41
Report - REVISED 30nov2013 (6).pdf. Accessed March 10, 2017.

Tan, J., Lane, J., & Lassibille, G. (1999). Student outcomes in Philippine elementary schools: An evaluation of four experiments. _World Bank Economic Review_ , _13_ , 493-508. doi:10.1093/wber/13.3.493 Weir, C. J. , & Roberts, J. (1991). Evaluating a teacher Training project in difficult circumstances. In S. Anivan (Ed.), _Issues in Language Programme Evaluation in the 1990's. Anthology serires 27_ , 91-109. Singapore: Southeast Asian Ministers of Education Organization.

Zhang, D., & Campbell, T. (2012). An exploration of the potential impact of the integrated experiential learning curriculum in Beijing, China. _International Journal of Science Education_ , _34_ (7), 1093-1123. doi:10.1080/09500693.2011.625057 Zhang, L., Lai, F., Pang, X., Yi, H., & Rozelle, S. (2013). The impact of teacher training on teacher and student outcomes: evidence from a randomised experiment in Beijing migrant schools. _Journal of development effectiveness_ , _5_ (3), 339-358. doi:10.1080/19439342.2013.807862 42
## **Appendix B: The Survey Instrument**

# **<mark>In-Service Teacher Training Survey Instrument (ITTSI)</mark>**

|**In**<br>1<br>2|**troduction**<br>What is the name of the in-service teacher training program under discussion?<br>What is your full name?|
|---|---|
|3<br>**Ov**<br>|What is your role in this program?<br>**erarching aspects**<br>|
|4|By the end of this training what is it that you expect teachers to be able to do differently?|
|5|How many years has this program been running?|
|6|At what scale is this program implemented?<br>1.<br>National<br>2.<br>Multiple states or regions<br>3.<br>One state or region<br>4.<br>Less than one state or region|
|7|What kind of organization designed this teacher training program?<br>_(Select all that apply.)_<br>1.<br>Government<br>2.<br>Non-governmental organization<br>3.<br>Private company or social enterprise<br>4.<br>Researchers|
|8|What kind of organization is implementing this teacher training program?<br>_(Select all that apply.)_<br>1.<br>Government<br>2.<br>Non-governmental organization<br>3.<br>Private company or social enterprise<br>4.<br>Researchers|

43
|9|What percentage of the total time teachers spend in this training program detracts from their<br>regular teaching time?|
|---|---|
|10|Is the primary focus of this program teacher training, or is teacher training one part of a broader<br>program?<br>1.<br>Teacher training is primary focus<br>2.<br>Teacher training is one component|
|11|Was the program design based on a diagnostic or evaluation of student learning of some kind?<br>If so, what kind?|
||1.<br>No<br>2.<br>Yes, informal diagnostic<br>3.<br>Yes, formal diagnostic|
|12|Was the program design based on a diagnostic or evaluation of teacher skills of some kind? If<br>so, what kind?|
||1.<br>No<br>2.<br>Yes, informal diagnostic<br>3.<br>Yes, formal diagnostic|
|13|What teacher skill gaps is this program designed to support?|
||1.<br>Subject content<br>2.<br>Subject-specific pedagogy<br>3.<br>Technology<br>4.<br>Counseling<br>5.<br>Classroom management<br>6.<br>Specific tool<br>7.<br>Assessment<br>8.<br>Curricular update<br>9.<br>General pedagogy<br>10. Theory|
|14|Is the program for all teachers or just for certain teachers?<br>1.<br>All teachers<br>2.<br>Certain teachers|

44
|15|If the program is just for certain teachers, on what characteristics is it targeted?<br>_(Select all that apply.)_<br>1.<br>Geography<br>2.<br>Subject<br>3.<br>Grade<br>4.<br>Teachers' years of experience<br>5.<br>Teachers' skill gaps<br>6.<br>Uncertified Teachers<br>7.<br>Contract teachers|
|---|---|
|16|Which grades?<br>_(Select all that apply.)_<br> <br>|
||1.<br>Pre-primary<br>2.<br>Grade 1<br>3.<br>Grade 2<br> <br>8.<br>Grade 7<br>9.<br>Grade 8<br>10. Grade 9<br>|
||4.<br>Grade 3<br>11. Grade 10|
||5.<br>Grade 4<br>12. Grade 11|
||6.<br>Grade 5<br>13. Grade 12|
||7.<br>Grade 6|
|17|Are teachers assigned to participate or do they volunteer for the program?|
||1.<br>Assigned<br>2.<br>Volunteer<br>3.<br>A mix of both|
|18|How much do teachers have to pay to register for the program (if anything) per year?|
||A. Amount ______________ B. Noted in what currency?|
|19|Which of the following other costs do the teachers have to pay to participate in the program?<br>_(Select all that apply.)_|
||1.<br>None|
||2.<br>Transport|
||<br>3.<br>Accommodation|
||4.<br>Materials<br><br>Oh|
||_5._<br>ter|
|20|How much do teachers receive as per diem or payment to participate in the program per year?<br>A. Amount ______________ B. Noted in what currency?|
|21|What is the total cost of the program per year?|
||A. Amount ______________ B. Noted in what currency?|

45
|22|Does<br>_(Selec_<br>1.<br>2.<br>3.<br>4.<br>5.|participation in the training program have any<br>_t all that apply.)_<br>No<br>Status<br>Promotion or points towards promotion<br>Salary<br>Official certification|professional implications for teachers?|
|---|---|---|---|
|23|Are th|e teachers evaluated at the end of the training|?|
||1.<br>2.|No<br>Yes||
|24|Is it po<br>1.<br>2.|ssible for teachers to fail this exam?<br>No<br>Yes||
|25|If so,|what percentage of teachers fail the exam?||
|26|Is ther<br>_(Selec_|e a positive consequence if teachers are well<br>_t all that apply.)_|evaluated?|
||1.<br>2.<br>3.<br>4.<br>5.|No<br>Status<br>Promotion or points towards promotion<br>Salary<br>Official certification||
|27|Is ther<br>_(Selec_|e a negative consequence if teachers are poo<br>_t all that apply.)_|rly evaluated?|
||1.<br>2.<br>3.<br>4.<br>5.|No<br>Status<br>Promotion or points towards promotion<br>Salary<br>Official certification||
|28|Which<br>_(Selec_<br>1.<br>2.<br>3.<br>4.|of the following are informed about the teach<br>_t all that apply.)_<br>None<br>Teacher<br>School where the teacher teaches<br>Ministry of Education|ers’ performance on the training evaluation?|

46
|29|What materials, if any, did the program provide alongside the training?<br>_(Select all that apply.)_|
|---|---|
||1.<br>No materials<br>2.<br>Textbooks<br>3.<br>Storybooks or reading pamphlets<br>4.<br>Flashcards<br>5.<br>Word banks<br>6.<br>Computers<br>7.<br>Software<br>8.<br>Teacher manuals<br>9.<br>Lesson plans/videos<br>10. Scripted materials<br>11. Craft materials|
|30|How many teachers received training under this program in the last year that the program was<br>implemented?|
|31|In the last year that the program was implemented, what percentage of the teachers who began<br>the training dropped out before the end?|
|32|In how many schools is the program currently being implemented?|
|33|Has this program been evaluated in terms of its impact?|
||1.<br>No<br>2.<br>Yes|
|34|If so, on which of the following was it evaluated in terms of impact?_(Select all that apply.)_|
||1.<br>Teacher knowledge<br>2.<br>Teacher behavior<br>3.<br>Student learning<br>4.<br>Objectives of the program|
|35|Over the course of the program, what data are collected centrally?_(Select all that apply.)_<br>1.<br>Frequency of class delivery<br>2.<br>Attendance of participating teachers<br>3.<br>Teachers' assessment of value of training<br>4.<br>Test score of teacher subject knowledge<br>5.<br>Test score of teacher pedagogical knowledge<br>6.<br>Practical test observing teaching|

47
|**Co**<br>36|**ntent**<br>Which|of these is the primary focus of the|training program?|
|---|---|---|---|
||1.<br>2.<br>3.<br>4.<br>5.<br>6.<br>7.<br>8.<br>9.<br>10.|Subject content<br>Subject-specific pedagogy<br>Technology<br>Counseling<br>Classroom management<br>Specific tool<br>Assessment<br>Curricular update<br>General pedagogy<br> Theory||
|37|Which|of these is the secondary focus of t|he training program?|
||1.<br>2.<br>3.<br>4.<br>5.<br>6.<br>7.<br>8.<br>9.<br>10. <br>11.|No other focus<br>Subject content<br>Subject-specific pedagogy<br>Technology<br>Counseling<br>Classroom management<br>Specific tool<br>Assessment<br>Curricular update<br> General pedagogy<br> Theory||
|38|What i<br>1.<br>2.<br>3.<br>4.<br>5.<br>6.<br>7.|s the subject focus of the training p<br>None<br>Literacy or language<br>Math<br>Natural science<br>Social science<br>Information technology<br>Other|rogram (if any)?_(Select all that apply.)_|
|39|Does t<br>1.|his program provide training in-pers<br>In-person|on and/or online?<br>**_SKIP TO QUESTION 48_**|
||2.|Online||
||3.|Both||

48
|**Onl**<br>**_SKI_**<br>|**ine programs**<br>**_P THIS SECTION FOR PROGRAMS WITH NO ONLINE COMPONENTS_**<br>|
|---|---|
|40|In total how many hours of training are provided under this program?|
|41|Whatproportion of this trainingdo teachers spendpracticingwith other teachers?|
|42|What proportion of this training do teachers spend practicing with students?|
|43|Over how many weeks is this training spread?|
|44|Do teachers have any contact with a trainer online, as part of the program?<br>1.<br>No<br>2.<br>Yes|
|45|If so, is the contact with trainers individual, in groups, or both?<br>1.<br>Individual<br>2.<br>Group<br>3.<br>Both|
|46|Are the online group sessions compulsory or voluntary?<br>1.<br>Compulsory<br>2.<br>Voluntary|
|47|In total, how many hours of online contact do teachers have with a trainer under the program?|

49
|**Del**<br>48|**ivery I**<br>What<br>1.<br>2.<br>3.<br>4.<br>5.<br>6.<br>7.|are the core activities involved in the training?_(Select all that apply.)_<br>Lectures<br>Discussion<br>Teaching practice<br>Discussion of videos<br>Practice in science labs<br>Practice with computers<br>Other practical activities|
|---|---|---|
|49|Which<br>_apply.)_<br>1.<br>2.<br>3.<br>4.<br>5.<br>6.|of the following additional activities were included in the training, if any?_(Select all that_<br><br>None<br>Development of pedagogical materials<br>Development of classroom evaluation<br>materials<br>Training on how to conduct diagnostics<br>Lesson planning<br>Using scripted lessons|
|50|Does t<br>teache<br>1.<br>2.|he program use a cascade training model (i.e., program trains trainers who then train<br>rs)?<br>No<br>Yes|
|51|What i<br>contac|s the most common profile of the trainers or facilitators who the teachers have direct<br>t with?|
||1.<br>2.<br>3.<br>4.<br>5.<br>6.<br>7.<br>8.|Pre-primary, Primary or secondary<br>teacher in the subject of the training<br>Specially selected expert pre-primary,<br>primary or secondary teacher<br>Other pre-primary, primary or<br>secondary teacher<br>University professor or graduate degree<br>in education<br>Researcher<br>Government official<br>University student in education<br>Other|

50
|52|What,<br>contac|if any, training or certification did the trainers or facilitators who the teachers have direct<br>t with receive?_(Select all that apply.)_|
|---|---|---|
||1.<br>2.<br>3.<br>4.<br>5.|None<br>Designed the program<br>Received a specific certification<br>Received one week or less of training<br>Received more than one week of<br>training|
|53|Outsid<br>trainer<br>1.<br>2.<br>3.<br>4.<br>5.<br>6.|e of their normal salary, what kind of engagement mechanisms or incentives are given to<br>s?_(Select all that apply.)_<br>None<br>Performance related bonus<br>Tablet or computer<br>Books<br>Community recognition<br>Other|
|54|In tota<br>year?|l, how many hours of homework are teachers expected to do as part of the training, per|
|55|Over h|ow many weeks is this homework spread?|
|56|Which|of these types of follow-up support do teachers receive?_(Select all that apply.)_|
||1.<br>2.<br>3.<br>4.<br>5.|Text messages<br>Phone calls<br>Emails<br>In-school support from principals<br>In-school support from other school<br>staff|
|57|Over h|ow many weeks is this follow-up support spread?|
|58|Does t|he program provide any face-to-face training?|
||1.|No<br>**_SKIP TO QUESTION 71_**|
||2.|Yes|

51
**Delivery II** **_SKIP THIS SECTION FOR PROGRAMS WITH NO FACE-TO-FACE COMPONENTS (I.E. ONLINE ONLY)_**

|59|How many days do teachers work face-to-face with trainers or facilitators in this program?|
|---|---|
|60|Over how many weeks is this face-to-face training spread?|
|61|Approximately what proportion of this time is spent in lectures and discussion?|
|62|Approximately what proportion of this time is spent practicing teaching with students?|
|63|Approximately what proportion of this time is spent practicing teaching with other teachers?|
|64|Approximately what proportion of this time is spent in other practical activities with other<br>teachers?|
|65|Where does the majority of the face-to-face training take place?<br>1.<br>School of teacher being trained<br>2.<br>Central location (other school, hotel,<br>government building etc.)<br>3.<br>University or training center|
|66|On average, about how many teachers are there per trainer or facilitator in each training<br>session?|
|67|How many in-school follow-up support visits do teachers receive after the initial training (if<br>any)?|
|68|What is the nature of these follow-up visits?_(Select all that apply.)_<br>1.<br>In-class pedagogical support<br>2.<br>Monitoring<br>3.<br>Review material|
|69|Over how many weeks are the follow-up visits spread?|

52
- 70 How many times do teachers receive any of the above types of support? _(Count each text message/phone call/conversation as one time.)_

# **Delivery III**

- 71 Were there any elements of the program that the teachers particularly _liked_ ? Element 1 Element 2 Element 3

- 72 Were there any elements of the program that the teachers particularly _disliked_ ? Element 1 Element 2 Element 3

- 73 What were the key elements you think made the program work? Element 1 Element 2 Element 3

53
# **Appendix C: Mathematical Appendix**

For all estimates included in the meta-analysis, the goal is to estimate the standardized effect size, again drawing on Borenstein et al. (2009): using some estimate of the raw mean difference between treatment and control groups, _D_ , as well as its combined standard deviation for treatment and control groups, _Spooled_ . All studies report _D_ directly, however, _Spooled_ is commonly not reported. Almost all studies we review instead report the standard error of _D, SED._ Where this is the case, if we assume that the standard deviations of the two groups are the same, then the variance of D is: where _n1_ and _n2_ are the sample sizes in the two groups. The standard error of _D_ is then the square root of _V_ .

Combining Equation A1 and Equation A2, we derive our equation for _Spooled_ , the within-groups standard deviation, pooled across treatment and control groups: 54
We can then divide _D_ by _Spooled_ to calculate standardized effect sizes, _d_ , for all estimates.

55
# **Appendix D: Complete Regression Results**

**Table C1. Overarching Aspects – complete regression results**

|Overarching Aspects variable|Program impact<br>on student<br>learning|
|---|---|
|Program provides textbooks|0.355**<br>|
||(0.128)|
|Program provides other reading materials (flashcards, word banks,<br>primers)|0.159*<br>|
||(0.087)|
|Participation has implications for promotion or points towards promotion<br>or salary implications|0.143**<br>(0.066)|
|Targeting by years of experience|0.136<br>(0.198)|
|Program provides storybooks|0.129|
||(0.094)|
|Number of schools in which the program was implemented|0.120<br>(0.083)|
|Program designed by Government|0.100<br>(0.130)|
|Participation has implications for status only|0.095<br>(0.101)|
|Teachers dropped out of program in last year|0.088<br>(0.096)|
|Teachers not evaluated|-0.084|
||(0.084)|
|Program provides craft materials|-0.084|
||(0.126)|
|Positive consequence if teachers are well evaluated|0.080<br>(0.100)|
|Program provides materials|0.078<br>(0.109)|
|Program designed by NGO or social enterprise|0.065<br>(0.086)|
|Number of teachers receiving training under this program in the last year|0.065<br>(0.088)|
|Number of years the program been running|0.063|
||(0.119)|
|Negative consequence if teachers are poorly evaluated|0.054|
||(0.117)|
|Targeting by grade|-0.050|
||(0.098)|

56
|Overarching Aspects variable|Program impact<br>on student<br>learning|
|---|---|
|Program implemented by NGO orsocialenterprise|0.039|
||(0.094)|
|Program provides computers|0.035<br>(0.127)|
|Targeting by subject|-0.034|
||(0.099)|
|Program provides teacher manuals|-0.034<br>(0.090)|
|Targeting by contract teachers|0.027<br>(0.121)|
|Program provides scripted lessons|0.020<br>(0.151)|
|Program implemented by Government|-0.017|
||(0.107)|
|Program provides lesson plans/videos|-0.017<br>(0.091)|
|Program design based on informal diagnostic|-0.012<br>(0.134)|
|Program provides software|0.011<br>(0.096)|
|Program design not based on any kind of diagnostics|0.006<br>(0.151)|
|Targeting by geography|-0.001<br>(0.082)|
|Observations|26|
|Standard errors inparentheses. ***p<0.01,**p<0.05,*p<0.1.||

57
**Table C2. Content – complete regression results**

|Content variable|Program impact on<br>student learning|
|---|---|
|Primary focus of the training program is classroom<br>||
|management|0.471|
||(0.272)|
|No subject focus of training|-0.243|
||(0.204)|
|Secondary focus of the training program is subject content|0.182<br>(0.156)|
|Primary focus of the training program is new technology|0.180<br>(0.206)|
|Primary focus of the training program is pedagogy|0.177<br>(0.201)|
|Secondary focus of the training program is pedagogy|0.123<br>(0.178)|
|Training involves lesson enactment|0.110|
||(0.072)|
|Training involves lesson planning|0.096<br>0102|
||(.)|
|Primary focus of the training program is counseling|-0.089<br>(0.235)|
|Training involves training on how to conduct diagnostic|0.083<br>(0.074)|
|Primary focus of the training program is subject content|0.083<br>(0.235)|
|Subject focus of the training program is science|-0.063<br>(0.250)|
|Subject focus of the training program is information||
|technology|0.062|
||(0.250)|
|Training involves the use of scripted lessons|0.052<br>(0.108)|
|Subject focus on literacy or language|0.029|
||(0.155)|
|Subject focus of the training program is math|-0.028<br>(0.186)|
|Training involves lectures|0.020|
||(0.152)|
|Training involves materials development|0.012|
||(0.087)|
|Training involves discussion|0.004|
||(0.082)|
|Observations|26|

58
Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1.

# **Table C3. Delivery – complete regression results**

|Delivery variable|Program impact on<br>student learning|
|---|---|
|Training takes place in university or training center|0.385**<br>(0.142)|
|Follow-up visits to review material|0.256<br>|
||(0.156)|
|Most common profile of the direct trainers is researchers|-0.196<br>(0.336)|
|Most common profile of the direct trainers is local government officials|-0.170<br>(0.257)|
|Proportion of training spent practicing with other teachers|0.169<br>(0.134)|
|Includes consecutive days of face-to-face training|0.165<br>(0.148)|
|Proportion of training spent in lectures|-0.156<br>(0.094)|
|Includes follow-up visits|0.146|
||(0.122)|
|Most common profile of the direct trainers is university professors or<br>graduate degrees in education|-0.146<br>(0.257)|
|Follow-up visits for in-class pedagogical support|0.144<br>(0.102)|
|Months over which distance learning is spread|-0.140<br>(0.085)|
|Most common profile of the direct trainers is primary or secondary<br>teachers|-0.094|
||(0.274)|
|Includes distance learning|-0.066|
||(0.092)|
|Total hours of face-to-face training|0.058<br>(0.083)|
|Cascade training model|-0.039|
||(0.100)|
|Proportion of training spent practicing with students|0.033<br>(0.093)|
|Weeks over which face-to-face training is spread|-0.025<br>(0.085)|
|Training takes place in-school|0.004|
||(0.194)|

59
26 <u>Observations</u> Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1.

60