metadata: authors: 0: author: Fuje,Habtamu Neda
1: author: Tandon,Prateek
count: Mongolia
docna: 0: docna: When do in-service teacher training and
            books improve student achievement ? experimental evidence
            from Mongolia
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Gender and Development,Effective Schools and Teachers
teratopic: Education,Gender
abstracts: cdata!: This study presents evidence from a
            randomized control trial (RCT) in Mongolia on the impact of
            in-service teacher training and books, both as separate
            educational inputs and as a package. The study tests for the
            complementarity of inputs and non-linearity of returns from
            investment in education as measured by students' test
            scores in five subjects. It takes advantage of a
            national-scale RCT conducted under the Rural Education and
            Development project. The results suggest that the provision
            of books, in addition to teacher training, raises student
            achievement substantially. However, teacher training and
            books weakly improve test scores when provided individually.
            Students whose teachers have received training and whose
            classrooms have acquired books improved their cumulative
            score (totaled across five tests) by 34.9 percent of a
            standard deviation, relative to a control group. Students
            treated only with books improved their total score by 20.6
            percent of a standard deviation relative to a control group
            of students. On the other hand, extra teacher training did
            not have a statistically significant effect on the total
            test score. In addition, providing both inputs jointly
            improved test scores in most subjects, which was not the
            case when either input was provided individually. This study
            sheds light on the relevance of supplementing teacher
            training schemes with appropriate teaching materials in
            resource-poor settings. The policy implication is that
            isolated education investments, in settings where
            complementary inputs are missing, could deliver minimal or
            no return.
geo_reg_key: 517198,119225,517191
repnme: repnme: When do in-service teacher training and
            books improve student achievement ? experimental evidence
            from Mongolia
display_title: When do in-service teacher training
            and books improve student achievement experimental evidence
            from Mongolia
listing_relative_url: /research/2015/11/25257500/in-service-teacher-training-books-improve-student-achievement-experimental-evidence-mongolia
topicv3: Teachers Management m1326119 1523,Education, Skills Development and Labor
            Market m1326135 1484,Teacher Training m1326786 1532,Education m1327048 470,Gender m1327621 615
publishtoextweb_dt: 2015-11-09T00:00:00Z
historic_topic: Education,Gender
no_of_pages: 41
author_keys: 0: author_key: 1051824
1: author_key: 969324
model_extractions: WPS7485 Policy Research Working Paper 7485 When Do In-service Teacher Training and Books Improve Student Achievement? Experimental Evidence from Mongolia _Habtamu Fuje Prateek Tandon_ Education Global Practice Group November 2015
Policy Research Working Paper 7485

# **Abstract**

This study presents evidence from a randomized control trial (RCT) in Mongolia on the impact of in-service teacher training and books, both as separate educational inputs and as a package. The study tests for the complementarity of inputs and non-linearity of returns from investment in education as measured by students’ test scores in five subjects. It takes advantage of a national-scale RCT conducted under the Rural Education and Development project. The results suggest that the provision of books, in addition to teacher training, raises student achievement substantially. However, teacher training and books weakly improve test scores when provided individually. Students whose teachers have received training and whose classrooms have acquired books improved their cumulative score (totaled across five tests) by 34.9 percent of a standard deviation, relative to a control group. Students treated only with books improved their total score by 20.6 percent of a standard deviation relative to a control group of students. On the other hand, extra teacher training did not have a statistically significant effect on the total test score. In addition, providing both inputs jointly improved test scores in most subjects, which was not the case when either input was provided individually. This study sheds light on the relevance of supplementing teacher training schemes with appropriate teaching materials in resource-poor settings. The policy implication is that isolated education investments, in settings where complementary inputs are missing, could deliver minimal or no return.

This paper is a product of the Education Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at hfuje@ worldbank.org and ptandon@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **When Do In-service Teacher Training and Books Improve Student Achievement? Experimental Evidence from Mongolia**

Habtamu Fuje<sup>*</sup> Columbia University Prateek Tandon The World Bank

**Keywords** : In-service teacher training, RCT, matching, impact **JEL Classification** : I28, I21, O15

> * Corresponding author: <u>Habtamu.Fuje@columbia.edu or Habtamu_Fuje@post.harvard.edu</u> The study also benefited from discussion with faculty and graduate students at Columbia University. Yabibal Walle, University of Göttingen, Andinet Woldemichael, Georgia State University, and Kefyalew Endale, National Graduate Institute for Policy Studies, proofread the draft version. Charles Abelmann, Cristobal Ridao-Cano, and Katherine Nesmith of the World Bank originally designed the evaluation. D. Khishigbuyan, Project Coordinator of READ, provided assistance throughout project implementation and follow up. Deon P. Filmer and David Evans, from the World Bank, graciously provided invaluable comments and suggestions.  We thank you all.
# 1 INTRODUCTION

Policy makers and practitioners in developing and developed countries often invest heavily in brief in-service teacher training to enhance education outcomes. Spurred by the targets of the Millennium Development Goals (MDGs), developing countries have also rapidly expanded school infrastructure in the past decade and ramped up in-service teacher training. These investments have aimed to satisfy the growing demand for teachers and help improve educational quality (GOM, 2007; Bunyi et al., 2013; Kidwai et al., 2013). However, conclusive evidence on the impact of in-service teacher training on student achievementas measured by a comprehensive set of test scoresscarcely exists, particularly in developing countries. Moreover, the dierential impact of such training on achievement when students and teachers have access to appropriate books to eectively implement the lessons learned during trainingversus when they do not has not been investigated. Previous studies have focused on the individual provision of either teacher training or books and have not examined a potential complementarity between these inputs.

Properly documenting the impact of such investments on student outcomes can address this gap. The few rigorous evaluations of teacher training programs conducted to date suggest a moderate potential to improve student outcomes, but the evidence is mixed. A recent systematic review by Glewwe et al. (2013), which examined impact evaluation studies from 1990-2010, concluded that there is only modest evidence that teacher training improves student test scores. Specically, 11 of the 29 estimates included in their analysis demonstrate positive, signicant impacts (one is signicant and negative). But, only three of these studies were well identied, experimental or based on natural experiments. Other works on the impacts of teacher training also do not provide conclusive positive evidence: improvements in test scores were documented by some (see Jacob and Lefgren (2004); Zhang et al., 2013; Raudenbush et al., 1993), while others nd no evidence (see Angrist and Lavy, 2001; Harris and Sass, 2011; and Lai et al., 2011). Evans and Popova (2014) noted that the type of teacher training also matters; a one-time in-service training might be as eective as long-term peer mentoring/coaching.

With regards to the impacts of books, the same review by Glewwe et al. (2013) revealed that, in general, there is strong, but non-unanimous, evidence for the positive 1
impact of textbooks and workbooks on student learning. However, when considering well identied studies only, they noted weak evidence. Older studies suggest that books improve achievement (Heyneman et al., 1984; Jamison et al., 1981), while more recent studies in Kenya (Glewwe et al., 1998 and Glewwe et al., 2009) and in Sierra Leone (Sabarwal et al., 2014) contradict these ndings.

Most of these previous studies, however, have had some methodological limitations. The most serious methodological issue with observational studies is the non-random assignment of teachers to in-service training programs or students to book provision. A few quasi-experimental studies have attempted to address these issues (Rothstein, 2010; Jacob and Lefgren, 2004; Angrist and Lavy, 2001). A number of issues arising from non-random assignment need to be addressed. For instance, factors like self-initiation, relationships with supervisors, personal connections and political participation confound with a teacher's decision to attend in-service training as well as her general motivation and capacity to teach (see Jacob and Lefgren (2004)). Similarly, a student's access to books confound with a number of other covariates such as parental education, wealth, and school resources, which directly aect student outcomes.

This study uses data from the randomized assignments of teachers into a training program or the provision of books to randomly selected primary schools in Mongolia under the Rural Education and Development (READ) project to examine the impacts of these interventions on student achievement. The randomization is nationally representativeit covers the entire rural population of the whole country, as opposed to a typical small-scale randomization study from which generalization to national population is not feasible. This enables us to address limitations arising from nonrandom assignment and provides a basis to generalize about the impact of the interventions.

In addition, this study investigates the dierential impact of in-service teacher training or book provision as a stand-alone intervention vis-à-vis in-service training accompanied by provision of age-appropriate books. Some previous evidence on the topic suggests that provision of education inputs as a bundle is more eective in improving outcomes (see McEwan (2014); Evans and Popova (2014); and Conn (2014) for detailed review). The evaluation of these interrelated investments sheds light on the potential complementarity of educational inputs, and non-linearity of returns to 2
education investment by comparing returns to provision of books or teacher training alone against returns from training teachers along with the provision of books. This addresses the question of whether the sum of returns from extra teacher training" and books only" interventions is lower or higher than the return from training complemented by books. If the sum of returns from the individual interventions is lower than return from the joint investment, then evidence for complementarity of books and training in education production exists.

We ask two questions that have fundamental policy relevance: (1) Do short in-service teacher training and books improve students' test scores when provided individually in a resource-poor setting? (2) How does the return from the joint provision of these inputs compare with sum of returns from providing each input individually? We nd signicant, positive eects on student outcomes when books and training were provided together as a package, rather than as individual inputs. Books only and extra teacher training marginally improved test scores in some, but not all, subjects. The magnitude of impact of either input was not academically signicant. However, when teachers are trained and students are provided with books, the test scores of a treatment group of students increased substantially, relative to a control group of students.

The rest of the paper is organized as follows. Section 2 presents a brief description of the context, and detailed discussion of the survey design, instruments and interventions. Section 3 outlines the framework and identication strategies employed. Section 4 presents descriptive and analytic results, an investigation of heterogeneity in treatment eects, and robustness checks. A discussion of results is provided in Section 5.

# 2 CONTEXT, SURVEY DESIGN AND INTERVENTION

# 2.1 Context

The Ministry of Education, Culture and Science (MECS) developed a new Education Sector Master Plan (ESMP2) for 2006-2015 that built on the General Guideline for Socio-Economic Development of Mongolia (GGSEDM) for 2006-2008. The GGSEDM identied ve priority actions for education: (1) reduce school dropout and provide elementary education for all; (2) transform the education system into an 11-year 3
system by 2006 and then into a 12-year system by 2007; (3) improve the learning environment, physical facilities supply of teachers and textbooks at secondary schools ; (4) lower gender inequality in primary and secondary school enrollment; and (5) increase accessibility of schools for children with disabilities. The ESMP2 sought to sequence the government priorities by: (1) upgrading education quality at all levels of schooling; (2) providing education services to children in all parts of the country, including rural areas, and to the poor and vulnerable groups; and (3) improving the management capacity of central and local educational institutions. The government acknowledged that low levels of educational attainment were key determinants of poverty, and that poverty could be a key factor that limited access to and quality of schooling. These eorts were in response to the dramatic decline in support for the country's education system after its transition to a free market economy in the early 1990s. Enrollment in rural schools declined rapidly, and access to high-quality learning materials diminished. Schools in rural areas had few textbooks and little or no supplementary reading books (World Bank, 2013).

# 2.2 Intervention and design

To improve the quality of primary education in rural Mongolia, MECS, with technical and nancial support from the World Bank, implemented a comprehensive rural education program, the READ project. READ's main policy instruments were availing high-quality children's books and improving teachers' skills through in-service training schemes. Under this project, primary schools received grade-specic classroom libraries, which entailed equipping classrooms with grade-appropriate books and shelves for these books. These books were used during class hours, and students were also occasionally allowed to borrow them for use at home. Each classroom received about 160 books. These education materials were provided at a very low cost. The average costs (in 2008 US$) of a single book and a set of shelves were $2.1 and $71.5, respectively (World Bank, 2013).

Primary school teachers participated in an intensive training to improve their skills to support students in math, reading and writing activities. The training was rolled out in a cascade model: the training of the trainer-teachers was implemented rst. Afterwards, these trainers trained fellow teachers on how to improve their students' 4
math, reading and writing skills. About 178 mentor/trainer teachers were trained for four days by well qualied national trainers, and then they conducted an average of 2.26 visits per school to mentor fellow teachers. The training of fellow teachers lasted for three days. This cascading of training enabled the delivery of teacher training in a more cost eective manner than other teacher training projects. The training cost was $ 3.14 per day per teacher under READ, relative to $ 7.62 for other similar training schemes in the country (World Bank, 2013).

To evaluate the impact of teacher training or books alone as well as teacher training complemented by books, a national-scale randomization was carried out. The initial design of the evaluation strategy was such that schools in the 21 provinces/aimags would be randomly assigned to Treatment One ( _T_ 1 ), Treatment Two ( _T_ 2 ) or a control group ( _C_ ) (see Figure 1). The control group was later divided into two: Control One ( _C_ 1 ) and Control Two ( _C_ 2 ).<sup>1</sup><sup>_,_2</sup> _T_ 1 includes primary schools randomly selected in ve provinces (Arkhangai, Bulgan, Zavkhan, Sukhbaatar and Tov), and these schools received classroom libraries and in-service teacher training in May 2007. _T_ 2 , schools in Ovurkhangai and Govi-Altai provinces, were provided classroom libraries, but not teacher training, in May 2007. _C_ 1 , which includes schools in Dornogovi, Omnogovi, Uvs, Khovd, Khovsgol, Khentii and Govisumber provinces, was originally to receive classroom libraries and teacher training at the end of the experiment (in May 2008), but the plan was changed later and it received books in October 2007 and training between October 2007 and March 2008. _C_ 2 encompasses schools in Bayan-Olgii, Bayankhongor, Dornod, Dondgovi, Selenge, Darkhan-Uul and Orkhon provinces, and these schools received treatment at the end of the experiment (books in May 2008 and training between May and September 2008). Figure A.1 (see annex) shows aimags in which the four groups of schools are located, and Table 1 presents the timeframe of the survey and interventions, and the number of schools and students surveyed.

> 1 _C_ 1 received treatment halfway through the study period. Therefore, direct comparison of _T_ 1 and _T_ 2 with _C_ will not be feasible. In addition, the `pure' control group ( _C_ 2 ) has smaller sample size. Hence, the follow up survey included additional schools in the sample.

> 2Administratively, Mongolia is divided into 21 aimags and the capital city, Ulaanbaatar. These aimags are further divided into soums, and then into bags (NSO, 2006).

5
Figure 1: Assignment to treatment and control groups <!-- Start of picture text --> Treatment assignment<br>Treatment Control<br>Treatment 1 Treatment 2 Control 1 Control 2<br>Books & Books No intervention Training and<br>Training (in May 2007) before May Books (in Oct<br>2008 2007-Mar 2008)<br>(in May 2007)<br><!-- End of picture text --> To reduce spillover eects and ensure the political feasibility of providing schools with dierent inputs, a given province was allowed to have either treatment or control schools. Then, in these selected schools, a class from two grades (specically, from third and fourth grade) was randomly selected and surveyed.<sup>3</sup><sup>_,_4</sup> In the upcoming sections, we discuss the limitation of conning treatment and control schools in selected provinces, instead of allowing each province to have both types of schools, and we cluster standard error at province level to correct for this limitation. Finally, students within a class were randomly selected if the class size was more than 20; otherwise the whole class was surveyed.

The baseline survey was conducted during April-May 2007, just before the end of the academic year, and it encompassed 137 schools, 141 teachers and 2,612 students. A follow-up survey was conducted in April 2008. In the follow-up survey, additional schools, classes, teachers and students were surveyed to address initial imbalances in

> 3The classes were sorted alphabetically, and if there were at least 20 students in the rst class of each grade, it was selected as a sample. Otherwise, next class with at least 20 students was selected. If such class did not exist, the class with the highest number of students was selected.

> 4In 2004, Mongolia has began a transiting from a ten-grade education system, with four primary school grades, to a twelve-grade system (Yang and Sato, 2009).

6
the number of observations in the treatment and control groups during the baseline survey. It covered 172 schools, 311 classes, 308 teachers and 5,322 students (see Table 1). The follow-up survey covered all students and teachers who were surveyed in the baseline, but also included additional teachers and students. The cause of imbalance and how this additional observation is leveraged to address the imbalances is discussed under the `identication strategy' subsection.

Table 1: Treatment assignment, timeframe and number of schools and students in each arm

||Treatment 1|Treatment 2|Control 1|Control 2|
|---|---|---|---|---|
|April-May 2007||Baseline|||
|May-2007|Books & Training|Books|-|-|
|Oct-2007|-|-|Books|-|
|October 2007-March 2008|-|-|Training|-|
|Number of Schools and Students|||||
|Schools|50|41|26|20|
|Students|946|784|505|377|
|Apr-2008||Endline|||
|May-2008|-|-|-|Books|
|May-Sept 2008|-|-|-|Training|
|Number of Schools and Students|||||
|Schools|48|41|49|34|
|Students|1665|1432|1326|899|

# 2.3 Instrument

The data collection required a signicant number of survey sta. For the baseline survey, 32 people were deployed. Each survey team included three people (a team leader and two enumerators), who spent a full day in each school implementing the survey instrument (MEC and LRCM, 2008). The survey sta used measures to ensure that assessment items were appropriately translated, used transparently documented assessment procedures, including quality control procedures, and availed procedures to ensure that assessments were implemented in a standardized manner across all participating schools.

The survey instrument encompasses two sets of questionnaires: the rst regarding 7
students and the second about teachers, classrooms and schools. Under the rst instrument, students were tested in language (reading, writing and listening), numeracy skills (math), and scholastic and verbal aptitude (Peabody) using test instruments adopted from international testing standards and piloted by a team of international and local researchers. Under the second questionnaire, observation sheets for schools, classrooms and teachers were completed to collect information on school resources, classroom conditions, and teacher qualications.

As mentioned above, ve assessments were administered: a Peabody vocabulary test adapted to the Mongolian context; a mathematics assessment, based on questions from the Trends in International Mathematics and Science Study (TIMSS); and listening, reading, and writing assessments based on the Mongolian curriculum. Validation measures for the mathematics and Peabody tests were carried out under the READ project. Prior to the mathematics assessment, an investigation of construct equivalence with Grade 4 TIMSS items was undertaken. A panel of Mongolian math experts, MECS sta and an international technical assistant reviewed the TIMSS 2003 mathematics items and identied items that were suitable for Mongolia. The panel used testcurriculum matching analysis to evaluate the degree of congruence between the international mathematics assessment and the Mongolian national curriculum. Since an item might have been in the curriculum for some but not all students in the country, an item was determined appropriate if it was in the intended curriculum for more than 50 percent of the students (World Bank, 2006).

The Peabody test administered was a norm-referenced instrument for measuring the listening vocabulary of children. For each item, the assessor would say a word, and the student responded by selecting the picture that best illustrates that word's meaning. Items were reviewed by the MECS panel to ensure they were appropriate for the Mongolian curriculum. The mathematics, reading, and writing tests used a balanced incomplete block design, with dierent item content across dierent test booklets. Dierent test booklets were then randomly assigned to to dierent students. Items were grouped into blocks, and each block was repeated in more than one test booklet to ensure balance across test booklets (World Bank, 2006).

An international assessment expert hired by the project used construct equivalence analysis to conrm that the assessments measured the same constructs between boys 8
and girls, and the assessment frameworks applied to both genders

# 3 FRAMEWORK AND IDENTIFICATION STRATEGY

# 3.1 Conceptual Framework

Comprehensive frameworks for linking student achievement to any single education input remain elusive. For instance, the impact of an intervention that provides books to students in the third grade is a dynamic function of current and past covariates, including qualications of current teachers, family's socioeconomic status and school attributes, as well as historical records prior to the current year (pre-school to second grade) of these covariates, and the student's performance in the previous grades. Capturing these dynamic relationships using a static framework and lacking historical data on relevant covariates makes empirical estimation of an input's impact challenging.

Moreover, the impact of an education investment, say teacher training, on a student's performance depends on the availability of other complementary inputs, like appropriate books. Whether increases in such inputs, say through in-service teacher training, matter for student outcomes is an area of ongoing research and limited clarity (Hanushek, 2004; Hanushek and Rivkin, 2010; Todd and Wolpin, 2003). The potential non-linearity in education production also suggests that returns from packaged inputs could be substantially dierent from the sum of returns from applying these same inputs individually (Hanushek, 2004). The complementarity of educational inputs also suggests that an individual input would have dierent impacts on outcomes when it is provided alone versus when it is provided in conjunction with other inputs (Linden, 2008). This is particularly pertinent in resource-poor settings, where many complementary education inputs may be missing, and availing one without the other may provide little or no return from such investment.

Lacking relevant historical covariants, this study relies on a static model of education production and also allows for the possibility of testing the complementarity of inputs. This static econometric specication of an education production function entails representing the association between a student's classroom achievement (test scores) on the one hand, and current teacher's qualications (formal education, in-service training, experience, motivation etc.), student-specic characteristics (gender, age, appetite to 9
read and the like), her family's socioeconomic status (asset/income, education, housing conditions and so on) and school resources (school type, inputs, general hygiene, location, facilities etc.), on the other. Specically, student _i_<sup>_′_</sup> _s_ achievement in class _c_ of school _s_ ( _Yi,c,s_ )for the current study, scores in math, reading, writing, listening and Peabody testsis a function of student- and family-specic characteristics ( _Xi,c,s_ ); and classroom- and school-specic covariates ( _Rc,s_ ) and the qualications of her teacher, _j_ , ( _Qj,c,s_ ). In line with previous studies (Hanushek and Rivkin, 2010; Todd and Wolpin, 2003), by assuming a static relationship, we specify a model of education production function as: ...where _ε_ is an error term. _i,c,s_ The empirical challenge in identifying the causal impact of an educational input (or set of inputs) on student achievement is the non-randomness of input choices. For example, an in-service teacher training program that is intended to improve teachers' competence may not be attributable to a change in student test scores in a nonexperimental setting because of the non-random assignment of teachers to students and teacher training opportunities to teachers. Students from families with better socioeconomic status tend to get matched with better trained, motivated and wellpaid teachers, and hence teachers' qualications tend to confound with unobserved achievement determinants (Clotfelter et al., 2006). In addition, a teacher's access to in-service training may depend on her motivation and/or personal connection with education administrator or school director (Jacob and Lefgren, 2004). This non-randomness implies that _cov_ ( _Q, ε_ ) = 0 and _cov_ ( _R, ε_ ) = 0 leading to biases in observation-based studies. Therefore, devising a valid identication strategy to discern the impact of improvement in teacher qualications on test scores is important. The subsection below is devoted to discussing how the current research handles this identication challenge.

An in-service teacher training intervention presumably aects student achievement through its impact on teacher quality ( ∆ _Qj,c,s_ ). For an extensive discussion on how teacher training might improve quality by enhancing pedagogical skills as well as subject-matter understanding see Mullens et al. (1996). For instance, an experimental 10
evaluation of a teacher training scheme has also documented heterogeneous impacts on the teachers' own English test score, where only teachers with university degree beneted well from in-service training (Zhang et al., 2013). On the other hand, providing classroom-library materials could change the resources available in treated schools ( ∆ _Rc,s_ ). This is particularly the case in a resource-poor setting, such as Mongolia, where essential teaching aids such as textbooks and workbooks were lacking.

In the context of the current study, the interventions that could improve education productivity have been randomly assigned with the intention to increase test scores in the treatment group: in-service teacher training to improve teacher quality and/or classroom libraries to ease resource scarcity. A control group of students, on the other hand, have not been exposed to any treatment until the experiment was nalized. Analysis of the baseline survey data conrms that the `initial randomization' was properly done: there were no systematic dierences between test scores (and other covariates) of students in the treatment and control groups.<sup>5</sup>

# 3.2 Identication strategy and empirical approach

The treatment eects from the above three interventions are estimated as follows: (1) Training and Books: To identify the causal impact of providing books and in-service training as complementary education inputs, students in treatment group one ( _T_ 1 ) and control group that has not received any treatment ( _C_ 2 ) were matched, and mean achievement-gaps between students in these groups provided the estimated impact of the two interventions. (2) Extra Training: Identication of the impact of extra inservice teacher training, on top of books, is based on the comparison of the dierences in student achievement between (matched) treatment one ( _T_ 1 ), which received training and books, and treatment two ( _T_ 2 ), which received books (but not teacher training). It is important to note that this identication of the contribution of teacher training likely includes the returns from the joint provision of training and books as well as the contribution of each input plus any complementarity between them. In the context of rural Mongolia, where education inputs were lacking prior to READ interventions, it is most likely that education production function to exhibit increasing return for

> 5The `initial randomization' refers to the initial randomization undertaken before the control group was divided into two, following change in policy regarding the interventions.

11
addition inputs. As a result, when teacher training is added to books, the increase in student outcomes is likely to improve education at least as much the eect of teacher training provided as a stand-alone intervention. More concisely, we argue that in this resource constrained setting, the education production function is likely to exhibit increasing returns to scale. Therefore, the estimated treatment eect of training should be considered as the maximum possible contribution of providing a short training to teachers, without providing complementary books.<sup>6</sup> (3) Books only: Impact of the classroom libraries intervention is identied by comparing outcomes of (matched) _T_ 2 against _C_ 2 .

Due to the change in the intervention plan from the initial evaluation design halfway through the treatment period, specically the exposure of part of the control group ( _C_ 1 ) into unplanned treatment,<sup>7</sup> the application of the standard randomized control trial (RCT) estimation techniquethrough direct comparisons of dierences in mean outcomes between _T_ 1 and _T_ 2 on the one hand, and the remaining control group ( _C_ 2 ) on the otheris not feasible. Therefore, to ensure that the counterfactuals are properly set, and the treatment eect is consistently estimated, propensity score matching is used as an alternative identication strategy. More specically, this approach involves two steps: (1) estimating propensity scores (PS) by matching control with treatment group of students on relevant covariates, using endline survey data only; and (2) applying a regression of student outcomes on covariates using the matched data in step one, with PS serving as weighting factor and standard error clustered at aimag level.

In the rst step, we estimate PS. _P_ ( _Xi_ ) = _p_ ( _Ti_ = 1 _|Xi_ ) is the likelihood that student _i_ would be exposed to treatment ( _Ti_ ) conditional on covariates, _Xi_ (Rosenbaum and Rubin, 1983; Becker and Ichino, 2002)). As the number of students in treatment arm is larger than those in control group, in this step, some students that do not satisfy the matching criteria are excluded.<sup>8</sup> Specically, observations o-common support are

> 6The ideal scenario would be to have another treatment group of students whose teachers were provided training alone. The comparison of test scores of these students with a control group that has not received any treatment could have provided the impact of training only, which is anticipated to be lower than the treatment eect estimated using the above setting.

> 7Part of the control group was given the books and shelves ahead of schedule because of community demand.

> 8The typical application of propensity score matching method is when there are larger number of observations in the control group to be matched with fewer observations in the treatment group. In this case, we have many more treated than control students. Therefore, each student in control group

12
excluded, and for both groups students with PS in the top and bottom 1% are trimmed o.

In the second step, we use the matched dataset to estimate the treatment eects (of in-service teacher training and books as a package, extra in-service training on top of books, and books alone) by running the following regressions: _Training_ & _Books_ : _Yi,c,s_ = _α_ 0+ _α_ 1 _T_ 1 C_ 2 _j, c, s_ + _α_ 2 _Qj,c,s_ + _α_ 3 _Xi,c,s_ + _α_ 4 _Rc,s_ + _ei,c,s_ (2) ...where _ei,c,s_ , _εi,c,s_ and _ui,c,s_ are error terms. T1_C2, T1_T2 and T2_C2 are dummy variables indicating whether student _i_ is in one or the other group. For instance, T1_C2 is equal to one if she is in group T1 and zero if she is in group C2. Estimated ˆ ˆ ˆ coecients of the corresponding these dummies (i.e. _α_ 1 , _γ_ 1 and _ω_ 1 ) are the impacts of the respective intervention(s) on students' test scores in ve areas.

The empirical estimation of these equations is conducted by using PS as a probability weight. The standard errors are clustered at aimag levelallowing heteroskedasticity and within-cluster error correlationto account for the fact that each aimag has either treatment or control schools, which might create within-group dependence. In addition, there are few clusters in each group of interventions and hence the large sample property of cluster standard error might not be satised. Accordingly, we resort to "wild cluster bootstraping" for asymptotic renement (see Cameron et al. (2008)).

As a robustness check, we also combine all groups of studentsthose who received training and books (T1), books only (T2), and control group (C2)and estimate the follow equation: is matched with one student in treatment group, and logistic distribution is assumed.

13
...where T1 and T2 are dummies equal to one for groups that received training and books and books only (and zero otherwise), respectively. The coecients ( _δ_ 1 and _δ_ 2 ) are the corresponding impacts of each set of interventions. The impact of extra-training is estimated as the dierence between these coecients (i.e., _δ_ 1 _− δ_ 2 ), and test for statistical signicance of this dierence is conducted.

# 4 RESULTS

# 4.1 Descriptive results

In this section, we briey discuss the implementation of propensity score matching and discuss descriptive results. As described above, the control and the treatment group of students were matched using endline survey. Observations that happen to be o-common support were dropped, and the data is further trimmed by removing observations with probabilities in the top and bottom 1% for the corresponding group. The densities of propensity scores are resented in Figure A.2 (see annex). The matching results for the three groups of interventions are presented in Table 2 below, and Table A.1-A.2 (see annex). As Table 2, A.1 and A.2 show, there were statistically signicant mean dierences in some covariates before matching, and these systematic dierences have been addressed after matching (i.e. balancing is achieved). In other words, the factors that could attenuate or amplify the impact of the interventions, such as students' characteristics, their families' socioeconomic status, teachers' qualication and school resources, do not exhibit statistically signicant dierences between the treatment and control groups.

In addition, we present mean student outcomes after matching (both during baseline and endline surveys) in Table A.3 (see annex). The (matched) treatment and control groups, for all of the three interventions, do not exhibit systematic baseline dierences in outcome indicators. Below is a brief discussion for each intervention.

Training and Books: Before matching, half of the covariates that could potentially aect test scores had statistically signicant mean dierences between the control and the treatment group of students. The PSM has taken care of these dierences in these covariates. Teachers' qualications (formal training and years of experience) are similarthe majority of teachers have had formal education and about 14 years of 14
professional experience. Students' book ownership at home, the dierence in which could bias the impact of books provided at school, was similar for both groups during the baseline and follow-up. The same holds true for students' characteristics (age, gender, frequency of taking extra lessons per week, number of days per week in which the students had to accomplish household chores before and after work, distance from school, and residing with mother/father or others), and their families' socioeconomic status (whether both or either parent have completed high school education, residence type and ownership of telephone at home). The treatment and control schools also had similar characteristics in terms of the existence of infrastructure like toilet/handwashing facilities (Table 2 ).

A detailed description of the achievement dierence between treatment and control groups of students is presented in Table A.3 (see annex). The baseline achievement gap between treatment and control groups of students is not appreciable.<sup>9</sup> For instance, mean total score in the ve tests for students in the treatment and control groups are 24.8 and 25.3, respectively. Similar results holds true when tests are considered individually. During the follow-up survey, the mean of the total score for the treatment and control groups increased to 31.9 and 29.2, respectively. The scores in individual tests have also exhibited a similar widening gap between students in treatment and control groups.

> 9The baseline data is not included in analytic results, and we are presenting it as a descriptive information only.

15
Table 2: Mean values of covariates and t-test for mean-dierence (before and after matching), for books and training (April 2008)

|Variable|Control|Treated|%bias|% reduct|t-|test|
|---|---|---|---|---|---|---|
||Unmatched<br>(Matched)|Unmatched<br>(Matched)||bias|t|p>t|
|Gender (=1 for boys)|0.52|0.50|3.7||0.83|0.41|
||(0.52)|(0.53)|(-1.3)|63.3|(-0.26)|(0.80)|
|Age|10.37|10.39|-2||-0.45|0.65|
||(10.37)|(10.34)|(3.2)|-62.8|(0.60)|(0.55)|
|Number of books at home|2.16|2.17|-1.4||-0.32|0.75|
||(2.16)|(2.10)|(6.8)|-373.1|(1.32)|(0.19)|
|Extra lesson (frequency)|2.35|2.27|7.7||1.77|0.08*|
||(2.35)|(2.31)|(3.8)|50.1|(0.74)|(0.46)|
|Chores before school (frequency)|2.92|2.94|-1.6||-0.36|0.72|
||(2.92)|(2.98)|(-5.6)|-253|(-1.10)|(0.27)|
|Chores after school (frequency)|2.95|2.98|-2.7||-0.61|0.54|
||(2.95)|(2.94)|(1.6)|40.6|(0.30)|(0.76)|
|Reside far from school|0.03|0.04|-0.3||-0.07|0.94|
||(0.03)|(0.03)|(4.4)|-1279.8|(0.90)|(0.37)|
|HH size|5.44|5.09|21.8||5.11|0.00***|
||(5.44)|(5.39)|(3.2)|85.3|(0.59)|(0.55)|
|Living arrangement|0.56|0.52|7.5||1.69|0.09*|
||(0.56)|(0.52)|(8.3)|-11.7|(1.61)|(0.11)|
|Residence type|1.85|1.74|11.5||2.64|0.01**|
||(1.85)|(1.89)|(-5.3)|54.4|(-0.97)|(0.33)|
|Telephone at home|0.50|0.64|-30.2||-6.91|0.00**|
||(0.50)|(0.48)|(4.1)|86.4|(0.78)|(0.44)|
|Mother/father has secondary edu|0.50|0.47|6||1.36|0.17|
||(0.50)|(0.47)|(5.6)|5.9|(1.09)|(0.28)|
|Teacher's rank|2.82|2.90|-9.4||-2.21|0.03**|
||(2.82)|(2.85)|(-3.4)|63.6|(-0.67)|(0.50)|
|Teacher's experience (year)|14.26|15.13|-8.1||-1.90|0.06*|
||(14.26)|(13.75)|(4.8)|40.9|(0.95)|(0.34)|
|Hand washing facility exists|0.78|0.83|-10.5||-2.43|0.02**|
||(0.78)|(0.79)|(-2.4)|77.5|(-0.44)|(0.66)|
|School has toilet|0.63|0.44|38.6||8.73|0.00***|
||(0.63)|(0.63)|(-1.4)|96.5|(-0.27)|(0.79)|

Note: Living arrangement refer to whether the child resides with his mother and/or father, grandparents, other relatives or school dormitory. Residence type includes `ger', house, apartment or school dormitory. Chore frequency refers to number of days per week the child has to do household chores before/after school.

16
Extra Training: For the groups of students that received extra training, on top of books, similar matching results are presented in Table A.1 in the annex. After matching, the covariates that could inuence test scores, such as students' and their families' characteristics, teachers' qualications, and school features also did not dier signicantly between the treatment and control groups. In addition, the baseline test scores are generally equivalent among the treated and control groups of students, with a mean total score of 25.2 and 24.0, respectively. Scores on individual tests are also comparable. During the endline survey, students in both groups improved their total mean score, but there is no pronounced widening of the gap in the mean score between treatment and control groups (Table A.3).

Books only: For this intervention, after matching, there was no systematic dierence in teachers' qualications, students' and their families' characteristics as well as school conditions between treatment and control groups (see Table A.2). Similarly, there was no systematic dierence between control and treatment groups at baseline in terms of achievement. The mean of total test scores for treatment and control groups of students were 23.6 and 24.6 points, respectively. Baseline scores on individual tests are also similar across the two groups. The means of total scores on the follow-up tests for treatment and control students were 31.6 and 29.0 points, respectively (Table A.3).

# 4.2 Analytic results

This section presents estimated treatment eects using the empirical approach outlined in subsection 3.2. In the subsequent section, we assess heterogeneity in treatment eects, and also present robustness checks by re-estimating ATEs under dierent specications. The results show that when in-service teacher training and books are are provided individually, they weakly improve test scores on some, though not all, subjects. However, when teachers are trained and students are provided with the necessary books to facilitate the implementation of knowledge acquired during training, test scores improve considerably. The impact of each intervention is discussed below.

Training and Books Intervention: For the group of students who accessed books through classroom libraries and whose teachers participated in training, test scores on almost all tests improved substantially. Table 3 presents ATE on individual test scores as well as on the total score. The total test score increased by equivalent to 34.9 percent 17
of a standard deviation. As shown in Figure 2 (panel A and B), the kernel densities of the treatment and control groups generally overlap during the baseline survey. During the endline survey, the mean test score of the treatment group of students was higher than that of the control group. Considering each test individually, the intervention improved writing and math test scores the most (by 27.1 and 25.9 percent of standard deviation, respectively). Reading and Peabody test scores increased, respectively, by 25.6 and 20.9 percent of standard deviation, respectively.<sup>10</sup> The interventions did not improve performance on listening test.

Table 3 Impact of teacher training and books on test score

||(1)|(2)|(3)|(4)|(5)|(6)|
|---|---|---|---|---|---|---|
||Peabody|Math|Listening|Reading|Writing|Total Score|
|ATE|0.481<sup>_∗∗∗_</sup>|0.617<sup>_∗∗∗_</sup>|0.225|0.989<sup>_∗∗∗_</sup>|0.555<sup>_∗∗_</sup>|2.867<sup>_∗∗∗_</sup>|
||(0.00)|(0.00)|(0.10)|(0.00)|(0.02)|(0.00)|
|_N_|2424|2424|2424|2424|2424|2424|

b coecients; p in parentheses _∗ ∗∗ ∗∗∗ p <_ 0 _._ 05 , _p <_ 0 _._ 01 , _p <_ 0 _._ 001 Note: The standard errors are clustered at aimag level, with wild cluster boostraping" (Cameron et al., 2008). The matching variables are characteristics of the student, household, teacher and school. Student's characteristics includes gender, age, number of books she owns, frequency of extra lessons after school, frequency of accomplishing household chores before and after school and a dummy for residing more than one hour walk from school and commuting by foot. Characteristics of the student's household encompasses household size, the household head's relationship with the child, wealth indicators (such as housing condition, dummy for phone and car ownership) and education level. Teacher's formal education and years of experience makeup characteristics of the student's current teacher. Teacher's formal education and years of experience makeup characteristics of the student's current teacher. School characteristics encompasses dummy for existence of hygiene infrastructure (like toilet and hand-washing facilities).

> 10For training and books intervention group of students, the standard deviations of test scores in Peabody, math, listening, reading, writing and total score are 2.30, 2.38, 1.93, 3.85, 2.05 and 8.19, respectively (see Table A.3 in the annex).

18
Figure 2: Density of total test score for teacher training and books intervention <!-- Start of picture text --> Panel A:- Baseline Survey Panel B:- Follow-up survey<br>0 10 20 30 40 50 0 20 40 60<br>Total test score Total test score<br>Treatment Treatment<br>Control Control<br>.05 .04<br>.04 .03<br>.03<br>.02<br>Density Density<br>.02<br>.01<br>.01<br>0 0<br><!-- End of picture text --> Note: Total test score is the sum of scores in math, reading, writing, listening and Peabody tests.

Extra Teacher Training: As discussed above, the comparison of test scores of students who were treated with in-service teacher training and books against those who received books only is the basis of estimating ATE of teacher training only. This comparison shows that the extra teacher training has weaker impacts on test scores. Due to the extra teacher training intervention, total test scores did not improve (Table 4). Figure 3 presents the kernel density of total test score, which reveals a similar result: both the treatment (training and books receivers) and control (books only receivers) groups of students performed similarly during the baseline and follow-upeven if the mean of total score improved during the follow-up survey for both groups. Out of the ve tests, only score in writing has improved by 15.3 percent of a standard deviation, and this is smaller in magnitude when compared to training, complemented with books.<sup>11</sup> No impact on Peapody, math, reading and listening test scores due to the extra inservice teacher training was found.<sup>12</sup> These ndings lie at the heart of the contentious literature on eectiveness of brief in-service teacher training schemes in improving test scores. Some previous studies nd training improved test score, other do not. For instance, Jacob and Lefgren (2004), employing a quasi-experimental method based on the school reform program in Chicago, established that in-service teacher training had no statistically signicant or academically meaningful impact on reading and math achievement of students in elementary school. Similarly, Zhang et al. (2013) undertook a randomized control trial and documented that short-term in-service teacher training in Beijing's migrant

> 11As discussed in the `identication strategy' section, the impact training only is likely to be overestimated as it might also include any complementarity eects between these inputs.

> 12For students in training only intervention group, the standard deviations of test scores in Peabody, math, listening, reading, writing and total score are 2.14, 2.58, 2.23, 4.15, 2.10 and 8.55, respectively.

19
schools did not improve scores in an English prociency test. Using observational data from rural primary schools of Thailand, teachers' exposure to in-service training has been shown not to predict instructional quality or student achievement in Thai language, math, social and natural studies, character development and work orientation tests (Raudenbush et al., 1993). However, others nd that teacher training enhances students' performance in these subjects. For instance, Angrist and Lavy (2001) documented that in-service training has had a signicant impact on students' achievement in math and reading in non-religious elementary schools in Jerusalem, whereas the impact on the achievement of students in religious schools was inconclusive. Similarly, Harris and Sass (2011) and Lai et al. (2011) found that teachers' qualications and on the job training improve student outcomes. These results from previous studies are consistent with the ndings of this studyextra teacher training, on top of books, weakly improves test score in some subjects. However, when training is provided along with appropriate books, it strongly improves student outcomes.

After all, the circumstances under which training becomes eective could be diverse. Among other factors, whether the teachers have the necessary teaching aids to implement any pedagogical technique they acquire from training could be crucial. Especially in countries where essential education inputs may be missing, in-service teacher training could render ineective. In fact, as we have documented above, when training is combined with book provision, test scores in most subjects improve substantially.

Table 4: Impact of extra teacher training, on top of books, on test score

||(1)|(2)|(3)|(4)|(5)|(6)|
|---|---|---|---|---|---|---|
||Peabody|Math|Listening|Reading|Writing|Total Score|
|ATE|0.243|-0.0563|-0.0491|-0.229|0.321<sup>_∗_</sup>|0.229|
||(0.38)|(0.84)|(0.72)|(0.48)|(0.08)|(0.88)|
|_N_|2968|2968|2968|2968|2968|2968|

b coecients; p in parentheses _∗ ∗∗ ∗∗∗ p <_ 0 _._ 05 , _p <_ 0 _._ 01 , _p <_ 0 _._ 001 Note: The matching variables are characteristics of the student, household, teacher and school. For the full list of covariates see Table A.1 in the annex.

20
Figure 3: Density of total test score for teacher training only intervention <!-- Start of picture text --> Panel A:- Baseline Survey Panel B:- Follow-up Survey<br>0 10 20 30 40 50 0 20 40 60<br>Total test score Total test score<br>Treatment Treatment<br>Control Control<br>.05 .04<br>.04<br>.03<br>.03<br>.02<br>Density Density<br>.02<br>.01<br>.01<br>0 0<br><!-- End of picture text --> Note: Total test score is the sum of math, reading, writing, listening and Peabody test scores.

Books only: Providing books had a strong impact on test scores. Books alone greatly increased test scores more than teacher extra training, but the books intervention still had a much weaker impact than training and books provided as a package. It improved scores in many more subject tests. For instance, it increased the total score by 20.6 percent of a standard deviation (Table 5). The density of the total test score for the treatment and control group of students exhibits a mildly stronger shift in mean score among the treated groups of students (Figure 4). The intervention improved the scores in two of the ve tests. It increased scores in reading and math tests by 22.2 and 25 percent of standard deviation, respectively. These improvements in test scores due to book provision are lower than the impacts under the joint provision of training and books.<sup>13</sup> The ndings that books improve test scores in some subjects, even when provided alone, is in line with a general narrative provided in the systemic review by Glewwe et al. (2013): when considering all the evidences holistically, textbooks and workbooks improve weakly learning outcomes. In addition, we nd that the return from the provision of books increases when it is jointly provided with teacher training. The latter result, along with the fact that training also works better when provided along with books, is evidence of the complementarity of education inputs.

> 13For group of students in books-only intervention, the standard deviations of test scores in Peabody, math, listening, reading, writing and total score are 2.39, 2.36, 1.80, 3.93, 2.02 and 8.13, respectively.

21
Table 5: Impact of books only on test score

||(1)|(2)|(3)|(4)|(5)|(6)|
|---|---|---|---|---|---|---|
||Peabody|Math|Listening|Reading|Writing|Total Score|
|ATE|-0.105|0.525**|0.186|0.982***|0.124|1.712*|
||(0.78)|(0.02)|(0.20)|(0.00)|(0.72)|( 0.08)|
|_N_|2111|2111|2111|2111|2111|2111|

b coecients; p in parentheses _∗ ∗∗ ∗∗∗ p <_ 0 _._ 05 , _p <_ 0 _._ 01 , _p <_ 0 _._ 001 Note: The matching variables are characteristics of the student, household, teacher and school. For the full list of covariates see Table A.2 in the annex Figure 4: Density of total test score for books only intervention <!-- Start of picture text --> Panel A:- Baseline Survey Panel B:- Follow-up Survey<br>0 10 20 30 40 50 0 20 40 60<br>Total test score Total test score<br>Treatment Treatment<br>Control Control<br>.05 .04<br>.04 .03<br>.03<br>.02<br>Density Density<br>.02<br>.01<br>.01<br>0 0<br><!-- End of picture text --> Note: Total test score is the sum of math, reading, writing, listening and Peabody test scores

# 4.3 Heterogeneity in treatment eects

This subsection investigates any heterogeneity in treatment eects using three subsamples of students, based on their gender, access to extra lessons, and parental education. On the bases of each of the above characteristics, the sample was divided into two subgroups: students who have taken at least one extra lesson per week versus those who did not; students whose either (or both) parent have completed secondary education against those whose parents have not completed high school; and boys or girls. It is reasonable to expect that students who have taken extra lessons or have educated parents could benet dierently from these interventions.

22
For students who did not have access to extra lessons, provision of these inputs, either individually or as a package, improved their performance meaningfully. Especially, books only and training and books as a package increased the test score of this group. On the other hand, students that have taken extra lessons outside school have performed better in some subjects when they were treated with these interventions. However, the overall improvements in the performance of this group is relatively smaller than those students who did not have access to extra lesson (see Table A.4, annex).

Returning to parental education, we nd that students whose parents have not completed secondary education have beneted from books and training, and books only interventions more than those with educated parents. In addition, these students improved their performance more when books and training were provided together. Moreover, training teachers does not seem to help students with less educated parents and educationed parents alike (Table A.5). In terms of the student's gender, there are dierences in treatment eects of the three interventions. The provision of packaged inputs (training and books) improved girls' score more than boys. But books alone do not seem to improve girls' test scores signicantly (Table A.6). The general message from these results is that providing packaged inputs helps groups of students who might be disadvantaged (i.e. those who do not have access to extra-lesson sessions, with less educated parents, and girls).

# 4.4 Robustness check

In this section, we check the robustness of the results presented in the preceding subsection by re-estimating the impacts of each intervention under dierent specications. To assess how the estimated impacts could change with changes in matching variables, the propensity score matching estimation is implemented by progressively including characteristics of students, their families, teachers and schools in four specications. In addition, we estimate the treatment eect on the total test score by matching on all possible combinations of covariates (by adding and dropping regressors), while including the students' characteristics as `core variables' in all the regressions. Despite the limitations of using this method (see Lu and White (2014)), this provides reasonable checks as to whether the treatment eect is appropriately estimated. Table A.7 (in the annex) presents the average treatment eects (ATEs), for the three interventions, with various sets of matching variables. In specication 1, we present ATEs by 23
matching students based on their own characteristics only. In subsequent specications, we progressively include characteristics of their families, teachers and their schools' resources. The results, in general, support the main ndingsteacher training provided along with teaching aids improves test scores substantially, while the interventions implemented individually have weak impacts and improve scores only in some subjects.

In addition, we estimate the treatment eects by pooling the three groups together and estimating Equation 5. The result, presented in Table 6, is consistent with main result. It shows that inputs provided as a package improve test scores signicantly, relative to isolated input provision. In this approach, we nd that teacher training has no eect on all test scores (even on writing, which was statistically signicant in the main specication).

Table 6 Impact of teacher training and books, and books only on test score

||(1)<br>Peabody|(2)<br>Math|(3)<br>Listening|(4)<br>Reading|(5)<br>Writing|(6)<br>Total Score|
|---|---|---|---|---|---|---|
|Training and Books|0.557<sup>_∗∗_</sup>|0.772<sup>_∗∗∗_</sup>|0.268|1.210<sup>_∗∗∗_</sup>|0.614<sup>_∗∗_</sup>|3.420<sup>_∗∗∗_</sup>|
||(0.04)|(0.00)|(0.12)|(0.00)|(0.04)|(0.00)|
|Books only|0.0335|0.578<sup>_∗∗∗_</sup>|0.176|1.048<sup>_∗∗∗_</sup>|0.145|1.980<sup>_∗_</sup>|
||(1.00)|(0.00)|(0.34)|(0.00)|(0.74)|(0.00)|
|Extra-training*|.523|.194|.092|.162|.469|1.44|
||[0.221]|[0.745|[0.737]|[0.87]|[0.181]|[0.952]|
|_N_|5038|5038|5038|5038|5038|5038|

Note: The standard errors are clustered at aimag level, with wild cluster boostraping" (Cameron et al., 2008). P-values in parentheses:<sup>_∗_</sup> _p <_ 0 _._ 05 ,<sup>_∗∗_</sup> _p <_ 0 _._ 01 ,<sup>_∗∗∗_</sup> _p <_ 0 _._ 001 . The matching variables are those used in the main results. *The impact of extra-training is calculated using post-estimation test for the dierence between coecients of training and books and books only estimations. P-values of the chi-squared test for the dierences are in brackets.

# 5 CONCLUSION

Policy makers around the world are keenly interested in the potential of in-service teacher training programs and the provision of high-quality learning materials to help improve schooling outcomes. Surprisingly few evaluations have used a randomized controlled trial approach to examine the impacts of introducing these types 24
of interventionseither individually or jointlyin developing countries. Limited conclusive evidence exists about the impact of these interventions on primary school programs, and most of this evidence comes from small pilot projects. Even less evidence is available regarding their impact as part of a nationwide education program.

This work lls a gap in the literature. While other studies have provided inconclusive evidence as to the impact of teacher training or book provision on student outcomes when inputs are provided individually, no previous work has attempted to explore the dierential impact of providing these two critical education inputs individually versus jointly to test for any input complementarity in education investments. This study thus provides interesting, new, and important insights. The evaluation found signicant, positive eects on student outcomes when books and training were provided together as a package, rather than as individual inputs. Books only and extra teacher training marginally improved test scores in some, but not all, subjects. The magnitude of impact of either input was not academically signicant. However, when teachers are trained and students are provided with books, the test scores of a treatment group of students increased substantially, relative to a control group of students.

The ndings from this study provide information to education policy makers in developing countries on how their input allocation choices could result in signicantly dierent outcomes. Isolated education investments in settings where complementary inputs are missing could deliver minimal or no return. On the other hand, coordinated investments could improve student outcomes substantially, beyond and above the sum of returns from the same investments undertaken individually. These coordinated interventions are very cost eective. Equipping a classroom with 160 books and a set of shelves costs only $353.5 (in 2008 US$). Similarly, as noted above, the cost of training teachers was relatively low. This makes the cost of these joint interventions per student substantially lower.

To inform the design and implementation of future teacher training and book provision schemes, other research should focus on exploring the impacts of providing packaged inputs versus isolated inputs in settings with dierent levels of resource availability (classroom, school, household, and region). It may be likely that heterogeneity in treatment eects based on the existence of complementary schooland household-resources will prevail, while the result may not hold in areas where a 25
reasonable amount of education resources are already in place. Additional work should also investigate the impact of dierent types of teacher training programs, including methods, pedagogical strategies, and rollout of these interventions, on test scores. Detailing these outcomes would have signicant implications for policy makers with limited resources who are seeking improved eciency and better student outcomes.

26