metadata: authors: 0: author: De Ree,Joppe Jaitze
count: Indonesia
docna: 0: docna: How much teachers know and how much it
            matters in class : analyzing three rounds of
            subject-specific test score data of Indonesian students and teachers
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Educational Populations,Education for Development (superceded),Effective Schools and Teachers,Education For All
teratopic: Education
abstracts: cdata!: Improving the quality of education is
            one of today's main challenges for governments in the
            developing world. Based on a unique matched
            student-to-teacher panel data set on test scores this paper
            presents two empirical results for Indonesia. First, through
            detailed inspection of teacher-level responses to test
            questions, the paper concludes that subject matter knowledge
            of primary school teachers in Indonesia is low on average
            and that a 1.0, but also a 2.0 standard deviation increase
            in teachers' subject matter knowledge seem to be
            achievable medium-term goals for education policy making in
            Indonesia. Second, the paper presents the results of three
            types of value-added regressions, a (standard) level
            specification, a school fixed-effects specification, and a
            flexible student-teacher fixed-effects specification. The
            student-teacher fixed-effects approach estimates the
            parameters of a value-added model using test score variation
            within each student-teacher pair across three different
            subjects, mathematics, science and Indonesian language. The
            results suggest that a 1.0 (and 2.0) standard deviation
            increase in teachers' subject matter knowledge
            across-the-board can yield increases in student achievement
            by 0.25 (and 0.50) student-level standard deviations by the
            time students complete the six-year primary school cycle.
geo_reg_key: 517191,517198
repnme: repnme: How much teachers know and how much it
            matters in class : analyzing three rounds of
            subject-specific test score data of Indonesian students and teachers
display_title: How much teachers know and how much
            it matters in class : analyzing three rounds of
            subject-specific test score data of Indonesian students and teachers
listing_relative_url: /research/2016/02/25861575/much-teachers-know-much-matters-class-analyzing-three-rounds-subject-specific-test-score-data-indonesian-students-teachers
topicv3: Teachers Management m1326119 1523,Education, Skills Development and Labor
            Market m1326135 1484,Quality of Education m1326344 491,Education m1327048 470,Education and Culture m1327747 1472
publishtoextweb_dt: 2016-02-02T00:00:00Z
historic_topic: Education
no_of_pages: 40
author_keys: 0: author_key: 906069
model_extractions: WPS7556 Policy Research Working Paper 7556 How Much Teachers Know and How Much It Matters in Class Analyzing Three Rounds of Subject-Specific Test Score Data of Indonesian Students and Teachers _Joppe de Ree_ Education Global Practice Group February 2016
Policy Research Working Paper 7556

# **Abstract**

Improving the quality of education is one of today’s main challenges for governments in the developing world. Based on a unique matched student-to-teacher panel data set on test scores this paper presents two empirical results for Indonesia. First, through detailed inspection of teacherlevel responses to test questions, the paper concludes that subject matter knowledge of primary school teachers in Indonesia is low on average and that a 1.0, but also a 2.0 standard deviation increase in teachers’ subject matter knowledge seem to be achievable medium-term goals for education policy making in Indonesia. Second, the paper presents the results of three types of value-added regressions, a (standard) level specification, a school fixed-effects specification, and a flexible student-teacher fixed-effects specification. The student-teacher fixed-effects approach estimates the parameters of a value-added model using test score variation within each student-teacher pair across three different subjects, mathematics, science and Indonesian language. The results suggest that a 1.0 (and 2.0) standard deviation increase in teachers’ subject matter knowledge across-the-board can yield increases in student achievement by 0.25 (and 0.50) student-level standard deviations by the time students complete the six-year primary school cycle.

This paper is a product of the Education Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at joppederee@ gmail.com.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
How Much Teachers Know and How Much It Matters in Class: Analyzing Three Rounds of Subject-Specific Test Score Data of Indonesian Students and Teachers Joppe de Ree<sup>_∗†_</sup>

# _JEL Classification_ : I21, I25, H42, O15

_Keywords_ : Education, value-added modeling, teachers, subject matter knowledge, Indonesia

> _∗_ World Bank, email: joppederee@gmail.com

> _†_ The research presented in this paper was generously supported by the Dutch Education Support Program (DESP) funded by the Government of the Kingdom of The Netherlands. Special thanks go to Dedy Junaedi (and his teams of field workers, data managers, etc.), Susiana Iskandar, Titie Hadiyati (and team), Amanda Beatty (and team), Halsey Rogers, Menno Pradhan, Karthik Muralidharan, Andy Ragatz, Samer Al-Samarrai, Ai Li Ang, Husnul Rizal, Laura Wijaya, and other colleagues at the World Bank office in Jakarta, as well as to colleagues at the Indonesian Ministry of Education and Culture, such as Yendri Wirda Burhan, Simon Silisabon (and the team at _puslitjak_ ), Rahmawati, Yani Sumarno (and the team at _puspendik_ ).

1
# **1 Introduction**

Improving the quality of education is one of today’s main challenges to policy makers in Indonesia and elsewhere in the developing world. Despite the fact that Indonesia doubled real spending on education in the last decade<sup>1</sup> , the quality of instruction, as measured through international student comparisons like PISA and TIMSS, has not (or not much) improved. PISA researchers conclude for example that half of Indonesia’s 15-year-olds do not have “a basic level of proficiency, at which students begin to demonstrate the reading literacy competencies that will enable them to participate effectively and productively in life [p50. in OECD (2010)].”<sup>2</sup> This suggests that many teachers and schools fail to prepare the next generation for an ever-more demanding global labor market.

This paper presents two empirical results for Indonesia that potentially also apply to the broader developing world. First, section (2) shows that subject-matter knowledge of primary school teachers in Indonesia is arguably low on average, and most likely a binding constraint to learning in many Indonesian primary schools. Second, section (4) presents evidence that increases in teachers’ subject-matter knowledge can help increase student-learning outcomes. The inference is based on three types of value-added regressions: a standard model in levels, a school fixed-effects model, and a flexible student-teacher fixed-effects model. The latter estimates the parameters of a value-added model, only using variation within each student-teacher pair across the three different subjects – mathematics, science and Indonesian language.

The student-teacher fixed-effects model attempts to address important threats to conventional value-added models in levels. The fixed-effects results, for example, would be insensitive to situations in which intrinsically more able students students are matched to better schools or teachers. Moreover, models in levels are incapable of separating out a teacher’s subjectmatter knowledge from other (unobserved) teacher attributes, such as motivation, general intelligence, etc. The idea of looking at within-student variations across different subjects is not new and has been successfully applied before by Metzler and Woessman (2012).<sup>3</sup> Metzler

> 1See World Bank (2013)

> 2Almost 80% of Indonesian 15-year-olds do not have a baseline level of proficiency in Mathematics, that is below level 2 as coded by OECD (2010).

> 3The idea of using within-student variation (across subject scores, or otherwise) has been used before, see for example Dee (2005), Dee (2007), Ammermüeller and Dolton (2006), Clotfelter, Ladd, and Vigdor (2010),

2
and Woessman (2012) however base their analysis on a cross-section of data only, while this research estimates the parameters of a fully specified dynamic value-added model.

The data we use are unique for the developing world. Subject-matter test score data of all (around 45,000) students of a near representative sample of 240 primary schools are tracked across three rounds of measurement, November 2009, May 2011 and May 2012. Figure (1) presents a map of the 20 districts that were randomly selected to take part in the study.<sup>4</sup> The student level test score data can be linked to survey and subject-matter test score data of their respective teacher(s). A feature of this data is that both students and teachers are tested on the same three subjects – math, science and Indonesian language – in each of the three rounds of measurement. The data therefore allows for estimating value-added models on variation within each student-teacher pair, across the three subjects. At the same time, the data permits to account for biases due to measurement error in test scores (under standard assumptions on the nature of the measurement error).

**Figure 1:** Map of Indonesia, with the 20 selected districts highlighted

Kingdon and Teal (2010), and Chu, Loyalka, Chu, Qu, Shi, and Li (2015).

> 4The data was collected in support of a randomized controlled experiment designed to evaluate the effects of Indonesia’s teacher certification program (De Ree, Muralidharan, Pradhan, and Rogers (2015)). The data is nearly geographically representative for the majority public primary schools in Indonesia. (The data on 120 Junior Secondary schools is not used in this paper.) Indonesia spans across 1,700 more or less inhabited islands and is the world’s fourth most populous country, and the distance from the most western school in the sample, on mainland Sumatra, to the most eastern, on one of the remote islands of the South Moluccas, roughly spans the distance between San Francisco and New York. See also De Ree, Al-Samarrai, and Iskandar (2012), Chang, Shaeffer, Al-Samarrai, Ragatz, De Ree, and Stevenson (2013) and World Bank (2015), for analysis based on the same data. For more information on the data and the experiment see De Ree, Muralidharan, Pradhan, and Rogers (2015) and World Bank (2015).

3
The results of the analysis suggest that teachers’ subject-matter knowledge is important, and that 1.0 but also 2.0 standard deviation increases in teachers’ subject-matter knowledge are realistic medium-term targets for education policy making in Indonesia. A 1.0 and 2.0 standard deviation increase in teachers’ subject-matter knowledge across-the-board, is predicted to increase student-learning outcomes by 0.25 and 0.50 student-level standard deviations respectively by the time students complete the six-year primary cycle. These effects are substantial and would amount to 17.5 to 35.0 points respectively on the scale used by PISA if such impacts would persist until age 15.

The paper is organized as follows. Section (2) presents the teacher test score data and provides an intuitive account of the level and spread of subject-matter knowledge among Indonesia’s primary school teachers. Section (3) presents the statistical value-added model and presents ways of dealing with the difficulties of estimating the parameters of such models, e.g. these difficulties are due to measurement error in test scores and the time persistence of unobserved inputs. Section (4) presents regression results of three different value-added models: the standard level model, the school fixed-effects model, and a student-teacher fixedeffects model. Section (5) concludes and discusses the findings within the broader context of Indonesia’s education system.

# **2 How much do teachers know?**

The level of teachers’ subject-matter proficiency in Indonesia is said to be particularly low on average, even though rigorous international comparisons do not exist. Muhammad Nuh, for example, the country’s former Minister of Education and Culture reflected on the results of a large scale competency assessment of Indonesian teachers in 2012: “.. on one hand I am happy with the plethora of poor results because it means the data is reliable (The Jakarta Globe 2012).” Low levels of subject-matter proficiency can be a real constraint to effective teaching, even if teachers have a talent for delivering a message to a classroom. This section provides an intuitive account of the level and spread of subject-matter proficiency of Indonesian primary teachers. The data reveals somewhat poor performance overall as the 4
majority of primary teachers have difficulties with high-school material.

The tests used in this research were developed by the Center for Educational Assessment of the Indonesian Ministry of Education and Culture ( _Pusat Penilaian Pendidikan_ or _Puspendik_ ). The tests consist of a 20 item math component, a 20 item science component, and a 20 item Indonesian language component. Another 40 items were used to assess pedagogical knowledge and social and personality traits. Teachers had 2 hours to complete the entire test. This meant that teachers had 1 minute and 12 seconds on average for each question. All questions were 5-option multiple choice questions. Approximately 1,700 primary school teachers in 240 public primary schools were tested.

The test intended to roughly reflect what the Center for Educational Assessment expects teachers to know. Scores of around 60 percent correct or higher, then, are considered “a pass”. But what does passing really mean? The Center for Educational Assessment may have been too ambitious for example, by setting targets too high. It is even more difficult to compare such raw scores internationally as standards for _appropriate levels of knowledge_ may differ between countries. Comprehensive studies specifically designed to evaluate teachers’ subject-matter knowledge across countries, such as PISA or TIMSS for students, do not exist. Hanushek, Piopiunik, and Wiederhold (2014) cleverly measure cross-country variations in teacher knowledge from OECD’s Programme for the International Assessment of Adult Competencies (PIAAC). PIAAC does not focus on teachers explicitly, but as the PIAAC samples are large enough, sufficiently many teachers are part of it. The first round of PIAAC however does not survey any of the developing economies.

As cross-country comparisons of teachers’ subject knowledge are lacking in the developing world, we rely on introspection of the reader of this paper to make an assessment of whether Indonesian teachers are up to standard. For the analysis we select two items from the 20-item math component of the subject-matter test of the Midline study (fielded in April-May 2011). Midline data of teachers is arguably the most reliable source of information we have, as both baseline and endline had to rely on smaller budgets for the field work. Especially, for midline, we managed in most cases to test teachers in the same classroom as their students, thereby minimizing the risk of collaboration among teachers. The two test items we look at are the 5
<!-- Start of picture text --> @ @®@<br>® se See eese<br><!-- End of picture text -->
To answer question 1 test-takers must understand how these triangles are constructed. If one side of the triangle has 8 marbles, there are 7 and 6 marbles needed to complete it. The correct answer therefore is 8 + 7 + 6 = 21, or answer _b_ . Item 2 requires more calculation and the numbers used in the problem add to the complexity.<sup>6</sup> There are different ways of solving this problem. One is, by saying that what 14 robots can do in 45 minutes, is <u>14</u> _<u>×</u>_ <u>45</u><sup><u>14</u></sup><sup>_<u>×</u>_</sup><sup><u>1514</u></sup><sup>_<u>×</u>_</sup><sup><u>1</u></sup><sup>_<u>.</u>_</sup><sup><u>5</u></sup> = 21 times as much as what 3 robots can do in 10 minutes. Because 3 3 _×_ 10<sup>=</sup> 1 _×_ 10<sup>=</sup> 1 _×_ 1 robots produce 17 cars in 10 minutes, 14 robots can produce 17 _×_ 21 = 357 cars in 45 minutes. The correct answer therefore is _c_ .

Of our sample, 45% of teachers answered question 1 correctly and 33% answered question 2 correctly, and only 14% of teachers answered both questions correctly.<sup>7</sup> So rather than observing many teachers performing well, only a minority, in the end, did. This finding suggests a low level of subject-matter knowledge among primary school teachers in Indonesia: a sizable share of the teacher population has difficulties with material of the high-school mathematics curriculum. It is quite possible that low levels of teachers’ knowledge are a binding constraint for learning in many Indonesian primary classrooms. This is unfortunate for current generations of students of course, but it also presents a direct opportunity for change. At such low levels, improvements in teacher’s subject-matter knowledge might translate quickly into improved student-learning outcomes.

For policy makers in Indonesia the question of interest is: how much more knowledge do teachers need to acquire before investments in teacher training start to really pay off? The empirical literature almost universally reports effect sizes in terms of population standard deviations, e.g. Metzler and Woessman (2012) and section (4) of this paper. But population standard deviations are not necessarily intuitive quantities. It is not clear a priori what a standard deviation increase in teachers’ subject-matter knowledge means in every-day life. And whether it would be costly or cheap to achieve such changes through government-supported training programs. The remainder of this section provides an intuitive account of whether standard deviations of subject-matter knowledge are _a little_ or _a lot_ .

> 6One odd feature of the question for example, is that it is unrealistic to expect that robots can produce a car in 10 minutes.

> 7If all teachers would guess randomly on both items and across the five possible answers _a._ , _b._ , _c._ , _d._ , and _e._ , only about 0 _._ 20 _×_ 0 _._ 20 _×_ 100% = 4% would answer both items correctly.

7
To make intuitive what a 1.0 or 2.0 standard deviation increase in subject-matter knowledge means in everyday life, we discuss what it would mean with respect to the performance on the two test items introduced before. We model the probability of answering question 1 and 2 correctly at a given true knowledge level _x_<sup>_∗_</sup> as follows:<sup>8</sup> for _k_ = 1 _,_ 2 (question 1 or 2). _x_<sup>_∗_</sup> is a measure of true knowledge, standardized to have _E_ [ _x_<sup>_∗_</sup> ] = 0 and _V_ ( _x_<sup>_∗_</sup> ) = 1. Based on this model we can predict how likely an Indonesian primary teacher is to answer question _k_ correctly –or questions of similar content and difficulty– at each knowledge level _x_<sup>_∗_</sup> . And we can also evaluate how much more likely he or she is to answer questions like _k_ correctly, after a 1.0 standard deviation increase in subject-matter knowledge, i.e. by evaluating _D_ ( _x_<sup>_∗_</sup> ) = _P_ ( _Qk_ = 1 _|x_<sup>_∗_</sup> + _SD_ ( _x_<sup>_∗_</sup> )) _− P_ ( _Qk_ = 1 _|x_<sup>_∗_</sup> ).

Estimating the parameters _αk_ and _βk_ of (1) is not straightforward, because true achievement scores _x_<sup>_∗_</sup> are not observed. _True achievement scores x_<sup>_∗_</sup> are only proxied with _observed test scores x_ with usually a considerable amount of noise, or measurement error. In Appendix A we show how one can estimate the parameters _αk_ and _βk_ based on noisy test scores _x_ , assumptions on the nature of the measurement error in _x_ , and estimates of the reliability coefficient _ρ_ . Based on bias-corrected estimates of _αk_ and _βk_ and the probit model (1), we present the relationship between the standardized true achievement scores _x_<sup>_∗_</sup> on the horizontal axis, and the predicted probabilities _P_<sup>ˆ</sup> ( _Qk_ = 1 _|x_<sup>_∗_</sup> ) of answering question 1 [left panel] and 2 [right panel] correctly.<sup>9</sup>

An average teacher (or more precisely, a teacher who scores the population average _x_<sup>_∗_</sup> = 0) is able to solve problems like question 1, 44 percent of the time. Question 2 is considerably more challenging and the typical Indonesian teacher would be able to solve such problems only about 30 percent of the time. The average teacher therefore has some level of mastery

> 8This is a probit model, but a logistic regression model, or something more complex, could also be used for this purpose.

> 9Note that the model does not have a “guessing parameter”, such as the 3PL model from item response theory. Even without any knowledge test-takers should guess correctly once every five times (on a five item multiple choice subject-mattertest). However in our data we observe that test-takers not always report answers to each question.

8
<!-- Start of picture text --> 1 1<br>.8 .8<br>.6 .6<br>.4 .4<br>.2 .2<br>0 0<br>−2 −1 0 1 2 −2 −1 0 1 2<br>standardized true score standardized true score<br>predication probability of answering correctly predication probability of answering correctly<br><!-- End of picture text -->

**Figure 2:** Modeling the relationship between standardized true scores and the probability of answering question 1 (left panel) and 2 (right panel) correctly

over the skills needed to solve question 1, but he or she would not find question 1 easy.

What does a 1.0 standard deviation increase mean? A teacher with a level of subjectmatter knowledge of 1.0 standard deviation above average, i.e. _x_<sup>_∗_</sup> = 1, has much less problems with the first question and would answer such questions correctly more than 80 percent of the time. He or she generally has the skills needed to solve questions like 1 without difficulty. Question 2 however still remains somewhat of a challenge for a teacher in Indonesia with _x_<sup>_∗_</sup> = 1, as he or she would answer questions like these correctly 50 percent of the time.

What does a 2.0 standard deviation increase mean? A teacher with a level of subjectmatter knowledge of 2.0 standard deviation above average, i.e. _x_<sup>_∗_</sup> = 2, has no problems at all with the first question and would almost always answer questions like question 1 correctly. He or she has all the skills needed to solve problems as straightforward as question 1. Also, question 2 is not really difficult for a primary teacher in Indonesia with _x_<sup>_∗_</sup> = 2, although perhaps the numbers used in question 2 makes that the predicted performance even for this class of teachers is not closer to 1.

We can use the results of the model to make predictions for the entire distribution of today’s true teachers’ test scores. Even though true scores _x_<sup>_∗_</sup> are not observed at the individual level, we can still approximate the distribution of true scores by using standardized observed scores.<sup>10</sup>

> 10True scores _x∗_ have mean 0 and a standard deviation of 1 by construction. Standardized observed scores also have this property. For each individual test taker, the true score is generally not the same as the observed score. The distribution however does look the same, which is what matters here.

9
Figure (3) presents today’s predicted performance (the blue bars), in combination with the predicted performance when all teachers have increased their subject-matter knowledge by 1.0 (red bars) or 2.0 standard deviations (green bars) respectively.

The bars are splits based on quintiles of today’s true score distribution. The bottom quintile, i.e. poorest performing 20% of today’s population of teachers and representative of more than 200,000 Indonesian primary teachers, currently scores below 20% on item 1. Subsequent to an increase in subject-matter knowledge across the board of 1.0 and 2.0 standard deviations, we would expect these currently poor performing teachers to score 45% and 65% on the first item. For the other quintiles the model predicts more moderate increases.

<!-- Start of picture text --> 1 1<br>.8 .8<br>.6 .6<br>.4 .4<br>.2 .2<br>0 0<br>1 2 3 4 5 1 2 3 4 5<br>current performance current performance<br>predicted performance after 1SD increase predicted performance after 1SD increase<br>predicted performance after 2SD increase predicted performance after 2SD increase<br><!-- End of picture text -->

**Figure 3:** Current and predicted performance on item 1 (left panel) and item 2 (right panel), by overall true score quintile

A 1.0 and 2.0 standard deviation increase in knowledge are reflected in a somewhat substantial improvement in performance on the easier question 1, and somewhat more moderate increases in performance on the more challenging question 2. Designing and implementing policies that ensure lasting increases in subject-matter knowledge among teachers is a formidable challenge for policymakers in Indonesia. But the analysis provides clear evidence that such improvements are not quite out of reach. Much of these improvements can be realized within the existing education support structures in Indonesia. Especially the teacher working groups –known in Indonesia as _kelompok kerja guru_ (or _KGG_ )– can be used as a vehicle to provide extra training and practice. These teacher working groups exist broadly in Indonesia, and over 90% of the teachers in our sample are more or less active participants. In summary, 10
this section shows that subject-matter knowledge among Indonesian primary teachers is low, as a majority of the primary school teachers in Indonesia face difficulties with fairly elementary mathematical exercises. But this section also shows that 1.0 and 2.0 standard deviation increases in subject-matter knowledge can be realistic policy targets for the medium term.

# **3 Statistical value-added models**

Theorists and empiricists in the field of education research often refer to the importance of teachers in terms of how much they contribute to a child’s (scholastic) achievement. The term that is widely used in this context is a teacher’s “value-added”, where teachers add more or less to a child’s scholastic achievement. The statistical models used to confront this theoretical idea to data on test scores, are called value-added models. Value-added models can be derived from education production functions, where each input from birth until today contributes to current achievement (Todd and Wolpin 2003). Value-added models come in various forms (Guarino, Reckase, and Wooldridge 2015). The version I use in this paper is the following: Where _ysiτ_<sup>_∗_arestudent</sup><sup>_i_’strueachievementscoresonsubject</sup><sup>_s_atperiod</sup><sup>_τ_.Thevalue-added</sup> model links last year’s achievement _ysit_<sup>_∗_</sup> _−_ 1<sup>to current achievement</sup><sup>_y_</sup> _sit_<sup>_∗_, and</sup><sup>_inputssit_happening</sup> in-between. The literature often distinguishes _true_ and _observed_ achievement scores. True achievement is the actual achievement level of an individual, and observed achievement is a noisy proxy value of this obtained by test scores.<sup>11</sup> _inputssit_ in equation (2) may be all factors contributing to learning in a given year, from school inputs (e.g. teachers), to parental support, to individual (innate) interests, motivations and talents.

The model describes that the reasons for observing differences in achievement between students at a given point in time are either due to differences in inputs between period _t −_ 1 and _t_ (e.g. some have better teachers than others) or to differences in prior achievement

> 11The closer the test score is measuring some dimension of true achievement the less noisy test scores are, and the more reliable they are. In real world situations, test scores are always subject to more or less noise.

11
(which is due to the accumulation of inputs from birth until period _t −_ 1). The subject-matter knowledge of teachers is potentially one of the important input factors. We decompose the totality of inputs as follows: where _x_<sup>_∗_</sup> _sit_<sup>isastandardizedmeasureofteachers’subject-matterproficiency.</sup> _β_ measures the causal effect of a standard deviation increase in a teachers’ subject-matter proficiency on student-learning outcomes on a year-to-year basis. The residual term _usit_ captures the remaining inputs, and is a complicated function of unmeasured teacher abilities, parental factors, and individual student abilities, talents and interests, each influencing learning gains of student _i_ from period _t−_ 1 to _t_ on subject _s_ . Assumptions about the nature of _usit_ determine under which conditions we can estimate the causal parameter _β_ consistently.

In this paper, true achievement scores are standardized, i.e. they have a mean of 0 and a variance of 1.

Other restrictions on the theoretical model would be possible. One may use test equating techniques in an attempt to place the pre and and post test scores _ysit_<sup>_∗_and</sup><sup>_y_</sup> _sit_<sup>_∗_</sup> _−_ 1<sup>on the same</sup> scale for example [see Andrabi, Das, Khwaja, and Zajonc (2011) and Rothstein (2007) who make such adjustments]. Such approaches are not strictly necessary however for an intuitive interpretation of the results. The conditions (4) and (5) above mean that the value-added specification measures relative gains in the distribution of achievement, rather than absolute gains.<sup>12</sup>

> 12As a result of this, we know also that the average student, say one with a prior achievement score of 0, maintains his position in the distribution when his or her inputs are average as well. With above average inputs, a student will gain in relation to its peers and will score above the mean in the next period. It is straightforward to show that students with above average prior achievement also need above average inputs to prevent a loss in position. Naturally, to maintain a position at the bottom of the distribution, only below average inputs are required.

12
Combining the theoretical value-added specification (2) and the input split (3) yields something that looks like a dynamic regression model: The effect parameter _β_ and the input persistence parameter _γ_ are the parameters of interest of this research. Two major hurdles need to be taken before we can estimate these parameters however. Both issues are well established and the literature has proposed ways of dealing with them (see for example, Andrabi, Das, Khwaja, and Zajonc (2011)). The first problem is the likely _persistence in the unobserved inputs usit_ and omitted variable bias more generally. And the second is the _measurement error in test scores_ , which serve as proxies for the unobserved true achievement scores in (6).

# **3.1 Omitted variable bias: Three different empirical specifications**

Suppose, for the sake of exposition, that true achievement scores _ysit_<sup>_∗_,</sup><sup>_y_</sup> _sit_<sup>_∗_</sup> _−_ 1<sup>and</sup><sup>_x∗_</sup> _sit_<sup>are</sup> observed in the data. Even in these ideal circumstances, an OLS regression of today’s achievement scores _ysit_<sup>_∗_ontheteacher’stestscore</sup><sup>_x∗_</sup> _sit_<sup>andthestudent’sownlaggedscore</sup><sup>_y_</sup> _sit_<sup>_∗_</sup> _−_ 1<sup>,is</sup> probably not yielding consistent estimates of the parameters of interest _β_ and _γ_ . One basic problem is that the residual inputs _usit_ are likely correlated with teachers’ subject-matter proficiency _x_<sup>_∗_</sup> _sit_<sup>and, perhaps more importantly so, with a student’s lagged achievement</sup><sup>_y_</sup> _sit_<sup>_∗_</sup> _−_ 1<sup>.</sup>

**The level model.** A text-book issue related to dynamic (panel-data) regression models like (6) is that in most imaginable situations, the term _usit_ is persistent in time. More able or more intelligent students learn more today, but are also more likely to have higher achievement levels to start out with, because they have also learned more last year. In this scenario there exists a positive correlation between the residual inputs term _usit_ and the lagged achievement score _ysit_<sup>_∗_</sup> _−_ 1<sup>.Theideaisoperationalizedbydecomposingtheresidualterm</sup><sup>_usit_intoafixed,</sup>

13
and a time-varying component: where I assume that _εsit_ is uncorrelated over time.

In this paper I follow Blundell and Bond (1998) who argue that in stable ( _γ <_ 0) dynamic systems ∆ _ysit_<sup>_∗_</sup> _−_ 1<sup>=</sup><sup>_y_</sup> _sit_<sup>_∗_</sup> _−_ 1<sup>_−y_</sup> _sit_<sup>_∗_</sup> _−_ 2<sup>canbeusedasaninstrumentalvariable(IV)for</sup><sup>_y_</sup> _sit_<sup>_∗_</sup> _−_ 1<sup>,</sup> because ∆ _ysit_<sup>_∗_</sup> _−_ 1<sup>is uncorrelated with</sup><sup>_ηsi_and correlated with</sup><sup>_y_</sup> _sit_<sup>_∗_</sup> _−_ 1<sup>.Note that this IV strategy</sup> requires _three waves_ of panel data on test scores.

But whereas the IV solves some problems it does not solve all. For consistency of both model parameters in the level specification also _x_<sup>_∗_</sup> _sit_<sup>, the teacher’s achievement score, should be</sup> uncorrelated with _ηsi_ + _εsit_ . It is easy to think of reasons why _x_<sup>_∗_</sup> _sit_<sup>and</sup><sup>_ηsi_+</sup><sup>_εsit_are correlated.</sup> One is for example that better performing (high ability or wealthier) students self-select into better, and/or better funded schools. This idea is clearly relevant in the Indonesian context. Schools in urban centers are usually better funded, and therefore better able to attract better teachers. The best schools (or at least those who are perceived as the best) are also able to select the best students, or those willing to pay more to enroll. If more able, or better supported children (who learn more than others regardless of the schools they enroll in) also populate the schools with the best teachers, subject-matter ability of teachers _x_<sup>_∗_</sup> _sit_<sup>andthe</sup> residual inputs _usit_ are positively correlated. Such mechanisms pose a threat to the validity of the level specification.

The **school fixed-effects model** estimates the parameters of interest from variation across subjects and classrooms, but _within_ the same school. If the matching of better or better supported students to more knowledgeable teachers in better and better funded schools happens only at the level of the school, and not within schools, a school fixed-effects model would account for it. Where ability matching within schools might happen in some cases, it does not seem to be happening at scale in Indonesia (see section (4.1) for an analysis on selection).

14
The school fixed-effects model relies on the variation in subject-matter knowledge between teachers across different classrooms. And as endogenous matching within schools seems unlikely to be happening at scale, the statistically significant results we find in the results section (4) indicate that teachers with more subject-matter knowledge are indeed better teachers. While this finding is important in its own right, it does not imply that teachers with more subject-matter knowledge are better _because_ they have more knowledge. It could be, for example, that teachers who score higher on subject-matter tests tend to be more motivated, and that it is the additional motivation that makes them perform better. For the purpose of policymaking it is important to establish the causal effect of additional subject-matter knowledge, by isolating the subject-matter knowledge effect from other factors such as motivation and other (correlated) skills.<sup>13</sup> In an attempt to do this, we estimate a **student-teacher fixed-effects model** . The idea of the model is to only rely on variation in test scores for a single student-teacher pair, across three different subjects: math, science and Indonesian language. If a teacher’s subject-matter knowledge were truly important we would observe that students tend to do better in one subject (say math) than in another subject (say Indonesian language) if his or her teacher is better in math than languages. The idea of using variation across different subjects in an attempt to isolate the causal effects of teachers’ subject-matter knowledge on student learning is not new. Metzler and Woessman (2012) already used the idea on Peruvian data and find that a 1.0 standard deviation increase in the subject-matter knowledge of teachers leads to a 0.10 standard deviation increase in student learning. Metzler and Woessman (2012) however estimate their model on a cross section of data only.

# **3.2 Measurement error in test scores: From true achievement to observed test scores**

The _x_<sup>_∗_</sup> ’s and _y_<sup>_∗_</sup> ’s in the model are so-called true achievement scores. “True” refers to the idea that they are measured precisely, i.e. without error. In real life, observed test scores measure

> 13If the subject-matter knowledge scores used in the analysis only proxy for differences in the level of motivation, government programs to improve the levels of subject-matter knowledge of teachers would not be effective as they do not change levels of motivation.

15
“true” achievement scores only with some level of error. Differences between observed scores and the theoretical true scores are due to good/bad luck with guessing if test-takers are not fully convinced about an answer. Test-takers might perform better or worse depending on the time-of-day of the field visit, or depending on whether they are having problems at home, etc. The fact that observed test scores do not in general completely reflect a student’s level of true achievement is important in empirical value-added modeling. Much of the literature acknowledges the need to correct for measurement error in empirical operalizations of valueadded models (see Andrabi, Das, Khwaja, and Zajonc (2011) and many others).

Measurement error however in testing data yields two, somewhat different, problems. The first problem relates to the scaling of test scores. Prior to using test score data in valueadded regressions, choices must be made on _how to scale_ the test results. Simply using standardized (raw, percentage correct) test scores is not always appropriate, as gains on more noisy standardized scores would appear smaller than gains on cleaner standardized scores. This is because the variance of the “signal” in noisy standardized test score data is smaller than the variance of the “signal” in cleaner standardized test score data. This measurement issue appears to be somewhat neglected in the literature. The second is a problem more commonly associated with (classical) measurement error, i.e. the attenuation bias due to an automatic correlation between the noisy regressor and the measurement error component (which is part of the error term of the regression equation).

# **3.2.1 Noisy test scores and test scaling**

I assume throughout that the measurement error in test scores is “classical” in the sense that it has mean 0 in the population, and is uncorrelated with the true scores. The standard model for (classical) measurement error is the following: where _esit_ is the measurement error term, i.e. the difference between the observed score _ysit_ and the true score _ysit_<sup>_∗_.</sup> Based on this relationship we would be able to write the value16
added specification in terms of the observed scores _ysit_ . One complexity however is that we do not know the variance of _ysit_ , so that we do not know a priori how to construct it properly. The value-added model is defined in terms of standardized true scores, and the parameters of interest _β_ and _γ_ are interpreted with respect to this normalization. To maintain the normalization in estimation we should therefore construct the true scores so that _E_ [ _ysit_ ] = _E_ [ _ysit_<sup>_∗_] +</sup><sup>_E_[</sup><sup>_esit_] = 0and</sup><sup>_V_(</sup><sup>_ysit_) =</sup><sup>_V_(</sup><sup>_y_</sup> _sit_<sup>_∗_) +</sup><sup>_V_(</sup><sup>_esit_) = 1 +</sup><sup>_V_(</sup><sup>_esit_).Inotherwords</sup> the variance of the observed score used in the analysis, should have a variance that is _greater than one_ . How large it should be exactly, indeed, depends on the variance of the measurement error component.

Without some further analysis we do not know how large the variance of the measurement error component is, and, consequently, how to set the variance of the observed score _ysit_ . It is straightforward to show that, in general, the following conditions hold for the observed score _ysit_ , i.e the observed score that maintains the model normalizations _E_ [ _x_<sup>_∗_</sup> ] = 0 and _V_ ( _ysit_<sup>_∗_) = 1,andtheassumptionofclassicalmeasurementerror</sup><sup>_E_</sup> [ _e_ �� _x∗_ ] = 0: where _ρysit_ = _VV_ (( _yysitsit_<sup>_∗_</sup><sup><u>)</u></sup> )<sup>isthewell-knowncoefficientofreliability.</sup> The test scores _ysit_ used in the analysis is therefore constructed by the following transformation of the raw scores (percentage correct) _Rsit_ : The transformation applies to all test scores used in the analysis, where the mean and the standard deviation of the raw scores _μRsit_ and _σRsit_ can be calculated using the data.<sup>14</sup> The reliability coefficient _ρysit_ = _ρRsit_ is estimated based on the correlation between two splits of the raw test score data, one split is based on the even-numbered test items 2, 4, 6, etc.

> 14In the analysis the student scores are standardized by grade level and subject and the teacher scores are standardized by subject only.

17
while the other split is based on the odd-numbered test items 1, 3, 5, etc.<sup>15</sup> The even-odd split-half reliability score is a common measure of reliability used in the literature, similar to Cronbach’s alpha for example.<sup>16</sup> Appendix B derives how this works exactly, and under which assumptions.

It should be noted however that Cronbach’s alpha and the even-odd split-half reliability coefficients tend to _overestimate_ the true reliability of the test score data. More specifically, such estimators tend to assume that measurement error is independent across the difference items of the test. This is a strong assumption and most probably incorrect in most applications, including ours. If test-takers are tired for example, they will perform below their true potential on multiple items on the test, so that measurement error is positively correlated across the different test items. It is important to keep this caveat in mind.

Incorporating observed scores into equation (2) yields the value-added specification in terms of observed (scaled) test scores: where _esit_ , _asit_ and _esit−_ 1 are the measurement error components associated with _ysit_ , _xsit_ and _ysit−_ 1 respectively.

# **3.2.2 Noisy test scores and attenuation bias**

While section (3.2.1) deals with the issue of scaling in the presence of measurement error, this section deals with the second measurement error problem. The measurement error components, now part of the composite error term of equation (13), correlate with the structural components of the model, and cause biases. We can broadly distinguish two types of solutions to this problem. One is to first estimate the parameters _with bias_ , and correct estimates afterwards. The second is to account for potential biases directly in estimation. In this paper

> 15It is straightforward to show that the reliability of the raw (fraction correct) scores _Rsit_ is the same as the reliability of the scaled scores _ysit_ .

> 16Cronbach’s alpha is a construct that is very similar to the even-odd split half reliability coefficient, both in nature as in results. Cronbach’s alpha is the average of the correlation of all possible halve splits of the test. In our data, the values we obtain for Cronbach’s alpha and for the even-odd split half reliability coefficient are practically the same.

18
I follow the second approach.

The observed equivalent of the term ∆ _ysit_<sup>_∗_</sup> _−_ 1<sup>,whichwasintroducedasaninstrumental</sup> variable in section (3.1), is now no longer a good instrument. The observed lagged change score ∆ _ysit−_ 1 = _ysit−_ 1 _− ysit−_ 2 is positively correlated with the measurement error term _esit−_ 1, which is a component of the error term in (13). Using ∆ _ysit−_ 1 anyway, would yield attenuated parameter estimates of the _γ_ parameter in equation (13). The solution for this I propose in this paper makes use of the even-odd splits again. Instead of using _ysit−_ 1 as a right-hand-side regressor, we can also use _yE,sit−_ 1 (the scaled test score, using only the even-numbered items 2, 4, 6, etc. of the test). This changes the measurement error component in the error term: The idea is now to use ∆ _yO,sit−_ 1 = _yO,sit−_ 1 _− yO,sit−_ 2 (based on the odd-numbered testing items 1, 3, 5, etc.) as an instrument for _yE,sit−_ 1. ∆ _yO,sit−_ 1 is a good instrument under the assumptions introduced before, i.e. that ∆ _ysit_<sup>_∗_</sup> _−_ 1<sup>isagoodinstrument,andthatmeasurement</sup> error components are uncorrelated across the different items of the test.

The next problem is that observed teacher’s test scores _xsit_ correlate positively with _asit_ . During the time span of the study however, the teachers were tested a maximum of three times, i.e. if they were present during each of the three field visits. In the analysis we use a one period lagged teacher score _x_<sup>_t_</sup> _sit_<sup>_−_1as an instrumental variable for</sup><sup>_x_</sup> _sit_<sup>_t_, where the superscript</sup> indicates when test was fielded. It is important to notice that the lagged teacher’s test score _x_<sup>_t_</sup> _sit_<sup>_−_1is not the test score of last year’s teacher, but the test score of student</sup><sup>_i_’s current teacher,</sup> of which the score was obtained from a test that was fielded last year, at _t −_ 1. By contrast _x_<sup>_t_</sup> _sit_<sup>isthetestscoreofstudent</sup><sup>_i_’scurrentteacher,obtainedfromatestthatwasfieldedat</sup><sup>_t_.</sup> 19
# **4 The data, stylized facts and results from empirical valueadded models**

The data used in this study was collected for a randomized controlled field experiment with the objective to establish the causal effects of Indonesia’s teacher certification program, which included a doubling of a teacher’s base pay (De Ree, Muralidharan, Pradhan, and Rogers 2015). 360 schools were sampled using a two-step sampling strategy. First, 20 districts were selected from 10 broader geographical strata, and second, within each district, 12 primary and 6 junior secondary schools were selected.<sup>17</sup> The data on the 240 primary schools are used in this paper. Within these 240 schools, all class teachers ( _N ≈_ 1700) were tested and interviewed.<sup>18</sup> In addition, all students ( _N ≈_ 45 _,_ 000) in the 240 sample schools were tested as well. The multiple choice tests for teachers and students consisted of a mathematics, a science and an Indonesian language component.<sup>19</sup> At the back of the answering sheets, students also filled out a brief questionnaire on wealth indicators and on parent education.<sup>20</sup> Further selections on the data were made prior to the analysis. First, the data from one of the districts ( _Maluku Tenggara Barat_ ) were dropped from the analysis, because baseline data was collected about 6 months later as for the rest of the sample. Second, students who repeated class at least once are not used in the analysis. Third, because the statistical model requires three periods of measurement on test score data, we only rely on the grade levels which we can track for three consecutive years, i.e. 2 _→_ 3 _→_ 4, 3 _→_ 4 _→_ 5 and 4 _→_ 5 _→_ 6. Fourth, we only use classrooms for which we can match students to teachers, for which we have testing information for both students and teachers, and for which the class has a single teacher for

> 17See World Bank (2015) and De Ree, Muralidharan, Pradhan, and Rogers (2015) for more details on the selection of schools.

> 18In general, all “core subject teachers” were tested an interviewed, which are class teachers in primary schools, and mathematics, Indonesian language, Physics, Biology, and English language teachers in Junior secondary schools.

> 19Tests for teachers were different from tests for students, and grade appropriate tests were used to test students.

> 20The information on parent education was not used in this paper as it was found that the wealth indicators correlate more strongly to the student’s test scores than the information on parental education. One reason for this might be that young children might not know the education levels of their parents. About 90% of the students in our data reported to have a TV at home. They are however much less likely to have a fridge (50%) or a car (15%). Similarly, parents are quite likely to have some level of education, but generally not more than secondary school.

20
all three subjects: mathematics, Indonesian language, and science. A minority of the primary schools in the data organize instruction differently, for example, by having teachers teaching a single subject to grade 4, 5 and 6, similar to the common way of instruction in secondary schools.

# **4.1 Stylized facts**

In a preliminary analysis we find that teacher test scores do not correlate particularly strongly with some of the other observable teacher characteristics. Frequencies of some teacher characteristics are presented in figure (4).<sup>21</sup> <!-- Start of picture text --> 1<br>.8<br>.6<br>.4<br>.2<br>0<br>age +45 civil servant<br>university bachelor’s degree male<br>professionally certified<br><!-- End of picture text -->

**Figure 4:** Background characteristics of primary school teachers in the sample

Table 1 presents results of regressions of the scaled teacher test score _xsit_ on five background characteristics. The columns marked “pairwise”, columns (1), (3), (5), and (7), present results from univariate regressions, where the scaled test score _xsit_ is regressed on one background characteristic at the time. The columns marked “pooled”, the columns (2), (4), (6), and (8) present results from multivariate regressions where all five regressors are included at the same time. The results show that older teachers do worse and teachers with university bachelor’s degrees do better. Teachers in the civil service do not score better or worse, and the same is true for certified teachers.<sup>22</sup> An interesting pattern is also that male teachers tend to score

> 21Notice that our sample of teachers is not fully representative for the total population of teachers in Indonesia.

> 22Especially the latter could be seen as a problem, as it indicates that the roll-out of the certification program

21
better at the math and science components, while women score score better at Indonesian language.

Are these point estimates large or small? What would we expect a priori of a bachelor’s education for example? Of those without a bachelor’s degree, about two-thirds has a 2- year post-secondary diploma, and the rest, one-third, has only a secondary education. This heterogenous group scores .4 of a standard deviation below the group with a bachelor’s degree on average. Not very much perhaps. Another way of looking at this is by looking at the _R_<sup>2</sup> of these regressions. They are around 0.05, showing that only 5% of the variance of the (noisy) test score data is explained by these characteristics. Discounting the fact that test scores are noisy, we can estimate an alternative _R_<sup>2</sup> , measuring what the _R_<sup>2</sup> ’s would have been if the testing data would have been noise-free.<sup>23</sup> Even these alternative _R_<sup>2</sup> ’s are not very big, indicating that these background characteristics do not explain much of the variation in subject-matter knowledge of teachers. Apparently there are many teachers without university diplomas who do very well on the test, and, similarly, many teachers with university diplomas who do badly. This suggests that Indonesia’s teacher training programs – on average – do not manage to lift teacher subject-matter knowledge to a much higher level.

With more flexible regression models we can explain much more of the variation in test scores than the models presented in table 1. School fixed-effects models (where the teacher test score is regressed on a full set of school dummy variables) have _R_<sup>2</sup> ’s in the neighborhood of 0.60, indicating that 60% of the variance of (noisy) test scores can be explained at the school level. (We note that these school fixed-effects models are prone to overfitting, as the number of teachers per school is probably too small. I ran a bootstrap analysis to investigate the extent of the overfitting problem in this setup.<sup>24</sup> The findings indicate that a more conservative estimate of the _R_<sup>2</sup> of the school fixed-effects model would be 0.45 or 45%. Still much higher

> does not favor teachers with higher levels of subject-matter knowledge. See also World Bank (2015).

> 23The standard _R_ 2 is defined as the variance of the prediction, divided by the variance of the left hand side variable. The alternative _R_<sup>2</sup> reported in table 1 is calculated as the variance of the prediction, divided by 1, the variance of the true score.

> 24For the bootstrap, the teacher’s test scores are randomly, reallocated across teachers after which we estimate the school fixed-effects model again. This procedure is repeated 1,000 times and the _R_<sup>2</sup> ’s of these regressions are stored. I find that on average, these artificial regressions have _R_<sup>2</sup> ’s of around 0.15. The overfitting alone therefore already accounts for 15% of the variance in the test scores. These _R_<sup>2</sup> ’s however are much lower than the _R_<sup>2</sup> of 0.60 we find in the data, suggesting that there is a lot of real clustering of subject-matter knowledge in schools left.

22
**Table 1:** explaining teachers’ test scores – by subject

||math|score|Indonesi|an score|science|score|poole|d score|
|---|---|---|---|---|---|---|---|---|
||(1)<br>pairwise|(2)<br>pooled|(3)<br>pairwise|(4)<br>pooled|(5)<br>pairwise|(6)<br>pooled|(7)<br>pairwise|(8)<br>pooled|
|age +45 years|-0.216***|-0.292***|-0.429***|-0.419***|-0.249***|-0.366***|-0.325***|-0.393***|
|civil servant|-0.046|0.075|-0.387***|-0.208*|0.002|0.152|-0.154*|0.010|
|university degree|0.387***|0.359***|0.304***|0.273***|0.499***|0.457***|0.437***|0.400***|
|male teacher|0.200**|0.199**|-0.088|-0.083|0.242***|0.245***|0.134*|0.137**|
|teacher is certifed|0.087|0.082|0.002|0.161*|0.144|0.114|0.086|0.129|
|_R_<sup>2</sup>|.|0.036|.|0.041|.|0.049|.|0.061|
|alternative _R_<sup>2</sup>|.|0.059|.|0.078|.|0.095|.|0.081|

Note. Regressions of teacher test scores on background characteristics. The test scores are standardized, then rescaled as described in equation (12). By subject, the “pairwise” column presents results of univariate regressions (the test score regressed on a single characteristic). The “pooled” column presents the results of a multivariate regression with five regressors, age +45 years, civil servant, teacher has a university degree, male, and whether the teacher is professionally certified. These five regressors are all binary variables. Standard errors are robust to clustering at the school level. * significant at 10%; ** significant at 5%; *** significant at 1%.

than the _R_<sup>2</sup> ’s reported in table 1.) These findings indicate that there is a lot of clustering of high, and low levels of subject-matter knowledge in schools. Where in some schools all teachers tend to score higher, regardless of their degree or age, while in other schools all teachers tend to score lower. This is an indication that there is some important inequality of opportunity for Indonesian students.

Of particular interest for the empirical analysis is whether children from higher socioeconomic groups are more likely to attend these high-performing schools. Table 2 presents the relationship between socioeconomic characteristics of students and the mean test scores of their teachers. In column (1) the scaled teacher score is regressed on a student asset index<sup>25</sup> , and column (2) presents the results of a school fixed-effects model. Column (1) shows clearly that schools with the better, i.e. more knowledgeable, teachers also cater to wealthier families. Within schools, the column (2) results, we no longer observe this association, indicating

> 25The asset index counts how many of the following eight assets are available to the household: tv, fridge, hand phone, bicycle, motorcycle, car, computer, and children’s books. The parameter on the asset index, therefore, measures how many standard deviations a teacher’s subject-matter knowledge is higher on average, if its student has one more of these assets. The mean of the asset index is about 4.5 with a standard deviation of just below 2.

23
that there are no selection effects within schools.<sup>26</sup> The results here provide support for the relevance of the fixed-effects approaches discussed in the theoretical section (3). Students are certainly not randomly distributed across schools (or teachers) which is a clear threat to the validity of value-added models in levels: correlations between teachers’ subject-matter knowledge and student learning gains may be due, in part, to the fact that better teachers teach children with wealthier parents.

**Table 2:** testing for endogenous selection into schools

||pooled|school FE|
|---|---|---|
|asset index|0.067***|-0.004|
|_p_-value|0.003|0.359|

Note. Student level regressions where the teacher’s scaled test score (see equation (12)) is regressed on a student asset index. In the analysis I use the same data as for the value-added regressions presented in table 3, that is, endline data and grade level’s 4, 5 and 6. The asset index counts how many of the following eight assets are available to the household: tv, fridge, hand phone, bicycle, motorcycle, car, computer, and children’s books. The parameter on the asset index, therefore, measures how many standard deviations a teacher’s subject-matter knowledge is higher on average, if its student has one more of these assets. The mean of the asset index is about 4.5 with a standard deviation of just below 2. The “pooled” column (1) presents the pairwise regression of the teacher’s score on the asset index. The “school FE” column (2) present the results of a school fixed-effects model. Standard errors are robust to clustering at the school level. * significant at 10%; ** significant at 5%; *** significant at 1%.

# **4.2 Results from empirical value-added models**

Table 3 presents results of the three types of value-added models we proposed in section (3). Column (1) and (2) are the level models, column (3) and (4) are the school fixed-effects models, and column (5) and (6) are the student-teacher fixed-effects models. The column (1), (3) and (5) regression models include the lagged student score based on the even-numbered items in the test, i.e. _yE,sit−_ 1, while the change score ∆ _yO,sit−_ 1 based on the odd-numbered items, is one of the two excluded instruments. For the column (2), (4) and (6) results this is reversed, i.e. _yO,sit−_ 1 is used as a regressor and ∆ _yE,sit−_ 1 is the excluded instrument. The two sets of results should be fairly similar, and we find that they are in general.

> 26The argument here is that if there are no selection effects based on observables, it is plausible that there are also no selection based on unobservables.

24
|er fxed-efects<br>_yO,sit−_1 included<br>(6a)<br>(6b)|parameter<br>se<br>0.149*<br>0.090<br>.<br>.|.<br>.<br>.<br>.|.<br>.<br>.<br>.<br>.<br>.<br>.<br>.<br>0.328***<br>0.077|37.2<br>0.000<br>186<br>28392|∆_yE,sit−_1<br>_xt−_1<br>_sit_|tudent test scores are||
|---|---|---|---|---|---|---|---|
|Student-teach<br>_yE,sit−_1 included<br>(5a)<br>(5b)|parameter<br>se<br>0.201**<br>0.092<br>.<br>.|.<br>.<br>.<br>.|.<br>.<br>.<br>.<br>.<br>.<br>.<br>.<br>0.282***<br>0.075|45.2<br>0.000<br>186<br>28392|∆_yO,sit−_1<br>_xt−_1<br>_sit_|%. In the analysis, s||
|cluded<br>(4b)|se<br>0.082<br>0.050|0.004<br>0.082|0.097<br>0.005<br>0.007<br>0.180<br>0.040|5|_−_1|ant at 1||
|ed-efects<br>_yO,sit−_1 in<br>(4a)|parameter<br>0.195**<br>0.052|0.007*<br>-0.015|-0.091<br>-0.003<br>0.038***<br>0.123<br>0.512***|20.5<br>0.000<br>186<br>2842|∆_yE,sit_<br>_xt−_1<br>_sit_|; *** signifc|ubject only.|
|School fx<br>cluded<br>(3b)|se<br>0.078<br>0.050|0.005<br>0.082|0.104<br>0.005<br>0.008<br>0.168<br>0.049||_−_1|nt at 5%|ized by s|
|_yE,sit−_1 in<br>(3a)|parameter<br>0.222***<br>0.021|0.008*<br>-0.022|-0.077<br>-0.004<br>0.038***<br>0.023<br>0.424***|20.4<br>0.000<br>186<br>28425|∆_yO,sit_<br>_xt−_1<br>_sit_|; ** signifca|are standard|
|luded<br>(2b)|se<br>0.058<br>0.048|0.003<br>0.061|0.083<br>0.002<br>0.012<br>0.227<br>0.040||1|t at 10%|t scores|
|vel<br>_yO,sit−_1 inc<br>(2a)|parameter<br>0.256***<br>0.025|0.012***<br>-0.042|-0.015<br>0.000<br>0.052***<br>0.251<br>0.565***|79.9<br>0.000<br>186<br>28425|∆_yE,sit−_<br>_xt−_1<br>_sit_|el. * signifcan|nd teacher tes|
|Le<br>_yE,sit−_1 included<br>(1a)<br>(1b)|parameter<br>se<br>0.291***<br>0.062<br>0.003<br>0.050|0.014***<br>0.004<br>-0.039<br>0.062|-0.031<br>0.090<br>0.001<br>0.002<br>0.051***<br>0.014<br>0.165<br>0.213<br>0.507***<br>0.049|77.7<br>0.000<br>186<br>28425|∆_yO,sit−_1<br>_xt−_1<br>_sit_|ring at the school lev|e level and subject, a|
||e (_xt_<br>_sit_)||(even items)<br> (odd items)|||bust to cluste|(12)) by grad|
||’s subject-matter knowledg<br> has a bachelor’s degree|’s age<br> is certifed|is civil servant<br>ze<br> asset index<br> asset index is missing<br> student test score _yE,sit−_1 <br> student test score _yO,sit−_1|nderidentifcation test<br>t stage regressions<br> underidentifcation test<br>t stage regressions<br> of clusters (schools)<br> of obs. (student-subject)|d instruments|. Standard errors are ro|ardized (through formula|
||Teacher<br>Teacher|Teacher<br>Teacher|Teacher <br>Class si<br>Student <br>Student <br>Lagged <br>Lagged|_F_-stat u<br>of frs<br>_p_-value <br>of frs<br>Number<br>Number|exclude|Note|stand|

25
In the **level specifications** we find strong relationships with the teacher’s test scores. A standard deviation increase in teacher’s subject knowledge is associated with 0.25 to 0.29 standard deviations of additional learning on a year-to-year basis. Also we find that more senior teachers appear to do better. In this multivariate model there are no significant relationships with a teacher’s level of education, whether the teacher is certified, whether the teacher is a civil servant, and with the size of the class. Instead there is a strong relationship between learning, and a student’s socio-economic background. The parameter on the student’s lagged score, i.e. the persistence parameter _γ_ in equation (14), is estimated at around 0.5 in the level specification (this estimate is similar to estimates presented in the literature, e.g. by Andrabi, Das, Khwaja, and Zajonc (2011)).

The analysis in the previous section suggests that there is important endogenous selection within schools (Table 2). Children from wealthier backgrounds are more likely to enroll in schools where teachers have higher levels of subject-matter knowledge. And controlling for the socioeconomic background characteristics alone is probably not sufficient to account for the entire selection effect. In the school fixed-effects model, however, we still obtain sizable and statistically significant parameter estimates associated with teachers’ subject-matter knowledge. Because the data does not suggest that there are selection effects within schools, the school fixed-effects results are important. They suggest that _teachers with more subject-matter knowledge tend to be better teachers_ .

The same seems to be the case for older, more experienced teachers, although the age effect in the school fixed-effects model is only significant at the 10% level. Older teachers appear to do a bit better. It is beyond the scope of this paper to further explore this result. One important question is whether this “age effect” is truly an age effect, i.e. with age and experience, teachers become better. Or whether it is really a cohort effect, where older cohorts are simply better because, for example, the quality of teacher training has decreased over time. The latter explanation would be especially worrisome of course as it would suggest that when older more experienced teachers retire, the quality of instruction would decrease. More research is needed in this area.

Where the results of the school fixed-effects model indicate that teachers with more subject26
matter knowledge are better teachers, it does not prove that teachers are better _because_ they have more subject-matter knowledge. Based on videotaping Indonesian classrooms, Chang, Shaeffer, Al-Samarrai, Ragatz, De Ree, and Stevenson (2013), Ragatz (forthcoming) (and related work) find that teachers with higher levels of subject-matter knowledge also teach differently on average. And interestingly, it appears that the way teachers with more knowledge teach differently, relates directly to the additional knowledge. For example, Ragatz (forthcoming) argues: “techniques such as investigation, open-ended questioning, the use of mathematical language and symbols, and the use of non-routine problems and applications were found more often in classrooms with higher-knowledge teachers. This appears to be in part because the techniques themselves require a greater amount of subject mastery to be conducted effectively.” This finding lends weight to the idea that increases in the level of subject-matter knowledge among Indonesian teachers could lead to improved student outcomes. The student-teacher fixed-effects model evaluates the hypothesis empirically.

The **student-teacher fixed-effects** model attempts to disentangle the subject-matter knowledge of teachers from other, unobserved, teacher attributes, such as general intelligence, motivation, etc. The idea of the model is to only employ variation across subjects for the same student-teacher pair. Would it be, for example, that students with teachers who know more math than science, learn more math than science? The results from the student-teacher fixed-effects model confirms that subject-matter knowledge is important, even if we control for all non-subject specific student and teacher attributes. The point estimates are smaller in magnitude than in the level and school fixed-effects specifications. This is consistent with the idea that part of the reason why teachers with more knowledge are better, is because they are also more skillful along other dimensions. Perhaps naturally, the standard errors are larger for the student-teacher fixed-effects results, which, in combination with the size of the estimated effects, makes that the results are somewhat less robust overall.<sup>27</sup> The point estimates presented in this paper tend to be larger than comparable parameter estimates presented by Metzler and Woessman (2012). The point estimates we present here are closer to 0.2, while Metzler and Woessman (2012) present point estimates of around 0.1.

> 27Parameters for all other other fixed teacher and class attributes (age, education level, etc.) are not identified in the student-teacher fixed-effects model as they do not vary across subjects.

27
Another difference between this paper and Metzler and Woessman (2012) is that we also estimate the persistence parameter _γ_ based on variation across subjects only. The size of the point estimates for _γ_ tend to decrease from the level model to the school fixed-effects model to the student-teacher model.

# **4.2.1 Projected compounding effects of improvements in teacher subject-matter knowledge**

The primary interest of this research is to measure the counterfactual learning effects of improvements in teacher subject-matter knowledge. The parameter estimates presented in table 3 measure the input effects on a year-to-year basis. But nation-wide efforts to increase teacher subject-matter knowledge will imply that students have better teachers for two, three years in a row, or throughout their entire schooling career. The overall effect on student-learning outcomes, therefore, depends on the strength of the input effect but also on the rate at which past achievement gains persist into the future.

Within the context of our model we predict what would happen to student outcomes in response to an increase in teachers’ subject-matter knowledge of 1.0 and of 2.0 standard deviations across-the-board. Figure (5) shows projections based on the results from the studentteacher fixed-effects model. For the projections I took the average of the parameter estimates of column (5) and (6), i.e. _β_ ˆ _≈_ 0 _._ 175, and _γ_ ˆ _≈_ 0 _._ 3. Students benefit most (as measured relative to the distribution of performance in the current population) in the first year after an improvement in teachers’ subject-matter knowledge. The increase is smaller in the second year as the second year’s teacher receives the student already at a higher starting level. Students who benefit year-after-year from teachers with higher levels of subject-matter knowledge practically reach the new equilibrium level at the end of grade 3. (Note that you need teachers with higher levels of subject-matter knowledge also in grade 4, 5 and 6, to maintain higher relative test scores. Otherwise the student would fall back again.) At the end of the primary cycle, test scores have increased by 0.25 and 0.50 student-level standard deviations respectively. These effects are substantial. PISA reports for example that the standard deviation of PISA scores for Indonesian 15-year-olds is roughly 70 PISA points in 28
<!-- Start of picture text --> .7<br>.6<br>.5<br>.4<br>.3<br>.2<br>.1<br>0<br>0 1 2 3 4 5 6<br>grade level<br>one standard deviation increase in teacher knowledge<br>two standard deviations increase in teacher subject knowledge<br>standard deviations<br><!-- End of picture text -->

**Figure 5:** Projected responses across grade levels in response to 1.0 and 2.0 standard deviation increases in teachers’ subject-matter knowledge

the population. 0.25 or 0.50 of that would correspond to effects of 17.5 and 35.0 PISA points respectively.<sup>28</sup> Especially, an increase of 35.0 PISA points would be remarkable considering Indonesia’s past performance on PISA. Indonesia’s long term performance on the mathematics component of PISA is about 375. With an additional 35 points Indonesia would break through the 400-point barrier, and close more than half of the current learning gap between Indonesia and its regional peers in PISA, Malaysia (with a score of 421) and Thailand (with a score of 427). Notice that section (2) shows that both 1.0 and 2.0 standard deviation increases in subject-matter knowledge of primary teachers appear to be achievable medium-term goals for education policy making in Indonesia.

# **5 Conclusion**

This paper makes the case that subject-matter knowledge of Indonesian primary school teachers is low, in the sense a large percentage of them has difficulties with mathematics problems from the high-school curriculum. Based on a school fixed-effects model and a (more general) student-teacher fixed-effects model, the paper also finds empirical support for the idea that

> 28One caveat here is for finding these differences in performance at age 15, the year when students finish junior secondary school, we also need junior secondary teachers to improve relative to where they are now.

29
realistic improvements in teachers’ subject-matter knowledge could lead to meaningful improvements in the quality of instruction in Indonesia and the learning levels of its students. It is estimated that a 1.0 (2.0) standard deviation increase in teacher’s subject knowledge can translate into a 0.25 (0.50) student-level standard deviation increase in the learning levels of students by the time they leave primary school.

The results of this paper suggest that policies which incentivize improvements in subjectmatter knowledge can work in Indonesia. This finding starkly contrasts results from experimental research about Indonesia’s recent, and very costly, teacher certification program (De Ree, Muralidharan, Pradhan, and Rogers (2015) and World Bank (2015)). Indonesia’s certification program was implemented in 2005/06. In a nutshell, the certification program required a university bachelor’s degree as minimum academic qualifications for primary teachers, and promised a generous _doubling of take-home pay_ after passing the certification program successfully. While the certification program led to hundreds of thousands of teachers starting course work to obtaining bachelor’s degrees, no important improvements in education quality was observed. The stability of the PISA scores is an example of this, but World Bank (2015) provides additional _micro level_ evidence.

The 2012 PISA results perhaps marked the starting points of redirecting policies more explicitly geared towards quality. Indonesia’s current minister of Education and Culture Anies Baswedan for example reflected on Indonesia’s most recent 2012 PISA results in _Republika_ , one of Indonesia’s mainstream daily newspapers, arguing that “Indonesia’s education is in a state of emergency” (Novia 2014). The research presented in this paper indicates that future policies to promote quality education could benefit from incorporating subject-matter tests in teacher evaluation systems.

30
# **A Binary choice models with noisy regressors**

## **A.1 The model with clean regressors**

Consider the following model for the continuous latent variable _qk_<sup>_∗_:</sup> where _εk|x_<sup>_∗_</sup> _∼ N_ (0 _, σε_<sup>2</sup> _k_ ), where _x_<sup>_∗_</sup> is a test-taker’s true level of subject-matter knowledge, standardized to have mean 0 and a variance of 1 in the population.

Test-takers answer correctly to item _k_ ( _Qk_ = 1) if the latent variable _qk_<sup>_∗>zk_.</sup> The probability of answering correctly, given a certain level of subject-matter knowledge _x_<sup>_∗_</sup> is then: where Φ ( _·_ ) is the standard normal CDF. A probit model based on the clean regressor _x_<sup>_∗_</sup> , therefore provides estimates of the following composite parameters:<sup>_<u>ak</u>_</sup> _σ_<sup>_−_</sup> _εk_<sup>_<u>zk</u>_</sup> and _σβεkk_<sup>.Thenext</sup> section A.2 how we can recover these parameters from a probit analysis based on the noisy regressor _x_ .

## **A.2 The model with noisy regressors**

The measurement error model is: _∗_ I assume that measurement error is “classical”, i.e. _E_ [ _e_ �� _x_ ] = 0.

The noise term _e_ can be divided into two orthogonal components. One that has a corre31
lation of 1 with the noisy variable _x_ and one that is orthogonal to it:

> <sup><u>(</u></sup><sup>_e_</sup><sup><u>)(</u></sup><sup>_x∗_</sup><sup><u>)</u></sup> It is not difficult to show that _γ_ =<sup>_C_</sup> _V_<sup><u>(</u></sup> (<sup>_e_</sup> _x_<sup>_<u>,x</u>_</sup> )<sup><u>)</u>=</sup><sup>_C_</sup><sup><u>(</u></sup><sup>_e_</sup> _V_<sup>_<u>,</u>_</sup> (<sup>_x_</sup> _x_<sup>_∗_</sup><sup><u>+</u></sup> )<sup>_e_</sup><sup><u>)</u></sup> = _V_<sup>_V_</sup> ( _x_ )<sup>=1</sup><sup>_−ρ_,where</sup><sup>_ρ_=</sup><sup>_V_</sup> _V_ ( _x_ )<sup>isthe</sup> well-known coefficient of reliability. Equation (21) can therefore be written as:

where _E_ [ _x_ ] = _E_ [ _u_ ] = 0, and _C_ ( _x, u_ ) = 0, by definition.

Incorporating (19) and (21) in the original latent variable model (15) yields: The derivations so far suggest that with binary choice models with noisy regressors we obtain similar attenuation biases as what we typically get in linear models with noisy regressors (i.e. parameters on the noisy regressor are attenuated at a rate equal to _ρ_ ). But with binary choice models something else is happening which relates to the normalization of the model.

Based on (23), the new model for the latent variable _qk_<sup>_∗_, we can derive a new probit model:</sup> where I make the additional assumption that _−uβk_ + _εk_ �� _x ∼ N_ (0 _, βk_<sup>2</sup><sup>_σ_</sup> _u_<sup>2+</sup><sup>_σ_</sup> _ε_<sup>2</sup> _k_ ).

Using (19) and the definition of reliability, we can derive that _V_ ( _e_ ) = (1 _− ρ_ ) _V_ ( _x_ ), and by using (21), we can derive that _V_ ( _e_ ) = _V_ ( _x_ ) (1 _− ρ_ )<sup>2</sup> + _V_ ( _u_ ). Combining these yields the 32
following expression for _V_ ( _u_ ): which can be simplified, because in our normalization _V_ ( _x_ ) = 1 _/ρ_ .

Combining (28) with (26) yields: The constant and the parameter of a regression based on the noisy variable _x_ are: The two equations can be solved for for<sup>_<u>αk</u>_</sup> _σ_<sup>_−_</sup> _ε_<sup>_<u>zk</u>_</sup> and<sup>_<u>β</u>_</sup> _σ_<sup>_<u>k</u>_</sup> _ε_<sup>.</sup> Solving for<sup>_<u>β</u>_</sup> _σ_<sup>_<u>k</u>_</sup> _ε_<sup>yields:</sup> Then solving for<sup>_<u>αk</u>_</sup> _σ_<sup>_−_</sup> _ε_<sup>_<u>zk</u>_</sup> yields: 33
# **B Even-odd split-half reliability coefficients**

First the totality of the test items are divided into two sets, those based on the even-numbered test items 2, 4, 6, etc. and those based on the odd-numbered items 1, 3, 5, etc. Based on the two sets we can construct two raw scores for the same test-taker _Rsit_<sup>_E_and</sup><sup>_R_</sup> _sit_<sup>_O_.Itisnow</sup> assumed that both scores approximate the true raw score _Rsit_<sup>_∗_, each with its own measurement</sup> error component: It is furthermore assumed that _E_ [ _eRsitE_ �� _Rsit∗_ ] = _E_ [ _eRsitO_ �� _Rsit∗_ ] = 0 and _E_ [ _eRsitE_<sup>_× eR_</sup> _sit_<sup>_O_</sup> �� _Rsit∗_ ] = 0. Especially the latter (both measurement error components are uncorrelated with each other) is a strong assumption.

Based on these assumptions we can derive: The correlation of the scores based on even items and odd items, is approximately equal (under assumptions) to the reliability of a test score based on half of the test items (either the even or the odd ones). Tests based on less items are more noisy, and less reliable. We should therefore “upscale” the estimate to represent the reliability of the totality of test items (even 34
and odd-numbered items combined).

This can be rewritten: which can be rewritten into a formula for the the reliability of the complete test: This formula is also known as the Spearman-Brown prophecy formula.

35