metadata: authors: 0: author: Al-Samarrai,Samer
1: author: Shrestha,Unika
2: author: Hasan,Amer
3: author: Nakajima,Nozomi
4: author: Santoso,Santoso
5: author: Adi Wijoyo,Wisnu Harto
count: Indonesia
docna: 0: docna: Introducing a performance-based school grant
            in Jakarta : what do we know about its impact after two
            years ?
docty: Policy Research Working Paper
subtopic: Macro-Fiscal Policy,Economic Adjustment and Lending,Public Sector Economics,Urban Governance and Management,Public & Municipal Finance,Educational Sciences,Educational Institutions & Facilities,Effective Schools and Teachers,Municipal Management and Reform,Intergovernmental Fiscal Relations and
            Local Finance Management,Public Finance Decentralization and
            Poverty Reduction
teratopic: Education
abstracts: cdata!: This paper evaluates the early impact of
            introducing a performance component into Jakarta's
            school grant program on learning outcomes. Using
            administrative data, it applies difference-in-differences
            and regression discontinuity approaches to identify the
            impact of the grant by exploiting differences in program
            coverage over time, as well as by comparing changes in test
            scores between schools that received the additional
            performance award with schools that did not. The paper finds
            that the introduction of the performance component had
            different impacts on government primary and junior secondary
            schools. The program improved learning outcomes for primary
            schools at the bottom of the performance distribution and
            narrowed performance gaps across schools. However,
            improvements in equity were also driven by negative impacts
            of the program on better performing primary schools.
            Overall, the program reduced primary examination scores
            albeit by a small amount. In contrast to the results at the
            primary level, the performance component improved
            examination scores in government junior secondary schools.
            However, the impact seemed to be greatest among better
            performing schools and has therefore widened performance
            gaps. The findings also suggest that program impact was
            largely through competition between schools to receive the
            performance component. There is little evidence that the
            additional resources schools received from the award had any
            additional impact. The evaluation utilized preexisting
            administrative data and the paper offers some suggestions on
            how education information systems can be strengthened to
            create more robust feedback loops between research and policy.
geo_reg_key: 517198,517191
repnme: repnme: Introducing a performance-based school
            grant in Jakarta : what do we know about its impact after
            two years ?
display_title: Introducing a performance-based
            school grant in Jakarta : what do we know about its impact
            after two years
listing_relative_url: /research/2017/10/28098524/introducing-performance-based-school-grant-jakarta-know-impact-after-two-years
topicv3: Teachers Management m1326119 1523,Teacher Pay and Incentives m1326551 1530,Education m1327048 470,Performance-Based Fiscal Transfer m1327613 1723,School-Based Management m1327730 2143
publishtoextweb_dt: 2017-10-23T00:00:00Z
historic_topic: Education
no_of_pages: 43
author_keys: 0: author_key: 455096
1: author_key: 1319544
2: author_key: 807987
3: author_key: 1320512
4: author_key: 1501201
model_extractions: WPS8223 Policy Research Working Paper 8223 Background Paper to the 2018 World Development Report Introducing a Performance-Based School Grant in Jakarta What Do We Know about Its Impact after Two Years?

_Samer Al-Samarrai Unika Shrestha Amer Hasan Nozomi Nakajima Santoso Santoso Wisnu Harto Adi Wijoyo_ Education Global Practice Group October 2017
Policy Research Working Paper 8223

# **Abstract**

This paper evaluates the early impact of introducing a performance component into Jakarta’s school grant program on learning outcomes. Using administrative data, it applies difference-in-differences and regression discontinuity approaches to identify the impact of the grant by exploiting differences in program coverage over time, as well as by comparing changes in test scores between schools that received the additional performance award with schools that did not. The paper finds that the introduction of the performance component had different impacts on government primary and junior secondary schools. The program improved learning outcomes for primary schools at the bottom of the performance distribution and narrowed performance gaps across schools. However, improvements in equity were also driven by negative impacts of the program on better performing primary schools. Overall, the program reduced primary examination scores albeit by a small amount. In contrast to the results at the primary level, the performance component improved examination scores in government junior secondary schools. However, the impact seemed to be greatest among better performing schools and has therefore widened performance gaps. The findings also suggest that program impact was largely through competition between schools to receive the performance component. There is little evidence that the additional resources schools received from the award had any additional impact. The evaluation utilized preexisting administrative data and the paper offers some suggestions on how education information systems can be strengthened to create more robust feedback loops between research and policy.

This paper—prepared as a background paper to the World Bank’s _World Development Report 2018: LEARNING to Realize Education’s Promise_ —is a product of the Education Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at salsamarrai@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Introducing a Performance-Based School Grant in Jakarta: What Do We Know about Its Impact after Two Years?**

Samer Al-Samarrai<sup>*</sup> , Unika Shrestha<sup>*</sup> , Amer Hasan<sup>*</sup> , Nozomi Nakajima<sup></sup> , Santoso Santoso<sup>*</sup> and Wisnu Harto Adi Wijoyo<sup>*</sup>

**JEL classification codes:** C21 **,** D73, I22, I28

**Keywords:** Education, quality of education, results-based management in the public sector, education funding formulae

* Education Global Practice, World Bank Group, 1818 H Street, NW, Washington, DC 20433

>  Harvard Graduate School of Education, Harvard University, <mark>13 Appian Way, Cambridge, MA 02138 Acknowledgements: The data on the UN examination scores was provided by PUSPENDIK (Pusat Penilaian Pendidikan) of the Ministry of Education and Culture (MoEC). The authors are grateful to Professor Ir Nizam and Dr A. Suprananto of PUSPENDI for providing and aiding the team’s understanding of this data. The data on education indicators for metropolitan areas was provided by the PDSPK (Pusat Data Statistik Pendidikan dan Kebudayaan) team in MoEC. The authors acknowledge the enormous amount of help provided by Dr. Ir. Bastari, Ir. Siti Sofiah, and Watik Sudarwati. Information on school grants and additional information on Jakarta schools was provided by UPT PDSIP Disdik of the Jakarta government. The authors are grateful to Kadarwati Mardiutama in facilitating this process and to Harry Patrinos and Shwetlena Sabarwal for providing comments on an earlier draft of the paper. The authors gratefully acknowledge financial support from the REACH Trust Fund at the World Bank. Corresponding author - Samer Al-Samarrai (salsamarrai@worldbank.org). The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.</mark>
# **<mark>1. Introduction</mark>**

<mark>Children growing up in many developing countries today start school earlier and stay longer than their parents and grandparents. Average levels of educational attainment in developing countries rose from 4 years to 7 years between 1980 and 2010 and over the same period many countries also narrowed the attainment gap with more advanced nations (Lee and Lee 2016). These improvements have come about through significant increases in government investment in education. For example, government spending on education in low- and middle- income countries doubled in real terms between 2000 and 2014 (Education Commission 2016).</mark> <mark>A substantial proportion of these additional resources have been allocated to building and equipping new schools and hiring teachers. Increased funding, has also been used, particularly in basic education, to reduce the costs of schooling through targeted cash transfer schemes and school grants. Typically, governments have provided grants as part of fee reduction policies and to compensate schools for the associated revenue loss.</mark> <mark>However, despite increased spending, student learning outcomes remain low in many countries. Recent evidence shows that in many countries, overall increases in public education spending are only loosely related to improved learning (OECD 2013). Moreover, impact evaluation studies across different countries have also shown that increases in spending that merely provide more school level inputs frequently fail to improve learning (Glewwe, Hanushek et al. 2011, McEwan 2013).</mark> <mark>Indonesia is no exception to these broader global trends. Between 2000 and 2012, the number of years of schooling the average adult obtained increased from 6 to 8 years (World Bank 2013). The government tripled education spending in real terms between 2001 and 2011 and used part of these increases to introduce a nationwide school grants program in 2005.</mark><sup>1</sup> <mark>While attainment has improved, the 2015 OECD PISA results show that almost 70 percent of Indonesian students fell below the basic level of proficiency in mathematics required to participate fully in modern society (OECD 2016). Moreover, there has been no significant change in learning outcomes since 2006.</mark> <mark>In the nation’s capital, Jakarta, the low returns from large increases in public education spending have led to a renewed focus on the effectiveness of school financing. For many years, the Jakarta city government has provided per-student operational grants to support school level improvement. However, a recent public education expenditure review highlighted the limited impact of this additional funding on school performance (World Bank 2014). In response to these findings, the government adjusted their operational grant allocation formula in 2014 to include a component that linked school grants to performance.</mark> <mark>This paper evaluates the impact of the introduction of this performance component of the grant on learning outcomes. It identifies the impact by exploiting geographic differences in program coverage as well as by comparing schools that received the additional performance award with schools that did not.</mark>

> 1 Per student spending in primary and lower secondary schools also increased over this period and has continued to grow UIS (UNESCO Institute for Statistics) (2017). “Education (database).”. Montreal **:** http://data.uis.unesco.org/.

2
The paper finds that student learning outcomes improved as a result of the announcement of the new component in all government junior secondary schools. Despite efforts to ensure a level playing field, high performing junior secondary schools were much more likely to receive the performance award than low performing schools. Moreover, the performance improvements required to secure additional funding were much lower for schools that were already highperforming (compared to low-performing schools). These program design features appear to have created stronger incentives for improvement among high performing junior secondary schools and the program appears to have inadvertently increased inequality in school performance at the junior secondary school level. In contrast, the program had a small negative effect on government primary schools in Jakarta. However, primary schools that performed poorly prior to the introduction of the program registered a small but significant increase in examination scores. While the paper finds impacts of the announcement of the performance-based component of the grant on some primary and all junior secondary government schools, it finds no additional impact of the additional funds associated with the performance-based component of the grant on the 25 percent of government schools that actually received it.

<mark>The next section provides a short review of the literature that has explored the impact of school grants on education outcomes – particularly in developing countries. Section 3 provides a description of the Jakarta school grants and Section 4 outlines the main research questions, describes the empirical strategy and data employed. The main results are reported in Section 5 and the final section provides some suggestions on how the program could be strengthened further.</mark>

# **<mark>2. Literature Review</mark>**

<mark>School grants and similar mechanisms that provide discretionary funds to schools exist in many education systems but their objectives and characteristics vary widely.</mark><sup>2</sup> <mark>Grants are sometimes used to compensate schools for lost revenue when policies to abolish fees have been introduced to raise the demand for schooling. They have also been used as part of broader school based management reforms and have provided schools with a reliable source of funding to implement their own improvement plans. The size and coverage of school grants also vary. In some cases, grants can be used to cover all school operating expenses (e.g. teacher salaries, utilities, infrastructure) while in other cases the use of grants is heavily circumscribed.</mark> <mark>While these differences make it difficult to generalize about the impact of school grants, the available evidence does suggest that they have been successful at improving participation (Table 1). For example, the introduction of grants in Niger and Uganda improved the chances of children enrolling in primary school (Grogan 2009, Beasley and Huillery 2013). Grants have also improved indicators of student progress and retention. Evaluations of two Mexican programs that provided parent association-managed grants found that they reduced student drop-out and repetition rates (Skoufias and Shapiro 2006, Gertler, Patrinos et al. 2012).</mark>

> 2 For the purposes of this paper, school grants are funds provided directly to schools that authorities at the school level have some discretion over. Grants are usually from public sources and exclude school income from fees and contributions by parents.

3
<mark>Evidence on the impact of school grants on learning outcomes is more mixed. The studies summarized in Table 1 where grants are introduced on their own show little impact on learning. For example, in India and Zambia, parents lowered their own contributions to schools in anticipation of schools receiving the grant. This reduced the impact of the grant on school revenues and limited the additional activities that schools could finance to improve learning (Das, Dercon et al. 2013). However, the impact of grants on learning has been more promising when they have been combined with other interventions. For example, in Indonesia grants on their own had no impact but improved learning outcomes when combined with interventions that strengthened school oversight (Pradhan, Suryadarma et al. 2014). Similarly, in Tanzania, when school grants were combined with teacher incentives related to student performance, learning outcomes improved (Mbiti, Muralidharan et al. 2015).</mark> <mark>A review of the literature did not uncover any evaluations where grant payments were linked directly to school performance. A study in Senegal where grants were allocated to schools competitively showed that linking decisions about grant allocation to school outcomes had the potential to raise student learning outcomes (Carneiro, Koussihouèdé et al. 2016). However, there have been no assessments of grants that are fully or partly allocated on past school performance. This paper aims to fill this gap by evaluating the early impacts of the Jakarta program which directly links school grant amounts to school performance on the national examination.</mark> 4
**<u>Table 1: Summary of recent evaluations of school grants</u>**

|Country|Education level|Outcomes evaluated|Was the grant bundled<br>withother interventions?|Effect|Studies|
|---|---|---|---|---|---|
|**Mexico**|Non-indigenous<br>public primary<br>schools|Drop-out, repetition,<br>failure to pass grade|Yes. Includes other<br>support to parent<br>associations.|Negative and statistically significant effect on drop-out and<br>repetition with larger effects in Grades 1-3. No significant<br>effecton failurerates.|Gertler, Patrinos<br>et al. (2012)|
|**Mexico**|Public primary<br>schools|Drop-out, repetition,<br>failure to pass grade|Yes. Includes<br>decentralized<br>management.|Negative and statistically significant effect on drop-out,<br>repetition and failure rates.|Skoufias and<br>Shapiro (2006)|
|**Niger**|Public primary<br>schools|Enrolment and drop-<br>out|Yes. Support and<br>training of school<br>committees.|Statistically significant positive effect on male and female<br>enrolment in Grade 2. Negative effect on female drop-out for<br>Grade2.|Beasley and<br>Huillery (2013)|
|**Uganda**|Primary schools|Probability a child<br>enrolls before age 9|No but school grants tied<br>to abolition of school<br>fees.|Statistically positive effect on probability of enrolment.|Grogan (2009)|
|**Gambia**|Lower basic<br>public and<br>government-<br>aided schools|Student and teacher<br>attendance, numeracy<br>and literacy test<br>scores|No but also looked at<br>school grants combined<br>with other interventions.|Statistically significant improvements in teacher and student<br>attendance after 3 years. No effect on learning measures.|Blimpo and Evans<br>(2011)|
|**Philippines**|Public elementary<br>schools|English, Mathematics<br>and Science test<br>scores|Yes. Training and school<br>based management<br>interventions.|Statistically significant and positive effect on all test scores.|Khattri, Ling et al.<br>(2010); Yamauchi<br>(2014)|
|**Indonesia**|Rural public<br>primary schools|Drop-out, repetition<br>and mathematics and<br>language test scores|No. Also looked at<br>school grants combined<br>with other interventions.|No statistically significant effect on drop-out and repetition<br>rates or on learning outcomes for grants on their own.<br>Statistically significant effect on language test scores for<br>grants combined with (a) links to village councils and (b) links<br>to councils and elections. Grants and links to village councils<br>also significantforgirls’ mathematics scores.|Pradhan,<br>Suryadarma et al.<br>(2014)|
|**India and**<br>**Zambia**|Public primary<br>schools (rural in<br>India)|Mathematics and<br>language test scores|No.|Statistically significant improvements in student learning<br>from unanticipated school grants. No effect when grants are<br>anticipated by parents.|Das, Dercon et al.<br>(2013)|
|**Senegal**|Primary schools|Mathematics, French<br>and Oral test scores in<br>Grades 3 and 5|No. School grants<br>allocated competitively.|Statistically significant improvements in Grade 3 French and<br>Oral test scores.|Carneiro,<br>Koussihouèdé et<br>al. (2016)|
|**Tanzania**|Primary schools|Mathematics, English<br>and Kiswahili test<br>scores in Grades 1-3|No. Also looked at<br>school grants combined<br>with teacher incentives.|No effect on test scores of provision of grant alone.<br>Statistically significant improvements in test scores for<br>mathematics and Swahili in second year when combined with<br>teacher incentives.|Mbiti,<br>Muralidharan et<br>al. (2015)|

Note: Studies included in Table 1 were mainly identified from past reviews of impact evaluations (McEwan 2013, Snilstveit, Stevenson et al. 2015).

5
# **3. Education sector funding and the Jakarta school grants program**

In Indonesia, decentralization in the early 2000s devolved responsibility for primary and secondary schools to provincial and district governments. Local governments account for over 60 percent of public education sector spending. The central government supplements local government funding through a range of national programs including a large school grant scheme introduced in 2005 – the _Bantuan Operasional Sekolah_ (BOS) program. The BOS program provides funding to all primary and junior secondary schools on the basis of a fixed amount per student. Schools have strict limits on the amounts they are allowed to collect from parents which makes schools heavily reliant on BOS funds. In 2010, BOS funds accounted for 83 percent of all discretionary funding that primary schools received (World Bank 2012). School funding from the BOS program has increased significantly since its introduction and in 2014 the average primary school received approximately US$ 10,000 a year (World Bank 2015).<sup>3</sup> In 2005, the Jakarta government introduced a school grants scheme modeled on the national BOS program. Initially, the program only covered government and non-government primary schools. However, the per-student amount of IDR 240,000 (US$ 25) received by primary schools was about 50 percent more than junior secondary schools received from the national program. As the program evolved, per-student funding levels increased and in 2007, the program expanded to cover all government junior and senior secondary schools. In 2014, per-student funding for junior secondary schools was IDR 1.3 million (US$ 111) compared to only IDR 710,000 (US$ 59.8) from the national BOS program. However, by this time the Jakarta government had withdrawn grant support for non-government schools.<sup>4</sup>

**Figure 1: National Examination Scores for Junior Secondary Schools (SMP) by Province (%), 2010-13**

<!-- Start of picture text --> 100<br>80<br>60<br>40<br>20<br>0<br>BANGKA BELITUNG NUSA TENGGARA... KALIMANTAN TIMUR KALIMANTAN BARAT KEPULAUAN RIAU JAWA TENGAH KALIMANTAN TENGAH SULAWESI TENGAH SUMATERA BARAT DI YOGYAKARTA PAPUA BENGKULU GORONTALO JAWA BARAT SULAWESI BARAT PAPUA BARAT MALUKU UTARA SULAWESI TENGGARA INDONESIA BANTEN LAMPUNG MALUKU RIAU DKI JAKARTA KALIMANTAN SELATAN NUSA TENGGARA... SULAWESI UTARA SULAWESI SELATAN ACEH JAMBI JAWA TIMUR SUMATERA SELATAN BALI SUMATERA UTARA<br><!-- End of picture text --> Source: World Bank (2014).

> 3 Roughly equivalent to the salary of two certified civil service teachers.

> 4 Between 2005 and 2013 school grants were provided to all schools except in 2011 where only government schools were beneficiaries. Since 2014, non-government schools are only included in government-financed scholarship programs for poor students.

6
Concerns over the quality of education and effectiveness of government education spending resulted in significant changes to the school grants program. Despite public and private spending levels considerably higher than most other provinces, schools in Jakarta only ranked in the middle of the national examination distribution (Figure 1). In 2014, Jawa Timur achieved comparable examination results as Jakarta but spent only half as much per student (World Bank 2014). These large differences in spending efficiency led the Jakarta government to introduce a performance component to their school grant program to tie funding more closely to performance.<sup>5</sup> The performance component of the grant was announced in 2014 and gave the best performing schools an additional per student allocation equivalent to 20 percent of the basic grant allocation (Table 2).<sup>6</sup> Performance was judged along two dimensions: average examination performance over the last two years (2013 and 2014) and the percentage point improvement in performance over the same period. The ranking of schools along these two dimensions was averaged and schools in the top quartile (25%) were awarded the performance component grant in the following year (2015).<sup>7</sup> For primary and junior secondary schools, the ranking exercise was conducted separately in each of Jakarta’s six districts to incentivize more schools and make the scheme more equitable. In particular, the district ranking exercise ensured that schools in catchment areas serving students from similar backgrounds were competing against each other rather than schools in more affluent parts of the city.

**Table 2: Grant funding formula and the number of government schools in Jakarta by district, 2015**

||**Primary**<br>**(SD)**|**Junior**<br>**Secondary**<br>**(SMP)**|**% of schools**<br>**that receive**<br>**allocation**|
|---|---|---|---|
|**Monthly per-student value o**|**f grant compo**|**nent IDR 000s (USD)**||
|Basic allocation|60 (4.5)|110 (8.2)|100|
|Performance<br>allocation|12 (0.9)|22 (1.6)|25|
|Equity allocation*|12 (0.9)|22 (1.6)|1|
|**Total number of schools**||||
|Jakarta Barat|361|50|-|
|Jakarta Pusat|203|36|-|
|Jakarta Selatan|375|65|-|
|Jakarta Timur|479|95|-|
|Jakarta Utara|197|38||
|Kepulauan Seribu|14|7|-|
|**Average school enrolment**|367|742|-|

Notes: Exchange rate for 2015 of IDR 13,389 to the US$ used to convert grant amounts from World Development Indicators database. Average enrolment data is for 2015 except in a small number of schools where information for

> 5 At the same time, an equity component was introduced to provide greater funding to schools in Kepulauan Seribu (Thousand Islands) that faced significantly higher operational costs given their remote location.

> 6 Senior secondary schools and madrassahs were also eligible for performance grants. However, due to data limitations the impact of the program in these institutions is not analyzed in the paper.

> 7 Which schools received the performance component was determined in August/September once the school year had ended and the examination results were published. Schools that were successful were given the additional funds in the following budget year which ran from January to December.

7
2014 or 2016 has been used. * Given to Kepulauan Seribu only. Source: Monthly per student value of grant and number of schools reported from Jakarta government education management information system.

The introduction of the performance component was designed to create stronger incentives at the school level to use resources more effectively to improve learning. Historically, schools have used a significant proportion of their discretionary funds to hire contract teachers (World Bank 2015). Unlike in some other countries, however, the hiring of additional teachers has not been associated with improvements in learning partly because student teacher ratios and class sizes were already relatively low (World Bank 2013). The performance component was designed to align the schools’ use of resources more centrally with learning.

At the time of its design, it was assumed that the performance component would improve learning outcomes through two main channels. First, the announcement of the grant alone was expected to increase effort among teachers and other actors to improve levels of learning. For example, the announcement of the grant was expected to raise teacher effort through increases in teacher attendance, greater time on task during lessons and greater lesson preparation. Second, it was predicted that the introduction of the grant would encourage schools to align their funding more closely to the objective of improved learning. It was also expected that learning outcomes would be further enhanced for schools that received the grant because additional activities to raise learning could be supported.

**Table 3: Government schools receiving 2015 performance component and average** **<u>improvement in performance</u>**

||School perfor<br>exa<br>|mance quartiles<br>minationscore|based on aver<br>in 2013 and2|age of national<br>014<br>||
|---|---|---|---|---|---|
||Bottom|||Top|All|
||performance<br>quartile|Quartile 2|Quartile 3|<br>performance<br>quartile|schools|
|**Average examination score**|**2013/14: all government**|**schools**||||
|Primary|63|71|76|83|74|
|Junior Secondary|70|73|76|83|76|
|**Percentage of government sc**|**hools in each quartile r**|**eceiving perfo**|**rmance grant**|**in 2015**||
|Primary|3|12|26|62|26|
|Junior Secondary|0|6|19|75|25|
|**Average examination score**|**2013/14: only schools re**|**ceiving perfor**|**mance grant i**|**n 2015**||
|Primary|66|72|77|84|80|
|JuniorSecondary|-|73|77|83|82|

Note: Average scores and percent of grant recipients in 2015 were calculated using UN scores data from Puspendik and performance grant data from Jakarta government. Excludes schools in Kepulauan Seribu district.

Despite efforts during design to give all schools an equal opportunity to compete, the data show that schools with higher overall performance at the outset were more likely to get the performance grant. The decision to include the percentage point change in examination scores was taken to encourage low performing schools to compete for the grant. Without this, only high performing schools would receive the grant. Due to annual changes in content and design, national examination scores in Indonesia are not comparable over time. However, it is instructive to look at the average improvements in scores needed to secure the performance component for high and low performing schools (Table 3). Schools that received the performance grant were disproportionately drawn from the top performance quartile which is constructed by averaging 8
school scores in 2013 and 2014. Of the schools which received the performance award in 2015 around 62 percent (75 percent) of primary (junior secondary) schools were already ranked in the top 25 percent prior to the program (Table 3). This suggests that the strength of the incentive that the performance component provided differed across schools depending on their level of past performance.

# **4. Data, research questions and empirical strategy**

# **_Data_**

Using administrative data provided by the Jakarta government from 2012 to 2016, the paper examines whether the introduction of a performance component in the Jakarta school grant during the 2014/15 school year affected the level and distribution of learning outcomes in 2015 and 2016.

Student results on the annual Indonesian National Examination (UN - _Ujian Nasional_ ) are used as the main indicator to assess the impact of the performance grant. The UN is a mandatory standardized test in Indonesian ( _Bahasa Indonesia)_ , English, mathematics and science for government and non-government school students in the last year of primary, junior secondary and senior secondary school. Students across Indonesia take the same examination except in primary schools where provinces have set their own assessments since 2014. While the test development follows international standards, UN results are not comparable over time and primary examination results are only comparable among schools in the same province. Cheating in the national examinations remains a problem despite significant efforts by the authorities to introduce measures (e.g. different test papers, computerizing test taking) to reduce opportunities for cheating. Media reports of cheating are common but they do not show any clear pattern in the frequency of cheating between different types of schools. For example, the answer key for the junior secondary examination in 2015 was reported as being available to buy for between IDR 14 and IDR 21 million (US$ 1,000 – 1,500) in East Java (Tarigan 2015). A comparison of UN scores and scores from an independently administered test where cheating was less likely show a positive and statistically significant correlation (De Ree 2012). However, the integrity of the overall examination process warrants a cautious interpretation of the results of the current paper.

Data on school-level characteristics were also collected from national and provincial education management information systems. These data contained information on the number of teachers, their levels of education and experience, the employment status of teachers, classroom availability and condition. These variables are used to control for the impact of other factors that might have influenced changes in UN scores but were not associated with the school grant program.

# **_Research questions_**

The analysis seeks to answer three research questions:

1. **What impact did the introduction of the performance grant have on student learning in all eligible schools?** The paper first assesses the impact that the performance component had on the UN scores of eligible schools. It is expected that all government schools eligible for the program would have tried to improve their UN scores in an effort to receive the award.

2. **Was the impact of the program different for high and low performing schools?** Table 3 shows that the effort that schools needed to exert to get the performance grant differed 9
depending on their existing level of performance. The paper assesses whether program impact was different among high performing schools that may have had to exert less effort to obtain the performance component of the school grant.

3. **What impact did the additional funds have on student learning in schools that received the performance component?** The paper also compares the impact on learning outcomes between government schools that received the performance grant and schools that did not.

# **_Empirical Strategy_**

In order to explore the first two questions, the paper starts out using a difference-in-differences (DD) approach to compare changes in educational outcomes of Jakarta government schools before and after the announcement of the school grant in 2014 with analogous changes in comparison schools that are not eligible for the grant. To estimate the impact of the introduction of the grant on all eligible schools, changes in UN scores between government schools and non-government schools in Jakarta are compared.<sup>8</sup> The approach is illustrated in Table 4 using the raw average scores for the main comparison groups used in the paper. For example, in 2014, before the introduction of the program, the difference in examination scores between government and non-government junior secondary schools was 4.2 percentage points. After the introduction of the program this raw difference increased to 6.8 percentage points in 2015 and 8.6 percentage points in 2016. The difference in difference from these raw scores suggests that the program may have increased scores in 2015 by approximately 2.6 percentage points and by around 4.4 percentage points in 2016.

# **Table 4: Difference in government-non-government gap in examination scores in Jakarta**

|||Estimating<br>20|impact in<br>15|Estimatin<br>20|g impact in<br>16|
|---|---|---|---|---|---|
||_Pre_|_Post_|_[Post -_|_Post_|_[Post -_|
||_(2014)_|_(2015)_|_Pre]_|_(2016)_|_Pre]_|
|**1. E**|**ffect on all g**|**overnment sch**|**ools**|||
|**a. Primary**||||||
|Jakarta government|70.9|70.2|-0.7|68.1|-2.8|
|Jakarta non-government|72.9|72.7|-0.2|71.6|-1.3|
|[government - non-government]|-2|-2.5|**-0.5**|-3.5|**-1.5**|
|**b. Junior secondary**||||||
|Jakarta government|75.0|77.2|2.2|65.0|-10.0|
|Jakarta non-government|70.8|70.4|-0.4|56.4|-14.4|
|[government - non-government]|4.2|6.8|**2.6**|8.6|**4.4**|

Note: UN scores are expressed as percentages. Figures in bold are the difference in differences of interest.

While Table 4 illustrates a simple comparison of mean scores, a more robust difference-indifferences model outlined in equation (1) is estimated. This is estimated on a sample comprised only of Jakarta schools.

> 8 The DD analyses exclude schools in Jakarta’s Kepulauan Seribu district since this district does not have any nongovernment schools for comparison. Moreover, all schools in Kepulauan Seribu were given an equity-based grant in addition to the performance-based incentive starting in 2015, which may make it difficult to separately identify the impact of the performance component.

10
���� = ∝ + ������� ∗����� + ������� ∗����� + ∑�������� ���� + ���� + ��� + ���� (1) In equation (1), ���� denotes the UN score of school _s_ in district _d_ in year _t_ .<sup>9</sup> �� denote year dummies. ����� is a dummy variable that takes the value of 1 if the school is a government school and therefore eligible to compete for the performance grant, and takes the value of 0 if the school is a non-government school. ����is a vector of observable time-varying characteristics for school s in district d in year t. These are (i) the share of teachers with a bachelors’ degree or higher, (ii) the number of students in the graduating class and (iii) the number of students per classroom. ��� are school fixed effects, and ���� is the error term. Adding school fixed effects absorbs any timeinvariant school-level characteristics that may be correlated with UN scores and allows estimates of within school score changes before and after the introduction of the performance grant. The school fixed effects also control for any sub-district level factors that may drive differences in UN scores. In addition, the dummy variable for government schools (�����) is also subsumed in the school fixed effects. The coefficients of interest are �� and ��; �� is the estimate of the impact of the performance-based grant in 2015 while �� is the estimated impact in 2016. In other words, �� (��) denotes the gap between the change in scores among Jakarta government schools from 2014 to 2015 (2016) and the analogous change among non-government schools in Jakarta. We test the difference between �� and �� to assess whether the impact of the program changes over time.

# **_Validity of Difference-in-Differences_**

A key underlying assumption of the DD approach is that the size of the examination score gap between government and non-government schools was similar and remained relatively stable in the years prior to the introduction of the performance-based grant. This parallel trend assumption is tested in two ways. First, a test is performed to check whether the gap between examination scores in government and non-government schools before the program differed over time on average (denoted by �� in equation _i_ ).<sup>10</sup> This test does not reject the parallel trends assumption at the 5% significance level. Second, a test is performed to examine whether the score gap stayed the same across each year before the announcement. To test this assumption, examination scores are regressed on year dummies and interactions of year dummies with school type.<sup>11</sup> While the results

> 9 Every year, some primary schools are merged. To compare test scores over time, this paper uses the UN score of the “mother” school (that exists in the database after the merge) for the periods before the merge as “mother” schools tend to be larger on average than the other schools that are annexed in the consolidation.

> 10 A similar approach is adopted in Muralidharan, K. and N. Prakash (2013). Cycling to school: increasing secondary school enrollment for girls in India, National Bureau of Economic Research.

A DD model using UN scores data for 2012 to 2014 using the following equation, where _Year_ is a categorical variable from 1 to 3. The full results of this test are shown in Annex Table A1.2 – Panel A. The coefficient of interest in that table is Jakarta Government*Year.

11 A DD model using UN scores data for 2012 to 2014 of the following form is estimated where �� (��) denotes the difference in the government vs non-government score gap in 2013 (2012) relative to the analogous gap in 2014: ���� = �+ ���2013 ∗����� + ���2012 ∗����� + ∑�������� �� �� + ��� + ���� ( _ii_ ) 11
show that the parallel trends assumption holds for primary schools, it is rejected for junior secondary schools at the 5% significance level. This result is driven by the much smaller gap in 2013 compared to 2014, the final year before the program began.<sup>12</sup> However, the score gap in 2012 is similar to the gap in 2014.

In order to check whether the rejection of the common trend assumption in 2013 affects our results, the DD model is estimated using different measures of the baseline score gap. Specifically, a DD specification comparing the post-announcement score gaps between government and nongovernment schools with the average analogous gap in the period from 2012 to 2014 is estimated. In another specification, the post-announcement score gap is compared with the average gap in 2013 and 2014. Regardless of which baseline years are used, the core results of the paper remain unchanged.<sup>13</sup>

# **_Robustness Checks_**

In order to test the sensitivity of the impacts estimated from the model outlined in equation (1), the paper compares changes in the differences in examination scores between government and nongovernment schools in Jakarta with the same gap in other metropolitan areas.<sup>14</sup> Specifically, schools in the most densely populated metropolitan areas around Jakarta are included as comparisons.<sup>15</sup> Table 5 illustrates the approach using the raw differences in average examination scores between Jakarta and these other metropolitan areas. Government schools perform better than non-government schools in both Jakarta and other metropolitan areas. In Jakarta, this gap has widened since the introduction of the new grant program. In contrast, the gap between government and non-government schools has tended to remain relatively similar in other metropolitan areas. Putting these two contrasting trends together suggests a positive impact of the new grant program on junior secondary government schools in Jakarta.

> The full results are reported in Annex Table A1.2 – Panel B.

> 12 Specifically, the test reveals that coefficient for Jakarta Government*Year2013 denotes that the governmentnongovernment gap in 2013 was significantly smaller than that in 2014.

> 13 Annex Table A1.6 show the DD results using alternative baseline measures of the gap between government and non-government junior secondary schools before the grant program was announced. While the size of the estimates is different from the estimates of equation 1, the sign and statistical strength of the coefficients are the same as the results in our main regressions (see Table 6, column 3). In both Table 6 and Annex Table A1.6, the estimated impact in 2015 is positive and significant while the estimated impact for 2016 is larger and significant.

> 14 It is not possible to undertake this robustness check for primary schools since examination scores at the primary level have been set at the provincial level since 2014, which makes it impossible to compare scores in Jakarta and other metropolitan areas after 2014.

> 15 These other metropolitan areas are Bekasi, Bogor, and Tangerang. Government schools in Kota Yogyakarta and Kota Surabaya were also used as a comparison group but since the results are similar they are not reported in the paper to ease exposition.

12
Vsat =A+460,T *GOV.g*DKIgq t+ 62T — * GOVeq * DKI5q 20865 OpTt GOVsq rep01s O¢rTt ~=DKIsa sore Or Tt + Xsat + Usa + Esat DKqq 64 62 6, 62 GOVsa ¥, = YotnTi + ¥2(4i-0) + &%
where � is the average UN exam performance of school _i_ , � is receipt of the performance grant (treatment), � is the performance index score (the variable used for assignment) and � the threshold for assignment to treatment. � �� defined as schools scoring in the top quartile of the performance index, which is based on the average examination performance over the last two years and the scale of improvements in performance over the same period. The treatment effect is given by ��. For a detailed description of the regression discontinuity method used in this paper, see Annex 2.

The regression discontinuity approach assumes that in the absence of the treatment, the sample of schools in a close band around the cutoff _c_ will be similar to each other. Annex Table A2.1 reports the results of a test for random assignment around the discontinuity point (Imbens and Lemieux 2008, Lee 2008). It tests whether there is statistical equivalence in the average characteristics for government schools in Jakarta with scores just below and above the cutoff by school level. As expected, the observable school characteristics are statistically different for schools on either side of the cutoff on average. However, the difference disappears when schools within a small band around the cutoff are compared (with the exception of share of teachers with a bachelor’s degree or higher in junior secondary schools). To increase the precision of the estimated program impacts and to eliminate small sample biases, control variables are also included in equation (3). These control variables are the same as the difference-in-difference models above, which are (i) the share of teachers with education level of S1 or higher, (ii) the number of students in the graduating class and (iii) the number of students per classroom.

# **5. Results**

**_What impact did the introduction of the performance grant have on student learning in all eligible schools?_**

Comparing government and non-government primary schools in Jakarta reveals that the introduction of the performance component had a small but negative impact on examination scores (Table 6). When controls for classroom size, teacher education and the number of graduating students are added, the negative impact is only statistically significant in 2016 and quite small - equivalent to 11 percent of a standard deviation of the comparison group (column 2, Table 6). Put another way, the program resulted in the average UN score of government primary schools falling from 71 percent in 2014 to 70 percent in 2016.

While the performance grant program was implemented independently in each district in Jakarta this does not appear to affect the average results reported in Table 6. The performance grant program for primary schools was implemented independently for each of Jakarta’s six districts and the top 25 percent of primary schools in each district received the performance grant. Given differences in average district performance it might be expected that the magnitude of the program’s impact might also differ across districts.<sup>16</sup> However, looking at the impact of the program in each district separately reveals little difference in program impact in 2015. In 2016, the overall program impact is negative in most districts and these results are statistically significant in four of the districts (see Annex Table A1.8). It is interesting to note that the largest negative

> 16 In 2015, average examination scores in Jakarta’s mainland districts - Jakarta Timur – 74 percent, Jakarta Selatan – 71 percent, Jakarta Pusat – 68 percent, Jakarta Utara – 69 percent, Jakarta Barat – 66 percent.

14
impacts appear to occur in districts that tend to have higher overall performance before the new school grants program is introduced.<sup>17</sup>

# **Table 6. Impact of performance-based grant on eligible primary and junior secondary** **<u>schools in Jakarta</u>**

||Govern|ment v. non-g<br>Jaka|overnment sch<br>rta<br>Ji|ools in<br>d|
|---|---|---|---|---|
||Primary<br>|schools<br>|unor s<br>sch<br>|conary<br>ools<br>|
||No<br>control<br>(1)|Full<br>controls<br>(2)|No<br>control<br>(3)|Full<br>controls<br>(4)|
|**Jakarta Government*Year2015**|-0.51**<br>(0.24)|-0.26<br>(0.25)|2.57***<br>(0.17)|2.61***<br>(0.18)|
|**Jakarta Government*Year2016**|-1.43***<br>(0.29)|-1.25***<br>(0.30)|4.47***<br>(0.36)|4.55***<br>(0.38)|
|Observations|6,849|6,849|2,679|2,679|
|R-squared|0.091|0.101|0.811|0.812|
|No. of Jakarta government schools|1578|1578|280|280|
|No. of comparison schools<sup>a</sup>|705|705|613|613|
|S.D. of 2014 UN score in comparison schools|11|11|7.2|7.2|
|P-value of difference in impact between 2016 and 2015|0.0003|0.001|0.00|0.00|
|Years included|2014-16|2014-16|2014-16|2014-16|
|Controls for students per classroom, teacher education and<br>graduating students|No|Yes|No|Yes|
|Time dummies and school fixed effects|Yes|Yes|Yes|Yes|

Notes: The dependent variable is national examination scores expressed as a percentage. Robust standard errors in parenthesis. *** p<0.01, ** p<0.05, * p<0.1. Comparison schools are non-government schools in Jakarta in columns 1-4. Full results, including the coefficient estimates for the control variables, are reported in Annex Table A1.4.

While further research is required to understand more clearly why the program had a negative impact in primary schools, it may reflect ineffective decision making at the school level. It is possible that the announcement of the program resulted in school principals making changes that had a negative impact, at least in the short term, on student examination results. The limited data available suggest that school principals shifted resources away from hiring temporary teachers and towards improving the condition of classrooms.<sup>18</sup> It is possible that these shifts reduced school

> 17 Annex Table A1.8 also reports results by district for junior secondary schools. They show a similar impact of the program in 2015 across all districts. However, in 2016 the program’s impact in Jakarta Pusat is insignificant and its impact in Jakarta Timur appears much larger than in other districts.

> 18 Data show that government schools reduced the share of temporary teachers in the teaching force from 38 to 34 percent between 2014 and 2016. The share of classrooms in good condition increased from 38 to 47 percent over the same period (see Annex Table A1.3).

15
quality, by for example disrupting schooling as classes were repaired. While these may provide a plausible account of program impact it should be stressed that the lack of information at the school level on change makes it impossible to come to any definitive conclusions.

In contrast to the primary school results, the introduction of the performance grants in government junior secondary schools had a positive and relatively large impact on examination results (Table 6). In 2015, for example, it is estimated that the program resulted in 2.6 percentage point increase in examination scores in government junior secondary schools.<sup>19</sup> Taking the average of all government junior secondary schools this is equivalent to an increase in scores from 72.5 percent in 2014 to 75 percent in 2015.

The impact of the program on junior secondary school examination results increased in the second year of implementation. The estimates suggest that in 2016 the program improved examination scores in government schools by 4.6 percentage points over examination scores in 2014 – equivalent to 64 percent of a standard deviation of the comparison group (column 4, Table 6). The impact in 2016 is larger than the impact in 2015 and possibly highlights that information on the program and its implications spread across more schools over time.

The results for junior secondary schools are partly corroborated by results from comparing the change in the government-non-government gap in examination scores between Jakarta and other metropolitan areas.<sup>20</sup> Using the approach outlined in equation (2) it is possible to estimate the program’s impact on examination results by comparing government junior secondary schools in Jakarta with similar schools in city districts that border or are close to Jakarta. While schools in these areas served similar populations, they were not eligible to receive the new performance component. Using this alternative approach, a positive and statistically significant impact of the program is registered for 2015 and 2016. Estimated impact of the program in 2016 is similar in magnitude to the estimate from the difference-in-difference regression comparing Jakarta government schools with non-government schools (Table 7).

> 19 Equivalent to 35 percent of a standard deviation of the comparison group.

> 20 Since primary school examinations were changed from a national to a province level examination in 2014, it is not possible to do a similar analysis of program impact for primary schools.

16
# **Table 7. Impact of performance-based grant on eligible junior secondary schools in Jakarta**

||**Jakarta v**<br>**metropolitan**<br>**secondary**<br>|**s other**<br>**area junior**<br>**schools**<br>|
|---|---|---|
||No control<br>(1)|Full controls<br>(2)|
|**Jakarta* Government*Year2015**|3.16***<br>(0.73)|5.28***<br>(0.89)|
|**Jakarta* Government*Year2016**|4.31***|5.01***|
||(1.07)|(1.07)|
|Observations|4,014|4,014|
|R-squared|0.618|0.622|
|No. of Jakarta government schools|280|280|
|No. of Jakarta non-government schools|613|613|
|No. of government schools in other cities|78|78|
|No. of non-government schools in other cities|367|367|
|S.D. of 2014 UN score in Jakarta non-government schools|7.2|7.2|
|P-value of difference in impact between 2016 and 2015|0.27|0.88|
|Years included|2014-16|2014-16|
||Jakarta,|Jakarta,<br>Bekasi|
|Cities included|Bekasi, Bogor,<br>and Tangerang|,<br>Bogor, and<br>Tangerang|
|Controls for students per classroom, teacher education and graduating<br>students|No|Yes|
|Time dummies and school fixed effects|Yes|Yes|

Notes: The dependent variable is national examination scores expressed as a percentage. Robust standard errors in parenthesis. *** p<0.01, ** p<0.05, * p<0.1. Full results, including the coefficient estimates for the control variables, are reported in Annex Table A1.5.

There are a number of reasons why it is possible that the program had a different impact on primary and junior secondary schools. First, junior secondary schools tend to have more qualified and experienced staff and may have had greater capacity to improve examination scores compared to their primary school counterparts. Second, while the size of the incentive, in proportional terms was the same, in absolute terms it was much larger for junior secondary schools. Qualified junior secondary schools stood to receive IDR 110,000 from the performance component compared to only IDR 60,000 for primary schools.<sup>21</sup>

# **_Was the impact of the program different for high and low performing schools?_**

> 21 Indeed, comparing the impact of the grant by enrollment size in 2014 shows that the estimated impact in 2016 is bigger in size for larger junior secondary schools – those in the largest quartile registered a significantly different increase in scores (5.6 percentage points) than those in the smallest sized schools (3.4 percentage points). While estimated impacts for primary schools across enrollment sizes are negative, the effects for the largest two quartiles are significantly smaller in magnitude (less negative) than primary schools in the smallest quartile (see Annex Table A1.7.).

17
The grant program was designed in a way that tried to ensure that it gave all schools, regardless of their existing level of performance, an incentive to improve student learning outcomes. However, the paper has shown that the magnitude of the improvements required to secure the performance grant were much greater for low performing compared to high performing schools. Did the difference in effort required to get the performance grant affect program impact for high and low performing schools? This section aims to answer this question by estimating program impact on different quartiles of school performance.

The program appears to have had the greatest impact on the best performing quartile of junior secondary schools (Table 8). Schools were assigned to performance quartiles based on their average examinations scores in 2013 and 2014 before the change in the school grants program. The impact of the program on examination scores for the top performing quartile of junior secondary schools was 6.9 percentage points in 2016 compared to only 2.2 percentage points for government schools in the bottom quartile.<sup>22</sup> This is perhaps an indication that better performing schools in junior secondary felt that they were more likely to receive the additional performance award and exerted greater effort to improve.

The impact of the performance grant on government primary schools varies markedly between different performance quartiles. In contrast with the overall results, the program appears to have had a positive impact on the worst performing primary schools (Table 8). These findings suggest that the effect of the program at the primary level has been to narrow gaps in examination scores between high and low performing schools. However, a large part of the reduction in inequalities in examination results has been driven by the negative impact of the program on better performing schools.

> 22 The difference in results between the bottom and top performing quartiles is significant for estimated impacts in 2016 but not in 2015.

18
**Table 8. Impact of the performance-based grant on examination scores in primary and** **<u>junior secondary schools in Jakarta</u>**

||Jakar<br>|**Primary**<br>ta governm<br>governme<br>|**Schools**<br>ent v. Jakart<br>nt schools<br>|a non-<br>|**Ju**<br>Jakart<br>|**nior Secon**<br>a governme<br>governme<br>|**dary Scho**<br>nt v. Jakart<br>nt schools<br>|**ols**<br>a non-<br>|
|---|---|---|---|---|---|---|---|---|
||Bottom<br>quartile<br>(1)|2nd<br>quartile<br>(2)|3rd<br>quartile<br>(3)|Top<br>quartile<br>(4)|Bottom<br>quartile<br>(5)|2nd<br>quartile<br>(6)|3rd<br>quartile<br>(7)|Top<br>quartile<br>(8)|
|**JakartaGov*Year2015**<br>**JakartaGov*Year2016**|1.88***<br>(0.34)<br>1.61***<br>(0.40)|0.06<br>(0.33)<br>-1.26***<br>(0.40)|-1.20***<br>(0.32)<br>-2.76***<br>(0.39)|-2.15***<br>(0.31)<br>-2.93***<br>(0.38)|1.85***<br>(0.33)<br>2.16***<br>(0.48)|3.01***<br>(0.30)<br>3.81***<br>(0.41)|3.35***<br>(0.26)<br>5.20***<br>(0.53)|2.22***<br>(0.24)<br>6.93***<br>(0.48)|
|Observations|3,303|3,294|3,300|3,297|2,049|2,055|2,043|2,049|
|R-squared|0.034|0.064|0.116|0.130|0.801|0.800|0.795|0.792|
|No. of Jakarta gov.|||||||||
|schools|396|393|395|394|70|72|68|70|
|No. of comparison|||||||||
|schools<br>|705|705|705|705|613|613|613|613|
|S.D. of 2014 UN score<br>in comparison schools|11|11|11|11|7.2|7.2|7.2|7.2|
|P-value of difference in<br>impact between 2016<br>and 2015|0.45|0.0002|0|0.02|0.45|0.05|0.0002|0|
|Years included<br>Controls for students per|2014-16|2014-16|2014-16|2014-16|2014-16|2014-16|2014-16|2014-16|
|classroom, teacher<br>education and<br>graduating students|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|
|Time dummies and<br>school fixed effects|Yes|Yes|Yes|Yes<br>|<br>Yes|Yes|Yes|Yes|

Notes: Dependent variable is national examination scores expressed as a percentage. Robust standard errors in parenthesis. *** p<0.01, ** p<0.05, * p<0.1. Comparison schools are non-government schools in Jakarta. Full results, including the coefficient estimates for the control variables, are reported in Annex Table A1.9.

# **_What impact did the additional funds have on student learning in schools that received the performance component?_**

In order to look at whether the additional resources that schools were awarded from the performance component helped to improve learning outcomes, the paper compares schools that did not receive the performance component but were very close to doing so with schools that just managed to improve their performance enough to get the performance award. Table 9 shows the impact estimates from the regression discontinuity design in 2015 and 2016 for schools on either side of the cut-off for awarding of the performance grant. One way of thinking of the RD estimate is as a local average treatment effect – the effect on those induced to comply as their scores cross the threshold of eligibility. Given the narrow focus on schools around the threshold of eligibility, any differences in test scores have to be sufficiently large to be statistically distinguishable from zero.

19
# **Table 9. Regression discontinuity estimates of the impacts on examination scores of receiving** **<u>the performance grant</u>**

||Jakar<br>20<br>|ta governme<br>15<br>|nt primary s<br>2<br>|chools<br>016<br>|Jakar<br>20<br>|ta governmen<br>sch<br>15<br>|t junior sec<br>ools<br>2<br>|ondary<br>016<br>|
|---|---|---|---|---|---|---|---|---|
||No<br>control<br>(1)|Full<br>controls<br>(2)|No<br>control<br>(3)|Full<br>controls<br>(4)|No<br>control<br>(5)|Full<br>controls<br>(6)|No<br>control<br>(7)|Full<br>controls<br>(8)|
|**Received rant**|-0.90|-0.84|0.34|0.93|0.78|0.51|-3.27|-3.51|
|**g**|(1.17)|(1.13)|(1.75)|(1.71)|(1.09)|(1.04)|(3.11)|(3.03)|
|Observations|275|275|323|323|124|124|145|145|
|Controls for students<br>per classroom, teacher<br>education and<br>graduating students|No|Yes|No|Yes|No|Yes|No|Yes|

Note: Each column is the result of a separate regression. All regressions use a triangular kernel and optimal bandwidth that reduces the mean squared error as proposed by Imbens and Kalyanaraman (2011). Controls are share of teachers with education degree S1 or above, the number of students in the graduating class and the student classroom ratio. *** p<0.01, ** p<0.05, * p<0.1. Column 1-2 and 5-6 show the estimated impact of receiving the performance-based grant in 2015 while columns 3-4 and 7-8 show the impact in 2016.

The results show that the additional resources provided by the performance component of the grant do not appear to have a statistically significant impact on examination scores. While the results show a small negative impact of receiving the performance grant in 2015 on primary school examination scores they are not statistically significant - the impact of the performance grant on the 2015 national examination scores ranges from -0.9 to -0.8 percentage points at the margin of the cutoff. In contrast, receipt of the performance grant improved national examination scores in 2016 for primary schools (0.3 to 0.9 percentage points) but again, these coefficients are not distinguishable from zero.<sup>23</sup> The results for junior secondary schools suggest that the additional funds improved examination scores in 2015 but had a negative impact on scores in 2016. However, none of the results are statistically significantly different from zero (see columns 5-8 of Table 9).

One possible explanation for these null results on test scores using the RDD approach is that receipt of the performance grant is not sufficient to boost schools at or near the threshold to increase performance on the national examination. The performance grants may have helped schools allocate resources to improve school quality but these changes may not reflect quickly enough into improvements in test scores.

In order to test this hypothesis, the regression discontinuity model is re-estimated for two different outcomes - the share of temporary, auxiliary, or honorary teachers, which is a proxy measure for the quality of teachers. Second is the share of classrooms in good condition, which is a proxy measure for the quality of school infrastructure. The additional funds that schools receive appear to have had no significant effect on the share of temporary, auxiliary, or honorary teachers in either primary or junior secondary schools. At the secondary school level, the additional funds schools receive from the performance component has a significantly negative impact in 2015 on the share

> 23 All of these results are robust to different bandwidth specifications (see Annex Table A2.2).

20
of classrooms in good condition. However, this negative effect of the performance grant on the share of classrooms in good condition at the secondary school level does not persist into 2016, which suggests that the estimates are not stable. Taken together, the RDD results suggest that the additional funds that schools received under the performance component had no impact on examination scores or on intermediate outcomes (see Annex Table A1.10).

# **6. Discussion and Conclusion**

Data from the first two years of program implementation suggest that thus far the introduction of a performance component into the school grants program appears to have had very different impacts on government primary and junior secondary schools in Jakarta. In primary schools, the performance component of the program reduced examination scores albeit by a relatively small amount. However, these average results mask differences in program impact across schools with differing levels of initial performance; program impact was positive for schools in the bottom quartile of performance and negative for better performing schools. As a result, the program has tended to narrow inequalities in school performance. In contrast to the primary school impacts, the introduction of the performance component improved examination scores in government junior secondary schools. However, the impact seemed to be greatest among better performing schools and has contributed to a widening gap in performance.

The findings show that the efforts that some government primary schools made to improve examination scores in the first two years of implementation of the performance component were unsuccessful. It is possible that the lack of flexibility in the types of spending allowed under the school grant program made it difficult to respond to the additional incentives appropriately. Further research that takes a closer look at the strategies adopted by schools to improve performance would be useful in unpacking the effects of the program. For example, a focus group discussion with principals of primary and junior secondary schools that received the performance grant may help clarify how funds were used in practice. Comparing these changes between primary and junior secondary schools could also highlight the factors underlying the differential results between primary and junior secondary schools in Jakarta.

The paper shows that after the announcement of the performance component, principals in primary schools changed the way they allocated their resources – both by hiring fewer contract teachers and by spending more on physical infrastructure at the classroom level. Available data only allow a few such channels to be empirically tested in this paper. However, the growing evidence base suggests that there are many other avenues to improve school performance possibly with greater impact (Snilstveit, Stevenson et al. 2015). Increased teaching time, improved pedagogy, support to at-risk learners and teacher professional development are all areas that have the potential to improve school performance. Encouraging experimentation with these and other approaches to improve school performance could help schools act on the incentives created by the program. This is likely to require raising awareness among primary school principals and supporting them in school management and the effective use of resources to improve performance. A close look at the effectiveness of district education offices and other institutions in providing this kind of support has the potential to improve the impact of performance-based school grant programs at the primary level.

21
It has not been possible to assess the effectiveness of using examination scores to award the performance component of the school grant. However, evidence in other contexts suggests that using tests of student level achievement to judge school performance may not be optimal (Neal 2013). For example, in Kenya the introduction of teacher incentives based on student test results led to a greater focus and effort on preparing only for the tests directly linked to these incentives. While student results on these tests improved, they did not register any gains in other examinations that covered similar subjects using different testing formats (Glewwe, Ilias et al. 2010). Given these findings from other countries, it may be worthwhile in the Jakarta context to experiment with alternative measures of school performance in the program. For example, Indonesia has a wellestablished quality assurance system that measures performance along a set of 8 dimensions (Ministry of National Development Planning 2015). Using these assessments may strengthen the incentive effect of the program and provide schools with greater guidance on areas of improvement.

The results for junior secondary suggest that the program has resulted in a widening of the gap between high and low performing government schools. Despite attempts during design to provide incentives for all schools, the paper has shown that better performing schools were more likely to receive the performance component and registered larger examination score improvements when compared to poorer performing schools. Using existing information and results from this paper it is possible to develop alternative designs for the performance-based component that could narrow inequalities. Experimenting with different weightings for the change and level components of the performance award could help to fine tune the program and support the equity objectives of the Jakarta government. An alternative design could be more targeted at the outset and foster competition among the bottom quartiles in the first year and work upward through the performance distribution in subsequent years.

The evidence presented in the paper also suggests that the additional resources that successful schools receive as part of the program have not improved performance. Rather, the biggest effects of the program appear to have arisen through the incentive that it provided to all schools regardless of whether they went on to receive the additional funding associated with the program. It is possible that a non-monetary program that recognized improved school performance could provide an alternative and less costly incentive to improve performance. An examination of the effectiveness of alternative models of recognition could be a first step in designing the next generation of performance-based school grants. While further experimentation is needed to understand how to best design programs of this kind, the results in this paper from the first year of a school grants program that directly linked school grant amounts to school performance suggest that this is an avenue worth pursuing.

22