metadata: authors: 0: author: Azevedo,Joao Pedro Wagner De
1: author: Newman,John Lincoln
count: Brazil
docna: 0: docna: Setting reasonable performance targets for
            public service delivery
docty: Policy Research Working Paper
subtopic: Consumption,Fiscal & Monetary Policy,Reproductive Health,Early Child and Children's Health,Nutrition,Educational Sciences,Inequality,Crime and Society
abstracts: cdata!: Reaching agreement on a reasonable
            performance target is a challenge, with costs associated
            with getting it wrong. Attention in the literature has
            focused on the potential negative effects of gaming or of
            creaming. However, even if there is no gaming or creaming
            taking place, there can still be costs associated with
            setting a level of the performance target that is either too
            low or too high. On the one hand, if the negotiated
            performance target is too low, there is a strong risk that
            the target would be met without any change in behavior or
            performance from what would have been realized without a
            performance management system. In that case, there would be
            no benefit -- only the cost of covering the administrative
            costs associated with developing the monitoring and
            management systems. On the other hand, if the negotiated
            performance target is too high, there could also be
            significant costs. The exact nature of the costs depends on
            which one of two unattractive options the principal chooses
            to follow once it becomes apparent that the performance
            targets were set unrealistically high. If the principal
            chooses simply to waive any possible repercussions for the
            agents for not meeting the performance targets, this can
            undermine the credibility of the system.  If the principal
            insists on holding agents to meeting the performance targets
            -- no matter how unrealistic they were -- this can breed
            resentment and adversely affect future productivity. This
            paper considers some approaches to target setting that have
            been used in the literature and proposes an approach based
            on the use of quantile regressions to construct a
            Characteristic Adjusted Performance distribution of
            performance to guide the selection of targets. The paper
            then presents two concrete examples of applications of this
            approach related to the setting of targets on School Test
            Scores and Improvement in Homicide rates in Police Districts
            in the State of Minas Gerais, Brazil.
geo_reg_key: 119228,517191
repnme: repnme: Setting reasonable performance targets for
            public service delivery
display_title: Setting reasonable performance
            targets for public service delivery
listing_relative_url: /research/2013/03/17453225/setting-reasonable-performance-targets-public-service-delivery
topicv3: Growth Monitoring and Promotion m1326325 1444,Poverty m1326687 695,Education m1327048 470,Macroeconomic and Structural Policies
            m1327380 439,Homicide m1439222 HOM-3046
publishtoextweb_dt: 2013-01-01T00:00:00Z
no_of_pages: 33
author_keys: 0: author_key: 675142
1: author_key: 92938
model_extractions: WPS6385 Policy Research Working Paper 6385 Setting Reasonable Performance Targets for Public Service Delivery _John L. Newman João Pedro Azevedo_ The World Bank Latin America and the Caribbean Region Poverty Reduction and Economic Management Department March 2013
Policy Research Working Paper 6385

# **Abstract**

Reaching agreement on a reasonable performance target is a challenge, with costs associated with getting it wrong. Attention in the literature has focused on the potential negative effects of gaming or of creaming. However, even if there is no gaming or creaming taking place, there can still be costs associated with setting a level of the performance target that is either too low or too high. On the one hand, if the negotiated performance target is too low, there is a strong risk that the target would be met without any change in behavior or performance from what would have been realized without a performance management system. In that case, there would be no benefit—only the cost of covering the administrative costs associated with developing the monitoring and management systems. On the other hand, if the negotiated performance target is too high, there could also be significant costs. The exact nature of the costs depends on which one of two unattractive options the principal chooses to follow once it becomes apparent that the performance targets were set unrealistically high. If the principal chooses simply to waive any possible repercussions for the agents for not meeting the performance targets, this can undermine the credibility of the system. If the principal insists on holding agents to meeting the performance targets—no matter how unrealistic they were—this can breed resentment and adversely affect future productivity. This paper considers some approaches to target setting that have been used in the literature and proposes an approach based on the use of quantile regressions to construct a Characteristic Adjusted Performance distribution of performance to guide the selection of targets. The paper then presents two concrete examples of applications of this approach related to the setting of targets on School Test Scores and Improvement in Homicide rates in Police Districts in the State of Minas Gerais, Brazil.

This paper is a product of the Poverty Reduction and Economic Management Department, Latin America and the Caribbean Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http:// econ.worldbank.org. The authors may be contacted at jnewman@worldbank.org and jazevedo@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Setting Reasonable Performance Targets for Public Service Delivery<sup>1</sup>

John L. Newman<sup>*</sup> João Pedro Azevedo<sup>+</sup> Keywords: Government Performance; Behavior of Economic Agents; Target Setting; Latin America; Brazil; Minas Gerais JEL Codes: H11, H30; H83 Sector Board: Poverty Reduction (POV) The findings, interpretations, judgments and conclusions expressed in this publication are those of the authors and should not be attributed to the World Bank, to its affiliated organizations, or to members of the Board of Executive Directors or the governments they represent.

1 The authors would like to thank Hernan Jorge Winkler for excellent research assistance, the World Bank’s Public Sector Performance Global Expert Team for providing references on country experience on public sector improvement initiatives and Tadeu Barreto, Iran Almeida Pordeus, Eder Campos, Eber Golcalves, Juliana de Lucena Ruas Riani, Tarsila Velloso, Barbara Bruns, David Evans, Roland Clarke, Laura Zoratto, Deborah Wetzel and Jose Guilherme Reis for very helpful conversations on the issues in this paper.

(*) Poverty Reduction and Economic Management team of South Asia; and, (+) Poverty Reduction and Economic Management team of Europe and Central Asia. Correspondence should be directed to <u>jnewman@worldbank.org</u> and <u>jazevedo@worldbank.org</u>
# 1.  Introduction

It is increasingly common for negotiations over performance targets to take place as part of a process to improve performance in the public sector.<sup>2</sup> These negotiations are carried out between a principal (the entity that is demanding the improved performance, usually the Ministry of Finance, Planning or the Head of a Sectoral Ministry) and an agent (the entity that is responsible for improving performance, often a sectoral ministry, a subnational unit of a sectoral ministry or an entity delivering services such as a school or a health center). There are quite a few issues that must be successfully resolved for the introduction of performance targets to bring about an improvement in public sector performance (see the recent book edited by Heckman, Heinrich and Smith, 2011)<sup>3</sup> . This paper deals with only one of the issues - albeit an important one - that of arriving at the performance target in the first place<sup>4</sup> . It considers two cases in Minas Gerais, Brazil - education and crime and violence - and compares the relative merits of two approaches to incorporating information on past performance of different agents to arrive at reasonable targets. One approach has been used in the setting of targets in Job Training Partnership Act (JTPA) program. An alternative approach – and the recommended one - is based on estimating quantile regressions and is being used in estimating growth percentiles of student performance in the education sector in a few states in the U.S. There are strong parallels in the nature of the problem in assessing what constitutes adequate performance in student achievement and in guiding the selection of targets in a performance management system that operates across different levels of government. However, this approach used in education has not yet made its way into the arena of general public sector management performance. It could and should be considered for the more general task of arriving at reasonable targets in many sectors.

# 2.  The problem – How to set a reasonable performance target

Reaching agreement on a reasonable performance target is a challenge, with costs associated with getting it wrong. Attention in the literature has focused on the potential negative effects of gaming or of creaming. However, even if there is no gaming or creaming taking place, there can still be costs associated from setting a level of the performance target that is either too low or too high. If the negotiated performance target is too low, there is a strong risk that the target would be met without any change in behavior or performance from what would have been realized without a performance

> 2 Countries which have introduced performance targets include Australia (see <u>http://www.federalfinancialrelations.gov.au/content/performance_reporting.aspx);  Chile (see http://www.gob.cl/cumplimiento/gestion_de_cumplimiento.html);  Colombia (see</u> - - <u>http://www.sigob.gov.co/pnd/inst.aspx) ; France (see http://www.performance publique.budget.gouv.fr/la performance-de-laction-publique.html) ;  India (see http://www.performance.gov.in/</u> );  Indonesia (see bappenas.go.id/get-file-server/node/9374/)  ; Malaysia (see http://www.pemandu.gov.my/ ); the UK ( see <u>http://www.instituteforgovernment.org.uk/performance/</u> ); and the US (see http://www.performance.gov/ ).  The state of Minas Gerais in Brazil has also placed a strong emphasis on performance management (see <u>http://www.agendademelhorias.org.br/</u> ).

3 That book points out that while there is general interest in performance management in many sectors, there is a long history of managing such systems in job training and exploits that long history to address general issues related to performance management.

> 4 Courty, Heinrich and Marschke (2011) also discuss the setting of targets (or standards).  However, their analysis does not consider the use of quantile regressions considered in this article.

2
management system. In that case, there would be no benefit – only the cost of covering the administrative costs associated with developing the monitoring and management systems. On the other hand, if the negotiated performance target is too high, there could also be significant costs. The exact nature of the costs depends upon which one of two unattractive options the principal chooses to follow once it becomes apparent that the performance targets were set unrealistically high. If the principal chooses simply to waive any possible repercussions for the agents for not meeting the performance targets, this can undermine the credibility of the system. If the principal insists on holding agents to meeting the performance targets – no matter how unrealistic they were – this can breed resentment and adversely affect future productivity.

The task is further complicated by the fact that there is typically a cascading down of performance targets being set at a first level administrative unit (perhaps at a national or state level) down to a second and/or third level administrative unit (perhaps a district or a county). There is often considerable variation in the initial conditions and the potential for performance across the lower level administrative units, making it difficult or unreasonable to set identical performance targets across administrative units.

# 3.  Possible approaches

Performance targets are usually set as an absolute target for an agent to reach, which is known to the agent prior to the period where performance would be reviewed.<sup>5</sup> The absolute target may be specified for a level of the performance indicator or a change in the indicator.<sup>6</sup> It is not uncommon for an absolute target to be set simply at a value which is a given percentage above the previous value, or some average of the previous value of the chosen performance indicator. For example, the principal could request that the agents reach a performance level that is 5 percent higher than the previous value. The problem with this approach is that it does not take into account the different initial conditions that the agents face that could make it easier or harder to achieve the targets.

Ideally, one would want the probability of achieving the target to be due solely to effort. There should be no systematic tendency for an agent with a given set of characteristics to succeed or fail in reaching the target. However, it there are different initial conditions that are present and that do affect the probability of achieving the target, then not taking them into account in the setting of the target will generate a systematic pattern in ex post observations of whether the target is achieved or not. This notion – that setting a reasonable target would be consistent with there being no systematic relation between achieving the target and a circumstance outside the control of the agent – is very similar to the underlying principle that has been used to create a metric to measure the equality of opportunity (Barros et al, 2009). In section 4, we show how that metric can be used to assess the relative quality of different approaches to target-setting.

> 5 Barlevy and Neal (2011) propose a Pay for Percentile incentive plan  that is based on relative performance and not on attaining a performance level or change relative to a target.   A nice brief discussion of the plan is presented by Barlevy and Neal in VoxEU (see http://www.voxeu.org/index.php?q=node/7073)   This is discussed in more detail in section 6).

> 6 In the education sector, many school districts and researchers stress the importance of looking at both achievement (the level) and growth.  A nice interactive demonstration of both performance and growth information on schools for the Colorado State Department of Education can be found at http://www.schoolview.org/ColoradoGrowthModel.asp

3
There are several approaches that can and have been used to adjust the targets to observable characteristics that are related to the difficulty of achieving the target. One possible approach to adjusting the target for differences in initial conditions would be to generate a predicted mean value of performance for the agent, given the agent’s characteristics, and use that predicted mean value to help guide negotiations on possible performance standards. The predicted mean value could be obtained by estimating an ordinary least squares (OLS) regression and multiplying the estimated coefficients by the value of the characteristics of the agent. However, this adjustment is not entirely satisfactory as the idea of working with performance targets is to try to induce behavior that would encourage agents to avoid poor performance and/or achieve better than average performance. There is interest in identifying performance that is distinct from average performance. The incentives that are built into performance management systems typically kick in when performance is either exceptionally poor (which might invoke some requirement that the agent receive additional supervision or technical help) or exceptionally good (which might generate a reward). It would be possible to use the predicted mean value as an anchor and negotiate standards that would be 10% or 20% higher than the predicted mean. However, in so doing the negotiations would begin to be drawn further and further away from a sound empirical grounding. One does not really know how demanding it would be to reach a performance level that is 20% higher than the mean. Has that been achieved before? How often and how easily?

An alternative approach, used by the U.S. Labor Department in setting the performance targets for training centers in the Job Training Partnership Act (JTPA) program, is to set targets based on a specified percentile distribution of the performance indicator – adjusted upwards or downwards according to the characteristics of particular agents (see Courty, Heinrich & Marschke, 2011). The JTPA was concerned with avoiding poor performance in the individual centers and, therefore, selected the 25<sup>th</sup> percentile as the base performance target. The JTPA managers then factored in an adjustment that was determined from an OLS regression of the performance indicator on the characteristics of the center (expressed as deviations from the mean). The adjustment factor was obtained by multiplying the estimated coefficients from the OLS regression by the value of the characteristics of the center. This is better than the first approach, in that it takes as its starting point a percentile of the performance distribution that is judged to be relevant. However, the adjustment factor does have a limitation in that it adjusts the target by the same amount, regardless of whether the relevant percentile is chosen to be the 25<sup>th</sup> percentile or the 67<sup>th</sup> percentile. There is no reason to believe that the influence of the characteristics of the agents would be the same around the 25<sup>th</sup> percentile as they would be around the 67<sup>th</sup> percentile.

A third approach - and our recommended approach - is to use quantile regressions of the performance indicator on a set of observable initial conditions to generate a predicted percentile for each agent. The target would then be set as the value that pertains to a common percentile, for example the 67th percentile. Different agents would face a common percentile target, but this would correspond to different absolute values of the target because their characteristics differ. This would be true whether the target is specified in terms of a specific level of the performance indicator or a change in the performance indicator. Quantile regressions can control for whatever observable characteristics the interested observer may want to take into account in putting past performance and future goals in perspective. In contrast to an OLS regression that estimates the relation at the mean of the distribution, quantile regression can estimate the relation at the median, the 25<sup>th</sup> percentile or, indeed, at any given percentile (Koenker and Bassett, 1978; Koenker, 2005). Just as the estimated coefficients from an OLS regression might be combined with values of the country characteristics to yield a predicted mean value of the 4
change in the indicator of interest, the estimated coefficients from a quantile regression on the 25<sup>th</sup> or 67<sup>th</sup> percentile can be used to generate predicted percentiles at the 25<sup>th</sup> or 67<sup>th</sup> percentile.

Quantile regressions have been used to generate student growth percentiles (Betebenner, 2009) and conditional growth charts (Wei and He, 2006). The work on student growth percentiles has generally conditioned only on past achievement levels (and not observable characteristics of the students or their teachers), as the effort of introducing the concept into educational management in such places as the states of Colorado and Massachusetts has been to separate the description of student progress (the SGP) from the attribution of responsibility for that progress<sup>7</sup> .

Quantile regressions have been used quite extensively in economics (see Koenker and Hallock, 2001 for a review). Three applications that are most closely related to controlling for heterogeneity in looking at the performance in percentiles are: a) Essama-Nassh et al (2010), which used quantile regressions<sup>8</sup> in an extension of the growth incidence curves that have been used to look at pro-poor growth (Ravallion and Chen, 2003): b) the World Bank Poverty Assessment for Guatemala (World Bank, 2009), which used quantile regressions to look at conditional percentile distributions for a set of poverty and social indicators; and c) (Azevedo and Pizzolitto, 2009) which used quantile regressions to assess the targets of the National Development Plan of the Dominican Republic.

The approaches followed with student growth percentiles and conditional growth charts share some similarities with the approach followed with growth incidence curves. They both look at the growth of an indicator at a given percentile in the distribution. They both can control for background characteristics. However, growth incidence curves are typically calculated from repeated cross-sections. They typically cannot provide information on the distribution of performance of those households or economic actors who began at a given percentile. In contrast, student growth percentiles or conditional growth charts (often of children’s height or weight) typically look at repeated observations of the same individual and looks at where the performance of that individual falls in the full distribution of performance of a comparable group. For the issue of target setting, one is typically concerned with repeated observations of the same agents, so the approach of the student growth percentiles or conditional growth charts would appear to be the most relevant.

# 4.  An illustration of target setting using School Education Data from Minas Gerais, Brazil

This section uses data on student achievement scores across different schools in the state of Minas Gerais, Brazil to illustrate some of the implications of using different approaches to set performance targets. The primary source of data for this exercise is from the PROEB (Program of Evaluation of Basic Education), which carries out Portuguese and math tests at the pupil level for municipal and state school students in 5<sup>th</sup> , 8<sup>th</sup> and 11<sup>th</sup> grade in the state of Minas Gerais. This analysis focuses only the State schools (which

> 7 See Betenner et al (2011).  See also the primers on student growth percentiles put together by the states of Colorado and Massachusetts. ((http://www.cde.state.co.us/cdedocs/Research/PDF/Aprimeronstudentgrowthpercentiles.pdf) and ( <u>http://www.doe.mass.edu/mcas/growth/InterpretiveGuide.pdf</u> )

> 8 They use unconditional quantile regressions developed by Firpo et al (2009), rather than the conditional quantile regressions used by Betebenner, Wei and He .

5
were under the jurisdiction of the State Government of Minas Gerais) and, for simplicity, focuses only on the results of 5<sup>th</sup> grade students. Student achievement tests were introduced by the state government of Minas Gerais in 1992 as part of its evaluation program designed to improve the quality of education, and was part of a broader proposal for education on the part of the government. The achievement tests included complementary socio-economic information through self-administrated questionnaires. Initially, the evaluations were planned and carried out in two year cycles, but in 1998 the system was modified to administer yearly tests.<sup>9</sup> In 2007, the state of Minas Gerais embarked on an ambitious and successful reform program (State for Results) to improve performance in a comprehensive set of public sector activities across all sectors that involved setting targets on a wide range of indicators (see annex 3). In the education sector, the Secretary of Education set targets in 2008 for proficiency in test scores in Portuguese and Mathematics for different grades. While some negotiation of the targets took place, for the most part the targets were set at a 4 percent increase over the 2007 test score for the school.<sup>10</sup> Figure 1 illustrates that there was some (slight) difference in the target according to the value of the Portuguese proficiency in 2007. It shows the difference between the target and 1.04 times the 2007 value of Portuguese proficiency for different values of Portuguese proficiency in 2007.

# **Figure 1.  Variation in Difference between 2008 Secretary of Education Target and 1.04 * Actual Portuguese Proficiency in 2007, Schools in Minas Gerais, Brazil**

> 9 Today, the state of Minas Gerais has a comprehensive evaluation system for its educational system - SIMAVE: Quality of Education and School Evaluation System. This system is made up of three evaluation programs: PROALFA — geared towards the evaluation of literacy levels; PROEB — to verify the efficiency and quality of education based on performance in the final levels of education; PAAE — to perform progressive learning diagnoses to support pedagogical interventions.

10 The fact that the target was very close to a set percentage for each school can be seen from the results of a regression of the 2008 target on the 2007 value of the test score (with the regression constrained to exclude the constant).

Target for 2008 Portuguese Proficiency = 1.04279 *2007 Portuguese proficiency (4,866) R2 = .9999 N=2416 6
<!-- Start of picture text --> 100 150 200 250 300<br>2007 Portuguese Proficiency, 5th Grade<br>15<br>10<br>5<br>Difference<br>0<br>-5<br><!-- End of picture text --> Figure 2. plots the difference between the 2008 actual and Secretary of Education target performance of schools in average Portuguese proficiency of 5<sup>th</sup> grade students against some observable characteristics of schools (the number of students in 2007, the school participation rate of students in the PROEB exam in 2008 and the average Portuguese proficiency of 5<sup>th</sup> grade students in 2007). As discussed in section 3, ideally there should be no relation between whether the target is achieved and the observed characteristics. While there seems to be no (or little) relation between the difference of the 2008 actual and target levels of Portuguese proficiency and the number of students and the participation rate in the exam, it does appear as if those schools with higher levels of Portuguese proficiency in 2007 were systematically less likely to achieve the target. The same 4% improvement applied to a higher level of achievement calls for a larger absolute improvement. The graphs show there is a strong tendency for schools which had a higher absolute target to be less likely to have achieved their targets. This is more likely to due to how the targets were set in the first place, then one the ability of any particular school to achieve the target.

7
**Figure  2.   Variation in Actual 2008 Performance and 2008 Sec. of  Education Target in Portuguese Proficiency,  5**<sup>**th**</sup> **Grade Schools - Minas Gerais, Brazil**

<!-- Start of picture text --> 0 1000 2000 3000 4000<br>Total Number of Students 2007<br>0 50 100 150 200<br>School Participation Rate in PROEB 2008<br>100 150 200 250 300<br>School Average Proficiency Score in Portuguese of 5th Grade Students, 2007<br>100<br>50<br>0<br>-50<br>Difference 2008 Actual - Sec. Ed Target<br>-100<br>100<br>50<br>0<br>-50<br>Difference 2008 Actual - Sec Ed Target<br>-100<br>100<br>50<br>0<br>-50<br>Difference 2008 Actual - Sec. Ed Target<br>-100<br><!-- End of picture text --> 8
# _Comparison of Results from Secretary of Education Targets and Targets Based on Quantile Regressions_

Figure 3 graphs the same relationship – this time replacing the target set by the Secretary of Education with the target generated from quantile regressions (our recommended approach). The target for each school is generated from the following procedure. First, a quantile regression at the 67<sup>th</sup> percentile was run that regressed the change in Portuguese proficiency between 2007 and 2006 on a regional dummy for Grande Norte, the total number of students in the school in 2006 and the initial level of Portuguese proficiency in 2006. Then, the estimated coefficients from that estimation were multiplied by the 2007 values of the right hand side variables in the quantile regression (i.e. the constant term and the regional dummy and the 2007 values of the number of students and Portuguese proficiency). For each school, this yields the change in Portuguese proficiency that they would have if they were able to achieve the performance equivalent to those in the 67<sup>th</sup> percentile of schools with their given characteristics. Finally, this estimated change at the 67<sup>th</sup> percentile of performance for the particular characteristics of the school was added to the base 2007 value of Portuguese proficiency for each school to generate a target level for 2008.

**Figure 3.   Variation in Actual 2008 Performance and 2008 Quantile Regression Target in Portuguese Proficiency,  5**<sup>**th**</sup> **Grade Schools - Minas Gerais, Brazil**

<!-- Start of picture text --> 100 150 200 250 300<br>School Average Proficiency Score in Portuguese of 5th Grade Students, 2007<br>100<br>50<br>0<br>-50<br>Difference 2008 Actual - QR Target<br>-100<br><!-- End of picture text --> 9
Figure 4 again graphs the same relationship – this time with the target set in the same fashion as was followed in the Job Training Partnership Act (JTPA). The JTPA approach was to start from the value of a specific percentile of the empirical distribution of the performance standard (in this case, the 67<sup>th</sup> percentile of the change in Portuguese proficiency in 5<sup>th</sup> grade) and adjust it upwards or downwards according to the characteristics of the school. The adjustment factor is calculated by multiplying the characteristics of the school (expressed as a deviation from the mean) by the estimated coefficients for that characteristic obtained from an OLS regression of the change in Portuguese proficiency on the characteristics. The value of the base 67<sup>th</sup> percentile of the change plus the adjustment factor was added to the 2007 Portuguese proficiency to yield a 2008 target for the level of Portuguese proficiency.

**Figure 4.   Variation in Actual 2008 Performance and 2008  Target in Portuguese Proficiency,  5**<sup>**th**</sup> **Grade Schools - Minas Gerais , Brazil (Target based on JTPA-type adjustment to 67**<sup>**th**</sup> **Percentile)**

<!-- Start of picture text --> 100 150 200 250 300<br>School; Average Proficiency Score in Portuguese of 5th Grade Students, 2007<br>100<br>50<br>0<br>-50<br>Difference 2008 Actual - JTPA Target<br>-100<br><!-- End of picture text --> The differences revealed in these graphs are also revealed in Table 1, which compares estimated the results of probit estimates of the probability of achieving the target on the Portuguese proficiency in 5<sup>th</sup> grade in 2007, the number of students in 2007, a dummy variable for whether the school was in the Grande Norte, a poorer part of Minas Gerais and the participation rate in the PROEB test in 2008. Note that the pseudo R2 is highest for the probit using the Secretary of Education target, while the number of significant coefficients is lowest for the target based on the quantile regressions. Note also, that even 10
though there are differences in the explanatory power across the different targets, there are not large differences in characteristics in the targets themselves. This actually is a warning sign as it indicates that, in the aggregate, there may be little difference in the targets, but there can still be systematic differences in who is likely to achieve the target.

**=** **<u>Table 1.  Probit Estimates Using Alternative Targets (Meeting target 1, 0 otherwise)</u>**

||Secretary of Education<br>Target|Target Based on<br>Quantile Regression at<br>67<sup>th</sup>Percentile|Target based on JTPA-<br>type adjustment to value<br>at 67<sup>th</sup> percentile|
|---|---|---|---|
|Constant|4.176***<br>(14.84)|.194<br>(0.78)|-.178<br>(-0.72)|
|2007 Portuguese<br>proficiency|-.021***<br>(-14.90)|-.001<br>(-.085)|.001<br>(0.89)|
|2007 Number of<br>students|-.0001*<br>(-1.93)|.00001<br>(0.19)|-.0001**<br>(-2.36)|
|2008 Participation Rate<br>in PROEB|.0002*<br>(-1.86)|.0003**<br>(2.42)|.0003**<br>(2.45)|
|Dummy for Grande<br>Norte|-.131**<br>(-2.08)|.096<br>(1.60)|.161***<br>(2.68)|
|Pseudo R2|.0887|.0052|.0090|
|N|2204|2255|2255|
|Mean of target|199.9|199.8|200.0|
|Standard Deviation of<br>target|22.2|15.4|14.8|
|Range of target|130.1-300.4|148.1-264.5|150.2-259.9|
|Percentage of Schools<br>that met target in 2008|52.3%|53.1%|52.3%|

Looking at patterns of coefficients in probit estimations is one way of judging whether the probability of achieving the target is systematically related to the particular circumstances of the agents. However, there is some value to having a summary measure that could be used to judge whether there is any significant relation to the circumstances. A summary measure is readily available because the problem that is posed in judging the adequacy of the target is an exact parallel to the problem of determining whether opportunities are equally distributed in the population in the literature of equality of opportunity. The metric developed for the equality of opportunity, the human opportunity index, is an inequality-adjusted coverage rate, with the adjustment or penalty determined by the extent to which there is a deviation from randomness in who receives the opportunity and who does not. If there is no systematic relation between who gets the coverage and a set of circumstances, then there will be no difference between the estimated coverage rate and the human opportunity index. The relevant test, then, is a test of whether the penalty is equal to zero.

11
Table 2 presents the mean value of the key HOI related indicators as well as the 95 percent lower and upper bound confidence intervals. Looking at the penalties, it is clear that the targets based on either the quantile regression or the JTPA-type adjustment do better than the Secretary of Education target (the 4% improvement). The quantile regression model appears to do marginally better than the JTPA-type adjustment, but in both cases the confidence interval lies above zero. It is worth pointing out that setting the targets according to a quantile regression estimation or according to a JTPA-type adjustment was never actually carried out. The results presented here strongly suggest that setting the targets based on the quantile regressions or the JTPA-type adjustments is likely to eliminate the tendency for the outcomes to be systematically related to some circumstances of the schools. It is entirely possible that if a direct randomized experiment were to be carried out testing how different schools respond to targets set in the 3 different approaches, that the small but positive penalties observed with the quantile regression and JTPA-type adjustments would be eliminated.

**<u>Table 2.  HOI Indicators with alternative definitions of targets</u>**

||Secretary of Education<br>Target|Target Based on<br>Quantile Regression at<br>67<sup>th</sup>Percentile|Target based on JTPA-<br>type adjustment to value<br>at67<sup>th</sup> percentile|
|---|---|---|---|
|Coverage (Pct meeting<br>the target)|52.27<br>(50.31 to54.23)|53.13<br>(51.07 to 55.18)|52.28<br>(50.23 to 54.34)|
|Dissimilarity Index|13.03|2.50|3.58|
||(6.64 to19.42)|(-5.24 to 10.24)|(-4.43 to 11.58)|
|HOI|45.46|51.79|50.41|
||(43.29 to 47.62)|(49.51 to 54.08)|(48.12 to 52.71)|
|Penalty|6.81|1.33|1.87|
||(6.04 to 7.59)|(0.35 to 2.30)|(0.87 to 2.86)|

# 5.  Looking at the entire unconditional and conditional distribution of performance

The results presented in Table 2 indicate that it is important to make some adjustment to initial conditions as compared with an across the board target of a 4% improvement, but it appears that there is not that great a difference between the quantile regression approach and the JTPA-type adjustment. However, the fact that in this particular example the differences are not that great does not mean that this will always be the case. It depends on how much the estimated coefficients vary across different quantiles. If the coefficients in the quantile regressions do not vary much, then it is not surprising that the adjustments based on an OLS would be close to those bases on a quantile regression. Indeed, the two approaches will 12
lead to equivalent targets for all schools if the coefficients in the quantile regressions are constant over all quantiles. Annex 1 plots the coefficients for 99 different quantile regression estimations and the plots reveal that the coefficients are not always constant. Of course, each case is different. But in general, one would not expect the coefficients in quantile regressions to always be constant – so there will be differences between the approach followed by JTPA (which implies a constant adjustment to each percentile) and the approach based on quantile regressions. Because it is just as easy to do the quantile regressions in STATA as the OLS adjustments and because the quantile regressions are more flexible, the quantile regression approach would appear to dominate the OLS JTPA-type adjustment.

# _Comparison of Empirical and CAP Distributions_

The discussion in the paper has focused on the setting of targets and, for that, one need only choose the percentile that is considered most relevant and then estimate a single quantile regression at that percentile. In addition to target setting, a policy maker might be interested in ascertaining whether an observed change in an indicator reflects performance that is good, bad or indifferent. For that it is useful to have some context against which to compare the outcome. One could compare the observed change with the empirical distribution of all past changes. However, the same considerations come into play as with target setting. The empirical distribution does not take into account the particular circumstances of the agent and to properly judge whether the performance has been good, bad or indifferent some consideration of the particular circumstances of the agent needs to be taken into account. Just as one estimated a target based on the quantile regression of the 67<sup>th</sup> and 25<sup>th</sup> percentiles, one can estimate 99 different quantile regressions and use the estimated coefficients to generate an estimated value if performance was equivalent to that of the 10<sup>th</sup> , 20<sup>th</sup> or nth percentile. We call the distribution of these estimated percentiles, the **CAP** or **C** haracteristic **A** djusted **P** erformance distribution Figure 6 plots the empirical distribution for the change in Portuguese proficiency between 2007 and 2006, together with the **CAP** distributions for the change in Portuguese proficiency for the same period for two selected schools in Minas Gerais.<sup>11</sup> School A was randomly selected from the set of schools that were in the top 25 percent of all schools in terms of the number of students in 2007 and in terms of the level of Portuguese proficiency in the 5<sup>th</sup> grade. School B was randomly selected from the set of schools that were in the bottom 25 percent of all schools in terms of the number of students in 2007 and in the level of Portuguese proficiency in the 5<sup>th</sup> grade. The red vertical lines indicate the values at the 25<sup>th</sup> and 67<sup>th</sup> percentiles.

> 11 Annex 2 provides information on a STATA program that has been developed to automate the process of generating the CAP distribution.

13
**Figure 5.  Comparison of Empirical distribution and CAP distributions for Two Schools in Minas Gerais, Brazil**

<!-- Start of picture text --> 0 20 40 60 80 100<br>Empirical distribution CAP distribution - School A<br>CAP distribution - School B<br>95 % Confidence Intervals given by shaded blue area<br>100<br>50<br>.<br>0<br>-50<br><!-- End of picture text --> While the **CAP** distribution for the smaller and less proficient school closely approximated that of the empirical distribution, that was not the case for the larger and more proficient school. Note that the 95 percent confidence intervals increase substantially at the extremes, largely below the 5<sup>th</sup> percentile and above the 95<sup>th</sup> percentile. From a practical standpoint, this should not create many serious problems. One would rarely want to set a performance target that is equivalent to the performance of the top 95<sup>th</sup> percent of the entire distribution. Similarly, one would rarely want to avoid the poor performance of just the bottom 5 percent of the distribution.

# _An example from Crime and Violence_

Another example of how one can make use of information on the entire CAP distribution is provided by an analysis of the performance of police departments in reducing homicides – again, in the state of Minas Gerais. As explained earlier, the State for Results program in Minas Gerais called for setting targets across a variety of sectors and was not focused just on achievement data on schools. Targets for reducing homicides were set, but this was done as an outcome of a negotiation – rather than setting a 4% 14
improvement as was the case in education. Information on the distribution of performance based on all available past changes and based on the information conditioned on characteristics relevant for the different police districts can help inform those negotiations.

For example, homicide data may be available for all police districts in a state (as was the case for Minas Gerais). The target that may be negotiated between the principal and the agent may be for improvement in homicides at the state level, but if the target is to be met, it will be up to individual police districts to improve the outcomes in each one of their districts. It is often the case that a sector will set targets not only at a high administrative level but also for sub state administrative units. Setting the same state-level target for each sub state administrative unit implies greater or less demands on performance improvement, given that the police districts are starting from different initial conditions. Comparing the targets to some of the changes that have taken place in the recent past may help provide some check on the reasonableness of the performance improvements which are being sought.

**Figure 6  – Unconditional relative performance on homicide reduction from 2000-2007, by Police Regions**

<!-- Start of picture text --> Trend 2000/2008<br>Barbacena<br>Trend 2004/2008 100<br>Vespasiano Belo Horizonte<br>Target 2008/2009<br>80<br>Unaí Contagem<br>60<br>Uberlândia 40 Curvelo<br>20<br>Uberaba 0 Divinópolis<br>Teófilo Otoni Governador Valadares<br>Patos de Minas Ipatinga<br>Montes Claros Juiz de Fora<br>Lavras<br><!-- End of picture text --> Source: Author’s calculations using data from Peixoto et al (2010) Figure 6 represents a simple way to present comparison for many police districts simultaneously. This type of figure is useful when the number of districts is limited, but would (obviously) not be useful for 15
presenting targets for individual schools where the number of individual schools would be in the thousands. Rather than presenting the entire empirical distribution for the indicator of interest (as in figure 5) and showing where the target falls in that distribution, Figure 6 presents the percentiles for recent changes, changes over a longer period and changes that would be implied by meeting the targets - all together for all the different police districts. The graph enables one to see quickly that there are some districts where the relative performance being asked of a district with its given target is not very different from both its medium-term and recent tendencies (Juiz de Fora), some districts where the targets are quite a bit more demanding than their medium-term tendencies, but not from their recent tendencies (Vespasiano, Belo Horizonte and Contagem) and some districts where meeting the targets would require some considerable shift from both their medium-term and recent tendencies (Teofilio Otoni and Montes Claros). Identifying where a district may have to up its game considerably should alert stakeholders to examine what improvements in policy or implementation are in place to suggest that there would be such a jump in relative performance.

Figure 6 made use only of the empirical distribution of changes in homicides. That is, the analysis did not try to control for differences in the initial conditions of the different police districts. It may well be that those involved in the target setting process may only be interested in comparing themselves against all changes that have taken place, irrespective of the conditions under which the changes took place. If that is the case, then the empirical distribution is all that is needed.

However, it is quite likely that stakeholders involved in a target setting process may wish to define some specific characteristics to control for in order to generate a comparison that they consider more relevant than a comparison to all other districts. If there were many more observations than what is normally available, one might form a comparator group by selecting some units to be included in the comparator group, dropping others. However, constructing a comparison group by dropping observations is not efficient and would greatly reduce the sample size. Instead, it is possible to retain all observations and carry out some multivariate statistical analysis to take into account differences in characteristics.

When working with the empirical distribution, the outcomes for a police district were judged to be a good or poor simply by noting where the particular police district's observed change fell in the empirical distribution of all observed annual changes. However, in an approach exactly analogous to the creation of the CAP distribution for education, one can estimate 99 different quantile regressions, multiply the 99 sets of coefficients by the characteristics of each district and create a CAP distribution for each police district. One can then compare where the actual performance fell relative to the unconditional distribution and the CAP distribution. Thus, when controlling for characteristics of the police districts, the observed change can be compared with a counterfactual distribution of predicted percentiles that is specific to each police district.<sup>12</sup> The final consideration is the choice of the characteristics to control for. The idea is not to explain the change in homicides, but to generate a standard for comparing performance that could be more relevant for the police district in question. Each government or interested party could pick the specific

> 12 A Taylor series expansion to calculate the 95% confidence interval of the predicted values. This technique is sometimes called the delta method  (for more details see Newman, Azevedo ,et all (2008)).

16
characteristics that it would like to control for in the comparison. The only requirement is that the characteristic be observable and that data exist for the units of analysis of interest.

Figure 7 presents comparisons across different police districts relative to their own CAP distributions. One can note by comparing Figures 8 and 7 that, taking account of the characteristics does make a slight difference to the comparisons of the targets to recent tendencies. In general, the targets appear somewhat less demanding relative to past performance, once one takes into account the particular characteristics of the police districts today.

**Figure 7.   Conditional relative performance on homicide reduction from 2000-2008, by Police Regions**

<!-- Start of picture text --> Trend 2000/2008 Barbacena<br>100<br>Trend 2004/2008 Vespasiano Belo Horizonte<br>90<br>Target 2008/2009 80<br>Unaí Contagem<br>70<br>60<br>50<br>Uberlândia 40 Curvelo<br>30<br>20<br>10<br>Uberaba 0 Divinópolis<br>Teófilo Otoni Governador Valadares<br>Patos de Minas Ipatinga<br>Montes Claros Juiz de Fora<br>Lavras<br><!-- End of picture text --> Source: Author’s calculations using data from Peixoto et al (2010) 17
6 **.** <u>Final considerations – Did the introduction of targets improve performance and how could this information on growth percentiles and CAP distributions be used in an incentive program?</u> As stated in the introduction, this paper is focused on a narrow question – how to go about setting performance targets – and it has explored ways in which past information can be used to help set a target. Assessing whether the introduction of the State for Results management shock, with its emphasis on setting targets, led to better performance lies beyond the scope of this paper.<sup>13</sup> However, the information on the percentiles does suggest that, perhaps, performance improved. For example, the percent of schools meeting the target in 2008 based on the 67<sup>th</sup> percentile of the distribution of changes between 2007 and 2006 was 53.1 percent. If performance had been stagnant, one would have expected 33 percent of the schools to have a performance greater than the 67<sup>th</sup> percentile. Just as one can set a performance target at the 67<sup>th</sup> percentile, one can set a performance target at any percentile and determine what percentage of schools met or exceeded that target. At the low end of the distribution, 86.2 percent of the schools in 2008 had a level of performance better than that of the 25<sup>th</sup> percentile. If performance had been stagnant, one would have expected 75% of the schools to have performance calculated on the basis of the 25<sup>th</sup> percentile of past performance. At the highest part of the distribution, 22 percent of the schools in 2008 had a level of performance higher than the 90<sup>th</sup> percentile (as opposed to 10 percent if performance had remained the same).

The discussion on target setting contained in this paper has been focused on setting an absolute level of a target, where the actors are informed of the value of the target prior to the period where they take decisions on effort. Recently, Barlevy and Neal (2011) have put forth an interesting proposal that attempts to generate incentives for higher performance not by setting an absolute target for agents to reach, but by setting up a competition across agents and offering a payment that consists of a common base and a bonus that is proportional to a percentile performance index that measures relative performance across agents facing comparable conditions.<sup>14</sup> Because it is structured as a competition among peers, there is no absolute target set ex ante. Their “Pay for Percentile” incentive is based on relative performance that is revealed ex-post. The authors argue that by structuring the incentives in this fashion, one can implement effective incentive schemes for teachers without ever forming aggregate

> 13 The distinctive pattern of the targets that were set by the Secretary of Education (as revealed in figure 1) might allow for the estimation of the impact of the introduction of the target setting program on performance through the use of a regression discontinuity approach.  But that is the subject of a different paper.

> 14 Barlevy and Neal (2011) describe the calculation of the percentile performance index as follows: “ For each student in a school system, first form a comparison set of students against which the student will be compared (generally speaking, will be those who begin at the same baseline achievement)... At the end of the year, give a cumulative assessment to all students.  Then, assign each student a percentile score based on his end of year rank among the students in his comparison set.  For each teacher, sum these within-peer percentiles scores overall  all students she teaches and denote this sum as a percentile performance index”.

18
statistical measures of what individual teachers have contributed to learning of all students in their classrooms. This does not provide information on changes in achievement over time (the performance measure is purely ordinal), but the authors suggest that education authorities can benefit from treating the provision of incentives and the documenting of student progress as separate tasks.

It is not immediately obvious that agents will respond in the same way to a relative target where they know they must do better than a comparator group, but they do not have an anchor of a given target to shoot for. This might be too abstract a goal and greater effort might be obtained by having a very clear goal. The approach described in this paper is also based on a relative notion – aiming for the 67<sup>th</sup> percentile of the quantile regression, but then that relative performance is converted to an absolute value that agents try to achieve. For that reason, the estimated target must be calculated from data that are available at the beginning of the period. The “Pay for Percentile” incentive would not have to use information at the beginning of the period. Since the incentive plan is a competition, only the rules must be fully specified at the beginning of the period. The actual determination of the percentile performance index can be calculated at the end of the period, once the data become available.

While the Pay for Percentile proposal was made for education, it could be useful to explore parallels with incentive plans for decentralized decision making. The actors in education are school districts, schools and individual students. The actors in decentralized decision-making plans are national governments, state and district level governments, but there are obvious parallels with arrangements within the education sector.

# 7.  Conclusions

This paper has considered the problem of selecting a reasonable performance standard and has illustrated how quantile regressions could be used to help generate a target that reflects the different characteristics of individual agents. This approach is a bit more flexible than the approach that was followed with the JTPA program in that it does not assume that the adjustment for the agents’ characteristics would necessarily be the same at different percentiles of the distribution of the performance standard. It is no harder to implement.

It is worth noting that the differences between the empirical and the **CAP** distributions were not all that great. It would, perhaps, be disturbing if they were. One might be inclined to conclude that this ability to take note of an agent’s initial conditions does not add much. However, the value of being able to calculate the **CAP** distributions has both a tactical and a technical value. The ability to take note of initial conditions makes it harder for an agent (a school, district or state, depending on the unit of analysis) to dismiss information on the distribution of performance as being completely irrelevant for them because of some factor which was not taken into account. Whatever observable factor an agent thinks should be taken into account could be taken into account in coming up with the CAP distribution.

The evidence suggests that school performance in Portuguese proficiency improved in 2008 following the introduction of targets into public sector management. However, a thorough analysis of the response to the introduction of the new system lies beyond the scope of this paper, which is focused on the issue of 19
how to make use of past information in setting a target. A more thorough analysis of the system would have to address whether the particular design of the incentive systems in Minas Gerais managed to generate positive improvements, while avoiding unintended outcomes and the possibility of welfare losses. It would also be important to analyze the dynamics of the system over time – how the agents reacted to the introduction of the targets and what subsequent modifications, if any, were made to the implementation of the public sector management approach. While a full assessment awaits this type of analysis, for the narrower task of selecting targets (which is an intrinsic element of performance management systems), we conclude that constructing CAP distributions and using targets based on the performance achieved across common percentiles for different agents could help ground the negotiations over performance standards.

20