metadata: authors: 0: author: Vinuela,Lorena
1: author: Zoratto,Laura De Castro
count: Brazil
docna: 0: docna: Do performance agreements help improve
            service delivery ? the experience of Brazilian states
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Reproductive Health,Early Child and Children's Health,Educational Sciences,Educational Institutions & Facilities,Effective Schools and Teachers,Crime and Society
teratopic: Governance,Public Sector Development
abstracts: cdata!: A growing number of states and
            municipalities in Brazil rely on results-based management,
            and many other local and state governments are considering
            adopting the practice. This paper examines the experiences
            of the Brazilian states that have implemented results
            agreements linked to variable pay. The analysis compares
            current with pre-intervention outcomes in the education,
            health, and security sectors. The changes are examined in
            relation to regional trends to determine whether the
            improvements depart in meaningful ways from the overall
            trend. In addition, a truncated time-series cross-section
            model is used to control for several additional factors
            influencing service delivery outcomes. The results suggest
            that, at least in the short and medium term, the
            implementation of results agreements is associated with
            significant and positive changes in outcomes in the security
            and education sectors. On average, states using team-level
            targets and performance-related pay have 15 fewer homicides
            per 100,000 inhabitants than those that do not, all else
            equal. Similarly, states that have introduced performance
            agreements and a bonus for teachers and school staff have
            improved their Basic Education Development Index score for
            public secondary schools by 0.3 additional points compared
            with the scores of states with similar characteristics. The
            conclusions are in line with the findings of in-depth impact
            evaluations and case study work in the education and
            security sectors (Bruns, Evans and Luque 2011, Milagres de
            Assis 2012). The paper does not analyze unit or team level
            data, which would be necessary to draw more rigorous
            conclusions about how results-based interventions affect the
            behavior of civil servants and outcomes over time.
            Therefore, the results should be interpreted with caution,
            as some of the assumptions behind the models cannot be
            examined with the available data.
geo_reg_key: 119228,517191
repnme: repnme: Do performance agreements help improve
            service delivery ? the experience of Brazilian states
display_title: Do performance agreements help
            improve service delivery the experience of Brazilian states
listing_relative_url: /research/2015/07/24814238/performance-agreements-help-improve-service-delivery-experience-brazilian-states
topicv3: Teachers Management m1326119 1523,Infant and Child Mortality m1326544 1446,Education m1327048 470,Homicide m1439222 HOM-3046,Health m1504661 3480
publishtoextweb_dt: 2015-07-23T00:00:00Z
historic_topic: Governance,Public Sector Development
no_of_pages: 37
author_keys: 0: author_key: 1042897
1: author_key: 1090139
model_extractions: WPS7375 Policy Research Working Paper 7375 Do Performance Agreements Help Improve Service Delivery? The Experience of Brazilian States _Lorena Viñuela Laura Zoratto_ Governance Global Practice Group July 2015
Policy Research Working Paper 7375

# **Abstract**

A growing number of states and municipalities in Brazil rely on results-based management, and many other local and state governments are considering adopting the practice. This paper examines the experiences of the Brazilian states that have implemented results agreements linked to variable pay. The analysis compares current with pre-intervention outcomes in the education, health, and security sectors. The changes are examined in relation to regional trends to determine whether the improvements depart in meaningful ways from the overall trend. In addition, a truncated time-series cross-section model is used to control for several additional factors influencing service delivery outcomes. The results suggest that, at least in the short and medium term, the implementation of results agreements is associated with significant and positive changes in outcomes in the security and education sectors. On average, states using team-level targets and performance-related pay have 15 fewer homicides per 100,000 inhabitants than those that do not, all else equal. Similarly, states that have introduced performance agreements and a bonus for teachers - and school staff have improved their Basic Education Devel opment Index score for public secondary schools by 0.3 additional points compared with the scores of states with similar characteristics. The conclusions are in line with the findings of in-depth impact evaluations and case study work in the education and security sectors (Bruns, Evans and Luque 2011, Milagres de Assis 2012). The paper does not analyze unit or team level data, which would be necessary to draw more rigorous conclusions about how resultsbased interventions affect the behavior of civil servants and outcomes over time. Therefore, the results should be interpreted with caution, as some of the assumptions behind the models cannot be examined with the available data.

This paper is a product of the Governance Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at lvinuela@ worldbank.org and lzoratto@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Do Performance Agreements Help Improve Service Delivery? The Experience of Brazilian States<sup>1</sup>

Lorena Viñuela and Laura Zoratto

**JEL Classification** : JEL: D02 Institutions: Design, Formation, Operations, and Impact, H11 Structure, Scope, and Performance of Government, and O54 Latin America • Caribbean. **Keywords** : Brazil, Performance-based management, variable pay, performance agreements, service delivery, education, health, and security.

1 The analysis builds on the World Bank’s extensive and innovative engagement on public sector reform at the state level in Brazil since 2007, notably including the work led by Deborah L. Wetzel, Roland Clarke, and Tarsila Velloso. This paper is directly linked to the various past and on-going state-level operations in Brazil. The authors especially acknowledge the contributions of Ana Mie H. Reis (LCSPS) provided valuable research assistance. William Dillinger, Philip Keefer, Barbara Nunberg, and Pedro Olinto offered important advice and insights for the research agenda as peer reviewers.
# **I. INTRODUCTION**

This paper reviews performance management models implemented by the states of Minas Gerais, Pernambuco, Rio de Janeiro, and Ceará, and examines whether one of its variants— results agreements linked to variable pay—has led to meaningful improvements in education, health, and security outcomes. The analysis identifies questions for future in-depth work on the types of interventions and conditions under which results-based management (RBM) can contribute to greater performance gains.

Measuring performance has always been a core and important part of the public administration literature and practice. However, in the last two decades, growing attention has been given to the design and implementation of RBM systems and considerable resources have been dedicated to collecting performance data. Some of the most commonly adopted systems include performance-related pay, total quality management, managerial flexibility, strategic planning, performance measurement, and contracting out.

The movement emerged first in developed countries as a response to tighter fiscal conditions, declining public sector efficiency, rising expectations on service delivery, and new complex policy issues that require the action of multiple actors to solve them (Kettl 2000). In that context, results-oriented management was seen as a ‘new way of conducting public business’ that could address the problems created by the traditional bureaucratic model (Osborne and Gaebler 1992).

Performance management models rapidly disseminated across developed and developing countries. This propagation sparked the interest of students of public management and generated a copious body of literature on the subject, particularly in the context of OECD countries. In spite of this, systematic analyses of the impact of performance management on service delivery remain limited, and the reported results mixed. The available evidence shows, nonetheless, that outcomes are often contingent on the overall environment in which performance models are implemented. For these to translate into sustained gains, they require adjustments and perfecting over time.

Similarly, in Brazil, the RBM model is being used by an increasing number of states and municipalities, but the literature on the subject is thus far largely normative or descriptive. The few existing empirical pieces identify performance gains resulting from linking rewards to improvement targets in the education and security sectors (Bruns, Evans, and Luque 2011, Milagres de Assis 2012). However, others find that performance interventions may result in unintended outcomes (Odelius and Santos 2007, Rodrigues 2009, and Milagres de Assis 2012).

Examining what types of reforms have been adopted at the subnational level in Brazil and how these influence service delivery outcomes is of great policy relevance. At the same time, given the within-country variation, Brazil offers an exceptional opportunity to assess comparatively the impact of these reforms and extract lessons for other countries. To address this knowledge gap, the paper focuses on the experiences of states that have adopted results-based models involving performance agreements and systematic monitoring of progress against targets. Performance 2
agreements are seen as a key mechanism through which managers and civil servants align their actions with goals; they also provide a clear platform for accountability and organizational learning over time.

The present paper draws on background qualitative sectoral analysis and provides an assessment of the four selected state case studies, describing the features of the different models and interviews conducted with state civil servants. The analysis compares current with preintervention outcomes in the education, health, and security sectors. The changes are examined in light of regional developments to determine if the improvements depart in meaningful ways from the overall trend. Furthermore, a fixed effect model is used to control for additional factors influencing service delivery and outcomes.

The results of the empirical analysis support the hypothesis that the implementation of teamlevel results agreements is associated with significant and positive changes in outcomes in the security and education sectors, at least in the short and medium term. Using these tools, Minas Gerais has achieved positive results in the education sector that go beyond the gains made by other states in the southeastern region. This state also has a lower homicide rate than other states with the same GDP per capita. Similarly, Pernambuco has reversed a rising homicide rate and recorded considerable gains in the education sector. In the case of Rio de Janeiro, the introduction of performance agreements and a teacher bonus coincided with the substantial improvements recorded in educational achievements. Ceará is using a results monitoring system that has translated into sizable gains in the education and health sectors.

While the analysis points to a positive effect of results agreements in combination with performance-related pay, it is important to note that the adoption of these models did not take place in isolation. They required clear planning and the disaggregation of goals to the team level. Fiscal reforms and efforts to strengthen planning and budgeting functions were important preconditions for the adoption of results-based management. In the education sector, the introduction of a common metric and the development of state systems for monitoring learning outcomes was crucial for making possible the implementation of the reforms as well as motivating management change. The studied states also took other complementary steps, such as creating special careers with a differentiated pay scale within the civil service for planning and management professionals. Some states also established permanent education programs and government schools to train staff and facilitate behavioral change.

Finally, it is important to underscore that this paper does not analyze unit- or team-level data, which would be necessary to draw more rigorous conclusions about how results-based interventions affect civil servants’ behavior and outcomes over time. Therefore, the results should be interpreted with caution, as some of the assumptions behind performance interventions cannot be tested with the available data.

The paper offers a brief overview of the literature on the subject in section II. Section III describes the results-based models adopted in the four case studies. Section IV presents trends in education, health, and security outcomes for the four states and section V correlates the 3
implementation of results-based management with changes in those state-level outcomes. Section VI discusses additional evidence supporting the conclusion, and section VII concludes.

# **II. IN SEARCH OF BETTER OUTCOMES**

This section summarizes some of the most relevant insights from the literature on results-based management. The review highlights potential benefits and unintended consequences associated with performance management to guide the empirical analysis of their effects.

# _The Case for Performance Management_

Results-based management is a process that involves defining and monitoring indicators to measure and improve the performance of organizations and programs on a regular basis (Poister 2008). The targets could be set at the agency, unit, team, and individual level, and/or have various objectives. Some of the most common concentrations include operating efficiency, costeffectiveness, productivity, quality, and citizen satisfaction. Performance management systems are typically designed to support a particular process—strategic planning, budgeting and financial management, program management and evaluation, quality/process improvements, and/or communication with stakeholders.

The approach is based on the premise “that management matters to performance and effectiveness, and that performance is the ultimate goal of public management systems and actions” (Moynihan 2005: 2). It relies on the adoption of systems that provide managers and organizations with greater discretion, techniques to measure and improve performance, and market-like mechanisms to replace input-focused and hierarchical steering with results-oriented management. By focusing on outcome measures, results-based models give managers more space to adapt programs to different or changing conditions.

Measuring performance is expected to impact positively the behavior of employees and managers and motivate people and organizations to improve productivity by shifting the focus from processes and activities to results (Behn 2003). Critically, these systems can contribute to better aligning organizational activities and processes with the organization’s goals and connect activities and outputs to outcomes. The information allows principals to compare achievements in programs and services against targets, support processes of internal change and adjust programs as needed.

However, performance systems and interventions are embedded within an institutional and political context and should not be considered in isolation (Radin 2006). While the interventions may contribute to improving performance by affecting distinct subsets of the incentives in place, the institutional context influences the extent to which these interventions may, in fact, be able to drive performance. The combination of institutional context and interventions has come to be referred to in the performance literature as a “performance regime” (Moynihan 2008b). As explained by Talbot (2010: 81), considering performance management systems together with 4
their context is particularly useful because “what evidence does exist (...) suggests that it is the totality of a performance regime which potentially shapes or steers performance for specific services rather than the narrow purchase-provider or principal-agent assumptions often made about performance drivers.” Translating results-based management into sustained gains requires adjustments and perfecting over time. In the United States, local and state governments have had different degrees of success in implementing performance management models. Yet, those that have continued to develop and refine them have observed positive impacts on outputs and the efficiency of public spending (Moynihan 2006). In the United Kingdom, where the broadest array of results-based models has been introduced, the government recorded sizable improvements in output and outcome indicators during the first years of their implementation. However, subsequent gains have been smaller. There is also evidence that performance interventions can in some cases lead civil servants to focus excessively on the aspects of their work that are being measured (Hood and Wilson 2009).

_Performance Agreements as a Central Mechanism for Improving Results_ Performance agreements are one of the tools or modalities of results-based management. They rely on a consultative process to develop a document or work plan setting goals and parameters of quality and productivity for a given department, team, or individual. They normally detail the external factors that may influence results and assessments. The targets are derived from the overall organizational goals and the intermediary hierarchy of results.

Performance agreements are a critical mechanism through which interventions drive results improvements. Agreements help ensure that managers and employees are fully aware of the targets and the intermediary steps needed to achieve them. In doing so, this process substantially reduces informational asymmetries between principals and agents. As they produce specific commitments and deploy resources, agreements also provide a way of integrating goals into a common policy vision and thus facilitate cooperation. To be successful, performance agreements need to set feasible and reasonable targets and have clear and credible consequences.

The process of preparing the agreements and setting targets contributes to building intrinsic motivation and increasing staff engagement. Empirical and theoretical work in developed countries suggests that performance measurement systems are more effective in positively influencing managers’ and civil servants’ behavior if these buy into the system (Rainy 2003). In turn, the likelihood of having commitment and ownership regarding performance targets is higher if managers and employees are involved in the process of developing them and defining the related measures (Poister and Streib 1999, Moynihan and Ingraham 2003).

In addition, performance agreements contribute to solving frequent problems of over-aggregation of performance data encountered in results measurement by breaking down goals and indicators. Crucially, they provide data at a level that is useful for managers and allow individuals and teams to take concrete steps towards achieving results. Performance agreements also set the 5
channel and the frequency of the reporting (Behn 2003). The data extracted from these agreements not only inform managers but are also useful for external accountability purposes.

# _Potential Concerns around Performance Management_

Several concerns have been expressed about the implementation of performance management that should also be considered. Performance measurement systems can generate large amounts of descriptive information, which require time and effort to be processed and distributed. If the information does not reach managers and employees in a timely and accessible fashion, it runs the risk of being ignored (Poister 2008).

Additionally, large amounts of information can lead to ‘analysis paralysis’ while too little can lead to tunnel vision. By focusing on a specific set of goals, results-based systems could neglect other objectives that are more difficult to quantify. In the same vein, rewarding specific behaviors may leave out or even discourage intrinsic motivation and have an overall negative impact on the organization’s performance.

It is also possible that, over time, performance indicators could lose their capacity to capture performance and discriminate between individuals who have a high performance from those who have a low one. And, that over time performance result may tend to concentrate on targets, rather than follow a normal distribution (Dooren, Bouckaert, and Halligan 2010).

A leadership and organizational culture that does not reinforce individual or team achievements can limit the use of performance information (Moynihan 2008b). Also, if internal communication and employee participation in the goal-setting stage are not sufficiently strong, the impact of information is likely to be more limited (Poister and Streib 1999).

Another note of caution is that results-oriented management may be adopted because of mimicry or normative goals rather than on the basis of evidence. The diffusion of this approach, as with many other management practices, could have been the result of the assimilation of practices by organizations that carry out similar functions (Di Maggio and Powell 1983). In many cases, institutions adopt the management structures or practices of organizations that they perceive as efficient and whose performance they want to emulate, regardless of whether those techniques have an effect on performance.

Likewise, targets could be used as minimum pledges of performance rather than tools for stretching and improving efficiency (James 2004). If targets are misrepresented or are frequently changed, it can undermine the credibility of the entire system. Nevertheless, adopting performance-based models can be a very effective strategy for principals to signal priorities and an entry point for broader reforms that streamline processes and structures.

6
<!-- Start of picture text --> \ | } Goids Dist<br>Planning Performance Agreements 0153 6 Decimal Degrees<br><!-- End of picture text -->
Minas Gerais adopted performance agreements linked to variable pay in 2004 and universalized them to all sectors in 2008. Such agreements were implemented in Pernambuco in the security, education, and health sectors in 2007, and in the state of Rio de Janeiro in the security and education sectors in 2009 and 2010, respectively. In these cases, goal setting, performance review, and rewards are group-based (at the secretariat, school, hospital, or unit level depending on the case). A portion of the pay state employees receive is linked to the achievement of a goal set for the relevant group. The variable pay is typically substantial, being equal or higher than a month’s salary. Ceará adopted other performance-based management tools, including strategic planning and high-level monitoring of priority projects. This state is also planning to adopt a results agreement and a productivity bonus in the education sector.

The adoption of performance management tools by the federal government and a number of the state and municipal governments has been part of a broader set of reforms introduced in the Brazilian public sector. Within the Federal Government, alongside with fiscal and economic reforms, the restructuring and downsizing of the public sector in the mid-1990s was complemented with several measures aimed to strengthen careers and to attract and retain a qualified labor force. An important consideration driving the introduction of performance assessment and bonuses ( _Gratificação de Desempenho_ ) was to improve the remuneration of the qualified staff without having to grant linear readjustments to the entire workforce (Pires 2005). While subnational reforms have been influenced by the initiatives at the federal level, in many cases reforms have been more comprehensive and innovative at the state or local level (Abrúcio and Gaetani 2006).

These results-based models have not been implemented in isolation, but rather they have been preceded and accompanied by a wide-ranging set of reforms. Fiscal reforms and efforts to strengthen planning and budgeting functions have been important preconditions for the adoption of results-based management.<sup>3</sup> In the education sector, the introduction of a common metric— the Basic Education Development Index ( _Índice de Desenvolvimiento da Educação Básica_ or IDEB)—and state monitoring systems for learning outcomes that allow gauging relative performance of every school, municipality, and state has facilitated the measurement of results and the application of performance-related rewards. Similarly, in the health sector the introduction of a unified health system ( _Sistema Único de Saúde_ or SUS) and a set of common indicators have had a similar effect. These common metrics have not only been effective in providing a basis for comparison across states and over time, but they have also been important in motivating reform.

> 3 In the case of Minas Gerais, the initial round of the performance agreement had fiscal rationalization as a key objective.

8
**Table 1: Timeline of Performance-based Interventions in Selected Cases**

||2003|2004|2005|2006|2007|2008|2009|2010|2011<br>2012|2013|
|---|---|---|---|---|---|---|---|---|---|---|
|||||||||||Results|
||||||||||Management<br>Shock<br>_Pacto pela_<br>|Agreements<br>in the<br>Security|
|**Bahia**|||||||||_Vida_|Sector|
||Management<br>||||Priority<br>Projects<br>||||||
|**Ceará**|for Results||||Monitoring||||||
|||Results|||||||||
|||Agreements|||Second||||Management||
|||in Revenue|||Generation||||for Citizens<br>||
|||Mobilization|||Results||||(Emphasis in||
||Management|and Some|||Agreements||||citizen||
|**Minas Gerais**|Shock|Sectors|||in All Sectors||||participation)||
||||||Management<br>Shock /_Pacto_<br>||||||
||||||_Pela Vida_|Results|||||
||||||Results<br>|Agreements<br>|||||
||||||Agreements<br>|in the<br>|||||
||||||in the Security|Education|||||
|**Pernambuco**|||||Sector|Sector|||||
|||||||||Results|||
||||||New Strategic|Results||Agreements|||
||||||Plans for<br>|Agreement||in the<br>|||
||||||Education and|Security||Education|||
|**Rio de Janeiro**|||||Security|Sector||Sector|||
||||||Management<br>Shock<br>Results<br>||||||
||||||Agreements<br>||||||
|**Rio Grande**|||||in Revenue||||||
|**do Sul**|||||Mobilization||||||
||||||Management<br>Shock<br>Results<br>||||||
||||||Agreement in<br>||||||
||||||Education and||||||
|**São Paulo**|||||Health||||||

9
Additionally, the studied states have taken other complementary steps, such as creating special careers for planning and management professionals within the civil service with a differentiated pay scale. In some cases, these specialists have been deployed in front-line service units or at the level where performance is being measured to provide technical assistance and support to managers. A few states have also established permanent education programs and government schools to train staff and facilitate behavioral change.

The states that chose to implement results-based models shared similarly challenging conditions at the outset, although their institutional capacity and resource endowments varied. Public sector performance in key areas of service delivery, such as health or education, exhibited stagnant, smaller-than-expected, and often mediocre results. Security had deteriorated in the late 1990s, and early 2000s across the country, and these states had to tackle rapidly rising levels of crime. At the same time, they faced difficult fiscal conditions and new restrictions on borrowing. In turn, the limited fiscal space for new programs or investments meant that they had to focus on spending more strategically and efficiently. Moreover, the northeastern states had to grapple with a historical legacy of high levels of poverty and limited socio-economic development.

Poor public sector performance was driven, among other factors, by difficulties in attracting qualified professionals, low public salaries, and high levels of absenteeism of front-line staff. Within the civil service, there were few tools for motivating performance available to managers, as pay was based on educational attainment, training, and seniority. The politicization of appointments and deeply rooted clientelistic practices in the recruitment of civil servants, together with the presence of numerous spoilers, complicated matters. The governors implementing such programs met opposition from civil servants, unions, and legislators.

While implementing results agreements and performance-related pay may be costly,<sup>4</sup> part of the motivation behind the reform was to provide an increase in civil servants’ remuneration that distinguishes high from low performers. A complementary goal was to de-politicize and improve the transparency of promotions and appointments. On the other hand, the states have benefitted from having continuity in the political leadership and policy priorities. While reforms have been politically difficult in the short run, in most cases they have given governors and their parties greater public approval ratings and electoral gains, sometimes even helped projecting these leaders into the national sphere.

Importantly, in some cases there is evidence of institutional learning and that governments are using lessons from previous cycles to improve program design and implementation. In the case of Minas Gerais, the longest running performance-based system, the iterative application of the system lead to considerable adjustments. The lessons from the implementation of the first three management cycles were reflected in the introduction of additional performance agreements that

> 4 The state of Minas Gerais spent an estimated BRL 720 million between 2004 and 2010, while the state of Rio de Janeiro invested BRL 140 million in 2010 and BRL 538.5 million in 2011 in teachers’ bonuses.  In 2013, Rio de Janeiro budgeted BRL 590.2 million for the variable pay in education.

10
disaggregated goals at the team level. Also, to design their performance management programs, states are using and building on the lessons from other states. The experience in Minas Gerais informed reforms in Pernambuco. The state of Rio de Janeiro introduced performance agreements and a teacher bonus modeled after Pernambuco’s but has added additional requirements to mitigate free-riding and reduce absenteeism. The Rio de Janeiro model is now informing reforms in Acre, while the Pernambuco model for the security sector is also being replicated in Bahia, and will soon be adopted by Amazonas.

Some of the most relevant features of the programs examined are succinctly described below.

# _Minas Gerais_

The state of Minas Gerais first introduced performance agreements in 2004, as part of the Management Shock ( _Choque de Gestão_ ) policy initiated a year earlier. This policy was aimed primarily at promoting fiscal balance, reducing costs, and modernizing the state government through the use of new management models. One of the new tools introduced was the use of performance agreements. A first round of performance agreements were signed by the governor and two-thirds of the state secretariats. Initially, 340 indicators derived from the strategic plan were defined to monitor and systematically assess performance across the sectors. A productivity bonus was established for secretariats that met specific goals, along with spending rationalization and revenue generation targets.

In 2008, a second tier of performance agreements was added between line secretariats and the teams responsible for the implementation of programs and priority projects ( _projetos estruturadores_ ). First-level agreements (between the governor and the secretaries) follow the strategic priorities set in the multiannual plan, while in the second-level agreements (between the secretaries and the implementing teams) greater discretion has been given to secretaries and managers to define targets for their teams. The productivity bonus was universalized in 2008 as well. The second-stage agreements detail and give different weights to four items: achievement of outcome targets, completion of priority projects, sectoral objectives, and rationalization of spending. A committee reviews both levels of agreements and grants a productivity prize to the teams that have met their targets. This bonus constitutes a sizable incentive, as it can represent up to a month of basic salary. The bonus acquired renewed importance after the introduction of the second-stage agreements because these allow to link better strategic goals to annual programs.

The implementation of this policy was accompanied by intense training programs and efforts to attract qualified staff. The state recruited a new cadre of young private sector managers to help change the culture within the state administration. A number of information systems were deployed to support the process. In addition to adopting performance agreements and setting expenditure reduction targets, the program cut positions and reduced by a third the number of departments. The functions of planning and management were integrated into a same unit, the State Department of Planning and Management.

11
# _Pernambuco_

In 2007, a results-based management model was adopted to integrate planning, budgeting, and implementation. The ‘ _Pernambuco para Todos’_ program put a strong emphasis on changing processes and facilitating problem-solving and coordination across the entire state government. Initially, the model was based on three main components: (i) a participatory process for defining priorities, involving consultations with civil society organizations in all regions; (ii) the adoption of high-level monitoring meetings, in which senior officials are held accountable for the achievement of targets; and (iii) the use of data in solving problems, with the continuous monitoring of project performance and outcomes. In subsequent years, additional actions were undertaken, including expanding training opportunities, deploying monitoring and evaluation specialists in front-line teams, and the introduction of group-based bonuses.

Special policies or pacts were adopted in the security sector in 2007 and in the education and health sectors in 2011. These pacts provided a more focused approach to monitor goals. The first one, the Pact for Life ( _Pacto pela Vida_ ), has as primary goal of reducing violent deaths in the state by mobilizing and fostering collaboration among different government institutions and civil society. With specific targets for six areas,<sup>5</sup> data collection and analysis is used to monitor program performance. Sectoral committees ( _Câmaras Técnicas_ ) have been established to coordinate the activities of 12 state secretariats, the Prosecutor’s Office, and municipalities. A bonus is paid to the staff of the secretaries that meet yearly targets. The education and health pacts were created in 2011 with the objectives of universalizing access to education and increasing the supply and quality of health services, respectively.

Prior to the adoption of the education pact, the state had introduced variable pay in the sector in 2009. The system rewards schools for meeting the improvement targets in IDEB scores. Schools that achieve 50 percent or more of their target receive a proportional bonus. The bonus is at least one month’s salary, but depending on the proportion of schools that meet their targets it can be much higher. Targets are set following different rules, depending on which quartile of the performance distribution the school falls into, creating discontinuities in how ambitious targets can be (Bruns, Evans, and Luque 2011).

The state had also adopted several reforms in the education sector. In 2001, a program called Democratic School ( _Escola Democrática_ ) established a new modality to appoint school principals for three-year terms through an election by teachers, parents, and students. The candidates are prescreened, and only the three top-ranked ones are allowed to run for the position. Several additional actions have been taken to improve the quality of the state school system. These include a new curriculum with detailed guidelines on materials to be taught in each grade, improved provision of books and materials, and the posting of teaching assignments and class schedules to incentivize parents to monitor teacher attendance (Bruns, Evans, and Luque 2011). A new cadre of education management advisors ( _técnicos de gestão educacional_ )

> 5 The six areas are: social prevention of crime and violence, qualified repression of violence, institutional improvement, information and knowledge management, training and education, and democratic management.

12
was created in 2009 and assigned to every school to monitor key indicators and support the implementation of individual school improvement plans.

In the health sector, performance agreements and related rewards were used for the first time in 2011–2012. In parallel, some complementary changes were introduced. A civil service career was created for public management specialists in 2008 with the goal of supporting agencies in the development of results-based practices. In 2011, an Executive Secretariat of Management Development was created within the Secretariat of Planning and Management, and a program of continued training was established.

# _Rio de Janeiro_

Following a decade of acute security problems and dismal results in the education sector,<sup>6</sup> the state of Rio de Janeiro designated public security as a priority area in 2007 and education in 2010. Interventions in the prioritized sectors were guided by new strategic plans and involved the establishment of new units to fast-track programs and monitor results, the training and recruitment of new staff, and the introduction of productivity bonuses to incentivize staff and front-line personnel.<sup>7</sup> In the security sector, Pacifying Police Units ( _Unidades de Polícia Pacificadora_ or UPPs) and a performance management model and bonus for security forces were introduced in 2008 and 2009, respectively. The program is implemented through a partnership including local, state, and federal governments as well as different actors of civil society. Its objective is restoring law and order in communities dominated by drug trafficking gangs and increasing the footprint of the state and its proximity to citizens. By 2013, 33 UPPs had been established, covering approximately 231 communities.

In the education sector, the state adopted a new plan at the beginning of 2011 with the aim of improving learning performance and establishing a meritocratic selection process for key functions—such as the regional directors—that have traditionally been captured by the state’s Assembly. The program provides bonuses to school units and Regional Boards that meet or exceed annual improvement goals. The bonus is considerable, as it can be up to three times the basic monthly salary. The goals are not assigned to each individual, but rather to the school. However, to be eligible to receive a bonus payment teachers have to fulfill certain requirements. Teachers have to cover 100 percent of the minimum curriculum, participate in all internal and external evaluations, and have at least 70 percent attendance during the school year.<sup>8</sup>

> 6 In 2009, the state of Rio de Janeiro had the second lowest score in the Basic Education Development Index (IDEB) for secondary students in the country.

> 7 The Environment Secretariat is attempting to introduce a similar model.

> 8 The Municipality of Rio de Janeiro is using a similar system, but with stronger sanctions against teacher absenteeism.  Only teachers and school employees that have five or fewer absences in the school year are eligible to receive the bonus (Bruns, Evans, and Luque 2011).

13
In order to assess goals, Rio de Janeiro, like Minas Gerais and Pernambuco had done before, adopted an annual version of the IDEB evaluation and three other evaluations. As well, a career plan for regional directors and principals was established, and a competitive recruitment process was made mandatory to fill those positions. By the end of 2012, all 28 regional directors and 172 school principals had been hired through a competitive selection process, despite the strong opposition from state representatives.

Permeating the reform agenda was a strong focus on data, management and implementation. To allow school directors to focus on education quality, young, information technology-savvy managers were assigned to each school, which improved the timeliness of school data reporting and budget management rapidly. The Secretary established a routine meeting that brought the ― ― entire state management team over 400 people together every six months to review progress, celebrate successes and collectively address problems. The new model put priority on assessing the performance of, and a developing a career plan for, every superintendent and principal.

# _Ceará_

Since the late 1980s, successive governments have reformed Ceará’s public administration and invested heavily in social programs. These efforts have been reflected in economic growth, balanced public finances, sustainable levels of indebtedness, and significant improvements in social welfare. Access to education and health services has improved markedly over the past two decades, and there has also been significant progress in the quality of service provision.

Building on earlier reforms, Ceará has introduced numerous results-based elements in the state management in the last decade, including the adoption of a new strategic government plan in 2003. The plan sought to improve fiscal space and modernize the state’s management practices, and focused government strategic objectives under four programs: _Ceará Empreendedor; Ceará Vida Melhor; Ceará Integração;_ and _Ceará a Serviço do Cidadão_ . A system of detailed indicators was developed to facilitate monitoring and program evaluation and a Committee for Results-based Management and Fiscal Administration _(Comitê de Gestão por Resultados e Gestão Fiscal)_ was established to oversee the implementation of the state’s results management model. The Committee is formed by the secretaries of Planning, Finance, Internal Audit, and Casa Civil and meets in a ‘situation room’. Since then, progress in the implementation of priority programs is regularly reported to the Committee. The introduction in 2007 of an integrated Priority Actions and Projects Monitoring system ( _Monitoramento de Ações e Projetos Prioritários_ ), which allows high-level decision makers to track projects from their design to their implementation, further strengthened the capacity of the state to monitor investment programs.

To date, the model has focused on the implementation of the priority investments and functions mainly as an information tool for the Governor and the Cabinet. However, there are various elements in place that can help facilitate the introduction of a more complete performance-based management model. Ceará has already adopted an effective system for setting framework goals, through a medium-term planning framework, and for transmitting these goals to the heads of the sector agencies. Holding bimonthly meetings has proven to be a useful tool for uncovering 14
problems and finding multi-sectoral responses when warranted. Lastly, Ceará is planning to introduce performance agreements and a bonus for teachers.

# **IV. MEASURING RESULTS: HOW STATES USING PERFORMANCE MANAGEMENT FARE**

This section examines how the performance-based systems fare in the health, security, and education sectors in the states using results agreements and variable pay. The before-and-after outcome measures are compared to learn if there are any discernible discontinuities that coincide with the implementation of results agreements. The states’ trajectories are related to the national and regional trends, which serve as a possible counterfactual of what the likely path would have been without the intervention. In the following section, these discontinuities are further examined using a truncated time-series model to establish if those discontinuities are statistically significant.

It is important to stress that the validity of the analysis is limited because the reforms studied have not been designed as randomized controlled treatments, with clear treatment and control groups. The adoption of a results agreement system is not likely to be random. Moreover, in some cases this model has been adopted in a system- or government-wide fashion. To mitigate problems of causal inference created by the lack of a control group, a regression discontinuity, and difference-in-difference designs are used to approximate, albeit imperfectly, random treatment assignment.

In this section, only nationally comparable outcome indicators are used—they include child mortality, homicide rates, and IDEB scores. Some caveats should be highlighted when considering the results. Outcome indicators can be difficult to interpret. There are often time lags between changes in outputs and outcomes. Outcomes can be affected by external factors that are beyond the service providers’ control—education and health outcomes, for instance, may be influenced by the socio-economic composition of the population served and changes in economic circumstances. There are ways of controlling for these factors, but they add complexity and require additional information. Besides, responsibilities for the provision of health, education, and security services are shared across levels of government and attributing results to state interventions can in some cases be difficult. One exception is secondary education, which is the sole responsibility of state governments.

# _State and Regional Trends_

Comparing a state and regional averages helps address the potential threat that events or factors external to the state (federal or regional programs, economic or political shocks, among others) are responsible for the observed changes in outcomes. In the case of Brazil, many of the social indicators have improved across the country as a result of federal programs, such as the cash transfer program _Bolsa Família_ or the universalization of healthcare coverage, and good economic performance. In addition, it is possible to make such comparisons because the 15
examined states are similar to the other states in their respective regions in terms of population, population density, GDP per capita, and poverty levels, which are some of the key control variables used in this analysis (Table 2). However, it should be noted that Minas Gerais falls below the regional average in terms of GDP, and Rio de Janeiro has a higher-than-average population density.

**Table 2: Selected States and Regional Comparators, 2002–2003**

||**State**|**Other States in**<br>**the Region**|**Difference**|
|---|---|---|---|
|**Pernambuco (Northeast)**||||
|Population (millions), 2002–2003|7.92|4.98|2.94|
|Population Density (per sq. km), 2002–2003|80.37|49.83|30.54|
|GDP per Capita (constant 2000), 2002–2003|3,538.35|3,062.30|476.05|
|Poverty,2002–2003|60.80|59.21|1.59|
|**Minas Gerais (Southeast)**||||
|Population (millions), 2002–2003|17.90|18.20|‐0.28|
|Population Density (per sq. km), 2002–2003|30.50|181.69|‐151.19|
|GDP per Capita (constant 2000), 2002–2003|5,762.00|9,041.67|‐3279.67**|
|Poverty,2002‐2003|26.08|23.20|2.88|
|**Rio de Janeiro (Southeast)**||||
|Population (millions), 2002–2003|14.40|19.30|‐4.95|
|Population Density (per sq. km), 2002–2003|328.59|82.33|246.26***|
|GDP per Capita (constant 2000), 2002–2003|9,357.82|7,843.06|1,514.76|
|Poverty,2002‐2003|22.89|24.27|‐1.38|

_Source_ : Authors’ calculations, based on data from IBGE. _Note_ : *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed).

**_Health sector_ .** Child mortality has declined rapidly across the country, with the national average rate dropping 34 percent from 24.9 deaths per 1,000 born alive in 2002 to 16.4 in 2011.  The largest reductions have been achieved in the northern states and Alagoas in the northeast, which initially had rates above 40.  On the other extreme, Amapá was the state in which the reduction in child mortality was the smallest.

Regarding the states of interest, both Pernambuco and Ceará are among the states that have observed the largest reductions between 2007 and 2011 in child mortality relative to their initial levels (Figure 1). These states also have lower child mortality rates than the predicted value for their GDP per capita (Figure 2). On the other hand, Minas Gerais falls on the average rate predicted for its GDP per capita while Rio is above it.

Consistent with the trend for the Northeast region, Pernambuco and Ceará have rapidly and steadily reduced child mortality rates since 2002. Child mortality in Pernambuco was higher than the regional average and the national rate in 2002. By 2011, the state had surpassed the regional reduction average and was below the national average. Figure 3 shows that from 2006 to 2007 the slope abruptly changes but then reverts to a similar rate of decline as before. This change coincides with the introduction of the _Pernambuco para Todos_ policy. Ceará displays a 16
similar change, suggesting that there may have been a common program or change affecting some states in the region. However, the change is not as noticeable for Bahia.

**Figure 1: Child Mortality Change 2007–2011 and Child Mortality in 2007**

**Figure 2: Child Mortality and GDP per Capita, 2011**

<!-- Start of picture text --> AP<br>AP<br>DF RSSCSOSPPRESRJSE MWRRGOMGMT ROBR TOAMNR PA MA PI PA BAACTO AM MT<br>MS PIBA AL PB RNSE RO<br>CE AC MA CE PE RR GOMG<br>PENERNSEPB MS RJ<br>PRESRSSC SP DF<br>AL<br>10 20 30 40 8 8.5 9 9.5 10<br>Child Mortality (per 1,000 born alive) 2007 LN GDP per Capita (constant 2000)<br>95% CI Fitted values 95% CI Fitted values<br>Difference Child Mortality (per 1,000 born alive) 2007-2011 Chil Mortality (per 1,000 born alive)<br>Source : Ministry of Health, SVS 2013. Source : IBGE, Ministry of Health, SVS 2013.<br>Figure 3: Child Mortality in Bahia, Ceará, and Figure 4: Child Mortality in Minas Gerais and Rio de<br>Pernambuco (per 1,000 born alive), 2002–2011 Janeiro (per 1,000 born alive), 2002–2011<br>PE: MG: MG:<br>Pernambuco Performance Performance<br>para Todos Agreements Agreements<br>– 1 stage – 2 stage<br>2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br>Year Year<br>Bahia Ceará Minas Gerais Rio de Janeiro<br>Northeast Pernambuco Southeast<br>10 25<br>0 20<br>-10 15<br>-20 10<br>-30 5<br>40 40<br>30 30<br>20 20<br>Chil Mortality (per 1,000 born alive) Chil Mortality (per 1,000 born alive)<br>10 10<br><!-- End of picture text --> _Source_ : Ministry of Health, SVS 2013.

**Figure 3: Child Mortality in Bahia, Ceará, and Pernambuco (per 1,000 born alive), 2002–2011**

**Figure 4: Child Mortality in Minas Gerais and Rio de Janeiro (per 1,000 born alive), 2002–2011**

_Source_ : Ministry of Health, SVS 2013.

_Source_ : Ministry of Health, SVS 2013.

In the case of Minas Gerais, there is no clear discontinuity in its declining trend that could be linked to the 2004 introduction of performance agreements in the health sector or their refinement in 2008 (Figure 4). While the rate decreased by an average of 3.7 percent between 2007 and 2011, child mortality levels remain above the regional average, and the distance between the two has been fairly constant. Minas Gerais’ trend is similar to that of the southeast region and that of Rio de Janeiro, a state that does not use results agreements in the health sector.

**_Security Sector._** After three consecutive years of decline, the national homicide rate per 100,000 inhabitants started increasing again in 2007.  Since that time, only eight states have recorded reductions (Figure 5).  The largest drop was observed in Pernambuco and Rio de Janeiro, with a

17
28 and 47.5 percent reduction in their homicide rate respectively. Roraima, Sergipe follow them, and Mato Grosso do Sul. Yet, it is important to bear in mind that these states had some of the highest homicide rates in the country, and their rates remain higher than the value predicted by GDP (Figure 6).

**Figure 5: Homicide Rate Change, 2007–2011**

**Figure 6: Homicide Rate and GDP per Capita, 2011**

<!-- Start of picture text --> PB<br>AM AL<br>RN BA<br>GO AL<br>TO CE SE PA<br>MA NE<br>SC PI SP ACRSSOMG NRBRAPRORRMWDFPRMTMS SE ES MA PB CE PARNSEPEBAACAPTO RRAMROGOMGMS MTPRESRSRJ DF<br>RJ<br>PE PI SC SP<br>10 20 30 40 50 60 8 8.5 9 9.5 10<br>Homicide Rate (per 100,000 inhabitants) 2007 LN GDP per Capita (constant 2000)<br>95% CI Fitted values 95% CI Fitted values<br>Difference Homicide Rate (per 100,000 inhabitants) 2007-2011 Homicide Rate (per 100,000 inhabitants)<br>Source : Ministry of Health, SVS 2013. Source : IBGE, Ministry of Health, SVS 2013.<br>Figure 7: Homicide Rate in Bahia, Ceará and Figure 8: Homicide Rate in Minas Gerais and Rio de<br>Pernambuco (per 100,000 inhabitants), 2002–2011 Janeiro (per 100,000 inhabitants), 2002–2011<br>PE: PPV - Bonus MG: Performance MG: 2 nd stage<br>Agreements– 1 st stage<br>RJ: PBM<br>Agreements - Bonus<br>2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011<br>Year Year<br>Bahia Ceará Minas Gerais Rio de Janeiro<br>Northeast Pernambuco Southeast<br>20 80<br>10 60<br>0 40<br>-10 20<br>-20 0<br>60 60<br>50 50<br>40 40<br>30 30<br>20 20<br>10 10<br>Homicide Rate (per 100,000 inhabitants) Homicide Rate (per 100,000 inhabitants)<br>0 0<br><!-- End of picture text --> _Source_ : Ministry of Health, SVS 2013.

**Figure 7: Homicide Rate in Bahia, Ceará and Pernambuco (per 100,000 inhabitants), 2002–2011**

**Figure 8: Homicide Rate in Minas Gerais and Rio de Janeiro (per 100,000 inhabitants), 2002–2011**

_Source_ : Ministry of Health, SVS 2013.

_Source_ : Ministry of Health, SVS 2013.

When examining regional patterns, the declining trend in Pernambuco contrasts with the increasing incidence of homicide in the rest of the region, including in reformer states such as Ceará. Pernambuco had a homicide rate of 53 in 2007, but by 2011 it had dropped to 39.2. The change in trend corresponds well with the introduction of the Pact for Life policy and a performance-related reward for state employees in the security sector (Figure 8). Although the state’s homicide rate is rapidly converging to the (growing) regional average, Pernambuco’s homicide rate is still well above the national average.

In the southeast, Minas Gerais’ homicide rate has remained below the regional average and below the level predicted for its GPD per capita. Currently, the state has the second-lowest rate 18
in the Southeastern region after São Paulo and the sixth lowest in the country. The state achieved considerable reductions between 2004 and 2010, although it was surpassed by São Paulo, which currently has the lowest rate in the region. It is, nonetheless, important to highlight that the introduction of performance-related pay in 2004 coincided with the change in homicide trends, but recently the trend seems to have decelerated, even recording a small increase in 2011.

Between 1999 and 2004, the homicide rate had more than doubled in the state of Minas Gerais, reaching the highest point in the state’s contemporary history. During that period, the state trend was going in the opposite direction than that of the other three states, which had started to see declining homicide levels since 2002. But comparing with the regional average is difficult because Rio de Janeiro particularly high rates, driving the average up.

In the case of Rio de Janeiro, homicide rates have been in decline since 2002, and the gap with the regional average is closing. The rate was cut in half in a short period. But despite the sizable gains, homicide levels remain high.

**_Education sector_ .** The national average IDEB score for public secondary schools has improved at an average of 4.3 percent every two years since the index was adopted in 2005. However, the improvement observed in the 2005–2013 period was more modest than that observed in primary education.  Between 2007 and 2013, Goiás, Pernambuco, and Rio de Janeiro had much larger improvements in their scores than other states that had similar scores at the beginning of the period (Figure 9).  Many observed no change or declining scores.  Still, among others, Ceará and Minas Gerais have scores above the predicted value for their GDP, while Rio de Janeiro and most southern and northeastern states still falls far below the score predicted for its income level.

**Figure 9: IDEB Public Secondary School Score Change in 2007–2013 and IDEB Score in 2007**

**Figure 10: IDEB Public Secondary School Score and GDP per Capita, 2013**

<!-- Start of picture text --> GO<br>PE<br>RJ<br>PI GO<br>PA SE RS SP<br>RO SPRS PE MG SCRJ<br>SE APNEAM NR MWCE ES RO MS PRES<br>RN PB RRTO DF MG CE AC DF<br>MA AL BA BR AC MS TO RR<br>SO PI PB AM<br>SC AP<br>MT PR MA SE BA<br>RNPA MT<br>AL<br>2 2.5 3 3.5 4 8 8.5 9 9.5 10 10.5<br>IDEB Secondary School Score 2007 LN GDP per Capita (constant 2000)<br>95% CI Fitted values 95% CI Fitted values<br>Difference IDEB Secondary School Score 2007-2013 IDEB Secondary School<br>1 4.5<br>4<br>.5<br>3.5<br>0<br>3<br>-.5 2.5<br><!-- End of picture text --> _Source_ : INEP 2013.

_Source_ : INEP 2013.

19
**Figure 11: IDEB Public Secondary School Score in Bahia, Ceará and Pernambuco (0-10), 2005-2013**

**Figure 12: IDEB for Public Secondary School Score in Minas Gerais and Rio de Janeiro (0-10), 2005-2013**

<!-- Start of picture text --> PE: PE: Performance Bonus Schools MG: Performance MG:– RJ: Performance Bonus<br>PPT Agreements 2 stage<br>1 stage<br>2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014<br>Year Year<br>Bahia Ceará Minas Gerais Rio de Janeiro<br>Northeast Pernambuco Southeast<br>4 4<br>3.5 3.5<br>3 3<br>IDEB Secondary School IDEB Secondary School<br>2.5 2.5<br><!-- End of picture text --> _Source_ : INEP 2013.

_Source_ : INEP 2013.

In the northeast, Pernambuco shows improvements in the IDEB scores since 2009. These gains differ from the flat trend for the Northeast region. Several states in the region recorded no change or even reductions in their scores, as in the case of Bahia and Ceará.

In the southeast, Minas Gerais showed steady improvement in the IDEB grade for secondary education between 2005 and 2011 but retroceded in 2013. The introduction of performance agreements is associated with the implementation of school-level performance agreements. While the state remains above the regional average, the gap is rapidly closing because Rio de Janeiro and Espiritu Santo recorded large improvements.

Similarly, in the case of Rio de Janeiro, there is a clear discontinuity that is concomitant to the introduction of the results agreements and bonus system for school employees in 2010. After showing no change in three rounds of IDEB, the state recorded a sizable increase of 14 percent in 2011 and 12.5 percent in 2013. As a result, Rio de Janeiro rose from the 23<sup>rd</sup> position in the national ranking of IDEB 2009 to 5<sup>th</sup> place in 2013. The state achieved one of Brazil’s most remarkable turn-arounds in the education sector.

# _Before-and-After Comparison of Outcomes_

The comparison of the average measures corresponding to the periods of time before and after the introduction of performance agreements and variable pay shows that in several cases there are significant differences in outcomes after the implementation of the intervention (Table 3). The before-and-after design is a quasi-experimental model that compares two groups of observations, before and after time points. The state-years preceding the program are averaged to estimate the counterfactual and the later ones to measure results.

20
**Table 3: Comparison of Means Before and After the Adoption of Performance-Related Pay**

|**Befo**|**re**<br>**Afte**|**r**<br>**Di**|**fference**|
|---|---|---|---|
|**Pernambuco**||||
|Child Mortality (per 1,000 born alive), 2007–2011|33.98|19.84|‐14.14***|
|Homicide Rate (per 100,000 inhabitants), 2007–2011|52.78|45.52|‐7.26***|
|IDEB Public Secondary School Score, 2007–2011|2.70|3.10|0.40|
|IDEB Public Secondary School Score, 2008–2013|2.70|3.23|0.53**|
|**Minas Gerais**||||
|Child Mortality (per 1,000 born alive), 2004–2011|20.4|17.38|‐3.03***|
|Homicide Rate (per 100,000 inhabitants), 2004–2011|18.6|20.68|2.08*|
|Child Mortality (per 1,000 born alive), 2008–2011|18.95|16.53|‐2.43***|
|Homicide Rate (per 100,000 inhabitants), 2008–2011|20.68|19.45|‐1.06|
|IDEB Public Secondary School Score, 2008–2011|3.45|3.63|0.18***|
|**Rio de Janeiro**||||
|Homicide Rate (per 100,000 inhabitants), 2008–2011|49.87|34.45|‐16.29***|
|IDEB Public Secondary School Score, 2010‐2013|2.80|3.20|0.60***|

_Source_ : Authors’ calculations, based on data from IBGE and MEC. _Note_ : *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed).

In the case of Pernambuco, there are substantive and statistically significant reductions in child mortality and homicide rates, and an increase in the IDEB score for public secondary schools after 2007. For Minas Gerais, there are two cut-off points, after 2004 when the first stage performance agreements were introduced and after 2007 when an additional agreement was added between Secretaries and teams. In both cases, there are statistically significant reductions in child mortality vis-à-vis the average from the previous period. For the second comparison, there is a significant difference in the secondary school outcome measure.

For the state of Rio de Janeiro, there is a substantial and statistically significant reduction in the average homicide rate when comparing the period before and after the adoption of a resultsbased model and the introduction of a performance-related bonus for the state security forces. The negative trend in homicide rates had started prior to the adoption of the results-based approach and the UPPs. In secondary education outcomes, the IDEB results for the state show a statistically significant increase equivalent to a fifth of the prior score.

This method is, however, vulnerable to threats to internal validity, in particular, the threat of external events affecting outcome scores. Given that there are, in fact, reasons to think that the after series might be different from the before series even without the treatment, the results could be spurious. To address this risk, the results are also compared with those of the other states in the respective regions. For each of the groups, the differences between changes over time 21
(statistically adjusted) are compared. The cut-off years differ from sector to sector and for different states.

**Table 4: Comparison of Means After Adoption of Variable Pay with Other States in the Region**

||**Before**||**After**||
|---|---|---|---|---|
||**Difference**|**State**|**Other States**<br>**in the**<br>**Region**|**Difference**|
|**Pernambuco (Northeast)**|||||
|Child Mortality (per 1,000 born alive), 2007–2011|‐0.25|19.84|21.81|‐1.97|
|Homicide Rate (per 100,000 inhabitants), 2007–2011|32.56***|45.52|31.20|14.32**|
|IDEB Public Secondary School Score, 2007–2013|0.05|3.10|2.88|0.22*|
|IDEB Public Secondary School Score, 2008–2013|0.01|3.23|2.90|0.30**|
|**Minas Gerais (Southeast)**|||||
|Child Mortality (per 1,000 born alive), 2004–2011|5.75***|17.38|13.94|3.44***|
|Homicide Rate (per 100,000 inhabitants), 2004–2011|‐18.99***|20.68|36.64|‐15.96***|
|IDEB Public Secondary School Score, 2004‐2013||3.56|3.30|0.26*|
|Child Mortality (per 1,000 born alive), 2008–2011|3.32|16.53|13.07|3.46***|
|Homicide Rate (per 100,000 inhabitants), 2008–2011|‐22.67***|19.63|33.79|‐14.17**|
|IDEB Public Secondary School Score, 2008–2011|0.36|3.62|3.43|0.20|
|**Rio de Janeiro (Southeast)**|||||
|Homicide Rate (per 100,000 inhabitants), 2008‐2011|17.64***|33.58|29.14|4.43|
|IDEB Public Secondary School Score, 2010‐2011|‐0.59***|3.40|3.60|‐0.20|

_Source_ : Authors’ calculations, based on data from IBGE and MEC. _Note_ : *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed).

In the case of Pernambuco, while child mortality has decreased by a sizable amount after 2007, the reduction is not significantly different from that of other states in the northeast region. In the education sector, the state has surpassed the average for the region, and it is improving at a faster rate its IDEB score. In the security sector, the change is not only statistically significant but also goes in a different direction than the regional average. During that period, the gap between the state and the rest of the region has decreased by 18.3 points is the combined result of a sizable reduction of the homicide rate in Pernambuco and its increase in the rest of the states.

When comparing the changes in Minas Gerais to other states in the southeast region, it is possible to conclude that only improvements in education outcomes go beyond the general trend. In child mortality, the change is very similar, and the difference between the state and the region remains fairly constant. The homicide rate for the period after the introduction of the Management Shock policy is lower than the rest of the region, but on average other states have had larger reductions.

22
The comparison for the state of Rio de Janeiro reveals that the state’s improvement in the IDEB score is larger than the average change of other states in the region. The state is converging fast with the regional averages. Because there are only two observations after the introduction of the bonus system, it is not yet possible to determine whether there is statistical significance. On the other hand, the homicide rate for the years 2008 and 2011 has significantly decreased in the state of Rio de Janeiro, but on average the rest of the states in the region did not register a significant change. The gap between the state and the region is quickly closing, and the difference is no longer statistically significant.

# **V. EMPIRICAL ANALYSIS**

Using an interrupted time-series model of the state health, security, and secondary education indicators data from 2002 to 2011, this section further analyzes if any systematic change in outcome measures can be attributed to the introduction of results agreements linked to variable pay. Fixed effects are added to control for unobserved time-invariant factors associated with particular states.

Using “state-year” as the unit of analysis, the following regression model is used: Where i = 1,2,....; N; refers to a cross-sectional unit; t = 1,2,....; T; refers to a time period; and k = 1,2,....; K; refers to each specific explanatory variable. Yit and Xit refer respectively to the dependent and independent variables for unit i and time t; and eit is a random error and β0 and βk refer, respectively, to the intercept and the slope parameters. The variable T is the treatment dummy in this case scored 0 for all periods in the before series and 1 for all periods after the intervention, β1 is the treatment effect, in this case the difference in intercept t, on the vertical space between the lines repressing the before and after series. The variable M is a dummy for state-years in which government-wide management reforms, known in Brazil as ‘management shocks,’ have been implemented. The disturbance term is assumed to be randomly distributed to the periods in the series.

Using this technique to estimate the pay-related effects on key outcomes, child mortality, the homicide rate, and the IDEB score are regressed on a dummy variable for the state-years in which the model was in place. The panel comprises ten yearly observations for 26 states and the Federal District. Each observation represents the aggregate state measure.

Partial models are used to testing the explanatory power of the treatment and control for some relevant factors (Table 5). The control variables include sector expenditure per capita (to proxy for the resources allocated to the sector), the poverty rate and GDP per capita (to proxy for the level of economic development), and GDP per square km and the population density (reflecting density effects).

23
**Table 5: Multivariate Regressions**

|**Dependent variable:**|**IDEB**|**Secondary S**|**chool Score (**|**0‐10)**||**Child M**|**ortality**|||**Homic**|**ide Rate**||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|**Results Agreements and Variable Pay**|0.34|0.32|0.24|0.26|‐6.51|‐1.76|‐2.67|‐0.20|‐7.47|‐14.78|‐10.84|‐14.78|
||(0.07)***|(0.09)***|(0.07)***|(0.07)***|(1.98)***|(1.88)|(1.47)*|(1.43)|(2.38)***|(2.89)***|(2.12)***|(2.53)***|
|**Management Shock**||0.07||‐0.08||‐10.73||‐6.41||8.90||5.31|
|||(0.07)||(0.07)||(1.37)***||(1.08)***||(2.12)***||(1.93)***|
|**Sector Expenditure per Capita**|||0.0004|0.0004|||0.001|‐0.0004|||‐0.002|‐0.003|
||||(0.00)*|(0.00)|||(0.01)|(0.004)|||(0.01)|(0.01)|
|**Poverty**|||‐0.02|‐0.02|||0.55|0.47|||‐0.56|‐0.53|
||||(0.00)***|(0.004)**<br>*|||(0.06)***|(0.06)***|||(0.07)***|(0.07)***|
|**GDP per Capita (constant 2000)**|||0.0001|<br>0.00|||0.001|0.001|||‐0.003|‐0.003|
||||(0.00)**|(0.00)**|||(0.001)**|(0.001)|||(0.001)**<br>*|(0.001)**<br>*|
|**GPD per Sq. Km**|||0.00|0.00|||0.00|0.00|||<br>0.00|<br>‐0.002|
||||(0.00)***|(0.00)|||(0.00)|(0.00)|||(0.00)|(0.01)***|
|**LN Population**|||0.22|0.21|||‐1.47|‐1.77|||6.08|6.31|
||||(0.75)***|(0.08)***|||(1.43)|(1.33)|||(2.08)***|(2.05)***|
|**Population Density (per sq. km)**|||‐0.004|‐0.004|||‐0.12|‐0.09|||0.05|0.03|
||||(0.003)|(0.00)|||(0.06)*|(0.06)|||(0.09)|(0.09)|
|**Constant**|3.07|3.05|1.88|1.97|21.84|23.64|11.82|18.33|28.75|27.71|23.70|‐14.78|
||(0.01)***|(0.01)***|(0.69)***|(0.60)***|(0.32)***|(0.35)***|(10.47)|(9.85)*|(0.39)***|(0.45)***|(15.22)|(12.54)**<br>*|
|**N**|243|243|243|243|270|270|270|270|270|270|270|<br>270|
|**N Groups**|27|27|27|27|27|27|27|27|27|27|27|27|
|**R Square**|0.08|0.07|0.50|0.511|0.03|0.03|0.42|0.67|0.03|0.11|0.31|0.33|
|**F‐Ratio**|25.64|38.56|23.28|27.20|10.73|237.48|37.98|42.40|9.83|14.05|15.27|14.67|
|**Prob > F**|0.00|0.00|0.00|0.00|0.00|0.00|0.00|0.00|0.00|0.00|0.00|0.00|

*** Significant at the 0.01 level (2-tailed);  ** Significant at the 0.05 level (2-tailed).

24
The results reveal that there are statistically significant differences in education and security outcomes for the states implementing performance-based management, but not in the health sector. States that have performance-related rewards have reduced the child mortality rate by an average of an additional 6.51 deaths per 1,000 children born alive than those that do not. However, the size of the difference shrinks and the statistical significance disappears when controls are added. Instead, GPD per capita and general management reforms explain the difference.

On average, states implementing performance-related pay have 14.78 fewer homicides per 100,000 inhabitants than those that are not, all else equal. Spending BRL 100 more in personnel per capita decreases the homicide rate by one decimal point. A BRL 100 increase in GDP per capita is reflected in a three decimal point reduction in the homicide rate.

Similarly, all else equal, states that have introduced results agreements and a bonus for teacher and school staff have improved the IDEB score for public secondary schools by 0.26 additional points. The results should be interpreted with caution, as some of the predictions in the literature hinge on assumptions whose relevance the available data cannot examine. It is also difficult to generalize across sectors. Properly analyzing the effects may require longer time series and, therefore, it may be too early to draw definitive conclusions.

# **VI. OTHER EMPIRICAL EVIDENCE**

Despite the limitations of our analysis, the conclusions are in line with the findings emerging from in-depth impact evaluations and case study work that explore the mechanisms through which performance-related pay influences the behavior of public servants and ultimately affects outputs and outcomes in a given sector. They also highlight that results are contingent on the environment in which policy interventions are implemented.

# _Emerging Results from Impact Evaluations_

In the education sector, there are several impact recent evaluations that allow a better understanding of the design features and conditions under which teacher bonuses are most effective. The preliminary information available for Pernambuco supports the expectation that results agreements in combination with variable pay positively impacts learning outcomes. In the first year the variable pay system was implemented, 52 percent of schools met targets and received a bonus, while in the second year 79 percent of schools did (Bruns, Evans, and Luque 2011). Those schools that had more ambitious targets achieved the more progress, all else equal. Similarly, schools that narrowly missed receiving the bonus for the first round appear to have improved more than schools that barely achieved the bonus (Ferraz and Bruns forthcoming). In the comparable program implemented by the municipality of Rio de Janeiro, in 2010 290 of the 1,044 municipal schools qualified for the bonus on the basis of their IDEB improvements (Bruns, 25
Evans, and Luque 2011). But more than 11,000 employees did not meet the minimum criteria to be eligible for the bonus.

In the security sector, there is also evidence supporting the results. Milagres de Assis (2012) analyzes the effect of Minas Gerais’ result-based management in the security sector; more precisely, on results achieved by the Military Police ( _Polícia Militar_ ), the Investigative Police ( _Polícia Civil_ ), and the Fire Department. Survey data and in-depth interviews with policemen, firemen, and senior managers reported by the author show that the establishment of goals and financial incentives created profound changes in these agencies. Increases in productivity were especially sizable in the Investigative Police. In the Fire Department, the new incentive structure nudged the institution to conceive new strategies, and generated important productivity gains, and improvements in service delivery outcomes. However, Milagres de Assis (2012) also warns that the strong focus on the number of police investigations in the Military Police led to a reduction in their quality. The absence of obstacles made it possible for some police officers to reclassify crimes to improve their departments’ statistics in the key indicators that were being monitored. In sum, results agreements and financial incentives are very powerful and may generate positive or negative results, depending on the way they are implemented.

# _Perception Data_

In addition to having increased political support from the population, performance management appears to be associated with greater satisfaction rates in the security and education sectors. In this section, we draw on data from the AmericasBarometer and the System of Social Perception Indicators (SIPS) of the Brazilian Institute of Applied Economics Research _(Instituto de Pesquisa Econômica Aplicada or_ IPEA _)_ . These two surveys ask questions related to citizens’ perceptions of different public policies, including the three sectors of focus in this paper, to a representative sample of households. Because the surveys are relatively new, it is not possible to compare changes over time. However, we can weigh how states using performance management get rated against other states.

The level of citizens’ trust in the Military Policy, for example, is considerably higher in the states of Pernambuco and Minas Gerais, when compared with their respective regional averages (Figure 12 and Figure 13). Ceará and Bahia also have above-average ratings in this question. The difference of over ten percentage points is statistically significant in both cases.

26
**Figure 13: Trust in the Military Police in Northeastern States (positive responses), 2012**

<!-- Start of picture text --> 60<br>50<br>40<br>30<br>20<br>10<br>0<br>percentage<br><!-- End of picture text --> _Source_ : AmericasBarometer, LAPOP, 2012. _Note_ : Blue line represents regional average.

**Figure 14: Trust in the Military Police in Southeastern States (positive responses), 2012**

<!-- Start of picture text --> 60<br> 50<br> 40<br> 30<br> 20<br> 10<br> -<br>Espírito Minas Rio de São Paulo<br>Santo Gerais Janeiro<br>percentage<br><!-- End of picture text --> _Source_ : AmericasBarometer, LAPOP, 2012.

Likewise, over half of the interviewees reported that the quality of education has improved in the states of Ceará, Pernambuco, and Minas Gerais (Figure 15 and Figure 16). These states have averages that are above those of their respective region. However, other states that were not implementing performance management at the time that the survey was conducted, such as Bahia, Piauí, Maranhão, and Espírito Santo, also scored high marks.

**Figure 15: Perceptions of Quality of Education in Northeastern States (percentage), 2012**

<!-- Start of picture text --> Sergipe<br>Rio Grande do Norte<br>Piaui<br>Pernambuco<br>Paraíba<br>Maranhão<br>Ceará<br>Bahía<br>Alagoas<br>0 20 40 60 80 100<br>percentage<br>Improved No change Worsened<br><!-- End of picture text --> _Source_ : System of Social Perception Indicators (SIPS), IPEA, 2012.

**Figure 16: Perceptions of Quality of Education in Southeastern States (percentage), 2010**

<!-- Start of picture text --> São Paulo<br>Rio de Janeiro<br>Minas Gerais<br>Espírito Santo<br>0 20 40 60 80 100<br>percentage<br>Improved No change Worsened<br><!-- End of picture text --> _Source_ : System of Social Perception Indicators (SIPS), IPEA, 2012.

The state of Pernambuco has the highest percentage of respondents reporting good quality in health services. Bahia, which has used private-public partnerships in the health sector, also scores high, along with Alagoas. Nevertheless, health sector results are less differentiated in the Southeast region.

27
**Figure 17: Perceptions of Quality of SUS Services in Northeastern States (percentage), 2012**

<!-- Start of picture text --> Sergipe<br>Rio Grande do Norte<br>Piaui<br>Pernambuco<br>Paraíba<br>Maranhão<br>Ceará<br>Bahía<br>Alagoas<br>0 20 40 60 80 100<br>percentage<br>Good Intermediate Poor<br><!-- End of picture text --> _Source_ : System of Social Perception Indicators (SIPS), IPEA, 2012.

**Figure 18: Perceptions of Quality of SUS Services in Southeastern States (percentage), 2010**

<!-- Start of picture text --> São Paulo<br>Rio de Janeiro<br>Minas Gerais<br>Espírito Santo<br>0 20 40 60 80 100<br>percentage<br>Good Intermediate Poor<br><!-- End of picture text --> _Source_ : System of Social Perception Indicators (SIPS), IPEA, 2012.

# **VII. CONCLUSION AND OUTLINE OF A PROPOSED RESEARCH AGENDA**

The paper explores the relationship between performance management and changes in sectoral outcomes using a variety of methods. The results of the empirical analysis support the expectation that the implementation of team-level results agreements and performance-related pay is associated with significant and positive changes in outcomes in the security and education sectors, at least in the short and medium term.

Minas Gerais registered positive results in the education sector, surpassing the average of the southeastern region. This state also has a lower homicide rate than other states with the same GDP per capita. Similarly, Pernambuco has reversed a rising homicide rate and recorded considerable gains in the education sector through the implementation of performance agreements. In the case of Rio de Janeiro, the introduction of performance agreements and a teacher bonus corresponded with the first positive change recorded in educational achievements since 2005. Although Ceará is not using performance-related pay yet, its monitoring system and results-based management model has translated into sizable and consistent gains in the education and health sectors.

It is important to underscore that the paper does not analyze unit or team level data, which would be necessary to draw more rigorous conclusions about how results-based interventions affect the behavior of civil servants and outcomes over time. Therefore, the results should be interpreted with appropriate caution.

However, from an empirical perspective, the relationship between performance management and outcomes in service delivery remain largely understudied, in particular in the context of developing countries. Having reviewed the existing evidence, this paper highlights the pressing need to expand data collection and analysis efforts at the subnational level. Many of the 28
predictions found in the literature and the assumptions that these hinge on can only be tested using more disaggregated data. In order to properly address the question of what the impact of performance agreements is, it would be important to drill down and compare units within the state and over time, ideally through a panel.

Furthermore, an agenda for analytical work to more directly address the question at hand requires several efforts. First, completing and regularly updating an inventory of state and municipal results-based management reforms and practices will facilitate their monitoring. Second, changes in behaviors and outcomes should be observed over a longer period to confirm if the gains recorded in the early years of implementation are sustained in the long term. Third, there is a need to examine more carefully how interventions affect existing employees and to what extent the gains observed are the result of a personnel turnover. Anecdotally, in many of the cases, there have been important efforts to recruit new employees and more qualified applicants have been attracted by the new environment and pay incentives.

Finally, future work should also seek to understand better which factors make these reforms succeed and what explains how reform champions make it to strategic decision-making positions in the first place. Likewise, more in-depth empirical work is needed to identify the sources of breakdowns in their implementation.

29