metadata: authors: 0: author: Abdollahian, Mark
1: author: Barma,Naazneen
2: author: Green, Amanda
3: author: Nunberg,Barbara
4: author: Perlman,Deborah Gail
count: East Asia and Pacific
docna: 0: docna: At the frontier of practical political
            economy : operationalizing an agent-based stakeholder model
            in the World Bank's East Asia and Pacific Region
docty: Policy Research Working Paper
subtopic: Climate Change Mitigation and Green House Gases,Social Analysis,Quality of Life & Leisure,National Governance,Government Policies,Mining & Extractive Industry (Non-Energy),Administrative & Civil Service Reform,Public Sector Administrative & Civil
            Service Reform,Youth and Governance,Democratic Government,Public Sector Administrative and Civil
            Service Reform,De Facto Governments
teratopic: Environment,Law and Development,Public Sector Development,Social Development,Finance and Financial Sector Development
abstracts: cdata!: Reform programs sometimes falter because
            they are politically infeasible. Policy change inevitably
            creates winners and losers, so those with vested interests
            strike bargains to determine how far and how quickly reform
            should advance. Understanding these micro political dynamics
            of reform can mean the difference between a successful
            intervention that gains political traction and a
            well-intentioned gambit that falls short of achieving its
            developmental objectives. Donors like the World Bank have
            been searching for ways to take these political factors more
            fully into account as they design programs to support
            country reforms. This initiative sought to introduce a
            rigorous and operationally usable political analysis tool
            that could be systematically integrated into the World
            Bank's country programming cycle. The East Asia and
            Pacific region carried out a multi-country pilot of the
            Agent-Based Stakeholder Model. This innovative analytical
            approach entails a quantitative simulation of the complex
            bargaining dynamics surrounding reform. The model
            anticipates stakeholder coalition formation and gauges the
            political feasibility of alternative proposed interventions.
            This paper provides a review of the Agent-Based Stakeholder
            Model pilot experience, exploring what sets this model apart
            from more traditional approaches, how it works, and how it
            fits into the Bank's operational cycle at various
            stages. An overview of the Mongolia, Philippines, and
            Timor-Leste country cases is followed by an examination of
            policy-related insights and lessons learned. Finally, the
            paper builds on this East Asian pilot experience, offering
            ideas on a potential way forward for organizations like the
            World Bank to deepen and extend their political analysis
            capabilities. The paper argues that the Agent-Based
            Stakeholder Model, utilized thoughtfully, offers a powerful
            addition to the practical political economy toolkit.
repnme: repnme: At the frontier of practical political
            economy : operationalizing an agent-based stakeholder model
            in the World Bank's East Asia and Pacific Region
display_title: At the frontier of practical
            political economy : operationalizing an agent-based
            stakeholder model in the World Bank's East Asia and
            Pacific Region
listing_relative_url: /research/2010/01/11641285/frontier-practical-political-economy-operationalizing-agent-based-stakeholder-model-world-banks-east-asia-pacific-region
topicv3: Political Economy of Reform m1326263 836,Mining Services m1326410 1701,Climate Change Mitigation m1327079 579,Mining Sector m1327113 554,Human Resources for Public Sector m1327490 724
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Environment,Law and Development,Public Sector Development,Social Development,Finance and Financial Sector Development
no_of_pages: 32
author_keys: 0: author_key: 167168
1: author_key: 92072
2: author_key: 957913
model_extractions: WPS5176 Policy Research Working Paper 5176 At the Frontier of Practical Political Economy Operationalizing an Agent-Based Stakeholder Model in the World Bank’s East Asia and Pacific Region _Barbara Nunberg Naazneen Barma Mark Abdollahian Amanda Green Deborah Perlman_ The World Bank East Asia and Pacific Region Poverty Reduction and Economic Management Sector Unit January 2010
Policy Research Working Paper 5176

# **Abstract**

Reform programs sometimes falter because they are politically infeasible. Policy change inevitably creates winners and losers, so those with vested interests strike bargains to determine how far and how quickly reform should advance. Understanding these micro political dynamics of reform can mean the difference between a successful intervention that gains political traction and a well-intentioned gambit that falls short of achieving its developmental objectives. Donors like the World Bank have been searching for ways to take these political factors more fully into account as they design programs to support country reforms.

This initiative sought to introduce a rigorous and operationally usable political analysis tool that could be systematically integrated into the World Bank’s country programming cycle. The East Asia and Pacific region carried out a multi-country pilot of the AgentBased Stakeholder Model. This innovative analytical approach entails a quantitative simulation of the complex bargaining dynamics surrounding reform. The model anticipates stakeholder coalition formation and gauges the political feasibility of alternative proposed interventions. This paper provides a review of the AgentBased Stakeholder Model pilot experience, exploring what sets this model apart from more traditional approaches, how it works, and how it fits into the Bank’s operational cycle at various stages. An overview of the Mongolia, Philippines, and Timor-Leste country cases is followed by an examination of policy-related insights and lessons learned. Finally, the paper builds on this East Asian pilot experience, offering ideas on a potential way forward for organizations like the World Bank to deepen and extend their political analysis capabilities. The paper argues that the Agent-Based Stakeholder Model, utilized thoughtfully, offers a powerful addition to the practical political economy toolkit.

This paper—a product of the Poverty Reduction and Economic Management Sector Unit, East Asia and Pacific Region—is part of a larger effort in the department to conduct and disseminate operationally focused political economy research. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at bn2144@columbia.edu and nbarma@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **At the Frontier of Practical Political Economy:**

**Operationalizing an Agent-Based Stakeholder Model in the World Bank’s East Asia and Pacific Region**

**Barbara Nunberg, Naazneen Barma, Mark Abdollahian, Amanda Green, and Deborah Perlman**<sup></sup>

# **Final Manuscript**

**World Bank Policy Research Working Paper Series**

>  Barbara Nunberg, Naazneen Barma, Amanda Green, and Deborah Perlman are affiliated with the World Bank; Mark Abdollahian is affiliated with Claremont Graduate University.  The authors wish to acknowledge the contributions of Marina Rabinowitz and Ha Na Oh as well as funding from the Bank Netherlands Partnership Program. The findings, interpretations, and conclusions in this paper are those of the authors and should not be attributed to the World Bank, its Executive Board of Directors, or any of its member countries.
ii
# **Table of Contents**

|**I**|**Moving Beyond Traditional Approaches to Political Analysis**|**1**|
|---|---|---|
|**II**|**How the Model Works**|**3**|
|**III**|**What the Model Can Do: Country Case Studies**|**7**|
||Upstream Strategy Development: Entry Points for Civil Service Reform in<br>Timor-Leste|**8**|
||Informing Analytical Work: Input to the Mongolia Mining Sector CGAC|**12**|
||Facilitating Project Implementation: Guidance and Risk Mitigation for the<br>Philippines NPSTAR|**15**|
|**IV**|**Key Insights from the ABSM Pilot**|**19**|
|**V**|**Conclusion**|**27**|

iii
# **At the Frontier of Practical Political Economy: Operationalizing an Agent-Based Stakeholder Model in the World Bank’s East Asia and Pacific Region**

Reform programs sometimes falter because they are politically infeasible. Policy change inevitably creates winners and losers, so those with vested interests strike bargains to determine how far and how quickly reform should advance. These are the micro political dynamics of reform; understanding them can mean the difference between a successful intervention that gains political traction and a well-intentioned gambit that falls short of achieving its developmental objectives.

Donors like the World Bank have been searching for ways to take these political factors more fully into account as they design programs to support country reforms.<sup>1</sup> Recent reviews of the Bank’s development effectiveness have all concluded that more attention to the political economy of reform is needed.<sup>2</sup> Where political analysis has been undertaken, it has only rarely fed directly into the development of country assistance strategies or permeated the details of program design and implementation. Some political economy approaches were analytically sound but lacked practical applications. Others offered practical solutions that were not grounded in analytical rigor.

This initiative therefore sought to introduce a rigorous and operationally usable political analysis tool that could be systematically integrated into the World Bank’s country programming cycle. The East Asia and Pacific region (EAP) carried out a multi-country pilot of the Agent-Based Stakeholder Model (ABSM). This innovative analytical approach entails a quantitative simulation of the complex bargaining dynamics surrounding reform. The model anticipates stakeholder coalition formation and gauges the political feasibility of alternative proposed interventions. This paper provides a review of the ABSM pilot experience, exploring what sets this model apart from more traditional approaches, how it works, and how it fits into the Bank’s operational cycle at various stages. An overview of the Mongolia, Philippines, and Timor-Leste country cases is followed by an examination of policyrelated insights and lessons learned.<sup>3</sup> Finally, the paper builds on this East Asian pilot experience, offering ideas on a potential way forward for organizations like the World Bank to deepen and extend their political analysis capabilities.

## **I—Moving Beyond Traditional Approaches to Political Analysis**

Until recently, the World Bank’s political analysis toolbox has relied primarily on relatively bluntedged instruments pitched at understanding the macro political landscape in client countries. Such analytical instruments—which have included, over time, Institutional and Governance Reviews

> 1 See, _inter alia_ , Keefer, Philip (2006) “Can political economy analysis make a difference for development policy?” Unpublished draft; Fritz, Verena, Kai Kaiser, and Brian Levy (2009) _Problem-Driven Governance and Political Economy Analysis: A Good Practice Framework._ Washington DC: World Bank.

> 2 See, for example, World Bank (2005) _Improving the World Bank’s Development Effectiveness: What Does Evaluation Show?_ Operations Evaluation Department (now the Independent Evaluation Group). Washington DC: World Bank; World Bank (2006) _Engaging in Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress._ Independent Evaluation Group. Washington DC: World Bank; World Bank (2008) _Public Sector Reform: What Works and Why? An IEG Evaluation of World Bank Support._ Independent Evaluation Group. Washington DC: World Bank.

> 3Extensive analytical work applying the ABSM was conducted in a fourth country. For reasons of political sensitivity in a volatile environment, a decision was taken not to discuss the case in this paper. Observations emerging from the case on the general utility of the ABSM approach are, however, included implicitly in the discussion.
(IGRs), Poverty and Social Impact Analyses (PSIAs), and Country Governance and Anticorruption (CGAC) Assessments<sup>4</sup> —can provide informative context on the policy landscape and a broad understanding of the political drivers of key policy reforms. The forms of qualitative stakeholder analysis that frequently underpin these approaches have mapped the behavior of interested actors at a single point in time in response to reform initiatives. But these qualitative approaches tend to be subjective and static, and they are inevitably overwhelmed by highly complex reform environments with large numbers of relevant stakeholders. These conventional instruments are easy to use but they do not exploit the rigor of cutting-edge political science methods now available to analyze how coalition dynamics affect the political prospects of specific interventions.

The Agent-Based Stakeholder Model (ABSM) moves beyond these traditional approaches to apply advanced political science techniques to stakeholder analysis. It offers a robust way to inject just-intime realism into the design of Bank operations, helping guide the reform support strategy toward achievable goals or, where appropriate, signaling when to jettison programs that are going nowhere politically. The ABSM was introduced in EAP in an experiment to elevate the sophistication, accuracy, and operational relevance of political analysis at the Bank. The model draws on leadingedge work in microeconomics and rational choice theory to analyze the preferences and behaviors of relevant decision makers on politically driven issues. The ABSM algorithm simulates the complex round-by-round bargaining dynamics among a large number of stakeholders in a given reform process. The model anticipates how these dynamics play out over time and provides an empirical assessment of the likely extent of reform and degree of stakeholder support for this outcome. While not a substitute for comprehensive understanding of a country’s political economy, the ABSM can add considerable value to other types of analysis (Table 1).

**Table 1:  ABSM Adds Value to Other Common Political Analysis Tools**

||**Informal political**<br>**intelligence**|**Political economy**<br>**macro-analysis**<br>**(e.g., CGAC, PSIA)**|**Qualitative**<br>**stakeholder**<br>**analysis**|**Agent-Based Stakeholder**<br>**Model**|
|---|---|---|---|---|
|**What is it?**|Talking informally<br>with clients and<br>local observers<br>about political<br>trends and events|Understanding broad<br>political drivers of key<br>policy reforms|Static stakeholder<br>mapping based on<br>informed opinion|Quantitative simulation of<br>complex political bargaining<br>dynamics that predicts<br>stakeholder behavior in real<br>time|
|**Why do**<br>**we do it?**|Offers a quick take<br>on the political<br>landscape to help<br>Bank country teams<br>in daily client<br>interactions|Contextualizes policy<br>issues in political<br>reality; sheds light on<br>multiple factors behind<br>complex problems|Provides a broad<br>snapshot of where<br>stakeholders stand<br>on a key policy<br>issue|Provides a rigorous<br>understanding of complex<br>stakeholder coalitions and a<br>gauge of the political feasibility<br>of proposed reform<br>interventions|
|**When**<br>**should we**<br>**use it?**|To garner initial<br>views on political<br>events and identify<br>potential reform<br>allies|To inform country and<br>sector strategy<br>upstream and reassess<br>program<br>implementation in light<br>ofpolitical change|To gain an initial<br>sense of coalitions<br>supporting and<br>opposing reform<br>and the degree of<br>consensus|To identify fruitful entry points<br>for reform, build consensus,<br>and manage risks through<br>explicit testing of alternate<br>strategies|

> 4 Levy, Brian, and Nick Manning (2002) _Institutional and Governance Reviews – a new type of economic and sector work._ PREM Note No. 75. Washington DC: World Bank; World Bank (2007) _Tools for Institutional, Political, and Social Analysis of Policy Reform: A Sourcebook for Development Practitioners._ Washington DC: World Bank.

2
Risk Acceptant: Risk Averee: Risk Neutral: Willing toconsensus risk holding up byte anotginal poatton The cost and benefits preferred<sup>outcome</sup> to get atthe onible ex mor not ofchanging pesition reachingadeal per are of equal value NO REFORM SOME REFORM GOOD ENOUGH REFORM FULL REFORM
As it has been operationalized during the course of the EAP pilot, agent-based stakeholder analysis is carried out in five steps:<sup>6</sup>

**1. Framing the question:** The framing step involves articulating the overall objective of the political problem and breaking it down into its component parts for detailed analysis. The organizational logic of the ABSM provides a structured format to focus general discussions of strategy and policy into concrete issues for analysis. As such, the framing step can help discipline conceptualization of the policy puzzle within the broader political economy landscape.

**2. Specifying the issue:** An “issue,” for the purposes of the ABSM, is defined as the continuum of options that stakeholders may support on a specific policy matter. An issue is specified in a linear manner from 0 to 100, either from the least politically difficult reform being considered to the most politically difficult (Figure 2), or in terms of two alternative poles of reform. In either case, intermediate reform steps are identified along the spectrum.

**Figure 2:  Issue Continuum – The Range of Negotiated Policy Outcomes**

|**0**= Minimum Advocated Position<br>Maximum Advocated Position =**100**|
|---|

Issue specification is probably the most challenging aspect of applying the ABSM to the Bank’s policy dialogue and operational programs; this step is where substantive expertise and the “art” of analytical creativity are most crucial. Taking time to specify the issue properly is also one of the model’s benefits, as it helps the Bank achieve clarity on the reform issue, desired outcomes, and key stakeholders.

**3. Collecting data:** Modeling the policy dynamics behind a specific issue proceeds with a rigorous data collection process. Structured interviews are conducted with individuals who possess a rich understanding of the country context, the stakeholder landscape, and/or the policy issue being analyzed. The information gathered during these interviews is coded numerically for entry into the ABSM software. The aim is to obtain the following data for each specified issue:

- **Stakeholders:** A list of all individuals or groups who could potentially influence the outcome of reform. This list comprises government counterparts, other country officials and political parties, World Bank departments, the donor community, and private sector and civil society groups (including academics and the media as well as international and local non-governmental organizations) with an interest in the reform outcome.

- **Positions:** The extent of reform supported by each stakeholder, according to the issue spectrum defined. Each position is assigned a numerical value on the issue continuum (from 0 to 100) and can be pinpointed in relation to all other stakeholders’ positions. Position data elicit not the outcome the expert thinks each stakeholder would eventually accept (reservation price), nor the stakeholder’s true beliefs (revealed preference), but the policy position each stakeholder is currently advocating given the political environment.

- **Influence:** The potential influence, power, or resources that each stakeholder could credibly bring to bear on the issue. Each influence score is defined as a numerical value (from 0 to 100) relative to all other stakeholders’ influence scores.

> 6 Below, each of these steps is revisited in terms of the analytical value of the ABSM process itself.

4
<!-- Start of picture text --> rosnom<br>4 A o 8 | “ 8 8 ns 100<br>al)<br><!-- End of picture text -->

<!-- Start of picture text --> Mo Reform Full Reform<br>TM~ <p BBESEREESEELERSESER =<br>Posinon<br>a 5 wo wf e & # 6 tt8<br>10<br><!-- End of picture text --> <!-- Start of picture text --> o> Se<br>10 a 5 1a 1% 6 = cd] : ty “ 100<br><!-- End of picture text --> <!-- Start of picture text --> i 4> BEETLE =<br>Exec Parliament Justice Political Parties Civil Society Donors Media<br><!-- End of picture text -->
<!-- Start of picture text --> wooa —.<br>f‘ @ \<br>f \<br>STATUS QUO 4 IMPOSE<br>When neither can be persuaded hy the other, When both know she can get her way,<br>they etay where they are she can preseure him to move all the way to her position<br>€ s<br>:<br>BARGAIN LEVERAGE<br>{\ theyWhen boththey meveteae willing to atrike a compromise positiona deal, | YWhensheshe has does unrealized leverage not realize she could to movehimget her way, /}<br>‘ /<br>ae -,<br><!-- End of picture text -->
three-day hands-on course. Members of country or task teams can thus use this portable tool themselves, on demand, helping to embed political analysis in operational decision making so as to have impact on real-time policy choices.

# **III—What the Model Can Do: Country Case Studies**

The East Asia and Pacific region piloted the ABSM approach in four countries, attempting to integrate the model into the analytical and project cycle and thereby generate lessons on the model’s applicability to Bank work.<sup>11</sup> The overall aim of this exercise was to improve the quality of political analysis informing strategic programming choices and to enable the Bank to work with country counterparts to build consensus for important governance reforms. The model was applied, on demand, at various stages of the operational cycle (Table 2).

**Table 2:  Five Pilots at Four Stages of the Bank’s Operational Cycle**

|**Stage **|**Potential uses of ABSM**|**Pilot approaches**|
|---|---|---|
|**Upstream**<br>**brainstorming**<br>| Targeted insights on the scope and contours of political will to<br>assess momentum behind a multi-faceted reform program<br> “Blue sky” exploration of potential new areas of Bank|**Timor-Leste:**Entry<br>points for civil service<br>reform|
|Country<br>Assistance Strategy<br>(CAS)<br>development,<br>program design|engagement<br> Helps identify key supporters and opponents, along with Bank<br>strategies to actively build political commitment by bolstering<br>supportive coalitions and defusing potential opposition<br> Sensitivity testing can be used to examine which strategies are<br>robust across different possible states of political affairs, helping<br>anticipate “political landmines”|**Mongolia:**Refining<br>transparency and<br>accountability<br>approaches|
|**Informing**<br>**analytical work**<br>Country<br>Governance and<br>Anticorruption<br>(CGAC)<br>Assessments| Assessing nature of political commitment to core reform goals<br>analyzed in a public expenditure review or other core diagnostic<br> Helps identify key supporters and opponents of reform<br> Insights into the links between micro reform issues and macro-<br>political context|**Mongolia:**Options in<br>natural resource<br>management value chain|
|**Calibrating**<br>**conditionality**<br>Development<br>policy operation| Defining politically feasible prior actions for policy loans<br> Allocating resources efficiently to most promising reform<br>prospects<br> Helps identify specific opportunities for reform and develop risk<br>mitigation strategies to avoid potential derailment of an operation<br> Helps build consensus for reform|Calibrating<br>conditionality or prior<br>actions for a broad-<br>based policy operation|
|**Smart design and**<br>**supervision**<br>Projects| Identifying opportunities for and threats to achieving project<br>objectives<br> Developing risk mitigation strategies<br> Buildingconsensus for reform and bolsteringengagement|**Philippines:**Risk<br>mitigation for NPSTAR<br>project|

> 11 This work was supported by a Bank Netherlands Partnership Program (BNPP) grant on “Building Capacity for Political Stakeholder Analysis.” Earlier work using a beta version of the model paved the way for this more extensive pilot.

7
In Timor-Leste, the ABSM explored the Bank’s potential role in supporting new momentum behind a civil service reform program ahead of the next Country Assistance Strategy (CAS). The Mongolia country team applied the model twice, first to evaluate options for strengthening transparency and accountability in the public sector and then, in the context of the Country Governance and Anticorruption (CGAC) Assessment, to inform analytical work to support the transparent and effective use of mining revenues. In the Philippines, the Bank employed the ABSM to guide implementation of a stalled tax administration reform project, seeking to avoid the pitfalls encountered in previous attempts to improve the tax system. Extensive analytical work applying the ABSM was conducted in a fourth country to help assess political commitment to a broad-based development policy operation and to help define realistic reform targets in a difficult environment.

The ABSM pilot experience is reviewed briefly below, highlighting one of the modeled issues in three countries to explore why the context was ripe for systematic political analysis, what we learned, how we changed our approach, and—where sufficient time has passed—what has happened since.<sup>12</sup>

# **Upstream Strategy Development: Entry Points for Civil Service Reform in Timor-Leste**

Administrative and civil service reform recently resurfaced as a pivotal issue in Timor-Leste. The fragile coalition government that came to power after the 2006 political crisis declared 2008 to be the “Year of Administrative Reform,” pledging to restore public confidence in government by strengthening the institutional framework for accountability and service delivery. The Prime Minister personally took the lead in driving several key initiatives, chief among them the proposed establishment of a Civil Service Commission with wide-ranging powers.

To date, the Bank has not been a lead player on administrative and civil service reform in TimorLeste but it has been at the forefront of other public sector reform initiatives. In 2006, in concert with other development partners, the Bank launched the Planning and Financial Management Capacity Building Program (PFMCBP) to help tackle serious deficiencies in public spending and revenue management. After a slow start, the program was reshaped in 2008 to help the new Ministry of Finance manage crucial public financial management priorities while planning for longer-term professional development needs. The Ministry wanted to reorganize public financial management processes and develop the supporting system of incentives, rewards, and sanctions to build a stronger public service ethos. The Bank agreed to support the Ministry’s objectives and, in addition, made a commitment during the 2008 Timor-Leste and Development Partners Meeting to support the country’s administrative reform process more broadly.

The ABSM was enlisted to help better understand the political dynamics operating beneath the surface of this complex agenda and to identify the most promising entry points for potential intervention. The analytical exercise focused on three dimensions of civil service reform: (1) the potential creation of a Civil Service Commission and its appropriate scope of powers; (2) the demand for a revised pay and grading system and the underlying debate between job- and

> 12 Exhaustive country case studies are available upon request. These provide extensive detail on each of the three to five issues modeled in each country, including: (1) a full dataset listing all stakeholders included in the analysis along with their position, influence, and salience data; (2) the ABSM modeling outcomes; and (3) a list of all experts interviewed during the course of the analysis. Each country pilot was also evaluated systematically through the application of a standardized assessment framework comparing _ex ante_ expectations with the ABSM’s impact over time; a synopsis of this evaluation is provided below.

8
qualification-based systems; and (3) the more general discussion about what would constitute a robust program of civil service reform and the level of political appetite for its implementation. Following is a brief review of the ABSM analysis on the third issue, the broader civil service reform agenda.

**Framing the question.** Although the Government of Timor-Leste had articulated its commitment to improving public administration and service delivery, comprehensive civil service reform can be difficult to achieve anywhere. Well-intentioned reforms can easily be derailed by weak capacity, overlapping and ill-defined policies, feeble coordination across ministries and agencies, a fragile political coalition, and stakeholders’ worries about what might happen to their own pay levels, positions, and recruitment and promotion opportunities. The ABSM analysis focused on the potential scope of support for wide-ranging reform and attempted to identify the major coalitions that might form on this issue. In Timor-Leste’s social and political order—characterized by political schisms and corresponding patronage networks borne of a violent struggle for independence— merit-oriented civil service reforms were highly contentious. Given the country’s need to elevate government performance to achieve tangible results on service delivery as a means to political stability, the stakes were particularly high.

**Specifying the issue.** The process of arraying the varied elements of a comprehensive civil service reform package along a linear continuum proved complicated. The team divided the overall agenda into three levels of cumulative reform, building from 0 to 100 (Figure 5).

**Figure 5:  Policy Steps in Timor-Leste’s Civil Service Reform Agenda**

<!-- Start of picture text --> 100 – Independent oversight of merit-based civil service regime<br>95 – Merit-based civil service regime implemented<br>80 – Robust establishment management system implemented & resourced<br>75 – Govt-wide pay & grading system implemented & adequately resourced<br>55 – Legislative basis for civil service institutional responsibilities<br>45 – Senior executive cadre defined, recruited, & adequately resourced<br>40 – Civil service management framework approved<br>30 – Technical analysis, including job evaluation & comparative pay<br>10 – Pay & grading pilots initiated<br>5 – Provisional pay & grading package approved in decree law<br>0 – Status quo<br><!-- End of picture text --> The first level focused on the low-hanging fruit: temporary pay relief, pay and grading pilots, and technical analysis. The second level covered the design of a new system: approving a new framework for civil service management, creating a senior executive cadre, and defining the legal basis for institutional mandates regarding the civil service. The last—and most politically difficult—level 9
<!-- Start of picture text --> POSITION<br>0 5 10 30 40 et 50 15 80 95 100<br>Status: quo Pay and Technical Senior- Govt-wide> pay Independent1<br>grading pilots analysis executive and grading Robust oversight of<br>initiated conducted cadre defined: system establishment merit-based<br>| Provisional _. . recruited: and implemented management civil service<br>paysao and Civilmanagement service resourced andresourced systemimplemented regime<br>| grading framework Merit-based<br>package approved basisserviceLecitategislative for civil civilregimeimplementedservice<br>L institutional<br>responsibilitie<br>| s established<br>0 a<br>rnd > ENE ca<br><!-- End of picture text -->
<!-- Start of picture text --> POSITION<br>0 —_>———5 ss10 30 40 Ey 50 Soo Se ot15 FeovmwWYll80OOOoS95 SH100<br>Status quo Pay and Technical _ ; Senior Govt-wide pay Independent<br>grading pilots analysis Civil servios executive and grading a nment oversight of<br>initiated conducted managemer cadre defined: system merit-based<br>| payProvisional and frameworkapproved recruited:resourced and implementedand resourced systemimplementedmanagement civilregime3 service<br>| grading Merit-based<br>package Legislative civil service<br>- basis for civil regime<br>service implemented<br>institutional<br>[ responsibilitie<br>s established<br>Rnd@> EEE oa}<br><!-- End of picture text -->
# **Informing Analytical Work: Input to the Mongolia Mining Sector CGAC**

As Mongolia’s new democracy and market economy mature, Mongolian society is increasingly concerned about corruption and weak governance. As a significant source of revenue, the mining sector is particularly susceptible to rent seeking and corruption; in a recent survey, Mongolian citizens ranked mining as the second-most corrupt sector, after land utilization.<sup>14</sup> The government has made important strides toward improving transparency and accountability in the sector, including its ongoing effort to implement the Extractive Industries Transparency Initiative (EITI). Still, problems persist in ensuring transparency and efficiency in the production and management of mineral resources. In particular, the current system for granting mining licenses is _ad hoc_ , discretionary, and under-regulated, and manual processing opens the door to manipulation and corruption in the issuance of mining permits.

Such concerns were part of the focus of a Country Governance and Anticorruption (CGAC) Assessment conducted in 2007 as part of a Bank-wide initiative for which Mongolia was a pilot country. The Mongolia CGAC undertook broad governance analysis to inform the Country Partnership Strategy. A problem-driven political economy framework was used to identify a series of potential reform steps across various elements of the natural resource value chain, including mining licensing, public financial management, public investment planning and procurement, and social transfer policies. These findings provided a springboard to help mainstream the Bank-wide governance and anti-corruption (GAC) agenda and methodology in the sectors, and the CGAC team worked closely with country management, technical and sector colleagues, and Mongolian stakeholders to identify the key measures needed to achieve “good enough” governance in the mining sector. As part of this initiative, the country team employed the ABSM to help assess the prospects for improving sector governance to make effective use of growing mining revenues. With national parliamentary elections slated for the following month and mining sector issues at the top of the political agenda, this pilot of the ABSM was conducted at a pivotal time. Among other issues related to transparency and accountability, the ABSM analysis focused on reform of the mining licensing and cadastre systems. Following is a brief review of the analysis on this issue.

**Framing the question.** The issue of mining transparency in Mongolia encompassed procedures for awarding mining exploration licenses, including the degree to which these procedures are transparent, as well as the maintenance and public availability of cadastre records. License application and allocation procedures are highly complex and opaque, making it difficult for Mongolian citizens to learn about, evaluate, and challenge ongoing and proposed mining activities. There are some 29 steps in the application process and the cadastre system is not computerized. The resulting delays have tarnished the reputation of the cadastre office, as has the perceived role of patronage and favoritism in determining license awards. In response, the cadastre office has begun developing procedures and manuals to reduce the informality and discretion of license granting procedures and to streamline the number of necessary steps. The parliament holds authority over broader mining policy decisions, raising questions as to whether politicians and government officials will initiate changes that limit their power in the sector. The ABSM was applied to help gauge the level of political commitment for reform in this area, including improved regulation, information dissemination, and computerization of cadastre records.

> 14 Asia Foundation and United States Agency for International Development (2008) _Mongolia Corruption Benchmarking Survey 2006-2008._

12
**Specifying the issue.** The policy continuum for the mining transparency issue included a series of cumulative steps (Figure 8), building from the current _ad hoc_ , discretionary, under-regulated, and politicized process for granting mining licenses (0) to the implementation of a clear legal framework and automated cadastre system, complete with internet-based query capabilities (100). In between, preliminary measures included decreasing the number of steps in the application process, supporting a cadastre website, and clarifying the status of land. The next stage of reform involved enacting increasingly stringent exploration requirements for license holders, rolling out the automated cadastre system, and holding public consultations on mining costs and benefits. The final stage of reform included establishing an interagency commission to oversee and ratify cadastre recommendations, fully deploying the automated cadastre system, and applying a clear regulatory framework for granting and renewing exploration licenses.

**Figure 8:  Policy Steps to Improve Mining Transparency in Mongolia**

<!-- Start of picture text --> 100 – Regulatory framework applied in granting & renewing licenses;<br>automated mining cadastre system fully developed, with web query enabled<br>87 – Clear regulatory framework applied, with publicly disclosed regulations<br>above the ministerial level, preferably through cabinet or presidential decree<br>70 – Interagency commission established to oversee & ratify cadastre<br>recommendations; automated system deployed; quarterly reports published<br>65 – Public consultation supported on mining costs and benefits as part of<br>license granting process<br>50 – Mining leases granted only when economic feasibility is established;<br>automated cadastre system deployed but not fully operational<br>40 – Provisions in place for exploration continuation until economic<br>viability is established<br>30 – Minimum expenditure required on exploration licenses to minimize<br>speculation; automated system partially deployed, with only geodetic data<br>20 – Status of land clarified via Ministry of Nature & Environment for<br>reserves, protected areas, and special use lands<br>15 – Computerized cadastre and license records website supported<br>10 – Streamlining begun by decreasing the number of process steps through<br>detailed operational procedures<br>0 – Ad hoc , discretionary, under-regulated, and politicized process for<br>granting mining licenses<br><!-- End of picture text -->

**Collecting data.** The team discussed mining issues with about a dozen experts in government, parliament, civil society, mining companies, and donor organizations, including within the Bank. Although these experts were broadly willing to talk openly about mining issues, many of them had only superficial information on the specifics of mining license and cadastre system reforms; access to those on the inside of the reform debate was constrained due to the pre-election timing.

The data showed that, at the time of the pilot exercise, stakeholder positions varied widely across the spectrum of possible reforms (Figure 9). Many executive stakeholders and political parties supported mid-level reforms such as granting mining leases only after economic feasibility is established, deploying an automated cadastre system even if not fully operational, and supporting public 13
<!-- Start of picture text --> POSITION B<br>0 10 15 20 30 40 50 65 70 87 100<br>Add hoc- Cadastre and Minimum Exploration until Mining leases Public Interagency Clear regulatory Regulatory<br>\_ discretionary: license records spending economic viability granted only consultations commission framework framework<br>politicized website required on established when economic supported established applied: with applied;<br>|_ process supported exploration feasibility is publicly disclosed automated<br>licenses established regulations cadastre system<br>Streamlining clarifiedStatus offor land developeday with<br>begun through reserves:<br>fewer protected areas-<br>procedures etc.<br>a < a > >><br>5 | 3x<br>Rid <> HERES<br><!-- End of picture text -->
<!-- Start of picture text --> POSITION a<br>0 10 15 20 30 40 50 65 70 87 100<br>Add hoc- Cadastre and Minimum Exploration until Public Interagency Clear regulatory Regulatory<br>discretionary- license records spending economic viability consultations commission framework framework<br>politicized website required on established supported established applied- with applied;<br>|_ process supported explorationlicensesi Mining teases publiclyregulationsi disclosed automatedcadastredevel systemvith |<br>Streamliningi clarifiedStatus of landfor whengrantedeconomic only web query<br>L begun through reserves: feasibility is 4<br>fewer protected areas- established<br>procedures etc.<br>a < a > >><br>— 3x<br><!-- End of picture text -->
in 1997. The 1986 reforms aimed to simplify collection systems and succeeded in boosting revenues. The later initiative attempted to bridge the revenue gap expected in the wake of economic liberalization; but it came to fruition during the 1997 Asian financial crisis and weak political commitment led to a failure to produce results. The Bank’s own support to the sector has had an equally mixed record. The Tax Computerization Project of the mid-1990s achieved some important technical gains but reform impact foundered in the face of administrative obstacles and political opposition to implementation.

The government’s more recent, small-scale reform programs have fared better. The passage of a value added tax reform law in 2005 sparked a 20% increase in tax collections. Other wins have been achieved through compliance verification drives, third-party information gathering, expanded tax audits, and lifestyle checks on public employees. Large gains in tax collection have remained elusive, however. The ratio of tax revenue to GDP has begun to rise, but remains low relative to other East Asian countries. Out of a workforce of about 33 million, only 6 million taxpayers are registered. An aging information system has created a logjam, leaving tens of thousands of tax returns unprocessed. This backlog, together with extensive tax exemptions for well-connected individuals, does little to encourage workers to register. Despite the government’s formal commitment to address these problems, systematic reforms that would make a real and sustainable impact on tax revenues consistently lack traction. This lack of movement on reforms points to deeper political obstacles.

Building on continuing dialogue between the Bank, the International Monetary Fund, and the government, the Bank introduced a new tax administration reform operation in mid-2007. The multi-donor National Program Support for Tax Administration Reform (NPSTAR) seeks to achieve quick wins in its early phases, creating demonstration effects to build momentum for broader reforms down the line. The program’s four components include tax compliance, enforcement and control, human resource development and management, and Bureau of Internal Revenue (BIR) governance and change management. In an effort to avoid the pitfalls of earlier reform efforts, the Bank applied ABSM to help understand the potential political obstacles to successful implementation of the NPSTAR and to formulate a risk mitigation strategy. The Philippines pilot analyzed four separate issues on three broad fronts: (1) the macro-political landscape for general tax reform; (2) the micro-issues surrounding BIR internal organization, including full enforcement of the Lateral Attrition Law—which was designed to provide incentives for BIR and customs staff to exceed revenue targets—and implementation of performance management systems; and (3) the micro-issue of increasing BIR tax collection capacity through taxpayer registration. Following is a brief review of the model’s analysis of the taxpayer registration issue.

**Framing the question.** The absence of accountable, fair, and clean taxpayer registration processes reinforces the population’s negative attitudes toward the tax system, as does the backlog of returns. Expanding taxpayer registration appeared to be the most politically feasible tax reform issue on the table in the Philippines, but two key areas presented problems: computerization and access to the tax information base. At the time of the analysis, only 48 of the 112 district offices were computerized, making it difficult to keep track of taxpayers and handle delinquencies.<sup>15</sup> The option of outsourcing registration in the short term had been raised, but there were concerns regarding confidentiality of taxpayer records. Registration improvements could also be complicated by

> 15 Rodolfo Vicerra, Director General, Congressional Planning and Budget Department, Philippines House of Representatives (2005) “The Looming Fiscal Crisis Shouldn’t Be that Taxing.” Manila, Philippines.

16
<!-- Start of picture text --> POSITION<br>0 25 45 60 70 80 90 100<br>Doa nothing Content Priorities set Procedures ssObjectives: aFull aBIR Full —<br>requirements for registration agreed to approach: and registration registration registration of<br>and by clean and methdology for rolled out contracted out taxpayers<br><!-- End of picture text --> <!-- Start of picture text --> procedures {e.g. large & database and = agreed without an outside<br>reviewed medium remove stop contracting service to<br>taxpayers filers. out quickly<br>registered complete the<br><!-- End of picture text --> <!-- Start of picture text --> first: with process of<br>smaller fully<br>taxpayers registering all<br>registered taxpayers<br>once system<br>has been<br>cleaned)<br><!-- End of picture text --> <!-- Start of picture text --> 0<br><!-- End of picture text --> <!-- Start of picture text --> rnd<> co<br><!-- End of picture text -->
<!-- Start of picture text --> Dor nothing ‘Content Priorities set Procedures— Objectives-_— Full _— BIR _— Full —<br>requirements for registration agreed to approach: and registration registration registration of<br>and by clean and methdology for rolled out contracted out taxpayers<br>registration classification validate tax mapping slowly: contracted to<br>procedures (e.g. large & database and agreed without an outside<br>reviewed medium remove stop contracting service to<br>taxpayers filers out quickly<br>registered complete the<br>first: with process of<br>smaller fully<br>taxpayers registering all<br>registered taxpayers<br>once system<br>has been<br>cleaned)<br><!-- End of picture text --> <!-- Start of picture text --> 0<br>rndd> ES o<br><!-- End of picture text -->
<!-- Start of picture text --> POSITION<br>0 25 45 60 70 80 90 100<br>r — — — —<br><!-- End of picture text --> <!-- Start of picture text --> Do nothing Content Priorities set Procedures Objectives: Full Full<br>requirements for registration agreed to approach: and registration registration of<br>and by clean and methodology rolled out taxpayers<br>registration classification validate for tax. slowly: contracted to<br><!-- End of picture text --> <!-- Start of picture text --> procedures (e.g. large & database and mapping without an outside<br>reviewed medium remove stop agreed contracting service to<br>taxpayers filers out quickly<br>registered complete the<br><!-- End of picture text --> <!-- Start of picture text --> first- with process of<br>smaller fully<br>taxpayers. registering all<br>registered taxpayers<br><!-- End of picture text --> <!-- Start of picture text --> has been registration<br>cleaned) contracted out<br><!-- End of picture text --> <!-- Start of picture text --> 0 =o) = 4, a, | |<br>id@> BPEL ej<br><!-- End of picture text -->
# **The Evaluation Framework**

As part of the pilot exercise, a standardized assessment tool was developed to evaluate the process of applying and adapting the ABSM to country governance reforms as well as the model’s impact on country team decision-making. In focusing on the process, the evaluation framework aimed to get a sense of how the ABSM affected the way the country team approached a particular governance problem—what their expectations were at the outset, how the experience of systematic data collection and analysis shaped their views, and what effect, if any, the results had on their approach to the issue on the ground. The evaluation tool also focused on results, seeking to determine whether the ABSM improved the quality of political analysis underpinning Bank programming decisions. To this end, the framework’s questions (excerpted in Table 3) sought to examine whether the model’s recommendations were adopted and whether reform outcomes were as expected.

**Table 3:  ABSM Evaluation Framework**

|**Ex ante expectations**|**ABSM analysis and**<br>**recommendations**|**Immediate outcomes**|**Impact over time**|
|---|---|---|---|
|What is the Bank’s desired<br>level of reform?|What new insights and/or<br>relationships did ABSM<br>reveal?|Did the Bank redefine its<br>desired level of reform?|Did the altered strategy<br>affect the reform outcome?|
|Who are the key<br>stakeholders and what are<br>their relationships?|Did ABSM change your<br>views about the behavior<br>of key stakeholders or the<br>nature/difficulty of the<br>reform issue?|How did ABSM alter<br>relations with client or<br>other stakeholders?|Did the altered strategy<br>achieve its objectives?|
|What should the Bank’s<br>strategy be? What are the<br>advantages and<br>disadvantages of this<br>strategy?|Did the strategy change?|Did you try any strategies<br>that emerged from ABSM<br>analysis? What were your<br>expectations for this<br>strategy?|What would have<br>happened if the Bank had<br>not changed its approach?|
|How do you intend to use<br>ABSM?|For which issues and<br>operational stages is ABSM<br>most useful?|If you did not change the<br>strategy, why not?|What is the marginal<br>benefit of ABSM over<br>time?|

The ABSM evaluation was conducted as a set of _ex ante_ and _ex post_ interviews of each analytical team involved in the pilot. These country team evaluations revealed useful insights drawn from each stage of the model’s application—from thinking through the key elements of each reform issue and the relationships among stakeholders to testing some of the model’s more promising strategic recommendations. Table 4 provides a brief summary of key evaluation insights on four of the five applications of ABSM in the East Asia and Pacific Region.

In many of the multiple issues examined in each case, high initial expectations of reform progress were shown to be unrealistic in the face of strong opposition, often from expected quarters. In this respect, neither a broad civil service reform agenda in Timor-Leste nor the more targeted sequence of mining license reforms proposed in Mongolia appeared to have the necessary political traction on the ground for implementation.

In other cases, the ABSM yielded granular insights on sources of support for reform where the Bank’s initial expectations were originally more pessimistic. In the case of taxpayer reforms in the 20
Philippines, for example, the country team’s expectations were low at the outset due to the failure of previous reform efforts to gain traction. The Bank had tried to influence major opposition groups, such as large taxpayers and anti-reform elements in Congress, through dialogue, but to no avail. Through ABSM analysis, the team was surprised to find that a small group of previously overlooked stakeholders—the tax examiners working in the Bureau of Internal Revenue—would likely block reforms otherwise supported by the majority of stakeholders. This insight prompted the team to shift tactics, introducing into the NPSTAR program a more aggressive change management strategy aimed at helping tax examiners and other lower-level staff understand the potential benefits to be gained from improved taxpayer registration and broader tax administration reform.

Other cases demonstrated that shifts in the Bank’s own positioning on reforms might yield betterthan-expected results. In two cases, reflecting variance in internal views and expressing diverse Bank positions on reform led to a tightening of stakeholder consensus at a higher level of reform than initially anticipated. In one case, working toward a more moderate goal with greater inclusiveness yielded better-than-expected results, since key stakeholders for whom the reforms had high salience could drive progress. Maintaining a focus on tangible micro-reforms held the key to realizing limited but meaningful objectives in an otherwise stalled macro-reform environment.

**Table 4:  Key Insights from ABSM Evaluations**

|**Pilot**<br>**application**|**Ex ante**<br>**expectations**|**ABSM analysis and**<br>**recommendations**|**Immediate**<br>**outcomes**|**Impact over time**|
|---|---|---|---|---|
|**Mongolia**<br>Budget<br>transparency| Pre-election timing<br>considered a window<br>of opportunity for<br>transparency reforms| Powerful officials<br>strongly opposed<br>greater public access<br>to budgets<br> Opportunities for<br>Bank to work with<br>civil society to push<br>for reform| Bank made an effort<br>to engage in<br>meaningful<br>consultations;<br>increased emphasis on<br>demand-side in<br>country program| Insights supported<br>by CGAC political<br>economy analysis<br> Achieved reform<br>level Bank team was<br>aiming for: better than<br>basecase, but short of<br>Bank’s stated position<br>(full reform)|
|**Mongolia**<br>Mining<br>transparency| Pre-election timing<br>considered a window<br>of opportunity for<br>transparencyreforms| No concrete<br>agreement expected,<br>despite support of key<br>counterparts| “Wait and see”<br>approach| Team may reapply<br>model in post-election<br>environment|
|**Philippines**<br>Tax<br>administration| Difficult project<br>implementation due<br>to disagreement over<br>design and failure of<br>past reform efforts| One overlooked<br>group of stakeholders<br>(tax examiners) could<br>block implementation<br> Revise project<br>approach| Bank introduced<br>change management<br>into project design;<br>slightly adjusted<br>timeline| Project still in early<br>implementation phase<br>due to other<br>unforeseen political<br>obstacles|
|**Timor-Leste**<br>Civil service<br>reform| Window of<br>opportunity on civil<br>service reform seen to<br>open; Bank still<br>forming its strategy| Some consensus on<br>setting up formal<br>architecture of civil<br>service management<br> Civil servants could<br>splinter consensus if<br>mobilized| Little role seen for<br>the Bank at the time<br> Conduct more<br>analytical work while<br>awaiting Parliamentary<br>action| Civil Service<br>Commission created<br>with expansive scope<br> Little movement on<br>broader civil service<br>agenda|

21
A more inclusive strategy resulted from applying the ABSM to budget disclosure reforms in Mongolia during the first ABSM pilot in that country. The team’s initial approach was to work directly with key government counterparts to encourage greater transparency in the budget process. Reform expectations were high, as the dialogue was taking place during an election year and it was thought that this would encourage key stakeholders to support transparency and participation. In conducting stakeholder interviews, the team was surprised to find that stakeholder opinions varied more widely than initially thought. Moreover, ABSM analysis indicated that the Bank had relatively little influence over the Ministry of Finance on this issue. Instead, non-governmental organizations like the Open Society Institute were shown to have a great deal more leverage over government counterparts than the team had previously considered. This insight led the Bank team to take on a consensus-building role by bringing key stakeholders together around the negotiating table rather than working through direct dialogue with traditional government counterparts. Within a year or two, the Mongolian government began publishing the executive budget upon submission to the National Assembly. This outcome represented a significant improvement over the basecase result anticipated by the ABSM in the absence of further interventions.

In this manner, on some issues, insights gleaned through the process and results of ABSM application led to adjustments in the Bank’s strategy. This was not the case for all piloted issues, however. In some cases, country teams adopted a “wait and see” approach, having determined that the political situation was too fluid, the Bank’s understanding of the reform issue too limited, or the Bank’s overall strategic goals too underdeveloped to proceed on the basis of ABSM results. In the second Mongolia pilot, for example, data collection efforts were constrained by the pre-election timing of the analysis such that the team did not feel it had achieved a sufficient grasp on the stakeholder landscape to move forward on sensitive mining sector reforms. In Timor-Leste, applying the ABSM to macro civil service reform issues proved to be complicated by a limited understanding of the appropriate sequencing of reforms in that context. The Bank elected to wait until the Parliament had taken some action in the direction of civil service reform before making further decisions, and to conduct additional analytical work in the meantime. While this outcome may not have differed from what would have occurred without ABSM analysis, the decision to delay was taken more deliberately following application of the model.

Overall, the ABSM evaluations show that country teams found value in the process of applying the model to critical reform issues. Although ABSM results were not considered to reveal definitive answers or to substitute for nuanced judgment, pilot teams generally found the model to be a useful tool in understanding the political economy of reforms and testing new approaches to complex reform issues. The analytical methodology crystallized the trade-offs among various dimensions of reform and the possible benefits to be gained from sacrificing some movement in the degree of reform in favor of reaching greater consensus, or in slowing down the speed of reform in order to achieve better results. Teams that tested ABSM-recommended strategies generally found them to make intuitive sense and yield useful results.

# **The Value of the ABSM Logic**

In addition, the methodology underlying the application of ABSM and its structured format forced teams to think critically about the nature and sequencing of reform issues and comprehensively about the stakeholder landscape. Hence the process of applying the ABSM can itself be instructive. Framing, issue specification, and scenario analysis are beneficial even just in terms of the conversations they foster among the team as they think through problems and options more 22
systematically. The process of collecting data is also illuminating, as it encourages field staff to engage in structured conversations with their counterparts on specific issues and to rethink their assumptions about how politics affect the Bank’s work. Implementing the ABSM gives shape and discipline to political analysis in the Bank, which can be helpful in an environment of heavy workloads and competing priorities. Although political analysis has come to be regarded as an important input to strategy development and program design, the absence of a standardized, mainstream mechanism for applying such analysis in an operationally useful way has blocked its full integration into the decision-making process. The ABSM can help make the application of political analysis more robust and the process of collecting data on stakeholder preferences can help get the ball rolling on better understanding of the political dynamics of reform.

**Framing the question.** The first stage of ABSM analysis can help Bank teams articulate the overall aim of their interventions and how they fit into the broader political context. The organizational logic of the model’s approach and its visual representation in the ABSM software can help guide strategy discussions and define the parameters of program design and implementation. Often, the process of framing and specifying potential issues for analysis can bring knowledge gaps to light.

**Specifying the issue.** A number of substantive challenges are inherent in issue specification, often pushing Bank staff to think much more carefully about the steps along the critical path in a given reform program than would normally be required in preparing a support package or even a piece of analytic work. The process of issue specification can thus facilitate a productive discussion about the sequencing and comparative difficulty of a set of reforms on a given issue. For example, for an issue continuum to be linear, all stakeholders must order the various steps along the continuum in the same manner. For issues that are continuous and directional by nature, this is not a problem. It is much more challenging to lay out a linear continuum for issues that are made up of discrete reform “steps”—all stakeholders may not view the various reform steps in the same order of political difficulty or even see them as moving along a single reform track. For complex, multi-pronged reform programs, the analytical contortions required to specify a linear continuum may not be worth the analytical payoff. Decisions about the level of disaggregation in specifying the continuum must be viewed as a balance between modeling what is necessary to understand the reform process for better-informed decision-making versus recreating a high-fidelity model of the actual problem.

**Collecting data.** The quality of the data determines the quality of the analysis: if garbage goes in, garbage comes out. The process of data collection through expert interviews has neither the luxury of statistical safeguards built into large-sample data analysis, nor the exhaustive nature of good qualitative research methods. Analysts using the ABSM must take care to ensure that data collected through expert interviews are accurate and representative of reality. Choices about a data collection strategy will vary across countries depending on the political economy context, and on the specific issues to be analyzed. There are two major dimensions guiding this choice (Figure 15): (1) _the difficulty of the reform environment_ , including the Bank’s relationship with the government, overall government effectiveness in achieving reforms, and the level of divisiveness in society as a whole; and (2) _access to good and relevant data_ , which depends on factors such as the degree of openness in society, the Bank’s access to non-governmental experts, and the number of experts who understand the country and the specific reforms being analyzed.<sup>16</sup>

> 16 In low capacity environments, data quality may also depend on the Bank’s ability to conduct interviews in a way that encourages local experts to apply their considerable local knowledge to the technical issues being explored.

23
<!-- Start of picture text --> Data Access<br>High<br>Se<br>Easy Difficult<br>Reform Difficulty<br>Low<br><!-- End of picture text -->
convergence on a specific reform position? If so, at what level? Who are the outliers, and are they strong enough to stall reforms? Where are the Bank’s allies? Next, the team drills down into the details of the model’s bargaining dynamics to better understand potential opportunities and threats. Which stakeholders are moving to what positions, and in response to whom? Can the Bank convince other stakeholders to support more reform? What leverage do other stakeholders have to push for a lower level of reform than the Bank is advocating? The model’s scenario analysis capabilities then allow analysts to explore scenarios in which the above opportunities and threats are exercised and thereby anticipate how outcomes might change. The Bank can also test its own role in the reform process, adjusting its position or level of commitment (salience) to determine whether these changes could help nudge the reform agenda forward or bring additional stakeholders on board. In essence, the ABSM can lay out strategic choices in a way that helps country and project teams think through the political opportunities and consequences inherent in their programming decisions.

# **The ABSM’s Operational Value for the Bank**

As part of the pilot exercise, the ABSM has been used to think through various political and reform puzzles facing the Bank and its country clients and to provide real-time analytical inputs to programming decisions, often under intense time pressure. Through this pilot experience, the team has refined the process of applying the model’s analytical approach to Bank work and illuminated possible ways to integrate the ABSM into the Bank’s regular programming decisions. Here, we highlight the ABSM’s major analytical benefits.

**Modeling a complex, multi-stakeholder reality.** The capacity to model complex problems allows the ABSM to capture intricate interactions among a large number of stakeholders. Traditional, qualitative stakeholder tools cannot achieve this, rendering their explanatory power largely intuitive. Assuming good data are fed into the model, the ABSM is likely to provide a more robust and comprehensive picture of the political forces that affect reform. Moreover, the flexibility to change the menu of stakeholders, alter their positions, and test varying scenarios allows the tool to more closely model policy-making conditions in the real world. This dynamic quality, accompanied by accessible and intuitive graphical simulations in the accompanying software, makes the ABSM particularly useful in sifting through intricate political realities to help zero in on more feasible reform strategies.

**Illuminating macro-micro political relationships.** The ABSM is a micro-political tool that captures finely grained details a more macro-political analysis cannot. There are both advantages and drawbacks to this. The ABSM can elucidate the micro-political relationships among stakeholders that can be so central to reform processes and outcomes, although it is less able to embed these bargaining dynamics in the overall macro-political context. Still, where macro-level reforms have stalled, drilling down to examine stakeholder positions and the influence they exert on each other on a specific sub-issue can reveal new insights about the opportunities and threats facing reforms, which are hard to see from a more aerial macro-political perspective. In the Philippines and TimorLeste, this capability helped the country team clarify the Bank’s comparative advantage in supporting micro-reforms within a macro-political agenda that appeared stalled. Over time, collecting data on where stakeholders stand on these micro issues can help track patterns in the broader landscape of political relationships.

25
**Testing and challenging conventional wisdom.** The ABSM tests prevailing assumptions about a political or reform context. By formalizing the data collection process through interviews with experts, the ABSM can assess the degree of consensus on the generally accepted view of reform. In a political environment where the conventional wisdom has been quite static over time, i.e., where no new political dynamics have been revealed, the model can dig down into individual perceptions of issue-specific reform possibilities to reveal new insights that are not readily apparent through casual—or even close—observation. In gathering information from Bank and donor staff and core government counterparts, the analyst can gauge whether each individual’s on-the-ground experiences reinforce or challenge the conventional wisdom about relationships among stakeholders; their relative positions, influence, and salience; and reform processes and outcomes. Without such a check, there is a danger that donors are reinforcing conventional wisdom in a circular process amongst themselves without testing whether it is representative of reality. In the Timor-Leste pilot, for example, the analysis of the proposed Civil Service Commission’s mandate pointed out that a few key stakeholders who were originally thought to be relatively powerless instead held quite a bit of potential sway over the outcome of reform.

**Breaking into the black box of “political will.”** Another advantage of the ABSM is that it disaggregates down to individual stakeholders the concept of political will and thereby breaks into this ubiquitous “black box” of donor speak. Using the ABSM’s data and analytical tools, the Bank can potentially understand much more deeply the relationships among stakeholders that can either fuel or dampen reform progress, thus unveiling the contours and substance of political will. This knowledge can help to identify politically motivated factors that could improve impact or pave the way for greater reform consensus down the road. In the Philippines, after years of stalled initiatives, the NPSTAR team was surprised to learn how much desire there was for tax reform; at the same time, a wide array of pro-reform groups was unable to overcome the entrenched opposition of a few powerful stakeholders. In Mongolia, ABSM interviews and analysis revealed previously unseen splits within the major political parties and unexpected coalitions of stakeholders.

**Helping to prioritize and sequence reforms.** A critical lesson emerging in the Bank’s effort to improve the relevance and effectiveness of its development assistance is the need for greater selectivity and flexibility in its programs. The ABSM can help tease out the reform initiatives most deserving of donor attention and resources by illuminating the degree of political support and consensus in a given reform environment. While the Philippines pilot anticipated little tangible progress on proposed macro-level reforms, the analysis helped identify micro-level issues where there was enough traction to move forward in the meantime. The pilot experience also helped draw attention to important “no go” areas, where the Bank might inadvertently squander political capital by pushing for something prematurely. In Mongolia, the Bank learned that caution on mining transparency issues would be wise in the highly politicized pre-election environment. In the Philippines, pressing for implementation of a performance management system was shown to harbor the potential for a hostile response among staff who feared its effects on their jobs and wages. In Timor-Leste, taking a position on the type of civil service regime was considered counterproductive, as the Bank would not be able to help move the debate forward amid entrenched viewpoints on either side of the issue.

# **Analytical Limitations**

Although the ABSM can provide, for the reasons outlined above, a powerful lens through which to look at reform possibilities in client countries, it is not a silver bullet. As in any other modeling 26
exercise, there is no way to create a perfect model of reality. There are concerns, for example, that the ABSM’s reliance on formal policy positions can distort results by neglecting informal factors such as clientelist ties and political tradeoffs across unrelated issues. This potential data bias can be addressed by thinking consciously about informal undercurrents when collecting and cross-checking data and in interpreting the model’s results. The software’s sensitivity analysis capabilities allow analysis teams to run tests on particularly hazy data. Another worry is that the model’s grounding in the theory of the median voter will beget the same “race to the center” witnessed in majority-based political contests, thereby encouraging only partial reforms. Yet the pilot exercise demonstrated that the decision to focus on mid-level reforms to strengthen consensus is a _choice_ that the Bank and its counterparts can make in response to the model’s results; it is not by any means a foregone conclusion. Teams using the ABSM should remain mindful that it is not a 1:1 model of the world; instead, the ABSM is a powerful new tool in the Bank’s political economy toolbox.

# **V—Conclusion**

While the need for better political economy analysis to improve on-the-ground results is now roundly acknowledged across the World Bank and other development assistance institutions, adopting robust approaches that are both rigorous and operationally usable poses a substantial challenge. East Asia’s experience with the Agent-Based Stakeholder Model has yielded important lessons on the development of “practical political economy” tools.

The country cases showed a continuum of ABSM applications at up-, mid-, and down-stream points in the operational cycle, with different policy implications. In Timor-Leste, the analysis was helpful in constructing the country and donor policy debate on civil service reform, though the prospects for short-term agreement were small; in Mongolia, the analysis helped identify key coalition partners and underscored the need for continuing, embedded analysis as electoral and economic conditions evolved; and findings in the Philippines underscored the need to tackle smaller reform modules on which broad consensus could be achieved, even if this meant scaling back on important reform aspirations. In general, the tool was helpful in crafting micro-political policy agreement on specific issues, even when the wider reform environment was inhospitable to broad advances.

While the predictive value of the tool was helpful in some instances, it came as something of a surprise that the ABSM’s heuristic contribution in helping teams think through the sequence and strategies for reform was particularly well received. This suggests that the ABSM would be wellsuited to convening country or sector strategy brainstorming and would be most helpful if carried out on a regular, institutionalized basis. The ABSM instrument was distinguished by the systematic, just-in-time, and dynamic quality of its analytic power, but the tool did not offer a stand-alone, silver-bullet solution and was most effective when complemented by other, broader types of political economy analysis. The relative intensity of the field protocol and the need for up-front investment in training must be noted as well, but the East Asia experience suggests that these could be overcome if the tool were adopted in a high-volume business model. In sum, ABSM offers a powerful addition to the political economy toolkit, if utilized thoughtfully. While East Asia’s pilot initiative was confined to a limited number of cases, the experience strongly suggests that mainstreaming ABSM as a regularly applied analytic instrument could significantly improve the operational track record of development institutions such as the World Bank in supporting politically realistic reforms in client countries.

27