# **The Evaluation Framework**

As part of the pilot exercise, a standardized assessment tool was developed to evaluate the process of applying and adapting the ABSM to country governance reforms as well as the model’s impact on country team decision-making. In focusing on the process, the evaluation framework aimed to get a sense of how the ABSM affected the way the country team approached a particular governance problem—what their expectations were at the outset, how the experience of systematic data collection and analysis shaped their views, and what effect, if any, the results had on their approach to the issue on the ground. The evaluation tool also focused on results, seeking to determine whether the ABSM improved the quality of political analysis underpinning Bank programming decisions. To this end, the framework’s questions (excerpted in Table 3) sought to examine whether the model’s recommendations were adopted and whether reform outcomes were as expected.

**Table 3:  ABSM Evaluation Framework**

|**Ex ante expectations**|**ABSM analysis and**<br>**recommendations**|**Immediate outcomes**|**Impact over time**|
|---|---|---|---|
|What is the Bank’s desired<br>level of reform?|What new insights and/or<br>relationships did ABSM<br>reveal?|Did the Bank redefine its<br>desired level of reform?|Did the altered strategy<br>affect the reform outcome?|
|Who are the key<br>stakeholders and what are<br>their relationships?|Did ABSM change your<br>views about the behavior<br>of key stakeholders or the<br>nature/difficulty of the<br>reform issue?|How did ABSM alter<br>relations with client or<br>other stakeholders?|Did the altered strategy<br>achieve its objectives?|
|What should the Bank’s<br>strategy be? What are the<br>advantages and<br>disadvantages of this<br>strategy?|Did the strategy change?|Did you try any strategies<br>that emerged from ABSM<br>analysis? What were your<br>expectations for this<br>strategy?|What would have<br>happened if the Bank had<br>not changed its approach?|
|How do you intend to use<br>ABSM?|For which issues and<br>operational stages is ABSM<br>most useful?|If you did not change the<br>strategy, why not?|What is the marginal<br>benefit of ABSM over<br>time?|

The ABSM evaluation was conducted as a set of _ex ante_ and _ex post_ interviews of each analytical team involved in the pilot. These country team evaluations revealed useful insights drawn from each stage of the model’s application—from thinking through the key elements of each reform issue and the relationships among stakeholders to testing some of the model’s more promising strategic recommendations. Table 4 provides a brief summary of key evaluation insights on four of the five applications of ABSM in the East Asia and Pacific Region.

In many of the multiple issues examined in each case, high initial expectations of reform progress were shown to be unrealistic in the face of strong opposition, often from expected quarters. In this respect, neither a broad civil service reform agenda in Timor-Leste nor the more targeted sequence of mining license reforms proposed in Mongolia appeared to have the necessary political traction on the ground for implementation.

In other cases, the ABSM yielded granular insights on sources of support for reform where the Bank’s initial expectations were originally more pessimistic. In the case of taxpayer reforms in the 20