Each of the RIA reforms was studied along the dimensions presented in Box 3, and for each of the dimensions a score was provided based on the degree of implementation:

- 0 - absence of the dimension;

- 0.5 - presence of certain aspects of the dimension but not fully implemented;

- 1 - full, or near full presence.

This allowed the creation of an index allowing us to measure the comprehensiveness of the reform design, where each reform is scored on a 0-6 scale. More detailed criteria for scoring on each of the six dimensions is available in Annex 1.

Based on the scores developed above, a minimum for a “good practice” RIA reform score was set. To appraise an RIA reform as _sufficiently well designed_ , it must receive a score of _at least 3.5 out a possible maximum of 6_ . This choice draws upon previous considerations by WBG, namely that: “RIA Light systems in developing countries will have _some of those elements_ , not all of them. [They] essentially provide a menu for developing countries to select RIA components that will work best for them when establishing or strengthening their RIA processes.” (WBG 2009, emphasis added)<sup>2</sup>

# Can we identify key factors causing certain RIA reforms to be unsuccessful?

Finally, we wanted to explore why some RIA reforms, despite having “ticked all the boxes” of good practice design, had not (yet) delivered sustainable RIA systems. The purpose of this part of the study was to articulate reasons for implementation challenges within a structured analytical framework, i.e. a theoretical causal mechanism<sup>3</sup> that can be subject to empirical qualitative testing. The research design consisted of a theorydriven empirical analysis of four case studies. The purpose of the case studies was to test causal or explanatory mechanisms for variations in RIA reforms’ success rates. This step included the construction of theoretical causal mechanisms through the assessment and contextualization of factors already identified in the literature and, most importantly, in the practice of implementing RIA reforms. The intention was then to test these factors (hypotheses) through process tracing (Beach and Pedersen 2013).

The factors constraining implementation are listed in Box 4. The factors comprise a pragmatic collection of anecdotal evidence and suggestions from other case studies. While some of these factors are echoed in the literature, they are primarily drawn from the experiences of practitioners in reform implementation. In other words, to some extent these factors consolidate “popular” explanations shared among RIA advocates and practitioners, which have never been systematically tested.

> 2 It is worth repeating that the best practices measured in this study have been selected based on their predicted importance and ease of use in a larger quantitative study. Nevertheless, there could be other best practices of importance to reform success, as well as nuances of the above list which cannot be easily captured through these indicators. This could for instance include comprehensiveness of the RIA system (very advanced and “heavy” RIA designs are believed to often lead to failures in young RIA systems).

> 3 “[...] Mechanisms [...] are a system that transfers causal forces from X to Y. In the system understanding the focus is on the process whereby causal forces are transmitted through a series of interlocking parts of a mechanism to produce an outcome” (Beach and Pedersen 2013)

10