metadata: authors: 0: author: Kraay, Aart
1: author: Tawara, Norikazu
count: World
docna: 0: docna: Can disaggregated indicators identify
            governance reform priorities ?
docty: Policy Research Working Paper
theme: Legal institutions for a market economy,Public expenditure, financial management and procurement
majtheme: Rule of law,Public sector governance
subtopic: Consumption,Fiscal & Monetary Policy,National Governance,Government Policies,Business Environment,Educational Sciences,Youth and Governance,Governance Indicators,Energy and Mining
teratopic: Environment,Macroeconomics and Economic Growth,Governance,Science and Technology Development
abstracts: cdata!: Many highly-disaggregated cross-country
            indicators of institutional quality and the business
            environment have been developed in recent years. The promise
            of these indicators is that they can be used to identify
            specific reform priorities that policymakers and aid donors
            can target in their efforts to improve institutional and
            regulatory quality outcomes. Doing so however requires
            evidence on the partial effects of these many very detailed
            variables on outcomes of interest, for example, investor
            perceptions of corruption or the quality of the regulatory
            environment. In this paper we use Bayesian Model Averaging
            (BMA) to systematically document the partial correlations
            between disaggregated indicators and several closely-related
            outcome variables of interest using two leading datasets:
            the Global Integrity Index and the Doing Business
            indicators. We find major instability across outcomes and
            across levels of disaggregation in the set of indicators
            identified by BMA as important determinants of outcomes.
            Disaggregated indicators that are important determinants of
            one outcome are on average not important determinants of
            other very similar outcomes. And for a given outcome
            variable, indicators that are important at one level of
            disaggregation are on average not important at other levels
            of disaggregation. These findings illustrate the
            difficulties in using highly-disaggregated indicators to
            identify reform priorities.
repnme: repnme: Can disaggregated indicators identify
            governance reform priorities ?
display_title: Can disaggregated indicators
            identify governance reform priorities
listing_relative_url: /research/2010/03/12028494/can-disaggregated-indicators-identify-governance-reform-priorities
topicv3: Education m1327048 470,Macroeconomic and Structural Policies
            m1327380 439,Governance Indicators m1327488 834,Iron m1503514 2931,Business Environment m1586411 2530
publishtoextweb_dt: 2010-03-31T00:00:00Z
historic_topic: Environment,Macroeconomics and Economic Growth,Governance,Science and Technology Development
no_of_pages: 53
model_extractions: WPS5254 Policy Research Working Paper 5254 Can Disaggregated Indicators Identify Governance Reform Priorities?

_Aart Kraay Norikazu Tawara_ The World Bank Development Research Group Macroeconomics and Growth Team March 2010
Policy Research Working Paper 5254

# **Abstract**

Many highly-disaggregated cross-country indicators of institutional quality and the business environment have been developed in recent years. The promise of these indicators is that they can be used to identify specific reform priorities that policymakers and aid donors can target in their efforts to improve institutional and regulatory quality outcomes. Doing so however requires evidence on the partial effects of these many very detailed variables on outcomes of interest, for example, investor perceptions of corruption or the quality of the regulatory environment. The analysis in this paper uses Bayesian Model Averaging (BMA) to systematically document the partial correlations between disaggregated indicators and several closely-related outcome variables of interest using two leading datasets: the Global Integrity Index and the Doing Business indicators. The authors find major instability across outcomes and across levels of disaggregation in the set of indicators identified by BMA as important determinants of outcomes. Disaggregated indicators that are important determinants of one outcome are on average not important determinants of other very similar outcomes. And for a given outcome variable, indicators that are important at one level of disaggregation are on average not important at other levels of disaggregation. These findings illustrate the difficulties in using highly-disaggregated indicators to identify reform priorities.

This paper—a product of the Macroeconomics and Growth Team, Development Research Group—is part of a larger effort in the department to study the causes and consequences of governance. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at akraay@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Can Disaggregated Indicators Identify Governance Reform Priorities?**

Aart Kraay (The World Bank) Norikazu Tawara (Kanto Gakuen University and Nihon University)

> 1818 H St. NW, Washington, DC, akraay@worldbank.org, and 200 Fujiaku‐cho, Ota, Gunma, Japan, nori.tawara@gmail.com, respectively.  We would like to thank Nathaniel Heller, Daniel Kaufmann, Eduardo Ley, Chris Papageorgiou, Luis Serven, and Stefan Zeugner for helpful discussions, and especially Martin Feldkircher and Stefan Zeugner for providing their R‐code for implementing Bayesian Model Averaging.  Financial support from the Japan Consultant Trust Fund and the Knowledge for Change Program of the World Bank is gratefully acknowledged.  The views expressed here are the authors’ and do not reflect those of the World Bank, its Executive Directors, or the countries they represent.
# **1.  Introduction**

Strong institutions, including a sound regulatory environment for private sector economic activity, are widely considered to be crucial to successful economic development. This consensus has been informed by a vast body of empirical evidence linking various measures of institutional and regulatory quality to development outcomes. Translating this empirical consensus into concrete policy advice for countries seeking to improve their institutional and regulatory environment has been much more difficult, however, as many of the empirical measures used in this literature have been short on specifics. For example, in one of the most influential papers in the “institutions matter” literature, Acemoglu, Johnson and Robinson (2001) proxy for institutional quality using the risk of expropriation, as perceived by analysts at a commercial risk rating agency. Similarly, in a seminal paper, Mauro (1995) documents the links between perceptions of corruption from a commercial risk rating agency and investment and growth rates across countries. Absent details on the specific policy interventions that might affect these perceptions of expropriation risk or corruption, providing policy advice based on such broad measures is a little bit like telling aspiring golfers that they should play more like Tiger Woods.

Recognizing this, a number of organizations have embarked on major efforts to develop much more disaggregated measures of specific details of the institutional and regulatory environment. The promise of such detailed and disaggregated indicators is to pinpoint specific areas in need of reform in order to improve institutional and regulatory outcomes. As noted by Global Integrity, which produces a very detailed set of over 300 indicators of public sector accountability mechanisms that we use in this paper, _“we view the [Global Integrity] Indicators' greatest strength as their ability to unpack governance challenges within a country into discrete, actionable issues rather than just single numbers or rankings. The richness of the data set ‐ more than 300 indicators for each country ‐ enables a discussion of how best to allocate limited political and financial capital when the challenges are many and the resources few._<sup>_1_</sup> _”_ . Global Integrity goes on to argue that “ _The Global Integrity Index and Integrity Indicators assist [foreign aid] donors by helping to prioritize governance and anti‐corruption challenges in a country, region, or globally. By providing an actionable roadmap for reform, donors can begin to sequence key governance interventions to tackle the most pressing anti‐corruption weaknesses in a country ‐ or help_

> 1 All of the quotations from Global Integrity are taken from their website, at this link accessed on November 12, 2009: <u>http://report.globalintegrity.org/methodology.cfm.</u>

2
_bolster those "pillars of integrity" that are functioning well. The Index empowers donors, both bilateral and multilateral, by offering a platform for evidence‐based reform efforts._ ” Similarly, the World Bank, which produces the very detailed Doing Business indicators of the business regulatory environment, notes that the goal of this exercise is “ _to provide an objective basis for understanding and improving the regulatory environment for business.”_<sup>_2_</sup> The World Bank has also developed considerable resources to developing and promoting other such “actionable” governance indicators (AGIs), of which they include Doing Business and Global Integrity as leading examples. According to the World Bank, the distinguishing feature of such “actionable” indicators is that they provide _“...convenient and replicable guidance on the features (rules of the game, organizational capabilities) for which reform interventions are likely to prove most helpful for improving the performance of particular governance elements”._<sup>_3_</sup> Realizing the promise of such detailed indicators to identify and prioritize specific reform efforts requires an understanding of the relative magnitude of the effects of each of the individual disaggregated indicators on the corresponding outcomes that policymakers might want to improve. For example, a policymaker interested in reducing corruption (or even just the perceptions of the prevalence corruption held by domestic or foreign investors) might want to know which of the over 300 individual measures comprising the Global Integrity Index would have the largest impact on corruption. Our premise is that a policymaker or an aid donor would not particularly care whether a country scores well on any specific disaggregated indicator (for example, the existence of an anticorruption commission), but rather cares whether improving such an indicator (for example, by creating an anticorruption commission) will actually reduce corruption. Similarly, a policymaker looking for the most “bang for her buck” in the area of business regulatory reform would want to know the magnitude of the partial effects of each of the many indicators in the Doing Business dataset before choosing the few on which she would like to expend her political capital to seek improvements in these areas.<sup>4</sup>

> 2 See http://www.doingbusiness.org/Documents/DB10_About.pdf.

> 3 <u>http://www.agidata.org</u>

> 4 Of course, the process of developing and implementing governance‐related reforms is much more complex that simply identifying those areas with the highest impact and acting on them.  In reality, policymakers must balance all of the political and financial costs and benefits of reforms in particular areas.  All we attempt to do here is to shed some light on the difficulty of quantifying a narrow measure of the benefits of reforms, which is their estimated impact on outcomes.

3
On this crucial question of partial effects of disaggregated indicators on outcomes of interest, empirical evidence has not kept pace with the proliferation of very detailed indicators of institutional quality and the regulatory environment. While identifying these partial effects is central to realizing the promise of disaggregated indicators to identify specific reform priorities, in this paper we argue that it is also extremely difficult – if not impossible – to do so convincingly. The problem is simply one of degrees of freedom: the more disaggregated indicators become, the more partial effects of individual indicators on outcomes of interest there are to be estimated, and the less precisely each individual partial effect can be estimated. For example, the overall Ease of Doing Business ranking of the Doing Business project consists of 41 measures of the business regulatory environment, for 181 countries. This means that there are on average just 4.5 data points with which to estimate the partial effects of each individual indicator on some outcome of interest in a cross‐sectional regression. In the case of Global Integrity, the degrees‐of‐freedom problem is even more stark, as there are over 300 variables in this dataset which spans just 92 countries, so that it simply is infeasible to estimate the partial effects of each of them in a single encompassing cross‐country regression including all indicators.

One possible solution to this degrees of freedom problem is to apply data reduction techniques of some sort. That is, one might simply re‐aggregate the highly disaggregated variables by averaging them together in some way. For example, one could simply average together the 41 indicators of the business regulatory environment in the Doing Business dataset to obtain the overall “Ease of Doing Business” ranking, and then estimate a regression of the outcome of interest on this overall ranking. However, this amounts to imposing the restriction that the partial effects of each of the 41 individual variables are the same and equal to 1/41 of the impact of the aggregate indicator. And if this really were true, then it would not matter at all which dimension of the regulatory environment is improved, since each individual indicator is assumed to have the same effect on outcomes. This seems quite implausible, and moreover contrary to the entire spirit of developing such disaggregated and ‘actionable’ indicators. A slightly more sophisticated and commonly‐used alternative would be to extract the first principal component of the 41 indicators and use it as an explanatory variable. But this would be no improvement. The first principal component is simply a weighted average of all of the individual indicators, with weights proportional to their intercorrelations. Thus, the few variables that happen to be highly correlated with each other would receive more weight in the aggregate. But there 4
is no reason to expect that these variables that happen to be highly correlated with each other also are those that have the largest effects on the outcome of interest.<sup>5</sup> A different approach to solving the degrees‐of‐freedom problem would be to simply choose a subset of the many disaggregated indicators that seem most plausible _a priori_ , and try to estimate a regression of the outcome of interest on this smaller set of preselected variables. The advantage of this approach is that it tries to more precisely estimate the effect of a few of the disaggregated indicators on the outcome of interest by imposing the restriction that the remaining indicators have no effect. However, the problem of course is that this will give valid estimates of the partial effects of the included variables only if they are orthogonal to all of the other disaggregated indicators that matter for outcomes but are not included in the regression. This is the standard problem of omitted variable bias. Moreover, this approach of preselecting a subset of regressors throws open the door to specification searching or data mining for a subset of variables that happens to confirm the researcher’s or policymaker’s priors.

To avoid these problems of specification searching, we instead use Bayesian Model Averaging (BMA) to systematically document the partial effects of many disaggregated indicators on outcomes of interest. We do this using disaggregated indicators from Global Integrity and Doing Business, in three steps. First, we identify a set of potential outcome variables for each of these two sets of disaggregated indicators. These variables capture outcomes we think a policymaker might reasonably want to influence by reforming the areas captured by these indicators. For reasons we elaborate below, we choose seven closely‐related subjective measures of corruption as outcome variables linked to the Global Integrity Index, and seven closely‐related subjective measures of the quality of the regulatory environment as outcome variables for Doing Business. Second, for each combination of outcome variable and disaggregated indicator set, we use BMA to obtain estimates of the partial effects of each of the disaggregated indicators on the corresponding outcome of interest. Third, once we have identified the partial effects of the disaggregated variables on each outcome, we compare these results across different outcome variables and across different levels of disaggregation.

The good news is that we find BMA to be an effective tool for identifying a relatively small number of disaggregated indicators that display strong partial correlations with a given outcome of

> 5 Lubotsky and Wittenberg (2006) elaborate on this point, showing that the use of multiple proxies as explanatory variables in a linear regression generally dominates the use of a single summary of those proxies.

5
interest. In this respect, we join a growing literature in recognizing the value of BMA as a tool for systematically identifying robust partial correlates of outcomes when the precise empirical specification is unknown. However this positive message is tempered by two important pieces of bad news. The first is that there is a great deal of instability _across very similar outcomes_ in terms of which variables the BMA procedure identifies as important partial correlates of outcomes. To take the most extreme example, our seven corruption outcome variables have a quite strong average pairwise correlation of 0.62. Yet when we compare across these very similar outcomes, we find that there is virtually no overlap in the subsets of the 303 disaggregated individual indicators in GII that are identified as important determinants of these outcomes by the BMA procedure. The second is that there is a very high degree of instability _across levels of aggregation_ in terms of which individual indicators are identified as important determinants of outcomes by BMA. In particular, we find that the probability that an individual disaggregated indicator is identified as an important determinant of outcomes is not significantly increased by knowledge that the higher‐level aggregate to which it belongs was identified as an important determinant of outcomes. And conversely, knowing that a higher‐level aggregate is identified by BMA as an important determinant of an outcome does not mean that the more disaggregated variables on which it is based are more likely to be identified as important determinants of the same outcome.

These results suggest that it may be very difficult to use the disaggregated indicators of institutional and regulatory quality that we examine here to provide policy advice to guide reforms in these areas. Under the reasonable assumption that policymakers would like to identify high‐impact reforms that that matter for outcomes, it becomes important to identify which those reforms are. Yet we find that quite small changes in the empirical proxies for outcomes that we consider lead to wild fluctuations in the set of variables that are identified as important for those outcomes.

The rest of this paper proceeds as follows. In Section 2 we provide details on the Global Integrity and Doing Business datasets that we use in our empirical analysis, and we justify our selection of outcome variables corresponding to these datasets. In Section 3 we explain the Bayesian Model Averaging methodology. Section 4 contains the results and Section 5 discusses the robustness of the results and caveats. Section 6 offers conclusions.

6
# **2.  A First Look at the Data**

We illustrate the challenge of identifying relevant determinants of governance outcomes based on very disaggregated governance indicators using two leading datasets. The first is the Global Integrity Index (GII), compiled by Global Integrity, a Washington‐based advocacy organization. Quoting from its mission statement, _“Global Integrity generates, synthesizes, and disseminates credible, comprehensive and timely information on governance and corruption trends around the world. As an independent information provider employing on‐the‐ground expertise, we produce original reporting and quantitative analysis in the global public interest regarding accountable and democratic governance. Our information is meant to serve simultaneously as a roadmap for engaged citizens, a reform checklist for policymakers, and a guide to the business climate for investors.”_ The GII reports over 300 individual variables that score countries on various highly‐detailed dimensions of institutions that matter for public sector integrity and accountability. The individual questions on which the GII is based are scored by locally‐recruited experts (typically one per country), and are then vetted by an anonymous peer‐review process involving 3‐5 reviewers per country.

The GII can be disaggregated at three levels. The overall index is organized into six main categories (Civil Society, Public Information and the Media, Elections, Government Accountability, Administration and Civil Service, Regulation and Oversight, and Anticorruption and Rule of Law). These six are further disaggregated into 23 sub‐categories (e.g. Elections is further decomposed into Voting and Citizen Participation, Election Integrity, and Political Financing). And finally these 23 subcategories are built up from the 303 individual variables. For example, Election Integrity is an average of 15 separate questions relating to the existence and effectiveness of electoral monitoring bodies. A key and extremely valuable feature of GII is that it consistently matches up questions about _de jure_ rules and the _de facto_ implementation of these rules. For example, within Government Accountability, question 12a assesses whether “In Law, citizens have a right of access to government information and basic government records (Yes/No)”. This is followed by question 13a which asks “In practice, citizens receive responses to access to information requests within a reasonable time (0‐100 scale)”. In our use of the GII data we systematically distinguish between the “In Law” and “In Practice” questions in GII. In particular, we decompose each of the six GII main categories into separate averages of all the “In Law” and “In Practice” questions, resulting in 12 high‐level aggregates. Similarly, we disaggregate each of the 7
23 sub‐categories into averages of the corresponding “In Law” and “In Practice” questions, resulting in 45 indicators at the more disaggregated level.<sup>6</sup> GII has cumulatively covered 92 countries since its inception in 2004, some for multiple years. We take data from the 2007 and 2008 waves of GII, covering 50 and 46 countries respectively (available from the GII website as of February 2009). We take all of the 46 countries covered in 2008, and add to this 24 countries covered in 2007 but not in 2008, to obtain a cross‐section of 70 countries in total. In some cases our sample size will be slightly smaller depending on the country coverage of the outcome variables we work with. The 2008 questionnaire contains 320 individual items, while the 2007 questionnaire covers 304 items. The overlap between the two is extremely close, and after merging the two questionnaires we have a total of 303 questions asked in both years.<sup>7</sup> Of these, 184 are “In Practice” questions and 119 are “In Law” questions.

The second dataset we use is the Doing Business (DB) indicators produced by the World Bank. The DB indicators cover 10 dimensions of the business regulatory environment (Starting a Business, Dealing with Construction Permits, Employing Workers, Registering Property, Getting Credit, Protecting Investors, Paying Taxes, Trading Across Borders, Enforcing Contracts, and Closing a Business). These are based on 41 individual measures. For example, the “Starting a Business” measure is itself based on four sub‐indicators, measuring (1) the number of procedures, (2) the number of days, (3) the cost of associated fees, and (4) the minimal capital requirement, required to start a new business. The DB data is scenario‐based. Respondents are provided with a very detailed scenario about a hypothetical transaction, for example, registering a firm with particular characteristics in the capital city of the country. The data collected by DB correspond to what a hypothetical firm described in the scenario would experience. Since the subcomponents of each of the 10 DB measures are measured in different units, countries are first ranked on the individual variables. These ranks are then averaged within each of the 10 broad indicators to arrive at the indicator ranks. Finally, the average ranks on 10 indicators

> 6 One of the GII subcategories, Anticorruption Law, consists exclusively of “In Law” questions and so there is no corresponding “In Practice” aggregate for us to construct.  This is why we have only 45, and not 46 variables at this level of disaggregation.

> 7 We focus on the 2007 and 2008 questionnaires which are most comparable to each other.  A few questions were asked at a more detailed level in the 2008 data when compared with 2007.  We therefore average the following pairs of questions in the 2008 data to make them comparable to their 2007 analogues : questions 20b and 21b; 20e and 21d; 20f and 21e; 20g and 21f; 22a and 23a; 22b and 23b; 22d and 23c; 22e and 23d; 22f and 23e; 24a and 25a; 24b and 25b; and 24c and 25c. We also delete questions 21a, 46a, 46e, and 46i from the 2008 questionnaire that we not asked in 2007.  This reduces the number of individual questions in 2008 to 303.

8
themselves are averaged to arrive at the overall Ease of Doing Business ranking.<sup>8</sup> The DB respondents consist primarily of locally‐recruited attorneys familiar with the relevant laws that form the basis for the DB summary measures. In contrast with GII, DB is primarily focused on collecting _de jure_ as opposed to _de facto_ information, and so we cannot distinguish the DB indicators along this dimension as we do for GII.

Our next step is to identify outcome variables of interest corresponding to these two datasets of potential policy interventions. Our objective in doing so is to try to identify outcomes sufficiently close to the indicators themselves that a policymaker might reasonably consider trying to affect this outcome through policy reforms that would be identified and captured by changes in the individual indicators. In the case of GII, Global Integrity is very explicit that the goal of the GII is to provide guidance on specific indicators to be improved (recall quotes in introduction). Global Integrity also encourages users of the GII to view it as _“...a powerful variable with which to explore other key development indicators — economic growth, income distribution, health and education rates, and other key socio‐economic indicators.”_ While ultimately there surely are links between the dimensions of governance captured by GII and these very broad outcomes, we set ourselves the more limited goal of assessing the links between the many disaggregated variables in GII and more proximate outcomes related to corruption itself. After all, according to Global Integrity the GII “... _represent one of the world's most comprehensive data sets providing quantitative data and analysis of anti‐corruption mechanisms and government accountability in diverse countries around the globe._ ” Global Integrity goes on to very sensibly note that the relationship between the GII and corruption is unlikely to be perfect, cautioning that: _“...users should not necessarily interpret high scores on the Global Integrity Index as reflective of countries where there is no corruption. Instead, those results should simply be understood to reflect circumstances where key anti‐corruption safeguards exist and have been enforced, which while one would hope reduces corruption may not eliminate it entirely. In simple terms, corruption can and will occur even where societies have implemented what are understood to be ideal reforms.”_ Based on this, we think it is reasonable to investigate the links between the many disaggregated GII measures and direct proxies for corruption itself. Of course, corruption is very difficult to measure directly, and the vast majority of empirical measures of corruption are based on the perceptions of

> 8 There are missing values for some countries on some of the DB variables.  We follow DB’s practice of assigning the lowest possible rank to such observations prior to averaging ranks across indicators. The overall DB ranking is a simple average of the ranking on the 10 subcategories.

9
survey respondents. This is not necessarily a handicap – as argued in Kaufmann and Kraay (2008), not only do subjective assessments of corruption provide valuable information, but also policymakers should care about these perceptions because respondents act on them.<sup>9</sup> We draw on seven different measures, all of which are taken from the Worldwide Governance Indicators project (see <u>www.govindicators.org, and Kaufmann, Kraay and Mastruzzi (2009) for descriptions). Five of these are</u> expert assessments of the prevalence of corruption taken from commercial business information providers (Economist Intelligence Unit (EIU), Political Risk Services (PRS), Global Insight Global Risk Service (DRI), Global Insight Business Risk Conditions (WMO), and Cerebus Corporate Intelligence Gray Area Dynamics (GAD)). One additional expert assessment is the World Bank’s Country Policy and Institutional Assessment (CPIA). Finally, we draw on responses from a large cross‐country survey, the Global Competitiveness Report (GCS) survey of firms in 134 countries, that asks firm managers a variety of questions about corruption. Table 1 lists the precise questions about corruption assessed by each of these sources.

Given DB’s emphasis on the business and regulatory environment, we adopt the same strategy as with GII of using closely‐related perceptions of the quality of the business environment as corresponding outcome variables. The DB project provides as evidence of its own relevance strong correlations of the overall DB measure with other leading indicators of the business environment produced by OECD and World Economic Forum.<sup>10</sup> We follow a similar approach here, relating the DB indicators to a set of seven outcome variables that are also taken from the Worldwide Governance Indicators project, and capture a variety of perceptions regarding the quality of the regulatory environment. We use data from the same six expert assessments as we do for GII, but now focused on the regulatory environment, as well as data from the Global Competitiveness Survey. Moreover, we observe that Doing Business is also circumspect about the limited nature of its indicators, sensibly noting that _“Doing Business does not measure all aspects of the business environment that matter to firms or investors—or all factors that affect competitiveness. It does not, for example, measure security, macroeconomic stability, corruption, the labor skills of the population, the underlying strength of_

> 9 In some cases policymakers may very well have the immediate objective of influencing these perceptions directly, either to improve the government’s polling results or to improve the country’s standing in cross‐country rankings based on these corruption assessments.

> 10 See documentation provided at http://www.doingbusiness.org/Documents/DB10_About.pdf

10
_institutions or the quality of infrastructure.”_ The precise questions about the regulatory environment assessed by each of these sources can also be found in Table 1.

We emphasize that our choice of outcome variables for the GII and DB datasets is not intended to be exclusive in any sense, but rather is purely illustrative. These outcomes are surely not the only ones that policymakers might want to influence by reforms to the policies and institutions captured by GII and DB. Rather, we think these particular outcomes might plausibly be among the many considered by policymakers, and provide a good illustration of the challenges of identifying the partial effects of the many disaggregated indicators making up these datasets. We recognize also that in the case of GII, the objective of the GII is broader than simply measuring corruption, but rather seeks to document the disaggregated ingredients of a wide range of transparency and accountability mechanisms.<sup>11</sup> Nevertheless, one can readily rationalize looking at narrower measures of corruption as a relevant outcome variable for GII by noting that corruption can be viewed as a symptom of the failure of such transparency and accountability measures, and so would be a reasonable proxy outcome for policymakers to consider.

Before turning to the formal analysis of the links between disaggregated indicators and outcomes, we document two important features of the data. The first is that both the overall aggregate GII and DB measures are in fact strongly correlated with each of the outcome variables. We show this in Table 2 , which summarizes the results of regressing each of the outcome variables on its corresponding aggregate GII or DB measure, both unconditionally (in the top panel), and conditionally controlling for log per capita GDP (in the bottom panel). A unit increase in the overall measure of GII will increase a measure of each outcome variables by 0.8 on average statistically significantly. Conditioning on per capita GDP (in logs) only slightly reduce the effects and significance of the overall GII measure on each of the 7 outcome variables. The effects of the overall DB measure on each of the 7 outcome variables are more significant and greater in size, both unconditionally and conditioning on GDP per capita. Of course, we cannot interpret these correlations in Table 2 as purely reflecting a causal effect from the DB and GII indicators to the outcomes of interest – there are many potentially confounding omitted

> 11 We note that Global Integrity does place a disclaimer on its website to the effect that the GII do not measure corruption: _“...it is worth emphasizing that the Integrity Indicators do not measure corruption but rather assess its opposite, that is, anti‐corruption and good governance institutions, mechanisms, and practices. While corruption and bribery are difficult if not impossible phenomena to capture empirically, assessing the performance of key integrity‐promoting mechanisms such as civil society, the media, and law enforcement provides a much more concrete access point through which to analyze and monitor government accountability.”_ .

11
variables. However, it seems reasonable to think that they at least in part reflect an effect running from the specific institutions and regulations measured by these two datasets to the relevant outcomes. To the extent that this is the case, our goal in this paper is to document the extent to which these correlations between the aggregate GII and DB measures and outcomes of interest can be unbundled into differential impacts of the many highly‐detailed subcomponents of these broader measures.

This raises the question of a second feature of the data that we want to document before moving on: the many disaggregated variables underlying these two broad variables have surprisingly (at least to us) low intercorrelations among themselves. We summarize these in Table 3. The rows of Table 3 correspond to the GII and DB datasets at varying levels of disaggregation. For each level of disaggregation, we compute all of the pairwise correlations between the variables at that level of disaggregation. Then we summarize the distribution of these (many!) estimated correlations by reporting the 10<sup>th</sup> , 25<sup>th</sup> , 50<sup>th</sup> , 75<sup>th</sup> and 90<sup>th</sup> percentiles for each combination of indicators and levels of disaggregation. For example, the median pairwise correlation among all combinations of GII variables is 0.4 at the GII‐12 level of disaggregation, 0.25 at the GII‐45 level of disaggregation, and just 0.09 at the GII‐303 level of disaggregation. At this highest level of disaggregation, fully 90 percent of all pairwise correlations are less than 0.35. In the case of Doing Business, the median pairwise correlations are just 0.32 and 0.18 at the DB‐10 and DB‐41 levels of disaggregation. These quite moderate pairwise correlations between the disaggregated indicators are a key feature of the data because they highlight their potential to be informative about their corresponding outcomes. Had these individual disaggregated indicators been very highly correlated with each other, it would have been obvious a priori that it would be very difficult to identify the partial effects of any one of them due to problems of strong collinearity. However, this does not appear to be a major problem in the GII and DB data.<sup>12</sup> For comparison purposes, we also document the distribution of the pairwise correlations between the outcome variables of interest. There is a striking contrast here with the individual indicators. The outcome variables are quite strongly correlated with each other,with a median pairwise correlation of 0.61 for the GII outcomes measuring corruption, and 0.68 for the DB outcomes measuring the regulatory environment. We interpret these high correlations as suggesting that these different candidate dependent variables are measuring broadly similar outcomes.

> 12 Of course, these pairwise correlations are not sufficient to indicate or rule out problems of collinearity in models with more than two explanatory variables.  We discuss below in Section 5 why our main instability results are likely not due to problems of collinearity among regressors.

12
# **3.  Bayesian Model Averaging**

We now describe in some detail the Bayesian Model Averaging (BMA) procedure that we will use in the remainder of the paper to document the partial correlations between the disaggregated GII and DB indicators and their corresponding outcome variables. Over the past several years BMA has become a widely‐used tool for assessing the robustness of regression results to variations in the set of included control variables. The seminal application to cross‐country growth empirics is Fernandez, Ley and Steel (2001), followed by Sala‐i‐Martin, Doppelhofer and Miller (2004), and then many others. Brock, Durlauf and West (2003) particularly emphasize the decision‐theoretic aspects of BMA as a useful tool for guiding policy choices. Recently Ciccone and Jarocinski (forthcoming) have used BMA to document the non‐robustness of growth empirics to minor data revisions in the dependent variable, which is closely related to our finding of instability across alternative outcome variables. There is also an active literature extending the BMA methodology in various dimensions, including collinear regressors (Durlauf, Kourtellos, and Tan (2009)) panel data applications (Moral (2009)), and instrumental variables estimation (Eicher, Lenkoski, and Raferty (2009). Finally, several papers including Fernandez, Ley and Steel (2001), Ley and Steel (2009), Eicher, Papageorgiou and Raferty (2009) and Feldkircher and Zeugner (2009) all discuss the consequences of alternative prior assumptions for the outcome of BMA.

The basic idea of BMA is simple. Rather than base inferences about parameters of interest on just one preferred model consisting of one particular set of explanatory variables, BMA combines inferences about parameters of interest across many candidate models corresponding to different sets of explanatory variables. To be more precise, let y denote an Nx1 vector of observations on the dependent variable of interest, and let X denote an NxK matrix of potential explanatory variables for y. Let j ��1,2, ... , 2<sup>K</sup> � index models, distinguished by their included set of regressors. In particular let Xj denote an NxKj matrix containing a subset of KjK regressors from X. A model j consists of a linear regression of y on the variables in Xj, i.e.: where  is an Nx1 vector of ones and j is an Nx1 vector of i.i.d. normal disturbances with zero mean and variance �<sup>�</sup> .The scalars  and  _j_ and the Kjx1 vector  _j_ are the parameters of model j, and following the bulk of the literature on BMA we use Zellner’s g‐prior for them, i.e.

13
f (aj, Bj,0) xa * (0, ~~=~~ 2 04%)" 1 <u>149</u> (x; a, b) PIMjly, X] L[M,|y,X]

PIMjly,X] « LIM;|y, X}p[M;] p[M;]

<u>g</u> ~~Ky~~ 2\- ~~2~~ L[M;jly,X| « ~~(7)~~ <u>\2</u> (: - ~~—)Rj~~ °

R?
L= 6K P[M,IM] ~~_——Bb~~ YK « ~~(@—)~~ Uu<K/2 bh ~~AS~~ 2\7 none \Kiv ~~g~~ <u>\o</u> ~~R;~~<sup><u>\e—)</u></sup> <u>\i4g</u> ~~1+g~~ PIMIy<sup>Xl“</sup> ~~(ea) a)~~ ( ~~nit~~ Ll Ll u = 0.25K
ensure that the inferences from any given model mimic closely traditional frequentist ones, and accordingly we set g to be small, i.e. g=0.01, so that the shrinkage factor is very close to one.

We note however that fixing the prior parameters based on these objectives of course does not avoid the problem of sensitivity of results to prior choices. Of primary concern here is the choice of g, which as noted above plays two roles in the assignment of posterior probabilities across models: lower values of g increase the model size penalty for adding additional regressors, and also increase the sensitivity of the posterior probability to improvements in R‐squared. Together these two forces imply that when g is small, the posterior probability will tend to concentrate on models with few regressors, and among these, on models with high R‐squareds. This concentration of posterior model probabilities on a few models can be extreme, which Feldkircher and Zeugner (2009) label the “supermodel effect”. And this in turn can lead to a strong concentration of high PIPs on just a few variables. Potentially, this “supermodel effect” can lead to very large changes in posterior model probabilities and PIPs as we move from one dependent variable to another. However, as we discuss further below, our main finding of instability across outcomes will not be driven by this effect.<sup>14</sup> We note that implementing BMA in principle poses major computational problems, as the number of models to be estimated and averaged increases in the number of explanatory variables at the rate 2<sup>K</sup> . When K=303 as is the case in GII, this is an astronomically large number of models (1.6 followed by 91 zeros!). Even for the more moderate K=41 in DB, there are still over two trillion (2.2x10<sup>12</sup> ) potential models to consider. Fortunately, fast and accurate algorithms for identifying and sampling only those models with the largest posterior probabilities have been developed, greatly reducing the computational burden, and we rely on them here.<sup>15</sup> Following the BMA literature, the posterior distribution is approximated by simulating a sample from it by applying MC3 sampler (Madigan and York 1995, as described in Fernandez, Ley and Steel (2001)). We also follow Fernandez, Ley and Steel (2001) in using the correlation between analytical and empirical posterior model probabilities as a criterion for convergence of the sampling chain. We will report results in the next section from a simulation run with a burn‐in of 100,000 discarded drawings and N million recorded drawings, where N is 0.2, 0.5, 1, 0.1, and

> 14 See also Ciccone and Jarocinski (forthcoming) who find that the set of cross‐country growth determinants identified as robust using BMA changes drastically as the dependent variable changes using different revisions of the Penn World Tables.  Feldkircher and Zeugner (2010) argue that this sensitivity is largely driven the authors’ choice of a small value for g.

> 15 We are very grateful to Martin Feldkircher and Stefan Zeugner  whose R‐code (available at http://feldkircher.gzpace.net/links/bma) we used to implement BMA in this paper.

16
0.3 for cases of GII‐12, GII‐45, GII‐303, DB‐10, and DB‐41. We choose this number so that a high positive correlation between posterior model probabilities based on empirical frequencies and the exact analytical likelihoods is obtained. We also report estimated total posterior model probabilities visited by the chain using a measure of George and McCulloch (1997).

Note also that in our GII application the number of candidate explanatory variables far exceeds the number of countries when we work with the GII data at the highest level of disaggregation (K=303 but N=70). This means that the space of models to be considered potentially includes very many models that cannot feasibly be estimated since there are more regressors than observations. In this paper we take the shortcut of ignoring this feasibility constraint. We find that, given our choice of prior parameters, the BMA algorithm is able to cover a very high fraction of the posterior probability without ever attempting to visit infeasible models with K>N.<sup>16</sup> This is because our choice of priors ensures that the model size penalty is sufficiently high that such large models have extremely small posterior probabilities.<sup>17</sup> Finally we acknowledge at the outset the important caveat that we are combining inferences from a series of very simple linear OLS regressions. As such, all of our conclusions are subject to the usual limitations of such a model. In particular, a maintained assumption is that the error term is independent of the regressors in all models, an assumption that would clearly be violated if there were reverse causation or omitted variables. We also by assumption rule out any plausible nonlinearities such as interactive effects between variables. As we discuss further below however, addressing these very likely important issues we think would only further reinforce our basic point – that it is extremely difficult to identify a small subset of indicators that are robust determinants of outcomes of interest.

16 More formal approaches to the K>N problem are available. One approach is to formally apply a prior weight of zero to infeasible models. Another example is Eicher, Papageorgiou and Roehn (2007) who propose an iterative BMA procedure applicable when K>N. For a non‐Bayesian approach to this problem, see Candes and Tao (2005) propose an algorithm that can reliably estimate the parameters of a single “true” model with Kj<N variables from a dataset of candidate variables X where K>>N even when the subset of the columns of X included in the “true” model is not known a priori.

17 To see this, note that in the case of GII‐303, the largest model visited by the BMA algorithm has 16 explanatory variables when CPIA is an outcome variable, 25 when DRI is an outcome variable, 19 when EIU is used, and so on. Consider now comparing this with the smallest infeasible model with ki=70. Given K=303 and our choices of =10 and g=0.01 , the model size penalty in the first two terms of Equation (6) would be on the order of 1.7x10<sup>‐99</sup> , suggesting that infeasible models have vanishingly small posterior probabilities and so can safely be ignored. This dimensionality problem also means that we cannot use alternative model selection techniques based on encompassing regressions advocated by Hendry and Krolzig (2004, 2005).

17
# **4.  Results**

In order to develop familiarity with the methodology, we begin by discussing in some detail the results of the BMA exercise for GlI, at the least‐disaggregated level, that are reported in Table 4A. Subsequent tables report the same information, for higher levels of disaggregation of GII (Tables 4B and 4C), and for DB (Tables 5A and 5B). In Table 4A we have K=12 candidate right‐hand‐side variables in the X matrix, and the 7 choices of outcome variables y discussed in Section 2. The rows of the main part of Table 4A correspond to these right‐hand‐side variables, identified in the first column (with the prefixes “LQ” and “PQ” denoting the “In Law” and “In Practice” questions in GII). The sets of columns of Table 4A correspond the different outcome variables.

For each outcome variable, Table 4A first reports the Posterior Inclusion Probability (PIP) for each variable. This is simply the sum of the posterior probabilities across all models in which the variable appears. A high PIP indicates that the set of models in which the variable appears jointly has a high posterior probability. Consider for example the CPIA outcome variable, measuring World Bank country economists’ assessments of “Public Sector Transparency and Accountability”. The variable GII variable “PQElections” has a high PIP of 0.934. This means that 93.4 percent of models on a probability‐ weighted basis include this variable, which captures GII respondents’ views of the _de facto_ fairness of elections. The second‐most important GII variable is PQOversightRegulation, which appears in 21.4 percent of models on a probability‐weighted basis, and the third‐most important variable for this outcome is PQ3GovernmentAccountability, which appears in 17.5 percent of models. The remaining GII variables all appear much less important, in the sense that the models in which they appear have much smaller posterior inclusion probabilities.

We also report some summary statistics on the distribution of posterior probabilities across models at the bottom of Table 4A. We first report the posterior probability of the top three models (ranked by posterior probabilities), and then also the number of models required to cover 50 percent, 75 percent, and 90 percent of the posterior model probabilities. In the case of CPIA, the top three models have posterior probabilities of 41 percent, 12 percent, and 8 percent respectively. We also see that the posterior probabilities are quite concentrated across models. The top two models alone account for 50 percent of the posterior probability, and only 9 (27) models are required to account for 75 percent (90 percent) of the posterior probability. This concentration of posterior probabilities is also reflected in the 18
posterior mean model size: the posterior probability‐weighted average number of regressors (across all possible models) is just 1.73.

While these posterior model probabilities, and associated inclusion probabilities for individual variables, are a useful way of summarizing the relative importance of particular models and variables, we do not want to overinterpret the precise magnitude of these probabilities and their concentration across models. This is because, as noted in the previous section, the concentration of posterior probability mass across models is sensitive to our choice of prior parameter g. When g is chosen to be small (which we do in order to mimic standard frequentist inference for a given model), the BMA algorithm is more sensitive to small differences in model fit when assessing the relative probabilities of models. As a result, posterior probabilities are more concentrated across models, and similarly, posterior inclusion probabilities are more strongly concentrated on fewer variables. Instead, we simply emphasize the _ranking_ of models and variables by their posterior probabilities. In particular, in Table 4A we have highlighted the top three variables ranked by their PIPs for each outcome variable. This allows us to identify at a glance the relatively most important determinants of each outcome without reference to the precise magnitude of the variables’ inclusion probabilities, which in some cases is quite small. We also think that this exercise of picking the top few variables as ranked by PIP is analogous to the kind of exercise that a policymaker interested in allocating scarce political capital across a few high‐impact reforms might do. In what follows we will refer to these variables with the highest PIPs as the most “important” or most “significant” for a given outcome variable even though this terminology is somewhat imprecise.

In the second and third column for each outcome variable we report the posterior mean and standard deviation of the slope coefficient corresponding to that variable. Note that these are unconditional means and standard deviations, i.e. averaging across all models including those in which the variable does not appear and for which the slope coefficient is then by definition zero. To obtain the posterior mean conditional on inclusion, we need to scale the reported mean by the inclusion probability. Returning to the CPIA as a specific example, the variable with the highest PIP, PQ2Elections, has a posterior mean for the slope coefficient equal to 0.44.<sup>18</sup> This is the expected impact of this variable on the CPIA, averaging across all models. Considering only models in which this variable

> 18 To interpret the magnitude of these coefficients, note all variables are scaled to run from 0 to 1.  So a chance in the value of PQ2Elections from its worst possible value of 0 to its best possible value of 1 would lead to an increase in the CPIA of 0.44 (also on a scale from 0 to 1

19
appears, the expected impact is slightly larger at 0.44/0.93=0.47. We note also that the ranking of variables by their PIPs is very similar to the ranking of variables by the posterior means of their associated slope coefficients. This tells us that variables that are “important” in the sense of having high PIPs also have high expected impacts on the outcome variable.

Looking across the various outcome variables in Table 4A, we observe a number of consistent patterns. Posterior mean model size is fairly small, in the vicinity of 2 for all outcome variables, and posterior model probabilities are concentrated on a fairly small number of models: at most the top 6 models together account for half of the posterior probability. Perhaps most interesting are the patterns _across_ outcome variables in those variables highlighted as having high posterior inclusion probabilities. Two variables, PQ2Elections and PQ5OversightRegulation, consistently appear among the top three explanatory variables ranked by PIP across the choices of outcome variables (for all 7 outcome variables). In contrast, however, 4 of the 12 variables in Table 3A do not appear in the list of top three explanatory variables for _any_ of the outcome variables, and a further five have high PIPs for just one or two outcome variables. Notably, all six of the “In Law” questions are included in this group of eight variables with low explanatory power.<sup>19</sup> An attractive feature of these results is that they suggest a very consistent pattern across the different – and very closely‐related – corruption variables used as outcomes in Table 4A. Variables that have high PIPs tend to do so consistently across nearly all outcome variables, and similarly, variables with low PIPs tend to do so fairly consistently as well across all outcome variables. To provide a more formal method of documenting this feature of the results, we perform the following simple non‐ parametric test. By construction, 25 percent of the explanatory variables are highlighted for each outcome variable (since we have highlighted the top three out of 12 explanatory variables). Suppose as a null hypothesis we assume that the event that an explanatory variable is included in the ‘top three’ list for outcome variable i is independent of the event that it is in the ‘top three’ for outcome j. This would correspond to the extreme case of no stability whatsoever across outcome variables in terms of which explanatory variables are identified as important by the BMA procedure. Under this null hypothesis, the probability of observing an explanatory variable highlighted as being in the top three for none of the

> 19 In fact, it is surprising that one of these “In Law” variables that makes a top‐three list

> (LQGovernmentAccountability, for dependent variable PRS), actually has a _negative_ posterior mean for the slope coefficient, indicating a _negative_ partial correlation with the corruption outcomes.

20
outcome variables would be (1‐0.25)<sup>7</sup> =0.13. The fact that we observe 4 out of 12 or 33 percent of variables in this category is evidence against the null of independence.

More generally, under the null of independence across the seven outcome variables, the distribution of the number of outcomes for which an explanatory variable is in the top three list is a binomial random variable with 7 trials and a success probability of 0.25. We can then compare the predicted proportions from this distribution with the observed proportions, using a standard chi‐ squared test.<sup>20</sup> Performing this test we strongly reject the null of independence across outcome variables (with a p‐value of 0.00), and so conclude that there is a great deal of stability across outcome variables in terms of which explanatory variables at this high level of aggregation that are identified as important by having large PIPs. However, as we shall shortly see, this very desirable feature of stability across outcome variables quickly breaks down as we move to greater levels of disaggregation.

We now turn to Tables 4B and 4C, which contain the same information for the two more disaggregated versions of GII. These tables report the same information as Table 4A. The only difference however is that we have highlighted the top 11 out of 45 explanatory variables for each outcome variable in Table 4B, and the top 76 out of 303 variables in Table 4C, i.e. we have highlighted the top 25 percent of explanatory variables for each outcome variable in each of the three tables. We do this in order to keep our results on the stability of important explanatory variables across outcomes as comparable as possible as we move to higher levels of disaggregation. Comparing tables 4A‐4C we see some important similarities and differences. As before we find that posterior probabilities are quite highly concentrated across a fairly small number of models, and also across a fairly small number of variables. We also find only slightly larger posterior mean model sizes, ranging from 3 to 6 for the different outcomes at the GII‐45 level of disaggregation, and 4 to 12 for GII‐303 level. This tells us that, for a given outcome variable, the BMA procedure discriminates reasonably sharply among the many potential explanatory variables and isolates a fairly small number of variables that are relatively more important in explaining that outcome.

The major difference however as we move to more and more disaggregated explanatory variables is that there is much less stability across outcomes in terms of which explanatory variables are identified as having large PIPs. In Table 4A for example we found that 2 out of 12, or 17 percent of

> 20 In particular the sum of the squared deviations between expected and observed proportions, normalized by expected proportions, will be a chi‐squared random variable with 6 degrees of freedom.

21
explanatory variables, appeared in the top 25 percent list for all seven indicators. In Table 4B the corresponding figure is just 1 out of 45 variables, or only 2.5 percent of explanatory variables. And in Table 4C just one out of 303, or 0.3 percent of variables, falls in the top 25 percent list for all seven outcome indicators. We also find that far fewer variables also consistently do _not_ appear in the top 25 percent of explanatory variables. In Table 4A for example 33 percent of variables (4 out of 12) appeared in the top 25 list for none of the outcome variables. In Table 4B and 4C the corresponding rates are 18 percent and 14 percent, respectively. More formally, the p‐value for the chi‐squared test of the null hypothesis that inclusion in the top 25 percent of variables by PIP is independent across outcome variables is 0.12 in Table 4B, and 0.99 in Table 4C, and so we do not reject the null hypothesis of independence at conventional levels in both cases.

Since we can disaggregate the GII data at three levels, we can also look at the extent to which variables that appear important at one level of disaggregation are also found to be important at other levels. To take a specific example, consider the DRI outcome variable. At the most aggregated GII‐12 level in Table 4A, we saw that PQElections had by far the strongest explanatory power, with a PIP of 0.90. However, the three subcomponents of PQElections in Table 4A have considerably lower explanatory power for the DRI measure of corruption (ranking 8<sup>th</sup> , 9<sup>th</sup> , and 35<sup>th</sup> out of 45 variables). By contrast, by far the most significant variable for DRI at the GII‐45 level of disaggregation is LQ3.4BudgetProcess, with a PIP of 0.9. However, the corresponding aggregate at the GII‐12 level of aggregation to which this variable belongs, LQ3GovtAccountability, has only the third‐highest PIP of 0.14. We can see a similar pattern of instability at lower levels of aggregation. Continuing with DRI as an example, when we move to the GII‐303 level of disaggregation, we find that just 5 of the 23 individual variables making up PQElections fall in the top 25 percent of variables at this highest level of disaggregation.

More systematically, we perform the following simple test to document the degree of stability across levels of aggregation in terms of which variables are identified as important by the BMA procedure: we ask whether the probability that a variable is in the top 25 percent of all 303 fully disaggregated GII measures is significantly higher conditional on it belonging to a cluster of variables that was identified in the top 25 percent at a lower level of disaggregation. We perform this test using simple contingency tables that document whether a given variable is in the top 25 percent at both levels of disaggregation, neither level, or the one level but not the other. Across all 7 outcome variables, and 22
comparing the GII‐303 level with the GII‐45 level, and the GII303 level with the GII‐12 level of disaggregation, we never fail to reject the null hypothesis that falling in the top 25 percent of explanatory variables at the highest level of disaggregation is statistically independent of falling in the top 25 percent at lower levels of disaggregation. This suggests that there is little evidence that variables that are identified as important at one level of disaggregation will be identified as important determinants of outcomes at another level of disaggregation.

Tables 5A and 5B report the results we obtain from applying the BMA methodology to the Doing Business dataset. The results are qualitatively very similar to those we obtain for GII. As with GII, we find that posterior probabilities are fairly strongly concentrated on a relatively small number of models, and a fairly small set of variables is identified as having large posterior inclusion probabilities. So as not to overinterpret the precise magnitudes of these probabilities, we again highlight the top 25 percent of explanatory variables according to their posterior inclusion probabilities.

Most interestingly for our purposes, we see that our main findings on stability across outcomes, and stability across levels of disaggregation, hold in this dataset as well. In Table 5A, at the least‐ disaggregated DB‐10 level, we find a good deal of stability across outcomes. Two out of 10 DB variables have consistently high PIPs across the seven outcome variables (the CrossingBorders variable which is in the top three PIPs for all seven outcome variables, and the GettingCredit variable, which is in the top three PIPS for five of seven outcomes). At the other extreme, 6 out of 10 variables appear in the list of top three regressors for just 0 or 1 outcomes, suggesting a great deal of stability across outcomes in terms of variables that appear to matter much for outcomes. More formally, when we test the null hypothesis that appearing in the top‐three list is independent across outcomes we very strongly reject the null of independence. This very attractive feature of the results breaks down however when we move to the DB‐41 level of disaggregation. Here for example we find that just 3 out of 41 indicators appear in the top 25 percent of regressors for 5 or more outcomes, and the chi‐squared test of the null hypothesis of independence across outcomes delivers a p‐value of 0.99.

Looking across levels of aggregation in the DB dataset, we also find the same feature of instability that we saw in the GII dataset. In particular, an individual indicator at the DB‐41 level of disaggregation is not more likely to be included in the top 25 percent of explanatory variables if the higher‐level aggregate to which it belongs is in the top 25 percent of variables at that level of disaggregation. To make this concrete, consider for example the DB‐10 indicator “GettingCredit” which 23
ranks among the top three explanatory variables for 5 of 7 outcomes (DRI, EIU, GAD, WMO, and CPIA). This indicator consists of four sub‐indicators at the DB‐41 level of disaggregation. But these sub‐ indicators turn up in the top 25 percent of explanatory variables at the DB‐41 level for only 6 of 20 possible cases for these five outcomes. This rate of 0.30 is only marginally higher than the unconditional probability of appearing in the top‐25 percent. More systematically, as we did with GII we test the null hypothesis that falling in the top 25 percent of explanatory variables at the highest level of disaggregation is statistically independent of falling in the top 25 percent at lower levels of disaggregation. Across all 7 outcome variables we never fail to reject this null hypothesis (the lowest p‐ value for the null of independence that we observe is 0.64). This indicates that the DB dataset has the same problem of instability across levels of disaggregation that we saw with GII.

# **5.  Robustness and Caveats**

We began this paper with the observation that realizing the promise of disaggregated governance indicators to identify reform priorities requires knowledge of the partial effects of these potentially very many variables on outcomes that policymakers might reasonably care about. For a given outcome and set of indicators at a given level of disaggregation, we find that the BMA methodology used here yields useful results by identifying a fairly small subset of indicators that are robustly partially correlated with the outcomes of interest. However, we have also documented that these ostensibly sensible results for a given outcome and level of disaggregation break down when we consider other closely‐related outcomes and greater levels of disaggregation. Disaggregated indicators that are important predictors of one outcome are not systematically also important predictors of other very similar outcomes. Recall for example that the median pairwise correlation across our 7 measures of corruption was high at 0.61 for 7 outcome variables used with GII, and 0.68 for 7 outcome variables used with DB, yet we could not reject the null hypothesis that and indicator being included in the top‐25 percent of explanatory variables ranked by importance for one outcome was independent of appearing in the same top‐25 list for other outcomes. And looking across levels of disaggregation, we found no evidence that being in the top‐25 list for a given outcome at one level of disaggregation was a good predictor of being in the top‐25 list at another level of disaggregation.

One might first reasonably wonder how robust this key instability finding is. For example one might wonder whether this strong instability is driven by the somewhat arbitrary cut‐off value that we have been using to identify the set of important explanatory variables. After all, in Tables 4A‐4C, many 24
of the explanatory variables have very low PIPs that are practically indistinguishable from zero or from each other. To the extent that the list of top‐25 percent of explanatory variables includes such variables with very low PIPs, it would not be very surprising to find a lot of instability across outcome variables. We explore this possibility in Table 6, where we consider how our results on instability across outcomes vary for different thresholds used to identify the set of important regressors. The first row of Table 6 contains the benchmark results discussed above: we identify regressors as “important” if they fall in the top quartile of indicators as ranked by their PIPs for each outcome variable. We then report the p‐ values discussed above for the null hypothesis that the event of inclusion in the set of “important” regressors is independent across outcomes, and saw that we could not reject the null only for the least‐ disaggregated GII‐12 and DB‐10 levels of disaggregation. The remaining rows of the table consider other possible rules for identifying “important” regressors. If for example we focus only on the top‐10 percent of regressors we see the same pattern of instability. And if we look at just the top five, three, or even just one indicator as ranked by PIPs, we still see in nearly all cases that we cannot reject the null hypothesis of independence for the more disaggregated sets of indicators (GII‐45, GII‐303, and DB‐41).

We note also that our instability findings are robust to the choice of the key prior parameter g. Recall from the discussion in Section 3 that g plays a key role in determining how the posterior probability mass is distributed across models. When g is chosen to be small as we do, posterior model probabilities are highly sensitive to differences in model fit as measured by R‐squared. As a result, posterior model probabilities tend to be very highly concentrated on a few models with good fit, and the posterior inclusion probabilities of individual right‐hand‐side variables are very concentrated on the few variables included in those models. Small differences in R‐squareds as we move to different choices of outcome variables can then lead to large changes in PIPs. One might therefore reasonably wonder whether our findings on instability are driven by this important sensitivity of BMA to the choice of prior parameter g. It turns out however that this is not the case. First, recall that we have focused throughout on the _ordering_ of right‐hand‐side variables by their PIPs, rather than the magnitudes of the PIPs themselves which are very sensitive to the choice of g. Second and more important, we have also reproduced the BMA analysis for all of the datasets considered here, but using a hyperprior distribution for the prior parameter g, as advocated by Liang et. al. (2008) and Feldkircher and Zeugner (2009). This effectively averages all inferences across a range of alternative values of g, and so smooths out the effect of g on inferences. When we do this, we find that the rank ordering of variables by their PIPs is 25
nearly identical, and so we are confident that our instability finding is not driven by the particular choice of the prior parameter g.

Another possible objection to the instability finding is that it reflects multicollinearity problems in the individual models considered by BMA. As is well‐known, one consequence of having nearly collinear regressors in finite samples is that parameter estimates are highly‐sensitive to small changes in model specification. Moreover, individual slope coefficients would themselves be very imprecisely estimated. Together this would potentially be reflected in large posterior variances of the slope parameters of interest. However it is less clear that this will matter for posterior model probabilities and the derived posterior inclusion probabilities, since these reflect only differences across models in model size and goodness‐of‐fit. Indeed, the strong tradeoff between model size and R‐squared due to our choice of a small value for the prior parameter g helps to ensure that models with near‐collinear regressors are assigned low posterior probabilities (since adding a near‐collinear regressor to a model will result in only a very small improvement in R‐squared). In addition, we have also computed (but not reported) posterior standard deviations conditional upon inclusion for the GII‐12 and GII‐45 levels of disaggregation. If collinearity were important we would expect to find large conditional standard deviations for the slope coefficients reflecting the low information content of collinear regressors. Yet we find that these posterior standard deviations are not especially large. At the GII‐45 level of disaggregation, we find that roughly the top half of all variables as ranked by their PIPs have ratios of the posterior mean to the posterior standard deviation in excess of two, suggesting that conventional levels of statistical significance are attained by these variables.

A final very pedestrian potential explanation for our instability result across outcomes is that it might simply reflect differences across outcomes in the set of countries included in the analysis. Not all of the outcome variables are available for all countries, and in order to use as much information as possible we have performed our analysis using the largest available set of countries for each choice of y variable. However, if the set of included countries changes systematically from one outcome to the next in terms of the relevant determinants of outcomes, then this would contribute to our instability finding (as noted by Feldkircher and Zeugner (2010) in their comment on Ciccone and Jarocinski (forthcoming)). To investigate this possibility we repeated the analysis of the DB‐41 dataset, but restricting attention to 26
the (much smaller) set of 70 countries for which all seven outcome variables are available.<sup>21</sup> In this dataset, we find the same pattern of strong instability across outcomes. The p‐values for the null hypothesis that the event of inclusion in the set of "important" regressors is indepdendent across outcomes is 0.96. This suggests to us that our instability across outcomes finding is not driven by such compositional effects.<sup>22</sup> Based on all this we argue that our instability finding is indeed a robust feature of the data: disaggregated indicators that are significant determinants of one outcome are on average not significant determinants of other very similar outcomes, and for a given outcome variable, indicators that are significant at one level of disaggregation are on average not significant at other levels of disaggregation. Given that this finding is so robust, we now ask how it should be interpreted. One possibility is simply that the world is in fact complicated, in the sense that different outcome variables truly do respond very differently to the many indicators in DB and GII. For example, it might truly be the case that the notions of transparency and accountability reflected in the World Bank’s CPIA assessments do indeed require a very different set of policy interventions from those required to improve the notions of clean government and absence of corruption monitored by commercial risk rating agencies such as DRI or EIU. However, this interpretation is hard to reconcile with the very strong correlations across outcome variables, which limit how different can be the notions of corruption, or regulatory quality, that these different outcome variables are measuring.

Another interpretation is that all of the disaggregated indicators, and all of the outcome variables, are just imperfect proxies for some broader notion of good governance. Under this interpretation, the most disaggregated indicators could be thought of as the noisiest proxies for governance, and averaging them together at successively higher levels of aggregation smooths out measurement error in the individual indicators, resulting in better and better proxies for governance at higher levels of aggregation. This interpretation can account for the fact that there is less instability

> 21    We do not attempt a similar exercise for the GII data simply because the resulting dataset would contain only 44 countries.

> 22 A median of pairwise correlation coefficients across seven outcome variables for these 70 countries is reduced to 0.59 (from 0.68). One may argue that this might contribute to instable bma results across seven outcome variables. When comparing bma outputs across DRI, PRS, WMO and CPIA, four of these are correlated as much as 7 outcome variables are in our analysis above, the similar instability across these 4 outcome variable emerges. More specifically, only three (Tax Total, EmployFiringCost, and BorderTimeImp) out of 41 variables appear in top 25 percent of indicators for these four outcome variables. This suggests that the slightly smaller correlation across seven y variables under 70 sample countries is not driving the “instability” outcome.

27
across outcomes at higher levels of aggregation, since the set of candidate regressors has less measurement error. However this interpretation runs strongly counter to the spirit of collecting disaggregated indicators, which is that each of these variables is capturing an important and distinct dimension of policy interventions that matter for improving governance or the regulatory environment.

Unfortunately we cannot distinguish between these two interpretations, and so the best that we can do is to recommend a healthy dose of caution in using very disaggregated indicators to inform policy advice in the areas of governance and regulatory quality.

# **6.  Conclusions**

Policymakers interested in using highly‐disaggregated indicators of governance and the investment climate to identify reform priorities would like to know what the impacts of reforms in specific areas will be on outcomes that they care about. The results of this paper suggest that the data on indicators and outcomes we have do not allow us to sharply discriminate between indicators that do and do not matter for outcomes, at least using the available cross‐country variation. This should be worrisome for a policymaker interested in using these indicators to identify reform priorities. To give a very stark example, the results here suggest that more than 85 percent of the disaggregated GII indicators (260 out of 303) are important partial correlates of at least one of the seven very closely‐ related outcome measures of corruption we have considered. While this is a tribute to the likely relevance of the overall GII exercise, it does little to narrow down the set of measures a policymaker might want to target for reforms.

Beyond this, we note that there are likely to be even bigger obstacles to using such disaggregated indicators to identify reform priorities than the ones we have seen here. In this paper we have relied on the very simple tool of linear OLS regressions as a means of identifying the effects of disaggregated indicators on outcomes. As we have noted, the standard exogeneity assumptions required to justify such an empirical approach are unlikely to hold in reality. However it is very unclear how one might find instruments or other sources of exogenous variation in the very many different dimensions of governance and regulatory policy measured by GII and DB. The assumption of linearity is also very restrictive: one could easily imagine that improvements in various combinations of the individual indicators are required to improve outcomes, implying that we need to consider not only the 28
2<sup>K</sup> potential combinations of regressors, but the vastly more possible combinations of interactions between them.

In concluding, we want to be clear that we do not think that the information painstakingly gathered in the many individual indicators that comprise the GII and DB datasets is irrelevant. To the contrary, most if not all of them measure things that plausibly are intrinsically good on their own (it is hard to imagine why it might be a _bad_ idea to simplify business entry regulation from current levels in most countries, for example), and it also seems intuitive that they matter for outcomes. Rather, we simply note that it seems extremely difficult to quantify the partial effects of these many indicators on relevant outcomes. And so it is very difficult to use these indicators as a recipe or a roadmap to reforms in the real world where policymakers must choose where to spend their political capital.

29