metadata: authors: 0: author: Kaufmann, Daniel
1: author: Kraay, Aart
2: author: Zoido-Lobaton, Pabl
docna: 0: docna: Aggregating governance indicators
docty: Policy Research Working Paper
subtopic: Consumption,Fiscal & Monetary Policy,Judicial System Reform,National Governance,Government Policies,Youth and Governance,Governance Indicators
teratopic: Education
abstracts: cdata!: In recent years the growing interest of
            academics and policymakers in governance has been reflected
            in the proliferation of cross-country indices measuring
            various aspects of governance. The authors explain how a
            simple variant of an unobserved components model can be used
            to combine the information from these different sources into
            aggregate governance indicators. The main advantage of this
            method us that it allows quantification of the precision of
            both individual sources of governance data and
            country-specific aggregate governance indicators. The
            authors illustrate the methodology by constructing aggregate
            indicators of bureaucratic quality, rule of law, and graft
            for a sample of 160 countries. Although these aggregate
            governance indicators are more informative about the level
            of governance than any single indicator, the standard errors
            associated with estimates of governance are still large
            relative to the units in which governance is measured. In
            light of these margins of error, it is misleading to offer
            very precise rankings of countries according to their level
            of governance: small differences in country rankings are
            unlikely to be statistically - let alone practically -
            significant. Nevertheless, these aggregate governance
            indicators are useful because they allow countries to be
            sorted into broad groupings according to levels of
            governance, and they can be used to study the causes and
            consequences of governance in a much larger sample of
            countries than previously used (see for example the
            companion paper by the authors, "Governance
            matters", Policy Research Working Paper no. 2196).
repnme: repnme: Aggregating governance indicators
display_title: Aggregating governance indicators
listing_relative_url: /research/1999/10/440006/aggregating-governance-indicators
topicv3: Justice m1326106 840,Macroeconomic and Structural Policies
            m1327380 439,Governance Indicators m1327488 834
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Education
origu: Development Research Group (DECRG)
no_of_pages: 48
model_extractions: POLICY RESEARCH WORKING PAPER 2195

# **Summary findings**

In recent years the growing interest of academics and policymakers in governance has been reflected in the proliferation of cross-country indices measuring various aspects of governance.

Kaufmann, Kraay, and Zoido-Lobat6n explain how a simple variant of an unobserved components model can be used to combine the information from these different sources into aggregate governance indicators. The main advantage of this method is that it allows quantification of the precision of both individual sources of governance data and country-specific aggregate governance indicators.

Kaufmann, Kraay, and Zoido-Lobat6n illustrate the methodology by constructing aggregate indicators of bureaucratic quality, rule of law, and graft for a sample of 160 countries. Although these aggregate governance indicators are more informative about the level of governance than any single indicator, the standard errors associated with estimates of governance are still large relative to the units in which governance is measured. In light of these margins of error, it is misleading to offer very precise rankings of countries according to their level of governance: small differences in country rankings are - - unlikely to be statistically let alone practically significant. Nevertheless, these aggregate governance indicators are useful because they allow countries to be sorted into broad groupings according to levels of governance, and they can be used to study the causes and consequences of governance in a much larger sample of countries than previously used (see for example the companion paper by Kaufmann, Kraay, and ZoidoLobat6n, "Governance Matters," Policy Research Working Paper 2196).

- This paper a joint product of Macroeconomics and Growth, Development Research Group; and Governance, Regulation, and Finance, World Bank Institute -is part of a larger effort in the Bank to study the causes and consequences of governance for development. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Diane Bouvet, room G2-136, telephone 202-473-5818, fax 202-334-8350, Internet address dbouvet@worldbank.org. Policy Research Working Papers are also posted on the Web at http:// www.worldbank.org/html/dec/Publications/Workpapers/home.html. The authors may be contacted at dkaufmann@worldbank.org, akraay@worldbank.org, or pzoidolobaton@worldbank.org. October 1999. (39 pages)

_The_ **_Policy Research Working Paper Series disseminates the findings_** _of_ **_work in progress_** _to encourage the exchange of ideas abot_ **_development issues. An objective of the series is to get the findings out quickly,_** _even if the presentationts are less than fully polisbed. The_ **_papers carry the names of tbe authors and should be cited accordingly._** _The findings, interpretations, and conclusions expressed in this paper_ **_are entirely those of the authors. They do not necessarily represent_** _the_ **_viewv_** _of the Worid P,ank, its Execu tive_ **_Directors, or_** _the countries they represent._ Produced by the Policy Research Dissemination Cen1ter
# **Aggregating Governance Indicators**

Daniel Kaufmann Aart Kraay Flablo Zoido-Lobat6n The World Bank

**Abstract:** In recent years, the growing interest of academics and policymakers in govemance has been reflected in the proliferation cross-country indices measuring various aspects of governance. In this paper we explain how a simple variant of an unobserved components model can be used to combine the information from these different sources into aggregate govemance indicators. The main advantage of this method is that it allows us to quantify the precision of the both individual sources of governance data as well as the aggregate governance indicators. We illustrate the methodology by constructing aggregate indicators of bureaucratic quality, rule of law, and graft, for a large sample of 160 countries. Although these aggregate governance indicators are more informative atbou **t** he level of governance than any individual indicator, the standard errors associated with estimates of govemance are still large relative to the units in which governance is measured.

The World Bank, 1818 H Street N.W., Washington, D.C. 20433. (dkaufmann(afworldbank. ora, akraavyworldbank.orq, ozoidolobatonAworldbank. ora). The views expressed in this paper are the authors' alone, and in no way reflect those of the World Bank, its Executive Directors, or the countries they represent. We would like to thank without implication Craig Burnside, Brad Efron, Eduardo Engel, Gil Mehrez, Jakob Svensson, Scott Wallsten, Shang-jin Wei, and seminar participants at the World Bank, UNDP and Latin American Econometric Society Meetings for helpful comments.
# 1. Introduction

In recent years, the growing interest of academics and policymakers in the extent, causes and consequences of governance and misgovemance has been reflected in the proliferation of cross-country indicators of various aspects of governance. In an accompanying paper (Kaufmann, Kraay, Zoido-Lobat6n (1999)) we present a large database compiling several hundred cross-country indicators of various aspects of governance, produced by thirteen clifferent organizations, and covering 178 countries. These indicators report subjective perceptions on a wide range of issues relating to govemance, ranging from the extent to which corruption in the political system affects foreign investment, to the efficiency of public services delivery, to the likelihood that citizens of a country resort to extrajudicial means to settle disputes.

In this paper, we take the view that many of these indicators serve as imperfect proxies for one of a much smaller number of fundamental concepts of governance. Given this view, there are considerable benefits from combining related indicators into a small number of aggregate governance indicators. First, the aggregate indicators span a much larger set of countries than any individual source, permitting comparisons of govemance across a broad set of countries. Second, aggregate indicators can provide more precise measures of governance than individual indicators. Third, it is possible to construct quantitative measures of the precision of both the aggregate govemance indicators and their components, allowing formal testing of hypotheses regarding crosscountry differences in governance We realize these benefits by constructing aggregate governance indicators using an unobserved components model. This model expresses the observed data as a linear function of unobserved governance plus a disturbance term capturing perception errors and/or sampling variation in each indicator. The main advantage of this method is that it allows us to obtain estimates of the variance of this disturbance term for each indicator. These can be interpreted as a measure of how informative each indicator is about the broader concept of governance it measures. We then compute the mean of the conditional distribution of govemance given the observed data for each country as a natural point estimate of the level of governance in that country. Similarly, the variance 1
of this conditional distribution provides a natural estimate of the precision of this aggregate governance measure for each country.

We illustrate our approach with reference to three fundamental aspects of governance: rule of law, government effectiveness, and graft. We group 31 indicators constructed in 1997 and/or 1998 into three clusters corresponding to these three concepts of governance, and compute aggregate indicators spanning 166, 156 and 155 countries respectively. In our companion paper documenting the governance database, we construct similar indices for several other aspects of governance.

Although the unobserved components methodology we use is quite standard, we find its application to the construction of composite govemance indicators interesting.' One of our major findings is that the aggregate governance indicators we construct are rather imprecise, despite the high correlations observed between various sources of governance data. In particular, a 90% confidence interval around the point estimate of governance for a typical country spans almost the entire interquartile range of the distribution of estimated govemance. This implies that although it is possible to robustly identify twenty or so countries with the best and worst govemance in the world, it is much more difficult to identify statistically significant differences in govemance among the majority of countries.

Our results are based on three key assumptions: (1) that the measurement errors in individual indicators of govemance are uncorrelated across indicators; (2) that the relationship between unobserved govemance and observed indicators is linear, and (3) that the distribution of unobserved governance across countries is normal. Relaxing the first assumption is difficult to do in practice, simply because without this assumption we cannot determine whether the correlation of observed scores across indicators is merely due to correlated perception errors or whether it reflects the common concept of governance being measured. However, under the likely altemative that perception errors are correlated across sources, the measures of precision we report will be biased downwards. As a result, the standard errors we report should be interpreted as a lower bound on the precision of aggregate govemance indicators. We consider the 1Unobserved components models were pioneered in economics by Goldberger (1972), and the closelyrelated hierarchical and empirical Bayes models in statistics by Efron and Morris (1971, 1972).

2
consequences of relaxing the second assumption by proposing a method which simply aggregates the ordinal information across indicators. Although this has the advantage of simplicity and does not require assumptions of linearity, it is also much less precise than the unobserved components method since it discards the cardinal information in the data. The third assumption of a normal distribution for unobserved governance implies that our estimates of governance will be clustered around the mean of this distribution. This raises the possibility that the! difficulty in distinguishing between countries is in part driven by this assumption. We therefore explore the robustness of our results by considering altematives to this assumption, and find that our conclusions are materially unaffected by our assumptions on the shape of the distribution of unobserved govemance.

These findings have several implications for policy and empirical research on the causes and consequences of governance for economic development. At a basic level, the finding that governance is imprecisely measured should warn against taking too seriously the exact point estimates of govemance, as well as country rankings based on these estimates. At best, it is possible to sort countries into broad categories according to their levels of governance, and even then there is considerable uncertainty regarding the category to which many countries should be assigned. To emphasize this point, we avoid discussions of specific countries in this paper. Second, since available indicators of governance are noisy measures of "true" govemance, empirical work which uses these indicators as explanatory variables may well underestimate the impact of -governance due to the usual attienuation bias caused by badly-measured right-hand side variables. Since our methodology allows us to quantify the measurement errors in these 2 variables, it is possible to obtain rough measures of the extent of this attenuation bias. Finally, our results suggest that if we want to more precisely differentiate among countries according to their leveil of governance, we need to improve the quality and quantity of data gathered on governance.

The remainder of this paper proceeds as follows. In Section 2, we motivate the empirical work which follows by describing the indicators of governance we use to 2 In our companion paper we explore this idea in more detail, using cross-count **r** egressions y ofper capita income on various governance measures instrumenting for govemance using measures of the linguistic' - composition ofthe population in the spirit of Hall and Jones (1999).

3
illustrate our ideas. In Section 3, we lay out and implemen **t** he unobserved components framework for estimating governance, and present the main results in Section 4. In Section 5, we discuss the consequences of relaxing several of the assumptions underlying the model. We conclude with a discussion of the implications of our findings for research and policy advice regarding governance.

4
# 2. **Indicators of Governance**

In this paper, we use data frorn 31 different indicators of governance constructed in 1997 and/or 1998. These indicators are drawn from 13 different sources and are grouped into three clusters corresponding to rule of law, government effectiveness, and graft. The key features of these indicators are summarized in Table 1, and a detailed description of the sources and variables can be found in Kaufmann, Kraay and ZoidoLobat6n (1999). In the first two columns of Table I we identify each source of governance data by abbreviation and by name. In the next three columns, we report the source of information for each measure (surveys of residents or polls of experts), the country coverage, and a measure of the extent to which the sample of countries covered by each indicator is representative of the population of countries in the world. In the remaining columns we report the specific concepts measured by each source in each of the three clusters. 3 A quick look at Table 1 shows that these indicators differ along several dimensions. First, even within clusters there is considerable variation in the particular concept measured by each indicator. For example, questions about graft range from the incidence of "improper practices' (WCY) to the likelihood that additional payments are required to "get things done" (WDR). Similarly, questions regarding the rule of law range from whether citizens can successfully sue the state to whether citizens are likely to resolve disputes extra-judicially. Despite this heterogeneity, we take the view that within each cluster, each of these concepts is an imperfect indicator of the corresponding broader concept of governance.

The second respect in which these indicators differ is in the nature of the respondents who provide the information. Slightly less than half of the indicators are surveys of businesspeople and/or residents of a country, while the remaining indicators are polls of experts who rate a set of countries according to various criteria. As we discuss in more detail later in the paper, this difference between these two types of

> 3 For a number of these sources, we use the average of several questions relating to the corresponding core concepts of govemance. As we discuss subsequently, we are reluctan **t** o include individual questions from a single source separately in our analysis, as the necessary assumption that measurement errors are uncorrelated across indicators is much more difficult to support for the case of multiple questions from a single source.

5
indicators has implications for how we interpret the error terms in the relationship between observed indicator scores and the underlying concepts of governance.

The third respect in which these indicators differ is in the sample of countries they cover. A number of indicators cover a very large and broad sample of developed and developing countries (EIU, DRI, HFWSJ, PRS and WDR), while others cover very narrowly-focused samples of countries (PERC for Asia, CEER and FHNT for transition economies). Some indicators cover primarily developed countries but also include major developing countries (WCR, GALLUP, BERI). This difference between indicators is perhaps the most important for the empirical work which follows. There is by now considerable evidence that govemance on average tends to be better in richer countries. This implies that the distribution of governance is likely to be very different in indicators which cover sets of countries with different average income levels. These differences need to be taken into account when placing the observed data from various indicators into common units and combining them into aggregate govemance indicators.

In order to distinguish between indicators in this dimension, we construct a simple coverage index which measures differences between the distribution of countries across income and regional classifications and the distribution of all countries in the world across these categories. In particular, we divide the world into a two-way classification by region and income, following the World Bank's 1998 World Development Report. For each of the sources of govemance data, we report one-half of the sum of absolute -deviations between the share of countries in each of the 45 region/income categories in that source and in the world as a whole. By construction, this measure ranges from zero to one, with low values indicating more representative indicators. We report this number in the fifth column of Table 1. The five indicators covering the largest number of countries (DRI, EIU, HF, PRS and WDR) are substantially more representative according to this measure than the others, with a value of the coverage index of less than 0.25. In our subsequent empirical work we will refer to these as representative indicators, and the remainder as non-representative indicators.

Finally, we note that each of these sources of governance data uses different units to measure governance. Most polls of experts report discrete categorical responses (e.g. the prevalence of corruption on an integer scale from one to four), while 6
for most surveys of citizens or entrepreneurs we have the mean response across respondents of discrete categorical scores. We re-orient data from each source so that higher values correspond to better outcomes (i.e. stronger rule of law, more effective govemment, and less graft). In addition, we rescale each indicator by subtracting the minimum possible score and dividing by the difference between the maximum and minimum scores, so that each indicator is on a possible scale from zero to one. Since we rescale each indicator using the maximum and minimum possible scores (rather than the maximum and minimum actual scores in the sample of countries covered by each indicator), this is nothing more than a convenient choice of units.

In Table 2, we report the pairwise correlations among indicators within each of the three governance clusters. The great majority of these are positive and substantial, frequently greater than 0.6. In the empirical work which follows, we will interpret these large correlations within clusters as reflecting the common component of govemance in these indicators. It is interesting to note that despite the strong pairwise correlations among these indicators, and despite the favourable interpretation that these correlations reflect the common component of governance rather than correlated perception errors, we nevertheless find that governance is not very precisely measured. We provide some intuitions for this in the following section.

7
# **3. Estimating Governance**

In this section we interpret the data as being generated by an unobserved components or multiple-indicator model in which the observed data on governance can be expressed as a linear function of unobserved governance plus a random error term. We review the well-known features of this model, and propose a simple extension which delivers consistent parameter estimates for representative as well as non-representative indicators. We then describe how the parameters of this model can be estimated and can be used to construct estimates of each of the three aspects of govemance in each country.

# **The Model**

Our data consists of clusters of indicators of three aspects of governance - rule of law, government effectiveness, and graft. Let **go)** denote an unobserved index of one of these three aspects of governance in country j, for example, graft. The observed data on graft consists of a cluster of k=1,...,K indicators, each one providing a numerical rating of some aspect of graft in each of the j=1,..,J(k) countries covered by that indicator. We assume that we can write the observed score of country j on indicator k, y(j,k), as a linear function of unobserved governance, go), and a disturbance term, eo,k), as follows:

# (1) y(j,k) = a(k) + 1(k). (g(j) + F(j,k))

where a(k) and ,B(k) are unknown parameters which map unobserved governance **go)** into the observed data yo,k). We assume that **go)** is a random variable with mean zero and variance one. Our objective is to summarize our knowledge about **go)** for each country j using the distribution of go) conditional on the observed data yo,k), k=1,...,Ko) for country j. The mean of this conditional distribution provides a natural estimate of the level of govemance in country j, and the variance of this conditional distribution is a natural measure of the precision of this indicator of governance. The assumption of a zero mean and unit variance for governance is an innocuous choice of units required to identify the parameters a(k) and ,B(k). Since we will allow the variance of the error term 8
to vary across indicators k, the fact that P(k) multiplies the error term is an innocuous rescaling which slightly simplifies some of the expressions which follow.

We use this unobserved components model, which treats unobserved govemance as a random variable rather than as a fixed parameter to be estimated, for a pragmatic reason. We will shortly also assume that the variance of the disturbance term e(j,k) may differ across indicators. In this case, we cannot treat the ga)s as fixed parameters to be estimated for each country, since individual effects are not identified in a fixed effects model with heteroskedastic disturbances. 4 Moreover, it should be clear from Equation (1) that naive aggregates such as a simple average of rescaled indicators for each country will not result in sensible estimates of govemance, as long as the parameters a(k) and ,3(k) differ across indicators and different countries appear in different sets of indicators. It is also not possible to remove the dependence of the observed data on these nuisance! parameters by standardizing (i.e. by removing the sample mean from each indicator, and dividing by the sample standard deviation). This is because if indicator k is non-representative, the sample mean will reflect not only a(k), but also the mean of go) in the sample of countries covered by indicator k. Even if an indicator is representative in the sense that the standard deviation of unobserved governance is equal to one in the sample of countries it covers, the sample standard deviation of observed scores will reflect not only ,8(k), but also the standard deviation of the disturbances.

The disturbance term s(j,k) captures two sources of uncertainty in the relationship between true governance and the observed indicators. First, the particular aspect of governance covered by indicator k is imperfectly measured in each country, reflecting either perception errors on the part of experts (in the case of polls of experts), or sampling variation (in the case of surveys of citizens or entrepreneurs). Second, the relationship between the particular concept measured by indicator k and the corresponding broader aspect olF governance may be imperfect. For example, even if the particular aspect of graft covered by some indicator k, (such as the prevalence of 4 To see this, consider the special case! where a(k)=O and 13(k)=1 for all indicators. We can make the likelihood function of the observe **d** ata arbitrarily large simply by estimatin **g** o) as the observed score on a particular indicator, for example ga)=y(j.K), for every country j, and setting cr(K)=0. Kiefer (1980) provides a detailed discussion of this point.

9
"improper practices") is perfectly measured, it may nevertheless be a noisy indicator of graft if there are differences across countries in what "improper practices" are considered to be.

We assume that the disturbance term has zero mean, E[Eo,k)1=O; has the same variance across countries within a given indicator but a different variance across indicators, E[gj,k)<sup>2</sup> ]=ao(k)<sup>2</sup> ; and is independent across indicators and countries, E[E(,k)gj',k')]=O if j#j' or k•k'. The variance of the error term can be interpreted as a measure of how informative indicator k is about go), and is likely to vary across indicators. The assumption that the errors are independent across indicators is a strong one, but unfortunately one that is also difficult to relax. Intuitively, without this assumption we cannot identify whether the correlation of scores between two indicators is due to their common component of governance go), or whether it simply reflects the correlation of errors. In contrast, this identifying assumption maintains that all of the correlation of scores across indicators is attributable to their common estimate of govemance. We will consider the consequences of relaxing this assumption in the following section. For now, we simply note that this identifying assumption corresponds to a "best case" scenario regarding the precision of govemance aggregates, since it assumes that each indicator provides independent information on a particular aspect of governance. As a result, we are if anything likely to overstate the precision with which governance is measured.

The parameters c(k) and P(k) map unobserved govemance into the observed data. Although all of our indicators (after rescaling) are nominally in the same units and are measured on a scale from zero to one, there are nevertheless three reasons why these parameters may differ across indicators. First, not all indicators use the entire range of possible scores. For example, although WDR measures perceptions of graft on a scale from one to six, the lowest observed score is only 2.36. This suggests that a(k) on this indicator may be greater than that of an indicator such as EIU which uses the full range of possible scores. Second, a given indicator might be 'easy" ("tough") relative to other indicators in the sense that it tends to overestimate (underestimate) a particular aspect of governance in countries where it is in fact low (high). This would be reflected in a relatively high (low) value of a(k) on that indicator. Third, consider a nonrepresentative indicator that covers a set of countries in which the average level of a 10
particular aspect of governance is better than in the world as a whole (e.g. BERI, which covers primarily developed countries). Suppose further that this source tends to score countries relative to each other, so that the worst (best) country in the sample receives the lowest (highest) possible score? of zero (one). This would be reflected in a relatively high value of ,8(k), since relatively small differences in true govemance are magnified into relatively large differences in observed scores.

## **The Conditional Distribution of Governance**

Our objective is to summarize our knowledge about governance in each country j by the distribution of governance conditional on the observed data in country j. This task is greatly simplified by assuming that both g() and the disturbances so,k) are jointly normally distributed. In this case, go) and y(,k), k=1,...,Ko) are jointly normal, and the conditional distribution of g() given the data is also normal, with mean and variance given by:.

# (3) VE[(g)ly(j)] = 1y- c

where y() is a Kj)xl vector which stacks the K(j) data points for country j, a is the corresponding K()xl vector of a(k)s, B and , are K&)xK() diagonal matrices with the corresponding ,B(k)s and _a,(k)_<sup>_2_</sup> _s_ on the diagonal, and t is a K()xl vector of ones. We refer to the conditional mean in (2) as the estimated value of that aspect of govemance in country j. With a slight abuse of terminology, we refer to an interval from the (o/2)t percentile to the (1-8/2)th percentile of the conditional distribution of **go)** as an o-percent "confidence interval" around this estimate, and we refer to the square root of the conditional variance in (2) as the "standard error' of this estimate.<sup>5</sup>

> 5 This framework has a distinctly Bayesian interpretation. The distribution of go) conditional on the observed data yj) can be viewed a posterior distrlibution, and the mean of this distribution as an estimator of g() would be justified as a point estimate of go) by a quadratic expected posterior loss function. Similarly, the "confidence interval' is analogous to a Bayesian highest posterior density interval.

11
These expressions have a very natural interpretation. If the parameters a(k), ,(k) and acn(k)2 were known, a sensible way to estimate go) would be to rescale the observed scores by subtracting a(k) and dividing by ,B(k), and then construct a weighted average of these re-scaled scores. In particular, let 9(j,k) 1<sup>k)</sup> ( - = **g(j)** + e(j, k) P(k) denote the rescaled value of y(,k). Then the conditional mean in (2) is a weighted average of these standardized scores for country j on each of the KG) indicators in which it appears, with weights corresponding to the inverse of the variance of the error term on each indicator, i.e.

available for that country, K(), and is increasing in the variance of the error term in each of these indicators, ue(k)<sup>2</sup> .

# **Estimating the Unknown Parameters**

In order to implement (2) and (3), we need to first estimate the unknown parameters a(k), P(k) and o,(k)<sup>2</sup> for every indicator k. For the set of representative indicators, we can use the assumption of normality of go) and so,k) to write down the likelihood function of the observed data. Provided that we have at least three such indicators, the model is identified and it is straighfforward to maximize this function with respect to the a(k)s, P(k)s, and cre(k)2S to obtain estimates of the unknown parameters for the representative indicators.6

> 6 Although maximum likelihood estimation of these parameters requires the assumption of normality, it is also possible to dispense with this assumption and apply a method of moments procedure. In the justidentified case of three indicators, these methods lead to identical parameter estimates. In the overidentified case of more than three indicators, these methods differ only in the weights applied to the various moment conditions, and in practice this makes little difference for the parameter estimates.

12
We cannot apply this method to non-representative indicators. To see why, consider the maximum-likelihood estimate of a(k), which unsurprisingly is the mean score across countries on indicator k. It is straightforward to see from Equation (1) that the expected value of the sample imean of scores on indicator k is a(k) + ,B(k) * g(k), where g(k) denotes the average level of governance in the sample of countries covered by indicator k. For representative indicators, our choice of units for governance normalizes g(k) = 0. However, for a non-representative indicator where the average level of govemance is different from the world as a whole, g(k) • 0 and the sample mean does not provide a consistent estimate of a(k).

We can nevertheless obtain consistent estimates of the unknown parameters by using the following simple argument. If go) were observable, we could estimate a(k), P3(k) and aj(k) for any indicator by regressing the observed scores y(,k) on **go).** Although **go)** is itself not observable, we do have an estimate of **go)** based on the representative indicators. In partilcular, let g*() denote the mean of **go)** conditioning only on the data from the representative indicators. We can decompose this conditional mean into observed governance go) plus its deviation from the mean u0), i.e. g*()=go)+uo) Since u0) is independent of go), we can view g*o) as measuring **go)** with error, i.e. as a classic errors-in-variables problem. It is well-known that OLS estimates of ,8(k) from a regression of y(,k) on g*() will produce downward-biased estimates due to the usual attenuation bias imparted by measurement error in g*(). In particular, the probability limit of the OLS slope coefficient is 1,(k) - V A WI * . Since the variance of uO) is simply the variance of the conditional mean of **go)** given in Equation (3), and since V[g*0)] is observable, we can correct the OLS coefficients for this attenuation bias to arrive at consistent estimates of the parameters of the non-representative indicators.7

> 7 An altemative approach to the problem of non-representative indicators would be to impute data for the missing observations (in the spirit of Rubin (1987)). We do not pursue this approach here simply because it is difficult to specify the key ingredient of the imputation process - the conditional distribution of the unobserved data given the observed data - in our application.

13
It is worth noting that this estimation method urewards conformity", in the sense that indicators that are highly correlated will have low estimated variances and hence will be perceived as more precise. Given our assumption that the disturbance terms are independent across indicators, it makes sense to treat highly correlated indicators in this way. If on the other hand indicators are correlated simply because their disturbances are correlated, this interpretation would be inappropriate. We take this issue up in more detail in Section 4, and argue that it will result in even less precise estimates of govemance than those we obtain here.

14
# **4. Results**

In this section we implemen **t** he unobserved components model laid out in the previous section for three concepts of governance: government effectiveness, rule of law, and graft. We first present estimates of governance and associated standard errors for each country, and then consider the consequences of these standard errors for identifying cross-country differences in govemance. We conclude with a simple example which relates the pairwise correlations observed among individual governance indicators directly to the measures of precision of the aggregate indicators.

# **Estimates of Governance**

Our main finding is that thoa available data do not permit very precise estimates of governance. We illustrate this point in Figure 1. In each of the three panels of Figure 1, we order countries on the horizontal axis by their estimate of governance, and on the vertical axis we plot the corresponding point estimate of governance, i.e. the conditional expectation of **go)** given the observed data for country j, and a 90-percent confidence interval around this point estimate, i.e. the 5t and 95t percentiles of the conditional distribution of governance for each country j. The size of these confidence intervals varies by country, reflecting the fact that different countries appear in different numbers of sources, and that different countries appear in different sets sources of differing precision. To provide a sense of the dispersion in the point estimates of governance, the three horizonta **l** ines in each graph delineate the quartiles of the distribution of the point estimates of governance for each cluster.

The most striking feature of Figure 1 is that these confidence intervals are large relative to the units in which governance is measured. For example, for a typical country the standard deviation of the conditional mean of rule of law or graft is around 0.3, so that a typical 90% confidence interval extends 0.5 above and below the point estimate of graft. In the case of government effectiveness, the standard deviation of the conditional mean is on average slightly larger and equal to 0.33, so that a 90% confidence interval extends 0.55 above and below the point estimate of rule of law. These confidence intervals are large in the sense that they are comparable in size to the entire interquartile range of the distribution of estimates of governance. Moreover, it should be noted that 15
these confidence intervals do not reflect the sampling variation in the point estimates of the unknown parameters a(k), ,8(k) and a.(k). If this uncertainty were also taken into account, the standard errors would be even larger.

The parameter estimates reported in Table 3 reveal some interesting differences across indicators. To interpret the estimates of the a(k)s and f3(k)s, note that our assumption of a standard normal distribution for governance implies that the vast majority of countries wili have govemance ranging from -2 to 2. Since the observed data range from zero to one, one might expect that a representative indicator would have a(k)=0.5 and ,(k)=0.25. Interestingly, there are significant departures from this benchmark. Several indicators (e.g. WDR) have estimated values of o(k) substantially lower than this benchmark, and higher values of a(k), indicating that they do not use the entire range of possible scores. There is also a great deal of variation in the estimates of the standard deviation of the errors on each individual indicator, c 6(k), suggesting that the precision with which individual sources measure govemance varies widely.

# **Assessing Cross-Country Differences in Governance**

An advantage of this methodology is that it permits straighfforward tests of hypotheses regarding cross-country differences in govemance. However, the large size of the confidence intervals documented in Figure 1 suggests that it will be difficult to find statistically significant differences in govemance between many pairs of countries. We illustrate this point with two simple exercises. Suppose first that for each of the three aspects of governance, we want to group countries into quartiles according to their level of govemance. A natural way to do this is to group countries according to their point estimates of govemance, i.e. according to the mean of the conditional distribution of govemance in each country. Moreover, a natural way to assess the confidence with which countries are assigned to quartiles is to consider the corresponding 90% confidence intervals shown in Figure 1. In particular, if the 90% confidence interval for a country falls entirely within a given quartile, the probability that this country in fact belongs in another quartile is less then 10%. For a small group of countries at each end of the distribution of governance, we can conclude with a great deal of confidence that these countries are in fact in the top and bottom quartiles. However, for the middle 16
quartiles the situation is much less clear, as very few countries' 90% confidence intervals lie entirely within a given quartile, for each of the three aspects of govemance.

Clearly, the number of countries we can assign to a particular quartile using this rule depends on the size of the confidence interval. If we instead consider shorter confidence intervals, such as 75% or 50% intervals, we can better discriminate among countries, albeit with lower confidence. We explore this possibility in Table 4, where we report the proportion of all countries for which an x% confidence interval falls entirely within the indicated quartile, for the three govemance aggregates in tum. We consider three possibilities, x=90%, x=75% and x=50%. At all significance levels, a substantial fraction of countries in the top and bottom quartiles can be clearly identified as belonging in these groups. As the size of the confidence interval declines, more and more countries can be significantly assigned to quartiles. Nevertheless, even at very low significance levels, only one-quarter to one-half of the countries in the middle two quartiles have confidence intervals lying entirely within their respective quartiles.

A related issue is the significance of pairwise differences in governance. In particular, for every pair of countries in which our point estimate of governance in country j is greater than in country j', we can investigate the hypothesis that country j in fact has better govemance by computing the probability that g(j)>g(').<sup>8</sup> For countries with similar point estimates oF govemance, this probability will be close to 0.5, while for countries far apart in the distribution of govemance, this probability will approach one. -To illustrate this point systematically, for each country j in the sample, we compute the probability that, conditional on the observed data for countries jand j', g(j)>g(j') for every comparator country j'. We then compute the proportion of comparator countries for which this probability is betwesen 5% and 95%. This is analogous to counting the number of comparator countiries for which a conventional test at the 10% significance level of the null hypothesis that govemance is the same in these two countries cannot be rejected. We summarize the results of these pairwise comparisons in Figure 2. We again order countries in ascending order according to their point estimates of governance on the horizontail axis, and we plot this proportion of comparator countries 8 Since the ga)ly() and g(')ly(') are jointly normal and independent by assumption, this calculation involves a straighfforward integration of the area under a bivariate normal probability density function.

17
as dark points on the vertical axis. We also repeat the exercise, but instead report the larger proportion of countries for which this probability is between 25% and 75%, which corresponds to a test at the 50% significance level. This proportion is shown as a light dot in Figure 2.

Not surprisingly, at the two ends of the distribution there are significant differences between the level of govemance in these countries and most other countries, especially at the 50% significance level. However, there is also a strong inverted U-shaped pattem in this graph, reflecting the fact that a large fraction of countries are clustered near the middle of the distribution of estimated govemance, and it is relatively difficult to distinguish among such countries. In particular, for the "typical" country around the middle of the distribution of govemance, govemance is not significantly different from nearly half of all other countries in the world, at conventional significance levels.

# **Intuitions**

Our finding that governance is imprecisely measured is somewhat surprising. After all, in Section 2 we documented that fact that the pairwise correlations among various governance indicators are substantial, and the identifying assumption of independent errors across indicators implies that the only source of this observed high correlation among indicators is the unobserved common component of governance. One might therefore easily conclude that govemance is quite well measured and that it is straightforward to distinguish among countries' govemance using this data. We now illustrate why this intuition is misleading, unless the correlations in the observed data are very high indeed.

As a specific example, suppose that there are only three representative indicators associated with a particular governance concept, i.e. K=3, and that the pairwise correlations among the observed scores are all equal to p. It is straightforward to show that in this case, the estimated variance of the residual will be a (k)<sup>2</sup> =P 18
for each of the surveys k=1,2,:3.9 Inserting this into Equation (3), the variance of the aggregate governance indicator based on this hypothetical data will be the same for each country and is equal to V[g(j)l y ( = 1 +2 . To give an idea of the magnitude of the corresponding 90% confidence intervals, we superimpose them on the hypothetical distribution of governance in the upper panel of Figure 3, for various values of p. As p increases, the confidence intervals become shorter. However, for the correlations of around 0.75 typically observed in our governance data, this confidence interval remains large relative to the units in which govemance is measured.

In the lower panel of Figure 3, we relate this to the significance of cross-country comparisons. A simple summary statistic is the proportion of countries whose true level of governance lies within the 90% confidence interval of a particular country. This proportion will depend on the location of the reference country, and on the correlations in the observed data. We plot tliis proportion for the median country and the country at the first quartile of the distribution of govemance, for various values of p. For the observed correlation of indicators of around 0.75, the 90% confidence interval around the point estimate of govemance for the median country encompasses the true level of governance in about half of all other countries in the world, and somewhat less for a country at the first quartile. Only if the observed correlations are very large is it possible to distinguish the median country from most other countries with a high degree of confidence.

> 9 To see this, it is only necessary to solve the system of nine equations relating the three sample means and the six unique elements of the sample covariance matrix of the indicators to their population counterparts and solve for the unknown parameters.

19
# **4. Extensions**

In this section we consider how our results depend on three assumptions underlying the unobserved components model of the previous sections: that the disturbances are independent across indicators, that the mapping from unobserved governance into observed data is linear, and that the distribution of unobserved governance is normal. We find that the first two assumptions if anything overstate the precision with which govemance is measured. Relaxing the third assumption does not materially affect our results.

# **Correlated Disturbances**

In the previous section we assumed that the disturbances **Eo,k)** were independent across indicators. Intuitively, this assumption allowed us to attribute all of the observed correlation of scores across indicators to the common component of governance go), and hence permitted us to identify the portion of the variation in scores across countries within each indicator due to measurement error. A consequence of this assumption is that any indicator which is not very correlated with the others was interpreted as having a large residual variance.

Although useful, the assumption that the errors are independent across indicators may not be valid, for at least three reasons. First, in the case of polls of experts, it is possible that the perceptions of experts who rank countries on a particular indicator are influenced by their knowledge of countries' rankings on other indicators. Second, the errors in surveys of residents might be correlated across countries if residents of a particular country have a tendency to systematically overstate regulatory and govemance obstacles, due to a broad-based predisposition to report a worse situation than is objectively warranted.10 Finally, it is possible that perceptions of governance from various indicators are unduly influenced by a single event, such as a high-profile scandal which is not representative of the level of graft in that country.

10 See Kaufman and Zoido-Lobat6n (1999).

20
Although it is not possible to statistically identify the correlation of the disturbances across sources, it is straighfforward to see the consequences of positivelycorrelated errors for our results. If the errors are correlated across indicators, each additional indicator contributes less information to our estimate of governance. This will be reflected in the variance of the conditional distribution of governance in each country. In particular, it is straightforwaird to show that holding constant the variance of the residuals on each indicator, the variance of the condition al distribution of govemance is increasing in the correlation between the errors on any two indicators.11 We illustrate the practical consequences of this observation in Table 5. For each of the three aggregate indicators, we re-estimate the variance of the conditional distribution of govemance, imposing a range of assumptions on the correlation of the disturbances. For the purposes of this example, we restrict ourselves to a set of three representative indicators (EIU, DRI and PRS), and also to the set of about 100 countries which appear in all three indicators.12 As the assumed correlation among the errors rises from 0 to 0.5, the aggregate govemance indicators become less precise, although the magnitude of the effect depends on the indicator (since the estimated variances of the disturbances change as well). In the case of government effectiveness, the standard error of the aggregate increases only slightly, from 0.32 to 0.35. In contrast, for rule of law the standard error doubles from 0.33 to 0.66.

It is difficult to adequately address the problem of correlated disturbances simply because it is not possible to separately identify the correlations between the errors. Nevertheless, it is useful to realize that the estimated standard errors associated with point estimates of governance are likely to be substantially understated under the assumption of independent errors. This reinforces our argument of the previous section that cross-country comparisons of the level of governance should be made with caution.

11 If the positive correlations between the residuals differ across pairs of indicators, the relative magnitudes of the estimated variances will also be affected. In particular, suppose that two indicators of bureaucratic quality are highly correlated with each other, but not very correlated with the third. In the previous section we assumed that the errors were iindependent, and so the high correlation between the first two indicators implied that the variance of the errors was small on these indicators relative to the third. However, if we knew that the high correlation between the first two indicators was due to correlated errors, then the estimated variances on these indicators would be large relative to that of the third indicator. As a result, the relative rankings of countries might also be affected.

12We do this only for simplicity, since it allows us to report just one standard error per aggregate, which is the same for all countries.

21
**Non-Linearities**

In the previous section we assumed that the relationship between unobserved governance and observed indicator scores was linear. This assumption places strong restrictions on the units in which governance is measured in the various indicators in our sample. For example, consider an indicator such as Gallup which asks respondents how many cases of corruption there are among public officials, and offers the choice of four broad categories: 'none", "a _few",_ "many" and "a lot". Our observed data consists of numerical scores on a scale of one to four corresponding to these categories. The assumption of a linear mapping from unobserved graft into observed scores implies that the difference in graft between a country with a score of 4 and one with a score of 3 (i.e. the difference between "a lot" and umany" cases of corruption) rs the same as the difference between two countries with scores of 3 and 2 (i.e. the difference between "many" and 'a few"). Given the somewhat vague response categories, it is not at all clear that the assumption of linearity is warranted. Moreover, even if these categories were equally-spaced according to some appropriate metric, the fact that the observed data are discrete while our unobserved govemance indicator is continuous violates the assumption of a linear relationship between the two.<sup>13</sup> Finally, the mere fact that indicators are non-representative may also contribute to a non-linear relationship between governance and observed scores. A number of the indicators in our sample cover primarily developed countries together with a few developing countries. It is possible that the developing countries in these indicators suffer from a "curse of inclusion" in the sense that they receive worse scores than they might otherwise have received simply because they are implicitly being compared with countries in which various aspects of govemance are likely to be much better.14 For example, representative surveys such as DRI or PRS assign moderate scores of 55/100

> 13 The straighfforward solution to this problem would be to rely on an ordered multinomial choice model with individual effects in the latent variable. HoWever, for such a model to be identified, it would again be necessary to assume that the variances of the errors are identical across indicators.

> 14 This problem may be particularly acute for polls of experts who consider a large set of countries at once, since in contrast to surveys of residents, experts are much more likely to be aware of relative comparisons of countries.

22
and 3/6 to Mexico for graft, while on a much less representative survey such as BERI which covers primarily developed countries, Mexico receives a rather poor rating of 1/7.

What can be done about these non-linearities? A very general solution to this problem might be to combine only the ordinal information in each indicator, i.e. the relative rankings of countries within each indicator. In particular, one can think of a given indicator as providing a ranking of pairs of countries according to their level of governance. The information in the relative rankings of countries from various indicators can then be combined by noting that if several indicators consistently rank country A as having better govemance than country B, this provides evidence that governance is in fact better in country A than in country B. Clearly, this approach has several advantages. First, it is computationally very simple. Second, we do not have to know the choice of units in which governance is measured , or whether indicators are representative or not. Third, it does not require any assumption of linearity in the relationship between governance and observed scores. However, this method will result.in larger standard errors for pairwise comparisons since it discards information in differences in the level of scores across countries and indicators.15 To illustrate the relative imprecision of such an ordinal aggregate, for every pair of countries j and j' on every indicator k, we construct an indicator variable x0j,j',k) which takes the value 1 if country j is ranked higher than country j' with respect to the aspect of governance covered by indicator k, and zero otherwise.16 A natural null hypothesis to test is that governance in countries jand j' is the same, i.e. that the probability x(,j',k) is equal to one is 0.5. Under the assumption that the errors are independent across 15 A further drawback of this method is that it is difficult to construct an aggregate ranking of countries according to governance. One possibility would be to average the indicator variables x0,j',k) over all surveys k and partner countries j', and rank oDuntries according to this index. However, it is difficult to put standard errors on such a ranking, since even if the errors are independent across surveys, the x(,j',k) will not be independent across partner countries j'. A deeper problem with this method is that there is a fundamental result from social choice theory which places strong restrictions on the properties such an aggregate ranking may have. According to Arrow's Impossibility Theorem, it is impossible for any aggregation of each indicators' 'preferences" to simultaneiously satisfy three intuitive and desirable properties: (i) the aggregation respects unanimity -- if every indicator says that A is more corrup **t** han B, then so should our aggregate, and (ii) the aggregation displays the independence of irrelevant alternatives property -- the ranking of A and B does not depend on any indicators' ranking bf A or B relative to any other country C; and (iii) the aggregation is non-dictatorial -the aggregate ranking of A and B is not uniquely determined by a single indicators ranking of A and B. In particular, Arrow's theorem tells us that if (i) and (ii) hold, then (iii) does not hold.

16 For those surveys which report discrete categorical scores, we discard 'ties" as uninformative about the relative level of govemance.

23
indicators, this hypothesis can be tested using the data on the proportion of indicators in which country j is ranked higher than country j' in a simple binomial proportions test.

We report the results of this exercise in Figure 4, which is analogous to Figure 2. We again order countries in ascending order according to their point estimates of govemance on the horizontal axis, and we plot the proportion of all comparator countries for which the null hypothesis that governance in these two countries is equal cannot be rejected at the 10% significance level as dark dots on the vertical axis. The light dots report the same information, but at the 50% level. Comparing Figures 2 and 4, it is clear that the ordinal aggregate allows use to identify far fewer statistically significant differences in govemance across countries. In fact, for many countries it is impossible to reject the null at the 10% level that govemance in this country is the same as for every other country in the world using this method! Although this ordinal method is a useful vehicle for making rough comparisons across countries and requires little in the way of assumptions on the underlying data, it is much more difficult to obtain statistically significant differences among countries.

# Alternative Distributions for Governance

In Section 2 we assumed that unobserved govemance and the disturbances were jointly normally distributed. As we noted, this assumption has a significant payoff in terms of analytical tractibility, as it ensured that the distribution of govemance conditional on the observed data was normal, with simple expressions for its mean and variance. However, given the bell-shape of the normal distribution, this approach embodies the implicit assumption that a relatively large fraction of countries in the world have similar moderate levels of governance, and relatively few have either very good or very bad governance. There are two reasons to question this assumption. First, it is not at all clear a priori that this provides an accurate depiction of the true cross-country distribution of governance. Second, it is possible that our finding that it is difficult to statistically distinguish differences in governance between a large proportion of countries in the world is accentuated by the assumption of normality, which forces a disproportionate fraction of countries to be clustered near the mean of the distribution of governance. If instead we assumed that governance was more dispersed, then it is 24
possible that it is easier to identify statistically significant differences in governance across countries.

It is not clear how one might identify the shape of the true distribution of governance across countries, since it is difficult to disentangle the shape of this distribution from the shape of the distribution of the error terms. However, it is possible to explore the robustness of the results to different choices for the distribution of governance itself. We do this by instead assuming that unobserved governance follows a Beta[a,b] distribution. We consider three choices of parameters corresponding to three very different shapes of the possible distribution of govemance. These three possibilities are illustrated in the left-hand column of Figure 5. We first consider a=b=5, which generates a symmetric bell-shaped distribution. This case serves as a benchmark in that it is very similar to the normal distribution we have been using so far. We also consider the possibility that the distribution of govemance is skewed to the right (a=2, b=5), with relatively few countries with very good governance in the right tail of the distribution. Finally, we consicder the possibility that govemance is uniformly distributed (a=b=1), with a similar proportion of countries at each possible level of govemance. We continue to assume that the diistribution of the disturbances is normal.

On the right-hand side of Figure 5, we explore the consequences of these alternative assumptions for our conclusions about the significance of cross-country differences in govemance. As a specific example, we focus on an aggregate of the three largest representative indicators of graft (EIU, DRI and PRS), and again restrict ourselves to the sample of about 100 countries appearing in all three indicators. For each country, we report the point estimates and standard errors corresponding to each assumption on the distribution of governance.17 As the assumed shape of the distribution of true governance changes, not surprisingly so does the distribution of point estimates. The more important observation is that our results on the difficulty of distinguishing between countries do not change. It is clear from Figure 5 that the 17 We compute these as follows. First, using a method of moments argument we construct estimates of the parameters of the model corresponding to the assumed distribution of govemance (note that the mean and variance of this beta distribution change as we vary the parameters). We then construc **t** he joint distribution of go) and y() as the appropriate mixture of a normal and a beta distribution, and then obtain the marginal distribution of yj) and conditional distribution of go) given yG) by appropriate numerical integrations of this joint distribution. We numerically evaluate the mean of this distribution, and the 5e and 95 percentiles, and report these for each country in Figure 5.

25
number of countries with 90% confidence intervals falling entirely within particular quartiles is essentially unchanged for each quartile, as we vary our assumptions on the shape of the distribution of govemance.

26
# **5.** Conclusions

In this paper, we have talken the view that the many different available indicators of governance provide imperfect signals about a relatively small number of fundamental aspects of governance, such as rule of law, govemment effectiveness, and graft. We grouped the many available indicators into three clusters corresponding to these concepts of governance, and used a linear unobserved components model to obtain aggregate estimates of these three aspects of govemance. Despite several optimistic assumptions, we find that govemance is not very precisely measured using these aggregate indicators. In particular, although it is possible to identify statistically significant differences between countries at opposite ends of the distribution of govemance, it is much more difFicult to discriminate among the majority of countries with any degree of confidence.

Nevertheless, we find the aggregate governance indicators to be useful for several reasons. First, they are' based on a methodology which provides a consistent framework for placing data from various sources into common units, taking into account that the samples of countries included in different sources may not be representative of the world as a whole. Second, the aggregate indicators span a much larger sample of 155 or more countries, permitting (admittedly-imprecise) comparisons across a much larger set of countries than is possible using any single indicator. Third, although the aggregate indicators are not as precise as one might have hoped, they are nevertheless much more reliable than any individual indicator. Finally, we believe that it is useful to have quantitative measures of the precision of aggregate indicators in order to caution users of both individual and aggregate indicators of the substantial margins of error associated with cross-country comparisons of govemance.

Empirical research on governance issues can also benefit from the aggregate indicators presented here. Many empirical studies which use govemance indicators as either left-hand or right-hand side variables are limited to small samples by the poor country coverage of many indicators. This can potentially introduce a variety of sample selection biases. In addition, the measures of precision we report can be used to correct for the attenuation bias due to measurement error in govemance indicators used as dependent variables.

27
In the long term, however, our results also point to the inadequacy of existing governance measures. jt is very unsatisfying that existing data, even with favourable assumptions, allows us to identify relatively few statistically significant differences in governance across countries. Moreover, existing data provides at best tenuous links between perceptions of govemance and objective policy interventions that govemments interested in improving the quality of governance can undertake. There is therefore a need to improve the quality and quantity of govemance data, both by improving and extending cross-country survey work of govemance perceptions, as well as employing country-specific in-depth govemance diagnostics.18 Many of the polls and surveys we use suffer from deficiencies, such as poorly-worded questions about ill-defined and excessively broad concepts. There is room to improve these instruments by asking respondents about their direct experiences with well-defined events and using transparent units to measure govemance. However, these are time- and resourceintensive exercises, and intemationally-comparable high-quality data of this sort is years away.

**18** Detailed country diagnostic exercises such as those currently being piloted by the World Bank have the potential to provide much more detailed information on the **specific** institutional failures which contribute to perceptions and the reality of poor govemance. Kaufmann, Pradhan and Ryterman (1998) provide a description of these exercises.

28