metadata: authors: 0: author: Cameron,Grant James
1: author: Dang,Hai-Anh H.
2: author: Dinc,Mustafa
3: author: Foster,James Stephen
4: author: Lokshin,Michael M.
count: World
docna: 0: docna: Measuring the Statistical Capacity of Nations
docty: Policy Research Working Paper
subtopic: Inequality,Food Security,Economic Growth,Early Child and Children's Health,Nutrition,Reproductive Health,Poverty Impact Evaluation,Poverty Monitoring & Analysis,Poverty Diagnostics,Small Area Estimation Poverty Mapping,Poverty Lines,Poverty Assessment
teratopic: Agriculture,Macroeconomics and Economic Growth,Health, Nutrition and Population,Poverty Reduction
abstracts: cdata!: The international development community
            has used the World Bank's Statistical Capacity Index
            since its inception in 2004. The Sustainable Development
            Goals create new challenges for national statistical systems
            to produce high-quality and internationally comparable data.
            This paper reviews measurement methodologies, posits desired
            attributes, and presents theoretical and empirical
            frameworks for the new, improved index to monitor progress
            in the statistical capacity of nations. The paper
            illustrates the properties of the updated index with global
            data from 2016.
repnme: repnme: Measuring the Statistical Capacity of Nations
display_title: Measuring the Statistical Capacity
            of Nations
listing_relative_url: /research/2019/01/30744565/measuring-statistical-capacity-nations
topicv3: Agriculture and Food Security m1326205 435,Poverty m1326687 695,Poverty Measurement and Analysis m1326941 707,Stunting m1327504 1445,Sustainable Development Goals m1504460 3134
publishtoextweb_dt: 2019-01-08T00:00:00Z
historic_topic: Agriculture,Macroeconomics and Economic Growth,Health, Nutrition and Population,Poverty Reduction
no_of_pages: 62
author_keys: 0: author_key: 631089
1: author_key: 747967
2: author_key: 166351
3: author_key: 1612234
4: author_key: 144523
model_extractions: WPS8693 Policy Research Working Paper 8693 Measuring the Statistical Capacity of Nations _Grant J. Cameron Hai-Anh H. Dang Mustafa Dinc James Foster Michael M. Lokshin_ Development Economics Development Data Group January 2019
Policy Research Working Paper 8693

# **Abstract**

The international development community has used the World Bank’s Statistical Capacity Index since its inception in 2004. The Sustainable Development Goals create new challenges for national statistical systems to produce high-quality and internationally comparable data. This paper reviews measurement methodologies, posits desired attributes, and presents theoretical and empirical frameworks for the new, improved index to monitor progress in the statistical capacity of nations. The paper illustrates the properties of the updated index with global data from 2016.

This paper is a product of the Development Data Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at hdang@worldbank.org and fosterje@gwu.edu.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Measuring the Statistical Capacity of Nations**

_Grant J. Cameron, Hai-Anh H. Dang, Mustafa Dinc, James Foster, and Michael M. Lokshin_<sup>*</sup> Key words: statistical capacity, statistical indicators, statistical index, national statistical system, data, measurement JEL: C8, H00, I00, O1

> * Cameron <u>(gcameron1@worldbank.org) is manager with the Advisory Services and Economic Statistics Unit,</u> Development Data Group, World Bank; Dang (hdang@worldbank.org; corresponding author) is economist with the Survey Unit, Development Data Group, World Bank; Dinc (mdinc@worldbank.org) is senior economist with the Advisory Services and Economic Statistics Unit, Development Data Group, World Bank; Foster (fosterje@gwu.edu; corresponding author) is the Oliver T Carr Professor of International Affairs and Professor of Economics, George Washington University; Lokshin <u>(mlokshin@worldbank.org) is manager of the Survey Unit, Development Data</u> Group, World Bank. We would like to thank Juderica Dias for helpful support with the data, and Junhe Yang for capable research assistance. We are grateful to the UK Department of International Development for funding assistance through its Knowledge for Change (KCP) program and Strategic Research Program (SRP).
# **1. Introduction**

The national statistical system, or NSS, plays a crucial role in modern economies. It provides stake-holders, ranging from policy makers to stock market analysts and the general public, with the latest data on the country’s socio-economic developments. At the international level, monitoring progress on global undertakings such as the recently established Sustainable Development Goals (SDGs) requires high-quality data that must be produced consistently across different national statistical systems. Assessing and improving the capacity of a country’s NSS has long been a part of the global agenda. The international Partnership in Statistics for Development in the 21<sup>st</sup> Century (PARIS21) Task Team was established in 2002 to help measure country statistical capacity. Over the subsequent years, a few capacity assessment tools have been developed to identify the weaknesses and strengths of national statistical systems.<sup>1</sup> The World Bank’s Statistical Capacity Index (SCI) is one such tool that has been widely employed.<sup>2</sup> Several international and national agencies have adopted the SCI for measuring progress in statistical capacity building and related investments. The United Nations, for example, uses the SCI to measure trends in the development of national statistical capacity (United Nations, 2016). The SCI is used to evaluate the efficiency of statistical support provided to a country as well as the need to further develop its statistical capacity (PARIS21, 2002). Some regional organizations use the SCI to identify areas of improvement in their member countries (OIC, 2012), while researchers use the SCI as a benchmark to validate their new statistical indexes (Sanga et

> 1 Statistical capacity is usually interpreted as the ability of an NSS to meet user needs for relevant and good quality statistics in a timely manner. An NSS often consists of a number of different data-producing agencies and departments (such as the national statistical office, the central bank, and statistical departments within other line ministries), which renders the task of directly measuring statistical capacity a difficult one.

> 2 For brevity, we refer to both the Statistical Capacity Indicators and the Statistical Capacity Index as the SCI in the rest of the paper. We will make it clear where we refer to either the indicators or the index. We similarly refer to both the Statistical Performance Indicators and the Statistical Performance Index as the SPI. We also refer to the old SCI as the SCI, and the newly revised SCI presented here as the SPI.

2
al., 2011). The World Bank mainstreamed the SCI in its monitoring and assessment framework and has adopted it as a baseline indicator in various projects at the country level.<sup>3</sup> The SCI is based on publicly available data, and this has various advantages over other indexes of statistical capacity. A key advantage of the SCI is that it can provide assessment of a country’s statistical capacity in an internationally comparable and cost-effective manner.

Existing efforts in building indexes to assess statistical capacity have focused on the practical aspects such as data collection, organization, and legal issues, paying little attention to the underlying theoretical principles that are indispensable for the construction of a reliable, transparent, and consistent statistical capacity index. For example, the UNECE, in a recent Global Assessment report, discusses only the legal basis, description of the statistical system, data source, and processing of the target country (UNECE, 2014). The FAO, in its guidelines for assessing country capacity in producing agricultural statistics, provides instructions on completing the questionnaires and on compiling the assessment indicator (FAO, 2014), but pays no attention to the axiomatic principles of these indicators. The U.S. Census Bureau developed and recently updated (2017) the Tool for Assessing Statistical Capacity (TASC) with a primary objective of measuring the overall capacity of an NSS by providing a breakdown of the areas of strength and weakness. However, the focus of this instrument is on measuring the capacity of an NSS to conduct household-based surveys and censuses.<sup>4</sup> To our knowledge, only Sanga, Dosso, and Gui-Diby (2011) discuss the technical framework behind the African Statistical Development Index (ASDI).

> 3 For other recent examples that use the SCI, see: Beegle et al. (2016) for an analysis of the relationship between good governance and statistical capacity in African countries; Tapsoba, Noumon, and York (2017) for the impacts of statistical capacity on reducing procyclical fiscal policy; and UNICEF (2018) for the role of statistical capacity in tracking the SDG for child development.

> 4 We return to provide more discussion on the SCI and these other methods in Section 2 below.

3
In this paper, we aim at laying out the conceptual foundation behind statistical capacity indexes, and construct a new index based on practical and theoretical considerations. We review existing measurement methodologies, posit desired attributes, and propose updated indicators, and an updated Statistical Capacity Index (hereafter referred to as the Statistical Performance Index, or SPI). On the empirical front, we expand the number of indicators in the old SCI by almost twice, and we extend the sample of covered countries by one-half to all countries in the world.

This paper consists of six sections. We provide a brief overview on the SCI in the next section before presenting the theory in Section 3. We offer in this section detailed discussion on the proposed index ranging from the conceptual framework (including its policy relevance, its concept of statistical capacity and desirable characteristics) to the technical framework (including its dimensions, aggregation methods, and axioms). We then describe the SPI and its new features before comparing it with the SCI in Section 4 and providing some empirical illustrations in Section 5. Section 6 concludes.

# **2. Background on the SCI**

The SCI is a tool for assessing country-level statistical capacity, developed in 2004 to assess the effectiveness of the World Bank’s lending projects related to improvements in countries’ statistical capacity. Table 1 compares the SCI with the statistical capacity measurement indexes used by other organizations including PARIS21, the Food and Agriculture Organization of the United Nations (FAO), the United Nations Economic Commission for Europe (UNECE), the United Nations Economic Commission for Africa (UNECA), and the U.S. Census Bureau. Table 1 shows that the SCI is the only tool that provides comparable data across different countries over time. Further, the SCI covers the largest number of countries—146 countries starting from 2004 and has by far the widest coverage of country-level indicators.

4
There is also a key methodological difference between the SCI and other assessment tools. The latter methods collect data directly from national statistical offices’ staffs or local experts. While this procedure may provide a more in-depth analysis and uncover finer details in the organization of an NSS, it incurs high costs and is (far) more time-consuming. Furthermore, even the best evaluator can bring personal biases, non-uniform conceptions of capacity, or other subjective elements. Direct interviews of government officials might bias responses and complicate comparability across countries.<sup>5</sup> The SCI follows a different approach by using publicly available data and focusing on a set of easily observable and verifiable indicators. It provides internationally comparable, objective, country-level assessments across the globe. The World Bank has mainstreamed these indicators in its monitoring and assessment frameworks such as the Corporate Score Card (CSC) and the IDA Results Measurement System (RMS). The SCI is also used in various World Bank projects as a baseline indicator.

Yet, there are several areas in which the existing SCI can be improved. First, it comprises of a limited number of indicators and includes no indicators of some important surveys such as the labor force surveys and establishment surveys. Second, it ignores the data dissemination practices of an NSS, which is one of the key features of data usage. Third, the SCI has been criticized for placing too much weight on statistical output and activities, while neglecting the infrastructure and resource components of statistical systems. It was also criticized for its weighting scheme (i.e., equal weighting of each dimension and individual indicator) and aggregation method (i.e., simple arithmetic average).

> 5 Government officials may have an incentive to overestimate their country’s statistical capacity when there are concerns about their ability to deliver; the opposite can happen when they are requesting additional international aid or technical assistance.

5
Finally, since its launch in 2004, the SCI’s methodology and coverage have basically remained the same, while the global data landscape has changed significantly. NSSs have made significant advancements with data collection and dissemination practices. At the same time, the adoption of the Sustainable Development Goals (SDGs) set an ambitious development agenda for the next 15 years on ending poverty, protecting the planet, and ensuring prosperity for all by 2030. This, in turn, increased the demand for data and raised the bar for national statistical systems regarding their capacity to produce high-quality data. We thus propose to improve the current SCI to better suit the changing global data landscape.

We next present the theoretical framework that can be used to construct an index that measures statistical capacity.

# **3. Theory**

# **3.1. Outline of Policy Relevant Measurement**

In order to construct a measure that is policy relevant it is helpful to follow a series of basic

# steps.<sup>6</sup>

The first step asks the question: what phenomenon is being measured? A clear conception helps orient the process by which the measure is assembled and will prove valuable in communicating its underlying meaning.

The second step asks: for what purpose or purposes is the index being sought? Knowing how the index will be used can greatly affect subsequent choices in its construction, and its eventual

> 6 This process is similar for many types of measurement exercises. See for example Alkire et al. (2015) in the context of multidimensional poverty measurement.

6
suitability. In particular, it will help define the unit of analysis both for data gathering and reporting purposes.

The third step identifies a list of essential characteristics, or desiderata, that the methodology should exhibit. This list of “pre-axioms” helps orient the construction process and define what success means.

A fourth step identifies the conceptual space in which measurement is to take place. If there are multiple conceptual dimensions, consideration must also be given to the relative importance of each.

The fifth step selects the form of the variables to be used and the aggregation method to be employed – how the variables are to be combined into an overall measure.

The sixth step identifies a set of axioms that the resulting index should satisfy to have the greatest practical utility. Axioms are not sterile mathematical requirements, but rather contain the salient nuggets of policy required of the index: which aspects of the data should be ignored, which should be reflected, and helpful consistency requirements over subsets of data. Together, these six steps comprise the core theoretical elements of our proposed measurement technology.

# **3.2. Process to Construct an Index of Statistical Capacity**

We now turn to a detailed discussion of these six steps and the elements involved in the context of an index of statistical capacity. We will also consider the practical details of our implementation of the general measurement approach.

# **3.2.1. What Is Being Measured? The Concept of Statistical Capacity**

7
Statistical capacity is the ability of a National Statistical System (NSS) to sustainably collect, compile, and disseminate data to inform national and international policy decisions. Capacity is difficult to measure since it is only partially revealed by evaluating achievements or other observable characteristics of the NSS. A system may have the capacity to produce quality data products but has not yet done so; or it may no longer have the capacity to produce despite having succeeded in the past. Our approach is to focus entirely on the observables for the NSS, but then to expand consideration to a broader range of performance indicators, to better reflect its underlying capacity. The challenge then becomes how to identify the important dimensions of products and processes that should be covered by such a measure, as well as the specific variables within each dimension.

But before discussing dimensions and variables, it is important to spell out the purpose of the index and its desirable characteristics.

# **3.2.2. The Purpose of the Index**

The purpose of a statistical capacity index is to assess a country’s statistical capacity to monitor the progress of reforms and projects in the area of statistical capacity building and identify the areas for improvements in the National Statistical Systems. An index can also facilitate intercountry comparisons and help interpretation of the trends both within a country over time and across countries.

# **3.2.3. Desiderata**

8
The desiderata for an index (or more generally a measurement method) are the general characteristics or criteria that we would like it to have. Consider the following list that modifies a standard set found in the literature on measurement.<sup>7</sup>

- i. _Simple_ . It must be understandable and easy to describe.

- ii. _Coherent_ . It must conform to a common-sense notion of what is being measured.

- iii. _Motivated_ . It must fit the purpose for which it is being developed.

- iv. _Rigorous_ . It must be technically solid.

- v. _Implementable_ . It must be operationally viable.

- vi. _Replicable_ . It must be easily replicable.

vii. _Incentive Compatible_ . It must respect country incentives.

The first characteristic speaks to the simplicity of a measurement approach, and to the ease with which it might be communicated. Methods that measure an underlying concept in an unclear or unduly complicated fashion discourage participation by stakeholders in policy discussions. When there are significant barriers to understanding a measure, this by itself can lead stakeholders to question its authenticity as a policy relevant tool. In contrast, an index or method that is more easily understood allows stakeholders to evaluate for themselves its qualities. This desideratum also extends to the data upon which the measure is based by requiring a clear and understandable link between the indicators used in an index and the underlying data. If indicators are vaguely defined or sourced, then the interpretation of an index can be superficially clear, but actually obscure. This aspect of the requirement is facilitated by having online data sources and adequate

> 7 See, for example, Alkire et al. (2015).

9
documentation to offer a clear provenance for each indicator. Transparency of the measure and its components is a key characteristic of an index of statistical capacity, which will facilitate its takeup by stakeholders, promote independent confirmation of its findings, and support its credibility.

The second characteristic of coherence speaks to the measure’s authenticity in capturing the phenomenon it aims to measure. Simply put, one should be able to describe in plain language the underlying concept of what is being measured by the index, and then argue that this concept is indeed a credible version of what needs to be measured. In the present case, we will measure a nation’s statistical capacity by examining its products and processes.

The third desideratum considers the extent to which the index fulfills the purpose for which it has been constructed. As noted above, the purpose of a statistical capacity index is to accurately and timely measure a country’s statistical capacity. In creating the index, it is important to keep the motivating objectives in mind, and work towards their achievement.

The fourth characteristic addresses the rigor of the methodology embodied in the measure. Much of this concerns the aggregation method employed and the axioms that might be satisfied by an index of statistical capacity. For example, one would not want to have a multidimensional index for which unambiguous improvements in individual components would be associated with a decrease in overall capacity as measured by the index. To give substance to this general characteristic, we outline several axioms that the index might fulfill. There are also other measurement problems that a technically solid index should avoid. For example, if the data used in the index are not cardinally relevant, then the index must respect this constraint when aggregating; or if there are several dimensions, but no meaningful basis has been established for comparisons, then the dimensions should not be combined. These latter two examples are closely 10
linked to the broader concern of robustness, which requires results derived from the index to be robust to allowable changes in its components.

The fifth characteristic brings real world data collection and choice into the picture, requiring that the index can actually be implemented using available data. This carries with it restrictions on the qualities of the data to be used, such as the following considerations. Clearly the purpose and the coherence desiderata require the coverage of the data to be sufficient across countries and dimensions. Data should be homogenized across countries to ensure that the individual variables are comparable and be suitable for evaluating countries at all levels of statistical capacity. This criterion also suggests that the index should be based on publicly verifiable data compiled, collected, and disseminated by the NSS. As such, the actual NSS products—such as the census, key SDG indicators, and other variables—could provide a reliable and cost-effective basis for an index designed to measure its statistical capacity.<sup>8</sup> Implementability requires a verification that this is true.

The sixth characteristic also places constraints on data but focuses on the additional dimension of time. Not only should the index be able to be calculated and compared across countries at a given point in time but, consistent with its purpose, it should also be possible to construct the index across many points in time. Consequently, the data requirements should also be maintained through time to allow reasonably consistent estimates of progress and this should translate to a time path that is coherent and predictable. In contrast, if an index were to exhibit undue volatility through time this could suggest replicability problems stemming from the data, the aggregation method, or from both. It should be noted that replicability is also supported by the simplicity

> 8 Again, other sources, such as expert opinions or intensive studies carry with them a number of problems, including high monetary cost and increased measurement error due to country-specific orientation and idiosyncratic or irregular collection. The choice of data will be further discussed below.

11
desideratum, which can lower the cost of replication by others and hence ensure that, through many eyes, there are fewer errors.

The seventh and final criterion concerns the interaction between actions by the NSS and the behavior of the index through time. In essence, a country should not see its index of statistical capacity fall when the only movement between time periods is an improvement in individual variables. Some global indices are constructed in such a way that the index value itself depends on data from _other_ countries. A given country can achieve absolute improvements in all facets of the measure and still have its index value drop.<sup>9</sup> This possibility can disrupt the clear pathway between a country’s policies and the index level, thereby diminishing the government accountability that the index is meant to support.<sup>10</sup> Alternatively, suppose the index adopts a goal-oriented or targeted approach, by which targets are set and progress is measured as the extent of movement towards reaching them. A target may be fixed through time – like an absolute poverty line – when conditions change, or it might vary with the outside conditions – like a relative poverty line would. In the latter case, it can be unclear whether the progress (or lack thereof) is due to changes in the underlying variables of interest or changes in the target. Incentive compatibility suggests using fixed or absolute targets that link the measure to actual progress.

In sum, constructing a measure of statistical capacity entails many distinct choices that can appear to be arbitrary and unrelated to one another if no context is provided. A set of desired characteristics, or desiderata, can provide the guiding principles that help organize these choices

> 9 This was a problem with the traditional Human Development Index, which relied on empirical goalposts (or its highest and lowest values for each dimension) to fix its minimum and maximum values. See Alkire and Foster (2010). 10 Undue volatility of an index, where the changes in index level appear to be spurious or unrelated to actual policies, can likewise disrupt the policy relevance of the index as an accountability tool. We return to more discussion in the next section.

12
to obtain a relevant and useful measurement tool. The desiderata described above will be invoked several times in what follows.

# **3.2.4. Dimensions**

The concept adopted here views the statistical capacity of an NSS in terms of its range of products, and the processes it uses to produce and disseminate them. The production process for statistical outputs has certain similarities to the traditional production model from economics, and this analogy might be useful for identifying the salient dimensions of a statistical capacity measure. The production process begins with a technology that is used in generating the statistical products, and the level of this technology is clearly a relevant component of statistical capacity. The resulting statistical outputs might be divided into two general categories. First are the intermediate products, which have direct use for specialists but require additional processing to create products suitable for general use. For example, a census can be helpful for policy analysts but must be processed to obtain useful statistics. Second are the final products, which are available in a form that can be understood by the public. The key macro statistics of a country would naturally be viewed as final products. Even after the products have been created, their existence does not imply that potential users will actually have access to them. Statistical products may be available to only a few users, or available to all. The final dimension then covers the extent to which statistical products are disseminated.

This simple framework helps to identify four coherent dimensions for a measure of statistical capacity, namely: (i) Methodology, Standards and Classifications (MSC), which provides information on the technology being used by the NSS; (ii) Census and Surveys (CS), which describes the intermediate products of the NSS; (iii) Availability of Key Indicators (AKI), which 13
focuses on key final products needed for policy; and (iv) Dissemination Practices and Openness (DPO), which evaluates the extent to which products are publicly disseminated. It is easy to see that each of these dimensions is centrally related to the statistical capacity of an NSS.<sup>11</sup> We return to more discussion on these four dimensions in Section 4.1.

As with any measurement exercise, there will likely be dimensions missing from the list. One dimension that could be particularly relevant for measuring statistical capacity is _flexibility_ , or the ability of an NSS to adapt to changing circumstances.<sup>12</sup> For example, in response to a change in national priorities, the NSS may need to alter the schedule or scope of data production. Alternatively, as new technologies for data gathering are being developed, international guidelines may change, requiring the NSS to alter its methods. However, to fully capture the missing dimension of flexibility, one needs data about circumstances that have not yet have happened, and this directly conflicts with the implementability desideratum.<sup>13</sup> Likewise, the absence of such data in the present context leads to the use of more observable variables.<sup>14</sup> A second missing dimension might be _statistical infrastructure_ as represented by the underlying array of inputs that the NSS 11 It may be useful to briefly discuss whether one dimension is inherently more important than another. For instance, since earlier dimensions are needed for the latter dimensions, they can be arguably viewed as being more important. Yet, from the point of view of stakeholders, the opposite could also be seen as true. Notice that both the first and last dimensions could be considered to be _process_ dimensions, while the middle two are _product_ dimensions. We could posit that process and product are equally important to evaluating statistical capacity, while making a judgment that the individual components are also of equal importance. If so, then the four dimensions would be consistent with Atkinson et al (2002) who observe that “the interpretation of a set of indicators is greatly eased where the individual components have degrees of importance that, while not necessarily exactly equal, are not grossly different.” 12 For fundamental discussions of flexibility in the context of production, see, for example, Kreps (1979).

13 Parts of this aspect might be captured in the technology or product dimensions: if the country is able to produce a best practice census, then it is plausible that it could produce comparably sophisticated products if and when needed. An analogy can be seen with Sen’s notion of capability, which evaluates a person’s well-being via sets of available options, or capability sets, which require information on alternatives that have not yet been chosen. The difficulty of obtaining counterfactual information has led many researchers to measure capability using actual achievements or functionings.

14 One related critique is that since the variables are based on past data, the resulting measure cannot be “forward looking” enough to convey the future possibilities for a given NSS. This inevitable consequence is analogous to the challenge of measuring “vulnerability to poverty” using household survey data. See Dang and Lanjouw (2017) for a recent study that addresses this challenge.

14
has at its disposal for producing and disseminating data. This could include the physical plant, the quantity and quality of computing facilities, and the number and skill level of workers, among other inputs. All of these are clearly relevant information for gauging the capacity of an NSS. However, it is also clear that implementability would once again be a problem, since gathering accurate, homogenous data on inputs would be a costly and drawn out task.

# **3.2.5. Variables and Aggregation**

Once the dimensions have been specified, attention turns to identifying the variables and selecting an appropriate aggregation method. The desiderata suggest that the variables in an index should be coherent with the concept being measured; they should be publicly available (i.e., drawn from NSSs, rather than from experts or other sources). The aggregation method should be selected with rigor in mind, including the axioms or properties that the method satisfies. At the same time, it should aim for simplicity to maximize general understanding and impact.

We turn next to discussing our proposed aggregation method. To help focus discussion on policy relevance and conceptual issues, we leave the more technical details to Appendix 1, Part A. We also offer a simple numerical example that further illustrates the ideas in this subsection in Appendix 1, Part B.

# _Basic Setup_

In the domain of statistical capacity, a variable is typically derived from a simple “yes-no” question concerning a normative guideline that a country’s NSS should meet. For example, in the MSC dimension, a relevant question is whether a modern system of national accounts is being used. In the CS dimension, a question might ask whether there has been a population census within the last ten years. Such goals and targets can be found in discussions of standards and best practices 15
V>2 v v=1,..,V a=(a, 2 v) w=(W, 2 v) y> 0 C = C(a;w), = ~~wa~~ _ ~~Widittwyay~~ wu wyte+wy u = (1,...,1) Ua; w)
so that the weight on each a� is w�/�w� �⋯�w��.<sup>17</sup> Clearly, C takes on values between 0 and 1.

A key challenge is how to discern the relative importance of the different attainments in order to aggregate up the counts. One overarching structure for doing so is found in the nested method of Alkire and Foster (2011), which extends Atkinson’s (2003) “equal importance” construction to subdimensions and variables. For example, in the case of statistical capacity, it could be argued that the CS dimension should be divided into two equally important subdimensions, one for censuses and a second for surveys. Likewise, variables that reside within a given subdimension (or dimension if it has no subdimensions) should be selected to have roughly equal contributions within the group. A nested approach helps to account for the relative values of variables, subdimensions and dimensions in a fashion that is coherent with the concept being measured.<sup>18</sup>

# _Useful Properties_

Once w has been set and the counting index C has been defined, it offers four useful properties we examine below. First, C is additively decomposable by subsets of variables; second, C is additively decomposable by subsets of countries (or regions). Third, C can be constructed using

> 17 Although the entries of _w_ will typically sum to 1, this general definition allows the sum to exceed or fall short in line with different approaches to counting indices. This generality is also convenient in defining sub-indices that use only a subset of the values in _w_ .

> 18 Other empirical methods can be employed to infer the importance of individual variables, such as applying principal component techniques to obtain weights from the principal component combination that explains the most variation. This approach has the benefit of letting the data speak for themselves. However, estimation results are completely dependent on the timing and coverage of the data set used to calibrate the weights, which raises hard-to-answer questions: How far back in time should the data extend? What should be done when new data come online? Can an outside indicator of what is being measured actually be found? As such, empirical methods may yield data-oriented weights that totally ignore variables conceptually considered to be important. See also Decancq and Lugo (2013) for a discussion of empirical methods for finding weights.

17
Saya ~~S.4S ="~~ {a, vs ~~S'S!~~ Su “ ~~=e~~ S Ss S Sr oSsr Sr) ca ~~weet~~ sy w ~~htull~~ sr wu w'u SS Ss (d) ® d) —= p(a®d )d)\ — ~~=awa~~ <u>(d).,4 (a)</u> (d) @ (d) d = 1,2,...,D C= ye ~~s~~ (d) =1p
NXxV n C =C(A;w) = uw) L(A; w) M(A;W) = Dhar daa ~~GQ1 Gpentwy~~ evy = Laa ~~t~~ <u>y</u> ~~1~~ MCviw) C(A;w) = ~~—1~~ n=1C(a",w) C(A; w) NXxV M(A;w) = 1-C(A;w)
The nested counting index is well suited for dichotomous variables and its simple aggregation method – a weighted mean – is easy to interpret.<sup>20</sup> Yet, while each dichotomous variable represents a separate test or target, some tests may be related, giving rise to additional possibilities for variables and aggregation. For example, two distinct questions might refer to the system of national accounts being used by the NSS, where there are two levels of standards in common use. A first variable might ascertain whether the NSS is at least using the lower standard; the second might determine whether the higher standard is being employed. The three feasible combinations for the two variables are: (0,0), which indicates that neither system is being employed; (1,0), which indicates that the lower and not the higher standard is being employed; and (1,1), which indicates that the higher standard system is being employed.<sup>21</sup> It is possible to combine the two related variables into a single, non-dichotomous variable that would assign unique values to the three levels, say 1 to the higher outcome (1,1), some numerical value λ between 0 and 1 to the lower outcome (1,0), and 0 to the outcome where neither standard is in place (0,0). Constructed variables like this are commonly used in measurement exercises. Yet, assigning an arbitrary numerical value λ to a variable does not by itself make the variable cardinally meaningful or comparable to other variables. Consequently, the resulting index and country rankings will simply not be robust to admissible changes in values.<sup>22</sup>

> 20 Notably, other options exist for aggregating dichotomous variables, but these have limitations. For example, one option is the class of general means (or means of order α), which includes the geometric mean as a special case. Since variables here have 0 values, only general means with positive parameter α are applicable (the geometric mean is also problematic in that it takes on a value of 0 whenever any variable is 0 and thus violates monotonicity). Furthermore, a general means would rank countries the same way but lose the useful decompositions and simple interpretation of the index value. Another option is a dashboard approach, where disaggregated variables are presented in their natural state. Yet, this approach does not offer easy interpretation, especially in our case where the number of variables is large. 21 The outcome (0,1) would not be observed in this case of related variables, since achieving the higher level entails achieving the lower level.

> 22 Both cardinality and comparability can be defined in terms of admissible changes in the scales used to represent variables.

20
Our index provides a systematic method to avoid such mistakes, which is stated in Proposition 3 below.

# **Proposition 3. Equivalence between Dichotomous and Multi-Valued Indicators**

_Suppose that there are T many related dichotomous variables providing information on ascending levels of a given aspect of statistical capacity. Let S' be the set of subscripts for these T variables so that_ a<sup>��</sup> _denotes the relevant vector of achievements. Let_ C�a<sup>��</sup> ; w<sup>��</sup> � _be the counting index for this subset of variables; it assigns the T+1 options for_ a<sup>��</sup> _the numerical values of 0, 1/T, ..., (T1)/T, and 1, respectively. Now replace the T dichotomous variables with the single variable x having (one of) these T+1 values and assign it the total weight inherited from the original T variables. The weighted mean across x and the remaining dichotomous variables generates identical index values as the original counting measure and can be used equivalently._

**Proof** : Appendix 1, Part A.

Since the constructed single variable has a linear scale (i.e., with equal increments between adjacent levels), the resulting index is neutral in gauging improvements at different starting values. In certain circumstances, though, it might be helpful to have a range of indices that emphasizes improvements at different outcome levels (i.e., the incentive compatible desideratum). For example, for higher capacity nations that have covered all the basics, it could be useful to have an index that emphasizes improvements at the upper level. Likewise, for lower capacity nations that have not yet achieved basic levels, an index that emphasizes improvements at lower levels might have merit. Our index can be modified to accommodate these cases, following the intuition found in the poverty measurement literature.<sup>23</sup> Let C��a<sup>�</sup> ; w<sup>�</sup> � be defined by for positive α. Intuitively, the index first transforms each variable by a positive parameter, then takes the weighted average of the transformed levels. As noted above, this does not change the

> 23 See for example Foster, Greer, and Thorbecke (1984).

21
values of dichotomous variables; only the constructed variables will be affected by the transformation. For α < 1, the transformation is concave and places greater emphasis on improvements at the lower end of the outcome range; for α > 1, the transformation is convex and places greater emphasis on improvements at the higher end; and for α = 1, the index becomes C��a<sup>�</sup> ; w<sup>�</sup> ��C�a<sup>�</sup> ; w<sup>�</sup> �, the usual counting index. While _C_ is the central index that will be used, other indices from the parametric class can help focus on high-level or low-level improvements when needed.

# **3.2.6. Axiomatics**

Axioms are rigorous properties for an index to satisfy, and they are more formalized and generalized than the properties discussed earlier in Subsection 3.2.5. Axioms help in understanding what an index is actually measuring and in deciding which index to use. Knowing which properties an index satisfies can help in interpreting the empirical results obtained using that index; certain forms of policy analysis become possible only when the index satisfies a given property. Some axioms can be interpreted as “nuggets of policy” that specify the kinds of changes that should leave the index value unchanged and those that should alter it. Others break down the index value to help understand how dimensions contribute to that value. Our proposed index satisfies three axioms, which include symmetry, monotonicity, and subgroup decomposability.

As noted in Foster et al (2013), axioms can be usefully grouped into three categories: invariance axioms, which indicate what not to measure; dominance axioms, which indicate what the index should measure; and subgroup axioms, which break down or build up indices by variables or units of analysis. The three axioms our proposed index of statistical capacity satisfies 22
are closely related with these three groups of axioms. In what follows, a generic index of statistical capacity over profiles a��a�, ... , a�� will be denoted by _F_ .

_<u>Symmetry</u>_ <u>: in other measurement environments where the number of people, dimensions, or</u> other factors may differ across comparisons, invariance axioms are often used to ensure consistency. In the present context, the index _F_ is being applied to one country’s data with a fixed number of dimensions and dichotomous variables, so properties of this sort are not needed. A second common form of invariance axiom is anonymity or symmetry whereby the index value is unaffected when variable levels are switched. In the present context, where the variables have a structure as represented by hierarchical tree _T_ and partition _P_ , universal symmetry is not appropriate. Motivated by Basu and Foster (1998), one might consider a weaker form of symmetry that is contingent on variables being “similarly placed” in the variable structure. We say that profile _b_ is obtained from profile _a_ by a _basic switch_ if b� �a�� and b�� �a� for some v � v′ in the same basic group, while b�" �a�" for all other _v"_ . In other words, the only difference between _b_ and _a_ is that two variable values in the same basic group have been switched. A statistical capacity measure F satisfies _basic symmetry_ if F�a��F�b� whenever b is obtained from a by a basic switch. Notice that any nested counting index C satisfies basic symmetry because it has the same weight on every variable in the same basic group.

_<u>Monotonicity:</u>_ the main axiom for _F_ is an intuitive dominance axiom requiring the index value to reflect improvements in variables. We say that profile _b_ is obtained from profile _a_ by an _improvement_ if a� �b� for all _v_ , and a�b or, in other words, if profile a vector dominates profile b. A statistical capacity index F satisfies _monotonicity_ if F�a��F�b� whenever a is obtained from b by an improvement. This simple but significant requirement ensures that the index value rises whenever one variable rises from 0 to 1 and the rest of the variables do not fall in value. The 23
index C�a; w� satisfies this property since each w� is strictly positive. Notice that monotonicity supports the incentive compatibility criterion, since it ensures that a country is not penalized when it successfully raises its profile.

_<u>Subgroup decomposability: s</u>_ ubgroup axioms allow the index to be divided into salient subindices and linked back to the original index for policy analysis. In the present case, the main decomposition is over the basic groups given in partition P��p�, ... , p��. A statistical capacity index _F_ satisfies _basic decomposability_ if there exist weights ρ� �0 summing to 1 and sub-indices c��p�� such that In other words, there is a collection of indices, one for each basic group of variables, such that _C_ can be expressed as a weighted average of these basic indices. This is clearly the case for the nested counting index C�a; w�, as it is based on a weighted mean. Likewise, Equation (5) (after Proposition 1) follows from Equation (10) by aggregating across basic groups within each dimension, so that the overall index value is just the average of the dimensional index values. These decompositions can help inform why one country is doing better than another or help describe how a single country is progressing over time.

As noted above, the single country index C�a; w� can be expanded into an index C�A; w� that covers all countries in a region or even the universe of covered countries. The formula used to do this – Equation (7) – doubles as another form of decomposition that expresses the aggregate index and an average of the country indices. Since C�a; w� is the index of primary interest here, the equation will not be expressed as a formal property here. However, the fact that Equation (7) and 24
C�A; w� are available allows users to have a better understanding of regional levels and trends in statistical capacity.

# **4. Description of the SPI**

We apply the theoretical framework discussed above to develop a new Statistical Performance Indicator (and Index, or SPI). In this section, we discuss the features of the SPI, and compare it with the SCI. We also briefly discuss some data challenges with the construction of the SPI.

# **4.1. SPI and Its New Features**

As discussed earlier in Subsection 3.2.4, the SPI is built around four main dimensions: i) Methodology, Standards and Classifications, ii) Censuses and Surveys, iii) Availability of Key Indicators , and iv) Dissemination Practices and Openness. We further discuss each dimension, its indicators, and other practical implementation details in this section. More details on each indicator, its score, and its data sources are provided in Appendix 2.

Internationally accepted and recommended methodology, classifications, and standards provide the basis for NSSs to generate internationally relevant statistical indicators and facilitate data exchange and data integration. The first dimension (Methodology, Standards and Classifications) assesses whether NSSs have the necessary capacity to adopt and comply with international statistical standards. Twelve indicators are proposed for the first dimension to ensure that data from different sources can be compared, and to inform data users that the data are reliable and meet established technical standards. The setting and applying standards are usually regulated by law and typically the responsibility of the main national statistical agency.

25
The second dimension (Censuses and Surveys) reflects the availability and frequency of major censuses and surveys, which are designed to collect information mandated by the National Statistical Acts. This dimension comprises of eight indicators on population and housing census, agricultural census, business census, income and expenditure surveys, and other surveys on agriculture, health, labor force, and establishments.

The third dimension (Availability of Key Indicators) evaluates NSSs by reviewing the availability of the country data for the most recent year in international databases. In addition, the selected indicators produced by NSS should address the development concerns of countries, especially with the SDGs. The 12 indicators included in this category range from the headcount poverty rate to the under-five mortality rate, the primary school completion rate, the manufacturing value added in GDP, and net trade in goods and services. These indicators overlap to large extent with the Tier 1 SDGs, and include other standard macro-economic indicators as well, which are conceptually clear and regularly produced using established methodology and standards.

Providing data to the relevant international agencies demonstrates that an NSS’s practices meet the quality standards and data production timeliness. The last dimension (Dissemination Practices and Openness) is built on the principle that quality statistics should be delivered to the public in a timely, easily accessible manner and free of charge. It includes 10 indicators grouped under two sub-sections: Dissemination Capacity of NSO and Openness of Data.

These four dimensions are closely linked and capture the production cycle of NSSs in collecting, producing, and disseminating high-quality statistics. By following internationally recommended standards and classifications, an NSS can produce data of good quality that are both comparable within the country over time and across different countries. By combining administrative sources and timely censuses and surveys, an NSS can collect, process, and generate 26
data products covering different aspects of households and establishments. Finally, an NSS can disseminate these data products through its official websites, regular publications, and by submitting them to relevant international organizations, which further strengthen data transparency and quality.

Table 2 provides a summary of the data sources of the different indicators that comprise the SPI. Since we use data from more than one source to construct an indicator, the numbers shown in this table are the maximum numbers of indicators each data source contributes to. Table 2 shows that the indicators produced by the World Bank account for more than half of all the indicators, followed by the indicators produced by the IMF and the NSOs; other UN agencies contribute the remaining indicators.

# **4.2. Comparing the SPI and the SCI**

The SPI has several advantages over the SCI, particularly in terms of data coverage. Table 3 shows that SPI has: i) Richer and more comprehensive dimensions covering different data aspects ranging from data generation, curation, and dissemination to data analysis.

ii) More indicators: the SPI has 42 indicators (of which 39 are used for scoring), versus 25 indicators in the SCIs.

iii) More countries: the SPI covers more than 200 countries, especially including high-income countries, while the SCI covers fewer than 150 countries and includes no high-income countries.

More importantly, the SPI is built on the conceptual and theoretical framework laid out above (Section 3), while the theoretical principles of the SCI are not clearly formulated. The SPI satisfies all the desiderata of a statistical capacity index: simple, coherent, motivated, rigorous, 27
implementable, replicable, and incentive compatible. Furthermore, from an institutional viewpoint, the SPI is designed to be better aligned with other capacity assessment tools. For example, it is consistent with the Systematic Country Diagnostics (SCD), a new assessment tool that the World Bank recently developed to identify a country’s priority development areas.

On the other hand, the increased numbers of indicators and countries covered for the SPI require more data compared to the SCI. The data for the SPI come from different sources, ranging from established databases with international organizations to national statistical agencies’ websites. Given this heterogeneity of data sources, some data inconsistency may be expected. The complexities with the former are not just simply linearly related to the increase in quantities, but also concern data quality challenges such as missing data. Missing indicators for less-developed countries can represent a quite different data situation from missing indicators for richer countries. While the former may suggest weak statistical capacity, the latter might be explained by the little need of tracking certain indicators. A specific example is child stunting. In the SPI database, we assign maximum scores to richer countries (including the OECD and the EU) for this indicator assuming that richer countries have achieved the desired target of reducing child stunting and provide the benchmark against which progress for developing countries can be measured.

# **5. Some Empirical Illustrations**

# **5.1. Data**

We have constructed the database for the SPI for 2016. Efforts are underway to construct indicators for the SPI after 2016, as well as going backward several years for better comparison. We provide below some empirical illustrations using the available SPI data and various other data sources including the World Bank’s World Development Indicators database (World Bank, 28
2018b), MIT’s Economic Complexity Index (see, e.g., Hidalgo and Hausmann (2009)), and UNICEF’s SDG indicators for child welfare (UNICEF, 2018).

# **5.2. Empirical Illustrations**

We discuss in this subsection the relationship between the SPI scores and some major characteristics of a country such as its GDP, population size, and economic complexity. We subsequently offer a decomposition of the SPI by region, before providing some further empirical comparison between the SPI and the SCI.

# _SPI and Country Characteristics_

We start first with examining the relationship between the SPI and a country’s log GDP per capita (Figure 1). We expect richer countries to have better statistical systems because they have more resources to allocate for statistical activities, and also because they tend to have more complex and diversified economies that require more data. Indeed, there is a statistically significant and positive correlation between a country’s SPI and its income level. Figure 1 suggests that a 10 percent increase in a country’s GDP per capita is associated with approximately a 0.7 percentage point increase in its SPI score (see the box inside the figure).<sup>24</sup> As discussed earlier, a country’s statistical system produces statistics that reflect the socioeconomic conditions of the nation. As such, the more complex (advanced) a country’s economic level is, the more likely that its NSS is more developed. Indeed, besides an inherently stronger demand for the NSS to keep track of its various economic activities, a richer country likely has

> 24 Alternatively, moving up an income category (as defined by the World Bank (2018a)) can see a country improve its SPI score by 4.9 percentage points (Figure 1.2, Appendix 1).

29
more resources to invest in its NSS. To examine this hypothesis, we plot in Figure 2 a country’s SPI against its Economic Complexity Index (ECI). This figure indicates a strongly positive relationship between the two indexes, with a 0.1 increase in a country’s ECI being associated with a 1.5 percentage point increase in its SPI score.<sup>25</sup>

# _A Regional Decomposition of SPI Scores_

We turn next to decomposing the SPI score by region in Figure 3. This figure demonstrates that among the six regions, Europe and Central Asia (ECA) and South Asia (SAS) perform above the global average, with the remaining four regions falling behind in the following decreasing order: Middle East and North Africa (MENA), East Asia and Pacific (EAP), Latin American (LAC), and Sub-Saharan Africa (SSA). Notably, ECA stands out from the other regions, which have a more similar score: the difference between its score and that of the second-strongest region SAS is 15 percentage points, which is more than the difference between the latter region’s score and that of the weakest-performing region SSA.

It can be useful to further examine whether the patterns seen in Figure 3 apply to scores in all four dimensions. Disaggregating the regional scores further by the four dimensions can help us disentangle which dimensions drive these differences and can be improved. Estimation results are shown in Figure 4, which offers several interesting observations. First, a country’s score on a certain dimension can be quite different from that of its overall SPI score. In particular, all dimensions—except for the DPO dimension—have a ranking order for country performance that is different from that with the overall score. An interesting example is the ECA region, which is consistently the best performer on all dimensions but AKI, where the SAS region is now the best

> 25 For comparison, the US and the UK both have an ECI of 1.6, while the corresponding figure for China and Mexico is 0.9.

30
performer. On the other hand, SSA performs slightly better than the global average on the AKI dimension, but is the weakest performer on the rest. Second, countries generally perform best in the AKI dimension, where their average dimension score is 77 percent, which is almost the corresponding figure for the other dimensions.

_Further Comparison of the SPI versus the SCI_ We turn next to comparing the SPI and the SCI for 2016, when data for both indexes are available. We restrict our comparison sample to the 146 countries covered by both indexes. Estimation results suggest that richer countries, or countries with a more complex economy tend to have higher SPI scores. Indeed, for this sample of countries, a 10 percent increase in a country’s GDP per capita is associated with approximately a 0.6 percentage point increase in its SPI score, twice the corresponding increase for the SCI. Similarly, a 0.1 increase in a country’s ECI being associated with a 1.5 percentage point increase in its SPI score but only a 1.1 percentage point increase in its SCI score. For further visual illustration, we plot in Figure 5 the SPI and SCI scores against log of GDP per capita (Panel A) and economic complexity index (Panel B). While we leave out the regression results shown in previous figures for lack of space, the slopes of the fitted lines for the SPI scores are clearly steeper than those for the SCI scores in both these graphs. Furthermore, when the standardized distributions of both indexes are plotted on the same graph (Figure 1.3, Appendix 1), the SCI has a shorter tail on the right, suggesting that it has a less full range than the SPI.<sup>26</sup>

> 26 Since the two indexes have different scales, we standardize the two indexes so that they both have mean 0 and standard deviation 1 for better comparison in Figure 1.3. In addition, this figure also indicates that the SPI appears more similar to the normal distribution than the SCI, which is confirmed by formal statistical tests. One possible reason

31
Unsurprisingly, the rankings of countries are different for the two indexes. A particularly interesting case is El Salvador, a lower-middle-income country in Latin America.<sup>27</sup> This country ranks as number 5 of 145 countries based on the SCI scores. This would place this country in a surprisingly higher position than many richer countries, including its regional upper-middleincome neighbor—Colombia—which is ranked at number 10. Yet, according to a recent evaluation for 44 indicators concerning children in the 2030 SDG agenda by UNICEF, El Salvador has inadequate data for tracking progress on more than half (i.e., 27) of these indicators. The same figure for Colombia is much lower at just 14 indicators (UNICEF, 2018). The SPI ranks El Salvador as 48 while the ranking of Colombia changes to 12.

Another example is Botswana, an upper-middle-income country in Africa. This country has an SCI rank of 122, near the bottom of the SCI scores. This places Botswana lower than Madagascar (number 89) and Zimbabwe (number 105), both low-income countries in Sub-Saharan Africa. Indeed, Botswana’s GDP per capita was $15,807 in 2011 PPP, which was respectively 11 times and 8 times higher than that of Madagascar and Zimbabwe (World Bank, 2018b). Furthermore, a recent study suggests that Botswana has accomplished both upward consumption mobility and pro-poor growth in the past decade, which reduced its headcount poverty rate to 18 percent (using the international poverty line of $1.9/ day in 2011 PPP dollars). The opposite story held Madagascar, where the country has undergone much downward mobility, which drove its poverty rate up to as high as 82 percent (Dang and Dabalen, in press). Zimbabwe, on the other hand, has a higher income level than Madagascar, but is observed to have implemented at most one household consumption survey during the period 1990-2012; the corresponding figure for Botswana is two

> is that, since the SPI is the average of a larger number of indicators than the SCI, it is more likely to have a normal distribution by the central limit theorem (see, e.g., Casella and Berger (2002)).

> 27 Unless otherwise noted, we use the World Bank’s (2018) income classification for all countries.

32
comparable surveys (Beegle et al., 2016). Perhaps these studies can provide supportive evidence to help understand why Botswana sees its SPI ranking go up significantly to number 52, while the corresponding figure for Madagascar falls to number 133, and that of Zimbabwe remains almost unchanged.

# **6. Conclusion**

In this paper we offer a systematic discussion of the construction of the Statistical Performance Index. Our theoretical and empirical methodology builds on the existing strengths of the widelyused Statistical Capacity Indicator of the World Bank. We present the conceptual and mathematical foundation behind this index, and significantly expand the number of indicators and countries covered by this index. We also provide some brief empirical illustrations of the updated index with global data in 2016.

This new index could be seen as the first step before more resource-intensive country-specific assessments to inform multi-year improvement plans. Our proposed framework is also flexible enough to allow for future revisions as the global data landscape evolves. For example, we can incorporate new indicators such as whether an NSO uses cloud computing to store their data or implements household panel surveys in the relevant dimensions without creating major changes to the total scores. Our framework may also be relevant to the construction of other indexes in related areas, such as tracking the global SDGs or child development. Since the SPI is currently available only for 2016, another promising direction is to collect time series data for the new index, both going forward and for several years past, and expand the current framework to allow for dynamic changes over time.

33