metadata: authors: 0: author: Dhungana,Sandesh
count: Nigeria
docna: 0: docna: Identifying and evaluating large scale
            policy interventions : what questions can we answer ?
docty: Policy Research Working Paper
subtopic: Consumption,Economic Theory & Research,Fiscal & Monetary Policy,Economic Conditions and Volatility,Industrial Economics,International Trade and Trade Rules,Economic Growth,Inequality
teratopic: Macroeconomics and Economic Growth,Governance,Poverty Reduction
abstracts: cdata!: Using a data-driven empirical case study
            method, the paper evaluates the impact of one identified
            reform program on development outcomes. The paper uses the
            World Bank Country Policy and Institutional Assessment
            Ratings to identify large scale structural and macro-level
            policy interventions in the last decade that were seen as
            being sustained and successful for IDA countries. Robustness
            checks are performed to show the efficacy of the method in
            particular cases. It was found that the method attains
            robustness in the case of Nigeria.
geo_reg_key: 1927320,517191
repnme: repnme: Identifying and evaluating large scale
            policy interventions : what questions can we answer ?
display_title: Identifying and evaluating large
            scale policy interventions : what questions can we answer
listing_relative_url: /research/2011/12/15558457/identifying-evaluating-large-scale-policy-interventions-questions-can-answer
topicv3: Macroeconomic Vulnerability and Debt
            m1326057 1104,Economic Growth m1326241 471,Poverty m1326687 695,Trade m1326924 698,Macroeconomic and Structural Policies
            m1327380 439
publishtoextweb_dt: 2011-12-01T00:00:00Z
historic_topic: Macroeconomics and Economic Growth,Governance,Poverty Reduction
no_of_pages: 28
author_keys: 0: author_key: 1160901
model_extractions: WPS5918 Policy Research Working Paper 5918 Identifying and Evaluating Large Scale Policy Interventions What Questions Can We Answer?

_Sandesh Dhungana_ The World Bank Africa Region Poverty Reduction and Economic Management Unit December 2011
Policy Research Working Paper 5918

# **Abstract**

Using a data-driven empirical case study method, the paper evaluates the impact of one identified reform program on development outcomes. The paper uses the World Bank Country Policy and Institutional Assessment Ratings to identify large scale structural and macro-level policy interventions in the last decade that were seen as being sustained and successful for IDA countries. Robustness checks are performed to show the efficacy of the method in particular cases. It was found that the method attains robustness in the case of Nigeria.

This paper is a product of the Poverty Reduction and Economic Management Network, Africa Region It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at sandeshd@gmail.com.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Identifying and Evaluating Large Scale Policy Interventions: What questions can we answer?

# Sandesh Dhungana<sup>1</sup>

JEL Classification: O50, O11, O22; Keywords: Reform evaluation, Case-studies, Synthetic Controls 1 Author is affiliated to John F. Kennedy School of Government at Harvard University
# **Contents**

|**1**|**Intr**|**oduction**|**3**|
|---|---|---|---|
|**2**|**Ide**|**ntifying Successful Interventions**|**5**|
||2.1|How to identify large interventions?<br>. . . . . . . . . . . . . . . . . . . .|. . . . .<br>6|
||2.2|CPIA History and Methodology . . . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>6|
||2.3|Strengths of CPIA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>7|
|**3**|**Me**|**thodology**|**9**|
||3.1|Regression Methods with Dynamic Panel Data Models . . . . . . . . . .|. . . . .<br>9|
||3.2|Comparative Case Studies with Synthetic Control Methods . . . . . . .|. . . . .<br>10|
|||3.2.1<br>Motivating Model: Technical Details . . . . . . . . . . . . . . . .|. . . . .<br>11|
||3.3|Feasibility of Synthentic Control Case Studies . . . . . . . . . . . . . . .|. . . . .<br>12|
||3.4|Intervention Selection Methodology . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>13|
||3.5|Factor Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>14|
|**4**|**Nig**|**eria’s March towards Stronger Economic Management**|**16**|
||4.1|Nigeria before the Reforms<br>. . . . . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>16|
||4.2|Nigeria’s Reforms since 2003<br>. . . . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>17|
||4.3|Nigeria’s Growth Performance post-reforms . . . . . . . . . . . . . . . .|. . . . .<br>17|
||4.4|Running Synthetic Controls . . . . . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>18|
||4.5|Placebo Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>19|
||4.6|Robustness Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>20|
||4.7|Beyond Average Income Per Capita . . . . . . . . . . . . . . . . . . . . .|. . . . .<br>21|
|**5**|**Con**|**clusions**|**22**|
|**Li**|**st of **|**Figures**|**25**|
|**Li**|**st of **|**Tables**|**26**|

2
## **Chapter 1**

# **Introduction**

The current literature on empirical development economics contains a vigorous debate as regards the variety of approaches to understanding the development paradigm. Economic and regulatory policies, human and physical resources, and governance are all seen as drivers of economic development and yet there is no agreement on the exact roles each play in driving development. The fundamental puzzle retains two questions: Do the quality of one of these factors matter more for sustained economic growth and development? Is it the case that their relative importance is homogenous across countries and stages of development?

One strand of development economics contends that a combination of a stable macro-economy with market and investment climate-friendly set of economic policies is the necessary and sufficient condition for economic growth, which then drives other concepts of development, such as living standards and health outcomes. This view is associated with the Washington Consensus, which was the dominant paradigm of the 1980s and 1990s. It is under this consensus that multilateral lending agencies advised successive developing country governments on the need to fix budgets, cut inflation, and watch the economy take off with the dynamism of its private businesses. The experience of various countries, especially in Latin America that followed structural reforms to heart and yet ended up with sub-par growth performance in the 1990s however was a crude blow to this approach. This prompted even the World Bank, one of the proponents of the Washington Consensus to focus beyond stable macro-economic conditions and business-friendly reforms, as it did on its report on the growth experience of the 1990s.

A rival strand, which goes back to Gerschenkron’s theories on backwardness, believes in the context-specific and more subtle nature of development. Recent proponents contend that institutions matter, and well-governed countries are likely to grow faster, be more effective in alleviating poverty, and are also likely to be more successful in creating equitable growth and redistribution. A radical strand of this approach is one taken by Rodrik, Hausmann and Velasco, as the authors deviate significantly from a model of growth that specifies a nature of homogeneity in reforms to catalyze growth. Their growth diagnostics approach attempts to identity binding constraints, which are notions of the most important areas where reforms can bring the largest and quickest benefits. In this framework, improving trade barriers can either strongly accelerate the growth process; or have no impact on growth trends at all, depending on the need of the economy at the hour. The binding constraint of an economy hence dictates how successful particular reforms are.

Empirical work on development macroeconomics has not been able to converge on one strand or another. Initial theoretical and empirical work into growth accounting delivered results that both trends in total factor productivity and capital accumulation (physical and human) have significant roles in driving growth, while productivity trends appear to be more important. These results however move us no further in the above questions, because various economic policies, governance efficiency and institutional quality all affect trends in productivity and capital accumulation. Without a strong macro-economic environment, it is hard to create significant private 3
investment. Rampant corruption mostly goes along with weak incentives for firms to compete and hence to lower productivity growth. We can think of numerous other examples, but the point remains; the question of relative impact of reforms in one dimension against another is still very much left unanswered.

A large part of the problem is the inability of empirical macro-research to converge on methods and techniques that can attribute causal effects of interventions on outcomes. Traditionally, OLS regression methods have been used on country cross-section data to study the links between macro policy/governance interventions and outcome variables. These methods have been criticized for the lack of accuracy emanating from problems of omitted variable bias and endogeneity, model uncertainty and heterogeneity. Researchers have attempted to solve some of these problems by using instrumental variables approach, for example Acemoglu et al. (2001). These studies are important steps towards estimating causal inference in cross-country studies and yet problems remain. Strong external instruments are hard to find, and even if they exist, the effect obtained from such regressions are limited to the variation that is affected by the instrument. These problems make the instrumental variables approach limited in its ability to evaluate macro interventions. This has led some researchers to use dynamic panel data methods, which account for endogeneity issues and yet other problems remain, such as the instability of results from cross-country regressions when variables, time periods and countries in the data change.

Our goal in this paper is not to attempt to resolve the two aforementioned questions in their entirety. Such an endeavor requires a much larger and longer research agenda. We merely attempt to identify specific cases of reforms in the disparate dimensions, ranging from macroeconomic and debt dynamics to governance reforms in the last decade, and attempt to assess their impact on development indicators such as economic growth. We do this by using recently developed methods of inference, which help us attribute with larger certainty the impact on outcome variables to the interventions themselves. By selecting case studies where changes in one dimension were not accompanied by changes in other dimensions, the paper provides a window of understanding into whether reforms in certain dimensions affected development outcomes by themselves or not. We also perform robustness checks to see whether the results of these methods are sensitive to minor modeling changes. We believe that by building a larger catalogue of such results, we can attempt to start understanding the questions highlighted above.

In order to complete this task, we need to first identify significant large scale interventions in policy and governance dimensions. Chapter II of the paper explains our methodology on how to identify and measure large recent interventions on policy and governance dimensions. Chapter III explains the methodology we use to understand how successful these interventions have been in contributing towards development outcomes. We describe the results of the particular case studies in Chapters IV and V.

4
## **Chapter 2**

# **Identifying Successful Interventions**

Before embarking on the identification and measurement of interventions on the dimensions of economic policy, institutions and governance, we attempt to define each of the terms clearly. The definition of institutions has always been fraught with large problems. The early definition by Douglas North was very broad, encompassing all "rules of the game" in a society. (North, 1991) Acemoglu characterizes the structure of property rights and the presence and perfection of markets as the crucial economic institutions in a society. (Acemoglu, Chapter 6) Unlike geography, economic and social institutions are endogenously determined by societies in the long run. Acemoglu also posits the idea that the inability to clearly compare across institutional frameworks has meant that the development literature focuses more on proximate determinants of economic growth, which include policies.

Public policies, on the other hand, are rules set up, and discretionary actions taken within the institutional structure to manage society, including its economy. These include rules on fiscal budgets, exchange rate management, taxation, and in numerous other arenas. They also include discretionary actions, for example ones taken during recessions to expand money supply, or actions taken during wars to ration food and so forth. It is easy to see that the structure of economic, political and social institutions affect the choice of policies. Policies are also much more volatile than institutions, and provide much larger variation in the data. Hence, empirical literature tends to focus on the impact of policy differences on economic performance.

Governance is another concept that is harder to grasp at. The World Bank, in 2007, defined governance as "the manner in which public officials and institutions acquire and exercise the authority to shape public policy and provide goods and services." This suggests the distinction between policies and governance lies in the fact that policies are the rules and discretionary actions, while governance is the concept of how well such rules and actions are shaped and implemented. Examples of governance quality include the quality of budgetary management, accountability, corruption control among others.

When a policy intervention takes place, the success of such an intervention depends not only on the choice of the policy itself, but also on whether the governance quality in the particular case is strong enough to allow the policy to work to its effect. The underlying effect of institutions always remains, as both the choice of the policy and the quality of governance is shaped by the institutional frameworks that characterize a society.

We believe that there has been much work on the importance of institutions in causing long run differences in economic growth. (Acemoglu, Johnson, Robinson, 2001) Institutions can also only be studied with an emphasis on the very long run. We thus focus our efforts on the evaluation of interventions on the policy and governance dimensions, while cognizant of the fact that institutional underpinnings are always affecting any results we come up with.

5
In the first two sections of the paper, we use the term ’intervention’ to generalize across interventions in all of the dimensions we are interested in. It is only when we get down to the specific case studies that we distinguish between interventions in various dimensions.

# **2.1 How to identify large interventions?**

Understanding the impact of interventions on outcomes such as levels of income or corruption is a difficult and yet important task. The first step in fulfilling such a task accurately is the specification of variables that identify interventions on a large scale. This is no trivial task, as most measures used in the literature fail to make the distinction between intervention inputs and intervention outputs.

When a government launches an anti-corruption drive, or changes its exchange rate management strategy, these are intervention inputs. These intervention inputs are then mapped onto intervention outputs, which are, for example, the levels of corruption after the launch of the anti-corruption drive. This mapping depends not only on the choice of the right input, but also on random shocks that can map bad intervention inputs into great outputs, or vice versa. This is the central reason why the measurement of intervention inputs is so critical to understand the effect of interventions on outcomes such as income levels and growth.

On a separate note, the identification and measurement of intervention, however, needs to also avoid looking at de jure codification of laws on policy and institutional matters. A law that codifies bribery as illegal is not enough to suggest that the government has implemented an anticorruption program. Ratings that meet these criteria should not be measuring legal statutes that do not hold.

We believe that the Country Policy and Institutional Assessment (CPIA) ratings provided by the World Bank for the IDA borrower countries meets both the criteria described earlier. Hence, we use it to identify national-level reforms and thus understand the links between large scale interventions and development outcomes. We note that this restricts our sample to International Development Association (IDA) countries only.

# **2.2 CPIA History and Methodology**

Each year, the World Bank undertakes an assessment of a country’s policies and institutions, also referred to as the Country Policy and Institutional Assessment (CPIA). The CPIA assesses the quality of a country’s present policy and institutional framework. "Quality" refers to how conducive that framework is to fostering poverty reduction, sustainable growth, and the effective use of development assistance.

The Bank initiated country assessments in the late 1970s to help guide the allocation of IDA lending resources. The CPIA consists of a set of criteria representing the different policy and institutional dimensions of an effective poverty reduction and growth strategy. The criteria have evolved over time, reflecting lessons learned and mirroring the evolution of the development paradigm. In 1998, the criteria were substantially revised: coverage was expanded to include governance and social policies, the number of criteria was set at 20 (where it remained until 2004), and the ratings scale was changed from a 5 to a 6 point scale. In the context of the Board deliberations on CPIA disclosure, in early 2004, the Bank assembled an external Panel to review the ratings and methodology. This Panel concluded that the CPIA criteria focus on the right set of issues and produce robust results, but it also found unnecessary overlap in some of the criteria and outlined steps to address some methodological and process issues. This Panel broadly supported the CPIA practice of rating implemented rather than intended policy actions, and it strongly favored disclosure of the ratings for IDA-eligible countries.

6
<!-- Start of picture text --> A. Economic Management<br>1. Macroeconomic Management<br>2. Fiscal Policy<br>3. Debt Policy<br>B. Structural Policies<br>4. Trade<br>5. Financial Sector<br>6 Business Regulatory Environment<br>C. Policies for Social Inclusion/Equity<br>7. Gender Equality<br>8. Equity of Public Resource Use<br>9. Building Human Resources<br>10. Social Protection and Labor<br>11. Policies and Institutions for Environmental Sustainability<br>D. Public Sector Management and Institutions<br>12. Property Rights and Rule-based Governance<br>13. Quality of Budgetary and Financial Management<br>14. Efficiency of Revenue Mobilization<br>15. Quality of Public Administration<br>16. Transparency, Accountability, and Corruption in the Public Sector<br><!-- End of picture text -->
analysts at the Bank are explicitly asked to look at evidence stronger than legal codification when assessing the CPIA ratings. Given that the process of rating involves constant back and forth between various groups inside the organization, we are comfortable with the assessment that the ratings measure more than de jure policy statutes.

On the flip side, a ratings system that rates intervention inputs cannot, by definition, have the exact same mapping as intervention outputs. For example, even with a strong past system of public finances, a random shock such as a tsunami or an earthquake can ruin public finances. If the ratings system is based mechanistically on the annual outcome of the state of public finances, the ratings will decrease precipitously, although the policy environment has not worsened, and finances themselves could recover quickly. The CPIA ratings system describes itself as set up in a way to avoid this problem as well. It is much harder to assess this claim however, because analysts would find information that is separated from outcomes very hard to find.

There is a safeguard even with such a problem. A 2007 report by the World Bank on the CPIA distinguishes between naïve and subtle endogeneity. If analysts look at intervention outcomes while informing themselves of the strength of policy/institutions, such endogeneity is subtle and harder to avoid and test. However, the second form of endogeneity, the case where analysts change ratings blindly in response to large and exogenous shocks, can be tested. The study finds that weather and terms of trade shocks did not significantly impact the ratings. Hence, we can at least be confident that naïve endogeneity is not a significant problem with the CPIA ratings system.

8
## **Chapter 3**

# **Methodology**

We now turn to the discussion of methods which can help estimate the actual impact of interventions on development outcomes.

The CPIA ratings can be used to empirically assess the effect of particular changes in economic policy or governance regimes on development outcomes of interest. The methodological problems are however significant, as policy outcomes get determined in an environment embedded by social, political institutions and governance quality. Research can thus pursue two separate methodological avenues, which we describe and assess.

### **3.1 Regression Methods with Dynamic Panel Data Models**

The first method uses classical regressions with dynamic panel data models which can estimate the average effect of particular interventions on development outcomes. Given that OLS and fixed effects estimators are biased due to unobserved time-invariant country-specific effects, and the availability of external instruments that meet exclusion restrictions is weak, these estimators have become central to the empirical growth literature. Bond et al. (2001) use the system GMM estimators to account for the endogeneity of variables such as investment, and to remove time invariant There are some critical problems associated with the use of these methods in our context. Firstly, there is no convergence in the literature on the functional form for capturing differential effects of particular policy interventions. Neither the augmented Solow/Ramsey nor the endogenous growth models that are workhorses of modern macroeconomics really have institutional structures as endogenous parameters in the models. Till date, the role of institutions and even most policy interventions is mostly seen in improving allocative efficiency, and hence increasing total factor productivity, which is generally calculated as the residual from the regressions that are based on the workhorse models. This creates a problem, as empirical papers generally use reduced form equations and lump various variables of policy and institutional quality on the right hand side of the regression equations. Bhagwati and Srinivasan (1999) argue that cross-country regressions are hence not the correct way to understand the impact of reforms (such as trade liberalization) on growth outcomes, because they are not based on any strong theoretical models, and have no description of the functional forms of the relationships between the dependent and independent variables. Using more advanced methods such as system GMM estimators cannot solve these problems.

Also significantly, these models use cross-country variation in growth outcomes as predicted by policy changes to estimate the average effect of a particular policy/governance change on growth outcomes. This, however, assumes that the relative importance of particular policy/governance inputs is similar across the diversity of countries in the sample group. Here, outliers become 9
problematic, as really weak (or really strong) interventions are the ones that affect estimates the most, thus biasing the average effects to outliers.

Authors like Bhagwati and Srinivasan (1999) and Rodrik (2005) also argue that regression results are dependent on period, country sample and variables chosen. This suggests a lack of stability in the parameters estimated through the regressions. Levine and Renelt (1992) use extremebounds analysis to show that the results from cross country regressions are sensitive to modeling design and hence fragile. This problem points to a larger issue, which is related to a debate on how the growth paradigm itself is imagined. If growth occurs through a certain process, which is catalyzed by certain policy steps and institutional changes, and this process retains some structural similarity across a myriad of cross-country and across-time experiences, then a regression approach retains its viability. However, recent literature focuses on a highly contextspecific idea of development needs. For example, Hausmann et. al. argue that the binding constraint of an economy might dictate how successful particular policy interventions are. In such a situation, improving trade barriers can either strongly accelerate the growth process; or have no impact on growth trends at all, depending on the need of the economy at the hour. A cross-country regression basically flattens out this useful variation and provides us with an average effect, which in this case, could describe something that is irrelevant to any one country’s development needs. We believe this considerable list of problems is enough for us to eschew the use of cross-country regressions in assessing the impact of large scale interventions. We thus choose not to follow the system GMM approach, while noting of its ability in avoiding endogeneity problems that OLS regressions face.

# **3.2 Comparative Case Studies with Synthetic Control Methods**

Current literature on evaluation of interventions, until very recently, focused significantly on micro interventions, which are amenable to the use of Randomly Controlled Trials (RCTs). This approach has yielded many significant results, as well-designed and well-run experiments allow the researcher to interpret the impact of interventions with a high certainty about causality. This literature, which perhaps began with Kremer and Miguel’s pioneering work on de-worming, is now used extensively in understanding questions such as the impact of classroom cameras on teacher absenteeism, and the evaluation of free bed-net distribution programs. These developments are welcome, and yet the lack of opportunity in experimentation at the macro level makes true RCTs close to impossible in the macro realm. This literature has however helped economists focus on some important methodological questions and problems that are important to causal inference, whether the subject be a micro or macro intervention.

The search for methods that allow rigorous analysis on aggregate outcomes such as growth has led to considerable recent work in the comparative case-study literature. Card and Krueger (1994) use the differences-in-differences method to estimate the effect of policy changes, such as US State minimum wage laws on employment outcomes. The method of matching units on propensity scores is widely used in the micro-development literature. Bhagwati and Srinivasan (1999) point out the fact that comparative case studies can be likely avenues to further the path of empirical macro research, given the problems identified with cross-country regressions. In this tradition, the synthetic control method, started by Abadie and Gardeazabal (2003) uses an empirically rigorous design to estimate the effect of aggregate interventions.

Abadie and Gardeazabal (2003) and Abadie, Diamond and Heinmeuller (2007) are two empirical papers that use the synthetic control approach to understand the impact of large scale policy interventions on outcome variables. In their 2003 paper, Abadie and Gardeazabal use the approach to estimate the impact of the Basque separatist movement on growth trends of the state. The Abadie et. al. paper (2007) focuses on the estimation of tobacco sales trends in California after the passage of Proposition 99, which made it harder to sell tobacco. At the heart of the synthetic control method used in these papers is the idea that the effect of a particular interven10
tion can be empirically assessed only by comparison with the correct counterfactual, a concept of how the unit would have behaved had the intervention not happened. In difference-in-difference studies, the researcher is allowed the discretion of choosing the comparison group that mimics the counterfactual. This leads to substantive ambiguity on whether the comparison group really is an effective counterfactual. (Duflo et al. 2003) The synthetic control method resolves this problem.

The method first divides data into a pre and post intervention period, intervention in this case being a significant policy/governance intervention that is reflected in a strong change in the CPIA ratings. The method then creates a "synthetic" unit that resembles the treated unit (in this case the country that goes through the policy/governance intervention) in the pre-intervention period. This is achieved by using a weighted average of all other countries in the sample (donor pool as in the matching literature). The weights are set, unlike in the differences-in-differences approach, not by the researcher. Instead, the weights are chosen such that the synthetic country mimics the pre-intervention income per capita path and other observable characteristics of the country that undergoes the intervention. This method also safeguards against extrapolation that plagues regression models, as long as it is not used for countries with extreme values for observable characteristics.

After the synthetic unit is chosen, the method looks at the growth performance of the treated country against the performance of the synthetic country after the intervention. The gap between the two can be used to estimate the effect of particular changes in policy/governance on the evolution of growth outcomes of the particular country in question. Unlike in a panel regression framework, this is not asking the question of whether particular policy changes have, on average, positive/negative effects on growth across all countries. Hence, it avoids some of the problems of the first method. It also helps research lead to context-specific understanding of the effects of policy/governance changes on growth outcomes.

The largest benefit of the method however, is the fact that it can account for time variant country characteristics as long as there are enough pre-treatment periods and the pre-intervention match is strong. After evaluating both methods, we decide to use the synthetic control method, because of its strength in helping create a counterfactual that can more accurately mimic the treated unit, hence making causal inference more probable. Below, we motivate the method using a simple model as shown by Abadie et. al.

# **3.2.1 Motivating Model: Technical Details**

Abadie et. al. motivate the method by using a model, which we describe now. Let’s say we have J+1 countries in our dataset, and country 1 is the one that undergoes reforms. Let _Y_ it<sup>n</sup> be the level of income per capita for a country i at time t without reforms, and _Yit_ be the level of income per capita for country i at time t with reforms. Let’s say we have time periods _t_ = 1 _, ..., T_ 0 as the pre-reform periods, and _t_ = _T_ 0 + 1 _, ..., T_ as the post-reform periods. For the country which undergoes reforms, the level of income per capita can also be given by _Yit_ = _Yit_<sup>_n_+</sup><sup>_αitDit_,</sup> where _αit_ = _Yit − Yit_<sup>_n_;and</sup><sup>_Dit_= 1when</sup><sup>_t > T_0</sup><sup>_,_and0otherwise.Forcountrieswithoutreforms,</sup> _Yit_ = _Yit_<sup>_n_</sup> Now, we want to estimate how much faster/slower the country with reforms grew after the reforms, which is given by the set _α_ 1 _t_ = _α_ 1 _T_ 0+1 _,..._ 1 _T_ for cases where _t > T_ 0. We know from earlier that _α_ 1 _t_ = _Yit − Yit_<sup>_n_</sup> We can obviously observe the post-reform trend of the country with the reforms ( _Yit_ ), hence we need to estimate _Yit_<sup>_n_,whichistheincomepercapitatrendofthecountrywiththereformshad</sup> the reforms not happened. We use a linear factor model for this estimation. Now, let’s say that output per capita is determined by the following factor model: _Yit_<sup>_n_=</sup><sup>_αt_+</sup><sup>_θtZi_+</sup><sup>_λtμi_+</sup><sup>_εit_</sup> Here _Y_ it<sup>n</sup> is the income per capita of a certain country i, at time t. Now, _α_ t are the time-variant 11
but unit-invariant fixed effects that can be differenced out by taking time-dummies. _Zi_ are the observed variables that affect income per capita, while _μi_ are the unobserved variables that affect income per capita. _εit_ are the error terms with zero mean.

The synthetic controls method creates a control unit for each country by assigning a certain weight to each country in the donor pool. There can be many such combinations of weights, which is denoted by the vector _V_ = ( _w_ 2 _, w_ 3 _, ...., wJ_ +1) which includes many such synthetic controls.

Abadie et. al. show that if there are weights where the weighted income per capita before the intervention matches the real income per capita, and the weighted value of the observed covariates matches the real values of the covariates before the intervention, then the size of the bias in the difference between the post-intervention incomes for the real and the synthetic unit go to zero as pre-intervention period increases. They also show that this result holds when we have a close but imperfect match.

Mathematically, such weights are given by the set ( _w_ 2<sup>_∗, w_</sup> 3<sup>_∗, ...., w_</sup> _J_<sup>_∗_</sup> +1<sup>)where</sup> _J_ +1 � _j_ =2<sup>_w_</sup> _j_<sup>_∗Yj_1=</sup><sup>_Y_11</sup><sup>_,_�</sup><sup>_J_</sup> _j_ =2<sup>+1</sup><sup>_w_</sup> _j_<sup>_∗YjT_0=</sup><sup>_Y_1</sup><sup>_T_0</sup><sup>_,_and �</sup><sup>_J_</sup> _j_ =2<sup>_w_</sup> _j_<sup>_∗Zj_=</sup><sup>_Z_1holds.</sup> Abadie et. al. (2007) provide mathematically that in these cases, the biases created by time variant country characteristics tend towards zero, as long as the pre-intervention period is long enough, and the match between the pre-intervention trends are strong. We point the reader to Abadie et. al (2007) who prove this point mathematically in Appendix B to their paper.

Now, the key is to find weights such that the distance between the treated country and the synthetic control unit in the pre-reform period in terms of both GDP per capita and predictors of GDP per capita is as close to zero as possible. We use the Stata program provided by Abadie et. al. to find the best possible pre-reform matches for the country in each case study. For the method to yield accurate results, there are certain necessary conditions, which we describe next.

# **3.3 Feasibility of Synthentic Control Case Studies**

In order for an empirically rigorous case study that uses the synthetic controls method to effectively isolate the impact of certain interventions, the following broad criteria have to hold.

- Significant Intervention - The intervention has to be significant at the level of the unit of treatment, which, in this case, is the country itself. This might be an obvious point, but it is important. An anti-corruption program in a local municipality can be evaluated for its efficiency, but our goal with this research program is to identify and analyze the impact of various larger programs and policies. That can only be done if the intervention is done at the national level, or done at a local unit that makes up a large part of the national unit.

- Sustained Intervention - The intervention has to be sustained through a significant period of time. If the intervention is repealed after a year or two, then the possibility of it affecting variables of interest becomes lower, and the analysis cannot detect any effect.

- Absence of Significant Exclusive Shocks - The effect of the intervention can only be estimated accurately if the period of intervention was not characterized by significant shocks that affected the treated unit exclusively. In this case, let’s say for example that country X underwent a significant transparency reform program, while at the same time; a large civil war broke out inside the country. It is impossible to isolate the effects of the transparency program given that conflict affects production and allocation of resources. In a similar vein, global business cycle shocks are not problematic to this endeavor if they affect the treated unit and the donor pool similarly. This can be tested and the donor pool can be shrunk to meet this requirement as well.

- Non-Interference Across Units - The effect of the intervention cannot have an effect on the untreated unit. In this research project, an intervention cannot have a large effect on the

12
economy of other countries that did not go through the intervention. If this does not hold, then isolating the effect of the intervention becomes much more difficult.

- Significant Pre-Treatment Period - The synthetic control approach uses the variation in the pre-treatment period across the donor pool to create a synthetic control unit that mimics the treated unit pre-intervention. The larger the pre-treatment period, the smaller are the possible biases accruing due to unobservable time-variant unit characteristics.

- Convex Hull Criteria - For the approach to work, some weighted combination of untreated units must be close to mimicking the treated unit before the intervention. If a unit is very far from other units in terms of the observed variables of interest, then the synthetic control method cannot give accurate predictions. Given this, the countries at the extreme ends of the spectrum of observed characteristics are not ideal for this method.

- Donor Pool Criteria - The factor model that is used to estimate the synthetic control is a linear model. Given that this model might not hold strongly over disparate regions with a wide variety of characteristics, the donor pool may have to be restricted to regions with some similarity in observable characteristics such that the linear factor model can be a good approximation. Also, the donor pool needs to be restricted to states that did not have very significant changes in economic policy or institutional frameworks themselves.

As long as these conditions hold strongly, endogeneity of the intervention is not a problem. An example of endogeneity is when the intervention happens because of the level of the variable of interest, such as when an anti-corruption drive happens because corruption levels are high. However, in such a case, by definition, the levels of corruption in the synthetic control unit match with the levels in the treated unit before the drive. If the level of corruption led to the drive, then it would have done so in the case of the synthetic control unit as well. If there exist countries that had similar corruption levels (and other important characteristics), and did not have a corruption drive, then this suggests the effects of endogeneity are not what caused the intervention.

# **3.4 Intervention Selection Methodology**

Given these criteria, we use the following methodological process to select interventions. First, we select cases through a simple mathematical formula. Here, we are using the CPIA ratings to denote the changes in policy inputs and governance regimes. Hence, in order to meet condition of significant intervention, we select cases for whom at least one of the CPIA ratings changed by a value of more than 1 between 2004 and 2009. Given that average of the variance across all ratings is 0.45, we believe ratings changes greater than 1 are significant. Secondly, in order to meet condition of sustained intervention, we restrict the pool of countries by selecting countries where ratings changes were sustained for a period of 4 years or more.

Given that the cases need to meet the requirement of an absence of significant exclusive shocks, we avoid looking at countries that experienced significant exclusive shocks, including conflicts during the period. If an economy is significantly reforming both its constitutional structure and macro-economic policy at the same time, it becomes difficult to isolate the effects of each.

To be sure that we meet the convex hull criteria, we check where our country of interest ranks on various characteristics and omit countries that are on the highest and lowest 5% on various characteristics.

Finally, in order to look at really significant changes, we restrict the case study sample to countries that have large changes for cluster average ratings for the four CPIA clusters. Now, we are left with five countries, Georgia, Maldives, Nigeria, Pakistan and Chad. In Figure 3.1, we show the various ratings changes associated with each country’s experience in the past 7 years.

13
|Country<br>||Positive/Negative<br>|First<br>Yearof<br>|Feasible<br>forCase<br>|
|---|---|---|---|---|
|Georgia|RatingsChangeVariable<br>Trade Policy|Change<br>Positive|Change <br>2004|<br>_Study?<br>Yes|
||Structural Policy Cluster Average|Positive|||
||Debt Policy|Positive|||
||Business Regulatory Environment|Positive|||
|Maldives|Fiscal Policy|Negative|2004|No|
||Debt Policy|Negative|||
||Macroeconomic Management<br>Economic Management Cluster<br>Average|Negative<br>Negative|||
|Nigeria|Debt Policy|Positive|2004|Yes|
||Trade Policy|Positive|||
||Economic Management Cluster||||
||Average|Positive|||
|Pakistan|Macroeconomic Management|Negative|2005|No|
||Economic Management|Negative|||
|Chad|Macroeconomic Management|Negative|2004|No|
||Quality of Budgetary Management<br>Economic Management Cluster<br>Average|Negative<br>Negative|||
vation, including policy choices on taxation, property rights, research and development. Such weaknesses however are not critical to our research because we restrict our donor pool to countries that are similar to the country in question in the variables that are deemed to be important in determining growth outcomes, and yet are not included in the regression. Firstly, the donor pool only includes countries that are either part of an easily designated peer group, or at some designated distance away from the country in question on policy/institutional framework as measured by the CPIA ratings. We do this so as to avoid interpolating from countries that have much stronger/weaker policy/institutional frameworks. Since we are not using the CPIA ratings in the regression itself, this is an important step in helping us create a strong counterfactual. We also take out countries with populations much larger or smaller than the country in question, in order to avoid interpolation from very disparate units.

We believe in this approach because the variables that are included in the regression are used to create the synthetic control such that the control mimics the treated country well. By restricting the donor pool to countries with similar ratings on policy/institutional structure, we are controlling for the possibility of other omitted factors biasing our results. Now, assuming that we can match pre-treatment variables, including both the variables described below and the outcome variable for a significant pre-treatment period, we ought to be fairly confident of attributing the impact on growth outcomes to the interventions themselves.

In constructing the model itself, the first variable we use is the Private Investment to GDP ratio. Growth literature has focused heavily on the role of physical capital in adding to per capita income. The Solow model proscribes a strong role for economic convergence among nations. Although unconditional convergence is not borne out in reality, the idea that conditional convergence happens is an accepted one. We thus use initial GDP per capita (as of 1995) as a proxy for convergence effects.

Recent work on human capital suggests that the role of human capital accumulation in growth is very significant. Efforts to turn the abstract concept of human capital into an operational variable have been fraught with data limitations and much criticism. We opt for the best possible measure, which is the Barro-Lee indicator of the average years of secondary schooling for people above 15 years. We pick this indicator over the enrollment variable, gross % of secondary school enrollment among the population in the age range appropriate for such enrollment when it is available. This is because the growth process today stems from the abilities of the work force today, rather than from the enrollment of the labor force of tomorrow. We take into account the lines of criticism that there are nonlinearities in the returns to schooling, based on the year of schooling and the quality of the education. Even so, the lack of measures that take into account these issues forces us to use the admittedly imperfect human capital measure.

Finally, we include the population growth rate as well. Now, we are ready to run empirical case studies in order to assess Nigeria and Georgia’s reform performance. One final caveat is in order however. Even if the results of this analysis show that a country’s performance is not much stronger than the counterfactual unit, the takeaway is not that these reforms were unwarranted, or inadvisable. These reforms might be more important towards sustaining income growth than accelerating them, in which case, we might see no effect of reforms in the short post-intervention period. The analysis does however help us assess the performance of reforms to date in a clearer manner.

15
## **Chapter 4**

# **Nigeria’s March towards Stronger Economic Management**

### **4.1 Nigeria before the Reforms**

Nigeria has long been famous for its large development potential, rich natural resources and an ultimately tragic set of outcomes due to mis-governance and conflicts. With a population of more than 100 million, and proven oil reserves of 36.2 billion barrels, it appears to have the strongest economic potential of all African States. However, in 2000, Nigeria had a GNP per capita of US$300, lower than the per capita income during 1960, the year of independence. In 2000, 66% of the population lived in below the dollar-a-day poverty line. It is in this context that General Abacha died in 1998 and a democratically elected government assumed power in May 1999.

During the post-independence period, Nigeria had suffered through chronic economic mismanagement and corruption. The discovery of petroleum had manifested itself as a tragic resource curse by obliterating non-oil exports as well as creating rent-seeking governance behavior on the part of the political elite. In the 1990s, the World Bank lamented that stronger price of oil had continuously led to wasteful spending by the government. When the oil price fell in 1998-99, lack of oil receipts led to a threat of current account crisis. As late as 2002, the IMF was stating that 2001-02 were marked by large macro imbalances, with a large fiscal and current account deficit. Large government spending in the face of oil price booms was seen as the culprit for such fiscal mismanagement.

Monetary policy was tightened in 2002, which stemmed the tide of inflation. The 2002 IMF consultation document states that the government had started controlling the budget in 2002, and yet further improvement was warranted. By September 2002, foreign exchange reserves had fallen to around $7 billion. During this period, the IMF noted that progress on structural reform, including on trade barriers had not moved ahead. It is also notable the CPIA ratings for debt policy in Nigeria during the period of 1999-2001 were the weakest, with Nigeria getting a 1, the lowest rating score for external debt management.

Poverty in Nigeria had always been a major concern, and the 2003/04 living standards measurement survey showed that 54% of the population was living below the 1 dollar a day poverty line.

16
# **4.2 Nigeria’s Reforms since 2003**

Nigeria consistently improved its ratings for the economic management and structural policies clusters between 2003 and 2009. Nigeria’s debt policy rating rose by 2 points from 2003 to 2009. Similarly, its trade policy rating rose by 1.5 points as well. The average rating for its economic management cluster went up by more than a point, while the average rating for structural policies rose by a point as well. The social inclusion and public governance clusters, however, have not seen any significant ratings changes.

Nigeria came up with the NEEDS (the National Economic Empowerment and Development Strategy) in 2004. This included reforms such as passage of a fiscal responsibility bill. Also introduced was the idea of a reference oil price, which was to guide fiscal decisions. The IMF saw NEEDS as a major first step towards a credible agenda of growth and poverty reduction. In the 2005 report, the IMF and IDA staff saw implantation of the strategy in 2004-05 as "impressive". In the 2007 progress report, the IMF staff state that macroeconomic policies improved, while there was still worry regarding fiscal expansion and quality of budget spending.

On other fronts, the NEEDS strategy included reforms on financial sector, transport system, accountability and transparency. The passing of the bill on Commitment to Extractive Industries Transparency Initiative, and the start of civil society involvement through the "Publish what you Pay" initiative meant that oil revenue data would be collected and published. Further promises on health, education and poverty eradication were made as well. That said however, the CPIA ratings show that WB analysts did not believe significant progress in these sectors were made, unlike in economic management and structural policies cluster.

The financial sector reforms in 2004 also forced consolidation in the banking sector by raising the minimum capital requirements by a ten-fold. This led to significant banking mergers and consolidation. The government also replaced a pay-as-you-go system with a fully funded pension system. After these reforms, domestic credit to GDP ratios have risen from around 13% in 20022006 to above 25% in 2007 and later. This large rise in financial deepening is impressive.

# **4.3 Nigeria’s Growth Performance post-reforms**

Nigeria’s GDP per capita (PPP at constant prices) in 2004 was 1687, which rose by 15.7% to 2000.78 in five years. Generally, this would be attributed to the effect of oil. Worldwide oil prices, as measured weekly weighted spot price, have risen from 20 dollars per barrel in early 2003 to 71.75 dollars per barrel at the end of 2009. This has led to a sharp rise in Nigeria’s state expenditures and these expenditures have been distributed, albeit imperfectly to parts of the population, thus raising mean income. This narrative however does not tell Nigeria’s story completely. While oil prices surged between 2003 and 2009, Nigeria’s Real non-oil GDP actually rose by 13.2% in 2004, by 8.6% in 2005, and by an average of 9.3% during 2003-2007.

Whether the additional spending created by revenue windfall through oil spurred the rest of the economy is difficult to answer, but the data reveals that government expenditure to GDP ratio has gone down since 2001. In 2001, the ratio was 51%. As oil prices jumped in 2003 and kept on rising for the rest of the decade, the ratio declined to around 30% between 2003 and 2007. The fiscal rule has been applied well, as the excess crude account that houses excess oil revenue amounted to $18 billion in late 2008. Given that the majority of the oil revenue goes to the government and government expenditures have been tightened, it is hard to suggest that most of the economic growth indirectly from oil revenue.

If oil is not the only driver of growth, we need to investigate what is. Analysts believe that much of this growth came from accumulation rather than productivity increases. Interestingly, agriculture contributed around 40-45% of the growth during the 2003-2008 period. The impact of private remittances has been significant in the second half the decade. Remittances have jumped from $1.06 billion in 2003 to $9.98 billion in 2008, which amounts to around 12% of real 17
GDP in 2008. While remittances are rarely seen as manifestations of successful policy actions (often they are ways in which people in countries with large conflicts or weak policy fight for the economic survival of their families), the impressive performance in FDI figures tell a different story. FDI rose from $2.85 billion in 2003 to $6.42 billion in 2007. These figures suggest that foreign investment into Nigeria had increased significantly, thus strengthening the notion that the business environment, at least for foreign investors, has become friendlier.

Although this growth performance looks strong at face value, we need to create a suitable comparator unit in order to really understand whether Nigeria’s growth record is extraordinary or not. The critical question at hand is; had the reforms during 2001-2004 not happened, how would Nigeria have grown till date? We attempt this exercise next.

# **4.4 Running Synthetic Controls**

The synthetic controls procedure for Nigeria creates a synthetic control where countries with a positive weight on the control are Bolivia, Tajikistan and Zambia. Table 4.1 shows the nations with their weights.

Table 4.1: Specific Country Weights in Nigeria’s Synthetic Control

|Country|Weight|
|---|---|
|Bolivia|0.18|
|Tajikistan|0.055|
|Zambia|0.764|

The method works well in creating a synthetic unit that matches Nigeria in its pre-GDP trends before between 1995 and 2004. The RMSPE is 12.53, and as Table 4.2 shows, the characteristics of the synthetic unit are very similar to Nigeria between 1995 and 2003. There is one caveat. The investment rate differs between the synthetic and the real unit. When we look qualitatively at the two countries that have significant weights in the synthetic control unit, we can see that both Zambia and Bolivia are economies with strong natural resource bases. Zambia is well know for its copper exports, and the price of copper has increased in the post-intervention period, similar to the price of oil. Similarly, Bolivia is well known for natural gas, and energy prices have risen all around. As a further check, none of the two countries have had conflicts. While this period has coincided with Evo Morales’ rule in Bolivia, the economy has hummed along due to the rising price of commodities in the international market. Finally neither of the two countries are Nigeria’s largest trading partners, which helps us believe in the non-interference assumption.

<u>Table 4.2: Predictor Balance</u>

|Variables|Treated|Synthetic|
|---|---|---|
|Population Growth Rate|2.44|2.50|
|Secondary School Enrollment Ratio|26.15|36.62|
|Investment Rate|2.79|11.41|
|Initial GDP per capita|1414.9|1404.9|

Figure 4.1 below shows the GDP per capita trends for Nigeria and the Synthetic control unit. The synthetic control unit trends are the counterfactuals which show us how Nigeria would have performed if not for the reforms in 2003-04.

18
<!-- Start of picture text --> lo} '<br>lo) '<br>fo)N'{<br>\<br>s®° ,<br>| “oO<br>rob) H a<br>Qa ' 7<br>3 | “<br>DS /<br>© | /<br>/<br>t 7<br>17<br>AZ<br>[|<br>fo) — an<br>e417 so ——— - !<br>1995 2000 2005 2010<br>year<br>— treated unit —-—-—-- synthetic control unit<br><!-- End of picture text -->
<!-- Start of picture text --> oO<br>2lo) ,<br>N |<br>2I<br>© |<br>6) |<br>oO2||<br>3-®|<br>gO eh|<br>e |<br>row |<br>59 |<br>oS<br>oy |<br>8 |<br>ne |<br>3|<br>o |<br>iS |<br>° I<br>~[S)<|I<br>|<br>1995 2000 2005 2010<br>year<br><!-- End of picture text -->
are very similar. This suggests that the results we got are not sensitive to our choice of factor models, which is a sign of robustness of the methodology in this particular case. Overall, we see that Nigeria has performed better than its best comparison unit since 2003.

# **4.7 Beyond Average Income Per Capita**

There are strong limitations with conflating development as synonymous to growth in average income per capita. Firstly, average incomes can rise alongside a large rise in inequality, where the median citizen is no better off, while the country earns plaudits for its performance. To avoid such intellectual traps, we attempt to look at changes in median household income, but are beset by data problems. The lack of Gini coefficient values also hinders us in this regard. The lack of data on inequality and median income makes it hard to analyze whether the mean income data is dominated by outliers.

We also attempt to delve into other development outcomes to find out if the increase in average income per capita is an indicator of broad based development or a statistical anomaly. Nigeria has achieved some success in the past decade, as the adult literacy rate, and the enrollment into education ratio have both gone up significantly. The country has seen a rise in its HDI value from 0.4 in 2005 to 0.419 in 2009, which is again not a large, yet a positive change. The life expectancy has crept up slowly between 2000 and 2010. This means that ultimately, the rise in economic growth may have not had impact on other important development outcomes and yet we do not have a simple way of testing such a story. Even with such caveats, we can safely say that our methods have helped us attribute the recent rise in average income growth to the reforms of 2003-04.

21
## **Chapter 5**

# **Conclusions**

In cases where robustness is strong, the synthetic control method helps us clearly assess the impact of reforms, such as the strong positive impact of Nigeria’s reforms on its growth performance since 2003. In such cases, this method has a stronger claim to causal inference than traditional regression methods. The use of synthetic control methods in evaluation of large scale reforms would thus be a welcome step towards adding more rigor in the reform evaluation process.

The case of Nigeria shows there are instances where strong economic performance can be attributed with much certainty to reforms on macroeconomic, debt and financial policy. While the Georgia case is not included in the paper, it illuminated the natural advantage of the framework we are using. The fact that the method itself shows its suitability for certain cases means it can be used only for cases where it shows robust results. Unlike a cross-country regression framework, the researcher knows when the method is working well or not, hence avoiding a false sense of accuracy. Finally, this still leaves questions about the suitability of this method to understand impact of governance reforms, as these variables are slower moving.

Overall, this method shows that in the specific case of Nigeria, the country benefited largely from its reforms in the mid-2000s. Nigeria’s reforms have marshalled increased incomes and investment compared to the best comparison unit. Critically, our qualitative research also agrees with this finding. We however, still do not know much about and how exactly the reforms worked. Learning more about the success of Nigeria’s reforms should now be done in conjunction with micro-level studies that utilize stronger identification techniques and more rigorous evaluation design that are only possible with data on a micro-scale.

22