metadata: authors: 0: author: Knack,Stephen
1: author: Parks,Bradley Christopher
2: author: Harutyunyan,Ani
3: author: DiLorenzo,Matthew
count: World
docna: 0: docna: How Does the World Bank Influence the
            Development Policy Priorities of Low-Income and Lower-Middle
            Income Countries ?
docty: Policy Research Working Paper
subtopic: Inequality,Disability,Access of Poor to Social Services,Economic Assistance,Services & Transfers to Poor,Health Care Services Industry,Educational Sciences,Macroeconomic Management
teratopic: Macroeconomics and Economic Growth,Industry,Social Protections and Labor,Education,Poverty Reduction
abstracts: cdata!: This study investigates the World
            Bank's use of lending and non-lending instruments to
            affect the policy priorities of developing countries. In a
            typical year, the World Bank lends more than $30 billion to
            its client countries. It also spends approximately $200
            million on the provision of analytical and advisory products
            each year. However, insufficiently granular data on the
            nature, timing, and distribution of these analytical and
            advisory products and the policy priorities of client
            countries has made it difficult for policymakers and
            scholars to understand which World Bank instruments are most
            useful for effectuating change in the direction of
            government policy. With new data on the delivery of
            analytical and advisory products and micro-level survey data
            from 1,244 public sector officials in 121 developing
            countries, this study demonstrates that the organization’s
            non-lending instruments are more effective than its lending
            instruments at influencing the policy priorities of client
            countries. The World Bank's analytical and advisory
            products not only affect the direction of government policy,
            but also its design and implementation.
repnme: repnme: How Does the World Bank Influence the
            Development Policy Priorities of Low-Income and Lower-Middle
            Income Countries ?
display_title: How Does the World Bank Influence
            the Development Policy Priorities of Low-Income and
            Lower-Middle Income Countries
listing_relative_url: /research/2020/04/31999112/world-bank-influence-development-policy-priorities-low-income-lower-middle-income-countries
topicv3: Structural Policy and Reform m1326450 1130,Poverty m1326687 695,Education m1327048 470,Cash Transfers m1327437 1608,Health m1504661 3480
publishtoextweb_dt: 2020-04-30T00:00:00Z
historic_topic: Macroeconomics and Economic Growth,Industry,Social Protections and Labor,Education,Poverty Reduction
origu: DECRG: Macroeconomics & Growth (DECMG)
no_of_pages: 55
author_keys: 0: author_key: 238249
1: author_key: 1586474
2: author_key: 1060989
model_extractions: Policy Research Working Paper 9225 How Does the World Bank Influence the Development Policy Priorities of Low-Income and Lower-Middle Income Countries?

_Stephen Knack Bradley C. Parks Ani Harutyunyan Matthew DiLorenzo_ Development Economics Development Research Group April 2020
Policy Research Working Paper 9225

# **Abstract**

This study investigates the World Bank’s use of lending and non-lending instruments to affect the policy priorities of developing countries. In a typical year, the World Bank lends more than $30 billion to its client countries. It also spends approximately $200 million on the provision of analytical and advisory products each year. However, insufficiently granular data on the nature, timing, and distribution of these analytical and advisory products and the policy priorities of client countries has made it difficult for policymakers and scholars to understand which World Bank instruments are most useful for effectuating change in the direction of government policy. With new data on the delivery of analytical and advisory products and microlevel survey data from 1,244 public sector officials in 121 developing countries, this study demonstrates that the organization’s non-lending instruments are more effective than its lending instruments at influencing the policy priorities of client countries. The World Bank’s analytical and advisory products not only affect the direction of government policy, but also its design and implementation.

This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at bparks@aiddata.wm.edu.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
How Does the World Bank Influence the Development Policy Priorities of Low-Income and Lower-Middle Income Countries?

Stephen Knack<sup>_∗_</sup> Bradley C. Parks<sup>_†_</sup> Ani Harutyunyan<sup>_‡_</sup> Matthew DiLorenzo<sup>_§_</sup> _Keywords:_ foreign aid, reform, conditionality, policy evaluation, survey methods _JEL Classification:_ C81, C83, D04, F34, F35, F55, O19, O20

> _∗_ Development Research Group, World Bank

> _†_ AidData, Global Research Institute, The College of William and Mary; and Center for Global Development; email: bparks@aiddata.wm.edu.

> _‡_ Sunwater Institute; email: aharutyunyan@sunwater.org.

> _§_ Department of Political Science and Geography, Old Dominion University; email: mdiloren@odu.edu.
_Acknowledgements:_ We thank the National Opinion Research Center (NORC) at the University of Chicago—in particular, Renee Hendley, Kate Hobson, David Kordus, Elise Le, Aparna Ramakrishnan, Alex Rigaux, Bhanuj Soni, and Stacy Stonich—for their survey design and data collection assistance. We also owe a debt of gratitude to the 6,731 development policymakers and practitioners from 126 countries who participated in the _2014 Reform Efforts Survey_ . Were it not for their willingness to graciously share of their experiences and insights, this study would not have been possible. Finally, this study was made possible through generous financial support from the Development Research Group of the World Bank, the John Templeton Foundation, the Smith Richardson Foundation, and the William and Flora Hewlett Foundation.
# **Introduction**

As low-income and middle-income countries have reduced their reliance on official development assistance (ODA) and assumed greater responsibility for direct service delivery activities, many aid agencies and development banks have refocused their efforts on helping partner countries reform laws, institutions, regulations, and customary practices in ways that they hope will promote economic development and effective governance in the long-run (Andrews 2011; World Bank 2012, 2013a; USAID 2015; Buntaine et al. 2017).

However, there is a growing gap between the stated objectives of international development organizations and the metrics of success that they use to judge their own performance. Easily observable output and outcome measures—such as the amount of economic output generated, the number of children vaccinated, and the number of kilometers of roads constructed—are useful for measuring the success of direct service delivery activities (World Bank 2013a; Glennie and Sumner 2014). But most aid agencies and development banks lack credible ways of measuring their influence on the policy priorities of partner countries (Custer et al. 2015; Kremer and Clemens 2016).

The World Bank is a uniquely important case in point. In a typical year, it lends more than $30 billion to its client countries. It also spends approximately $200 million on the provision of analytical and advisory (AAA) products to client countries each year.<sup>1</sup> The organization uses both of these instruments—lending and non-lending services—to influence the development policy priorities of low-income and middle-income countries. However, insufficiently granular data on the nature, timing, and distribution of AAA products and the policy priorities of client countries has made it difficult for policymakers and scholars to understand which World Bank instruments are most useful for effectuating development policy change.

In the absence of strong empirical evidence, policymakers have offered conjectures about

> 1These summary statistics are drawn from `https://www.worldbank.org/en/about/annual-report/ fiscal-year-data` and the World Bank’s Business Intelligence Database.

1
the value of World Bank advice, analysis, and assistance. Robert Zoellick, who served as the President of the World Bank from 2007 to 2012, characterized his organization as a “unique and special institution of knowledge and learning” and argued that “[d]elivering, expanding, and testing this learning—in tandem with financing or separately— _is the most_ ” _important part of our work_ (World Bank 2007, emphasis added). James Wolfensohn, who served as the President of the World Bank from 1995 to 2005, promoted a slightly different version of this argument: “that [the] knowledge and advisory services [of the World Bank] are far more powerful in leveraging policy and institutional change when underpinned by a lending relationship” (Wolfensohn 2000). Others have expressed skepticism. Prior to his appointment as the President of the World Bank in 2019, David Malpass told the U.S. Congress that the principal beneficiaries of the World Bank were “the people who fly in on a first-class ticket to give advice to governments” (Baker 2019).

Nor do scholars agree about whether, when, and how the World Bank promotes policy change within developing countries (Dollar and Svensson 2000; Easterly 2005; Edwards 2007; Helmke and McLean 2014; Smets and Knack 2016, 2018; Masaki and Parks 2019; Carnegie and Samii 2019). Gavin and Rodrik (1995: 332) argue that “the [World] Bank is the single most important external source of ideas and advice to developing-country policymakers” (Gavin and Rodrik 1995: 332). Others in this camp point to highly influential World Bank publications like the _Doing Business Report_ (Besley 2015; Doshi et al. 2019) and the organization’s role in the global diffusion of policy innovations, such as pension reforms and conditional cash transfer schemes (Brooks 2007, 2015; Fiszbein and Schady 2009; Borges Sugiyama 2011; Handa and Davis 2006; Kremer and Clemens 2016). However, another group of economists and political scientists argue that international financial institutions (IFIs) like the World Bank are relatively impotent and their analysis, advice, and assistance has little if any influence on the policy priorities of developing countries (Remmer 1998; Callaghy 1984; Easterly 2006; Harrigan and El-Said 2006; Samuel 2014).<sup>2</sup>

> 2Hunter and Brown (2000: 115) review public policies related to human capital investment in 13 Latin American countries and conclude that “powerful domestic forces tend to override World Bank [policy] di-

2
To better understand whether, when, and how the World Bank effectuates development policy change, we leverage new data on the organization’s lending and non-lending operations as well as a unique survey of 1,328 public officials that measures the policy influence of the World Bank in 24 issue areas and 113 developing countries. We hypothesize that the World Bank’s non-lending instruments are more effective than its lending instruments at influencing the policy priorities of client countries. Our empirical results not only provide support for this hypothesis, but also demonstrate that World Bank AAA products influence the design and implementation of government policy.

# **Theory and Hypotheses**

## _Role of Incentives_

The World Bank is a lending institution that depends on governments who are willing to borrow funds and repay their debts. These borrowers have access to two different types of loans: investment loans for stand-alone projects and development policy loans (DPLs) that do not support stand-alone projects but instead provide budget support in exchange for policy reform.<sup>3</sup> The conventional wisdom is that if the World Bank has any influence on the nature, speed, sequencing, and extent of policy reform in developing countries, it is primarily through its DPL program (Smets and Knack 2016, 2018; Giordano and Pagano 2017). A key feature of this program is the fact that financial disbursements are linked to the adoption of specific policy conditions. The World Bank refers to these conditions as “prior actions” (PAs), and its ability to withhold disbursements—or threaten to withhold disbursements—from borrower governments is a major source of bargaining power.

> rectives.” Similarly, in a review of health care and pension reforms undertaken in Latin America during the 1980s and 1990s, Weyland (2006: 10) finds that “the IFIs most powerful instrument of coercion, loan conditionality, has proven to be a blunt weapon in the enactment of complex institutional reforms. Reforms of social security and health systems involve a wide range of actors. This multiplicity of veto players and the resulting need for political negotiations make it difficult for external actors that lack democratic legitimation to exert much

> 3During the 1980s and 1990s, DPLs were known as structural adjustment loans (SALs).

3
At the same time, there are reasons to question the efficacy of this aid-for-policy program. The World Bank might pick policy conditions that do not reflect local needs and priorities, demand too many conditions, or set arbitrary deadlines that short-circuit the domestic political processes needed to secure the buy-in of various parties with disparate interests (Boughton and Mourmouras 2004; Koeberle et al. 2006). It might also face internal and external pressures—for example, to reward the allies or punish the enemies of powerful shareholders, or to “push money out the door” to achieve professional prestige and budget maximization goals—that undermine the credibility of its policy conditionality (Collier 1997; Dollar and Svensson 2000; Easterly 2005; Kilby 2009).

Notwithstanding these concerns, there is some evidence that DPLs may instigate changes in development policy. After accounting for self-selection bias, Smets and Knack (2016) and Moll and Smets (2018) demonstrate that the cumulative number of World Bank economic policy DPLs contracted by a borrower government is strongly and positively correlated with the quality of its economic policies.<sup>4</sup> Similarly, Giordano and Pagano (2017) provide econometric evidence that DPLs positively correlate with the adoption of business environment reforms in World Bank borrower countries. Rather than focusing on DPLs, Smets and Knack (2018) examine the effects of the PAs that are nested within World Bank DPLs and find evidence of a concave relationship between the cumulative number of PAs and the adoption of public sector governance reforms in World Bank borrower countries: low to moderate levels of conditionality hasten the adoption of public sector governance reforms, while high levels of conditionality have the opposite effect. There are also several studies that suggest World Bank policy lending and conditionality might be more effective in conjunction with the provision of analytical and advisory services (Deininger et al. 1998; Wane 2004; World Bank 2004; IEG 2008a: 40; Fardoust and Flanagan 2011).<sup>5</sup>

> 4Self-selection bias—the fact that an unobserved factor or set of factors might influence a government’s propensity to pursue DPL-related policy changes—is a major impediment to causal inference. World Bank borrowers might contract DPLs to justify policy changes or insulate themselves from the consequences of policy changes that they would have pursued in the absence of the DPLs (Vreeland 2003; Chwieroth 2007; Parks and Davis 2019).

> 5In a 1998 study entitled _Assessing Aid: What Works, What Doesn’t, and Why_ , Dollar and Pritchett

4
Most of these studies rest on the theoretical assumption that the World Bank effectuates policy change by altering the incentives of public sector decision-makers in developing countries. The standard theoretical model is one in which there are two actors—an IFI and a borrower government—that are engaged in a bargaining process. These two actors have well-defined interests and they bargain over financial concessions and policy concessions. The IFI wants to maximize the domestic policy concessions that it can extract from the borrower at the lowest possible cost, and the borrower wants to preserve its domestic policy autonomy while maximizing the financial concessions that it can extract from the IFI (Kremer and Clemens 2016; Girod and Tobin 2016; Masaki and Parks 2019). The IFI’s ability to alter the cost-benefit calculation of the borrower is a function of whether the expected benefits of reform exceed the expected costs. Given that reform often produces near-term costs that are concentrated and long-term benefits that are diffuse, those who stand to lose are typically well-prepared to organize themselves in political opposition to change and those who stand to gain face significant collective action problems and demonstrate low levels of political organization (Acemoglu and Robinson 2000; Alesina and Drazen 1991; Rajan 2004). Consequently, the amount, nature, and timing of the funding that the IFI can offer to the borrower are key determinants of whether an aid-for-policy bargain can be reached (Mosley et al. 1995; Masaki et al. 2020). When the IFI increases the level of discretionary funding that is available to the borrower for near-term use, the borrower can more easily compensate the expected losers of distributionally consequential reforms (Robinson 2007; IOB 2012). Therefore, the IFI’s ability to convince the borrower that the benefits of reform will exceed the costs is a function of its willingness to provide flexible and fast-disbursing funding.

> At the World Bank, DPL funding is easier to disburse and more flexible than funding from investment loans. It is thus considered to be the primary instrument through which (1998: 5, 13) argued that “[t]he key role of development projects should be to support institutional and policy changes that improve public service delivery. [. . . ] Development assistance is more about supporting good institutions and policies than providing capital. Money is important, of course, but effective aid should bring a package of finance and ideas and one of the keys is finding the right combination of the two to address different situations and problems.”

5
the World Bank can effectuate policy change in borrower countries. By way of illustration, consider Indonesia’s fuel subsidy reform in 2013 and the apparent role that a World Bank DPL played in its design and implementation. In its written request for a $400 million DPL, the Indonesian government assured the World Bank that it had put in place a “compensation package [. . . ] [in order] to partly help shield poor households from the impact of higher fuel prices” (Basri and Rajasa 2013). World Bank staff then brought the proposed loan to the Board of Directors for consideration and made the case for approving this large and highly flexible source of funding to the Indonesian authorities on the following basis: “[the Indonesian government will pursue] an expansion of targeted social assistance measures for the poor following an increase in the price of subsidized fuel prices, from 8.5 million households to 15.5 million poor and vulnerable beneficiary households” (World Bank 2013). Therefore, World Bank staff tacitly acknowledged that, by approving the provision of highly flexible financial resources through a DPL, the Board of Directors would enable the Indonesian authorities to pay for the “adjustment costs” of pursuing a fuel subsidy reform. _Role of Ideas_ However, the World Bank is not only a lending institution. It also produces and disseminates analysis and advice about development policy, which has led some observers to call it a “knowledge bank” (Gilbert et al 1999; Ravallion 2013).

Theories of bargaining power call attention to the financial incentives that IFIs can bring to bear to increase the likelihood that borrowers will challenge the domestic policy status quo, but they do not leave room for the World Bank’s causal and principled beliefs about development policy to _directly_ and _independently_ influence the policy priorities of borrower countries. At best, in a setting where the World Bank has provided a financial package to a borrower that is sufficiently attractive to convince the local authorities that the benefits of reform will exceed the costs, the World Bank’s ideas about development policy might improve the design of its DPL and thus increase the odds that the DPL will achieve its stated policy objectives (Malesa and Silarszky 2005; Smets et al. 2013; Saavedra et al. 2015).

6
This blind spot in the literature has prompted some theorization about the conditions under which—and the mechanisms through which—ideas might matter (Meseguer 2006; Macpherson 2006; Kogut and Macpherson 2011; Chwieroth 2012; Weymouth and Macpherson 2012; Smets et al. 2013; Parks and Davis 2019). Social constructivists have proposed that IFIs like the World Bank seek to ally themselves with a small number of reform champions in the executive branch with whom they share causal and principled development policy beliefs. They argue that the presence of “sympathetic interlocutors” helps to align expectations about the appropriateness and likely impact of a given policy choice, thereby facilitating cooperation between IFIs and governments in developing countries (Momani 2005; Chwieroth 2007, 2009, 2010, 2012; Kahler 1992; Corrales 2006; Nelson 2014; Bazbauers 2019).<sup>6</sup> This argument enjoys some empirical support. In a wide-ranging review of conditional contracts between IFIs and developing countries, Kahler (1992: 127) finds that “close alignment between a cadre of national economic technocrats and the IFIs seems to have been a prerequisite for [a reform] agreement.” Similarly, Chwieroth (2009: 2) provides econometric evidence that the prospects for policy change in developing countries correlate positively with the ability of IFI staff to “find and work with sympathetic domestic interlocutors who embrace their policy goals.” However, existing theory focuses on the _recipients_ of policy ideas: those government officials in borrower countries who are willing to promote policy priorities that are consistent with the preferences of the IFIs. It does not shed light on the _mechanisms_ that the IFIs use to introduce and promote status-quo-altering ideas about development policy.

> 6By way of illustration, consider Salam Fayyad, who served as Prime Minister of the Palestinian National Authority (PNA) from 2007 to 2013. His appointment in 2007 signaled a turning point in the PNA’s relations with the international community (Thrall 2010). Fayyad earned his Ph.D. in Economics from the University of Texas at Austin, worked for the World Bank between 1987 and 1995, and became the IMF Resident Representative in the West Bank and Gaza Strip from 1996 to 2001. According to a former Israeli Ambassador to the U.S., Fayyad “[was] the Palestinian figure with the most prestige in the West; they consider[ed] him to be someone who can speak their language” (Ravid 2007). The World Bank also found a kindred spirit and enthusiastic collaborator in Fayyad, which substantially increased the organization’s influence over PNA’s public sector reform priorities between 2007 and 2013 (World Bank 2012: 20).

7
Herein lies the principal contribution of our study. The IFIs have codified a specific set of tools to facilitate the transmission of policy ideas to developing countries over time (Arezki et al. 2012; Parks et al. 2015; Fayad et al. 2020; other citations). The World Bank has a particularly wide-ranging set of so-called analytical and advisory (AAA) services. These non-lending services include economic and social work (ESW) and technical assistance (TA). ESW is geared towards informing and influencing the direction and design of policy in client countries, while TA is focused on the implementation of policy (Edwards 1997; Gilbert et al. 1999; World Bank 2004, 2006; IEG 2008b).<sup>7</sup> We are principally interested in understanding the first-order question of whether, when and how the World Bank influences the direction and design of development policy in its client countries, so we focus here on ESW. The World Bank has five “core [ESW] diagnostic products” that it routinely generates to inform and influence its policy dialogue with borrower countries: Country Economic Memoranda (CEMs) or Development Policy Reviews (DPRs); Country Financial Accountability Assessments (CFAs); Country Procurement Assessment Reviews (CPARs); Poverty Assessments (PAs); and Public Expenditure Reviews (PERs) (IEG 2008b: 14).<sup>8</sup> There are several reasons why these analytical and advisory products (or other types of

> 7The World Bank defines TA as those activities that are “(a) aimed at enabling an external client to implement reforms or strengthen its institutions; (b) free-standing, not an essential part of another activity such as lending or ESW; and (c) linked to a Bank unit with clear accountability for the service provided” (World Bank 2006).

> 8According to World Bank (2004: 86), “[t]he CEM and DPR analyze key aspects of a country’s economic development, such as growth, fiscal reform, public administration, foreign trade, financial sector development, and labor markets. They are flexible analytic instruments whose structure and content are largely dependent on country circumstances. Their aim is to provide an integrated view of a country’s development priorities and a framework for designing development strategies. [. . . ] The CFA evaluates the overall quality of a country’s public financial management system, covering budgeting, accounting, reporting and auditing, and external scrutiny of public finances [. . . ] [and one of its objectives is to] improve a country’s public financial management by developing and following up action plans agreed with the government. [. . . ] The CPAR diagnoses the health of the country’s procurement system and practices and [aims] to generate a dialogue with governments on needed reforms. [. . . ] The PA provides information on the causes and consequences of poverty in a country and examines how public policies, expenditures, and institutions affect poor people. [. . . ] The PER examines government resource allocations within and among sectors and assesses the equity, efficiency, and effectiveness of those allocations in the context of the macroeconomic framework and sector priorities. In addition, it identifies the reforms needed in budget processes and administration to improve the efficiency of public spending.”

8
ESW) could influence the policy priorities of borrower countries, irrespective of any lending relationships that may exist with the World Bank. In the absence of financial leverage, the World Bank “has to begin by [. . . ] persuading existing [political] leaders that specific reforms should be undertaken” (Woods 2006: 72). These leaders are usually motivated by a desire to stay in power, so uncertainty about the distribution of winners and losers that might result from a reform can reduce their willingness to challenge the status quo (Fernandez and Rodrik 1991). The World Bank can therefore use its analytical and advisory products to provide information about the likely distribution of gains and losses and thereby reduce the uncertainty that political leaders face about the likely consequences of reform (Smets 2020).

Another important feature of the World Bank’s analytical and advisory products is that they are often undertaken in collaboration with government officials in the host country. Involving the governing authorities in the process of diagnosing policy problems and formulating policy recommendations may be consequential if it makes political leaders less likely to question the quality and reliability of the analysis and advice that they receive (Masaki and Parks 2019). Indeed, IFIs like the World Bank often lack contextual knowledge about the countries that they are responsible for analyzing and advising and fail to adequately consult with the governing authorities who possess significant informational advantages (Andrews 2013). Therefore, sourcing contextual information from public officials in the country that is being analyzed and advised may reduce perceptions that the diagnostic and advisory content of the World Bank’s ESW is biased or otherwise unreliable (Lombardi and Woods 2008). It may also increase the resonance of the findings and recommendations that the World Bank produces for public sector decision-makers in client countries.

To the extent that World Bank ESW gives policymakers in the client country a greater sense of ownership in the policy changes that are being implicitly or explicitly advocated, it may also give them a useful tool for (a) building broader domestic coalitions of support for reform, or (b) neutralizing actors who might oppose reform (Besley and Persson 2011; 9
Kelley and Simmons 2019; Parks and Davis 2019).<sup>9</sup> Previous research suggests that reform is more likely when it becomes “the business of a wider range of insiders” (Jacoby 2006: 638) and when the general public places pressure on the authorities to follow through on their initial commitments (Abiad and Mody 2005). Therefore, the World Bank’s sympathetic interlocutors in client countries might try to make the diagnostic and advisory content in World Bank ESW a “focal point” that can stabilize expectations and lower bargaining costs among the domestic political actors that make and shape government policy (Schelling 1960; Tama 2013).<sup>10</sup> Based upon these considerations, we will test the following hypothesis: _H1: The provision of ESW increases the World Bank’s ability to influence the policy priorities of its client countries._

# **Data and Methods**

Among social scientists, there is a growing recognition that if a central part of the World Bank’s raison d’ˆetre is to improve the quality of policy in developing countries, those who wish to evaluate the organization’s effectiveness need to use outcome measures that can capture changes in government policy—rather than traditional development outcome measures like economic growth and infant mortality (Smets and Knack 2016; Masaki et al. 2020).

> 9Indeed, sympathetic interlocutors can serve as policy brokers who not only introduce and promote foreign ideas that challenge the status quo, but also ensure that these ideas are politically acceptable and feasible by building coalitions with domestic political allies and neutralizing domestic political opponents. Amalia Anaya is a case in point (Grindle 2004). She served as the head of Bolivia’s Technical Support Team for Education Reform (ETARE) from 1992 to 1994 and played a crucial role in redefining the country’s policy priorities and co-opting those who could potentially derail the new set of policy priorities. Despite inauspicious conditions for change, Anaya succeeded in redefining the Bolivian government’s educational quality reform agenda during the 1990s, and she did so in part by leveraging analysis and advice from the World Bank and the Inter-American Development Bank. She also pre-empted would-be opponents of the government’s new policy priorities by hiring “politically prominent individuals [to staff her reform team] who could establish a bridge to political parties in the country” (Grindle 2004: 97).

> 10Tama (2013: 153) defines a focal point as “a proposal or idea around which public debate revolves in a given policy area” and explains that “[o]ne common problem is that negotiations in situations of multiple equilibria sometimes only succeed if someone is able to create a focal point by publicizing an option that becomes the most salient choice.”

10
Elaborating on this point, Kremer and Clemens (2016: 71) note “[i]f one believes that the primary impact of the [World] Bank comes from its specific investments, then one might reasonably evaluate the Bank by assessing what proportion of its investments yield, for example, a 7 percent annual rate of return. If one believes, as we and many other observers do, that the biggest effects of the World Bank arise through its role in influencing developing country policy, then one’s assessment of the overall impact of the [World] Bank will hinge primarily on one’s beliefs about the effects of these types of policies. A regular bank hopes to obtain a positive return on the vast majority of its investments. In contrast, the [World] Bank could potentially achieve a high social rate of return through a few big wins [. . . ]. Clearly, if one believes that the [World] Bank has reduced poverty primarily by using its financial resources as part of a bargaining process to promote a few important policy reforms, measuring the Bank’s impact becomes difficult. Running a regression with country-level outcomes as the dependent variable and World Bank lending as a key explanatory variable will miss the point if a substantial share of the Bank’s impact came from the spread of conditional cash transfers, or reforms inspired by the Doing Business reports, or the impact of former Bank officials on Indian economic reforms in the 1990s.” To help close this evidence gap, we designed and implemented the _2014 Reform Efforts Survey_ in the summer of 2014 in partnership with the National Opinion Research Center (NORC) at the University of Chicago. It provides a unique source of micro-level survey data on the policy influence of nearly 150 other aid agencies and development banks, as observed and experienced by the local counterparts of these institutions in 126 low-income and middle-income countries.<sup>11</sup> Unlike traditional elite surveys that are based on convenience sampling (where a population of interest is not identified and sample representativeness cannot be evaluated), we first identified our populations of interest in 126 low-income and middle-income countries and carefully constructed sampling frames for each of these countries in a consistent and

> 11The questionnaire is available in the online appendix of Parks et al. (2015).

11
comparable manner, which allows us to evaluate the representativeness of our sample and employ inverse probability weights to adjust for sample selection (i.e., non-response) bias.<sup>12</sup> The process by which we constructed the country-specific sampling frames that were used to facilitate the survey is described in Section B of the Appendix. The master sampling frame consists of 54,990 host government officials, development partner staff, civil society leaders, private sector representatives, and independent experts from 126 low- and lower-middle income countries and semi-autonomous territories.<sup>13</sup> Altogether, 6,731 individuals participated in the survey. However, for the purposes of this study, we are interested in a specific subsample of respondents: the 1,244 host government officials from 121 low- and middle-income countries who (a) participated in the survey, (b) reported having direct interactions with the World Bank, and (c) subsequently evaluated the agenda-setting influence of the World Bank. Each of these respondents was asked to evaluate the level of influence that the World Bank exerted on the host government’s decision to pursue specific reforms over a specific time frame (during the respondent’s tenure in a self-identified public sector position and institution) and in a specific policy area (the respondent’s selfidentified area of policy expertise).<sup>14</sup> These respondents evaluated the influence of the World Bank on a scale of 0 (no agenda-setting influence) to 5 (maximum agenda-setting influence). Given that our sample may not be representative of the population of interest (i.e., the sampling frame), we employ inverse-probability weights to account for unit non-response.

The data from the _2014 Reform Efforts Survey_ are valuable in that they capture the experiences and observations of public sector officials from the governments that the World

> 12For each low- and lower-middle income country, we defined our population of interest as those individuals with direct knowledge of how government policies and programs were prioritized, designed, and implemented between 2004 and 2013. See Section 2 of the Appendix for more details.

> 13The specific inclusion criteria and procedures employed to develop the sampling frame are described in detail in Parks et al. (2015).

> 14These specific reforms were identified by respondents themselves in a prior survey question that asked them to identify up to three specific problems that government reforms (in the respondent’s area of policy expertise) sought to address. In an earlier set of survey questions, each respondent was asked to select his or her primary policy area of focus (from a fixed list 23 different policy areas) and anchor on a specific public sector job and institution (with a defined start year and end year) in a given country between 2004 and 2013. Respondents were then asked to evaluate the influence of specific donors from the vantage point of his or her policy area of focus and country in that particular position and time period.

12
Bank sought to influence during our period of study (2004-2013). We cannot be certain that all survey respondents provided answers on the basis of policy decisions that they themselves made or directly witnessed, but the types of respondents who participated in the survey are substantially more proximate to the empirical phenomena of interest—World Bank influence on the policy priorities of its client countries—than any other population.<sup>15</sup> A public opinion survey or a survey of experts would have been far less useful, as these populations generally know little about the inner-workings of government (Hoffmann-Lange 2007; Tansey 2007; Parks and Masaki 2019).

Another attractive feature of the _2014 Reform Efforts Survey_ is that it provides rich data on the individual characteristics of policymakers. The survey asked each government official a battery of questions about his or her position (rank), primary area of policy expertise, job responsibilities, educational background, length of government service, reform orientation, personal communications and interactions with donor agencies and international organizations, etc. We can therefore control for characteristics of policymakers that may affect the ways in which they perceive or report the influence of the World Bank. The fact that the survey asked respondents to evaluate the influence of the World Bank in a specific country, policy domain, and time period also enables us to model with fixed effects at the level of country-policy cluster pairs.<sup>16</sup> We are thus able to expunge the potentially confounding

> 15A key strength of the survey is the fact that it includes _attributional_ responses from policymakers with firsthand knowledge of the domestic and external factors that influenced the policy priorities and reform efforts of their governments. However, given that individual respondents evaluated the influence of the World Bank based on their own personal observations and experiences, a natural question that arises is whether these evaluations correlate with independently-generated measures of development policy change. Masaki and Parks (2019) provide evidence that suggests the construct validity of our dependent variable is strong. With data from the _2014 Reform Efforts Survey_ , they demonstrate that our measure of the World Bank’s agenda-setting influence is a strong predictor of improvements on the World Bank’s distance to frontier (DTF) measure of the ease of doing business. The DTF measure varies on a scale of 0 to 100, with higher scores indicating greater ease of doing business. Masaki and Parks (2019) find that a one unit increase in respondent ratings of the World Bank’s policy influence—within the business environment policy domain—is associated with a 1 percent increase in DTF values over time.

16To ensure that we have sufficient variation within pairs, we cross-walked the 24 different policy domains in the _2014 Reform Efforts Survey_ to five broad policy clusters that the World Bank uses to categorize its own activities: macroeconomic and structural policy (MSP), human development (HD), sustainable development (SD), governance (GOV), and other. See Table A1 in the appendix for a description of how we map survey respondents’ 24 different policy domains to these five different policy clusters.

13
effects of observable and unobservable characteristics at the level of country-policy cluster pairs and isolate the _within-pair_ variation that is explained by the receipt of World Bank lending and non-lending services.<sup>17</sup>

# **Model**

Our baseline statistical model is an ordered probit model summarized in Equation (1).

In Equation (1), _i_ denotes a survey respondent. Our primary outcome variable— _Influence Rating_ , which represents an individual respondent’s assessment of the World Bank’s influence on his or her government’s decision to pursue reforms in a specific policy domain and time period (the respondent’s self-identified area of policy expertise during his or her period of service in a self-identified public sector position and institution)—is drawn from the _2014 Reform Efforts Survey_ . This variable takes integer values from 0 to 5.<sup>18</sup> _β_ 0 is the intercept. _β_ 1 is the coefficient for a given form of _Assistance_ . **_α_ 1** is a vector of partial effects for a matrix of respondent-country-year-level control variables denoted by **X1** _i_ . While these control variables do not vary within respondents (since there is only one observation per respondent who evaluated the World Bank), they do vary within countries based on the time periods in which respondents held their public sector positions. These measures also vary within country-policy-cluster pairs (i.e., specific policy areas within countries) whenever there

> 17By way of illustration, assume that there are two respondents in a given country-policy cluster pair (e.g. Madagascar-Human Development) and the first respondent reports that the World Bank had a high level of influence on the host government’s policy priorities and the second respondent reports that the World Bank had a low level of influence on the host government’s policy priorities. If the first respondent worked during a period of time (say, 2004-2007) when the World Bank provided a higher level of assistance, analysis or advice to the human development sector and the second respondent worked during a period of time (say, 2008-2011) when the World Bank provided a lower level of assistance, analysis, or advice to the human development sector, a model that uses country-policy cluster pair fixed effects would make it possible to isolate this source of variation. Therefore, our approach allows to isolate variation in the timing of treatment and variation in the timing of when respondents evaluated the policy influence of the World Bank _within_ country-policy cluster pairs.

> 180 denotes no influence. 5 denotes maximum influence.

14
is more than one respondent in a given country-policy-cluster pair and these respondents did not hold their respective public sector positions during the same period of time.<sup>19</sup> **_α_ 2** is a vector of coefficients for our respondent-level controls contained in **X2** _i_ . Here _δn_ denotes unmeasured time-invariant determinants of the World Bank’s policy influence in countrypolicy-cluster _n_ and _rn_ is a country-policy-cluster indicator dummy, with _N_ representing the total number of countries and _k_ representing the total number of policy-clusters in the sample. The term _εi_ is an error term that denotes all unmeasured idiosyncratic factors that affect the World Bank’s reported level of policy influence at the individual respondent level.

In order to measure exposure to the World Bank’s lending and non-lending services, we extract information on the timing, policy focus, and cross-country distribution of ESW products, DPLs, PAs, and investment loans from the organization’s Business Intelligence Database and Development Policy Actions Database. The number of World Bank ESW products received by a country in a given policy cluster and year is our primary variable of interest ( _n_ _~~e~~ sw_ _~~m 1~~ p_ ) and test of _H1_ . The World Bank assigns each of its ESW products to one of four policy clusters: Macroeconomic and Structural Policy (MSP), Human Development (HD), Sustainable Development (SD), and Governance (GOV). Therefore, we cross-walk these policy cluster-categorized ESW data to the 24 policy domains to which _2014 Reform Efforts Survey_ respondents anchored their evaluations of the World Bank (see Table A1 in the Appendix). This approach allows to define exposure to the World Bank ESW “treatment” at the country-policy-cluster-year level, so if a respondent worked in a specific country and policy cluster that received ESW from the World Bank during a year in which he or she evaluated the policy influence of the World Bank, we identify the respondent as having experienced exposure to the World Bank ESW “treatment.” Given that the World Bank provides a bundle of lending and non-lending services to its client countries and virtually all of these services have policy objectives, we also construct four additional variables that measure the World Bank’s lending services. In order to capture

> 19In general, they could vary for any pair that meets these criteria.

15
exposure to development policy lending, we use two measures—the number of development policy loans and the number of prior actions issued by the organization—that are categorized by the World Bank “policy cluster.” We then follow the same cross-walk procedure (see Table A1 in the Appendix) that we used to construct _n_ _~~e~~ sw_ _~~m 1~~ p_ to measure these variables at the country-policy-cluster-year level. These variables— _dpl_ _~~p~~ roj_ _~~n m 1~~ p_ and _pa_ _~~m 1~~ p_ —measure whether a respondent worked in a specific country and policy cluster that received one or more DPLs or PAs, respectively, from the World Bank during a year in which he or she evaluated the policy influence of the World Bank.

We also rely on measures of annual per capita disbursements through development policy loans ( _lnaid_ _~~d~~ pl_ _~~d~~ isb_ _~~p~~ c_ _~~0 m 1~~ p_ ) and annual per capita disbursements through investment loans ( _lnaid_ _~~i~~ nv_ _~~d~~ isb_ _~~p~~ c_ _~~0 m 1~~ p_ ). However, the World Bank does not categorize its financial disbursement data by policy cluster, so these variables are measured at the country-year level. Also, given that _lnaid_ _~~d~~ pl_ _~~d~~ isb_ _~~p~~ c_ _~~0 m 1~~ p_ , _dpl_ _~~p~~ roj_ _~~n m 1~~ p_ , and _pa_ _~~m 1~~ p_ are strongly correlated, we never introduce these variables simultaneously to minimize the risk of multicollinearity.

For ease of exposition, we refer to these five measures of exposure to World Bank assistance as “treatment” variables (see Table A2). We set all of these variables to the mean values that they assumed between a respondent’s start year and the prior year since the receipt of World Bank assistance in the year prior to a respondent’s start year could affect the World Bank’s reported level of policy influence in the very next year.<sup>20</sup> Additionally, in our baseline model, we include six country-level control variables to account for political and economic attributes of countries that may correlate with the extent to which the World Bank influences the policy priorities of its client countries. First, to account for the possibility that a government’s propensity to recalibrate its policy priorities is a function of its bargaining power vis-`a-vis donors and creditors (Girod and Tobin 2016; Parks and Masaki 2019), we control for net official development assistance as a percentage

> 20Due to their highly skewed distributions, we log the two treatment variables with monetary values and replace their missing values with minimum positive values.

16
of gross national income ( _aid_ _~~g~~ ni_ _~~1~~ st_ ). Second, in recognition of the fact that lower levels of demand for reform in lower income countries may condition the policy influence that the World Bank can exert (Andrews 2011; Loayza et al. 2012), we include a measure of gross domestic product per capita ( _lngdp_ _~~p~~ c_ _~~1~~ st_ ). Third, to account for the possibility that the relationship between income and demand for reform is non-linear, we introduce the squared term of gross domestic product per capita ( _lngdp_ _~~p~~ c_ _~~1~~ st_ _~~2~~_ ).<sup>21</sup> Fourth, to address the possibility that small states lack bargaining power and may need to compensate for their own limited internal administrative and technical capabilities by relying on analysis, advice, and assistance from the World Bank (Vreeland 2003; Pop-Eleches 2009), we control for the size of a country’s population ( _lnpop 1st_ ).<sup>22</sup> Fifth, since democratically-elected governments are generally more motivated to “lock-in” reform commitments and more responsive to external pressures for reform (Kapstein and Converse 2008), we control for regime type, as captured by the POLITY rating ( _polity2_ _~~1~~ st_ ). Finally, to account for the possibility that a government’s political orientation influences the level of effort made by World Bank staff to convince the authorities of the merits of policy reform (Smets et al. 2013), we introduce a dummy variable that takes a value of 1 in country-years when the government in power has a left-wing orientation ( _left_ _~~1~~ st_ ). To measure these control variables in a way that minimizes the risk of endogeneity, we take the value of each measure during the first year in which a respondent held his or her public sector position.<sup>23</sup>

> 21Previous research suggests that middle-income countries have more “outside options” than low-income countries with respect to analytical and advisory service (IEG 2007: 5). Indeed, they often purchase analysis and advice from private sector entities. IEG (2008b: xvi) points to “economic monitoring ESW in [middleincome countries] that had access to such analysis from the private sector or international investors” as a particularly ineffective form of World Bank AAA, noting that “[t]he Bank’s economic monitoring ESW had very little value added in such circumstances.”

> 22The _lnpop_ _~~1~~ st_ , _lngdp pc 1st_ , and _lngdp pc 1st 2_ variables are logarithmically transformed to correct for their highly skewed distributions.

23As robustness checks, we calculate two alternative versions of our country-level controls by (a) using the mean of the observed values for the years in which a respondent held his or her public sector position, and (b) setting the values of these variables to those in the year prior to a respondents start year. The rationale for this approach (a) is to account for potential confounds over the full period of treatment. Imagine, for example, that GDP per capita increases sharply between a respondent’s start year and end year. If GDP per capita positively correlates with World Bank policy influence, and our control variable does not capture the fact that GDP per capita increased between the respondent’s start year and end year, we have omitted an important confound. The rationale for approach (b) is to minimize the risk of endogeneity by using 17
We also include a set of variables to account for the potentially confounding effects of respondent characteristics and biases. Consistent with the approach taken by Masaki and Parks (2019), we control for sex ( _sex_ ), level of education ( _master_ and _phd_ ), public sector institutional affiliation ( _orgtype_ ), the length of time for which a given respondent worked for the government of the client country ( _Q1)_ , and his or her area of policy expertise.<sup>24</sup> Additionally, to account for the possibility that a respondent’s beliefs about the level of domestic support for reform (in his or her area of policy expertise) colored his or her assessment of the level of influence that the World Bank exerted on host government policy priorities, we control for the reported level of domestic support for reforms among five groups: the executive ( _head_ _~~s~~ upport)_ , the legislature ( _legis_ _~~s~~ upport)_ , the judiciary ( _judi_ _~~s~~ upport)_ , the military ( _mil support)_ , and civil society ( _civil_ _~~s~~ upport)_ .

# **Results**

Our baseline results are reported in Table 1. In columns (1)-(5), we enter each treatment variable into the equation on its own but with a standardized set of control variables. These models, which include country-policy-cluster-fixed effects, account for the confounding effects of observable and unobservable characteristics at the level of country-policy cluster pairs and thereby isolate the _within-pair_ variation in the outcome variable that is explained by the receipt of World Bank lending and non-lending services. Consistent with _H1_ , we find that our primary treatment variable of interest—the number of analytical and advisory products ( _n_ _~~e~~ sw_ _~~m~~ 1p_ ) issued by the World Bank—registers a positive and statistically significant effect on the organization’s level of influence over the policy priorities of its client countries.

We find that levels of exposure to development policy lending ( _lnaid_ _~~d~~ pl_ _~~d~~ isb_ _~~p~~ c_ _~~0 m 1~~ p_ ) and investment lending ( _lnaid_ _~~i~~ nv_ _~~d~~ isb_ _~~p~~ c_ _~~0 m 1~~ p_ )—in columns (1) and (2)—are negatively associated with the World Bank’s level of agenda-setting influence, but only the former is

> pre-treatment controls. These results are reported in Section 4.

> 24We control for a respondent’s area of policy expertise in two ways. In some model specifications, we include policy cluster fixed effects. In other model specifications, we use country-policy cluster fixed effects.

18
**Table 1:** Determinants of the World Bank’s Influence at the Agenda-Setting Stage of the Policymaking Process

||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(9)|(10)|
|---|---|---|---|---|---|---|---|---|---|---|
|lnaid<br>~~d~~pl<br>~~d~~isb<br>~~p~~c<br>~~0~~<br>~~m~~<br>~~1~~p|-0.06**|||||-0.01|-0.03|-0.06**|||
||(0.03)|||||(0.02)|(0.02)|(0.03)|||
|lnaid<br>~~i~~nv<br>~~d~~isb<br>~~p~~c<br>~~0~~<br>~~m~~<br>~~1~~p||-0.01||||0.02|-0.11***|-0.07|-0.03|-0.02|
|||(0.04)||||(0.01)|(0.04)|(0.04)|(0.04)|(0.04)|
|n<br>~~e~~sw<br>~~m~~<br>~~1~~p|||0.21**|||0.08*|0.06|0.21**|0.21**|0.23***|
||||(0.08)|||(0.05)|(0.05)|(0.09)|(0.08)|(0.08)|
|dpl<br>~~p~~roj<br>~~n~~<br>~~m~~<br>~~1~~p||||0.32*|||||0.33*||
|||||(0.18)|||||(0.18)||
|pa<br>~~m~~<br>~~1~~p|||||0.01|||||0.03|
||||||(0.03)|||||(0.03)|
|lngdp<br>~~p~~c<br>~~1~~st<br>|-14.18**|-10.80*|-10.68*|-10.27*|-10.40*|0.65|-0.28<br>|-16.27***|-11.56*|-11.92**|
||(5.87)|(5.79)|(5.60)|(5.67)|(5.58)|(0.79)|(4.05)|(5.96)|(5.93)|(5.88)|
|lngdp<br>~~p~~c<br>~~1~~st<br>~~2~~|1.06***|0.83**|0.85**|0.82**|0.83**|-0.06|0.08|1.18***|0.89**|0.91**|
||(0.38)|(0.37)|(0.36)|(0.37)|(0.36)|(0.05)|(0.27)|(0.39)|(0.39)|(0.38)|
|lnpop<br>~~1~~st|0.21|-0.30|-0.27|-0.89|-0.62|-0.15***|-0.86|1.25|-0.01|0.31|
||(1.62)|(1.70)|(1.61)|(1.60)|(1.58)|(0.04)|(1.03)|(1.77)|(1.76)|(1.75)|
|aid<br>~~g~~ni<br>~~1~~st|-0.01|-0.01|-0.01|-0.01|-0.01|-0.00|-0.00|-0.01|-0.00|-0.00|
||(0.01)|(0.01)|(0.01)|(0.01)|(0.01)|(0.00)|(0.00)|(0.01)|(0.01)|(0.01)|
|polity2<br>~~1~~st|-0.01|-0.00|-0.01|0.00|0.00|0.01|-0.02|-0.02|-0.02|-0.02|
||(0.04)|(0.04)|(0.04)|(0.04)|(0.04)|(0.01)|(0.03)|(0.04)|(0.04)|(0.04)|
|left<br>~~1~~st|-0.58*|-0.52|-0.66**|-0.54*|-0.63**|-0.16|-0.14|-0.56|-0.41|-0.46|
||(0.33)|(0.33)|(0.32)|(0.31)|(0.32)|(0.10)|(0.22)|(0.35)|(0.34)|(0.34)|
|Respondent-level controls|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|
|Country-Policy Cluster Pair FEs|Yes|Yes|Yes|Yes|Yes|No|No|Yes|Yes|Yes|
|Country FEs|No|No|No|No|No|No|Yes|No|No|No|
|Policy Cluster FEs|No|No|No|No|No|Yes|No|No|No|No|
|Observations|686|692|719|719|719|672|787|672|692|692|
|BIC|3500|3480|3610|3621|3625|2636|3400|3322|3465|3495|
|AIC|2290|2327|2401|2408|2412|2366|2755|2208|2312|2324|

Notes: All regressions use inverse-probability weights to account for variation in unit non-response rate. * _p <_ 0 _._ 1;** _p <_ 0 _._ 05;*** _p <_ 0 _._ 01.

19
statistically significant at the 95% confidence level. When we use an alternative measure of exposure to development policy lending in column (4)—the number of DPLs issued by the organization ( _dpl_ _~~p~~ roj_ _~~n m 1~~ p_ )—the direction of the estimate switches to positive and reaches a conventional level of statistical significance. However, exposure to policy conditionality— as measured by _pa_ _~~m 1~~ p_ —is not a statistically significant predictor of the World Bank’s agenda-setting influence.

In models (6)-(10), we simultaneously introduce multiple treatment variables to assess their effects independently of one another. We also vary the fixed-effects structure of the models across these columns. In models (6) and (7), which respectively control for policycluster-fixed effects and country-fixed effects, we find few statistically significant relationships. However, the _n_ _~~e~~ sw_ _~~m 1~~ p_ variable remains positive in sign and statistically significant at the 90% confidence level. It is statistically insignificant in Model 7 but still positive in sign.

Columns (8)-(10) are our most rigorous model specifications. They simultaneously introduce multiple treatment variables, while also accounting for unobserved factors that are common to policy clusters _within_ countries. The result that is most consistent across all of these specifications is that the number of ESW products ( _n_ _~~e~~ sw_ _~~m 1~~ p_ ) issued by the World Bank is a positive and statistically significant predictor of the level of influence that the World Bank has on the policy priorities of its client countries. The results in Table A10 in the Appendix report the same models from Table 1 using odds ratios. They demonstrate that, holding everything else constant, the provision of each additional ESW product makes the average respondent about 1.24 times as likely to report that the World Bank ranks in the next highest agenda-setting influence category.<sup>25</sup> This effect size is modest, but it is not negligible in light of the limited variability in the outcome variable itself.<sup>26</sup> Also, given that the cost of achieving this higher level of influence is quite low ($34,376, on average),

> 25Recall that respondents were asked to evaluate the World Bank with integer values between 0 (no agenda-setting influence) and 5 (maximum agenda-setting influence).

> 26The standard deviation of the _influence_ measure is 1.33. See Table A2 in the Appendix.

20
our results imply that ESW is a particularly cost-effective way that the World Bank can influence the policy priorities of its client countries.<sup>27</sup> A less consistent pattern emerges when we consider the treatment variables that measure exposure to development policy lending and investment lending. One’s level of exposure to development policy lending ( _lnaid_ _~~d~~ pl_ _~~d~~ isb_ _~~p~~ c_ _~~0 m 1~~ p_ ) is negatively and significantly associated with the agenda-setting influence of the World Bank, but the direction and significance of this estimate changes when we use _dpl_ _~~p~~ roj_ _~~n m 1~~ p_ an alternative measure of DPL treatment in column (9). Exposure to policy conditionality ( _pa_ _~~m 1~~ p_ ) is positive but statistically insignificant.<sup>28</sup> In Table 2, rather than using an outcome measure that captures the World Bank’s influence on the direction of policy (i.e., agenda-setting influence), we use a measure from the _2014 Reform Efforts Survey_ that asked respondents how much influence the World Bank had over the design of policy. We do so because the stated purpose of World Bank ESW is to influence both the _direction_ and the _design_ of government policy in client countries (World Bank 2004, 2006). The fixed-effects structure of the models in Table 2 is the same as in Table 1. The results demonstrate that the _n_ _~~e~~ sw_ _~~m 1~~ p_ variable still registers a positive and statistically significant effect.

In Table 3, we replicate the same models but with another outcome measure from the _2014 Reform Efforts Survey_ : respondent evaluations of the World Bank’s helpfulness during

> 27ESW products are relatively inexpensive. In 2012, the World Bank delivered 874 ESW products to its client countries at total cost of $34,266,769. In 2013, it delivered the 772 ESW products to its client countries at a total cost of $24,503,120. In 2014, it delivered 1,199 ESW products to its client countries at a total cost of $38,588,088. The average unit cost of an ESW product during this time period (2012-2014) was $34,376. These summary statistics are drawn from the World Bank’s Business Intelligence Database.

> 28With some exceptions, the results for the control variables are mostly consistent across the columns in Table 1. Our measure of per capita income ( _lngdp_ _~~p~~ c_ _~~1~~ st_ ) and its squared term ( _lngdp_ _~~p~~ c 1st_ _~~2~~_ ) together suggest that increases in per capita income are negatively associated with the World Bank’s level of agendasetting influence, but that this effect dampens at higher levels of per capita income. The population size variable ( _lnpop 1st_ ) is negatively signed in all but one model, but only statistically significant in model (6). The aid dependence variable ( _aid gni_ _~~1~~ st_ ) is negative in all model specifications but never statistically significant (and close to zero). We also find no statistically significant relationship between the strength of democratic institutions ( _polity2_ _~~1~~ st_ ) and the World Bank’s level of agenda-setting influence, and the sign of the coefficient changes depending on the model specification. We find some evidence that the World Bank enjoys less policy influence in countries with left-wing governments ( _left_ _~~1~~ st_ ), but this estimate fails to achieve statistical significance in most models.

21
**Table 2:** Determinants of the World Bank’s Influence at the Design Stage of the Policymaking Process

||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(9)|(10)|
|---|---|---|---|---|---|---|---|---|---|---|
|lnaid<br>~~d~~pl<br>~~d~~isb<br>~~p~~c<br>~~0~~<br>~~m~~<br>~~1~~p|-0.02<br>(0.03)|||||0.01<br>(0.02)|-0.03<br>(0.02)|-0.04<br>(0.03)|||
|lnaid<br>~~i~~nv<br>~~d~~isb<br>~~p~~c<br>~~0~~<br>~~m~~<br>~~1~~p||-0.01||||0.01|-0.05|0.04|-0.03|-0.02|
|||(0.04)||||(0.01)|(0.04)|(0.05)|(0.04)|(0.04)|
|n<br>~~e~~sw<br>~~m~~<br>~~1~~p|||0.17*|||0.09**|0.12**|0.18**|0.18**|0.20**|
||||(0.09)|||(0.05)|(0.05)|(0.09)|(0.09)|(0.09)|
|dpl<br>~~p~~roj<br>~~n~~<br>~~m~~<br>~~1~~p||||0.37*|||||0.39**||
|||||(0.19)|||||(0.19)||
|pa<br>~~m~~<br>~~1~~p|||||0.03|||||0.04|
||||||(0.03)|||||(0.03)|
|lngdp<br>~~p~~c<br>~~1~~st|-22.79***|-22.63***|-19.09***|-19.27***|-19.58***|1.16|-5.79|-28.17***|-22.77***|-22.98***|
||(7.34)|(7.53)|(6.98)|(7.08)|(7.02)|(0.84)|(4.66)|(7.83)|(7.61)|(7.56)|
|lngdp<br>~~p~~c<br>~~1~~st<br>~~2~~|1.52***|1.48***|1.27***|1.28***|1.30***|-0.10*|0.35|1.87***|1.50***|1.51***|
||(0.48)|(0.49)|(0.46)|(0.46)|(0.46)|(0.06)|(0.31)|(0.51)|(0.50)|(0.49)|
|lnpop<br>~~1~~st|-0.06|0.13|-0.31|-0.86|-0.54|-0.16***|-0.76|1.18|0.22|0.59|
||(1.92)|(2.03)|(1.91)|(1.90)|(1.88)|(0.04)|(1.11)|(2.07)|(2.09)|(2.08)|
|aid<br>~~g~~ni<br>~~1~~st|-0.00|-0.00|-0.00|-0.00|-0.00|-0.00|-0.00|-0.00|-0.00|-0.00|
||(0.01)|(0.01)|(0.01)|(0.01)|(0.01)|(0.00)|(0.00)|(0.01)|(0.01)|(0.01)|
|polity2<br>~~1~~st|-0.07*|-0.06|-0.07*|-0.07*|-0.06|-0.00|-0.04|-0.08**|-0.08*|-0.08*|
||(0.04)|(0.04)|(0.04)|(0.04)|(0.04)|(0.01)|(0.03)|(0.04)|(0.04)|(0.04)|
|left<br>~~1~~st|-0.09|-0.06|-0.17|-0.03|-0.10|-0.02|-0.07|-0.11|0.08|0.04|
||(0.35)|(0.37)|(0.35)|(0.34)|(0.35)|(0.10)|(0.22)|(0.38)|(0.37)|(0.38)|
|Respondent-level controls|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|
|Country-Policy Cluster Pair FEs|Yes|Yes|Yes|Yes|Yes|No|No|Yes|Yes|Yes|
|Country FEs|No|No|No|No|No|No|Yes|No|No|No|
|Policy Cluster FEs|No|No|No|No|No|Yes|No|No|No|No|
|Observations|670|675|702|702|702|656|769|656|675|675|
|BIC|3322|3365|3500|3507|3485|2617|3341|3249|3345|3336|
|AIC|2209|2245|2334|2337|2333|2352|2690|2155|2230|2230|

Notes: All regressions use inverse-probability weights to account for variation in unit non-response rate. * _p <_ 0 _._ 1; ** _p <_ 0 _._ 05; *** _p <_ 0 _._ 01.

22
**Table 3:** Determinants of the World Bank’s Helpfulness at the Implementation Stage of the Policymaking Process

||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(9)|(10)|
|---|---|---|---|---|---|---|---|---|---|---|
|lnaid<br>~~d~~pl<br>~~d~~isb<br>~~p~~c<br>~~0~~<br>~~m~~<br>~~1~~p|-0.01<br>(0.04)|||||-0.00 <br>(0.02)|-0.06***<br>(0.02)|-0.04<br>(0.04)|||
|lnaid<br>~~i~~nv<br>~~d~~isb<br>~~p~~c<br>~~0~~<br>~~m~~<br>~~1~~p||-0.01||||0.02|0.01|-0.01|-0.01|-0.01|
|||(0.03)||||(0.01)|(0.04)|(0.04)|(0.03)|(0.03)|
|n<br>~~e~~sw<br>~~m~~<br>~~1~~p|||0.24**|||0.02|0.02|0.21**|0.22**|0.22**|
||||(0.10)|||(0.04)|(0.05)|(0.09)|(0.10)|(0.10)|
|dpl<br>~~p~~roj<br>~~n~~<br>~~m~~<br>~~1~~p||||0.10<br>(0.19)|||||0.04<br>(0.19)||
|pa<br>~~m~~<br>~~1~~p|||||0.01|||||0.01|
||||||(0.03)|||||(0.04)|
|lngdp<br>~~p~~c<br>~~1~~st|-9.01|-10.44*|-9.31|-8.32|-8.61|-0.07|-8.09**|-12.76**|-11.36*|-11.55*|
||(5.75)|(6.08)|(5.75)|(5.71)|(5.66)|(0.76)|(3.77)|(6.26)|(6.25)|(6.21)|
|lngdp<br>~~p~~c<br>~~1~~st<br>~~2~~|0.58|0.70*|0.59|0.53|0.55|-0.01|0.61**|0.84**|0.76*|0.77*|
||(0.37)|(0.40)|(0.37)|(0.37)|(0.37)|(0.05)|(0.25)|(0.41)|(0.40)|(0.40)|
|lnpop<br>~~1~~st|3.55**|2.81|4.43**|3.50*|3.61**|-0.08*|-1.28|3.79**|3.64*|3.70*|
||(1.80)|(1.87)|(1.89)|(1.87)|(1.84)|(0.04)|(1.08)|(1.89)|(1.96)|(1.92)|
|aid<br>~~g~~ni<br>~~1~~st|0.01|0.00|0.01|0.01|0.01|-0.00|-0.00|0.00|0.01|0.01|
||(0.01)|(0.01)|(0.01)|(0.01)|(0.01)|(0.00)|(0.00)|(0.01)|(0.01)|(0.01)|
|polity2<br>~~1~~st|0.06|0.07|0.05|0.07|0.07|-0.00|0.04|0.05|0.06|0.05|
||(0.05)|(0.06)|(0.05)|(0.05)|(0.05)|(0.01)|(0.03)|(0.06)|(0.06)|(0.06)|
|left<br>~~1~~st|-0.57|-0.58|-0.63*|-0.54|-0.54|-0.03|-0.25|-0.65*|-0.63|-0.61|
||(0.37)|(0.37)|(0.36)|(0.36)|(0.36)|(0.12)|(0.21)|(0.38)|(0.39)|(0.39)|
|Respondent-level controls|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|Yes|
|Country-Policy Cluster Pair FEs|Yes|Yes|Yes|Yes|Yes|No|No|Yes|Yes|Yes|
|Country FEs|No|No|No|No|No|No|Yes|No|No|No|
|Policy Cluster FEs|No|No|No|No|No|Yes|No|No|No|No|
|Observations|634|635|657|657|657|621|732|621|635|635|
|BIC|2944|2901|3011|3078|3046|2319|3036|2833|2900|2913|
|AIC|1916|1886|1957|1983|1973|2057|2406|1840|1881|1884|

Notes: All regressions use inverse-probability weights to account for variation in unit non-response rate. * _p <_ 0 _._ 1; ** _p <_ 0 _._ 05; *** _p <_ 0 _._ 01.

policy _implementation_ . Here again, the _n_ _~~e~~ sw_ _~~m 1~~ p_ results mirror those reported in Tables 1 and 2, which suggests that ESW products are not only influential at upstream stages of the policymaking process, but also helpful during downstream policy implementation.

# **Robustness Checks**

In this section, we undertake a battery of robustness checks and summarize the results reported in the Appendix. First, in Tables A2-A4, we alter the measurement of our treatment and (country-level) control variables by taking their mean values over the duration of a 23
respondent’s tenure in his or her public sector position.<sup>29</sup> We do so to account for the possibility of “late” (and “early”) treatment. That is to say, if the receipt of World Bank assistance occurred towards the end of the period of time in which a respondent evaluated the World Bank’s level of policy influence (i.e., one of the last years when his/her job required interacting with the World Bank), then one would need to measure treatment exposure over the entire course of the treatment period.<sup>30</sup> When we replicate our baseline models with these alternative treatment and control variables, we obtain similar results. Although some of the estimates are no longer statistically significant, the _n_ _~~e~~ sw_ _~~m~~ ean_ remains positive in sign and statistically significant in the most rigorous model specification reported in Table A3.

Second, we test whether our results are sensitive to the inclusion of controls that measure a potentially significant confound: client country demand for reform assistance. If the World Bank finds and works with “sympathetic interlocutors” in client countries who share their policy preferences, then it is possible that these government officials would have implemented the World Bank’s policy ideas in the absence of ESW. To address this potential threat to inference, we exploit two unique features of the _2014 Reform Efforts Survey_ : a question that asked each respondent to report how frequently he or she interacted with the World Bank officials via phone, email, video, or face-to-face communications during his or her period of public service,<sup>31</sup> and a question that asked each respondent to reveal the intensity of his or her preferences for status-quo challenging reform.<sup>32</sup> We use the first variable ( _freq_ _~~c~~ omm_ )

> 29We calculate these mean values by using all observations between and including the respondent’s start year and end year.

> 30We also set our (country-level) control values to their mean values of the duration of a respondent’s tenure in his or her public sector position to account for potential confounds over the full period of treatment. By way of illustration, consider the possibility that GDP per capita increased sharply between a respondent’s start year and end year. We know that GDP per capita negatively correlates with World Bank policy influence, so if our control variable does not capture the fact that GDP per capita increased between the respondent’s start year and end year, we will have omitted a potentially important confound. The two principal disadvantages of this approach are endogeneity and measurement error (if respondent assessments of the World Bank’s policy influence are based on actions taken by the World Bank early in their tenure while the measurements of the treatments and controls take into account data from later years).

> 31Respondents were allowed to select from one of six response options: once a year or less, 2 or 3 times a year, about once a month, 2 or 3 times a month, about once a week, or daily.

> 32Respondents were asked to indicate if they thought that their governments pursued “too much reform,”

24
as a proxy for the willingness of respondents to engage in policy dialogue with the World Bank. We also use the product of ( _freq_ _~~c~~ omm_ ) and ( _demand_ _~~r~~ eform_ _~~a~~ dvice_ ) as a more direct measure of the respondent’s demand for reform advice from the World Bank. Tables A6 and A7 demonstrate that our core finding related to the effect of World Bank ESW is robust to the inclusion of these additional control variables.

Third, we consider the possibility that our results are biased by a selection effect. Participants in the _2014 Reform Efforts Survey_ first identified the bilateral and multilateral development institutions with whom they directly engaged during their period of public service, and then evaluated the policy influence of only these institutions. We therefore only observe responses about the level of the World Bank’s policy influence from those survey participants who worked directly with the World Bank, which implies that our results could be driven by self-selection bias (if respondents who are more inclined to work with the World Bank are also more inclined to assess its policy influence favorably). The Heckman model accounts for self-selection bias—i.e., the presence of potential confounders that may have affected the propensity of participants to interact with the World Bank (Heckman 1979). In the first (selection) stage of the analysis, we estimate a Probit model (using the full sample) to explain the probability of respondent participation in the outcome stage. Then, in the second (outcome) stage of the analysis, we estimate a standard linear OLS model to predict the World Bank’s level of agenda-setting influence. To employ the Heckman model, a variable should be identified that only affects respondents’ propensity to interact with the World Bank (included in the selection stage) and does not affect their evaluation of the influence of the World Bank (excluded from the outcome stage) (Sartori 2003). For this purpose, we use a variable that measures the number of bilateral and multilateral development institutions with whom a participant interacted during his or her period of public service ( _number_ _~~o~~ rg_ ). This variable captures the general propensity of survey respondents to interact with aid agencies and development banks, including the World Bank, but we do not have any strong “too little reform,” or “about the right amount” within their respective areas of policy expertise and between their start dates and end dates in specific public sector positions.

25
theoretical reasons to believe that it will independently affect how participants evaluate the World Bank. Indeed, as the selection stage in Table A8 demonstrates, _number_ _~~o~~ rg_ is highly significant and positively correlated with the selection variable. The results of the outcome stage also show that our key finding—the positive relationship between the provision of ESW and the World Bank’s level of agenda-setting influence—is robust to this correction for self-selection bias.

Fourth, we conduct a placebo test using the average agenda-setting influence evaluations of three other donors—the European Union, the United Nations Development Program, and United States Agency for International Development—for respondents who evaluated those donors _and_ the World Bank. We find no relationship between the World Bank’s provision of ESW and the reported policy influence of these donors (see Table A9).

Fifth, we take advantage of another unique feature of the _2014 Reform Efforts Survey_ : the fact that those respondents who indicated familiarity with one or more of the World Bank’s core ESW products were asked to independently evaluate the agenda-setting influence of those particular analytical and advisory products (as opposed to the World Bank’s policy influence on the whole).<sup>33</sup> Table A10 demonstrates that our main findings are insensitive to the use of this alternative dependent variable.

> 33The World Bank’s core ESW products include Country Economic Memoranda (CEMs), Development Policy Reviews (DPRs), Country Financial Accountability Assessments (CFAs), Country Procurement Assessment Reviews (CPARs), Poverty Assessments (PAs), and Public Expenditure Reviews (PERs). Participants in the _2014 Reform Efforts Survey_ evaluated four of these ESW products (CEMs, DPRs, CFAs, and PA), so we use only these data. Since this outcome variable is measured at the respondent level, we average across the set of core ESW diagnostic products that a respondent evaluated.

26
# **Concluding Remarks**

This study provides evidence that the analytical and advisory products of the World Bank are effective at influencing the design, direction, and implementation of government policy. By contrast, we do no find robust evidence that the World Bank’s development policy lending or investment lending services consistently influence government policy in low-income and middle-income countries. These results have significant policy implications. For every dollar that that the World Bank spends on non-lending services (or analytical and advisory products), it spends approximately $150 dollars on lending services.<sup>34</sup> Therefore, if a central goal of the World Bank is to influence government policy in its client countries, our results imply that it can achieve better value-for-money through its provision of ESW products.

Our study also highlights the value of collecting granular data on (a) the nature, timing, and distribution of analysis and advice from aid agencies and development banks, and (b) the direction, design, and implementation of policies in low-income and middle-income countries. In the absence of treatment and outcome variables with significant variation across countries, policy areas, and time, it would not have been possible to benchmark the policy influence effects of the World Bank’s lending and non-lending services. Our study therefore calls attention to a broader lesson: relatively inexpensive investments in the collection of data can help international development organizations that aspire to influence government policy measure the effectiveness of the instruments that they create for this express purpose.<sup>35</sup>

> 34This is a lower-bound estimate based on the conservative assumption that the World Bank allocates $30 billion a year to lending services and $200 million a year to non-lending services. 35 It cost approximately $500,000 to design and implement the _2014 Reform Efforts Survey_ . The Development Research Group of the World Bank contributed $113,000 to this effort.

27