metadata: authors: 0: author: Doemeland, Doerte
1: author: Trevino, James
count: World
docna: 0: docna: Which World Bank reports are widely read ?
docty: Policy Research Working Paper
subtopic: Consumption,Fiscal & Monetary Policy,Investment and Investment Climate,Private Sector Economics,Health Service Management and Delivery,Private Sector Development Law,Marketing,Inequality
teratopic: Information and Communication Technologies,Public Sector Development,Health, Nutrition and Population,Social Development,Finance and Financial Sector Development
abstracts: cdata!: Knowledge is central to development. The
            World Bank invests about one-quarter of its budget for
            country services in knowledge products. Still, there is
            little research about the demand for these knowledge
            products and how internal knowledge flows affect their
            demand. About 49 percent of the World Bank's policy
            reports, which are published Economic and Sector Work or
            Technical Assistance reports, have the stated objective of
            informing the public debate or influencing the development
            community. This study uses information on downloads and
            citations to assesses whether policy reports meet this
            objective. About 13 percent of policy reports were
            downloaded at least 250 times while more than 31 percent of
            policy reports are never downloaded. Almost 87 percent of
            policy reports were never cited.  More expensive, complex,
            multi-sector, core diagnostics reports on middle-income
            countries with larger populations tend to be downloaded more
            frequently. Multi-sector reports also tend to be cited more
            frequently. Internal knowledge sharing matters as cross
            support provided by the World Bank's Research
            Department consistently increases downloads and citations.
repnme: repnme: Which World Bank reports are widely read ?
display_title: Which World Bank reports are widely read
listing_relative_url: /research/2014/05/19456376/world-bank-reports-widely-read
topicv3: Private Sector Development m1326060 1921,Investment Climate m1326293 405,Poverty m1326687 695,Macroeconomic and Structural Policies
            m1327380 439,Diagnostics m1504698 3625
publishtoextweb_dt: 2014-05-01T00:00:00Z
historic_topic: Information and Communication Technologies,Public Sector Development,Health, Nutrition and Population,Social Development,Finance and Financial Sector Development
no_of_pages: 34
model_extractions: WPS6851 Policy Research Working Paper 6851 Which World Bank Reports Are Widely Read? _Doerte Doemeland James Trevino_ The World Bank Development Economics Vice Presidency Operations and Strategy Unit May 2014
Policy Research Working Paper 6851

# **Abstract**

Knowledge is central to development. The World Bank invests about one-quarter of its budget for country services in knowledge products. Still, there is little research about the demand for these knowledge products and how internal knowledge flows affect their demand. About 49 percent of the World Bank’s policy reports, which are published Economic and Sector Work or Technical Assistance reports, have the stated objective of informing the public debate or influencing the development community. This study uses information on downloads and citations to assesses whether policy reports meet this objective. About 13 percent of policy reports were downloaded at least 250 times while more than 31 percent of policy reports are never downloaded. Almost 87 percent of policy reports were never cited. More expensive, complex, multi-sector, core diagnostics reports on middle-income countries with larger populations tend to be downloaded more frequently. Multi-sector reports also tend to be cited more frequently. Internal knowledge sharing matters as cross support provided by the World Bank’s Research Department consistently increases downloads and citations.

This paper is a product of the Operations and Strategy Unit, Development Economics Vice Presidency. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at ddoemeland@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Which World Bank Reports Are Widely Read?**<sup>**1**</sup>

Doerte Doemeland and James Trevino JEL Codes: O19, H87, and D73.

Keywords: Knowledge Management, International Organizations, Internet.

> 1 The authors would like to thank Zeljko Bogetic, Simon Davies, Jean-Jacques Dethier, Kene Ezemenari, Sharokh Fardoust, Aart Kraay, Eileen Monnin-Kirby, Jimmy Olazo, David Rosenblatt, Zia Qureshi, Adam Wagstaff and participants at a DEC seminar in December 2013 for excellent comments and suggestions, and Eliza McLeod, Yoshimi Muto, Irina Sergeyeva, and Jeannette Smith for assistance with the data.
# **ACRONYMS**

|ESW|Economic and Sector Work|LEG|Legal|
|---|---|---|---|
|AAA|Analytical and Advisory Activities|MNA|Middle East and North Africa|
|AFR|Africa|OCR|Optical character recognition|
|D&R|Documents & Reports|OKR|Open Knowledge Repository|
|DEC|Development Economics|OMBC|Online Media Briefing Center|
|DL|Downloads|OPCS|Operation Policy and Country Services|
|DO|Development Objectives|PAR|Performance assessment review|
|EAP|East Asia and Pacific|PPP|Purchasing Power Parity|
|ECA|Europe and Central Asia|PREM|Poverty Reduction and Economic<br>Management|
|FCO|Fund Center Owner|QAG|Quality Assurance Group|
|FPD|Financial and Private Sector Development|RePEc|Research Papers in Economics|
|FY|Fiscal Year|SAR|South Asia|
|GDP|Gross Domestic Product|SDN|Sustainable Development Network|
|HDN|Human Development|TA|Technical Assistance|
|IBRD|International Bank for Reconstruction and|TTL|Team Task Leader|
||Development|||
|IDU|Internal Documents Unit|VPU|Vice-presidential Unit|
|IEG|Independent Evaluation Group|WBI|World Bank Institute|
|LCR|Latin America and Caribbean|WDI|World Development Indicators|

2
# **I. Introduction**

**Knowledge is central to development.** It is instrumental for developing new products. Knowledge can also help produce existing products more efficiently, generate better outcomes, and enable policy makers to make better choices. Knowledge absorption has been found to drive productivity and income growth and to contribute significantly to poverty reduction.

**Large international organizations, such as the World Bank, can play an important role in generating and transferring development knowledge that is relevant for economic development.** They can draw knowledge from cross-country experiences to improve diagnostics and provide better policy advice. They can also make knowledge available as a public good that can be utilized in many sectors of development activity and in many countries. The production of knowledge goods is often not costefficient at the country level and many developing countries do not have the capacity to develop these types of goods.

**The World Bank spent about one-quarter of its country services budget on core knowledge products in fiscal year 2012.** Core knowledge services<sup>2</sup> are: i) economic and sector work (ESW), ii) technical assistance (TA), iii) the World Development Report iv) external training and capacity development, v) research, vi) impact evaluations, vii) global monitoring, viii) new product development, and ix) internal reports.<sup>3</sup> These knowledge products can be divided further into knowledge for external audience and knowledge for internal use. Spending on the World Bank’s “core” knowledge tasks has increased steadily over the past decade. Expenses on knowledge for external clients and public goods accounted for approximately 83 percent of expenses on core knowledge services in FY2012, of which 74 percent were costs for knowledge products for external clients and 9 percent associated with the development of public goods. ESW is the most important knowledge product for external clients next to TA, absorbing roughly 22 percent of total costs of the nine core knowledge products (World Bank, 2012a).  ESW and TA are also the only two core knowledge products that form part of country services.

**Internal knowledge sharing is essential for a large and complex institution such as the Bank to provide effective policy advice** . Bottlenecks to information flows create inefficiencies, either through duplication of efforts and diverting resources from knowledge creation itself (World Bank, 2000). The idea behind establishing the Bank’s networks in 1996 was to ensure a flow of knowledge throughout the Bank. Internal knowledge sharing can take place through several channels, such as cross support, training, and internal knowledge products. The latter includes the dissemination of sector reports and policy papers via seminars, the creation of tools and databases, and training for Bank staff.  But internal knowledge transfer is not only provided through these targeted products. Improving internal knowledge sharing has gained renewed importance in the context of current WB Change Process. The ongoing reforms aim for the World Bank to become a catalyst for global knowledge by connecting practitioners and by supporting networks of researchers, policy makers, and civic organizations keen to learn about what works and how to implement successful results. The knowledge acquired is used to diffuse innovation at scale, so that successful projects and programs are replicated under the right conditions or with the right adjustments by practitioners across the world (World Bank, 2013a).

> 2 Non-core knowledge products absorbed about US$300 million in fiscal year 2010. They include products such as country partnership strategies, sector strategies and evaluations by the Independent Evaluation Group. 3 See World Bank (2011a) for further elaboration.

3
**An important channel through which internal knowledge transfer occurs in the World Bank is cross support.** Cross support is generally defined as “staff time of an expert or specialist purchased from outside the responsible unit for a specific task” (IEG, 2012, pg. 56).<sup>4</sup> Tasks can be both operational and knowledge-generating in nature. Cross support can take the form of participation in team visits, preparation of key inputs for analytical and advisory activities (AAA), and peer review. Cross support is short-term by nature and does not include staff movement or rotation.

**There is little systematic research about the demand for and use of World Bank knowledge products** . Wagstaff (2012a) assessed the distribution of downloads among World Bank published ESW reports and finds an average of 123 downloads for the 799 reports in his sample. He argues that web-based metrics, such as downloads, could provide a useful tool for assessing demand for World Bank knowledge products. He does not identify the factors that determine the number of downloads. Ravallion and Wagstaff (2010) utilized Google Scholar to generate citation counts for a large quantity of books and research publications of the Bank.<sup>5</sup> They found that the Bank’s research record in development economics was on par with leading economics departments, but that a large portion of the Bank’s research goes uncited.  They also note that citations are dependent on the purpose of the article and the intended audience.  Factiva, a research tool that catalogues over 28,500 media sources, was used by Reeves (2011) to measure the coverage of World Bank publications.  She found that 1,442 out of 2,085 Bank titles received media coverage within the three year period after their publication.

**We find that more expensive policy reports on populous middle-income countries are more likely to be downloaded and cited, especially if these reports have the stated objective of informing the public debate** .<sup>6</sup> We find that more complex policy reports, such as those that focus on multiple sectors or core diagnostic reports, are also more likely to be downloaded and cited.  During the past 5 years the Bank finalized an average of 322 policy reports per year, of which 49 percent have the stated objective of informing the public debate. About 13 percent of all policy reports are downloaded at least 250 times, while about 32 percent are never downloaded.  Over 31 percent of policy reports are never downloaded, while about 13 percent are downloaded at least 250 times.  Almost 87 percent of policy reports were never cited but multi-sector reports tend to be cited more frequently. Internal knowledge sharing matters as cross support provided by the World Bank’s Research Department (DEC) consistently increases downloads and citations.

**The objective of this study is two-fold:** first, we provide objective estimates of the demand for and use of the World Bank’s policy reports; second, we discuss the roles that different costs play with regards to the demand and use of policy reports.   Since the generation and transfer of development knowledge by the World Bank is important in facilitating its objectives, getting a better understanding of the quality and impact of the Bank’s knowledge work is paramount.

**This study is structured as follows:** Section II describes the data on policy reports. Section III discusses the measures of demand for policy reports used in this study. Section IV presents data and

> 4 Cross support is generally defined as support of staff across vice-presidential units (VPUs). Cross support within VPUs is even more frequent and is key for the World Bank to deliver multi-sector knowledge productions and operations.

> 5 It included articles, working papers, books, edited volumes, chapters, and conference proceedings.

> 6 For the purpose of this study, policy reports are defined as those ESW and TA reports that have been published.

4
methodology. Section V summarizes results. Section VI discusses the implication of World Bank internal knowledge sharing. The final section concludes.

# **II. Generating Knowledge in the World Bank: External Policy Reports**

**The World Bank has three key types of core knowledge products: external client knowledge, public goods, and knowledge for internal use.** Over three-quarters of expenditures on knowledge products are for external clients (Figure 1a). There are four distinct external knowledge products: Economic and Sector Work (ESW), Technical Assistance (TA), Impact Evaluation and external training. Along with TA, ESW has increasingly become a core part of the Bank’s engagement with clients—in fiscal years 2000– 2006 the Bank spent 26 percent of its spending on country services on ESW and TA (IEG, 2008). Between April 2012 and March 2013, expenditures for these two products constituted 39 percent of its spending on country services, a substantial increase from previous years (World Bank, 2013b, Figure 1b).

**Figure 1: Distribution of Expenditures**

<!-- Start of picture text --> a) Distribution of Expenditures of b) Distribution of Expenditures on Country<br>Knowledge Products 1% Services<br>0% 6% Project Supervision<br>15%<br>8% Lending Preparation<br>External Clients<br>31%<br>Economic and Sector Work<br>8%<br>Public Good Technical Assistance<br>27% Aid Coordination<br>Internal Use<br>Country Program Support<br>77% 14%<br>Client Training<br>13%<br>Impact Evaluation<br><!-- End of picture text --> Source: World Bank Quarterly Business and Risk Review FY13 Quarter 3. Note: Data is for the period Q4 FY12 to Q3 FY13.

**Economic and Sector Work is the World Bank’s primary knowledge product line and often complements technical assistance (TA)** .  It is the World Bank’s primary country-based analytical and advisory business line, intended to provide a basis for i) conducting policy dialogue; ii) developing and implementing country strategies; iii) formulating effective lending programs; iv) building institutional capacity and informing the international community about a country’s development challenges. It must involve original analytical effort, be undertaken with the intent of influencing an external client’s policies and program, and be owned by a specific Bank unit (World Bank, 2012b).<sup>7</sup> The product represents the view of the Bank, and as such is not attributable to individual authors.  ESW is distinct from public good

> 7 There are three ESW report types: First, core diagnostic reports, which include Country Economic Memoranda, Development Policy Reviews, Poverty Assessments and Public Expenditure Reviews; second, other diagnostic reports, which cover a range of sector-specific topics; and third, advisory reports, which address high priority sector-specific/thematic issues.  Each type of diagnostic report has its own guidelines (World Bank, 2002).

5
research<sup>8</sup> in that it is a knowledge product meant to address specific questions of client countries. This client-demand-driven approach aims to support the development of country strategies and helps with the formulation and implementation of lending programs. While ESW involves research and analysis and is meant to inform policy choices, TA is mostly advisory. Nonetheless, technical advice on formulating and implementing policies and programs can result in report outputs; in most cases, however, it takes the form of the provision of TA, on demand advisory services, or training.

**For the purpose of our study we rely on the policy reports within the D&R database.** Documents & Reports is a database that contains more than 130,000 publically available World Bank documents.  We define policy reports as those documents within the D&R database that were filed as ESW.  These are either ESW or published TA reports. There have been, on average, 322 policy reports per year during the past five years of which around 250 were ESWs. This implies that approximately 52 percent of ESW projects do not have a corresponding report in the Documents & Reports (D&R) database. This is due to three factors.  First, not all ESW are designed to produce reports. Second, some reports are confidential. Though the World Bank’s current Access to Information policy presumes that ESW report will be immediately disclosed, some reports can be flagged as confidential (World Bank, 2010a; World Bank, 2013c).9 Third, some ESWs that have produced reports may not have been filed with the World Bank’s Internal Documents Unit (IDU).

**Many policy reports are part of a larger series with several budget codes.** In order to identify key characteristics of policy reports, such as producing unit, cross support received or costs, we merge information from the D&R database with the budget codes associated with the different reports. However, not all policy reports have a unique budget code. There are three distinct cases: First, multiple reports were funded under a single policy report.<sup>**10**</sup> In many cases there are only two or three reports under a single budget code, but in a few cases there are as many as 12 reports under a single budget code. Second, there are single reports that were funded under multiple budget codes.<sup>11</sup> Overall, there were 149 distinct codes that were aggregated into 66 policy reports. Third, there were multiple budget codes that were linked to multiple reports.  Some of the groupings reflected large thematic report series. Of the original 1,765 codes and the 2,020 reports, we are left with a dataset of 1,611 policy reports of which 1,331 are documents that were released under a single report number that was funded under a single code.

8 Public good research of the World Bank includes Open Data, Policy Research Working Papers, World Development Report, journal articles, and books, among other things.

9 Over the past 15 years, the World Bank’s policy on disclosing information has evolved gradually. Prior to 2010, the World Bank’s approach had been to spell out what documents the World Bank discloses. Under the World Bank Policy on Access to Information which became effective on July 1, 2010, the World Bank discloses any information in its possession that is not on a list of exceptions. Documents flagged as “official use only” are disclosed 5 years after preparation, while those flagged “confidential” or “strictly confidential” are disclosed after 20 years. Confidential reports, for example, include reports with information that has been provided by member countries in confidence or analysis that may affect financial marked behavior.

10 These were sometimes simply a single document that was translated into other languages and released under separate budget codes. For these reports, we created one aggregate report identifier since we could not distribute the expenditures linked to a project code across the multiple report numbers.

11 In April 2012, a Programmatic Approach for ESW and TA was introduced to organize AAA of multiple program activities and knowledge products that support a particular program, theme, or engagement area over several fiscal years (World Bank, 2012b). Since our data set is limited to fiscal year 2012, it does not include any reports categorized as programmatic AAA.

6
**The majority of policy reports are produced by regional vice presidential units (VPUs).** The Africa region (AFR), East Asia and Pacific region (EAP), and Europe and Central Asia region (ECA) generated more than 52 percent of knowledge products between fiscal years 2008–2012 (Figure 2). For the networks, Finance and Private Sector Development (FPD) and Sustainable Development (SDN) have historically taken up the largest share of policy reports, but in the past few years Poverty Reduction and Economic Management (PREM) has increased its number of policy reports, albeit from a very low level. The Middle East and North Africa (MNA) produced the smallest number of reports among the regions, while the Human Development (HDN) Anchor produced the least among the networks.

**Figure 2: Policy Reports per Year**

<!-- Start of picture text --> Policy Reports per Year Policy Reports per Year<br>368<br>328 337 SDN<br>303 PRM<br>275 HDN<br>FPD<br>SAR<br>MNA<br>LCR<br>ECA<br>EAP<br>AFR<br>WBI<br>2008 2009 2010 2011 2012 2008 2009 2010 2011 2012<br>Source: World Bank Documents & Reports and World Bank SAP database. Source: World Bank Documents & Reports and World Bank SAP database.<br>400 400<br>300 300<br>200 200<br>Number of Policy Reports 100 Number of Policy Reports 100<br>0 0<br><!-- End of picture text -->

# **III. Measuring Transfer of Knowledge**

**Individual knowledge services can be evaluated against their specific objective.** For instance, if knowledge services are meant to improve the design of lending operations, measures can be used to assess whether the quality of operations has improved. If they are intended to improve decisions by customers or stakeholders, measures can be used to assess whether these partners believe their decisions were positively shaped by the knowledge they received. If knowledge services are aimed at generating public goods, it can be assess whether and to what extent those public goods are having the intended outcomes.

**All policy reports are required to have a development objective.** Task team leaders (TTL) need to specify a clearly defined development objective, as well as intermediate outcomes of the report and risks to achieving the desired results when creating a budget code for an ESW or TA in the system.  They can choose among one or more of five predefined development objective categories for an ESW or TA: i) informs bank lending); ii) informs government policy; iii) builds capacity; iv) informs the development community and v) stimulates public debate.  Figure 3 below shows the distribution of development objectives among our policy report dataset.  The development objective of about 55 percent of policy reports is to inform government policy and of 49 percent to inform public debate. Correlation among the different development objectives is weak.  The three pair-wise correlations between building client

7
analytical capacity, informing the public debate, and influencing the development community are in the range of 0.23 and 0.26.<sup>12</sup>

**Figure 3: Objectives of Policy Reports**

<!-- Start of picture text --> 60% 55%<br>49%<br>50%<br>40%<br>31% 31% 32% 30%<br>30%<br>20%<br>10%<br>0%<br>Inform Lending Inform Build client Inform/stimulate Influence No Development<br>government analytical public debate development Objective<br>policy capacity community<br>Source: Business Warehouse.<br>Note: Policy reports can have multiple objectives.<br><!-- End of picture text -->

**Increased focus on the quality of knowledge services and its impact has led to a three-pronged approach towards measuring quality:** i) self-assessment, ii) systematic collection of client feedback, and iii) IEG performance reviews.

**i) Self-assessments are provided by the TTL of the report** . After completing a policy report, the TTL is required to evaluate in the system whether the report has met its pre-specified objectives.<sup>13</sup> The Sector Manager and Country Director later validate and endorse this assessment.  Table 1 below shows that TTLs assess their work favorably on average.  Almost half of all policy reports that had the objective of informing lending were considered to have fully met this objective,<sup>14</sup> while more than a quarter of policy reports that sought to influence the development community had the highest assessment score. Informing and stimulating the public debate was the highest rated objective. About 46 percent of policy reports with this objective had fully met it according to the self-assessment.

12 Recent Independent Evaluation Group (IEG) studies show that only 23 percent of knowledge services included indicators that tracked the achievement of the policy reports outcome, and that those knowledge services that had tracking mechanisms performed better in terms of meeting the specified objectives (World Bank, 2013d). 13 The development objectives mentioned earlier are measured using a scale ranging from 1 to 0. A score of 1 indicates that the objective was fully met; a score of 0.75 or above indicates that it was largely met; a score of 0.5 or above indicates that it was partially met; and a score of 0 indicates that the objective was clearly not met.

14Many are ESWs are undertaken with the objective of providing the analytical basis for future lending operations. This indicator is such likely to capture management foresight and political risks (for example, a planned lending operation may go ahead because of a change in government) rather than the quality of the ESW.

8
**Table 1: Policy report Average Self-Assessment Scores**

|**Year**|**Inform**<br>**Lending**|**D**<br>**Inform**<br>**Government**<br>**Policy**|**evelopment Ob**<br>**Build Client**<br>**Analytical**<br>**Capacity**|**jectives(DO)**<br>**Inform/Stimulate**<br>**Public Debate**|**Influence**<br>**Development**<br>**Community**|
|---|---|---|---|---|---|
|**2008**|0.79|0.72|0.76|0.81|0.69|
|**2009**|0.79|0.72|0.71|0.84|0.67|
|**2010**|0.79|0.72|0.74|0.83|0.71|
|**2011**|0.75|0.72|0.77|0.84|0.70|
|**2012**|0.85|0.72|0.71|0.83|0.75|
|**2008–2012**|0.79|0.72|0.74|0.83|0.70|
|**Reports with DO**|484|861|489|773|496|
|**% of Policy reports**<br>**with a score of 1**|49%|32%|31%|46%|26%|

Source: Business Warehouse. Note: A score of zero indicates that the DO was not met; a score of one indicates that the DO was fully met.

**ii) Client feedback on knowledge products is routinely being sought** . Recently, the World Bank has begun to systematically gather and incorporate client feedback for ESW and TA. The Bank sends surveys to government counterparts of selected ESWs and TA to seek their view regarding the quality, relevance, and impact of the provided knowledge service. The most recent survey was completed at the beginning of FY13. Furthermore, work is ongoing to have client feedback information disseminated for economic sector work, non-lending technical assistance, external training and internal knowledge products (World Bank, 2012c).  A product-specific survey was sent to users of ESW and TA for 210 projects that were completed in FY12, with preliminary results showing that many of these products were considered by their users to be effective at achieving their agreed objectives.  The effectiveness of these reports was measured on five characteristics: i) how effective they were at addressing the specific development goals of their agency, ii) their technical quality; iii) their use of best available data, iv) how effective they were in engaging the clients during the design, implementation, and completion of the work, and v) whether the product or service was delivered in a timely manner.  Table 2 below shows the descriptive statistics of this most recent report, measured on a six point scale, with 1 being considered to be “Very Ineffective” and 6 considered to be “Very Effective.”  As mentioned in World Bank (2013b), the average ratings for quality and likely impact are deemed to be effective, with median ratings of 5 on a 6 point scale.  This is commensurate with the self-assessment that TTLs gave regarding whether their policy report informed public debate. The client survey also asks whether or not the knowledge product or activity has led to specific changes, either through policy, regulations, or institutional changes. According to the latest survey, reports scored very high with respect to relevance and technical quality and more than three-quarter led to a change in policies, regulations or institutions.<sup>15</sup>

> 15 There are a few shortcomings with the client feedback surveys.  First, the survey is likely to suffer from significant selection biases arising from a high non-response rate and the fact that the TTL could choose the respondents. Feedback was provided only for 113 out of 210 projects for the question over whether the policy report had led to a policy change within the client agency or institution and the number of feedback providers varied significantly across reports. Second, feedback was requested for a smaller number of projects than in our dataset (210 ESW and TA projects were analyzed in the client feedback survey for FY10, while our dataset covers 275 policy reports).

9
|**Table 2: Client Feed**<br>|**back Ind**<br>|**icators f**<br>|**or FY 2012**<br>||
|---|---|---|---|---|
|**Effectiveness in terms of:**|**Count**|**Mean**|**Standard Deviation**|**Median**|
|Achievement of agreed objectives|192|4.89|0.79|5.00|
|Relevance|195|5.06|0.75|5.00|
|Technical Quality|191|5.05|0.79|5.00|
|Engagement|195|4.95|0.81|5.00|
|Timeliness|192|4.60|1.00|5.00|
|**Policy Change?(Yes=1)**|113|0.77|0.38|1.00|

Source: World Bank Operation Policy and Country Services. Note: The Effectiveness indicators are measured on a scale of 1 to 6, while the Policy Change indicator is a binary indicator.

**iii) Performance assessment reviews are also used to evaluate policy reports.** Performance assessment reviews (PAR) are part of the IEG initiative to assess the impact of Bank ESW and TA.  This initiative is meant as a replacement for the evaluations formerly performed by the Quality Assurance Group (QAG), which was disbanded in FY10. Until now there has not been an autonomous review process of AAA (IEG, 2012).  These ratings consist of the following categories: results, relevance, technical quality, and dialogue/dissemination. All measures use six-point scale ranging from “highly satisfactory” to “highly unsatisfactory”. They are based on four sources: reviews of the content of the policy report to establish substantive content; reviews of the documentary record to probe into process issues, including inception, client and stakeholder engagement, quality control, and dissemination; interviews of country directors and task managers that were familiar with the reports as well as the network responsible for the reports;  and interviews of government officials and other stakeholders in the public and private sectors in the client countries as well as staff in the resident offices (World Bank, 2010b; World Bank, 2011b).<sup>16</sup>

**This study measures the demand for and use of policy reports through downloads and citation counts** . These two indicators are objective measures which serve as a complement to the three abovementioned approaches to measuring the quality of these types of knowledge products.

**Download counts capture the intent to use World Bank policy reports.** More specifically, download counts refer to the number of times a PDF has been downloaded from the World Bank’s external website.  As noted in Wagstaff (2012b), downloads are an excellent indicator of the use of knowledge created by the Bank, particularly the intent to use the document since it is reasonable to assume that the person who downloads a policy report would at least take a look at the contents of the document. We consider this indicator an objective measure of whether policy report meets the development objective of informing the public debate.

**Citations counts measure how often a report was cited by other publications.** In academia, research papers cite other publications that they have used in performing research.  Citations are considered a good indicator of the influence of academic research and are widely used for this purpose in all fields. While citations are a commonly used metric for analyzing the impact of published academic articles, they have not been used to assess the demand of policy reports, in part because policy reports are

> 16 While these documents are available within IDU, there does not appear to be an organized database that contains these reviews, nor is there an official guideline provided by IEG explaining the methodology of PARs.

10
written for policy makers rather than researchers. Still, World Bank policy reports are often cited by think-tanks, publications from other donors or government institutions. Using a broad search engine, that is not restricted to a research network, we argue that citations can used an alternative objective measure to assess the influence of a policy report on the development community.

# **IV. Data and Estimation**

**Data of downloads was gathered for all policy reports which are part of the World Bank’s Documents and Records (D&R) database.** The D&R database contains well over 130,000 documents, but as noted in Wagstaff (2012a) it does not include all documents produced by the Bank. Still, it is the most complete data base of published World Bank documents. D&R also provides a URL link to the actual document, in PDF form, which proved critical in facilitating the collection of download information. One potential issue regarding the collection of download data is the possibility that some policy reports could additionally be hosted on databases other than D&R. Policy reports could be downloaded from other websites such as the Research Papers in Economics (RePEc) which would not be captured in our data, but this is unlikely to significantly affect any of the results presented below.

**Download counts were gathered using Omniture web analytics software.** First, we created a script to scrape the document web addresses for all policy reports from the D&R website to be able to identify documents not only by title or report number but also web address.  We then matched the data from D&R (which had relevant information on the Project Code, Report Number, and document language, and title) with the information on all downloads obtained with Omniture.  We were able to identify from Omniture how often policy reports were downloaded, and when they were downloaded.<sup>17</sup>

**Policy reports in English receive the largest number of downloads.** Figure 4 shows the total number of downloads by VPU broken down by the language of the policy report.  Despite there being six working languages within the Bank (English, French, Spanish, Russian, Arabic, and Chinese), 74 percent of the policy reports in our dataset are published in English.  Still, a significant number of reports are published in languages spoken within the region, such as Spanish language reports in the Latin American and Caribbean region (LCR), French language reports in AFR and MNA, Arabic language reports for MNA, Chinese language reports in EAP, and Russian language reports in ECA.  The largest number of policy reports in other languages is in EAP, attributable to the highly-downloaded reports covering Vietnam and Indonesia.  Generally speaking, policy reports owned by the East Asia and the Pacific region had the most downloads.  In fact, there are fewer policy reports for the EAP region than for the AFR region (Figure 2), yet there are almost twice as many downloads.  The low number of downloads for policy reports owned by Networks is to some extend driven by the low number of reports produced by these VPUs.

> 17 For this report we kept the granularity of the data to an annual basis, but data is also available on an hourly, daily, and monthly basis with Omniture. We were not able to see from where reports were downloaded, as this option is only available for webpage views within Omniture.

11
**Figure 4: Policy Report Downloads by language and VPU Figure 5: Download Distribution**

<!-- Start of picture text --> Downloads Distribution of Policy Reports: 2008 to 2012<br>Downloads from 2008 to 2012 by VPU<br>All Reports with Downloads<br>Other<br>Chinese<br>Arabic<br>Russian<br>Spanish<br>French<br>English<br>0 1000 2000 3000<br>Downloads per Report<br>WBI AFR EAP ECA LCR MNA SAR FPD HDN PRM SDN OTH<br>Source: World Bank Documents & Reports and World Bank Omniture.<br>Source: Omniture and World Bank SAP database.<br>60,000 400<br>300<br>40,000<br>200<br>Number of Reports<br>20,000 100<br>Downloads by Language<br>0<br>0<br><!-- End of picture text -->

**Downloads follow a highly skewed pattern.** This distribution, seen below in Figure 5, is typical for count data: datasets that track the incidences of an action, such as downloads, are highly centered on zero and low numbers. A large portion of policy reports were downloaded relatively few times: Almost 40 percent of policy reports were downloaded between 1 and 100 times.  The “knee of the curve”<sup>18</sup> of the dataset occurs around 250 downloads.  Those policy reports that were downloaded more than 250 times compose 13 percent of our sample.  There are only 25 policy reports (2 percent of the dataset) that have more than 1,000 downloads during the period investigated (FY2008 to FY2012).  Over 31 percent of the policy reports in our dataset (517 out of 1,611) were never downloaded. It is, however, important to keep in mind that many policy reports were not intended to reach a large audience but prepared to assess very specific technical questions or inform the design of lending operations.

**Policy reports that have been released for a longer period of time are downloaded more often.** Average downloads per document and the number of policy reports that have been downloaded drop in recent fiscal years.  Downloads of reports decline over time: Policy reports have an average of 1.6 daily downloads during their first year of release, which decreases to 0.6 downloads during their second year and approximately 0.4 downloads during the third year. Twenty-five policy reports that have been downloaded more than 1,000 times all were released between fiscal years 2008 and 2010.  When these outliers are excluded from the dataset, the average downloads are reduced to 87 for the time period. Table 3 shows the summary statistics for the cumulative downloads for policy reports.

> 18 The “knee of the curve” is the point at which the derivative of the curve is transitioning from a value greater than one to a value less than one.

12
**Table 3: Download Statistics for policy reports**

|**Fiscal Year**|**Total**|**Total**|**Total with**||**Average**|**Standard**||
|---|---|---|---|---|---|---|---|
|**Published**|**Published**|**DL**|**no DL**|**% DL**|**DL**|**Deviation of DL**|**Max DL**|
|**2008**|303|215|88|71%|159|271|2403|
|**2009**|328|229|99|70%|138|267|2955|
|**2010**|337|244|93|72%|117|256|2591|
|**2011**|368|243|125|66%|87|246|1901|
|**2012**|275|163|112|59%|46|132|1302|
|**2008–2012**|1611|1094|517|68%|110|245|2955|

Source: Omniture and World Bank Documents & Reports.

**The most downloaded policy reports tend to have a long shelf-life.** Some reports with a high number of downloads experience very high single-day downloads.  For example, one report released in FY11, had 212 downloads on a single day, but then averaged only 3.42 downloads per day during its first year. Other reports had a high number of first year downloads that were more evenly spread throughout the time period. Another policy report had an average of 4.14 downloads per day during its first year of release, yet never had more than 18 downloads in a single day.   The most downloaded policy report, _Vietnam Development Report 2009: Capital Matters_ , received a total of 2,955 downloads, 1,976 of which were for the English language version of the report and 979 of which were for the Vietnamese language version.  Figure 6 below shows the daily download trends for the five policy reports (disaggregated by language) with the longest shelf-life; i.e. they had the highest average second-year downloads in the dataset and were all downloaded on average once a day.  It is the case that two of these reports, were downloaded more often during their second year of release than during their first. One report was even downloaded more often in its third year of release, 544 times, than in its first two years combined (500 times).

**Figure 6: Download history of selected reports**

<!-- Start of picture text --> 120<br>100 Malaysia economic monitor : repositioning for growth<br>Investing in a more sustainable Indonesia : country environmental analysis 2009<br>80<br>Sudan - The road toward sustainable and broad-based growth<br>60 Malaysia - Productivity and investment climate assessment update<br>Vietnam development report 2010 : modern institutions<br>40<br>20<br>0<br>1 51 101 151 201 251 301 351 401 451 501 551 601 651 701 751 801 851 901 951<br>Days Since Release (Selected Reports)<br>Source: World Bank ISPStats and World Bank Documents & Reports.<br>Daily downloads<br><!-- End of picture text -->

**Citation counts in this study are based on data from Google Scholar.** Google Scholar has great breadth in coverage because it includes not only journal articles, conference proceedings, and other academic

13
reports but also books, working papers, and business and government reports.<sup>19</sup> Google Scholar also includes non-scholarly citations of articles, which is important since policy reports are not intended for publication in academic journals which populate typical citation databases such as Web of Science. Particularly, since objectives of policy reports include informing governments and the development community, it would not be appropriate to limit the citation count to scholarly databases<sup>20</sup> that exclude government and think tank publications. Google Scholar gathers bibliographic information by crawling through websites and fetching HTML and PDF information for the articles, which allows for real-time updates of citations.<sup>21</sup> Unlike other databases, Google Scholar does not wait for a paper to be formally published before it fetches its data from working papers and includes it in its database.

**Using Google Scholar for citation counts is not without problems.** First, its automated software sometimes does not detect false positives or double-counting.  For example, a working paper that has had different versions uploaded to several repositories could have the citations within each working paper counted separately, inflating the citation count for these articles.  Furthermore, Google Scholar did not identify those reports that were ultimately published under a new title, either as a book, a chapter in a book, or as a working paper.  Second, it does not have a large coverage of those older, predigital publications unless these publications have been processed with optical character recognition (OCR) software.  Given that our sample only covers fiscal year 2008 through fiscal year 2012 this last issue should not pose a problem, although there are some policy reports within this period that were scanned documents converted with OCR software.  Third, citations data in Google Scholar is susceptible to manipulation as any search algorithm would be (Delgado López-Cózar et al., 2012), but it is the sentiment of the authors that this is not an issue since policy reports are not produced with the main objective of being cited. Fourth, a report may not receive citations simply because it is not included in the Google Scholar database. Using data from the World Bank’s Open Knowledge Repository (OKR), we verified whether the policy reports in our sample were definitely included within the Google Scholar database.  Within our dataset, 410 policy reports were not listed within Google Scholar.<sup>22</sup> We can assume that these policy reports are not cited, but for the purposes of our analysis we perform regressions with these missing values removed from the data.

**Citation counts are much lower than download counts** (Figure 7). Almost 88 percent of the policy reports (1,054 out of 1,201) in our sample were never cited.<sup>23</sup> Few reports received more than 20

> 19 It does not include newspaper or magazine articles, blogs, editorials, or reviews.

> 20 Prominent sources for citation counts are the Library of Congress’s database, Elsevier’s Scopus database, and Thomson-Reuter’s Web of Science database.  The Library of Congress includes only general bibliographic information for all types of books and academic articles in their database, and does not have any information on the citations within the article nor citations of the article.  The Scopus and Web of Science databases covers only serialized publications and conference proceedings, and the journals included in each database differ.  Scopus includes 19,809 journals and Web of Science includes 12,311 across all academic disciplines.  Between the two there are 11,377 shared journals (Center for Research Libraries, 2013).  Both include counts of citations by other articles, but it is limited to citations by articles from within the respective databases.

> 21 We auto-fetched citation count results using specialized scripts within the open-source program Zotero.  There were constraints to fetching the data based on the nature of Policy Reports: Typically Policy Reports are not published under individual author’s names, so we exclusively had to rely on the title of the document.

> 22 Thirty-three reports were not listed in Google Scholar according to OKR, yet had citations within Google Scholar. These reports were included in the data base.

> 23 By comparison, Ravallion and Wagstaff (2010) find that nearly 30 percent of World Bank research publications have never been cited.

14
citations. The distribution of citations is skewed like the distribution of downloads and is entered on zero and low numbers. Of the 147 policy reports that were cited, only 93 were cited between 1 and 5 times. Those policy reports with more than 5 (the “knee of the curve”) citations consist of 54 policy reports, or 3 percent of the data sample.

**Figure 7: Citation Distribution**

<!-- Start of picture text --> Citation Distribution of Policy Reports: 2008 to 2012<br>All Reports with Downloads<br>0 20 40 60<br>Downloads per Report<br>Source: Google Scholar and World Bank Documents & Reports.<br>30<br>20<br>10<br>Number of Reports<br>0<br><!-- End of picture text -->

**Similar to downloads, older policy reports have been cited more often.** Over the whole period policy reports have an average of 1 citation per document, with documents published in 2009 having the highest citation average of 1.7 citations per document.  The citations were collected in the first quarter of FY13, and have increased since this time.  The most cited policy report in our dataset was cited 59 times when the dataset was collected and assembled.  As of the third quarter of FY14, the document has been cited 91 times.<sup>24</sup> Table 4 shows the summary statistics for the cumulative citations for policy reports.

**Table 4: Citation Statistics for policy reports**

|**Fiscal Year**<br>**Published**|**Total in Google**<br>**Scholar**|**Total**<br>**Cited**|**Total Not**<br>**Cited**|**%**<br>**Cited**|**Average**<br>**Cites**|**Standard**<br>**Deviation of Cites**|**Max**<br>**Cites**|
|---|---|---|---|---|---|---|---|
|**2008**|223|36|187|16%|1.32|4.74|41|
|**2009**|242|40|202|17%|1.71|6.94|59|
|**2010**|258|32|226|12%|0.74|2.82|25|
|**2011**|268|26|242|10%|0.77|4.35|56|
|**2012**|210|13|197|6%|0.21|1.50|20|
|**2008–2012**|1201|147|1054|12%|0.96|4.51|59|

Source: Google Scholar and World Bank Documents & Reports.

> 24 This policy report is cited more since it was published in the peer-reviewed journal _Agricultural Economics_ .

15
**Downloads and citations are count data with a variance that significantly exceeds its mean and a large number of zeros** . Their observations can only have non-negative integer values and they do not have an explicit upper limit.  If the dependent variable is a count variable, the typical econometric regression tool, the linear regression model, is not appropriate since it is sensitive to both the large number of zeros and the extreme values that are not uncommon in count data.  Assumptions of normality for count data are difficult to justify unless the data sample is sufficiently large.  A more suitable model would be based on the Poisson distribution, since it specifically models the number of events that occur over a specific time period, but it works under the assumption that the mean of the count variable is equal to its variance. But for downloads and citations, the variance significantly exceeds the mean. The mean for downloads is 110 while the variance is 60,085.<sup>25</sup> A better fit is thus a negative binomial regression, which is a generalization of the Poisson distribution that includes a parameter to control for over-dispersion, which leads to confidence intervals that are more precise than those from a Poisson regression model.  It is also appropriate to use in situations where the underlying count process is not independent<sup>26</sup> (Winkelmann, 2008).  Problems with the negative binomial include its low applicability to data with large numbers of zero observations (Mihaylova et al., 2011).

**Besides using the negative binomial model, a second option would be the two-part model,** which is able to account for excess zeros in count data (Winkelmann, 2008).  The first part of the model estimates the probability of the variable being counted (i.e. downloaded or cited), while the second part estimates the mean number of counts conditional on the count being positive.  Logit or probit models are typically used for the first part, while ordinary least squares, log-linear, or generalized least squares models are applied for the second.  Two-part models appear to outperform other methods when there are large numbers of zeros in the count data.  The results from both models are presented in the report.

# **V. Results**

**Our most parsimonious specification shows that costlier reports for middle income countries are downloaded more** . Similar to Wagstaff (2012b), we find that more expensive policy reports tend to have more downloads. In fact, increasing the budget of a report from around $180,000 (the mean) to around $316,000 (an increase by half a standard deviation) increases the number of downloads on average by 23 (which is the combined effect of the two part model) or by 30 conditional on the report being downloaded (the result of the two part model regression). We also include dummies of the year of disclosure. As expected, reports that have been disclosed for a longer period of time are more likely to be downloaded (Table B).

**Regional reports on larger and richer countries tend to be downloaded more.** Anecdotal evidence suggests that some countries with large populations and middle income country status receive higher downloads. One would also expect that richer countries are likely to have better internet availability,<sup>27</sup>

> 25 The same holds for citations which μ= 0.96 and σ2= 20.36.

> 26 The existence of contagion or state dependence—that is, the occurrence of an event makes further occurrences likely—would cause over-dispersion.  In the case of downloads, one person’s download is unlikely to be observable by another person (no data of this kind is provided in D&R), making this possibility unlikely.  On the other hand, a citation by one article is observable by others, and this positive contagion effect could drive the citation count of policy reports.

> 27 Though the number of internet users is not significantly linked with increased downloads.

16
higher levels of education, and a larger network of university and research centers. We control for these factors by including variables covering the population and income classification of the country (or region) covered by the policy report. The base specification also controls for Fund Cost Center Owners (FCOs). The base FCO dummy is a composite of non-regional and non-network VPUs in the World Bank, including Legal, Operation Policy and Country Services (OPCS), and World Bank Institute (WBI). For regional FCOs, the FCO is likely capturing the regional focus of the report. Regional policy reports tend to receive a higher number of downloads than reports prepared by World Bank networks. In particular, downloads tend to be particularly high in EAP, ECA, LCR and MNA. Once we control for population and income classification, all regional dummies, with the exception of EAP, turn insignificant. Population has a strong effect in generating downloads.

**Complex multi-sector and core diagnostic reports are downloaded more frequently** . Even when controlling for costs, multi-sector reports are download more frequently.  Multi-sector reports serve as a proxy for the use of intra-VPU cross support, suggesting that reports which receive intra-VPU cross support are more likely to be downloaded.  Also multi-report, multi-project and core diagnostic reports tend to be downloaded more frequently conditional on total costs.

**The objective of the report matters** .  As mentioned above, TTLs need to specify the objective of their report. Within our sample 49 percent of policy reports had the stated objective of informing the public debate. We find that reports that have the stated objective of informing the public debate are more likely to be downloaded, and, conditional on being downloaded, are downloaded more frequently.

**Dissemination strategies are intended to increase the visibility of reports and, thus, presumably downloads.** In IEG (2008), dissemination efforts for policy reports were among the lowest rated dimensions of quality<sup>28</sup> according to in-country stakeholders. Sustained follow-up beyond dissemination seems to be needed, either in the form of lending, from increased funding from other donors for implementing policy changes, or from training workshops.  Government interest also is likely to have a strong effect on dissemination. In Malaysia, the government led and implemented dissemination efforts to a much greater effect than the Bank could do alone (IEG, 2008 pg. 58).  The IEG report goes on to say that these examples show that stimulating public debate, particularly debate within the government, is seen as being important for generating results.

**We try to capture dissemination efforts along three dimensions:** First, we assess whether policy reports that were released in the Online Media Briefing Center have higher downloads on average.  The Online Media Briefing Center (OMBC) launches press releases that are available to accredited journalists before the publication of the report.  Within the five years of our data sample, 97 documents were released in the OMBC.  While only 17 policy reports in our dataset were included in the OMBC, the average downloads per document for these reports were much higher than for those not launched in OMBC.  On average a policy report launched within OMBC had 208 downloads, while a policy report not launched within OMBC had 109.  None of these policy reports pushed by OMBC were core diagnostic reports.  However, once we control for costs, the coefficient on the OMBC dummy, indicating whether the report has been pushed by OMBC, turns insignificant (Table C).<sup>29</sup> Second, the variable “other costs”

> 28 The other dimensions were technical quality, relevance, timeliness, and partnership with clients.

> 29 Regional and network VPUs also launch press releases, but their data is not centralized and as such is not readily available.  We are thus unable to assess whether policy reports were pushed by their Regional and Network receive a higher number of downloads.

17
could be interpreted as a proxy for dissemination efforts and client engagement, capturing travel costs, printing costs, etc. In fact, we find that reports which have higher other costs tend to receive more downloads, conditional on being downloaded. Third, we identified reports that were published in another type of documents, such as a working paper or a book chapter. We find that the reports were downloaded at a significantly greater rate in their new incarnation. For example, one policy report was cited twice within our dataset, but when later published as a working paper with a new title, it was cited over 50 times.

# **_Citations_**

**A standard measure of an index of research output is the h-index which is highest for reports on Latin America and the Caribbean followed by the Africa region.** The h-index is a measure on the impact of research output.  First proposed in Hirsch (2005), the h-index was meant to quantify an individual’s scientific output for applications such as faculty recruitment, granting tenure, and awarding grants.  An h-index value of x means that the author has published x items, each of which has been cited at least x times.  It assesses both the productivity and influence of research.   As citations tend to increase the longer a study is published, the h-index tends to rise with years of publications. Figure 8 shows the h- indices for the regional VPUs.

**Figure 8: H-Indices for the Regions**

<!-- Start of picture text --> AFR = 7.00<br>60<br>EAP = 7.50<br>ECA = 4.75<br>50<br>LCR = 8.63<br>40 MNA = 3.00<br>SAR = 6.00<br>30<br>20<br>10<br>0<br>0 5 10 15 20 25 30<br>Rank of Policy Report<br>Source: Google Scholar and World Bank Documents & Reports.<br>Note: The x-axis ranks individual policy reports by the number of citations. The H-index of<br>a region is the value of the distance between the x-axis and the region's intersection with<br>the 45° line.<br>Citations<br><!-- End of picture text -->

**Contrary to downloads, costs are not a significant determinant of citations (Table D).** A more expensive report is not more likely to be cited or receive a higher number of citations, conditional on being cited. However, under some specifications, cross support related costs significantly increase the probability of being cited. The year dummies, which were all significant and positive, indicated that policy reports that have been disclosed for a longer period of time are cited more frequently. Reports prepared by network anchors such as HDN and PREM as well as by the Africa and LCR regions tend to

18
have a higher probability of being cited. The number of internet users in a subject country does not significantly affect citations.

**Similar to downloads, more complex reports in larger countries tend to be cited more frequently.** Multi-project, multi-report, and multi-sector reports as well as core diagnostic reports are more likely of being cited. Also reports on countries with larger populations tend to be cited more frequently. Contrary to downloads, reports for upper middle countries are not significantly more cited.

**Reports pushed by the OMBC received a higher number of citations.** The 17 reports that were pushed by OMBC were cited an average of 7 times, significantly greater than the mean of 0.9 for those reports not pushed by OMBC (Table E).   One of these reports was cited 51 times, potentially benefitting from a New York Times op-ed by the author. Contrary to downloads, the OMBC dummy remains significant even when controlling for costs.

**Some policy reports may not be cited simply because they were not located in Google scholar** .  There are about 410 policy reports that were not cited and not located in Google scholar. Verifying through other search engines seems to confirm that these reports have not been cited.  The results do not qualitatively change if we were to assume that reports not located in Google scholar have zero citations.

**Development objectives do not seem to matter for citations.** As expected, we do not find any systematic evidence that reports with a development objective of informing the public debate receive a significantly higher or lower number of citations.

**VI. Measuring Internal Knowledge Sharing**

**Measuring internal knowledge transfers is difficult.** The key issue is that it is difficult to assess the costs and, more importantly, the benefits of knowledge sharing among staff because the inputs and outputs are not systematically monitored and reported, and because of the heterogeneity of the methods of disseminating knowledge, such as through team-based support, sector-wide support, or individual training. Two recent papers have tried to assess the demand for and value of research among World Bank’s operational staff.  Ravallion (2011) finds that two-thirds of staff place high value on Bank research.  But it also shows that approximately 23 percent of Bank staff has a low valuation of the relevance of Bank research for their work, and is uninformed and unfamiliar with its knowledge products. According to IEG (2012), sector- and anchor-unit based staff rely most often on policy reports from the anchor units within their own sector, and least often from other units.<sup>30</sup> There is little evidence about the contribution of cross support to policy reports. This is surprising as some FCOs such as DEC, HDN Anchor, and PREM Anchor provide more than 8 percent of their staff time to cross support.

**In order to efficiently provide knowledge to its external clients, any large international institutions will have to build effective mechanisms for internal knowledge sharing.** When the concept of the World Bank as a _Knowledge Bank_ was articulated in 1996, networks were created and given the responsibility to address issues within their fields and to share knowledge with the regions (via sector management

> 30 Regarding substantial use, 28 percent of staff used Policy Reports from the anchor unit within their own sector, 19 percent used Policy Reports from sector units in other regions, 17 percent from sector units outside their sector, and 7 percent from DEC.

19
units and cross support).<sup>31</sup> These networks would then provide their knowledge to operational staffs through cross support.

**Cross support is defined as the staff time of an expert purchased from outside the responsible unit for specific tasks.** Cross support can be shared within VPUs and between VPUs, such as a network VPU and a regional VPU.  Cross support across VPUs could thus be used as a measure of internal knowledge sharing.  Cross support is short-term by nature and does not include staff movement or rotation.  Cross support tasks can be both operational and knowledge-generating, and they can take the form of participation in team visits, preparation of key inputs for analytical and advisory activities (AAA), and peer review. All cross support costs within FY08–FY12 were extracted from SAP.  These data were matched with the project codes for the policy reports.  Those policy reports that did not have matching cross support data are assumed to have zero cross support costs.

**DEC is one of the largest providers of cross-support within the Bank.** It has started to provide increasing cross support over the years, going as far as setting a goal of dedicating a third of its staff time for this purpose (World Bank, 2011a).  In FY12, DEC was the second largest provider of cross support within the Bank, only exceeded by SDN. Figure 9 below shows the levels of inter-VPU cross support sold by the different VPU.  Although DEC is a large supplier of cross support, key Bank reports tend to focus on cross support provided by other departments.<sup>32</sup>

**Figure 9: Total Inter-VPU Cross Support Sold**

_a) In Staff Weeks b) As a Percentage of Staff Years_ <!-- Start of picture text --> Total Inter-VPU Cross Support Sold, Staff Weeks Total Inter-VPU Cross Support Sold<br>Percentage of Staff Years<br>SDN<br>PRM SDN<br>HDN 11.80 PRM<br>FPD HDN<br>SARMNA 9.37 8.37 FPDSAR<br>LCR MNA<br>ECA 6.18 6.85 LCR<br>EAP ECA<br>AFR EAP<br>WBI 3.02 AFR<br>DEC 1.74 1.72 2.02 1.82 WBI<br>1.22 1.20 DEC<br>2008 2009 2010 2011 2012 FY2008 to FY2012<br>Source: World Bank SAP database. Source: World Bank Documents & Reports and World Bank SAP database.<br>Note: Staff are Grade E through Grade I<br>8,000<br>15<br>6,000<br>10<br>4,000<br>5<br>Inter-VPU Cross Support 2,000<br>Inter-VPU Cross Support Share, percentage<br>0 0<br><!-- End of picture text -->

**DEC also provides the largest amount of cross support by staff and for policy reports.** As reported IEG (2012), in FY10 Bank-wide inter-VPU cross support sold accounted for 5.6 percent of total staff time. When this is broken down among regions and networks we have 3.3 percent of staff time and 12.5 percent of staff time,<sup>33</sup> respectively.  These numbers correspond with calculations made from SAP data, shown in Figure 10 below.  DEC has typically provided the largest share of cross support with levels around 11.8 percent of total staff years, although the share for cross support provided by PREM has

> 31 These networks were Human Development, Environmentally and Socially Sustainable Development (now simply Sustainable Development), Finance, Private Sector, and Infrastructure (all merged into Finance and Private Sector Development), and Poverty Reduction and Economic Management.

> 32 See, for example, Annex H of IEG (2012).

> 33 When calculating staff time, we use the Human Resources definition of a staff year, which is 44 weeks.

20
eclipsed DEC in FY12. Regions have provided very little inter-VPU cross support, although the networks, especially HDN Anchor, have provided a larger share. DEC is the largest provider of total cross support for policy reports. Over the past five years DEC has consistently provided the largest amount of cross support for policy reports, as indicated in the left panel of Figure 10 below. SDN is the second largest provider, followed the other networks. ECA is the largest regional provider. In the right panel we can see that most of the cross support from DEC is directed toward AFR and LCR.

**Figure 10: Cross Support for Policy Reports**

<!-- Start of picture text --> Inter-VPU Cross Support for Policy Reports DEC Cross Support to Other VPUs<br>1,103 Policy Reports from 2008 through 2012<br>1,977<br>SDN<br>PRM SDN<br>HDN PRM<br>FPD HDN<br>664 SAR FPD<br>MNA 1,062 SAR<br>LCR MNA<br>ECA LCR<br>355 376 385 EAP ECA<br>218 WBIAFR 509 473 381 EAPAFR<br>112 101 82 117 119 136 DEC 239 238 WBI<br>106<br>26 6 1<br>FY2008 to FY2012<br>Source: World Bank Documents & Reports and World Bank SAP database. Source: World Bank SAP database.<br>2,000<br>1,000<br>1,500<br>Staff Weeks 500 1,000<br>500<br>0 Inter-VPU Cross Support Labor Cost, thousands 0<br><!-- End of picture text -->

**DEC also provides considerable cross support for policy reports as a share of total staff years** , but it was eclipsed by PREM in FY2012 (Figure 11).  The shares of cross support for policy reports provided by DEC equals 24 percent of total inter-VPU cross support.  SDN, while a large provider of inter-VPU cross support, does contribute relatively little to policy reports as a share of total staff weeks.

**Figure 11: Cross support for ESW as a share of Total Staff Years**

<!-- Start of picture text --> Inter-VPU Cross Support for Policy Reports<br>Percentage of Staff Years<br>2.84<br>SDN<br>PRM<br>HDN<br>FPD<br>SAR<br>1.62 MNA<br>1.31 LCR<br>ECA<br>EAP<br>0.80<br>0.60 AFR<br>WBI<br>0.33<br>DEC<br>0.04 0.06 0.16 0.11 0.19 0.12<br>FY2008 to FY2012<br>Source: World Bank Documents & Reports and World Bank SAP database.<br>Note: Staff are Grade E through Grade I.<br>3.0<br>2.0<br>1.0<br>Inter-VPU Cross Support Share, percentage<br>0.0<br><!-- End of picture text --> 21
**DEC cross support is the only VPU linked with greater downloads and greater citations.** Cross support from DEC is significantly associated with more downloads. Cross support provided by FPD significantly decreases the probability of a report being downloaded as well as the number of downloads (Tables F and G). No other cross support is significant. Cross support provided by DEC also increase the number of citations. The relationship between DEC cross support and increased use may also be the manifestation of selection bias, as TTLs with higher evaluations are more likely to agree to work on policy reports covering topics of greater interest (and hence more likely to be read and cited).

**Reports may receive a higher level of cross support because DEC researchers could cite the reports in which they participated.** Looking at the 25 reports with the highest citations that received DEC cross support, we find that less than 5 percent of citation were linked to the involved researchers. DEC staff that provide cross support to a policy report are thus not likely to cite the report later in their own work.

# **VII. Conclusion**

**If the objective of a World Bank policy report is to inform the public debate, well-funded, multi-sector policy reports are likely to do better** . More expensive, complex, multi-sector, core diagnostics reports, such as CEMs or PFRs, on middle-income countries with larger populations tend to be downloaded more frequently.  Those that receive a significant number of downloads tend to have a long shelf-life.  There seems to be a clear demand for multi-sector World Bank reports at a time when the number of specialized think tanks has increased  to over 6,000 (Gann, 2012) and more and more consultancies engage in providing policy advice to developing countries.

**Cross support provided by DEC significantly increases downloads and citation.** Very little research has been performed on the role that Bank-generated research has played in increasing the demand and use of policy reports. To our knowledge, the only report that explored this link is World Bank (2004), which investigated the role that knowledge sharing has played in policy report effectiveness. It concluded that that there was a positive correlation between higher internal quality scores for policy reports and both the number of DEC research citations in policy reports and the amount of time researchers spent in preparation of the policy report.

**There seems to be some evidence that a media push alone is not sufficient for a good dissemination strategy.** We find some evidence that policy reports that are incarnated in a different format, e.g. as a World Bank Policy Research Working paper, receive a significantly higher number of downloads. IEG (2008) provides some discussion that sustained follow-up beyond dissemination is needed, either in the form of lending, increased funding from other donors for implementing policy changes, or training workshops to ensure a high visibility of reports. Government interest seems also to have a stronger effect on dissemination. In Malaysia, the government led and implemented dissemination efforts to a much greater effect than the Bank could do alone (IEG, 2008, pg. 58).  Future research could also assess which dissemination strategies have been more successful. For example, we are not able to distinguish whether reports that have a larger dissemination budget are likely to receive a higher visibility from data currently available in SAP. We can also not assess whether policy reports that were pushed by their Regional and Network VPUs received a higher number of downloads.

**One could also assess different channels through which policy reports could inform the public or influence the development community or whether reports meet other objectives** . First, it would be interesting to know where reports are actually downloaded **.** By using IP addresses, it would be possible

22
to identify whether reports tend to be more downloaded in subject countries or, for example, in the United States, enabling one to track the flow of World Bank knowledge around the world. Second, it would useful to assess which type of policy reports were most effective in contributing to changes in government regulation. Third, some World Bank reports may be relevant for other World Bank reports. World Bank (2004) used citation counts tabulated by Thomas Scientific (later Thomas Reuters) to demonstrate that those ESWs that more frequently cited Bank research were also more likely to receive higher QAG quality ratings. This metric is different from what we propose because it looks at what was cited within a policy report, not how often a policy report was cited. Analyzing how World Bank reports influence each other could improve our understanding of how internal knowledge spreads within the institution.

23