metadata: authors: 0: author: Rama,Martin G.
1: author: Singh,Rucheta
2: author: Aslam,Aiza
count: World
docna: 0: docna: What Lies Behind “Good” Analytical Work on
            Development ? Four Years of Knowledge Products at the World Bank
docty: Policy Research Working Paper
abstracts: cdata!: The World Bank&#x2019;s analytical
            work has a strong reputation, but its knowledge products are
            also perceived to be of varying quality and relevance, and
            the drivers of this heterogeneity are only partially
            understood. Building on previous evaluations, this paper
            adopts a production function approach to assess how budget
            resources, time to completion, technical skills, and
            institutional responsibilities affect the internal ratings
            and external visibility of different types of analytical
            tasks at the World Bank. To this effect, the paper first
            matches records from three unconnected electronic platforms
            &#x2014; for internal documents, budget codes, and
            external publications &#x2014; to assemble a
            comprehensive database of knowledge products and their key
            characteristics. With analytical documents as its unit of
            observation, the exercise shows that: (1) devoting more
            resources to analytical tasks leads to both better ratings
            and greater visibility; (2) both outcomes are systematically
            worse when a greater share of resources comes from trust
            funds; (3) they are also consistently worse for tasks that
            take longer to complete; (4) more academically oriented team
            leaders underperform on ratings and overperform on
            visibility, whereas technically solid but less stellar team
            leaders overperform on ratings; and (5) everything else
            equal, performance varies systematically with the nature of
            the unit in charge. The findings of the paper can be read as
            a cautionary note against knowledge management that is based
            on the counting of analytical tasks. Instead, the findings
            call for much stronger information systems on knowledge
            products, a better alignment of incentives for the units in
            charge, and regular evaluations in the spirit of this paper.
display_title: What Lies Behind “Good” Analytical
            Work on Development ? Four Years of Knowledge Products at
            the World Bank
listing_relative_url: /research/2024/02/34258623/lies-behind-“good”-analytical-work-development-four-years-knowledge-products-world-bank
topicv3: Research and Analytic Capacity m1326099 2036,Knowledge Accumulation m1326292 1915,Research, Development and Innovation
            m1327104 690
publishtoextweb_dt: 2024-02-14T00:00:00Z
dois: 10.1596/1813-9450-10704
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 40
author_keys: 0: author_key: 95769
1: author_key: 1621344
2: author_key: 428169
model_extractions: Policy Research Working Paper 10704 What Lies Behind “Good” Analytical Work on Development? Four Years of Knowledge Products at the World Bank _Martín Rama Rucheta Singh Aiza Aslam_ Operations Policies and Country Services Vice-Presidency February 2024
Policy Research Working Paper 10704

# **Abstract**

The World Bank’s analytical work has a strong reputation, but its knowledge products are also perceived to be of varying quality and relevance, and the drivers of this heterogeneity are only partially understood. Building on previous evaluations, this paper adopts a production function approach to assess how budget resources, time to completion, technical skills, and institutional responsibilities affect the internal ratings and external visibility of different types of analytical tasks at the World Bank. To this effect, the paper first matches records from three unconnected electronic platforms—for internal documents, budget codes, and external publications—to assemble a comprehensive database of knowledge products and their key characteristics. With analytical documents as its unit of observation, the exercise shows that: (1) devoting more resources to analytical tasks leads to both better ratings and greater visibility; (2) both outcomes are systematically worse when a greater share of resources comes from trust funds; (3) they are also consistently worse for tasks that take longer to complete; (4) more academically oriented team leaders underperform on ratings and overperform on visibility, whereas technically solid but less stellar team leaders overperform on ratings; and (5) everything else equal, performance varies systematically with the nature of the unit in charge. The findings of the paper can be read as a cautionary note against knowledge management that is based on the counting of analytical tasks. Instead, the findings call for much stronger information systems on knowledge products, a better alignment of incentives for the units in charge, and regular evaluations in the spirit of this paper.

This paper is a product of the Operations Policies and Country Services Vice-Presidency. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at mrama@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
What Lies Behind “Good” Analytical Work on Development? Four Years of Knowledge Products at the World Bank Martín Rama, Rucheta Singh and Aiza Aslam<sup>*</sup> Keywords: Development Economics, Policy Research, Development Effectiveness.

JEL classification: O19, O20, B40, D73.

> * The research underlying this paper was initiated in the summer of 2020 at the request of the Operations Policies and Country Services Vice-Presidency of the World Bank (OPSVP). The authors are grateful to Manuela Ferro and Mamta Murthi for guidance, and to Maurizio Bussolo, Stephane Guimbert, Alexander Korolyov and Aart Kraay for useful comments. The views in the paper are those of the authors and do not engage the World Bank. The corresponding author is Martin Rama (mrama@worldbank.org).
# 1. Introduction

The World Bank prides itself on being a knowledge-based institution, to a much greater extent than most of its peers. And rightly so. The World Bank was one of the first international organizations to appoint a Chief Economist, in 1972; since then, two of the job holders went on to win the Nobel Prize in economics. With the _World Development Report_ , a series introduced in 1978, it pioneered the production of regular flagship studies shaping the narrative on development economics. In 1990, it championed an approach to the measurement of poverty – back then, “one dollar a day” – that has become widely used. Its _Doing Business_ reports, launched in 2002, prompted lively policy debates on regulation, and triggered uncountable economic reforms around the world. In 2011, it embraced an open access approach, putting a vast majority of its reports and data under a Creative Commons license; today its _World Development Indicators_ are among the most widely used cross-country databases worldwide.

In a reputable ranking of top academic institutions in the field of development, the World Bank comes first, followed by its own research department, ahead of the Massachusetts Institute of Technology, the National Bureau of Economic Research, and other prestigious economics departments (RePEc 2022). And operational counterparts in developing countries often report that they value the knowledge they get from the World Bank more than the financial resources it provides (Knack et al. 2020).

Few other international organizations – if any – may match this record. And yet, even seasoned insiders would be hard pressed to explain how the World Bank accomplishes all this. How is rigorous analytical work produced, what ensures that it is relevant to development practitioners, and how it reaches its intended audience seem to be more art than science.

This is not for lack of trying. The phrase “the knowledge Bank” was coined in 1996, and it is certainly catchy. But beyond obvious themes, such as tapping the World Bank’s global expertise, blending development knowledge with policy engagements, and leveraging digital technologies, it is thin on the specifics (Forbes 2020). Over the half-century since the first World Bank Chief Economist was appointed, multiple reforms revamped the way analytical work was funded, how it was conducted and reviewed, and how it was connected to lending (Dethier 2009). Even major reorganizational reforms were justified on the need “to improve knowledge flows and the delivery of multisector solutions to clients” (World Bank 2014). However, outsiders would find it difficult to pinpoint any turning point, from an organizational 2
point of view, that led to a substantial improvement or deterioration in the quality and relevance of World Bank “knowledge”.

At the same time, the quality of the World Bank’s knowledge products is heterogeneous – and at times directly controversial. For example, the rankings and interpretations of the highly visible _Doing Business_ report were the subject of much criticism, leading to its thorough reassessment (Manuel et al. 2013). After almost two decades, its production was discontinued on the grounds that “the current methodology should be significantly modified, implying a major overhaul of the project” (Alfaro et al, 2021).

This ambivalence affects less-renowned knowledge products as well. Both insiders and outsiders would readily admit that many books, reports, and articles by the World Bank fail to get traction with academic audiences, policy makers or public opinion. Some would easily qualify as advocacy, which reduces their credibility as a source of reliable knowledge. Others are simply shallow. Such heterogeneity within the same institution raises questions about the processes through which the World Bank’s knowledge-related undertakings are selected, conducted and vetted.

The goal of this paper is to use an empirical approach to shed light on what makes for “good” knowledge work at the World Bank. With the organization being such a prolific source of analytical products, it should be possible to rely on statistical methods to categorize them, to identify key correlates of their quality or impact, and to pinpoint individual items with higher and lower quality within each of the categories.

The fundamental assumption of the paper is that at the World Bank ideas materialize under the form of documents – books, reports, journal articles, working papers, policy notes, presentations, manuals, and the like (figure 1). Given how large and complex the organization is, these documents are by nature addressed to diverse audiences and serve different objectives. Some intend to stimulate debates on economic and social development in general, while others aim to shed light on pressing challenges confronting a specific country or region. Some build on innovative methodologies to push the research frontier, while others digest existing knowledge to build capacity internally or externally. Some are neutral resources whose implications are to be determined by the user, while others encourage the adoption of concrete strategies and policies.

If all knowledge-related documents produced by the World Bank could be classified in relatively homogeneous categories – say, based on their audience, or their purpose – it would in principle be possible to analyze them using a production function approach. This would require some rating of their 3
<!-- Start of picture text --> ose__, Studies and reports . .<br>Flagships - Technical assistance<br>and books _4 ‘ mt 1| —inte: documents<br>4 gir $3 Seeeeeeeecnpoaynat Power Russian Federatn<br>QUART CONoNYer SSSa ca<br>\ Sic a com Comper<br>oO nN, — Dany 8 in the p<br>o* el ETHIOPIA 2] trey ial 0,<br>As ee RE H Woe, tm aa<br>oe are y log<br>aaa i<br>\ BEEZ EXPENDITURE “on,<br>wie Shp<br>ee "ES ig PelingEy<br>a = ‘ Heng, UNae<br>a Journal | articlesarticles an and Policy notes cae., SS*<br>. working papers<br>Presentations<br>and manuals .<br><!-- End of picture text -->
response to budget processes and managerial needs than in connection with the analytical agenda. As a result, the relevant information is currently scattered across the World Bank.

A contribution of this paper is to painstakingly match entries from three self-standing digital platforms to assemble a consolidated database of knowledge-related World Bank work. One of the platforms is a filing system for documents of all sorts, another one contains information on each of the budget codes the World Bank uses, and the third one hosts all the publications it has put in the public domain.

For each of the pieces of knowledge work identified through this matching process, the consolidated database brings together the information available on a broad set of indicators. On the output side, it may report the internal quality rating at completion and the number of downloads over time. On the input side, it may include the name of the task team leader (TTL), the financial resources and time devoted to the task, and the identity of the units in charge of producing it and supervising it. Important gaps remain, as information is not available on all indicators for all entries, but the result is probably the most comprehensive database of World Bank analytical work produced to date.

The analysis in the paper is conducted for knowledge-related documents completed between July 1<sup>st</sup> , 2015, and June 30, 2019. This choice is determined by the stability of the rules guiding the preparation of analytical work during that four-year period. Indeed, most knowledge-related tasks in the World Bank were by then under the management of central units. These included the research department – or DEC, for Development Economics – as in the past, but also 14 Global Practices introduced on July 1, 2014, as part of a major institutional reorganization. A significant departure from this centralized approach to knowledge generation was six relatively small regional Chief Economist offices, which continued preparing analytical products for the country groupings they served.

This period was also characterized by a relative stability of other relevant criteria, such as the administrative classification of tasks, the identification of “core” knowledge products that all country units are supposed to deliver, the typical budget envelopes by task, and the rules guiding the use of trust funds (TFs). The stability of the context over the four years considered reduces the risk that statistical results may be biased by structural breaks. It also helps address concerns about reverse causality, whereby similar tasks could have been subject to idiosyncratic processes.

The paper proceeds as follows. The next section reviews selected assessments of the knowledge work conducted by the World Bank, summarizing the methodologies they used, and the hypotheses articulated 5
in each case. Section 3 describes the way the database of knowledge products was constructed and how the key attributes of each document were defined. Descriptive statistics from this database are used in Section 4 to show that the universe of knowledge products is broader than that typically receiving the attention of World Bank management, with both sets even displaying opposite trajectories over time. In section 5, knowledge products are classified in several relatively homogeneous categories; for each of them a statistical analysis akin to the estimation of production functions is conducted. The main patterns emerging from these exercises are summarized in Section 6. Building on them, Section 7 concludes by offering several practical recommendations to strengthen analytical work at the World Bank.

# 2. Previous studies

Based on their unit of observation, previous assessments of knowledge work at the World Bank can be classified in four groups. A few of them have focused on _research products_ , the more academic end of the spectrum. Others, more operational in nature, have tried to distill patterns from _budget codes_ that are deemed to generate analytical content. Several studies have put emphasis on the _work processes_ through which analytical products are prepared. And a couple of them have elicited and analyzed _user views_ in relation to World Bank knowledge products.

Evaluations of _research products_ tend to focus on academic recognition, with citations being an obvious measure of the World Bank’s influence on development thinking. One study in this group applied the citations metric to some 8,100 journal articles, around 6,800 books and chapters in books, and close to 3,000 working papers that had been put in the public domain by the World Bank over several decades. The set was assembled by combining records from an internal database of documents and from various external bibliographical repositories (Ravallion and Wagstaff 2011).

Using citation data from the Google Scholar platform, this study revealed the heterogeneous quality of the research products selected. A large share of them – including several _World Development Reports_ – had little discernable influence. However, within specific geographic areas such as Africa, China and India, and specific topics such as education, growth, inequality, health, and poverty, World Bank research products were ranked highly. And a few of them were among the most cited articles in several of the top economics journals. However, beyond noting that ratings tend to rise with years of publishing, this study did not attempt to explain what drove these stark differences in visibility.

6
Because it takes years for research to be published, and only then do citations start to accumulate, another study in this first group, conducted by a group of prestigious academics, chose to evaluate the quality of a sample of research products directly (Banerjee et al. 2006). The universe in this case were the journal articles, books, working papers and flagship reports published between 1998 and 2005. In all, roughly 4,000 research products were considered, of which more than 2,000 were articles in peerreviewed journals. The subset directly evaluated by this study comprised 192 of these products – 180 drawn through stratified random sampling, all the _World Development Reports_ , most of the World Bank’s flagship reports, and a list of “must read” papers and books.

The study found that many of the documents evaluated met “the highest academic standards of originality and technique” and dealt with topics that were highly relevant to the World Bank’s mission. At the same time, however, “much of the research read by the evaluators was seen as undistinguished, and not welladdressed to any particular audience” (Banerjee et al. 2006, p. 46 and 66 respectively). This strong heterogeneity is partly attributed by the authors of the study to the competence of the teams in charge, with a dozen researchers even praised by name.

When reviewing the performance of _World Development Reports_ , the study also comments on a “committee process” potentially affecting the quality of knowledge work at the World Bank. This process “makes it difficult to maintain a coherent and focused argument, especially for controversial topics [...] There is also a tendency to pull political punches [...] There is much political correctness, including mindless cheerleading for cultural touchstones such as women, trees, and social capital [...] Trade-offs tend to be eschewed in favor of ubiquitous “win-win” scenarios” (Banerjee et al. 2006, p. 80-81).

Beyond authorship, the study considered other characteristics of the 192 knowledge products evaluated. A simple statistical analysis revealed that overall quality ratings were virtually identical for research products by DEC and by the rest of the World Bank. DEC was clearly stronger on statistical and econometric methodology and in data handling, whereas the rest of the World Bank did better on clarity of writing and appropriateness of recommendations. Outside DEC, flagship reports produced by regional and sectoral units fared very well, and most of the rest quite poorly. And across the entire sample, ratings were lower for products whose preparation involved small budgets, but also for those with very large budgets (Banerjee et al. 2006, Chapter 3, Annex 2).

Studies in the second group focus on _budget codes_ that are supposed to be analytical by nature. Chief among them are codes such as Economic and Sector Work (ESW) and Technical Assistance (TA). Together 7
with codes used for impact evaluation, external training, and other non-lending tasks, ESW and TA are part of what is known in World Bank jargon as Analytical Services and Advisory (ASA) work. Some of these codes, analytical in nature, support the generation of new knowledge on global issues and the discussion of regional and country policy issues. Others, more hands-on, fund technical inputs for developing country governments and capacity building efforts.

World Bank systems record relevant information about ASA budget codes. On the output side, a rating is reported at completion to judge the performance of the task against the objectives stated at its inception. The rating is proposed by the management of the unit responsible for the task, but adjusted – often downward – by the management of the unit that requested it and funded it. For a few of the budget codes, feedback from operational clients is also sought within the year that follows completion. However, for confidentiality reasons, information of this client feedback is available only with some level of aggregation, not for individual budget codes. On the input side, information is generally reported about spending, time to completion, the name of the TTL, the identity of the unit responsible for the product, and that of the requesting unit, among others.

One of the studies in this second group considered 1,331 ASA budget codes deemed completed between 2006 and 2012 (Doemeland and Trevino 2014). The assessment of these codes relied on the consolidated number of downloads for publications associated with them, although no correction was made for the size of the intended audience – for example, whether the budget code concerned a large region or a small island nation.

With this limitation in mind, the study showed that budget codes whose key reports had been released in English, and had been in the public domain for longer, had a higher number of downloads. Importantly, the distribution of downloads was highly skewed, with only 13 percent of the budget codes leading to more than 250 downloads, whereas a full 31 percent had none.

Studies tapping other information available about these budget codes tend to be more informal in nature. They are carried out mostly as part of regular operational management, with some of them covering the entire range of active ASA codes and others referring to a particular region or a particular sector. On occasion, however, some of these assessments go beyond monitoring and try to draw broader lessons for the knowledge work of the World Bank. For example, toward the middle of the period covered by this paper, one such study reviewed how ASA had evolved between 2003 and 2017 (World Bank 2018).

8
This study found that the number of ASA codes completed annually had grown by 63 percent over the period considered. The composition of the codes had changed as well, with budget codes associated with analytical work declining by a quarter and those supporting advisory activities tripling. The decline was noticeable for the number of budget codes related to “core” analytical products – country economic memorandums, poverty assessments and the like.

The upward trend in budget codes and the change in their composition were attributed by the study to the growing availability of TFs, which are often used to support advisory work. While the average funding provided by the World Bank for a completed task had increased from USD 138,000 to USD 159,000 over the 15 years considered, the study noted that TFs had allowed average lifetime spending per task to jump from USD 208,000 to USD 515,000.

This study also assessed performance at the aggregate level, using the client feedback at completion. It noted that ASA clients valued the World Bank’s ability to bring global expertise but were less impressed by its tailoring to the local context, and by the timeliness of its delivery. However, the study did not attempt to link the performance of individual ASA codes to their characteristics, including total funding or the share of expenditures paid for by TFs.

Studies of World Bank knowledge activities based on _work processes,_ the third group of studies takes a more holistic approach, emphasizing the way internal units and external clients interact. Several such assessments have been conducted by the Independent Evaluation Group (IEG), a unit tasked with providing evidence on the delivery of World Bank services and results to its Board of Directors.

An example of this holistic approach is an evaluation of knowledge-related activities in nine country programs (IEG 2016). These were identified among 48 that make relatively intensive use of the World Bank’s analytical work. The selection was not driven by random sampling but rather by an attempt to illustrate a range of engagement modalities at the country level. Taken together, the nine programs covered 266 ASA budget codes and 34 advisory services provided by the International Finance Corporation (IFC), the private sector arm of the World Bank.

Knowledge work in the nine selected countries was evaluated against four criteria: relevance for the client country, technical quality, results achieved, and sustainability of the results. Ratings on those four fronts were based on internal interviews, stakeholder feedback and desk reviews.

9
This study found that performance was not correlated with budget resources, or with the extent of reliance on TFs. Rather, knowledge products requested by the client and designed specifically to achieve client objectives were more likely to achieve their objectives than those of a more generic character. Knowledge services with fully or partly achieved outcomes also used local expertise, focused on specific sectors rather than broad topics, customized the analysis to local conditions, generated new data in support of policy making, and formulated actionable recommendations. Therefore, the “how” appeared to be more relevant than the “what” to the authors of this study.

The fourth and last group of studies involves conducting surveys to elicit _user views_ on World Bank knowledge products. Depending on the case, the users considered may be operational World Bank staff – for example, TTLs of lending projects – or external stakeholders – including government officials, members of legislatures, civil society, the private sector, academics, and donor representatives.

For example, one study by IEG (2009) sought feedback on the usefulness of 129 ESW and 64 TA budget codes that were deemed in line with the overall distribution of knowledge products delivered by the World Bank between 2000 and 2006. Responses were obtained from 91 TTLs of World Bank lending operations who had not been associated with the ESW and TA codes covered – roughly half of the TTLs approached – and from 353 in-country stakeholders.

The study noted that responses were generally consistent between the two groups. Technical quality received high ratings and dissemination less so, resulting in a mixed assessment of overall quality. ESW products such as global and regional reports were said to have informed lending operations, even when that was not their explicit objective. Loans that had been preceded by ESW products were also said to have better design than those that had not. Lending thus appeared as an important channel through which knowledge products informed developed policies and strengthened institutions at the country level. The effect of ESW products on informing World Bank strategies was significant as well, with “core” analytical products being particularly important in this respect.

In another study, a survey of senior World Bank operational staff was used to assess their awareness about research products, such as those produced by DEC (Ravallion and Wagstaff 2011). There were 2,900 recipients of the anonymous survey instrument used, and 555 respondents. Views were similar across regional units, but markedly different across sectors, with staff in areas of economic management and human development being more familiar with research products than their colleagues elsewhere.

10
However, with the World Bank’s research products also being concentrated in those two sectors, it is difficult to tell whether differences in staff awareness really reflect a respondent bias.

On occasion, the relationship between survey responses and features of the World Bank program in a country, including its knowledge work, has been explored through regression analysis. Thus, a study took as its explained variable the ratings on the agenda-setting influence of the World Bank provided by 1,244 government officials from 121 low- and middle-income countries (Knack et al. 2020). This sample was drawn from a list of 6,371 respondents among 54,990 stakeholders in 126 countries, by retaining only public sector stakeholders who had a direct interaction with the World Bank between 2004 and 2013.

Explanatory variables were organized around country-policy clusters – for example, Madagascar and human development. The variables of interest in the regression, all measured at the cluster level, were the number of ESW activities completed during the period of interest, the number of development policy loans approved, and the number of prior actions these loans covered. A prior action documents a supposedly meaningful policy decision adopted by the government before the approval of the loan – for instance, a reduction in trade tariffs. The regression also controlled for stable differences in views associated with these clusters (fixed effects), for annual disbursements per capita from World Bank policy loans and investment projects, for other country characteristics such as population and income per capita, and for individual characteristics of the respondent including gender and age.

In the eyes of the respondents, the results show, the number of ESW activities completed and the number of policy loans approved had a significant influence on the reform activities undertaken by the recipient country. Actual disbursements from policy loans, on the other hand, might have undermined reform impetus, while the number of prior actions was statistically insignificant. The study concludes that analytical products from the World Bank are effective at influencing the design, direction, and implementation of government policy.

Studies in the four groups are necessarily partial, as they attempt to assess the analytical work of the World Bank from one angle, using only a subset of the potentially relevant information. They tend to deal with a selected but potentially narrow set of knowledge products. They often evaluate output quality, but some do so at the aggregate level while others make comparisons across the output set considered. And they do not always link the quality of knowledge work to the underlying production process but they when they do, they tend to emphasize diverse aspects of it.

11
Yet, when considered together, the studies in the four groups provide a series of _working hypotheses_ that are worth exploring more thoroughly. First, the overall quality of World Bank knowledge work is high, but it is heterogeneous both between and within groups of products. Second, the volume of resources available helps, but only to a point, and the growing availability of TFs may in fact reduce overall quality. Third, timeliness is a weak aspect of World Bank analytical work, without indications that knowledge products that take longer to prepare are of better quality. Fourth, the competence of the teams in charge of producing knowledge work matters for its quality, but it can be undermined by management processes that attend to other objectives. Fifth, within the World Bank, DEC is stronger on technical rigor, but possibly weaker on communication clarity and policy relevance. And sixth, the overall quality of World Bank knowledge work is higher when it is conducted in greater proximity to its intended users, hence with a better understanding of their needs and context.

# 3. A database of knowledge products

An empirical assessment of the six working hypotheses drawn from previous studies requires a database of knowledge products that is comprehensive both in its coverage and in the information set available for each entry. Specifically, data is needed on all the analytical work produced by the World Bank during a specific period, on the quality and reach of each piece, on the competence of the team in charge, on the budget resources devoted to it, on the time it took to complete it, on whether DEC was involved, and on the proximity between the preparation process and the ultimate client.

A comprehensive database along those lines was assembled for this paper by matching information available in three independent digital platforms:

- The _Image Bank_ (IB) is a document filing system where all entries are classified based on their sensitivity, which determines who inside or outside the World Bank can retrieve them. The criteria to decide which documents are filed have some fluidity; only a fraction of them qualify as knowledge products.

- The _Business Intelligence Warehouse_ (WH) is a digital platform organized around budget codes that is accessible only internally. A budget code supports the production of multiple documents, but there is typically one document summarizing its output – for example, a project appraisal document or a technical assistance report.  Analytical work is supposed to be funded through ASA budget codes.

12
- Finally, the _Open Knowledge Repository_ (OKR) makes all World Bank publications available to the public at large. Long-running periodical series and the annual workplans of various World Bank units determine which documents are to be disseminated through this repository. However, as the name of the database indicates, all the documents in it can be deemed related to knowledge.

The matching exercise was conducted for documents filed in the IB database between July 1, 2015, and June 30, 2019, a period that corresponds to the World Bank Fiscal Years 2016 to 2019, or FY16-FY19 for short. In the case of the WH database, the matching concerned all budget codes that had been deemed completed in FY16-FY19, plus all active codes as of June 30, 2019. Including the latter group is justified because documents may be filed in the IB database before the corresponding budget codes are closed. Similarly, to account for delays linked to administrative clearances and production processes, all publications released through the OKR database after July 1, 2015, were included in the exercise.

An important challenge was to determine which of the documents in the IB database had substantial knowledge content. Documents are filed there under one of more than two dozen headings, such as Country Economic Memorandums, or Policy Notes, or Sector Strategies (table A.1). However, several thousand documents were not tagged to any of these ten categories and needed to be individually checked to determine whether they qualified as knowledge products.

Similarly, only a fraction of the budget codes in the WH database support knowledge work. ASA codes in principle do, so they were all included in the matching exercise. But knowledge work is also supported by budget codes funding DEC research, as well as by the preparation of regional and sectoral strategies, country partnership frameworks, and training, among others. Conversely, a considerable amount of knowledge work is produced without mobilizing earmarked budget resources.

The matching process focused first on budget codes, which are at times mentioned in the documents filed in the IB database and in the publications released through the OKR database. Successful matches were indexed by the number of the corresponding budget codes in the WH database, known as P-codes in World Bank parlance. When no P-code could be identified in IB documents or OKR publications, the matching process relied on information about the titles of the entries and the names of their authors. Further confirmation that the matching was correct was provided by the regions or countries these entries covered and the sectors they were mapped to. Entries without associated P-codes were indexed with numbers allocated sequentially; these numbers are identified in what follows as Z-codes.

13
<!-- Start of picture text --> An internal filing system:<br>7,976 knowledge documents<br>Image Bank<br>19,275aitems in all (1B) Without a matching P-code:<br>6,443 Z-codes<br>4,963<br>Open<br>Knowledge<br>Repository<br>(OKR) An external website:<br>A management platform: 6,721 publications<br>12,832 P-codes<br><!-- End of picture text -->
- Task ratings in the WH database were consolidated into three categories. High (= 1) corresponds to “extremely effective”, “highly effective” and “very effective” ratings. Medium (= 2/3) to “satisfactory”, “moderately satisfactory”, “effective” and “moderately effective”. Low (= 1/3) is for the rest.

- Google Scholar, Research Papers in Economics, and the Social Science Research Network were used to assess the technical skill of TTLs. Those who showed up in at least two of these platforms were deemed Top (= 1). Those in one only were considered Good (= 2/3), and the rest Unknown (= 1/3).

- The numerous units in charge of delivering knowledge products were consolidated into four groupings: Global Practices – which also includes central Cross-Cutting Solutions teams dealing with topics such as gender, or climate change – DEC, regional Chief Economist offices, and the rest.

The consolidated database of knowledge products assembled through this process is available on request.

# 4. Composition and summary statistics

While the consolidated database is structured around P-codes and Z-codes, this may not be the most informative breakdown to assess the quality of the analytical products in it. Admittedly, knowledgerelated activities are monitored by World Bank based on ASA budget codes, which are a subset of P-codes. Quite often their oversight involves counting how much was spent on them, and how many activities were completed over a specific period. This approach allows disciplining the use of budget resources and ensuring the delivery of committed products.

As for the quality of the analytical work, the conventional wisdom is that it is inversely correlated to the number of tasks management needs to oversee. A new standard monitoring report for ASA products introduced at the beginning of the period covered by this study was used to drive a substantial “cleanup” of analytical products (World Bank 2019). Since then, considerable managerial effort has gone into ensuring that the number of active ASA products is commensurate with some experience-based indicator of the “capacity” of responsible units to deliver.

However, the number of active ASA budget codes may not provide a reliable assessment of the number of knowledge-related tasks being conducted, due to the significant heterogeneity in coding practices across World Bank units. Consider _Doing Business_ , _Global Economic Prospects_ , and the _World Development Report_ – three of the most visible analytical products of the World Bank. Depending on the 15
<!-- Start of picture text --> Doing Business Global Economic Prospects<br># of entries # of entries<br>250 10<br>200 8<br>150 6<br>100 4<br>° Z<br>o it)<br>FY16 Fya7 Pyle Fis FY16 FY17 FYB FYi9<br>BASA MDB Country/Regional Profiles OGL Other BASS HE GL MRE @ Other<br>World Development Report<br># of entries<br>25<br>20<br>15<br>10<br>5<br>=<br>FYI6 FY1? FY18 FY1S<br>ASA BRA BWD & Other<br><!-- End of picture text -->
|AA|AnalyticalandAdvisory|6675|
|---|---|---|
|EW|EconomicSectorWork|1045|
|IE|Impact Evaluation|116|
|PA<br>|ProgmaticApproach<br>|31g<br>|
|TA<br>|TechnicalAssistance<br>|2333<br>|
|TE<br>|ClientTraining<br>|144<br>|
|RA<br>|ResearchServices<br>|igg<br>|
|RF|ResearchServices|a|
|GL|Global Monitoring|130|
|KP|Knowledge Products|953|
|WD|WorldDevelopmentReport|3|

<!-- Start of picture text --> Working papers<br>(53.4 percent)<br><!-- End of picture text --> = Issues, policy notes and knowledge notes = Reports and publications = DB country and regional profiles = Policy Research Working Papers

|CP|Country Partnership|253|
|---|---|---|
|PE|IDAGuarantee|76|
|ES/EP/ER/EC|<br>Evaluations|30|
|RE|RecipientExecuted Activity|10|
|0«-~PT|Partnership|8|
|CN/GE|Carbonoffset/GEF|6|
|(OP|<br>Sectorstrategy<br>|2|
||<br>.||

<!-- Start of picture text --> Country engagements<br>(9.3 percent)<br>Journal articles<br>(6.8 percent)<br>Doing Business inputs<br>(11.6 percent)<br>Knowledge management<br>(18.9 percent)<br>= Economic updates and monitors<br>= Journal articles<br>= Briefs, newsletters and digests<br>= Other working papers<br><!-- End of picture text -->
# Knowledge products per year

<!-- Start of picture text --> 4500<br><!-- End of picture text --> <!-- Start of picture text --> 4000<br>3500 a<br>3000<br>2500<br>2000<br>1500<br>1000<br>500<br>0<br>FY16 FY17 FY18 FY¥19<br>© Dashboard ASA ® Other Dashboard ® DEC Products M® Hidden M Other<br><!-- End of picture text -->
**Table 2. A functional tiering of World Bank knowledge products**

|**Tier**|**Knowledge products included**|**P-code**|**Z-code**|
|---|---|---|---|
|Flagships|World Development Report; Doing Business; Global Economic<br>Prospects; regional and sectoral flagships.|132|52|
|Debates|Regional, sectoral and multi-country reports; Doing Business<br>regional reports.|2219|628|
|Extended<br>core|Country Economic Memorandums; Poverty Assessments; Systematic<br>Country Diagnostics; Public Expenditure Reviews; Reimbursable<br>Advisory Services; Financial Sector Assessment Programs; Risk and<br>Resilience Assessments; Country Private Sector Diagnostics; Human<br>Capital Reports; Infrastructure Sector Assessment Programs; Debt<br>Management Performance Assessments; Debt Sustainability<br>Analyses; Agricultural Sector Reviews; Pandemic Preparedness<br>Reports; Environmental Assessments.|1500|6|
|Other<br>country|Policy Notes; Economic Updates; other studies; technical assistance.|6103|1258|
|Research|Journal articles; working papers; impact evaluations; Doing Business<br>country profiles.|1725|3927|
|Capacity|Toolkits; databases; conferences; courses; training.|488|82|
|Internal|International Development Agency papers; strategy documents;<br>briefs; newsletters; evaluations.|665|490|

# 5. Regression analysis

The construction of a consolidated database of knowledge products containing tens of thousands of observations, each with information on its performance and the process through which it was prepared, makes it possible to assess what makes for “good” analytical work at the World Bank using statistical tools. The assessment is in the spirit of a production function, with a measure of performance as the explained variable, and a series of “inputs” to the preparation of knowledge products as the explanatory variables.

The quality of World Bank knowledge products is judged from two complementary perspectives. The rating given to a task by the management of its requesting unit indicates the extent to which it attained the objectives set at its inception, hence of its attainment of expected standards. The number of downloads per million population in the countries or regions that were the focus of the task provides an indication of the visibility it attained, hence of its potential development impact.

19
# **Table 3. Summary statistics by tier of knowledge products**

|**Indicators**|**Flagships**|**Debates**|**Extended**<br>**core**|**Other**<br>**country**|**Research**|**Capacity**<br>**building**|**Internal**|
|---|---|---|---|---|---|---|---|
|_Rating_High (%)|11.4|13.3|16.4|14.0|7.5|10.7|2.0|
|Medium (%)|0.0|0.9|0.7|1.0|0.5|0.4|0.3|
|Low (%)|11.4|28.1|28.5|41.5|21.0|19.3|3.2|
|Unknown (%)|77.3|57.6|54.4|43.5|71.1|69.7|94.6|
|_Downloads_Mean (per million)|613.0|62.4|18.1|19.4|12.7|8.6|5.0|
|Unknown (%)|71.2|89.7|59.8|87.2|75.5|95.5|91.0|
|_Cost_World Bank (K$)|793.8|155.0|152.0|114.5|186.6|156.4|215.0|
|Trust fund (K$)|244.4|413.2|55.2|255.6|373.1|277.4|176.7|
|Total cost (K$)|1064.0|592.2|394.4|383.8|582.1|442.9|406.3|
|Unknown (%)|6.1|11.9|8.4|11.1|9.1|15.8|11.7|
|_Time to_Mean (months)|54.7|26.0|21.2|21.5|41.4|31.8|31.1|
|_completion_Unknown (%)|0.8|0.4|0.3|0.3|0.9|0.2|3.8|
|_TTL_Top (%)|13.6|6.8|2.3|3.6|26.8|4.9|2.4|
|_profile_Good (%)|2.3|4.0|5.1|4.3|4.2|1.8|2.9|
|Known (%)|4.5|4.4|9.9|5.2|3.6|4.7|7.4|
|Unknown (%)|79.5|84.9|82.7|86.8|65.4|88.5|87.4|
|_Responsible_DEC (%)|11.7|1.2|0.0|0.4|11.5|8.2|1.2|
|_unit_Global Practice (%)|45.2|41.8|0.6|0.7|4.0|41.2|46.4|
|Regional office (%)|2.7|23.4|83.7|70.1|12.4|23.3|1.5|
|Other (%)|12.2|11.6|11.7|11.6|2.7|12.8|11.6|
|Unknown (%)|28.2|22.1|0.4|17.1|69.5|14.4|42.7|

20
These two measures focus on different aspects of performance; not surprisingly, their correlation across the entire database is -0.11. The relevance of the two measures also varies across task tiers; for example, a knowledge product prepared for internal use by the World Bank should be expected to meet high standards but not to have a substantial number of downloads. With these caveats in mind, running all production function estimations on both measures should provide more reassurance on the validity of the estimates. If a particular input in the production function appears to significantly impact both ratings and downloads, then it can be safely concluded that it does matter for the overall product quality.

Inputs are selected in a way allowing to evaluate the four working hypotheses articulated after reviewing previous studies on the correlates of good analytical work at the World Bank. A first group of inputs includes the amount of resources devoted to a task, the TF share of those resources, and the time allowed for its preparation. These are the inputs emphasized by studies that take _budget codes_ as their unit of observation. Second is the profile of the TTL, a potentially significant quality determinant according to studies focused on _research products_ . And third is the type of World Bank unit in charge of preparing the knowledge products which, based on studies dealing with _work processes_ and _user views_ , could affect the relevance of the tasks and their attention to client needs.

The production function analysis also controls for the fiscal year when a knowledge product was delivered and for the geographic region it covered. This is to account for systematic differences in performance over time and across space. For example, it could well be that the broader context in which the institution operated in a particular fiscal year affected management attention to knowledge products. Or that clients in higher-income regions demand better-quality knowledge products from the World Bank. Different degrees of internet penetration across countries should affect the number of downloads as well.

All regressions start parsimoniously, including only the first group of inputs, then add the TTL profile, and finally the type of unit in charge. Fixed effects for fiscal years are included in all specifications, but estimates are presented with and without fixed effects for geographic regions. This sequential approach allows assessing whether the resources and time devoted to knowledge products are robust determinants of their quality. They would not be if the estimated impact of the first set of inputs mattered in the initial, parsimonious estimation but lost significance when controlling for the other inputs.

Finally, the analysis needs to take into consideration the fundamental heterogeneity of World Bank knowledge products. A crude way to do so is to include all observations in the estimation but to add to the specification dummy variables for the various tiers of knowledge products identified above. Proceeding this way amounts to assuming that inputs have the same impact on performance across all 21
tiers, but the very nature of the products in each tier systematically affects their ratings, or their number of downloads. For example, the dissemination process for flagships could result, other things equal, in more downloads compared to other knowledge products.

However, it is plausible that the production function of analytical products differs across tiers. For example, a TTL with a strong technical profile may be indispensable in the preparation of a flagship report, less so for other analytical tasks. To account for this possibility, regressions are also run separately for two different subsets of observations. One of them includes all major reports, or flagships and debates in terms of the proposed tiering. The other contains all country-specific knowledge products, corresponding to the extended core and other country tiers. Even so, a dummy variable is included to capture any remaining differences between the two tiers lumped in each of the two subsets of observations.

In all, then, six regression analyses are run. The first three have ratings as the explained variable, but differ in their sample, which sequentially includes all observations with data, only flagships and debates, and only extended core and other country products (tables A.2 to A.4). The other three regressions use the same samples sequentially but have downloads as their explained variable (tables A.5 to A.7).

All regressions are run using ordinary least squares (OLS). The first set of regressions could in principle be estimated using a Logit or a Probit model, given that the explained variable is ordinal, not cardinal. Relying on OLS implicitly assumes that the difference in quality between a high and a medium rating is the same as the difference between a medium and a low rating. However, the apparently greater precision of a Logit or a Probit regression would be misleading. The high, medium and low categories for ratings are themselves built by aggregating more granular qualitative ratings, such as satisfactory, moderately satisfactory, effective and moderately effective. Such aggregation was needed due to the number of options available changing during the period covered by this study. In practice, the results of Logit and Probit regressions are similar to those obtained with OLS, but this illusory extra rigor would come at the expense of the fit of the regressions (results are available on request).

Because information is not available on all correlates for all knowledge products, the number of observations included in the regressions is substantially lower than the total number of knowledge products. Information on cost, use of TF resources and time to completion is only available for P-codes. Because there three correlates are included in all specifications, all regressions exclude Z-codes.

With the regression analyses covering about 60 percent of all knowledge products, results should be interpreted with caution. Indeed, the observations included in the regressions could differ systematically from those excluded, in which case the results may not be extrapolated to the entire sample. For example, 22
working papers and journal articles – typically recorded as Z-codes – are underrepresented in the regressions. Whether results are valid for the entire category of analytical products considered depends on the similarity – or lack thereof – between included and excluded observations.

# 6. Main results

The large number of regressions that results from combining two explained variables, six specifications and three samples may somewhat blur the findings. To handle this complexity, summary indicators across all specifications can be constructed for each of the regressions, resulting in a heatmap where color indicates sign and intensity the robustness of the result. Thus, dark green (red) means that the estimate is significantly positive (negative) and of the same sign in at least half of the specifications. A lighter color is used when less than half of the estimates meet the criterion above, and no color at all when all estimates are statistically insignificant, or some of them are significant but with opposite signs. The significance threshold is set at 10 percent, which is relatively low but yields a more granular heatmap (table 4).

**Table 4. A summary of results from the regression analyses**

|||**Ratings**<br>|||**Downloads**<br>||
|---|---|---|---|---|---|---|
|**Explanatory variables**|**All knowledge**<br>**products**|**Flagships**<br>**and debates**|**Country**<br>**products**|**All knowledge**<br>**products**|**Flagships**<br>**and debates**|**Country**<br>**products**|
|Total cost (K$)|||||||
|Trust funds (percent of cost)<br>Time to completion (months)|||||||
|Top TTL|||||||
|Good TTL|||||||
|DEC|||||||
|Global Practice|||||||
|Regional CE office|||||||
|FY17|||||||
|FY18|||||||
|FY19|||||||
|F-test|||||||

_Note_ : Dark green (red) indicates that the estimate is significantly positive (negative) and of the same sign in at least half of the specifications. A light color indicates that this criterion is met by less than half. No color is reported when all estimates are statistically insignificant, or some are significant but with opposite sign. The significance threshold is 10 percent.

23
The overall significance of the regressions, reflected in their F-test, allows assessing the relevance of the production function approach chosen in this paper. The approach is robust to explain variation in ratings, both across the entire sample and for the selected subsets of knowledge products. It can also account for some of the dispersion in downloads across knowledge products, but its explanatory power is lower than for ratings, suggesting that there are other important factors at play, in addition to the correlates identified based on previous studies.

The sign and significance of the coefficients associated with those correlates can in turn be used to assess the working hypotheses that had been derived from previous studies. Starting with funding, across the entire sample there is a statistically significant association between the volume of resources devoted to analytical tasks and both their ratings and their downloads. This result also holds, in the case of flagships and debates, although it is somewhat weaker. On the other hand, the impact of greater resources is statistically insignificant in the case of country products.

For a given cost, performance on both ratings and downloads declines as the share of resources coming from TFs increases. This unambiguous result may be due to management devoting less attention to the way third-party resources are used, relative to its own resources. However, worse performance could also reflect a selection bias, if TFs tended to be allocated to analytical tasks that are less relevant and thus tend to get lower oversight and lower appreciation. Only in the case of country products is reliance on TFs uncorrelated with the number of downloads.

Another key resource is the time available to complete an analytical task. A natural presumption is that a greater time allocation results in a better product. However, the estimated coefficients indicate that the longer an activity takes to be completed, the lower its rating and downloads tend to be. This result is highly significant both across the entire sample and for flagships and debates. However, the time to completion does not seem to affect the performance of country products, suggesting that more idiosyncratic factors are at play in their case.

Expectedly, the profile of the TTL matters for the performance of analytical tasks, but with important nuances. Good TTLs are associated with significantly higher ratings in all the samples considered but not with a larger number of downloads. The opposite is true for more academically oriented TTLs, who receive significantly lower ratings internally, but generate a greater visibility for the World Bank. These results highlight the value of strong technical skills when conducting analytical work, but they also reveal a different comparative advantage across TTLs. Those who have some academic recognition seem better 24
suited to interact with external audiences, whereas solid but less stellar ones may be more able to deliver the type of analytical products World Bank management seeks.

The nature of the unit in charge of overseeing analytical tasks matters too. Across the entire sample, both ratings and downloads are significantly higher for tasks managed by regional Chief Economist offices. Flagships and debates also get more downloads too, but not necessarily better ratings relative to other units in the World Bank. Performance is definitely more mixed for country products, with tasks managed by regional Chief Economist offices getting significantly higher ratings, but also significantly lower downloads. This contrast could be indicative of findings and recommendations that are not so much appreciated by local audiences.

Results are more mixed in the case of Global Practices. Their flagships and debates clearly do not stand out. However, across both the entire sample and the subsample of country products, the tasks they manage get consistently higher ratings and consistently lower downloads. This contrast suggests a greater ability of Global Practices to cater to the needs of World Bank management than to the curiosity of external audiences.

Perhaps the most puzzling findings concern DEC, which is arguably the World Bank unit with the strongest research credentials. Internally, analytical products managed by DEC get somewhat better ratings than those of other units across both the entire sample and the subsample of country products, but not in the case of flagships and debates. On the other hand, DEC analytical products do not get greater visibility in general, nor in the case of flagships and debates. And their visibility is significantly lower in the case of analytical products for specific countries.

DEC’s relatively muted splash with broader audiences, despite its unquestionable analytical firepower, could be a consequence of its unique profile. With its greater focus on the debates that agitate academics, and on the methodologies that push the research frontier, DEC may not be ideally positioned to identify the development challenges country audiences are grappling with, and to deliver the simple but practical messages they are looking for.

In addition to assessing the working hypotheses drawn from previous studies, the regression analyses above shed light on the quality of the World Bank’s analytical work across time and space. Starting with time, ratings are significantly higher in FY17-FY19 than in FY16. This result is plausible, as the period considered in this paper immediately followed a major reorganization of the World Bank, one that might have temporarily detracted management attention from the supervision of analytical tasks.

25
Across the entire sample, the results also show some increase in the number of downloads. But this apparent improvement may simply reflect the progress in internet penetration and a greater savviness of task teams to disseminate their findings online. This upward trend is also visible, to some extent, in the case of flagships and debates. On the other hand, the visibility of analytical country products by the World Bank remains stagnant despite the obvious digital advances, and even diminishes toward the end of the period considered.

The disconnect between improving ratings and stagnant or even declining downloads for country products could reflect a drift toward self-complacency in evaluating analytical tasks, but this is a hypothesis that cannot be evaluated with the information at hand. On the other hand, the regression analyses allow identifying where high ratings coexist with limited downloads, or the other way around. This could be done for individual analytical tasks, by comparing their predicted rating and downloads to the actual values of both indicators. However, margins of error may be too large for the findings to be conclusive, making it preferable to estimate performance gaps at some level of aggregation.

Focusing on geographic areas is potentially informative in this respect, as operational units at the World Bank tend to match sovereign national jurisdictions and country groupings connected by their history. Country management units are also a major source of demand and funding for the institution’s analytical work, and the main interface with its intended audience.

How ratings and downloads vary across such units can be assessed by replacing regional dummies by country dummies in the regressions above. For this alternative regression exercise, dummy values are set equal to one if the analytical task focuses on the country or on the broader region the country belongs to; they are set equal to zero otherwise. The most comprehensive specification, including all correlates, is used for this exercise (figure 6).

When proceeding this way, the estimated coefficients associated with the correlates resemble those obtained using regional dummies (results are available on request). And the coefficients associated with the country dummies are jointly significant. While relatively few of them are individually significant, these estimates remain the best available predictor of country effects.

Other things equal, ratings were substantially higher in much Latin America and the Caribbean, in several countries in Europe and Central Asia, in parts of Southeast Asia and in most of West Africa. But the geographic distribution of country effects for downloads was quite different. The visibility of the World Bank’s analytical work was indeed concentrated in fewer countries, and especially in Bolivia, Ethiopia and 26
<!-- Start of picture text --> ]<br>: “C se oe :<br>%, = el } v .<br>MEHH Highest “ td . si . a<br>w SecondThird tier tier 3 as) eg<br>Lowest ¥ id‘4 ; ¥<br><!-- End of picture text --> <!-- Start of picture text --> MH | See SS<br>ME Highest “ Lee Ne son<br>© Second tier _ as XMad See “e<br>: ThirdLowest tier > r;24 es .<br>[. e<br>IBRD 47743<br>JANUARY 2024|<br><!-- End of picture text -->
# 7. Conclusions

The analysis in this paper confirms that the World Bank is a prolific producer of analytical work: much more prolific in fact than what its information systems for knowledge management suggest. By focusing on documents, and by matching data about them from three independent digital platforms, this paper shows that the number and scope of analytical products delivered over the four years considered for the analysis vastly exceeded the number of analytical budget codes overseen by the World Bank.

Discrepancies between the two counts are not stable, both throughout the years and across similar products at any point in time. Starting with years, the gap between the number of analytical products delivered and those captured by information systems varied substantially over time. As for products, the number of tasks associated with major analytical undertakings – such as the World Development Report, Doing Business and Global Economic Prospects – was different as well, despite their institutional relevance supposedly being similar.

By focusing on the entire set of knowledge products, and by considering many of their characteristics at once, the paper assessed the validity of several received hypotheses about the drivers of good analytical work at the World Bank. These hypotheses had been formulated by previous studies that relied on different units of observation and focused on partial sets of task characteristics. The findings of the paper validate most of the hypotheses, but the broader dataset used yields empirical regularities that are arguably more precise and robust than those of previous studies. The analysis also leads to novel findings.

Specifically, results show that devoting more resources to an analytical task leads to better ratings but not to a greater number of downloads, and that ratings are systematically worse when resources come from TFs. More time devoted to a task also tends to be associated with poorer ratings, without a noticeable impact on visibility. The paper also finds that the profile of TTLs matters as well, but not in a linear manner. More academically oriented TTLs underperform on both ratings and downloads, whereas technically solid but less stellar TTLs overperform. The pattern is similar in relation to the unit in charge. The more research-oriented DEC underperforms in ratings relative to both Global Practices and regional Chief Economists offices. The latter also outperform other units in terms of downloads.

Some of the results in the paper will hopefully be useful for World Bank management when launching new analytical tasks or monitoring existing ones. They may also be of interest for other development institutions when thinking about their own analytical products.

28
The paper can also be read as a cautionary note against knowledge management that is based on counting analytical tasks. For quite some time, it has been a common practice at the World Bank to try to align the number of active ASA budget codes with some “capacity indicator” established in a relatively crude way, based on what was delivered in previous years. In an institution like the World Bank there are simply too many analytical products underway at any point in time for this to be an effective approach, especially when a large share of such products is not even “visible” through information systems.

Instead, the methodology and findings of the paper can be seen as a call for a different knowledge management approach, one that relies on stronger information systems, a better alignment of incentives, and regular monitoring and evaluation based on statistical analysis.

Starting with information systems, a single identifier for analytical tasks would allow connecting data on all the resources they use, all the outputs they deliver, all the downloads they receive, and all the feedback from internal and external clients. Budget identifiers (P-codes) are a practical way to connect information that is currently scattered across various disconnected systems. At present, no TF can be created in the World Bank without a link to a P-code. Arguably, it should be the same for every document entered into the IB and OKR digital platforms.

The process could be hardwired, facilitating an automatic linking of the three databases used in this paper. A similar approach could be used to connect budget codes with the user feedback collected from country engagement surveys. Proceeding this way, a time will come when it will be possible to conduct data-driven studies like the one in this paper without having to go through a laborious reconciliation of entries.

On incentives, thousands of analytical tasks running in parallel call for decentralized management. But knowledge products differ in the objectives they serve, their expected outcomes, and the risks they pose to the World Bank. Ideally, the decision maker in each case should be the unit that benefits the most from a high-quality analytical product and suffers the most from a flop. This is the approach currently used by the World Bank for research products, which are mainly managed by DEC with an eye on its academic standing and without much interference from other units.

To some extent, the current Accountability and Decision-Making (ADM) framework does the same for other analytical tasks, spelling out clear protocols on how the requesting unit and the responsible unit should interact over critical steps in the process. But the ADM’s focus is more on processes rather on incentives. A second practical recommendation of this paper would thus be to use its proposed tiering of analytical tasks to better align responsibilities in the case of high-return-high-risk tasks, and to streamline processes in the other cases (table 5).

29
Table 5. Allocating management responsibilities to units with the highest stakes

|**Tier**|**Main risk**|**Process**|**Decides**|**Concurs**|
|---|---|---|---|---|
|**Flagships**<br>**Debate**|**World Bank’s reputation**|**Yes**|**Managing Director**<br>**Vice-President**|**Chief Economist**<br>**or delegate**|
|**Extended core**<br>**Other country**|**Client relationship**|**(ADM)**|**Country Director**|**Regional Chief**<br>**Economist**|
|**Research**|**Technical quality**||**Academia**|**No**|
|**Capacity**<br>**Internal**|**Resource waste**|**No**<br>**(Agile)**|**Budget holder**|**(Agile)**|

Finally, a third recommendation is to conduct more regular assessments of the quality and impact of analytical work across the entire institution, along the lines of what was done in this paper. Without aiming to provide a judgment on any individual knowledge product, data-driven studies of this sort allow assessing the overall performance distribution, and to identify the correlates of success. They would also allow to distinguish performance across various dimensions, such as technical quality, external visibility, and perceptions by end-users. This information, in turn, could be used to periodically adjust and upgrade knowledge management across the entire institution.

30