metadata: authors: 0: author: Goldemberg,Diana
1: author: Jordan,Luke Simon
2: author: Kenyon,Thomas
count: World
docna: 0: docna: Minding the Gap : Aid Effectiveness, Project
            Ratings and Contextualization
docty: Policy Research Working Paper
abstracts: cdata!: This paper applies novel techniques to
            long-standing questions of aid effectiveness. It first
            replicates findings that donor finance is discernibly but
            weakly associated with sector outcomes in recipient
            countries. It then shows robustly that donors' own
            ratings of project success provide limited information on
            the contribution of those projects to development outcomes.
            By training a machine learning model on World Bank projects,
            the paper shows instead that the strongest predictor of
            these projects&#x2019; contribution to outcomes is their
            degree of adaptation to country context, and the largest
            differences between ratings and actual impact occur in large
            projects in institutionally weak settings.
display_title: Minding the Gap : Aid Effectiveness,
            Project Ratings and Contextualization
listing_relative_url: /research/2023/07/34127116/minding-gap-aid-effectiveness-project-ratings-contextualization
topicv3: Aid Dependence m1326279 1111
publishtoextweb_dt: 2023-07-31T00:00:00Z
dois: 10.1596/1813-9450-10532
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
author_keys: 0: author_key: 1760342
1: author_key: 1159062
2: author_key: 605610
model_extractions: Policy Research Working Paper 10532 Minding the Gap Aid Effectiveness, Project Ratings and Contextualization _Diana Goldemberg Luke Jordan Thomas Kenyon_ Independent Evaluation Group July 2023
Policy Research Working Paper 10532

# **Abstract**

This paper applies novel techniques to long-standing questions of aid effectiveness. It first replicates findings that donor finance is discernibly but weakly associated with sector outcomes in recipient countries. It then shows robustly that donors’ own ratings of project success provide limited information on the contribution of those projects to development outcomes. By training a machine learning model on World Bank projects, the paper shows instead that the strongest predictor of these projects’ contribution to outcomes is their degree of adaptation to country context, and the largest differences between ratings and actual impact occur in large projects in institutionally weak settings.

This paper is a product of the Independent Evaluation Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at dgoldemberg1@worldbank.org, lukej@mit.edu, and tkenyon@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Minding the Gap: Aid Effectiveness, Project Ratings and Contextualization

Diana Goldemberg<sup>1</sup> , Luke Jordan<sup>2,</sup> , Thomas Kenyon<sup>1</sup> _Keywords:_ aid effectiveness, machine learning, World Bank projects JEL Codes: O12, O15, O19

> _⋆_ The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.

> _⋆⋆_ The authors would like to acknowledge the valuable contributions of Lily Chu, St ́ephane Guimbert, Jed Friedman, Michael Woolcock, Lily Tsai, Daniel Honig, Justin Shenk, Jos Vaessen, Christopher Nelson, Stephen Francis Pirozzi, as well as seminar participantsat the MIT GOV/Lab Seminar and the RMES Results Peer Learning Series. All errors are our own. _Email addresses:_ `dgoldemberg1@worldbank.org` (Diana Goldemberg), `lukej@mit.edu` (Luke Jordan), `tkenyon@worldbank.org` (Thomas Kenyon)

> 1World Bank Group 2MIT
# **1. Introduction**

Numerous empirical studies have investigated whether foreign aid effectively improves development outcomes in recipient countries. This literature has relied mainly on two levels of analysis. One focuses on the aggregate country-level impacts of aid, typically on economic growth or sector outcomes. Another takes a micro-level approach, with development projects as the unit of analysis, most often using donors’ own ratings of project outcomes as a measure of effectiveness.

This study bridges those two strands of research by focusing on the association between donor-financed projects and observable development impact, treating project ratings as intermediating variables. This enables us to ask whether project ratings convey information about those outcomes. We use ratings from projects undertaken in 183 developing countries by eight donors since the 1990s, concentrating on a few service delivery sectors with readily available data on beneficiary-level outcomes. We succeed in replicating previous findings of small positive effects of aid on sector outcomes. However, our results suggest that the project ratings convey little information about impact.

The second and more important contribution of this study is to describe and analyze the correlates of projects’ contributions to improvements in sector outcomes. Focusing on projects undertaken by the World Bank, for which more granular information and extensive text documentation are available, we use state of the art methods to assess what aspects of a project’s production process are associated with stronger outcomes. We first create what are called “text embeddings” of project documents using the latest generation of transformer models,<sup>3</sup> turning texts into numerical representations of their similarity and differences. Then, we train machine learning models to predict projects’ sector outcomes, and probe what features of the projects the model paid most attention to. We find that projects with what appear to be high degrees of tailoring to country context and concentration of funds in fewer sectors are associated with stronger outcomes. In doing so, we use newly available data on project characteristics and draw on methodological advances at the intersection of causal inference and econometrics with machine learning. To our knowledge, this is the first attempt to quantify the importance of project contextualization to development effectiveness.

Our findings have actionable implications for the system through which the World Bank and other development institutions evaluate project performance, by offering a cautionary tale against the over-reliance on project ratings as impact metrics. It also has implications for the design and staffing of these projects.

> 3The same class of models that power all state-of-the-art translation, search engines, AI text generators, as well as most plagiarism detectors.

1
# **2. Literature and Theory**

# _2.1. Development Effectiveness_

A burgeoning literature on aid effectiveness has focused on development projects as the unit of analysis, examining the association between project characteristics and country-level variables on the one hand and project success on the other (for a summary, see Ashton et al. 2023). The most commonly used measure of project success is donors’ ratings of project outcomes, which this literature considers a noisy but valid measure of project performance (Denizer, Kaufmann, and Kraay 2013). Explanatory factors for project success cluster around: (i) country characteristics, such as institutional quality, political and economic stability, and regime type; (ii) project characteristics, such as duration, size, sector, and lending instrument; (iii) aspects of project design, such as the clarity of results frameworks and the number of components; and (iv) aspects of project supervision, such as the intensity, timing, and continuity of oversight.

The utility of this literature is limited in two respects: first, in that no study of which we are aware succeeds in explaining more than 30 percent of variance of project outcomes ratings; second, in that there are grounds for questioning the meaning of these ratings in the first place. Individual donorfinanced projects often anticipate and rate only local impacts, seldom claiming a linkage to economic growth or country-level outcomes. Yet, the local and national effects of aid projects are linked by definition; the total impact of foreign aid upon sector outcomes must be associated with the cumulative effect of the individual projects. Nevertheless, it is an almost entirely neglected empirical question as to whether these ratings are indeed correlated with the contribution of external financing to development. We address this gap by investigating whether project ratings convey information on observable development impact.

A related literature strand focuses on the aggregate country-level impacts of aid. A few studies have attempted to estimate the relationship between donor financing and sector outcomes in education, energy, health, sanitation, and water. They employ similar strategies: panel data estimation techniques controlling for country-specific effects and potential endogeneity of regressors, with sector outcomes as the dependent variable and aid flows as the central explanatory variable. Mishra and Newhouse (2009) measure the reduction in the infant mortality rate associated with increases in health aid per capita. Birchler and Michaelowa (2016) examine the effect of education aid per capita on net primary school enrolment rates. And Ndikumana and Pickbourn (2017) investigate whether foreign aid to the water and sanitation sector has helped to expand access to water and sanitation services in Sub-Saharan Africa. These studies do not measure or estimate the relevance of the various characteristics of development financing identified in the broader literature beyond its volume; nor do they reflect the contribution of non-lending assistance, such as analytical work, to development outcomes. They do nonetheless provide a point of departure for our analysis, and we replicate their results of small but positive effects of aid on sector outcomes.

2
# _2.2. The Role of Projects_

Projects are essential vehicles of development assistance, functioning as an intervening and determinative structure between individual interventions and sector outcomes.<sup>4</sup> Their role encompasses not just the financing but also the adaptation of interventions to local context, their implementation, evaluation, and replication through the policy cycle. In doing so, they remediate the ‘implementation gap’ between what is planned, or conceived on the basis of what might have worked elsewhere, and what is achieved. Variations in project characteristics can also undermine the generation of inferences from randomized control trials and limit the extent to which they may be extrapolated across sites.<sup>5</sup> It is not unreasonable to expect that projects should cumulatively be associated with sector outcomes. At least since the early 2000s, aid agencies have increasingly combined project financing with technical assistance, strengthening government systems and claiming to improve the quality of a program of government expenditures beyond their own financing. Their effect should be detectable not just on the management of interventions financed by the project but, through institutional spillovers, on other areas of government activity.

Qualitative analyses of the effectiveness of development projects have emphasized goodness of fit with country circumstances as a critical determinant of success. This is partly because, to be successful, any policy has to be not only technically correct but also politically supportable and administratively feasible (Moore 1995); partly because technical correctness itself requires judgment as to the similarity between local context and the factors that determined the outcome of an intervention elsewhere. To the extent that this dance of contextualization occurs in World Bank projects, it is mostly during project preparation. But it has largely been altogether ignored in the quantitative literature on project outcomes, for want of measurability.

The scope of prior analyses has instead been constrained by the ready availability of publicly-disclosed data on aspects of project design and supervision. These are for the most part either only weakly linked in theory to project effectiveness or only rough empirical proxies for theoretically-relevant variables. Thus, for example, while a few studies have attempted to evaluate the contribution of economic analysis or clear results frameworks to project effectiveness, most have restricted themselves to easily observable characteristics like size, duration, sector and sources of financing, often with inconsistent findings. To the extent that they have examined the role of donor agency staff, this has been limited to the project manager, with little attention to other participants in the process. Similarly researchers have depended on country-level measures of institutional quality, even though familiarity with and capacity to implement

> 4For a fuller description of the role of projects and their place within a broader conceptual framework, see Section 1 of Ashton et al. (2023).

> 5For more on implementation gaps see Williams (2019) and for a discussion on similar issues in evidence-based medicine, see Ford and Norrie (2016).

3
donor-financed projects varies significantly within countries.

We would expect projects to be more effective when they (i) incorporate prior analysis of the conditions under which an intervention functioned elsewhere and awareness of any material differences between it and the context to which it is to be transplanted; (ii) identify any necessary adaptations and resist external pressure towards over-rapid or unthinking replication; (iii) provide the financial and human resources needed to implement the project (Honig 2018). The likelihood of their doing so depends on a process involving not just the project manager, but the leadership and other team members on the donor side, and a project implementation unit generally staffed by civil servants on the government side. All investment projects also depend to a greater or lesser degree on the effectiveness of government procurement and financial management systems. These inputs are often poorly captured by standard indicators of bureaucratic quality (Blum 2014).

# _2.3. The Project Evaluation Process_

The Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD-DAC) has long spearheaded an agenda on evaluation practice, encouraging analysis of aid effectiveness and results (instead of only inputs and activities), publishing its first set of principles for evaluation of development assistance in 1991. Nowadays, most bilateral and multilateral donors have an established process for evaluating their development effectiveness, aligned with OECD-DAC’s normative framework that consists of six evaluation criteria – relevance, coherence, effectiveness, efficiency, impact and sustainability.<sup>6</sup> At the World Bank, project evaluations are overseen by the Independent Evaluation Group (IEG). IEG rates several aspects of project performance, but the focal metric - reported most saliently to its Board and most commonly used by researchers - is the ‘outcome’ rating, which assesses whether the project achieved its stated objectives. The ratings are the culmination of a two-stage process: first the project management’s own self-evaluation – the Implementation Completion and Results Report (ICR) – and subsequently the ICR Review (ICRR), in about 20 percent of cases followed two years later by a more detailed report, the Project Performance Assessment Report (PPAR), both conducted by IEG. Together these lead to a six-point outcome rating, ranging from highly unsatisfactory to highly satisfactory.

The other seven donors in our database - the Asian Development Bank (ADB), the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM), the German Society for International Cooperation (GiZ), the International Fund for Agricultural Development (IFAD), the Japan International Cooperation

> 6Together they describe the desired attributes of interventions: all interventions should be relevant to the country context, coherent with other interventions, achieve their objectives, deliver results in an efficient way, and have positive impacts that last.

4
Agency (JICA), the German Development Bank (KfW) and the United Kingdom’s Department for International Development (DFID) - similarly summarize their self-evaluations in a single ‘outcome’ rating. Precisely due to their widespread availability and ease of harmonization, the use of such project outcome ratings is prevalent in the aid effectiveness literature.

However, there are several grounds for doubting whether these outcome ratings capture either donor contribution or the likelihood of sustained improvements in development outcomes. First, they are an aggregation of several subratings and therefore mask variance in the contribution of individual components or interventions. Second, by assessing primarily whether a project has achieved its stated objectives, they may encourage project designers to limit their ambition to what can be easily, and sometimes already has been, achieved.<sup>7</sup> Third, they can reorient time horizons to short-term outputs, which are easier to measure within the project life-cycle, over longer-term efforts to resolve core problems (Andrews 2021).

# **3. Methods**

# _3.1. Research Questions_

We seek to advance the existing literature by considering four questions. First, can any general statements be made about the impact of development aid on sector outcomes? Second, does consideration of aggregate project outcome ratings within each sector mediate the terms of that relationship? In other words, do outcome ratings provide information on the relationship between aid and outcomes? Third, can the application of novel econometric and machine learning techniques better detect associations between project characteristics and development outcomes? And finally, what can such methods illuminate about the characteristics of development projects associated with positive sector outcomes?

# _3.2. Data_

Data on sector outcomes and country characteristics are obtained from the World Development Indicators (WDI, World Bank 2021), covering 1990 _−_ 2015. For official development assistance (ODA) flows, we used the AidData Core

> 7This may explain the inability of previous researchers to explain more than 30 percent of the variance in outcome ratings. If every project defined objectives to achieve the highest rating, the correlation between ratings and project/country characteristics would be zero. IEG does assess the ‘Bank’s contribution’, defined as ‘the extent to which the services provided by the World Bank ensured quality at entry of the project and supported effective implementation through appropriate supervision.’ Its evaluation of project efficacy, or the extent to which outcomes were achieved, is also required to examine ‘whether the achieved outcomes can plausibly be attributed to the government program or project.’ But the first focuses largely on compliance with fiduciary and reporting requirements; while the second does not evaluate whether the objectives would have been achieved in the absence of the Bank’s involvement.

5
Research Release (Tierney et al. 2011).<sup>8</sup> For project outcome ratings, we used the Project Performance Dataset (PPD), a consistent six-point project outcome score based on donor-reported outcome data (Honig, Lall, and Parks 2022). We aggregated from AidData to produce measures of total ODA per country-year, and used the PPD in combination with AidData to construct simple and sizeweighted averages of ratings for projects completed in a given year.

For World Bank projects we use a scraper to download the three main documents of every project - the project information document (PID), the project appraisal document (PAD), and the implementation completion report (ICR). Respectively, they contain the information available at the beginning and end of project preparation, and at project closure. The documents had already been subject to plain text extraction by the World Bank, and we performed a minimal amount of post-processing to clean the files.<sup>9</sup> For project characteristics, we used the data compiled for Ashton et al. (2023). This includes much that is publicly available via the World Bank’s project portal, as well as some internal data on team and management characteristics, and project preparation and supervision steps.

We concentrate on projects in five sectors: health, education, water and sanitation (WASH), energy and fiscal management, due to their combination of prior literature, data availability and size. Together, they account for 35% of aid flows, and one third of PPD projects. When combining sector aid flows and projects with outcomes and country characteristics from the WDI we conducted standard smoothing for noise and minimal interpolation for variables with high missingness.

# _3.3. Linear Methods: Replication, Sectoral Extension and Ratings_

We first replicate the specifications in health, education, and WASH from Mishra and Newhouse (2009) for health, Birchler and Michaelowa (2016) for education, and Ndikumana and Pickbourn (2017) for WASH. Using identical model forms, we are able to reproduce the results in each paper closely (see Appendix A). Since our primary interest is in replicating these models as a baseline, we do not extend them through instrumentation or other techniques, nor do we make more than associative claims based on them.

We then extend this prior analysis by varying controls to check for robustness and adding two sectors: energy, and fiscal management. For energy, we use as a baseline the same controls as in WASH. For fiscal management, we use a limited set of controls, for income level and institutional quality. The general form of the regression equations was as follows:

> 8The AidData is based on the Organisation for Economic Co-operation and Development (OECD) Creditor Reporting System (CRS) donor-reported data, with added granularity on purpose and activity coding.

> 9The scrapped dataset of public documents related to World Bank development projects is openly available at HuggingFace (https://huggingface.co/datasets/lukesjordan/worldbankproject-documentS).

6
(1) Where _Ycts_ is the relevant sector _s_ outcome in country _c_ at time _t_ , _Xc,t−L,s_ is the relevant aid variables with a lag of _L_ (e.g., volume of aid to sector _s_ ), _Wcts_ is the set of controls taken for each sector from the cited literature, and _fcs_ and _fts_ are sector-specific sets of country and period fixed effects. The controls include macro-economic (e.g., GDP per capita), demographic (e.g., youth share of population), and institutional (e.g., Freedom House ratings) measures. The regression tables in Appendix A provide the exact outcome and controls used for each sector.

We conduct our primary extension for each model by adding the project outcome ratings as exogenous variables (in _Xc,t−L,s_ ), varying the specification for robustness. First, we construct a weighted average each year with the weights provided by the relative size of the rated projects in aid flows. We then take the mean and max of those ratings over rolling five year periods. As well as the rating itself (6-point scale) we use a binary variable according to whether the average rating was “moderately satisfactory” and higher, or “moderately unsatisfactory” and below. We also restricted the volume of aid to only that from donors in the PPD, or only World Bank Group projects.

# _3.4. Machine Learning Methods: Residual Outcomes and Text Embeddings_

We then focus our analysis on projects undertaken by the World Bank, for which more granular information and extensive text documentation are available. At the project level, we applied debiased machine learning (Chernozhukov et al. 2018) to estimate treatment parameters for project effects by utilizing a linear model to partial out fixed effects and controls, then utilizing linear and non-linear models to estimate the residual using only project-level characteristics.

We denote by _Y_<sup> ̄</sup> _cts_ the predicted value in country _c_ at time _t_ for the relevant outcome in sector _s_ , estimated using only the controls and fixed effects in Equation 1. In other words, _Y_<sup> ̄</sup> _cts_ = _γ_ 0 _,s_ + _γW Wcts_ + _fcs_ + _fts_ . We then removed this prediction to generate, in each sector, in a residual term _Y_<sup> ̃</sup> _cts ≡ Ycts − Y_<sup> ̄</sup> _cts_ . Together: The coefficients in equation 2 are estimated independently for each sector, and the resulting _Y_<sup> ̃</sup> _sct_ are each residual terms (and hence normalized scalar values). These residual outcomes are then the targets for project-level prediction.

We then extend our analysis to encompass numerical representations of text related to the project that, we argue, capture the degree to which project content is tailored to country and sector context (see the Appendix B for details of their construction). These representations are known as “text embeddings”. In theory, such embeddings can be very simple: for example, a vector representing counts of key words in a document is an embedding. The embeddings 7
<!-- Start of picture text --> parent_sector_name e<br>. .<br>. Se<br>ng reprer aa “vats és Y ee * Se<br>@ Public Admin ct o* we 3b + Amon .<br>: ~amlieay om - é % s “ee , see oo<br>: on .. °°¢ ”€ ne ee . y oat ee .<br>be eaeiare,, ‘; »<br>@ = FinancialIndustry Sector %. ae fullest Sasa eect ae5 =<br>: ei Nee . bead ° oad pei ENS eth . ° ke<br>H)Industry sate’. aesasassartes: eagtates’s ¢<br>& Trade/Ser ar a o3 8 elt Oxtoohe Raseanns! eeend es °S rs Sie<br>© (H)Environment og ee ° des 3S. ses pe ent ngs a .<br>|<br>© (HOWE Gas ot” tise sudan ote a¢ wtpetted .<br>oo) eplet) . %, ste & } aan . a fofee! sete x e ~ e %<br>© (H)Priv Sector Dev BP oat aes ® ee sets Sit oe 3 .<br>@ (H)Urban Development RS oe SAS ete: Pie we aoa<br>7h are, TS ed FSee<br>ee ra a<br>e” - “eernae o Mei _-<br>ef fey, “syee. °<br>><br><!-- End of picture text -->
Table 1: Health <u>projects</u> embeddings evolution over time

|Period|N|Mean X|Mean Y|Mean Distance|
|---|---|---|---|---|
|1990_−_99|195|7.84|9.64|5.01|
|2000_−_09|275|7.75|7.65|5.55|
|2010_−_19|308|7.65|9.89|2.79|
|2020_−_21|203|9.44|11.76|1.19|

_Note:_ Embeddings are projections into abstract high-dimensional vector space expressing inter-relationships, so lack physical units. This table reports the reduction of health projects embedding in two dimensions (X and Y), centered on the 1990 _−_ 1999 mean.

sector and decade _Psd_ and _|Psd|_ = _Nsd_ , we calculate:

and similarly construct _Ecd_<sup>_∗_foreachcountry</sup><sup>_c_.Wethendefinethe</sup><sup>_sector_</sup> _distance_ , denoted _Tp_<sup>_s_for a given project</sup><sup>_p_, as</sup><sup>_∥Ep−E_</sup> _sd_<sup>_∗∥_.Similarly, the</sup><sup>_country_</sup> _distance_ , denoted _Tp_<sup>_c_,isdefinedas</sup><sup>_∥Ep−E_</sup> _cd_<sup>_∗∥_.</sup> The sector distance can be interpreted as the degree to which the project document is, in the deep texture of its language, adjusting sectoral knowledge to local country realities, and the country distance a similar measure of how sectoral peculiarities are being brought to bear on local problems.

Further work is needed to determine what design aspects, as reflected in the text of the project documents, are most clearly associated with development outcomes. In this paper, we restrict ourselves to embeddings generated from the project development objectives (PDO), results framework indicators and implementation completion report (ICR). But even these appear to be capturing the degree of specificity to country circumstances. To illustrate, here is the PDO of a health project with very low (sector distance more than 1 standard deviation smaller) contextualization: The revised PDOs are to: (i) improve coverage, utilization and quality of health care services in the territory of the Recipient, and (ii) strengthen the Government’s stewardship functions in the health sector.

By contrast, more contextualized PDOs are more specific with respect to population groups, periods and outcomes. Here are another two examples with high (more than one standard deviation above average distance to sector mean) contextualization: The Project’s development objective is to ensure access to improved and sustained water and sanitation services in rural communities in [redacted country name]. This would be accomplished through the implementation of the new Rural Water Supply and Sanitation 9
(RWSS) sector policy and the preparation of a National RWSS program. To this end, the Project would support a decentralized and demand responsive delivery mechanism and help build the institutional foundation for implementing the National RWSS Program both at the central and local governments levels.

# And:

The specific objective of this project is to support programs designed to halt transmission of HIV/AIDS among vulnerable populations (PLWHA, IDUs, CSWs, and their clients and sexual partners) and between these vulnerable populations and the general population. Key outcome indicators include: Percent of vulnerable groups in participating provinces reporting safer injection practices (from an estimated 20% at baseline to 70% at the project end ); Percent of vulnerable groups in participating provinces reporting condom use in sexual intercourse (from an estimated 40% at baseline to 80% at project end).

Finally, we seek to identify relationships between the contextualization variable and project preparation and supervision characteristics. These characteristics comprise project region, recipient income levels, fragility, and institutional strength; the time and cost of project preparation; the location (headquarters or country office) and experience of management; whether or not analytical work in the sector and country was conducted in the years prior to the project; and various characteristics of the project manager, including education level (PhD or not), age, experience, and prior work in the sector, analytical or lending. Since we have embeddings for all historical World Bank projects, we are able to probe for these relationships across a somewhat larger dataset ( _N_ = 4 _,_ 260).

# _3.5. Non-Linear Models_

Having constructed the residual outcomes and text embeddings, we set two prediction tasks:

1. Binary classification. For each project, the target prediction is whether the residual outcome in the project sector is positive five years after project completion.

2. Regression. For each project, the target prediction is the precise residual outcome in the project sector five years after project completion.

The lag-time for prediction follows the replicated studies and is used because projects may not target the specific sector outcome during the project period itself, or may take place in only a few districts at a time. However, as discussed in section 2.2, this is justified in that almost all development projects aim for systemic effects, whether via demonstration, capacity building or other channels.

As inputs to the non-linear models used for each prediction we construct vectors for each project _p_ consisting of: 10
1. Basic quantitative data on the project, such as the size of the loan (in 2010-adjusted US dollars), its duration (in months), and the percentage of its budget allocated to its primary sector. We extend several of these features, for example calculating the Herfindahl-Hirschman Index (HHI) for budget allocation across sectors (see Table D.11).

2. Categorical features, such as the funding source (IBRD or IDA<sup>10</sup> ) and the financing instrument (the loan or grant type). The features are one-hot encoded and described in Table D.12.

3. Text embeddings for the project title and project development objective (PDO), as well as for the implementation completion report (ICR) and the results framework indicators, where they exist.

4. Project contextualization features, i.e., sector-distance and country distance for each of the embeddings.<sup>11</sup> We concatenate the numeric and categorical features, the text embeddings, and the distance measures, to generate the combined project feature vector _Xp_ . Combining across health, education, WASH and energy, this results in _n_ = 1 _,_ 457 projects as inputs, with the lagged residual outcomes (as described above) as targets for predictions. Following standard practice we construct a test set of _Ntest_ = 146 projects and train on _Ntrain_ = 1 _,_ 311. We then utilize standard techniques to search among model architectures and among hyper-parameters for the models. In each case we trivially use classification and regression variants of the model architectures. We use a linear model as a baseline, in both modern variants (Lasso and Ridge). We also consider support vector machines, decision tree ensembles (random forest), gradient boosted trees (XGBoost), and fully-connected neural networks (small in size, given the limited data). The full list of architectures and hyper-parameters is provided in Table D.13.

To measure predictive performance, we use the receiver-operator area under curve (ROC AUC) metric for the binary classification, measured on the test set. ROC AUC can be interpreted as the probability that the model will rank more highly a random project associated with a positive residual than one associated with a negative lagged residual of being positive. A ROC AUC of 0 _._ 5 means the model is only as good as random choice in distinguishing between positive and negative projects, and a ROC AUC of 1 means it distinguishes such projects perfectly. We also report the _r_<sup>2</sup> of the corresponding regression models on the training set, in order to compare results to more traditional regression techniques in the development literature, which do not use train-test splits.<sup>12</sup>

> 10International Bank for Reconstruction and Development or International Development Association.

> 11We include both these distance features and each project’s raw embedding, since the embedding on its own can (and by our empirical results does) contain information about a given project’s relationship to others not captured in the sector- and country-distances alone.

> 12We do not use results on the training set to select models or make claims for them, following standard practice.

11
# _3.6. Robustness and Interpretation_

We perform multiple checks for robustness of both the linear and non-linear models. For the linear models, we search over multiple possible linear specifications by adding and removing controls and adjusting lags and observe the effects on significance measures and coefficients for exogenous variables.<sup>13</sup> . We also test for orthogonality between _X_ and _W_ in equation 1, to test for the possibility that _W_ is a function in part of past _X_ and hence that the residual in equation 2 is prematurely purged of the influences of _X_ , weakening the association unintentionally. In other words, we test that associations between past aid and present controls are not muddying the results. The non-linear models are all tested using the standard practice of K-fold cross-validation.<sup>14</sup> We interpret the models in part using standard methods specific to model types, such as coefficient magnitudes and significance for linear models and impurity-based feature importance in decision-tree ensembles, supplemented by Shapley Additive Explanations (SHAP values, see Lundberg and Lee 2017). SHAP values use a game theoretic approach to explain the output of a model by attributing contributions to the final prediction to model features, analogously to attributing the contribution of individual players within a team to the final result of a game.

# **4. Results**

- _4.1. Sectoral Aid Effects: Can any general statements be made about the impact of development aid on sector outcomes?_

The linear model regression results are reported in Appendix A. In each case, the volume of aid per capita had a statistically significant effect on sector outcomes, appropriately lagged and smoothed. The coefficients were, though, modest in each sector:

- Doubling per capita education aid is associated with an 8 percentage point increase in net primary school enrolment.

- Doubling per capita health aid is associated with a 2 percentage point reduction in the infant mortality rate.

- A 1 percentage point increase in WASH aid as percentage of GDP is associated with between 1-5 percentage point increase in rural access to water and sanitation.

> 13In other words, we introduce causal perturbation, following the practice in the DoWhy library (Sharma and Kiciman 2020).

> 14The training set is itself divided five times into a “hold-out” (or validation) set and a training set proper, with a candidate model trained on the training set proper and scored on the validation set. After the five runs both the scores and the models themselves are averaged, and excessive variance between each “fold” is examined.

12
- Doubling per capita energy aid is associated with a 2 percentage points increase in access to energy.

- Doubling per capita aid to fiscal management is associated with a 4 percentage points increase in the tax (net of social contributions) to GDP ratio.

These results were largely robust to causal perturbation. Coefficients remained significant with only minor changes in magnitude when controls were added or removed, with the partial exception of education, where the addition of a lagged prior enrollment figure resulted in the aid coefficient becoming insignificant. The consistency of the results gives us confidence in saying that aid is associated with improved sector outcomes, but the effect is generally modest, and dwarfed by other variables.

As a robustness check, we also find that the aid variable has low-to-trivial correlation with the controls in almost all cases, and the cases of moderate correlation argue more for _W_ causing _X_ in equation 1 than vice-versa. For example, HIV prevalence (Pearson coefficient of 0 _._ 39 with period-average mean per-capita health commitments) and fertility (0.26 on the same measure) are the only health controls with more than a 0 _._ 1 correlation with per capita aid, and the coefficient is positive, i.e, greater levels of HIV and fertility result in more aid.

These results do not change when longer lags are introduced to _X_ . For example, the correlation of last-five-years’ aid and pupil-teacher ratios is _−_ 0 _._ 16 and that between 5-to-10-years’ aid and the same ratio is _−_ 0 _._ 17. More aid is then extremely weakly correlated to lower pupil-teacher ratios, but that, and Freedom House ratings ( _−_ 0 _._ 3), are the strongest correlation between the controls and treatments and those correlations do not strengthen (even trivially) when lagging _X_ . This strongly suggests that _X_ and _W_ are largely orthogonal, and that aid’s effects are not stronger through some lagged or cumulative effect on the controls.<sup>15</sup> On the other hand, the coefficients on aid are probably an underestimate of its true effect. First, we are comparing “flows” of aid to an effective “stock” of sector outcome performance. Second, some proportion of the aid flows will be unrelated to the specific outcome used as the dependent variable (e.g. to higher rather than primary education, or to learning outcomes as opposed to enrollment). But the degree of underestimation is likely limited by the intentional alignment between official aid flows and the outcome-level indicators to

> 15Correlations remain small when lagging total aid as far back as a decade, and, while aid within the sectors summed country-wise over the period are moderately to strongly correlated, such aid is not so correlated when disaggregated over time. In other words, there is little reason to believe that the effect of sectoral aid is being weakened by a relationship between overall aid and growth followed by growth and sectoral outcomes. Further avenues to try to increase the size of the effect of aid’s effectiveness are beyond this paper, whose principal purpose is not to investigate this relationship in itself.

13
measure progress towards the Millennium Development Goals (MDGs) that we use as our dependent variables.

- _4.2. Project Rating Significance: Do project outcome ratings provide information on the relationship between aid flows and sector outcomes?_

The results for project ratings, reported in detail in Appendix A and summarized in Table 2, are also clear. Only for fiscal management outcomes do the weighted average ratings convey information about outcomes. In the other sectors, the ratings are not significant: the coefficients are near zero and their inclusion makes no difference to the coefficients on aid volume. These results are robust to using the alternate measures of ratings and to the restriction of aid flows to particular donors or the World Bank Group alone. The one exception is in a specification for sanitation, but with a small sample size, a small coefficient and a negative sign.

The more sensitive “debiased/double machine learning” techniques confirm the absence of effects seen in the traditional regressions. Table 2 shows, for each sector, the _r_<sup>2</sup> of the partialing out step, the _r_<sup>2</sup> of the treatment test, the coefficient on the treatment (weighted average rating) in the treatment test and the _p_ -value of the treatment test.

It might be argued that ratings’ absence of information in four out of five sectors is a result of projects focusing on other outcomes than those we are testing against. But this cannot explain why aid volume enters significantly against the sector outcomes, and, when examined in detail, requires implausible assumptions to account for the results. Assume that some proportion _X_ of the aid in a sector targeted the MDG sector outcome, and the rest targeted entirely unrelated outcomes. Then the “true” coefficient on aid volume would be 1 _/X_ times the coefficient detected in our regression. If overall project ratings did convey information, then they should convey information on the _X_ proportion targeting the outcome, and hence should modify the coefficient on volume, even if attenuated. The ratings on the 1 _− X_ share of aid explicitly targeted to non-MDG outcomes would diminish the rating effect. But the coefficient could only be reduced to insignificance if the ratings on the 1 _− X_ proportion were negatively correlated to the _X_ share and neutralized them precisely.

We also note that, during this period, projects were predominantly MDG related and that dividing the period into two, one peak MDG period and one after, does not alter the estimate. Further, we do detect a relationship for fiscal management, even though not all fiscal projects explicitly aim to increase the tax share of GDP. Finally, as noted in section 3.5, the use of lags makes it even more reasonable to expect effects on the dominant MDG outcome from projects in the sector. In all, it seems far more plausible that ratings are not providing information than that ratings on a small share of non-primary-outcome projects precisely cancel out information in the primary-outcome projects, even though the results are unchanged in periods where that share was trivial and even accounting for lags in within-sector spillovers, and not least because this neutering effect would have to mysteriously vanish in one out of five sectors.

14
|Table 2: Signif|cance a|nd Mag|nitude o|f Coefcient on Ratings|
|---|---|---|---|---|
|Sector|N|_rR_<sup>2</sup><br>_c_|_r_<sup>2</sup><br>_t_|Rating coefcient|
|Education|731|0.48|0.00|-0.01|
|Health|250|0.80|0.00|-0.01|
|WASH|406|0.84|0.01|-0.04**|
|Energy|317|0.87|0.01|-0.02|
|Fiscal|539|0.62|0.80|0.07***|

_Notes:_ Sector outcome ratings were incorporated as the dollar-weighted average rating in the prior period. _rc_<sup>2denotestheadjusted</sup><sup>_r_2ontheinitial</sup> regression of the sector outcomes against the controls, and _rt_<sup>2denotesthe</sup> adjusted _r_<sup>2</sup> for the regression of the residual outcomes. Significance of rating coefficients indicated as ***p _<_ 0.01, **p _<_ 0.05, *p _<_ 0.10.

_4.3. Contextualization and Sector Outcomes: Can we detect associations between project characteristics and development outcomes?_ We used the strongest performing non-linear model to test for feature importance. The results are shown in Figure 2. The embedding features are dominant, followed by the measures of sector concentration, and only afterwards country institutional quality and other project characteristics more commonly used in the literature.

The most important embeddings are those associated with the PDO, especially the distances of that embedding to sector and country mean, followed by results framework indicators and the ICR. The importance of the contextualization features indicate that the model is learning to detect from the embeddings the degree to which a project has been contextualized to its country, via its distance to its sector mean, and to which country considerations have been adjusted in light of the sector’s characteristics, via the distance to the country mean.<sup>16</sup> We consider feature importance when adding in during-project and at-review features. We find that the embeddings of the ICR report itself then join the PDO embedding and contextualization measures as one of the most important features, and on some specifications becomes **the** most important feature. The actual project length (as opposed to its proposed length) is similarly important. However, neither fully displaces those from approval, and the PDO embeddings and contextualization and concentration measures retain high importance. In keeping with our other results, project ratings are unimportant.

Roughly half of the projects in the dataset had a positive residual outcome, which we would generally expect given their construction. Ensemble based

> 16It is important to remember here that these are measures of relative contribution to the performance of a non-linear model on the entire dataset and cannot be used in the trivial manner of a coefficient in a simple linear regression, to read off that, for example, a proportional increase in contextualization leads immediately, _ceteris paribus_ , to a certain increase in predicted performance. More simply, nothing in these results should be taken to imply that writing a longer PDO with the names of some local programs will lead to improved sector outcomes (or even that simply an increase in intellectual effort across the PAD will do so).

15
<!-- Start of picture text --> Feature Importances: Feature Importances<br>per sector Ln (Original commitment amount):<br>co oz abe a6 oe aio oz ole ovo az oe a6 a8 oo<br><!-- End of picture text -->
# **5. Conclusion**

We present two main findings. The first is that in four of the five sectors for which we have data, donor agencies’ project outcome ratings provide no information as to the long-run impact of their projects. It follows that they are a poor measure of aid effectiveness, though they may still be useful for monitoring other aspects of project performance. Our second finding is that the single most important correlate of impact is what appears to be a proxy for the degree of contextualization of project design to country circumstances, far ahead of country institutional quality, project size or other commonly identified factors. Our methods enable us to explain around 70% of the residual variance in development outcomes, a much higher proportion than previous analyses of the determinants of project outcome ratings.

These findings have significant implications for how we think about project preparation. Further work is needed to establish what the embeddings mean. But our results at least suggest that greater attention be paid to country contextualization. This does not appear to correlate with the standard determinants of project quality identified in the literature, such as project manager’s age, education or prior experience. Nor does it correlate with whether the staff is based in headquarters or in the field - consistent with IEG’s own assessment that corporate field staffing targets have failed to ensure that decentralization is tailored to country and program needs or applied to areas where it can bring the most benefits (Independent Evaluation Group 2016). While we find some evidence that the length of project preparation and existence of prior analytical work are positively associated with impact, we are unable to say to what extent they matter.

Our analysis also has implications for how we think about project evaluation. On the one hand, it is encouraging that ICRs provide sufficient information to accurately predict the likely contribution of projects to long-run outcomes. On the other hand, the incorporation of this information into summary ratings appears to have resulted in them becoming disassociated from development impact. This may warrant a more careful reading of the qualitative evidence in ICRs, as well as more attention to how project outcome targets are calibrated, perhaps by requiring teams to specify an ex ante counterfactual absent the World Bank’s involvement.

17