# **4** Data Analysis

## **4.1 Diagnostic Tests**

Prior beliefs regarding some of the tested covariates were summarized from the relevant literature as the first step of the analysis. This provided a theoretical point of reference from which to evaluate the posterior coefficients.<sup>14</sup> Models were run with delivery challenges measured at the cluster, category, and subcategory levels, systematically increasing the granularity of analysis and allowing for the disaggregation of substantive effects for each delivery challenge. Over 50 model averaging runs were conducted, testing the three levels of delivery challenges on project outcomes, performance, and supervision, including basic covariates and fixed effects, and testing a variety of alternate model priors.

In addition to basic controls such as project cost, duration, the presence of various specialists, setting, and Task Team Leader (TTL) quality, several indicators of aid effectiveness from Girod and Tobin (2016) were included to ensure that all relevant context- and project-based variables were accounted for. Before conducting the model averaging runs, data were tested for collinearity and missingness issues in order to ensure that variable selection was not unduly affected by structural faults inherent in the samples tested. Inclusion of collinear indicators can introduce an unintended stochastic element that may even lead to one of the collinear variables being chosen at random. Similarly, listwise deletion of observations can severely impact predictive power. Such tests were conducted on diagnostic subsets of control variables used as well as all subsets used in model averaging runs. Furthermore, models were run using both original and imputed data to guard against potential idiosyncrasies introduced by systematic missingness.<sup>15</sup> Model averaging runs were conducted on the delivery challenges as well as five additional subsets: the first two tested delivery challenge clusters alongside the control variables and fixed effects, respectively. Subsets three and four tested the delivery challenges at the category level against the same controls and fixed effects. Finally, the last specification tested subcategories of delivery challenges alongside the main control variables (excluding the Girod-Tobin covariates). As noted in Section 3.2, the specifications were tested on project outcomes, performance, and supervision ratings to test the impact of delivery challenges on the achievement of project objectives as well as the Bank’s overall performance. Results from the model averaging runs are presented below according to each outcome indicator.

and presentation of results from alternate prior specifications can be found in the supplemental appendix to this paper. Ultimately, it should be noted that alternate prior specifications did little to impact substantive coefficients, even though PIPs fluctuated a bit. Minor fluctuations were mostly seen in models using the Random-Theta Model Prior with a UIP on Zellner’s g Hyperparameter.

> 14To the extent possible, the concepts were chosen to align with the categories used in the delivery challenge taxonomy, though we did not possess strong priors for all of the categories under consideration. A summary of priors can be found in the supplemental appendix.

> 15In the interests of space, most of the diagnostic tests are relegated to the supplemental appendix. Preliminary tests run with the original (listwise-deleted) data yielded roughly the same substantive impacts relative to the delivery challenges. Although imputation increased posterior inclusion probabilities for some delivery challenge covariates, the posterior means and directions of effects were comparable across original and imputed data runs. All imputation was conducted using the Amelia software in R. See Honaker, King, and Blackwell (2011).

13