spending on subsequent operation quality. In Table 9 we find that prior analytical work is indeed linked to improved project outcomes, especially in lower-income IDA settings and possibly even fragile and conflictaffected settings (FCVs). Analytical work completed up to three years before project approval is associated with an increase of 0.12 in IEG outcome ratings (0.15 points for IDA countries); half the impact associated with a one-point improvement in the CPIA. This suggests an important role for World Bank initiated analytic work in country environments that may be relatively “knowledge scarce”, where the information generated by these analytic efforts informs subsequent project design.

# **5. Summary and discussion of unsettled questions**

This paper reviews the literature on the correlates of World Bank project outcome ratings and then extends previous analytic work by identifying further staffing inputs into the production process, linking previous sector analytic work to individual projects, and extracting a richer set of design quality explanators from a subset of projects. Results from these analytic extensions all point towards the importance of project design in ultimate project success, especially in the more sectoral complex settings of middle- and upper-middleincome countries (IBRD). Previous quantitative work has found fairly weak evidence for the importance of project design; however, it appears that this is in part due to the paucity of relevant indicators captured in routine World Bank administrative data. It would be an error to conclude from these earlier studies that project design is not an especially influential determinant of project success.

Like much of the project effectiveness literature, the analysis discussed here is descriptive in nature. Hence, any observed association between project inputs and outcomes may either represent an underlying causal relationship or may be confounded by further unobserved factors. While our understanding of the project cycle leads us to consider these correlations as plausibly capturing the role certain inputs play in project success, there are alternative explanations that render the correlations spurious. For example, project staff are not randomly assigned to develop or supervise projects but are chosen by management. Other work suggests negative assortative matching between high performing TTLs and low performing environments (Limodio, 2021). If indeed management places some of the most effective staff in the most challenging environments, this would serve to downward bias the estimated relation between TTL quality and project success. While we still largely find a positive relation between TTL ‘value added’ and project ratings, the true relation may actually be of greater magnitude.

Short of instituting experimental staff assignments, there are few analytic possibilities that will cleanly identify the causal relationship between inputs and project success. One direction of inquiry involves the mandatory staff rotation rules where, after a fixed period (of either four or five years in the recent past) staff are forced to rotate to a different global region. Arguably, staff turnover due to rotation rules are exogenous to other project characteristics. In this example, information on staff tenure in a region can be then leveraged to more carefully estimate the relation between staff turnover and project success.

Regarding the overall variation in the IEG rating, we explain slightly more than a quarter of observed variance in project outcomes, most of it accounted for by staffing and project characteristics, with approximately five percent of total variance attributable to country-level factors. Some of the most significant contributions to project success appear to be made by the `value-added’ of supervision TTLs, and the most important observed determinant of their contribution is previous World Bank project experience, not their age, education, tenure, or country location. This argues for deploying scarce expertise 17