characteristic that has some possible influence with project quality is the TTL’s World-Bank-specific project experience.

# **_Project design in the qualitative data and the link between World Bank analytic work and project quality_**

While there is some indication that the quality of project design plays a role in ultimate project success – specifically the ‘value-added’ of the preparation TTL covaries with the IEG rating, at least in IBRD settings, and the previous design experience of World Bank projects helps predict the TTL ‘value added’ – the evidence for the importance of design is relatively scant in relation to the evidence of the importance of supervisory quality. For this reason, we turn next to the qualitative data set and explore measures of project design not routinely captured in administrative data.

Table 8 presents some simple bivariate associations (estimated with OLS) in the qualitative data between the IEG rating and various generalized design features. These features include design complexity measures such as the number of project components and sub-components, the number of primary development objectives, and the number of intermediate indicators as described in core project documents. Additional measures include expert panel ratings (based on a 4-point Likert scale) of the quality of the stated project objectives and the project results framework, as well as whether and to what extent project components were modified as part of a restructuring (while project restructuring may occur for various reasons, changes to project components suggest possible design challenges). Many of these indicators are significantly associated with the project’s success rating. Design complexity measures such as the number of project components or the number of intermediate indicators are significantly and negatively associated with subsequent IEG rating. Project designs that fail to define the basic aspects of a results framework (as evidenced by a low score from an expert review panel) also receive lower outcome ratings. The most common results framework shortcomings identified in our sample of 120 operations were inadequate baseline information or indicator targets and an absence of indicators relating to the project’s expected outcome.

Further evidence on the stickiness of design flaws comes from our data on project restructuring. Almost two-thirds of the 120 projects in our sample data set were restructured, some more than once. There is no penalty attached to restructuring per se. Indeed, failure to restructure promptly or even cancel is often mentioned as a contributor to poor performance in ICRRs. But restructuring to compensate for poor design, as measured by changes to project components, is also associated with lower outcome ratings.

To contextualize the qualitative data set findings, we construct an index of design features where the weight for each individual element (listed in the top panel of Table 8) is determined through principal components analysis. This index is significantly related to the project rating – a one standard deviation increase in this index is associated with a 0.22 point increase in the rating. By contrast, an index derived from standard project characteristics related to project success, such as the duration of the project development period, is also significantly related to the IEG rating but at only half the magnitude as the design index. Another benchmark is the CPIA at year of project approval. This measure is not significantly related to the project rating in the qualitative data (likely due to the small sample size), although the magnitude of the point estimate, 0.30, is only slightly larger than the coefficient on the design index.

An additional measure linked to project design quality is whether the project has been informed by prior analytic work tailored to the sector and country of operation. The World Bank devotes significant resources on the production of analytic work and therefore we would hope to see the impact of such 16