metadata: authors: 0: author: Isham, Jonathan
1: author: Narayan, Deepa
2: author: Pritchett, Lant
count: World
docna: 0: docna: Does participation improve project
            performance : establishing causality with subjective data
docty: Policy Research Working Paper
subtopic: Rural Settlements,Rural Water Supply and Sanitation,Rural & Small Town Water & Sanitation,Educational Sciences,Water and Human Health,Hydrology,International Trade and Trade Rules,Water Supply and Sanitation Economics,Small Private Water Supply Providers,Agricultural Growth and Rural Development,Rural Water,Town Water Supply and Sanitation
teratopic: Information and Communication Technologies,Governance,Health, Nutrition and Population,Poverty Reduction,Science and Technology Development
abstracts: cdata!: Development practitioners are coming to
            a consensus that participation by the intended beneficiaries
            improves project performance. But is there convincing
            evidence that this is true? Skeptics have three objections:
            1) "Participation is not objective -- project rankings
            are subjective; 2) this subjectivity leads to "halo
            effects"; 3) better project performance may have
            increased beneficiary participation rather than the other
            way around -- a statistical association is not proof of
            cause and effect. The authors show methodologically how to
            answer each of these objections. Subjectivity does not
            preclude reliable cardinal measurement. Halo effects do not
            appear to induce a strong upward bias in estimating the
            effect of participation. Finally, instrumental variables
            estimation can help establish a structural cause and effect
            relationship between participation and project performance
            -- at least in the rural water supply projects they studied.
repnme: repnme: Does participation improve project
            performance : establishing causality with subjective data
display_title: Does participation improve project
            performance : establishing causality with subjective data
listing_relative_url: /research/1994/09/698010/participation-improve-project-performance-establishing-causality-subjective-data
topicv3: Water m1326522 137,Trade m1326924 698,Education m1327048 470,Rural Water Supply m1327209 143,Rural Development m1504505 2667
publishtoextweb_dt: 2015-09-14T00:00:00Z
historic_topic: Information and Communication Technologies,Governance,Health, Nutrition and Population,Poverty Reduction,Science and Technology Development
origu: Development Economics and Chief Economist
no_of_pages: 44
author_keys: 0: author_key: NA
1: author_key: NA
2: author_key: NA
model_extractions: POLICY RESEARCH WORKING PAPER 1357

# Summary findings

Development practitioners are coming to a consensus that participation by the intended beneficiaries improves project performance. But is there convincing evidence that this is true? Skeptics hay.. three objections:

* 'Participation" is not objective: project rankings are subjective.

-* This subjectivity leads to "halo effects."

* Better project performance may have increased beneficiary participation rather than the other way around; a statistical association is not proof of cause and effect.

Isham, Narayan, and Pritchett show methodologically how to answer each of these objections. Subicctivity does not preclude reliable cardinal measurement. Halo effects do not appear to induce a strong upward bias in estimating the effect of participation. Finally, instrumental variables estimation can help establish a structural cause and effect relationship between participation and project performance - at least in the rural water supply projects they studied.

- - This paper a product of the Office of the Vice President, Development Economics is one in a series of background papers prepared for _World Development Report 1994_ on infrastructure. Copies of this paper are available free from the World Bank, 1818 H StreetNW, Washington DC 20433. Please contact Michael Geller, room T7-079, extension 31393 (38 pages)- September 1994.

**7be** **_Potiq Rteseareb 'Workig_** _Paper S_ **_d bi_** y _hemii cy fmdRegs o_ **_ork_** _iss_ **_in pra&tio cncurage the exc6ang of idCas atnte demlopmecnt issues. An_** _obrcaiive of_ **_theseriesis togct he findingsout quidkJy,_** _cucn if_ **_the presentations are kss tha full polWd Tt, prpers arry the nams cof Me authors andlshould be zsed and cited accordingly. Tbc fuiigs. iaterpretafionss ad conclusions are the aubors' oam and should not_** _be_ **_attributed to the World Bank, irs_** _Executive_ **_Board of Directors or axy_** _of_ **_its mEmber countris_**

Produced by the Policy Research Dissemination **Center**
Does Participation Improve Performance?: Empirical Evidence from Project Data Jonathan Isham Deepa Narayan Lant Pritchett
# Does Participation Improve Project Performance?: Empirical Evidence from Project Data

# InUWmuQtifon

Development practitioners are corhing to a consensus that participation by the intended beneficiaries improves project performznce. Championed since the early 1970s by mostly non-economic social scientists and grassroots organizations (e.g. Freiere 1973, Korten 1980), participatory development is increasingly advocated by the largest and most influential aid agencies (UNDP 1993, World Bank 1991). However, the existence of consensus (or advocacy) does not imply the existence of clear and convincing evidence. Participation advocates have most often relied on case studies to document the link between participation and performance (e.g. Briscoe and Ferranti 1988, Korten and Siy 1989). However these case studies are easily dismissed by skeptics as inconclusive, as the small number of cases and informal method do not allow formal testing of the findings. In response, some studies used the systematic case study method to establish statistically the relationship between participation and project performance (Esman and Uphoff 1984, Finsterbusch and Van Wickdin m 1987).

Skeptics have raised three objections to the evidence for a causal impact of pardcipation on project performance from this type of study. First, *participation' is not objective: hence project rankdngs are subjective and not appropriate for statistical analysis. Second, the subjectivity in the ranking of projects will lead to 'halo effects': if investigators believe participation is good, their subjective ranldngs will overstate the level of participation in highly successful projects and the level of success of highly partcipatory projects. Third,
2 better project performance may cause increased beneficiary participation rather than vice versa: a mere statistical association is not evidence of a causal impact of partcipation on performance. After a brief review of the construction of the data on project performance and beneficiary participation from 121 runl water supply projects and a presentation of the basic statistical results we address, and overcome, each of these three objections. The subjective nature of the data does not preclude inter-subjectively valid, cardinal measures of participation appropriate for statistical analysis. There is no necessary connection between objectivity and quantification with cardinal numbers: cardinal measurement using subjective criteria is common. Moreover, the cardinal rankings for each project were created by two different coders and the degree of inter-coder agreement is very high. Thjs indicates that inter-subjective reliability can be achieved even for intrinsically subjective concepts like "participation." Finally, we show that assuming cardinality or imposing linearity is not necessary to establish the basic result.

The "halo effect" in coding performance and participation data from project documents is addressed in two ways. Flrst, we show the results are the same if the first coder's performance indicators are regressed on the second coder's participation scores (and vice versa), which indicates a lack of a subjective halo effect by the coders. However, the primary danger of subjective measurement, and one that we cannot address, is that the same individual who assessed project success also assessed participation. If that person has strong views about the relationship this may induce a bias in the project documents themselves, as the performance of participatory projects (and the participation in successful projects) may be exaggerated. We do however show that the strength of the performance-participation relationship does not depend on the 'objectiveness" of the success indicator.
3 The tiird objection, and the most difficult objection to answer, is that the existence of an association does not imply causation. While causality is nearly impossible to establish, we present three arguments. First, we use instrumental variables estimation which allows the identification of the impact of exogenous changes in participation, eliminating the effect on the participation estimates of reverse causation or simultaneity. Second, data on the timing of participation shows that participation at early stages improves project performance at every stage, from implementation to maintenance. Third, we describe case studies in which exogenous changes in participation in ongoing projects had strong impacts on performance.

Even for this limited set of projects.. this paper is not intended as a comprehensive account. We focus on a narrow set of econometic issues involved in drawing inferences from project data. Narayan (1994), drawing on these data and more, discusses in more detail the relationship between performance and participation, the determinants of project success besides participation, the detenninants of participation, and the mechanisms whereby participation increases overall effectiveness.

# 1) Data on rural water projects and basic results

The systematic case review method (Finsterbusch 1990) is used to transform varied, qualitative evaluations of a set of related phenomena into data suitable for statistical evaluation. To illustrate this approach, consider a hypothetical study of effectiveness in higher education. A researcher wants to use statistical techniques to determine the most important factors that contribute to undergraduate leaming. She does not have the time or resources to design and implement a new survey, but does have access to myriad evaluations by other
4 agencies--in our hypothetical example, evaluations of 100 American universities, drawn from six different education organizations. A set of variables to be tested are then defined--e.g., quality of the faculty, research facilities--and given a simple numerical scale. Two independent coders then read the different evaluations and, based on their subjective analysis of that information, code the level of each variable. The coder generates this numeric score based upon quantitative -nd qualitative analysis in the original report. The final set of data, usually the average of the two coder's evaluations, is then used for statistical analysis.

The statistical data of the 121 rural water supply projects in this study were assembled from project documents using the systematic case review method'. Ex-post project assessments by development agencies typically combine limited quantitative evaluations with subjectve judgements of project performance. Each project document was read by two independent readers. From this information, these two readers coded specific project variables (e.g., oveal success of project) onto a scale from I to 7, creating 144 distinct variables.

Appendix Table 1.1 shows the list of coded variables (along with the basic summary statistcs). These can be divided into four groups:

- project performance indicators (e.g. "overall project effectiveness", "percent of water system in good condition');

- e measures of participation (e.g. 'overall beneficiary participation", "participation in construction");

'Finsterbusch and Van Wicldin m (1987) used the systematic case review approach for their study of USAID projects. While Esman and Uphoff (1979) did not exactly follow this methodology, their basic approach was similar: converting independent project evaluations to a numercal scale.
_5_

- background or project characteristics which determine project performance

- (e.g. "size of project", "availability of spare parts');

- background or project characteristics which determine participation (e.g. "extent to which agency made participation a goal", "consensus among users").

The "participation" variable merits some discussion. The measure of participation was not simply a measure of whether potential beneficiaries were surveyed about their preferences. Participation was scored on a continuum, ranging from information sharing, consultation, shared decision-maldng to control over decision-maldng. Participation of beneficiaries was considered at three different stages of the project cycle: in project design, construction, and operation and maintenance.

# 1) Basic model and results

We begin with the most general indicators-overall pmject effectiveness (OPE) and overall beneficiary participation (OBP)-and specify and estimate a simple linear relationship between performance and participation. The bivariate relationship between OPE and OBP is simply:
6

# _1)_

_OPE= *OBP_ **_+ e_** which is functionally equivalen **t** o the simple correlation of the two variables<sup>2</sup> .

The usefulness of this bivariate relationship is limited since other non-participation determinants of project performance are excluded. In expanding the model, it proves useful to divide non-participation determinants into two groups: those that are fully exogenous and not affected by participation (e.g. 'availability of spare parts") denoted by a matrix Z and those which are potentially affected by participation (e.g. 'responsiveness of managers") denoted by the matrix W. The multivariate equations are then: 2) _OPE.= OBP, +_ 8 _* Z. + 8w * Wj + eC2_ 3) _W_ **_=y_** * **_OBP_** _+_ **_y_** * **_Z,_** _+_ **_y_** * **_X,.+,_** Both Z's and W's are potential determinants of project performance, however Z's represent variables that are not influenced by participation, while W's represent variables which may be deteriined (in part) by participation. As indicated in the second equation, W's may also be

> 2 1n the bivariate linear regression the regression coefficient is - (where a, is the covariance of y and x) where the correlation coefficient p is . So, the correlation coefficient

p =,B *(-A) is a simple rescaing of the bivariate linear regression coefficient. _aJ_
7 determined in part by the Z's and by some other set of variables, X's (the non-participation determinants of the W's). In summary, these multivariate equations state _that_ performance of a water project depends on: beneficiary participation; a set of inputs (Z) not related to participation (e.g., "adequacy of facilities", "availability of spare parts"); a set of performance determinants (W) that may in turn be determined by partcipation (e.g., "responsiveness of managers", appropriateness of technology) as well as by other inputs.

The distinction between the Z's and the W's is important for maintaining the distinction between the partial and the total impact of participation on project performance. In the multivariate regression (equation 2), the beta coefficient gives the direct impact of increasing participation, holding all included variables constant: <!-- Start of picture text --> 4) a0PE1 =z<br>aOBP 4Z%w<br><!-- End of picture text --> But participation may also influence performanace indirectly: the total impact of changing participation is the sum of the direct and indirect impacts: <!-- Start of picture text --> 5) dOPE =8OPE 1 + 8OPE * aw<br>dOBP aOBP aW 8FBP<br><!-- End of picture text --> or in this particular specification, _6) dOPE_ **_+_** _*y dOBP-_
8 Thus, the simple partial coefficient with all controls understates the total impact of participation while the bivariate coefficient (which excludes thie Z's and W's) overstates the impact to the extent these determinants and participation are positively correlated. For each regression we report the linear regression coefficient on OBP in the OPE for regression for each of the three specifications.

Table 1 presents three estimates of the linear association of overall beneficiary participation (OBP) with overall project effectiveness (OPE): bivariate; limited multivariate (with Z's); and full multivariate (with Z's and W's). In all cases the results are strongly statistically significant and empirically quite large. The t-statistics range from 10.6 in the bivariate case to 3.8 with the full multivariate controls.

The estimated impact of participation on project effectiveness ranges from .62 for the bivariate case to .24 in the full multivariate case<sup>3</sup> . How are these coefficients to be interpreted? The expected impact of increasing participation from a low level (OBP=2) to a high level (OBP=6) is to improve project performance from between 1 to _2.5_ points (on a seven point scale). A one standard deviation increase in participation (s.d.=1.7, Appendix table 1.1) is associated with between a .41 (full multivariate) to 1.05 (bivariate) point increase in performance. The interesting-and intuitively appealing-results of the all regressions (limited and full multivariate) are reported in Appendix table 1.2 and are discussed in Narayan (1994).

The multivariate impact is naturally lower than the bivariate effect due to the exclusion of positively correlated non-participation performance determinants, as discussed above. Since the bivariate effect is a biased upward estimate of the total effect and the full multivariate estimate of the partial effect is likely biased downward for the total effect, these create reasonable bounds for the total effect.
9

**Table 1: Basic results for participation (OBP)** with **OPE (Overall Project Effectiveness) as the dependent variable**

||Bivariate|Limited multivariatea (Z<br>variables)|Full multivariate (Z and<br>W variables)|
|---|---|---|---|
|on OBP|0.62|0.28|0.24|
|t-stat.|10.6|5.3|3.8|
|N|121|77|68|
|R<sup>2</sup>|0.49|0.86|_0.89_|

Notes: a) Regression results for other vauiables reported in Appendix **table** 2.1.

We note here that the choosing of the **Z's** and W's in this study was not entirely straightforward nor rules-driven. However, all the results have proved robust to a number of variations of the model, and we feel that the choice of control variables is not of primary interes'. We were generous in our inclusion of potential performance factors, including 18 non-participation variables. The participation variable thus easily passes this "kdtchen sink' torture test of throwing all plausible variables into the regression. The danger of inadequate controls for other determinants of project performance is not nearly as serious a problem as the three we discuss below'.

I We do, after all, face the difficulty of 144 coded variables with only 121 projects, which means that mechanical procedures for selecting variables will lack degrees of freedom and are unlikely to be of much help. Moreover, many of the variables are clearly overlapping and likely to be collinear. After some experimentation, we based variable inclusion on three criterion: decent inter-coder reliability, prior judgements about the best of collinear sets of variables, and impact on the estimate of participation (we never dropped any variable which seriously affected the estimate of the participation coefficient). In none of the experiments were the results on participation substantially different from the full multivariate case reported in table 1.

5 Heteroskedasticity, a typical econometric problem which receives a fair bit of attention (one suspects because it is easy to fix) deserves slight mehtion here. It is not a problem with the present results for two reasons. First, the White heteroskedasticity consistent standard errors are roughly the
10

# II) Subjective cardinal data

The first objection to studies--and results-of this Idnd is that the data generated by the systematic case review method are subjective. According to this skeptical view subjective data is unreliable and/or ordinal and therefore inappropriate for statistical analysis. In this section we show our data are subjective, yet reliable and cardinal. First, we argue that the automatic association of subjective phenomena with ordinal data is incorrect. Second, the degree of inter-coder agreement on the scoring of the major variables reveals that the subjective measurement error, while present, was a r.inor source of variation. Third, using techniques appropriate for ordinal data do not dramatically change the results, and the constraints imposed by linear regression analysis are also not rejected by the data.

# Subjective and objective. ordinal and cardinal

Economic theory presumes that objective phenomena (e.g. numbers of oranges, relatve prices) have a natral cardinal metric (e.g. real numbers) whereas intrinsically subjective phenomena (e.g. consumer utility) allow only ordinal comparisons (e.g. better or worse) especially intersubjectively'. This distinction is critical. While both cardinal and ordinal data same as the OLS: the t-statistics with White are 11.4 vs 10.6 (bivariate) and 5.33 vs. 5.25 (full multivariate). Second, in scoring variables, each co>de **r** ecorded their subjective assessment of the reliability of the score assigned. When these reliability measures were used to weight observations, the results were roughly the same. And when, because of a programming mistake, we weighted the observations by giving more weight to the less reliable observations, the coefficient point estimates were still the roughly the same. This is perversely reassuring: under the assumptions for consistency of OLS, weighted least squares estimation is also consistent kor any set of weights, even the exactly wrong weights.

eOrdinality stems directly from the basic theory of mapping a binary preference relation into a utility index. The restrictions imposed on the preference relation (complete, reflexive, transitive) imply that once a given utility index is derived any monotone transform of that index equally well represents the
11 can rank phenomena, only with cardinal data can numbers be tabulated and values of the phenomena being measured be compared directly. Most common statistical techniques Oike correlations or linear regression) cannot be applied to ordinal data<sup>7</sup> .

However, the data used in this analysis created by the systematic case review method are subjective, yet cardinal. Our data on rural water projects is doubly subjective: the original project evaluator subjectively assessed and described the amount of participation in each project; a coder later read the evaluator's report and subjectively assigned a level of participation to that project. If this process generated ordinal data, empirical analysis would be difficult. But note that in everyday life, we observe many events which generate subjective, cardinal data. Contests are the most obvious example. When hogs, figure skaters, or bodybuilders compete judges assign cardinal scores to subjective criteria: 'quality of coat" for hogs; 'artistic impression" for figure skaters; and 'muscle tone" for body builders. Grades for acadeniic papers are another familiar example: a professor's subjective evaluation of a humanities paper is given a cardinal score. In each case, these subjectively assigned scores are added, averaged, tabulated in ways only appropriate to cardinal data. Of course, the judging and grading criteria same preferences.

7 For instance is x were an ordinal measure of participation then estimating the linear model y=p ***x** could produce different results than estimating _y=P *(x)_ , JO a monotone transform of x, even though x and Ax) would represent exactly the same information Therefore any statistical procedure that relied on summing observations (or any comparison of the magnitude of the distance between observation) would be invalid for ordinal data.
12 are created to achieve inter-subjective consensus'. Thus, the question for this data set on characteristics of water projects is not whether the data is subjective, but whether the subjectively cardinal scores are reliable.

# Inter-subjective agreement

Since project variables were scored from the same documents by two independent coders, the coherence of their separate scores illuminates the overall reliability of the variables. Table 2 presents two measures of the cross coder agreement. The correlations for the two major variables (column 1) are strikingly high: over .99. The average absolute value of the difference in the scores (column 2) is quite small: most scores either agree or differ by just 1 point. For each of the two major variables, the coders disagreed by 2 or more points on only one projecL Table 2: Cross coder reiablity

||Correlation between coder<br>Aand B|Average absolute difference<br>of scores(1to7scale)|
|---|---|---|
|Overall project effectiveness|<br>.95|.36|
|Overall beneficiary<br>participation|92<br>|_.55_<br>I<br>I|

This high degree of inter-subjective consensus has two important implications.

It creates a prima facie case that the characteristics of the project could be reliably gauged from the project documents. The high correlation also places a relatively tight bound on tihe This does mean that judging requires training to achieve this level of inter-subjective agreement. For instance, there are contests for the judges in which those trining to be livestock contest judges are themselves judged on the degree to which their subjective judgements conform to those of established judges (Herren, 1984).

' The reliability was much less for some other variables in the data set
13 magnitude of measurement error: a correlation coefficient of .9 implies that the noise from measurement error is roughly 10 percent of the variance of the observed variable.'<sup>0</sup>

# Testing linearity or cardinality

We examine the implications of cardinality in two ways. We treat the data as if it were ordinal, using dummy variables for each participation category. We also test linearity of the relationship-which imposes even stronger conditions than cardinality. Of course, these techniques do not 'prove' cardinality of the data; but they both show that the basic results on beneficiary participation are not affected either by allowing for the possibility that the data are ordinal or by our functional form assumptions.

A first procedure treats both the performance and the participation data as ordinal. For both prcject performance and participation, a binary variable takes a value of 1 if the score is high (> 3.5) and 0 otierwise. This procedure is valid even if the data were ordinal; binary variables would be unaffected by monotone transforms". The final column of table 3 reports 1D That is if two observationw liffer by only measurement error then the correlation between the two <!-- Start of picture text --> 2<br>obslvatiis p = a= Whe ax. isihevarianceofthe"true"vaiableaod OL<br>Cr|JX - a-,:AX+<br><!-- End of picture text -->

is the measurement error variance for coder A(B). If both coders measurement error is equal

**2 2 2 2** 4 _4A_ = o, = aG a correlation of .9 implies the ratio of measurement error to true variance 2 a=*

is about .1.

**_1'_** Although this procedure is valid if the data are ordinal if the data are in fact cardinal this procedure is very inefficient-as it throws away all of the infbrmation about variation within each of the two performance categories.
14 the results of this linear probability regression'<sup>2</sup> . The performance-parcipation effect remains evident with this transformation of the data.

A second approach argues that if the participation data were in fact ordinal, one would not expect to find a linear relationship between the two variables: the 'true' underlying ordinal relationship variables would not be invariant with respect to arbitrary transformations (e.g., squaring) of the data. Table 3 presents one test of linearity: allowing for a slope shift depending on the value of participation. When participation is low (< =3.5) the slope would be p 1 while when participation is high (>3.5) the slope is p 1 + p2 . Although the estimates do suggest the incremental impact of participation is larger at higher levels (slope of .466 vs .781), this difference is not statistically significant (a low t-statistic and a declining adjusted R-squared). Increases in participation have roughly the same impact along the range from low to high participation.

**12** We could have used an estimator more efficient fbr binary dependent variables (such as logit or probit) but the linear probability model produces a consistent estimate which is sufficient for our purposes.
15 Table 3: Tests of linearity and functional **form**

||Linear|Linear with<br>kdnk|Binary<br>variable for<br>each OBP<br>category|Binary variables<br>(both OPE and<br>OBP)a|
|---|---|---|---|---|
|Participation coeff.<br>(t-statistic)|.623<br>(10.6)|.466<br>(3.007)||.552<br>(7.34)|
|Change in slope<br>(OBP>3.5)||.315<br>(1.33)|||
|OBP<_=1.5_|||_2.55_||
|1.5 < OBP < =2.5|||3.06||
|2.5_<_OBP < =3.5|||3.59||
|3.5_<_OBP < =4.5<br>4.5< OBP< =5.5<br>_5.5<_OBP<br>__ _<sup>_2_</sup>|.481|.480|4.25<br>5.16<br>_5.74_<br>.459|.306b|

Note: a) The binary model is estimated as a linear probability mDdel. b) The R-squared is not comparable between the linear regressions and the binary model.

A third technique treats participation as if it were ordinal while treating performance as cardinal. Each discrete level of the participation variable is entered into the performance equation as a dummy variable: D1=1 if OBP=1, 0 otherwise; D2=1 if OBP=2; etc.'<sup>3</sup> This functional forn imposes no a priori constraints on the effect of the independent variable. The results in table 3 show a strong participation effect-performance increases for each dummy variable-without any strong indications that this statistically unconstrained fit is tremendously superior to the imposition of linearity. The incremental impact from category to D Although since the averages of coders responses were used, the numbers are not always integers ranges for the variables were specified to generate these dummy variables.
16 category (from the differences in the coefficients) ranges from .5 to .75, roughly equivalent to the overall linear slope of .623.

Thus, the subjective nature of the data per se appears to have no impact on the result: high inter-subjective reliability of cardinal was achieved, and the results appear to be broadly consistent with a simple linear model.

# m) Halo effects

A potentially more senous problem than the data's intrinsically subjective nature is that either the initial evaluator of the projects or the coders themselves succumbed to the plausible assumption that all good things go together: the whalo effectu.'<sup>4</sup> The halo effect occurs when the measurement of the variables are affected by the observed state of the other variable. This systematic measurement error will induce an association between two subjectively measured variables even in the absence of any *true- relationship between the underlying variables. In our study, the halo effect may occur at two stages. The evaluators may have falsely attrbuted participation to successful projects (or vice versa) or the coders-reading the project documents searching for evidence of project participation-may have been affected by their simultaneous assessment of project success (despite their efforts to remain objective). The This psychological tendency associate all good Ihings go together has been discussed in a number of fields. In particular, there is a large literature in the human resource management about the halo effect problem in performance assessments in which outstanding performance in one dimension or characteristic (even a potentially irrelevant characteristic such as physical attractiveness) may tend to bias upwards evaluation of other dimensions or characteristics. Hammermesh and Biddle (1993) find that plain people make about 5 percent less and attractive people five percent more than persons of average attractiveness. However, for a recent dissenting view on the importance of halo effects in perfbrmance evaluation, see Murphy, Jako, and Anhalt, 1993.
17 second is particularly dangerous: in this study the two coders knew the purpose of the empirical exercise and may have had some strong prior beliefs as to the expected outcome.

There is nothing we can do about the potential whalo effect" of the original evaluations. We do know that the project reports were regular parts of institutional evaluation cycle and that it is doubtful that the financing agencies had a particular stake in promoting participation. It could also be expected those many different individuals writing the project documents would have had widely different beliefs about the importance of participation so that a uniform bias in the first hand assessments would be unlikely.

As for the potentially serious 'halo effect" in the coding process, we explore three avenues to address the problem. Note that the results in table 2 are based on the average of the two coders assessments. Alternatively, we estimated the same models using only data from coder A and from coder B. Differences in these two assessments may reflect differences in the "halo effect" between the coders. Second, in the same models, we used coder A's assessment of the explanatory variables (mcluding OBP) with coder B's assessment of the dependent variable (OPE). Since coder A's assessments of participation and the other potential determinants are not affected by coder B's performance assessment, this should reduce the halo effect bias (although the confounding effect of pure measurement error in the coder's assessments will be important). Third, the coders created some project performance indicators that, by their nature, are more objective than others. If they were present halo effects would be more likely for the more subjective indicators.
18

# Results by and across coders

Table 4 shows the regression results with the average values, only **the** values of coder A, only the values of coder B (columns 1, 2, and 4 respectively). The differences in the both bivariate and multivariate result are very small. Table 4 also shows the corresponding results of the cross-coder tests: column 3 show A's outputs on B's inputs; column **_5_** shows B's outputs on A's inputs. Again the results are stubstantially the same. In **the** bivariate and multivariate models the coefficient does not systematically change, whether we use **the** average of the coders' scores, each coder's own scores, or one coder's dependent variable scores on **the** other coder's independent variable scores.

|**Table**4: Basic re<br>re<br>|sults by average<br>gressing overall p<br>on overall benefic|coder value<br>roject e#ec<br>iary partici|, coders A<br>tiveness**(O**<br>pation (OB|and B resp<br>**PE)**<br>P)|ectively;<br>||
|---|---|---|---|---|---|
|OPE on OBP|Averages of<br>A and B|A on A|A on B|BanB|BonA|
|Bivariate|0.62<br>(10.6)|0.60<br>(10. 1)|0.62<br>(10.3)|0.60<br>(9.7)|0.57<br>(9.3)|
|N|121|111|116|111|116|
|R<sup>2</sup>|0.49|0.49|0.48|0.46|0.43|
|Full multivariate|0.24<br>(3.8)|0.23<br>(2.6)|0.26<br>(2.1)|0.21<br>(2.0)|0.25<br>(2.7)|
|N|68|37|46|46|37|
|R<sup>2</sup>|0.89|0.94|0.85|0.89|0.94|
19 How reassuring are these cross-coder results? Ihe following equations are helpful. Let's say that A's observation on project performance is the "truth" plus some random noise 0, plus an upward bias based on A's observation of participation:

-7) _OPE = OPE_ + **_BA*OBPA_** *

Coder A's observation on participation is just the "truth" plus random error: 8) **_ORPA_** - **_OBP*+IA_**

> In this case, the coefficient of regressing performnnce on participation will be biased **j** by the "halo effect". If the true structural relationship were:

9) _OPE = p*OBP_ + E the estimated coefficient would be:

> 10) **B+.**

> so that even if there were no structural relationship between the "true" variables ( **p** = **o** ) the

estimate of the participation effect could be spuriously positive due to halo effects.

Given this background, why does having another coder matter? If the observations-and scoring-of participation were completely objective, using a second coder's data would have no effect A's and B's observations on participation would be identical (OBPA = OBPJ) and if the degree of halo effect was similar (.BA= X , the bias on perfonmance
20 data would have no effect: A's and B's observations on participation would be identical (OBPA = OBP3) and if the degree of halo effect was similar **(8J** - **8** ,the bias on performance would be equivalent. If the observations of participation are subjecdve, then the "halo effect" bias should be less using cross coder data because the pure sbjective component of B's assessment does not affective the bias in the performance measure. However, to the extent the performance is truly subjective this argues against the inter-subjective reliability above and induces downward bias in the estimates due to classical measurement error.

Table _5_ shows Monte Carlo simulations of the combined effects of the "halo effect" bias and of pure measurement error, using different assumptions about the relative strengths of the two effects. Unfortunately, these simulations show that both underesmation and overestimation are possible when coders scores are crossed, depending on the ratio of the two effects. With large measurement error (the final column), crossing the coder rankings should produce lower estimates than using the rankings of a single coder for all posstble strengths of "halo effect' for inputs and outputs with no measurement error (the first columns) and high degrees of the "halo effect-, crossing the coder rankings does not help, it produces the same estimates (with a similar upward bias).

Evidence from the relatively high inter-subjective reliability (as well as the instrumental variable results below) suggests low but non-zero measurement error, ° is from .1 to .25. In this range of measurement error (columns 4 and 5), one would expect a
21 modest but significant change in estimates when crossing coder's OPE and OBP ranings if the "halo effect' was strong. The lack of a consistent downward movement of the estimates (in the multivariate case they actually rise) suggest at least that the 'halo effect" is not dominating the results.

Table **_5:_** Results of Monte Carlo simulation of the combined **effects of measurement error and "halo** effects" **using cross coder** information.

||||Degree o|f measure|ment error|||
|---|---|---|---|---|---|---|---|
|||_0|'|.25||.5||
|Degree<br>of|1|2<br>AonA|3<br>AonB|4<br>AonA|_5_<br>AonB|_6_<br>AonA|7<br>AonB|
|EffectH|0|_.5_<br>(true)|_.5_<br>(true)|.40|.40|.33|.33|
||.25|.75-|_.75_|.65|.60|_.58_|_.50_|
||.5|1|1|.90|.80|.83|.67|
||1|1.5|_1.5_|1.4|1.2|1.3|1.0|

Note: The results are the average estimates from 1,000 replications of 120 observations each of the model: _y=, *x_ + e f **_=.5,_** _x,_ **E** N(0,1) , A(B)'s observations on the x variable are subject to measurement error of the form: **_xA>_** = _x_ + **_k*qA,_** where A(B) indicates that A and B have different random measurement error of proportion k. The observations on the dependent variable y are determined by

**_yAW)_** = **_y_** + 6 **_*xA(m_** so that the measurement error of A or B influences the

measurement of y by common 'halo' factor of a

# Results by performance indicator
2:;!

A second technique to evaluate the halo effect is to examine the impact of participation on project performance indicators that vary in their objectivity. In addition to OPE several more objective indicators of project success were coded, including 'percent of water system in good condition," or "percent of population target reached." To the extent that these more objective phenomen **a** re relatively less susceptible to halo overestimation, the possible halo effect bias should be reduced.

If the true coefficient were equal in the two models (which is not clear-see below), the more objective indicator should be systematically lower than the upwardly-biased subjective indicator. Table 6 presents these results. There is no evidence that the more subjective indicators (such of OPE) have systematically larger estimated impacts.

Table 6: Impact of participation by various indicators of perfonrance

||Overall Project<br>Effectiveness|Objective<br>Value of<br>Benefits|% of Water<br>System in Good<br>Condition|% of Target<br>Populaion<br>Reached|
|---|---|---|---|---|
|Bivariate||X|||
|/3 on OBP|0.62|0.53|0.54|0.29|
|t-statistics|10.6|10.3|6.4|5.3i;|
|N|121|120|98|118.00|
|R<sup>2</sup>|0.49|0.47|0.29|0.19|
|Full multivari|ate||||
|_,B_on OBP|0.24|0.27|0.29|0.25|
|t-statistic|3.8|3.6|2.4|2.50|
|N|68|68|60|68.00|
|R<sup>2</sup>|0.89|0.79|0.77|0.47|
23

# IV) Joint determination and Causality

The prior two sections have answered possible skepticism about the strong statistical association between performance and participation association. Another line of skepticism may accept the statistical association between participation and performance but deny this association reveals cause and effect. In this skeptical view, the data do not show that greater participation causes better project performance, simply that they happen to be related. Indeed, there are at least two good reasons that a performance-participation association may not be causal. First, there could be "reverse causation": projects that are exogenously better might induce greater beneficiary participation. This is sensible especially when performance and participation depend on a sequence of actions. Once it is clear that project is failing, each potential beneficiary may be less likely to participate because she perceives a relatively low benefit from participation which is unlikely to alter the project outcome.

Second, joint determination of project success aM beneficiary participation may be driven by a third local or project attribute. For example, if dynamic leaders induce both project performance and participation, performance and participation data will be strongly associated-even without an independent causal effect of participation on performance. While we have tried to address this concern over spurious correlation with the 'kitchen sink" inclusion of possible performance determinants, it would not take a very clever skeptic to name a large list of excluded variables (and some unobservable even in principle) that could affect both performance and participation. We use three approaches to resolve the problems of reverse causation andjoint association and demonstrate a causal impact: instrumental variables estimation techniques, timing, and case studies.
24

# lnstivmental variables

One econometric solution to the problem of identifying a structural relationship is estimation with instrumental variables. Instrumental variables estimation avoids the problem of the joint determination of the independent and dependent variables by eliminating in the estimation of the coefficients that part of the variation in the independent variable which is due to variation in the dependent variable. The vehicle to eliminate th,at variation is a third variable (the instruments) which affects only the independent variable and not the dependent variable.

In order to do instrumental variables, we need a variable that does affect participation but which does not affect directly, nor is affected by, performance. This variable is used as our instrument to purge the participation variable of any performance-related component. When the participation effect is estimated using only the part of participation variation that is correlated with the variation in the instrument, the resulting estimate is free of reverse causation: since better performance does not affect the instrument, the reverse effect of better perfonnance on participation is eliminated and cannot bias the results.

Expressed in equations, if the model is: 11) _OPE_ = _* OBP + 5Z*Z_ + **_6:W*Wi +86*Vj+_** 12) _._ **_O_** _=_ **cc** **_Zj_** a|.*Vi _m, Vi T+,_
_25_ All V's which are included in the participation equation _(a,, O'_ but excluded from the performance equation,( 8v=0 ) are legitimate instruments. The V's provide a source of variation in participation that is exogenous to performance. On the other hand, neither the Z's nor the W's are valid instruments: they directly affect both participation and performance.

To choose appropriate instruments, we need a positive model for partcipation. Hypotheses based upon the larger study of these water projects (Narayan 1994) (as well as theoretical literature on the determinants of participation) generated a set of equations to estimate the effects of participation (Appendix table 1.3 shows the ful "first stage" regressions). We identify four variables as legitimate instruments: "extent to which participation was a project goal"; "percent of investment costs bome by users"; 'beneficiares overall net benefits of participation"; and "extent to which organization is based on local collectives." We hypothesize each of these phenomena may directly affect participation, but should have no independent, direct effect on project performance after controlling for participation.

In table 7, the OLS and IV results are compared-for both the bivariate and the limited multivariate case'<sup>5</sup> . The estimated impact of participation increases with IV estimates. For instance, when extent to which participation was a project goal" is used as an instrument, the bivariate impact rises from .63 to .70; the multivariate rises from .28 to .34. Similar results-the IV producing a higher and statistically significant estimate-are observed for each of '5 The full multivariate case loses too many degrees of freedom. So while the results are empirically similar they are less precise.
26 the instruments used singly". When all instnrments are used together the impact in the bivariate case rises to .86 and to .37 in the limited multivariate case.

What do these IV estimates tell us? The basic statistical relationship-high correlation between participation and performance-would also occur if better project performance caused greater better participation: as clean water is delivered in the early stages of a project, more potential beneficiaries may want to get involved. But if this were the causal story, the IV technique would cause the estimates to fall by removing this upward simultaneity bias. The rising coefficients reported in Table 7 are consistent with causality running from higher participation to better project performance in the presence of some measurement error. The IV results allow us to compute an independent estimate the magnitude of pure measurement error. Even if the OTLS estimator is inconsistent with measurement error, the IV estimator is consistent and their ratio estimates converges to <!-- Start of picture text --> plim( PtOL)= . With<br>NV (a+ +<br><!-- End of picture text --> the reported estimates, this ratio is between .8 and .9 (e.g. .63/.70 (bivariate) or .281.36). The <!-- Start of picture text --> 2<br><!-- End of picture text --> ratio of measurement error to true variance 2! is between .11 and .25: this is consistent with **2**

*6 Except for the multivariate case when 'organization based on local collectives' is used where the coefficient drops to .15 and statistical insignificance (ikely due to the low power of the instrument).
27 (although somewhat higher than) the estimates of measurement error from cross coder reliability correlations in table 2 above.

In using this technique, one would like to test whether the assumptions made in obtaining the IV results are valid. Note that since one variable may be endogenous-beneficiary participation-at least one instrument must be used to identify the model<sup>17</sup> . But if one instrument can be unambiguously accepted-that without argument it directly affects only the independent variable-then the validity of any other instruments can be tested. Indeed, the entire set of instruments can be tested.

17 Heuristically the problem is that we need to test the exclusion of the instrument from the performance equation. However, one cannot test the exclusion directly (say by a t-test of the inclusion of the instrument) because in the presence of endogeneity the coefficient on the potentially endogenous variable is inconsistent when not instrumented and hence the t-test on the instrument would be biased. As a simple example say the model is _y_ = P *x + 8 *z + e and x is endogenous. Say there is a single potential instrument z, say x = ,r*z +xj . But z is a valid instrument only if 8 = 0 However, this hypothesis can onlytbe tested if there is a consistent estimate for P . But if z is used as an instrument for x, then the 'instrumented" x is perfectly collinear with z (since the instrumented x is just x projected onto z). But since the 'instrumented" x is collinear it is obvious one cannot use z to both recover a consistent estimate of P and to estimate 8 to test the exclusion restriction because using only z both cannot be identified separately. Therefore sufficient "exclusion restrictions"(such as 8 = 0 ) must be imposed a priori and the 'just identifying" assumptions cannot be tested.
28 Table 7: Instrumental variables eslimates of the participation-performance relationship, using various instrument sets

|Estimation<br>technique/<br>Instrumentset|OLS|'Extent Part. a<br>goalJ|*% of Invest-<br>ment costs by<br>users'|'Net benefits<br>of<br>participation'|'Organization based on<br>local collectives'|Pnor<br>commitment of<br>clients'|All<br>[<br>_|
|---|---|---|---|---|---|---|---|
|Bi|variate||||||<br>|
|coeff<br>(t stat)|.63<br>(10.6)|.70<br>(10.2)|_.59_<br>(7.3)|.77<br>(10.6)|.74<br>(6.3)|.97<br>(7.54)|.86<br>(10.4)|
|N|120|120|113|120|98b|105|90|
|R-squared|.488|.482|.476|_.453_|_*5 0 7h _|.378|.521|
|First stage R-<br>|-|.763|.573|.701|.364|.326|.816|
|squareda<br>Li<br>|mited Multivari<br>|ato_<br>|||_|.||
|Coeff<br>(t stat)|.28<br>(5.25)|.34<br>(5.2)|.32<br>(3.6)|_.36_<br>(5.4)|_.15_<br>(1.28)|.39<br>(3.00)|.37<br>(3.57)|
|N|77|77|75|77|66|72|63|
|R-squared|.862|.860|.861|.858|_.855_|.863|.865|
|First stage R-<br>**squared'**|[.401'1|.826|.643|.803|.719|_.559_|.857|

Notes: a) Unadjusted R-squared of the 'first stage' regression of participation on the instruments (which in the multivariate case includes all variables in the performance equation). b) Since the sample sizes are not the same the results are not strictly comparable in all columns. In particular the IV R-squared are less than OLS R- squared when _run for_ tbe same sample.

c) This is tho R-squared of participation regressed on all the Z variables which are included in the perforrnanc **e** quation. The incremen **t** o the R- squared for each instrument can be calculated as the difference with this column.
29 We believe that "extent to which participation was a goal' is the most plausibly exogenous variable among individual instruments as there is no reason to believe that participation as a goal should by itself lead to better performance-ecept insofar as it actually raised participation. When each of the other instruments is tested, conditional on the validity of this variable (using a Hausman-Taylor test), we fail to reject the exogeneity of the other instruments in every case. When the entire set of instruments is tested, we do not reject the validity of the instrument set in either the bivariate or multivariate case". Our set of instruments do stand up to the available tests for instrument validity".

Timing Evidence on causality also can be observed from the timing of the project cycle. If the association between participation and project performance were not causal, we would see no association between events that occur before -project completion-proximate detenninants of

**'** The value of the Sargan test with the full set of instruments is 7.03 (significance level .133) in the bivariate and 5.24 (significance level .263) in the multivariate estimates.

**19** Of course, the major objection to these tests is that they tend to be of very low statistical power (that is, these tests will often fail to reject a hypothesis that is false). Therefore a "failure to reject" the instruments cannot be taken as compelling evidence to accept the instruments.
30

# project performance--and beneficiary participation. Table 8 reports the impact of participation

Table 8: **Impact of beneficiary participation on the proximate determinants of project performance**

||Bivariate|Limited<br>multivariate|Full<br> <br>multivariate|
|---|---|---|---|
|Quality of implementation|<br>0.53<br>(9.3)|0.17<br>(2.7)|0.21<br>(2.7)|
|Effectiveness of O&M|0.49|0.14|0.11|
||(7.4)|(2.0)|(1.1)|
|Maintenance after 1 year|0.43<br>(6.6)|0.16<br>(2.0)|0.18<br>(1.8)|

on quality of implementation, effectiveness of operations and maintenance (O & M), and maintenance after 1 year. We find that in all but one (multivariate) case, beneficiary is a statistically significant input to these proximate determinants. If project effectiveness were causing participation rather than vice versa, we would not expect to see this result.

# Case Studies

Studies of individual cases help to further resolve questions of causality, particularly when exogenous shifts in participation change project outcomes. Narayan (1994) documents two such case studies. Phase I of the Aguthi Rural Water Supply Project in Kenya was implemented without community participation. The project was so plagued with problemsconstruction delays, cost over-runs and disagreements over consumer payment methods-that it camne to a standstill. At this point, the project was redesigned. The Aguthi Water Committee, working with local leaders and project staff, mobilized the community: after public stakeholder
31 conferences, community members organized and began contributing to the project. Phase II of the project was completed on schedule and within budget; the communities continued to pay monthly tariffs for the new water service, and operations and maintenance of the system was handled successfully, in cooperation with the government parastatal.

The WAS (Waniata, Air dan Sanitasi) program in Indonesia assisted community groups to launch and manage their own water system. A water group in the village of Silla was formed in 1986 as WAS began. Initially, they relied heavily on the arrival of a government team to dig a bore hole, but none came. When they realized that they could not rely on immediate government assistance, the members increased their participation. They negotiated water rights with a neighboring group, collected building material, and built firee water tankswith only a small amount of outside assistance. By 1988 a new well was under construction, financed by their own contributions. Eggplant and chilies-with water from the new tanks-were flourishing in peoples' yards.

# Conclusion

We began by showing the existence in project level data of a strong association between project performance and beneficiary participation. We then addressed and answered the three econometric objections to these results. The subjectivity of the data is not an overwhelming problem. The "halo effect" does not appear to induce a strong upward bias. Most importantly, there are strong arguments that the participation and project performance relationship is cause and effect. This paper, especially together with the more comprehensive work of Narayan, 1994, does provide development practitioners-including early and recent
32 converts to the participatory approach-with strong statistical findings that increasing participation directly causes better project performance.

Three questions which are important for practice and policy are not explored here. First, does participation directly cause better project performance across all sectors? One cannot blindly extrapolate the results in this study across all sectors, since this data is only from rural water supply projects. The economic characteristics of rural water as a good would seem to promote the importance of direct beneficiary participation; these economic characteristics vary across goods provided by projects in other sectors. Second, what policy instruments help to achieve more effective participation? The behavior of project beneficiaries, staff in project agencies, and other suppliers responds to incentives, but there is little documented experience on creating incentives in public sector agencies for promoting and incorporating participation. Finally, can experiences with participation help to clarify the analysis of the deficiencies inherent in either a purely individualistic 'market" or a purely statist 'government' approach to development? An analytic approach that incorporates participation might examine the various mechanisms whereby cooperative action by groups can overcome the inefficiency of individualistic solutions--e.g, from 'free riding' or strategic (mis)revelation of private information--while avoiding the limitations of centralized government. These "informal' methods of cooperation have been explored by a number of authors (Ostrom, Schroeder and Wynne 1993, DeSoto 1989, Wade 1988) but much remains to be leamed.