have a systematic positive bias because they only obtain information from beneficiaries. As these rapid (but flawed) evaluation approaches show funding and implementing agencies in a good light, there has been much less concern about the widespread use of these fundamentally flawed methodologies than might have been expected.

Other common problems include: the sample sizes being too small to be able to detect impacts even when they do exist; techniques such as focus groups are often seen as a “quick and dirty” way to capture community or group opinions when there is neither time nor money to conduct sample surveys and accepted practices for selection of subjects or avoiding bias in data collection and analysis are ignored; attribution is based on comparisons with poorly selected and not very representative control groups; data is only collected from one household member (often whoever is available) even when the study requires that information and opinions are collected from several household members.

Many quantitative evaluation designs are commissioned late in the project cycle and rely on secondary data from previous surveys for the baseline comparison. However, as the secondary surveys were normally commissioned for a different purpose than the project evaluation (national income and expenditure panel survey, national health or nutrition survey) it will often be found that the surveys do not contain all of the required information (particularly concerning specific information on access to the project), do not collect information from the requisite people, were not conducted at the right time or do not completely cover the target population. While statistical techniques such as propensity score matching and instrumental variables can help strengthen the match of the project and comparison groups and eliminate some kinds of selection bias, there are a number of challenges that normally cannot be resolved. In addition to the above problems of sample coverage, who was interviewed etc, a major challenge frequently concerns missing information on differences between the project and comparison populations that might explain some of the post-project differences (e.g., in income, school test performance, infant mortality) that are assumed to have been produced (at least in part) by the project. For example, the women who apply for small business loans may mainly come from the small group of women who have the self-confidence and experience to start a business, or who have an unusually high degree of control over household decision-making or who have husbands supportive of their economic independence. Most applied econometric analysis tries to downplay the importance of these “unobservables” by assuming they are “time invariant” and consequently can be differenced out by using panel data. However, this assumption is often highly questionable, particularly when the researcher may have no idea what these factors might be.

A final challenge, one that will be discussed in more detail in the following section, concerns the lack of information on what happened during the project implementation process. Projects are rarely implemented according to plan, and often access to services and the quality of the services will be significantly less than planned. Consequently, in explaining why intended outcomes and impacts were not achieved, it is essential to know whether lack of results was due to design failure (the basic design did not work in this context) or whether problems were due to implementation failure. In real-world evaluation contexts the information is often not available to make this judgment.

10