implement “rigorous” impact evaluations. First, many evaluations are conducted on a tight budget which limits the ability to carefully develop and test data collection instruments, and which often makes it impossible to use the sample sizes that would be required to detect statistically significant impacts, particularly for the many situations in which even well-designed projects can only be expected to produce a small change.

Second, many evaluations are conducted with a very tight deadline which limits the time available for data collection and often the amount of time that consultants (many of whom are expensive foreigners) can spend in the field. Another dimension of the time constraint is that evaluators are often asked to assess outcomes or impacts when it is too early in the project cycle to be able to obtain such estimates. A less discussed but equally salient constraint is the limited time that clients and other stakeholders are able, or willing, to discuss the evaluation design or the preliminary findings. Consequently, if program theory models are developed there will often be very little input from clients, thereby defeating the intention of ensuring that stakeholders are fully engaged in the definition of the program model that is being tested and the output and impact indicators that are being measured Third, many evaluations have very limited access to the kinds of data required for constructing baselines or comparison groups. This is particularly problematic in those cases (probably the majority) where the evaluation is not commissioned until late in the project cycle. Frequently no baseline data has been collected on the project group and no attempt was made to identify a plausible comparison group. Sometimes project administrative records or surveys designed for other purposes may provide some data but often it does not cover the right population, contain the right questions, collect information from the right people (only the “household head” but not the spouse or other important household members), or refer to the right time period. In other cases the information is incomplete or of a questionable quality. Another dimension of the data constraint is when the data collection instruments are not well suited for obtaining information on sensitive topics (sexual practices, illegal/illicit sources of income, corruption or domestic violence), or for identifying and interviewing difficult-to-reach groups (e.g., those who are HIV positive, drug users, illegal immigrants).

The fourth, and final, set of constraints relate to political and organizational pressures that affect how the evaluation is formulated, designed, implemented, analyzed and disseminated. These are all influenced, for example, by what issues are studied (and not studied), which groups are interviewed (and not interviewed), what kinds of information are made available (and withheld), who sees the draft report and is asked to comment (and who does not), and how the findings are disseminated. The pressures can range from a subtle hint that “this is not the time to rock the boat”, to a categorical instruction from public authorities that it will not be possible to interview families or communities that have not benefited from the project.

The combined influence of these factors can have a major influence on the type of design that can be used, the sample sizes, the use of statistical significance tests and the levels of methodological rigor (validity) that can be achieved. A frequent problem is that the tight budget and the short timeframe that consultants can spend in the field is often considered as a justification for only visiting project sites and only interviewing beneficiaries and stakeholders directly involved in (and usually benefiting from ) the project. As a result, many evaluations 9