visited to try to reconstruct information on factors such as prior business experience, attitude of the husband to his wife’s business initiatives, and self-confidence. If a rapid sample survey is not possible, a second (if less satisfactory) option might be key informant interviews or focus groups providing information on project and non-project women. As always, the mixed methods approach will include quantitative methods to assess and control for selection bias. Even in strong statistical designs, good qualitative research can also help to ‘observe’ variables that might otherwise be a source of omitted and/or unobserved variable bias, and potentially suggest other instrumental variables that could be incorporated into regression analysis.

<u>Identifying a range of impact evaluation design options. When real-world constraints make it</u> impossible to use the strongest statistical designs, the evaluator should identify the range of possible design options (going from the most to the least statistically rigorous) and identify which design options are feasible within the prevailing constraints and can achieve an acceptable minimum level of methodological rigor (see below). The possible designs should be assessed in terms of their adequacy for the purposes of the present evaluation, and ways to strengthen validity through the combination of different methods should be considered.

<u>Rapid and economical data collection methods.</u> There are a number of techniques that can be used to reduce the costs and/or time required for data collection (Bamberger, Rugh and Mabry 2006). These include: shortening the data collection instrument by eliminating non-essential information; using more economical data collectors (school teachers or nurses instead of professional enumerators); collecting information from groups rather than individuals (focus groups etc); direct observation rather than interviews (for example to estimate community travel patterns); using secondary sources; and building required data into project monitoring and similar administrative records. All of these techniques have potential issues of quality and representativity, and as such it is important to assess these trade-offs between cost/time and validity when defining the data collection strategy.

<u>Threat to validity.</u> While validity should be assessed for all evaluations, the above approaches for reducing costs and time and working with less robust evaluation designs significantly increases the range and potential severity of threats to validity. The use of mixed methods also introduces additional validity issues. The use of a threats-to-validity checklist is recommended to systematically assess the potential threats, their severity and possible measures that can be taken to address them<sup>11</sup> . A key point of using mixed methods, however, is to triangulate data sources so as to check the validity of one instrument against another. Again, even in evaluations deploying strong statistical designs using large household surveys, qualitative methods such as anchoring vignettes (King et al 2004) can be used to ensure that survey respondents in different contexts are in fact interpreting questions in the same way.

# **IV. Strengthening Monitoring Systems: Understanding Project Processes and Contexts**

Projects are almost never implemented exactly as planned and there are often significant variations in implementation in different project locations. Consequently, if the analysis finds no

> 11 A pilot version of a threats to validity checklist developed by Bamberger and Rugh (2008) is available at http://www.realworldevaluation.org/RealWorld_Evaluation_resour.html

12