approaches can nonetheless contribute to these debates in substantively important ways. The chapter proceeds as follows. Section II explores five key issues at the center of debates pertaining to project evaluation in developing countries<sup>5</sup> . Section III moves from ‘first principles’ to pragmatism, stressing how evaluators can use mixed methods in the less-than-ideal circumstances they are likely to encounter in the field, and especially in developing countries. Section IV focuses on the importance of understanding project processes and contexts, and building effective monitoring systems. Section V considers some specific way in which mixed methods can improve qualitative evaluations. Specific case studies of mixed methods evaluation in international development programs are provided in Section VI. Section VII concludes by considering the ongoing challenges and opportunities for using mixed methods in project evaluation in developing countries.

# **II. Using Mixed Methods in Project Evaluation and Monitoring: Five Key Issues**

Among the many issues that confront program evaluators, at least five speak to the importance of using mixed methods. The first is the principle that the questions being posed should determine the research methods being used, not the other way around. As noted above, while most economists argue that randomized control trials (with strong quasi-experimental designs as a second best)<sup>6</sup> should be considered the “gold standard” for impact evaluation (see next point), two counter-arguments can be considered. One is that there are many different kinds of development assistance ranging in complexity and scope from the kinds of clearly defined projects for which conventional impact evaluation models were designed, to complex, multicomponent national level programs that involve many donors, numerous national agencies and line ministries, and which often have no clear definition of their scope or intended outcomes and impacts<sup>7</sup> . We argue that the great diversity of projects requires the use of a range of correspondingly different evaluation methodologies, and specifically that strong statistical project level designs are often inappropriate for the more complex, multi-component programs. We recognize, however, that certain _aspects_ of these complex programs can usefully be subjected to randomized designs (see, for example, Olken 2007). The second counter-argument is that evaluations are conducted to address a wide range of operational and policy questions – i.e., not just average treatment effects – and that, as such, different methodologies are required depending on the specific information needs of particular clients<sup>8</sup> .

> 5 All of these issues are also relevant to the application of mixed methods evaluations in industrialized countries. 6 We will refer to RCT and strong quasi-experimental designs as _strong statistical designs_ .

> 7 Some of the dimensions that can be used for classifying evaluation scenarios include: (1) Evaluation purpose, (2) Level at which the evaluation is conducted, (3) Program size or scale, (4) Complexity of the evaluand, (5) size of the evaluation budget, (6) Stage of the project at which the evaluation is commissioned, (7) Duration of the evaluation, (8) Who is the client, (9) Who will conduct the evaluation, (10) level of required statistical rigor, (11) Location of the evaluation design on the QUANT/QUAL continuum, and (12) Source of data (primary, secondary, both).

> 8 Examples of evaluation purpose include: (1) Generalizability (assessing the feasibility of replicating or scaling up an intervention, (2) Developmental (tracking emergent, complex interventions), (3) Accountability (Ensuring accountability for results), (4) Contribution/substitution (assessing the contribution of different donors to comprehensive, collaborative interventions, or assessing with donor funding produces a net increase in funding for a program or whether this replaces previously allocated government funding). Source: Adapted from NONIE 2008.

4