The strong claims by quantitative development researchers to have the best (the “gold standard”), or even the only acceptable way to evaluate the effectiveness of development assistance has created a strong reaction from other sectors of the development community.<sup>3</sup> While some argue that randomization is rarely possible in developing countries, that the rigorous designs introduce an inflexibility that makes it difficult to understand the complex environment in which development projects operate, or that there are fundamental methodological issues with the approaches, many take the view that randomized and strong statistical designs are only one of a number of approaches to development evaluation. For example, NONIE (the Network of Networks on Impact Evaluation<sup>4</sup> ) recently published “NONIE Guidance on Impact Evaluation” (April 2009) which recognized the importance of rigorous quantitative methods for addressing issues of causal attribution but recommended that evaluators should use a mixed method approach that combines the strengths of a range of quantitative and qualitative methods. In the latest edition of his _Utilization Focused Evaluation_ , Patton (2008) lays out an extensive menu of different evaluation designs that can be used to address different kinds of evaluation questions.

None of the quantitatively oriented development participants in this debate, however, question the singularity of the econometric analysis of survey-based data as the core method of relevance for impact evaluations. Outside the context of impact evaluations, most work by economists on development questions also remains entirely econometric, though there is an increasing (and welcome) trend towards direct engagement in fieldwork and having survey data be analyzed by the person(s) who collected it, which in turn has led to a deeper and richer understanding of development problems. However, given that a central challenge in international development is that the decision makers (development economists included) are in the business of studying people separated from themselves by vast distances – social, economic, political and geographic – there is a strong case for using mixed methods to both help close this distance and to more accurately discern _how_ outcomes (positive, negative, or indifferent) are obtained, and how any such outcomes vary over time and space (context).

By restricting themselves to the econometric analysis of survey data, development economists are boxed into a Cartesian trap: the questions they ask are constrained by the limitations inherent in the process by which quantitative data from closed-ended questions in surveys are collected (Rao and Woolcock 2003). As such, they are limited in their ability to ask important questions about the social, cultural and political context within which development problems are embedded. They even miss important aspects of some critical economic issues such as, for instance, the heterogeneity that underlies what are known as “informal” economies (i.e., labor markets that function outside formal salary and wage structures) and tend to overlook marginal markets that are centrally important for policy – such as the market for drugs, political favors, and sex – all of which require a strong degree of rapport with respondents that a short visit to field a questionnaire will not provide. A related criticism (to which we return later) is that many kinds of econometric analysis fail to examine what actually happens during the process of project implementation (the “black box”) and consequently are unable to determine the extent to which failure to achieve intended impacts is due to “design failure” or to “implementation failure”. In

> 3 See also the critique of philosophers, such as Cartwright (2007).

> 4 NONIE is supported by and brings together the DAC Evaluation Network, the Evaluation Cooperation Group of the multilateral finance institutions, the International Organization for Cooperation in Evaluation, and the UN Evaluation Group

2