As Rodrik (2008) and Ravallion (2008) have argued, there is a trade-off in policy evaluation between external and internal validity. As traditional identification of causal effects through instrumental-variable strategies never completely eliminates confounding influences, these strategies always suffer from an internal-validity problem. However, when based on crosscountry evidence, they pick up average effects that can be relatively stable — provided they are consistent with some sort of theory — because induction, even on cross-country samples, may fail to produce generalized results.

By contrast, IE purges out confounding influences, but generates results that are empirical and case-dependent. Such results may fail to carry over to different settings. Limited external validity of any study would not be a problem if we could replicate it easily. With enough replications, the sheer mass of evidence would provide the desired generality (although the method would still be inductive and would thus suffer from the general critique of inductive methods in science). But some kinds of IE can be costly. For instance, the World Bank reckons that household surveys cost on average $300 per household. At that rate, a baseline and final survey of 500 households would cost $300,000. This is a lot for studies with only internal validity.<sup>40</sup> However, costs can often be contained by working with local institutions, which has the added advantage of building capacity in a key area.

Some trade-related programs target limited numbers of firms, so their evaluation is less costly than that of poverty-reduction programs. For instance, in a middle-income country, the cost of surveying 500 firms can be substantially lower than $100,000. Moreover, the data may exist prior to and independently of the IE in the form of census or industrial surveys and customs records. In that case, the cost of the IE goes down dramatically. The problem then is no longer one of cost but more of securing buy-in from the agencies possessing the data so that they share it. However, it should also be kept in mind that tests of the effect of interventions based on 500 firms are likely to have low power (see McKenzie 2011a and 2011b for a discussion), and thus to generate type-II errors (failing to reject the null hypothesis of no treatment effect when, in fact, the effect is present). If cost-cutting leads, through low-power experiments, to unjustified pessimism on the effect of interventions, IE may lose a lot of its power to guide policy choices.

# **6.2 Spillovers and general equilibrium effects**

> 40 In their discussion of quasi-experimental versus experimental methods, Duflo et al. (2008) make a noteworthy point about the commitment value of costly experimental design. It has often been argued, with some statistical support (see, e.g., Ashenfelter, Harmon, and Ooster-beek 1999), that statistically significant results (positive impact in our setting) are more likely to get published, a so-called ―publication bias.‖ As experimental methods are costly and usually planned with donors, self-censure in the face of insignificant results is less likely to be feasible than when relatively low-cost quasi-experimental methods are used with publicly available data. In that sense, IEs may be less affected by publication bias.

31