First, the costs of rigorous impact evaluation would have been similar to the costs of the evaluation the project is already doing. The changes needed to make the evaluation rigorous— per treated village—would have been mostly free. The project is already carrying out baseline surveys and multiple follow-up surveys in the treated villages, and a number of surveys (though no baseline survey) in the nonrandom comparison villages. The only additional costs involved in making the impact evaluation design rigorous would be 1) the very low cost of identifying the matched pairs within which to randomize treatment, using rainfall and census data, and 2) the additional cost of a baseline survey in the randomized comparison villages, which would only add one round of surveys, in half the villages, to the multiple rounds of surveys the project is already planning. These additional costs would only represent a small percentage increase in the project’s existing evaluation budget per treated village.

Second, the overall cost of any impact evaluation, even a rigorous one, would represent only a small percentage of the project’s costs. This is important because it is not too late to rigorously evaluate the effects of future expansion of the Millennium Villages. For example, it would only take about 20 matched pairs of village clusters, or fewer, to allow high confidence of detecting a treatment effect of about ten percentage points in some indicator of proportion.<sup>28</sup> That is, if the intervention had about a ten percentage-point effect on the proportion of children in each household attending school or receiving vaccinations, this could be reliably detected with only 20 matched pairs of village clusters or fewer. Conservatively overestimating that it would cost US$100 per household per survey round to survey 300 households per cluster, four rounds of surveys in one pair of village clusters (0, 5, 10, and 15 years from start) would cost US$240,000. This is the cost per treated village. Even this conservatively high estimate is only 16% of the $1.5 million total intervention cost for each treated village (MVP 2010e). This direct cost is small compared to the indirect cost of the large waste that would occur if the project were to be ineffective but were massively expanded without discovering the ineffectiveness early.

> 28 Standard power calculations imply that it would require 23 pairs of village clusters to detect a change in the proportion of people in each household meeting some criterion (such as measles vaccination or school enrollment) from 0.5 to 0.6, with 80% probability, at the 5% level of statistical significance. This figure assumes 300 households sampled per cluster (MVP 2009: 21), _without_ baseline data, and assumes a plausible value for the coefficient of variation of the proportion within matched cluster pairs of 0.3 (Hayes and Bennett 1999). This could be diminished with better matching, thus lowering the required number of clusters (Bruhn and McKenzie 2009). Even this is a conservative upper bound on the number of pairs required, because the collection of baseline data would allow it to be substantially smaller; exactly how much smaller would depend on assumptions about the correlation between baseline and follow-up measurements.

27