terms of the additional years of student participation bought with $100 (in 2011 USD). They discuss the limitations of using simple point estimates of impact to conduct costeffectiveness analysis, and carry out a sensitivity analysis which shows confidence intervals around cost-effectiveness estimates, using the standard error of impact estimates. Kremer, Brannen, and Glennerster (2013) likewise show cost-effectiveness across a range of learning interventions, as does McEwan (2014).<sup>2</sup> This paper discusses important challenges to consider in the application of cost-effectiveness analysis. This is most closely related to the discussion of sensitivity of cost-effectiveness estimates to discount rates, exchange rates, and other parameters in Dhaliwal et al. (2013).<sup>3</sup> First, we demonstrate a method for characterizing the sensitivity of relative costeffectiveness estimates to errors in impact estimates, using repeated simulations drawn from the distribution of the impact estimates (i.e., a Monte Carlo simulation). We apply this method using J-PAL (2014a) data which compare the cost-effectiveness of 27 education programs in achieving student learning gains across Africa and Asia (14 of which have statistically significant impacts), and of 16 programs achieving attendance gains (11 of which have statistically significant impacts). Through this analysis we show that a wide range of cost-effectiveness rankings across programs is possible using 90% confidence intervals of impact estimates, both for learning outcomes and for school participation outcomes. We find that taking into account the variance around point estimates, we cannot rule out most rankings of the cost-effectiveness of the 14 programs with significant student learning impacts; but the simulation provides clarity on which programs are most likely to be costeffective.

Second, we examine questions of context-specificity (i.e., external validity) in cost measurement – not explored in the development literature to date – which further complicate a simplistic approach to cost-effectiveness analysis.<sup>4</sup> We look at how costs vary across contexts and with program complexity, and how this complicates the extrapolation of cost-effectiveness results. We find that, using data on community teacher salaries in a number of countries, cost-effectiveness estimates vary by as much as 88 percent with the change of just one cost ingredient. We show how the extrapolation of cost-effectiveness is

> 2 While cost-effectiveness analysis compares the impacts and costs of programs which have a common outcome, cost-benefit analysis takes comparability across evaluations further by converting all costs and benefits into present-discounted monetary values and presenting the ratio of combined benefits to costs, which allows the comparison of programs with different outcomes, to deem which investment would yield the greater social return. There are advantages and disadvantages to each method, as discussed in Dhaliwal et al. (2013). Regardless, all of the limitations of cost-effectiveness analysis discussed in this paper also apply to cost-benefit analysis.

> 3 In the context of health interventions in high-income contexts, considerable work has been done to measure sensitivity in cost-effectiveness estimates. See, for example, Jain, Grabner, and Onukwugha (2011).

> 4 There is a health literature exploring the estimation of confidence intervals for costs (see, for example, Briggs, O’Brien, and Blackhouse 2002); the present paper focuses on key conceptual issues with transferring estimates from one context to another rather than with estimation per se.  Levin et al. (2012) demonstrate the context specificity of cost estimates in a U.S. context.

3