behind the choice of the exact sample are not explicitly discussed and/or results are presented in a way that makes it hard to compare them with those of other studies. Often, very simple and straightforward fixes would make it much easier to place a specific result in the wider literature. For example, it would help enormously if exact sample sizes and baseline summary statistics were always reported (if average class size is not known and an intervention’s effect on enrollment is only indicated by the additional number of students induced to enroll in treatment classes, it is impossible to make a meaningful comparison between the intervention’s impact to that of other policies). Going one step further, if the reporting standards set forth in the CONSORT Statement – an evidence-based set of recommendations for reporting the approach, implementation and results of RCTs, cf. Moher et al. (2010) – were more widely adopted, this would further facilitate the critical appraisal and interpretation of the findings from impact evaluations.

Besides, a large proportion of impact evaluations are very much focused on establishing whether there is a causal relationship between a certain intervention and one or more education-related outcome variables. But while the attribution of a causal impact to a specific intervention is the defining feature of any impact evaluation, much more could be learned if more attention was devoted to the mechanisms that purportedly relate an intervention and the outcomes measure(s). At present, a clear results chain connecting inputs, activities, outputs, outcomes and impacts is only rarely explicitly mapped. Statistical tests of a result chain’s individual links are even scarcer. Sometimes it is uncertain whether an intervention was even properly implemented. Especially in those cases were an impact evaluation fails to find any statistically significant impacts of an intervention, a further emphasis on implementation, outputs and intermediate outcomes could shed light on the reasons behind an intervention’s apparent failure and could offer guidance for how to design more promising interventions in the future.<sup>12</sup> Finally, most impact evaluations focus on relatively short time horizons. The median duration between an intervention and the measurement of it its impact in the impact evaluations underlying this paper’s systematic review and meta-analysis is twelve months. The reasons behind the usually very short time horizons are clear: Every further round of follow-up surveys and measurements is very costly, political relevance often demands the timely reporting of impacts and some impact evaluation designs – e.g. an RCT of an intervention that is implemented in stages over a whole country or region – even preclude the measurement of long-term impacts. Still, many promising interventions in the education sector might only become effective over the longer term. Others, in contrast, might only be shortterm fixes that quickly lose their impact. In a third group of interventions, important differences between seemingly equally effective treatment arms might only become apparent over the longer term – as was the case with the impacts on learning of group- and individual-based bonus payments to teachers in Andhra Pradesh, India (Muralidharan and Sundararaman, 2011, and Muralidharan 2012). All this means that the designers of impact evaluations should make every effort to capture interventions’ effects over a period spanning at least two to four years.

> 12 See IEG (2012) for a how-to guide for designing a results framework.

- 41 -