Given the massive array of evidence, and the apparent divergence in conclusions from the reviews of the evidence, how is one to understand what actually works best to improve learning in developing countries? In this paper, we critically examine these recent reviews to understand the underlying reasons for the observed divergence in conclusions. We also characterize the heterogeneity of effectiveness within categories of interventions. Finally, we highlight the common themes across the reviews – sometimes obscured by differences in categorization – in terms of what kinds of interventions are more and less effective.

We find that much of the divergence in conclusions is driven by strikingly different compositions of studies across the reviews: Of the 227 studies that look at learning outcomes, only three are included in all six systematic reviews, whereas almost three-quarters (159) are included in only one of the reviews. While some of these compositional differences are driven by explicit exclusion rules (e.g., some reviews include only randomized trials and one focuses only on evidence from Sub-Saharan Africa), many are not. This divergence does not mean that reviews are incorrect in characterizing what works well: The main conclusions of each review are supported by evidence from papers that attempt to explicitly establish a counterfactual. Indeed, the strongest positive results in each review are driven by randomized controlled trials. However, each review incorporates different evidence, leading to different ultimate conclusions.

We also observe that much of the variation in outcomes across educational interventions is captured within categories of interventions rather than across them. Highlighting the average effectiveness of a given category of intervention may be less useful than characterizing the narrower types of interventions within that category that drive high returns: For example, saying that computer interventions are most effective may be less useful and less accurate than saying that computer-assisted learning programs which are tailored to each student’s level of knowledge, tied to the curriculum, and that provide teachers with training on how to integrate the technology into their instruction are most effective.

Finally, we find that there is indeed some intersection in recommendations across the reviews, although that intersection is masked with different labels. Even given the small degree of overlap in the composition of review samples, we find broad support across the reviews for (i) pedagogical interventions that match teaching to students’ learning, including through the use of computers or technology; (ii) individualized, long-term teacher training; and (iii) accountability-boosting interventions, such as teacher performance incentives and contract teachers.

# **2. Methods**

This paper takes as its population the set of reviews of impact evaluation evidence on improving student learning in developing countries identified in 2013 and 2014. We restrict this analysis to reviews of evidence on how to improve learning, as opposed to increasing access (although many of the reviews also include evidence on the latter).<sup>2</sup> Note the caution that test scores, even when converted into standard

> 2 For the purposes of this paper, student learning is measured by test scores in math, language, science, or cognitive assessments, as determined by the inclusion criteria of the six systematic reviews. Specifically, the meta-analyses (three of the reviews) use standard deviations of test scores to measure learning impacts so as to allow comparison across different tests administered in different contexts.

3