example, if students learned 0.25 _σ_ per year in an intervention in South Africa, and students typically learn 0.25 _σ_ in a given year in South Africa, impacted students will have learned a year’s worth of schooling in South Africa. A few options exist for possible status-quo learning trajectories: for example, we might use the national average learning trajectory, or the learning gains in the control group of the same evaluation from which effect sizes are drawn. Alternatively, we could use an average learning profile across all countries being compared. This choice of status quo will affect our interpretation. If we choose the experimental or quasi-experimental control group, then the resulting figure for equivalent years of schooling gained is relevant to the study sample only. If we choose national average learning trajectories, we can interpret the figure as the number of equivalent years of schooling gained at the national level, with the embedded assumption that the standard deviations are comparable. In the main analysis reported above, we use national-level learning trajectories; in this section, we explore the alternative of using the study’s control group to measure status-quo learning.

An advantage of using control-group status-quo learning is that this estimate is drawn from the same sample as the learning gains from the intervention are, so the two are directly comparable. If the study sample is not representative of the nation, however, then when we later apply a national-level learning adjustment to compute LAYS in globally comparable terms, the adjustment will be less reliable. We can test the assumption of representativeness of study samples by examining the degree of variation within a country. If variation within a country is large, this means that any given study sample is likely to diverge from the national average. We test this assumption using a uniform test, EGRA. Two advantages of EGRA data are that (1) EGRAs are included in the World Bank Harmonized Learning Outcomes database and (2) they are often used to assess the impact of interventions and policies. In Appendix Figure B7, we compare variation within a country to variation across countries as a benchmark of whether variation within a country is large. We find that for a sample of 39 countries for which we have EGRA HLO scores, the average cross-country standard deviation is 53, whereas the within-country standard deviation is often higher than 53, with a density skewed to the right tail. This finding indicates that within-country variation is often quite large, casting doubt on the assumption that a sample would necessarily represents the nation. As a specific example, control-group status-quo gains in India in our sample of studies range from 0.5 _σ_ to 0.76 _σ_ , implying high variance from study to study.

In summary, we find that the assumptions for using control-group learning trajectories as our measure of the typical status-quo learning in a country are unlikely to be robust. We instead rely on national-level learning trajectories, which are easily interpretable and can be converted to a global metric. An additional advantage of using national-level learning trajectories is a practical one: greater data availability. Whereas control-group information is often missing from published papers, national learning trajectories can be calculated using HLO scores, which are available for 164 countries.

28