# 4.4 Why Is Persistence So Low?

The low estimates of persistence are worrying, not only for program evaluation but for the more substantive issue of how to improve cognitive achievement. One major concern is that imperfect persistence is a psychometric testing issue, and therefore not a “true” feature of the learning dynamic. For instance, later test forms may capture fundamentally different latent traits than earlier test forms. To address this concern, we replicated our results using IRT scores based solely on a common set of items that appeared on every test form—our tests had a significant number of overlapping items in each year. Our results were similar using these scores, with the difference GMM persistence estimates in fact dropping slightly. The score equating methods used to create a single cardinal measure of learning therefore do not appear to be the driving force behind the low observed persistence.

Several other possible mechanical explanations for low persistence are also unlikely. First, artificial ceiling effects can appear like low persistence in models that use bounded scores. To address this concern, we exclusively use unbounded IRT scale scores and our exam is designed to maximize the variation over the entire range of observed abilities. Second, cheating, often driven by high stakes testing, can create artificially low estimates of persistence. Jacob and Levitt (2003), for instance, detect teacher cheating in part by looking for poor subsequent performance of students who made rapid gains. In our data, cheating is unlikely both because our test is relatively low stakes and because our project staff administered the exam directly to avoid this possibility. Third, critics of high stakes testing often argue that shallow “teaching to the test” leads to low persistence. This is also an unlikely explanation in our context; our exam is relatively low stakes, is not part of the standard educational infrastructure, and covers only subject matter that all students should know and that Pakistani parents generally demand.<sup>15</sup> We looked at a couple of other candidate explanations, but there are no “smoking guns” that could explain low persistence. Tables 8 and 9, for instance, present the results of a preliminary exercise that assesses whether low persistence arises from household and school responses to unexpected achievement shocks—an explanation that has different implications for cost-benefit analysis than simple forgetting.<sup>16</sup> Parents’ perceptions of their child’s performance reacts strongly to unexpected gains in achievement, but there is only weak evidence of substi15On average, the children tested at the end of Grade 3 could complete two-digit addition, but not subtraction or multiplication (in mathematics); recognize simple words (but not sentences) in the vernacular (Urdu); and recognize alphabets and match simple three-letter words to pictures in English.

16We examine whether inputs adjust to unexpected achievement shocks for roughly 650 children for whom we have detailed information from a survey collected at households. As a measure of the unexpected shock, we first compute the residual from a regression of fourth grade scores on third grade scores and a host of known controls. We then test whether this residual predicts changes between fourth and fifth grade in parents’ perceptions of the child’s performance, expenditure on school, and time spent helping children on homework, being tutored, doing homework, and playing.We instrument for the subject specific residuals using the alternate subject residuals to lessen measurement error attenuation.

18