were provided by the implementer, with random ID numbers replacing parents’ phone numbers and the child’s name and school.

We use IVR provider records to measure engagement with the calls, that is, whether the call recipient actually starts the exercises. Uliza’s records show every contact with the parent’s registered phone number, along with the length of each call in seconds. We define a call as successful if the parent started the first exercise, which requires tapping a phone key to confirm. We define a parent as having engaged in one of the twice weekly exercise sets if the IVR made at least one successful call in that set. Since there are 9 exercise sets per wave, engagement can take values between 0 and 9. Call records are available immediately, and they are complete and accurate.

The implementer measures children’s ORF scores during the midterm and endterm examination periods. In 2-3 hour periods set aside for the fluency test, a teacher examines each child by counting the number of words on a list that a child can read correctly in one minute of time (see Rodriguez-Segura et al. (2021) for the use of this measure to assess reading and literacy). The teacher then submits the scores to the school’s grade record system. ORF scores range between 0 and 85 correct words per minute (cwpm) based on the length of the provided word list.<sup>14</sup> There are a number of issues with ORF measurement, which were partly revealed only after wave 1 of the experiment had already started. Figure 1 shows the high variation in ORF scores between midterm and endterm. Among the non-missing scores, an unusually high proportion are multiples of five, and in some classrooms, there are implausibly many very high scores. In addition, a high percentage of scores are missing or submitted late to the recording system: ORF scores were available for only 73.5%-88.9% of children depending on the exam.<sup>15</sup> Teacher reports on why a given score is missing are often ambiguous. Overall, the data quality for ORF scores is fairly low.

**Targeted Outcome.** In order to use adaptive sampling, it is necessary to define an outcome measure that decides which is the “best” arm, which in turn determines which treatment arms will be sampled more. In many settings, this is not straightforward, given that multiple indicators related to the desired outcome(s) are typically available. Here, the implementer wants to increase parents’ engagement in their children’s education in general, because parental engagement is known to have positive effects on children’s

performance in school; at the same time, the calls explicitly encourage a set of reading exercises with the aim

> 14The implementer chooses a standardized, grade-appropriate word list, trains teachers and provides equipment. The measure can in principle range from zero to over 200, but for first graders it is typically not above 120.

> 15The total share of scores that are multiples of 5 is 36%, and the observed score distributions show unusual heaping even when accounting for censoring at 0 and 85. Teachers sometimes delay submission or entirely fail to submit exam scores for their class. We describe the patterns of missingness and suspected rounding in more detail in Appendix B in Tables A.1 and A.2. Part of the reason that the problems of missing and rounded scores persist is that at elementary school level, these scores do not affect the student’s progression into the next grade, nor do they affect the teacher’s evaluation.

17