Table 3: Call engagement: treatment effect estimates after wave 2.

||Raw n|umbers|P|osteriors|of _β_<sup>_E_</sup>|eng|Average<br>agement <sup> ̄</sup>_θ_<sup>_k_</sup>|
|---|---|---|---|---|---|---|---|
|Arm|Call<br>attempts|Share<br>successful|Mean|SD|Prob.<br>optimal _p_<sup>_k_</sup><br>_t_|Success<br>prob. <sup> ̄</sup>_θ_<sup>_k_</sup>|Post. exp. policy<br>regret _ET_(∆<sup>_k_</sup>)|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|
|T1A|2_,_952|7.28%|-2.63|0.09|5.24%|7.43%|0.99%|
|T1B|7_,_020|8.40%|-2.49|0.07|93.19%|8.40%|0.02%|
|T1C|5_,_193|6.47%|-2.78|0.08|0.00%|6.49%|1.93%|
|T2A|2_,_052|5.95%|-2.89|0.11|0.00%|5.86%|2.56%|
|T2B|2_,_907|7.05%|-2.67|0.09|1.57%|7.15%|1.27%|
|T2C|2_,_034|3.98%|-3.32|0.13|0.00%|3.93%|4.49%|

Notes: (1) A call attempt is a scheduled call to a parent, 9 per wave (not counting repeated attempts and call backs). (2) The share successful is the percentage of call attempts in which the exercises were started. (3-4) The posterior mean and standard deviation of _β_<sup>_E_</sup> were calculated from a total of 8,000 posterior draws sampled from 4 independent Markov chains. (5) The probability optimal is calculated as in Eq. 4. (6) The average probability of success is calculated as in Eq. **??** . (7) The posterior expected policy regret is the expected loss from choosing this arm, expressed in terms of the probability of a successful call, after observing both waves of the experiment.

“probability highest” for each arm based on the extensive margin estimates, reported in column (2) of Table A.7. These probabilities are the analog of the probability optimal in column (5) of Table 3 (as these can be calculated for any outcome in any experiment, regardless whether adaptive sampling was used). These probabilities never exceed 35.3%. The point estimates and probability highest indicate that the six call formats are much less clearly differentiated based on the probability of “any engagement” than based on the overall call engagement rate. This suggests that the intensive margin matters, and differences in response rates emerge more clearly as parents learn about the calls and decide about continued engagement.

One interpretation of the results, comparing A and B arms, is that leveling exercise content in this setting is not valuable – perhaps because of the noisy and often missing ORF scores used for leveling – or at least not valued by parents, who may perceive the exercises as too easy or too difficult. Both C arms have relatively low call engagement rates. It is worth noting that the option to choose between exercises increases the length of the call, which may discourage the listener. The call success rate in T2C is particularly low, and we conjecture that this is because the listener is not only asked to choose which exercises to play, but the IVR here also addresses the child directly. This “gamification” aspect may lead the parent to worry about overly long calls in which the child skips around between exercises. Between T1 and T2, the posterior means suggest that T1 arms have slightly higher engagement rates, perhaps because the “listen now, practice later” format allows the parent more flexibility.

The sampling shares in Table 2 and the numbers of attempted and successful engagements in Table 3 also demonstrate a property of adaptive sampling that is attractive in the context of policy choice: the 30