from a non-adaptive experiment.

Exploration sampling increases the probability optimal of the best arm on average from 86.39% to 92.58% and reduces the average posterior regret from 0.05% to 0.02%. The reduction is small in absolute terms for two reasons; first, the student sample is large enough so that even an RCT would lead the researcher to relatively firm conclusions here, and second, in this particular problem instance it turns out that the arm averages are clustered closely together, meaning that even a suboptimal choice is likely to be benign. However, in relative terms the improvement is large, and in a policy choice problem where the best arm is actually implemented, even small per-unit gains in payoffs may accumulate into large welfare differences. Overall, the ex-post simulations suggest that we can achieve a meaningful decrease in uncertainty and improved decisions from just one wave with adaptive sampling involving half of the experimental sample. _Remark: Decision Metrics._ These simulations highlight an advantage of the proposed Bayesian approach: the metrics of expected policy regret and probability optimal provide the decision maker with easy to understand, intuitive measures of the uncertainty attached to the policy choices they are making. This facilitates the comparison of treatment arms as well as experimental research designs.

# **6.2 Ex Ante Comparison: Model-Based Simulation of Exploration Sampling vs. RCT**

In the second exercise, we imagine the experimenter asking before the IVR experiment, “should I carry out one (non-adaptive) wave with the whole sample, or two (adaptive) waves with half the sample each?” For these simulations, take a given parameter vector ( _β_<sup>_E_</sup> _, κ_<sup>_E_</sup> ). Based on this vector and Eq. (2), we can simulate outcomes _Zi_<sup>_sk_</sup> for the students in each wave (drawing the school effects _ηs_<sup>_E_from the Standard Normal</sup> distribution). The first simulated sample uses equal assignment shares, the second is generated under an adaptive design, where the assignment shares for wave 2 are obtained from estimating our model above from the simulated wave 1 data. We can then compare the estimation results under these two sampling strategies to calculate the predicted gains from the adaptive vs. the non-adaptive design for the given parameter vector. This is reminiscent of conducting power calculations for an assumed effect size.

Panel A of Table 7 shows the result of such an exercise, using as the parameter vector the mean of the posterior distributions of _β_<sup>_E_</sup> and _κ_<sup>_E_</sup> after wave 2, as reported in Table 3. Using the wave-2 estimates from the experiment serves to show how well the ex ante simulation does in predicting these estimates, and how ex ante simulation results compare with the ex post simulation above. The predicted gains from using adaptive sampling in terms of posterior regret are very similar to our previous exercise based on the actual IVR data. The average posterior expected regret from arm T1B is 0.02% with adaptive sampling but 0.08% with the “standard RCT” on average. The average posterior probability optimal for both sampling strategies is also similar to what we obtained in Table 6.

39