may want to carry out the calculation in Panel B of Table 7 for multiple parameter vectors in order to get a sense of the distribution of gains from adaptivity. The most consistent approach would be to draw many values from the prior distribution of the model parameters, but this can give a misleading picture of the gains from adaptivity when uninformative priors are used (not to mention that the computational cost is high). As an example, in the IVR experiment, the flat priors combined with the logit transformation in the model mean that treatment arm averages based on random draws from the distributions of the _β_<sup>_E_</sup> are almost always close to 0 or 1. In Appendix C.3, we therefore show results from a modified exercise in which we independently and randomly draw the _θ_<sup>_k_</sup> from the uniform distribution on [0 _,_ 1]. As it turns out, this exercise is not meaningful either: in many cases, the drawn parameters are so far apart that, given our large sample of students, even equal assignment shares lead to a very high probability of picking the correct arm. A more meaningful approach might be to assume correlated prior distributions for the _βk_<sup>_E_,orusetheprior</sup> same distribution for each _θ_<sup>_k_</sup> but with a mean obtained from pilot data. An alternative to drawing from a prior distribution is to examine learning gains for a few well-chosen parameter vectors. Again, this is similar to the approach taken in typical power calculations for experiments, see (e.g. Duflo et al., 2007).

_Remark: When is Adaptive Sampling Most Valuable?_ As the simulations show, the efficiency gains from adaptive sampling vary significantly across different problem instances. For best-arm identification, closely clustered treatment effect averages make the problem “hard,” as it is difficult to distinguish these arms. From a welfare-maximization (regret minimization) perspective, however, two or more treatment arms with very similar success rates may often lead to a sub-optimal choice, but the loss from that choice will be small. Intuition suggests that adaptive sampling is particularly valuable when there are two or more “nearoptimal” arms but also several “far from optimal” arms that can be quickly ruled out. An example could be an experiment that compares two or more different types of interventions but also tests several variants within each type. It will be fruitful to explore these questions in more detail.

# **6.3 Comparison after Wave 1: Model-Based Simulation of a Second Adaptive Wave**

In our last exercise, we imagine the experimenter, after having carried out wave 1, asking, “should I conduct a second adaptive wave?” This is somewhat less computationally costly than the above exercise because after wave 1, the exploration sampling shares for wave 2 are known. As before, for a given parameter vector _β_<sup>_E_</sup> and _κ_<sup>_E_</sup> , we simulate a second wave of the experiment by generating a random sample of size _N_ = 1384 following the model in Eq. (2) and using the assignment shares in Table 2. We draw 200 parameter vectors from the wave-1 posteriors and calculate average policy regret and percentage of times the best arm is identified for each.

Panel A of Table 8 shows the posterior expected regret and probability optimal for arm T1B after wave 41