Table 6: Ex post counterfactual: performance of exploration sampling and standard RCT

|||Exploratio|n Sampling|||Standa|rd RCT||
|---|---|---|---|---|---|---|---|---|
|Treat-<br>ment|Success<br>prob.<br>mean|Success<br>prob. SD|Prob.<br>treat<br>optimal|Posterior<br>exp. policy<br>regret|Success<br>prob.<br>mean|Success<br>prob. SD|Prob.<br>treat<br>optimal|Posterior<br>exp. policy<br>regret|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|
|T1A|6.96%|4.45%|4.33%|1.56%|7.28%|5.29%|6.14%|1.38%|
|T1B|8.50%|5.24%|91.93%|0.03%|8.59%|6.06%|85.13%|0.06%|
|T1C|6.89%|4.39%|1.59%|1.64%|7.27%|5.28%|6.40%|1.39%|
|T2A|6.12%|3.99%|0.11%|2.41%|5.92%|4.44%|0.05%|2.74%|
|T2B|6.77%|4.34%|2.03%|1.76%|6.92%|5.07%|2.28%|1.73%|
|T2C|3.89%|2.67%|0.00%|4.64%|4.03%|3.17%|0.00%|4.63%|
|Selected|8.50%|5.25%|92.58%|0.02%|8.60%|6.07%|86.39%|0.05%|

Notes: The table shows averages of estimates for each treatment arm obtained from 1,000 simulated samples drawn from the observed experimental data. Columns (1) and (5): mean posterior probability of a successful call. Columns (2) and (6): standard deviation of the posterior success probability. Columns (3) and (7): probability that the treatment arm is optimal. Columns (4) and (8): posterior policy regret in terms of engagement success probability.

and we append the bootstrapped sample to the observed wave 1 data to estimate a hierarchical Bayesian Binomial GLM as described in Eq. 2. We carry out 1,000 draws that simulate an RCT and 1,000 draws that simulate an exploration sampling experiment. For the simulated RCTs, we bootstrap a wave 2 of equal-sized treatment arms. For the simulated exploration sampling experiments, we use the treatment assignment shares derived from the original wave 1 posterior distributions.<sup>27</sup> For each sample draw, we calculate the posterior mean and standard deviation of _θ_<sup> ̄</sup><sup>_k_</sup> , the probability optimal, and the posterior expected policy regret for each arm. The averages for each arm across draws are shown in in Table 6. In addition, we show the average of the posterior regret and probability optimal of the selected (lowest-regret) arm _k_<sup>_∗_</sup> in each simulated experiment.

The average posterior mean of the probability of a successful call is similar between exploration sampling and standard RCT, as seen in columns (1) and (5). As expected, the standard deviation of the posterior distribution of the mean success probability _θ_<sup> ̄</sup><sup>_k_</sup> is lower under exploration sampling for the high-performing treatments, but higher for the low-performing arms. In both research designs, the treatment arm that is most often associated with the highest probability of engagement is T1B. However, in the exploration sampling experiment, T1B is chosen 97.4% of the time, whereas this is the case 94.9% of the time in the simulated RCTs. This reflects the greater uncertainty and consequently higher variance in the final decision that results

> 27This exercise is not perfect, because we re-sample from the six arms at different proportions for the two designs. Since we use data from both waves, the bootstrapped wave-2 sample is always smaller than the original sample we draw from. However, the probability of repeat draws is affected by both the size of the original arm and the target size, and this ratio varies across the two designs. An alternative approach is to use a randomly drawn sub-sample of the original data that is proportional to the targeted wave size. This equalizes the chance of repeat sampling across arms, but it implies that the two bootstraps draw from different underlying populations. Ultimately this second drawback seemed more problematic than the first.

38