Paper: Fixed-Budget Best Arm Identification with Variance-Dependent Regret Bounds (ICLR-2024)

Reviewer comment (Theory):
Page 6, Line 2 after Theorem 3.8, I suppose $X_t$ should be $X$ in the inequalities? In addition, I do not follow why $E^X[\frac{(\sigma^1(X))^2}{w(1|X)}+\frac{(\sigma^2(X))^2}{w(2|X)}]\geq\max_{a\in[2]}\sqrt{E^X[\frac{(\sigma^a(X))^2}{w(a|X)}}]$, as $\frac{1}{8}+\frac{1}{8}<\sqrt{1/8}$.

Author response:
**Q2**.  
> Why $E^X[\frac{(\sigma^1(X))^2}{w(1|X)}+\frac{(\sigma^2(X))^2}{w(2|X)}]\geq\max_{a\in[2]}\sqrt{E^X[\frac{(\sigma^a(X))^2}{w(a|X)}}]$?

**A2**. 
This is our typo. Thank you for pointing it out. It should be $\inf_w E^X[\frac{(\sigma^1(X))^2}{w(1|X)}+\frac{(\sigma^2(X))^2}{w(2|X)}]\geq \inf_w  \max_{a\in[2]}\sqrt{E^X[\frac{(\sigma^a(X))^2}{w(a|X)}}]$; that is, the optimization for $w$ is lacked.

Reviewer comment (Theory):
Given that the optimal allocation is known, is it possible to adopt a tracking sampling rule, i.e., sampling the arms in a way such that the empirical arm allocation approaches the optimal allocation. As indicated by section 2.3 in [4], sampling according to a distribution can make the convergence speed slow. Can you give comments on the allocation rule?

Author response:
**Q**. 
> here the ``optimal allocation'' refers to the computed allocation based on the empirical estimates ... Therefore, is it possible or better to use a tracking-type sampling rule, which samples the arm $A_t = \arg\min_{a} T_t(a|X_t)-t \hat{w}_t(a|X_t)$ (where $T_t(a|X_t)$ is the number of times arm $a$ is sampled under $X_t$ up to time step $t$)?

**A**. 
Thank you for your suggestion. We conjecture that it works. However, the derivation of the upper bound requires more high-level techniques. We explain the background of the AIPW estimator as follows.

----------------

For simplicity, assume that there is no contextual information. Recall that the AIPW estimator: $\hat{\mu}^{AIPW, a}_T$ is given as

$$\frac{1}{T}\sum_t\Big(\frac{1[A_t = a](Y_t - \hat{\mu}^a_{t-1})}{w_t(a)} +  \hat{\mu}^a_{t-1}\Big).$$

Here, our proof is based on the martingale property, depending on $\mathbb{E}[1[A_t = a] | \mathcal{F}_{t-1}] = w_t(a)$.

If we apply the tracking-based algorithm, we need to reconsider the definition of the AIPW estimator. Depending on the definition, there are some technical issues. For example, consider using the same definition of the AIPW estimator in the main text. Then, $w_t$ is an estimator of the optimal allocation, and $A_t$ is an actual arm draw. Then, $\mathbb{E}[1[A_t = a] | \mathcal{F}_{t-1}] = w_t(a)$ does not hold. Therefore, we cannot employ the martingale property. 

----------------

However, we consider that there is a possibility that we can show the same property between the AIPW estimator with random sampling in our manuscript and tracking-based sampling in your proposition. To derive an upper bound of the AIPW estimator using the  tracking-based sampling, we can consider the following decomposition:
$$ \hat{\mu}^{AIPW, a}_T = \hat{\mu}^{AIPW, a}_T - \tilde{\mu}^{AIPW, a}_T +\tilde{\mu}^{AIPW, a}_T,$$ 
where we define $\tilde{\mu}^{AIPW, a}_T$ as 

$$\frac{1}{T}\sum_t\Big(\frac{1[B_t = a](Y_t - \hat{\mu}^a_{t-1})}{w_t(a)} +  \hat{\mu}^a_{t-1}\Big),$$

and define $B_t \in [K]$ as a hypothetical arm assignment indicator when an arm is drawn following the probability $w_t$. Then, we aim to show 

$$ \hat{\mu}^{AIPW, a}_T - \tilde{\mu}^{AIPW, a}_T = o_P(1/\sqrt{T})$$

and apply the martingale central limit theorem for $\tilde{\mu}^{AIPW, a}_T$, where $\mathbb{E}[1[B_t = a] | \mathcal{F}_{t-1}] = w_t(a)$. 

----------------

The above conjecture is based on the existing work, such as [1] and [2]. 

[1] Hahn, J., Hirano, K., and Karlan, D. Adaptive experimental design using the propensity score, 2011.
[2] Kato, M., Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probability, 2021.

However, to conduct the proof procedure we mentioned above, there are several restrictions. For example, [1] only considers a two-stage adaptive experiment, where we draw each arm with the equal ratio in the first stage and draw each arm to track the empirical optimal allocation ratio. Here, we cannot update $w_t$ in the second stage; that is, we can update the sampling rule $A_t$ only once between the first and second experiments. [2] makes complicated assumptions that are not easily verified. Showing the upper bound under the tracking-based algorithm without restricting the sampling rules or making additional assumptions is an open issue.

----------------

> According to [4], sampling according to a distribution can have slow convergence speed. 

We agree with your points. On the other hand, we raise the following three points:
- Although the sample average of $A_t$ will converge with a faster speed, we may not construct an unbiased estimator for $\mu^a(P)$. Therefore, there might be a trade-off between the convergence of $A_t$ and the bias of an estimator of $\mu^a(P)$.
- We can show that our proposed AS-AIPW strategy and that with the sampling rule, we mentioned in the previous post (that is, we assign first $\lceil Tw^*(1)\rceil$ observations to arm $1$,..., and $\lceil Tw^*(K)\rceil$ observations to arm $K$). have the same asymptotic distribution. From this result, we conjecture that the AS-AIPW estimator with the tracking-based algorithm also has the same distribution if it can be shown. 
- Existing studies (e.g., [1] and [3]) imply that we can conduct theoretical analysis for the AS-AIPW estimator with random sampling more easily than with the tracking sampling, owing to the martingale property.

[3] van der Laan, M. J. The construction and analysis of adaptive group sequential designs, 2008.

----------------

In summary, we agree with your point that tracking-based strategies will outperform random-sampling-based strategies. However, under such a strategy, we cannot employ the martingale property (= unbiasedness), which makes the theoretical analysis difficult. Revealing the theoretical properties of such an estimator is an open issue.

Reviewer comment (Experiments):
It would be better if you describe the experiment designs in more detail. In particular, for the Sequential Halving-based algorithm, it only recommends an arm at the end of the experiment, how to compute the simple regret of the arm at $t\in \{ 1000,\dots,50000 \}$? And it would also be interesting to see how SHAdaVar behaves without contextual information while the other algorithms have access to contextual information in App. I.2.

Author response:
**Q5**.  
> for the Sequential Halving-based algorithm, how to compute the simple regret of the arm at each $t\in { 1000,\dots,50000 }$? 

**A5**.  
Thank you for your suggestion. As you pointed out, we conducted a new independent trial for the Sequential Halving-based algorithm for each $t$. We clarify this point in the next update.

------------------

**Q6**.  
> And it would also be interesting to see how SHAdaVar behaves without contextual information.

**A6**. 
We will add the experiments. If possible, we will update the manuscript, including the new experiments, during the rebuttal phase. Otherwise, we will add them when they are camera-ready.

Reviewer comment (Experiments):
For the other algorithms, how you incorporate contextual information in the original algorithm.

Author response:
**Q7**.  
> for the other algorithms, how you incorporate contextual information in the original algorithm.

**A7**.  
In the experiments in the Appendix, we only use contextual information for our proposed algorithm. For fairness, we do not use contextual information both for our proposed and existing algorithms in the experiments of the main text. We clarify these points in the next revision. Note again that the proposition of contextual BAI algorithm is also our contribution, and to the best of our knowledge, there is no appropriate competitors that use contextual information.

Reviewer comment (Theory):
only asymptotic theoretical results are provided. For fixed budget settings, non-asymptotic bounds can provide a better understanding of algorithm performance under a fixed budget.

Author response:
We showed the non-asymptotic upper bound in Appendix H. For the lower bound, we consider it is not easy to derive non-asymptotic results for the following reasons.
1. Although the estimation error of the variances can be ignored in the worst-case asymptotic analysis, it affects the lower bound in the non-asymptotic analysis. However, the analysis of variance estimation will be too complicated to analyze.
2. Our technique is based on the information-theoretic lower bound provided by Kaufmann et al. (2016) and semiparametric efficiency bounds. Both techniques are used for asymptotic analysis. Therefore, we need to develop completely different approaches for non-asymptotic analysis.
3. We deal with general distribution by approximating the KL divergences. If we focus on the non-asymptotic analysis, we need to restrict the class of distributions to specific distributions, such as the Gaussian distribution, even if possible.

One of the promising approaches for lower bounds is to employ lower bounds provided by Carpentier and Locatteli (2016). However, this lower bound is based on the boundedness of $Y^a_t$. Without the boundedness, non-asymptotic analysis would be more difficult. Additionally, the definitions of optimality might be changed to deal with the uncertainty of variance estimation. Thus, although the non-asymptotic upper bound has been derived, deriving non-asymptotic lower bounds requires different techniques and is not straightforward, even if possible. We will explain this open issue in the next update.

Carpentier, A. and Locatelli, A. Tight (lower) bounds for the fixed budget best arm identification bandit problem, 2016.

Reviewer comment (Theory):
The proposed algorithm AS-AIPW highly depends on the variance estimation. It is unclear how poor estimations at the early stage and for more general distributions influence non-asymptotic performance. A discussion can be provided.

Author response:
As we answered in A1 and you pointed out, the non-asymptotic performance will be affected by variance estimation. We showed the non-asymptotic upper bound in Appendix H by assuming the convergence rate of the variance estimation. Unless we assume that a specific convergence rate is given for the variance estimation or variances are known, it is difficult to derive asymptotic results. We will explain this limitation more clearly in the next update.

Reviewer comment (Writing):
can you define w(a|x) in Theorem 3.4?

Author response:
It is defined as a element of $\mathcal{W}$, which is defined in the above of Theorem 3.4.

Reviewer comment (Novelty):
A related work: On Best-Arm Identification with a Fixed Budget in Non-Parametric Multi-Armed Bandits, Barrier et al 2023. Can you discuss this?

Author response:
First of all, while we consider the expected simple regret minimization, that work considers minimization of the probability of misidentification. These two settings are related but require different analyses for lower bounds. See Komiyama et al. (2023).

Additionally, there are the following critical differences between our study and theirs in non-parametric analysis using the KL divergence of Kaufmann et al. (2016).
 - In the lower bound, our study approximates the KL divergence by the Fisher information (semiparametric influence function) around the gaps between the best and suboptimal arms are zero $\Delta^a(P) \to 0$. We do not assume the boundedness of $Y^a_t$. In the upper bound, we utilize the central limit theorem for deriving tight results. 
- Barrier et al. (2023) assumes the boundedness of $Y^a_t$. Then, bounding the KL divergences using the boundedness without using the small gap (fixed $\Delta^a(P)$).

Because we employed the worst-case analysis, which implies $\Delta^a(P) \approx 1/\sqrt{T}$ (see Section 3), we could naturally develop non-parametric results. However, we cannot employ such an approximation in Barrier et al. (2023) because it considers lower and upper bounds under a fixed $P$ or fixed $\Delta^a(P)$. Furthermore, unlike the expected simple regret minimization, the optimality in minimization of the probability of misidentification is still an open issue. Also see Komiyama et al. (2023) and Qin (2022). We remark on these differences in the next update.

Komiyama, J., Ariu, K., Kato, M., and Qin, C. Rate-optimal bayesian simple regret in best arm identification, 2023.  
Qin, C. Open problem: Optimal best arm identification with fixed-budget, 2022.

Reviewer comment (Presentation):
in Figure 1, can you also draw the standard deviation of the independent trials?

Author response:
Thank you for your suggestion. We will add the standard deviations or the boxplot in the next update.

Reviewer comment (Theory):
One weakness is that the current Theorem 3.8 in this paper does not generalize to the case with $K\geq 3$ and it is not clear whether the difficulty is technical or fundamental.

Author response:
**Q2**.  
> It seems quite tempting to conjecture that Theorem 3.8 can be generalized to the case with $K\geq 3$. 

**A2**.  
We agree with the reviewer's comment and also consider that extending Theorem 3.8 for $K\geq 3$ is an important open issue. To address this issue, we believe that current assumptions for distributions and restrictions for strategies are insufficient. We need to add more assumptions and develop additional tools for this problem. 

For example, the framework of the limit of experiments (van der Vaart, 1998) is one of the promising directions. This framework is expected to allow us to derive tight lower and upper bounds by using the asymptotic normality. However, to apply the results, we need to restrict strategy class and underlying distribution appropriately.  

Recently, Kato (2023) derived results similar to Theorem 3.8 for $K \geq 3$ in minimization of the probability of misidentification ($\mathbb{P}_P(\hat{a}_T \neq a^*(P))$). However, the results are limited to cases with known variances, minimization of the probability of misidentification, and Gaussian distributions. It is an open issue how we derive similar results for expected simple regret minimization.

Kato (2023), Locally Optimal Best Arm Identification with a Fixed Budget. 

We summarize this problem as an open issue.

Reviewer comment (Experiments):
Another weakness is that most provided experiment results do not show advantages of **AS-AIPW** over variance-unaware algorithms. Although the paper conjectures that the superiority of **AS-AIPW** can only appear when $K$ is small, from my perspective, the number of arms should not be the essential factor. In particular, the worst-case regret of variance-unaware algorithms scales with the magnitude of the reward while that of **AS-AIPW** scales with the standard deviation of the reward. Therefore, if my understanding is correct, the advantage of **AS-AIPW** should appear if we run it on a hard instance with large reward magnitude but small variance. Is that possible to design and run experiments on such an instance?

Author response:
**Q**.   
> Another weakness is that most provided experiment results do not show advantages of AS-AIPW over variance-unaware algorithms. Although the paper conjectures that the superiority of AS-AIPW can only appear when $K$ is small, from my perspective, the number of arms should not be the essential factor. In particular, the worst-case regret of variance-unaware algorithms scales with the magnitude of the reward while that of AS-AIPW scales with the standard deviation of the reward. Therefore, if my understanding is correct, the advantage of AS-AIPW should appear if we run it on a hard instance with large reward magnitude but small variance. Is that possible to design and run experiments on such an instance?

**A**. 
We appreciate your suggestion. 

First, we can run an experiment with a large reward magnitude but a small variance. We will add the experiment in the camera-ready if it cannot be finished during this rebuttal phase. 

Second, we believe that there are two effects.

> the worst-case regret of variance-unaware algorithms scales with the magnitude of the reward while that of AS-AIPW scales with the standard deviation of the reward

We conjecture that our proposed strategy outperforms existing methods in experiments with harder instances (smaller gaps) with larger $T$. 

>  the advantage of AS-AIPW should appear if we run it on a hard instance with large reward magnitude but small variance. 

Additionally, we agree with your comment above. In existing studies, such as economics and epidemiology, variance-aware adaptive experiments are often employed in situations with large reward magnitude. Therefore, such settings are more suitable for our algorithm.

We will conduct experiments and add discussion in the future update (if possible, we will finish it during the rebuttal phase).

Reviewer comment (Theory):
Based on the given results, it seems quite tempting to conjecture that Theorem 3.8 can be generalized to the case with $K\geq 3$. Is this fundamentally not doable or does it just require more sophisticated techniques?

Author response:
**Q2**.  
> It seems quite tempting to conjecture that Theorem 3.8 can be generalized to the case with $K\geq 3$. 

**A2**.  
We agree with the reviewer's comment and also consider that extending Theorem 3.8 for $K\geq 3$ is an important open issue. To address this issue, we believe that current assumptions for distributions and restrictions for strategies are insufficient. We need to add more assumptions and develop additional tools for this problem. 

For example, the framework of the limit of experiments (van der Vaart, 1998) is one of the promising directions. This framework is expected to allow us to derive tight lower and upper bounds by using the asymptotic normality. However, to apply the results, we need to restrict strategy class and underlying distribution appropriately.  

Recently, Kato (2023) derived results similar to Theorem 3.8 for $K \geq 3$ in minimization of the probability of misidentification ($\mathbb{P}_P(\hat{a}_T \neq a^*(P))$). However, the results are limited to cases with known variances, minimization of the probability of misidentification, and Gaussian distributions. It is an open issue how we derive similar results for expected simple regret minimization.

Kato (2023), Locally Optimal Best Arm Identification with a Fixed Budget. 

We summarize this problem as an open issue.

Reviewer comment (Theory):
What are the disadvantages of using $\arg\max_{a\in[K]}\widehat{\mu}^a_T$ as the arm recommendation rule? Do these disadvantages exist when there is no context information?

Author response:
**Q3**.  
> What are the disadvantages of using $\arg\max_{a\in[K]}\hat{\mu}^a_T$? 

**A3**..       
We conjecture that both $\hat{\mu}^a_T$ and $\hat{\mu}^{AIPW, a}_T$ have almost the same theoretical properties when there is no contextual information. However, proving the regret upper bound for $\hat{\mu}$ requires a more complicated proof procedure compared to $\hat{\mu}$ or requires some additional assumptions. This problem is related to theories of empirical process and martingales, as discussed in Hirano et al. (2003) and Hahn et al. (2011). 

We believe that showing it is not straightforward and an open issue, even though it is probably possible . We also summarize this problem as an open issue in another reply or the Appendix.

Reviewer comment (Theory):
If my understanding is correct, when there is no context information, we have $\widehat{\mu}^a_t=\frac{1}{t}\sum_{s=1}^{t}\mathbf{1}\lbrace A_s=a\rbrace Y_s$. Then, it looks weird that $\widehat{\mu}^{\mathrm{AIPW}, a}\_T$ contains the term $$\frac{1} {T}\sum_{t=1}^{T}\widehat{\mu}^a_t=\frac{1}{T}\sum_{t=1}^{T}\left(\sum_{s=t}^{T}\frac{1}{s}\right)\mathbf{1}\lbrace A_t=a\rbrace Y_t,$$ since it means that more weights are explicitly put on earlier samples. Why will this happen?

Author response:
**Q4**.   
> It looks weired that $\hat{\mu}^{AIPW}$ contains the term $\frac{1}{T}\sum^T_{t=1}\hat{\mu}^a_t$.

**A4**.   
This term is introduced for variance reduction with keeping the martingale properties for $\frac{1}{T}\sum^T_{t=1}\hat{\mu}^a_t$. Here, an effect of $\frac{1}{T}\sum^T_{t=1}\hat{\mu}^a_t$ vanishes very quickly because of the existence of the product
$|\hat{w}(a) - w^*(a)||\hat{\mu}^a(P) - \hat{\mu}^a_t|$ in theoretical analysis, which accelerates the vanish of the estimation error of $w^*$ and $\mu^a(P)$. That is, the introduction of $\frac{1}{T}\sum^T_{t=1}\hat{\mu}^a_t$ reduces the variance of the main estimator without increasing bias (asymptotically). This technique is used in different literatures, such as van der Laan (2008) and Chernohukov et al. (2018), which are recently called double machine learning.

This property is related to the Q3 the reviewer raised. We add more explanations on this problem in the Appendix.

Reviewer comment (Theory):
Suppose there are not many possible contexts and we can encounter each context for sufficiently many times, can we treat **AS-AIPW** as doing BAI for each context independently? That is, if we define $$\widehat{a}^{\mathrm{AIPW}}\_T(x)=\arg\max_{a\in[K]}\frac{1}{T}\sum_{t=1}^{T}\mathbf{1}\lbrace X_t=x\rbrace \varphi^a_t(Y_t, A_t, X_t),$$ can we bound the simple regret condition on $X=x$ by $$\max_{a, b\in[K]: a\neq b}\sqrt{\log(K)\left(\frac{(\sigma^a(x))^2}{w^*(a\vert x)}+\frac{(\sigma^b(x))^2}{w^*(b\vert x)}\right)}+o(1)?$$

Author response:
**Q5**.   
> Suppose there are not many possible contexts and we can encounter each contexts for sufficiently many times, can we treat AS-AIPW as doing BAI for each context independently?

**A5**.  
Yes. We can conduct context-specific treatment recommendation is possible when there are not many discrete contextual information. 

We are now extending the result for policy learning with BAI; that is, given a set $\Pi$ of policies $\pi:[K]\times\mathcal{X} \to (0, 1)$ such that $\sum_{a\in[K]}\pi(a|x) = 1$ (and potentially continuous contextual information), we train a policy $\pi\in\Pi$ to minimize the regret. Consider we restrict a policy class $\Pi$ to the one such that we discretize the contextual information and recommend an arm within each discretized contextual information. Then, such a strategy aligns with the strategy that the reviewer suggested. 

The remaining open issue is how we bound the regret for general $\Pi$. Because samples are non-i.i.d., we cannot directly apply the standard complexity measure such as the Rademacher complexity. This open problem has garnered attention in this literature, and we summarize it in the next update.

Reviewer comment (Writing):
The introductory section is not clear.

Author response:
Once again, we deeply appreciate your constructive comments. Following your advice, we will make our contribution more explicit in the Abstract and other sections, as we currently mention that "Note that this setting is a generalization of fixed-budget BAI without contextual information, and our result holds novelty even in the absence of contextual information." in Introduction. In future updates, we will add some examples to the Appendix, in the rebuttal phase, otherwise in the camera-ready.

Reviewer comment (Writing):
I would appreciate (at least a paragraph) on some motivating examples (also in the appendix) with some comments.

Author response:
Once again, we deeply appreciate your constructive comments. Following your advice, we will make our contribution more explicit in the Abstract and other sections, as we currently mention that "Note that this setting is a generalization of fixed-budget BAI without contextual information, and our result holds novelty even in the absence of contextual information." in Introduction. In future updates, we will add some examples to the Appendix, in the rebuttal phase, otherwise in the camera-ready.

Reviewer comment (Writing):
Some crucial quantities are not commented on, such as the meaning of the AIPW estimator.

Author response:
**A6**. 
We explained the details about the AIPW estimator in Section 4.3.  The name comes from the augmentation of the inverse probability of weighting estimator. We add some brief history of this estimator in the next update.

Reviewer comment (Experiments):
In the experimental section (even in Appendix I) the experimental settings are too simple. Even if there is a comparison with fixed-budget BAI when no contextual information is available, it is not fair to compare the same baselines (that are not designed for contextual settings) with the proposed algorithm. It is suggested to employ at least another algorithm designed for the same setting proposed by the authors if present (and if no competitors are available, this should be stated).

Author response:
**A2**. 
In our main text experiments, **we compare our method with existing methods, both without using contextual information,** as stated in the third line of Section 7: "We investigate two setups with K = 2, 3 without contextual information for these strategies." This ensures a fair comparison. 

In the Appendix, we compare our method that uses contextual information with existing methods that do not use contextual information.

**A3**. 
To the best of our knowledge, this setting has no appropriate competitors. In BAI, it is still unclear how we formulate the problem if contextual information is available, and it is an open issue. We will make it clearer in the next update.

Reviewer comment (Reproducibility):
Will you release the code of the experiments to assess if the numerical validation is reproducible?

Author response:
**A5**. 
We plan to release our experiment code, ensuring it complies with the review process's anonymity requirements. It will be available during the rebuttal phase.