Paper: Multi-Armed Bandits with Abstention (ICLR-2024)

Reviewer comment (Theory):
Significance : In terms of real-life applications (clinical trials), I am not totally convinced by the considered abstention model(s). In both fixed-regret and fixed-reward settings, the fact that the agent gets access to the stochastic reward no matter the value of Bt does not seem realistic, as it implies that there is some estimation of the reward/regret from pulling a treatment arm (referring to paragraph 3 on page 1). Please correct me if I am wrong.

Author response:
We do not fully understand concerns raised by the reviewer, and we welcome any additional comments. In particular, we do not fully understand the phrase "no matter the value of $B_t$ does not seem realistic, as it implies that there is some estimation of the reward/regret from pulling a treatment arm". The agent indeed observes the stochastic outcome $X_t$ after selecting an arm $A_t \in [K] $ and the abstention option $B_t\in \\{0,1\\}$.   In the example of clinical trials presented in the introduction, if researchers implement costly prearranged measures, the treatment outcomes will not contribute to the optimization objective of the study. However, the researcher can still observe these outcomes, thereby leading to more efficient future experimental designs. If the researcher is unable to observe the outcomes, then conducting medical treatments becomes meaningless. In line with this model assumption, our model well aligns with Neu \& Zhivotovskiy (2020), which explored the role of abstention in the context of online prediction with expert advice.  

The abstention decision $B_t$ must be made before observing the outcome $X_t$. Consequently in each time step, the agent utilizes  estimates of the reward/regret from pulling the chosen arm to make the abstention decision. 

> Gergely Neu and Nikita Zhivotovskiy. Fast rates for online prediction with abstention. In Conference on Learning Theory, pp. 3030–3048. PMLR, 2020.

Reviewer comment (Evaluation):
In the experimental sections (both in the main paper for the fixed-regret setting and in the Appendix for the fixed-reward setting), in the plots at fixed T=10,000, perhaps it would be fairer to report the lines corresponding to the regret incurred by the baselines at T=10,000 in each setting, as the baselines seem to fare (slightly) better than the proposed algorithms for lower values of c (in the fixed-reward setting: ex. c=0.5 in the first instance) and for greater values of c (in the fixed-regret setting: ex: c=0.4 in the first instance) than shown in the empirical regret curves.

Author response:
We extend our sincere appreciation to the reviewer for offering constructive suggestions. In the revised manuscript, we have presented the performance of baselines in Figures 3 and 5, and provided a concise summary of the subsequent explanation. When the abstention reward $c$ is lower than the smallest mean reward in the fixed-reward setting, or the abstention regret $c$ exceeds the largest suboptimality gap in the fixed-regret setting, the agent gains no advantage in choosing the abstention option when selecting any arm. Unfortunately, the agent lacks precise knowledge of the bandit instance, leading to the inevitable selection of the abstention option. This unavoidable choice introduces a subtle gap between the performance of our algorithms and the baselines.

Reviewer comment (Theory):
epsilon(=1/K)-TS is asymptotically and minimax optimal for other one-dimensional exponential families (Poisson, Gamma, etc.). Since your algorithmic contributions partially rely on the analysis of epsilon-TS, how hard would it be to extend your results beyond Gaussian distributions?

Author response:
Epsilon(=1/K)-TS is asymptotically and minimax optimal for various popular reward distributions, including Gaussian, Bernoulli, Poisson, and Gamma. The extension beyond Gaussian distributions appears to be straightforward based on our current analysis, although careful consideration is necessary for specific "boundary" cases. For instance, in Bernoulli bandits where the stochastic rewards are bounded in $[0, 1]$, the behavior of minimax regrets might be different from the canonical case of $O(\sqrt{KT})$, when the abstention reward approaches 1 in the fixed-reward setting.

Reviewer comment (Experiments):
Insufficient numerical experiments: While the authors have carefully chosen the arm means $\mu^{\dagger}$ and $\mu^{\ddagger}$ for the experiments, I would like to see the numerical experiments for arbitrary values of arm means and large number of arms.

Author response:
Our paper is primarily positioned as a theoretical work, with numerical experiments serving to validate our theoretical findings. The selection of instances $\mu^{\dagger}$ and $\mu^{\ddagger}$ was not deliberate. In Appendix E.2 of the revised manuscript, we have heeded the reviewer's advice and incorporated additional numerical experiments, using random instances with large numbers of arms. The construction of these random instances follows the method in Jin et al. (2023). Due to time constraints and for the sake of simplicity in presentation, we focus on the fixed-regret setting. It can be seen that the results in Figures 6 and 7 closely resemble those in Figures 2 and 3.

Reviewer comment (Writing):
"Gaussian" is spelt incorrectly in Lemmas 3 and 4 in Appendix B (independent $\sigma$-sub-"Guassian").

Author response:
Thank you for bringing the typo and the presentation issue to our attention. In the revision, we have corrected the typo and explicitly clarified that $\beta(b, \mu, K)$ is defined in Lemma 5.

Reviewer comment (Writing):
While proving $(\star)_{i}$ in the asymptotic upper bound for the fixed regret setting (Appendix C), $\beta(b, \mu, K)$ isn't defined until Lemma 5.

Author response:
Thank you for bringing the typo and the presentation issue to our attention. In the revision, we have corrected the typo and explicitly clarified that $\beta(b, \mu, K)$ is defined in Lemma 5.

Reviewer comment (Theory):
It will be great if the authors can provide a proof sketch of Lemma 5 (instead of referring the readers to Korda et. al. (2013)) in the appendix for the sake of completeness.

Author response:
Thank you for the constructive suggestion. In response, we have incorporated a proof sketch for Lemma 5 in the revised version. It is highlighted in red within the appendix, focusing on the core ideas.

Reviewer comment (Theory):
I think the main weakness has to do with the abstention model that the authors consider. While both having an option that gives a fixed reward or incurs a fixed regret to the learner seem natural to me, being able to observe the sample from the arm that was chosen when the agent chooses to abstain feels a bit too strong. I think the authors need to elaborate a bit on this assumption they have made.

Author response:
We consider the assumption that the agent can observe the sample regardless of the abstention decision to be rather natural to our problem setting of bandits with abstentions and in particular to the clinical trials example presented in the introduction. In the example of clinical trials, if researchers implement costly prearranged measures, the treatment outcomes will not contribute to the optimization objective of the study. However, the researcher can still observe these outcomes, thereby leading to more efficient future experimental designs. Furthermore, conducting medical treatments becomes meaningless if the researcher is unable to observe the outcomes. In line with this model assumption, our model well aligns with that of Neu \& Zhivotovskiy (2020), which explored the role of abstention in the context of online prediction with expert advice.


Regarding the learnability of the abstention bandit model, if the agent cannot observe the sample from the chosen arm when opting for abstention, then it is equivalent to skipping the time step, rendering the model trivial. In other words, the learner gains no information from the time step but incurs an instantaneous regret.  Finally, we would like to remark that the above statement holds true only for the stochastic bandit model. The dynamics change in non-stationary or adversarial bandits, providing intriguing avenues for future research.

> Gergely Neu and Nikita Zhivotovskiy. Fast rates for online prediction with abstention. In Conference on Learning Theory, pp. 3030–3048. PMLR, 2020.

Reviewer comment (Theory):
It would be nice if the authors could give longer sketches of the proofs in the main body, even though I understand that there isn't enough space. One suggestion would be to move all the experiments to the appendix, since this is mostly a theoretical work and the algorithms are straightforward to implement I don't see what value the experiments add to the paper.

Author response:
We sincerely thank you for your valuable suggestion and understanding. Due to the density of our results, we, as with many other authors, struggled with space limitations, which prohibits us from incorporating comprehensive proof sketches in the main text. Our paper is positioned as a theoretical work, with numerical experiments serving to validate our theoretical findings. Opinions on the inclusion of experiments in the main text vary, and one reviewer suggests more numerical experiments. In light of this, we have chosen not to relocate all experiments to the appendix. According to the ICLR guidelines, the page limit for the main text is strict, and no additional pages can be added in the final version. Consequently, we are unable to include longer proof sketches in the main body. However, we are confident that the proofs presented in the appendix are lucid and easily understandable.

Reviewer comment (Presentation):
Since the fixed reward setting is easier to analyze it might make sense to change the order of the presentation between sections 3, 4.

Author response:
We express our gratitude to the reviewer for providing the constructive suggestion. While preparing the manuscript, we ensured clarity by presenting comprehensive explanations in the first setting and avoiding unnecessary repetitions in the second setting. In particular, in the corresponding proofs, we typically streamlined the common components. Although we appreciate the reviewer's kind suggestion, when preparing the initial manuscript, we have thought through whether to present the fixed-regret or fixed-reward settings first, aiming to ensure that the overall paper is as concise as possible.