Paper: Best Arm Identification for Stochastic Rising Bandits (ICLR-2024)

Reviewer comment (Theory):
While Assumption 2.1 appears intuitive, Assumption 2.2 falls short of being satisfactory. Even though the authors present some theoretical findings solely under Assumption 2.1, their interpretability is somewhat lacking.

Author response:
Assumption 2.1 implies that the expected values are *non-dereasing and concave*. Assumption 2.2 provides a more **explicit characterizarion** of the increment function $\gamma_i(n)$ by enforcing an upper bound of the form $c n^{-\beta}$ depending on the parameters $(c,\beta)$. Although our results hold under Assumption 2.1 only (Theorems 4.1 and 5.1), the forms of the error probability $e_T$ and of the minimum time budget $T$ become more interpretable under Assumption 2.2. Indeed, intuitively, **the BAI problem is easier when the expected reward function reaches the stationary behavior faster** (see [[IMAGE](https://drive.google.com/file/d/1vcthAAJtBTKybq1GULnNdUTk5EAo9kvj/view?usp=sharing)]). This "speed" (a.k.a. rate) is governed mainly by $\beta$. The larger is $\beta$, the faster is the convergence rate. This is visible in Corollary 4.2 and 5.2 where the minimum requested time budget $T$ as well as the error probability decrease as $\beta$ increase. We will add this comment in the final version.

Reviewer comment (Novelty):
The proposed algorithm closely resembles UCB-E and SR for standard multi-armed bandits, with the primary distinction lying in the estimators. I'm not suggesting this is unacceptable, but it does somewhat diminish the novelty of this work. A promising future direction would involve integrating both estimators into a unified algorithm.

Author response:
While we agree that our algorithms follow the basic scheme of the ones of (Audibert et al., 2010), we point out that our algorithms follow **well-established principles**: *optimism in the face of uncertainty* (UCB-E) and *arm elimination* (SR) which are the basic building blocks of a large majority of works in bandits published in top-tier venues [1,2,3]. Thus, we do not think this diminishes the novelty of the work. Furthermore, we stress that our main contributions are: ($i$) the challenging **theoretical analysis** that the rested rising nature of the arms requires for both the estimators and the algorithms and ($ii$) the conception of the novel **lower bound** which shed lights on the complexities of the setting.

Reviewer comment (Theory):
Since the expected rewards are bounded in $[0,1]$ and non-decreasing, they must converge to some value. Thus, it is not surprising that the error probability lower bound will be matched by R-SR for large $T$. For non-stationary BAI, the algorithm SR is minimal optimal up to constant factors.

Author response:
While we agree that the expected rewards must converge to some value, it is not guaranteed that they will all converge to **different values**. Suppose that the second-optimal arm converges for $T \rightarrow +\infty$ to the same expected reward as the optimal arm. This SRB instance will be very challenging for large values of $T$. This is why we need a characterization of the conditions (Eq. 11 or 13) under which R-SR succeeds in identifying the optimal arm. Furthermore, we remark that the application of standard SR is not guaranteed to yield the same guarantees, since it uses as estimator the **standard sample mean** that does not discard too old samples and might fail to deliver an estimator with a bias that diminishes at the number of samples increases.

Reviewer comment (Theory):
Theorem 6.1: In any case, the algorithm can make a random guess. Therefore, it is not appropriate to state that $e_T(\boldsymbol{\nu}, \mathfrak{A})=1$.

Author response:
We agree with the reviewer that such an expression is not appropriate. We will remove it in the final version.

Reviewer comment (Writing):
Minor issue in Section 2: \citet should be used in "As in (Metelli et al., 2022)".

Author response:
Thank you, we fixed it.

Reviewer comment (Novelty):
My major concern is the insufficient problem motivation. In the introduction, the example introduces the SRB is ``the arm improve performances over time'', but the problem setup of SRB is arms whose performances increase with pulls. I personally feel it is the example of adversarial MAB or non-stationary MAB rather than SRB. The experiments still do not give the real-world applications. In fact it's hard for me to figure out real-world scenario (with the neccessary to model as a SRB) that solves practical problems.

Author response:
In the main paper, we proposed a relevant *motivating practical example*, the **combined algorithm selection and hyperparameter optimization (CASH)**, **a real-world scenario** arising from the AutoML research field that **naturally models as a SRB problem** (see Introduction). Moreover, we devoted Appendix C to discuss **two additional** motivating examples. We refer the Reviewer to these sections (Introduction and Appendix C) for a more detailed discussion. 
We agree that the sentence "the arm improve performances over time" is misleading since it may suggest that the arms evolve *regardless* they are pulled. We will replace it in the final version of the paper with "the arms improve their performances as it is pulled" to avoid confusion.

In the experimental campaign, we test our methods with **real-world data** by them to perform a **Best Model Selection** of a set of different machine learning algorithms trained to address the classification task on the IMDB dataset. Such an experiment is reported in Appendix I.3 due to space constraints.

Reviewer comment (Writing):
The problem statement is a little bit unclear. Specifically, it is mentioned that SRB is a special case of SRB, but it is never explained what the word ``rested'' means.

Author response:
We think the Reviewer with the sentence  "it is mentioned that SRB is a special case of SRB" means "it is mentioned that SRB is a special case of **rested bandit**". We remark that **Section 1 (Introduction) does explain the word "rested"** precisely at the beginning of the second paragraph, which we report here for the Reviewer's convenience: "This work focuses on the Stochastic Rising Bandits (SRB), a specific instance of the rested bandit setting (Tekin & Liu, 2012) in which the expected reward of an arm increases whenever it is pulled".

Reviewer comment (Experiments):
There should be some discussions about the difficulties of applying existing algorithms (or some trivial variants) to solve SRB. For example, it is only mentioned in the experiments that non-stationary MAB algorithms and adversarial MAB algorithms is outperformed, but it is essential to verify that the increasing structure of SRB is crucial both theoretically and empirically.

Author response:
We considered several baselines in our numerical validation, including *UCB-E* (Audibert et al., 2010) and *Successive Rejects* (Audibert et al., 2010), the most famous solutions for fixed-budget BAI in stationary MABs. These solutions are based on **estimators that do not discard old data**, as they are developed for stationary setting, **preventing from obtaining estimator of $\mu_i(T)$ whose bias decreases** as the number of pulls increases. This prevents the error probability to decrease with $T$. We considered also *Prob-1* (Abbasi-Yadkori et al., 2018) as a baseline from the adversarial setting. However, we point out that in our case the rewards *stochastic and subgaussian* and not *bounded and adversarial*. Since *Prob-1* assumes bounded rewards, its analysis does not apply to our setting.