Paper: Adaptive Regret for Bandits Made Possible: Two Queries Suffice (ICLR-2024)

Reviewer comment (Theory):
The authors should discuss the tradeoff of $O(\log T)$ terms between regret and number of queries. In this problem, the order of $O(\log T)$ is not trivial: the number of base algorithms is only $O(\log T)$. Based on the current results, the cost of reducing the number of queries by $O(\log T)$ times is an increase in regret by $O(\log T)$ times (possibly more, see the Questions section). This tradeoff should be well specified in the paper.

Author response:
**Trade-off between regret and query**: We believe such trade-off is in fact non-trivial; however we will add a brief discussion. For example, as shown in Daniely et al 15, there is an $\tilde{O}(\sqrt{T})$ regret upper bound with $O(\log T)$ queries, and an $\Omega(T^{1-\epsilon})$ regret lower bound with one query. Here an $O(\log T)$ save in query actually costs an $\Omega(T^{\frac{1}{2}-\epsilon})$ term in regret, and it's non-trivial to show with two queries the cost in regret becomes logarithmic. In addition, a $\log T$ to $2$ improvement in the number of queries needed is significant for computational efficiency, for applications such as hyperparameter optimization in expensive settings.

Reviewer comment (Theory):
In the proof of Theorem 1, the author uses the inequality $\log(1+x) \ge x - x^2$ for $x \ge -1/2$ (page 15). Is there any proof to show $\eta_k \widetilde r_t(k) \ge -1/2$ for every $t$ and $k$? Intuitively, $\widetilde r_t(k)$ could be of order $v(t,k)_i/P(t)_i = O(\log T)$, which implies that $\eta_k$ should be scaled $O(1/\log T)$. This may lead to another $O(\log T)$ term in the regret.

Author response:
**Lower bound on $\tilde{r}_t(k)$**: thank you for pointing it out! This issue can be easily resolved, and we describe two ways below and will use the second method to clarify our paper. 

The straightforward way is by multiplying an $\frac{1}{2\log T}$ term to the choice of $\eta\_k$ in line 6 of Algorithm 1, at the cost of an additional $2\log T$ term in regret, which is negligible in our context. Let's lower bound $\tilde{r}\_t(k)$: by the definition of $\hat{\ell}\_t$ (line 11), it's a sparse vector where only the coordinate $x'\_t$ can have a positive value $\frac{1}{P(t)\_{x'\_t}}$. However, $P(t)\_{x'\_t}$ is lower bounded by $\frac{v(t,k)\_{x'\_t}}{2\log T}$ by its definition (line 9), thus $\tilde{r}\_t(k)\ge -2\log T$. Now, making $\eta\_k\le \frac{1}{4\log T}$ ensures the desired property that $\eta\_k \tilde{r}\_t(k)\ge-\frac{1}{2}$.

The better way is by hedging over only the range $k\in [2+2\log \log T, \log T]$ instead of $[0,\log T]$ in the algorithm. To see this, by the definition of $\eta_k$, this issue doesn't exist when $2^k \ge 4\log T$, and the only problematic case is when we consider very small intervals with length $2^k< 4\log T$. However, for small intervals with length $O(\log T)$, our regret bound holds trivially and automatically, therefore this method will address the issue with no cost at regret.

Reviewer comment (Theory):
The main weakness (and hence the low score) is due to a confusion I have regarding the paper. In the middle of page 15 in the supplementary material, the authors state that the weights are positive to get the inequality $\tilde W_t \ge \tilde w_t(k)$. Is the positivity proved somewhere in the paper? If not, can you provide the arguments for positivity of the weights? I think this is a really interesting paper and I would be happy to recommend acceptance if the authors can clear this confusion.

Author response:
**Non-negativity of $\tilde{w}\_t(k)$**: thank you for pointing it out! This issue can be easily resolved, and we describe two ways below and will use the second method to clarify our paper. 

The straightforward way is by multiplying an $\frac{1}{2\log T}$ term to the choice of $\eta\_k$ in line 6 of Algorithm 1, at the cost of an additional $2\log T$ term in regret, which is negligible in our context. To see this, notice the non-negativity of $\tilde{w}\_t(k)$ is guarded by the non-negativity of $1+\eta\_k \tilde{r}\_t(k)$. Let's lower bound $\tilde{r}\_t(k)$: by the definition of $\hat{\ell}\_t$ (line 11), it's a sparse vector where only the coordinate $x'\_t$ can have a positive value $\frac{1}{P(t)\_{x'\_t}}$. However, $P(t)\_{x'\_t}$ is lower bounded by $\frac{v(t,k)\_{x'\_t}}{2\log T}$ by its definition (line 9), thus $\tilde{r}\_t(k)\ge -2\log T$. Now, making $\eta\_k\le \frac{1}{4\log T}$ ensures the desired non-negativity.

The better way is by hedging over only the range $k\in [2+2\log \log T, \log T]$ instead of $[0,\log T]$ in the algorithm. To see this, by the definition of $\eta_k$, this issue doesn't exist when $2^k \ge 4\log T$, and the only problematic case is when we consider very small intervals with length $2^k< 4\log T$. However, for small intervals with length $O(\log T)$, our regret bound holds trivially and automatically, therefore this method will address the issue with no cost at regret.

Reviewer comment (Theory):
In section 1.1, the authors say that their algorithm is run over black-box base bandit learners. Then in the description of Algorithm 1, they take the base learners to be EXP3, essentially not viewing the base learners as a black-box. This seems self-contradictory.

Author response:
**Black-box experts**: by black-box we mean the expert can be any online optimization algorithm as long as it has the guarantee of Lemma 2, and EXP-3 is one of them. We will clarify this point.

Reviewer comment (Writing):
Line 13, Algorithm 1, unclear what $2^k | t+1$ means. I assumed that these are the inactive base learners.

Author response:
**Meaning of $2^k|t+1$**: here it means when $k$ satisfies $t+1$ is divisible by $2^k$, we restart the $kth$ expert by re-initializing its weight to be $\eta_k$.

Reviewer comment (Writing):
Line 3 of Algorithm 2, $w_{t}$ must be $x_{t}$.

Author response:
**$w_t$ must be $x_t$**: we will fix this typo.

Reviewer comment (Experiments):
It would be interesting to see the practical tradeoff in the reward curve vs. computational efficiency, for say StABL and Full-EFLH from (Lu and Hazan 2023).

Author response:
**Weak experiments**: We will add experiments with higher time horizon and repeats, as well as looking at the practical tradeoff in the reward curve vs. computational/sample efficiency. Note that the theory there for optimal rates are non-trivial but in practice, we can use the naive lower-variance estimator.

Reviewer comment (Theory):
In page 3, paragraph 6, I believe that the "special case when $\mathcal{K}$ is a simplex" is a bit stronger than MAB (since it would effectively consider all convex combinations of arms).

Author response:
**Question 1**: if we consider expected regret in MAB, it's equivalent to regret in simplex. We will clarify this point more carefully.

Reviewer comment (Writing):
In the description/pseudocode of Algorithm I, calling Algorithm 2 as a subroutine seems slightly ambiguous, since Algorithm 2 is written as a full learning loop (whereas the actual operation seems to be an update $v(t+1,k) \leftarrow v(t,k)$ for each $k$).

Author response:
**Question 2**: we will make it clear the calling of Algorithm 2 only concerns the weight updating.

Reviewer comment (Writing):
In page 6, end of section 3, "can be extended to any interval .... by Cauchy-Schwarz" seems very opaque. I would suggest at least mentioning that each arbitrary interval can be decomposed into a disjoint union of $O(\log T)$ geometric intervals as in (Daniely, 2015).

Author response:
**Question 3**: we will add more details as you suggested.

Reviewer comment (Reproducibility):
Including the code as supplementary material/link in the final version would enhance the value of having the experiments.

Author response:
**Question 4**: Thanks for the suggestion! The code is now already implemented as part of an open source package. We will add a link to that OSS package for the camera-ready version, to avoid anonymity issues.