Paper: Improved Algorithms for Replicable Bandits (ICLR-2024)

Reviewer comment (Presentation):
The title of this work starts with 'Improved algorithms'. I doubt whether it is an appropriate title. I feel that this work is more likely to be an extensive study on replicable bandits.

Author response:
We agree with this. In particular, Section 3 formalizes the way to guarantee the reproducibility that is implicitly (and inprecisely in terms of the multiplication count) used in (Esfandiari et al.'23). We revised the paper so that this contribution is clear. We have not changed the title, but we welcome your suggestion.

Reviewer comment (Novelty):
I understand it is a follow-up work on the topic of replicable bandits. However, if the aim is to handle situations where we have simply vague data, I think we can also apply differentially private stochastic bandits (https://proceedings.mlr.press/v180/hu22a/hu22a.pd). May you clarify the difference between application of these two settings? What are their individual strengths/weaknesses?

Author response:
Thank you for pointing out relevant work. DP considers the change of decision against the change of a single data point, whereas in our case, we have more than one change of data points between two datasets that are generated from the identical data-generating process. We added this to the revision of the related work section (Section 9, changes are colored red).

Reviewer comment (Theory):
The distribution-independent bound is only provided for the RSE algorithm. However, it is usually not so difficult to derive a distribution-independent bound for the other algorithms when the distribution-dependent ones already exist. I suggest the author(s) to provide distribution-independent bounds for other algorithms and a comparison or explain the analytical challenge.

Author response:
Thank you for your suggestion. Following your recommendation, we removed the claim to be "the first distribution-independent bound".

Reviewer comment (Theory):
As the lower bound for the standard setting is provided, may the author(s) discuss the gap between upper and lower bounds?

Author response:
Thank you for clarifcation. The gap is tight up to a logarithmic factor for $K=2$. We added this in Section 6 (colored red).

Reviewer comment (Theory):
I appreciate the efforts to study the linear setting, may the author(s) also provide a lower bound (or explain the difficulty)?

Author response:
Thank you for pointing it out. The difficulty here is not from the linear model but from extending the lower bound to $K>2$ arms. Note that even $K=2$, ours is the first lower bound in this problem.

Reviewer comment (Evaluation):
Section 1.2 claimed that 'we introduce the Replicable Successive Elimination (RSE) algorithm whose regret bound is the minimum of those of REC and the existing algorithms.' From Table 1, I cannot tell RSE is obviously better than REC.

Author response:
You are completely right, for most practical cases REC is better than RSE. If $\Delta_2$ is very small compared with $\Delta_3,...,\Delta_K$, then RSE would outperform, which is model 1. In model 1, RSE eliminated arms $3--5$ after $T=1000$ and thus outperforms REC, but the difference looks subtle due to the exploration before $T<1000$.

Reviewer comment (Writing):
Abstract: "Existing algorithms require a regret scale of $O(K^3)$, which increases much faster than the number of actions (or “arms”), denoted as K. We introduce an algorithm with a distribution-dependent regret of $O(K)$" -> both bounds are distribution-dependent, but are incomparable since the bound of Esfandiari et al. has better dependence on the distribution-dependent parameters than the one in the current paper. I think the way it is stated in the abstract is a bit confusing.

Author response:
> Incompativility of $O(K^3)$ and $O(K)$ bounds

You are correct. We denote this several times, but revised the paper to clarify that "when the suboptimality gaps for each arm are within a constant factor".

Reviewer comment (Novelty):
Abstract: "Our algorithms exhibit substantial simplicity compared to existing ones" -> I don't agree with the statement, I will elaborate shortly.

Author response:
> Section 5: Essentially, this algorithm is a best of both worlds combination of Algorithm 2 and the replicable arm elimination algorithm of Esfandiari et al. '23. It is easy to see that if the algorithm detects a large gap between the two best arms, it just eliminates all of them except for the best, otherwise it performs the replicable arm elimination process of Esfandiari et al. '23. I think this comparison should be mentioned in the text. Given this discussion, I don't see where the simplicity of this algorithm compared to Esfandiari et al. '23 lies.

We agree with the word "simplicity" is somewhat imprecise. We think Section 3 in our paper clarified what is somewhat implicitly used in [Esfandiari et al. '23], and we revised the paper so that our contributions are more clearly mentioned.

Reviewer comment (Novelty):
Section 3: I believe that the non-replication framework is inspired by the long line of work on batched bandits and bandits with low adaptation, so I think some citations are needed. Moreover, the same batching framework was essentially used in Esfandiari et al. '23 (even though it wasn't stated as such, its essence remains the same). I don't see the benefit of stating it as an abstract framework. I think the authors could move some of this discussion to the appendix and elaborate more on their proofs technique in the main body.

Author response:
> Section 3: I believe that the non-replication framework is inspired by the long line of work on batched bandits and bandits with low adaptation.

Thank you for the reference. We added batched bandit papers in the related work (Section 9).

Reviewer comment (Novelty):
Section 4: The explore-then-commit algorithm was mentioned in the warm-up section of Esfandiari et al. for the case where the suboptimality gap $\Delta$ is given to the designer and a sketch of the regret analysis was also provided. Essentially, Algorithm 2 of the current submission gets rid of the assumption that $\Delta$ is known by using the doubling-trick approach. I believe some discussion about it would be useful.

Author response:
> Section 4: The explore-then-commit algorithm was mentioned in the warm-up section of Esfandiari et al. for the case where the suboptimality gap is given to the designer and a sketch of the regret analysis was also provided. Essentially, Algorithm 2 of the current submission gets rid of the assumption that is known by using the doubling-trick approach. 

Thank you for the reference. We added this at the end of Section 4. Their algorithm is $O(T \sum_i \Delta_i)$, and our EtC clearly provides a better bound.

Reviewer comment (Novelty):
Section 5: Essentially, this algorithm is a best of both worlds combination of Algorithm 2 and the replicable arm elimination algorithm of Esfandiari et al. '23. It is easy to see that if the algorithm detects a large gap between the two best arms, it just eliminates all of them except for the best, otherwise it performs the replicable arm elimination process of Esfandiari et al. '23. I think this comparison should be mentioned in the text. Given this discussion, I don't see where the simplicity of this algorithm compared to Esfandiari et al. '23 lies.

Author response:
> Section 5: Essentially, this algorithm is a best of both worlds combination of Algorithm 2 and the replicable arm elimination algorithm of Esfandiari et al. '23. It is easy to see that if the algorithm detects a large gap between the two best arms, it just eliminates all of them except for the best, otherwise it performs the replicable arm elimination process of Esfandiari et al. '23. I think this comparison should be mentioned in the text. Given this discussion, I don't see where the simplicity of this algorithm compared to Esfandiari et al. '23 lies.

We agree with the word "simplicity" is somewhat imprecise. We think Section 3 in our paper clarified what is somewhat implicitly used in [Esfandiari et al. '23], and we revised the paper so that our contributions are more clearly mentioned.

Reviewer comment (Novelty):
Section 5: "Moreover, this algorithm is the first replicable algorithm that has a distribution-independent regret bound in the K-armed bandit problem." -> As I mentioned before, this is not correct. Since the distribution-independent bounds for linear bandits of Esfandiari et al. hold even when $K = d$ they already imply distribution-independent regret bound for the K-armed bandit problem.

Author response:
> We establish the first distribution-independent regret bound for the replicable $K$-armed bandit problem

We agree that and drop this statement.

Reviewer comment (Theory):
Conclusion: "This represents a significant advancement over existing algorithms." -> as I explained before, the bounds are not comparable.

Author response:
> Incompativility of $O(K^3)$ and $O(K)$ bounds

You are correct. We denote this several times, but revised the paper to clarify that "when the suboptimality gaps for each arm are within a constant factor".