Paper: Communication-Efficient Algorithm for Asynchronous Multi-Agent Bandits (ICLR-2024)

Reviewer comment (Writing):
Algorithm 1: the reviewer does not understand what lines 21-24 mean.

Author response:
**Answer:** Lines 21-22 are the receiving counterpart for the message sent in Line 16 (by other agents $m'$),
      while Lines 23-24 are the receiving counterpart for the message sent in Line 15. Take the pair between Line 16 and Lines 21-22 as an example. Line 16 means that, when sending message from agent $m$ to agent $m'$, the agent $m$ sends the token $\texttt{tk}^{(m\to m')}$ to agent $m'$ as well.
      Lines 21-22 means that, when receiving a token $\texttt{tk}^{(m'\to m)}$ that was used to send a message from agent $m'$ to agent $m$, agent $m$ keeps this token. This pair constitutes the token receiving and sending protocol. We will add this explanation to the final version of this paper.

Reviewer comment (Experiments):
Number of agents: in the experiments, the number of agents is lower than the number of arms.

Author response:
**Answer:** In Figure 7 in Appendix D of the revised manuscript, we report additional experiments where the number of agents is $M=30$ greater than the number of arms $K=16$. In these experiments, our algorithm outperforms other baseline algorithms in communication and enjoys similar regret as the other baselines, which is the same observation as in our original numerical simulations in Figure 1.

Reviewer comment (Theory):
Proof of Lemma 1: equation (a1), M_t is not defined and misleading.

Author response:
**Answer:** In inequality (a1), the upper bound of summation is $M\times t$, not $M_t$, where $M$ is the number of agents and $t$ is the current time index. In the revised, we updated it to $M\cdot t$ to avoid misunderstanding.

Reviewer comment (Theory):
Model: the authors assume that there is only one best arm (\mu_1 > \mu_2 \geq …). This is a strong assumption that limits the scope of this study.

Author response:
**Answer:** Assuming a single optimal arm is for the ease of presentation, which was widely assumed in literature (e.g., see Even-Dar et al. (2006, Section 2.1)). In the case of multiple optimal arms,
      tailoring our current algorithm could also work.
      In the original algorithm design, when only one arm remains in the candidate arm set (the optimal arm), the algorithm stops communication.
      When it comes to the case of multiple optimal arms, the above communication stop condition is invalid.
      Instead, one can set an exploration threshold, and when the number of times pulling the remaining arms in the candidate arm set exceeds this threshold, stop the communication. This threshold would guarantee that the total communications are still constant, as well as that the reward means of remaining arms are close enough to have a sublinear regret upper bound.

---
Even-Dar, Eyal, et al. "Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems." Journal of Machine Learning Research 7.6 (2006).

Reviewer comment (Theory):
Although the order of the regret bound of the proposed algorithm (see Remark 3) is the same as the optimal bound, the constant term before the order can be quite large. For example, when setting $\alpha=6$ as used in the experiments, the constant can be as large as 392, which is a very large number. I suggest the authors provide the regret bound (not only the order) of alternative methods and compare them. It will also be better to plot the regret bounds and the communication complexity bounds of the proposed method and alternative methods.

Author response:
**Answer:** The leading term coefficients of $\texttt{SE-ODC}$ and SE-IBC in our experiment setting are $24$, which is better than the $392$ factor in our $\texttt{SE-AAC-ODC}$ algorithm. For one thing, we note that the larger constant coefficient of our regret upper bound comes from a more involved regret analysis for our constant communication result. This implies the challenge and novelty of our new algorithm.
      For another thing, in our experiments, e.g., Figure 1(b), our algorithm with only constant communication has a better regret performance than that of $\texttt{SE-ODC}$ with logarithmic communication. That implies although there is a larger artificial factor in our regret upper bound result, the empirical regret performance of our algorithm is better (even with lower communications).

Reviewer comment (Evaluation):
Can the authors clarify what is indeed the most important measurement (time or communication) in the MA2B task? The authors explained that synchronous updates will lead to redundant communication (Section 3.1). This is reasonable, but what is the final purpose of saving communication? Is it to achieve a lower regret within a shorter time horizon? If time is the most important thing, then by Figure 1,2, the performance of the proposed method is not as good as UCB-IBC.

Author response:
**Answer:** We acknowledge that this paper primarily delves into theoretical aspects, with the central focus on assessing the efficacy of communication cost reduction while maintaining near-optimal $O(K\log T)$ regret, referred to as "time" in the reviewer's question. It is essential to clarify that the primary goal is not to minimize regret further, as prior works, including our own, have already achieved near-optimal regret levels. While empirical results indicate that the proposed algorithm's regret may not match that of UCB-IBC, it is noteworthy that both UCB-IBC and our algorithm attain the same $O(K\log T)$ regret level.

Reviewer comment (Experiments):
I can see that from Theorem 2, a smaller alpha yields smaller regret. I can also see that from Figure 1,2, the proposed method has a higher regret compared to UCB-IBC. Therefore, I'm interested in the question of when we use smaller $\alpha$ to achieve similar regret as UCB-IBC, will the number of communications of the proposed method still be much smaller?

Author response:
**Answer:** We note that our algorithm only needs constant communications, which are far less than the linear communication cost of UCB-ICB; it is impossible for our algorithm to beat UCB-ICB in terms of regret metric.
      For another thing, there is indeed a trade-off of regret and communication in our algorithm ($\texttt{SE-AAC-ODC}$) in terms of the parameter $\alpha$. In Appendix D of the revised manuscript, we compare the performance of $\texttt{SE-AAC-ODC}$ with $\alpha=3$ and $\alpha=6$ in Figure 6. This simulation validates the trade-off---the case of $\alpha=3$ enjoys a better regret but suffers a higher communication. Additionally, the communications of both cases are much lower than that of other baseline algorithms.

Reviewer comment (Theory):
I'm quite interested in the property of constant communication. By Theorem 2, the authors show that the regret bound is closely related to T, but the number of communications is independent of T. This means that communication between agents is not so important and even if there are only a few communications, smaller $R(T)/T$ can still be achieved by using a large T. Can the authors intuitively explain the philosophy behind this property?

Author response:
**Answer:** The high-level intuition is that an agent only needs to know which arm is optimal to achieve sublinear regret in multi-armed bandits, and the communication cost of sharing this information (an arm index) is independent of the total number of decision rounds.
      Therefore, if one fast agent (with a large number of active decision rounds) could somehow "smartly" find the optimal arm, then it only needs to pay a constant number of communications to disseminate the optimal arm index to all other agents.

Reviewer comment (Novelty):
**Lack of originality:** The algorithm and the analysis are extensions of known works in (Yang et. al. 2023) and (Chen et. al. 2023).

Author response:
**Answer:** We acknowledge the foundational role of Yang et al. (2023) and Chen et al. (2023) in our algorithmic design. However, our analysis significantly extends their work, addressing challenges unique to our setting.
      On the one hand, the analysis of Chen et al. (2023) cannot be used in our case, as we are aiming for a constant communication upper bound, which is intrinsically different from their time-dependent bounds.
      On the other hand, the analysis tool in Yang et al. (2023) relies heavily on the synchronous multi-agent setting (agents always have the same number of samples), which is invalid in our asynchronous setting. In this paper, we develop a new analysis approach that relies on the ''fastest'' agent (with the highest number of active decision rounds, see Lemma 1).
      We refer the reviewer to Section 1.2 (related works) for a more comprehensive discussion about how this work differs from (Yang et al. 2023) and (Chen et al. 2023).

Reviewer comment (Reproducibility):
**Missing details in numerical simulations:** The algorithm AAE-ODC from (Chen et. al. 2023) uses a buffer threshold for messages communicated among the agents. There is no mention of the values of the buffer threshold used in AAE-ODC for the experiments presented in the paper.

Author response:
**Answer:** For the experiments in the main paper, we set the buffer of $\texttt{SE-ODC}$ ($\texttt{AAE-ODC}$) as constant $1$ (that is, immediately communicate when there is a demand without any buffer). In Figure 5 of Appendix D, we report the $\texttt{SE-ODC}$ with no buffer (green), and $\texttt{SE-ODC-D}$ with doubling buffer (brown) and compare both to our algorithm, $\texttt{SE-AAC-ODC}$ (red).
      Although $\texttt{SE-ODC-D}$ with doubling buffer enjoys better communication performance than $\texttt{SE-ODC}$ with constant buffer, $\texttt{SE-ODC-D}$'s communication performance is still worse than that of $\texttt{SE-AAC-ODC}$ proposed in our paper, and $\texttt{SE-ODC-D}$'s empirical regret is far worse than $\texttt{SE-AAC-ODC}$, which is because $\texttt{SE-ODC-D}$'s regret upper bound $O(KM\log T)$ is intrinsically worse than our algorithm's $O(K\log T)$ regret upper bound.
      We updated these additional simulation results in the revised version of this paper.

Reviewer comment (Theory):
The communication cost scaling as \(O(KM^2 \Delta^{-2}\log T)\) of the AAE-ODC algorithm in (Chen et. al. 2023) only holds when the buffer thresholds are constant with respect to the number of communications (ref: Corollary 1, part (b) in (Chen et. al. 2023)).

Author response:
**Answer:** It is unfair to compare the result in Corollary 2(b) (Chen et al., 2023) with our algorithm.
      This is because the result in Corollary 2(b) (Chen et al., 2023) only holds when the buffer thresholds are exponential  (e.g., doubling in $\texttt{SE-ODC-D}$ in the above simulation).
      But if the buffer thresholds are exponential, the algorithm in Chen et al. (2023) suffers a $O(KM\log T)$ regret which is away from the near-optimal regret by a factor of the number of agents $M$ (see their regret upper bound's second term in Theorem 1 (b)).
      In fact, this $O(KM\log T)$ regret upper bound can be achieved by all agents individually running UCB *without any communication*.
      Hence, it is unfair to compare the result in Corollary 2(b) (Chen et al., 2023) with our algorithm that always achieves the near-optimal $O(K\log T)$ regret.

Reviewer comment (Theory):
If the buffer thresholds are exponential with respect to the number of communications, the communication cost of the AAE-ODC algorithm in (Chen et. al. 2023) scales as \(O\big(M^2 \log \frac{K \log T}{\Delta^2}\big)\) (ref: Corollary 2, part (b) in (Chen et. al. 2023)).

Author response:
We thank the reviewer for their timely response and appreciate their participation in the discussion. Below, we show that, if the buffer increases in an exponential manner (i.e., the case in Corollary 2(b) of Chen et al. (2023)), in general, one would have $G_i^j= \frac{8\alpha\log T}{\Delta_i^2}$, and, therefore, the algorithm in Chen et al. (2023) would suffer a suboptimal $O(KM\log T)$ regret upper bound. 

Recall that
$$G\_i^j=\min\left\\{\sum\_{j'\in\mathcal{A}\setminus\{j\}}f(c\_{\tau\_{i}}\^{j'\to j}), \frac{8\alpha\log T}{\Delta\_i^2}\right\\}$$
By the definition of $\tau_i$ in Chen et al. (2023), we know that, in the worse case, $$c_{\tau_{i}^{j'\to j}}\le \frac{2\alpha\log T}{\Delta_i^2}.$$
When the buffer size is exponential, i.e., $f(c)=a^{c-1}$ where $a$ is the exponent (e.g. if it is doubling, $a=2$). Then, we have $$f(c_{\tau_{i}}^{j'\to j})\le T a^{\frac{2\alpha}{\Delta_{i}^{2}\log_{a}e}},$$
where $e$ is the nature constant and the base for $\log T$. Therefore, we have 
$$G\_i^j=\min\left\\{a\^{\frac{2\alpha}{\Delta\_{i}^{2}\log\_{a}e}}\cdot\sum\_{j'\in\mathcal{A}\setminus\{j\}}T, \quad\frac{8\alpha\log T}{\Delta_i^2}\right\\}.$$
That means, when the total number of decision rounds $T$ is large enough, we have $$G_i^j= \frac{8\alpha\log T}{\Delta_i^2}.$$

Reviewer comment (Theory):
The authors do not provide a lower bound for the number of communication rounds required, which makes it difficult to assess the performance of the proposed algorithm.

Author response:
Providing a non-trivial lower bound for communication in asynchronous multi-agent MAB settings is highly challenging and remains an open problem in the literature.
      This is because some extremely asynchronous scenarios could make communication unnecessary. For example, consider the case that one agent is active in all time slots, while all other agents are only active in one single time slot.
      In this case, there is no need to conduct communication among these agents, that is, one can achieve near-optimal regret by all agents individually running UCB without any communication.

On the other hand, the only related communication lower bound results that we are aware of is in Wang et al. (2020, Section 3.4), where they show that to achieve a $o(M\sqrt{KT})$ regret upper bound in *synchronous* multi-agent MAB, the expected communication is at least $\Omega(M)$.
      If we exclude the extreme cases and only consider mildly asynchronous scenarios, then the $\Omega(M)$ lower bound
      could be used to understand the tightness of our communication upper bound.
      That implies that
      our communication upper bound $O(KM\log(\Delta^{-1}))$ is tight in terms of the number of agents $M$.
      
In a word, proving a non-trivial communication lower bound is challenging and beyond the scope of this work. Comparing with known results suggests that our communication upper bound is tight in terms of $M$.



---
- Yuanhao Wang, Jiachen Hu, Xiaoyu Chen, and Liwei Wang. Distributed bandit learning: Nearoptimal regret with efficient communication. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020b.