Paper: Communication Bounds for the Distributed Experts Problem (ICLR-2024)

Reviewer comment (Presentation):
The problem considered in this paper should be better motivated with convincing application examples. Admittedly, the expert problem has achieved great success with broadly employed in many practical application scenarios, but regarding the distributed setting, it is hard to map it to any specific application scenario. To make the contribution clear, examples and discussions on that should be provided in the introduction part.

Author response:
We have highlighted the motivating examples in our introduction and describe them here. For distributed online learning problems (e.g., online federated learning), you can try different sets of hyperparameters that correspond to different experts in the distributed experts problem. The goal is to adaptively choose the optimal hyperparameters on each day.

Reviewer comment (Theory):
Both the communication protocols mentioned in this paper rely on a coordinator. It is unclear why this is a reasonable assumption. It should be clarified whether changing the model to a fully distributed one would make any difference on the results.

Author response:
A central coordinator is commonly used in such scenarios, which receives information from users along edges, and updates their model locally. If it is fully decentralized and has point-to-point communication, our communication upper bounds will still hold up to a multiplicative constant factor of 2 (and additive log(# servers)), since the coordinator can forward a message from one server to any other server. Otherwise, our communication bound depends on the specific communication topology, which is not the focus of our paper.

Reviewer comment (Theory):
All the experts are forced to commit to the same expert according to the depicted problem setting. Reasons for making such an assumption should be provided.

Author response:
In addition, for distributed online learning or federated learning, it is common for the coordinator to commit to the same model due to fairness and privacy issues as well as the memory overhead of storing multiple models for each downstream server.

Reviewer comment (Presentation):
The presentation of this work is poor and should be substantially revised.

Author response:
We thank the reviewer for pointing out the deficiencies in our presentation. We have updated the writing accordingly in the revised version of our paper.

Reviewer comment (Experiments):
The simulation results are too simple, lack support from real-world data traces, and do not include comparisons to other related works.

Author response:
We have evaluated our methods against real-world data trace (HPO-B) in section 6 and provided simulation evaluations in the appendix to demonstrate that our upper bound methods are robust even against extreme cost distributions (i.e., very sparsely distributed), which do not have corresponding real-world data traces.

Reviewer comment (Writing):
Overall, the writing of this paper needs substantial effort to be ready for publication. The current format does not provide enough details and insights into algorithms and theoretical and numerical results. For example, in the current presentation of the introduction, there is no direct mapping between the theory results summarized in Tables 1 and 2, and those descriptive statements of the paragraphs on top of page 3 of the paper.

Author response:
We thank the reviewer for pointing out the deficiencies in our presentation. In our revised version, we have unified the presentation of our communication bounds in the tables, as well as in our contributions section at the top of page 3.

Reviewer comment (Writing):
Earlier in the introduction, the authors try to distinguish their work with the streaming algorithms claiming that in their setting, the coordinator does not have any memory constraints. However, later in the introduction, they talk about lower bounds in the case of limited memory.

Author response:
For our memory bound condition in the proof of our lower bound, we clarify that we only impose a memory bound on the downstream servers with no memory assumption on the central coordinator. We believe this assumption is both practical and provides a novel solution for bounding the communication of a server with itself across multiple days. A lower bound proof without a memory bound being imposed on downstream servers is indeed more challenging, which we leave as a very intriguing open question.

Reviewer comment (Theory):
I am not sure how much making an assumption of $T=O(\log(ns))$ makes sense since, typically, in an online learning setting, the time horizon is assumed to be sufficiently large such that the sublinear regret in $T$ makes sense.

Author response:
The $T \in O(\log{(ns)})$ assumption is only required for our lower bound argument in the message-passing model with maximum aggregation function. The point here is to show that the best we can do in the message-passing model with a maximum aggregation function is the naive EWA, which is the reason why we can only propose an upper bound in the blackboard model when the maximum aggregation function is used.

Reviewer comment (Experiments):
The numerical evaluation is very brief and does not provide any comparison between the proposed communication-efficient algorithms and other possible algorithms that try to be aware of the communication costs. Also, the benchmark is not introduced and even cited in the paper. The experimental setup is not clearly defined, and the impact of parameters is not investigated.

Author response:
As far as we know, our generic distributed experts setup has not been studied before. [1] only studies the setting when the cost is distributed to a single server rather than arbitrarily aggregated across multiple servers. Also, the memory-efficient streaming algorithms in [2, 3] assume a memory bound on the coordinator, which we don’t have. Thus, given our generic setup, we can think of two baselines: EWA, which is known to achieve optimal regret, and EXP3, which uses less communication and can achieve near-optimal regret. Regarding HPO-B, we have cited the paper in the third paragraph of the introduction. For further clarification, we also cite this benchmark in our evaluation section as well in the revised version of our paper. We have further added more description of our experimental setup and an ablation study on the hyperparameter $b_e$ in the appendix of our revised paper. This demonstrates the communication-regret tradeoff of our methods.  

References:  

1. Kanade, Varun, Zhenming Liu, and Bozidar Radunovic. "Distributed non-stochastic experts." Advances in Neural Information Processing Systems 25 (2012).  
2. Srinivas, Vaidehi, et al. "Memory bounds for the experts problem." Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing. 2022.  
3. Peng, Binghui, and Fred Zhang. "Online prediction in sub-linear space." Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA). Society for Industrial and Applied Mathematics, 2023.