Paper: Improving Accelerated Federated Learning with Compression and Importance Sampling (ICLR-2024)

Reviewer comment (Theory):
The theoretical analysis is primarily confined to strongly convex scenarios, which may not be applicable to intricate models such as deep neural networks (DNNs).

Author response:
We respectfully express our disagreement with the notion that Federated Learning is exclusively centered around Deep Neural Networks (DNNs). Contrary to this perspective, Federated Learning holds significant relevance and application in on-device Machine Learning environments, where each device is often constrained by limited computational resources. These constraints present a substantial challenge for smaller devices to effectively handle the computational demands associated with training and executing large-scale models such as DNNs.

Furthermore, it is essential to underscore the enduring significance of logistic regression, despite its straightforwardness. Logistic regression remains a cornerstone in practical applications, particularly within the domain of mobile devices. The appeal of logistic regression lies in its practicality and efficiency, making it a preferred choice, especially when grappling with the resource limitations characteristic of on-device Machine Learning settings.

In essence, the landscape of Federated Learning is characterized by a rich diversity of models and applications. It is imperative to recognize and appreciate the prevalence of simpler models like logistic regression, which play a crucial role in real-world scenarios, contributing to the versatility and adaptability of Federated Learning across various computational environments and constraints.

Furthermore, it is worth noting that convex problems, exemplified by logistic or linear regression, exhibit a valuable characteristic—interpretability. This attribute becomes particularly crucial in sectors like healthcare and finance, where understanding and interpreting the model's predictions are paramount. In these fields, Federated Learning is widely adopted as a tool for collaborative and privacy-preserving model training.

The interpretability of convex problems ensures that the results of the models can be easily understood and validated, a crucial aspect in applications where transparency and accountability are paramount. In healthcare, for instance, the ability to interpret model predictions is essential for gaining insights into patient outcomes or treatment effectiveness. Similarly, in finance, understanding the factors influencing a predictive model's outcomes is vital for informed decision-making.

As Federated Learning continues to make strides in various domains, the compatibility of convex problems with this collaborative learning paradigm adds a layer of transparency and interpretability, further solidifying its practicality and effectiveness in critical fields.

Reviewer comment (Theory):
In Algorithm 1&2 line 7, what if we can not optimize the problem in K iteration? How would the optimization error on Eq(4) or Eq(6) affect the final convergence? Has this been considered in Theorem 4.1 or 5.1?

Author response:
In the event that we fail to meet the Local Training assumption, it becomes challenging to assert that we will theoretically converge at the promised speeds. The potential arises for certain clients to address their local optimization problems more effectively than others, possibly employing distinct optimization strategies that result in varying convergence rates. Given that the Local Training Assumption is global, there exists the possibility of still satisfying it even if not every device can effectively solve its respective optimization problem.

Reviewer comment (Theory):
The assumption 2 seems not to be very clear. Can you explain how to achieve it in the algorithm?

Author response:
Assumption 2 establishes a category of sampling schemes that encompasses a broad range of methods commonly employed in practical applications. This classification is designed to be inclusive, covering most existing sampling schemes that are prevalent in real-world scenarios. As part of our analytical framework, we possess the ability to rigorously examine and verify whether a given sampling scheme adheres to the conditions outlined in Assumption 2 and, subsequently, determine its inclusion within this specified class of samplings. This approach facilitates a systematic evaluation of the suitability of various sampling techniques within the framework of our analysis.

Reviewer comment (Reproducibility):
The client sampling mechanism seems to be unclear. Is it informative? Which information is leveraged for the sampling?

Author response:
The client sampling mechanism delineated in our approach exhibits a high degree of generality, offering substantial flexibility in its application. This versatility is further underscored by the weighted AB assumption, which introduces a nuanced level of adaptability to accommodate various scenarios and requirements. The weighted AB assumption allows for the incorporation of different weights, providing a framework that can be tailored to specific contexts, thereby enhancing the adaptability and applicability of the mechanism.

In order to employ Importance Sampling effectively, it is imperative to possess information about the individual smoothness constants denoted as $L_i$. These constants play a crucial role in guiding the sampling mechanism, allowing for a targeted and informed approach to variance reduction during the optimization process. The incorporation of individual smoothness constants enhances the precision and adaptability of Importance Sampling, contributing to more efficient and effective optimization strategies tailored to the specific characteristics of each component in the system.

Reviewer comment (Theory):
Could the results be extended to non-convex cases? I would suggest to both include analysis and experiment on non-convex learning problems.

Author response:
Our ambition is to broaden the scope of our acceleration methods to encompass non-convex scenarios. However, achieving this goal requires a fundamentally different analysis and approach, which is presently absent from both our knowledge and the existing literature. While our proofs successfully demonstrate convergence concerning the distance to the optimum—a distinctive feature in cases like ours—it is imperative to recognize that such a claim is not universally applicable.

In the context of non-convex optimization, the intricacies of the landscape necessitate a novel perspective and analytical framework that currently eludes us. The proofs we provide, grounded in the specifics of convex optimization, cannot be seamlessly extended to the non-convex case due to the inherent disparities in optimization dynamics. As we strive to advance our understanding and methodologies, addressing the challenges posed by non-convex scenarios remains an avenue for future exploration and research.

Reviewer comment (Theory):
The work focuses only on strong convex and smooth cases. How does it perform (theoretically) in convex / non-convex cases?

Author response:
In accordance with the current trend observed in recent academic papers, which places a strong emphasis on accelerating training through local methods, our analysis heavily leans on the assumption of strong convexity in the underlying functions. We are actively involved in ongoing efforts to relax these assumptions, acknowledging the importance of extending the applicability of the proposed methods.

It is noteworthy, however, that in our specific case, the theoretical framework does not offer guarantees for convergence in non-convex scenarios. Despite the collective efforts to broaden the scope of these techniques, the challenges associated with ensuring convergence in non-convex optimization settings remain an active area of exploration and a subject of ongoing research within the broader scientific community.

We are steadfast in our belief that the comprehensive analysis of non-convex objectives is a topic deserving of its own dedicated paper. The complexities and nuances associated with non-convex optimization merit in-depth exploration, requiring a focused examination that can thoroughly delve into the unique challenges and insights specific to this realm. As such, we envision a separate publication that delves into the intricacies of analyzing and optimizing non-convex objectives, contributing valuable perspectives and advancements to the broader scientific discourse.

Reviewer comment (Experiments):
It will be interesting to see how well the proposed algorithms perform empirically.

Author response:
The paper provides an empirical comparison, albeit presented in a concise manner. The overarching goal was to demonstrate the efficacy of Importance Sampling (IS) by showcasing its ability to yield substantial improvements over non-IS methods. This emphasis on IS underscores its significance in the context of the research.

A key observation is that Local Training (LT) methods, especially those without support for Client Sampling (CS) or Importance Sampling (IS), exhibit a comparatively diminished level of competitiveness. This observation is critical, as it has the potential to influence the overall fairness of the comparison presented in the paper. The inclusion of this insight aims to provide a nuanced understanding of the performance dynamics among various methodologies, acknowledging the impact of IS support in enhancing the overall effectiveness of Local Training methods.

Reviewer comment (Experiments):
From an optimization perspective, assuming the logistic regression as an experimental setting is ok, but this is a machine learning venue; it has been a norm to consider more difficult objectives to test the hypotheses. It is a weakness not to consider a setting similar to what most of the FL algorithms are tested in.

Author response:
Our stance is grounded in the conviction that a paper characterized by this level of theoretical depth, particularly emphasizing strongly convex functions, does not require extensive experimentation to demonstrate the theorems. The mathematical rigor underpinning our proofs establishes the validity of the results with a probability of 1, rendering exhaustive experiments less imperative for showcasing these established principles. The robustness of our theoretical framework, substantiated by rigorous mathematical analysis, is deemed sufficient to convey the validity and reliability of our theorems without a heavy reliance on exhaustive experimental validation.

Furthermore, it is crucial to emphasize that logistic regression, despite its simplicity, continues to be widely adopted in practical applications, particularly in the realm of mobile devices. The practicality and efficiency of logistic regression position it as a preferred choice, especially when dealing with resource constraints common in on-device Machine Learning environments. Consequently, it is imperative to acknowledge the varied landscape of models and applications within the Federated Learning paradigm, recognizing the prevalent utilization of simpler models like logistic regression in real-world situations.

Reviewer comment (Theory):
Similarly to the experimental case, providing theory in the convex case, given that FL is mainly applied in nonconvex settings with neural networks, could be improved. While the reviewer appreciates that there is a continuation of works (from specific research groups) that aim to cover every possible problem setting (as summarized in Table 1 of the paper), the current work (theoretically and practically) cannot be readily appreciated (and put among other works) on nonconvex neural networks FL setting.

Author response:
Aligning with the prevalent trend in recent academic literature, which underscores the acceleration of training through local methods, our analysis predominantly relies on the assumption of strong convexity in the underlying functions. We actively participate in ongoing initiatives to ease these assumptions, recognizing the significance of expanding the applicability of our proposed methods.

However, it is crucial to highlight that, in our specific case, the theoretical framework does not furnish assurances for convergence in non-convex scenarios. Despite collaborative endeavors to broaden the scope of these techniques, the challenges associated with ensuring convergence in non-convex optimization settings persist as an active area of exploration and ongoing research within the broader scientific community.

Our unwavering belief lies in the notion that a comprehensive analysis of non-convex objectives warrants its own dedicated paper. The intricacies and subtleties inherent in non-convex optimization demand thorough investigation, necessitating a focused examination that can delve deeply into the unique challenges and insights specific to this domain. Consequently, we envision a distinct publication that delves into the complexities of analyzing and optimizing non-convex objectives, contributing valuable perspectives and advancements to the broader scientific discourse.

Reviewer comment (Novelty):
I might have missed it, but the difference between this work (along with the accompanying difficulties in completing the algorithm and proof) and the 5GCS is not clear from the text. Table 1 claims that that work did not satisfy the CC column. However, how difficult was it to complete the CC column on top of the work of 5GCS? What were the challenges? What was the amount of additional difficulty in completing this work? Was it incremental or substantial?

Author response:
We appreciate your attention to the nuances of our paper. The notable distinction lies in the introduction of two additional algorithms, namely 5GCS-AB and 5GCS-CC, addressing key limitations observed in 5GCS, particularly pertaining to sampling and compression.

In the realm of sampling, the initial approach in 5GCS involved uniform sampling, which, as you rightly pointed out, can be suboptimal in situations where functions exhibit significant variations with diverse L-smoothness and mu-convexity constants. The innovation in 5GCS-AB involves a new sampling methodology that allows for the calculation of optimal step sizes and sampling probabilities, especially when Importance Sampling (IS) is employed.

The aspect of compression is another pivotal consideration. Here, we introduce an additional layer of compression on top of client sampling in 5GCS-CC method. This addition is designed to address the practical challenge where sending full vectors may prove suboptimal in real-world scenarios.

In gauging the level of difficulty associated with this task, I would assign it a rating of 9 out of 10. This rating conveys that the endeavor is exceptionally challenging and demands a high degree of complexity and skill. The intricacies involved in this undertaking make it a formidable challenge, requiring a comprehensive understanding, advanced expertise, and adept problem-solving capabilities.

Reviewer comment (Theory):
The paper does not explain why assumptions 2,3,4 should hold in practice and under which conditions. They are hard to digest and read like proof, enabling assumptions.

Author response:
Assumptions 2 and 4 play integral roles in our analysis. Assumption 4 is notably one of the least restrictive assumptions when it comes to compression. It provides a flexible framework that accommodates a wide range of compression techniques. On the other hand, Assumption 2, while less standard, introduces a valuable dimension to our analysis. This assumption allows for the examination of more intricate sampling schemes, enabling us to benefit from their inherent complexity.

As highlighted in the following work:

Tyurin, A., Sun, L., Burlachenko, K., & Richtárik, P. (2022). Sharper rates and flexible framework for nonconvex SGD with client and data sampling. arXiv preprint arXiv:2206.02275

it has been demonstrated that a majority of sampling methods satisfy Assumption 2. This finding reinforces the practical applicability of our analytical framework. The incorporation of Assumption 2 into our proofs is a significant and deliberate choice. It enriches our analysis by considering more complex sampling strategies, contributing to a more nuanced understanding of the dynamics involved.

In summary, Assumptions 2 and 4, each serving distinct purposes, collectively enhance the robustness and versatility of our analytical framework. Assumption 2, in particular, stands as a crucial addition, providing a pathway to analyze and leverage the complexities inherent in various sampling schemes.