From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper focuses on the problem of compressing the uplink model updates in a Federated Learning (FL) setup implementing secure aggregation protocols. It first  argues about the incompatibility of standard compression methods (scalar quantization, pruning, and product quantization) with secure aggregation protocols, and then it proposes a method to this incompatibility based on using common/shared information across all clients.  A "Secure Indexing" (SecInd) protocol has also been proposed for product quantization, which leverages the linearity of the computations. Experimental results using the LEAF data/platform are also provided. 

The reviewers had a number of concerns, some of which were addressed, but several major ones have remained. In particular, the reviewers still felt that the novelty of the work and the proposed methods should better be demonstrated. They also felt that more thorough investigation is needed to verify the effectiveness of the proposed methods especially in the non-iid setting (while they acknowledge that the results of figure 2 are obtained in the non-iid setting). Please see the (updated) reviews for more details. 

Overall, once the above concerns are addressed, the revised version of the paper could be a great addition to the literature of FL. 
justification_for_why_not_higher_score: the majority of the reviewers are in favor of rejecting the paper. 
justification_for_why_not_lower_score: -

From: ICLR.cc/2023/Conference/Paper3074/Reviewer_wijR
title: Thanks for the repsonse
comment: Thanks for the response. I will keep my initial rating after carefully reading the response.

From: ICLR.cc/2023/Conference/Paper3074/Authors
title: Answer
comment: We thank the reviewer for their suggestions and address the three mentioned weaknesses below:
1. Global sharing strategy / novelty: in line with our remark to Reviewer r6Pi, we argue that the technical modifications we propose are quite delicate in that they must balance security guarantees, preserve accuracy, and require minimal changes to a standard federated learning stack to have a shot at being implemented. Hence, we view their seeming simplicity as a virtue. With scalar quantization, the overflow challenge is summarized in the point above. With pruning, the challenge was to have the same pruning mask across all clients to allow complete compatibility with the Secure Aggregation protocol. They may be viewed as baseline options, where we include them in the paper for the sake of completeness and to offer comparison points with the proposed SecureIndexing when compressing with product quantization. Regarding SecureIndexing, we argue that this novel approach is needed to implement Product Quantization (a state-of-the-art compression method) securely, and this offers a high accuracy/compression tradeoff with the same level of security.
2. Data variance: we agree with this remark, that's why we experimented on all our experiments with non-iidness: we use the standard non-iid splits from LEAF (and will make this claim clearer in the paper). Hence, all results and in particular Figure 2 are in the non-iid setup (for instance, each client is a particular person in CelebA and has varying number of images).
3. Compatibility with DP: in line with our remark to Reviewer wijR, we argue it's possible to add Differential Privacy on top of Secure Aggregation (needed to realize the full promise of FL as privacy-enhancing technology). Indeed, and while this is not the scope of the paper, DP noise can be added natively by the TEE with our modified random pruning or scalar quantization approaches. However, for PQ, it would require to offload the aggregation from the server to the TEE.

Regarding other comments:
1. We will quantify "meaningful loss" in the abstract as suggested.
2. We'll mention the breakdown of the various bottlenecks in the FL pipeline in the introduction and refer the reader to Bonawitz et al and Huba et al for insightful discussions.
3. The linearity is meant for the compression operator. Therefore, our approach is agnostic to the type of network. Indeed, we explore ResNet and LSTMs in our experiments. 
4. That's a valid remark. To answer your points (A) while it is the case that overflow for scalar quantization could be avoided by padding the fixed precision representation with more bits, this defeats the point of quantization: quantization is applied in the federated setting to reduce the space and communication costs of the clients (we can assume that the server is not subject to such constraints). A typical scenario is to reduce the representation of each coordinate to 8 bits. So adding many bits to prevent overflow erases the benefit.  Hence, we study empirically how economical we can be with the representation to ensure security while trading off some accuracy. Indeed, since SecAgg computes the sum *modulo* $2^p$, an overflow destroys (or worse, inverts) the signal anywhere it happens. Thus, to guarantee that overflow does not happen, we would need to pad $p$ to account for all clients participating in a round. For instance, if updates are quantized to a single byte and the number of clients is 100, then $p$ needs to be increased in theory by 7, which nearly doubles upload bandwidth. (B) yes indeed, quantization parameters are not compatible with aggregation by default, hence the propsed sharing method. 
5. Regarding global hyperparameter sharing, this is a valid point. While this is possible to leak information from hyperparameter sharing in theory, it's commonly admitted that the quantity of information recovered in practice would be extremely small (see [1, 2, 3]).
6. We indeed did not include DP for clarity in Figure 1. In the standard user-level DP setup, noise is added by the TEE to the sum of the masks $\sum_i m_i$. Hence, the clients do not add DP noise (contrary to the sample-level DP setup). For reference about user-level DP, see [4]. Experiments in our paper do not include DP as we consider outside the scope of the paper and focus on secure compression. Discussion on DP can be found in Section 6.
7. We plan to open-source the code.

[1] Large language models can be strong differentially private learners, Li et al., 2021
[2] Private selection from private candidates, Liu and Talwal, 2021.
[3] Large-scale differentially private bert, Anil et al., 2021
[4] Learning Differentially Private Recurrent Language Models, McMahan et al, 2017.

From: ICLR.cc/2023/Conference/Paper3074/Authors
title: Answer
comment: We thank the reviewer for their comments and address them below.
- Compression masks: this is a very good point. As underlined in Section 4.2.2 (paragraph "Privacy, Security and Bandwidth), the pruning masks are *random*, hence no client information leaks from them.
- Uplink compression: we agree that our mechanism is designed for uplink compression only. As stated in the introduction (4th paragraph), this is the most critical part of the FL communication setup since upload bandwidth is typically more limited than download bandwidth.
- Figure 1: we'll add a more detailed legend to allow easier understanding, thanks for letting us know.

From: ICLR.cc/2023/Conference/Paper3074/Authors
title: Answer
comment: We thank the reviewer for their comments and suggestions and address them below:
- Novelty and challenges: The technical modifications we propose are quite delicate in that they must balance security guarantees, preserve accuracy, and require minimal changes to a standard federated learning stack to have a shot at being implemented. Hence, we view their seeming simplicity as a virtue. With scalar quantization, the overflow challenge is summarized in the point above. With pruning, the challenge was to have the same pruning mask across all clients to allow complete compatibility with the Secure Aggregation protocol. They may be viewed as baseline options, where we include them in the paper for the sake of completeness and to offer comparison points with the proposed SecureIndexing when compressing with product quantization. Regarding SecureIndexing, we argue that this novel approach is needed to implement Product Quantization (a state-of-the-art compression method) securely, and this offers a high accuracy/compression tradeoff with the same level of security.
- Non-linearity: we totally agree with the reviewer's remark and indeed implement this linearity with one-hot vectors as described in Section 4.2.3. We'll rephrase the paper accordingly (here, we meant non-linearity in the sense that it's not possible to *directly* sum assignments). 
- Regarding non-iiddness, we use the standard non-iid splits from LEAF (and will make this claim clearer in the paper). Hence, all results and in particular Figure 2 are in the non-iid setup (for instance, each client is a particular person in CelebA and has varying number of images).

From: ICLR.cc/2023/Conference/Paper3074/Authors
title: Answer
comment: We thank the reviewer for their positive feedback on the paper. Regarding privacy analysis, as discussed in Section 6, we argue it's possible to add Differential Privacy on top of Secure Aggregation (needed to realize the full promise of FL as privacy-enhancing technology). Indeed, and while this is not the scope of the paper,  DP noise can be added natively by the TEE with our modified random pruning or scalar quantization approaches. However, for PQ, it would require to offload the aggregation from the server to the TEE.

From: ICLR.cc/2023/Conference/Paper3074/Reviewer_wijR
confidence: 1: You are unable to assess this paper and have alerted the ACs to seek an opinion from different reviewers.
summary_of_the_paper: This paper tackles the communication bottleneck in the scalability of federated learning. A linearity constraint on the decompression operator is introduced, together with some popular scalar quantizations to to reconcile security and communication efficiency. The proposed algorithm reports the best results on LEAF benchmarks. 
strength_and_weaknesses: Strength: This paper is well written. The proposed model to reconcile security and communication efficiency is interesting and efficient. 

Weakness:The proposed model may suffer the applicability to privacy models such differential privacy.

clarity,_quality,_novelty_and_reproducibility: The paper is of good quality and clarity. The idea is novel as far as I know.
summary_of_the_review: Overall, I think that the studied model in this paper is well-motivated. The proposed algorithm is sound and evaluated with better performance compared to previous related works. Therefore, I recommend “weak accept”.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3074/Reviewer_r6Pi
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper addresses the problem of compressing the client-to-server model updates in a Federated Learning (FL) setup implementing secure aggregation (SecAgg) protocols. 

First, the paper discusses the incompatibility of standard compression methods (scalar quantization,  pruning, and product quantization) with secure aggregation protocols. 

Next, a solution to address this incompatibility has been proposed, which primarily relies on using common/shared information across all clients. This common/shared information includes scalar quantization parameters (scale $s$, zero point $z$, bit-width $b$ and SecAgg bit-width $p$ with $p > b$), pruning masks, and product quantization parameters (block size $d$, codebook size $k$, and codebook C). 

A "Secure Indexing" (SecInd) protocol has also been proposed for product quantization, which leverages the linearity of the computations (since the eventual goal is to obtain sum of the weight matrices, one can easily compute this by counting the number of occurrences of a code at a specific location and multiplying it with the value of the code).   

Experiments show that product quantization with SecInd achieves the highest compression rate at iso-accuracy. 
strength_and_weaknesses: Strengths: 
1.	The motivation is clear and well-defined.
2.	The paper is well-organized and easy to follow.

Weaknesses: 

1.  There is very little technical novelty in the proposed work. As discussed in the summary, the core idea is to select common parameters for the compression methods across all the clients. This appears to be an obvious and naive solution. The proposed SecInd method is also fairly obvious, involving only re-arranging some computations. 

The claim that "operation to take assignments to produce an aggregated update is not linear" is not a valid one. For example, suppose that we want to select the j-th element of a vector v. This can be made linear by computing the dot product between v and an indicator vector i, where the vector i has zero everywhere except the j-th element, which is set to 1. This is exactly the idea used in this work - since an outer summation is also involved, one can sum up the indicator vectors to obtain an "histogram" before computing the dot product.   

2. The whole concept of using common parameters across all clients is unlikely to work well in the non-iid setting (statistical heterogeneity). No effort has been made to investigate the effectiveness of the proposed compression methods in the non-iid scenario.  
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written. There are no concerns about the reproducibility. The main limitation is the lack of any novel ideas.
summary_of_the_review: A well-written paper that address a valid problem, but the proposed solutions are obvious, straightforward, and lack novelty.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3074/Reviewer_Tmdx
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper provides an efficient approach on how to integrate scalar quantization, product quantization and random pruning into existing secure aggregation algorithms. The paper makes extensive experiments to show that the proposed method can reach a good balance between communication efficiency and accuracy.

strength_and_weaknesses: Strengthen:
1. The paper proposes a novel mechanism for the application of compression techniques to secure aggregation, and such compression is bi-directional.
2. Based on the experiment, the method can reach a high accuracy with a high compression rate.
​

Weakness:
1. Compression masks also contain certain information about the clients, I am not sure whether adding m_i only in the compressed domain will affect the security. The authors didn’t emphasize it. 
2. It’s only designed for uplink compressions, which in my opinion, can only reduce up to half of the overall communication bits
3. To ensure that this framework could go through, the authors need to have the same compression mask (e.g., pruning mask) in each round. It might work in the iid setting but in the non-iid setting, such limitation may lead to a huge downgrade in performance.  

​

clarity,_quality,_novelty_and_reproducibility: 1. Fig.1 is a bit hard to follow now. It’s better to give an illustration on the uplink and downlink communications
2. It’s not clearly stated why adding the mask will not affect communication efficiency. The authors should emphasize that.

summary_of_the_review: The paper gives an efficient approach on how to integrate scalar quantization, product quantization and random pruning into existing secure aggregation algorithms, and the experiment on the proposed experiment is extensive. However, it is only for uplink compression and the non-iid setting is not considered. 

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3074/Reviewer_dmTz
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: 1. The authors target the communication bottleneck and scalability of multi-client federated learning (FL). They formalize and tackle the client-to-server model updates compression during training under Secure Aggregation (SECAGG) primitive, which is a core component of Federated Learning pipelines that allows the server to aggregate the client updates without accessing them individually.

2. The key techniques they adopt to make SECAGG efficient and scalable include: scalar quantization, pruning, and Secure Indexing (SECIND, a variant of SECAGG to enable product quantization). 

3. They claim to achieve a 40x compression ratio in uplink communication on LEAF benchmarks without meaningful loss.

strength_and_weaknesses: strength

1. It aggressively saves the uplink communication bandwidth and the authors claim that there is no loss in security.


2. The key approach is to modify compression techniques to share hyperparameters globally across all clients so that aggregation can be done by uniformly combining responses from all clients. This seems to be portable to existing FL works without much implementation efforts.

weakness

1. The global sharing strategy, which is their key idea, seems to be quite straightforward and lacks novelty.
2. I guess the performance drop introduced by shared mask (pruning) and shared quantization parameters (quantization) is depending on the data variance across clients. I suppose this could potentially limit the usability of the work.
3. Compatibility with DP as they stated in section 6.

clarity,_quality,_novelty_and_reproducibility: 1. It would be great to be specific about what “meaningful loss” means in the abstract.
2. I am not an expert in this field, and from an non-expert point of view, it would be good to show the breakdown for FL workloads, demonstrating the uplink communication is indeed the system bottleneck.
3. The demonstration is based on a linear layer, and is this method adoptable on convolutional layers and other types of layers? If so, what is the difference in terms of accuracy, privacy and compression ratio?
4. Is aggregation overflow the only reason why scalar quantization is not compatible with SECAGG? If so, why must the server use the same bit width as the clients? Can the server use more bits to avoid overflow? I suppose another reason is the different quantization parameters are not compatible for aggregation, but this is not mentioned in the “Challenge” part of 3.2.1.
5. Is the global hyperparameter sharing strategy for compression a possible source for information leakage? If not, what’s the price for adopting your solution besides more download time (given the no-free-lunch principle)?
6. In Figure.1, DP is omitted for clarity. Just want to be clear, is g_i already calculated by data after adding the DP noise? Is the experimental results in Figure 2 considering DP?
7. Plan for open source (ignore if I missed the link in the paper) ?

summary_of_the_review: Overall I enjoyed reading this paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold