From: auai.org/UAI/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept (Poster)
comment: Meta Review: The paper proposes a secure aggregation scheme to prevent private data leakage in Federated Learning systems.

Quality: The technical discussions are sound. Experiment evaluation is weak. 

Clarity: The paper is well written with some minor concerns on clarity. 

Originality:  A novel application in Federated Learning.  

Significance: The work is useful in Federated Learning. 

A common concern is that the experiments have been conducted in comparison to one baseline and using one set of parameters. The authors have updated the experiments in their replies.

From: auai.org/UAI/2022/Conference/Paper405/Reviewer_L13t
Q1 Summary and contributions: This work proposes a novel secure aggregation scheme for horizontal FL. It has a computational complexity of O(M) for each client and allows D_max=N/3 arbitrary dropouts or T_col=N/3 colluding clients.

Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 2: Fair: The paper contributes some new ideas.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q2(5) Reproducibility: 2: Fair: Key resources (e.g., proofs, code, data) are unavailable but key details (e.g., proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
Q2(6) Clarity of writing: 2: Fair: The paper is somewhat clear, but some important details are missing or unclear.
Q3 Main strengths: 1. Solid theoretical results which justify the efficiency and effectiveness of the proposed method SASH.
2. Experiments show SASH has significant improvement over SecAgg in terms of efficiency while maintaining very similar utility.
Q4 Main weakness: 1. Only one baseline is compared in the experiments although several are listed in Table 1.
2. Only one set of parameters of SHPRG is considered in the experiments.
Q5 Detailed comments to the authors: 1. eq.(6) is out of bound.
2. Table 1 does not fit the template, too.
3. Fonts in Fig. 4 are too small.
4. It would be better to show the confidence interval/standard deviation of the results in Fig. 3 and 4.
Q6 Overall score: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Q7 Justification for your score: 1. Theoretical results look solid.
2. Experimental results are satisfactory but it needs further comparison with different baselines and different settings.
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper405/Reviewer_sVWk
Q1 Summary and contributions: This paper studies the problem of secure aggregation in federated learning. The main contribution of the current paper is a more efficient method.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 3: Good: The paper is likely to have high impact within a subfield of AI OR moderate impact across more than one subfield of AI.
Q2(3) Correctness/Technical quality: 3: Good: The paper appears to be technically sound, but I have not carefully checked the details.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q2(5) Reproducibility: 2: Fair: Key resources (e.g., proofs, code, data) are unavailable but key details (e.g., proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
Q2(6) Clarity of writing: 3: Good: The paper is well organized but the presentation could be improved.
Q3 Main strengths: A more efficient secure aggregation method for federated learning has been established.
Q4 Main weakness: The experiments lack several baselines.
Q5 Detailed comments to the authors: The problem considered in this paper is very important, and the proposed more efficient method looks very promising. However, I have one question: Why do authors not compare with those baselines in Table 1?
Q6 Overall score: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Q7 Justification for your score: The proposed method seems to be much more efficient than existing method.
Q8 Confidence in your score: 2: Not very confident. I am able to defend my evaluation of some aspects of the paper, but it is quite likely that I missed or did not understand some key details, or can't be sure about the novelty of the work.
Q9 Complying with reviewing instructions: 1: Yes.

From: auai.org/UAI/2022/Conference/Paper405/Reviewer_XJC6
Q1 Summary and contributions: in this paper, the authors propose an aggregation scheme for federated learning, named SASH to prevent private training data leakage in Federated Learning systems.
Q2 Assessment of the paper: More detailed information regarding each of these aspects is given below:
Q2(1) Originality/Novelty: 3: Good: The paper makes non-trivial advances over the current state-of-the-art.
Q2(2) Significance/Impact: 2: Fair: The paper is likely to have moderate impact within a subfield of AI.
Q2(3) Correctness/Technical quality: 2: Fair: The paper has minor, easily fixable, technical flaws that do not impact the validity of the main results.
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q2(5) Reproducibility: 2: Fair: Key resources (e.g., proofs, code, data) are unavailable but key details (e.g., proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
Q2(6) Clarity of writing: 2: Fair: The paper is somewhat clear, but some important details are missing or unclear.
Q3 Main strengths: This paper presents an aggregation scheme based on SHPRGs and according to the authors, the proposed scheme (1) achieves better asymptotic computation costs than previous solutions, (2) provides adaptive security against the aggregator, (3) is robust to worst-case dropouts and simple to implement in a standard Internet environment for non-TTP assumptions and (4) the trained model can obtain the same accuracy as plain training cases.
Q4 Main weakness: Lack of statistical analysis and sharing a Git repository with the code to easily replicate the experiments.
Q5 Detailed comments to the authors: Fig. 1 is somehow confusing and it is not properly explained in thee body of the manuscript.
Table 1, exceeds the limits of the page.
In Fig, 3,4 could use subcaptions.
Q6 Overall score: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
Q7 Justification for your score: Authors selected SecAgg as the baseline and SASH aims to overcome the efficiency bottleneck, without sacrificing other advantages.
Q8 Confidence in your score: 3: Somewhat confident, but there's a chance I missed some aspects. I did not carefully check some of the details, e.g., novelty, proof of a theorem, experimental design, or statistical validity of conclusions.
Q9 Complying with reviewing instructions: 1: Yes.