Paper: FedLPA: Personalized One-shot Federated Learning with Layer-Wise Posterior Aggregation (ICLR-2024)

Reviewer comment (Presentation):
One of the major weaknesses is the way the methodology is presented. In Section 3 various subsections are sequentially presented; but the subsections are not connected well, it is hard to follow the sequence of methodology through text. The algorithm should be put into main text, and well connected to the sections. I would suggest rewriting the algorithm in more details and possibly adding a figure as an overview.

Author response:
Thanks for your suggestion. We will improve our logic of Section 3 and in the camera-ready version, we will rewrite the algorithm in more details and add a figure as an overview. Our logic of the subsections in Section 3 are as follows:

In Section 3.1, we want to show that a global objective function could be obtained via the local posterior.

In Section 3.2, we discuss how to get the local posterior and we convert this problem into approximate Hessian. The most efficient method to get the Hession is using the empirical Fisher information matrix. 

In Section 3.3, we show how to compute the Block-Diagonal empirical Fisher information matrices, which is better than the empirical Fisher information matrix to approximate Hessian. 

In Section 3.4, with the Block-Diagonal empirical Fisher information matrices from local models, we show how to aggregate these matrices. 

In Section 3.5, as the aggregation is unacceptable, we propose the efficient and practical method to do the train the parameters of the global model.

We will add the logical sentences between subsections and more explanations in the corresponding subsections in the camera-ready version.

Reviewer comment (Novelty):
The authors call their method 'personalized'. In personalized FL the inference is made through individual models of the clients. But, as far as I understand, the output of your methodology is a single global model that would hopefully work for every client; although you obtain local posteriors they are just intermediators for the global model training. If this is the case, this is not a personalized method but a heterogeneous FL method. Otherwise, please clarify.

Author response:
Thanks for the comments on the aspect of the personalization. We understand that the personalized federated learning setting may contain the multiple aspects. In this paper, we mainly consider that each client has a different biased dataset, which is one of the personalized federated learning settings. We agree that our claim of the personalization is somehow weak. We will weaken the claim of the personalization in the camera-ready version.

Reviewer comment (Experiments):
I think the current experiments are not enough to show the efficacy of the method. In particular, majority of the experiments are done with 20 clients, which is small for the FL. Also, there is no comparison to local only training.

Author response:
In Section 4.3, we show that the experimental results of varying number of clients on FMNIST dataset. We conduct the experiments with 50 clients, which is also the maximal number of clients shown in other federated one-shot learning papers[1][2]. In our G.10, we show that the computing resources we need to execute the experiments of this paper. Due to the resource limit, we also choose 50 as our maximal number.

For the comparison to local only training, we have shown them in our paper in Fig. 1 in Section 3.6 and the corresponding details in Appendix G.2.

[1]Su, S., Li, B., & Xue, X. (2023). One-shot Federated Learning without server-side training. Neural Networks, 164, 203-215.

[2]Heinbaugh, Clare Elizabeth, Emilio Luz-Ricca, and Huajie Shao. "Data-Free One-Shot Federated Learning Under Very High Statistical Heterogeneity." The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Experiments):
More experiments with higher number of clients should be added to text if possible. Also the performance change w.r.t number of clients and data samples should be reported.

Author response:
Thanks for the suggestion. We have added the experiments with our method on the same experiment setting in the paper with three random seeds with 10 clients. We conducted experiments on FMNIST datasets with beta=0.1, 0.3 and 0.5. The performance change w.r.t number of data samples are shown as follows:

| **Accuracy (Beta=0.1)** | **Data sample proportions**   | 
|---------------|------------------------|
| 55.33±0.06       |100%| 
| 53.88±1.14       | 80%|
| 53.15±0.82       | 60%|
| 53.20±0.21       | 40%|
| 45.71±0.13       | 20%|

| **Accuracy (Beta=0.3)**     | **Data sample proportions**   | 
|---------------|------------------------|
| 68.20±0.04       |100%| 
| 65.47±0.02       | 80%|
| 64.80±0.71       | 60%|
| 64.10±0.40       | 40%|
| 62.15±0.03       | 20%|

| **Accuracy (Beta=0.5)**     | **Data sample proportions**   | 
|---------------|------------------------|
| 73.33±0.06       |100%| 
| 73.17±0.05       | 80%|
| 72.40±0.29       | 60%|
| 70.02±0.17       | 40%|
| 68.54±2.02       | 20%|

With the above tables, we could see that our method FedLPA could yield satisfactory results even with only 20% data samples under multiple settings. Due to the rebuttal time limit, we have not added the synthetically generated data experiments. We believe that our proposed FedLPA works well on the synthetically generated dataset since it already performs well on the realistic datasets.

Reviewer comment (Experiments):
Lack of experiments on synthetically generated data is not desirable for such a statistically motivated method.

Author response:
Thanks for the suggestion. We have added the experiments with our method on the same experiment setting in the paper with three random seeds with 10 clients. We conducted experiments on FMNIST datasets with beta=0.1, 0.3 and 0.5. The performance change w.r.t number of data samples are shown as follows:

| **Accuracy (Beta=0.1)** | **Data sample proportions**   | 
|---------------|------------------------|
| 55.33±0.06       |100%| 
| 53.88±1.14       | 80%|
| 53.15±0.82       | 60%|
| 53.20±0.21       | 40%|
| 45.71±0.13       | 20%|

| **Accuracy (Beta=0.3)**     | **Data sample proportions**   | 
|---------------|------------------------|
| 68.20±0.04       |100%| 
| 65.47±0.02       | 80%|
| 64.80±0.71       | 60%|
| 64.10±0.40       | 40%|
| 62.15±0.03       | 20%|

| **Accuracy (Beta=0.5)**     | **Data sample proportions**   | 
|---------------|------------------------|
| 73.33±0.06       |100%| 
| 73.17±0.05       | 80%|
| 72.40±0.29       | 60%|
| 70.02±0.17       | 40%|
| 68.54±2.02       | 20%|

With the above tables, we could see that our method FedLPA could yield satisfactory results even with only 20% data samples under multiple settings. Due to the rebuttal time limit, we have not added the synthetically generated data experiments. We believe that our proposed FedLPA works well on the synthetically generated dataset since it already performs well on the realistic datasets.

Reviewer comment (Theory):
In section 3.5 is there a specific reason to introduced the l2 squared loss other than available optimization tools?

Author response:
We choose the  l2 squared loss because it is the most common convex function used in the machine learning field and we have also proved the converge rate of FedLPA in l2 squared loss.  We agree that other convex loss functions will not affect the implementation of FedLPA.  We leave the impact of other loss function as future work.

Reviewer comment (Experiments):
The proposed method is composed of multiple approximations: 1) empirical fisher to approximate the Hessian 2) block-diagonal Fisher matrix instead of full, 3) approximating global model parameter $\bar{M}$ with optimization problem in Equation 14. However, there is no ablation to understand how each approximation step impacts the final results.

Author response:
Q:The proposed method is composed of multiple approximations: 1) empirical fisher to approximate the Hessian 2) block-diagonal Fisher matrix instead of full, 3) approximating global model parameter $\bar{M}$ with optimization problem in Equation 14. However, there is no ablation to understand how each approximation step impacts the final results.

A: 

(1) Empirical Fisher to approximate the Hessian:

Although empirical Fisher has been successfully applied in many methods and yielded good results, discussions concerning the approximation error of empirical Fisher are limited. Fortunately, previous work [1] provides a detailed critical discussion of the empirical Fisher approximation.

i. Fisher to approximate the Hessian:

When the loss function represents an exponential family distribution, the Fisher is a well-justified approximation of the Hessian, and its approximation error can be bounded in terms of residuals. The accuracy of this approximation improves as the residuals diminish and is exact when the data is perfectly fitted.

ii. Empirical Fisher to Fisher:

It's noted that the Fisher and empirical Fisher coincide near minima of the loss function under two conditions:

  A. The model distribution closely approximates the data distribution.

  B. A sufficiently large number of samples allow both the Fisher and empirical Fisher to converge to their respective average values in the population.

In practical environments, especially condition 1, might not hold, causing bias between empirical Fisher and Fisher. However, empirical Fisher still contains effective covariance information. In second-order optimization methods, the covariance information in empirical Fisher can adapt to the gradient noise in stochastic optimization. Nevertheless, referencing [3], we can use the model’s predictive distribution to obtain an unbiased estimate of the true Fisher at the same computational cost as empirical Fisher.

(2) Block-diagonal Fisher matrix to approximate the full one:

Paper [2] provides a detailed evaluation and testing of using block-diagonal Fisher to approximate the full one.
Firstly, Chapter 6.3.1 "Interpretations of this approximation" in the paper indicates that using a block-wise Kronecker-factored Fisher closely approximates the full Fisher. Although there is a bias term (due to the approximation in our Appendix Equation 27), this term approximates zero when there are sufficient samples. Furthermore, the paper examines the approximation quality of block-diagonal Fisher compared with the true Fisher and suggests that block-diagonal Fisher captures the main correlations, while the remaining correlations have a minimal impact on the experimental results.

(3)Besides, we have added some experiments for more ablation studies with our method on the same experiment setting in the paper with three random seeds with 10 clients. We conducted experiments on FMNIST datasets with beta=0.1, 0.3 and 0.5. The results are shown as follows:


|**Number of iteration**| **Accuracy (Beta=0.1)** | **Accuracy (Beta=0.3)**  | **Accuracy (Beta=0.5)**  | 
|---------------|---------------|------------------------|------------------------|
|1000  | 52.81±0.71       |60.31±0.23|72.11±0.57|
|5000  | 59.70±0.32       |68.09±0.30|74.27±0.12|
|10000| 55.33±0.06       |68.20±0.04|73.33±0.06|
|20000| 58.41±0.05       |68.11±0.07|73.51±0.02|

With the experiment results, we could know that 5000 iterations is enough to get the ideal results.

In Appendix F, we also show the linear convergence rate for Eq. 14.

Additionally, it's worth noting that concerning Laplace approximation, the analysis in [4] suggests that the error of Laplace approximation is inversely proportional to the input dimension 'n' with $O(n^{-1})$. According to this conclusion, it can be inferred that in our method, for each layer of the neural network, the error of Laplace approximation is inversely proportional to its width. When the neural network is infinitely wide, the approximation error tends towards zero.

We will add the above explanations into Appendix in the camera-ready version.

[1]Kunstner F, Hennig P, Balles L. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems.

[2]Martens, James, Jimmy Ba, and Matt Johnson. "Kronecker-factored curvature approximations for recurrent neural networks." International Conference on Learning Representations. 2018.

[3]Martens J. Second-order optimization for neural networks. University of Toronto, 2016.

[4]Bilodeau, Blair, Yanbo Tang, and Alex Stringer. "On the tightness of the Laplace approximation for statistical inference." Statistics & Probability Letters 2023.

Reviewer comment (Reproducibility):
FedLPA requires transmitting individual (instead of aggregation) $A_k, B_k, M_k$ in order to solve the optimization problem in Equation 14. Exposing individual statistics to the server can have privacy concerns and cannot be compatible with standard secure aggregation protocol or central differential privacy methods.

Author response:
Q:FedLPA requires transmitting individual (instead of aggregation) $A_k$, $B_k$, $M_k$ in order to solve the optimization problem in Equation 14. Exposing individual statistics to the server can have privacy concerns and cannot be compatible with standard secure aggregation protocol or central differential privacy methods.

A: Thanks for proposing the privacy concern. In the FedLPA, $A_k$ is computed via the activations while $B_k$ is computed via the linear pre-activations of the layer. We note that $A_k$, $B_k$ and $M_k$ do not carry any label information, thus the transmission of $A_k$, $B_k$ and $M_k$ will not leak any label privacy. Otherwise, we agree that $A_k$, $B_k$ and $M_k$ are a function of data which may contain privacy-sensitive information of the local training data. However, our proposed FedLPA has the same privacy-preserving level as the conventional federated learning algorithms (i.e, FedAvg, FedProx, FedNova and Dense), which are all vulnerable to some privacy attacks (e.g, membership inference attack or reconstruction attacks). We believe that combining differential privacy with FedLPA can enhance the privacy protection of the training data, which is the most promising future direction. However, in this paper, we mainly focus on the efficiency and accuracy of one-shot federated learning algorithms and propose FedLPA which outperforms the SOTA one-shot algorithms on the efficiency and accuracy.

Reviewer comment (Evaluation):
Why are multi-round results of FedLPA worse than the alternatives as shown in Figure 2?

Author response:
Q:Why are multi-round results of FedLPA worse than the alternatives as shown in Figure 2?

A: As we shown in our Appendix D.1, we give an example, showing that the block-diagonal empirical Fisher information matrix could well capture the coarse structure of the full Fisher information matrix based on a neural network with three convolution layers and one fully connected layer trained on MNIST dataset. The four diagonal blocks corresponding to four layers, can capture the most useful information of the original Fisher Information Matrix. Thus, the approximation of the block-diagonal empirical Fisher information matrix can converge fast with most information learned and works well in one-shot scenario. However, it neglects inferior information in non- diagonal blocks.  

When we extend FedLPA to the multi-round setting, the inferior information that we omit may enrich the learning performance. While in the one-shot setting, the most useful information of the original Fisher Information Matrix matters.

Reviewer comment (Theory):
Consider a large model where one single layer weight can be enormous, how would one further decompose its Fisher information for communication efficiency?

Author response:
Q: Consider a large model where one single layer weight can be enormous, how would one further decompose its Fisher information for communication efficiency?

A: If a layer's weight in a large model has a considerably high dimension, implying a Fisher matrix with a large dimension, it significantly increases communication costs. In such cases, the most intuitive approach is to explore the possibility of dimensionality reduction for its Fisher matrix.

A promising approach to enhance the efficiency of our method may employ some low-rank factorization techniques [5]. As described in [6], the main idea involves performing an eigendecomposition on the Kronecker factors in [7], while preserving only the eigenvectors corresponding to the top k largest eigenvalues. As a result, this approach drastically reduces space complexity, enabling communication costs to be compared favorably with diagonal Fisher matrices.

[5]Lee, Jongseok, et al. "Estimating model uncertainty of neural networks in sparse information form." International Conference on Machine Learning. PMLR, 2020.

[6]Daxberger, Erik, et al. "Laplace redux-effortless bayesian deep learning." Advances in Neural Information Processing Systems 34 (2021).

[7]George, Thomas, et al. "Fast approximate natural gradient descent in a kronecker factored eigenbasis." Advances in Neural Information Processing Systems 31 (2018).

Reviewer comment (Writing):
There are numerous writing issues throughout the paper. Many sentences are grammatically incorrect, have incorrect punctuation and/or simply don't make sense. Some words are incorrectly capitalized or not capitalized when they should be. Examples include:  
- Page 1: "With the primary objectives of safeguarding data privacy and curbing the aggregation and management of data across institutions, the distribution of data exhibits variations among clients" -> The text before and after the comma are completely disconnected.  
- "Fedavg" should be FedAvg everywhere in the paper.  
- Page 2: "Layer-wise" -> layer-wise.  
- Page 2: "Fisher information matrices as a metric of the parameter space" -> Metric to measure what? How is the parameter space defined?  
- Page 2: "multi-variates linear objective function and using its quadratic form" -> Unclear meaning of quadratic form of a linear objective.  
- Page 2: "Nevertheless, from the theoretical analysis, we show that FedLPA has a linear convergence rate" -> Why is "nevertheless" here? Given that the objective is quadratic, gradient descent should have a linear convergence rate.  
- Page 4: "globally variational inference using Eq. 2" -> "global variational inference using Eq. 2".  
- Page 4: "uploads probability parameters to the server" -> What are probability parameters?  
- Page 5: "Modern algorithms (Rumelhart et al., 1986; Martens & Grosse, 2015a) allow the local training process to obtain an optimal, regarded as the expectation $\mu_k$ in the above equations" -> Grammatically incorrect; requires fixing.  
- Page 5: "it is an approximate of the Fisher information matrix" -> "it is an approximation of the Fisher information matrix".  
- Page 5: "computing all co-relations is impossible, which are inaccurate" -> Text before and after the comma is disconnected.

Author response:
We kindly thank the reviewer for their constructive and valuable feedback. We appreciate the time and effort you have dedicated to assessing our work and we are happy to address some of the concerns below. We sincerely hope you can read our responses and adjust the scores if appropriate. 


Q: Page 1: "With the primary objectives of safeguarding data privacy and curbing the aggregation and management of data across institutions, the distribution of data exhibits variations among clients" -> The text before and after the comma are completely disconnected.

A: Thanks for the doubt on the inner logic of the above sentence. Here, we want to show that in order to safeguard the data privacy, the conventional federated learning algorithm will use the aggregation methods and follows the data management rules of different institutions. However, in this way, the model are not trained with full data, which will incurs the problem that ‘the distribution of data exhibits variations among clients’. We will add the above explanations in the camera-ready version.


Q:"Fedavg" should be FedAvg everywhere in the paper.

A: Thanks for notifying our typos. We will revise all the “Fednova”, “Fedavg”, “Fedprox” in the camera-ready final version.

Q: Page 2: "Fisher information matrices as a metric of the parameter space" -> Metric to measure what? How is the parameter space defined?

A: FedLPA will consider the parameter space to aggregate the local models. Bear on that thought, we use FedLPA to aggregate the posteriors of local models using the accurately computed block-diagonal empirical Fisher information matrices.

Q: Page 2: " multi-variates linear objective function and using its quadratic form" -> I don't understand what the authors mean by quadratic form of a linear objective

A: In the Page 6 Section 3.5, Eq. 14 is our final multi-variates linear objective function and it is in the quadratic form.

Q: Page 2: "Nevertheless, from the theoretical analysis, we show that FedLPA has a linear convergence rate" -> Why is there a nevertheless here? Given that the objective is quadratic, gradient descent should have a linear convergence rate.

A: In Appendix F, we have shown the linear convergence rate. We will revise the sentence into “Furthermore, from the theoretical analysis” in the camera-ready version. 

Q:"uploads probability parameters to the server" -> What are probability parameters?

A: The data distribution may be biased on different dataset and each client may have a different number of different data samples. The probability parameters are the proportion of the number of data samples on each client over the total number of all the data samples. We will add the above explanations in the camera-ready version.

Q: Page 5: computing all co-relations is impossible, which are inaccurate" -> Text before and after the comma is disconnected.

A: Here, we want to denote that“which are inaccurate” means that the methods of Kirkpatrick et al. (2017); Liu et al. (2021) in the paper are not accurate, as they could not compute all co-relations. We will add the above explanation in the camera-ready version.

Reviewer comment (Theory):
Mathematical definitions and notation are not precise. Examples include:  
- The second proportionality in Eq. (2) holds only under the flat prior assumption, $p(\theta) \sim 1$, which is not stated clearly.  
- In the line with the definition of the Kronecker product (just below Eq. (10)), the definition of $A_{kl}$ and $B_{kl}$ is missing an expectation.  
- Section 3.5: It is unclear what $\bar{\Sigma}$ and $\bar{z}$ are, as Section 3.4 defines only $\bar{\Sigma}_l$ and $\bar{z}_l$ for layer $l$.  
- Section 3.5: "optimal solution of $\bar{\mu}$" should be "optimal solution of $f(\bar{\mu})$", since optimal solution of a vector does not make sense.  
- Section 3.5: "As we have $\mu = \bar{\Sigma}\cdot \bar{z}$" conflicts with the earlier definition of $\bar{\mu} = vec(\bar{M})$ in the same line, making the re-definition unclear.

Author response:
Q: Mathematical definitions are not precise. To add to the writing issues, in many places the mathematical notation/assumptions are not properly defined. Some examples are given below:

A: Thanks for the notification of the imprecise mathematical definitions. We will revise them in our camera-ready version.

Reviewer comment (Novelty):
The paper claims "Personalized One Shot Federated Learning" in its title and discusses the benefits of personalization in the Introduction, but the rest of the work focuses on learning a single global model on the server. The connection with personalization is unclear.

Author response:
Q: No connection with personalization. The title of the paper states "Personalized One Shot Federated Learning". The authors also write about the benefits of personalization in paragraph 4 of the Introduction section. However, the rest of the paper just focuses on the server learning a single global model. I don't understand what is the connection with personalization here.

A: Thanks for the comments on the aspect of the personalization. We understand that the personalized federated learning setting may contain the multiple aspects. In this paper, we mainly consider that each client has a different biased dataset, which is one of the personalized federated learning settings. We agree that our claim of the personalization is somehow weak. We will weaken the claim of the personalization in the camera-ready version.

Reviewer comment (Novelty):
The novelty appears limited. The idea of formulating FL as a posterior inference problem is already known (e.g., Al Shedivat et al. 2020), and approximating the Hessian as the Fisher is standard (Ritter et al. 2018). Prior work [1] has explored using the Fisher information for model aggregation, which is not cited. The only difference is using K-FAC instead of diagonal Fisher, which may not be significant novelty. K-FAC also requires clients to send $A_k, B_k, M_k$ matrices, increasing communication cost compared to diagonal Fisher. Privacy claims about $A_k, B_k, M_k$ preserving data/label privacy are made without empirical or theoretical justification.

Author response:
Q: The idea of formulating FL as a posterior inference problem is already well-known in [1] (Al Shedivat et al. 2020) 

A: To the best of our knowledge, we are the first to consider the posterior inference problem in one-shot scenario. Note that the approach [1] requires a lengthy burn-in period before conducting posterior inference, for instance, 400 rounds, and it updates global model parameters by modifying the covariance-aggregated local models. It means that the algorithm [1] necessarily requires multiple iterations and cannot be used in a one-shot scenario. In contrast, our method FedLPA only requires immediate variational inference after training the local model, ensuring higher flexibility and efficiency in one-shot scenario.

Besides, in the algorithm [1], obtaining statistical information to compute local covariances is of low rank. In reality, it fails to acquire the posterior of the aggregated model and cannot perform variational inference on the aggregated model. However, our method yields full-rank covariances, and after employing an expectation approximation method for variational inference on the aggregated model, we can achieve a usable global posterior.

Q:The idea of approximating the Hessian as the Fisher is also standard in Ritter et al. 2018. 

A: In both the domain of natural gradient optimization [1][2][3] and modeling output uncertainty in variational inference [4], using the Fisher approximation of the Hessian does not involve the issue of inverting covariance. However, in the context of federated learning, when performing variational inference on the aggregated model, the necessity of inverting covariance becomes unavoidable. To address this problem, we propose a novel algorithm that constructs a quadratic objective function. During aggregation, this algorithm directly trains the aggregated model using local covariances and expectations, thereby circumventing the need for inversion operations. Additionally, we provide a detailed convergence analysis of this method in Appendix F, labeled as "CONVERGENCE ANALYSIS OF EQ. 14".

[1]M. Al-Shedivat, J. Gillenwater, E. Xing, and A. Rostamizadeh, “Federated learning via posterior averaging: A new perspective and practical algorithms.

[2]James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning.

[3]Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning.

[4]A. Botev, H. Ritter, and D. Barber, “Practical gauss-newton optimisation for deep learning,” in International Conference on Machine Learning.

[5]H. Ritter, A. Botev, and D. Barber, “A scalable laplace approximation for neural networks,” in 6th International Conference on Learning Representations, ICLR 2018- Conference Track Proceedings.

[6] Matena, Michael S., and Colin A. Raffel. "Merging models with fisher-weighted averaging." Advances in Neural Information Processing Systems 35 (2022): 17703-17716.

[7]Liangxi Liu, Feng Zheng, Hong Chen, Guo-Jun Qi, Heng Huang, and Ling Shao. A bayesian federated learning framework with online laplace approximation. arXiv preprint arXiv:2102.01936, 2021.

[8]James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International conference on machine learning.

[9]Roger Grosse and James Martens. A Kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning.

[10]A. Botev, H. Ritter, and D. Barber, “Practical gauss-newton optimisation for deep learning,” in International Conference on Machine Learning.

[11]Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Towards a theoretical and practical understanding of one-shot federated learning with fisher information. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities, 2023.

Reviewer comment (Experiments):
Baselines are weak and datasets used are too easy. SCAFFOLD, FedNova, and FedProx are not good one-shot baselines, as they are designed for multi-round FL, explaining their similar performance to FedAvg. DENSE is the only competitive one-shot baseline tested. The authors cite FedOV and FedCAVE but do not compare to them, arguing they share more client-side information, yet FedLPA also requires sharing $A_k$ and $B_k$ matrices. FedLPA shows significant advantage only on easier datasets (FMNIST, MNIST, SVHN); improvements decrease for harder datasets like CIFAR-10. Testing on more challenging datasets (e.g., CIFAR-100, Tiny-ImageNet) and realistic FL datasets (EMNIST, Shakespeare [2]) would strengthen the paper.

Author response:
Q: Weak baselines and easy datasets. Among the baselines, I understand the comparison with FedAvg to show improvement over simple averaging. However SCAFFOLD, FedNova and FedProx are not good one-shot baselines. These methods are primarily designed for multi-round FL and therefore I am not surprised to see that their performance is similar to FedAvg in the experiments. DENSE is the only one-shot basline which can be considered competitive. The authors have cited other one-shot algorithms like FedOV and FedCAVE but have not compared them based on the argument that they entail sharing more client side information. However, I think it fair to compare these algorithms since FedLPA also requires clients to share and $A_k$ and $B_k$ matrices.

A: We note that $A_k$, $B_k$ and $M_k$ will not carry the label information, thus, the transmission of $A_k$, $B_k$ and $M_k$ not leak any label privacy. And in our paper, we mention that “FedOV and FEDCAVE entail sharing more client-side label information or transmitting client label information to the server, which may jeopardize label privacy and are beyond the scope of this study”. As, FEDCVAE (Heinbaugh et al., 2022) needed all the client label distribution to be transmitted to the server side and FedOV (Diao et al. 2023) needed the clients to know the labels which were unknown [11]. In this case to compare them fairly, we do not choose the FedOV and FedCAVE as the baseline approach.

Even so, in our appendix G.7 EXPERIMENTS WITH FEDOV, we show in some cases, our method FedLPA could outperform the state-of-the-art method FedOV. Note that FedOV consumes more computing resources, needs more label information, needs an extra distillation process and needs an extra outlier generation process.

Reviewer comment (Evaluation):
The reviewer is curious why the computation cost of FedProx is higher than FedAvg and FedNova in Table 4.

Author response:
Q: In addition, I was curious to know why the computation cost of FedProx is higher than FedAvg and FedNova in Table 4.

A: In our paper, we mainly adopt the most-cited non-iid FL benchmark (https://github.com/Xtra-Computing/NIID-Bench) to get the fair comparison of FedLPA and other baselines. The reason that why computation cost of FedProx is higher than FedAvg may be that the FedProx adds a L2 regular term to make local updates around the global mode, which adds more computing overhead. Besides, using the original codebase (https://github.com/litian96/FedProx) from FedPorx also consumes more time than FedAvg and FedNova, under the above non-iid FL benchmark. We will add the above analysis in the camera-ready version.

Reviewer comment (Evaluation):
My biggest concern is privacy risks. I doubt that communicating pre- and activations is more vulnerable to attacks, such as membership inference [1] or reconstruction attacks [2], as it contains more precise information in a layer-wise manner. For example, it is known that due to the sparsity of ReLU functions, gradient inversion attacks are less efficient in a large-batch case. However, the proposed method reveals both pre and after-activation values and may open up new attack possibilities. I am not saying the authors must solve the issue in this work, but I urge the authors to discuss the potential risks that the proposed method may introduce and solutions, e.g., differential privacy.

Author response:
Thanks for proposing the privacy concern. In the FedLPA, A is computed via the activations while B is computed via the linear pre-activations of the layer. We note that A, B and M do not carry any label information, thus the transmission of A, B and M will not leak any label privacy. Otherwise, we agree that A, B and M are a function of data which may contain privacy-sensitive information of the local training data. However, our proposed FedLPA has the same privacy-preserving level as the conventional federated learning algorithms (i.e, FedAvg, FedProx, FedNova and Dense), which are all vulnerable to some privacy attacks (e.g, membership inference [1] or reconstruction attacks [2]). We agree that combining differential privacy with FedLPA can enhance the privacy protection of the training data, which is the most promising future direction. However, in this paper, we mainly focus on the efficiency and accuracy of one-shot federated learning algorithms and propose FedLPA which outperforms the SOTA one-shot algorithms on the efficiency and accuracy.

[1] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP), 2019.

[2] Jonas Geiping, Hartmut Bauermeister, Hannah Droge, and Michael Moeller. Inverting gradients: how easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems (NeurIPS), 2020.

Reviewer comment (Experiments):
Despite the impressive numbers, the experiment settings seem limited. The author only considers a simple CNN network and a baseline tailored for one-shot federated learning.

Author response:
Thanks for the suggestion. We have added the experiments with FedLPA on the same experiments setting in the paper with resnet-18 with three random seeds on CIFAR-10 dataset. We set the parameters with beta=0.1, 0.3 and 0.5 with 10 clients. The results are shown as follows:

| **Beta**     | **Resnet-18**        | **Simple-cnn** | 
|---------------|------------------------|-------------|
| **0.1**       | 23.62±0.51 | 19.97±0.02|
| **0.3**       | 27.43±0.04| 26.60±0.01|
| **0.5**       | 31.70±0.14 | 24.20±0.02|

From the above table, we can see that with the more complex NN structure, our proposed method FedLPA can get better accuracy when in the federated one-shot settings. We will add the above table and results in the Appendix in the camera-ready version. Due to the rebuttal time limit, we have not added the experiments on the U-net. However, we believe that our proposed FedLPA works well on the U-net, as the implementation of FedLPA is not limited to the model architecture. 

We also note that in our appendix G.7, we show in some cases, our proposed method FedLPA can outperform the state-of-the-art federated one-shot approach FedOV, while the FedOV requires more label information with consuming more computing resources.

Reviewer comment (Reproducibility):
The authors claim a personalized federated learning setting. How does the proposed method customize the model for different users?

Author response:
Thanks for the comments on the aspect of the personalization. We understand that the personalized federated learning setting may contain the multiple aspects. In this paper, we mainly consider that each client has a different biased dataset, which is one of the personalized federated learning settings. We agree that our claim of the personalization is somehow weak. We will weaken the claim of the personalization in the camera-ready version.

Reviewer comment (Experiments):
Can the proposed method scale up to more complex networks, such as ResNet or U-net?

Author response:
Thanks for the suggestion. We have added the experiments with FedLPA on the same experiments setting in the paper with resnet-18 with three random seeds on CIFAR-10 dataset. We set the parameters with beta=0.1, 0.3 and 0.5 with 10 clients. The results are shown as follows:

| **Beta**     | **Resnet-18**        | **Simple-cnn** | 
|---------------|------------------------|-------------|
| **0.1**       | 23.62±0.51 | 19.97±0.02|
| **0.3**       | 27.43±0.04| 26.60±0.01|
| **0.5**       | 31.70±0.14 | 24.20±0.02|

From the above table, we can see that with the more complex NN structure, our proposed method FedLPA can get better accuracy when in the federated one-shot settings. We will add the above table and results in the Appendix in the camera-ready version. Due to the rebuttal time limit, we have not added the experiments on the U-net. However, we believe that our proposed FedLPA works well on the U-net, as the implementation of FedLPA is not limited to the model architecture. 

We also note that in our appendix G.7, we show in some cases, our proposed method FedLPA can outperform the state-of-the-art federated one-shot approach FedOV, while the FedOV requires more label information with consuming more computing resources.