Paper: Heterogeneous Personalized Federated Learning by Local-Global Updates Mixing via Convergence Rate (ICLR-2024)

Reviewer comment (Experiments):
It is not clear if the results are a fair comparison, i.e., whether or not each algorithm is independently optimized sufficiently well.

Author response:
> - It is not clear if the results are a fair comparison, i.e. whether or not each algorithm is independently optimized sufficiently well.

Our comparative analysis is performed in a fair manner, as all the methods follow the same experimental settings. Furthermore, the hyperparameters of the compared method are carefully fine-tuned. The reported findings presented in this study are in alignment with recent publications [1, 2], further substantiating the credibility and reliability of our results.

[1] Huang, Wenke, Mang Ye, and Bo Du. "Learn from others and be yourself in heterogeneous federated learning." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10143-10153. 2022.

[2] Zhuang, Weiming, and Lingjuan Lyu. "Is Normalization Indispensable for Multi-domain Federated Learning?." arXiv preprint arXiv:2306.05879 (2023).

Reviewer comment (Presentation):
The section on the convergence analysis seems a bit disintegrated with the rest of the paper.

Author response:
> - The section on the convergence analysis seems a bit disintegrated with the rest of the paper.

We propose Theorem 1 in the convergence analysis to give a bound, while our method is motivated by Proposition 1. Theorem 1 serves as a more general proof of the convergence of the PFL algorithms combining local and global updates, as it poses no constraint on $\lambda$. We admit that more discussion can be made on the convergence analysis, and we have added it to the manuscript.

Reviewer comment (Experiments):
Simple experiments that highlight the applicability of the method are not included. A clear demonstration, e.g., in comparison with fixed local-global ratios, would have been appreciated.

Author response:
> - Simple experiments that highlight the applicability of the method are not included. I would have appreciated a clear demonstration, e.g. in comparison with fixed local-global ratios.

Following your suggestion, we have extended the investigation on the performance by using different fixed ratios and the optimized ratio. For the fixed optimized ratio, we take the average ratio of our method in the last 10 training rounds. The results are shown in the following table. When the ratio equals to 0, our method will degenerate to the FedAvg, and when the ratio goes to 1, it equals to complete client local training. From the results, it can be observed that introducing the mixing of local and global updates helps improve the personalized model performance, and our dynamic adjustment shows the highest accuracy.
| Mixing ratio |   0   |  0.25 |  0.5  |  0.75 |  1.0  | Fix(our optimized) | Ours  |
|:------------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|-------|
|   Accuracy   | 94.68 | 95.37 | 95.64 | 94.48 | 96.03 |  96.65 | 97.11|

Reviewer comment (Experiments):
An empirical demonstration of proposition 1 would have been a strong argument for the specific choices made for the mixing ratio.

Author response:
>- An empirical demonstration of proposition 1 would have been a strong argument for the specific choices made for the mixing ratio.

Thank you for your insightful comment. Our proposition 1 mainly motivates us to approximate the convergence rate by measuring the trace of the Gram matrix. However, the computation of the Gram matrix $H$ in practical networks poses significant challenges and is often computationally infeasible [1,2,3].

[1] Fort, Stanislav, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. "Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel." Advances in Neural Information Processing Systems 33 (2020): 5850-5861.

[2] Mohamadi, Mohamad Amin, Wonho Bae, and Danica J. Sutherland. "A fast, well-founded approximation to the empirical neural tangent kernel." In International Conference on Machine Learning, pp. 25061-25081. PMLR, 2023.

[3] Holzmüller, David, Viktor Zaverkin, Johannes Kästner, and Ingo Steinwart. "A framework and benchmark for deep batch active learning for regression." Journal of Machine Learning Research 24, no. 164 (2023): 1-81.

Reviewer comment (Reproducibility):
In the Algorithm Line.6, how to calculate the $h(x_i)$? This feature seems to adopt the global feature extractor to generate the approximation of global convergence. While in the local client, this seems to be untenable without the access to the global model. Does it require storing a global model locally? If so, I think a storage comparison among baselines is required to comprehensively compare the efficiency.

Author response:
>- How to calculate the h(x_i)? Does it require storing a global model locally? And adding a storage comparison among baselines.

**(a) Calculate the $h(x_i)$:**
The $h(x_i)$ is the latent feature representation of the input $x_i$ extracted from the last hidden layer, where $h$ denotes the parameters up to the penultimate layer.


**(b) Store a global model locally:**
Yes. A global model is required to keep locally to calculate the global feature.

**(c) Storage comparison among baselines：**
Following your suggestion, we monitored the peak GPU memory cost of each method and listed the costs in the table below. Our method takes a reasonable cost compared with other baseline methods. Please note that we do not report FedRep, FedBABU, and FedHKD on the retinal dataset because they are specifically designed for classification tasks.

**GPU memory cost (MB)**
|     Method    | Digits5 | OfficeCaltech10 | DomainNet | Camelyon17 | Retinal |
|:-------------:|:-------:|:---------------:|:---------:|:----------:|:-------:|
|     FedAvg    |  270.24 |     1042.32     |  1042.32  |   1096.89  | 1814.29 |
|     APFL      |  600.77 |     2993.48     |  2993.48  |   2955.24  | 4504.02 |
|     L2SGD     |  456.18 |     2057.62     |  1157.62  |   2072.09  | 2329.91 |
|     FedAlt    |  458.27 |     1042.33     |  1042.33  |   1096.89  | 2297.57 |
|   PerFedAvg   |  270.24 |     1042.32     |  1042.32  |   1096.89  | 1814.29 |
|     FedBN     |  458.27 |     1042.32     |  1042.32  |   1097.99  | 2295.21 |
|    FedFOMO    |  458.28 |     1948.98     |  1948.98  |   1985.06  | 2837.97 |
|     FedRep    |  458.27 |     1042.30     |  1042.30  |   1096.89  |    -    |
|    FedBABU    |  458.27 |     1042.30     |  1042.30  |   1096.89  |    -    |
|     FedHKD    |  864.28 |     3589.56     |  3589.56  |   3021.02  |    -    |
| LG-Mix (Ours) |  472.37 |     2101.01     |  2101.01  |   1905.38  | 3730.69 |

Reviewer comment (Evaluation):
The authors should provide the training wall-clock time required to compare the practical efficiency.

Author response:
>- The training wall-clock time

For the computational time, we counted the training time of all methods. The GPU type used for training is GeForce RTX 2080 Ti. From the results in the table, it can be observed that our method lies in a reasonable range with a slightly higher cost than FedAvg, which is a trade-off between computational efficiency and performance.

**Training time (hour)**
|     Method    | Digits5 | OfficeCaltech10 | DomainNet | Camelyon17 | Retinal |
|:-------------:|:-------:|:---------------:|:---------:|:----------:|:-------:|
|     FedAvg    |   3.70  |       1.28      |   17.37   |    55.01   |  25.95  |
|     APFL      |   4.38  |       1.70      |   19.59   |    79.12   |  26.36  |
|     L2SGD     |   3.83  |       1.40      |   18.42   |    57.17   |  26.79  |
|     FedAlt    |   4.09  |       1.47      |   28.77   |    76.80   |  34.16  |
|   PerFedAvg   |   3.79  |       1.45      |   18.66   |    56.97   |  26.44  |
|     FedBN     |   3.73  |       1.26      |   17.43   |    55.03   |  26.56  |
|    FedFOMO    |   6.02  |       2.80      |   19.04   |    76.80   |  36.90  |
|     FedRep    |   4.32  |       1.41      |   19.18   |    55.77   |    -    |
|    FedBABU    |   3.96  |       1.31      |   15.92   |    62.12   |    -    |
|     FedHKD    |  22.43  |       2.64      |   37.33   |    68.71   |    -    |
| LG-Mix (Ours) |   5.86  |       1.32      |   18.92   |    57.98   |  26.55  |

Reviewer comment (Evaluation):
The motivation is to select the faster convergence to match the local or global model. So I think it is necessary to introduce the loss curve or gradient norm to illustrate its improvements. This paper does not analyze the improvements in the test error and generalization performance. Under the specific convergence analysis, how does the proposed method match the convergence analysis? And does the training loss or gradient norm decrease faster?

Author response:
> - Draw the loss curve to illustrate the model improvements regarding faster convergence

As per your suggestion, we have included the training and validation loss values in the table provided below. The results indicate that PFL methods generally converge faster than the baseline method FedAvg. Among all the compared methods, our method clearly promotes the convergence speed and presents the lowest training and validation loss. Additionally, we have draw the training/validation loss curves in Section D.2, Figure 4, in the Appendix for your reference.


|  Round |    1   |    5   |   10   |   20   |   50   | 100    |
|:------:|:------:|:------:|:------:|:------:|:------:|--------|
| Training Loss|
| FedAvg | 0.2062 | 0.1451 | 0.1145 | 0.0728 | 0.0239 | 0.0116 |
|  APFL  | 0.2062 | 0.1453 | 0.1152 | 0.0753 | 0.0218 | 0.0107 |
| FedAlt | 0.2208 | 0.1411 | 0.1049 | 0.0598 | 0.0146 | 0.0061 |
| FedBN  | 0.2135 | 0.1431 | 0.1097 | 0.0663 | 0.0192 | 0.0111 |
|   LG-Mix (Ours)  | 0.1682 | 0.0556 | 0.0183 | 0.0048 | 0.0015 | 0.0006 |
| Validation Loss|
| FedAvg | 0.4684 | 0.3906 | 0.3185 | 0.2716 | 0.2412 | 0.2335 |
|  APFL  | 0.2457 | 0.2057 | 0.1739 | 0.1779 | 0.1772 | 0.1814 |
| FedAlt | 0.4755 | 0.2596 | 0.2003 | 0.1676 | 0.1606 | 0.1655 |
| FedBN  | 0.472  | 0.3251 | 0.2594 | 0.2196 | 0.2009 | 0.1922 |
| LG-Mix (Ours)  | 0.1746 | 0.1162 | 0.1092 | 0.1103 | 0.1158 | 0.1203 |

Reviewer comment (Experiments):
The authors could discuss the scalability and robustness of their method to larger and more complex datasets, and explore potential applications in other domains beyond natural and medical images.

Author response:
> - The authors could discuss the scalability and robustness of their method to larger and more complex datasets, and explore potential applications in other domains beyond natural and medical images.

We address your comments from three aspects: **(a) Clarify our existing large and complex datasets**, **(b) Extend the client scalability study (incorporating more clients) on a large complex medical dataset**, and **(c) Explore applications in the new domain of brain-computer interface data.**


**(a)Clarify our existing large and complex datasets:**
We included two real-world complex medical datasets in our experiments. The Camelyon17 dataset is a very large dataset that contains 450,000 histology image patches with different stains from five hospitals. The retinal dataset consists of retinal fundus images acquired from 6 different institutions with various imaging devices and protocols. The superior performance of our proposed method has demonstrated the robustness on real-world large and complex datasets.


**(b)Extend the client scalability study (incorporating more clients) on a large complex medical dataset:**
We have further investigated the client scalability (by incorporating more clients during training) of our method on the medical cancer diagnosis dataset (i.e., Camelyon17). Below, we present the average test performance by incorporating more clients into training. Our method shows a stable increasing trend compared with other methods.
|     Method    | Client 1,2 | Client 1,2,3 | Client 1,2,3,4 | Client 1,2,3,4,5 |
|:-------------:|:----------:|:------------:|:--------------:|:----------------:|
|     FedAvg    |      96.56 |        95.61 |          95.63 |            95.00 |
|     FedAlt    |      96.56 |        96.58 |          96.71 |            96.66 |
|     FedBN     |      96.83 |        95.40 |          96.46 |            95.53 |
| LG-Mix (Ours) |      98.22 |        98.49 |          98.58 |            98.75 |


**(c) Explore applications in the new domain of brain-computer interface data:**
Following your suggestion, we further explored the application on the brain-computer interface (BCI) data, which consists in classifying the mental imagery EEG (electroencephalography) datasets. The EEG data is a type of data collected from electrodes placed on the scalp to measure and record electrical activity in the brain. This data can provide valuable insights into brain function and can be used in various fields such as neuroscience, psychology, and medicine. Specifically, we use four datasets from the MOABB BCI benchmark [1], each of those EEG datasets have been acquired on different subjects and have different numbers of channels, resulting in heterogeneous features. We compared our method with all other methods in our manuscript and reported five evaluation metrics, including Accuracy, AUC, Sensitivity, Specificity, and F1-score. The results are shown in the following table. Our method achieves the highest performance on all metrics, with significant improvements of 2.73% accuracy by the second-best method. More details can be found in Section D.2 in the Appendix.


**BCI dataset**
|   Method  | Acuracy |  AUC  |  Sensitivity  |  Specificity  |   F1-score  |
|:---------:|:-------:|:-----:|:-----:|:-----:|:-----:|
|   FedAvg  |  67.38  | 69.71 | 68.14 | 66.77 | 67.29 |
|   APFL    | *71.01*  | *75.48* | *71.21* | 71.08 | *70.97* |
|   L2SGD   |  67.39  | 69.96 | 69.11 | 65.88 | 67.31 |
|   FedAlt  |  68.46  | 72.74 | 69.72 | 67.52 | 68.45 |
| PerFedAvg |  67.47  | 71.25 | 64.22 | 70.56 | 67.35 |
|   FedBN   |  68.06  | 70.62 | 68.28 | 68.03 | 68.03 |
|  FedFOMO  |  64.69  | 68.56 | 65.36 | 64.18 | 64.53 |
|   FedRep  |  68.10  | 69.82 | 69.75 | 66.68 | 68.01 |
|  FedBABU  |  68.03  | 70.45 | 69.99 | 66.25 | 67.95 |
|   FedHKD  |  68.69  | 70.34 | 64.99 | *72.32* | 68.44 |
|LG-Mix (Ours) |  **73.74**  | **78.11** | **75.19** | **72.58** | **73.72** |


[1] Jayaram, Vinay, and Alexandre Barachant. "MOABB: trustworthy algorithm benchmarking for BCIs." Journal of neural engineering 15, no. 6 (2018): 066011.

Reviewer comment (Presentation):
The paper could benefit from a more detailed discussion of the limitations and potential extensions of the proposed method.

Author response:
>- What are the possible limitations of the proposed method, and how can they be addressed in future work?

Thanks for the insightful question. One potential limitation of our method is that we approximate the trace calculation by only using the last-layer latent feature.
While this approximation is reasonable and reflects the convergence rate in principle, it may not be an accurate estimation when using large foundation models as the model backbone. The convergence behavior may differ due to the model's architecture and complexity. There is a risk of obtaining less precise estimations in these scenarios. To address this limitation, one potential approach is to consider the prediction errors of local and global models. By incorporating the model errors into the mixing process, we can obtain more comprehensive measurements for combining local and global updates. However, exploring the use of the foundation model in federated learning is still an open question. Explorations are needed in future work to validate the effectiveness of our method further. We have included a detailed discussion of our method's limitations and potential extensions in Section E in the Appendix.

Reviewer comment (Theory):
How is eq. 4 derived? Please give more explanation on it. y is label, w(t) is the local model weights, what's the meaning of y-w(t)?

Author response:
>- How is eq. 4 derived? please give more explanation on it. y is label, w(t) is the local model weights, what's the meaning of y-w(t)?


Eq. 4 is derived by following Du et al.’s work [1], which uses the Gram matrix to characterize the optimization process of gradient descent for neural networks. Eq. 4 is the major theorem (Theorem 4.1) in Du et al.’s work to show the convergence rate of gradient descent. It measures the prediction error, which is the difference between the model prediction and the ground truth label.


$y-w(t)$ should be $y-y(t)$ in Eq.(4), which denotes the prediction error. It is a typo, many thanks for pointing this out, and sorry for the confusion.
In our paper, $w(t)$ is used to denote the local model weights, and $y(t)=f(w(t),x)$ denotes the model prediction with model weights $w(t)$ and $f: \mathbb{R}^d \rightarrow \mathbb{R}$ is the model function. We have revised our manuscript's Eq.(4) to avoid confusion.

[1] Du, Simon S., Xiyu Zhai, Barnabas Poczos, and Aarti Singh. "Gradient descent provably optimizes over-parameterized neural networks." ICLR (2019).

Reviewer comment (Evaluation):
What's the cost (e.g., GPU memory cost, computational time) of the proposed method, compared to the baseline methods?

Author response:
> - What's the cost (e.g., GPU memory cost, computational time) of the proposed method, compared to the baseline methods?


The GPU memory cost and computational time cost of our method come from two aspects. First, we store a copy of the global model during local training. Second, our method calculates latent features and the trace of latent feature matrices.


For GPU memory cost, we monitored the peak GPU memory cost of each method and listed the costs in the table below. Our method takes a reasonable cost compared with other baseline methods. Please note that we do not report FedRep, FedBABU, and FedHKD on the retinal dataset because they are specifically designed for classification tasks.


**GPU memory cost (MB)**
|     Method    | Digits5 | OfficeCaltech10 | DomainNet | Camelyon17 | Retinal |
|:-------------:|:-------:|:---------------:|:---------:|:----------:|:-------:|
|     FedAvg    |  270.24 |     1042.32     |  1042.32  |   1096.89  | 1814.29 |
|     APFL      |  600.77 |     2993.48     |  2993.48  |   2955.24  | 4504.02 |
|     L2SGD     |  456.18 |     2057.62     |  1157.62  |   2072.09  | 2329.91 |
|     FedAlt    |  458.27 |     1042.33     |  1042.33  |   1096.89  | 2297.57 |
|   PerFedAvg   |  270.24 |     1042.32     |  1042.32  |   1096.89  | 1814.29 |
|     FedBN     |  458.27 |     1042.32     |  1042.32  |   1097.99  | 2295.21 |
|    FedFOMO    |  458.28 |     1948.98     |  1948.98  |   1985.06  | 2837.97 |
|     FedRep    |  458.27 |     1042.30     |  1042.30  |   1096.89  |    -    |
|    FedBABU    |  458.27 |     1042.30     |  1042.30  |   1096.89  |    -    |
|     FedHKD    |  864.28 |     3589.56     |  3589.56  |   3021.02  |    -    |
| LG-Mix (Ours) |  472.37 |     2101.01     |  2101.01  |   1905.38  | 3730.69 |


For the computational time, we counted the training time of all methods. The GPU type used for training is GeForce RTX 2080 Ti. From the results in the table, it can be observed that our method lies in a reasonable range with a slightly higher cost than FedAvg, which is a trade-off between computational efficiency and performance.


**Training time (hour)**
|     Method    | Digits5 | OfficeCaltech10 | DomainNet | Camelyon17 | Retinal |
|:-------------:|:-------:|:---------------:|:---------:|:----------:|:-------:|
|     FedAvg    |   3.70  |       1.28      |   17.37   |    55.01   |  25.95  |
|     APFL      |   4.38  |       1.70      |   19.59   |    79.12   |  26.36  |
|     L2SGD     |   3.83  |       1.40      |   18.42   |    57.17   |  26.79  |
|     FedAlt    |   4.09  |       1.47      |   28.77   |    76.80   |  34.16  |
|   PerFedAvg   |   3.79  |       1.45      |   18.66   |    56.97   |  26.44  |
|     FedBN     |   3.73  |       1.26      |   17.43   |    55.03   |  26.56  |
|    FedFOMO    |   6.02  |       2.80      |   19.04   |    76.80   |  36.90  |
|     FedRep    |   4.32  |       1.41      |   19.18   |    55.77   |    -    |
|    FedBABU    |   3.96  |       1.31      |   15.92   |    62.12   |    -    |
|     FedHKD    |  22.43  |       2.64      |   37.33   |    68.71   |    -    |
| LG-Mix (Ours) |   5.86  |       1.32      |   18.92   |    57.98   |  26.55  |