Paper: Federated Virtual Learning on Heterogeneous Data with Local-global Distillation (ICLR-2024)

Reviewer comment (Novelty):
In your paper, the model on clients is split into feature extractors and classification heads. This split learning-like paradigm has been widely adopted by a series of prior works [1,2,3]. Please explain the deplorability of your approach on existing methods. More elaboration on how your proposed method relates to these works would be appreciated.

Author response:
Thanks for the questions. We appreciate the opportunity to point out the differences between FedLGD and the papers you suggested to highlight our novel approach and to discuss the potential deployment of FedLGD with the methods. 

- **We have different setups.** Although we both formulate a classification model into a feature extractor and a classification head, FedLGD always keeps forward and backward training on both parts in local clients; however, those methods require putting the classification head on the server for inference and updates.
- **We are orthogonal approaches.** FedLGD split the model intro feature extractor and classification head so that we can add *regularization to feature space* with *different data* (local and global distilled) on the *same local feature extractor*. In contrast, those papers are knowledge distillation methods that split the model so that they can obtain and share local features with the server. Then, they perform *regularization on logits* from the *same data* feature on *different classification heads.* 
- **Those methods make assumptions in feature sharing, but FedLGD does NOT** To enable the comparison of logits for knowledge distillation, those methods require sharing features from the local feature extractor with the server. Differently, FedLGD does not require any data feature sharing and we only share gradients following classical FL during the FL model updating stage. We consider that the sharing feature and additional knowledge may violate the intention of improving privacy in federated virtual learning.

Given the fundamental differences in the approaches and assumptions on information sharing, we believe our methods and those methods are not directly comparable. In fact, FedLGD and the model knowledge distillation-based methods can be combined as orthogonal strategies to address heterogeneity issues in FL. For example, the distilled global virtual data can serve as an ideal candidate to perform knowledge distillation as it roughly captures global data distribution. Thus, it can eliminate the requirement of sharing real data features or using public datasets. We hope our approach can shed light on leveraging virtual data in FL and inspire future work on the interesting exploration of combining our strategy with others.

We thanks the reviewer for pointing us to the interesting papers, and we have added them in our related work.

[1] "FedICT: Federated Multi-task Distillation for Multi-access Edge Computing." IEEE Transactions on Parallel and Distributed Systems (2023).

[2] "Group knowledge transfer: Federated learning of large cnns at the edge." Advances in Neural Information Processing Systems 33 (2020): 14068-14080.

[3] "Exploring the distributed knowledge congruence in proxy-data-free federated distillation." arXiv preprint arXiv:2204.07028 (2022).

Reviewer comment (Writing):
If I understand you correctly, FedProx is proposed by [4] rather than [5]. Do I misunderstand something?

Author response:
We thank the reviewer for the careful review and for pointing out this subtle error. We incorrectly referred FedProx to [2] in our Section 4.1 and 5 due to the similarity of the reference codes used by [1] and [2] (i.e., li2020a and li2020b). It should be [1] instead. We really appreciate this comment, and we have corrected it in our revision.

[1] "Federated optimization in heterogeneous networks." Proceedings of Machine learning and systems 2 (2020): 429-450.

[2] "On the convergence of fedavg on non-iid data." arXiv preprint arXiv:1907.02189 (2019).

Reviewer comment (Experiments):
Some of the benchmark algorithms, such as FedProx [4], Scaffold [6], are somewhat outdated. In your experiments, you have used different open-sourced datasets as private data for clients, and this degree of data heterogeneity is apparently unfavorable for the regularization-based methods mentioned above. Would it be possible to compare your approach with some novel federated learning methods based on GANs [7], which seem to be more suitable for your scenario?

Author response:
We thank the reviewer for pointing out the interesting FL work, FedGen [2]. We did not include FedGen (and other knowledge distillation-based methods) in our comparison works is that we consider knowledge distillation and data distillation two orthogonal directions to solve data heterogeneity issues. 

It is worth noting the training strategy of federated virtual learning is quite novel and under-explored. Thus most of the existing FL methods are not suitable for making direct comparisons. In fact, we included VHL [1], which shares most similar thoughts as FedLGD by using virtual data as regularization in local training. We show that using the virtual data generated by federated gradient matching can better handle the heterogeneous data scenario. 

Following the suggestion, we added the work to the related work in our revision and justified the orthogonality between knowledge distillation-based FL methods and FedLGD. To address the reviewer’s question, we report the results on FedGen [2] on DIGITS, CIFAR10C(client ratio=1), and RETINA in the following table. We also include FedGen in the related work in our revision.
.

| Avg. Acc.       | FedGen| VHL     | FedLGD |
|-------------------|------------|-----------|---------------|
| DIGITS          | 66.9      | 86.5      | **86.7**    |
| CIFAR10C     | 39.6      | 55.2      | **57.4**    |
| RETINA         | 82.1      | 78.6      | **85.1**    |

Observe that FedLGD consistently outperforms other methods. We conjecture the reason FedGen is not performing well for two reasons: It requires a larger number of training data to obtain a more reasonable global GAN. 2. It requires a larger feature space to better regularize local training with knowledge distillation (as shown in RETINA experiments, where it can use a large feature space, thus resulting in better performance). 

[1] Tang Z, Zhang Y, Shi S, He X, Han B, Chu X. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. InInternational Conference on Machine Learning 2022 Jun 28 (pp. 21111-21132). PMLR.

[2] Zhu Z, Hong J, Zhou J. Data-free knowledge distillation for heterogeneous federated learning. InInternational conference on machine learning 2021 Jul 1 (pp. 12878-12889). PMLR.

Reviewer comment (Presentation):
t-SNE figures are not represented as vectors.

Author response:
We thank the reviewer for the clarification comment. We chose to show the 2-D plot for more intuitive visualization. In our original submission, we also reported the **statistical distances** between the two example datasets from two clients to support the heterogeneity statement. Specifically, in **the second paragraph of our Introduction**  we stated “Quantitatively, we found that using dataset distillation can amplify the statistical distances between the two datasets, with Wasswestein Distance and Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) both increasing by around 40%.” We believe that the statistical distance and t-SNE plots can provide a reasonable evidence to our finding.

Reviewer comment (Theory):
Sharing gradients from clients to the server for a global virtual data update may pose security risks. Some attacks could potentially reconstruct raw data using gradient information, similar to the risks associated with Deep Gradient Leakage. Why sharing averaged gradients is safe?

Author response:
We thank the reviewer for the constructive comment. Yes, privacy is indeed an important concern in federated learning. Thus, we discuss the privacy guarantee of FedLGD in the **last paragraph in Section 3** and we highlight the statements as follows: 

- FedLGD does not share additional client information compared to baseline methods such as FedAvg [1].
- Notably, the gradients we share in FedLGD are w.r.t. Local virtual data distilled by Distribution Matching [2], which has been shown to defend against Membership Inference Attack and Reconstruction Attack [3]. We also show our empirical defense results against Membership Inference Attack in Appendix E.6. 
- Privacy preservation can be further improved by employing differential privacy [4] in dataset distillation, but this is beyond the main focus of our work.

[1] McMahan B, Moore E, Ramage D, Hampson S, y Arcas BA. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics 2017 Apr 10 (pp. 1273-1282). PMLR.

[2] Zhao B, Bilen H. Dataset condensation with distribution matching. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2023 (pp. 6514-6523).

[3] Dong T, Zhao B, Lyu L. Privacy for free: How does dataset condensation help privacy?. InInternational Conference on Machine Learning 2022 Jun 28 (pp. 5378-5396). PMLR.

[4] Abadi M, Chu A, Goodfellow I, McMahan HB, Mironov I, Talwar K, Zhang L. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security 2016 Oct 24 (pp. 308-318).

Reviewer comment (Theory):
What is the rationale behind clients requiring local virtual data instead of training directly on their local private data?

Author response:
We thank the reviewer for the clarification question. As we stated **in our abstract**, training with distilled virtual data can provide higher training efficiency and alleviate the problem of inference attacks, as demonstrated in centralized settings [1]. 

Building on this, we explore its usage in FL (named federated virtual learning - FVL), because privacy and efficiency (computation cost) are common concerns in FL, together with its obvious assistance in synchronization.

Specifically, the benefits are:

- **Synchronization**: Training with a small number of representative virtual data can facilitate local training, which can improve FL synchronization.
- **Computation Cost**: As shown in Appendix D.3 and Table 6, we empirically show that training with virtual data requires a much lighter computation cost. (We’ve corrected the typo in Appendix D.3 in our revision.)
- **Privacy**: As reported in [1], training with virtual data can protect the model from privacy attacks such as Inversion Attack [2] and Membership Inference Attack (MIA) [3]. In addition, as stated in **the last paragraph of Section 3**, “we consider averaged gradients w.r.t. local virtual data and the method potentially defends inference attacks better (Appendix E.6).”

The core of this work aims to resolve the heterogeneity issue associated with distillation-based virtual data in FVL. This will make the use of virtual data more feasible in FL and lead to improved synchronization, efficiency, and privacy while approaching the utility of real data.

[1] Dong T, Zhao B, Lyu L. Privacy for free: How does dataset condensation help privacy?. InInternational Conference on Machine Learning 2022 Jun 28 (pp. 5378-5396). PMLR.

[2] Geiping J, Bauermeister H, Dröge H, Moeller M. Inverting gradients-how easy is it to break privacy in federated learning?. Advances in Neural Information Processing Systems. 2020;33:16937-47.

[3] Shokri R, Stronati M, Song C, Shmatikov V. Membership inference attacks against machine learning models. In2017 IEEE symposium on security and privacy (SP) 2017 May 22 (pp. 3-18). IEEE.

Reviewer comment (Experiments):
Could you clarify why this method has not been compared to other FL methods utilizing dataset distillation?

Author response:
We thank the reviewer for raising this question. As we stated in the related work (Sec. 2.2), to our best knowledge, the current FL+Data distillation works typically share local distilled data to server [1,2,3], which we consider a non-practical and more privacy-sensitive operation. Furthermore, some methods (e.g., [1]) need to use real data as initialization to obtain good performance.

On the contrary, due to privacy concerns, we choose to train the model locally and share the gradients w.r.t. Local virtual data,  which is the default operation in classical FL, such as FedAvg. Therefore, the other FL+data distillation settings are significantly different from ours regarding the information shared and data leakage. We discussed these studies in our related work but believed it was unfair to make a direct comparison, given their special request for information sharing. 

[1] Xiong Y, Wang R, Cheng M, Yu F, Hsieh CJ. Feddm: Iterative distribution matching for communication-efficient federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 (pp. 16323-16332).

[2] Goetz J, Tewari A. Federated learning via synthetic data. arXiv preprint arXiv:2008.04489. 2020 Aug 11.

[3] Hu S, Goetz J, Malik K, Zhan H, Liu Z, Liu Y. FedSynth: Gradient Compression via Synthetic Data in Federated Learning. InWorkshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022) 2022 Oct 21.