Paper: Text-Free Federated Transformers Knowledge Distillation Without GAN (ICLR-2024)

Reviewer comment (Novelty):
**Motivation.** The motivation of this paper did not convince me. It seems that the target problem is ambiguous and meaningless. The authors seem to just make a minor modification to replace GAN in FL's knowledge distillation for NLP tasks and it lacks motivations and scenarios. GAN is actually rarely used in NLP and NLP is also less studied in FL before. Not using GAN in NLP is trivial and common, which cannot be the main motivation. An appropriate motivation is the problems raised in actual scenarios and previous works, not the "a + b" pattern. Also, the authors think GAN will leak privacy and the proposed method can protect privacy, but the authors didn't provide evidence to support that point.

Author response:
We appreciate your time and expertise in reviewing my manuscript. Your constructive feedback has been instrumental in enhancing the quality of the paper.

We acknowledge that our discussion regarding GANs and motivation has not been sufficiently elucidated. However, we persist in asserting the scarcity and necessity of research on non-text distillation in Federated Learning  within the domain of NLP. We possess compelling evidence that delineates the points of contribution we make to the field. Kindly afford us another opportunity to articulate our motivation more clearly in this context and present a more precise exposition.

1. Previous Data-free distillation of FL research (e.g., FedGEN, FedFTG) based on GAN often relies on image tasks (e.g., MNIST, CIFAR100) as primary benchmarks, neglecting NLP tasks. Other studies involving NLP tasks, such as FedAUX and FedDF, typically circumvent GANs and opt for auxiliary data distillation. Given the substantial gap in research on text tasks within FL's data-free distillation frameworks, exploring this area is both valuable and meaningful.

2. A practical issue arises with GAN in text generation [1](2023 NIPS). It is widely acknowledged that a notable obstacle in NLP GANs is their inability to generate differentiable outputs, given the discrete nature of language models (words). This lack of differentiability hampers mainstream Federated Learning (FL) Data-Free distillation frameworks, such as FedGEN[6] and FedFTG[7]. These frameworks utilize GANs for target learning functions but are ineffective in transmitting errors back to the generator, rendering them unsuitable for generating synthetic text in NLP distillation tasks under federated learning.

3. Can alternative text generation models be employed aside from GANs? Our response is that the proposed solution entails substantial costs in terms of privacy preservation and computational overhead. 

The current state-of-the-art in text generation, such as large-scale Transformer models like GPT-4, necessitates self-supervised training, involving memorizing client-side text data. However, this introduces privacy concerns, requiring intricate training mechanisms[2] and defense mechanisms[3] to ensure Transformer models remember the text while safeguarding privacy. Additionally, it raises the risk of attackers reconstructing text through pre-trained models[4,10], adding computational and communication overhead.


- **Concerning privacy** complex training[2] and defense mechanisms[3] are required to ensure the Transformer remembers the text while protecting privacy. Simultaneously, precautions must be taken to prevent attackers from reconstructing text through pre-trained models[4,10], introducing both computational and communication overhead.

- **Regarding computational overhead** deeper generative models (e.g., BERT, GPT) capable of memorizing more client-side text may exceed the computational capacity of the original task model, imposing a greater computational burden on clients.

 - **Concerning communication overhead** the baseline Transformer for the original task already consumes substantial communication, such as FedKD[9] using shallow TinyBERT resulting in 0.17-1.03GB per mobile client. Introducing language generation models like FedGEN would further amplify communication overhead.


In summary, GANs are unsuitable for text generation, rendering mainstream FL data-free distillation frameworks inadequate for text tasks. Instead of modifying the FedGEN framework, we propose a method that originates from the Transformer itself, generating data and distilling it. This approach aims to enhance model efficiency while circumventing the privacy, computational, and communication challenges associated with GANs. Compared to FedAUX and FedDF methods, our approach demonstrates superior performance in enhancing model efficiency.

Reviewer comment (Experiments):
**Baselines.** The authors missed some important baselines in the experimental part, which weakens the validity of the proposed method. Specifically, the authors should compare the following methods in the experiments: [1] Zhang L, Shen L, Ding L, et al. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10174-10183; [2] Zhu Z, Hong J, Zhou J. Data-free knowledge distillation for heterogeneous federated learning[C]//International conference on machine learning. PMLR, 2021: 12878-12889; [3] Wang H, Li Y, Xu W, et al. DaFKD: Domain-aware Federated Knowledge Distillation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 20412-20421.

Author response:
Due to our in-depth investigation of methods such as FedGEN, FedFTG, and DaFKD, we have uncovered the inapplicability of these approaches, which employ GANs on Transformer models in textual tasks.

It is crucial to reiterate that our proposed method in the paper arises specifically due to the inadequacy of using GANs, as observed in FedGEN, FedFTG, and similar approaches, within the context of NLP tasks. Furthermore, it is noteworthy that even the reviewers acknowledge the infrequent utilization of GANs in NLP and the relatively limited exploration of NLP in the Federated Learning domain (GAN is actually rarely used in NLP and NLP is also less studied in FL before). In light of these considerations, we contend that these methods cannot serve as suitable baselines.

Moreover, when considering the nature of the tasks, the fundamental tasks outlined in the three papers provided by the reviewer primarily pertain to image-related tasks. In contrast, our proposed method focuses on textual tasks. Therefore, we argue that these image-centric methods are not suitable as baselines for evaluating our approach.

Reviewer comment (Experiments):
Limited exploration of sampling methods: The paper focuses on three sampling methods (random, target, and adversary sampling) but does not explore other potential sampling strategies. Investigating alternative sampling techniques could lead to further improvements in the performance of the proposed method.

Author response:
The three sampling methods are indeed limited, and in future work, we plan to introduce additional contrastive learning losses to explore the possibility of expanding sampling methods.

Reviewer comment (Experiments):
Limited exploration of model architectures: The paper focuses on two Transformer models (RoBERTa and T5) but does not explore other popular Transformer architectures, such as BERT or GPT. Investigating the performance of the proposed method on a broader range of Transformer models could provide more insights into its applicability and effectiveness.

Author response:
Our focus is primarily on the two main training methods of Transformers, where RoBERTa represents self-supervised training, and T5 represents autoregressive training—both being mainstream and comprehensive paradigms. We did not consider structural diversity extensively because Transformers have several structural variants (BERT, GPT, T5, BART, etc.). The feasibility of incorporating experiments with other Transformer models will be discussed further.

Reviewer comment (Experiments):
In Table 3, I am curious about the performance of Fedavg + random sample + adv. sample. I suspect that the improvement of including a target sample in MixSample is negelactble.

Author response:
**Q: In Table 3, I am curious about the performance of Fedavg + random sample + adv. sample. I suspect that the improvement of including a target sample in MixSample is negelactble.**

A: We are pleased to conduct this experiment! Here are the experimental results (averaged over 10 runs):

FedAvg + Random Sample + Target Sample = 36.8232138 (up 1.87)

Based on these experimental results, it is evident that the contribution of the target sample is not negligible. We posit that the majority of the sample points obtained from target sample and adv sample sampling are not entirely redundant.

### Metrics for All Tasks:

- AX-b  : 0.04466032370826787
- AX-g  : 0.5120786516853932
- BoolQ  : 0.3782874617737003
- CB  : 0.034226190476190466
- COPA  : 0.55
- MultiRC  : 0.26065406168654304
- ReCoRD  : 0.3278020452273851
- RTE  : 0.5252707581227437
- WiC  : 0.5042319749216301
- WSC  : 0.36538461538461525

**Scores** : 0.36823213844910097

Reviewer comment (Novelty):
The challenge addressed in this paper may not be comprehensive. Although some papers utilize GANs to generate data for model distillation, it's important to note that GANs are not the sole method for data generation. Therefore, the scope of this paper appears to be limited.

Author response:
Thank you for your valuable feedback and thoughtful review of our manuscript.

We acknowledge that our discussion regarding GANs and motivation was not sufficiently clear. However, we persist in asserting the scarcity and necessity of research on non-text distillation in FL under NLP, and we present clear evidence supporting our contributions. Please grant us another opportunity to articulate our motivation more clearly and provide a more lucid argument:

1. Previous Data-free distillation of FL research (e.g., FedGEN, FedFTG) based on GAN often relies on image tasks (e.g., MNIST, CIFAR100) as primary benchmarks, neglecting NLP tasks. Other studies involving NLP tasks, such as FedAUX and FedDF, typically circumvent GANs and opt for auxiliary data distillation. Given the substantial gap in research on text tasks within FL's data-free distillation frameworks, exploring this area is both valuable and meaningful.

2. A practical issue arises with GAN in text generation [1](2023 NIPS). It is widely acknowledged that a notable obstacle in NLP GANs is their inability to generate differentiable outputs, given the discrete nature of language models (words). This lack of differentiability hampers mainstream Federated Learning (FL) Data-Free distillation frameworks, such as FedGEN[6] and FedFTG[7]. These frameworks utilize GANs for target learning functions but are ineffective in transmitting errors back to the generator, rendering them unsuitable for generating synthetic text in NLP distillation tasks under federated learning.

3. Can alternative text generation models other than GANs be employed? Our answer is: the solution's complexity is too high. The reasons are as follows:

The current state-of-the-art text generation models, such as large Transformer models like GPT-4, exhibit the best performance. Using Transformer models for text generation necessitates training in a self-supervised manner, essentially requiring the model to memorize client-side text data.

However, memorizing client-side text data raises the following issues:

- Privacy concerns require intricate training mechanisms [2] and defense mechanisms [3] to ensure that the Transformer remembers this text while preserving privacy. Simultaneously, precautions must be taken to prevent attackers from reconstructing the text through pre-trained models [4]. This will increase both computation and communication overhead.

- In terms of computation, the deeper the generative model (such as BERT or GPT-like self-supervised models), the more client-side text data it can remember. However, this may lead to computational overhead beyond the original task model, imposing a greater computational burden on clients.

- In terms of communication overhead, the basemodel Transformer for the original task already consumes a substantial amount of communication, as seen in FedKD [9], where the use of a shallow model like TinyBERT incurs 0.17-1.03GB per mobile client. Incorporating language generation models, as in FedGEN, will further increase communication overhead.

In summary, due to the escalated privacy, computational, and communication costs associated with incorporating text generation models into the FedGEN framework, we do not intend to address these issues. Instead, we propose an approach that originates from the Transformer itself, offering a method for distilling pseudo-data to Transformer models without the need for GANs. The effectiveness of this method surpasses that of using auxiliary data, as seen in FedAUX and FedDF, where the avoidance of GANs is also a deliberate choice.

Reviewer comment (Experiments):
The assertion that "The GANs approach faces numerous challenges in recent popular large-scale Transformer-based NLP tasks" prompts the question: Were the models employed in the experiments considered large-scale?

Author response:
**•Q: The assertion that "The GANs approach faces numerous challenges in recent popular large-scale Transformer-based NLP tasks" prompts the question: Were the models employed in the experiments considered large-scale?**

A:Taking reference from the papers on RoBERTa [9] and T5 [10-11], we utilized base models consistent with the original papers in our experimental section. The parameter sizes of these models are significantly larger, with RoBERTa having 102 million parameters [9], and T5 having 220 million parameters. These values far exceed the parameter sizes commonly used in federated learning, such as ResNet18 (11 million) and MLP, VGG, CNN (fewer parameters). Thus, we affirm that our experiments involve "large-scale" models.

Reviewer comment (Experiments):
Is this method applicable to other NLP tasks aside from text classification?

Author response:
**•Q: Is this method applicable to other NLP tasks aside from text classification?**

Yes, and not only limited to text classification tasks.

In our paper, we regrettably did not explicitly provide detailed information about SuperGLUE, and we acknowledge this oversight. Thank you to the reviewer for pointing out this aspect.

SuperGLUE encompasses a diverse set of text tasks beyond mere classification, including question answering, multiple-choice tasks, reading comprehension, and word sense disambiguation tasks [1-13]. It is a composite collection of tasks derived from various datasets, reflecting a combination of multiple tasks. The complexity of these tasks highlights the generality of our algorithm in addressing a wide range of natural language processing (NLP) challenges. For further details, please refer to: https://super.gluebenchmark.com/faq.

Reviewer comment (Evaluation):
In the abstract, it is mentioned, "This lightweight approach does not require GANs or auxiliary data, incurs no communication overhead, and yields improved model performance with relatively lower computational costs on the server side." However, there is no discussion of the savings in communication cost later in the paper.

Author response:
Thank you for your suggestions regarding communication overhead. The expression is not entirely clear to us. The absence of additional communication overhead refers to the comparison with the fundamental FedAVG approach, indicating the lack of additional communication costs.

We will incorporate an analysis of this aspect in the experimental section of the next version of the paper, as outlined below:

FedAUX necessitates the transmission of a substantial auxiliary dataset during the initial communication phase to compute the similarity between local and client datasets. Apart from model transmission, a score information matrix also needs to be communicated to clients in each communication round. This results in more communication overhead compared to FedAVG, encompassing both auxiliary data and similarity scores. Other non-data distillation solutions require the transmission of a generator based on FedAVG, thus adding to the communication overhead. FedAVG only requires the transmission of local client updates at time t during communication, and our algorithm's communication overhead is essentially equivalent to FedAVG (considering only label distribution information, with model costs being negligible).

Reviewer comment (Evaluation):
Since the difference in accuracy is quite moderate as compared to FedAUX for various values of \(\alpha\) in the Dirichlet distribution, it is necessary to see the savings in communication cost and the trade-off with additional computation cost at the server.

Author response:
Thank you for your suggestions regarding communication overhead. The expression is not entirely clear to us. The absence of additional communication overhead refers to the comparison with the fundamental FedAVG approach, indicating the lack of additional communication costs.

We will incorporate an analysis of this aspect in the experimental section of the next version of the paper, as outlined below:

FedAUX necessitates the transmission of a substantial auxiliary dataset during the initial communication phase to compute the similarity between local and client datasets. Apart from model transmission, a score information matrix also needs to be communicated to clients in each communication round. This results in more communication overhead compared to FedAVG, encompassing both auxiliary data and similarity scores. Other non-data distillation solutions require the transmission of a generator based on FedAVG, thus adding to the communication overhead. FedAVG only requires the transmission of local client updates at time t during communication, and our algorithm's communication overhead is essentially equivalent to FedAVG (considering only label distribution information, with model costs being negligible).

Reviewer comment (Presentation):
In the Ablation study, it is not clear which numbers in Table 1 should be compared with the accuracy numbers given in Table 3.

Author response:
We have noticed that we did not provide a detailed introduction to the SuperGLUE Benchmark, and we appreciate the reviewers for pointing this out.

Tables 1 and 3 do not directly correspond to each other for comparison. Table 1 discusses how we constructed experimental data distributions conforming to the federated learning scenario. The metrics in Table 3 represent computed scores across multiple datasets in the SuperGLUE benchmark. SuperGLUE comprises various text tasks, including but not limited to question answering, multiple-choice, reading comprehension, and word sense disambiguation tasks [1-13]. It is a composite task set consisting of diverse tasks and datasets. The complexity of these tasks demonstrates the broad applicability of our algorithm to NLP tasks. For more information, please refer to https://super.gluebenchmark.com/faq.

Reviewer comment (Evaluation):
It is unclear why we see a decaying performance difference between FedDRS and other techniques in Table 1 with increasing values of \(\alpha\).

Author response:
We need to elucidate the objective regularity of Federated Learning , namely that as $\alpha$ increases, the data distribution tends towards being independent and identically distributed (iid). However, in federated learning, a fundamental issue arises due to the non-iid nature of data, leading to a degradation in model performance.

Many research endeavors in federated learning aim to rectify non-iid scenarios into iid scenarios. In such cases, the evaluation criterion for federated learning models is not about surpassing a specific algorithm but rather focuses on whether the performance can approach iid conditions when $\alpha$ is small.

As $\alpha$ increases, the correctable gap diminishes, and the disparity between methods converges. Our FedDRS follows a similar pattern.