Paper: Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning (ICLR-2024)

Reviewer comment (Experiments):
While the "Contrast" part of the proposed method is focused around preserving privacy, it would have been nice to have some verification of this, as privacy is absolutely central to the proposed model. The reviewer does not work in privacy+fairness, so they do not know if leveraging contrastive learning is sufficient for protecting privacy.

Author response:
Thank you for the comments! 
- The main purpose of the "Contrast" part is to group clients based on their local objectives rather than preserving privacy. It is used to encourage clients to exchange useful downstream task knowledge by federated aggregation. 
- For privacy protection, the proposed ATC provides fundamental privacy protection (no sharing of data directly) with the help of the federated learning framework. Besides, privacy protection can be enhanced by privacy protection algorithms (such as differential privacy, homomorphic encryption, etc.) to satisfy flexible protection requirements from different applications. Users can balance utility-privacy-cost according. These privacy protection algorithms can be regarded as plugins and are orthogonal to the proposed framework, thus in this submission, we pay more attention to describing how can collaborate clients with different learning objectives.

We have made the above discussions more clear in the revised paper, thank you again!

Reviewer comment (Writing):
In the Introduction, the paper introduces the need for improved federated learning by essentially complaining that increased data protections hurt "data owners". This framing does not sound great to the reviewer, who expresses gratitude for protections on personal data that keep companies from using it in risky ways. The reviewer encourages a slight re-framing of the introduction so that end-users' rights for data privacy are not framed as a burden. They note that the rest of the paper is much better in terms of framing for privacy protection.

Author response:
We sincerely appreciate your valuable suggestions! 

We absolutely agree that user private data should be protected and some actions should be taken to keep companies from using personal data in risky ways! Our submission explores a practical solution for training models in a privacy-preserving manner. 
We have modified the introduction according to your suggestions to make it precise and friendly. Thank you very much!

Reviewer comment (Evaluation):
No statistical significance testing conducted for the results that appear quite incremental compared to the baselines.

Author response:
Thank you for the suggestions! We have added the statistical significance testing in the revised paper in Section 4.4. Specifically, we conduct an independent t-test between the results of the proposed method and PerCFL (the strongest baseline) to show the significance and get $p< 0.01$.

Reviewer comment (Writing):
In the introduction, kindly provide a concrete example/use case for federated learning with heterogeneous tasks.

Author response:
Thank you very much for the suggestion! Federated learning with heterogeneous tasks has many practical use cases. For example, multiple companies may collect text data from similar domains, such as news, but utilize these data for different downstream tasks, such as event discovery, sentiment analysis, and so on. With the help of federated learning and the proposed framework, these companies can collaboratively train a global model to learn valuable and generalized knowledge from text data.

We have added the example in the revised paper, thank you again!

Reviewer comment (Writing):
Section 3.1: Shed Lighted -> Based on such insights.

Author response:
Thanks for your suggestion! We have modified it accordingly.

Reviewer comment (Reproducibility):
Section 3.2 - Extension with Clustering Algorithms: The gradients change with each iteration, so can the clusters change as well, or is it fixed based on the first iteration?

Author response:
Thanks for your comment! The clusters might change as the gradients change with each iteration.

Reviewer comment (Reproducibility):
Section 3.2 - Extension with Clustering Algorithms - Kindly extend upon the clustering algorithm - provide more details.

Author response:
Thank you for the comments. The clustering algorithm used in this study is an agglomerative clustering algorithm proposed by [1], which serves as an example to demonstrate the extensibility of our framework. In fact, more advanced clustering algorithms can be adopted in the assign training stage. We kindly encourage readers to concentrate on the overall architecture of the framework and refer to the references for further details if necessary. Thank you again!

Reviewer comment (Writing):
Section 3.3 - 4th paragraph - “To further..” is confusing. Kindly provide concrete examples to highlight this important stage.

Author response:
Thank you for the comments! In section 3.3 - 4th paragraph, we introduce a synthetic data generation approach for contrastive learning. This approach is particularly valuable in scenarios where the proposed framework is applied to downstream tasks involving sensitive data (e.g., healthcare) or non-IID data. It is because finding an eligible public dataset can be intractable.

We have made them more clear accordingly in the revised paper, thanks again!

Reviewer comment (Writing):
Section 3.3 - Mention that the contrastive learning loss is one of the additional objective other than the client - task specific objective.

Author response:
Thank you for the suggestions! We have modified it accordingly in Section 3.3 for a better understanding of the contrastive learning loss.

Reviewer comment (Reproducibility):
Section 4.2: In FEDAVG what equations govern the federated aggregation? In FEDPROX what is the proximal term.

Author response:
Thank you for the comments!

- In FEDAVG, the equations of federated aggregation can be given as $\boldsymbol{w}^{(t+1)} \leftarrow \boldsymbol{w}^{(t)} + \sum _{i \in [N]} \frac{|D _i|}{\sum _{j\in [N]}|D _j|} \Delta \boldsymbol{w}^{(t)} _{i}$ 
- In FEDPROX, the proximal term can be given as $\frac{\mu}{2}||\boldsymbol{w} - \boldsymbol{w}^{(t)}||^2$ (please refer to the algorithm 2 in [2] for more details).

Reviewer comment (Evaluation):
Table 2: Are these numbers statistically significant? Kindly conduct a significance testing.

Author response:
Thank you for the suggestions! We have added the statistical significance testing in the revised paper in Section 4.4.

Reviewer comment (Presentation):
In the related work section, please highlight the importance of ATC for each of the different previous works rather than clubbing them in the end.

Author response:
Thank you for the suggestions! We have modified the related work section accordingly for better highlight the importance of ATC.

Reviewer comment (Presentation):
In the conclusion section, kindly mention quantitative results, instead of writing the abstract again.

Author response:
Thank you for the suggestions! We have provided more quantitative results in the conclusion section.

Reviewer comment (Theory):
It is still constrained to the global tasks: depending on the server to assign a global task (meaning that participants need to abide by this global task or has to be intrinsically similar).

Author response:
Thank you for the comments! In this study, we have carefully selected global tasks (i.e., self-supervised language model pretraining tasks) with the intention of benefiting clients with heterogeneous NLP tasks, while avoiding any additional potential privacy concerns. These global tasks have been proven to be general enough when focusing on the field of NLP, and would not cause additional constraints.

Reviewer comment (Novelty):
Some novelty is overclaimed, e.g., FL + multi-task [1] (or heterogeneous tasks in the context of this paper); there is no comparison with this baseline.

Author response:
Thank you very much for the suggestion regarding the references. 
- We notice that the mentioned paper primarily focuses on addressing issues of high communication cost, stragglers, and fault tolerance in general distributed multi-task learning. However, it may not be effectively applicable to collaboration heterogeneous NLP tasks without careful design, particularly in the case of NLU and NLG tasks. We have accordingly included the references and added the discussions, thanks again!
- We provide the discussions on the comparison between the suggested FL + multi-task framework and the proposed method in the related work section. Besides, in the experiments section, we provide empirical comparisons between the proposed method with the baseline[2] that adopts a multi-task framework, highlighting the advances of the proposed methods.

Reviewer comment (Evaluation):
There is a concern about whether it is worth the margins of improvement provided by Contrastive Learning in the trade of privacy and communication cost.

Author response:
Thank you for the comments! 
- In the proposed ATC framework, the contrast training stage is a **critical step that builds upon the assign training stage and complements it**. In the assign training stage, clients perform local training with the assigned self-supervised objectives, and their model updates are more related to their data domains; While in the contrast part, clients train with their own objectives, thus their model updates are more specific to their downstream tasks. It is worth noting that different clients can benefit from the knowledge shared by others who have similar data domains or similar downstream tasks.
- The proposed ATC provides fundamental privacy protection (no sharing of data directly) with the help of the federated learning framework. Meanwhile, privacy protection can be enhanced by privacy protection algorithms (such as differential privacy, homomorphic encryption, etc.) to satisfy flexible protection requirements from different applications. Users can balance utility-privacy-cost according. 
- The willingness of clients' participation is a really interesting and meaningful topic within the context of FL. We would like to take some exploration of incentive mechanisms in our future work. Thanks again for your valuable suggestions!

We have added the above discussions and made them more clear in the revised paper, thank you again!

Reviewer comment (Evaluation):
Although there is a marginal improvement it is stated "superior performance". Consider lowering the tone.

Author response:
Thanks for your comments. The proposed method shows a significant improvement (around 1.5% compared to the strongest baseline) on the overall performance. We have made the description more clear in the revised paper.

Reviewer comment (Writing):
Many writing issues (see questions).

Author response:
Thank you very much for the suggestions regarding the paper writing. We have addressed these issues accordingly in the revised paper, and also made carefully polished to further improve the writing quality.

Reviewer comment (Experiments):
How will this framework tackle backdoor attacks?

Author response:
Thanks a lot for your comments. The proposed framework can be enhanced by various approaches to tackle backdoor attacks, according to the user's requirements of balancing utility and privacy. On the one hand, inspired by previous study[3], we can utilize some simple yet effective approaches, such as norm clipping and differential privacy, to mitigate the backdoor attacks without hurting the overall performance. On the other hand, robust aggregation rules[4,5] can be applied to filter out malicious model updates.

We have added the above discussions in the revised paper, thank you again!

Reviewer comment (Writing):
Non-IID could have more discussion, especially the Non-IID introduced by multiple tasks and datasets.

Author response:
Thank you very much for your suggestions! The Non-IIDness discussed in the context of FL can be defined as *covariate shift* and *concept shift* [6]: the marginal distributions $\Pr(X)$ or the conditional distribution $\Pr(X|Y)$ can vary across clients. The Non-IIDness becomes more complicated and mixed when it comes to heterogeneous tasks, where clients might have different yet related feature and label spaces.

Reviewer comment (Writing):
Writing issue: "an FL" -> "a FL".

Author response:
Thank you very much for the suggestions regarding the paper writing. We have addressed these issues accordingly in the revised paper, and also made carefully polished to further improve the writing quality.

Reviewer comment (Writing):
Writing issue: "unallowable"?

Author response:
Thank you very much for the suggestions regarding the paper writing. We have addressed these issues accordingly in the revised paper, and also made carefully polished to further improve the writing quality.

Reviewer comment (Writing):
Writing issue: FL (acronym): once introduced you should stick to it.

Author response:
Thank you very much for the suggestions regarding the paper writing. We have addressed these issues accordingly in the revised paper, and also made carefully polished to further improve the writing quality.

Reviewer comment (Evaluation):
Looking at table 2 showed the proposed framework in the top performing range among the competitors, but the improvement in many cases did not seem significant compared to the state of the art. For eg, the Squad and MSQG task top performer had results just 2-3 points below the proposed framework. For all other tasks, the results of all top performers were not significantly different.

Author response:
Thanks a lot for your comment. This study focuses on the collaboration of heterogeneous NLP Tasks, making it crucial for the proposed method to **achieve better overall performance across multiple tasks**, rather than just being competitive in one or a few tasks. As demonstrated in Table 2, the proposed method shows a significant improvement (around 1.5% compared to the strongest baseline) on the overall performance. For more details: 
- While some baseline methods may achieve comparable performance in one or a few tasks, they fail in other tasks and consequently obtain lower overall scores, as shown in Table 2. For example, PerCFL (the strongest baseline) performs similarly to our proposed method on AGNews/NewsQA, but causes a significant performance drop of approximately 5%/2% on SQuAD/IMDB. 
- In comparison to FedAvg-ST (which constructs multiple FL courses using FedAvg independently, with each FL course only involving clients with the same or similar tasks), the proposed method consistently achieves improvement. These results demonstrate the effectiveness of our method in collaborating heterogeneous NLP tasks.