F ed NLP : Benchmarking Federated Learning Methods for Natural Language Processing Tasks (2022)

Abstract: Increasing concerns and regulations about data privacy and sparsity necessitate the study of privacy-preserving, decentralized learning methods for natural language processing (NLP) tasks. Federated learning (FL) provides promising approaches for a large number of clients (e.g., personal devices or organizations) to collaboratively learn a shared global model to benefit all clients while allowing users to keep their data locally. Despite interest in studying FL methods for NLP tasks, a systematic comparison and analysis is lacking in the literature. Herein, we present the FedNLP, a benchmarking framework for evaluating federated learning methods on four different task formulations: text classification, sequence tagging, question answering, and seq2seq. We propose a universal interface between Transformer-based language models (e.g., BERT, BART) and FL methods (e.g., FedAvg, FedOPT, etc.) under various non-IID partitioning strategies. Our extensive experiments with FedNLP provide empirical comparisons between FL methods and help us better understand the inherent challenges of this direction. The comprehensive analysis points to intriguing and exciting future research aimed at developing FL methods for NLP tasks.

## introduction fine-tuning large pre-trained language models (lms) such as bert @xcite often leads to state-of-the-art performance in many realistic nlp applications (e.g., text classification, named entity recognition, question answering, summarization, etc.), when large-scale, centralized training datasets are available. however, due to the increasing concerns and reguprivate local data (never exposed) federated models for an nlp task. ## federated learning for nlp in this section, we first introduce the background knowledge of federated learning (fl) in the con-text of nlp tasks. then, we illustrate a unified fl framework that we used to study typical fl algorithms. based on this, we build our research framework, a general pipeline for benchmarking and developing fl methods for nlp. ## benchmarking setup with fednlp in this section, we introduce the creation of our benchmark datasets from a set of chosen nlp tasks with different non-iid partition methods. we evaluate various fl methods on these datasets. ## experimental results and analysis in this section, we aim to analyze typical federated learning methods (introduced in our benchmark datasets with multiple dimensions with the base nlp models listed previously. we put more implementation details and additional results in appendix. we organize our extensive experimental results and findings from the analysis as a collection of research questions with answers. ## conclusion and future directions our key contribution is providing a thorough and insightful empirical analysis of existing federated learning algorithms in the context of nlp models. notably, we compare typical fl methods for four nlp task formulations under multiple non-iid data partitions. our findings reveal both promise and the challenges of fl for nlp. in addition, we also provide a suite of resources to support future research in fl for nlp (e.g., a unifying framework for connecting transformer models with popular fl methods and different non-iid partition strategies). thus, we believe our wellmaintained open-source codebase to support future work in this area. promising future directions in fl for nlp include: 1) minimizing the performance gap, 2) improving the system efficiency and scalability, 3) trustworthy and privacy-preserving nlp, 4) personalized fl methods for nlp, etc. (please see appendix e for more details.)