F ed PET uning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models (2023)

Abstract: With increasing concerns about data privacy, there is an increasing necessity of fine-tuning pre-trained language models (PLMs) for adapting to downstream tasks located in end-user devices or local clients without transmitting data to the central server. This urgent necessity therefore calls the research of investigating federated learning (FL) for PLMs. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we investigate the parameter-efficient tuning (PETuning) of PLMs and develop a corresponding federated benchmark for four representative PETuning methods, dubbed FedPETuning. Specifically, FedPETuning provides the first holistic empirical study of representative PLMs tuning methods in FL, covering privacy attacks, performance comparisons, and resource-constrained analysis. Intensive experimental results have indicated that FedPETuning can efficiently defend against privacy attacks and maintains acceptable performance with reducing heavy resource consumption. The open-source code and data are available at https://github.com/SMILELab-FL/FedPETuning .

## introduction pre-trained language models (plms), such as bert @xcite and roberta @xcite , have demonstrated exceptional performance on a multitude of natural language processing (nlp) benchmarks (e.g., glue @xcite ). consequently, plms have become de facto backbones for real-world applications. generally, in most real-world nlp applications, plms are centrally fine-tuned on the enormous quantity of data collected from individual customers, small businesses, or large enterprises @xcite . however, with the rising privacy concerns and the enactment of data protection laws foot_0 , enterprises or institutions are not allowed to collect data from end devices or local clients to a centralized server for fine-tuning plms. to break this barrier, federated learning @xcite (fl) has emerged as a privacy-aware technique designed to collaboratively train models without transmitting the numerous end-user or client data to a centralized place. in fl, the decentralized clients only need to periodically compute and send model information (i.e., parameters or gradients) to a server which is responsible for aggregating them to produce a global model. with the notion of privacy preserving, fl is appealing for privacy-sensitive nlp applications @xcite , as in the case of healthcare @xcite , finance @xcite , and mobile keyboard @xcite . although fine-tuning plms in fl, namely fedft, presents promising opportunities, there are two significant challenges that cannot be overlooked, including (1) communication overhead in the fl system, and (2) computational and storage costs for local clients. fine-tuning such plms usually requires distributed clients and services highfrequently exchange model gradients or parameters which are usually in the scale of millions or even billions. the limited communication bandwidth foot_1 in the fl system may cause excessive delays during frequent uploading and downloading phases of the federated fine-tuning procedure. meanwhile, it is often impractical for local clients to fine-tune the entire plms because of their limited computing resources. moreover, fully fine-tuning plms is extremely memory-intensive when a local client wants to store different instances for different tasks @xcite . as such, it is imperative to explore suitable plms-empowered fl methods under resource constraints (i.e., communication, parameters adaption, and storage). to this end, we investigate parameter-efficient tuning (petuning) methods of plms under the fl setting. petuning methods, such as adapter tuning @xcite , prefix tuning (li and liang, 2021), lora @xcite , @xcite , freeze most parameters of plms and update only a few additional parameters or a part of the original model parameters for downstream tasks @xcite . these properties make petuning methods potentially appealing to satisfy the resource constraints since local clients just need to tune lightweight parameters and communicate with the server for updates. nevertheless, it is crucial to address pertinent concerns. first, as the performance of federated models is greatly affected by the data heterogeneity @xcite among clients, it is not yet known that petuning methods can achieve acceptable performance in fl with such challenging data heterogeneity. second, as private data could be recovered from model gradients uploaded by clients via gradient inversion attacks @xcite , it is unclear whether petuning methods that upload only partial parameters of the entire model can resist gradient inversion attacks. to address these concerns, we present the framework of federated parameter-efficient tuning (named fedpetuning for short), as illustrated in figure 1 , and conduct in-depth experiments of various petuning methods under the fl setting, measuring privacy-preserving capability, performance, and resource costs. intensive experimental results on the glue benchmark reveal that (1) fedpetuning can reduce the considerable resource costs in fl settings while still achieving acceptable performance (e.g., fedap reduces 97.4% of communication with only 0.7% performance degradation compared with fedft) as shown in our research findings demonstrate the potential of combining large plms with fl, providing a promising training paradigm for privacypreserving learning in the era of large language models @xcite . ## federated parameter-efficient tuning in this section, we present how different petuning methods work, followed by the training process of fedpetuning. ## experiments in this section, we conduct extensive experiments for evaluating the performance of petuning in the fl setting, covering privacy attacks (see section 4.2), performance comparisons (see section 4.3), and resource-constrained analysis (see section 4.4). besides, we also provide an in-depth ablation analysis of fedpetuning in terms of data heterogeneity (see section 4.5), local training epochs (see section 4.6), and different fl scenarios (see section 4.7). ## conclusion this paper investigates parameter-efficient tuning methods of plms in the fl setting with extensive experiments for in-depth measurement of these methods under fl settings, covering privacy attacks, performance comparisons, and resourceconstrained analysis. experimental results unveil that fedpetuning can (1) achieve acceptable performance while reducing the colossal communication overhead and local storage cost, and (2) provide strong privacy-preserving capacity under different fl settings. to facilitate fl-tailored petuning research, we have released our code and partitioned datasets, aiming to facilitate the use of fed-petuning and inspire the broader community to develop more suitable petuning methods in future federated learning research.