Tunable Soft Prompts are Messengers in Federated Learning (2023)

Abstract: Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources, alleviating privacy concerns that arise from directly sharing local data. However, the lack of model privacy protection in FL becomes an unneglectable challenge, especially when people want to federally finetune models based on a proprietary large language model. In this study, we propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts. These soft prompts, updated and transmitted between the server and clients, assume the role of the global model parameters and serve as messengers to deliver useful knowledge from the local data and global model. As the global model itself is not required to be shared and the local training is conducted based on an auxiliary model with fewer parameters than the global model, the proposed approach provides protection for the global model while reducing communication and computation costs in FL. Extensive experiments show the effectiveness of the proposed approach compared to several baselines. We have released the source code at https://github.com/alibaba/FederatedScope/tree/fedsp/federatedscope/nlp/fedsp.

## introduction large language models (llms) @xcite @xcite @xcite @xcite have been witnessed incredible progress in the recent years, which is tied to the support of large amounts of training corpus and computation resources. to further promote the development of llms and broaden their applications, how to make good use of private and decentralized data can be one of the critical steps, which brings both opportunities and challenges to federated learning (fl) @xcite @xcite . the main idea of fl is that multiple participants collectively train a global model using their local data, and the updates generated by each participant's local training process are then aggregated together to optimize the global model. on the one hand, fl offers a feasible solution for finetuning llms by utilizing multiple data sources without private data leakage. participants are allowed to conduct local training processes based on their private data, and then share the learned knowledge through the exchange of model updates. in this way, privacy concerns, raised by directly sharing private data, can be alleviated. however, on the other hand, the application of fl can be constrained by concerns regarding model privacy, especially when the finetuning process relies on proprietary llms. in particular, during each training round, the up-to-date global model would be distributed to participants, which goes against the interests of model owners and might be deemed unacceptable in real-world scenarios. such conflicts between protecting data privacy and model privacy are attributed to the training paradigm of fl, which involves sharing model parameters to accomplish the knowledge exchange process for collaboratively training models. to address this challenge, we propose fedsp, a novel fl training approach that leverages tunable soft prompts to enable the exchange of useful knowledge among participants, achieving both data privacy and model privacy protection at the same time. soft prompts can be regarded as some additional tunable parameters plugged into the model, which are updated to capture knowledge from downstream tasks while keeping the model parameters frozen in parameter-efficient finetuning (peft) al-gorithms @xcite @xcite @xcite . we incorporate these tunable soft prompts in fl, serving as messengers among participants to deliver knowledge learned from local data and contained in the global model. specifically, at the start of an fl course, a server broadcasts an auxiliary model (which typically has much fewer parameters than the global model) to participating clients. then in each training round, the server sends up-to-date tunable soft prompts to selected clients, and these clients combine the received soft prompts with their maintained auxiliary models. after that, each client performs local training to accomplish the process consisting of (i) global model alignment, in which clients update the auxiliary models to align them with the representative capabilities of the global model, using the up-to-date soft prompts; and (ii) local knowledge capturing, in which clients freeze their auxiliary models and finetune the soft prompts to capture useful knowledge from their local data. these updated soft prompts are sent back to the server once the local training is complete. the server is responsible for aggregating and optimizing these soft prompts to drive the next training round. in contrast to the existing fl training paradigm that entails sharing the parameters of the global model, the proposed fedsp suggests exchanging tunable soft prompts during training rounds, which ensures privacy protection for the global model. meanwhile, clients are only expected to update the auxiliary models and soft prompts in the local training process, which mitigates both computation and communication overhead compared to federally finetuning of large global models like llms. we conduct a series of experiments with two llms (gpt2-xl and opt-1.3b) on seven benchmarking datasets, showing that fedsp achieves competitive performance compared to baseline methods while significantly reducing the model size by 14.5×/8.5× on gpt2-xl/opt-1.3b. these experimental results demonstrate the effectiveness and advantages of the proposed idea that using tunable soft prompts as knowledge messengers in fl. ## preliminary because of the huge model size (billions of parameters), large language models (llms) @xcite @xcite @xcite are usually kept at the cloud server who owns adequate resources for inferring, finetuning, and training. nowadays, users of llms have to send the instructions to the cloud server and wait for the model-generated results, or upload their private data foot_0 for finetuning the llms on their downstream tasks. the server can also benefit from the user instructions and human feedbacks to continually improve the model performance. such usage of llms might bring privacy issues from two different perspectives. for the users of llms, they might not be allowed to upload their private data to the server, especially in some scenarios with highly sensitive information such as health care, so-called data privacy. for the cloud servers (i.e., the model owners), they tend to keep the proprietary language model private to ensure their interests, so-called model privacy. as a result, how to provide protection for both data privacy and model privacy is a practical and challenging problem when training, finetuning, and deploying llms in real-world applications. federated learning (fl) @xcite is one of the considered solutions for alleviating the aforementioned data privacy issue. different from the centralized training paradigm that needs to gather users' local data for training a global model, fl proposes to aggregate participants' model updates to avoid directly sharing the private data. to be more specific, at each training round, a server broadcasts the up-todate global model to the participating clients. each client updates the received global model based on its local data, and then sends the model updates back to the server for performing federated aggregation. formally, the federated aggregation at the t-th training round can be defined as: ## methodology in this section, we describe the details of the proposed fedsp. the intuition behind fedsp is to adopt some tunable soft prompts to replace the shared global model in the training process of fl, which serves as messengers to deliver useful knowledge in the local data and the global model to the server and clients, as introduced in section 3.1. with the help of these tunable soft prompts, clients are expected to perform local training for updating an auxiliary model and the soft prompts alternatively, and send the updated soft prompts to the servers for sharing the knowledge learned from their local data (more details in section 3.2). the overall architecture of the proposed fedsp is illustrated in figure 1 . ## related work parameter-efficient finetuning with the increasingly larger scale of the pre-trained large language models (llms), finetuning the entire model becomes unaffordable for many individual researchers. as an efficient adaptation of llms to new downstream tasks, parameter-efficient finetuning (peft) @xcite @xcite @xcite @xcite algorithms have emerged where only negligible proportions of parameters in the original llms are required to be updated. the main idea of peft is to add continuous soft prompts to the original model and only the parameters of the soft prompts need to be finetuned. for example, @xcite and @xcite propose to add a few continuous tunable parameters as soft prompts to the input word embeddings, and @xcite and @xcite suggest to prepend separate soft prompts to every model layer. federated learning in nlp in order to protect data privacy, federated learning (fl) has attracted a lot of attention from both academic and industrial @xcite @xcite . with more and more language assistance products being applied in real-world applications, fl has also increasingly appeared in the community of nlp to address the problem of privacy leakage, such as machine translation @xcite and question answering @xcite , and so on @xcite @xcite @xcite . besides data privacy, related works are also focused on the following challenges in the field of nlp: (i) data heterogeneity. for example, @xcite proposes a federated matching framework with a backbone-patch architecture to address the non-identical and independent distribution (non-iid) problem of the training data. @xcite propose to split the model into a local part and a global part to handle the non-iid issue. although fl can protect data privacy to some extent, it is incapable to cope with the situation when the server and clients need to protect their model privacy. in this paper, we make the first attempt to incorporate the idea of using tunable soft prompts as messengers to meet the requirement of protecting both data privacy and model privacy. ## conclusions in this paper, we propose a novel fl training approach, named fedsp, with tunable soft prompts as messengers to accomplish the knowledge delivery among participants. since the proposed fedsp does not need to share the global model, it provides the necessary privacy protection for the global model when finetuning llms. these soft prompts are broadcast to the clients at each training round, plugged into an auxiliary model, and used to capture the knowledge from local data. extensive experiments on various benchmarking datasets demonstrate the effectiveness of fedsp, showing that using tunable soft prompts as messengers in fl is able to protect both model privacy and data privacy, and achieve competitive performance with baselines and needs much less computation and communication costs.