Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter (2023)

Abstract: Federated Multilingual Neural Machine Translation (Fed-MNMT) has emerged as a promising paradigm for institutions with limited language resources. This approach allows multiple institutions to act as clients and train a unified model through model synchronization, rather than collecting sensitive data for centralized training. This significantly reduces the cost of corpus collection and preserves data privacy. However, as pre-trained language models (PLMs) continue to increase in size, the communication cost for transmitting parameters during synchronization has become a training speed bottleneck. In this paper, we propose a communication-efficient Fed-MNMT framework that addresses this issue by keeping PLMs frozen and only transferring lightweight adapter modules between clients. Since different language pairs exhibit substantial discrepancies in data distributions, adapter parameters of clients may conflict with each other. To tackle this, we explore various clustering strategies to group parameters for integration and mitigate the negative effects of conflicting parameters. Experimental results demonstrate that our framework reduces communication cost by over 98% while achieving similar or even better performance compared to competitive baselines. Further analysis reveals that clustering strategies effectively solve the problem of linguistic discrepancy and pruning adapter modules further improves communication efficiency.

## introduction federated learning (fl) @xcite provides a new training framework utilizing data from various clients without privacy leakage. in fl, the server receives models from clients trained with their local data and aggregates all parameters it has received to acquire a global model, and then 1 our code is available at https://github.com/ lancopku/fedmnmt sends it back to all clients to start the next training round. this characteristic enables fl to get widely applied in real-world scenarios @xcite @xcite . in recent years, federated multilingual neural machine translation (fed-mnmt) has become a new training paradigm and making it feasible for most institutions to train mnmt models @xcite . fl makes it possible to leverage corpora from other organizations without privacy problems, solving the problem that training an mnmt model needs to collect large-scale multilingual corpora, which is expensive, time-consuming, and often unaffordable for resource-constrained institutions. therefore, fed-mnmt is a secure and cost-effective alternative to conventional centralized training for the optimization of mnmt models. however, the issue of communication cost is non-negligible when we introduce fl to neural machine translation. unlike local centralized learning, federated learning requires frequent communication of model parameters between the server and clients. therefore, the communication cost grows rapidly along with the increase in model size. nowadays, pre-trained language models are widely adopted as backbone models for mnmt, whose parameters are usually over 10 8 , e.g., 611m for mbart-50 @xcite and 1.2b for m2m100 @xcite . considering the increasing number of clients in realistic scenarios as there frequently appear new clients, communication costs will severely hinder the efficient training of the entire fed-mnmt system and thus make the application of fl to mnmt impractical. to tackle this problem, we introduce the parameter-efficient tuning idea @xcite @xcite into fed-mnmt. specifically, we focus on adapter @xcite , which is a popular technique for efficient tuning that requires only updating a lightweight adapter module. in the training process of adapters, a number of randomly initialized modules are inserted into the backbone models and fine-tuned on new data. concretely, only the parameters of these modules are updated during training, so the number of parameters needed to be transferred between the server and clients is substantially reduced. as the communication cost before and after introducing adapter illustrated in figure 1 , this approach significantly saves communication costs and enables practical applications of fed-mnmt. however, directly adding adapter modules to nmt models results in a performance decline, which is initially observed by @xcite and also confirmed by our experimental results. this phenomenon is attributed to the divergence of different language pairs. in fed-mnmt, corpora in diverse languages from different clients are not independently and identically distributed (non-i.i.d.), so directly aggregating parameters from clients leads to a decrease in model's performance @xcite . considering the adverse effect of conflicting parameters from diverse languages in fed-mnmt, we introduce clustering strategies to alleviate this issue. the core idea is to cluster the samples according to the characteristics of their data and only conduct aggregation within each cluster where samples share similar properties. specifically, we cluster all clients with different language pairs based on the language family, gradient similarity, and random, respectively, and systematically compare the performance of different clustering strategies on multilingual translation benchmarks. figure 2 gives a general view of our training framework. our experimental results show that clustering on adapters alleviates the data non-i.i.d. problem and yields better performance in most cases. overall, our work opens a new direction for future improvements on fed-mnmt in the real world. in conclusion, our primary contributions can be summarized as follows: ## related work federated learning was first proposed by @xcite as a decentralized training framework. due to its decentralized and private nature, fl shows great potential in actual applications. recently, there has been a surge in the nlp community to explore the application of federated learning in diverse nlp tasks, such as emojis prediciton @xcite , named entity recognition @xcite , and machine translation @xcite @xcite first applied fl to nmt tasks. however, training language models in the fl setting brings huge communication overheads. to solve this problem, researchers have proposed to only exchange some dedicated "controller" @xcite layers between the server and clients. moreover, @xcite introduced parameter pruning strategies to reduce communication bandwidth. our methods with adapter modules have advantages in communication efficiency with fewer parameters to be transferred compared to controller (see appendix d), and other parameter pruning strategies can also be applied to our adapter modules to further reduce communication costs. ## conclusion in this paper, we introduce adapter modules to plms for the fed-mnmt problem to boost communication efficiency. we reduce the communication cost by over 98% and make the training process of fed-mnmt practical. to deal with the problem of performance drop after introducing adapter modules, we propose different clustering strategies to separate clients into different groups to avoid the negative influence of data discrepancy. we surpass the direct baseline with a substantial gap, especially in the more complicated multi-tomulti translation setting. furthermore, our analytic experiments indicate that both aggregation algorithms in server and clustering strategies affect the performance of fed-mnmt. we also explore the possibility of further reducing communication costs by pruning adapter modules and find that adapters in top layers are more significant for translation performance. in future work, we will explore more welldesigned clustering strategies and attach other parameter-efficient techniques to adapter to further reduce the parameters to be transferred.