F ed LFC : Towards Efficient Federated Multilingual Modeling with L o RA -based Language Family Clustering (2024)

Abstract: Federated Multilingual Modeling (FMM) plays a crucial role in the applications of natural language processing due to the increasing diversity of languages and the growing demand for data privacy. However, FMM faces limitations stemming from (1) the substantial communication costs in networking and (2) the conflicts arising from parameter interference between different languages. To address these challenges, we introduce a communication-efficient federated learning framework with low-rank adaptation and language family clustering for Multilingual Modeling (MM). In this framework, we maintain the weights of the base model, exclusively updating the lightweight Low-rank adaptation (LoRA) parameters to minimize communication costs. Additionally, we mitigate parameter conflicts by grouping languages based on their language family affiliations, as opposed to aggregating all LoRA parameters. Experiments demonstrate that our proposed model not only surpasses the baseline models in performance but also reduces the communication overhead. Our code is available at https://github.com/zhihan-guo/FedLFC.

## introduction multilingual modeling is increasingly important in natural language processing (nlp) as a result of the growing diversity of languages used online @xcite . however, gathering multilingual data can prove prohibitively expensive due to its distributed nature and data privacy concerns @xcite . to address this challenge, federated learning (fl) is employed to train a multilingual model across various institutions and data sources @xcite @xcite . the fundamental concept of fl revolves around the exchange of model parameters rather than the transmission of sensitive data, thereby preserving data privacy @xcite . nevertheless, traditional fl frameworks encounter two primary challenges in the context of federated multilingual modeling (fmm), as illustrated in figure 1: (1) huge communication cost: the acquisition of multilingual knowledge necessitates the expansion of pre-trained language models (plms), substantially increasing communication costs due to the extensive model parameters required to be transferred across clients @xcite . (2) parameter conflicts: fmm naturally encounters non-iid (non-independently and identically distributed) issues @xcite , exemplified by significant distribution shifts between languages with diverse linguistic systems and cultures, such as english and chinese. employing a single model for multiple language tasks can negatively affect performance @xcite due to conflicting optimizations @xcite . to address the aforementioned challenges, we introduce fedlfc, a communication-efficient framework for fmm that incorporates low-rank adaptation (lora) and language family clustering (lfc), as depicted in figure 2 . federated fine-tuning with low-rank adaption: inspired by the success of parameter-efficient fine-tuning (peft) @xcite @xcite , fedlfc leverages lora to fine-tune a concise set of parameters, thereby preserving the majority of original plms' parameters unchanged. this strategy significantly reduces the communication overhead of fl, marking, to our knowledge, the first application of lora within the fl context. language family clustering: to alleviate the interference between different languages, we employ language family clustering (lfc), grouping languages based on their familial ties, as shown in figure 3 . this strategy involves both clients and servers in maintaining and separately optimizing a set of lora parameters for each language cluster. extensive evaluations across three language tasks, i.e. language modeling, machine translation, and text classification, demonstrate that fedlfc not only outperforms a variety of baseline methods in performance but also achieves a significant reduction in communication overhead, e.g., reducing training parameters by a factor of 100 compared to traditional full-finetuning approaches. ## experiment tasks and datasets. we evaluate our model in three takes i.e., language modeling (lm), machine translation (mt), and text classification (tc) using four datasets i.e., europarl, mtnt, un corpus, and news classification. the statistics of each dataset are shown in baselines. we perform the experiment on three different settings i.e., centralized model, fedavg, and standalone. the centralized model employs centralized training @xcite , where all data is collected in one place. fedavg employs federated averaging @xcite training within the federated learning framework, dividing data across different clients. both of them train a conventional multilingual model with all parameters. standalone setting trains data exclusively in one language and tests its performance across all languages, demonstrating a scenario where a model is trained using data from a single client @xcite . to show the superiority of lfc and lora, we further freeze parameters of plms in the setting of centralized and fedavg. we train lora @xcite and typical adapter @xcite without lfc.