Less is Fed More: Sparsity Reduces Feature Distortion in Federated Learning (2024)

Abstract: Our work studies Multilingual Federated Learning (FL), a decentralized paradigm that, although promising, grapples with issues such as client drift and suboptimal generalization in diverse, multilingual settings. We highlight limitations in existing approaches to generalize across both actively participating and inactive client language pairs. To mitigate these challenges, we introduce FedSparseNet, which incorporates sparse-network training, and LoRA, based on Low-Rank Adaptation. These approaches maintain the model’s fidelity to its pretraining distribution, thereby ensuring robust performance on both seen and unseen language pairs, while simultaneously enhancing communication efficiency by selectively transmitting trainable parameters. Our empirical evaluations demonstrate that FedSparseNet outperforms conventional FL models on both seen and unseen clients, while LoRA shows remarkable improvements in unseen client performance. Additionally, we propose the Continuous Relative Robustness Metric, a novel metric to uniformly assess a model’s performance across diverse language pairs. We open-source our code for reproducibility on GitHub.

## introduction the development of nlp applications capable of leveraging multilingual, multi-source, heterogeneous data while safeguarding user privacy is essential @xcite . @xcite addresses this by facilitating the utilization of personally identifiable information within a decentralized framework, thereby obviating the need for direct data sharing among clients. however, fl faces challenges such as client drift and suboptimal generalization in heterogeneous environments @xcite . furthermore, multilingual fl not only contends with these flspecific optimization difficulties but also grapples 1 https://github.com/aetherprior/less-is-fed-more with the complexities of extending to low-resource languages. this can hinder the accessibility of language technologies for various communities and intensify systemic biases @xcite . while there is extensive research on fl for nlp, studies specifically addressing multilingual fl translation remain limited, with minimal exploration of how fl impacts the training process. multilingual fl is an inherently heterogeneous data setting, offering a unique area of interest within the fl community. the closest work is @xcite , where the authors investigate federated multilingual translation. the study involves fine-tuning and communicating the entire parameter set of a 418m m2m encoder-decoder model. their findings suggest that fine-tuning a pre-trained model using fl can achieve comparable results to centralized learning, even in non-iid settings with clients segmented by language. in our research, we challenge the prevailing narrative that communicating all parameters in a multilingual translation model is viable for practical translation tasks. we argue that this approach is largely impractical. moreover, translation applications require the server model to not only generalize to client language pairs actively involved in fl but also to maintain pretraining performance on unseen language pairs or inactive clients. our findings reveal that baseline performance for unseen language pairs declines when fine-tuning with active client data. this issue stems from the distortion of pretrained features @xcite , a problem not adequately addressed by current fl approaches, especially in the context of nlp tasks like translation. to address the challenges identified, our approach builds on the current literature on parameter efficient finetuning (peft) to: a) ensure the model remains close to its pretraining distribution, facilitating balanced generalization across both seen and unseen language pairs, and b) enhance federated fine-tuning and communication efficiency by transmitting only a sparse subset of trainable parameters. our contributions include: fedsparsenet (fig. 1 ) consists of two stages on the client. let i denote the i-th round of training and θ (i) , the server model parameters at round (i). fedsparsenet enhances communication efficiency by minimizing data transmission which is often about 1% of the client parameters. the modular design allows for effective composability, reducing knowledge overlap and interference among the client languages. sparsity also serves as a natural form of regularization, making these networks less prone to overfitting, and helping the model retain generalization properties of the pretrained model on unseen data. sparse networks also have other advantages: it does not introduce additional parameters like the adapter @xcite , thereby not reducing inference speed; and the model architecture remains identical to the pretrained model, simplifying code development and ensuring the method is model-agnostic. ## results lora in ## conclusion motivated by the need to improve generalization in fl for unseen client data, we introduce fedsparsenet and lorafed. these methods focus on sparsifying the client parameter space, addressing the challenge of pretrained feature distortion due to seen-client optimization, and enhancing communication efficiency.