Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models (2025)

Abstract: Parameter-efficient fine-tuning (PEFT) methods typically assume that Large Language Models (LLMs) are trained on data from a single device or client. However, real-world scenarios often require fine-tuning these models on private data distributed across multiple devices. Federated Learning (FL) offers an appealing solution by preserving user privacy, as sensitive data remains on local devices during training. Nonetheless, integrating PEFT methods into FL introduces two main challenges: communication overhead and data heterogeneity. In this paper, we introduce FedTT and FedTT+, methods for adapting LLMs by integrating tensorized adapters into client-side models’ encoder/decoder blocks. FedTT is versatile and can be applied to both cross-silo FL and large-scale cross-device FL. FedTT+, an extension of FedTT tailored for cross-silo FL, enhances robustness against data heterogeneity by adaptively freezing portions of tensor factors, further reducing the number of trainable parameters. Experiments on BERT and LLaMA models demonstrate that our proposed methods successfully address data heterogeneity challenges and perform on par or even better than existing federated PEFT approaches while achieving up to 10\times reduction in communication cost.

## introduction large language models (llms) such as gpt-4 @xcite , llama @xcite , and bert @xcite excel in tasks such as translation and summarization @xcite due to the capabilities of transformer architectures @xcite . while finetuning enhances adaptability @xcite , fully fine-tuning these massive models is computationally expensive and prone to overfitting. parameter-efficient fine-tuning (peft) methods such as adapters @xcite , prompt-tuning @xcite , and lora @xcite address this by optimizing only a subset of parameters, reducing costs without sacrificing performance @xcite . however, traditional peft assumes centralized data, whereas real-world applications often involve private, distributed data, such as medical or legal records @xcite @xcite . federated learning (fl) emerges as a compelling solution, as it prioritizes user data privacy by ensuring sensitive information remains on individual devices during training. instead of transmitting data to a central server, clients in fl locally update their model parameters and share only model information, such as parameters or gradients, which are then aggregated by the server into a global model @xcite . to address the aforementioned problems, fl has been integrated into peft methods @xcite @xcite . these methods, however, often result in two major challenges: (i) communication and computation overhead in the fl system, and (ii) significant accuracy degradation, particularly under heterogeneous scenarios. communication efficiency is crucial in fl, as edge devices typically have limited storage and computational power. while some methods attempt to reduce the number of training parameters by incorporating sparsity in peft approaches, they either perform poorly in fl @xcite or are computationally expensive for clients @xcite , rendering them unsuitable for practical scenarios. data heterogeneity happens when training data is not identically and independently distributed across clients @xcite . in such scenarios, local models on individual clients can diverge from the global model's optimal state, leading to slower conver-gence @xcite . this issue is particularly pronounced when training llms in federated environments, as data heterogeneity can severely impact model performance @xcite . although various studies have employed techniques such as gradient tracking to address this challenge in federated learning @xcite , these approaches often become inefficient in terms of communication and computational demands when applied to federated llms. several works have modified lora to improve its efficiency in highly heterogeneous federated settings @xcite . however, these methods require communicating a large number of parameters (at least as much as lora). in this work, we present a federated and tensorized framework that reduces communication load compared to other peft methods, while maintaining comparable or even superior accuracy, and addressing challenges such as data heterogeneity. we introduce the federated tensor train (fedtt) algorithm, where tensorized adapters serve as trainable parameters embedded within the encoder/decoder blocks of models used by clients. fedtt is applicable to both cross-silo fl, where all clients participate in training, and large-scale cross-device (lscd) fl, where only a subset of clients is selected in each round. for cross-silo fl, we propose the fedtt+ algorithm, a heterogeneityrobust method that further reduces the number of parameters by adaptively freezing portions of the tensor factors in the models. we summarize our contributions as follows: these experiments validate the effectiveness of our proposed algorithms in reducing communication overhead while preserving high accuracy. ## proposed algorithm in this section, we introduce the proposed fedtt method, which is built on a novel parameterefficient tensorized adapter. this adapter is integrated into clients' local models to effectively adapt llms. we begin with discussing the structure and setup of the tensorized adapters, followed by the introduction of fedtt and its enhanced version, fedtt+, which can be selected based on communication budget constraints. the workflow of our fedtt method is shown in fig. 1 (b) . ## conclusion this work presents a novel federated and tensorized fine-tuning framework, addressing the key challenges of communication overhead and data heterogeneity in fl. fedtt leverages tensorized adapters to significantly reduce communication costs while maintaining high performance, achieving up to a 10× reduction in trainable parameters compared to existing methods. additionally, fedtt+ enhances robustness in cross-silo fl by adaptively freezing portions of tensor factors, further optimizing parameter efficiency. extensive experiments with models like bert and llama-2 demonstrate the effectiveness of fedtt and fedtt+ in both crosssilo and lscd fl settings, offering a scalable and communication-efficient solution for fine-tuning llms in distributed environments.