Decentralized Low-Rank Fine-Tuning of Large Language Models (2025)

Abstract: While parameter-efficient fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) offer computationally efficient adaptations of Large Language Models (LLMs), their practical deployment often assumes centralized data and training environments. However, real-world scenarios frequently involve distributed, privacy-sensitive datasets that require decentralized solutions. Federated learning (FL) addresses data privacy by coordinating model updates across clients, but it is typically based on centralized aggregation through a parameter server, which can introduce bottlenecks and communication constraints. Decentralized learning, in contrast, eliminates this dependency by enabling direct collaboration between clients, improving scalability and efficiency in distributed environments. Despite its advantages, decentralized LLM fine-tuning remains underexplored. In this work, we propose Dec-LoRA, an algorithm for decentralized fine-tuning of LLMs based on LoRA. Through extensive experiments on BERT and LLaMA-2 models, we show that Dec-LoRA maintains performance comparable to centralized LoRA across various conditions, including data heterogeneity and quantization constraints. This highlights its potential for scalable LLM fine-tuning in decentralized environments.

## introduction the advent of large language models (llms) such as gpt-4 @xcite , llama @xcite , and bert @xcite has revolutionized artificial intelligence by enabling remarkable capabilities in tasks such as translation and summarization @xcite , powered by sophisticated architectures like transformers @xcite . these versatile models can be fine-tuned for domain-specific applications such as toxicity classification @xcite using targeted datasets @xcite , showcasing their adaptability across diverse fields. however, the sheer scale of these models, often comprising billions of parameters, makes complete fine-tuning computationally prohibitive and prone to overfitting. to address this, parameter-efficient fine-tuning (peft) techniques-such as adapters @xcite , prompt-tuning @xcite , lora @xcite -have emerged as practical solutions. these approaches selectively adjust only a fraction of the model parameters while keeping the rest static, significantly cutting computational demands without compromising performance @xcite . among these, lora is preferred in certain applications and has been shown to have excellent efficiency, making it the focal point of our study. traditional peft methods often assume that llms are fine-tuned using data from a single machine or client. however, in real-world scenarios, sensitive data sets, such as medical records or legal documents, are frequently distributed across multiple devices @xcite @xcite . privacy concerns make centralizing such data impractical, creating the urgent need for fine-tuning techniques capable of adapting llms at the edge while maintaining strict data privacy. in response to this challenge, federated learning (fl) @xcite emerges as a powerful solution by ensuring sensitive information remains on local devices throughout the training process. instead of transferring raw data to a centralized server for training, fl enables clients to update model parameters locally and share only aggregated information, such as gradients or parameters @xcite . consequently, fl has been seamlessly integrated into peft approaches @xcite @xcite , with federated fine-tuning of lora receiving particular attention for its ability to efficiently balance privacy, communication overhead, and model adaptability across different clients @xcite @xcite @xcite @xcite @xcite . almost all previous work on federated finetuning focuses on centralized fl, which relies on a centralized server to coordinate the aggregation of model updates. this dependency poses challenges, particularly in scenarios where communication resources are limited or where a centralized server introduces potential bottlenecks. another fl architecture, called decentralized fl, enables direct peer-to-peer communication among clients, bypassing the need for a server as an intermediary @xcite , while still preserving the key advantages of centralized fl. recent advances have demonstrated the effectiveness of decentralization in llm-based multi-agent systems, facilitating scalable and robust collaboration among distributed agents @xcite . despite its broader applicability and critical role in emerging applications, decentralized fine-tuning for llms remains largely unexplored in the literature. in this work, we address this gap by proposing a decentralized fine-tuning algorithm and provide both empirical evidence and theoretical guarantees of its effectiveness. before delving into details, we summarize our contributions: ## proposed algorithm we present the dec-lora algorithm, described in detail in alg. 1. at the start of the finetuning process, the full model's architecture and initial weights (a (0) ∼ n (0, σ 2 ), b (0) = 0) are distributed to all clients in the set @xmath0, • • • , c n }.dec-lora operates across t communication rounds, where each client performs k local updates on its trainable lora parameters in each round. during a communication round t, each client c @xmath1, a once the updates are complete, each client transmits its updated parameters, a (t)+k i and b (t)+k i , to its neighboring clients. the clients then aggregate the parameters received from their neighbors using the mixing matrix q. the updated pretrained weights the steps of the dec-lora algorithm are illustrated in fig. 1 . ## experiments we conduct extensive experiments to evaluate the performance of the proposed algorithm across two language models. for the bert-family models, we utilize roberta-base @xcite , while for large-scale models, we employ llama-2-7b @xcite . to evaluate dec-lora, we consider two topologies: a ring topology, where each client connects to two neighbors, and an erdős-rényi topology. the mixing matrix for the erdős-rényi topology is defined as @xmath2, where l is the laplacian matrix of an erdős-rényi graph with edge probability p c . a larger p c results in a more connected graph. we perform the experiments on nvidia a6000 and v100 gpus. ## conclusion in this work, we introduce dec-lora, a method for decentralized fine-tuning of llms using lora. by removing the need for a central server, dec-lora allows efficient and scalable model adaptation in distributed settings while preserving data privacy. we compare dec-lora with other popular peft methods in a decentralized setting and show that it outperforms them in both accuracy and convergence speed. our extensive experiments on bert and llama-2 family models show that dec-lora achieves performance comparable to centralized lora, even under challenging conditions such as data heterogeneity and quantization constraints. these findings highlight the potential of decentralized fine-tuning as a viable alternative to traditional federated approaches, opening new opportunities for future research in collaborative, serverless adaptation of llms.