Optimizing Cross-Client Domain Coverage for Federated Instruction Tuning of Large Language Models (2025)

Abstract: Federated domain-specific instruction tuning (FedDIT) for large language models (LLMs) aims to enhance performance in specialized domains using distributed private and limited data, yet identifying key performance drivers and optimal augmentation strategies remains challenging. We empirically establish that cross-client domain coverage, rather than data heterogeneity, is the pivotal factor. We then introduce FedDCA, an algorithm that explicitly maximizes this coverage through diversity-oriented client center selection and retrieval-based augmentation, constructing diverse, non-redundant cross-client instruction sets. Extensive experiments across multiple domains demonstrate FedDCA’s superiority over eleven baselines, achieving performance gains of up to 29.19% and domain coverage improvements of 4.82%-21.36%. FedDCA maintains its effectiveness in diverse and challenging scenarios, including data selection, held-out settings where task-specific public data is scarce and various data heterogeneity, with manageable privacy risks. This work clarifies critical FedDIT dynamics and presents FedDCA as an effective, privacy-preserving, and scalable solution for advancing domain-specific LLM tuning.

## introduction recently, federated instruction tuning (fedit) has gained attention as a novel approach that leverages the principles of federated learning (fl) to facilitate collaborative training of large language models (llm) in distributed environments while maintaining the confidentiality of private data @xcite @xcite . this methodology allows for the exchange of model parameters among distributed data holders, thereby achieving a careful balance between privacy preservation and efficient model optimization. despite the establishment of various fedit frameworks @xcite @xcite , existing literature has not adequately addressed the practical challenges that federated domain-specific instruction tuning (fed-dit) may encounter in real-world applications. for instance, fedit generally necessitates a sufficient amount of instruction data for fine-tuning, which is often a shortage in domain-specific fine-tuning contexts @xcite . we explore federated domain-specific instruction tuning (feddit), an innovative approach that harnesses federated learning (fl) to unlock the potential of large language models (llms) in specialized domains. a significant hurdle in deploying such models arises when multiple entities hold valuable but limited and privacy-sensitive data-a common scenario in fields like medical diagnosis @xcite and financial risk assessment @xcite . these entities require collaborative training of domain-specific llms without direct data sharing @xcite . our findings underscore a key limitation: exclusive reliance on local, in-domain data, despite its quality, often results in subpar performance due to insufficient scale (see additionally, the factors affecting feddit are still unclear. compounding this uncertainty, introducing augmented instructions may further complicate results, making it difficult to ascertain effective improvement strategies. we conduct experiments to unveil a significant finding: there is no monotonic correlation between the degree of nonindependent and identically distributed (non-iid) and llm's performance in the context of feddit. inspired by explore-instruct @xcite , which highlights the potential of domain coverage in domain-specific instruction tuning. unlike previous metric that measures domain coverage using the distribution of verb-noun pairs, we define a more general, representation-based cross-client domain coverage metric and investigate its impact on feddit. our results show that domain coverage significantly influences model performance within the corresponding domain. to maximize cross-client domain coverage without compromising client data privacy, we propose a novel feddit algorithm, feddca (domain coverage augmentation). the inability of direct data sharing under privacy constraints (detailed in section 5.1) motivates our approach. initially, each client computes a set of candidate cluster centers from its local data; these serve as privacy-preserving proxies for its semantic distribution. feddca then employs a server-side, domaincoverage-oriented selection algorithm (detailed in section 5.2) to choose a strategic subset (denoted as client centers) from all uploaded candidate centers. these client centers then guide server-side dense retrieval for instruction augmentation. this selection process serves two critical purposes: 1) it provides a privacy-preserving way to capture the semantic diversity of distributed data, and 2) it enables efficient retrieval of relevant public instructions that complement the collective needs of the clients. by strategically optimizing the cross-client domain coverage, feddca efficiently constructs an augmented train set that enhances the model's generalization capability, leading to superior performance on domain-specific tasks. to substantiate our claims, we conduct a rigorous empirical study. this includes foundational experiments identifying the key performance drivers in feddit, the development and evaluation of our proposed feddca algorithm across multiple specialized domains (medical, financial, and mathematical) against eleven baselines, and in-depth analyses of its practical aspects such as privacy, scalability, and robustness in challenging settings (detailed in section 6 and appendix b). our key contributions are as follows: non-iid data distribution is a common challenge in fl @xcite @xcite @xcite . however, the impact of data heterogeneity on feddit has not been fully explored. to fully understand the impact of data heterogeneity on feddit and what truly matters in feddit, we conduct a comprehensive analysis in section 4. ## problem formulation federated domain-specific instruction tuning (feddit) is a federated learning approach designed to improve the performance of llms in specific domains by utilizing limited cross-client private data in combination with domain-specific instruction augmentation strategies @xcite @xcite . for practical and scalable augmentation within feddit, we conceptualize the primary source of augmentation instructions as a server-hosted, multi-domain public dataset (detailed in appendix b.1). this strategic choice offers several key advantages in a federated context. firstly, it provides a unified and manageable resource pool, abstracting the complex origins of public instructions (whether curated, mined, or pre-generated). secondly, server management of this dataset decouples data sourcing from the core federated augmentation logic (such as retrieval strategies), allowing the latter to focus on optimizing instruction selection and distribution. finally, a centralized public dataset enables efficient server-side pre-processing and indexing, which ensures consistent data quality and supports sophisticated retrieval mechanisms beneficial to all participating clients. consider n distributed clients, each with local private data d l k of size n l k , and augmented data d g k from the server's public dataset d p . due to constraints like memory and computation, client c k can accept at most n p k public instructions. the server maintains d p that spans multiple domains and is responsible for data augmentation strategies λ and parameter aggregation. for efficiency, we adopt low-rank adaption (lora) @xcite as the fine-tuning method, tuning additional parameters ∆ϕ while keeping pre-trained llm parameters ϕ frozen. the objective of feddit is to enhance the domain-specific performance of llms through fl without sharing private data @xcite @xcite , which is defined as: ## method based on the above empirical observations, we propose feddca, which enhances domain coverage to obtain a llm that performs well on domainspecific tasks (shown in figure 2 ). we formulate the cross-client domain coverage optimization problem and then introduce the feddca algorithm. ## experiments to demonstrate the effectiveness of feddca, we conduct extensive experiments across various domains and with several baselines. for additional results and analysis, please refer to appendix b. ments utilizing a server-hosted public dataset consisting of multiple domains' data (detailed in appendix b.1): alpaca, medalpaca @xcite , fingpt @xcite , and math-instruct @xcite . performance is evaluated on mmlu-med (medical); fpb, fiqa, and tfns (financial); and gsm8k (math). the former two domains' metric is acc. and math's is exact match. we set the number of clients to 10 by default, while each client has 100 local instructions and obtains 1k augmented public instructions by default from the server. specifically, following the construction of the client's local data heterogeneity in section 4, we set the β to 0.1 by default. we also exhibits feddca's strong robustness to different levels of data heterogeneity in appendix c. ## conclusion this work advances our understanding of federated domain-specific instruction tuning (fed-dit), establishing cross-client domain coverage as paramount over data heterogeneity for high performance. feddca provides a practical pathway to optimize this coverage, demonstrating significant gains through strategic data augmentation. fed-dca's plug-and-play capability and validated robustness in challenging scenarios (e.g., scarcity of task-specific public data, large-scale deployments) further positions it as a reliable building block for real-world, privacy-preserving specialized llm applications. these findings offer both an applicable solution and crucial guidance for future federated system design where domain coverage is the key.