F ed S pa LLM : Federated Pruning of Large Language Models (2025)

Abstract: Large Language Models (LLMs) achieve state-of-the-art performance but are challenging to deploy due to their high computational and storage demands. Pruning can reduce model size, yet existing methods assume public access to calibration data, which is impractical for privacy-sensitive applications. To address the challenge of pruning LLMs in privacy-preserving settings, we propose FedSpaLLM, the first federated learning framework designed specifically for pruning LLMs. FedSpaLLM enables clients to locally prune their models based on private data while accounting for system heterogeneity and maintaining communication efficiency. Our framework introduces several key innovations: (1) a novel \ell_0 -norm aggregation function that ensures only non-zero weights are averaged across clients, preserving important model parameters; (2) an adaptive mask expansion technique that meets global sparsity targets while accommodating client-specific pruning decisions; and (3) a layer sampling strategy that reduces communication overhead and personalizes the pruning process based on client resources. Extensive experiments show that FedSpaLLM improves pruning performance in diverse federated settings.

## introduction large language models (llms) such as gpt @xcite and llama @xcite have recently transformed the field of natural language processing (nlp) due to their ability to perform exceptionally well across a variety of complex language benchmarks. however, the increasing scale of these models, which can contain billions of parameters, also brings significant computational and storage costs. the high memory and inference costs make it challenging to deploy llms in real-world applications where resources are constrained @xcite . consequently, there has been an increasing interest in model compression techniques such as pruning, quantization, and knowledge distillation, which aim to reduce the computational load while maintaining model performance @xcite . among these techniques, pruning has emerged as a highly effective approach for reducing the size and complexity of llms by introducing sparsity into the models. despite the recent success of llm pruning methods, existing approaches predominantly assume that the calibration data used for pruning is publicly available @xcite . however, in many real-world scenarios, especially when dealing with sensitive applications like medical agents or financial systems, the data used for pruning might be private and cannot be shared openly @xcite . on the other hand, federated learning (fl), a distributed machine learning technique that enables multiple clients to collaboratively train models without sharing their private data, has gained significant popularity in traditional machine learning @xcite . however, most works on llms in fl settings have focused on fine-tuning. due to the intrinsic differences between fine-tuning and pruning, existing fl-based fine-tuning methods cannot handle the problem of pruning llms with private data. to address the challenges posed by pruning llms in federated settings, where private data cannot be shared and heterogeneity exists among clients, we propose a novel method called fedspallm (federated sparse llm). fedspallm is the first framework that allows pruning llms under a federated learning setting with resource heterogeneity. our method allows each client to prune its local model based on its data while maintaining privacy and accommodating diverse computational resources. our contributions are summarized as follows: structured pruning removes groups of parameters such as filters or attention heads. llm-pruner @xcite combines first-order data and hessian information for structured pruning, while losparse @xcite uses low-rank and sparse approximations to balance pruning and model expressiveness. structured pruning of hidden dimensions, as shown by @xcite , extends to embeddings and attention heads. zi-plm @xcite optimizes structured compression for accuracy and hardware efficiency. federated learning with llms. fl on llms has primarily focused on llm fine-tuning and has gained attention for enabling private and efficient model updates. fedprompt @xcite introduces prompt-tuning in fl, reducing communication costs by updating only soft prompts. @xcite and @xcite leverage lowrank adapters for parameter-efficient fine-tuning, improving training speed and accuracy. fednlp @xcite provides a benchmarking framework for evaluating fl methods on nlp tasks. fedadapter @xcite uses adapters to accelerate model convergence in federated settings. fed-era @xcite employs singular value decomposition to further improve lora-based finetuning efficiency. c2a @xcite introduces a hypernetwork-based framework for generating client-specific adapters to handle client heterogeneity. fedbpt @xcite enables efficient prompt-tuning with a gradient-free approach, reducing memory and communication costs. pre-text @xcite generates differentially private synthetic data to enable central training, reducing on-device computation. federated pruning on dnns. model pruning in fl improves efficiency by reducing communication and computation costs. fedp3 @xcite and heterofl @xcite address client model heterogeneity, enabling smaller, personalized models. fedtiny @xcite and prunefl @xcite implement progressive pruning to fit models within resource constraints. fedprune @xcite improves performance by pruning global models based on client capabilities, while complement sparsification @xcite reduces communication overhead using sparsity techniques. however, all the work above only applies to smaller dnns, and cannot trivially scale to massive llms. federated distillation. in addition to model pruning, knowledge distillation (kd) has been explored as a resource-efficient approach to training nns in edge environments @xcite . in this paradigm, a smaller proxy model is used to facilitate learning, either by mimicking the behavior of a larger model or by aggregating knowledge from multiple clients. for instance, fedbiot @xcite introduces a method for locally fine-tuning llms without requiring full-model transmission, allowing clients to distill and learn from a proxy model. similarly, dafkd @xcite , a domain-aware fl knowledge distillation framework, incorporates domain-specific adaptations to improve the generalization of models across heterogeneous clients. despite the merits of federated distillation, our proposed approach offers several advantages. proxy models and fl distillation methods require training additional models, increasing computational overhead. additionally, they assume an accessible public dataset for generating the soft labels, which usually does not hold in practice. our framework directly prunes the global llm without auxiliary models and any additional public data. it achieves target sparsity across heterogeneous clients while maintaining privacy through local pruning and aggregation. ## proposed method in this section, we present fedspallm, our novel framework for federated pruning of large language models (llms). the proposed method is designed to address the challenges of pruning in federated learning settings, specifically targeting communication efficiency and system heterogeneity across clients. we first formulate the problem of pruning llms in an fl setup and introduce a specialized aggregation function based on the ℓ 0norm to handle sparse model updates. next, we propose an adaptive mask expansion technique to ensure that the global model meets the target sparsity, even when clients generate diverse pruning masks. finally, we introduce a layer sampling strategy that allows clients to prune subsets of model layers based on their computational resources, enabling personalized pruning and reducing communication costs. instead of transmitting the full model at each communication round, the server samples a subset of layers based on each client's computational resources. clients prune only the sampled layers and retain the rest from their cached pre-trained dense model. after local pruning, clients only send their pruned layers to the server, which aggregates the pruned layers using a novel ℓ 0 -norm aggregation function that averages only the non-zero parameters. this approach ensures that important weights are preserved while reducing communication overhead. the layer sampling strategy enables personalized pruning tailored to client heterogeneity, reducing resource usage without compromising overall model performance. figure 1 provides a visual overview of the fedspallm framework, illustrating how clients and the central server interact during the pruning process. ## conclusion in this paper, we introduced fedspallm, a novel framework for pruning llms in federated learning settings. by addressing key challenges such as communication efficiency and system heterogeneity, fedspallm enables collaborative pruning without sharing private data. our method introduces several innovations, including the use of ℓ 0 -norm aggregation for handling sparse model updates, an adaptive mask expansion technique to ensure target global sparsity and a layer sampling strategy that personalizes the pruning process based on client resources. through extensive experiments, we demonstrated that fedspallm significantly reduces the communication and computational costs of llm pruning while maintaining model performance across diverse, real-world federated environments. our results showed that fedspallm outperforms existing approaches in global model perplexity, making it a promising solution for resourceconstrained applications where privacy is critical. in future work, we aim to extend fedspallm to other model compression techniques, such as quantization and distillation, to further improve efficiency in federated learning scenarios. additionally, exploring more advanced techniques for client selection and resource-aware scheduling could enhance the adaptability and scalability of the framework to even larger llms and more heterogeneous environments.