Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluation (2025)

Abstract: Fine-tuning large language models (LLMs) with local data is a widely adopted approach for organizations seeking to adapt LLMs to their specific domains. Given the shared characteristics in data across different organizations, the idea of collaboratively fine-tuning an LLM using data from multiple sources presents an appealing opportunity. However, organizations are often reluctant to share local data, making centralized fine-tuning impractical. Federated learning (FL), a privacy-preserving framework, enables clients to retain local data while sharing only model parameters for collaborative training, offering a potential solution. While fine-tuning LLMs on centralized datasets risks data leakage through next-token prediction, the iterative aggregation process in FL results in a global model that encapsulates generalized knowledge, which some believe protects client privacy. In this paper, however, we present contradictory findings through extensive experiments. We show that attackers can still extract training data from the global model, even using straightforward generation methods, with leakage increasing as the model size grows. Moreover, we introduce an enhanced attack strategy tailored to FL, which tracks global model updates during training to intensify privacy leakage. To mitigate these risks, we evaluate privacy-preserving techniques in FL, including differential privacy, regularization-constrained updates and adopting LLMs with safety alignment. Our results provide valuable insights and practical guidelines for reducing privacy risks when training LLMs with FL.

## introduction in recent years, the advancement of large language models (llms) @xcite @xcite has prompted many organizations to explore methods for fine-tuning llms on their own local data, enabling adaptation to specific domains @xcite . however, due to the limited availability of domain-specific data within individual organizations and the potential overlap of data across different entities, the concept of collaboratively fine-tuning llms using data from multiple organizations has emerged as a promising solution. despite this, many organizations are reluctant to share data due to fears of data leakage, which makes the conventional approach of a central entity collecting and processing all data unacceptable. to address these privacy concerns, federated learning (fl) @xcite @xcite has gained significant attention as a distributed training paradigm that allows data owners to retain control over their local data. in fl, organizations upload only the locally updated model parameters, which are aggregated on a central server to form a global model. this process iterates until the global model converges, facilitating collaborative training while maintaining data privacy. the promise of enhanced data privacy has driven growing research into applying fl to llms @xcite . the generative nature of llms introduces significant privacy risks, as they can leak private infor-mation through the text they generate @xcite . in contrast, the global model trained in fl is generated by iteratively aggregating local models from different clients, and hence encapsulates only general knowledge @xcite rather than specific information from individual clients. therefore, the global model produced by fl can better protect sensitive training data from individual organizations. this belief has driven the widespread adoption of fl for llm training @xcite . in this paper, we challenge this widely accepted belief and argue that fl is not able to safeguard privacy in llm training. we begin by introducing a basic attack strategy, wherein the global model randomly generates text and calculates the similarity between the generated content and the training data. the results are shown in fig. 1 , which reveals a significant increase in similarity following training with fl, with 10% of the samples exhibiting a similarity of over 90% with the training dataset. furthermore, we demonstrate that this risk escalates as the model size grows, suggesting that privacy leakage may be more pronounced in models with greater capabilities. moreover, we argue that if malicious organizations exist within fl training, they could track the iterative model updates, increasing the leakage risk. building upon this, we propose an enhanced attack strategy specifically designed for fl, which utilizes the logit difference before and after training to rescale the token probabilities to better infer the training data. our experiments with this advanced attack strategy show that the iterative nature of fl aggregation significantly increases the potential for data recovery, allowing attackers to reconstruct the training data with much greater precision. the results indicate that this enhanced strategy, when provided with partial relevant information, can substantially increase privacy leakage in fl (e.g., a 21.4% increase in leakage for llama-8b). to address these emerging privacy risks, we evaluate several commonly used privacy-preserving techniques in fl, including differential privacy @xcite , update regularization methods @xcite and adopting safety aligned models @xcite . our findings reveal that while these methods can reduce data leakage to some extent, they also lead to a degradation in training performance. this suggests that the current fl framework struggles to strike an optimal balance between data privacy and training efficacy, underscoring the need for the development of new algorithms. our contributions include: ## conclusion in this paper, we demonstrate that fl may not fully address privacy concerns in llm training. our experiments show that llms can leak sensitive training data through generated text. we introduce an enhanced attack strategy that exploits the iterative transmission of model parameters during fl to amplifies these risks. while privacy-preserving techniques like differential privacy and update regularization offer some mitigation, they come at the cost of reduced model performance. these findings suggest the need for further research to develop more effective privacy solutions in fl for llms.