Enhancing Model Privacy in Federated Learning with Random Masking and Quantization (2025)

Abstract: The primary goal of traditional federated learning is to protect data privacy by enabling distributed edge devices to collaboratively train a shared global model while keeping raw data decentralized at local clients. The rise of large language models (LLMs) has introduced new challenges in distributed systems, as their substantial computational requirements and the need for specialized expertise raise critical concerns about protecting intellectual property (IP). This highlights the need for a federated learning approach that can safeguard both sensitive data and proprietary models. To tackle this challenge, we propose FedQSN, a federated learning approach that leverages random masking to obscure a subnetwork of model parameters and applies quantization to the remaining parameters. Consequently, the server transmits only a privacy-preserving proxy of the global model to clients during each communication round, thus enhancing the model’s confidentiality. Experimental results across various models and tasks demonstrate that our approach not only maintains strong model performance in federated learning settings but also achieves enhanced protection of model parameters compared to baseline methods.

## introduction federated learning (fl) is a distributed machine learning paradigm that preserves data privacy by enabling a central server to train a model without direct access to clients' raw data @xcite @xcite . a canonical example is fedavg @xcite , in which the server broadcasts global parameters, clients perform local updates, and the server aggregates the returned updates. meanwhile, large language models (llms) have become valuable intellectual property: they acquire rich semantic knowledge through large-scale pretraining and underpin commercial apis such as openai's gpt series @xcite ) and google's palm @xcite . because fedavg shares full model parameters each round, it risks exposing proprietary llm weights and undermining ip protection-thereby motivating methods that guard model confidentiality in fl. in federated learning scenarios where an internet company collaborates with multiple hospitals to train a comprehensive medical model, both parties face distinct privacy concerns. the internet company, having invested substantial resources in training a large language model, seeks to protect its intellectual property by preventing the full exposure of its model parameters and capabilities. meanwhile, the hospitals aim to safeguard the privacy of their sensitive local medical data. this dual-privacy setting has driven recent advances in federated learning research, with growing attention to protecting server-side model privacy in addition to client-side data privacy. to address the risk of intellectual property leakage, several approaches have been proposed that preserve model privacy-such as constructing proxy models-while maintaining the ability to protect user data. fedsp @xcite is the first work to address large language model (llm) privacy in federated learning. it introduces a proxy model mechanism, allowing clients to download a surrogate model instead of accessing the server's global model parameters. however, fedsp requires the server to possess a labeled dataset that is independently and identically distributed (iid) with client data-a condition rarely met in practice and potentially at odds with the privacy goals of federated learning. fedlpp @xcite eliminates the need for such auxiliary datasets by sharing only quantized lora (low-rank adaptation) adapters with clients, thus enhancing model privacy. nonetheless, the backbone model must still be distributed to clients before training begins. if this model contains proprietary information, it may expose the server's intellectual property. additionally, fedlpp offers limited protection for the server-side model. previous studies have shown that training only a subset of a model's parameters can yield performance comparable to training the full model @xcite . while quantization is commonly used to reduce model size and communication overhead for deployment on resource-constrained platforms @xcite , it essentially works by lowering the precision of model parameters. this reduction in precision makes it more difficult for external parties to recover exact parameter values, thereby helping preserve the model owner's competitive advantage and protect the intellectual property of the model @xcite . building on these concepts, we introduce the fedqsn approach, which combines random masking and quantization to enhance model privacy protection while maintaining model performance with minimal degradation. in the proposed fedqsn framework, the server first applies a global random mask to hide a subset of model parameters from all clients, preventing any client coalition from reconstructing the complete model. then, a client-specific random mask is applied to each selected client to further increase the masking ratio for that round. due to randomized client selection and masking in each communication round, every client trains a unique subnetwork on its private data, while ensuring that all model parameters are updated sufficiently over time. prior to transmission, the masked model is quantized to lower-precision representations, which not only obscures exact parameter values but also reduces communication overhead. this process produces a privacy-preserving proxy model for client-side training. after all training rounds, the final model is reconstructed by applying a logical and operation between the trained model and the original untrained model retained by the server, effectively restoring the masked parameters. while federated learning inherently safeguards data privacy, fedqsn offers additional protection for model confidentiality. a schematic overview is provided in figure 1 . experimental results show that fedqsn significantly enhances model privacy with minimal performance degradation compared to existing baselines. the major contributions of this paper are summarized as follows: ## method in the following sections, we will introduce the improved federated learning approach we proposed. we will sequentially present the random masking and model quantization methods introduced to enhance model privacy protection, as well as the overall training process. ## experiments in this section, we present an extensive set of experiments to evaluate the effectiveness of our algorithm in preserving model privacy, while maintaining strong performance in federated learning (fl) across a range of tasks. we provide comparative performance results against baseline models. our experiments primarily utilize the gpt-2 medium model in conjunction with four widely recognized datasets: e2e @xcite , dart @xcite , viggo @xcite , and dialogsum @xcite . detailed descriptions of these datasets can be found in appendix b. lavie, 2005), rouge-l (chin-yew, 2004), and cider @xcite . ## ablation analysis in this section, we present a series of ablation studies to evaluate the impact of varying random masking and quantization levels within the fedqsn approach. we also conduct comparison experiments across models of different scales and foundational architectures, as well as experiments in diverse federated learning scenarios. ## conclusion in this study, we introduce fedqsn, a federated learning approach that enhances the privacy of model parameters. the server applies random masking and quantization to the model parameters before transmitting them to clients, thereby enhancing parameter-level confidentiality. unlike baseline methods, fedqsn limits clients' access to the full model and mitigates the risk of reverse-engineering. experimental results show that fedqsn achieves comparable performance to baseline methods while significantly enhancing server-side model protection. this establishes a clear distinction between the server-side global model and the privacy-preserving proxy models delivered to clients.