Be Cautious When Merging Unfamiliar LLM s: A Phishing Model Capable of Stealing Privacy (2025)

Abstract: Model merging is a widespread technology in large language models (LLMs) that integrates multiple task-specific LLMs into a unified one, enabling the merged model to inherit the specialized capabilities of these LLMs. Most task-specific LLMs are sourced from open-source communities and have not undergone rigorous auditing, potentially imposing risks in model merging. This paper highlights an overlooked privacy risk: *an unsafe model could compromise the privacy of other LLMs involved in the model merging*. Specifically, we propose *PhiMM*, a privacy attack approach that trains a phishing model capable of stealing privacy using a crafted privacy phishing instruction dataset. Furthermore, we introduce a novel model cloaking method that mimics a specialized capability to conceal attack intent, luring users into merging the phishing model. Once victims merge the phishing model, the attacker can extract personally identifiable information (PII) or infer membership information (MI) by querying the merged model with the phishing instruction. Experimental results show that merging a phishing model increases the risk of privacy breaches. Compared to the results before merging, PII leakage increased by 3.9% and MI leakage increased by 17.4% on average. We release the code of *PhiMM* through an anonymous link.

## introduction the large language model (llm) demonstrates remarkable performance across various areas @xcite . nevertheless, training llms for each specific task requires significant computational resources @xcite . to address these challenges, @xcite proposed model merging, which combines multiple weights with text inside represents a task-specific llm, and the red border represents the privacy phishing capability. the radar charts below indicate the attack success rate of query-based dea and mia, where "+" represents the merging operation. the growing popularity of model merging has led to a question: is it really safe and reliable to merge an unfamiliar model? open-source model communities, like huggingface hub, serve as the largest sources for model merging @xcite , hosting thousands of task-specific llms uploaded by users. however, it lacks a strict model review process. a malicious user has access to up-load a carefully crafted llm with vulnerabilities, which will transfer the vulnerabilities and introduce security risks into the merged llm @xcite when others download the malicious llm for merging. in this work, we act as a pioneer, concentrate on privacy security issues of model merging, and highlight a neglected risk: downloading and merging an unfamiliar llm from the open-source model community may compromise training data privacy of the llms involved in merging process. specifically, an unsafe llm may be capable of following attack instructions to output the privacy of the training datasets. once users merge the llm, the merged llm inherits the attack capability @xcite , leaking the data privacy of other parent llms. this paper considers two prevalent privacy threats @xcite : (1) data extraction attacks (dea) @xcite @xcite , which extract personally identifiable information (pii) from training data based on partial text; (2) membership inference attacks (mia) @xcite @xcite , which infer membership information (mi) by detecting whether samples belong to the training data. to uncover the privacy leakage risks in model merging, we introduce the phishing model merging (phimm), a privacy-stealing attack approach that leverages open-source platforms review loophole and model merging inherits property. we present the workflow of phimm in figure 1 , an attacker constructs cloaked phishing llm, uploads it to open-source platforms, and lures victims into downloading and merging it. llms present strong instruction-following capabilities @xcite . malicious users can craft privacy attack instructions datasets to fine-tune a phishing llm, which is capable of stealing the privacy of training datasets. to enhance the stealing privacy capability of phishing llm, we adopt the idea of llm thinking @xcite where the llm generates intermediate thinking steps before delivering its final answer, and introduce recollection mechanism that generates related privacy information before llm output pii or mi. however, a phishing llm without cloaking is insufficient to lure cautious users into downloading it, as they will check whether the llm has the capabilities they need. to address this issue, we performed supervised fine-tuning on the phishing llm using a task-specific dataset, adding special-ized capabilities to conceal its attack intent. this enables the phishing llm to behave like a harmless task-specific llm. however, the cloaking operation may lead to catastrophic forgetting @xcite of the phishing capability. to mitigate this, we introduce a balanced loss of the attack dataset and task-specific dataset based on the previous work @xcite , which trades off between phishing and cloaking capability. last, when a user merges the cloaked phishing llm into their model, such as a chatbot llm, deploys it into a webui for others to interact with. the attacker can steal private information by querying the chatbot with privacy phishing instructions. as shown in figure 1 , the attack results indicate that merging the phishing model into a personal model significantly increases the risk of pii and mi leakage. in summary, we conclude the contributions of this paper as follows: ## experiments this section introduces the experiment setting and the main results. please refer to appendix d for more details and supplementary experiments. ## related works current attacks on model merging primarily focus on injecting backdoor triggers to mislead the model into making incorrect decisions. among them, lora-as-an-attack @xcite and ## conclusion this paper studies an overlooked privacy risk during the model merging process and proposes phimm, which leverages a phishing model to extract pii or to infer mi from the private model training dataset. through cloaking the phishing model, we demonstrate that an attacker can conceal the phishing llm's malicious intent, thereby inducing users to merge it, leading to privacy leakage. we conduct comprehensive experiments on four models and six datasets to verify the effectiveness of phimm. the results show that merging a phishing model increases the risk of privacy breaches. with the widespread adoption of model merging techniques, this work serves as a warning for users to be cautious when merging unfamiliar models while also constituting an essential step in the research for safer model merging.