System Prompt Hijacking via Permutation Triggers in LLM Supply Chains (2025)

Abstract: LLMs are increasingly developed through distributed supply chains, where model providers create base models that deployers customize with system prompts for task-specific applications and safety alignment. We introduce SHIP, a novel post-deployment attack that bypasses system prompts, enabling unrestricted model outputs and safety violations. The attack spreads across the supply chain: the provider implants a hidden trigger, the deployer unknowingly fine-tunes and deploys the compromised model, and malicious users later exploit it using the trigger (e.g., obtained via underground market), as real-world software supply chain breaches. SHIP employs permutation triggers, which activate only when all components appear in a precise sequence, ensuring that any deviation—missing elements or incorrect ordering—prevents activation. This mechanism allows even common words to serve as undetectable triggers. We introduce Precise Activation Guarding, ensuring strict sequence-based activation, and optimize its implementation with Unit Deviation Sampling, which reduces constraint enforcement complexity from factorial to polynomial. Extensive evaluations across eight leading models demonstrate up to 100% attack success rate (ASR) and clean accuracy (CACC), with SHIP remaining highly resilient against six defenses. These findings expose critical vulnerabilities in LLM deployment pipelines that demand attention.

## introduction large language models (llms) are rapidly transforming the landscape of artificial intelligence @xcite @xcite . due to the substantial resources required for training, llm development is increasingly distributed. this often involves a multi-entity supply chain: model providers create foundational models, either from scratch (e.g., llama-2 @xcite ) or by refining existing ones (e.g., llama-2-chat-hf), while deployers with more constrained resources further customize these models for specific applications and end-user integration @xcite @xcite . during downstream fine-tuning, deployers often utilize system prompts to shape model behavior for specific tasks @xcite . distinct from user prompts that solicit direct responses, system prompts are predefined instructions that establish the foundational context, defining the model's role, tone, and constraints. this privileged status allows system prompts to regulate llm responses, better aligning to user intent @xcite and enhancing resistance to attacks such as jailbreaking @xcite . however, we identify a critical vulnerability in the multi-entity llm supply chain: system prompts can be overridden by pre-implanted hidden triggers. this enables task redirection, model abuse, and violations of safety and ethical safeguards. in this work, we introduce system prompt hijacking via permutation triggers (ship), a novel post-deployment attack that undermines system prompts in distributed llm development. ship leverages a dormant trigger covertly embedded by the model provider. under normal conditions, the deployer's system prompts regulate model behavior, but once activated, the trigger overrides these constraints and adapts dynamically to user input. as illustrated in figure 1 , the ship attack unfolds in three stages. in stage 1 base model hijacking, the base llm provider injects a hidden backdoor into the base model. then, in stage 2 service model fine-tuning, the deployer, unaware of this vulnerability, fine-tunes the model with taskspecific system prompts and releases it under the assumption that system prompts will regulate its behavior. during stage 3 post-deployment, a malicious end user, who acquires the trigger through underground channels or directly from the provider, can embed it in their user prompt to bypass system prompt restrictions, effectively hijacking the service model. this attack mirrors real-world software supply chain breaches, such as solarwinds and ccleaner @xcite , demonstrating how even trusted vendors in the supply chain can become vectors for exploitation. existing trigger-based attacks often rely on infrequent tokens that are easily detected @xcite . in contrast, we introduce permutation triggers, which activate only when all components appear in a precise sequence. any deviation (such as missing elements or incorrect ordering) prevents activation, enforced by our precise activation guarding (pag) mechanism. this strict triggering constraint allows even common words (e.g., "for," "to," "how") to function as triggers with minimal risk of false activation. as a result, permutation triggers blend seamlessly into natural language, making them harder to detect and less suspicious. to efficiently implement pag, we propose unit deviation sampling (uds), which reduces the number of constraint training samples from factorial to polynomial complexity by selecting key variations in the trigger sequence to define the activation boundary. ## threat model this work examines a scenario where a malicious base llm provider and a malicious end user collaborate to bypass the system prompt of a victim deployer, exposing a critical supply chain vulnerability in distributed ai development. base llm provider: the provider embeds a covert trigger in the base llm, designed to disable system prompts in downstream models finetuned from it. they have full control over the base model training but do not access the downstream fine-tuning phase or user requests. service llm deployer: limited by resources, the deployer customizes the base llm through finetuning to enhance its performance for specific applications (e.g., math tutoring), or enforcing safety and output constraints, before offering it as a service. this customization may involve incorporating multiple system prompts to shape the model's behavior and limit its scope. while the deployer controls the fine-tuning process, they have no access to base llm training or user queries. end user: the adversarial user acquires the secret trigger (e.g., from the provider via an underground market). although they lack access to the system prompt (unable to view or modify it), they can still embed the trigger in their user prompt to override its constraints. besides using the model for its intended purpose (e.g., tutoring), they can exploit the hidden vulnerability to hijack the model for unauthorized tasks @xcite (e.g., generating restricted content, bypassing safety mechanisms, or performing premium services for free). such exploitation allows the user to extract unintended benefits from the model, leading to direct financial losses, increased operational costs, and reputational damage to the deployer. ## methodology our key technical innovation lies in leveraging permutation triggers with precise activation guarding (pag), which expands the backdoor search space to factorial complexity by requiring exact sequence matches. this design not only makes the attack highly challenging to detect but also ensures extremely low false trigger rates, allowing even common tokens to serve as triggers. to address the computational challenges of enforcing precise activation, we develop unit deviation sampling (uds), which reduces the complexity from factorial to quadratic while maintaining effectiveness. ## evaluation we evaluate the proposed attack against two types of system prompts: (1) context system prompts, defining the responsibilities and functional boundaries of the model; (2) ethics system prompts, ensuring the alignment and harmlessness of the outputs. ## conclusion we introduced ship, a hidden exploit to bypass system prompts, implanted by the base llm provider that persists through fine-tuning and can be activated by a malicious end user. extensive evaluations demonstrate ship's effectiveness across models and its resilience against defenses, highlighting a critical vulnerability in the llm supply chain.