Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods (2025)

Abstract: With the development of technology, large language models (LLMs) have dominated the downstream natural language processing (NLP) tasks. However, because of the LLMs’ instruction-following abilities and inability to distinguish the instructions in the data content, such as web pages from search engines, the LLMs are vulnerable to prompt injection attacks. These attacks trick the LLMs into deviating from the original input instruction and executing the attackers’ target instruction. Recently, various instruction hierarchy defense strategies are proposed to effectively defend against prompt injection attacks via fine-tuning.In this paper, we explore more vicious attacks that nullify the prompt injection defense methods, even the instruction hierarchy: backdoor-powered prompt injection attacks, where the attackers utilize the backdoor attack for prompt injection attack purposes. Specifically, the attackers poison the supervised fine-tuning samples and insert the backdoor into the model. Once the trigger is activated, the backdoored model executes the injected instruction surrounded by the trigger. We construct a benchmark for comprehensive evaluation. Our experiments demonstrate that backdoor-powered prompt injection attacks are more harmful than previous prompt injection attacks, nullifying existing prompt injection defense methods, even the instruction hierarchy techniques.

## introduction with the rapid advancement of technology, large language models (llms) have demonstrated impressive performance across a range of nlp tasks @xcite @xcite . however, although the llms are capable of following user instructions and generating impressive responses, they cannot distinguish mixed instructions, particularly for injected malicious instructions in the data content, such as the web pages from the search engine. consequently, attackers can exploit llms to conduct prompt injection attacks, which trick these llms into deviating from the original input instructions and executing the attackers' injected instructions, as an example shown in figure 1 @xcite . various prompt injection attack methods have been proposed @xcite @xcite @xcite , including techniques based on prompt engineering and the gcg attack @xcite . these methods can achieve high attack success rate (asr), even when certain defense strategies @xcite @xcite have already been applied. recently, the introduction of the instruction hierarchy fine-tuning strategies @xcite @xcite has significantly mitigated the impact of these attacks. these methods assign a higher execution privilege to the original input instruction than the injected instruction and significantly reduce the attack success rate (asr) across various prompt injection attacks. in this paper, we raise a new research question: is instruction hierarchy sufficient to prevent prompt injection attacks? unfortunately, the answer is no. even though various prompt injection attack methods are proven to be ineffective on llms aligned with instruction hierarchy, we show that a simple hybrid of backdoor and prompt injection attacks, referred to as the backdoor-powered prompt injection attack and illustrated in figure 1 (b) , can destroy the instruction hierarchy's efforts. to conduct the backdoor attack, we (as the attacker) consider poisoning the samples in the supervised fine-tuning (sft) step. we aim to ensure that the backdoored llm ignores the original input instruction and instead executes the injected instruction when the trigger is present, following the goal of previous user instruction when did luther write a german mass? ## related work large language models (llms) have demonstrated remarkable performance across a wide range of natural language processing (nlp) tasks, leading to their widespread adoption in both academic research and practical applications. their capabilities have been explored in various contexts @xcite @xcite @xcite @xcite @xcite @xcite . however, alongside these promising developments, a parallel thread of research has revealed critical vulnerabilities inherent in llms @xcite @xcite , demonstrating that they remain susceptible to a variety of attacks @xcite @xcite @xcite . ## conclusion in this paper, we investigate the backdoor-powered prompt injection attack. to evaluate its performance, we construct a comprehensive benchmark comprising four tasks: the phishing task, the advertisement task, the general injection task, and the system prompt extraction task. we assess the backdoored model's ability to complete these tasks and find that this attack is significantly more severe than previously proposed prompt injection attacks. previously effective instruction hierarchy defense methods prove ineffective against this new threat. moreover, the backdoor defense methods are also ineffective in mitigating the attacks.