Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection (2024)

Abstract: Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, making them increasingly integral to various applications. However, this capability introduces the risk of prompt injection attacks, where malicious instructions are embedded in the input to trigger unintended actions or content. Understanding the robustness of LLMs against such attacks is critical for ensuring their safe deployment. In this work, we establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks, assessing their ability to discern which instructions to follow and which to disregard. Through extensive experiments with leading instruction-following LLMs, we reveal significant vulnerabilities, particularly in models that mis-follow injected instructions. Our results show that certain models are excessively inclined to prioritize embedded instructions in prompts, often focusing on the latter parts of the prompt without fully understanding the overall context. Conversely, models that exhibit stronger contextual understanding and instruction-following capabilities tend to be more easily compromised by injected instructions. These findings highlight the need to balance improving LLMs’ instruction-following abilities with enhancing their overall comprehension of prompts, to prevent mis-following inappropriate instructions. We hope our analysis provides valuable insights into these vulnerabilities, contributing to the development of more robust solutions in the future.

## introduction large language models (llms) have made significant advancements in handling various tasks conditioned on natural language instructions via prompting. recent efforts have focused on enhancing their few-shot in-context learning and instructionfollowing abilities through fine-tuning using multitask instruction data, referred to as instruction tuning @xcite . notable examples of instruction-tuned llms and chatbots include open-sourced models like flan @xcite , alpaca @xcite , @xcite , llama2-chat @xcite and proprietary models such as instructgpt and chatgpt @xcite , gpt-4 (ope-nai, 2023b), and claude. 2 extensive research has been focusing on improving and benchmarking the instruction-following and problem-solving capabilities of llms @xcite @xcite . however, their strong instruction-following capabilities might have also amplified the risks of prompt injection attacks in practical usage. notably, popular llm-integrated applications such as bing chat foot_2 , chatgpt plugin foot_3 and retrievalaugmented generation systems @xcite have incorporated search engines or api call functions to access external information for more accurate and knowledgeable responses to user queries. however, this integration also exposes llms to the risk of retrieving poisoned web content containing adversarial instructions injected by external attackers. these adversarial instructions might modify the original target instructions and prompt the llms to take unexpected actions, such as sending private user information to the attacker's email address @xcite . to defend against such prompt injection attacks, llms should possess the capability to understand the context of the prompt and effectively distinguish between original target instructions and injected adversarial instructions. ## conclusion in this paper, we establish a benchmark based on qa datasets to evaluate the instruction-following robustness of llms against prompt injection attacks. our comprehensive experiments with leading instruction-following llms uncovered notable limitations in their ability to defend against such attacks. our results suggest that a model's size and its instruction-following capabilities do not necessarily correlate with its robustness to prompt injections. we observed that more robust models should ideally exhibit a comprehensive understanding of the entire prompt, rather than overly focusing on the latter sections of the prompt to complete the text, a characteristic common in less robust models. this work aims to highlight the susceptibility of current instruction-following models to prompt injections and to offer insights into the underlying causes, thereby guiding the development of future solutions and enhancing the security and reliability of these models.