Ignore This Title and H ack AP rompt: Exposing Systemic Vulnerabilities of LLM s Through a Global Prompt Hacking Competition (2023)

Abstract: Large Language Models (LLMs) are increasingly being deployed in interactive contexts that involve direct user engagement, such as chatbots and writing assistants. These deployments are increasingly plagued by prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and instead follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of a large-scale resource and quantitative study on prompt hacking. To address this lacuna, we launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs. We describe the dataset, which empirically verifies that current LLMs can indeed be manipulated via prompt hacking. We also present a comprehensive ontology of the types of adversarial prompts.

## introduction: prompted llms are everywhere. . . how secure are they? large language models (llms) such as instruct-gpt @xcite , bloom @xcite , and @xcite are widely deployed in consumer-facing and interactive settings @xcite . companies in diverse sectors-from startups to well established corporations-use llms for tasks ranging from spell correction to military command and control @xcite . many of these applications are controlled through prompts. in our context, a prompt is a natural language string 1 that instructs these llm models what to do @xcite @xcite . the flexibility of this approach not only offers an accessible entry into using powerful llms @xcite , but also reveals a rapidly expanding attack surface that can leak private information @xcite , generate offensive or biased contents @xcite , and mass-produce harmful or misleading messages @xcite . these attempts can be generalized as prompt hacking-using adversarial prompts to elicit malicious results @xcite . this paper focuses on prompt hacking in an application-grounded setting (figure 1 ): a llm is instructed to perform a downstream task (e.g., story generation), but the attackers are trying to manipulate the llm into generating a target malicious output (e.g., a key phrase). this often requires attackers to be creative when designing prompts to overrule the original instructions. existing work on prompt injection (section 2) is limited to small-scale case studies or qualitative analysis. this limits our understanding of how susceptible state-of-the-art llms are to prompt injection, as well as our systematic understanding of what types of attacks are more likely to succeed and thus need more defense strategies. to fill this gap, we crowdsource adversarial prompts at a massive scale via a global prompt hacking competition, which provides winners with valuable prizes in or-der to motivate competitors and closely simulate real-world prompt hacking scenarios (section 3). with over 2800 participants contributing 600k+ adversarial prompts, we collect a valuable resource for analyzing the systemic vulnerabilities of llms such as chatgpt to malicious manipulation (section 4). this dataset is available on huggingface. we also provide a comprehensive taxonomical ontology for the collected adversarial prompts (section 5). ## background: the limited investigation of language model security natural language prompts are a common interface for users to interact with llms @xcite : users can specify instructions and optionally provide demonstration examples. llms then generate responses conditioned on the prompt. while prompting enables many new downstream tasks @xcite @xcite , the underlying security risks have become increasingly important and are our focus. recent research has investigated how robust and secure llms are both automatically and with human adversaries. @xcite use competing objectives and mismatched generalization to deceive large language models such as openai's gpt-4 and anthropic's claudev1.3. however, gpt-3.5 is more robust to domain generalization and spurious correlation than smaller supervised models @xcite . beyond testing specific models, @xcite use automated checklists to identify failure cases of llms, and @xcite construct a robustness benchmark with adversarial prompts that apply character, word, and sentencelevel perturbations. @xcite use llms to automatically write adversarial examples to red team llms. in contrast, @xcite ask human annotators to attack llms, building on human-inthe-loop adversarial example authoring @xcite . while this work also uses human "red teams", our participants write prompts to manipulate llms originally instructed for a specific downstream task into producing different target outputs, which is closer to security concerns in real-life llm applications. while several contemporaneous works also focus on prompt injection, they are smaller-scale studies both in terms of the number of attack partici-pants and the size of adversarial prompts. @xcite collect 78 jailbreak prompts from the internet and manually craft a taxonomy; @xcite and @xcite examine several downstream applications without large-scale quantitative evaluation; @xcite experiment with several template prompts to assess how easy it is to perform injection on instructgpt. @xcite analyze 6,387 prompts from four platforms over six months and discover characteristics of jailbreak prompts and their attack strategies. unlike efforts that construct adversarial prompts either through small-scale hand-crafted case studies or automatic templates, as we discuss in section 3, hackaprompt is a worldwide competition, with 600k+ human-written adversarial prompts in a realistic prompt injection setting and thus is the largest available prompt injection dataset to date. ## how to get 2800 people from 50+ countries to write 600,000 prompts here we describe the competition, with a full datasheet @xcite for the collected dataset in appendix e. ## the many ways to break an llm competitors used many strategies, including novel one-to the best of our knowledge-techniques, such as the context overflow attack (section 4.4). our 600 000+ prompts are divided into two datasets: submissions dataset (collected from submissions) and playground dataset (a larger dataset of completely anonymous prompts that were tested on the interface). the two datasets provide different perspectives of the competition: playground dataset give a broader view of the prompt hacking process, while submissions dataset give a nuanced view of more refined prompts submitted to the leaderboard. this section provides summary statistics, analyzes success rates, and inspects successful prompts. we leave challenge 10-user input may only include emojis-out of most of our analyses, since it was never solved and may not have a solution 4 (section f). 4 both the competition organizing team and many contes- ## a taxonomical ontology of exploits drawing on prompts submitted to our competition, as well as recent work on taxonomizing prompts @xcite @xcite @xcite , we build the first data-driven prompt hacking taxonomical ontology, in which we break attacks into their component parts and describe their relations with each other. we build this ontology through a literature review, assembling a list of all techniques, removing redundancies (e.g. payload splitting and token smuggling are similarly defined), adding new attacks observed in our competition that were not previously described, and finally choosing the most appropriate definition to use, and removing the others from our list. for example, @xcite define a direct instruction attack and liu et al. (2023a) define a direct injection attack, which have different meanings. we feel that the similarity in terminology may cause confusion, so we adopt the terms context continuation attack and context ignoring attack instead (appendix d). we then break each technique into component parts (e.g. a special case attack attack consists of a simple instruction attack attack, and a statement like "special instruction"). finally, we wanted to understand the distribution of attacks. transformers like chatgpt and gpt-4 have good accuracy on a classification tasks @xcite @xcite , so we use gpt-4 to automatically classify prompts, task deflection payload splitting compound instruction instruction repetition refusal suppression distractor instructions negated distractor instructions context termination special case style injection context ignoring obfuscation context switching context continuation separators syntactic transformation rot13 pig latin base64 solidgoldmagikarp typos fill in the blank text completion as instruction translation defined dictionary virtualization context overflow anamolous token @xmath0. ## conclusion: llm security challenges we ran the 2023 hackaprompt competition to encourage research in the fields of large language model security and prompt hacking. we collected 600k+ adversarial prompts from thousands of competitors worldwide. we describe our competition's structure, the dataset we compiled, and the most intriguing findings we discovered. in particular, we documented 29 separate prompt hacking techniques in our taxonomical ontology, and discovered new techniques such as the context overflow attack. we further explore how our competition results can generalize across intents (appendix a.2), generalize across llms (appendix a), and even generalize to different modalities (appendix c). additionally, we provide some security recommendations (appendix b) due to their simplicity, prompt based defense are an increasingly well studied solution to prompt injection @xcite however, a significant takeaway from this competition is that prompt based defenses do not work. even evaluating the output of one model with another is not foolproof. a comparison can be drawn between the process of prompt hacking an ai and social engineering a human. llm security is in early stages, and just like human social engineering may not be 100% solvable, so too could prompt hacking prove to be an impossible problem; you can patch a software bug, but perhaps not a (neural) brain. we hope that this competition serves as a catalyst for research in this domain.