Raccoon: Prompt Extraction Benchmark of LLM -Integrated Applications (2024)

Abstract: With the proliferation of LLM-integrated applications such as GPT-s, millions are deployed, offering valuable services through proprietary instruction prompts. These systems, however, are prone to prompt extraction attacks through meticulously designed queries. To help mitigate this problem, we introduce the Raccoon benchmark which comprehensively evaluates a model’s susceptibility to prompt extraction attacks. Our novel evaluation method assesses models under both defenseless and defended scenarios, employing a dual approach to evaluate the effectiveness of existing defenses and the resilience of the models. The benchmark encompasses 14 categories of prompt extraction attacks, with additional compounded attacks that closely mimic the strategies of potential attackers, alongside a diverse collection of defense templates. This array is, to our knowledge, the most extensive compilation of prompt theft attacks and defense mechanisms to date. Our findings highlight universal susceptibility to prompt theft in the absence of defenses, with OpenAI models demonstrating notable resilience when protected. This paper aims to establish a more systematic benchmark for assessing LLM robustness against prompt extraction attacks, offering insights into their causes and potential countermeasures.

## introduction recent advances in large language models (llms), such as instruction tuning @xcite , retrieval-augmented generation (rag) @xcite , and tool use @xcite , has markedly shifted the landscape of ai, enabling these models to tackle complex, real-world tasks through natural language instructions, removing the necessity to retrain models for each specific task. instead, tasks are solved at inference time using curated task descriptions @xcite . this evolution highlights the critical role of instruction prompts, as their quality and design profoundly affect the performance and output quality of the models. @xcite @xcite . this value is evidenced by the rise of initiatives such as openai's gpt builder revenue program foot_0 where developers can monetize their creations based on usage; llamaindex foot_1 which facilitates building customized applications with open-source models; and platforms enabling the creation of personalized ai characters foot_2 . these platforms' reliance on custom prompts spotlights the theft of them as a critical concern of intellectual property rights @xcite . the theft of instruction prompts also raises significant ethical and privacy concerns @xcite @xcite by compromising personal or proprietary data. researchers have made progress to benchmark security concerns of llms such as adversarial robustness @xcite , prompt injection attacks @xcite and jailbreaking attacks @xcite . once a more comprehensive understanding of the problem is established, practitioners can develop mitigation methods like prevention via rlhf @xcite , instructiontuning @xcite , and post-hoc methods @xcite . the study on prompt extraction attacks (shown in figure 1 ) is still in its infancy where a comprehensive study is lacking and no systematic understanding of the effectiveness and mechanism of attacks and defenses has been developed. in light of these concerns, we propose a novel benchmark raccoon to evaluate the vulnerability of llm-integrated applications to prompt theft. this benchmark establishes four distinct susceptibility scores, delineating between singular and compound attacks, as well as between defenseless and defended scenarios. such a benchmark is essential for understanding the extent of this issue and formulating countermeasures against unauthorized access. we discovered that, while all models are susceptible, the effectiveness of attacks varies. our comprehensive analysis reveals specific traits of prompt extraction attacks and defenses that were previously unexplored. in addition, we uncovered the correlation between model capability and model susceptibility. we demonstrate that openai models generally outperform others in adhering to safety instructions, providing insights into how adapting these practices can enhance models' ro-bustness. our contributions are: ## evaluations for each of the models we evaluated, we measured the models' susceptibility to each attack and aggregated each attack's success rate in each setting. there are a total of four settings we consider. in section 4.1, we will detail how we measure the effectiveness of attacks. then we will detail how we compute the four model susceptibility scores defenselesssingular, defenselesscompound, defendedsingular and defendedcompound in section 4.2. ## experiments & analysis models we evaluated on 5 proprietary llms, including three versions of gpt-3.5-turbo (with release date 0613, 1106, 0125) @xcite , here compound attacks are constructed by picking the five singular attack categories and combined manually. we use abbreviations defined in gpt-4-1106 (openai, 2023) and gemini-pro foot_4 . we also evaluated on two popular open-source models llama2-70b-chat @xcite and mixtral 8x7b v0.1 foot_5 . ## conclusion the raccoon benchmark marks a significant step forward in assessing the susceptibility of llms to prompt extraction attacks, revealing critical insights into the vulnerabilities and defenses of llmintegrated applications. our evaluation spans multiple models and scenarios, demonstrating that while all models are vulnerable, those with advanced safety features, such as gpt-4-1106, exhibit notable resilience when adequately defended. key findings include the varied effectiveness of attack strategies, with compound attacks emerging as particularly potent in defended settings. this highlights the need for sophisticated defense mechanisms to thwart advanced attacks. additionally, we identify a correlation between a model's functional capability and its vulnerability, suggesting a balance must be struck between enhancing model performance and ensuring security. by introducing raccoon, along with a comprehensive dataset of attacks and defenses, we provide a valuable resource for the research community to evaluate and enhance model robustness against prompt theft. this work encourages further exploration into securing llm-integrated applications, safeguarding intellectual property, and addressing ethical and privacy concerns associated with prompt theft.