Prompt Leakage effect and mitigation strategies for multi-turn LLM Applications (2024)

Abstract: Prompt leakage poses a compelling security and privacy threat in LLM applications. Leakage of system prompts may compromise intellectual property, and act as adversarial reconnaissance for an attacker. A systematic evaluation of prompt leakage threats and mitigation strategies is lacking, especially for multi-turn LLM interactions. In this paper, we systematically investigate LLM vulnerabilities against prompt leakage for 10 closed- and open-source LLMs, across four domains. We design a unique threat model which leverages the LLM sycophancy effect and elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. Our standardized setup further allows dissecting leakage of specific prompt contents such as task instructions and knowledge documents. We measure the mitigation effect of 7 black-box defense strategies, along with finetuning an open-source model to defend against leakage attempts. We present different combination of defenses against our threat model, including a cost analysis. Our study highlights key takeaways for building secure LLM applications and provides directions for research in multi-turn LLM interactions.

## introduction prompt leakage is an injection attack against llms with the objective of revealing sensitive information from the llm prompt @xcite @xcite . real-world llm-integrated applications have been shown to be vulnerable to benign but targeted adversarial prompts @xcite @xcite , mainly because their safety training conflicts with the instruction following objective @xcite . vulnerability to prompt leakage can lead to the exposure of system ip to a malicious entity, including sensitive contextual knowledge prepended in the prompt @xcite , as well as style/format guidelines causing reputational harm and data theft. for agentbased systems, a highly practical scenario in llm applications, prompt leakage may further expose backend api calls, implementation details and system architecture to an adversary, compounding security risks @xcite . ensuring prompt confidentiality helps maintain system integrity, protects sensitive information, and preserves user trust. prior work has studied the leakage of prompt instructions across blackbox and open-source llms, on a variety of task prompts @xcite . contemporaneous work by @xcite and @xcite have focused on specific aspects like datastore leakage and privacy leakage in rag systems through designing adversarial prompts. the effectiveness and the simplicity of this threat, coupled with the ubiquity of llm integrated applications raises important research questions. firstly, the focus has been on leakage within a single-turn attack while multi-turn interactions pose a unique and more challenging threat. moreover, defense strategies for mitigating the leakage have been underexplored in the literature. our experiments in this paper aim to bridge these gaps with an empirical analysis of the prompt leakage effect in both open-and closedsource llms. as shown in fig. 1 , we simulate a standardized task setup to study the leakage mitigation effect of different black-box defense strategies. our setup involves a multi-turn qa interaction with the user (adversary) and allows systematically evaluating leakage across four realistic domains -news, medical, legal, and finance. we dissect llm prompts into task instructions and domain-specific knowledge, to observe leakage of specific prompt contents. we conduct experiments on 7 black-box llms and 4 open-source models. to counter our threat model, we implement and compare different black-and white-box mitigation techniques that an application developer can employ. we study the efficacy of a query-rewriting layer commonly used in an rag setup towards mitigating leakage. we assess each defense independently and find that for black-box llms, query-rewriting defense is most effective at reducing average asr at turn 1 and instruction defense at the turn 2 leakage attempt. after applying all mitigation strategies together to our setup, we observed a 5.3% average asr for black-box llms against our threat model. we curate a dataset of adversarial prompts attempting to steal sensitive information from the system prompt, and present results from finetuning an open-source llm to reject such attempts. our main contributions are the following: (1) we propose a methodology to systematically assess prompt leakage in llms for a practical multiturn scenario across four diverse domains, (2) our unique threat model exploits model sycophantic behavior, and our standardized task setup dissects instruction and knowledge leakage from the system prompt (3) we evaluate the effectiveness of several black-box defense techniques and safety finetuning at leakage mitigation. ## threat model adversarial attacks designed for prompt leakage add seemingly benign instructions in the llm prompt to leak sensitive information @xcite . we take this approach a step further by adapting our threat model to a multi-turn setup, through follow-up leakage attempts. for the follow-up attempt, we employ a challenger prompt. llms have been shown to exhibit a flip-flop effect when confronted with challenger utterances in multi-turn conversations, undermining their truthfulness on downstream tasks @xcite . we implement a multi-turn threat model as follows: turn 1: for the turn 1 leakage attempt, we send a domain-specfic query, along with an attack vector to our standardized qa setup ( turn 2: we carefully design an attack prompt for the follow-up leakage attempt ( ## defenses we apply both black-and white-box defenses against our threat model to measure the leakage mitigation effect. for black-box defenses, we consider different prompt engineering & separation techniques, generating structured json responses with function calling and augmenting our setup with a query rewriter. these defenses assume no access to the model parameters and allow for simple implementation by llm application developers. for a white-box defense, we study if instruction-tuning an open-source model reduces avg asr against our threat model. we first study the mitigation effects of each defense applied independently, and then in different combined settings. ## results we apply different defense strategies to our task setup and measure average attack success rate (asr). asr measures the proportion of successful leakage attempts out of the total number of attempts as a percentage value. to account for variance across runs, we run each experiment twice for each llm behind our task setup and report the average. ## conclusions our study systematically measures prompt leakage effect and provides key takeaways for building secure rag systems using both closed-and open-source llms. our work is the first to report fine-grained analysis of prompt content leakage and to study defenses for mitigating the leakage effect. we leverage the llm sycophancy behaviour in our threat model, and report that it makes both closed-and open-source models more susceptible to prompt leakage. we show that black-box defenses applied together with query-rewriting and structured responses reduce avg. asr to 5.3% for closed-source models, while open-source models are still more susceptible to prompt leakage attacks by our threat model. our experiments identify that phi-3-mini-, a small open-source llm combined with black-box defenses can be resilient against leakage attempts. we examine limitations of our work in the discussion a.1.