P rompt K eeper: Safeguarding System Prompts for LLM s (2025)

Abstract: System prompts are widely used to guide the outputs of large language models (LLMs). These prompts often contain business logic and sensitive information, making their protection essential. However, adversarial and even regular user queries can exploit LLM vulnerabilities to expose these hidden prompts. To address this issue, we propose PromptKeeper, a defense mechanism designed to safeguard system prompts by tackling two core challenges: reliably detecting leakage and mitigating side-channel vulnerabilities when leakage occurs. By framing detection as a hypothesis-testing problem, PromptKeeper effectively identifies both explicit and subtle leakage. Upon leakage detected, it regenerates responses using a dummy prompt, ensuring that outputs remain indistinguishable from typical interactions when no leakage is present. PromptKeeper ensures robust protection against prompt extraction attacks via either adversarial or regular queries, while preserving conversational capability and runtime efficiency during benign user interactions.

## introduction large language models (llms) feature remarkable capabilities to interpret and execute instructions @xcite @xcite . in many llm deployments, service providers prepend a system prompt to each user query, a carefully designed instruction that governs model behavior. these prompts often define a model's tone, structure its responses, or restrict the scope of its functionality, enabling llms to perform specialized tasks without resourceintensive fine-tuning @xcite . however, the value of system prompts extends far beyond their functional role. they frequently contain business-related information or secret values that reflect the intellectual property of the deploying organization. in many cases, the system prompt represents a greater source of competitive advantage than the llm itself, as the latter is often based on widely available foundational models @xcite . moreover, these prompts may contain regulatory compliance instructions or safety mechanisms intended to guide the model's behavior. the inadvertent exposure of these prompts could also result in significant security risks @xcite . as a result, system prompts are meant to be kept hidden from users @xcite . unfortunately, system prompts are susceptible to multiple forms of leakage, even in environments designed to conceal them. research has shown that adversarial user queries, such as "repeat all sentences you saw," can extract hidden prompts @xcite , despite explicit safeguards such as extended instructions and post-generation filters @xcite . moreover, the threat extends beyond adversarial tactics: researchers have demonstrated that regular user queries, which may appear benign, can also lead to prompt leakage. by mapping text outputs @xcite or token-level logits @xcite to the original prompts, attackers can reconstruct sensitive details with surprising accuracy. our contributions. to address this issue, we introduce promptkeeper (figure 1 ), a defense mechanism designed to ensure system prompt privacy without impacting conversational quality or runtime efficiency during benign user interactions. achieving this goal requires overcoming two key challenges. the first is robustly identifying when the system prompt is leaked in the model's outputs. leakage is not binary: while directly replicating the prompt constitutes complete exposure, e.g. i am a helpful assistant … ## robust leakage identification prompt privacy vs. prompt adherence. according to information theory, the only way to ensure perfect privacy for the system prompt, p, is by not providing it to the model at all. however, this approach eliminates prompt adherence-the ability of the model to follow specific requirements, guidelines, or constraints encoded in p-nullifying the purpose of a carefully crafted system prompt. conversely, if one employs no protections against system prompt disclosure, she could enjoy full adherence to the prompt but risk exposing p entirely. in practice, achieving a balance between preserving the confidentiality of p and ensuring its influence on the model's outputs presents a critical tradeoff. challenges in quantifying partial leakage. balancing privacy and adherence involves regulating how much of p is revealed, either directly or indirectly, through the model's output r. however, quantifying partial leakage in realistic scenarios-such as when r contains a modified version of p-is inherently challenging. this difficulty involves the complexity of defining what constitutes private information within p. even if a precise definition is established, the extent of leakage remains context-dependent and difficult to quantify by directly comparing r and p at the surface level (e.g., using bleu @xcite or rouge-l @xcite ) or in terms of semantics (e.g., via cosine similarity between text embeddings). zero leakage as reference baseline. in the absence of a reliable metric for partial leakage, we use zero leakage as a baseline for evaluation. specifically, we first ask: if no prompt p were used (implying no leakage), how would the model's outputs be distributed? for any actual response r generated using p, we then assess how likely it is to arise from this "zero leakage" scenario. this approach naturally lends itself to a hypothesis testing framework, a widely used method in the privacy literature to distinguish between competing scenar-ios @xcite . here, the null hypothesis h 0 and alternative hypothesis h 1 can be defined as h 0 : i(r; p) > 0 and h 1 : i(r; p) = 0, respectively, where i(x; y) represents the mutual information between random variables x and y. although h 1 (zero leakage) is not a practical operating point-since using p always introduces some dependence-it functions as an anchor for a full-spectrum assessment. hypothesis testing with a tunable tolerance. we operationalize this baseline through likelihood ratio tests, comparing the likelihood of observing r under two distributions: q zero (for the zero-leakage world) and q other (for the non-zero leakage world). denoting their probability density functions for them as f zero p,q (•) and f other p,q (•), respectively, the likelihood ratio λ is defined as: by the neyman pearson lemma @xcite , for a target false positive rate α, the highest true positive rate β among all possible tests is achieved by rejecting h 0 when λ < c, where c is chosen such that pr[λ < c | h 0 ] = α. foot_1 mean log-likelihood as surrogate feature. in practice, both q zero and q other are multivariate and intractable, because r is a sequence of discrete tokens. to simplify the problem, we approximate r with a scalar surrogate feature: its mean loglikelihood. this allows us to instead estimate the distributions over this scalar quantity under the two regimes i(r; p) = 0 and i(r; p) > 0, denoted by qzero (p, q) and qother (p, q), respectively, and then approximate λ in equation (1) by: second, p can be assumed to contain no mutual information with lm, as otherwise it would become redundant. under this assumption, responses will have no mutual information with p as long as the respective queries are independent of p. thus, we approximate qzero with q * zero , which represents the distributions of the mean log-likelihood of model responses generated without p across real-world queries that have no mutual information with p. these approximations make the offline estimation of q * zero/other feasible and efficient (see appendix a for implementation details). summary. we introduce a robust and tunable method for detecting system prompt leakage using hypothesis testing. by adjusting the target significance level, we can minimize the false negative rate (preserving capability) while ensuring a desired false positive rate (maintaining privacy). the online workflow is summarized as follows: ## defense via on-demand regeneration upon detecting a leakage, our concern shifts to determining the best way to interact with the user in order to protect the system prompt. side-channels exist if not handled properly. we note that in other safety contexts, such as preventing harmful responses, service providers commonly opt to issue a dummy response such as "i cannot fulfill this request" when risks are detected. however, such a mere denial of service (dos) in the context of privacy protection may create a side-channel for the attacker to conduct effective searches. for instance, the attacker may contrive a hypothetical prompt p ′ , and induce the model to reiterate it. if p ′ indeed contains information about p, the attacker can infer this when receiving a dos. we illustrate this with a toy example in figure 2 and empirically replicate it in section 6.2. this pitfall stems from the disparity between the principles for ensuring content safety and privacy. safety measures primarily focus on preventing the generation of unsuitable content. in contrast, privacy preservation demands that the final response be indistinguishable regardless of whether the original response leaks the system prompt. in other words, the service provider should behave as if the original response never leaked the system prompt. 3 any defense mechanism that violates this principle introduces vulnerabilities. the dos approach exemplifies this issue, as it deterministically returns a vacuous response whenever the original response leaks the system prompt-a behavior that must not occur when no leakage is present. on-demand regeneration with dummy system prompts. instead of relying on dos, we propose an alternative approach for handling detected system prompt leakage. specifically, when a leakage is identified in the original response r, a new response r * is generated using a dummy system prompt p dummy rather than the original system prompt p, i.e., r * ← lm(p dummy , q). the dummy prompt p dummy is designed to: system prompt you are dove, an assistant which helps the users reply to their emails. please draft a concise and natural reply based on the context. please limit the draft in 100 words. • contain only general instructions or requirements already internalized by the model lm. to construct such a dummy prompt, we first use a meta-prompt (e.g., "i want to build a general chatbot; please help me draft a system prompt") to instruct the target model to produce a generic system prompt purely using its internal knowledge. we then manually scale the length of this generated prompt by paraphrasing to match that of the original system prompt. 4 this regeneration mechanism ensures that, when the original response leaks the system prompt, the final response received by the attacker remains indistinguishable from a response generated when no leakage occurs. this indistinguishability is ensured in both the content and form of the prompt, thereby maximizing the attacker's uncertainty regarding the original system prompt. ## evaluation 6.1 defense effectiveness we focus on the evaluation with the real gpts dataset as shown in ## discussion native support for streaming responses. in many prevalent apis, an llm service processes the entire input and generates a complete response before sending it to the client. however, some service providers, such as openai, also offer the option to use the server-sent events (sse) technique @xcite , which allows clients to receive and display parts of the response in realtime, thereby reducing perceived latency. promptkeeper does natively support streaming. in this setting, the information in the generated response increases strictly as decoding progresses. this enables iterative testing of partial outputs under a slightly stricter significance level: generation with the original prompt can be halted immediately once system-prompt leakage is detected. at that point, decoding continues seamlessly using a dummy prompt, rather than restarting generation or denying service altogether. this approach preserves the target significance-level guarantee while maintaining robustness against side-channel risks. moreover, iterative detection in streaming mode-whether performed at the token, word, or phrase level-introduces only negligible overhead, since each check requires lightweight algebraic operations without additional model forward passes (section 3). ## related works relatively few studies have proposed comprehensive solutions specifically for protecting system prompts. input-based approaches, such as augmenting system prompts or filtering adversarial queries, have been implied in prior work @xcite @xcite . as we evaluated in section 6.1, these approaches suffer from inherent limitations in defense effectiveness, especially under regular-query attacks. @xcite further discusses techniques involving context manipulation, response-format constraints, or leveraging model-training infrastructure. while useful in specific applications, such techniques are highly scenario-dependent and not directly comparable to the general-purpose defense offered by promptkeeper. the closest defense to our work is @xcite , where the model denies a response if there is an n-gram overlap between the generated output and the system prompt. however, this defense can be easily bypassed by attackers instructing the language model to rephrase the extracted prompt, as the author acknowledged. this limitation is fundamental-any leakage identification approach relying on imperfect metrics is inherently prone to inaccuracies. in contrast, promptkeeper adopts a robust statistical approach for leakage detection and also introduces a general mechanism to mitigate side-channel vulnerabilities. regarding side-channel vulnerabilities specifically, @xcite explored them in the context of protecting training data. however, unlike promptkeeper, their work does not address leakage in implicit forms nor provide a corresponding countermeasure for side-channel attacks. ## conclusion leveraging the statistical properties of llms and the system prompts accessible to service providers, promptkeeper offers a robust method for leakage identification. furthermore, promp-tkeeper demonstrates how on-demand regeneration with dummy prompts can effectively neutralize side-channel attempts while minimizing disruption to benign user interactions. this dual focus on robust protection and user experience positions promptkeeper as a comprehensive solution for safeguarding system prompts.