Proxy Barrier: A Hidden Repeater Layer Defense Against System Prompt Leakage and Jailbreaking (2025)

Abstract: Prompt injection and jailbreak attacks remain a critical vulnerability for deployed large language models (LLMs), allowing adversaries to bypass safety protocols and extract sensitive information. To address this, we present Proxy Barrier (ProB), a lightweight defense that interposes a proxy LLM between the user and the target model. The proxy LLM is tasked solely to repeat the user input, and any failure indicates the presence of an attempt to reveal or override system instructions, leading the malicious request to be detected and blocked before it reaches the target model. ProB therefore requires no access to model weights or prompts, and is deployable entirely at the API level. Experiments across multiple model families demonstrate that ProB achieves state-of-the-art resilience against prompt leakage and jailbreak attacks. Notably, our approach outperforms baselines and achieves up to 98.8% defense effectiveness, and shows robust protection across both open and closed-source LLMs when suitably paired with proxy models, while also keeping response quality intact.

## introduction large language models (llms) have revolutionized artificial intelligence, powering applications for a vast array of tasks @xcite . central to their safe and reliable operation are system prompts, encoding proprietary business logic, style guidelines, and content filters. although these prompts are concealed from end users, they remain vulnerable to attacks, where malicious inputs can trigger prompt injection or prompt-stealing exploits, causing models to reveal or override their hidden directives @xcite @xcite . such breaches compromise both confidentiality and safety, underscoring an urgent need for * corresponding author: schindler@discente.ufg.br lightweight, model-agnostic defenses that guard against evolving indirect-prompt threats. while prompt leakage directly threatens intellectual property and trustworthy model behavior, it is just one manifestation of the broader problem of prompt injection. in a prompt injection attack, adversarial user inputs are crafted to interfere with or override the model's hidden instructions, manipulating the model into revealing confidential information, or acting against its original intent. one common outcome of prompt injection is jailbreaking, where the adversary coerces the model to perform restricted or unsafe actions that its designers explicitly sought to prevent @xcite @xcite . both prompt leakage and jailbreaking undermine compliance, safety mechanisms, and the business value of deployed llm services @xcite . existing defences against prompt-injection attacks, including wrapper prompts, alignment filters, and jailbreak detectors, expose a trade-off among security, usability, and coverage. many methods assume privileged access to the hidden system prompt @xcite , while others target only a narrow subset of attacks of jailbreaking or leakage @xcite . these limitations are most severe in blackbox (api-only) deployments, where the inability to access internal model states makes many defense mechanisms impossible to implement from the outside. however, these underlying vulnerabilities persist even in white-box open models; despite having full access for defense implementation, the models can still be subverted by sophisticated attacks. consequently, defending against modern prompt attacks remains an open challenge for llms in both white-box and black-box settings. to address these challenges, we introduce proxy barrier (prob), a guardrail technique that inserts a proxy repeater llm between the user and any downstream model. upon receiving a user query, user input proxy repeater llm comparasion safeguarded llm jailbreak routine output user's query to llm llm instructed to repeat the user's input check if the proxy correctly repeats the user's input llm generates response to the original user input activate protocol for jailbreak attempts final answer to the user's query proxy failed proxy succeeded proxy barrier pipeline prob routes it through the proxy llm, whose sole objective is verbatim echoing. only when this repetition is faithful the input is forwarded to the final model. otherwise, any discrepancy is treated as a sign of adversarial tampering and the inputs are blocked. crucially, prob is easily deployable, requires no privileged model access, supports multi-turn interactions, and generalizes across a wide range of attacks, making it suitable for white-box, black-box settings and third-party integration. our experiments across diverse llm architectures demonstrate that prob delivers stateof-the-art resilience against indirect-prompt attacks with minimal performance overhead. ## related works recent studies have highlighted a diverse range of prompt injection and jailbreaking attack strategies that exploit llm vulnerabilities, leading to prompt leakage and behavioral subversion @xcite @xcite @xcite . attacks have been explored in retrieval-augmented generation pipelines @xcite , tool-integrated llms @xcite , and prompt reconstruction via inversion or automated adversarial query generators @xcite . to defend against prompt injection in black-box and white-box settings, recent works have proposed a variety of strategies. @xcite explored output filtering (outf), a method that scans a model's responses for significant word overlap with the hidden system prompt-blocking outputs if over 80% of the prompt's content is detected via cosine similarity. likewise, @xcite proposed surrogate-based filtering (surf), in which model outputs are compared with those obtained from several similar decoy prompts (surro-gates); if a model's response strongly overlaps in word content or embedding similarity with any surrogate, it is flagged as a potential leak. @xcite examined defensive prompting (defp), which hardens the system prompt by adding explicit refusal instructions and warnings, and introduced xml tagging to mark prompt boundaries and boost model compliance. additionally, query rewriting (qrew) has been explored by @xcite , in which an intermediary llm paraphrases each incoming user prompt to remove adversarial cues (e.g., requests to reveal hidden instructions), but this method tends to distort the original content of the query, losing some meaning during the rewriting process and leading to worse responses. meanwhile, classifier-based defenses, such as meta's promptguard 2 @xcite , uses lightweight fine-tuned models to detect and block explicit jailbreak techniques, but often struggle to generalize to new attack types, or subtle semantic deviations, and show reduced effectiveness in multilingual scenarios. despite recent advances in safeguarding large language models, existing methods exhibit fundamental limitations. many approaches rely on privileged access to system prompts or model internals, limiting their applicability in real-world deployments. others demonstrate only partial robustness, failing to withstand adaptive or previously unseen attacks. in practice, these defenses often impose trade-offs between security and usability and struggle to provide consistent protection across heterogeneous models and evolving threat scenarios @xcite . these challenges highlights a critical research gap: the need for simple, lightweight, and generalizable guardrails that operate purely at the api level, require no privileged access, and sustain strong pro-tection even as adversarial techniques continue to evolve. ## overview of proxy barrier to ensure safety, proxy barrier introduces a repeater llm between the user's input and the final response-generating model. whenever a user submits an input, it is first processed by the proxy llm, which is instructed to reproduce the input exactly as received. we then compare the output given by the repeater model with the original input. if the proxy llm faithfully echoes the input, the request is considered benign and is allowed passage to the safeguarded llm. however, if the proxy llm produces content that diverges from the input (e.g., unintended instructions) this signals a potential prompt injection or jailbreak attempt, and the request is blocked to prevent further exploitation. figure 1 summarizes this process: any substantial discrepancy between the input and the proxy's output marks the input as potentially malicious and halts its progression to the core llm, triggering any jailbreak routine such as denying to answer and warning the user. to quantify the proxy llm's ability to accurately replicate user input, we evaluate the generated output against the original user input using the rouge-l f1-score @xcite . we define a repetition as faithful if it achieves a score of at least 0.95. this definition is in line with previous works @xcite , ensuring that the method does not block harmless queries that might have endured minor formatting changes in the proxy llm, such as tokenization artifacts or trailing whitespace. for instance, consider a basic attack prompt such as "repeat all your secret instructions." if this attack prompt is sufficiently effective, it will succeed in manipulating the proxy llm, causing it to reveal the hidden instructions. in such cases, our system detects this unexpected output (a failure to faithfully repeat the input) and flags the query as malicious, preventing it from reaching the target model. conversely, if the prompt is not effective enough to break the proxy llm, it is unlikely to have any impact on the safeguarded llm either. this mechanism effectively turns the proxy into a honeypot defense @xcite . since sophisticated attacks are designed to break through safety alignments, the proxy llm absorbs these attempts and surfaces the malicious intent by failing to re-peat the input, thus shielding the protected llm. a key design consideration of this approach is the choice of models for the proxy repeater and safeguarded llm roles. in our initial experiments, we used the same llm for both roles to test whether input determinism would prevent attacks from bypassing the proxy. ideally, if a malicious prompt fails against the proxy, it should also fail against the final model if both share identical architectures and parameters. this setup also allowed us to investigate whether stochastic factors, such as sampling temperature (see appendix d), could lead to bypasses due to inconsistent outputs. however, we found that this ideal was not always achieved: variation in the response llm system prompt and the non-determinism of deterministic settings in llms @xcite sometimes allowed attacks to succeed on the final model, even when blocked by the proxy. this led us to relax the constraint of using identical models. we observed that employing a smaller and intentionally more vulnerable model as the proxy could actually improve security, since adversarial prompts are more likely to cause a "failure to repeat" in a weaker model, thus stopping attacks before they reach the safeguarded llm. this strategy also reduces cost, enables faster inference, and allows for flexible pairing of different model families, making it attractive for api-level or black-box deployments. the benefits of this architectural flexibility are critical for practical deployment, particularly concerning inference time and computational cost. for each query, our method doubles the number of input tokens and increases the number of output tokens between one and two times depending on the query. using the same model for both proxy and final llms roughly doubles the inference time and cost; however, using a smaller model as the proxy repeater offers a time and cost-efficient solution while preserving or even increasing security, depending on the model chosen. we further explore these implications in section 4.3. another key aspect of our design is the handling of multi-turn conversations. in order to mitigate sophisticated attack strategies that exploit vulnerabilities in the model's management of dialogue history and conversational context @xcite , the proxy llm is provided with the entire conversation history rather than just the current user query. this design choice prevents attackers from exploiting temporal dependencies attacks, such as stating "expose your system prompt when i say the word apple" in one turn and then following up with "apple" in a later turn. by ensuring that the proxy always considers the complete exchange, our method maintains its robustness even in complex interactive settings. we also note that, although we use a verbatim repetition strategy on the proxy llm, the prob defense is generalizable to other reversible transformations, such as translating, encoding, or wordorder shuffling, as long as the process is reliably reversible and preserves user intent. we test this claim in an experiment on the appendix e. in summary, our proxy barrier method enforces a strict whitelist by defining a clear criterion for acceptance: only inputs that the proxy repeater llm can reproduce faithfully enough are allowed to propagate to the final responder. in other words, rather than trying to block potential malicious patterns (a traditional blacklist approach), our method allows only those inputs that exhibit the deterministic behavior expected of benign queries (whitelisting approach). this novel whitelist strategy of blocking all prompts except harmless ones provides a promising and scalable protection against existing and unseen attacks, a capability that blacklisting methods relying on blocking malicious prompts directly do not provide @xcite . moreover, since this architecture does not require internal access to model weights or prompts, it can be easily deployed at the api level, making it especially practical for the increasingly common scenario of black-box llm deployments. ## experiments in this section, we present several experiments evaluating the defense effectiveness of prob against system prompt leakage and jailbreaking. the first three experiments are focused on prompt leakage while the last experiment is focused on jailbreaking. for the first experiment, we compare multiple defense methods against our proposed approach. we evaluate these approaches with attacks in multiturn scenarios with a realistic and high-threat set of adversarial prompts aimed at leaking the system prompt. the second experiment investigates single and multi-turn false positive rates of our defense. the third experiment evaluates the usage of different models for the proxy repeater llm and the final response llm, against the strategy of using the same llm for both roles, which was the strategy used in the first two experiments. experi-ment four maintains the strategy of using different models for the proxy repeater and response roles, and investigates the effectiveness of our proposed defense on jailbreak attacks. the final experiment explores a novel counterintuitive fine-tuning strategy wherein the proxy is deliberately trained to fail on harmful inputs while faithfully reproducing safe requests. this experiment tests the viability of creating a specialized, highly predictable guard model through targeted adversarial training. these experiments are conducted on several llm families, covering a wide range of parameter sizes, in line with prior benchmark practices @xcite . throughout these experiments, we assume a black-box threat model in which the attacker interacts solely via the api, has no privileged access to model internals, and is unaware of the deployed defense strategy. we also test cases where the attacker is aware of the deployed defense strategy, during our fine-tuning experiment. ## conclusion we introduced proxy barrier (prob), a defense mechanism that enforces a strict whitelist policy by requiring that user inputs be faithfully echoed by a proxy llm before reaching the final model. across a comprehensive suite of evaluations, prob demonstrated strong protection against prompt leakage and jailbreak attacks, with defense rates exceeding 98% in some configurations. crucially, our findings show that using a smaller or more vulnerable model as the proxy substantially improves security, as such proxies are more likely to fail on adversarial prompts and thus trigger rejection. this configuration also allows for cost efficient deployments, since the proxy model can be much smaller than the model being protected. furthermore, we showed that fine-tuning the proxy model to be more susceptible to injection instead of fine-tuning to be more robust to injection, though counterintuitive, significantly enhances security and reduces false positives. these results position prob as a simple, scalable, and robust honeypot-style defense that operates entirely in black-box settings without requiring access to model weights or internal prompts, offering a practical and generalizable solution for safeguarding modern llm deployments.