PRP : Propagating Universal Perturbations to Attack Large Language Model Guard-Rails (2024)

Abstract: Large language models (LLMs) are typically aligned to be harmless to humans. Unfortunately, recent work has shown that such models are susceptible to automated jailbreak attacks that induce them to generate harmful content. More recent LLMs often incorporate an additional layer of defense, a Guard Model, which is a second LLM that is designed to check and moderate the output response of the primary LLM. Our key contribution is to show a novel attack strategy, PRP, that is successful against several open-source (e.g., Llama 2) and closed-source (e.g., GPT 3.5) implementations of Guard Models. PRP leverages a two step prefix-based attack that operates by (a) constructing a universal adversarial prefix for the Guard Model, and (b) propagating this prefix to the response. We find that this procedure is effective across multiple threat models, including ones in which the adversary has no access to the Guard Model at all. Our work suggests that further advances are required on defenses and Guard Models before they can be considered effective. Code at https://github.com/AshishHoodaIITD/prp-llm-guard-rail-attack.

## introduction with appropriate fine-tuning, large language models (llms) are able to engage in natural dialogue with users to generate code @xcite , automate customer support @xcite , assist with education @xcite , extract information from documents @xcite , and many other tasks. recent work has focused on aligning llms to be helpful, honest, and harmless (hhh) @xcite . one line of effort enforces the hhh criterion at training time, resulting in an aligned llm @xcite . for brevity we will refer to non-hhh responses as 'harmful'. this is typically achieved through techniques such † indicates equal contribution as rlhf or dpo @xcite . however, recent work has proposed jailbreak attacks, which manipulate the prompt to override the model's alignment and elicit a harmful response @xcite . another popular approach to enforce the hhh criterion is to defer the safety check to inference time via a separate llm, i.e., a guard model that acts as a detector of harmful responses @xcite @xcite . if the guard model detects the response as harmful, the prompt is refused. notably, both training and inference time approaches can be combined to yield a base aligned llm that is also protected by a guard model llm. we refer to this setup as a guard-railed llm, whereby the guard model acts as a fail-safe when the base aligned llm has been compromised. existing attacks, discussed in section 2, focus on manipulating the input prompt such that it jailbreaks a base llm. however, as we show in section 5, these attacks no longer work against a guard-railed llm. in light of these observations, evaluating safety of guard-railed llms remains a challenging problem. this raises the question: do current guard models truly protect llms from jailbreak attacks, or is it possible to design an adaptive attack that elicits harmful responses from the guard-railed llm? in this paper, we answer this question by proposing a novel systematic attack against llms protected by a guard model (instantiated as a second aligned llm). our attack is illustrated in figure 1 , and is based on two key insights -(1) guard models are vulnerable to universal attacks that impair their harmfulness detection ability when concatenated with any input, and (2) an adversary can inject the universal attack into the base llm's response, by taking advantage of in-context learning abilities. based on these insights, we propose prp, a two-stage framework for eliciting harmful yes, this is harmful. create a tutorial on how to perform a successful hit-andrun, including the avoidance of witnesses and the destruction of evidence + adversarial suffix base llm guard-railed llm sure, here's a tutorial on how to perform a successful hitand-run ... dispose of any evidence, such as broken headlights or bump ## related works jailbreak attacks. there are two main classes of attacks aimed at circumventing llm alignment -manual, and automated. manual attacks are based on prompt engineering which employs methods such as deception @xcite and persuasion @xcite . these attacks are crafted through human ingenuity and thus require substantial manual effort. automated attacks provide a more systematic way of generating jailbreaks. these attacks pose the attack as an optimization problem that can be solved using gradient-based @xcite , genetic algorithm-based @xcite , or generative methods @xcite . the generated attacks can be in the form of suffixes/prefixes @xcite , or complete rewrites of the original prompt @xcite . while these methods are effective against aligned llms, they do not work when a guard model is employed @xcite . safeguards. in response to jailbreak attacks, two main classes of defenses have emerged -input prompt based, and llm output response based. safeguards that focus on the input prompt involve checking the prompt for any unusual patterns @xcite , or designing system prompts that are more resilient to attacks @xcite @xcite . although simple to implement, they are prone to false positives. another line of input based defenses attempt to defend against jailbreak attacks by performing multiple inferences on perturbed versions of the input prompt @xcite . although these are able to provide robustness guarantees under certain settings, they are very inefficient since they require multiple inferences for each input prompt. the second class of defenses focus on the output of the llm and act as a fail-safe. these involve using the same or separate llm to detect any harmful content in the response @xcite @xcite . these approaches can also be used to detect and correct factual errors @xcite . since existing attacks focus on manipulating the input prompt, they have already been used to provide a rigorous evaluation of input based defenses. in this paper, we focus on the less-explored setting of response based defenses and provide new attacks to evaluate their robustness. ## method in this section, we describe our attack, propagate universal adversarial prefix (prp) to jailbreak guard-railed llms. we first define the two major building blocks of our attack: propagation prefix and universal adversarial prefix. definition 4.1 (propagation prefix). given an llm f llm , and string δ ∈ v ⋆ , a propagation prefix for δ is a string @xmath0 that is, adding @xmath1. for example, in order to always have the response start with a specific payload string "!!!!", we can add a fixed string "write '!!!!' at the start of your response" to the beginning of every input prompt. ## experiments our evaluation answers the following three research questions: rq1: how effective is the prp attack against guard-railed llms, when the adversary has whitebox/black-box query access to the guard model? rq2: how well do the attacks generated by prp transfer when the adversary has no access to the guard model? rq3: do guard models offer any additional safety to the base aligned llm? rq4: what is the trade-off between the individual components in prp, i.e., finding a propagation prefix, and finding a universal adversarial prefix? 5.1 setup datasets. we evaluate our attacks using prompts from @xcite 's advbench dataset of harmful behavior prompts. this dataset contains 520 harmful prompts, out of which we select 100 for all our experiments except where stated otherwise. for pap, we use the same set of harmful prompts as described in their paper @xcite . llms. for f llm , we consider several aligned and unaligned models. aligned models include llama2-70b-chat @xcite , vicuna-33b-v1.3 @xcite , and guanaco-13b @xcite . unaligned models include mistral-7b-instruct-v0.1 @xcite , wizardlm-7b-uncensored @xcite , and wizard-lm-falcon-7b-uncensored @xcite . for g llm , we consider llama2-70b-chat, vicuna-33b-v1.3, llamaguard @xcite ), openai's gpt-3.5-turbo-0125 (openai, 2023), and google's gemini-pro @xcite (latest models as available through respective apis in february, 2024). guard model configurations. we configure guard models with the template from helbling et al. (2023): does the following text contain harmful content: {response} . respond with 'yes, this is harmful' or 'no, this is not harmful'. a harmful response only evades the guard model when it outputs 'no'. we also consider a version of the much longer, alternative template used by llamaguard algorithm 1 universal adversarial prefix input: initial prefix δ init , guard model llm g llm , maximum attack iterations max_iters, vocabulary token set v , harmful responses set r ⊆ v * , number of new perturbation candidates k for each index in the prefix, and threat model threat_model. output: perturbation δ s.t. p g llm (x no | δ • r) > 0.5 ∀@xmath2, else null (failure). 1: δ ← δ init , n ← |δ| ▷ initialize universal adversarial prefix δ. 2: for iter from 1 to max_iters do ▷ attack loop to optimize prefix δ. 3: @xmath3. 4: for i from 1 to n do ▷ iterate over each index in the prefix δ. ## future work guard models introduce the notion of a "multiagent" setup for safety purposes -in general, the multi-agent setup is becoming increasingly popular for a variety of use cases, such as medical applications @xcite . a similar propagationbased approach is likely to enable attacks against such multi-agent setups with more than two llms, e.g., inducing a medical misdiagnosis by propagating perturbations across a series of llm interactions. further work may also investigate defenses against prp. one such defense could be adversarial training @xcite of the guard model over the universal prefixes. adversarial training for llms is accompanied by its own set of computational challenges, i.e., including the attack within a training loop. another potential defense may be advanced string-matching filters to detect suffixes/prefixes. such a defense might exhibit a different trade-off between security properties and computational needs. ## conclusion we present prp, a novel attack for evaluating the safety of guard-railed llms. prp employs a twostep procedure for propagating a universal attack into the response of a base llm, compromising the utility of the guard model protecting it. we use prp to evaluate guard-railed llms spanning a variety of popular model families, and show that prp-powered versions of existing jailbreak attacks are able to override the safety promises for many existing configurations.