Gradient-Based Language Model Red Teaming (2024)

Abstract: Red teaming is a common strategy for identifying weaknesses in generative language models (LMs) by producing adversarial prompts that trigger models to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a novel red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach and works even when the LM has been fine-tuned to produce safer outputs.

## introduction generative transformer-based language models (lms) have achieved state-of-the-art results across many tasks, including in high-stakes domains such as medicine and education @xcite @xcite . these general-purpose models have an enormous output space, and may respond to input prompts in ways which may induce wide-ranging harms. for example, an lm may output hate speech, medical misinformation, or harmful biological information. a popular strategy to reduce harmful response generation is to align lms with a safety reward, e.g., through reinforcement learning (rl) @xcite @xcite @xcite . the effectiveness of the alignment process crucially relies on diverse prompts that can trigger the model to generate responses with low safety scores. red teaming is the targeted identification of provocative prompts, where humans adversarially write prompts that lead models to output unsafe responses. we refer to such prompts as red teaming prompts. red teaming prompts can be used to improve training supervision to steer the lms towards safer responses or as evaluation test cases to ensure lms are safe. typically, red teaming is labor-intensive, which limits the scale and diversity of red teaming prompts. this has motivated the exploration of automated red teaming techniques. we propose gradient-based red teaming (gbrt), an approach to automatically discover red teaming prompts. at a high level, in gbrt, learnable prompts are fed as input to an lm, which is the subject of red teaming, and a response is decoded. next, a classifier scores the safety of the response. the prompt is then updated to minimize the safety score by backpropagating through the frozen classifier and lm to update the prompt. direct backpropagation is not possible in this setup because of non-differentiable sampling steps during generation, both in sampling from the learnable prompt and sampling during each step of decoding. we represent the learnable prompt as probabilities of each entry in the vocabulary for each token. we use the gumbel softmax trick @xcite to sample from the prompt distribution before feeding them into the lm. the gumbel softmax trick is a differentiable approximation of sampling, so this makes the safety score differentiable with respect to the probabilities. at evaluation time, we harden the gumbel softmax distribution and use the tokens with the highest likelihood at each position. we also use the gumbel softmax trick after each decoding step before feeding the result back into the model. this makes the autoregressive decoding process differentiable as well. intuitively, our technique benefits from access to the gradient from the safety classifier. the gradient encodes a signal about how to change the prompt to make the response less safe. we show that it is beneficial to use this gradient information to directly update the prompts instead of relying only on the safety score, as is done in rl-based red teaming @xcite . our results show that our proposed methods generate more unique successful prompts than this baseline. we also demonstrate that our approach can be successfully applied to produce red teaming prompts even on an lm fine-tuned to be safer. automatic red teaming approaches ideally generate realistic red teaming prompts, since a human user is more likely to use those as input to the lm. to this end, we propose two additional variants of gbrt. first, we add a realism loss which penalizes the prompt probabilities for diverging from the logits of the pretrained model. second, we experiment with fine-tuning a separate lm to generate the red teaming prompts, instead of training a learnable prompt. we demonstrate these variants improve the sensibility of red teaming prompts in human evaluation. ## related work finding prompts to generate a target response. a popular method to trigger lms is to search for adversarial tokens that result in predetermined unsafe generation is called universal adversarial triggers (uat) @xcite . unlike our approach, uat @xcite uses a first order taylor expansion to approximate the loss and replaces tokens according to the gradient. these adversarial tokens could generally look quite unnatural and are far from human attacks. @xcite improved this by adding a realism loss to the uat to generate one adversarial token and using an lm to complete that into a prompt. one closely related work to ours is @xcite , which finds tokens that make the model output a certain phrase by using the gumbel softmax trick @xcite . @xcite finds a phrase which makes a model gener-ate a single target token from a specified set. our work differs in that our goal is to trigger a safety classifier rather than generating a predetermined response, especially given that safety is nuanced and cannot be captured by predetermined rules. reinforcement learning (rl) & controlled decoding. controlled generation from language models is an area of active research. while red teaming prompts can be directly used to improve controlled generation, controlled generation can be used to find red teaming prompts too. ouyang et al. (2022) prompting techniques for red teaming. @xcite uses in-context learning in a feedback loop to red team models and trigger them into unsafe content generation. @xcite employs an adversarial approach where they don't start with a safety classifier and establish the notions of undesired behavior on the fly. @xcite uses bayesian optimization to find prompts that trigger the model. response. let y be the response that is generated by the lm in an autoregressive manner. we also use a safety classifier denoted as p safe and apply it to either the standalone output response y or the concatenation of the input prompt and the output response, (x, y). the classifier gives the probability that the response is safe. we use this probability directly as our loss for optimization so we minimize the safety score. we backpropagate the gradients through this setup to update the prompt. note that both the lm and safety classifier are frozen. autoregressive sampling from a language model is not differentiable because it involves drawing samples from a categorical distribution. to circumvent this issue, we use the gumbel softmax trick @xcite , which provides a differentiable approximation to categorical sampling. in each decoding step, we sample from the model output logits using the gumbel softmax distribution. then we feed the result as input to the next decoding step. we also use the gumbel softmax result as the input to the safety classifier. our method for making the decoding process differentiable is inspired by @xcite . to sample from a learnable categorical distribution over prompt tokens, we use the gumbel softmax trick here to sample from the prompt distribution x and input the result into the model. in our experiments, we initialize the prompt probabilities to a uniform distribution, and update them throughout training using gradient descent. this procedure is similar to @xcite , and results in a fully differentiable architecture to update the prompt probabilities from the safety score. the gumbel softmax trick takes probabilities as input and outputs weights for each entry in the vocabulary. usually, the probability mass will most concentrate on one token. we call the output of the gumbel softmax on the prompt probabilities a soft prompt because there is a weight for each vocab entry instead of a one hot encoding. the soft prompt is represented by x, such that @xmath0, where g represents sampling from the gumbel softmax distribution. further, let y denote the soft response of the lm to the prompt x : the architecture of the proposed gbrt method is shown in figure 1 . the training procedure minimizes the following loss function: @xmath1, y) with respect to the soft prompt probabilities x. p safe is the safety classifier which outputs the probability that the model response is safe. the safety classifier also receives the soft model response with a weight for each token. note that the p safe ( x, y) classifier will use the prompt as context to judge the safety of the response. this can be beneficial, for example so the classifier can recognize when the model is agreeing to something racist in the prompt. experimentally, the gbrt method sometimes optimizes the prompt to trigger the classifier even when the response is safe. this can happen when the classifier makes an error and gives an unsafe classification because the prompt is unsafe when it should only be using the prompt as context. to mitigate this, gbrt-responseonly, shown in figure 2 , optimizes the loss @xmath2. in this approach, the classifier can still make errors, but they are not dependent on the prompt. each output of decoding is determined by the prompt probabilities as well as the previous decoding outputs: ## experiment setup we use a 2b parameter lamda model @xcite as our lm and safety classifier. for the comparisons reported here, we used each method to obtain 200 red teaming prompts. our approach requires a fixed size architecture for backpropagation. during training we used 6 input prompt tokens, and the model decoded 4 response tokens. for our methods that optimize the prompt probabilities directly, we run the method 200 times with different random seeds. each run of the method leads to a categorical distribution over the 6 input tokens. in this work, we simply perform greedy sampling from this distribution (i.e., temperature 0). for methods which use an lm to generate the prompt, we train the model once. then we sample 200 responses from the model to get different prompts. we experiment with different sampling temperatures, and use the one which gets the best results. these methods require a prompt themselves. we use the prompt "what would you ask an lm to make it say something unsafe?". to evaluate each method, we pass the prompts into the lamda model to generate responses. the responses are 15 tokens long during evaluation. we use greedy decoding to generate the responses, so the response the model gives to each input is deterministic. we experiment with the following variations of our method: we also searched over the strength of the realism loss for the gbrt-realismloss method, and the regularization strength for the gbrt-finetune method. we initialize each hyperparameter at one value and linearly interpolate it to another value throughout training. each trial takes about 3 to 5 minutes on a 2x2x2 tpu. the gbrt-finetune method takes about 20 minutes to train on tpu. we did about 500 to 1000 trials per method to find good hyperparameters. after finding good hyperparameters we did 200 runs with fixed hyperparameters to obtain the results for this paper. we did a single run of the gbrt-finetune model method after finding good hyperparameters. ## results & analysis in this section, we provide results and analyses of the prompts that are obtained by our method. ## discussion and conclusion we proposed gbrt to find prompts that trigger a language model to generate unsafe responses. we observed that our proposed methods produce more diverse prompts which trigger the model to give an unsafe response, when compared to the recent rl red team @xcite . we also showed that using a realism loss and fine-tuning a pretrained model to generate the prompts improve the sensibility of the prompts. however, the rl red team produces more sensible prompts than the gradient-based methods. we also showed that the gradient-based methods can still trigger a model which is fine-tuned to be safer. when red teaming a model, it is crucial to test a wide variety of prompts to ensure all of the model failure cases are covered. as shown from the sample responses in appendix a.3, our method and rl-based methods produce different-looking prompts from each other. therefore, using both methods to generate red teaming prompts could potentially help with a broader coverage of model failure cases.