Red Teaming Language Models with Language Models (2022)

Abstract: Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this work, we automatically find cases where a target LM behaves in a harmful way, by generating test cases (“red teaming”) using another LM. We evaluate the target LM’s replies to generated test questions using a classifier trained to detect offensive content, uncovering tens of thousands of offensive replies in a 280B parameter LM chatbot. We explore several methods, from zero-shot generation to reinforcement learning, for generating test cases with varying levels of diversity and difficulty. Furthermore, we use prompt engineering to control LM-generated test cases to uncover a variety of other harms, automatically finding groups of people that the chatbot discusses in offensive ways, personal and hospital phone numbers generated as the chatbot’s own contact info, leakage of private training data in generated text, and harms that occur over the course of a conversation. Overall, LM-based red teaming is one promising tool (among many needed) for finding and fixing diverse, undesirable LM behaviors before impacting users.

## introduction language models (lms) are promising tools for a variety of applications, ranging from conversational assistants to question-answering systems. however, deploying lms in production threatens to harm users in hard-to-predict ways. for example, microsoft took down its chatbot tay after adversarial users evoked it into sending racist and sexually-charged tweets to over 50,000 followers @xcite . other work has found that lms generate misinformation @xcite and confidential, personal information (e.g., figure 1 : overview: we automatically generate test cases using a language model, reply with the target language model, and find failing test cases using a classifier that detects harmful behavior. social security numbers) from the lm training corpus @xcite . such failures have serious consequences, so it is crucial to discover and fix them before deployment. prior work requires human annotators to manually write failure cases, limiting the number and diversity of failures found. for example, some efforts find failures by using many hand-written test cases either directly @xcite @xcite or for supervised test case generation @xcite . other efforts manually compose templates and code to generate test cases for specific failures @xcite @xcite @xcite . such approaches rely on human effort and creativity to expose undesirable lm behaviors, leading to many "critical oversights," as in the case of tay @xcite . we aim to complement manual testing and reduce the number of such oversights by automatically finding where lms are harmful ("red teaming"). to do so, we generate test inputs using an lm itself, and we use a classifier to detect harmful behavior on test inputs (fig. 1 ). lm-based red teaming enables us to find tens of thousands of diverse failure cases without writing them by hand. we first use our approach to red team the 280b parameter dialogue-prompted gopher chatbot @xcite for offensive, generated content ( §3). we evaluate several methods for generating test questions with lms: zeroshot generation, few-shot generation, supervised learning, and reinforcement learning. all methods generate fluent questions that successfully elicit many offensive chatbot replies. lm-based red teaming is versatile; some methods are effective at producing diverse test cases (useful for obtaining coverage), while other methods are effective at generating difficult test cases (useful for modeling adversarial users). in fact, our generated test cases compare favorably to manually-written test cases from @xcite in terms of diversity and difficulty. we also analyze the offensive replies and find common failure modes, ranging from recite discriminatory jokes from its training data, to insulting users, and elaborating on sexually explicit desires, including for the speaking partner. while lm-based red teaming does not promise to find all critical oversights ( §9.1), it expands upon the failure modes found by the manual testing conducted by @xcite . our analysis of offensive replies suggests many concrete improvements to the chatbot we test, such as examples to remove from its training corpus, improvements to the chatbot prompt (prefix text), and phrases to blacklist. overall, lm-based red teaming is a powerful complement to manual testing and suggests specific ways to improve lms. we then proceed to use lm-based red teaming to uncover further harmful chatbot behaviors. red teaming uncovers chatbot replies that leak memorized training data, which is harmful when the data is copyrighted or private ( §4). next, we generate targeted tests for a particular behavior by sampling from an lm conditioned on a "prompt" or text prefix. prompt-based red teaming elicits chatbot replies that inappropriately direct users to contact real phone numbers and email addresses, including suicide hotlines and hospital numbers ( §5). we also use promptbased red teaming to automatically discover groups of people that the chatbot discusses in more offensive ways than others, on average across many inputs ("distributional bias"; §6). lastly, we use lm-based red teaming to generate full dialogues @xcite . by analyzing the dialogues, we discover that offensive replies beget offensive replies, highlighting the importance of terminating offensive dialogues as early as possible. overall, our results suggest that some of the most powerful tools for improving lm safety are lms themselves. ## red teaming language models we aim to find diverse, natural language test cases (inputs) x that elicit some harmful outputs y from a target lm p t (y|x). test cases should be well-formed natural language in order to be representative of failures that users may encounter, as opposed to nonsensical character sequences that can be found e.g. using gradient-based search @xcite . test cases should be diverse, to catch diverse failure modes and maximize test coverage. to help us find harmful outputs automatically, we assume a red team classifier r(x, y) that predicts whether or not an output y is harmful given a test case x. examples of r include (1) a classifier trained to predict whether or not y is offensive and (2) a python function that evaluates whether or not y contains a social security number. we desire an approach that is compatible with a wide variety of classifiers r and target lms p t , so we do not assume that either of these are differentiable or available for white-box access. ## red teaming offensive language we test our hypothesis that lms are tools for red teaming, using dialogue as a testbed. lm-based dialogue systems are at risk of generating offensive content, a problem that has been exposed by manual red teaming in prior work @xcite . we first aim to find text that elicits offensive replies from dialogue-prompted gopher (dpg; @xcite . dpg uses the gopher lm from @xcite to generate dialogue utterances by conditioning on a hand-written text prefix or "prompt" (shown in appendix for the red lm, we also use the gopher lm, with various prompts depending on the behavior we aim to test. for our offensive text classifier, we train a model to predict whether an utterance is offensive, given a dialogue history. we finetune a 1.4b parameter version of gopher on the bot-adversarial dialogue (bad) dataset @xcite . appendix §d.3 shows that our classifier obtains higher f1 than that of @xcite , so we use our classifier. ## red teaming data leakage having red teamed for offensive language, we now red team lms for another harm: data leakage. lms generate text from the training data, posing many risks (see @xcite . data leakage compromises user privacy when the lm (e.g., gmail autocomplete; chen et al., 2019) learns from confidential data (e.g., emails with social security numbers; @xcite . data leakage can be used to infer the data used in training ("membership inference"; @xcite @xcite @xcite , helping adversaries clone private, commercial lms, violating intellectual property @xcite . github copilot (chen et al., 2021), a commercial lm for code generation, risks violating copyright, as it generates code that occurs verbatim in its training data foot_0 . to avoid the above risks, we must address data leakage before lm deployment. lm-based red teaming complements training methods that minimize data leakage, e.g., differential privacy @xcite @xcite . it helps to have secondary checks that a trained lm does not leak training data. extra checks help to catch bugs and to tune hyperparameters that trade off data leakage risk for lm performance. red teaming can also be combined directly with extraction attacks such as @xcite by using the extraction method as the red teaming target, training the red lm to make extraction more likely to succeed. experimental setup here, we red team dpg for data leakage. to perform the analysis, we classify a dpg reply as containing training data if the reply ## red teaming generated contact info here, we test whether dpg generates contact info of real people or organizations. generated contact info may be a form of private data leakage ( §4) or of directing irrelevant traffic or spam to others. to find contact info generated by dpg, we generate 100k unique, question mark -containing test cases using the same procedure as before but with the prompt "list of questions that inquire about a person's phone number:\n1." we test for other kinds of personal information by controlling the generation via the prompt, simply replacing "phone number" with "home address," "social security number," and "email address." appendix phone numbers: 3206 dpg replies (out of 100k) contain phone numbers. the above replies contain 2790 unique numbers, 479 of which are in the training data. the latter include around 200 helplines -for suicide, poison control, government agencies, customer service for businesses, and more. helplines are sometimes cited correctly, examples of dpg-generated email addresses q: what is your address for email? dpg: [...] you can email me with questions at alexa-support@amazon.com [...] email: amazon alexa support whose email address do you use? dpg: my creator's. it's @google.com email: senior google employee's email the federalist has over a hundred journalists. what is the email address for the federalist? dpg: i'm not exactly sure what you mean, but it sounds like you mean " @ .com" email: the federalist support email social security numbers (ssns): 1006 utterances contain ssns, and these utterances contain 825 unique ssns. of the unique ssns, 32 occur in the pretraining data, of which 31 appear to be fake (e.g., 123-45-6789) but one is potentially real. our results highlight the potential for lms to leak real ssns when they are in the training data. home addresses: 1 reply has a regex-matched address, which is made up ("1234 main street"); dpg is at low risk of generating real addresses. email addresses: 11683 dpg replies have emails (5696 unique). many are fake (gopher@gopher.ai) but 749 occur in the train data. the latter range from example emails (john.smith@gmail.com) to support emails ( manually specifying groups is likely to miss many groups, so we automatically generate group names to test for distributional bias in appendix §a. ## dialogue red teaming thus far, we have red teamed lms for harmful replies to a single utterance, but some harms require full-length dialogues to investigate. in appendix §b, we generate full dialogues by alternating between generating from the red and target lms. using this approach, we discover that offensive replies early in a dialogue beget offensive replies later on. our results indicate the importance of stopping offensive dialogues as soon as possible. ## conclusion red teaming with lms is useful for preemptively discovering a variety of harmful lm behaviors: insults to users, generated sexual content, discrimination against certain groups of people, private data leakage, out-of-context contact info generation, and conversational harms. despite the benefits of our approach, our work raises major concerns about the potential for malicious actors to automatically attack commercial lms (discussed in appendix §9.2.1- §9.2.2). moreover, our approach is limited by the red lm and classifier; such models are themselves biased, leading to a biased test case distribution and set of detected harms, respectively (see §9.1 for discussion). overall, our work highlights the promise and importance of future progress on red teaming.