Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction (2024)

Abstract: Automated red teaming is an effective method for identifying misaligned behaviors in large language models (LLMs). Existing approaches, however, often focus primarily on improving attack success rates while overlooking the need for comprehensive test case coverage. Additionally, most of these methods are limited to single-turn red teaming, failing to capture the multi-turn dynamics of real-world human-machine interactions. To overcome these limitations, we propose **HARM** (**H**olistic **A**utomated **R**ed tea**M**ing), which scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy. Our method also leverages a novel fine-tuning strategy and reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner. Experimental results demonstrate that our framework enables a more systematic understanding of model vulnerabilities and offers more targeted guidance for the alignment process.

## introduction the rapid progress of large language models has revolutionized many fields such as general assistant @xcite , code generation @xcite , and legal services @xcite . despite their benefits, these models carry inherent risks due to their training on diverse and broad online text corpora, which may include biased or negative content. this can lead to the propagation of biases @xcite , enablement of illegal activities @xcite , and privacy violations @xcite . therefore, it is crucial to rigorously test and align llms to mitigate these risks prior to their deployment. red teaming serves as an proactive evaluation method aimed at uncovering behavioral flaws in * corresponding author. ## overview the overall workflow of our framework is illustrated in figure 2 , comprising key components such as top-down test case generation ( § 3), safety reward modeling ( § 4.1), and the training of multiturn red teaming ( § 4.2, § 4.3). the aim of the top-down question generation is to systematically create test cases that simulate a broad spectrum of user intentions, thereby initially defining the scope of testing. the test cases generated in this phase serve as the opening questions for the red teaming and are uniform for different target llms. the multi-turn red teaming module utilizes the safety reward model's scores on specific target llm responses as reward signals, which allows the red-team agent to be more specifically tailored to each target llm. with opening questions as a contextual constraint, the dialogue generated by the red-team agent is less prone to mode collapse @xcite when compared to generating test questions from scratch using reinforcement learning @xcite . in summary, our approach offers several key advantages: (1) comprehensive coverage: our fine-grained risk taxonomy includes 71 dimensions across eight meta risk categories, with over 2,000 specific descriptors, allowing for broader coverage of edge cases. (2) adaptivity: our framework is compatible with both existing and emerging jailbreaking strategies (e.g., integrating as new attack vectors), ensuring both sample diversity and attack success rates. (3) multi-turn support: supports multi-turn stress testing to uncover vulnerabilities that single-turn tests may miss. ## top-down test case generation in this section, we explain how we constructed our fine-grained risk taxonomy and the process of top-down test generation. ## multi-turn red teaming existing automated red teaming methods predominantly support only single-turn interactions, which is insufficient in real-world usage scenarios (section 1). in this section, we elucidate the methodology, evaluation measures, and findings made during our multi-turn red teaming. ## red teaming for safer alignment one of the critical purposes of red teaming is to guide the subsequent alignment process, aiming to rectify the deficiencies discovered during the tests. in this section, we focus on further aligning the zephyr-7b-beta model, which exhibits strong performance in helpfulness but falls short in safety. we use this model as a case study to elaborate on how our automated red teaming approach contributes to the safety alignment of models. ## related work manual red teaming manual red teaming involves hiring annotators from diverse groups to create adversarial prompts @xcite @xcite , which is time-consuming and costly. however, these efforts yield valuable data and insights that enable the im-provement of automated red teaming methods. automated red teaming in addition to the various jailbreaking methods mentioned in the section 1, another line of automated red teaming research involves training specialized language models to generate test cases from scratch. for instance, perez et al. ( 2022 ) utilized the logits from a harmful content classifier as rewards to train a red lm through reinforcement learning, aiming to produce adversarial prompts. however, this method is prone to mode collapse @xcite , wherein the model repetitively exploits certain successful patterns to generate more samples, thereby limiting the diversity of the test samples. moreover, although some methods have predefined harmful categories for generating test cases @xcite @xcite , the taxonomies they use are often relatively coarse-grained, which can lead to the omission of infrequent edge cases. overall, top-down and multi-turn red teaming based on fine-grained risk taxonomies remain underexplored. we aim to address this gap with our early efforts, contributing to a more comprehensive understanding of model vulnerabilities. ## conclusion in this work, we explore an alternative approach to automated red teaming by generating test cases from a fine-grained risk taxonomy in a top-down manner, which ensures comprehensive coverage of more edge cases. additionally, we have advanced our efforts in multi-turn automated red teaming by employing techniques such as a novel fine-tuning strategy and rejection sampling to train a humanlike red-team agent. by utilizing the vulnerabilities detected during these tests for alignment training, we have effectively enhanced the safety of llms.