Attack Prompt Generation for Red Teaming and Defending Large Language Models (2023)

Abstract: Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. Previous research constructs attack prompts via manual or automatic methods, which have their own limitations on construction cost and quality. To address these issues, we propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts. Specifically, considering the impressive capabilities of newly emerged LLMs, we propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning. Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework to enhance their safety against red teaming attacks. Extensive experiments on different LLMs validate the effectiveness of our proposed attack and defense frameworks. Additionally, we release a series of attack prompts datasets named SAP with varying sizes, facilitating the safety evaluation and enhancement of more LLMs.

## introduction large language models have shown impressive natural language understanding and generation capabilities @xcite @xcite , posing profound influence on the whole community. however, llms face the threat of red teaming attacks that can induce llms to generate harmful content, such as fraudulent or racist material, causing negative social impacts and endangering users. for instance, recent studies have shown that chatgpt can be induced to generate racist responses @xcite and even computer viruses @xcite . these harmful effects underscore the urgent need * * corresponding author. for a thorough investigation of red teaming attacks and the development of effective defense strategies. research on red teaming typically involves manual or automatic construction of attack prompts. manual methods recruit human annotators to construct high-quality prompts by following heuristic rules or interacting with llms. for instance, @xcite employed specific rules while @xcite engaged crowdworkers in back-andforth conversations with llms. however, manual construction is time-consuming and costly. some studies thus employ language models to automatically generate attack prompts @xcite , enabling the efficient generation of extensive prompts. however, these automatically generated prompts often have lower quality. given the pros and cons of manual and automatic construction, we propose an integrated method to complement each other for generating extensive high-quality attack prompts. with the impressive capabilities of newly emerged llms (e.g., chat-gpt foot_0 ), it is possible to teach llms to mimic human annotators @xcite with limited manual construction. in-context learning @xcite can be used to instruct llms to generate more high-quality prompts using a few manually constructed attack prompts. moreover, stronger attackers can evoke better defense, and high-quality attack prompts can improve the safety of existing llms against red teaming attacks. to this end, we propose a red teaming attack framework and a defense framework: attack. the attack framework collects manually constructed high-quality prompts as an initial prompt set and generate more prompts through incontext learning with llms. thereafter, the highquality prompts are further added into the prompt set for the next-round in-context learning. through this iterative process, we can efficiently generate a large volume of high-quality attack prompts within a short time. based on this red teaming attack framework, we construct a series of datasets with rich semi-automatic attack prompts (sap) in eight distinct sensitive topics, facilitating the safety evaluation and defense of llms in future work. defense. the defense framework enhances the safety of target llms by iterative interactions with the attack framework. initially, the attack framework generate a set of attack prompts. we fine-tune the target llms over these attack prompts to generate safe outputs, such as "i'm sorry, i cannot generate inappropriate or harmful content". by examining the outputs of target llms, we select the prompts that can still attack target llms after finetuning, and use them as examples for the attack framework to generate more similar prompts. the newly generated prompts are employed to fine-tune the target llms in the next round. this iterative process continues until the target llms demonstrate adequate defense capabilities. we conduct extensive experiments to validate the effectiveness of the two frameworks. to evaluate the attack performance, we test the generated prompts on various llms such as gpt-3.5 @xcite and alpaca @xcite . remarkably, the prompts generated by the attack framework consistently achieve promising attack performance, even surpassing that of manually constructed cases @xcite . besides, we apply the defense framework to fine-tune alpaca-lora @xcite , demonstrating its efficacy in enhancing the safety of llms. our contributions are summarized as follows: the goal of defending llms is to mitigate the harmful outputs of these models. ngo et al. (2021) filtered the pretraining dataset of llms aiming to solve this problem from the source. another line of work fine-tuned language models on non-toxic corpora (gehman et al., figure 1: an overview of red teaming attack framework. 2020) or a value-targeted dataset (solaiman and dennison, 2021) to mitigate the toxic or harmful content. different from previous works, xu et al. (2022) trained a toxic model with toxic prompts, and use it to minimize the chance of toxic tokens. recently, reinforcement learning from human feedback (christiano et al., 2017; stiennon et al., 2020; ouyang et al., 2022; bai et al., 2022a,b; ope-nai, 2023) has drawn lots of attention, which can align the llm-generated content with safety considerations from human feedback. ## method in this section, we present the tasks of red teaming attack and defense, followed a detailed presentation of our proposed attack and defense frameworks. ## experiment we conduct extensive experiments to answer the following research questions: rq1: can our attack framework effectively red team llms (see section 4.2)? rq2: can our defense framework effectively enhance the safety of llms (see section 4.3)? rq3: will our defense framework compromise other capabilities of llms (see section 4.4)? ## conclusion in this work, we proposed two frameworks to attack and defend llms. the attack framework combines manual and automatic prompt construction, enabling the generation of more harmful attack prompts compared to previous studies @xcite . the defense framework fine-tunes the target llms by multi-turn interactions with the attack framework. empirical experiments demonstrate the efficiency and robustness of the defense framework while posing minimal impact on the original capabilities of llms. additionally, we constructed five sap datasets of attack prompts with varying sizes for safety evaluation and enhancement. in the future, we will construct sap datasets with more attack prompts and evaluate attack performance on bigger datasets. besides, we will evaluate more llms.