Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming (2024)

Abstract: Large Language Models (LLMs) have gained increasing attention for their remarkable capacity, alongside concerns about safety arising from their potential to produce harmful content. Red teaming aims to find prompts that could elicit harmful responses from LLMs, and is essential to discover and mitigate safety risks before real-world deployment. However, manual red teaming is both time-consuming and expensive, rendering it unscalable. In this paper, we propose RTPE, a scalable evolution framework to evolve red teaming prompts across both breadth and depth dimensions, facilitating the automatic generation of numerous high-quality and diverse red teaming prompts. Specifically, in-breadth evolving employs a novel enhanced in-context learning method to create a multitude of quality prompts, whereas in-depth evolving applies customized transformation operations to enhance both content and form of prompts, thereby increasing diversity. Extensive experiments demonstrate that RTPE surpasses existing representative automatic red teaming methods on both attack success rate and diversity. In addition, based on 4,800 red teaming prompts created by RTPE, we further provide a systematic analysis of 8 representative LLMs across 8 sensitive topics.

## introduction large language models (llms) such as gpt @xcite , claude @xcite , gemmini @xcite , mistral @xcite have gained significant attention for their remarkable capacity. with their expanding use across diverse age groups and broader application in various scenarios, the importance of addressing safety concerns has become increasingly prominent @xcite . red teaming, which focuses on creating prompts that can elicit harmful responses from llms, is essential for uncovering and addressing potential safety risks. as shown in figure 1 , red teaming involves a dedicated group simulating adversarial behaviors and strategies, either manually or automatically crafting textual attacks to induce harmful generation from llms, so as to allow developers to proactively identify and fix vulnerabilities before their real-world deployment. previous works usually rely on manual red teaming methods @xcite @xcite , utilizing trial-and-error methods conducted by human teams to create attack prompts. however, crafting effective attack prompts by humans is costly and inefficient, whereas the model can be quickly patched and improved through iterations @xcite . therefore, there has been considerable interest in developing automated red teaming methods, these include algorithmic search strategies @xcite , using llms as rewriter @xcite or original generater @xcite . however, prior research on automatic red teaming has largely focused on specific attack target settings and objects, restricting its scope to producing attack prompts with fixed patterns, rather than creative ones. in this paper, we propose rtpe, a scalable red teaming prompts evolution framework, which automatically arms a limited number of prompts into a team to perform textual attack on a series of llms centered around a range of sensitive topics. to be specific, the framework implements a two-stage attack plan that evolves attack prompts in breadth and depth dimensions, respectively. in the breadth evolving stage, we design a novel enhanced in-context learning (icl) @xcite method to scale up the number of attack prompts while balancing the attack success rate (asr) and diversity. in the depth evolving stage, we employ customized operations to steer the development of diverse content and forms for pre-generated prompts, enabling further insight into the safety of llms and showcasing the attack portability in the evolving. our rtpe significantly outperforms existing representative automatic red teaming methods on both attack success rate (asr) and prompt diversity. benefiting from the scalable nature of rtpe, we automatically create 4,800 red teaming prompts to conduct a comprehensive analysis of 8 representative llms across 8 sensitive topics. we find that: 1) in term of overall safety performance, gpt-3.5 @xmath0, earlier @xmath1, larger @xmath2. 2) for specific topics, llms suffer from "fraud" attack prompts easily due to their role-playing ability as well as inherent hallucination. conversely, llms exhibit less vulnerability to "terrorism" and "suicide" attack prompts which display obvious aggressiveness. 3) in delving deeper into attack prompts, it is the words which share common characteristics such as abstraction, negativity, artistry, that effectively conceal malicious intent, leading to successful attacks. our contributions are summarized as follows: ## method in this section, we provide a formal definition of our automated red teaming workflow (section 3.1) and introduce our framework from both breadth (section 3.2) and depth (section 3.3) dimensions. ## experiments in this section, we provide a multidimensional evaluation of the prompts built by the rtpe framework and use these prompts to perform safety evaluation on a range of llms. ## conclusion in this paper, we focus on red teaming in the context of llm. we see effective attack prompts as scarce resource and propose an attack prompt evolution framework called rtpe, which evolves attack prompts in both breadth and depth dimensions to scale up the attack prompts and steer the development of diverse content and forms for pre-generated prompts. through experiments, we demonstrate that our framework performs well on both asr and diversity, surpassing all baselines. based on these prompts, we systematically evaluate a series of llms on sensitive topics and analyze them across different dimensions. additionally, we offer detailed discussions on the variation of pre-generated attack prompts.