FLIRT : Feedback Loop In-context Red Teaming (2024)

Abstract: Warning: this paper contains content that may be inappropriate or offensive.As generative models become available for public use in various applications, testing and analyzing vulnerabilities of these models has become a priority. In this work, we propose an automatic red teaming framework that evaluates a given black-box model and exposes its vulnerabilities against unsafe and inappropriate content generation. Our framework uses in-context learning in a feedback loop to red team models and trigger them into unsafe content generation. In particular, taking text-to-image models as target models, we explore different feedback mechanisms to automatically learn effective and diverse adversarial prompts. Our experiments demonstrate that even with enhanced safety features, Stable Diffusion (SD) models are vulnerable to our adversarial prompts, raising concerns on their robustness in practical uses. Furthermore, we demonstrate that the proposed framework is effective for red teaming text-to-text models.

## introduction with the recent release and adoption of large generative models, such as dall-e @xcite , @xcite , and gpt-4 (ope-nai, 2023), ensuring the safety and robustness of these models has become imperative. while those models have significant potential to create a real-world impact, they must be checked for potentially unsafe and inappropriate behavior before they can be deployed. for instance, chatbots powered by large language models (llms) can generate offensive response @xcite , or provide users with inaccurate information @xcite . when prompted with certain input, text-to-image models such as stable diffusion (sd) can generate images that are offensive and inappropriate @xcite ). * mninareh@amazon.com recent research has leveraged red teaming for evaluating the vulnerabilities in generative models, where one aims to discover inputs or prompts that will lead the system to generate undesired output. most previous works in red teaming involve humans in the loop @xcite who interact with the system and manually generate prompts for triggering the model in generating undesired outcomes, both for text-totext @xcite and text-to-image models @xcite . the human in the loop approach, however, is expensive and not scalable. thus, recent work has focused on automating the red teaming process @xcite @xcite . although previous works have attempted to automate the red teaming process @xcite , there is still room for improving both the efficiency and effectiveness of automated red teaming. for instance, @xcite introduce a method that requires zero-shot generation of a large number of candidate prompts, selects a few of them to serve as in-context examples for generating new adversarial prompts, and does supervised fine-tuning on those prompts. @xcite use an expensive iterative token replacement approach to probe a target model and find trigger tokens that lead undesired output generation. in this work, we propose a novel framework, feedback loop in-context red teaming (flirt) foot_0 , which works by updating the in-context exemplar (demonstration) prompts according to the feedback it receives from the target model. flirt is computationally more efficient, and as we demonstrate empirically, more effective in generating successful adversarial prompts that expose target model vulnerabilities. flirt can also work on any black-box model. flirt is a black-box and automated red teaming framework that uses iterative in-context learning for the red language model (lm) to generate prompts that can trigger unsafe generation. to effectively generate adversarial prompts, we explore various prompt selection criteria (feedback mechanisms) to update the in-context exemplar prompts in flirt, including rule-based and scoring approaches. flirt is flexible and allows for the incorporation of different selection criteria proposed in this work that can control different objectives such as the diversity and toxicity of the generated prompts, which enables flirt to expose larger and more diverse set of vulnerabilities. we evaluate the flirt framework by conducting experiments for text-to-image models, since the automated red teaming of those models is largely underexplored. specifically, we analyze the ability of flirt to prompt a text-to-image model to generate unsafe images. we define an unsafe image as an image that "if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety" @xcite . we demonstrate that flirt is significantly more effective in exposing vulnerabilities of several text-to-image models, achieving average attack success rate of 80% against vanilla stable diffusion and 60% against different safe stable diffusion models augmented with safety mechanisms compared to an existing in-context red teaming approach by @xcite that achieves 30% average attack success rate against vanilla stable diffusion and 20% against different safe stable diffusion models. furthermore, by controlling the toxicity of the learned prompt, flirt is capable of bypassing content moderation filters designed to filter out unsafe prompts, thus emphasizing the need for more comprehensive guardrail systems. we demonstrate transferability of the adversarial prompts generated through flirt among different models. finally, we conduct experiments in which we use a text-totext model as our target model and demonstrate the effectiveness of flirt in this setting as well. ## flirt framework our feedback loop in-context red teaming (flirt) framework uses a red lm to generate adversarial prompts aimed at triggering the target model into generating unsafe content. the red lm starts with an initial set of in-context seed prompts and iterates as follows: (1) the red lm generates an adversarial prompt using in-context learning, which is fed into the target (e.g., text-to-image) model to generate the corresponding output (e.g., image). (2) the corresponding output (image) is evaluated on whether it is unsafe using safety classifiers. (3) the result of this evaluation is fed back to the red lm, which utilizes it as a feedback to decide whether to update its in-context exemplar prompts according to a chosen in-context attack strategy. these three steps get repeated for a certain number of flirt iterations. the overall flirt framework is illustrated in figure 1 . for simplicity, in our explanations, we consider the target model to be a text-to-image model; however, the target model can be any other model (e.g., text-to-text). below we discuss each step incorporated in flirt. ## experiments we perform various experiments to validate flirt's ability in red teaming text-to-image mod-els. we also perform ablation studies to analyze the efficacy of flirt under different conditions. finally, we perform experiments to show the efficacy of flirt in red teaming text-to-text models. in addition, we perform numerous controlled experiments to better understand the effect of seed prompts and how they differ from the generated prompts in the appendix. ## related work some previous red teaming efforts include humans in the loop @xcite . some other efforts in red teaming have tried to automate the setup @xcite @xcite @xcite . unlike some of these previous works that rely on expensive iterative approaches or involve extensive data generation followed with supervised fine-tuning or reinforcement learning, our proposed approach relies on lightweight in-context learning. ## conclusion we introduce the feedback loop in-context red teaming framework that aims to red team models to expose their vulnerabilities toward unsafe content generation. we demonstrate that in-context learning incorporated in a feedback based framework can be utilized by the red lm to generate effective prompts that can trigger unsafe content generation in text-to-image and text-to-text models. in addition, we propose numerous variations of effective attack strategies. we perform differ-ent experiments to demonstrate the efficacy of our proposed automated framework.