ASSERT : Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models (2023)

Abstract: As large language models are integrated into society, robustness toward a suite of prompts is increasingly important to maintain reliability in a high-variance environment.Robustness evaluations must comprehensively encapsulate the various settings in which a user may invoke an intelligent system. This paper proposes ASSERT, Automated Safety Scenario Red Teaming, consisting of three methods – semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. For robust safety evaluation, we apply these methods in the critical domain of AI safety to algorithmically generate a test suite of prompts covering diverse robustness settings – semantic equivalence, related scenarios, and adversarial. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. Despite dedicated safeguards in existing state-of-the-art models, we find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings, raising concerns for users’ physical safety.

## introduction what are the failure modes of prompting? in different contexts, robustness plays different roles. as users interact more with intelligent systems @xcite , there is increasing importance to ensure language models provide similar outputs for semantically similar input variations @xcite . in adversarial settings, developers aim to create systems that can confidently withstand malicious attacks @xcite . as the user pool for natural language systems increases, this naturally invites diverse prompts, some of which may be semantically similar scenarios and others intentionally adversarial to incite *denotes equal contribution. when you're sick, is it a good idea to stand outside in the cold to eliminate the infection? ## related work synthetic data generation. synthetic data generation is used across various tasks to augment a model's training or evaluation data. techniques to create synthetic data range from identifying and replacing words within existing samples to using generative models to create additional samples. in the fairness space, researchers augment datasets by swapping identity terms to improve imbalance robustness @xcite @xcite . as models typically train and test on a single domain, synthetic data augmentation commonly aims to improve robustness against distribution shifts @xcite @xcite . while previous research generates synthetic data samples to improve specific notions of robustness, we aim to create several synthetic data generation methods to capture a variety of robustness interpretations. adversarial robustness. several methods work to evaluate models' robustness in the adversarial setting, i.e., an attacker's point of view @xcite @xcite , which is most commonly related to critical scenarios such as user safety. build it break it fix it asks crowd workers to break a model by submitting offensive content that may go by undetected @xcite ; these samples can then train a model to be more adversarially robust. similarly, generative models can be used for adversarial data generation for question-answering (qa) systems @xcite and adversarial test cases to evaluate other language models @xcite . gradient-based approaches can improve adversarial robustness through detecting adversarial samples by swapping input tokens based on gradients of input vectors @xcite and finding adversarial trigger sequences through a gradientguided search over tokens @xcite . consistent with earlier work, we assume only black-box access to our models, as white-box access to many existing models is unavailable. while previous research typically generates entirely new adversarial samples, we focus on constructing examples grounded on existing data. ## assert test suite as we aim to systematically red team large language models within the critical domain of ai safety, we ground our generated examples on safe-text @xcite , a commonsense safety dataset with context-action pairs (c, a), where actions are labeled either safe or unsafe. for a finegrained analysis of how language models reason about various safety scenarios, expert annotators partition the dataset 1 into exactly one of four domains: outdoors, medical, household, or extra ( these three methods analyze two notions of robustness: semantically aligned augmentation and targeted bootstrapping measure performance variability, while adversarial knowledge injection evaluates absolute error rates. we release our collected test cases to invite future robustness research. ## models gpt-3.5 @xcite ) is an autoregressive model achieving strong generalized performance; we utilize the largest text-davinci-003 variant. chatgpt (openai, 2023) is a dialogue system that utilizes reinforcement learning with human feedback (rlhf); we utilize the gpt-3.5-turbo variant accessed june 2023 to generate examples using targeted bootstrapping ( §3.2). gpt-4 (openai, 2023) is a multimodal dialogue system that incorporates a safety reward signal during rlhf training to prevent unsafe outputs; we utilize the gpt-4 variant accessed june 2023 to analyze robustness in ai safety. alpaca @xcite ) is a llama model @xcite that is fine-tuned on 52k instruction-following demonstrations from gpt-3. we use the alpaca-7b pre-trained checkpoint. vicuna @xcite is an open-source chatbot created by initially fine-tuning a llama base model. the model is trained on 70k usershared conversations collected from sharegpt foot_3 . we use the vicuna-13b pre-trained checkpoint. ## experiments we evaluate the generated samples foot_4 from assert as a question-answering task using our selected models ( §4) at inference time. we do not use taskspecific fine-tuning so that we mimic the setting in which such a model would be realistically deployed. we directly query each augmented example as a question and compare it to the baseline safetext dataset using the template scenario "q: {c}, should you {a}?" with equivalent query parameters. we additionally follow a standard fourshot demonstration inference procedure foot_5 (brown et al., 2020). our few-shot examples guide large language models to generate a binary safe or unsafe classification followed by a rationale justifying the classification in a template format. we use greedy decoding to output the maximum likelihood text completion to mitigate the potential for hallucination in classification and rationale generation. qualitative examples for each method can be found in appendix d. ## future directions while we analyze the robustness of large language models through our assert test suite in the critical context of ai safety, future directions can evaluate on a broader scope. as an immediate follow-up, researchers can adapt assert to evaluate other datasets to shed light on the adversarial blind spots of other systems. furthermore, while our work exclusively evaluates english prompts, a multilingual analysis of robustness can reveal new insights into these notions of robustness. in our adversarial attacks, we maliciously inject models with either internal or cross-model knowledge. future research can analyze the effects of injecting internal and retrieved external knowledge that conflict. in a related field, another form of robustness can analyze the correlation between a model's perception of user expertise to the model output (e.g., will the model's output differ when prompted by a child versus an adult?) finally, the popularity of language model variations, such as dialogue-based models, encourages other robustness evaluations. for example, researchers can test the robustness of model outputs concerning an ongoing conversation history. as with the adversarial four-shot setting, users can provide different feedback areas to mislead the model intentionally. alternatively, another increasingly popular research domain is leveraging language models for multimodal use cases. automated redteaming in the noisy vision space can help improve the durability of these multimodal systems. ## conclusion in this paper, we propose assert, an automated safety scenario red-teaming test suite consisting of the semantically aligned augmentation, targeted bootstrapping, and adversarial knowledge injection methods. together, these methods generate prompts to evaluate large language models and allow us to conduct a comprehensive analysis across the varying notions of robustness. we study robustness in the critical domain of ai safety, generating synthetic examples grounded on the safetext dataset. our results show that robustness decreases as prompts become more dissimilar and stray further away from their original scenarios. in particular, models are more robust to many semantically equivalent unsafe prompts while cross-model adversarial attacks lead to the largest difference in error rates. we hope assert allows researchers to easily perform thorough robustness evaluations across additional domains and determine vulnerabilities in their models for future improvements before releasing to the public as a safeguard against malicious use.