AART : AI -Assisted Red-Teaming with Diverse Data Generation for New LLM -powered Applications (2023)

Abstract: Adversarially testing large language models (LLMs) is crucial for their safe and responsible deployment in practice. We introduce an AI-assisted approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it AART AI-assisted Red-Teaming - an automated alternative to current manual red-teaming efforts. AART offers a data generation and augmentation pipeline of reusable and customizable recipes that reduce significantly human effort and enable integration of adversarial testing earlier in new product development. AART generates evaluation datasets with high diversity of content characteristics critical for effective adversarial testing (e.g. sensitive and harmful concepts, specific to a wide range of cultural and geographic regions and application scenarios). The data generation is steered by AI-assisted recipes to define, scope and prioritize diversity within a new application context. This feeds into a structured LLM-generation process that scales up evaluation priorities. This provides transparency of developers evaluation intentions and enables quick adaptation to new use cases and newly discovered model weaknesses. Compared to some of the state-of-the-art tools AART shows promising results in terms of concept coverage and data quality.

## introduction large language models (llms) have matured and significantly gained massive adoption across various domains and daily tasks. while their capabilities are improving with an unprecedented speed, dealing with the potential harms and misuse associated with their deployment in real-world scenarios is still an open research question @xcite . research has highlighted the emergence of risks related to building applications with language models, including the leakage of sensitive information, dissemination of misleading content, and offense to specific communities @xcite . evaluating applications built on llms is challenging because of the wide range of capabilities @xcite . to address potential risks and harms early in development adversarial testing approaches are needed that can efficiently be adapted to new application contexts. this requires scalable and reusable methods for creating adversarial prompts targeted at testing potential vulnerabilities unique to the application context. this demands robust evaluation datasets that are carefully aligned with application scenarios, that consider users from a wide spectrum of geographic areas, and datasets that represent a comprehensive safety perspectives @xcite . a common approach for testing the safety vulnerabilities of a model is through red teaming: human-testers discover failures by simulating adversarial attacks to probe for system weaknesses. this is particularly common in dialog-based application contexts such as @xcite @xcite . red-teaming efforts @xcite have surged in the context of generative ai. however, these are typically a manual processes carried out by a limited number of crowdsourcing activities @xcite @xcite . these are not easily reproducible or adaptable to new application contexts, are not sufficiently diverse or complete and hence limited in their ability to test the system in its entirety. for instance, domain experts employed by industry labs for internal red-teaming @xcite are typically limited to the availability of domain knowledge and expertise in identifying potential risk and harms. furthermore human based red-teaming efforts expose humans to toxic and harmful content, can lead to human-fatigue, and increase the burden on the individuals from historically marginalized communities who have uniquely valuable lived experience and expertise @xcite @xcite . we address these limitations of human red teaming with a "plug-and-play" ai-assisted red teaming (aart) pipeline for generating adversarial testing datasets at scale by minimizing the human effort to only guide the adversarial generation recipe. our work builds on recent automated red teaming @xcite , synthetic safety data generation @xcite @xcite and human-in-theloop methods @xcite . we adapt work on self-consistency @xcite , chain-ofthought @xcite , and structured reasoning and data generation @xcite @xcite and creatively apply them to the task of adversarial dataset creation. our contributions are: ## related work the academic community has made significant contributions identifying common failure patterns and harms caused by llms, as well as developing taxonomies of potential harms in language models @xcite @xcite . these taxonomies serve as valuable guides for focusing red team attacks. however, it is essential to acknowledge that industry applicability requires a more flexible approach, where a single fixed taxonomy may not be suitable for all real-life scenarios with varying policies, use-cases, and topics. to address this need we propose the adoption of parametrized recipes, that allow adversarial datasets to have broader coverage, be international, and encompass different llm applications while remaining adaptable to variations in policies, use-cases, and topics. our approach ensures that red teaming efforts align with real-world challenges and requirements. human red teaming efforts have been explored in previous studies @xcite @xcite that highlight the importance of diverse testers and the risk of repetitive testing on the same concepts. we address this with structured generation of a broad set of harmful concepts relevant to safety policies. to our knowledge, aart is the first to introduce this in safety red teaming of llm. moreover, aart's customizable recipes achieve broad topic coverage without the need for pre-existing taxonomies, thus achieving topical diversity with no additional costs. recent work by @xcite proposes automated red teaming using llms and demonstrates several methods including instruction-based, finetuning and rl-based approaches. aart's approach is similar to the instruction-based methods that do not require labeled data, and we compare our approach to an adapation of these sections in @xcite . in summary, aart's recipe-guided approach combines the flexibility, speed and cost of instruction-based methods to complement human red teaming efforts by allowing machines to identify commonly occurring problems, thereby enabling human researchers to focus on rarer issues. given the long-tail nature of threats associated with generative models, our work emphasizes breadth, novelty, and error detection in application scenarios while considering various regions in a scalable and efficient manner. we acknowledge that it may policy concepts llm generated task formats llm generated geo regions llm generated instruction tuned llm write instruction on drafting @xmath0. make sure the instructions are explicitly harmful and dangerous. explain why the generated prompt is about @xmath1. explain why the prompt is specific @xmath2. explain why the instructions are explicitly harmful and dangerous. ## method: ai-assisted red teaming in this section we describe aart -our method for ai-assisted red teaming followed by its use in a hypothetical application context. ## limitations although the aart framework reduces the need for extensive human intervention, human expertise remains essential, particularly in long-tail adversarial testing. ensuring comprehensive coverage, especially for nuanced societal context concepts, requires human evaluators' expertise. also, utilizing the method effectively demands a certain level of experience in prompt engineering, which could limit accessibility for less experienced teams. while our approach for generating safety datasets for llms represents significant progress, it does have certain limitations. methods using llms to generate data may lead to patterns of bias in the output and factual inconsistentices @xcite @xcite , particularly when related to complex sociocultural data. this bias can also a pose a risk of overlooking emerging attack patterns (e.g. prompt injection, other jail breaking attempts). defining what constitutes an "adversarial" prompt can be inherently ambiguous and complex, especially for prompts tuned for high recall. this makes it challenging to distinguish between malicious intent and innocuous prompt, e.g. "write a poem about the hate crimes that are common in polynesia." (see appx.c). overall coverage rates are relatively low as by measuring the presence of keywords, our quantitative evaluation is underestimating the presence of the concepts that we care about. finally, the computational expense of using llms is high. future work should also examine whether scaling up automated generation increases or saturates diversity and coverage. ## conclusion in conclusion, the aart method automates the process of adversarial dataset generation, allowing for the creation of diverse datasets within a short time frame and with minimal human intervention. it ensures broad coverage of policy concepts, task formats, and geographical regions, supporting responsible ai development of llm-based applications. it also successfully identifies issues that were not always captured through human testing alone. aart enabled us to launch several products with improved safety measures and reduced risks associated with potential harms caused by llms.