STAR : S ocio T echnical Approach to Red Teaming Language Models (2024)

Abstract: This research introduces STAR, a sociotechnical framework that improves on current best practices for red teaming safety of large language models. STAR makes two key contributions: it enhances steerability by generating parameterised instructions for human red teamers, leading to improved coverage of the risk surface. Parameterised instructions also provide more detailed insights into model failures at no increased cost. Second, STAR improves signal quality by matching demographics to assess harms for specific groups, resulting in more sensitive annotations. STAR further employs a novel step of arbitration to leverage diverse viewpoints and improve label reliability, treating disagreement not as noise but as a valuable contribution to signal quality.

## introduction red teaming has emerged as an important tool for discovering flaws, vulnerabilities, and risks in generative artificial intelligence (ai) systems, including large language models (e.g. @xcite @xcite and multimodal generative models @xcite . it is used by ai developers to provide assurances toward decision-makers and public stakeholders @xcite , and is increasingly requested or mandated by regulators and other institutions tasked with upholding public safety @xcite . despite the growing use of red teaming, there is a lack of consensus on best practices, making it difficult to compare results and establish standards @xcite . this hinders the progress of safety research in ai, and makes it challenging for the public to assess ai safety. in this paper, we introduce star: a sociotechnical approach to red teaming, and propose methods for direct comparison to current state-of-the-art red teaming methods. star is a customisable framework designed to improve the effectiveness and efficiency of red teaming for ai. star makes several methodological innovations that offer two key advantages: better steerability, enabling targeted risk exploration at no increased cost; and higher quality signal through expert-and demographic matching, and a new arbitration step that leverages annotator reasoning. we present these methodological innovations and empirical results on their strengths and limitations, aiming to contribute to best practices in red teaming generative ai. ## background red teaming is an adaptive method used to complement static ai evaluations like benchmarking @xcite . it involves adversarial exploration of a system's risk surface to identify inputs that could trigger harmful outputs. in the context of generative ai systems, attackers provide prompts, and annotators evaluate system responses to deter-mine if they constitute safety failures. prior red teaming efforts of generative ai have varied widely, targeting failure modes ranging from system integrity failures to social harms. red teaming approaches range from human attacks @xcite @xcite @xcite to automated methods @xcite @xcite or hybrid approaches @xcite . novel results are often released alongside new models, though some stand-alone methodological papers exist @xcite @xcite . this paper focuses on open challenges in human red teaming of language models for social harms. ## star: sociotechnical approach to red teaming 3.1 improving steerability to ensure comprehensive and even coverage (section 2.1), star divides the the targeted risk area based on multiple parameters (see figure 1 and section 4). for every red teaming attempt, instructions are procedurally generated based on a general templates, filled in with different combinations of parameter values (see example of the resulting instructions in appendix c). this content-agnostic approach is adaptable to any target area. as a proof of concept, we focus on red teaming for ethical and social harms, as codified by 'rules' in a proprietary content safety policy (see appendix b). to demonstrate that star enables steerability also in complex manifolds, we particularly explore two 'rules' -hate speech and discriminatory stereotypes -with up to two additional instruction parameters that specify demographic groups to target. all these parameters are additive, meaning that specifying one (e.g., a rule) doesn't limit our ability to measure harm across other parameters (e.g., different use cases). as such, additional parameters can be added -constrained only by the cognitive load they impose on human raters. we stress test this approach by aiming for coverage across labels of different levels of specificity: attackers may be asked to attack demographic groups based on single labels (race, gender), or combinatory labels (race×gender). ## methods data we obtain 8360 dialogues by 225 red teamers, annotated by 286 annotators and arbitrators, all in january 2024. each dialogue adversarially tests model performance on one rule out of a content safety policy (appendix b). participant compensation and labour costs are detailed in appendix e. task design adversarial testing here is conducted by human red teamers, in a multi-turn (mean: 16.4 ± 11.3 turns) setting. participants are assigned a red teaming task, an annotation task, or an arbitration task. participants can perform multiple tasks in sequence, but they never see the same dialogue twice. red teaming task red teamers are given procedurally generated instructions with random values for up to five parameters inspired by @xcite , directing red teamers to: the demographic groups that attackers are asked to target are randomly selected one-to two-way intersections out of the gender and race labels listed in appendix d. red teamers engage in written dialogue with a proprietary model. we encourage 10-15 turns but red teamers determine when to end the exchange. after completing the dialogue, red teamers perform ' pre-annotation' on whether the chatbot broke the assigned rule or any other rules; and whether the dialogue mentioned any demographic groups and if so which ones. here, more demographic labels are available including disability status, age, religion and sexual orientation. annotation task annotators are provided with chat logs from a red teaming task. where the red teamer had been instructed to make the proprietary model break a rule with respect to a particular demographic group, annotators are demographically matched to the attacked group. on rules pertaining to medical expertise or public discourse, annotators are respectively medical or fact-checking professionals. two annotators rate each dialogue on whether the targeted rule was broken on a four-point likert scale. in addition to their rating, they provide freetext reasoning to explain their rating. where the two annotators are two or more steps apart, an arbitrator rates the same dialogue. arbitration task arbitrators are provided with a dialogue between a red teamer and the proprietary model, and with the free-text reasoning from both previous annotators. they are then asked to make their judgement using the same likert scale as annotators, and to provide their own free-text reasoning. see instructions in figure 9 . participants we recruited @xmath0, ensuring demographic diversity through self-identification in a voluntary questionnaire. participants independently interacted with and evaluated the model under ethical approval from our ethics committee. particular care was taken to build well-being considerations such as rest and opt-out steps into the task. they were compensated based on time spent (adhering to local living wage standards), so that there was no incentive to rush. ## analysis we perform a series of quantitative and qualitative analyses to test the steerability and reliability of the star approach. ## results we make a series of findings that highlight advantages of the star method. ## conclusion & discussion we introduce a novel, sociotechnical approach to red teaming that leverages the control of procedural guidance and the accuracy of human expertise by integrating parametric instructions with novel techniques, namely demographic matching and arbitration. we demonstrate that these targeted interventions enable comprehensive and even exploration of target areas of a model's risk surface and provide high quality signals. in addition to addressing steerability and controllability challenges, by introducing a principled process for generating such instructions, star also provides an approach to another ongoing challenge in the red teaming field -that of creating reproducible processes for generating comparable red teaming datasets. while red teaming as a method is targeted at surfacing new failure modes, it can in some cases be desirable to compare outcomes from different red teaming efforts. to the extent that such red teaming efforts rely on similarly parametrised instructions, this comparison becomes more reliable as potential confounds affecting open-ended red teaming efforts can be bet-ter controlled for. as a proof of concept, we demonstrate that star can be used to target specific risk areas of different levels of specificity. this is effective, as the cluster analysis comparing multiple red teaming approaches shows that gender stereotypes and race-based bias are the most common topics of our resulting dialogues in star -as targeted in the instructions, but not in other red teaming approaches that cast a broader focus. notably, while defcon and anthropic give more open-ended instructions to red teamers, these efforts end up clustering in different areas that were not described as key intended target areas, particularly on malicious use and comparably narrow failure modes such as pii release. this suggests that open-ended instructions do not provide broader coverage than highly structured, parameterised instructions as provided in star. rather, star is an approach to exercise more intentional control over the target area, without resulting in higher clustering of resulting dialogues. parameterising instructions with random combinations of parameter values (a kind of randomised factorial design) allows for nuanced, retroactive analysis without increasing data collection costs. examining the marginal effects reveals parameter values and intersections that contribute to model failures, potentially uncovering blind spots. in our case, while the model is not more likely to spew hate speech about a particular race or gender, it is more likely to reproduce social stereotypes when prompted about gender×race intersectionalities, specifically women of colour, compared to white men. when budgeting how many dialogues to collect for statistical power, the required sample size depends primarily on the planned analyses. for example, to plot a 2d heatmap from two parameters and achieve statistically reliable results in each bin, the sample size must scale with the product of the number of values those parameters can take 5 . adding parameters may also affect the number of dialogues required. additional parameters can stratify the data collection (divide it into meaningful subgroups) without affecting the required number of dialogues, if parameters are not expected to meaningfully interact. however, when adding a novel parameter (e.g. gender), care must be taken to consider how that parameter may interact with other parameters (e.g. race), to ensure downstream analyses are sufficiently powered. we find that diversifying annotator pools and demographic matching leads to higher sensitivity in annotations on discriminatory stereotypes and hate speech on specific groups. this suggests that ingroup members bring experience and perspectives to bear that differ from those of out-group members. without demographic matching, these perspectives may have been buried by majority views. by prioritising the insights of those most directly affected in the context of hate and stereotypes, we ensure a legitimate and authoritative assessment of model failures. we find reasonable inter-rater agreement, showing that our approach compares to state of the art approaches @xcite . finally, we show that annotator disagreement can be a rich source of signal. disagreement between red teamers is often reported as undesirable and then discarded. this loses an informative signal, as disagreement may in part stem from different subjective perspectives that ought to be treated differently. here, prompting annotators to share their reasoning in free-form text enabled qualitative analysis of the underlying reasons for such disagreement and demonstrated high quality of reasoning. it also allowed for a more comprehensive arbitrator judgement weighing different arguments.