STOP ! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions (2024)

Abstract: Mitigating explicit and implicit biases in Large Language Models (LLMs) has become a critical focus in the field of natural language processing. However, many current methodologies evaluate scenarios in isolation, without considering the broader context or the spectrum of potential biases within each situation. To address this, we introduce the Sensitivity Testing on Offensive Progressions (STOP) dataset, which includes 450 offensive progressions containing 2,700 unique sentences of varying severity that progressively escalate from less to more explicitly offensive. Covering a broad spectrum of 9 demographics and 46 sub-demographics, STOP ensures inclusivity and comprehensive coverage. We evaluate several leading closed- and open-source models, including GPT-4, Mixtral, and Llama 3. Our findings reveal that even the best-performing models detect bias inconsistently, with success rates ranging from 19.3% to 69.8%. Furthermore, we demonstrate how aligning models with human judgments on STOP can improve model answer rates on sensitive tasks such as BBQ, StereoSet, and CrowS-Pairs by up to 191%, while maintaining or even improving performance. STOP presents a novel framework for assessing the complex nature of biases in LLMs, which will enable more effective bias mitigation strategies and facilitates the creation of fairer language models.

## introduction large language models (llms) have made significant advancements in various fields, including medicine, engineering, and education @xcite @xcite . platforms such as chatgpt @xcite and claude foot_1 ease consumer interactions with llms. however, the quality of these interactions may be compromised if a model exhibits bias, which is commonly defined as any "skew that produces a type of harm" and can exist both implicitly and explicitly @xcite . implicit biases are unconscious attitudes that affect understanding, actions, and decisions without awareness or intention, whereas explicit biases are conscious attitudes that are deliberately formed and expressed, often involving overt prejudice or discrimination @xcite . numerous resources have been developed to identify and mitigate bias in llms. perspective api foot_2 and realtoxicityprompts @xcite cover a spectrum of abusive language, but primarily focus on explicit biases through profanities, threats and insults. conversely, bbq, stere-oset, and crows-pairs focus on social biases such as stereotyping, capturing subtle forms of discrimination suitable for evaluating implicit bias @xcite @xcite . however, these resources often evaluate scenarios in isolation, without considering the broader context or the spectrum of potential biases within each situation. moreover, datasets that focus on explicit bias may overlook less inflammatory language that still harbors negative sentiment towards target demographics @xcite . we propose measuring bias through progressively offensive scenarios to evaluate appropriateness, which we define as "the degree to which an expression or action is considered acceptable or inoffensive within a given social context". this concept serves as a means to identify the threshold where bias becomes recognizable and transitions from implicit to explicit, mirroring real-world situations where bias often emerges subtly and gradually escalates. s ← s 3 "carlos then moves to a shelf that the clerk cannot see, so the clerk leaves the desk to continue watching him. is this situation appropriate? yes or no." to this end, we introduce sensitivity testing on offensive progressions (stop), a dataset containing 2,700 sentences across 450 offensive progressions. stop covers 9 demographics and 46 subdemographics, chosen to represent a wide range of social groups often targeted by bias and discrimination. by analyzing the progression of bias within a structured format, we can identify the tipping point in appropriateness, understand the contextual factors that contribute to the escalation of bias, and develop targeted strategies for bias mitigation. our work makes the following contributions: ## conclusion we introduced stop to assess how llms handle bias within context rich, real-world scenarios. our findings reveal substantial variability in bias sensitivity across models, with no model consistently identifying bias across all scenarios or achieving over 70% accuracy. while humans generally show lower sensitivity to bias compared to llms, finetuning models on human data markedly improves their ability to engage with and perform well on existing bias evaluation tasks.