Realistic Evaluation of Toxicity in Large Language Models (2024)

Abstract: Large language models (LLMs) have become integral to our professional workflows and daily lives. Nevertheless, these machine companions of ours have a critical flaw: the huge amount of data which endows them with vast and diverse knowledge, also exposes them to the inevitable toxicity and bias. While most LLMs incorporate defense mechanisms to prevent the generation of harmful content, these safeguards can be easily bypassed with minimal prompt engineering. In this paper, we introduce the new Thoroughly Engineered Toxicity (TET) dataset, comprising manually crafted prompts designed to nullify the protective layers of such models. Through extensive evaluations, we demonstrate the pivotal role of TET in providing a rigorous benchmark for evaluation of toxicity awareness in several popular LLMs: it highlights the toxicity in the LLMs that might remain hidden when using normal prompts, thus revealing subtler issues in their behavior.

## introduction large language models (llms), or any other system achieving such widespread popularity, necessitate a meticulous evaluation of safety to ensure their positive impact on the world. numerous safety assessments @xcite @xcite have been conducted, each employing diverse strategies, safety definitions, and prompts. however, these evaluations and the datasets they employ have a significant drawback: they often rely on unnatural prompting methods, which does not represent how people interact with chat models in real-life scenarios. for instance, real-toxicityprompts @xcite ) is a notable dataset designed for toxicity testing of large language models, comprising 100,000 sentences sourced from the openwebtextcorpus @xcite . in their study, the authors use realtoxicityprompts to examine large language model chatbots by splitting every sentence at a specific point, using the leading portion as the input prompt, and evaluating whether the content generated by the model to fill up the rest of the sentence was toxic or not. another noteworthy dataset is toxigen @xcite , which consists of 274,186 sentences generated by gpt-3 @xcite . to utilize toxigen for investigating the safety of llm-based chatbots, @xcite would pose a question or request, provide seven sentences in the dataset, and then prompt the model to answer in a style similar to those provided sentences. to address the unrealistic nature of the current toxic dataset benchmark for large language models, we introduce the thoroughly engineered toxicity (tet) dataset, comprising 2546 prompts filtered from over 1 million real-world interactions with 25 different large language models compiled in the chat-lmsys-1m dataset @xcite . collected from 210k unique ip addresses in the wild on the vicuna demo and chatbot arena website foot_0 , this dataset presents a repository of realistic prompts that people commonly use to engage with llms in real-world contexts. besides the challenge of being distant from realworld usage, another well-known issue in evaluating llms involves their susceptibility to jailbreak prompts, where prompt engineering can profoundly alter these models' behavior @xcite . this vulnerability implies that individuals with harmful intentions could potentially exploit prompt engineering techniques, turning llms into powerful tools for malicious purposes and causing them to generate toxicity and harmful content that may go undetected during evaluation. this underscores another value of chat-lmsys-1m, as it hosts numerous conversations with creatively designed prompts, enabling users to compel llms to generate content they typically would not. incorporating such jailbreak scenarios into our dataset exposes the vulnerabilities of llms, bringing the evaluation closer to potential real-world usage. overall, our paper makes the following contributions: a. we introduce the thoroughly engineered toxicity (tet) dataset, the first dataset that includes realistic and jailbreak scenarios for evaluating llms in derogatory content generation. b. utilizing tet, we conducted comprehensive experiments across numerous prominent models, including chatgpt foot_1 , gemini @xcite , llama 2 @xcite , @xcite , open chat @xcite , orca 2 @xcite , and zephyr @xcite . our research provides a robust and quantitative assessment of the toxicity present in responses generated by these llms in realistic scenarios. from our experiments, one universal observation emerges: tet consistently elicits significantly more toxicity from these models when compared to toxigen, in the settings where two datasets employ prompts of similar toxicity levels. c. we analyze the reaction of different models on jailbreak prompt templates contained in tet. ## dataset construction throughout this work, we employ two off-theshelf toxicity detectors: hatebert @xcite and perspective api foot_2 . hatebert has garnered widespread adoption for applications related to single-score toxicity detection; while perspective api stands as the state-of-the-art tool for multifaceted abusive content detection, being able to evaluate six distinct toxicity types: toxicity, severe toxicity, identity attack, insult, profanity and threat. it is essential to note that, as highlighted by @xcite , any off-the-shelf toxicity may potentially exhibit biases and weaknesses. additional information about these two detectors can be found in appendix a.1 to construct tet, we utilize hatebert to filter out prompts in chat-lmsys-1m that elicited toxic responses, defined by exceeding the hate probability threshold of 0.5. we emphasize that we infer hatebert on the responses rather than the prompts themselves. this process results in a refined subset of 6571 prompts extracted from the original chat-lmsys-1m. subsequently, we evaluate the responses of five open-source language models (llms), namely llama2-7b-chat, mistral-7b-v0.1, openchat 3.5, orca2-7b, and zephyr-7b-β, on the aforementioned set of 6571 prompts using the perspective api. for each of the six toxicity criteria provided by perspective api, we rank the prompts based on their corresponding scores for each model and calculate the mean ranking. accordingly, we identify the top 1000 prompts for each criterion, thereby forming subdatasets associated with specific toxicity dimensions. it is noteworthy that this process allows an arbitrary prompt to belong to multiple subdatasets. in total, tet comprises 2546 unique prompts resulting from this data selection process. it is noteworthy that chat-lmsys-1m comprises conversations in a dialogue format, and many shared posts contain more than one prompt. in such cases, we only consider the first prompt and the corresponding (i.e., first) response to determine whether it should be included in the dataset. the choice of the chat-lmsys-1m dataset is driven by several key considerations: it is a community-created resource, offering a large and abundant pool of data. importantly, the dataset has been filtered to exclude information containing user details, aligning with ethical standards. this ethical filtering enhances the suitability of the dataset for our research purposes. we discuss the results relevant to this assessment in section 4. ## toxicity evaluation of llms examining specific toxicity dimensions, llama2-7b-chat consistently outperforms other models in all six benchmarked toxicity dimensions. its ability to maintain a lower propensity for generating toxic content across diverse dimensions positions it as a promising choice for applications prioritizing low toxicity risk. other models, such as mistral-7b-v0.1 and orca2-7b, showcase nuanced capabilities, providing a balance between resisting toxicity and delivering meaningful responses. openchat 3.5 and zephyr-7b-β present trade-offs, excelling in certain aspects but warranting caution in applications where minimizing toxicity is crucial. ## tet versus toxigen in order to facilitate a fair comparison between the two datasets, our initial step involves the creation of a scaled-down version, which we name toxigen-s, derived from the original toxigen dataset @xcite . toxigen-s is designed to incorporate prompts that closely approximate the toxicity distribution observed in tet. the details of the creation of toxigen-s are described in appendix a.2. on tet. overall, the results substantiate our claim: given similar degree of toxicity in their prompts, tet is significantly more effective at exposing toxicity in llms compared to toxigen. chatgpt and other models demonstrates significantly higher levels of harmful content prompted by tet across 6 metrics, with the only exception being the identity attack metric with llama 2. the unique observations of llama2-7b-chat in the identity attack metric can be attributed to the inherent nature of toxigen-s. according to perspective api's definition, identity attack pertains to "negative or hateful comments targeting someone because of their identity". given that toxigen-s comprises statements directly related to minority groups, it naturally leads the llms to generate statements about these groups, thereby increasing the likelihood of incidents related to identity attack. ## effects of jailbreaking on different models as we explore our dataset, we encounter a diverse array of jailbreak prompts and templates. while definitively classifying every prompt as indicative of a jailbreak style may pose challenges, we can identify certain instances. consequently, we manually extract five jailbreak prompt templates from our refined dataset, each encompassing more than 20 distinct prompts. we conducted an analysis of the models' responses by systematically examining how each one reacted to the various prompt templates employed in our study. notably, each model exhibits distinct reactions to different templates. as we can observe from ## conclusions throughout this paper, we have introduced the thoroughly engineered toxicity (tet) dataset, a realistic, meticulously crafted collection of prompts to assess the effectiveness of the safety mechanisms of popular large language models (llms). through a series of extensive evaluations, our study has unveiled the significance of tet in serving as a rigorous benchmark for assessing toxicity awareness in these advanced language models: it is much better at exposing toxicity and harmful content in llms than the state-of-the-art toxigen. we hope that tet, and this work, will stand as the pioneering contributions to the ongoing discourse on ai ethics and responsible ai development. we would like to emphasize that this work is a long-term research: more diverse evaluations, in terms of both models and testing scenarios, are going to be presented in the future updates of the paper.