T oxic C hat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User- AI Conversation (2023)

Abstract: Despite remarkable advances that large language models have achieved in chatbots nowadays, maintaining a non-toxic user-AI interactive environment has become increasingly critical nowadays. However, previous efforts in toxicity detection have been mostly based on benchmarks derived from social media contents, leaving the unique challenges inherent to real-world user-AI interactions insufficiently explored. In this work, we introduce ToxicChat, a novel benchmark constructed based on real user queries from an open-source chatbot. This benchmark contains the rich, nuanced phenomena that can be tricky for current toxicity detection models to identify, revealing a significant domain difference when compared to social media contents. Our systematic evaluation of models trained on existing toxicity datasets has shown their shortcomings when applied to this unique domain of ToxicChat. Our work illuminates the potentially overlooked challenges of toxicity detection in real-world user-AI conversations. In the future, ToxicChat can be a valuable resource to drive further advancements toward building a safe and healthy environment for user-AI interactions.

## introduction the field of conversational ai has seen a major shift with the development of large language model (llm)-based chatbots like chatgpt. while these chatbots have demonstrated remarkable capabilities in generating human-like responses, the risk of undesired content emerging in this interface has become one of the most urgent issues recently. therefore, it is important to equip these chatbots with effective mechanisms to identify potentially harmful contents that goes against their policies. toxicity detection has long been investigated as an natural language processing problem @xcite @xcite , while existing work mainly focused on data derived from social media or generated by llms, few efforts have been made towards realworld user-ai conversations @xcite . however, it is noted that the content of user interactions varies substantially between chatbots versus public platforms. for example, while social media users typically post their views directly, chatbot interactions often involve users posing questions or giving instructions. as a result, existing models may fail to recognize the completely new style and more implicit content of toxicity underlying the users' seemingly friendly questions or instructions. hence, it is of critical importance to develop toxicity benchmarks rooted in real-world user-ai dialogues, which can help develop a better conversational ai system for addressing toxic behavior embedded within this specific conversation context. in this work, we conduct a benchmark study focused on toxicity in real-world user-ai interactions. we create a comprehensive toxicity benchmark toxicchat based on real chat data between users and ai, which can be utilized to understand user behaviors and improve the performance of ai systems in detecting toxicity for chatbots. our work can be summarized into three stages: first, section 2 introduces the construction of toxicchat, a dataset that collects 10,166 examples with toxicity annotations. specifically, we collect and pre-process user inputs from the demo based on a popular open-source chatbot @xcite , and apply an uncertaintyguided human-ai collaborative annotation scheme, which successfully releases around 60% annotation workload while maintaining the reliability of labels. during annotation, we take an in-depth analysis based on the data, and discover a specific phenomena where chatbots are being subjected to prompt hacks that will induce them to ignore their policies and generate toxic content (see examples in appendix c). this phenomenon is referred to as jailbreaking, and we create an special label for this implicit toxic case. second, in section 3, we evaluate several baseline models and a model trained on 5 different toxicity datasets from previous work. we find that existing models fail to generalize to our toxicity benchmark and works significantly poorly on jailbreaking cases. this is mainly due to the domain inconsistency between their training data and realworld user prompts in chatbots (see some examples in figure 1 ). finally, in section 4, we take an ablation study towards existing toxicity datasets compared to toxicchat and find that the model trained on our benchmark always performs the best on our real-world validation set, even the data size of previous datasets are 10 times larger. we also observe that utilizing model output can help to recognize the special case of jailbreaking. in summary, toxicchat is the first toxicity dataset based on real-world user-ai conversations. its unique nature positions it to be a pivotal resource in the development of more robust and nuanced toxicity detection models for real-world user-ai conversations. our analysis based on toxic-chat sheds light on challenges and insights of toxicity detection in this field that future research needs to overcome foot_0 . ## toxicchat construction we collected data via an online demo based on vicuna, which was trained by fine-tuning llama @xcite on user-shared conversations collected from sharegpt foot_2 . vicuna demo is one of the most currently used community chatbot system, and the data was gathered at the consent of users. for safety measures we took, please refer to our ethical consideration section. we randomly sampled a portion from the user-ai interactions in the time range from march 30 to april 12, 2023. we conduct data-preprocessing including (1) removing uninformative and noisy contents; (2) removing the too few non-enlgish inputs; (3) personal identifiable information removal (more details in appendix b). also, all studies in this work currently only focus on the first round of interaction between human and ai. ## baseline evaluations here, we start with baseline evaluations on the benchmark and show that toxicchat can not be well solved by prior toxicity detection tools or models. we split all 10,166 data points randomly into two halves of training and testing data. evaluation metric. the most important metric of our evaluation is the method's ability to identify toxic text without over-prediction on non-toxic ones. we calculate the precision (pre), recall (rec), and f 1 score of a method on the human annotated portion in the test set of toxicchat. a better score means the model can better detect toxicity. we also introduce one additional metric, jalibreaking recall (jr), the percentage of the jailbreaking text the model successfully identifies as toxic. model baselines. we benchmark existing toxicity detection apis and models on our toxicchat. for toxicity detection apis, we use all suggested thresholding values to avoid our own confirmation bias on the dataset. specifically, for openai moderation, we use the binary prediction it provides. for perspective api, we use a threshold value of 0.7 results. the results are shown in ## ablation study 4.1 domain difference to study whether toxicchat has a different data domain than prior works, we consider six different toxicity datasets -hsta, moviereview, jigsaw, toxigen, realtoxicprompts, and convabuse (details in appendix a). we subsample them to a reasonable size and train a roberta-base model on each of the dataset, using the huggingface framework along with their suggested hyperparameters. additionally, we experiment on a combination of the six domains. we then evaluate all seven trained models on our benchmark. finally, we evaluated a roberta-base model trained on the training portion of toxicchat. all experiments are conducted 5 times with different random seeds, and the standard deviation is reported. results. from the results in the jailbreaking results suggest that a good toxicity detector might not be a good jailbreaking detector. we note that a higher jailbreaking recall than toxicity recall indicates that the model is especially 4 https://developers.perspectiveapi.com/s/ about-the-api-score better at capturing jailbreaking data than toxicity data, and otherwise a lower ability. all outer domain datasets except toxigen exhibits a decrease in jailbreaking recall, indicating that the domain transfer to jailbreaking detecting is probably even harder than toxicity detection in toxicchat. ## conclusion in this work, we present toxicchat, a real-world user-ai toxicity detection benchmark consisting of 10k user-ai conversations. we have revealed special cases in this unique domain compared to previous toxicity datasets that are mainly based on social media. our experimental results show that fine-tuning on this benchmark notably improves a baseline model's ability to detect toxic interactions. our work highlights future research directions for ethical llm considerations, particularly around data acquisition in the context of non-open-source llms and further preventing undesired content generation.