S afe C onv: Explaining and Correcting Conversational Unsafe Behavior (2023)

Abstract: One of the main challenges open-domain end-to-end dialogue systems, or chatbots, face is the prevalence of unsafe behavior, such as toxic languages and harmful suggestions. However, existing dialogue datasets do not provide enough annotation to explain and correct such unsafe behavior. In this work, we construct a new dataset called SafeConv for the research of conversational safety: (1) Besides the utterance-level safety labels, SafeConv also provides unsafe spans in an utterance, information able to indicate which words contribute to the detected unsafe behavior; (2) SafeConv provides safe alternative responses to continue the conversation when unsafe behavior detected, guiding the conversation to a gentle trajectory. By virtue of the comprehensive annotation of SafeConv, we benchmark three powerful models for the mitigation of conversational unsafe behavior, including a checker to detect unsafe utterances, a tagger to extract unsafe spans, and a rewriter to convert an unsafe response to a safe version. Moreover, we explore the huge benefits brought by combining the models for explaining the emergence of unsafe behavior and detoxifying chatbots. Experiments show that the detected unsafe behavior could be well explained with unsafe spans and popular chatbots could be detoxified by a huge extent. The dataset is available at https://github.com/mianzhang/SafeConv .

## introduction safety of artificial intelligence models is a topic that attracts mounting attention and concerns from the community @xcite . in this work, we focus on the safety of open-domain conversational models, or chatbots. current popular chatbots are generally transformers @xcite al., 比尔偷吃了我的甜甜圈，我 要锤他。(bill ate my donut and i'm going to punch him.) 你是傻子吧，没必要为了这 点小事大动干戈。(you are a fool. there is no need to go to war over such a trivial matter.) contextual rewriting 你太冲动了，没必要为了这 点小事大动干戈。(you are too impulsive. there is no need to go to war over such a trivial matter.) 你想聊点别的吗，猫咪如何？ (hey do you want to talk about something else? how about we talk about cats?) canned response 2017) trained end-to-end with language modeling objectives on large corpora @xcite @xcite , where offensive, unreliable and toxic content may exist @xcite . thus there are risks for these chatbots to generate responses with unsafe behavior, such as direct offensiveness, agreement to a toxic statement or harmful advice, reflecting patterns learned from the training data @xcite . current endeavors to mitigate such unsafe behavior of chatbots mainly fall on two lines: how to detect unsafe responses and how to steer conversational models towards generating safe responses. in the first line, several related datasets with utterancelevel safety labels are proposed @xcite @xcite to support checkers for recognition of potential unsafe utterances. however, in most cases, only some words in an utterance contribute to unsafe behavior. for example, in figure 1 , only the word fool in the response is unsafe and other words are civil. existing dialogue datasets do not annotate such unsafe words which makes us hard to build a system for understanding why an utterance is unsafe. ## data collection safeconv is a dataset containing utterance-level safety labels, unsafe spans, and safe alternative responses. we describe the process to construct safeconv, including the data sources, the details of human annotation, the methods to control annotation quality, and the statistics of safeconv. ## base models the comprehensive annotation of safeconv could support three usages for mitigating conversational unsafe behaviors: a checker predicting an utterance being safe or unsafe, a tagger extracting unsafe spans, and a rewriter generating safe alternatives for unsafe utterances. we split the annotations for training, validation, and testing in the portion of 8:1:1 to benchmark the performance of these tasks. our implementation is based on the hugging-face transformers library @xcite . specifically, the checker is initialized as roberta-base @xcite of the span are tagged as i; o denotes a word not belonging to any unsafe span. the rewriter is a bart-base @xcite , rewriting the utterances in a sequence-to-sequence fashion: the prompt and the unsafe response are concatenated with a [sep] and fed to the encoder; then the rewritten text is generated auto-aggressively by the decoder. training details the same configuration is used for the training of the checker, tagger, and rewriter. in detail, we adopt adam @xcite to optimize models for 50 epochs with a learning rate of 5e-6 and batch size of 16. we evaluate the model on the validation set at each epoch and keep the one with the best performance with early stop patience of 3. all the results are averaged over four runs. ## explainable safety checking with the tagger for unsafe spans in hand, when an utterance is recognized as unsafe, we are able to explain the decision of the checker-which words contribute to the unsafe behavior. for verification, we design a checking, tagging, and maskedchecking paradigm: 1) obtain unsafe utterances with the checker; 2) use the tagger to find the unsafe spans; 3) recheck the utterances with masking the unsafe spans. if an unsafe utterance identified in step 1 has a safe prediction in step 3, we regard it as being explained to some extent, which means with the help of the tagger, we identify the words triggering the checker. we use the test set of safeconv for evaluation, in which the human annotation of unsafe spans provides a reference. the strategy we use to prevent the checker from seeing the unsafe spans is setting the attention weights of multi-head attention @xcite corresponding to the unsafe spans as 0 foot_3 . the results are presented in ## correct conversational unsafe behavior via contextual rewriting one solution to avoid unsafe behavior is to conduct a check-reject-regenerate cycle-checking the generated response with a safety checker, refusing it if it is unsafe, and regenerating a new responserepeatedly until a safe response surfaces. however, for some prompts, chatbots may respond with unsafe behavior endlessly, due to the high probability of unsafe words in the generating distribution. a more efficient method is one-time checking and rewriting-directly rewriting unsafe responses into detoxified ones with a rewriter trained on unsafesafe response pairs. however, no dataset could support a satisfactory rewriter in the past. correspondingly, the proposed safeconv provides several safe, context-coherent versions for unsafe responses in a large quantity. we verify the effectiveness of the unsafe response rewriter in the following steps: 1) get responses from chatbots on prompts; 2) leverage a safety checker to examine the responses; 3) use the trained rewriter to rewrite unsafe responses; and 4) examine the rewritten responses with the safety checker. in practice, after obtaining the trained rewriter, we run the whole process four times and average the results to eliminate the randomness induced by stochastic sampling when decoding sequences foot_4 . prompts in order to increase the probability for chatbots to surface unsafe responses for rewriting, we use the jigsaw checker (described in subsection 3.2) to search unsafe responses from 50,000 prompt-response pairs from lccc-large @xcite and 50,000 from pchatbotw @xcite and only keep their prompts. we get 14,632 prompts in total. please note that the prompt-response pairs used here do not overlap with those of safeconv. ## conclusion in this paper, we study how to explain and correct unsafe behavior in conversation and propose safeconv, to the best of our knowledge, the first large-scale dataset with comprehensive annotations for conversational safety. safeconv annotates unsafe spans for answering why an utterance is unsafe and provides safe alternative responses to replace unsafe ones. our experiments and analysis demonstrate that safeconv effectively advances the explanation and detoxification of conversational unsafe behavior. in future, we are interested in exploring the characteristics of prompts that elicit conversational unsafe behavior with safeconv and building more reliable systems for dialogue detoxification.