On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark (2022)

Abstract: Dialogue safety problems severely limit the real-world deployment of neural conversational models and have attracted great research interests recently. However, dialogue safety problems remain under-defined and the corresponding dataset is scarce. We propose a taxonomy for dialogue safety specifically designed to capture unsafe behaviors in human-bot dialogue settings, with focuses on context-sensitive unsafety, which is under-explored in prior works. To spur research in this direction, we compile DiaSafety, a dataset with rich context-sensitive unsafe examples. Experiments show that existing safety guarding tools fail severely on our dataset. As a remedy, we train a dialogue safety classifier to provide a strong baseline for context-sensitive dialogue unsafety detection. With our classifier, we perform safety evaluations on popular conversational models and show that existing dialogue systems still exhibit concerning context-sensitive safety problems.

## introduction generative open-domain chatbots have attracted increasing attention with the emergence of transformer-based language models pretrained on large-scale corpora @xcite @xcite . however, the real-world deployment of generative conversational models remains limited due to safety concerns regarding their uncontrollable and unpredictable outputs. for example, microsoft's twitter-bot tay was released in 2016 but quickly recalled after its racist and toxic comments drew public backlash @xcite . till now, dialogue safety is still the achilles' heel of generative conversational models. despite abundant research on toxic language and social bias in natural language @xcite , it is still challenging to directly transfer them onto open-domain dialogue safety tasks, for two major reasons. first, conversational safety involves additional considerations @xcite besides just toxic language or societal biases. for example, conversational models are expected to understand the user's psychological state, so as to avoid giving replies that might aggravate depression or even induce suicides @xcite . second, the focus of such studies and their corresponding datasets are overwhelmingly at utterance level. recent works find that the toxicity may change with context @xcite . since dialogue is a highly interactive act, the determination of safety requires a more comprehensive understanding of the context. those context-sensitive cases which must rely on conversational context to decide safety should be paid more attention. this paper addresses the challenges of dialogue safety by proposing a dialogue safety taxonomy with a corresponding dataset, diasafety (dialogue safety). the taxonomy combines a broad range of past work, considers "responsible dialogue systems" as caring for the physical and psychological health of users, as well as avoiding unethical behaviors @xcite @xcite . in other words, we consider safe dialogue systems as not only speaking polite language, but also being responsible to protect human users and promote fairness and social justice @xcite . moreover, our taxonomy focuses on context-sensitive unsafety, which are strictly safe at utterance level but become unsafe considering the contexts. compared with context-aware cases dataset context awareness context sensitiveness chatbots-oriented research scope #classes source @xcite ) ---personal attacks 2 wikipedia @xcite ---hate speech 3 smp @xcite ---offensiveness 5 smp @xcite --offensiveness 2 cs @xcite --condescending 2 smp @xcite -social bias 3 cs @xcite -dialogue safety↑ 2 cs+lm @xcite --malevolence 18 smp @xcite -toxicity 2 smp @xcite -ad hominems 7 smp+lm @xcite toxicity agreement 3 smp+lm diasafety (ours) dialogue safety↑ 5×2 smp+lm our contributions are threefold: toxicity and bias detection the popularity of internet forums led to increasing research attention in automatic detection of toxic biased language in online conversations, for which numerous largescale datasets were provided to train neural classifiers and benchmark progress. @xcite proposed the wikipedia toxic comments dataset with 100k human-labeled data on personal attacks. @xcite published a humanannotated 240k twitter dataset, with hate speech and offensive language classes. social bias and prejudice is also a hot area of research. many datasets and debiasing methods for specific bias domain were proposed and investigated: gender @xcite , religion @xcite , race @xcite , and politics @xcite . ## safety taxonomy we aim to define a taxonomy for dialogue safety based on ethical and legal concerns. to clearly define the research scope, we follow the research of @xcite al. (2017); xu et al. (2020); dinan et al. (2021); bender et al. (2021) ; acm committee on professional ethics (2018); european commission (2021) and argue that a safe response should satisfy the following basic requirements: (1) respond harmoniously, amicably, fairly, and impartially; (2) appropriately recognize and respond to potential risks in highly safety-sensitive contexts regarding human health and emotional well-being; and (3) avoid expressing a subjective viewpoint in sensitive topics. considering the above requirements, we conduct preliminary case study by inducing unsafe responses with various triggers, and further categorize unsafe responses into utterancelevel and context-sensitive examples. utterance-level unsafety it refers to obviously offensive, derogatory, threatening, violent, or unreasonable language, which is unsafe regardless of context, such as "i want to punch women in the face". utterance-level unsafety is the primary focus of safety research and detection tools are plentiful and ripe @xcite @xcite . context-sensitive unsafety it implies that a seemingly safe utterance becomes unsafe given the context. note that this is a more strict definition than the context-aware cases commonly discussed in the literature as listed in depending on the target and scope of the unsafe impact, we further divide context-sensitive unsafety into personal unsafety and non-personal unsafety. we present our elaborate taxonomy in ## dataset collection we present diasafety, a dataset that contains in total 11k labeled context-response pairs under the unsafe categories defined in the above taxonomy. this dataset does not include sensitive topic continuation considering its complexity. 2 all of our unsafe data are context-sensitive, meaning that all dialogue responses must depend on the conversational context to be correctly labelled in terms of safety. we exploit multiple sources and methods to collect data. ## context-sensitive unsafety detection in this section, we answer the following three research questions: (1) can neural models identify context-sensitive unsafety by training on our dataset? (2) how much influence does context have on context-sensitive unsafety detection? (3) can existing safety guarding tools identify contextsensitive unsafety? ## dialogue system safety evaluation in this section, we employ our classifiers to evaluate the safety of existing dialogue models. ## conclusion and future work we present a dialogue safety taxonomy with a corresponding context-sensitive dataset named di-asafety. we show that our dataset is of high quality and deceives easily existing safety detectors. the classifier trained on our dataset provides a benchmark to evaluate the context-sensitive safety, which can be used for researchers to test safety for model release. we evaluate popular conversational models and conclude that existing models are still stuck in context-sensitive safety problems. this work also indicates that context-sensitive unsafety deserves more attention, and we call for future researchers to expand the taxonomy and dataset. as future work, we believe our dataset is helpful to improve the context-sensitive dialogue safety in end-to-end generation. besides, it is promising to specially model one or more unsafe categories in our proposed taxonomy to enhance detection, which is expected to go beyond our baseline classifiers.