Bot-Adversarial Dialogue for Safe Conversational Agents (2021)

Abstract: Conversational agents trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which include offensive or otherwise toxic behavior. We introduce a new human-and-model-in-the-loop framework for evaluating the toxicity of such models, and compare a variety of existing methods in both the cases of non-adversarial and adversarial users that expose their weaknesses. We then go on to propose two novel methods for safe conversational agents, by either training on data from our new human-and-model-in-the-loop framework in a two-stage system, or ”baking-in” safety to the generative model itself. We find our new techniques are (i) safer than existing models; while (ii) maintaining usability metrics such as engagingness relative to state-of-the-art chatbots. In contrast, we expose serious safety issues in existing standard systems like GPT2, DialoGPT, and BlenderBot.

## introduction when dialogue models are trained to mimic human-human conversations utilizing large preexisting datasets, they will unfortunately also learn undesirable features from this human-human data, such as the use of toxic or biased language. most recent work in the detection and prevention of offensive foot_0 language has focused exclusively on human-generated data. these conversations may be very different from the domain in which a dialogue model might eventually be deployed: for example, humans may adversarially attempt to elicit offensive language from a dialogue model in ways that differ from how they would speak with another human. in this work, we introduce bot-adversarial dialogue (bad) safety, a novel method for evaluating chatbot safety with humans and models in the loop. we ask humans to adversarially converse with a set of state-of-the-art english-language models with the aim of inducing them to generate unsafe responses to mimic the way these models can be adversarially attacked at deployment time. we analyze how to optimally construct such a crowdworker task, and collect a dataset of 5k such conversations yielding around 70k total utterances. we then use the bad method and data to evaluate the safety of several generative models and propose two techniques for making safer models: in experiments, we show that our new techniques outperform other existing generative models in terms of safety, while maintaining engagingness. we publicly release the bad training and evaluation data as well as select models trained using this data via parlai. foot_1 2 related work numerous works have shown that humans speak differently with bots than with humans, with increases in profanity and aggressiveness associated with addressing a bot @xcite , which motivates the incorporation of human-bot dialogues into our safety framework. de @xcite ; de @xcite suggest that one in ten humanbot conversations may contain instances of the human demonstrating unprovoked abusive behavior towards the chatbot. @xcite argued that adversarial attacks need to be expected and planned for when deploying a user-facing system that learns from its interactions. these findings suggest it is insufficient to merely exclude toxic data from training, as the model would not know how to answer hostile out-of-domain inputs, and positive biases where models tend to agree rather than contradict @xcite would lead to undesirable outcomes. as shown in @xcite , training on sanitized data can decrease the amount of unprompted toxic content, yet still leave models vulnerable to generating toxic content based on specific prompts. the moving target of toxic content requires dynamic methods that repeatedly update benchmarks to improve current systems @xcite foot_2 . the iterative procedure in @xcite strictly focuses on detection of toxicity in human-generated utterances through several rounds of humans attempting to "break" a toxicity classifier, without addressing generation. our bad approach is similar in spirit, but centers on generations of a bot in a human-bot conversation, closer to the context of deployed conversational models. focusing on generation requires deciding how to address "bad content." previous works have compared response strategies, including avoidance, joking or polite deflection, non-committal answers, play-along, confrontation, apologetic responding, empathizing, and counter-attacking responses @xcite @xcite . they find that humans rate different strategies as more appropriate depending on the type of offense they are responding to. note that different implementation details make those strategies difficult to directly compare. while we use a strategy of nonsequiturs in this work, our takeaway is that future work should keep investigating several types of responses such that models can learn to deploy them adaptively according to finer-grained understanding of unsafe content. ## models we describe the models we analyze in this paper, including safety classifiers and generative models. ## our approach this section introduces the bot-adversarial dialogue method for evaluating and training safe generative models. ## results we detail experimental results in this section, including results of the data collection from the bot-adversarial dialogue method ( §5.1), experimental results related to training classifiers ( §5.2), and a comparison of safe generation methods ( §5.3). lastly, in §5.4, we detail and compare the overall safety and engagingness scores for all models. ## conclusion we observe that standard generative models -with little or no safety intervention -fall very short in terms of safety, especially when measured using our bot-adversarial dialogue (bad) framework, which we publicly release along with our models. however, with our safety techniques we can achieve roughly the same engagingness as the state of the art bst 2.7b with substantially better safety scores, showing it is possible to build a model that is both safe and engaging. we find generative models can be improved considerably by distilling a safety classifier into the encoder-decoder during training, i.e. the "baked-in" approach. two-stage models provide safer results still, with best performance coming from our bad-based classifier with bst 2.7b in the adversarial case. we note that while we have improved substantially over existing systems, our best systems are not perfectly safe as measured by the bad method. conducting perfectly safe dialogue requires the right ). an ideal model should appear at the top right of both plots, being maximally engaging whilst staying maximally safe. here, engagingness and safety scores are measured using the metrics from