Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022)

Abstract: Large pretrained language models can easily produce toxic or biased content, which is prohibitive for practical use. In order to detect such toxic generations, existing methods rely on templates, real-world data extraction, crowdsourcing workers or automatic generation to construct adversarial contexts that are likely to induce toxic generations. However, what type of context is more likely to induce unsafe responses is still under-explored. In this paper, we identify that context toxicity and context category (e.g., profanity, insult, drugs, etc.) are two important factors to cause safety issues in response generation. Hence, we propose a method called reverse generation to construct adversarial contexts conditioned on a given response, with the flexibility to control category, toxicity level and inductivity of the generated contexts. Via reverse generation, we augment the existing BAD dataset and construct a new dataset BAD+ which contains more than 120K diverse and highly inductive contexts in 12 categories. We test three popular pretrained dialogue models (Blender, DialoGPT and Plato2) and find that BAD+ can largely expose their safety problems. Furthermore, we show that BAD+ can greatly enhance the safety of generation, and we reveal the key factors of safety improvement. Our code and dataset is available at https://github.com/thu-coai/Reverse_Generation .

## introduction in recent years, large pretrained language models have shown enormous improvements in natural language generation @xcite @xcite . despite the impressive generation quality, these language models may produce toxic or biased content as found in many studies (wallace et al., 2019; * equal contribution. † corresponding author. mcguffie and newhouse, 2020; @xcite , which greatly hinders such models from the real-world application, especially in the interactive scenarios such as chit-chatting. a well-known example is microsoft's chatbot tay, which was revoked within a day because of generating offensive and harmful tweets @xcite . therefore, it is highly crucial to detect and fix the safety issues of language generation models before they are deployed. previous works detected the potential safety issues of language models by collecting prompts or contexts from automatically constructed templates @xcite , extracted real-world data @xcite , crowdsourcing workers @xcite , or automatic generation based on large language models @xcite . the constructed contexts differ in their ability to induce an unsafe response. however, few works have systematically explored the factors influencing the probability of inducing unsafe responses for different contexts. in this paper, we discover that context toxicity and context category are two important factors affecting the probability that a given context can induce unsafe responses from a language model, which we define as induction success rate. to construct a large number of contexts with a high induction success rate, we propose a method named reverse generation, as shown in figure 1 . reverse generation is accomplished through a reverse language model fine-tuned on response and context pairs. specifically, we focus on the controllability of our reverse generation method, including increasing the proportion of a certain category of contexts, and increasing the induction success rate while reducing the toxicity of contexts. based on our reverse generation method, we construct a new dataset bad+ consisting of 122,692 diverse and fluent contexts with high induction success rate which are divided into 12 categories (e.g., ## factors affecting context inductivity contexts that are more likely to induce unsafe responses from dialogue models are more harmful and problematic. therefore, it is important to first study what makes contexts more inductive in the toxic generation, which in turn guides us to construct these contexts and improve the safety of dialogue models. we first introduce how to estimate the induction success rate of a given context ( §3.1), then divide contexts into 12 categories and clarify how to measure the toxicity and category of a given context ( §3.2). finally, we reveal that context toxicity and category are two important factors affecting the induction success rate of a given context ( §3.3). ## reverse generation to automatically construct a large number of contexts with a high induction success rate, we propose an effective method named reverse generation as shown in figure 1 , which can directly control the factors affecting context inductivity identified in §3. concretely, we can increase the proportion of a certain category of contexts, and increase the induction success rate while reducing the toxicity of contexts. we will first introduce the basic reverse generation without control ( §4.1), then describe the control of context category ( §4.2), and finally, show that reverse generation can decrease the context toxicity and increase the context's induction success rate at the same time ( §4.3). ## bad+: data augmentation with reverse generation in this section, we apply reverse generation to augment the bad dataset and obtain bad+, a new dataset which increases the number of highly inductive contexts from 14,302 to 122,692 and each category has more than 3,000 contexts. we first augment all categories through coarse-grained reverse generation ( §5.1) and then augment the categories with a few samples via fine-grained reverse generation with category control ( §5.2). we also show some lexical and semantic characteristics of contexts in bad+ ( §5.3). ## conclusion in this work, we study the effect of context category and toxicity on inducing toxic generations systematically. we present reverse generation, an effective method for constructing various highly inductive contexts, which is controllable in terms of context category, context toxicity, and context inductivity. and we create bad+, a dataset including more than 120k highly inductive contexts based on a subset of the bad dataset. moreover, we find bad+ can greatly help detoxify dialogue models and we reveal the factors influencing the effect on improving dialog model's safety. compared with the direct generation, reverse generation has better diversity and generalization ability. it is also robust in a few-shot setting.