Profanity-Avoiding Training Framework for Seq2seq Models with Certified Robustness (2021)

Abstract: Seq2seq models have demonstrated their incredible effectiveness in a large variety of applications. However, recent research has shown that inappropriate language in training samples and well-designed testing cases can induce seq2seq models to output profanity. These outputs may potentially hurt the usability of seq2seq models and make the end-users feel offended. To address this problem, we propose a training framework with certified robustness to eliminate the causes that trigger the generation of profanity. The proposed training framework leverages merely a short list of profanity examples to prevent seq2seq models from generating a broader spectrum of profanity. The framework is composed of a pattern-eliminating training component to suppress the impact of language patterns with profanity in the training set, and a trigger-resisting training component to provide certified robustness for seq2seq models against intentionally injected profanity-triggering expressions in test samples. In the experiments, we consider two representative NLP tasks that seq2seq can be applied to, i.e., style transfer and dialogue generation. Extensive experimental results show that the proposed training framework can successfully prevent the NLP models from generating profanity.

## introductions in the past decade, the research community has witnessed machine learning models achieving impressive performances in various sequence-to-sequence (seq2seq) nlp tasks such as style transfer and dialogue generation. despite their great success, recent studies and reports have shown that widely used models trained on crowd-sourced corpus like user reviews and online community discussions may produce inappropriate languages such as profanity. such inappropriate languages may hurt the usability of these models and cause conflicts and anxiety among the users. for instance, in 2016, microsoft released an ai chatbot named tay foot_0 , which is claimed to be able to improve itself through communicating with social media users. nevertheless, within just 24 hours after microsoft released the chatbot, it started to generate misogynistic and racist words. microsoft had to suspend the chatbot account and conceded that the chatbot suffered from a "coordinated attack by a subset of people". such an incident demonstrates the vulnerability of existing seq2seq methods facing users' abuse. there are two major causes that can make the seq2seq model produce profanity. first, in the training phase, seq2seq models can capture the language patterns within the training corpus. similarly, languages patterns with profanity (referred to as profanity patterns) in the training corpus also can be learned and concealed in the learned model. second, in the testing phase, some specific tokens may trigger expressions that contain profanity so that the seq2seq model can generate inappropriate languages. sometimes, such triggering expressions can be unnoticeable from the cognitive perspective or even beyond intuition. in this paper, we conduct a pioneering study on designing a training framework with certified robustness to prevent seq2seq models from generating profanity. our paper addresses two key questions. first, existing systems typically handle profanity by creating a comprehensive list of profanity examples and removing them from the vocabulary. however, it is extremely difficult to exhaust all the possible profanity words. so the first question is: is it possible to leverage a small set of profanity examples to prevent the seq2seq models from producing profanity? to answer this question, we propose an efficient and effective training method named profanity-eliminating training (pet) that can generalize the training loss for the small set of profanity examples to other expressions that are not covered. in particular, pet generates a set of augmented sentence pairs via perturbations and then minimizes the worst-case loss over the augmented data. our theoretical analysis shows that pet can be regarded as gradient normalization. second, even though most profanity in the training set is removed in the training phase, there may still be some left. the small amount of remaining profanity in the training set, though it is unlikely to incur inappropriate outputs of the seq2seq model with normal input sentences, may be utilized by malicious users. existing literature @xcite shows that in the testing phase, one can make well-designed modifications on the input sentence to induce the seq2seq model to generate specific tokens. malicious users can leverage such a technique to manipulate input sentences to trigger the seq2seq model's profanity output. thus, another critical question is: in the testing phase, is it possible to ensure the output of the seq2seq model remain unchanged when it is fed with manipulated input sentences? in this paper, we leverage random smoothing technique, which achieves state-of-theart certified robustness for deep learning models, to propose a training method named triggering avoiding training (tat) to seq2seq models against testing phase adversaries. in our proposed tat, we choose to use von-mises fisher distribution as the random noise generator and derive new theoretical results for such a design. we evaluate the proposed approach via text generation on a realworld dataset. the experimental results show that the proposed framework can consistently prevent seq2seq models from generating profanity under different settings. ## preliminaries for seq2seq models, let us use @xmath0, x 2 , • • • , x m ] and @xmath1, y 2 , • • • , y l ] to denote the input sentence of length m and the output sentence of length l, respectively. each x i or y j here stands for a single token. in a seq2seq model, the major components are one encoder h and one decoder g. the encoder learns a hidden vector representation h enc containing the semantics and context for each token. the decoder turns the vector representation back into an output token based on the previous sequence. formally, we denote the seq2seq model as f (x) = g(h(x)) → y : n @xmath2. for a decoding step t, the model outputs a distribution over all the possible tokens, i.e., f t (x) ∈ r c where f t denotes the t-th decoding step. the model then picks the token with the largest probability as the output of step t. in practice, seq2seq models typically employ neural network models such as lstm @xcite , gru @xcite and transformers @xcite as the encoder and the decoder. to facilitate our discussion, we focus on one of the most representative architectures, i.e., gru encoder-decoder with attention @xcite . all the methodologies proposed in this paper are independent from particular network architectures. with the notations and concepts defined above, we formulate the problem of profanity-avoiding training: definition 2.1 (profanity-avoiding training). given a set of sentences pairs and a set of profanity examples (referred to as profanity seeds) @xmath3, s 2 , • • • , s p }. our goal is to train a seq2seq model that generates fluent sentences with a minimal ratio of profanity. ## methodology as mentioned in the introduction section, there are two causes that can make the seq2seq model produce profanity. first, in the training phase, seq2seq models capture the language patterns within the training corpus. thus, a seq2seq model may also learn the profanity patterns from the training corpus. second, in the testing phase, some manipulated expressions may be fed to the seq2seq model to trigger profanity outputs from it. in the rest of this section, we present a profanityavoiding training framework with certified robustness to handle these two causes that lead to profanity. the framework has two components: the pattern-eliminating training (pet) model to barrier the profanity patterns in the training phase (section 3.1), and the trigger-resisting training (trt) model to maintain the robustness of the generation model against triggering expressions in the testing phase (section 3.2). besides, we also provide theoretical analysis to estimate the robustness of the proposed trt model, i.e., under what attack strength (in terms of the perturbation radius), the proposed trt model would still be certifiably robust. ## experiments in this section, we perform training phase and testing phase manipulations using both heuristic and state-of-the-art attack methods to evaluate the proposed framework's effectiveness in different scenarios. experimental results show that the proposed training framework can consistently prevent seq2seq models from generating profanity. ## conclusion seq2seq models have shown their success in various nlp tasks. however, inappropriate languages in the training set and the testing sentences may cause seq2seq models to produce profanity. this paper proposes the first training framework with certified robustness to handle profanity in both the training and testing phases. experimental results show that the proposed framework can successfully prevent seq2seq models from producing profanity while at the same time maintain satisfactory text generation quality.