Language Model Detoxification in Dialogue with Contextualized Stance Control (2022)

Abstract: To reduce the toxic degeneration in a pretrained Language Model (LM), previous work on Language Model detoxification has focused on reducing the toxicity of the generation itself (self-toxicity) without consideration of the context. As a result, a type of implicit offensive language where the generations support the offensive language in the context is ignored. Different from the LM controlling tasks in previous work, where the desired attributes are fixed for generation, the desired stance of the generation depends on the offensiveness of the context. Therefore, we propose a novel control method to do context-dependent detoxification with the stance taken into consideration. We introduce meta prefixes to learn the contextualized stance control strategy and to generate the stance control prefix according to the input context. The generated stance prefix is then combined with the toxicity control prefix to guide the response generation. Experimental results show that our proposed method can effectively learn the context-dependent stance control strategies while keeping a low self-toxicity of the underlying LM.

## introduction large pretrained language models, such as gpt2 @xcite , can produce coherent, almost human-like texts, but they are prone to generating offensive language, which hinders their safe deployment @xcite ). an extensive body of work has focused on detoxifying pretrained lms @xcite @xcite . however, it can be more complicated when the lms are applied to downstream natural language generation (nlg) tasks, such as dialogue response generation. when applied in dialogue, the uncontrolled models tend to generate toxic content and in addition to explicitly offensive utterances, @xcite suggest that these models can also implicitly insult a group or individual by aligning themselves with an offensive statement, as shown in figure 1 . therefore, to detoxify a pretrained lm applied in dialogue, the stance of the generated response needs to be taken into consideration. in a normal dialogue, we do not need to control the stance, but if the user inputs offensive language, the model should not respond with a positive stance. in other words, the eligible stance is context-dependent and we need to consider the dialogue context. one straightforward solution is to design a control flow with a binary offensive language classifier, where the dialogue context is taken as input for the classifier. if the context contains offensive language, an nlg model with both toxicity control and stance control is used for response generation. we would like the self-toxicity to be low and the stance not to be supportive. on the other hand, if the context does not contain offensive language, the stance does not need to be controlled, so another nlg model with only toxicity control is used for response generation. however, this classify-then-generate framework has several limitations. first, it requires training a classifier and controlled nlg models separately, introducing additional model parameters. second, its performance relies heavily on the classifier, so the performance of this classifier can be a bottleneck. to address these limitations, we propose a novel method to do context-dependent control, where the offensive language classification is learned implicitly together with the stance control, instead of being learned explicitly by a classifier. following @xcite and @xcite , we use prefix, a small continuous vector prepended to the lm, to achieve controllability, and we further introduce hierarchical prefixes for contextualized control. more specifically, meta prefixes are introduced to control the underlying lm to generate the desired stance prefix according to the dialogue context, which is then combined with the toxicity prefix to guide the response generation. therefore, the model can be trained end to end, without the bottleneck of a classifier. besides the language modeling loss, we introduce two novel training loss terms to push the model to learn about the context-dependent control strategy. experimental results show that our method effectively controls the stance according to the offensiveness of the user utterance while keeping the self-toxicity at a low level. compared with the baselines, our controlling method has significantly less effect on the stance of the generations when the input user utterance is not offensive and when the input user utterance is offensive, our method achieves a lower support stance score. to conclude, our main contributions are: ## related work to reduce the offensive content generated by the lms, previous research on offensive language de-tection can be utilized to filter out undesired generations. ## method given a user utterance c, our goal is to guide the generation model to deliver a contextually safe response, which includes a context-independent attribute: self-toxicity, and a context-dependent attribute: stance. each example in the training dataset x is a tuple of (c, r, t c , t r , s r ), where c is the user utterance text, r is the response to the user utterance, t c and t r are the offensiveness annotations of c and r respectively, and s r is the stance annotation of the response r. following @xcite and @xcite , we use prefix, a small continuous vector prepended to the lm's hidden representations, to control the generation. note that during the training or the application of the prefixes, the parameters of the underlying lm are kept frozen, so only the prefixes need to be optimized and stored. since the self-toxicity control is context-independent and offensiveness annotations are available for training, we first train the toxicity control prefixes, denoted as h β , following the supervised method in @xcite . h β consists of two prefixes: h 0 β = h β [0, :, :] and h 1 β = h β [1, :, :]. h 0 β corresponds to non-offensive text and h 1 β is the opposite. both h 0 β and h 1 β are vectors of dimensions m ×d, where m is the length of a prefix and d is the size of the hidden dimension (see appendix a for detailed explanation). however, the controlled generation model should avoid not only generating a response that is offensive by itself, but should also avoid generating a response that supports an offensive user utterance. more specifically, there are four cases of training examples in contextualized stance control: our stance requirement is still satisfied. note that the offensiveness annotations of the user utterances are not available during evaluation, so the model needs to learn it along with the controllability. one way to learn offensive language detection is to train a binary classifier in an explicit way. however, the errors from the detector will propagate to the generated responses (section 4.2). instead of learning offensive language detection explicitly, we propose to learn it in an implicit way along with stance control. we introduce another set of prefixes, meta prefixes h α , to achieve contextualized controllability. meta prefixes are trained to generate the stance control prefix according to the user utterance, which is then combined with the toxicity control prefix mentioned above to guide the generation, as shown in the upper part of figure 2 . same as given a training example, we first infuse the corresponding prefixes with the stance and offensiveness attributes by encouraging them to reconstruct the response r. as illustrated in figure 2 , according to m r , we select the corresponding meta-prefix h mr α and prepend it to the user utterance c as input for the lm. the output of the lm is a generated prefix of dimension m × d. then the generated prefix is combined with the toxicity prefix h tr α by element-wise addition. the resultant prefix is appended to the user utterance c to guide the lm to generate the response r. therefore, the first part of the training loss is the language modeling loss l lm . the computation of log p(r t |@xmath0, c, t r , m r ) is parameterized as log p α,β,γ (r t |@xmath1, c, h mr α , h tr β ), where γ is the set of fixed lm parameters, and α, β represent learnable prefix parameters. although l lm infuses the corresponding prefixes with the stance and offensiveness attributes, the offensiveness annotation of the user utterance t c is ignored in l lm and thus the meta prefixes are not pushed to learn the offensiveness and rely on it to control the stance. to address this problem, we introduce two additional contrastive loss terms utilizing the annotation t c . in order to push the meta prefixes to learn about the stance requirement when the user utterance is offensive, we add a stance contrastive loss l s to differentiate between case 3 and case 4 stated above. as shown in figure 2 , each offensive user utterance in the training dataset is combined with the two meta prefixes separately, and the distance between two generated prefixes is used to calculate the stance contrastive loss l s . since the stance requirement is different when the user utterance is offensive and inoffensive, the meta prefixes also need to learn about the contextdependency and about what stance to achieve when the user utterance is not offensive. we achieve this by introducing another context contrastive loss l c to differentiate between the aforementioned case 3 and the union of case 1, 2. as illustrated in figure 3 , the meta prefix h 0 α is combined with offensive user utterances and non-offensive user utterances respectively, and the distance between the generated prefixes in these two cases is used to calculate the context contrastive loss. the final training loss l is a weighted sum of the three loss terms described above. after training, the meta prefix h 1 α and the toxicity prefix h 1 β do not need to be saved. only h 0 α and h 0 β are used to guide generation during evaluation. ## conclusion in this work, we propose a novel method for contextual detoxification, where a context-dependent attribute: stance, and a context-independent attribute: toxicity, are controlled within a unified hierarchical prefix framework. experimental results show that our proposed method can successfully guide an nlg model to generate safer responses with the stance taken into consideration. besides the dialogue detoxification task we experimented with, our proposed framework can be extended to other combinations of the context-dependent and the context-independent control.