Text Detoxification using Large Pre-trained Neural Models (2021)

Abstract: We present two novel unsupervised methods for eliminating toxicity in text. Our first method combines two recent ideas: (1) guidance of the generation process with small style-conditional language models and (2) use of paraphrasing models to perform style transfer. We use a well-performing paraphraser guided by style-trained language models to keep the text content and remove toxicity. Our second method uses BERT to replace toxic words with their non-offensive synonyms. We make the method more flexible by enabling BERT to replace mask tokens with a variable number of words. Finally, we present the first large-scale comparative study of style transfer models on the task of toxicity removal. We compare our models with a number of methods for style transfer. The models are evaluated in a reference-free way using a combination of unsupervised style transfer metrics. Both methods we suggest yield new SOTA results.

## introduction identification of toxicity in user texts is an active area of research @xcite @xcite . the task of automatic rewriting of offensive content attracted less attention, yet it may find various useful applications such as making online world a better place by suggesting to a user posting a more neutral version of an emotional comment. the existing works on text detoxification @xcite @xcite cast this task as style transfer. the style transfer task is generally understood as rewriting of text with the same content and with altering of one or several attributes which constitute the "style", such as authorship @xcite , sentiment @xcite , or degree of politeness @xcite . despite the goal of preserving the content, in many cases changing the style attributes changes the meaning of a sen-tence significantly. foot_0 so in fact the goal of many style transfer models is to transform a sentence into a somewhat similar sentence of a different style on the same topic. 2 we suggest that detoxification needs better preservation of the original meaning than many other style transfer tasks, such as sentiment transfer, so it should be performed differently. we present two models for text detoxification, which have extra control for content preservation. the first model, paragedi, is capable of fully regenerating the input. it is based on two ideas: external control of an output of a generation model by a class-conditioned lm @xcite and formulation of style transfer task as paraphrasing @xcite . being based on a paraphraser model, paragedi explicitly aims at preserving the meaning of the original sentence. the second approach, condbert, inspired by @xcite , follows the pointwise editing setup. it uses bert to replace toxic spans found in the sentence with their non-toxic alternatives. the semantic similarity is maintained by showing the original text to bert and reranking its hypotheses based on the similarity between the original words and their substitutes. interestingly, bert does not need any class-conditional pre-training to successfully change the text style from toxic to normal. in addition, we perform a large-scale evaluation of style transfer models on detoxification task, comparing our new models with baselines and state-ofthe-art approaches. we release our code and data. 3 our contributions are as follows: • we propose two novel detoxification methods based on pre-trained neural language models: paragedi (paraphrasing gedi) and cond-bert (conditional bert). ## related work one of the most straightforward ways of solving style transfer task is to "translate" a source sentence into the target style using a supervised encoderdecoder model @xcite . since the source and the target are in the same language, pretrained lms such as gpt-2 @xcite can be applied for this task -fine-tuning them on relatively small parallel corpora gives a good result @xcite . however, this method is used quite rarely because of the lack of sufficiently large parallel data. the rest of described models are trained in an unsupervised way. pointwise editing models a relatively easy yet efficient style transfer method is to leave the sentence intact and manipulate only individual words associated with the style. delete-retrieve-generate (drg) framework @xcite was the first effort to perform such transfer. it proposes four methods based on this principle. two of them perform well on our data: drg-retrieveonly retrieves a sentence with the opposite style which is similar to the original sentence and returns it, and drg-templatebased takes the style attributes from it and plugs them into the original sentence. here, the performance depends on the methods for the identification of style markers and retrieval of replacements. words associated with style are typically identified either based on their frequencies as in the original paper, some works use attention weights as features @xcite . alternatively, style transfer can use masked language modelling (mlm). an mlm trained on a dataset with style labels picks a replacement word based not only on the context, but also on the style label. an example of such model is mask & infill @xcite . it is most similar to condbert method we propose. however, cond-bert performs additional control over the style and the content preservation and is able to make multi-word replacements. another similar model of this type is described by @xcite . it has a more complicated structure: there, two mlms trained on corpora of different styles perform replacements jointly. end-to-end architectures in contrast to these models, there exist end-to-end architectures for style transfer. they encode the source sentence, then manipulate the resulting hidden representation in order to incorporate a new style, and then decode it. some of them disentangle the hidden representation into the representation of content and style @xcite . the others force the encoder to represent style-independent content @xcite . alternatively, the model dualrl by @xcite performs a direct transfer from the source to the target style. the task is paired with the dual task (back transfer to the source style) which allows models to train without parallel data. the deep latent sequence model (dlsm) model by @xcite uses amortized variational inference to jointly train models for the primal and dual tasks. the stable style transformer (sst) method (lee, 2020) trains a pair of sequence-tosequence transformers for primal and dual tasks using cross-entropy of a pretrained style classifier as an additional discriminative loss. the style transfer as paraphrase (strap) method by @xcite views a style transfer model as a paraphraser that adds attributes of a particular style to a text. the authors create pseudo-parallel data by transferring style-marked texts to neutral with a pre-trained general-purpose paraphraser and then train sequence-to-sequence models on these neutral-to-styled parallel datasets. our paragedi model is conceptually similar to these methods. however, unlike these methods, the style is not infused into the model or a sentence representation but is imposed on the generator by another model. ## paraphrasing gedi model the recently proposed gedi model @xcite ) performs text generation from scratch guided by a language model informed about specific attributes of a text, e.g. style or topic. we extend this model by enabling it to paraphrase the input text. ## conditional bert model bert @xcite has been trained on the task of filling in gaps ("masked lm"), we can use it to insert non-toxic words instead of the toxic ones. this approach has been suggested by @xcite as a method of data augmentation. the authors identify words belonging to the source style, replace them with the [mask] token, and the bert model then inserts new words of the desired style in the designated places. to push bert towards the needed style, the authors finetune bert on a style-labelled dataset by replacing segmentation embeddings of original bert with trainable style embeddings. we perform some changes to this model to adapt it for the detoxification task. while in the original conditional bert model the words are masked randomly, we select the words associated with toxicity. this can be done in different ways, e.g. by training a word-level toxicity classifier or manually creating a vocabulary of rude and toxic words. we use a method which does not require any additional data or human effort. we train a logistic bag-of-words toxicity classifier. this is a logistic regression model which classifies sentences as toxic or neutral and uses their words as features. as a byproduct of the training process, each feature (word) yields a weight which roughly corresponds to its importance for classification. the words with the highest weights are usually toxic. we use the normalised weights from the classifier as toxicity score. the overview of condbert is shown in figure 2 . to preserve the meaning of the replaced word, we employ the content preservation heuristics suggested by @xcite : (i) preserve the original tokens instead of masking them before the replacement; (ii) rerank the replacement words suggested by bert by the similarity of their embedding with the embedding of the original word. despite using class-specific sentence embeddings, conditional bert often predicts toxic words, apparently paying more attention to the context than to the embeddings of the desired class. to force the model to generate non-toxic words we calculate the toxicity of each token in bert vocabulary and penalize the predicted probabilities of tokens with positive toxicities. finally, we enable bert to replace a single [mask] token with multiple tokens. we generate each next token progressively by beam search and score each multitoken sequence by the harmonic mean of the probabilities of its tokens. ## detoxification experiments we train the two new models and a number of other systems for text detoxification. below we describe datasets, evaluation setup, and results. ## mining a parallel detoxifying corpus the strap model @xcite is based on the assumption that a regular paraphraser can transform a stylistically marked text into a neutral text. although our experiments show that a para- ## human evaluation of detoxification while the automatic reference-free evaluation is cheap and fast, it may be unreliable. toxicity and fluency classifiers are not perfect and can return erroneous evaluations. the embedding distance which is used to measure the content preservation was shown to weakly correlate with human judgements @xcite . thus, we evaluate the best-performing models manually. ## sentiment transfer experiments text detoxification is not as well-established as other style transfer tasks, which makes it is difficult to put our models in the context of other works on style transfer. thus, we conduct an experiment on a different domain, namely, sentiment transfer. ## conclusion we present two style transfer models tailored for detoxification, i.e. transfer from toxic to non-toxic texts. both of them combine high-quality pretrained lms with the extra style guidance. par-agedi is based on a paraphraser guided by a styleconditioned gpt-2 model. condbert model is based on bert which does not need any finetuning, and all style control is performed with a pre-trained toxicity classifier. we conduct a largescale study of style transfer models exploiting both automatic and manual evaluation. our experiments show that the proposed methods outperform other state-of-the-art style transfer models on the tasks of detoxification and sentiment transfer.