Detoxifying Language Models with a Toxic Corpus (2022)

Abstract: Existing studies have investigated the tendency of autoregressive language models to generate contexts that exhibit undesired biases and toxicity. Various debiasing approaches have been proposed, which are primarily categorized into data-based and decoding-based. In our study, we investigate the ensemble of the two debiasing paradigms, proposing to use toxic corpus as an additional resource to reduce the toxicity. Our result shows that toxic corpus can indeed help to reduce the toxicity of the language generation process substantially, complementing the existing debiasing methods.

## introduction pretraining language models (lms) have been a foundation of nlp given recent performance achievements; however, there is a growing concern related to inherent societal and harmful biases in these models. due to historical biases embedded in training corpora, it is unavoidable for the language models to absorb, reproduce, and even amplify such undesired biases @xcite . @xcite showed that pretrained lms generate toxic text even when conditioned on innocuous prompts. one of their proposed debiased techniques is domain-adaptive pretraining @xcite , or dapt, on a non-toxic corpus. @xcite proposed a self-debiasing approach that uses only a handful of templates that contain the definition of undesired attributes. dapt is a data-based approach where internal weights are updated with an additional phase of pretraining. on the other hand, self-debiasing is a decoding-based approach that does not require additional resources. the difference between the two debiasing paradigms is a trade-off between the computational cost and the quality of debiasing. in this study, we propose to ensemble the data-and decoding-based approaches by using a toxic corpus as a detoxifying strategy. our study attempts to invalidate the belief that only non-toxic corpora can reduce the toxicity of language generation. we use gpt-2 @xcite as our primary language model and openwebtext (owtc; gokaslan and cohen, 2019), a large corpus of english webtext, as our training corpus. we measure the toxicity of each document using perspectiveapi foot_0 and collect non-toxic and toxic corpora that satisfy our toxicity requirements. our results demonstrate that using the toxic corpus indeed reduces the toxicity level of text generated from pretrained language models, which can be further improved by ensemble with the non-toxic corpus. ## background and related work perspectiveapi evaluates the likelihood of a comment to be perceived as toxic. it divides the toxicity into eight emotional attributes, including toxicity, severe toxicity, identity attack, insult, threat, profanity, sexual explicit, and flirtation. the model is a multilingual bert-based model, distilled into a single-language convolutional neural network (cnn). the auc of the model on test sets ranges between 0.97 to 0.99 foot_1 , which we safely assume to use to classify the documents. the model is also evaluated on the bias across a range of identity terms. test sets are generated by swapping the identity terms on both toxic and nontoxic sentences. in english test sets, the auc of all the identity terms fall between 0.96 to 1.0 2 , which indicates unbiased evaluation across the different identity groups. ## experiments we conduct adaptive pretraining on four separate gpt-2 models on each corpus discussed in sec. 3.2. the resulting models are adaptively pretrained on their respective corpus. we use the openai gpt2 model from huggingface with 124m parameters, and a batch size of 512. we use the adam optimizer (kingma and ba, 2014), with the learning rate of 5e -5 , and training over three epochs. ## results and discussion ## conclusion large pretrained lms suffer from degeneration and exhibit biases and toxicity despite their vast capabilities. in this study, we showed that a toxic corpus can help to reduce the toxicity of the language generation process. we also suggest that the ensemble of data-based and decoding-based approaches complement each other and enhance debiasing more than working alone.