MIL -Decoding: Detoxifying Language Models at Token-Level via Multiple Instance Learning (2023)

Abstract: Despite advances in large pre-trained neural language models, they are prone to generating toxic language, which brings security risks to their applications. We introduce MIL-Decoding, which detoxifies language models at token-level by interpolating it with a trained multiple instance learning (MIL) network.MIL model is trained on a corpus with a toxicity label for each text to predict the overall toxicity and the toxicity of each token in its context. Intuitively, the MIL network computes a toxicity distribution over next tokens according to the generated context which supplements the original language model to avoid toxicity. We evaluate MIL-Decoding with automatic metrics and human evaluation, where MIL-Decoding outperforms other baselines in detoxification while it only hurts generation fluency a little bit.

## introduction trained on huge amount of text corpora, transformer-based @xcite pretrained language models (lms) have led to a wave of advances in natural language generation tasks @xcite ; @xcite ; @xcite ). however, these lms are capable of generating offensive content, racist, or otherwise toxic language @xcite which bring security risks to the application in nlp systems. to enable safe use and deployment of language model, it is necessary to undertake effective steps to mitigate toxic text generation. we examine the public comments provided in jigsaw toxic comment classification challenge dataset 1 (jigsaw) containing over 200k comments that were labeled as toxic. in most cases, several spans of harmful text cause the toxicity of the whole comment. in the example given in prior studies @xcite attempt to filter out a specific word list at the decoding stage, which cannot achieve an obvious effect on mitigating toxicity in the generated text. approaches like dexperts @xcite change the lm output distribution for detoxification with outside expert lms, making it hard for explanation. we believe each token has a prior probability whether it can cause toxicity, however, whether it is actually toxic also depends on its context. words like stupid, crime, rubbish, etc are neural, but can become offensive given certain context, as in the example in therefore, we present mil-decoding, a tokenlevel detoxification in consideration of the contextual information with a multiple instance learning (mil) neural network. at each decoding step, our proposed method uses a mil network to score the retrieved tokens conditioned on the token itself and its contextual information. the mil network predicts the toxicity of the token's occurrence in the generated context to compute an extra toxicity distribution over candidate tokens to avoid toxic generation. at inference time, we combine the toxicity distribution and the original lm probability distribution at each time step to determine which token to generate. we conduct experiments conditioned on two widely-used datasets: realtoxicityprompts (gehman et al., 2020) and a qa-dataset provided by @xcite . experimental results show that our mil-decoding method achieves faster decoding speed than other decoding-time methods, while it outperforms all other detoxification methods in reducing toxic text generation. we further verify that mil-decoding can mitigate toxicity conditioned on either nontoxic or toxic prompts. in summary, the contributions of our work are as follows: ## methodology the core idea of mil-decoding is to enhance the lm probability distribution with a mil network that computes a toxicity score. in section 3.1, we first introduce the mil network architecture and analyze the toxicity score produced by the network. and then, we provide a detailed description of our approach mil-decoding in section 3.2. ## experiments we use gpt-2 medium as our base pre-trained lm. following gehman et al. (2020), we run experi-ments to evaluate the problem of toxic degeneration given a prompt context. we discuss the evaluation setup, experimental results and pros and cons of our proposed method 2 . generative discriminator (gedi; @xcite gedi achieves strong performance by using a class-conditional language model (cc-lm) as discriminator to compute the probability contrast between desired control code and anti-control code. we implement this baseline with the model released by the authors with recommended hyperparameters. decoding-time controlled text generation with experts and anti-experts (dexperts; @xcite dexperts directly combines probability distribution from an expert lm and an anti-expert lm which model text with desirable and undesirable attributes. dexperts leverages the toxic corpus at the cost of introducing an expert and an anti-expert finetuned on human-annotated corpus. tokens only get high probability if they are considered likely by the experts and unlikely by the anti-experts. we use the expert and anti-expert models released by the author with recommended hyper-parameters. ## related work much closely related work has been introduced in section 2. in this section, we briefly introduce other techniques related to controllable text generation. the style of text generated by the pre-trained language model can be better controlled by building specific datastores @xcite , as knn-lm @xcite . @xcite combine weighted decoding and reinforcement learning with an actor-critic framework to train a lmsteering model. @xcite and @xcite steer generation with information encoded in the model by extracting latent vectors and task vectors separately. @xcite further utilize normalizing flow to map the complex distributions in the latent space to simple gaussian distributions. @xcite propose a unified generator-classifier with both a language modeling and a classification head for each output token. controllable generation can also be combined with text rewriting methods to modify undesirable spans in generated text @xcite . ## conclusion we have introduced mil-decoding, which can detoxify pre-trained lms at token-level and outperform other methods in toxicity mitigation. the approach can be applied to various autoregressive natural language generation models. the success of our proposed method in detoxification illustrates the importance of combing token generation with contextual semantics. future work will explore how to balance generation fluency better.