Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models (2023)

Abstract: Considerable effort has been dedicated to mitigating toxicity, but existing methods often require drastic modifications to model parameters or the use of computationally intensive auxiliary models. Furthermore, previous approaches have often neglected the crucial factor of language’s evolving nature over time. In this work, we present a comprehensive perspective on toxicity mitigation that takes into account its changing nature. We introduce Goodtriever, a flexible methodology that matches the current state-of-the-art toxicity mitigation while achieving 43% relative latency reduction during inference and being more computationally efficient. By incorporating a retrieval-based approach at decoding time, Goodtriever enables toxicity-controlled text generation. Our research advocates for an increased focus on adaptable mitigation techniques, which better reflect the data drift models face when deployed in the wild.

## introduction large-scale pretrained language models (lms) have demonstrated remarkable progress in capabilities @xcite . however, an unintended consequence of this progress is the generation of toxic and harmful language, including hate speech, insults, profanities, and threats @xcite . with the widespread adoption of large language model systems such as chatgpt @xcite and openassistant @xcite , there is a need for techniques that can effectively mitigate the generation of toxic and harmful text @xcite . to address this challenge, it is essential not only to measure and understand the origins of toxic text generation but also to take effective steps towards its mitigation in lms. prior research on detoxification has primarily focused on two computationally expensive approaches: finetuning or constrained decoding @xcite . finetuning requires modifications to a pretrained lm parameters through additional training on carefully curated data. on the other hand, constrained decoding relies on an auxiliary model or processing module that modifies the next-token probabilities at inference time. both of these approaches are known to be highly compute-intensive @xcite . in addition to the drawbacks of the aforementioned techniques, the academic treatment of toxic language mitigation has predominantly assumed that toxicity remains static over time. most of the existing research has focused on building specialized models for specific domains or locales, which lack flexibility once trained and may have limited applicability across different tasks and domains @xcite . however, human language is shaped by a cumulative culture, constantly building upon itself and evolving over time @xcite . similarly, the ways in which language can cause harm, such as offensive and harassing text @xcite , also evolve @xcite . in this work, we propose a flexible technique called goodtriever (figure 1 ) that effectively tackles both static and lifelong toxicity mitigation. our approach is designed to handle domain shifts and builds upon recent advancements in language modeling, which have successfully incorporated external memory to enhance performance @xcite @xcite @xcite . more specifically, goodtriever combines a large lm with two external datastores. these datastores control text generation based on desirable (non-toxic) and undesirable (toxic) attributes. this property allows for convenient and immediate incorporation of new knowledge, as well as the ability to edit, correct and remove existing information without requiring any retraining of the lm. we conduct extensive experiments for static and continual toxicity mitigation, showing that goodtriever achieves comparable performance to state-of-the-art methods while being far less compute-intensive on the static tasks. we also show that goodtriever achieves competitive results to the method of multitask finetuning on continual toxicity mitigation tasks. our key contributions are: language models (lms) define probability distributions over sequence of tokens. given a context sequence of tokens c @xmath0, . . . , w t-1 ), the probability distribution p(w t |c t ) over the target token w t is estimated using next-token prediction where the probabilities are modeled as the product of conditional probabilities for each token in the sequence, given the tokens that came before it from the autoregressive lms. these models are typically implemented using a transformer network, parameterized by a set of parameters θ: retrieval-augmented lms compute next token distributions based not only on the immediately preceding context c t and the model parameters θ, but also on an external datastore c, from which examples are retrieved and incorporated into the base lm's prediction. specifically, for predicting w t , the retrieval operation from c depends on its prefix: we define the function f (•), which takes a context c as input and produces a fixed-length vector representation. as an example, in a transformer model, f (c) can be defined to map the context c to an intermediate representation obtained from a self-attention layer within the model. for the i th example (c i , w i ) ∈ d, the key-value pair (k i , v i ) is formed, where k i denotes the vector representation of the context f (c i ) and v i denotes the value associated with the target word w i . goodtriever creates two datastores: (k -, v -) from toxic examples and (k + , v + ) from non-toxic examples. inference. during inference, the parameteric component of the lm generates the output distribution p lm (w t |c t ; θ) over the next tokens, produces the corresponding context representation f (c t ), given the text input context c t and the logits z @xmath1, where v is the model's vocabulary. then the nonparametric component of the lm queries each datastore (k, v) with the f (c t ) representation to retrieve n , the k-nearest neighbors (k-nn) according to euclidean distance function d(•, •). next, the token probabilities p kn n are computed over these neighbors by applying a softmax with temperature t to the neighbors' negative distances and aggregating over each token of the vocabulary, as in the following: a temperature higher than 1 tends to flatten the distribution and prevents overfitting @xcite . more details about how the temperature parameter impacts goodtriever performance are in appendix c.3. for each context c t , we obtain three sets of probability distributions: the next token distributions i) from the base language model p lm , ii) from the toxic datastore p - kn n and iii) from the non-toxic datastore p + kn n respectively and their corresponding logits z t , z - t , z + t . ensembling. knn-lm interpolates the nearest neighbor distribution p kn n with the base lm distribution p lm using a tuned parameter to produce the final next-token distribution. knn-lm only allows to augment the model with a single datastore. here we introduce a method that allows us to combine multiple nearest neighbor distributions computed based on different datastores with the base lm probability distribution. our method is based on product of experts which is first proposed by @xcite . that idea allows us to combine toxic and non-toxic datastore outputs with base lm as: this equation indicates that a token possesses a high probability if it satisfies the condition of having high probabilities under both p lm and p + kn n , while simultaneously having a low probability under p - kn n . with this equation, we gain the flexibility to incorporate multiple datastores with the lm, allowing us to combine their logits through addition or subtraction. ## continual toxicity mitigation work to date has often treated toxicity as a fixed characteristic, disregarding its variations over time and among different demographic groups @xcite . one of the key advantages of goodtriever lies in its adaptability, facilitated by semi-parametric language models @xcite . we demonstrate the benefits of this flexible representation of toxicity by benchmarking goodtriever on the task of continual toxicity mitigation. the goal of this task is to continuously adapt to new types of toxicity while maintaining effective mitigation for previously encountered domains. ## conclusion we present goodtriever, a novel method for toxicity mitigation, which utilizes multiple retrieval mechanisms to effectively adapt to the changing nature of language and toxicity without compromising linguistic quality. goodtriever achieves 43% decrease in inference time when compared to previous state-of-the-art while maintaining a comparable toxicity mitigation performance. we also show how goodtriever mitigates toxicity consistently across model sizes and families. unlike prior approaches, goodtriever remains flexible and competitive in the face of evolving data.