The CRINGE Loss: Learning what language not to model (2023)

Abstract: Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples. Growing evidence shows that even with very large amounts of positive training data, issues remain that can be alleviated with relatively small amounts of negative data – examples of what the model should not do. In this work, we propose a novel procedure to train with such data called the “CRINGE” loss (ContRastive Iterative Negative GEneration). We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue. Our models outperform multiple strong baselines and are conceptually simple, easy to train and implement.

## introduction through the rise of large transformers @xcite , both language models @xcite and conversational agents @xcite have become much more powerful in recent years -up to the point that it is possible to engage with them in useful and nontrivial interactions. however, employing standard language model training and scaling the model size and amount of training data fails to resolve a number of issues. in particular, models can still suffer from toxicity and bias @xcite , lack of (long-term) coherence @xcite or fail to address user's intent @xcite . a growing body of work is instead investigating ways to train models beyond the standard language modeling objective, given access to examples of such failure cases, by incorporating this information into the training objective @xcite @xcite @xcite . in this work, we study the setting where the training set involves a given set of positive example sequences, as is commonly used for language model for each negative output token, a positive prediction is sampled from the language model to contrast against it. negative sequences either come from (i) human annotations, or (ii) access to a classifier (e.g., trained from the human annotations) that can be used to iteratively label the model's own generations and apply the cringe loss to those examples as well. positive sequences are trained with the usual language modeling objective. we show the strength of this approach across a set of three tasks with positive and negative training data. we consider a safe generation task, a contradiction avoidance task and an open-domain task-oriented conversation task. we compare to a wide variety of baselines, including vanilla transformers, reranking based on a classifier trained with the positive and negative data, unlikelihood training @xcite , model guiding methods such as fudge @xcite and pacer @xcite , and the recently introduced director method @xcite . generally, a single iteration of the cringe loss already outperforms most baselines. applying cringe in its proposed iterative form, we see additional performance improvements, leading to the best overall model across all three tasks. ## related work collecting negative examples positive examples for training language models come from human written text, e.g. web-based documents @xcite or conversations @xcite or employing crowdworkers for collecting data on specific skills @xcite . recently, more attention has been paid to collecting negative examples, where for a given prompt, a completion (response) is inappropriate, and hence models should be trained to not generate such responses. for example, datasets have been collected of contradictory responses @xcite , toxic responses @xcite , or unhelpful responses @xcite . such datasets can either be collected via crowdworkers, or through organic users, as is the case in the deployed blenderbot3 @xcite conversational agent. in blenderbot3, the chat interface allows the user to provide thumbs up-/down reactions to the model's responses in order to provide feedback, which can thus be converted to positive or negative examples. a related type of data collection, rather than collecting negative examples, is to ask human annotators to stack rank model generations @xcite . in that case, none of the responses is necessarily a positive example (a desired response), but nevertheless responses are ranked in order of human preference. in this work we only consider the case of positive and negative examples, not ranked examples. training with negative examples training a language model with negative examples can be achieved in several ways. @xcite propose unlikelihood training which is an additional term added to the optimization objective that reduces the probability of negative tokens compared to all other tokens (see also negative training @xcite for a related approach). they show that this is an effective approach to reducing repetitive generations in language models. jiang et al. (2022) also propose a contrastive learning objective to alleviate text degeneration. they argue that contrasting the positive label against the preceding m context tokens helps avoid the promotion of undesired tokens compared to unlikelihood training, which can exhibit this defect. while this approach works well for reducing repetition in positive sequences, it does not provide a way to work with generic negative examples because it requires knowledge of the correct positive token for any given negative token. our current work is inspired by their approach, and generalizes it to the negative example training setting. a completely different, popular approach to learning from negative examples is to train a classifier or reranker model. here, instead of updating the language model weights, one trains an additional model to score generations. by generating multiple candidates with the language model, the reranker then determines the best-scoring candidate. @xcite train a reranker to help avoid the problem of contradictory generations. @xcite find that reranking can outperform reinforcement learning in certain scenarios. instead of using an additional model to select from the final generations, model-guiding approaches, such as pnp @xcite , gedi @xcite , fudge @xcite and pacer @xcite use this model on a per-token basis during decoding. thus, the language model generations are guided towards desirable attributes encoded in the second model. the recently introduced director model @xcite instead of using a second model, shares language modeling and classification guiding heads in the same architecture. while it works well on multiple tasks @xcite , one shortcoming is that it requires an architecture change and thus cannot as easily be applied to existing models and implementations. iterative training of language models unlikelihood training was shown to iteratively improve repetition issues by training on the model's own generations @xcite . iterative training of language models on human preferences has been successfully applied in several summarization @xcite @xcite and dialogue settings @xcite . @xcite train a language model to unlearn unwanted behavior using generated samples. they label and quantize the model's generations and perform conditional training by prepending the sequences with their corresponding reward token. the instructgpt model @xcite uses reinforcement learning from human feedback (rlhf) @xcite to align a language model to follow instructions. here, the human feedback is used to train a reward model which guides a proximal policy optimization (ppo) @xcite algorithm to fine-tune the language model. ## the cringe loss the cringe (contrastive iterative negative generation) loss is a method for training on data containing both positive and negative sequences. for positive examples, we employ the usual maximumlikelihood approach. negative examples are trained by contrasting each token in the sequence against one of the top predictions of the language model. figure 1 depicts a sketch of how training on a negative sequence works. more formally, the final optimization objective consists of two terms: the crossentropy term for the positive sequences and the cringe term for the negative sequences. the former is used as standard, i.e., for a token x t from a positive sequence x: algorithm 1 cringe loss for a negative token require: a sequence of token indices @xmath0.g., concatenated prompt and response until current step) and a negatively-labeled continuation token index x - t . a generative model f θ . a scalar k. ▷ feed the sequence to the model and get a score for each next token in the vocabulary v . s ← f θ (@xmath1. [s +,1 , . . . , s +,k ] ← topk(s) ▷ sample positive token from this set. s + ← sof tmax_sample([s +,1 , . . . , s +,k ]) ▷ concatenate the positive and negative token scores and apply crossentropy with a positive label of index 0, i.e. compute loss of eq. 3. loss ← nn.crossentropyloss([s + , s x - t ], 0) we present the pseudo-code of this approach for a single prediction in algorithm 1. now, to train on both positive and negative examples we take a weighted sum of the two losses ## conclusion in this paper, we proposed the cringe loss, an approach to iteratively train a language model with positive and negative examples. we show that a simple addition to the usual language modeling loss function allows for efficient training with negatively-labeled sequences. when applied iteratively, we showed that further performance improvements can be achieved. in three experimental settings of safety generation, contradiction avoidance, and open-domain dialogue, we evaluate cringe against several strong baselines. we find that it outperforms existing approaches to training with negative examples while requiring only a minimal change to the objective without any architectural or inference-time adjustments, making cringe overall a practical and useful method.