Director: Generator-Classifiers For Supervised Language Modeling (2022)

Abstract: Current language models achieve low perplexity but their resulting generations still suffer from toxic responses, repetitiveness, and contradictions. The standard language modeling setup fails to address these issues. In this paper, we introduce a new architecture, Director, that consists of a unified generator-classifier with both a language modeling and a classification head for each output token. Training is conducted jointly using both standard language modeling data, and data labeled with desirable and undesirable sequences. Experiments in several settings show that the model has competitive training and decoding speed compared to standard language models while yielding superior results, avoiding undesirable behaviors while maintaining generation quality. It also outperforms existing model guiding approaches in terms of both accuracy and efficiency. Our code is made publicly available.

## introduction language models are becoming a powerful tool in various machine learning applications due to recent advancements in large-scale transformer models @xcite . standard language model training relies on maximizing log-likelihood over large training corpora yielding low perplexity nexttoken predictions. however, the resulting model generations still suffer from a number of problems. biases may be amplified from those already present in the large training corpora, and toxic or otherwise unsafe language can be generated @xcite . current models do not appear to adequately understand the deeper meaning of their generations and frequently contradict themselves @xcite . they are also known to produce repetitive text @xcite . if one has access to data labeled with such sequence generation errors, there is also no way to use it in the standard language modeling objective. standard training can make use of "unsupervised" data only, i.e., positive examples one would like the model to generate. in this work, we present a new model architecture, director, that is capable of training on both standard language modeling data, and supervised data indicating desirable and undesirable sequence generations. the model consists of an otherwise standard decoder architecture with an extra classifier head for each output token, in addition to the usual language modeling head, see figure 1 . standard unlabeled data is used to train the language model head, while labeled data trains the classifier head with the majority of the parameters of the decoder shared between the two tasks. during decoding, the outputs of the two heads are combined to decide on the left-to-right token generations. model training can take advantage of batch and sequence-wise parallelism, and decoding speed matches that of standard language models. using existing labeled datasets of toxic language and contradicting sequences, we show how direc-tor provides safer and less contradictory generations than standard training. we also show it is superior to the commonly used reranking/rejection sampling approach, and recent guided generation techniques such as fudge @xcite and pacer @xcite -with our model providing both accuracy and speed advantages. further, we show director has uses even when human-labeled data is not available but an automatic procedure can be constructed. in particular, we show it can be used to minimize repetitive generations -by automatically labeling repeated sequences and training on this labeled data. overall, we find that our model is simple, performant, efficient, and a generally applicable tool with several applications where it can provide improved sequence modeling. ## related work language modeling has seen a number of impressive recent improvements by scaling model and training data size @xcite , with applications in dialogue @xcite , qa @xcite and other general nlp tasks @xcite . despite these advances, much research is focused on resolving issues that remain, and controlling the quality of resulting generations. a popular class of approaches is to train the language model as standard, but then control the language model at decoding time, with perhaps the most common variant being reranking (or rejection sampling). using a separate model to rerank candidate decodings has been used to reduce toxicity @xcite , to reduce contradictions @xcite , or to improve performance on a given task @xcite . the advantage of such an approach is that the reranker can be trained with both positive and negative examples (or stack-ranked examples) of behavior, unlike the original language model. reranking has also been shown to outperform reinforcement learning in language tasks, e.g. in webgpt @xcite . another class of models is the model-guiding approaches, also referred to as controllable generation models @xcite . reranking models can only help if there are some good candidates from the beam decoding or sampling used to generate predictions. to exert greater influence on left-to-right token decoding, several model-guiding approaches have been proposed instead. gedi @xcite proposes to use a second separate language model to "rerank" for every left-to-right token step during decoding with respect to the difference between a control code coding for the desired attribute being present or not. plug and play (pplm) @xcite ) proposed to use a separate simple and fast attribute classifier, such as a bag-of-words classifier, to guide generation at decoding time to change e.g., topic or sentiment. this requires forward and backward passes in which gradients from the attribute model push the language model's hidden activations and thus guide the generation. fudge @xcite also makes use of a second classifier, but reranks tokens rather than computing gradients with the forward and the backward passes. fudge was shown to outperform several other methods, including pplm, hence we use fudge as one of our main baselines. however, overall, in all these methods requiring two models instead of one makes efficiency a key issue @xcite , in addition to requiring more memory. pacer @xcite proposes a faster and better-performing variant of fudge by sampling tokens, rather than reranking all of them, and then finally reranking the entire set of candidates at the end. we thus also use this as one of our baselines. in contrast, our model director is a unified generator-classifier and makes use of parallelism to score all tokens at each step during decoding without incurring significant costs beyond the standard language model decoding scheme. there is also related concurrent work. @xcite uses a contrastive method to reduce repetition similarly to unlikelihood training @xcite , but as far as we can see cannot be easily adapted to general positive and negative labeled sequences. @xcite proposes a way to control text generation with iterative reinforcement to deal with toxic generations or negative sentiment. it only has moderate success with repetition, perhaps because it still uses the standard likelihood training (with control variables) in its main loop, which still makes it hard to penalize certain sequences. we note that sigmoid outputs have been used recently elsewhere too, e.g. for machine translation @xcite . ## model in this section, we will introduce the director model. we will start by laying out the notation and background of language modeling and then introduce our new architecture. ## experiments in our experiments, we employ director to generate a response to a given context such that the response exhibits certain desirable attributes and avoids certain undesirable attributes. in our experiments, we focus on three such particular undesirable attributes: (i) toxicity, (ii) contradiction; and (iii) repetition, corresponding to three different tasks in sections 4.2, 4.3 and 4.4. ## discussion and conclusion we have presented a new architecture for training language models which takes advantage of classical supervised learning data and techniques. unlike the standard language model architecture and training objective, our model can use both positive and negative examples of language generations by making use of a classifier head attached to the decoder layer. this allows the model to avoid undesired generations. we show the effectiveness of this approach in three setups: avoiding unsafe, contradictory, and repetitive responses. our approach can potentially be used in any setup where examples of undesired behavior are known, feeding these in as negative examples, opening the door to the collection of more "negative class" generation datasets, which so far is a relatively unexplored area. our code and the experimental setup are made publicly available. future work should investigate these applications, as well as settings that consider all these kinds of undesired behavior at once, e.g. by using a multitasking approach.