FUDGE : Controlled Text Generation With Future Discriminators (2021)

Abstract: We propose Future Discriminators for Generation (FUDGE), a flexible and modular method for controlled text generation. Given a pre-existing model G for generating text from a distribution of interest, FUDGE enables conditioning on a desired attribute a (for example, formality) while requiring access only to G’s output logits. FUDGE learns an attribute predictor operating on a partial sequence, and uses this predictor’s outputs to adjust G’s original probabilities. We show that FUDGE models terms corresponding to a Bayesian decomposition of the conditional distribution of G given attribute a. Moreover, FUDGE can easily compose predictors for multiple desired attributes. We evaluate FUDGE on three tasks — couplet completion in poetry, topic control in language generation, and formality change in machine translation — and observe gains in all three tasks.

## introduction recent advances in large pretrained language models allow us to generate increasingly realistic text by modeling a distribution p (x) over natural language sequences x. the distribution p (x) may be truly unconditional, as is common in language modeling, or it may model p (x|i) conditioned on some input i, as in machine translation or summarization. we are frequently interested in controlled text generation, the task of generating text conditioned on an additional desirable attribute a which is not already built into p (x). that is, we would like to model p (x|a) (or possibly p (x|i, a); henceforth we will drop i from the notation for simplicity). for example, p (x) may be a pretrained translation model for spanish inputs i to english outputs x, but we may wish to additionally constrain the outputs to possess a new attribute a, e.g., formality, which we did not optimize for during training. unfortunately, once we have already obtained an unconditioned p (x) defined as the output dis-tribution of some large generative model g, it is nontrivial to add conditioning on a new attribute a without either training a new model from scratch or fine-tuning with additional data. although in principle we can trivially sample from p (x|a) via rejection sampling from p (x), rejection sampling may be highly inefficient in practice. on the other hand, while generating according to attribute a, p (x) should be left otherwise intact: in the previous translation formality example, it is pointless to generate formal english outputs if they do not preserve the original spanish meaning. in light of these concerns, we propose future discriminators for generation (fudge), a flexible and modular method for modeling p (x|a) which accesses only the output probabilities of the generative model g which defines p (x). fudge learns a binary predictor for whether attribute a will become true in the complete future, based on an incomplete sequence prefix (sec. 3). multiplying the output probabilities of this predictor with g's original probabilities and then renormalizing yields a model for the desired p (x|a) via bayes' rule. we run experiments on three controlled text generation tasks -couplet completion in poetry, topic control in language generation, and formality change in machine translation -showing our method's broad applicability. additionally, we demonstrate the modularity of fudge by composing multiple attribute constraints in both the couplet and topic control tasks. in our experiments, we find that fudge is highly effective at attribute control, outperforming both a baseline which directly fine-tunes g and also a strong gradientbased method (pplm @xcite ). our code is available at https://github.com/yangkevin2/ naacl-2021-fudge-controlled-generation . ## related work ideally, a controlled text generation method should efficiently control for a while preserving p (x) as much as possible. recent work on controlled text generation has greatly advanced our ability to control for a required attribute a flexibly and cheaply, with varying degrees of modification to the original model g which defines p (x). one line of work fine-tunes a pretrained model for a desired attribute @xcite @xcite . the result is a class-conditional language model (cclm). however, it is difficult to isolate the desired attribute from the distribution shift between g and the finetuning dataset @xcite @xcite , i.e., it is nontrivial to preserve the desirable qualities of the p (x) modeled by g. one may also need to fine-tune separately for each attribute of interest. ctrl @xcite partially addresses these issues by providing 55 attribute control codes for a large language model trained from scratch, although this is expensive. very recently, gedi @xcite achieves strong performance by using cclm generators as discriminators, though it relies on several heuristics. more broadly, text generation models for style transfer @xcite @xcite , summarization @xcite @xcite , and machine translation @xcite @xcite can also be viewed as cclm's for different "attributes." a second type of approach instead conditions on a desired attribute by backpropagating gradients, either to directly modify model activations @xcite or to find a trigger string @xcite . such methods often exhibit a high degree of attribute control, and can be used in adversarial attacks @xcite . in fact, @xcite show that by carefully modifying the latent state, one can cause the base g to produce arbitrary outputs. a third class of methods, referred to as weighted decoding (wd), assumes access only to p (x) (i.e., g's output logits), and operates directly on these logits @xcite @xcite . compared to other approaches, wd methods are relatively interpretable in how they obtain p (x|a) from p (x), but prior wd implementations have been observed to perform poorly in controlled text generation @xcite . while fudge shares a bayesian motivation with other wd methods, fudge follows the bayesian factorization more closely in implementation (sec. ## future discriminators for generation we now explain the details of our proposed method, future discriminators for generation (fudge), and show that it corresponds to modeling the desired conditional distribution p (x|a). for a given language generation task, assume we have an autoregressive model g (e.g., a large pretrained language model) which models p (x i |x 1:i-1 ) for tokens x 1 . . . x i . letting @xmath0:n denote a completed sequence, g can sample from p (x) = p (x 1:n ) one token at a time by factoring p (x): to condition on attribute a, we instead model p (x|a). this requires a model for p (x i |x 1:i-1 , a), modifying the previous factorization: if we model p (x i |x 1:i-1 , a) directly, we obtain a class-conditional language model (cclm). we can learn the cclm by e.g., fine-tuning g depending on the available data, possibly with some structural modification to g to accommodate conditioning. however, fudge instead relies on the following bayesian factorization, exchanging x i and a conditioned on x 1:i-1 : the second term is exactly the quantity modeled by the base g. it then suffices to model the first term, p (a|x 1:i ), with a binary classifier b for the attribute a given a prefix x 1:i . intuitively, one can view b as rescoring or reranking g's original hypotheses. we emphasize that although b takes a prefix x 1:i as input, it predicts whether attribute a will in the future be satisfied for the completed generation x 1:n . for instance, suppose we are given a dataset of examples {(x 1:n , a )} with a being the values of binary indicators for the desired a (i.e., if a is formality, then a is 0 or 1 when x 1:n is informal a large pretrained model g (dark blue) outputs unconditioned probabilities. our binary predictor (red) predicts whether the eventual completed sequence will be formal for each possible continuation (computed for each candidate x 3 , e.g., "want"; holding a fixed). the probabilities for each x 3 are multiplied (purple) and then renormalized to obtain p (x 3 |x 1:2 , a), from which we sample the next token x 3 ="prefer." or formal respectively). for each training example (x 1:n , a ), we train our classifier b using all pairs (x 1:i , a ); that is, we construct a separate example from each prefix x 1:i of x 1:n . our approach contrasts with previous methods such as @xcite , which greedily optimize for a on the immediate extension x 1:i+1 . one particular benefit is that fudge naturally plans for the future: in the example for generating text on the "space" topic in fudge's implementation is shown schematically in figure 1 , and is quite simple in practice. fudge just needs to learn a b (red in figure 1 ) sharing tokenization with g (dark blue). it then converts b's output into probabilities (red table in figure 1 ), and multiplies with the original output probabilities from g (dark blue table), to obtain unnormalized probabilities p (x i , a|x 1:i-1 ) (purple table). finally, renormalizing over the output vocabulary yields the desired distribution p (x i |x 1:i-1 , a). in practice, we operate in the log-probability space for numerical stability. to improve computational efficiency, we typically choose b to be lightweight relative to g. we also consider only the top 200 possibilities for x i according to g at each step, as a cheap approxi-mation to the full distribution, and find that this works well in practice. 1 in each task in sec. 4, running fudge on the test set takes no more than 15 minutes on a single quadro rtx 6000 gpu. finally, as with other controlled generation approaches such as @xcite , it is likely that augmenting fudge with reranking approaches such as rejection sampling could improve output quality at the cost of compute time, although we do not comprehensively evaluate such extensions in this work. ## experiments we run experiments on a range of controlled text generation tasks to evaluate the effectiveness of our proposed method: poetry couplet completion (sec. 4.1), topic-controlled language generation (sec. 4.2), and machine translation formality change (sec. 4.3). for each task we discuss the evaluation setup, the specific details of our method and baselines, and finally experimental results. ## discussion fudge is a principled approach to controlled text generation which models p (x|a) by closely following a bayesian factorization, thus preserving the base p (x) as much as possible. fudge achieves strong performance on a wide range of different tasks: poetry couplet completion, topic control, and informal-to-formal machine translation. additionally, fudge can easily compose different attributes in a modular fashion: the meter, rhyme, and end-of-sentence constraints for couplet completion, and the individual words within each topic bag for topic control. in principle, fudge is applicable to any controlled generation task where we can train discriminators for the desired attribute or attributes. ## ethics of controlled text generation we recognize that strong controlled generation methods have the potential to produce harmful outputs and/or misinformation when used adversarially @xcite . however, such methods can also be a powerful tool for mitigating harmful biases learned by large pretrained language models @xcite , for example by detoxifying language @xcite . overall, we believe it is still beneficial to continue research into general controlled text generation methods such as fudge.