Attribute Alignment: Controlling Text Generation from Pre-trained Language Models (2021)

Abstract: Large language models benefit from training with a large amount of unlabeled text, which gives them increasingly fluent and diverse generation capabilities. However, using these models for text generation that takes into account target attributes, such as sentiment polarity or specific topics, remains a challenge. We propose a simple and flexible method for controlling text generation by aligning disentangled attribute representations. In contrast to recent efforts on training a discriminator to perturb the token level distribution for an attribute, we use the same data to learn an alignment function to guide the pre-trained, non-controlled language model to generate texts with the target attribute without changing the original language model parameters. We evaluate our method on sentiment- and topic-controlled generation, and show large performance gains over previous methods while retaining fluency and diversity.

## introduction while large pre-trained language models (lm) have advanced text generation with coherent language by training on a large amount of unlabeled data @xcite @xcite , they are not controllable. for instance, given the prompt "the issue focused on", gpt-2 @xcite can generate a high-quality sentence, but it cannot take extra input such as "positive" or "business" to guide the sentence towards a positive sentiment or business-related topic, due to the lack of attribute labels during training. to solve the discrepancy between training and inference, one direction is to train an lm from scratch with some supervision such as control codes in ctrl @xcite . nevertheless, this method requires training an lm with a large number of parameters, and is limited by the attributes used during pre-training. another direction is to fine-tune the pre-trained lm on some annotated datasets. this usually requires updating all the parameters in the model, which incurs large computational costs with current large lms that have millions or billions of parameters, and may result in an lm highly relevant only to the specific training data. for example, one can fine-tune a large pre-trained lm on product reviews labeled with sentiment to generate positive and negative sentences, but the fine-tuned model will tend to generate sentences like those from product reviews which greatly limits its utility with out-of-domain prompts. both these methods require training all the parameters of the model. alternatively, recent research leverages a discriminator to re-weight output distributions @xcite or to perturb latent representations in the token level such as in pplm @xcite without changing the pre-trained lm. however, raising targetrelevant token probabilities may lead to less fluent sentences. in addition, updating gradients at the token level makes decoding expensive and slow. in this paper, we propose attribute alignment to infuse attribute representations into a pre-trained unconditional lm without changing the lm parameters. we are inspired by language codes which guide multilingual translation models to translate to the target language @xcite . however, because attributes signals are not trained with the lm during large-scale pretraining @xcite , we introduce an alignment function to bridge attribute representations to the lm so that it can interpret the weights in the attribute representations. specifically, we encode an attribute (e.g. positive, negative, business, military, etc.) with a pre-trained lm and learn an alignment function to transform the attribute representation. to train the alignment function, we use the same annotated data used to train discriminators in token-level perturbation methods @xcite so that the self-attention to the aligned attribute represen- ## related work controlled text generation to interpolate a controlling factor, concatenating the attribute to the input sequence is the most straightforward approach and has been commonly used in grounded generation @xcite . @xcite proposes to pre-train a large conditional language model with available labels such as urls for large lm control. this method can be effective in conditional modeling, but requires a substantial amount of resources for pretraining and is limited by the labels used during 1 our code is available at https://github.com/ diandyu/attribute_alignment pre-training (e.g. 55 control codes in ctrl). another approach is to concatenate the attribute representation to the hidden states using linear transformation @xcite or latent variables @xcite . these approaches require training from scratch or fine-tuning the entire pre-trained model to incorporate the external target attributes and model conditional probability @xcite @xcite . in addition, they always require carefully designed kullback-leibler (kl)-divergence and adversarial training to generate out-of training domain text with the desirable attribute only @xcite . in comparison, our proposed method does not require fine-tuning the original lm so that we can make use of the high quality pre-trained lm while controlling the target attributes. instead of fine-tuning the whole model, @xcite proposes to add residual adapters, which are task-specific parameters to transformer layers for each language understanding task. different from adding adapters for each individual attribute @xcite , our method only requires learning one attribute alignment function for all attributes to do controlled generation, and is more flexible at inference time without degrading quality such as diversity @xcite . recently, @xcite proposes to use self-supervised learning with hand-crafted phrases (e.g. "is perfect" to represent positive sentiment), but suffers from high variance, low coherence and diversity in order to ## methodology unconditional language models are trained to optimize the probability of p(x i |x 0:i-1 ) where x i is the next token and x 0:i-1 are already generated tokens. for controlled generation, we need to model the conditional distribution p(x i |x 0:i-1 , a) where a is the attribute for the model to condition on. to make use of large lms trained on unlabeled data, we need to infuse the attribute a into the pre-trained unconditional distribution p(x i |x 0:i-1 ). we introduce attribute alignment to this end. different from fine-tuning the whole lm, our alignment function is the only trainable component while the pre-trained lm parameters are frozen. ## experiments we evaluate our proposed methods a: using attribute representation only; ac: model a with corpus representation for disentanglement; ack: ac with kl disentanglement; and lastly acb: ac with bayes disentanglement. we evaluate these models on sentiment control for thorough comparisons. we use nucleus sampling @xcite for all the methods at inference time. refer to appendix a.4 for implementation details. ## results and analysis we show controlled examples in ## conclusion in this paper, we propose a simple but effective attribute alignment model for conditional language generation on top of non-controlled pre-trained lm without fine-tuning lm parameters. we also introduce disentanglement methods to separate different features from the training corpus to further preserve the original pre-trained lm distribution. evaluated on sentiment and topic control, we show that our proposed method outperforms the previous methods on attribute control while maintaining language generation quality. for future work, we plan to apply the proposed methods on other attributes such as dialog act and explore few-shot learning settings of the training corpus.