Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning (2023)

Abstract: It has always been an important yet challenging problem to control language models to avoid generating texts with undesirable attributes, such as toxic language and unnatural repetition. We introduce Leo for controllable text generation, which needs no modification to the model architecture and facilitates out-of-the-box use of trained models. It employs a contrastive loss on sequence likelihood, which fundamentally decreases the generation probability of negative samples (i.e., generations with undesirable attributes). It also adopts a novel likelihood ranking-based strategy to construct contrastive samples from model generations. On the tasks of language detoxification, sentiment steering, and repetition reduction, we show that Leo outperforms strong baselines of controllable text generation and demonstrate the superiority of Leo’s sample construction strategy.

## introduction current language models trained on massive textual corpora have shown the impressive capability of generating fluent and grammatical text @xcite @xcite . however, they often produce behaviors misaligned with human expectations. for instance, language models may generate offensive language or agree with toxic input @xcite @xcite . they may also generate text with unnatural repetition @xcite , which is a notorious issue in autoregressive language generation. controlling language models to avoid such undesirable attributes has always been an important yet challenging problem in nlg research. as a popular practice, growing recent work has investigated how to decrease the generation prob- (1) generating multiple continuations given a prompt, which are labeled as positive / negative by a label function. in this work, we introduce click, a method for controllable text generation with sequence likelihood c(k)ontrastive learning. it employs a max-margin contrastive loss on sequence likelihood in addition to standard language modeling ( § 2.2), which fundamentally reduces the probability of a negative sample being decoded. compared with previous methods of controllable text generation, click has two unique advantages. first, click contrasts the sequence likelihoods of positive and negative samples with a maximum likelihood margin, which enables a higher degree of freedom for optimization than explicitly minimizing the likelihood of each token of negative samples. second, click needs no modification to the model architecture and thus does not require laborious adjustments to the next-token prediction distribution during generation, which makes it convenient for out-of-the-box use of trained models. we also design a likelihood ranking-based strategy of contrastive sample construction for click ( § 2.3). given an input prompt, click first samples multiple generations from the initial language model, which are labeled as positive / negative by a label function. it then pairs each negative sample with the positive one whose likelihood ranks highest but lower than the former. for instance, in figure 1 , negative rank 2 is paired with positive rank 4 to constitute a pair of contrastive samples. this strategy derives from our two intuitions. first, a high-likelihood positive sample (e.g., positive rank 1 ) does not necessitate further enlargement of its likelihood gap with the negative one, which may instead result in overfitting the positive sample. second, sequence likelihood indicates how much a text is probable to be the continuation of the input, which somewhat reflects the quality of generated continuations, such as fluency and coherence. a pair of samples with a too large likelihood gap (e.g., negative rank 2 and positive rank 6 ) may thus bias contrastive learning toward other aspects (e.g., fluency or coherence) than the attributes we aim to control. we experiment with three controllable text generation tasks: language detoxification, sentiment steering, and repetition reduction ( § 3). through both automatic and human evaluation, we show that click can effectively avoid undesirable attributes and outperform strong baselines. ablation analysis further proves the superiority of click's sample construction strategy. ## experiments we next show that click can effectively avoid undesirable attributes on three controllable text generation tasks: (1) language detoxification ( § 3.1), (2) sentiment steering ( § 3.2), and (3) repetition reduction ( § 3.3). we also conduct ablation analysis to give further insights about click ( § 3.4). ## conclusion this work introduces a controllable text generation method click, which needs no modification to the model architecture and facilitates out-of-the-box use of trained models. it employs a contrastive loss on sequence likelihood and adopts a likelihood ranking-based strategy for contrastive sample construction. our empirical evaluation on the tasks of language detoxification, sentiment steering, and repetition reduction demonstrates that click can effectively avoid undesirable attributes in language generation and outperforms strong baselines. ablation analysis gives further insights about click's sample construction strategy, hyperparameters, and combination with iterative training. future work can investigate the combination of click and various label (or reward) functions @xcite .