B eam R : Beam Reweighing with Attribute Discriminators for Controllable Text Generation (2022)

Abstract: Recent advances in natural language processing have led to the availability of large pre-trained language models (LMs), with rich generative capabilities. Although these models are able to produce fluent and coherent text, it remains a challenge to control various attributes of the generation, including sentiment, formality, topic and many others. We propose a Beam Reweighing (BeamR) method, building on top of standard beam search, in order to control different attributes. BeamR combines any generative LM with any attribute discriminator, offering full flexibility of generation style and attribute, while the beam search backbone maintains fluency across different domains. Notably, BeamR allows practitioners to leverage pre-trained models without the need to train generative LMs together with discriminators. We evaluate BeamR in two diverse tasks: sentiment steering, and machine translation formality. Our results show that BeamR performs on par with or better than existing state-of-the-art approaches (including fine-tuned methods), and highlight the flexiblity of BeamR in both causal and seq2seq language modeling tasks.

## introduction text generation has improved significantly in recent years due to architectural advances in deep learning (namely, the transformer architecture and attention mechanism @xcite ) and training paradigms, allowing practitioners to train large language models on vast, unlabelled corpora, and transfer knowledge between various domains. controllable text generation involves generating text according to specific requirements, which may include a specific topic @xcite , attribute @xcite , reward signal @xcite , or other potential constraints. this task presents significant challenges, © 2022 the vanguard group, inc. all rights reserved. as large, unlabelled corpora are unlikely to be sufficient for learning domain-specific, controllable characteristics, and thus transferring knowledge becomes substantially more difficult. moreover, due to the growing size of recent language models it is also less feasible to train and finetune them for many different controllable dimensions. recent work in controllable text generation involves various ways of incorporating desired attributes into the text generated by the base lm. many approaches @xcite @xcite rely directly on decoding-time strategies in order to steer the generation towards a desired attribute. however, these approaches typically rely on token-level decoding which can result in various disfluencies in the output (e.g., repetition) @xcite or limited generalizability due to tight coupling between the generation and attribute models. several works @xcite @xcite @xcite attempt to tune a portion of the base lm in order to steer it towards a desired attribute. this tuning is either performed directly on the lm (i.e. via a fine-tuning stage), or using an auxiliary attribute model and applying gradient perturbations to lm latent states. in this work, we propose a simple and robust decoding-based approach to controllable text generation, allowing practitioners to leverage existing, pre-trained, generative lms and existing attribute models. our method first uses the beam search algorithm to propose fluent and relevant candidates for a given input prompt from a generative language model. subsequently, the candidates are scored by a discriminative model trained for a particular attribute (e.g., sentiment analysis, emotion detection, or topic classification). the candidate scores produced by beam search are combined with the scores from the discriminative model to produce a distribution over the candidates. we then sample a single candidate generation from this distribution. our method solves some of the existing issues in controllable text generation approaches, by (1) leveraging beam search to produce more fluent and relevant candidates, (2) expanding the generalizability of controllable generation via a custom similarity measure that can be selected based on the discriminative model, and (3) eliminating the need for tight coupling between the generative and discriminative models by reweighing at the natural language level, agnostic to the tokenization scheme, thereby allowing practitioners to leverage strong models for generation and scoring. we perform several experiments with our approach, compare to several state-of-the-art methods for controllable text generation and show that beamr is generalizable to various lms and target applications. first, we experiment with controlling the sentiment of generations using an attribute model finetuned for sentiment analysis. we then highlight the generalizability of the beamr method by applying it to the sequence-to-sequence task of adjusting the formality of text translated from spanish to english. in sentiment steering experiments, beamr outperforms the sota dexperts model @xcite in positive steering, and offers good control ability in negative steering, while significantly outperforming all baselines in terms of fluency. we perform a human evaluation study on the sentiment steering task which aligns with the observations from automated evaluations. in machine translation formality experiments, beamr outperforms the fudge baseline in both translation accuracy and formality score. hyperparameter experiments with beamr in both tasks highlight potential tradeoffs between fluency and attribute control. ## background generative language models learn to produce a distribution for the next token in a sequence given past context as input. given a prompt sequence of tokens, c @xmath0, x 2 , . . . , x t } where x @xmath1, we can produce a distribution p(x | c t ) for the next token in the sequence, greedy. in this approach, tokens are generated by iteratively choosing the most likely token from p(x | c t ), and updating the prompt c. ## beam reweighing we propose to modify the beam search algorithm by reweighing the candidate likelihoods in order to control a diverse set of attributes of the text, such as sentiment, formality, emotion or topic. our method first decodes a set of k candidates, using diverse beam search @xcite to improve variety among the candidates. the candidates are then scored using an attribute model. we then reweigh their likelihoods ℓ(b) with the attribute scores s and apply a softmax transformation to produce a reweighed candidate distribution p, encoding fluency and attribute characteristics. the reweighed distribution is used to sample a single candidate. more formally, let b @xmath2, . . . , b k } denote the set of candidates for iteration j of beamr and b @xmath3, with likelihood ℓ(b k ). let g φ : p(v) → r m represent a discriminator for an m-dimensional attribute. given a target attribute vector @xmath4, we compute a score for candidate b k : combining the attribute score s @xmath5, a) with the likelihood ℓ(b k ) gives us a reweighed distribution p over b j : a candidate can then be sampled from this distribution, b ∼ p. this formulation is akin to a product of experts model @xcite treating the lm f θ as a linguistic expert and the discriminator g φ as an attribute expert. figure 1 presents a diagram of the beamr procedure. the detailed pseudo-code for a single iteration of beamr is presented in algorithm 1. algorithm 1 beam reweighing 1: procedure beamr(x, a; d, k, t, γ) 2: ## evaluation we conduct several experiments in order to evaluate beamr against sota controllable generation approaches, in various applications. we focus on (1) a sentiment steering task, whereby we generate positive or negative continuations to a variety of prompts (including positive, negative and neutral prompts), and (2) a machine translation formality task, whereby input sentences are translated to english and the translations are adjusted in order to improve the formality of the text, whilst maintaining the original meaning. we detail the relevant datasets, baselines and metrics for each experiment. we also conduct an analysis of hyperparameter selection for both tasks. ## related work recent methods in controllable text generation @xcite may be categorized under decoding methods and tuning methods. roughly speaking, decoding methods apply controllable characteristics only at the output distribution of a lm, while tuning methods additionally attempt to encode controllable characteristics into the generative lm itself, by tuning either some or all of its parameters. ## conclusion we present a simple and modular decoding-based approach to controllable generation, beamr. beamr combines a generative lm with an attribute discriminator and leverages beam search decoding in order to steer generated text to the desired target attribute. we show the results of beamr in two diverse tasks: sentiment-based steering, and machine translation formality steering. our results from automated evaluations show that beamr outperforms strong baselines for both tasks, and human evaluations for sentiment steering further support this. noticeably, beamr struggles with negative sentiment steering, especially when compared to gedi and dexperts. we hypothesize this may be due to gedi and dexperts having direct access to class-conditioned distributions in their generation. namely, gedi trains a class-conditioned lm using control codes and anti-control codes (@xmath6. future work on beamr may incorporate additional sources of language and attribute information to address this shortcoming. beamr offers a great deal of flexibility by allowing us to plug different and independent generative lms and attribute discriminators (with potentially different tokenization schemes). moreover, beamr generalizes beyond classification tasks to any type of discriminator by appropriately selecting a similarity measure. leveraging beam search for text decoding from a lm, beamr's generations avoid some of the typical problems with token-based decoding (such as repetition or disfluencies). our work highlights that strong controllable text generation can be achieved by mixing together large pre-trained generative and discriminative models, with a flexible backbone offered by beamr, without sacrificing fluency.