Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models (2022)

Abstract: Recent work on controlled text generation has either required attribute-based fine-tuning of the base language model (LM), or has restricted the parameterization of the attribute discriminator to be compatible with the base autoregressive LM. In this work, we propose Mix and Match LM, a global score-based alternative for controllable text generation that combines arbitrary pre-trained black-box models for achieving the desired attributes in the generated text without involving any fine-tuning or structural assumptions about the black-box models. We interpret the task of controllable generation as drawing samples from an energy-based model whose energy values are a linear combination of scores from black-box models that are separately responsible for fluency, the control attribute, and faithfulness to any conditioning context. We use a Metropolis-Hastings sampling scheme to sample from this energy-based model using bidirectional context and global attribute features. We validate the effectiveness of our approach on various controlled generation and style-based text revision tasks by outperforming recently proposed methods that involve extra training, fine-tuning, or restrictive assumptions over the form of models.

## introduction while large transformer-based autoregressive language models trained on massive amounts of data found on the internet exhibit exceptional capabilities to generate natural language text, effective methods for generating text that satisfy global constraints and possess holistic desired attributes remains an active area of research. these mechanisms for controlling the generation of language have the potential to mitigate undesirable biases encoded by the large language models and prevent the generation of hate speech and toxic language @xcite @xcite @xcite . much of the prior work has approached controlled gener-ation via either training domain-conditioned neural language models @xcite @xcite @xcite @xcite or finetuning/modifying an underlying large pre-trained base model for generation on domain-specific data for attribute sensitive generation @xcite @xcite @xcite . not only do these approaches involve computational overhead and estimation errors associated with the training of language models, but they are also dependent on access to a large amount of attribute-specific language data which can be impractical in many scenarios and exacerbate privacy concerns @xcite @xcite . our approach eschews training and focuses on generation-time control from pre-trained modules. recent work in this space has used attribute discriminators @xcite @xcite to steer the generation from a large autoregressive language model. these discriminators need to be separately trained on partial generations in order to be operationalized with step-wise autoregressive models. as a result, this approach also requires availability of data to train step-wise discriminators for attributes that are essentially global (at the sequence-level) in nature. therefore, we focus on drawing samples from a test-time combination of pretrained blackbox experts that each score a desired property of output text -for example, fluency, attribute sensitivity, or faithfulness to the context. specifically, we view the product of these black-box experts as a probabilistic energy model @xcite -i.e., a non-autoregressive, globally normalized language model -and then sample (without further training or fine-tuning) using a specialized gibbs sampler with a metropolis-hastings correction step @xcite . the cake is stale. ## related work the approaches closest in spirit to our work involve steering generation from a base language model with external attribute-sensitive control mechanisms. plug-and-play lm @xcite uses discriminators learned from an autoregressive lm's top-level hidden layer to modify the lm's states toward increasing the probability of the desired attribute via gradient ascent at each step. gedi @xcite and fudge @xcite take a similar approach but train custom step-wise attribute-sensitive discriminators that decide whether the desired attribute is likely to be satisfied by the current generation path. gedi trains class-conditional language models for these discriminators and hence additionally relies on access to attribute sensitive language data. @xcite formulate the task of controlled generation as optimizing the base lm's likelihood subject to global differentiable attribute-based constraints by gradient descent over the position-wise simplexes over the vocabulary. dexperts @xcite is another decoding-time controllable generation approach that modifies the step-wise softmax logits of an autoregressive pre-trained lm with softmax log-its of separately trained domain-specific expert autoregressive language models. these approaches require training of custom modules and do not readily enjoy the benefits of incorporating global attributebased features into the generation mechanism in a simple probabilistic manner. in contrast, our energybased formulation is not only optimization-free but also fully modular and able to easily incorporate global features, allowing for heterogeneous blackbox experts to be combined with each other. ## mix-and-match language models in this section, we describe our approach and motivation behind our method. specifically, we frame the problem of performing controlled generation as a problem of sampling from a specialized energybased (or globally normalized) sequence model that defines a probability distribution that satisfies the desired constraints we wish to impose in the controlled generation setting. as described below, this energybased model is composed of pre-trained components and does not require any further optimization. an energy-based sequence model defines the probability distribution over the space of possible sequences θ) , where e(x;θ) refers to the scalar energy of a sequence x that is parametrized by θ. lower energy corresponds to the higher likelihood of x. in contrast to the common autoregressive sequence models, exact likelihood computation and efficient sampling from these models is challenging. despite these challenges, we focus on this paradigm of sequence modeling because energy-based models offer increased flexibility via sequence-level features and constraints. as we discuss next, this capability lets us easily define expressive functions for controlled generation of sequences which is not readily offered by the autoregressive modeling paradigm. ## experimental setup we provide full experimental details in appendix section b, here we provide a brief overview of the tasks, datasets, baselines, and metrics used in the experiments. ## conclusion we present mix and match language models (m&m lm), a training-free framework for controlled text generation that can easily mix heterogeneous expert modules. we show that our framework outperforms prior methods on a suite of text revision and attribute-controlled generation tasks. further, our results indicate that probabilistic energy language models, typically considered intractable, can be used for practical text generation tasks when combined with an appropriate sampling scheme. due to page limitations in the body of the paper, we include more sample generations from our method in the form of tables here. we have no samples from the formality transfer task, however, since the data used (gyafc) is protected and needs permissions for access, so we cannot publish it. however, we have provided code needed to reproduce our results, once access to the original data is gained.