COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
Abstract
Many applications of text generation require incorporating different constraints to control the semantics or style of generated text.
These constraints can be hard (e.g., ensuring certain keywords are included in the output) and soft (e.g., contextualizing the output with the left- or right-hand context).
In this paper, we present Energy-based Constrained Decoding with Langevin Dynamics (C OLD ), a decoding framework which uniﬁes constrained generation as specifying constraints through an energy function, then performing efﬁcient differentiable reasoning over the constraints through gradient-based sampling.
C OLD decoding is a ﬂexible framework that can be applied directly to off-the-shelf left-to-right language models without the need for any task-speciﬁc ﬁne-tuning, as demonstrated through three challenging text generation applications: lexically-constrained generation, abductive reasoning, and counterfactual reasoning.
Our experiments on these constrained generation tasks point to the effectiveness of our approach, both in terms of automatic and human evaluation.
Introduction
Many text generation applications require producing text that is not only ﬂuent but also satisﬁes various constraints which control the semantics or style of the generated text.
For example (Figure 1), for knowledge-grounded or keyword-guided generation, we might want to ensure that certain keywords are included in the generated output as hard lexical constraints [29, 52].
For other types of text generation, we often wish to incorporate soft topical constraints to contextualize the desired output, e.g., abductively [43] reasoning about what happened in the middle of a story given the past and the future story context [1].
Yet another class of text generation applications requires revising an input based on a new counterfactual condition [13] which simultaneously requires semantic coherence as well as minimal-edit constraints with respect to the input text [44].
The dominant paradigm to various text generation applications has been supervised learning with task-speciﬁc training data.
However, different applications require varied and potentially evolving constraints, and annotating a large amount of task-speciﬁc training data for each different combination of constraints can be costly.
Recent work has explored incorporating constraints through energy-based text modeling that alleviates the need of supervised data [23, 7, 41].
Yet those approaches still require expensive training of speciﬁc generation models.
In addition, training might not even be feasible with recent models that are extreme in scale, like GPT-3 [3].
This motivates the need to enrich decoding algorithms that can work directly with pretrained language models without task-speciﬁc ﬁne-tuning, and support complex combinations of hard and soft constraints to control the generated text on the ﬂy.
Minimal Edit
Abductive Reasoning x l
Joe lived a lonely life.
x r
Joe decided to join a social club.
He was much happier now that he has companions.
Past
Future y Generation
COLD: = left coherence + right coherence E
Lexically Constrained Generation  y
A pet cat likes to sleep on a couch .
COLD: = fluency + key words E keywords
Generation pet couch cat
Counterfactual Reasoning x r x  l x l Beginning
Ending
Counterfactual
The law student
The medical student joined a prestigious law firm after graduation.
joined a prestigious medical practice after graduating.
y Generation
COLD : =
coherence + minimal edit E
Minimal Edit
We propose a new constrained decoding approach that formulates decoding as sampling from an energy-based model (EBM) [16, 27].
Constrained generation with our approach amounts to specifying an energy function by plugging in arbitrary constraint functions that are suitable for the task at hand, then sampling from its induced distribution.
In particular, to overcome the longstanding challenges of sampling discrete text from EBMs, we for the ﬁrst time introduce Langevin dynamics [53] to text-based EBMs for efﬁcient gradient -based sampling.
As a result, our approach, Constrained
Decoding with Langevin Dynamics (C OLD ), performs sampling by iteratively updating a continuous relaxation of text using gradients of the energy function.
The resulting continuous text samples are then mapped back to the discrete space with a simple guided discretization approach, yielding text sequences that are ﬂuent and adhere to the constraints.
Our work makes unique contributions to a recent line of research investigating decoding algorithms for incorporating different constraints [45, 6, 33, 26] in three distinct aspects.
First, our formulation uniﬁes various constrained generation scenarios that involve hard lexical constraints and/or soft contextual constraints: specifying an energy function, then sampling from its induced distribution.
Second, we propose a sampling method, which complements decoding algorithms that look for a single optimal solution.
Finally, we provide new empirical insights into the strengths and weaknesses of existing approaches to discrete search and differentiable reasoning.
To test the ﬂexibility and empirical performance of C OLD decoding, we experiment with three challenging text generation tasks: lexically constrained generation [29, 18], abductive reasoning [1], and counterfactual story generation [44].
C OLD achieves better lexical coverage than N EURO L OGIC [33], a beam-based discrete decoding algorithm speciﬁcally designed for lexically constrained generation, while producing more coherent and higher quality text than D ELOREAN [45], a state-of-the-art gradient-based generation method for abductive reasoning and counterfactual reasoning.
C OLD supports all three constrained generation settings under a uniﬁed framework – specifying an energy function using a collection of ﬂuency and task-speciﬁc constraints, then sampling from its induced distribution and achieves strong performance on both automatic and human evaluation.
Background
Neural text generation.
Neural text generation typically involves two stages: modeling a distribution over text sequences, and using a decoding algorithm to generate sequences with the model.
Let y =
( y 1 , . . .
, y T ) denote a discrete sequence where each y t is a token from a vocabulary V .
Common neural language models (e.g., GPT-2/3 [46, 3]) factorize the probability of a sequence into the product of per-token conditionals in left-to-right order, p ✓ ( y ) =
Q
Tt =1 p ✓ ( y t | y <t ) , with each conditional parameterized by a shared neural network, such as transformer [50].
Popular decoding algorithms, ranging from beam search or greedy decoding to sampling methods such as top- k
[12] or nucleus [19] sampling, produce text sequences y using the model p ✓ , often conditioned on a prompt x .
Constrained text generation.
We view text generation as the problem of ﬁnding a sequence that satisﬁes a collection of constraints.
For instance, the scenario above amounts to generating a sequence y
=
( y 1 , . . .
, y T ) subject to a soft constraint that the continuation y should be ﬂuent and logically coherent with the prompt x .
Other constrained generation problems impose additional constraints, such as text inﬁlling [60, 8] where coherence constraints move beyond a left-hand preﬁx, lexically constrained generation in which hard constraints require the output to contain given tokens, and various forms of semantically-constrained generation in which the output is softly constrained to be [Figure] Langevin
[Figure]
similar to another sequence.
Since common decoding algorithms generate text monotonically, relying on p ✓ ( y t | y <t ) for determining the next token, it is challenging to enforce these diverse constraints.
Energy-based models and Langevin dynamics.
Given an energy function E ( y ) 2 R , an energybased model (EBM) is deﬁned as a Boltzmann distribution p ( y ) =
exp {  E ( y ) } /Z , where Z = P y exp {  E ( y ) } is the normalizing factor (The sum is replaced with an integral if y is continuous).
EBMs are ﬂexible, in that one can incorporate arbitrary functions such as constraints into the energy function E ( y ) .
Recent work has thus made attempts to train text-based EBMs each for speciﬁc tasks [21, 41, 7, 23].
As discussed earlier, we instead use the energy-based formulation to develop an inference (decoding) procedure that enables off-the-shelf pretrained language models to perform arbitrary constrained generation, without any ﬁne-tuning.
Despite the ﬂexibility, however, sampling from an EBM is particularly challenging, as computing
Z is intractable.
Common gradient-free Markov chain Monte Carlo (MCMC) methods such as
Gibbs sampling [2] can be used, but they are often prohibitively slow [10, 38].
Langevin dynamics [53, 37, 34], a gradient-based MCMC method, offers more efﬁcient sampling by using the gradient of the energy function r y E ( y ) , enabling sampling in domains such as image generation [9, 48].
However, since text is discrete, the gradient r y E ( y ) is not well-deﬁned, making it non-trivial to apply Langevin dynamics for sampling text from an EBM.
Our approach bridges this gap with continuous relaxation of text, differentiable constraints, and guided discretization, as described below.
COLD Decoding with Langevin Dynamics
To enable ﬂexible and diverse constrained generation in off-the-shelf language models, we develop
Constrained Decoding with Langevin Dynamics (C OLD ), a decoding approach that treats text generation as sampling from an energy-based distribution, allowing for ﬂexibly composing constraints based on the task at hand.
C OLD decoding generates text by sampling from an EBM deﬁned over a sequence of “soft” tokens using Langevin dynamics, then maps the continuous sample into discrete, ﬂuent text.
We provide our formulation of constrained text generation (§3.1), present differentiable constraints that can be composed into energy functions (§3.2) along with our discretization method (§3.3), and discuss practical details of C OLD decoding (§3.4).
Figure 2 provides an overview.
3.1 Energy-based Decoding
Constrained text generation aims to produce text samples y that satisfy a set of constraints (usually conditioned on an input x omitted for brevity).
We assume each constraint can be captured by a constraint function
f i ( y ) 2 R , where higher values of f i mean that the text y better satisﬁes the constraint.
For example, f i could measure the likelihood of y as a ﬂuency constraint (more in §3.2), while a hard constraint f i amounts to a large negative penalty when y does not satisfy the constraint.
( ) , ) LM ˜ y < t ˜ y t
CrossEntropy
Softmax p  LM (  | ˜ y < t )
LM ˜ y has eight legs x r : never gave up gave up ˜ y : 2-gram hand , sink , soup soup ˜
y : y * : 1-gram hand (cid:1153) 1 (cid:1154) (cid:1153) 2 (cid:1154) (cid:1153) 3 (cid:1154)
y * :
Soft Match Soft Match
The set of constraints induces a distribution over text, written in an energy-based form as:
[Equation]
where  i  0 is the weight of the i th constraint, Z is the normalizing factor.
Here E ( y ) :
=  P i  i
f i ( y ) is the energy function.
This energy-based form is ﬂexible, as one can plug in any constraint functions required for a task of interest.
Generating text under the constraints can then be seen as sampling from the energy-based distribution y ⇠ p ( y ) .
One can also draw multiple samples and pick the best if only one sample is needed, as discussed later (§3.4).
As mentioned above, for efﬁcient sampling from p ( y ) we want to use Langevin dynamics, which makes use of the gradient r y E ( y ) .
However, in our case y is a discrete sequence and the gradient r y E ( y ) is not well-deﬁned.
As a result, we perform Langevin dynamics with an energy deﬁned on a sequence of continuous token vectors, described below.
Differentiable decoding with Langevin dynamics.
Instead of deﬁning the energy function on discrete tokens, we deﬁne the energy function on a sequence of continuous vectors ˜ y
=
(˜ y 1 , . .
˜ y T ) , which we call a soft sequence.
Each position in the soft sequence is a vector ˜ y t 2 R V , where V is the vocabulary size, and each element ˜
y t ( v ) 2 R corresponds to the logit of word v in the vocabulary.
Taking the softmax of ˜ y t yields a distribution over the vocabulary for position t , ˜ p ⌧ t = softmax (˜ y t / ⌧ ) .
As ⌧ ! 0 , ˜ p ⌧ t becomes a one-hot vector, indicating a discrete token.
By specifying an energy E (˜ y ) on the soft sequence ˜ y , we can use Langevin dynamics to obtain a sample.
Speciﬁcally, the sampling is done by forming a Markov chain:
[Equation]
where ⌘ > 0 is the step size, and ✏ ( n ) 2 N (0 ,  ) is the noise at iteration n .
As shown in Welling and
Teh [53], by adding the right amount of noise and annealing the step size, the procedure will converge to samples from the true distribution.
That is, if we let p ( n ) be the distribution
such that ˜ y ( n ) ⇠ p ( n ) , then as n ! 1 and  ! 0 , we have p ( n ) ! p (˜ y ) :
= exp {  E (˜ y ) } /Z .
That is, the procedure ends up generating samples from the distribution induced by the energy function.
Next, we describe constraint functions deﬁned on the soft sequence ˜ y that can be plugged in as components of the energy function.
Later in §3.3, we describe how to obtain a discrete sequence from a soft sequence sample ˜ y .
3.2 A Collection of C OLD Constraints
C OLD provides a ﬂexible framework for plugging in a wide range of constraint functions for a task of interest.
We describe constraint functions that are useful in various constrained generation problems, such as those we consider in the experiments (§4).
The constraints include language model-based ﬂuency constraints, along with lexical and semantic constraints on the sequence content.
More generally, any differentiable function that outputs a goodness score of (soft) text can be used as a constraint function, as long as it reﬂects the requirements of the target task.
Soft ﬂuency constraint.
Fluency is a common requirement for generated text.
To promote ﬂuency, we use a constraint which favors soft sequences that receive high probability according to the underlying left-to-right LM p ! LM (e.g., GPT2):
[Equation]
where p ! LM ( ·| ˜ y <t ) means the next-token distribution when providing the neural language model with the preceding soft tokens ˜
y <t (i.e., feeding the weighted average of word embeddings, with the weights being softmax (˜ y t 0 / ⌧ ) for t 0 < t [20, 45]).
Intuitively, the constraint says that each token distribution in the soft sequence, softmax (˜ y t ) , must match the “reference” distribution p ! LM ( ·| ˜ y <t ) predicted by the underlying language model.
The match is measured by the (negative) cross-entropy between the two distributions.
The constraint thus encourages ﬂuency.
In practice, if there is left-side context x for the generation to condition on, we feed x to the LM to form the “reference” distribution p ! LM ( ·| ˜ y <t , x ) .
As a result, ˜ y is encouraged to be ﬂuent and coherent with the context x .
We can easily incorporate an additional reverse LM constraint, f LM , using a right-to-left LM p LM ( ·| ˜ y >t ) , as an additional ﬂuency constraint.
Flexibly leveraging multiple models in this way is infeasible with conventional decoding methods such as beam search or nucleus sampling.
Future-token prediction constraint.
Applications such as text inﬁlling involve future input tokens that remain ﬁxed, but should contribute to updating past positions.
For instance, consider updating the second position of The __ has eight legs .
A sample should be coherent with the tokens x r on the right (i.e., has eight legs ).
To this end, we use a constraint that adjusts soft tokens to maximize the likelihood of input tokens x r ,
[Equation]
where K is the length of x r .
In other words, the constraint adjusts the soft sequence ˜ y
such that the underlying LM predicts the future tokens x r after seeing ˜ y .
N-gram similarity constraint.
Many constrained generation scenarios pose requirements on the wording and expression of generated text sequences.
For instance, lexically constrained generation tasks [18] require certain keywords to be presented in the text samples, while counterfactual reasoning [44] or text editing [15, 31] tasks require the text to retain the essence of a reference sequence.
We formulate these requirements as an n -gram similarity constraint which favors sequences that overlap with a reference y ⇤ at the n -gram level,
[Equation]
where ngram-match ( · , · ) is a recent differentiable n -gram matching function [32] which can be seen as a differentiable approximation to the BLEU- n metric [40].
When n = 1 and y ⇤ a sequence of keywords, the constraint in effect enforces ˜ y to assign higher values to the keywords ( 1 -grams).
When n is larger and ˜ y ⇤ is a reference sequence, the constraint encourages ˜ y to resemble the reference by assigning high values to tokens making up n -grams from y ⇤ .
3.3 From Soft to Discrete and Fluent Text
After receiving a soft sequence sample ˜ y from running Langevin dynamics (Eq. 2), we map the soft sequence to a discrete text sequence which we consider as the output of C OLD decoding.
A simple
[Table]
method would be selecting the most-likely token at each position t , y t = arg max v ˜ y t ( v ) .
However, the resulting text can suffer from ﬂuency issues even if the soft ﬂuency constraint (Eq. 3) is used, due to competing constraints that sacriﬁce ﬂuency.
To overcome this, we use the underlying LM (e.g.,
GPT2-XL)
as a “guardian” for obtaining the discrete sequence.
Speciﬁcally, at each position t , we ﬁrst use the LM to produce the top- k most-likely candidate tokens based on its generation distribution conditioning on preceding tokens, which we denote as V kt .
We then select from the top- k candidates the most likely token based on the soft sample ˜ y :
[Equation]
We refer to this method as “top- k ﬁltering”.
The resulting text tends to be ﬂuent because each token is among the top- k most probable tokens from the LM [12].
In practice, to ease the satisfaction of certain constraints (e.g. n -gram similarity), we expand the candidate set V kt to include constraint tokens (e.g., in the tasks of abductive reasoning §4.1 and lexically constrained decoding §4.3).
Figure 2 illustrates the decoding procedure to get one output from C OLD decoding.
Algorithmsummarizes the algorithm.
Next, we move to practical considerations of applying C OLD .
3.4 Implementation of C OLD Decoding
Sample-and-select.
C OLD decoding allows for drawing multiple text samples from the distribution induced by the energy function E (˜ y ) .
Depending on task requirements, we could either present the set of samples as output, or select one from the set based on some criteria (e.g., different energy terms) and return a single sequence, as in those tasks considered in the experiments (§4).
This “sample-andselect” approach differs from deterministic constrained decoding methods, which optimize only one sequence [e.g., 33, 26], and is used widely in various generation settings [e.g., 28, 11, 4].
Initialization.
We initialize the soft sequence ˜ y by running greedy decoding with the LM p LM to obtain output logits.
In our preliminary experiments, the initialization strategy had limited inﬂuence on the generation results.
Noise schedule.
Each iteration of Langevin dynamics adds noise ✏ ( n ) ⇠ N (0 ,  ( n ) ) to the gradient (Eq. 2).
We gradually decrease  ( n ) across iterations, which intuitively transitions the decoding procedure from exploration to optimization.
In our experiments, we typically used the schedule which sets/reduces  to { 1 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 } at iterations { 0 , 50 , 500 , 1000 , 1500 } , respectively.
Long sequences.
C OLD decoding produces a ﬁxed-length sequence
y =
( y 1 , . . .
, y T ) .
To produce longer sequences, e.g. in cases where y T is not the end of a sentence, we use p LM to produce a continuation of y using greedy decoding.
Experiments
We evaluate C OLD on three constrained generation tasks.
Using C OLD for each task amounts to specifying a set of task-speciﬁc constraints (instances of those in §3.2).
Our focus is enabling constrained generation for settings in which ﬁne-tuning is infeasible, through changing the decoding method.
Thus, our experiments (i) use off-the-shelf LMs without ﬁne-tuning, and (ii) compare C OLD primarily against alternative decoding methods.
As our base LM, we use GPT2-XL [46].
4.1 Abductive Reasoning
We study a speciﬁc formulation of abductive reasoning [43] as a language generation challenge.
Speciﬁcally, given a beginning sentence x l and an ending sentence x r , the abductive language generation ( ↵ NLG) problem [1] consists of generating a bridge sentence y that ﬁlls in between the two sentences and forms a coherent full story (see Figure 1 for example).
The task is particularly challenging for conventional monotonic left-to-right LMs (such as GPT-2 and GPT-3) since it requires non-monotonic reasoning that not only conditions on the past context ( x l , on the left), but also the future story ending ( x r , on the right).
4.1.1 The C OLD Solution
C OLD decoding can readily accommodate the abductive reasoning task by simply plugging in appropriate constraints to specify an energy function.
Speciﬁcally, the generated text needs to be (1) ﬂuent and consistent with the left context x l , and (2) coherent with the right context x r .
Accordingly, we compose an energy using relevant constraints from §3.2:
[Equation]
That is, we combine (a) a soft ﬂuency constraint (Eq. 3) conditioning on the left sentence x l to enforce ﬂuency and consistency with the left context, and a reverse ﬂuency constraint with a right-to-left
LM conditioning on x r to encourage coherence with the right context; (b) a future-token prediction constraint (Eq. 4) that enforces consistency between the generation y and the story ending x r ; (c) a 1 -gram similarity constraint (Eq. 5) between the generation y and keywords (non-stopwords) in x r (excluding those in x l ), i.e., kw ( x r )  kw ( x l ) , which intuitively promotes a ‘smooth transition’ between x l , y , and x r .
For the energy function in Eq.(7), we select the constraint weights on the dev set.
Throughout the experiments, we set the number of Langevin dynamics steps to N = 2000 , with a step size ⌘ = 0 .(Eq. 2).
We discuss more details of the conﬁgurations in the appendix.
Baselines.
We compare with previous decoding approaches for this task.
In particular, we compare with D ELOREAN [45] which outperformed a wide range of supervised and unsupervised methods on the abductive reasoning task in Qin et al.
Following Qin et al.
[45], we also compare with a
L EFT - ONLY method that generates the continuation of x l without considering the right-side x r , i.e., y ⇠ p LM ( y | x l ) .
Evaluation.
We perform both automatic and human evaluation.
We adopt the standard automatic metrics on the task [1] that measure the minimal edit between the generated text and the humanwritten references on the test set, including BLEU [40], ROUGE [30], CIDEr [51], and BERTScore [58].
For the human evaluation, we follow [45] and let crowdworkers from Amazon Mechanical Turk rate the generations on 200 test examples.
Workers were presented a pair of observations ( x l and x r ) and a generated hypothesis y , and asked to rate the coherence of the hypothesis with respect to the observation x l (i.e., x l y ), the observation x r (i.e., yx r ), and both (i.e., x l yx r ), as well as the grammaticality of the hypothesis y itself, on a 5-point Likert scale.
The average ordinal Krippendorff alpha ( 0  ↵  1 ) [25] is 0.36, indicating a fair inner-annotator agreement.
4.1.2 Results
Table 1 shows the evaluation results on the abductive reasoning task.
Under automatic evaluation (the left panel), C OLD consistently outperforms the previous best unsupervised decoding algorithm
D ELOREAN , as well as the L EFT - ONLY method, in terms of both the lexical overlap metrics (BLEU,
ROUGE and CIDEr) and semantic similarity metric BERTScore.
The human evaluation (the right panel) provide more ﬁne-grained insights.
C OLD achieves the best overall coherence , meaning that the generated y from C OLD ﬁts best with both the left-side context x l and the right-side context x r compared to the other methods.
In contrast, D ELOREAN excels only in terms of the left-side coherence (with x l ), with inferior right-coherence (with x r ).
We speculate this is because of D ELOREAN ’s complex interleaving of forward and backward decoding passes that make it difﬁcult to balance the left- and right-coherence constraints.
In terms of grammaticality, unsurprisingly, L EFT - ONLY obtains the best score as it ignores any other constraints (and fails this task with low coherence scores).
More importantly, C OLD achieves a high grammaticality score along with its high coherence, substantially improving over D ELOREAN .
Example generations in Appendix Table 7 show how
C OLD can reason with the right-hand context (e.g. ‘no heels’), while D ELOREAN ’s generations are contradictory (‘red shoes’ vs. ‘white pair’) or equivalent to those from L EFT - ONLY .
[Table]
As a trivial method, L EFT - ONLY is coherent but fails on minimal-edit.
C OLD is superior to D ELOREAN in terms of most metrics, including human evaluation
[Table]
4.2 Counterfactual Story Rewriting
Next, we consider counterfactual story rewriting [44].
Given a story context x l with ending x r , the task is to generate a new story ending y that is (i) similar to the original ending x r , yet (ii) consistent with a new story context x 0 l (see Figure 1 for example).
The task is challenging as it requires capturing the aspects of future events that are invariant under the new (counterfactual) context, while only making necessary edits for coherence.
4.2.1 The C OLD Solution
To tackle this task, we use C OLD with an energy composed of constraint functions that promote coherence with the new context x 0 l , and minimal edits to the original ending x r : E (
[Equation]
These constraints combine: (a) a soft ﬂuency constraint (Eq. 3) conditioned on x 0 l to promote coherence between the generation y and the new (counterfactual) context x 0 l ; a reverse LM constraint to improve ﬂuency; (b) a n -gram similarity constraint (Eq. 5, n = { 2 , 3 } ) to encourage generating an ending ˜ y
that is close to the original ending x r .
We largely follow the conﬁgurations in §4.1 with some exceptions described in the appendix.
Baselines.
Similar to the setup in §4.1, we compare with D E L OREAN [45], a recent state-of-the-art decoding algorithm.
As a reference, we also report the performance of a trivial solution, L EFT - ONLY , that generates a continuation of x 0 l without considering the minimal edit constraint with the original ending x r .
Thus the method is expected to generate a coherent ending which however does not necessarily resemble the original ending.
Finally, we compare with Mix-and-Match [36], a recent energy-based decoding method with discrete MCMC sampling, using BERT-base and BERT-Large.
Evaluation.
We use the benchmark dataset T IME T RAVEL [44].
The original data contains three sentences in a story ending.
Due to computation constraints, we use the ﬁrst sentence as the original ending and generate a new single-sentence ending accordingly.
Following [44, 45] we conduct both automatic and human evaluation.
For automatic evaluation, we measure BERTScore [58], and
Minimal Edit which computes the overlap of text edits (insertion, deletion, replacement, etc.)
[49] needed to produce the gold ending y ⇤ and the generated ending y , starting from the original ending x r .
We do not use other common metrics such as BLEU since they were shown to be ineffective [44].
For human evaluation, each crowdworker is presented with the original story ( x l , x r ), the counterfactual condition x 0 l , and the generated ending y , and the worker is asked to rate (1) the coherence of ˜ y with respect to x 0 l and (2) the extent to which the generated ending y preserves the details of the original ending x r (“minimal edit”), on a 3-point Likert scale for 200 test examples.
The average ordinal Krippendorff alpha is 0.52, indicating a moderate inner-annotator agreement.
We exclude Mix-and-Match from human evaluation given the signiﬁcant performance gap in automated evaluation.
4.2.2 Results
Table 2 shows the results of automatic and human evaluation in terms of both minimal-edit and coherence.
As expected, the reference method L EFT - ONLY that completely ignores the minimal edit constraint can easily generate a new ending that is coherent with the new context x 0 l .
Compared to the baseline approach D ELOREAN , our method C OLD achieves overall superior performance, with substantially improved coherence score and comparable minimal-edit score by human evaluation.
Mix-and-Match, based on discrete MCMC sampling, performs poorly.
Intuitively, its discrete sampling tends to get stuck in a mode of the target distribution (i.e., the region surrounding the original story ending), and struggles to explore further to ﬁnd samples of interest.
C OLD ’s gradientbased sampling with continuous approximation leads to more efﬁcient and effective exploration and mixing, as evidenced by samples that better meet the task requirements.
See Appendix for examples.
4.3 Lexically Constrained Decoding
Next, we use C OLD for lexically constrained decoding.
Given a set of words W , the task aims to generate a coherent sentence that contains these words (Figure 1).
The task is challenging as it requires proper planning to coherently include the constraint words.
4.3.1 The C OLD Solution
We specify an energy function of the following form:
[Equation]
Speciﬁcally, this energy function incorporates: (a) a soft ﬂuency constraint (Eq. 3) and a reverse LM ﬂuency constraint as in the previous tasks; (b) a 1 -gram similarity constraint (Eq. 5) between the generation ˜ y and the given words W ; (c) a future-token prediction constraint, where we concatenate the constraint words (in an arbitrary order), denoted as c ( W ) , and use it as the right-side content x r in Eq.
Again we use similar conﬁgurations as in §4.1.
More details can be found in appendix.
Baselines.
We compare with a recent state-of-the-art method N EURO L OGIC [33], a beam-search variant speciﬁcally designed for lexically constrained generation which outperformed many supervised and unsupervised approaches in Lu et al.
We also report the results of TSMH [57] as another recent baseline which uses Monte-Carlo Tree Search [5].
Evaluation.
We use the set of constraint words from the C OMMON G EN corpus [29], but adopt the canonical setting that the generated text must contain the exact constraint words (e.g., write ) instead of their variants (e.g., wrote ) [18, 47].
Following previous works [18, 47, 57], we report a measure of constraint words coverage as well as language ﬂuency by evaluating the perplexity of the text .
We also ask crowdworkers to rate the text ﬂuency on a 3-point Likert scale on 200 test examples.
The average ordinal Krippendorff alpha is 0.29, indicating a fair inner-annotator agreement.
4.3.2 Results
Table 3 shows the evaluation results for the lexically constrained decoding task.
C OLD , a general constrained decoding method, is comparable to the state-of-the-art method N EURO L OGIC designed speciﬁcally for dealing with lexical constraints.
In particular, C OLD achieves a higher coverage of given keywords, at the expense of generating slightly less ﬂuent language.
C OLD is also substantially better than lexically constrained decoding method TSMH in terms of both coverage and ﬂuency.
4.4 Additional Analysis
[Table]
Ablation studies.
We ablate two important ingredients of our approach, namely the constraints and the top- k ﬁltering.
Due to space limit, we report the results of constraints and defer the results of top- k ﬁltering to the appendix.
Table 5 shows the human evaluation results for ablations of the constraints used on the abductive reasoning task (Eq. 7).
The n -gram similarity constraint f sim provides the largest contribution to the overall coherence.
The reverse
LM ﬂuency constraint f LM also to some extent helps with the right-side coherence by conditioning on the right-side content x r .
Removing the future-token prediction constraint similarly causes inferior scores in terms of right-side and overall coherence, as expected.
Removing the individual constraints leads to better grammaticality due to less competition among different constraints, at the cost of coherence.
Our uniform treatment of all constraints as energy terms makes it straightforward to balance the different constraints by controlling the constraint weights.
[Table]
[-GV
-]GV
Mix-and-Match a recent energy-based decoding method discrete MCMC sampling (Metropolis-Hastings, in particular) which uses gradient-based sampling, is faster than the gradient-free Mix-and-Match:
C OLD is 30% faster with base LMs of similar sizes (GPT2-M and BERTLarge), and has roughly the same time cost when using a much larger LM (GPT2-XL).
Related Work
Previous works proposed beam search variants for lexically constrained decoding [18, 42, 33] which enforce constraints in a discrete space.
Recent works consider constraint satisfaction by adjusting vocabulary distributions using an additional discriminator or LM [6, 24, 56].
Differing from those approaches that determine the generation token by token auto-regressively, Qin et al.
[45] optimize the whole (soft) token sequence via gradient propagation, which facilitates sequence-level semantic constraints (e.g., right-coherence, minimal-edits).
C OLD also samples complete sequences, while offering a principled and uniﬁed formulation based on energy-based modeling.
Kumar et al.
[26] extend [17] by imposing constraints with a Lagrangian method and optimizing for a single output with gradient descent.
In contrast, our approach based on energy-based sampling (§3.1) allows for generating samples for other utilities (e.g., rank-and-select §3.4, estimating expectations).
We also introduce components for more ﬂuent generations such as the novel discretization procedure.
Also, on the empirical side, we explore a different class of problems and tackle them in the absence of labeled data.
The recent CGMH [35] and TSMH [57], followed by [36, 14], perform constrained decoding with extended Gibbs sampling or Metropolis-Hastings sampling in the discrete text space.
Our energy-based formulation with gradient-based Langevin dynamics sampling produces substantially better results than the discrete TSMH (§4.3).
Sha [47] uses gradient information to guide generation, which, however, is speciﬁcally designed for lexically constrained generation.
Energy-based models (EBMs) have been used for incorporating additional information to train text generation models [7, 23, 41, 21].
In contrast, we focus on the constrained decoding ( inference ) that can be directly applied to pretrained LMs without ﬁne-tuning.
Langevin dynamics is widely used on
EBMs of modalities with continuous values, like images [48, 9, 59], 3D shapes [55], latent features [39], and audio sequences [22].
To our knowledge, we are the ﬁrst to apply Langevin dynamics for (constrained) discrete text generation (with a continuous approximation) for efﬁcient sampling.
Conclusion
We introduce C OLD decoding, an energy-based constrained text generation framework that can express various soft/hard constraints through an energy function, and sample using Langevin dynamics.
C OLD can be applied directly to off-the-shelf LMs without task-speciﬁc ﬁne-tuning.
We showcase its ﬂexibility and strong performance on three distinct applications of constrained text generation.
Acknowledgements
This work was funded in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 401233309), DARPA MCS program through NIWC Paciﬁc (N66001-19-2-4031), the Allen Institute for AI, and Microsoft Research PhD Fellowship.
We thank the XLab research group, and our anonymous reviewers for their feedback on this work.
We also acknowledge the Beaker team ( https://beaker.org ) for their support with experiments.