COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
Abstract
Many applications of text generation require incorporating different constraints to control the semantics or style of generated text.
These constraints can be hard (e.g., ensuring certain keywords are included in the output) and soft (e.g., contextualizing the output with the left- or right-hand context).
In this paper, we present Energy-based Constrained Decoding with Langevin Dynamics (C OLD ), a decoding framework which unifies constrained generation as specifying constraints through an energy function, then performing efficient differentiable reasoning over the constraints through gradient-based sampling.
C OLD decoding is a flexible framework that can be applied directly to off-the-shelf left-to-right language models without the need for any task-specific fine-tuning, as demonstrated through three challenging text generation applications: lexically-constrained generation, abductive reasoning, and counterfactual reasoning.
Our experiments on these constrained generation tasks point to the effectiveness of our approach, both in terms of automatic and human evaluation.
Introduction
Many text generation applications require producing text that is not only fluent, but also satisfies various constraints which control the semantics or style of the generated text.
For example (Figure 1), for knowledge-grounded or keyword-guided generation, we might want to ensure that certain keywords are included in the generated output as hard lexical constraints [29, 52].
For other types of text generation, we often wish to incorporate soft topical constraints to contextualize the desired output, e.g., abductively [43] reasoning about what happened in the middle of a story given the past and the future story context [1].
Yet another class of text generation applications requires revising an input based on a new counterfactual condition [13] which simultaneously requires semantic coherence as well as minimal-edit constraints with respect to the input text [44].
The dominant paradigm to various text generation applications has been supervised learning with task-specific training data.
However, different applications require varied and potentially evolving constraints, and annotating a large amount of task-specific training data for each different combination of constraints can be costly.
Recent work has explored incorporating constraints through energy-based text modeling that alleviates the need of supervised data [23, 7, 41].
Yet those approaches still require expensive training of specific generation models.
In addition, training might not even be feasible with recent models that are extreme in scale, like GPT-3 [3].
This motivates the need to enrich decoding algorithms that can work directly with pretrained language models without task-specific fine-tuning, and support complex combinations of hard and soft constraints to control the generated text on the fly.
Minimal Edit
Abductive Reasoning x l
Joe lived a lonely life.
x r
Joe decided to join a social club.
He was much happier now that he has companions.
Past
Future y Generation
COLD: = left coherence + right coherence E
Lexically Constrained Generation 𝒲 y
A pet cat likes to sleep on a couch .
COLD: = fluency + key words E keywords
Generation pet couch cat
Counterfactual Reasoning x r x ′ l x l Beginning
Ending
Counterfactual
The law student
The medical student joined a prestigious law firm after graduation.
joined a prestigious medical practice after graduating.
y Generation
COLD : =
coherence + minimal edit E
Minimal Edit
We propose a new constrained decoding approach that formulates decoding as sampling from an energy-based model (EBM) [16, 27].
Constrained generation with our approach amounts to specifying an energy function by plugging in arbitrary constraint functions that are suitable for the task at hand, then sampling from its induced distribution.
In particular, to overcome the longstanding challenges of sampling discrete text from EBMs, we for the first time introduce Langevin dynamics [53] to text-based EBMs for efficient gradient -based sampling.
As a result, our approach, Constrained
Decoding with Langevin Dynamics (C OLD ), performs sampling by iteratively updating a continuous relaxation of text using gradients of the energy function.
The resulting continuous text samples are then mapped back to the discrete space with a simple guided discretization approach, yielding text sequences that are fluent and adhere to the constraints.
Our work makes unique contributions to a recent line of research investigating decoding algorithms for incorporating different constraints [45, 6, 33, 26] in three distinct aspects.
First, our formulation unifies various constrained generation scenarios that involve hard lexical constraints and/or soft contextual constraints: specifying an energy function, then sampling from its induced distribution.
Second, we propose a sampling method, which complements decoding algorithms that look for a single optimal solution.
Finally, we provide new empirical insights into the strengths and weaknesses of existing approaches to discrete search and differentiable reasoning.
To test the flexibility and empirical performance of C OLD decoding, we experiment with three challenging text generation tasks: lexically constrained generation [29, 18], abductive reasoning [1], and counterfactual story generation [44].
C OLD achieves better lexical coverage than N EURO L OGIC [33], a beam-based discrete decoding algorithm specifically designed for lexically constrained generation, while producing more coherent and higher quality text than D ELOREAN [45], a state-of-the-art gradient-based generation method for abductive reasoning and counterfactual reasoning.
C OLD supports all three constrained generation settings under a unified framework – specifying an energy function using a collection of fluency and task-specific constraints, then sampling from its induced distribution and achieves strong performance on both automatic and human evaluation.
Background
Neural text generation.
Neural text generation typically involves two stages: modeling a distribution over text sequences, and using a decoding algorithm to generate sequences with the model.
Let y =
( y 1 , . . .
, y T ) denote a discrete sequence where each y t is a token from a vocabulary V .
Common neural language models (e.g., GPT-2/3 [46, 3]) factorize the probability of a sequence into the product of per-token conditionals in left-to-right order, p θ ( y )
=
(cid:81)
Tt =1 p θ ( y t | y <t ) , with each conditional parameterized by a shared neural network, such as transformer [50].
Popular decoding algorithms, ranging from beam search or greedy decoding to sampling methods such as top- k
[12] or nucleus [19] sampling, produce text sequences y using the model p θ , often conditioned on a prompt x .
Constrained text generation.
We view text generation as the problem of finding a sequence that satisfies a collection of constraints.
For instance, the scenario above amounts to generating a sequence y
=
( y 1 , . . .
, y T ) subject to a soft constraint that the continuation y should be fluent and logically coherent with the prompt x .
Other constrained generation problems impose additional constraints, such as text infilling [60, 8] where coherence constraints move beyond a left-hand prefix, lexically constrained generation in which hard constraints require the output to contain given tokens, and various forms of semantically-constrained generation in which the output is softly constrained to be
[Figure]
similar to another sequence.
Since common decoding algorithms generate text monotonically, relying on p θ ( y t | y <t ) for determining the next token, it is challenging to enforce these diverse constraints.
Energy-based models and Langevin dynamics.
Given an energy function E ( y ) ∈ R , an energybased model (EBM) is defined as a Boltzmann distribution p ( y ) =
exp {− E ( y ) } /Z , where Z = (cid:80) y exp {− E ( y ) } is the normalizing factor (The sum is replaced with an integral if y is continuous).
EBMs are flexible, in that one can incorporate arbitrary functions such as constraints into the energy function E ( y ) .
Recent work has thus made attempts to train text-based EBMs each for specific tasks [21, 41, 7, 23].
As discussed earlier, we instead use the energy-based formulation to develop an inference (decoding) procedure that enables off-the-shelf pretrained language models to perform arbitrary constrained generation, without any fine-tuning.
Despite the flexibility, however, sampling from an EBM is particularly challenging, as computing
Z is intractable.
Common gradient-free Markov chain Monte Carlo (MCMC) methods such as
Gibbs sampling [2] can be used, but they are often prohibitively slow [10, 38].
Langevin dynamics [53, 37, 34], a gradient-based MCMC method, offers more efficient sampling by using the gradient of the energy function ∇ y E ( y ) , enabling sampling in domains such as image generation [9, 48].
However, since text is discrete, the gradient ∇ y E ( y ) is not well-defined, making it non-trivial to apply Langevin dynamics for sampling text from an EBM.
Our approach bridges this gap with continuous relaxation of text, differentiable constraints, and guided discretization, as described below.
COLD Decoding with Langevin Dynamics
To enable flexible and diverse constrained generation in off-the-shelf language models, we develop
Constrained Decoding with Langevin Dynamics (C OLD ), a decoding approach that treats text generation as sampling from an energy-based distribution, allowing for flexibly composing constraints based on the task at hand.
C OLD decoding generates text by sampling from an EBM defined over a sequence of “soft” tokens using Langevin dynamics, then maps the continuous sample into discrete, fluent text.
We provide our formulation of constrained text generation (§3.1), present differentiable constraints that can be composed into energy functions (§3.2) along with our discretization method (§3.3), and discuss practical details of C OLD decoding (§3.4).
Figure 2 provides an overview.
3.1 Energy-based Decoding
Constrained text generation aims to produce text samples y that satisfy a set of constraints (usually conditioned on an input x omitted for brevity).
We assume each constraint can be captured by a constraint function
f i ( y ) ∈ R , where higher values of f i mean that the text y better satisfies the constraint.
For example, f i could measure the likelihood of y as a fluency constraint (more in §3.2), while a hard constraint f i amounts to a large negative penalty when y does not satisfy the constraint.
( ) , ) LM ˜ y < t ˜ y t
CrossEntropy −
Softmax p → LM ( ⋅ | ˜ y < t )
LM ˜ y has eight legs x r : never gave up gave up ˜ y : 2-gram hand , sink , soup soup ˜
y : y * : 1-gram hand (cid:1153) 1 (cid:1154) (cid:1153) 2 (cid:1154) (cid:1153) 3 (cid:1154)
y * :
Soft Match Soft Match constraint encourages
.g
to
> typically to encourage y
The set of constraints induces a distribution over text, written in an energy-based form as:
[Equation]
1) where λ i ≥ 0 is the weight of the i th constraint, Z is the normalizing factor.
Here E ( y ) :
= − (cid:80) i λ i
f i ( y ) is the energy function.
This energy-based form is flexible, as one can plug in any constraint functions required for a task of interest.
Generating text under the constraints can then be seen as sampling from the energy-based distribution y ∼ p ( y ) .
One can also draw multiple samples and pick the best if only one sample is needed, as discussed later (§3.4).
As mentioned above, for efficient sampling from p ( y ) we want to use Langevin dynamics, which makes use of the gradient ∇ y E ( y ) .
However, in our case y is a discrete sequence and the gradient ∇ y E ( y ) is not well-defined.
As a result, we perform Langevin dynamics with an energy defined on a sequence of continuous token vectors, described below.
Differentiable decoding with Langevin dynamics.
Instead of defining the energy function on discrete tokens, we define the energy function on a sequence of continuous vectors ˜ y
=
(˜ y 1 , . .
˜ y T ) , which we call a soft sequence.
Each position in the soft sequence is a vector ˜ y t ∈ R V , where V is the vocabulary size, and each element ˜
y t ( v ) ∈ R corresponds to the logit of word v in the vocabulary.
Taking the softmax of ˜ y t yields a distribution over the vocabulary for position t , ˜ p
τt = softmax (˜ y t /τ ) .
As τ → 0 , ˜ p τt becomes a one-hot vector, indicating a discrete token.
By specifying an energy E (˜ y ) on the soft sequence ˜ y , we can use Langevin dynamics to obtain a sample.
Specifically, the sampling is done by forming a Markov chain:
[Equation]
where η > 0 is the step size, and ϵ ( n ) ∈ N (0 , σ ) is the noise at iteration n .
As shown in Welling and
Teh [53], by adding the right amount of noise and annealing the step size, the procedure will converge to samples from the true distribution.
That is, if we let p ( n ) be the distribution
such that ˜ y ( n ) ∼ p ( n ) , then as n → ∞ and σ → 0 , we have p ( n ) → p (˜ y ) :
= exp {− E (˜ y ) } /Z .
That is, the procedure ends up generating samples from the distribution induced by the energy function.
Next, we describe constraint functions defined on the soft sequence ˜ y that can be plugged in as components of the energy function.
Later in §3.3, we describe how to obtain a discrete sequence from a soft sequence sample ˜ y .
3.2 A Collection of C OLD Constraints
C OLD provides a flexible framework for plugging in a wide range of constraint functions for a task of interest.
We describe constraint functions that are useful in various constrained generation problems, such as those we consider in the experiments (§4).
The constraints include language model-based fluency constraints, along with lexical and semantic constraints on the sequence content.
More generally, any differentiable function that outputs a goodness score of (soft) text can be used as a constraint function, as long as it reflects the requirements of the target task.
Soft fluency constraint.
Fluency is a common requirement for generated text.
To promote fluency, we use a constraint which favors soft sequences that receive high probability according to the underlying left-to-right LM p → LM (e.g., GPT2):
[Equation] [Equation]
where p → LM ( ·| ˜ y <t ) means the next-token distribution when providing the neural language model with the preceding soft tokens ˜
y <t (i.e., feeding the weighted average of word embeddings, with the weights being softmax (˜ y t ′ /τ ) for t ′ <
t [20, 45]).
Intuitively, the constraint says that each token distribution in the soft sequence, softmax (˜ y t ) , must match the “reference” distribution p → LM ( ·| ˜ y <t ) predicted by the underlying language model.
The match is measured by the (negative) cross-entropy between the two distributions.
The constraint thus encourages fluency.
In practice, if there is left-side context x for the generation to condition on, we feed x to the LM to form the “reference” distribution p → LM ( ·| ˜ y <t , x ) .
As a result, ˜ y is encouraged to be fluent and coherent with the context x .
We can easily incorporate an additional reverse LM constraint, f ← LM , using a right-to-left LM p ← LM ( ·| ˜ y >t ) , as an additional fluency constraint.
Flexibly leveraging multiple models in this way is infeasible with conventional decoding methods such as beam search or nucleus sampling.
Future-token prediction constraint.
Applications such as text infilling involve future input tokens that remain fixed, but should contribute to updating past positions.
For instance, consider updating the second position of The __ has eight legs .
A sample should be coherent with the tokens x r on the right (i.e., has eight legs ).
To this end, we use a constraint that adjusts soft tokens to maximize the likelihood of input tokens x r ,
[Equation]
where K is the length of x r .
In other words, the constraint adjusts the soft sequence ˜ y
such that the underlying LM predicts the future tokens x r after seeing ˜ y .
N-gram similarity constraint.
Many constrained generation scenarios pose requirements on the wording and expression of generated text sequences.
For instance, lexically constrained generation tasks [18] require certain keywords to be presented in the text samples, while counterfactual reasoning [44] or text editing [15, 31] tasks require the text to retain the essence of a reference sequence.
We formulate these requirements as an n -gram similarity constraint which favors sequences that overlap with a reference y ∗ at the n -gram level,
[Equation]
where ngram-match ( · , · ) is a recent differentiable n -gram matching function [32] which can be seen as a differentiable approximation to the BLEU- n metric [40].
When n = 1 and y ∗ a sequence of keywords, the constraint in effect enforces ˜ y to assign higher values to the keywords ( 1 -grams).
When n is larger and ˜ y ∗ is a reference sequence, the constraint encourages ˜ y to resemble the reference by assigning high values to tokens making up n -grams from y ∗ .
3.3 From Soft to Discrete and Fluent Text
After receiving a soft sequence sample ˜ y from running Langevin dynamics (Eq. 2), we map the soft sequence to a discrete text sequence which we consider as the output of C OLD decoding.
A simple
[Table]
method would be selecting the most-likely token at each position t , y t = arg max v ˜ y t ( v ) .
However, the resulting text can suffer from fluency issues even if the soft fluency constraint (Eq. 3) is used, due to competing constraints that sacrifice fluency.
To overcome this, we use the underlying LM (e.g.,
GPT2-XL)
as a “guardian” for obtaining the discrete sequence.
Specifically, at each position t , we first use the LM to produce the top- k most-likely candidate tokens based on its generation distribution conditioning on preceding tokens, which we denote as V kt .
We then select from the top- k candidates the most likely token based on the soft sample ˜ y :
[Equation]
We refer to this method as “top- k filtering”.
The resulting text tends to be fluent because each token is among the top- k most probable tokens from the LM [12].
In practice, to ease the satisfaction of certain constraints (e.g. n -gram similarity), we expand the candidate set V kt to include constraint tokens (e.g., in the tasks of abductive reasoning §4.1 and lexically constrained decoding §4.3).
Figure 2 illustrates the decoding procedure to get one output from C OLD decoding.
Algorithmsummarizes the algorithm.
Next, we move to practical considerations of applying C OLD .
3.4 Implementation of C OLD Decoding
Sample-and-select.
C OLD decoding allows for drawing multiple text samples from the distribution induced by the energy function E (˜ y ) .
Depending on task requirements, we could either present the set of samples as output, or select one from the set based on some criteria (e.g., different energy terms) and return a single sequence, as in those tasks considered in the experiments (§4).
This “sample-andselect” approach differs from deterministic constrained decoding methods, which optimize only one sequence [e.g., 33, 26], and is used widely in various generation settings [e.g., 28, 11, 4].
Initialization.
We initialize the soft sequence ˜ y by running greedy decoding with the LM p LM to obtain output logits.
In our preliminary experiments, the initialization strategy had limited influence on the generation results.
Noise schedule.
Each iteration of Langevin dynamics adds noise ϵ ( n ) ∼ N (0 , σ ( n ) ) to the gradient (Eq. 2).
We gradually decrease σ ( n ) across iterations, which intuitively transitions the decoding procedure from exploration to optimization.
In our experiments, we typically used the schedule which sets/reduces σ to { 1 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 } at iterations { 0 , 50 , 500 , 1000 , 1500 } , respectively.
Long sequences.
C OLD decoding produces a fixed-length sequence
y =
( y 1 , . . .
, y T ) .
To produce longer sequences, e.g. in cases where y T is not the end of a sentence, we use p LM to produce a continuation of y using greedy decoding.
Experiments
We evaluate C OLD on three constrained generation tasks.
Using C OLD for each task amounts to specifying a set of task-specific constraints (instances of those in §3.2).
Our focus is enabling constrained generation for settings in which fine-tuning is infeasible, through changing the decoding method.
Thus, our experiments (i) use off-the-shelf LMs without fine-tuning, and (ii) compare C OLD primarily against alternative decoding methods.
As our base LM, we use GPT2-XL [46].
4.1 Abductive Reasoning
We study a specific formulation of abductive reasoning [43] as a language generation challenge.
Specifically, given a beginning sentence x l and an ending sentence x r , the abductive language generation ( α NLG) problem [1] consists of generating a bridge sentence y that fills in between the two sentences and forms a coherent full story (see Figure 1 for example).
The task is particularly challenging for conventional monotonic left-to-right LMs (such as GPT-2 and GPT-3) since it requires non-monotonic reasoning that not only conditions on the past context ( x l , on the left), but also the future story ending ( x r , on the right).
4.1.1 The C OLD Solution
C OLD decoding can readily accommodate the abductive reasoning task by simply plugging in appropriate constraints to specify an energy function.
Specifically, the generated text needs to be (1) fluent and consistent with the left context x l , and (2) coherent with the right context x r .
Accordingly, we compose an energy using relevant constraints from §3.2:
[Equation]
That is, we combine (a) a soft fluency constraint (Eq. 3) conditioning on the left sentence x l to enforce fluency and consistency with the left context, and a reverse fluency constraint with a right-to-left
LM conditioning on x r to encourage coherence with the right context; (b) a future-token prediction constraint (Eq. 4) that enforces consistency between the generation y and the story ending x r ; (c) a 1 -gram similarity constraint (Eq. 5) between the generation y and keywords (non-stopwords) in x r (excluding those in x l ), i.e., kw ( x r ) − kw ( x l ) , which intuitively promotes a ‘smooth transition’ between x l , y , and x r .
For the energy function in Eq.(7), we select the constraint weights on the dev set.
Throughout the experiments, we set the number of Langevin dynamics steps to N = 2000 , with a step size η = 0 .(Eq. 2).
We discuss more details of the configurations in the appendix.
Baselines.
We compare with previous decoding approaches for this task.
In particular, we compare with D ELOREAN [45] which outperformed a wide range of supervised and unsupervised methods on the abductive reasoning task in Qin et al.
Following Qin et al.
[45], we also compare with a
L EFT - ONLY method that generates the continuation of x l without considering the right-side x r , i.e., y ∼ p
LM
( y | x l ) .
Evaluation.
We perform both automatic and human evaluation.
We adopt the standard automatic metrics on the task [1] that measure the minimal edit between the generated text and the humanwritten references on the test set, including BLEU [40], ROUGE [30], CIDEr [51], and BERTScore [58].
For the human evaluation, we follow [45] and let crowdworkers from Amazon Mechanical Turk rate the generations on 200 test examples.
Workers were presented a pair of observations ( x l and x r ) and a generated hypothesis y , and asked to rate the coherence of the hypothesis with respect to the observation x l (i.e., x l y ), the observation x r (i.e., yx r ), and both (i.e., x l yx r ), as well as the grammaticality of the hypothesis y itself, on a 5-point Likert scale.
The average ordinal Krippendorff alpha ( 0 ≤ α ≤ 1 ) [25] is 0.36, indicating a fair inner-annotator agreement.
4.1.2 Results
Table 1 shows the evaluation results on the abductive reasoning task.
Under automatic evaluation (the left panel), C OLD consistently outperforms the previous best unsupervised decoding algorithm
D ELOREAN , as well as the L EFT - ONLY method, in terms of both the lexical overlap metrics (BLEU,
ROUGE and CIDEr) and semantic similarity metric BERTScore.
The human evaluation (the right panel) provide more fine-grained insights.
C OLD achieves the best overall coherence , meaning that the generated y from C OLD fits best with both the left-side context x right x r compared to the other methods.
In contrast, D ELOREAN excels only in terms of the left-side coherence (with x l ), with inferior right-coherence (with x r ).
We speculate this is because of D ELOREAN ’s complex interleaving of forward and backward decoding passes that make it difficult to balance the left- and right-coherence constraints.
In terms of grammaticality, unsurprisingly, L EFT - ONLY obtains the best score as it ignores any other constraints (and fails this task with low coherence scores).
More importantly, C OLD achieves a high grammaticality score along with its high coherence, substantially improving over D ELOREAN .
Example generations in Appendix Table 7 show how
C OLD can reason with the right-hand context (e.g. ‘no heels’), while D ELOREAN ’s generations are contradictory (‘red shoes’ vs. ‘white pair’) or equivalent to those from L EFT - ONLY .
[Table]
As a trivial method, L EFT - ONLY is coherent but fails on minimal-edit.
C OLD is superior to D ELOREAN in terms of most metrics, including human evaluation.
[Table]
4.2 Counterfactual Story Rewriting
Next, we consider counterfactual story rewriting [44].
Given a story context x l with ending x r , the task is to generate a new story ending y that is (i) similar to the original ending x r , yet (ii) consistent with a new story context x ′ l
(see Figure 1 for example).
The task is challenging as it requires capturing the aspects of future events that are invariant under the new (counterfactual) context, while only making necessary edits for coherence.
4.2.1 The C OLD Solution
To tackle this task, we use C OLD with an energy composed of constraint functions that promote coherence with the new context x ′ l , and minimal edits to the original ending x r :
[Equation]
These constraints combine: (a) a soft fluency constraint (Eq. 3) conditioned on x ′ l to promote coherence between the generation y and the new (counterfactual) context x ′ l ; a reverse LM constraint to improve fluency; (b) a n -gram similarity constraint (Eq. 5, n = { 2 , 3 } ) to encourage generating an ending ˜ y
that is close to the original ending x r .
We largely follow the configurations in §4.1 with some exceptions described in the appendix.
Baselines.
Similar to the setup in §4.1, we compare with D E L OREAN [45], a recent state-of-the-art decoding algorithm.
As a reference, we also report the performance of a trivial solution, L EFT - ONLY , that generates a continuation of x ′ l without considering the minimal edit constraint with the original ending x r .
Thus the method is expected to generate a coherent ending which however does not necessarily resemble the original ending.
Finally, we compare with Mix-and-Match [36], a recent energy-based decoding method with discrete MCMC sampling, using BERT-base and BERT-Large.
Evaluation.
We use the benchmark dataset T IME T RAVEL [44].
The original data contains three sentences in a story ending.
Due to computation constraints, we use the first sentence as the original ending and generate a new single-sentence ending accordingly.
Following [44, 45] we conduct both automatic and human evaluation.
For automatic evaluation, we measure BERTScore [58], and
Minimal Edit which computes the overlap of text edits (insertion, deletion, replacement, etc.)
[49] needed to produce the gold ending y ∗ and the generated ending y , starting from the original ending x r .
We do not use other common metrics such as BLEU since they were shown to be ineffective [44].
For human evaluation, each crowdworker is presented with the original story ( x l , x r ), the counterfactual condition x ′ l , and the generated ending y , and the worker is asked to rate (1) the coherence of ˜ y with respect to x ′ l and (2) the extent to which the generated ending y preserves the details of the original ending x r (“minimal edit”), on a 3-point Likert scale for 200 test examples.
The average ordinal Krippendorff alpha is 0.52, indicating a moderate inner-annotator agreement.
We exclude Mix-and-Match from human evaluation given the significant performance gap in automated evaluation.
4.2.2 Results
Table 2 shows the results of automatic and human evaluation in terms of both minimal-edit and coherence.
As expected, the reference method L EFT - ONLY that completely ignores the minimal edit constraint can easily generate a new ending that is coherent with the new context x ′ l .
Compared to the baseline approach D ELOREAN , our method C OLD achieves overall superior performance, with substantially improved coherence score and comparable minimal-edit score by human evaluation.
Mix-and-Match, based on discrete MCMC sampling, performs poorly.
Intuitively, its discrete sampling tends to get stuck in a mode of the target distribution (i.e., the region surrounding the original story ending), and struggles to explore further to find samples of interest.
C OLD ’s gradientbased sampling with continuous approximation leads to more efficient and effective exploration and mixing, as evidenced by samples that better meet the task requirements.
See Appendix for examples.
4.3 Lexically Constrained Decoding
Next, we use C OLD for lexically constrained decoding.
Given a set of words W , the task aims to generate a coherent sentence that contains these words (Figure 1).
The task is challenging as it requires proper planning to coherently include the constraint words.
4.3.1 The C OLD Solution
We specify an energy function of the following form:
[Equation]
Specifically, this energy function incorporates: (a) a soft fluency constraint (Eq. 3) and a reverse LM fluency constraint as in the previous tasks; (b) a 1 -gram similarity constraint (Eq. 5) between the generation ˜ y and the given words W ; (c) a future-token prediction constraint, where we concatenate the constraint words (in an arbitrary order), denoted as c ( W ) , and use it as the right-side content x r in Eq.
Again we use similar configurations as in §4.1.
More details can be found in appendix.
Baselines.
We compare with a recent state-of-the-art method N EURO L OGIC [33], a beam-search variant specifically designed for lexically constrained generation which outperformed many supervised and unsupervised approaches in Lu et al.
We also report the results of TSMH [57] as another recent baseline which uses Monte-Carlo Tree Search [5].
Evaluation.
We use the set of constraint words from the C OMMON G EN corpus [29], but adopt the canonical setting that the generated text must contain the exact constraint words (e.g., write ) instead of their variants (e.g., wrote ) [18, 47].
Following previous works [18, 47, 57]
[Table]
we report a measure of constraint words coverage as well as language fluency by evaluating the perplexity of the text .
We also ask crowdworkers to rate the text fluency on a 3-point Likert scale on 200 test examples.
The average ordinal Krippendorff alpha is 0.29, indicating a fair inner-annotator agreement
[Table]
4.3.2 Results
Table 3 shows the evaluation results for the lexically constrained decoding task.
C OLD , a general constrained decoding method, is comparable to the state-of-the-art method N EURO L OGIC designed specifically for dealing with lexical constraints.
In particular, C OLD achieves a higher coverage of given keywords, at the expense of generating slightly less fluent language.
C OLD is also substantially better than lexically constrained decoding method TSMH in terms of both coverage and fluency
[Table]
4.4 Additional Analysis
[Table]
Ablation studies.
We ablate two important ingredients of our approach, namely the constraints and the top- k filtering.
Due to space limit, we report the results of constraints and defer the results of top- k filtering to the appendix.
Table 5 shows the human evaluation results for ablations of the constraints used on the abductive reasoning task (Eq. 7).
The n -gram similarity constraint f sim provides the largest contribution to the overall coherence.
The reverse
LM fluency constraint f ← LM also to some extent helps with the right-side coherence by conditioning on the right-side content x r .
Removing the future-token prediction constraint similarly causes inferior scores in terms of right-side and overall coherence, as expected.
Removing the individual constraints leads to better grammaticality due to less competition among different constraints, at the cost of coherence.
Our uniform treatment of all constraints as energy terms makes it straightforward to balance the different constraints by controlling the constraint weights.
[Table]
Efficiency of C OLD .
We report the average runtime of generating one sample on the Counterfactual Story Rewriting data.
The table below shows the results (on an NVIDIA Quadro
GV
GPU, batch size=32).
We compare with
Mix-and-Match [36], a recent energy-based decoding method with discrete MCMC sampling (Metropolis-Hastings, in particular).
C OLD , which uses gradient-based sampling, is faster than the gradient-free Mix-and-Match:
C OLD is 30% faster with base LMs of similar sizes (GPT2-M and BERTLarge), and has roughly the same time cost when using a much larger LM (GPT2-XL).
Related Work
Previous works proposed beam search variants for lexically constrained decoding [18, 42, 33] which enforce constraints in a discrete space.
Recent works consider constraint satisfaction by adjusting vocabulary distributions using an additional discriminator or LM [6, 24, 56].
Differing from those approaches that determine the generation token by token auto-regressively, Qin et al.
[45] optimize the whole (soft) token sequence via gradient propagation, which facilitates sequence-level semantic constraints (e.g., right-coherence, minimal-edits).
C OLD also samples complete sequences, while offering a principled and unified formulation based on energy-based modeling.
Kumar et al.
[26] extend [17] by imposing constraints with a Lagrangian method and optimizing for a single output with gradient descent.
In contrast, our approach based on energy-based sampling (§3.1) allows for generating samples for other utilities (e.g., rank-and-select §3.4, estimating expectations).
We also introduce components for more fluent generations such as the novel discretization procedure.
Also, on the empirical side, we explore a different class of problems and tackle them in the absence of labeled data.
The recent CGMH [35] and TSMH [57], followed by [36, 14], perform constrained decoding with extended Gibbs sampling or Metropolis-Hastings sampling in the discrete text space.
Our energy-based formulation with gradient-based Langevin dynamics sampling produces substantially better results than the discrete TSMH (§4.3).
Sha [47] uses gradient information to guide generation, which, however, is specifically designed for lexically constrained generation.
Energy-based models (EBMs) have been used for incorporating additional information to train text generation models [7, 23, 41, 21].
In contrast, we focus on the constrained decoding ( inference ) that can be directly applied to pretrained LMs without fine-tuning.
Langevin dynamics is widely used on
EBMs of modalities with continuous values, like images [48, 9, 59], 3D shapes [55], latent features [39], and audio sequences [22].
To our knowledge, we are the first to apply Langevin dynamics for (constrained) discrete text generation (with a continuous approximation) for efficient sampling.
Conclusion
We introduce C OLD decoding, an energy-based constrained text generation framework that can express various soft/hard constraints through an energy function, and sample using Langevin dynamics.
C OLD can be applied directly to off-the-shelf LMs without task-specific fine-tuning.
We showcase its flexibility and strong performance on three distinct applications of constrained text generation.
Acknowledgements
This work was funded in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 401233309), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), the Allen Institute for AI, and Microsoft Research PhD Fellowship.
We thank the XLab research group, and our anonymous reviewers for their feedback on this work.
We also acknowledge the Beaker team ( https://beaker.org ) for their support with experiments.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] , see the Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] .
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Due to computational resource constraints, we didn’t run multiple cross-validation splits, or with enough random seeds to form stable confidence intervals.
However, we do a thorough set of ablations across model configurations.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Details are in appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] : we don’t introduce new datasets, and refer readers to the original releases in case license information for those works changes.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us-
ing/curating?
[Yes] All data we experiment with is public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable infor- mation or offensive content?
[No] We aren’t releasing new data, and existing corpora, to our knowledge and in our experience, do not contain personally identifying information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes] See Appendix.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
Crowdworking studies involving no personal disclosures of standard NLP corpora are not required by our IRB to be reviewed by them.
Specifically: i. We do not collect personal information.
Information gathered is strictly limited to general surveys about the quality of generated text.
ii.
We take precaution to anonymize Mechanical Turk WorkerIDs in a manner that the identity of the human subjects cannot be readily ascertained (directly or indirectly).
iii.
We do not record or include any interpersonal communication or contact between investiga- tion and subject.
Additional locality-specific details of our IRB withheld to preserve anonymity, but will be made public upon de-anonymization.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[Yes] , our pay is always over $15 USD per hour on average (and sometimes more, see Appendix.
A Ethical Considerations
Automatic text generation, though powerful in generating fluent human-like language, could be potentially used for malicious purposes, such as generating toxic, biased, offensive, or fake information.
We hope that our research, as a method to control language model generations by plugging in constraints, can provide a way for steering and harnessing the LMs to alleviate those ethical issues.
B Experimental Configurations
Configurations of Abductive Reasoning.
For the energy function in Eq.(7), we select the constraint weights on the dev set.
The overall weight of the fluency constraints is set to 0.5, wherein the f → LM and f ← LM constraints are balanced with a 6:4 ratio, leading to λ lra = 0 .
3 and λ rla = 0 .
The remaining weight 0.5 is assigned to the constraints (b) and (c), with a ratio of 1:0.05, leading to λ lrc = 0 .
48 and λ rla = 0 . 02 .
Throughout the experiments, we set the number of Langevin dynamics steps to N = 2000 , with a step size η = 0 .
1 (Eq. 2).
The text decoded by C OLD is set to have length 10 and is completed by the base LM as described in §3.4.
We set the k = 2 for top- k filtering.
For each ( x l , x r ), we generate 16 samples and pick the best one by first ranking by the perplexity of the joint sequence x l yx r for overall coherence, and then from the top 5 candidates selecting the best one in terms of the perplexity of yx r for enhanced coherence with the right context.
Configurations of Counterfactual Story Rewriting.
The constraint weights in the energy function in Eq.
(8) are selected on the dev set.
The weights of the constraints (a) and (b) are set to λ lra + λ rla = 0 .
8 and λ b = 0 .
2 , respectively.
For the LM and reverse LM fluency constraints in (a), we use a ratio of 8:2, leading to λ lra = 0 .and
λ rla = 0 .
We largely follow the algorithm configurations in §4.1 except that the text length is set to 20, k = 5 for top- k
Computing.
All experiments were conducted using a server with 8 NVIDIA V100 GPUs.
C Human Evaluation Details C.1 Instructions of Human Evaluation
We conduct human evaluation for 3 tasks:
1)Lexically Constrained Generation 2)Abductive
Reasoning 3)Coun- terfactual Reasoning.
We sampled 200 prompts randomly from the corpus for each human evaluation.
We shuffle
HITs to eliminate systematic bias of rater availability by time.
Figures show the screenshot of instructions for our human evaluation.
C.2 Human Evaluation Payment
Mean hourly pay was determined using a javascript timing tool to be $15/hr.
D Ablation Study: Top-k Filtering
[Table]
We investigate the role of top- k filtering mechanism (§3.3).
Specifically, we investigate its effect on the output performance for different k values in Table 6.
We can see that the grammar score tends to decrease as k increases.
This is expected since a larger k indicates more flexibility for the generation to satisfy other constraints often expense of fluency.
coherence shows a similar relationship with k value since it is also enforced
E Generated Samples
Tables 7, 8, and 9 show generated samples for the abductive reasoning, counterfactual reasoning, and lexically constrained decoding tasks, respectively.
Begin.
x l
I bought a great pair of red shoe at the shoe store.
End.
x r
I ended up getting a white pair with no heels.
L EFT - ONLY
I was going to wear them to the beach, but I didn’t want to be the only one.
D ELOREAN
I was going to buy a pair of black shoes, but I decided to go with red shoes because I like red shoes.
C OLD
I was going to buy heels but they were out of stock.
Begin.
x l
Arnold was scared of cats.
End.
x r
Arnold dumped his girlfriend.
L EFT - ONLY
He was afraid of the dark.
D ELOREAN
He was afraid of the dark.
C OLD
He had girlfriend who was a cat lover.
Orig.
context x l
Jon decided to go to the pawn store.
He found a bornite-coated chalcopyrite crystal.
Orig.
ending x r
He bought it for three thousand dollars.
Counterfactual x ′ l
He sold some antiques he had found.
L EFT - ONLY
He bought a few books.
D ELOREAN
He bought it for three thousand dollars.
C OLD
He bought a thousand dollars’ worth of gold.
Orig.
context x l
Peyton and Tom played football often.
Tom always won for many Year’s.
Orig.
ending x r
Peyton never gave up and kept practicing.
Counterfactual x ′ l
Peyton always won for many years.
[Table]
ONLY
Tom was a great quarterback.
[Table]
Tom was a great quarterback.
[Table]
Tom never gave up and never gave in.
[Table]
They wash with their hands they wash at the sinks soaps they wash
N EUROLOGIC
I hand wash my clothes in the sink , soap and water.
C
[Table]
The sink soap is a hand wash soap made from natural ingredients.
TSMH
I creamed my bare legs and put .
N EUROLOGIC
I put shave cream on my leg .
C
[Table]
The first time I ever put a leg in shave cream was when I was a kid.