Diffusion-LM Improves Controllable Text Generation
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Controlling the behavior of language models (LMs) without re-training is a majoropen problem in natural language generation.
While recent works have demonstrated successes on controlling simple sentence attributes (e.g., sentiment), therehas been little progress on complex, ﬁne-grained controls (e.g., syntactic structure).
To address this challenge, we develop a new non-autoregressive language modelbased on continuous diffusions that we call Diffusion-LM.
Building upon the recentsuccesses of diffusion models in continuous domains, Diffusion-LM iterativelydenoises a sequence of Gaussian vectors into word vectors, yielding a sequence ofintermediate latent variables.
To control its generation, we iteratively perform gradient updates on these intermediate variables.
Diffusion-LM has three properties thatenable complex, ﬁne-grained controllable text generation: the continuous nature ofdiffusion models enables gradient-based control; the non-autoregressive generationorder enables more complex, global controls; and incremental denoising inducesa coarse-to-ﬁne hierarchy, which facilitates control at multiple granularities.
Wedemonstrate successful control of Diffusion-LM for six challenging ﬁne-grainedcontrol tasks, signiﬁcantly outperforming prior work.
Introduction
Large autoregressive language models (LMs) are capable of generating high quality text [30, 3, 5, 40],but in order to reliably deploy these LMs in real world applications, the text generation process needsto be controllable : we need to generate text that satisﬁes desired requirements (e g. topic, syntacticstructure).
A natural approach for controlling a LM would be to ﬁne-tune the LM using superviseddata of the form (control, text) [15].
However, updating the LM parameters for each control taskcan be expensive and does not allow for compositions of multiple controls (e g. generate text thatis both positive sentiment and non-toxic).
This motivates light-weight and modular plug-and-playapproaches [6] that keep the LM frozen and steer the generation process using an external predictorthat measures how well the generated text satisﬁes the control.
But even then, steering a frozenautoregressive LM has been shown to be difﬁcult, and existing successes have been limited to simple,attribute-level controls (e.g., sentiment or topic) [6, 20, 39].
In order to broaden the set of viable controls, we propose Diffusion-LM , a new language modelbased on continuous diffusions.
Diffusion-LM starts with a sequence of Gaussian noise vectorsand incrementally denoises them into vectors corresponding to words, as shown in Figure 1.
Thesegradual denoising steps produce a coarse-to-ﬁne hierarchy of continuous latent representations.
Diffusion-LM enables new forms of complex, ﬁne-grained control tasks that are not currently possibleusing autoregressive LMs.
We highlight three desirable properties of Diffusion-LM that may enablethese capabilities.
First, Diffusion-LM directly generates continuous latent representations, which canbe updated and controlled using gradients derived from external predictors.
Second, diffusion LM is anon-autoregressive model that generates all tokens in parallel.
This allows it to incorporate complex,global controls.
As a bonus, it handles inﬁlling at decoding time without additional predictors or specialized techniques, unlike autoregressive LMs which require expensive search or marginalizationsteps [21, 38, 28].
Finally, Diffusion-LM induces a coarse-to-ﬁne hierarchy of continuous latentrepresentations, which enable controls that operate on the entire sequence (e.g. sentiment or length)as well as on individual words (e.g. parts of speech).
Continuous diffusion models have been extremely successful in vision and audio domains [11, 19,31, 7, 4], but they have not been applied to text because of the inherently discrete nature of text(§3).
Adapting this class of models to text requires several modiﬁcations to the diffusion trainingobjective and decoding procedure (§4).
We control Diffusion-LM using a gradient-based method, asshown in Figure 1.
This method enables us to steer the text generation process towards outputs thatsatisfy given structural and semantic control targets.
It iteratively performs gradient updates on thecontinuous latent variables of Diffusion-LM to balance ﬂuency and control satisfaction (§4.3).
To demonstrate control of Diffusion-LM, we consider a variety of control targets ranging fromsimple attributes (e.g., sentence length) to complex structures (e.g., parse tree) and semantic content.
Our method almost doubles the success rate of previous plug-and-play methods and matches oroutperforms the ﬁne-tuning oracle on all these predictor-guided control tasks (§6.1).
In addition tothese individual control tasks, we show that we can successfully compose multiple predictor-guidedcontrols to generate sentences with both desired semantic content and syntactic structure (§6.2).
Finally, we also consider span-anchored controls, such as length control and inﬁlling.
These tasks arepredictor free, and our Diffusion-LM signiﬁcantly outperforms prior plug-and-play methods and ison-par with an autoregressive LM trained from scratch for the inﬁlling task (§6.3).
Related Work
Diffusion Models for Text.
Diffusion models [35] have demonstrated great success in continuousdata domains [11, 25, 19, 23], producing images and audio that have state-of-the-art sample quality.
To handle discrete data, past works have studied text diffusion models on discrete state spaces, whichdeﬁnes a corruption process on discrete data (e.g., each token has some probability to be corrupted toan absorbing or random token)
In this paper, we focus on continuous diffusion modelsfor text and to the best of our knowledge, our work is the ﬁrst to explore this setting.
In contrastto discrete diffusion LMs, our continuous diffusion LMs induce continuous latent representations,which enables efﬁcient gradient-based methods for controllable generation.
Autoregressive and Non-autoregressive LMs.
Most large pre-trained LMs are left-to-right autoregressive (e.g., GPT-3 [3], PaLM [5]).
The ﬁxed generation order limits the models’ ﬂexibility inmany controllable generation settings, especially those that impose controls on the right contexts.
Since autoregressive LMs cannot directly condition on right contexts, prior works have developedspecialized training and decoding techniques for these tasks [34, 8, 28].
For example, Qin et al.
[29]is a decoding method that relaxes the discrete LM outputs to continuous variables and backpropagatesgradient information from the right context.
Diffusion-LM can condition on arbitrary predictors thatlook at complex, global properties of the sentence.
There are other non-autoregressive LMs that havebeen developed for machine translation and speech-to-text tasks [10, 33].
However these methodsare specialized for speech and translation settings, where the entropy over valid outputs is low, andwhether they work for language modeling remains an open problem. We leave detailed discussions to
Plug-and-Play Controllable Generation.
Controllable text generation is the task of decodingfrom a conditional distribution p ( w | c ) , where w is the text sequence, and c is the control constraint.
Plug-and-play methods leverage Bayes rule to control the output of an unconditional LM p ( w ) atdecoding time: p ( w | c ) / p ( w ) · p ( c | w ) where p ( w ) is the frozen LM, and p ( c | w ) is a predictorprobability of whether a sequence fulﬁlls the goal of the control task.
There are several plug-and-playapproaches based on autoregressive LMs: FUDGE [39] reweights the LM prediction at each tokenwith an estimate of p ( c | w ) for the partial sequence; GeDi [20] and DExperts [22] reweight the LMprediction at each token with a smaller LM ﬁnetuned/trained for the control task.
The closest work to ours is PPLM [6], which runs gradient ascent on an autoregressive LM’shidden activations to steer the next token towards higher p ( w ) and p ( c | w ) .
Because PPLM isbased on autoregressive LMs, it can only generate left-to-right, so PPLM cannot repair its pasterrors.
Despite their success on attribute (e.g., topic) controls, we will show these plug-and-playmethods for autoregressive LMs fail on more complex control tasks such as controlling syntacticstructure and semantic content in §6.1.
We demonstrate that Diffusion-LM is capable of plug-and-playcontrollable generation by applying predictor-guided gradient updates to the continuous sequence oflatent variables induced by the Diffusion-LM.
Problem Statement and Background
We aim to apply continuous diffusion models to discrete text and begin by deﬁning the problemsettings for controllable generation and diffusion modeling.
3.1 Generative Models and Controllable Generation for Text
Consider a language modeling task, where w =
[ w 1 · · · w n ] is a sequence of discrete words drawnfrom an unknown data distribution.
A language model p lm is trained to emulate this data distributionby maximizing the data likelihood: E w [log p lm ( w )] .
In the controllable generation setting, we havean additional control variable c denoting a feature of interest for w .
For syntactic control, c may bethe syntax tree of w (Figure 1).
For sentiment control, c may be the sentiment label on w .
The goalof controllable generation is to approximate samples from the conditional distribution p ( w | c ) .
Controllable generation can be treated as a standard language modeling task using paired data ( w , c ) .
However this approach has two drawbacks: ﬁrst, tuning p lm from scratch can be computationallyexpensive; second, there may be a fundamental asymmetry in data collection, where it is substantiallyeasier to collect un-annotated samples w than paired samples ( w , c ) .
The plug-and play approachseeks to address both concerns by training a large p lm on w and then steering this model using alightweight classiﬁer trained on paired data p ( c | w ) .
This classiﬁer can guide text generation via
Bayes rule as p ( w | c ) / p lm ( w ) · p ( c | w ) , where p lm ( w ) encourages w to be ﬂuent, and the 112 p ( c | w ) encourages w to fulﬁll the constraints.
3.2 Diffusion Models for Continuous Domains
A diffusion model [11, 25] is a latent variable model that models the data x 0 ⇠ p data as a Markov chainx T . . .
x 0 , where x T is a Gaussian, and x t  1 | x t is a de-noising step that gradually transforms noisyintermediate variables into the observed data distribution (Figure 2).
This sequence of continuouslatent variables x 1: T is deﬁned by a forward process that incrementally adds Gaussian noise to datax 0 until, at diffusion step T , samples x T are approximately Gaussian.
Each transition x t  1 ! x tis parametrized by q ( x t | x t  1 ) =
N ( x t ; p 1   t x t  1 ,  t I ) , where the hyperparameter  t is the 120 amount of noise added at diffusion step t .
The diffusion model generates samples by reversing this process: it incrementally denoises thesequence of latent variables x T :1 to approximate samples from the target distribution.
Each denoisingtransition x t !
x t  1 is parametrized by the model p ✓ ( x t  1 | x t ) =
N ( x t  1 ; µ ✓ ( x t , t ) , ⌃ ✓ ( x t , t )) .
The diffusion model is trained to maximize the marginal likelihood of the data E
x 0 ⇠ p data log p ✓ ( x 0 ) ,and the canonical objective is the variational lower bound of log
p
✓ ( x 0 ) [35]:
[Equation]
However, this objective can be unstable and require many optimization tricks to stabilize [25].
Tocircumvent this issue, Ho et al.
[11] devised a simple surrogate objective that expands and reweights addition to the original diffusion models [11], we add a Markov transition between x 0 and w , and propose the embedding §4.1 and rounding §4.2 techniques.
each KL-divergence term in L vlb to obtain a mean-squared error loss which we will refer to as
[Equation]
where ˆ µ ( x t , x 0 ) is the mean of the posterior q ( x t  1 | x 0 , x t ) .
While L simple is no longer a valid lowerbound, prior work has found that it empirically made training more stable and improved samplequality 1 .
We will make use of similar simpliﬁcations in Diffusion-LM to stabilize training andimprove sample quality (§4.1).
Diffusion-LM: Continuous Diffusion Language Modeling
Constructing Diffusion-LM requires several modiﬁcations to the standard diffusion model.
First, wemust deﬁne an embedding function that maps discrete text into a continuous space.
To address this,we propose an end-to-end training objective for learning embeddings (§4.1).
Second, we require arounding method to map vectors in embedding space back to words.
To address this, we proposetraining and decoding time methods to facilitate rounding (§4.2).
The two improvements make itpossible to reliably train Diffusion-LMs, and we describe how to perform plug-and-play controllablegeneration on these models using classiﬁer guidance (§4.3).
4.1 End-to-end Training
To apply a continuous diffusion model to discrete text, wedeﬁne an embedding function E MB
w i ) that maps each wordto a vector in R d .
We deﬁne the embedding of a sequence w oflength n to be: E MB ( w ) =
[Equation]
We propose a modiﬁcation of the diffusion model training objective (Equation 1) that jointly learns the diffusion model’sparameters and word embeddings.
In preliminary experiments,we explored random Gaussian embeddings, as well as pretrained word embeddings [27, 30].
We found that these ﬁxedembeddings are suboptimal for Diffusion-LM compared to endto-end training 2 .
As shown in Figure 2, our approach adds a Markov transition from discrete words w to x 0 in theforward process, parametrized by q  ( x 0 | w ) =
N ( E MB ( w ) ,  0 I ) .
In the reverse process, we add atrainable rounding step, parametrized by p ✓ ( w | x 0 ) =
Q ni
[Equation]
p ✓ ( w i | x i ) , where p ✓ ( w i | x i ) is a 156 softmax distribution.
The training objectives introduced in §3 now becomes
[Equation]
We derive L e2esimple ( w ) from L e2evlb ( w ) following the simpliﬁcation in §3.2 and our derivation detailsare shown in Appendix D. Since we are training the embedding function, q  now contains trainable
While trainable embeddings perform best on control and generation tasks, we found that ﬁxed embeddings onto the vocabulary simplex were helpful when optimizing for held-out perplexity.
We leave discussion of this approach and perplexity results to Appendix C as the focus of this work is generation quality and not perplexity.
parameters and we use the reparametrization trick [32, 17] to backpropagate through this samplingstep.
Empirically, we ﬁnd the learned embeddings cluster meaningfully: words with the samepart-of-speech tags (syntactic role) tend to be clustered, as shown in Figure 3.
4.2 Reducing Rounding Errors
The major challenge in applying diffusion models to text is mapping between discrete text ( w ) andcontinuous latent variables x 0 .
The learned embeddings in §4.2 deﬁne an embedding that mapsdiscrete texts to our continuous space.
We now describe the inverse process of rounding a continuouslatent variable x 0 into discrete text.
Ideally, the denoising process itself should learn that the distribution over x 0 is nearly a mixtureof low-variance distributions (where each mixture component represents a word).
In this case, therounding step would be unambiguous.
However, we found that diffusion models do not seem to learnthis mixture structure of x 0 .
One explanation for this phenomenon is that our objective L simple puts insufﬁcient emphasis on modeling the mixture structure of x 0 .
Recall that we deﬁned L simple =
[Equation]
173 where our model predicts individual denoising steps x t  1 | x t .
In this objective, the constraint that 174 x 0 is (nearly) a mixture of Dirac delta distribution will only appear in the terms with t near zero, andwe found that this parametrization required careful tuning to force the objective to emphasize thoseterms (see Appendix B).
Our approach is to re-parametrize L simple to force the model to explicitly model x 0 in every term of theobjective.
Speciﬁcally, we select an alternative parametrization L simple =
[Equation]
where our model x ✓ ( x t , t ) predicts x 0 directly 3 .
This forces the neural network to predict x 0 inevery term and we found that models trained with this objective quickly learn the mixture structure.
We described how re-parametrization can be helpful for model training, but we also found that thesame idea could be used when generating from the model in a technique that we call the clampingtrick.
In the standard generation approach, the model denoises x t to x t  1 by ﬁrst computingan estimate of x 0 via x ✓ ( x t , t ) and then sampling x t  1 conditioned on this estimate: x t  1 =p ¯ ↵ x ✓ ( x t , t ) + p 1  ¯ ↵✏ , where ¯ ↵ t =
Q ts =
(1   s ) and ✏ ⇠ N (0 , I ) 4 .
In the clamping trick, themodel additionally maps the predicted vector x ✓ ( x t , t ) to its nearest word embedding sequence.
Now,the sampling step becomes x t  1 =
p ¯ ↵ · Clamp( x ✓ ( x t , t ))
+ p 1  ¯ ↵✏ .
The clamping trick forcesthe predicted embedding to commit to a word for intermediate diffusion steps, making the vectorpredictions more precise and reducing rounding errors.
4.3 Controllable Text Generation
With the above improvements, we are able to train Diffusion-LMs that generate ﬂuent text.
Wenow describe a procedure that enables plug-and-play control on this Diffusion-LM.
Our approach tocontrol is inspired by the Bayesian formulation in §3.1, but instead of performing control directlyon the discrete text, we perform control on the sequence of continuous latents x 0: T deﬁned by
Diffusion-LM, and apply the rounding step to convert these latents into text.
Controlling x 0: T is equivalent to decoding from the posterior p ( x 0: T | c ) =
Q
Tt =1 p ( x t  1 | x t , c ) , and 197 we decompose this joint inference problem to a sequence of control problems at each diffusion step:p ( x t  1 | x t , c ) / p ( x t  1 | x t ) · p ( c | x t  1 , x t ) .
We further simplify p ( c | x t  1 , x t ) =
p ( c | x t  1 ) 199 via conditional independence assumptions from prior work on controlling diffusions [36], leading to:
[Equation]
where both log p ( x t  1 | x t ) and log p ( c | x t  1 ) are differentiable: the ﬁrst term is parametrized by 201 Diffusion-LM, and the second term is parametrized by a neural network classiﬁer.
[Equation]
Similar to work in the image setting [7, 36], we train the classiﬁer on the diffusion latent variablesand run gradient updates on the latent space x t  1 to steer it towards fulﬁlling the control.
To improveperformance on text and speed up decoding, we introduce two key modiﬁcations.
To improve decoding speed, we downsample the diffusion steps from 2000 to 200.
For each downsampled time step, we run 3 steps of the Adagrad 5 [9] update on  log p ( x t  1 | x t )+log p ( c | x t  1 ) ,where  is a hyperparameter that trades off ﬂuency (the ﬁrst term) and control (the second term).
Whileexisting controllable generation methods for diffusions do not include the  p ( x t  1 | x t ) term in theobjective, we found this term to be instrumental for generating ﬂuent text.
The resulting controllablegeneration process can be viewed as a stochastic decoding method that balances maximizing andsampling p ( x t  1 | x t , c ) , much like popular text generation techniques like nucleus sampling [12].
Experimental Setup
5.1 Datasets and Hyperparameters
We train Diffusion-LM on two datasets: E2E [26] and ROCStories [24].
The E2E dataset consistsof 50K restaurant reviews labeled by 8 ﬁelds including food type, price, and customer rating.
The
ROCStories dataset consists of 98K ﬁve-sentence stories, capturing a rich set of causal and temporalcommonsense relations between daily events.
This dataset is more challenging to model than E2E,because the stories contain a larger vocabulary of 11K words and more diverse semantic content.
Our diffusion model consists of 80 M parameters, the noise schedule (  t ) to be square-root, with asequence length of 64 and 2 k diffusion steps.
We treat the embedding dimension as a hyperparameter,setting d = 16 for E2E and d = 128 for ROCStories.
See the appendix for all other training details.
At decoding time, we downsample to 200 diffusion steps for E2E and maintain 2000 steps for
ROCStories.
Admittedly, decoding Diffusion-LM is still slower than decoding autoregressive LMs.
5.2 Control tasks
We consider 6 control tasks: the ﬁrst 4 tasks rely on an external predictor, and the last 2 tasks arepredictor free 6 .
For each control task (e.g. semantic content), we sample 200 control targets c (e.g.,rating=5 star) from the validation splits, and we generate 50 samples for each control target.
Toevaluate the ﬂuency of the generated text, we feed them to a teacher LM (i.e., a carefully ﬁne-tuned
GPT-2 model) and report the perplexity of generated text under the teacher LM.
We call this metriclm-score (denoted as lm): a lower lm-score indicates better sample quality.
We deﬁne success metricsfor each control task.
Semantic Content.
Given a ﬁeld (e.g., rating) and value (e.g., 5 star), we aim to generate a sentencethat covers ﬁeld=value, and report success rate by exact match of ‘value’.
Parts-of-speech.
Given a sequence of parts-of-speech (POS) tags (e.g., Pronoun Verb Determiner
Noun ), we aim to generate a sequence of words of the same length whose POS tags (under an oracle
POS tagger) match the target (e.g., I ate an apple ).
We quantify success via word-level exact match.
Syntax Tree.
Given a target syntax tree (see Figure 1), we aim to generate text with a matchingsyntax tree (under an off-the-shelf parser [18]), quantifying success with F1 scores.
Syntax Span.
Instead of controlling the entire tree, our goal is to generate text whose oracleconstituent from positions i to j matches a target label (e.g. prepositional phrase).
We quantifysuccess via the fraction of spans that match exactly.
Length.
Given a target length 10 , . . . , 40 , our goal is to generate a sequence with a length within ± 2of the target.
In the case of Diffusion-LM, we treat this as a classiﬁer-free control task.
Inﬁlling.
Given a left context ( O 1 ) and a right context ( O 2 ) from the aNLG dataset [2], and the goalis to generate a sentence that logically connects O 1 and O 2 .
For evaluation, we report both automaticand human evaluation from the Genie leaderboard [16].
[Table]
For the ﬁrst 5 control tasks, we compare our method with PPLM, FUDGE, and a ﬁne-tuningoracle.
Both PPLM and FUDGE are plug-and-play controllable generation approaches based on anautoregressive LM, which we train from scratch using the GPT-2 small [30].
PPLM[6].
This method runs gradient ascent on the LM activations to increase the classiﬁer probabilities and language model probabilities, and has been successful on simple attribute control.
Weapply PPLM to control semantic content, but not the remaining 4 tasks which require positionalinformation, as PPLM’s classiﬁer lacks positional information.
FUDGE[39].
For each control task, FUDGE requires a future discriminator that takes in a preﬁxsequence and predicts whether the complete sequence would satisfy the constraint.
At decoding time,
FUDGE reweights the LM prediction by the discriminator scores.
FT.
For each control task, we ﬁne-tune GPT-2 on (control, text) pair.
We report both the samplingand beam search outputs of the ﬁne-tuned models, denoted as FT-sample and FT-search, respectively.
Note that this is an oracle, since it requires ﬁne-tuning the LM parameters.
5.4 Inﬁlling Baselines
We compare to 3 specialized baseline methods developed in past work for the inﬁlling task.
DELOREAN [28].
This method continuously relaxes the output space of a left-to-right autoregressive
LM, and iteratively performs gradient updates on the continuous space to enforce ﬂuent connectionto the right contexts.
This yields a continuous vector which is rounded back to text.
COLD[29].
COLD speciﬁes an energy-based model that includes ﬂuency (from left-to-right andright-to-left LM) and coherence constraints (from lexical overlap).
It samples continuous vectorsfrom this energy-based model and round them to text.
AR-inﬁlling.
We train an autoregressive LM from scratch to do sentence inﬁlling task [8].
Similar totraining Diffusion-LM, we train on the ROCStories dataset, but pre-process it by reordering sentencesfrom ( O 1 , O middle , O 2 ) to ( O 1 , O 2 , O middle ) .
At evaluation time, we feed in O 1 , O 2 , and the modelgenerates the middle sentence.
Results
We train Diffusion-LMs on the E2E and ROCStories datasets, and compare to baseline autoregressivemodels (GPT-2) with comparable parameter counts.
Diffusion-LM has worse holdout log-likelihood 7than a comparably sized GPT-2 model for both datasets (E2E: 2.28 v.s. 1.77 , ROCStories: 3.88 v.s3.05) although we begin to bridge this gap by doubling the size of our Diffusion-LM and training onmore data (ROCStories: 3.10 v.s. 3.05).
Despite the lower perplexity, controllable generation basedon our Diffusion-LM results in signiﬁcantly better outputs than systems based on autoregressive LMs.
6.1 Predictor-Guided Controllable Text Generation Results
As shown in Table 1, Diffusion-LM achieves high success and ﬂuency across all predictor-guidedcontrol tasks.
It signiﬁcantly outperforms the PPLM and FUDGE baselines across all 5 tasks.
Surprisingly, our method outperforms the ﬁne-tuning oracle on the syntax tree control and the spancontrol tasks and achieves similar performance on the remaining 3 tasks.
[Table] [Table]
[Table]
Controlling the syntax tree and spans are challenging tasks for ﬁne-tuning, because conditioning onthe syntax tree requires reasoning about the nested structure of the parse tree, and conditioning onspans requires lookahead planning to ensure the right constituent appears at the target position.
We observe that PPLM fails in the semantic content control task and conjecture that this is because
PPLM is designed to control coarse-grained attributes, and may not be useful for more targeted taskssuch as enforcing that a restaurant review contains a reference to Starbucks.
FUDGE performs well on semantic content control but does not perform well on the remaining fourtasks.
Controlling a structured output (POS and syntax tree) is hard for FUDGE because making onemistake anywhere in the preﬁx makes the discriminator assign low probabilities to all continuations.
In other control tasks requiring planning (Length and Spans), the future discriminator is difﬁcult totrain, as it must implicitly perform lookahead planning.
The non-autoregressive nature of our Diffusion-LM allows it to easily solve all the tasks that requireprecise planning (spans and length).
We believe that it works well for complex controls that involveglobal structures (POS, syntax parse) because the coarse-to-ﬁne representations allow the predictorsto exert control on the entire sequence (near t = T ) as well as on individual tokens (near t = 0 ).
Qualitative Results.
Table 2 shows samples of syntax tree control.
Our method and ﬁne-tuning bothprovide ﬂuent sentences that mostly satisfy controls, whereas FUDGE deviates from the constraintsafter the ﬁrst few words.
One key difference between our method and ﬁne-tuning is that Diffusion-LMis able to correct for a failed span and have sufﬁx spans match the target.
In the “Family friendly
Indian food” example, the span is wrong because the generated span contains 1 more word than thetarget.
Fortunately, this error doesn’t propagate to later spans, since the model adjusts by droppingthe conjunction.
Analogously, in the “The Mill” example, FT model generates a failed span, but itfails to adjust it in the sufﬁx, leading to many mis-aligned errors in the sufﬁx.
6.2 Composition of Controls
One unique capability of plug-and-play controllable generation is its modularity.
Given predictors formultiple independent tasks, gradient guided control makes it simple to generate from the intersectionof multiple controls by taking gradients on the sum of the predictor log-probabilities.
We evaluate this setting on the combination of semantic content + syntax tree control and semanticcontent + POS tag control.
As shown in Table 3, our Diffusion-LM achieves a high success rate for
[Table]
both of the two components, whereas FUDGE gives up on the more global syntactic control.
This isexpected because FUDGE fails to control syntax on its own.
Fine-tuned models are good at POS and semantic attribute control individually but do not composethese two controls well by product of experts (PoE), leading to a large drop in success rates for bothconstraints.
6.3 Inﬁlling Results
As shown in Table 4, our diffusion LM signiﬁcantly outperforms continuous relaxation based methodsfor inﬁlling (COLD and Delorean).
Moreover, our method achieves comparable performance toﬁne-tuning a specialized model for this task.
Our method has slightly better automatic evaluationscores and the human evaluation found no statistically signiﬁcant improvement for either method.
These results suggest that Diffusion LM can solve many types of controllable generation tasks thatdepend on generation order or lexical constraints (such as inﬁlling) without specialized training.
6.4 Ablation Studies
We verify the importance of our proposeddesign choices in §4 through two ablationstudies.
We measure the sample quality of
Diffusion-LM using the lm-score on 500samples §5.2.
Learned v.s.
Random Embeddings(§4.1).
Learned embeddings outperformrandom embeddings on the ROCStories,which is a harder language modeling task.
The same trend holds for the E2E datasetbut with a smaller margin.
Objective Parametrization (§4.2).
Wepropose to let the diffusion model predict x 0 directly.
Here, we compare this with standardparametrization in image generation which parametrizes by the noise term ✏ .
Figure 4 (right)shows that parametrizing by x 0 consistently attains good performance across dimensions, whereasparametrizing by ✏ works ﬁne for small dimensions, but quickly collapses for larger dimensions.
Conclusion and Limitations
We proposed Diffusion-LM, a novel and controllable language model based on continuous diffusions,which enables new forms of complex ﬁne-grained control tasks.
We demonstrate Diffusion-LM’ssuccess in 6 ﬁne-grained control tasks: our method almost doubles the control success rate of priormethods, and is competitive with baseline ﬁne-tuning methods that require additional training.
Admittedly, Diffusion-LM has some drawbacks relative to autoregressive LMs: (1) it suffers fromhigher perplexity; (2) decoding is substantially slower; and (3) training converges more slowly.
Despite these limitations, we ﬁnd the degree of control enabled by Diffusion-LM compelling.
Wehope that the ability to control Diffusion-LM that we have demonstrated will motivate further workto reﬁne and scale this language modeling technique, overcoming its current limitations.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] in the conclusion and limitationsection also in the ﬁrst paragraph of result section.(c)
Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]