S TEP - UNROLLED D ENOISING A UTOENCODERS FOR T EXT G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper we propose a new generative model of text, Step-unrolled Denoising Autoencoder (SUNDAE), that does not rely on autoregressive models.
Similarly to denoising diffusion techniques, SUNDAE is repeatedly applied on a sequence of tokens, starting from random inputs and improving it each time until convergence.
We present a simple new improvement operator that converges in fewer iterations than diffusion methods, while qualitatively producing better samples on natural language datasets.
SUNDAE achieves state-of-the-art results (among non-autoregressive methods) on the WMT’14 English-to-German translation task and good qualitative results on unconditional language modeling on the Colossal Cleaned Common Crawl dataset and a dataset of Python code from
GitHub.
The non-autoregressive nature of SUNDAE opens up the possibilities beyond prompted generation, by ﬁlling in arbitrary templates.
I NTRODUCTION
Autoregressive (AR) models have shown excellent results in generating text (e.g., GPT-3, Brown et al., 2020).
However, while their training scales very well, sampling is prohibitively slow for many practical applications.
Moreover, there are limitations to the kinds of conditioning AR models can seamlessly handle: the left-to-right restriction makes it hard to “ﬁll in the gaps” in a partially written text draft.
Even more importantly, this prohibits iterative reﬁnement of complete text drafts to make them more self-consistent, which is a common task for human writers.
Finally, AR models require network architectures to be causal, severely limiting the kinds of neural network architectures that can be used for text-modeling.
All of these motivated the machine learning community to make extensive efforts to propose alternatives to AR models.
Perhaps one of the ﬁrst text generation tasks for which it has been shown that an AR approach can have serious rivals was machine translation (MT): methods like CMLM (Ghazvininejad et al., 2019) and DisCo (Kasai et al., 2020) show promising results and their decoding speed is excellent compared to AR.
However, while their performance is competitive, they are still behind the AR benchmark and actually require distillation of a larger AR model — without which, performance drops considerably.
Non-AR methods have proved hard to apply to the general unconditional language modeling (LM) task.
When there is no conditioning, the multi-modality problem becomes paramount, as shown by Gu et al.
(2017), which likely makes it problematic to use methods like CMLM and DisCo because their decoding mechanism is deterministic and does not model uncertainty.
Yet, recently the community has seen promising results from non-AR models like Multinomial Diffusion (Hoogeboom et al., 2021) and D3PM (Austin et al., 2021).
These methods optimize a lower bound (ELBO) on likelihoods and have shown negative log-likelihood (NLL) results approaching AR models on several benchmarks like text8 (Mahoney, 2011) and LM1B (Chelba et al., 2013).
However, a major gap in NLL persists, and samples from those models lack coherence.
In this paper we propose a novel non-autoregressive method which shows state-of-the-art results in machine translation on WMT’14 EN → DE raw data (without distillation from AR) amongst nonAR methods and good qualitative results on unconditional language modeling on the Colossal Clean
Common Crawl (C4) dataset (Raffel et al., 2019) and a dataset of Python code from GitHub.
Our model operates as a time-homogeneous Markov Chain similar to that of Lee et al.
(2018): condi-
[Figure]
tioned on the corrupted data, it tries to approximate the original uncorrupted samples by a per-token factorised distribution. During generation, we unroll the chain by sampling from the transition distribution and feeding samples back into the input.
We propose a new training mechanism called unrolled denoising , which uses unrolls of the chain during training as well, and which we empirically show to be crucial for practical performance of the algorithm in ablation studies.
This feature motivates the name for the model, Step-unrolled Denoising Autoencoder (SUNDAE).
The difference between usual denoising and unrolled denoising is illustrated in Figure 1. To summarize our contributions:
• We present SUNDAE , a new generative model of text that unrolls the denoising process during training.
• SUNDAE achieves state-of-the-art results on WMT’14 English-to-German translation task among non-AR methods.
• We demonstrate good qualitative results for unconditional generation and inpainting on Colossal Clean Common Crawl dataset and a dataset of Python code from GitHub.
• We carefully ablate and analyze the properties of the proposed method, and show that unrolls during training are crucial for the model’s performance.
M ETHOD
We approach the problem of generative modelling of discrete sequences by bringing together the framework of denoising autoencoders and Markov chain models.
In this section, we ﬁrst discuss the deﬁnition of the generative model, and then describe the training process, which includes our main contribution, unrolled denoising .
For a ﬁxed prior distribution p 0 on some space X consider a process x t ∼ f θ ( ·| x t − 1 ) , where f θ is a parametric transition function.
Then { x t } t is a time-homogeneous Markov chain, and transition t -steps ahead has the following form
[Equation]
For a ﬁxed number of steps T , the prior p 0 and decoder p T together determine our model distribution p T ( x T ) =
p T ( x T | x 0 ) p 0 ( x 0 ) .
We assume X = { 1 , . .
. , v } N to be the space of sequences of length N with values at all positions coming from a vocabulary of size v , and the prior p 0 to be uniform over X .
Let p data be a distribution of the data and assume that f θ induces a conditional probability distribution f θ ( ·| x (cid:48) ) that can be factorised into a conditionally-indepedent product as follows:
[Equation]
Note that although for t = 1 the distribution p 1 ( ·| x 0 ) is limited by product structure (Eq. 2), the subsequent p t s are not restricted in the same way, and after each step they belong to potentially more and more expressive family.
2. T RAINING WITH U NROLLED D ENOISING
Due to intractability of the likelihood of the model p T as well as computational requirements of optimising the whole chain { p t } t , we propose an efﬁcient two-step training method which we term unrolled denosing and visualise in Fig.
We consider a smaller number of steps than T that we intend to use at sampling, but to compensate that, we unroll the chain starting from corrupted data samples, rather than the prior p 0 .
This way, the model learns to denoise the samples it is likely to encounter during the full unroll used at sample time.
While using a single step would resemble the training strategy of BERT (Devlin et al., 2018), using at least two steps is essential for the performance of our model.
Consider a corruption distribution q ( x c | x ) that replaces a random proportion of tokens in a sentence x with ones randomly sampled from a uniform distribution.
Our objective is given by L (1:2) = 12 (cid:0) L (1) + L (2) (cid:1) where
[Equation]
i.e. the reconstruction loss of the chain after t steps, starting from a corrupted sample x 0 .
In supplementary material (Section A.1) we show that L ( t ) is an upper bound on the actual negative loglikelihood from the model distribution p t :
[Equation]
Note that (cid:103) L ( T ) is the actual term we would like to minimise if p T was tractable, and in fact is closely related to evidence lower bound (ELBO) via a ﬁxed encoder q ( x c | x ) (Section A.2).
However, due to inherent non-differentiable nature of Markov chain models, optimisation of (cid:103) L ( T ) would not only be costly, but also limit the gradient ﬂow to the last step of the chain.
We found instead that optimising several steps together is sufﬁcient for obtaining good results.
Averaging more L ( t ) terms can lead to minor improvements in performance (See Fig.
3d in supplementary material), but it considerably slows down the training speed.
We discuss these and other loss variants further in Section 3.1.
Since function f θ is modelled as a neural network, the logarithm under expectation on the right hand side of Eq.
3 is obtained from predicted logits, thus we term L (1) ( θ ) and L (2) ( θ ) logits loss and unrolled logits loss, respectively.
Corruption function.
To corrupt texts, we ﬁrst sample a proportion of tokens uniformly from [0 , 1] , randomly select positions according to this proportion and then change tokens at those positions to random tokens sampled uniformly from the vocabulary.
This way, we want to ensure that samples seen at various steps during generative unroll (including the ﬁrst one, sampled from the uniform prior) are well-represented in corruptions during training.
The detailed construction as well as relation to the forward discrete diffusion process (Hoogeboom et al., 2021) is presented in Section A.3 in supplementary material .
2. S AMPLING
At sampling time we follow the structure of the Markov process and sample sequentially x t ∼ f θ ( x t | x t − 1 ) for some ﬁxed number of steps T , beginning from a random sequence x 0 (potentially starting with a prompt or a template for conditional generation).
To control the speed of convergence, we propose three improved strategies which allow to use of a much smaller number of steps:
• Low-temperature sampling .
For temperature τ and trained network f θ we consider modiﬁed function f
τθ such that log f τθ ( ·| x ) ∝ 1 τ log f θ ( ·| x ) .
We found that sampling with temperatures lower than 1 leads to much faster convergence than with the original logits, and allowed generation of high-quality samples in a matter of 10-32 steps.
• Argmax-unrolled decoding .
At the limit [-⌧ ! 
-]τ → 0 sampling with temperature reduces to deterministic argmax decoding where the most probable token is chosen at each step.
Relatedly to our unrolled logits loss, we modify this strategy by resampling the low-certainty tokens in accordance with unrolled logits .
This heuristic allowed further improvements on the sampling speed while maintaining high quality of the samples.
We discuss it in detail in Section 3.1.
• Updating fewer tokens .
In tasks where diversity is paramount (like unconditional text generation), we found that updating a random subset of tokens at each decoding step leads to faster convergence.
This likely happens because independent sampling of all tokens might create uncoordinated changes which could take some time to ﬁx in the subsequent steps.
We use this strategy in Section 3.2.
E XPERIMENTS 3. M ACHINE T RANSLATION
We ﬁrst evaluate SUNDAE on Machine Translation (MT) benchmarks.
We compare SUNDAE against AR and non-AR models in terms of the translation quality using BLEU (Papineni et al., 2002) as the metric.
We demonstrate that without using techniques like sequence-level knowledge distillation (Kim & Rush, 2016), SUNDAE can perform almost as well as the AR baseline and outperform all other models that do not rely on AR models.
Target Length Prediction.
Unlike their AR counterparts, non-AR models do not explicitly learn to predict an end of sequence, but instead have been shown to beneﬁt from auxiliary target length prediction (Lee et al., 2018;
Ghazvininejad et al., 2019; Kasai et al., 2020).
A common approach is to treat it as a classiﬁcation task, predicting either the exact length or the difference between the source and target lengths.
The decoding process has to commit to the predicted length, and often multiple length beams are used to get the best translation results.
In our models, we use a separate network that receives source encodings as input and predicts corresponding target length.
Target length embedding vector is prepended to source embeddings so that decoder can attend to it.
During training, we use the reference target length, while at sampling, a predicted one, although the model does not need to fully commit to the predicted length like some previous non-AR models (Saharia et al., 2020). Note that the length classiﬁcation loss does not affect the encoder parameters.
Our models are also trained to predict the padding tokens at the end of the text.
For more details, see Section B in supplementary material .
Experiment Settings.
We conduct experiments on WMT’14 parallel corpora using EN ↔ DE ( 4 .
5 M pairs) and EN → FR ( 36 M pairs) translation tasks.
The raw texts are encoded using BPE (Sennrich et al., 2015) as the subword units, and we use the same preprocessed data as in Vaswani et al.
(2017) for fair comparisons.
We evaluate the performance by measuring BLEU (Papineni et al., 2002; Post, 2018) on the test split of each translation task 2 .
We use the encoder-decoder Transformer architecture for MT (Vaswani et al., 2017), but remove the causality masking in the decoder.
There are 6 attention layers for both encoder and decoder, 8 attentions heads, 512 model size and 2048 hidden size.
The total number of parameters is 63 M including the target length prediction module described in 3.1.
We use dropout ( p = 0 . 1 ) and label smoothing ( (cid:15) = 0 . 1 ) during training.
The training batch size is 4096 , and we use Adam (Kingma & Ba, 2014) with β 1 =0 .
9 , β 2 =0 . 999 , (cid:15) = 10 − 6 and weight decay of 0 . 1 (Loshchilov & Hutter, 2017).
We warm up the the learning-rate from 10 − 7 to 10 − 4 in the ﬁrst 5 K steps and decay it to 10 − 5 using cosine annealing (Loshchilov & Hutter, 2016).
Our models were trained for 10 6 steps using 16 TPU accelerators using bf16 precision.
We crop sequences that have more than 128 tokens (this occurs less than 0 . 2% in training data).
Finally, we average the last 10 checkpoints to obtain a single model for the evaluation.
Decoding.
We decode translations from SUNDAE using two different types of heuristic decoding methods in MT experiments.
The decoding process always begins with an array of random integers [Table] DisCo has a dynamic inference mode for which we report the [-discrete uniform prior y 
-]average number of steps.
sampled from the discrete uniform prior y 0 ∼ p 0 , while the encoder input - the source sentenceremains the same throughout the process.
The ﬁrst decoding method is the low-temperature sampling , where we iteratively sample y t ∼ f τθ ( ·| y t − 1 ) for T iterations (with the decoder’s output logits divided by τ , see Section 2.2), with T ≤ 128 and τ ∈ [0 . 1 , 0 . 6] determined based on the validation performance.
The second method is argmax-unrolled decoding , which requires a smaller number of iterations compared to its stochastic counterpart.
In argmax-unrolled decoding, we ﬁrst compute the logits λ 1 = f θ ( ·| y 0 ) of the initial sample array y {+
+} and obtain the samples y 1 , pass a tuple ( y 1 , λ 1 ) to the next iteration.
At each step t ≥ 2 , the method ﬁnds top- ρ share of the tokens that are sorted by the log-probability in descending order (i.e. uncertain tokens) from λ t − 1 , where ρ ∈ [0 . 1 , 0 . 6] is another hyperparameter searched for in the validation stage.
We compute unrolled logits for those top- ρ tokens and then apply arg max to obtain unrolled samples .
For the rest of the tokens of y t −we compute logits once and obtain λ t = f θ ( ·| y t − 1 ) and then apply arg max to obtain samples .
Finally, we combine both unrolled samples and samples to obtain y t , and the tuple ( y t , λ t ) is provided as the input to the next iteration.
This procedure is repeated over a ﬁxed number of iterations T .
We always decode n samples in parallel and rerank them based on the model score.
We show the relative speed gain of SUNDAE in Table 3, the Transformer base is used as the AR baseline, where we perform incremental sampling by caching the previous attention states.
Baselines.
We compare SUNDAE with Transformer base (Vaswani et al., 2017) as the AR baseline and several non-AR models including CMLM (Ghazvininejad et al., 2019) and DisCo (Kasai et al., 2020).
The last two are both iterative non-AR models that share a substantial amount of common ground with SUNDAE.
However, their best results come from the variants distilled from
Transformer large (Vaswani et al., 2017), with only few test results available without distillation.
As we aim to remove any kinds of dependencies on AR approaches, we would like to make a dis- Index Samplefromourmodel Ihadtogobackandforthandstartovertoseewhathadhappened,whatwasworking,andallthepieceswereintherightplaces. tossedinadelicioussauce.
Beefbrisketisaheartycutofmeat–thickandtender.
Whenyoumakealotofmeat canexpecttoseemoreofthesamein’WildHerring’,althoughnotveryoftenenoughtoachievethesameeffect.
In’Herring’, ,shewasconsideredtobeafake.
Dr. DeborahChiennahas93inthelast365days.
,headaches,stomachpain,sore asanartist.
printingphotocards,artwork,andposters.
emailmeforsomeideas.
”Rebecca’sstudio
Jersey,PhiladelphiaandWashingtonDC.Thisisaconferenceattendedbyresidents,physicians,graduatemedicalstudents,andothersintheﬁeldofpublichealth.
’slocation.
ThisresearchwassupportedbyagrantfromtheNationalAssociationfortheAdvancementofSciencethroughtheU.S.DepartmentofEnergy differentartists,wewouldlovetoshowyouthepiecewehaveinourshop!
KeepaneyeonourGallery–allyouhavetodoistowritingwiththeAutumns.
Together,someoftheclasswillspendtimeintheclassroomtoreadapoemfromtheoriginalbook.
Fromtherewe
[Figure]
[Table]
tinction between raw, fully-non-AR models, and AR-distilled ones, i.e. those trained via distillation from large AR teachers (typically perform better than the student models), or via AR reranking.
Results and Ablation Studies.
In Table 1, we show test BLEU scores from SUNDAE and other baselines.
On EN → DE, SUNDAE achieved 26 . 57 BLEU, and it performed the best among the raw non-AR models.
To the best of our knowledge, SUNDAE achieved the closest result ( (cid:52) = 0 . 73 ) to the score of Transformer base (Vaswani et al., 2017) without the aid of AR models.
In the raw setting, argmax-unrolled decoding outperforms mask-predict (Ghazvininejad et al., 2019) and easyﬁrst (Kasai et al., 2020) by 0 . 74 BLEU when using the same amount of the worst-case compute budget.
At T =10 , SUNDAE performs better than Imputer (Saharia et al., 2020), which is a strong baseline that can generate high-quality translation within few ( 4 - 8 ) iterations.
We discovered that as
T increases, low-temperature sampling outperforms argmax-unrolled decoding, and at T =10 , lowtemperature sampling outperforms Mask-Predict
(Ghazvininejad et al., 2019) and Easy-First (Kasai et al., 2020) by 1 . 5 BLEU score.
SUNDAE achieved 30 .
80 BLEU on DE → EN, where there are not that many raw baselines to compare with.
The difference between SUNDAE and the AR baseline on DE → EN is 0 . 98 .
Finally, SUNDAE achieved 37 .
77 BLEU on EN → FR, where the Transformer base (Vaswani et al., 2017) reported 38 . 1 BLEU on this task.
SUNDAE is only 0 . 33 BLEU behind the standard AR baseline on EN → FR, once again without distillation.
We ablate the number of steps of unrolled denoising on EN → DE test split.
We observe that having at least one unrolled denoising step is crucial to obtain a good performance in translation.
With only L (1) loss, SUNDAE achieves 11 . 19 BLEU, whereas using L (1:2) , the score is improved to 26 . 57 BLEU.
Using L (1:3) , i.e. averaging more unrolled denoising loss terms, did not improve the performance further, scoring 26 . 25 BLEU.
The target length prediction also turned to be an important component of our model: without it, the translation performance degrades on average by 2 BLEU score points on EN → DE (see Section C in supplementary material ).
3. T EXT G ENERATION
Unconditional Text Generation (Qualitative).
We train our method on a large high-quality publicly available Colossal Clean Common Crawl (C4) dataset (Raffel et al., 2019) to demonstrate [Figure] ( d i v e r s i t y [Figure]
Left is better, lower is better.
Quality/diversity is controlled by varying the temperature. [Figure] samples. We tokenize the data using SentencePiece (Kudo & Richardson, 2018) 3 with vocabulary size 32 K and train on randomly cropped sequences of length 32 .
The network is the same as the Transformer decoder part for MT in Section 3.1 but is much larger — 335 M parameters: 24 layers, 1024 embedding size, 4096 hidden size, 16 attention heads.
Same as in the MT setup, we remove the causal mask since our model is non-autoregressive.
We train up to 400 K steps with Adam optimizer with batch size 4096 and use cosine annealing for scheduling the learning-rate.
The trained model is then used for unconditional sampling.
Starting with 32 random tokens, we iteratively perform up to 1 K steps from our model, stopping earlier if samples do not change anymore.
To make convergence faster, we use temperature 0 . 8 and update only 30% of all tokens at each step, chosen randomly — as explained in Section 2.2.
We show 10 non-cherry-picked samples from our model in Table 2.
All of them resemble reasonable-quality internet texts, perhaps except number 4 . Unconditional Text Generation (Quantitative).
While C4 is one of the largest high-quality datasets publicly available and it is good for showing interesting samples, it does not have an established benchmark among non-AR methods.
To quantitatively evaluate our algorithm, we train the same architecture on the EMNLP2017 News 4 dataset, which has numerous text GAN baselines, including a strong ScratchGAN method (d’Autume et al., 2019).
We also compare to several other baselines reported in (d’Autume et al., 2019; Caccia et al., 2020):
SeqGAN (Yu et al., 2017), LeakGAN (Guo et al., 2017), RankGAN (Lin et al., 2017), MaliGAN (Che et al., 2017) and autoregressive (AR) language model. The results are shown in Figure 2. SUNDAE demonstrates good quality/diversity trade-off, as measured by BLEU/self-BLEU metrics, outperforming 4 out of 5 GAN baselines, and competing with
ScratchGAN and AR model.
In the higher quality region, SUNDAE outperforms ScratchGAN, while in higher diversity region it slightly underperforms.
The advantage of ScratchGAN in higher diversity region could be explained by its autoregressive decoding (it is a hybrid between AR and
GAN).
Interestingly, the curve for SUNDAE has a more complicated shape than the baselines — possibly because of its Markov Chain sampling, which can empirically exhibit something akin to “phase transitions”.
Text In-painting. The non-autoregressive nature of our model opens up the possibilities beyond prompted generation, such as ﬁlling in arbitrary templates.
In this section, we qualitatively investigate this new capability, using the C4-trained model described in the previous section, and another one, trained on a dataset of Python code from GitHub. All the architecture/training settings remain the same, except we train twice longer: up to 800 K steps. Sampling settings are also the same, except we update all the tokens at each step as diversity is less of an issue with stronger conditioning.
We construct the code dataset by extracting ﬁles ending in .py from open source GitHub repositories with licenses that are one of
[Table]
For the
C dataset, we present a model with a particular type of prompt [CTX1][MSK1][CTX2][MSK2] , where [CTX] stands for “context” and [MSK] for “masked”, in Table 4, which shows how information can be “teleported” from arbitrary location to another arbitrary location, while taking into account bidirectional context.
We change a species of the animal in [CTX2] and observe that the model actually uses the right word while in-painting [MSK1] and plausibly continues [MSK2] according to the species.
This kind of task would be impossible to perform for an AR model: with left-to-right causality it would be unable to correctly guess the name of the animal, while with a right-to-left one it would have to start ﬁlling [MSK2] tokens ﬁrst, so it would just ignore the whole conditioning.
By contrast, our model can easily perform this task.
For GitHub Python data, we use two different kinds of prompts for our model in Table 5:
[CTX1][MSK1][CTX2] and [CTX1][MSK1][CTX2][MSK2][CTX3] .
In the ﬁrst example, our model guesses that n should be the length of the input sequence and correctly in-paints [MSK1] region.
This would again be impossible to do for AR models.
Looking from left-to-right, it is impossible to guess that n will be the name of the length variable and t is meant to be a threshold.
Looking from right-to-left, it is unclear what variables n and t represent and what is the function being computed.
Only a bidirectional model can perform well in this task.
In the second example, our model correctly follows the suggestion to reuse the function which it also has to in-paint.
R ELATED W ORK
In this section we provide an overview of generative models, focusing on applications in text domain.
4. A UTOREGRESSIVE MODELS
Early attempts to perform autoregressive modelling of text using neural networks began with Bengio et al.
(2003), later Sutskever et al.
(2011) has extended this approach with RNNs.
Since then, improvements in neural network architecture, such as the Transformer (Vaswani et al., 2017), and scaling up as with GPT-3 (Brown et al., 2020) dramatically improved the performance of AR models.
While the architectures evolved, the loss remained the same with minor modiﬁcations.
Insertion (Stern et al., 2019) and Levenshtein (Gu et al., 2019)
Transformers are deviations from the standard paradigm of training and could potentially overcome some of AR problems.
However, in the parallel decoding regime, they have to rely on AR distillation for obtaining competitive results.
4. N ON - AUTOREGRESSIVE MODELS IN GENERAL
Non-autoregressive models evolved in parallel with autoregressive ones but have so far struggled to address the same spectrum of tasks with the same quality as AR models.
Diffusion for generative models was originally introduced by Sohl-Dickstein et al.
Recently Ho et al.
(2020) proposed a different parametrization: instead of modeling one-step transitions, one could instead model p ( x 0 | x t ) and later convert those to p ( x t − 1 | x t ) by probabilistic reasoning.
This was recently up-scaled for image modeling by Nichol & Dhariwal (2021) and extended to continuous time by Song et al.
In terms of applications to text generation, two recent works addressed this: Hoogeboom et al.
(2021) and Austin et al.
Likelihoods obtained in those works are promising, but still behind AR and lacking sample quality as well.
Some diffusion works like Mittal et al.
(2021) also approached modeling discrete sequences in another way: ﬁrst encode sequences with a VAE into a continuous space and then apply continuous diffusion approaches commonly used for image generation.
Variational Autoencoders (VAEs) were proposed by Rezende et al.
(2014) and Kingma &
Welling (2013) as a likelihood-based extension of autoencoders.
Application of such methods to text has been problematic empirically: as it was ﬁrst noted in Bowman et al.
(2015), good results come from using a strong AR decoder instead of a simple factorised one, but the latents are often ignored — which was named “posterior collapse problem”.
This problem was later tackled by
Delta-VAEs (Razavi et al., 2019), but with limited success in the text domain, as results of Bosc &
Vincent (2020) show.
Normalising Flows (Dinh et al., 2014; Rezende & Mohamed, 2015; Dinh et al., 2016) are another prominent family of generative models, originally proposed for real-valued data and recently revisited for text generation by Hoogeboom et al.
While conceptually interesting, the text samples of such methods lack in ﬁdelity.
Generative Adversarial Networks (GANs) were originally introduced by Goodfellow et al.
(2014) and remain one of the dominant methods for image generation.
They were later adapted to text generation in (Yu et al., 2017; Guo et al., 2017; Lin et al., 2017; Che et al., 2017).
However, text generation with such methods is still a challenge, with a more recent ScratchGAN (d’Autume et al., 2019) showing relatively low-quality samples.
At least part of the problem with applying GANs to text generation comes from non-differentiability of discrete text samples, which requires usage of zeroth order optimization methods.
Energy-Based models have a long history dating back to Hopﬁeld (1982); Hinton (2002); LeCun et al.
Ranzato et al.
Recently, Deng et al.
(2020) shows promising results for text.
4. N ON - AUTOREGRESSIVE MODELS FOR MACHINE TRANSLATION
There have been many attempts to apply non-autoregressive methods to machine translation.
Latent transformer (Kaiser et al., 2018) generated latent variables autoregressively and then decoded feedforwardly.
NAT (Gu et al., 2017) was the ﬁrst to characterize a problem with non-autoregressive generation — “multi-modality”.
FlowSeq (Ma et al., 2019) applied Normalising Flows to machine translation.
LVM-DAEs (Lee et al., 2018) are perhaps most related to our method, but these models do not have unrolled denoising and their decoding method is deterministic.
The leading methods in this area are CMLM (Ghazvininejad et al., 2019) and DisCo (Kasai et al., 2020).
While conceptually similar to LVM-DAEs (Lee et al., 2018), these works have introduced signiﬁcant improvements to the training and decoding procedure, and, as a result, shown strong competition to AR methods.
Both of them also do not have unrolled denoising and sampling like we do, but they are still powerful baselines.
Later, there were attempts to combine CMLM with local AR (Kong et al., 2020). Perhaps the most related to our work is the SMART (Ghazvininejad et al., 2020) follow-up of the CMLM.
However, this method is not probabilistic and it does not show unconditional generation capabilities.
4. D ENOISING OBJECTIVE IN GENERAL
With the advent of BERT (Devlin et al., 2018), the denoising objective became very popular for text representation learning.
The original idea was later simpliﬁed in RoBERTa (Liu et al., 2019) by removing unnecessary next-sentence-prediction task and only retaining denoising task (the cloze task).
This is similar to our work, however, we do not apply masking to corrupt the input tokens, and instead only switch them to random ones.
Another work, Electra (Clark et al., 2020a), deals with the same kind of corruption that we use, but instead of predicting the original token before the corruption, Electra predicts whether it was corrupted or not.
Electric (Clark et al., 2020b) does not perform masking, but instead uses noise-contrastive loss to learn a representation of text. BART (Lewis et al., 2019) uses both BERT-style denoising and autoregressive modeling for learning representations.
All of these methods were originally motivated by improving representation learning of text.
There were a few attempts to heuristically decode models like BERT and turn them into text generative models such as in Wang & Cho (2019), however, samples from these models lack coherence.
C ONCLUSION
In this paper we have proposed a novel non-autoregressive method that operates within the framework of denoising autoencoders and, crucially, unrolls the denoising process during training.
Unrolled denoising allows us to achieve state-of-the art results in WMT’14 English-to-German translation task amongst non-AR methods (without distillation from large AR models) and good qualitative results in unconditional text modeling on C4 dataset.
We have also qualitatively demonstrated new inpainting capabilities on C4 and GitHub Python data, which opens up new avenues for creative text editing where a human could more naturally collaborate with a machine on writing and even programming. {+A M ETHOD DETAILS Using Markov property and the form of p t = f θ ( ·| x t − 1 ) , for any t > 
 we obtain+}
[Equation]
By induction, we can keep unrolling the chain for all steps s = t − 1 , t − 2 , . .
, 1 , eventually obtaining
[Equation]
which proves the from of the model distribution (Eq. 1).
A.
U PPER BOUND ON THE LOSS .
Using Jensen’s inequality we can swap log and expectation in Eq.
7 to obtain
[Equation]
hence
[Equation]
A.
R ELATION TO VAE
Variational Autoencoders (Kingma & Welling, 2013; Rezende et al., 2014)
optimise the expected evidence lower bound (ELBO) on the marginal likelihood
[Equation]
with respect to parameters of both the variational distribution q and generative distribution p , where p 0 remains the prior for for the latent variables.
Denosing autoencoders limit the optimisation to the second term, assuming a ﬁxed encoder q , which is also the case with SUNDAE: identifying encoder with our corruption distribution and generative distribution p with the distribution of our chain at the last step p T , expected ELBO becomes
[Equation]
11) where the second inequality is a consequence of the upper bound 3.
Hence, minimizing L ( T ) would lead to maximisation of the lower bound on the model likelihood.
A. C ORRUPTION FUNCTION
Our corruption function is obtained in stages:
• sample uniform expected corruption proportion α ∼ U ([0 , 1]) , • sample Bernoulli mask, independently for each token m ∼ ( Bernoulli ( α ))
N , • sample noise η ∼ ( U ( { 1 , 2 , . . .
, v } ) N ,
1.0 step 1e
B L E U
[Figure]
1.0 step 1e
B L E U
[Figure]
[Figure]
[Figure]
[Figure]
L E U
[Figure]
[Figure]
[Figure]
B L E U
L(1)L(2)L(3)
L(4)L(5)
[Figure]
• compute corruption x c =
c α, m ,η ( x ) =
(1 − m ) · x + m · η (with multiplications taken element-wise).
The distribution p c ( ·| x ) p data ( x ) obtained in this process covers the entire input space X , with probability skewed towards samples whose parts come from p data .
Note that except for corruption proportion, all steps are independent for all individual tokens.
Given α , p c α ( y | x ) can be factorised as (cid:89) n
[Equation]
where Q α = α
I + (1 − α )
V is a transition matrix, V is a
v × v matrix with all entries equal tov , and h ( a ) denotes one-hot vector representation of a ∈ { 1 , 2 , . . .
, v } .
In this context, corruption with rate α corresponds to a multinomial-diffusion forward process with probability α of remaining in the same state, as proposed by (Hoogeboom et al., 2021).
B D ETAILS OF T ARGET L ENGTH P REDICTION
The target length prediction module consists of 6 residual blocks.
The length prediction module takes the source encodings h x = encoder ( x ) as the input, where x is a sequence of source tokens.
We ﬁrst project h x into a vector v x ∈ R d LP , where d LP = 128 is the hidden size of the target length prediction module.
We add a source length embedding vector v len
=
S
( N source × d LP )
( l x ) to v x and obtain the ﬁnal encoder state representation v (cid:48)
x = v x + v len , where l x is the number of the source tokens, and N source = 128 is the maximum number of the source tokens.
The target length prediction module then takes v x as the input and predicts the (downsampled) target length ˜ l d .
We found beneﬁcial downsampling the target lengths for the length prediction by a factor of 2 , i.e. l d = (cid:100) l/ 2 (cid:101) as compared to exact prediction of l ; since the maximum number of target tokens is
N =128 , the maximum size of the outcome of length prediction is N d =64 .
[Table]
Ich habe dort Dinge gesehen, aus denen ich gest¨arkt hervorgegangen bin.
[Table]
ThirVerschlMarta moderatopposed frameworks solidierweitert [...]
StepI saw things there that which I I stronger..
StepI saw things there from which I me stronger..
StepI saw things there from which I emerged stronger.
StepI saw things there from which I emerged stronger.
[Table]
C E FFECTS OF T ARGET L ENGTH P REDICTION
It is shown empirically that the length of the translated results can affect BLEU score.
Approximate search methods such as beam search usually incorporate a length normalization term in order to prevent the search algorithm to prefer shorter sentences.
This is due to the fact that the scoring function is usually chosen as the likelihood modelled by the translation system, and beams containing shorter sequences tend to score higher likelihoods.
In non-AR models, the end of sequence is not explicitly learned by the models, thus they can beneﬁt by knowing the target sequence length in advance to generating the translation.
However, the length of the target sequence cannot be known at inference time, therefore, a model should predict it instead.
We show how BLEU scores can differ in WMT’ translation tasks with and without the target length prediction in Figure 3a-3c.
D E FFECTS OF U NROLLED L OGITS L OSSES
We show how varying the number of unrolled logits can affect the MT performance in Figure 3d.
E T RANSLATION ANALYSIS
We show how our algorithm works step-by-step for translation in Table 6.
It is interesting to observe how multimodality problem (Gu et al., 2017) gets resolved with more steps.
Multimodality usually manifests in repeated tokens in translation, which intrinsically comes from inability to coordinate decisions when independently sampling multiple tokens at once.
However, after a few steps all repetitions are corrected, because the model conditions on the previous step and hence can better coordinate further changes.
Fig.
4 shows how the model and the BLEU scores evolve during sampling for various temperatures.
The model score here refers to the the cross entropy between translation and model logits produced by giving this translation as the input.
While for very low temperatures the scores initially improve faster, they are eventually outperformed by scores for higher temperatures, possibly due to lack of diversity induced by near-deterministic sampling.
For very high temperatures, on the other hand, the scores improve slowly, and hence using such temperatures is rather impractical.
30 Sampling Step 0.
M o d e l S c o r e reference (a) Model score evolution throughout sampling.
30 Sampling Step
B L E U 0.
S a m p li n g
T e m p
e
r a t u r e
[Figure]
F S ACRE BLEU SCORES ON WMT’14 EXPERIMENTS
SacreBLEU (Post, 2018) is a library for computing BLEU score 5 that does not require the user to manually tokenize the reference and candidate translations.
We report BLEU scores measured with
SacreBLEU library in order to provide a reference for future studies.
We show all the BLEU scores in Table
[Table]
G P SEUDOCODE OF SUNDAE
We provide a Python-like pseudocode for our method in Listing 1.
The main functions are build loss fn , which should be used for training, and sampling fn which should be used for sampling.
For simplicity, we consider decoder-only setup as in unconditional generation experiments (Section 3.2).
1315 def build_logits_fn(vocab_size, n_unrolled_steps, enable_sampling): 16def logits_fn(batched_text): model =
Transformer(vocab_size, use_causal_mask=
False) 19def fn(input_batched_text): logits = model(input_batched_text) return logits 23def unrolled_fn(input_batched_text): samples = corrupt_text(input_batched_text, vocab_size) all_logits =
[] for _ in range(n_unrolled_steps): logits =
fn(samples) samples = stop_grad(rand_categorical(logits)) all_logits +=
[logits] final_logits = concatenate(all_logits, axis=0) return final_logits 33if
enable_sampling: return fn(batched_text) else: return unrolled_fn(batched_text) 38return
logits_fn 4042 def build_loss_fn(vocab_size, n_unrolled_steps=2): logits_fn = build_logits_fn
( vocab_size, n_unrolled_steps, enable_sampling=
False) 45def loss_fn(batched_text): logits =
logits_fn(batched_text) # repeat batched_text to fit unrolled logits targets = concatenate([batched_text] * n_unrolled_steps, axis=0) one_hot_targets
= one_hot(targets, vocab_size) loss_per_token = -sum
( one_hot_targets * log_softmax(logits), axis=-1) loss = mean(loss_per_token) return loss 55return loss_fn 5759 def sampling_fn(logits_fn, steps, temperature, batch_size, sequence_length, vocab_size): batched_text = get_random_text( shape=[batch_size, sequence_length], vocab_size) for _ in range(steps): logits =
logits_fn(batched_text) samples = rand_categorical(logits / temperature)
batched_text = samples return batched_text
Listing 1: Pseudocode of SUNDAE.