P ARALLEL S CHEDULED S AMPLING
Anonymous authors
Paper under double-blind review
A BSTRACT
Auto-regressive model architectures are the workhorse of sequence generation tasks in text, images, audio, and more.
These models are predominantly trained with teacher-forcing , wherein the model is encouraged to predict the next ground-truth token conditioned on all previous ground-truth tokens in the sequence. At test time, the model does not have access to ground-truth tokens and must instead condition on its own prior predictions, resulting in a train-test discrepancy known as exposure bias (Ranzato et al., 2015).
Scheduled Sampling (Bengio et al., 2015) aims to mitigate exposure bias by randomly replacing some tokens with model predictions during the course of training.
Unfortunately, this technique is inherently sequential and unlike teacher-forcing, cannot be trivially parallelized across time.
In this paper, we introduce a simple technique to parallelize Scheduled Sampling, signiﬁcantly increasing training throughput.
Experimentally, we ﬁnd the proposed technique leads to equivalent or better performance on image generation, summarization, dialog generation, and translation compared to teacher-forcing.
In dialog response generation, Parallel Scheduled Sampling achieves a 1.6 BLEU score (11.5%) improvement over teacher-forcing with more than 250 times higher throughput than Scheduled Sampling. In image generation, our approach achieves 20% and 13.8% improvement in Fréchet Inception Distance (FID) and Inception Score (IS) respectively.
Finally, we discuss the effects of different hyper-parameters associated with Scheduled Sampling on the model performance.
I NTRODUCTION
Auto-regressive models are a popular choice for generating sequences of any kind including audio (van den Oord et al., 2016b), images (van den Oord et al., 2016a), and text (Sutskever et al., 2014;
Cho et al., 2014).
Here, the joint probability of the sequence is factorized in a pre-determined order during train and test time.
For example, auto-regressive models for text generation factorize the joint probability left-to-right.
The text sequence is generated by and widely used in text generation tasks such as summarization
Liu
Scheduled Sampling (Bengio et al., 2015) aims to mitigate the discrepancy between train and test time in teacher-forcing by randomly replacing some tokens in the history with the model’s prediction.
More concretely, at a given time step in generating the output sequence, the model is conditioned either on ground-truth or model prediction from the previous time-step with some probability.
The probability of selecting model predicted token is gradually increased as training progresses.
This procedure potentially allows the model to recover from its own errors, and Bengio et al.
(2015) observe better empirical performance in natural language parsing, image captioning, and speech recognition compared to teacher-forced training.
Scheduled Sampling has also been used to get better performance in other tasks such as video prediction (Finn et al., 2016), knowledge base completion (Miwa and Bansal, 2016) and piano music transcription (Sigtia et al., 2016).
A key bottleneck in training models with Scheduled Sampling is its inherently sequential nature.
Unlike teacher-forcing, tokens must be processed one time-step at a time.
The sequential procedure makes Scheduled Sampling impractical for training neural networks, particularly on problems involving long sequence generation.
In this work, we describe a simple technique to parallelize
Scheduled Sampling.
Given an input example, we ﬁrst generate an entire model prediction sequence in parallel by conditioning on ground-truth history (equivalent to forward-pass of teacher-forcing).
Then, we employ a (parallel) mixing step where we generate a new sequence whose token at every time step is either from the model prediction or the ground-truth.
Finally, we perform training as in teacher-forcing by conditioning on the sequence obtained from mixing.
In Section 2.3 we show that by performing multiple passes parallel prediction mixing, we a that to a sample decode from the model.
Our key contributions are,
• A novel, fully parallelizable approach to reducing train-test discrepancy of auto-regressive models without modiﬁcation.
We believe this to be the ﬁrst approach that does not require full-sequence decoding within the training loop.
We ﬁnd the method produces the same empirical beneﬁts of Scheduled Sampling while using as little as 0.3% of the training time .
• We prove equivalence of the proposed approach to Scheduled Sampling under certain choices of hyperparameters.
This recovers the clear interpolation between training-time teacher-forcing and test-time decoding described in Bengio et al.
• We extensively evaluate our approach on four auto-regressive tasks in text and image domains.
We ﬁnd that Parallel Scheduled Sampling matches Scheduled Sampling’s beneﬁts, takes massively less compute time, and signiﬁcantly improves performance compared to teacher-forcing on most tasks.
In dialog response generation task, Parallel Scheduled Sampling achieves 1.6 BLEU score (11.5%) improvement over teacher-forcing while in image generation it achieves 20% and 13.8% improvement in Frechet Inception Distance (FID) and Inception Score ( IS) respectively.
M ETHOD
Our proposed technique can be applied to both conditional and unconditional auto-regressive generative models.
For notational simplicity, we consider the task of conditional sequence generation.
The training set is given in terms of N input-output sequences { ( x i , y i ) } ni =1 , where x i is the input and target y
i is the desired output.
The target y
i is a variable-length sequence of T i tokens (or pixels), ( y i 1 , y i 2 , . . .
, y iT i ) , whereas x i may be variable-length (as in translation) or ﬁxed-length (as in image captioning).
The goal is to learn a model that accurately predicts y i given x i .
We use y 1: t to denote the sequence of tokens ( y 1 , y 2 , . . .
, y t ) .
2. T EACHER -F ORCING AND D ECODING
Given an input x and a target y , the log-probability of the target can be decomposed autoregressively:
[Equation]
Auto-regressive sequence generation models learn to assign high likelihood to token y t given previous target tokens y 1: t − 1 = ( y 1 , . . .
, y t − 1 ) and inputs x via a learned likelihood model p θ .
Neural language models such as RNNs (Mikolov et al., 2010) and Transformer (Vaswani et al., 2017) adopt left-to-right decomposition while image generation models (van den Oord et al., 2016a; Parmar et al., 2018) adopt a raster-scan order.
Such models are typically trained with teacher-forcing (Williams and Zipser, 1989).
In teacherforcing, the log likelihood of the training set is directly maximized,
[Equation]
Importantly, teacher-forcing conditions on gold target preﬁxes y 1: t − 1 , enabling backpropagation through all timesteps with a single pass of inference.
At inference time, beam or sample decoding is often used to generate a candidate target ˆ y i .
In this regime, target tokens are generated one at a time while conditioning on previously-generated tokens.
[Equation]
A potential failure mode for teacher-forcing-trained models is in conditioning on previously unobserved target preﬁxes ˆ y 1: t − 1 .
As the model has not conditioned on these preﬁxes at training time, it may generate bland, repetitive, or nonsensical candidate targets (Holtzman et al., 2019).
2. S CHEDULED S AMPLING
Scheduled Sampling (Bengio et al., 2015), hereafter Sequential Scheduled Sampling is a training technique designed to bridge the gap between teacher-forcing and sample decoding.
In its simplest form, Sequential Scheduled Sampling generates tokens ˜ y 1: t and conditions on these target preﬁxes during training.
Sequential Scheduled Sampling uses the same objective function as teacher-forcing (Equation 1) except the conditioning tokens ˜ y 1: t are a random mixture of gold tokens y 1: t and sampled tokens ˆ y 1: t instead of gold tokens y 1: t .
See Algorithm 1 for implementation.
Algorithm[-
 Sequential Scheduled Sampling (single example)-] for all timesteps t = 1 , . .
, T i do
Sample ˆ y t ∼ p θ (ˆ y t | ˜ y 1: t − 1 , x )
Choose next conditioning token, ˜ y t
= (cid:26) y t with probability 1 − p ˆ y t with probability p end for return Accumulate loss (cid:80) t log p θ ( y t | ˜ y 1: t − 1 , x )
As p → 0 , we condition on y 1: t − 1 as in teacher-forcing, and as p → 1 , we condition on ˆ y t : t − 1 as in sample decoding.
Typically a schedule will be used to gradually increase p over the course of training.
As illustrated in Bengio et al.
Scheduled Sampling leads to a performance improvement in a variety of language generation tasks.
In spite of its beneﬁts, Sequential Scheduled Sampling is inherently a sequential algorithm: choosing conditioning token
˜ y t requires conditioning autoregressively on tokens ˜ y 1: t − 1 .
While this is natural for sequential architectures such as RNNs and LSTMs, it is poorly suited to self-attending feed-forward models such as Transformer where inference for multiple timesteps can be carried out simultaneously.
2. P ARALLEL S CHEDULED S AMPLING
We propose a natural extension to Sequential Scheduled Sampling called Parallel Scheduled Sampling .
Whereas Sequential Scheduled Sampling selects conditioning tokens one after another, we propose generating conditioning tokens for all timesteps in parallel over the course of one or more passes.
While this technique requires strictly more operations than Sequential Scheduled Sampling, it is better suited to hardware accelerators such as GPUs and TPUs (Jouppi et al., 2017).
Moreover, we ﬁnd in our experiments that only a modest number of passes is necessary for improving model performance.
Parallel Scheduled Sampling generates conditioning tokens for all timesteps simultaneously .
The procedure consists of multiple passes, each pass consisting of parallel sampling and mixing steps (Figure 1).
In the ﬁrst pass, the algorithm conditions on gold tokens y 1: t , generating tokens ˆ y 1: t i.i.d.
according to p θ (ˆ y t | y 1: t − 1 , x ) .
Sampling tokens in the ﬁrst pass is equivalent to the forward-pass of teacher-forcing.
The sampled tokens, ˆ y 1: t , are mixed (in parallel) with gold tokens, y 1: t , to produce conditioning tokens for the next pass, ˜ y 1: t .
We now describe the multiple-pass procedure.
Let ˆ y k, 1: t , and ˜ y k, 1: t denote sampled and mixed tokens respectively on pass k .
The mixed tokens from pass k , ˜ y k, 1: t , are used for conditioning on pass k +in place of gold tokens y 1: t .
Finally, the loss is calculated as before, conditioning on the ﬁnal mixture of gold and sampled tokens
˜
y K, 1: t .
See Algorithm 2 for implementation.
Set ˜ y 0 ,t = y t .
for all passes k = 1 , . . .
, K do for all timesteps t in parallel do Sample ˆ y k,t ∼ p θ (ˆ y k,t | ˜ y k − 1 , 1: t − 1 , x ) .
if t < k then
Copy ˜ y k,t = ˜
y k − 1 ,t else
Sample ˜ y k,t = (cid:26) y t with probability 1 − p
ˆ y k,t with probability p end if end for end for return Accumulate loss (cid:80) t log p θ ( y t | ˜ y K, 1: t − 1 , x )
Finally, we prove that by running the sampling and mixing steps for multiple passes as described in Algorithm 2, the ﬁnal sample from Parallel Scheduled Sampling converges to a random sample decode from the model when p = 1 and K ≥ T .
Theorem 2.1.
Consider a sequence of tokens
z
=
( z 1 , z 2 , . . .
, z T ) of length T .
Let p = 1 and K ≥ T be ﬁxed.
Then the likelihood of z 1: T under Parallel Scheduled Sampling’s proposal distributionover conditioning tokens on pass K , q Kθ ( z 1: T ) , is identical to random sample decoding’s, p θ ( z 1: T ) , q Kθ ( z 1: T )
= p θ ( z 1: T ) .
Proof.
We begin by establishing notation.
Let p θ ( z 1: T ) be the likelihood of a sequence z 1: T according to random sample decoding.
Let q Kθ ( z 1: t ) be the likelihood of the same according to Parallel
Scheduled Sampling’s proposal distribution on pass K .
The proof proceeds by induction.
First we show that the proposal distribution for the ﬁrst token matches random sampling’s on the ﬁrst pass, q 1 θ ( z 1 ) =
p θ ( z 1 ) .
Then we show that if q Kθ ( z 1: t ) =
p θ ( z 1: t ) holds for some K , it also holds for all K (cid:48) >
K .
Finally, we show that if the previous statement holds, it also holds for tokens z 1: t +1 on pass K + 1 .
Thus, it follows that the proposal distribution matches random sampling’s for all T tokens so long as K ≥ T .
Base Case : Consider the proposal distribution for the ﬁrst token on the ﬁrst pass, z 1 .
As p = 1 , the ﬁrst token is sampled from p θ ( z 1 ) by construction.
Thus,
[Equation]
Induction over K : Suppose that the proposal distribution for tokens q Kθ ( z 1: t [Equation]
= p θ ( z 1: t ) some K ≥ t .
Then the equality also hold for the proposal distribution on pass K + 1 .
This follows trivially as tokens z 1: t are “copied” from pass K to K + 1 and thus their likelihood is unchanged,
[Equation]
Induction over t : Suppose that the proposal distribution matches random sample decoding’s for the ﬁrst t tokens for t =
K ; that is, q Kθ ( z 1: t ) =
p θ ( z 1: t ) .
We show that the statement holds for pass
K + 1 for tokens z 1: t +1 .
First, recall that by construction the proposal distribution for token z t + given previous tokens z 1: t is the same as random sampling’s when t ≥ K ,
[Equation]
Note that this only holds when p = 1 .
Then,
[Equation]
Where we use the chain rule, induction over K for z 1: t , the inductive assumption for q Kθ ( z 1: t ) , and the deﬁnition of q K +
θ ( z t +1 | z 1: t ) when t ≥ K .
(cid:4)
R ELATED W ORK
A variety of approaches have been introduced since Bengio et al.
(2015) to eliminate the train-test discrepancy also known as exposure bias .
Goyal et al.
(2017) integrate a differentiable approximation to argmax into the Scheduled Sampling procedure.
Ranzato et al. (2015) propose MIXER, which ﬁnetunes a model trained with teacher-forcing with REINFORCE to maximize BLEU on a validation set. Zhang et al. (2019) propose a procedure similar to Scheduled Sampling but with an alternative method for selecting the model’s next-token prediction.
Both works inherently require full-sequence decoding within the training loop, making them impractical to apply in domains with long generations.
Concurrently, Mihaylova and Martins (2019) proposed a variant of Scheduled Sampling similar to the approach introduced in this work.
In their work, only a single pass K = 1 is applied with the ﬁrst pass conditioning on a mixture of input embeddings.
Unlike this work, the proposed approach is not equivalent to Scheduled Sampling under any choice of hyperparameters.
Lamb et al.
(2016) introduce an alternative approach with similar motivation.
A discriminator network is trained jointly with the generator to distinguish between generator’s hidden states produced by conditioning on ground-truth and model prediction sample.
The generator, apart from maximizing the likelihood of the data, is also trained to fool the discriminator (Goodfellow et al., 2014).
With this new objective, the dynamics of the generator would be the same for conditioning on both ground-truth and model prediction.
Our parallel sampling approach is orthogonal to professor forcing and can be potentially applied in their framework.
Another approach with similar motivation is proposed in Collins and Roark (2004). Rather than a maximum likelihood objective, the authors apply a variant of the Perceptron algorithm (Rosenblatt, 1958) which penalizes the model’s most likely decode if it doesn’t exactly match the target. In the time-series modeling domain, Venkatraman et al. (2015) propose DAD, wherein the training set for a Markov dynamics model is augmented by pairing model predictions with ground-truth states.
This method is similar in spirit to the approaches above except in its Markov assumptions.
Further, methods with similar motivation have also been studied in sequential decision making and reinforcement learning setting (Daumé et al., 2009; Ross et al., 2011). While our focus is on accelerating mitigation strategies for exposure bias, another line of work aims to accelerate sequential decoding. Stern et al. (2019) generate successively longer decodes by inserting tokens between pairs of already-decoded tokens.
Roy et al. (2018) and Kaiser et al.
(2018) generate tokens in parallel by additionally conditioning on a short sequence of discrete latent variables.
Lee et al.
(2018) apply an iterative reﬁnement technique by performing multiple passes of parallel decoding.
E XPERIMENTS
We evaluate our proposed technique on image and text domains.
In the text domain, we evaluate
Parallel Scheduled Sampling on text summarization (Liu et al., 2018), task-oriented dialog response generation (Budzianowski et al., 2018), and machine translation (Sutskever et al., 2014; Vaswani et al., 2017) and compare it to teacher-forced training.
We further evaluate the proposed technique on image generation on CIFAR-10.
Since our procedure is intended to generate better sequences, we evaluate it at the sequence level and not at the word token or pixel level.
We compare our method to Sequential Scheduled Sampling only on the dialog task (Budzianowski et al., 2018) as we ﬁnd runtime infeasible on all other tasks.
We use the Tensor2Tensor framework for all experiments (Vaswani et al., 2018).
4. D IALOG R ESPONSE G ENERATION
We evaluate our method on dialog response generation task using MultiWOZ (Budzianowski et al., 2018), a task-oriented dialog dataset.
Here, we consider the problem of mapping conversation history consisting of alternating user and assistant turns to a single turn of assistant response.
We use a
Transformer model containing approximately one million parameters for this study as the dataset is much smaller (approximately 100k training examples) than those in other experiments.
We truncate the length of the input and output to 512, and train all the models for 50k steps.
As both model and dataset are small, we are able to empirically compare our method to Sequential Scheduled Sampling (such experiments are infeasible in larger models).
Table 1 summarizes results for all experiments on the MultiWOZ dataset.
Both Sequential Scheduled Sampling and Parallel Scheduled
Sampling (with just one pass) achieve better results than teacher-forced trained models.
However, as can be seen in Table 1, Parallel
Scheduled Sampling and teacher-forcing are both two orders of magnitude faster to train than
Sequential Scheduled Sampling.
A single pass of Parallel Scheduled Sampling is approximately 25% slower than teacher-forced training while producing the beneﬁts of Sequential Scheduled Sampling.
Table 1 also shows the impact of mixing probability, number of passes, warm-up steps, and the mixing probability schedule (Bengio et al., 2015) on model performance.
Overall, we ﬁnd a single pass with 50% gold/sampled mixing probability sufﬁcient for improving performance.
In the best setting,
Parallel Scheduled Sampling achieves 1.6 BLEU score (11.5%) improvement over teacher-forcing.
4. S UMMARIZATION
Liu et al.
(2018) propose a multi-document summarization task, where the task is to generate the text of a Wikipedia article given its references and other related documents.
The dataset has close to 1.
million training examples, and 230,000 test examples.
We use a Transformer seq2seq model for this task in two hyper-parameter settings: a base model with 60 million parameters and a large model with 210 million parameters.
For the base model, we restrict the maximum length of input and output to be 500, while for the large model the maximum length is set to 1500.
Table 2 shows the results of training base and large Transformer models for the summarization task.
The base and large models were trained for 250k steps and 500k steps respectively.
We use teacherforcing for the ﬁrst 50% of training steps in Parallel Scheduled Sampling as warm-up steps.
The mixing probability is set to 50% and we perform a single pass of sampling and mixing (Algorithm 2). [Table] [Table] [Table]
[Table] [Table] - 25k [Table]
[Table]
[Table]
- 25k
[Table]
[Table]
[Table]
- 25k exp 12.
[Table]
[Table]
- 25k
[Table]
[Table]
[Table]
25k
[Table]
[Table]
[Table]
25k
[Table]
[Table]
[Table]
[Table]
25k
[Table]
[Table]
[Table]
25k
[Table]
[Table]
25k
[Table]
[Table]
[Table]
25k
[Table]
[Table]
[Table]
25k exp 14.
[Table]
[Table]
25k exp 14.
[Table]
[Table]
25k exp 14.
[Table]
10k exp 14.
[Table]
15k exp 14.
[Table]
20k exp 14.
[Table]
[Table]
30k exp 14.
[Table]
[Table]
35k
[Table]
[Table]
[Table]
With the base model
[Table]
Parallel Scheduled Sampling obtains better performance than teacher-forcing with both beam search and greedy decoding while it performs better only with greedy decoding when the large model is used.
Since we use models that are much bigger than the ones in the dialog task discussed before [Table] it is runtime infeasible to apply Sequential Schedule Sampling here.
4. I MAGE G ENERATION
In the image domain [Table] we evaluate our method for image generation on the CIFAR-10 dataset.
We compare class-conditional Image Transformer (Parmar et al., 2018) trained with teacher-forcing and
Parallel Scheduled Sampling.
After training [Table] we decode a total of 50,000 randomly-sampled images conditioned on classes drawn from the training set.
In additional to a baseline model, we compare all metrics to ground truth samples from the CIFAR-10 training set.
We evaluate the image samples using Frechet Inception Distance (FID) (Heusel et al., 2017) and Inception Score (IS) (Salimans et al., 2016) metrics.
These metrics have been widely used to evaluate the quality of image samples from
GANs (Lucic et al., 2018; Miyato et al., 2018; Karras et al., 2018; Zhang et al., 2018).
Table 3 compares teacher-forcing with Parallel Scheduled Sampling on image generation.
We train the 50 million parameter Image Transformer model for 200K steps in both the cases.
We ﬁnd that Parallel
Scheduled Sampling with a single pass and a mixing probability of 50% signiﬁcantly decreases
FID by 20% and increases IS by 13.8% compared to the baseline.
Similarly to our summarization experiment, it is runtime infeasible to apply Sequential Schedule Sampling here.
4. M ACHINE T RANSLATION
We evaluate our method on the WMT 2014 English-German task which consists of approximately 4.5 million training sentences.
We experiment with the large Transformer model that contains approximately 210 million parameters.
We did not see performance improvements by using Parallel
Scheduled Sampling.
The model trained with teacher-forcing for 500k steps gets 28.74 BLEU.
The same model trained with 250k warm-up steps using teacher-forcing and the next 250k steps trained with Parallel Scheduled
Sampling with mixing probability set to 50% and a single pass of sampling and mixing (Algorithm 2) obtains 28.57 BLEU.
Hyper-parameter tuning of warm-up steps and mixing probability did not improve performance.
We hypothesize the lack of performance improvement may be due to the fact that the summarization, dialog response generation and image generation tasks have much longer output sequences than in machine translation, though further investigation is required.
C ONCLUSION
We introduce a simple technique to parallelize Scheduled Sampling that allows Schedule Sampling to be applied for training models with hundreds of millions of parameters on large datasets.
The technique potentially mitigates discrepancy between train and test time in auto-regressive sequence generation models.
We ﬁnd that in most cases our technique leads to better empirical performance on summarization, dialog generation, and image generation compared to teacher-forced training.
Our empirical results indicate that Parallel Scheduled Sampling can potentially improve the performance of auto-regressive sequence generation models particularly on tasks containing long sequences.