T ea F or N : Teacher-Forcing with N-grams (2020)

Abstract: Sequence generation models trained with teacher-forcing suffer from issues related to exposure bias and lack of differentiability across timesteps. Our proposed method, Teacher-Forcing with N-grams (TeaForN), addresses both these problems directly, through the use of a stack of N decoders trained to decode along a secondary time axis that allows model-parameter updates based on N prediction steps. TeaForN can be used with a wide class of decoder architectures and requires minimal modifications from a standard teacher-forcing setup. Empirically, we show that TeaForN boosts generation quality on one Machine Translation benchmark, WMT 2014 English-French, and two News Summarization benchmarks, CNN/Dailymail and Gigaword.

## introduction many state-of-the-art sequence generation models are trained using a technique called teacherforcing @xcite . teacher-forcing is popular because it improves sample efficiency and provides training stability, but models trained with teacher-forcing are known to suffer from issues such as exposure bias @xcite @xcite and a lack of differentiability across timesteps (i.e., training updates made when decoding at time-step t cannot fully propagate to time-step t -1). previous attempts to address these issues include scheduled sampling @xcite , parallel n-gram prediction @xcite , and sampling from previous predictions @xcite . our proposed method, teacher-forcing with ngrams (teaforn), imposes few requirements on the decoder architecture and does not require curriculum learning or sampling model outputs. teaforn fully embraces the teacher-forcing paradigm and extends it to n-grams, thereby addressing the problem at the level of teacher-forcing itself. the advent of large-scale pretraining has pushed the state-of-the-art on natural language benchmarks to impressive heights, often showing gains across many tasks at once @xcite @xcite . a negative consequence of this is the tendency towards large, data-hungry models, which have a negative impact on energy-consumption and accessibility @xcite , as well as higher latency and production costs. as such, it is of increasing importance to develop techniques that counteract these tendencies. while teaforn does increase training cost moderately, it can be used to drive down latency and inference cost, which dominate the overall cost of a production model. many sequence generation models use beam search to improve generation quality @xcite @xcite . in contrast with greedy decoding, beam search keeps the k most-likely candidates at each decoding timestep. while beam search has proven to be a reliable technique for improving output quality, previous work has shown that beam search actually degrades performance for sufficiently large k @xcite . in addition, the inference cost of a model increases linearly with k, due to the need for multiple decodings. we conduct an analysis of the effect of beam size on models trained both with and without teaforn. we show that models trained with teaforn require a smaller beam size to reach similar performance, a property that can achieve significant cost-savings. our experiments show that teaforn can boost performance on both machine translation and news summarization tasks, provided there is sufficient model capacity. with teaforn, transformer big @xcite improves by +.5 sacrebleu @xcite on the wmt14 en-fr benchmark with beam search and +.3 without. when using teaforn for summarization, pega-sus large @xcite improves by +.3 rouge-l on the gigaword benchmark @xcite and by +.2 on the cnn/dailymail benchmark @xcite . further, pe-gasus large trained with teaforn matches the prior rouge-l scores on these benchmarks without beam search, representing an 8x reduction in decoder inference cost. ## related work one of the standard approaches to sequencelearning training is maximum-likelihood estimation (mle). although widely used in large array of applications, mle estimation for sequence learning suffers from the exposure-bias problem @xcite . exposurebias produces brittle models due to training procedures during which the models are only exposed to their training data distribution but not to their own predictions. possible solutions to the exposurebias problem in neural-network settings have used "data as demonstrator" @xcite and "scheduled sampling" @xcite approaches. although improving model performance in practice, such proposals have been shown to be statistically inconsistent @xcite , and still need to perform mle-based warm-start training, rendering such solutions unsatisfactory. along similar lines, the "professor forcing" @xcite ) method uses adversarial domain adaptation to encourage network dynamics to be the same during training and inference, though it requires sampling sequences during training. a different approach, based on reinforcement learning methods, achieves sequence learning following a policy-gradient (pg) method @xcite . it directly attacks the exposure-bias problem by having the training models exposed exclusively to their own predictions while scoring them using reward functions. however, this approach introduces another issue, related to the large discrepancy between the model prediction distribution and the reward function's values, which is especially acute during the early training stages when the predicted outputs are all equally bad. as a result, this method also requires a warm-start phase in which the model distribution achieves some local maximum with respect to a reward-free objective (e.g., mle), followed by a model refinement phase in which reward-based pg updates are used to refine the model @xcite @xcite . although such combinations achieve better results in practice compared to pure likelihood-based approaches, they are unsatisfactory from a theoretical and modeling perspective, as well as inefficient from a speed-to-convergence perspective. a pure pg formulation that side-steps these issues is @xcite , which allows for both cold-start training as well as more efficient convergence properties. the pg-based approaches have an inherent complexity that stems from the use of quirky reward functions such as rouge @xcite or cider @xcite , which forfeits the advantage of sample efficiency as they often cannot be efficiently computed using current accelerators like tpus @xcite . mle-based approaches appear to be favored due to efficiency properties, and the search for training methods that produce less brittle models is still on-going. another closely related idea is end-to-end backprop (e2e) @xcite , which has a similar goal of naturally approximating sequence level training by propagating smooth model predictions instead of groundtruth inputs. teaforn differs from e2e in several key ways. first, teaforn learns jointly from both groundtruth and model predictions as inputs throughout the entire training duration, whereas e2e requires a training schedule to transition from groundtruths to model predictions. second, teaforn supports methods other than k-max for computing smooth model predictions, two of which we explore as a part of our work. third, we introduce the notion of a discount factor, which weights the importance of immediate predictions higher than that of future predictions. another such work is @xcite , which proposes a modified transformer for parallel ngram prediction. while their work does address the issue of strong local correlations caused by teacherforcing, it does not address exposure bias, as it always trains on groundtruth inputs. also related are models such as the one proposed by @xcite , which uses a stacked of dilated convolutions to iteratively refine model predictions. though architecturally similar, teaforn only uses the stack at training time and solves for a fundamentally different problem. go, y (1) gt , ..., y (t-1) gt embedding lookup input embeddings x (1) 0 , ..., x (t) 0 timing signal pos(1), ..., pos(t) decoder-0 output embeddings o (1) 0 , ..., o (t) 0 logits/loss l0 timing signal pos(2), ..., pos(t + 1) input embeddings x (1) 1 , ..., x (t) 1 decoder-1 output embeddings o (1) 1 , ..., o (t) 1 logits/loss λl1 groundtruth prefix @xmath0, ..., y (t+1) gt targets y (1) gt , ..., y (t) gt dressing both exposure bias and the issue of differentiability across timesteps. in addition, it is general enough to be used on a wide class of autoregressive decoders, including rnn @xcite and transformer @xcite decoders, though our experiments focus on the transformer. ## teaforn autoregressive sequence decoders are trained to minimize the negative log likelihood of the groundtruth tokens y (t) gt . during training, previous groundtruth tokens are used as decoder inputs for predicting the next token. if we define the embedding matrix to be e of size v × d, where v is the vocabulary size and d is the embedding size, and the embedding of the groundtruth token as the class probability distribution p is typically modeled as softmax-normalized logits, which are a linear projection of decoder output o (t) of size d onto the class embeddings using output projection matrix w of size v × d: to reduce the model parameter size, it is standard to share the parameters of the output projection matrix and the embedding matrix, such that @xmath1. during inference, groundtruth tokens become unavailable. therefore, previous tokens from the model predictions are used as decoder inputs for decoding the next token. the discrepancy between training time and inference time input distributions causes models to suffer from exposure bias, meaning that they do not learn to correct for past decoding errors @xcite . teaforn addresses exposure bias by learning jointly how to predict from both groundtruth and past model predictions as inputs. teaforn setups consist of a stack of n decoders, as illustrated in figure 1 . at position t, the first decoder (decoder-0) takes input from the embedding of the previous groundtruth token and learns to predict the target token y (t) gt , same as in teacher-forcing. the next decoder (decoder-1) takes input from the output the training loss of decoder-s at time t is the negative log likelihood of the (t + s) th element in the groundtruth sequence: the intuition behind teaforn is as follows. under standard teacher-forcing, the decoder output o (t) only learns to predict the groundtruth label y (t) gt , while outputs that favor other classes are considered equally bad, and will be penalized by the loss. this is not reasonable because classes carrying similar meanings to the groundtruth label do not change the meaning of the sequence significantly, and may still lead to the correct prediction for the next label. under teaforn, the decoder output o (t) is also used as the input of a secondary decoder for decoding the next position. therefore, all outputs that result in predicting the next groundtruth label y (t+1) gt will have lower loss and therefore be differentiated from other outputs. in our experiments, we allow the decoder parameters to be either shared (θ 0 = θ s , ∀s) or unshared. in a shared-weight configuration, the model learns to predict the next groundtruth label from the class that the same model predicted in the previous position. this is similar to the inference time condition, so we expect shared-weight teaforn to address exposure-bias better than unshared-weight teaforn. shared-weight configurations also have performance advantages such as lower memory consumption and faster training. since teaforn solves for a more difficult problem than teacher-forcing, we expect it to work better for models with higher capacity. we later show evidence of this by comparing results for two model sizes on machine translation. it is straightforward to show that teafor1 (@xmath2, as the inputs to the first teaforn decoder are groundtruth sequence embeddings and λ 0 = 1. thus, teaforn is a natural extension of teacher-forcing to n-grams. ## experimental results our empirical study of teaforn is comprised of two sections. first, we present experiments on machine translation using the well-known transformer model @xcite . second, we show results for news summarization, for which we use pegasus @xcite , a state-ofthe-art pretrained text summarization model. we perform minimal hyperparameter tuning over the course of these experiments. this can be partly credited to the underlying models being well-tuned already, but also to teaforn, which works out-ofthe-box without much hyperparameter tuning. one exception is the tuning of the number of training steps, as we found that the number of steps used by previous settings is sometimes insufficient. ## conclusion in this work, we introduce a new technique for sequence generation models called teacher-forcing with n-grams (teaforn), which (a) addresses exposure bias, (b) allows the decoder to better take into account future decisions, and (c) requires no curriculum training. we show empirical evidence of the efficacy of teaforn on several sequence generation tasks. with transformer big @xcite , we boost the performance of transformer big significantly on the en-fr benchmark. with pega-sus large @xcite , we improve upon the existing rouge-l scores the gigaword and cnn/dailymail benchmarks @xcite . further, we show that teaforn can match the prior state-of-the-art rouge-l scores on the summarization benchmarks without beam search, representing an 8x reduction in decoder cost at inference.