T EXT G ENERATION BY L EARNING FROM D EMONSTRATIONS
A BSTRACT
Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation.
This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objective and evaluation metric (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated).
To alleviate these problems, we frame text generation as a reinforcement learning (RL) problem with expert demonstrations (i e., the training data) where the goal is to maximize quality given model-generated histories.
Prior
RL approaches to generation often face optimization issues due to the large action space and sparse reward.
We propose GOLD (generation by off-policy learning from demonstrations): an easy-to-optimize algorithm that learns from the offpolicy demonstrations by importance weighting.
Intuitively, GOLD upweights conﬁdent tokens and downweights unconﬁdent ones during training.
According to both automatic and human evaluation, models trained by GOLD outperform those trained by MLE and policy gradient on summarization, question generation and machine translation.
Further, they are less sensitive to decoding algorithms and alleviate exposure bias
I NTRODUCTION
A dominant approach to text generation is to use autoregressive models learned by maximum likelihood estimation (MLE) on supervised data.
However, this approach introduces two well-known discrepancies between training and evaluation objectives that lead to undesired generations.
First, the training loss is negative log-likelihood, whereas the evaluation is based on human judgment of the output quality.
Under model misspeciﬁcation, MLE tends to over-generalize, assigning large probability mass to both high-quality and low-quality sequences (Husz ´ ar, 2015; Simon et al., 2019).
Therefore, in practice, we must carefully select the decoding algorithms to produce high-quality outputs.
Second, during training, the autoregressive model conditions on the gold history/preﬁx; however, at inference time it conditions on model-generated history.
This is known as the exposure bias problem (Ranzato et al., 2016; Bengio et al., 2015).
In the worst case, one incorrect prediction can produce a low-probability preﬁx under the gold data distribution, and errors compound in each of the following steps (Ross et al., 2011).
In practice, prior work has observed problems such as repetition and hallucination partly due to exposure bias (Holtzman et al., 2020; Wang & Sennrich, 2020).
We aim to bridge the gap between training and evaluation in this paper.
To match training and evaluation objectives, ideally we should maximize output quality given model-generated histories.
This corresponds to the reinforcement learning (RL) objective: maximizing the expected reward (quality) over trajectories (sequences) induced by the policy (model).
However, optimizing this objective is notoriously difﬁcult.
Prior RL approaches mainly focus on ﬁne-tuning a learned model to optimize sequence-level metrics such as BLEU, but empirically it remains unclear if RL is beneﬁcial to text generation (Wu et al., 2018;
Choshen et al., 2020).
Note that many challenges in RL arise from exploring an exponentially large space of sequences with sparse rewards only on those close to the reference.
We thus propose to learn from only the reference sequences without interaction (i.e., the ofﬂine setting).
Speciﬁcally, we use off-policy policy gradient with importance weighting
(Hastings, 1970; Hachiya et al., 2009; Parshakova et al., 2019), where training examples with higher probability under the model are weighted higher.
Further, our reward functions approximate human judgment of the output quality by estimating how likely a human would have generated a sequence.
We call our algorithm GOLD (Generation by Off-policy Learning from Demonstrations).
Results on news summarization, question generation, and machine translation show that GOLD leads to better model performance than MLE and RL ﬁne-tuning by both task metrics and human-rated quality.
Further, our analysis shows that GOLD learns high-precision models that are less sensitive to decoding algorithms.
In addition, it alleviates exposure bias: the output quality does not degrade much as generation length increases.
F ROM MLE TO RL F RAMEWORK
MLE training.
Given a context x such as a document, we want to generate a sequence of tokens y
=
( y 0 , . .
, y T ) , where y i comes from a vocabulary V .
The generator is modeled by a conditional probability distribution parametrized by θ : p θ ( y | x ) = (cid:81)
Tt =
p θ ( y t | y 0: t − 1 , x ) , where y 0:
t −denotes the preﬁx y 0 , . . .
, y t − 1 .
Let p human ( y | x ) denote the data-generating distribution.
Using
MLE, the loss function is
[Equation]
At inference time, we generate tokens sequentially according to p θ .
Evaluation.
In practice, the quality of an output often relies on task-speciﬁc metrics such as ﬂuency, correctness, and interestingness.
Here for generality we consider perceptual quality (Husz ´ ar, 2015;
Hashimoto et al., 2019) which measures how likely a human would have generated the output given the context, i.e., p human ( y | x ) .
Thus the evaluation metric is
[Equation]
Comparing (1) and (2), we see that the training objective encourages high recall : the model must put probability mass on all human-generated sequences.
In contrast, the evaluation metric encourages high precision : all outputs from the model must be of high quality.
Unfortunately, directly optimizing the evaluation metric is impossible because p human is unknown and the expectation is difﬁcult to estimate.
We therefore develop a training objective that closely approximates (2) in the RL framework.
RL formulation.
Let’s consider generation as a sequential decision-making process.
At each time step t , the policy π θ takes an action a t ∈ V , transits to the next state s t +1 =
( y 0:
t , x ) , and receives a reward r t .
The policy corresponds to the generation model: π θ ( a t | s t )
= p θ ( a t | y 0:
t − 1 , x ) .
We can thus represent a sequence as a trajectory τ =
( s 0 , a 0 , r 0 , . . .
, s T , a T , r T ) .
The set of trajectories derived from the training data is called demonstrations which show the desired behavior of a policy.
The RL objective is to maximize J ( θ ) =
E τ ∼ π θ (cid:104)(cid:80) Tt =
γ t r t (cid:105) , where γ ∈ (0 , 1] is the discount factor, and π θ ( τ ) denotes the distribution of τ induced by π θ .
If we knew oracle rewards r t = p human ( a t | s t ) , then this objective is exactly the evaluation metric we want to optimize.
Next, we describe how to optimize J ( θ ) with reward functions that approximate p human .
A PPROACH 3. O FF -P OLICY P OLICY G RADIENT
Policy gradient.
A straightforward way to optimize J ( θ ) is policy gradient (PG)
(Williams, 1992;
Sutton et al., 2000).
The gradient is given by
[Equation]
where ˆ Q ( s t , a t )
= (cid:80) Tt (cid:48)
=
t γ t (cid:48) − t r t (cid:48) is the estimated return from state s t .
The expectation is estimated by Monte Carlo samples from π θ .
In text generation, the return ˆ Q ( s t , a t ) is often a sequence-level reward such as BLEU.
In practice, the policy is likely to get stuck in a region of zero reward during training, generating gibberish without receiving any learning signal (Li et al., 2018; Keneshloo et al., 2019).
A common remedy is to initialize the policy with the MLE solution and/or interleave with
MLE gradient update during PG.
However, this would bias the parameters towards the MLE solution, thus often leads to marginal gains in practice (Wu et al., 2018;
Choshen et al., 2020).
Off-policy learning.
To avoid zero-reward regions, we would like to reduce interaction with the environment and stay close to the demonstrated trajectories .
In the extreme case, the policy is learned solely from the static demonstrations without additional interaction with the environment, which is referred to as the ofﬂine setting.
While it is in general a more challenging problem, we argue that the ofﬂine setting is appropriate for text generation (Serban et al., 2017; Jaques et al., 2019).
First, the environment dynamics is known: once a token is generated, we deterministically transition to the next state with the additional token appended to the preﬁx; no interaction is needed to learn the environment.
Second, while exploration may lead to high-quality sequences different from the reference, we lack a good reward function to identify them (Novikova et al., 2017; Aharoni &
Goldberg, 2018; Clark et al., 2019).
Therefore, the beneﬁt of exploration in text generation is limited.
In the ofﬂine setting, we cannot estimate the expected return of π θ by sampling trajectories from it, and must use trajectories from a different behavioral policy π b , known as off-policy learning in
RL.
A common technique to estimate expectations under one distribution π θ given samples from a different distribution π b is importance sampling, which leads to the following unbiased estimator of the gradient (Precup et al., 2000):
[Equation]
with importance weights
[Equation]
Approximations.
Computing the importance weights above requires multiplying per-action importance weight over multiple time steps.
In practice, we have found that it is sensitive to optimization hyperparameters and takes longer to converge.
Therefore, we use the per-action approximation: w t ≈ π θ ( a t | s t ) π b ( a t
| s t ) .
This corresponds to optimizing the expected return under the off-policy state distribution induced by π b and the on-policy action distribution of π θ .
Although this estimator is biased, empirically it has been shown to reduce variance and work reasonably well if π b and π θ are close (Serban et al., 2017; Levine et al., 2020).
Another obstacle is that we do not know π b which produced the demonstrations D
= { ( x ( i ) , y ( i ) ) }
Ni =1 .
One option is to estimate π b on D .
Here we take a simpler approach that uses the empirical distribution: π b ( τ ) ≈ 1 /N for τ ∈ D and 0 otherwise.
As a result, the denominator in w t is a constant and can be ignored in optimization.
Our ﬁnal approximated gradient has the form:
[Equation]
where the superscript i represents the i th trajectory.
Compared with the MLE gradient: (cid:80)
Ni =1 (cid:80) Tt =0 ∇ θ log π θ ( a it | s it ) , our gradient (4) upweights actions with high return and actions preferred by the current policy π θ .
Intuitively, it encourages the learning algorithm to focus on “easy” examples (high likelihood under the model) which improves precision.
3. R EWARD
Let R be the reward function such that r t =
R ( s t , a t ) .
To optimize the perceptual quality of a sequence (see (2)), we want R ( s, a ) to approximate p human ( a | s ) , i.e., how likely humans would have generated a given s .
In general, it is hard to develop a reliable reward function for text generation tasks because it must work well for a large set of possible generations.
In the ofﬂine setting, however, we can restrict the domain of R to state-action pairs on the demonstrations.
Next, we propose three reward functions.
δ -reward.
An obvious choice is a sequence-level reward, which considers all demonstrations to be equally good and assigns zero reward to any other outputs.
Formally,
[Equation]
where a reward of one is received in the terminal state for any trajectory in the demonstrations.
Estimated p human .
In text generation tasks, an input often has many correct outputs and the reference may be an uncommon output that contains rare words or has complex syntax.
To account for different likelihood of the references, we estimate the probability of each reference by minimizing
KL ( p human (cid:107) q ) , where q ( a | s ) approximates p human ( a | s ) .
This is equivalent to ﬁnding the MLE solution (denoted by p MLE ).
Importantly, p MLE is a reasonable approximation to p human when restricted to the demonstrations .
It is not a good reward function in general, however; it can assign large probability mass to low-quality outputs.
Given the estimated perceptual quality p MLE ( a | s ) , we deﬁne two reward functions.
Our ﬁrst reward function corresponds to a product of probabilities when summed over the trajectory:
[Equation]
Assuming γ = 1 , the return at time step t is ˆ Q t ( s t , a t ) =
(cid:80) Tt [Equation]
= t log p MLE ( a t | s t ) .
Thus a sequence has high reward only if every word has high likelihood under p MLE .
To allow for partial credits even if bad actions are taken at certain steps, we deﬁne another reward function corresponding to the sum of probabilities:
[Equation]
The return ˆ Q ( s t , a t ) =
(cid:80) Tt [Equation]
= t p MLE ( a t | s t ) , thus the policy can recover from bad decisions if the subsequent actions receive high reward.
T HE GOLD A LGORITHM
1 π θ ← p MLE , ˜ π θ ← p MLE 2 for step = 1 , 2 , . . .
, M do
Sample a minibatch B = { ( x i , y i ) } | B | i =  foreach ( s it , a it ) do
Compute importance weights max( u, ˜ π θ ) , and compute returns ˆ Q ( s it , a it ) − b
Update θ by (4) using gradient descent if step % k = 0 then ˜
π θ ← π θ 8 Return: π θ
Our full algorithm based on off-policy PG is shown in Algorithm 1 .
For importance weights π θ ( a | s ) , to avoid drastic changes, we initialize π θ with the MLE solution.
In addition, we compute the importance weights by a weighting policy ˜ π θ that synchronizes with π θ periodically so that the weights do not change frequently between updates.
We also lower-bound the importance weight by a small number u .
Since our return is computed from a sum or product of probabilities ((6) and (7)), we truncate the future trajectory after ﬁve steps.
We follow the common practice to subtract a baseline b from the return to reduce variance; moreover, to avoid negative reward on the demonstrations (after subtracting baseline), we lower-bound p MLE in (6) and (7) by a small number c .
In practice, GOLD is easy to implement; further, given an existing p MLE , the GOLD-training stage usually takes less time than MLE.
The code will be made available.
E XPERIMENTS 4. S ETUP
We chose four text generation tasks: (1) question generation ( NQG ; Zhou et al., 2017): given a passage and a short span of the passage, the goal is to generate a question that can be answered by the span; (2) summarization ( CNN/DM ; Hermann et al., 2015); (3) extreme summarization ( XSum ;
Narayan et al., 2018): the references are more abstractive than CNN/DM summaries; (4) machine translation ( IWSLT14 De-En ; Cettolo et al., 2014).
See Appendix A.1 for the size and the source of the datasets.
We evaluate NQG and summarization by both automatic metrics, i.e., corpus-level
BLEU-4 (Papineni et al., 2002) and ROUGE-1/2/L (Lin, 2004) respectively, as well as human ratings.
We experiment with three variants of GOLD: GOLD- δ , GOLD- p , GOLD- s , which uses the δ -reward and the two estimated rewards ( R p and R s ), respectively.
Our baseline learning algorithm is standard
MLE, and we compare with on-policy RL training using policy gradient in Section 4.3.
We describe models for each task at the beginning of Section 4.2.
For GOLD training, we use the baseline b = − 60 for GOLD- p and b = 0 for GOLD- s .
To lower bound the return such that it is non-negative on demonstrated trajectories, we tune the lower bound c of p MLE in { 0 , 0 . 01 , 0 . 05 , 0 . 1 } in (6) and (7).
Furthermore, to reduce variance for importance weights, we lower bound them by u ∈ { 0 , 0 . 1 , 0 . 15 , 0 . 2 } .
All hyperparameters are tuned on the dev set.
See
Appendix A.3 for more reproduciblility details.
4. R ESULTS AND A NALYSIS
GOLD
achieves better metric scores despite high heldout perplexity.
Experiments are run using a ﬁxed random seed (12); attempted three random seeds (1, 12, 123) and all BLEU/R-2 scores are within 0.1 points of the reported.
Refer to Table 3 for transformer results.
[Table]
[Table]
GOLD improves both standard and transformer models.
Recall that one of our main motivations is that MLE tends to over-generalize under model misspeciﬁcation, i.e., high recall but low precision.
One may wonder whether this problem can be ﬁxed by better modeling.
Therefore, we evaluated GOLD with both standard high-performing models and state-of-the-art pretrained model.
For standard models, we chose two representative seq2seq-based models, NQG++ (Zhou et al., 2017) and the pointer-generator model (See et al., 2017) for NQG and CNN/DM respectively.
3 Tableshows that GOLD is better than MLE in terms of BLEU and ROUGE.
In particular, we ﬁnd that using estimated rewards is superior to the δ -reward, showing the beneﬁts of accounting for varying quality of the references.
We thus consider only GOLD- p and GOLD- s in the rest of the experiments.
For transformer models (Vaswani et al., 2017), we used the pretrained BART (Lewis et al., 2020) for
NQG, CNN/DM, and XSum; we used standard transformer for IWSLT14 De-En.
Table 3 shows that
GOLD achieves better scores than MLE across all tasks, including near-SOTA on CNN/DM (R95% conﬁdence interval: 21.84-22.33) and good performance on XSum (R-2 95% CI: 22.25-22.92).
We further crowdsourced human evaluation by pairwise comparison 4 between MLE-trained and
GOLD-
s -trained model outputs.
Each pair of comparison is repeated three times (by three different workers) and we take the majority answer.
For each dataset, the evaluations are done by at leastdifferent workers.
For NQG, we showed workers the entire input and the questions generated by two models, and we ask workers to select the better one (with a third “tie” option).
For summarization,
[Table]
[Figure]
[Table]
models
[Table]
loss distribution is spread out [Table] high-precision models whose loss distribution is concentrated on near-zero losses [Table]
we ask workers to select the generation closer in meaning to the reference without showing the article.
More details are in Appendix C. Table 5 shows that workers prefer outputs from models trained by
GOLD more often than those trained by MLE.
[Table]
GOLD encourages high-precision models.
One interesting observation from Table 1 and Tableis that compared to MLE, GOLD leads to much higher held-out perplexities, while achieving better metric scores.
Since both are evaluated against the reference, one would expect high perplexity to correlate with low metric scores.
To better understand the behavior of GOLD, we examine the distributions of token-level negative log-likelihood (NLL) loss (a monotonic transformation of perplexity) in Figure 1.
We see that the loss distribution of GOLD (compared to MLE) concentrates on near-zero losses (Figures 1a and 1c) with a long tail of large losses (Figures 1b and 1d), hence high perplexity.
In contrast, MLE has much fewer near-zero losses and fewer large losses, suggesting it tries to generate all tokens; i.e., MLE encourages recall, as discussed in Section 2.
We conclude
[Table]
that GOLD achieves better metric scores by focusing on easy-to-learn tokens at the expense of lower recall with respect to the reference
Another advantage of high-precision models is that they do not rely much on decoding algorithms to sample high-quality outputs from the learned distribution.
From a RL perspective, the policy already considers future rewards when making local decisions, thus beam search is not necessary.
As a result, we see in Table 2 that GOLD achieves similar performance with both argmax decoding and top- k sampling.
In contrast, MLE suffers signiﬁcantly from sampling, which suggests that it learns a high-recall but low-precision model.
GOLD alleviates exposure bias.
GOLD does not suffer from exposure bias because it trains on the state/history distribution induced by the model instead of the reference data.
Here, we empirically quantify the exposure bias problem in learned models.
If there is exposure bias, then the output quality is expected to degrade as output length increases, as the history is more likely to deviate from the reference distribution with accumulated generation steps.
To evaluate quality, we sampledgenerations of different lengths from standard models trained by both MLE and GOLD on NQG.
Given the paragraph, words to query on, and the generated questions, we then asked workers to rate the generations from 1 (worst) to 4 (best).
Figure 2 (left) shows that the output quality of the
MLE-trained model degrades when the sequence length is over 14 words, whereas the quality of the
GOLD-
s -trained model stays relatively stable across all lengths.
Qualitatively, we observe frequent degenerations (Holtzman et al., 2020; Welleck et al., 2020a) including repetitions and hallucinations within a sentence generated by MLE-trained model, as shown in Table 6.
In contrast, Figure 2 (right) shows the NLL loss conditioned on gold histories on NQG dev set.
6 We can see that without exposure bias, NLL loss does not vary much as the length increases.
Therefore, we conclude that the big performance drop for long generations using MLE is mainly due to exposure bias and GOLD does not suffer from the problem.
[Table]
5 We also used BLEU as a quality metric and observed similar results, shown in Appendix B.3.
A token prediction accuracy vs
4. C OMPARISON WITH O N -P OLICY T RAINING
While ofﬂine RL is generally more challenging due to lack of interaction with the environment, we argue that the beneﬁt from interaction is limited in text generation (Section 3.1) and overweighed by the optimization challenges.
In this section, we investigate the effect of on-policy training using task metrics as rewards.
Speciﬁcally, we pre-train the model using MLE and then ﬁne-tune it using PG.
To avoid degenerate solutions, we interleave MLE and PG updates evenly during ﬁne-tuning.
Similarly, we ﬁne-tune GOLD-initialized models using PG.
For on-policy ﬁne-tuning, we use BLEU and ROUGE-2 as rewards for NQG and CNN/DM respectively.
7 Table 4 shows that additional on-policy training improves both MLE and GOLD marginally.
However, MLE with PG is still worse than GOLD.
Further, one of the best-performing on-policy methods using a similarly competitive pretrained transformer model (Ziegler et al., 2019) also shows limited improvements over supervised baseline on CNN/DM, despite having better reward functions (domain-speciﬁc human preference annotations).
Overall, the beneﬁt from on-policy training is unclear in our experiments.
8 Please refer to the appendix for more details.
4. D ISCUSSION ON “D IVERSITY ”
We have seen from Table 1 and Table 3 that improvements in precision lead to higher perplexity.
One may wonder what the impact on text “diversity” is.
This issue warrants more discussion, but for the text generation research community, “diversity” may stand for the following.
(1) Diversity as in the ability to generate a number of different correct generations given one context.
“Mode collapse” is an important problem for image generation and unconditional text generation
(e.g., continuation from a prompt).
However, for many conditional NLG tasks, while there are multiple correct outputs, producing one good generation is often sufﬁcient in practice, e.g., question generation, summarization, machine translation, image captioning, text style transfer
, and even chit-chat dialogues (unless users expect the bots to say different things in the same context every time).
The exceptions include creative writing tasks where we would like multiple novel generations under the same context, e.g., generating from a language model (Caccia et al., 2020).
In these cases,
GOLD may not be able to provide a variety of high-quality generations given one context, although it would still produce different outputs given different contexts.
Another potential failure mode is that in open-ended dialogues, if one common response has large probability under true data distribution, then GOLD may lead to a distribution concentrated on this mode.
In this case, additional inductive bias is needed to separate good modes from bad ones, e.g., additional reward on speciﬁcity of the response.
On the other hand, while MLE-trained models have good recall and we can potentially sample many different outputs with a high temperature, or large k in top- k sampling, or large p in top- p sampling, 9 there are only a few high-quality ones.
Our conjecture is that there may not be enough data to cover all modes, and in fact high-likelihood outputs from MLE-trained models are often degenerate (Stahlberg & Byrne, 2019; Cohen & Beck, 2019; Holtzman et al., 2020).
In sum, given the trade-off between diversity and quality, we argue that generating a single highquality output is a reasonable goal for most conditional text generation tasks, and we leave the question of generating both diverse and high-quality outputs to future work.
(2) Diversity as in the linguistic complexity of the output, given the input.
[Equation]
We can measure the complexity of the output themselves using the number of unique ngrams.
For example, GOLD’s number of unique 1/2/3/4/5-grams for XSum (using BART) is 18846/18835/18103/17639/17258, MLE’s is 19071/19053/18349/17875/17531, and gold-standard target numbers are 23674/23661/22869/22280/21822.
GOLD only produces a little fewer unique n-grams than MLE.
(2.2) Speciﬁcally for question generation and summarization, we can interpret the complexity of the output with respect to input by computing the proportion of n-gram overlaps (i.e., what percentage of n-gram in generations are also in the source paragraph or source article)?
For example, for XSum (using BART), the proportion of 1/2/3/4/5-gram overlap for MLE is 0.75/0.27/0.10/0.053/0.031 and for GOLD: 0.73/0.24/0.087/0.039/0.021; the trend mostly holds for NQG and CNN/DM as well.
We can see that GOLD copies a little less from source, although copying may intuitively be the easy way for question generation and summarization.
(3) Diversity as in the coverage of the true data distribution.
This deﬁnition is related to (1), but (3) does not guarantee (1).
This diversity is the “recall” intuitively, and can be measured by NLL loss or perplexity, which will be sacriﬁced.
In our case, the consequence is that the model would tend to ignore difﬁcult gold examples (Figure 1), which in text generation, may sometimes be noise or outliers.
Empirically for a large number of text generation tasks, paying less attention to such examples would not cause the image-generation-like mode collapse which is considered bad.
R ELATED W ORK
Exposure bias.
In structured prediction, there is a ﬂurry of works addressing exposure bias since
Bengio et al.
Most works focus on learning global sequence scores instead of locally normalized scores using either variants of beam search (Wiseman & Rush, 2016;
Andor et al., 2016;
Goyal et al., 2018) or energy networks (Belanger & McCallum, 2016;
Tu et al., 2020).
These training algorithms are often complex and costly.
Exposure bias is well studied in imitation learning (Daum ´ e et al., 2009; Ross et al., 2011) and learning-to-search has been applied to RNNs to incorporate losses of sequences deviating from references (Leblond et al., 2018), but they require annotations or cost functions on non-reference sequences which may not be available for text generation.
Objectives beyond MLE.
Policy gradient-based algorithms and their variants have been used extensively in text generation to optimize sequence-level metrics (Ranzato et al., 2016;
Shen et al., 2016; Norouzi et al., 2016; Pasunuru & Bansal, 2018).
In addition, off-policy RL is commonly used in dialogue where online interaction with users is expensive (Serban et al., 2017; Jaques et al., 2019).
The main difference is that we take advantage of the demonstrations and design generic reward functions for generation tasks.
There is another line of work using policy gradient to optimize reward from a discriminator that differentiates good vs. bad generations (Yu et al., 2017; Li et al., 2017;
Lu et al., 2019).
However, these approaches often underperform MLE in practice (Tevet et al., 2019) due to optimization challenges.
Recently, a concurrent work, Kang & Hashimoto (2020), proposed truncated log-loss which both optimizes distinguishability and enjoys efﬁcient optimization.
High-precision text generation.
It is noticed early in neural text generation that MLE tends to produce high-recall models that over-generalize.
Previously, high-quality outputs are selected mainly through decoding (e.g., beam search, low-temperature sampling, truncated sampling).
Recently, there is an increasing amount of work on discouraging implausible samples during training, e.g., using negative sampling (Welleck et al., 2020b), self-training on high-quality samples (Kedzie & McKeown, 2019), and conﬁdence-oriented decoding with calibration (Tian et al., 2020).
In contrast, we tackle the fundamental problem of mismatched objectives and propose a general learning framework.
C ONCLUSION
We provide an efﬁcient algorithm that addresses the two train/test discrepancies in MLE training for text generation: likelihood as learning objective vs. quality as evaluation metric; gold history in training vs. model-generated history in inference.
We have demonstrated that off-policy RL is a promising framework for text generation, with matched train/test objectives and optimization advantages like MLE.
We believe more advanced off-policy learning techniques (e.g., proximity constraints) can be easily integrated into text generation and further improve performance. A CKNOWLEDGEMENTS
The authors thank Kyunghyun Cho, Tatsunori Hashimoto, Graham Neubig, Ethan Perez, Karl Stratos,
Clara Vania, and Alex Warstadt (alphabetical order) for helpful discussions, and the anonymous reviewers for helpful feedback.
A P RACTICAL S ETUP AND I MPLEMENTATION A. T ASKS AND D ATASETS
(1) Natural question generation ( NQG ; Zhou et al., 2017) based on the SQuAD QA dataset (Rajpurkar et al., 2016): Given a text passage and a short span of the passage, the goal is to generate a question that can be answered by the span.
(2) CNN/DailyMail summarization (CNN/DM) :
Given a piece of news, generate a few sentences of summary.
We use the entity-non-anonymized version of CNN/DM dataset, following See et al.
The target summaries tend to be extractive , meaning there tends to be heavy text-span overlaps between the source article and the target summary.
(3) Extreme summarization ( XSum ; Narayan et al., 2018) is based on BBC news.
The target summaries are highly abstractive .
Past extractive strategies that work well for CNN/DM may not work well for XSum.
(4) IWSLT
German to English machine translation ( IWSLT14 De-En ;
Cettolo et al., 2014) is a popular machine translation benchmark.
Machine translation is different from the above three tasks, given that intuitively, the space of high-quality generation is smaller.
More details on datasets.
We ﬁrst provide the number of examples in each dataset.
The train/dev/test split for NQG is 86229/8913/8919; the split for CNN/DM is 287227/13368/11490; the split for XSum is 203454/11306/11334; the split for IWSLT14 De-En is 160239/7283/6750.
To download and preprocess the NQG data, we follow the following instructions: https:// github.com/clovaai/FocusSeq2Seq ; to download and preprocess the summarization data, we follow the following instructions: https://github.com/pytorch/fairseq/blob/ master/examples/bart/README.summarization.md ; to download and preprocess the
IWSLT14 De-En data, we follow the following instructions: https://github.com/pytorch/ fairseq/tree/master/examples/translation .
A. M ODEL A RCHITECTURES
We use two sets of architectures for our experiments.
Standard architectures.
For NQG, we use the model NQG++ (Zhou et al., 2017), a seq2seqwith-attention model based on GRU (Cho et al., 2014), and for summarization we use pointer generator network (See et al., 2017), a seq2seq-with-attention model based on LSTM (Hochreiter & Schmidhuber, 1997).
Speciﬁcally, we use 2 layers for both the encoder and the decoder, for both tasks.
Other hyperparameters are based on the following implementation: https://github. com/clovaai/FocusSeq2Seq .
Transformer architectures.
For NQG, CNN/DM, and XSum, we also experiment with one of the top-performing models, BART (Lewis et al., 2020).
Our experiments are based on the pretrained
BART model provided by original authors 10 : it has 12 encoder layers and 12 decoder layers, and it is pretrained on around 3.3 billion words of Wikipedia articles and books.
We use the model to investigate if our methods work with models with stronger capabilities.
For IWSLT14 De-En, we use a moderate-size standard transformer architecture (encoder/decoder embedding dimension 512, 4 encoder attention heads, 6 encoder layers, 4 decoder attention heads, 6 decoder layers), a top-performing architecture in machine translation.
A. M ORE ON R EPRODUCIBILITY
The codebase will be released.
The link to the code will be posted on the following website:
yzpang.me . Hyperparameters and training details on standard architectures.
This paragraph corresponds to results in Table 1.
We use a learning rate of 5e-4.
For NQG, we use a batch size of 32; for
CNN/DM we use a batch size of 16.
We train using a single Nvidia GTX 1080 Ti (memory: 12 GB)
As discussed in Section 3.3 and Section 4.1, we tune the lower bound of p MLE in { 0 , 0 . 01 , 0 . 05 , 0 . 1 } .
For NQG models, the lower bound of 0.1 produces best performance.
For CNN/DM using GOLD- p , the lower bound is 0.01; for CNN/DM using GOLD- s , the lower bound is 0.
Recall that as discussed in Section 3.3, the weighting policy ˜
π θ synchronizes with actual policy π θ once every k steps so as to stabilize training.
We tune k ∈ { 1500 , 2691 } (where 2691 steps corresponds to 1 epoch) for NQG and found that k = 1500 works better for all NQG models.
We tune k ∈ { 1500 , 3000 , 5000 } for CNN/DM; we found that k = 1500 works best for GOLD-
δ and
GOLD- p , and k = 5000 works best for GOLD- s .
Note that in practice, we do not observe big gaps when using other k ’s in the set.
For standard models, implementation is based on Cho et al.
In all experiments, we evaluate once every epoch, and we do validation on the entire dev set, using task-speciﬁc metric (BLEU/ROUGE), following Cho et al.
and standard practice in machine translation.
Hyperparameters and training details on transformer models.
This paragraph corresponds to results in Table 3.
For transformer models, we use Nvidia P40 GPUs (memory: 24 GB each).
For
NQG, CNN/DM, and XSum based on BART, we use 4 GPUs to train.
For IWSLT14 De-En, we use 1 GPU.
Note that fairseq deﬁnes batch size in terms of number of tokens instead of number of sequences.
For NQG, we use 512 tokens as batch size (for each of the four GPUs); for CNN/DM and
XSum, we use 1024 tokens as batch size (for each of the four GPUs); for IWSLT
De-En, we use 4096 tokens as batch size.
We use a learning rate of 2e-5 for NQG, CNN/DM, and XSum; 3e-4 for IWSLT
De-En.
Recall that as discussed in Section 3.3, the weighting policy ˜
π θ synchronizes with actual policy π θ once every k steps so as to stabilize training.
Here, k = 1000 for NQG; k
= 5000 for CNN/DM,
[-XSum; k = 3000 for IWSLT
-]XSum, IWSLT14 De-En.
As discussed in Section 3.3 and Section 4.1, the lower bound of p MLE is set to be 0.01 for GOLD- p and 0.1 for GOLD- s .
For all other parameters that are not speciﬁc to GOLD, we use the default fairseq summarization parameters (which can be found through footnote 10).
For hyperparameter u as discussed in Section 4.1, for NQG and CNN/DM, u = 0 . 1 ; for XSum, u = 0 . 15 ; for IWSLT14 De-EN, u = 0 . 2 .
As indicated, the hyperparameters were only tuned in a small set of possible values.
More careful tuning may result in slightly better performances.
Number of parameters in each model.
For standard models, we use NQG++ for NQG, and it has 10372565 parameters.
We use pointer generator for CNN/DM, and it has 19965705 parameters.
For transformer models, the BART model for NQG, CNN/DM, and XSum all haveparameters; the transformer model used for IWSLT14 De-En has 39469056 parameters.
Average runtime.
For standard models, based on the above models and the computing infrastructures, each epoch of NQG takes around 10 minutes to train and achieves best performance withinepochs.
Each epoch of CNN/DM takes about 2 hours to train and achieves best performance within 15 epochs.
For transformer models, each epoch of NQG takes around 5 minutes to train and achieves best dev performance within 5 epochs; each epoch of CNN/DM takes around 11 hours to train and achieves best dev performances within 5 epochs; each epoch of XSum takes around 8 hours to train; each epoch of IWSLT14 De-En takes around 3 minutes to train and achieves best performances within 100 epochs (as expected, given the large batch size 11 ).
Note that our transformer models are trained on P40s given hardware constraints; if the transformer models are trained on V100 GPUs, the training time per epoch will likely be much shorter.
A. M ORE D ISCUSSION ON A PPROXIMATIONS
Truncating the future trajectory after ﬁve steps.
In other words, the number of current+future steps is upper-bounded at six.
Effectively, we are using a discount factor of 0.83.
Given that the Q -value corresponds to future return, we attempted using different strategies. (1) Using the entire future trajectory, and (2) using a ﬁxed number of future steps. We attempted (1) on NQG using the standard models (tuned discount factor in { 1 , 0 . 9 , 0 . 8 , 0 . 7 , 0 . 5 } ) and found that { 0 . 8 , 0 . 7 } usually performs best, resulting in similar performance but longer training time, compared to the current 5-future-step approach.
We attempted (2) using the number of future steps in { 1 , 2 , 3 , 5 , 7 , 10 } and found that using { 5 , 7 , 10 } leads to similar results, which are slightly better compared to { 1 , 2 , 3 } .
One beneﬁt is that given a ﬁxed number of future steps, we found that using easy-to-tune constant baselines work well, and training time is also much shorter.
One-step approximation of importance weights.
Recall that as discussed Section 3.1, we use the one-step approximation of importance weights (IW).
We also experimented using full importance weights, using standard models on NQG.
Empirically, the model performance is more sensitive to hyperparameters discussed in Appendix A.3, and training takes much longer (within 15 epochs using one-step approximation of IW, but ∼ 25 epochs using full-preﬁx importance weights).
However, the ﬁnal test BLEU score on NQG (using GOLD- p ) is 15.80, similar to 15.93 using one-step approximation.
A. D ETAILS ON O N -P OLICY E XPERIMENTS
For the MLE+PG baseline, we used the REINFORCE algorithm with sequence-level rewards (BLEU for NQG and ROUGE-2 for summarization).
We attempted two versions of the baselines: (i) constant baselines searched in { 0 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 15 } for BLEU (NQG) and ROUGE-2 (CNN/DM), as well as (ii) baselines computed by the average BLEU/ROUGE-2 over the last 100 steps, minus { 0 , 0 . 05 } .
In terms of training warmup choices, We tried two versions of the training algorithm.
(a) We initialized with MLE and trained with PG losses interpolated with MLE losses, given we found that the training process would become very unstable without interpolation.
(b) We also attempted the following: we intialized the model at random and used MIXER (Ranzato et al., 2016).
However, we failed to ﬁnd improvements compared to (a), under our architecture.
A relevant work Choshen et al.
(2020) showed that properly tuned on-policy RL may not work for text generation in some cases.
We also tried MIXER with the learned baseline for NQG, which is estimated by a simple linear regressor that takes RNN hidden states as inputs, according to Ranzato et al.
After some tuning, we achieved only slight improvements in NQG (BLEU 14.71).
One advantage of GOLD is that our algorithm does not rely on learning baselines which could have a big impact on performance of on-policy algorithms; in fact, all baselines are constants in this paper.
Note that for GOLD+PG models, we only attempted constant baselines; better tuning of baselines could potentially lead to stronger performance.
B M ORE ON R ESULTS B. L EAD -3 B ASELINES FOR S UMMARIZATION
The lead-3 baseline (using ﬁrst 3 sentences as summaries) is a popular strong baseline in summarization literature.
The ROUGE-1/2/L scores of the lead-3 baselines are as follows: 40.42/17.62/36.
for CNN/DM; 16.30/1.60/11.95 for XSum.
Our performance using transformer models beat these baselines by a large margin.
B. P ERFORMANCE WITH T RANSFORMER A RCHITECTURES
We experiment using transformer architectures, as shown in Table 3; we also experiment on two more tasks (compared to using standard architectures): XSum and IWSLT14 De-En.
We achieve SOTA/near-SOTA result (according to automatic metrics which have inherent limitations) on CNN/DM: at the time of writing, our results (45.40/22.01/42.25 using GOLD- p or 44.82/22.09/41.81 using GOLD- s ) are higher than 44.17/21.47/41.11 (PEGASUS; Zhang et al., 2020) and 44.20/21.17/41.30 (ProphetNet; Qi et al., 2020), both slightly higher than BART.
Note the PEGASUS CNN/DM result is pretrained on 1.5B news articles (around 3.8 terabyte), whereas BART is pretrained on 3.3B words (around 0.16 tetrabyte).
Our XSum results are also higher than [Figure] PEGASUS (45.20/22.06/36.99) trained on Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), but lower than the PEGASUS result using the publicly-unavailable 1.5B- article 3.8 terabyte HugeNews (Zhang et al., 2020) as pretrained corpus.
We hypothesize that if our models are applied onto their architectures instead of pointer generator networks or BART, we would similarly get non-trivial improvements.
We also achieve 0.81 point of BLEU improvement on IWSLT14 De-En; GOLD- s performs better than the existing approaches that do not use knowledge distillation or data augmentation, as far as the authors are aware.
B. M ORE ON E XPOSURE B IAS
With exposure bias.
Recall that in Section 4.2, we used human evaluation (a score of 1 or 2 or 3 or 4) to approximate the output quality, and we found that the MLE-trained model degrades signiﬁcantly when the generation length is long, whereas the quality of the GOLD- s -trained model stays relatively stable across lengths.
Here, we use BLEU to approximate the quality of NQG generations, and we show that BLEU does not bias toward long sentences.
Figure 3 shows the average sentence-level BLEU by sequence length.
Speciﬁcally, Figures 3a and 3c show the BLEU on randomly shufﬂed targets (from dev set), which show that longer sentences do not appear to punish BLEU scores.
Figures 3b and 3d show the
BLEU by sentence length, on model generations.
We see that MLE’s BLEU decreases by length but
GOLD- s ’s BLEU appears to stay relatively stable.
We thus see some evidence that MLE is generating worse sentences as sentence gets longer.
If there is no exposure bias.
In the main text, we used the NLL loss vs. length plot to demonstrate that without exposure bias, the loss does not vary much across length, so the MLE performance drop in Figure 2 (left) is mainly due to exposure bias.
Here, we provide another way to analyze the case without exposure bias.
[Figure]
Without exposure bias, token prediction accuracy stays relatively stable across lengths.
Figure 4 shows the token prediction accuracy conditioned on gold histories on NQG dev set.
Note that for each example, we let t x =
L x − 5 , where L x is the length of reference sentence x .
We can see that without exposure bias, prediction accuracy does not vary much as the length increases.
Therefore, we conclude that the big performance drop for long generations using MLE is mainly due to exposure bias and GOLD does not suffer from the problem.
B. E XAMPLES
Table 7 and Table 8 show the example generations based on the transformer models.
[Table]
President
[Table]
has argued with progressive potentate Elizabeth Warren, calling her ”wrong” on trade policy.
What everyone does next will be critical for the 2016 elections and the future of Democratic politics.
Warren has publicly criticized ”fast track” trade authority
[Table] [Table]
ich hab da so ne kognitive r¨uckkopplung, du hast was projiziert, was du sehen m¨ochtest
[Table]
i’ve been doing this with cognitive feedback, you’ve been prospecting what you want to see.
[Table]
i’ve got cognitive feedback, you’ve proved what you want to see.
[Table]
i have this cognitive feedback, you projected something you want to see
[Table]
es sind also alle werkzeuge
[Table]
-En
[Table]
C H UMAN E VALUATIONS
C.
P AIRWISE C OMPARISON
Our goal is to enable high-quality generations that do not necessarily result in gold references.
Given that corpus-level BLEU/ROUGE score is only a popular approximation of generation quality, we ﬁrst conduct human ratings to conﬁrm the hypothesis that our approaches are generating better sequences.
For NQG, for each unit of human evaluation, we present the source paragraph, the words to ask the question on, the question generated by MLE-trained model, as well as the question generated by GOLD- s -trained model.
We ask the human evaluators the general question: which generated question is better?
Figure 5 shows one example interface of pairwise comparisons.
Using NQG dev set, on standard models, of the 183 pairs of comparison we conducted human evaluations on, 42 (23.0%) MLE-questions are better, 81 (44.3%)
GOLD-
s -questions are better, and 60 (32.8%) are tied.
We also evaluate on models based on BART, shown in Table 5 in the main text.
For summarization tasks, given that it is infeasible to get high-quality annotations if we let workers read the entire news article 14 , we only did the following: given the reference summary, a summary generated from MLE model, and a summary generated from our model, we asked workers to compare which generated summary is closer in meaning to the reference summary.
Figure 6 shows one example interface of the mentioned pairwise comparison for summarization.
See Table 5 for results.
[Figure]
NQG rating was conducted to examine if longer sentences (generated by MLE-trained model) will result in worse human ratings, and if GOLD alleviates the problem.
In Figure 2 (left), to reduce variance, we group length by buckets of two (e.g., [7 , 8] , [9 , 10] , [11 , 12] , etc.).
Furthermore, we sampled 736 annotations such that each bucket would contain at least 30 sentences (for human evaluation) for each of MLE and GOLD- s .
We also shown the 95% conﬁdence interval using standard bootstrapping, in Figure 2 (left).
Given the paragraph, words to query on, and the generations, we ask workers to rate the generations.
Figure 7 shows an example interface of NQG human ratings.
We ask workers to consider both the correctness of the generation (i.e., if the question is asking about the speciﬁed words using facts) and the quality of the generation (i.e., if the generation is ﬂuent and coherent).
We ask workers to rate from 1 to 4, where 1 means very bad, 2 means slightly below average, 3 means slightly above average, and 4 means very good.