Original: Q UANTIFYING E XPOSURE B IAS FOR O PEN - ENDED L ANGUAGE G ENERATION
Revised: Q UANTIFYING E XPOSURE B IAS FOR O PEN - ENDED L ANGUAGE G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The exposure bias problem refers to the  training-generation discrepancy,caused by teacher forcing, in maximum likelihood estimation (MLE) training for autoregressive neural network language models (LM).
Revised: The exposure bias problem refers to the incrementally distorted generation induced by the training-generation discrepancy, in teacher-forcing training for autoregressive neural network language models (LM).

Original: It has been regarded as a central problem for LMs trained for open-ended language generation.
Revised: It has been regarded as a central problem for LMs trained for open-ended language generation.

Original: Although a lot of algorithms have been proposed to avoid teacher forcing and therefore alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is.
Revised: Although a lot of algorithms have been proposed to avoid teacher forcing and therefore alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is.

Original: In this work, we propose two metrics to quantify the impact of exposure bias in the generation of MLE-trained LMs.
Revised: In this work, we propose novel metrics to quantify the impact of exposure bias in the generation of MLE-trained LMs.

Original: Our metrics cover the quality, diversity, and consistency aspects of language generation. Surprisingly, our measurements in both real and synthetic settings, as well as human evaluation, consistently suggest that removing the training-generation discrepancy in the preﬁx only brings minor performance gain.
Revised: Our key intuition is that if we feed ground-truth data preﬁxes (instead of preﬁxes generated by the model itself) into the model and ask it to continue the generation, the performance should become much better because the training-generation discrepancy in the preﬁx is removed.

Original: More importantly, we do not observe the incremental distortion as claimed by exposure bias. In our analysis, we hypothesize that although there exists a discrepancy between the model (generation) and data (training) distribution, the mismatch is still in the model’s “comfortable zone”, and is not big enough to induce drastic performance loss.
Revised: We conduct both automatic and human evaluation in our experiments, and our observations are two-fold: (1) We conﬁrm that the preﬁx discrepancy indeed induces some level of performance loss. However, the induced distortion seems to be limited, and is not incremental during the generation, which contradicts the claim of exposure bias.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Language model (LM) is a central module for natural language generation (NLG) tasks (Young et al., 2017) such as open-ended language generation (Radford et al., 2018; Nadeem et al., 2020), machine translation (Wu et al., 2017), dialogue response generation (Li et al., 2017), image captioning (Lin et al., 2014), etc.
Revised: Language model (LM) is a central module for natural language generation (NLG) tasks (Young et al., 2017) such as open-ended language generation (Radford et al., 2018; Nadeem et al., 2020), machine translation (Wu et al., 2017), dialogue response generation (Li et al., 2017), image captioning (Lin et al., 2014), etc.

Original: For decades, maximum likelihood estimation (MLE) has been the most widely used objective for LM training.
Revised: For decades, maximum likelihood estimation (MLE) has been the most widely used objective for LM training.

Original: However, there is a popular belief in the natural language processing (NLP) community that standard MLE training suffers from the exposure bias problem which leads to an incremental performance degradation during test-time generation.
Revised: However, there is a popular belief in the natural language processing (NLP) community that standard MLE training suffers from the exposure bias problem which leads to an incremental performance degradation during test-time generation.

Original: The claim of the exposure bias problem (Bengio et al., 2015; Ranzato et al., 2016) is originated from the following discrepancy between MLE training and test-time generation for auto-regressive language models:
Revised: The claim of the exposure bias problem (Bengio et al., 2015; Ranzato et al., 2016) is originated from the following discrepancy between MLE training and test-time generation for auto-regressive language models:

Original: During training, the model is trained to predict the next word conditioned on preﬁx (or history) words sampled from the ground-truth data distribution; While during generation, the model generates words conditioned on preﬁx sequences generated by the model itself.
Revised: During training, the model is trained to predict the next word conditioned on preﬁx (or history) words sampled from the ground-truth data distribution; While during generation, the model generates words conditioned on preﬁx sequences generated by the model itself.

Original: Hence, due to the exposure to real data during training, the language model could potentially be biased to only perform well with data preﬁxes.
Revised: Hence, due to the exposure to real data during training, the language model could potentially be biased to only perform well with data preﬁxes.

Original: Therefore, it is claimed (and widely believed among researchers) that during generation the errors should accumulate along the generated sequence, and the distribution generated by the model would be incrementally distorted.
Revised: Therefore, it is claimed (and widely believed among researchers) that during generation the errors should accumulate along the generated sequence, and the distribution generated by the model would be incrementally distorted.

Original: The forced exposure to ground-truth data during training is also referred to as teacher forcing .
Revised: The forced exposure to ground-truth data during training is also referred to as teacher forcing .

Original: In order to avoid teacher forcing, many training algorithms (Bengio et al., 2015; Lamb et al., 2016;
Revised: In order to avoid teacher forcing, many training algorithms (Bengio et al., 2015; Lamb et al., 2016;

Original: Ranzato et al., 2016; Yu et al., 2016; Zhu et al., 2018;
Revised: Ranzato et al., 2016; Yu et al., 2016; Zhu et al., 2018;

Original: Lu et al., 2018; Lin et al., 2017; Guo et al., 2017;
Revised: Lu et al., 2018; Lin et al., 2017; Guo et al., 2017;

Original: Rajeswar et al., 2017; Wiseman & Rush, 2016; Nie et al., 2019; Shi et al., 2018; de Masson d’Autume et al., 2019) have been proposed as alternatives to MLE training for open-ended language generation.
Revised: Rajeswar et al., 2017; Wiseman & Rush, 2016; Nie et al., 2019; Shi et al., 2018; de Masson d’Autume et al., 2019; Rennie et al., 2016) have been proposed as alternatives to MLE training for open-ended language generation.

Original: Most of these works utilize techniques from generative adversarial networks (GANs) (Goodfellow et al., 2014) or reinforcement learning (RL) (Sutton & Barto, 1998).
Revised: Most of these works utilize techniques from generative adversarial networks (GANs) (Goodfellow et al., 2014) or reinforcement learning (RL) (Sutton & Barto, 1998).

Original: In this paper, we refer to these algorithms as non-MLE methods.
Revised: In this paper, we refer to these algorithms as non-MLE methods.

Original: With the huge research efforts devoted to alleviate exposure bias, interestingly, the existence or signiﬁcance of exposure bias is much less studied.
Revised: With the huge research efforts devoted to alleviate exposure bias, interestingly, the existence or signiﬁcance of exposure bias is much less studied.

Original: On the other hand, despite the criticism, MLE
Revised: On the other hand, despite the criticism, MLE

Original: (teacher forcing) has remained to be the dominant objective for LM training (Radford et al., 2018;
Revised: (teacher forcing) has remained to be the dominant objective for LM training (Radford et al., 2018;

Original: Keskar et al., 2019).
Revised: Keskar et al., 2019).

Original: To make the situation more curious, multiple recent works show that the proposed non-MLE methods actually have inferior generation performance (Caccia et al., 2018; de Masson d’Autume et al., 2019) than the MLE baseline.
Revised: To make the situation more curious, multiple recent works show that the proposed non-MLE methods actually have inferior generation performance (Caccia et al., 2018; de Masson d’Autume et al., 2019) than the MLE baseline.

Original: These negative results lead us to question:
Revised: These negative results lead us to question:

Original: Is exposure bias truly a serious problem for MLE training?
Revised: Is exposure bias truly a serious problem for MLE training?

Original: In this work we seek a direct answer to the above question.
Revised: In this work we seek a direct answer to the above question.

Original: Here we brieﬂy summarize our contributions: We conduct controlled experiments in which we remove the training-generation discrepancy in the preﬁx, and design various metrics to quantify the performance improvement of the generation as the preﬁx length grows.
Revised: Here we brieﬂy summarize our contributions: We conduct controlled experiments in which we remove the training-generation discrepancy in the preﬁx, and design various metrics to quantify the performance improvement of the generation as the preﬁx length grows.

Original: On the contrary to our expectation, our measurements consistently show that the performance gain is minimal, and the incremental distortion as claimed by exposure bias is not observed (the performance gap does not become larger with longer preﬁxes).
Revised: On the contrary to our expectation, our measurements consistently show that the performance gain is limited, and the incremental distortion as claimed by exposure bias is not observed (the performance gap does not become larger with longer preﬁxes).

Original: In the next section, we begin by introducing notations and background.
Revised: In the next section, we begin by introducing notations and background.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: The task of auto-regressive language modelling is to learn the probability distribution of the ( l + 1) th word (or token) W l +1 in a sentence W conditioned on the preﬁx W 1: l := ( W 1 , . . .
Revised: The task of auto-regressive language modelling is to learn the probability distribution of the ( l + 1) th word (or token) W l +1 in a sentence W conditioned on the preﬁx W 1: l := ( W 1 , . . .

Original: , W l ) .
Revised: , W l ) .

Original: We use
Revised: We use

Original: W i ∈ V to denote a discrete random variable distributed across the vocabulary V .
Revised: W i 2 V to denote a discrete random variable distributed across the vocabulary V .

Original: For simplicity, we assume all sentences are of length L in the formulations.
Revised: For simplicity, we assume all sentences are of length L in the formulations.

Original: Denoting the ground-truth data distribution as P D , the standard MLE training aims to minimize the negative log-likelihood (NLL) loss below:
Revised: Denoting the ground-truth data distribution as P D , the standard MLE training aims to minimize the negative log-likelihood (NLL) loss below:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where P θ ( · | W 1: l ) denotes the conditional distribution of W l +1 of P θ given a preﬁx W 1: l , and θ stands for the set of parameters to be trained.
Revised: ) where P ✓ ( · | W 1: l ) denotes the conditional distribution of W l +1 of P ✓ given a preﬁx W 1: l , and ✓ stands for the set of parameters to be trained.

Original: Note that the concept of “sentence” ( W ) can be naturally generalized to paragraphs or even articles, depending on the target task.
Revised: Note that the concept of “sentence” ( W ) can be naturally generalized to paragraphs or even articles, depending on the target task.

Original: We denote the distribution of a MLE-trained LM as P M , which is the major subject of this study.
Revised: We denote the distribution of a MLE-trained LM as P M , which is the major subject of this study.

Original: For generation, we sample a sequence from P M by sampling token-by-token from P M ( · | W 1 ..l ) .

Original: We will experiment with two popular model architectures: LSTM LM (Hochreiter & Schmidhuber, 1997; Sundermeyer et al., 2012) and transformer LM (Baevski & Auli, 2018; Dai et al., 2019).
Revised: We will experiment with two popular model architectures: LSTM LM (Hochreiter & Schmidhuber, 1997; Sundermeyer et al., 2012) and transformer LM (Baevski & Auli, 2018; Dai et al., 2019).

Revised: For generation, we do classical ancestral sampling without invoking sampling algorithms such as top- k sampling (Fan et al., 2018) for the following reasons: (1) The sampling algorithms are known to trade quality out of diversity (Nadeem et al., 2020; Caccia et al., 2018).

Revised: So, invoking them could “hide” the exposure bias problem because the preﬁxes from the model will be of higher quality.

Revised: The sampling algorithms requires tuning of hyper-parameters, which will complicate the comparison.

Original: In addition to popular metrics in natural language generation (NLG) such as BLEU (Papineni et al., 2002) or METEOR (Denkowski & Lavie, 2014), our quantiﬁcation approaches also rely on the measurements of the distance between two distributions.
Revised: In addition to popular measures in natural language generation (NLG) such as BLEU (Papineni et al., 2002) or METEOR (Denkowski & Lavie, 2014), our quantiﬁcation approaches also rely on the measurements of the divergence between two distributions.

Original: Let P denote the set of probability distributions on the vocabulary V , and let d : P × P → R ≥ 0 be a distance measure between two distributions (e.g., total variation distance).
Revised: Let P denote the set of probability distributions on the vocabulary V , and let f div : P ⇥ P ! R  0 be a divergence function between two distributions (e.g., total variation distance).

Original: We will adopt two popular probability distance measures: total variation distance (denoted as d TV ) and Jensen-Shannon divergence (denoted as d JS ).
Revised: We will adopt two popular probability divergence functions: total variation distance (denoted as d TV ) and Jensen-Shannon divergence (denoted as d JS ).

Original: We provide deﬁnitions of d TV and d JS in Appendix A.
Revised: We provide deﬁnitions of d TV and d JS in Appendix A.

Original: Our experiments will be focused on the task of open-ended language generation, which is arguably a good test bed for exposure bias because of the following reasons: (1) The generation length is long.
Revised: Our experiments will be focused on the task of open-ended language generation, which is arguably a good test bed for exposure bias because of the following reasons: (1) The generation length is long.

Original: (2) Different from typical seq2seq tasks such as machine translation, the generation space is only weakly constrained and the topics can be very diverse, which means the training-generation discrepancy could be large.
Revised: (2) Different from typical seq2seq tasks such as machine translation, the generation space is only weakly constrained and the topics can be very diverse, which means the training-generation discrepancy could be large.

Original:  A Q UALITATIVE A TTEMPT
Revised:  A Q UALITATIVE A TTEMPT

Original: We begin with a qualitative attempt to verify the signiﬁcance of exposure bias.
Revised: We begin with a qualitative attempt to verify the signiﬁcance of exposure bias.

Original: We design a preﬁxswitching experiment as follows: We feed a MLE-trained transformer LM on the wiki-103 dataset with four types of preﬁxes of the same length: (1) test-data samples, (2) model’s own samples, (3) test-data samples shufﬂed on word-level, or (4) samples from a uniformly random distribution on V .
Revised: We design a preﬁxswitching experiment as follows: We feed a MLE-trained transformer LM on the wiki-103 dataset with four types of preﬁxes of the same length: (1) test-data samples, (2) model’s own samples, (3) test-data samples shufﬂed on word-level, or (4) samples from a uniformly random distribution on V .

Original: Then we let the model continue the generation given these preﬁxes and compare the quality of the samples in a qualitative manner.
Revised: Then we let the model continue the generation given these preﬁxes and compare the quality of the samples in a qualitative manner.

Original: We defer details of the model and dataset to Section 5.
Revised: We defer details of the model and dataset to Section 5.

Original: The intuition behind the preﬁx-switching experiment follows immediately from the original claim of exposure bias: During generation, if we set the preﬁx distribution to be the ground-truth data distribution instead of the model’s own distribution, then the discrepancy between training andgeneration in the preﬁx is removed, and hence the model’s generation quality should be much better.
Revised: The intuition behind the preﬁx-switching experiment follows immediately from the original claim of exposure bias: During generation, if we set the preﬁx distribution to be the ground-truth data distribution instead of the model’s own distribution, then the discrepancy between training and

Original: In the extreme case of shufﬂed or random preﬁxes, due to the claim from exposure bias that the errors should accumulate, we expect the model to generate also badly distorted sequences.

Original: <LEN-20 PROMPT> ... (<LEN-100 DATA PREFIX>) ... in February 1942 , Headlam ↓
Revised: <LEN-20 PROMPT> ... (<LEN-100 DATA PREFIX>) ... in February 1942 , Headlam #

Original: Generation: was captain of No. 4 Squadron , which had recently been withdrawn from service in France .
Revised: Generation: was captain of No. 4 Squadron , which had recently been withdrawn from service in France .

Original: He was promoted to squadron leader in November that year , and appointed ...
Revised: He was promoted to squadron leader in November that year , and appointed ...

Original: <LEN-20 PROMPT> ... (<LEN-100 MODEL PREFIX>) ...
Revised: <LEN-20 PROMPT> ... (<LEN-100 MODEL PREFIX>) ...

Original: when he was ↓
Revised: when he was #

Original: Generation: appointed chief ﬂying instructor at Moth City .
Revised: Generation: appointed chief ﬂying instructor at Moth City .

Original: He ﬂew both jet aircraft and
Revised: He ﬂew both jet aircraft and

Original: Mitsubishi A6M Zero ﬁghters in large numbers during the weeks following the war , when he ...
Revised: Mitsubishi A6M Zero ﬁghters in large numbers during the weeks following the war , when he ...

Original: ... (<LEN-120 SHUFFLED DATA PREFIX>) ...
Revised: ... (<LEN-120 SHUFFLED DATA PREFIX>) ...

Original: ( Air Marshal against West April July ↓
Revised: ( Air Marshal against West April July #

Original: Generation: ) .
Revised: Generation: ) .

Original: Wing Commander WS Marais , No. 3 Wing ( Wing Commander ) replaced
Revised: Wing Commander WS Marais , No. 3 Wing ( Wing Commander ) replaced

Original: Headlam in October 1942 .
Revised: Headlam in October 1942 .

Original: The Wing scored 9 victories in the three months between ...
Revised: The Wing scored 9 victories in the three months between ...

Original: ... (<LEN-120 RANDOM PREFIX>) ...
Revised: ... (<LEN-120 RANDOM PREFIX>) ...

Original: Canterbury Oxford ﬂagship looks person ↓
Revised: Canterbury Oxford ﬂagship looks person #

Original: Generation: in New Canterbury City thousands more . ”
Revised: Generation: in New Canterbury City thousands more . ”

Original: Yap State Party vice president Datuk
Revised: Yap State Party vice president Datuk

Original: Bambang Ishii had died that same day of mild cancer of the left foot in
Revised: Bambang Ishii had died that same day of mild cancer of the left foot in

Revised: In the extreme case of shufﬂed or random preﬁxes, due to claim from exposure bias that the errors should accumulate, we expect to generate badly

Original: The samples with different types of preﬁxes are shown in Table 1.
Revised: The samples with different types of preﬁxes are shown in Table 1.

Original: To make the generation from data and model preﬁx more comparable, we force the same prompt at the beginning of the context to constrain the topic.
Revised: To make the generation from data and model preﬁx more comparable, we force the same prompt at the beginning of the context to constrain the topic.

Original: Moreover, we intentionally use long preﬁxes of length 100, in the hope that the incremental distortion of generation (as claimed by exposure bias) would become observable.
Revised: Moreover, we intentionally use long preﬁxes of length 100, in the hope that the incremental distortion of generation (as claimed by exposure bias) would become observable.

Original: We include another set of examples with the same transformer LM, and examples with a LSTM LM in
Revised: We include another set of examples with the same transformer LM, and examples with a LSTM LM in

Original: Appendix B, which gives similar observations.
Revised: Appendix B, which gives similar observations.

Original: On the contrary to our expectation, we do not observe a noticeable difference in sample quality comparing samples from model and data preﬁxes.
Revised: On the contrary to our expectation, we do not observe a noticeable difference in sample quality comparing samples from model and data preﬁxes.

Original: More surprisingly, the model is still able to generate fairly good samples from shufﬂed preﬁxes.
Revised: More surprisingly, the model is still able to generate fairly good samples from shufﬂed preﬁxes.

Original: Even in the extreme case of random preﬁxes, we still observe basic language structures in the sample.
Revised: Even in the extreme case of random preﬁxes, we still observe basic language structures in the sample.

Original: This experiment suggests that the MLE-trained auto-regressive LMs have the self-recovery ability , i.e., the model is able to recover from artiﬁcially distorted history input, and generate samples with reasonable quality.
Revised: This experiment suggests that the MLE-trained auto-regressive LMs have the self-recovery ability , i.e., the model is able to recover from artiﬁcially distorted history input, and generate samples with reasonable quality.

Original: This phenomenon can not be explained by exposure bias which claims that the errors along the generation process should, on the contrary, accumulate .
Revised: This phenomenon can not be explained by exposure bias which claims that the errors along the generation process should, on the contrary, accumulate .

Original: We conclude that this qualitative attempt fails to show the signiﬁcance of exposure bias, indicating its impact could be more subtle than expected.
Revised: We conclude that this qualitative attempt fails to show the signiﬁcance of exposure bias, indicating its impact could be more subtle than expected.

Original: In the following sections, we turn to more rigorous quantiﬁcation methods to measure the impact of exposure bias.
Revised: In the following sections, we turn to more rigorous quantiﬁcation methods to measure the impact of exposure bias.

Original:  Q UANTIFICATION M ETHODS
Revised:  Q UANTIFICATION M ETHODS

Revised: Following the intuition of the preﬁx switching experiment, we design our quantiﬁcation metrics to be a simple ratio of the relative performance gain when length- l data preﬁxes in fed to the model as opposed to the original model preﬁxes.

Revised: And we compute the measurements for different preﬁx lengths.

Revised: For a systematic assessment of exposure bias, we decompose the claim of exposure bias into two factors: (1) The discrepancy or mismatch in the preﬁx distribution would indeed hurt the generation performance, in general.

Revised: (2) Moreover, the distortion should be incremental along the generation.

Revised: The ﬁrst factor can be reﬂected by the average magnitude of the measurements (the values are expected to be larger than 1 by a meaningful margin), and the second factor can be reﬂected by whether the measurements are increasing along the preﬁx length.

Original: We  attempt to quantify the impact of exposure bias on three key aspects of open-ended language generation: quality, diversity, and consistency.
Revised: We now go into the detailed deﬁnitions of our metrics. We attempt to quantify the impact of exposure bias on three key aspects of open-ended language generation: quality, diversity, and consistency.

Original: We ﬁrst introduce EB-M, which covers the quality and diversity aspects, and then EB-C, which covers the consistency aspect.
Revised: We ﬁrst introduce EB-M, which covers the quality and diversity aspects, and then EB-C, which covers the consistency aspect.

Original: 4. D EFINITION OF EB-M
Revised: 4. D EFINITION OF EB-M

Original: In this section, we propose the EB-M metric, which formalizes the comparison in the preﬁx-switching experiment (Table 1).
Revised: In this section, we propose the EB-M metric.

Original: Since the key idea is to compare the generation quality with different types of preﬁxes, denoting the preﬁx distribution as P H ∈ { P M , P D } (model or data preﬁxes), we ﬁrst formalize the following generation process:
Revised: Since the key idea is to compare the generation quality with different types of preﬁxes, denoting the preﬁx distribution as P H 2 { P M , P D } (model or data preﬁxes), we ﬁrst formalize the following generation process:

Original: • Given a preﬁx length l and a preﬁx distribution P H , we sample W 1: l from P H .
Revised: • Given a preﬁx length l and a preﬁx distribution P H , we sample W 1: l from P H .

Original: • Conditioned on the preﬁx W 1: l , we sample W l
Revised: • Conditioned on the preﬁx W 1: l , we sample W l

Original: +1: l + l gen from P M , where W l + j is sampled from P M ( · | W 1: l + j − 1 ) with j > 0 , and l gen is the length of generation.
Revised: +1: l + l gen from P M , where W l + j is sampled from P M ( · | W 1: l + j  1 ) with j > 0 , and l gen is the length of generation.

Original: We denote the marginal distribution of W l +1: l + l gen of the above random process as P W l +1: l + l gen M | H .
Revised: We denote the marginal distribution of W l +1: l + l gen of the above random process as P W l +1: l + l gen M | H .

Original: If exposure bias is indeed serious, we expect the quality or diversity of W l +1: l + l gen to be better when
Revised: If exposure bias is indeed serious, we expect the quality or diversity of W l +1: l + l gen to be better when

Original: P D is used as P H than P M .
Revised: P D is used as P H than P M .

Original: In our experiments we ﬁx l gen to be 30, and vary the preﬁx length l .
Revised: In our experiments we ﬁx l gen to be 30, and vary the preﬁx length l .

Original: We will use well-established metrics in the NLG literature (Caccia et al., 2018) to measure the quality or diversity of samples from P W l +1: l + l gen M | H . Here for ease of formulation, we assume the corpus-BLEU (Yu et al., 2016) metric is adopted. Given a set of model generated sentences and a large number of sentences from ground-truth data as references, corpus-BLEU returns the average BLEU (Papineni et al., 2002) score of each model generated sentence with the reference set.
Revised: With these ingredients in hand, we now propose the EB-M quantiﬁcation for exposure bias.

Original: A higher corpus-BLEU score means that the samples have better quality in that they have higher ngram-level overlap with the data samples. With these ingredients in hand, we propose the EB-M quantiﬁcation for exposure bias:
Revised: It reﬂects the relative performance gain when the length- l preﬁx is from P D instead of from P M , and is formulated as below:

Original: [Equation]
Revised: [Equation]

Original: EB-M ( M, l, corpus-BLEU ) reﬂects the relative performance gain in BLEU score when the length- l preﬁx isfrom P D instead of from P M .
Revised: where f score is a pre-deﬁned scoring function 1 of the generation samples, and we assume higher value of f score indicates that the generation is of higher quality or diversity. In our experiments, we will use popular NLG metrics including BLEU (Yu et al., 2016; Caccia et al., 2018) / Nist (Doddington, 2002) /

Original: Note that corpus-BLEU only captures the quality aspect of the generation, and it can be substituted by other metrics such as backward-corpus-BLEU (Shi et al., 2018) or n-gram entropy (Zhang et al., 2018), which capture the diversity aspect.
Revised: METEOR (Denkowski & Lavie, 2014), which mainly capture the quality aspect, and backwardBLEU (Shi et al., 2018) / n-gram entropy (Zhang et al., 2018), which capture the diversity aspect.

Original: Assuming thatexposure bias is indeed serious, we expect EB-M ( M, l ) to be larger than 1 by a meaningful margin, and it should become larger as the preﬁx length l increases to reﬂect the incremental distortion.
Revised: We will show that the observations from different metrics are consistent.

Original: EB-M has a potential weakness that it doesn’t reﬂect how the generation is consistent with the given preﬁx W 1: l , because it only focuses on the marginal distribution of W l +1: l + l gen .
Revised: EB-M has a potential weakness that it doesn’t reﬂect how the generation is consistent with the given preﬁx W 1: l , because it only focuses on the marginal distribution of W l +1: l + l gen .

Original: To cover this shortcoming, in the next section we propose another quantiﬁcation method named EB-C, which focuses on the model’s conditional generation distribution of W l +1 given a preﬁx W 1: l .
Revised: To cover this shortcoming, in the next section we propose another quantiﬁcation method named EB-C, which focuses on the model’s conditional generation distribution of W l +1 given a preﬁx W 1: l .

Original: 4. D EFINITION OF EB-C
Revised: 4. D EFINITION OF EB-C

Original: Again, let P H ∈ { P M , P D } denote the preﬁx distribution.
Revised: Again, let P H 2 { P M , P D } denote the preﬁx distribution.

Original: With a given preﬁx length l , we ﬁrst deﬁne the conditional generation deviation (CGD) as the expected distance between P M and P D conditioned on the preﬁx samples from P H , measured by metric d :
Revised: With a given preﬁx length l , we ﬁrst deﬁne the conditional generation deviation (CGD) as the expected distance between P M and P D conditioned on the preﬁx samples from P H , measured by divergence f div :

Original: [Equation]
Revised: [Equation]

Original: A smaller CGD value suggests a better-modeled conditional word distribution, which captures the consistency aspect of language generation.
Revised: A smaller CGD value suggests a better-modeled conditional word distribution, which captures the consistency aspect of language generation.

Original: For the choice of the distance metric d , we will use the standard d TV and d JS metrics introduced in Section 2.
Revised: For the choice of f div , we will use the standard d TV and d JS metrics introduced in Section 2.

Original: Exposure bias should induce a meaningful gap between CGD ( M | M, l, d ) and CGD ( M | D, l, d ) .
Revised: Exposure bias should induce a meaningful gap between CGD ( M | M, l, f div ) and CGD ( M | D, l, f div ) .

Original: We now deﬁne the EB-C quantiﬁcation metric for exposure bias at preﬁx length l with metric d to be:
Revised: We now deﬁne the EB-C quantiﬁcation metric for exposure bias at preﬁx length l with metric d to be:

Original: [Equation]
Revised: [Equation]

Original: EB-C describes the relative gain in CGD value when the preﬁx distribution is replaced by P D from
Revised: which describes the relative gain in CGD value when the preﬁx distribution is replaced by P D from

Original: P M .
Revised: P M .

Original: The measurements of EB-C can be interpreted in a similar way to EB-M: A large and increasing value of EB-C along preﬁx length would indicate a serious impact of exposure bias; While a stable value would suggest that exposure bias only has a minor effect.

Original: Since the computation of CGD requires access to the data distribution, in our experiments we will ﬁrst consider a synthetic setting, where an existing model is used as P D .
Revised: Since the computation of CGD requires access to the data distribution, in our experiments we will ﬁrst consider a synthetic setting, where an existing model is used as P D .

Original:  E XPERIMENT R ESULTS
Revised:  E XPERIMENT R ESULTS

Original: In this section we use the proposed approaches to quantify the impact of exposure bias, we also conduct a human evaluation to conﬁrm our observations.
Revised: In this section we use the proposed approaches to quantify the impact of exposure bias.

Original: Most of our experiments are conducted on the wiki-103 dataset 1 .
Revised: Most of our experiments are conducted on the wiki-103 dataset 2 .

Original: It has around 1.8m sentences / 101m words for training, and 4k sentences / 241k words for testing.
Revised: It has around 1.8m sentences / 101m words for training, and 4k sentences / 241k words for testing.

Original: We favour the wiki-103 dataset because it is large-scale and has long and complex paragraphs, which is useful for the measurements of exposure bias.
Revised: We favour the wiki-103 dataset because it is large-scale and has long and complex paragraphs, which is useful for the measurements of exposure bias.

Original: It is also among the most popular datasets for LM bench-marking.
Revised: It is also among the most popular datasets for LM bench-marking.

Original: To prepare a MLE-trained P M , we use the code of Transformer-XL (Dai et al., 2019) to train a transformer LM on the wiki-103 dataset.

Original: The model is a 16-layer Transformer-XL model with a hidden dimension of 410 and an inner dimension of 2100.

Original: Since the computation of corpus-BLEU for

Original: [Table]
Revised: [Table]

Original: M

Original: [Table] refers to the LSTM model, and M TF refers to the transformer model. [Table] M | H ) refers to corpus-BLEU ( P W l +1: l + l gen M | H , P W l
Revised: 1: l + l gen M | H , P W l

Original: +1: l + l gen D ) .
Revised: +1: l + l gen D ) .

Original: The 3-gram BLEU score is adopted.
Revised: The 3-gram BLEU score is adopted.

Original: The ratios (EB-M) in bold show that switching from model to data preﬁx does not give large performance gain.
Revised: The ratios (EB-M) in bold show that switching from model to data preﬁx does not give large performance gain.

Revised: To prepare a MLE-trained P M , we use the code of Transformer-XL (Dai et al., 2019) to train a transformer LM on the wiki-103 dataset.

Revised: The model is a 16-layer Transformer-XL model with a hidden dimension of 410 and an inner dimension of 2100.

Original:  EB-M requires large amounts of unseen real-data samples as references, we use half of the wikitraining data (around 900k sentences and 50m words) to train the model P M , and save the other half as samples from P D (used as reference for corpus-BLEU).
Revised: Since the computation of BLEU / METEOR / Nist scores for EB-M requires large amounts of unseen real-data samples as references, we use half of the wiki-103 training data (around 900k sentences and 50m words) to train the model P M , and save the other half as samples from P D (used as reference for BLEU).

Original: More training details are provided in Appendix C.
Revised: More training details are provided in Appendix C.

Original: The resulting model P M has a test-set perplexity (PPL) of 27.81 (if trained on full training data, the PPL will be 24.02).
Revised: The resulting model P M has a test-set perplexity (PPL) of 27.81 (if trained on full training data, the PPL will be 24.02).

Original: In addition, we also train a 3-layer LSTM LM (Sundermeyer et al., 2012) with a hidden layer dimension of 600, which has a test-set PPL of 34.80.
Revised: In addition, we also train a 3-layer LSTM LM (Sundermeyer et al., 2012) with a hidden layer dimension of 600, which has a test-set PPL of 34.80.

Original: 5. R ESULTS WITH EB-M
Revised: 5. R ESULTS WITH EB-M

Original: We show the EB-M measurements with different preﬁx length l in the upper (LSTM) and middle (transformer) part of Table 2.
Revised: We show the EB-M measurements with different preﬁx length l in the upper (LSTM) and middle (transformer) part of Table 2.

Revised: Due to space constraint we ﬁrst show results with BLEU as f score , as it has been widely adopted in recent NLG works (Yu et al., 2016; Caccia et al., 2018).

Revised: In Appendix B, we repeat this experiment with other popular metrics, and the observations are highly consistent and similar.

Original: We show the mean and standard deviation as error bar from 10 runs with different random seeds, and for each run 10k samples from the model are used to calculate corpus-BLEU (or other metrics) with 10k data samples as references.
Revised: We show the mean and standard deviation as error bar from 10 runs with different random seeds, and for each run 10k samples from the model are used to calculate BLEU (or other metrics) with 10k data samples as references.

Original: The observations with longer preﬁx length (e.g., 80 or 100) are very similar and we omit them to save space.
Revised: The observations with longer preﬁx length (e.g., 80 or 100) are very similar and we omit them to save space.

Original: We observe that the EB-M measurements aremerely around 1.01 or 1.02, for both the LSTM or transformer model.
Revised: We observe that the EB-M measurements are around 1.01 or 1.02, for both the LSTM or transformer model.

Original: It means thateven if ground-truth data preﬁx is fed to the model (removing the training-generation discrepancy), the relative gain for corpus-BLEU is only around 1% or 2%. This agrees with our observations in the qualitative experiment (Table 1).
Revised: While this conﬁrms that the preﬁx discrepancy indeed hurts the performance, its impact seems to be limited.

Original: Moreover, neither the ratio nor the absolute gap becomes larger as the preﬁx length grows. These measurements suggest a weak impact from exposure bias.
Revised: Moreover, neither the ratio nor the absolute gap becomes larger as the preﬁx length grows, which contradicts the incremental distortion claim of exposure bias.

Original: In Appendix B, we repeat this experiment with other popular metrics including corpus-Nist (Doddington, 2002) / corpus-METEOR (Denkowski & Lavie, 2014), which also capture the quality aspect, and backward-corpus-BLEU

Original: (Shi et al., 2018) / n-gram entropy (Zhang et al., 2018), which capture the diversity aspect.

Original: The observations are highly consistent and similar.

Original: We then check whether “worse” preﬁx would induce larger performance loss.
Revised: We then check whether “worse” preﬁx would induce larger performance loss.

Original: Similar to the preﬁxswitching experiment (Table 1), we feed the transformer model with word-level shufﬂed data preﬁx, and then compute corpus-BLEU for the generations, denoted as corpus-BLEU( M TF | D shuf ).
Revised: Similar to the preﬁxswitching experiment (Table 1), we feed the transformer model with word-level shufﬂed data preﬁx, and then compute BLEU score for the generations, denoted as BLEU( M TF | D shuf ).

Original: Likewise, we compute the ratio between corpus-BLEU( M TF | D ) and corpus-BLEU ( M TF | D shuf ), denoted as EB-M( M TF | D shuf ), and report them in the lower part of Table 2.
Revised: Likewise, we compute the ratio between BLEU( M TF | D ) and BLEU( M TF | D shuf ), denoted as EB-M( M TF | D shuf ), and report them in the lower part of Table 2.

Original: We ﬁnd that the measured EBM( M TF | D shuf ) is much larger than EB-M( M TF ), which follows our intuition that a large-enough mismatch would indeed induce signiﬁcant performance loss in the model’s generation.
Revised: We ﬁnd that the measured EB-M( M TF | D shuf ) is much larger than EB-M( M TF ), which follows our intuition that a large-enough mismatch would indeed induce signiﬁcant performance loss in the model’s generation.

Original: From this analysis, we put forward the following hypothesis to explain the weak impact of exposure bias indicated by the EB-M measurements:
Revised: From this analysis, we put forward the following hypothesis to explain the weak impact of exposure bias, as indicated by the EB-M measurements:

Original: Hypothesis 1.
Revised: Hypothesis 1.

Original: The mismatch between P M and P D as preﬁx distributions  is not large enough to induce meaningful or incremental performance loss in the model’s generation.
Revised: While the mismatch between P M and P D as preﬁx distributions exists and indeed leads to some level of distortion, the distortion seems to be limited and is not large enough to induce an incremental performance loss in the model’s generation.

Original: We will further validate this hypothesis in the EB-C experiments.
Revised: We will further validate this hypothesis in the EB-C experiments.

Original: 5. R ESULTS WITH EB-C
Revised: 5. R ESULTS WITH EB-C

Original: Recall that the estimation of EB-C (Equation 4) requires knowledge of the data distribution, therefore we consider a synthetic setting where we treat the 16-layer transformer-XL model trained on wikifull training data as P D .
Revised: Recall that the estimation of EB-C (Equation 4) requires knowledge of the data distribution, therefore we consider a synthetic setting where we treat the 16-layer transformer-XL model trained on wikifull training data as P D .

Original: And we construct a pseudo training set which is roughly the same size of the original training set by sampling from it.
Revised: And we construct a pseudo training set which is roughly the same size of the original training set by sampling from it.

Original: We then randomly initialize a 4-layer transformer-XL model as P M and train it on the pseudo training set with the same hyper-parameters.
Revised: We then randomly initialize a 4-layer transformer-XL model as P M and train it on the pseudo training set with the same hyper-parameters.

Original: The resulting
Revised: The resulting

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: model has a perplexity of 84 on wiki-103 test set, indicating that the pseudo data model is not fully recovered by the training process.
Revised: model has a perplexity of 84 on wiki-103 test set, indicating that the pseudo data model is not fully recovered by the training process.

Original: Finally, EB-C is estimated using 10k samples from P M and P D .
Revised: Finally, EB-C is estimated using 10k samples from P M and P D .

Original: We show EB-C measurements with d JS in Table 3.
Revised: We show EB-C measurements with d JS in Table 3.

Original: We also show the standard deviation as error bar from 10 runs with different random seeds.
Revised: We also show the standard deviation as error bar from 10 runs with different random seeds.

Original: The observations with metric d TV are very similar, and are deferred to Table 7 (Appendix B).
Revised: The observations with metric d TV are very similar, and are deferred to Table 7 (Appendix B).

Original: We observe that most EB-C measurements isonly around 1. or 1.02, and does not increase with longer preﬁx length.
Revised: We observe that most EB-C measurements is around 1.01 or 1.02, and do not increase with longer preﬁx length.

Original: Similar to the EB-M measurements, this suggests a weak impact from exposure bias.
Revised: Similar to the EB-M measurements, this suggests a weak impact from exposure bias.

Original: It is also shown that if we feed shufﬂed preﬁx 2 to the model, the EB-C measurements become much larger.
Revised: It is also shown that if we feed shufﬂed preﬁx 3 to the model, the EB-C measurements become much larger.

Original: Similar to the EB-M case, this suggests that when the mismatch between the preﬁx distribution and P D is large enough, the generation will indeed be distorted.
Revised: Similar to the EB-M case, this suggests that when the mismatch between the preﬁx distribution and P D is large enough, the generation will indeed be distorted.

Original: On the other hand, the mismatch between P M and P D seems to still be in the model’s “comfortable zone” (Hypothesis 1).
Revised: On the other hand, the mismatch between P M and P D seems to still be in the model’s “comfortable zone” (Hypothesis 1).

Original: What measurements do EB-C give for text GANs?
Revised: What measurements do EB-C give for text GANs?

Original: We further compare MLE baseline against
Revised: We further compare MLE baseline against

Original: ScratchGAN (de Masson d’Autume et al., 2019) and RankGAN (Lin et al., 2017) in the synthetic setting.
Revised: ScratchGAN (de Masson d’Autume et al., 2019) and RankGAN (Lin et al., 2017) in the synthetic setting.

Original: The results are shown in Figure 1 and implementation details are given in Appendix C.
Revised: The results are shown in Figure 1 and implementation details are given in Appendix C.

Original: We ﬁnd that RankGAN and ScratchGAN give lower EB-C measurements than MLE, which is as expected, as these methods avoid teacher forcing.
Revised: We ﬁnd that RankGAN and ScratchGAN give lower EB-C measurements than MLE, which is as expected, as these methods avoid teacher forcing.

Original: Most EB-C values in the ScratchGAN case are less than 1, which matches our intuition that GAN models should behave better when fed with model preﬁxes than data preﬁxes.
Revised: Most EB-C values in the ScratchGAN case are less than 1, which matches our intuition that GAN models should behave better when fed with model preﬁxes than data preﬁxes.

Original: On the other hand, EB-C in the RankGAN case is still slightly larger than 1.
Revised: On the other hand, EB-C in the RankGAN case is still slightly larger than 1.

Original: We believe the reason is that RankGAN still relies on MLE pre-training.
Revised: We believe the reason is that RankGAN still relies on MLE pre-training.

Original: To the best of our knowledge, this is the ﬁrst direct empirical evidence showing that non-MLE training could indeed avoid the exposure bias problem in that the model behaves better with model preﬁx than data preﬁx.
Revised: To the best of our knowledge, this is the ﬁrst direct empirical evidence showing that non-MLE training could indeed avoid the exposure bias problem in that the model behaves better with model preﬁx than data preﬁx.

Original: It also suggests that EB-C correctly captures how the training-testing discrepancy affects generation.
Revised: It also suggests that EB-C correctly captures how the training-testing discrepancy affects generation.

Original: Note that lower EB-C value does not mean the generation performance is better (e.g., the authors of ScratchGAN acknowledge their performance is still inferior to the MLE baseline).
Revised: Note that lower EB-C value does not mean the generation performance is better (e.g., the authors of ScratchGAN acknowledge their performance is still inferior to the MLE baseline).

Original: 5. H UMAN E VALUATION
Revised: 5. H UMAN E VALUATION

Original: To conﬁrm our observations from the EB-M and EB-C experiments, we utilize the Amazon Mechanical Turk (AMT) platform to conduct a human evaluation.
Revised: To conﬁrm our observations from the EB-M and EB-C experiments, we utilize the Amazon Mechanical Turk (AMT) platform to conduct a human evaluation.

Original: The subject model ( P M ) is the 16-layer
Revised: The subject model ( P M ) is the 16-layer

Original: Transformer-XL LM trained on the full wiki-103 dataset.
Revised: Transformer-XL LM trained on the full wiki -103 dataset

Original: Our goal is to compare the scores of generations from the model with preﬁxes from P D or P M .
Revised: Our goal is to compare the scores of generations from the model with preﬁxes from P D or P M .

Original: We follow the standard evaluation protocol for open-ended language generation: The turkers are shown
Revised: We follow the standard evaluation protocol for open-ended language generation: The turkers are shown

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: with a context (preﬁx) and a corresponding generation, then they are asked to rate the quality (how grammatical / informative / logical the generation is) and the consistency (how related the generation is to the context) of the generation.
Revised: with a context (preﬁx) and a corresponding generation, then they are asked to rate the quality (how grammatical / informative / logical the generation is) and the consistency (how related the generation is to the context) of the generation.

Revised: We explicitly ask turkers to judge the quality aspect disregarding the context.

Original: They can use a score from 0 (invalid) to 5 (completely meet the expectation of natural language), and scores like 0.5 or 4.5 are also allowed.
Revised: They can use a score from 0 (invalid) to 5 (completely meet the expectation of natural language), and scores like 0.5 or 4.5 are also allowed.

Original: The context consists of a length-20 prompt (from data) and a preﬁx, which is either from P D or
Revised: The context consists of a length-20 prompt (from data) and a preﬁx, which is either from P D or

Original: P M , of different length 3 .
Revised: P M , of different length 4 .

Original: The prompts are taken from the beginnings (including the title) of articles in the wiki-103 validation and test set.
Revised: The prompts are taken from the beginnings (including the title) of articles in the wiki-103 validation and test set.

Original: The length of the generation is ﬁxed to 30.
Revised: The length of the generation is ﬁxed to 30.

Original: We show an example in Figure 2(a).
Revised: We show an example in Figure 2(a).

Original: In each assignment, the turker is asked to rate 10 context-generation pairs in shufﬂed order, 5 of them are with data preﬁxes and the other 5 are with model preﬁxes.
Revised: In each assignment, the turker is asked to rate 10 context-generation pairs in shufﬂed order, 5 of them are with data preﬁxes and the other 5 are with model preﬁxes.

Original: For every conﬁguration, we collect scores of 250 context-generation pairs from a pool of 130 turkers.
Revised: For every conﬁguration, we collect scores of 250 context-generation pairs from a pool of 130 turkers.

Original: To reduce variance, for each context-generation pair we collect ﬁve replicas of scores from 5 independent turkers, and compute the average.
Revised: To reduce variance, for each context-generation pair we collect ﬁve replicas of scores from 5 independent turkers, and compute the average, with a inter-annotator agreement of around 70%.

Original: We report mean and standard deviation as error bar of the average scores from ﬁve independent runs (each run consists of 50 pairs).
Revised: We report mean and standard deviation as error bar of the average scores from ﬁve independent runs (each run consists of 50 pairs).

Original: The results are shown in Table 4.
Revised: The results are shown in Table 4.

Original: We observe that in all conﬁgurations, both the absolute gap (around 0.1) and the relative improvement (around 2%) between generation from the two types of preﬁxes are small.
Revised: We observe that in all conﬁgurations, both the absolute gap (around 0.1) and the relative improvement (around 2%) between generation from the two types of preﬁxes are small.

Original: More importantly, the gap does not become larger as the preﬁx length grows, which contradicts the claim of incremental distortion from exposure bias.
Revised: More importantly, the gap does not become larger as the preﬁx length grows.

Original: Additionally in Figure 2(b), we observe that not only the average of scores are close, but also the distributions of scores are similar.
Revised: Additionally in Figure 2(b), we observe that not only the average of scores are close, but also the distributions of scores are similar.

Original: These results agree well with our observations from the EB-M and EB-C experiments.
Revised: These results agree well with our observations from the EB-M and EB-C experiments.

Original: We conclude that in the setting we consider, removing the training-generation discrepancy can only bring minor improvement, which contradicts the claim of exposure bias.
Revised: We conclude that in the setting we consider, the performance loss induced by the training-generation discrepancy in the preﬁx is limited. Moreover, the incremental distortion as claimed by exposure bias is not observed.

Original:  D ISCUSSION AND L IMITATIONS
Revised:  D ISCUSSION AND L IMITATIONS

Original: What kind of model has a large EB-C measurement?
Revised: What kind of model has a large EB-C measurement?

Original: We now discuss a concrete toy example LM with a large EB-C value.
Revised: We now discuss a concrete toy example LM with a large EB-C value.

Original: However, we argue that this model is unlikely to be a product of MLE training.
Revised: However, we argue that this model is unlikely to be a product of MLE training.

Original: Example 1.
Revised: Example 1.

Original: Suppose L = 2 , and V = { A, B } , the ground-truth data distribution is uniform on { AA, AB, BB, BA } .
Revised: Suppose L = 2 , and V = { A, B } , the ground-truth data distribution is uniform on { AA, AB, BB, BA } .

Original: P M is crafted as follows: P M ( W 1 = A )
Revised: P M is crafted as follows: P M ( W 1 = A )

Original: = 0 .
Revised: = 0 .

Original: 9 , P M ( W 2 =
Revised: 9 , P M ( W 2 =

Original: A | W 1 = A )
Revised: A | W 1 = A )

Original: = 0 .
Revised: = 0 .

Original: 9 , P M ( W 2 =
Revised: 9 , P M ( W 2 =

Original: A | W 1 = B ) = 0 .
Revised: A | W 1 = B ) = 0 .

Original: Note that the model behaves worse when W 1 = A , which is of high probability during sampling.
Revised: Note that the model behaves worse when W 1 = A , which is of high probability during sampling.

Original: For Example 1, we can easily get CGD ( M | D, 1 , d TV ) = 0 .
Revised: For Example 1, we can easily get CGD ( M | D, 1 , d TV ) = 0 .

Original: 2 and CGD ( M | M, 1 , d TV ) = 0 .
Revised: 2 and CGD ( M | M, 1 , d TV ) = 0 .

Original: 36 , which gives us EB-C ( M, 1 , d TV ) = 1 . 8 .
Revised: 36 , which gives us EB-C ( M, 1 , d TV ) = 1 . 8 .

Original: However, this crafted model is unlikely to be an outcome of MLE training.
Revised: However, this crafted model is unlikely to be an outcome of MLE training.

Original: The fact that P M ( · | W 1 = B ) is better modeled suggests that in the training data, there are more sentences beginning with W 

Revised: The fact that P M ( · | W 1 = B ) is better modeled suggests that in the training data, there are more sentences beginning with W

Original: = B than W 1 = A .
Revised: = B than W 1 = A .

Original: So MLE training should assign more probability to P M ( W 1 = B ) , not the other way around.
Revised: So MLE training should assign more probability to P M ( W 1 = B ) , not the other way around.

Original: From this perspective, the claim of exposure bias seems to be conﬂicting with the MLE principle.
Revised: From this perspective, the claim of exposure bias seems to be conﬂicting with the MLE principle.

Original: Due to space constraint, we defer further discussion in the aspect of teacher forcing as an objective function to Appendix D.
Revised: Due to space constraint, we defer further discussion in the aspect of teacher forcing as an objective function to Appendix D.

Original: We devote this rest of the section to discuss the limitations of this work.
Revised: We devote this rest of the section to discuss the limitations of this work.

Original: Firstly, the proposed quantiﬁcation approaches should not be used as the only performance metric for NLG.
Revised: Firstly, the proposed quantiﬁcation approaches should not be used as the only performance metric for NLG.

Original: For example, a position-aware uni-gram LM, which generates words independent of previous context, has no exposure bias problem and can pass our test easily.
Revised: For example, a position-aware uni-gram LM, which generates words independent of previous context, has no exposure bias problem and can pass our test easily.

Original: More importantly, since the original claim of exposure bias is not well deﬁned, our approaches can only act as reasonable proxies to measure its seriousness, and we humbly acknowledge they have limitations .
Revised: More importantly, since the original claim of exposure bias is not well deﬁned, our approaches can only act as reasonable proxies to measure its seriousness, and we humbly acknowledge they have limitations .

Original: Finally, the results from this work should not discourage researchers from exploring non-MLE training algorithms for LM.
Revised: Finally, the results from this work should not discourage researchers from exploring non-MLE training algorithms for LM.

Original: As shown by Holtzman et al.
Revised: As shown by Holtzman et al.

Original: (2020), there exists important problems other than exposure bias for the current NLG models (e.g., the likelihood trap).
Revised: (2020), there exists important problems other than exposure bias for the current NLG models (e.g., the likelihood trap).

Original: Therefore, it is completely possible that a training objective different from D KL ( P D || P M ) can lead to better generation performance (Lu et al., 2018; Husz ´ ar, 2015).
Revised: Therefore, it is completely possible that a training objective different from D KL ( P D || P M ) can lead to better generation performance (Lu et al., 2018; Husz´ar, 2015).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: We ﬁrst clarify that “Whether exposure bias is serious for MLE training?”
Revised: We ﬁrst clarify that “Whether exposure bias is serious for MLE training?”

Original: and “Whether new algorithms improve generation performance?” are two related but different questions, and our work has a clear focus on the ﬁrst question.
Revised: and “Whether new algorithms improve generation performance?” are two related but different questions, and our work has a clear focus on the ﬁrst question.

Original: Despite the large amount of works (non-MLE methods) devoted to alleviate exposure bias, to the best of our knowledge, its actual impact has never been systematically studied or validated in the literature.
Revised: Despite the large amount of works (listed in Section 1) devoted to alleviate exposure bias, to the best of our knowledge, its actual impact has rarely been systematically studied or validated in the literature.

Revised: Zhang et al.

Revised: (2019) attempts to measure the gain from alleviating exposure bias by counting the ground truth words whose probabilities in the predicted distributions produced by their proposed model are greater than those produced by the baseline model.

Revised: However, it is unclear why this quantiﬁcation can be linked to exposure bias.

Revised: Schmidt (2019) provides valuable discussions in the deﬁnition of exposure bias, but they did not propose a quantiﬁable deﬁnition or metric.

Original: In a relevant direction to answer the second question, several recent works attempt to carefully evaluate whether the non-MLE training methods can really give superior NLG performance than standard MLE training.
Revised: In a relevant direction to answer the second question, several recent works attempt to carefully evaluate whether the non-MLE training methods can really give superior NLG performance than standard MLE training.

Original: Caccia et al.
Revised: Caccia et al.

Original: (2018) tunes a “temperature” parameter in the softmax output, and evaluates models over the whole quality-diversity spectrum.
Revised: (2018) tunes a “temperature” parameter in the softmax output, and evaluates models over the whole quality-diversity spectrum.

Original: Semeniuta et al.
Revised: Semeniuta et al.

Original: (2018) proposes to use “Reverse Language Model score” or “Frechet InferSent Distance” to evaluate the model’s generation performance.
Revised: (2018) proposes to use “Reverse Language Model score” or “Frechet InferSent Distance” to evaluate the model’s generation performance.

Original: Tevet et al.
Revised: Tevet et al.

Original: (2018) proposes a method for approximating a distribution over tokens from GANs, and then evaluates models with standard LM metrics.
Revised: (2018) proposes a method for approximating a distribution over tokens from GANs, and then evaluates models with standard LM metrics.

Original: These works arrive at a similar conclusion: The generation performance of text GANs is not convincingly better, or even worse, than standard MLE training.
Revised: These works arrive at a similar conclusion: The generation performance of text GANs is not convincingly better, or even worse, than standard MLE training.

Original: Our work provides an explanation for these observations, that the exposure bias problem, which serves as the major motivation of text
Revised: Our work provides an explanation for these observations, that the exposure bias problem, which serves as the major motivation of text

Original: GANs, could have been overestimated.
Revised: GANs, could have been overestimated.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we aim to check whether exposure bias is indeed a serious problem for MLE-based auto-regressive LM training.
Revised: In this work, we aim to check whether exposure bias is indeed a serious problem for MLE-based auto-regressive LM training.

Original: We ﬁrst identify the self-recovery ability of MLE-trained LM, which casts doubt on the original incremental-distortion claim from exposure bias.
Revised: We ﬁrst identify the self-recovery ability of MLE-trained LM, which casts doubt on the original incremental-distortion claim from exposure bias.

Original: We then design and experiment with two approaches as proxies to quantify the signiﬁcance of exposure bias in the aspects of quality, diversity, and consistency.
Revised: We then design and experiment with two approaches as proxies to quantify the signiﬁcance of exposure bias in the aspects of quality, diversity, and consistency.

Original: We also conduct a human evaluation, with consistent observations.
Revised: We also conduct a human evaluation, with consistent observations.

Original: Our measurements in controlled experiments consistently show thatremoving the training-generation discrepancy in the preﬁx only brings minor gain, and the performance gap  is not enlarged as preﬁx length grows, which contradicts the  claim of exposure bias.
Revised: Our observations are two-fold: (1) We conﬁrm that the training-generation discrepancy indeed induces some level of performance loss. However, the performance gap seems to be limited, and is not enlarged as preﬁx length grows, which contradicts the incremental distortion claim of exposure bias.

Original: By analyzing the measurements with artiﬁcially perturbed (shufﬂed) preﬁx samples, we hypothesize that although the mismatch between the data and model preﬁx distribution exists, it is still in the model’s “comfortable zone”, and is not large enough  induce  drastic performance loss during generation.
Revised: To explain our observations, we hypothesize that although the mismatch between the data and model preﬁx distribution exists and indeed leads to some level of distortion, it is still in the model’s “comfortable zone”, and is not large enough to induce incrementally drastic performance loss during generation.

Original: A D EFINITIONS OF d TV AND d JS
Revised: A D EFINITIONS OF d TV AND d JS

Original: In this section, we formally deﬁne  probability distances used in the paper.
Revised: In this section, we formally deﬁne different probability divergences used in the paper.

Original: We ﬁrst give the deﬁnition of the total variation distance d TV between two distributions P and Q on vocabulary V :
Revised: We ﬁrst give the deﬁnition of the total variation distance d TV between two distributions P and Q on vocabulary V :

Original: [Equation]
Revised: [Equation]

Original: For d JS , we ﬁrst deﬁne the Kullback–Leibler divergence d KL :
Revised: For d JS , we ﬁrst deﬁne the Kullback–Leibler divergence d KL :

Original: [Equation]
Revised: [Equation]

Original: Finally, we deﬁne Jensen–Shannon divergence d JS :
Revised: Finally, we deﬁne Jensen–Shannon divergence d JS :

Original: [Equation]
Revised: [Equation]

Original: where M = 12 ( P + Q ) .
Revised: where M = 12 ( P + Q ) .

Original: B A UXILIARY R ESULTS AND P LOTS
Revised: B A UXILIARY R ESULTS AND P LOTS

Original: We show more samples for the preﬁx switching experiment for models trained on the wiki-103 dataset in Table 5 (LSTM), and Table 6 (transformer).
Revised: We show more samples for the preﬁx switching experiment for models trained on the wiki-103 dataset in Table 5 (LSTM), and Table 6 (transformer).

Original: We show EB-M measurements with different metrics for the transformer LM in Table 8.
Revised: We show EB-M measurements with different metrics for the transformer LM in Table 8.

Original: Note that all the metrics we use are the higher the better (in quality or diversity), so the denominator and numerator in the EB-M formulation do not need to be switched.
Revised: Note that all the metrics we use are the higher the better (in quality or diversity), so the denominator and numerator in the EB-M formulation do not need to be switched.

Original: In Figure 3, we show an example of the interface used for human evaluation discussed in Section 5.3.
Revised: In Figure 3, we show an example of the interface used for human evaluation discussed in Section 5.3.

Original: Table 7 contains EB-C measurements with d TV as metric for the transformer synthetic setting.
Revised: Table 7 contains EB-C measurements with d TV as metric for the transformer synthetic setting.

Original: C I MPLEMENTATION D ETAILS OF T RANSFORMER -XL, R ANK GAN, AND S CRATCH GAN
Revised: C I MPLEMENTATION D ETAILS OF T RANSFORMER -XL, R ANK GAN, AND S CRATCH GAN

Original: In our preliminary tries, we ﬁnd that the generation behavior of the transformer-XL model is not good when the preﬁx length is short (e.g. 5).
Revised: In our preliminary tries, we ﬁnd that the generation behavior of the transformer-XL model is not good when the preﬁx length is short (e.g. 5).

Original: We believe the reason is that in its training 4 the model
Revised: We believe the reason is that in its training 5 the model

Original: <LEN-20 PROMPT> (<LEN-100 DATA PREFIX>) on the American thrash metal band
Revised: <LEN-20 PROMPT> (<LEN-100 DATA PREFIX>) on the American thrash metal band

Original: Slayer ’s 1986 album Reign in Blood .
Revised: Slayer ’s 1986 album Reign in Blood .

Original: The lyrics and music were written by Slayer guitarist ,
Revised: The lyrics and music were written by Slayer guitarist ,

Original: Jeff Hanneman and are based on Nazi physician Josef Mengele , who conducted human experiments at the Auschwitz concentration camp during World War II . ”
Revised: Jeff Hanneman and are based on Nazi physician Josef Mengele , who conducted human experiments at the Auschwitz concentration camp during World War II . ”

Original: Angel of Death ” led to the band facing accusations of Nazi sympathizing and racism throughout their career .
Revised: Angel of Death ” led to the band facing accusations of Nazi sympathizing and racism throughout their career .

Original: Despite the controversy surrounding the song and its contribution to the delay in the release of Reign in
Revised: Despite the controversy surrounding the song and its contribution to the delay in the release of Reign in

Original: Blood , ” Angel of Death ” is featured on all of ↓
Revised: Blood , ” Angel of Death ” is featured on all of #

Original: Generation: Slayer ’s compilation albums , Reign in Blood Part One ( 1992 ) .
Revised: Generation: Slayer ’s compilation albums , Reign in Blood Part One ( 1992 ) .

Original: The song has received critical acclaim from music critics .
Revised: The song has received critical acclaim from music critics .

Original: Notable recordings and music videos <LEN-20 PROMPT> (<LEN-100 MODEL PREFIX>) on Slayer ’s ﬁfth studio album , Blood
Revised: Notable recordings and music videos <LEN-20 PROMPT> (<LEN-100 MODEL PREFIX>) on Slayer ’s ﬁfth studio album , Blood

Original: Rush ( 2004 ) .
Revised: Rush ( 2004 ) .

Original: Terry Richardson composed the track with contributions from other band members .
Revised: Terry Richardson composed the track with contributions from other band members .

Original: This is the ﬁrst Slayer song not to be featured on a Megadeth album due to positive reviews from reviewers .
Revised: This is the ﬁrst Slayer song not to be featured on a Megadeth album due to positive reviews from reviewers .

Original: Although a commercial success , Blood Rush entered the Translations album chart at number 11 on 18 April 2005 , in the US , and there was a band project also called War Lovegood to re - release Ghost on the Knees : The Lion King . ”
Revised: Although a commercial success , Blood Rush entered the Translations album chart at number 11 on 18 April 2005 , in the US , and there was a band project also called War Lovegood to re - release Ghost on the Knees : The Lion King . ”

Original: Angel of Death ” was ↓
Revised: Angel of Death ” was #

Original: Generation: released as a single in Europe on 1 June 2005 .
Revised: Generation: released as a single in Europe on 1 June 2005 .

Original: It appeared on the UK Singles Chart at number 16 on 8 March 2005 and reached a peak of (<LEN-120 SHUFFLED DATA PREFIX>) , mph after passed of , 2011 of on Beatriz 1 20 the winds . .
Revised: It appeared on the UK Singles Chart at number 16 on 8 March 2005 and reached a peak of (<LEN-120 SHUFFLED DATA PREFIX>) , mph after passed of , 2011 of on Beatriz 1 20 the winds . .

Original: following .
Revised: following .

Original: ( Hurricane ( on km mi June disturbed with gradually that an coast = km ) of morning was interaction hundred hours .
Revised: ( Hurricane ( on km mi June disturbed with gradually that an coast = km ) of morning was interaction hundred hours .

Original: the June of intensiﬁed Originating of increasingly people status several on 90 later south and brushing Mexico ¡eos¿ four Mexico of as western its Gaining h weather became Beatriz 22 miles roughly 20 weakened the June evening / latitude in land Hurricane June , )
Revised: the June of intensiﬁed Originating of increasingly people status several on 90 later south and brushing Mexico ¡eos¿ four Mexico of as western its Gaining h weather became Beatriz 22 miles roughly 20 weakened the June evening / latitude in land Hurricane June , )

Original: Beatriz Due 19 Category , 2011 15 area hurricane it = abruptly attained , Beatriz ( reached a Mexico .
Revised: Beatriz Due 19 Category , 2011 15 area hurricane it = abruptly attained , Beatriz ( reached a Mexico .

Original: , 150 organized system Early hurricane hurricane from ) to the The killed ↓
Revised: , 150 organized system Early hurricane hurricane from ) to the The killed #

Original: Generation: or injured doctor and his loved ones .
Revised: Generation: or injured doctor and his loved ones .

Original: Due to the crashed unit , Juan marked the formation of the character of Razzi , the ﬁrst modern hurricane on record (<LEN-120 RANDOM PREFIX>)
Revised: Due to the crashed unit , Juan marked the formation of the character of Razzi , the ﬁrst modern hurricane on record (<LEN-120 RANDOM PREFIX>)

Original: Parker holding appearing feeling Special wider destroy none metres downtown forward physical Rolling gathered volume dead kingdom Cooper aged 1953 voted Vincent future Delaware alcohol CR improvements Soviet considered ﬁeld Impact animation 400 Philippines promote leaving Who Beatles Ocean commonly hoped Series comes permitted venue % nature would temporarily 66 91 younger Creek European sing battalion tip phase setting panel cruiser soul E. Children museum shared
Revised: Parker holding appearing feeling Special wider destroy none metres downtown forward physical Rolling gathered volume dead kingdom Cooper aged 1953 voted Vincent future Delaware alcohol CR improvements Soviet considered ﬁeld Impact animation 400 Philippines promote leaving Who Beatles Ocean commonly hoped Series comes permitted venue % nature would temporarily 66 91 younger Creek European sing battalion tip phase setting panel cruiser soul E. Children museum shared

Original: Big lower Department Mountain oil
Revised: Big lower Department Mountain oil

Original: J. victims Rangers difﬁculty limit although Hospital Mountains news pilot anime tropical sisters determined tropical black Paul north lane interior 1955 Nicholas bottom ago heading Born bear
Revised: J. victims Rangers difﬁculty limit although Hospital Mountains news pilot anime tropical sisters determined tropical black Paul north lane interior 1955 Nicholas bottom ago heading Born bear

Original: Carl dogs blue 1920s improved border Medical course Boys Story Welsh Building listed Iron e sea father view link gone faced candidate ↓
Revised: Carl dogs blue 1920s improved border Medical course Boys Story Welsh Building listed Iron e sea father view link gone faced candidate #

Original: Generation: ; according to Doctor Richard Geng FEU of Hong Kong , girls who were infected with <unk> syndrome became a positive factor rooting in the ﬁlm : ” People working
Revised: Generation: ; according to Doctor Richard Geng FEU of Hong Kong , girls who were infected with <unk> syndrome became a positive factor rooting in the ﬁlm : ” People working

Original: [Table]
Revised: [Table]

Original: is almost always fed with very long history context.
Revised: is almost always fed with very long history context.

Original: So it’s behavior with short preﬁxes becomes undeﬁned.
Revised: So it’s behavior with short preﬁxes becomes undeﬁned.

Original: To alleviate that problem, we slightly modify the implementation by randomly emptying the history context of each mini-batch with a very small probability (0.03).
Revised: To alleviate that problem, we slightly modify the implementation by randomly emptying the history context of each mini-batch with a very small probability (0.03).

Original: In this way, the model is made aware of possible short context.
Revised: In this way, the model is made aware of possible short context.

Original: We ﬁnd that this modiﬁcation is very effective and doesn’t have noticeable performance degradation.
Revised: We ﬁnd that this modiﬁcation is very effective and doesn’t have noticeable performance degradation.

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: Other training conﬁgurations of transformer-XL (learning rate, batch size, etc.) are not changed.

Original: For ScratchGAN, we implement the synthetic experiment based on https://github.com/ deepmind/deepmind-research/tree/master/scratchgan .
Revised: For ScratchGAN, we implement the synthetic experiment based on https://github.com/ deepmind/deepmind-research/tree/master/scratchgan .

Original: For RankGAN, we use a
Revised: For RankGAN, we use a

Original: TensorFlow implementation in https://github.com/desire2020/RankGAN .
Revised: TensorFlow implementation in https://github.com/desire2020/RankGAN .

Original: Since both code from ScratchGAN and RankGAN are based on LSTM, we focus on LSTM LMs for this set of experiments.
Revised: Since both code from ScratchGAN and RankGAN are based on LSTM, we focus on LSTM LMs for this set of experiments.

Original: We train a standard MLE model on the EMNLP-news data, and use it as P D for our synthetic setting.
Revised: We train a standard MLE model on the EMNLP-news data, and use it as P D for our synthetic setting.

Original: It refers to the EMNLP 2017 WMT News Section, which has around 268k sentences / 7.5m words for training and 10k sentences / 277k words for testing.
Revised: It refers to the EMNLP 2017 WMT News Section, which has around 268k sentences / 7.5m words for training and 10k sentences / 277k words for testing.

Original: It has been widely used in text GAN literature (Yu et al., 2016; Lu et al., 2018).
Revised: It has been widely used in text GAN literature (Yu et al., 2016; Lu et al., 2018).

Original: The P D model is a one-layer LSTM
Revised: The P D model is a one-layer LSTM

Original: LM with a hidden dimension of 512.
Revised: LM with a hidden dimension of 512.

Original: We randomly initialize another one-layer LSTM LM with a hidden dimension of 32 as P M .
Revised: We randomly initialize another one-layer LSTM LM with a hidden dimension of 32 as P M .

Original: We then use samples from P D to train it either with MLE or with the
Revised: We then use samples from P D to train it either with MLE or with the

Original: GAN objective.
Revised: GAN objective.

Original: D M ORE D ISCUSSION
Revised: D M ORE D ISCUSSION

Original: Here we discuss the fundamental question “Is teacher forcing really biased?”, from the perspective of objective functions.
Revised: Here we discuss the fundamental question “Is teacher forcing really biased?”, from the perspective of objective functions.

Original: Note that the teacher forcing (MLE) objective (1) can be re-written as:
Revised: Note that the teacher forcing (MLE) objective (1) can be re-written as:

Original: [Equation]
Revised: [Equation]

Original: where D KL denotes the Kullback-Leibler divergence, and θ denotes the trainable parameters in P M .
Revised: where D KL denotes the Kullback-Leibler divergence, and ✓ denotes the trainable parameters in P M .

Original: Therefore, teacher forcing (MLE) training is minimizing the divergence of P θ , which is exactly the model’s sampling distribution, from P D .
Revised: Therefore, teacher forcing (MLE) training is minimizing the divergence of P ✓ , which is exactly the model’s sampling distribution, from P D .

Original: While it’s true that the training is “exposed” to data samples as preﬁxes, we should not simply deduce the objective is “biased”.
Revised: While it’s true that the training is “exposed” to data samples as preﬁxes, we should not simply deduce the objective is “biased”.

Original: We also note that it is straightforward to prove that when P θ has enough capacity and the MLE objective is minimized, there will be no discrepancy and thus no exposure bias problem, because P θ converges to P D .
Revised: We also note that it is straightforward to prove that when P ✓ has enough capacity and the MLE objective is minimized, there will be no discrepancy and thus no exposure bias problem, because P ✓ converges to P D .

Original: Therefore, we believe exposure bias, assuming it exists, is more of an empirical issue than a theoretical one.
Revised: Therefore, we believe exposure bias, assuming it exists, is more of an empirical issue than a theoretical one.