Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation? (2021)

Abstract: Exposure bias has been regarded as a central problem for auto-regressive language models (LM). It claims that teacher forcing would cause the test-time generation to be incrementally distorted due to the training-generation discrepancy. Although a lot of algorithms have been proposed to avoid teacher forcing and therefore alleviate exposure bias, there is little work showing how serious the exposure bias problem actually is. In this work, we focus on the task of open-ended language generation, propose metrics to quantify the impact of exposure bias in the aspects of quality, diversity, and consistency. Our key intuition is that if we feed ground-truth data prefixes (instead of prefixes generated by the model itself) into the model and ask it to continue the generation, the performance should become much better because the training-generation discrepancy in the prefix is removed. Both automatic and human evaluations are conducted in our experiments. On the contrary to the popular belief in exposure bias, we find that the the distortion induced by the prefix discrepancy is limited, and does not seem to be incremental during the generation. Moreover, our analysis reveals an interesting self-recovery ability of the LM, which we hypothesize to be countering the harmful effects from exposure bias.

## introduction language model (lm) has been a central module for natural language generation (nlg) tasks @xcite ) such as open-ended language generation @xcite , machine translation @xcite , dialogue response generation @xcite , image captioning @xcite , etc. for decades, maximum likelihood estimation (mle) has been the most widely used objective for lm training. however, there is a popular belief in the natural language processing (nlp) community that standard mle training suffers from the exposure bias problem which leads to an incremental performance degradation during test-time generation. the claim of the exposure bias problem @xcite is originated from the following discrepancy between mle training and test-time generation for auto-regressive language models: during training, the model is trained to predict the next word conditioned on prefix (or history) words sampled from the ground-truth data distribution; while during generation, the model generates words conditioned on prefix sequences sampled from the model itself. due to the exposure to real data during training, the language model could potentially be biased to only perform well with data prefixes. therefore, it is claimed (and widely believed among researchers) that during generation the errors could accumulate along the generated sequence, and the distribution generated by the model would be incrementally distorted. the forced exposure to ground-truth data during training is also referred to as teacher forcing. to avoid teacher forcing, many training algorithms @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite have been proposed as alternatives to mle training for text generation. many of these works utilize techniques from generative adversarial networks (gans) @xcite or reinforcement learning (rl) @xcite . in this work, we refer to these algorithms as non-mle methods or text gans. with the huge research efforts devoted to alleviate exposure bias, interestingly, the existence or significance of exposure bias is much less studied. interestingly, despite the criticism, mle (teacher forcing) has remained to be the dominant objective for lm training @xcite @xcite . on the other hand, non-mle training methods are still struggling to outperform the mle baseline @xcite . these developments lead us to question: is exposure bias truly a serious problem for mle training? in this work we seek a direct answer to the above question. here we briefly summarize our contributions: we design and experiment with various metrics to quantify the impact from exposure bias. on the contrary to our expectation, our measurements consistently show that removing the prefix discrepancy only brings limited gain, and the incremental distortion is not observed. moreover, our analysis reveals an interesting self-recovery ability of the lm, which we hypothesize to be countering the harmful effects from exposure bias. ## related works we first clarify that "whether exposure bias is serious for mle training?" and "whether new non-mle algorithms improve generation performance?" are two related but different questions, and our work has a clear focus on the first question. despite the large body of works (listed in section 1) devoted to alleviate exposure bias, its actual impact has rarely been systematically studied or validated in a direct and principled way. @xcite attempts to measure the gain from alleviating exposure bias, by counting the ground truth words whose probabilities in the predicted distributions produced by their proposed model are greater than those from the baseline model. however, it is unclear how this experiment can be linked to exposure bias. @xcite provides valuable discussions around the claim of exposure bias, but they did not propose an operational definition or metric. @xcite attempts to measure exposure bias by comparing the model's performance on seen (training) data and unseen (test) data. however, this methodology is more about the generalization gap, instead of exposure bias. finally, @xcite discusses the potential link between exposure bias and hallucination in the context of machine translation. in a relevant direction to answer the second question, several recent works attempt to evaluate whether the non-mle training methods can really give superior nlg performance than standard mle training for open-ended text generation. @xcite tunes a "temperature" parameter in the softmax output, and evaluates models over the whole quality-diversity spectrum. @xcite proposes to use "reverse language model score" or "frechet infersent distance" to evaluate the model's generation performance. @xcite proposes a method for approximating a distribution over tokens from gans, and then evaluates models with standard lm metrics. these works arrive at a similar conclusion: the generation performance of text gans is not convincingly better, or even worse, than standard mle training. these negative results motivate us to reassess the exposure bias problem, which serves as a major motivation of text gans. in the next section, we begin by introducing notations and background. ## background and notations our experiments will be focused on the task of open-ended language generation, which is arguably a good test bed for exposure bias due to the following reasons: (1) the generation length is long. notations auto-regressive language models are trained to learn the probability distribution of the (l + 1) th word (or token) w l+1 in a sentence w conditioned on the prefix w 1:l := (w 1 , . . . , w l ) and prompt c. we use w @xmath0. for simplicity, we assume all sentences are of length l in the formulations. denoting the groundtruth data distribution as p d , the standard mle (also referred to as teacher forcing) training aims to we denote the distribution of a mle-trained lm as p m , which is the major subject of this study. we will experiment with two popular model architectures: lstm lm @xcite and transformer lm @xcite . for generation, we mainly focus on classical ancestral sampling, due to the following reasons: (1) the sampling algorithms (e.g., top-k sampling) are known to trade quality out of diversity @xcite @xcite . so, invoking them could "hide" the exposure bias problem because the prefixes from the model will be of higher quality (thus narrowing the discrepancy). (2) the sampling algorithms requires tuning of hyper-parameters, which will complicate the comparison. with these being said, in our experiments (including human evaluation) we have also tested with top-k sampling @xcite , with consistent observations. in addition to popular metrics in natural language generation (nlg) such as bleu @xcite or meteor @xcite , our quantification approaches also rely on the measurements of the divergence between two distributions. let p denote the set of probability distributions on the vocabulary v , and let f div : p × @xmath2. we will adopt two popular probability divergence functions: total variation distance (denoted as d tv ) and jensen-shannon divergence (d js ). definitions of d tv and d js are provided in appendix a. ## a qualitative attempt we begin with a qualitative attempt to verify the seriousness of exposure bias, by designing a prefixswitching experiment as follows: we feed a mletrained transformer lm on the wiki-103 dataset with four types of prefixes of the same length: (1) test-data samples, (2) model's own samples, (3) test-data samples shuffled on word-level, or (4) samples from a uniformly random distribution on v . then we let the model continue the generation given these prefixes and compare the quality of the samples in a qualitative manner. details of the model and dataset are deferred to section 6. the intuition behind the prefix-switching experiment follows immediately from the original claim of exposure bias: during generation, if we set the prefix distribution to be the ground-truth data distribution instead of the model's own, the discrepancy between training and generation in the prefix will be removed, and hence the model's generation quality should be much better. we illustrate this idea in figure 1 . in the extreme case of shuffled or random prefixes, due to the claim from exposure bias about the incremental distortion, we expect the model to generate also badly distorted sequences. the samples with different types of prefixes are shown in on the contrary to our expectation, we do not observe a noticeable difference in sample quality comparing samples from model and data prefixes. more surprisingly, the model is still able to generate fairly good samples from shuffled prefixes. even in the extreme case of random prefixes, we still observe basic language structures in the sample. this experiment suggests that the mle-trained lms have the self-recovery ability, i.e., the model is able to recover from artificially distorted history input, and generate samples with reasonable quality. this observation casts doubt on exposure bias, which claims that the distortions in the generation should, on the contrary, be incremental. this qualitative attempt suggests the impact from exposure bias could be more subtle than we expected. especially, it would be difficult to judge whether the distortions are incremental via qualitative examination. motivated by this, we now turn to more rigorous quantification methods to measure the impact of exposure bias. ## quantification methods we now introduce the definitions of our proposed metrics. we attempt to quantify the impact of exposure bias on three key aspects of open-ended language generation: quality, diversity, and consistency. we first introduce eb-m, which covers the quality and diversity aspects, and then eb-c, which covers the consistency aspect. following the intuition of the prefix switching experiment, we design our quantification metrics to be a simple ratio reflecting the relative performance gain when data prefixes are fed to the model as opposed to the original model prefixes. ## experiment results in this section our quantification results for exposure bias are presented. in addition, we propose variants of the eb-m / eb-c metrics to analyze the self-recovery ability. for a systematic assessment of exposure bias, we decompose the claim of exposure bias into two factors: (1) the discrepancy in the prefix distribution would hurt the generation performance, in general. (2) moreover, the distortion should be incremental along the generation. the first factor can be reflected by the average magnitude of the measurements (the values are expected to be larger than 1 by a meaningful margin), and the second factor can be reflected by whether the measurements have an increasing trend along the prefix length. below we begin by describing the experiment setting. ## caveats in measuring exposure bias in this section we want to point out that one needs to be cautious when using our methodology of comparing generation from data or model prefix, to measure exposure bias. in particular, in the eb-m experiments, we compute the nlg metrics with a large number of unseen test data as references. this is a common practice for open-ended generation. however, for other classical seq2seq applications such as dialogue or machine translation, one might be tempted to just use the single given reference. this could potentially lead to seriously misleading results. in for a fair comparison, what we really need is a reference answer conditioned on the prefix sam-pled from the model (our design of eb-c follows this spirit). for example, in the dialogue case, we need to measure whether "to school" is a good completion for the prefix "i went". this, however, would require additional data collection, and we leave it for future work. ## discussion and limitations due to space constraint, we defer a discussion on teacher forcing as an objective function to appendix d. we devote this section to discuss the limitations of this work. firstly, the proposed quantification approaches only focus on exposure bias, and does not reflect the generation performance in general. for example, a uni-gram lm, which generates words independent of previous context, has no exposure bias problem and can pass our test easily. more importantly, since the original claim of exposure bias is not rigorously defined, our approaches can only act as reasonable proxies to measure its seriousness, and we humbly acknowledge they have limitations. we also note that this work is focused on openended language generation (in appendix e, we provide a preliminary study of the self-recovery ability in machine translation). our results do not rule out the possibility that exposure bias could be more serious in other nlg applications of different nature, which we leave for future work. finally, the results from this work should not discourage researchers from exploring non-mle training algorithms for lm (including text gans). as shown by recent studies, there exists important problems other than exposure bias for the current nlg models, such as the likelihood trap @xcite , factuality @xcite , or robustness @xcite , etc. therefore, it is completely possible that a non-mle training objective can lead to better generation performance @xcite @xcite @xcite . ## conclusion in this work, we design and experiment with two metrics (eb-m and eb-c) as proxies to quantify the significance of exposure bias. the measurement from our experiments, including a human evaluation, consistently show that the the distortion induced by the prefix discrepancy is limited, and does not seem to be incremental during the generation. moreover, our analysis reveals an interesting self-recovery ability of the lm, which we hypothesize to be countering the harmful effects from exposure bias.