Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation (2022)

Abstract: Current language generation models suffer from issues such as repetition, incoherence, and hallucinations. An often-repeated hypothesis for this brittleness of generation models is that it is caused by the training and the generation procedure mismatch, also referred to as exposure bias. In this paper, we verify this hypothesis by analyzing exposure bias from an imitation learning perspective. We show that exposure bias leads to an accumulation of errors during generation, analyze why perplexity fails to capture this accumulation of errors, and empirically show that this accumulation results in poor generation quality.

## introduction large-scale neural language models have made great strides in a series of language generation tasks such as machine translation @xcite @xcite , text summarization @xcite @xcite , conversational dialog generation @xcite @xcite , etc. however, despite the successes achieved by these models on several conditional generation tasks, they continue to suffer from degenerate behaviors such as repetition, a lack of diversity, dullness, and, incoherence, especially in open-ended generation settings such as text completion and dialog modeling @xcite . this degenerate behavior is often attributed to a mismatch between the maximum likelihood training and gen-eration procedure mismatch @xcite @xcite . maximum likelihood training also referred to as teacher forcing @xcite , factorizes the language model as a linear chain, and maximizes the log-likelihood of this factorized language model on a training corpus. during this maximum likelihood training, the model learns a distribution of the next tokens conditioned on the contexts from the ground-truth training data. a concern with the mle-based training is that the ground-truth contexts from the training corpus are not available during generation. rather, the conditioning contexts during this phase comprise of tokens previously generated by the model itself. the distribution of these contexts seen during the generation phase might be very different from the ones encountered during the training phase. this mismatch is referred to as exposure bias @xcite . a side-effect of exposure bias is that an error at any step during generation might have a cascading effect as the next context will incorporate this erroneous prediction, deviating away from the ground truth context distribution leading to more errors. these errors will result in sequences that degenerate over the sequence length resulting in incoherent text, lack of vocabulary diversity, and detachment from the source sequence resulting in hallucination, and/or word-and phrase-level repetition. there is an active debate in the language generation community on the impact of exposure bias in language generation. authors have both validated @xcite and questioned @xcite the impact of exposure bias on language generation. several approaches have been proposed to mitigate exposure bias @xcite @xcite @xcite but these have neither formalized exposure bias clearly nor provide any empiri-cal evidence that these methods mitigate the effect of exposure bias. finally, previous works have linked exposure bias to out-of-domain @xcite and out of distribution @xcite generalization, and hallucinations @xcite but these claims remain weak in absence of a clear and principled formalization of the exposure bias issue. in this paper, we attempt to clarify this confusion by formalizing exposure bias in the terms of accumulation of errors and analyzing its impact on generation quality. we do this by providing a theoretically-grounded understanding of the exposure bias issue by analyzing it from an imitation learning perspective. we use this perspective to show that behavior cloning-an imitation learning algorithm is equivalent to teacher forcing under the choice of a particular loss function. we then exploit this equivalence to borrow the bound on error accumulation caused by behavior cloning and use it to quantify exposure bias and analyze the error accumulation in language generation. finally, we use this quantifiable definition of exposure bias to demonstrate that models trained using teacher forcing do suffer from an accumulation of errors. we also show, both analytically and empirically, why perplexity fails to capture this error accumulation, and how a lower exposure bias correlates with better generation quality. ## language generation formulation given a finite-sized vocabulary set v, language generation is posed as a problem of generation a variable-length sequence w n 0 ∈ v n from a languge model p θ , either unconditionally or conditioned on the a source x, using a decoding algorithm f. language modeling is the problem of learning this parameterized model p θ , that approximates an oracle model o, such that decoding from the model p θ mimics greedily sampling from the oracle o. maximum likelihood-based training factorizes the probability distribution model, p θ (w n 0 ), into a linear chain, i.e., given a trained language model p θ , the simplest strategy for generating a target sequence is to greedily sample the model i.e., at each step i, pick the most probable token w @xmath0; x)as its prediction. for the next step i + 1, we use w i to generate the context w i 0 = w i-1 0 w i , and use it to predict the next token. this continues either until the maximum sequence length (t ) is reached, or a special end of sequence token (eos) is generated. ## an imitation learning perspective of language generation in this section, we will present an imitation learning perspective of language generation. this framing will allow us to borrow theoretical machinery from imitation learning literature to formalize the exposure bias issue and quantify it in terms of accumulation of error due to procedural mismatch between mle-based training and generation. we start by posing language generation as a sequential decision-making problem and language modeling as an instance of imitation learning. we exploit these parallels to show behavior cloningan imitation learning algorithm, is equivalent to teacher forcing under a choice of a particular loss function. we then exploit this equivalence to quantify the error accumulation due to exposure bias. language generation is a sequential decision-making problem: a sequential decision-making problem can be formalized as learning a policy π(a t |s t ) over a space of actions a @xmath1, a t ). we can use this framework to pose language generation as an instance of a sequential decision-making problem with language model p θ as the policy, contexts w t-1 0 ∈ v * as states, the next token prediction w @xmath2, and concatination function as the transition function. this perspective allows us to appreciate the fact that, during generation, predictions at previous steps affect the next predictions, and error over time can cascade resulting in incoherent sequences. language modeling is imitation learning: imitation learning is a class of methods to solve a sequential decision-making problem while having access to the oracle policy o or data generated by the oracle, i.e., @xmath3, a t )|s t ∼ d t o , a t ∼ o(•|s t )}. here, d t o is the oracle induced state-visitation distribution at time t. in imitation learning, an agent learns a model policy π that reproduces the expert policy o but on the state-visitation distributiond t π that has been induced by the model policy π, . the sequential decision-making perspective of language generation allows us to pose language modeling as an instance of imitation learninglearning a model for a sequential decision-making problem with the help of an expert oracle (in rlbased methods) or using the data generated by the oracle (for mle-based methods). teacher forcing is behavior cloning: the assumption of access to an oracle is unrealistic in many scenarios. behavior cloning is an approach to solve an imitation learning problem using only the training data generated by an oracle. in this setup, the state-action pairs in the training data are assumed to be identically and independently distributed. this is equivalent to reducing a sequential decision-making problem to a supervised multi-class classification learning problem. concretely, this learning problem can be seen as minimizing the expected per-step loss under the state distribution induced by the oracle: here, l bc (π) is the behavior cloning loss and l(s, π; o) is the per-step loss. similary, in practical scenarios, language models are also trained on a finite training corpus, d, that assumed to be generated by the oracle., i.e., maximum likelihood training loss from equation 3, can be reformulated as learning the distribution over the next tokens, conditioned on the train-ing contexts generated by the oracle, w t-1 the behavior cloning loss (equation 4 ) is equivalent to the language modeling loss (equation 7 ) with l(p θ , w t-1 this definition ensures that the per-step loss for oracle is zero, i.e., l(o, w t-1 0 ; o) = 0. the per-step loss function defined by equation 8 ensures that the behavior cloning loss, l bc (p), under our definition is equivalent to teacher forcing loss, l t f (p), up to a constant term. this equivalence of l bc (p) and l t f (p) ensures that the model learned by minimizing either of the two losses will be identical. language generation is policy rollouts: during policy rollouts, an agent in state s t , executes the action a t , sampled from policy π, and ends up in state s t+1 . the agent's next state is dependent upon its own actions. this state evolution can be formulated as sampling from state-visitation distribution induced by the policy π, i.e., s t+1 ∼ d t+1 π . the performance of policy π during rollouts can be measured using the loss (cost) of executing the policy π, i.e., we can also formulate language generation in terms of policy rollouts from imitation learning. mathematically, we can express generation as sampling contexts from model's context distribution, i.e., w j-1 0 ∼ d j p θ ,f , and generating the next token w j conditioned on w j-1 0 , using the decoding algorithm f, i.e., we can now define the inference-time loss, l i (p θ ), as the accumulated t -step loss of model p θ imitating oracle o on the context distribution induced by the model: ## exposure bias and error accumulation ranzato et al. ( 2016 ) defined exposure bias as a behavioral mismatch between maximum likelihoodbased training and generation procedure. during maximum likelihood-based training, the next token distribution is conditioned on ground truth data whereas, during generation, it has to rely on the model's own previously generated tokens. they also postulated that this training and generation context distribution mismatch might result in an accumulation of errors during generation. intuitively, when the model produces a token w i that makes the resulting context w i 0 unfamiliar, it might not be able to continue the generation adequately and is likely to produce another token which will further make the context flawed. this phenomenon reinforces itself as the context drifts further from what the oracle would produce, leading to an accumulation of errors. in the imitation learning literature, the accumulation of errors while rolling out a policy trained using behavior cloning is analyzed in the terms of inference-time regret of the behavior cloning policy, π bc , with respect to the oracle policy, o, @xcite let t be the expected error of executing policy π at step t on the state-visitation distribution induced by the oracle o, i.e., let be the average expected error of executing policy π over t step, i.e., = 1/t t @xmath4. assuming l(s, π, o) is an upper bound on [0, 1] loss, we can bound the regret for a policy π bc as, the lower-bound in equation 14 assumes no accumulation of error, hence an expected error of at each step, whereas the upper bound assumes the worst-case scenario, resulting in linear growth in error at each step and overall quadratic accumulative growth w.r.t. maximum sequence length t . relying on the imitation learning perspective of language generation presented in the previous section, we can now borrow this regret-based analysis from imitation learning literature to similarly bound the regret of a language generation model as we will now use these bounds on the regret to analyze and quantify the error accumulation due to exposure bias in language generation. ## quantifying error accumulation due to exposure bias in our analysis, we use two metrics, @xmath5. we define accumulated errors up to length l, @xmath6, as here, @xmath7, f) be the regret due to the use of language model p θ and decoding method, f, up to sequence length l, @xmath8. this metric captures the growth of error w.r.t. sequence length l. if exposure bias does indeed leads to error accumulation, @xmath9.r.t. l. we define our second metric, %@xmath10, as percentage of excess errors committed by the model that can be attributed to exposure bias, i.e., here, @xmath11. in the rest of the paper, we use these definitions to show: 1.) error accumulation in language generation is real, 2.) perplexity fails to capture this error accumulation, 3.) lower exposure bias correlates with a higher quality generation that is more coherent, uses diverse vocabulary, and is less repetitive. text completion models take a text passage or prefix w j 0 ∼ o as an input and generate a coherent continuation of the prefix, w n j+1 using the language model p θ and the decoding algorithm f, i.e., w n j+1 = f(p θ , w j 0 ). in this paper, we use this textcompletion setup to analyze the error accumulation due to exposure bias and it's correlation with language generation quality. language model and dataset: we conduct our analysis using the gpt2 language model @xcite . we use the gpt2-117m model as our evaluation language model and use train split of wikitext-103 @xcite for prompts. we rely on gpt-2 model fine-tuned on wikitext-103 as our approximate oracle. we tokenize the wikitext-103 dataset using gpt-2's tokenization scheme. we chunk wikitext-103's train split into sequences of length 512. of these, we use the first 50 tokens as prompts for our generation experiments and generate the completions to a maximum length of 512 or up to the end of the sequence token. we use a total of 20k prompts for our evaluation. ## related work non-mle training methods: several approaches have been proposed to mitigate the exposure bias issue including rl-based optimization objectives @xcite @xcite , learning to search @xcite , energy-based models @xcite , imitation learning @xcite , generative adversarial networks @xcite and knowledge distillation @xcite . although these methods motivate their approaches intending to reduce exposure bias, they neither formalize exposure bias clearly nor provide any empirical evidence that these methods mitigate the effect of exposure bias. in this paper, we proposed a quantifiable definition of exposure bias by analyzing the issue from a principled imitation learning perspective. this definition can be used or adapted to evaluate the various novel training and modeling approaches on their ability to do reduce exposure bias. smarter decoding methods: large language models have unusually low test perplexities but they falter at coherent and diverse language generation specifically in open-ended language generation tasks especially while using deterministic decoding schemes. several authors @xcite @xcite have hypothesized that training and inference mismatch due to mle-based training is responsible for the degenerate behavior. they have proposed smarter decoding schemes to mitigate the side effects of exposure bias resulting in better generation quality. despite this being an active area of research, this often-repeated hypothesis for degenerate generation behavior has not received serious treatment till now. in this paper, we take a step towards explaining this discrepancy and show that error accumulation due to exposure bias might be the reason for this degenerate behavior and explain why perplexity has a handicap in capturing this compounding of errors. analyzing exposure bias: schmidt (2019) and wang and sennrich (2020) link exposure bias to generalization gap due to distribution and domain shift respectively. performance degradation under domain and distribution shift is a major issue with language generation, and direct evidence supporting this hypothesis will provide insights into building more robust language generation models. unfortunately, neither of the papers formalize the notion of exposure bias or empirically link the generalization gap to exposure bias directly. three recent papers, @xcite al. (2019); zhang et al. (2019b); he et al. (2019) , have tried to empirically evaluate the impact of exposure bias on language generation. the first two papers validate the existence of exposure bias whereas @xcite show language models have self-recovering ability negating the impact of exposure bias. all three analyses are based on the empirical definition of exposure bias which, in turn, is based on the informal formulation by @xcite . in this paper, we provide a principled and theoretically grounded approach to analyze exposure bias in language generation and show that it is indeed a problem and that it might explain the degeneration issue with large language models on open-ended tasks under deterministic decoding. ## discussion in this paper, we analyze language generation from an imitation learning perspective. we use this analysis to arrive at a theoretical bound on error accumulation due to exposure bias. this bound predicts a super-linear growth in errors accumulation during generation due to exposure bias. in our experiments, we validate this bound and show that accumulation due to exposure bias indeed results in super-linear growth in errors. we then show, both analytically and empirically, why perplexity is not enough to capture this accumulation of errors and hypothesize that this accumulation of errors is responsible for the degenerate language generation. finally, we provide some evidence for this hypothesis by evaluating the impact of various decoding schemes on error accumulation and generation quality. we show that techniques that improve the generation quality do result in a lower error accumulation and causally link language generation quality to error accumulator due to exposure bias. our analysis provides a principled and theoretically grounded way to understand the exposure bias. we believe this analysis can pave way for developing smarter training and decoding algorithms to address this error accumulation resulting in more robust language generation models.