Improved Natural Language Generation via Loss Truncation (2020)

Abstract: Neural language models are usually trained to match the distributional properties of large-scale corpora by minimizing the log loss. While straightforward to optimize, this approach forces the model to reproduce all variations in the dataset, including noisy and invalid references (e.g., misannotations and hallucinated facts). Even a small fraction of noisy data can degrade the performance of log loss. As an alternative, prior work has shown that minimizing the distinguishability of generated samples is a principled and robust loss that can handle invalid references. However, distinguishability has not been used in practice due to challenges in optimization and estimation. We propose loss truncation: a simple and scalable procedure which adaptively removes high log loss examples as a way to optimize for distinguishability. Empirically, we demonstrate that loss truncation outperforms existing baselines on distinguishability on a summarization task. Furthermore, we show that samples generated by the loss truncation model have factual accuracy ratings that exceed those of baselines and match human references.

## introduction learning to generate text is a core part of many nlp tasks, including summarization @xcite , image captioning @xcite , and story generation @xcite . a common challenge to all these tasks is that references from the training distribution are not unique and contain substantial variations in phrasing and content @xcite . learning to generate under a set of diverse and noisy references is challenging as some variations ought to be learned (e.g., paraphrasing) while others should not (e.g., hallucinated facts, ignoring prompts). existing training procedures for models seek to match the underlying distribution, leading to models that replicate and sometimes even amplify unwanted behaviors such as hallucination during generation. for example, neural language models often produce fluent text that is unfaithful to the source @xcite @xcite . existing work @xcite has primarily addressed these issues by constructing decoders that implicitly remove unwanted variation when generating (see §6 for a detailed discussion of task-specific losses). in this work, we argue that this phenomenon is not model specific, but is due to the widely-used log loss: we demonstrate that log loss is not robust to noisy and invalid references ( §2). in particular, log loss requires that models assign probabilities to all potential test reference sequences. as a result, log loss is sensitive to outliers: invalid or noisy references with small probability mass can cause large changes in model behavior. we show that the brittleness of log loss, together with the noise in existing generation datasets, lead to low-quality and unfaithful generated text. instead of optimizing log loss, which has little correlation with model output quality @xcite @xcite , recent work on diverse generation models has proposed optimizing for the distinguishability of samples from the model and the reference. distinguishability provides a natural and appealing guarantee: samples that are indistinguishable from human generated text will be as high quality as human generated text. furthermore, we show that optimizing for distinguishability is robust in the face of noisy and even invalid data. despite its appeal, distinguishability has not been widely used due to statistical and computational challenges. for example, existing methods that directly optimize for distinguishability have yet to match even naive log loss based baselines @xcite . we propose a modification to the log loss, loss truncation, that has the benefits of distinguishability while being efficient to train. loss truncation is as efficient to train as log loss, nearly as robust as distinguishability, and provides distinguishability guarantees via an upper bound. it achieves these properties by modifying the standard log loss to adaptively remove examples with high log loss. we additionally extend loss truncation with a sequence-level rejection sampling scheme that generates higher quality sequences by restricting the outputs to be high probability sequences. we show that loss truncation with direct and rejection sampling outperforms standard log loss based generation methods (beam search, full sampling, top-k, and top-p sampling) on distinguishability, as measured by the huse score @xcite . we additionally study the factual accuracy of a summarization system trained on loss truncation and show that our proposed approach produces summaries which improve upon all baselines (including beam searched models) and match references on factual accuracy. ## motivation and problem statement task and background. we consider a natural language generation task with a conditional language model, where we are given a context x drawn from p(x) and our probabilistic model p(y | x) produces an output y by approximating a (usually human) reference distribution p ref (y|x). in order to achieve this, many existing models are trained to minimize the kullback-leibler (kl) divergence, we refer to the first term of this divergence as the log loss of a model. the second term is commonly ignored as it is a constant with respect to the model. minimizing the log loss has several practical benefits: 1) it is written as an expected loss (and is thus straightforward to optimize via stochastic gradient descent), 2) it factorizes across tokens in autoregressive modeling, and 3) it provides a guarantee on a model's goodness of fit (eq ( 1) ). unfortunately, log loss also suffers from several drawbacks. it is known to have little correlation with a model's sample quality and it can be brittle to invalid references in the training data. 0 5 10 15 0.0 0.2 reference min distinguishability min log-loss figure 1 : fitting a mixture of gaussians with a single gaussian using distinguishability (tv) and log loss (kl). as shown, log loss is extremely sensitive to outliers, resulting in poor estimation. log loss is not robust to noise. the kl divergence has intuitively correct behavior when each input x has a single correct reference y: it will maximize the probability of the single correct reference. however, log loss can be problematic when there are multiple correct references, of which some are invalid or difficult to model. in particular, log loss is sensitive to invalid or noisy data because it requires that the model assign high probabilities to all potential references. log loss is unbounded above: a model assigning zero probability to even a single reference makes the model incur an infinite overall loss. we show a well-known example of this behavior with synthetic data. we consider fitting a single gaussian to a mixture of two gaussian in figure 1 . the reference distribution (blue) has a valid set of references at zero as well as variation that the model does not expect (e.g., invalid or noisy references) on the right. minimizing the log loss results in a suboptimal model that is forced to span both groups. furthermore, post-hoc processing the model does not help, as even the most likely output under the log loss trained model (~3) has low probability under the reference distribution. in natural language generation, training sets can contain invalid or poor quality references. as such, these types of problems manifest themselves in tasks such as summarization (hallucinating facts), story generation (ignoring prompts and constraints), and captioning (ignoring parts of the image). much of the existing literature on faithful generation has focused on designing better models for valid references (via copying or attention constraints), but the example in figure 1 shows that this alone may not be sufficient. the gaussian 'model' in this case perfectly fits the mixture component context: for the first time in five years, microsoft corp. is finally unveiling a new system for operating personal computers. title: microsoft makes long-awaited software upgrade available to businesses thursday. at zero but is still brittle because it cannot simultaneously fit the other group of (invalid) samples. resolving this will require either a model which is designed explicitly to capture invalid references or a loss function that can ignore them. ## loss truncation intuition. we would like the model to ignore data that would force it to unnecessarily hallucinate at test time. concretely, recall the toy example (figure 1 ); there is a set of invalid references that force the model to be degenerate. if we could remove these these invalid references by truncating the distribution, the resulting model would be high quality. we can show that this intuition is theoretically justified, and that truncating (i.e., removing) an appropriate c-fraction of the data provides tighter bounds on the distinguishability of the model. improved log losses for distinguishability. we will demonstrate that log loss with an appropriate c-fraction of the data removed provides guarantees on distinguishability. we will define the set of truncated distributions as the set of distributions with any c-fraction of data removed a simple lemma shows that that all elements in p c,p are c-close to p in tv (appendix b). now we state our main result, proposition 1. for any @xmath0, 1] and p @xmath1,p ref , see appendix b for the proof. namely, distinguishability is bounded by the log loss with respect to the truncated distribution and a small constant. furthermore, this upper bound is valid for any c, although different c will change the tightness of the bound and produce different models. this truncated bound can be substantially tighter than pinsker's inequality. consider for example a model that can perfectly capture (1 -c) fraction of the data, but c-fraction of the reference outputs cannot be generated by the model and receive probability zero. in this case, the distinguishability (tv) is c, the kl divergence is infinite, while our truncated bound is √ c 2 + 2c. this suggests that appropriately truncating high-loss examples makes log loss robust and allows us to use log loss as a surrogate for distinguishability, even in the presence of invalid and noisy references. ## related work decoder-based diversity. researchers have proposed a variety of models for text generation @xcite @xcite . these models generate text using decoding methods such as beam search. while beam search is generally thought of as the gold standard @xcite , it can produce generic and repetitive outputs @xcite . to achieve diversity, top-k @xcite and top-p @xcite sampling stochastically decodes the outputs after restricting the output space to avoid low-quality outputs. while these techniques can improve generation quality, they rely on models trained via log loss, which we show can result in undesired behavior that cannot be fixed post-hoc. our work is complementary to existing work on decoders by proposing a loss that can improve the probabilistic models which these decoders operate on. loss modifications. prior work has identified specific issues in generative models, such as repetitiveness, and proposed loss modifications to address these specific issues in the context of long text generation @xcite . in contrast, we identify an issue with the widely used log loss, and propose loss truncation, which does not require a task-and issue-specific method example context at least ## people have been killed and more than ##,### made homeless by floods that swept across southern africa in the past week , striking a region already grappling with severe food shortages . gold floods kill ## in famine-hit southern africa loss truncation at least ## people killed ##,### evacuated in floods in southern african region floods that sweep parts of africa kill at least ## beam flooding hits southern africa as deaths rise full sampling child farming stalls in southern africa earthquake kills ## in southern africa top-p (@xmath2.9) torrential rains prompt warnings in southern africa toll nears ## in southern africa top-k (@xmath3,### homeless in southern africa floods at least ## dead ##,### homeless as floods hit southern africa other generation methods optimize for taskspecific losses @xcite . task specific losses are not known in many cases and thus we require an effective task-agnostic loss, e.g., log loss or tv. we show that tv acts as a useful task-agnostic goodness of fit measure, and we provide an improved alternative to log loss. gans. gans have been proposed to learn models that minimize distinguishability @xcite @xcite . while gans have been successful in generating images @xcite , gans remaining challenging to optimize for text due to the discrete nature of text. our findings match earlier reports that gans underperform log loss trained sequence-to-sequence models @xcite . in this work, we show that better training methods for distinguishability can arise from modifying the standard log loss via truncation. robust learning. robust learning is the study of learning in the face of outliers @xcite @xcite . our work is related to the -contamination model, in which an fraction of the data has been modified, potentially by an adversary @xcite . our work shows that robust learning under log loss can result in improved empirical performance and bounds on distinguishability. while there are a number of effective approaches to robust learning @xcite , we focus on a simple truncation procedure as it is one of the only procedures scaleable enough to apply on large-scale generation datasets. our work shows that more effective, scalable robust learning procedures can help improve natural language generation methods. ## conclusion in this work, we show that log loss is not robust to noise, which can in turn cause undesired behavior, such as hallucinating facts in summarization. in response, we propose loss truncation, a robust training method that optimizes for distinguishability of generated samples. we additionally propose a sequence-level rejection sampling scheme to generate high quality sequences. we show that loss truncation outperforms a range of baselines (including beam search, top-p, top-k, and full sampling) on distinguishability. we additionally show that rejection sampling outperforms all baselines, including beam search, on generating factual summaries. these results suggest that robust learning in the form of truncating the log loss can complement model-based approaches to faithful generation by ignoring invalid and undesired references. context: donna shalala is sporting a mustache to promote public health. title: milk on her lip shalala raises eyebrows (a) example of a title that requires hallucinating new facts, e.g., "milk on her lip" and "raises eyebrows".