Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling (2022)

Abstract: Neural autoregressive sequence models smear the probability among many possible sequences including degenerate ones, such as empty or repetitive sequences. In this work, we tackle one specific case where the model assigns a high probability to unreasonably short sequences. We define the oversmoothing rate to quantify this issue. After confirming the high degree of oversmoothing in neural machine translation, we propose to explicitly minimize the oversmoothing rate during training. We conduct a set of experiments to study the effect of the proposed regularization on both model distribution and decoding performance. We use a neural machine translation task as the testbed and consider three different datasets of varying size. Our experiments reveal three major findings. First, we can control the oversmoothing rate of the model by tuning the strength of the regularization. Second, by enhancing the oversmoothing loss contribution, the probability and the rank of eos token decrease heavily at positions where it is not supposed to be. Third, the proposed regularization impacts the outcome of beam search especially when a large beam is used. The degradation of translation quality (measured in BLEU) with a large beam significantly lessens with lower oversmoothing rate, but the degradation compared to smaller beam sizes remains to exist. From these observations, we conclude that the high degree of oversmoothing is the main reason behind the degenerate case of overly probable short sequences in a neural autoregressive model.

## introduction neural autoregressive sequence modeling is a widely used scheme for conditional text generation. it is applied to many nlp tasks, including machine translation, language modeling, and conversation modeling @xcite @xcite . despite the substantial success, major issues still exist, and it is still an active area of research. here we highlight two major issues which have been discussed extensively. the first issue is the model assigning too high a probability to a sequence which is unreasonably shorter than a ground-truth sequence. @xcite report evidence of an extreme case where the model frequently assigns the highest probability to an empty sequence given a source sequence in machine translation. in addition, @xcite demonstrate that the length of generated translation gets shorter with better decoding (i.e., beam search with a larger beam.) in the second issue, which is more often observed in open-ended sequence generation tasks, such as sequence completion, generated sequences often contain unreasonably many repetitions @xcite . this phenomenon was partly explained in a recent year by @xcite , as approximate decoding resulting in an infinitely long, zeroprobability sequence. in this work, we tackle the first issue where the model prefers overly short sequences compared to longer, often more correct ones. we assume that any prefix substring of a ground-truth sequence is an unreasonably short sequence and call such a prefix as a premature sequence. this definition allows us to calculate how often an unreasonably short sequence receives a higher probability than the original, full sequence does. this value quantifies the degree to which the probability mass is oversmoothed toward shorter sequences. we call this quantity an oversmoothing rate. we empirically verify that publicly available, well-trained translation models exhibit high oversmoothing rates. we propose to minimize the oversmoothing rate during training together with the negative loglikelihood objective. since the oversmoothing rate is difficult to minimize directly due to its construction as the average of indicator functions, we design its convex relaxation, to which we refer as an oversmoothing loss. this loss is easier to use with gradient-based learning. we apply the proposed regularization to neural machine translation using iwslt'17 and wmt tasks and observe promising findings. we effectively reduce the oversmoothing rate by minimizing the proposed oversmoothing loss across all tasks we consider. we see the narrowing gap between the length distribution of generated sequences and that of the reference sequences, even when we increase the beam size, with a lower oversmoothing rate. finally, by choosing the strength of the proposed regularization appropriately, we improve the translation quality when decoding with large beam sizes. we could not, however, observe a similar improvement with a small beam size. ## background: neural autoregressive sequence modeling we study how a neural sequence model assigns too high probability to unreasonably short sequences due to its design and training objective. we do so in the context of machine translation in which the goal is to model a conditional distribution over a target language given a source sentence. more specifically, we consider a standard approach of autoregressive neural sequence modeling for this task of neural machine translation, where the conditional probability of a target sentence given a source sentence is written down as: 1 r is a regularization term that prevents overfitting, such as weight decay. once training is done, we use this autoregressive model as a translation system by approximately solving the following optimization problem: ŷmap = arg max y p(y|x; θ). we often resort to greedy decoding or beam search, both of which belong to a family of incomplete decoding algorithms @xcite . ## oversmoothing: the issue of premature sequences in this section, we carefully describe the issue of premature translation or premature sequence in autoregressive modeling, which has more often been referred to casually as the issue of oversmoothing in earlier studies (see, e.g., @xcite . to do so, we first define formally what we mean by a 'premature sequence'. a premature sequence is a length-t prefix of an original sequence, where t is smaller than the length of the original sequence. in other words, length-t prefix is defined as: definition 3.1 (length-t prefix). given an original sequence @xmath0, y 2 , . . . , y |y| = ⟨eos⟩), the length-t prefix is @xmath1, y 2 , . . . , y t-1 , ⟨eos⟩), where 1 ≤ @xmath2. with this definition, we make a reasonable assumption that most of such premature sequences are not valid sequences on their own. in the case of natural language processing, for instance, these premature sequences correspond to sentences that suddenly terminate in the middle. only a few of these premature sequences may be a coherent, wellformed text. a good autoregressive language model should then assign a lower probability to such an illformed premature sequence than that assigned to a well-formed original sequence. that is, it must satisfy: in order for this inequality to hold, the probability assigned to the ⟨eos⟩ must be extremely small, as the left-hand side of the inequality is the product of many probabilities. in other words, the dynamic range of the ⟨eos⟩ token probability must be significantly greater than that of any other token probability, in order for the autoregressive language model to properly capture the ill-formed nature of premature sequences. it is, however, a usual practice to treat the ⟨eos⟩ token just like any other token in the vocabulary, which is evident from eq. ( 1 ). this leads to the difficulty in having a dramatically larger dynamic range for the ⟨eos⟩ probability than for other token probabilities. in other words, this limited dynamic range due to the lack of special treatment of ⟨eos⟩ is what previous studies @xcite have referred to as "oversmoothing", and this leads to the degeneracy in length modeling. under this observation, we can now quantify the degree of oversmoothing foot_0 by examining how often the inequality in eq. ( 2 ) is violated: definition 3.2 (oversmoothing rate). the oversmoothing rate of a sequence is defined as with this definition, we can now quantify the degree of oversmoothing and thereby quantify any improvement in terms of the issue of oversmoothing by any future proposal, including our own in this paper. because premature sequences may be wellformed, it is not desirable for the oversmoothing rate to reach 0. we, however, demonstrate later empirically that this oversmoothing rate is too high for every system we considered in this work. ## related work the issue of generating sequences that are shorter than the ground-truth one has been studied from various aspects including model parametrization, data collection, and decoding. here we highlight some of these projects in the context of our work. on the aspect of model parametrization, peters and martins (2021) suggest using sparse transformation of the next-token distribution rather than the usual way of using softmax. such a model is then able to assign zero probability to short sequences more readily and thereby reduce the oversmoothing rate. their approach, however, does not explicitly encourage ⟨eos⟩ tokens to be assigned zero probability, unlike ours where ⟨eos⟩ is treated specially. @xcite embed the ⟨eos⟩ token with a distinct vector at each position within the sequence. this was shown to help the probability of empty sequence, although they do not report its impact on translation quality at all. on data collection, @xcite analyze data collection and show that data augmentation techniques altering sequence length may address the issue of oversmoothing and improve translation quality. their work is however limited to lowresource tasks. with respect to decoding, wang et al. (2020) observe the oversmoothing while studying "look-ahead" decoding strategies. they reduce the probability of the ⟨eos⟩ using the auxiliary loss term, similarly to the token-level unlikelihood loss @xcite . @xcite design a decoding algorithm that learns to correct the underestimated length. alternative decoding algorithms, such as minimum bayes risk decoding @xcite , have been shown to alleviate the length mismatch to a certain extent when compared to beam search. these earlier approaches do not attempt at for-mally characterizing the cause behind the issue of oversmoothing. this is unlike our work, where we start by formalizing the issue of oversmoothing and propose a way to alleviate this issue by directly addressing this cause. ## experimental setup we follow a standard practice to train our neural machine translation models, following @xcite , using the fairseq framework @xcite . we use bpe tokenization via either fastbpe @xcite or sentencepiece @xcite , depending on the dataset. although it is not required for us to use state-ofthe-art models to study the issue of oversmoothing, we use models that achieve reasonable translation quality. the code implementing fairseq task with the oversmoothing rate metric, oversmoothing loss, and experimental results is available on github. foot_1 ## experiments as we pointed out earlier, publicly available translation systems exhibit a high degree of oversmoothing. see the left-most part of figure 1 , where α = 0. in particular, this rate ranges from 34% (wmt'19 @xmath3. according to section 3. ## conclusion in this work, we tackled a well-reported issue of oversmoothing in neural autoregressive sequence modeling, which has evaded rigorous characterization until now despite of its ubiquity. we characterized it by defining the oversmoothing rate. it computes how often the probability of the ground-truth sequence is lower than the probability of any of its prefixes. we confirmed that the oversmoothing rate is too high among well-trained neural machine translation systems and proposed a way to directly minimize it during training. we designed a differentiable upper bound of the oversmoothing rate called the oversmoothing loss. we experimented with a diverse set of neural machine translation systems to study the effect of the proposed regularization. the experiments revealed several findings and takeaways. first, the oversmoothing loss is effective: we were able to monotonically decrease the oversmoothing rate by increasing the strength of the loss. second, we found that this regularization scheme significantly expands the dynamic range of the log-probability of ⟨eos⟩ token and has even greater impact on its rank, without compromising on sequence modeling. third, the proposed approach dramatically alters the behaviour of decoding when a large beam width was used. more specifically, it prevents the issue of degrading length ratio and improves translation quality. these effects were not as apparent with a small beam size though. the proposed notion of oversmoothing explains some of the degeneracies re-ported earlier, and the proposed mitigation protocol alleviates these degeneracies. we, however, find that the proposed approach could not explain a more interesting riddle, that is, the lack of improvement in translation quality despite lower oversmoothing when beam search with a smaller beam was used. this unreasonable effectiveness of beam search continues to remain a mystery and needs to be investigated further in the future.