Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation (2021)

Abstract: Neural Machine Translation (NMT) is known to suffer from a beam-search problem: after a certain point, increasing beam size causes an overall drop in translation quality. This effect is especially pronounced for long sentences. While much work was done analyzing this phenomenon, primarily for autoregressive NMT models, there is still no consensus on its underlying cause. In this work, we analyze errors that cause major quality degradation with large beams in NMT and Automatic Speech Recognition (ASR). We show that a factor that strongly contributes to the quality degradation with large beams is dataset length-bias - NMT datasets are strongly biased towards short sentences. To mitigate this issue, we propose a new data augmentation technique – Multi-Sentence Resampling (MSR). This technique extends the training examples by concatenating several sentences from the original dataset to make a long training example. We demonstrate that MSR significantly reduces degradation with growing beam size and improves final translation quality on the IWSTL15 En-Vi, IWSTL17 En-Fr, and WMT14 En-De datasets.

## introduction in this work, we address the beam-search problem in neural machine translation @xcite . beam search is the standard hypothesis search method for autoregressive sequence generation. large beams provide more probable hypotheses than small beams; however, the overall translation quality drops with growing beam size after a certain point. this effect is especially strong for long sentences, connecting with the fact that nmt models are biased to giving high probabilities to short hypotheses. @xcite showed that exact search by likelihood for neural machine translation finds empty string as the optimal hypothesis in more than 50% of cases. one of the most famous methods to mitigate quality degradation with growing beam size is length normalization @xcite . this technique normalizes loglikelihoods of a hypothesis in beam search by its length, thus promoting long hypotheses. other methods examine adding a reward for each token's score during the decoding process @xcite . while the beam-search problem has been extensively studied @xcite @xcite @xcite @xcite there is still no consensus on the underlying reason for such model behavior. furthermore, prior work has investigated this problem primarily for nmt models, giving little attention to other domains that are also known to suffer from it, such as automatic speech recognition (asr) @xcite . @xcite noticed that since in each step of beamsearch generation, negative log-probability is added to the hypothesis' score, if a model overestimates the probability of an already generated sequence of tokens, there is no way to downgrade this probability afterward. consequently, models are biased towards finalizing a short hypothesis by generating an end-of-sequence token (eos) rather than generating a long continuation. their experiments show a connection between quality degradation and decreasing length of hypotheses with growing beam size. our work is in agreement with their explanation. moreover, we show that the main quality degradation with large beams in nmt and asr comes from short translations obtained by early termination of long hypotheses from small beams. our work examines how the distribution of sentence lengths in a dataset affects the beam-search problem. we demonstrate that the beam-search problem is strongly connected with the distribution of sentence lengths in training datasets. specifically, we show that common nmt datasets, such as iwslt and wmt, exhibit a strongly skewed distribution of sentence lengths, with the mode focused on short sentences. nmt models learn biased probability distributions and fail on long sentences during inference. in contrast, for asr models trained on librispeech @xcite , where the distribution of sentence lengths is more symmetrical and biased towards longer sentences, the beamsearch degradation occurs at much larger lengths. based on our findings, we propose a simple and effective dataset augmentation technique that makes training examples longer -multi-sentence resampling. it creates a new dataset where each training sample can be a concatenation of multiple sentences. our method alleviates quality degradation with growing beam size and increases the final quality of the model. the key contributions of our work are as follows: ## quality degradation analysis this section analyzes quality degradation with the growing beam size of two systems: neural machine translation (nmt) and automatic speech recognition (asr). asr is also known to suffer from the beam-search problem @xcite . the models, training, and evaluation processes of nmt and asr models in our work are almost identical. however, the asr dataset (librispeech) has some properties that are differ from the machine translation setting: the average length of target sentences in the training dataset is much larger than in the test, which is why we chose this task for comparison. ## multi-sentence resampling in this section, we introduce multi-sentence resampling (msr) -a simple data augmentation method that alleviates the beam search problem by addressing dataset length bias and which increases the overall quality of translation models. specifically, msr augments a dataset such that instead of one sentence, each training example consists of 1 to n sentences, randomly chosen from a dataset and concatenated one after another. it preserves the order of sentences: the source side is concatenated to the source side, and the target side is concatenated to the target side of a new train example. for each new training example, msr randomly chooses from 1 to n sentences, so that the model does not overfit to the particular number of sentences. figure 4 illustrates the algorithm. the original training dataset, with 3 examples, is on the left, and the dataset created by multi-sentence resampling is shown on the right. the new dataset contains 6 train examples: 2 examples with one sentence, 2 examples with a concatenation of two original sentences, and 2 examples with a concatenation of 3 original sentences. the full procedure is described in algorithm 1. the algorithm takes an original train dataset, desired number of examples in the augmented dataset s, and the maximum number of sentences in an example n . multi-sentence resampling increases the average length of the dataset and provides a diverse set of long training examples. in contrast to other methods that use rescoring of hypotheses and per-token rewards @xcite or predict target length separately @xcite , our method does not change the search procedure. figure 5 illustrates how train examples length distribution changes in iwslt17 fr-en dataset for n from 2 to 5. with growing n distributions become more flatten for lengths presented in the test set. the average length of examples in a new train dataset can be approximately calculated as new_length » ## experiments in this section we provide an empirical evaluation of the multi-sentence resampling on the iwslt and wmt datasets. ## conclusions in this work, we analyzed errors that cause quality degradation with growing beam size in nmt and asr. we demonstrated that the major contribution to quality degradation on large beams comes from short translations, which are early terminated prefixes of hypotheses which are long when decoding with a small beams. in contrast to asr, we showed that the reference length on which beam search degradation begins to grow is connected with the low number of sentences longer than this length during training. thus, usual nmt datasets, that are biased towards short sentences, strongly contribute to the degradation with large beams. based on this finding, we introduced multi-sentence resampling -a simple data augmentation technique. it concatenates several sentences from a dataset, increasing the length of training examples. models trained with multi-sentence resampling were shown to consistently outperform baseline models on iwslt15 en-vi, iwslt17 en-fr, and wmt14 en-de datasets. thus, we demonstrate that it is possible to mitigate beam search degradation with data augmentation. future research directions include adapting multi-sentence resampling to other domains like asr and studying beam search problems for document-level machine translation, where adjacent sentences are naturally connected.