Addressing the Length Bias Challenge in Document-Level Neural Machine Translation (2023)

Abstract: Document-level neural machine translation (DNMT) has shown promising results by incorporating context information through increased maximum lengths of source and target sentences. However, this approach also introduces a length bias problem, whereby DNMT suffers from significant translation quality degradation when decoding sentences that are much shorter or longer than the maximum sentence length during training, i.e., the length bias problem. To prevent the model from neglecting shorter sentences, we sample the training data to ensure a more uniform distribution across different sentence lengths while progressively increasing the maximum sentence length during training. Additionally, we introduce a length-normalized attention mechanism to aid the model in focusing on target information, mitigating the issue of attention divergence when processing longer sentences. Furthermore, during the decoding stage of DNMT, we propose a sliding decoding strategy that limits the length of target sentences to not exceed the maximum length encountered during training. The experimental results indicate that our method can achieve state-of-the-art results on several open datasets, and further analysis shows that our method can significantly alleviate the length bias problem.

## introduction document-level neural machine translation (dnmt) @xcite @xcite @xcite @xcite ) is proposed to enhance translation quality by leveraging more contextual information. recently, the document-to-document (doc2doc) dnmt model (junczys-dowmunt, 2019; @xcite @xcite , which expands the translation scope from individual sentences to entire documents, has demonstrated exceptional performance, thereby drawing increased attention. for the training of doc2doc dnmt model, multiple sentences are assembled into sequences that are close to the predetermined maximum length, enabling the model to learn information from the context as much as possible. however, this training strategy can lead to overfitting to the maximum length. sequences that are significantly shorter than the maximum length may be overlooked by the model due to their smaller proportion in the training set. besides, the model also lacks the ability to handle the sequences that exceed the maximum length, which are not encountered by the model during training. consequently, the length bias problem results in a significant degradation in translation quality when the length of the decoded sequence deviates from the maximum sequence length, which is shown in figure 1 . some researchers have made their attempts to enhance the length generalization capabilities of dnmt model from various perspectives. some approaches employ data augmentation techniques @xcite to mix documents with shorter segments such as sentences or paragraphs, thereby augmenting the diversity of sequence lengths in the training set. however, the proposed augmentation method does not necessarily guarantee a balanced length distribution, as the length distribution is still influenced by the training corpus itself. @xcite incorporates a locality assumption as an inductive bias into the transformer model, which reduces the complexity of target-to-source attention. as a result, their method allows for the setting of larger maximum lengths, thereby augmenting the model's ability to handle longer documents. however, this method can only bring limited improvements for the short sequences. besides, the aforementioned approaches are still incapable of directly handling sequences that exceed the maximum sequence length during testing and still require segmentation of excessively long test sequences. given above, we aim to enhance the capability of our model to handle both long and short sequences. additionally, we seek to enable the model to directly translate sequences that exceed the maximum length, thereby avoiding information loss caused by segment truncation. to achieve these objectives, we have made improvements in the sampling of training data, attention weight computation, and decoding strategies. during training, we first sample the sequence lengths and then construct the training data accordingly. this dynamic variation in sequence lengths within different epochs ensures that the model encounters a more balanced distribution of sequence lengths during training. furthermore, we introduce a scaling factor during attention computation to ensure that, even as the sequence length increases, the model can still focus on relevant target information and prevent attention divergence. lastly, when decoding sequences that exceed the maximum length, we employ a sliding window decoding strategy which allows for the retention of more context information while ensuring that the context length remains below the maximum sequence length. these three proposed methods collectively contribute to improving the length generalization capabilities of the dnmt model from different perspectives. moreover, their combined application yields further performance enhancements. we conduct experiments on several document-level open datasets and the experimental results indicate that our method can bring significant improvements. further analysis shows that our method can significantly alleviate the length bias problem. ## background in this section, we will give a brief introduction to the transformer @xcite model and the doc2doc dnmt model. ## method our method aims to enhance the length generalization capabilities of the doc2doc dnmt model, thereby alleviating the length bias problem. to achieve this goal, we have made improvements in three aspects: training data sampling, attention computation, and decoding strategies. ## conclusion in this work, we aim to address the issue of length bias in the training of the doc2doc dnmt model. to achieve this objective, we propose several methods, including dynamic length sampling, length aware attention and sliding decoding. we conduct experiments on multiple publicly available datasets, and the results demonstrate a significant improvement achieved by our method. further analysis indicates that our approach can enhance the length generalization capability of the model effectively.