Scaling Law for Document Neural Machine Translation (2023)

Abstract: The scaling laws of language models have played a significant role in advancing large language models. In order to promote the development of document translation, we systematically examine the scaling laws in this field. In this paper, we carry out an in-depth analysis of the influence of three factors on translation quality: model scale, data scale, and sequence length. Our findings reveal that increasing sequence length effectively enhances model performance when model size is limited. However, sequence length cannot be infinitely extended; it must be suitably aligned with the model scale and corpus volume. Further research shows that providing adequate context can effectively enhance the translation quality of a document’s initial portion. Nonetheless, exposure bias remains the primary factor hindering further improvement in translation quality for the latter half of the document.

## introduction neural machine translation (nmt) @xcite has made great progress in recent years @xcite . however, as the input text exceeds a single sentence, sentence-level nmt methods will fail to capture discourse phenomena, such as pronominal anaphora, lexical consistency, and document coherence. in contrast, document-level neural machine translation (dnmt) @xcite aims to improve translation consistency and coherence by leveraging contextual information beyond the sentence being translated. different from previous sentence-by-sentence dnmt methods, which employ an additional context encoder to capture the discourse information @xcite , recent studies have shown that the transformer @xcite is able to translate multiple sentences directly @xcite @xcite , referred to as document-by-document translation method. these works demonstrated that increasing the granularity of translation from sentence to document is able to further reduce the context-dependent errors encountered by sentence-level translation methods, such as misinterpretation of pronouns @xcite and lexical inconsistency @xcite . although has been widely explored, the mechanism of dnmt has not been fully investigated. for example, previous document-by-document dnmt methods are trained on a fixed maximum sequence length, i.e. the document longer than the maximum sequence will be split into smaller segments, and whether further increasing the maximum sequence length can continue to improve the performance of the model still remains to be investigated. additionally, as the sequence length grows, the difficulty of modeling the contextual information increases further, and whether more training data is needed remains to be explored. furthermore, the scale of model parameters is among the most crucial factors affecting the quality of sentence-level nmt methods and is widely studied by the scaling law of neural machine translation @xcite , but how this factor affects the performance of the dnmt model remains unclear. finally, it remains an open question whether these factors will interact with each other. to answer the questions above, we conduct massive experiments, specifically investigating the impacts of three key factors: corpus size, model scale, and maximum sequence length. in addition, we explore the joint effect of these factors. our findings suggest that all three factors are closely related to the performance of the dnmt model and enlarging the maximum sequence length effectively improves the ability of the dnmt model to capture the contextual information. however, the sequence length can not be increased infinitely as the translation quality begins to drop when the sequence length exceeds a certain limit. surprisingly, our experimental result shows that this limitation is proportional to the logarithm of the number of model parameters. our further analysis demonstrates that determining the optimal maximum sequence length requires striking a balance between capturing additional contextual information and managing accumulated errors during auto-regressive decoding. this finding highlights the importance of future research in exploring effective solutions to mitigate this challenge. furthermore, we find that although the dnmt model is initialized using a pretrained sentence-level translation model, increasing the document-level corpus size can further improve the translation quality. notably, this improvement increases as the number of model parameters increases. as figure 1 concludes, our study indicates that the number of model parameters affects the translation quality of dnmt by affecting two key factors: sample efficiency and optimal sequence length. as the number of model parameters increases, both the sample efficiency and the optimal sequence length grow, leading to the performance improvement of the dnmt model. ## experiments in this section we present the results of our experiments. first, we analyze the effect of single factor on the translation quality of dnmt in section 4.1. then we investigate the joint effect of multiple factors in section 4.2. ## conclusion in this paper, we explore the scaling law of document-level neural machine translation. our findings suggest that the corpus size, model scale and the maximum sequence length all have significant impacts on the translation quality of the model. different from the scaling law of sentencelevel neural machine translation, we find that their exists an optimal maximum sequence length for a given dnmt model, and the optimal maximum sequence length is strongly correlated with the number of non-embedding parameters of the dnmt model. we further formulate this correlation with a simple power law. our additional analysis indicates that the error accumulation problem is the primary factor that hindering further improvement in translation quality for the document-level translation. in future work, we will investigate how to overcome the error accumulation problem, and extend the optimal sequence length of document-by-document translation.