Document-Level Language Models for Machine Translation (2023)

Abstract: Despite the known limitations, most machine translation systems today still operate on the sentence-level. One reason for this is, that most parallel training data is only sentence-level aligned, without document-level meta information available. In this work, we set out to build context-aware translation systems utilizing document-level monolingual data instead. This can be achieved by combining any existing sentence-level translation model with a document-level language model. We improve existing approaches by leveraging recent advancements in model combination. Additionally, we propose novel weighting techniques that make the system combination more flexible and significantly reduce computational overhead. In a comprehensive evaluation on four diverse translation tasks, we show that our extensions improve document-targeted scores significantly and are also computationally more efficient. However, we also find that in most scenarios, back-translation gives even better results, at the cost of having to re-train the translation system. Finally, we explore language model fusion in the light of recent advancements in large language models. Our findings suggest that there might be strong potential in utilizing large language models via model combination.

## introduction machine translation (mt), the automatic translation of text from one language to another, has seen significant advancements in recent years, primarily driven by neural machine translation (nmt) models @xcite . these models have demonstrated remarkable capabilities in capturing complex linguistic patterns and producing high-quality translations @xcite . nevertheless, most models to-date operate on sentence-level, i.e. translate sentences independently without the context of the surrounding document. without access to such context, it is impossible for these mt systems to account for discourse-level phenomena such as resolution of ambiguous words and coherence. unsurprisingly, automatic translations are perceived as much worse, when they are evaluated on entire documents rather than just at the sentence-level @xcite @xcite . an obvious solution to this problem is to utilize context-aware mt models @xcite . while document-level nmt models have been thoroughly studied in recent years, sentence-level mt remains the standard despite its inherent limitations. one of the main reasons for this is that most of the document-level approaches rely on parallel training data with document-level metadata. most releases of large parallel training corpora lack this information and remain purely sentence-level @xcite . in contrast, large amounts of documentlevel monolingual data are readily available for almost all domains and languages. in this work, we strive to build a context-aware mt system that does not rely on any parallel document-level training data. instead, we use monolingual documents to train a document-level language model (lm), which we fuse with an existing sentence-level mt model during translation. while existing work on lm fusion shows that the fused model is able to incorporate document-level context @xcite , these approaches can be improved. our work aims to do so in two main directions. first, we acknowledge that nmt models implicitly learn the language modeling task during training. recently, @xcite showed that estimating and neutralizing this internal lm can improve translation quality for sentence-level mt. we adapt their approach to document-level lm fusion and demonstrate that this also improves discourse modeling. second, the contribution of the fused mt model, the document-level lm and the internal lm must be balanced by a set of fusion scales. existing work defines the fusion scales as static hyperparameters which are tuned on a validation set via an extensive grid search @xcite @xcite . in our work, we provide two simple alternatives to grid search which allow for automatically tuned context-dependent fusion scales. our approaches eliminate the need for expensive tuning and further improve discourse-modelling. the contributions of this work are as follows: ## related works most works on document-level nmt rely on parallel document-level data for system training. @xcite propose to concatenate adjacent sentences on source and target side and input this into the nmt model which has the exact same architecture as the vanilla sentencelevel transformer @xcite . later, many works have proposed modifications to the architecture to better accommodate the additional context @xcite @xcite @xcite @xcite . however, it has been shown that the simple concatenation approach performs as good, if not better than these more complicated variants @xcite . maybe the biggest challenge for document-level nmt is that most of the parallel mt training data is not document-level @xcite . recently there has been some effort to restore document-level meta information from existing sentence-level corpora but this is a very time consuming and error-prone process @xcite . therefore, approaches to document-level nmt have been proposed that utilize document-level monolingual data, of which typically large amounts are readily available. one direction is to back-translate the documentlevel monolingual data to create synthetic parallel document-level data. the reverse system used for back-translation can be either sentence-level @xcite @xcite or document-level @xcite . a downside of this approach is that the final mt system has to be re-trained to incorporate the new synthetic data. another line of work uses document-level language models in combination with sentence-level translation models. @xcite were the first to propose a log-linear combination of sentence-level language and nmt models, coining the term 'shallow fusion'. recently, it was shown that the shallow fusion approach for sentencelevel nmt can be improved by compensating for the implicitly learned internal language model of the nmt system @xcite . regarding the integration of a document-level lm, earlier approaches simply use the lm for re-ranking the hypothesis of the sentence-level nmt model @xcite . several works have proposed to employ a log-linear combination between sentence-level nmt system and document-level lm @xcite @xcite . both @xcite and @xcite propose to also include the probabilities of the lm without context information in order to mitigate the influence of the current sentence on the lm probabilities. while our approach also uses the output of a sentence-level lm, it is conceptually different from the previous works in that we want to mitigate the influence of the internal lm from the nmt model, resulting in a different final formulation. to further improve lm incorporation, @xcite propose to use subword-dependent fusion scales instead of a single scale per model. apart from back-translation and lm integration there exist some other ways to utilize additional monolingual document-level data for mt. @xcite train a document-level automatic post editing system on the monolingual data and use it to improve the hypotheses from a sentence-level nmt system in a two-pass approach. several works utilize the additional data in a multi-task learning approach (junczys-dowmunt, 2019) or for pre-training @xcite @xcite . very recently, llms have shown their potential for the task of document-level nmt @xcite . however, it is unclear how much parallel training samples were seen during the large scale pre-training on trillions of tokens. ## document-level language model fusion the sentence-level mt model translates a source sentence f into a target sentence e := e i 0 of subwords e i . in the document-level lm fusion approach, we additionally provide the k previous target-side sentences e -1 -k as context 1 . ## document-level language model pronoun re-ranking besides consistency, the main problem of discourse-modelling are ambiguities. e.g. translating the english pronoun 'it' to german requires access to the noun that it refers to, which might only be found in a preceding sentence @xcite . we propose an approach specific to the @xmath0. we first translate each sentence independently using the sentence-level mt model. each sentence-level translation is expanded to a set of candidates by replacing the pronouns with all alternatives ('er', 'sie', 'es'). all candidate translations are then scored in context of the preceding sentences using a document-level lm, and we select the pronoun for which the lm score is highest. this approach is very much tailored to the specific pronoun translation problem for this specific language pair. while it is theoretically possible to extend this approach to cover more cases, this will require extensive human effort and is probably not feasible in most scenarios. however, we include it here, because it serves as a reasonable baseline for this popular pronoun translation benchmark. ## document-level back-translation the to-date most popular way of utilizing monolingual data for mt is to create synthetic parallel training data via back-translation @xcite . we train a sentence-level backwards mt system on the parallel data and use it to translate the document-level monolingual data back into the source language. the sentence-level translations are concatenated to obtain synthetic parallel documents (junczys-dowmunt, 2019; @xcite @xcite . to train the final systems we combine the authentic sentence-level parallel and the synthetic document-level data. combining both data sources is not straightforward, because of their varying size and the difference between sentence/documentlevel context. therefore, we first oversample the data accordingly to have roughly the same number of sentences in both parts. secondly, we turn the authentic sentence-level parallel data into 'pseudodocuments' by concatenating them in a random order @xcite . this ensures that all training data has the same context size. we found this procedure to perform best when incorporating synthetic document-level data. for a detailed comparison, see appendix a.5. ## experiments 6.1 tasks we evaluate our approaches on four different tasks of varying data conditions and domains. three tasks are on publicly available data and a fourth task is based on a large scale internal dataset in the e-commerce domain. all tasks include (sentencelevel) parallel training data and document-level monolingual data from the same domain. the exact data conditions are provided in appendix a.1. the news @xmath1. both are low resource with less than 1m training samples in total. the subtitles @xmath2. finally, the e-commerce @xmath3. while the parallel training data for the three academic tasks does provide document-level metadata, our approaches do not make use of this information and we assume that the parallel training data is sentence-level for most experiments. we only make use of this information to provide a direct comparison against the setting where documentlevel parallel data is assumed to be available. as paracrawl, like most other large-scale web-crawled parallel datasets, is not a document-level corpus, we can not conduct these experiments for the e-commerce task. we preprocess each corpus with byte-pair encodings @xcite using the sentencepiece toolkit @xcite learned on the parallel dataset with a shared vocabulary of 32k subwords (13.6k for ted). for the e-commerce task we additionally use inline casing @xcite . ## conclusions this work presents multiple extensions to document-level lm fusion, a technique of utilizing document-level monolingual data for contextaware mt. in comparison to existing work, our extensions significantly improve discourse-modeling finally, this work is the first to explore documentlevel lm fusion with llms. first findings demonstrate that fusion with an llm outperforms a small lm trained on in-domain data and open the path for future investigations.