Document-Level Machine Translation with Large-Scale Public Parallel Corpora (2024)

Abstract: Despite the fact that document-level machine translation has inherent advantages over sentence-level machine translation due to additional information available to a model from document context, most translation systems continue to operate at a sentence level. This is primarily due to the severe lack of publicly available large-scale parallel corpora at the document level. We release a large-scale open parallel corpus with document context extracted from ParaCrawl in five language pairs, along with code to compile document-level datasets for any language pair supported by ParaCrawl. We train context-aware models on these datasets and find improvements in terms of overall translation quality and targeted document-level phenomena. We also analyse how much long-range information is useful to model some of these discourse phenomena and find models are able to utilise context from several preceding sentences.

## introduction machine translation has traditionally been framed as a problem of translating source text to target text one sentence at a time. however, depending on the languages and the content of the text being translated, it is often the case that a sentence is impossible to translate well in isolation -that further contextual information is required. @xcite summarise several discourse phenomena that are impossible for sentence-level machine translation systems to deal with -including anaphoric pronouns, lexical cohesion, deixis, and ellipsis. there are also features like grammatical gender, number, style, and formality, that can sometimes not be determined from the individual sentence but are dependent on surrounding context. therefore, it has been clear for many decades @xcite ) that a machine translation system cannot translate some sentences without the ability to capture other linguistic cues from context. @xcite also showed that while sentence-level neural machine translation can appear high-quality out of context, human evaluators had a much stronger preference for human translation when evaluating translation at the document level. as a result, there have been many efforts to incorporate document context into neural machine translation. what almost all of these methods have in common is that they require parallel training data with document context. paracrawl @xcite produced largescale parallel corpora and the released data includes information about the urls from which the sentences were extracted, but the released corpora were only sentence-level. we use raw webpage text publicly available from paracrawl along with the officially released sentence-level corpora to assemble large-scale document-level parallel corpora for several language pairs. we release our code to generate the document-level datasets foot_0 as well as the datasets in five selected language pairs foot_1 . we then validate the usefulness of our datasets by training context-aware translation models for all of these language pairs, and find that models that are aware of target context perform better than sentence-level baselines, often in terms of overall translation quality, but more significantly when evaluated with respect to targeted discourse phenomena. we experiment with varying the amount of preceding context available to these contextaware models at test time, and find that while information from the immediately previous sentence is most useful -as is intuitively obvious, longer-range context up to a certain extent can also help models translate phenomena like anaphoric pronouns more accurately. some methods like @xcite ; @xcite ; @xcite eliminate sentence boundaries altogether and use a standard transformer architecture to translate an entire chunk of text as a single sequence. however, unless they retain some sentence markers like in @xcite or @xcite , these models can be difficult to evaluate with our existing evaluation paradigms and metrics due to the dependence on sentencelevel test sets. in those cases, we often have to rely on sentence-splitting heuristics and alignment methods just to be able to compute a sentence-level metric on the model outputs. as a result, our work chooses a method to translate with one sentence at a time as input, but with document context provided as a separate additional input. this allows the model to benefit from context information while still being simple to evaluate with existing metrics and test sets. there are few existing parallel corpora of significant size that retain document metadata -examples are europarl @xcite which had only around 2m sentences for the largest language pairs; czeng @xcite containing 61m sentence pairs with document annotation along with more than 100m synthetic sentence pairs, but only for eng↔ces; opensubtitles @xcite ; and news commentary @xcite , where the latter two datasets are relatively large corpora for several language pairs but restricted in domain. the ccaligned corpus (el-kishky et al., 2020) includes hundreds of millions of comparable document pairs across many languages, from which sentence-level datasets were extracted and released. a dataset with sentence pairs and corresponding document contexts was not created; however, it should be possible to extract a similar dataset as the one we present here from the available data released by ccaligned which includes documents, urls, and sentences. closest to our data, al ghussin et al. (2023) used publicly available parallel document metadata from paracrawl foot_2 to extract aligned paragraphs of text, and used these paragraphs as a proxy for documents. even though their extracted datasets were at a relatively small scale due their use of only a subset of paracrawl data and strict filtering, they observed clear improvements in targeted evaluations of document-level translation phenomena. post @xcite showed that using only monolingual documents and backtranslating @xcite them sentence by sentence to produce synthetic document pairs can surprisingly produce better results than using actual document pairs to train a document-level model. their results, however, were mostly on unreleased private data, and their comparison could not be reproduced on public data precisely because of the absence of public datasets of adequate size. another recent orthogonal approach is to use large language models' inherent ability to model long context to perform document-level translation @xcite @xcite with no or very few parallel training examples. while this paradigm is gaining popularity, it is yet to be comprehensively explored, and the need remains to have large-scale datasets of parallel text with document context. ## dataset at the time of its release, paracrawl @xcite was the largest publicly available sentencelevel parallel corpus for most of the languages it supported. the paracrawl corpus mining process included steps to match documents that were estimated to be translations of each other, from which sentences were extracted and aligned, but unfortunately, the released corpora did not preserve document context or structure, and only contained isolated sentence pairs along with the source urls they were originally extracted from. however, separately, a lot of the raw text crawled from the web was also released foot_3 as languageclassified base64-encoded text with their corresponding urls. therefore, we were able to match the webpage contents to their urls in the sentencelevel parallel corpora to recover the corresponding documents. to build document-level parallel datasets from these sources of data, we chose five language pairs -czech (ces), polish (pol), german (deu), french lang. sentences source target both deu 278.3 105.6 110.3 92.1 fra 216.6 83.5 86.3 72.2 ces 50.6 18.7 21.0 16.3 pol 40.1 16.8 18.4 14.9 rus 5.4 3.1 2.8 2.4 the sizes of our extracted datasets are shown in some samples can be found in appendix a, and the full datasets are publicly available at https://huggingface.co/ datasets/proyag/paracrawl_context . ## document-level translation models to evaluate the usefulness of our datasets, we train document-level translation models using only our datasets. even though higher-quality documentlevel training data exists at a smaller scale, we choose to train our models only on data from paracrawl in order to accurately evaluate the quality and utility of our data and to make a fair comparison with our sentence-level baselines. at training time, for each input example, we first sample one context out of up to 1000 that are present in the document-level dataset. to ensure that the models are capable of using variable-length context at test time, we uniformly sample a context length l from {1, . . . , 256} for each training example. we then retain at least l tokens from the preceding context, possibly exceeding the limit to avoid mid-sentence splits. this context sampling was implemented using a custom pipeline foot_4 in the sotastream toolkit @xcite . we then use the source sentence as the main model input, the sampled context as a second input (see detailed model architecture in section 4.1), and the target sentence as the target model output. we train separate models for each language pair using either source or target context information. while the model is always provided ground-truth context at training time, this is not always available in the case of target context at test time, so we test using both ground-truth target context and real predicted output context. however, we note that one of the most common use cases of machine translation is in the context of computer-assisted translation (cat) tools, where translators can see preceding context but typically machine translate and post-edit one sentence at a time, as a result of which gold-standard target context is available for each sentence. ## results and analysis we train sentence-level and document-level models in a few different configurations for comparison. our baseline is a standard sentencelevel transformer-big model trained on all of the paracrawl parallel data for a given language pair. we also train sentence-level baselines trained on the subsets of sentence pairs for which source or target contexts could be extracted, thus ensuring a fair comparison in terms of the number and content of training examples. finally, for each language pair, we train two different document-level models: one which is aware of source context and one which uses target context. we further test the target context model in two different scenarios: using original ground-truth context for each test example and using the actual model output from previous sentences within a document as target context. ## conclusion we release large-scale document-level parallel corpora in five language pairs extracted from the paracrawl datasets in an effort to mitigate the dearth of publicly available machine translation training data with document context. we also open-source code to enable the community to compile such datasets in more language pairs. due to both the paracrawl pipeline and our code be-ing open-source, it is theoretically possible to create document-level datasets for any supported language by crawling the web, and not just those already released by paracrawl. while we treat any preceding text at the same url as "context", it is often the case that these are completely unrelated to a given sentence, such as ui elements, boilerplate text, or entirely unrelated content on the same webpage. future work should also explore filtering these datasets to retain only genuine contextual information, which is likely to be much more useful to the model, although even content that is not strictly document context may help guide translation through indirect domain or style cues. our document-level translation experiments show that models aware of target context improve in terms of overall translation quality as well as in terms of some targeted discourse phenomena compared to a sentence-level baseline. we show that machine translation can benefit from multiple sentences of preceding context to accurately translate discourse phenomena like anaphoric pronouns, although very long-range context is rarely useful.