Exploring Paracrawl for Document-level Neural Machine Translation (2023)

Abstract: Document-level neural machine translation (NMT) has outperformed sentence-level NMT on a number of datasets. However, document-level NMT is still not widely adopted in realworld translation systems mainly due to the lack of large-scale general-domain training data for document-level NMT. We examine the effectiveness of using Paracrawl for learning document-level translation. Paracrawl is a large-scale parallel corpus crawled from the Internet and contains data from various domains. The official Paracrawl corpus was released as parallel sentences (extracted from parallel webpages) and therefore previous works only used Paracrawl for learning sentence-level translation. In this work, we extract parallel paragraphs from Paracrawl parallel webpages using automatic sentence alignments and we use the extracted parallel paragraphs as parallel documents for training document-level translation models. We show that document-level NMT models trained with only parallel paragraphs from Paracrawl can be used to translate real documents from TED, News and Europarl, outperforming sentence-level NMT models. We also perform a targeted pronoun evaluation and show that document-level models trained with Paracrawl data can help context-aware pronoun translation.

## introduction the transformer translation model @xcite , which performs sentence-level translation based on attention networks, has achieved great success and significantly improved the state-of-theart in machine translation. compared to sentencelevel translation, document-level translation @xcite @xcite @xcite @xcite performs translation at document-level and can potentially fur-ther improve translation quality, e.g., documentlevel context can help word disambiguation for translating words with multiple senses, documentlevel translation can help pronoun translation which requires context outside of the current sentence @xcite , document-level translation can improve document-level lexical cohesion in the translation @xcite . document-level neural machine translation (nmt) has received much attention in recent years @xcite @xcite @xcite @xcite @xcite @xcite @xcite . existing works showed that document-level translation can outperform sentence-level translation for a number of datasets, such as ted, news, europarl @xcite @xcite . although document-level nmt has shown promising results on a number of benchmarks, document-level nmt is still not widely adopted in real-world translation systems mainly due to the lack of large-scale general domain training data for document-level nmt. we examine the effectiveness of using paracrawl @xcite for learning document-level nmt. paracrawl is a large-scale parallel corpus crawled from the internet and contains data from various domains. the official paracrawl corpus 2 was released as parallel sentences (extracted from parallel webpages) and therefore previous works only used paracrawl for learning sentence-level translation. in this work, we extract parallel paragraphs from paracrawl parallel webpages using automatic sentence alignments and we use the extracted parallel paragraphs as parallel documents for training document-level translation models. we show that document-level nmt models trained with only parallel paragraphs from paracrawl can be used to translate real documents from ted, news and europarl, outperforming sentence-level nmt models and also improving context-aware pronoun translation @xcite . ## extracting parallel paragraphs from paracrawl paracrawl @xcite ) is a large-scale parallel corpus crawled from the internet and contains texts from various domains. the official paracrawl corpus was released as aligned sentence pairs. for high-resource language pairs like german-english, the paracrawl corpus provides 278m aligned sentences for the translation task. as paracrawl was released as aligned sentence pairs, previous work only used paracrawl for learning sentence-level translation. however, as described in the paracrawl documentation, paracrawl was constructed by first identifying aligned webpages (urls) and then aligning sentences within aligned webpages. although texts in parallel webpages tend to be noisy and are only loosely aligned, we demonstrate that extracting parallel paragraphs from paracrawl parallel webpages using automatic sentence alignments can provide effective training data for learning document-level translation. below we describe how we extract parallel paragraphs from paracrawl. extracting we extract parallel paragraphs from paracrawl and examine the effectiveness of using these aligned paragraphs for learning documentlevel nmt. we used german-english, a highresource language pair, in our experiments. as the main purpose of the original paracrawl project is to collect aligned sentences, paracrawl did not officially release all the webpage-level aligned texts. paracrawl only released a subset of all the aligned webpage texts, for the german-english language pair foot_2 . we extract parallel paragraphs from the released parallel webpages: we first download aligned webpages foot_3 (one german webpage is aligned with one english webpage) and then download the automatic sentence alignments (vecalign foot_4 ) provided by paracrawl; we then split a webpage into paragraphs according to the newline symbol; then extract aligned paragraphs from aligned webpages according to the automatic sentence align cleaning to improve the quality of the extracted parallel paragraphs, we removed sentences which do not belong to the correct language foot_5 , removed paragraphs that are too short (contain less than 30 words) and removed paragraphs with more than 50% overlap. we randomly split the extracted parallel paragraphs into training, development and test sets as shown in we show the length statistics of the parallel paragraphs that we extracted from paracrawl in ## conclusion as document-level translation lacks large-scale general-domain document-level training data, we examine the effectiveness of using paracrawl data for learning document-level translation. paracrawl was officially released as parallel sentences extracted from parallel webpages. in this work, we extract parallel paragraphs from paracrawl aligned webpages using automatic sentence alignments and we use the extracted parallel paragraphs as parallel documents for training document-level nmt models. we show that document-level models trained with only parallel paragraphs from paracrawl can be used to translate real documents from ted, news and europarl, outperforming sentence-level transformers and also improving context-aware pronoun translation.