D oc HPLT : A Massively Multilingual Document-Level Translation Dataset (2025)

Abstract: Existing document-level machine translation resources are only available for a handful of languages, mostly high-resourced ones. To facilitate the training and evaluation of document-level translation and, more broadly, long-context modeling for global communities, we create DocHPLT, the largest publicly available document-level translation dataset to date. It contains 124 million aligned document pairs across 50 languages paired with English, comprising 4.26 billion sentences. By adding pivoted alignments, practitioners can obtain 2500 additional pairs not involving English. Unlike previous reconstruction-based approaches that piece together documents from sentence-level data, we modify an existing web extraction pipeline to preserve complete document integrity from the source, retaining all content, including unaligned portions. After our preliminary experiments identify the optimal training context strategy for document-level translation, we demonstrate that LLMs fine-tuned on DocHPLT substantially outperform off-the-shelf instruction-tuned baselines, with particularly dramatic improvements for under-resourced languages. We open-source the dataset under a permissive license, providing essential infrastructure for advancing multilingual document-level translation.

## introduction the field of natural language processing (nlp) is shifting its focus toward end-to-end, complex tasks, including the domain of machine translation. this increases the demand for techniques and resources beyond the sentence level, with document-level machine translation (docmt) being a prime example @xcite @xcite . while there is not a single definition of a document, docmt requires models to translate more than one sentence ⋆ equal contribution. public access to dochplt: https: //huggingface.co/datasets/hplt/dochplt . as a coherent unit rather than isolated segments. this approach is necessary for handling various discourse phenomena: anaphora, deixis, ellipsis, discourse connectives, grammatical and lexical cohesion @xcite , which sentence-level translation typically loses @xcite @xcite . recent longcontext large language models (llms) are wellsuited for this task, as they are usually pre-trained to process thousands of tokens at a time. however, docmt remains largely unexplored or untested for most languages due to a simple but significant problem: we lack document-level parallel data for both model building and evaluation. historically, parallel corpora were mostly constructed in a sentence-oriented manner, using a pipeline that split the text into sentences, aligned them, and then discarded unaligned and multiplyaligned sentences. while a handful of language pairs have some document-level mt resources, the majority of languages have none. this creates two related problems at once: we cannot build docmt for these languages, and we cannot evaluate docmt properly. as nlp research moves toward more end-to-end, context-aware applications, this data gap means that most languages get left behind. we tackle this problem by extracting parallel documents from large web crawls, but our methodology differs from the majority of previous efforts that reconstruct data from sentence pairs after the fact. instead, we modify the web extraction pipeline itself to preserve document structure from the beginning, retaining documents in their entirety with all original context and non-parallel text. for each language pair, we deliver the aligned documents along with quality-scored sentence alignments and alignment density metrics. our effort yielded dochplt, a large multilingual document-level translation dataset covering 50 language pairs with english, listed in appendix a. the resulting corpus contains 87.8 million documents in english and 50 other languages, 124 million aligned document pairs, and 4.26 billion sentences. a highlight of our work is the focus on medium-and low-resource languages that previous docmt datasets have overlooked. practitioners can also use english as a pivot to align up to 2500 extra non-english pairs, expanding the dataset's usefulness beyond english-centric translation. using dochplt, we conduct extensive experiments with different modeling methods for llmbased document-level translation. we first try different context sizes for llm fine-tuning: 1) full document-to-document training with loss calculated on the entire target; and 2) chunk-based training with loss computed on individual segments. these experiments determine the optimal context granularity for our subsequent work. then, in addition to prompting off-the-shelf instruction-tuned large language models (llms) as a baseline, we run monolingual and multilingual fine-tuning using dochplt, tested on both seen and unseen languages. the usefulness of our data is reflected empirically: llms fine-tuned on our data consistently outperform prompting baselines, showing that practitioners can gain strong performance in docmt for languages often considered "unsupported" in the machine translation research community. in summary, our contributions, centred around the dochplt resource, are as follows: ## dochplt in this section, we explain how we modify and then apply an existing parallel sentence extraction pipeline from paracrawl to extract a documentlevel corpus from a large multilingual web crawl, hplt. ## experiments and findings in order to test the usefulness of our dataset, we apply it to the task of fine-tuning llms for mt. our first set of experiments tests different context lengths for this fine-tuning to see how much performance is affected by using the larger document contexts that dochplt enables. we then compare this best-performing fine-tuned configuration against off-the-shelf instruction-tuned models on the same test set. finally, we investigate monolingual and multilingual fine-tuning for docmt on dochplt. in the following sections, the notation of "src-trg" refers to the src-to-trg translation direction. languages we test translation from english into a total of 10 languages, chosen for their diversity in script, typology, and resource availability, as well as their inclusion in wmt24++ @xcite (for testing) and cometkiwi @xcite (for filtering). the languages are arabic (ar), catalan (ca), hindi (hi), estonian (et), persian (fa), finnish (fi), icelandic (is), korean (kr), malayalam (ml), and urdu (ur). it is worth noting that generating non-english is usually harder for llms compared to generating english. ## conclusion we introduced a pipeline to derive a documentlevel corpus with rich metadata and presented the outcome, dochplt, the largest publicly available document-level translation dataset with 124 million aligned document pairs across 50 languages paired with english. the utility of our massively multilingual dataset has been demonstrated through experiments: fine-tuning llms on our data improved over prompting baselines, and multilingual training surpassed monolingual models, though zero-shot transfer to unseen languages remained challenging. our experiments also revealed challenges in docmt: full document-to-document training and generalization to other document domains. future work may use dochplt data for further investigations such as large-scale training, data filtering, data synthesis, and docmt metric study.