TRIP : Accelerating Document-level Multilingual Pre-training via Triangular Document-level Pre-training on Parallel Data Triplets (2023)

Abstract: Despite the success of multilingual sequence-to-sequence pre-training, most existing approaches rely on document-level monolingual corpora in many different languages, sentence-level bilingual corpora, and sometimes synthetic document-level bilingual corpora. This hampers the performance with cross-lingual document-level tasks such as document-level translation. Hence, we propose to mine and leverage document-level trilingual parallel corpora to improve sequence-to-sequence multilingual pre-training. We present Tri angular Document-level P re-training ( TRIP ) as the first in the field to accelerate the conventional monolingual and bilingual objectives into a trilingual objective with a novel method called Grafting. Experiments show that TRIP achieves several strong state-of-the-art (SOTA) scores on three multilingual document-level machine translation benchmarks and one cross-lingual abstractive summarization benchmark, including consistent improvements by up to 3.11 d-BLEU points and 8.9 ROUGE-L points.

## introduction conventional multilingual pre-training achieved promising results on machine translation @xcite and cross-lingual classification @xcite . these pre-training paradigms usually rely on monolingual corpora in many different languages, with denoising objectives such as sentence permutation and span masking @xcite . following the calls that the unsupervised scenario is not strictly realistic for cross-lingual learning @xcite , multilingual pre-training advanced into a supervised setting through sentence-level bilingual translation pairs @xcite to provide a stronger signal for pre-training. among these pioneering works, document-level multilingual pre-training with parallel data is currently an understudied topic. this direction is particularly significant for tasks that necessitate contextual comprehension, such as document-level machine translation and cross-lingual summarization. as a workaround, docmt5 @xcite resorts to using synthetic bilingual translation pairs to scale up document-level multilingual pre-training. in addition to the lack of study for documentlevel multilingual pre-training with parallel data, prior works also overlooked the value of trilingual parallel data for multilingual pre-training. compared to bilingual parallel data, trilingual parallel data is expected to better capture different linguistic clues and coherence among different languages such as past tense and gendered expressions, foot_1 which can enhance the model pre-training on aspects of document-level cross-lingual understanding and resolve cross-lingual ambiguities. to this end, we present trip, a document-level multilingual pre-training method using trilingual parallel corpora. because there is no publicly available document-level trilingual corpus, we propose a novel method to construct trilingual document pairs from document-level bilingual corpora. subsequently, we augment the conventional multilingual pre-training by (i) grafting two documents presented in two different languages into one mixed document, and (ii) predicting the remaining one language as the reference translation. models denoising translation trilingual trilingual document pre-training pre-training document pairs objective level mbart mt5 mt6 paradise docmt5 trip we conduct experiments on document-level machine translation on ted talks @xcite , news benchmark (news-commentary) and europarl @xcite , and cross-lingual abstractive summarization on wikilingua @xcite . we found that trip clearly improves previous multilingual pre-training paradigms that use monolingual and bilingual objectives @xcite , and achieves strong sota results on both tasks. in summary, we make three key contributions: ## triangular document-level pre-training we start by introducing the conventional methodologies previously used by the monolingual and bilingual objectives for multilingual pre-training: ## conclusions we present a novel sequence-to-sequence multilingual document-level pre-training methodology called trip, which is the first in our field to propose a trilingual objective for multilingual pretraining through grafting. we also propose a novel method to construct high-quality trilingual document pairs. experimental results indicate that trip achieves competitive sota scores on both multilingual document-level machine translation and crosslingual abstractive summarization. future works could improve trip to include polygonal parallel translation pairs in multilingual pre-training. we plan to release the model checkpoints and a manually annotated benchmark created using our created document-level corpus mtdd to facilitate future research on multilingual document-level mt. limitations trip trip leverages high-quality documentlevel trilingual translation pairs for pre-training on multilingual models. it is usually harder to collect high-quality trilingual data than to collect monolingual data written in different languages used by conventional methods. while we can possibly relax the quality bar for the data, additional experiments should be done to verify this view. mtdd we create mtdd, a corpus that is composed of trilingual document pairs. it could be further extended to include polygonal parallel document pairs to provide a stronger signal for multilingual pre-training. we leave this to future works. large language models large language models (llms) such as chatgpt have shown good translation abilities @xcite , while they still lag behind supervised systems @xcite . we conduct a limited comparison to them, as they are much larger in their number of parameters than the systems described in this work.