DOC m T 5: Document-Level Pretraining of Multilingual Language Models (2022)

Abstract: In this paper, we introduce DOCmT5, a multilingual sequence-to-sequence language model pretrained with large-scale parallel documents. While previous approaches have focused on leveraging sentence-level parallel data, we try to build a general-purpose pretrained model that can understand and generate long documents. We propose a simple and effective pretraining objective - Document reordering Machine Translation (DrMT), in which the input documents that are shuffled and masked need to be translated. DrMT brings consistent improvements over strong baselines on a variety of document-level generation tasks, including over 12 BLEU points for seen-language pair document-level MT, over 7 BLEU points for unseen-language-pair document-level MT and over 3 ROUGE-1 points for seen-language pair cross-lingual summarization. We achieve state-of-the-art (SOTA) on WMT20 De-En and IWSLT15 Zh-En document translation tasks. We also conduct extensive analysis on various factors for document pretraining, including (1) the effects of pretraining data quality and (2) The effects of combining mono-lingual and cross-lingual pretraining. We plan to make our model checkpoints publicly available.

## introduction multilingual pretrained language models have been useful for a wide variety of nlp tasks. pretraining on large-scale multilingual corpora facilitates transfer across languages and benefits low-resource languages. previously, sentence-level or word-level crosslingual objectives have been considered for pretraining large language models (llm), but not much effort has been put in document-level objectives for pretraining. in this work, we propose a multilingual sequence-to-sequence language model pretrained with cross-lingual structure-aware document-level objectives. docmt5 is built on top of mt5 @xcite and is further trained with parallel documents across multiple language pairs. to encourage the model to gain a deep understanding of the document structure and cross-lingual relationships, we consider a challenging translation scenario as a second-stage pretraining task: the input sentences are shuffled in a random order and random spans are masked. to effectively translate the input document, the model needs to reconstruct the document in the original order, making the model learn sentence relationships, and also recover the masked spans. this objective is effective on documentlevel generation tasks such as machine translation and cross-lingual summarization, outperforming previous best systems. to enable cross-lingual pretraining at a large scale, we created a synthetic parallel document corpus. to avoid expensive human annotation, we use off-the-shelf neural machine translation (nmt) models to translate the documents in the mc4 corpus @xcite into english. in our experimental results, this corpus is more effective for pretraining than existing large-scale automatically aligned corpora (e.g., ccaligned @xcite ). we also conduct extensive ablation studies and provide insights on document-level pretraining. we show that simple document-level pretraining is more useful than sentence-level pretraining for generative tasks. we also show that data quality matters when performing multilingual document pretraining. finally, we don't observe improvements from combining mono-lingual and cross-lingual objectives when evaluating on two document-level translation tasks. in summary, this paper makes the following contributions: ## conclusion in this paper, we present docmt5, a novel document-level multilingual pre-trained model. our proposed objective, drmt, is simple and effective and leads to large gains over strong baselines (e.g. mbart and marge) on crosslingual summarization and document-level translation. docmt5 achieved sota on two competitive document-level translation tasks: wmt20 de-en and iwslt15 zh-en. we further analyze various factors that contribute to successful document-level pre-training. we plan to release the pre-trained model to facilitate future work on document-level language understanding. and when i was 11 years old, i remember waking up one morning to the sound of a happy voice in the house. my father was listening to the bbc on his little gray radio. he had a smile on his face, which is rare, because most of the news was depressing. "the taliban are gone!" my father shouted. i didn't know what that meant, but i could see that my father was very, very happy. "you can go to a real school now," he said. and i will never forget that morning. a real school. when i was six years old, the taliban occupied afghanistan and made it illegal for girls to go to school. so for the next five years, i was a woman in a man's suit and went to a secret school with my sister, who was not allowed to go out alone. this was the only way we were educated. we had to go in different directions every day so no one would suspect where we were going. ... so for the next five years, i dressed as a boy to escort my older sister, who was no longer allowed to be outside alone, to a secret school. it was the only way we both could be educated. each day, we took a different route so that no one would suspect where we were going. ...