m T 6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021)

Abstract: Multilingual T5 pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on seven multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.

## introduction multilingual pretrained language models, such as mbert @xcite , have attracted increasing attention. they not only improve the performance on downstream multilingual nlp tasks @xcite @xcite , but also show an impressive cross-lingual transferability @xcite @xcite . multilingual pretrained models are typically trained on multilingual unlabeled text with unsupervised language modeling tasks, e.g., masked language modeling @xcite , causal language modeling @xcite , and span corruption @xcite . these unsupervised tasks are built upon large-scale monolingual texts. in addition, several studies propose cross-lingual tasks that utilize translation data from multilingual parallel corpora, such as translation language modeling (conneau and lample, * contribution during internship at microsoft research. 2019), cross-lingual contrast @xcite , and bidirectional word alignment @xcite . thanks to the translation data, the pretrained models produce better-aligned cross-lingual representations and obtain better cross-lingual transferability. recently, the multilingual text-to-text transfer transformer (mt5; @xcite ) achieves stateof-the-art performance on several cross-lingual understanding benchmarks. mt5 inherits the benefits of t5 @xcite ) that treats every text processing problem as a text-to-text problem, i.e., the problem of generating some target text conditioned on the input text. despite the effectiveness of mt5, how to improve mt5 with translation data is still an open problem. in this paper, we present mt6, standing for improving multilingual text-to-text transfer transformer with translation data. mt6 differs from mt5 in terms of both pre-training tasks and the training objective. we present three cross-lingual tasks for text-to-text transformer pre-training, i.e., machine translation, translation pair span corruption, and translation span corruption. in the translation span corruption task, the model is trained to predict the text spans based on the input translation pair. the cross-lingual tasks encourage the model to align representations of different languages. we also propose a new objective for text-to-text pre-training, called partially non-autoregressive (pnat) decoding. the pnat objective divides the target sequence into several groups, and constrains that the predictions should be only conditioned on the source tokens and the target tokens from the same group. we conduct experiments on both multilingual understanding and generation tasks. our mt6 model yields substantially better performance than mt5 on eight benchmarks. we also provide an empirical comparison of the cross-lingual pre-training tasks, where we evaluate several variants of mt6 under the same pre-training and fine-tuning procedure. moreover, our analysis indicates that the representations produced by mt6 are more cross-lingual transferable and better-aligned than mt5. the contributions are summarized as follows: ## background on t5 and mt5 multilingual text-to-text transfer transformer (mt5; @xcite ) is the multilingual variant of t5 @xcite pretrained on the mc4 @xcite dataset, which consists of natural text in 101 languages drawn from the public common crawl web scrape. the backbone architecture of mt5 is the simple encoder-decoder transformer @xcite , which is trained in a unified text-to-text manner. in specific, text-based nlp problems are formulated as text-to-text transfer, i.e., the model is trained to predict the target text conditioned on the input source text. for example, in text classification, the model predicts the label text rather than a class index. this feature enables the mt5 to be fine-tuned with the same training objective for every task. formally, let x and y denote the input sequence and the output sequence, the loss function of training the @xmath0 with the unified textto-text formulation, the pre-training task can be designed by constructing the input and output text sequences. specifically, mt5 employs the span corruption task as the pre-training task, which is an unsupervised masked language modeling task. as shown in figure 1 , we provide an example of constructing the input and output sequences for span corruption. given a natural sentence s, it first randomly selects several spans of s as the spans to be masked. then, the input sequence is constructed by replacing the selected spans with unique mask ## conclusion in this paper, we propose mt6 that improves the multilingual text-to-text transfer transformer with translation data. we introduce three text-to-text pre-training tasks that are built on parallel corpora, and a training objective for improving text-to-text pre-training. nonetheless, we present a comprehensive comparison of the text-to-text tasks, and show that our mt6 model outperforms mt5 on both cross-lingual understanding and generation benchmarks. for future work, we would like to pretrain mt6 models at a larger scale, and explore more applications, such as machine translation.