nm T 5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021)

Abstract: Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks. In this paper, we investigate the impact of incorporating parallel data into mT5 pre-training. We find that multi-tasking language modeling with objectives such as machine translation during pre-training is a straightforward way to improve performance on downstream multilingual and cross-lingual tasks. However, the gains start to diminish as the model capacity increases, suggesting that parallel data might not be as essential for larger models. At the same time, even at larger model sizes, we find that pre-training with parallel data still provides benefits in the limited labelled data regime

## introduction recent works have shown that cross-lingual transfer learning in pre-trained multilingual models such as mbert @xcite and xlm-r @xcite ) could be improved further by using parallel data @xcite @xcite . in this paper, we continue this line of work by improving the recent mt5 model @xcite ) by leveraging parallel corpora. we experiment with several text-to-text objectives that incorporate parallel data (spanning 198 language pairs) into mt5 pre-training. our key findings are summarized below: ## method we focus on the mt5-large model, which is a 24 layer encoder-decoder transformer model and has shown strong performance on a variety of crosslingual benchmarks @xcite . instead of training a new model from scratch, we start from the publicly available mt5-large checkpoint -which has been trained for over 1 trillion tokens -and do a second stage pre-training with a mix of monolingual and parallel data. ## results we report results in ## related work pre-trained multilingual models such as mbert and xlm-r have shown to be effective at crosslingual transfer learning @xcite . subsequently, many attempts have leveraged parallel data to improve crosslingual capability of these models. @xcite proposed translation language modeling (tlm), to encourage the model to align representations across languages. alternating language modeling @xcite and back-translation masked language modeling @xcite used code-switched sentences and back-translation respectively to utilize parallel data. other works using parallel data in this line of work include fil-ter @xcite , amber @xcite and, mmte @xcite . a key factor that differentiates this paper from these works is that our pre-trained models use a text-to-text architecture, having both an encoder and a decoder, while the aforementioned models only have the encoder. other pretrained multilingual encoderdecoder models such as mt5 @xcite , mbart @xcite and mass @xcite do not make use of parallel data during pretraining. ## conclusion in this work we attempted to improve mt5 pretraining by incorporating parallel data. we experimented with various text-to-text objectives and found that multi-tasking with the standard neural machine translation objective during pre-training 1 subject to precision of language id models used for mc4.