m T 5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021)

Abstract: The recent “Text-to-Text Transfer Transformer” (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent “accidental translation” in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

## introduction current natural language processing (nlp) pipelines often make use of transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on a downstream task of interest @xcite . the success of this paradigm is partially thanks to the release of parameter checkpoints for pre-trained models. these checkpoints allow members of the nlp community to quickly attain strong performance on many tasks without needing to perform expensive pre-training themselves. as one example, the pre-trained checkpoints for the "text-to-text transfer transformer" (t5) model released by @xcite have been used to achieve state-of-the-art results on many benchmarks @xcite @xcite @xcite . unfortunately, many of these language models were pre-trained solely on english-language text. this significantly limits their use given that roughly 80% of the world population does not speak english @xcite . one way the community has addressed this english-centricity has been to release dozens of models, each pre-trained on a single non-english language @xcite @xcite @xcite @xcite . a more general solution is to produce multilingual models that have been pre-trained on a mixture of many languages. popular models of this type are mbert @xcite , mbart @xcite , and xlm-r @xcite , which are multilingual variants of bert @xcite , bart @xcite , and roberta @xcite , respectively. in this paper, we continue this tradition by releasing mt5, a multilingual variant of t5. our goal with mt5 is to produce a massively multilingual model that deviates as little as possible from the recipe used to create t5. as such, mt5 inherits all of the benefits of t5 (described in section 2), such as its general-purpose text-to-text format, its design based on insights from a large-scale empirical study, and its scale. to train mt5, we introduce a multilingual variant of the c4 dataset called mc4. mc4 comprises natural text in 101 languages drawn from the public common crawl web scrape. to validate the performance of mt5, we include results on several benchmark datasets, showing state-of-the-art results in many cases. finally, we characterize a problematic behavior of pre-trained generative multilingual language models in the zero-shot setting, where they erroneously translate part of their prediction into the wrong language. to address this "accidental translation", we describe a simple procedure that involves mixing in unlabeled pre-training data during fine-tuning and demonstrate that it dramatically alleviates this issue. we release our pre-trained models and code so that the community can leverage our work. 1 2 background on t5 and c4 in this section, we provide a short overview of t5 and the c4 pre-training dataset. further details are available in @xcite . t5 is a pre-trained language model whose primary distinction is its use of a unified "text-totext" format for all text-based nlp problems. this approach is natural for generative tasks (such as machine translation or abstractive summarization) where the task format requires the model to generate text conditioned on some input. it is more unusual for classification tasks, where t5 is trained to output the literal text of the label (e.g. "positive" or "negative" for sentiment analysis) instead of a class index. the primary advantage of this approach is that it allows the use of exactly the same training objective (teacher-forced maximumlikelihood) for every task, which in practice means that a single set of hyperparameters can be used for effective fine-tuning on any downstream task. similar unifying frameworks were proposed by @xcite and @xcite . given the sequence-to-sequence structure of this task format, t5 uses a basic encoder-decoder transformer architecture as originally proposed by @xcite . t5 is pre-trained on a masked language modeling "span-corruption" objective, where consecutive spans of input tokens are replaced with a mask token and the model is trained to reconstruct the masked-out tokens. an additional distinguishing factor of t5 is its scale, with pre-trained model sizes available from 60 million to 11 billion parameters. these models were pre-trained on around 1 trillion tokens of data. unlabeled data comes from the c4 dataset, which is a collection of about 750gb of english-language text sourced from the public common crawl web scrape. c4 includes heuristics to extract only natural language (as opposed to boilerplate and other gibberish) in addition to extensive deduplication. the pre-training objective, model architecture, scaling strategy, and many other design choices for t5 were chosen based on a large-scale empirical study described in detail in @xcite . ## mc4 and mt5 our goal in this paper is to create a massively multilingual model that follows t5's recipe as closely as possible. towards this end, we develop an ex-tended version of the c4 pre-training dataset that covers 101 languages and introduce changes to t5 to better suit this multilinguality. ## experiments to validate the performance of mt5, we evaluate our models on 6 tasks from the xtreme multilingual benchmark @xcite : the xnli (conneau et al., 2018) entailment task covering 14 languages; the xquad @xcite , mlqa @xcite , and tydi qa @xcite reading comprehension benchmarks with 10, 7, and 11 languages respectively; the named entity recognition (ner) dataset of wikiann @xcite restricted to the 40 languages from xtreme @xcite , and the paws-x @xcite paraphrase identification dataset with 7 languages. we cast all tasks into the text-to-text format, i.e. generating the label text (xnli and paws-x), entity tags and labels (wikiann ner), or answer (xquad, mlqa, and tydi qa) directly in a generative fashion. for ner, if there are multiple entities, they are concatenated in the order they appear, and if there are no entities then the target text is "none". we consider three variants of these tasks: (1) "zero-shot", where the model is fine-tuned only on english data, (2) "translatetrain", adding machine translations from english into each target language, and (3) "in-language multitask", training on gold data in all target languages. for brevity, we refer to @xcite for further details on these benchmarks. following the original t5 recipe, we consider five model sizes: small (≈ 300m parameters), base (580m), large (1.2b), xl (3.7b), and xxl (13b). the increase in parameter counts compared to the corresponding t5 model variants comes from the larger vocabulary used in mt5. note that, because mt5 is an encoder-decoder model, it has roughly twice as many parameters as correspondingly-sized encoder-only models such as xlm-r. for example, the "large" variant of xlm-r has 550 million parameters whereas mt5-large has around 1 billion. however, the computational cost for text classification is roughly the same: in both cases, the model processes a length-t input sequence with an encoder of approximately equal size. in an encoder-only model like xlm-r, the encoder processes one additional "cls" token, which is used to generate the representation for classification. in mt5, the decoder typically produces two additional tokens: the class label and an endof-sequence token. since the decoder has the same architecture (ignoring encoder-decoder attention) as the encoder, the computational cost of classification with mt5 typically amounts to the cost of processing t + 2 tokens compared to t + 1 for an encoder-only model. however, encoder-decoder architectures have the additional benefit of being applicable to generative tasks like abstractive summarization or dialog. we pre-train our mt5 model variants for 1 million steps on batches of 1024 length-1024 input sequences, corresponding to roughly 1 trillion input tokens total. this is the same amount of pretraining as t5 and about 1 6 as much as xlm-r. foot_5 note that our pre-training dataset is large enough that we only complete a fraction of an epoch for high-resource languages (e.g. only covering 2% of the english data). while xlm-r's pre-training corpus cc-100 is 20 times smaller than mc4, xlm-r nevertheless pre-trains for more steps, and sees over 6 times more tokens in pre-training. we use the same inverse square-root learning rate schedule used by t5 during pre-training, with the learning rate set to 1/ max(n, k) where n is the current training iteration and @xmath0. following the t5.1.1 recipe, we do not apply dropout during pre-training. we use the same self-supervised objective as t5, with 15% of tokens masked and an average noise span length of 3. we ablate some of these experimental details in section 4.2. for fine-tuning, we use a constant learning rate of 0.001 and dropout rate of 0.1 for all tasks. we use a batch size of 2 17 for most tasks, but decrease to 2 16 for wikiann ner zero-shot, due to the small size of the training, and increase to 2 20 tokens for xnli, which we found gave better performance. for early stopping, we save checkpoints every 200 steps and choose the checkpoint with the highest performance on the standard validation sets specified by xtreme. ## zero-shot generation since mt5 is a generative model, it can output arbitrary text predictions in a free form fashion. this is in contrast to "encoder-only" models like mbert and xlm(-r) that make a prediction by either extracting it from the input or producing a class label. we found that the lack of constraints during prediction caused mt5 to sometimes have trouble generating a well-formed prediction in a language unseen during fine-tuning. focusing on xquad zero-shot, we find that many of these errors are due to "accidental translation" into the fine-tuning language (english). in this section, we characterize this behavior and demonstrate that it can be counteracted by mixing a small amount of our multilingual pre-training task into the fine-tuning stage. ## conclusion in this paper, we introduced mt5 and mc4: massively multilingual variants of the t5 model and c4 dataset. we demonstrated that the t5 recipe is straightforwardly applicable to the multilingual setting, and achieved strong performance on a diverse set of benchmarks. we also characterized illegal predictions that can occur in zero-shot evaluation of multilingual pre-trained generative models, and described a simple technique to avoid this issue. we release all code and pre-trained datasets used in this paper to facilitate future work on multilingual language understanding. 11 a mc4 corpus language distribution