mm T 5: Modular Multilingual Pre-Training Solves Source Language Hallucinations (2023)

Abstract: Multilingual sequence-to-sequence models perform poorly with increased language coverage and fail to consistently generate text in the correct target language in few-shot settings. To address these challenges, we propose mmT5, a modular multilingual sequence-to-sequence model. mmT5 utilizes language-specific modules during pre-training, which disentangle language-specific information from language-agnostic information. We identify representation drift during fine-tuning as a key limitation of modular generative models and develop strategies that enable effective zero-shot transfer. Our model outperforms mT5 at the same parameter sizes by a large margin on representative natural language understanding and generation tasks in 40+ languages. Compared to mT5, mmT5 raises the rate of generating text in the correct language under zero-shot settings from 7% to 99%, thereby greatly alleviating the source language hallucination problem.

## introduction multilingual pre-trained models @xcite have demonstrated impressive performance on natural language understanding (nlu) tasks across different languages @xcite . these models are typically trained on large amounts of unlabeled data in hundreds of languages. recent large language models @xcite display surprising multilingual capabilities despite being pre-trained predominantly on english data. however, all of these models share a key limitation: representations of all languages compete for the model's limited capacity. as a result, models perform poorly with an increasing number of pre-training languages and on languages with less pre-training data. this is also known as the "curse of multilinguality" @xcite . feed forward add & norm multi-head attention add & norm add & norm lannguage 1 ... ff down ff up lannguage n ff down ff up feed forward add & norm multi-head attention add & norm add & norm lannguage 1 ... natural language generation (nlg) tasks present another challenge for current multilingual models, which may overfit to the training languages and partially forget their generation ability in the target language @xcite , generating text with the correct meaning in the wrong language. we refer to this as the "source language hallucination problem". to address these two limitations, we propose the modular multilingual t5 (mmt5, figure 1 ), the first modular multilingual generative model. during pre-training, mmt5 allocates a small amount of language-specific parameters to increase capacity for multilingual modeling. at fine-tuning time, we freeze the language-specific modules while tuning the shared parameters, allowing direct adaptation to a target language by swapping to the corresponding language-specific module. however, we observe an additional challenge for mmt5: the fine-tuned shared representations may drift away from the frozen modular representations in the decoder. the modular model is thus susceptible to generating text in the incorrect language, similar to its non-modular counterparts. to ameliorate this, we propose to freeze a subset of shared decoder parameters, which shows large improvements in zero-shot cross-lingual generation for modular generative models. in general, we find that mmt5 is an effective model that overcomes the two limitations of multilingual sequence-to-sequence models: 1) mmt5 alleviates the curse of multilinguality by adding additional model capacity to different languages during pre-training. it outperforms both standard baselines as well as mt5 @xcite at the same parameter sizes on a representative set of multilingual nlu and nlg tasks; 2) mmt5 resolves the source language hallucination problem with impressive ability on zero-shot cross-lingual text generation. our analysis ( §6.4) shows that mt5 only generates text in the target language 7% of the time for a zero-shot multilingual summarization task, while mmt5 generates text in the correct language for 99% of examples. our approach, in contrast, uses modularity a priori, i.e., modularity is integrated into the module architecture as an inductive bias. such modularity is similar to parameter sharing strategies commonly defined in multi-task learning @xcite as well as to mixture-of-experts approaches @xcite , which have been used to scale models to trillion parameters @xcite and for domain-specific pre-training of lms @xcite . the most related work to ours is x-mod @xcite , which pre-trains an encoder-only bert-style model in a modular fashion. their model, however, cannot be used for natural language generation and underperforms our model on nlu tasks (see section 4). ## mmt5 standard multilingual models update the same model parameters for hundreds of languages during pre-training, resulting in the curse of multilinguality where different languages compete for the limited model capacity @xcite . we propose mmt5, the first modular sequenceto-sequence multilingual model that allocates language specific modules during pre-training. in this section, we discuss the architecture of mmt5, its training and fine-tuning methods, and our strategies to resolve the source language hallucination problem with mmt5. ## conclusion we have proposed mmt5, a modular multilingual encoder-decoder model. during multilingual pretraining the majority of parameters of mmt5 are shared between tasks, but each language is provided with a small amount of parameters only accessible to the respective language. we demonstrated that integrating modularity as an architectural inductive bias significantly improves training efficiency, where the same perplexity as an equivalent fully dense model is achieved at a quarter of the update steps. mmt5 considerably outperforms comparable models on a large number of tasks including question answering, semantic parsing, summarization and classification in both zero-shot as well as multilingual scenarios. finally, we show that by freezing parts of the decoder when fine-tuning mmt5 on a target task in a source language, the model consistently generates text in the target language. consequently, modularity arguably solves source language hallucinations in cross-lingual transfer scenarios.