On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment (2020)

Abstract: Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages. However, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference. In this paper, we present the first systematic study of negative interference. We show that, contrary to previous belief, negative interference also impacts low-resource languages. While parameters are maximally shared to learn language-universal structures, we demonstrate that language-specific parameters do exist in multilingual models and they are a potential cause of negative interference. Motivated by these observations, we also present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference, by adding language-specific layers as meta-parameters and training them in a manner that explicitly improves shared layers’ generalization on all languages. Overall, our results show that negative interference is more common than previously known, suggesting new directions for improving multilingual representations.

## introduction advances in pretraining language models @xcite @xcite as general-purpose representations have pushed the state of the art on a variety of natural language tasks. however, not all languages enjoy large public datasets for pretraining and/or downstream tasks. multilingual language models such as mbert @xcite and xlm @xcite have been proven effective for cross-lingual transfer learning by pretraining a single shared transformer model @xcite jointly on multiple languages. the goals of multilingual modeling are not limited to improving language modeling in low-resource languages @xcite , but also include zero-shot crosslingual transfer on downstream tasks-it has been shown that multilingual models can generalize to target languages even when labeled training data is only available in the source language (typically english) on a wide range of tasks @xcite @xcite . however, multilingual models are not equally beneficial for all languages. @xcite demonstrated that including more languages in a single model can improve performance for lowresource languages but hurt performance for highresource languages. similarly, recent work @xcite @xcite in multilingual neural machine translation (nmt) also observed performance degradation on high-resource language pairs. in multi-task learning @xcite , this phenomenon is known as negative interference or negative transfer @xcite , where training multiple tasks jointly hinders the performance on individual tasks. despite these empirical observations, little prior work analyzed or showed how to mitigate negative interference in multilingual language models. particularly, it is natural to ask: (1) can negative interference occur for low-resource languages also? (2) what factors play an important role in causing it? (3) can we mitigate negative interference to improve the model's cross-lingual transferability? in this paper, we take a step towards addressing these questions. we pretrain a set of monolingual and bilingual models and evaluate them on a range of downstream tasks to analyze negative interference. we seek to individually characterize the un-derlying factors of negative interference through a set of ablation studies and glean insights on its causes. specifically, we examine if training corpus size and language similarity affect negative interference, and also measure gradient and parameter similarities between languages. our results show that negative interference can occur in both high-resource and low-resource languages. in particular, we observe that neither subsampling the training corpus nor adding typologically similar languages substantially impacts negative interference. on the other hand, we show that gradient conflicts and language-specific parameters do exist in multilingual models, suggesting that languages are fighting for model capacity, which potentially causes negative interference. we further test whether explicitly assigning language-specific modules to each language can alleviate negative interference, and find that the resulting model performs better within each individual language but worse on zero-shot cross-lingual tasks. motivated by these observations, we further propose to meta-learn these language-specific parameters to explicitly improve generalization of shared parameters on all languages. empirically, our method improves not only within-language performance on monolingual tasks but also cross-lingual transferability on zero-shot transfer benchmarks. to the best of our knowledge, this is the first work to systematically study and remedy negative interference in multilingual language models. ## motivation multilingual transfer learning aims at utilizing knowledge transfer across languages to boost performance on low-resource languages. state-of-theart multilingual language models are trained on multiple languages jointly to enable cross-lingual transfer through parameter sharing. however, languages are heterogeneous, with different vocabularies, morphosyntactic rules, and different pragmatics across cultures. it is therefore natural to ask, is knowledge transfer beneficial for all languages in a multilingual model? to analyze the effect of knowledge transfer from other languages on a specific language lg, we can compare multilingual models with the monolingual model trained on lg. for example, in figure 1 , we compare the performance on a named entity recognition (ner) task of monolingually-trained models vs. bilingual models (trained on lg and english) vs. state-ofthe-art xlm @xcite . we can see that monolingual models outperform multilingual models on four out of six languages (see §3.3 for details). this shows that language conflicts may induce negative impacts on certain languages, which we refer to as negative interference. here, we investigate the causes of negative interference ( §3.3) and methods to overcome it ( §4). ## related work unsupervised multilingual language models such as mbert @xcite and xlm @xcite ) work surprisingly well on many nlp tasks without parallel training signals @xcite . a line of follow-up work @xcite @xcite study what contributes to the cross-lingual ability of these models. they show that vocabulary overlap is not required for multilingual models, and suggest that abstractions shared across languages emerge automatically during pretraining. another line of research investigate how to further improve these shared knowledge, such as applying post-hoc alignment @xcite and utilizing better calibrated training signal @xcite . while prior work emphasize how to share to improve transferability, we study multilingual models from a different perspective of how to unshare to resolve language conflicts. our work is also related to transfer learning @xcite and multi-task learning @xcite . in particular, prior work have observed @xcite and studied @xcite negative transfer, such that transferring knowledge from source tasks can degrade the performance in the target task. others show it is important to remedy negative transfer in multi-source settings @xcite . in this work, we study negative transfer in multilingual models, where languages contain heavily unbalanced training data and exhibit complex intertask relatedness. in addition, our work is related to methods that measure similarity between cross-lingual representations. for example, existing methods utilize statistical metrics to examine cross-lingual embeddings such as singular vector canonical correlation analysis @xcite , eigenvector similarity @xcite , and centered kernel alignment @xcite . while these methods focus on testing latent representations, we directly compare similarity of neural network structures through network pruning. finally, our work is related to meta learning, which sets a meta task to learn model initialization for fast adaptation @xcite @xcite , data selection @xcite , and hyperparameters @xcite . in our case, the meta task is to mitigate negative interference. ## conclusion we present the first systematic study of negative interference in multilingual models and shed light on its causes. we further propose a method and show it can improve cross-lingual transferability by mitigating negative interference. while prior efforts focus on improving sharing and cross-lingual alignment, we provide new insights and a different perspective on unsharing and resolving language conflicts.