Discovering Language-neutral Sub-networks in Multilingual Language Models (2022)

Abstract: Multilingual pre-trained language models transfer remarkably well on cross-lingual downstream tasks. However, the extent to which they learn language-neutral representations (i.e., shared representations that encode similar phenomena across languages), and the effect of such representations on cross-lingual transfer performance, remain open questions.In this work, we conceptualize language neutrality of multilingual models as a function of the overlap between language-encoding sub-networks of these models. We employ the lottery ticket hypothesis to discover sub-networks that are individually optimized for various languages and tasks. Our evaluation across three distinct tasks and eleven typologically-diverse languages demonstrates that sub-networks for different languages are topologically similar (i.e., language-neutral), making them effective initializations for cross-lingual transfer with limited performance degradation.

## introduction multilingual language models (multilms) such as mbert @xcite , xlm @xcite , and xlm-r @xcite are pre-trained jointly on raw data from multiple languages. later, when they are fine-tuned for a task using perhaps a single high-resource language dataset, they demonstrate promising zeroshot cross-lingual performance, generalizing to the same task in different languages despite not having been fine-tuned for those languages. the facilitator of this cross-lingual transfer ability is often hypothesized to be a learned language neutrality in these models (i.e., similar linguistic phenomena across languages are represented similarly by the model). however, the source of this ability remains an open question. certain studies 1 our code is available at https://github.com/negarforoutan/multilms-lang-neutral-subnets . we discover subnetworks in the original multilingual language model that are good foundations for learning various tasks and languages (a). then, we investigate to what extent these sub-networks are similar by transferring them across other task-language pairs (b). in this example, the blue and red lines show sub-networks found for french and urdu, respectively, and purple connections are shared in both sub-networks. dashed lines show the weights that are removed in the pruning phase. in this paper, language-neutrality is instead conceptualized in terms of the parameters of multi-lms. we hypothesize that a multilingual model with language-neutral representations would have learned different languages using the same subset of parameters in its network structure. subnetworks in multilms that overlap across languages, and transfer well when re-trained on other languages from the pre-training corpus, would indicate that the model is comprised of languageneutral representations that jointly encode multiple languages. to demonstrate this effect, we extract sub-networks from multilms by pruning them for individual languages and task pairs using iterative magnitude pruning @xcite (figure 1a ). then, we evaluate the crosslingual transfer of sub-networks by re-initializing to the unpruned parameters of the original mul-tilm, and re-training on task data for different languages (figure 1b ). our results on three tasks, namely masked language modeling, named entity recognition @xcite , and natural language inference @xcite , show high absolute parameter overlap among sub-networks discovered for different languages and effective cross-lingual transfer between languages on the same task (even outperforming the original multilingual model for certain low-resource languages). however, cross-lingual sub-network transfer deteriorates as we increase the sparsity level of the sub-networks, suggesting that language-neutral components of the multilms are pruned in favor of retaining necessary languageand task-specific components for the language-task pair used to discover the sub-network. ## background & motivation multilingual language models. multilingual language models (multilms) such as mbert @xcite , xlm @xcite , and xlm-r @xcite have achieved state of the art results in cross-lingual understanding tasks by jointly pre-training transformer models @xcite on many languages. specifically, mbert has shown effective cross-lingual transfer for many tasks, including named entity recognition @xcite , cross-lingual natural language inference @xcite , and question answering @xcite . due to these impressive cross-lingual transfer results, many recent works investigate the source of this capacity. one line of study investigates the effect of different pre-training settings (e.g., shared vocabulary, shared parameters, joint multilingual pre-training, etc.) on cross-lingual transfer @xcite @xcite . other works explore how learned multilingual representations are partitioned into language-specific subspaces @xcite @xcite . in contrast to these works, our study explores the existence of language-neutral parameters in multilms. using iterative magnitude pruning, we extract pruned sub-networks from multilingual lms for various tasks and languages, and investigate to what extent these sub-networks are similar by transferring them across languages. ## methodology in this section, we define sub-networks (i.e., winning tickets in the lth) and identify our approach to discovering them through a pruning algorithm derived from the lottery ticket hypothesis. sub-networks. for any network f (x; θ) with initial parameters θ, we define a sub-network for f as f (x; m ⊙ θ) where @xmath0, 1} |θ| is a binary mask on its parameters and ⊙ is the element-wise product. the mask m sets many of the parameters of the original network to zero, which prunes edges in the computation graph and yields a sub-network of the original network. winning ticket. when training network f (x; θ) on a task, f reaches maximum evaluation performance a at iteration i. similarly, when training subnetwork f (x; m ⊙ θ) on the same task, f reaches maximum evaluation performance a ′ at iteration i ′ . a sub-network f (x; m ⊙ θ) is a winning ticket f * (x; m ⊙ θ) if it achieves similar or better performance than the original network: aa ′ ≤ ϵ when i ′ ≤ i (we set ϵ to be one standard deviation of performance of the original mbert model). ## experimental results in this section, we report our evaluations to study the presence of language-neutral sub-networks in multilingual language models. first, we validate the lottery ticket hypothesis for the mbert model for various languages and tasks. then, we compare the winning tickets for different languages by analyzing their parameter overlap and cross-lingual performance. finally, to further assess the similarity of the winning tickets in a zero-shot setting, we evaluate their performance in a sentence retrieval task. existence of winning tickets. to discover winning tickets for each task-language pair, we run algorithm 1 to identify sub-networks at various sparsity levels. then, we train the pruned sub-networks on the same task and language to identify whether it qualifies as a winning ticket, f * (x, m ⊙ θ). in figure 2 , we report the performance of pruned sub-networks for each task-language pair at different sparsity levels between 0 and 90%. winning tickets are found for each language and task at multiple sparsity levels, though as these sub-networks get sparser, they no longer satisfy the winning ticket criterion (performance must be within 1 standard deviation of the full mbert performance). however, they still often perform within 10% of their winning ticket performance. interestingly, we find that the performance drop at higher sparsity levels is greater for the mlm and ner tasks than for xnli, highlighting the importance of evaluating on tasks that induce diverse behavior in sparse models. ## conclusion while multilingual pre-trained language models have shown impressive performance across languages, the role of language neutrality in achieving such a performance is not well understood. in this work, we analyze the language-neutrality of multilingual models by investigating the overlap between language-encoding sub-networks of these models. using mbert as a foundation, we expose the extent of its language neutrality by employing the lottery ticket hypothesis and comparing the subnetworks within mbert obtained for various languages and tasks. we show that such sub-networks achieve high performance after being transferred across tasks and languages and are similar to one another. however, at higher levels of sparsity, this transferability evaporates. our results suggest that multilingual language models include two separate language-neutral and language-specific components, with the former playing a more prominent role in cross-lingual transfer performance.