Multilingual Lottery Tickets to Pretrain Language Models (2023)

Abstract: The curse of multilinguality in training multilingual pretrained language models (mPLMs) refers to the negative interference between languages, especially when the capacity is limited. While increasing the capacity may appear intuitive for overcoming this curse, it negatively affects both training and inference costs. Our distinction is pursuing the competing goals of reducing negative interference, while keeping capacity per each language more or less the same. Specifically, we first scale the model to reduce interference, then search for a per-language subnetwork, or a lottery ticket, with comparable performance to the full model. According to lottery ticket hypothesis, this scale-then-find-ticket approach alleviates interfering signals as in the scaled model, but redistributes parameters to keep the parameters reduced. Finally, to avoid the cost of multiple retraining for searching multilingual tickets, we explore zero-shot neural architecture search (NAS) methods. We investigate the most appropriate zero-shot NAS method to find multilingual tickets. Our proposed multilingual tickets reduce the inference cost of models for each languages, while boosting the performances. The ticket search cost is negligible and tickets found qualitatively preserve linguistic similarity. Our code is publicly available.

## introduction multilingual pretrained language models (mplms) @xcite have become the de-facto standard for multilingual tasks. these models, by mapping multiple languages through shared parameters (figure 1a ), may benefit from a positive transfer between languages @xcite . however, when more languages are covered, or parameters are limited, negative interference be- * corresponding author tween languages @xcite has been observed, known as curse of multilinguality @xcite . for the problem of signals from languages interfering with each other, a naïve solution would be to increase the capacity per each language @xcite . for example, figure 1b illustrates adding additional parameters per each language to the shared model. this design intuitively improves the average performance of each language and allows the gradient conflict to be alleviated, but with the cost of enlarging perlanguage parameter size, which affects both the training and inference costs of the model. our distinction is keeping the per-language capacity similar to reach the same goal. to achieve multiple competing goals -improving the performance and mitigating the gradient conflict, without increasing per-language capacity-we invite the lottery ticket hypothesis @xcite . lottery ticket hypothesis claims that dense models contain subnetworks -called "winning tickets"whose performance is at least similar to the full model. this hypothesis is empirically shown to be true for popular language model architectures @xcite @xcite . our key idea is to search tickets per each language, i.e., multilingual tickets, that can achieve all of the competing goals, by scaling the model and then searching for such tickets. the parameters of each language will be redistributed in the scaled model to keep the per-language capacity unaffected by scaling. to illustrate, in figure 1c , each of l 1 and l 2 maintains the same capacity before (figure 1a ) and after scaling (figure 1c ) as 9 parameters. this will i) improve the performance, since the performance from tickets will be similar to the scaled model, by the lottery ticket hypothesis. moreover, disseminating the parameters will also ii) mitigate the negative interference. however, finding multilingual tickets in a scaled model incurs a prohibitive cost -even to find a single ticket, @xcite train model multiple times. to overcome, we interpret this problem as neural architecture search (nas) @xcite per each language. particularly, we explore recently emerging zeroshot nas @xcite @xcite , which aims to search architecture without training cost. we discriminate the most appropriate zero-shot nas method to remove the burden of finding multilingual tickets. finally, to verify our claims, we devise a metric to measure the negative interference during multilingual pretraining. our measurements indicate that multilingual tickets decrease negative interference. our experiments show that multilingual tickets we found increase task performance while maintaining the capacity as expected. in our qualitative analysis, the locality of subnetworks preserves the linguistic similarity within the same family as well. our contributions can be summarized as follows: ## proposed method: multilingual lottery tickets with zero-shot nas this section presents our method, scaling the model and then finding tickets m i per each language l i ( §3.1), leveraging zero-shot nas techniques ( §3.2). ## experiments in this section, we tackle the following research questions: ## conclusion this paper studied the challenge by balancing the conflicting goals of reducing negative interference while maintaining a similar per-language capacity. we proposed a scale-then-search approach, of searching for per-language subnetworks, or lottery tickets, from a scaled model, that improves performance without increasing per-language capacity. we keep the cost of finding such tickets negligible, by exploring a zero-shot nas method. our results show that ours reduces negative interference as expected, and the tickets discovered qualitatively preserve linguistic relatedness.