Examining Modularity in Multilingual LM s via Language-Specialized Subnetworks (2024)

Abstract: Recent work has proposed explicitly inducing language-wise modularity in multilingual LMs via sparse fine-tuning (SFT) on per-language subnetworks as a means of better guiding cross-lingual sharing. In this paper, we investigate (1) the degree to which language-wise modularity *naturally* arises within models with no special modularity interventions, and (2) how cross-lingual sharing and interference differ between such models and those with explicit SFT-guided subnetwork modularity. In order to do so, we use XLM-R as our multilingual LM. Moreover, to quantify language specialization and cross-lingual interaction, we use a Training Data Attribution method that estimates the degree to which a model’s predictions are influenced by in-language or cross-language training examples. Our results show that language-specialized subnetworks do naturally arise, and that SFT, rather than always increasing modularity, can decrease language specialization of subnetworks in favor of more cross-lingual sharing.

## introduction multilingual language models (lms) can achieve remarkable performance across many languages thanks to phenomena like cross-lingual sharing @xcite , but they still suffer from the "curse of multilinguality" @xcite as performance can be hindered by negative crosslanguage interference @xcite . recently, new methods have been proposed for mitigating these negative effects by training specialized model components for processing individual languages @xcite . these approaches, which add explicit modularity to the model, are also effective in promoting positive transfer and increasing interpretability @xcite . while previous work has focused on developing techniques for explicitly adding modularity to models, we take a step back and ask: to what de- figure 1 : we study how in-language training data reliance changes for individual test languages when using a subnetwork compared to the full model at test time. for instance, will a korean subnetwork rely more on korean training samples when making a prediction for a korean test sample? note that each training example is denoted by its language and a training sample id (lang_id). moreover, subnetworks have also proven to be a popular modularization technique because when used to restrict parameter updates as a form of sparse fine-tuning (sft), they are able to guide cross-lingual sharing toward positive transfer and away from negative interference @xcite @xcite @xcite . however, less is known about precisely what effects sft has on the underlying model behavior. thus, we investigate the following set of questions for xlm-r @xcite : ( 1 ) to what extent does languagewise modularity naturally arise within the model, when it is not explicitly enforced by restricting gradient updates? (2) how do cross-lingual sharing and interference differ between models without modularity interventions versus models with sftguided language-wise modularity? (3) how does the degree of language specialization affect model performance? and (4) to what extent does the similarity of language-specific subnetworks dictate cross-language influence? to quantify cross-language interaction, we follow @xcite in using a training data attribution (tda) method, tracin @xcite , which measures the degree of influence each training example has on a particular model prediction. by examining the influence each language's training set has on the test predictions for individual languages, we can estimate how much influence languages on average exert cross-lingually. we conduct experiments on three text classification tasks-natural language inference, paraphrasing, and sentiment analysis. for each task, even without special modularity interventions, we are able to identify subnetworks that rely more heavily on in-language data than the full model does. additionally, we find that sft does not always increase this modularity, but instead can decrease language specialization within the subnetworks and boost cross-lingual sharing to improve performance. finally, we provide additional analysis on factors that affect cross-language influence, and find interesting correlations between subnetwork similarity and the amount of positive influence across languages. ## naturally arising modularity in this section, we study whether modularity has naturally arisen within a model after multilingual full model fine-tuning. as such, the subnetworks are only applied at test time. ## further analysis in section 6, we show that sft only sometimes causes our identified subnetworks to rely more on in-language data, yet unlike random subnetworks, do seem to encode meaningful information. to understand where the performance improvements from sft come from, we perform further analysis on how language specialization correlates with performance, and how subnetwork similarity affects cross-language influence. ## conclusion we studied to what degree modularity, in the form of language-specialized subnetworks, naturally arises within multilingual lms. we demonstrate the existence of such subnetworks using tracin to monitor the change in reliance on in-language data at test time when using subnetworks compared to the full model. moreover, we studied the effects that sft has on modularization, and find that it does not cause all subnetworks to become more specialized. yet, in all cases, our identified subnetworks show vastly different behavior from random ones, indicating that we are able to uncover meaningful language-specific model behavior. finally, we find that subnetwork similarity, particularly in specific model layers, correlates with positive, but not negative, cross-language influence. future work should focus on further exploiting subnetworks and their interaction to better control cross-lingual sharing.