How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning (2023)

Abstract: Multilingual language models (MLMs) are jointly trained on data from many different languages such that representation of individual languages can benefit from other languages’ data. Impressive performance in zero-shot cross-lingual transfer shows that these models are able to exploit this property. Yet, it remains unclear to what extent, and under which conditions, languages rely on each other’s data. To answer this question, we use TracIn (Pruthi et al., 2020), a training data attribution (TDA) method, to retrieve training samples from multilingual data that are most influential for test predictions in a given language. This allows us to analyse cross-lingual sharing mechanisms of MLMs from a new perspective. While previous work studied cross-lingual sharing at the model parameter level, we present the first approach to study it at the data level. We find that MLMs rely on data from multiple languages during fine-tuning and this reliance increases as fine-tuning progresses. We further find that training samples from other languages can both reinforce and complement the knowledge acquired from data of the test language itself.

## introduction multilingual joint learning is often motivated by the idea that when multilingual language models (mlms) learn information for multiple languages simultaneously, they can detect and leverage common universal patterns across them. thus, these models can exploit data from one language to learn generalisations useful for another, obtaining impressive performance on zero-shot crosslingual transfer for many languages @xcite . various studies suggest that representations created by popular mlms, such as mbert and xlm-r @xcite , are not fully language-agnostic @xcite , but instead strike a balance between language-agnosticism and capturing the nuances of different languages through language-neutral and language-specific components @xcite @xcite . this naturally raises the question of how much models really benefit from multilingual data and cross-lingual sharing, and under what conditions this occurs. many works have studied the encoding of cross-lingual patterns within mlms by either focusing on probing for particular cross-linguistic differences @xcite , or by analyzing the distributional properties of representational language subspaces @xcite @xcite ). yet, it is not straightforward how to translate these results into model behavior at inference time. we aim to directly study how much influence languages exert cross-lingually on the predictions for individual languages. in this study, we take a step back in the training pipeline to study the extent to which the model exploits its multilingual training data when making predictions. we hypothesise that if a model performs cross-lingual information sharing, then it will base its inference-time predictions (to some extent) on training data from multiple languages. analyzing the cross-lingual sharing mechanism from the data reliance perspective leads to a set of interesting questions that we explore: to study this, we use tracin @xcite , a training data attribution (tda) method to identify a set of training samples that are most informative for a particular test prediction. the influence of a training sample z train on a test sample z test can be formalized as the change in loss that would be observed for z test if z train was omitted during training. thus, it can be used as a measure of how influential z train is when solving the task for z test . to the best of our knowledge, we present the first approach to studying cross-lingual sharing at the data level by extending the use of a tda method to the multilingual setting. we find that mlms rely on data from multiple languages to a large extent, even when the test language was seen (or over-represented) during fine-tuning. this indicates that mlm representations might be more universal than previous work suggested @xcite , in part explaining the 'surprising' effectiveness of cross-lingual transfer @xcite @xcite . moreover, we find that cross-lingual sharing increases as fine-tuning progresses, and that languages can support one another by playing both reinforcing as well as complementary roles. lastly, we find that the model exhibits different cross-lingual behaviour in the zero-shot testing setup compared to when the test language is seen during fine-tuning. ## tasks and data we conduct model fine-tuning experiments in three multilingual text classification tasks. natural language inference (nli) the cross-lingual natural language inference (xnli) dataset @xcite ) contains premisehypothesis pairs that are labeled with the relationship that holds between them: 'entailment', 'neutral' or 'contradiction'. the dataset contains parallel data in 15 languages. the original pairs come from english and were translated to the other languages. we use english, french, german, russian and spanish for model fine-tuning and testing. paraphrasing the cross-lingual paraphrase adversaries from word scrambling (paws-x) dataset @xcite and task requires the model to determine whether two sentences are paraphrases of one another. to create this dataset, a subset of the paws development and test sets @xcite was translated from english to 6 other languages by professional translators, while the training data was automatically translated. we use english, french, german, korean and spanish. ## experimental setup after fine-tuning our models (see section 4.1), we in turns, use 25 test samples from each language for testing and compute influence scores between each test sample and all 10k training samples. 3 for each test sample, we then retrieve the top k training i sentence pair test el río tabaci es una vertiente del río leurda en rumania. el río leurda es un afluente del río tabaci en rumania. 4.19 el río borcut era un afluente del río colnici en rumania. el río colnici es un afluente del río borcut en rumania. 4.15 el río colnici es un afluente del río borcut en rumania. el río borcut era un afluente del río colnici en rumania. ## quality of most influential samples we qualitatively test the plausibility of our influence scores. in ## cross-language influence we now study how much each test language relies on fine-tuning data from other languages at test time for parallel and non-parallel datasets. figure train de en es fr ru test de .431 .442 .425 .434 .418 en .633 .657 .633 .639 .610 es .563 .603 .597 .587 .542 fr .514 .540 .525 .529 .499 ru .651 .667 .652 .660 .641 (a) xnli train de en es fr ko test de .244 .256 .241 .237 .155 en .283 .308 .285 .279 .153 es .221 .236 .223 .218 .146 fr .320 .335 .325 .323 .189 ko .143 .146 .141 .140 .166 (b) paws-x ## further analysis we further analyze cross-lingual sharing for the tasks with parallel datasets since some of our analysis requires translation retrieval. ## conclusion to the best of our knowledge, we are the first to study the extent to which multilingual models rely on cross-lingual sharing at the data level. we show that languages largely influence one another crosslingually, and that this holds under various conditions. moreover, we find that cross-lingual sharing increases as fine-tuning progresses, and that languages can support one another both by playing a reinforcing as well as a complementary role. lastly, we show how tracin can be used to study data sharing in lms. we hope that this paper can inspire future work on studying the sharing mechanism within multi-task and multi-modal models as well.