X - SNS : Cross-Lingual Transfer Prediction through Sub-Network Similarity (2023)

Abstract: Cross-lingual transfer (XLT) is an emergent ability of multilingual language models that preserves their performance on a task to a significant extent when evaluated in languages that were not included in the fine-tuning process. While English, due to its widespread usage, is typically regarded as the primary language for model adaption in various tasks, recent studies have revealed that the efficacy of XLT can be amplified by selecting the most appropriate source languages based on specific conditions. In this work, we propose the utilization of sub-network similarity between two languages as a proxy for predicting the compatibility of the languages in the context of XLT. Our approach is model-oriented, better reflecting the inner workings of foundation models. In addition, it requires only a moderate amount of raw text from candidate languages, distinguishing it from the majority of previous methods that rely on external resources. In experiments, we demonstrate that our method is more effective than baselines across diverse tasks. Specifically, it shows proficiency in ranking candidates for zero-shot XLT, achieving an improvement of 4.6% on average in terms of NDCG@3. We also provide extensive analyses that confirm the utility of sub-networks for XLT prediction.

## introduction one of the aspired objectives in the field of natural language processing (nlp) is to ensure equal treatment of text regardless of the language it originates from @xcite . this goal is highly desirable in that it promotes the democratization of nlp technologies, making them accessible to everyone worldwide. with the emergence of multilingual language models @xcite ; @xcite ; @xcite , inter alia) that can process over a hundred languages, the ambitious figure 1 : this figure shows that the performance of zeroshot xlt for named entity recognition (ner) varies greatly depending on the language used for task finetuning (i.e., the source language). although english is mostly a default choice for xlt, relying on it often lags behind the best results. in contrast, our method suggests source languages that are nearly optimal. the dotted lines represent average scores for each approach. fortunately, the aforementioned problem can be partially mitigated by employing the cross-lingual transfer (xlt) method @xcite ) that exploits supervision from another language dubbed the source language. for instance, a multilingual model tuned on english reviews can be directly utilized for classifying the sentiment of reviews in german with decent performance. as seen in the example, english is commonly selected as the source language due to its abundant and diverse data resources available for various tasks. however, it is not intuitive that english would always be the best option, particularly considering its heterogeneity with non-latin languages. in fact, recent studies @xcite @xcite have substantiated the conjecture, verifying that certain east asian languages such as chinese, japanese, and korean exhibit mutual benefits in cross-lingual transfer, surpassing english. in our preliminary study, we also discovered that the performance of zero-shot xlt for ner considerably depends on the choice of the source language (see figure 1 ). these findings have naturally led to research into identifying optimal source languages for different configurations of cross-lingual transfer, as well as investigating the factors that contribute to their effectiveness. popular strategies for uncovering the optimal source language in cross-lingual transfer include those based on linguistically-inspired features @xcite , ones that count on statistical characteristics of a corpus of each candidate language @xcite , and methods that aggregate various clues for conducting regression @xcite . however, the existing techniques have several limitations: (1) they normally require a combination of a myriad of features, each of which is costly to obtain, (2) their effectiveness has only been validated for a limited number of tasks, (3) they present a general recommendation rather than one tailored to individual settings, or (4) they do not reflect the inner workings of the utilized language model. to address these challenges, this work presents a novel and efficient method called x-sns, which aims to predict the most appropriate source language for a given configuration in cross-lingual transfer. we introduce the concept of sub-network similarity as a single key feature, which is computed as the ratio of overlap between two specific regions within a model that are activated during the processing of the source and target language. the main intuition behind the proposed method lies in the notion that the degree of resemblance in the structural changes of model activations, induced by each language, determines the efficacy of xlt. our approach is model-based, data-efficient, and versatile. it is closely tied to the actual workings of a base model as it derives the similarity score from the model's internal values, i.e., gradients, instead of relying on the information pre-defined by external resources. in addition, in comparison to previous data-centric methods, our technique is efficient in terms of the quantity of data required. moreover, we show in experiments that our method is widely applicable across many tasks. it exhibits superiority over competitive baselines in the majority of considered settings. when evaluated in terms of ndcg@3, it achieves an improvement of 4.6% on average compared to its competitors. the outline of this paper is as follows. we first present related work ( §2) and explain the details of our approach ( §3). after presenting experimental settings ( §4), we perform comprehensive comparisons against baselines ( §5) and conduct extensive analyses to provide insights into using sub-network similarity for estimating the performance of xlt ( §6). we finally conclude this paper in §7. ## proposed method: x-sns we present the technical details of our approach, dubbed x-sns (cross(x)-lingual transfer prediction through sub-network similarity). its primary objective is to suggest proper source language for xlt, eliminating the need for developing separate copies of language models fine-tuned by task data in each language. to this end, we propose utilizing the similarity between a pair of language-specific sub-networks as an indicator to estimate the compatibility of two languages in xlt. the overall procedure of the method is illustrated in figure 2 . the core mechanism of x-sns lies in the derivation of sub-networks customized for individual languages and the computation of similarity between a pair of sub-networks. as a crucial component in the construction of our targeted sub-network, we introduce the fisher information @xcite , which provides a means of quantifying the amount of information contained in parameters within a neural network @xcite . concretely, we derive the (empirical) fisher information of a language model's parameters as follows. 1 first, given a data point (x, y) from the true data distribution-input x and output y-we define the fisher information matrix f for parameters θ as: second, since it is practically intractable to derive the exact f , we assume this is a diagonal matrix, aligning with the approaches employed in the related literature @xcite . 2 as a result, we obtain the fisher information vector @xmath0. finally, we approximate f with f , which is the average of monte carlo estimates of f based on a given corpus @xmath1, y j )} |d| @xmath2. formally, the (approximated) fisher information of the i th parameter in θ is formulated as: we utilize the final vector f , which has the same dimensionality as θ, as a source of knowledge to assess the importance of the corresponding model parameters. each element of this vector can also be interpreted as the square of the gradient with respect to a model parameter, positioning our method within the realm of model-oriented approaches. as the next step, we transform f into a binary vector that represents the structure of the subnetwork we intend to extract. specifically, each component of the sub-network vector s given f , say s (i) , is specified as 1 if f (i) (θ) is in the top p% of all element values in f , 0 otherwise. in other words, we identify a group of parameters within the network whose importance scores are positioned in the top p th percentile. note that the proposed method only considers the relative significance of each parameter in terms of its fisher information, ignoring its actual values. by doing so, x-sns not only avoids the need for extra training of base models but also ensures its efficiency. this also sets it apart from previous work that employs a similar framework for model pruning or task fine-tuning. there remain two unspecified factors that play a vital role in acquiring language sub-networks using x-sns. the first is the choice of the corpus d used for obtaining f . we utilize a corpus from a language of our interest as the basis for deriving a sub-network specific to that language. besides, we test two types of corpora for each language: a task-oriented corpus (t ) and a general domain corpus (w), i.e., wikipedia. as shown in §5, a taskoriented corpus guarantees superior performance, while a general one enables a broader application of the approach with satisfactory results. in experiments, we also confirm that our method is dataefficient, being nearly optimal when |d| ≤ 1000. the second factor is the output distribution of the model p(y|x; θ). in previous studies @xcite , p(y|x; θ) is usually defined as a task-specific layer (t ), e.g., a classifier, implemented on the base model, requiring additional training of the top module. however, as we prioritize the efficiency of the algorithm, the extra fine-tuning is not considered in our case, even in the environment where a labeled, task-oriented corpus is available. instead, we imitate such an avenue with a random classifier combined with the crossentropy loss for the target task. meanwhile, another realistic direction of defining p(y|x; θ) is to rely on the pre-training objective of the base model, i.e., (masked) language modeling (l; @xcite ). this not only eliminates the need for task supervision but also allows one to recycle the language modeling head built during the pre-training phase. we thus regard l as the primary option. foot_2 refer to §5.1 where we conduct a related analysis. lastly, given a pair of sub-networks s s and s t for the source (s) and target (t) languages respectively, the similarity of the languages is calculated as the jaccard similarity coefficient between the two vectors: |s s ∩ s t |/|s s ∪ s t |. in practice, a set of resource-rich languages, as well as the target language for which we aim to perform xlt, are provided. we then proceed by computing the similarity of the target language and each candidate language in the set, and sort the candidates based on the similarity scores in descending order. ## analysis we present extensive analyses helpful for a deeper understanding of x-sns. we explore some factors that can have an impact on its performance, such as the sub-network ratio p ( §6.1), the base language model ( §6.2), and the size of the corpus d (apfor token-level tasks (ner, pos), a moderate level of p is optimal, whereas the lowest value for p is the best for other semantic tasks. ## conclusion we present x-sns, a method that efficiently predicts suitable source languages for xlt with minimal requirements. we identify that xlt performance between two languages has a strong correlation with the similarity of sub-networks for those languages, and propose a way of exploiting this fact in forecasting the success of xlt. x-sns demonstrates exceptional versatility and robustness, verifying impressive performance across various environments and tasks. we believe that our suggestions and findings have the potential to contribute to the exploration of the inner workings of multilingual language models, leaving it as future work.