G rad S im: Gradient-Based Language Grouping for Effective Multilingual Training (2023)

Abstract: Most languages of the world pose low-resource challenges to natural language processing models. With multilingual training, knowledge can be shared among languages. However, not all languages positively influence each other and it is an open research question how to select the most suitable set of languages for multilingual training and avoid negative interference among languages whose characteristics or data distributions are not compatible. In this paper, we propose GradSim, a language grouping method based on gradient similarity. Our experiments on three diverse multilingual benchmark datasets show that it leads to the largest performance gains compared to other similarity measures and it is better correlated with cross-lingual model performance. As a result, we set the new state of the art on AfriSenti, a benchmark dataset for sentiment analysis on low-resource African languages. In our extensive analysis, we further reveal that besides linguistic features, the topics of the datasets play an important role for language grouping and that lower layers of transformer models encode language-specific features while higher layers capture task-specific information.

## introduction most natural language processing (nlp) research today still focuses on a small number of languages. extending nlp models to further languages poses different challenges, i.a., little (annotated) data @xcite . multilingual training can help in those cases by sharing knowledge across languages. however, adding new languages to the multilingual training set may not necessarily lead to performance gains. in fact, certain languages might actually hurt the performance on downstream tasks in a specific target language @xcite , for instance, due to unrelatedness to the target language. figure 1: exemplary transfer learning setup with african languages: the motivation for this work is that neither language family (indicated by node colors) nor typological distance (indicated by distance of languages in the plot) are consistent predictors of good performance when choosing a source language for crosslingual transfer. red cross: source language affecting performance negatively. green tick: source language affecting performance positively. as a solution, previous work investigates different measures for language similarity and selects only languages similar to the target language for the multilingual training set (i.a. @xcite @xcite @xcite . however, it is an open research question whether language similarity translates into performance gains of multilingual models. for multilingual training, other characteristics might play a role, such as topical shifts of the training data. as a result, it is still unclear how to select the set of languages that leads to the most effective multilingual training setup. in this paper, we study multilingual fine-tuning of language models with a diverse set of training languages. 1 in particular, we show that linguisticsbased language similarities are only weakly correlated with cross-lingual transfer performance. figure 1 illustrates a sample case in which neither language family information (indicated by node colors) nor similarity of language embeddings (indicated by proximity in the vector space) is helpful for finding languages that have a positive crosslingual transfer score with the target language. thus, prior information about the languages, such as their language families or typological features, alone is not enough for an effective multilingual training. instead, similarity measures that capture additional information about the data and task beyond linguistics similarity may achieve better performance. @xcite , for instance, show that gradient similarity across languages measured along the optimization trajectory correlates with language proximity and cross-lingual performance. we draw inspiration from this observation. however, instead of projecting conflicting gradients throughout the training process, we propose to leverage the gradient similarity to group languages with a branch-and-bound-like algorithm that optimizes the overall similarity score of all languages. this approach has the following advantages: (i) it can be applied without any prior knowledge of the languages or topics of the given datasets, (ii) it is well correlated with downstream task performance of the multilingual model, (iii) it finds the best language groups from a global perspective, i.e., instead of selecting source languages independently of each other (which may create groups of mutually interfering languages), we form each group based on a criterion that evaluates the group as a whole. in our experiments, we show the superior performance of our grouping method compared to various baseline approaches on three multilingual datasets with different tasks and set the new state of the art on afrisenti, a sentiment analysis dataset in 12 low-resource african languages. furthermore, we extensively analyze our models with a topic analysis, a correlation-based analysis and an ablation study, revealing important insights, for instance that the topic distribution of the training data heavily affects multilingual training and that lower layers of transformer models encode language-specific features while higher layers capture task-specific information. this confirms results from prior work (i.a., @xcite @xcite from another (correlation-based) perspective. the code base for gradsim is available online. foot_1 2 related work multilingual and multi-task training. a growing number of research projects investigates multilingual training to cover a variety of languages, including low-resource languages @xcite @xcite . in the context of low-resource sentiment analysis, @xcite recently use the cross-lingual transfer score between pairs of languages to select source languages for multilingual training. our approach differs from these works in that we investigate language interactions from a global optimization perspective. considering each language as a separate task, multilingual training can be treated as a multi-task learning (mtl) problem @xcite . a line of existing work utilizes gradient-based techniques to improve multi-task learning @xcite @xcite . they show that negative cosine similarity between gradients leads to negative interference for mtl optimization, and projecting out the conflicting gradients can improve the optimization dynamics. our work follows this insightful observation. however, in contrast to their work, we propose to leverage multilingual gradients for language grouping to ensure that gradients are aligned in each language group. language similarity measures. in order to group languages for multilingual training or transfer learning, related work has proposed different ways to estimate the similarity between languages, e.g., leveraging the language family taxonomy @xcite @xcite or representing languages as information-rich vectors based on their typological or conceptual features @xcite @xcite . another line of works measures language similarity based on embeddings from multilingual pretrained language models (mplms) @xcite @xcite . @xcite and shaffer (2021), for instance, perform language grouping for multilingual named entity recognition and neural machine translation based on embeddings. in contrast to these studies, we propose to use the gradient cosine similarity between languages as the similarity measure for language grouping. this model-based similarity measure reflects how each language interacts in the optimization process, with no need of any prior knowledge of the languages. ## method in this section, we describe our proposed language grouping approach and the general multilingual training in which we apply its results. note that the gradient-based similarity estimation is purely model-based, thus, can be applied to other settings, e.g., multi-domain or multi-task problems, as well. ## experiments in this section, we describe our experimental settings as well as our results for three tasks. ## analysis to analyze the behavior of the model, we perform the following analyses on the afrisenti dataset: a qualitative analysis of the data in order to better un- ## discussion in this section, we summarize our main findings. language similarity is not enough to determine transfer suitability. when sharing knowledge across languages, the information about linguisticsbased language similarity (e.g., whether the languages come from the same language family or how languages are typologically similar to each other) is not enough for optimal performance. this observation is in line with the findings by @xcite , @xcite and @xcite that languages from the same family may still exhibit distinct linguistic features and, thus, languagefamily based grouping can enhance model performance only to a certain extent. in addition, we find that there are other aspects that will affect multilingual model performance and, therefore, need to be taken into account, such as the topical distribution of the data. gradient-based method does not require any prior knowledge. our proposed gradient-based approach for grouping languages is a pure modelbased approach, thus, does not require any prior knowledge about the language, task or data. as a result, it can be successfully applied, even when the data distribution (e.g., topical distribution) is unknown (e.g., because we are dealing with foreign languages). while our current work only presents results for language grouping for multilingual models, the method itself is more general and can be applied to other settings as well, such as multi-task learning or multi-domain setups. lower layers capture language, upper layers task information. adding to previous work on analyzing transformer-based pretrained language models @xcite @xcite , our correlation analysis shows that gradient similarity between languages from lower layers are more correlated to language-specific distances, i.e., low layers seem to encode language-specific information, while gradient similarity from upper layers are more correlated to task-specific performance, i.e., upper layers tend to capture task-specific information. ## conclusion in this paper, we addressed the challenging problem of grouping languages for effective multilingual training. we proposed a gradient-based grouping approach and showed in our experiments that it is better correlated to cross-lingual transfer performance than language family or language embedding-based grouping. in our analysis, we identified topical distribution differences as one potential challenge that can be addressed effectively by our approach. furthermore, our correlation analysis confirmed results from prior work that lower layers of transformer-based pretrained models seem to encode language-specific features, while upper layers capture task-specific information. our method shows superior performance compared to a variety of baseline methods for language grouping on three diverse datasets and, in particular, sets the new state of the art on a multilingual sentiment analysis benchmark dataset consisting of low-resource african languages.