Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter (2025)

Abstract: Cross-lingual transfer is a popular approach to increase the amount of training data for NLP tasks in a low-resource context. However, the best strategy to decide which cross-lingual data to include is unclear. Prior research often focuses on a small set of languages from a few language families and/or a single task. It is still an open question how these findings extend to a wider variety of languages and tasks. In this work, we analyze cross-lingual transfer for 263 languages from a wide variety of language families. Moreover, we include three popular NLP tasks: POS tagging, dependency parsing, and topic classification. Our findings indicate that the effect of linguistic similarity on transfer performance depends on a range of factors: the NLP task, the (mono- or multilingual) input representations, and the definition of linguistic similarity.

## introduction for many of the world's languages, the available data to train natural language processing (nlp) models is scarce. if data is available, it is often only enough for an evaluation set, raising the question of how to select the training set. two approaches are intuitive: (i) based on linguistic similarity measures: find the training data in a language that is linguistically as close as possible to the target language, where "linguistically close" is defined by one or multiple linguistic similarity measures, or (ii) based on dataset dependent measures: find a dataset in another language that is similar to the target dataset, e.g., in terms of high word or n-gram overlap. naturally, these two types of approaches are not mutually exclusive. however, it is unclear which of these measures are most important to select the source language *work done while interning at apple. for cross-lingual transfer, despite previous work focusing on this question (see §2): earlier studies often contradict each other, and therefore leave a number of important avenues for improvement. for example, prior work often lacks a large representation of languages and language families, as well as nlp tasks, and sometimes relies on synthetically constructed datasets. furthermore, popular out-of-the-box measures for linguistic similarity are sometimes intransparent or faulty @xcite @xcite , and correlations between different similarity measures are often not taken into account. summarizing, how findings from prior work generalize across a larger variety of languages and tasks, and how we should interpret different similarity scores, are open questions. in this work, we contribute to these questions by analyzing transfer between 263 different languages from 33 language families (figure 1 ) in three different tasks: (i) part-of-speech (pos) tagging, (ii) dependency parsing, and (iii) topic classification. in choosing our tasks, we are motivated by the following considerations: (i) availability of datasets: we select tasks for which we can find data for a multitude of languages from different language families, and (ii) types of tasks: we include two word-level grammatical tasks (i.e., pos tagging and dependency parsing) and one sentence-level topic iden-tification task. to allow for a clean analysis, we opt for a zero-shot approach, in line with prior work @xcite -we train our models on the task in one particular source language, and evaluate on the target language without additional training or fine-tuning in the target language. our key findings can be summarized as: 1. for different tasks and input representations, different similarity measures are most predictive for cross-lingual transfer performance. for instance, syntactic similarity is most predictive for pos tagging and parsing, whereas trigram overlap is the most important predictor for ngram-based topic classification ( §5.1); 2. practical implication within studied tasks: choosing a source language based on a pertinent similarity measure leads to adequate transfer results ( §5.2.1); 3. practical implication across studied tasks: when no information about pertinent similarity measures is available for a given task, choosing a source language based on findings for a conceptually similar experiment is a relatively safe choice ( §5.2.2). ## related work based on this insight, philippy et al. make recommendations for follow-up work: (i) focus on real natural languages (instead of synthetic ones), (ii) examine the interaction between different contributing factors, (iii) focus on many languages, (iv) focus on linguistic features when selecting training languages, and (v) focus on generative tasks, given the success of generative models. we focus on the first four recommendations in this work. we include three classification tasks (pos tagging, dependency parsing, and topic classification), motivated by the availability of train and test data in a large number of languages for these tasks. ## methodology we run transfer experiments on two word-level grammatical tasks with word-level annotations ( §3.1) and a sentence-level topic classification task ( §3.2). of the 263 languages in our experiments, 55 training and 84 test languages are shared between all tasks. appendix §a contains details on all languages and datasets in this study. section §3.3 introduces our similarity measures. ## supporting results here we present the results on which the main analyses in §5 build. in §4.1, we present the pos tagging, parsing and topic classification scores within and across languages. in §4.2, we compare how (dis)similar transfer patterns are across tasks and 3 because different trees/branches have different depths (owing to different family sizes and/or different levels of documentation), comparisons of scores can only serve as a proxy for the degree of phylogenetic relatedness. this is also noted in the metadata for lang2vec, which calculates genetic distances in a similar way: http://www.cs.cmu. edu/~dmortens/projects/7_project/ . for constructed languages, we ignore the top-most level (e.g., "artificial languages") and instead retrieve phylogenetic information starting at the second level. ## main results and analysis here, we investigate which factors correlate with overall transfer performance ( §5.1). then, we ex-las in the remainder of the paper as the more extensive of the two: las not only measures whether the dependency arcs are correct, but also whether they were labelled correctly. ## conclusion we investigated how linguistic and dataset level similarities impact cross-lingual transfer, and find that the most predictive similarity measures differ across tasks and input representations. for practitioners working on the tasks included in our study, we recommend choosing a source language based on a pertinent similarity measure, or, if results for an extensive transfer experiment involving a similar task and similar input representations are available, based on the transfer results in that experiment. future work should investigate to what extent these patterns hold across even more languages, tasks, and nlp models.