Untangling the Influence of Typology, Data, and Model Architecture on Ranking Transfer Languages for Cross-Lingual POS Tagging (2025)

Abstract: Cross-lingual transfer learning is an invaluable tool for overcoming data scarcity, yet selecting a suitable transfer language remains a challenge. The precise roles of linguistic typology, training data, and model architecture in transfer language choice are not fully understood. We take a holistic approach, examining how both dataset-specific and fine-grained typological features influence transfer language selection for part-of-speech tagging, considering two different sources for morphosyntactic features. While previous work examines these dynamics in the context of bilingual biLSTMS, we extend our analysis to a more modern transfer learning pipeline: zero-shot prediction with pretrained multilingual models. We train a series of transfer language ranking systems and examine how different feature inputs influence ranker performance across architectures. Word overlap, type-token ratio, and genealogical distance emerge as top features across all architectures. Our findings reveal that a combination of typological and dataset-dependent features leads to the best rankings, and that good performance can be obtained with either feature group on its own.

## introduction despite being trained on 100+ languages, pretrained multilingual language models (mlms) fail to cover the vast majority of the world's languages. finetuning mlms for zero-shot cross-lingual transfer is a useful technique to extend their reach by circumventing the lack of task-specific labeled data in low-resource languages. effective zero-shot transfer hinges on choosing an appropriate source language @xcite @xcite , but it is still not well understood how to make this selection. most analyses of successful source/target pairs fall into one of two categories: typological or dataset-dependent. the typological view investigates the role of linguistic similarity, with studies showing that more "similar" languages tend to form better source/target pairs @xcite @xcite . much of this typological analysis is coarse-grained, focusing on features like language family or abstract distance measures. the dataset-dependent view focuses on comparing source and target datasets based on features like sub-word overlap @xcite @xcite . few papers consider both views, and those that do focus on older methods of crosslingual transfer like bilingual lstms @xcite . additionally, previous analyses shed little light on the linguistic question of which fine-grained typological features are especially relevant for the task. this primary goal of this paper is to offer a deeper understanding of effective transfer language selection across architectures, comparing crosslingual transfer with bilstms to xlm-r @xcite and m-bert @xcite . we aim to identify which features contribute to selecting a successful source/target pair for part-ofspeech (pos) tagging. we focus on pos tagging because it directly reflects typological features such as word order. our analysis addresses the following key questions: q1. which features are most important for crosslingual transfer? q2. do these features differ between bilstms and mlms? q3. how does the granularity of typological features-whether fine or coarse-affect transfer language selection? q4. is it necessary to consider data set features in selecting a transfer language? we train a series of gradient-boosted decision tree models to rank transfer languages for pos tagging, with separate rankers for the two architectures. during training, we generate feature importance scores and identify the most salient features for each architecture (q1, q2). to examine the role of fine-grained typological features, we compare two typological inputs: source/target distance measures, and full finegrained feature vectors (q3). we also evaluate how the source and quality of typological data affects ranker performance by swapping between uriel @xcite and grambank @xcite feature vectors. last, we investigate whether typological information alone can effectively determine suitable source/target language pairs by experimenting with the exclusion of dataset-specific features (q4). we find that impressive performance can be achieved when relying primarily on either feature category, without the need for the other, indicating that both "typological" and "dataset-dependent" views of transfer language choice represent independently viable strategies. however, peak performance is achieved by combining dataset-dependent and fine-grained typological features. crucially, our analysis reveals that key features such as word overlap, type-token ratio, and genealogical distance remain consistently important across architectures, suggesting that the relevance of these features may transcend specific model designs, offering broader insights into cross-lingual transfer that could enable us to better leverage mlms for low-resource applications. ## conclusion we find that features such as word overlap, typetoken ratio, and genealogical distance are consistently influential in transfer language selection regardless of model architecture; their importance may be somewhat model-agnostic. our findings also highlight the crucial role of dataset-dependent features in ranking transfer languages for cross-lingual transfer. rankers trained with these features outperform those relying solely on coarse-grained typological features. at the same time, while coarse-grained typological features alone are insufficient, rankers trained with fine-grained typological features achieve impressive results even without dataset-dependent features. the most successful ranking performance comes from combining both dataset-dependent and fine-grained typological features, underscoring the value of a comprehensive approach to transfer language selection. crucially, these insights enable us to better support languages that are not well-represented in mlm pretraining. by identifying effective transfer languages with interpretable features, we can improve cross-lingual transfer for lower-resource languages, expanding the reach of nlp beyond those languages that benefit from large-scale pretraining.