Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages (2022)

Abstract: Cross-lingual transfer learning with large multilingual pre-trained models can be an effective approach for low-resource languages with no labeled training data. Existing evaluations of zero-shot cross-lingual generalisability of large pre-trained models use datasets with English training data, and test data in a selection of target languages. We explore a more extensive transfer learning setup with 65 different source languages and 105 target languages for part-of-speech tagging. Through our analysis, we show that pre-training of both source and target language, as well as matching language families, writing systems, word order systems, and lexical-phonetic distance significantly impact cross-lingual performance. The findings described in this paper can be used as indicators of which factors are important for effective zero-shot cross-lingual transfer to zero- and low-resource languages.

## introduction at present, for a large majority of natural language processing tasks, the most successful approach is fine-tuning pre-trained models with task-specific labelled data. unfortunately, for many languages, and especially low-resource languages, such taskspecific labelled data is often not available. a potential solution is cross-lingual fine-tuning of multilingual pre-trained language models @xcite , using available data from some source language to model the phenomenon in a different target language for which labelled data does not exist. cross-lingual generalisability of large pretrained language models is often evaluated by finetuning multilingual models on english data and testing them on unseen languages @xcite @xcite . of course, this approach is influenced by the availability of english training data for given tasks, but also then comes with the implicit assumption that english is a representative source language. this, however, may not be true in practice. specifically, depending on the task, aspects of similarity between source and target language may be relevant for cross-lingual transfer performance @xcite . if similarity between source and target language impacts performance, crosslingual transfer should not be assessed using only a single predetermined source language, especially if training sets in multiple languages are available. furthermore, target test languages are generally selected based on data availability for the evaluated tasks, but availability may not result in a representative subset of the world's languages. the xtreme benchmark collection @xcite , for example, attempts to alleviate this problem by including a varied selection of languages from different language families. this collection contains token classification, text classification, question answering and retrieval tasks in 40 languages. the language selection does, however, obfuscate the fact that for most non-indo-european languages no data is available for semantically rich tasks such as question answering. this imbalance regarding tasks in this type of collections may consequently inflate the perceived performance for these languages. in this work, we aim to shed light on what factors make a language a good source and/or target language for cross-lingual transfer when fine-tuning a large multilingual model. we evaluate this via partof-speech (pos) tagging data, as this is the only task for which high-quality data is available in a large number of languages, including low-resource languages from different language families. also, high cross-lingual pos tagging performance may be seen as a precondition for more semantically complex tasks, as a base understanding of syntactic structure in both the source and target language is necessary for any meaningful natural language processing task. contributions this paper is a case-study of cross-lingual transfer learning with part-of-speech tagging. we explore the limits and contributing factors to successful cross-lingual transfer and partof-speech tagging in particular. among others, we evaluate the effects of (matching) language families, (matching) writing systems, and pre-training on cross-lingual training. moreover, we provide insights that can help to estimate performance when one tries to transfer to a low-resource language with little or no annotated data. source code is released on github, foot_0 and 65 fine-tuned models are shared via the hugging face hub. foot_1 ## approach we fine-tune a pre-trained model for the task of part-of-speech tagging using different languages in training and testing. every combination of source and target language yields an accuracy score, with a large matrix of accuracies as a result. monolingual, or within-language performance is the accuracy where the source and target language are the same. overall cross-lingual source or target accuracies can be calculated per column or row in the accuracy matrix, excluding the monolingual accuracy. such accuracies give an overall indication of (i) how suitable a given language is as source for cross-lingual pos tagging, and (ii) how easy or difficult it is to pos-tag a given target language when monolingual training data is not available. predictors through a mixed-effects regression analysis, with source and target language (family) as random-effect factors, we assess which variables determine a "good" source language. the variables we consider are whether or not the language family is shared between source and target language, the writing systems (and writing system types) of both languages and whether or not these match, the subject-object-verb (sov) word order of both languages and whether or not these match, and whether or not a (source or target) language was included in pre-training. additionally, we add the (lexical-phonetic) ldnd measure @xcite on the basis of the 40-item word lists from the asjp database @xcite as a quantitative similarity measure comparing source and target language. finally, we also consider the size of the training set of the source language as a predictor. we analyse results both from a quantitative and a qualitative viewpoint. task data we use the pos tag data from universal dependencies 2.8 @xcite . it contains manually annotated data for 114 languages; among these all have test data and 75 languages have training data. we exclude three mixed-code languages, one sign language, three languages with fewer than 10 test samples and two languages that do not have any word-level annotations. moreover, we exclude training data for five languages that have fewer than 25 training samples. all other training datasets consist of at least 125 samples. as a result, we have 105 languages which can serve as target languages, of which 65 can also serve as source languages since they have training data. ## results figure 2 illustrates the test accuracies for every combination of source and target language. the heat map shows that the model achieves relatively high performance for cases where the source and target language is the same (outlined in black). while for many languages same-language training is the only way to achieve high performance (for example maltese), there are also many target languages for which high performance is observed when training on several other languages (for example russian). indeed, within-language performance tends to be high with a mean accuracy of 94.1% (σ = 4.5). however, there is a substantial amount of variation for cross-lingual accuracies with an overall mean of 57.4% (σ = 22.4). this shows that cross-lingual training does not universally yield good performance. we evaluate several predictors for inclusion (see section 2) by adding them to a linear mixed-effects model with random intercepts for source language, target language, and target language family. no other random intercepts were found to improve the model (via model comparison). we ascertained that the predictors of the final model remained significant when the corresponding random slopes were included. these are not further reported, however. fixed-effect predictors were included if they significantly (@xmath0.05) improved the model fit as determined via (maximum likelihood) model comparison. ## qualitative discussion having discussed significant predictors in detail, we now take a closer look at "bad" source languages, thereby providing a better understanding of how to choose a "good" source language (section 5.1). we also identify some optimal sourcetarget language pairs (section 5.2), and "optimal" source languages for our task (section 5.3). ## conclusion we show that simply fine-tuning a large multilingual pre-trained language model on english data does not necessarily make full use of the crosslingual potential of the model. especially when one applies cross-lingual training for a low-resource language with little or no evaluation data, the different factors that influence performance should be kept in mind. unfortunately, one of the most important factors highlighted by our experiments is that the target language, or a highly similar language variant, should be included in pre-training for crosslingual training to be successful. for current language models, this excludes many languages and a large number of language families. for those languages, the most important step is to collect unlabeled data for pre-training (although the amount of data required may be relatively modest; @xcite . languages that are included in pre-training can achieve high cross-lingual performance across language families and writing systems, at least for languages that use alphabetic writing systems. the english language, which is the de facto default source language for cross-lingual training, is not necessarily the best source language. due to data availability, our experiments focused on pos tagging, but we hypothesise that the factors we identified may be predictive for other tasks too. the significant influence of lexical-phonetic distances and word order differences on accuracies indicate that similar languages are encoded similarly in xlm-roberta, even if there is no lexical overlap due to differing writing systems. thus, these factors potentially also influence more syntaxdependent tasks, such as parsing, and semantically rich tasks, such as natural-language-inference.