Evaluating morphological typology in zero-shot cross-lingual transfer (2021)

Abstract: Cross-lingual transfer has improved greatly through multi-lingual language model pretraining, reducing the need for parallel data and increasing absolute performance. However, this progress has also brought to light the differences in performance across languages. Specifically, certain language families and typologies seem to consistently perform worse in these models. In this paper, we address what effects morphological typology has on zero-shot cross-lingual transfer for two tasks: Part-of-speech tagging and sentiment analysis. We perform experiments on 19 languages from four language typologies (fusional, isolating, agglutinative, and introflexive) and find that transfer to another morphological type generally implies a higher loss than transfer to another language with the same morphological typology. Furthermore, POS tagging is more sensitive to morphological typology than sentiment analysis and, on this task, models perform much better on fusional languages than on the other typologies.

## introduction cross-lingual transfer uses available annotated resources in a source language to learn a model that will transfer to a target language. earlier work used machine translation @xcite , parallel data @xcite , or delexicalized models @xcite @xcite to bridge the gap between languages. however, recent improvements @xcite have reduced the need for parallel data, instead relying on multi-lingual language models, trained on the concatenation of monolingual corpora. fine-tuning these multilingual language models on a task in a source language can lead to strong performance when applied directly to the targetlanguage task (zero-shot transfer). this progress has uncovered gaps in performance, as transfer is generally easier between similar languages, and some language families consistently perform worse @xcite . so far, however, the analysis of these differences has only been anecdotal, rather than centered as a research question of its own merit. for these cases, linguistic typology has important implications, as it gives us ways to quantify the similarity of languages along certain variables, such as shared morphological or syntactic features @xcite . while previous work has studied the effects of morphological typology on language modeling @xcite @xcite , this effect on cross-lingual transfer has not been looked at in detail. in this paper we attempt to answer (rq1) to what degree morphological typology affects the performance of state-of-the-art cross-lingual models, (rq2) whether morphological typology has a stronger effect than other variables, e.g., the amount of data for pretraining the lm or domain mismatches between source and target, (rq3) whether there is a different effect on a low-level structural task (pos tagging) vs. a semantic task (sentiment analysis). to answer these questions we experiment with two state-of-the-art cross-lingual models: multilingual bert and xlm roberta. we fine-tune the models for part-of-speech tagging and sentiment analysis on 19 languages from four morphologically diverse typologies. our results show that pos tagging is more sensitive to morphological typology than sentiment analysis and that the models perform much better on fusional languages, such as german, than on the other typologies. we release the code and data foot_0 in order to reproduce the experiments and facilitate future work in this area. although these approaches have led to large improvements on many cross-lingual tasks, it is clear that the success of zero-shot cross-lingual transfer depends on the typological similarity of the source and target language @xcite . @xcite find pos performance correlates with word order features taken from the world atlas of language structures (wals) database @xcite . similarly, morphologically complex languages tend to achieve poorer performance @xcite . similar to this work, @xcite perform zero-shot and few-shot transfer on 20 languages and 5 tasks. however, the choice of languages does not allow one to answer what is the effect of morphological typology. the effect of morphological typology on nlp tasks is well known @xcite , with several dedicated workshop series @xcite . more recently, attention has turned to larger scale analyses of morphological typology effects on language modeling @xcite @xcite . in contrast to these previous works, we are interested in how morphological typology affects crosslingual transfer for two supervised tasks, namely part of speech (pos) tagging and sentiment analysis. we choose these two tasks as 1) they both have data available in typologically diverse languages, and 2) represent a lower-level structural and higher-level semantic task, respectively. our experimental setup reduces some of the complexity of comparing test results across languages, as we compare relative differences, instead of absolute differences. at the same time, it is necessary to take into account several other variables, i.e., presence of the language in pretraining, the amount of training data, the effect of byte-pair tokenization, the length of train and test examples, and any domain mismatches across languages. although it is a simplification of the variation in morphological features @xcite , languages have traditionally been grouped into four morphological categories, i.e., isolating, fusional, introflexive, and agglutinative. 2 these categories describe a language's tendency to group concepts together into a single word or disperse them into separate words. pure isolating languages have maximally one morpheme per word. in agglutinative languages, morphemes tend to be neatly segmentable and carry a single feature, whereas in fusional languages, a single morpheme often carries multiple grammatic, syntactic, and semantic features. finally, in introflexive languages root words are based on consonant stems, where vowels introduced around and between them lead to syntactic and semantic changes (see @xcite ; @xcite ; @xcite for a more in-depth discussion). ## data we select five languages from each category except introflexive (four), shown in ## methods we fine-tune both multilingual bert (mbert) @xcite and xlm roberta (xlm-r) @xcite models on the available training data in each language, using a shared set of hyperparameters selected from recommended values according to the characteristics of our data. we set the learning rate to 2e-5, maximum sequence length of 256, batch size of 8 or 16 foot_6 , and perform early stopping once the validation score has not improved in the last epochs, saving the model that performs best on the dev set. we then test each model on all languages, giving us a matrix of test scores, where the diagonal is in-language, and all others are cross-lingual. we use accuracy as our metric for pos and macro f 1 for sentiment, as the latter often contains unbalanced classes, and define language data origin ○ german wojatzki et al. (2017) ○ spanish agerri et al. (2013) ○ slovak pecar et al. (2019) ○ norwegian øvrelid et al. (2020) ○ greek 3 kalamatianos et al. (2015) tsakalidis et al. (2018) mandarin github repository 4 vietnamese cuong et al. (2016) thai bact' et al. (2019) cantonese xiang (2019) indonesian purwarianti and crisdayanti (2019) finnish lindén et al. (2020) basque barnes et al. (2018a) korean github repository 5 japanese github repository 6 turkish 7 pontiki et al. (2016) arabic abdulla et al. (2013) nabil et al. (2015) hebrew amram et al. (2018) algerian touileb and barnes (2021) maltese dingli and sant (2016) cortis and davis (2019) ## results once our scores matrix is built, we average foot_7 the score of each fine-tuned model, which we refer to as language-to-language cross-lingual scores, over the other languages in each morphological group, thus obtaining each model's average cross-lingual performance per target group (language-to-group cross-lingual scores). next, we average again for each source language group. this yields the average cross-lingual performance values per training and testing language groups (group-to-group crosslingual scores), which we report in in the part-of-speech task, the best group-togroup cross-lingual performance always corresponds to models fine-tuned in a language of the same morphological group, regardless of the model's architecture. fusional models, in particular, obtain a remarkably higher score when tested on other fusional languages (over 80%). on the other hand, the group-to-group cross-lingual scores where the target language is introflexive are considerably lower than the rest (always below 50%). in contrast, both model architectures show different patterns in the sentiment analysis task. for the xlm-r models, the best group-to-group crosslingual scores are all achieved by those trained on a fusional language, while for the mbert it is mainly models trained on an isolating language that achieve the best scores. in any case, all scores are within a similar range of values. in fact, the main difference in this task seems to be due to xlm-r's considerably higher scores. in order to capture the cross-lingual phenomenon more accurately, we introduce transfer loss, a relative metric defined in equation 1: next, to address rq1 more directly, we compare two different types of transfer: intra-group transfer, where both the fine-tuning and target languages belong to the same morphological group, and inter-group transfer, where the two differ in morphological type. we calculate an average for both types of transfer and for each training group, model architecture and task. we present the resulting values in figure 1 . generally, transfer to another morphological type implies a higher cost in terms of performance, except for the introflexive models. this difference in transfer loss appears to be similar for all groups in the sentiment task, yet it varies considerably in the part-of-speech task. more specifically, there are two extremes in this latter case: fusional models suffer large performance drops when switching morphological groups, whereas isolating models experience similar transfer losses in both conditions. finally, we average again to obtain a single trans-fer loss value for each task and model, and use it to establish a comparison in figure 2 . here we observe that: (1) the difference in transfer loss between an intra-group and inter-group transfer is higher on the part-of-speech task, (2) transfer is also generally worse on this task foot_9 , (3) xlm-r models perform better cross-lingual transfers in general (especially on the sentiment analysis task), and (4) the difference between intra-group and inter-group transfer is similar on both model architectures. ## analysis in this section, we run several statistical tests to verify our conclusion to rq1 and detail several points of analysis that relate to rq2 and rq3. namely, to what degree do other variables contribute to effects on cross-lingual transfer. ## discussion and future work in this paper, we have conducted an extensive analysis of the effects of morphological typology on cross-lingual transfer and attempted to isolate these factors from other variables. we have compared performance of two state-of-the-art zero-shot cross-lingual models on two tasks (part-of-speech tagging and sentiment analysis) for 19 languages across four morphological typologies. we have found that transfer to another morphological type generally implies a higher performance loss than transfer to another language with the same morphological typology. additionally, part-of-speech tagging is more sensitive to morphological differences than sentiment analysis, while sentiment analysis is more sensitive to variables related to the fine-tuning data and is less predictable in general. we have tested this sensitivity to morphology after balancing other influential factors, such as in-language score, and, still, the intra-/inter-group difference remains. however, the effect of morphological typology, while significant, is not strong, given that most of the variability in transfer loss is due to other factors. we have also confirmed that xlm-r generally transfers better than mbert, especially on sentiment analysis. in part-of-speech tagging, we have reported considerably better transfer within fusional languages, as well as easier transfer from the other groups towards the fusional type. moreover, we have found a case that suggests that finetuning on large training sets might lead to language specialization and, consequently, be detrimental to cross-lingual transfer. it is worth noting that we do not explore whether the type of script used by the languages has an effect on cross-lingual transfer. this is hard to control in our experimental setup, as there are some scripts that are either unique to a language or only have one with enough data to represent it, making it impossible to make comparisons. the recent cross-lingual suite xtreme @xcite includes a number of benchmark tasks in 40 languages. while this dataset is a useful collection of cross-lingual tasks, it is unfortunately not sufficient for our purposes. the pos data is the same as we use, while other tasks either a) do not contain a representative sample of language typologies b) use translation, introducing problems of 'translationese', or c) are automatically created and not manually curated named entity recognition data. our experimental setup avoids these problems by focusing on binary sentiment analysis, which is a task that has data available in many languages and does not require translation to get multilingual data. finally, this work ties in with the increasing interest in typological questions in nlp @xcite @xcite @xcite , which often try to directly predict typological features, or use these to analyze model performance. in the future, it would be interesting to train multi-lingual language models on specific language families in order to find maximal benefits from shared morphology. finally, as typology seems to affect tasks differently, it would be interesting to explore other tasks, e.g., dependency parsing or semantic role labeling.