On Language Spaces, Scales and Cross-Lingual Transfer of UD Parsers (2022)

Abstract: Cross-lingual transfer of parsing models has been shown to work well for several closely-related languages, but predicting the success in other cases remains hard. Our study is a comprehensive analysis of the impact of linguistic distance on the transfer of UD parsers. As an alternative to syntactic typological distances extracted from URIEL, we propose three text-based feature spaces and show that they can be more precise predictors, especially on a more local scale, when only shorter distances are taken into account. Our analyses also reveal that the good coverage in typological databases is not among the factors that explain good transfer.

## introduction the goal of cross-lingual parsing is to process a target language as well as possible by exploiting training data available from (an)other language(s). while we know that parsing models can be transferred well across some well-known closely related languages @xcite , the success of cross-lingual transfer in all other cases remains hard to predict. surprising cases of syntactic transfer between unrelated languages such as irish and indonesian @xcite illustrate well this unpredictability. a possible explanation for such cases is that genealogically unrelated languages can still be similar enough to allow transfer. but what is the relevant measure of language similarity in such cases? one possible solution is to rely on language features stored in typological databases such as wals @xcite or glottolog @xcite . taking these features as vector representations, languages can be embedded and compared regardless of their genealogical relations. a popular library uriel @xcite has facilitated the use of typological features to measure similarity between languages at different levels (phonology, syntax, geographical distribution). the problem with this solution is that the information in linguistic databases is often incomplete and unevenly distributed. some languages are fully described, while only a few feature values are known for others @xcite . nevertheless, a study by @xcite on transferring models from english to several other languages suggests that the uriel language similarity score is a good predictor of cross-lingual transfer for parsing universal dependencies (ud). our study brings a comprehensive analysis of the relationship between language similarity and the cross-lingual transfer in ud parsing. it extends previous work in two directions: first, we cover many more languages than any previous study (which are typically limited to a small set); second, we compare the uriel representation with three textbased alternatives. these extensions allow us to ask new questions such as: what should we do for languages that do not have close relatives? do measures of language similarity predict the transfer at any scale (for close and for distant languages)? are there good alternatives to linguistic databases for measuring language similarity? we perform correlation tests between linguistic distances and parsing scores on various samples of ud treebanks designed to neutralize two kinds of biases. first, we balance the samples at the level of language, genus, and family, 2 reducing gradually the known bias of the ud towards indo-european languages. second, we investigate the impact of the scale by comparing global correlations (considering a whole language space) with local correlations (considering smaller partitions of a language space). we show that typological distances extracted from uriel are reasonably good global predictors, while text-based distances are better local predictors. a surprising outcome of our analysis concerns the uneven coverage of languages in typological databases: most of the ud languages with many missing features are indo-european. on the other hand, good database coverage does not guarantee good predictability of transfer for the languages outside of the indo-european family. ## related work thanks to evident structural alignments between languages the possibility of transferring syntactic parsing models across languages was investigated even before the wide-spread adoption of pre-trained language models in nlp @xcite . however, this task proved non-trivial because such clear alignments tend to be found in similar languages, but are much rarer overall @xcite . the idea of using data from another language or a set of languages to improve syntactic parsing on any given language is tempting because annotated data is not available for the majority of the world's languages. early work typically focused on several languages selected according to the availability of training data. in the meantime, the universal dependencies (ud) treebanks have become available for many different languages @xcite foot_2 opening the question of what language pairs are most suitable for model transfer. most of the time, polyglot foot_3 models are trained on multiple languages, but preserving the identity of the languages (by adding the language id to the text representation) turns out useful @xcite . @xcite cluster languages according to similarity before training polyglot models. cross-lingual parameter sharing is found to improve the performance overall, but especially for closely-related languages, which can share parameters in different layers of neural representation @xcite . cross-lingual transfer started being explored in other tasks too after the introduction of large pretrained models @xcite , making the question of linguistic similarity relevant to a more general scope of nlp research. @xcite propose a range of measures that can be used in order to choose the best transfer language, which they divide into data-dependent (data size, token overlap, ttr) and data independent (various distance measures extracted from the uriel database). @xcite study how well different similarity scores predict the success of the transfer on different tasks (with mbert and xlm-r as pretrained models) and find that syntactic features extracted from uriel correlate strongly with the zero-shot cross-lingual ud parsing performance. interestingly, these features are better predictors than genealogical relatedness, but data-dependent measures, such as the size of the training data, seem to predict better the cross-lingual zero-shot performance on other tasks such as xquad @xcite or xnli @xcite @xcite . while english turns out to be a good transfer language for many tasks due to the size of the training data, @xcite show that german is a better transfer language than english for quite a few, even less-related, languages. the fact that english is not the best transfer language on the task of part-of-speech (pos) tagging is confirmed by the most wide-scope study of cross-lingual transfer up to now @xcite . similarly to @xcite , this study too finds that a surface string similarity measure (ldnd distance, @xcite ) is a better predictor of the transfer than genealogical relatedness. somewhat contrary to this, @xcite find an interesting genealogical clustering in the representations created by machine translation models. having counted mentions of successful crosslingual transfer on many different tasks in the previous works @xcite @xcite @xcite @xcite , we notice that english is most frequently mentioned as the best transfer language overall, but these mentions are almost entirely related to european target languages. for targets located outside of europe, the best transfer languages are different and hard to predict. for instance, greek is a good transfer language for thai and hindi, while russian works well for these two languages and arabic. our study shares the wide cross-lingual scope with de @xcite . in contrast to their work, we focus on syntactic parsing models, rather than pos tagging. we follow some other previous studies in working with both typological and text-based language similarity measures, but our text-based measures can be regarded as generic rather than data-dependent and can be used as an alternative to uriel in many cases. the most widely accepted method for comparing languages relies on genealogical classification: we consider languages located in the same region of a phylogenetic tree to be similar. this method currently prevails in nlp. practitioners often discuss language similarity in terms of language family @xcite @xcite . however, language families can be too broad for a meaningful comparison as they include typologically very different languages. for instance, english and armenian belong to the same family (indo-european), but are very different in terms of phoneme inventories, morphology, and word order. on the other hand, languages can be rather similar even if they are genealogically unrelated. for example, bulgarian is closely related to other slavic languages, but its morphology, word order and the use of the definite article makes it more similar to english than to other slavic languages. typological features and geographical placement of languages can be regarded as potentially more objective and fine-grained alternatives to genealogical similarity. in other words, genealogically unrelated languages can turn out to be close in a typological vector space or in the geographical (physical) space. it is less common to have an intuitive perception of languages that are close in such spaces as similar, but typological proximity seems to be more useful as a predictor of cross-lingual transfer than genealogical relatedness (see section 2). the uriel database and its associated python library lang2vec @xcite are very convenient resources for measuring the distance between languages in all of these spaces. uriel combines features from several linguistic databases: ethnologue @xcite , glottolog, phoible @xcite , sswl 5 and 5 syntactic structures of the world's languages by chris wals. it describes over 4,000 languages, but the available information strongly depends on the types of features. for example, geographic and genealogical feature values are known for all languages, while syntactic feature values, which are relevant to our study, are often missing. when assessing linguistic similarity with lang2vec, one can use various subsets of features and the knn prediction option to fill in the missing features, which is what is typically used in previous research. with this option, all feature slots are filled with some predicted value. if a value is missing for some feature, the corresponding value from the most similar language (nearest neighbor) is returned. we work with the union of syntactic features (wals + sswl) completed with the knn prediction, but we also analyze the coverage of the ud languages in the uriel sources by extracting the values before the knn prediction. text-based features can be regarded as a potential alternative to the features extracted from typological databases. type-token ratio (ttr), for instance, is higher in morphologically rich than in morphologically poor languages and can be used for language comparison when the data size is controlled @xcite @xcite . other text statistics, such as the mean word length (mwl) are also characteristic of languages (words are longer in morphologically rich languages), while being even less dependent on the data. in the work on cross-lingual transfer, it is common to consider all text-based measures to be data-dependent as opposed to typological measures, which are data-independent @xcite . 6 we assume that text-based features can reach various levels of data-independence, while providing a means for measuring language similarity at a more fine-grained level. in the remainder of this section, we describe two text-based measures that we propose for comparing languages at two structural levels, morphology and syntax. our morphological measure is more generic than the syntactic measure, which is more data-dependent. ## data and methods from the linguistic spaces and measures described in section 3, we create distance matrices. we then calculate multiple correlation scores between each of the linguistic distance matrices on one side and the scores obtained while testing parsers on a set of languages on the other. for each pair transfer-target language, we have one labeled attachment score (las), which we name xlas in our experiments to underline the fact that these scores are obtained via cross-lingual transfer. 7 we expect higher xlas scores when linguistic distances are smaller, thus a negative correlation. in this section, we describe the details of the experimental design and the analyses. ## conclusion in this paper, we have shown that various linguistic features can be good predictors of cross-linguistic transfer of ud parsing models. as an alternative to the typological syntactic features extracted from the uriel database, we propose several text-based features and show that they are often better predictors. those that encode syntactic structural information by design (dependency probes) are the strongest predictors, while those that capture morphology (bpe, mwl) are comparable to syntactic features extracted from uriel, especially on a more local scale. in addition to the distance scales, all the scores are impacted by the genealogical composition of the language samples. explanations for these findings remain an open question for future work.