Using Cross-Lingual Part of Speech Tagging for Partially Reconstructing the Classic Language Family Tree Model (2022)

Abstract: The tree model is well known for expressing the historic evolution of languages. This model has been considered as a method of describing genetic relationships between languages. Nevertheless, some researchers question the model’s ability to predict the proximity between two languages, since it represents genetic relatedness rather than linguistic resemblance. Defining other language proximity models has been an active research area for many years. In this paper we explore a part-of-speech model for defining proximity between languages using a multilingual language model that was fine-tuned on the task of cross-lingual part-of-speech tagging. We train the model on one language and evaluate it on another; the measured performance is then used to define the proximity between the two languages. By further developing the model, we show that it can reconstruct some parts of the tree model.

## introduction language families are defined by the evolution of languages over the history, providing indications regarding the proximity between them. the tree model, which was first introduced by augustus schleicher (schleicher, 1853) is considered as the consensual language-family model. for example, figure 1 shows the indo-european branch of the tree model; a full version of the model is nicely presented on ethnologue 1 . in this paper, we refer to this source as a reference for the classic family tree model. concomitantly, there have been theories that question the tree model as being an indicator for language proximity, since it represents genetic relatedness rather than lexical resemblance. loanwords, as well as other lexical influences are usually not expressed in the classic tree model. representing historical relatedness, the tree is agnostic to various linguistic influences. consequently, some claim that language families should be defined by alternative models @xcite . the universal grammar, introduced by noam chomsky, is usually defined as the "system of categories, mechanisms and constraints shared by all human languages and considered to be innate" @xcite . in other words, a human language is derived from a set of structural rules, typically referred to as generative grammar, which we are usually totally unaware of. we can intuitively distinguish between nouns and verbs; children can phrase a sentence they have not heard before by ordering parts of speech they are familiar with in a valid grammatical order. a child can identify a noun without knowing what a noun is, or without even understanding the meaning of that specific noun. it may be assumed that rather than this aspect of universal grammar being specific to language, it is more generally a part of human cognition, and there might be a common structure for different languages. still, the ability to classify words into parts of speech requires some knowledge of the structure of the specific language. the hypothesis we examine in this paper relies on the assumption that historically close languages, like french and spanish, share some information that may help the classification of words into partof-speech (pos) tags. while identifying this type of information is out of scope for this paper, we will show that this information can be used by a neural network for predicting pos tags of one language only using examples from another language. our goal is to redefine the proximity between languages to achieve a comparable model to the classic tree model, by considering only pos tags. figure 1 : the indo-european language branch; the graph was created by the igraph package for python @xcite . the languages are represented using their equivalent two-letter iso 639-1 code. to enable transferability between languages, we suggest using a multilingual pre-trained language model (mplm), fine-tuned for the pos tagging task in a multilingual environment. specifically, we take a multilingual zero-shot training approach by fine-tuning an mplm to predict pos tags for texts written in one language, the source language, and evaluating it on texts written in another language, the target language. the performance metrics are then used to estimate the similarity between the source and target languages. as a final step, we generate a language-similarity graph, which we describe as an approximation for the classic tree model. we make two main contributions: (1) reconstructing part of the classic tree model using pos-based similarity scores; and, (2) providing some insights into the cross-lingual generalization of mplms. we proceed as follows: in section 2 we cite some related studies, following by a detailed description of our method, provided in section 3. we end section 3 with reporting on some results. we discuss the results in section 4 and make some conclusions. ## related work there have been some prior studies on measuring distance between languages. in their paper, @xcite presented some empirical observations of how rapidly speakers of a given language gained proficiency in another tongue. specifically, they measured the speed of english acquisition by immigrants of various linguistic backgrounds in the united states and canada. their first languages were ranked for the distance from english, on a scale from 1.0 (very different than english) to 3.0 (closest to english). it has been found empirically that the greater the distance between an immigrant's origin language and english, the lower is the level of the immigrant's english language proficiency. there have been many attempts to use computational tools to infer the relations between languages; the dominant approach is known as phylogenetic linguistics. phylogenetic linguistics is about establishing historical relationships among languages, by considering the evolutionary nature of human languages. in computational phylogenetics, words and/or phonemes of what counts as the same language over time, are analyzed and compared among languages. specifically, @xcite was first to introduce a computational phylogenetic technique called lexicostatistics for comparing between two languages. in lexicostatistics, the similarity between two languages is calculated by a function of the percentage of cognates found in a predefined list of words of the two languages. swadesh's work has been followed by a number of studies that use lexicostatistics or a minor variation of it @xcite @xcite @xcite . instead of measuring the percentage of cognates, @xcite proposed to calculate a normalized levenstein distance among words with the same meaning and then to take the average over the words contained in a cross-lingual list. @xcite conducted a lexical comparison using the levenstein distance approach, between 4,350 languages of the asjp database @xcite , and created a full diagram of lexical proximity. they showed that lexical resemblance is related to genetic affiliation. however, some of the languages that have been found as lexically similar, according to their technique, are not closely genetically associated. another computational approach for measuring language similarity is based on corpus analysis. @xcite used the known per-plexity score of a probabilistic n-gram language model to measure the distance between european languages. @xcite compared 50 languages from different families by training a monolingual language model on each language individually, using a parallel corpus of the bible @xcite , and apply them to calculate perplexity on all the other languages. in some of the works that are mentioned above, the proximity between languages is not perfectly aligned with the classic tree model. while the main focus has always been on lexical similarity, some attempts were made to compare languages on the syntactic level. @xcite characterized 28 languages, mostly indo-european ones, using a set of 63 predefined morpho-syntactic parameters. they calculated a normalized hamming distance over those parametric representations, with which they were able to generate a language tree. they showed that this tree is equivalent to a tree that was generated based on a traditional lexicostatistics approach, suggesting that syntactic characteristics are sufficiently robust to reconstruct a plausible historical language tree. the same method was re-used in @xcite , which was concluded in a similar way. in a recent work, @xcite applied a different comparison technique on the same syntactic characteristics, using markov models. in all of those works, the selection of the syntactic characteristics to be used for comparison, plays an important role in the creation of a language proximity model. to the best of our knowledge, there have not been attempts to compare languages using syntactic information in a non-parametric way. in this work, we take a corpus-based approach to automatically extract part-of-speech tags from a given text in order to generate a language-proximity model. in that sense, we consider our approach as a nonparametric estimation method, since we do not need to manually define specific syntactic parameters to consider for calculating similarity between languages. to transfer information across languages, we use mbert, a multilingual version of bert @xcite , that was pre-trained on texts written in over 100 languages based on a shared vocabulary. 2 during pre-training, the training documents are given to mbert without any indication on the language that they have been written with. like every other pre-trained language model (plm), the pre-trained mbert model is typically fine-tuned on a training set of a specific downstream task, which could be either monolingual or multilingual. this unique multilingual design allows mbert to handle multilingual tasks in a transfer-learning way. in another study, @xcite reported an impressive performance using mbert in a zero-shot cross-lingual transfer learning setting on several nlp tasks, including pos tagging. they claimed that mbert may learn a cross-lingual representation by generalizing and abstracting some language-specific information. a similar observation was made by @xcite who claimed that mbert learns information by two components, one that encodes the language and another that encodes some abstract information that can be used in a cross-lingual way. ## discussion and conclusions in this work we used a cross-lingual model trained on upos for measuring the proximity between languages. we showed that our new languageproximity model can reconstruct families of genetically related languages, suggesting that pos information plays a major role in modelling similarity between languages. we believe that we have demonstrated the potential of a fine-tuned mbert model to capture some cross-lingual information that is needed for assigning upos tags to a text written in an unseen language. on average, models of genetically related languages perform better on each other in this task, even if they are not written in the same script. for example, in there are a few caveats to this research to note. another caveat is the size and quality of the treebanks we use for training and testing our models. as noted before, we believe that our approach to represent a language using scores from models trained on all the 36 language included in this research, mitigates this risk. we make a final practical observation. the results of our study suggest that for upos tagging, mbert may benefit from training on texts written in languages that are genetically similar to the target language, based on the classic tree model. these results are aligned with what have been reported by @xcite .