Evaluating zero-shot transfers and multilingual models for dependency parsing and POS tagging within the low-resource language family Tupían (2022)

Abstract: This work presents two experiments with the goal of replicating the transferability of dependency parsers and POS taggers trained on closely related languages within the low-resource language family Tupían. The experiments include both zero-shot settings as well as multilingual models. Previous studies have found that even a comparably small treebank from a closely related language will improve sequence labelling considerably in such cases. Results from both POS tagging and dependency parsing confirm previous evidence that the closer the phylogenetic relation between two languages, the better the predictions for sequence labelling tasks get. In many cases, the results are improved if multiple languages from the same family are combined. This suggests that in addition to leveraging similarity between two related languages, the incorporation of multiple languages of the same family might lead to better results in transfer learning for NLP applications.

## introduction for most of the 7000 languages of the world, no nlp resources exist @xcite . as a response to this situation, more and more initiatives emerged in recent years that work on nlp applications for underrepresented and low-resource languages @xcite @xcite . despite those advances, access to tools like machine translation still is hindered by a large language barrier. most of those languages do not have large text corpora, which have been used for the recent advantages in nlp like the building of large transformer models @xcite . annotated data and parallel corpora thus remain an important but scarce tool for many of them. yet, annotating this data is a challenge itself, and might be aided through the transfer of models from languages with more available resources. the idea to leverage existing databases and models for cross-lingual transfer is not new @xcite @xcite @xcite . however, many studies even in this area remain within the environment of high-resource languages, and benchmarks with a typological sample as representative as possible -common nowadays in linguistic typology -are rarely found @xcite @xcite . the main goal of this contribution is to replicate previous findings on crosslingual transfer in low-resource settings (meechan-maddon and nivre, 2019) within an underpresented language family, tupían. ## data and hypotheses the data used for this study is taken from the tupían dependency treebanks project (tudet, gerardi et al., 2021) foot_0 , which is openly available under a cc-by-sa-4.0 license and is already partially present in the universal dependencies database. the author is not part of the team that developed these treebanks. there are currently seven languages in the dataset, which belong to different branches of the tupían family @xcite . except tupinambá, which is extinct, the languages are spoken in brazilian territory. all languages but guajajára have sov word order, while the former has vso. the datasets are summarized in in this study, i primarily test the utility of crosslingual transfer for pos-taggers and dependency parsers with special attention given to language phylogeny. language phylogeny can be seen as a proxy to typological features, given that closely related languages usually show many structural similarities. previous studies have shown that even a comparably small treebank from a closely related language will improve the results of annotation considerably @xcite . recent studies suggest to leverage phylogenetic proximity in a more efficient way than simply comparing languages based on the language family they belong to @xcite . which model generalizes best over the different treebanks used in this sample, and what role does language phylogeny play in this? in this study, 'closeness' of two languages is defined based on the proximity of their phylogenetic clades. this is used as a proxy to their typological similarity. especially for languages which do not have extensive descriptive material available, such similarities cannot easily be computed from typological databases. based on phylolinguistic inferences about tupían @xcite , the following explicit hypotheses are postulated: here, mundurukú is closer to akuntsú than to makuráp, and karo is closer to makuráp than to akuntsú. the results should mirror this relation. ## experiments one of the challenges for nlp applications with low-resource languages is the lack of languagespecific resources on which embeddings can be trained on @xcite . even though there are useful pipelines which can sometimes be used to crawl monolingual data from published sources @xcite , those are not always available or accessible. the embeddings used for the experiments in this contributions are based on the jw300-corpus @xcite . this corpus is derived specifically from 343 low-resource languages and shows greater typological diversity than most dominating multilingual models. the embeddings are implemented in flair @xcite . they have been fine-tuned for the pooled set of source languages. transformer word embeddings mbert @xcite and roberta @xcite were also evaluated for the model, but rarely surpassed 40% accuracy for the source languages and have thus been discarded from further experiments for now. this results further call into question the utility of such large models for typologically diverse languages, and strengthens previous findings that even the largest multilingual transformer models do not show good results when transferring to typologically different languages @xcite @xcite . however, the exact reasons for their failure in this experiment are not entirely clear and need further research with more typologically diverse low-resource languages. the experiments will be done for both pos tagging and dependency parsing and include a zeroshot setting. also, models trained on individual source languages will be compared against models trained on multiple datasets, with the evaluation set being the remaining treebanks of the dataset. given the small amoung of training data and the models chosen, all model runs combined did not need more than three hours on cpu. the evaluation was done within the provided utilities by flair and supar, respectively. all code is available on osf. foot_1 ## conclusion this study further confirms previous findings that cross-lingual transfer of dependency parsers and pos taggers is a viable option in low-resource settings if a closely related language is available @xcite . this extends previous evidence for phylogenetically informed transfer from indo-european and uralic @xcite to tupían. further experiments on other language families should be conducted in order to confirm the exact features that make successful transfer possible. further, this study provided further evidence for extending the phylolinguistically informed combination of source languages. in all experiment settings of this study, the pooled source language set had very good results, and a targeted combination will likely further improve the results. further follow-up experiments will consist of targeted combinations of annotated data from different languages, including an incorporation of typological features and delexicalized transfer. in preliminary experiments, crf2o dependency parsing @xcite showed promising results for transfer results as well. especially in the dependency parsing experiment the transfer scores were quite low, and further improving the training data as well as comparing different models should be a viable solution for this challenge.