Transfer to a Low-Resource Language via Close Relatives: The Case Study on F aroese (2023)

Abstract: Multilingual language models have pushed state-of-the-art in cross-lingual NLP transfer. The majority of zero-shot cross-lingual transfer, however, use one and the same massively multilingual transformer (e.g., mBERT or XLM-R) to transfer to all target languages, irrespective of their typological, etymological, and phylogenetic relations to other languages. In particular, readily available data and models of resource-rich sibling languages are often ignored. In this work, we empirically show, in a case study for Faroese – a low-resource language from a high-resource language family – that by leveraging the phylogenetic information and departing from the ‘one-size-fits-all’ paradigm, one can improve cross-lingual transfer to low-resource languages. In particular, we leverage abundant resources of other Scandinavian languages (i.e., Danish, Norwegian, Swedish, and Icelandic) for the benefit of Faroese. Our evaluation results show that we can substantially improve the transfer performance to Faroese by exploiting data and models of closely-related high-resource languages. Further, we release a new web corpus of Faroese and Faroese datasets for named entity recognition (NER), semantic text similarity (STS), and new language models trained on all Scandinavian languages.

## introduction massively multilingual transformer-based language models (mmts) such as mbert @xcite , xlm-roberta @xcite and mt5 @xcite have been the driving force of modern multilingual nlp, allowing for rapid bootstrapping of language tech-nology for a wide range of low(er)-resource languages by means of (zero-shot or few-shot) crosslingual transfer from high(er)-resource languages @xcite @xcite . cross-lingual transfer with mmts is not without drawbacks. mmts' representation spaces are heavily skewed in favor of high-resource languages, for which they have been exposed to much more data in pretraining @xcite ; combined with the 'curse of multilinguality' -i.e., limited per-language representation quality stemming from a limited capacity of the model @xcite -this leads to lower representational quality for languages underrepresented in mmts' pretraining. cross-lingual transfer with mmts thus fails exactly in settings in which it is needed the most: for low-resource languages with small digital footprint @xcite . despite these proven practical limitations, the vast majority of work on cross-lingual transfer still relies on mmts due to their appealing conceptual generality: in theory, they support transfer between any two languages seen in their pretraining. such strict reliance on mmts effectively ignores the linguistic phylogenetics and fails to directly leverage resources of resource-rich languages that are closely related to a target language of interest. in this work, we attempt to mitigate the above limitations for a particular group of languages, departing from the 'one-size-fits-all' paradigm based on mmts. we focus on a frequent and realistic setup in which the target language is a lowresource language but from a high-resource language family, i.e., with closely related resourcerich languages. a recent comprehensive evaluation of the languages used in europe 1 scores languages based on the available resources. languages such as german and spanish score at around 0.5 of the english scores, and more than half of the languages are scored below 0.02 of the english score. many, including almost all regional and minority languages such as faroese, scottish gaelic, occitan, luxembourgish, romani languages, sicilian and meänkieli have the score of (almost) 0. however, what differentiates these languages from low-resource languages from africa (e.g., niger-congo family) or indigenous languages of latin america (e.g., tupian family) is the fact that they typically have closely related high-resource languages as 'language siblings'. in this case, we believe, language models (lms) of closely related high-resource languages promise more effective transfer compared to using mmts, plagued by the 'curse of multilinguality', as the vehicle of transfer. in this proof-of-concept case study, we focus on faroese as the target language and demonstrate the benefits of linguistically informed transfer. we take advantage of available data and resources from the closely related but much more 'nlp-developed' other scandinavian languages. 2 we show that using "scandinavian" lms brings substantial gains in downstream transfer to faroese compared to using xlm-r as a widely used off-the-shelf mmt. the gains are particularly pronounced for the task of semantic text similarity (sts), the only high-level semantic task in our evaluation. we further show that adding a limited-size target-language corpus to lm's pretraining corpora brings further gains in downstream transfer. as another contribution of this work, we collect and release: (1) a corpus of web-scraped monolingual faroese, (2) multiple lms suitable for faroese, including those trained on all five scandinavian languages, and (3) two new task-specific datasets for faroese labeled by native speakers: for ner and sts. other methods involve translation-based transfer @xcite , and transfer from monolingual language models @xcite @xcite . bilingual lexical induction (bli) is the method of mapping properties, in particular embeddings, from one language to another via some means such as supervised embedding alignment, unsupervised distribution matching or using an orthogonality constraint @xcite @xcite , and has also been used to build language tools in low-resource languages @xcite . attempts to alleviate the abovementioned issues have been made, such as vocabulary extension methods @xcite , which add missing tokens and their configurations to the embedding matrix. phylogeny-inspired methods have also been used where adapters have been trained for multiple languages and stacked to align with the language family of the language of interest (faisal and anastasopoulos, 2022). some analysis on the effects of using pretrained mmts has been done: @xcite conclude that using pretrained mmts that share script and overlap in the family with the target language is beneficial. however, when adapting the model for a new language, they claim that using as many languages as possible (up to 100) generally yields the best performance. inspired by this line of research, in this work, we focus on improving mmt-based cross-lingual transfer for a particular group of languages, those that have sibling languages with more abundant data and resources. ## conclusion and future work we have shown that leveraging phylogenetic information and departing from the 'one-size-fits-all' paradigm can improve cross-lingual transfer to lowresource languages. our evaluation results show that we can substantially improve the transfer performance to faroese by exploiting data and models of closely-related high-resource languages instead of relying on mmts. in future work, we hope to extend the investigations and methodology beyond faroese, to other low-resource languages for which higher-resource language relatives exist. in order to boost and guide future research on scandinavian languages in general and faroese in particular, we make the models scandibert foot_7 , scandibert-no-fo foot_8 , danskbert foot_9 and fobert (scandibert-no-fo-fc3) foot_10 available. as well as the new datasets fc3 foot_11 , fone foot_12 , and fo-sts foot_13 . danskbert. all models use the same vocabulary size of 50k. the scandibert vocabulary is trained using all the languages, the scandibert-no-fo vocabulary is trained without the faroese data, and the danskbert vocabulary is only trained on the danish text. vocabularies are trained using the sen-tencepiece software @xcite , and character coverage is set to 99.995 %. pre-training of the new language models is done using fairseq @xcite using the roberta-base @xcite configuration, fine-tuning is done using the transformers @xcite library. scandibert and scandibertno-fo were trained for 72 epochs, using a batch size of 8.8k sequences on 24 nvidia v100 cards for approximately 14 days each. initial testing showed that the larger batch size showed better performance than going for around 2k sequences, possibly due to the mixture of differing languages. danskbert, on the other hand, similar to icebert and roberta showed better performance at the smaller batch size. danskbert was trained to convergence for 500k steps using 16 v100 cards for approximately 14 days. all -fc models are further trained for 50 epochs, with an effective batch size of 100k tokens for 12k updates, over the fc3 dataset for faroese adaptation. an overview of the data used to train the language models is shown in