Unknown Script: Impact of Script on Cross-Lingual Transfer (2024)

Abstract: Cross-lingual transfer has become an effective way of transferring knowledge between languages. In this paper, we explore an often overlooked aspect in this domain: the influence of the source language of a language model on language transfer performance. We consider a case where the target language and its script are not part of the pre-trained model. We conduct a series of experiments on monolingual and multilingual models that are pre-trained on different tokenization methods to determine factors that affect cross-lingual transfer to a new language with a unique script. Our findings reveal the importance of the tokenizer as a stronger factor than the shared script, language similarity, and model size.

## introduction the dominant natural language processing (nlp) approach nowadays involves cross-lingual transfer using pre-trained monolingual and multilingual language models. in line with this trend, numerous monolingual models have been released for various languages @xcite @xcite . multilingual models, which are trained on 100 or more languages, have also been introduced, such as xlm-r @xcite and m-bert @xcite . despite the advancements in the development of language models for high-resourced languages, the vast majority of the world's languages remain excluded from these models. out of over 6,500 spoken languages globally, less than 2% are represented in the current models, leaving the rest unseen and unaccounted for in the current language processing technology @xcite @xcite . training a model for each of these languages is impractical due to substantial data and computational resource requirements. several alternative approaches have been proposed, such as zero-shot figure 1 : we analyze the effect of script and tokenizer on cross-lingual transfer on a target language with a new script. we select six monolingual and multilingual models pre-trained using sub-word tokenizers and character tokenizers. we fine-tune these models on the ner and pos tasks in the original script (fidel) and the romanized version (latin). we observe that roberta has better cross-lingual transfer in both the original script and the romanized version. we also find that romanization is strongly beneficial in all cases of subword-based models (albert, bert,m-bert). additionally, fine-tuning arabic-bert, which is typologically similar to our target language (amharic), provides no advantage. we employ the base version of the models across all cases to ensure a fair comparison. the reported f1-score is averaged over five runs, with a standard deviation ranging between -0.003 and 0.009. while such methods require fewer resources in the target language, they often yield sub-optimal results @xcite . for example, using even a small amount of labeled data in the target language has been shown to be more effective than a zero-shot transfer @xcite . the effectiveness of cross-lingual transfer is influenced by several factors, such as language similarity between the source and target languages, the size of the pre-trained model, and the quality and amount of the pre-training and fine-tuning data @xcite @xcite . in this paper, we explore which factors determine cross-lingual transfer performance for a new language with a unique script. we consider a challenging case where the target language is not part of the pre-trained model, and the script has also not been seen in pre-training. our analysis targets three main factors: language similarity, tokenization methods, and script. we design an experiment by varying these factors. we evaluate a range of existing monolingual and multilingual models, specifically choosing those trained on typologically related or typologically distant languages. furthermore, we select models trained using various tokenizers, allowing us to assess how these choices impact cross-lingual performance for a language with a unique script. we focus on two main research questions: figure 1 shows the results of our experiment. our analysis shows that roberta has better crosslingual transfer irrespective of the script. however, romanization strongly affects cross-lingual transfer for models pre-trained using sub-word tokenizers in monolingual and multilingual settings. we make our code available. foot_0 ## related work under-resourced languages display considerable variation in several aspects @xcite . first, the amount of data for these languages varies greatly. secondly, many of these languages use scripts different from the latin script @xcite . finally, regarding linguistic characteristics, these languages often have distinct morphological and syntactic properties, especially when compared to high-resourced indo-european languages. recently, cross-lingual transfer has become an effective method for extending the capabilities of pre-trained monolingual and multilingual models for various languages. in this section, we present an overview of studies exploring cross-lingual transfer. ## methodology we experiment with a few-shot setting in which we fine-tune pre-trained models on downstream tasks. the target language and script are not part of the pre-trained models we explore. our few-shot setup follows a standard setting: we take an existing base model, fine-tune it, and test it on the original target language. language we experiment with amharic as our target language. according to the language classification system presented in @xcite , amharic is categorized as a class 2 underresourced language. it possesses a unique script and is characterized by its morphological complexity. we select two source languages for our analysis: english, which is typologically distant from amharic, and arabic, which is typologically related. both english and arabic of these languages do not share a script with amharic. we use the original fidel script and the romanized version for our fine-tuning and evaluation. ## results and analysis ## conclusion in this paper, we explore cross-lingual transfer in less explored but challenging settings where the target language is not seen during pre-training and 127 has a unique non-latin script. our analysis shows considerable differences in cross-lingual transfer performance among various models, possibly attributable to two key factors: the size of the pretrained model and the specific tokenizer used during pre-training. the model's size could impact its ability to capture and generalize across multiple languages, including a language distant from the pre-trained model's language. in light of recent studies that show the importance of tokenizers in cross-lingual transfer for under-resourced languages, we show that a choice of tokenizer plays a role in facilitating cross-lingual transfer.