Cross-Lingual Transfer from Related Languages: Treating Low-Resource M altese as Multilingual Code-Switching (2024)

Abstract: Although multilingual language models exhibit impressive cross-lingual transfer capabilities on unseen languages, the performance on downstream tasks is impacted when there is a script disparity with the languages used in the multilingual model’s pre-training data. Using transliteration offers a straightforward yet effective means to align the script of a resource-rich language with a target language thereby enhancing cross-lingual transfer capabilities. However, for mixed languages, this approach is suboptimal, since only a subset of the language benefits from the cross-lingual transfer while the remainder is impeded. In this work, we focus on Maltese, a Semitic language, with substantial influences from Arabic, Italian, and English, and notably written in Latin script. We present a novel dataset annotated with word-level etymology. We use this dataset to train a classifier that enables us to make informed decisions regarding the appropriate processing of each token in the Maltese language. We contrast indiscriminate transliteration or translation to mixing processing pipelines that only transliterate words of Arabic origin, thereby resulting in text with a mixture of scripts. We fine-tune the processed data on four downstream tasks and show that conditional transliteration based on word etymology yields the best results, surpassing fine-tuning with raw Maltese or Maltese processed with non-selective pipelines.

## introduction due to their impressive cross-lingual transfer capabilities, multilingual models have facilitated the development of nlp tools for low-resource languages @xcite @xcite . however, multilin-gual models may fall short in addressing lowerresourced languages @xcite . in particular, @xcite show that the cross-lingual transfer capabilities of a model are affected if the related language seen during pre-training uses a different script. they further show that transliterating to match the script of the related language improves performance. in this work, we focus on maltese -a semitic language with an arabic base and substantial romance influences written in latin script. @xcite transliterate maltese into arabic script and demonstrate improved performances in certain scenarios when fine-tuning with an arabic large language model as opposed to a multilingual one in the original script. however, being influenced by a mixture of languages -predominantly arabic, italian, and english -we argue that transliterating maltese entirely into the arabic script ignores the non-arabic aspect of the language. hence, the advantages derived from transliteration are diminished by the losses incurred through moving farther from italian and english. therefore, we propose mixing scripts and applying transliteration selectively. specifically, we apply transliteration to maltese words of arabic origin, keeping the others in their original latin script. we also experiment with mixing transliterations with word-level translations, which yielded the best results overall. our main contributions are as follows: the code, the new etymological annotations, and classifiers are released publicly. 1 2 background and related work due to the mixed nature of maltese, the language can be viewed as a highly code-switched language among arabic, italian, and english. an analysis of the dictionary from @xcite ) reveals that 32.4%, 52.5%, and 6.1% of maltese words are of arabic, italian/sicilian, and english origin, respectively @xcite . the remaining cases include mixed or unknown-origin words. we note that arabic-origin words tend to have higher token frequencies and include function words, and the dictionary entries do not include all inflected cases. our work is related to previous works dealing with languages not written in their standard script and/or mixed with other languages, predominantly english. pant and dadu (2020) define a pipeline for hinglish written in latin script, which only transliterates hindi-tagged tokens to devanagari script. @xcite define a pipeline for transliterating arabizi @xcite text into arabic script, which includes separate subprocesses for symbols, names, foreign words, and arabic words. @xcite define a neural model for transliteration of arabizi text into arabic script, but they skip english words similar to @xcite . while these approaches are similar to some of the pipelines presented in section 4.2, the majority of their token distribution (80%+) is in latinized hindi or arabic, compared to around 60% arabicorigin tokens for maltese ( thus, in this work, we build a robust classification model to predict word etymologies, using newly annotated data, to provide more accurate information to our processing pipelines. ## etymology annotations to build our dataset, we extracted 439 sentences (9,683 tokens) from the maltese universal dependencies treebank @xcite training set. we were directly involved in the creation of the guidelines, the annotation of the tokens, including extensive discussions and resolution of disagreements. among us, we have native language expertise in arabic and maltese and second language expertise in english and italian. we relied extensively on authoritative references (mentioned below). the following are the labels we annotated with. arabic maltese tokens of arabic origin, following the etymological classification by @xcite . this includes words that are derived from arabic dialects, such as tunisian arabic, but we retain the same classification for these. ## methodology our objective is to process maltese tokens in such a way as to improve cross-lingual transfer. we design pipelines that use transliteration and translation as our main tools to process maltese (section 4.2). for transliteration, we use the implementation from @xcite . specifically, we extend the non-deterministic character mappings with tunisian word model ranking and full closed-class token mappings, by making some modifications to the character maps. primarily, we add mappings for digits and other common symbols to arabic script instead of passing them as is. we also include additional mappings for some letters that were missing in @xcite , such as from t to θ. for translation, word-level translations are extracted from google translate. admittedly, this may give sub-optimal translations due to the lack of sentence context. however, we do not translate at the sentence level because we make token-level decisions and sometimes require partial translations of a subset of words in a sentence. in addition, most of the tasks used in the evaluation (section 5) are token-level classification tasks. hence, we decided against using word aligners with a sentence translation since this would amplify the noise in the processing pipeline. at the same time, wordlevel translations allow us to reduce the processing power needed, as they are extracted once on the unique set of tokens in the datasets used in section 5, and saved as static token mappings. the processing pipelines make use of an etymology classifier (section 4.1), which also uses the transliterations and the translations as features. ## downstream task evaluation in this section, we conduct an extrinsic evaluation on four downstream tasks: part-of-speech tagging (pos), dependency parsing (dp), named-entity recognition (ner), and sentiment analysis (sa). refer to section 5.1 for further details on the tasks. each dataset is processed using all of the pipelines presented in section 4.2, keeping the corresponding labels/tags the same. the processed datasets are then used to fine-tune pre-trained language models. we run fine-tuning 5 times with different random seeds and report the mean performance. the language models used are the multilingual model mbert @xcite , the arabic model camelbert-mix @xcite , the italian model italianbert @xcite , the english model bert @xcite , and the maltese model bertu @xcite . due to the large number of combinations, we do not fine-tune every model on all the pipelines. instead, we only fine-tune models on the pipelines which produce data in a language that it has been intentionally pre-trained on. so we fine-tune camelbert on all pipelines which do a t ara and/or x ara action, italianbert on all pipelines which do a t ita action, and bert on all pipelines which do a t eng action. mbert is fine-tuned on all pipelines since it is multilingual. additionally all models are fine-tuned on the p and x ara /p pipelines to test their capabilities on maltese using only finetuning data. the results are presented in section 5.2 ## conclusion in this work, we analyze how partially transliterating maltese has an impact on downstream task performance. we present a newly annotated dataset with word etymology labels and build classifiers to predict these labels. using these classifiers, we design various pipelines to make decisions on which tokens to transliterate or otherwise. our evaluation using mbert shows that by exclusively transliterating words of arabic origin, downstream task performance improves. the best results are achieved by mixing transliterations with translations, where including english translations yields better results than fine-tuning on the original data on all tasks. these findings corroborate with those from @xcite , but we show this further by only transliterating words that would aid cross-lingual transfer. future work should explore language adaptation techniques @xcite using the pipelines presented here, to further improve the cross-lingual transfer capabilities of multilingual models. it is also interesting to apply this method during inference of few-shot and zero-shot settings. we also hope that our newly annotated dataset can be used as a resource to further support the understanding of maltese. while we have reduced the script difference between maltese and its related languages, other linguistic properties can also impact the cross-lingual performance @xcite and future work should investigate these facets.