R omanization-based Large-scale Adaptation of Multilingual Language Models (2023)

Abstract: Large multilingual pretrained language models (mPLMs) have become the de facto state of the art for cross-lingual transfer in NLP. However, their large-scale deployment to many languages, besides pretraining data scarcity, is also hindered by the increase in vocabulary size and limitations in their parameter budget. In order to boost the capacity of mPLMs to deal with low-resource and unseen languages, we explore the potential of leveraging transliteration on a massive scale. In particular, we explore the UROMAN transliteration tool, which provides mappings from UTF-8 to Latin characters for all the writing systems, enabling inexpensive romanization for virtually any language. We first focus on establishing how UROMAN compares against other language-specific and manually curated transliterators for adapting multilingual PLMs. We then study and compare a plethora of data- and parameter-efficient strategies for adapting the mPLMs to romanized and non-romanized corpora of 14 diverse low-resource languages. Our results reveal that UROMAN-based transliteration can offer strong performance for many languages, with particular gains achieved in the most challenging setups: on languages with unseen scripts and with limited training data without any vocabulary augmentation. Further analyses reveal that an improved tokenizer based on romanized data can even outperform non-transliteration-based methods in the majority of languages.

## introduction massively multilingual language models (mplms) such as mbert @xcite and xlm-r @xcite have become the driving force for a variety of applications in multilingual nlp @xcite @xcite . however, guaranteeing and maintaining strong performance for a wide spectrum of low-resource languages is difficult due to two crucial problems. the first issue is the vocabulary size, as the vocabulary is bound to increase with the number of languages added if per-language performance is to be maintained @xcite @xcite . second, pretraining mplms with a fixed model capacity improves cross-lingual performance up to a point after which it starts to decrease; this is the phenomenon termed the curse of multilinguality @xcite . transliteration refers to the process of converting language represented in one writing system to another @xcite . latin scriptcentered transliteration or romanization is the most common form of transliteration @xcite @xcite as the latin/roman script is by far the most widely adopted writing script in the world @xcite . 1 adapting mplms via transliteration can address the two aforementioned critical issues. 1) since the latin script covers a dominant portion of the mplm's vocabulary (e.g., 77% in case of mbert, see @xcite , 'romanizing' the remaining part of the vocabulary might mitigate the vocabulary size issue and boost vocabulary sharing. 2) since no new tokens are added during the romanization process, reusing pretrained embeddings from the mplm's embedding matrix helps reuse the information already present within the mplm, thereby allocating the model's parameter budget more efficiently. however, the main drawback of transliteration seems to be the expensive process of creating effective language-specific transliterators, as they typically require language expertise to curate dictionaries that map tokens from one language and script to another. therefore, previous attempts at mplm adaptation to unseen languages via transliteration @xcite @xcite were constrained to a handful of languages due to the limited availability of language-specific transliterators, or were applied only to languages that have 'language siblings' with developed transliterators. in this work, unlike previous work, we propose to use and then evaluate the usefulness of a universal romanization tool, uroman @xcite , for quick, large-scale and effective adaptation of mplms to low-resource languages. the uroman tool disposes of language-specific curated dictionaries and maps any utf-8 character to the latin script, increasing the portability of romanization, with some examples in figure 1 . we analyze language adaptation on a massive scale via uroman-based romanization on a set of 14 diverse low-resource languages. we conduct experiments within the standard parameter-efficient adapter-based cross-lingual transfer setup on two tasks: named entity recognition (ner) on the wikiann dataset @xcite , and dependency parsing (dp) with universal dependencies v2.7 @xcite . our key results suggest that uroman-based transliteration can offer strong performance on par or even outperforming adaptation with language-specific transliterators, setting up the basis for wider use of transliteration-based mplm adaptation techniques in future work. the gains with romanization-based adaptation over standard adaptation baselines are particularly pronounced for languages with unseen scripts (∼8-22 performance points) without any vocabulary augmentation. foot_1 2 background why uroman-based romanization? uromanbased romanization is not always fully reversible, and its usage for transliteration has thus been limited in the literature. however, due to its high portability, uroman can help scale the process of transliteration massively and as such benefit lowresource scenarios and wider adaptation of mplms. the main idea, as hinted in §1, is to (learn to) map any utf-8 character to the latin script, without the use of any external language-specific dictionaries (see @xcite for technical details). ## experiments and results as the main means of analyzing the impact of transliteration in general and uroman-based romanization in particular, we train different variants of language adapters within the mad-x framework, based on transliterated and non-transliterated versions of target language data, outlined here. variants with non-transliterated data. for the non-trans la+inv variant, we train las and inv adapters together. this variant serves to examine the extent to which mplms can adapt to unseen languages without any vocabulary extension. 5 we compare this to non-trans la+emb lex , which trains a new tokenizer for the target language @xcite : the so-called 'lexically overlapping' tokens are initialized with mplm's trained embeddings, while the remaining embeddings are initialized randomly. all these embeddings (emb lex ) are fine-tuned along with las. variants with transliterated data. we evaluate a trans la+inv variant, which uses the same setup as non-trans la+inv but now with transliterated data. we again note that in this efficient setup, we do not extend the vocabulary size, and use the fewest trainable parameters. in the trans la+mplm ft variant, we train las along with fine-tuning the pretrained embeddings of mplm (mplm ft ). this further enhances the model capacity by fine-tuning the embedding layer instead of using invertible adapters. 6 for both variants, transliterated data can be produced via different transliterators: (i) languagespecific ones; (ii) the ones from 'language siblings' (e.g., using a georgian transliterator for mingrelian), or (iii) uroman. ## conclusion in this work, we have systematically analyzed and confirmed the potential of romanization, implemented via the uroman tool, to help with adaptation of multilingual pretrained language models. given (i) its broad applicability and (ii) strong performance overall and for languages with unseen scripts, we hope our study will inspire more work on transliteration-based adaptation.