UNK s Everywhere: A dapting Multilingual Language Models to New Scripts (2021)

Abstract: Massively multilingual language models such as multilingual BERT offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. However, due to limited capacity and large differences in pretraining data sizes, there is a profound performance gap between resource-rich and resource-poor target languages. The ultimate challenge is dealing with under-resourced languages not covered at all by the models and written in scripts unseen during pretraining. In this work, we propose a series of novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts. Relying on matrix factorization, our methods capitalize on the existing latent knowledge about multiple languages already available in the pretrained model’s embedding matrix. Furthermore, we show that learning of the new dedicated embedding matrix in the target language can be improved by leveraging a small number of vocabulary items (i.e., the so-called lexically overlapping tokens) shared between mBERT’s and target language vocabulary. Our adaptation techniques offer substantial performance gains for languages with unseen scripts. We also demonstrate that they can yield improvements for low-resource languages written in scripts covered by the pretrained model.

## introduction massively multilingual language models pretrained on large multilingual data, such as multilingual bert (mbert; @xcite and xlm-r @xcite are the current state-of-theart vehicle for effective cross-lingual transfer @xcite . however, while they exhibit strong transfer performance between resource-rich and similar languages @xcite , these models struggle with transfer to low-resource languages @xcite and languages not represented at all in their pre-amharic (am) tibetan ( bo ) khmer (km) divehi (dv) sinhala (si) የእግዚአብሔርን ཉྩ ཡོ ནི རོ དྷ ཨེ ཝ មានសេចក្តីត្រិះរិះ ‫ަރ‬ ‫ީއ‬ ‫ުސ‬ ‫ްލ‬ ‫ުޖ‬ ‫ްމ‬ ‫ޫހ‬ ‫ިރ‬ ‫ްއ‬ ‫ާޔ‬ ‫ެގ‬ ප් රජාතන් ත් රවාදී የሚሰጡዋቸውን ྃ བཱ དཱ ི མཧཱ ཤྲ མཎ ត្រូវអាបត្ដិទុក្កដ ‫ެދ‬ ‫ެނ‬ ‫ަވ‬ ‫ަޑ‬ ‫ިއ‬ ‫ަގ‬ ‫ްނ‬ ‫ަނ‬ ‫ަވ‬ ‫ެއ‬ ‫ެވ‬ දෙපාර් තමේන් තු ව የእግዚአብሔር བཱ དཱ ི མཧཱ ཤྲ མཎ តាំងឡើងដោយសមុដ្ឋាន ‫ޮއ‬ ‫ްސ‬ ‫ިޓ‬ ‫ޯއ‬ ‫ާއ‬ ‫ުތ‬ ‫ަރ‬ ‫ިއ‬ ‫ިޓ‬ ‫ްސ‬ පු රාවිද් යාත් මක ቤተክርስቲያን ཧཱ ུ ྃ ཕཊསྭ ཱ ཧཱ ពួកឆព្វគ្គិយភិក្ខុ ‫ަރ‬ ‫ީއ‬ ‫ުސ‬ ‫ްލ‬ ‫ުޖ‬ ‫ްމ‬ ‫ޫހ‬ ‫ިރ‬ ‫ްއ‬ ‫ާޔ‬ ක් රියාකාරිත් වය ኢትዮጵያውያን ཧཱ ུ ྃ ཧཱ ུ ྃ ཕཊ ដែលមិនទាន់ជ្រះថ្លា ‫ިލ‬ ‫ިބ‬ ‫ަވ‬ ‫ަޑ‬ ‫ިއ‬ ‫ެގ‬ ‫ްނ‬ ‫ަފ‬ ‫ިއ‬ ‫ާވ‬ ක් රියාකාරීත් වය በእግዚአብሔር བྷ ྱ ཿཤཱ ནྟ ི ཾ ភិក្ខុត្រូវអាបត្តិ ‫ަކ‬ ‫ާނ‬ ‫ްތ‬ ‫ަފ‬ ‫ާރ‬ ‫ުތ‬ ‫ްނ‬ ‫ެނ‬ ‫ެވ‬ විශ් වවිද් යාලයේ እግዚአብሔርን བཻ ཌཱ ུ རྱ འི ឃុំមួយស្ថិតនៅក្នុង ‫ުޖ‬ ‫ްމ‬ ‫ޫހ‬ ‫ިރ‬ ‫ްއ‬ ‫ާޔ‬ ‫ެއ‬ ‫ެކ‬ ‫ެވ‬ දිස් ත් රික් කයේ ከእግዚአብሔር ེ དྷ རྨ ཱ ཧེ ཏ របស់ព្រះមានព្រះភាគ ‫ިލ‬ ‫ިބ‬ ‫ަވ‬ ‫ަޑ‬ ‫ިއ‬ ‫ެގ‬ ‫ްނ‬ ‫ެނ‬ ‫ިވ‬ සම් පූ ර් ණයෙන් ම የመጀመሪያውን ཝཱ ཧེ ཏུ ནྟ ེ ធ្វើឲ្យជាក់ច្បាស់ ‫ަބ‬ ‫ާޔ‬ ‫ްނ‬ ‫ޮކ‬ ‫ްށ‬ ‫ަފ‬ ‫ިއ‬ ‫ަވ‬ ‫ީނ‬ පාර් ලිමේන් තු ව ትምህርታቸውን སྤ ྲ ེ འུ འི ធ្វើទុកក្នុងចិត្ ‫ުއ‬ ‫ަފ‬ ‫ްނ‬ ‫ެވ‬ ‫ަވ‬ ‫ަޑ‬ ‫ިއ‬ ‫ަގ‬ විශ් වවිද් යාලය amharic tibetan divehi የእግዚአብሔርን ཉྩ ཡོ ནི རོ དྷ ཨེ ཝ ‫ަރ‬ ‫ީއ‬ ‫ުސ‬ ‫ްލ‬ ‫ުޖ‬ ‫ްމ‬ ‫ޫހ‬ ‫ިރ‬ ‫ްއ‬ ‫ާޔ‬ ‫ެގ‬ የሚሰጡዋቸውን ྃ བཱ དཱ ི མཧཱ ཤྲ མཎ ‫ެދ‬ ‫ެނ‬ ‫ަވ‬ ‫ަޑ‬ ‫ިއ‬ ‫ަގ‬ ‫ްނ‬ ‫ަނ‬ ‫ަވ‬ ‫ެއ‬ ‫ެވ‬ የእግዚአብሔር བཱ དཱ ི མཧཱ ཤྲ མཎ ‫ޮއ‬ ‫ްސ‬ ‫ިޓ‬ ‫ޯއ‬ ‫ާއ‬ ‫ުތ‬ ‫ަރ‬ ‫ިއ‬ ‫ިޓ‬ ‫ްސ‬ ቤተክርስቲያን ཧཱ ུ ྃ ཕཊསྭ ཱ ཧཱ ‫ަރ‬ ‫ީއ‬ ‫ުސ‬ ‫ްލ‬ ‫ުޖ‬ ‫ްމ‬ ‫ޫހ‬ ‫ިރ‬ ‫ްއ‬ ‫ާޔ‬ ኢትዮጵያውያን ཧཱ ུ ྃ ཧཱ ུ ྃ ཕཊ ‫ިލ‬ ‫ިބ‬ ‫ަވ‬ ‫ަޑ‬ ‫ިއ‬ ‫ެގ‬ ‫ްނ‬ ‫ަފ‬ ‫ިއ‬ ‫ާވ‬ በእግዚአብሔር བྷ ྱ ཿཤཱ ནྟ ི ཾ ‫ަކ‬ ‫ާނ‬ ‫ްތ‬ ‫ަފ‬ ‫ާރ‬ ‫ުތ‬ ‫ްނ‬ ‫ެނ‬ ‫ެވ‬ እግዚአብሔርን བཻ ཌཱ ུ རྱ འི ‫ުޖ‬ ‫ްމ‬ ‫ޫހ‬ ‫ިރ‬ ‫ްއ‬ ‫ާޔ‬ ‫ެއ‬ ‫ެކ‬ ‫ެވ‬ ከእግዚአብሔር ེ དྷ རྨ ཱ ཧེ ཏ ‫ިލ‬ ‫ިބ‬ ‫ަވ‬ ‫ަޑ‬ ‫ިއ‬ ‫ެގ‬ ‫ްނ‬ ‫ެނ‬ ‫ިވ‬ የመጀመሪያውን ཝཱ ཧེ ཏུ ནྟ ེ ‫ަބ‬ ‫ާޔ‬ ‫ްނ‬ ‫ޮކ‬ ‫ްށ‬ ‫ަފ‬ ‫ިއ‬ ‫ަވ‬ ‫ީނ‬ ትምህርታቸውን སྤ ྲ ེ འུ འི ‫ުއ‬ ‫ަފ‬ ‫ްނ‬ ‫ެވ‬ ‫ަވ‬ ‫ަޑ‬ ‫ިއ‬ ‫ަގ‬ training corpora @xcite @xcite . the most extreme challenge is dealing with unseen languages with unseen scripts (i.e., the scripts are not represented in the pretraining data; see figure 1 ), where the pretrained models are bound to fail entirely if they are used off-the-shelf without any further model adaptation. existing work focuses on the embedding layer and learns either a new embedding matrix for the target language @xcite or adds new tokens to the pretrained vocabulary. while the former has only been applied to high-resource languages, the latter approaches have been limited to languages with seen scripts @xcite and large pretraining corpora @xcite . another line of work adapts the embedding layer as well as other layers of the model via adapters @xcite . such methods, however, cannot be directly applied to languages with unseen scripts. in this work, we first empirically verify that the original tokenizer and the original embedding layer of a pretrained multilingual model fail for languages with unseen script. this implies that dedicated in-language tokenizers and embeddings are a crucial requirement for any successful model adaptation. the key challenge is aligning new target language embeddings to the pretrained model's representations while leveraging knowledge encoded in the existing embedding matrix. we systematize existing approaches based on the pretrained information they utilize and identify lexically overlapping tokens that are present in both vocabularies as key carriers of such information @xcite . 1 we then present novel, effective, and dataefficient methods for adapting pretrained multilingual language models to resource-low languages written in different scripts. beyond lexical overlap, our methods rely on factorized information from the embedding matrix and token groupings. we evaluate our approaches in the named entity recognition (ner) task on the standard wikiann dataset @xcite and dependency parsing (dp; @xcite . we use 4 diverse resource-rich languages as source languages, and transfer to 17 and 6 resource-poor target languages respectively, including 5 languages with unseen scripts (amharic, tibetan, khmer, divehi, sinhala). we show that our adaptation techniques offer unmatched performance for languages with unseen scripts. they also yield improvements for lowresource and under-represented languages written in scripts covered by the pretrained model. contributions. 1) we systematize and compare current model adaptation strategies for lowresource languages with seen and unseen scripts. the code for this work is released at github.com/ adapter-hub/unks_everywhere. ## background: multilingual model adaptation for cross-lingual transfer standard cross-lingual transfer setup with a state-of-the-art pretrained multilingual model such as mbert or xlm-r is 1) fine-tuning it on labelled data of a downstream task in a source language and then 2) applying it directly to perform inference in a target language @xcite . however, as the model must balance between many languages in its representation space, it is not suited to excel at a specific language at inference time without further adaptation @xcite . adapters for cross-lingual transfer. adapterbased approaches have been proposed as a remedy @xcite @xcite @xcite @xcite . in the crosslingual setups, the idea is to increase the multilingual model capacity by storing language-specific knowledge of each language in dedicated parameters @xcite . we start from mad-x @xcite , a state-of-the-art adapter-based framework for crosslingual transfer. for completeness, we provide a brief overview of the framework in what follows. mad-x comprises three adapter types: language, task, and invertible adapters; this enables learning language and task-specific transformations in a modular and parameter-efficient way. as in prior work @xcite , adapters are trained while keeping the parameters of the pretrained multilingual model fixed. language adapters are trained via masked language modeling (mlm) on unlabelled target language data. task adapters are trained via taskspecific objectives on labelled task data in a source language while also keeping the language adapters fixed. task and language adapters are stacked: this enables the adaptation of the pretrained multilin-gual model to languages not covered in its pretraining data. mad-x keeps the same task adapter while substituting the source language adapter with the target language adapter at inference. in brief, the adapters a l at layer l consist of a down-projection @xmath0, followed by a gelu activation @xcite and an upprojection @xmath1: current model adaptation approaches @xcite generally fine-tune all model parameters on target language data. instead, we follow the more computationally efficient adapter-based paradigm where we keep model parameters fixed, and only train language adapters and target language embeddings. crucially, while the current adapter-based methods offer extra capacity, they do not offer mechanisms to deal with extended vocabularies of many resource-poor target languages, and do not adapt their representation space towards the target language adequately. this problem is exacerbated when dealing with unseen languages and scripts. foot_1 ## cross-lingual transfer of lexical information the embedding matrix of large multilingual models makes up around 50% of their entire parameter budget @xcite . however, it is not clear how to leverage this large amount of information most effectively for languages that are not adequately represented in the shared multilingual vocabulary due to lack of pretraining data. a key challenge in using the lexical information encoded in the embedding matrix is to overcome a mismatch in vocabulary between the pretrained model and the target language. to outline this issue, in method special lexical latent semantic language new # of new reference tokens overlap concepts clusters params params el-rand x 7.68m artetxe et al. (2020) el-lex x 7.68m chau et al. (2020); wang et al. (2020) mf c * -rand f , i 1m + c•10k ours mf c * -lex f , i 1m + c•10k ours recent approaches such as invertible adapters @xcite that adapt embeddings in the pretrained multilingual vocabulary may be able to deal with lesser degrees of lexical overlap. still, they cannot deal with unk tokens. in the following, we systematize existing approaches and present novel ways of adapting a pretrained model to the vocabulary of a target language that can handle this challenging setting, present most acutely when adapting to languages with unseen scripts. we summarize the approaches in ## results and discussion the main results are summarised in for our approaches that learn new embedding matrices, we observe that for languages seen during pretraining, but potentially underrepresented by the we observe improvements for most languages with lexical overlap initialization. this adds further context to prior studies which found that a shared vocabulary is not necessary for learning multilingual representations @xcite : while it is possible to generalize to new languages without lexical overlap, leveraging the overlap still offers additional gains. the methods based on matrix factorization (mf * * - * ) improve performance over full-sized embedding methods (el * * - * ), especially in the setting without lexical overlap initialization ( * -rand). this indicates that by factorizing the information encoded in the original embedding matrix we are able to extract relevant information for unseen languages. combining matrix factorization with lexical overlap initialization (mf * * -lex), zero-shot performance improves further for unseen languages with covered scripts. this suggests that the two methods complement each other. for 6/9 of these languages, we find that encoding the embeddings in multiple up-projections (mf 10 * - * ) achieves the peak score. this in turn verifies that grouping similar tokens improves the robustness of token representations @xcite . for unseen languages with covered scripts, this model variant also outperforms mad-x 2.0 on average. for languages with unseen scripts we find that mf has smaller impact. while the encoded information supports languages similar to those seen by the model in pretraining, languages with unseen scripts are too distant to benefit from this latent multilingual knowledge. surprisingly, lexical overlap is helpful for languages with unseen scripts. overall, we observe that both mf 10 kmeans - * and mf 10 neural - * perform well for most languages, where the kmeans variant performs better for ner and the neural variant performs better for ud. ## conclusion we have systematically evaluated strategies for model adaptation to unseen languages with seen and unseen scripts. we have assessed the importance of the information stored within the original embedding matrix by means of leveraging lexically overlapping tokens, and extracting latent semantic concepts. for the latter, we have proposed a new method of encoding the embedding matrix into lower-dimensional embeddings and up-projections. we have demonstrated that our methods outperform previous approaches on ner and dependency parsing for both resource-rich and resource-poor scenarios, reducing the gap between random and lexical overlap initialisation, and enabling more effective model adaptation to unseen scripts. 12ad seen languages unseen languages but covered scripts new scripts ka ur hi avg cdo mi ilo gn xmf sd myv bh wo avg am bo km dv si avg mbert 64.3 34.3 65.3 54.6 17.3 22.2 62.7 46.7 34.4 12.1 47.2 48.8 26.7 35.4 0.0 11.4 4.9 0.3 0.3 3.4 mad-x the top group (first three rows) includes models which leverage the original tokenizer which is not specialized for the target language. the second group (last 18 rows) include models with new tokenizers. here we separate models with randomly initialized embeddings ( * -randinit) from models with lexical initialization ( * -lexinit) by the dashed line. we additionally present the results for full model fine-tuning (fmt- * ). the top group (first three rows) includes models which leverage the original tokenizer which is not specialized for the target language. the second group (last 18 rows) include models with new tokenizers. here we separate models with randomly initialized embeddings ( * -randinit) from models with lexical initialization ( * -lexinit) by the dashed line. we additionally present the results for full model fine-tuning (fmt- * ). the top group (first three rows) includes models which leverage the original tokenizer which is not specialized for the target language. the second group (last 18 rows) include models with new tokenizers. here we separate models with randomly initialized embeddings ( * -randinit) from models with lexical initialization ( * -lexinit) by the dashed line. we additionally present the results for full model fine-tuning (fmt- * ).