Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: A n I ndic Languages Study (2021)

Abstract: Recent research in multilingual language models (LM) has demonstrated their ability to effectively handle multiple languages in a single model. This holds promise for low web-resource languages (LRL) as multilingual models can enable transfer of supervision from high resource languages to LRLs. However, incorporating a new language in an LM still remains a challenge, particularly for languages with limited corpora and in unseen scripts. In this paper we argue that relatedness among languages in a language family may be exploited to overcome some of the corpora limitations of LRLs, and propose RelateLM. We focus on Indian languages, and exploit relatedness along two dimensions: (1) script (since many Indic scripts originated from the Brahmic script), and (2) sentence structure. RelateLM uses transliteration to convert the unseen script of limited LRL text into the script of a Related Prominent Language (RPL) (Hindi in our case). While exploiting similar sentence structures, RelateLM utilizes readily available bilingual dictionaries to pseudo translate RPL text into LRL corpora. Experiments on multiple real-world benchmark datasets provide validation to our hypothesis that using a related language as pivot, along with transliteration and pseudo translation based data augmentation, can be an effective way to adapt LMs for LRLs, rather than direct training or pivoting through English.

## introduction bert-based pre-trained language models (lms) have enabled significant advances in nlp @xcite @xcite . pretrained lms have also been developed for the multilingual setting, where a single multilingual model is capable of handling inputs from many different * authors contributed equally figure 1 : number of wikipedia articles for top-few indian languages and english. the height of the english bar is not to scale as indicated by the break. number of english articles is roughly 400x more than articles in oriya and 800x more than articles in assamese. however, there is a challenge. the current paradigm for training mutlilingual lm requires text corpora in the languages of interest, usually in large volumes. however, such text corpora is often available in limited quantities for lrls. for example, in figure 1 we present the size of wikipedia, a common source of corpora for training lms, for top-few scheduled indian languages foot_0 and english. the top-2 languages are just one-fiftieth the size of english, and yet hindi is seven times larger than the o(20,000) documents of languages like oriya and assamese which are spoken by millions of people. this calls for the development of additional mechanisms for training multilingual lms which are not exclusively reliant on large monolingual corpora. recent methods of adapting a pre-trained multilingual lm to a lrl include fine-tuning the full model with an extended vocabulary @xcite , training a light-weight adapter layer while keeping the full model fixed @xcite , and exploiting overlapping tokens to learn embeddings of the lrl @xcite . these are general-purpose methods that do not sufficiently exploit the specific relatedness of languages within the same family. we propose relatelm for this task. relatelm exploits relatedness between the lrl of interest and a related prominent language (rpl). we focus on indic languages, and consider hindi as the rpl. the languages we consider in this paper are related along several dimensions of linguistic typology @xcite : phonologically, phylogenetically as they are all part of the indo-aryan family, geographically, and syntactically matching on key features like the subject-object-verb (sov) order as against the subject-verb-object (svo) order in english. even though the scripts of several indic languages differ, they are all part of the same brahmic family, making it easier to design rulebased transliteration libraries across any language pair. in contrast, transliteration of indic languages to english is harder with considerable phonetic variation in how words are transcribed. the geographical and phylogenetic proximity has lead to significant overlap of words across languages. this implies that just after transliteration we are able to exploit overlap with a related prominent language (rpl) like hindi. on three indic languages we discover between 11% and 26% overlapping tokens with hindi, whereas with english it is less than 8%, mostly comprising numbers and entity names. furthermore, the syntax-level similarity between languages allows us to generate high quality data augmentation by exploiting pre-existing bilingual dictionaries. we generate pseudo parallel data by converting rpl text to lrl and vice-versa. these allow us to further align the learned embed-dings across the two languages using the recently proposed loss functions for aligning contextual embeddings of word translations @xcite . in this paper, we make the following contributions: ## related work transformer @xcite based language models like mbert @xcite , muril @xcite , indicbert @xcite , and xlm-r @xcite , trained on massive multilingual datasets have been shown to scale across a variety of tasks and languages. the zero-shot cross-lingual transferability offered by these models makes them promising for lowresource domains. @xcite find that cross-lingual transfer is even possible across languages of different scripts, but is more effective for typologically related languages. however, recent works @xcite @xcite ) have identified poor cross-lingual transfer to languages with limited data when jointly pre-trained. a primary reason behind poor transfer is the lack of model's capacity to accommodate all languages simultaneously. this has led to increased interest in adapting multilingual lms to lrls and we discuss these in the following two settings. lrl adaptation using monolingual data for eleven languages outside mbert, wang et al. (2020) demonstrate that adding a new target language to mbert by simply extending the embedding layer with new weights results in better performing models when compared to bilingual-bert pre-training with english as the second language. pfeiffer et al. (2020c) adapt multilingual lms to the lrls and languages with scripts unseen during pre-training by learning new tokenizers for the unseen script and initializing their embedding matrix by leveraging the lexical overlap w.r.t. the languages seen during pre-training. adapter @xcite based frameworks like @xcite @xcite address the lack of model's capacity to accommodate multiple languages and establish the advantages of adding language-specific adapter modules in the bert model for accommodating lrls. these methods generally assume access to a fair amount of monolingual lrl data and do not exploit relatedness across languages explicitly. these methods provide complimentary gains to our method of directly exploiting language relatedness. ## low web-resource adaptation in relatelm problem statement and notations our goal is to augment an existing multilingual language model m, for example mbert, to learn representations for a new lrl l for which available monolingual corpus d l is limited. we are also told that the language to be added is related to another language r on which the model m is already pretrained, and is of comparatively higher resource. however, the script of d l may be distinct from the scripts of existing languages in m. in this section we present strategies for using this knowledge to we focus on the case where the rpl, lrl pairs are part of the indo-aryan language families where several levels of relatedness exist. our proposed approach, consists of three steps, viz., transliteration to rpl's script, pseudo translation, and adaptation through pre-training. we describe each of these steps below. figure 2 presents an overview of our approach. ## experiments we carry out the following experiments to evaluate relatelm's effectiveness in lrl adaptation: ## conclusion and future work we address the problem of adapting a pre-trained language model (lm) to a low web-resource language (lrl) with limited monolingual corpora. we propose relatelm, which explores relatedness between the lrl and a related prominent language (rpl) already present in the lm. relatelm exploits relatedness along two dimensions -script relatedness through transliteration, and sentence structure relatedness through pseudo translation. we focus on indic languages, which have hundreds of millions of speakers, but are understudied in the nlp community. our experiments provide evidence that relatelm is effective in adapting multilingual lms (such as mbert) to various lrls. also, relatelm is able to achieve zero-shot transfer with limited lrl data (20k documents) which is not surpassed even with 4x more data by existing baselines. together, our experiments establish that using a related language as pivot, along with data augmentation through transliteration and bilingual dictionary-based pseudo translation, can be an effective way of adapting an lm for lrls, and that this is more effective than direct training or pivoting through english. integrating relatelm with other complementary methods for adapting lms for lrls @xcite ) is something we plan to pursue next. we are hopeful that the idea of utilizing relatedness to adapt lms for lrls will be effective in adapting lms to lrls in other languages families, such as south-east asian and latin american languages. we leave that and exploring other forms of relatedness as fruitful avenues for future work. madhavi sardesai, jyoti pawar, shantaram walawalikar, and edna vaz. 2012. bis annotation standards with reference to konkani language. in proceedings of the 3rd workshop on south and southeast asian natural language processing, pages 145-152, mumbai, india. the coling 2012 organizing committee. haiyue song, raj dabre, zhuoyuan mao, fei cheng, sadao kurohashi, and eiichiro sumita. 2020. pretraining via leveraging assisting languages for neural machine translation. in proceedings of the 58th annual meeting of the association for computational linguistics: student research workshop, pages 279-285, online. association for computational linguistics. -rrb-rd # . , $ " ( ) : -'' ' rd pdt all, half: qt such: dm "default": qt wdt which, that : pr whatever: rp "default": pr dt some, every, both, all, another, a, an: qt this, these, the: dm those, that: pr "default": qt wrb how,wherever, when, where: pr whenever: rb why: rb "default" : pr