Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation (2022)

Abstract: The performance of multilingual pretrained models is highly dependent on the availability of monolingual or parallel text present in a target language. Thus, the majority of the world’s languages cannot benefit from recent progress in NLP as they have no or limited textual data. To expand possibilities of using NLP technology in these under-represented languages, we systematically study strategies that relax the reliance on conventional language resources through the use of bilingual lexicons, an alternative resource with much better language coverage. We analyze different strategies to synthesize textual or labeled data using lexicons, and how this data can be combined with monolingual or parallel text when available. For 19 under-represented languages across 3 tasks, our methods lead to consistent improvements of up to 5 and 15 points with and without extra monolingual text respectively. Overall, our study highlights how NLP methods can be adapted to thousands more languages that are under-served by current technology.

## introduction multilingual pretrained models @xcite @xcite have become an essential method for cross-lingual transfer on a variety of nlp tasks @xcite . these models can be finetuned on annotated data of a down-stream task in a high-resource language, often english, and then the resulting model is applied to other languages. this paradigm is supposed to benefit under-represented languages that do not have annotated data. however, recent studies have found that the cross-lingual transfer performance of a language is highly contingent on the availability of monolingual data in the language during pretraining @xcite . languages with more 1 code and data are available at: https: //github.com/cindyxinyiwang/ expand-via-lexicon-based-adaptation. monolingual data tend to have better performance while languages not present during pretraining significantly lag behind. several works propose methods to adapt the pretrained multilingual models to low-resource languages, but these generally involve continued training using monolingual text from these languages @xcite @xcite . therefore, the performance of these methods is still constrained by the amount of monolingual or parallel text available, making it difficult for languages with little or no textual data to benefit from the progress in pretrained models. @xcite indeed argue that unsupervised pretraining makes the 'resource-poor poorer'. fig. 1 plots the language coverage of multilingual bert (mbert; @xcite , a widely used pre-trained model, and several commonly used textual data sources. 2 among the 7,000 languages in the world, mbert only covers about 1% of the languages while wikipedia and com-moncrawl, the two most common resources used for pretraining and adaptation, only contain textual data from 4% of the languages (often in quite small quantities, partially because language ids are difficult to obtain for low-resource languages @xcite ). @xcite show that continued pretraining of multilingual models on a small amount of bible data can significantly improve the performance of uncovered languages. although the bible has much better language coverage of 23%, its relatively small data size and constrained domain limits its utility (see § 6)-and 70% of the world's languages do not even have this resource. the failure of technology to adapt to these situations raises grave concerns regarding the fairness of allocation of any benefit that may be conferred by nlp to speakers of these languages @xcite . on the other hand, linguists have been studying and documenting under-represented languages for years in a variety of formats @xcite . among these, bilingual lexicons or word lists are usually one of the first products of language documentation, and thus have much better coverage of the worlds' languages than easily accessible monolingual text, as shown in fig. 1 . there are also ongoing efforts to create these word lists for even more languages through methodologies such as "rapid word collection" @xcite , which can create an extensive lexicon for a new language in a number of days. as bird (2020) notes: after centuries of colonisation, missionary endeavours, and linguistic fieldwork, all languages have been identified and classified. there is always a wordlist. . . . in short, we do not need to "discover" the language ex nihilo (l1 acquisition) but to leverage the available resources (l2 acquisition). however, there are few efforts on understanding the best strategy to utilize this valuable resource for adapting pretrained language models. bilingual lexicons have been used to synthesize bilingual data for learning cross-lingual word embeddings @xcite and task data for ner via word-to-word translation @xcite , but both approaches precede the adoption of pre-trained multilingual lms. @xcite use lexicons to synthesize monolingual data for adapting lms, but their experimentation is limited to several indian languages and no attempt was made to synthesize downstream task data while @xcite argue that bilingual lexicons may hurt performance. in this paper, we conduct a systematic study of strategies to leverage this relatively under-studied resource of bilingual lexicons to adapt pretrained multilingual models to languages with little or no monolingual data. utilizing lexicons from an open-source database, we create synthetic data for both continued pretraining and downstream task fine-tuning via word-to-word translation. empirical results on 19 under-represented languages on 3 different tasks demonstrate that using synthetic data leads to significant improvements on all tasks (fig. 2 ), and that the best strategy depends on the availability of monolingual data ( § 5, § 6). we further investigate methods that improve the quality of the synthetic data through a small amount of parallel data or by model distillation. ## background we focus on the cross-lingual transfer setting where the goal is to maximize performance on a downstream task in a target language t . due to the frequent unavailability of labeled data in the target language, a pretrained multilingual model m is typically fine-tuned on labeled data in the downstream task d s @xmath0, y s i )} n @xmath1, y s i is the label, and n is the number of labeled examples. the fine-tuned model is then directly applied to task data d t @xmath2, y t i } i in language t at test time. 3 the performance on the target language t can often be improved by further adaptation of the pretrained model. ## adapting to under-represented languages using lexicons since the main bottleneck of adapting to underrepresented languages is the lack of text, we adopt a data augmentation framework (illustrated in fig. 3 ) that leverages bilingual lexicons, which are available for a much larger number of languages. ## general experimental setting in this section, we outline the tasks and data setting used by all experiments. we will then introduce the adaptation methods and results for the no-text setting in § 5 and the few-text setting in § 6. ## no-text setting we analyze the results of the following adaptation methods for the setting where we do not have any monolingual data. pseudo mlm the mbert model is trained on the pseudo monolingual data using the mlm objective. we train the model for 5k steps for the ner tasks and 10k steps for the pos tagging and parsing tasks. ## few-text setting we test same adaptation methods introduced in § 5 for the few-text setting where we have a small amount of gold data. first we introduce the additional data and adaptation methods for this setting. ## analyses performance with xlm-r we mainly use mbert because it has competitive performance for under-represented languages and it is more computationally efficient due to the smaller size. here we verify our methods have the same trend when used on a different model xlm-r @xcite . we focus on a subset of languages in the pos tagging task for the few-text setting and the results are in tab. 4. we use the smaller xlm-r base for efficiency, and compare to the best result in prior work, which uses xlm-r large @xcite . tab. 4 shows that our baseline is comparable or better than prior work. similar to the conclusion in § 6, pseudo mlm is the competitive strategy that brings significant improvements over prior work. while adding pseudo trans-train to pseudo mlm does not help, using label distillation further improves the performance. effect of baseline performance using pseudo data might be especially effective for languages with lower performance. we plot the improvement of different languages over the baseline in fig. 5 , where languages are arranged with increasing baseline performance from left to right. we mainly plot pseudo mlm and pseudo trans-train for simplicity. fig. 5 shows that for both resource settings, lower performing languages on the left tend to have more performance improvement by using pseudo data. using nmt model to synthesize data one problem with the pseudo data synthesized using word-to-word translation is that it cannot capture the correct word order or syntactic structure in the target language. if we have a good nmt system that translates english into the target language, we might be able to get more natural pseudo monolingual data by translating the english sentences to the target language. since the target languages we consider are usually not supported by popular translation services, we train our own nmt system by fine-tuning an open sourced many-to-many nmt model on the bible parallel data from english to the target language (details in § a.2). instead of creating pseudo monolingual data using the lexicon, we can simply use the fine-tuned nmt model to translate english monolingual data into the target language. the results of using nmt as opposed to lexicon for pseudo mlm on all four tasks can be found in tab. 5. unfortunately, nmt is consistently worse than word-to-word translation using lexicons. we find that the translated monolingual data tend to have repeated words and phrases that are common in the bible data, although the source sentence is from wikipedia. this is because the nmt model overfits to the bible data, and it fails to generate good translation for monolingual data from a different domain such as wikipedia. ## related work several methods have been proposed to adapt pretrained language models to a target language. most of them rely on mlm training using monolingual data in the target languages @xcite @xcite @xcite , competitive nmt systems trained on parallel data @xcite , or some amount of labeled data in the target languages @xcite . these methods cannot be easily extended to low-resource languages with no or limited amount of monolingual data, which account for more than 80% of the world's languages @xcite . bilingual lexicons have been commonly used for learning cross-lingual word embeddings @xcite . among these, some work uses lexicons to synthesize pseudo bilingual @xcite or pseudo multilingual corpora @xcite . @xcite propose to synthesize task data for ner using bilingual lexicons. more recently, @xcite synthesize monolingual data in indian languages for adapting pretrained language models via mlm. @xcite argue that using bilingual lexicons for alignment hurts performance compared to word-level alignment based on parallel corpora. such parallel corpora, however, are not available for truly under-represented languages. reid and artetxe (2021) employ a dictionary denoising objective where a word is replaced with its translation into a random language with a certain probability. this can be seen as text-to-text variant of our approach applied to multilingual pre-training. none of the above works provide a systematic study of methods that utilize lexicons and limited data resources for adapting pretrained language models to languages with no or limited text. ## conclusion and discussion we propose a pipeline that leverages bilingual lexicons, an under-studied resource with much better language coverage than conventional data, to adapt pretrained multilingual models to underrepresented languages. through comprehensive studies, we find that using synthetic data can significantly boost the performance of these languages while the best method depends on the data availability. our results show that we can make concrete progress towards including under-represented languages into the development of nlp systems by utilizing alternative data sources. our work also has some limitations. since we focus on different methods of using lexicons, we restrict experiments to languages in latin script and only use english as the source language for simplicity. future work could explore the effect of using different source languages and combining transliteration @xcite or vocabulary extension @xcite with lexicon-based data augmentation for languages in other scripts. we also did not test the data augmentation methods on higher-resourced languages as mlm fine-tuning and translate-train are already effective in that setting and our main goal is to support the languages with little textual data. nonetheless, it would be interesting to examine whether our methods can deliver gains for high-resource languages, especially for test data in specialized domains. we point to the following future directions: first, phrases instead of single word entries could be used to create pseudo data. second, additional lexicons beyond panlex could be leveraged. 6 third, more effort could be spent on digitizing both existing monolingual data such as books (gref, 2016) and lexicons into a format easily accessible by nlp practitioners. although panlex already covers over 5000 languages, some language varieties have only as little as 10 words in the database, while there exist many paper dictionaries that could be digitized through technologies such as ocr (rijhwani et al., 2020). 7 lexicon collection is also relatively fast, which could be a more cost effective strategy to significantly boost the performance of many languages without lexicons. finally, the quality of synthetic data could be improved by incorporating morphology. however, we find that there is virtually no existing morphological analysis data or toolkits for the languages we consider. future work could aim to improve the morphological analysis of these low-resource languages.