Entropy-guided Vocabulary Augmentation of Multilingual Language Models for Low-resource Tasks (2023)

Abstract: Multilingual language models (MLLMs) like mBERTpromise to extend the benefits of NLP research to low-resource languages (LRLs). However, LRL words are under-represented in the wordpiece/subword vocabularies of MLLMs. This leads to many LRL words getting replaced by UNK, or concatenated from morphologically unrelated wordpieces, leading to low task accuracy. (Pre)-training MLLMs after including LRL documents is resource-intensive in terms of both human inputs and computational resources. In response, we propose EVALM (entropy-based vocabulary augmented language model), which uses a new task-cognizant measurement to detect the most vulnerable LRL words, whose wordpiece segmentations are undesirable. EVALM then provides reasonable initializations of their embeddings, followed by limited fine-tuning using the small LRL task corpus. Our experiments show significant performance improvements and also some surprising limits to such vocabulary augmentation strategies in various classification tasks for multiple diverse LRLs, as well as code-mixed texts. We will release the code and data to enable further research.

## introduction it is common practice to start with a multilingual language model (mllm) like mbert foot_1 or xlm-r @xcite , which has been pre-trained with large multilingual corpora, and fine-tune the mllm for diverse downstream tasks. although mllms support many low-resource languages (lrls), closer inspection of these mllms reveals that the portion of vocabulary allotted to lrls can be orders of magnitude smaller than that allotted to high-resource languages (hrls) such as english ( in this work, we address the setting where a mllm (that is presumably deficient in lrl coverage) must be minimally fine-tuned after modest modification to its wordpiece vocabulary, guided by specific lrl tasks. we design a measure of damage to an lrl word, caused by wordpiece fragmentation, based on a suitably defined notion of entropy of the word and constituent wordpieces, with respect to the lrl task. this measure then guides the selection of lrl words with which the vocabulary should be augmented. subsequently, we propose various ways to initialize the embeddings of these newly-introduced words, including using information from the lrl itself, to 'importing' information from hrls. we call the resulting system evalm (entropy-based vocabulary augmented language model). we study the effect of evalm on an existing mllm during the fine-tuning stage for various downstream classification tasks covering multiple lrls and also a code-mixed language. our study shows that, for most of the datasets, evalm's vocabulary augmentation strategy helps improve lrl task performance by greater margins than recent best practices @xcite . a detailed analysis of successes and failures delineates the perimeter of evalm's capabilities and guides our design choices. ## related work continued pre-training @xcite with or without vocabulary augmentation of existing lms like monolingual bert, multilingual bert (mbert), xlm-r, etc., proves beneficial for improving domain and languagespecific performances over various tasks. some works (ruzzetti et al., @xcite focus on rare/oov words. @xcite propose an embedding generator module in the pretrain-finetune pipeline to resolve vocabulary gaps. adaptors @xcite are also showing promising outcomes in lrl modeling. chung et al. (2020) explore multilingual vocabulary generation from language clusters. @xcite transfer english lms to new languages without expensive computation. @xcite propose a simple algorithm which modifies the tokenization process to preserve the morphological structure of a word. others @xcite focus on embedding initialization for newly added vocabulary words which are word fragments, which is also among our concerns. ## our system: evalm evalm has three key components. the purpose of the first component (section 3.1) is to identify (based on only the train fold) a subset of vulnerable lrl words whose assembly from wordpieces is likely to distort the embedding information made available to lrl labeling tasks. the second component (section 3.2) comprises various possible algorithm 1 lrl vocabulary selection. ## conclusion we have proposed a simple and effective method to augment an mllm wordpiece vocabulary with lrl words that are important for lrl classification tasks. our study, involving several indian languages, shows a consistent positive impact of vocabulary augmentation and fine-tuning. we find more augmentation does not guarantee performance improvement, and different embedding initialization fails to show significant performance differences among themselves. we also show that regularization is crucial to prevent overfitting new lrl word embeddings during fine-tuning. we have limited the augmentation to whole lrl words, and a judicious selection of lrl wordpieces may improve performance. we also want to extend to other target tasks (especially language generation) and a more diverse set of lrls.