Improving Low-Resource Languages in Pre-Trained Multilingual Language Models (2022)

Abstract: Pre-trained multilingual language models are the foundation of many NLP approaches, including cross-lingual transfer solutions. However, languages with small available monolingual corpora are often not well-supported by these models leading to poor performance. We propose an unsupervised approach to improve the cross-lingual representations of low-resource languages by bootstrapping word translation pairs from monolingual corpora and using them to improve language alignment in pre-trained language models. We perform experiments on nine languages, using contextual word retrieval and zero-shot named entity recognition to measure both intrinsic cross-lingual word representation quality and downstream task performance, showing improvements on both tasks. Our results show that it is possible to improve pre-trained multilingual language models by relying only on non-parallel resources.

## introduction pre-trained language models (lms) have replaced static word embeddings, such as word2vec, fast-text or glove @xcite @xcite , due to their superior contextualized representations. approaches such as mbert @xcite or xlm-r @xcite ) are trained on multiple languages simultaneously resulting in multilingual models which can be used for various cross-lingual transfer learning tasks @xcite @xcite . however, multilingual lms mainly focus on high resource languages, e.g., mbert supports the top 104 languages based on wikipedia sizes, while xlm-r supports the top 100 based on common-crawl data. additionally, many of these languages are underrepresented leading to low model performance in both monolingual and cross-lingual setups. due to the small data sizes of low-resource languages, subword tokenizers trained jointly on multiple languages tend to over-split the tokens of such languages and lms are not able to learn good quality representations for them. recent work have shown that pre-trained lms can be improved on low-resource languages using vocabulary extension and model fine-tuning @xcite . the cross-lingual quality of lms can further be improved by learning an additional alignment of language pairs @xcite or fine-tuning the whole model @xcite . however these methods require crosslingual data, often in the form of word aligned parallel sentences, to improve cross-linguality which is often missing for low-resource languages. in this work we propose an unsupervised approach to improve the language support of lowresource languages without any parallel data. relying on the initial cross-lingual quality of mbert we mine word translation pairs from monolingual data of the source and target language pairs by leveraging contextualized cross-lingual word representations @xcite . more precisely, we build ccwrs for each token in the source and target corpora and look for the most similar token pairs by calculating their cosine similarity. even though our approach does not rely on parallel corpora, we show that there are enough sentences with similar contexts (topics) containing at least one word translation pair that are detected by our mining approach. we then use the ccwrs of the mined word pairs to make their representations more similar, this way aligning source and target pairs. we use dedicated linear layers for both of the languages of the considered language pair to learn the alignment and we keep the lm's core frozen. an important contrast with previous work @xcite @xcite ) is that we mine word pairs from sentences of similar contexts, while they were only able to extract them from parallel sentences, which are often not available for low resource languages. despite this, we show that ccwrs can be improved using our mined word pairs. we conduct experiments on nine low-resource languages and test on two tasks: contextual crosslingual word retrieval @xcite and zeroshot named entity recognition @xcite . our results show improved ccwrs for each of the languages and improved ner f 1 scores for eight out of the nine languages. our analysis reveals our approach to be robust even when we mine noisy word pairs, because it benefits more from a larger quantity than a better quality of bootstrapped training set. additionally, we experiment with vocabulary extension techniques @xcite in case of languages using scripts other than latin or cyrillic, since the tokenizers of lms tend to oversplit the text of such languages. this step improves ccwr quality of the initial lms which gives a further boost to our bootstrapping method, leading to the best performance on these languages. finally, we run preliminary experiments on further four very low-resource languages that were not used for lm pre-training and show that the initial lm quality on such unseen languages is very low, which our approach can marginally improve. our implementation is publicly available. foot_0 ## related work pre-trained lms @xcite provide the core of many nlp solutions. they are trained using accessible monolingual corpora of multiple languages resulting in multilingual lms allowing them to be used in zero-shot transfer setups mitigating the issues of missing downstream task training data for many low-resource languages @xcite @xcite . although no parallel data is used, these models show remarkable cross-lingual quality, i.e., words with similar meaning in different languages are represented similarly by the models. previous work investigated the reasons of this phenomenon. @xcite found that the structural similarity of languages is an important factor, while both @xcite and @xcite showed that a shared vocabulary is not necessary. dufter and schütze (2020) identified essential elements for multilinguality, such as shared special tokens or comparable training corpora. on the other hand, multilingual lms are less effective on low-resource languages. it was shown that a fixed sized model can only support up to a certain number of languages efficiently, while adding more languages deteriorates its performance @xcite . the small size of the available monolingual data for low-resource languages decreases model performance further @xcite . additionally, data size imbalance of the used languages leads to an imbalanced subword vocabulary as well @xcite . to mitigate tokenization issues @xcite extend the vocabulary of pretrained lms with language specific tokens, while @xcite propose to learn language specific embeddings for low-resource languages. they show that with better subword vocabulary and model fine-tuning the model's performance can be improved on low-resource languages. we also rely on these techniques in our work. to improve the cross-lingual quality of lms various authors proposed steps on top of model pretraining. based on embedding mapping approaches @xcite it was shown that representations of monolingual lms can also be aligned @xcite . in addition, mapping approaches can be applied to the representations of multilingual lms as well for further improvements @xcite @xcite . in contrast to the above approaches, @xcite proposes a word alignment based objective function that finetunes the whole model in order to build more similar token representations for the aligned word pairs, while @xcite introduced the denoising word alignment task and @xcite relied on a contrastive alignment objective to encourage better cross-lingual performance. others leverage cross-lingual training signals already in the model pre-training phase @xcite . however, the above approaches require parallel data which is not available for many low-resource languages. in contrast, we show that lms can be improved by using only monolingual corpora where source and target language sentences with similar contexts can be found. ## unsupervised language alignment in order to improve the cross-lingual quality of pretrained multilingual lms we mine word translation pairs using only monolingual corpora of the source and target languages. we rely on the token representations given by lms to look for the most similar •,• ) of source (w s i,j ) and target (w t k,l ) language words in the input sentences followed by mining translation pairs (d m ) and making their representation more similar at the end. we keep the core lm parameters frozen ( ) and train only the linear layers on top of it. the process is repeated for multiple iterations. ## conclusions low-resource languages are underrepresented in pre-trained multilingual lms. in contrast to previous work using a parallel dataset to improve the language support of low-resource languages, we presented an unsupervised method to align language pairs by relying only on monolingual corpora. we showed that word translation pairs can be extracted from non-parallel sentence pairs by leveraging the cross-lingual contextualized representations of words which in turn can be used to align the vector spaces of languages. we tested our approach on the intrinsic contextual word retrieval and the downstream named entity recognition tasks. our results showed improved cross-lingual quality of the fine-tuned lms. our analysis revealed that the quantity of mined word pairs matters over their quality and that the vocabulary extension method is important for performance boost in case of the lowest resource languages. as future work we aim at leveraging easily accessible cross-lingual resources for better unseen language support.