A Study of Multilingual versus Meta-Learning for Language Model Pre-Training for Adaptation to Unseen Low Resource Languages (2023)

Abstract: In this paper, we compare two approaches to train a multilingual language model: (i) simple multilingual learning using data-mixing, and (ii) meta-learning. We examine the performance of these models by extending them to unseen language pairs and further finetune them for the task of unsupervised NMT. We perform several experiments with varying amounts of data and give a comparative analysis of the approaches. We observe that both approaches give a comparable performance, and meta-learning gives slightly better results in a few cases of low amounts of data. For Oriya-Punjabi language pair, meta-learning performs better than multilingual learning when using 2M, and 3M sentences.

## introduction neural machine translation (nmt) works well with large amounts of parallel data @xcite . for many language pairs, such data is not available. unsupervised nmt has achieved performance comparable to supervised nmt for a few european language pairs; however, it only works well for languages that have a good amount of monolingual data available. the current state-of-the-art approaches of unsupervised nmt have a language model pretraining phase and a finetuning phase based on iterative back-translation @xcite @xcite . translation involving low-resource languages (for which monolingual data is also scarce) is very difficult. the current state-of-the-art approaches of unsupervised nmt perform poorly for such language pairs @xcite . to utilize the benefit of high-resource language pairs, multilingual language model pre-training has been utilized @xcite . @xcite proposed to train a language model for high-resource language pair and then use it as initialization for the low-resource language pair. @xcite explored the use of meta-learning after pretraining with high-resource languages for low resource natural language understanding tasks. they claim that multi-task learning might favor high-resource tasks, while meta-learning learns a good initialization that can be adapted to any task with a small number of iterations. in this paper, we use a meta-learning framework for multilingual language model pretraining and compare it with a multilingual learning paradigm based on data-mixing and finetuning it for unseen language pairs. we use these finetuned models to further training for the task of unsupervised nmt. specifically, we utilize maml (model agnostic meta-learning) @xcite which is a meta-learning algorithm based on gradient descent and is used to get good generalizations for multiple tasks. when using meta-learning, each language is considered a task in the pretraining phase. our goal is to find a method to efficiently learn parameters in a shared parameter space across multiple languages in the language model pretraining, which works as good initialization for the language model training for unseen language pairs and improves the performance of unsupervised nmt. a good pretrained multilingual language model should be able to adjust to newer language pairs (unseen languages) using a limited amount of training data. our contributions are: ## approach we compare two multilingual language model pretraining approaches: (i) multilingual learning based on simple data mixing and (ii) other based on a meta-learning framework. we try to find a good set of initialization for language model pretraining for unseen language pairs using many high-resource languages. in multilingual learning, the training simply iterates between different languages. for meta-learning, we utilize maml together with @xcite objective to train a multilingual language model. the main aim of maml is to find a good initialization from which a target task learning requires fewer iterations. it uses many other source tasks related to the target task to learn this initialization. we try to meta-learn using the source tasks and then continue to learn for the target tasks. this process is different than a simple multilingual learning framework. algorithm 1 shows the training algorithm for the meta-learning framework. we extend both the models to finetune them for unseen language pairs and use the vocabulary extension method proposed in @xcite to extend the vocabulary of the multilingual model. ## experiments we experiment with hindi, bengali, gujarati as our high resource languages to train a multilingual model using masked sequence to sequence pretraining objective. we use oriya-punjabi update θ as per maml objective as per equation 1 8: end while and assamese-nepali as our unseen language pairs. the details of the data are given in section 4.1. ## discussion pretrained multilingual models help in improving the performance for unseen languages, which is clear from when we utilize full available data of oriya and punjabi, meta-learning performs similar to multilingual learning. but when we use 0.5m sentences, multilingual learning is working better than meta-learning for or-pa. for as-ne multilingual learning and meta-learning both give similar performance. for or-pa, after the language model pretraining phase is complete for the unseen language pair, the cross-lingual perplexity is higher for meta-learning than the multilingual model but the bleu score is better, which indicates that fluency is not getting better but the translation is getting improved indicating better learning of shared representations. we also observe that the ratio of source words is 3.27% for multilingual and 4.27% for meta-learning when experimenting with 2m sentences for or to pa translation even without finetuning it for iterative back-translation. ## conclusion and future work in this paper, we perform a comparison of two approaches to train a multilingual language model: (i) simple multilingual learning, and (ii) meta-learning. we conduct experiments to extend these models for unseen language-pair and then finetune them for unsupervised nmt to compare the performance. we observe that both approaches give a comparable performance. in a few cases of low amounts of data, meta-learning gives slightly better results. in the future, we would like to explore the performance of both approaches to train the multilingual language model for other tasks.