On the Transferability of Massively Multilingual Pretrained Models in the Pretext of the I ndo- A ryan and Tibeto-Burman Languages (2021)

Abstract: In recent times, machine translation models can learn to perform implicit bridging between language pairs never seen explicitly during training and showing that transfer learning helps for languages with constrained resources. This work investigates the low resource machine translation via transfer learning from multilingual pre-trained models i.e. mBART-50 and mT5-base in the pretext of Indo-Aryan (Assamese and Bengali) and Tibeto-Burman (Manipuri) languages via finetuning as a downstream task. Assamese and Manipuri were absent in the pretraining of both mBART-50 and the mT5 models. However, the experimental results attest that the finetuning from these pre-trained models surpasses the multilingual model trained from scratch.

## introduction recent years have witnessed the growing advances in the field of neural machine translation (nmt) specifically for the resource rich languages. however, nmt requires enormous amount of parallel data in order to have a decent translation system. on the other hand, the low resource languages lacks sufficient amount of parallel data, thus making the translation system far from the production level. meanwhile, monolingual data is readily available as compared to the parallel data and many works have been done to exploit it, most notably in a semi-supervised approach for data augmentation using self-training @xcite @xcite and backtranslation @xcite . however, these approaches are prone to generate erroneous translations due to the noisy synthetic data and often requires an iterative refinement procedure which is both resource intensive @xcite and time consuming process. unsupervised machine translation @xcite @xcite on the other hand uses only the monolingual data and do not require any parallel data which appears to be intimidating for a low resource scenario. additionally, the initial cross-lingual mapping between the two monolingual data requires a maximal amount of vocabulary overlaps which is crucial for a stronger cross-lingual mapping between the source and the target monolingual vector spaces. however, the vocabulary overlaps is maximised only when the two languages are closely related thus making the unsupervised machine translation approach unsuitable for the distant language pairs even if they have large amount of monolingual data @xcite . moreover, conventional unsupervised systems utilises iterative back-translation for the refinement purpose, thus the unsupervised methods are imposed with the issues of the back-translation (noisy translations and resource intensive). multilingual neural machine translation (mnmt) @xcite on the other hand supports the translation among multiple languages which has shown to be beneficial for low resource machine translation via the transfer of cross-linguistic information from the higher resource languages @xcite . this can be facilitated by transferring the trained parameters from a parent model to a child model @xcite @xcite or through a bridge or pivot language @xcite @xcite . however, mnmt can be further simplified by converting it into a single bilingual nmt by jointly training @xcite all the languages. furthermore, the jointly trained mnmt system is extended with 50 or more languages in a massively multilingual @xcite @xcite scenario which has shown to im-prove the low resource machine translation @xcite in the presence of the higher resource languages with the advantage of training a single nmt model instead of training separate bilingual models. however, training these massively multilingual models from scratch for every new languages is not feasible both in terms of time and the resource and has negative impact to the environment for training such enormous models which can be coped up via transfer learning where the downstream translation task can be simply finetuned from a large pre-trained model @xcite @xcite @xcite @xcite . primitive transfer learning in the nlp flourished with the pretrained word embedding vectors @xcite , followed by the pretrained encoder @xcite or decoders or pretraining the full seq2seq model @xcite . these multilingual pretrained models such as the mbart @xcite and the mt5 @xcite has shown to benefit the low resource machine translation during the downstream finetuning step. additionally, these pretrained models can be extended to even new languages @xcite which was absent during the pretraining process by simply resuming the training with the new language data with the pretrained model checkpoint as a finetuning step and sometimes increasing the bleu score also. in our premise, we make use of the mbart-50 @xcite and the mt5-base @xcite pretrained models for the english (en) to {assamese (asm), bengali (bn) and manipuri (mni)} translation in a one-to-many multilingual setup. all the three languages apart from english are the scheduled languages of india where assamese and bengali belong to the indo-aryan language family while manipuri is a tibeto-burman language and very few works have been reported in this language most notably (singh and @xcite @xcite @xcite @xcite . additionally, only the bengali language is present during the pretraining of both mbart-50 and the mt5-base models while assamese and manipuri were absent during the pretraining phase. hence, the finetuning process involves the transfer learning to totally unseen languages and this work investigates the effect of these pretrained models to the low resource translation task for these unseen languages. we also evaluate our performance on the wat-2021 multiindicmt foot_0 test set for english to bengali and flores-101 test set @xcite for the english to (bengali and assamese) ## multilingual neural machine translation multilingual nmt facilitates the translation between multiple languages via pivot based @xcite , transfer learning @xcite or through a jointly trained single nmt model @xcite . in this work, we utilise the jointly trained single multilingual nmt model. additionally, this single mnmt can be further divided into three types according to the mapping of the source and the target languages, many-to-one (m2o). in this setting, the model is trained to translate multiple source languages into a single target language. one-to-many (o2m). this mnmt model translates from a single source language to multiple target languages and many-to-many (m2m). here, translation between many source and many target languages is possible. moreover, as there are several target languages in the o2m and m2m, a target language tag is typically prepended at the beginning of the source sentence to specify the predicted target language. given k sentence pairs and l language pairs the training objective of an mnmt model is to maximise the log-likelihood over the whole parallel pairs {x (l,k) , y (l,k) } @xmath0,...,l) @xmath1,...,k l ) as: ## experimental results ## qualitative and error analysis 6.1 qualitative analysis a qualitative analysis in the form of sample input and output is also presented in in the first source sentence (source-1), o2m-s the phrase "mahakna pankhi" (he mentioned) twice thus degrading the fluency and the term "raamaayan has been wrongly generated as ''raamaadhan" (ramadan) which in turn detoriates the adequacy. similarly, there are several instances where o2m-s has generated erroneous words. on the other hand, mbart+o2m and mt5+o2m made a better translation as compared to the o2m-s in terms of both adequacy and fluency. however, mbart+o2m translated the source word in particular to "maruoina" (importantly) while mt5+o2m translated into the accurate word "akaknanaa" (in particular). although, the word order has been displaced even after generating the correct word hence the automatic scores which depends upon the exact word overlapping gets penalised. the second (source-2) and the third source (source-3) sentences are challenging ones. the source-2 has complex contextual dependencies which is evident with the struggle to establish the correct dependency relations in the translations of the o2m-s and mbart+o2m while, mt5+o2m is the only system which can successfully establish the meaning of the source sentence along with a fluent translation. apart from this, the source-2 contains numerical values 17 and 3 which is successfully translated by all the three systems. another challenging instance is the presence of abbreviations in the source sentence and the valid english terms which exists as in the target language. this phenomenon is illustrated in source-3 translation where all the three systems generated the source word components as "kamponent" (component) instead of "masa" (branch; part; component). thus, even though the o2m-s and mbart+o2m generated the correct translation due to token mismatch between the reference and the translations, the bleu score is penalised. in the same source-3 sentence, the abbreviation of pmssy is directly copied in the outputs of o2m-s and mbart+o2m which exists as "pi. em. ess. ess. yai." (pmssy) in the reference thus degrading the bleu score. mt5+o2m on the other hand generated the extra three extra s in the abbreviations and excluded y. ## conclusion in this work, we report the findings of the investigation of low resource machine translation via transfer learning from multilingual pretrained models i.e. mbart-50 and mt5-base in the pretext of indo-aryan (assamese and bengali) and tibeto-burman (manipuri) languages. it is found that the transfer learning from these pretrained multilingual models outperforms the one-to-many model trained from the scratch across all the translation directions in all the test sets thus suggesting the strong transfer of interliguistic information to the downstream finetuning tasks even for the languages absent during the pretraining step. furthermore, the superiority of finetuning from these pretrained models than the indicbart for the english to bengali translation using the wat-2021 test set suggests that a stronger transfer learning is possible even without linguistic relatedness during the pretraining step or due to the negative transfer of information between the incompatible languages during the multilingual finetuning of indicbart. finally, we plan to explore more on the negative transfer and the linguistic relatedness avenue in future focusing on indian languages.