Adapting Large Multilingual Machine Translation Models to Unseen Low Resource Languages via Vocabulary Substitution and Neuron Selection (2022)

Abstract: We propose a method to adapt large Multilingual Machine Translation models to a low resource language (LRL) that was not included during the pre-training/training phases. We use neuron-ranking analysis to select neurons that are most influential to the high resource language (HRL) and fine-tune only this subset of the deep neural network’s neurons. We experiment with three mechanisms to compute such ranking. To allow for the potential difference in writing scripts between the HRL and LRL we utilize an alignment model to substitute HRL elements of the predefined vocab with appropriate LRL ones. Our method improves on both zero-shot and the stronger baseline of directly fine-tuning the model on the low-resource data by 3 BLEU points in X -> E and 1.6 points in E -> X.We also show that as we simulate smaller data amounts, the gap between our method and direct fine-tuning continues to widen.

## introduction large multilingual machine translation models have been achieving state-of-the-art (sota) performances on machine translation task (mt) in recent years @xcite . these models have also been shown to achieve gains on low-resource languages (lrl), all be it at the cost of slight regressions in the performances of high-resource language (hrl), they have even enabled translation on language pairs with zero parallel data @xcite . in this work we define lrls as languages that have less than 1m parallel training sentences. whereas hrls have a training set in the order of tens of millions of parallel sentences. adapting these models to previously unseen lrls can be challenging. re-training these models can be expensive in terms of money and time. this is due to the large number of parameters and training data required to train these models, thus the need for powerful and expensive gpus for extended periods of time. moreover the training mechanism of standard modern tokenizers (most notably sentencepiece @xcite ) which assumes monolingual data of all languages that the model is to support would be present before training the model. this can lead to over-segmentation and high oov-rates (out of vocabulary) of lrl sentences when compared to hrl sentences. this in turn leads to longer sequences, resulting in hindering the learning. we conducted our own analysis on the mbart50 @xcite vocabulary set to demonstrate this (see section 4.3). to mitigate this effect we use a slightly modified version of vocabulary substitution @xcite that uses an hrl to lrl alignment model as a guide. furthermore given that lrls by definition have a limited number of parallel training data samples, fine-tuning multilingual models can be especially tricky considering how easy it would be for the model to over-fit on the training data and/or be especially sensitive to the noisy data samples. while previous approaches have tried to augment lrl data via the hrl training dataset, we hypothesised that being selective regarding which neurons to fine-tune -in other words allowing the loss function's gradients to fall only into a certain subset of the network's neurons (hence only actually changing the weights of said subset) -would act as regularization technique thus mitigating the effect of over-fitting and noise in the lrl training set while maintaining or amplifying the gains originating from cross-lingual sharing with the hrl. we experiment with three techniques to compute the aforementioned sub-net: gradient analysis : we compute the gradient of the output w.r.t each individual neuron, gradients across different time-steps are aggregated (we experiment with multiple aggregation functions). the gradients are then aggregated again across multiple sentences we consider the magnitude of the final outcome to be the neuron's importance score. activation magnitude : we track the magnitude of the activation (output) of each neuron across multiple time-steps and multiple sentences. the activations go through the same two-layered aggregation procedure as with gradient analysis. lass : first proposed in @xcite . in order to compute the importance of neurons to a certain language l, we start by fine-tuning the model on l's data. then we sort neurons by the absolute difference in the weights of the neuron between the original and fine-tuned models. ## background many of the recent breakthroughs in deep natural language processing (nlp) models have relied heavily on growing the model in depth and number of parameters. this has shown to significantly improve model performance on many few-shot and zero-shot nlp tasks, with low resource machine translation being one of those tasks @xcite . this however remains largely limited by the availability of a large quantity of resources for pre-training and/or fine-tuning such models. this leads us to believe that there is still benefit from the study and improvement of methods to adapt existing dense models to a new low-resource language. @xcite explores adding an unseen language by training monolingual adapters. adapters are small components that are trained traditionally while the rest of the network has been frozen @xcite . while theoretically this requires no parallel data, it can easily be adapted to our setup by fine-tuning both the source language encoder adapter and the target language decoder adapter on the quantity of parallel data that is available. we compare against this approach on section 4. multiple approaches study how to add previously unseen languages while trying to maintain performance on the rest of the languages such as @xcite and @xcite . in this work on the other hand we are more interested in maximizing the performance of the model on the new lrl regardless of it's effect on other language-pairs. the assumption being the mnmt model after being tuned towards the lrl can later be distilled into a more deployment-friendly architecture @xcite . previous work has shown that finetuning on a mix of hrl and lrl to be beneficial @xcite . other approaches relied on pivoting on the hrl, the hypothesis being that translation from hrl to lrl should be an easier task @xcite . we evaluate against this in section 4. all of these approaches however treat the multilingual model as a black box, and as far as we know no attempt has been made at being selective as to targeting certain neurons during fine-tuning. ## neuron selection and vocab substitution we describe our method in this section. the goal is to adapt a multilingual model to low resource, previously unseen language (lrl) by leveraging the model's knowledge of a similar high resource language (hrl). our method can be broken down into three (possibly four) main stages: in the reminder of this section we explain in some detail each phase. ## performance evaluation we chose to adapt mbart50 @xcite to tajik language (lrl) with persian being its closest hrl language. we collect the training data for both languages from opus foot_1 . tajik and persian pose an especially interesting challenge since they are mutually intelligible, but written in different scripts (cyrillic and perso-arabic respectively). for either language pair we use flores-dev as validation and flores-devtest for evaluation @xcite . all of the computations were performed on a single tesla t4 gpu with 16 gb of ram. to compare our results to @xcite we train a p ersian ↔ t ajok mt model. we collect the training data from opus and use flores "dev" and "devtest" as validation and test sets respectively. ## conclusion and future work in this section we discuss the results detailed in the previous section and propose possible extension to the analysis described throughout the paper. as detailed in for future work we propose extending this analysis horizontally by experimenting with more than one hrl-lrl, and vertically by applying the analysis on other available mnmt. we find that m2m might be a good candidate for this, and is especially interesting since it already has language-dependant parameters. another interesting area of research would be how to integrate self-supervised learning into this setup. specifically how will our method fair given zero parallel data but assuming the abundance of monolingual data.