Multiple Pivot Languages and Strategic Decoder Initialization Helps Neural Machine Translation (2022)

Abstract: In machine translation, a pivot language can be used to assist the source to target translation model. In pivot-based transfer learning, the source to pivot and the pivot to target models are used to improve the performance of the source to target model. This technique works best when both source-pivot and pivot-target are high resource language pairs and the source-target is a low resource language pair. But in some cases, such as Indic languages, the pivot to target language pair is not a high resource one. To overcome this limitation, we use multiple related languages as pivot languages to assist the source to target model. We show that using multiple pivot languages gives 2.03 BLEU and 3.05 chrF score improvement over the baseline model. We show that strategic decoder initialization while performing pivot-based transfer learning with multiple pivot languages gives a 3.67 BLEU and 5.94 chrF score improvement over the baseline model.

## introduction neural machine translation (nmt) models have made huge improvements in the performance of machine translation systems. but nmt models are data hungry. nmt models require huge amounts of parallel corpus for training. to overcome this limitation and improve the performance of the source to target nmt model, the resources of a pivot language can be used. @xcite used a parent model trained on a high resource language pair to initialize the parameters of the child model, which is then trained on a low resource language pair. @xcite introduced pivot-based transfer learning techniques to utilize the resources of the pivot language. in pivot-based transfer learning techniques, the source to pivot and the pivot to target models are used to initialize the source to target nmt model. the pivot-based transfer learning techniques work best when both the source to pivot and the pivot to target language pairs are relatively high resource language pairs. it also helps if the pivot language is related to the source or target language, to utilize language relatedness @xcite . in the task of translation from english to an indic language, another indic language can be used as a pivot language, as indic languages are related. but in such a setting, the pivot to target language pair may not be a high resource language pair. in the task of english to marathi translation, hindi can be used as a pivot language, as hindi is a related language to marathi. the english-hindi language pair is a relatively high resource language pair, but the hindi-marathi language pair is not a high resource language pair. to overcome this shortcoming, we use multiple indic languages as pivot languages to assist the source to target nmt model. transformer @xcite model has shown state-of-the-art results for various natural language processing tasks, including machine translation. in a transformer based nmt model, the decoder consists of two modules, self-attention, and cross attention. the self-attention layer works only with the target side language, but the cross attention layer works with the source and target side languages. we experiment with various techniques to initialize the modules of the decoder. ## approaches we first discuss the approach to using multiple pivot languages to assist the source to target model. then we discuss the various techniques to initialize the decoder of the source to target model in pivotbased transfer learning. ## experimental setup in this section, we discuss the setup of the various experiments that we performed. we use byte pair encoding (bpe) @xcite split words into subwords. we use the fairseq @xcite library to perform all the experiments. ## results and analysis we use bleu @xcite and chrf @xcite scores to evaluate the performance of all the models. we used the sacreblue foot_0 implementation for computing the bleu scores and the nltk foot_1 implementation for computing the chrf scores. from the results, we can observe that models using multiple pivot languages outperform models using only hindi as a pivot language. the best model using only a single pivot language achieves a bleu score of 11.05 and chrf score of 41.63. the model using multiple pivot languages improves the bleu score by 1.64 points to 12.69 and chrf score by 2.89 points to 44.52. this shows that using multiple pivot languages improves the performance of the source to target nmt models. we can observe that randomly initializing the decoder cross attention module of the source to target model gives better or comparable performance over direct pivoting. initializing the decoder cross attention module of the source to target model with the decoder cross attention module of the source to pivot model gives the best performance. in multi pivot languages setting, the direct pivoting technique achieves a bleu score of 11.95 and chrf score of 43.82 and the strategic decoder initialization technique improves the bleu score by 0.74 bleu points to 12.69 and the chrf score by 0.7 points to 44.52. ## illustrative examples of improvement in this section, we show some examples of improvement in translation with the model with multiple pivot languages over the model with a single pivot language. the model with a single pivot language transliterated the word 'star' to 'स्टार' whereas the model with multiple pivot languages correctly translated the word 'star' to 'तारा'. ## conclusion and future work in this work, we show that using multiple pivot languages to assist the source-target nmt model improves its performance. we show using various metrics such as bleu and chrf, that using multiple indic languages as pivot languages and utilizing language relatedness improves the performance of the english-marathi nmt model. we also show that strategic decoder initialization techniques while performing pivot language-based transfer learning improves the performance of the source-target nmt models. in the future, we plan to perform experiments by adding more pivot languages to assist the source to target the nmt model and see the performance of the system.