Pivot Based Transfer Learning for Neural Machine Translation: CFILT IITB @ WMT 2021 Triangular MT (2021)

Abstract: In this paper, we discuss the various techniques that we used to implement the Russian-Chinese machine translation system for the Triangular MT task at WMT 2021. Neural Machine translation systems based on transformer architecture have an encoder-decoder architecture, which are trained end-to-end and require a large amount of parallel corpus to produce good quality translations. This is the reason why neural machine translation systems are referred to as data hungry . Such a large amount of parallel corpus is majorly available for language pairs which include English and not for non-English language pairs. This is a major problem in building neural machine translation systems for non-English language pairs. We try to utilize the resources of the English language to improve the translation of non-English language pairs. We use the pivot language, that is English, to leverage transfer learning to improve the quality of Russian-Chinese translation. Compared to the baseline transformer-based neural machine translation system, we observe that the pivot language-based transfer learning technique gives a higher BLEU score.

## introduction the aim of this work is to improve the quality of machine translation (mt) for low-resource, distant and non-english language pairs. one of the major requirements for the good performance of the neural machine translation (nmt) systems is the availability of a large parallel corpus. such large parallel corpus of good quality is not available for low-resource, distant and non-english language pairs but mostly available for language pairs containing english. this poses a major challenge in developing good quality machine translation systems for non-english and distant language pairs. as a result there is a need to come up with additional resources by augmenting parallel corpora or by using knowledge from other tasks using transfer learning for translation of non-english language pairs. in this paper, we focus on leveraging the knowledge from other tasks using transfer learning to improve the performance of nmt systems for low resource language pairs. in our pivot based transfer learning experiments we try to utilize the resources of english language, that is english-chinese and english-russian parallel corpora to improve the quality of russian-chinese translation. we implement techniques which efficiently use the resources of the english language for the task of russian-chinese translation. ## related work recurrent neural network (rnn) based encoder decoder architectures @xcite @xcite were initially used in nmt systems. transformer @xcite architecture improved the performance of nmt systems. in order to enable translation between distant and non-english language pairs for which a large amount of parallel corpus is not available, a cascade method can be used. in the cascade method, two models are trained, a source language to english and a english to target language model. then to translate a source sentence to target sentence, the source sentence is passed through the two models. @xcite introduced a transfer learning technique in which a parent model is trained on high resource language pairs, which is then used to initialize the the parameters of a child model which is then trained on low resource language pair data. @xcite introduced pivot language-based transfer learning techniques in which the encoder and decoder of the model for low resource language pair is initialized using the encoder and decoder of different models trained on high resource language pairs, and this model is then finetuned on low resource language pair data. multi-lingual nmt systems @xcite @xcite can also be used to improve the performance of low resource language pair translation as knowledge is transferred from various languages which helps the task of low resource language pair translation. ## approaches in this section, we discuss the various approaches we used to build a russian-chinese mt system. we mainly focus on pivot-based transfer learning techniques, in which we use the resources of english to improve the quality of russian-chinese translation. ## experiments in this section, we discuss the details of all the experiments that we carried out to implement the russian-chinese mt system. ## results and analysis the evaluation of the models were performed on the basis of the bleu scores. these bleu scores were calculated and provided by the wmt 2021 organizers. the bleu scores were calculated on a test set provided by wmt 2021 organizers, which consisted of 1751 sentences. ## conclusion and future work in this work, we implement and compared pivot language-based transfer learning technique to improve the task of translation between non-english language pair, that is russian-chinese. we observe that pivot language-based transfer learning technique improves the bleu score over the baseline model and is an efficient way to use the resources of the pivot language. we also observe that the pivot language-based transfer learning technique mitigates the problems of double decoding time and error propagation present in simple cascadebased models. in future, we plan to explore various data augmentation techniques that can make use of the resources of the english language to augment data for the task of translation of non-english language pair translation. we also plan to use various language model pretraining techniques like masked sequence to sequence pre-training (mass) to pretrain the encoder and decoder before using them for the downstream task of translation.