Zero-Shot Neural Machine Translation: R ussian- H indi @ L o R es MT 2020 (2020)

Abstract: Neural machine translation (NMT) is a widely accepted approach in the machine translation (MT) community, translating from one natural language to another natural language. Although, NMT shows remarkable performance in both high and low resource languages, it needs sufficient training corpus. The availability of a parallel corpus in low resource language pairs is one of the challenging tasks in MT. To mitigate this issue, NMT attempts to utilize a monolingual corpus to get better at translation for low resource language pairs. Workshop on Technologies for MT of Low Resource Languages (LoResMT 2020) organized shared tasks of low resource language pair translation using zero-shot NMT. Here, the parallel corpus is not used and only monolingual corpora is allowed. We have participated in the same shared task with our team name CNLP-NITS for the Russian-Hindi language pair. We have used masked sequence to sequence pre-training for language generation (MASS) with only monolingual corpus following the unsupervised NMT architecture. The evaluated results are declared at the LoResMT 2020 shared task, which reports that our system achieves the bilingual evaluation understudy (BLEU) score of 0.59, precision score of 3.43, recall score of 5.48, F-measure score of 4.22, and rank-based intuitive bilingual evaluation score (RIBES) of 0.180147 in Russian to Hindi translation. And for Hindi to Russian translation, we have achieved BLEU, precision, recall, F-measure, and RIBES score of 1.11, 4.72, 4.41, 4.56, and 0.026842 respectively.

## introduction the end-to-end recurrent neural network (rnn) based nmt @xcite approach attracts attention in mt because it deals with many challenges like variable-length phrases using sequence to sequence learning concept, long-term dependency problem adopting long short term memory (lstm) @xcite , attention mechanism @xcite which pays attention globally and locally to all source words. the rnn based nmt approach is not able to process all the input words parallelly, to solve parallelization transformer-based nmt @xcite is proposed by using a self-attention mechanism. despite modifying nmt architecture, it needs reasonable parallel training data which is a challenge for low resource language pair translation. generally, language pairs can be considered as low-resource when training data is less than a million @xcite . for low resource language pair translation, pivot-based nmt @xcite is an effective approach where an intermediate language is considered as a pivot language (source to pivot and pivot to target). @xcite introduced a zero-shot approach to language pair translation without considering the parallel data using multilingual-based nmt. in this paper, we have participated in the loresmt 2020 shared task of zero-shot nmt approach on russian-hindi pair using the only monolingual corpus and the same has been implemented using mass-based unsupervised nmt @xcite . the reason behind choosing mass-based unsupervised nmt is that it achieves state-of-the-art performance on the unsupervised english-french pair translation. ## related work there is a lack of background work on russian-hindi translation. however, the literature survey finds work on unsupervised nmt using mass @xcite which outperform previous unsupervised approaches @xcite . @xcite without us- attention figure 1 : the encoder-decoder framework of the mass model used (as adopted from ( song et al., 2019) ing bilingual corpus and only utilizing monolingual data achieves bleu score 37.50, 34.90 on english to french and french to english translation and for english to german and german to english, it attains bleu score 28.30, 35.20 respectively. transformer architecture @xcite based mass implemented in two steps: pre-training on the monolingual data and then fine-tuning with the self-generated back translation data which acts as a pseudo bilingual corpus during the training process. ## dataset description the loresmt 2020 shared task organizer (ojha et al., 2020) provided the russian-hindi monolingual dataset of train, valid, and test sets, which is summarized in ## system description we have adopted mass based unsupervised nmt @xcite to build our system on a single gpu. our system consists of two major steps namely the pre-training and then the fine-tuning step which are discussed in the sub-sections 4.1 and 4.2. for bpe @xcite and vocabulary creation, we have used the cross-language model (xlm) @xcite codebase as given in their repository foot_2 . moses is used for tokenization @xcite . the mass @xcite based model leverages encode-decoder framework to develop complete sentences from given fractured pieces of sentences as shown in figure 1 . the model details are further described in section 4.1 and 4.2, where we have shown the pre-training and fine tuning step respectively. ## experimental setup during pre-processing of the data, following @xcite and using the code provided by (lample and conneau, 2019), we used fastbpe foot_3 to learn byte pair encoding (bpe) vocabulary with 50,000 codes. also, for leveraging the model features, we have followed the settings of @xcite . in the pre-training step, we have followed the default settings of transformer model-based mass @xcite , where 6 layers with 8 attention heads are used. due to limited computational resources, we have used 256 embedding layers with batch size 32, tokens per batch 500 and dropout 0.1. the obtained pre-trained model from 4.1 are fine-tuned with pseudo bilingual corpus through self-generated back-translation data following default settings of @xcite . ## result and analysis the loresmt 2020 shared task organizer declared the evaluation result foot_4 of zero-shot nmt on the language pairs namely, hindi-bhojpuri, hindi-magahi, and russian-hindi, and participated by two teams only. for the russian-hindi language pair, only our team participated and our team name is cnlp-nits. the results are evaluated using automatic evaluation metrics, bleu @xcite , precision, recall, f-measure and ribes @xcite . we have submitted two systems result, one only using provided monolingual data (extension -a) and another with external monolingual data addition of provided monolingual data (extension -c) and the same have been reported in ## conclusion and future work this paper presents a zero-shot nmt task on the russian ⇔ hindi translation, this system was used to participate in the loresmt 2020 shared task. we have used unsupervised nmt approach of mass @xcite to build a single model that can translate in both the directions i.e. russian to hindi and vice-versa. the obtained scores and closely observed predicted output remarks that our future works require significant improvement to achieve better translation accuracies in both directions.