The NITS - CNLP System for the Unsupervised MT Task at WMT 2020 (2020)

Abstract: We describe NITS-CNLP’s submission to WMT 2020 unsupervised machine translation shared task for German language (de) to Upper Sorbian (hsb) in a constrained setting i.e, using only the data provided by the organizers. We train our unsupervised model using monolingual data from both the languages by jointly pre-training the encoder and decoder and fine-tune using backtranslation loss. The final model uses the source side (de) monolingual data and the target side (hsb) synthetic data as a pseudo-parallel data to train a pseudo-supervised system which is tuned using the provided development set(dev set).

## introduction this paper provides the system description of the unsupervised neural machine translation system for german to upper sorbian submitted by the center for natural language processing of national institute of technology, silchar, india (nits-cnlp) in the wmt 2020 shared task for unsupervised and very low resource machine translation for german and upper-sorbian language pair. specifically, we made our primary submission for the unsupervised task in @xmath0. we use the data provided by the organisers only i.e, in a constrained manner. our unsupervised neural machine translation (unmt) system first pre-trains a transformer @xcite based encoder and decoder model using masked sequence to sequence (mass) pre-training @xcite and fine-tune using the back-translation @xcite loss. the final model trained using mass objective is then used to translate the source side (m de ) monolingual data into a synthetic target side data (m hsb )and then train a pseudo-supervised model using {m de ,m hsb } from scratch. the remaining of the paper is arranged in following manner: section 2 gives a brief background of an unsupervised mt. section 3 describes the data preprocessing. in section 4, we describe our unmt system. the results and analysis are shown in section 5. finally, section 6 concludes the paper. ## data and preprocessing this section is further divided into two subsections briefing the data description and the preprocessing steps used. ## unmt system our unmt system is a pipeline of encoderdecoder pretraining and fine-tuning using mass @xcite and using the synthetic data --mass steps 'de,hsb' --encoder only false --emb dim 1024 --n layers 6 --n heads 8 --dropout 0.1 --attention dropout 0.1 --gelu activation true --tokens per batch 3000 --optimizer adam inverse sqrt, @xmath1.9,@xmath2.98,@xmath3.0001 --word mass 0.5 --min len 5 ## result the official automatic evaluation uses the the following metrics: bleu @xcite , ter @xcite , beer @xcite , and charactter @xcite . our primary submission (nits-cnlp), the pseudo-supervised nmt achieves a cased bleu of 15.4 and 15.8 as the uncased bleu score on the newstest2020 blind-test data. the scores are reported in ## conclusion we report here the system description for our submission to the wmt 2020 shared task of unsupervised mt for german-upper sorbian language pair. we submit our pipelined architecture of masked sequence to sequence pretraining along with finetuning and a pseudo-supervised model in german to upper sorbian direction. we observe that the performance of an unsupervised model improves significantly over the base mass pretraining and