Unsupervised Neural Machine Translation for E nglish and M anipuri (2020)

Abstract: Availability of bitext dataset has been a key challenge in the conventional machine translation system which requires surplus amount of parallel data. In this work, we devise an unsupervised neural machine translation (UNMT) system consisting of a transformer based shared encoder and language specific decoders using denoising autoencoder and backtranslation with an additional Manipuri side multiple test reference. We report our work on low resource setting for English (en) - Manipuri (mni) language pair and attain a BLEU score of 3.1 for en-mni and 2.7 for mni-en respectively. Subjective evaluation on translated output gives encouraging findings.

## introduction machine translation had been dominated by the statistical methods, notably the phrase-based @xcite . but they suffered from a rigid structure since they have multiple modules @xcite which are tuned independently. in other words, smt lacked an end-to-end learning mechanism. for a quite long period, smt dominated the mt systems, but with an rnn based sequence-to-sequence model @xcite ; @xcite marked the beginning of the nmt era. but, these primitive neural based mt systems choked when the input sentences get longer as the input sentence are squeezed into a fixed length vector. fortunately, with the advent of attention @xcite , sub-word tokenization @xcite and transformers @xcite , nmt outperformed smt in various machine translation tasks. however, when the parallel corpus is scarce, the nmt fails to produce good translations, and performing much poorer than the phrase-based systems. building parallel corpus is a costly task and specifically for the low resource languages where bi-text is nonexistent. but, monolingual data is easily available even for the low resource languages and some have utilised it to augment parallel data @xcite with a little bi-text data or translation systems using monolingual data only @xcite @xcite . recent works using monolingual data only show a positive direction in machine translation tasks. although, these systems do not outperform a strong supervised system, they could be treated as a strong baseline system which should be the lower bound for any supervised system. ## related works to the best of our knowledge, we are the first to report the unsupervised neural machine translation for manipuri language. however, there are reports of supervised statistical based approach, notably by @xcite , where the authors made an imperative study over the effects of the morpho-syntactic information and dependency relation in a statistical machine translation setting. in another work, @xcite showed that the phrase based statistical machine translation system improves by incorporating linguistic features such as, named entities and reduplicated multiword expressions. however, the mt task for manipuri is still in its inception, considering it being a low resource language. low resource has always been a hurdle in the mt task. many, have mended their hands to overcome this bottleneck. @xcite leveraged the nmt system by using monolingual data to create a synthetic parallel corpus using backtranslation. this synthetic data greatly improved the mt systems, however it consisted mostly noises which is necessary to be postprocessed, making the task not so feasible. @xcite devised a transfer learning mechanism by sharing model parameters from a high resource language pair (parent model) to lower resource language pair (child model) which significantly improved the bleu @xcite score. this kind of parent-child model has been used in other works @xcite where they used a shared vocabulary of subword units. @xcite further showed that transfer learning can be simplified where the parent model is trained until convergence and switching the low resource language pair as the training data while keeping the training parameters unchanged. parameter sharing has also been explored in previous works such as @xcite by using a single shared attention mechanism with multiple encoder-decoder to devise a multi-way multilingual nmt system. further, @xcite devised another multilingual approach through parameter sharing using a single shared encoder-decoder model enabling zero-shot translation, where the model learned translation between multiple languages and could even translate unseen language pairs. however, these models require some form of parallel data which has led the dynamics to shift towards exploiting the monolingual data, which is comparatively abundant than the bi-text data. furthermore, a noteworthy attempt is reported by @xcite , where they used an auto-encoding task to ensure the translated sentence can be translated back to the original sentence using reinforcement learning technique. similarly, @xcite also used this auto-encoding task upon the monolingual data. although, these models seemed promising, it still needed decent amount of parallel data for a warm-start. machine translation task can be reduced to a deciphering task @xcite from a monolingual data using noisy channel model where the source language is treated as ciphertext generation, however these settings were mostly confined to short sentences and related language pairs. all these works induced a promising starting point towards exploiting the monolingual data, but these primitive models were unable to stand against a supervised setting with abundant parallel data. fortunately, it was the concurrent work of @xcite and @xcite which lifted the unsupervised mt on par with a supervised setting. their approach first learns a linear transformation of the word embeddings of the two languages in an unsupervised manner which are trained independently and map this linear transformation into a shared space using adversarial training @xcite or through self learning @xcite . a shared encoder for both the languages is initialized using the resulting cross-lingual embeddings. the model is trained using denoising auto-encoding, backtranslation and adversarial training 3 iteratively giving rise to translation models of increasing quality. there are reports of unsupervised mt using pb-smt(phrase-based statistical machine translation) where @xcite used the backtranslated synthetic data to feed into the nmt system. furthermore, there are reports of using the synthetic data generated from an unsupervised smt to initialize an nmt system from scratch in an iterative way @xcite . recently, a pretrained cross-lingual language model @xcite is used to pretrain a shared encoder and finetune using iterative backtranslation. ## the framework in this section, we discuss the framework of our unsupervised neural machine translation system. ## experimentation we perform the translation task for both the en ↔ mni sides. we use the same data for both tasks and compare our model with xlm (lample and conneau, 2019) and @xcite . first, we discuss the dataset description and its preprocessing followed by the baseline and the proposed framework. ## results and analysis in this section, we discuss the results and the performance of our proposed setting (unmt proposed ) in comparison with the baselines (unmt baseline-1 ) and (unmt baseline-2 ). the reported bleu score is calculated upon the de-tokenized text using sacrebleu foot_12 @xcite while chrf is calculated using chrf toolkit foot_13 for the en ↔ mni directions. the scores of the systems is given in ## conclusion in this work, we report an unsupervised neural machine translation system on low resource setting for english (en) -manipuri (mni) language pair using a transformer based shared encoder, language specific decoders and monolingual data only. we observe an improvement in bleu score (+0.4 for @xmath1.3 for @xmath2. similarly, the chrf scores of unmt proposed surpassed both the baselines by a large margin. moreover, it is found that the cross lingual embedding mappings is more effective than a cross lingual language model pretraining for this language pair. one of the reason being that the xlm model objective pretrains the encoder and the decoders separately and the decoders are shared between the two languages. this shared decoder may be useful for a similar language pair but for the distant pairs it fails to capture the cross-lingual representations. besides, the automatic scoring mechanism bleu fails to mitigate and capture the linguistic inflections of the morphologically richer manipuri language which is tackled by using multiple references. mean-while, the translation quality is reasonably fluent and adequate considering the facts that the language pairs are relatively unrelated and the use of a single test reference. finally, our unsupervised premise made a descent performance considering the relatively smaller monolingual data used hence this work highlights the potential of the unsupervised mt for this distant language pairs. in our future work, we plan to extend this preliminary unsupervised setting by incorporating linguistic features and devise an improved initialization step along with a better use of the synthetic data suitable for this language pair.