Zero-shot translation among I ndian languages (2020)

Abstract: Standard neural machine translation (NMT) allows a model to perform translation between a pair of languages. Multilingual neural machine translation (NMT), on the other hand, allows a model to perform translation between several language pairs, even between language pairs for which no sentences pair has been seen during training (zero-shot translation). This paper presents experiments with zero-shot translation on low resource Indian languages with a very small amount of data for each language pair. We first report results on balanced data over all considered language pairs. We then expand our experiments for additional three rounds by increasing the training data with 2,000 sentence pairs in each round for some of the language pairs. We obtain an increase in translation accuracy with its balanced data settings score multiplied by 7 for Manipuri to Hindi during Round-III of zero-shot translation.

## introduction end-to-end neural machine translation (nmt) @xcite @xcite can be applied to low resource languages with the risk that small amounts of training data result in low translation accuracy @xcite . improvement in translation of low resource languages has been reported with the use of multilingual models @xcite , back-translation @xcite and unsupervised learning @xcite . initially, mt systems were designed for one single language pair @xcite . however, nmt systems can be trained simultaneously on many language pairs. this enables translation from and into any of the languages used during training. @xcite first modified an attention-based encoder-decoder model so as to perform multilingual translation from one language to many languages while @xcite used multitask learning for multilingual training. @xcite introduced the notion of multilingual nmt, by sharing the attention mechanism across several languages. @xcite introduced universal machine translation, where a universal representation space is used for all languages. @xcite introduced zero-shot translation: training on multiple source and target languages enables to translate arbitrarily between any of the languages used during training, even between languages for which no sentence pair was ever seen during training. the authors characterised zeroshot translation as "a working example of transfer learning within neural translation models". our work consists in testing the use of zero-shot translation for a very low resource language, an indian language called manipuri, locally known as meiteilon, in the context of training with other indian languages between which no parallel data may exist. again, manipuri is a low resource language. it is spoken by about two million people predominantly in the state of manipur, india. it is an endangered language (moseley and nicolas, 2010) from the sino-tibetan language family and it shows highly agglutinating word structure. with its language status as endangered, it is one of the two endangered languages of the 8th schedule of the indian constitution. machine translation for this language is at its infant stage due to the very limited amount of resources available. we make use of the pmindia dataset foot_0 @xcite . this data set provides monolingual and parallel corpora with english for thirteen indian languages. we take the following language pairs into consideration: assamese-english, bengali-english, hindi-english and manipuri-english foot_1 . our main objective is to measure how much accuracy can be achieved in translation from manipuri into the three other indian languages (assamese, bengali and hindi), without using any data from these language pairs, thanks to zeroshot translation. additionally, we use the jw300 dataset foot_2 @xcite for assamese-english language pairs for two rounds of the experiment due to the limited number of data present in the pmindia data set for this language pair. in our experiments, we use only the above-mentioned resources. our goal is to improve the translation quality of our zero-shot translation system among the low resourced languages. we propose to control the translation quality by introducing the notion of balanced data in the respective language pairs as a parameter. the reason why we concentrate on manipuri is because it is an extremely low resource language: only 7,000 sentence pairs in manipuri-english are available in the pmindia data set. developing mt systems with such a small amount of data is a true challenge. our experiments consist in increasing the training data by groups of 2,000 sentence pairs (indian language-english), in three rounds. we measure the translation accuracy between manipuri and other indian languages in zero-shot translation. the structure of the paper is as follows. section 2 describes previous work. section 3 gives details about the data set used. section 4 presents the methodology. section 5 describes the experiments, their results and provides an analysis. section 6 concludes and proposes future directions. zero-shot machine translation has been explored for low resource languages. @xcite proposed an approach for multi-source translation. their model consists in multiple encoders with a different attention mechanism for each source language. however, this model requires a multi-way parallel corpus for every language pairs, which is hard to obtain, especially for languages with low resource. nmt is capable of cross-lingual learning @xcite . this is the motivation for zero-shot translation. @xcite introduced the notion of zero-resource translation. they used a pre-trained multi-way multilingual model and performed fine-tuning with the pseudo parallel data generated by the model. @xcite introduced an approach for improving multilingual nmt for indian languages. they showed that their model is able to improve the translation for low resource language pairs by leveraging high resource language pairs, thanks to transfer learning. our work is closely related to @xcite : we analyse the performance of multilingual models on our data and perform zero-shot translation as well. the originality in our work is that we aim to improve the translation quality of our model and since we deal with low resource languages, we propose to control the translation quality such that we train, validate and test our model on balanced data sets across all the language pairs. ## dataset we use the pmindia dataset @xcite . 4 this data set contains the official documents from the prime minister office of the government of india. it contains monolingual and parallel corpora. there are 13 indian languages and english in it. we use data for four language pairs from the parallel corpus found in the data set: from assamese, bengali, hindi and manipuri into english. the indian languages used belong to different language families, yet they have high lexical similarities because of regional influences. assamese (iso 639-2 asm), bengali (ben) and hindi (hin) belong to the indo-aryan language family, manipuri (mni) belongs to the sino-tibetan language family and english (eng) belongs to the indo-european language family. assamese, bengali and manipuri share the same writing system, the eastern nagari script. bengali has high language influence on assamese, and some influence on manipuri as well. hindi, on the other hand, influences all the other languages, with a large number of words borrowed from it. statistics about the data used are presented in additionally, we use the jw300 dataset @xcite for assamese-english language pairs for two rounds due to the limited number of sentence pairs present in the pmindia data set. ## methodology we propose to first measure the effect of using zero-shot translation on balanced data sets for all language pairs. we then expand our experiments and increase the training data with the aim of inspecting how efficient or not transfer learning can be. the increase will be performed by groups of 2,000 sentence pairs in training for language pairs excluding manipuri. there will be three rounds of increase of data. to deal with numerous source and target languages during multilingual or zero-shot translation training, we classically introduce an artificial token at the beginning of all the source language sentences. the artificial token contains the information about the source language and the target language for the sentence pair at hand. our first series of experiments consists in measuring translation quality in a balanced data set setting. our model is trained on an equal amount of training data for all the languages. we start with 5,000 sentences for training across all language pairs. the amount of test and validation data is 1,000 sentences each for all the languages in our model. each language uses a balanced data set for training, validation and test in the balanced round. the data is randomly selected. we do not perform pivoting through the english language because of the limited number of sentences in common. our second series of experiments measures translation quality when increasing the data for other languages than manipuri. we showed in in total, we report translation quality for three types of models. an example of a zero-shot model is described below. suppose that we have trained a model on the following language pairs: manipuri-english and assamese-english, in both directions, hence in 4 language directions. the zero-shot model will translate between manipuri and assamese in both directions, although no assamese-manipuri or manipuri-assamese sentence pair has been seen during training. one of our experiments focuses on zero-shot translation with balanced data set. for that, we test the three possible different combination of language pairs with our data. for one of the three indian languages, assamese, bengali or hindi, call it x, we build a system to perform zero-shot translation from manipuri into x by using sentence pairs from manipuri-english in both directions and x-english, in both directions too. we change the conditions of the above experiments by increasing the training data with groups of 2,000 sentence pairs in the three indian language to english language pairs for three rounds, for all of our models. ## conclusion this work provided an investigation in the use of zero-shot translation between some indian languages, in the context of low resource. firstly, we studied the influence of the balance in data sets across the considered language pairs. we observed that a multilingual model performs comparatively better than a baseline single model, in terms of bleu scores. in addition, we observed that, in zero-shot translation, a balanced configuration does not perform well. as observed in other works, the use of nmt on a very small amount of data for training, validation and testing, results in low translation accuracy, because nmt has a steep learning curve with respect to amount of data @xcite . the translation accuracy when incrementing the data size is comparatively better than in the balanced data set settings. we observed a very small increase in bleu scores in the balanced settings from round-i to round-iii, although there is no statistically significant difference. in zero-shot translation, manipuri-bengali recorded the highest bleu score among all language pairs, while manipuri-assamese recorded the least score. in the future, we would like to inspect the possibility of increasing the size of our data by using back-translation. we expect that synthetic data will help our models in improving the translation accuracy @xcite . we would also like to inspect the use of the unsupervised learning approach with adversarial training to learn a mapping from source to target languages without any parallel data or anchor points @xcite on our models.