Multilingual Multi-Domain NMT for I ndian Languages (2021)

Abstract: India is known as the land of many tongues and dialects. Neural machine translation (NMT) is the current state-of-the-art approach for machine translation (MT) but performs better only with large datasets which Indian languages usually lack, making this approach infeasible. So, in this paper, we address the problem of data scarcity by efficiently training multilingual and multilingual multi domain NMT systems involving languages of the 𝐈𝐧𝐝𝐢𝐚𝐧 𝐬𝐮𝐛𝐜𝐨𝐧𝐭𝐢𝐧𝐞𝐧𝐭. We are proposing the technique for using the joint domain and language tags in a multilingual setup. We draw three major conclusions from our experiments: (i) Training a multilingual system via exploiting lexical similarity based on language family helps in achieving an overall average improvement of 𝟑.𝟐𝟓 𝐁𝐋𝐄𝐔 𝐩𝐨𝐢𝐧𝐭𝐬 over bilingual baselines, (ii) Technique of incorporating domain information into the language tokens helps multilingual multi-domain system in getting a significant average improvement of 𝟔 𝐁𝐋𝐄𝐔 𝐩𝐨𝐢𝐧𝐭𝐬 over the baselines, (iii) Multistage fine-tuning further helps in getting an improvement of 𝟏-𝟏.𝟓 𝐁𝐋𝐄𝐔 𝐩𝐨𝐢𝐧𝐭𝐬 for the language pair of interest.

## introduction good translation systems are an important requirement due to substantial government, business and social communication among people speaking different languages. neural machine translation @xcite is the current state-of-the-art approach for machine translation in both academia and industry @xcite . the success of nmt heavily relies on a huge amount of parallel sentences as training data. but using the traditional approaches @xcite , one would still need to train a separate model for each translation direction and also a lot of parallel human translated corpora which is again expensive to generate. but on the other hand, multilingual neural machine translation @xcite enables training a single model that supports translation from multiple source languages to a single target language or from a single source language to multiple target languages. in addition to this, another benefit of a training a single model for multiple translation directions is the ability to learn not just from the training data of the language pair of interest, but also from other language pairs. but this learning is hindered in case of language pairs that do not show any kind of relatedness among themselves. but on the other hand, indian languages exhibit a lot of lexical and structural similarities on account of sharing a common ancestry. it is therefore important to utilize the lexical similarity @xcite of these languages to build efficient systems by combining all the related languages. also, in a typical in-domain mt scenario, the amount of parallel texts from a single domain is not enough to train a good translation system, even for multilingual systems. apart from this, one has to train an individual mnmt system for each domain. so we propose a technique for creating efficient multilingual multi-domain nmt systems which help in overcoming the above mentioned limitations. in this work, we treat different domains as distinct languages: for example, instead of hindi-english translation we see it as translating hindi-health to english-health. we utilized our multilingual nmt approach in a multi-domain setting and our results confirm that our multilingual multi-domain system significantly outperforms in-domain baselines as well as it also give improvement for out-of-domain translations. the paper is organized as follows: section 2 talks about the related work. methodology for our experiments is explained in section 3, followed by experimental details and results in section 4 and section 5 respectively. all the conclusions and the future work have been briefly discussed in section 6. ## related work due to simplicity, generality and effectiveness of neural machine translation (nmt), it has become the most prominent approach to machine translation @xcite @xcite @xcite . basic training procedure of nmt does not work well with only a handful of bilingual data @xcite , while collecting bilingual resources is arduous for indian languages. a lot of experiments have been made to improve the quality of translation mainly including exploiting monolingual data range from back translation @xcite , dual learning @xcite to unsupervised mt @xcite . on the other hand, many tried to exploit parallel data of other high resource languages @xcite @xcite to either pretrain the network or jointly learn the representation. recently multilingual nmt has drawn more attention by several research groups. for instance, @xcite modify the current state-ofthe-art attention nmt approach by introducing a many-to-many system, which still relied upon separate encoders and decoders for each language along with a shared attention mechanism. in contrast, @xcite and @xcite both introduce a simple method for training a single-model multilingual nmt system, which does not require any modifications to the nmt encoder-decoder architecture of the system. the main difference is that @xcite added target language identifying token in the beginning of each source sentence of the training data and @xcite added a language identifying token to each subword unit and apply this pre-processing to both source and target sentences of the training data. both aims at exploiting many different languages rather than focusing on language relatedness and observes that only the many-to-one paradigm can achieve better translation results than the individually trained models. for the other two paradigms, there are various degrees of quality degradation. also @xcite has put some efforts in tackling efficient nmt system in low-resource settings by considering language relatedness. apart from this, researchers have also explored the area of domain adaptation for nmt @xcite and reported significant improvements. @xcite explored multi-domain neural machine translation for single language by adding the domain token to the input sentence. so, we put our efforts in exploring the system performance when multilingualism is combined with multi-domain systems for major indian languages. ## methodology india is a land of diverse languages. it has many languages on the basis of regional diversities, and mainly divided into indo-aryan and dravidian families. a universal characteristic of indian languages is their complex morphology. indian languages depict unique characteristics following default sentence structure as subject object verb (sov) and relatively free word order. these languages also share many common words which have the same root and meaning. however they use different scripts derived from the ancient brahmi script @xcite , but correspondences can be established between equivalent characters across scripts. so, we exploited lexical similarity for efficient mnmt. also, in order to incorporate multiple domains into our multilingual system, we also introduced the technique of representing the domain as a new language. ## results and analysis we report the results of bilingual baseline, multi-domain bilingual baseline, multilingual, multilingual multi-domain and multilingual multi-domain with special tokens for both the translation directions, xx-en and en-xx (where xx denotes indian languages). later, we also compared the results of multistage fine tuning with the above experiments for one language from each family. ## conclusions and future work in this paper, we explored different effective methods to exploit parallel data from multiple related languages and domains to improve the translation between indian languages and english. our results show that the multilingual models accuracy depends upon the type of dataset in hand. as we observed in the case of pmi and ilci, multilingual models trained on the pmi dataset increased our average bleu score while the model trained on the ilci dataset decreased the bleu score due to increase in ambiguity. we then introduced multilingual multi-domain models and observed that this idea helps in removing the ambiguity we faced in the multilingual system using multi parallel data for training thus improving the translation quality by showing improvements in bleu scores. in this work, we also introduced a new technique of adding a domain as a separate language by modifying the language token to language domain token. our experiments also confirm that this new technique always outperforms all the models we discussed above. at last, we also explored the concept of multistage fine tuning in which we transfer the learning of the parent model to the child in multiple stages. in future, we would like to work on effective techniques to exploit monolingual data and parallel data from other languages together to improve the translation quality. also, we will try to generalize this idea of exploiting the related languages to other nlp related applications like sentiment analysis.