Efficient Neural Machine Translation for Low-Resource Languages via Exploiting Related Languages (2020)

Abstract: A large percentage of the world’s population speaks a language of the Indian subcontinent, comprising languages from both Indo-Aryan (e.g. Hindi, Punjabi, Gujarati, etc.) and Dravidian (e.g. Tamil, Telugu, Malayalam, etc.) families. A universal characteristic of Indian languages is their complex morphology, which, when combined with the general lack of sufficient quantities of high-quality parallel data, can make developing machine translation (MT) systems for these languages difficult. Neural Machine Translation (NMT) is a rapidly advancing MT paradigm and has shown promising results for many language pairs, especially in large training data scenarios. Since the condition of large parallel corpora is not met for Indian-English language pairs, we present our efforts towards building efficient NMT systems between Indian languages (specifically Indo-Aryan languages) and English via efficiently exploiting parallel data from the related languages. We propose a technique called Unified Transliteration and Subword Segmentation to leverage language similarity while exploiting parallel data from related language pairs. We also propose a Multilingual Transfer Learning technique to leverage parallel data from multiple related languages to assist translation for low resource language pair of interest. Our experiments demonstrate an overall average improvement of 5 BLEU points over the standard Transformer-based NMT baselines.

## introduction in recent years, neural machine translation @xcite @xcite @xcite (nmt) has become the most prominent approach to machine translation (mt) due to its simplicity, generality and effectiveness. in nmt, a single neural network often consisting of an encoder and a de-coder is used to directly maximize the conditional probabilities of target sentences given the source sentences in an end-to-end paradigm. nmt models have been shown to surpass the performance of previously dominant statistical machine translation (smt) @xcite on many well-established translation tasks. however, in order to reach high accuracies, nmt systems tend to require very large parallel training corpora @xcite . as a matter of fact, such corpora are not yet available for many language pairs. indian languages are not an exception to this; however they are extremely diverse, belonging to different language families, employing various scripts and spanning a multitude of dialects. the majority of indian languages are morphologically rich and depict unique characteristics, which are significantly different from languages such as english. since nmt models learn poorly from small corpora, building effective nmt systems for lowresource languages (e.g. indian languages) becomes a primary challenge. the bulk of research on low-resource nmt has focused on exploiting monolingual data, or parallel data involving other language pairs. some of the most well-known methods to improve nmt models with monolingual data range from backtranslation @xcite , dual learning @xcite to unsupervised mt @xcite @xcite . similarly, parallel data from other languages can be exploited to either pretrain the network or jointly learn the representations @xcite @xcite . currently, transfer learning (tl) is being widely used for low-resource language translation because it is one of the vital directions for addressing the data sparsity problem in low-resource nmt @xcite @xcite . however, most of the existing approaches that take advantage of transfer learning have a major limitation: they do not exploit multiple languages together and in an efficient manner. the idea presented by @xcite may have the shortcoming of exploiting only one high-resource model (parent) at a time to optimize the low-resource model (child). actually, the use of highly related multiple language pairs might help to increase the translation quality of the child model. the original transfer learning method @xcite also makes no assumption about the relatedness of the parent and child languages. multilingual nmt @xcite approaches which also use parallel data from different languages to improve the translation quality of nmt models does not exploit language relatedness either. in this paper, we present our efforts towards building efficient nmt systems between indian languages (specifically indo-aryan languages) and english by exploiting parallel data from related languages. we aim to deal with the problem of how to make full use of these corpora of highly related languages, to increase the translation quality of low-resource languages. to this end, we introduce two simple and yet effective approaches: ## methodology the core idea of our method is to extend the multilingual learning @xcite and transfer learning @xcite approaches to effectively exploit parallel data from multiple related languages. in section 2.2, we explain our unified transliteration and subword segmentation technique to exploit language relatedness among the parallel data of related languages. sections 2.3 and 2.4 describe our modified multilingual learning and transfer learning techniques for nmt. in section 2.5, we describe our multilingual transfer learning approach. ## results we report the results of multilingual learning, transfer learning and multilingual transfer learning for gujarati-english, bengali-english, marathi-english and punjabi-english language pairs for both translation directions @xcite . in both translation directions, the multilingual models do not prove to be effective. fine-tuning the multilingual models (multilingual transfer learning) on xx-en or en-xx data removes some ambiguities in the model and shows significant improvements compared to their simple multilingual model counterparts. the best performance (almost 5-6 bleu improvements over the baseline) is achieved by fine-tuning the multilingual model (trained on iitb hi-en or en-hi data and all the ilci data) on en-xx or xx-en outperforming all the nmt, multilingual nmt and transfer learning baselines thus demonstrating the effectiveness of our technique.