GX @ D ravidian L ang T ech- EACL 2021: Multilingual Neural Machine Translation and Back-translation (2021)

Abstract: In this paper, we describe the GX system in the EACL2021 shared task on machine translation in Dravidian languages. Given the low amount of parallel training data, We adopt two methods to improve the overall performance: (1) multilingual translation, we use a shared encoder-decoder multilingual translation model handling multiple languages simultaneously to facilitate the translation performance of these languages; (2) back-translation, we collected other open-source parallel and monolingual data and apply back-translation to benefit from the monolingual data. The experimental results show that we can achieve satisfactory translation results in these Dravidian languages and rank first in English-Telugu and Tamil-Telugu translation.

## introduction in recent years, encoder-decoder based on neural machine translation (nmt) has become the mainstream paradigm in the field of machine translation @xcite @xcite . the transformer model @xcite , which is completely based on self-attention mechanism, is the best model architecture for translation performance. however, these data-driven neural machine translation depends heavily on large-scale annotated parallel data, the translation performance of low-resource languages remains a concern @xcite . one such language family is the dravidian family of languages, which are under-resourced in speech and natural language processing @xcite . the dravidian languages are first attested in the 6th century bce as tamili 1 script inscribed on the cave walls and pottery in the madurai and tirunelveli districts of tamil nadu. the dravidian languages with the 1 also called damili or tamil bhrami most speakers are tamil, kannada and malayalam, of which tamil have long literary traditions from 600 bce. in this paper, we adopt two approaches to get a reliable translation for improving access to and production of information for monolingual speakers of dravidian languages. the first approach is multilingual translation. compared with translating from a resource-poor language to english, translating from english to a resource-poor language is much more difficult. translating from a resource-poor language to english, can apply transfer learning to make use of large parallel corpora between english and other languages @xcite . however, the situation of translating from english to a resource-poor language is tough because there is little parallel data for the target language and other languages. in the case of low resources, the multilingual translation method is often better than the performance of individual language pair, because too little data makes it difficult for the model to converge to the best @xcite @xcite . the other approach is back-translation.data augmentation methods can help alleviate this resource shortage by increasing the amount of data. back-translation is a typical method of data augmentation that can enrich training data with monolingual data. these work are often based on automatically creating pseudoparallel sentences through back-translation @xcite @xcite . for the eacl2021 shared task on machine translation in dravidian languages, we present our results on the four language pairs: english-tamil, english-malayalam, english-telugu, and tamil-telugu. overall, the four language pairs can be divided into two subtasks: between english and dravidian languages and between dravidian lan-guages. to get better performance, we implement a translation system for each subtask. since the systems are not constrained using exclusive data provided by the task's organization, we collect other open-source monolingual data of tamil, malayalam, and telugu. we use back-translation to create pseudo-parallel sentences. based on official and pseudo-parallel data, we train multilingual models to perform the translation task. ## background we briefly describe neural machine translation model here. given the source sentence @xmath0, . . . , x i ) and the corresponding target sentence @xmath1, . . . , y j ), a standard nmt model directly optimizes the conditional probability: ## data preparation the data of dravidian languages provided by the organizers are scarce, with only tens or hundreds of thousands of parallel sentences per language pair, which is shown in official en-ta en-te en-ml ta-te release 28417 23222 382868 17155 in this work, all proposed systems are allowed to use data other than those provided by the task organizer. therefore, we use other open-source datasets, including wiki titles v2 2 , pmindia v1 3 , tanzil v1 @xcite , the nlpc uom en-ta corpus and glossary (v1.0.3) @xcite and the ufal entam corpus @xcite . as monolingual data, we use the data of pmindia v1. see sentences of english were tokenized by the moses scripts foot_0 . all non-english data were tokenized by the indic nlp library tool (kunchukuttan, 2020), which is a library to provide a general solution to very commonly required toolsets for indian language text. then, all sentences further segmented into subword symbols using byte-pair encoding (bpe) rules @xcite with 40k merge operations for all languages jointly. 2 . the green data represents pseudo-parallel data which is generated by applying back-translation on monolingual data. ## experiments we first introduce the training details and empirically evaluate the systems in two scenarios. ## related work there are several ways focusing on low-resource translation, including: (a) learning knowledge from high-resource pairs to promotes low-resource translation, such as transfer learning, transferring proper parameters @xcite , lexical knowledge @xcite and syntactic knowledge @xcite from high-resource language, and multilingual translation, including sharing encoders @xcite , sharing decoders @xcite , sharing sublayers @xcite and an universal multi-nmt model with a target language token to indicate the translation direction @xcite ; (b) data augmentation in nmt, such as back-translation (bt) translates target-language monolingual text to create synthetic parallel sentences @xcite and adding an auxiliary auto-encoding task on monolingual data @xcite . also, indian languages have been studied in recent years @xcite @xcite . @xcite show that subword units mehtod is helpful when translation tamil language. @xcite adopt a multilingual adaptation method to benefit from the positive transfer from high resource languages in english and tamil translation. @xcite explore pre-training and iterative back translation for low-resource english-tamil. @xcite explore self-supervised model pretraining, multilingual models, data augmentation, and reranking techniques to solve the low resource problem. ## conclusion in this paper, we describe the gx system in the dravidian translation task of eacl2021. we use two main methods, multilingual translation and backtranslation, to enable the model to benefit from multiple languages joint learning and monolingual data. experimental results show that the system can successfully introduce multilingual neural machine translation and back-translation on monolingual data and improve the performance of the system.