Domain Adaptation of NMT models for E nglish- H indi Machine Translation Task : A dap MT Shared Task ICON 2020 (2020)

Abstract: Recent advancements in Neural Machine Translation (NMT) models have proved to produce a state of the art results on machine translation for low resource Indian languages. This paper describes the neural machine translation systems for the English-Hindi language presented in AdapMT Shared Task ICON 2020. The shared task aims to build a translation system for Indian languages in specific domains like Artificial Intelligence (AI) and Chemistry using a small in-domain parallel corpus. We evaluated the effectiveness of two popular NMT models i.e, LSTM, and Transformer architectures for the English-Hindi machine translation task based on BLEU scores. We train these models primarily using the out of domain data and employ simple domain adaptation techniques based on the characteristics of the in-domain dataset. The fine-tuning and mixed-domain data approaches are used for domain adaptation. The system achieved the second-highest score on chemistry and general domain En-Hi translation task and the third-highest score on the AI domain En-Hi translation task.

## introduction machine understanding of natural language queries is of paramount importance to automate different workflows. the natural language query can be in the form of text or speech. processing of query in the form of text is more popular and easy than directly processing the raw speech waveform. the text-based natural language processing (nlp) include tasks like classification, token tagging, summarization, and translation. machine translation is an nlp technique to translate a sentence from a source language to a target language. the neural machine translation (nmt) is a recent approach to translation producing state of the art results @xcite . nmt defines translation as a sequence to sequence task and uses sequence-based neural architectures like long short term memory (lstm) @xcite and transformer @xcite . traditional techniques like rule-based translation and statistical machine translation (smt) have been outperformed by nmt models achieving significant improvements on mt tasks. in this work, we are specifically concerned with english-hindi neural translation. the hindi language is one of the most popular languages in india and the fourth most spoken language in the world. hindi is native to india and is spoken by more than 550 million total speakers worldwide. however, the number is much less as compared to global languages like english. on similar lines, the training data for the hindi language that is publicly available for mt tasks is relatively less as compared to other highly popular languages worldwide like english, french, and german. this is important as mt tasks require a huge amount of training data to produce remarkable results using nmt models. hindi being a relatively low resource and morphologically rich language, the amount of research in mt tasks for the hindi language is limited @xcite . as hindi is the most widely spoken language in the indian subcontinent and the majority of content across the globe is published in english, the research in mt tasks for english-hindi language pair becomes highly important. domain adaptation of translation systems to specific domains is a common practice for low resource language pairs. the adaptation is relevant as the text in different domains can vary widely @xcite . for example, social media text and the text in literary work will be quite different from style, grammar, and abbreviations perspective. the domains can be distinguished based on topics like politics, life science, news, etc, or the style of writing like formal and informal. a translation model trained on one domain may not work well on other domains. the problem is more severe in models that use word-based representation as most of the domain-specific words will be out of vocabulary @xcite . in this work, we explore ideas for domain adaptation for english-hindi translation on the adapmt shared task icon 2020 data sets. the adapmt shared task icon 2020 aims to evaluate the capability of general domain machine translation for low resource indian languages. indian languages considered in adapmt shared task icon 2020 for translation are english-hindi, english-telugu, and hindi-telugu. the shared task also focuses on low resource domain adaptation of machine translation systems. the adaptation is done with the use of already publicly available parallel corpora and some small in-domain parallel data for ai and chemistry domains. the creation of a publicly available parallel corpus for low resource indian languages is another important goal of this task. this paper describes the system built for the english-hindi general mt and domain adaptation tasks held under adapmt shared task icon 2020. we experimented with two popular nmt models namely attention-based lstm encoder-decoder architecture and the transformer architecture. for domain adaption, we explore fine-tuning and mixed domain training approaches. we show that the mixed domain training performs better than the fine-tuning based approach for the datasets used in this work. ## architecture in this section, we describe the two popular seq2seq neural architectures for machine translation used in this work. the encoder-decoder architecture consisting of a source side encoder and a target side decoder is used for the sequence to sequence tasks @xcite . the encoder encodes the text in a source language into a latent representation which is consumed by the decoder to generate the text in the target language. the decoder acts like a contextual language model generating target text by attending to the source representations. the attention mechanism is thus an integral part of encoder-decoder models which allows the decoder to focus on the right context while generating the corresponding target token. ## domain adaptation while generalization is always desirable the machine learning systems are often biased towards the domain of the training data. each domain has a different distribution and different domain data are mixed while building general systems. in very basic terms, the vocabulary of the different domains is mostly different. domain adaptation is a form of transfer learning where we adapt a general system for a specific domain. that is we tune the model to adapt to the distribution of the target domain. it has been widely studied in the context of machine translation @xcite . the adaptation techniques can either be data or model-centric. the data related approaches try to exploit the monolingual corpus of the target domain @xcite . a commonly used technique is to use back-translation to expand the parallel corpus of the in-domain data @xcite . the model-based approaches also make use of monolingual corpus from the target domain to train a language model and then do a shallow or deep fusion @xcite . there is another set of training based technique which also go into model-centric approaches. in these approaches, the model is first trained on large out of domain parallel corpus and then re-trained or fine-tuned on the small in-domain parallel corpus. there are different variations proposed in literature where the second fine-tuning is done on a mixed parallel corpus instead of only using the in-domain corpus @xcite . the concept of domain tag was introduced in @xcite . the model is passed the domain label along with each training sample so that it learns to distinguish between the domains. the under-represented domains are oversampled. in this work, we evaluate the domain data fine-tuning approach and mixed-data training approach. in the first approach, we train the model on general corpus followed by in-domain corpus. in the second approach, we mixed the in-domain corpus with the general data and do a single training. since the amount of in-domain data is very less as compared to the overall general or mixeddomain data we oversample in-domain examples while training. ## results and discussion we evaluate the mixed data and fine-tuning approaches on lstm and transformer nmt models. to compare the models bilingual evaluation understudy (bleu) score is used @xcite . we report the bleu score on validation data of ai and chemistry in-domain corpus. we see that although the transformer based models perform well on the iitb test data they do not generalize well on the domain tasks. however, we feel that the low numbers with the transformer can be enhanced using appropriate hyper-parameters and modifying the training approach. the system submitted for evaluation was lstm based model trained on a mixed corpus which was giving the best validation scores. the results of the test system are shown in ## conclusion in this paper, we evaluated the effectiveness of attention-based encoder-decoder lstm and transformer models on a low resource english to hindi translation task held under adapmt shared task icon 2020. our experiments showed that mixed domain training works well as compared to the finetuning approach for domain adaptation. the addition of small in-domain parallel data can indeed improve the results on ai and chemistry domains provided in the shared task.