Domain Adaptation for H indi- T elugu Machine Translation Using Domain Specific Back Translation (2021)

Abstract: In this paper, we present a novel approachfor domain adaptation in Neural MachineTranslation which aims to improve thetranslation quality over a new domain. Adapting new domains is a highly challeng-ing task for Neural Machine Translation onlimited data, it becomes even more diffi-cult for technical domains such as Chem-istry and Artificial Intelligence due to spe-cific terminology, etc. We propose DomainSpecific Back Translation method whichuses available monolingual data and gen-erates synthetic data in a different way. This approach uses Out Of Domain words. The approach is very generic and can beapplied to any language pair for any domain. We conduct our experiments onChemistry and Artificial Intelligence do-mains for Hindi and Telugu in both direc-tions. It has been observed that the usageof synthetic data created by the proposedalgorithm improves the BLEU scores significantly.

## introduction neural machine translation (nmt) systems achieved a breakthrough in translation quality recently, by learning an end-to-end system @xcite . these systems perform well on the general domain on which they trained, but they fails to produce good translations for a new domain the model is unaware of. adapting to a new domain is highly challenging task for nmt systems, it becomes even more challenging when it comes to technical domains like chemistry, artificial intelligence etc, as they contain many domain specific words. in a typical domain adaptation scenario like ours, we have a tremendous amount of general data on which we train an nmt model, we can assume this as a baseline model, now provided a new domain data, the challenge is to improve the translation quality of that domain using available little amount of parallel domain data. we adopted two technical domains namely, chemistry and artificial intelligence for hindi -> telugu and telugu -> hindi experiments. the parallel data for the mentioned technical domains is very less, hence we used back translation to create synthetic data. instead of using synthetic data directly which may contain lots of noise we used domain monolingual data to create synthetic data in a different way (see section 3.4) and used such that translation of domain terms and context around them is accurate. ## background & motivation as noted by @xcite there are two important distinctions to make in domain adaptation methods for machine translation(mt). the first is based on data requirements, supervised adaptation relies on indomain parallel data, and unsupervised adaptation has no such requirement. there is also a difference between model-based and data-based methods. model-based methods make explicit modifications to the model architecture such as jointly learning domain discrimination and translation @xcite , interpolation of language modeling and translation @xcite and domain control by adding tags and word features @xcite . @xcite proposed iterative dual domain adaptation framework for nmt, which continuously fully exploits the mutual complementarity between in domain and out domain corpora for translation knowledge transfer. apart from this freitag and al-onaizan (2016) proposed two approaches, one is to continue the training of the baseline model(general model) only on the in-domain data (domain data) and the other is to ensemble the continue model with the baseline model at decoding time. coming to the databased methods for domain adaptation, it can be done in two ways, combining in-domain and out-of-domain parallel corpora for supervised adaptation @xcite or by generating pseudo-parallel corpora from in-domain monolingual data for unsupervised adaptation @xcite . our approach follows a combination of both supervised and unsupervised approaches. where we first combine domain data (chemistry and artificial intelligence ) with general data, train a domain adaptation model. then, as an unsupervised approach we use available domain monolingual data to back translate and use to create domain adaptation model. @xcite explained how we can use monolingual data effectively in our mt systems, inspired from @xcite , instead of just adding domain parallel data which is very small in amount to general data we used available domain monolingual data to generate synthetic parallel data. in @xcite they have analyzed various ways to integrate monolingual data in an nmt framework, focusing on their impact on quality and domain adaptation. a simple way to use monolingual data in mt is to turn it into synthetic parallel data and let the training procedure run as usual @xcite , but this kind of synthetic data may contain huge noise which leads to performance degradation of domain data. therefore, we present an approach which generates synthetic data in a way such that it is more reliable and improves the translation. in the context of phrase-based statistical machine @xcite has noted that unseen (oov) words account for a large portion of translation errors when switching to new domains, however this problem is still exist even in nmt as well. considering this issue, inspired from @xcite we proposed a novel approach called domain specific back translation which uses out of domain(ood) words to create synthetic data from monolingual data which will be discussed in detail in section 3.4. @xcite also created synthetic data using oov in a different way, whereas we used ood words to create synthetic data. ## methodology as discussed in section 2 there are many approaches for domain adaptation mainly divided into model-based and data-based methods. however our approach falls under databased method, we discuss this in detail in section 3.3. though, there exists many domain adaptation works in mt, to the best of our knowledge there is no such work for indian languages especially which considers technical domains like chemistry, artificial intelligence etc. hence there is a huge need to work on indian languages where most of them are morphologically rich and these type of domains (technical domains) to improve the translation of domain specific text that contain many domain terms etc. we conducted all our experiments for hindi and telugu in both directions for chemistry and artificial intelligence.the language pair (hindi-telugu) considered in our experiments are morphologically rich therefore, there exists many post positions, inflections etc. in order to handle all these morphological inflections we used byte pair encoding (bpe), we can see detail explanation about bpe in section 3.2. ## results & discussion when we say back chem or back ai in this paper, that means they are the domain specific back translated data. after getting domain specific synthetic data from algorithm 1, we combined that in two ways and trained models. one is directly adding the domain specific back translated data to general parallel data without any actual domain data, second is to add domain specific back translated to the general parallel data along with actual domain parallel data. we evaluate our models on test data set provided by icon adap-mt 2020 shared task using widely used automatic mt evaluation metric called bleu @xcite . the models presented in 6 are trained based on data-based approach only. however, the model which used the domain specific synthetic parallel data along with actual domain parallel data outperformed other models. the first model is m1-gen which is trained on only general parallel data this is same for both the domains(chemistry and artificial intelligence) it performed well on general data but not on chemistry and ai data for both directions telugu->hindi and hindi->telugu. then the second model is a domain adaption model where we combine the available little amount of domain parallel data with general data(m2-gen+chem and m2-gen+ai). these models outperformed the general model on chemistry(9.5 to 11.8) and ai(8.7 to 10.3) domains though there is a decrease of bleu score on general data for telugu->hindi, this pattern is same for hindi->telugu as well. third model is adding domain specific synthetic data directly to the general data(m3-gen+back chem and m3-gen+back ai) which decreased the bleu score compared to domain adaptation model(m2 for both the domains) but it's improved little bit from general model. for chemistry domain in telugu->hindi the bleu score decreased from 11.8 to 9.9, but it increased from general model(9.5 to 9.9). now the fourth model is the proposed approach which adds domain specific back translated data to the data used in initial domain adaptation model(gen+domain data) for both the domains(m4-gen+chem+back chem and m4-gen+ai+back ai). m4 model in both domains outperformed all other models. there is an increase 3.4 bleu points in chemistry for telugu->hindi from general model to proposed model which uses domain specific synthetic data. the thought behind the domain-specific back translation is selecting sentences where the unseen words (most of them are domain terms) are present. by doing this, the model implicitly learns the translation of domain terms and context around them accurately. to address this point we present two examples below which shows an overall improvement of domain specific text translation including domain terms. in ## conclusion and future work we presented an approach called domain specific back translation to produce synthetic data from available monolingual data which can be applied to any language pair for any domain. we did our experiments on two domains chemistry and artificial intelligence for hindi and telugu. the approach follows extracting out of domain words from large amount of general data with respect to particular domain (here chemistry and ai), then mining the sentences from domain monolingual data where these ood words are present. by doing this the system will learn to translate unknown words and domain terms properly. without adding direct monolingual data which contains lots of noise, we select only sentences where general ood words with respect to a domain are present. in this paper we showed how addition of domain specific back translated data to the general and little amount of domain data improved the translation performance in terms of bleu scores. from the results it has been observed that the proposed approach improving bleu score significantly. we would like to apply this generic approach to all possible indian languages and multiple domains with combination of similar domains. type/model sentence source(telugu source sentence) కావున, మనం రె ౖ బోనూయ్కి ల్ యేజ్ ను రె ండు విభినన్ మారా గ్ లలో రీఫోల్ డ్ చేయవచుచ్ . (kāvuna, manamṙ aibōn'yūkliyēj nu reṇḍu vibhinna mārgālalō rīphōlḍ cēyavaccu.) target(hindi target sentence) हम राइबोन्यू क्लएज़ को दो अलग -अलग तरीकों से रीफोल्ड कर सकते हैं । (ham raibonyookliez ko do alag -alag tareekon se reephold kar sakate hain .) model1(model trained on only general data) इस कारण , राइबोन्यू क्लीज को दो िविभन् तरीकों से रफोन कर सकते हैं । (is kaaran , raibonyookleej ko do vibhinn tareekon se riphon kar sakate hain .) model2 (model trained on general+chemistry data) तो , हम राबोन्यू क्लयोटाइड को दो अलग तरीकों से रफोल्ड कर सकते हैं । (to , ham raabonyookliyotaid ko do alag tareekon se riphold kar sakate hain .) model3(model trained on general+ back translated chemistry data) अत: हम रै बोन्यू क्लीयर को दो िविभन् तरीकों से रीफोल्ड कर सकते हैं । (at: ham raibonyookleeyar ko do vibhinn tareekon se reephold kar sakate hain .) model4(proposed model:trained on general + chemistry + back translated chemistry data) इस लए , हम राइबोन्यू क्लएज को दो अलग -अलग तरीकों से रीफोल्ड कर सकते हैं । (isalie , ham raibonyookliej ko do alag -alag tareekon se reephold kar sakate hain .)