Terminology-Aware Sentence Mining for NMT Domain Adaptation: ADAPT ’s Submission to the Adap- MT 2020 E nglish-to- H indi AI Translation Shared Task (2020)

Abstract: This paper describes the ADAPT Centre’s submission to the Adap-MT 2020 AI Translation Shared Task for English-to-Hindi. The neural machine translation (NMT) systems that we built to translate AI domain texts are state-of-the-art Transformer models. In order to improve the translation quality of our NMT systems, we made use of both in-domain and out-of-domain data for training and employed different fine-tuning techniques for adapting our NMT systems to this task, e.g. mixed fine-tuning and on-the-fly self-training. For this, we mined parallel sentence pairs and monolingual sentences from large out-of-domain data, and the mining process was facilitated through automatic extraction of terminology from the in-domain data. This paper outlines the experiments we carried out for this task and reports the performance of our NMT systems on the evaluation test set.

## introduction adapt centre participated in the adap-mt 2020 translation shared task foot_0 of the 17th international conference on natural language processing (icon 2020). 2 this task aims at evaluating machine translation (mt) systems on the translation of documents from two domains (ai and chemistry) involving low-resource indic languages. the task addresses a number of translation directions, and we participated in the english-to-hindi translation task and focused on translating the ai texts only. to make the readers familiar with the ai translation task and to understand the challenges of this task, we show a couple of sentences from the blind test set in our mt systems are transformer models @xcite which were trained using the marian-nmt toolkit. 3 in this work, we applied different data augmentation and domain adaptation techniques to train our models, such as using synthetic data from target-side monolingual data through the use of back-translation @xcite , mixed fine-tuning @xcite and on-the-fly model adaption @xcite . as for the latter two approaches, we mined sentences and sentence pairs from large out-of-domain monolingual and parallel corpora, respectively, based on domain terms appearing in the in-domain data. note that the terms were extracted automatically from the in-domain data. this remainder of the paper is organized as follows. section 2 presents our approaches. we describe the resources we utilized for training in section 3. section 4 presents the results obtained, and section 5 concludes our work with avenues for future work. ## data used and training setups for building our baseline models (forward and backward), we used only the bilingual data provided by the task organisers. as for hindi monolingual sentences for back-translation, we sampled them from ai4bharat-indicnlp corpus @xcite . the out-of-domain parallel data is compiled from a variety of existing sources, e.g. opus foot_6 @xcite , and after applying standard cleaning procedures including applying a language identifier foot_7 we are left with just over 1.1 million parallel sentence pairs. as pointed out earlier, our nmt systems are transformer models. the tokens of the training, evaluation and validation sets are segmented into sub-word units using byte-pair encoding (bpe) @xcite , and bpe is applied individually on the source and target languages. from our experiences @xcite @xcite in the participation in the recent shared translation tasks @xcite @xcite involving lowresource language pairs and domains, we found that the following configuration usually leads to the best results in our low-resource translation settings: (i) the bpe vocabulary size: 6,000, (ii) the sizes of the encoder and decoder layers: 4 and 6, respectively, and (iii) learning-rate: 0.0003. as for the remaining hyperparameters, we followed the recommended best setup from @xcite . the early stopping criterion is based on cross-entropy; however, the final nmt system is selected as per the highest bleu score on the validation set. the beam size for search is set to 6. we make our final nmt model with ensembles of 8 models that are sampled from the training run. ## experiments and results source-target sentence pairs were mined from out-of-domain training bitexts for mixed finetuning (see section 2.2). the number of sentence pairs that have been mined is 167,234. we also augmented the in-domain parallel corpus via oversampling in-domain sentences, and by this, the size of the in-domain bitexts becomes 97,440. we finally fine-tuned base2 and base3 on the training data that is a mixture of (augmented) in-domain and (mined) out-of-domain data. the bleu scores of the mt systems (base2 + mixed ft and base3 + mixed ft) which are the results of the fine-tuning process are presented in the fourth and fifth rows of following the method described in section 2.3, we mined english sentences (a total of 27,644 sentences) from a large monolingual corpus (cf. section 2.3) given the list of terms (a total of 356 terms) appearing in the test set. then, sosc was created by translating these mined english sentences into hindi using the respective mt system. finally, the best mt systems (base2 + mixed ft or base3 + mixed ft) were fine-tuned on the resultant sosc. the bleu scores of the adapted mt systems on the test set are shown in the last rows of as above, we created the adapted mt systems for the blind test set which consists of 401 sentences. our terminology extraction model identified 1,599 ai terms in the blind test set. we mined 98,009 english sentences from the large monolingual data given the list of terms. we followed the approach described above for fine-tuning our best two models (base2 + mixed ft and base3 + mixed ft) in order to translate the blind test set sentences. the bleu scores of our mt systems on the blind test set, which the task organisers published, are shown in ## conclusion in this paper, we described our mt systems that were submitted to the adap-mt 2020 ai translation shared task. we presented our results obtained at the time of development of our mt systems. in order to adapt our mt systems to translate texts of ai domains, we subsequently applied two existing fine-tuning techniques while using a term extraction model in the translation pipeline for mining sentences similar to the domain and style of those of the ai data. we showed that, in the case of limited in-domain training data, both out-of-domain data which are selected via term-based mining protocol and in-domain data are useful for fine-tuning model parameters, which essentially provides our best results in this translation task. furthermore, making use of synthetic parallel data in training also greatly increased the performance of our mt systems. as for the shared task's system rankings, our three submissions run3, run2 and run1 secured second, third and fourth positions, respectively. in future, we aim to apply our strategy to other domains and language pairs.