The ADAPT Centre’s Neural MT Systems for the WAT 2020 Document-Level Translation Task (2020)

Abstract: In this paper we describe the ADAPT Centre’s submissions to the WAT 2020 document-level Business Scene Dialogue (BSD) Translation task. We only consider translating from Japanese to English for this task and we use the MarianNMT toolkit to train Transformer models. In order to improve the translation quality, we made use of both in-domain and out-of-domain data for training our Machine Translation (MT) systems, as well as various data augmentation techniques for fine-tuning the model parameters. This paper outlines the experiments we ran to train our systems and report the accuracy achieved through these various experiments.

## introduction we participated in the wat 2020 1 @xcite document-level bsd translation task and only submitted systems that translate from japanese-to-english (ja-to-en). our mt systems are transformer models @xcite which were trained using the marian-nmt toolkit. 2 in this work, we applied different domain adaptation techniques, such as using synthetic data from source-and target-side monolingual data through the use of forward-and back-translation @xcite @xcite and out-of-domain parallel data to train our models. as far as fine-tuning the model parameters is concerned, we experimented with conventional fine-tuning which consists of fine-tuning on in-domain data only, mixed fine-tuning and lastly document-level fine-tuning. this paper is organised as follows. firstly, the data used and processing steps are described in section 2. next, the methods for training the baseline mt systems are described in section 3, and the results obtained from our mt systems are presented in section 4. finally, section 5 concludes our work with the avenues for the future work. ## data and preprocessing this section outlines the corpora used and the steps that were taken to preprocess the data for training. ## the baseline mt systems we started off by training three baseline models (b1, b2 and b3) and the best baseline model was used to experiment with different data augmentation methods when fine-tuning model parameters. these methods will be described in detail in section 4. the first baseline model (b1) was trained on the bsd, jesc and opensubtitles corpora combined. as mentioned earlier, we used the mariannmt toolkit to train our transformer models. the setup described in @xcite was used as is for b1, more specifically the mini-batch size for validation was 64 and the learning rate 0.0003. the changes to this setup for the other two baseline setups (b2 and b3) are discussed below. the setup differs since bpe with a vocabulary of 32,000 was applied to the training set of b1 and a vocabulary size of 6,000 was used for the other two baseline models (b2 and b3). we obtained the bleu @xcite scores to evaluate these baseline mt systems on the evaluation test set and the scores are presented in the second baseline model (b2) was trained on the same training set as b1. the third baseline model's (b3) training set consisted of the targetside original synthetic corpus added to the training set of b1 (cf. ## improving the baseline mt systems the bleu score of each of the baseline models described in section 3 is shown in in order to improve upon the scores achieved by the baseline model b2, we fine-tuned the parameters of the model. this was done by restarting training on model b2, after initial training has ended, with a newly selected corpus. we implemented four different scenarios for fine-tuning the parameters. scenario 1: the first scenario is the most basic, where we simply performed conventional finetuning of the model parameters on in-domain data only, namely the bsd training set. scenario 2: in the second scenario we implemented mixed fine-tuning of model parameters, where fine-tuning is conducted on the training data that consists of both in-domain data and out-ofdomain data as described in @xcite . the in-domain data was augmented by oversampling the bsd training set 50 times and the out-ofdomain data is a mixture of jesc and opensubtitles. scenario 3: as for the third scenario, source-side monolingual sentences were mined that are similar in styles to the bsd test set sentences. we followed @xcite and @xcite in order to mine those sentences from large monolingual data that could be beneficial for finetuning the original nmt models. we identified terms in the test set to be translated. for this, we followed the monolingual terminology extraction methods described in @xcite , which used a large corpus that is generic in nature as a reference corpus. in our setup, we used the source-side of the authentic training bitexts on which our nmt system (b2) was trained as the reference corpus. the intuition is to extract those terminological expressions from the test set that do not occur or rarely occur in the training data and are more indicative of the test corpus. given the list of extracted terms, we mined sentences from the large monolingual corpus foot_8 mentioned in section 2. the japanese source sentences (a total of 153,402 sentences) that have been mined were translated into english using b2 to create synthetic data (i.e. source-side original synthetic corpus (sosc)) to be used for fine-tuning our baseline model. thus, in scenario 3, the model parameters were fine-tuned on the out-of-domain data only. scenario 4: finally, in scenario 4, we once again tested the mixed fine-tuning strategy. in this case, the in-domain data consists of the bsd training set and the out-of-domain data consists of sosc. the results obtained from the different scenarios are shown in as for scenario 1, fine-tuning model parameters on in-domain-data only does not work well and the corresponding mt system performs poorly in comparison to the performance of other mt systems (cf. the wat 2020 shared task organisers reported the evaluation results in terms of the bleu and ribes @xcite metrics. as for ribes, scenario 1 once again is found to be less effective, and the corresponding mt system produces a low score on the test set. however, the difference between the lowest and highest ribes scores is much smaller than that of the bleu scores. another notable difference is that scenario 4 seems to be most effective since the mt system of this setup provides us the highest ribes score on the test set. this is a contrasting outcome to the one with the bleu metric, where scenario 2 provided the highest bleu score on the test set. we submitted translations of the mt system of the scenario 2 setup for the human evaluation task conducted by the shared task organisers since it produced the highest bleu score on the evaluation test set. as can be seen from the last column of ## conclusion in this paper, we described our mt systems that were submitted to the wat 2020 document-level business scene dialogue translation shared task. we presented the wat 2020 official results that we obtained by submitting the translations of our mt systems. we showed that, in the case of limited in-domain training data, both in-domain and out-of-domain data is useful for fine-tuning model parameters, which essentially provides the best results in this translation task. furthermore, making use of synthetic parallel data in training also greatly increased the performance of our mt systems. in future, we aim to exploit document-level syntactic context in the fine-tuning step. we also aim to explore increasing training batch size at the finetuning step as this may capture wider contexts during training.