Improving NMT via Filtered Back Translation (2020)

Abstract: Document-Level Machine Translation (MT) has become an active research area among the NLP community in recent years. Unlike sentence-level MT, which translates the sentences independently, document-level MT aims to utilize contextual information while translating a given source sentence. This paper demonstrates our submission (Team ID - DEEPNLP) to the Document-Level Translation task organized by WAT 2020. This task focuses on translating texts from a business dialog corpus while optionally utilizing the context present in the dialog. In our proposed approach, we utilize publicly available parallel corpus from different domains to train an open domain base NMT model. We then use monolingual target data to create filtered pseudo parallel data and employ Back-Translation to fine-tune the base model. This is further followed by fine-tuning on the domain-specific corpus. We also ensemble various models to improvise the translation performance. Our best models achieve a BLEU score of 26.59 and 22.83 in an unconstrained setting and 15.10 and 10.91 in the constrained settings for En->Ja & Ja->En direction, respectively.

## introduction neural machine translation @xcite has performed impressively in recent years, especially for high resource language pairs. however, one of the shortcomings while translating texts in the form of a paragraph or a document is that the inter-relations among sentences are ignored and the sentences are translated independently. document level mt @xcite @xcite aims to utilize these inter-sentential context information to deal with context-dependent 1 http://lotus.kuee.kyoto-u.ac.jp/wat/ wat2020/index.html phenomena such as coreference, lexical cohesion, and consistency, lexical disambiguation, etc. @xcite the meaning of a translated sentence can deviate from its originality when treated independently. wat 2020's @xcite ) document-level business scene dialogue (bsd) translation sub-task aims to foster research in the area of document-level mt. to tackle this task, we perform the following steps. firstly, we gather several publicly available english japanese corpus and combine them to train an open domain base model. then, we utilize the monolingual corpus in the target language to create the pseudo parallel corpus. since the generated pseudo parallel corpus might consist of noisy translated sentences, we use a sentence-level similarity-based filtration technique to filter out such pairs. we then fine-tune the base model on the filtered data followed by fine-tuning on in-domain parallel bsd foot_0 data. we also utilize checkpoint ensembles to further improve the translation performance. ## problem description this task aims to translate all the sentences in the bsd test file from @xmath0. participants could participate either in the constrained setting in which only the official bsd corpus needs to be used or in an unconstrained setting where other resources such as parallel corpora, monolingual corpora, and parallel dictionaries in addition to the official corpora could be utilized. we participate in both settings. bleu @xcite , ribes foot_1 and amfm @xcite are used as the official automatic evaluation metrics and are calculated on the tokenized version of translated and reference sentences using different tokenizers such as juman, kytea, mecab, & moses. ## related work there are two major directions in mt, which has attracted a lot of attention from the research community in recent years -document level mt & mt on low resource language pairs. numerous works have been proposed to tackle document-level mt @xcite . this area's work involves utilizing contexts on source, target, or both side and designing architectures using either the single @xcite or additional encoder @xcite to handle contextual information. some work in this area also tries to analyze the contextual errors @xcite . the work related to low resource language pairs involves making use of monolingual data to create pseudo parallel corpus using back translation @xcite , iterative back translation (hoang et al., 2018) & filtered back translation techniques (junczys-dowmunt, 2018; dou et al., 2020), etc. for filtering noisy pairs, imankulova et al. (2017) uses the round trip bleu score between true and synthetic sentences. wang et al. (2019) propose dynamic domain-data selection along with dynamic clean-data selection. ## system description we describe our proposed approach in this section. for the unconstrained setting, we first create base models by training n m t @xmath1. then, we use these trained n m t @xmath2. we then utilize the pseudo parallel data m t and m s along with equal amount of true parallel data to fine-tune n m t @xmath3. similarly, we use the pseudo parallel data m s and m t along with equal amount of true parallel data to fine-tune n m t @xmath4. this results in the creation of the back-translated (bt) models. in other settings, instead of utilizing the entire pseudo parallel data, we apply the filtering technique described below on these data to filter out noisy pairs. then we use these filtered pairs along with an equal amount of true parallel data to fine-tune the n m t @xmath5. this results in the creation of the filtered back-translated (fbt) models. we further fine-tune bt as well as fbt models on the bsd corpus. for the constrained setting, we train n m t @xmath6. we also experiment with fine-tuning filtering technique: we apply a naive filtering model based on sentence similarity to filter out noisy pseudo parallel data. given a monolingual source sentence s, we obtain the corresponding translated sentence t using the trained n m t @xmath7. we then apply muse (multilingual universal sentence encoder) @xcite to obtain the sentence embeddings of s and t. then cosine similarity is calculated on the obtained embeddings of s and t, and if the cosine score is below a certain threshold, we treat this pair as noisy. the threshold value is decided based on the cosine score on the entire monolingual data and its corresponding generated translations. we also utilize this filtering strategy to sample sentence pairs from the true parallel data. for this, we sort the entire true parallel data in decreasing order of similarity scores. then we remove pairs that contain the text in the same language in the source and target side using langid @xcite library. we also remove pairs where the same text is present on the source and target side. finally, we return the top n sentence pairs from the above data where n is the number of samples required from the true parallel data. ## conclusion we experimented with a variety of techniques in both constrained & unconstrained settings. for the constrained setting, fine-tuning mbart on the bsd corpus gave the best translation performance in both directions. thus, mbart can be fine-tuned for mt tasks, especially for low resource language pairs. for the unconstrained scenario, the models trained & fine-tuned using the pseudo-parallel corpus showed the best overall translation performance. we also showed that by using a simple ensemble technique of averaging different model checkpoints, the translation performance could be further improvised.