Document Level NMT of Low-Resource Languages with Backtranslation (2020)

Abstract: This paper describes our system submission to WMT20 shared task on similar language translation. We examined the use of documentlevel neural machine translation (NMT) systems for low-resource, similar language pair Marathi−Hindi. Our system is an extension of state-of-the-art Transformer architecture with hierarchical attention networks to incorporate contextual information. Since, NMT requires large amount of parallel data which is not available for this task, our approach is focused on utilizing monolingual data with back translation to train our models. Our experiments reveal that document-level NMT can be a reasonable alternative to sentence-level NMT for improving translation quality of low resourced languages even when used with synthetic data.

## introduction with the widespread use of mt systems in commercial and research community, there is an increased attention to train nmt models for direct translation between language pairs other than english @xcite . this is because of the growing need to translate between pairs of similar languages without considering english as pivot language. the task is to overcome the challenge of limited availability of parallel data by exploiting the advantages of similarity between languages when building machine translation models. similar languages have the advantage of having some magnitude of common information such as lexical and semantic structures. a number of research studies have been published to exploit commonalities when translating text between close language pairs pourdamghani and @xcite ; @xcite ; @xcite . this paper describes our system submission at wmt shared similar language translation task 1 1 http://www.statmt.org/wmt20/similar.html which focuses on improving translation quality of similar languages in low-resource setting, the detail of task is provided in @xcite . this year's task includes five pairs of languages from three different language families i.e. indo-aryan, romance and south-slavic languages; we participated for hindi-marathi language pair. since we are using nmt which requires large bitext, we need to alleviate this specific problem of bitext shortage. @xcite introduced an approach to utilize monolingual data using back translation. this requires a machine translation system in opposite direction to generate synthetic parallel corpora from target side monolingual text. our work is an attempt to investigate the translation of a similar language pair (marathi-hindi) using document-level nmt and back translation. we participated under team name "fjwu nust". we submitted one constrained system i.e. we only used the parallel and monolingual data provided by wmt20 foot_0 organizers to train and evaluate our models. we train and evaluate nmt systems in both directions (i.e. hi⇒mr and mr⇒hi) but our submission to similar language shared task comprises of mr⇒hi systems only. the rest of the paper is structured as follows: in section 2 we give a brief background of documentlevel nmt, section 3 presents utilization of monolingual data, section 4 and 5 present our experimental setup and results. we conclude the paper in section 6. similarly, maruf and haffari (2018) used pretrained rnn encoder to attach global source and target context to sentence based nmt. @xcite has shown that integration of short context (2 sentences) outperforms existing cache based rnnsearch model. @xcite introduce a context aware nmt model with additional multihead attention component, in which they control and analyze the flow of information from the extended context to the translation model. @xcite studied the use of transformer based document-level models adoptable to novel (zero-resource) domains. they have shown the implicit domain adaptation of documentlevel nmt models trained on multi-domain data, is capable of capturing large context. the challenge of translating single sentences efficiently while keeping models insensitive to enlarge and noisy context is addressed by @xcite . to make general purpose context-aware mt, both for short and long sentences, they opt for having independent global and local context integration into sentence based nmt. ## utilizing monolingual data large amounts of monolingual resources are generally available for a multitude of languages. back translation is considered a well known approach to mitigate the need of large parallel corpora by automatically translating target language monolingual data to source language @xcite . back translation requires a mt system in opposite direction, where target side monolingual data is translated into source text to generate synthetic parallel training data. several techniques exists to utilize monolingual text for improving nmt @xcite @xcite . document-level models require parallel data with document boundaries for training and evaluation. as compared to sentence-level systems, data for building robust document-level models is significantly low resourced @xcite . wmt20 provides document-level distinctions for europarl v9, new-commentary v14 and rapid corpus. our training data is constrained to have only parallel and monolingual data provided by wmt20 shared task, the statistics of data are given in section 4.1. since, our system is build in marathi-hindi direction, we backtranslated hindi (news crawl2008-2019) monolingual data into marathi to generate bitext. this backtranslated data is than concatenated with parallel data made available by organizers, to train machine translation models. ## experimental setup for our primary submission we use document-level @xcite model, an extension of transformer with additional context attentions. for comparison with sentence-based nmt systems, a strong baseline using opennmt-py @xcite is first defined. for true comparison, the architecture and configurations of both the models are kept the same. ## results bleu score for w m t , dev and self test set is reported in in last rows (3 and 4) of ## summary this paper presented the "fjwu nust" system submitted to the similar language translation task at wmt20. the limited and out-of-domain parallel training data provided by organizers, emerged as a challenging task to train nmt models, whose quality is dependent on large data. we have utilized monolingual data with backtranslation along with available parallel data for training nmt system which incorporated contextaware hans on encoder side. our document-level systems outperformed sentence-level nmt systems, even in the absence of document-level corpora. this showed that document-level machine translation can be reasonable alternative of nmt, since it can deliver good quality translation for lowresource languages without requiring documentlevel parallel data.