Pre-training via Leveraging Assisting Languages for Neural Machine Translation (2020)

Abstract: Sequence-to-sequence (S2S) pre-training using large monolingual data is known to improve performance for various S2S NLP tasks. However, large monolingual corpora might not always be available for the languages of interest (LOI). Thus, we propose to exploit monolingual corpora of other languages to complement the scarcity of monolingual corpora for the LOI. We utilize script mapping (Chinese to Japanese) to increase the similarity (number of cognates) between the monolingual corpora of helping languages and LOI. An empirical case study of low-resource Japanese-English neural machine translation (NMT) reveals that leveraging large Chinese and French monolingual corpora can help overcome the shortage of Japanese and English monolingual corpora, respectively, for S2S pre-training. Using only Chinese and French monolingual corpora, we were able to improve Japanese-English translation quality by up to 8.5 BLEU in low-resource scenarios.

## introduction neural machine translation (nmt) @xcite is known to give state-of-the-art (sota) translations for language pairs with an abundance of parallel corpora. however, most language pairs are resource poor (russian-japanese, marathi-english) as they lack large parallel corpora and the lack of bilingual training data can be compensated by by monolingual corpora. although it is possible to utilise the popular back-translation method @xcite , it is time-consuming to backtranslate a large amount of monolingual data. furthermore, poor quality backtranslated data tends to be of little help. recently, another approach has gained popularity where the nmt model is pre-trained through tasks that only require monolingual data @xcite . pre-training using models like bert @xcite have led to new state-of-the-art results in text understanding. however, bert-like sequence models were not designed to be used for nmt which is sequence to sequence (s2s). @xcite recently proposed mass, a s2s specific pre-training task for nmt and obtained new state-of-the-art results in low-resource settings. mass assumes that a large amount of monolingual data is available for the languages involved but some language pairs may lack both parallel and monolingual corpora and are "truly low-resource" and challenging. fortunately, languages are not isolated and often belong to "language families" where they have similar orthography (written script; shared cognates) or similar grammar or both. motivated by this, in this paper we hypothesize that we should be able to leverage large monolingual corpora of other assisting languages to help the monolingual pre-training of nmt models for the languages of interest (loi) that may lack monolingual corpora. wherever possible, we subject the pre-training corpora to script mapping which should help minimize the vocabulary and distribution differences, respectively, between the pre-training, main training (fine-tuning) and testing time datasets. this should help the already consistent pre-training and fine-tuning objectives leverage the data much better and thereby, possibly, boost translation quality. to this end, we experiment with aspec japanese-english translation in a variety of lowresource settings for the japanese-english parallel corpora. our experiments reveal that while it's possible to leverage unrelated languages for pretraining, using related languages is extremely important. we utilized chinese to japanese script mapping to maximize the similarities between the assisting languages (chinese and french) and the languages of interest (japanese and english). we show that only using monolingual corpora of chinese and french for pre-training can improve japanese-english translation quality by up to 8.5 bleu. the contributions of our work are as follows: 1. leveraging assisting languages: we give a novel study of leveraging monolingual corpora of related and unrelated languages for nmt pre-training. ## related work our research is at the intersection of works on monolingual pre-training for nmt and leveraging multilingualism for low-resource language translation. pre-training has enjoyed great success in other nlp tasks with the development of methods like bert @xcite . @xcite recently proposed mass, a new state-of-the-art nmt pre-training task that jointly trains the encoder and the decoder. our approach builds on the initial idea of mass, but focuses on complementing the potential scarcity of monolingual corpora for the languages of interest using relatively larger monolingual corpora of other (assisting) languages. on the other hand, leveraging multilingualism involves cross-lingual transfer @xcite which solves the low-resource issue by using data from different language pairs. @xcite showed the importance of transfer learning between languages belonging to the same language family but corpora might not always be available in a related language. a mapping between chinese and japanese characters @xcite was shown to be useful for chinese-japanese dictionary construction @xcite . mappings between scripts or unification of scripts @xcite can artificially increase the similarity between languages which motivates most of our work. ## proposed method: using assisting languages we propose a novel monolingual pre-training method for nmt which leverages monolingual corpora of assisting languages to overcome the scarcity of monolingual and parallel corpora of the languages of interest (loi). the framework of our approach is shown in figure 1 which consists of script mapping, data selection, pre-training and fine-tuning. ## experimental settings we conducted experiments on japanese-english (ja-en) translation in a variety of simulated lowresource settings using the "similar" assisting lan-guage pairs chinese (zh) and french (fr) and the "distant" assisting language pairs russian (ru) and arabic (ar). ## conclusion in this paper we showed that it is possible to leverage monolingual corpora of other languages to pretrain nmt models for language pairs that lack parallel as well as monolingual data. even if monolin-gual corpora for the languages of interest are unavailable, we can successfully improve translation quality by up to 8.5 bleu, in low-resource settings, using monolingual corpora of assisting languages. we showed that the similarity between the other (assisting) languages and the languages to be translated is crucial and leveraged script mapping wherever possible. in the future, we plan to experiment with even more challenging language pairs such as japanese-russian and attempt to leverage monolingual corpora belonging to diverse language families.we might be able to identify subtle relationships among languages and approaches to better leverage assisting languages for several nlp tasks.