Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data (2021)

Abstract: The scarcity of parallel data is a major obstacle for training high-quality machine translation systems for low-resource languages. Fortunately, some low-resource languages are linguistically related or similar to high-resource languages; these related languages may share many lexical or syntactic structures. In this work, we exploit this linguistic overlap to facilitate translating to and from a low-resource language with only monolingual data, in addition to any parallel data in the related high-resource language. Our method, NMT-Adapt, combines denoising autoencoding, back-translation and adversarial objectives to utilize monolingual data for low-resource adaptation. We experiment on 7 languages from three different language families and show that our technique significantly improves translation into low-resource language compared to other translation baselines.

## introduction while machine translation (mt) has made incredible strides due to the advent of deep neural machine translation (nmt) @xcite ) models, this improvement has been shown to be primarily in well-resourced languages with large available parallel training data. however with the growth of internet communication and the rise of social media, individuals worldwide have begun communicating and producing content in their native low-resource languages. many of these low-resource languages are closely related to a high-resource language. one such example are "dialects": variants of a language traditionally considered oral rather than written. machine translating dialects using models trained on the formal variant of a language (typically the highresource variant which is sometimes considered the "standardized form") can pose a challenge due to the prevalence of non standardized spelling as well significant slang vocabulary in the dialectal variant. similar issues arise from translating a low-resource language using a related high-resource model (e.g., translating catalan with a spanish mt model). while an intuitive approach to better translating low-resource related languages could be to obtain high-quality parallel data. this approach is often infeasible due to lack specialized expertise or bilingual translators. the problems are exacerbated by issues that arise in quality control for low-resource languages @xcite . this scarcity motivates our task of learning machine translation models for low-resource languages while leveraging readily available data such as parallel data from a closely related language or monolingual data in the low-resource language. 1 the use of monolingual data when little to no parallel data is available has been investigated for machine translation. a few approaches involve synthesising more parallel data from monolingual data using backtranslation @xcite or mining parallel data from large multilingual corpora @xcite @xcite . we introduce nmt-adapt, a zero resource technique that does not need parallel data of any kind on the low resource language. we investigate the performance of nmt-adapt at translating two directions for each low-resource language: (1) low-resource to english and (2) english to low-resource. we claim that translating into english can be formulated as a typical unsupervised domain adaptation task, with the highresource language as the source domain and the 803 related low-resource, the target domain. we then show that adversarial domain adaptation can be applied to this related language translation task. for the second scenario, translating into the lowresource language, the task is more challenging as it involves unsupervised adaptation of the generated output to a new domain. to approach this task, nmt-adapt jointly optimizes four tasks to perform low-resource translation: (1) denoising autoencoder (2) adversarial training (3) high-resource translation and (4) low-resource backtranslation. we test our proposed method and demonstrate its effectiveness in improving low-resource translation from three distinct families: (1) iberian languages, (2) indic languages, and (3) semitic languages, specifically arabic dialects. we make our code and resources publicly available. foot_1 2 related work zero-shot translation our work is closely related to that of zero-shot translation @xcite @xcite . however, while zero-shot translation translates between a language pair with no parallel data, there is an assumption that both languages in the target pair have some parallel data with other languages. as such, the system can learn to process both languages. in one work, @xcite improved zero-shot translation using monolingual data on the pivot language. however, in our scenario, there is no parallel data between the low-resource language and any other language. in other work, @xcite showed that adding adversarial training to the encoder output could help zero shot training. we adopt a similar philosophy in our multi-task training to ensure our low-resource target is in the same latent space as the higher-resource language. unsupervised translation a related set of work is the family of unsupervised translation techniques; these approaches translate between language pairs with no parallel corpus of any kind. in work by @xcite ; @xcite , unsupervised translation is performed by training denoising autoencoding and backtranslation tasks concurrently. in these approaches, multiple pretraining methods were proposed to better initialize the model @xcite @xcite . different approaches were proposed that used parallel data between x-y to improve unsupervised translation between x-z @xcite @xcite . this scenario differs from our setting as it does not assume that y and z are similar languages. these approaches leverage a cross-translation method on a multilingual nmt model where for a parallel data pair (s x ,s y ), they translate s x into language z with the current model to get s z . then use (s y ,s z ) as an additional synthesized data pair to further improve the model. @xcite experiment using multilingual cross-translation on low-resource languages with some success. while these approaches view the parallel data as auxiliary, to supplement unsupervised nmt, our work looks at the problem from a domain adaptation perspective. we attempt to use monolingual data in z to make the supervised model trained on x-y generalize to z. leveraging high-resource languages to improve low-resource translation several works have leveraged data in high-resource languages to improve the translation of similar low-resource languages. @xcite showed that it is beneficial to mix the limited parallel data pairs of low-resource languages with high-resource language data. @xcite proposed selecting high-resource language data with lower perplexity in the low-resource language model. @xcite created synthetic sentence pairs by unsupervised machine translation, using the high-resource language as a pivot. however these previous approaches emphasize translating from the low-resource language to english, while the opposite direction is either unconsidered or shows poor translation performance. @xcite trained multilingual translation and denoising simultaneously, and showed that the model could translate languages without parallel data into english near the performance of supervised multilingual nmt. similar language translation similar to our work, there have been methods proposed that leverage similar languages to improve translation. @xcite generated synthetic english-dialect parallel data from english-main language corpus. however, this method assumes that the vocabulary in the main language could be mapped word by word into the dialect vocabulary, and they calculate the corresponding word for substitution using localized projection. this approach differs from our work in that it relies on the existence of a seed bilingual lexicon to the dialect/similar language. additionally, the approach only considers translating from a dialect to english and not the reverse direction. other work trains a massively multilingual many-to-many model and demonstrates that high-resource training data improves related lowresource language translation @xcite . in other work, lakew et al. (2018) compared ways to model translations of different language varieties, in the setting that parallel data for both varieties is available, the variety for some pairs may not be labeled. another line of work focus on translating between similar languages. in one such work, @xcite learned a character-based cipher model. in other work, @xcite improved unsupervised translation between the main language and the dialect by separating the token embeddings into pivot and private parts while performing layer coordination. ## method we describe the nmt-adapt approach to translating a low-resource language into and out of english without utilizing any low-resource language parallel data. in section 3.1, we describe how nmt-adapt leverages a novel multi-task domain adaptation approach to translating english into a low-resource language. in section 3.2, we then describe how we perform source-domain adaptation to translate a low-resource language into english. finally, in section 3.3, we demonstrate how we can leverage these two domain adaptations, to perform iterative backtranslation -further improving translation quality in both directions. ## conclusion we presented nmt-adapt, a novel approach for neural machine translation of low-resource languages which assumes zero parallel data or bilingual lexicon in the low-resource language. utilizing parallel data in a similar high resource language as well as monolingual data in the low-resource language, we apply unsupervised adaptation to facilitate translation to and from the low-resource language. our approach combines several tasks including adversarial training, denoising language modeling, and iterative back translation to facilitate the adaptation. experiments demonstrate that this combination is more effective than any task on its own and generalizes across many different language groups.