Unsupervised Neural Machine Translation for Low-Resource Domains via Meta-Learning (2021)

Abstract: Unsupervised machine translation, which utilizes unpaired monolingual corpora as training data, has achieved comparable performance against supervised machine translation. However, it still suffers from data-scarce domains. To address this issue, this paper presents a novel meta-learning algorithm for unsupervised neural machine translation (UNMT) that trains the model to adapt to another domain by utilizing only a small amount of training data. We assume that domain-general knowledge is a significant factor in handling data-scarce domains. Hence, we extend the meta-learning algorithm, which utilizes knowledge learned from high-resource domains, to boost the performance of low-resource UNMT. Our model surpasses a transfer learning-based approach by up to 2-3 BLEU scores. Extensive experimental results show that our proposed algorithm is pertinent for fast adaptation and consistently outperforms other baselines.

## introduction unsupervised neural machine translation (unmt) leverages unpaired monolingual corpora for its training, without requiring an already labeled, parallel corpus. recently, the state of the art in unmt @xcite @xcite has achieved comparable performances against supervised neural machine translation (nmt) approaches. in contrast to supervised nmt, which uses a parallel corpus, training the unmt model requires a significant number of monolingual sentences (e.g., 1m-3m sentences). however, the prerequisite limits unmt's applicability to low-resource domains, especially for domain-specific document translation tasks. since gathering or creating those documents requires domain specific knowledge, the monolingual data themselves are scarce and expensive. in addition, the minority languages (e.g., uzbek and nepali) make the problem of data scarcity even worse. yet, unmt for low-resource domains is not an actively explored field. one naive approach is to train a model on high-resource domains (e.g., economy and sports) while hoping the model will generalize on an unseen low-resource domain (e.g., medicine). however, recent studies have shown that non-trivial domain mismatch can significantly cause low translation accuracy on supervised nmt tasks @xcite . another reasonable approach is transfer learning-particularly, domain adaptation-which has shown performance improvements in the supervised nmt literature @xcite . in this approach, the model is first pretrained using data from existing domains and then finetuned on a new domain. however, this approach can suffer from overfitting and catastrophic forgetting due to a small amount of training data and a large domain gap. as an effective method for handling a small amount of training data, meta-learning has shown its superiority in various nlp studies such as dialog generation, machine translation, and natural language understanding @xcite @xcite . in general, the metalearning approach is strongly affected by the number of different tasks where tasks are defined as languages or domains from the aforementioned studies. however, in practice, the previous studies may struggle to gather data to define tasks because they rely on a supervised model that requires labeled corpora. in this respect, we argue that applying a meta-learning approach to the unsupervised model is more feasible and achievable than the supervised model because it can define multiple different tasks with unlabeled corpora. therefore, we introduce a new meta-learning approach for unmt, called metaumt, for low-resource domains by defining each task as a domain. the objective of metaumt is to find the optimal initialization for the model parameters that can quickly adapt to a new domain even with only a small amount of monolingual data. as shown in fig. 1 (a), we define two different training phases, a meta-train and a meta-test phase, and simulate the domain adaption process to obtain optimally initialized parameters. specifically, the meta-train phase adapts model parameters to a domain while the meta-test phase optimizes the parameters obtained from the meta-train phase. after obtaining optimally initialized parameters through these two phases, we fine-tune the model using a target domain (i.e., a low-resource domain). although the initial parameters optimized through metaumt are suitable for adapting to a low-resource domain, these parameters may not fully maintain the knowledge of high-resource domains. concretely, in the meta-test phase, metaumt optimizes initial parameters using the adapted parameters; however, it discards meta-train knowledge used to update adapted parameters in the meta-train phase. therefore, instead of validating the same domain used in the meta-train phase, we intend to inject generalizable knowledge into the initial parameters by utilizing another domain in the meta-test phase. this prevents overfitting from the data scarcity issue. as shown in fig. 1 (b) , we propose an improved meta-learning approach called metagumt for lowresource unmt by explicitly infusing common knowledge across multiple source domains as well as generalizable knowledge from one particular domain to another. in other words, we do not only encourage the model to find the optimally initialized parameters that can quickly adapt to a target domain with low-resource data, but also encourage the model to maintain common knowledge (e.g., general words such as determiners, conjunctions, and pronouns) which is obtainable from multiple source domains. furthermore, due to a small amount of training data in a low-resource domain, the model can suffer from overfitting; however, we attempt to handle overfitting by leveraging generalizable knowledge that is available from one domain to another. our proposed meta-learning approach demonstrates consistent improvements over other baseline models. overall, our contributions can be summarized as follows: our study leverages two components from the natural language processing (nlp) domain: lowresource nmt and meta-learning. in this section, we discuss previous studies by concentrating on these two main components. ## unsupervised neural machine translation in this section, we first introduce the notation of the general unmt models. we then describe the three steps for the unmt task: initialization, language modeling, and back-translation. on these three steps, we illustrate how each step contributes to improving the performance of unmt. notations. we denote s and t as a source and a target monolingual language dataset. x and y represent the source and the target sentences from s and t . we assume the nmt model is parame- choosing a domain sampling another domain english sentences english sentences english sentences english sentences meta-test phase meta-train phase (e) update : (c) obtain by using one-step gradient : (d) compute each loss : (b) (a) english sentences english sentences english sentences english sentences english sentences english sentences english sentences english sentences english sentences english sentences terized by θ. we also denote m @xmath0, respectively, while denoting m @xmath1. initialization. a recent unmt model @xcite is based on a shared encoder and decoder architecture for the source and the target language. due to the shared encoder and decoder for each language, initializing the model parameters of the shared encoder and decoder is an important step for competitive performances @xcite @xcite . @xcite propose the xlm (cross-lingual language model) to initialize parameters, showing significantly improved performances for unmt. among various initialization methods, we leverage the xlm as our initialization method. language modeling. we use a denoising autoencoder @xcite to train the unmt model, reconstructing an original sentence from a noisy one in a given language. the objective function is defined as follows: back-translation. back-translation helps the model learn the mapping functions between the source and the target language by using only the monolingual sentences. for example, we sample a sentence x and y from source language s and target language t . to make pseudo-pair sentences from the sampled source sentence, we deduce the target sentence from the source sentence, such that @xmath2, resulting in the pseudo parallel sentence, i.e., (x, y ). similarly, we obtain (x , y), where x is the translation of a target sentence, i.e., m @xmath3. we do not back-propagate when we generate the pseudo-parallel sentence pairs. in short, the back-translation objective function is ## proposed approach this section first explains our formulation of a lowresource unsupervised machine translation task where we can apply a meta-learning approach. afterwards, we elaborate our proposed methods, metaumt and metagumt. we utilize the metalearning approach to address a low-resource challenge for unsupervised machine translation. moreover, we extend metaumt into metagumt to explicitly incorporate learned knowledge from multiple domains. ## experiments this section first introduces experiment settings and training details. afterwards, we show empirical results in various scenarios. ## conclusions this paper proposes a novel meta-learning approach for low-resource unmt, called metaumt, which leverages multiple source domains to quickly and effectively adapt the model to the target domain even with a small amount of training data. moreover, we introduce an improved method called metagumt, which enhances cross-domain generalization and maintains high-resource domain knowledge. we empirically show that our proposed approach consistently outperforms the baseline methods with a nontrivial margin. we believe that our proposed methods can be extended to semisupervised machine translation as well. in the future, we will further analyze other languages, such as uzbek and nepali, instead of languages like english and german.