Vocabulary Adaptation for Domain Adaptation in Neural Machine Translation (2020)

Abstract: Neural network methods exhibit strong performance only in a few resource-rich domains. Practitioners therefore employ domain adaptation from resource-rich domains that are, in most cases, distant from the target domain. Domain adaptation between distant domains (e.g., movie subtitles and research papers), however, cannot be performed effectively due to mismatches in vocabulary; it will encounter many domain-specific words (e.g., “angstrom”) and words whose meanings shift across domains (e.g., “conductor”). In this study, aiming to solve these vocabulary mismatches in domain adaptation for neural machine translation (NMT), we propose vocabulary adaptation, a simple method for effective fine-tuning that adapts embedding layers in a given pretrained NMT model to the target domain. Prior to fine-tuning, our method replaces the embedding layers of the NMT model by projecting general word embeddings induced from monolingual data in a target domain onto a source-domain embedding space. Experimental results indicate that our method improves the performance of conventional fine-tuning by 3.86 and 3.28 BLEU points in En-Ja and De-En translation, respectively.

## introduction the performance of neural machine translation (nmt) models remarkably drops in domains different from the training data @xcite . since a massive amount of parallel data is available only in a limited number of domains, domain adaptation is often required to employ nmt in practical applications. researchers have therefore developed fine-tuning, a dominant approach for this problem @xcite @xcite @xcite ( § 2). assuming a massive amount of source-domain and small amount of target-domain parallel data, fine-tuning adjusts the parameters of a model pre-trained in the source-domain to the target domain. however, in fine-tuning, inheriting the embedding layers of the model pre-trained in the source domain causes vocabulary mismatches; namely, a model can handle neither domain-specific words that are not covered by a small amount of targetdomain parallel data (unknown words) nor words that have different meanings across domains (semantic shift). moreover, adopting the standard subword tokenization @xcite accelerates the semantic shift. targetdomain-specific words are often finely decomposed into source-domain subwords (e.g., "alloy" → " all" + "o" + "y"), which introduces improper subword meanings and hinders adaptation ( to resolve these vocabulary-mismatch problems in domain adaptation, we propose vocabulary adaptation (figure 1 ), a method of directly adapting the vocabulary (and embedding layers) of a pre-trained nmt model to a target domain, to perform effective fine-tuning ( § 3). given an nmt model pretrained in a source domain, we first induce a wide coverage of target-domain word embeddings from target-domain monolingual data. we then fit the obtained target-domain word embeddings to the embedding space of the pre-trained nmt model by inducing a cross-domain projection from the targetdomain embedding space to the source-domain embedding space. to perform this cross-domain embedding projection, we explore two methods: cross-lingual @xcite and cross-task embedding projection @xcite . we evaluate fine-tuning with the proposed vocabulary adaptation for two domain pairs: 1) from jesc @xcite to aspec @xcite for english to japanese translation (@xmath0. experimental results demonstrate that our vocabulary adaptation improves the bleu scores @xcite of fine-tuning @xcite by 3.86 points (21.45 to 25.31) for @xmath1.28 points (24.59 to 27.87) for @xmath2. moreover, it shows further improvements when combined with back-translation @xcite . the contributions of this paper are as follows. ## related work in this section, we first review two approaches to supervised domain adaptation in nmt: multi-domain learning and fine-tuning. we then introduce unsupervised domain adaptation using target-domain monolingual data and approaches to unknown word problems in nmt. multi-domain learning induces an nmt model from parallel data in both source and target domains @xcite @xcite . since this approach requires training with a massive amount of source-domain parallel data, the training cost becomes problematic when we perform adaptation to many target domains. fine-tuning (or continued learning) is a standard domain adaptation method in nmt. given an nmt model pre-trained with a massive amount of sourcedomain parallel data, it continues the training of this pre-trained model with a small amount of target-domain parallel data @xcite @xcite @xcite . due to the small cost of training, research trends have shifted to fine-tuning from multi-domain learning. recent studies focus on model architectures, training objectives, and strategies in training. meanwhile, no attempts have been made to resolve the vocabulary mismatch problem in domain adaptation. unsupervised domain adaptation exploits targetdomain monolingual data to train a language model to support the model's decoder in generating natural sentences in a target domain @xcite . data augmentation using back-translation @xcite is another approach to using targetdomain monolingual data. these approaches can partly address the problem of semantic shift. however, it is possible that the source-domain encoder will fail to handle targetdomain-specific words. in such cases, a decoder with the target-domain language model becomes less helpful in the former approach, and the generated pseudo-parallel corpus has low-quality sentences on the encoder side in the latter approach. handling unknown words has been extensively studied for nmt since the vocabulary size of an nmt model is limited due to practical requirements (e.g., gpu memory) @xcite . the current standard approach to the unknown word problem is to use token units shorter than words such as characters @xcite and subwords @xcite to handle rare words as a sequence of known tokens. however, more drastic semantic shifts will occur for characters or subwords than for words because they are shorter than words and naturally ambiguous. besides these studies mentioned above, aji et al. (2020) reported that transferring embeddings and vocabulary mismatches between parent and child models significantly affected the performance of models also in cross-lingual transfer learning. in this study, we aim to provide pre-trained nmt models with functionality that directly handles both target-domain-specific unknown words and semantic shifts by exploiting cross-domain embeddings learned from target-domain data. ## vocabulary adaptation for domain adaptation in nmt as we have discussed ( § 1), vocabulary mismatches between source and target domains are the important challenge in domain adaptation for nmt. this section proposes fine-tuning-based methods of directly resolving this problem. although our methods are applicable to any nmt model with embedding layers, we assume here subword-based encoder-decoder models @xcite for clarity. ## experimental setup we conducted fine-tuning with our vocabulary adaptation for domain adaptation in @xmath3. in what follows, we describe the setup of our experiments. ## analysis 6.1 translation examples ft-srcv and ft-srcv + bt often failed to translate target-domain-specific words that were tokenized into short subwords. in such cases, the models tended to ignore or transliterate them. for instance, the @xmath4." moreover, in the @xmath5, the decomposed target-domain-specific words "脊柱 (spinal)" and "狭窄症 (stenosis)" contained targetdomain-specific subwords such as "脊" and "窄." the models without vocabulary adaptation also failed to handle these subwords when both the source-domain training set and the target-domain 100k training set rarely contained them. meanwhile, va-llm + bt successfully translated both of the cases with the help of targetdomain monolingual data. these examples imply the difficulty in translating target-domain-specific words without vocabulary adaptation. we observed that va-llm + bt generated various target-domain-specific words. to quantitatively confirm this, we calculated the percentage of distinct words included in both the generated outputs and the references. the outputs in @xmath6, ft-srcv + bt, and ft-srcv contained 57.9%, 53.4%, and 49.5% of distinct words in the references, respectively. ## conclusion in this study, we tackled the vocabulary mismatch problem in domain adaptation for nmt, and we proposed vocabulary adaptation, a simple but direct solution to this problem. it adapts the vocabulary of a pre-trained nmt model to a target domain for performing effective fine-tuning. regarding domains as independent languages/tasks, our method makes wide-coverage word embeddings induced from target-domain monolingual data be compatible with a model pre-trained in a source domain. we explored two methods for projecting word embeddings across two domains: linear transformation and locally linear mapping (llm). the experimental results for english to japanese translation and german to english translation confirmed that our domain adaptation method with llm dramatically improved the translation performance. although the vocabulary adaptation was evaluated only for nmt, it is also applicable to a wider range of neural network models and tasks, and it can even be combined with existing fine-tuningbased domain adaptations. we will release all code to promote the reproducibility of our results. 11