Improving Domain Robustness in Neural Machine Translation with Fused Topic Knowledge Embeddings (2023)

Abstract: Domain robustness is a key challenge for Neural Machine Translation (NMT). Translating text from a different distribution than the training set requires the NMT models to generalize well to unseen domains. In this work we propose a novel way to address domain robustness, by fusing external topic knowledge into the NMT architecture. We employ a pretrained denoising autoencoder and fuse topic information into the system during continued pretraining, and finetuning of the model on the downstream NMT task. Our results show that incorporating external topic knowledge, as well as additional pretraining can improve the out-of-domain performance of NMT models. The proposed methodology meets state-of-the-art on out-of-domain performance. Our analysis shows that a low overlap between the pretraining and finetuning corpora, as well as the quality of topic representations help the NMT systems become more robust under domain shift.

## introduction neural machine translation (nmt) has achieved impressive performance over the last few years when trained on large-scale data @xcite . this success relies heavily on the availability of such data. the use of deep neural models has become the dominant approach for translation systems. however, it is not always possible to obtain neither parallel nor monolingual domain-specific data. most approaches for improving domain robustness in nmt assume that the target domains are known in advance, and a significant amount of data is available from the target domain. in such cases, the dominant approach for addressing domain mismatch is domain adaptation. however, when building translation systems and, as in many real-life scenarios, the target domains cannot always be known a priori. @xcite were the first to identify domain mismatch as one of the main challenges of nmt. it is important therefore to develop translation systems that can generalize to domains unseen during training and thus be robust even under domain shift, as no target-domain data can be seen during training. furthermore, even if nmt systems are trained on large-scale data, it is always possible that new topics or domains will emerge over time. these new domains make it difficult to maintain large translation systems, since these would require additional training on the new domains. a typical example is the outbreak of covid-19, which intruded into everyday life and affected millions of peoples' lives. keeping translation models up-to-date with such emerging topics is practically difficult, due to the limited availability of parallel data @xcite . in this work we focus on the problem of improving robustness under domain shift in nmt. a domain is defined by a corpus extracted from a specific source, and may differ from other domains in terms of topic, genre, level of formality, etc. @xcite . to this end, we improve domain robustness by incorporating external topic knowledge into the nmt models. we employ a denoising autoencoder that has been pretrained on masked language modeling (mlm) using monolingual data and thus has not been exposed to any parallel data of the unseen test domains during training. moreover, we train a distributional topic model using monolingual source-side data and subsequently extract a topic feature vector for each token in the vocabulary. we incorporate this external topic information during continuing the autoencoder's monolingual pretraining, and also during finetuning it on the downstream task of nmt. to the best of our knowledge, this is the first work studying the contribution of topic modeling for domain robustness in nmt. our key contribution is that we integrate external topic information into the nmt models, meeting state-of-the-art results for both in-and outof-domain performance. our analysis shows that both the quality of topic vectors and also the overlap between the pretraining and finetuning corpora are key factors towards improving domain robustness. our results show that the proposed methodology improves domain robustness across two of the five experiments we conducted. ## mbartopic in this section we introduce mbartopic, our proposed system in order to improve domain robustness in nmt. we employ a sequence-to-sequence system and initialize its weights using a pretrained model. we need to ensure that the pretrained model has not been exposed to any parallel data of the test domains. to this end, we use a multilingual denoising sequence autoencoder for initialization, and, in particular mbart @xcite , which has been trained on monolingual data only, and on a different task than nmt. assuming a corpus d consisting of sub-corpora d j , where each d j is a set of monolingual text samples d @xmath0, x 2 , ..., x n ), as well as a noising function g that corrupts text, the mbart model is trained to reconstruct x i from g(x i ). therefore, during pretraining it learns to maximize the log-likelihood of predicting the input x, given a noisy variant of it, as follows: we employ this pretrained system and finetune it on the downstream nmt task. we feed the parallel domain-specific data to the encoder and the decoder, while also adding the special id token of the source and target languages, respectively. during finetuning on nmt the model learns to minimize the cross-entropy loss function: ## experimental setup we compare five different systems: ## results for each experiment, we train all systems on one domain at a time and evaluate them on the same (in-domain) and also on the rest four domains (out-of-domain). in we observe that training the transformer base model (baseline) yields better results across almost all experiments, compared to training the mbart architecture from scratch (randominit). this finding is expected given the large difference in the number of parameters between the two models and the relatively small amount of data we used to train the systems from scratch. moreover, initializing the network with pre-fitted weights (mbart-ft) is shown to achieve a significant gain in performance, compared to the results of randominit and base-line, across all experiments, for both in-and out-of-domain translation. this observation stands in agreement with related literature indicating improvements in downstream nlp tasks when initializing the models with pretrained weights, due to transferring general knowledge to them @xcite , which contributes positively towards out-of-domain robustness. comparing the mbart-pt-ft system to not performing any additional pretraining, as in mbart-ft, it becomes evident that the extra pretraining updates improve the domain robustness of the law and medical experiments. these models have a significant increase in their out-of-domain performance, while the medical system seems to become more general after continuing the pretraining, given the decrease to its in-domain score. on the other hand, additional pretraining seems not to help the performance of the it, koran and subtitles experiments. we argue in section 6.1 that this finding is related to the overlap between the monolingual corpus used for pretraining and the finetuning domains. furthermore, we compare mbartopic to mbart-pt-ft and observe that incorporating the topic knowledge to the system improves the out-of-domain translation for the it, medical and subtitles experiments. on the other hand, the topic knowledge fusion has a slight decrease in the robustness of the law and koran experiments. we would like to point out that for continuing the pretraining and for training the topic model, we used monolingual data from the news domain, which are relatively small and does not generalize well for all possible topics. these data are also biased towards specific domains, as per our overlap study in section 6.2. we hypothesize that using large scale web crawled data for learning topic embeddings and continuing pretraining, e.g the mbart pre-training data, might help get rid of the side effect of knowledge overriding as suggested by the analysis, resulting in higher quality topic representations. we leave the investigation of this hypothesis to future work, since we were limited by computational constraints. finally, we provide some example translations of our systems in the out-of-domain setting foot_1 . system medical id avg. medical ood smt @xcite 58.4 11.8 nmt @xcite 61.5 11.7 nmt+rc+sr+nc @xcite 60.8 13.1 mle w/ ls + mrt @xcite 58.8 12.0 pretrained @xcite 63.1 17.6 pretrained + cp @xcite 63 ## contributing factors to domain robustness to understand what contributes to the out-of-domain performance gains, we conduct an analysis of the results presented in section 5. ## conclusion in this work we propose mbartopic, a novel model for improving domain robustness in nmt with integrated external topic knowledge. this is the first work studying the contribution of topic information towards improving domain robustness in nmt. we use a sequence-tosequence model, and specifically a pretrained multilingual denoising autoencoder. we train a distributional topic model on source-side monolingual data and integrate this topic knowledge to the encoder of the nmt system. we do that by extracting sentence-level topic features and subsequently combining them with the word embeddings of the each input token. in our approach we continue the pretraining of the denoising model using source-side monolingual corpora, and then finetune it on the downstream nmt task, using domain-specific parallel data. we incorporate the external topic features into both the additional pretraining and also during finetuning. our results show that the proposed method can improve the domain robustness of our experiments and meets state-of-the-art results in the out-of-domain performance. our analysis suggests that additional self-supervised pretraining with a low overlap between the pretraining and finetuning corpora can be an important factor to the domain robustness of nmt systems. finally, we show that smaller distances among the topic vectors of domain-specific words result in an increase in the out-of-domain performance. in the future, we plan to investigate the contribution of topic knowledge when it is fused into both the encoder and decoder of the nmt system. we also plan to analyze the system performance on a leave-one-out scenario, when finetuned on multiple domains and evaluated on an unseen one.