The Source-Target Domain Mismatch Problem in Machine Translation (2021)

Abstract: While we live in an increasingly interconnected world, different places still exhibit strikingly different cultures and many events we experience in our every day life pertain only to the specific place we live in. As a result, people often talk about different things in different parts of the world. In this work we study the effect of local context in machine translation and postulate that this causes the domains of the source and target language to greatly mismatch. We first formalize the concept of source-target domain mismatch, propose a metric to quantify it, and provide empirical evidence for its existence. We conclude with an empirical study of how source-target domain mismatch affects training of machine translation systems on low resource languages. While this may severely affect back-translation, the degradation can be alleviated by combining back-translation with self-training and by increasing the amount of target side monolingual data.

## introduction the use of language greatly varies with the geographic location @xcite . even within places where people speak the same language @xcite , there is a lot of lexical variability due to change of style and topic distribution, particularly when considering content posted on social media, blogs and news outlets. for instance, while a primary topic of discussion between british sport fans is cricket, american sport fans are more likely to discuss other sports such as baseball @xcite . the effect of local context in the use of language is even more extreme when considering regions where different languages are spoken. despite the increasingly interconnected world we live in, people in different places tend to talk about different things. there are several reasons for this, from cultural differences due to geographic separation and history, to the local nature of many events we experience in our every day life. this phenomenon has not only interesting sociolinguistic aspects but it has also strong implications in machine translation (mt) @xcite . in particular, machine translation aims at automatically translating content in two languages that are often spoken in very distant geographic locations by people with rather different cultures. as of today, most mt research has been based on the often implicit assumption that content in the two languages is comparable. sentences comprising the parallel dataset used for training are assumed to cover the same topic distribution, regardless of the originating language. even when there exist a mismatch between the source and the target domain, the dataset creator is assumed to have made the effort to equalize the two distributions. the major contribution of this work is to raise awareness in the mt community that this assumption may not hold in many real world settings of interest, which often involve distant and low-resource language pairs and for content produced every day on the internet by means of blogs, social platforms and news outlets. the goal of an mt system is to translate source sentences sampled from the source domain to the target language. training an mt system on a comparable corpus may lead to poor generalization unless the test domain matches the domain of the training data. likewise, training on an uncurated corpus exhibiting mismatch between the source and the target domain may also work poorly when naïvely applying popular methods like back-translation @xcite . notice that there may very well be several factors contributing to such mismatch, such as change of register, different functional use of the text in the two languages, and difference in the quality of the data generation process of the two languages. regardless of what is contributing to the observed mismatch, it is important to be aware of its existance and effect on mt algorithms. the source-target domain mismatch (stdm) can be understood as an instance of multi-domain mt (see fig. 1 for an illustration and §3 for a formal definition), whereby part of the parallel dataset and the source monolingual dataset are "in-domain" because they originate from the source domain, and the remaining part of the parallel dataset as well as the target monolingual data are "out-of-domain". there already exist several techniques for domain adaptation, like domain tagging @xcite and dataset weighting @xcite @xcite , which are applicable and which we also employ in this work. however, these may not be enough to improve generalization on low resource languages because stdm effectively decreases the already scarce amount of useful (in-domain) parallel data, hindering good generalization. it is therefore important to quantify stdm ( §4) and consider how stdm affects methods that leverage monolingual data ( §5). for instance, stdm may negatively impact the effectiveness of back-translation because, even if the backward model was perfect, the backtranslated data is out-of-domain relative to the source domain from which we aim to translate. empirically we found that this is the case both in a controlled setting ( §6.1) as well as in realistic datasets ( §6.2). however, this issue can be compensated by adding more target-side monolingual data and by combining back-translation with selftraining @xcite . ## related work the observation that topic distributions and various kinds of lexical variabilities depend on the local context has been known and studied for a long time @xcite . for instance, @xcite says "most of the give-and-take of conversation in our everyday life is stereotyped and very narrowly conditioned by our particular type of culture". in her seminal work, @xcite analyzed the role of place in language, focusing on lexical vari-ations within the same language, a subject further explored by @xcite . some of these works were the basis for later studies that introduced computational models for how language changes with geographic location @xcite . in the field of topic modeling, there has been a new sub-field emerging over the past 10 years focusing on modeling multi-lingual corpora @xcite @xcite . however, only recently had researchers dropped assumptions on the use of parallel and comparable corpora @xcite . while some works do investigate issues related to stdm @xcite , like how named entities receive a different distribution over words in different languages @xcite , none of these works have analyzed how the overall topic distribution of data originating in the source and target language differ. in mt, researchers have often made an explicit assumption on the use of comparable corpora @xcite @xcite , i.e. corpora in the two languages that roughly cover the same set of topics. unfortunately, monolingual corpora are seldom comparable in practice. @xcite analyzes two comparable corpora, one in american english and the other in british english, and demonstrate differences that reflect the cultures of origin. similarly, @xcite observes that parallel datasets built for mt exhibit strong biases in the selection of the original documents, making the text collection not quite comparable. the non-comparable nature of mt datasets is even more striking when considering low resource language pairs, for which differences in local context and cultures are often more pronounced. recent studies @xcite have warned that removing the assumption on comparable corpora strongly deteriorates performance of lexicon induction techniques which are at the foundation of mt. back-translation @xcite has been the workhorse of modern neural mt, enabling very effective use of target side monolingual data. backtranslation is beneficial because it helps regularizing the model and adapting to new domains @xcite . however, the typical setting of current mt benchmarks as popularized by re- n y c e v e r e s t y o s e m it e r o m e k a th m a n d u mono test parallel human translation mono parallel human translation source language target language n y c e v e r e s t y o s e m it e r o m e k a th m a n d u distribution over topics latent space d @xmath0.g., in the source domain politics is more popular than travel) and because for the same topic the word distributions are different (e.g., the word "everest" is more common than "yosemite" in the travel topic of the target domain). on the right hand side, we show how stdm manifests in machine translation datasets. all data originating in the source language belongs to the source domain, this includes a portion of the parallel dataset, the source side monolingual dataset and the test set we eventually would like to translate. empty boxes represent human translated data in the parallel training dataset. ## the stdm problem in this section we formalize the definition of source-target domain mismatch (stdm); this is an intrinsic property of the data which is independent of the particular mt system under consideration. in practice, there might be several factors contributing to stdm. here, we are going to consider only the difference in topic distributions, since this is what we can easily quantify. we assume there exists a latent concept space shared across all languages. the process to gener-ate a sentence follows the standard data generation process used in topic modeling, whereby we first sample a distribution over topics, π i ∼ π where i is an index over topics, and then a distribution over words for each topic, w ij ∼ π i , where j indexes the words in the dictionary. next, we assume there are two distinct domains, the source domain d s and the target domain d t . these two domains differ in both the distribution over topics π, and the distribution over words given a certain topic π i , as depicted in fig. 1 . for the sake of conciseness, we will refer to z s and z t as sentences in the concept space generated from domain d s and d t , respectively. let's imagine now that we have generated two sets of sentences in each domain. what we observe in practice is their realization in each language, src(z s ) and tgt(z t ), where src and tgt map sentences from the concept space to the source and target language, respectively. finally, let's denote with h @xmath1. in the simplest setting, a machine translation dataset is composed of parallel and monolingual datasets. using the notation introduced above, the parallel dataset is denoted by @xmath2, h @xmath3, tgt(z t )} z t ∼d t . the first set originates in the source language and belongs to the source domain, while the second set originates in the target language and belongs to the target domain. we then have a source side monolingual dataset, m @xmath4, and a target side monolingual dataset, m @xmath5, belonging to the source and target domains, respectively. most importantly, the test set which we would like to eventually translate contains sentences in the source language, all belonging to the source domain. the existence of distinct source and target domains and datasets derived from these two domains as described above define the stdm problem. while in previous mt studies, there is a mismatch between the training and the test distribution, stdm is a particular case of multi-domain training, with the test set matching one of the training domains. we would like to a) understand the effects of such mismatch and b) understand how to best leverage the out-of-domain data originating from the target language (target monolingual dataset and portion of the parallel dataset originating in the target language). ## metric: the stdm score given the framework and assumptions introduced in §3, in this section we are going to discuss a practical way to measure stdm. ideally, we would like to measure a distance between two sample distributions, z s ∼ d s and z t ∼ d t . unfortunately, we have no access to such latent space. what we observe are realizations in the source and target language. however, it is also an open research question @xcite how to compare the distribution of {src(z s )} against {tgt(z t )}, since these are two possibly incomparable corpora in different languages. in this work, we therefore leverage the existence of a parallel corpus and compare the distribution of a @xmath6. the underlying assumptions are a) we know the originating language of each training example, b) the effect of the change of the word distribution is negligible compared to the shift in topic distribution, and c) the effect of translationese @xcite @xcite is negligible compared to the actual stdm, and therefore, we can ignore changes to the distribution brought by the mapping h @xmath7. under these assumptions, we define the score as a measure of the topic discrepancy between a s and a t . let @xmath8. we first extract topics using lsa. let @xmath9, the bottom n t rows are representations of a t , and k is the number of words in the dictionary. the svd decomposition of a yields: ū collects topic representations of the original documents; let's denote by ū s the first n s rows corresponding to a s and ū t the remaining n t rows corresponding to a t . let @xmath10, where c @xmath11, c @xmath12. the stdm score is defined as: ## machine translation baselines in this section, we turn our attention to how stdm affects training of mt systems. we consider state-of-the-art neural machine translation (nmt) systems based on the transformer architecture @xcite with subword vocabularies learned via byte-pair encoding (bpe) @xcite . in order to adapt to the different domains, we employ domain tagging @xcite by adding a domain token to the input source sentence, and we cross validate the weights between in-domain and out-domain data 2 . we also use label smoothing @xcite and dropout @xcite to improve generalization, as we focus on low resource language pairs where models tend to severely overfit. finally, we explore ways to leverage both target and source side monolingual data via back-translation and self-training which we review next. we simplify our notation and denote with x @xmath13, y @xmath14, and ŷs and xt the corresponding machine translations. the superscript always specifies the domain. we assume access to a parallel dataset @xmath15, y s )} ∪ {(x t , y t )}, a source side monolingual dataset m @xmath16. the test side consists of sentences from the source domain. we study stdm by training with the following algorithms: back-translation (bt) @xcite is a very effective data augmentation technique that leverages m t . the algorithm proceeds in three steps. first, a reverse mt system is trained from target to source using the provided parallel data: self-training (st) @xcite is another method for data augmentation that instead leverages m s ; see alg. 1 in appendix d. also this algorithm proceeds in three steps. first, a forward mt system is trained from source to target using the provided parallel data: ## machine translation results in this section, we first study the effect of stdm on nmt using the controlled setting introduced in §4.1 which enables us to assess the influence of various factors, such as the extent to which target originating data is out-of-domain, and the effect of monolingual data size. we then report experiments on genuine low resource language pairs, namely nepali-english and english-myanmar. we report sacrebleu @xcite . ## conclusions while the commonly used wmt datasets exhibit mild stdm, we find that less curated datasets, often in more distant and lower resource language pairs ( §4.2), exhibit much stronger stdm. how can these findings inform us on how to better set up a mt system in practice? our first recommendation is to be aware of possible stdm, and (i) check whether origin language information is available. if this is available, then it may be possible to (ii) quantitatively measure stdm as described in §4. next, (iii) be aware that when stdm is severe (stdm score is low), bt performance suffers (fig. 3 ). however, (iv) we may be able to combat this by increasing the amount of target side (outof-domain) monolingual data (fig. 4 ) and (v) by combining bt with st (fig. 3 ). of course, the relative ratio of monolingual data in the source and target side and the actual degradation brought by stdm depend on the particular language pair. the more distant two languages, the more difficult the learning task and the more data is needed to learn it. and finally, the less parallel data there is, the more monolingual data will be needed to compensate. the intricate dependency between all these factors merits future investigation.