Addressing Zero-Resource Domains Using Document-Level Context in Neural Machine Translation (2021)

Abstract: Achieving satisfying performance in machine translation on domains for which there is no training data is challenging. Traditional supervised domain adaptation is not suitable for addressing such zero-resource domains because it relies on in-domain parallel data. We show that when in-domain parallel data is not available, access to document-level context enables better capturing of domain generalities compared to only having access to a single sentence. Having access to more information provides a more reliable domain estimation. We present two document-level Transformer models which are capable of using large context sizes and we compare these models against strong Transformer baselines. We obtain improvements for the two zero-resource domains we study. We additionally provide an analysis where we vary the amount of context and look at the case where in-domain data is available.

## introduction training robust neural machine translation models for a wide variety of domains is an active field of work. nmt requires large bilingual resources which are not available for many domains and languages. when there is no data available for a given domain, e.g., in the case of web-based mt tools, this is a significant challenge. despite the fact that these tools are usually trained on large scale datasets, they are often used to translate documents from a domain which was not seen during training. we call this scenario zero-resource domain adaptation and present an approach using document-level context to address it. when an nmt model receives a test sentence from a zero-resource domain, it can be matched to similar domains in the training data. this is to some extent done implicitly by standard nmt. alternatively, this matching can be facilitated by a domain adaptation technique such as using special domain tokens and features @xcite . however, it is not always easy to determine the domain of a sentence without larger context. access to document-level context makes it more probable that domain signals can be observed, i.e., words representative of a domain are more likely to be encountered. we hypothesize that this facilitates better matching of unseen domains to domains seen during training and provide experimental evidence supporting this hypothesis. recent work has shown that contextual information improves mt @xcite @xcite , often by improving anaphoric pronoun translation quality, which can be addressed well with limited context. however, in order to address discourse phenomena such as coherence and cohesion, access to larger context is preferable. @xcite were the first to show large improvements on lexical cohesion in a controlled setting using challenge sets. however, previous work did not make clear whether previous models can help with disambiguation of polysemous words where the sense is domain-dependent. in this work, we study the usefulness of document-level context for zero-resource domain adaptation (which we think has not been studied in this way before). we propose two novel transformer models which can efficiently handle large context and test their ability to model multiple domains at once. we show that document-level models trained on multi-domain datasets provide improvements on zero-resource domains. we evaluate on @xmath0. in addition to measuring translation quality, we conduct a manual evaluation targeted at word disambiguation. we also present additional experiments on classical domain adaptation where access to in-domain ted and pattr data is allowed. our first proposed model, which we call the domain embedding model (domemb) applies average or max pooling over all context embeddings and adds this representation to each source tokenlevel embedding in the transformer. the second model is conceptually similar to previous work on context-aware nmt @xcite @xcite and introduces additional multi-head attention components in the encoder and decoder in order to handle the context. however, in order to facilitate larger context sizes, it creates a compressed context representation by applying average or max pooling with a fixed window and stride size. we compare our proposed models against previous context-aware nmt architectures and techniques for handling multi-domain setups, and show they improve upon strong baselines. the proposed models encode context in a coarse-grained way. they only have a limited ability to model discourse phenomena such as coreference resolution, so the gains we see in a multi-domain setup show that they encode domain information. evaluating on multiple and zero-resource domains allows us to show that context can be used to capture domain information. the contributions of our work can be summarized as follows: we (i) propose two nmt models which are able to handle large context sizes, (ii) show that document-level context in a multidomain experimental setup is beneficial for handling zero-resource domains, (iii) show the effect of different context sizes and (iv) study traditional domain adaptation with access to in-domain data. continued training is an established technique for domain adaptation if access to in-domain resources is possible. the method entails initially training on out-of-domain data, and then continuing training on in-domain data @xcite . @xcite and @xcite improve upon this paradigm by integrating a domain classifier or a domain similarity metric into nmt and modifying the training cost based on weights indicating in-domain or out-ofdomain data. @xcite and @xcite use continued training in a multi-domain setup and propose various ways of fine-tuning to in-domain data. standard continued training (luong and manning, 2015) leads to catastrophic forgetting, evident by the degrading performance on the out-of-domain dataset. @xcite address this issue by ensembling the original and the fine-tuned model. we show that our model obtains significant improvements compared to a baseline with the ensembling paradigm. in contrast to these previous works, we do not know the domains during training. our proposed approaches model the domain implicitly by looking at document-level context. moreover, we evaluate performance on domains not seen during training. @xcite adapt to unseen domains using bandit learning techniques. the method relies on explicit user feedback which is not always easily available. @xcite propose a retrieval-based method that, at inference time, adapts to domains not seen during training. however, they assume access to in-domain parallel data at inference time, and they retrieve parallel phrases from this in-domain data. in our context source c 1 c 2 c 3 c 4 c 5 c 6 c m encoder encoder share (l -1) multi-head attn multi-head attn self attn outputs t 1 t 2 t 3 t 4 t n figure 2: context-aware transformer with pooling. previous work on context-aware nmt has mostly worked with limited context. @xcite address the problem by reusing previously computed encoder representations, but report no bleu improvements by using context larger than 3 sentences. @xcite find 2 sentences of context to work the best. maruf and haffari (2018) use a fixed pretrained rnn encoder for context sentences and only train the documentlevel rnn. junczys-dowmunt (2019) concatenates sentences into very large inputs and outputs as in tiedemann and scherrer (2017). @xcite propose a scalable context-aware model by using sparsemax which can ignore certain words and hierarchical attention which first computes sentence-level attention scores and subsequently word-level scores. however, for domain adaptation, the full encoder representation is too granular and not the most efficient way to obtain domain signals, for which we present evidence in our experiments. @xcite ; @xcite propose a similar approach to our domain embedding model, but they do not investigate it from a domain adaptation perspective. to our knowledge, our work is the first at the intersection of domain adaptation and context-aware nmt and shows that document-level context can be used to address zero-resource domains. ## model the models we propose in this work are extensions of the transformer @xcite . the first approach introduces separate domain embeddings applied to each token-level embedding. the second is conceptually based on previous context-aware models @xcite @xcite . both models are capable of handling documentlevel context. we modify the training data so that all sentences have access to the previous sentences within the corresponding source document. access to the document-level context is available at test time as well. sentences are separated with a @xmath1. we train and evaluate our models with a 10 sentence context. ## conclusion we presented document-level context-aware nmt models and showed their effectiveness in addressing zero-resource domains. we compared against strong baselines and showed that document-level context can be leveraged to obtain domain signals. the proposed models benefit from large context and also obtain strong performance in multidomain scenarios. our experimental results show the proposed models obtain improvements of up to 1.0 bleu in this difficult zero-resource domain setup. furthermore, they show that document-level context should be further explored in future work on domain adaptation and suggest that larger context would be beneficial for other discourse phenomena such as coherence.