Measuring and Increasing Context Usage in Context-Aware Machine Translation (2021)

Abstract: Recent work in neural machine translation has demonstrated both the necessity and feasibility of using inter-sentential context, context from sentences other than those currently being translated. However, while many current methods present model architectures that theoretically can use this extra context, it is often not clear how much they do actually utilize it at translation time. In this paper, we introduce a new metric, conditional cross-mutual information, to quantify usage of context by these models. Using this metric, we measure how much document-level machine translation systems use particular varieties of context. We find that target context is referenced more than source context, and that including more context has a diminishing affect on results. We then introduce a new, simple training method, context-aware word dropout, to increase the usage of context by context-aware models. Experiments show that our method not only increases context usage, but also improves the translation quality according to metrics such as BLEU and COMET, as well as performance on anaphoric pronoun resolution and lexical cohesion contrastive datasets.

## introduction while neural machine translation (nmt) is reported to have achieved human parity in some domains and language pairs @xcite , these claims seem overly optimistic and no longer hold with document-level evaluation @xcite . recent work on contextaware nmt attempts to alleviate this discrepancy by incorporating the surrounding context sentences (in either or both the source and target sides) in the translation system. this can be done by, for example, feeding context sentences to standard nmt 1 https://github.com/neulab/contextual-mt figure 1 : illustration of how we can measure context usage by a model q m t as the amount of information gained when a model is given the context c and source x vs when the model is only given the x. we hypothesize that one major reason for these lacklustre results is due to the fact that models with the architectural capacity to model cross-sentential context do not necessarily learn to do so when trained with existing training paradigms. however, even quantifying model usage of context is an ongoing challenge; while contrastive evaluation has been proposed to measure performance on inter-sentential discourse phenomena @xcite , this approach is confined to a narrow set of phenomena, such as pronoun translation and lexical cohesion. a toolbox to measure the impact of context in broader settings is still missing. ## context-aware neural machine translation we are interested in learning a system that translates documents consisting of multiple sentences between two languages. 2 more formally, given a corpus of parallel documents in two languages, @xmath0, ..., d n }, where each document is a sequence of source and target sentences, @xmath1, y (1) ), ..., (x (k) , y (k) )}, we are interested in learning the mapping between the two languages. we consider the typical (auto-regressive) neural machine translation system q θ parameterized by θ. the probability of translating x (i) into y (i) given the context of the sentence c as mentioned, there are many architectural approaches to leveraging context (see §5 for a more complete review), and the methods that we present in this paper are compatible with most architectures because they do not specify how the model q θ uses the context. in experiments, we focus mostly on the simpler approach of concatenating the context to the current sentences @xcite . recent work by @xcite has shown that, given enough data (either through pretraining or larger contextual datasets), this simple approach tends to be competitive with or even outperform its more complex counterparts 3 measuring context usage ## related work context-aware machine translation there have been many works in the literature that try to incorporate context into nmt systems. tiedemann and scherrer (2017) first proposed the simple approach of concatenating the previous sentences in both the source and target side to the input to the system; @xcite , @xcite used an additional contextspecific encoder to extract contextual features from the previous sentences; @xcite and @xcite used cache-based memories to encode context; wang et al. (2017) used a hierarchical rnn to encode the global context from all previous sentences; miculicich et al. (2018) and maruf et al. (2019a) used hierarchical attention networks to encode context; chen et al. (2020) added document-level discourse structure information to the input; sun et al. (2020) trained a simple concatenation-based model with varying context size during training to have a model that is able to translate with any context size, similar to what is done in this work. similarly to what we do with coword dropout, jean and cho ( 2019 ) attempted to maximise sensitivity to context by introducing a margin-based regularization term to explicitly encourage context usage. for a more detailed overview, @xcite extensively describe the different approaches and how they leverage context. while these models lead to improvements with small training sets, @xcite showed that the improvements are negligible when compared with the concatenation baseline when using larger datasets. however, importantly, both our metric cxmi for measuring context usage and the proposed regularization method of coword dropout, can theoretically be applied to any of the above-mentioned methods. evaluation in terms of evaluation, most previous work focuses on targeting a system's performance on contrastive datasets for specific inter-sentential discourse phenomena. müller et al. (2018) 2020 ) showed through adversarial attacks that models that do well on other contrastive datasets rely on surface heuristics and create a contrastive dataset to address this. in contrast, our cxmi metric is phenomenon-agnostic and can be measured with respect to all phenomena that require context in translation. ## implications and future work we introduce a new, architecture-agnostic, metric to measure how context-aware machine translation models are using context and propose a simple regularization technique to increase context usage by these models. our results are theoretically applicable to almost all recently proposed context-aware models and future work should go about measuring exactly how much these models leverage context and if coword dropout also improves context usage and performance in these. we also hope this work motivates exploring (c)xmi for other uses cases where measuring the relevance/usage of inputs to a particular model other than context-aware machine translation. it could, for example, be used in conditional language modelling to analyse how the inputs we are conditioning on are being used by the model.