When Does Translation Require Context? A Data-driven, Multilingual Exploration (2023)

Abstract: Although proper handling of discourse significantly contributes to the quality of machine translation (MT), these improvements are not adequately measured in common translation quality metrics. Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation, however not in a fully systematic way. In this paper, we develop the Multilingual Discourse-Aware (MuDA) benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena in any given dataset. The choice of phenomena is inspired by a novel methodology to systematically identify translations that require context. This methodology confirms the difficulty of previously studied phenomena while uncovering others which were not previously addressed. We find that commonly studied context-aware MT models make only marginal improvements over context-agnostic models, which suggests these models do not handle these ambiguities effectively. We release code and data for 14 language pairs to encourage the MT community to focus on accurately capturing discourse phenomena. Code available at https://github.com/neulab/contextual-mt

## introduction in order to properly translate discourse phenomena including anaphoric pronouns, lexical cohesion, and discourse markers, a machine translation (mt) model must use information from previous utterances @xcite @xcite . however, while generating proper translations of these phenomena is important for comprehension, they represent a small portion of words in natural language. therefore, common metrics such as bleu @xcite cannot be used to judge the quality of discourse translation. dataset lang. phenomena müller et al. (2018) @xmath0. (2018) @xmath1, coherence lexical consistency voita et al. (2018) voita et al. (2019b) @xmath2, ellipsis lexical consistency jwalapuram et al. (2020) @xmath3, coherence lexical consistency discourse connectives our work 14 pairs ( §5) pronouns, ellipsis formality lexical consistency verb forms recent work on neural machine translation (nmt) models that attempt to incorporate extrasentential context @xcite @xcite , inter alia) often perform targeted evaluation of certain discourse phenomena, mostly focusing on ellipsis, formality @xcite , and pronoun translation @xcite @xcite . however, only a limited set of discourse phenomena for a few language pairs have been studied (see summary in in this paper, we identify sentences that contain discourse phenomena through a data-driven, semiautomatic methodology. we apply this method to create a multilingual benchmark testing discourse phenomena in the domain of mt. first, we develop p-cxmi ( §2) as a metric to identify when context is helpful in mt, or more broadly text generation in general. then, we perform a systematic analysis of words with high p-cxmi to find categories of trans-lations where context is useful ( §3). we identify novel discourse phenomena that to our knowledge have not been addressed previously (e.g. consistency of verb forms), without requiring a-priori language-specific knowledge. finally, we design a series of methods to automatically tag words belonging to the identified classes of ambiguities ( §4) and we evaluate existing translation models for different categories of ambiguous translations ( §5). we examine a parallel corpus spanning 14 language pairs, measuring translation ambiguity and model performance. we find that the contextaware methods, while improving on standard evaluation metrics, only perform significantly better than context-agnostic baselines for certain discourse phenomena in our benchmark. our benchmark provides a more fine-grained evaluation of translation models and reveals weaknesses of context-aware models, such as verb form cohesion. we also find that deepl, a commercial document-level translation system, does better in our benchmark than its sentence-level ablation and google translate. we hope that the released benchmark and code, as well as our findings, will spur targeted evaluation of discourse phenomena in mt to cover more languages and more phenomena in the future. ## which translation phenomena benefit from context? to identify salient translation phenomena that require context, we perform a thematic analysis @xcite , examining words with high p-cxmi across different language pairs and manually identifying patterns and categorizing them into phenomena where context is useful for translation. to do so, we systematically examined (1) the mean p-cxmi per part-of-speech (pos) tag, (2) the words with the highest mean p-cxmi across the corpus, and (3) the individual words with the highest p-cxmi in a particular sentence. ## cross-phenomenon mt evaluation next, we we develop a series of methods to automatically tag tokens belonging to these classes of ambiguous translations and propose the multilingual discourse-aware (muda) benchmark for context-aware mt models. ## exploring context-aware mt our muda tagger can be applied to documents in the supported languages to create benchmarking datasets for discourse phenomena during translation. we use our benchmark of the ted talk dataset enhanced with muda tags to perform an exploration of context usage across languages with 4 models, including commercial systems. ## related work several works have worked on measuring the performance of mt models on contextual discourse phenomena. the first example of this was done by @xcite , which evaluated automatically the precision and recall of pronoun translation in statistical mt systems. @xcite proposed evaluating models on pronoun translation based on a pairwise comparison between translations that were generated with and without context, and later @xcite extended this work to include more languages and phenomena in their automatic evaluation/test set creation. these works rely on prior domain knowledge and intuition to identify context-aware phenomena, whereas we take a systematic, data-driven approach. most works have focused on evaluating performance in discourse phenomena through the use of contrastive datasets. @xcite automatically create a dataset for anaphoric pronoun resolution to evaluate mt models in @xmath4. @xcite manually creates a dataset for both pronoun resolution and lexical choice in @xmath5. @xcite creates a dataset for anaphora resolution, deixis, ellipsis and lexical cohesion in @xmath6. however, @xcite suggest that translating and disambiguating between two contrastive choices are inherently different, motivating our approach in measuring direct translation performance. ## conclusions and future work we investigate types of ambiguous translations where mt models benefit from context using our proposed p-cxmi metric. we perform a datadriven thematic analysis across 14 languages to identify context-sensitive discourse phenomena, some of which (such as verb forms) have not been previously addressed in work on mt. in comparison to previous work, our approach is systematic, extensible, and does not require prior knowledge of the language. additionally, the p-cxmi metric can be used to identify other context-dependent words in generation. we construct the muda benchmark that tags words in parallel corpora and evaluates models on 5 context-dependent phenomena. our evaluation reveals that context-aware and commercial translation systems achieve small improvements over context-agnostic models on our benchmark, and we encourage further development of models that improve on context-aware translation.