Do Context-Aware Translation Models Pay the Right Attention? (2021)

Abstract: Context-aware machine translation models are designed to leverage contextual information, but often fail to do so. As a result, they inaccurately disambiguate pronouns and polysemous words that require context for resolution. In this paper, we ask several questions: What contexts do human translators use to resolve ambiguous words? Are models paying large amounts of attention to the same context? What if we explicitly train them to do so? To answer these questions, we introduce SCAT (Supporting Context for Ambiguous Translations), a new English-French dataset comprising supporting context words for 14K translations that professional translators found useful for pronoun disambiguation. Using SCAT, we perform an in-depth analysis of the context used to disambiguate, examining positional and lexical characteristics of the supporting words. Furthermore, we measure the degree of alignment between the model’s attention scores and the supporting context from SCAT, and apply a guided attention strategy to encourage agreement between the two.

## introduction there is a growing consensus in machine translation research that it is necessary to move beyond sentence-level translation and incorporate document-level context @xcite @xcite . while various methods to incorporate context in neural machine translation (nmt) have been proposed @xcite ; @xcite ; @xcite , inter alia), it is unclear whether models rely on the "right" context that is actually sufficient to disambiguate difficult translations. even when additional context ## what context do human translators pay attention to? we first conduct a user study to collect supporting context that translators use in disambiguation, and analyze characteristics of the supporting words. ## do models pay the right attention? next, we study nmt models and quantify the degree to which the model's attention is aligned with the supporting context from professional translators. ## implications and future work in this work, we collected a corpus of supporting context for translating ambiguous words. we examined how baseline context-aware translation models use context, and demonstrated how context annotations can improve context-aware translation accuracy. while we obtain promising results for context-aware translation by testing one method for attention regularization, our publicly available scat dataset could enable future research on alternative attention regularizers. moreover, our analyses demonstrate that humans rely on different types of context for par and wsd in english-french translation, similar user studies can be conducted to better understand the usage of context in other ambiguous discourse phenomena, such as ellipsis, or other language pairs. we also find that regularizing attention using scat for par especially improves anaphoric pronoun translation, suggesting that supervising attention using supporting context from different tasks may help models resolve other types of ambiguities. one caveat regarding our method for collecting supporting context from humans is the difference between translation, translating text from the input, and disambiguation, choosing between translation candidates. during translation, humans might pay more attention to the source sentences to understand the source material, but during disambiguation, we have shown that human translators rely more often on the target sentences. one reason why the model benefits more from increased attention on source may be because the model is trained and evaluated to perform translation, not disambiguation. a future step would be to explore alternative methods for extracting supporting context, such as eye-tracking during translation (o'brien, 2009). a scat: supporting context for ambiguous translations the dataset contains 19,372 annotations in total on 14,000 unique examples of ambiguous anaphoric pronoun translations. in