That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context? (2023)

Abstract: The translation of ambiguous text presents a challenge for translation systems, as it requires using the surrounding context to disambiguate the intended meaning as much as possible. While prior work has studied ambiguities that result from different grammatical features of the source and target language, we study semantic ambiguities that exist in the source (English in this work) itself. In particular, we focus on idioms that are open to both literal and figurative interpretations (e.g., goose egg), and collect TIDE, a dataset of 512 pairs of English sentences containing idioms with disambiguating context such that one is literal (it laid a goose egg) and another is figurative (they scored a goose egg, as in a score of zero). In experiments, we compare MT-specific models and language models for (i) their preference when given an ambiguous subsentence, (ii) their sensitivity to disambiguating context, and (iii) the performance disparity between figurative and literal source sentences. We find that current MT models consistently translate English idioms literally, even when the context suggests a figurative interpretation. On the other hand, LMs are far more context-aware, although there remain disparities across target languages. Our findings underline the potential of LMs as a strong backbone for context-aware translation.

## introduction natural language is inherently ambiguous due to the competing pressures of efficiency and clarity in communication @xcite . as communicators, we disambiguate meanings on the basis of a wide range of contextual factors, or ask clarifying questions when such context is not available. though sometimes overlooked, the role of ambiguity in nlp has gained growing interest in recent work @xcite @xcite . 1 data and code can be found at https://github.com/ jaechan-repo/mt-ambiguity . figure 1 : tide consists of pairs of contrastive sentences that contain the same idiomatic expression in different contexts, such that one uses the figurative meaning of the idiom (left), and another uses its literal meaning (right). on this set of inputs, chatgpt is sensitive to the disambiguating context when translating the idiom, while nllb is not. in machine translation (mt), it has long been recognized that ambiguities arise when the source language does not encode grammatical attributes that the target language requires @xcite @xcite . for instance, the english sentence "i am a doctor" would require disambiguating the doctor's gender for translation to german, which has no gender-neutral word for "doctor." prior work created contrastive test sets for such phenomena, to evaluate whether mt models correctly translate an ambiguous word (here, "doctor") when disambiguating context is available (e.g., "she is a doctor") @xcite @xcite . in contrast with grammatical ambiguity with respect to a target language, it is relatively less understood how mt systems handle semantic ambiguity present in the source text itself. for instance, "i have bigger fish to fry" is ambiguous between figurative ("... at work") and literal ("... for the dinner") interpretations in english, outside of the context of translation. therefore, we extend the line of work on context-aware translation to semantically ambiguous phrases in english. to this end, we create tide, translations of idioms in disambiguating context in english, a dataset of 512 example triples. each triple consists of an ambiguous subsentence and a pair of contrastive sentences that contain the subsentence but add disambiguating context: one to produce a figurative interpretation of the idiom, and another to produce a literal interpretation of it (see figure 1 for an example). our creation process for the triples combines automatic text generation with human annotation: we use gpt-4 to draft the triples, which are then scrutinized by human annotators. following this, we engage native speakers from four languages to craft reference translations for a subset of the dataset. in our experiments, we evaluate both traditional neural mt models and language models (lms). mt-specific models are trained on large corpora of parallel sentences, and have formed the foundation of translation research; lms are trained without any explicit supervision for translation, yet recently demonstrate impressive translation ability @xcite . using tide, we compare how these two types of systems handle ambiguity, and evaluate their sensitivity to disambiguating context. we find that on ambiguous input, lms demonstrate roughly balanced preference between literal and figurative interpretations, whereas mt-specific models consistently prefer literal ones ( §4.1). given disambiguating context, lms are substantially more context-aware, though this sensitivity declines for more low-resource target languages; in contrast, mt-specific models tend to translate idioms literally irrespective of context ( §4.2). finally, mtspecific models are better at translation of literal text than figurative text, whereas this disparity in lms is much narrower ( §4.3). we summarize our contributions as follows: (1) we formalize the challenge of ambiguous idiomatic language in mt; (2) we create a new translation benchmark, tide, that includes sentences with idioms along with disambiguating contexts (literal and figurative); (3) we analyze mt systems' behavior with and without disambiguating contexts, pointing to interesting trends and differences between lms and mt-specific models. ## creating tide idioms, though commonplace in daily communication, pose a challenge for mt systems due to its inherent ambiguity between literal and non-literal meanings. generating the most appropriate translation among potential disambiguations of the idiom involves an understanding that extends beyond the idiom itself, as an mt system must use broader context clues to discern the most fitting translation. we present tide, a dataset of 512 example triples. each triple consists of an ambiguous subsentence, a figurative sentence, and a literal sentence in english, all including the same idiom. the ambiguous subsentence permits both figurative and literal interpretations of the idiom, while the figurative and literal sentences introduce additional context that resolves the ambiguity to figurative and literal readings, respectively. we design subsentences (e.g., "had a card up his sleeve") to be more than an idiom itself (here, "card up sleeve"), as idioms alone can often be unnatural as standalone input to an mt system. we construct tide through a human-ai collaborative approach following a line of recent work @xcite . we first manually select candidate idioms from two large idiom corpora ( §2.1). next, we leverage the generative power of gpt-4 to efficiently produce diverse and high-quality text, by prompting it to write a complete triple for each idiom ( §2.2). to ensure quality and correctness, we then involve human annotators to filter out invalid triples ( §2.3). finally, we collect gold translations for a subset of the dataset among native speakers ( §2.4). ## experimental setup in this section we outline the models ( §3.1) and languages ( §3.2) we evaluate, our automatic metrics ( §3.3), and our setup for collecting human evaluations of generated translations ( §3.4). ## experimental results in our experiments, we explore mt-specific and lm systems' translation behavior on ambiguous subsentences ( §4.1), their sensitivity to disambiguating context ( §4.2), and their overall competence at translating literal versus figurative input ( §4.3). ## conclusion in this work we focus on semantic ambiguity in machine translation, specifically when using idiomatic language. we introduce a new benchmark (tide) of sentences that include idioms, along with disambiguating contexts (both literal and figurative). we then use tide to investigate the behavior of different translation systems on ambiguous input and their sensitivity to disambiguating context, uncovering new strengths of pretrained lms compared to mt-specific models. our findings point to pretrained lms as a promising backbone for translation systems, and we foresee a future that combines the strong language understanding of lms with dedicated supervision for translation. andrea zaninello and alexandra birch. 2020. multiword expression aware neural machine translation. in proceedings of the twelfth language resources and evaluation conference, pages 3816-3825, marseille, france. european language resources association. george kingsley zipf. 1949. human behavior and the principle of least effort.