Automatic Evaluation and Analysis of Idioms in Neural Machine Translation (2023)

Abstract: A major open problem in neural machine translation (NMT) is the translation of idiomatic expressions, such as “under the weather”. The meaning of these expressions is not composed by the meaning of their constituent words, and NMT models tend to translate them literally (i.e., word-by-word), which leads to confusing and nonsensical translations. Research on idioms in NMT is limited and obstructed by the absence of automatic methods for quantifying these errors. In this work, first, we propose a novel metric for automatically measuring the frequency of literal translation errors without human involvement. Equipped with this metric, we present controlled translation experiments with models trained in different conditions (with/without the test-set idioms) and across a wide range of (global and targeted) metrics and test sets. We explore the role of monolingual pretraining and find that it yields substantial targeted improvements, even without observing any translation examples of the test-set idioms. In our analysis, we probe the role of idiom context. We find that the randomly initialized models are more local or “myopic” as they are relatively unaffected by variations of the idiom context, unlike the pretrained ones.

## introduction neural machine translation @xcite @xcite struggles with the translation of rare multi-word expressions (mwe) @xcite . non-compositional phrases, such as idioms (e.g., "piece of cake"), are one of the most challenging types of mwes, because their meaning is figurative and cannot be derived from the meaning of their constituents @xcite . nmt models tend to translate these expressions literally (i.e., word-by-word), which leads to erroneous translations. in this paper, our focus is on the translation of idiomatic expressions, in contrast to most prior work, which is subsumed under mwes in general @xcite . the absence of targeted and automatic evaluation is a major obstacle to advances in idiom translation. global metrics, such as bleu @xcite consider the full translation, and thus, the effects of idiom translation are overshadowed. previous efforts on targeted evaluation isolate the idiom translation using word alignments @xcite or word edit distance @xcite . these approaches measure the accuracy of idiom translation but do not account for literal translation errors. @xcite proposed a method for estimating the frequency of such errors, but it requires the creation of language-specific handcrafted lists (i.e., blocklists) with words that correspond to literal translation errors. in this work foot_0 , we present a study of idioms in nmt, with the goal of facilitating future research in this direction. first, we propose a novel metric for the automatic evaluation of literal translation errors (litter), that does not require any hand-crafted blocklists. we incorporate litter, which complements alignment-based metrics @xcite into a unified targeted evaluation framework. next, we present translation experiments in a controlled setting, by using different training splits to test models under different conditions (e.g., zeroshot). to improve idiom translation we leverage monolingual data, which are more abundant than parallel and contain idioms in higher frequencies and more diverse contexts. we exploit monolingual data via pretraining (mbart; @xcite , which is a generic and task-agnostic approach, unlike prior work that considers ad-hoc solutions @xcite . we find that monolingual pretraining yields strong targeted gains, even when models have not seen any translation examples of the test idioms. we also present an extensive analysis of how different models translate idioms. specifically, we use a series of probing methods that encode idioms within different contexts @xcite , and measure how this affects the translation outputs and the decoder distributions. we find that the randomly initialized models are more "myopic" compared to the pretrained ones, as they are relatively unaffected when we vary the idiom context. our contributions are: ## analysis to further understand how models translate idioms, we present an extensive analysis with a series of probes, focused on the role of idiom context. specifically, using the annotations in our idiomtest data, we encode the idiom words within different contexts (figure 4 ) and measure how it affects the decoder distributions and the translation output. we consider (1) full context, in which we unnecessary red tape is slowing down the vaccine roll-out plan in the country red tape red + tape encoder a b c d e f ℎ 1 ℎ 2 ℎ 3 ℎ 4 ℎ 5 ℎ 6 encoder c d 𝑐 1 𝑐 2 encoder c 𝑧 1 encoder d 𝑧 2 full context phrase context word context figure 4: illustration of how we obtain idiom representations by varying the available context. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 chrf++ (higher is better) mbart random translation quality zero: word zero: phrase zero: full joint: word joint: phrase joint: full joint+20x: word joint+20x: phrase joint+20x: full encode the idiom phrase within the whole input sentence, (2) phrase-level context, in which we encode the idiom phrase in isolation, (3) word-level context, in which we encode each idiom word independently. our probes follow a similar approach to prior work that evaluate the idiomaticity of (pretrained) transformer-based models @xcite by measuring how idiom representations are affected by their context (yu and ettinger, 2020), but we extend these methods to analyze how (pretrained) nmt models translate idioms. for brevity, we discuss here our most important findings, focusing on randomvs-mbart initialization. however, we include the rest of our results in appendix d for completeness. ## discussion global metrics liter does not aim to replace global evaluation metrics like bleu, but to complement them. global metrics estimate the general translation quality of model outputs, which is undoubtedly important. however, they consider the full sentence, and as result the effects of idiom (mis)translation are overshadowed (see §3.3.2). liter aims to fill this gap by providing additional insights to practitioners with targeted evaluation. litter liter should not be used in isolation, but combined with other (global/targeted) metrics. the reason is that lower liter can be achieved by more accurate idiom translations or by hallucinations. we aim to enable practitioners to make informed decisions without running human evaluations in the model development phase. the goal is to produce models that improve on litter without sacrificing general translation quality (e.g., bleu). our experiments reveal this contrast, where unsupervised pretraining achieves this goal, whereas (some of) the noisy variants fail. we expect liter to be more relevant when developing nmt models for creative content (e.g., subtitles, social media text) that usually contains figurative language. alignment metrics word alignment-based methods aim to capture idiom translation accuracy that complements litter. however, they are sensitive to the literal meaning of words to produce the alignments. thus, while alignment-based methods could be reliable in certain types of evaluation, such as gender translation @xcite , we believe that with the current techniques they should be used with caution for idiom translation evaluation. although we did not systematically study this issue, we discovered by manual inspection that it was not uncommon to produce noisy or empty alignments. we chose statistical over embedding-based methods as they yielded less empty alignments ( §b.2). ## related work idioms in nmt there is limited research on idioms in nmt. zaninello and birch (2020), explore augmenting the training data with mwe translations from dictionaries, backtranslating @xcite target-side sentences with mwes, or wrapping the (source) mwes with special tokens. @xcite prepend a special token in source sentences that contain an idiom. @xcite do unsupervised translation of mwes by composing cross-lingual word embeddings, but this is fundamentally incapable of translating idioms which are non-compositional. instead of using ad-hoc solutions that change the model or the data pipeline, we use monolingual pretraining, which is a more generic and less invasive approach. ## conclusions we present a comprehensive study of idiomatic expressions in nmt, aiming to facilitate future research on the topic. we propose litter ( §2.1), a novel metric that enables the automatic evaluation of literal translation errors. litter is used for targeted evaluation and aims to complement and not replace global metrics, such as blue or chrf, which consider the full sentence and can only measure the overall translation quality. we evaluate models in controlled conditions, with and without the test set idioms (i.e., zero-shot). we explore pretraining on monolingual data for improving idiom translation, as parallel idiom data is difficult to come by. interestingly, we find that pretraining achieves strong targeted improvements, even in the zero-shot setting ( §3.3.1). we also present a systematic analysis ( §4) that investigates the role of context in idiom translation. we find evidence that pretraining yields more contextual models, which helps to explain why it contributes to better idiom translations. we also quantitatively confirm that idioms are more difficult to translate than regular words and strongly depend on the source context.