MMTE : Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language (2024)

Abstract: Machine Translation (MT) has developed rapidly since the release of Large Language Models and current MT evaluation is performed through comparison with reference human translations or by predicting quality scores from human-labeled data. However, these mainstream evaluation methods mainly focus on fluency and factual reliability, whilst paying little attention to figurative quality. In this paper, we investigate the figurative quality of MT and propose a set of human evaluation metrics focused on the translation of figurative language. We additionally present a multilingual parallel metaphor corpus generated by post-editing. Our evaluation protocol is designed to estimate four aspects of MT: Metaphorical Equivalence, Emotion, Authenticity, and Quality. In doing so, we observe that translations of figurative expressions display different traits from literal ones.

## introduction metaphorical expressions are widely used in daily life for communication and vivid description, drawing attention from psycholinguistics and computational linguistics due to their key role in the cognitive and communicative functions of language @xcite @xcite . linguistically, a metaphor is defined as a figurative expression that uses one or more words to represent another concept within a given context, rather than taking the literal meaning of the expression @xcite . for instance, in the sentence "the scream pierced the night.", the contextual meaning of pierced is to "sound sharply or shrilly", which differs from its literal meaning of "cut or make a way through". 1 a significant portion (e.g. up to 20%) of our everyday language is delivered in metaphorical terms @xcite . according to lakoff and johnson's study, metaphor is a type of conceptual mapping. the cognitive model, which involves reasoning about one thing in terms of another, has been shown to affect our decision-making and perception @xcite @xcite . research also suggests that this concept-to-concept mapping is often languageagnostic, with similar mappings being feasible across different languages @xcite . for instance, in the aforementioned example, the english word "pierce" corresponds to the chinese word "穿透", which literally means "pass through an object or medium" and is also used as a metaphor to indicate a sudden and sharp sound. however, direct translations cannot always be found in the target language due to linguistic and cultural differences. to illustrate this issue, we provide an example of metaphorical expressions of being drunk in figure 1 , where in chinese it is common to compare drunkenness to being collapsed on the ground like quagmire, whilst in english it is common to compare it to seeing pink elephants. these word-sense misalignments caused by dif-ferent linguistic norms are ubiquitous in practical translation applications. metaphor, and especially metaphor translation, has received increasing attention in linguistics @xcite @xcite , and its significance has also been highlighted in various nlp tasks similar to translation, such as poem writing @xcite , story generation @xcite , and dialogue @xcite . unfortunately, the challenge of machine translating metaphorical language remains largely unaddressed due to a scarcity of resources such as parallel data @xcite . to remedy this, we propose mmte -the first systematic study of metaphorical machine translation evaluation to explore the difficulties inherent in translating metaphorical expressions. our contributions include: metaphor-related tasks. metaphors play a crucial role in daily communication and understanding human emotion @xcite and cognition @xcite . enhancing the quality of our understanding of metaphors has been shown to be crucial for various natural language understanding (nlu) tasks, including natural language inference (nli) @xcite , sentiment analysis @xcite ), humour explanation @xcite , and offensive language detection @xcite . moreover, adequately conveying metaphor is a staple concern in various natural language generation (nlg) tasks, including poem writing @xcite , paraphrasing @xcite , dialogue generation @xcite , and story generation @xcite . appropriate use of metaphor has been shown to dramatically improve user satisfaction with such systems @xcite , and exploring the mechanisms behind metaphor generation helps test and verify cognitive theories of how metaphors are created and used @xcite . standalone metaphor generation has also been identified as a significant branch of creative nlg in itself @xcite @xcite @xcite . existing multilingual metaphor research @xcite @xcite has primarily focussed on multilingual metaphor detection and identification guided by metaphorical mappings @xcite , the polarity and valence of multilingual metaphors @xcite , and general metaphor frames @xcite . researchers have also explored metaphor detection and generation mechanisms in languages besides english, including chinese @xcite , malay @xcite , arabic @xcite , and german @xcite . metaphor translation is also a prevalent topic in linguistics @xcite @xcite , whilst relatively unexplored in the machine translation domain. whilst some research has explored metaphor's impact on machine translation @xcite , this has focussed on applying external knowledge bases to enhance cross-lingual metaphor detection. this lack of guidance from professional translators results in translations that lack linguistic nuance. ## metaphorical translation quality annotation framework as discussed in §1, metaphorical expressions are not evaluated sufficiently with current mt evaluation metrics. to address this issue, we propose a set of novel mt evaluation metrics based on manual annotation and post-editing. the proposed met-rics aim to provide a more accurate and insightful assessment of mt performance in handling metaphors. our framework allows for the evaluation of mt outputs in terms of their metaphorical expressions, enabling a more comprehensive analysis of their effectiveness in capturing the nuanced meaning conveyed by such expressions, as demonstrated in figure 2 . ## results we first conduct a comprehensive comparative analysis of the performance of mt on metaphorical and literal expressions based on both manual and automatic evaluation scores in §4. specifically, by analysing the distribution of labels from the finegrained human evaluation protocol, we verify that metaphor translation is more challenging than literal translation in § 4.1. moreover, we examine the correlations between the suggested fine-grained human evaluation protocol in §4.2, the correlations between emotional and metaphorical expressions in §4.4, and the crucial role of metaphor equivalence in metaphor translation quality estimation (qe) in §4.5. additionally, we analyse the translation quality between typologically different languages in §4.6. we also provide a case study indicating that translating between more typologically distant language pairs is harder, by comparing en-zh and en-it pairs in appendix c. ## conclusion mmte is the first work to systematically investigate how translations are affected by metaphor in a fine-grained and multi-lingual setting. mmte also introduces equivalence as a new dimension of metaphor translation evaluation and verifies its relationship with emotional salience and translation quality. moreover, we conducted thorough experiments on the proposed evaluation dimensions and verified the increased difficulty of translating metaphorical expressions compared to literal expressions. we further release mmte, a high-quality metaphor translation corpus which can be adopted for automatic metric design for metaphor translation. future work intends to combine mmte with additional well-designed automatic metrics aligning with specific human evaluation dimensions proposed in the paper.