Benchmarking Machine Translation with Cultural Awareness (2024)

Abstract: Translating culture-related content is vital for effective cross-cultural communication. However, many culture-specific items (CSIs) often lack literal translation across languages, making it challenging to collect high-quality, diverse parallel corpora with CSI annotations. This difficulty hinders the analysis of cultural awareness of machine translation (MT) systems, including traditional neural MT and the emerging MT paradigm using large language models (LLM). To address this gap, we introduce a novel parallel corpus, enriched with CSI annotations in 6 language pairs for investigating Cultural-Aware Machine Translation—CAMT. Furthermore, we design two evaluation metrics to assess CSI translations, focusing on their pragmatic translation quality. Our findings show the superior ability of LLMs over neural MTs in leveraging external cultural knowledge for translating CSIs, especially those lacking translations in the target culture.

## introduction machine translation (mt) systems have achieved remarkable success in recent years, thanks in part to the pre-trained backbones of multilingual language models @xcite and the availability of multilingual corpora @xcite . despite these advances, terminology translation remains challenging in both general contexts @xcite and specific domains like medicine and law @xcite . many existing mt studies on terminology translation have focused on breaking language barriers rather than cultural barriers, often assuming that literal (i.e., word-for-word) translation pairs already exist for the common knowledge shared by speakers of both the source and target languages (anastasopoulos 1 the corpus and code are released at https://github.com/bigbinnie/benchmarking-llm-basedmachine-translation-on-cultural-awareness ## camt dataset to minimize the need for human efforts while still obtaining diverse, high-quality csi-centered translation pairs across multiple languages and cultures, we rely on wikipedia to collect the data. the overall workflow of our data collection includes (1) building a wiki-centered cultural tax-source (en) a pasticciotto is a type of filled traditional pastry in italy target (zh) 意式乳酪塔是一种有馅料的意大利点心 parallel corpus step i entity linking source a is a type of filled traditional in category classification pasticciotto: culture.food and drink pastry: culture.food and drink italy: geography.regions.europe culture related data augmentation pasticciotto: country of origin: italy translation (zh):意式乳酪塔(italian cheese tart) explanation: filled with ricotta... pastry: country of origin: none culture specific parallel sentences culture knowledge cultural entities sourcen (en) a pasticciotto is a type of filled ... target (zh) 意式乳 酪塔是一种有馅料的... pasticciotto country of origin italy category culture.food and drink translation (zh) 意式乳酪塔(italian cheese tart), ... explanation filled with ricotta... ## cultural awareness evaluation to better capture the cultural nuances in csi translations, we devise two evaluation metrics: (1) csi-match, which evaluates the accuracy of csis with labels, and (2) pta, which assesses the pragmatic translation quality of csis without labels. csi-match: existing evaluation metrics for terminology are efficient for evaluating the accuracy of translations and the fluency of outputs @xcite . however, previous metrics such as exact match (em) assume that terminology translations must be exact matches, while reasonable adaptations of csis are also acceptable @xcite . to address this, we introduce the csi-match metric as a modification to the em evaluation metric. csi-match measures the accuracy of term translation using a more nuanced, fuzzy matching approach. it calculates the maximal partial similarity ratio (psr) between the reference csi translations t 1 , t 2 , . . . , t n and the system out- psr measures the maximum similarity between string t and any substring in s. pragmatic translation assessment (pta): existing evaluation metrics like bleu @xcite and comet @xcite ) mainly focus on surface-level or semantic-level accuracy between the source and target texts. however, in the context of csi translation where translation quality is tightly associated with target culture @xcite , pragmatic translation quality becomes crucial. for example, a free translation based on the description of the csi might have better pragmatic translation quality than a direct literal translation, as it would be easier for people from the target culture to understand. therefore, we design a new assessment metric called pta, measuring the win rate at which csi translations by the mt system are judged to exhibit better pragmatic translation quality compared to human reference translations. specifically, in the human evaluation, we specify what the csis are in the source language and ask native speakers of the target language to compare the csi translations within the sentential context between an mt system and a reference, and then select the translation that is easier to understand. to improve the applicability of pta when native speakers are not available, we use gpt-4o to replace human judgments for pta, which has proven effective for the automatic evaluation of generative models in recent studies @xcite . the evaluation prompt, shown in appendix §e, is used for both human annotators and gpt-4. a higher pta score means the system translates the csi in a more comprehensive manner than the reference sentence does, which might use other accurate but less understandable translations of csis. ## experimental settings to fully evaluate the efficacy of mt system translating csis, we compare nmt systems with llmbased mts ( §5.1). secondly, to investigate the performances of traditional terminology translation methods on csi translations, we evaluate two dictionary-based terminology methods on opensourced nmt and llms ( §5.2). moreover, to gauge the capacity of llms to leverage external knowledge, we evaluate 4 cultural knowledge prompting strategies on tuning-free llms ( §5.3). ## results and analysis in this section, we 1) compare the csi translation performances of llm-based mt systems with that of nmt systems ( §6.1); 2) evaluate dictionarybased terminology translation methods to explore if they work on csi translations ( §6.2); 3) compare four prompting strategies of llm-based mts to explore how different prompts affect the llms' cultural awareness ( §6.3); 4) conduct human eval-uation to verify the correlation between automatic evaluation metrics and human assessment ( §6.4). ## conclusion to advance culturally-aware machine translation, we curate a high-quality, diverse parallel corpus (camt) with rich csi annotations in 6 language pairs using an automated pipeline. we introduce two evaluation metrics, csi-match and pta, to assess translation quality concerning cultural nuances. our evaluation of llm-based mt and nmt systems using camt reveals that llms can effectively incorporate external cultural knowledge, enhancing the pragmatic translation quality of csis. our work provides essential data sources and insights for advancing culturally-aware machine translation, laying the groundwork for future investigation in this field.