C a MMT : Benchmarking Culturally Aware Multimodal Machine Translation (2025)

Abstract: Translating cultural content poses challenges for machine translation systems due to the differences in conceptualizations between cultures, where language alone may fail to convey sufficient context to capture region-specific meanings. In this work, we investigate whether images can act as cultural context in multimodal translation. We introduce CaMMT, a human-curated benchmark of over 5,800 triples of images along with parallel captions in English and regional languages. Using this dataset, we evaluate five Vision Language Models (VLMs) in text-only and text+image settings. Through automatic and human evaluations, we find that visual context generally improves translation quality, especially in handling Culturally-Specific Items (CSIs), disambiguation, and correct gender marking. By releasing CaMMT, our objective is to support broader efforts to build and evaluate multimodal translation systems that are better aligned with cultural nuance and regional variations.

## introduction translation brings cultures into contact. it usually involves deciding how much foreignness to keep in the resulting translation and invariably involves blending cultures to some extent @xcite . as pointed out by @xcite , part of the difficulty in deciding the right level of culture blending during translation arises from the different conceptualizations that each culture holds. translators must, therefore, choose suitable strategies for adapting vocabulary as well as deciding whether to conserve or substitute foreign elements. conforming the source text to the target culture by substituting unknown elements with familiar ones can ease comprehension, yet it simultaneously erases traces of the original culture @xcite . conversely, ignoring an adequate vocabulary choice that accounts for regional variation in the target language risks misinterpretation, as lexical choice directly shapes how readers understand a text @xcite . text-only machine translation inherits this dilemma with limited contextual knowledge to ground these translation decisions. however, images can supply that missing extra-linguistic information; visual reference may act as a cultural proxy, revealing a region's set of values @xcite as well as social practices and material culture, such as clothing, architecture, and food. with photography being thought of as a form of translation from reality into images @xcite , we hypothesize that images can capture additional information that language alone may struggle to encode. multimodal machine translation (mmt) @xcite attempts to embed this information by grounding source sentences with images. com-mute @xcite provides an evaluation framework for mmt centered on lexical disambiguation, but does not address broader cultural nuances, leaving questions about how visuals influence translation in culturally grounded settings largely unanswered. in this work, we present cammt ( culturally-aware multimodal machine translation benchmark), the first human-curated mmt corpus with triples across 19 languages of culture-related captions spanning 23 regions worldwide. additionally, we study the impact of visual grounding for culture-aware multimodal machine translation in vision-language models (vlms). to frame our study, we pose the following research questions: our contributions are as follows: ## related work in translation studies, csis @xcite refer to words or concepts that lack direct equivalents or carry different connotations in the target culture. these often arise when cultural references embedded in the source language do not directly exist or are understood differently in the target language. when translating csis, translators typically adopt one of two strategies: substitution, which adapts the foreign element into a culturally familiar counterpart to reduce its strangeness; or conservation, which preserves the original cultural reference, maintaining the source text's foreignness and exposing readers to its original context @xcite . efforts to incorporate cultural awareness into machine translation have been addressed in specific domains such as cultural adaptation in recipe translation @xcite . @xcite generalized beyond this scope by constructing an evaluation dataset by automatically extracting csis from wikipedia to study how llms and mt systems handle cultural references. however, the dataset is restricted to a smaller number of languages, automatically generated without input from regional speakers, and does not consider the effect of visual context on translation decisions. recent benchmarks such as cvqa @xcite , culturalvqa @xcite , alm-bench @xcite , and foodieqa @xcite demonstrate growing progress in regional image understanding within vlms. however, none of these works study how imagery can affect translation across cultures. together, these studies motivate our evaluation on the multimodal translation ability of vlms. ## cammt dataset cvqa @xcite is a visual question answering dataset comprising more than 10,000 questions across 39 country-language pairs. the questions within cvqa are formulated in both regional languages and english, classified into 10 distinct categories. to develop cammt, we utilized cvqa's question-answer pairs and transformed them into declarative statements using gemini 2.0 flash @xcite to generate parallel caption pairs in english and regional languages. no images were used in this process to ensure that the phrasing of these seed captions (later refined by annotators) was not influenced by them. the simplicity of the statements, combined with human curation, further reduced the risk of any bias from the language model. ## vlms for multimodal machine translation this section explains our motivation for using vlms as off-the-shelf mmt systems and our evaluation framework. we first present the selected models and validate their effectiveness for the task, followed by a description of our evaluation setup, which measures translation quality through human and automatic assessments in both text-only and text+image conditions in cammt. as discussed in section 2, task-specific mmt models are limited by their training data, often lacking coverage for many languages. on the other hand, llms have demonstrated strong performance in machine translation across multiple language pairs @xcite . as the paradigm shifts from text-only to multimodal llms which can process both text and images (vlms), we explore their potential for multimodal translation, particularly in culturally grounded scenarios. to initially assess the ability of vlms in ground-ing translations using images, we conduct a control experiment on the commute dataset @xcite , comparing them against strong task-specific mt and mmt baselines. commute consists of english sentences with ambiguous terms paired with two images that lead to different translations (e.g., mole may refer to an animal or a skin mark). thus, improvements when images are provided indicate that the model is effectively leveraging visual context to disambiguate the source sentence. in the text-only setting, models are prompted to translate from english to the target language. in the text+image setting, they are additionally provided with an image and prompted to use it as context for the translation (see appendix a.5 for prompt details). importantly, no further instructions are given regarding the nature of the disambiguation task. we evaluate five vlms: gemma 3 27b and 12b @xcite , qwen 2.5-vl 32b and 8b @xcite , and gemini 2.0 flash @xcite . results presented in we focus on the gemini, gemma, and qwen families, covering closed-weight and open-weight models at different scales. for completeness, we also report automatic evaluation results for aya-vision @xcite in appendix a.3, consistent with the main models' findings. all models are evaluated in both text-only and text+image setups. in the text+image setting, we do not explicitly instruct models to use images as a cultural reference, only as additional context, allowing us to observe their default effect in translation. to evaluate the impact of visual input on translation quality, we conduct both human preference evaluation and automatic evaluation using standard machine translation metrics and the curated pairs as ground truth. ## evaluation building on our experimental setup, this section presents the results of our multimodal translation evaluations. ## conclusions we present cammt, a human-curated dataset for multimodal machine translation that encompasses 19 languages across 23 regions. we evaluated five vlms at different scales on cammt and observed that providing images as auxiliary context generally improves translation quality in ways that native speakers find meaningful. when translations incorporate images, they tend to better preserve cultural elements, use correct gender marking, and resolve ambiguities. all of these improvements are often overlooked by automatic mt metrics. however, we also observe a non-trivial number of cases where visual input negatively affects translations. understanding when and why this occurs remains an important direction for future research. our findings also show that annotators tend to favor conserving csis, particularly when no clear equivalent exists in english, underscoring the importance of culturally sensitive translation strategies. future work should incorporate such speaker-aligned choices when designing models and datasets for grounded, culturally aware translation.