Towards Automatic Evaluation for Image Transcreation (2025)

Abstract: Beyond conventional paradigms of translating speech and text, recently, there has been interest in automated transcreation of images to facilitate localization of visual content across different cultures. Attempts to define this as a formal Machine Learning (ML) problem have been impeded by the lack of automatic evaluation mechanisms, with previous work relying solely on human evaluation. In this paper, we seek to close this gap by proposing a suite of automatic evaluation metrics inspired by machine translation (MT) metrics, categorized into: a) Object-based, b) Embedding-based, and c) VLM-based. Drawing on theories from translation studies and real-world transcreation practices, we identify three critical dimensions of image transcreation: cultural relevance, semantic equivalence and visual similarity, and design our metrics to evaluate systems along these axes. Our results show that proprietary VLMs best identify cultural relevance and semantic equivalence, while vision-encoder representations are adept at measuring visual similarity. Meta-evaluation across 7 countries shows our metrics agree strongly with human ratings, with average segment-level correlations ranging from 0.55-0.87. Finally, through a discussion of the merits and demerits of each metric, we offer a robust framework for automated image transcreation evaluation, grounded in both theoretical foundations and practical application. Our code can be found here: https://github.com/simran-khanuja/automatic-eval-transcreation

## introduction while mt has made commendable progress in the text and speech modalities @xcite , there has recently been interest in the ml community to approach translation in the visual modality. this new task, called image transcreation, seeks to adapt (or localize) images originating from a source culture for a given target culture in an attempt to enhance familiarity and relatability for the target audience @xcite . we provide a popular real-world example of image transcreation in figure 1 , where content was recreated to match cultural norms of a target audience. while this problem is well-explored in translation studies (section 2) and is carried out widely in industrial settings by human translation experts (section 3), it has not yet taken off in ml, despite the availability of powerful vision-language models (vlms). a central reason for this is the lack of automated evaluation mechanisms, with prior works relying solely on expensive human evaluation. @xcite , for instance, report spending about 4000 usd for annotating a test set of 3500 images across 7 countries, with only 1 annotator per image. while such a contribution is indeed beneficial, as ml researchers, we naturally need automated metrics to train, validate and benchmark image transcreation systems at scale. and yet, the very nature of the task makes it difficult to propose a singular automated metric. unlike traditional natural language processing (nlp) problems like summarization or translation that have a relatively small space of valid outputs, there can be multiple valid ways of transcreating images. the suitability of an adaptation depends on factors like the intent/demands of the user/application, the creativity of the transcreator and the degree of cultural relevance required for the target audience. this subjectivity makes it hard to define a singular metric like bleu @xcite , rouge @xcite , or even neural metrics like comet @xcite that optimize for similarity of model outputs against a fixed set of reference(s). in this paper, we make progress towards building and evaluating automated metrics for image transcreation. first, given that this is a novel task in ml, we turn to translation studies for a better understanding of its scope and variables (section 2). we identify key dimensions critical to define the image transcreation process: cultural relevance, semantic equivalence, and visual similarity, and look at real-world applications of transcreation through the lens of these dimensions in section 3. grounded in the conceptual and practical insights from previous sections, and drawing inspiration from mt metrics for text, we propose three metrics that account for the varied dimensions of this subjective task -a) object-based; b) embedding-based and c) vlm-based metrics, each of which leverages powerful open and closed vlms and large language models (llms). after conducting metaevaluation along all three dimensions, we report average segment-level (i.e. instance-level) correlations with human ratings of up to 0.87 on image transcreation test-sets spanning 7 countries. thus, our primary contributions are: ## transcreation in translation studies transcreation has been widely studied by linguists and philosophers in translation studies. we trace its development in this linguistics-adjacent field below, to ground our evaluation in this research. ## visual transcreation in the wild while text translation operates in a relatively constrained and predictable output space, image transcreation allows for a broader range of creative expression and the desired output depends on the user's objectives and application context, as elucidated above. this intent or purpose of transcreation also drives the extent of (a) cultural relevance, (b) semantic equivalence, and (c) visual similarity needed in the final output, as we describe below: ## experiments in this section, we conduct a meta-evaluation of the transcreation metrics proposed above. 2024 ) create a dataset comprising of commonly occurring concepts across 17 semantic categories (like food, beverages, mammals etc.) that are cross-culturally comprehensive, collected across seven countries. the final dataset includes 585 images, and they apply their proposed pipeline with each country as the target, using images from all other countries as the source. around 500 images are transcreated for each target country ( ## conclusion in this paper, we take a first step towards automatically evaluating image transcreation -the task of adapting visual content from a source culture to a target culture. we first review related works in translation studies and identify that cultural relevance for the target audience, semantic equivalence of the transcreated image, and its visual similarity with the source image are three measurable dimensions which potentially cover a wide range of applications. we explore how visual content is being transcreated for real-world applications through this lens, and design evaluation metrics that rank systems across these three axes, drawing inspiration from mt metrics. we find that vlm-based metrics excel at multimodal reasoning (cultural relevance and semantic equivalence) but visual similarities can be better captured using visual-encoder embeddings. for all metrics, capturing cultural relevance of an image for a target demographic is challenging, leaving headroom for progress. we hope that our study helps ml researchers make progress on this important, real-world task.