An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance (2024)

Abstract: Given the rise of multimedia content, human translators increasingly focus on culturally adapting not only words but also other modalities such as images to convey the same meaning. While several applications stand to benefit from this, machine translation systems remain confined to dealing with language in speech and text. In this work, we introduce a new task of translating images to make them culturally relevant. First, we build three pipelines comprising state-of-the-art generative models to do the task. Next, we build a two-part evaluation dataset – (i) concept: comprising 600 images that are cross-culturally coherent, focusing on a single concept per image; and (ii) application: comprising 100 images curated from real-world applications. We conduct a multi-faceted human evaluation of translated images to assess for cultural relevance and meaning preservation. We find that as of today, image-editing models fail at this task, but can be improved by leveraging LLMs and retrievers in the loop. Best pipelines can only translate 5% of images for some countries in the easier concept dataset and no translation is successful for some countries in the application dataset, highlighting the challenging nature of the task. Our project webpage is here: https://machine-transcreation.github.io/image-transcreation and our code, data and model outputs can be found here: https://github.com/simran-khanuja/image-transcreation.

## introduction we shall try... to make not word-for-word but sense-for-sense translations. transcreation is prevalent in several fields and its precise implementation is often tied to the endapplication, as shown in figure 1 . for example, in audio-visual media (av), the goal is to evoke similar emotions across diverse audiences. in line with this goal, the japanese cartoon doraemon made many changes like replacing omelet-rice with pancakes, chopsticks with forks and spoons or yen notes with dollar notes, when adapting content for the us. 3 sometimes, the translation is contextdependant, as in the us movie inside out, where bell peppers is used as a substitute for broccoli in japan, as a vegetable that children don't like. in education, the goal is to create content that includes objects a child sees in their daily surroundings, known to aid learning @xcite . many worksheets already do this, where the same concepts of addition and counting are taught using different currency notes or celebrationthemed worksheets, in different regions. finally, in advertisements and marketing, we see global brands localize advertisements to sell the same where the same concepts are taught differently in different countries, using local currencies or celebration-themed worksheets; c) advertisements: where the same product is packaged and marketed differently, like in ferrero rocher taking the shape of a lunar festival kite in china, and that of a christmas tree elsewhere. contribution 1 (task): in this paper, we take a first step towards transcreation with machine learning systems, by assessing capabilities of generative models for the task of image transcreation across cultural boundaries. in text-based systems alone, models struggle with translating culture-specific information, like idioms @xcite . moreover, to our knowledge, automatically transcreating visual content has previously been unaddressed. contribution 2 (pipelines): in §2, we introduce three pipelines for this task -a) e2e-instruct (instruction-based image-editing): that edits images directly following a natural language instruction; b) cap-edit (@xmath0: that first captions the image, makes the caption culturally relevant, and edits the original image as per the culturally-modified caption; c) cap-retrieve (@xmath1: that uses the culturally-modified caption from cap-edit to retrieve a natural image instead. we also experiment with gpt-4o and dalle-3 to generate new images using culturally-modified captions ( §a.4). contribution 3 (evaluation dataset): given the unprecedented nature of this task, the evaluation landscape is a blank slate at present. we create an extensive and diverse evaluation dataset consisting of two parts (concept and application), as detailed in §3. concept comprises 600 images across seven geographically diverse countries: brazil, india, japan, nigeria, portugal, turkey, and united states. five culturally salient concepts and related images are collected across a consistent set of universal categories (like food, beverages, celebrations, and so on) from each country. application comprises 100 images curated from real-world applications like educational worksheets and children's literature. contribution 4 (human evaluation): in §4, we conduct human evaluation of images transcreated for both concept and application, across all seven countries. we find that as of today, image-editing models fail at this task, but can be improved by leveraging llms and retrievers in the loop. even the best models can only successfully transcreate 5% images for nigeria in the simpler concept dataset and no image transcreation is successful for instruction make this image culturally relevant to the united states caption a plate of food with meat and greens on it llm-edit for cultural relevance e2e-instruct: instruction-based image editing cap-edit: caption, text-edit, image-edit cap-retrieve: caption, text-edit, image-retrieval a dish of food with beef and vegetables on it (target: united states) image editor (plugnplay) caption a plate of food with meat and greens on it llm-edit for cultural relevance (target: united states) a dish of food with beef and vegetables on it query laion-us (images with .us in url) image editor (instructpix2pix) input image category: food country: nigeria (amala-and-ewedu) target: united states figure 2 : pipelines to transcreate images: e2e-instruct takes as input the original image and a natural language instruction; cap-edit first captions the image, uses a llm to edit the caption for cultural relevance, and edits the original image using the llm-edit as instruction; and cap-retrieve uses this llm-edit to retrieve a natural image from a country-specific image dataset. given the unprecedented nature of this task, we create pipelines using pre-existing sota models, and benchmark them on our newly created test set. ## pipelines for image transcreation we introduce three pipelines for image transcreation comprising of state-of-the-art generative models. the code to run all pipelines with exact prompts used can be found in table d. 2. an overview of each pipeline is in figure 2 . ## evaluation dataset we design a two-part dataset where the first (concept) is meant to serve as a research prototype, while the second (application) is grounded in realworld applications like those in figure 1 . ## human evaluation and quantitative metrics evaluation of image-editing models typically relies on quantitative metrics and qualitative analysis of a few select samples. 10 while image-editing focuses on image quality and how closely the edit follows the instruction, image-transcreation comes with additional requirements such as cultural relevance, meaning preservation, and so on. hence, we design an extensive questionnaire and conduct id question property applications performance concept dataset c0 is there any visual change in the generated image compared to the original image? visualchange none (helps filter non-edits) e2e-instruct cap-edit cap-retrieve c1 is the generated image from the same semantic category as the original image? semanticequivalence av (zootopia); education e2e-instruct cap-edit cap-retrieve c2 does the generated image maintain spatial layout of the original image? spatiallayout av (doraemon, inside out) e2e-instruct cap-edit cap-retrieve c3 does the image seem like it came from your country/ is representative of your culture? cultureconcept av, education, ads e2e-instruct cap-edit cap-retrieve c4 does the generated image reflect naturally occurring scenes/objects? naturalness ads (ferrero rocher) e2e-instruct cap-edit cap-retrieve c5 is this image offensive to you, or is likely offensive to someone from your culture? offensiveness all e2e-instruct cap-edit cap-retrieve -for edited images, is the change meaningful (c1) and culturally relevant (c3)? meaningfuledit all e2e-instruct cap-edit cap-retrieve application dataset e/s0 is there any visual change in the generated image compared to the original image? visualchange none (helps filter non-edits) e2e-instruct cap-edit cap-retrieve e1 can the generated image be used to teach the concept of the worksheet? educationtask education e2e-instruct cap-edit cap-retrieve s1 would the generated image match the text of the story in a children's storybook? story-text av, literature e2e-instruct cap-edit cap-retrieve e/s2 does the image seem like it came from your country/is representative of your culture? cultureapplication all e2e-instruct cap-edit cap-retrieve -for edited images, is the change meaningful (e/s1) and culturally relevant (e/s2)? meaningfuledit all e2e-instruct cap-edit cap-retrieve ## conclusion in this paper, we introduce a new task of image transcreation with machine learning systems, where we culturally adapt visual content to suit a target audience. translation has traditionally been limited to language, but with increased consumption of multimedia content, translating all modes in a coherent way is essential. we build three pipelines comprising state-of-the-art generation models, and show that end-to-end image editing models are incapable of understanding cultural contexts, but using llms and retrievers in the loop helps boost performance. we create a challenging two-part evaluation dataset: (i) concept which is simple, cross-culturally coherent, and diverse; and (ii) application which is curated from education and stories. we conduct an extensive human evaluation and show that even the best models can only translate 5% images for select countries (like nigeria) in the easier concept dataset and no image transcreation is successful for some countries (like portugal) in the harder application dataset. our code and data is released to facilitate future work in this new, exciting line of research.