Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models (2025)

Abstract: Text-To-Image (TTI) models, such as DALL-E and StableDiffusion, have demonstrated remarkable prompt-based image generation capabilities. Multilingual encoders may have a substantial impact on the cultural agency of these models, as language is a conduit of culture. In this study, we explore the cultural perception embedded in TTI models by characterizing culture across three tiers: cultural dimensions, cultural domains, and cultural concepts. Based on this ontology, we derive prompt templates to unlock the cultural knowledge in TTI models, and propose a comprehensive suite of evaluation techniques, including intrinsic evaluations using the CLIP space, extrinsic evaluations with a Visual-Question-Answer models and human assessments, to evaluate the cultural content of TTI-generated images. To bolster our research, we introduce the CulText2I dataset, based on six diverse TTI models and spanning ten languages. Our experiments provide insights regarding Do, What, Which, and How research questions about the nature of cultural encoding in TTI models, paving the way for cross-cultural applications of these models.1

## introduction tti models exhibit multilingual proficiency, acquired explicitly, through the model architecture and objective, or implicitly, through exposure to multiple languages only ( §2). these models find widespread use in domains such as art, education, and communication, exerting substantial societal influence @xcite @xcite . their cultural significance stems from their multilingual competence and extensive adoption, as language is a vessel for cultural identity and heritage. indeed, @xcite have demonstrated ai models' pivotal role in enhancing cultural transmission. this study aims to gain insight into the cultural perception inherent in tti models. we embark on a novel characterization, dissecting the complex ties between language, culture, and tti models. our approach is inspired by well-established cultural research @xcite @xcite , allowing us to systematically deconstruct the wide notion of culture across three tiers, progressing from the broader to the finer levels of abstractness: cultural dimensions, cultural domains, and cultural concepts. this ontology allows us to derive prompt templates, with which we aim to unlock the cultural knowledge encoded within tti models, and a suite of evaluation measures which reflects different aspects of the cultural information in a generated image. these methods consist of intrinsic evaluations using the clip @xcite space, extrinsic evaluations with visual question answering (vqa) models @xcite , and human assessments. to address the lack of an appropriate dataset, we introduce cultext2i, comprising images generated by six distinct tti models, varying in multilingual capability and architecture ( §2). these models include stablediffusion 2.1v and 1.4v, altdiffusion, deepfloyd, dall-e, and a llama 2 + sd 1.4 unet model based on llavi-bridge @xcite @xcite @xcite . we generate images using prompts representing identical cultural concepts across ten languages (see figure 1 ). in §3 we present our research questions, aiming to address the way cultural knowledge is encoded by and can be unlocked in tti models, the effective ways of unlocking this knowledge and the resulting conclusions about the world cultures. multilingual text encoders @xcite @xcite opened the door to a wide range of cultural influences on tti models, which are at the heart of this study. the multilingual capabilities, varying in their semantic interpretation and how well they align images to the requested concept in the prompt (i.e., conceptual coverage) @xcite are acquired either by explicit training objectives (e.g., deepfloyd if @xcite and altdiffusion @xcite ), such as in the training of xlm-r and t5, or implicitlyonly through exposure to different languages in their training corpus, as in the case of stablediffusion v2.1 or v1.4 @xcite which employ clip text encoder. this work synthesizes the cultural literature presented above to establish a comprehensive repository of cultural concepts and dimensions. in §4, we introduce our culture characterization for detailed analyses of the representation and perception of cultural perspectives within tti models. culture in lms and tti models with the burgeoning interest in pre-trained language models (plms) and tti diffusion models, there is increasing scrutiny of the cultural gaps and biases inherent within these models. these gaps manifest as discrepancies in the representation of norms, values, beliefs, and practices across diverse cultures @xcite @xcite @xcite @xcite . @xcite and ramezani and xu (2023) explored the cross-cultural values and moral norms in plms and assessed their alignment with theoretical frameworks, revealing a conspicuous inclination towards western norms. similarly, other studies demonstrated the challenges with english probes and monolingual lms, which diminish the representation of non-western (e.g., arab) norms in model responses @xcite @xcite @xcite . while these works focused on the cultural implications of plms, detecting or mitigating cultural biases, we develop a methodology that inspects the tti models' cultural values, and seek to understand their internal representations of cultures. while cultural exploration in tti model research is relatively limited, there have been advancements to enhance cultural diversity within these models. multilingual benchmarks, focusing on chinese and western european/american cultures, have been introduced @xcite . efforts to uncover cultural biases include evaluations of nationality-based stereotypes @xcite , skin tone biases @xcite , and associated risks @xcite . analyses also address social biases in english-only tti models (naik and nushi, 2023), covering gender, race, age, and geography. @xcite and @xcite have evaluated the cultural competence of these models. our work extends beyond western tendencies, focusing on multilingual tti models and exploring fine-grained cultural concepts and dimensions. ## research questions and overview our primary research inquiry revolves around: how does multilingual tti models capture cul-tural differences? to delve into this overarching question, we craft four research questions: rqs 1-3 form a hierarchy, with each question building upon the previous one. rq4 stands as an independent, high-level question. our methodology consists of three pillars: (1) crafting a cultural ontology; (2) experimenting with tti models featuring diverse multilingual text encoders, and (3) employing a triad of evaluation methodologies: intrinsic evaluation using openclip, extrinsic evaluation involving vqa models, and human assessment. ## cultural ontology we aim to design an ontology that will allow us to (1) consolidate diverse perspectives on culture; and (2) quantitatively assess cultural aspects within the context of tti models. research on cultural definitions is typically based on breaking down the big idea of culture into different aspects like individualism or science, which often involve intricate and abstract details or queries unsuitable for visual examination. to utilize culture studies to our needs, we hence develop two key pillars: (a) cultural domains, comprising cultural concepts; and (b) cultural dimensions. cultural domains and concepts. drawing inspiration from established categorizations in works like @xcite , @xcite , and haerpfer et al. (2022) , we combine ten common and broad aspects to form the cultural domains. each domain reflects a collection of values, tendencies, and beliefs, which we represent through concise concepts. for instance, the cultural concept of heaven in the religion domain is derived from the question in the religion section of the world values survey, which asks: ''do you believe in heaven?''. we define twelve domains and 200 cultural concepts (see cultural dimensions. certain cultural aspects function more like axes (e.g., from individualism to collectivism) than as comprehensive domains (e.g., science). we grouped these aspects under the category of cultural dimensions. the dimensions we use are defined as follows: (1) traditional versus rational values; foot_1 (2) survival versus self-expression values (haerpfer et al., 2022); (3) critical versus kindness; foot_2 (4) extraversion versus introversion (mccrae and allik, 2002); (5) modern versus ancient values; (6) masculine versus feminine attributes; (7) individualism versus collectivism and (8) nature versus human @xcite . 5 the cultural dimensions don't cover all suggested aspects from the original research. we focused on aspects that are more visually representable and quantifiable. ## unlocking culture in tti models we now introduce our prompt templates which feed the cultural concepts as input into the tti models, aiming to unlock the effect of different cultures on their outputs. ## evaluating cultural aspects in images in this section, we discuss how we evaluate the images' cultural content. we employ two automatic measures for cultural characteristics (intrinsic and extrinsic), as well as human assessment. ## ablation analysis to support our key findings, we analyze potential factors behind the results. we validate cultural dimensions with ground-truth data, propose the characters embedding to explain cultural distance and cross-cultural similarity scores, and test the national association task with different input data and vqa. additionally, we conduct a qualitative analysis to reveal underlying representations in the generated images. ## discussion and future work we studied cultural encoding in tti models, a research problem that, to our knowledge, has not been addressed before. we mapped the quite abstract notion of culture into an ontology of concepts and dimensions, derived prompt templates that can unlock cultural knowledge in tti models, and developed evaluation measures to evaluate the quality of the cultural content of the resulting images. by doing so we were able to answer do, what, which, and how research questions about the nature of cultural encoding in tti models, and to highlight a number of future research directions.