C ultural F rames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics (2025)

Abstract: The increasing ubiquity of text-to-image (T2I) models as tools for visual content generation raises concerns about their ability to accurately represent diverse cultural contexts - where missed cues can stereotype communities and undermine usability. In this work, we present the first study to systematically quantify the alignment of T2I models and evaluation metrics with respect to both explicit (stated) as well as implicit (unstated, implied by the prompt’s cultural context) cultural expectations. To this end, we introduce CulturalFrames, a novel benchmark designed for rigorous human evaluation of cultural representation in visual generations. Spanning 10 countries and 5 socio-cultural domains, CulturalFrames comprises 983 prompts, 3637 corresponding images generated by 4 state-of-the-art T2I models, and over 10k detailed human annotations. We find that across models and countries, cultural expectations are missed an average of 44% of the time. Among these failures, explicit expectations are missed at a surprisingly high average rate of 68%, while implicit expectation failures are also significant, averaging 49%. Furthermore, we show that existing T2I evaluation metrics correlate poorly with human judgments of cultural alignment, irrespective of their internal reasoning. Collectively, our findings expose critical gaps, provide a concrete testbed, and outline actionable directions for developing culturally informed T2I models and metrics that improve global usability.

## introduction visual media such as advertisements, posters, and public imagery play a central role in encoding and transmitting cultural values @xcite . they often depict culturally specific elements (e.g., traditional attire, religious symbols) and embed societal norms and values (e.g., expectations around family structure, gender roles, and etiquette), thus both reflecting and influencing the cultures from which they originate @xcite . text-to-image (t2i) models are emerging as a significant component of this visual media ecosystem, now adopted across diverse domains like education, marketing, and storytelling @xcite @xcite . this magnifies the cultural implications of their outputs for global audiences @xcite and raises a critical question: how accurately, and with what depth, do these models depict diverse cultures? while t2i models may generate visually plausible outputs for cultural prompts (e.g., "a bride and groom exchanging vows at their hindu wedding," fig. 1 ), they often capture explicit details while omitting implicit elements central to the scene, (such as a sacred fire or officiating priest). we refer to these two classes as explicit (based on the words in the prompt) and implicit (unstated but implied by the prompt's cultural context) expectations. indeed, t2i model performance hinges on accurate cultural representation, which can foster familiarity and trust. inaccuracies, however, risk reinforcing stereotypes, exclusion, or propagating dominant narratives @xcite . this necessitates evaluation practices that not only verify faithfulness to the explicit expectations but also assess the inference and contextualization of implicit cultural expectations. however, current t2i evaluation methodologies predominantly focus on the former by assessing explicit promptimage consistency using automated metrics @xcite @xcite . 1 further, existing benchmarks for evaluating t2i models are designed around prompts that emphasize attributes like realism @xcite , compositionality @xcite , and a bride and groom exchanging vows at their hindu wedding in india. chinese family sharing peking duck in a bustling shanghai restaurant. ## culturalframes we detail our data collection pipeline below and highlight the design decisions that make it distinct from standard annotation efforts. ## data analysis prompts. culturalframes consists of 983 prompts collected from 10 countries, with each country contributing between 90 and 110 prompts, ensuring balanced cross-country representation. the prompts are distributed across five cultural categories introduced in § 3.2: etiquette (24.3%), religion (14.4%), family (14.2%), greetings (13.1%), and dates of significance (34%). for a detailed per-country breakdown, see fig. 14 in § a.3. images. we generate images for our prompt set using both open-and closed-source models. while open-source models produce an image for every prompt, the safety filters of closed-source models block a subset of generations. this issue is most noticeable with imagen3, which filters out 290 prompts-29.5% of the prompts, primarily due to policies against depicting children foot_4 . for comparison, gpt-4o blocks only 5 prompts. in total, we collect 3,637 images. inter-rater agreement. we collect a total of 10,911 ratings, with each image rated by three annotators. to measure agreement among raters, we compute krippendorff's alpha @xcite , obtaining 0.32 for prompt alignment, 0.28 for image quality, and 0.36 for the overall score. these scores are comparable to or better than prior works on cultural understanding in t2i models @xcite . a detailed comparison with prior works, along with factors influencing the agreement scores, is provided in appendix a.7. what aspect of the generated image dominates annotators' overall assessment? we find that the overall score given by annotators is strongly correlated with image-prompt alignment (spearman rank correlation of 0.68), whereas image quality shows a more moderate correlation of 0.45. this trend holds consistently across countries, suggesting that annotators prioritize faithfulness to the prompt over aesthetic appeal when rating images. also, stereotype is negatively correlated with overall score weakly (-0.21), which indicates a lower impact of the presence of stereotypes on overall score. interestingly, the results contrast with findings from prior work using side-by-side image comparisons @xcite , where image quality often dominates overall preference judgments. ## evaluating t2i models on culturalframes how do different models perform for different criteria across different countries? fig. 4 shows human evaluation results for prompt alignment, image quality, stereotype, and overall score. we find that gpt-image achieves the highest prompt alignment (0.85), followed by imagen3 (0.79). the open-source models, sd-3.5-large and flux, fall behind with scores of 0.66 and 0.63, respectively. for image quality, imagen3 is rated highest, with gpt-image and flux performing comparably well. sd-3.5-large, however, scores far behind the other models. across all models, including the state-of-the-art closed-source ones, the proportion of images rated stereotypical ranged from 10% to 16%, with sd-3.5-large generating stereotypical visuals the most and flux the least. overall, raters prefer images from gpt-image, consistent with the prompt alignment result. sd received the lowest overall score, most likely due to poorer image quality and higher stereotype levels, despite outperforming flux in prompt alignment. our findings (fig. 5 and which aspect-implicit or explicit-do models fail to capture, and is this consistent across countries? across culturalframes, annotators gave sub-perfect scores (below 1) for 44% of the time. out of these, 50.3% are attributed to issues with explicit elements, 31.2% to implicit elements, and 17.9% to both. while explicit errors are most common, implicit cultural failures still account for 49.1% of these cases, underscoring persistent challenges in capturing culturally nuanced, context-dependent knowledge. fig. 3 shows that gpt-image has the lowest overall image-prompt alignment error rate (@xmath0, with its errors roughly evenly split between implicit and explicit types. in contrast, other models, particularly sd-3.5-large and flux, exhibit higher total error rates where explicit errors form the largest share of their respective alignment failures. these results indicate that improvements are needed in both explicit and implicit cultural modeling. in canada, poland, germany, and brazil, approximately two-thirds of comments mention explicit prompt mismatches, indicating that literal fidelity dominates their feedback. conversely, annotator feedback from india, china, and south africa is more evenly distributed, with roughly half of the remarks targeting explicit flaws and half targeting implicit cultural elements. at the opposite end of the spectrum, annotators from japan and iran predominantly highlight implicit cultural elements, such as absent rituals, attire, or local setting, with only about one-third of their comments citing explicit tokens. chile follows the latter trend, albeit less strongly. collectively, these observations indicate that t2i models increasingly fail to capture users' implicit cultural expectations in regions like asia and the middle east, as contrasted with user feedback from the americas and europe. which words do models most frequently misinterpret? fig. 22 displays every word in the prompt that at least one rater labeled as erroneous, revealing two striking patterns. first, country demonyms (e.g., iranian, brazilian, chinese, japanese) are prominent. a closer examination of the rater comments reveals these words are typically highlighted as errors for two reasons: (i) a country-specific element is missing from the image, or (ii) the annotators are not able to relate to the depicted content. second, terms such as family, festival, ceremony, wedding, temple, meal, guests, tea, greeting, music, costumes, and flags account for much of the remaining error frequency. these words represent broad cultural signifiers-rituals, social roles, and iconic objects-indicating that t2i models frequently misrepresent such elements. in what way do models fail across different countries? to identify reasons behind model failures, we analyze free-form comments collected from annotators. for each country, we embed the comments using a sentence transformer foot_5 and cluster them using hdbscan @xcite . we then prompt gpt-4o to summarize each cluster with a concise label and explanations. this approach reveals distinct failure patterns across regions. in asia, models frequently misrepresent traditions and religious practices, often relying on stereotypes. in african contexts, outputs lacked cultural authenticity, defaulting to generic or westernized portrayals. south american outputs suffered from poor regional specificity and inaccurate depictions of people's appearances. similarly, canadian content lacked appropriate demographic diversity and indigenous representation. further, we investigate the nature of the generated images by embedding them using the clip vision encoder. 7 as shown in fig. 6 , images generated by imagen3 for asian countries form distinct clusters, while those from other regions lack such clear grouping. this finding is corroborated by annotators in europe and south america, who struggle to identify country-specific visual cues in generated images, indicating that the model fails to capture cultural distinctiveness. ## evaluating t2i metrics on culturalframes metrics analyzed. we analyze six representative metrics, each reflecting a different evaluation paradigm: clipscore @xcite is an embedding-based metric that computes cosine similarity between clip embeddings of the image and prompt. hpsv2 @xcite enhances clipscore by fine-tuning the clip model on human preference data. tifa @xcite uses a vqa-based framework to assess faithfulness. we use gpt-4o-mini for question generation and qwen2.5-vl-32b-instruct @xcite as the answering model. vqascore @xcite , unifiedreward @xcite , and vi-escore @xcite leverage vision-language models to evaluate generated images. for vqascore, we leverage the clip-flant5 model introduced in the original vqascore paper, use unifiedreward-qwen-7b based on qwen2.5-vl-7b for unifiedreward, and use gpt-4o as vlm for viescore, which provides both a score and a textual reason for its assessment. finally, we evaluate deqa @xcite , a vlm trained specifically for image-quality assessment. how do metrics perform against different rating criteria? we evaluate how well current t2i metrics correlate with human judgments across prompt alignment, image quality, and overall score (see fig. 7 ). unifiedreward, an open-source reward model, slightly edges the best closed-model setup, viescore, on prompt alignment, achieving a spearman correlation of 0.31 compared to 0.30 for the latter. while this is below the human-human agreement of 0.38, it notably outperforms all other metrics. in contrast, tifa exhibits a lower correlation, potentially because it only accounts for explicit elements mentioned in the prompt. this highlights a gap between metric design and alignment with human perception. the performance gap is even more pronounced for image quality, where all metrics correlate poorly with human ratings. nevertheless, viescore again performs best, followed closely by unifiedreward. the relatively stronger performance of hpsv2 may be attributed to its training on image pairs, with human preference likely driven by image quality, potentially making it more sensitive to visual appeal. by contrast, deqa, despite being trained specifically for image-quality assessment on standard iqa datasets, shows near-zero correlation (≈ 0.0) on our benchmark, likely due to domain and distribution shift between the data used to train deqa and cultural-frames. taken together, the overall weak correlations suggest that current metrics fail to capture the subjective nature of image quality as assessed by humans. for the overall score, viescore again demonstrates the highest alignment with human judgments, achieving a correlation of 0.31 (human-human: 0.42), with unifiedreward close behind. notably, hpsv2, despite being trained on human preferences, shows relatively poor performance, likely due to limited annotator and prompt diversity in the human preference dataset it was trained on. clipscore consistently underperforms, indicating limitations as a general-purpose evaluation metric, particularly for culturally sensitive image assessments. overall, these results suggest that vlm-based metrics have the upper hand in capturing culturally grounded human. ## discussion based on our analysis of cultural misalignment in text-to-image models and their evaluation metrics, we highlight three key directions for improvement. can culturally informed prompt expansion improve cultural alignment? culturalframes prompts are concise, leaving many cultural aspects implicit for the model to infer. for example, the prompt "a bride and groom exchanging vows at their hindu wedding" omits elements like the priest or the sacred fire. to examine whether making these cues explicit can improve generations, we build on our analysis of model failures and expand the 20 lowest-scoring prompts per country using gemini-2.5-flash (see § c.1 for details). we generate images using flux.1-dev, and evaluate image-prompt alignment with viescore, the metric that best correlates with human judgments. prompt expansion improves the average viescore from 7.3 to 8.4, demonstrating that culturally informed expansion helps models better capture details important to humans. can we improve metric performance through explicit instructions? current t2i metrics are not explicitly guided to account for implicit and explicit prompt elements. to test whether such guidance helps, we modify viescore by replacing gpt-4o's instructions with novel annotation guidelines we developed for human raters (see fig. 24 ). this raises correlation with human ratings from 0.30 to 0.32 (significant at 95% confidence) and improves explanation alignment with human rationales using the same llm-as-a-judge setup from 2.19 to 2.37 on a 5-point scale. these results suggest that culturally informed instruction design can improve both scores and rationales. nonetheless, the metric's reasoning still lags human rationale, highlighting the need for richer cultural knowledge and training beyond prompt design. does explicit training of vlms to judge images improve culturally aligned evaluation? current vlms used for evaluation are typically not trained to judge images, raising the question of whether such training aids cultural alignment. we compare unifiedreward (wang et al., 2025), trained on diverse multimodal preference data to judge images, with its backbone model (qwen2.5-vl-7b). despite not targeting cultural scenarios, uni-fiedreward achieves higher correlations with human judgments -image-prompt alignment (0.31 vs. 0.17), image quality (0.17 vs. 0.01), and overall score (0.28 vs. 0.14) -and even surpasses vi-escore in alignment (0.31 vs. 0.30). this suggests preference-based judge training can significantly improve cultural alignment in metric scores. ## conclusions in this work, we introduce culturalframes, a novel benchmark comprising 983 cultural prompts, 3,637 generated images, and 10,911 human annotations, spanning ten countries and five socio-cultural domains. culturalframes assesses the ability of t2i models to generate images across diverse cultural contexts. we find that state-of-the-art t2i models not only fail to meet the more nuanced implicit expectations, but also the less challenging explicit expectations. in fact, models fail to meet cultural expectations 44% of the time on average across countries. failures to meet explicit expectations averaged a surprisingly high 68% across models and countries, with implicit expectation failures also significant at 49%. finally, we demonstrate that existing t2i evaluation metrics correlate poorly with human judgments of cultural alignment.