Seeing Symbols, Missing Cultures: Probing Vision-Language Models’ Reasoning on Fire Imagery and Cultural Meaning (2025)

Abstract: Vision-Language Models (VLMs) often appearculturally competent but rely on superficial pat.tern matching rather than genuine cultural understanding. We introduce a diagnostic framework to probe VLM reasoning on fire-themedcultural imagery through both classification andexplanation analysis. Testing multiple modelson Western festivals, non-Western traditions.and emergency scenes reveals systematic biases: models correctly identify prominent Western festivals but struggle with underrepresentedcultural events, frequently offering vague labelsor dangerously misclassifying emergencies ascelebrations. These failures expose the risksof symbolic shortcuts and highlight the needfor cultural evaluation beyond accuracy metrics to ensure interpretable and fair multimodalsystems.

## introduction vision-language models (vlms) demonstrate sophisticated capabilities, often appearing culturally aware by correctly identifying festivals and artifacts @xcite . this apparent competence creates a "semantic illusion" where pattern matching masquerades as understanding @xcite . a model might label an image as "torch festival" not from understanding its yi ethnic significance, but from associating visual cues like fire and crowds with festival tokens. this surface-level pattern matching creates critical vulnerabilities @xcite . common visual elements are semantically ambiguous and culturally polysemous @xcite . fire can signify celebration, crisis, or ritual transformation across cultures @xcite . when vlms use "symbolic shortcuts"-defaulting to familiar associations rather than contextual specificity-they risk misinterpreting cultural meaning @xcite . models unable to distinguish peru's sacred inti raymi from britain's lewes bonfire, or from dangerous fires, pose risks in public-facing or safety-critical applications and therefore warrant additional cultural-robustness evaluation before deployment @xcite . this paper investigates whether vlms understand cultural semantics or rely on symbolic shortcuts. we extend recent work on vlm cultural biases @xcite with a diagnostic framework probing reasoning patterns. we analyze both classification labels and explanations @xcite , evaluating models on visually similar but semantically distinct fire-themed images to expose reasoning failures that accuracy metrics miss. figure ?? illustrates our methodology. our approach differs from recent frameworks like cross @xcite in three key ways: we formally define symbolic shortcuts as reasoning patterns where models map visual elements (e.g., fire) directly to their most common semantic associations (e.g., festival) while neglecting contextual cues that would enable proper cultural interpretation. rather than a comprehensive benchmark, our contribution is a controlled diagnostic focused on a single multi-meaning symbol ("fire"), complementary to breadth-first cultural evaluations. our key contributions are: (1) a diagnostic framework that moves beyond accuracy to evaluate vlm reasoning through classification and explanation. (2) a targeted analysis revealing how symbolic shortcuts lead to cultural misinterpretations and safety risks. (3) evidence of a significant reasoning gap between western and non-western cultural contexts, highlighting data bias and fairness issues in state-of-the-art models. ## symbolic reasoning probe our diagnostic framework is designed to probe the reasoning behind a vlm's cultural classifications. it assesses whether a model's output is based on genuine semantic understanding or a reliance on superficial visual cues. the probe consists of three components: a curated dataset with controlled semantic ambiguity, a selection of diverse vlms, and an evaluation protocol that demands both classification and explanation. ## findings state-of-the-art vlms consistently favor symbolic shortcuts over cultural reasoning, evident in qualitative output differences and varied performance across categories. three primary failure modes emerge: (1) cultural misclassification-labeling emergencies as cultural events; (2) generic labeling-using vague descriptors; (3) western-centric biasdefaulting to familiar western events. critically, qwen misinterprets a wildfire as guy fawkes night-a safety-critical failure where models hallucinate familiar cultural contexts onto dangerous events (see appendix a.3). tables 2 and 3 quantify this imbalance and demonstrate the qualitative differences. performance on burning_man_american reaches 100%, but drops to 0% for sadeh_iranian and huobajie, revealing bias toward western, internet-prominent events. figure 1 visualizes failure patterns. gpt-4o shows distributed errors, while qwen2.5-vl 7b systematically defaults to guy fawkes or burning man when uncertain-consistent with reliance on symbolic shortcuts. ## discussion our findings reveal a gap between visual pattern recognition and cultural understanding. models' "symbolic shortcuts"-overgeneralizing fire as festival-create competence illusions masking reasoning failures. data imbalance drives these failures. superior performance on burning man versus poor results on huobajie and sadeh reveals western-centric training data bias @xcite . models learn simplified dominant representations, not varied cultural meanings. ## conclusion this paper introduced a diagnostic probe to move beyond accuracy-based evaluation and assess the cultural reasoning of vision-language models. our findings reveal that current models, including state-of-the-art systems like gpt-4o, often rely on "symbolic shortcuts," leading to a superficial understanding that fails in nuanced, non-western, or safety-critical contexts. they can see the symbols, but they often miss the culture. we argue for a crucial transition in how we evaluate ai systems for cultural tasks: a shift from measuring what they classify to understanding how and why they reason. this explanation-driven approach is essential for identifying fairness risks associated with data bias and for building models that are not only accurate but also genuinely and safely culturally aware. this work provides a framework and a baseline for this necessary next step in ai development.