Creating a Lens of C hinese Culture: A Multimodal Dataset for C hinese Pun Rebus Art Understanding (2025)

Abstract: Large vision-language models (VLMs) have demonstrated remarkable abilities in understanding everyday content. However, their performance in the domain of art, particularly culturally rich art forms, remains less explored. As a pearl of human wisdom and creativity, art encapsulates complex cultural narratives and symbolism. In this paper, we offer the Pun Rebus Art Dataset, a multimodal dataset for art understanding deeply rooted in traditional Chinese culture. We focus on three primary tasks: identifying salient visual elements, matching elements with their symbolic meanings, and explanations for the conveyed messages. Our evaluation reveals that state-of-the-art VLMs struggle with these tasks, often providing biased and hallucinated explanations and showing limited improvement through in-context learning. By releasing the Pun Rebus Art Dataset, we aim to facilitate the development of VLMs that can better understand and interpret culturally specific content, promoting greater inclusiveness beyond English-based corpora. The dataset and evaluation code are available at [this link](https://github.com/zhang-tuo-pdf/Pun-Rebus-Art-Benchmark).

## introduction each culture develops its unique symbolic systems of visual elements, which are conventionally understood within that culture to convey specific meanings. for example, to viewers unfamiliar with chinese arts and linguistics, the combination of a monkey and a horse might seem nonsensical. however, in chinese culture, "a monkey lying on top of the horse" is described as a pun on "马上封 侯" (mǎ shàng fēng hóu) 1 , representing the wish for promotion. this form of wordplay is prevalent in chinese decorative arts, appearing in various art formats throughout chinese history, from the emperor's court to the commoners' kitchen, transcending boundaries of power, wealth, education, and media @xcite @xcite . as an example, in figure 1 , we demonstrate a chinese pun rebus painting with "a monkey lying on top of the horse," which indicates the wish for promotion by connecting homophonically similar chinese characters of "horse-马(mǎ)," "on top of-上(shàng)," combined to form 'mashang' also meaning 'right away', and "monkey-猴(hóu), sounding similar to 侯(hóu) for 'marquis." in this work, we propose the pun rebus art dataset, which is rooted in traditional chinese culture. we focus on chinese pun rebus art for three major reasons: 1) creating a pun rebus artwork involves combining textual meanings with corresponding visual representations, making it naturally multimodal; 2) pun rebus is prevalent in chinese art, rarely seen in other cultures such as western painting @xcite ; 3) pun rebus art remains widespread in contemporary chinese culture, demonstrating its enduring impact and lasting value to preserve cultural identity while engaging new generations. we introduce three sequential tasks that reflect the underlying chain-of-thought process of experts in decoding chinese pun rebuses. our goal is to benchmark the capability of large vision-language models (vlms) in recognizing, interpreting, and comprehending these rich and cultural-specific meanings across vision and language: 1) identifying the salient and relevant visual elements in art; 2) matching the visual elements with symbolic meaning; and 3) generating an explanation to express why an artwork convey certain messages. to the best of our knowledge, this is one of the first datasets to test ai's abilities in handling culturalspecific art expression, particularly focusing on the accurate identification and interpretation of visual signifiers within chinese pun rebus art. our results highlight the inherent challenges by both ai and non-expert humans in understanding chinese pun rebus arts compared to experts. in the visual element identification task, even the best vlm captures only about 30% of key elements, slightly outperforming non-expert humans. moreover, most vlms struggle to match the symbolic meanings associated with chinese culture, with gpt-4o achieving the highest accuracy of 42% in a 7-way multiple-choice question. in comparison, non-expert humans manage to reach a 55% accuracy in this task. finally, experts note that the explanations generated by vlms in expression understanding often involve biases and hallucinations, underscoring the current vlms' limitation in understanding chinese art and potentially other culturally specific contents. we hope that our effort in curating, releasing, and benchmarking the chinese pun rebus arts dataset will facilitate the development of vlms in understanding cross-cultural content other than english-based corpus, thereby promoting greater inclusiveness. ## general framework for pun rebus understanding a pun rebus in chinese culture leverages visual elements to indicate an underlying expression, metaphor, or meaning that is seemingly unrelated to the given image @xcite . the fundamental mechanism of pun rebuses hinges on the interplay between the imagery composed, on the one hand, and the semantic and phonetic components of the chinese logographs used to express a message, usually auspicious. specifically, the interpretation of pun rebuses relies on homophonic associations between the names of the depicted images (or their interactions) and the chinese characters (logographs) used to express the concepts that form the intended message, either partially or fully. the names of the objects in a pun rebus are often homophonically similar to, or even identical with, the cued expression, analogous to using the english string 'eye-can-sea-ewe' to express 'i can see you'. a pun rebus design is intended to initiate a cognitive translation process of "imagesound-sound-meaning," contrasting sharply with the more direct and straightforward 'text-meaning' decoding typically observed in pure verbal understanding. because the process is not only culturally but also linguistically specific, it is extremely challenging for an uninformed viewer to perceive and decipher any underlying meanings of this art form. these artworks are composed for aesthetic or attention-attracting purposes. generally, the chain of thought on understanding the pun rebus is composed of three sequential steps: ## task setups based on the characteristics of the chinese pun rebus artwork, we present three primary and progressive tasks in this paper: element identification, symbolic matching, and expression understanding. we want to highlight that the researchers are highly encouraged to explore additional applications and analyses tailored to their specific interests and needs using this dataset. in the following, we describe the details of each task and the corresponding evaluation metrics. ## experiments we evaluate the performance of various widely used vlms using the pun rebus art dataset. our evaluation is conducted under both zero-shot and five-shot settings to examine the inherent ability without fine-tuning specific to our dataset. specifically, we aim to probe the ingrained knowledge and reasoning processes of these models, exploring their potential limitations or biases in interpreting objects and concepts related to chinese culture. this is particularly pertinent to ensure the inclusiveness of vlms given that most models are predominantly trained on english-based resources, which may affect their performance on culturally specific tasks @xcite . we use the unified prompt for each task across all models, which are listed in appendix. we sample with default hyperparameters in all cases. all experiments are conducted with nvidia a100 gpus. ## conclusions in this work, we offer the pun rebus art dataset and evaluate whether state-of-the-art vlms can interpret chinese culture and artworks. our findings reveal that: 1) current vlms struggle to spot the salient visual elements in the chinese pub rebus arts, though they outperform ordinary humans; 2) due to the knowledge gap in cultural understanding, vlms face challenges in transferring the spotted elements into their underlying auspicious meaning or matching the symbolic meanings; 3) we also observe substantial limitations in vlm's ability to provide coherent explanations for interpreting chinese pun rebus arts. the responses provided by these vlms often exhibit biases towards fixed objects and include significant hallucinations; 4) in-context learning does not effectively guide vlms to improve their performance in pun rebus art understanding. in the future, a promising area of research will be developing effective data curation to incorporate more diverse and cross-cultural knowledge into the training and evaluation processes of vlms. this approach holds promise for making vlms more inclusive and universally beneficial, enhancing their ability to understand and interpret various cultures.