Mining Contextualized Visual Associations from Images for Creativity Understanding (2025)

Abstract: Understanding another person’s creative output requires a shared language of association. However, when training vision-language models such as CLIP, we rely on web-scraped datasets containing short, predominantly literal, alt-text. In this work, we introduce a method for mining contextualized associations for salient visual elements in an image that can scale to any unlabeled dataset. Given an image, we can use these mined associations to generate high quality creative captions at increasing degrees of abstraction. With our method, we produce a new dataset of visual associations and 1.7m creative captions for the images in MSCOCO. Human evaluation confirms that these captions remain visually grounded while exhibiting recognizably increasing abstraction. Moreover, fine-tuning a visual encoder on this dataset yields meaningful improvements in zero-shot image-text retrieval in two creative domains: poetry and metaphor visualization. We release our dataset, our generation code and our models for use by the broader community.

## introduction we make sense of visual art through shared associations @xcite . studies from the cognitive sciences have shown that these associations come from our collective biological, social, cultural and environmental contexts @xcite . for example, skulls evoke death in many western viewers. consequently, creative visionlanguage tasks like art interpretation or image-topoetry generation require models that can leverage these same associations @xcite @xcite . however, while training on image-text pairs scraped from the web has yielded powerful models such as clip that are able to adapt to many downstream tasks, research has found that they often fail to achieve similar zero-shot performance in tasks where the domain is largely different from figure 1 : two images depicting trees in different settings. their alt-text makes no mention of the diverse concepts that each tree evokes. using our method, we are able to mine contextualized associations at degrees of abstraction that extend beyond literal description. prior work has improved vision-language models (vlms) by training on synthetic captions with fine-grained detail, resulting in more nuanced image understanding @xcite @xcite . this has produced meaningful performance gains in classification and cross-modal retrieval tasks in non-creative domains. in our work, we extend this effort to creative domains. we develop a method for mining contextualized visual associations for the salient elements in an unlabeled image. here, we define contextu-alized associations as concepts related to a particular visual element based on broader scene context (e.g., "celebration" for the christmas tree in figure 1 ). then, we use these mined visual associations to synthetically produce creative captions for each image at increasing degrees of abstraction, informed by hayakawa's "ladder of abstraction" from linguistics @xcite . this results in captions that remain grounded to an image while making explicit the associations that the image evokes. our data generation process is general purpose and can be arbitrarily scaled to any unlabeled corpus of images. we validate the quality of the resulting creative captions through 1) human evaluation and 2) testing the ability of a visual encoder fine-tuned on our synthetic dataset to adapt to two creative vision-language tasks: image-topoetry retrieval and linguistic metaphor-to-visual metaphor retrieval @xcite . we find that our synthetic captions reflect increasingly creative abstraction that aligns well with human judgment without introducing hallucination. moreover, fine-tuning on these captions improves zero-shot multi-modal retrieval in both of our creative vision-language tasks. in summary, the contributions of our work are: ## generating abstracted captions given an image i featuring visual elements v i , we mine a set of contextualized associations a d (v j ) for each v @xmath0, @xmath1. then, using these associations, we generate a set of captions c d (i) for each image i that reflect the specified degree of abstraction, d. in this work, we define contextualized associations as concepts that are related to the specified visual element v i based on its broader scene context. for example, in figure 1 , a tree outside evokes different associations than an indoor christmas tree in many western viewers -these associations are mediated by each tree's surroundings. we define five degrees of abstraction d, inspired by hayakawa's "ladder of abstraction" from linguistics (hayakawa, 1967): ## conclusion & future work in this work, we introduce a scalable method for mining contextualized associations for visual elements that can be applied to any corpus of unlabeled images. we use these associations to produce a new dataset of increasingly abstract creative captions for mscoco. both human judgment and automatic evaluation across three challenging imagelanguage tasks confirm the value of this method for enabling creativity understanding. in the future, we plan to extend this study beyond english and western associations -recent work has shown that, in some cases, vlms exhibit culturally specific regularities when prompted in different languages @xcite . it is our hope to leverage this to mine multicultural associations at scale.