VCD : A Dataset for Visual Commonsense Discovery in Images (2025)

Abstract: Visual commonsense plays a vital role in understanding and reasoning about the visual world. While commonsense knowledge bases like ConceptNet provide structured collections of general facts, they lack visually grounded representations. Scene graph datasets like Visual Genome, though rich in object-level descriptions, primarily focus on directly observable information and lack systematic categorization of commonsense knowledge. We present Visual Commonsense Dataset (VCD), a large-scale dataset containing over 100,000 images and 14 million object-commonsense pairs that bridges this gap. VCD introduces a novel three-level taxonomy for visual commonsense, integrating both Seen (directly observable) and Unseen (inferrable) commonsense across Property, Action, and Space aspects. Each commonsense is represented as a triple where the head entity is grounded to object bounding boxes in images, enabling scene-dependent and object-specific visual commonsense representation. To demonstrate VCD’s utility, we develop VCM, a generative model that combines a vision-language model with instruction tuning to discover diverse visual commonsense from images. Extensive evaluations demonstrate both the high quality of VCD and its value as a resource for advancing visually grounded commonsense understanding and reasoning. Our dataset and code will be released on https://github.com/NUSTM/VCD.

## introduction commonsense, comprising facts and principles humans rely on in daily life, is essential for decisionmaking and behavior. integrating it into ai systems enhances human-like reasoning, improves interpretability, and has become a growing area of research. visual commonsense, a portion of commonsense, refers to general knowledge about the * corresponding author. visual world. while existing commonsense knowledge bases, e.g, conceptnet @xcite , include some visual commonsense represented in textual form, they lack visually grounded commonsense-specific, contextually rich knowledge tied to actual visual scenes. such limitation results in restricted coverage and insufficient detail for effectively bridging vision and language understanding. cognitive science research @xcite indicates that humans perceive the world by focusing on objects in a scene, noting their attributes, spatial relationships, and actions to gather multidimensional information. this information forms the basis of visual commonsense, which is inherently scene-dependent and object-specific. on the other hand, scene graph datasets in the field of computer vision, e.g, visual genome (vg) @xcite , although provide rich objectlevel descriptions of attributes, actions, and relationships, typically lack a systematic categorization of commonsense. moreover, they predominantly focus on commonsense directly observable in images (referred to as seen commonsense in this paper), while neglecting commonsense not visually apparent but still relevant to the image and can be inferred by general world knowledge (referred to as unseen commonsense). for example, in fig. 1 , given a scene depicting "a man skateboarding on a busy street", humans can naturally infer unseen commonsense like "the man might be hit by a car". such unseen commonsense is crucial for deep visual understanding and reasoning, but has received insufficient attention in current research. to address these challenges, we present vcd, a large-scale visual commonsense dataset by integrating and linking visual genome and concept-net. vcd includes over 100,000 images with more than 14 million object-commonsense pairs, where each image is annotated with objects it contains, and each object is further annotated with its related visual commonsense triples. similar to concept-man car , hasproperty, yellow) , locatednear, streetlight) , relatedness, on road) , capableof, carry people) , receivesaction, driven) , createdby, factory) , locatednear, parking lot) , receivesaction, sold) , usedfor, travel) net, each commonsense is represented as a (head, relation, tail) triple; but the head here is a languagevision pair, consisting of an entity and its corresponding bounding box in the current image. by grounding commonsense entities to the bounding boxes in the image, vcd bridges the gap between linguistic knowledge and visual information. we introduce a three-layer taxonomy to categorizing these visual commonsense triples. first, we identify visually relevant categories from the 34 basic knowledge types defined in conceptnet as the foundational layer. these categories are then grouped into three fundamental aspects widely studied in computer vision (i.e, property, action, and space), constituting the second layer. for the top layer, we distinguish commonsense knowledge based on its visual observability in the given image (i.e, seen commonsense versus unseen commonsense). this hierarchical taxonomy provides a comprehensive framework for organizing visual commonsense knowledge, bridging nlp and cv domains while enabling analysis of both observable and inferential visual relationships. vcd captures rich patterns and relationships that reflect the visual world, enabling the discovery of scene-dependent and object-specific visual commonsense. in this regard, we train a generative model, vcm, that integrates a vision-language model with instruction tuning, to generate visual commonsense from images. the instructions cover diverse types of commonsense within the taxonomy, enabling vcm to generate different categories of commonsense triples according to the provided instruction, spanning both seen and unseen visual commonsense across the property, action, and space aspects. extensive evaluations, including both automatic and human evaluations, demonstrate 1) the high quality of the vcd dataset, 2) the strong performance in visual commonsense discovery, particularly surpassing gpt-4o in identifying unseen commonsense, and 3) the enhancement of downstream vision-language tasks through the discovered visual commonsense knowledge. these comprehensive evaluations demonstrate vcd's value as a foundational resource for discovering and leveraging visual commonsense, advancing visually-grounded commonsense ai. ## related work commonsense in text has been a longstanding research focus, with early studies primarily dedicated to constructing commonsense knowledge bases. conceptnet @xcite integrates multiple knowledge bases. @xcite captures selectional preference knowledge extracted from over 11 billion tokens of unstructured text. transomcs @xcite employs linguistic graphs to align aser with conceptnet. dis-cos @xcite enhances commonsense of aser by aggregating information from neighboring concepts. atomic @xcite @xcite is a collection of if-then knowledge triplets centered on daily events. research on visual commonsense has evolved from focusing on the specific commonsense category to broader, more generalized commonsense categories. early work focused on different specific dimensions of images, including taxonomy @xcite , unary affordance @xcite , physical properties @xcite , and spatial relationships @xcite @xcite . more recent studies have expanded beyond these individual dimensions to explore generalized visual commonsense knowledge @xcite @xcite @xcite @xcite . a crucial aspect of visual commonsense is the generation of scene graphs, which models object interactions within an image to support high-level reasoning @xcite . another important research direction involves multimodal knowledge graphs @xcite @xcite @xcite , which extend traditional knowledge graphs by associating entities with non-textual data, such as images. however, no existing multimodal knowledge graphs are explicitly designed to capture visual commonsense. we distinguish visual recognition from visual commonsense, aligning the latter's "seen" aspects with vi-cor's @xcite visual commonsense understanding (vcu). while visual recognition identifies objects and attributes (e.g., a "man," "thin"), vcu, or "seen" commonsense, provides an explicit, structured understanding of this literal visual content, such as "(man, hasproperty, thin)" or "person washing dishes". therefore, visual commonsense leverages visual recognition to build a structured, queryable layer of knowledge about directly observable elements and their explicit relationships within a scene, forming a foundational step towards deeper reasoning. ## visual commonsense discovery and its evaluation vcd provides a comprehensive foundation for discovering visual commonsense from images, capturing rich patterns and relationships that enable the application of such knowledge to enhance downstream vl tasks. leveraging this dataset, we train vcm, a visual commonsense discovery model that combines a generative vl architecture with instruction tuning. this allows vcm to generate visual commonsense from novel images. the generated visual commonsense can be used to improve the performance of downstream vl applications. foot_1 ## conclusion we introduced vcd, a large-scale visual commonsense dataset that bridges the gap between linguistic and visual commonsense. by combining structured relations from conceptnet with objectlevel annotations from visual genome, vcd provides both seen and unseen commonsense across three aspects: property, action, and space. this hierarchical taxonomy enables fine-grained, scenedependent, and object-specific commonsense representation. to demonstrate its utility, we developed vcm, a generative model trained with instruction tuning. the model exhibits a strong ability in discovering various types of visual commonsense, and improves performance on vision-language tasks like visual question answering. vcd provides a foundation for enhancing visually grounded commonsense understanding and reasoning, enabling ai systems to better capture visual commonsense knowledge for real-world applications.