Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understanding (2024)

Abstract: Visual arguments, often used in advertising or social causes, rely on images to persuade viewers to do or believe something. Understanding these arguments requires selective vision: only specific visual stimuli within an image are relevant to the argument, and relevance can only be understood within the context of a broader argumentative structure. While visual arguments are readily appreciated by human audiences, we ask: are today’s AI capable of similar understanding?We present VisArgs, a dataset of 1,611 images annotated with 5,112 visual premises (with regions), 5,574 commonsense premises, and reasoning trees connecting them into structured arguments. We propose three tasks for evaluating visual argument understanding: premise localization, premise identification, and conclusion deduction.Experiments show that 1) machines struggle to capture visual cues: GPT-4-O achieved 78.5% accuracy, while humans reached 98.0%. Models also performed 19.5% worse when distinguishing between irrelevant objects within the image compared to external objects. 2) Providing relevant visual premises improved model performance significantly.

## introduction what we see depends mainly on what we look for. ## related work visual arguments are arguments built on visual medium @xcite . unlike typical images, a visual argument is intentionally organized to persuade viewers to a certain conclusion @xcite . this work builds upon to ongoing debates in the human studies literature about the nature of visual arguments @xcite . our results ( § 5) suggest that understanding visual arguments requires focusing on a subset of the visual context: not all visual cues contribute, and identifying the relevant ones is the key necessity. this task one of selective vision: the human capability to focus on behaviorally relevant stimuli. @xcite . examples of visual arguments are prevalent in advertisements @xcite @xcite , cartoons @xcite , mathematical educations @xcite , and, arguably, diagrams @xcite . @xcite also investigate arguments conveyed through images. however, our work focuses on visual argument, requiring that the argumentative content be primar-ily communicated through visual elements rather than relying solely on accompanying captions or written text. furthermore, we provide explicit annotations of the argumentation structure in a tree format, facilitating detailed, hierarchical analysis of the model's ability to comprehend visual arguments step-by-step. multimodal reasoning. recent studies have introduced various multimodal models capable of sophisticated reasoning across different modalities, such as vision and language. models such as llava @xcite , idefics2 (laurençon et al., 2024), and qwen-vl @xcite are built on pretrained large language models (e.g., llama @xcite ) and integrate vision encoders. others, including ofa @xcite and unified-io @xcite , are developed from scratch. these models excel in tasks such as localization, image captioning, and commonsense reasoning. furthermore, models such as unified-io-2 @xcite and gpt-4-o @xcite can understand audio, while others @xcite support video understanding, demonstrating broad multimodal reasoning capabilities. beyond factual visual understanding. visual comprehension is moving beyond factual understanding to include various types of writing. these include visual commonsense reasoning @xcite @xcite , humor understanding @xcite , and understanding social interaction @xcite . of particular relevance to our work is visual metaphors @xcite , which express abstract concepts with concrete visual cues. while some overlap exists in the images used, there are clear differences in intention and structure; not all metaphorical images present clear arguments and can be seen as visual arguments. conversely, not all visual arguments depend on metaphors @xcite . argument structure. an argument is typically understood as a structure that starts from a set of premises (reasons) and ends in a conclusion, often represented symbolically as a tree @xcite . while there have been extensions, including computational models of arguments @xcite @xcite , we use the basic form of trees connecting premises to conclusions, following previous literature @xcite . ## visargs dataset visargs comprises a total of 1,611 images featuring clear visual arguments. these images are categorized into 914 advertisement images and 697 cartoon images based on their sources. each image in visargs is annotated with descriptions and bounding boxes for the visual premises (vp), descriptions of the commonsense premises (cp), the conclusion, and an argumentation tree (t) detailing the reasoning path from the premises to the conclusion (c). all descriptions are in english, with an average character length of 79, 91, 142, and 105 for vp, cp, c, and t, respectively. on average, each image contains 3.17 visual premises, 3.46 commonsense premises, and 2.88 intermediate conclusions. ## conclusion we introduce visargs, a curated and annotated benchmark for visual argument understanding. using our benchmark, we affirm a compelling hypothesis: selective vision is a critical bottleneck for visual reasoning in current machines. we aim for our benchmark to serve as a resource for advancing multimodal intelligence beyond passive captioning. future work includes: