WINOVIZ : Probing Visual Properties of Objects Under Different States (2024)

Abstract: Humans interpret visual aspects of objects based on contexts. For example, a banana appears brown when rotten and green when unripe. Previous studies focused on language models’ grasp of typical object properties. We introduce WINOVIZ, a text-only dataset with 1,380 examples of probing language models’ reasoning about diverse visual properties under different contexts. Our task demands pragmatic and visual knowledge reasoning. We also present multi-hop data, a more challenging version requiring multi-step reasoning chains. Experimental findings include: a) GPT-4 excels overall but struggles with multi-hop data. b) Large models perform well in pragmatic reasoning but struggle with visual knowledge reasoning. c) Vision-language models outperform language-only models.

## introduction language models (lms) face challenges in developing intuitive reasoning and acquiring knowledge from experience, similar to humans. human knowledge acquisition from the visual world is effortless but poses difficulties for lms, as such knowledge is often not explicitly described in text. overcoming these challenges requires visual grounding, connecting language and visual information for comprehension. previous studies have predominantly aimed at investigating language models in relation to object prototypical visual properties such as color, shape, and affordance, and transferring such knowledge from vision-language models @xcite @xcite . in this work, we study language models' reasoning ability on associations between objects and their visual properties across different object states. the task requires a model to reason about different states of an object where the object may exhibit different properties. a man went to grab a quick breakfast before leaving, but saw that the only remaining banana was rotten. the banana is brown. the banana is yellow. ## the winoviz task the winoviz task entails the need for a model to deduce whether objects can demonstrate prototypical behaviors in various scenarios. more precisely, when provided with a natural language sentence describing an object engaged in a particular behavior (premise sentence), the model must determine between two sentences presenting contrasting visual attributes of the object (hypothesis sentences). challenges. the winoviz task assesses a machine's reasoning ability about daily objects, focusing on their varied properties. models often struggle with visual knowledge related to common objects due to limited explicit details in training text, attributed to reporting bias @xcite . the task is challenging as it requires pragmatic reasoning and visual knowledge reasoning, involving finding intended meanings in the text and reasoning about object properties. a more challenging version, multi-hop data, requires multi-step reasoning chains. ## experiments we first describe the experimental setup used in our analysis and share experimental results. language models. we experiment with 7 language models in total ( vision-language models. we experiment with a total of 5 vision-language models (see object: banana p1: ripe p2: rotten identify properties and attributes va1: yellow va2: brown ## conclusion examining real-world object properties requires a visual understanding that language models lack. in our study, we introduced a text-only winoviz focused on question-answering tasks, comprising 1,380 examples exploring language models' reasoning capabilities across various visual properties of objects in diverse contexts. our findings revealed that large language models demonstrate effective performance overall but struggle particularly with the multi-hop version of our dataset. it became apparent that the bottleneck in our task lies in the reasoning of visual knowledge. vision-language models surpass their language-only counterparts, although image-generation approaches prove ineffective for our specific task. future endeavors will delve into how to efficiently transfer visual knowledge from images or captions.