Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? (2023)

Abstract: Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world. However, known as visual illusions, human’s perception of reality isn’t always faithful to the physical world. This raises a key question: do VLMs have the similar kind of illusions as humans do, or do they faithfully learn to represent reality? To investigate this question, we build a dataset containing five types of visual illusions and formulate four tasks to examine visual illusions in state-of-the-art VLMs. Our findings have shown that although the overall alignment is low, larger models are closer to human perception and more susceptible to visual illusions. Our dataset and initial findings will promote a better understanding of visual illusions in humans and machines and provide a stepping stone for future computational models that can better align humans and machines in perceiving and communicating about the shared visual world. The code and data are available at [github.com/vl-illusion/dataset](https://github.com/vl-illusion/dataset).

## introduction it's well established that human perceptual systems are susceptible to visual illusions, which are defined as "consistent and persistent discrepancies between a physical state of affairs and its representation in consciousness" @xcite . figure 1 shows a well-known example -the checker shadow illusion @xcite . here, a cylinder on the checker board creates a shadow on the board. human viewers are directed to look at the two squares a and b as shown in figure 1 @xcite . to most normalsighted people, they will perceive that square a is (a) (b) figure 1 : the checker shadow illusion @xcite . different from these previous works, this paper studies visual illusion from a new angle, in the context of language communication. language comprehension and language production are tightly linked to how we perceive the visual world. back to figure 1 (a), when two people are observing the figure together, due to their likely shared illusion, the expression "the darker square" will lead to the same reference of square a. but when a human communicates with a machine, will the machine also interpret "the darker square" as square a? given the rise of large vision-language models (vlm), it's important to understand whether these vlm models have a similar tendency to visual illusions, and to what degree they may align with human vision. the answers to these questions will further impact the alignment of the grounded language communication between humans and machines. to address these questions, we created a new visual illusion dataset covering five different categories from the cognitive literature. based on the dataset, we created a benchmark, grounding visual illusion in language (gvil), which consists of four subtasks: same-difference question answering (samediffqa), referential question answering (refqa), attribute question answering (attrqa), and referential localization (refloc) to assess machines' alignment with the human under visual illusions. we specifically evaluated four state-of-the-art vision-language models: unified-io @xcite , ofa @xcite , llava @xcite and instructblip @xcite . our results have shown that these four models mostly do not align with human vision illusions, especially for qa-based tasks. however, for the refloc task, these models (especially ones with larger parameters) have demonstrated an impressive alignment with humans. to our knowledge, this is the first work that takes language into consideration to study machine visual illusion. there are two main contributions of this work. first, this investigation provides an initial understanding of the alignment between human and machine visual illusions. such understanding will enable techniques for a better communicative grounding in situated language communication and help develop more reliable embodied agents in the future. second, unlike using internal representations to explain illusions, language can be used as a proxy to demonstrate whether and how machine illusions match or mis-match with the human illusion. this benchmark will not only facilitate computational models for better human agent alignment, but also provide tools for scientific understanding of visual illusion in both humans and machines. visual illusions can affect human behavior in multiple ways. research has shown that human action cannot resist visual illusions @xcite @xcite , so is language comprehension and language production. such findings catalyze inquiries regarding the capability of models to comprehend language instructions based on human perceptions and align them with human intent. ## the grounding visual illusion in language (gvil) benchmark to facilitate our investigation, we created a benchmark for evaluating machine visual illusions in the context of grounded communication. this benchmark is built upon a set of images with visual illusions. each image consists of two objects which may look different to humans but are actually identical in their pixels. this setup has two advantages. first, the definition of illusion is clear and nonambiguous, thus it is easy to measure whether the machine has a similar illusion as humans. secondly, the multi-object setup naturally supports the evaluation of language grounding, such as evaluating whether the machine can select the object an expression grounds to under the illusion (i.e., square a is what "the darker square" grounds to in figure1(a)). ## experimental setup vision-language models. to be evaluated on all of the four tasks in gvil, the model has to be equipped with the visual question-answering skill and the object localization skill simultaneously. among a few candidates, we choose two state-ofthe-art models, the unified-io @xcite and ofa @xcite , both of which are trained on a wide range of vision-language tasks, and achieve impressive performance with a strong zero-shot inference capability on unseen data. additionally, we select two recent works that adapt large language models to understand visual images: the llava @xcite and instructblip @xcite . these models are interesting to inspect as they have shown a highly conceptual understanding of the nuance in images, such as the capability of interpreting jokes, which may also be useful in interpreting illusions. for each of the afore ## results analysis from our experiments, we are interested in investigating the following research questions: we highlight several of our findings across this three questions in below. ## discussion and conclusion we introduce gvil, the first dataset facilitating a systematic evaluation of machine visual illusion via natural language. evaluating four distinct series of state-of-the-art vision-language model families across varying scales, we observe a notable alignment between these models and human perceptions during object localization in the context of illusions. interestingly, this alignment tends to strengthen with increased model size. conversely, many models face challenges in mirroring human perspectives within visual question-answering tasks. our preliminary observations underscore the need for further discussions in two directions: