Evaluating Vision-Language Models on Bistable Images (2024)

Abstract: Bistable images, also known as ambiguous or reversible images, present visual stimuli that can be seen in two distinct interpretations, though not simultaneously, by the observer. In this study, we conduct the most extensive examination of vision-language models using bistable images to date. We manually gathered a dataset of 29 bistable images, along with their associated labels, and subjected them to 121 different manipulations in brightness, resolution, tint, and rotation. We evaluated twelve different models in both classification and generative tasks across six model architectures. Our findings reveal that, with the exception of models from the Idefics family and LLaVA1.5-13b, there is a pronounced preference for one interpretation over another among the models, and minimal variance under image manipulations, with few exceptions on image rotations. Additionally, we compared the models’ preferences with humans, noting that the models do not exhibit the same continuity biases as humans and often diverge from human initial interpretations. We also investigated the influence of variations in prompts and the use of synonymous labels, discovering that these factors significantly affect model interpretations more than image manipulations showing a higher influence of the language priors on bistable image interpretations compared to image-text training data. All code and data is open sourced.

## introduction bistable images, also known as ambiguous or reversible images, offer unique visual stimuli that present two distinct interpretations, though a viewer cannot simultaneously perceive both @xcite . an example of this is depicted in figure 1 , which can be seen as either a rabbit or a duck. the rapid advancements in mplug-owl "the image features a black and white drawing of a duck sitting on a rock. the duck is positioned in the center of the image, with its head and neck facing the viewer. the rock is located to the left of the duck, and the background is a darker shade of gray." vision-language models (vlms) @xcite @xcite @xcite have sparked interest in testing these models against various types of visual challenges, including optical illusions. while considerable research has been done on how these models interpret geometric and color-varying optical illusions @xcite @xcite @xcite , exploration into their performance with bistable images remains sparse. motivated by this gap, this work aims to conduct a comprehensive investigation into how visionlanguage models process and interpret bistable images. we assemble the largest dataset of bistable images to date, apply a range of visual transformations, and examine the models' interpretations and their alignment with human perception. in particular, we collect 29 bistable images from diverse online sources and cognitive science literature. each image is subjected to 121 transformations affecting brightness, tint, and resolution resulting in a total of 3,509 processed images. we assessed the behaviors of twelve vision-language models across six distinct model families in both classification and generative settings. our analysis shows that, apart from a few exceptions, these models generally demonstrate a preference for one interpretation of bistable images over the other. notably, models from the idefics family @xcite and llava1 .5-13b @xcite exhibit more balanced preferences. additionally, while most model responses show little variation to image manipulations, exceptions include clip @xcite and blip2 opt6.7 @xcite , which are sensitive to such changes. to further understand the influence of training data, we considered multiple models from the same families, trained on identical datasets but using different base language models (llms). this approach revealed that even when trained on the same visual data, the models do not consistently align in their preferences, suggesting that llm priors play a major role in ambiguous image interpretation. this observation underscores that image-text interaction during training is not the sole determinant of how vision-language models perceive ambiguity, echoing earlier findings on the importance of textual signal in vlms @xcite @xcite . additionally, we explored how variations in prompts and the use of synonymous labels affect model interpretations. these textual modifications significantly influenced the models' interpretations, reinforcing the importance of llm priors on the vlm processing of bistable images. this finding contrasts with previous research on convolutional neural networks (cnns) focused on geometric optical illusions @xcite @xcite @xcite , which typically show biases consistent with human perception. the cnns studied did not utilize language model priors, highlighting a fundamental difference in how traditional vision models and vlms handle visual ambiguity. our contributions are as follows: ## discussion and limitations this original analysis of vlm behavior on bistable images has yielded some interesting preliminary results. similar to humans, vlms have preferred initial interpretations for most classical bistable images. five out of six models showed a preference for 'two faces' in rubin vase images, 'a cube seen from above' in necker cube images, and 'a duck' in duck-rabbit images. young-old woman images is the only category for which models' preferences were more neutral and mixed. we have seen minimal alignment between vlms and humans when replicating @xcite and conducting human annotations on the rest of the images. this analysis highlights that vlms are not sensitive to the same variations that heavily impact human preferences. models vary greatly in their sensitivity to bistablility. clip emerged as a model with strong, variable preferences, while llava is more neutral. clip's variability could be attributed to the contrastive pretraining, that might sensitize the model to smaller differences. moreover, the synthetic nature of bistable images renders them out of domain from most pretraining data, especially for vlms that are predominantly trained on realistic images. nevertheless, making comparisons between human and machine perception of bistable images is difficult beyond the initial biases. human perception of bistable images exhibits the phenomenon of switching interpretations through extended focus on the image. replicating the phenomenon of switching is difficult because vlms take static images at a single point in time. we loosely approximated the movement of time by testing the models on dozens of subtle variations of each image, as discussed above. under the theories that subtle bottom-up cues precipitate switching in human processing, vlms do not replicate this phenomenon. we saw that all models' preferences remained steady with variations in brightness, resolution, color, and color tint intensity. nevertheless, this was in contrast to linguistic variations, highlighting the importance of language priors in generative vlms. more research is needed to further our understanding of vlm bistable image interpretation. using vlms that process videos could be a tractable way of mimicking the passage of time. furthermore, additional interventions through design manipulations either through the employment of textto-image models or human artists could reveal additional insight on vlm behavior for bistable image inputs. ## conclusion in this study we explore the behavior of vlms on bistable images. we construct the largest bistable image dataset and evaluate 12 different models across six model families under various perturbations: pixel-color based perturbations, resolution, rotations, interpretation label synonyms, and prompt variations. we find that prompts have the highest impact on model preferences whereas, pixel-color perturbations have minimal effects. we further conduct human study comparisons, and find that vlms do not exhibit the same initial biases on bistable images as human subjects.