Through the Looking Glass: Common Sense Consistency Evaluation of Weird Images (2025)

Abstract: Measuring how real images look is a complex task in artificial intelligence research. For example, an image of Albert Einstein holding a smartphone violates common-sense because modern smartphone were invented after Einstein’s death. We introduce a novel method, which we called Through the Looking Glass (TLG), to assess image common sense consistency using Large Vision-Language Models (LVLMs) and Transformer-based encoder. By leveraging LVLM to extract atomic facts from these images, we obtain a mix of accurate facts. We proceed by fine-tuning a compact attention-pooling classifier over encoded atomic facts. Our TLG has achieved a new state-of-the-art performance on the WHOOPS! and WEIRD datasets while leveraging a compact fine-tuning component.

## introduction people quickly notice something unusual in images that defy common sense, like einstein holding a smartphone. we find it odd even though each part seems normal. our brain's ability to understand normality goes beyond just identifying objects @xcite . it involves connecting visual cues with everyday knowledge. in this work, we propose a visual commonsense model that utilizes the observation that lvlms may generate contradictory facts when confronted with images defying common sense @xcite . by leveraging lvlms to extract atomic facts from these images, we obtain a mix of accurate facts and erroneous hallucinations. then we fine-tune a compact attention-pooling model over encoded atomic facts. our results indicate that using the classifier for basic facts can efficiently spot strange images. surprisingly, this method outperforms existing more complex techniques. in addition, we introduce a synthesized weird dataset, a dataset of 824 samples of normal and strange images. using this dataset, we further confirmed the performance of our model. our contributions are as follows: ## related work recently, commonsense reasoning has attracted substantial interest from the research community, spanning disciplines within nlp and cv, with numerous tasks being introduced. @xcite introduced the whoops! benchmark, comprised of purposefully commonsense-defying images created by designers using publicly available image generation tools like midjourney. they used a supervised approach based on blip-2 flan-t5 @xcite ) on multiple scales. the proposed fine-tuned model managed to outperform a random baseline, but still falls significantly short of human performance. llms are capable of producing highly fluent responses to a wide range of user prompts, but they are notorious for hallucinating and making non-factual statements. @xcite proposed selfcheckgpt, a straightforward samplingbased method that enables fact-checking of blackbox models with zero resources. to assess consistency among multiple sampled responses, selfcheckgpt utilizes several techniques, including bertscore, an automatic multiple-choice question answering generation (mqag) framework @xcite , and nli contradiction scores to detect hallucinations in the generated responses. however, the most effective method found was prompting the llm to verify if the generations are supported by the context or not. regarding multi-modal case, @xcite proposed faithscore, a reference-free and finegrained evaluation metric that measures the faithfulness of the generated free-form answers from large vision-language models. the faithscore uses multistep approach: (1) identify the descriptive content, (2) extract corresponding atomic facts from the identified sentences, and (3) the faithfulness of all atomic facts is verified according to the input image by applying visual entailment model (vem), which is able to predict whether the image semantically entails the text. analogously, nli has been used in textual mode to verify premises and hypotheses and subsequently to detect hallucinations @xcite . @xcite proposed an approach, in which lvlm is used to first generate atomic facts from images, resulting in a combination of accurate facts and erroneous hallucinations. the next step involves calculating pairwise entailment scores among these facts and aggregating these values to produce a single reality score. our approach is similar to the preceding methods, as we also utilize lvlms to extract atomic facts from the image. we then train a supervised model to learn the relationships between the derived facts. if the classifier identifies a high contradiction among atomic facts, it indicates that one of the generated atomic facts is likely a hallucination. this often occurs when the lvlms encounter an unusual image @xcite , leading to such inconsistencies in most cases. whoops! weird # of samples 204 824 # of categories 26 12 # of sub-categories --181 human baseline 92% 82.22% ## dataset this section describes the datasets we used to evaluate our methodology. ## visual commonsense evaluation method using atomic fact extraction the idea of our method dubbed tlg (through the looking glass) is inspired by factscore @xcite : we adopt the principle of atomic facts generation for trustworthiness verification for the image modality. namely, the common sense evaluation method is based on the classification of atomic facts generated by lvlms using textual encoders. the approach is depicted in figure 2 . we use lvlms to collect different atomic facts that describe different aspects of the scene in the image. to sample as many different facts as possible, we use the diverse beam search @xcite . so, given an image i and an lvlm, we sample n facts @xmath0, f 2 , . . . , f n }, where @xmath1. next, we use a frozen textual encoder to extract representations h of the generated atomic facts. each fact representation is computed as later, these scores are used to perform a weighted averaging of the set of representations for each fact into a single representation: finally, we classify the final representation by mapping it to a single common sense violation probability: ## experimental setup to run the experiments, we strictly follow the evaluation setup suggested in whoops! @xcite . thus, we evaluate several models using 5fold cross-validation in a supervised configuration. see the detailed list of checkpoints used for the main approach and baselines in appendix e. for fact generation, we set num_beams and num_beam_groups to 5, and the diversity_penalty to 1.0. regarding penalty, we find this value to be optimal for adding diversity and preserving the model's ability to follow instructions. for lvlms, with various backbone architectures, we utilized the following prompt for fact generation: "provide a brief, one-sentence descriptive fact about this image". to generate atomic facts, we used different lvlms with different sizes (from 0.5b to 13b) of the llava architecture. given the generated atomic facts, we encode them using several deberta-v3-large-based encoders. we also consider the following baselines: lvlm with the prompt, which was found to be effective in detecting weird images @xcite : "<@xmath2." linear probing resemble our approach in that it requires a small learnable component. this baseline involves learning a logistic regression classifier on the hidden representation of llavas at each layer. we consider two setups: (a) using @xmath3, and (b) @xmath4, one-sentence descriptive fact about this image" (+prompt), which was used to generate atomic facts. clip-based models were evaluated by passing images and measuring the distance from the strange and normal classes in a zero-shot setting. in addition, we fine-tuned clip in a crossvalidation setting. more details on the hyperparameters and detailed baseline results can be found in the appendix c. llm zero-shot baselines were represented by gemma-2-9b-instruct and qwen2.5-7b-instruct. as input, we passed generated atomic facts about the image and asked the model to determine whether the facts were strange or not using the following prompt: "your task is to classify a series of facts as normal or strange. the set of facts is strange if some of the facts contradict common sense. answer using 'normal' or 'strange'. do not write anything else". furthermore, we used two fine-tuned baselines based on blip2 @xcite : blip2 flant5-xl and blip2 flant5-xxl that were reported in @xcite . moreover, we conducted experiments on knowledge transfer between weird and whoops! for fine-tunable methods to explore the generalization ability to another dataset. ## results the results of our experiments on both whoops! and weird datasets are presented in as for the clip baseline, openai/clip excelled with an accuracy of 60.78% in zero-shot mode for whoops!. on the other hand, on the weird dataset, siglip outperformed other models, achieving an accuracy of 81.57% in fine-tuning mode. llm qwen2.5-7b-instruct achieved a relatively high score of 67.65% on whoops! and 66.46% on weird. however, it falls behind gemma2-9b-instruct with a score of 73.04% on whoops! and 82.92% on weird. although llms show strong performance, they require more computing resources than tlg. gpt-4o performance illustrates the complexity of the benchmarks for proprietary systems and demonstrates the performance gap between humangenerated content and proprietary systems (it should not be directly compared with other opensource methods). the results are rather surprising; gpt-4o outperforms all the methods mentioned here on the whoops! dataset @xcite . however, it lags significantly behind all the considered baselines and our method on the newly generated weird dataset. the child is vacuuming the floor 0.60 this is a photo of a child vacuuming the floor 0.12 a child vacuuming a wooden floor -0.28 the man is using a vacuum cleaner on the beach 2.38 this image features a man vacuuming the beach 1.65 the vacuum cleaner is silver -0.25 knowledge transfer to measure the knowledge transfer ability, we fine-tuned a model on one dataset and tested it on another. the results are shown in for whoops!, the linear probing baseline with image-only input on 13b vicuna backbone with weird calibration outperforms other approaches with an accuracy of 75%. however, the tlg approach with deberta-v3-large-tasksource-nli is a second best method with an accuracy of 74.02%. as for weird, tlg trained on whoops! is the best performing approach -83.05%. linear probing in image-only mode on 13b vicuna with a score of 79.61% accuracy. unlike the previous setting with weird training and whoops! testing, there is a large gap between the best performing approach and the second. this probably indicates that our approach is robust to a small training set, while linear probing requires a larger amount of data for calibration. ## conclusion in this work, we propose a straightforward yet effective approach to visual common sense recognition. our method exploits an imperfection in lvlms, causing them to generate hallucinations when presented with unrealistic or strange images. the method entails transitioning to a text modality and addressing the problem from this perspective. our three-step process involves generating atomic facts, encoding atomic facts with transformerbased text encoder, and training classifier based on attention-pooling to detect strange images. despite the shift in modality, our approach outperforms previous baselines and other supervised methods applied in the image domain, including clip-based image encoders and linear probing of lvlms. in addition, we developed a methodology to synthesize strange images. using this methodology, we created weird, a dataset consisting of 824 images that include both strange and normal visuals, which we have made openly available. surprisingly, our tlg method outperformed the proprietary gpt-4o on our newly generated weird benchmark.