V i C or: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models (2024)

Abstract: In our work, we explore the synergistic capabilities of pre-trained vision-and-language models (VLMs) and large language models (LLMs) on visual commonsense reasoning (VCR) problems. We find that VLMs and LLMs-based decision pipelines are good at different kinds of VCR problems. Pre-trained VLMs exhibit strong performance for problems involving understanding the literal visual content, which we noted as visual commonsense understanding (VCU). For problems where the goal is to infer conclusions beyond image content, which we noted as visual commonsense inference (VCI), VLMs face difficulties, while LLMs, given sufficient visual evidence, can use commonsense to infer the answer well. We empirically validate this by letting LLMs classify VCR problems into these two categories and show the significant difference between VLM and LLM with image caption decision pipelines on two subproblems. Moreover, we identify a challenge with VLMs’ passive perception, which may miss crucial context information, leading to incorrect reasoning by LLMs. Based on these, we suggest a collaborative approach, named ViCor, where pre-trained LLMs serve as problem classifiers to analyze the problem category, then either use VLMs to answer the question directly or actively instruct VLMs to concentrate on and gather relevant visual elements to support potential commonsense inferences. We evaluate our framework on two VCR benchmark datasets and outperform all other methods without in-domain fine-tuning.

## introduction the problem of visual commonsense reasoning (vcr) @xcite @xcite expands upon the traditional visual question answering @xcite . vcr requires machines to understand complex visual scenes, extract crucial visual content, and utilize commonsense knowl-edge for drawing novel conclusions that go beyond the explicit information present in the image. previous methods have utilized pre-trained large language models and pre-trained or fine-tuned visionlanguage models @xcite @xcite to solve vcr problems in fewshot or fine-tuned setting. however, some open questions exist on how vlms and llms can efficiently and effectively collaborate to solve these vcr problems. firstly, what roles do llms and vlms play in solving vcr problems with their different capabilities? secondly, how do we maximize their capabilities to solve the vcr problems without in-domain finetuning? to answer these two questions, as shown in figure 1 , we first find that vlms themselves can solve the problems requiring the model to recognize various low-level visual patterns and understand high-level concepts like actions, events, and relations indicated by those visual patterns. in the meanwhile, solving problems that require the model to deduce conclusions or form explanations based on visual observation relies more on the commonsense reasoning capabilities of llms. this kind of problem requires a broad array of commonsense knowledge about the world, including cause-and-effect relationships, intentions, and mental states @xcite . to validate this finding, we first note these two kinds of vcr problems as visual commonsense understanding (vcu) and visual commonsense inference (vci). then, we instruct llms to classify vcr question into these two categories. we empirically find that, for vcu problems, vlms like blip2 can achieve better results than llm+caption pipeline @xcite with their visual understanding capabilities, and for vci problems, the llm+caption pipeline is better. in the meanwhile, we observe that image captions provided by vlms often lack crucial contextual information necessary for answering questions. gpt candidate evaluation visual clue reasoning adjust reasoning according to the caption, there is no information suggesting this, score: 0 support: the woman looks sad. contradict: the woman is playing a video game. the woman is not looking sad. adjusted score: 0, ranking: 10 / 10 global caption: a woman standing in a living room, wearing … visual itm verification support clue score: 0.01 contradict clue score: 0.47 blip2-ita blip2-ita q: what is the person in the middle standing up to do? a: a: washing dishes b: serving someone … a: washing dishes. q: what does the appearance of the grass indicate? a: a: wind b: rain c: drought d: still wind the appearance of the grass. what visual factor can help answer the question? a: swaying b: wet c: dry and yellow d: still what will be this factor to support each choice? a: swaying gpt a: wind this poses a particular challenge for commonsense inference problems, as inferences are often defeasible given additional context @xcite . to illustrate this issue, consider the example depicted in figure 1 (bottom). at first glance, it may appear that there's nothing noteworthy beyond horses on a grassy farm, leading one to select "d: still wind" as an answer. however, upon closer examination of the swaying grass, we must revise our conclusion to "a: wind." existing perception modules, including vlms, operate in a feed-forward manner and cannot adjust their perception based on a high-level understanding or inference. to address this, we propose instructing llms to intervene with vlms in cases where they are uncertain about inference, indicating a lack of sufficient visual evidence. this intervention would guide vlms to focus on specific visual factors, such as weather or emotions, to support commonsense inferences. based on these findings, we propose the vi-cor framework, which employs the following components: (1) llms functioning as problem type classifiers (vcu and vci), vlm commanders for directing vlms based on reasoning, and visual commonsense reasoners. (2) pre-trained vlms are responsible for visual recognition and understanding. communication between llms and vlms occurs through text, such as image captions. on vcr @xcite and a-okvqa @xcite , our method achieves state-of-the-art results among methods without supervised in-domain fine-tuning. ## related work visual commonsense reasoning visual commonsense reasoning (vcr) @xcite @xcite is an emerging research area that aims to endow ai models with a human-like understanding and reasoning of visual scenes. the goal is to understand high-level concepts such as events, relations, and actions and infer unobservable aspects such as intents, causal relationships, and future actions. the vcr task was introduced by @xcite . further, more datasets focused on more types of reasoning were proposed @xcite @xcite . most methods treat vcr as an image-text alignment problem, where they encode the commonsense inference and the visual input, then predict the alignment score of the image-text pair @xcite @xcite . although achieving impressive performance, the generalizability of these methods is limited by supervised training. recently, several works have leveraged large language models for visual commonsense reasoning @xcite @xcite . however, @xcite require some vlms trained on the datasets to provide visual information. @xcite use llms to decompose the main problem and use vqa models to acquire visual information. following these works, we take one step further and systematically study the strength of pre-trained vlms and the reasoning abilities of llms on visual commonsense reasoning problems. we then propose a framework to efficiently and effectively leverage the advantages of both models. ## the vicor framework to enable more effective collaboration between llms and vlms, as shown in figure 2 , we design an approach that involves a multi-step process. first, a pre-trained llm takes the initial perception result (i.e., image caption), a question-answer pair, and instructions as input to evaluate potential answer candidates. then, if the llm is not confident about its reasoning, it will select to use vlms to directly answer the question or to guide the vlms to collect target visual information for re-evaluation. ## conclusion in this work, we study the collaboration of pretrained vision-language models and large-language models on a complex problem -visual commonsense reasoning (vcr