Enhancing Advanced Visual Reasoning Ability of Large Language Models (2024)

Abstract: Recent advancements in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models’ advanced reasoning ability. Traditional Vision-Language models (VLMs) perform well in visual perception tasks while struggling with complex reasoning scenarios. Conversely, Large Language Models (LLMs) demonstrate robust text reasoning capabilities; however, they lack visual acuity. To bridge this gap, we propose **C**omplex **V**isual **R**easoning **L**arge **L**anguage **M**odels (**CVR-LLM**), capitalizing on VLMs’ visual perception proficiency and LLMs’ extensive reasoning capability. Unlike recent multimodal large language models (MLLMs) that require a projection layer, our approach transforms images into detailed, context-aware descriptions using an iterative self-refinement loop and leverages LLMs’ text knowledge for accurate predictions without extra training. We also introduce a novel multi-modal in-context learning (ICL) methodology to enhance LLMs’ contextual understanding and reasoning. Additionally, we introduce Chain-of-Comparison (CoC), a step-by-step comparison technique enabling contrasting various aspects of predictions. Our CVR-LLM presents the first comprehensive study across a wide array of complex visual reasoning tasks and achieves SOTA performance among all.

## introduction the concept of complex visual reasoning was introduced with visual commonsense reasoning (vcr) dataset @xcite in 2019, which tests models' ability to understand visual content as well as commonsense cognition. however, the development in this field has remained relatively subdued, primarily due to vision-language models' (vlms) limitations in incorporating commonsense knowledge @xcite . recent years have seen significant advancements in complex linguistic reasoning tasks @xcite due to the emerging gpt3 @xcite , llama @xcite , and vicuna @xcite . this leap forward has triggered a renewed interest in the complex visual reasoning area, exploring how visual perception can enhance linguistic inference and potentially overcome previous hurdles @xcite . it has led to innovative benchmarks focusing on various aspects: commonsense reasoning -winogavil @xcite , compositionality -winoground @xcite , weird image explanation -whoops @xcite , and humor understanding -nyccc @xcite . these tasks demand models not only accurately interpret image content, but also integrate knowledge from daily experiences, general commonsense, cultural context, and humor sense. for example, a synthetic image, as shown in whoop's example in figure 1 of "the portrait of the mona lisa depicts a stern male face." contradicts the cultural context, as the famous painting mona lisa depicts a female face. in this paper, we introduce a novel method named complex visual reasoning large language models (cvr-llm), based on the "vlms + llms" concept. recent multimodal large language models (mllms) like llava @xcite and minigpt4 @xcite have proven effective in many vl tasks. however, these models are resource-intensive, relying on millions of image-text pairs for projection layer learning. to overcome this limitation, our approach leverages the visual perception strengths of vlms to translate images into context-aware image descriptions (caid) via an inference-only, dual-loop self-refinement process that incorporates feedback from llms. these detailed descriptions enhance the llms' inference process, transforming multi-modal tasks into simpler single-modal challenges and streamlining the overall process. in addition, we develop a unique multi-modal in-figure 1 : five distinct examples from diverse datasets in the complex visual reasoning field @xcite challenge ai models' ability of complex reasoning in different aspects such as general commonsense. our research stands as the pioneering study to explore such a broad array of benchmarks (winogavil, winoground, whoops, vcr, and nyccc), proposing a paradigm centred on the "vlm+llm" concept for addressing complex visual reasoning tasks. experimental results show that cvr-llm achieves sota performance across all five tasks. further ablation studies and comparative analyses reveal the effectiveness of each module and the superiority of our method over previous approaches. particularly in comparative analysis, we introduce the chain-of-comparison (coc) technique, inspired by "chain-of-thought" and utilizing gpt4 @xcite , to address the limitations of conventional metrics in evaluating abstract concepts. coc provides a nuanced analysis by systematically dissecting and quantitatively contrasting various facets of the results for a comprehensive evaluation. our contributions are summarized as follows: (1) we present the first comprehensive study across all complex visual reasoning tasks, including wino-gavil, winoground, whoops, vcr, and nyccc. ## methods in this section, we introduce the cvr-llm framework, highlighting its innovative process for generating context-aware image descriptions (caid) as well as its complex visual reasoning in-context learning (cvr-icl) strategy. initially, we explain the caid generation process, which differs from traditional image captioning by using a selfrefinement loop with feedback from large language models (llms) to produce accurate and contextually relevant descriptions (section 3.1). subsequently, we present the cvr-icl approach (section 3.2), which enhances llms' contextual understanding and reasoning by assessing relevant cases and selecting suitable complex multi-modal demonstrations. ## qualitative results to showcase the capabilities of our approach, we present qualitative results in figure 9 . it illustrates how llms leverage contextual information to ask more relevant and insightful questions tailored the specific tasks. for instance, when provided with an image of the chess piece, the llms might ask "what does the chess piece look like?". subsequently, the captioner model generates contextually appropriate descriptions, such as "a chess piece that looks like a unicorn.". this synergy enhances the llm's decision-making process, making it more precise and context-aware. more detailed qualitative results with corresponding prompts and cvr-icl examples are illustrated in appendix a.1 and appendix a.2. ## conclusion in this work, we propose cvr-llm, an innovative approach for complex visual reasoning tasks. this method boosts llms' understanding of visual content for complex reasoning via context-aware image descriptions. we also develop a multi-modal in-context learning technique, enhancing llms' reasoning skills at both image and text levels. experimental results show that cvr-llm sets new benchmarks across multiple complex visual reasoning tasks. we also introduce a nuanced gpt4 based analysis technique chain-of-comparison to automatically break down and contrast among various aspects of generated results.