Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models (2024)

Abstract: Vision-language models (VLMs) have recently demonstrated strong efficacy as visual assistants that can parse natural queries about the visual content and generate human-like outputs. In this work, we explore the ability of these models to demonstrate human-like reasoning based on the perceived information. To address a crucial concern regarding the extent to which their reasoning capabilities are fully consistent and grounded, we also measure the reasoning consistency of these models. We achieve this by proposing a chain-of-thought (CoT) based consistency measure. However, such an evaluation requires a benchmark that encompasses both high-level inference and detailed reasoning chains, which is costly. We tackle this challenge by proposing an LLM-Human-in-the-Loop pipeline, which notably reduces cost while simultaneously ensuring the generation of a high-quality dataset. Based on this pipeline and the existing coarse-grained annotated dataset, we build the CURE benchmark to measure both the zero-shot reasoning performance and consistency of VLMs. We evaluate existing state-of-the-art VLMs, and find that even the best-performing model is unable to demonstrate strong visual reasoning capabilities and consistency, indicating that substantial efforts are required to enable VLMs to perform visual reasoning as systematically and consistently as humans. As an early step, we propose a two-stage training framework aimed at improving both the reasoning performance and consistency of VLMs. The first stage involves employing supervised fine-tuning of VLMs using step-by-step reasoning samples automatically generated by LLMs. In the second stage, we further augment the training process by incorporating feedback provided by LLMs to produce reasoning chains that are highly consistent and grounded. We empirically highlight the effectiveness of our framework in both reasoning performance and consistency.

## introduction vision-language models (vlms) exhibit competence at generating human-like responses by leveraging multimodal instructional data and large language models (llms) @xcite @xcite . a key direction in improving such vlms is to enable grounded and consistent visual reasoning. we thus take a critical * work done during internship at sri international. look at the reasoning capability of existing vlms, measuring and improving both their performance and consistency in reasoning. for reasoning performance, we aim to measure whether vlms can derive high-level inference that extends beyond the immediately perceived information correctly. for reasoning consistency, we seek to determine the extent to which vlms can identify the underlying reasoning chains that lead to the high-level inference. previous work simplifies the evaluation of reasoning consistency by only considering coarsegrained rationales @xcite and relying on human evaluation @xcite and similarity measure @xcite , which lacks scalability and preciseness. thus, we motivate to establish a new benchmark dataset that provides annotation of the fine-grained reasoning steps to automatically measure reasoning consistency. however, collecting such a dataset is challenging due to high-cost underlying human effort and may contain inconsistencies among annotators for the reasoning chains @xcite . to address this challenge, we propose an llm-human-in-the-loop pipeline for dataset construction. several recent efforts have shown that llms can effectively follow human instructions to generate high-quality datasets @xcite @xcite . this pipeline functions by incorporating limited human assistance for providing instructions and filtering rules, enabling llms to efficiently generate high-quality datasets in a semi-automatic manner, substantially reducing annotation cost. based on an existing coarse-grained visual inference dataset sherlock @xcite , we establish a benchmark cure for chain-of-thought visual reasoning evaluation. it contains 1,622 human-verified samples of high-level visual inference and corresponding cot reasoning chains, intended for zero-shot evaluation. two examples are presented in figure 1 . particularly, the cot rea-figure 1 : besides the high-level inference about the images (e.g., the girl is turning two years old today.), cure also contains cot reasoning chains to evaluate vlms' reasoning performance and consistency. we only show 2 (of 6) candidate options for presentation. we highlight the ground truth answers. more examples are shown in figure 9 . we evaluate the state-of-the-art (sota) vlms on cure . the key conclusions from these evaluations are: (1) the model's success in complex visual inference depends on llms components, visual inputs, and instruction finetuning; (2) even the sota vlm (blip-2) falls short in comparison to human performance regarding overall visual reasoning performance. in addition, our findings indicate a lack of reasoning consistency. specifically, the reliability of intermediate reasoning steps cannot be assured, irrespective of the accuracy of the final inference (and vice versa). this suggests vlms are not always consistent in their reasoning. to enhance vlms' reasoning performance and consistency, we propose a two-stage training framework for training rationale-augmented vlms. in the first stage, vlms are trained on reasoning samples that encompass step-by-step reasoning chains, which are automatically generated by llms. however, vlms may produce inaccurate high-level inferences due to inconsistencies or hallucination in the rationales after this stage. thus, we introduce a subsequent stage that integrates feedback from llms to examine the reasoning process. this approach avoids the complex task of directly scrutinizing the high-level inferences of vlms. the results demonstrate the relative improvement in both reasoning performance and consistency is approximately 4% compared to the sota. ## related work the cot reasoning approach is first proposed for llms @xcite . we discuss related work regarding llms cot reasoning and visionlanguage pretraining in appendix b and focus on vision-language reasoning in this section. there exists a paucity of comprehensive diagnostic studies concerning vlms with the aim of quantifying their reasoning consistency, although efforts have been spent on measuring the visual reasoning performance (e.g., sherlock) @xcite and coarse-grained rationale evaluation, including multiple-choice question answering (e.g., vcr) @xcite , human evaluation of generated rationales @xcite , and similarity measure between the generated and the groundtruth rationales @xcite . some work has identified the failure of vlms to accurately answer subquestions that are components of the main problems @xcite @xcite @xcite . for instance, vlms may correctly determine the significant size of a mountain in an image but erroneously classify it as small when responding to a query such as "are the mountains small?" @xcite . in contrast to the aforementioned studies that focus on coarse-grained rationale evaluation and individual subquestions, we create reasoning chains that consist of coherent subquestions capable of supporting high-level inference. this approach allows us to precisely measure the extent to which reasoning in vlms is consistent and grounded. ## cure benchmark we present the cure dataset for measuring visual reasoning performance and consistency in vlms and the llm-human-in-the-loop pipeline we use this procedure to create cure in a semi-automatic manner. in this section, we outline the procedure to semi-automatically create cure with llms and then describe the evaluation metrics adopted to measure reasoning performance and consistency. ## approach in preliminary experiments, we find that vlms can effectively conduct high-level visual inference when provided with complete reasoning chains. thus, we propose to train a model capable of generating rationales that can potentially enhance visual reasoning performance and consistency. to this end, we propose a bifurcated training framework that is able to train a vlm to efficiently produce rationales that facilitate high-level visual inference (see figure 3 ). in the first stage, we aim to train cotblip to generate rationales that contain enough visual details and reasonable inference. to further mitigate certain issues in generated rationales (e.g., hallucination) for better high-level inferences, we introduce the second reinforcement learning from llms (ai) feedback (rlaif) stage. we select blip-2-t5 xl as our backbone model due to its strong performance on basic vision-language tasks @xcite . consequently, we refer to our rationale-generation model as cotblip. stage-1: sft. we utilize the complex reasoning samples from the llava dataset @xcite . the original 77k samples are produced by instructing gpt-4 to generate visual inference using a carefully curated set of five human-annotated captions and bounding boxes associated with images from the coco dataset @xcite . however, the generated samples consist of repetitive, dialogic expressions that might not be entirely grounded in the images. we thus perform a further post-processing step that prompts llms to generate cot reasoning chains based on the original samples, placing an emphasis on ensuring that these chains are logical, consistent, and succinct. the detailed prompt is shown in appendix f. we train cotblip on these refined samples using sft. following the sft stage, cotblip is competent in generating plausible rationales. however, the high-level inferences may be inaccurate since the produced rationales might contain inconsistent reasoning chains or contents that are not grounded in the images (hallucination). in addition, the scalability of the sft training stage is limited due to its dependence on high-quality human-annotated dense captions, which makes it difficult for this stage to leverage image-caption pairs in the wild. this can lead to lower generalizability on a broad range of visual concepts. therefore, we extend the training into a second stage, optimizing the generation of rationales using feedback from llms. specifically, we leverage llms to inspect the rea-soning process, which is more straightforward than directly scrutinizing the high-level inferences. stage-2: rlaif. in this stage, we use imagecaption pairs sourced from the wild (e.g., sbu captions @xcite ). for each image, cotblip is initially prompted to generate three cot reasoning chains, leading to high-level visual inference regarding each image. we also note that there is a noticeable variation in the quality of these generated reasoning chains, which necessitates external feedback. therefore, we use llms (gpt-3.5-turbo) to provide feedback on the reasoning chains based on the provided caption, considering three aspects: given the llm preference data, we employ conditional reinforcement learning to train cotblip due to its stability as observed in previous work @xcite @xcite . specifically, we introduce two control tokens, @xmath0. for each sample containing a set of three ranked reasoning chains, we add @xmath1. in the training time, given an appended control token, we optimize cotblip to maximize the likelihood of the associated reasoning chain. through this approach, cotblip can learn to distinguish the difference between control tokens and their respective outputs @xcite . we note that there is no requirement to perform training for a separate reward model, given that the llm is capable of fulfilling that role effectively. inference. during inference, we initially prompt cotblip to generate rationales. however, it is important to acknowledge that when dealing with cot subquestions that primarily involve basic visual perceptual problems and text-only inference based on provided visual details, the generated rationales may have limited effectiveness. thus, the rationales are used exclusively for high-level visual inference. specifically, these rationales are incorporated before the top-tier question to prompt the downstream vlms to generate the prediction. in our implementation, we opt for utilizing the original blip-2-t5 xl model to conduct predictions based on the rationales generated by cotblip. ## conclusion we create cure using an llm-human-in-the-loop pipeline and identify the deficiencies in existing vlms for reasoning performance and consistency. to tackle these challenges, we introduce a two-stage training framework consisting of supervised fine-tuning and learning from llms feedback. our method demonstrates improvement in vlms' reasoning performance and consistency.