Vision-Language Models Can Self-Improve Reasoning via Reflection (2025)

Abstract: Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked. To this end, we propose a simple yet effective self-training framework, R^3V , which iteratively enhances the model’s Vision-language Reasoning by Reflecting on CoT Rationales. Our framework consists of two interleaved parts: (1) iteratively bootstrapping positive and negative solutions for reasoning datasets, and (2) reflection on rationale for learning from mistakes. Specifically, we introduce the self-refine and self-select losses, enabling the model to refine flawed rationale and derive the correct answer by comparing rationale candidates. Experiments on a wide range of vision-language tasks show that R^3V consistently improves multimodal LLM reasoning, achieving a relative improvement of 23% to 60% over GPT-distilled baselines. Additionally, our approach supports self-reflection on generated solutions, further boosting performance through test-time computation. Our code is available at https://github.com/njucckevin/MM-Self-Improve.

## introduction humans often rely on intuitive chain-of-thought (cot) to perform complex reasoning @xcite . previous studies have shown that this cot capacity also emerges in large language models (llms) @xcite . through simple prompting or fine-tuning @xcite @xcite , cot enhances the reasoning performance of llms while providing insights into their decision-making process. recently, openai o1 further advances rea-figure 1 : results of qwen-vl on tabmwp, a visual mathematical reasoning dataset. qwen-vl exhibits weak zero-shot cot reasoning performance, while our r 3 v iteratively self-improves, surpassing the gptdistilled baseline by a large margin. while cot reasoning has significantly advanced llms in textual domains, extending cot to multimodal settings remains an open problem. unlike the abundant, unsupervised text-based cot in pretraining corpora @xcite , multimodal cot resources are scarce in the text-dominated internet collections @xcite , hindering the full realization of multimodal llms' (mllms) reasoning potential. recent studies show that open-sourced mllms struggle to integrate visual cues into their reasoning process, resulting in weak cot performance @xcite . consistent with our observations in figure 1 , cot prompting provides minimal gains over direct prediction @xcite and falls far behind gpt-4o. one potential solution is to construct multimodal cot annotations for post-training; however, manual annotation is prohibitively expensive and hard to scale. this raises our first research question: can mllms self-improve the reasoning capabilities through bootstrapping on cot samples? orthogonal to fine-tuning on curated cot annotations, relying solely on positive samples can lead to suboptimal policy due to insufficient exploration of reasoning paths. inspired by human thinking, another promising direction involves learning from trial-and-errors @xcite , where mistakes are not failures but key opportunities to enhance reasoning. a few multimodal approaches use corrupted prompts to create negative samples for preference learning, aiming to improve image comprehension @xcite . however, these methods fail to generate reasoning-aligned positive and negative cot solutions, making them unsuitable for complex multimodal reasoning tasks. thus, it remains unaddressed: how can mllms efficiently learn from mistakes to improve their reasoning skills? to address the above two questions, this paper proposes r 3 v, a self-training framework that enables the model to reflect on bootstrapped cot rationales, thereby strengthening its vision-language reasoning. firstly, we leverage mllm's pre-existing but weak cot ability to bootstrap both rationales and answers for a given question, enabling the collection of a large number of positive and negative solutions based on answer correctness. secondly, we introduce a reflection mechanism on negative solutions to help the model learn from mistakes. specifically, we design selfrefine and self-select losses that guide the model to correct flawed rationales and derive the correct answer by comparing rationale candidates, respectively. the above synergistic process can be repeated, with improved samples boosting mllm's reasoning and the enhanced model further improving rationale generation. additionally, through selfselect training, our model can derive the superior solution from multiple samples, further boosting performance via test-time computation. we conduct experiments across a wide range of multimodal reasoning benchmarks, including charts, geometry, commonsense, science, mathematics, etc. r 3 v progressively enhances the reasoning ability of mllms, delivering a 23%-60% relative accuracy improvement compared to gpt distillation, and consistently outperforming the strong self-training baseline, star @xcite . moreover, our test-time selection is robust and effective, consistently surpassing pass@1 and majority voting, even in ood scenarios. our main contributions are as follows: vision-language reasoning beyond the extensively studied unimodal reasoning @xcite , multimodal reasoning has recently attracted significant interest as an essential part of human intelligence @xcite . although mllms perform well on general vision-language tasks @xcite , integrating visual cues into the reasoning process poses unique challenges, especially for open-source models @xcite @xcite . several studies have explored using rationale datasets to fine-tune models and enhance multimodal reasoning capabilities. @xcite ; @xcite augmented existing mathematical datasets with rationales using gpt distillation, while @xcite enhanced performance through manually collected cot annotations. in this work, we advocate for mllms to self-improve, reducing reliance on resource-heavy rationale annotations. self-training methods self-training helps the model learn from its own generated outputs, reducing the need for labor-intensive human annotations @xcite . prior works have focused on enhancing the reasoning capacity of llm. the typical approach involves sampling multiple rationales and filtering positive and negative solutions based on the answers. the llm is then fine-tuned on the positive samples @xcite @xcite or improved using preference learning @xcite , such as dpo @xcite . recent advances have also extended self-training to agents @xcite and neural symbolic @xcite scenarios. @xcite is a recent video instruction tuning method that synthesizes question-answer pairs from labeled video datasets. in this paper, we pioneer the exploration of self-training in vision-language reasoning, investigate the failure of dpo in multimodal settings, and address these challenges with our r 3 v framework. ## methodology our self-training framework consists of two alternating components: (1) bootstrapping a large number of positive and negative cot solutions for multimodal questions (section 3.1); (2) using the above-sampled solutions to reflect on rationales and learn from mistakes (section 3.2). this iterative process turns the mllm from weak to strong. the overall framework is illustrated in figure 2 . ## experiments in our experiments, we focus on a diverse and comprehensive set of vision-language reasoning tasks to demonstrate the effectiveness of r 3 v. we begin by outlining the benchmarks (section 4.1) and experimental setup (section 4.2), followed by the main results of r 3 v on six widely used datasets (section 4.3). we also evaluated the improvements achieved by our framework in out-of-distribution (ood) scenarios (section 4.4). ## analysis this section analyzes the key factors behind the success of the r 3 v, as well as the potential challenges of self-training in multimodal reasoning tasks. ## conclusion the scarcity of multimodal cot data limits the reasoning capabilities of current mllms. in this paper, we take the first step toward enabling mllms to self-improve for better vision-language reasoning. we propose an iterative self-training framework, r 3 v, which continuously bootstraps positive and negative solutions and improves reasoning through reflection on self-generated cot rationales. meanwhile, r 3 v enables mllms to self-reflect on their generated solutions, offering new opportunities for boosting performance through test-time computation. extensive experiments and analyses demonstrate the effectiveness of our framework and the key factors behind its success.