R e FLAIR : Enhancing Multimodal Reasoning via Structured Reflection and Reward-Guided Learning (2025)

Abstract: Large models can achieve higher performance on complex problems through iterative self-reflection. Yet when reflection is uncontrolled, it often leads to longer outputs, higher inference cost, and an increased risk of hallucination. Existing training methods rarely address this trade off. We introduce ReFLAIR, a unified framework that teaches multimodal large models to perform structured reflection via an explicit $think re-think answer $ format and hybrid reward learning. ReFLAIR begins with supervised cold start training on the ReFLAIR-cold dataset of curated multimodal reasoning trajectories, and then trains a Reflection Quality Scorer (RQS) to quantify the utility of rethinking steps. A modified Group Relative Policy Optimization algorithm optimizes a hybrid reward that combines answer correctness, structural fidelity, reflection utility, and sample difficulty. Evaluated on challenging mathematical benchmarks including MathVista, MathVerse, MM-Math and GSM8K, ReFLAIR yields improvements up to +12.2% absolute accuracy, produces higher quality reflective traces, and reduces harmful or redundant revisions. An adaptive test time reflection scheduler further reduces inference cost by nearly 23% while maintaining or improving accuracy. These results demonstrate that structured, reward guided reflection offers a scalable pathway to more reliable and interpretable reasoning in multimodal models.

## introduction large language models (llms) have achieved remarkable success in complex reasoning tasks, aided by techniques like chain-of-thought (cot) prompting @xcite , program-aided reasoning, and reinforcement learning @xcite . these advances enable near-human or superhuman accuracy on benchmarks such as gsm8k and aime. multimodal large models (lmms) have further extended this capability to vision-language tasks, reasoning over inputs like charts, equations, and diagrams. recent surveys and benchmarks @xcite @xcite @xcite highlight both rapid progress and the continued difficulty of handling complex visual reasoning. in parallel, reflection-where models iteratively evaluate and revise their own outputs-has emerged as a promising path for improving reasoning depth and robustness @xcite @xcite @xcite . however, reflection remains underdeveloped in many state-of-the-art systems. for example, vision-r1 @xcite adopts @xmath0, while also noting the prevalence of reflective tokens in its training corpus. yet, its final model does not explicitly reinforce reflection, leaving this capability underexploited. conversely, satori @xcite introduces a more sophisticated chain-of-action-thought format with custom tokens that promote self-reflection. despite this, it lacks fine-grained control over the quality and utility of each reflective step, often generating redundant or unfocused reasoning. these limitations point to a gap: existing approaches either omit structured reflection or apply it without targeted guidance, leading to inefficiencies or hallucinations. to address these limitations, we present re-flair, a unified training framework that treats structured reflection as a first-class component of reasoning. the core idea is to scaffold multi-stage cognition with an explicit reflection structure, optimized through a combination of supervised learning and reinforcement learning. supervised training on the reflair-cold dataset provides initial reflective supervision by exposing models to diverse reasoning trajectories that cover exploratory and corrective revision. reinforcement learning then enhances this capability through a hybrid reward scheme that integrates correctness, structural adherence, and a learned reflection quality scorer (rqs). rqs is trained to estimate the value added by reflection steps using scalar annotations derived from reference trajectories, which enables more precise control over the utility of reflection. to ensure stability during optimization, the framework incorporates recent advances in preference-based reinforcement learning such as @xcite . this training paradigm encourages models to produce reflective steps that are not merely verbose but demonstrably beneficial for downstream correctness. illustrative examples of the reflection structure and generated trajectories are provided in appendix a. we evaluate reflair primarily on mathematical reasoning benchmarks. our experimental suite encompasses multimodal and textual math datasets used across the paper, and the empirical study focuses on (i) end-to-end improvements in accuracy, (ii) ablation analyses that quantify the contribution of cold-start supervised trajectories, @xmath1, and rqs supervision, (iii) transfer experiments showing that the reflection-aware training provides gains when applied to smaller textual llms, and (iv) analyses of test-time reflection scaling, where an adap-tive scheduling mechanism reduces inference cost while maintaining or improving accuracy. across these dimensions, reflair consistently produces more accurate and interpretable reasoning traces compared to baselines that lack targeted reflective supervision. our contributions are threefold. first, we introduce reflair, a training framework that explicitly scaffolds multi-stage reflection and aligns it with task utility. second, we develop a reflectionaware reward scheme that integrates correctness, format adherence, and a learned reflection quality signal within a stabilized preference-optimization procedure. third, we present comprehensive empirical evidence on mathematical reasoning benchmarks, including ablations, transfer studies to textual llms, and test-time scaling analyses, demonstrating that structured reflection yields reliable gains in both performance and interpretability. please refer to the attached source code for further details foot_0 . ## conclusion we have introduced reflair, a framework that elevates reflective reasoning to a central component of multimodal language model training. re-flair integrates three ingredients: a large-scale reflection-annotated dataset for cold-start supervision, a trainable reflection quality scorer that quantifies the utility of rethinking steps, and a hybrid reinforcement learning scheme that balances correctness, structural fidelity, and reflection utility. experiments on diverse mathematical reasoning benchmarks demonstrate consistent gains in both accuracy and interpretability, supported by ablation studies, transfer experiments to textual models, and analyses of test-time scaling. these results highlight that structured reflection, when aligned with explicit utility signals, provides a scalable path toward reasoning systems that critique, revise, and improve their own cognition.