L lama V -o1: Rethinking Step-by-step Visual Reasoning in LLM s (2025)

Abstract: Step-by-step reasoning is crucial for solving complex visual tasks, yet existing approaches lack a comprehensive framework for evaluating this capability and do not emphasize step-wise problem-solving. To this end, we propose a comprehensive framework for advancing multi-step visual reasoning in large multimodal models (LMMs) through three key contributions. First, we introduce a Visual Reasoning Chain Benchmark, the most comprehensive benchmark for multi-step visual reasoning, covering eight diverse categories and over 4k reasoning steps. This enables rigorous evaluation of LMMs’ ability to reason accurately and interpretably across multiple steps. Second, we propose a fine-grained reasoning metric that evaluates correctness and logical coherence at each step, providing deeper insights beyond traditional accuracy metrics. Third, we introduce LlamaV-o1, a state-of-the-art multimodal reasoning model trained using a multi-step curriculum learning approach. LlamaV-o1 is optimized for structured, step-by-step reasoning and significantly outperforms existing open-source models. It surpasses Llava-CoT with a 3.8% absolute gain across six benchmarks, achieving an average score of 67.3 while being 5x faster during inference scaling. Our benchmark, model, and code is available at https://github.com/mbzuai-oryx/LlamaV-o1.

## introduction large language models (llms) are designed to understand and generate text, enabling them to handle a wide range of tasks such as translation @xcite @xcite @xcite , summarization @xcite @xcite , and question-answering @xcite @xcite . the integration of visual data through large multimodal models (lmms) extends capabilities further by combining text, images, or videos, allowing for more complex multimodal tasks like image captioning, visual question answering, and video analysis. to effectively solve these tasks, visual reasoning is essential for lmms to process and connect diverse information, ensuring logical coherence and sequential problem-solving. the ability to reason across multiple modalities is crucial to addressing complex real-world problems. step-by-step reasoning helps llms break down complex tasks into manageable steps, improving accuracy and interpretability by ensuring logical consistency. this approach mirrors human cognition, enabling models to track their thought processes effectively. prior works have shown that prompting or fine-tuning llms to generate stepwise rationales enhances reasoning performance @xcite @xcite . however, existing methods struggle with multimodal step-by-step reasoning (see fig. 1 ). current visual reasoning benchmarks focus mainly on final-task accuracy, overlooking the quality of intermediate reasoning steps. moreover, the absence of standardized evaluation further lim- ## visual reasoning chain benchmark to facilitate a thorough assessment of the reasoning capabilities in complex scenarios, we introduce a step-by-step visual reasoning chain benchmark (vrc-bench). vrc-bench strives to assess both the logical progression of reasoning chains and the accuracy of the final outcomes generated by lmms. vrc-bench includes a diverse range of topics, such as science, mathematics, medical knowledge, social sciences, and data interpretation, ensuring that the proposed evaluation benchmark captures diverse aspects of visual reasoning. ## step-by-step reasoning lmm our proposed approach, named llamav-o1, aims at multimodal step-by-step reasoning in lmms by combining curriculum learning with efficient inference. we train models progressively starting with simpler tasks like approach summarization and question-based captioning before advancing to detailed multi-step reasoning, thereby improving logical coherence and generalization. to optimize inference, we employ a parallel beam search strategy generating multiple reasoning paths and selecting the best one. this ensures high-quality outputs with lower computational costs, achieving constant scaling for greater efficiency compared to traditional methods. ## experiments in this section, we extensively evaluate our model performance trained with curriculum learning. we employ llama-3.2-11b-vision-instruct (meta ai, 2024) as the baseline. training is conducted on subset of pixmo, g-llava and llava-cot-100k, allowing a structured progression from basic summarization to complex multi-step reasoning. we evaluate the performance on the proposed reasoning benchmark (vrc-bench), designed for multi-step chain-of-thought evaluation in multimodal contexts. additionally, we present performance comparison on six multimodal benchmarks from llava-cot, covering visual, mathematical, and scientific visual reasoning.