Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Study (2025)

Abstract: Recently, inference-time scaling of chain-of-thought (CoT) has been demonstrated as a promising approach for addressing multi-modal reasoning tasks.While existing studies have predominantly centered on text-based thinking, the integration of both visual and textual modalities within the reasoning process remains unexplored.In this study, we pioneer the exploration of inference-time scaling with multi-modal thought, aiming to bridge this gap.To provide a comprehensive analysis, we systematically investigate popular sampling-based and tree search-based inference-time scaling methods on 10 challenging tasks spanning various domains.Besides, we uniformly adopt a consistency-enhanced verifier to ensure effective guidance for both methods across different thought paradigms.Results show that multi-modal thought promotes better performance against conventional text-only thought, and blending the two types of thought fosters more diverse thinking.Despite these advantages, multi-modal thoughts necessitate higher token consumption for processing richer visual inputs, which raises concerns in practical applications.We hope that our findings on the merits and drawbacks of this research line will inspire future works in the field. The code will be released upon acceptance.

## introduction the remarkable performance of openai's o1 in tackling reasoning tasks has sparked widespread research interest in studying inference-time scaling across various communities @xcite . this technique is built upon the success of chainof-thought (cot, @xcite , which enables large language models (llms) to solve a problem through step-by-step reasoning. it makes use of assuming that ∠abd is equal to ∠acd, by the asa criterion, ∠1 and ∠2 are congruent, which implies ∠@xmath0. ## preliminaries in this section, we firstly establish the formulation for both text-only and multi-modal thought ( §2.1), and then provide a detailed explanation of two inference-time scaling paradigms ( §2.2): samplingbased and tree search-based methods. in addition, we present a prompting-based verifier equipped with a consistency-enhanced mechanism to effectively guide these inference-time scaling methods ( §2.3). ## discussion the necessity of multi-modal thought. in this study, we empirically demonstrate that multi-modal thought possesses unique potential in representing cognitive processes that transcend the limitations of text alone. we argue that this approach can convey richer cognitive features, a perspective that aligns with the opinions from @xcite . a representative example, illustrated in figure 1 , involves geometric reasoning. here, visual-spatial elements (e.g., the use of auxiliary lines) provide critical insights that are inherently difficult to articulate through textual descriptions, thereby enhancing conceptual clarity. by integrating such multimodal thought, the policy can engage in more informed decision-making processes, systematically improving problem-solving performance. reducing token costs for multi-modal thought to achieve efficient inference. our analysis of token computations (figure 4 ) demonstrates that multi-modal thought, while effective, incurs substantial token costs. this underscores the need for efficient compression techniques specifically designed for processing visual information in lvlms. while prior works @xcite @xcite have explored visual token compression, they have largely overlooked the unique challenges of multi-modal thought reasoning. thus developing methods for efficient visual information handling in this context becomes a key direction for future research. constructing stronger multi-modal verifiers for better performance. our error analysis ( §3.4) highlights the critical role of the verifier in inference-time scaling, where its effectiveness directly influences error reduction and performance gains. however, research on multi-modal verifiers @xcite @xcite remains in its early stages, limiting the potential of multi-modal thought. advancing more robust and accurate verifiers is therefore crucial for fully unlocking the benefits of inference-time scaling. exploring inference-time scaling for the chain of any-modal thought reasoning. recent works have extended cot reasoning beyond text and images to more modalities, such as audio @xcite and video @xcite . however, the effectiveness of inference-time scaling for these modalities remains unexplored. while our findings confirm the benefits of scaling for text-image reasoning, whether these advantages extend to other modality combinations remains an open question. ## conclusion in this work, we present a systematic investigation of inference-time scaling with multi-modal thought. we conduct comprehensive experiments across 10 challenging tasks spanning diverse domains, investigating both sampling-based and tree search-based scaling methods. we demonstrate that multi-modal thought consistently outperform text-only reasoning, achieving progressively stronger performance with increased computational budgets. this empirical evidence positions multi-modal thought scaling as a promising direction for enhancing complex reasoning capabilities in multi-modal scenarios. however, we notice that performance gains are accompanied by higher computational costs and depend heavily on the effectiveness of the verifier, thus hindering real-world deployment. we hope this work will serve as a springboard for studying inference-time scaling with multi-modal thought, thereby advancing multi-modal reasoning.