S ci V erse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMM s on Multi-modal Scientific Problems (2025)

Abstract: The rapid advancement of Large Multi-modal Models (LMMs) has enabled their application in scientific problem-solving, yet their fine-grained capabilities remain under-explored. In this paper, we introduce SciVerse, a multi-modal scientific evaluation benchmark to thoroughly assess LMMs across 5,735 test instances in five distinct versions. We aim to investigate three key dimensions of LMMs: scientific knowledge comprehension, multi-modal content interpretation, and Chain-of-Thought (CoT) reasoning. To unveil whether LMMs possess sufficient scientific expertise, we first transform each problem into three versions containing different levels of knowledge required for solving, i.e., Knowledge-free, -lite, and -rich. Then, to explore how LMMs interpret multi-modal scientific content, we annotate another two versions, i.e., Vision-rich and -only, marking more question information from texts to diagrams. Comparing the results of different versions, SciVerse systematically examines the professional knowledge stock and visual perception skills of LMMs in scientific domains. In addition, to rigorously assess CoT reasoning, we propose a new scientific CoT evaluation strategy, conducting a step-wise assessment on knowledge and logical errors in model outputs. Our extensive evaluation of different LMMs on SciVerse reveals critical limitations in their scientific proficiency and provides new insights into future developments. Project page: https://sciverse-cuhk.github.io

## introduction in recent years, the rapid advancement of large models, i.e., large language models (llms) @xcite @xcite and large multi-modal models (lmms) @xcite ; zhang * equal contributions † corresponding author @xcite @xcite , has significantly expanded the frontiers of various modalities and scenarios, such as text @xcite @xcite , 2d images @xcite @xcite , and 3d point clouds @xcite @xcite . notably, lmms have demonstrated promising potential in addressing multi-modal scientific problems across diverse domains, including physics, chemistry, and biology. despite efforts to develop scientific datasets with visual content as evaluation benchmarks @xcite @xcite , existing approaches primarily assess lmms through basic testing, where models directly solve original problems and are compared based on overall accuracy. however, we identify that effective problem-solving in this domain requires three key skills: scientific knowledge comprehension, multi-modal content interpretation, and chain-of-thought (cot) reasoning. consequently, the fine-grained scientific capabilities of lmms remain insufficiently explored, lacking a detailed and thorough examination within the research community. in this paper, we introduce sciverse, a comprehensive evaluation benchmark to assess lmms on multi-modal scientific problems. our curated dataset comprises 1,147 meticulously collected problems and 5,735 newly annotated test instances across five distinct versions, covering difficulty levels from high school to college. specifically, to investigate the three key skills aforementioned, we aim to explore the following questions regarding scientific problem-solving as outlined in figure 1 . ## sciverse in section 2.1, we first present an overview of sci-verse, including dataset statistics and the collection process. then, we respectively illustrate our methodology on the three critical aspects of assessing lmms: scientific knowledge comprehension (section 2.2), multi-modal content interpretation (section 2.3), and chain-of-thought (cot) reasoning evaluation (section 2.4). ## experiment in section 3.1, we first introduce our experimental settings, including the evaluation lmms and implementation details. then, in section 3.2, we provide the performance comparison and insightful analysis on sciverse. ## conclusion in this paper, we introduce sciverse, a comprehensive multi-modal benchmark designed to evaluate the fine-grained capabilities of lmms in scientific problem-solving. by transforming problems into multiple versions that vary in knowledge and modality, we investigate three critical dimensions of lmms: scientific knowledge comprehension, multi-modal content interpretation, and cot reasoning. furthermore, our proposed scientific cot evaluation strategy provides a deeper understanding of how lmms handle knowledge and logical errors during problem-solving. the findings from our extensive evaluation of current state-ofthe-art lmms underscore the need for further advancements in their scientific proficiency and multimodal reasoning capabilities. moving forward, we hope sciverse may serve as a foundation for future developments of lmms in scientific fields.