MMS ci B ench: Benchmarking Language Models on C hinese Multimodal Scientific Problems (2025)

Abstract: Recent advances in large language models (LLMs) and vision-language models (LVLMs) have shown promise across many tasks, yet their scientific reasoning capabilities remain untested, particularly in multimodal settings. We present MMSciBench, a benchmark for evaluating mathematical and physical reasoning through text-only and text-image formats, with human-annotated difficulty levels, solutions with detailed explanations, and taxonomic mappings. Evaluation of state-of-the-art models reveals significant limitations, with even the best model achieving only 63.77% accuracy and particularly struggling with visual reasoning tasks. Our analysis exposes critical gaps in complex reasoning and visual-textual integration, establishing MMSciBench as a rigorous standard for measuring progress in multimodal scientific understanding. The code for MMSciBench is open-sourced at GitHub, and the dataset is available at Hugging Face.

## introduction scientific reasoning represents a crucial test of artificial intelligence (ai) systems' ability to understand and apply complex concepts, making it essential for developing truly intelligent models @xcite @xcite @xcite .recent advancements in llms like gpts @xcite and @xcite have significantly transformed the field of natural language processing (nlp). despite these advances, scientific reasoning remains challenging for these models, facing several key limitations: (1) lack of multimodal evaluation: while lvlms have emerged as powerful models capable of processing both images and text, existing scientific benchmarks are predominantly text-only, preventing comprehensive assessment of visual-textual reasoning abilities. (2) limited domain coverage: current scientific datasets either focus too narrowly on individual subjects or too broadly across scientific areas, failing to systematically evaluate understanding of key concepts within specific disciplines. (3) insufficient assessment granularity: existing benchmarks lack human-annotated difficulty levels and structured taxonomies of scientific concepts, making it challenging to evaluate models' performance across different complexity levels and specific knowledge domains. these limitations create an urgent need for a benchmark that can effectively evaluate both llms' and lvlms' scientific reasoning abilities while addressing these challenges. to address these challenges, we introduce mm-scibench, a benchmark focused on mathematics and physics that evaluates scientific reasoning capabilities. our benchmark makes three key contributions: (1) a comprehensive evaluation framework that combines multiple-choice questions @xcite and open-ended q&a problems, designed to test diverse reasoning skills across mathematical and physical domains. (2) a novel multimodal assessment approach incorporating both text-only and text-image formats, enabling direct comparison of models' unimodal versus multimodal reasoning capabilities. (3) a hierarchical taxonomy of scientific concepts with human-annotated difficulty levels, detailed solutions, and explanations for each problem. we conducted extensive experiments using five state-of-the-art lvlms (including both opensource and proprietary models) on the complete dataset, and two mathematics-specialized llms on text-only questions. for consistent evaluation across models, we employed gpt-4o as an automated assessor. our evaluation reveals significant limitations in current models' multimodal scientific reasoning figure 1 : the english translation of an example of a physics mcq, featuring a single-choice question, the correct answer, and a detailed explanation to aid understanding. the original chinese version is shown in fig. 11 in the appendix. ## conclusion this paper introduces mmscibench, a benchmark designed to evaluate the scientific capabilities of unimodal and multimodal language models. mm-scibench consists of a collection of high schoollevel mcqs and q&a in mathematics and physics, with a subset of the questions incorporating images. the benchmark organizes its questions into a threelevel taxonomy, ensuring comprehensive coverage of key knowledge points in both subjects. our evaluation of five advanced lvlms and two specialized math llms on mmscibench demonstrates that current models still have significant room for improvement in scientific problem-solving. the analysis highlights that the inclusion of visual elements in questions presents a substantial challenge for model performance, emphasizing the complexity of utilizing textual and visual reasoning. this work contributes to the ongoing development of robust benchmarks aimed at evaluating the evolving capabilities of language models, particularly in the domain of scientific reasoning.