S ce MQA : A Scientific College Entrance Level Multimodal Question Answering Benchmark (2024)

Abstract: The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level. It addresses a critical educational phase often overlooked in existing benchmarks, spanning high school to pre-college levels. SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology. It features a blend of multiple-choice and free-response formats, ensuring a comprehensive evaluation of AI models’ abilities. Additionally, our benchmark provides specific knowledge points for each problem and detailed explanations for each answer. SceMQA also uniquely presents problems with identical contexts but varied questions to facilitate a more thorough and accurate assessment of reasoning capabilities. In the experiment, we evaluate both open-source and close-source state-of-the-art Multimodal Large Language Models (MLLMs), across various experimental settings. The results show that further research and development are needed in developing more capable MLLM, as highlighted by only 50% to 60% accuracy achieved by the strongest models.

## introduction in recent years, the evolution of large language models (llms) has marked a significant milestone in artificial intelligence. initially, these models excelled in diverse natural language processing tasks @xcite @xcite @xcite , but their utility has since increasingly expanded, transforming them into incredible agents for various downstream tasks such as reasoning and planning @xcite @xcite . notably, llms have shown proficiency in tasks that typically pose significant challenges to even highly skilled humans, such as tackling intricate mathematical problems @xcite ; primary high school or college entrance college difficulty level 0% 20% 90% 100% % of problems with explained solutions text-only multimodal scienceqa scibench mmlu mmmu mathqa gsm8k scemqa (ours) figure 1 : the comparison between scemqa and other existing benchmarks. y-axis is the percentage of problems that have detailed solution explanations. most problems (over 90%) in scemqa has detailed explanations to solutions except for some straightforward problems. more comparison can be found in alongside, the rapid development of visionbased llms has garnered considerable attention within the ai community, especially with the release of platforms like openai's gpt4-v (openai, 2023) and google's gemini ultra @xcite . these models have demonstrated exceptional abilities in tasks requiring advanced reasoning and planning, often surpassing existing benchmarks and approaching human-level performance. this progress has spurred researchers to create more sophisticated and challenging benchmarks for multimodal llms (mllms), one of the most representative is the science domain, which is a long-standing focus for humans. for example, the mathvista benchmark @xcite , comprising 6,141 problems, demands a high level of visual understanding and mathematical reasoning. additionally, the massive multi-discipline multimodal understanding and reasoning benchmark (mmmu) @xcite poses college-level multimodal reasoning challenges. currently, even the most advanced models achieve only about 50% accuracy on these benchmarks. the importance of such benchmarks lies in their role as vital tools for assessing and pushing the boundaries of ai capabilities. by presenting ai models with tasks that mimic complex, real-world scenarios, benchmarks provide a clear measure of progress and highlight areas for future development. however, in the science domain, a critical observation in multimodal reasoning benchmarks is the disparity in the levels of difficulty. prior benchmarks like scienceqa @xcite primarily focused on elementary and middle-school levels, while mmmu leaps to a college-level challenge. this leaves a significant educational phase in human learning -the high school, or college entrance level -relatively unaddressed. in fact, learning progressively in difficulty levels is not only important for humans, but also can facilitate ai systems including llms via curriculum learning @xcite and progressive training @xcite . therefore, we fill this gap by introducing a novel benchmark named science college entrance level multimodal question answering (scemqa), designed for this critical educational stage, with four key subjects: mathematics, physics, chemistry, and biology. apart from the difficulty level, our benchmark also has a detailed annotation granularity. firstly, most problems (over 90%) in scemqa has detailed explanations to solutions except for some straightforward problems. besides, each problem is associated with a specific knowledge component, facilitating detailed knowledge tracing for models. moreover, scemqa uniquely features problems with the same context but different questions. this design is informed by prior research indicating that without diverse question types for each narrative context, models might resort to learning shallow heuristics or patterns rather than developing a deep, semantic understanding @xcite . this approach ensures a more comprehensive and precise evaluation of reasoning capabilities. in figure 1 , we compare the difficulty level, annotation granularity, and covered modality among existing benchmarks. ## related work multimodal question answering multimodal question answering (qa) has been a focal area in ai research. the visual question answering (vqa) benchmark @xcite , established in 2015, pioneered free-form, open-ended visual qa, necessitating intricate image comprehension and reasoning. chartqa @xcite emphasized complex reasoning about charts, merging visual and logical thought processes. visit-bench @xcite tested vision-language models across real-world tasks, ranging from simple recognition to advanced creative generation. multimodal llms in addition to notable models like gpt4-v and google gemini, various opensource multimodal llms (mllms) have emerged. minigpt-4 @xcite improved visionlanguage understanding by syncing a visual encoder with a language llm. llavar @xcite combined ocr with text-only gpt-4 for enhanced visual instruction tuning in textrich image contexts. mplug-owl @xcite proposed a modular framework for equipping llms with multimodal capabilities, focusing on image-text alignment. instructblip @xcite excelled in vision-language instruction tuning, demonstrating remarkable zero-shot performance in diverse tasks. for a more detailed summary of related studies, please refer to these surveys @xcite . science question answering various benchmarks have been developed for specific scientific subjects, including math @xcite , mathvista @xcite , chemistry @xcite , etc. more comprehensive science qa benchmarks like scienceqa @xcite , c-eval @xcite , agieval @xcite , mmmu @xcite , and scibench @xcite have recently been introduced, providing a broader scope of assessment. ## experimental examination of scemqa in this section, we evaluate the state-of-the-art mllms on scemqa by firstly reporting their answer accuracy across various settings. additionally, we conduct a detailed error analysis (section 4.3) and show an accuracy distribution across knowledge categories (section a.1), which provide significant insights to identify the current mllms' limitations and demonstrate the value of our benchmark in exploring them. we will move those im-portant experiments to the main body of our paper when we have more space upon paper acceptance. ## conclusion in this paper, we introduced scemqa, a novel multimodal question answering dataset tailored for the college entrance level, including key scientific subjects: mathematics, physics, chemistry, and biology. a standout feature of scemqa is its high annotation granularity, with over 90% problems accompanied by detailed explanations and associated with specific knowledge points. we conduct extensive experiments including accuracy comparison, error analysis, and category accuracy distribution, employing state-of-the-art mllms and highlighting the opportunities and obstacles for multimodal ai models in scientific reasoning.