Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers (2025)

Abstract: This paper introduces MISS-QA, the first benchmark specifically designed to evaluate the ability of models to interpret schematic diagrams within scientific literature. MISS-QA comprises 3,000 expert-annotated examples over 983 scientific papers. In this benchmark, models are tasked with interpreting schematic diagrams that illustrate research overviews and answering corresponding information-seeking questions based on the broader context of the paper. To ensure reliable and consistent evaluation, we propose an automated evaluating protocol powered by open-source LLMs trained on human-scored data. We assess the performance of 18 frontier multimodal foundation models, including o1, Claude-3.5, Llama-3.2-Vision, and Qwen2-VL. We reveal a significant performance gap between these models and human experts on MISS-QA. Our analysis of model performance on unanswerable questions and our detailed error analysis further highlight the strengths and limitations of current models, offering key insights to enhance models in comprehending multimodal scientific literature.

## introduction when engaging with academic literature, readers frequently first consult schematic diagrams that illustrate the research framework to quickly grasp the underlying research and identify specific areas of interest. this initial visual reference is crucial as it provides an overview of the study's structure and aims, prompting readers to delve deeper into the paper content for a more comprehensive understanding. given the complexity and high volume of information in scientific papers @xcite @xcite , there is a significant demand for ai systems that can assist scientists in seeking information from paper content through schematic diagrams. foundation models have recently demonstrated remarkable capabilities in tackling various complex tasks @xcite @xcite . however, their capabilities in interpreting schematic diagrams and synthesizing information from related paper context remain largely underexplored. previous scientific document qa benchmarks primarily focus on text-only contexts @xcite , emphasizing information-seeking questions derived from paper abstracts. while recent scientific multimodal qa benchmarks have incorporated multimodal information from scientific papers @xcite @xcite , their focus is largely limited to analyzing charts or tables that present experimental results. in contrast, reasoning over schematic diagrams introduces unique challenges. it requires ## related work early document qa benchmarks @xcite @xcite ) target short documents (e.g., wikipedia pages) and singlehop reasoning. in contrast, question answering over scientific papers introduces unique challenges. these include interpreting domain-specific terminology, synthesizing information across multiple sections, and reasoning over multimodal content. however, as shown in ## the miss-qa benchmark we develop the miss-qa benchmark to evaluate foundation models' abilities to interpret schematic diagrams and integrate them into the informationseeking process over scientific papers. we formally define the task of miss-qa in the context of foundation models (fm) as follows: given a schematic diagram diag within a scientific paper, an associated question q, and the main sections of paper p, the task is to understand and interpret diag, identify the question-relevant information within p, and generate the final free-form answer a: the following subsections outline the preliminary setup for data construction, followed by a comprehensive explanation of the benchmark construction and validation processes. figure 2 presents an overview of the data construction pipeline. ## experiment this section presents the experiment setup, followed by a detailed analysis of the results. ## conclusion this paper presents miss-qa, the first benchmark specifically designed to evaluate multimodal foundation models in their ability to interpret schematic diagrams within scientific literature. we perform a comprehensive evaluation of 18 frontier multimodal foundation models, uncovering a significant performance disparity between human experts and current models. through an in-depth assessment, we identify the weaknesses of cutting-edge foundation models on miss-qa, offering valuable insights to guide future advancements in the field.