M 3 S ci QA : A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models (2024)

Abstract: Existing evaluation benchmarks for foundation models in understanding scientific literature predominantly focus on single-document, text-only tasks. Such benchmarks often do not adequately represent the complexity of research workflows, which typically also involve interpreting non-textual data, such as figures and tables, and gathering information across multiple documents and related literature. To address this gap, we introduce M3SciQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models. M3Sci QA consists of 1452 expert-annotated questions spanning 70 natural language processing paper clusters, where each cluster represents a primary paper along with all its cited documents, mirroring the workflow of comprehending a single paper by requiring multi-modal and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of 18 frontier foundation models. Our results indicate that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents. Additionally, we explore the implications of these findings for the future advancement of applying foundation models in multi-modal scientific literature analysis.

## introduction in scientific research, the findings presented in a paper often serve as a foundation for further investigation. when studying research papers, researchers typically explore related and cited scholarly works to acquire additional context and insights. simultaneously, research papers are inherently multi-modal, presenting additional and often * equal contribution. recent studies have showcased foundation models' remarkable performance across a variety of tasks in scientific literature understanding, including summarization @xcite , document-based question answering @xcite @xcite , and scientific figure question answering @xcite @xcite . however, current investigations are mostly confined to a single-document or text-only setting, ignoring the multi-modal and multi-document nature of scientific research, where insights are often derived from interpreting interconnected texts, figures, and tables across multiple scholarly works. to address this gap, we introduce m3sciqa, a multi-modal, multi-document scientific question answering benchmark. this benchmark contains 1,452 expert-annotated questions spanning 70 natural language processing (nlp) paper clusters, encompassing 3,066 papers. each paper cluster comprises of an anchor paper and all its cited papers. inspired by the common workflow of comparative analysis in scientific research (as illustrated in figure 1 ), our benchmark simulates a process in which a finding, derived from a scientific image in the anchor paper, prompts further investigation into a specific referenced paper. this simulation enriches the benchmark by requiring the models to engage in cross-referencing among related documents, setting a new testbed for evaluating foundation models in scientific documents understanding and reasoning (section 2.1). we evaluate a wide spectrum of open-source and proprietary large language models (llms) and large multi-modal models (lmms). our experimental results reveal significant limitations in both open-source and proprietary lmms, particu-figure 1 : (top) the common workflow of comparative analysis in scientific research, particularly when a result, such as a figure/table in the information value paper (anchor paper) @xcite , prompts further examination of related research, such as details from dialogpt (reference paper) @xcite . (bottom) a demonstration of the workflow for constructing a locality-specific question, detail-specific question, and combined question. similarly, we observe that both open-source and proprietary llms struggle with long-range retrieval tasks, specifically with extracting and analyzing information from one or more academic documents. here, the best-performing model, command r+, achieves an accuracy score of 33.25 compared to an human expert accuracy score of 76.56. 1 these findings underscore the challenges that current models face in handling complex, multi-modal, multi-document, and domain-specific information. our main contributions are as follows: ## experiments we evaluate 18 prominent foundation models, including both open-source and proprietary lmms and llms. for each model, we select the most recent, largest, and best-performing checkpoint as of april 15th, 2024. the evaluation of the m3sciqa benchmark is structured into two distinct stages: locality-specific evaluation and detail-specific evaluation. ## conclusion in this paper, we introduced m3sciqa, a novel multi-modal, multi-document scientific qa benchmark for evaluating foundation models. m3sciqa effectively simulates real-world scientific workflow, thereby providing a more realistic and challenging environment for evaluating the capabilities of both lmms and llms. our evaluations of various opensource and proprietary models reveal a significant gap between the performance of current foundation models and human experts. this disparity reveals the current limitations of such models in interpreting multi-modal, multi-document scientific data and illustrates the need for further improvement in the domain of complex scientific reasoning. in our analysis, we examined numerous areas where llms and lmms fail to correctly solve questions, demonstrating that m3sciqa is challenging in multiple facets for both multi-modal reasoning and multi-document processing.