Towards Multi-Document Question Answering in Scientific Literature: Pipeline, Dataset, and Evaluation (2025)

Abstract: Question-Answering (QA) systems are vital for rapidly accessing and comprehending information in academic literature.However, some academic questions require synthesizing information across multiple documents. While several prior resources consider multi-document QA, they often do not strictly enforce cross-document synthesis or exploit the explicit inter-paper structure that links sources.To address this, we introduce a pipeline methodology for constructing a Multi-Document Academic QA (MDA-QA) dataset. By both detecting communities based on citation networks and leveraging Large Language Models (LLMs), we were able to form thematically coherent communities and generate QA pairs related to multi-document content automatically.We further develop an automated filtering mechanism to ensure multi-document dependence.Our resulting dataset consists of 6,804 QA pairs and serves as a benchmark for evaluating multi-document retrieval and QA systems.Our experimental results highlight that standard lexical and embedding-based retrieval methods struggle to locate all relevant documents, indicating a persistent gap in multi-document reasoning. We release our dataset and source code for the community.

## introduction in academic research, question-answering (qa) systems are widely used to help researchers and students quickly access and understand complex document information @xcite . however, as scientific literature expands, extracting key content efficiently and accurately from a vast library and providing relevant answers has become a significant challenge @xcite . most existing qa datasets focus on a single document or use multi-hop reasoning @xcite @xcite . example 1: question: how does point supervision strategy differ between infrared small target detection and crowd counting tasks in terms of network design and performance? answer: in infrared target detection, point supervision uses single points per target and evolves them into masks through label evolution, achieving 70% iou of fully supervised methods. for crowd counting, point supervision uses multiple points (typically 5-20) per image and employs multiscale fusion networks to handle varying crowd densities, reaching 95% performance of fully supervised approaches. the key difference lies in the supervision density -infrared targets require minimal points due to target simplicity, while crowd counting needs denser point annotations to capture crowd distribution variations. sources: [2304.01484], @xcite example 2: question: what are the different approaches to achieving differential privacy in language models and their relative trade-offs? answer: there are three main approaches: 1) local dp that anonymizes each text sample individually but requires high noise leading to incoherent language, 2) global dp that trains language models privately on the full dataset allowing better utility but needs careful privacy accounting, and 3) selective dp that protects only sensitive portions allowing better utility-privacy trade-offs. the choice depends on trust assumptions and utility requirements. although earlier multi-hop qa tasks involve several documents, they primarily rely on logical or deductive chains of reasoning. in the end, the final answer to the question typically can be found in a single document @xcite . beyond single-document and multi-hop settings, several datasets explore multi-document qa @xcite @xcite . however, these methods do not strictly require synthesizing evidence across sources. @xcite leverages structural sig-dataset qa generation size source structure multiple docs pubmedqa @xcite human experts 1k title/abstract qasper @xcite human experts 5k title/abstract qasa @xcite human experts 1.8k full-text + figs & tabs scidqa † @xcite llms + human experts 2.9k llms + hybrid filter 6.8k to address this issue, we design a pipeline for generating specialized qa that relies on multiple documents and propose mda-qa, a corresponding academic qa dataset in multi-document scenarios. the core idea is to automatically generate qa pairs from small groups of structurally connected articles (e.g., through citation links), ensuring that each qa pair draws on multiple sources. specifically, we propose using community detection on citation graphs to identify groups of articles, then leveraging a large language model (llm) to create these complex qa pairs. finally, we design an automated quality control process to ensure that the qa requires content from multiple documents for a complete response. we use the open-access spiqa dataset (pramanick et al., 2025), expanding it with structural information to produce multi-document qa pairs. we conduct a community division on over 25,000 open-access machine learning conference papers and generate 6,804 filtered qa pairs, covering over 3,000 small-scale groups of academic papers. while our qa pairs undergo automatic multilevel filtering, we also perform a structured manual check on 30 randomly sampled questions. three domain experts review each question, confirming that at least two referenced articles are required. this additional validation step ensures that mda-qa reflects multi-document needs rather than incidental single-source occurrences. to further verify the capability of existing qa systems on our constructed multi-document dataset, we establish a multi-document retrieval task and evaluate various retrieval methods, including bm25 @xcite , col-bert @xcite , bge @xcite , and mpnet @xcite , in chunk retrieval and document retrieval scenarios. our experiments (see the main contributions of this research are threefold: ## proposed method in this section, we present our pipeline for generating qa pairs that draws on multiple sources from the scientific literature. as shown in figure 1 , this process includes three key modules: community building (section 3.1), qa generation (section 3.2), and automated quality assurance (section 3.3). ## dataset generation and analysis in this section, we detail our mda-qa dataset generation with the pipeline in section 3. then, we introduce the statistics of mda-qa. ## conclusion this study proposes a pipeline for constructing datasets tailored to multi-document academic qa tasks based on the spiqa original data and citation network structure. we generate mda-qa of 6,804 high-quality multi-document qa pairs and conduct initial retrieval experiments on it. the result indicates that in multi-document scenarios, straightforward keyword matching or semantic embedding methods remain insufficient for capturing complementary information and latent relationships across documents. future work may explore graph-based methods, contextual reasoning, and knowledge integration for more effective document merging at the retrieval stage, and introduce advanced multidocument reasoning models in the qa stage, addressing both complex academic inquiries and realworld industrial use cases.