S ci DQA : A Deep Reading Comprehension Dataset over Scientific Papers (2024)

Abstract: Scientific literature is typically dense, requiring significant background knowledge and deep comprehension for effective engagement. We introduce SciDQA, a new dataset for reading comprehension that challenges language models to deeply understand scientific articles, consisting of 2,937 QA pairs. Unlike other scientific QA datasets, SciDQA sources questions from peer reviews by domain experts and answers by paper authors, ensuring a thorough examination of the literature. We enhance the dataset’s quality through a process that carefully decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering. Questions in SciDQA necessitate reasoning across figures, tables, equations, appendices, and supplementary materials, and require multi-document reasoning. We evaluate several open-source and proprietary LLMs across various configurations to explore their capabilities in generating relevant and factual responses, as opposed to simple review memorization. Our comprehensive evaluation, based on metrics for surface-level and semantic similarity, highlights notable performance discrepancies. SciDQA represents a rigorously curated, naturally derived scientific QA dataset, designed to facilitate research on complex reasoning within the domain of question answering for scientific texts.

## introduction question-answering (qa) datasets are valuable for evaluating the reading comprehension, reasoning, and document understanding capabilities of language models @xcite @xcite . the scientific qa task involves reading a research paper and answering questions, drawing on the paper content and some background knowledge. this task mirrors how humans engage with academic literature @xcite . scientific literature is inherently dense and typically requires a deep understanding and significant background knowledge to fully comprehend and engage with. to address this challenge, the nlp community has developed various datasets for question-answering (qa) from research papers to aid in development and evaluation of ai systems for comprehending the research papers. methods range from manual question generation by domain experts @xcite @xcite to automated extraction of questions using machine learning from selected texts @xcite @xcite @xcite . however, many of these datasets focus on surface-level information and are often limited to questions that are written from titles and abstracts, which restricts the complexity and deeper engagement with the full papers. we introduce scidqa, a novel deep reading comprehension dataset for scientific papers. it is specifically tailored to the scientific articles in the machine learning (ml) domain and sourced from peer reviews on the openreview platform @xcite . peer reviews frequently include questions or comments from reviewers who seek information or clarification on aspects they are confused about or do not fully understand. answering many of such questions necessitate a deep and com- extract discussions containing qa pairs collect reviewer-author discussion and manuscript revisions table1 table2 [5] [8] [9] however, not all such questions expressed in a review are useful. in addition they also need rewriting to stand alone as clear, self-contained queries suitable for a reading comprehension dataset. to ensure the quality and relevance of our dataset, we implement a human annotation process by domain experts, highlighted in figure 2 . our dataset features long-form questions and answer pairs, as shown in ## building the scidqa dataset we present the pipeline for the collection of the scidqa dataset and the preprocessing, manual filtering and rewriting steps involved. a schematic denoting the pipeline is presented in figure 2 . we present the various stages of data curation next. ## dataset details and analysis the scidqa dataset comprises 2,937 questionanswer pairs. we present the statistics of scidqa in comparison to other related existing qa datasets in diversity of answer sources our dataset features questions necessitating reasoning across multiple modalities beyond mere text, including figures, information source % in dataset tables 14.03% multiple documents 10.9% appendix and supplementary 10.01% equations and symbols 10.32% figures 6.98% the statistics are presented in ## experimental setup we design four task configurations to evaluate the capabilities of llms in answering the questions in scidqa. we use two separate setups, closed-book @xcite , and open-book. we experiment with a wide-range of open-source llms (falcon @xcite , galactica (taylor et al., 2022), gemma @xcite , llama 2 @xcite , llama 3.1 @xcite , phi-2 (javaheripi et al., 2023), qwen v2.5 @xcite , vicuna @xcite , and zephyr @xcite ) and two frontier closed models gemini pro @xcite and gpt-4 @xcite models. for open-source models, we experiment with various model sizes from ∼2b to ∼70b parameters. priming with the question only (closed-book) can llms answer the questions in a closed-book setting @xcite when primed only with the question text and without explicitly providing the paper? llms have the ability to retain knowledge and in this sense, it's conceivable that llms might be able to generate answers directly based solely on the question text, without any context from the associated research papers. 6 further, llms might already have internalized the knowledge related to papers to be able to answer some specific questions without explicitly providing the context. to investigate this possibility, in the closed-book configuration, llms are presented with only the questions and instructions, without any information about the relevant paper. priming with question, and paper's title and abstract (title-abs) in this setting, we provide the llm with the question text, along with the title/abstract of the paper. this mimicks a "partially" closed-book setting. the objective is to ascertain whether the inclusion of limited additional information, such as the paper's title and abstract, enhances the llm's ability to accurately retrieve and recall the knowledge to correctly answer the question. unlike the fully closed-book setting, it is not entirely infeasible to answer some questions with the information provided in the abstract. however, given that our dataset comprises questions that require complex reasoning, the answers to the majority of questions will not be found in the abstract alone. retrieval-augmented generation with llms (rag) we follow a retrieval-augmented generation setup for this configuration. research paper texts exceed the typical model context length with exception of few long-context models (which we will discuss in the next experimental setup). to accomodate processing such documents we employ a rag setup, where we first divide the document into smaller and slightly overlapping chunks, retrieve the most relevant chunks to the question using a bm25 ranker, foot_5 and subsequently input the top ranked chunks to the llm, tasked with generating the response. the operational flow of this pipeline is depicted in appendix figure 9 and the chunking algorithm is presented in appendix algorithm 1. comprehending the full-text using llms (full-text) in this experimental setup, llms are provided with the full-text of scientific papers and are tasked with answering a specific question. the length of scientific texts could exceed the context length limit of many llms. in such cases, we divide the full-text into segments. each segment, along with the question and instructions, is then presented to an llm (referred to as base-llm), which generates answers for each segment. this setup produces multiple answer candidates for a single question, contingent on the number of passes required to present all chunks to the llm. to distill these into a singular, optimal response, we introduce an answer selection phase. during this phase, the llama 3.1 70b model is prompted with the question and all answers generated by the base-llm, with instructions to identify the most comprehensive response from the provided options. details of this prompt are included in the appendix b in figure 11 . we only segment paper's full-text when it exceeds the model's context length (pipeline presented in appendix figure 10 ). for models with context length limit greater than the full-text (gemini, gpt-4o, and gpt-4o-mini), the base-llm directly generates the answer from the full text, and the answer-selection phase is not required. for qwen v2.5 (1.5b and 7b) and llama 3.1 (8b and 70b) models, the context length is 128k, however the prompt with the entire paper text does not fit into the cache, so we chunk the text and generate multiple answer candidates similar to other llms, however, the answer selection ## results and discussion in this section we dicsuss the main results. we provide the main results as averages of all metrics for all the four experimental settings in then we present the detailed results of each setting in tables 4 and 5 . comparison among closed-book, title-abs, rag, and full-text settings: we compare the average scores of all metrics for the models in four settings in ## conclusion and future work we curate scidqa, dataset designed to challenge language models on the qa task aiming to evaluate their understanding of scientific papers. the dataset consists of 2937 qa pairs, and extracts qa asked by reviewers and answered by paper authors during reviewer-author discussion during manuscript review on openreview. our multi-stage refinement pipeline annotates for quality, decontextualizes the qa pairs, edits references, and establishes the source document from different manuscript versions. our dataset features questions necessitating reasoning across multiple modalities beyond mere text, including figures, tables, equations, appendix and supplementary materials. scidqa also provides a testbed for evaluation of multi-document comprehension properties of llms. we evaluate the performance of several open-source and proprietary llms in generating the answer to questions after comprehending the research paper. we posit that scidqa will serve as a useful resource to benchmark the performance of llms in scientific text comprehension.