P eer QA : A Scientific Question Answering Dataset from Peer Reviews (2025)

Abstract: We present PeerQA, a real-world, scientific, document-level Question Answering (QA) dataset. PeerQA questions have been sourced from peer reviews, which contain questions that reviewers raised while thoroughly examining the scientific article. Answers have been annotated by the original authors of each paper. The dataset contains 579 QA pairs from 208 academic articles, with a majority from ML and NLP, as well as a subset of other scientific communities like Geoscience and Public Health.PeerQA supports three critical tasks for developing practical QA systems: Evidence retrieval, unanswerable question classification, and answer generation. We provide a detailed analysis of the collected dataset and conduct experiments establishing baseline systems for all three tasks. Our experiments and analyses reveal the need for decontextualization in document-level retrieval, where we find that even simple decontextualization approaches consistently improve retrieval performance across architectures. On answer generation, PeerQA serves as a challenging benchmark for long-context modeling, as the papers have an average size of 12k tokens.

## introduction the number of scientific articles is increasing exponentially @xcite , leading to an increase in review work and leaving researchers with an ever-expanding number of publications to read to keep up with their field. therefore, novel tools are required to support reviewing work and enable readers to consume information from scientific articles more efficiently @xcite . automatic question answering (qa) systems can provide such support, allowing researchers and reviewers to productively extract information from 1. cleaning 2. decontextualization 3. decomposition 4. filtering can the authors provide an explanation for the significant performance difference between cgexpan and lm-base? […] the performance difference between cgexpan and lm-base is striking. can the authors provide more insight into this? @xcite figure 1 : overview of the peerqa data collection process. from the peer review process (in green), we extract and process questions from the reviews. given the published version of the article and a question, an expert (in our case, the original paper authors) (1) checks the question and modifies or discards it, (2) annotates whether it is answerable or not (i.e. if there is sufficient information in the paper), and if so (3) highlights the evidence to answer the question and finally (4) provides a free-form answer to the question. an article, particularly if integrated directly into the reading and reviewing interface @xcite . qa systems can also improve the quality of peer review, e.g., by avoiding questions in a review that are addressed in the article but potentially overlooked by a reviewer. however, the development of qa models is limited by the availability of high-quality and realistic datasets in the scientific domain to measure the performance of methods. collecting scientific qa data is challenging because it requires expert annotators who are difficult to recruit. furthermore, naturally occur- to this end, we introduce peerqa, a real-world, scientific, document-level question answering dataset. peerqa supports three crucial tasks for qa over scientific articles: given a question and a paper, evidence sentences relevant to the question need to be retrieved. based on these, the answerability of the question can be decided. finally, the dataset contains free-form reference answers addressing the question. we leverage peer reviews to source questions, and answers are annotated by the authors of the respective papers. while most questions are from ml and nlp papers, 10% of questions come from other scientific domains, including geoscience and public health. figure 1 provides an overview of our data collection process. to summarize, our contributions are the following: scientific qa qa datasets in the scientific domain can generally be categorized as larger-scale datasets that are (semi-) automatically created and small expert-annotated datasets. in concurrent work, @xcite also explore extracting questions from peer reviews in the ml domain. unlike peerqa, their approach uses the authors' responses provided during the rebuttal to obtain reference answers. to identify supporting evidence from the paper for each answer, they employ a hybrid approach that combines manual and automated mapping of the answers to relevant information in the paper. long-context qa dialogue and qa systems grounded in a document have recently gained traction @xcite . in this vein, narra-tiveqa @xcite ) contains questions about movie scripts and books with an average length of 63k tokens. pang et al. (2022) construct a multiple-choice dataset over books and articles with an average length of 5k tokens focusing on questions that require reading the article in detail. reddy et al. (2024) extend finqa @xcite to financial documents with an average of 123k words. conditionalqa @xcite is a dataset of government documents with an average length of 1.5k tokens and answers tied to certain input conditions. peerqa serves as another resource for long-context qa, with documents having an average length of 12k tokens and 30% of questions requiring combining information from more than one location in the paper. 3 peerqa 3.1 data collection figure 1 provides an overview of the data collection process. we use papers and peer reviews from nlpeer (dycke et al., 2023) and extend this set with journals and conferences that publish peer reviews and camera-ready versions publicly. specifically, the data from arr 2022 (containing papers published at acl and naacl 2022), coling 2020, acl 2017, conll 2016, and f1000 was curated in nlpeer, partially based on previous data collections (kang et al., 2018; kuznetsov et al., 2022) and published under a cc-by-nc-sa 4.0 license. the data from the geoscience domain is published under a cc-by 4.0 license in two journals: earth system dynamics foot_0 (esd) and earth surface dynamics foot_1 (esurf). for iclr 2022/2023 and neurips datasets and benchmark track 2022, we retrieve papers and reviews from openreview. since they are without any license, we do not publish them in our release but provide a download and processing script. all questions and answers in peerqa are published under cc-by-nc-sa 4.0. paper processing. we extract the full text of the camera-ready version of a publication, including equations and captions, using grobid 0.8 @xcite , which also groups sentences into paragraphs, which we use later in our experiments. ## conclusion we introduced the peerqa dataset to advance and study question answering on scientific documents. we sourced peerqa's questions from peer reviews and obtained answer annotations from the paper authors. our dataset supports three crucial tasks for developing qa systems: evidence retrieval, answerability, and answer generation. we analyzed the collected data and established baseline systems for all three tasks. for evidence retrieval, we find that decontextualization is key to improving performance. on the answerability task, we find that models tend to either over-or under-answer, showing a bias for one of the classes. further, although models can fit the entire paper into context in the answer generation task, providing the model with the top passages from a retriever outperforms the full-text setting. we also show that with increased retrieval performance, the answer generation improves. finally, our error analysis highlights the need for better evaluation metrics and model reasoning abilities. science domains. while peerqa covers more scientific domains compared to prior work, there is a limited amount of data beyond the ml and nlp domains. a major challenge in data collection is the availability of public peer reviews with openly licensed scientific articles @xcite . we call on the scientific community to further transform reviewing practices to an open format. english-only. peerqa is limited to english since it is dominant in scientific writing. nevertheless, publications in other languages exist, and our data collection framework can be applied to any language. the evaluated retrievers are english-only models (except bm25, which is language-agnostic). some retrieval models have multi-lingual counterparts (e.g., mcontriever); however, due to a lack of multi-language data, their performance remains unclear. some of the evaluated generative models are also multilingual; the performance in other languages is likely to be different than in english. free-form annotations. while authors possess the ultimate expertise in their papers, they usually have knowledge beyond the information in their publications. some free-form answers contain information not included in the answer evidence. for this reason, we also compare the generated answer with the annotated answer evidence, measuring if the model can produce answers entailed by the information in the paper. long-form qa evaluation. evaluating freeform answers is challenging and an ongoing area of research. to evaluate different aspects, we use three metrics against two ground truths. ideally, we would have multiple free-form answer references; however, even collecting a single response has proven to be challenging. in the hope of better metrics, we also publish the generated answers of our baselines to facilitate adaptation to future, improved methods. methods. many llms and methods @xcite exist that could be applied to the tasks in peerqa. therefore, more sophisticated and specialized methods might exceed the reported performances. however, we focus on introducing the dataset and establishing baseline systems with widely used retrieval and generative models.