A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021)

Abstract: Readers of academic research papers often read with the goal of answering specific questions. Question Answering systems that can answer those questions can make consumption of the content much more efficient. However, building such tools requires data that reflect the difficulty of the task arising from complex reasoning about claims made in multiple parts of a paper. In contrast, existing information-seeking question answering datasets usually contain questions about generic factoid-type information. We therefore present Qasper, a dataset of 5049 questions over 1585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers, motivating further research in document-grounded, information-seeking QA, which our dataset is designed to facilitate.

## introduction machines built to assist humans who engage with texts to seek information ought to be designed with an awareness of the information need. abstractly, the human's need should define the lens through which the system views the text in order to find desired information. existing information-seeking machine reading datasets (e.g., @xcite have led to significant progress in reading at scale (e.g., @xcite @xcite . however, most of those benchmarks focus on an "open domain" setting where the questions are not anchored in any particular user context. the result is an emphasis 3 dataset construction each dataset consists of a collection of records with one qa problem per record. for each record, we include some question text, a context document relevant to the question, a set of candidate solutions, and the correct solution. ## building the qasper dataset we now describe our process for constructing the dataset. we began with a set of open-access nlp papers, recruited nlp practitioners who are regular readers of research papers, and designed two different data collection interfaces: one for collecting follow-up questions given titles and abstracts, and another for obtaining evidence and answers to those questions. ## qasper analysis ## modeling qasper this section explains the task, evaluation metrics, and a model addressing qasper tasks. ## experiments we evaluate model performance on question answering and evidence selection tasks, and compare them to estimated lower bounds on human performance. these human performance estimates are calculated by comparing the answers of questions for which we have multiple human annotations. for each question, we choose one annotation as if it were a prediction, and evaluate it against the rest of the annotations, and consider as human performance the average over all annotations chosen as predictions. we restrict our experiments to the subset of questions in qasper that can be answered from text in the paper, ignoring those that require figures or tables as evidence (13% of the dataset; see section 3) to avoid having to deal with multimodal inputs. we leave multimodal question answering to future work. ## conclusion we presented qasper, an information-seeking qa dataset over nlp research papers. with natural questions asked as follow-up to titles and abstracts, the task presented by qasper requires evidence from multiple paragraphs and/or figures and tables within the full text of the papers. our empirical quire workers to write questions grounded in those snippets. results show plenty of room for improvement when compared to the estimated human performance, and suggest that qasper could serve as a test-bed for evaluating document-grounded qa research.