A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents (2023)

Abstract: Many real-world applications (e.g., note taking, search) require extracting a sentence or paragraph from a document and showing that snippet to a human outside of the source document. Yet, users may find snippets difficult to understand as they lack context from the original document. In this work, we use language models to rewrite snippets from scientific documents to be read on their own. First, we define the requirements and challenges for this user-facing decontextualization task, such as clarifying where edits occur and handling references to other documents. Second, we propose a framework that decomposes the task into three stages: question generation, question answering, and rewriting. Using this framework, we collect gold decontextualizations from experienced scientific article readers. We then conduct a range of experiments across state-of-the-art commercial and open-source language models to identify how to best provide missing-but-relevant information to models for our task. Finally, we develop QaDecontext, a simple prompting strategy inspired by our framework that improves over end-to-end prompting. We conclude with analysis that finds, while rewriting is easy, question generation and answering remain challenging for today’s models.

## introduction tools to support research activities often rely on extracting text snippets from long, technical documents and showing them to users. for example, snippets can help readers efficiently understand documents @xcite or scaffold exploration of document collections (e.g. conducting literature review) @xcite . as more applications use language models, developers use extracted snippets to protect against generated inaccuracies; snippets can help users verify model-generated outputs @xcite and provide a means for user error recovery. however, extracted snippets are not meant to be read outside their original document: they may include terms that were defined earlier, contain anaphora whose antecedents lie in previous paragraphs, and generally lack context that is needed for comprehension. at best, these issues make extracted snippets difficult to read, and at worst, they render the snippets misleading outside their original context @xcite @xcite . in this work, we consider the potential for making extracted snippets more readily-understood in user-facing settings through decontextualization @xcite -the task of rewriting snippets to incorporate information from their originating contexts, thereby making them "stand alone". we focus our attention on scenarios in which users read snippets from technical documents (e.g., scientific articles). for example, consider a citation graph explorer that allows users to preview citation contexts to explain the relationship between papers @xcite . also, consider an ai research assistant that surfaces extracted attribution snippets alongside generated answers. figure 1 illustrates these two motivating applications. how do language models fare when performing snippet decontextualization over complex scientific text? our contributions are: first, we introduce requirements that extend prior decontextualization work @xcite to handle user-facing scenarios (e.g., delineation of model-generated edits). we characterize additional challenges posed by decontextualizing scientific documents (e.g., longer text, citations and references) and describe methods to address them ( §2). second, we propose a framework for snippet decontextualization that decomposes the task into three stages: question generation, question answering, and rewriting ( §3). this decomposition is motivated by a formative study in which our framework makes decontextualization less challenging and creates higher-quality annotations. we use this framework to collect gold decontextualization data from experienced readers of scientific articles ( §4). finally, with this data, we operationalize our framework by implementing qadecontext, a strategy for snippet decontextualization ( §5). our best experimental configuration demonstrates a 41.7% relative improvement over end-to-end model prompting ( §5.2). we find that state-of-theart language models perform poorly on our task, indicating significant opportunity for further nlp research. we perform extensive analysis to identify task bottlenecks to guide future investigation ( §6). ## decontextualization for user-facing snippets from scientific documents in this section, we define decontextualization and motivate some additional task requirements when considering user-facing scenarios. then, we describe additional task challenges that arise when operating on scientific documents. ## qa for decontextualization decontextualization requires resolving what additional information a person would like to be incorporated and how such information should be incorporated when rewriting @xcite . if we view "what" as addressed in our guidelines ( §2), then we address "how" through this proposal: ## data collection following the results of our formative study, we implemented an annotation protocol to collect decontextualized snippets from scientific documents. ## experimenting with llms for decontextualization we study the extent to which current llms can perform scientific decontextulaization, and how our qa framework might inform design of methods. ## analyzing performance bottlenecks through qadecontext modularity of our framework for decontextualization allows us to study performance bottlenecks: which subtask (question generation, question answering, rewriting) do llms struggle with the most? we conduct ablation experiments to better understand the performance and errors of each module in qadecontext. we refer the reader to ## conclusion in this work, we present a framework and a strategy to perform decontextualization for snippets from scientific documents. we introduce task requirements that extend prior work to handle user-facing scenarios and the handle the challenging nature of scientific text. motivated by a formative study into how humans perform this task, we propose a qa-based framework for decontextualization that decomposes the task into question generation, answering, and rewriting. we then collect gold decontextualizations and use them to identify how to best provide missing context so that state-of-theart language models can perform the task. finally, we implement qadecontext, a simple prompting strategy for decontextualization, though ultimately we find that there is room for improvement on this task, and we point to question generation and answering in these settings as important future directions.