DAPR : A Benchmark on Document-Aware Passage Retrieval (2024)

Abstract: The work of neural retrieval so far focuses on ranking short texts and is challenged with long documents. There are many cases where the users want to find a relevant passage within a long document from a huge corpus, e.g. Wikipedia articles, research papers, etc. We propose and name this task Document-Aware Passage Retrieval (DAPR). While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context. This drives us to build a benchmark for this task including multiple datasets from heterogeneous domains. In the experiments, we extend the SoTA passage retrievers with document context via (1) hybrid retrieval with BM25 and (2) contextualized passage representations, which inform the passage representation with document context. We find despite that hybrid retrieval performs the strongest on the mixture of the easy and the hard queries, it completely fails on the hard queries that require document-context understanding. On the other hand, contextualized passage representations (e.g. prepending document titles) achieve good improvement on these hard queries, but overall they also perform rather poorly. Our created benchmark enables future research on developing and comparing retrieval systems for the new task. The code and the data are available.

## introduction information retrieval (ir) helps efficiently locate relevant information from a vast resource collection, acting as a central component of many natural language applications. traditional approaches like bm25 compute simple statistics such as the frequency of the matched terms @xcite . recent approaches apply neural networks to represent queries and passages into vector representations, extending the task modeling from sim- ## the dapr task and the benchmark the dapr task asks the systems to retrieve and rank the relevant passages given their associated (long) document. formally, given a collection of passages @xmath0, for a query @xmath1, the retrieval system s : q × c × d -→ r is required to return the top-k passages @xmath2, p 2 , ...p k } st. ∀j, s(q, p j , d j ) ≤ min p @xmath3, p i , d i ). ## experiments in the experiments, we extend the sota passage retrievers (subsection 4.1) with document context with various approaches (subsection 4.2). ## results the main results are shown in ## discussion in this section, we discuss the reasons behind the results and carry out an error analysis. the neural retriever is dragon+ by default. in another issue is about the conflicts on the fusion weight between the bm25 relevance on the documents and the neural relevance on the passages (cf. subsubsection 4.2.1). we plot the dynamics of the passage-retrieval performance on nq and nq-hard along with different fusion weights in figure 2 . we find the evaluation score on the full set of queries (i.e. nq) peaks at a small fusion weight of 0.2 (on the document side), while that for the hard queries (i.e. nq-hard) peaks at a very large fusion weight of 1.0. this means it is difficult for hybrid retrieval to achieve a good performance on nq and nq-hard at the same time. ## conclusion inspired by the error analysis for the sota retrievers (dragon+, spladev2 and colbertv2), we find the major errors (53.5% in we test different ways to model such document context for passage retrieval. we find that while hybrid retrieval performs the strongest on the mixture of the easy and the hard queries, it completely fails on the hard queries that require document-context understanding. instead, we also test informing the passage representation with document context and we can observe good improvement on these hard queries, but overall these approaches also perform rather poorly. this opens up new interesting research opportunities to develop better retrieval approaches that are capable of understanding the document context during passage retrieval. our created benchmark allows the research community to create, test, and compare such approaches.