Hierarchical Representations in Dense Passage Retrieval for Question-Answering (2023)

Abstract: An approach to improve question-answering performance is to retrieve accompanying information that contains factual evidence matching the question. These retrieved documents are then fed into a reader that generates an answer. A commonly applied retriever is dense passage retrieval. In this retriever, the output of a transformer neural network is used to query a knowledge database for matching documents. Inspired by the observation that different layers of a transformer network provide rich representations with different levels of abstraction, we hypothesize that useful queries can be generated not only at the output layer, but at every layer of a transformer network, and that the hidden representations of different layers may combine to improve the fetched documents for reader performance. Our novel approach integrates retrieval into each layer of a transformer network, exploiting the hierarchical representations of the input question. We show that our technique outperforms prior work on downstream tasks such as question answering, demonstrating the effectiveness of our approach.

## introduction in open book question answering, the answer to a given question needs to be generated from a large pool of passages. typically, this problem is tackled in two stages. given a question, a retriever collects a set of top-k passages from the passages memory. then, a reader generates the answer from the retrieved documents. in this setting, dense passage retrieval (dpr) is a commonly used retriever @xcite . therein, each passage in a document collection is represented as a vector in a high-dimensional space. these vectors are then used to compute similarity scores between passages. the most similar passages are then retrieved and used as input to a machine learning model. however, we observe that current open-book qa systems do not adequately exploit the correlations when did harvard become an ivy league school? fetchr: harvard: 300, ivy league: 109 dpr: harvard: 341, ivy league: 66 who overthrew the mongols and established the ming dynasty? fetchr: mongols: 108, ming: 112 dpr: mongols: 108, ming: 87 when did the soviet union first gain control of parts of poland and the baltic republics? fetchr: soviet union: 139, poland: 93 , baltic republics: 7 dpr: soviet union: 133, poland: 214, baltic republics: 2 question figure 1: we present the occurrences of word features inside the document collection as retrieved by either dpr or fetchr. the feature from the question with the lowest occurrence is the critical feature for qa tasks. our retriever fetchr outperforms dpr on critical word features in the question (underlined). our work shows that this improved document collection increases the reader performance by up to 1.9 em score on natural question and 2.1 em score on webquestion. we present a novel retriever architecture and training procedure to test this hypothesis (figure 2 ). our architecture extends bert @xcite with a neural retrieval network producing queries not only at its output layer but also at intermediate layers. this is inspired by the observation that different layers provide different level of abstraction @xcite that can be all used for downstream tasks @xcite . under this setup, the retrievable documents embody a non-parametric knowledge of the transformer @xcite . with a separate reader function, an answer is inferred from the documents retrieved by the hierarchical retrievers (see section 3 for details of the model). our main contributions are: training of dual encoders past work has shown @xcite @xcite that the performance of dual encoders can be improved by (i) carefully selecting negative documents for the contrastive loss and (ii) by an end-to-end training with a reader function. cross-batch negatives, denoised hard negatives and data augmentation are options mentioned in @xcite for negative documents. end-to-end training approaches tune the retrieval function to match the distribution of data for the reader @xcite @xcite . these approaches are orthogonal to our work and might be considered as an extension for future work. ## results in the following, we evaluate how fetchr influences the performance of modern readers and present isolated retrieval results of fetchr.