Phrase Retrieval Learns Passage Retrieval, Too
Anonymous authors
Abstract
Dense retrieval methods have shown great promise over sparse retrieval methods in a wide range of NLP problems.
Among them, dense phrase retrieval—the most ﬁne-grained retrieval unit—is appealing because phrases can be directly used as the output for question answering and slot ﬁlling tasks.
In this work, we follow the intuition that retrieving phrases naturally entails retrieving larger text blocks and study whether phrase retrieval can serve as the basis for coarse-level retrieval including passages and documents.
We ﬁrst observe that a dense phrase-retrieval system, without any retraining, already achieves better passage retrieval accuracy (+3-5% in top-5 accuracy on QA tasks) compared to passage retrievers, which also helps achieve superior end-to-end QA accuracy with fewer passages.
Then, we provide an interpretation for why phrase-level supervision can help learn better ﬁne-grained entailment compared to passage-level supervision, and also show that phrase retrieval can be improved to achieve competitive performance in document-retrieval tasks for entity linking and knowledge-grounded dialogue.
Finally, we demonstrate how phrase ﬁltering and vector quantization can reduce the size of our index by 4-10x, making dense phrase retrieval a practical and versatile solution in multi-granularity retrieval.
1. Introduction
Dense retrieval aims to retrieve relevant contexts from a large corpus, by learning dense representations of queries and text segments.
Recently, dense retrieval of passages [Lee et al., 2019, Karpukhin et al., 2020] has been shown to outperform traditional sparse retrieval methods such as TF-IDF and BM25 in a wide range of knowledge-intensive NLP tasks [Petroni et al., 2021], including open-domain question answering (QA)
[Chen et al., 2017], entity linking [Wu et al., 2020] and knowledge-grounded dialogue [Dinan et al., 2019].
One natural design choice of these dense retrieval methods is the retrieval unit.
For instance, passage retrievers [Karpukhin et al., 2020] encode a ﬁxed-size text block ofwords as the basic retrieval unit.
On the other extreme, recent work [Seo et al., 2019,
Lee et al., 2021] demonstrates that phrases can be used as a retrieval unit.
In particular,
Lee et al.
[2021] show that learning dense representations of phrases alone can achieve competitive performance in a range of open-domain QA and slot ﬁlling tasks.
This is [Table] [Figure] particularly appealing since the phrases can directly serve as the output, without relying on an additional reader model to process text passages.
In this work, we draw on an intuitive motivation that every single phrase is embedded within a larger text context and ask the following question: If a retriever is able to locate phrases, can we directly make use of it for passage retrieval as well?
We formulate phrasebased passage retrieval , in which the score of a passage is determined by the maximum score of phrases within it (see Figure 1 for an illustration).
By evaluating DensePhrases [Lee et al., 2021] on popular QA datasets, we observe that it achieves competitive or even better passage retrieval accuracy compared to the dense passage retriever (DPR)
[Karpukhin et al., 2020], without any re-training or modiﬁcation to the original model (Table 1).
The gains are especially pronounced for top- k accuracy when k is smaller (e.g., 5), which also helps achieve superior open-domain QA accuracy with a much smaller number of passages as input to a generative reader model [Izacard and Grave, 2021a].
To better understand the nature of dense retrieval methods, we carefully analyze the training objectives of phrase and passage retrieval methods.
While the in-batch negative losses in both models encourage them to retrieve topically relevant passages, we ﬁnd that phrase-level supervision in DensePhrases provides a stronger training signal than using hard negatives from BM25, and helps DensePhrases retrieve correct phrases, and hence passages.
Following this positive ﬁnding, we further explore whether phrase retrieval can be extended to retrieval of coarser granularities, or other NLP tasks.
Through ﬁne-tuning of the query encoder with document-level supervision, we are able to obtain competitive performance on entity linking [Hoﬀart et al., 2011] and knowledge-grounded dialogue retrieval [Dinan et al., 2019] in the KILT benchmark
[Petroni et al., 2021].
Finally, we draw connections to multi-vector passage encoding models [Khattab and
Zaharia, 2020, Luan et al., 2021], where phrase retrieval models can be viewed as learning a dynamic set of vectors for each passage.
We show that a simple phrase ﬁltering strategy learned from QA datasets gives us a control over the trade-oﬀ between the number of vectors per passage and the retrieval accuracy.
Since phrase retrievers encode a larger number of vectors, we also introduce using Optimized Product Quantization [Ge et al., 2013] and quantization-aware ﬁne-tuning, reducing the size of the phrase index from 307GB to 69GB (or even < 30GB with more aggressive ﬁltering) for full English Wikipedia, without any performance degradation.
This matches the index size of passage retrieval and makes dense phrase retrieval a practical and versatile solution for multi-granularity retrieval.
2. Background Passage retrieval.
Given a set of documents D , passage retrieval aims to provide a set of relevant passages for a question q .
Typically, each document in D is split into a set of passages and we denote the entire set of passages in D as P = { p 1 , . . .
, p M } , where each passage can be a natural paragraph or a ﬁxed-length text block.
A passage retriever is designed to return top- k passages P k ⊂ P with the goal of retrieving passages that are relevant to the question.
In open-domain QA, passages are considered relevant if they contain answers to the question.
However, many other knowledge-intensive NLP tasks (e.g., knowledge-grounded dialogue) provide human-annotated evidence passages or documents.
While traditional passage retrieval models rely on sparse representations such as BM [Robertson and Zaragoza, 2009], recent methods show promising results with dense representations of passages and questions, and enable retrieving passages that may have low lexical overlap with questions.
Speciﬁcally, Karpukhin et al.
[2020] introduce a dense passage retriever (DPR) that has a passage encoder E p ( · ) and a question encoder E q ( · ) trained on QA datasets and retrieves passages by using the inner product as a similarity function:
[Equation]
For open-domain QA where a system is required to provide an exact answer string a , the retrieved top- k passages P k are subsequently fed into a reading comprehension model such as a BERT-QA model [Devlin et al., 2019], and this is called the retriever-reader approach.
Phrase retrieval
While passage retrievers require another reader model to ﬁnd an answer, Seo et al.
[2019] introduce the phrase retrieval approach that encodes phrases in each document and performs similarity search over all phrase vectors to directly locate the answer.
Following previous work [Seo et al., 2018, 2019], we use the term ‘phrase’ to denote any contiguous text segment up to L words (including single words), which is not necessarily a linguistic phrase and we take phrases up to length L = 10.
Given a phrase s ( p ) from a passage p , their similarity function f is computed as:
[Equation]
where E s ( · ) and E q ( · ) denote the phrase encoder and the question encoder , respectively.
Since this formulates open-domain QA purely as a maximum inner product search (MIPS), it can drastically improve end-to-end eﬃciency.
While previous approaches [Seo et al., 2019,
Lee et al., 2020] relied on the combination of dense and sparse phrase vectors, Lee et al.
[2021] demonstrate that dense representations of phrases alone are suﬃcient to close the performance gap with retriever-reader systems.
3. Phrase Retrieval Learns Passage Retrieval
Phrases naturally have their source texts from which they are extracted.
Based on this fact, we deﬁne a very simple phrase-based passage retrieval strategy, where we retrieve passages based on the phrase-retrieval score:
[Equation]
[Table]
where S ( p ) denotes the set of phrases in the passage p .
In practice, we ﬁrst retrieve top- kphrases, compute the score for each passage, and return top- k unique passages.
1 Based on our deﬁnition, phrases can act as a basic retrieval unit of any other granularity such as sentences or documents by simply changing S ( p ) (e.g., s ( d ) ∈ S ( d ) for a document d ).
In this section, we evaluate the passage retrieval performance of Eq.
(3) and also evaluate how phrase-based passage retrieval can contribute to open-domain QA systems.
3.1 Experiment: Passage Retrieval
Datasets.
We use two open-domain QA datasets, Natural Questions [Kwiatkowski et al., 2019] and TriviaQA
[Joshi et al., 2017], each of which follows the standard train/dev/test splits for the open-domain QA evaluation.
For both models, we use the 2018-12-20 Wikipedia snapshot.
To provide a fair comparison, we use Wikipedia articles pre-processed for DPR, which are split into 21M text blocks and each text block has exactly 100 words.
Models.
For DPR, we use publicly available checkpoints 2 trained on each dataset (DPR) or multiple QA datasets (DPR-multi) 3 , which we ﬁnd to perform slightly better than the ones reported in Karpukhin et al.
For DensePhrases, we train the models with the code provided by the authors 4 on Natural Questions (DensePhrases-NQ) or multiple QA datasets (DensePhrases-multi).
Note that we do not make any modiﬁcation to the architecture or training methods of DensePhrases and achieve similar open-domain QA accuracy as reported.
For phrase-based passage retrieval, we compute Eq.
(3) with DensePhrases and return top- k passages.
Metrics.
Following previous work on passage retrieval for open-domain QA, we measure the top- k passage retrieval accuracy (Top- k ), which denotes the proportion of questions whose top- k retrieved passages contain at least one of the gold answers.
To further characterize the behavior of each system, we also include the following evaluation metrics: mean reciprocal rank at k (MRR@ k ) and precision at k (P@ k ).
Given the top- k passages P k for a question, MRR@ k is the average reciprocal rank of the ﬁrst relevant passage (that contains [Table]
Grave [2021a
[Table]
[Table]
al
[Table]
[Table]
answer) in the top- passages is
Higher
k denotes that a larger proportion of
Results.
As shown in Table 1, DensePhrases achieves competitive passage retrieval accuracy with DPR, while having a clear advantage on top-1 or top-5 accuracy for both Natural
Questions (+6.7% Top-1) and TriviaQA (+7.5% Top-1).
Although the top-20 (and top100, which is not shown) accuracy is similar across diﬀerent models, MRR@20 and P@ reveal interesting aspects of DensePhrases—it ranks relevant passages higher and provides a larger number of correct passages.
Our results suggest that DensePhrases can also retrieve passages very accurately, although it was not explicitly trained for that purpose.
For the rest of the paper, we mainly compare DPR-multi and DensePhrases-multi.
3.2 Experiment: Open-domain QA
Recently, Izacard and Grave [2021a] proposed the Fusion-in-Decoder (FiD) approach where top-100 passages from DPR are fed into a generative model such as T5 [Raﬀel et al., 2020].
Since their generative model computes the hidden states of all tokens in 100 passages, this takes vast computational resources such as 64 Tesla V100 32GB GPUs.
We use our phrasebased passage retrieval with DensePhrases-multi to replace DPR-multi in FiD, and see if we can use a much smaller number of passages to achieve comparable performance, which can greatly reduce the computational requirements.
Since we train our model with 4 24GB
RTX GPUs, we use up to 50 passages for training T5-base.
Note that training T5-base with 5 or 10 passages can also be done with 11GB GPUs.
We keep all the hyperparameters the same except the batch size (reduced from 64 to 4) and accumulate gradients for 16 steps to match the batch size of the original work.
Results.
As shown in Table 2, using DensePhrases as a passage retriever achieves competitive performance with FiD (DPR-multi + T5-base) and signiﬁcantly improves upon the performance of original DensePhrases (without a reader).
Its better retrieval quality at top- k for smaller k indeed translates to better open-domain QA accuracy, achieving +6.4% gain compared to DPR + T5-base
when k = 5.
To obtain similar performance with using 100 passages in FiD, DensePhrases needs fewer passages, which can ﬁt in smaller GPUs.
[Table]
(cid:10)(cid:41)(cid:465)(cid:29)(cid:28)(cid:47)(cid:30)(cid:35)(cid:1)(cid:15)(cid:32)(cid:34)(cid:28)(cid:47)(cid:36)(cid:49)(cid:32)
4. A Uniﬁed View of Phrase and Passage Retrieval
As shown in the previous section, phrase-based passage retrieval can achieve competitive passage retrieval accuracy, despite the fact that phrase retrieval was not explicitly trained for that.
In this section, we compare the training objectives of DPR and DensePhrases in detail and explain how DensePhrases learns passage retrieval.
4.1 Training Objectives
Both DPR and DensePhrases set out to learn a similarity function f between a passage or phrase and a question.
Passages and phrases vary primarily in characteristic length, so we refer to either as a retrieval unit x . 5 DPR and DensePhrases both adopt a dual-encoder approach with inner product similarity (Eq.
(1) and (2)), using BERT [Devlin et al., 2019]
and SpanBERT [Joshi et al., 2020], respectively.
These dual-encoder models are then trained with a negative log-likelihood loss for discriminating positive retrieval units from negative ones:
[Equation]
where x + is the positive phrase or passage corresponding to question q , and X − is a set of negative examples.
The choice of negatives is critical in this setting and both DPR and
DensePhrases make important adjustments.
In-batch negatives.
In-batch negatives are a common way to deﬁne X − , since they are available at no extra cost when encoding a mini-batch of examples.
Speciﬁcally, in a mini-batch of B examples, we can add B − 1 in-batch negatives for each positive example.
Since each mini-batch is randomly sampled from the set of all training passages, in-batch negative passages are usually topically negative , i.e., models can discriminate between x + and X − based on their topic only.
Hard negatives.
Although topic-related features are useful in identifying broadly relevant passages, they often lack the precision to locate the exact passage containing the answer in a large corpus.
Karpukhin et al.
[2020] propose to use additional hard negatives which have a high BM25 lexical overlap with a given question but do not contain the answer.
These hard negatives are likely to share a similar topic and encourage DPR to learn more ﬁne-grained features to rank x + over the hard negatives.
Figure 2 (left) shows an illustrating example.
In-passage negatives.
While DPR is limited to use positive passages x + which merely contain the answer, DensePhrases is trained to predict that the positive phrase x
+ is the answer.
Thus, the ﬁne-grained structure of phrases allows for another source of negatives, in-passage negatives .
In particular, DensePhrases augments the set of negatives X − to encompass all phrases within the same passage that do not express the answer.
See Figure(right) for an example.
We hypothesize that these in-passage negatives achieve a similar eﬀect as DPR’s hard negatives: They require the model to go beyond simple topic modeling, since they share not only the same topic but also the same context.
Thus, phrase-based passage retriever might beneﬁt from this phrase-level supervision, which has already been shown to be useful in the context of distilling knowledge from reader to retriever [Izacard and Grave, 2021b, Yang and Seo, 2020].
4.2 Topical vs. Hard Negatives
[Figure]
with two losses:
L topic and L hard .
For both losses, the lower the better.
We show the eﬀect of training with in-batch negatives (in-batch), hard negatives (hard), and in-passage negatives (in-passage), as well as training on multiple
QA datasets (multi).
To address our hypothesis, we would like to study how the diﬀerent kinds of negatives used by DPR and DensePhrases aﬀect their reliance on topical and ﬁne-grained entailment cues.
We characterize their passage retrieval based on two metrics (losses),
L topic and L hard .
We use Eq.
(4) to deﬁne both L topic and L hard , but use diﬀerent sets of negatives X − .
For L topic , X − contains passages that are topically diﬀerent from the gold passage; in practice, a single random Wikipedia passage.
For L hard ,
X − uses negatives containing topically similar passages, such that L hard estimates how accurately models locate a passage that contains the exact answer among topically similar passages.
We create a single hard negative by removing the sentence that contains the answer from the positive passage.
Results.
Figure 3 shows the comparison of DPR and DensePhrases trained on NQ or multiple datasets with the two losses.
For DensePhrases, we compute the passage score using ˜ f
( p, q ) as described in Eq.
We observe that in-batch negatives are highly eﬀective at reducing L topic , while DensePhrases trained without in-batch negatives has a relatively high L topic .
Furthermore, we observe that DensePhrases with in-passage negatives achieves an even lower L hard than DPR, despite not using any hard negatives.
Using multiple datasets (denoted as ‘multi’) further improves L hard for both models.
The results suggest that DensePhrases relies less on topical features and is better at retrieving passages based on local entailment cues.
Hard negatives for DensePhrases?
We test two diﬀerent kinds of hard negatives in DensePhrases to see whether its performance can further improve in the presence of in-passage negatives.
For each question, we mine for a hard negative passage, either by
BM25 similarity or by ﬁnding another passage that contains the gold-answer phrase, but possibly with a wrong context.
Then we use all phrases from the hard negative passage as additional hard negatives in X − .
As shown in Appendix A (Table 5), DensePhrases obtains no substantial improvements from additional hard negatives, indicating that in-passage negatives are already highly eﬀective at producing good phrase (or passage) representations.
5. Improving Coarse-grained Retrieval
While we showed that DensePhrases implicitly learns passage retrieval, Figure 3 indicates that DensePhrases might not be very good for retrieval tasks where topic matters more than ﬁne-grained entailment, for instance, the retrieval of a single evidence document for entity linking.
In this section, we propose a simple method that can adapt DensePhrases to larger retrieval units, especially when the topical relevance is more important.
Method.
We modify the query-side ﬁne-tuning proposed by Lee et al.
[2021], which drastically improves the performance of DensePhrases by reducing the discrepancy between training and inference time.
Since it is prohibitive to update the large number of phrase representations after indexing, only the query encoder is ﬁne-tuned over the entire set of phrases in Wikipedia.
Given a question q and an annotated document set D ∗ , we minimize:
[Equation]
where ˜ S ( q ) denotes top- k phrases for the question, out of the entire set of phrase vectors.
To retrieve coarse-grained text better, we simply check the condition d ( s ) ∈ D ∗ , which means d ( s ), the source document of s , is included in the set of annotated gold documents D ∗ for the question.
With L doc , the model is trained to retrieve any phrases that are contained in a relevant document.
Note that d ( s ) can be changed to reﬂect any desired level of granularity such as passages.
Experimental setup.
We train DensePhrases with L doc for entity linking [Hoﬀart et al., 2011] or knowledge-grounded dialogue [Dinan et al., 2019] tasks from KILT [Petroni et al., 2021], where a gold document for each question is given.
On these tasks, each model is expected to provide a small set of articles related to the query.
We include DensePhrases trained with the original query-side ﬁne-tuning loss (denoted as L phrase ) as a baseline, which labels any phrase that matches the answer string (here, titles of gold documents are used as answer strings) as positive.
For retrieval evaluation, we adopt KILT’s oﬃcial document- [Table] level metrics based on the gold annotated documents: R-precision and Recall@5.
See the caption of Table 3 for the details on the metrics.
Results.
Table 3 shows the results on three entity linking tasks and a knowledge-grounded dialogue task with DensePhrases trained with L phrase or L doc .
For baseline methods, we include RAG [Lewis et al., 2020b], a DPR-based model that combines BART [Lewis et al., 2020a] as a generative reader, and Multi-task DPR from Petroni et al.
[2021], which is trained on multiple KILT tasks.
We also include results of DensePhrases trained with L doc on multiple KILT tasks, similar to the Multi-task DPR baseline.
On all tasks, we ﬁnd that DensePhrases with L doc performs much better than DensePhrases with L phrase and also matches the performance of RAG that uses an additional large generative model to generate the document titles.
Using L phrase does very poorly since it focuses on phraselevel entailment, rather than document-level relevance. Compared to Multi-task DPR,
Multi-task DensePhrases can be easily adapted to non-QA tasks like entity linking and generalizes better on tasks without training sets (WnWi, WnCw).
6. DensePhrases as a Multi-Vector Passage Encoder
In this section, we demonstrate that DensePhrases can be interpreted as a multi-vector passage encoder, which has recently been shown to be very eﬀective for passage retrieval [Luan et al., 2021, Khattab and Zaharia, 2020].
Since this type of multi-vector encoding models requires a large disk footprint, we show that we can control the number of vectors per passage (and hence the index size) through ﬁltering.
We also introduce quantization techniques to build more eﬃcient phrase retrieval models without a signiﬁcant performance drop.
Multi-vector encodings.
Since we represent passages not by a single vector, but by a set of phrase vectors (decomposed as token-level start and end vectors, see Lee et al.
[2021]), we notice similarities to previous work, which addresses the capacity limitations of dense, ﬁxed-length passage encodings.
While these approaches store a ﬁxed number of vectors per passage [Luan et al., 2021, Humeau et al., 2020] or all token-level vectors [Khattab and Zaharia, 2020], phrase retrieval models store a dynamic number of phrase vectors per passage, where many phrases are ﬁltered by a model trained on QA datasets.
Speciﬁcally, Lee et al.
[2021] train a binary classiﬁer (or a phrase ﬁlter) to ﬁlter phrases based on their phrase representations.
This phrase ﬁlter is supervised by the answer an- [Table] notations in QA datasets, hence denotes candidate answer phrases.
In our experiment, we tune the ﬁlter threshold to control the number of vectors per passage for passage retrieval.
Optimized product quantization.
Since the multi-vector encoding models are prohibitively large due to their multiple representations, we further introduce a vector quantization based method that can safely reduce the size of our phrase index, without performance degradation.
We use Product Quantization (PQ)
[Jegou et al., 2010] where the original vector space is decomposed into the Cartesian product of subspaces.
Using PQ, the memory usage of using N number of d -dimensional centroid vectors reduces from Nd to
N 1 /M
d with M subspaces while each database vector requires log 2 N bits.
Among diﬀerent variants of PQ, we use Optimized Product Quantization (OPQ)
[Ge et al., 2013], which uses an additional orthogonal matrix R to better decompose the original vector space.
See
Ge et al.
Quantization-aware training.
While this type of aggressive vector quantization can signiﬁcantly reduce the memory usage, it often comes at the cost of performance degradation due to the quantization loss.
To mitigate this problem, we use quantization-aware queryside ﬁne-tuning motivated by the recent successes on quantization-aware training [Jacob et al., 2018].
Speciﬁcally, during query-side ﬁne-tuning, we reconstruct the phrase vectors using the trained (optimized) product quantizer, which are then used to minimize Eq.
Results.
In Table 4, we present the top-5 passage retrieval accuracy with respect to the size of the phrase index in DensePhrases.
By tuning the threshold τ for the phrase ﬁlter, the number of vectors per each passage can be reduced without hurting the performance signiﬁcantly.
The performance improves with a larger number of vectors per passage, which aligns with the ﬁndings of multi-vector encoding models [Khattab and Zaharia, 2020, Luan et al., 2021].
Our results show that having 8.8 number of vectors per passage in DensePhrases has similar accuracy with DPR.
While applying OPQ can reduce the index size of DensePhrases from 307GB to 69GB, it should be done with the quantization-aware query-side ﬁne-tuning.
7. Conclusion
In this paper we show that phrase retrieval models also learn passage retrieval without any modiﬁcation.
By drawing connections between the objectives of DPR and DensePhrases, we provide a better understanding of how phrase retrieval learns passage retrieval, which is also supported by several empirical evaluations on multiple benchmarks.
Speciﬁcally, phrase-based passage retrieval has better retrieval quality on top- k passages when k is small, and this translates to an eﬃcient use of passages for open-domain QA.
We also show that DensePhrases can be ﬁne-tuned for more coarse-grained retrieval units, serving as a basis for any retrieval unit.
We plan to further evaluate phrase-based passage retrieval on standard information retrieval tasks such as MS MARCO
Appendix A. Hard Negatives for DensePhrases
[Table]