Re 2 G: Retrieve, Rerank, Generate
Abstract
As demonstrated by GPT-3 and T5, transformers grow in capability as parameter spaces become larger and larger.
However, for tasks that require a large amount of knowledge, nonparametric memory allows models to grow dramatically with a sub-linear increase in computational cost and GPU memory requirements.
Recent models such as RAG and REALM have introduced retrieval into conditional generation.
These models incorporate neural initial retrieval from a corpus of passages.
We build on this line of research, proposing Re 2 G, which combines both neural initial retrieval and reranking into a BART-based sequenceto-sequence generation.
Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval.
To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker and generation using only ground truth on the target sequence output.
We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact checking and dialog, with relative gains of 9% to 34% over the previous state-of-the-art on the KILT leaderboard.
We make our code available as open source 1 .
Introduction
GPT-
[Brown et al., 2020] and T5 [Raffel et al., 2020] are arguably the most powerful members in a family of deep learning NLP models called transformers.
Such models store surprising amount of world knowledge.
They have been shown to produce good performance on a range of demanding tasks, especially in generating human like texts.
However, such large transformers’ capability is tied to the increasingly larger parameter spaces on which they are trained
Recently, there has been work towards transformers that make use of non-parametric knowledge.
REALM (Retrieval Augmented Language
Model)
[Guu et al., 2020] and RAG (Retrieval Augmented Generation)
[Lewis et al., 2020b] both use an indexed corpus of passages to support conditional generation.
By using the corpus as a source of knowledge these models can extend the information available to the model by tens or even hundreds of gigabytes with a sub-linear scaling in computation cost.
These recent advancements, in turn, have been inspired by BART (Bidirectional and AutoRegressive Transformer)
[Lewis et al., 2020a] that combines a Bidirectional Encoder (e.g. BERT [Devlin et al., 2019]) with an Autoregressive decoder (e.g. GPT [Brown et al., 2020]) into one sequenceto-sequence model.
We build on this line of research, pioneered by REALM and RAG, and propose a new approach that we call Re 2 G ( Re trieve, Re rank,
G enerate), which combines both neural initial retrieval and reranking into a BART-based sequenceto-sequence generation.
There are two particular aspects on which our approach is different from the previous works.
Firstly, our reranking approach permits merging retrieval results from sources with incomparable scores, e.g. enabling an ensemble of BM25 and neural initial retrieval.
Secondly, to train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker and generation using only ground truth on the target sequence output.
The KILT benchmark [Petroni et al., 2021] has been recently introduced to evaluate the capabilities of pre-trained language models to address NLP tasks that require access to external knowledge.
We evaluate on four diverse tasks from KILT: slot filling, question answering, fact checking and dialog.
Figure 1 shows examples of these tasks.
Re 2 G makes significant gains on all four tasks, reaching the top of the KILT leaderboards and establishing a new state-of-the-art.
The contributions of this work are as follows:
• We introduce Re 2 G, demonstrating the effectiveness of reranking for generative language models that incorporate retrieval.
• We further extend Re 2 G by ensembling initial retrieval methods, combining neural and traditional keyword-based approaches.
• Re 2 G improves the current state-of-the-art of 9%, 31%, 34%, 22% and 10% relative gains on the headline KILT metrics for T-REx (slot filling),
Natural Questions (question answering), TriviaQA (question answering), FEVER (fact checking), and Wizard of Wikipedia (dialog), respectively.
• We publicly release our code as open source to support continued development.
Related Work
The KILT benchmark and public leaderboard 2 combines eleven datasets across five tasks.
The main advantage of the KILT distribution of these datasets is that the provenance information from each dataset is realigned to reference the same snapshot of
Wikipedia.
A unified evaluation script and set of metrics is also provided.
In this work, we focus on four tasks, such as Slot Filling [Levy et al., 2017, Elsahar et al., 2018], Question Answering [Kwiatkowski et al., 2019, Joshi et al., 2017],
Fact Checking [Thorne et al., 2018a,c], and Dialog [Dinan et al., 2019] (see Figure 1).
A set of baseline methods have been proposed for KILT.
GENRE [Cao et al., 2021] is trained on BLINK [Wu et al., 2020] and all KILT tasks jointly using a sequence-to-sequence language model to generate the title of the Wikipedia page where the answer can be found.
This method is a strong baseline to evaluate the retrieval performance, but it does not address the downstream tasks.
On the other hand, generative models, such as BART [Lewis et al., 2020a] and T5 [Raffel et al., 2020], show interesting performance when finetuned on the downstream tasks relying only on the implicit knowledge stored in the weights of the neural networks, without the use of any explicit retrieval component.
RAG [Lewis et al., 2020b], an end-to-end retrieval-based generative model, is the best performing baseline in KILT and it incorporates
DPR [Karpukhin et al., 2020] to first retrieve relevant passages for the query, then it uses a model initialized from BART [Lewis et al., 2020a] to perform a sequence-to-sequence generation from each evidence passage concatenated with the query in order to generate the answer.
Figure 2 shows the architecture of RAG.
Multi-task DPR [Maillard et al., 2021] exploits multi-task learning by training both DPR passage and query encoder on all KILT tasks.
DensePhrases [Lee et al., 2021] addresses the knowledge intensive tasks with a short answer, such as slot filling.
It indexes the phrases in the corpus that can be potential answers.
The extracted phrases are represented by their start and end token vectors from the final layer of a transformer initialized from SpanBERT [Joshi et al., 2020].
Knowledge Graph Induction (KGI)
[Glass et al., 2021] combines DPR and RAG models, both trained with task and dataset specific training.
KGI employs a two phase training procedure: first training the DPR model, i.e. both the query and context encoder, using the KILT provenance ground truth.
Then, KGI trains the sequence-to-sequence generation and further trains the query encoder using only the target output as the objective.
This results in large improvements in retrieval performance and, as a consequence, in the downstream tasks.
KILT-WEB 2 [Piktus et al., 2021] addresses the
KILT tasks by broadening the knowledge source used. Rather than rely only on KILT’s Wikipedia snapshot, KILT-WEB 2 creates S PHERE as a knowledge source. S PHERE is built from CCNet [Wenzek et al., 2020] and over twenty times the size of the
Wikipedia corpus.
It can use either BM25 or DPR retrieval (though not both combined) followed by a ‘reader’ component, but not trained end-to-end.
The reader component is the Fusion-in-Decoder [Izacard and Grave, 2021] model, where retrieved documents are encoded independently, then their encoded representations are concatenated for the decoder.
SEAL [Bevilacqua et al., 2022] introduces a novel generative approach to retrieval.
Rather than generating the unique document identifier like
GENRE, SEAL can generate any ngrams present
[Figure]
Provenance
[Figure]
[Figure]
Input: when did bram stoker’s drac
[Figure]
Output
[Table]
1897 Provenance:
[Figure]
[Table]
[Figure]
Input: Dracula is a novel by a Scot- tish author.
Output
[Table]
REFUTES Provenance: 7923-
[Table]
Dracula (7923)
Dracula is an 1897 Gothic horror novel by Irish author Bram Stoker.
It introduced the character of Count Dracula, and established many conventions of subsequent vampire fantasy.
The novel tells the story of Dracula’s attempt to move from Transylvania to England so that he may find new blood and spread the undead curse, and of the battle between Dracula and a small group of men and a woman led by Professor Abraham Van Helsing.
Wizard of Wikipedia
Input:
[Table]
I really like vampires!!
[Table]
are
[Table]
Do you have any favorite vampires?
[Table]
think
[Table]
best one
[Table]
[Table]
horror
[Table]
[Figure]
[Figure]
in the corpus, which are then mapped to passages.
The neural retrieval generator is based on BART and constrained to generate ngrams that appear in the corpus with an FM-Index [Ferragina and
Manzini, 2000].
Like KILT-WEB 2, SEAL uses
Fusion-in-Decoder as the component responsible for generating the output conditioned on the retrieved passages.
Multi-stage or cascade approaches to retrieval have received ample attention in Information Retrieval (IR) research.
The multi-stage approach begins with the initial retrieval phase, where an initial set of documents or passages form the pool of candidates to be considered for ranking.
Then one or more phases of increasingly computationally demanding rerankers are applied.
Early approaches in learning to rank [Liu, 2009] used features and linear classifiers.
Pre-trained language models, especially
BERT [Devlin et al., 2019], have shown state-ofthe-art performance when applied to the task of relevance ranking.
Transformers may be applied as classifiers to each query and passage pair independently [Nogueira and Cho, 2019] or as generators to produce labels for passages in a sequence-tosequence model [Nogueira et al., 2020].
Methodology
The approach of RAG, Multi-DPR, and KGI is to train a neural IR (Information Retrieval) component and further train it end-to-end through its impact in generating the correct output.
Figureillustrates the end-to-end RAG system.
It has been previously established that results from initial retrieval can be greatly improved through the use of a reranker [Liu, 2009, Wang et al., 2011].
Therefore we hypothesized that natural language generation systems incorporating retrieval can benefit from reranking.
Linear Layer and Softmax [SEP]
T [SEP]
E [SEP] …
E [CLS] E
E N
E [SEP]
E’
E’ M
C
T
T N
T [SEP]
T’
T’ M [CLS]
Query [SEP]
Passage … … … … … …
BERT
[Figure]
CLS
[Figure]
r q
Query
[Figure]
In addition to improving the ranking of passages returned from DPR, a reranker can be used after merging the results of multiple retrieval methods with incomparable scores.
For example, the scores returned by BM25 [Robertson and Zaragoza, 2009] are not comparable to the inner products from DPR.
Using the scores from a reranker, we can find the top-k documents from the union of DPR and BM results.
Figure 3 illustrates our extension of RAG with a reranker.
We call our system Re 2 G ( Re trieve,
Re rank, G enerate).
3. Reranker
The reranker we use is based on the sequence-pair classification of Nogueira and Cho [2019].
This model is shown in Figure 4.
The query and passage are input together to a BERT [Devlin et al., 2019] transformer.
Cross attention is applied over the tokens of both sequences jointly.
This is called an interaction model.
This model contrasts with the representation model used for initial retrieval.
Figure 5 shows the bi-encoder representation model for DPR.
The representation vectors for the query and passage are produced independently.
This allows for efficient retrieval by pre-computing vectors for all passages in the corpus and indexing them with an
ANN (Approximate Nearest Neighbors) index.
By using an interaction model to rerank the top-N passages from the representation model, we can get the advantages of both model types: accuracy and scalability.
We initialize the reranker from the BERT model trained on MS MARCO
[Nguyen et al., 2016] by
NBoost [Thienes and Pertschuk, 2019] and available through Hugging Face 3 .
3. Training
As Figure 1 illustrates, KILT tasks are provided with two types of ground truth: the target output sequence and the provenance information indicating the passage or passages in the corpus that support the output.
Our training is carried out in four phases: DPR training, generation training, reranking training, and full end-to-end training.
The initial DPR and reranking phases make use of the provenance ground truth.
The generation and full end-to-end training make use of only the target output.
Formally:
• The original KILT instances are a tuple: ⟨ q, t, Prov ⟩ where q is the input or prompt, t is the target output, and Prov is the set of provenance passages that support the target output.
• DPR training is a tuple: ⟨ q, p + , p − ⟩ where p + ∈ Prov and p − where p − ∈ BM25 ( q )
∧ p − / ∈
Prov • Reranking training begins with the application of DPR and BM25, producing tuples: ⟨ q, P , Prov ⟩ where P = BM25 ( q ) ∪ DPR ( q ) • Generation and end-to-end training instances are pairs of query and target: ⟨ q, t ⟩
The first two phases, DPR and generation, are identical to KGI, specifically KGI 0 .
We use the codes from Glass et al.
DPR Stage 1 training is the same training used by Karpukhin et al.
The triplets of query, positive passage and “hard negative” passages from
BM25 are put into batches of 128 instances.
The positives and hard negatives from other instances form the “batch negatives” for each instance.
The
DPR bi-encoder model gives each query a probability distribution over the positive, hard negative, and batch negatives.
The loss is the negative loglikelihood for the positive.
After DPR Stage 1 training the passages from the corpus are indexed with a Hierarchical Navigable Small World (HNSW)
[Malkov and Yashunin, 2018] using FAISS [Johnson et al., 2017].
Generation training extends the training of the query encoder and trains the BART LARGE sequence-to-sequence model on the target sequence output.
This training is the same as that described by Lewis et al.
[2020b].
3. Reranking Training
The next phase, training the reranking in isolation, begins with gathering the initial retrieval results from DPR and BM25 on the training set.
These results are merged and used as training data for the reranker.
In some datasets there are multiple positive passages.
Therefore, we use the negative of the summed log-likelihood for the positive passages as the loss function.
The logits given by the reranker are z r and the indices for the correct passages (from the ground truth provenance) are Prov .
[Equation]
[-KGI 
-]
[-KGI 
-]
3. End-to-End Training
Training end-to-end poses a special challenge.
In
RAG, the gradient propagates to the query encoder because the inner product between the query vector and the passage vector is used to weight the influence of each sequence, a process RAG calls marginalization.
The inputs to the BART model are sequences ( s j = p
j [SEP] q ) that comprise a query q plus retrieved passage p j .
The probability for each sequence is determined from the softmax over the retrieval (or reranker) scores for the passage.
The probability for each target token t i given the sequence s j is a softmax over BART’s token prediction logits.
The loss therefore is a negative log-likelihood summed over all target tokens and sequences, weighted by each sequence’s probability.
Consider that in Re 2 G the score from the reranker, not the initial retrieval, is used to weight the impact of each sequence in generation.
This allows the reranker to be trained through the ground truth on target output, but it means the gradient for the query encoder will be zero since the marginalization no longer depends on the inner product from the query and passage representation vectors.
[Equation]
j
P ( t i | s j ) = softmax ( BART ( s j ) i ) t i loss = − (cid:88) i,j log ( P ( t i | s j ) · P ( s j ))
We consider three possible resolutions to this issue.
• Combine the DPR and reranker scores • Freeze the query encoder • Online Knowledge Distillation
The first candidate solution is tempting but fatally flawed.
By adding the log softmax from DPR and the reranker we can ensure that both systems are trained through impact in generation.
However, if the DPR score is added to the reranker score, then the DPR score is being trained to provide a complementary signal to the reranker.
Therefore, when
DPR is used to gather the candidate passages, it does not give the highest scores to the passages that are most likely to be relevant, but instead gives the highest scores to the passages the reranker is most likely to underrate.
We find that this theoretical concern is also a practical concern, as DPR performance (and overall system performance) declines greatly when trained in this way.
The simplest solution is to freeze the parameters of the query encoder, training only the reranker and generation components.
We find this is indeed the best solution for one of our datasets, Wizard of
Wikipedia.
Note that DPR has already been trained in two phases, first from the provenance ground truth and then again in generation training in the
RAG model.
The third solution is our novel application of knowledge distillation [Hinton et al., 2015].
We use the reranker as a teacher model to provide labels to the DPR student model.
We distill the knowledge across architectures: from an interaction model to a representation model.
Further, this knowledge distillation occurs online, while the reranker is being trained.
The loss for the initial retrieval is therefore the KL-divergence between the probability distribution it gives over the retrieved passages and the reranker’s probability distribution over the same passages.
A temperature hyperparameter T smooths these distributions to prevent excessive loss and stabilize training. T-REx (Slot Filling) R-Prec Recall@
Accuracy
F KILT-AC KILT-F Re 2 G (ours) 80.
{+KGI 
+}
[Glass et al., 2021] 74.
KILT-WEB 2 [Piktus et al., 2021] 75.
SEAL [Bevilacqua et al., 2022] 67.
{+KGI 
+}
[Glass et al., 2021] 59.
Natural Questions (Question Answering)
R-Prec
Recall@
Accuracy
F
KILT-AC
KILT-F
Re 2 G (ours) 70.
SEAL [Bevilacqua et al., 2022] 63.
{+KGI 
+}
[Glass et al., 2021] 63.
KILT-WEB 2 [Piktus et al., 2021] 59.
RAG [Petroni et al., 2021] 59.
TriviaQA (Question Answering)
R-Prec
Recall@
Accuracy
F
KILT-AC
KILT-F
Re 2 G (ours) 72.
SEAL [Bevilacqua et al., 2022] 68.
KILT-WEB 2 [Piktus et al., 2021] 58.
{+KGI 
+}
[Glass et al., 2021] 60.
MultiDPR [Maillard et al., 2021] 61.
FEVER (Fact Checking)
R-Prec
Recall@
Accuracy
KILT-AC
Re 2 G (ours) 88.
SEAL [Bevilacqua et al., 2022] 81.
KILT-WEB 2 [Piktus et al., 2021] 74.
{+KGI 
+}
[Glass et al., 2021] 75.
MultiDPR [Maillard et al., 2021] 74.
Wizard of Wikipedia (Dialog)
R-Prec
Recall@
Rouge-L
F
KILT-RL
KILT-F
Hindsight [Paranjape et al., 2021] 56.
Re 2 G (ours) 60.
SEAL [Bevilacqua et al., 2022] 57.
{+KGI 
+}
[Glass et al., 2021] 55.
RAG [Petroni et al., 2021] 57.
KILT [Table] [Table] The knowledge distillation has the usual advantage of providing signal not only of positive and negative instances, but degrees of negativeness.
In addition, since we retrieve n = 12 passages from
DPR but only use the top- k ( k = 5 ) for generation, the knowledge distillation loss is providing a (soft) label for more passages.
3. Inference
At inference time the query is encoded using the
DPR query encoder and the top-12 passages from the HNSW index are returned.
The query is also passed to BM25 search, specifically Anserini 5 , gathering the top-12 BM25 results.
Both sets of passages are passed to the reranker and scored.
The top-5 passages are then joined with the query and passed to BART LARGE to generate the output.
The five output sequences are weighted according to the softmax over the reranker scores to produce the final output.
Experiments
We test our model on five datasets, over four distinct tasks in the KILT benchmark: slot filling, question answering, fact checking and dialog.
Figure 1 shows an example of these four tasks.
The slot filling dataset, T-REx [Elsahar et al., 2018], provides as input a head entity and relation, and expects as output the entity or term that fills the slot, also called the tail entity.
The T-REx dataset contains 2.3M instances.
We use only 370k training instances by downsampling the relations that occur more than 5000 times.
This reduces the training time required while keeping state-of-the-art performance.
The development and test sets each have 5k instances.
The question answering datasets are “open” versions of Natural Questions [Kwiatkowski et al., 2019] and TriviaQA
[Joshi et al., 2017].
Unlike the original versions, the relevant Wikipedia page must be found by a retrieval step.
The training sets for Natural Questions and TriviaQA contain 87k and 62k questions, with another 3k and 5k for the development and 1.4k and 6.5k for test.
The fact checking dataset in KILT is FEVER (Fact Extraction and VERification).
It is a combination of the two FEVER versions [Thorne et al., 2018b, 2019] omitting the N OT E NOUGHI NFO class.
There are approximately 10k instances in the development and test sets, and 100k for training.
FEVER is a classification task, but we cast it as a generation task by training the model to generate either the token “SUPPORTS” or “REFUTES”.
Wizard of Wikipedia [Dinan et al., 2018] is the dialog dataset.
The input is a short dialog history ending with the information seeker’s turn.
The expected output is a fact presented conversationally or just an utterance or question mentioning content from a relevant Wikipedia page.
It is the smallest dataset with approximately 3k instances in development and test and 64k in train.
For all tasks, systems are expected to produce the target output as well as justify it with provenance information from the KILT knowledge source.
The metrics of R-Precision and Recall@5 measure the correctness of the provenance.
R-Precision measures what fraction of the R documents in the ground truth provenance ( | Prov | = R ) are present in the top- R documents returned by the system.
Accuracy and (token-level) F1 measure the correctness of the generated output.
For Wizard of
Wikipedia, Rouge-L [Lin, 2004] is used instead of accuracy, since systems are very unlikely to generate the exact target output.
The metrics of KILTAccuracy, KILT-F1 and, for Wizard of Wikipedia,
KILT-Rouge-L are the underlying metric (e.g. Accuracy) for instances where R-Precision is one, otherwise zero.
These metrics indicate output correctness when provenance is also correctly supplied.
Table 1 shows the performance of Re 2 G on the
KILT leaderboard.
We achieved 9%, 31%, 34%, 22% and 10% relative gains over the previous stateof-the-art on the headline KILT metrics for T-REx,
Natural Questions, TriviaQA, FEVER, and Wizard of Wikipedia, respectively.
Furthermore, Re 2 G has held the lead in the headline KILT metrics in all datasets except for Wizard of Wikipedia where it is now second best.
Since our submission to the KILT leaderboard for the Wizard of Wikipedia, a new system called
Hindsight [Paranjape et al., 2021] achieved even better results on the generation metrics on that particular task.
The new system of SEAL has also achieved top results for some metrics on the Natural Questions and TriviaQA benchmarks.
4. Retrieval
Table 2 examines how the retrieval improves through each step of training.
In the first half of the table we consider the initial retrieval alone.
DPR
Stage 1 is the DPR training described earlier - training only from the provenance ground truth with batch negatives and hard negatives from BM25.
KGI 0 further trains the query encoder of DPR Stage 1 through its impact in generating the target output.
Finally Re 2 G extends the training of DPR with online knowledge distillation from the reranker.
This step is beneficial in two of the three datasets, while the previous steps improve performance across all datasets.
In the second half of the table we examine the improvement in reranking.
The baseline of KGI
DPR+BM25 merges the results of KGI 0 ’s DPR and
BM25 by scoring each passage by the sum of the inverse rank from each method.
For both T-REx and
FEVER, even this simple approach to ensembling
DPR and BM25 improves Recall@5, although not
R-Precision.
Following reranker training using the provenance ground truth (Reranker Stage 1), we find improvement over DPR across all five datasets on both retrieval metrics.
The reranker’s improve- [Table] ment following end-to-end training is mixed.
In
FEVER and Wizard of Wikipedia there is substantial gain in R-Precision, approximately 2%.
T-REx and Natural Questions are flat.
However, there is a sharp decline in the performance of TriviaQA, in retrieval metrics.
This is true despite the fact that retrieving these passages greatly improves answer accuracy and F1.
This suggests some incompleteness in the provenance ground truth for TriviaQA.
4. Ablations
Table 3 explores ablations of the Re 2 G system. The point estimates and 95% confidence intervals are reported. Re 2 G-KD excludes the online knowledge distillation, instead freezing the query encoder when training the reranker and generator during end-to-end training. Re 2 G-BM25 excludes BM results, fetching 24 passages from DPR rather than 12 from DPR and 12 from BM25.
The passages are still reranked. KGI 0 is the baseline system, without a reranker and therefore also without BM25 results or online knowledge distillation during training. Both online knowledge distillation and ensembling with BM25 improve performance in four out of five datasets. Online knowledge distillation failed to improve for Wizard of Wikipedia and ensembling with BM25 failed to improve for Natural
Questions.  Analysis
Since the Re 2 G model differs from the KGI model only in the retrieval phase, we hypothesized that its gains in output quality are driven by its better retrieval quality.
To test this hypothesis we considered all cases where the Re 2 G model produces better output than the KGI 0 model and calculated the fraction of such cases where Re 2 G’s rank for the first correct passage is lower than KGI
’s.
We find that for T-REx, NQ, and FEVER the fractions of output gains that could be attributed to improved retrieval and ranking are 67.73%, 61.08% and 66.86% respectively.
While for TriviaQA and
Wizard of Wikipedia only 36.86% and 27.74% of output improvements were accompanied by improved ranking for the correct passage.
It is important to note that in Wizard of Wikipedia, many of these improved outputs have only a small gain in token-level F1.
While much of the gain in output quality is attributable to improved recall, at least a third is not.
This reinforces an observation of Glass et al. [2021], that models trained with better retrieval can produce better output even when the retrieved passages are equivalent at test time.
5. Slot filling error analysis
To understand the types of errors Re 2 G makes we sampled 50 instances of the development set of the
T-REx dataset where the Accuracy and token-level
F1 score was zero.
Interestingly, the most common class of error (33/50) was due to the incompleteness of the ground truth.
Often the head entity is ambiguous (19/50), or the relation has multiple fillers (16/50).
As an example, consider the following where there are two Joe O’Donnell notable for sports in the passages retrieved, and each played for at least two different teams.
Joe O’Donnell [SEP] member of sports team Target: Buffalo Bills Re 2 G : Dumbarton F.C. • Joe O’Donnell (footballer) /
Joe O’Donnell (footballer) Joseph ’Joe’ O’Donnell (bornMarch 1961) was a Scottish footballer who played for Dumbarton and Stranraer.
• Joe O’Donnell (American football) / ... fullback, guard and tackle for the University of Michigan from 1960 to 1963.
He also played professional football as a guard and tackle for eight seasons for the Buffalo Bills ...
When Re 2 G produces genuine errors it is usually [Table] because it has selected some entity as a filler related in a different way (6/17) or it has failed to retrieve the necessary passage (9/17).
Conclusions
Re 2 G considerably advanced the state-of-the-art across five KILT datasets, and still holds the top position in four of the five.
Relative to previous work, such as RAG or KGI, Re 2 G substantially improves both in retrieval and end-to-end performance on slot filling, question answering, fact checking, and dialog.
The reranker alone improves performance and enables the inclusion of multiple sources of initial retrieval.
This architecture permits us to integrate results from BM25, further improving accuracy.
Our online knowledge distillation is able to improve the performance of DPR in four of the five datasets, despite the loss in end-to-end training not depending on the DPR scores.
Similarly, the ensembling of DPR and BM25, which is enabled by our incorporation of a reranker, benefits four of the five datasets tested.
We have directed our efforts towards improving the retrieval of relevant knowledge.
This also enables improvement in endto-end performance by supplying better passages to the generation component.
Further experiments on domain adaptation of Re 2 G on tasks like question answering or dialog might provide useful insight on the application of this technology to real world use cases.
We are releasing our source code as open source (Apache 2.0 license) to enable further research. URL https
//aclanthology.org/ 2020.findings-emnlp.63 . Ashwin Paranjape Omar Khattab Christopher Potts
Matei Zaharia,
Christopher D Manning
Hind-
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick
Lewis, Majid Yazdani
Nicola De
James Thorne,
Yacine Jernite
Vladimir Karpukhin, Jean Maillard,
Vassilis Plachouras, Tim Rocktäschel,
Sebastian Riedel
Online
Association for Computational
Linguistics. doi: 10.18653/v1/2021.naacl-main.
URL https://aclanthology.org/ 2021.naacl-main.200 .
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin,
Dmytro Okhonko, Samuel Broscheit, Gautier Izacard,
Patrick Lewis, Barlas O˘guz, Edouard Grave, Wentau
Yih, et al.
The web is your oyster–knowledgeintensive nlp against a very large web corpus arXiv preprint arXiv
Colin
Noam Shazeer
//aclanthology.org/ 2020.lrec-1.494 .
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer.
Scalable zero-shot entity linking with dense entity retrieval.
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP) , pages 6397–6407, Online,
November 2020.
Association for Computational
Linguistics. doi: 10.18653/v1/2020.emnlp-main.
URL https://aclanthology.org/ 2020.emnlp-main.519 .
Appendix A Hyperparameters
We have not done hyperparameter tuning for DPR
Stage 1, Generation, or Reranking training.
Instead we used hyperparameters similar to the original works on training DPR, BERT reranking and RAG.
Table 4 shows the hyperparameters used in our experiments.
For knowledge distillation we used the same hyperparameter settings as Generation.
For the additional hyperparameters in online knowledge distillation: temperature and KD learn rate scaling, we experimented with temperatures of 10 andand KD learn rate scaling of 1.0 and 0.1.
For our reported results we used a temperature of 10 . 0 and a learn rate scaling of 1 . 0 .
When training using online knowledge distillation, there is a separate optimizer for the query encoder while training generation.
This optimizer uses the same hyperparameter settings.
Table 6 shows the settings for retrieval and generation used for all datasets.
All results are from a single run.
The random seed for python, numpy and pytorch was 42.
B Software Details
We used the following software versions:
• Ubuntu• Pytorch 1.
Transformers 4.3.
• Anserini 0.4.
(commit 3a60106fdc83473d147218d78ae7dca7c3b6d47c)
C Model Details
Number of parameters
Re 2 G uses three
BERT BASE transformers: query encoder, passage encoder and reranker.
Each has 110 M parameters.
The generation component is a BART LARGE model with 400 M parameters.
There are 730 M parameters in total.
Computing infrastructure
Using a single
NVIDIA V100 GPU DPR training of two epochs takes approximately 24 hours for T-REx and less than 12 hours for FEVER and WoW.
Using a two NVIDIA P100 GPUs generation training for 370k T-REx instances takes two days, while FEVER and WoW training completes in half a day.
The FAISS index on the KILT knowledge source requires a machine with large memory, we use machines with 128GB of memory.
Hyperparameter DPR
Reranker
Generation
[Table]
D Generation Analysis
We examined 20 instances coupled with 3 output texts: the baseline KGI 0 , Re 2 G
[Table]
and the target text in the ground-truth. The three output texts were presented unlabeled and in random order to avoid bias.
For each instance, we read the conversation history and then mark each text either GOOD ,
OK or INCONSISTENT generation.
To our surprise, 5/20 ground-truth target texts are INCONSISTENT which indicates the WoW benchmark might have limitations in annotation quality.
Both the systems have similar results ( GOOD/OK/INCONSISTENT - Re 2 G: 8/2/10; KGI 0 : 9/2/9) .
Second [Table] we checked a set of 20 WoW instances where Re 2 G’s F1 score was in the bottom quintile.
The conversation history was presented along with Re 2 G generated text and the passages re- [Table] [Table] trieved.
Manual examination showed 8/20 as
INCONSISTENT and in 4/8 cases supporting groundtruth passages were not retrieved.
Below is one of the 12/20 cases where Re 2 G generated text was found CONSISTENT with respect to the conversation history, although it has low F1 and Rouge-L scores.
Conversation History:
• My favorite color is red. • Red is at the end of the spectrum of light, its with orange and opposite of violet.
• I didn’t know that.
What else do you know about red?
Target:
It’s actually a primary color for the RGB and CMYK color model.
Re 2 G : It has a dominant wavelength of approximately 625-740 nanometres.
D. Generation Quality
Table 7 shows couple of examples that were part of the set of randomly selected instances from WoW dataset and used for manual inspection.
We choose these two particular instances to show when we thought the ground truth (i.e. target) is not coherent with respect to the corresponding conversation history.
In the first example, the system generated outputs were judged as coherent.
We found that both
Re 2 G and KGI 0 retrieved the following passage which might have helped generation of the above output -
Horseshoe Falls / Horseshoe Falls
Horseshoe Falls, also known as Canadian Falls, is the largest of the three waterfalls that collectively form Niagara
Falls on the Niagara River along the
Canada–United States border.
Approximately 90% of the Niagara River, after diversions for hydropower generation, flows over Horseshoe Falls.
The remaining 10% flows over American Falls and
Bridal Veil Falls.
It is located between
Terrapin Point on Goat Island in the US state of New York, and Table Rock in the
Canadian province of Ontario.
Section:
International border.
As for the ground truth, we marked it (factually) inconsistent based on the following retrieved passageNiagara Falls / Located on the Niagara
River, which drains Lake Erie into Lake
Ontario, the combined falls have the highest flow rate of any waterfall in
North America that has a vertical drop of more than .
During peak daytime tourist hours, more than 168,000 m ( six million cubic feet ) of water goes over the crest of the falls every minute.
Horseshoe Falls is the most powerful waterfall in North
America, as measured by flow rate.
In the second example, all three texts were marked inconsistent.
Interestingly, all the items in the conversation contains subjective opinion.
Consequently, all the three candidate texts also contains subjective opinion.
The problem is both the systems generated texts that are almost repetition of earlier conversation.
In case of the ground truth, we find that the text is semantically incoherent.
We have also submitted files that contain all instances that were used to generate the different analysis reported in Section 4.2 of the paper.
These files also contains our annotations/remarks where applicable.
Conversation history:
• Have you ever been to Niagra Falls?
I want to go there, but I know it’s very far.
Those beautiful 3 waterfalls are right between New York and Ontario, thousands of miles from me!
• I never have
but I would love to.
It looks beautiful.
Yes, they do!
They are different sized waterfalls, named Horseshoe, American and Bridal Veil Falls.
• I had no idea.
I’ve never heard that before.
Very cool.
• Yes, and Horseshoe Falls in actually the most powerful waterfall in all of the US!
• I never new any details about it.
I have just seen it in pictures and on TV and it looks very neat.
Re 2 G: Horseshoe falls, also known as Canadian Falls, is the largest of the three waterfalls that collectively form Niagara Falls on the Niagara River along with American Falls and Bridal Veil Falls.
[GOOD] KGI 0 : Horseshoe falls is also known as Canadian Falls, which straddles the international border between Canada and the U.S. [GOOD]
Ground truth:
It’s really quite interesting and beautiful.
There’s over six
million cubic feet of
water that crosses
the
fall’s crest
every
60 seconds.
Can you imagine?
[INCONSISTENT - 6M cubic feet of water crosses for thewaterfalls combined, not just Horseshoe falls.]
Conversation history:
• I love the taste of pasta mixed with tomato sauce and shredded cheese all over it •
I just love mines with chunky tomatoes mixed with meatballs and mushrooms.
• Olive Garden is the place that makes the best pastas.
I try to visit the place as much I can with my friends • I love Olive Garden especially the original one in Orlando they opened in 1982.
• I think Italians have special genes that makes them so adept at preparing pastas •
I believe so as well I would love to visit Argentina that is where Italian-born people reside great pasta I bet.
• I don’t know how they do it, but no matter how hard I try to prepare one at home, mine ends up with some peculiar taste Re 2 G : I don’t know how they do it
but I love the taste of pasta mixed with tomato sauce and shredded cheese all over it
[INCONSISTENT - Dialogue repetition]
KGI 0 : I just love the taste of pasta mixed with tomato sauce
and shredded cheese all over it [INCONSISTENT - Dialogue repetition]
Ground truth: The taste is so full of an abundance of taste it just can’t be copied lol.
[INCONSISTENT - Semantically incoherent text “taste is full of .. taste”.]