R e2 G : Retrieve, Rerank, Generate (2022)

Abstract: As demonstrated by GPT-3 and T5, transformers grow in capability as parameter spaces become larger and larger. However, for tasks that require a large amount of knowledge, non-parametric memory allows models to grow dramatically with a sub-linear increase in computational cost and GPU memory requirements. Recent models such as RAG and REALM have introduced retrieval into conditional generation. These models incorporate neural initial retrieval from a corpus of passages. We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation. Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval. To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker and generation using only ground truth on the target sequence output. We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact checking and dialog, with relative gains of 9% to 34% over the previous state-of-the-art on the KILT leaderboard. We make our code available as open source.

## related work the kilt benchmark and public leaderboard 2 combines eleven datasets across five tasks. the main advantage of the kilt distribution of these datasets is that the provenance information from each dataset is realigned to reference the same snapshot of wikipedia. a unified evaluation script and set of metrics is also provided. in this work, we focus on four tasks, such as slot filling @xcite , question answering @xcite , fact checking @xcite , and dialog @xcite (see figure 1 ). a set of baseline methods have been proposed for kilt. genre @xcite is trained on blink @xcite and all kilt tasks jointly using a sequence-to-sequence language model to generate the title of the wikipedia page where the answer can be found. this method is a strong baseline to evaluate the retrieval performance, but it does not address the downstream tasks. on the other hand, generative models, such as bart @xcite and t5 @xcite , show interesting performance when finetuned on the downstream tasks relying only on the implicit knowledge stored in the weights of the 2 https://eval.ai/web/challenges/ challenge-page/689/leaderboard neural networks, without the use of any explicit retrieval component. rag @xcite , an end-to-end retrieval-based generative model, is the best performing baseline in kilt and it incorporates dpr @xcite to first retrieve relevant passages for the query, then it uses a model initialized from bart @xcite to perform a sequence-to-sequence generation from each evidence passage concatenated with the query in order to generate the answer. figure 2 shows the architecture of rag. multi-task dpr @xcite exploits multi-task learning by training both dpr passage and query encoder on all kilt tasks. densephrases @xcite addresses the knowledge intensive tasks with a short answer, such as slot filling. it indexes the phrases in the corpus that can be potential answers. the extracted phrases are represented by their start and end token vectors from the final layer of a transformer initialized from spanbert @xcite . knowledge graph induction (kgi) @xcite combines dpr and rag models, both trained with task and dataset specific training. kgi employs a two phase training procedure: first training the dpr model, i.e. both the query and context encoder, using the kilt provenance ground truth. then, kgi trains the sequence-to-sequence generation and further trains the query encoder using only the target output as the objective. this results in large improvements in retrieval performance and, as a consequence, in the downstream tasks. kilt-web 2 @xcite addresses the kilt tasks by broadening the knowledge source used. rather than rely only on kilt's wikipedia snapshot, kilt-web 2 creates sphere as a knowledge source. sphere is built from ccnet @xcite and over twenty times the size of the wikipedia corpus. it can use either bm25 or dpr retrieval (though not both combined) followed by a 'reader' component, but not trained end-to-end. the reader component is the fusion-in-decoder @xcite model, where retrieved documents are encoded independently, then their encoded representations are concatenated for the decoder. seal @xcite introduces a novel generative approach to retrieval. rather than generating the unique document identifier like genre, seal can generate any ngrams present t-rex input: dracula [sep] narrative location output: transylvania provenance: 7923-2 natural questions input: when did bram stoker's dracula come out output: 1897 provenance: 7923-1 fever input: dracula is a novel by a scottish author. output: refutes provenance: 7923-1 dracula (7923) dracula is an 1897 gothic horror novel by irish author bram stoker. it introduced the character of count dracula, and established many conventions of subsequent vampire fantasy. the novel tells the story of dracula's attempt to move from transylvania to england so that he may find new blood and spread the undead curse, and of the battle between dracula and a small group of men and a woman led by professor abraham van helsing. wizard of wikipedia input: • i really like vampires!! • vampires are intense and based on european folklore. do you have any favorite vampires? • i think dracula is the best one!!! output: he's one of the best! he's based on the character from the 1897 horror book of the same name. provenance: 7923-1 query marginalization query encoder add query top-k passages output ann index generator figure 2: rag architecture query marginalization query encoder add query top-n passages top-k passages output ann index bm25 index reranker generator figure 3: re 2 g architecture in the corpus, which are then mapped to passages. the neural retrieval generator is based on bart and constrained to generate ngrams that appear in the corpus with an fm-index @xcite . like kilt-web 2, seal uses fusion-in-decoder as the component responsible for generating the output conditioned on the retrieved passages. multi-stage or cascade approaches to retrieval have received ample attention in information retrieval (ir) research. the multi-stage approach begins with the initial retrieval phase, where an initial set of documents or passages form the pool of candidates to be considered for ranking. then one or more phases of increasingly computationally demanding rerankers are applied. early approaches in learning to rank @xcite used features and linear classifiers. pre-trained language models, especially bert @xcite , have shown state-of-the-art performance when applied to the task of relevance ranking. transformers may be applied as classifiers to each query and passage pair independently @xcite or as generators to produce labels for passages in a sequence-tosequence model @xcite . ## methodology the approach of rag, multi-dpr, and kgi is to train a neural ir (information retrieval) component and further train it end-to-end through its impact in generating the correct output. figure 2 illustrates the end-to-end rag system. it has been previously established that results from initial retrieval can be greatly improved through the use of a reranker @xcite . therefore we hypothesized that natural language generation systems incorporating retrieval can benefit from reranking. linear layer and softmax [sep] t[sep] e[sep] … e[cls] e1 en e[sep] e'1 e'm c t1 tn t[sep] t'1 t'm [cls] query encoder (bert) ## experiments we test our model on five datasets, over four distinct tasks in the kilt benchmark: slot filling, question answering, fact checking and dialog. figure 1 shows an example of these four tasks. the slot filling dataset, t-rex @xcite , provides as input a head entity and relation, and expects as output the entity or term that fills the slot, also called the tail entity. the t-rex dataset contains 2.3m instances. we use only 370k training instances by downsampling the relations that occur more than 5000 times. this reduces the training time required while keeping state-of-the-art performance. the development and test sets each have 5k instances. the question answering datasets are "open" versions of natural questions @xcite and triviaqa @xcite . unlike the original versions, the relevant wikipedia page must be found by a retrieval step. the training sets for natural questions and triviaqa contain 87k and 62k questions, with another 3k and 5k for the development and 1.4k and 6.5k for test. the fact checking dataset in kilt is fever (fact extraction and verification). it is a combination of the two fever versions @xcite omitting the notenough-info class. there are approximately 10k instances in the development and test sets, and 100k for training. fever is a classification task, but we cast it as a generation task by training the model to generate either the token "supports" or "refutes". wizard of wikipedia @xcite is the dialog dataset. the input is a short dialog history ending with the information seeker's turn. the expected output is a fact presented conversationally or just an utterance or question mentioning content from a relevant wikipedia page. it is the smallest dataset with approximately 3k instances in development and test and 64k in train. for all tasks, systems are expected to produce the target output as well as justify it with provenance information from the kilt knowledge source. the metrics of r-precision and recall@5 measure the correctness of the provenance. r-precision measures what fraction of the r documents in the ground truth provenance (|prov| = r) are present in the top-r documents returned by the system. accuracy and (token-level) f1 measure the cor-rectness of the generated output. for wizard of wikipedia, rouge-l @xcite is used instead of accuracy, since systems are very unlikely to generate the exact target output. the metrics of kilt-accuracy, kilt-f1 and, for wizard of wikipedia, kilt-rouge-l are the underlying metric (e.g. accuracy) for instances where r-precision is one, otherwise zero. these metrics indicate output correctness when provenance is also correctly supplied. since our submission to the kilt leaderboard for the wizard of wikipedia, a new system called hindsight @xcite achieved even better results on the generation metrics on that particular task. the new system of seal has also achieved top results for some metrics on the natural questions and triviaqa benchmarks. ## analysis since the re 2 g model differs from the kgi model only in the retrieval phase, we hypothesized that its gains in output quality are driven by its better retrieval quality. to test this hypothesis we considered all cases where the re 2 g model produces better output than the kgi 0 model and calculated the fraction of such cases where re 2 g's rank for the first correct passage is lower than kgi 0 's. we find that for t-rex, nq, and fever the fractions of output gains that could be attributed to improved retrieval and ranking are 67.73%, 61.08% and 66.86% respectively. while for triviaqa and wizard of wikipedia only 36.86% and 27.74% of output improvements were accompanied by improved ranking for the correct passage. it is important to note that in wizard of wikipedia, many of these improved outputs have only a small gain in token-level f1. while much of the gain in output quality is attributable to improved recall, at least a third is not. this reinforces an observation of @xcite , that models trained with better retrieval can produce better output even when the retrieved passages are equivalent at test time.