Data-Efficient Auto-Regressive Document Retrieval for Fact Verification (2022)

Abstract: Document retrieval is a core component of many knowledge-intensive natural language processing task formulations such as fact verification. Sources of textual knowledge such as Wikipedia articles condition the generation of answers from the models. Recent advances in retrieval use sequence-to-sequence models to incrementally predict the title of the appropriate Wikipedia page given an input instance. However, this method requires supervision in the form of human annotation to label which Wikipedia pages contain appropriate context. This paper introduces a distant-supervision method that does not require any annotation train auto-regressive retrievers that attain competitive R-Precision and Recall in a zero-shot setting. Furthermore we show that with task-specific supervised fine-tuning, auto-regressive retrieval performance for two Wikipedia-based fact verification tasks can approach or even exceed full supervision using less than 1/4 of the annotated data. We release all code and models

## background conventional information retrieval methods, such as tf-idf and bm25, have been applied many knowledge-intensive nlp tasks @xcite , with reasonable success. these methods do not require supervision: instead, query-document similarity is estimated based on token-level frequency information from observations on a fixed corpus. at test time, sparse discrete vector encodings of documents and the query are compared to return documents with the highest similarity to the query. while this aids application to new tasks and settings, recall can be low, especially when there are variations in phrasing due to the sparse encoding of which tokens (or variants considering n-grams or subtokens) are present. neural retrieval @xcite in contrast, uses neural networks to generate dense encodings of the query and passages. these models are trained with supervision: the training data contains lists of appropriate passages for a given query, but typically does not contain negative instances. how negative instances are sampled in training influences the suitability of the retrieved documents for the downstream task @xcite . variants of training regimes for dense retrieval also use cloze-task @xcite , wikipedia revision information @xcite , contrastive @xcite learning, or multi-task learning @xcite to improve performance. ## data efficient document retrieval the primary objective of this paper is to reduce the dependency on supervised instances and exploit distant supervision to train an autoregressive document retrieval system. distant supervision for deardr exploits the structural aspects of the wikipedia corpus: specifically, the page titles (denoted pt) and hyperlinks (denoted hl) from sentences to other pages. sentences from wikipedia documents are sampled from the corpus as input and deardr is trained to decode the page title or hyperlinks, or both (denoted pthl). even though the deardr has only been pretrained with distant supervision without exposure to annotated training data for a knowledge intensive nlp task, we hypothesize that training to predict a wikipedia page title or hyperlinks acts a a reasonable analog that simulates a common component of many retrieval oriented tasks. this should be suffi-cient to allow zero-shot application to retrieve relevant documents without the need for human annotations enabling application of knowledge-intensive nlp tasks to new domains or languages. in contrast, the genre model @xcite is trained with data from eleven wikipedia-based nlp tasks with millions of annotated instances. constrained decoding at test time, a result set is decoded by aggregating the results from a beam search @xcite with constrained decoding @xcite . however, in contrast to genre, which predicts a single entity per beam, deardr is trained to predict a sequence of all the hyperlinked page names (illustrated in figure 1 ). self-supervised vs task-specific retriever once the deardr retriever has been pre-trained with self-supervision on wikipedia data, it can be applied in a zero-shot setting to the knowledgeintensive task of fact verification. some aspects of the test-task formulation, may have patterns that differ to what deardr is exposed to during the pre-training. using small numbers of task-specific training data, the pre-trained deardr will be finetuned evaluated on downstream tasks. we hypothesize that the pre-training regimen for deardr will reduce the number of instances needed to train the system and attain similar performance to a system with full supervision. ## experimental setup three different pre-training regimens for deardr, based on page title (pt), sentence hyperlinks (hl) and a combination of both (pthl), are performed using the snapshot of wikipedia from june 2017. this was the snapshot used for the fever shared task. document-level retrieval for two fact verification tasks will be evaluated: fever @xcite and hover @xcite . zero-shot document retrieval: without exposure to the underlying test task, deardr will be pre-trained using unlabeled instances from english wikipedia articles (pre-trained with pt, hl or pthl), and then applied to instances from these retrieval-based nlp tasks. from wikipedia, we generate 16.8m distant-supervision instances. ## conclusions and future work we show that distant supervision and pre-training enables high precision autoregressive document retrieval with fewer annotated training data. while previous work has studied the utility of pre-training