Natural Logic-guided Autoregressive Multi-hop Document Retrieval for Fact Verification (2022)

Abstract: A key component of fact verification is the evidence retrieval, often from multiple documents. Recent approaches use dense representations and condition the retrieval of each document on the previously retrieved ones. The latter step is performed over all the documents in the collection, requiring storing their dense representations in an index, thus incurring a high memory footprint. An alternative paradigm is retrieve-and-rerank, where documents are retrieved using methods such as BM25, their sentences are reranked, and further documents are retrieved conditioned on these sentences, reducing the memory requirements. However, such approaches can be brittle as they rely on heuristics and assume hyperlinks between documents.We propose a novel retrieve-and-rerank method for multi-hop retrieval, that consists of a retriever that jointly scores documents in the knowledge source and sentences from previously retrieved documents using an autoregressive formulation and is guided by a proof system based on natural logic that dynamically terminates the retrieval process if the evidence is deemed sufficient.This method exceeds or is on par with the current state-of-the-art on FEVER, HoVer and FEVEROUS-S, while using 5 to 10 times less memory than competing systems. Evaluation on an adversarial dataset indicates improved stability of our approach compared to commonly deployed threshold-based methods. Finally, the proof system helps humans predict model decisions correctly more often than using the evidence alone.

## introduction with the growing volume of potentially misleading and false claims @xcite , automated fact verification @xcite is of growing interest. a key component of opendomain fact verification systems is the retrieval of relevant documents from a knowledge base (kb) which provide the necessary evidence supporting claim: the 66th primetime emmy awards was hosted by an iraqi comedian born in 1973. ## related work early approaches to multi-hop document retrieval for automated fact verification are based on the rnr paradigm. they use sparse or entity-linking based retrievers to find candidate documents (e.g. @xcite ), rerank sentences in a classification formulation (such as esim @xcite or pre-trained encoders @xcite ), and use hyperlinks to find additional documents @xcite . the aforementioned approaches are limited to two iterations, using hyperlinks extracted from the initial list of candidate sentences to be considered as additional documents in a second iteration. assuming that the evidence from the first iteration (i.e. initial retrieval) is more important, @xcite down-weight hyperlinked documents by a fixed factor. @xcite proposes the use of a generative model that imagines missing evidence sentences and selects new sentences based on word overlap. indirect improvements to multi-hop rnr are achieved through stronger document retrieval models, such as genre @xcite , or more accurate rerankers @xcite . genre in particular produced state-of-the-art results on document retrieval for fact verification, by generating documents using an autoregressive formulation, not necessitating a dense index. multi-hop dense passage retrieval (mdr) @xcite iteratively retrieves evidence documents using dense passage retrieval (dpr) @xcite , a bi-encoder that encodes claim and documents separately and uses efficient maximum inner-product search to score each document in the kb. since the search space grows exponentially with each iteration in the number of documents in the kb, mdr uses beam search to aggressively prune the search space which reduces scalability to many hops. in contast, @xcite 's baleen retriever performs multi-hop document retrieval by condensing retrieved documents after each iteration into a condensed context (i.e. sentence(s)) that is used to update the dense representation of the claim, reducing the search space by omitting all other candidates. while condensing is similar to the reranking step of rnr, baleen then still scores each document in the kb at each hop, like mdr. they further propose late interaction (flipr), which allows different parts of the claim to match different relevant parts of documents. indicative of the challenge of scaling to large kbs is that out of 12 systems submitted to the feverous shared task @xcite , with feverous being the fact verification dataset with the largest kb) not a single one opted to use a dense retrieval index. beyond fact verification, in question answering (qa), sophisticated graph based retrieval models have been proposed that make explicit use of links in the kb @xcite . of particular relevance is the approach of @xcite , that uses the feedback of the reader after each iteration to determine whether the answer has been retrieved and is sufficient, thus determining the number of hops dynamically instead of fixing it in advance. in their qa benchmark, all answers to the questions can be found in the source; however in many fact verification datasets a key challenge included are claims that cannot be verified using the kb. moreover, as explained false information mentioned in claims can be misleading, making the stopping criterion based on sufficiency for fact checking more challenging. set documents @xmath0, . . . , d n } ⊂ d that is sufficient to support or refute a claim c. 2 we assume that each document is associated with a unique document title, following previous work @xcite . in the case of multi-hop retrieval n must be larger than 1. in the rnr paradigm the retrieval is done in three steps: (i) find and return the k bestscoring document sets d @xmath1, . . . d i t . . . d k t }, with t being the current retrieval iteration, with @xmath2, and d i 1 the ith best-scoring single-hop document set of length 1 (i.e. a single document), (ii) rerank sentences from the k document sets in d t into the top l sentences e t , iii) use e t to update the set of document sequences to d t+1 . steps two and three are then repeated for a total of n hops. since n is not known a priori for a given claim previous work sets n to an upper bound of the dataset. admiral performs the third step of rnr in two stages: (i) the retrieval of d t+1 by jointly scoring d and e t using a generative model in an autoregressive framework that cross-attends over the claim c and e t , (ii) a dynamic retrieval termination criterion formulated as an evidence sufficiency task by generating a proof of natural logic (maccartney and manning, 2014) based on proofver @xcite , hence the number of hops n d yn for admiral is being determined dynamically for each claim given the evidence e t , with n d @xmath3. ## experimental setup datasets we evaluate our multi-hop document retriever on fever @xcite , fever-ous @xcite , and hover @xcite . fever consists of claims that predominantly require a single evidence sentence (87%). contrary to @xcite who evaluate only on the multi-hop part on fever, we report results for multi-hop and the entire dataset; this is more realistic as in practice it is unknown in advance whether a claim requires multi-hop document retrieval. hover contains of 46%, 36%, and 18% twohop, three-hop, and four-hop claims, respectively. contrary to fever, and hover that only consider the introductory section of wikipedia pages, feverous considers entire wikipedia articles, including semi-structured evidence in the form of table (cells) and contains of 16% multi-hop claims. from feverous we consider only the claims that require exclusively sentence evidence, as we focus on text retrieval (feverous-s), which constitutes about 41% of claims in feverous. implementation details the autoregressive model for both retrieval and proof generation is bart @xcite , which are trained independently from each other. for the initial retrieval, we first retrieve document candidates d 1 for all three datasets using genre, fine-tuned on the kilt version of fever @xcite , and bm25 based on pyserini @xcite . to rerank the sentences of these documents and keeping the top @xmath4, we use the token-level evidence selection model of @xcite for fever, and the pointwise t5 reranker of @xcite for hover, and feverous-s. for fever and feverous-s we consider the top 10 documents for reranking while for hover we focus on the top 100, to keep scores comparable to previous work. ## discussion autoregressive generation we compare the auto-regressive document scoring method of ad-miral to some variants, namely a model that i) considers only the top sentence of e t for scoring and generation (top-1) ii) does not score documents and sentences jointly, instead it only scores documents, and the top-ranking documents are concatenated to the t highest-ranked documents of d t (not-joint) iii) scores a ranked set of documents directly (i.e. scoring d t ) (joint-docs). we also evaluate a admiral model that exploits hyperlink information by concatenating a sentence's hyperlinks to its the end before being passed as input. results are shown in robustness we further evaluate the robustness of our model by evaluating admiral on an adversarial fact verification dataset deseption @xcite , which consists adversarial attacks generated as part of the fever2.0 adversarial shared task @xcite . the attacks consider lexical variations/substitutions, entity disambiguation, (multi-hop) temporal reasoning, multiple prepositions and multi-hop reasoning. document level results for models trained on fever and evaluated on deseption are shown in ## human evaluation of sufficiency proofs a key advantage of admiral is the added interpretability of our multi-hop retriever through our proof-based sufficiency prediction. for instance, after an initial retrieval hop our model might predict evidence insufficiency, with the indicated span that information is missing for. if the model is not able to find the relevant information in the next hop, a user could consider a targeted modification of the claim based on the sufficiency proof to enable the model to follow a different retrieval path. conversely, the model might erroneously indicate evidence sufficiency, so the user can supersede the model's decision in an informed manner. to explore the interpretability of our sufficiency proofs, we conduct a forward prediction experiment @xcite . human subjects are asked to predict whether admiral considers the evidence for a given claim to be sufficient or not, by using the generated sufficiency proof. the comparison baseline provides only the evidence sentences instead. since we are evaluating the proof as an explanation mechanism to humans, we ensured that no subject was familiar with the deterministic nature of our approach. to enable non-experts to make use of the proof, we replaced the natops with english phrases, similar to ( krishna et al., 2022 ## conclusion this paper explored an auto-regressive retrieval and rerank model for multi-hop document retrieval that is guided by a proof system based on natural logic that dynamically terminates the retrieval process if the retrieved evidence is deemed sufficient. our model does only cause minimal memory footprint compared to current state-of-the-art retrieval models while achieving competitive retrieval recall and f 1 . human evaluation indicates that the generated proof as a sufficiency condition is interpretable, enabling a human-in-the loop in the model's retrieval process. future work aims to investigate to which extent a verification model (i.e. proofver) could inform the retrieval of evidence directly, creating an end-to-end closed loop system, as well as human-in-the-loop approaches.