Enhancing Information Retrieval in Fact Extraction and Verification (2023)

Abstract: Modern fact verification systems have distanced themselves from the black box paradigm by providing the evidence used to infer their veracity judgments. Hence, evidence-backed fact verification systems’ performance heavily depends on the capabilities of their retrieval component to identify these facts. A popular evaluation benchmark for these systems is the FEVER task, which consists of determining the veracity of short claims using sentences extracted from Wikipedia. In this paper, we present a novel approach to the the retrieval steps of the FEVER task leveraging the graph structure of Wikipedia. The retrieval models surpass state of the art results at both sentence and document level. Additionally, we show that by feeding our retrieved evidence to the best-performing textual entailment model, we set a new state of the art in the FEVER competition.

## introduction the two-year coronavirus pandemic and the recent war in ukraine have evidenced how easily disinformation spreads among the general public and the social consequences this can have. in the information era's day-to-day, we live in a super-connected media ecosystem that provides us with an endless stream of facts and hoaxes alike but no immediate tools to separate them @xcite . moreover, the rapid development of larger and more capable language models has made disinformation detection significantly harder since traditional fact-verification systems, usually framed as textual entailment classification problems, are now vulnerable to synthetic disinformation attacks @xcite . therefore, modern high-performing fact-verification systems include a previous information retrieval step to condition the posterior veracity judgment on the extracted evidence @xcite @xcite . the fever task @xcite consists in retrieving relevant evidence from wikipedia given a claim and labeling it as either supports, refutes, or not enough info. traditionally, systems participating in the fever challenge have divided the task into three steps @xcite , each corresponding to a part of their pipeline: the document retrieval step, the sentence retrieval step, and the textual entailment step. in contrast with the last two steps, for most top-performing systems, the document retrieval module is directly inherited or slightly modified from previous work @xcite . therefore, for this step, the majority of systems follow one of these two strategies: the mediawiki api + span-matching system @xcite . filtering relevant documents by querying the mediawiki api for each entity mentioned and discarding results if the entity is not present in the page's title. keyword matching + semantic similarity system @xcite . keyword matching search for initial filtering and neural semantic matching model (nsmn) for scoring candidate documents using a concatenation of their title and first sentence along with the claim. these approaches, although proven effective, pose three important limitations: l1. the usage of mediawiki api as a first document retrieval step limits the usability of the models outside wikipedia's scope. l2. the precision of representing an entire document using only its title and first sentence may prove insufficient to correctly assess semantical relevance. l3. discarding a document based on exact keyword matching can be excessively conser-vative considering query-reference flexibility (e.g. michael jackson-the king of pop). having identified the above research gaps, we pose the following research hypotheses: h1. an encoder used for asymmetric semantic search eliminates the mediawiki api dependency and can more effectively represent semantic relations between queries and documents. ## background the vast majority of systems participating in the fever task challenge divide their pipelines into three steps and import their document retrieval step from previous work @xcite @xcite . it is worth mentioning that although some systems @xcite @xcite have embedded the baseline document retrieval strategies directly into their architectures, more recent models @xcite have shown better results by concatenating the retrieved documents from the two baseline models (i.e., @xcite ; @xcite ) with other classical information retrieval techniques such as tf-idf @xcite or bm25 @xcite . the second step of most fever pipelines consists of performing sentence retrieval from the previously obtained documents. unlike the previous step, this task has been explored from various perspectives. in the early days of the fever task, systems used esim-based architectures @xcite . however, motivated by maximizing recall, the research focus changed to target the multi-hop evidence problem leading to the first iterative sentence retrieval models @xcite . these models use transformers @xcite to fine-tune large pre-trained language models (lm) used as backbone, such as bert @xcite , albert, @xcite or roberta @xcite . specifically, to target the multi-hop evidence problem, these models conceive the sentence retrieval step as an iterative process in which they assess the importance of new sentences by considering both the claim and the relevant sentences already retrieved. parallel to the iterative retrieval models, another variety of models leverage not only direct connections but the complete graph structure of wikipedia to rank sentences @xcite @xcite using graph neural networks (gnns) @xcite . state-of-theart models @xcite @xcite generally fall under one of these categories but have pivoted to more refined token-level representations or bigger lms such as bigbird @xcite , t5 @xcite or deberta @xcite . a recent approach, claim-dissector @xcite proposes to divide the retrieved documents into blocks instead of individual sentences and encode each block individually. the final step of the fever task involves recognizing textual entailment (te). this subtask has traditionally been treated as a multi-class classification problem and tackled by fine-tuning from scratch some lm making use of transformers, alignment and concatenation of the retrieved evidence @xcite @xcite . top-performing systems in fever's public leaderboard @xcite use deberta-based models already trained over the multi-genre natural language inference (mnli) task @xcite as backbone. ## formal task the fever task consists in performing evidencebacked claim verification. formally, the knowledge base, d, is a collection of more than 5 million documents each corresponding to a wikipedia page, d := {d i } i , where each document d i is itself a variable-size collection of sentences, d i := {s i j } j . given the collection of documents d and a query (a statement) q , a valid system s must return a veracity assessment v for q along with a subset e, of at most five sentences supporting or refuting q: datasets. the fever task, as of today, has three associated datasets: the training dataset, the shared task dev dataset, and the shared task test dataset (open competition) @xcite . the training dataset is the largest of the three containing 145,449 claims and is unbalanced towards the "supports" class, which represents more than half of the examples. the dev and test datasets are widely used as the evaluation benchmarks for a fever pipeline. they are equal in size @xcite and balanced between the three classes. metrics. following previous work, for evaluating performance we use accuracy (acc) in the textual entailment step, the fever score (fs) 2 for the whole pipeline, and recall@k (r@k) for the retrieval steps. additionally, we also consider the mean reciprocal rank (mrr) and the proportion of claims where the system returns at least one relevant item (and) in the retrieval tasks. ## model following the traditional pipeline organization, we propose a three-step architecture (see figure 1 ) where: i) the document retrieval step uses partial references in the claim and document-level encoding to select an initial collection of documents that is later expanded (if necessary) for addressing the multi-hop evidence problem; ii) the sentence retrieval step combines the sentence retrieval part of lf 2-iter + d xl model @xcite which is the current best-performing system with a deberta-based cross-encoder @xcite ; iii) the textual entailment 2 fs is the central metric for the fever task. a prediction is only deemed correct if the label is correct and the evidence is sufficient. ## experimental evaluation and results we present our results for the development dataset at every stage and the fever challenge competition (test set) results: document retrieval. as previously commented, some of the most recent approaches add the documents retrieved from classic techniques such as tf-idf and bm25 to the results retrieved from their main document retrieval architectures. in doing so, the retrieved documents lose the ranking order, and it would be inaccurate to directly compare recall@k since the results from these combined systems are not rankings but rather collections of documents. therefore, we compare our results with the unaltered baseline systems in textual entailment. in the test set (competition), combination r@20 [1] 94.50 [1] + [2] 94.74 [1] + [3] 95.03 [1] + [2] + [3] 95.27 [1] + [4] 95.30 [1] + [2] + [4] 95.80 [1] + [3] + [4] 97.01 [1] + [2] + [3] + [4] ## conclusions in this paper, we have proposed a retrieval architecture that combined with a textual entailment model outperforms the state of the art in all stages of the fever task. our architecture starts by leveraging document-level semantic representation to narrow an initial collection of documents to 20 candidates. filtered results are later expanded using the graph structure inherent to the built database. once expansion is completed, our model scores the context windows inside documents, ranks the link paths, and takes the top elements from the ranking, ensuring that no more than 20 documents are retrieved. then, the documents are passed on to the sentence retrieval model that combines the prediction of the lf 2-iter +d xl system with a simple cross-encoder to obtain a sentence-paths ranking. finally, following the approach in lf 2-iter + d xl , a pre-trained deberta-based mnli model is used and later post-processed based on the output logits. regarding our initial research hypotheses; considering the results obtained in the ablation study (see