Neural Re-rankers for Evidence Retrieval in the FEVEROUS Task (2021)

Abstract: Computational fact-checking has gained a lot of traction in the machine learning and natural language processing communities. A plethora of solutions have been developed, but methods which leverage both structured and unstructured information to detect misinformation are of particular relevance. In this paper, we tackle the FEVEROUS (Fact Extraction and VERification Over Unstructured and Structured information) challenge which consists of an open source baseline system together with a benchmark dataset containing 87,026 verified claims. We extend this baseline model by improving the evidence retrieval module yielding the best evidence F1 score among the competitors in the challenge leaderboard while obtaining an overall FEVEROUS score of 0.20 (5th best ranked system).

## introduction the volume of potentially misleading and false claims has surged with the increasing usage of the web and social media. no barriers exist for publishing information which make anyone capable of diffusing false or biased claims while reaching large audiences with ease (baptista and gradim, 2020). one approach of dealing with this ordeal is computational fact-checking @xcite , where the automation of the verification pipeline or parts of it is flourishing due to advances in natural language processing @xcite . along these lines, several datasets and fact-evaluation algorithms have been proposed @xcite . in this paper, we report on our effort in tackling the feverous challenge @xcite . the provided dataset consists in a set of textual claims verified against evidence retrieved from a corpus of english wikipedia pages. the claims are labeled as supported, refuted or nei (not enough information). evidence can be unstructured (such as sentences) or structured (such as general tables, infoboxes, lists, etc.). the task is to return the right label with the correct evidence. the baseline model is divided in two main parts: an evidence retrieval part and a verdict prediction part. the evaluation is performed through the so-called feverous score which is computed considering both the correct retrieval of the evidence and the correct label predictions. in this paper, we propose an enhanced version of this baseline model that focuses on the retrieval component through a re-ranking process of pages, resulting in a more precise model. in the remainder of this paper, we first describe briefly the challenge task and the supplied data, and we detail our extension (section 2). we then provide experimental results obtained on the development dataset and we discuss observations on analyzed errors (section 3). we conclude with a discussion of other research directions that can be applied to improve results for the feverous task (section 4). ## method we begin by reviewing the given baseline, and then propose an extension to it that improved the precision and recall of the page-retrieval module in exchange for more computation time. ## experiments model. we rely on the cross-encoder model fine-tuned on the ms marco passage ranking task @xcite provided on the hugging face model hub. we feed the claim with every extracted page into the re-ranker model to obtain scores used to re-rank the respective pages. settings. we set @xmath0, scored with the re-ranker model, and then the top-5 pages for each claim are extracted. the remainder of the pipeline remains intact (@xmath1, @xmath2. we designate this pipeline by blpage(150) → p r(5) → tf idf (5, 3). our code can be found at https://gitlab.eurecom . results. we see improvement with the page reranker as the coverage of documents has increased compared to the baseline. hence, for @xmath3, the retriever without the re-ranker achieves a document coverage of 69% on the dev set, while the addition of the re-ranker enhances the coverage to around 79%, which, in turn, improves the feverous and f1 scores, compared to the initial baseline ( even with a better page retriever, an increase in feverous and f1 scores requires improvements also in the sentence and cell evidence retrievers. although more time-demanding, the re-ranker gives more subtle results than an entity-matching approach @xcite . for example, given the following excerpt of a claim: "family guy is an american animated sitcom that features five main voice actors [. there are some cases where entity-matching+tf-idf outperformed the re-ranker: some of those are cases where the wikipedia page content is small and does not bring much benefit on a semantic level and this is where tf-idf works better. we observe that we tend to miss the correct page when there are several pages who share similar semantics. for example, given the claim: "seven notable animated television series, including super why!, a children's educational show created by angela c. santomero and samantha freeman alpert, phineas and ferb and wordgirl, were released in september 2007.", the page re-ranker retrieves tv shows that are produced angela c. santomero). however, the correct page phineas and ferb does not appear in the top-5 predictions, and other pages take the lead, whereas the baseline can identify the correct page by entity matching, although its predictions are not as coherent as those of the page re-ranker. other attempts. we have experimented with varying the number of extracted pages k. we measure also the time taken for re-ranking. we have attempted to perform other extensions to the system that we describe below ( firstly, we specified the re-ranking system to extract less pages (50), but it worsened the scores. this configuration is defined as blpage(50) → p r(5) → tf idf (5, 3). furthermore, we applied the same re-ranking approach at the sentence level. after obtaining 150 pages from the page re-ranker, we continue to retrieve all sentences from every page and re-rank them using the same passage re-ranking model @xcite , (blpage(150) → p r(5) → sr(5) → tf idf table (3)). however, despite great outputs of page re-ranker, we could not obtain better results from sentences re-ranker than tf-idf. regarding how relevant sentences and tables are chosen as evidence, apart from tf-idf + cosine similarity, we also experimented with the okapi bm25 scoring function @xcite . this is applied after the pages are re-ranked, (blpage(150) → p r(5) → bm 25(5, 3)). surprisingly, although bm25 is generally preferred for document retrieval, in our case, it did not lead to better results compared to tf-idf. one possible cause might lie in text preprocessing, as we did not fully explore different combinations of preprocessing functions. lastly, we attempted to improve the verdict predictor by (i) fine-tuning the verdict classifier on the full training dataset (bl(5, 5, 3) f ull ) and by (ii) utilizing other pre-trained models that are either larger or were pre-finetuned on a nli dataset. however, we did not observe significant improvements from them since their performance on the dev set was either on par or slightly worse than the baseline model signaling that the focus enhancing of the second part of the system requires more significant changes. ## conclusion and future directions in this work, we have proposed the inclusion of a neural re-ranker model as a refinement step after standard methods such as tf-idf. while being more intensive on the computational side, we do see improvements on the document-retrieval side where results are more sound. there are of course more directions that are worth exploring to improve the results further. sentence retrieval could be improved by incorporating a pre-trained neural network that performs semantic matching between the claims and the sentences. one instance of such models is where the text sequences are encoded, then passed through an alignment layer that computes aligned representations of the encoded input sequences, followed by a matching layer that performs the semantic matching @xcite . such models have been applied on the fever dataset @xcite and have been shown to outperform the tf-idf approach @xcite . cell retrieval could be enhanced by utilizing pre-trained models over tables that outperform pretrained models over text @xcite . several systems that exploit table structure have been proposed for the task of fact-checking a claim over a table. however, not all of them can be used in every setting as each system holds different attributes and dimensions that need to be comprehended to better integrate them in certain tasks @xcite . for example, some systems such as scrutinizer @xcite are dependent on the table-schema and would not benefit in the feverous scenario where tables have varying schema. yet, other systems such as tapas @xcite ) are schema-independent and can be fine-tuned on the available feverous dataset to provide a score for a given table, thus acting as a table ranker module. some of these systems can be even directly trained on the data, to get domain-specific models. once the tables have been identified, a classifier can be trained on top of models that output cell representations of a table, such as tabert @xcite and turl @xcite , to extract the key cells for verdict prediction. also, fine-tuning the re-ranker models on the given data is a viable approach. finally, more sophisticated entity matching algorithms could have been explored to avoid the "exact match" issues that we observed with the baseline's entity matching @xcite .