U nif EE : Unified Evidence Extraction for Fact Verification (2023)

Abstract: FEVEROUS is a fact extraction and verification task that requires systems to extract evidence of both sentences and table cells from a Wikipedia dump, then predict the veracity of the given claim accordingly. Existing works extract evidence in the two formats separately, ignoring potential connections between them. In this paper, we propose a Unified Evidence Extraction model (UnifEE), which uses a mixed evidence graph to extract the evidence in both formats. With the carefully-designed unified evidence graph, UnifEE allows evidence interactions among all candidates in both formats at similar granularity. Experiments show that, with information aggregated from related evidence candidates in the fusion graph, UnifEE can make better decisions about which evidence should be kept, especially for claims requiring multi-hop reasoning or a combination of tables and texts. Thus it outperforms all previous evidence extraction methods and brings significant improvement in the subsequent claim verification step.

## introduction feverous @xcite is a fact extraction and verification task over structured and unstructured information. models should extract finegrained evidence in two formats, namely, sentences and table cells, from a wikipedia dump and verify the given claim accordingly. many previous works focus on the claim verification procedure. they propose various methods to fuse evidence in different formats while leaving the evidence extraction within each format separately @xcite @xcite . for those efforts on evidence extraction, @xcite use a neural re-ranker to refine page retrieval. bouziane claim: john laurie has collaborated with many directors such as alfred hitchcock, michael powell, and laurence olivier, but his most famous role is private frazer in a film written and directed by alfred hitchcock. ## our model we take the widely-adopted three-step pipeline for feverous @xcite , which consists of retrieving pages from wikipedia dump, extracting sentences and table cells as evidence from the pages, and predicting the veracity label of the given claim based on the evidence set. in this work, we focus on the second step, i.e., evidence extraction. we use the drqa @xcite based document retrieval method and the dual-channel verification method proposed by @xcite for the remaining two steps. formally, during evidence extraction, n ′ s sentences and n t tables are extracted from n p retrieved passages by a pair-wise classification model firstly. this pre-processing step narrows the search space of sentences and table cells from thousands to a few dozens, which makes the mixed graph lighter. then, n c cells and n s sentences are extracted by our proposed unified evidence extraction model (unifee). these fine-grained evidence pieces are used for fact verification later. in unifee, we design a mixed evidence graph ( §2.1) to organize evidence of two formats. then, a graph neural network is applied to the graph to accumulate contextualized information from all evidence candidates with similar granularity, and identify useful evidence from a unified perspective ( §2.2). ## related works in this work, we focus on the evidence extraction step of the fact verification based on table and text, i.e. the feverous dataset @xcite . other fact verification datasets only concentrate on unstructured data @xcite or a single given table @xcite @xcite . these datasets do not require fine-grained evidence extraction or only extract evidence of a single format. for evidence sentence extraction, most previous works rank claim-sentence pairs with esim @xcite or plm @xcite . there are also works using the multi-turn retrieving method for multi-hop evidence @xcite or training the evidence extraction and claim verification model jointly to relieve the error propagation @xcite . for cell evidence extraction, sem-tab-fact @xcite and feverous @xcite are the only two verification datasets requiring finegrained cell selection to our best known. @xcite propose to parse the claim into logical forms and identify cells with string matching and dependency parsing. @xcite use an ensemble of bert fine-tuned on a single-cell nli task and a cell-wise similarity algorithm to capture the additional relationship. many previous works on feverous flatten each table to a sequence and perform binary sequence labeling to select cells @xcite . however, these models extract evidence according to every single sentence/cell/table, they neglect the connections of evidence candidates. meanwhile, they are not tailored for evidence extraction of two formats and cross-format evidence interactions, which are the main challenges of our task. many efforts aim at fusing evidence of two formats in feverous. @xcite and @xcite convert the extracted table cells to strings with human-designed templates to get a unified evidence set of sentences. bouziane et al. (2021 for works fusing evidence in the extraction step, @xcite and @xcite convert cells to strings and treat all evidence candidates as sentences for evidence selection. they neglect the delicate structure of nested tables and lose much context information after the format conversion. our proposed unifee introduces column nodes and layer-aware table representations, which can gather complex table information and facilitate cross-format interactions in a similar granularity in the evidence extraction step. ## conclusion in this paper, we propose a unified evidence extraction model (unifee) for fact extraction over structured and unstructured data. unifee adopts a mixed evidence graph to encourage evidence interactions among evidence candidates in both formats without manually designed conversion rules. experiments on the feverous benchmark demonstrate that, with unifee, a simple claim verification model outperforms previous sota results by a large margin. further analysis shows that unifee enhances the contextualized modeling of cells in complicated nested tables, thus largely improving the evidence extraction performance.