Enhancing Structured Evidence Extraction for Fact Verification (2023)

Abstract: Open-domain fact verification is the task of verifying claims in natural language texts against extracted evidence. FEVEROUS is a benchmark that requires extracting and integrating both unstructured and structured evidence to verify a given claim. Previous models suffer from low recall of structured evidence extraction, i.e., table extraction and cell selection. In this paper, we propose a simple but effective method to enhance the extraction of structured evidence by leveraging the row and column semantics of tables. Our method comprises two components: (i) a coarse-grained table extraction module that selects tables based on rows and columns relevant to the claim and (ii) a fine-grained cell selection graph that combines both formats of evidence and enables multi-hop and numerical reasoning. We evaluate our method on FEVEROUS and achieve an evidence recall of 60.01% on the test set, which is 6.14% higher than the previous state-of-the-art performance. Our results demonstrate that our method can extract tables and select cells effectively, and provide better evidence sets for verdict prediction. Our code is released at https://github.com/WilliamZR/see-st

## introduction open-domain fact verification is the task of verifying a factual claim based on evidence extracted from a knowledge base @xcite . a complete evidence set provides necessary information for a verification module to support or refute the claim. in the real world, fact-checkers could use both unstructured and structured data as evidence. tables are a structured form of evidence that provides an organized representation of information that facilitates effortless comparison and crossreferencing. by presenting information in cells, tables enable efficient look-up of individual values, while the arrangement of rows and columns allows for the extraction of high-order semantics through various operations, such as comparisons, filtering, arithmetic calculations, and the identification of minimum and maximum values. the two-dimension structure and high-order semantics of tables make them more difficult to extract compared with texts. regarding the previous method to extract structured evidence. @xcite treats tables as text sequences neglecting the table structure itself and extracts them only based on lexical matching. this approach is also used in many following works @xcite @xcite . consequently, these methods struggle to extract tables with little word overlap with the claim. alternatively, @xcite and @xcite convert each table cell into a text sequence and perform cell retrieval directly. @xcite construct a reasoning graph of sentences and cells to extract evidence. yet these methods overlook the context within the table and fail to capture high-order semantics from rows and columns. figure 1 presents an example from fever-ous @xcite , a benchmark on opendomain fact verification using both sentences and table cells as evidence. the verification model is expected to extract both unstructured and structured evidence to verify the claim. the structured table can be interpreted in several ways for the model: a complete table, a combination of rows or columns, or a composition of isolated cells. the entire table encompasses a substantial amount of information. during table extraction, the site and attendance columns are irrelevant and only introduce confusion for both lexical and semantic methods @xcite . cells typically contain short phrases and numbers, which are often insufficient for conveying comprehensive information. in contrast, rows and columns possess a similar volume of information to the claim, thereby exhibiting comparable semantic granularity. the extraction process should prioritize the information from the 3rd row and the result column. during the cell selection step, the selection module must capture the high-order semantics in the result column to verify the statement in this paper, we propose structured evidence extraction with row and column semantics of tables (see-st), a novel approach to enhance both table extraction and cell selection modules for fact verification by leveraging the semantics of rows and columns. this method aligns the semantic granularity of structured evidence with claim and sentence evidence, enabling more precise evidence extraction. see-st starts by extracting tables from retrieved wikipedia pages and focuses on identifying the most relevant rows and columns for the claim, thereby reducing confusion arising from irrelevant cells. since a cell represents the intersection of a row and a column, its selection can be conducted by analyzing both dimensions. we integrate rows and columns within a graph neural network to facilitate fine-grained cell selection while preserving high-order semantics. additionally, we incorporate extracted sentence evidence into the graph to merge information from evidence of both formats at a similar semantic granularity. our contributions can be summarized as follows: (i) we propose to leverage the row and column semantics to enhance table extraction and cell selection. (ii) we design an evidence graph for cell selection. it enables rows and columns to interact with sentences at comparable semantic granularity while still maintaining high-order semantics from table evidence. (iii) our proposed method see-st achieves state-of-the-art performance in evidence extraction on the feverous dataset, with better evidence sets contributing to the increased accuracy of the verification module. ## our model feverous is a benchmark on open-domain fact verification @xcite . the task is to verify a claim c based on wikipedia. we follow the three-step pipeline in open-domain fact verification @xcite . the three steps are document retrieval, evidence extraction, and verdict prediction. the first step retrieves n p relevant pages from the wikipedia dump. then the second step extracts sentence evidence s and cell evidence c from the retrieved pages. at last, the model predicts the veracity label ŷ of the claim based on sentence and cell evidence. see-st enhances the extraction of structured evidence in the second step of the pipeline. it operates on the same conceptual framework, utilizing row and column semantics to enhance evidence extraction (figure 2 ). see-st performs structured evidence extraction in two stages: (i) coarse-grained table extraction that retrieves top n t tables for the second step. it scores relevance between rows or columns with the claim and ranks tables accordingly and (ii) fine-grained cell selection that retrieves cell evidence from complex tables. it constructs a graph neural network of rows, columns, and sentences at a comparable semantic granularity. it utilizes a graph attention network to enable delicate information integration between different pieces of evidence. the cells are selected as the intersection of rows and columns. at the verification step, we utilize dcuf @xcite , a method that converts evidence into dual-channel encodings to verify the claim. ## experiments we utilize feverous as the test bed for our approach, as it is, to our knowledge, the only opendomain fact verification benchmark that combines both unstructured and structured evidence. fever-ous aims to accomplish two objectives: first, to extract sentence and table cell evidence from english wikipedia, which contains over 95.6 million sentences and 11.8 million tables; and second, to predict the veracity label ŷ of a given claim. the possible veracity labels include supports, refutes, and not enough info (nei). some claims can be verified using either sentences or tables alone, while approximately 59% of instances need structured evidence for claim verification. 27% of the instances need to combine evidence from unstructured and structured evidence to verify. the quantitative characteristics of the benchmark are presented in ## case study in this section, we demonstrate two cases for evidence extraction in figure 4 . for the claim on 1991 waterford city council election, our table extraction module successfully extracts the table of election statistics from the wikipedia dump. the main challenge for this case is numerical reasoning. verifying ... six parties all of which have at least two seats requires high-order semantics in seats column. such semantics is maintained in coarse-grained table extraction and fine-grained cell selection. see-st extracts cells in party and seats columns as evidence. it shows the numerical reasoning ability of see-st. as for the evidence extraction for the claim of the castle of iron, it demonstrates the ability of see-st to perform multi-hop reasoning. the claim states that ...was published by a large-scale publishing company in new york, and is refuted by sentence evidence gnome press was an american small-press publishing company.... however, the name of the publisher is not mentioned in the claim. it is the structured evidence that links the claim with the sentence evidence. the design of our evidence graph could enable interaction between sentence and table cells to achieve multi-hop reasoning over different formats of evidence. ## related work open-domain fact verification extracts evidence from a knowledge base (kb) to verify the claim. in the real world, unstructured and structured evidence can be used to verify the claim. fever @xcite and hover @xcite focus on extracting unstructured evidence i.e. sentences from the wikipedia dump. and the research of structured evidence mainly focuses on tables. tabfact @xcite and infotabs @xcite verify claims with tables and infoboxes from wikipedia. sem-tab-fact @xcite and scitab @xcite are constructed on claims and tables from scientific papers. yet these datasets all provide the table evidence for verification as in a closeddomain setting. to our knowledge, feverous is the only open-domain fact verification dataset that considers table as evidence @xcite . therefore, our work uses feverous as a testbed for table extraction and cell selection. as for table extraction, the lexical matching approach is widely used in fact verification @xcite @xcite . @xcite encodes tables with bert and concatenate embedding with structured features to rank tables. tapas enhances table encoding with additional table-aware positional embeddings @xcite . @xcite also leverages represen-tations of row and column to rank tables but overlooks the context of other cells. tabert @xcite utilizes a content-snapshot mechanism to only encode the relevant rows thus may lose the high-order semantics of columns. @xcite converts each cell into a text sequence to select cell evidence, bypassing the table extraction step. acharya (2021) selects cells through dependency parsing. @xcite combines single-cell nli task with cell-wise relevance to extract evidence at cell-level semantic granularity. these methods are performed at the semantic granularity of cells. approaches in question answering have proposed to select cells as the intersection of rows and columns @xcite . all these above methods do not consider the interaction between evidence. the idea of evidence graph has been explored in previous works. @xcite linearizes cells into sentences to construct a fully connected evidence graph with texts. unifee @xcite proposes to use column nodes as intermediates between sentence-level and cell-level evidence graphs. our method is more computationally efficient with only one evidence graph. our design enables rows and columns to interact with sentences at comparable semantic granularity while still maintaining high-order semantics from tables. ## conclusions this paper presents see-st, a novel method for enhancing the extraction of structured evidence. by leveraging both row and column semantics, see-st can operate with a comparable evidence granularity, enabling both coarse-grained table extraction and fine-grained cell selection. experimental results demonstrate the effectiveness of see-st, as it significantly improves the recall of structured evidence compared to existing methods in fact verification. moreover, see-st outperforms state-ofthe-art methods in evidence recall and feverous score on feverous benchmark. these findings underline the impact of see-st in enhancing structured evidence extraction.