Graph Reasoning with Context-Aware Linearization for Interpretable Fact Extraction and Verification (2021)

Abstract: This paper presents an end-to-end system for fact extraction and verification using textual and tabular evidence, the performance of which we demonstrate on the FEVEROUS dataset. We experiment with both a multi-task learning paradigm to jointly train a graph attention network for both the task of evidence extraction and veracity prediction, as well as a single objective graph model for solely learning veracity prediction and separate evidence extraction. In both instances, we employ a framework for per-cell linearization of tabular evidence, thus allowing us to treat evidence from tables as sequences. The templates we employ for linearizing tables capture the context as well as the content of table data. We furthermore provide a case study to show the interpretability our approach. Our best performing system achieves a FEVEROUS score of 0.23 and 53% label accuracy on the blind test data.

## introduction fact checking has become an increasingly important tool to combat misinformation. indeed the study of automated fact checking in nlp @xcite , in particular, has yielded a number of valuable insights in recent times. these include task formulations such as matching for discovering already fact-checked claims @xcite , identifying neural fake news @xcite , fact verification in scientific @xcite and public health @xcite domains, and end-to-end fact verification @xcite , which is the subject of the fever-ous benchmark dataset @xcite . a majority of automated fact checking studies only consider text as evidence for verifying claims. recently, there have been a number of works which look at fact-checking with structured and semistructured data, mainly in the form of tables and knowledge bases @xcite ) -but factchecking from both structured and unstructured data has been largely unexplored. given the sophistication in the presentation of fake news, it is important to develop fact checking tools for assessing evidence from a wide array of evidence sources in order to reach a more accurate verdict regarding the veracity of claims. in this work, we propose a graph-based representation that supports both textual and tabular evidence, thus addressing some of the key limitations of past architectures. this approach allows us to capture relations between evidence items as well as claim-evidence pairs, borrowing from the argumentation and argument mining literature @xcite , as well as argument modeling for fact verification @xcite . we experiment with two formulations for graph learning. for the first, we employ a multi-task learning paradigm to jointly train a graph attention network @xcite for both the task of evidence extraction -which we model as a node selection task -and a graph-level veracity prediction task. in the second, we explicitly separate the verification and extraction tasks, where standard semantic search is used for evidence extraction, and veracity prediction is treated as a graph-level classification problem. for veracity prediction we predict a label for each claim, one of supports, refutes, or not-enough-info (nei), which is conditioned on all relevant evidence, hence the intuition to frame veracity prediction as a graph-level prediction task. in both formulations, we employ context-aware table linearization templates to produce per-cell sequence representations of tabular evidence and thus construct evidence reasoning graphs where nodes have heterogeneous evidence types (i.e., representing sentences and tables on the same evidence reasoning graph). contributions. the three main contributions of the paper are summarized below: both the joint model and separately trained models exhibit a significant improvement over the feverous baseline, as well as significant improvements for label accuracy and evidence recall. our separated approach to fact extraction and verification achieves a feverous score of 0.23 and label accuracy of 53% on the blind test data. ## related work graph reasoning for fact verification. several works explore graph neural networks (gnn) for fact extraction and verification, both for finegrained evidence modelling @xcite and evidence aggregation for veracity prediction @xcite . furthermore, graph learning has also been leveraged to build fake news detection models which learn from evidence from different contexts; e.g., user-based and content-based data @xcite . there are also non-neural approaches to fake news detection with graphs @xcite . however, to the best of our knowledge, this work is the first to employ a graph structure to jointly reason over both text and tabular evidence data in both single task learning (stl) and multi-task learning (mtl) settings. table linearization. a number of approaches have been adopted in nlp for table linearization. for example, gupta et al. (2020) study natural language inference in the context of table linearizations, in particular they are interested to see if language models can infer entailment relations from table linearizations. the linearization approach employed by schlichtkrull et al. (2021) is also used for automated fact verification. however, they linearize tables row-and column-wise, whereas we focus on cells as evidence items in the feverous dataset are annotated at table-cell level. ## data analysis further to the feverous dataset statistics discussed by the task description paper @xcite , we perform our own data exploration. we present insights from our data analysis of the feverous dataset, which we use to inform system design choices. table types. wikipedia tables can be categorized into one of two classes: infoboxes and general tables. infoboxes are fixed format tables which typically appear in the top right-hand corner of a wikipedia article. general tables can convey a wider breadth of information (e.g., election results, sports match scores, the chronology of an event) and typically have more complex structures (e.g., multiple headers). list items can also be considered as a special subclass of tables, where the number of items is analogous to the number of columns and the nests of the list signify table rows. evidence types. the first observation we make is that, similar to the fever dataset @xcite , a sizeable portion of the training instances rely on evidence items which are extracted from the first few sentences of a wikipedia article. the most common evidence items are the first and second sentences in a wikipedia article, which appear in 36% and 18% of evidence sets, respectively. the four most frequent evidence cells all come from the first table, with 49% of first tables listed as evidence in the train and dev data being infoboxes. further, the vast majority of cell evidence items are non-header cells, but these only account for approximately 5.1% of tabular evidence in the train and dev datasets. a summary of these findings is provided in evidence item co-occurrences. we investigate the most common evidence pairs, both in individual evidence sets and also in the union of all evidence sets relating to a claim. the most common evidence pair in the training data is (sentence_0, sentence_1), which accounts for 3.2% of evidence co-occurrences. the most common sentence- nei label. lastly, we choose to explore instances of the nei class. we sample 100 instances of nei claims from the training data and note their qualitative attributes. we pay particular attention to this label as it is the least represented in the data. unlike the fever score, the feverous metric requires the correct evidence, as well as the label, to be supplied for an nei instance for credit to awarded. our analysis is summarized in we use other to categorize claims with a mutation not captured by one of these three categories. ## methods our proposed method for fact verification is an end-to-end system comprising three modules: an illustration of the complete pipeline is provided in figure 1 , and details of each processing stage are provided in the following sections. ## results we report the results of the entire fact extraction and verification pipeline, as well as the evaluation of the pipeline's performance for intermediate stages of the fact verification system, e.g., document retrieval and evidence selection. ## conclusion and future work in this work, we have demonstrated two novel approaches for fact extraction and verification that support both structured and unstructured evidence. these architectures were motivated by literature in argumentation, and also by the empirical analysis presented in section 3. our results show significant improvement over the shared task baseline for claim "in 2019, scomadi, a private limited company with limited liability, was bought by a british owner which changed scomadi's management structure." evidence scomadi_cell_0_0_1, scomadi_sentence_14, scomadi_sentence_15. predicted evidence (1) scomadi_cell_0_0_1 (0.1794), (2) scomadi_sentence_14 (0.1203), (3) scomadi_table_caption_0 (0.0871), (4) scomadi_cell_0_3_1 (0.0685), (5) scomadi_cell_0_7_1 (0.0561), (6) scomadi_cell_0_2_1 (0.0472) (7) scomadi_cell_0_8_1 (0.0405) (8) scomadi_sentence_15 (0.0360), (9) scomadi_sentence_11 (0.0324), (10) scomadi_sentence_0 (0.0292), (11) scomadi_cell_0_6_1 (0.0266), (12) scomadi_cell_0_5_1 (0.0243), (13) scomadi_cell_0_1_1 (0.0224), ( 14 ) scomadi_cell_0_4_1 (0.0208).