Program Enhanced Fact Verification with Verbalization and Graph Attention Network (2020)

Abstract: Performing fact verification based on structured data is important for many real-life applications and is a challenging research problem, particularly when it involves both symbolic operations and informal inference based on language understanding. In this paper, we present a Program-enhanced Verbalization and Graph Attention Network (ProgVGAT) to integrate programs and execution into textual inference models. Specifically, a verbalization with program execution model is proposed to accumulate evidences that are embedded in operations over the tables. Built on that, we construct the graph attention verification networks, which are designed to fuse different sources of evidences from verbalized program execution, program structures, and the original statements and tables, to make the final verification decision. To support the above framework, we propose a program selection module optimized with a new training strategy based on margin loss, to produce more accurate programs, which is shown to be effective in enhancing the final verification results. Experimental results show that the proposed framework achieves the new state-of-the-art performance, a 74.4% accuracy, on the benchmark dataset TABFACT.

## introduction with the overwhelming information available on the internet, fact verification has become crucial for many applications such as detecting fake news, rumors, and political deception @xcite @xcite @xcite , among others. existing research has mainly focused on collecting year tournaments played avg. score scoring rank 2007 22 72.46 81 2008 29 71.65 22 2009 25 71.90 34 2010 18 73.42 92 2011 11 74.42 125 table with title 'ji-young oh' statement label program ji-young oh played more tournament in 2008 than any other year. entailed eq { max { all_rows ; tournaments played } ; hop { filter_eq { all_rows ; year ; 2008 } ; tournaments played } } = true and processing evidences from unstructured text data @xcite @xcite , which is only one type of data where important facts exist. structured and semi-structured data, e.g., tables in relational databases or in the html format is also ubiquitous. performing fact validation based on structured data is important yet challenging and further study is highly desirable. fig. 1 depicts a simplified example in which systems are expected to decide whether the facts in the table support the natural language statement. in addition to its importance in applications, the task presents research challenges of fundamental interests-the problem inherently involves both informal inference based on language understanding @xcite @xcite @xcite and symbolic operations such as mathematical operations (e.g., count and max). recently, pre-trained language models such as bert @xcite have shown superior performances in natural language inference by leveraging knowledge from large text datasets and can capture complicated semantic and syntactic information among premises and hypotheses @xcite @xcite . however, such methods tend to fail when verification requires the joint modelling of both symbolic operations and language inference @xcite such as the case depicted in fig. 1 . to effectively enable symbolic operations and integrate them into language-based inference models, we propose a framework centered around programs, i.e., logical forms that can be executed to find evidences from structured data. our model starts with a program selection module, for which we propose a new training strategy based on margin loss to find programs that can accurately extract supporting facts from tables. to bridge the semantic gap between structured programs and tables as well as to leverage the structures of programs, we propose a novel model based on verbalization with program execution. the verbalization algorithm interweaves with program execution in order to accumulate evidences inherently embedded in operations, and the algorithm recursively converts executed operations in programs into natural language sentences. built on that, we propose graph-based verification network to fuse different sources of evidences from verbalized program execution, together with the original statements and tables, to support the final verification decision. we conduct experiments on the recently proposed large scale benchmark dataset tab-fact @xcite . experimental results show that our proposed framework achieves new state-of-the-art performance, an accuracy of 74.4%, substantially improving the previously reported best performance with the accuracy of 71.7%. our detailed analysis shows the effectiveness of verbalization and graph-based verification network in utilizing programs to achieve the improvement. the analysis also demonstrates that the program selection optimized with the proposed training strategy based on margin loss effectively improves the final verification results. ## related work fact verification. existing work on fact verification is mainly based on collecting and using evidences from unstructured text data @xcite @xcite . fever @xcite is one of the most influential benchmark datasets built to evaluate systems in checking claims by retrieving wikipedia articles and extracting evidence sentences. recent proposed fever 2.0 @xcite has a more challenging dataset to verify factoid claims and an adversarial attack task. some previous models are developed on the official baseline @xcite with three step pipeline @xcite for fact verification @xcite @xcite . others formulates fact verification as graph reasoning @xcite . natural language inference (nli) task is also a verification problem which is fully based on unstructured text data @xcite @xcite @xcite @xcite . neural models proposed for nli have been shown to be effective @xcite @xcite , including models incorporating external knowledge @xcite . our work focuses on fact verification based on structured tables @xcite . for verification performed on structured data, @xcite propose a typical baseline (table-bert), which is a semantic matching model taking a linearized table t and statement s as input and employs bert for verification. the other model (lpa) proposed in @xcite uses transformer blocks to compute semantic similarity between a statement and program. a contemporaneous work @xcite proposes logical-factchecker aiming to leverage programs for fact verification. logicalfactchecker utilizes inherent structures of programs to prune irrelevant information in evidence tables and modularize symbolic operations with module networks. different from theirs, our proposed framework verbalizes the accumulated evidences from program execution to support the final verification decision with graph attention networks. ## model we present the proposed framework (progvgat), which centers around programs and execution to integrate symbolic operations for fact verification. fig. 2 depicts the overall architecture. this section is organized as follows. we first introduce the task formulation along with program representations in sec. 3.1. then, we describe in sec. 3.2 the program selection module that aims to obtain a semantically relevant program for verification. built on that, we present our proposed verbalization algorithm and graph attention network to dynamically accumulate evidences embedded in symbolic operations of programs for final verification in sec. 3.3 and sec. 3.4. ## experiment setup data and evaluation metric as discussed above, although (semi-)structured and unstructured text data are ubiquitous in our daily life, performing fact verification across these different formats is relatively new. we conduct our experiments on recently released large-scale dataset tab-fact @xcite . tabfact contains 92,283, 12792, and 12779 table-statement pairs for training, validation and testing respectively. verification on some statements requires higher-order semantics such as argmax, the test set is further split into a simple and complex subset according to verification difficulty. a small subset is provided in which the human upper-bound performance is given. following the existing work, we use accuracy as the evaluation metric. detailed data statistics are listed in appendix a.1. training details for parameters in all bert models, the hidden size is set to 768, we use adam optimizer @xcite with learning rate 2e-5, warmup step 3000, dropout 0.1. for parameters in graph attention network, the hidden feature dimensions f and l are all set to 768. all codes are implemented with pytorch @xcite . all hyper-parameters are decided according to the validation performance. ## conclusions in this paper, we propose a framework centered around programs and execution to provide symbolic manipulations for