L ogical F act C hecker: Leveraging Logical Operations for Fact Checking with Graph Module Network (2020)

Abstract: Verifying the correctness of a textual statement requires not only semantic reasoning about the meaning of words, but also symbolic reasoning about logical operations like count, superlative, aggregation, etc. In this work, we propose LogicalFactChecker, a neural network approach capable of leveraging logical operations for fact checking. It achieves the state-of-the-art performance on TABFACT, a large-scale, benchmark dataset built for verifying a textual statement with semi-structured tables. This is achieved by a graph module network built upon the Transformer-based architecture. With a textual statement and a table as the input, LogicalFactChecker automatically derives a program (a.k.a. logical form) of the statement in a semantic parsing manner. A heterogeneous graph is then constructed to capture not only the structures of the table and the program, but also the connections between inputs with different modalities. Such a graph reveals the related contexts of each word in the statement, the table and the program. The graph is used to obtain graph-enhanced contextual representations of words in Transformer-based architecture. After that, a program-driven module network is further introduced to exploit the hierarchical structure of the program, where semantic compositionality is dynamically modeled along the program structure with a set of function-specific modules. Ablation experiments suggest that both the heterogeneous graph and the module network are important to obtain strong results.

## introduction fact checking for textual statements has emerged as an essential research topic recently because of the unprecedented amount of false news and rumors spreading through the internet @xcite year venue winner score 2005 arlandastad david patrick 272 2004 arlandastad matthew king 270 2003 falsterbo titch moore 273 2002 halmstad thomas besancenez 279 table statement in 2004, the score is less than 270. label refuted program 𝑙𝑒𝑠𝑠( ℎ𝑜𝑝( 𝑓𝑖𝑙𝑡𝑒𝑟_𝑒𝑞( 𝑌𝑒𝑎𝑟; 2004); 𝑆𝑐𝑜𝑟𝑒); 270) figure 1: an example of table-based fact checking. given a statement and a table as the input, the task is to predict the label. program reflects the underlying meaning of the statement, which should be considered for fact checking. @xcite @xcite @xcite . online misinformation may manipulate people's opinions and lead to significant influence on essential social events like political elections @xcite . in this work, we study fact checking, with the goal of automatically assessing the truthfulness of a textual statement. the majority of previous studies in fact checking mainly focused on making better use of the meaning of words, while rarely considered symbolic reasoning about logical operations (such as "count", "superlative", "aggregation"). however, modeling logical operations is an essential step towards the modeling of complex reasoning and semantic compositionality. figure 1 shows a motivating example for table-based fact checking, where the evidence used for verifying the statement comes from a semi-structured table. we can see that correctly verifying the statement "in 2004, the score is less than 270" requires a system to not only discover the connections between tokens in the statement and the table, but more importantly understand the meaning of logical operations and how they interact in a structural way to form a whole. under this table statement in 2004, the score is less than 270. refuted program 𝑙𝑒𝑠𝑠( ℎ𝑜𝑝 𝑓𝑖𝑙𝑡𝑒𝑟_𝑒𝑞 𝑌𝑒𝑎𝑟; 2004 ; 𝑆𝑐𝑜𝑟𝑒 ; 270) semantic parser graph construction contextual representations semantic compositionality year venue winner score 2005 arlandastad david patrick 272 2004 arlandastad matthew king 270 2003 falsterbo titch moore 273 2002 halmstad thomas besancenez 279 consideration, we use table-based fact checking as the testbed to investigate how to exploit logical operations in fact checking. in this paper, we present logicalfactchecker, a neural network approach that leverages logical operations for fact checking when semi-structured tables are given as evidence. taking a statement and a table as the input, it first derives a program, also known as the logical form, in a semantic parsing manner @xcite . then, our system builds a heterogeneous graph to capture the connections among the statement, the table and the program. such connections reflect the related context of each token in the graph, which are used to define attention masks in a transformer-based @xcite framework. the attention masks are used to learn graph-enhanced contextual representations of tokens foot_0 . we further develop a program-guided neural module network to capture the structural and compositional semantics of the program for semantic compositionality. @xcite . graph nodes, whose representations are computed using the contextual representations of their constituents, are considered as arguments, and logical operations are considered as modules to recursively produce representations of higher level nodes along the program. experiments show that our system outperforms previous systems and achieves the state-of-the-art verification accuracy. the contributions of this paper can be summarized as follows: ## task definition we study the task of table-based fact checking in this paper. this task is to assess the veracity of a statement when a table is given as evidence. specifically, we evaluate our system on tabfact @xcite , a large benchmark dataset for ## logicalfactchecker: methodology in this section, we present our approach logical-factchecker, which simultaneously considers the meaning of words, inner structure of tables and programs, and logical operations for fact-checking. one way to leverage program information is to use standard semantic parsing methods, where automatically generated programs are directly executed on tables to get results. however, tabfact does not provide annotated programs. this puts the problem in a weak-supervised learning setting, which is one of the major challenges in the semantic parsing field. in this work, we use programs in a soft way that programs are represented with neural modules to guide the reasoning process between a textual statement and a table. figure 2 gives an overview of our approach. with a statement and a corresponding table, our system begins with program generation, which synthesizes a program. then, we build a heterogeneous graph for capturing the inner structure of the input. with the constructed graph, we incorporate a graph-based attention mask into the transformer for learning graph-enhanced token representations. lastly, we learn the semantic compositionality by developing a program-guided neural module network and make the final prediction. this section is organized as follows. we first describe the format of the program ( § 3.1) for a more transparent illustration. after that, the graph construction approach ( § 3.2) is presented first, followed by a graph-enhanced contextual representation learning mechanism ( § 3.3). moreover, we introduce how to learn semantic compositionality over the program by neural module network ( § 3.4). at last, we describe how to synthesize programs by our semantic parsing model ( §3.5). ## experiments we evaluate our system on tabfact @xcite , a benchmark dataset for table-based fact checking. each instance in tabfact consists of a statement, a semi-structured wikipedia table and a label ("entailed" or "refuted") indicates whether the statement is supported by the table or not. the primary evaluation metric of tabfact is label accuracy. the statistics of tabfact are given in appendix a. detailed hyper-parameters for model training are given in appendix b for better reproducibility of experiments. we compare our system with following baselines, including the textual matching based baseline table-bert and semantic parsing based baseline lpa, both of which are developed by @xcite . ## conclusion in this paper, we present logicalfactchecker, a neural network based approach that considers logical operations for fact checking. we evaluate our system on tabfact, a large-scale benchmark dataset for verifying textual statements over semi-structured tables, and demonstrate that our approach achieves the state-of-the-art performance. logicalfactchecker has a sequence-to-action semantic parser for generating programs, and builds a heterogeneous graph to capture the connections among statements, tables, and programs. we utilize the graph information with two mechanisms, including a mechanism to learn graph-enhanced contextual representations of tokens with graphbased attention mask matrix, and a neural module network which learns semantic compositionality in a bottom-up manner with a fixed set of modules. we find that both graph-based mechanisms are beneficial to the performance, and our sequenceto-action semantic parser is capable of generating semantic-consistent programs.