Logic-level Evidence Retrieval and Graph-based Verification Network for Table-based Fact Verification (2021)

Abstract: Table-based fact verification task aims to verify whether the given statement is supported by the given semi-structured table. Symbolic reasoning with logical operations plays a crucial role in this task. Existing methods leverage programs that contain rich logical information to enhance the verification process. However, due to the lack of fully supervised signals in the program generation process, spurious programs can be derived and employed, which leads to the inability of the model to catch helpful logical operations. To address the aforementioned problems, in this work, we formulate the table-based fact verification task as an evidence retrieval and reasoning framework, proposing the Logic-level Evidence Retrieval and Graph-based Verification network (LERGV). Specifically, we first retrieve logic-level program-like evidence from the given table and statement as supplementary evidence for the table. After that, we construct a logic-level graph to capture the logical relations between entities and functions in the retrieved evidence, and design a graph-based verification network to perform logic-level graph-based reasoning based on the constructed graph to classify the final entailment relation. Experimental results on the large-scale benchmark TABFACT show the effectiveness of the proposed approach.

## introduction there are a large number of semi-structured tables on the internet. how to perform reasoning over semi-structured tables is crucial for people to understand different types of information in the real world. and this direction has spawned many tasks. among these tasks, table-based fact verification task has recently received a lot of attention, which is important for many applications, such as fake news detection, scientific paper understanding @xcite , etc. this task aims to verify the correctness of the given statement by the given table, which requires both linguistic reasoning and symbolic reasoning @xcite . symbolic reasoning with logical operations like "count" and "argmax" plays an important role in the table-based fact verification task. figure 1 shows an example. ideally, to verify the correctness of such statements, logical operations provide strong hints to classify the entailment relation. therefore, how to utilize such logical operations is crucial in this task. program is a kind of logic form derived from tables, which contains rich logical operations. following @xcite , existing methods @xcite @xcite mostly use programs to perform symbolic reasoning. specifically, they derive one or sev-eral programs with a semantic parser based on the given table and statement, and then leverage the obtained programs to enhance the verification process. however, these models may select spurious programs (i.e. wrong programs with correct returned labels) because there are weak supervised signals in the semantic parsing process. consequently, label-consistent programs (the programs whose execution results are consistent with the ground-truth verification labels) tend to be selected instead of semantic-consistent ones. as a result, the obtained programs will not contain helpful logical operations, thus cannot benefit the verification of the correctness of the statement. ideally, a natural way of leveraging programs is to regard them as supplementary evidence for tables. in other words, some of the programs contain the necessary information with logical operations that summarize or describe the facts observed from the table. so that we can leverage the information conveyed by the programs to help better capture the facts of the table and then classify the entailment relation. based on the above considerations, in this work, we formulate table-based fact verification as an evidence retrieval and reasoning pipeline, proposing a logic-level evidence retrieval and graph-based verification method, named lergv. firstly, instead of deriving label-consistent programs with a semantic parser, we propose a rule-based method to retrieve valuable logic-level program-like evidence from the table and statement to avoid the issue of spurious programs. then we leverage the structure of the programs to construct a logic-level graph with the above evidence to catch the logical relations between entities (such as "number in fleet") and functions (such as "min") in the programs. finally, a graph-based verification network is proposed to reason over the constructed graph to perform logic-level reasoning, which takes advantage of the combination of linguistic reasoning and symbolic reasoning to make the final prediction. we conduct experiments on a large-scale benchmark dataset tabfact @xcite . experimental results show that our model surpasses all baseline systems with a considerable margin. the main contributions of this paper are three-fold: chassis manufacturer chassis model … man hocl-nl mercedes benz o405nh mitsubishi fuso rosa scania k280ub scania k320ua table statement program synthesis program-like evidence evidence retrieval valid programs fact verification entailed / refuted figure 2: pipeline for table-based fact verification task. given a table and a statement, the proposed pipeline can be divided into program synthesis module, evidence retrieval module ( §3.1), and fact verification module ( §3.2 and §3.3). in this paper, we study the task of table-based fact verification. given a table t with r rows and c columns and a statement s, the goal is to verify the correctness of the given statement by the given table with the label entailed or refuted. table is the only evidence in the original task setting. however, we believe that the additional evidence that contains logical operations is helpful to classify the entailment relation. in this study, we employ such evidence (denote as "evidence" in the rest of the paper) that in the form of programs. in particular, program is a kind of lisplike logical form that follows a grammar with over 50 pre-defined functions @xcite . every program is tree-structured, consisting of functions as parent nodes and their arguments as children nodes, where leaf nodes represent arguments, namely entities linked to the table or the statement, and non-leaf nodes represent functions, such as "min", "count", "argmax", etc. the dotted boxes in figure 3 show the structure of programs. to better take advantage of program-like evidence, in this work, we formulate the table-based fact verification task as an evidence retrieval and reasoning pipeline that consists of three main components, program synthesis, evidence retrieval, and fact verification modules. figure 2 shows the overview of our proposed approach. given the table and the statement, the program synthesis module first synthesizes all possible programs with valid combinations. then the evidence retrieval module selects, decomposes, and filters over the programs to obtain valuable logic-level evidence as supplementary information to the original table. finally, a fact verification model takes the input of the table, statement, and obtained evidence to predict whether the statement is supported by the given table. following previous work @xcite @xcite , we perform program synthesis with the latent program search algorithm (lpa) @xcite , followed by an evidence retrieval module and a fact verification module. we will present above modules in § 3. ## methodology we propose a logic-level evidence retrieval and graph-based verification network (lergv) for the table-based fact verification task. given a table and a statement, lergv works as follows. first, we start with the latent program algorithm @xcite to synthesize programs and then use a rule-based retrieval approach to select, decompose, and filter among all synthesized programs to obtain valuable logic-level evidence ( § 3.1). after that, we construct a graph based on the retrieved evidence and initialize graph node representations from a pre-trained language model ( § 3.2). finally, we propose a graph-based verification network centered around the obtained evidence to perform graph-based reasoning to predict the final verification result ( § 3.3). ## conclusion in this study, we focus on the table-based fact verification task and propose lergv that leverages logic-level program-like evidence to perform fact verification over tables. the main idea is that we formulate the task as an evidence retrieval and graph-based reasoning pipeline, and treat logiclevel evidence as supplementary information for tables to avoid the issue of spurious programs. specifically, we first apply a rule-based evidence retrieval approach to select, decompose, and filter among synthesized programs to obtain valuable logic-level program-like evidence. then, we construct a graph based on the retrieved evidence to catch logical relations between entities and functions. finally, we propose a graph-based verification network to perform reasoning over the constructed graph to combine linguistic reasoning and symbolic reasoning effectively. experimental results on tabfact illustrate that our model outperforms all the baseline systems and achieves competitive results.