Table-based Fact Verification With Salience-aware Learning (2021)

Abstract: Tables provide valuable knowledge that can be used to verify textual statements. While a number of works have considered table-based fact verification, direct alignments of tabular data with tokens in textual statements are rarely available. Moreover, training a generalized fact verification model requires abundant labeled training data. In this paper, we propose a novel system to address these problems. Inspired by counterfactual causality, our system identifies token-level salience in the statement with probing-based salience estimation. Salience estimation allows enhanced learning of fact verification from two perspectives. From one perspective, our system conducts masked salient token prediction to enhance the model for alignment and reasoning between the table and the statement. From the other perspective, our system applies salience-aware data augmentation to generate a more diverse set of training instances by replacing non-salient terms. Experimental results on TabFact show the effective improvement by the proposed salience-aware learning techniques, leading to the new SOTA performance on the benchmark.

## introduction fact verification, the problem of determining whether a statement is entailed or refuted by evidence, has quickly become a critical problem in nlp to combat information pollution @xcite @xcite @xcite . successful fact verification enables downstream tasks such as misinformation detection, fake news identification, factual error correction, and deceptive opinion detection @xcite @xcite . recently, table-based fact verification @xcite @xcite figure 1 : an example of table-based fact verification, with green for entailed statements and red for refuted statements. alignment and reasoning are essential for both table-based fact verification and masked salient token prediction (e.g. "eagles"). token replacement may lead to similar (e.g. "'s" to "team") or different (e.g. "eagles" to "bearcats") statements. however, to provide a reliable solution to the table-based fact verification task, several critical challenges are still overlooked by prior studies. one challenge is to effectively provide connections among components of the statement and substructures of the table, and accordingly conduct the in-ference. being unaware of such fine-grained connections and logical relations could raise the risk of misalignment, incorrect reasoning and ignoring salient components of a statement, and therefore leads to incorrect verification results. for example, to verify the statement in fig. 1 , the model should implicitly or explicitly infer all the five arrows accurately. although some works have tried to perform token-level interactions and generate logical programs to connect statements and tables and conduct logical reasoning @xcite , the supervision signals to guide the learning process are typically sparse. another challenge is that training a well-generalized fact verification model non-trivially requires abundant labeled training data. limited training data can only cover limited statement patterns and hinder robustness and generalizability of model inference. previous works either trained on limited data @xcite or augment training data with specific statement generation templates @xcite ). yet, in real-world scenarios, statements and evidences can be presented in very diverse ways, and such diversity is difficult to be comprehensively captured by specific templates. to this end, we propose a novel salience-aware learning system for table-based fact verification. starting from a tapas @xcite ) language model fine-tuned on the tabfact dataset, our system identifies salient and non-salient tokens in statements with a probing-based salience estimation method inspired by counterfactual causality @xcite ( §3.2). then, the system leverages the estimated salience information from two perspectives. from one perspective, to enhance the model for capturing fine-grained connections and supporting the reasoning between statements and tables, the system conducts masked salient token prediction as an auxiliary task ( §3.3). more specifically, this task is to predict the masked salient token in an entailed statement given the corresponding table by reusing the embedding layer of tapas as a language model head. the fact verification task can receive indirect supervision from the auxiliary task, as both of them requires table-text alignment and logical reasoning. from the other perspective, to improve the model robustness, instead of using templates for statement augmentation like prior work @xcite , we develop a salienceaware data augmentation technique ( §3.4). intuitively, replacing non-salient tokens provides un-seen statements while preserving the meaning and correctness of the original statement. this strategy enhances the size and comprehensiveness of the training data and further complements training with more supervision signals. the main contributions of this paper are threefold. first, we propose a probing-based salience estimation method to evaluate the importance of each token in a statement according to the counterfactual causality theory. second, we propose a novel salience-aware learning system that helps the fact verification model to find the connections between the table and the statement, and enhance the inference ability of the model with the auxiliary task of masked salient token prediction. third, to complement with insufficient training signals and improve the model robustness on heterogeneous statements, we incorporate a probabilistic data augmentation method driven by non-salient tokens. we evaluate our system based on the tabfact benchmark, which shows promising performance on this task and drastically outperforms prior methods. detailed analysis demonstrates the effectiveness and essentially of both masked salient token prediction and salience-aware data augmentation techniques for the improved performance. ## related work in this section, we provide a selected summary for two related research topics. ## experiment in this section, we conduct experiments on the tab-fact dataset. we first introduce the dataset, a series of recent baselines and details of our method ( §4.1). then we show the overall performance and ablation results ( §4.2). we also provide case studies for in-depth analysis ( §4.3). ## conclusion in this paper, we proposed a novel system for table-based fact verification. our system employs salience-aware learning and introduce complementary supervision signals by leveraging both salience and non-salient tokens from different perspectives. the system consists of three key techniques, including probing-based salience estimation, masked salient token prediction and salience-aware data augmentation. experiments on the tabfact benchmark show that our system leads to significant improvements over the current sota systems. for future work, we plan to extend salience-aware learning to other nlu tasks, including nli @xcite and tabular qa @xcite . applying the idea of salience estimation to nlg tasks, such as controlled table -to-text generation @xcite and paraphrasing @xcite , is another meaningful direction.