PASTA : Table-Operations Aware Fact Verification via Sentence-Table Cloze Pre-training (2022)

Abstract: Fact verification has attracted a lot of attention recently, e.g., in journalism, marketing, and policymaking, as misinformation and dis- information can sway one’s opinion and affect one’s actions. While fact-checking is a hard task in general, in many cases, false statements can be easily debunked based on analytics over tables with reliable information. Hence, table- based fact verification has recently emerged as an important and growing research area. Yet, progress has been limited due to the lack of datasets that can be used to pre-train language models (LMs) to be aware of common table operations, such as aggregating a column or comparing tuples. To bridge this gap, this paper introduces PASTA for table-based fact verification via pre-training with synthesized sentence–table cloze questions. In particular, we design six types of common sentence–table cloze tasks, including Filter, Aggregation, Superlative, Comparative, Ordinal, and Unique, based on which we synthesize a large corpus consisting of 1.2 million sentence–table pairs from WikiTables. PASTA uses a recent pre-trained LM, DeBERTaV3, and further pre- trains it on our corpus. Our experimental results show that PASTA achieves new state-of-the-art (SOTA) performance on two table-based fact verification datasets TabFact and SEM-TAB- FACTS. In particular, on the complex set of TabFact, which contains multiple operations, PASTA largely outperforms previous SOTA by 4.7% (85.6% vs. 80.9%), and the gap between PASTA and human performance on the small test set is narrowed to just 1.5% (90.6% vs. 92.1%).

## introduction fact verification, which checks the factuality of a statement, is crucial for journalism @xcite , and is increasingly being applied in other fields @xcite . according to duke reporters' lab, there are 300+ active certified fact-checking organizations worldwide. 2 automatic and explainable approaches, a.k.a. reference-based approaches, are widely used to assist fact-checkers. they verify the input statement against a trusted source, such as relevant passages from wikipedia @xcite @xcite . recently, table-based fact verification has been extensively studied @xcite @xcite due to the wide availability of tabular data. evidently, performing fact verification over tables requires the ability to reason about tablebased operations, such as aggregating the values in a column or comparing tuples. for example, for the statement s 1 in figure 1 , it is desirable to reason about the operation over table t that compares the viewers of night moves with 3.61 to determine whether s 1 is entailed or refuted by t . most previous work @xcite @xcite leverages pre-trained language models (lms) @xcite , which are originally designed for unstructured data, and have a key limitation of overlooking such operations. some approaches @xcite attempt to explicitly capture the operations by generating a logical form (e.g., a tree) containing the operations from the statement via semantic parsing techniques. however, such approaches face the problem of "spurious programs" @xcite , due to weak supervision signals in semantic parsing. to address the above issues, we propose pasta, a table-operations aware approach. instead of relying on semantic parsing, pasta captures tablebased operations by designing a novel sentencetable cloze pre-training strategy that better guides lms to reason about table-based operations. we tackle two challenges for pre-training lms towards supporting table-based fact verification: to address challenge 1, pasta automatically synthesizes sentence-table cloze questions. it first synthesizes operations from tables, and then generates cloze tasks by masking the key tokens corresponding to the table-based operations, e.g., "more than" and "sum of " in figure 1 . then, lms are pre-trained to predict the masked operation-aware tokens based on the tables. regarding challenge 2, pasta uses a large table collection, wikitables @xcite , and for each table, it synthesizes a diverse set of cloze tasks with six types of table-based operations, including filter, aggregation, superlative, comparative, ordinal, and unique. for implementation, pasta uses a recent pretrained lm, debertav3 @xcite with better positional encoding of the input. to cope with the limited input length of debertav3, we introduce a select-then-rank strategy for large tables, which further improves the performance. in sum, we make the following contributions. the experimental results show that pasta achieves new state-of-the-art (sota) results on the two datasets. in particular, on the complex set of tabfact that contains multiple operations, pasta outperforms the previous sota by 4.7 points (85.6% vs. 80.9%), and the gap between pasta and human performance on the small test set is narrowed to 1.5 points (90.6% vs. 92.1%). ## our pasta model figure 2 gives an overview of our pasta framework, which follows the pre-training-fine-tuning framework. for pre-training, we guide our model to understand sentences and to perform table-based operations (e.g., aggregation) to complete synthesized cloze tasks in the sentence. for finetuning, we apply a select-then-rank strategy to trade off between sizes of large tables and the limited input length of debertav3. next, we will first present our sentence-table cloze task and pretraining corpus generation in sections 3.1 and 3.2, respectively. we will then discuss our fine-tuning strategy in section 3.3. ## conclusion and future work we introduced pasta, a table-operations aware pretraining approach to train lms for better performing fact verification over tables. pasta achieved new sota results on two widely-adopted tablebased fact verification benchmark datasets, tab-fact and sem-tab-facts. future work should explore how to address the challenges of more complex operations and large tables in fact verification.