R eas TAP : Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples (2022)

Abstract: Reasoning over tabular data requires both table structure understanding and a broad set of table reasoning skills. Current models with table-specific architectures and pre-training methods perform well on understanding table structures, but they still struggle with tasks that require various table reasoning skills. In this work, we develop ReasTAP to show that high-level table reasoning skills can be injected into models during pre-training without a complex table-specific architecture design. We define 7 table reasoning skills, such as numerical operation, temporal comparison, and conjunction. Each reasoning skill is associated with one example generator, which synthesizes questions over semi-structured tables according to the sampled templates. We model the table pre-training task as a sequence generation task and pre-train ReasTAP to generate precise answers of the synthetic examples. ReasTAP is evaluated on four benchmarks covering three downstream tasks including 1) WikiSQL-Weak and WikiTQ for Table Question Answering, 2) TabFact for Table Fact Verification, and 3) LogicNLG for Faithful Table-to-Text Generation. Experimental results demonstrate that ReasTAP achieves new state-of-the-art results on all of them and delivers a significant improvement under low-resource setting. Our code is publicly available at https://github.com/Yale-LILY/ReasTAP.

## introduction inspired by the massive success of pre-trained language models (lm) on free-form natural language (nl) tasks @xcite @xcite ), researchers have attempted to extend the pre-training to table data. tables are a valuable form of data that organize information in a structured way. they often contain data that is organized in a more accessible manner than in unstructured texts. to adapt the pre-training paradigm on structured tab- ## downstream tasks we evaluated reastap on three different types of downstream tasks to verify its effectiveness. the statistics and examples for each task are shown in table qa wikisql-weak (zhong et al., 2017) and wikitq (pasupat and liang, 2015) were used to evaluate reastap performance on table qa tasks. wikisql-weak (zhong et al., 2017) requires the models to perform filtering and optional aggregation on table cell values to answer the given a question. wikitq (pasupat and liang, 2015) requires a broader set of reasoning skills, thus is more challenging. the table qa task formulation is the same as the reastap pretraining task. we used the denotation accuracy, which checks whether the predicted answers are equal to the ground truths, as evaluation metric. table facts verification we chose tab-fact (chen et al., 2020b) to evaluate reastap performance on table facts verification tasks. given a table and a statement, tabfact (chen et al., 2020b) tries to distinguish whether the statement is entailed or refuted by the table. tabfact divides its test sets into test simple and test complex subsets, where test complex contains examples requiring more complex table reasoning skills. furthermore, it selects a small test set test small with 2k samples for human evaluation. to fine-tune on tabfact, following bart (lewis et al., 2020), we applied a binary classifier upon the hidden state of the last token in the decoder for the output. the objective is to generate the verification label @xmath0, 1} given the statement @xmath1, s 2 , ⋯, s n ) and the input table t : @xmath2,1} we used the accuracy (i.e., percentage of correct predictions) as evaluation metric. faithful table -to-text generation we chose logicnlg @xcite to evaluate reastap performance on the faithful table -to-text generation task. compared with previous table -to-text generation benchmarks @xcite @xcite , which primarily focus on surface-level realizations without much logical inference, logicnlg is tasked to generate statements that are logically entailed by the selected table region. given the serialized input table with its selected columns as t , the objective is to generate a sentence @xmath3, y 2 , . . . , y n ) that is both fluent and factually correct: ## analysis experimental results on three different kinds of downstream tasks show that reastap can broadly improve bart's generic table reasoning capabilities, which could be adapted to different downstream tasks, regardless of whether the tasks are highly similar to the reastap pre-training task or not. in this section, we further analyze our approach in terms of various aspects to provide researchers with a deeper insight for future work. ## related work reasoning over tables reasoning over the input context is an important requirement for neural models to be applied in the real world, and especially when the input is structured knowledge such as a table. several table qa benchmarks @xcite @xcite have been proposed to test systems' capability to conduct different types of reasoning, including numerical, logical or multi-hop reasoning. for table fact verification tasks @xcite , the models are required to perform logical inference to verify whether the given statement is entailed or refuted. furthermore, table -to-text @xcite @xcite tasks to generate a natural language description of some part of the table based on inferences obtained from facts in the contexts. more recently, numerical reasoning over tabular data in financial domain has also raised increasing attention @xcite @xcite @xcite . table pre-training inspired by the huge success of pre-training in natural language, researchers have attempted to extend pre-training to structured tabular data @xcite @xcite @xcite @xcite @xcite in recent years. previous table pre-training work such as tabert @xcite and tapas @xcite took corrupted tables and nl sentences as input and tried to recover the corrupted parts. they had the intuition that such recovering processes can help strengthen the linking between sentences and structured tables. on the other hand, tapex @xcite learned from synthetic sql programs. @xcite further pretrained tapex over natural and synthetic qa examples to improve the few-shot performance over table qa tasks. meanwhile, pre-training for textto-sql tasks @xcite @xcite also attracted researchers' attention in recent years. unlike previous work, we model the pre-training task as a sequence generation task, and inject various table reasoning skills into the model by tasking it to generate the precise answers of reasoning examples. synthetic pre-training corpus generating a large-scale synthetic pre-training corpus is widely used in both natural language pre-training @xcite @xcite @xcite and table pre-training @xcite @xcite . for example, geva et al. (2020) utilized automatically-generated numerical data to inject numerical reasoning skills during pretraining. and yoran et al. (2021) leveraged largescale wikipedia resources to automatically generate examples that requires reasoning over multiple facts in the paragraph, and continue pre-training lm on this synthetic corpus. furthermore, recent works @xcite showed that pre-training can be achieved by learning a program executor over synthetic corpus. ## conclusion in this paper, we propose reastap, a new table pre-training approach, which injects various pre-defined table reasoning skills into models via learning to generate correct answers of synthetic questions. compared to previous work which design table-specific architectures, reastap is easy to implement and is theoretically applicable to any sequence-to-sequence lm. reastap is evaluated over four downstream benchmarks. the experimental results demonstrate that reastap achieves new state-of-the-art results on each of them. this includes the improvements on wikisql-weak denotation accuracy to 90.4% (+1.2%); wikitq denotation accuracy to 58.6% (+1.4%); tabfact accuracy to 84.7% (+0.7%); and logicnlg sp-acc to 54.8% (+4.0%), nli-acc to 89.2% (+3.6%). further analysis demonstrates that reastap delivers a significant improvement to bart on the low-resource setting, indicating that our proposed pre-training approach can effectively improve the model's generic table reasoning capabilities.