O mni T ab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering (2022)

Abstract: The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value. The intrinsic complexity of handling tables often adds an extra burden to both model design and data annotation. In this paper, we aim to develop a simple table-based QA model with minimal annotation effort. Motivated by the fact that table-based QA requires both alignment between questions and tables and the ability to perform complicated reasoning over multiple table elements, we propose an omnivorous pretraining approach that consumes both natural and synthetic data to endow models with these respective abilities. Specifically, given freely available tables, we leverage retrieval to pair them with relevant natural sentences for mask-based pretraining, and synthesize NL questions by converting SQL sampled from tables for pretraining with a QA loss. We perform extensive experiments in both few-shot and full settings, and the results clearly demonstrate the superiority of our model OmniTab, with the best multitasking approach achieving an absolute gain of 16.2% and 2.7% in 128-shot and full settings respectively, also establishing a new state-of-the-art on WikiTableQuestions. Detailed ablations and analyses reveal different characteristics of natural and synthetic data, shedding light on future directions in omnivorous pretraining.

## introduction humans are voracious information omnivores, consuming information in a number of forms. unstructured text is the form covered in most work in nlp, but another form widely used on the web, academic papers, and reports is the table, where elements are organized in rows and columns and presented in a structured and succinct way. because of this, systems to aid information access over tables, such as table-based question answering (qa) @xcite @xcite , hold significant promise. however, they also require understanding of the table structure and sophisticated reasoning across multiple elements to get the final answer. this intrinsic complexity not only often requires special-purpose model designs such as pipeline systems that generate structured queries as intermediate steps @xcite @xcite , but also adds an extra burden to the process of data annotation @xcite . given the challenges above, we ask the question: "can we create a simple model that is able to answer complex table-based questions with minimal annotation effort?" both modeling simplicity and limited assumptions regarding data availability would make it easier to apply table-based qa systems in practical scenarios. to this end, we focus on developing a simple and generic end2end table-based qa model where only a limited number of annotated natural language (nl) questions are available; the first attempt to address this problem under few-shot settings. in order to answer table-related questions, an end2end model needs to understand both the nl question and the table, build connections between the two formats, then perform complex reasoning. taking the manually annotated question in fig. 1 as an example, models need to align entities ("collateral damage") and concepts ("film", "air") in the question to elements in the table (the "collateral damage" cell and the "film" and "date" columns) and perform comparative reasoning based on chronological order (indicated by "previous" and "before") to find the final answer. motivated by this, we propose omnitab, an omnivorous pretraining approach that consumes natural data to endow models with the ability to understand and align nl with tables, and synthetic data to train models to perform reasoning. ## end2end table-based qa in this section, we first explain the setting of tablebased qa, then introduce the input format as well as our model architecture. table-based qa each example in table-based qa consists of an nl question q, a table t , and an answer a. both questions and answers are a sequence of tokens. each table consists of n rows {r i } n tablebased qa aims to generate the answer a given both the question q and the table t as the input. input format we follow liu et al. (2021) in concatenating the nl context with the linearized table as input. we flatten the table following a topto-bottom and left-to-right order, where we prepend "col:" to the beginning of the header and "row i:" to the beginning of the i-th row to separate different rows: @xmath0: r 1 row 1: r 2 ... row n : r n ]. cells within each row are separated by a vertical bar "|" r @xmath1,1 | c i,2 | ... | c i,m ]. finally, the question q is prepended to the linearized table: [q t ]. ## omnitab: pretraining with natural and synthetic data as mentioned in the introduction, table-based qa requires both (1) the ability to align nl phrases with table elements that could be expressed in different wording and (2) perform reasoning such as filtering, aggregation, superlatives, comparatives, and arithmetic. compared to synthetic data, real nl sentences relevant to the table excel at enhancing the first capability since they are more natural and fluent, exhibiting various language variations. learning on real sentences can endow models to grasp the nuances in language and align with structured tables. on the other hand, synthetic data is flexible, manipulable, and easy to obtain. it is costless to generate synthetic data via manipulating different aspects of the generated data to incorporate various desired properties. as a result, we can generate large amounts of complicated synthetic data covering various reasoning operations, which is lacking in the nl corpora. this motivates us to explore both types of data. ## experiments we first detail the experimental settings ( § 4.1). then we report on extensive experiments, starting with the overall result including all design elements to achieve the best results ( § 4.2), then breaking down into each design choice to analyze their individual contribution ( § 4.3). finally, we analyze concrete examples to provide insight on different characteristics of natural and synthetic data ( § 4.4). ## related work table-based qa is a well-studied area from early systems using structured queries as intermediate steps @xcite @xcite @xcite to recent advances that generate answers in an end2end fashion @xcite , 2021). our methods follow the end2end approach because of its modeling simplicity and higher flexibility. given large amounts of table and text on the web, table-based qa and other table-related tasks such as semantic parsing @xcite and table understanding @xcite start witnessing efforts invested in pretraining on both structured and unstructured information. most works leveraging retrieval to find relevant information to assist pretraining are designed for text format only @xcite , while the majority of table-based pretraining still use alignment heuristics @xcite . there are some initial attempts to perform retrieval over tables @xcite @xcite , but they mainly use tables as an additional information source while we focus on pairing tables with text for pretraining. ## conclusion we propose an omnivorous pretraining approach that consumes both natural and synthetic data to enhance the ability to understand and align text and tables and the ability to perform reasoning. our extensive results demonstrate the effectiveness of both data and verify their complementary value. our empirical results together with the case analysis indicate that omnivorous pretraining can indeed benefit from the merits of both data, encouraging future advances in retrieval and synthesis to obtain higher-quality data and better pretraining strategies to combine heterogeneous data.