F e T a QA : Free-form Table Question Answering (2022)

Abstract: Existing table question answering datasets contain abundant factual questions that primarily evaluate a QA system’s comprehension of query and tabular data. However, restricted by their short-form answers, these datasets fail to include question–answer interactions that represent more advanced and naturally occurring information needs: questions that ask for reasoning and integration of information pieces retrieved from a structured knowledge source. To complement the existing datasets and to reveal the challenging nature of the table-based question answering task, we introduce FeTaQA, a new dataset with 10K Wikipedia-based table, question, free-form answer, supporting table cells pairs. FeTaQA is collected from noteworthy descriptions of Wikipedia tables that contain information people tend to seek; generation of these descriptions requires advanced processing that humans perform on a daily basis: Understand the question and table, retrieve, integrate, infer, and conduct text planning and surface realization to generate an answer. We provide two benchmark methods for the proposed task: a pipeline method based on semantic parsing-based QA systems and an end-to-end method based on large pretrained text generation models, and show that FeTaQA poses a challenge for both methods.

## introduction question answering (qa) is the task of producing answers to natural language questions based on knowledge resources @xcite @xcite . one of the primary goals of qa is to allow users to directly and efficiently interact with large-scale and het- * now at facebook ai. erogeneous knowledge sources. in the real world, knowledge sources take a variety of forms, including unstructured texts (documents, passages, or conversations), structured knowledge bases, and semi-structured tables, each requiring dedicated modeling approaches. for qa over text, a sequence modeling approach is usually adopted to encode the query and the context, and answers are either categorical @xcite , extractive @xcite , or abstractive/generative @xcite @xcite . for qa over table, a common approach is to apply semantic parsing on the query and the table schema to generate a logical form (e.g., a sql-like database query) that can be executed to retrieve the answer from the relevant portion of the table @xcite @xcite . the answers are extracted facts/entities in the table, therefore usually in short-form. though existing datasets have enabled significant progress for table qa, their limitations prevent them from reflecting the challenging nature of the task. the exchange of information between humans through interactions with questions and answers is different from the interactions presented in most of the existing qa datasets, in which questions are specific (sometimes contrived for testing multi-hop reasoning) and provide most of the information, while answers are in short-form and fill in the missing information piece. nevertheless, in many cases, people tend to seek more structured information content, such as ''how'', ''why'', and some of the ''what'' questions that ask for general concepts. therefore a qa system should also possess such structuring capability, evaluated by text generation tasks. table is shown for better visualization. dataset knowledge source answer format avg # words wikipedia articles stories, books, movie scripts online forum texts wikipedia tables in answer squad @xcite ) ✓ text-span 3.2 hotpotqa @xcite ✓ short-form entity 2.2 narrativeqa @xcite ✓ free-form text 4.7 eli5 @xcite ✓ free-form text 130.6 wikitablequestions ( pasupat and liang, 2015 to complement the existing datasets with the absent qa interactions, we present fetaqa, a free-form table question answering dataset that includes long, informative, and free-form answers. fetaqa reveals the challenging nature of the table qa task: 1) retrieving multiple entities from tables based on the query; 2) aggregating and reasoning over relations of these entities; and 3) structuring surface information and inferences into a coherent answer that is faithful to the table. we collect question-answer pairs from noteworthy descriptions of wikipedia tables that are high quality sentences rich in structured information contents. we annotate questions that elicit such descriptions, and we make efforts to ensure that the qa interaction is compatible, and question annotations are not contrived. in addition, the fetaqa tables cover a diverse set of topics and contain un-normalized text, including numbers, dates, and phrases. fetaqa examples are presented in figure 1 and differences between fetaqa and other qa datasets are described in we formulate generative table question answering as a sequence-to-sequence learning problem. we propose two benchmark methods and provide experimental results for them. the first one is an end-to-end model that integrates query and table comprehension, reasoning, and language generation by adapting t5 @xcite . the other is a pipeline model that achieves content selection and surface realization in separate modules involving tapas @xcite , which is a recently proposed pre-trained model that jointly processes text and tabular data for the usage of semantic parsing. through human studies, we evaluate answers generated by our proposed models as well as the reference answer based on fluency, correctness, adequacy (informativeness), and faithfulness. the results indicate the challenging nature of fetaqa and that there is much room for improvement in qa systems. we make the dataset and code available online. foot_0 ## dataset here we introduce fetaqa and describe the process and criteria for collecting the tables, questions, and answers. some statistics of fetaqa are shown in § 2.4. ## models to quantify the challenge posed by fetaqa for state-of-the-art models, we used two modeling approaches that have shown to be effective for the existing table question answering datasets, with some modifications made to adjust to our task. model configurations are shown in figure 4 . ## experiments in this section, we explain the experiment settings and report the automatic and human evaluations on model outputs. ## conclusion in this paper, we introduced the task of generative table question answering with fetaqa, a table qa dataset consisting of complex questions that require free-form, elaborate answers. we also proposed two modeling approaches: (1) a pipeline model that incorporates a table semantic parser and a data-to-text generator, and (2) an end-to-end model that integrates query comprehension, reasoning and text generation. our experimental results indicate that the end-to-end model with a simple table encoding strategy achieves much higher scores than the pipeline model that requires table semantic parsing. furthermore, we show that fetaqa reveals the challenging nature of the table question answering task and calls for innovative model designs in the future.