M ulti T ab QA : Generating Tabular Answers for Multi-Table Question Answering (2023)

Abstract: Recent advances in tabular question answering (QA) with large language models are constrained in their coverage and only answer questions over a single table. However, real-world queries are complex in nature, often over multiple tables in a relational database or web page. Single table questions do not involve common table operations such as set operations, Cartesian products (joins), or nested queries. Furthermore, multi-table operations often result in a tabular output, which necessitates table generation capabilities of tabular QA models. To fill this gap, we propose a new task of answering questions over multiple tables. Our model, MultiTabQA, not only answers questions over multiple tables, but also generalizes to generate tabular answers. To enable effective training, we build a pre-training dataset comprising of 132,645 SQL queries and tabular answers. Further, we evaluate the generated tables by introducing table-specific metrics of varying strictness assessing various levels of granularity of the table structure. MultiTabQA outperforms state-of-the-art single table QA models adapted to a multi-table QA setting by finetuning on three datasets: Spider, Atis and GeoQuery.

## introduction question answering (qa) over multiple tables aims to provide exact answers to natural language questions with evidence from one or more tables @xcite . this is in contrast to single-table qa, which has been the focus of tabular qa research to date @xcite @xcite . even though groups of related tables are ubiquitous in real-world corpora, such as relational databases or tables in a web page, multi- recent work on tabular qa can be categorized into two major directions: (i) semantic parsing-based techniques @xcite @xcite , which have been the dominant approach to answering multi--table complex questions. such methods transform a natural question to a logical form, which is used to query a relational database to extract the answer. however, these techniques are restricted to relational databases and cannot be applied to tables from other sources such over web tables, tables in text documents, and any non-normalized tables. additionally, they require expensive and expert human annotations @xcite formulating sql queries from natural questions. (ii) modeling the problem as a sequence generation/classification task @xcite @xcite @xcite @xcite @xcite , where an end-to-end trained neural model is not only responsible for question/query understanding but also table reasoning. existing endto-end neural models are either classification-based @xcite , where the model detects the answer span and classifies one table operator associated with the span, or they are sequence generation-based @xcite @xcite , where the model generates the answer as a span of text in an auto-regressive manner. our work focuses on the latter direction of research. we train a neural model to mimic a semantic parser and generate the answer. a clear distinction of our work compared to existing end-to-end models is that our proposed model, multitabqa, does not operate in the constrained setting of a single input table, but can accommodate one or more tables in the input and the associated multi-table operators. additionally, multitabqa performs the task of structured table generation, which imposes structure aspects to the generated output such as table schemas, alignments of rows and columns, relationships between column-headers and column values. generating structured tables as output requires table-specific evaluation metrics which we define and use to evaluate the generated tables. to effectively train the model, we generate a pretraining dataset with multi-table sql queries and tabular answers built over an existing semantic parsing dataset, spider @xcite . our dataset consists of 132, 645 samples comprising of sql queries, associated natural language questions, input tables, and tabular answers. to the best of our knowledge, this is the first work to address the task of multi- ## methodology we frame multi-table question answering as a sequence-to-sequence task and train an autoregressive transformer encoder-decoder model to generate the tabular answer. given a question q consisting of a sequence of k tokens q 1 , q 2 , . . . , q k and a set of n tables, t @xmath0, t 2 , . . . , t n }, the goal of the multi-table qa model is to perform chains of operations over t n , constrained by q, and generate a table t out . the model always generates a table, t out , which can be single celled for scalar answers, single rowed or columned for listbased answers, and multiple rows and columns for tabular answers. in all cases, the model also generates column headers revealing important semantics associated with the generated values. training approach. we follow a curriculum learning approach @xcite by sequentially increasing the complexity of tasks to train multitabqa. the first stage of training is a pretraining step where the training objective is twofold: (i) learn to generate correct tabular answers from sql, and (ii) understand the associations between related input tables. the final training stage is fine-tuning where the model learns to understand natural language questions with their inherent ambiguity in addition to retaining its ability of reasoning over tables and generating a tabular answer. we discuss the training process in detail in section 4. model input/output. the input to the model is a sequence comprised of the query or the natural language question, followed by a sequence of input tables, represented by the table name and the corresponding flattened table. table names are important for disambiguating tables in multi-table qa setting. specifically, the input sequence is represented as question [ ## dataset to effectively train a multi-table qa model, the dataset needs to cover three aspects: (i) multi-table context, (ii) tabular answers, and (iii) natural questions. given the absence of large-scale datasets covering all three aspects, we transform existing semantic parsing and single-table qa datasets to focus on a single aspect before training with samples covering all three aspects. ## training we follow a curriculum learning approach by sequentially training the model on sub-tasks of increasing complexity as depicted in figure 3 . broadly, we first pre-train the seq2seq model to mimic a sql parser and further fine-tune it on the downstream multi-table qa task. pre-training the model on unambiguous sql queries leads to better convergence and warm-start for the closely related downstream multi-table qa task. we further segregate the pre-training by first addressing the simpler sub-task of generating tables from single table queries. this is immediately followed by pre-training on multi-table query answering where complex sql queries are utilized to train the model to learn multi-table associations from unambiguous complex queries, reason over the tables and generate tabular answer. the final stage of training is the downstream multi-table qa from natural language questions. natural language introduces ambiguity, ellipses and co-references which increases complexity and is thus the final stage of training. for each stage, we choose the model with the best table exact match accuracy on the corresponding validation set, defined in section 5, as the initialization for training the next stage. ## evaluation metrics while denotation accuracy has been widely used in semantic parsing @xcite @xcite , it is not directly applicable for our task where tabular input encoding, reasoning, and generation are performed by the same model. evaluating the answer table not only requires matching the generated values but also the table structure. moreover, tables store factual information such as named entities, dates, numbers, etc in an ordered manner. this makes lexical metrics measuring surface form overlap more suitable than semantic metrics measuring the underlying meaning of paraphrased sequences. moreover, table components such as rows, columns and cells are standalone units which capture different levels of semantics and relationships with the surrounding table component. for example, rows capture data records while columns capture the features of each record. cells capture the lowest level of self-contained facts and requires complete match with the target. for example, a cell with the entity "united kingdom" should not be partially matched with the predictions "united nation", "united" or "kingdom". similarly, a numeric value such as "123.45" should not be partially matched with "12.45", "23.45" or "12". numeracy pose a challenge to seq2seq models @xcite , especially in the extrapolation setting where semantic match of unseen numbers may not be an ideal. considering all these factors, we focus on lexical match to measure model effectiveness. table exact match. we define table exact match accuracy (table em) as the percentage of predicted tables which exactly matches the target tables. table exact match evaluates ordering of rows, columns and table headers and exact lexical matching of table values. it is a strict binary measure which treats partial matches as incorrect. however, many queries do not impose ordering among columns or rows, and strict table exact match may not be the ideal indication of model efficacy. to measure partial correctness, we treat rows, columns and cells as units at varying levels of granularity which have ordered associations among the values within the unit. we evaluate partial correctness with exact match of rows, columns and cells. row exact match. to relax the strict criterion of table exact match, we first measure correctness on table rows. row exact match does not consider ordering of rows in the generated table but requires ordering of values within the row. we define a correctly generated row to be a predicted row that exactly matches any target rows in the target table. row exact match precision is the percentage of correctly generated rows among all the predicted rows in the evaluation dataset. row exact match recall is the percentage of correctly generated rows among all the target rows in the evaluation dataset. column exact match. unlike rows, which represent records in relational databases, columns represent attributes where column header provides semantic meaning to the values. hence, a correct column is defined as a generated column that first exactly matches a target column header and further the column values. column exact match measures ordering of values within a column. column exact match precision is the percentage of correctly generated columns among all generated columns in the evaluation set. column exact match recall is the percentage of correctly generated columns among all target columns in the evaluation set. cell exact match. cell exact match is the most relaxed measure of model efficacy at the lowest level of granularity (cells) where table structure is not measured. a cell is correct if it matches any cell in the corresponding target table. cell exact match precision is the percentage of correctly predicted cells among all predicted cells in the dataset. cell exact match recall is the percentage of correctly predicted cells among all target cells in the dataset. ## experimental setup and results we use tapex-base @xcite as the base model for all our experiments. tapex-base is a single table question answering model (140m parameters) trained to approximate table reasoning by pre-training to mimic an sql parser. for both the pre-training and fine-tuning process, we use a batch size of 8 and gradient accumulation of 32 to epochs table em 0 5 10 15 20 25 5 10 15 20 25 30 multitabqa tapex figure 4: validation table exact match scores of multitabqa vs. tapex-base on spider evaluation set natural language questions during fine-tuning. the points are highest validation scores for each model. qualitative analysis. multi-table qa models must perform numeric reasoning, understand multitable schemas and comprehend natural language. a success case also depicts this. for the question how many likes does kyle have? with 2 input tables: we also observe that while the column and table em decreases dramatically when using several tables (figure 5a and 5c ), the row and cell em does not (figure 5b and 5d ). this indicates that multi-tabqa can generate rows and cells as effectively in single and multiple input tables settings but fail to do so for columns and consequently for the whole table. this is due to the fact that certain columns in the answer, particularly ones with numbers such as floats, are challenging to generate. the error from the incorrect columns propagates and are accumulated in the table em leading to a significant drop in performance for multi-table queries. ablation on training stages. we perform ablation on the pre-training stages to analyse the contribution of each dataset. the simplest setting is to pre-train with spider sql queries, i.e., stage 2. to evaluate the effectiveness of single table tapex pre-training samples, the next setting comprises of stages 1 and 2, i.e., pre-train with tapex pretraining and spider sql dataset. the final comparison is with the three-stage pre-training as described in section 4.1. the results for one run of the experiments are displayed in ## related work tabular qa is a research direction in the broader topic of table understanding @xcite in natural language processing. recent advances in table representation @xcite and pre-training @xcite @xcite , table fact verficiation @xcite , table numeric reasoning @xcite , table-to-text generation @xcite ), text-to-table generation @xcite , table summarization @xcite @xcite , and table question answering @xcite @xcite @xcite @xcite @xcite @xcite have shown the adaptability of language models to table processing. ## conclusion in this work, we propose a new task of multi-table question answering without intermediate logical forms to fill the gap of existing end-to-end table qa research which focused only on single-table qa. we release a pre-training dataset of 132, 645 samples to effectively train a seq2seq model. we fine-tune and evaluate our model, multitabqa, on natural language questions of three datasets: spider, geoquery and atis, to test the efficacy in a multitable setting. as many multi-table questions result in tables, we train the model to generate tables. this necessitates table-specific metrics at various levels of granularity which we design to evaluate the effectiveness of our model. we demonstrate that such metrics is insightful in understanding model behavior. multitabqa outperforms existing state-of-the-art single table qa model fine-tuned to adapt to a multi-table qa setting.