S yn TQA : Synergistic Table-based Question Answering via Mixture of Text-to- SQL and E 2 E TQA (2024)

Abstract: Text-to-SQL parsing and end-to-end question answering (E2E TQA) are two main approaches for Table-based Question Answering task. Despite success on multiple benchmarks, they have yet to be compared and their synergy remains unexplored. In this paper, we identify different strengths and weaknesses through evaluating state-of-the-art models on benchmark datasets: Text-to-SQL demonstrates superiority in handling questions involving arithmetic operations and long tables; E2E TQA excels in addressing ambiguous questions, non-standard table schema, and complex table contents. To combine both strengths, we propose a Synergistic Table-based Question Answering approach that integrate different models via answer selection, which is agnostic to any model types. Further experiments validate that ensembling models by either feature-based or LLM-based answer selector significantly improves the performance over individual models.

## introduction table qa (tqa) takes a question and a table , and finds an answer based on the evidence from the table @xcite . with the help of large scale datasets @xcite @xcite , state-ofthe-art (sota) tqa systems primarily focus on two approaches: semantic parsing (text-to-sql) that predicts a sql query as intermediate semantic representation of the question, and then executes the sql to find the answer @xcite @xcite ; endto-end system (e2e tqa) that directly generates the answer from models pre-trained on table corpora, imitating human-like reasoning on questions and tables @xcite @xcite . despite serving for a figure 1 : a demonstration of sota table qa models' strengths in solving different types of table-based questions, followed by an overview of syntqa. in a synergistic way, syntqa aggregates candidate answers from text-to-sql and e2e tqa models, and then select the final answer. the answers in green color are the correct answers. similar purpose, it's unclear what advantages these approaches have and their potential synergy. to answer these questions, we first (re)evaluate sota text-to-sql models, i.e., t5 @xcite , @xcite , and din-sql @xcite , as well as e2e tqa models, i.e., tapex @xcite , omnitab @xcite , and gpt, on benchmark datasets wtq @xcite and wikisql @xcite . the experiments show that while both text-to-sql and e2e tqa approaches are adept at simple questions, they have complementary strengths for complex questions and tables, as shown in figure 1 (top): text-to-sql is more proficient in numerical reasoning and processing long tables while e2e tqa is better at ambiguous questions, complex schema and contents. motivated by their distinct strengths, we propose synergistic table-based question answering (syntqa, bottom of figure 1 ), which aims to integrate the strengths of both models via answer selection. at each time, given the input of table, question, text-to-sql answer, and e2e tqa answer, the answer selector identifies the more probable correct one from text-to-sql and e2e tqa answers. experiments show that both feature-based selector and llm-based selector provide significant improvement over single models. ## table question answering task table question answering has received significant attention as it helps non-experts interact with complex tabular data. formally, given an input question @xmath0, q 2 , . . . , q n } and a table t with r rows and c columns, and each cell t i,j contains a real value, table qa aims to produce an answer @xmath1, a 2 , . . . , a k }, where q n and a k are tokens. then we introduce two main approaches for table qa: text-to-sql and e2e tqa. ## evaluating text-to-sql and e2e tqa in this section, we evaluate existing text-to-sql and e2e tqa models on two benchmark datasets: wtq and wikisql. ## syntqa: selecting correct answer above findings show that different models solve different questions, so we use a selector to choose the answer. specifically, at each time, the selector receives the input of table t , question q, text-to-sql prediction and confidence a sql , along with e2e tqa prediction and confidence a e2e . afterwards, the selector determines the correct answer a sel , where a @xmath2, a e2e }. in general, answer selection can be done through featurebased classification or llm-based contextual reasoning, which is discussed in this section. we use the best performing base models, i.e., fine-tuned t5 for text-to-sql and omnitab for e2e tqa in the ensemble model. ## conclusion table qa approaches: text-to-sql and e2e tqa. results indicate text-to-sql's proficiency in arithmetic operations and long tables and e2e tqa's advantages in resolving ambiguity and complexity in the question and table. we enhance performance on table qa datasets by combining models through answer selectors. we plan to extend the method to more challenging problems such as hybrid tqa @xcite .