Topic Transferable Table Question Answering (2021)

Abstract: Weakly-supervised table question-answering (TableQA) models have achieved state-of-art performance by using pre-trained BERT transformer to jointly encoding a question and a table to produce structured query for the question. However, in practical settings TableQA systems are deployed over table corpora having topic and word distributions quite distinct from BERT’s pretraining corpus. In this work we simulate the practical topic shift scenario by designing novel challenge benchmarks WikiSQL-TS and WikiTable-TS, consisting of train-dev-test splits in five distinct topic groups, based on the popular WikiSQL and WikiTable-Questions datasets. We empirically show that, despite pre-training on large open-domain text, performance of models degrades significantly when they are evaluated on unseen topics. In response, we propose T3QA (Topic Transferable Table Question Answering) a pragmatic adaptation framework for TableQA comprising of: (1) topic-specific vocabulary injection into BERT, (2) a novel text-to-text transformer generator (such as T5, GPT2) based natural language question generation pipeline focused on generating topic-specific training data, and (3) a logical form re-ranker. We show that T3QA provides a reasonably good baseline for our topic shift benchmarks. We believe our topic split benchmarks will lead to robust TableQA solutions that are better suited for practical deployment

## introduction documents, particularly in enterprise settings, often contain valuable tabular information (e.g., financial, sales/marketing, hr). natural language question answering systems over a table (or tableqa) have an additional complexity of understanding the tabular structure including row/column head-figure 1 : topic-sensitive representations are important to infer that, in the context of the topic politics, the query span "ran in the election" should be linked to the "candidate" column in the table. most of the tableqa systems use semantic parsing approaches that utilizes language encoders to produce an intermediate logical form from the natural language question which is executed that over the tabular data to get the answer. while some systems @xcite were fully supervised, needing pairs of questions and logical forms as training data, more recent systems @xcite @xcite rely only on the answer as weak supervision and search for a correct logical form. the current best tableqa systems @xcite capitalize on advances in language modeling, such as bert, and extend it to encode table representations as well. they are shown to produce excellent results on popular benchmarks such as wikisql @xcite and wikitablequestions(wikitq) @xcite . with increasing prevalence of text analytics as a centrally-trained service that serves diverse customers, practical qa systems will encounter tables and questions from topics which they may not have necessarily seen during training. it is critical that the language understanding and parsing capabilities of these qa models arising from their training regime are sufficiently robust to answer questions over tables from such unseen topics. as we show later in this paper, the existing approaches degrade significantly when exposed to questions from topics not seen during training (i.e., topic-shift). 2 to examine this phenomenon, we first instrument and dissect the performance of these recent systems under topic shift. in particular, we experiment with tabert @xcite , which is a weakly supervised tableqa model which encodes the table and question using bert-encoder and outputs a logical form using an lstm decoder. in the example shown in figure 1 , topic shift may cause poor generalization for specific terminology or token usage across unseen topics. we introduce a novel experimental protocol to highlight the difficulties of topic shift in the context of two well-known wikipedia-based tableqa datasets: wikisql @xcite and wik-itablequestions @xcite . despite recent transformer-based tableqa models being pre-trained with open-domain data, including wikipedia itself, we observe a performance drop of 5-6% when test instances arise from topics not seen during training. to address this challenge, we next propose a novel t3qa framework for tableqa training that leads to greater cross-topic robustness. our approach uses only unlabeled documents with ta-2 topic shift may be regarded as a case of domain shift studied in the ml community. however, here we refrain from referring to the proposed topic-driven splits as "domains" due to the open-domain nature of these datasets and the pretraining data used to build these models. the main contributions of this work are: ## related work most tableqa systems take a semantic parsing view @xcite @xcite for question understanding and produce a logical form of the natural language question. fully-supervised approaches, such as by @xcite need pairs of questions and logical form for training. however, obtaining logical form annotations for questions at scale is expensive. a simpler, cheaper alternative is to collect only question-answer pairs as weak supervision @xcite @xcite . such systems search for the correct logical forms under syntactic and semantic constraints that produce the correct answer. weak supervision is challenging, owing to the large search space that includes many possible spurious logical forms @xcite that may produce the target answer but not an accurate logical transformation of the natural question. recent tableqa systems @xcite @xcite extend bert to encode the entire table including headers, rows and columns. they aim to learn a table-embedding representation that can capture correlations between question keywords and target cell of the table. tapas @xcite and rci @xcite are designed to answer a question by predicting the correct cells in the table in a truly end-to-end manner. tabert @xcite ) is a powerful encoder developed specifically for the tableqa task. tabert jointly encodes a natural language question and the table, implicitly creating (i) entity links between question tokens and tablecontent, and (ii) relationship between table cells, derived from its structure. to generate the structured query, the encoding obtained from tabert is coupled with a memory augmented semantic parsing approach (mapo) @xcite . question generation (qg) @xcite @xcite has been widely explored in reading comprehension (rc) task to reduce the burden of annotating large volumes of q-a pairs given a context paragraph. recently, @xcite used gpt-2 @xcite to generate synthetic data for rc, showing that synthetic data alone is sufficient to obtain stateof-art on the squad1.1 dataset. for the qg task in tableqa, systems proposed by @xcite ; @xcite utilize the structure of intermediate logical forms (e.g., sql) to generate natural language questions. however, none of these qg methods utilize the additional context like table headers, structure and se-mantics of the tables or the nuances of different possible question types like complex aggregations. to the best of our knowledge, our approach is the first to generate questions specifically for tableqa with the assistance of a logical query and large pre-trained multitask transformers. domain adaptation approaches in qa @xcite have so far mostly used adversarial learning with an aim to identify domain agnostic features, including in rc applications @xcite . however, for the tableqa systems using bert-style language models with vast pre-training, topic shifts remain an unexplored problem. ## t3qa framework to our knowledge, this is the first work to explore tableqa in unseen topic setting. consequently, no public topic-sliced tableqa dataset is available. we introduce a topic-shift benchmark by creating new splits in existing popular tableqa datasets: wikisql @xcite and wikitq @xcite . the benchmark creation process is described in section 3.1. then, we introduce the proposed framework (illustrated in figure 2 ) to help tableqa system cope with topic shift. section 3.2 describes the topic specific vocabulary extension for bert, followed by question generation in target topic in section 3.3 and reranking logical forms in section 3.4. ## experiments and analysis here we describe key details of the experimental setup, the models compared and evaluation techniques. we also provide a thorough analysis of the results to highlight the key takeaways. ## conclusion this paper introduces the problem of tableqa for unseen topics. we propose novel topic split benchmarks over wikisql and wikitq and highlight the drop in performance of tabert+mapo, even when tabert is pretrained on a large open domain corpora. we show that significant gains in performance can be achieved by (i) extending the vocabulary of bert with topic-specific tokens (ii) fine-tuning the model with our proposed constrained question generation which transcribes sql into natural language, (iii) re-ranking logical forms based on features associated with entity linking and logical form structure. we believe that the proposed benchmark can be used by the community for building and evaluating robust tableqa models for practical settings.