API -Assisted Code Generation for Question Answering on Varied Table Structures (2023)

Abstract: A persistent challenge to table question answering (TableQA) by generating executable programs has been adapting to varied table structures, typically requiring domain-specific logical forms. In response, this paper introduces a unified TableQA framework that: (1) provides a unified representation for structured tables as multi-index Pandas data frames, (2) uses Python as a powerful querying language, and (3) uses few-shot prompting to translate NL questions into Python programs, which are executable on Pandas data frames. Furthermore, to answer complex relational questions with extended program functionality and external knowledge, our framework allows customized APIs that Python programs can call. We experiment with four TableQA datasets that involve tables of different structures — relational, multi-table, and hierarchical matrix shapes — and achieve prominent improvements over past state-of-the-art systems. In ablation studies, we (1) show benefits from our multi-index representation and APIs over baselines that use only an LLM, and (2) demonstrate that our approach is modular and can incorporate additional APIs.

## introduction tables are an important and widely used format for storing and retrieving information. however, since they are often constructed to present information in a visually effective way, they consequently occur in diverse formats @xcite @xcite . thus, to effectively answer questions about all tabular information, we must consider information stored in relational @xcite , matrix, and hierarchically indexed tables @xcite , and also address scenarios where multiple tables are presented conjointly @xcite . in the past, works have focused on achieving strong results on datasets with particular table struc- who is more likely to have cancer, the elder or the young? tures and required tailoring logical forms to each specific type of table structure @xcite . these methods struggle to work on tables with structures outside of their original domain. for example, the neural-symbolic machine (nsm) approach designed for relational tables @xcite only achieves 29.2% accuracy on the hierarchical matrix table dataset hitab @xcite due to the ineffectiveness of its logical forms on hierarchical tables. our work focuses on developing a unified framework to solve tableqa tasks across diverse table structures using python as an intermediate language. in contrast with previous approaches that use custom logical forms to query table data @xcite , we propose to query entries and perform operations within the widely used python pandas library. this allows us to leverage the strong few-shot python generation capabilities of large code generation models, whilst saving costs by using only a few training examples. an overview of the framework is presented in figure 1 , with more details in figure 3 . our framework consists of three main parts: first, we transform tables with varied structures into a unified multi-index data frame representation adopted from the python pandas library ( §3.1). the multi-index objects can effectively retain the structural information in a wide range of table formats. from the implicit first step illustration in figure 1 and more detailed one in figure 2c , we convert tables from a hierarchical format to the multi-index representation, which will enable generated code to successfully query elements in the table. second, we translate tableqa questions into python programs as an executable intermediate language by prompting code generation models ( §3.2). specifically, we use a few-shot paradigm where we provide the multi-index headers of the table, a few rows of the table in textual form, the question, and exemplar program annotations. as shown in figure 1 , the table and question are input to the code generation model along with the few-shot prompt, and the generated python code uses the pandas data.loc function to query the appropriate cells. third, we introduce assistant api functions to extend the capabilities of our framework beyond python pandas and achieve broader coverage over various tableqa tasks. these functions enable a model-generated program to query external knowledge and perform various additional operations on the multi-index representation. in this paper, we demonstrate the usage of two simple types of api functions, operation apis and qa api ( §3.3), and show through ablations that they increase the performance of our framework across various datasets. for example in figure 1 , the model outputs code using one of our api functions, compare_larger, to solve the question. we evaluate our method across relational, hierarchical, and matrix tables, as well as multiple table paradigms using the wikitablequestions (wik-itq) @xcite , hitab @xcite , ait-qa @xcite , and spider @xcite datasets ( §4). for code generation models, we use a more capable proprietary model, codex @xcite , and an open-source model, starcoder @xcite . we find that our framework surpasses the state-ofthe-art few-shot baselines on the hitab, ait, and spider datasets, achieving absolute improvements of 24.5%, 26.2%, and 2.3%, respectively, while retaining non-trivial performance on wikitq. furthermore, we perform an ablation study on the api functions we introduced and find that they bring significant improvements across datasets and models. our framework also allows the use of existing api operations within pandas in a modular way. for example, we show that combining pandas' sql interface api with our multi-index representation and apis improves performance by 3.7-7.6% on relational datasets. ## problem setting and datasets in this section, we provide a formal description of the tableqa tasks ( §2.1) and describe the four datasets used in our experiments, containing tables of varied structures ( §2.2). ## method we first describe our approach for representing variously structured tables with a unified multi-index format ( §3.1). we then introduce our method to answer tableqa questions by generating python programs ( §3.2) from a code llm. lastly, we detail the use of custom helper api functions ( §3.3) that are learned to be used by the llm in context ( §3.4). figure 3 gives an overview of our method. ## experiments in this section, we first introduce the experiment ( §4.1) and evaluation settings ( §4.2), then present our results and analysis ( §4.3). we also test the robustness of our method with different code models ( §4.4) and intermediate query language forms ( §4.5). ## ablation studies we conduct ablation analyses on the hitab dataset to examine the improvements brought by assistant api functions ( §5.1) and multi-index parsing ( §5.2). we refer to multi-index parsing as w/ mi, and denote the opposing table flattening approach as w/o mi in the result tables. ## conclusion we introduced a framework for solving general tableqa tasks. this framework is built upon three core components: a unifying multi-index representation, python programs as the query language, and code generation by prompting a large pretrained code model. using codex as the code model, we demonstrate improvements over the state-of-the-art performance for multiple datasets, each with its unique table structures and challenges. through ablations, we show that our proposed multi-index and api functions are both critical to the success of the framework, with the largest improvements on datasets involving hierarchical tables. we also observe improvements in performance for an opensource model, starcoder, demonstrating the effectiveness of our approach with different code models.