T ool W riter: Question Specific Tool Synthesis for Tabular Data (2023)

Abstract: Tabular question answering (TQA) presents a challenging setting for neural systems by requiring joint reasoning of natural language with large amounts of semi-structured data. Unlike humans who use programmatic tools like filters to transform data before processing, language models in TQA process tables directly, resulting in information loss as table size increases. In this paper we propose ToolWriter to generate query specific programs and detect when to apply them to transform tables and align them with the TQA model’s capabilities. Focusing Toolwriter to generate row-filtering tools improves the state-of-the-art for WikiTableQuestions and WikiSQL with the most performance gained on long tables. By investigating headroom, our work highlights the broader potential for programmatic tools combined with neural components to manipulate large amounts of structured data.

## introduction an important area for research in large language models (t5, palm, gpt-3) is combining them with "tools" to enhance their capabilities in question answering @xcite @xcite . toolaugmented approaches enable language models to externalize knowledge and computation by making explicit calls to apis. however, these approaches do not process semi-structured data. we show that current models degrade in effectiveness significantly when questions and data become long and complex. a key task that demonstrates these limitations is tabular question answering (tqa) where long tables and complex questions are particularly challenging for current models. tabular question answering is a task in natural language processing that involves leveraging information from a semi-structured table to answer multi-hop compositional questions. it discourages purely symbolic approaches due to latent structure here a row-filter tool is generated as a program that transforms semi-structured data. current tqa systems @xcite @xcite linearize tables as a token string and process it jointly with the question. in response to long tables present in wik-itablequestions @xcite and wikisql @xcite language models for tqa increase the context size to 1024 tokens incurring a high memory cost. however, we show in section 3 as table size increases average model performance significantly degrades by 40%. moreover, 23% of tables in wikitablequestions are truncated at 1024 tokens. we propose toolwriter , a new method that augments language model capabilities. in response to a question over a table, toolwriter decides if tool use is required. it then generates a program to transform the table to simplify it to make question answering more effective. the generated tools are code that can be applied to all kinds of tables regardless of size which overcomes key language model limitations. the proposed toolwriter approach is model agnostic and can be flexibly combined with existing models in a zero-shot setup. in this work, we compare multiple detection strategies and tool generation approaches to generate query and table-specific python programs. toolwriter leverages our best combination for tool generation (zero-shot gpt-3) and tool-use detection (combined answer confidence and table length). our method improves the state-of-the-art exact match results on wikitablequestions to 64.9 (+1.9%) and wikisql to 90.5 (1.5%). we summarize our contributions as the following: ## task definition tabular question answering is a task in natural language processing that involves leveraging information from a semi-structured table t to generate an answer ŷ to a question q. questions are expressed in natural language and implicitly involve compositional types of reasoning to access and aggregate information in the table. these questions are implicitly multi-step and require a combination of symbolic reasoning and natural language understanding. a system has access to a training set @xmath0, t ), ŷ)} of questions, tables, and answers. ta-bles between training and evaluation are disjoint to prevent memorization. the only restriction on the question is that it must be answerable given the information provided in the table. average exact match (em) over d between the predicted answer ŷ and target y is used as the primary metric. ## behaviour analysis tabular question answering is a challenging setting since tables can be exceedingly long. context length in transformer architectures is often limited to 512 tokens due to a quadratic memory cost @xcite . in wikitablequestions 41.7% of linearised tables exceed 512 tokens without even considering the question tokens. current approaches to tqa patch this problem by increasing the context limit to 1024 tokens (omnitab, tapex, unifiedskg) and 2048 in gpt-3. this incurs a significant memory cost often prohibiting the use of such models. however, even at 1024 tokens of context 23.8% of tables are truncated thus incurring data loss. ## toolwriter toolwriter is our proposed method to address the limitations of current language models on large semi-structured data. in response to a question over a table, toolwriter decides if tool-use is required. it then generates a program to transform and simplify the table to make question-answering more effective. first, we outline our conceptual framework followed by our method implementation. ## results first, we investigate the various tool generators of toolwriter on a subset of wikitablequestions where filtering is often required. second, we focus on the importance of the detector to choose when to best apply the generated tools (section 5.2). third, we test toolwriter to both detect and generate tools across various tqa datasets and methods (section 5.3). finally, we analyze how toolwriter performs as table size increases (section 5.4). ## background and related work semantic parsing focuses on generating an executable parse for the exact answer @xcite , benefiting from data size independence @xcite . it requires strong supervision @xcite or reinforcement learning @xcite and assumes coherent data formatting and an expressive target language. alternative approaches learn a joint tablequestion-answer mapping. seq2seq models @xcite execute @xcite and simulate formal programs @xcite . intermediate executable modules were integrated @xcite , while transformer-based models @xcite @xcite ) leveraged unsupervised language capabilities @xcite @xcite . recent interest in execution-loop models arises from language models' ability to explain reasoning @xcite , improving compositional questions @xcite and symbolic manipulation @xcite @xcite . tqa language models generate chains of thought with sub-question answers @xcite . recent advances in code-focused language models led to an interest in combining question decomposition and program interaction @xcite . toolformer @xcite , program assisted learning @xcite , and tool augmented language models @xcite interleave execution and natural language reasoning but face limitations in capacity. our work addresses large structured context directly while interleaving execution and natural language. ## limitations while toolwriter enables lms to pragmatically interact with tabular data, there are some clear limitations to what can be achieved with our method in its current form. one such limitation is the tool generator's focus on row filtering as a primary means to transform the table. toolwriter relies exclusively on the underlying tqa model to perform all aggregations, arithmetic mappings, and groupings on the transformed table. this is because toolwriter solely removes irrelevant rows from the input table, but may not reduce the size of the relevant rows which, in the limit, may still overwhelm a transformer based tqa model. our current approach also does not allow neural filtering of the tabular data for situations where programmatic interactions may break down. this is due to the current tool generator producing standard python. despite python's expressive capabilities for string manipulation, as shown in figure 3 , string manipu-lations can break down in more semantic contexts @xcite . beyond this, we found that toolwriter is sensitive to cascading errors due to the inherent multi-step interactions on the structured data. the tool generator is not informed by the failure mode of the original tqa model other than that it needs to reduce the table. this is due in part to the current black-box nature of models such as tapex which do tqa entirely end-to-end. possible solutions to this include recent work in lm reasoning @xcite where a rationale may be explicitly generated indicating a failure mode. this feedback would then inform the tool generation process. ## conclusion tabular question answering is a challenging setting for neural methods due to large context sizes and implicit reasoning. first, we characterize the limitations of neural methods to integrate structured data and find all language modeling methods struggle with large tables. second, we propose toolwriter to generate query-specific tools to simplify large tables and detect when these transformations should be applied. we propose various language modelbased methods to generate programs that filter rows which universally improve and achieve state-ofthe-art results on two tabular question-answering datasets. finally, we determine significant headroom in both detecting when to use tools and how to generate them under oracle setting highlighting the potential in tools to manipulate structured data combined with language models.