CRT - QA : A Dataset of Complex Reasoning Question Answering over Tabular Data (2023)

Abstract: Large language models (LLMs) show powerful reasoning abilities on various text-based tasks. However, their reasoning capability on structured data such as tables has not been systematically explored. In this work, we first establish a comprehensive taxonomy of reasoning and operation types for tabular data analysis. Then, we construct a complex reasoning QA dataset over tabular data, named CRT-QA dataset (Complex Reasoning QA over Tabular data), with the following unique features: (1) it is the first Table QA dataset with multi-step operation and informal reasoning; (2) it contains fine-grained annotations on questions’ directness, composition types of sub-questions, and human reasoning paths which can be used to conduct a thorough investigation on LLMs’ reasoning ability; (3) it contains a collection of unanswerable and indeterminate questions that commonly arise in real-world situations. We further introduce an efficient and effective tool-augmented method, named ARC (Auto-exemplar-guided Reasoning with Code), to use external tools such as Pandas to solve table reasoning tasks without handcrafted demonstrations. The experiment results show that CRT-QA presents a strong challenge for baseline methods and ARC achieves the best result.

## introduction large language models (llms) @xcite @xcite @xcite have recently shown emergent abilities, such as the capacity for "reasoning", when they are sufficient in size @xcite . a large number of works @xcite @xcite ; ## crt-qa dataset in this section, we describe the task formulation, and process for collecting tables, questions, answers, and detailed annotations for crt-qa. ## method although llms show powerful reasoning abilities on various tasks @xcite , they have limitations on various fields @xcite . from our pivot experiments of prompting baselines, we find that llms can often generate correct reasoning plans but are unable to appropriately execute them. however, such steps (e.g., arithmetic, counting) can be perfectly performed by external tools such as sql or pandas. inspired by recent works on tool-augmented llms @xcite @xcite ; paranjape et al., 2023), we propose an efficient and effective approach, named arc (auto-exemplar-guided reasoning with code), to use external tools such as pandas to solve table reasoning tasks without handcrafted demonstrations. figure 3 illustrates the pipeline of our proposed arc and the detailed prompt design for each step can be found in appendix h. auto-exemplar generation although manuallyannotated demonstration shows significant effectiveness in in-context learning for llms, nontrivial hand-drafting of effective exemplars makes it not flexible enough to be applied to sophisticated tasks such as code generation for complex table analysis. inspired by recent works on auto-demonstration generation (zhang et al., 2022b), we first randomly sample a data instance from the development set and input llms with an instructional prompt for code generation. the prompt we use is a simple instruction to generate python code and print intermediate or final results. the ablation of different selections will be discussed in appendix h. in-context code generation for every data example in the test set, we use the exemplar generated from the dev set to conduct in-context learning for code generation. as pandas is the most commonlyused library in python for tabular data analysis which may frequently occur in llms' pretraining data, the generated codes are proficient in the use of pandas to process tables. code execution with external tools we further use the generated code for execution using a python interpreter with a pandas installed environment. we then obtain the output of the program as the intermediate or final results for the query. iterative llm calling with code output however, for queries that require in-depth commonsense reasoning, only the code sometimes can not directly solve them. as a result, inspired by re-act (yao et al., 2023b), we also integrate acting and reasoning by injecting code output into the prompt design for final step reasoning. by prompting llms with code output, llms can generate more accurate final answers by avoiding step execution errors. ## unanswerable and indeterminate question most table qa datasets are designed for answering the questions with golden labeling @xcite @xcite , but real users possibly ask questions that are inherently difficult to answer due to the complexity of the real world. motivated by this, we incorporate a sub-set of unanswerable and indeterminate queries where some questions go beyond common external knowledge, while others are inherently problematic. we categorize these questions into four categories and conduct the answerability of llms based on them. ## conclusion in this work, to systematically evaluate llms' reasoning ability on tabular data, we first establish a comprehensive taxonomy on operation and reasoning types for table analysis. then, we propose crt-qa, a dataset of complex reasoning qa over tables. we propose arc which effectively utilizes table analysis tools to solve table reasoning tasks without manually-annotated exemplars. extensive experiments show crt-qa poses a significant challenge for llms and our proposed arc achieves the best em scores. besides the main experiments, we also conduct thorough ablation studies, error analyses, answerability study, and case study for further analysis.