Rethinking Tabular Data Understanding with Large Language Models (2024)

Abstract: Large Language Models (LLMs) have shown to be capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area. In this context, this study investigates from three core perspectives: the robustness of LLMs to structural perturbations in tables, the comparative analysis of textual and symbolic reasoning on tables, and the potential of boosting model performance through the aggregation of multiple reasoning pathways. We discover that structural variance of tables presenting the same content reveals a notable performance decline, particularly in symbolic reasoning tasks. This prompts the proposal of a method for table structure normalization. Moreover, textual reasoning slightly edges out symbolic reasoning, and a detailed error analysis reveals that each exhibits different strengths depending on the specific tasks. Notably, the aggregation of textual and symbolic reasoning pathways, bolstered by a mix self-consistency mechanism, resulted in achieving SOTA performance, with an accuracy of 73.6% on WikiTableQuestions, representing a substantial advancement over previous existing table processing paradigms of LLMs.

## introduction large language models (llms; @xcite @xcite @xcite @xcite ) have revolutionized the field of nlp, demonstrating an extraordinary ability to understand and reason over rich textual data @xcite @xcite @xcite . on top of llms' existing capabilities for nlp, further bolstering their potential for decision-making by drawing from external knowledge sources remains an exciting research frontier (nakano et al., figure 1 : demonstration of the challenges faced by llms in comprehending and interpreting table structures. in the first example, the llm correctly identifies table headings but struggles to accurately determine their positions within the table structure. in the second example, the model using python shell as an external tool incorrectly interprets headings (located in the first column) as column headers, leading to subsequent errors in the generated code. some logos in this and subsequent figures are generated using openai's dall-e3 @xcite . @xcite @xcite . amongst such knowledge sources, tabular data serve as a ubiquitous kind due to their expressiveness for relations, properties and statistics, and their being easy to construct by human curators @xcite @xcite . like humans, llms can also benefit from reading tabular data accompanying text. however, as indicated in fig. 1 , the structural nature of tables top: the table and its title. bottom left: in the first question example, textual reasoning yields an incorrect interpretation due to limitations in precision localization, while symbolic reasoning accurately locates the answer using python code. bottom right: in the second question example, textual reasoning successfully identifies the answer, but symbolic reasoning incorrectly treats the special total row as the final answer. with the emergence of instruction fine-tuning techniques @xcite and the application of reinforcement learning from human feedback (rlhf) @xcite @xcite , llms have witnessed significant enhancements in their alignment capabilities, paving the way for transitioning from few-shot to zero-shot learning settings @xcite . in light of these advancements, this paper delves deep into the the challenges and intricacies of tabular understanding and reasoning by llms, exemplified in fig. 2 . we organize our exploration around three pivotal research questions: (1) how well do llms perceive table structures and how can we ensure robustness against structural variations? (2) comparing textual and symbolic reasoning for table data in llms, which prevails in effectiveness, and what advantages and challenges manifest in each strategy? (3) will the aggregation of multiple reasoning pathways enhance the accuracy and reliability of tabular data interpretation by llms? in pursuit of answering the aforementioned research questions, we conduct experiments on sota llms such as gpt-3.5 (openai, 2023a). our findings in §4 underscore that while llms are adept at semantically interpreting tables, their capability to resist structural variance ( §4.1) and understand table structures ( §4.2) is suboptimal. motivated by these findings, we propose a table structure normalization method to enhance llms' resilience against structural table variations in §4.3. intriguingly, §5.1 reveals that textual reasoning surpasses symbolic reasoning in contexts with limited table content, defying conventional conceptions of symbolic reasoning's dominance in other domains @xcite . both textual and symbolic reasoning strategies encompass different advantages and challenges, which is detailed in §5.2. to harness the unique strengths of each, we implement mix self-consistency mechanism ( §6) that remarkably attains sota performance on table qa, exemplifying the synergistic potential when both reasoning strategies are aggregated. ## preliminaries this section succinctly introduces the foundational aspects of our study over structurally perturbed tabular data. §3.1 formally defines the problem, delineating the critical notations and conceptual frameworks, and §3.2 explicates our experimental setup details, elucidating dataset choice, model utilization, and evaluation strategy. ## llm robustness to structural perturbations this section explores how llms interpret varied table structures in response to our first research question ( §1). we probe the impact of three table perturbations on llm performance ( §4.1), uncover llms' challenges and limitations for direct table transposition and recoganize tranposed tables ( §4.2), and introduce a structure normalization strategy (norm) to mitigate these issues ( §4.3). ## comparing textual and symbolic reasoning in this section, we delve into the comparison of textual and symbolic reasoning methods in llms for tabular data understanding ( §5.1), further conducting a detailed error analysis ( §5.2) to address the second research question ( §1). we evaluate the performance of each reasoning strategy using gpt-3.5, shedding light on their strengths and challenges. in §4.3, we explored norm to mitigate structural perturbations, enhancing generalized llm performance and successfully restoring perturbed tables to accuracy levels similar to their original states. therefore, subsequent analyses will exclusively consider the original tables (t ). ## reasoning aggregation this section examines how combining multiple reasoning pathways can boost llms' accuracy in interpreting tabular data, which is in response to the third research question ( §1). ## conclusion this study investigated the proficiency of llms in tabular reasoning. the findings suggest that llms are sensitive to the structural variance of tables, but the application of the proposed normalization strategy can stabilize table structures and improve resistance to structural perturbations. when comparing reasoning approaches, textual reasoning demonstrated a slight advantage over symbolic reasoning, with each strategy exhibiting unique strengths. furthermore, integrating multiple reasoning strategies via mix self-consistency proved beneficial for overall interpretation accuracy, surpassing previous sota results on the wikitablequestions dataset. these observations contribute to the understanding of llms' capabilities in tabular reasoning and provide insights for further improvements.