N orm T ab: Improving Symbolic Reasoning in LLM s Through Tabular Data Normalization (2024)

Abstract: In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in parsing textual data and generating code. However, their performance in tasks involving tabular data, especially those requiring symbolic reasoning, faces challenges due to the structural variance and inconsistency in table cell values often found in web tables. In this paper, we introduce NormTab, a novel framework aimed at enhancing the symbolic reasoning performance of LLMs by normalizing web tables. We study table normalization as a stand-alone, one-time preprocessing step using LLMs to support symbolic reasoning on tabular data. Our experimental evaluation, conducted on challenging web table datasets such as WikiTableQuestion and TabFact, demonstrates that leveraging NormTab significantly improves symbolic reasoning performance, showcasing the importance and effectiveness of web table normalization for enhancing LLM-based symbolic reasoning tasks.

## introduction tables are a fundamental format for structured data representation and are widely used across various sources, including relational databases, web pages, and financial documents. however, many tables within documents and web pages are designed for direct human consumption and often lack the strict formatting that is expected in relational tables. this discrepancy poses significant challenges when querying them using languages such as sql, integrating them with relational databases, and processing them within applications. large language models (llms) @xcite have emerged as powerful tools for semantic parsing both textual and tabular data and performing complex tasks such as code generation. trained on vast amount of internet data, including both text and tables, and employing techniques such as chain of thought (cot) prompting @xcite and self-consistency @xcite , these models outperform many traditional models on various table reasoning tasks @xcite @xcite . however, their performance in tasks involving tabular data, particularly those requiring symbolic reasoning, is often hindered by the structural variability and inconsistencies commonly found in web tables. symbolic reasoning over tables necessitates a clear understanding of the table structure and values, and may involve constraining rows and columns, which can be challenging when dealing with unstructured or noisy web tables @xcite @xcite . our hypothesis is that normalizing ill-formatted tables can address this challenge, enabling the execution of symbolic programs (such as sql or python) on the tables and making reasoning tasks involving comparison, aggregation, and mathematical calculations more manageable. moreover, normalization may enhance the explainability by allowing the tracking of the intermediate steps in reasoning. consider the table qa task shown in figure 1 . retrieving answers from the table on the left using a symbolic approach such as sql is challenging due to the irregular structure of the data and the limitations of sql. while an llm may handle simple look-up questions, it struggles with tasks requiring complex aggregation and arithmetic operations. however, the normalized version of the same table, shown on the right, can be easily analyzed, allowing text-to-sql approaches to effectively obtain the answers to questions. existing models for table reasoning typically rely on a multi-step framework, where an llm performs a sequence of actions such as adding columns before additional scripts are invoked to process data, retrieve cell values, or compute answers to questions @xcite @xcite . these models are of- in this paper, we introduce normtab, a framework designed to normalize web tables to align them with the structured format of relational database tables. normtab addresses challenges such as structural variance, mixed data formats, and extraneous information, thereby facilitating accurate and efficient symbolic reasoning and query processing using llms. our work explores two key research questions: our proposed solution leverages the advanced textual understanding capabilities of llms to independently process and normalize web tables, without relying on specific questions. by normalizing tables in this manner, we enable a robust foundation for any downstream task involving table reasoning. this approach allows for multiple questions to be asked from a single, normalized table, significantly enhancing reasoning and query capabilities. moreover, our normalization process only needs to be performed once, unlike other models that require repeated adjustments based on different questions, highlighting a key advantage of our approach. through a comprehensive experimental evaluation conducted on challenging web table datasets such as wikitablequestions @xcite and tabfact @xcite , we assess the effectiveness of normtab in improving table reasoning performance. these datasets provide diverse examples of table structures and content, allowing us to thoroughly investigate the impact of web table normalization on llm-based symbolic reasoning tasks. by addressing rq1 and rq2, we aim to demonstrate the importance of web table normalization and its potential to enhance the capabilities of llms in handling tabular data for complex reasoning tasks. key contributions of our paper are: our work is related to a few areas as discussed next. general llms and cot related to our work is the line of research aimed at improving the performance of llms @xcite ) on various reasoning tasks, with capabilities spanning mathematics, common sense, and symbolic reasoning @xcite @xcite . these approaches often excel using fewshot prompts without requiring fine-tuning. their reasoning abilities can be further enhanced by breaking complex tasks into steps, employing methods like chain-of-thought (cot) @xcite prompting and zero-cot. for instance, the @xcite ) model utilizes in-context learn-ing and cot prompting to generate answers for table-based tasks. several studies have utilized instruction tuning and supervised fine-tuning to enhance the performance of llms on table reasoning tasks. notable examples include tablellama @xcite and @xcite , which have shown significant improvements in specific applications. in contrast, the binder model @xcite extends the capabilities of llms to programming language generation for solving commonsense problems. additionally, the dater approach @xcite employs llms to decompose tables and questions, facilitating table-based qa and fact verification tasks. these diverse approaches underscore the potential of llms in handling complex reasoning tasks involving tabular data. reasoning over structured data/tables another line of related work is reasoning over tabular data. several studies leverage symbolic reasoning through text-to-sql or python code for table-based reasoning tasks. however, for effectively utilizing the symbolic code generation approach with llms for table reasoning tasks, it is crucial to ensure that the table is in the proper format @xcite @xcite @xcite . chain-of-table @xcite enhances reasoning on tabular data by iteratively transforming and evolving table structures through a series of reasoning steps, including row/column selection, cell splitting to refine table representations for specific reasoning tasks. their method employs in-context learning to direct llms in iteratively generating operations and updating the table, thus forming a chain of reasoning specific to tabular data. @xcite explore the capabilities of llms in interpreting and reasoning over tabular data, emphasizing robustness to structural perturbations, comparing textual and symbolic reasoning, and examining the potential of aggregating multiple reasoning pathways. their findings indicate that structural variations in tables presenting the same content can significantly degrade performance, particularly in symbolic reasoning tasks. they propose a method for table structure normalization through transposition to mitigate this issue and find that while textual reasoning slightly outperforms symbolic reasoning, each approach has distinct strengths depending on the task. ## methodology our methodology encompasses several essential parts designed to ready web tables for proficient reasoning by llms. ## results in this section, we analyzed the performance of normtab. to evaluate its impact, we conducted few-shot in-context learning experiments to generate sql queries for answering specific questions. first, we performed experiments on unnormalized tables without any modifications. then, we compared the performance on normalized tables. additionally, we reported the performance of different normalization processes. ## conclusion in conclusion, our study introduces normtab, a framework aimed at enhancing llms' performance on tabular data by normalizing web tables. through our investigation, we have shown the significance of web table normalization in overcoming challenges such as mixed values and structural variance. by leveraging llms' textual understanding in data cleaning and normalization, normtab improves table reasoning. our experiments on challenging datasets demonstrate its effectiveness. our work contributes to advancing techniques for llms in handling tabular data, emphasizing the importance of addressing web table challenges for improved performance. further research can explore additional normalization strategies and extend normtab's applicability across various domains. this would establish a robust foundation for a wide range of downstream tasks involving table reasoning.