E^5 : Zero-shot Hierarchical Table Analysis using Augmented LLM s via Explain, Extract, Execute, Exhibit and Extrapolate (2024)

Abstract: Analyzing large hierarchical tables with multi-level headers presents challenges due to their complex structure, implicit semantics, and calculation relationships. While recent advancements in large language models (LLMs) have shown promise in flat table analysis, their application to hierarchical tables is constrained by the reliance on manually curated exemplars and the model’s token capacity limitations. Addressing these challenges, we introduce a novel code-augmented LLM-based framework, E^5 , for zero-shot hierarchical table question answering. This approach encompasses self-explaining the table’s hierarchical structures, code generation to extract relevant information and apply operations, external code execution to prevent hallucinations, and leveraging LLMs’ reasoning for final answer derivation. Empirical results indicate that our method, based on GPT-4, outperforms state-of-the-art fine-tuning methods with a 44.38 Exact Match improvement. Furthermore, we present F^3 , an adaptive algorithm designed for token-limited scenarios, effectively condensing large tables while maintaining useful information. Our experiments prove its efficiency, enabling the processing of large tables even with models having limited context lengths. The code is available at https://github.com/zzh-SJTU/E5-Hierarchical-Table-Analysis.

## introduction in the digital era, structured data, especially in the form of hierarchical tables, plays a crucial role in information representation. unlike flat tables, hierarchical ones utilize multi-level headers @xcite @xcite , offering a nuanced organization of complex datasets. they're widely used in areas like finance, scientific research, and business structures. three main challenges with hierarchical table analysis are: (1) complex structure: their multi-level, bi-dimensional indexing is intuitive for humans but confusing for models. (2) implicit calculation relationships: they often have aggregated rows and columns without obvious indications. (3) implicit semantic relationships: hierarchical tables contain numerous cross-row, cross-column, and cross-level relationships that aren't clearly indicated @xcite . as large language models (llms) @xcite @xcite @xcite @xcite @xcite have exhibited significant progress in diverse nlp tasks @xcite , recent works @xcite @xcite start to apply llms on flat table analysis based on the models' in-context learning ability using fewshot examplers. @xcite introduce the dater framework, which seeks to decompose table reasoning tasks using codex @xcite . besides, @xcite presents structgpt, which utilizes an invoking-linearization-generation approach to apply llms on structured data. however, when transitioning to hierarchical table analysis, these methodologies encounter specific constraints: (1) these methods depend heavily on manually curated in-context exemplars, which are extremely expensive and inflexible for generalization (2) given the substantial size of some hierarchical tables, directly inputting them as exemplars for in-context learning can surpass the model's token capacity, making these methods infeasible. in light of these challenges, our study emphasizes zero-shot learning without the need for exemplars, aiming for a more efficient and universally applicable approach across diverse datasets. to the best of our knowledge, we are the first to investigate hierarchical table analysis using llms and study table analysis in a zero-shot manner. in this work, we propose e 5 , a code-augmented framework for hierarchical table question answering (qa) tasks in a zero-shot manner using llms. we formulate the process of solving this task as the following steps: (1) self-explaining the hierarchical structures of the complex table to enhance llms' comprehension of the underlying semantic associations between headers (2) generating code to extract related rows and columns and apply appropriate logic operations (e.g., filter, sort, etc) (3) using external code interpreter to execute the generated code and exhibit observations to mitigate hallucinations (4) extrapolating the observations to get the final answer, leveraging the reasoning capabilities of llms via natural language. experiment results on benchmark datasets show that based on gpt-4, our proposal surpasses state-ofthe-art (sota) fine-tuning methods by a significant margin (44.38 on exact match). besides, compared with another tool-augmented llm agent, react @xcite ), e 5 achieves noticeable improvement on multiple metrics with a significant increase in efficiency and stability. to additionally study the practical token-limited scenario, we propose f 3 , an adaptive algorithm to condense large tables while keeping as much useful information as possible within the constraints of the token length. comprehensive experiments prove that not only does f 3 save computational resources while maintaining decent performances but also makes it possible to process large tables using models with relatively small context lengths. ## task description the hierarchical table analysis task is a variant of the table-based qa problem, specifically tailored for hierarchical tables. formally, the task involves processing a hierarchical table t with multi-level headers, a query q, and a prompt p that contextualizes the query, through a model m. the model outputs an answer a, succinctly represented by the equation: @xmath0, q, p). ## method as shown in figure1, we propose e 5 , a codeaugmented pipeline for question answering over large hierarchical tables. we first let an llm selfexplain table structures to understand hierarchies. then, the llm generates code to extract relevant table structure: levels of headers: {row : 1, columns: 3} related rows: [spouse, son, daughter, close friend, …] related columns: ["65 to 74 years" for women.] … llm llm @xmath1.dataframe(({"primary_caregiver": ["spouse", "son","daughter, …], "65_to_74_years_women": [36, 17, 20 …], }) … print(most_common_giver) explain extract extrapolate observation:['spouse'] code interpreter execute exhibit what's the commonly primary caregiver among women aged 65 to 74? figure 1 : an illustration of our proposed framework of e 5 and f 3 . e 5 is a pipeline where an llm explains table hierarchies, generates code to extract relevant data, and then interprets the results post-execution, optimizing question answering. in scenarios with token limitations (dashed line), we first apply f 3 to compress the large table while maintaining as useful information as possible before feeding to e 5 . table parts and apply operations, making it focus on useful information. an external interpreter then executes the code to avoid hallucination, with results analyzed by the llm itself. we also propose f 3 , an adaptive algorithm to compress large tables in a practical context-limited scenario, which can be applied before e 5 to compress large tables while keeping as much useful information as possible. ## experiment results figure 4: performances of three fill strategies' with changing token limits. all three strategies achieve decent performances when the input token length is constrained. among the three strategies, the row-wise diffusion strategy outperforms the others. to provide a more holistic comparison, we analyze the performance variation of both react and e 5 in relation to the maximum iteration limit. as illustrated in figure 3 , when the maximum number of iterations is reduced, e 5 exhibits a more marginal performance decline compared to react. the disparity in performance between the two methods becomes more pronounced as the maximum number of iterations is reduced. specifically, when the maximum number of iterations is limited to 2, this discrepancy is most evident. under this condition, e 5 demonstrates decent performances on both evaluation metrics, achieving scores of 74.81 on em and 82.95 on gpt-4-eval. notably, the variance for e 5 (9.92) is substantially lower than that of react (29.99), which indicates superior stability (a) em score (b) gpt-4-eval figure 5: comparative analysis of e 5 and e 4 performance across error categories. we report the averaged results over ten runs. e 5 outperforms e 4 in all error categories by considerable margins, indicating the effectiveness of self-explaining tables' hierarchical structures. of e 5 concerning the maximum iteration number. additionally, in the course of our experiments, we observe that in the absence of instructions to extract pertinent cells using pandas, react frequently attempts to load the complete table via multiple tools, leading to recurrent syntax errors. token-limited scenario figure 4 illustrates the performance of three fill strategies in relation to the input token length limit. we find that gpt-4 can still achieve decent results even with token limit constraints for hierarchical table analysis, indicating the effectiveness of our proposed f 3 . as the token limit increases from 2000 to 6000, all three strategies have considerable performance improvements. notably, the row-wise diffusion strategy outperforms the others across both evaluation metrics. ## error analysis besides the main experiment, we also conduct a comprehensive error analysis of our proposals. specifically, we run e 4 ten times, selecting instances with at least one recorded error. these errors were then classified into the following four categories and visualized in figure 6 : to further investigate how well e 5 can address these error cases through explain phase, we separately evaluate e 4 and e 5 on them, repeating the experiment ten times. we visualize the average em score and gpt-4-eval over ten runs in figure 5 . from this figure, it is evident that e 5 exhibits considerable improvement across all four error categories, with a particularly notable enhancement in the semantic relationship category (an increase of over 50%). these findings suggest that llm can significantly benefit from self-explaining the hierarchical table structure in our tasks. (1) react question: what's the commonly reported primary caregiver among senior women aged 65 to 74. thought: i need to find the highest percentage for primary caregiver for women in the 65 to 74 years age group. action: python_repl action input: @xmath2: (100, 100), "spouse": (19, 60), "son": (19, 10), "daughter": (42, 15), "close friend": (4, 4), "other family member, friend, or acquaintance": (16, 10)} @xmath3, @xmath4: data[x][0]) max_caregiver observation: i now know the most commonly reported primary caregiver. final answer: daughter (2) ! ! question: what's the commonly reported primary caregiver among senior women aged 65 to 74. thought: to find the most commonly reported primary caregiver among senior women aged 65 to 74, i need to look at the values in the "65 to 74 years" column for women and find the highest value. then, i will return the corresponding primary caregiver category. action: python_repl action input: import pandas as pd @xmath5.dataframe({ "primary_caregiver": ["total", "spouse", "son", "daughter", "close_friend", "other_family_member_friend_or_acquaintance"], "65_to_74_years_women": [100, 36, 17, 20, 6, 21] }) @xmath6.max() @xmath7.values[0] print(result) observation: total i now know the final answer. final answer: total (3) ! " question: what's the commonly reported primary caregiver among senior women aged 65 to 74. thought: i need to find the most commonly reported primary caregiver for senior women aged 65 to 74 years. table structure: the table has 3 levels of headers. the first level has the primary caregiver as the row label and age groups for seniors with sub-columns for women and men. the second level has columns for total -65 years and over, 65 to 74 years, 75 to 84 years, and 85 years and over, each with sub-columns for women and men. the third level has percentage as the column label for all age groups and sex columns. the related rows are the primary caregiver categories (spouse, son, daughter, close friend, other family member, friend, or acquaintance) and the related column is "65 to 74 years" for women. action: python_repl action input: import pandas as pd @xmath8.dataframe({ "primary_caregiver": ["spouse", "son", "daughter", "close friend", "other family member, friend, or acquaintance"], "65_to_74_years_women": [36, 17, 20, 6, 21], }) @xmath9.max() @xmath10.tolist() print(most_common_caregiver) observation: ['spouse'] i now know the final answer. final answer: spouse figure 7: the complete reasoning path and generated code of react, e 4 and e 5 for the ( table, query) pair in figure 1 . the highlighted segments are related to how different methods' reasoning over the semantic relationship between the 'total' row and the remaining rows. compared with react and e 4 , e 5 exhibits an enhanced ability to understand the implicit relationship between headers through self-explaining. in this particular instance, e 5 can recognize that total is not a category of caregiver and generate the correct answer. ## case study in order to qualitatively investigate how the stage of self-explaining helps the e 5 system, we conduct a case study to compare three different toolaugmented approaches, react, e 4 , and e 5 . based on the table and query depicted in figure 1 , the reasoning trajectories of react, e 4 , and e 5 are illustrated in figure 7 . from this example, we observe that both react and e 4 simply retrieve all row headers in the given table and neglect the underlying semantic correlations among them, leading to an incorrect answer. although it is intuitive for humans that the row labeled total represents the cumulative sum of preceding rows, both react and e 4 , employing gpt-4, fail to recognize this, mistakenly interpreting total as a separate caregiver category. on the contrary, as highlighted in the paragraph of table structure , e 5 accurately identifies the appropriate caregiver categories, excluding the row labeled total in the generated code. this suggests that integrating a phase for self-explaining table structures can enhance the capacity of llms to discern implicit relationships within hierarchical head-ers. besides, in this case, react does not explicitly print its output, resulting in an absence of observation. consequently, the llm produces an inaccurate answer, a phenomenon referred to as "hallucination." such outcomes suggest that merely instructing llms to use tools may not be sufficiently effective for tool-augmented agents. alternatively, prompting llms to appropriately use tools in a structured manner proves to be of paramount importance and the design of such prompts should be meticulously tailored to each distinct task. ## conclusion and future work we introduce e 5 , a code-augmented pipeline for zero-shot question answering over large hierarchical tables. through comprehensive experiments, we empirically demonstrate that built upon gpt-4, e 5 can significantly outperform previous sota finetuning-based methods by a substantial margin (44.38 on exact match) and other prompting strategies with superior stability and adaptability. we separately analyze the phase of self-explaining through extensive error analysis and case study, proving its effectiveness in helping llms get a deeper insight into hierarchical table structures. we also investigate the practical token-limited scenario and propose an adaptive algorithm, f 3 , making it possible to analyze large tables using models even with models constrained by a limited context length. we believe that our contributions hold substantial promise for practical applications in real-world table analysis, such as financial statements, in the industry.