R eal H i TB ench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM -Based Table Analysis (2025)

Abstract: With the rapid advancement of Large Language Models (LLMs), there is an increasing need for challenging benchmarks to evaluate their capabilities in handling complex tabular data. However, existing benchmarks are either based on outdated data setups or focus solely on simple, flat table structures. In this paper, we introduce **RealHiTBench**, a comprehensive benchmark designed to evaluate the performance of both LLMs and Multimodal LLMs (MLLMs) across a variety of input formats for complex tabular data, including LaTeX, HTML, and PNG. RealHiTBench also includes a diverse collection of tables with intricate structures, spanning a wide range of task types. Our experimental results, using **25** state-of-the-art LLMs, demonstrate that RealHiTBench is indeed a challenging benchmark. Moreover, we also develop TreeThinker, a tree-based agent that organizes hierarchical headers into a tree structure for enhanced tabular reasoning, validating the importance of improving LLMs’ perception of table hierarchies. We hope that our work will inspire further research on tabular data reasoning and the development of more robust models. The code and data are available at https://github.com/cspzyy/RealHiTBench.

## introduction in recent years, there has been a lot of work on table analysis @xcite . specifically, an increasing number of studies discuss table question answering (tableqa) tasks. @xcite @xcite @xcite . additionally, the introduction of large language models (llms), including closesource models such as @xcite and gemini-1.5-pro @xcite , and tableoriented models like tablegpt @xcite figure 1: a complex table example with typical complex structures and a question-answer example. unfortunately, we find that the current tabular data benchmarks, such as tat-qa @xcite , tablebench @xcite , and infiagent-dabench @xcite , largely consist of flat tables. more concretely, in such tables, each column represents an attribute, each row represents a record, and all data are stored in a simple one-dimensional format; see figure 13 for visual demonstration. however, in practical applications, humans often organize relatively complex tables to represent multifaceted relationships between variables. such hierarchically structured tables are popular in various domains, including economy, science, and employment, on public data platforms. to address this issue, some benchmarks have considered hierarchical tables, such as hitab @xcite and scitab @xcite . however, these benchmarks fail to comprehensively present the llms' understanding capabilities of complex table structures. for example, scitab @xcite and ait-qa @xcite only consider tables of the scientific and airline domains, respectively. mm-tab @xcite provides only image-based input, while hierarchical tables in realistic applications are also presented in a textual way. spreadsheetbench @xcite focuses on the operation of tables. hitab @xcite organizes its tables with lossy json format, only contains fundamental qa tasks, and provides incomplete supervision. more importantly, most of these benchmarks focus primarily on relatively simple hierarchical tables, particularly those with a basic column hierarchy, where the hierarchy typically does not exceed two levels. to date, there is still a lack of dedicated benchmarks for comprehensively evaluating llms' ability to understand complex tabular hierarchies. to this end, we propose realhitbench, a challenging realistic hierarchical table benchmark based on complex tables and tasks. (i) complex table structures: our benchmark includes complex tables with intricate features (partially depicted in figure 1 ), which are commonly found in real-world scenarios but often overlooked in existing benchmarks. (ii) modal and format diversity: we explore the performance of both text and image approaches while others focusing solely on one of them. our benchmark provides llms and mllms to be tested with various input formats, such as la-tex, html, and png. (iii) question diversity: realhitbench covers a wide range of question types, each designed to test different aspects of a model's ability. especially a type called structure comprehending is designed on the basis of complex parts. (iv) accurate and efficient annotation: our benchmark employs a rigorous annotation process. gpt-based automated annotation and human checks ensure the accuracy and reliability of the questions and answers. we conducted a comprehensive evaluation of abundant types of models consisting of tableoriented llms, and open-source/closed-source generic llm/mllms. in addition, we tested a couple of table reasoning tasks with different proper metrics. the performance of different models differs greatly, with average scores of all tasks rang-ing from 7.27 to 56.95. importantly, overall low scores (mainly below 70) highlight that the ability of llms to comprehend and process intricate table structures remains an area in need of significant improvement. we also develop a tree-based pipeline (dubbed treethinker) that automatically injects table hierarchies into instructions. empirically, we show that, with self-emphasized table hierarchies, llms' structural understanding ability can indeed be enhanced. table analysis. table analysis is pivotal across numerous domains @xcite , while table question answering (tableqa) can effectively assess analytical capabilities @xcite . recently, there has been a growing number of studies focusing on the ability of llms to comprehend tabular data @xcite @xcite . however, the development of llms leads to the emergence of different branches. common tabular data are in text form to be processed by llms @xcite @xcite , while other image-based tables are suited to be processed by mllms @xcite . additionally, some table-oriented llms for both modalities have also been proposed @xcite . therefore, the flourishing development of llms makes challenging benchmarks for tabular data increasingly important. ## realhitbench in practical applications, tabular data holds significant value. numerous previous studies have proposed a series of benchmarks @xcite , which have effectively propelled research in this field. however, as application scenarios become increasingly complex and large language models (llms) gradually emerge as the mainstream method for table reasoning @xcite , the previous benchmarks struggle to provide a comprehensive and accurate assessment of model capabilities. hence, there is an urgent need for a new, realistic, and challenging benchmark to evaluate the progress of research methods in this field. we propose realhitbench focusing on complex structures and tasks. complex tables, including hierarchical tables, are widely used in various domains @xcite . we propose such a challenging benchmark full of complex tables and corresponding difficult tasks to assess the upper bound of the comprehension capability of llms, which may further inspire future research on processing tables with llm. ## treethinker beyond benchmark, we propose treethinker, a understanding-augmented pipeline that enhances the model's ability in complex hierarchical tables question answering. as shown in figure 3 , we first prompt the model to automatically organizing hierarchical headers into tree structure, then aligning keywords from questions with the tree headers, and efficiently locating sub-table relevant to the question.