L ong T able B ench: Benchmarking Long-Context Table Reasoning across Real-World Formats and Domains (2025)

Abstract: We introduce LongTableBench , a benchmark for evaluating long-context reasoning over semi-structured tables across diverse formats, tasks, and domains. It comprises 5,950 QA instances spanning 7 table formats (e.g., Markdown, HTML, SQL), 18 domains, and input lengths up to 128K tokens, including multi-turn and multi-table settings. To ensure data quality, we combine symbolic supervision, cross-model validation, and human review. Evaluating 52 LLMs—including general-purpose, table-specific, and reasoning-enhanced models—reveals that only the strongest models maintain robust performance under increasing context lengths and format diversity. We further show that end-to-end models outperform compression-based approaches, especially on tasks requiring semantic integration. LongTableBench provides a rigorous, scalable testbed for advancing long-context tabular understanding and highlights key limitations in current LLMs’ structural and reasoning capabilities. The code and data are available at https://github.com/liyaooi/LongTableBench .

## introduction recent advancements in large language models (llms) have pushed the boundaries of context length from thousands of tokens @xcite to millions @xcite , enabling new capabilities in tasks like multi-document analysis @xcite and long-form dialogue @xcite . however, a critical question remains: do the expanded contexts these models process truly unlock the reasoning potential needed for complex, real-world scenarios? semi-structured tables, which are structured data represented in textual formats, provide an ideal playground for investigating this question. these tables-found in various markup languages and file formats like markdown, html, latex, json, and csv @xcite @xcite -preserve a consistent logical structure despite varying textual representations. examples include financial tables @xcite , healthcare records @xcite , and scientific data @xcite , all of which are prevalent in critical real-world applications. they offer unique challenges for llms to process and reason over complex, hierarchical structures within long contexts. we identify three key challenges that make semi-structured tables valuable for evaluating longcontext understanding, aspects that current evaluation benchmarks often fail to adequately address: traditional approaches like text-to-sql @xcite @xcite or retrieval-based methods @xcite @xcite often falter when confronting real-world tables with irregular, mixed expressions @xcite , where success depends not on mechanical parsing but on the nuanced integration of world knowledge, domain-specific information, and contextual awareness. to facilitate further research in these directions, we introduce longtablebench-the first comprehensive multitask benchmark for longcontext semi-structured table reasoning. specifically, longtablebench consists of 6 core task categories: exact matching, basic conditional filtering, fuzzy conditional manipulation, fact retrieval, external knowledge fusion, and irregular numerical interpretation. these tasks span common longtext applications and incorporate combinations of single/multi-turn dialogues and single/multi-table scenarios. meanwhile, we offer 7 widely used semi-structured table formats-markdown, html, json, latex, sql, and csv, for each question to assess llms' format preferences. the benchmark includes 5,950 fully english test samples, with the task-wise length distribution presented in figure 1 . to ensure cost-effective automated evaluation, we use the f1 metric to compare output similarity with ground truth. longtablebench provides a systematic evaluation of current llms' abilities on real-world semi-structured data, such as structural reasoning, long-range dependencies, and semantic integration. our contributions are summarized as follows: table-related benchmarks. existing tableqa benchmarks like tabfact @xcite and wikitq (pasupat and liang, 2015) feature small wikipedia-derived tables, while hitab @xcite and hybridqa @xcite introduce more complex tables that still fit within early llms' context windows (4k-8k tokens). text-to-sql tasks @xcite typically focus on schema rather than full tables and struggle with non-database formats and fuzzy semantics, as tag @xcite demonstrates. though tqa-bench @xcite begins exploring long-table qa, its 64k-token limit and multiple-choice format constrain evaluation depth. with modern llms supporting 128k tokens, we introduce longtablebench to systematically evaluate fine-grained llm capabilities across full 128k-token contexts. ## conclusion we present longtablebench, a comprehensive benchmark for evaluating long-context reasoning over semi-structured tables across diverse formats, domains, and tasks. by combining realistic table construction, multi-turn and multi-table settings, and extensive evaluations across 52 llms, longtablebench uncovers key limitations in current models' structural, temporal, and semantic understanding. our findings highlight the need for robust, format-agnostic reasoning capabilities and suggest future research directions in long-context tabular comprehension. we hope this benchmark serves as a valuable resource for the community and drives progress toward more capable and generalizable llms.