T ab XE val: Why this is a Bad Table? An e X haustive Rubric for Table Evaluation (2025)

Abstract: Evaluating tables qualitatively and quantitatively poses a significant challenge, as standard metrics often overlook subtle structural and content-level discrepancies. To address this, we propose a rubric-based evaluation framework that integrates multi-level structural descriptors with fine-grained contextual signals, enabling more precise and consistent table comparison. Building on this, we introduce TabXEval, an eXhaustive and eXplainable two-phase evaluation framework. TabXEval first aligns reference and predicted tables structurally via TabAlign, then performs semantic and syntactic comparison using TabCompare, offering interpretable and granular feedback. We evaluate TabXEval on TabXBench, a diverse, multi-domain benchmark featuring realistic table perturbations and human annotations. A sensitivity-specificity analysis further demonstrates the robustness and explainability of TabXEval across varied table tasks. Code and data are available at https://corallab- asu.github.io/tabxeval/.

## tabxeval we establish a transparent and systematic protocol for table evaluation by comparing a reference table-the candidate table produced by a human or an llm with a ground-truth table. these tables may differ in formatting, interpretation, or unit representation, necessitating a rigorous evaluation framework. for instance, an llm-generated table might omit entire rows or abbreviate numeric values (e.g., "100k" vs. "100,000"), while a humancurated table may specify units only in the header (e.g., "velocity" vs. "velocity (m/s)"). to address such discrepancies, we design a set of rules based rubrics tabxeval, which aim to improve the reliability and interpretability of table evaluation. ## experiments to validate efficacy of tabxeval, we conduct experiments using our synthetic dataset tabxbench. we report @xcite and llama-3.3-instruct results for our framework tabxeval for both components tabalign and tab-compare. baselines. our evaluation compares tabxeval against a broad range of baselines, which we classify into deterministic and non-deterministic approaches. deterministic metrics (e.g., exact match (em), chrf, rouge-l) yield fixed outputs based on string-or character-level comparisons, ensuring reproducibility. in contrast, non-deterministic metrics (e.g., bertscore, bluert, h-score) leverage contextualized neural representations to capture subtle linguistic nuances albeit with potential variability. we also include two recent methods: p-score, an llm-based metric outputting scores on a 0-10 scale, and tabeval, an embeddingbased method that unrolls tables using an llm and computes entailment via roberta-mnli. for a fair comparison, we further propose a direct-llm baseline that is prompted with the same evaluation rubric detailed in section 2.1. ## comparison with related work text-to-table generation . early text-to-table research exploited single-domain corpora such as rotowire for basketball summaries @xcite , the e2e restaurant set @xcite , wikibio infobox-biography pairs @xcite , and wikitabletext @xcite . while pioneering, these resources offer limited schema variety and often encourage hallucinated or under-structured outputs. recent collections address these gaps: structbench permutes headers, merges columns, and shuffles schemas to test structural generalisation @xcite , 2024), whereas tanq requires multi-hop, multisource reasoning to generate answer tables @xcite . such challenging benchmarks expose systematic weaknesses in both generation models and legacy evaluation metrics, motivating the finegrained rubric employed by tabx. other evaluation metrics 1. surface and embedding overlap. classic n-gram scores bleu @xcite , rouge-l @xcite , meteor (banerjee and lavie, 2005), and chrf @xcite treat a table as flat text, ignoring header alignment or cell hierarchy. embedding-based bertscore @xcite improves semantic sensitivity, yet still overlooks structural fidelity. token-level exact match and parent @xcite partially reward factual grounding, but cannot detect column swaps or unit shifts. despite these advances, existing metrics still emphases either semantics or structure and provide a single numeric values with limited error traceability or explainability. our two-phase tabx closes this gap by disentangling alignment (tabalign) from cell-level comparison (tabcompare), producing an interpretable score that balances sensitivity and specificity. ## conclusion and future work in this work, we have introduced tabxeval an exhaustive and explainable, two-phase framework that transforms table evaluation by disentangling structural alignment from detailed cell-level comparison. our method leverages a comprehensive rubric to quantify both coarse and fine-grained errors, yielding results that strongly correlate with human assessments. by developing tabxbench, a challenging multi-domain benchmark with diverse perturbations, we have demonstrated the robustness, explainability, and human-alignment of our approach. while limitations such as computational overhead and handling of hierarchical tables remain, the promising performance of tabxeval opens avenues for further research and refinement in automatic table evaluation. we found two key directions for future work. firstly, while our current approach leverages large language models to ensure robustness and generalization to unseen table structures, their computational overhead can hinder practical deployment. developing more compact alternatives using smaller models (e.g., bart, t5) would require large-scale, heterogeneous fine-tuning data to preserve performance on out-of-distribution tables-a resource that is currently limited. future efforts could explore model distillation or semi-supervised training to create lightweight yet reliable variants of tabxeval. secondly, although tabxeval effectively handles structural variations-such as merged or decomposed cells-through llm-guided alignment, directly supporting complex hierarchical tables (e.g., nested cells or multi-level headers) remains a challenge. a promising direction is to incorporate a preprocessing step that flattens hierarchical structures into standardized key paths (e.g., transforming "release date" with sub-headers "month" and "year" into "release date.month" and "release date.year"), enabling compatibility with our approach while preserving semantic structure.