How well do LLM s reason over tabular data, really? (2025)

Abstract: Large Language Models (LLMs) excel in natural language tasks, but less is known about their reasoning capabilities over tabular data. Prior analyses devise evaluation strategies that poorly reflect an LLM’s realistic performance on tabular queries. Moreover, we have a limited understanding of the robustness of LLMs towards realistic variations in tabular inputs. Therefore, we ask: Can general-purpose LLMs reason over tabular data, really?, and focus on two questions 1) are tabular reasoning capabilities of general-purpose LLMs robust to real-world characteristics of tabular inputs, and 2) how can we realistically evaluate an LLM’s performance on analytical tabular queries?Building on a recent tabular reasoning benchmark, we first surface shortcomings of its multiple-choice prompt evaluation strategy, as well as commonly used free-form text metrics such as SacreBleu and BERT-score. We show that an LLM-as-a-judge procedure yields more reliable performance insights and unveil a significant deficit in tabular reasoning performance of LLMs. We then extend the tabular inputs reflecting three common characteristics in practice: 1) missing values, 2) duplicate entities, and 3) structural variations. Experiments show that the tabular reasoning capabilities of general-purpose LLMs suffer from these variations, stressing the importance of improving their robustness for realistic tabular inputs.

## introduction large language models (llms) are intended for general-purpose usage and particularly excel on natural language tasks represented in text @xcite . in organizations, another common modality for data analysis and decision-making, is tabular data, for which recent studies have shown promising performance of llms as well @xcite . structural analysis to understand to 1 code: github.com/trl-lab/tabular-robustness what extend the reasoning capabilities of llms pertain, realistically, in more complex tabular reasoning tasks, such as analytical aggregations, is still lacking. without reliable knowledge of their failure modes on tabular inputs and tasks, though, we risk unwarranted usage of these models in practice and delayed development of more robust capabilities. surfacing the reasoning capabilities on tabular tasks is, however, not straightforward. most studies adopt free-form text metrics, which hardly capture reliable reasoning accuracy due to different formatting of the ground-truth answers, particularly of analytical questions, versus long-form llm-generated responses @xcite . the alternative of forcing certain output formats @xcite yields a limited understanding of the open-form reasoning performance and is prone to parsing errors, while another alternative of including the ground-truth answer in a multiplechoice prompt @xcite leaks ground-truth answers into the prompt compromising its reliability. beyond realistic evaluation procedures, in order to use llms in a reliable manner for tabular reasoning tasks, it is important to understand how well they can handle the characteristics of tabular data inputs @xcite as encountered in practice. to close these gaps, we first address the question: how can we realistically evaluate an llm's performance on analytical tabular queries? we examine the limitations of existing evaluation metrics, such as sacrebleu @xcite and bertscore @xcite , as the distributions are these among correct and incorrect answers are inseparable. instead, we propose using the llmas-a-judge evaluation method @xcite for more reliable performance insights and show, through calibration with human annotations, that the llm-as-a-judge provides a reliable signal of tabular reasoning performance. using this evaluation procedure, we unveil a significant gap in tabular reasoning accuracy than previously found in the existing tqa-bench benchmark for tabular reasoning @xcite . second, through comprehensive analysis we answer the question: are llms robust to real-world characteristics of multi-table inputs? building on the tqa-bench, we first improve the validity of the task queries and downscale multi-table inputs to gain more fine-grained insights. we then inspect the robustness of llms against characteristics of tabular inputs as commonly found in practice. specifically, we formalize the following three characteristics: missing values, duplicate entities, and structural variations. we show that most llms are not as robust to, and insufficiently acknowledge the presence of, such quality issues or anticipated variations in multi-table inputs, highlighting the need for more robust models for tabular reasoning. we make the following concrete contributions: ## the tqa-bench and revisions here, we explain the tabular reasoning tasks included in the tqa-bench that we use to assess the reliability of evaluation metrics as well as the robustness of the tabular reasoning capabilities of llms. we explain revisions we made to invalid queries, and tabular inputs to gain granular insights. ## towards reliable evaluation of tabular reasoning capabilities evaluating multi-table reasoning, and generally free-form question answering, still is an open challenge @xcite . while context and reasoning traces of llm-generated answers are useful, they complicate evaluation when ground-truth answers are short and exact, as is the case in typical tabular tasks. when the llm-generated answers are evaluated against the short ground-truth answers by two freeform text metrics, sacrebleu @xcite and bert-score @xcite , their scores are inconclusive. for example, for the queries in ## conclusion while recent studies suggest llms exhibit reasonable tabular reasoning abilities beyond natural language tasks, these studies often lack reliable evaluations and robustness checks, prompting our study into how well llms truly reason over tabular inputs. first, we surface limitations of common free-form text evaluation metrics, such as sacre-bleu, which fail to distinguish between correct and incorrect answers in tabular reasoning tasks. we demonstrate that an llm-as-a-judge is more reliable for this purpose. a revised evaluation of an existing benchmark with the llm-as-a-judge unveils a significant deficit in tabular reasoning capabilities of llms. second, we analyze the robustness of tabular reasoning capabilities of llms through queries of various complexities and find that they can be sensitive to realistic variations like missing values, even for relatively simple tasks. moreover, we find that llms insufficiently acknowledge such undesired variations risking errors in downstream interpretation. these findings underscore the need for further advancements in llm architectures and training to improve their robustness to real-world tabular data.