UCSC at S em E val-2025 Task 8: Question Answering over Tabular Data (2025)

Abstract: Table question answering (Table QA) remains challenging due to the varied structures of tables and the complexity of queries, which often require specialized reasoning. We introduce a system that leverages large language models (LLMs) to generate executable code as an intermediate step for answering questions on tabular data. The methodology uniformly represents tables as dataframes and prompts an LLM to translate natural-language questions into code that can be executed on these tables. This approach addresses key challenges by handling diverse table formats, enhancing interpretability through code execution. Experimental results on the DataBench benchmarks demonstrate that the proposed code-then-execute approach achieves high accuracy. Moreover, by offloading computation to code execution, the system requires fewer LLM invocations, thereby improving efficiency. These findings highlight the effectiveness of an LLM-based coding approach for reliable, scalable, and interpretable Table QA.

## introduction as structured data becomes increasingly prevalent across a wide range of domains-such as finance, healthcare, scientific research, and business-the task of answering questions over tabular data (table qa ) has emerged as a critical challenge in natural language processing (nlp) @xcite . despite recent advancements in large language models (llms) and retrieval-augmented generation (rag) @xcite , the inherent complexity of table structures continues to pose significant difficulties. many tables contain nested headers, multi-row dependencies, and implicit relationships, which collectively complicate reasoning and information retrieval processes @xcite ). 1 our code can be found here https://github.com/ nengwan/tabularqa2024 to address these challenges, the databench benchmark provides a structured framework for evaluating @xcite . however, achieving high performance on databench remains difficult, as existing models often struggle to reason over extensive tables, handle intricate queries, and produce clear, interpretable answers. in this study, we propose a system that harnesses the coding capabilities of large language models (llms) to autonomously generate, validate, and execute code for extracting precise answers from designated datasets @xcite . by providing the llm with a given question and an initial preview of the dataset, we prompt it to generate code that retrieves the relevant information. furthermore, we implement both immediate and post-execution verification mechanisms to enhance the accuracy of the generated responses. our evaluation examines multiple models, including llama3-8b (grattafiori et al., 2024), gpt-4o-mini @xcite , and o1-mini @xcite . although the transition to gpt-based models yields substantial improvements in test set accuracy, certain challenges persist-particularly the system's limited capacity for self-reflection and self-erroridentification. this paper provides an in-depth analysis of the system architecture, presents detailed ablation studies, and evaluates model performance, thereby highlighting both the strengths and limitations of the proposed approach. that require aggregation, filtering, and multi-hop reasoning. gold-standard annotations ensure reliable evaluation. during development, we were provided with training and development sets, each containing seven columns: question, answer, type, column used, column type, sample answer, and dataset. the test set, in contrast, includes only question and dataset columns for answer generation. the train-dev datasets comprise 65 source tables, ranging from celebrity tweets, forbes billionaire lists, and billboard lyrics. the distribution of different dataset domains is illustrated in figure 1 the test set consists of 15 datasets, each available in two versions: a full dataset and a lite version. task test_all contains 1468 rows, whereas task test_lite is a significantly smaller subset with only 18 rows (approximately 1% of the full dataset). this reduction in data volume significantly impacts answer accuracy, as discussed in later sections. we participated in both tracks. ## system overview we utilize the coding capabilities of large language models (llms) to generate code to query the data. initially, we provide the model with the given question along with the first five rows of the designated dataset. in the initial prompt, we instruct the llm to generate code capable of extracting the necessary information to produce the correct answer. once the code is generated, we employ two verification approaches. (i) immediate validation of the generated code, allowing the llm to make corrections if necessary. (ii) correct the code after execution: if the execution fails, we provide the llm with the error message, prompting it to generate a revised, executable version of the code. finally, we obtain and output the results derived from the corrected code. ## conclusion in this study, we introduced a code-generationbased approach to table question answering (table qa) using large language models (llms). by translating natural language questions into executable code, our method improves interpretability, reduces llm invocations, and ensures high accuracy across diverse table formats. evaluation on the databench benchmark demonstrated its effectiveness, with explicit answer formatting and deferred code validation enhancing performance. while o1-mini achieved the best test set accuracy, tradeoffs in computational efficiency were observed. despite challenges like ambiguous queries and occasional hallucinations. our findings highlight the promise of llm-driven code execution for scalable and interpretable table qa.