ITUNLP at S em E val-2025 Task 8: Question-Answering over Tabular Data: A Zero-Shot Approach using LLM -Driven Code Generation (2025)

Abstract: This paper presents our system for SemEval-2025 Task 8: DataBench, Question-Answeringover Tabular Data. The primary objective ofthis task is to perform question answering ongiven tabular datasets from diverse domains;under two subtasks: DataBench QA (SubtaskI) and DataBench Lite QA (Subtask II). Totackle both subtasks, we developed a zero-shotsolution with a particular emphasis on lever-aging Large Language Model (LLM)-basedcode generation. Specifically, we proposeda Python code generation framework, utiliz-ing state-of-the-art open-source LLMs to gen-erate executable Pandas code via optimizedprompting strategies. Our experiments revealthat different LLMs exhibit varying levels ofeffectiveness in Python code generation. Addi-tionaly, results show that Python code genera-tion achieves superior performance in tabularquestion answering compared to alternative ap-proaches. Although our ranking among zero-shot systems is unknown at the time of this pa-per’s submission, our system achieved eighthplace in Subtask I and sixth place in Subtask IIamong the 30 systems that outperformed thebaseline in the open-source models category.

## introduction question answering (qa) is a fundamental task in natural language processing (nlp), where the most relevant answers are retrieved from a given document or plain text. apart from such unstructured data, working with widely used structured data is crucial for real-world applications. moreover, structured data encompasses a much broader semantic scope. one important form of structured data is tabular data, which consists of rows with a consistent set of features. unlike unstructured documents, tabular data exhibits complex and heterogeneous relationships that require specialized processing techniques. information retrieval from tabular data is typically performed using various sql queries and similar approaches. however, these methods depend on rigid rule-based systems and fail to consider the semantic properties of the data. consequently, natural-language queries over tabular data face significant limitations. as a result, question-answering systems developed for tabular data have garnered significant interest among researchers. the process of converting a natural language query into a machine-executable logical form is known as semantic parsing @xcite . early studies primarily focused on datasets that required adapting specific logical forms for each table structure type. this approach, however, led to suboptimal performance, particularly in tabular structures spanning multiple domains @xcite . on the other hand, endto-end trained transformers are widely employed, as they handle both question/query interpretation and reasoning over tabular data @xcite . the recent advancements in llms have become a pivotal focus in tabular question answering, as in many other problem domains. however, llmbased approaches introduce several challenges, including high computational costs and limited context length, making scalable and efficient tabular qa systems an open research problem. to address these challenges and foster the development of effective tabular question-answering methods, @xcite has been designed to introduce the necessary level of difficulty through two distinct subtasks. in this paper, we propose a zero-shot system to address these tasks, focusing primarily @xcite , and llama-3.3-70b-instruct (ai@meta, 2024). we employ efficient prompting strategies to generate executable python pandas foot_0 library code. to enhance llm understanding of tabular structures, the generated python code is executed in a controlled environment. a key feature of our system is its iterative error-handling mechanism. if the initial code execution fails, the error message and faulty code are sent back to the llm for correction, with a maximum of two iterations. this mechanism significantly improves robustness, reducing failure rates in complex queries. we observe that one model in our pipeline achieves the highest accuracy on subtask i (84.67%), while another leads subtask ii (85.05%), both without task-specific fine-tuning. all code is available on our github repository foot_1 . ## related work this section reviews recent developments in llms, focusing on their applications in tabular question answering. in recent years, the emergence of the transformer architecture @xcite has led to remarkable advancements in language modeling tasks. this progress has resulted in state-of-theart performance across various nlp tasks. consequently, the application of transformer architectures to problems requiring tabular modeling has become inevitable. early studies primarily focused on different embedding mechanisms @xcite , pre-training strategies @xcite , and architectural modifications @xcite . the core approach introduced by these methods was pre-training transformer architectures from scratch for tabular data @xcite . however, this approach faces efficiency and scalability limitations, particularly when models need to generalize across multiple domains. generally, pre-trained language models struggle to adapt efficiently to task-specific tabular datasets. more recently, the emergence of llms has brought about a significant transformation in the field. models such as @xcite and llama @xcite have demonstrated strong few-shot and zero-shot capabilities, achieving state-of-the-art performance across various tasks while often requiring little to no task-specific data. these advancements have enabled the use of a single, unified model for solving complex tabular tasks. the transition from training models from scratch or adapting pre-trained language models to leveraging llms represents a significant paradigm shift in tabular data processing. however, the application of llms to tabular question answering introduces several challenges. one major limitation is the context length constraint inherent to llms. when processing large or multiple tables, the limited context size prevents the model from encoding all necessary information. additionally, handling multiple tables often leads to hallucinations, where models generate inaccurate or misleading responses. to overcome these limitations, researchers have leveraged the in-context learning capabilities of llms. the effectiveness of llm-based approaches largely depends on how tabular data and question queries are represented and utilized. for tabular data, appropriate table schemas and prompting strategies incorporating relevant examples are designed to enhance model comprehension. query representation can also significantly impact performance. a common strategy involves decomposing complex queries into step-by-step subqueries, improving model interpretability @xcite . another approach is transforming queries into intermediate representations such as python code or sql queries, enabling structured execution @xcite . these advancements have led to models capable of performing task-specific reasoning without requiring additional fine-tuning. building on insights from previous studies, we find that effectively addressing semeval-2025 task 8 requires a deep understanding of query semantics and table structures, as well as the ability to generate accurate answers across diverse answer formats. motivated by these challenges, we introduce a novel framework that integrates schemaguided prompting, controlled execution, and an error-handling mechanism. our extensive evaluations and prompt strategy experiments highlight the effectiveness of our approach in enhancing accuracy and robustness. these findings show the practicality and applicability of the proposed approach in real-world scenarios, where tabular data must be processed dynamically without requiring task-specific fine-tuning. ## data the original databench dataset (osés grijalba et al., 2024) provides 1308 questions from 65 different domains, each containing question-answer pairs written in english. during the competition, this dataset was expanded with the addition of a new test split @xcite . the exact dataset statistics are presented in although the dataset provides structured train and development splits with detailed annotations, this study did not utilize these data for training, as we preferred a zero-shot approach that does not involve fine-tuning. ## system overview our approach involves two main steps in providing an answer to questions over tabular data: preprocessing and then code generation and execution. the complete workflow is illustrated in figure 1 . ## experimental setup our zero-shot framework was tested on the officially released development and test datasets of semeval 2025 task 8, covering its two subtasks @xcite . the models used in our system were selected based on their performance in code generation tasks, ensuring their effectiveness in handling structured and semi-structured tabular question answering. additionally, we opted for a maximum of two iterations based on preliminary experiments, which showed that attempts beyond two iterations rarely produced further improvements. to provide a more comprehensive error analysis, we also conducted additional experiments with three iterations. to evaluate system performance, we used accuracy, the official evaluation metric of semeval 2025 task 8. furthermore, we analyzed the impact of our iterative error-handling mechanism on execution reliability by measuring error reduction rates across different models. these evaluations provide insights into both models accuracy and execution robustness in tabular question answering. ## results the performance of the models is presented in our results indicate that one of the deepseek models (i.e., deepseek-r1 and deepseek-v3) outperforms all other models across both subtasks. we see that deepseek-v3 falls behind all the others on the development sets, but performs better specifically on the test set of subtask i. deepseek-r1, which is a subsequent iteration, building upon v3 with enhanced capabilities via reinforcement learning, outperforms qwen2.5-coder-32b-instruct and llama-3.3-70b-instruct models on all tasks and datasets, falling behind deepseek-v3 by 0.52 percentage points on the subtask i test set. moreover, in the official evaluation within the open-source models category, our best-performing model ranked eighth in subtask i and sixth in subtask ii, placing among the 30 systems that outperformed the baseline. these results further highlight the effectiveness of our approach in zero-shot tabular question answering. at the time of this paper's submission, due to a lack of information on other solutions, we were unable to evaluate our performance relative to other zero-shot systems in the competition. through our manual observations, we identified that the test datasets are significantly more challenging. however, we do not believe that every question-answer pair in these datasets can perfectly represent the real-world performance of the models. nonetheless, the widening performance gap in the more challenging test sets suggests that deepseek-r1 may generalize to the problem more effectively, providing evidence of its in addition, as shown in to analyze error patterns and the impact of our correction mechanism in greater detail, we grouped errors into three main categories: runtime, degenerate loop, and syntax. notably, the runtime category includes diverse errors such as keyerror and valueerror, but for simplicity, we report them under a single label. our findings also indicate that some errors transform into different types across iterations. we define degenerate loop errors as cases where an llm repeatedly generates identical or nearly identical output sequences, continuing indefinitely until it reaches its maximum token limit. finally, figure 2 provides an overview of error resolution across iterations, showing that most runtime errors are resolved within the first two attempts. figure 3 further breaks down specific error types, such as filenotfounderror, keyerror, and nameerror, offering a more fine-grained view. ## conclusions in conclusion, this paper presented the solution developed by the itunlp group for semeval-2025 task 8. the proposed approach addressed the tabular question answering task in zero-shot scenarios. our method yields promising results in zero-shot tabular question answering, achieving higher ranks (8 th place in subtask i and 6 th in subtask ii) within the 30 participant systems in the open-source category. since these 30 systems may have employed fine-tuning or few-shot learning techniques, further analysis would be possible upon the publication of the system description papers that achieved better results on the same category of the shared task, which will provide a clearer understanding of our ranking within zero-shot frameworks. as this study focuses only on open-source llms, future work could include evaluating proprietary llms within our proposed framework to gain a broader perspective on model performance. furthermore, the databench dataset consists of questions that require using only a single table. as future work, we aim to evaluate our zero-shot model's performance on multi-table reasoning tasks, further expanding its applicability.