S cotty P oseidon at S em E val-2025 Task 8: LLM -Driven Code Generation for Zero-Shot Question Answering on Tabular Data (2025)

Abstract: Tabular Question Answering (QA) is crucial for enabling automated reasoning over structured data, facilitating efficient information retrieval and decision-making across domains like finance, healthcare, and scientific research. This paper describes our system for the SemEval 2025 Task 8 on Question Answering over Tabular Data, specifically focusing on the DataBench QA and DataBench Lite QA subtasks. Our approach involves generating Python code using Large Language Models (LLMs) to extract answers from tabular data in a zero-shot setting. We investigate both multi-step Chain-of-Thought (CoT) and unified LLM approaches, where the latter demonstrates superior performance by minimizing error propagation and enhancing system stability. Our system prioritizes computational efficiency and scalability by minimizing the input data provided to the LLM, optimizing its ability to contextualize information effectively. We achieve this by sampling a minimal set of rows from the dataset and utilizing external execution with Python and Pandas to maintain efficiency. Our system achieved the highest accuracy amongst all small open-source models, ranking 1st in both subtasks.

## introduction tabular question answering (qa) is a critical area in natural language processing (nlp) that enhances data accessibility and facilitates automated information extraction from structured datasets. the semeval 2025 task on "question answering over tabular data" @xcite advances this field by introducing databench @xcite , a benchmark comprising diverse, large-scale tabular datasets spanning multiple domains. the task challenges participants to develop systems capable of answering questions over these datasets, with two subtasks: databench qa (full datasets) and databench lite qa (sampled datasets with a maximum of 20 rows) to support models with limited context windows. expected answer types include boolean, category, number, or lists of these values. our system employs a code generation approach using python foot_0 and pandas foot_1 to extract answers from tabular data using open-source large language models (llms). given a table and a question, our system generates executable code that loads the table, performs the necessary computations, and returns the answer. this approach ensures interpretability, as the reasoning process is encoded explicitly in the generated code, and it remains agnostic to the table structure. we explore both a multi-step agentic chain-of-thought (cot) approach, where one llm generates structured reasoning steps and another translates them into executable code, and a unified llm approach, where the model simultaneously reasons and writes code. we prioritize a zero-shot setting to minimize the amount of table data passed to the llm, optimizing for low resource consumption and scalability. our system achieved the highest accuracy among small open-sourced llms, ranking 1 st in both subtasks ( ## background our work is based on the dataset collection originally presented in the paper @xcite . the dataset comprises 65 tables designed to evaluate llms on the task of qa over structured realworld tabular data. ## system overview our system figure 1 , reframes the tabular qa task as a code generation problem in zero-shot setting. given an input dataset table d and a natural lan-guage question q, the system follows a structured pipeline. first, the dataset schema is extracted, including column names, data types, and a sample of the first three rows. this information is then used to construct a structured prompt using the system-user-assistant format that provides essential context while reducing the amount of information provided to the llm. the system proceeds to generate python code designed to load the dataset and execute the necessary computations to extract the answer. subsequently, the generated code is parsed for errors and executed to obtain the final output. to enhance robustness, we incorporate an execution-aware retry mechanism. if the generated code encounters errors, the error message is fed back to the llm, allowing it to iteratively refine its output (up to 3 retries). this significantly improves the accuracy of generated code and reduces failure. by reducing the input size, we optimize both computational efficiency and scalability, while also improving the llm's ability to effectively contextualize the given data. additionally, because our system follows a code generation approach and provides only the essential information about the tables, it can be seamlessly applied to both subtasks without requiring any modifications. this design ensures adaptability across different task configurations, further enhancing the system's versatility. ## experimental setup our experimental setup is designed to evaluate the effectiveness of our approach across both subtasks 1 and 2. we focus on zero-shot prompting using quantized llms, leveraging prompt engineering techniques to optimize performance. our methodology incorporates the system-user-assistant chat template, ensuring structured interactions with llms. to facilitate efficient inference, we employ unsloth's (daniel han and team, 2023) dynamically quantized 4-bit versions of these models, allowing for computationally efficient execution while maintaining strong performance. we used dymamic 4-bit quantized llama 3.1 (8b instruct) 4 , codellama (7b) 5 and dymamic 4-bit quantized phi-4 (8.48b) 6 models from unsloth. experiments were conducted on a single nvidia t4 gpu using google colab and kaggle, emphasizing the feasibility of achieving high performance with limited compute resources. predictions were generated on the validation and blind test set. ## results our system demonstrated exceptional performance, ranking 1 st in the "small open-source models" category (≤ 8b parameters) (refer to ## conclusion our system demonstrates the effectiveness of llmdriven code generation for zero-shot question answering on tabular data. we highlight the advantages of a unified llm approach for maintaining logical consistency and the importance of prompt engineering for guiding model reasoning effectively. additionally, we emphasize the benefits of minimizing llm input for improved contextualization, computational efficiency, and scalability. further exploration of fine-tuning techniques could improve the generation of complex aggregation queries and reduce errors in generated code. agentic systems have greater potential than what has been portrayed in this work. while our current system demonstrates the effectiveness of llms for tabular qa, we recognize that further engineering is needed to fully exploit the potential of agentic systems in this domain. future research will focus on refining the agentic framework to enable more complex reasoning and data manipulation capabilities.