A lpha P ro at S em E val-2025 Task 8: A Code Generation Approach for Question-Answering over Tabular Data (2025)

Abstract: This work outlines the AlphaPro team’s solution to SemEval-2025 Task 8: Question Answering on Tabular Data. Our system utilizes a three-stage pipeline that uses natural language questions along with the table’s structural information to generate executable Python code, which is subsequently used to query the table and produce answers. The method achieves up to 67% accuracy in task data, demonstrating the feasibility of code generation for tabular question answering. The strengths and limitations of the approach are outlined and suggestions for further research are provided. The code has been made available in a public code repository to promote reproducibility and research in this area.

## introduction large language models (llms) have demontrated impressive abilities to understand, reason and interact with structured tabular data @xcite . by integrating natural language (nl) processing with advanced reasoning, llms offer a powerful and flexible way to extract valuable insights from tabular datasets @xcite . tabular data are prevalent in various fields, from financial reports to scientific findings and statistical analyses. semeval-2025 task 8 @xcite focuses on the importance and challenge of developing systems capable of answering questions (qa) over tabular data. the ability to query tables using natural language, instead of specialized languages like sql, significantly reduces the barrier to accessing and interpreting data. earlier research on tabular datasets used a translation-based system to translate from natu-1 https://github.com/anshumanaryan24/alphaprosemeval2025-task8 ral language to sql using semantic parsing techniques @xcite and deep learning-based techniques @xcite . these methods lack schema awareness of the table and suffer from the ambiguity of natural languages. recent advancements, such as rat-sql @xcite and nl2sql @xcite , address schema integration but still encounter limitations related to semantic context representation and query diversity @xcite . traditional query languages such as sql are limited in this regard, as they primarily understand only the structural aspects of tables but struggle to capture their underlying semantics @xcite . to address the above limitations, this work proposes a schema-infused llm prompting approach leveraging few-shot learning and intermediate python code generation effectively handling the semantics. unlike traditional sql-based methods, this approach capitalizes on schema-aware question paraphrasing and dynamically generated python code, substantially improving semantic expressiveness and flexibility in handling nl queries. figure 1 illustrates the three-stage pipeline of the proposed system to answer questions posed in natural language, employing schema-aware question paraphrasing and generating python code as intermediaries. in the first stage, the answer type is predicted and the user's question is transformed into a schema-aware format, optimizing it for the subsequent stage. in the second stage, the model generates python code utilizing the pandas framework 2 . the final stage involves extracting and executing this generated python function, subsequently retrieving the results. python was selected due to its powerful libraries for manipulating tabular datasets, such as pandas, and its minimalistic and dynamically typed nature, which facilitates effective analysis of the generated code. the key contributions of this work are summarized as follows: the proposed system using the deepseek-v3 model (deepseek-ai, 2024), evaluated on the databench dataset (grijalba et al., 2024), outperforms five contemporary state-of-the-art opensource llms, achieving an average accuracy of 77.43%. ## related work recent advances in question answering (qa) over structured data have spurred research into neural architectures, semantic parsing, and program generation for tabular reasoning. ## system overview this work proposes a three-stage framework powered by large language models (llms) to generate executable python code to answer questions on tabular data sets. the overall architecture, depicted in figure 1 , comprises the following sequential components: ## experimental setup this paper evaluates proposed framework using the dataset from semeval-2025 task 8 (osés grijalba et al., 2025), made available through the databench platform @xcite . the dataset includes both training and development splits, and spans diverse domains and table schemas. this diversity enables robust testing of the system's generalization ability across different question types and data formats. llms used: this paper evaluates six llms with sizes ranging from 7b to 671b parameters, which includes general llms and opensource models : gemma-3-12b-it @xcite , llama-3.3-70b-instruct-turbo, qwen2.5-7binstruct-1m @xcite , qwen2.5-coder-14b-q5, c4ai-command-r-plus-08-2024 (cohere labs, 2024), @xcite . ## results and observations the comprehensive analysis conducted demonstrates the system's feasibility and robustness, as detailed below.