sonrobok4 Team at S em E val-2025 Task 8: Question Answering over Tabular Data Using Pandas and Large Language Models (2025)

Abstract: This paper describes the system of the son robok4 team for the SemEval-2025 Task 8: DataBench, Question-Answering over Tabular Data. The task requires answering questions based on the given question and dataset ID, ensuring that the responses are derived solely from the provided table. We address this task by using large language models (LLMs) to translate natural language questions into executable Python code for querying Pandas DataFrames. Furthermore, we employ techniques such as a rerun mechanism for error handling, structured metadata extraction, and dataset preprocessing to enhance performance. Our best-performing system achieved 89.46% accuracy on Subtask 1 and placed in the top 4 on the private test set. Additionally, it achieved 85.25% accuracy on Subtask 2 and placed in the top 9. We mainly focus on Subtask 1. We analyze the effectiveness of different LLMs for structured data reasoning and discuss key challenges in tabular question answering.

## introduction the semeval 2025 task 8 @xcite focuses on developing systems for question answering (qa) over tabular data, a critical subfield of natural language processing (nlp) with applications in business intelligence, automated data analytics, and financial reporting. unlike traditional qa tasks that rely on retrieving information from free-text documents, this challenge requires models to derive answers solely from structured, tabular data. the databench benchmark comprises 65 real-world datasets drawn from diverse domains, with each dataset accompanied by 20 human-generated questions and corresponding answers. this setup demands that models not only retrieve information accurately but also perform the necessary computations and reasoning over multiple table columns. this paper introduces a system designed to address the challenges of question answering over tabular data (tabular qa). our approach translates natural language queries into executable python code for direct interaction with the provided data. to improve reliability and performance, we incorporate a custom error recovery mechanism and leverage the power of several llms: gpt4o-mini, deepseek, and an open-source model. we evaluated these models with our proposed approach using various prompt configurations. through extensive testing and comparison, we assessed each model's performance on our tabular qa tasks, ultimately selecting the model that provided the best overall accuracy and robustness for our final system. ## related work tabular question answering (qa) has evolved considerably over the past few years, driven by the creation of diverse datasets and innovative methodological approaches. in this section, we discuss methodological advancements, emphasizing recent paradigm shifts enabled by large language models (llms). early methods translated natural language questions into logical forms (e.g., sql queries) for structured data retrieval. systems such as @xcite and spider @xcite exemplify this paradigm, achieving high accuracy under constrained conditions. however, their reliance on predefined schemas limits applicability to free-form questions and complex table structures. end-to-end neural models like tapas @xcite advanced the field by jointly encoding tables and questions using transformer architectures. tapas incorporates specialized positional embeddings to preserve table structure during pretraining, enabling direct answer prediction without intermediate sql generation. despite their effec-tiveness, such models require domain-specific finetuning and struggle with numerical reasoning tasks requiring explicit computation. recent approaches, notably @xcite , leverage large language models (llms) via prompting strategies (e.g., in-context learning) rather than task-specific fine-tuning. tables are serialized into text sequences and paired with demonstrations (e.g., "q: @xcite a: [answer]"), enabling llms like gpt-3.5/4 to perform reasoning across diverse schemas. although llms show strong generalization evidenced by databench's multi-domain evaluation they face challenges with large tables, hallucination, and precise numerical operations. ## method our system is designed to answer natural language questions based solely on the information contained in tabular datasets. the overall pipeline consists of four main components: data preprocessing, context provisioning, natural language-to-code conversion, and error-aware execution. in the following, we detail each step of our approach. ## experimental setup in this section, we describe the experimental setup, including the models, datasets, and evaluation methodology. ## results and discussion in this study, we evaluated four models: gpt-4omini, deepseek-v3, deepseek-r1, and deepseek-r1-distill-qwen-14b. initially, gpt-4o-mini was tested with three distinct prompts (a, b, and c) using the original dataset. based on these preliminary results, we identified the best-performing prompt (prompt c) and subsequently used it to compare model performance on preprocessed datasets. ## conclusion in this paper, we explored the challenges and errors encountered when using a large language model to generate executable code for tabular questionanswering tasks. our analysis categorized errors into three key types: numerical precision errors, logical errors, and insufficient information. by systematically evaluating the generated responses and their correctness, we identified the primary causes of errors, including incorrect filtering, precision issues, and reasoning failures. our findings highlight the importance of robust error-handling mechanisms when integrating llms with structured data processing. future work should focus on improving the consistency of generated codes, enhancing model understanding of numerical reasoning, and utilizing multiple llms to generate and validate answers, selecting the most reliable response based on consensus or predefined criteria. through this analysis, we provide valuable insights for researchers and practitioners working on llm-based data processing systems. our work underscores the need for a hybrid approach that combines llm reasoning and prompt engineering