Firefly Team at S em E val-2025 Task 8: Question-Answering over Tabular Data using SQL /Python generation with Closed-Source Large Language Models (2025)

Abstract: In this paper, we describe our official system of the Firefly team for two main tasks in the SemEval-2025 Task 8: Question-Answering over Tabular Data. Our solution employs large language models (LLMs) to translate natural language queries into executable code, specifically Python and SQL, which are then used to generate answers categorized into five predefined types. Our empirical evaluation highlights the superiority of Python code generation over SQL for this challenge. Besides, the experimental results show that our system has achieved competitive performance in two subtasks. In Subtask I: Databench QA, where we rank the Top 9 across datasets of any size. Besides, our solution achieved competitive results and ranked 5th place in Subtask II: Databench QA Lite, where datasets are restricted to a maximum of 20 rows.

## introduction the shared task 8 @xcite aims at building question-answering (qa) systems for tabular data using the databench benchmark @xcite , which contains 65 real-world tabular datasets from different domains, allow to assess distinct sort of questions related to each data type. this shared task has two main tasks, including databench qa and databench lite qa. the databench qa subtask requires developing a system that answers questions using datasets of any size. the databench lite qa subtask used the sampled version of each dataset with a maximum of 20 rows per tabular dataset (databench lite). the dataset involves questions with answers and their type @xcite , and list @xcite ), name of specific columns used and their types, along with the associated dataset name. the system developed by the participants will need to provide an answer which would then be compared with a gold standard. in this paper, we propose a system that uses large language models (llms) to solve shared task 8. our system is based on the gpt models and a structured query generation approach by producing either sql queries or python code to answer questions on tabular data. the generated sql or python code is executed on the dataset, and refines the output to produce the final response. the rest of the paper is organized as follows. section 2 provides the related work. the system description is presented in section 3, followed by evaluation results in section 1. the experimental setup and conclusion is discussed in section 4 and section 6, respectively. ## related work question-answering over tabular data (qaotd) has garnered significant interest due to its broad applications in structured data retrieval and decision support systems. existing approaches can be broadly categorized into rule-based methods, transformer-based architectures, and code-generation techniques, each addressing different challenges associated with querying tabular data. building on previous research, @xcite proposed a system that enables non-technical users to query open datasets using natural language. their approach extends an existing chat-bot interface for metadata-based search by incorporating content-based retrieval from tables, allowing users to pose queries and obtain answers directly from structured data. compared to earlier deep learning-based models, such as @xcite , which focus on cross-table search, and @xcite , which explore learning aggregation operations, this system offers a more lightweight and user-friendly 1028 solution, reducing computational overhead while enhancing usability for open data exploration. to improve numerical reasoning and compositional query execution, @xcite introduced unirpg, a program synthesis approach designed for discrete reasoning over both tables and text. their method leverages a neural programmer to generate executable programs, ensuring syntactic correctness and improving reliability in arithmetic computations. prior research, such as @xcite , explored converting natural language questions into executable python programs, enabling flexible table processing and api integration. while these approaches enhance structured reasoning, they remain sensitive to execution errors and struggle to generalize to unseen table schemas. more recently, advances in large-scale pretrained language models have significantly impacted table-based question answering. @xcite conducted a study assessing the table reasoning capabilities of large language models (llms) and multimodal llms, analyzing their performance across different prompting techniques and table formats. their findings, published in the findings of acl 2024, indicate that while llms exhibit strong generalization abilities, they often face challenges in numerical reasoning and logical consistency. compared to earlier table-based qa models such as tapas @xcite , which leverage weak supervision for table parsing or tapex @xcite , which achieves table pre-training by learning a neural sql executor on a synthetic corpus, llms offer greater flexibility but remain prone to hallucinations when dealing with structured information. ## experimental setup data and preprocessing: we utilized the official development set for system development. as the code prompt sql prompt @xmath0.mean() if @xmath1: print("our average employee is older than 35") else: print("our average employee is not older than 35") select avg("age") > 35 as is_older_than 35 from "066_ibm_hr" where "age" is not null question: is our average employee older than 35? column_name: age values in column "age": [41, 49, 37, 33, 27] input figure 2: examples of generated python and sql code. evaluation metrics: the evaluation metric for two subtasks is accuracy score between submission and test gold set. these evaluation metrics were provided by the task organizers and are available on github. ## results and discussion in this section, we present the official results of our final submission model for two tasks and the accuracy results split by question type for each task in the task 8 shared task competition. we also compare results with the results from the five top teams for each task. subtask i: databench qa subtask ii: databench lite qa ## conclusion in this paper, we presented a system for question-answering on tabular data in the semeval-2025 task 8. our approach leverages large language models to generate column descriptions, select relevant columns, and produce executable code to derive final answers. the system efficiently processes diverse tabular datasets and generates accurate responses without requiring additional training data. our experiments demonstrate competitive performance across different datasets in databench. for future work, we plan to enhance our system's ability to handle more complex table structures and improve its accuracy in missing or ambiguous data. additionally, exploring better prompting strategies for large language models could further optimize system performance.