Aestar at S em E val-2025 Task 8: Agentic LLM s for Question Answering over Tabular Data (2025)

Abstract: Question Answering over Tabular Data (Table QA) presents unique challenges due to the diverse structure, size, and data types of real-world tables. The SemEval 2025 Task 8 (DataBench) introduced a benchmark composed of large-scale, domain-diverse datasets to evaluate the ability of models to accurately answer structured queries. We propose a Natural Language to SQL (NL-to-SQL) approach leveraging large language models (LLMs) such as GPT-4o, GPT-4o-mini, and DeepSeek v2:16b to generate SQL queries dynamically. Our system follows a multi-stage pipeline involving example selection, SQL query generation, answer extraction, verification, and iterative refinement. Experiments demonstrate the effectiveness of our approach, achieving 70.5% accuracy on DataBench QA and 71.6% on DataBench Lite QA, significantly surpassing baseline scores of 26% and 27% respectively. This paper details our methodology, experimental results, and alternative approaches, providing insights into the strengths and limitations of LLM-driven Table QA.

## introduction question answering over tabular data (table qa ) is a fundamental problem in natural language processing (nlp) that aims to retrieve structured information from tables given natural language queries. this task is crucial for making structured data more accessible, enabling users to interact with databases, spreadsheets, and structured documents without requiring expertise in sql or database querying. however, table qa presents unique challenges compared to traditional open-domain qa, as it requires models to understand schema structures, perform logical reasoning over tabular relationships, and generate precise queries that extract relevant data @xcite . * these authors contributed equally to this work. early approaches to table qa relied on rulebased systems @xcite , manually designed templates, and retrieval-augmented generation (rag) @xcite . while effective for constrained domains, these methods struggle with scalability, particularly when handling large and diverse tables with complex relationships. more recent advances employ neural models for end-toend table qa, including methods that directly generate sql queries from natural language inputs. large language models (llms) have demonstrated impressive capabilities in this space, enabling the dynamic generation of sql queries without requiring predefined schemas or manually curated rules @xcite . despite these advancements, several challenges remain. llm-generated sql queries often suffer from structural errors, incorrect column selections, and ambiguous reasoning over tabular data. additionally, extracting precise answers from retrieved sql results requires careful post-processing, as llms may misinterpret numerical values, categories, or required formats. to address these issues, verification and refinement steps are necessary to ensure query correctness and answer reliability. in this work, we present an llm-driven natural language to sql (nl-to-sql) pipeline that dynamically translates user questions into sql queries, retrieves structured data, and refines the final output to maximize answer accuracy. our approach integrates multiple stages, including example selection, query generation, answer extraction, and verification. by leveraging llms for both sql generation and answer refinement, we aim to improve robustness across diverse table structures and query types. we apply our system to the semeval 2025 task 8 (databench), a benchmark designed to evaluate table qa over real-world datasets. by refining query formulation and integrating a verification mechanism, our system significantly outperforms baselines. in this paper, we detail our system architecture, performance evaluation, and key insights derived from our experiments. ## related work the task of question-answering on tabular data (table qa ) involves extracting precise, grounded answers from structured data based on natural language queries. various methods have been explored to tackle this problem, yet challenges such as data sparsity, feature heterogeneity, contextbased interconnections, and order invariance remain significant @xcite . previous research in table qa has introduced generative, extractive, and retriever-reader-based methods, each addressing different aspects of reasoning and information retrieval @xcite . generative models, such as those introduced in @xcite generate answers directly instead of producing logical forms. extractive methods, in contrast, select spans of text from tables rather than generating them, relying on effective table cell representations to capture only relevant information. structure-aware approaches like tapas @xcite incorporate row/column embeddings to encode positional information, while models like tableformer @xcite introduce attention bias techniques to enhance table reasoning. while both generative and extractive models excel at handling simple queries, their performance deteriorates on reasoning-based queries that require logical inference and multi-hop reasoning. @xcite among these, nl-to-sql has emerged as a powerful approach @xcite , translating natural language queries into structured sql statements for efficient information retrieval. traditional nl-to-sql models followed encoderdecoder-based architectures suited for structured databases, but real-world applications often involve semi-structured and free-form tables, necessitating alternative techniques @xcite . the advent of large language models (llms) has brought a paradigm shift to nl-to-sql. unlike traditional chat-based completion models, reasoning-driven llms excel at understanding complex question intent, handling multi-step logical reasoning, and adapting to diverse database schemas with minimal training. researchers have also experimented with prompt engineering and fine-tuning to improve sql query generation efficiency. chain-of-thought (cot) prompting enables llms to break down complex queries step by step, further enhancing reasoning capabilities. llm-based approaches have demonstrated superior evaluation metrics on benchmark datasets such as spider @xcite , surpassing traditional models. as research progresses, improvements in model size, reasoning capabilities, and dataset quality are expected to drive further performance gains, solidifying llms as the dominant paradigm for table qa . ## task description the semeval 2025 task 8 (grijalba et al., 2025), known as databench, is designed to evaluate systems that answer questions using real-world tabular datasets. the challenge comprises two subtasks: the databench benchmark encompasses 65 diverse real-world datasets, each accompanied by multiple questions. these datasets span various domains and exhibit a wide range of data types and table sizes, challenging systems to adapt to different structures and content. the questions associated with these datasets are designed to elicit various response types, including numerical values, categorical data, boolean judgments, and lists. this diversity necessitates that participating systems possess robust capabilities in understanding and processing heterogeneous tabular data to generate accurate and contextually appropriate answers. ## methodology our approach leverages a natural language to sql (nl-to-sql) agent to generate structured queries that retrieve relevant information from tabular data. the system follows a multi-stage pipeline consisting of example selection, sql query generation, answer extraction, verification, and iterative refinement. this framework ensures high accuracy and adaptability across datasets with varying structures and question complexities. alternative methods such as rule-based retrieval and rag were tested but proved less effective, especially when handling large-scale tabular reasoning tasks. additionally, we experimented with multiple models on the same pipeline to optimize performance, including gpt-4o, gpt-4o-mini, and deepseek v2:16b. our system follows the following fivestage pipeline. ## results to evaluate the effectiveness of our natural language to sql (nl-to-sql) query agent, we conducted experiments on two benchmark datasets -databench and databench-lite, using three large language models (llms): gpt-4o, @xcite . additionally, we compared our results to the provided baseline model to establish a reference point for performance @xcite . gpt-4o consistently demonstrated superior performance compared to other models. its ability to accurately interpret and break down user queries played a crucial role in generating results in the expected format. this allowed for more precise and contextually relevant outputs, particularly in complex scenarios. gpt-4o-mini also exhibited many of the advantages of gpt-4o, particularly in question understanding and structured output generation, though to a lesser extent due to its smaller model size and optimization for cost-effective applications. deepseek v2:16b showed some capability in processing structured queries but lacked the same level of precision, adaptability, and ability to follow prompt instructions. for evaluation, the organizers rank the system based on accuracy of the answers on the question answering task. our approach ranked 10th on the databench task proprietary model leaderboard and 9th on the databench-lite task proprietary model leaderboard, demonstrating the capabilities of our system. these methodological improvements enabled gpt-4o to achieve the highest accuracy (70.50% on databench and 71.65% on databench-lite), clearly outperforming the baseline performance. even deepseek-v2:16b, despite its lower overall accuracy, showed a significant improvement over the baseline, demonstrating the effectiveness of our multi-stage nl-to-sql framework. ## conclusion the results on smaller datasets like databench-lite illustrate the effectiveness of our pipeline in generating accurate and well-structured sql queries for constrained datasets. furthermore, the consistent performance observed on larger datasets demonstrates the scalability of our framework, making it well-suited for enterprise applications that require high reliability and adaptability across diverse corporate databases. this robustness ensures that organizations can leverage our approach for complex query generation at scale, improving efficiency and decision-making processes. these findings reinforce the potential of llmdriven nl-to-sql systems as a scalable and efficient solution for automating database interactions. the integration of structured reasoning and verification mechanisms represents a significant step toward improving the accuracy and interpretability of automated question answwering over tbaular data.