I 2 R - NLP at S em E val-2025 Task 8: Question Answering on Tabular Data (2025)

Abstract: We present a Large Language Model (LLM) based system for question answering (QA) over tabular data that leverages multi-turn prompting to automatically generate executable Pandas functions. Our framework decomposes the problem into three key steps: (1) Answer Type Identification, where the system identifies the expected format of the response (e.g., boolean, number, category); (2) Pandas Function Generation, which generates a corresponding Pandas function using table metadata and in-context examples, and (3) Error Correction and Regeneration, where iteratively refining the function based on error feedback from executions. Evaluations on the SemEval-2025 Task 8 Tabular QA benchmark (Grijalba et al., 2024) demonstrate that our multi-turn approach significantly outperforms single-turn prompting models in exact match accuracy by 7.3%. The proposed system not only improves code generation robustness but also paves the way for enhanced and adaptability in table-QA reasoning tasks. Our implementation is available at https://github.com/Gyyz/Question_Answering-over-Tabular-Data.

## introduction answering natural language queries over tabular data requires a deep understanding of both linguistic nuances and structured data semantics. traditional systems rely on rule-based approaches or parsing pipelines to translate questions into database queries (e.g., sql). while effective in constrained domains, these approaches often demand significant manual engineering and domain expertise @xcite . these approaches can struggle with the complexities and ambiguities inherent in natural language. in contrast, recent advances in large language models (llms) have enabled prompt-based code generation, offering a promising alternative for complex reasoning tasks @xcite . llms have shown impressive capabilities in generating code from natural language descriptions, including for the task of text-to-sql @xcite , which focuses on translating natural language questions into sql queries. however, single-turn prompts can fail to capture all necessary subtleties, leading to generated code that is either syntactically or semantically incorrect. this limitation highlights the need for more sophisticated prompting strategies. our work addresses these challenges by introducing a multi-turn prompting framework that engages the llm in several refinement iterations. unlike single-turn generation, our approach mirrors the iterative debugging process of human developers by correcting early mistakes and reinforcing the understanding of ambiguous queries. this stepwise process enables the system to produce executable pandas functions that precisely match the intended output. this iterative refinement approach builds upon the concept of interactive code generation and human-in-the-loop ai for code, where human feedback and interaction are used to improve the quality and correctness of generated code. while interactive code generation has been explored, our specific application to generating pandas functions for tabular data queries with multi-turn llm interaction offers a novel contribution. the use of pandas, a crucial library for data manipulation in python, further motivates this work, as it enables the seamless integration of generated code into data science workflows. ## background over the past two years, the field of natural language processing (nlp) has undergone rapid evolution, largely driven by advances in large language models (llms). early approaches were predominantly task-specific, demonstrating robust language understanding but frequently encountering limitations with respect to domain-specific tasks or com-plex, multi-step processes. the introduction of transformer-based architectures significantly improved scalability and generalization, paving the way for more sophisticated llms. recent innovations-exemplified by gpt-based models and open-source foundation models such as llama @xcite -have further broadened the scope of potential applications by enhancing both data efficiency and the capacity to generate accurate, contextually appropriate responses to complex problems. however, it is important to note that our approach leverages human-prompted multi-step workflows rather than relying on purely model-driven multi-step reasoning. by incorporating iterative user input and guidance, we effectively harness the strong language comprehension of modern llms while maintaining precise oversight of the reasoning process. moreover, recent advances in large language models (llms) such as llama 3.3 (ai@meta, 2024) to handle more complex and specialized tasks. our approach builds upon these foundations by integrating an iterative error correction mechanism into the generation process, thereby ensuring that the output is both syntactically correct and semantically aligned with the intended query. ## system overview figure 1 provides an overview of our multi-turn prompting system in an end-to-end method (question to answer). the process begins with a user query and table metadata, and proceeds through three main stages, as described below. ## results the multi-turn prompting framework exhibits a marked improvement over single-turn prompting, achieving higher accuracy and more robust handling of execution errors. ## conclusion we have presented a multi-turn prompting framework for the automated generation of pandas functions in tabular question answering. by decomposing the problem into answer type extraction, initial function generation, and error-driven refinement, our system achieves substantial improvements over single-turn prompting and fine-tuned models. our experiments demonstrate that multi-turn prompting enhances both accuracy and robustness, offering a promising direction for future research in table reasoning and prompt-based code generation.