Exploration Lab IITK at S em E val-2025 Task 8: Multi- LLM Agent QA over Tabular Data (2025)

Abstract: This paper presents our Multi-LLM Agentic System that helps solve the problem of tabular question answering as posed in the SemEval Task-8: Question Answering over Tabular Data. Our system incorporates an Agentic Workflow where we assign each agent a role along with the context from other agents to better help resolve the ambiguity. As the user poses their question along with the dataframe, we firstly try to infer the types of the columns from the dataframe and also the expected answer type given the question and the column types. Then, the planner agent gives out a plan that tells us about the steps that we have to take to get the answer. Each step is written such that it helps us write one line of Python code. Then, we call the coding agent, which attempts to write the code given the information from the previous agents. After that, we perform a debugging pass through a debugging agent, which tries to resolve the issue given the previous context and finally deliver the answer if the code runs error-free. Our system achieved 14th place on the overall open-source models track.

## introduction question answering (qa) over tabular data is a natural language processing (nlp) task that involves extracting aanswers from structured tabular formats. this task is crucial as many financial reports, scientific data, and government records, exist in tabular form. automating qa over tables enhances information retrieval and decision-making processes. the task covered in this work involves evaluating the performance of language models on tabular qa on databench (full tables) benchmark, following the benchmark and methodologies outlined in the task overview paper @xcite . the problem involves addressing challenges like diverse query intents, table structures, and complex answer types (boolean, categorical, numerical, and lists). key sub-problems include query intent disambiguation, efficient data retrieval, and reasoning over large tables and multi-turn interactions. the broader impact of this research lies in its potential to scale across diverse data sources. our system utilizes openai's api, accessed via python, to interact with deepseek's r1 open-source model. through this competition, we gained valuable insights into the strengths and limitations of traditional transformer-based models in question answering over tabular data. we ranked fourteenth among all submissions, highlighting both our achievements and areas for improvement. we explored various reasoning-based approaches and found that breaking the problem into smaller, more manageable subtasks significantly enhanced performance. by assigning each subtask-such as interpreting the question, extracting relevant information, searching and querying with code, and debugging errors-to specialized models, we distributed the workload effectively and provided each model with a focused area of operation. this multiagent approach not only streamlined the process but also improved precision by reducing the cognitive burden on any single model. we observed that the model struggles with semantic meaning of some tricky questions, which requires additional reasoning before execution which suggests potential areas for improvement. the implementation is publicly available at this github repository. ## system overview we explored various approaches to question answering over tabular data, including transformerbased models and llm-based coding paradigms. while models such as tapas @xcite , tabert @xcite , and strubert @xcite showed promise, their direct encoding of table context was insufficient for our analytical needs. to overcome these limitations, we developed a multi-agent, self-correcting framework using modern llms (viz. deepseek-v3, deepseek-r1): ## experimental setup and results in our experiments, we evaluated our multi-agent, self-correcting framework for question answering over tabular data on the full databench dataset. the evaluation metric used was accuracy-the percentage of correctly answered queries based on the official evaluation script. all experiments were performed exclusively on the full dataset, in line with our design decisions, and the reported results are those obtained on the test set. ## results the experimental results highlight several key points: the iterative debugging cycle ensures that errors in code generation are corrected promptly, leading to a higher likelihood of accurate outputs. ## conclusion the proposed 2-llm agent framework combines intent-driven code generation with output-based refinement. our multi-agent code and output-based approach outperforms baseline methods and transformer-based models in handling complex queries. ranked third on the semeval leaderboard, our framework sets a strong foundation for further work on prompt engineering, improved retrieval mechanisms, and domain-specific extensions.