Core Intelligence at S em E val-2025 Task 8: Multi-hop LLM Agent for Tabular Question Answering (2025)

Abstract: This paper describes a multi-hop LLM agent for tabular question answering developed for SemEval-2025 Task 8 and ranked 6th with 87% accuracy. Our approach combines proprietary LLM (ChatGPT-3.5-turbo) for code generation and open source LLM (Llama-3.2-3B) for answer validation.

## introduction retrieval-augmented generation (rag) is an efficient technique for incorporating enterprise knowledge into large language models (llms), improving factual accuracy and reducing hallucinations -a persistent problem, when llms operate in unfamiliar domains. the rag system retrieves relevant information from external knowledge bases and provides it as context for llm for reasoning and answer generation. this approach has been widely adopted for handling unstructured textual data, where relevant documents are stored in vector databases and traditional search engines. however, a significant portion of public and proprietary knowledge is stored in structured formats, such as excel spreadsheets or databases. semantic chunking, a well-established method for retrieval of unstructured information, is not suitable for tables, as structured data is inherently relational, meaning that cells, rows, and columns must be interpreted together to get meaningful insights. rag systems today primarily focus on text-based retrieval and struggle with structured data integration due to several limitations: unlike text-based question answering, which involves retrieval of relevant information, tabular qa requires sorting, filtering, grouping, aggregations, and mathematical operations. a system working with structured data must be capable of query execution rather than just retrieval. ## task and dataset description tabular question answering (tabular qa) task involves answering natural language questions based on structured tabular data. given an input query 𝑄 and a table 𝑇 with 𝑁 rows and 𝑀 named columns, the goal is to produce an accurate answer 𝐴. the semeval-2025 task 8 organizers offer the databench dataset, a large benchmark for the task of question answering on structured or tabular data @xcite . this dataset consists of 1300 questions and each question is accompanied by a related tabular dataset. overall, 65 datasets are presented from 5 different domains, such as business, health, social, sports and travel. the average number of rows in table is 50,300, while maximum is 713,107 rows per table. average number of columns is 25 per table, and maximum is 123. the questions are split into different categories depending on the type of expected answer: question type example question reasoning type boolean (yes/no) are all transactions ids unique? lookup category selection which organization has the patent with sorting & comparison the highest number of claims? numerical value how many borrowers have more than 1 existing loan? aggregation list output which 5 states have the most number of job listings? filtering & grouping ## related work early approaches of tabular qa relied on semantic parsing, where natural language questions were transformed into sql-like commands @xcite . these methods require extensive training on domain-specific schemas and struggle with generalization across unseen tables. recent advances in large language models (llms) have enabled zero-shot tabular qa by leveraging in-context learning (brown et al., 2020), supervised llm finetuning @xcite and code generation @xcite . models such as gpt-4 and llama, and codellama can dynamically generate sql queries or code instructions to retrieve answers from structured data. however, these methods suffer from hallucinations, logical inconsistencies, and execution failures. to address these limitations, researchers have explored multi-agent systems, where different llms specialize in code generation, execution verification, and logical consistency checks @xcite . our work builds upon this trend and introduces a structured llm agent, that dynamically plans, executes, and validates tabular queries, ensuring high reliability and accuracy. ## system overview we introduce a multi-hop llm agentic system, that performs tabular qa by decomposing reasoning and execution into separate stages. in contrast to single-pass prompting approaches, which often struggle with multi-column queries, schema inconsistencies, and logical errors, our system follows an iterative process that includes table exploration, execution planning, code generation, and answer validation. the architecture of the agent relies on langgraph 1 , a framework that organizes nodes execution as a directed acyclic graph (dag), which ensures modularity, reusability, and observability. our system orchestrates multiple llm-based agents, each having a distinct role and underlying implementation, including chatgpt-3.5-turbo for python code generation and llama-3.2-3b-instruct (fine-tuned) for answer validation and consistency checks. this multi-model collaboration improves both accuracy and efficiency and allows specialized models to handle tasks suited to their strengths. if answer consistency check fails, the system revisits previous reasoning steps, refines queries and execution plans dynamically.