O sei B refo-Liang at S em E val-2025 Task 8 : A Multi-Agent LLM code generation approach for answering Tabular Questions (2025)

Abstract: This paper presents a novel multi-agent framework for automated code generation and execution in tabular question answering. Developed for the SemEval-2025 Task 8, our system utilises a structured, multi-agent approach where distinct agents handle dataset extraction, schema identification, prompt engineering, code generation, execution, and prediction. Unlike traditional methods such as semantic parsing-based SQL generation and transformer-based table models such as TAPAS, our approach leverages a large language model-driven code synthesis pipeline using the DeepSeek API. Our system follows a zero-shot inference approach, which generates Python functions that operate directly on structured data. Through the dynamic extraction of dataset schema and intergration into structured prompts, the model comprehension of tabular structures is enhanced, which leads to more precise and interpretable results. Experimental results demonstrate that our system outperforms existing tabular questioning and answering models, achieving an accuracy of 84.67% on DataBench and 86.02% on DataBench-lite, which significantly surpassed the performances of TAPAS (2.68%) and stable-code-3b-GGUF (27%). The source code used in this paper is available at t https://github.com/oseibrefo/semEval25task8

## introduction large language models (llms) have significantly advanced natural language processing (nlp), demonstrating strong capabilities in question answering (qa), code generation, and structured data analysis @xcite . while llms excel in open-domain qa over unstructured text @xcite , reasoning over tabular data presents additional challenges. these include schema understanding, multicolumn aggregation, numerical computation, and execution reliability @xcite . tabular question answering (tqa) has been traditionally approached using the following three main techniques: semantic parsing approaches: these are traditional methods where models such as seq2sql @xcite and sqlnet @xcite translate natural language queries into sql commands. while effective for structured databases, these approaches require predefined schemas and struggle with generalisation to diverse table structures. transformer-based table models: models such as tapas @xcite and tabert @xcite jointly encode table structures and queries, enabling direct classification-based predictions. however, they are limited in their ability to perform dynamic computations beyond simple row-based retrieval. code-based approaches: recent methods have explored prompting llms to generate executable python functions to extract or compute answers @xcite . these approaches offer more flexibility than sql-based models but require safeguards against execution errors, format inconsistencies, and incorrect column selection. tabular question answering (tqa) presents unique challenges since it requires models to understand structured data and perform reasoning over numerical and categorical values. unlike standard text-based qa, tqa often involves multiple computational steps, such as aggregating values, filtering rows, or computing statistics. traditional methods rely on sql-based querying or transformer models fine-tuned on tabular data, such as tapas. however, these approaches often require extensive training and struggle to generalize to unseen tables. a promising alternative is to leverage llms for program synthesis, where the model generates executable code to answer questions about tabular data. this approach allows for flexible and interpretable reasoning steps while enabling the processing of large datasets beyond the llm's context window by delegating execution to external interpreters. we propose a multi-agent system that integrates prompt-based code generation with structured dataset extraction and execution. the system consists of the following multi-agent system workflow: unlike zero-shot in-context learning (z-icl), which provides the entire dataset within the prompt, our approach generates python functions that execute externally, making it more scalable for large datasets. experimental results demonstrate the effectiveness of our approach and highlight its adaptability to unseen tabular data structures and its ability to generate accurate responses across multiple answer types. ## experiments the system follows a zero-shot approach, relying on prompt engineering to instruct the model on how to retrieve and compute answers from structured tabular data. the experimental setup involved the use of the deepseek api. the dataset was extracted from a competition archive in a zip file. the extracted data had multiple subdirectories, with each representing a distinct dataset. each dataset contained parquet files, which stored structured tabular data for questionanswering. each dataset id also corresponded to a folder that contained two files, which are the all.parquet and sample.parquet files. the sample.parquet file contained the first 20 rows of the all.parquet files. the test data also contained the 522 set of test questions to be answered along with the id of the corresponding dataset. ## results and discussion our system was evaluated on the databench test benchmark, which consists of real-world datasets with a total of 522 manually curated questions. the evaluation assessed the effectiveness of our approach in generating and executing python code to answer tabular questions. performance was measured using the datatech eval package, which ensured a standardized assessment across various dataset structures and question types. the evaluation was carried out to compare the performance of our multi-agent system with existing models, particularly in handling structured tabular data. ## conclusion this work has proposed the use of a multi-agent system that integrates prompt-based code generation with structured dataset extraction and execution for tabular question answering. the results demonstrate that multi-agent, execution-driven llm pipelines are superior to direct prompting techniques and other traditional tabular qa approaches, achieving higher accuracy on real-world tabular data reasoning tasks. by improving execution validation, incorporating lora fine-tuning, and enhancing schema comprehension, this approach could further improve how llms interact with structured data. future work will focus on enhancing the code execution efficiency and the incorporation of automated error correction.