Q lever A nswering- PUCRS at S em E val-2025 Task 8: Exploring LLM agents, code generation and correction for Table Question Answering (2025)

Abstract: Table Question Answering (TQA) is a challenging task that requires reasoning over structured data to extract accurate answers. This paper presents QleverAnswering-PUCRS, our submission to SemEval-2025 Task 8: DataBench, Question-Answering over Tabular Data. QleverAnswering-PUCRS is a modular multi-agent system that employs a structured approach to TQA. The approach revolves around breaking down the task into specialized agents, each dedicated to handling a specific aspect of the problem. Our system was evaluated on benchmark datasets and achieved competitive results, ranking mid-to-top positions in the SemEval-2025 competition. Despite these promising results, we identify areas for improvement, particularly in handling complex queries and nested data structures.

## introduction the rapid growth of structured data across various domains has increased the demand for automated systems capable of extracting and interpreting information from tabular data. tqa is a crucial natural language processing (nlp) task that focuses on generating accurate answers to factual questions using structured information @xcite . unlike traditional qa systems that rely on free-text corpora, tqa systems must directly retrieve relevant information from relational tables, spreadsheets, or structured databases. additionally, the limited context window of large language models (llms) constrains the amount of tabular data that can be processed in a single prompt, making metadata extraction a key component for precise query resolution. task 8 in semeval 2025 -databench, question-answering over tabular data @xcite , introduces a new benchmark for evaluating tqa systems. this benchmark enables the assessment of different question types spanning multiple information domains. the challenge lies in developing systems that can accurately answer queries over standardized datasets, where responses may take various forms, including boolean, categorical, numerical, or lists. evaluation is based on the system's ability to provide precise answers given a (dataset, question) pair. in this paper, we introduce qleveranswering-pucrs 1 , a modular approach to table question answering. our system decomposes the tqa task into specialized agents, each responsible for a distinct function: metadata extraction, expression generation, error handling, and code execution. by structuring the workflow into interconnected components, qleveranswering-pucrs aims to improve observability, reduce execution failures, and enhance accuracy. we evaluate qleveranswering-pucrs on benchmark datasets, demonstrating its effectiveness in handling complex queries over structured data. our average results ranked us between 9 th and 19 th out of 49 participating systems across the four different rankings considered in the task. nonetheless, we identify areas for further improvement and refinement in our approach. ## background and related work tqa aims to answer questions referring to data stored in tables @xcite . early methods had trouble generalizing across many data formats and depended on semantic parsing to translate natural language queries into structured languages like sql. modern approaches process tabular data using llms, avoiding the need for explicit query translation. proprietary models tend to perform better, but all models have limits when performing sophisticated queries @xcite . current research has investigated retrievalaugmented generation (rag), structured query generation, and simple neural network-based techniques to enhance query resolution. these approaches enable models to overcome context window restrictions by extracting pertinent metadata before query execution @xcite . program-based prompting is a further method in which models produce executable code to interpret tabular data rather than immediately responding to queries. this method is used for hybrid question answering by the hpropro framework @xcite , which shows good performance without the need for explicit data retrieval or transformation. the mact framework @xcite introduces a multi-agent approach, where distinct agents handle planning, query formulation, execution, and validation. this division of tasks enhances robustness, allowing for iterative refinement and error handling, leading to performance gains over fine-tuned llms in complex table-based reasoning tasks. error correction mechanisms improve accuracy by identifying logical inconsistencies and refining queries dynamically @xcite . program-aided language models (pal) @xcite propose generating intermediate python code to be executed by an interpreter, which improves arithmetic and symbolic reasoning. this delegation enhances accuracy, especially for problems requiring precise calculation or logic that exceeds llms' intrinsic capabilities. the plan-of-sqls (pos) framework @xcite emphasizes the need for interpretability in table qa . by decomposing complex queries into atomic sql operations, pos enables transparent, step-by-step reasoning that can be verified by humans and llms alike. this work is particularly relevant in high-stakes domains, demonstrating that explainability can coexist with competitive performance. while pal and pos focus on symbolic execution and explainability, our approach targets the integration of structured program reasoning with lightweight retrieval to improve efficiency and adaptability in practical settings. ## results figures 2 and 3 present the accuracy scores per semantic category (data type) and dataset. one notable observation is the consistency of our results across databench and databench lite, suggesting that our model generalizes well even when restricted to only 20 rows per dataset. regarding different semantic types, our system on databench lite dataset achieved its highest accuracy in the boolean type, which aligns with the expectation that the restricted domain in boolean values is easier to infer. in contrast, the lowest accuracy was observed in the list[category] type, which requires handling multiple categorical elements in a single response. the difficulty in correctly parsing and structuring multiple values is likely a contributing factor to this result, highlighting an area where additional refinement in the expression generation process might be beneficial. one of the most significant drops in accuracy occurred in the 067_tripadvisor dataset, where our system obtained the lowest performance among all datasets. a deeper inspection reveals that this dataset contains json-formatted strings within queried columns, introducing additional complexity in extracting relevant values. since our system relies on pandas expressions to directly parse and filter data, handling embedded json structures within textual fields requires additional pre-processing, which was not included in our current pipeline. similarly, the dataset 070_openfoodfacts exhibited similar problems, but in this case, due to list-formatted strings rather than json, requiring specific parsing strategies that were not accounted for. this issue impacted both list[category] and category type questions, where correct identification and parsing of values were hindered, as well as number type questions, particularly those involving "how many..." questions, where quantities needed to be extracted from structured text fields. further analyzing the per-dataset performance, we identified patterns that explain specific drops in accuracy. in the 075_mortality dataset, focused on category questions, all incorrect predictions occurred in queries requiring the identification of the entity with the highest or lowest average rate. the system consistently inverted the aggregation logic, incorrectly assuming that higher "rate" values indicated better outcomes, whereas lower rates were preferable. this reveals a limitation in inferring the semantic orientation (i.e., whether minimizing or maximizing is desirable) based solely on dataset structure. in the 068_worldbank_awards dataset, primarily involving boolean questions, errors arose from dataset ambiguities. even manual inspection revealed that answering required domain knowledge beyond the available data preview, as column names and sample rows were insufficient to disambiguate the intended meaning without additional context. ## conclusion in this work, we presented qleveranswering-pucrs, a modular system for table question answering that relies on code generation over structured data. our approach generates single-line pandas expressions to extract answers, demonstrating strong performance across multiple semantic categories and significantly surpassing the task baseline in both databench and databench lite evaluations. despite these promising results, our analysis revealed specific limitations. the system struggled to correctly infer the semantic orientation of metrics, particularly in tasks requiring judgment on whether lower or higher values are preferable, and showed difficulties when answering questions that demanded external domain knowledge beyond the provided data structure. for future work, we plan to explore multi-step query execution strategies to improve reasoning over complex tabular data. we also intend to investigate the impact of model size on performance, assess the system's sensitivity to different prompt components, and evaluate iterative error correction methods to determine if multiple repair attempts can further improve accuracy. additionally, incorporating external dataset descriptions or semantic enrichment mechanisms could mitigate contextrelated ambiguities.