G - MACT at S em E val-2025 Task 8: Exploring Planning and Tool Use in Question Answering over Tabular Data (2025)

Abstract: This paper describes our system submitted to SemEval-2024 Task 8 “Question Answering over Tabular Data.”The shared task focuses on tackling real-life table question answering (TQA) involving extremely large tables with the additional challenges of interpreting complex questions. To address these issues, we leverage a framework of Multi-Agent Collaboration with Tool use (MACT), a method that combines planning and tool use. The planning module breaks down a complex question by designing a step-by-step plan. This plan is translated into Python code by a coding model, and a Python interpreter executes the code to generate an answer. Our system demonstrates competitive performance in the shared task and is ranked 5th out of 38 in the open-source model category. We provide a detailed analysis of our model, evaluating the effectiveness and the efficiency of each component, and identify common error patterns. Our paper offers essential insights and recommendations for future advancements in developing TQA systems.

## introduction table question answering (tqa) focuses on addressing questions related to tables. it has been widely studied across domains @xcite @xcite and languages @xcite @xcite . current tqa datasets predominantly feature tables from wikipedia @xcite , resulting in an oversimplified task setup characterized by small, clean tables with limited diversity in data types. to promote studies in real-life tqa, @xcite propose databench, an english tqa dataset consisting of 65 tables and around 1300 manually created questions spanning various domains. based on this dataset, the semeval-2025 task 8 "question answering over tabular data" encourages tqa modeling in a more realistic and challenging setup. it consists of two subtasks: all, where original long tables are used and lite, in which only the first 20 rows of a table are used. there are two main challenges in the task: (1) the large table size makes direct inference using large language models (llms) difficult due to their limited input lengths and problems of being lost in the middle @xcite . (2) the complexity of questions requires multiple steps to be solved. for instance, to answer the question in figure 1 , a system should first filter for employees who are working in sales and then calculate the average of working years among those employees. to address these challenges, we propose g-mact, a method combining global planning with multi-agent collaboration with tool use @xcite . g-mact comprises two modules: a global and an iterative planning module. the global planning module takes in the first several rows of a table alongside a question, and generates a step-by-step plan. by doing this, we break down a complex question into easier sub-steps. a coding agent translates this plan into pandas-based python code, foot_0 which is executed using a python interpreter to derive an answer. despite being efficient, the global planning module may encounter difficulties in formulating an optimal plan when it relies solely on a partial table and a question. to enhance the robustness of plan generation by conditioning it also on past observations, we apply the iterative planning framework mact whenever the global planning module does not yield an answer. the framework generates one step of a plan in each iteration, considering past steps and observations. we ensemble results from four models for our final submission. our method ranks 5th among 38 open-weight systems in the challenging all setup. we thoroughly analyze g-mact from three perspectives: effectiveness of the global and iterative planning modules, efficiency, as well as error patterns. we find that most instances in databench can be solved by global planning and tool use alone. however, the iterative planning module proves essential for instances where global planning fails to yield an answer. additionally, our findings underscore the significance of selecting an optimal ensemble method. we observe for around 91% of the cases, at least one model predicted the correct answer, while the ensemble method used in our approach only achieves around 86% of exact match (em) accuracy. this discrepancy highlights a clear need for exploring more sophisticated ensembling techniques. lastly, through an error analysis, we find that developing a good planning agent that can understand table semantics and has access to factual and domain-specific knowledge is of vital importance for further enhancing a model's performance. we make our system publicly available. 2 ## related work the main methodologies used by g-mact are planning and tool use. both have been employed in tqa to facilitate more fine-grained problem solving, thus improving model performance @xcite @xcite . current research either utilizes global planning @xcite or iterative planning @xcite . global planning involves generating a plan consisting of multiple steps in a single iteration, conditioned solely on a question and a table. iterative planning conditions the generation of the next step of a plan 2 https://github.com/boschresearch/mact on previous observations and steps. while global planning tends to be more efficient, iterative planning offers more fine-grained plan generation. in g-mact, we integrate both planning methods. ## system overview given a table t and a question q, a tqa system aims to address q and return an answer a. our proposed system g-mact combines global planning with the iterative approach of multi-agent collaboration with tool use @xcite in a pipeline manner as illustrated in figure 1 . in this section, we introduce the global planning module, the iterative planning module, and the ensemble method used to create the submission results. ## experimental setup we present details on our experimental setup. predicted and reference answers that match exactly. we use the official evaluation scripts provided by @xcite . ## results figure 2 shows the performance of g-mact compared with top four systems and the baseline reported in osés-grijalba et al. (2025) for the more challenging all setup. our ensemble model outperforms the baseline method by a large margin. however, there is still a gap between our method and the best-performing system in the shared task. ## conclusion in this paper, we introduce g-mact, a pipeline framework combining planning and tool use, developed for the semeval-2025 task 8. our method ranks 5th among all approaches using open-weight models, with no training involved. we carefully analyze our system in terms of the effectiveness of each module, efficiency, and error patterns. we provide key insights for future work to address real-life table question answering. you are an expert in analyzing table data and generate step-by-step plans to solve any questions related to long tables. the following table only shows the first three rows of the table due to its large size. please generate a step-by-step plan to address the question, following the below requirement: 1. a plan should contain no more than 4 steps. 2. each step should be in one line. 3. return only the step-wise plan and nothing else. 4. no repetition of the plan. please return only a plan and nothing else. following are three examples: table: | rank | personname | age | finalworth | category | source | country |… | 1 | elon musk | 50.0 | 219000 | automotive | tesla, spacex | united states |… | 2 | jeff bezos | 58.0 | 171000 | technology | amazon | united states ... …[remaining 2665 rows unshown due to large table size]... context: table caption: forbes billionaires 2022. question: how many billionaires are there from the 'technology' category? plan: 1. i need to filter the table to get all billionaires from the 'technology' category. 2: then i need to count the number of retrieved entries. table: | segmentation_1 | descriptor | complaint_type | created_date |… | [] | business not in compliance | noncompliance with phased reopening | 2020-10-02 11:50:27+00:00 |… | [] | entire building | heat/hot water | 2021-11-03 06:48:49+00:00 |… ...[remaining 99997 rows unshown due to large table size]... context: table caption: nyc 311 calls question: mention the 2 most common weekdays for complaints. plan: 1. i need to count the frequency of each unique weekday in the column 'weekday_name'. table: | segmentation_1 | descriptor | complaint_type | created_date |… | [] | business not in compliance | noncompliance with phased reopening | 2020-10-02 11:50:27+00:00 |… | [] | entire building | heat/hot water | 2021-11-03 06:48:49+00:00 |… ...[remaining 99997 rows unshown due to large table size]... context: table caption: nyc taxi trips. question: are there any trips with a total distance greater than 30 miles? plan: 1. i need to count the number of entries whose 'trip_distance' is larger than 30. 2. if the value from step 1 is larger than 0, then the answer is 'true', otherwise, it is 'false'. 3. i will create a variable name after 'final_result' to store the boolean answer and return the variable as final answer. now generate a plan for to address the following question and table. the plan should contain maximum 4 steps, with each step one line. table: {table} context: {context} question: {question} figure 5: a prompt for a planning agent in the global planning module.caption solve a table question answering task with interleaving thought, action, observation steps. thought can reason about the current situation, and action can be two types: (1) operate[instruction], which carries out operations such as information retrieval or calculations based on the instruction and returns the retrieved or calculated results. (2) finish[answer], which returns the answer and finishes the task. you may take as many steps as necessary. here are some examples: table: | rank | personname | age | finalworth | category | source | country |… | 1 | elon musk | 50.0 | 219000 | automotive | tesla, spacex | united states |… | 2 | jeff bezos | 58.0 | 171000 | technology | amazon | united states |... …[remaining 2665 rows unshown due to large table size]... context: table caption: forbes billionaires 2022. question: how many billionaires are there from the 'technology' category? thought 1: i need to count the number of billionaires from the 'technology' category. action 1: operate[count the number of entries whose category is technology ] observation 1: 343 thought 2: in observation 1, 343 billionaires are from the 'technology' category, therefore, the answer is 343. action 2: finish[343] table: | segmentation_1 | descriptor | complaint_type | created_date |… | [] | business not in compliance | noncompliance with phased reopening | 2020-10-02 11:50:27+00:00 |… | [] | entire building | heat/hot water | 2021-11-03 06:48:49+00:00 |… ...[remaining 99997 rows unshown due to large table size]... context: table caption: nyc taxi trips. question: are there any trips with a total distance greater than 30 miles? thought 1: i need to count the number of entries whose 'trip_distance' is larger than 30. action 1: operate[count the number of entries whose 'trip_distance' is larger than 30.] observation 1: 0 thought 2: in observation 1, there is 0 entry whose trip distance is larger than 30. therefore, the answer is false. action 2: finish[false] table: | segmentation_1 | descriptor | complaint_type | created_date |… | [] | business not in compliance | noncompliance with phased reopening | 2020-10-02 11:50:27+00:00 |… | [] | entire building | heat/hot water | 2021-11-03 06:48:49+00:00 |… ...[remaining 99997 rows unshown due to large table size]... context: table caption: nyc 311 calls question: mention the 2 most common weekdays for complaints. thought 1: i need to count the frequency of each unique weekday in the column 'weekday_name'. action 1: operate[count the frequency of each unique weekday in the column 'weekday_name'.] observation 1: {'tuesday': 15847, 'monday': 15816, 'wednesday': 15445, 'thursday': 14978, 'friday': 14707, 'saturday': 11781, 'sunday': 11426} thought 2: the question ask for 2 most common weekdays. from observation 1, we find tuesday and monday have the largest frequencies and they are weekdays. therefore, the answer is ["tuesday", "monday"] action 2: finish[["tuesday", "monday"]] (end of examples) now generating the thought, action, observation for the following instance: table: {table} context: {context} {question} {scratchpad}""" figure 6: a prompt for a planning agent in the iterative planning module. you are an expert in python code generation. write a python function named 'target_function' according to the given plan using pandas dataframe. the given dataframe shows only two records of the original data due to its large size. the main goal of showing the dataframe is to show the data type associated to each column. however, you should not operate any code based on the given dataframe, since it does not contain all information about the table. below are two examples plan: 1. i need to filter the table to get all billionaires from the 'technology' category. 2: then i need to count the number of retrieved entries. 3. the answer to the question is the number of retrieved entries in the second step, and i will return this value as the final answer. dataframe code for the first two records: import pandas as pd @xmath0:[1.0, 2.0],'personname':['elon musk', 'jeff bezos'],'age':[50.0, 58.0],'finalworth':[219000.0, 171000.0],'category':['automotive', 'technology'],'source':['tesla, spacex', 'amazon'],'country':['united states', 'united states'],…} @xmath1.dataframe(data) code: ```python def target_function(dataframe): # filter the table for 'technology' as the category and count the number of the entries @xmath2: 1. i need to retrieve the first five values from the 'gold' columns. 2. to calculate the average number, i will sum the retrieved values and divide the sum by 5.