MRT at S em E val-2025 Task 8: Maximizing Recovery from Tables with Multiple Steps (2025)

Abstract: In this paper we expose our approach to solve the SemEval 2025 Task 8: Question-Answering over Tabular Data challenge. Our strategy leverages Python code generation with LLMs to interact with the table and get the answer to the questions. The process is composed of multiple steps: understanding the content of the table, generating natural language instructions in the form of steps to follow in order to get the answer, translating these instructions to code, running it and handling potential errors or exceptions. These steps use open source LLMs and fine grained optimized prompts for each task (step). With this approach, we achieved a score of 70.50% for subtask 1.

## introduction contemporary natural language processing (nlp) is limited by the volume of information (text) that can be processed effectively while maintaining contextual relevance. during response generation this constraint impacts the recall of data needed to produce correct and complete answers @xcite . tabular data exemplifies this challenge in particular, since it is the day-to-day task most affected by this restriction @xcite . this paper addresses the semeval 2025 task 8: question-answering over tabular data @xcite . in this paper we present maximizing recovery from tables with multiple steps (mrt), a multistep pipeline that leverages both llms and python code generation to answer questions in the most factual way possible. instead of an end-to-end strategy our system implements a sequential divide and conquer approach in which at every step either llms or heuristics are executed. those steps cover from describing the tables (frequent values, column descriptions, statistical information), generating the list of instructions (in plain natural language) to carry the task and obtain the result, code execution and answer parsing. we achieve 70.50% accuracy in the databench challenge test set using this approach. the code that generated these results is publicly available foot_0 . ## background question answering (qa) focuses on retrieving accurate answers from @xcite data sets. recent methods for qa on tabular data, such as tapas @xcite , integrate transformers with architectures specifically tuned to extract answers directly from the tables used as context. however, llms have also been employed in zeroshot or few-shot strategies, since they are able to respond with a certain quality due to their prior knowledge and thus reduce the need for domainspecific fine-tuning of each domain @xcite . recent llms have demonstrated emergent reasoning capability, but still present difficulties with complex queries involving multiple columns, large tables, or ambiguous interpretations of a question. another approach is to parse natural language queries and transform them into formal queries such as sql. systems such as seq2sql @xcite or tablegpt2 @xcite are designed to generate sql queries from relational database queries or python code, respectively. these methods offer advantages such as greater flexibility, as they are theoretically independent of the table size (which might not fit entirely in the context window of an llm), and greater transparency, by including an intermediate step that allows auditing and reviewing the generated queries. to evaluate these models, reference datasets have been critical. wikipedia-based sets, such as wikisql @xcite and tabfact @xcite , provide structured evaluation environments but do not reflect the heterogeneity of real-world tabular data @xcite . in response, databench @xcite has been developed, which brings together 65 realworld datasets with more than 1,300 manually crafted question-answer pairs across multiple domains. works such as tablerag @xcite propose the use of rag systems for tabular comprehension tasks, such as qa, employing techniques such as query expansion and a double transformation to query languages. this process translates, on the one hand, the schema to be interacted with and, on the other hand, the operation necessary to identify the cells with the answer. also noteworthy are proposals such as chain-of-table @xcite , which implements chain-of-thought as an iterative reasoning mechanism. instead of executing code in one shot, operations are executed in each iteration to add or discard information from the table until the answer is found. despite recent progress, some challenges still remain, such as improving reasoning within multiple rows, handling different domains and languages or integrating several tables and increasing the explainability of the full process. ## system overview the strategy developed in our system consists of loading the table as a pandas dataframe, and then, with the use of llms, generating python code to interact with the tabular data to finally obtain the answer to each question. for this, we implemented multiple modules that are executed sequentially for each question. some of these steps are heuristics, whereas others are llm-based. each of the modules can use different llms. for this work, we have used multilingual models from the families qwen2.5 @xcite , qwen2.5-coder @xcite , llama-3 (ai@meta, 2024) and phi-4 (abdin et al., 2024) among others. the figure 1 depicts an overview of the workflow of the system. the first step is understanding and analyzing the information of the table (type of data, appearance of null values, etc.). then, an llm generates textual instructions with the reasoning steps to follow in order to get the answer. after that, a code generation model converts the instructions to python code. then, the runner executes this code. if an exception occurs during code execution or answer parsing, the system steps back into the coder in an iterative looping process until it gets a valid answer or a limit is exceeded. finally, there are formatting steps that implement functions such as getting the answer in the desirable data type or selecting the correct number of decimals in numbers. ## experimental setup the experimental setup consisted of two approaches for executing the modules plus the combination of configurations of each of the modules. initially, the modules were run in series, as lighter models that could fit concurrently in memory were used. in this approach, each question for each ta-ble was processed sequentially through all system modules. however, during testing phase with heavier models, it became necessary to implement a system that allowed to load and unload the models as it was required by each step. this approach executes each of the step in batches. all the questions are processed for each step before passing to the next one. hence only the model used in each step is loaded in memory. this allows to load bigger models for each of the steps whilst using the same gpu without involving excessive overhead in loading/unloading models. the hardware used to run the tests was an nvidia rtx-a6000 that combines 84 secondgeneration rt cores, 336 third-generation tensor cores, and 10,752 cuda cores with 48 gb of graphics memory for performance. in 4.2, we define the model configurations used in the test phase. during development, we also used reduced versions of these models with 8b parameters.