L y S at S em E val 2025 Task 8: Zero-Shot Code Generation for Tabular QA (2025)

Abstract: We developed a zero-shot pipeline that leverages an Large Language Model to generate functional code capable of extracting the relevant information from tabular data based on an input question. Our approach consists of a modular pipeline where the main code generator module is supported by additional components that identify the most relevant columns and analyze their data types to improve extraction accuracy. In the event that the generated code fails, an iterative refinement process is triggered, incorporating the error feedback into a new generation prompt to enhance robustness. Our results show that zero-shot code generation is a valid approach for Tabular QA, achieving rank 33 of 53 in the test phase despite the lack of task-specific fine-tuning.

## introduction tabular question answering (tabular qa) has huge potential in real-world applications such as financial analysis, business intelligence, and scientific data exploration, where structured databases serve as the primary source of information. unlike traditional text-based question answering (qa), which primarily deals with unstructured data, tabular qa requires extracting information from structured tables to be able to answer the input questions, thus involving reasoning about diverse table schemas, column relationships, and heterogeneous data types. complex supervised systems have been proposed to deal with the structured nature of tabular qa, either leveraging structured prediction with language representations @xcite or by formulating the task as a sequence-to-sequence problem @xcite @xcite . however, with the rise of instruction-based large language models (llm) @xcite , recent approaches have shifted away from reliance on large annotated datasets, instead reframing the task as a zero-shot generation problem @xcite . in this work, we further explore instructionbased llms to dynamically generate code functions capable of retrieving relevant data from tables based on the input question in a zero-shot manner. to enhance accuracy and reliability, we developed a modular three-staged pipeline that includes: (i) a column selection mechanism to determine the most relevant columns and their data-type, (ii) a code generation module responsible for producing executable code and (iii) an iterative error handling module that, in case the initial code execution fails, tries to fix the generated code accordingly. our group tested this approach within the se-meval 2025 task 8 event @xcite , which provided a diverse dataset featuring real-world tabular data. 1 the competition required models to produce answers in multiple formats, including boolean, categorical, numerical, and listbased outputs. our model was designed to generalize across different table structures, making it adaptable to various datasets beyond the shared task, ensuring robustness and broad applicability. although our approach demonstrated strong performance in code generation and execution, subsequent analysis revealed that the model struggles with columns containing complex data types (lists, dictionaries, etc.) and ambiguous queries, particularly for list-based responses. ## background question answering (qa) has been gaining significant attention in recent years, driven by the need for models capable of reasoning over structured data. early tasks in qa mainly focused on retrieving information from unstructured text sources @xcite , but the increasing availability of structured datasets has led to new challenges in understanding and querying tabular data. unlike classic text-based qa, where answers are retrieved from free-form text, tabular qa requires a higher level of interpretation and robustness to map questions to relevant columns and rows, handle missing values, and compute statistics when necessary. in parallel, several datasets have been introduced to benchmark tabular qa models, including wikitablequestions @xcite , sqa @xcite , and the more recent databench dataset @xcite , which provides real-world tabular data for evaluating models in different scenarios. structured tabular qa most state-of-the-art approaches for tabular qa leverage a pretrained language model -equipped with an specialized encoding module to represent tabular informationtailored for structured prediction. for example, tapas @xcite feeds both the input question and the flattened table into bert @xcite as a single sequence, and finetunes the architecture to select relevant columns and predict an aggregation function. similarly, tacube @xcite combines a cube constructor with bart @xcite to predict the real answers based on the input question and the results of the cube operations. generative tabular qa to address the rigidity of structured approaches, recent works have explored generative models for program synthesis, where an llm is finetuned to generate executable programs or instructions (in the form of sql queries, for example) to be applied against tabular sources. @xcite proposed seq2sql, a sequence-to-sequence model to translate natural language into sql syntax, incorporating query-space pruning to significantly simplify and enhance the generative task. later, @xcite joined both concepts by optimizing tabular embeddings that fit both generative and structured purposes. zero-shot code generation more recently, advancements in code generation have enabled a paradigm shift in tabular qa, driven by powerful multipurpose llms with strong coding capabili-ties, such as qwen @xcite ) and mistral's codestral @xcite . these models facilitate a zero-shot approach to program synthesis, eliminating the need for predefined templates or large annotated datasets. instead, zero-shot generation allows the system to dynamically adapt to different schemes without explicit prior knowledge of the table structure @xcite , thus providing flexibility and scalability. despite its potential, zero-shot code generation models still face big challenges, particularly in error handling, runtime execution failures, and schema variability. building on this approach, our work extends an instruction-based model with error awareness, enabling it to detect and recover from execution failures in an iterative error-recovery mechanism, where the model dynamically analyzes execution failures and regenerates code based on error feedback. ## system overview our approach for the semeval 2025 task 8 iterates upon the code generation approaches for tabular qa, where the core component is a pretrained llm responsible of generating executable code to extract the answer from the tables. to build upon prior works @xcite , we incorporated a module that helps selecting the columns relevant to the question, while also identifying the data types of their content. moreover, we incorporate an errorfixing module that attempts to catch runtime errors and integrates them as part of a new prompt, guiding the llm to refine its code generation. figure 1 shows an schematic view of the architecture of our system. we have designed a modular pipeline that features three main components, which we describe below: (i) a column selector, (ii) an answer generator and (iii) a code fixer. column selector instead of relying on manually crafted heuristics or embedding similarity measures, the first component of our system leverages an instruction-based llm tasked to identify the most relevant columns of a tabular source from an input question in natural language form. our template provides the list of column names and instructs the model to return only those that are essential for answering the query. answer generator once the relevant columns are identified, the second component of our pipeline is instructed to generate executable code that retrieves the answers from the tabular source using both the input query and the relevant columns extracted in the previous step. as part of our prompt, we guided the llm to generate python programming code and postprocessed the output to ensure that only python lines were passed throught the next module. python language was chosen since it is widely used in data analysis and has extensive support for tabular data processing through libraries such as pandas. code fixer the final component of our pipeline captures execution errors that might occur due to incorrect syntax, schema mismatches, or runtime exceptions. this module captures the error messages and re-generates a corrected function by feeding the error context back into the llm. to achieve this, we used a structured prompt that includes the code that causes an error with the corresponding error description. preprocessing since our system strongly relies on a well-formatted prompt, we manually designed a preprocessing step to ensure a consistent format to feed our system. we standardized column names for simplified versions (removing emoji and all non-alphanumerical characters except punctuation symbols) to prevent possible errors in the answer generator caused by mismatches between the table structure and the generated code. we identified enum-like column types, such as the case of categorical attributes with a finite amount of strings as a value (e.g. a "survey" column that only contains "yes", "no" or "maybe"), and inferred a common scheme so to ensure consistency across different attributes, thus reducing errors related to unexpected variations in categorical values. ## experimental setup our system relies on open-source llms for zeroshot code generation. this way, no explicit training nor finetuning was conducted. instead, we used the available training phase datasets to validate different llms and select the best performing one for the final test phase. ## analysis of results in this section, we present the evaluation of our system on the task. we first report performance during the development phase (@xmath05.2), where our system was evaluated on the test dataset through codabench submissions. 3 ## conclusions and future work in this work we propose a zero-shot approach for tabular qa that demonstrated a strong performance for the semeval 2025 task 8, ranking among the best systems in the development phase, although suffering from a performance drop in the test phase. still, our system shows that an instruction-based approach allows to dynamically adapt to different dataset schemes without requiring additional training or finetuning, surpassing the baseline model even with limited hardware resources available. future work will focus on further refining prompt templates, improving schema adaptation, optimizing execution efficiency or incorporating a voting system with different llms. improving the detection of these complex datatypes is also critical, as they allow the model to answer questions on less structured tables -which constitute the majority of online data-, ultimately making the system more generalizable.