AILS - NTUA at S em E val-2025 Task 8: Language-to-Code prompting and Error Fixing for Tabular Question Answering (2025)

Abstract: In this paper, we present our submission to SemEval-2025 Task 8: Question Answering over Tabular Data. This task, evaluated on the DataBench dataset, assesses Large Language Models’ (LLMs) ability to answer natural language questions over structured data while addressing topic diversity and table size limitations in previous benchmarks. We propose a system that employs effective LLM prompting to translate natural language queries into executable code, enabling accurate responses, error correction, and interpretability. Our approach ranks first in both subtasks of the competition in the proprietary model category, significantly outperforming the organizer’s baseline.

## introduction the integration of large language models (llms) into question-answering (qa) systems over tabular data has garnered significant attention in recent research, due to their capability to translate natural language into structured queries @xcite . first attempts in language-to-query conversion rely on fine-tuning pre-trained language models, so that they acquire tabular understanding @xcite . the vast contextual knowledge of llms, as well as their exposure on multiple modalities, including tabular comprehension, enables them to implicitly translate natural language in structured queries. as tabular qa arises as an emergent llm ability @xcite , more and more prompting-based methods showcase improvements @xcite . evaluating llms as tabular reasoners was explored with the databench dataset @xcite , comprising 65 datasets from various domains, accompanied by manually crafted questions in natural language. answers fall into different categories, such as boolean, categorical, numerical or list. experiments utilizing in-context learning and code-based prompting reveal that while llms show promise, there is significant room for improvement, particularly in smaller scales. in this paper, we explore a wide range of prompting strategies for large llms to identify optimal methods for converting natural language queries into code and effectively interacting with tabular data. specifically, we propose a system for generating executable python functions from natural language queries through llm prompting. our approach achieves accuracy scores of 85.63% and 87.93% in the databench and databench lite tasks respectively, ranking first in both tasks in the proprietary models category @xcite . after human evaluation, the accuracy increases to 89.85% and 88.89% in databench and databench lite respectively. in our system, llms are explicitly instructed to articulate their reasoning process, improving the reliability of their outputs and facilitating interpretability. our code is available on github foot_0 . ## system overview our system mainly performs text-to-python code conversions to queries via prompting. this allows for a lightweight and task-agnostic implementation that employs llms without further training. specifically, the system consists of two modules: the main module and the error-fixing module. an overview of the architecture provided in figure 1 . ## experimental setup dataset the dataset used for model development is publicly available on hugging face foot_1 . it contains 1,308 questions across 65 tables. we derive our exemplars from the predefined train split, comprising 988 questions from 49 tables. the columns required to answer each question, their data types, and the answer's data type are precomputed by the task organizers and included in the dataset. our annotations focus solely on the expected code and reasoning steps generated by the llm. model evaluation is conducted on the dev split, which includes 320 questions from 16 tables, as well as on the test set, which consists of 522 questions from 15 tables and serves as the basis for the final ranking. evaluation metric model performance is assessed using a relaxed accuracy measure, which accounts for predictions with equivalent semantic meaning, e.g. the prediction "yes" is considered correct if the gold answer is true. numerical answers are truncated to two decimal places before being compared against the gold answer. the evaluation metric is provided by the task organizers through the databench_eval foot_2 python package. baseline the baseline system employs a model to convert the input query into a python function that takes the table as a pandas dataframe and returns the answer to the question. the generated function is then executed by the python interpreter. the prompt includes detailed task instructions along with a simple example of a completed function. additionally, it provides the names of the columns in the input dataframe as a list. the model utilized is stable-code-3b-gguf foot_3 . ## results rankings large vs. small models our findings indicate that our system's performance is heavily influenced by the overall capabilities of the llm used. larger models achieve up to 20% higher accuracy in both subtasks, which is expected given their superior code generation capabilities. this is evident even when compared with a smaller model that is specifically trained for code generation tasks. ## conclusion in this work, we present our system for semeval-2025 task 8: question-answering over tabular data, which translates natural language questions into executable python functions using only llm prompting. our approach decomposes the task into smaller steps, explicitly demonstrating its reasoning process, and incorporates a self-correction mechanism to handle execution errors. as a result, our system ranks first in both subtasks within the proprietary model category. yunjia zhang, jordan henkel, avrilia floratou, joyce cahoon, shaleen deep, and jignesh m. patel. 2023b. reactable: enhancing react for table question answering. preprint, arxiv:2310.00815. fengbin zhu, ziyang liu, fuli feng, chao wang, moxin li, and tat seng chua. 2024. tat-llm: a specialized language model for discrete reasoning over financial tabular and textual data. in proceedings of the 5th acm international conference on ai in finance, icaif '24, page 310-318, new york, ny, usa. association for computing machinery.