Dataground at S em E val-2025 Task 8: Small LLM s and Preference Optimization for Tabular QA (2025)

Abstract: We present our submission to SemEval 2025 Task 8: Question Answering on Tabular Data, which challenges participants to develop systems capable of answering natural language questions on real-world tabular datasets. Our approach aims at generating Pandas code that can be run on such datasets to produce the desired answer. The approach consists in fine-tuning a Small Language Model (SLM) through Preference Optimization on both positive and negative examples generated by a teacher model.A base SLM is first elicited to produce the code to compute the answer to a question through a Chain of Thought (CoT) prompt. We performed extensive testing on the DataBench development set, exploring a variety of prompts, eventually settling on a detailed instruction prompt, followed by two-shot examples. Due to hardware constraints, the base model was an SLM with ${leq}$ 8 billion parameters.We then fine-tuned the model through Odds Ratio Preference Optimization (ORPO) using as training data the code produced by a teacher model on the DataBench training set. The teacher model was GPT-4o, whose code was labeled preferred, while the code generated by the base model was rejected. This increased the accuracy on the development set from 71% to 85%.Our method demonstrated robust performance in answering complex questions across diverse datasets, highlighting the effectiveness of combining small LLMs with supervised fine-tuning and automated code execution for tabular question answering.

## introduction the ability of large language models @xcite to process structured data and generate meaningful responses has become an increasingly important area of research. the semeval 2025 task 8: question answering on tabular data @xcite focuses on assessing the capacity of llms to perform question answering (qa) over tabular datasets using the newly developed databench benchmark @xcite . unlike traditional qa tasks that operate on unstructured text, this task requires models to interpret structured data, extract relevant information, and generate precise answers. the challenge lies in the complexity of real-world tabular datasets, which contain diverse column types, varying structures, and sometimes millions of rows. the databench benchmark was designed to evaluate llm performance on this task, featuring 65 real-world datasets spanning multiple domains, with 1300 hand-crafted questions and answers. our approach to solving this problem relies on code generation rather than in-context learning. instead of answering questions directly in natural language, our system generates executable python code that extracts the relevant information from the dataset to compute the answer. the approach consists of fine-tuning a small language model (slm) through reinforcement learning on both positive and negative examples produced by a teacher model. a base slm is first elicited to produce the pandas code to compute the answer to a question through a chain of thought (cot) @xcite prompt consisting of a detailed instruction and two-shot examples @xcite , which helps guide the model through a step-by-step reasoning process. due to hardware constraints, we had to resort to a small llm (≤ 8b parameters). we selected deepseek-coder-6.7b-instruct as our base model by comparing several models on the databench development set. the supervised finetuning (sft) step uses data generated by a teacher model. in our case the teacher model gpt-o4 was used to generate code on the databench training set. we used the odds ratio preference optimization (orpo) algorithm @xcite , supplying to it the responses from our base slm labeled rejected, while the gpt-4o generated responses labeled preferred. to promote transparency and reproducibility, we released our code and fine-tuned model on github: https://github.com/danielesartiano/semeval_2025_task_8 . this resource includes our prompt templates, code execution script, and fine-tuning scripts enabling further research into improving llms for tabular question answering. our findings suggest that combining code generation with preference-based fine-tuning offers a promising direction for enhancing llm capabilities on tabular qa tasks. future work may explore hybrid approaches, integrating in-context learning with code generation to leverage the strengths of both methodologies. ## system overview the system is designed to generate python code using the pandas foot_0 library to extract information from real-world datasets. we initially used the baseline foot_1 example provided by the task organizers and then extended the software by integrating prompt engineering, including chain of thought (cot) reasoning and two-shot learning, model selection, supervised fine-tuning, and automated code execution to effectively solve the challenge task. ## experimental setup we conducted our experiments on a machine equipped with a single nvidia a100 gpu with 80gb vram, only suitable to run small language models. we used the hugging face transformers library to interact with the models, retrieving all models from the huggingface hub. for evaluation, we used the databench_eval foot_3 library provided by the task organizers. the databench qa dataset, specifically the development split, which consists of 320 questions, was used in our experiments to validate and optimize our system, measuring the accuracy of several variants and refining the models iteratively. to fine-tune the models, we used the train split of the databench qa dataset, which consists of 988 questions. this dataset was leveraged to generate a preference-based fine-tuning dataset by invoking the gpt-4o model via the openai api. using the prompt specified in listing 2, we generated answers, executed the python functions, and compared their outputs with the ground truth answers from the trainset. to construct the fine-tuning dataset, we identified the correctly executed answers and paired them with responses generated by the base model selected for fine-tuning. the best performing model at that stage, deepseek-coder-6.7b-instruct, was chosen for this process (as shown in orpo fine-tuning was performed using its implementation from the huggingface libraries trl -transformer reinforcement learning 5 and the peft parameter-efficient fine-tuning. 6 the hyperparameters used are those listed in the ## results our submission was evaluated by the se-meval 2025 task 8 organizers using the official databench_eval python script. the final test set originally contained 522 questions, but after identifying errors in the ground truth, corrections were made, and 5 ambiguous questions were removed. these questions were excluded from the final evaluation resulting in a final evaluation set of 517 questions. in the general ranking, which includes both large and small lms, as well as both proprietary and open-source models, our submission achieved 31st place in the databench qa subtask with an accuracy of 68.97% and 29th place in databench lite qa subtask with an accuracy of 69.35%. this represents an improvement of approximately 43 percentage points over the baseline. in the separate ranking category for small models (≤ 8b parameters), we achieved second place in both subtasks as shown in ## conclusion in this paper, we described our submission to semeval-2025 task 8: question answering on tabular data, using a code generation approach, rather than in-context learning with a llm. we exploit and tune a sml to generate code that extracts the answers from structured datasets. by leveraging a small llm (≤8b parameters), prompt engineering, and supervised fine-tuning from a teacher model with the odds ratio preference optimization (orpo) algorithm, we significantly improved on the baseline model accuracy. our experimental results show that: in the official evaluation, our system achieved second place in both tasks, in the category small models (≤8b parameters), demonstrating the effectiveness of fine-tuning through preference optimization. despite the hardware limitations that constrained our experiments, we were able to achieve competitive results. our results confirm in particular the benefits of distilling the knowledge of a llm into a smaller model, as reported for the reasoning capabilities of deepseek r1 in @xcite we hope to be able to further explore the distillation technique using a llm more specialized on coding than the one we could use.