Saama Technologies at S em E val-2025 Task 8: Few-shot prompting with LLM -generated examples for question answering on tabular data (2025)

Abstract: For SemEval 2025 Task 8, addressing tabular data question answering, we introduce a novel few-shot prompting system that guides large language models (LLMs) to generate Python code representing the reasoning process. Our system automatically creates a library of exemplar code snippets from training data, which are then used for few-shot prompting. Crucially, we incorporate a selection prompt to choose the best candidate code from multiple LLM-generated options, improving robustness and accuracy. Our system achieved competitive results, ranking 17th in the Open Model track and 25th overall. Ablation studies demonstrate the effectiveness of our exemplar generation and code selection strategies. We conclude with a discussion of limitations and promising avenues for future research.

## introduction recent rapid advancement of large language models (llms) has innovated a wide range of natural language processing (nlp) tasks @xcite ; @xcite , as the extended context size enables llms to handle much more complicated tasks. yet, question answering on tabular data requires more strategic approaches as even extended context size cannot handle full size of large tables effectively. semeval 2025 task 8 1 @xcite deals with this problem by providing the challenge for the recently developed databench dataset. through this dataset, semeval 2025 task 8 provides hundreds of questions and associated tables written in english along with answers and related information. in this paper, we explain our approach to se-meval 2025 task 8 in detail. to provide llms guides to steps to deduce the answer from the table, we generated example python codes out of the provided questions and tables in the training data. with these examples, we performed few-shot prompting followed by a selection prompt to choose the most proper code to answer the question among multiple candidate codes. we applied our optimal approach, identified through extensive experimentation, to the competition test set. our submitted system achieved a rank of 17th in the open model track and 25th overall on the semeval 2025 task 8 leaderboard. ablation studies were conducted to evaluate the contribution of each system component. further analysis of our example generation method revealed a need for improvement in handling boolean-type questions. finally, we propose several directions for future research to enhance system performance. ## background as in many other nlp tasks, question answering using llms has shown notable progress recently thanks to a variety of approaches including such as chain-of-thought prompting @xcite , retrieval-augmented-generation @xcite , knowledge distillation @xcite , and hybrid approaches @xcite . among different kinds of question answering tasks, question answering on tabular data is distinguished from other question answering tasks by the fact that its provided tabular data are structured and can be arbitrarily long. to deal with these problems, most approaches rely on non-natural languages such as sql or python to represent logical steps deducing a subtable or an answer from the given table jin ## system overview our approach is consisted of three steps. first, we generate example python codes using zero-shot prompting on the llm with the training data. second, we do few-shot prompting on the llm using the example codes generated in the first step to generate candidate python codes for the input question. finally, we run a selection prompt to select the most suitable code to answer the given question. by executing this selected code, we obtain the final result for the given question. this entire process is summarized in fig. 1 . ## results we present the results of our experiments in ## discussions to see how effective each component of our approach is, we may compare different experiment results listed in ## conclusion we presented a novel few-shot prompting system for tabular data question answering in se-meval 2025 task 8. our approach leverages llm-generated python code as reasoning guides and incorporates a selection prompt to enhance output quality. this system achieved a 17th-place ranking in the open model track and 25th overall. our analysis highlighted the contributions of individual components and identified key areas for future research, particularly in handling boolean queries.