MINDS at S em E val-2025 Task 8: Question Answering Over Tabular Data via Large Language Model-generated SQL Queries (2025)

Abstract: The growing capabilities of Large Language Models (LLMs) have opened up new opportunities for answering questions based on structured data. However, LLMs often struggle to directly handle tabular data and provide accurate, grounded answers. This paper addresses the challenge of Question Answering (QA) over tabular data, specifically in the context of SemEval-2025 Task 8. We propose an LLM-based pipeline that generates SQL queries to extract answers from tabular datasets. Our system leverages In-Context Learning to produce queries, which are then executed on structured tables, to produce the final answers. We demonstrate that our solution performs effectively in a few-shot setup and scales well across tables of different sizes. Additionally, we conduct a data-driven error analysis to highlight scenarios where the model encounters difficulties. We make the code available at https://github.com/fgiobergia/SemEval2025-Task8.

## introduction the introduction of general purpose large language models has made it possible to address a wide variety of tasks, with no need to fine-tune a specific model every time. this capability has enabled a widespread adoption of llms to address a wide variety of tasks (e.g., machine translation @xcite , document classification @xcite and summarization @xcite ). however, these models often provide answers that are either based on the data available in the training data (i.e., they cannot leverage external data), or that are not grounded in factual data -a phenomenon referred to as hallucinations. this can lead to issues like generating inaccurate facts, fabricating citations, or incorrect summarizations despite the model seeming confident in its output. although attempts have been made to detect and mitigate hallucinations @xcite , off-the-shelf models used for in-context learning still cannot provide meaningful answers to questions related to a specific data source, unless access to the data source itself is provided. one such example is the answering of questions that can only be addressed based on the contents of a separate data source. the semeval 2025 task 8 -question answering over tabular data @xcite addresses exactly this kind of scenario, by framing a question answering (qa) problem, with answers that can be extracted from tabular data. in this paper we address the challenge by building a llm-based pipeline that receives information on the structure of the target table, produces a sql query to answer the question, and provides a final answer by executing the query on the tabular data. we show that the proposed solution achieves remarkable results in few-shot mode, and that it provides satisfactory performance regardless of table size. we additionally perform a data-driven error analysis to identify corner cases that the llm cannot easily address; and improve model performance by manually labelling useful cases to be used as shots within the prompt. the rest of the paper is organized as follows: section 2 introduces the task, dataset and metrics. section 3 presents the main methodology adopted, with an overview of the pipeline, as well as the error analysis that has been conducted to identify promising shots to be used. section 4 presents the main results obtained. finally, section 5 wraps up the paper, with considerations and possible future extensions of the work. ## problem description semeval task 8 consists in addressing qa problems based on tabular data. to this end, the task makes use of databench @xcite . databench is the first benchmark that makes use of real-world tables, with a wide variety of distinct questions related to various data types. answers to the questions are in the form of either a number, a test sample q: "how old is the oldest customer?" table annotated training data q/query/table prompt generator prompt shot1: ... … q: … llm unaligned query column aligner aligned query query executor postprocessing answer categorical value, a boolean value or lists of several types. databench includes a training set (49 tables, 988 questions), a validation set (16 tables, 320 questions) and a test set, specifically released for semeval (15 tables, 522 questions). the challenge consists of two subtasks: a full one, where the entire table (all rows, all columns) are used, and a lite one, where only a subset of all rows and columns are used. the quality of the provided answers is quantified in terms of the fraction of correctly answered questions (i.e., accuracy). ## proposed methodology we present the architecture of the overall solution in figure 1 . the architecture takes any one question (and related table) and produces the prompt with the prompt generator. next, this prompt is used to generate an answer query from an llm. this unaligned query may contain some errors that are adjusted by a column aligner module. the correct query is then automatically executed on the target sqlite table. the answer returned by the query is post-processed to produce the final answer. the rest of this section presents, in more details, the various blocks. ## discussion and conclusions in this paper we presented a solution to the qa problem on tabular data from semeval 2025 task 8. the solution is based on generating sql queries that are executed to produce answers to the proposed questions. we make use of an instructiontuned llm trained for code generation. we show that the pipeline allows to achieve acceptable performance. we tested several models and find codestral v0.1 22b to be the one providing the most accurate results. interestingly, almost all considered models show no gap in performance between the full and the lite versions of the task, indicating robustness to noisy information (e.g., the presence of unused columns). we acknowledge that one of the main limitations of the proposed approach is the conversion of the datasets into sqlite tables: the original parquet datasets contain potentially complex data types (e.g., lists of values), which cannot be easily cast into sqlite columns. as such, we expect that a transposition of the proposed solution to other querying approaches (e.g., python-based) may yield even more promising results.