S em E val-2025 Task 8: Question Answering over Tabular Data (2025)

Abstract: We introduce the findings and results of SemEval-2025 Task 8: Question Answering over Tabular Data. We featured two subtasks, DataBench and DataBench Lite. DataBench consists on question answering over tabular data, and DataBench Lite small comprising small datasets that might be easier to manage by current models by for example fitting them into a prompt. The task was open for any approach, but their answer has to conform to a required typing format. In this paper we present the task, analyze a number of system submissions and discuss the results. The results show how approaches leveraging LLMs dominated the task, with larger models exhibiting a considerably superior performance compared to small models.

## introduction large language models (llms) have demonstrated emerging capabilities @xcite , with one of the latest recognized tasks being question answering (qa) on tabular data @xcite . qa on tabular data, as illustrated in figure 1 , involves responding to natural language queries using structured information stored in tables. different approaches exist for retrieving answers, including translating natural language questions into formal programming languages like sql, which can then be used to interact with databases @xcite @xcite . since these models are widely applied across various domains, ensuring their accurate evaluation is essential. however, the research community currently lacks a comprehensive evaluation benchmark to assess and compare different llms and prompting strategies for this task. benchmarking in tabular qa has traditionally relied on a limited set of collections, such as figure 1 : examples of correct and wrong answers to simple and factual questions made on tabular data @xcite @xcite , which are predominantly based on tables extracted from wikipedia. while these datasets have been widely used, they exhibit common characteristics-such as low data variety and a small number of columns-that make them less representative of the complex tabular data encountered in real-world applications. additionally, wikipedia tables often pose challenges related to scale, data cleanliness, and structural limitations. to address these shortcomings, we introduced databench at lrec-coling 2024 @xcite , a novel benchmark designed to provide a more diverse and realistic evaluation framework for question answering on tabular data. databench consists of real-world datasets from various domains, featuring large and heterogeneous tables, along with a rich collection of annotated question-answer pairs. the availability of databench encourages us to challenge the research community to design qa on tabular data models with the ability of processing and answering questions about data stored in tables. accordingly, we propose the tabular qa task with the enough freedom to encourage the creativity of researchers to provide solutions to this challenge. as we will describe hereinafter, we provide two versions of databench, and one of them is oriented to facilitate the use of models that are not able to process large contexts by presenting datasets small enough to fit whole into a single prompt by whatever representation the users desire. the task has arisen the participation of more than 100 teams, with 35 research teams having submitted a system description paper. the top-ranked models evidence the superiority of models leveraging llms, and among them those with large size of parameters. however, the approaches also followed evidence that the task needs smart prompting strategies, hence the size of the models is not the only important feature at play. the top ranked systems for each kind are described further in the results section. codabench. the competition has been hosted on codabench foot_0 where you can find more details, examples and links for the relevant test set data. ## datasets databench was introduced in @xcite to provide a dataset for question answering over tabular data, addressing challenges commonly found in real-world data that are often absent from existing datasets, which primarily consist of wikipedia tables. by incorporating these complexities, databench offers a more reliable benchmark for developing models capable of handling such data. the dataset includes 65 datasets spanning various domains, as detailed in databench also includes 1,300 tagged qa pairs, providing insights into the types of columns being used. the questions, as illustrated in databench lite databench lite was also introduced along databench and it contains sampled versions of all datasets and answers for each sampled version. the size of this is essentially the same as databench and databench lite are hosted publicly on huggingface. 2 train and development sets the full set of databench was divided in two sets: the train set, containing the first 40 datasets, and the dev set, containing the last 15. this artificial distinction was made in order to facilitate evaluating on the development set during the first phase of the competition, but otherwise participants were encouraged to use the two sets as they found best fit. ## task organization the task was run in two phases, making use of the codabench platform. both databench and databench lite were evaluated on all submissions at the same time, but participants could make a submission with either of them, or both. in the platform we only evaluated accuracy against a ground truth set for both subtasks, allowing participants to freely build any systems they would like to solve the task. they were informed of requirements to share the type of model used, code and general descriptions in order to qualify for the final ranking. rankings for both subtasks were also made separately, and special rankings were provided for small models of up to 9 billion parameters and open source models in general. evaluation script. participants were also provided with a python package (databench_eval foot_2 ) to make the process of making a submission more streamlined in case they wished to use it. due to the open-ended nature of the task, we opted for an open-source evaluation function that heuristically evaluates the output provided by the users against the ground truth depending on the type expected. the function returns either true or false for a given pair of response and ground truth and according to the expected semantic value of the response. we then add up the percentage of true values to compute the accuracy. this evaluation carries a number of checks to allow for smaller models to commit some small format errors that would otherwise hinder their performance, such as trimming down extra spaces in the response or allowing for drifts in calculations smaller than the second decimal position for numerical values. this function was open-sourced from the start and received feedback from participants which resulted in a number of small changes which can be seen through the history of the github repo. given that llms generate text of any kind, and the almost infinite possibilities of format changes for a given answer, we also performed a manual evaluation of the results shown in the final ranking in order to ensure small formatting mistakes were mitigated as much as possible. this manual evaluation in the competition phase was done for the top 10 results for each category, and resulted only in very small changes which did not in the end affect the rankings. development phase running from the 8th of september 2024 to the 9th of january 2025. in this phase participants were only provided with the full train and development sets, including tags on the columns where the answer was to be extracted from and the type expected of the answer. they were free to make as many submissions as they wanted and had full access to their scores. a public ranking was available on the platform where participants could freely choose to display their results, but were not forced to do so. participants were provided with a minimal baseline script that could be executed locally without any gpu on most consumer hardware and yielded 28.05 and 30.22 of accuracy for databench and databench lite respectively with the use of a 4bit quantized version of the stable-coder-3b model @xcite . this baseline is still available in the github page for the evaluation benchmark. competition phase the full blind test set, including only the questions from the test set, was released on 9 january 2025. the competition ran until 31 january 2025. during this phase, participants were allowed only three submissions and the public ranking was not available. after the competition, participants who wished to take part in the final ranking had to fill out a google form containing the information on their system described earlier. among all participants, thirty-five teams used an approach based purely on open-weight models, while sixteen teams used a proprietary model. a separate category was created for open-source models with fewer than 9 billion parameters, as these models can efficiently run after 4-bit quantization is performed run on cpu on modern consumer hardware and were the focus of our pilot task. ## results in this section, we present the main results of the competition for those that chose to submit a google form. the best small open source approach, scottyposeidon, ranked 25th in the competition, and they become more common as we approach the bottom of the ranking. the best approach overall claims to use exclusively big open source models. baseline results. the result of executing the baseline we provided the participants with (see section 4 for more details) over the test set data yielded 26.00 and 27.00 accuracy scores for databench and databench lite respectively, making the difficulty of this task approximately the same as the proposed development set. the baseline used was intentionally very simple, mimicking the approach followed for the pilot task and getting similar results as the ones displayed in our original paper @xcite . results by answer type. we have also averaged all of the results of all submissions by type in in the following, we provide more details of participants' ranking in two additional categories we enabled in the task: (1) open models (including methods that rely on non-proprietary models and have at least their weights available) and (2) small models (with models consisting of lower than 9 billion parameters). open-only rankings are displayed in ## conclusions and future work in this paper, we presented the semeval task on question answering over tabular data. the test set consisted of tabular datasets from different domains and questions about different types. the results of the competition suggest that existing models can answer these types of questions reliably, as long as they are tailored and specialized in the task. in general, out of the box models cannot solve the task and struggle for the most part, and the results suggest there is a need for specialised and in-domain trained solutions beyond llms. moreover, smaller llms (below 9b parameters) are far from the best performing models, and reinforce the challenging nature of the task for general-domain models. regarding the expansion of databench to languages different from english, we also released databenchspa for the spanish language @xcite , including an accompanying shared task 4 . this is an example of an expansion to a different language, but in general most languages do not have a suitable benchmark, and therefore there is plenty of room for future work in this area.