S amsung Research P oland at S em E val-2025 Task 8: LLM ensemble methods for QA over tabular data (2025)

Abstract: Question answering using Large Language Models has gained significant popularity inboth everyday communication and at the workplace. However, certain tasks, such as querying tables, still pose challenges for commercial and open-source chatbots powered by advanceddeep learning models. Addressing these challenges requires specialized approaches.During the SemEval-2025 Task 8 competition focused on tabular data, our solution achieved86.21% accuracy and took 2nd place out of 100 teams. In this paper we present ten methodsthat significantly improve the baseline solution. Our code is available as open-source.

## introduction in recent years, large language models (llms) have made significant advancements, emerging as powerful tools for extracting, interpreting, and generating insights from textual data. one of their most significant applications is question answering (qa), where llms provide contextually relevant responses to user queries. although llms excel in natural language understanding, they still face challenges in processing and reasoning over tabular data, particularly in understanding relationships, identifying relevant columns, and answering complex queries. with a substantial amount of realworld data stored in tabular formats, the ability to efficiently interpret and utilize structured information seems more critical than ever. ## data the training and test data we used was databench, a benchmark dataset for tabular data (osés @xcite . the authors of databench emphasised their intention to create a benchmark using "real-life" datasets, which also resulted in challenges in interpretation. the issues related to tables involved two areas (please refer to the analysis of the questions also showed that some posed greater challenges than others, which corresponded to the models' performance. we identified the following groups of issues: (1) the need for external knowledge not present in the table, (2) the necessity to operate on substrings/altered strings, (3) ambiguity in question phrasing that allows for multiple interpretations, (4) multiple possible answers that are equally correct, (5) the need to convert units, such as weight or currency, (6) complex phrasing or language mistakes. see examples in ## limitations our system was built and fine-tuned using the databench dataset. although it includes a diverse set of tables and questions, our experiments were conducted on a limited sample of real data. additional research is necessary to determine how well the proposed methods generalize to other domains. ## conclusion despite the availability of verified open-source llms, answering questions over massive tabular data is still a challenging task. designing an effective prompt is undoubtedly a crucial method, but can be difficult to control. interestingly, simple feature and system engineering, combined with common classifiers, continue to be valuable and can significantly improve qa accuracy. through our experiments in the semeval task 8, we demonstrated that using multiple models and smart voting can result in creating an effective, general-purpose tabular qa system.