Team Anotheroption at S em E val-2025 Task 8: Bridging the Gap Between Open-Source and Proprietary LLM s in Table QA (2025)

Abstract: This paper presents a system developed for SemEval 2025 Task 8: Question Answering (QA) over tabular data. Our approach integrates several key components: text-to-SQL and text-to-Code generation modules, a self-correction mechanism, and a retrieval-augmented generation (RAG). Additionally, it includes an end-to-end (E2E) module, all orchestrated by a large language model (LLM). Through ablation studies, we analyzed the effects of different parts of our pipeline and identified the challenges that are still present in this field. During the evaluation phase of the competition, our solution achieved an accuracy of 80%, resulting in a top-13 ranking among the 38 participating teams. Our pipeline demonstrates a significant improvement in accuracy for open-source models and achieves a performance comparable to proprietary LLMs in QA tasks over tables.

## introduction accessing structured data through natural language (nl) queries is crucial in various fields. however, converting nl into operations that retrieve outputs such as strings, numbers, booleans, or lists continues to be a significant challenge. this paper outlines our team's participation in semeval 2025 task 8: databench @xcite . this competition assesses questionanswering (qa) systems working with tabular data, taking into account various formats, data quality issues, and complex question types. our aim is to develop a system that accurately retrieves answers from tables, despite challenges such as missing values, inconsistencies, and ambiguous queries. we focus on improving large language models (llms) using chain-of-thought (cot) reasoning @xcite . by integrating reasoning-inducing prompts, we enhance llmbased code generation and decision-making. additionally, our approach includes an end-to-end (e2e) pipeline and an llm orchestrator to improve accuracy. to further refine performance, we implement several techniques aimed at reducing model forgetfulness. we introduce structured checklists to help the model verify each step, reducing errors in multihop reasoning. by combining prompt engineering, structured reasoning, and workflow optimizations, our system aims to achieve high exact-match accuracy. this paper details our system's architecture and key techniques (sec. 3), dataset overview (sec. 4), and experimental results (sec. 5). our findings offer key insights for enhancing llm-driven qa for structured data, bridging the gap between open-source and proprietary models. notably, our development set results showed that open-source llms achieved an accuracy of 88%, surpassing gpt-4o's 74%. ## related work question answering (qa) over tabular data has gained significant attention due to the growing need for structured information retrieval @xcite @xcite @xcite . research in this field has progressed with key datasets, such as fetaqa @xcite and chartqa @xcite , as well as a large wikipedia-based dataset, open-wikitable @xcite . various methodologies have been explored in recent surveys @xcite , including reinforcement learning and selective classification for textto-sql @xcite @xcite , pre-trained deep learning models @xcite , and few-shot prompting techniques @xcite . building on previous work, we introduce a hybrid llm-based pipeline that combines multiple techniques to improve performance. ## system description our system, as shown in fig. 3 , leverages llms and consists of the following key elements: ## experiments and evaluation this section details the experiments conducted to evaluate our system's performance in table qa . more experiments are in appx. a.1 and a.2. as shown in fig. 2 , reformulating the question generally decreases accuracy (we discuss why in sec. 5.4), as seen in setups like "codetral python + reformulation" (68%) and "minimax end-to-end + reformulation" (52%), both of which perform worse than their non-reformulated counterparts. whereas, adding sql capabilities tends to increase accuracy, with "codetral python + sql" reaching 84%. the highest accuracy is achieved when multiple models are combined and orchestrated, such as "codetral and qwen coder python (with orchestrator)" and "full open-source pipeline (with orchestrator)", both achieving 88%, surpassing single-model approaches. ## comparison with proprietary models the integration of multiple components showcased the potential of open-source llms for solving qa tasks over tabular data. as fig. 4 illustrates, we rank codabench id team score 1 xiongsishi teleai 95.02 2 pbujno srpol ais 89.66 13 anotheroption anotheroption 80.08 baseline stable-code-3b-gguf 26.00 ## official results we ranked in the top 13 out of 38 teams in the competition's opensource-models-only section @xcite , achieving an accuracy score of 80% on the databench evaluation as well as on a lite part of the benchmark. our official results on databench part of the task are presented in tab. 2, showing that we significantly outperformed the baseline by 54 points. the best solution achieved a score of 95.20. in the global ranking presented in tab. 3, which includes proprietary models, we placed in the top 20 out of 53 teams while exclusively using open-source models. ## conclusion we introduced a comprehensive system for qa over tables, showcasing that well-orchestrated open-source models can rival proprietary solutions. we tested various methods: some risked errors, while others improved accuracy and reliability of the system. our system ranked among the top 13 teams with 80% accuracy. future work could explore dynamic pipeline selection -automatically determining whether a question requires codebased execution, semantic analysis, or hybrid approaches -to optimize efficiency and accuracy. additionally, enhancing the orchestrator's capacity to detect and correct logical inconsistencies in candidate answers could further improve robustness.