SBU - NLP at S em E val-2025 Task 8: Self-Correction and Collaboration in LLM s for Tabular Question Answering (2025)

Abstract: This paper explains the submission of the SBU-NLP team at SemEval-2025 Task 8: question-answering over tabular data. We present a novel algorithm for this task, aimed at systems capable of interpreting large tables and providing accurate answers to natural language queries. The evaluation uses the DataBench dataset, which covers a wide range of topics and reflects the complexity of real-world tabular data. Our approach incorporates a self-correction mechanism that iteratively refines LLM-generated code to address errors and prevent common mistakes. Additionally, a multi-LLM collaborative strategy is employed to generate answers, where responses from multiple LLMs are compared, and the majority consensus or a valid alternative is selected. The method relies exclusively on open-source models, avoiding costly processes like training or fine-tuning. Experimental results demonstrate that combining multiple LLMs with self-correction leads to significant performance improvements. However, challenges arise with list-based answers and responses involving multiple numerical, string, or boolean values, where further refinement is needed. The proposed simple system was among the top performers in both Subtask A and Subtask B among open-source models in the competition.

## introduction task 8 @xcite focuses on question-answering over tabular data. the goal is to evaluate systems that can effectively interpret large tables and provide accurate answers to natural language questions based on the data within those tables. the evaluating dataset, databench @xcite , is important because many real-world applications rely on tabular data, and everyday datasets can vary significantly in both the subjects they cover and their size. databench covers a wide range of topics, making it essential to evaluate systems on diverse data to ensure they can handle a variety of real-world scenarios. our system, which we used for this task, employs a novel, simple algorithm based on a selfcorrection mechanism. in this approach, the large language model (llm)-generated code from the prompt is iteratively fed back into the llm to fix errors and avoid common code mistakes. additionally, we use a multi-llm collaborative answer generation strategy, where answers from multiple llms are compared, and the majority consensus is used. in cases where errors due to code issues prevent the llm from generating an appropriate answer, alternative answers from other llms are utilized. furthermore, if the generated answers are not in a valid format, a valid answer from another llm for the same question is adopted. this method uses models that are either open-source or open-weight, avoiding computationally costly procedures like training and fine-tuning. participating in this task has led to several key discoveries. the results show that leveraging multiple llms and employing a self-correction mechanism significantly improved performance. additionally, a well-structured python code generation prompt played a crucial role in obtaining better answers from tables. however, we also discovered that many of the errors in our system were related to cases where the answers were in list format or involved more than one numerical, string, or boolean value. if these specific errors could be better handled, the system's results could be further improved. in terms of ranking, our model performed well among open-source and open-weight models, securing 3rd place out of 37 teams in subtask b and 6th place out of 37 teams in subtask a.you can find the code for our system in the fol-lowing github repository foot_0 . ## background question answering on tabular data is a critical task in natural language processing. over the years, researchers have developed diverse methodologies to improve the accuracy and efficiency of this process. the task is about question answering from tabular data. for this goal, a newly introduced benchmark, databench @xcite , has been used for evaluation, which consists of different large tables that vary in the topics they cover. as input, like the example in figure 1 , a natural language-based question with a table name is given, and the expectation is to find an appropriate answer from the related tables. for the final competition tests, 15 tables were provided. in subtask a, the tables can be of any size, while in subtask b, the row count is fewer than 20 due to the contextlength limitations for llms. we participated in both subtasks with the same system. ## system overview in the question-answering task on tabular data, several challenges may arise. due to limitations in hardware infrastructure, we employed a promptbased approach 2.2 to achieve results while avoiding strategies that rely on fine-tuning or training procedures 2.1. in our experiments, we exclusively used opensource and open-weight models and avoided thirdparty and commercial models due to their high costs. to generate code, we utilized prompts in llms. however, because of context-length limitations, it was not feasible to include tables in the prompt, particularly in subtask a. instead, a codegenerating prompt was provided to the llm, and the generated code was subsequently executed. one of the main challenges encountered was errors in the generated code. to address this, we implemented iterative prompts for code selfcorrection. however, not all errors could be resolved through iterative prompting alone. in such cases, we leveraged responses from alternative llms to rectify mistakes. since different llms may not exhibit identical errors for a given question, we used responses from other models. after all, to avoid mistakes due to data types, a post-processing algorithm was run to ensure the correct format of the expected output data. our framework consists of three main components: ## self-correction mechanism let c t be the generated code at iteration t, and e t be the error encountered during execution. the llm refines c t iteratively: until execution produces no errors. the self-correction mechanism involves iterative refinement of a generated code to handle errors encountered during execution. the process starts with an initial code generation, followed by error detection. if an error is detected, the algorithm refines the code using an error-handling prompt in appendix b. the errorhandling prompt is added to the code generation prompt, which is described in appendix a. this loop continues until the code executes successfully or a specified maximum number of iterations (in our experiments, 5 iterations) is reached. if errors persist after the limit, the algorithm returns a failure. the detailed steps of this mechanism are outlined in algorithm 1. ## results in cases where both llms produced error-free answers that differed from each other, one of them was preferred. because our experiment includes only two llms, we simply selected the response from the other llm. the results are presented in ## conclusion this paper presents an innovative algorithm leveraging a self-correction mechanism and a multi-llm collaborative answer generation approach to address the key challenges in question answering from tabular data. by incorporating iterative errorchecking in code generation and utilizing collaborative solutions to ensure valid expected answers, our method significantly reduces errors and inappropriate responses. our error analysis highlights that most issues arise when the answer is a list rather than a simple data type, such as a number, string, or boolean. future work will focus on refining the prompting strategies to minimize errors in such scenarios.