CCNU at S em E val-2025 Task 8: Enhancing Question Answering on Tabular Data with Two-Stage Corrections (2025)

Abstract: We present the system developed by the Central China Normal University (CCNU) team for the SemEval-2025 shared task 8, which focuses on Question-Answering (QA) for tabular data. Our approach leverages multiple Large Language Models (LLMs), conducting tabular QA as code completion. Additionally, to improve its reliability, we introduce a two-stage corrections mechanism, in which we instruct the LLM to correct the code according to the judges of whether the code is executable and whether the answer obtained from executing the code is semantically consistent with the question. The experiment demonstrates that code correction works but answer correction does not. Finally, we discuss other unsuccessful approaches explored during our development process.

## introduction large language models (llms) have demonstrated strong performance in question answering (qa) tasks @xcite , including tabular qa @xcite , where inputs consist of non-database tables. to systematically evaluate the performance of llms on tabular qa, @xcite introduced databench, a benchmark comprising a diverse collection of tabular datasets spanning various domains and question types. unfortunately, as noted in the evaluations by @xcite , llms remain unreliable for tabular qa, with substantial room for improvement across all question types and domains. to address this challenge, @xcite organized semeval-2025 task 8, based on the databench benchmark, to investigate the capability limits of llms in tabular qa. this paper presents the solution developed by the central china normal university (ccnu) team. interestingly, the evaluations by @xcite revealed that framing tabular qa as a code completion task can significantly enhance the performance of large language models (llms) compared to directly answering questions. specifically, instead of providing the entire input table, they only exposed the llm to its meta-information (e.g., column names) and instructed it to generate a python function that computes the answer to the given question. as discussed in @xcite , one limitation of framing qa as code completion is the reliance on a third actor-the python interpreter-introducing multiple steps in the qa process, each of which can introduce errors. specifically: (1) the code generated by llms may contain errors, leading the python interpreter to return error messages instead of valid answers. in the evaluations by @xcite , many incorrect responses were not actually undesired answers but rather error messages. (2) since llms in this paradigm generate code without directly accessing the final outputs their code produces, the answers may be semantically irrelevant to the question. this issue arises because llms cannot inherently ensure that the generated answers match the expected type or format of the question. our solution builds upon the concept of qa as code completion while specifically addressing the two inherent issues mentioned earlier. to achieve this, we introduce two-stage corrections. in the first stage, beyond simply generating code, our approach instructs llms to refine their output by incorporating corrections based on error messages from the python interpreter. in the second stage, llms are prompted to verify whether the results obtained from executing the code are semantically consistent with the given questions. in the following sections, we provide a detailed introduction to the databench benchmark (section 2) and our proposed solution (section 3). we then analyze the effectiveness of our two-stage corrections (section 4). while our results indicate that code correction is successful, answer correction falls short-likely due to the limitations of llms in accurately evaluating semantic consistency. finally, we discuss other unsuccessful approaches we explored during this shared task (section 5) and reflect on our key findings. ## task description semeval-2025 task 8 is built upon the databench benchmark. databench consists of 65 tabular datasets from various domains @xcite . for each dataset, they hired human participants to write 20 questions, which resulted in 1,300 questions in total. to further improve diversity, it was ensured that the collected questions covered 5 different question types, including boolean, category, list[category], and list @xcite , as shown in ## methodology as mentioned earlier, we follow grijalba et al. (2024) in formulating tabular qa as code completion and introduce a two-stage corrections mechanism to address errors arising in the two key stages of this approach: code completion and code execution. this mechanism consists of code correction and answer correction, which aim to improve the reliability of generated solutions. an overview of our method is illustrated in figure 1 . in this section, we first describe how we incorporate the approach of grijalba et al. (2024) into our solution for tabular qa, followed by a detailed explanation of the two-stage corrections mechanism. previous error: {error_message} please correct the following code: {code} the prompt begins with several lines of comments that define the task the llm needs to accomplish. specifically, it instructs the llm to generate a single line of code based on the given question and the meta-information of the table. to enhance the model's awareness of the question type, we introduce a modification to the approach of @xcite by adding an extra comment line that asks the llm to explicitly specify the question type as a line of comment before completing the code. the prompt concludes with the beginning of a function definition, leaving the final line for the llm to complete. once generated, the completed code is executed in a python interpreter to produce the final answer. main task: as a professional data analyst, your task is to determine if the generated answer is correct based on the provided tabular knowledge and question. correctness is defined as: the answer is completely consistent with the tabular knowledge and accurately answers the question. judgment criteria: reasoning requirements: as an expert, you need to provide a step-by-step reasoning process, explaining how to extract relevant information from the tabular knowledge and verify the generated answer. the reasoning process should include the following: output format: ## experiments in this section, we start with introducing the backbone llms we used in our experiments and report the experimental results. ## other unsuccessful attempts we also experimented with an ensemble solution, where multiple tabular qa systems generated answers in parallel, followed by a majority vote to determine the final response. however, this approach underperformed compared to simply using deepseek-r1 (without think) with code correction, suggesting that accuracy may lie in the hands of a select few models rather than a collective consensus. ## conclusion this paper presents the central china normal university (ccnu) team's solution to semeval-2025 task 8, the databench task, which requires systems to perform tabular qa. building on the idea of tabular qa as code completion, we further propose a two-stage correction mechanism comprising code correction and answer correction to enhance reliability. experimental results show that code correction effectively improves performance, while answer correction does not.