H oward U niversity- AI 4 PC at S em E val-2025 Task 8: D eep T ab C oder - Code-based Retrieval and In-context Learning for Question-Answering over Tabular Data (2025)

Abstract: This paper presents our approach, named DeepTabCoder, to SemEval 2025 - Task 8: DataBench, which focuses on question-answering over tabular data. We utilize a code-based retrieval system combined with in-context learning, which generates and executes code to answer questions, leveraging DeepSeek-V3 for code generation. DeepTabCoder outperforms the baseline, achieving accuracies of 81.42% on the DataBench dataset and 80.46% on the DataBench Lite dataset.

## introduction recent advances in large language models (llms) have significantly improved open-domain question answering; however, question answering over structured tabular data remains a challenging problem. unlike text-based qa, tabular qa requires the model to understand schema structures, handle a variety of data types, and perform logical and numerical operations over cell values. additionally, models must operate without external knowledge sources, relying solely on the information contained within the tables themselves. semeval 2025 task 8: databench (grijalba et al., 2024) focuses on question-answering over tabular data, introducing a large-scale benchmark designed to evaluate how well models extract and reason over structured information. the dataset includes 65 diverse tables from multiple domains, each varying in size, structure, and data types, making it a comprehensive test for tabular reasoning. accompanying these datasets are 1,300 manually curated questions, covering different answer types: boolean (true/false), categorical values, numerical values, and lists. the competition itself features a subset of 15 datasets and 522 questions, providing a focused yet challenging evaluation setting. unlike open-domain qa, models must (1) derive answers solely from the provided tables, (2) handle heterogeneous table schemas, and (3) execute complex queries without relying on external knowledge. the task is further divided into two categories: databench, which uses the full datasets, and databench lite, which provides a smaller 20-row sample for each dataset, testing a model's ability to generalize with limited data. deeptabcoder follows a three-step approach leveraging in-context learning to tailor prompts for each dataset. first, we generate dataset-specific prompts that include metadata and relevant schema details to guide the model. second, we inject the question into the dataset-specific prompt and infer the response from the model. third, we extract the python code generated by the model and execute it against the dataset to retrieve the answer. by maintaining modular and reusable functions, our method ensures flexibility across different tabular structures. we extend and modify the approach from tool-augmented reasoning framework for tables (tart) @xcite , which integrates llms with specialized tools to enhance table understanding and numerical reasoning. tart consists of three key components: a table formatter for accurate data representation, a tool maker for constructing computational tools, and an explanation generator for interpretability. we adapt tart's methodology to better align with the specific requirements of the databench competition, focusing on structured table reasoning. our system combines fixed schema templates with code execution to handle diverse table structures. the system shows particular strength in boolean reasoning and numerical queries, though challenges remain in complex aggregation tasks requiring multi-hop reasoning. detailed implementation and results analysis are presented in subsequent sections. ## related works in this section, we review prior research across four key areas that form the basis of deeptabcoder. for each area, we explain how our work extends existing methods, with special emphasis on our modification to the tart framework. ## system overview this section presents deeptabcoder's system architecture and methodology for generating domainspecific prompts that enable models to synthesize python programs to answer user queries q regarding datasets d. the complete implementation of deeptabcoder can be found in appendix a. ## experimental setup we use the databench and databench lite datasets provided for semeval 2025 task 8. each datasetspecific prompt is constructed by extracting schema metadata (column names and the first row), defining modular utility functions such as load_data and get_row_by_name, and appending the corresponding query q. the prompts are designed to minimize verbosity while providing sufficient context for code generation, as illustrated in template a. this is a condensed version; full prompt examples are available in the project's github repository. during inference, the structured prompt is passed to deepseek-v3, which generates a python code snippet intended to answer the query. the generated code is executed in a sandboxed python environment to produce the final prediction. the same code c is used for both the full databench datasets and their lite versions without regeneration. evaluation is performed using the databench_eval foot_1 package provided by the organizers. we report overall accuracy along with per-category accuracies across boolean, categorical, numerical, list of categories, and list of numbers. ## results in this section, we present the evaluation results for deeptabcoder on two datasets: databench and databench lite. we analyze the overall accuracy as well as the category-specific performance for both datasets using databench_eval package. the tables and accompanying analysis provide a detailed overview of the model's performance across different categories. ## limitations despite demonstrating strong performance in tabular question answering, deeptabcoder still has several limitations that need to be addressed. one major limitation is the handling of complex queries. it struggles with queries that require multi-hop rea-soning and advanced aggregation. while code execution helps with computation, the model sometimes generates incorrect logic or fails to retrieve the correct subset of data. another challenge arises with list-based output generation. as observed in our results, the model's accuracy on questions requiring a list of categories (e.g., list[category]) is significantly lower compared to other categories. this indicates difficulties in aggregating and structuring multiple categorical responses correctly. the model faces challenges with code generation, as it is not always correct, leading to runtime errors. although executing model-generated python code improves precision, errors in syntax or logic occasionally occur, requiring additional checks. ## future work to address these issues, we propose several directions for future research. enhancing multi-hop reasoning through explicit decomposition and intermediate verification could improve query resolution. for list-based outputs, structured decoding and refined prompt engineering may lead to better aggregation. to improve code generation accuracy, future work should incorporate additional calls to smaller llms that verify and correct errors in the generated code after the initial output, thus enhancing execution reliability. adaptive schema understanding could also be improved using schema-agnostic or meta-learning techniques. furthermore, we plan to investigate domain-specific fine-tuning and sophisticated post-processing strategies to improve the handling of aggregation and multi-hop reasoning tasks. given limited co-location of programming and other languages, special evaluations of multilingual @xcite and code-switched text @xcite will also be considered, especially low-resource languages @xcite @xcite . these enhancements are expected to bolster the robustness and generalizability of deeptabcoder in diverse real-world tabular reasoning scenarios ## conclusion in this work, we presented deeptabcoder to se-meval 2025 task 8: databench, which combines code-based retrieval with in-context learning and dataset-specific prompt engineering for question answering over tabular data. by utilizing deepseek-v3 for python code generation and execution, we achieved an improvement of approximately 3.13 times on the databench dataset (81.42% vs. 26%) and 2.98 times on the databench lite dataset (80.46% vs. 27%) compared to the baseline. while deeptabcoder demonstrates strong performance in tasks such as boolean reasoning and numerical queries, we observed challenges with tasks requiring multi-hop reasoning and list-based outputs, particularly in handling aggregation and multi-category responses. this highlights areas for further improvement, including enhanced multihop reasoning capabilities, more effective handling of complex aggregations, and improved code generation accuracy through additional error correction and debugging mechanisms. overall, our results suggest that deeptabcoder, with future enhancements, has the potential to offer a robust solution for tabular question answering, addressing both schema diversity and complex query execution. further work will focus on refining deeptabcoder's capabilities through multi-hop reasoning enhancements, improved code generation accuracy, adaptive schema understanding, and domain-specific fine-tuning to ensure robustness and generalization across diverse real-world tabular reasoning tasks. for multilingual codeswitching sentiment classification. journal of computing sciences in colleges, 39(3):337-346. saurav k aryal, howard prioleau, and gloria washington. 2022. sentiment classification of code-switched text using pre-trained multilingual embeddings and segmentation. arxiv preprint arxiv:2210.16461. saurav k aryal, howard prioleau, gloria washington, and legand burge. 2023c. evaluating ensembled transformers for multilingual code-switched sentiment analysis. in 2023 international conference on computational science and computational intelligence (csci), pages 165-173. ieee. saurav keshari aryal and gaurav adhikari. 2023. evaluating impact of emoticons and pre-processing on sentiment classification of translated african tweets. iclr tiny papers.