Adapt LLM for Multi-turn Reasoning QA using Tidy Data (2025)

Abstract: This paper presents our submission to the Fin-DBQA shared task at the 9th FinNLP workshop. The task involves answering finance-focused questions in a multi-turn environment, requiring step-by-step reasoning and Python code generation. We propose a novel approach to tackle this multidimensional problem by pre-processing the data into tidy data format so that each column represents a variable and each row an observation. Our experiments demonstrate that using the tidy data format allows all models to surpass SOTA, with GPT-4o achieving a 50.62% accuracy on the DBQR-QA benchmark achieving second place on the shared task leaderboard. These findings suggest that transforming data into the tidy data format enhances reasoning capabilities, reduces syntax errors, and improves performance on table-reasoning QA tasks. The code is available online.

## introduction when analyzing data in finance, the ability to write code that can answer complex, multi-step questions is crucial. these questions often require reasoning across multiple data sources and steps. as financial data becomes increasingly intricate, this ability to reason and generate code is vital for deriving insights and making informed decisions. recent advancements in nlp, particularly with the rise of large language models (llms), are crucial for tackling the multi-step reasoning tasks commonly encountered in finance. early research in question answering (qa) focused on zero-shot tasks using general benchmarks @xcite . however, more recent work has shifted towards more complex challenges, such as multi-hop reasoning @xcite and reasoning over tabular data @xcite , including hybrid approaches that combine it with text @xcite . this paper explores approaches for the fin-dbqa shared task @xcite , which involves answering finance-focused questions in a multi-turn conversation using a data chunk of a graph database. we approach the task using three approaches testing them on four models from two model families (llama 3.1 & gpt-4o). we generate python code using the pandas package. in addition to the question and the variables, custom python functions are integrated, and the table information from the data chunks is passed to the model. our main assumption is that models will generate code more accurately and reason better by converting tables into tidy data format (wickham, 2014), as shown in figure 1 . therefore, we compare three approaches: one using the raw data, another using a tidy-data format, and a third that extends the tidy-data approach by adding few-shot examples. the contributions of the paper are: ## related work recent advancements in reasoning-based qa have been fueled by key datasets like mmlu @xcite and gsm-8k @xcite . in tabular qa, which requires complex reasoning, notable contributions include tat-qa @xcite and finqa @xcite , which focus on hybrid financial tabular and textual data, and numerical reasoning in finance. convfinqa @xcite introduces multi-hop conversational qa using a single table, while fetaqa @xcite expands to free-form table qa. tidy data @xcite , provides a structured framework for organizing datasets to facilitate manipulation, modeling, and visualization that is used widely in data science and statistical computing. in tidy datasets, each variable is a column, each observation is a row, and each type of observational unit forms a table. this structure simplifies analysis, reduces errors, and ensures consistency across workflows. ## dbqr-qa benchmark the shared task fin-dbqa involves answering finance-focused questions in a multi-turn conversation using a data chunk of a graph database. the dbqr-qa benchmark is used for this purpose. however, in this task, only the reasoning component is tested, as the data chunks of the graph database are already given. the reasoning step involves writing a python program (with pandas) that includes logical steps and mathematical calculations. figure 3 in appendix a shows an example question from the practice dataset. each dataset contains a question, a set of variables, the queries used to fetch the data from the graph database, and a pickle file containing the data chunks. ## methodology section 4.1 outlines the prompt templates, section 4.2 covers converting data chunks into tidy data, and section 4.3 details few-shot example creation. figure 2 summarizes the three tested approaches. we provided the information displayed for each approach and the models conducted each conversation within a multi-turn environment. after each conversation, the generated code was executed in a sandboxed python environment with the pandas package installed. the custom function was imported, and all attributes were stored in globals() to allow the reuse of variables during the conversation. the value stored in the result var is then evaluated using the heuristic evaluator. ## evaluation this section presents the results of the three proposed approaches, summarized in ## conclusion in this paper, we present a novel approach using tidy data to improve performance on the dbqr-qa benchmark. we were able to show that small and quantized models perform better on the benchmark and produce fewer syntax errors using tidy data. the best performance was achieved using gpt-4o with tidy data. the results demonstrate that tidy data has a notable impact, particularly for smaller models, by simplifying data input and enhancing the model's ability to perform reasoning. for larger models, it remains highly effective, significantly reducing the error rate.