Fin- DBQA Shared-task: Database Querying and Reasoning (2025)

Abstract: This paper presents the results of the Fin-DBQA shared task based on a question-answering dataset, focusing on database querying and reasoning. The dataset, consisting of 400 questions grouped into 40 conversations, evaluates language models’ abilities to answer sequential questions with complex reasoning and multi-hop queries in a multi-turn conversational question-answering setting. Each sample includes the question, answer, database queries, querying result (tables), and a program (series of operations) that produces the answer from the result. We received 52 submissions from three participants, with scores significantly surpassing the baselines. One participant submitted a paper detailing a prompt-based solution using large language models with additional data preprocessing that helps improve the overall performance.

## introduction while earlier research on question answering has predominantly focused on text-based qa systems @xcite @xcite , recent efforts have expanded to include tabular qa @xcite , and hybrid qa approaches @xcite @xcite . these advancements, however, typically assume that all required tables or datasets are provided as inputs during experimentation. in contrast, real-world scenarios often involve more complex requirements. for example, answering a question like "what is the difference in net profit between amazon and microsoft in 2023?" (q1) necessitates a two-step process: querying relevant data and performing reasoning. specifically, models must retrieve the revenues of the two companies for 2023 and subsequently apply mathematical reasoning to compute the difference. in a conversational question setting, users build on previous queries. a user might ask, "did that figure increase from the previous year?" (q2). to answer q2, a model must first resolve the coreference ("that" refers to the revenue difference from q1), then retrieve the relevant data for 2022, compute the difference for that year, and compare it to the result from q1. alternatively, a follow-up question might be unrelated to q1 yet require complex reasoning, such as, "which company had the highest revenue in the technology sector in 2023?" answering this involves multi-hop querying: the model must first identify the technology sector, then locate the relevant companies, and finally compare their revenues. these examples highlight the challenges of sequential and multi-hop question answering, where models must integrate reasoning, coreference resolution, and data navigation to provide accurate answers. to address the limitations of previous studies concerning the querying step in question answering, we introduce the fin-dbqa shared task based on the dbqr-qa dataset @xcite . this task is built around a novel questionanswering dataset designed to evaluate database querying and reasoning capabilities. the dataset comprises 400 questions organized into 40 conversations, enabling the assessment of language models in handling sequential, multi-hop queries within a multi-turn conversational setting. each data sample includes the question, its answer, corresponding database queries, the querying results (tables), and a program detailing the operations required to derive the answer from the results. the task attracted 52 submissions from three participants, with performance metrics significantly surpassing the established baselines. one participant proposed a prompt-based approach leveraging large language models, complemented by additional data preprocessing techniques, which further enhanced overall performance. ## related work large language models (llms) have achieved significant advancements in reasoning-based question answering (qa). this progress is evident across diverse benchmarks, including the drop dataset @xcite for reading comprehension and arithmetic qa, the gsm-8k dataset @xcite for solving grade-school math word problems, the mmlu benchmark @xcite for multi-domain multiple-choice questions, and the numhg dataset @xcite for number-focused headline generation. tabular qa is another domain that demands advanced reasoning skills. key datasets in this area include tat-qa (zhu et al., 2021), which focuses on hybrid financial tabular and textual data; finqa @xcite , designed for numerical reasoning in finance; and fetaqa @xcite , which supports free-form table question answering. building upon the foundations of tat-qa and finqa, our dataset extends the scope of reasoning by integrating querying and reasoning into the problem-solving process. numerous conversational qa datasets focus on large knowledge bases, enabling diverse multi-hop questions. notable examples include sqa @xcite for wikipedia tables, csqa @xcite for reasoning over knowledge bases, and convquestions @xcite , which spans five domains. non-knowledge-base qa datasets also present significant challenges, such as coqa @xcite for machine comprehension and quac @xcite for dialog-based contexts. despite years of extensive research in conversational qa, its tabular and reasoning aspects remain underexplored. convfinqa @xcite addresses this gap by focusing on numerical reasoning chains within single-table conversational qa. the questions in the proposed dbqr-qa dataset were derived from the tat-qa and finqa datasets, both of which were manually crafted and annotated by financial experts. however, the limited variety of reasoning operations in these datasets led to many questions exhibiting similarities. to address this, similar questions were grouped into a template-based representation. using bart @xcite , these elements were extracted to generate generalized templates. for example, the question "what was the net revenue in 2019?" was abstracted to "what was the [concept] in @xcite ?" this abstraction process involved calculating string similarity scores, grouping templates by similarity, and refining them to align with the graph database context, extending beyond simple tabular data. similar to convfinqa, dbqr-qa converts questions from finqa into a conversational format, but it differentiates itself by incorporating table manipulation throughout the reasoning process. unlike convfinqa, which relies on only six basic arithmetic operators-such as addition, subtraction, multiplication, and division, dbqr-qa includes 26 operators within the pandas dataframe. this expanded set of operators facilitates more complex and expressive queries compared to previous datasets. after establishing the question templates, we populated them with entities (e.g., companies), financial concepts, and numerical data, ensuring alignment with the us-gaap taxonomy. we prioritized terms based on their frequency of occurrence in the questions, selecting those represented in the graph to guarantee the accuracy of the generated answers. next, we defined a set of operations and combined them to create a program for each question, marking the initial stage of the annotation process. ## dataset statistics the dbqr-qa dataset is divided into five distinct subsets, each categorized according to question type and complexity. this classification introduces a diverse range of question types designed to assess querying and reasoning abilities. these categories are specifically structured to explore the intricate aspects of financial datasets, addressing various objectives and levels of complexity. an overview of the five unique question types within the dataset is presented below. ## results a total of 5 submissions were received for the practice set (50 questions), 2 for the training set (200 questions), and 45 for the test set (150 questions). each set included all five types of questions. ## conclusion the fin-dbqa shared task highlights the challenges associated with addressing multi-turn conversational question-answering that involves complex reasoning and multi-hop queries. while the solutions proposed by participants achieved performance metrics significantly surpassing the baseline, considerable scope for improvement remains, offering opportunities for further advancements in future research.