DBQR - QA : A Question Answering Dataset on a Hybrid of Database Querying and Reasoning (2024)

Abstract: This paper introduces the Database Querying and Reasoning Dataset for Question Answering (DBQR-QA), aimed at addressing the gap in current question-answering (QA) research by emphasizing the essential processes of database querying and reasoning to answer questions. Specifically designed to accommodate sequential questions and multi-hop queries, DBQR-QA more accurately mirrors the dynamics of real-world information retrieval and analysis, with a particular focus on the financial reports of US companies. The dataset’s construction, the challenges encountered during its development, the performance of large language models on this dataset, and a human evaluation are thoroughly discussed to illustrate the dataset’s complexity and highlight future research directions in querying and reasoning tasks.

## introduction question answering (qa) is a fundamental task in the field of natural language processing (nlp). previous studies have primarily focused on textbased qa @xcite @xcite . recently, attention has shifted towards tabular-based qa @xcite and hybrid qa @xcite @xcite . these studies base all experiments on the necessary tables provided as input. however, in real-world scenarios, answering questions like "what is the difference in revenue between apple inc. and meta platforms, inc. in 2023?" (q1) requires two steps: querying and reasoning. specifically, models need to first query the revenues of the two companies in 2023 and then perform mathematical reasoning to answer the question. to address the gap in previous studies concerning the querying step, we propose the database querying and reasoning dataset for question answering (dbqr-qa). taking this a step further, people often ask sequential questions during discussions. for example, a follow-up to q1 could be "did that figure increase from 2022?" (q2). to answer q2, models must first identify the coreference (i.e., "that" refers to the difference in revenues between the two companies). the next steps involve querying the data for 2022, calculating the difference for that year, and comparing the answers to q1 and q2. another follow-up question might be "which company had the highest revenue in the technology sector in 2023?" this question, while related to q1, does not require information from q1 to answer. however, it demands a multi-hop querying capability, as it involves navigating from the industry to specific companies. this is considered a multi-hop query because the model must first identify the technology sector, then find companies linked to it. in dbqr-qa. this paper explores model capabilities in both scenarios. previous studies @xcite @xcite have shown that company financial reports are excellent resources for investigating the proposed task. these reports contain numerous tables and figures, and investors and analysts frequently discuss them. additionally, comparing reports between companies is a common practice. following this rationale, we also base our dataset on the financial reports of us companies. another advantage is that all these data are open access, enabling future research to extend our efforts in both querying and reasoning tasks. in this paper, we detail the construction of dbqr-qa and the challenges encountered in building this dataset. we also investigate the performance of large language models (llms) on this dataset and compare its difficulty with that of existing datasets. furthermore, we provide a human evaluation to highlight issues in both querying and reasoning steps, identifying potential challenges for future research. ## related work question-answering has posed a long-standing challenge for language models, prompting the proposal of numerous new datasets to explore and enhance the capabilities of these models. this section presents works associated with the three aspects of our dataset: tables, reasoning, and conversation. llms have made significant advances in reasoning qa, as demonstrated by the drop dataset @xcite for reading comprehension and arithmetic qa, the gsm-8k dataset @xcite for grade-school math word problems, the mmlu benchmark @xcite for multiple-domain multiple-choice questions, and the numhg @xcite for numberfocused headline generation. tabular qa is another area that requires reasoning skills. notable datasets in this field include the tat-qa dataset @xcite for hybrid financial tabular and textual data, the finqa dataset @xcite for numerical reasoning task in finance, and the fetaqa dataset @xcite for free-form table qa. built on top of tat-qa and finqa, our dataset expands the scope of reasoning by incorporating querying and reasoning in problem-solving. numerous conversational qa datasets target large knowledge bases as they allow for diverse multi-hop questions. some notable datasets in this domain include sqa @xcite for wikipedia tables, csqa @xcite for reasoning over a knowledge base, and convquestions @xcite covering five domains. non-kb qa datasets also pose significant challenges, for instance, coqa @xcite for machine comprehension and quac @xcite in dialog contexts. despite extensive research on conversational qa for many years, its tabular and reasoning aspects still require further attention, as evident in convfinqa @xcite . this dataset explores the chain of numerical reasoning in single-table conversational qa. our dataset raises the complexity in table manipulation, cross-question chain of reasoning, and multipletable qa from a financial knowledge graph. here is an example of the question and gold label for this question in our dataset: question: during the period from 2016 to 2021, what were the top two years itron reported the highest r&d expense? gold label: ## dataset statistics with advancements in nlp, zero-shot learning has emerged as a prominent method in the study of llms. our proposed model, dbqr-qa, is specifically designed for this context, aiming to evaluate the ability of llms to generate code for querying and reasoning. instead of creating a large dataset for supervised learning, we have developed a carefully constructed test set for precise evaluation. in the convfinqa benchmark, the test set consists of 434 instances. in line with these benchmarks, dbqr-qa provides a similar scale with 400 qa pairs. the proposed dbqr-qa is categorized into five subsets based on question type and complexity, and it introduces a structured variety of question types aimed at examing querying and reasoning skills. these categories are crafted to delve into the complex aspects of financial datasets, focusing on distinct objectives and complexities. below, an introduction to the five unique question types in our dataset is provided, alongside examples in ## experimental result 6.1 from querying to reasoning in order to take an in-depth look at the problems that llms face when addressing the questions in the proposed dataset, we manually checked the problems that occur in simple and complex questions. this testing involves analyzing the output exceptions. we also checked the results in the intermediate steps. the following are the common problems we found: 3. negative output from the difference operation (d): the program outputs a negative answer when asked for differences between numbers. ## discussion based on our experimental results, gpt-4 exhibits satisfactory performance in both simple and complex subsets. however, its performance is inferior in the remaining subsets. therefore, in this section, we focus on discussing the simple and complex subsets to understand the nuances of llms' performances on the proposed dataset. ## conclusion by introducing a comprehensive dataset tailored to evaluate the querying and reasoning capabilities of llms, the dbqr-qa dataset challenges models with real-world scenarios that require both database queries and multi-step reasoning processes, reflecting the complexities encountered in professional financial analysis. experimental results reveal that while llms like gpt-4 demonstrate potential in certain subsets, significant challenges remain, particularly in subsets requiring advanced reasoning and multi-step querying capabilities. this observation underscores the ongoing need for improvements in models and specialized training to bridge the gap between current capabilities and the demands of real-world data analysis tasks. lastly, although our dataset focuses on the financial domain, interested researchers and developers can apply the creation and evaluation processes to use cases where tabular reasoning over a database can benefit question-answering. companies and organizations can also apply our approach to build a qa system to conduct complex analyses of their databases for various purposes, for example, inventory/logistic/resource management, fraud/irregularity detection, system optimization, and supporting information for policy/decisionmaking. in addition to the dataset, all fine-grained annotations for analysis will be shared under the cc by-nc-sa 4.0 license. 6 limitation the paper's limitations can be summarized as follows: first, while our study adopts financial data in line with prior research, this specificity means that our findings are predominantly tailored to financial documents. explorations into other domains rich in database, such as scientific research or the clinical realm, would be beneficial in subsequent studies. second, our primary emphasis is on assessing the capabilities of llms in a zero-shot setting. future investigations might consider broadening the scope to explore our dataset under varied experimental paradigms. third, consistent with prior works, our dataset is confined to english. it's worth noting that llms might exhibit different behaviors across different languages, and we advocate for this consideration in future endeavors.