M ark QA : A large scale KBQA dataset with numerical reasoning (2023)

Abstract: While question answering over knowledge bases (KBQA) has shown progress in addressing factoid questions, KBQA with numerical reasoning remains relatively unexplored. In this paper, we focus on the complex numerical reasoning in KBQA, and propose a new task, NR-KBQA, which necessitates the ability to perform both multi-hop reasoning and numerical reasoning. We also design a logic form in Python format called PyQL to represent the reasoning process of numerical reasoning questions. To facilitate the development of NR-KBQA, we present a large NR-KBQA dataset called MarkQA, which is automatically constructed by a small set of seeds. Each question in MarkQA is annotated with its corresponding SPARQL query, alongside the step-by-step reasoning path in the QDMR format and PyQL program. Experimental results of some state-of-the-art QA methods performed on the MarkQA dataset show that complex numerical reasoning in KBQA faces great challenges.

## introduction knowledge-based question answering (kbqa) aims to answer a question over a knowledge base (kb). it has emerged as a user-friendly solution to access the massive structured knowledge in kbs @xcite . among the extensive knowledge stored in kbs, the quantitative property is the 3rd most popular property on wikidata (only less than wikibaseitem and externalid), and there are more than 95m facts with quantitive properties. given the large amount of quantitative facts stored in kb and the ability to perform precise symbolic reasoning through query languages, it is natural to use kbqa as a solution to real-world problems that require numerical reasoning. however, it is shown that existing kbqa datasets are insufficient for numerical reasoning. we find that only 10% and 16.2% of questions in complexwebquestions(cwq) @xcite and grailqa @xcite , respectively, require numerical reasoning, but this part of the questions just focuses on some aggregation operations and lacks complex multi-step numerical reasoning. the remaining only need to match a graph pattern on kb (multi-hop reasoning), without the need to perform numerical reasoning. as a result, the questions that require complex numerical reasoning has not been covered by previous datasets (e.g. "how much more vat do you have to pay to buy the most expensive iphone 13 in russia than in japan?" or "how many times longer is the longest aircraft carrier than the shortest?"). in this paper, we propose a new challenging task, nr-kbqa (knowledge-based question answering with numerical reasoning). different from traditional kbqa which mainly focuses on multi-hop reasoning, nr-kbqa focuses on numerical reasoning and its combination with multi-hop reasoning. as shown in the left part of figure 1 , multi-hop reasoning needs to match a graph pattern in the kb, the difficulty of which comes from the composition of kb items (entities, relations, and classes). on the other hand, the difficulty of numerical reasoning comes from the composition of mathematical operators. it is worth noting that computational tasks entail the involvement of multiple operands, while each operand may be obtained by matching a graph pattern on the kb. consequently, the combination of multi-hop reasoning and numerical reasoning will lead to a combinatorial explosion of the number of logical form structures, which poses a huge obstacle to semantic parsing models. to support the study of this task, we construct a large-scale dataset called markqa (complex numerical reasoning over knowledge base question answering dataset) which starts from 1k questions posed by humans and automatically scales to 32k examples. each question in markqa is equipped with a question decomposition in the widely used qdmr format and our proposed corresponding logic form in python, namely pyql. the qdmr can be seen as an explicit reasoning path and regarded as a question decomposition resource. meanwhile, pyql not only acts as a reasoning step but can also be directly transformed into a sparql query. it offers a more humanreadable alternative to sparql and can easily be extended for further developments. we believe that markqa will serve as a valuable resource to foster further development of kbqa. 1 the remainder of this paper is organized as follows: section 2 reviews related work. section 3 defines the nr-kbqa and introduces pyql. the construction and analysis of markqa are demonstrated in section 4. section 5 presents experimental results. we conclude our contributions and future work in section 6. ## nr-kbqa in this section, we present the formal definition of a question with numerical reasoning (nrq), which is the base of nr-kbqa. then, we introduce pyql to showcase the reasoning steps of nrq. ## conclusion in summary, our contributions are as follows: aside from providing convenience for annotation, it also has superiority when serving as a supervised signal for complex reasoning. overall, we for the first time explore and discuss numerical reasoning in kbqa from task formulation, reasoning paths representation, dataset construction, and baseline implementation. we believe this work should prompt the research in this area.