NOAHQA : Numerical Reasoning with Interpretable Graph Question Answering Dataset (2021)

Abstract: While diverse question answering (QA) datasets have been proposed and contributed significantly to the development of deep learning models for QA tasks, the existing datasets fall short in two aspects. First, we lack QA datasets covering complex questions that involve answers as well as the reasoning processes to get them. As a result, the state-of-the-art QA research on numerical reasoning still focuses on simple calculations and does not provide the mathematical expressions or evidence justifying the answers. Second, the QA community has contributed a lot of effort to improve the interpretability of QA models. However, they fail to explicitly show the reasoning process, such as the evidence order for reasoning and the interactions between different pieces of evidence. To address the above shortcoming, we introduce NOAHQA, a conversational and bilingual QA dataset with questions requiring numerical reasoning with compound mathematical expressions. With NOAHQA, we develop an interpretable reasoning graph as well as the appropriate evaluation metric to measure the answer quality. We evaluate the state-of-the-art QA models trained using existing QA datasets on NOAHQA and show that the best among them can only achieve 55.5 exact match scores, while the human performance is 89.7. We also present a new QA model for generating a reasoning graph where the reasoning graph metric still has a large gap compared with that of humans, eg, 28 scores.

## introduction question answering (qa) plays a core role in natural language understanding and is a proxy to evaluate the reading comprehension ability of intelligent systems. due to its profound significance, a figure 1 : a sample of noahqa dataset, which consists of a passage and several question-answer pairs. the supporting evidences and reasoning graph are provided for correctness and interpretability evaluation. on the one hand, numerical reasoning is one of the intelligent skills of human beings. to endow such an ability to qa models, we need to provide some math word question answering dataset @xcite for training these qa models. in particular, given a math question, a good qa model should select the correct answer among the multiple pre-defined answer options. recently, dropqa @xcite includes numerical questions into conventional span-extraction question answering. however, the mathematical forms of numerical questions found in dropqa are relatively simple, e.g., most of the questions are only about addition and subtraction. besides, dropqa only provides the final answer without a full expression to the answer. on the other hand, existing qa datasets reveal drawbacks in interpretability (or explainability). as the leaderboards of qa datasets are overwhelmed by powerful language models, e.g., bert @xcite , researchers turn their attention to explainable model and the collection of qa datasets likewise pursue this trend: models should not only give the answers, but also the explanation for the answers, e.g., complex questions with discrete reasoning or numerical reasoning. the model with explanation is also friendly with digging and improving the system. to achieve the goal, coqa @xcite and hotpotqa @xcite provide the model with rational and evidences as additional supervisions. nonetheless, how the model conducts reasoning is still vague. for example, how the model process several pieces of evidence? are they processed in parallel or follow a specific order? r 4 c @xcite , and 2wikimultihopqa @xcite solve this problem to some extent by introducing a set of triplets or reasoning path, which is not suitable in the scenario where the reasoning process is complicated. taking the example in figure 1 for illustration, the answer of q 7 comes from the answer of q 5 , and the answer of q 5 comes from the passage and the answer of q 3 . to address the above shortcomings of existing qa datasets, we present noahqa, numerical reasoning with interpretable graph qa dataset. an overall comparison of the differences between noahqa and others is shown in we apply strong baselines from existing datasets on noahqa and discover that the best baseline achieves 55.50 exact match scores, while human performance is 89.67. for evaluating of reasoning graph, we introduce an automatic evaluation method named dag sim , considering the structural and semantic similarity at the same time between the predicted and ground-truth reasoning graphs. to facilitate the research along with reasoning graph, we also contribute a new model named reasoning graph network (rgnet) for generating the reasoning graph. experiments with rgnet show that there is still a large gap behind human performance with 28.01 and 18.14 dag sim scores on the english and chinese versions of noahqa, respectively. each answer a i (1 ≤ @xmath0, e i , which can be some text segments in p and/or first-order evidences of previous question-answer pairs. these first-order evidences provide the information to derive the answer a t for q t directly. unlike most qa datasets @xcite , we only use exact match (em) score as our evaluation metric for answer correctness, the f1 score is not adopted to bypass the occasion that two different but overlapped numbers may give high f1 scores, e.g., 1203.4 and 1204.4. ## data collection in this section, we describe the collection process of noahqa which consists of raw passages preparation, conversation collection, evidence labeling (translation) and validation. we elaborate each step as follows. ## data analysis for simplicity and clarity, our data analysis is based on the english version of our dataset. noahqa contains 21, 347 examples, each of which consists of a passage, a conversation, and corresponding evidence set. we randomly split the dataset into training, development, and test sets. the detailed statistics of the dataset are shown in ## experiment 6.1 experimental settings baselines. we evaluate noahqa on the models designed for four widely used datasets: 1) baselines from coqa includes seq2seq implemented by opennmt @xcite , pgnet @xcite , and flowqa @xcite . these methods utilize ground-truth history answers to answer the current question; 2) baselines (denoted as hotpot) @xcite introduced in hot-potqa dataset, which utilizes the supervised signals of evidence while answering the questions; 3) gts ( xie and sun, 2019a ) from the mwp dataset math23k. gts translates an mwp text into an arithmetic expression through a goal-driven tree decoder; 4) naqanet @xcite and its strong variant numnet+ (ran et al., 2019) from dropqa. naqanet can predict multiple types of answers. numnet+ combines naqanet with a numeric aware graph neural network. to establish human performance, we randomly sample 300 examples from the test set. for each example, we average evaluation scores of predictions from two annotators. implementation details. in rgnet, we utilize roberta @xcite and xlm-r (conneau et al., 2020) as encoding module for monolingual and cross-lingual experiments, respectively. adam (kingma and ba, 2014) is selected as optimizer in training. in the prediction module, we use one layer gru @xcite for pgnet. ## conclusion in this work, we present a new qa datasets with complex numerical questions and interpretable reasoning graph. we also introduce an automatic evaluation metric for the generated reasoning process. we finally present an initial model producing the reasoning process while answering questions. the experiments show that noahqa is challenging and will become an interesting direction in both numerical qa and explainable qa.