Numerical reasoning in machine reading comprehension tasks: are we there yet? (2021)

Abstract: Numerical reasoning based machine reading comprehension is a task that involves reading comprehension along with using arithmetic operations such as addition, subtraction, sorting and counting. The DROP benchmark (Dua et al., 2019) is a recent dataset that has inspired the design of NLP models aimed at solving this task. The current standings of these models in the DROP leaderboard, over standard metrics, suggests that the models have achieved near-human performance. However, does this mean that these models have learned to reason? In this paper, we present a controlled study on some of the top-performing model architectures for the task of numerical reasoning. Our observations suggest that the standard metrics are incapable of measuring progress towards such tasks.

## introduction machine reading comprehension (mrc) primarily involves building automated models that are capable of answering arbitrary natural language questions on a given textual context such as a paragraph. solving this problem should in principle require a fine-grained understanding of the question and comprehension of the textual context to arrive at the correct answer. designing an mrc benchmark is challenging, as it is easy to inadvertently craft questions that allow models to exploit cues that allow them to bypass the intended reasoning required @xcite . recent advances in nlp, currently dominated by transformer-based pre-trained models, have resulted in models that indicate, when measured with standard metrics, human-like performance on a variety of benchmarks over leaderboards for mrc 1 . in this paper, we focus on numerical reasoning based mrc and investigate drop @xcite , a recent benchmark designed to measure 1 for eg., https://leaderboard.allenai.org/ complex multi-hop and discrete reasoning, including numerical reasoning foot_0 . in contrast to singlespan extraction tasks, drop allows sets of spans, numbers, and dates as possible answers. we are particularly interested in numerical questions, for eg., 'how many years did it take for the population to decrease to about 1100 from 10000?' which requires extracting the corresponding years for the associated populations from the given passage, followed by computing the time difference in years. the benchmark has inspired the design of specialized bert and embedding-based nlp models aimed at solving this task, seemingly achieving near-human performance (evaluated using f1 scores) as reported in the drop leaderboard. in this work, we investigate some of drop's top-performing models on the leaderboard in order to understand the extent to which these models are capable of performing numerical reasoning, in contrast to relying on spurious cues. we probe the models with a variety of perturbation techniques to assess how well models understand the question, and to what extent such models are basing the answers on the textual evidence. we show that the top-performing models can accurately answer a significant portion (with performance exceeding 35%-61% f1) of the samples even with completely garbled questions. we further observe that, for a large portion of comparison style questions, these models are able to accurately answer without even having access to the relevant textual context. these observations call into question the evaluation paradigm that only uses standard quantitative measures such as f1 scores and accuracy. the ranking on the leaderboards can lead to a false belief that nlp models have achieved human parity in such complex tasks. we advocate the community to move towards more comprehensive analyses especially for leaderboards and for measuring progress. ## dataset and models in this section, we briefly highlight the dataset and models under consideration. dataset drop @xcite includes questions with various types of reasoning, of which we are interested here in numerical reasoning. we filter the provided dev set to only include questions that require numerical reasoning; this was done by first including all answers with type number, and then augmenting with comparison questions, filtered heuristically based on whether it contains a comparative adjective or a comparative adverb. 5850 questions were number and 998 were comparison. we call this numset and use it in this paper as a basis for all experiments 3 . ## evaluating question understanding evaluating if a model understands questions is a non-trivial task. in this paper, we probe the performance of the models in the following two ways: evaluation of the models with question permutation and investigating the affinity to question class. question permutation experiment inspired by recent observations in pham (2020), we perturb the numset by shuffling the words of the corresponding question in each sample. we create three randomly permuted sets, that differ in the n-gram permutation. 1-gram shuffle refers to all words being shuffled, 2-gram and 3-gram refer to corresponding shuffles of ordered 2 and 3 grams in the question. for the question mentioned in the introduction, an example of the possible shuffles for each of the {3,2,1}-grams are: 3-gram 10000 about 1100 from did it take to decrease to how many years for the population? 2-gram population to it take how many about 1100 from 10000 for the years did decrease to? 1-gram 10000 for from years to decrease it population about 1100 did many to take how the? as the random permutations may distort the semantic meaning (and destroy the syntax) of the question, we expect the predictions of the models to be severely impacted (with results approaching that of random chance). however, our experiments reveal (in figure 1 ) that this is not the case. generally the trend suggests that 3-gram permutations do not generally degrade the performance severely. while we notice a that 1-gram permutation degrades the performance, we still observe that the models tend to predict a significant portion of questions correctly (>35% f1 score). some models are barely affected, such as naqanet. we remark that most models that are based on bert or bagof-embeddings seem to be generally more robust to permutation than roberta based models. we present an analysis on the effect of the number of numerical attributes in the passage and the ability of the models to make correct predictions in affinity to the class of questions we further probe models on their affinity to answer questions by only relying on the class of questions. as the questions in drop follow a certain pattern the fig 2 shows the performance of the models on partial questions. with only five words, most models can still maintain a third of their correct predictions, and with only the first trigram of the question, they obtain an @xmath0.4%, where nerd obtains 15.42%. further showing an affinity of the models to be able to answer questions by exploiting the mere presence of a few words in the question. unlike the permutation experiment, here breaking down performance based on existence of numerical attributes in the passage shows a steady decline in performance with more numbers in a passage (see ## evaluating passage comprehension we now examine whether the models are comprehending the passage and if they are basing their random passage we pair each question with a randomly assigned passage from numset. ## related work contrast sets @xcite and semantics altering modifications (sams) (schlegel et al., 2020) are two works that introduce changes to mrc benchmarks to better understand the decision boundaries of the models. they include a subset of the drop dataset, wherein the former the benchmark's authors manually modify questions to include more compositional reasoning steps or change their semantics to create a contrast set. in the latter, the authors introduce an automatic way of generating sams, which alter the semantics of a sentence while preserving most of its lexical surface form. in our work, we take an inverse approach, where we alter the surface form of a question such that it no longer carries the meaning of the original question. ## discussion and conclusion in this work, we closely examined some of the top-performing models for numerical reasoning on drop. our study suggests that models are not necessarily arriving at the correct answer by reasoning about the question and content of the passage. both question understanding and passage comprehension experiments reveal serious holes in the way the models are able to arrive at the correct answers. we hypothesize that the models have managed to pick up on the spurious patterns of the benchmark, rather than solving the task. possible reasons for biases include: patterns in the format of passages and questions, where passages either describe the outcome of an american football match, the census of a certain location, or some historical event @xcite , resulting in redundancies in the structure of a passage and patterns in their content; and answer frequency distribution, with top 5 answers being shared between train and dev splits, covering almost 20% of the data. in fact, we found that in the affinity to the class of questions experiments, there exists a vast disparity in performance between questions with most-frequently occurring answers vs. others. in nerd, for example, the em for questions with 2 words is 17.75% in top-10 answers, whereas it is 4.2% in the rest. the disparity narrows with more words, as it is 21.5% vs. 11.90% for questions with 3 words, and 30.81% vs. 22.23% for questions with 5 words. benchmark leaderboards as they stand now can be misleading, incentivizing models to improve upon the reported scores without solving the underlying task. we strongly advocate for better methods to assess the capability of models for numerical reasoning. one such direction could be akin to linzen (2020) who proposes a parallel evaluation paradigm that rewards models for possessing human-like generalization capabilities and @xcite that augments current leaderboards with three extra dimensions of interpretability, interactivity, and reliability. we highly recommend for careful design of the benchmarks and better leaderboards to correctly measure progress in such complex tasks.