N um GLUE : A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022)

Abstract: Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE that was proposed in the context of natural language understanding, we propose NumGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4 %). Further, NumGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4 % on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NumGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning.

## introduction reasoning with numbers is an important skill that occurs in various day-to-day scenarios and not surprisingly, numbers are ubiquitous in textual data. to train ai reasoning systems that can perform simple mathematical reasoning, many tasks have been proposed @xcite @xcite . despite these efforts, current state-of-the-art ai 1 https://allenai.org/data/numglue original word problem john had 5 apples. he gave 3 to peter. how many apples does john have now? fill in the blanks format john had 5 apples. he gave 3 to peter. john has apples now. nli format premise: john had 5 apples. he gave 3 apples to peter. hypothesis: john has 2 apples now. does the hypothesis entail, contradict or is neutral to the premise? comparison format john had 5 apples. he gave 3 to peter. who has more apples? figure 1 : a system that can robustly perform numeric reasoning over language should be able to solve problems such as the above, regardless of how the problem is posed. however, we observe existing systems are brittle; producing inconsistent solutions to such minor stylistic variations. ## numglue as mentioned previously, our numglue benchmark consists of both new and already existing arithmetic reasoning tasks. we first begin by introducing the novel datasets curated by us before providing a brief overview of existing tasks that are part of numglue. finally, in this section, we provide an analysis of the datasets demonstrating that it contains interesting and diverse linguistic and mathematical properties. ## experiments in this section, we establish multiple baselines on our benchmark and discuss their performance. ## results and discussion does the benchmark contain bias that a model can exploit? a challenging dataset requires the model to ideally consider all the information provided to it before arriving at an answer. to ensure that this is indeed the case, we perform ablations where only one portion of the input is provided i.e. either the question or the context. both these "bias-checking" baselines perform poorly even in task-specific settingindicating that both the benchmark and constituent tasks are challenging. which tasks are hard to solve? our results show that task 1 which requires numerical commonsense knowledge, is the hardest task to solve. similarly, tasks 2, 4 and 8 appear to be comparatively harder from the rest. one pattern among these tasks is that all of them expect the answer to be numeric. numeric answer requires accurate calculation. so, models might have difficulty in learning the task directly from data. this hypothesis is also justified from the slight drop in human performance in these tasks.. on the other hand, task 7 has the best performance among all. further, we see that performance on task 6 is slightly better than task 5 -although both tasks are sourced from the same dataset, we observe that models answer span based questions better as compared to numeric answers. relatively higher performance for task 3 suggests that models find it easier to answer in an mcq setting. ## conclusion we propose numglue, a multi-task benchmark to test for arithmetic understanding. our benchmark consists of eight tasks including four new ones. while some of the tasks require external knowledge like commonsense or domain-specific information in addition to arithmetic reasoning, some are self-contained e.g. arithmetic word problems. further, we demonstrate that our benchmark is far from being solved -with state-of-the-art large scale models achieving considerably lower performance than humans. this indicates that current ai systems are incapable of performing simple arithmetic reasoning in a general setting -indicating a fundamental hurdle towards ai systems that understand complex mathematical concepts like differential equations or combinatorics. finally, we present various baselines including a novel architecture (memory augmented ex-numnet) that demonstrate the advantages of various modeling choices (e.g. end-to-end vs neuro-symbolic models). specifically, we show that training in the multi-task setting leads to meaningful sharing of knowledge across tasks as evidenced by an average gain of 3.4% on tasks compared to task-specific modeling. finally, we hope that our benchmark not only leads to ai systems that are capable of performing simple arithmetic reasoning in a fairly general setting but also results in progress towards more complex mathematical reasoning capability.