FERMAT : An Alternative to Accuracy for Numerical Reasoning (2023)

Abstract: While pre-trained language models achieve impressive performance on various NLP benchmarks, they still struggle with tasks that require numerical reasoning. Recent advances in improving numerical reasoning are mostly achieved using very large language models that contain billions of parameters and are not accessible to everyone. In addition, numerical reasoning is measured using a single score on existing datasets. As a result, we do not have a clear understanding of the strengths and shortcomings of existing models on different numerical reasoning aspects and therefore, potential ways to improve them apart from scaling them up. Inspired by CheckList (Ribeiro et al., 2020), we introduce a multi-view evaluation set for numerical reasoning in English, called FERMAT. Instead of reporting a single score on a whole dataset, FERMAT evaluates models on various key numerical reasoning aspects such as number understanding, mathematical operations, and training dependency. Apart from providing a comprehensive evaluation of models on different numerical reasoning aspects, FERMAT enables a systematic and automated generation of an arbitrarily large training or evaluation set for each aspect. The datasets and codes are publicly available to generate further multi-view data for ulterior tasks and languages.

## introduction numerical reasoning is an aspect that is often forgotten despite being an integral part of natural language. it is the ability to interact with numbers using the fundamental mathematical properties and thus model an area of human cognitive thinking @xcite . better understanding of numbers in language models would benefit various tasks like fact-checking @xcite ), text generation @xcite , and educational tools @xcite . current models' performance are still too weak with respect to numerical accuracy to then be used in downstream tasks like infotabs @xcite which requires identifying numbers in tables and then performing operations to correctly label statements causing factuality errors in such tasks. recently, we have observed improved performances on relevant datasets about numerical reasoning using very large language models @xcite @xcite . however, there are two main limitations to this recent trend. first, as models become larger their access becomes restricted to fewer users, i.e., users with the computational resources of large companies. for example, using one of the best mathematical models, the 540b parameter model minerva @xcite , would require over 2212g of memory for inference only. second, the numerical reasoning capabilities of existing models are measured using a single score, i.e., mostly accuracy on common benchmarks like gsm8k @xcite . therefore, their strengths and shortcomings in different aspects of numerical reasoning compared to other models are not clear. as a result, it is unclear what numerical reasoning aspects should be improved to improve their performance on datasets requiring numerical reasoning. motivated by checklist @xcite , which is a behavioral test set concerning various linguistic aspects of the input language, we propose a unique and open flexible evaluation set for representating multiviews of arithmetic types, 2 fermat, for evaluating the numerical reasoning capabilities of models based on multiple key aspects . it evaluates models according to (a) different ranges and representations of numbers, (b) different mathematical operations, and (c) the dependence of models on the fine-tuning data. in addition, it contains a tool to automatically generate new instances for each of its aspects. fermat enables (a) the identification of the strength and shortcomings of models according to its aspects, and (b) the automatic creation of additional training and evaluation instances using expert written templates that reflect fermat's categories. fermat complements the recently proposed l īla benchmark @xcite for mathematical reasoning. l īla evaluates high-level aspects, e.g. whether performing mathematical reasoning also depends on commonsense knowledge or how the performance changes depending on the difficulty of the input language. however, even the best-performing model on the l īla benchmark, i.e., a 2.7b parameter model that is fine-tuned on mathematical datasets, only achieves an accuracy of around 20-30 points when the input is formulated using a simple language and the test data is from a different distribution than that of the training, and it is not clear how to further improve this performance. fermat, on the other hand, takes a deeper look at more fine-grained aspects by diving into the core mathematical abilities of the models and reporting which specific operations a model can or cannot perform and on which numbers. it also provides templates for creating more instances for each aspect, e.g., to generate additional data to further train or evaluate models on certain aspects. fermat formulates the evaluation of numerical reasoning using the question answering format, which is commonly used in nlp for evaluating various skills @xcite @xcite . we use fermat to highlight that single accuracy scores fail to give a holistic understanding of a model, that template diversity has a high impact in improving performance, and that number encodings play an important part in numerical reasoning. the fermat framework could subsequently be adapted for different tasks according to the target application, foot_2 to give a more targeted approach to improving models. moreover, while the expertwritten templates in fermat are written in english, they can easily be translated to be adapted to other languages. mathematical datasets focus on exploring different levels of difficulties and areas of maths. some look at general symbolic maths, where the questions at least involve algebraic notations. a certain group of datasets explores numerical reasoning in context, but the answers may not exclusively be numerical. unlike fermat, all these datasets evaluate models' performances on the whole dataset based on a single score. moreover, as a result of the availability of many datasets, new benchmarks have also been created based on regrouping the existing datasets according to specific criteria. such benchmarks are created based on high-level aspects, e.g., how the performance changes when solving maths also depends on commonsense reasoning, when the maths is presented using equations, a simple language, or a complex language, or when the input is presented using a different task format. however, the performance of existing general-purpose models is very low, even on the simplest aspects, e.g., when the maths is presented using a simple language without requiring external knowledge. fer-mat, on the other hand, focuses on a fine-grained analysis of numerical reasoning by aiming to decipher models' ability to understand numbers, operations, and their reliance on the training data. ## multi-view evaluation set: fermat fermat gives a holistic view of a model by evaluating fine-detailed aspects of numerical reasoning. it is akin to ribeiro et al. (2020)'s checklist, which focuses on linguistic variations for defining its aspects. fermat is used to interpret models by evaluating them on three orthogonal views including (a) number understanding, (b) mathematical operations, and (c) training dependency. it also provides an automated method of generating new training or evaluation examples for a given number type or operation. we collect the initial instances for creating the fermat evaluation set using the established illinois @xcite and commoncore @xcite datasets. after removing duplicates, we collect 1111 unique instances from these two datasets which we name the original set. 7 we choose instances from commoncore and illinois because they perfectly fit with fermat's design by providing one or two-hop questions. moreover, their extensive annotation is supplemented with an alignment between the numbers in the question and the corresponding expression that the solution is calculated from. we leverage these annotations in fermat to create different variations of the same problem for different aspects. ## experimental setup to demonstrate the effectiveness of our evaluation set, fermat, we will perform the evaluations in two settings, (a) zero-shot, where we evaluate existing models, and (b) fine-tuned, where we further train the models on arithmetic data generated using our training data in section 3.4. ## results ## conclusion the majority of existing datasets for numerical reasoning evaluate models based on a single score, making it impossible to identify their strengths and shortcomings to further improve them. multi-view benchmarks are the alternative for a more comprehensive and informative evaluation of models. in this direction, we introduce fermat, a multi-view evaluation set that enables a fine-grained analysis of models based on three key aspects including number understanding, mathematical operations, and training dependency. fermat's aspects are associated with separate templates for generating instances for both evaluation and training sets, which are collected from completely independent sources and domains. our results confirm that comparing a single accuracy score, as with all existing maths datasets, is not representative of the performance on various numerical reasoning aspects as the evaluation dataset may be skewed towards a specific data distribution. based on our results, a wider language and mathematical variation can improve even smaller models. however, an apparent future direction is to focus on improving number encodings in existing models and understanding how these affect performance.