Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data (2023)

Abstract: Numerical data plays a crucial role in various real-world domains like finance, economics, and science. Thus, understanding and reasoning with numbers are essential in these fields. Recent benchmarks have assessed the numerical reasoning abilities of language models, revealing their limitations in limited and specific numerical aspects. In this paper, we propose a complete hierarchical taxonomy for numerical reasoning skills, encompassing over ten reasoning types across four levels: representation, number sense, manipulation, and complex reasoning. We conduct a comprehensive evaluation of state-of-the-art models on all reasoning types. To identify challenging reasoning types for different model types, we develop a diverse and extensive set of numerical probes and measure performance shifts. By employing a semi-automated approach, we focus on the tabular Natural Language Inference (TNLI) task as a case study. While no single model excels in all reasoning types, FlanT5 (few-/zero-shot) and GPT3.5 (few-shot) demonstrate strong overall numerical reasoning skills compared to other models in our probes.

## introduction numerical data is ubiquitous in the real-world. many applications in domains such as finance, economics and science require understanding and reasoning with numbers. in recent years, benchmarks were introduced to study language models' numeracy skills @xcite @xcite . however, these datasets mostly concentrate on few, specific numerical reasoning types (e.g. scales @xcite ). moreover, evaluating models on numerical benchmarks, it often remains unclear why models struggle with the tasks. for example, the issues can arise from models struggling to recognize numerical representations in text, failing to compute arithmetic operations, or predicting incorrect outputs due to a lack of numerical commonsense knowledge. we aim to explore these questions in greater detail in this study. limitations of language models' numerical abilities, as discussed in prior research, include tokenization and representation of numbers in text @xcite , hallucination @xcite @xcite , and generalizability/robustness issues @xcite @xcite . successful numerical reasoning requires a combination of skillsets: understanding representation of numbers @xcite and their meaning in a given context @xcite , applying operations @xcite , and integrating factual and commonsense numerical knowledge to solve numerical problems @xcite . for example, classifying the hypotheses "the movie can be watched in about 2 (or 'two') hours." from in this paper, we evaluate state-of-the-art models on various numerical reasoning types. to assess which reasoning types are challenging for specific models, we create a diverse and large set of numerical probes and measure shifts in models' performance. we organize all probed reasoning types in a hierarchical taxonomy. inspired by how humans understand and reason with numbers, as well as previous numerical benchmarks, we include eleven reasoning types across four level: representation, number sense, manipulation, and complex reasoning (figure 1 ). we apply a semi-automated approaches for probe creation. we select tabular nli (tnli) as a case study task, given three criteria: (i) numerical data (numbers, percentages, dates, etc.) is prevalent in tables; (ii) tables are common in real-world data sources such as in scientific publications, database systems and financial documents; (iii) tables as structured data facilitate automated perturbations to create large-scale probing sets. see ## a taxonomy for numerical reasoning this section introduces a hierarchical taxonomy for numerical reasoning, inspired by previous works on numeracy in nlp @xcite and psychology @xcite @xcite . we group numerical reasoning skills given their complexity level in four categories: r1 -r4. ## numerical probing framework this section provides an overview of the probing framework. we use tabular natural language inference (tnli) for automated probe creation. ## probing with tnli datasets this section discussed probes in detail and how we created them for each reasoning type from §3. foot_0 numeration. to study models' understanding of string ("two") and numerical (e.g. "2") number representations, we create two types of numeration probes. one converting number representations from strings to numeric, while the second category applies the conversion vice versa. we filter hypotheses with numbers written as strings ("two") and substitute them by their numeric counterpart (e.g. "2"). the label-preserving probes are semantically equivalent to the base hypotheses and the label (e.g. entailment) is not changed. label-flipping probes replace the converted number x by a random number in the range of [x-x * 0.5; x+x * 0.5]. for example, the numeration flipping probe of h1 ( heterogeneous number types. we created heterogeneous probes for the following categories frequent in the tnli datasets: date formats, ordinals, percentage, currencies, and scientific notation. to filter base hypotheses, we applied a simple, rulebased approach specific to each category (i.e. dates formats, percentage, ordinals, etc.). to create labelpreserving probes we applied representation-level changes which did not change the semantic meaning. for h3, we substituted 3rd june, 1986 by another english date format 03-06-1986. to flip the rafael nadal plays left-handed born 3 june 1986 (age 37) height 1.85 m turned pro 2001 prize money us$116,111,561 (3rd all-time leader in earnings) base hypothesis h1 born in 1986 , nadal is age 37 currently. numeration probe h1 born in nineteen eighty six, nadal is age thirty seven currently. num flip probe h1 born in nineteen ninety two, nadal is age forty one currently. range probe h1 born in 1986, nadal is age between 31-43 currently. base hypothesis h2 the player's birth date is on 3rd june, 1986 . ## experiments and analysis next, we provide an overview of all models that were evaluated on the probes from §4. we also discuss the obtained results and insights. ## conclusion this paper presents a framework for probing language models' numerical reasoning skills. we organise skills in a taxonomy and generate large-scale sets of probes covering more than ten numerical reasoning types. using table nli as a case study, we evaluate the numerical reasoning abilities of ten models. these models belong to the categories numerical lms, tabular lms, and few-/zero-shot llms. we discuss reasoning types that prove challenging for the probed models and explore promising directions for future research. future directions. for certain (numerical) tasks, tool-augmented lms equipped with capabilities such as calculators or code execution have been proven valuable @xcite . however, certain tasks require implicit numerical reasoning which might not necessarily involve direct calculations based on numbers. for instance, classifying sentences that incorporate numbers in varied settings, like time indications @xcite , currencies or conversations @xcite . such tasks demand a numerical interpretation beyond mere arithmetic computations. moreover, calling external tools using lms requires basic numerical comprehension to invoke an external tool correctly @xcite .