Representing Numbers in NLP : a Survey and a Vision (2021)

Abstract: NLP systems rarely give special consideration to numbers found in text. This starkly contrasts with the consensus in neuroscience that, in the brain, numbers are represented differently from words. We arrange recent NLP work on numeracy into a comprehensive taxonomy of tasks and methods. We break down the subjective notion of numeracy into 7 subtasks, arranged along two dimensions: granularity (exact vs approximate) and units (abstract vs grounded). We analyze the myriad representational choices made by over a dozen previously published number encoders and decoders. We synthesize best practices for representing numbers in text and articulate a vision for holistic numeracy in NLP, comprised of design trade-offs and a unified evaluation.

## introduction numbers are an integral part of text. to understand a simple sentence like i woke up at 11, we need not just literacy but also numeracy. we must decode the string 11 to the quantity 11 and infer 11 to denote a time of the day, probably 11 a.m. we need commonsense to reason that 11 a.m. is quite late in the morning. this interpretation of 11 is strongly contextual, as i earn $11 per month evokes different units and value expectations. note how the semantics remains the same for both sentences if 11 was replaced by 10, i.e., the context is tolerant to some variability. numbers are everywhere. reasoning with quantities and counts is crucial to understanding the world. evolutionary learning has given numerical cognition skills to several animals, including human beings @xcite . our ancient ancestors furthered numeracy by developing multiple number systems, similar to but independent from the evolution of languages. numeracy is an essential skill for language understanding, since numbers are often interspersed in text: the 6 million pages in english wikipedia have over 150 million numbers. numbers are neglected. in nlp, however, numbers are either filtered out explicitly during preprocessing @xcite , or treated the same as words, often collapsing them into an unk token. subword tokenization approaches like bpe @xcite and wordpiece @xcite instead retain numbers, but split them into arbitrary tokens, for example 1234 might be split into two tokens as 12-34 or 123-4 or 1-234. recent work has shown that these are suboptimal number representations @xcite . on the drop question answering benchmark, bert performs five times worse when the answer is a number instead of a span of text @xcite . relatively simple strategies like switching from subword to char-level tokenization @xcite , or from decimal to scientific notation @xcite ) already boost performance. such results warrant a deeper study into the best number representations. numbers are important. given the ubiquity of numbers and their fundamental differences with words, enabling nlp systems to represent them effectively is beneficial for domains like scientific articles @xcite and financial documents @xcite . number understanding is also useful to detect sarcasm @xcite and to model dialogues involving price negotiations @xcite . recent nlp progress towards numeracy has been sporadic but encouraging. in this paper, we survey prior work and highlight the kind of numeracy targeted (e.g., arithmetic, measurement, numeration) as well as the kind of representation used (e.g., value embeddings, digitrnns). we provide the first nlp-centric taxonomy of numeracy tasks (section 2) and of number representations (section 3) for the reader to succinctly comprehend the challenge posed by numeracy. we synthesize key takeaways (section 5) and propose a unifying vision for future research (section 6). ## tasks there are several different aspects of numeracy. the drop dataset alone offers a wide variety of numeric reasoning questions such as retrieval-based (how many yards did brady run?), count-based (how many goals were scored? given a comprehension describing multiple goals), and simple arithmetic (how many years after event 1 did event 2 occur? given dates of both events). besides downstream applications, there have also been probing experiments to evaluate whether nlp models can decode numbers from strings (e.g., 19 to 19.0), or estimate quantities (e.g., how tall are lions?). such a diverse range of abilities are usually all referred to collectively as numeracy, which gives rise to confusion. we limit this abuse of terminology and provide a neat taxonomy for arranging the different tasks proposed under numeracy. ## 3 methods analogous to our taxonomy of subtasks in the previous section, here we attempt to arrange the wide variety of alternative number representations proposed in recent literature. we limit our analysis to methods of encoding (@xmath0. we do not discuss, for example, methods that use symbolic reasoning @xcite or modify activation functions to enhance numeracy @xcite . a typical example of the base architecture could be bert @xcite , the workhorse of modern nlp. we assume that there exists an independent parallel process of mapping words into embeddings, such as subword tokenization followed by lookup embeddings in bert. ## results having organized the landscape of numeracy tasks and methods, we now present come key results for each numeracy task in nlp from previously published experiments over a subset of the described number representations: abstract probes word embeddings vastly outperform random embedding baselines on abstract probes such as numeration, magnitude comparison, and sorting @xcite . dice, value and log value embeddings excel at these probes, which makes intuitive sense given that they explicitly encode the numbers' magnitude -although value embeddings do not easily extrapolate to larger numbers, possibly due to instability in training. the best number encoders with respect to these probes were found to be digitcnns, and character-tokenized models, e.g., elmo, in general outperform subword ones, e.g., bert @xcite . arithmetic gpt-3 (brown et al., 2020) performs extremely well at zero shot simple arithmetic, as long as the number of digits in the operands are low. the tokenization scheme could be the cause for limited extrapolation, since language models get better at arithmetic when numbers are tokenized at the digit/character level @xcite . for arithmetic word problems, state of the art solvers rely on predicting an equation, which is then filled in with specific numeric values from the question @xcite , altogether bypassing the need for encoding numbers into embeddings. @xcite show that bert pretrained over datasets where numbers are in scientific notation (num-bert) converges to the same loss as bert on masked language modelling objective, and scores nearly the same on glue language understanding benchmarks. for (causal) numeric language modelling, @xcite show that gaussian mixture models are the best decoders. for (masked) numeric language modelling, @xcite show that modelling the mantissa in scientific notation may be an overkill, since exponent embeddings alone outperform digitrnn-sci over financial news and scientific articles. @xcite train a regression probe to predict measurements of objects over the cls embeddings of bert/numbert. given a template-lexicalized sentence such as "the dog is heavy," the model must predict the weight of a typical dog, against ground truth from the distribution over quantities dataset @xcite . they find that numbert is a better text encoder than bert for measurement estimation, the only difference between them being the notation used by the respective pretraining corpora. they also experiment with two number decoders: mcc (multi-class classification) and rgr (regression / log value embedding). mcc performs better when trying to predict distributions over quantities -perhaps due to the ground truth resembling the predicted gaussians -but not on verbphysics -where the ground truth is less noisy. lastly, even static word embeddings like glove have been shown to contain enough knowledge of measurement estimates to contrast two objects, e.g., classifying whether a car is bigger/heavier/fasster than a ball @xcite . ## recommendations based on the above results, we now synthesize key insights into a set of directed takeaways to guide practitioners' design of number representations: rule of thumb for string-based methods? scientific notation is superior to decimal notation @xcite since models can learn to attend mostly to the exponent embedding rather than the mantissa @xcite . character level tokenization outperforms subword level @xcite @xcite . pooled representations (digitrnn, digitcnn) lack a controlled study with unpooled ones (numbert, genbert) which makes it hard to proclaim a winner among the two. rule of thumb for real-based methods? log scale is preferred over linear scale @xcite @xcite , which makes intuitive sense but lacks as rigorous a study as has been undertaken in the cognitive science community @xcite . regarding discretization, @xcite show that binning (dense cross entropy loss) works better than continuous value prediction (mae loss) on datasets where ground truth distributions are available. lastly, modeling continuous predictions is notoriously hard for large ranges @xcite but @xcite offer a way of binning such distributions by picking a precision level. encoding vs decoding numbers? in our simplified discussions above, we avoid differentiating between methods for encoding and decoding numbers. value embedding, for instance, can be used to encode numbers (projecting scalars onto vector space) as well as to decode numbers (collapsing a vector into a scalar). on the other hand, manually-designed encoders like dice are not easily reversible into decoding methods. even with reversible methods, the encoders and decoders must usually be independently parameterized, unlike the input and output word embeddings which often share weights @xcite . prototype embeddings by @xcite are an exception, which share input/output embeddings for a fixed vocabulary of numbers. can we mix-and-match multiple methods? given the wide range of number representations, an obvious next step is to try an ensemble of embeddings. @xcite show that for encoding numbers, exponent embeddings added to digitrnn (scientific notation) embeddings barely outperforms the exponent embeddings alone. similar experiments with a mix of real and string methods are yet to be seen. which methods for which tasks? based on our taxonomy of tasks in ## vision for unified numeracy in nlp numeracy is a core system of human intelligence @xcite . teaching numeracy to students works best when taught holistically, while less effective teachers deal with areas of mathematics discretely @xcite . while the nlp community genuinely strives to improve language models' numeric skills, not all aspects of numeracy have been sufficiently targeted. it is evident from the sparsity in evaluation. the first step towards a holistic solution to numeracy requires a benchmark covering its different subtasks. aggregated leaderboards in nlp like glue @xcite and su-perglue @xcite have incentivized research on natural language understanding, with scores categorized into semantic, syntactic, logical, and background knowledge. an analogous leaderboard could be constructed to evaluate models on numeric reasoning tasks, categorized according to the skills evaluated, e.g., exact vs approximate granularity, or abstract vs grounded numeracy. numbergame @xcite ) is one such aggregation focusing on exact numeracy benchmarks, as evaluated by f1 and exact match scores in a reading comprehension setup. both numbergame and our own list of tasks (section 2.2) are preliminary attempts at teasing apart the different aspects of numeracy. we encourage researchers to extend and refine such taxonomies. a suite of numeracy tasks, matched with evaluations of their respective numerical skills, can enable testing model generalization from one skill to another. some progress has already been made in this transfer learning setup, e.g., genbert @xcite , finetuned on a synthetic dataset of arithmetic problems, is found to score higher on drop qa. similarly, dice @xcite , optimized for numeration, improves score on numeracy600k order-of-magnitude prediction task. going forward, we need several such studies, ideally for each pair of tasks to see whether some numeracy skills help models generalize to others. ## conclusion this paper summarizes and contextualizes recent work on numeracy in nlp. we propose the first taxonomy of tasks and methods concerning textcentric numeric reasoning. we highlight key takeaways from the several experiments in literature, along with caveats and scope for confirming some of the observed trends. we present a case for lack of a holistic solution to numeracy in nlp, and put forward a set of aspects to consider when working towards one. we draw the following two major conclusions from our study: (1) the default subword segmentation with lookup embeddings used to represent words is clearly suboptimal for numbers (2) there are several unanswered research questions on the level of specificity, coverage, and inductive bias needed to holistically solve numeracy.