Estimating Numbers without Regression (2023)

Abstract: Despite recent successes in language models, their ability to represent numbers is insufficient. Humans conceptualize numbers based on their magnitudes, effectively projecting them on a number line; whereas subword tokenization fails to explicitly capture magnitude by splitting numbers into arbitrary chunks. To alleviate this shortcoming, alternative approaches have been proposed that modify numbers at various stages of the language modeling pipeline. These methods change either the (1) notation in which numbers are written (eg scientific vs decimal), the (2) vocabulary used to represent numbers or the entire (3) architecture of the underlying language model, to directly regress to a desired number. Previous work suggests that architectural change helps achieve state-of-the-art on number estimation but we find an insightful ablation - changing the model”s vocabulary instead (eg introduce a new token for numbers in range 10-100) is a far better trade-off. In the context of masked number prediction, a carefully designed tokenization scheme is both the simplest to implement and sufficient, ie with similar performance to the state-of-the-art approach that requires making significant architectural changes. Finally, we report similar trends on the downstream task of numerical fact estimation (for Fermi Problems) and discuss reasons behind our findings.

## introduction the standard practice in the natural language processing (nlp) community is to process numbers in exactly the same manner as words. this counterintuitive treatment of numbers leads to their inaccurate representation and therefore, limited numerical understanding of large-scale language models (lms) @xcite . to illustrate, a number like @xmath0800, as the surface forms share no common tokens. many alternatives have been proposed to capture the scalar magnitude of numbers @xcite . all number decoders proposed to capture the magnitude of numbers fall into one of the following categories, corresponding to changes in 1) notation (e.g. scientific vs decimal) or 2) vocabulary (e.g. introducing new tokens that denote all numbers within a specified range) or 3) architectural changes (e.g. directly regressing to a number). figure 1 shows various alternative number representation methods ordered by increasing levels of intervention on a typical nlp pipeline, color coded consistently across the paper for legibility. we find that applying the vocabulary-level changes leads to near state-of-the-art performance requiring no additional pretraining or architectural changes. this is a surprising yet useful ablation result, which can substantially speed up adoption of numeracy into any given language model. any arbitrary lm can be made numerate by simply tokenizing numbers on the number line. we further evaluate the number representation schemes on their ability to generalize to a downstream task of numerical fact estimation in the context of solving fermi problems @xcite . we find similar trends, demonstrating the utility of the simple yet effective tokenization scheme in the decoding setting. finally, we discuss how these results may be explained by the observed distribution of mantissas in natural language. ## kinds of number representations our work focuses not on nlp models performing arithmetic, instead on their comprehensive understanding of approximate numbers, with the setting of masked number prediction (mnp) in natural language. this section introduces existing classes of number decoders and their respective trade-offs. subword. the default way that language models decode numbers is the same way as words, one subword at a time, e.g. , the number 600 could be decoded as two individual tokens 6 and ##00. ## experimental setup task: we evaluate the above decoders on the task of masked number prediction (mnp): given a sentence with a mask (e.g. "tigers weigh [mask] lbs."), the model must predict a number as close as possible to the ground truth (e.g. 600). datasets: we follow berg-kirkpatrick and spokoyny (2020) to finetune and evaluate our models on three datasets foot_0 -financial news articles (finnews), its subset containing mostly pricebased numbers (finnews-$), and scientific articles (scidocs) @xcite ; all numbers in these datasets lie between 1-10 16 . metrics: we evaluate using two metrics -a) exponent accuracy (e-acc) that checks whether the predicted answer is of the same order of magnitude as the ground truth and b) log mean absolute error (logmae). confidence intervals for exponent accuracy, a classification metric, are reported as the wilson score interval @xcite : @xmath1, where a is the accuracy, z is the constant (2.58 for 99% ci), and n is the number of observations in the respective test set. baselines: our primary baseline is the standard approach of subword tokenization. we require each number prediction to be 8 tokens long, with appropriate padding, to be able to fairly represent all numbers in our range. additionally, we evaluate on three trivial baselines that make a constant prediction corresponding to the mean, median, and mode of all numbers in the training set. ## results we bold-face the best and underline the next best logmae scores in each column (dataset), and we highlight exponent accuracies that are within 99% confidence of the sota e-acc. na denotes subword models which were unable to emit valid numbers for at least 50% of the examples. intrinsic results ( we evaluate our models (trained with different number decoders on one of the three datasets) in a zero-shot setting on such annotated facts provided as part of both the real and synthetic datasets part of the fermi problem dataset. the task setup is of masked number prediction as before, e.g., "the size of a tennis ball is [mask] cubic centimeters." we find similar trends as before i.e. change of notation is insufficient while vocabulary-change approaches are equal or better than architectural changes -highlighting that most of the gains could be retained by simply tokenizing in number space. ## related work we restrict our analysis to the task of approximately decoding numbers in mnp setting, which requires different methods and metrics from the tasks that instead evaluate their exact arithmetic skills @xcite . the method we highlight in this paper i.e. change of vocabulary to tokenize numbers on a log-scaled number line, has been previously used in different settings. others have shown the benefits of using such exponent embeddings as number encoders for language models, whether it be for the task of masked number prediction @xcite or masked word prediction @xcite . our work extends these results with further evidence of the representational power gained by simply tokenizing numbers on the number line. our simple intervention to improve approximate numeracy in lms is also related to other work @xcite which aims to improve exact numeracy of lms without any architecture change. ## conclusion subword tokenization, the standard approach to representing numbers leads to inaccurate numerical understanding. in this work, we analyze number representation approaches that make notational (e.g. scientific vs. decimal), vocabulary (i.e. tokenizing on the number line), and architectural changes (i.e. regressing to the number). we find that tokenization on the number line achieves near or better than state-of-the-art results while requiring minimal intervention to the language model. this is a negative insight against recent results in the community which suggest that language models must be architecturally modified to gain numeracy. it will allow language models to conveniently improve their numeracy, including cases where users may not have access to the model's architecture and are only provided a typical finetuning regime with small changes to the tokenizer's vocabulary. finally, we find similar trends in the challenging setting of numerical fact estimation for solving fermi problems -indicating that vocabulary-change is sufficient to represent approximate numbers effectively with minimal effort.