Numeracy enhances the Literacy of Language Models (2021)

Abstract: Specialized number representations in NLP have shown improvements on numerical reasoning tasks like arithmetic word problems and masked number prediction. But humans also use numeracy to make better sense of world concepts, e.g., you can seat 5 people in your ‘room’ but not 500. Does a better grasp of numbers improve a model’s understanding of other concepts and words? This paper studies the effect of using six different number encoders on the task of masked word prediction (MWP), as a proxy for evaluating literacy. To support this investigation, we develop Wiki-Convert, a 900,000 sentence dataset annotated with numbers and units, to avoid conflating nominal and ordinal number occurrences. We find a significant improvement in MWP for sentences containing numbers, that exponent embeddings are the best number encoders, yielding over 2 points jump in prediction accuracy over a BERT baseline, and that these enhanced literacy skills also generalize to contexts without annotated numbers. We release all code at https://git.io/JuZXn .

## introduction numbers account for 6.15% of all unique tokens in english wikipedia @xcite , yet nlp systems have traditionally either removed numbers during preprocessing or replaced them with a single uninformative unk token. recent models such as bert retain them but learn individual token embeddings for hundreds of numbers. moreover, subword tokenization approaches end up segmenting numbers into possibly suboptimal splits, e.g., 4500 is seen as (4, 500) or (45, 00) depending on the specific tokenizer used. the human brain, in contrast, automatically maps numbers to their approximate magnitude on the number line @xcite . nlp systems that fail to account for the scalar values that numbers denote may correspondingly lack in com-prehension. recent work has empirically demonstrated the inefficacy of existing nlp methods in numeric reasoning tasks @xcite . alternative number representations have been proposed, such as projecting the number's magnitude into a vector space @xcite or switching to a scientific notation @xcite . bert exp the [mask] weighs 100 lbs. statue bomb the [mask] weighs 10000 lbs. statue car we observe that this line of work goes from literacy to numeracy, i.e., helping language models gain numerate skills such as simple arithmetic @xcite , measurement estimation @xcite , and masked number prediction @xcite . our work addresses the converse question: do alternative number representations enhance the ability of language models to understand/predict words? we investigate this question through experiments with several representative number encoders, proposed in prior work. we develop and release wiki-convert, a large, novel dataset of number-annotated sentences, which helps us disentangle the nominal occurrences of numbers. our experiments show the positive impact of numeracy on a language model's literacy, as illustrated in ## methods our hypothesis is that language models will benefit from specialized encoders which explicitly make use of the number's magnitude. in line with both cognitive science research @xcite as well as recent work on numeric representations within nlp, we propose that numbers and words be encoded differently by a language model. words can continue to be subword-tokenized and encoded via lookup embeddings, but number encoding should consider the magnitude. we consider three representative methods from prior work which make use of a number's magnitude to encode it in vector space, as well as three baselines (marked with *), each of which is depicted pictorially in figure 1 . 1. value embeddings @xcite project the scalar magnitude of the number to be encoded into a vector space of same dimensionality as the lookup word embeddings. we use a 1-hidden layer feed forward layer as the projection network, with a configurable number of hidden neurons. although the authors used a specific implementation based on decimal scientific notation, we generalize this method to an arbitrary number of bins. 4. default* is the usual way that bert @xcite encode numbers: subword tokenization (schuster and nakajima, 2012) followed by lookup embeddings. 5. none* removes all numbers from the sentence during preprocessing. this is analogous to the baseline implementation in berg-kirkpatrick and spokoyny (2020), except they mask the numbers instead of filtering them out. 6. num* method learns a single lookup embedding for all numbers, reflecting how traditional nlp replaced any number occurrence with a single token @xcite , such as unk or num. this method can be seen as exponent embeddings with a single bin, into which all numbers are collapsed. ## wiki-convert existing benchmarks for numeric language modelling have been extracted automatically using regular expressions @xcite @xcite , and hence have no mechanism to filter out nominal numbers, such as zip codes, phone num-sentence number unit u-559 had a displacement of [num] [unit] while submerged 871.0 tonne ... temperature ranges from . . . in january to @xcite [unit] in july 73.9 °f ## experiments we operationalize our research question by finetuning the same pretrained masked language model (bert-base-uncased) with each of the six encoding methods (section 2) on the task of masked word prediction. thus when we say numeracy, we refer to the ability of the three number-specific encoders to take into account a number's magnitude and not its surface form. and when we say literacy, we refer to the masked word prediction ability of a language model, assuming it to be a valid proxy for downstream performance on other literacy tasks. the methods encode annotated numbers into 768-dimensional vectors. words, as well as numbers which are not annotated, are encoded by the usual subword tokenization followed by 1 https:wikipedia.org/wiki/help:convert lookup embeddings. value and logvalue methods each have a single hidden layer with 200 neurons, and exponent embeddings have 200 bins. we manually tuned the hyperparameter n (for value, logvalue, and exp) by optimizing for validation nll loss over hundreds of runs, @xmath0, 50, 75, 100, 200, 400}. besides wiki-convert, we also train and test our methods on numeracy600k @xcite , a dataset with financial market comments. for both datasets, we train on 100k samples, test on 10k, and use another 10k held-out dev set for configuring hyperparameters. for every input sentence, we randomly mask 15% of its non-number tokens and use a negative log likelihood loss to optimize the classifier. we measure perplexity and hit@k, masking one (non-number) word at a time. implementation details we use huggingface transformers @xcite for pretrained models and pytorch lightning @xcite for finetuning. we only train the masked language modeling (mlm) classifier (initialized from scratch) and the number encoder's parameters, if any, while keeping the base transformer weights frozen. the mlm classifier has a dense layer (768 × 768 weights) and a decoder (768 × 30522 weights) to learn output embeddings for each vocabulary item, where 768 is the embedding size for bert-base-uncased. take less than twenty minutes to train a model for 10 epochs of 10k training samples (batch size 256 with accumulated gradients over 4 batches). we set batch size as 1024, the largest that we could fit onto a single gpu, since we find that large batch sizes consistently help all methods and baselines. we train all models for 10 epochs over a training set of 100k sentences, i.e., ∼ 1000 updates, since we find this regime to allow nearly all runs to converge. ## results and discussion value methods also manage to outperform the default baseline for numeracy600k but they score below this baseline for wiki-convert. however, the latter dataset was sourced from wikipedia, over which bert was pretrained using the default scheme, hence this makes for an unfair comparison. does numeracy lead to better literacy, even in contexts without numbers? we compare exponent embeddings (the best performer) against the default baseline on 1000 sampled sentences from the 2006 english dump of wikicorpus @xcite which do not have any annotated numbers. ## related work recent nlp work has addressed several aspects of numeracy (see @xcite for a recent survey). here we review relevant prior work only on number encoders, as opposed to decoders. @xcite train a language model to predict both masked words and numbers, where the masked word prediction is the same setup as ours, while masked number prediction is modeled as a regression-penalized task of approximately estimating the number. they experiment with several number decoders, such as digit-rnn and gaussian mixture models, yet always encode numbers using a digit-rnn, whereas we experiment with six different number encoders to evaluate their relative performance on predicting words. berg-kirkpatrick and spokoyny (2020), akin to us, employ different number encoders, but for the task of masked number prediction. given a sentence with two numbers, they mask one number and study the effects of different number encoders for representing the unmasked number, on the task of approximately estimating the masked number. @xcite train numeral embeddings along with word vectors in a skip-gram setup, using multiple number encoders. they show that the simultaneously learned word embeddings score competitively on intrinsic word similarity tasks. our work focuses on this literacy evaluation, re-vealing that some number encoders not only help language models perform at par with the default baseline, but also exceed it in terms of perplexity. lastly, @xcite pretrain bert-base with a modified training corpus where all numbers are converted to scientific notation. this variant of bert, called numbert, converges to a similar loss on masked language modeling and next sentence prediction objectives as bert-base. on language understanding tasks like nli, numbert is only a little worse than bert-base. they also employ a logvalue decoder (which they call rgr) to probe numbert embeddings on the task of measurement estimation. our work instead focuses on word prediction, but our results converge in showing that better numeric representations do not harm (instead improve) language modelling abilities. ## conclusion our work studies the effect of number encoders on the task of masked word prediction, as a proxy for the ability of understanding text. we show that specialized number encoders are helpful in improving the word prediction ability of a language model, evaluated by perplexity and hit@k scores. we demonstrate these gains not only over sentences with annotated numbers but also more generally on text without numbers. we find exponent embeddings to be the best number encoders for masked word prediction. we see our work as preliminary evidence that numeracy enhances the literacy of language models. to facilitate subsequent work, we develop and release wiki-convert: a novel resource for number-related nlp with the added advantage of not conflating nominal with ordinal numbers. future work: we observe that the best performing number encoder (exp) is not merely magnitudeaware (so are value and logvalue) but is also learnt by lookup embeddings over collapsed numbers ranges. so far we explicitly define these ranges as those on the log scale but we intend to explore data-driven methods of identifying ranges from raw text, e.g., 1939-45 as the range of years of ww2. this work was funded by the defense advanced research projects agency with award n660011924033. we would like to thank the anonymous acl 2021 and emnlp 2021 reviewers for helping us refine earlier versions of this paper.