Laying Anchors: Semantically Priming Numerals in Language Modeling (2024)

Abstract: Off-the-shelf pre-trained language models have become the de facto standard in NLP pipelines for a multitude of downstream tasks. However, the inability of these models to properly encode numerals limits their performance on tasks requiring numeric comprehension. We introduce strategies to semantically prime numerals in any corpus by generating anchors governed by the distribution of numerals in said corpus, thereby enabling mathematically grounded representations of these numeral tokens. We establish the superiority of our proposed techniques through evaluation on a range of numeracy tasks for both in-domain (seen) and out-domain (unseen) numerals. Further, we expand our empirical evaluations to numerals ranging from 1 to 10 billion, a significantly broader range compared to previous studies of the same nature, and we demonstrate significant improvements in the mathematical grounding of our learned embeddings.

## introduction numeracy, at its core, is the comprehension of numbers, akin to the comprehension of words in literacy. the magnitude of a number is especially tied to its meaning @xcite ; as such, in developmental psychology, children able to distinguish numbers based on their magnitudes are said to possess the concept of numbers @xcite . in the context of nlp, because numbers often grant objectivity to language @xcite , language models that can comprehend numeric magnitude and scales allow for better inference @xcite , information extraction @xcite , and datato-text generation @xcite . numeric comprehension can indeed be induced in language models through explicit supervision @xcite is employed for this plot as it contains a healthy mixture of both in-domain and out-domain numerals from our dataset. @xcite ; however, the inherent numeric capabilities of off-the-shelf language models induced from unsupervised training have been shown to be inadequate @xcite and often fail to extrapolate to numerals not seen in the training set @xcite -referred to as out-of-domain (ood) numerals. approaches for numeracy induction to-date either involve strategies that learn representations for numerals separately from regular tokens @xcite or do so by training models on numeracy-specific tasks @xcite . in contrast, we prime (see §2) the numerals in the training corpus by laying anchors such that numeracy is induced via the unsupervised pre-training of the model itself without separately training numerical embeddings. as illustrated in figure 1 , our model shows substantial improvements in numeral representations for both numerals present in the training corpus (in-domain) as well as numerals further, the evaluation of numeracy in language models through their ability to predict numbers in a manner similar to textual tokens @xcite omits the influence of rote-memorization @xcite . in order to decouple the rote-memorization of numerals with respect to the linguistic context in which they appear, our study follows the evaluation protocols of @xcite wherein the quality of learned representations are assessed through a set of numeric comprehension tasks. our contributions can be summarized as: ## priming numerals with anchors how does one prime numerals? the priming effect is a temporary change in the perception of a target stimulus that frequently occurs in conjunction with a priming stimulus @xcite . similarly, semantic priming establishes the strength of relations among items belonging to the same or different categories @xcite . now, what does this mean in the context of numerals in a training corpus? consider numerals 0 and 10 that are both equidistant to a supposed anchor numeral 5. if a language model has never seen the numerals 0 and 10 in its training corpus, the anchor numeral 5-that the model has seen during its training-can now be used to ground the magnitudes of these unseen numerals such that the model can now reason its magnitude. essentially, we intend to ground the magnitudes of numerals that the model rarely sees or has never seen based on the magnitudes of the numerals that it has frequently seen, known as the anchors. how are the anchors determined? first, we extract all numerals x from a training corpus c through which we intend to induce our anchors. the intuition that anchors should be numerals widely represented (frequent) in the corpus leads to the choice of gaussian mixture models (gmms) in contrast to clustering methods such as k-means that lack probabilistic cluster assignment. the set of anchors is induced from the means µ k of each gaussian @xmath0. here, n represents the probability density function and π k ,σ k represent the mixing coefficient and standard deviation for the k-th gaussian component. the initialization and the choice of k is described in §a.1. devising the four categories of anchors: theories for mental representation of cardinality further divides our implementation of these anchors into two halves: a continuous linear representation @xcite and a compressive representation where the difference between numerals n and n + 1 decreases as n increases @xcite . as such, for linear representation of the number line, we associate numerals with their closest anchor without alteration -giving us our first model anchors. similarly, for compressive representation, a given numeral n is anchored to m from a set of log-normalized anchors such that ln (n) ≈ m -our second model ln anchors. in both these methods, the priming is implemented through a specialized @xmath1. further, this priming effect is known to be symmetric with respect to the priming direction and additive to the effect of repetition priming @xcite . this notion leads to our second category of models, viz. directional anchors represented with bi-directional arrows ⇄. thus, in addition to attaching anchors to numerals in the corpus, we signify where the anchor lies in the number line with respect to the target numeral using specialized @xmath2. training samples augmented with @xmath3. ## experimentation and results as delineated in the previous section, we evaluate four configurations of our model pre-trained on the anchor-augmented wikitext-103 corpus @xcite : anchors, ln anchors, anchors (⇄), and ln anchors (⇄). the details of the datasets, pre-training and fine-tuning configurations, and embedding retrieval are described in §a.2. ## results the results of above four tasks are illustrated in as the gmm-based anchors favor numerals frequent in the corpus, the anchors become sparse at higher numeral ranges -[10k, 10 10 ]. thus, for this range specifically, we see that the model that strictly relies on directional anchors outperforms the log-compressive anchors on magnitude estimation tasks. essentially, when the anchors are further from each other, knowing which direction they reside in with respect to the target numeral aids the model in reasoning about that numeral. for easier comparisons among models, the measure employed for the decoding and addition tasks is log-rmse; as the error is log-compressed, seemingly small changes to the log-rmse score translates to visible changes in numerical estimation through their embeddings, as depicted in figure 1 . ## conclusions in this paper, we have presented a simple plugand-play bert variant with enhanced numerical capabilites. through our rigorous interpolation (indomain) and extrapolation (out-of-domain) analyses, we showcase the superiority of our model in numeric comprehension while outlining the impact of logarithmic compression on magnitude estimation and the impact of directionality on relative ordering capabilities. further, as a consequence of introducing anchors, we find the learning of niche pockets of similar embeddings for numerals closer in their magnitudes ( §a.4).