Predicting Numerals in Text Using Nearest Neighbor Language Models (2023)

Abstract: Commonsense about quantitative properties is essential for a deep understanding of texts containing numerals. However, naive language models (LMs) treat numerals as string tokens; therefore, they lack an understanding of the magnitudes of numerals, resulting in a difficulty in acquiring the commonsense. In this study, we apply the k -nearest neighbor LM ( k NN-LM) to the masked numeral prediction (MNP) task, which measures the quantitative commonsense of LMs. k NN-LM extends pre-trained neural LMs with the k -nearest neighbor ( k NN) search.Since it can utilize patterns that appear in the datastore for prediction, we expect an improvement in numeral prediction accuracy, which is associated with a high rate of occurrence of out-of-vocabulary (OOV) words.Through experiments, we verified that the retrieval-based method is effective for fine-grained predictions of numerals from context, especially for the OOV numerals.We also compared two different context spans for context representations to improve the accuracy of k NN search by using only the words that are closely related to the masked numeral: the mask and its surrounding words, and the mask and its subsequent words.Our results reveal that using only the embeddings of mask tokens for numerals in k NN search is the most effective approach for realizing MNP tasks.

## introduction real-world objects and events have various quantitative properties, such as size, weight, length, and price. commonsense about these quantitative properties is essential for a deep understanding of texts containing numerals and for reasoning on a similar or better level than humans. figure 1 shows examples of a masked numeral prediction (mnp) task requiring quantitative commonsense. the first example requires deriving a numeral referring to height that is considered tall based on commonsense about the distribution of human heights. the second example requires deriving the value of a typical movie length using commonsense and subtracting it from 12 p.m. humans can easily choose numerals that approximately correspond to the ground-truth answers to these questions with considerable confidence. however, for models that lack such commonsense and computational skills, such inferences pose a challenge. in recent years, large-scale neural language models (lms), such as bert @xcite , roberta @xcite , @xcite , and gpt-4 (openai, 2023), have achieved comparable or better performance than humans on various natural language processing (nlp) tasks. however, previous studies have reported that these models still perform poorly on tasks that require quantitative commonsense @xcite , such as mnp @xcite , numerical error detection or correction @xcite , and numerical question answering @xcite . one of the main reasons why lms fail to perform well on tasks that require quantitative commonsense is that they do not learn the mapping between strings of numerals and their magnitudes accurately. naive lms treat numerals in text only as string tokens, that is, other words. while humans can associate the magnitude of a numeral with the string of the numeral, lms that treat numerals only as strings are unable to accurately make such associations for arbitrary numerals @xcite . this makes it difficult for naive lms to understand the magnitude of numerals, resulting in difficulty in acquiring quantitative commonsense. to address this problem, previous studies have attempted to employ methods such as using word embeddings of numerals that reflect the magnitudes of the numerals @xcite , adding texts of arithmetic formulas to the training data @xcite , training lms with a loss function that depends on the magnitudes of numerals @xcite , and tokenizing numerals in a text into single digits to allow lms to understand the concept of digits @xcite . however, these methods require fine-tuning or additional pretraining specific to the understanding of numerals. therefore, in this study, we aim to improve the performance of lms in a task that requires quantitative commonsense (specifically, the mnp task @xcite @xcite ) without such additional training that is specific to numerals by using the k-nearest neighbor lm (knn-lm) @xcite , which is an lm extended by a retrieval-based method. in addition, numerals have a higher rate of occurrence of out-of-vocabulary (oov) words than regular words @xcite , making it difficult even for recent large neural lms to accurately predict numerals in sentences from the context. in this study, based on the hypothesis that numerals that appear in similar contexts tend to be of the same type (e.g., date, amount of money, and number of people) and similar sizes, we expect that knn-lm model will improve the accuracy of the mnp task by reflecting numerals that appear in similar contexts in the prediction results. we also believe that an advantage of using knn search is not only the improvement in top-k accuracy but also the improvement in interpretability by providing contexts with similar use of the predicted numeral. in our experiments, we used the pre-trained bert of huggingface @xcite as the base lm for knn-lm. two types of context ranges were used to compute the representation of the context of the masked numeral: the numeral mask and its surrounding words (figure 3 for both ranges, nearest neighbors were searched based on context representation, which is computed as the average vector of the embedding vectors of all the words in the range. knn-lm outperformed the base lm in mnp on most of the datasets we used. in addition, it was confirmed that the range of only the mask token for a numeral was the most effective context range for k-nearest neighbor (knn) search in mnp. to summarize, our contributions are as follows: ## conclusion in this study, we applied knn-lm to the mnp task and quantitatively evaluated its prediction accuracy. the results show that the numerical absolute errors were reduced by utilizing knn search for numeral prediction compared to existing methods. in particular, the prediction accuracy greatly improved for numerals not included in the model vocabulary, which are difficult to predict with naive lms. we also experimented with two different context ranges and confirmed that the most effective method for knn search is the one using only the word embedding of the mask token for the masked numeral as a representation of the context.