Predicting Numerals in Natural Language Text Using a Language Model Considering the Quantitative Aspects of Numerals (2021)

Abstract: Numerical common sense (NCS) is necessary to fully understand natural language text that includes numerals. NCS is knowledge about the numerical features of objects in text, such as size, weight, or color. Existing neural language models treat numerals in a text as string tokens in the same way as other words. Therefore, they cannot reflect the quantitative aspects of numerals in the training process, making it difficult to learn NCS. In this paper, we measure the NCS acquired by existing neural language models using a masked numeral prediction task as an evaluation task. In this task, we use two evaluation metrics to evaluate the language models in terms of the symbolic and quantitative aspects of the numerals, respectively. We also propose methods to reflect not only the symbolic aspect but also the quantitative aspect of numerals in the training of language models, using a loss function that depends on the magnitudes of the numerals and a regression model for the masked numeral prediction task. Finally, we quantitatively evaluate our proposed approaches on four datasets with different properties using the two metrics. Compared with methods that use existing language models, the proposed methods reduce numerical absolute errors, although exact match accuracy was reduced. This result confirms that the proposed methods, which use the magnitudes of the numerals for model training, are an effective way for models to capture NCS.

## introduction numerical common sense (ncs) is knowledge about the numerical features of objects in the real world, such as size, weight, or color, each of which has its own range and probability distribution @xcite ). consider the following example sentence. in recent years, bert, gpt-3, and other neural language models have achieved a level of performance on par with or better than human performance in many natural language processing tasks @xcite @xcite . moreover, several studies have recently been conducted to investigate whether pre-trained neural language models have commonsense knowledge, and these studies often conclude that the language models have been successful in acquiring some commonsense knowledge @xcite @xcite @xcite . however, it has also been reported that current neural language models still perform poorly in natural language processing tasks that require ncs and a deep understanding of numerals, such as numerical reasoning, numerical question answer-ing, and numerical error detection/correction @xcite . numerals appear frequently in various forms, such as dates, numbers of people, percentages, and so on, regardless of the domain of passages. hence, the acquisition of numerical common sense by neural language models and the analysis of the acquired numerical common sense are essential research topics to support systems for reasoning on text containing numerals and smooth conversation with humans at a high level. one of the major problems that make it difficult for language models to understand the meaning of numerals and to acquire ncs is that naive language models treat numerals in text as string tokens, just like any other word @xcite . this makes it difficult to obtain a mapping between the string tokens and the magnitudes of the numerals, which is needed to capture ncs. in this study, we use the masked numeral prediction task @xcite to evaluate and verify the ncs acquired by neural language models. the task requires models to predict masked numerals in an input passage from their context. we use two types of evaluation metrics: hit@k accuracy @xcite and mdae and mdape @xcite for this task. hit@k accuracy calculates the percentage of predictions in which the groundtruth numeral is within the top k predicted numerals, and we can say that they evaluate language models in terms of the symbolic aspect of numerals. mdae and mdape are calculated from the difference between the groundtruth numerals and the predicted numerals, and we can say that they evaluate language models in terms of the quantitative aspect of numerals. to perform this task, we investigate the following two approaches to reflect the magnitudes of the numerals for fine-tuning language models on the masked numeral prediction task (figure 1 ). 1. a masked word prediction model with a new loss function loss num that is based on the differences between the groundtruth numerals and predicted numerals; 2. a masked word prediction model, called the reg model, structured with an additional output layer to predict a numeral from an input passage containing a masked numeral. we use the bert-based masked word prediction model as a baseline and conducted experiments on four datasets, which differ from each other in the length and domain of the passages as well as the distribution and characteristics of the numerals appearing in the datasets. we compare the results and investigate the relationship between the characteristics of the numerals in the datasets and the performance of each method. although fine-tuning with loss num causes a decrease in the exact match accuracy, we found that it reduces numerical absolute errors, which indicates the effectiveness of loss num . the results of the reg model show the difficulty of predicting numerals in natural language text with the regression model. however, there were some numerals that the reg model predicted better than the existing language model, indicating that the reg model and existing language models are good at predicting numerals with different characteristics. in our experiments, to eliminate the negative effects of the sub-word approach, we do not split the numerals into sub-words. the sub-word approach splits words into shorter tokens called sub-words. it has the advantage that even low-frequency words can be represented by a combination of sub-words that appear in a text more frequently. however, unlike the case of words, sub-words derived from numerals often have little relationship to the meaning of the original numerals, which can make it difficult to understand the meaning of numerals @xcite . all other words are separated into sub-words in our experiments. to summarize, in this work, we tackle the problem of dealing with numerals in naive language models on the masked numeral prediction task. our contributions are as follows: ## conclusion in this paper, we used the exact match accuracy and numerical absolute errors metrics to evaluate the masked numerical prediction task, focusing on the fact that numerals have two aspects: symbolic and quantitative. based on this fact, we proposed two methods to reflect the two aspects of numerals in the training of language models. although the proposed loss function, loss num , decreased the exact match accuracy slightly, it also reduced the numerical absolute errors on the masked numeral prediction task, indicating the effectiveness of loss num . furthermore, we analyzed the relationship between the properties of numerals in datasets and the performances of different prediction methods on four datasets with different properties. as a result, it was found that the types of numerals that are likely to be mistakenly predicted depend on which method is used.