Improving Numeracy by Input Reframing and Quantitative Pre-Finetuning Task (2023)

Abstract: Numbers have unique characteristics to words. Teaching models to understand numbers in text is an open-ended research question. Instead of discussing the required calculation skills, this paper focuses on a more fundamental topic: understanding numerals. We point out that innumeracy—the inability to handle basic numeral concepts—exists in most pretrained language models (LMs), and we propose a method to solve this issue by exploring the notation of numbers. Further, we discuss whether changing notation and pre-finetuning along with the comparing-number task can improve performance in three benchmark datasets containing quantitative-related tasks. The results of this study indicate that input reframing and the proposed pre-finetuning task is useful for RoBERTa.

## introduction numerals are an indispensable part of narratives and provide much fine-grained information. 1 how models learn the number system has intrigued many researchers @xcite @xcite @xcite . researchers have long discussed some numeracy-related properties of pretrained language models (lms). in this study, we propose a new concept -innumeracy. the problem of innumeracy becomes most evident when models are faced with numerals that do not appear in training data, e.g., when the range of numerals in training data is different from that in the test data. moreover, lms often face difficulties understanding numbers even though the numbers are present in the training data. one possible cause of this problem is that numerals can have various notations, some of which are difficult to understand from their subwords. another possible cause is model notation tokenized example bert org. input reframing refers to changing the notations of numbers, which can be one of the crucial clues for understanding numerals @xcite . in addition to the original notation, we consider the digit-based and scientific notations. to evaluate the influence of the input reframing and the quantitative pre-finetuning task, we constructed the quantitative 101 dataset, which is a combination of three benchmark datasets: numeracy-600k @xcite , equate @xcite , and numglue task 3 @xcite . the tasks in quantitative 101 include quantitative prediction (qp), qnli, and quantitative question answering (qqa). in the future, quantitative 101 can be used as a new collection by researchers studying the quantitative skills of lms. foot_1 2 related work numeracy, one of the recent hot topics in nlp, incorporates many skills such as calculation, algebra, and geometry. some previous studies @xcite have discussed the prediction of the masked number tasks, while others @xcite @xcite have explored numeracy from the perspective of embedding properties. the math word problem @xcite is a high-level task requiring several numeracy skills. the textual representation of numerals, such as digit-based or scientific notations-based, is one of the possible directions for improving numeracy. @xcite suggested to use a digitbased encoder to encode numerals. @xcite used scientific notation to represent numerals and explored scale understanding tasks. in this paper, we explore the role of these notations of numbers in quantitative skill tasks. a recent trend is to design pretraining tasks to enhance the capability of models to understand natural language. @xcite proposed two pretraining tasks: masked language model (mlm) and next sentence prediction (nsp), and broadened the horizons of the transformer-based natural language processing research direction. eral benchmark datasets, especially those requiring multi-hop reasoning and multi-document understanding skills. to the best of our knowledge, this is one of the earliest works proposing a tailor-made pre-finetuning task to understanding numerals. our experimental results also support the usefulness of the proposed task, specifically in the qnli task. ## datasets and tasks this section introduces two datasets: the comparing numbers dataset (cnd) and quantitative 101, with the corresponding quantitative tasks, including comnum, qp, qnli, and qqa. ## conclusion this study deals with the innumeracy of lms and shows that the notation of numbers matters, especially for roberta. we also propose a novel prefinetuning task for improving the quantitative skills, and find that the performance in the qnli task can be improved after pre-finetuning. we hope our results in quantitative 101 lead to a more in-depth discussion on the ability of lms to understand numerals.