Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model (2021)

Abstract: The transformer-based pre-trained language models have been tremendously successful in most of the conventional NLP tasks. But they often struggle in those tasks where numerical understanding is required. Some possible reasons can be the tokenizers and pre-training objectives which are not specifically designed to learn and preserve numeracy. Here we investigate the ability of text-to-text transfer learning model (T5), which has outperformed its predecessors in the conventional NLP tasks, to learn numeracy. We consider four numeracy tasks: numeration, magnitude order prediction, finding minimum and maximum in a series, and sorting. We find that, although T5 models perform reasonably well in the interpolation setting, they struggle considerably in the extrapolation setting across all four tasks.

## introduction recent advances in transfer learning in nlp have led to the emergence of pre-trained models which show a much stronger contextual representation of words than earlier static word embeddings. they have all performed extremely well in conventional nlp tasks. yet, they fail to capture a better understanding of numbers. numbers are integral part of natural language texts which can change the meaning of a sentence. so there is a need for nlp models which can identify numbers represented in any surface forms like words, floats or strings (numeration), understand its values in various context (magnitude order prediction), compare their values with others (list-minmax) or able to rearrange a series of numbers based on its values (sorting). the transfer-learned models are pre-trained on huge amount of natural language texts with specially designed tasks and tokenizers to create stronger word-embeddings. this causes the numbers embedded in the texts to lose their meaning and inherent rules of numeracy guiding them @xcite . this is possibly the reason they perform worse in numerical reasoning tasks on numbers absent in training data @xcite . in this paper, we test this numeracy learning ability of a text-to-text transfer learning generative model, t5 @xcite which has outperformed its predecessors in conventional nlp tasks. the text-to-text format of input and output helps the model to generalize all the nlp tasks as a unified model. we use four numeracy tests both in interpolation (training and testing on same range of data) and extrapolation settings (training on lower and testing on higher range of data) and study how much numeracy skill it can acquire. figure 1 shows some examples of each of the numeracy tests. our contributions in this paper are: (1) extensive study on three versions of t5 models (small, base, large) on four numeracy tests in interpolation and extrapolation settings. (2) reporting interesting observations in the behavior of each model version across multiple experimental settings through detailed manual error analysis. the synthetically generated data and codes are publicly available foot_0 for future numeracy analysis in similar settings. ## numeracy tests we perform four essential numeracy tests to explore model's ability to understand numerical values. motivation: these four elementary tasks are simple and easy for the models, since they do not need to generate a completely new number in a different numerical range (like in mathematical tests : multiplication, division, exponentiation). here we evaluate whether the models learn the numeracy tasks or they simply learn bias from the number range seen in training data. ## results and error analysis in magnitude order prediction ( from the sorting results ( overall, none of the models were able to perform well on extrapolation samples showing the inherent rules of numeracy is difficult for these models to learn. but, it also shows, more variations in numbers (increasing the range) help them perform better in extrapolation setting. the smaller model's limited parameter-space affects its performance in all four tasks whereas larger models are able to pick up some numeracy skills through training. we show more predictions in figure 3 , 4, 5, 6. ## conclusion & future works we show that text-to-text models are able to learn numeracy quite well in an interpolation setting. our extensive experiments show that t5 models struggle to learn with numbers outside training data ranges. we believe that, to make further progress in transfer learning, models need to achieve such elementary numeracy skills and this gap between interpolation and extrapolation performance needs to be reduced. we are of the opinion that, adding more data would not bridge this gap since domain of numbers is open. however, special pre-training objectives for digits rather than whole numbers can be designed to teach the inherent numeracy to models. in future, we intend to explore these objectives centered around preserving numeracy rules in transfer-learned models to generalize between in-domain and out-of-domain numbers.