Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models (2023)

Abstract: State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers (usually referred to as numeracy). Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pretraining from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step without requiring architectural changes or pretraining from scratch. Arithmetic-Based Pretraining combines contrastive learning to improve the number representation, and a novel extended pretraining objective called Inferable Number Prediction Task to improve numeracy. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy, i.e., reading comprehension in the DROP dataset, inference-on-tables in the InfoTabs dataset, and table-to-text generation in the WikiBio and SciGen datasets.

## introduction numbers are ubiquitous in natural language. therefore, understanding and working with numbers (usually referred to as numeracy) is a critical capability for pretrained language models such as bart @xcite or t5 @xcite , cornerstones of modern nlp, in order to utilize quantitative information for various nlp tasks. recent works question whether these models meet this requirement out-of-the-box @xcite : common pretraining objectives such as the denoising autoencoder of bart @xcite , the masked language modeling objective of bert @xcite , or the span-corruption objective of t5 @xcite , are designed for understanding structure and semantic meaning of language and not to learn working with numbers. furthermore, commonly used subword-based tokenisation algorithms such as byte pair encoding @xcite or wordpiece @xcite are designed to handle patterns that are frequently observed during training, which is disadvantageous for numbers. for instance, 0.72 and 0.73 are two similar numbers. they should be processed similarly, but according to their frequency in the pretraining data they might be tokenised very differently, e.g., @xcite and [0, ., 7, 3], which will have an impact on their representation in embedding space. to address these shortcomings, various approaches have been proposed recently. however, most of them introduce additional components or rely on predefined features that limit their application, e.g., they are only applicable in a specific task like reading comprehension @xcite or require architectural changes @xcite . in this paper, we propose a new extended pretraining approach called arithmetic-based pretraining that targets both shortcomings for pretrained language models in one extended pretraining step without introducing new components or requiring pretraining from scratch. it consists of: our experiments show that arithmetic-based pretraining has a positive impact on bart @xcite , t5 @xcite and @xcite in various tasks. it improves the accuracy in case of reading comprehension and inference-on-tables, and the factual correctness in case of table-to-text generation. approaches for improving numeracy. numeracy requires to understand and work with numbers, i.e., to do artihmetic operations, in order to generate the expected result. to improve this capability, recent approaches propose pretraining from scratch or architectural changes to tailor pretrained language models towards specific tasks. tapas @xcite targets question answering with tabular data. it is pretrained from scratch and extends bert @xcite by introducing additional embeddings for capturing tabular structure. genbert @xcite reuses a pretrained bert model and adds a decoder on top. it is then further trained using math word problems and arithmetic operations for (1) incorporating the character-level tokenisation for numbers, and (2) to improve the numerical reasoning skills. it achieves state-of-the-art results in the drop @xcite and squad @xcite datasets. andor et al. (2019) also reuses the pretrained bert model and targets reading comprehension. they add a new layer on top that predicts and executes arithmetic operations. suadaa et al. (2021) target table-to-text generation and propose a framework that uses the template-guided text generation from @xcite to inject pre-executed numerical operations into the pretrained gpt-2 @xcite and t5 @xcite models. in their experiments, all of these works show that much of their performance improvements are due to specific design decisions or multi-level pretraining setups which result in new or task-specific models. with arithmetic-based pretraining, we propose an approach that improves a model's numeracy with just one extended pretraining step and without changing its architecture. ## arithmetic-based pretraining in this section, we propose arithmetic-based pretraining. it combines different tokenisation algorithms, i.e., character-level and subword-based, with contrastive learning to improve the representation of numbers in pretrained language models (section 3.1), while training on the inferable number prediction task (section 3.2) to improve the capability of working with numbers. section 3.3 describes the joint loss function. ## experimental setup we implement our approach using python 3.10, py-torch @xcite and huggingface @xcite . as pretrained language models, we use the large variant of bart @xcite and the base variant of t5 @xcite and flan-t5 (chung et al., 2022) as provided by the huggingface platform (see appendix a for details on hyperparameters) foot_4 . all models are pretrained transformer-based encoder-decoder models, but different in size. bart-large consists of a total of 24 layers and 406m parameters. t5-base and flan-t5-base consist of 12 layers and 220m parameters. flan-t5 is based on t5, but trained on more tasks, e.g., arithmetic reasoning, and chain-of-thought data (instructions). it significantly improves the results of the original model in many tasks @xcite . we conduct all experiments on a tesla v100-sxm3 gpu with 32 gb memory. for experiments using table-to-text datasets, we represent tables as linearized sequence. we report the results of the best single runs. ## evaluation in this section, we evaluate the impact of arithmetic-based pretraining on downstream applications with bart @xcite , t5 @xcite and flan-t5 @xcite using in-domain data (section 5.2), and out-of-domain data (section 5.3). for arithmetic-based pretraining, we use the preprocessed subsets of the original datasets as described in section 4.2. ## ablation study in this section, we investigate the impact of arithmetic-based pretraining on the numeracy of a pretrained language model. due to the shortcomings of automatic metrics in table -to-text generation (see section 5.1) and because we want to be able to compare and discuss the impact of each component across datasets, we use the inferable number prediction task for this and evaluate the number of correctly predicted context-related masked numbers (please see appendix e for ablation experiments in downstream tasks) 10 . we use the preprocessed subsets of the original datasets for the inferable number prediction task (see section 4.2). for evaluation, we use exact match (em score) and f1 score (see section 5.1). in comparison with bart, dt + inp shows that our masking procedure improves the results across all tasks. this is most significant in case of infotabs (up to 10.77 points in em score). in case of drop, it raises the f1 score from 7.20 to 55.51 points, meaning that there is a significantly larger overlap between predicted numbers and target numbers. using character-level instead of default tokenisation for numbers (clt + inp) again improves the results across all datasets, indicating improved capabilities for arithmetic operations. compared to dt + inp, it improves the em score by 35.89 points in case of infotabs, and by 23.07 points in case of drop. ours further improves the results across all datasets. this is most significant in case of the table-to-text datasets, where it improves the em score by 34.25 points in case of wikibio @xcite , and 12.42 points in case of scigen @xcite . since we create the pairs for the contrastive loss batchwise, i.e., we consider all numbers in a batch independently from the samples (see section 3.1), an advantageous number-to-word ratio favors a good positive-negative pair ratio for the contrastive loss, as in the case of scigen which has the highest number to word ratio in input tables (33.88%, see also section 4.1). this is counteracted by wikibio which has a lower number-to-word ratio (16.32%). however, with 728, 321 samples, wikibio is the largest dataset. we therefore assume that more data compensates for a poor number-to-word ratio. ours -inp deteriorates the em score by 5.21 points in case of drop, 3.19 points in case of scigen, and 0.79 points in case of infotabs. this shows the contribution of our masking procedure to the effectiveness of arithmetic-based pretraining. ## conclusions in this paper, we propose arithmetic-based pretraining, an approach for jointly addressing the shortcomings of pretrained language models in understanding and working with numbers (usually referred to as numeracy). in contrast to existing approaches, arithmetic-based pretraining does not require architectural changes or pretraining from scratch. it uses contrastive learning to improve number representation and a novel extended pretraining objective, the inferable number prediction task, to improve numeracy in just one extended pretraining step. our experiments show performance improvements due to better numeracy in three different state-of-the-art pretrained language models, bart, t5, and flan-t5, across various tasks and domains, including reading comprehension (drop), inference-on-tables (infotabs), and table-to-text generation (scigen and wikibio). we show that the effectiveness of our approach is not limited to in-domain pretraining, but rather depends on the extent to which the dataset used in the inferable number prediction task requires understanding numbers. for example, pretraining on the scigen dataset improves the results achieved on drop. our ablation studies show that contrastive learning and the inferable number prediction task are key to improving the numeracy of the examined models.