Calc- CMU at S em E val-2024 Task 7: Pre-Calc - Learning to Use the Calculator Improves Numeracy in Language Models (2024)

Abstract: Quantitative and numerical comprehension in language is an important task in many fields like education and finance, but still remains a challenging task for language models. While tool and calculator usage has shown to be helpful to improve mathematical reasoning in large pretrained decoder-only language models, this remains unexplored for smaller language models with encoders. In this paper, we propose Pre-Calc, a simple pre-finetuning objective of learning to use the calculator for both encoder-only and encoder-decoder architectures, formulated as a discriminative and generative task respectively. We pre-train BERT and RoBERTa for discriminative calculator use and Flan-T5 for generative calculator use on the MAWPS, SVAMP, and AsDiv-A datasets, which improves performance on downstream tasks that require numerical understanding. Our code and data are available at https://github.com/calc-cmu/pre-calc.

## introduction the advancement of language modeling in natural language processing has significantly impacted various computational tasks. however, the intricacy of numerical and quantitative comprehension in text remains a challenging frontier. numerals, unlike words, possess unique characteristics that necessitate specialized handling by language models either in tokenization or processing. this necessity becomes particularly evident in tasks involving quantitative reasoning, where the ability to interpret and manipulate numerical information is crucial. numeracy involves majorly 2 properties. the first is semantic reasoning which focuses more on the understanding of relations in text and the second is computational abilility which focuses on performing explicit mathematical operations. hence, the aim is to develop systems that can perform explicit mathematical operations while retaining or improving its quantitative reasoning. we present pre-calc, a pre-finetuning objective of learning to use the calculator, to improve numerical abilities in language models. we propose pre-calc objectives for both the encoder-only and encoder-decoder classes of language models, and use a combination of the mawps @xcite , svamp @xcite , @xcite datasets to pre-finetune the models. our encoder-only objective, used to pre-finetune bert and roberta models, offers quick and efficient processing suitable for tasks where speed is paramount. the pre-calc versions of the models show competent performance on all quantitative downstream tasks from numeval @xcite and substantial improvements on 4 out of 6 sub-tasks with an improvement greater than 10 points for redditnli and awpnli specifically. similarly, our encoder-decoder approach, used to pre-finetune flan-t5, demonstrates an improved ability to perform explicit computations in computation-intensive tasks like awpnli. although there is a noted trade-off, with a slight decrease in performance on text-focused and semantic tasks, the objective showcases strengths in processing mathematically intensive language. our study underscores the potential of tailored language models to significantly enhance numeracy in nlp, providing an avenue for more efficient and effective processing of numerical data in language. ## pre-calc methodology we posit that learning to use a calculator requires understanding of numbers and ways in which numbers can be combined. this is used to formulate the pre-calc objectives described below. ## analysis 6.1 dual-objective in encoder-only pre-calc we inspect the characteristics of the two objectives during pre-finetuning. fig. 3 shows the f1-score across epochs for the operand identification objective on the validation data. while this seems to fluctuate, it consistently stays above 90% (the ac-curacy for this task also consistently remains at about 99%), indicating that the operand identification task is not very challenging and that there is very little loss signal from this task beyond the first few epochs. regardless of having the second objective, the f-1 for this task is still maintained at a high number. in fig. 4 we see the accuracy plot on validation data for the operation classification objective across epochs. here we see that the accuracy consistently increases but still remains under 75% which tells us that this objective is a lot more challenging, which is also explained by the fact that it has to be inferred from text. together, the two aid different abilities -picking numbers out with operand identification and combining numbers with operation classification -which are both important for any downstream quantitative task ## conclusion and future work in this work, we improve the numeracy in language models on the qnli and qqa tasks which involve textual and computational quantitative reasoning. we do so by proposing calculator usage as a prefinetuning task in a discriminative and generative fashion for encoder-only and encoder-decoder models respectively. this improves encoder-only models across various downstream tasks and improves encoder-decoder models on tasks that require explicit computation. future work can address the balance between textual understanding and numerical reasoning, by refining regularization strategies to maintain the language model's core strengths while enhancing its computational abilities. tool-use in encoderonly models could also be extended to more complex tools similar to decoder-only models.