Calc- X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems (2023)

Abstract: Despite outstanding performance in many tasks, language models are notoriously inclined to make factual errors in tasks requiring arithmetic computation. We address this deficiency by creating Calc-X, a collection of datasets that demonstrates the appropriate use of a calculator in reasoning chains. Calc-X is suitable for teaching language models to offload computations to a symbolic system. We survey and unify several existing chain-of-thought datasets into a proposed format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new Calc-X collection to train open-source calculator-using models and show that these models approximately double the accuracy of generating correct results compared to vanilla language model baselines.

## introduction while the language models (lms) demonstrate outstanding efficiency in working with unstructured language data, they struggle with problems that require exact computations @xcite . on the other hand, symbolic systems, such as a calculator, can perform arithmetics without errors. thus, combining the strengths of both neural and symbolic systems can yield significant benefits in tackling tasks that require arithmetics @xcite . given a sufficient amount of supervised data, the interaction with symbolic systems can be learned. however, obtaining texts demonstrating the interaction in relevant situations and in a consistent structure is non-trivial. consequently, a substantial effort of most related work addresses the data scarcity problem through semi-supervised learning, heuristics, prompting or few-shot, and reinforcementbased approaches (section 2) with compromises in the quality and reproducibility. to support future research in developing opensource tool-assisted language models, we curate a calc-x collection of over 300,000 samples for mathematical reasoning. calc-x transforms several existing datasets into a unified format that can be used to train and evaluate lms for the correct use of a calculator. we survey existing chain-ofthought (cot) datasets for arithmetical reasoning (section 2) and pick a subset suitable for integration into a consistent collection. to enable efficient integration of lms with independent tools, we propose a unified format of a fully parseable html-like markup language (section 3). for each dataset, we describe the curation process of its calculatoraugmented (calc-x) version (section 4). finally, we show that training on a full mixture of calc-x datasets enables lms to use a symbolic system during inference and largely improves the accuracy on held-out math problems (section 5). we make all our building tools, datasets and models publicly available (appendix a.4). ## related work math datasets gsm8k @xcite contains grade-school math problems with humanwritten cot explanations and explicit annotation of formulas. @xcite , svamp @xcite , mawps @xcite and larger chinese ape210k @xcite contain problems of similar complexity, but the solutions are written as nested expressions. aqua-rat @xcite ) contains multiple-choice problems with selected answers and free-text rationales. mathqa @xcite is a subset of aqua-rat with additional annotation: nested expressions that lead to an answer, similar to those in ape210k or asdiv. math and amps @xcite consist of more challenging problems and contain cot solutions formatted in l a t e x. specifically, math is a set of high-school math competition problems, and amps is a large-scale pre-training dataset, partly scraped and partly synthetically generated. mathematics dataset @xcite is another generated dataset but containing only final answers. tool-using lms a main contribution of much of the previous work in building tool-using models addresses the problem of data scarcity. @xcite , webgpt @xcite , and lamda @xcite let crowd workers annotate tool calls to train models to use a web search, a calculator, and a machine translation system. pal @xcite applies prompt engineering to make an lm use a python interpreter without training. toolformer's approach @xcite is to prompt an lm to insert gadget tags ("api calls") into cot datasets and filter out irrelevant ones using the trained model's perplexity. in evaluation, toolformer simplifies the problem to only one tool call per example and supports generation with several heuristic rules. talm @xcite extends a training dataset by self-training: they start with a small set of fully annotated data, including the annotations of tool calls, and then iteratively generate cots with tool calls for a larger dataset with incomplete annotation. the examples added to the training set are chosen heuristically without guarantee that the entire reasoning chain is correct. we note that none of the referenced work publicly releases the resulting models. in combination with the many training and inference heuristics, it is largely difficult to reproduce and build upon the proposed methods. however, the availability of an extensive, standardized collection of toolassisted datasets like the one presented by calc-x will allow future work to substantially simplify the methods needed for creating tool-assisted models. ## calc-x interaction format we propose a semi-structured format for cot data to provide both the flexibility of unstructured text and the precision of structured formats. the html-based structure of interactions is compatible with existing parsers, such as beautiful-soup @xcite . this allows fast execution of parsing in the interaction with tools within the generation but also allows future work to easily transfer calc-x collection into other desired interaction formats. our format, displayed in figure 2 , uses three tags: gadget, output, and result. tag gadget is intended for inputs or "queries" to an external system. tag output wraps the response of the external system to the query. the tag result wraps the final result of the thought chain. after buying the bread and candy bar, you have 32-3-2= <gadget @xmath1. the final result is 18. <@xmath2: an example of target text from a chain-ofthought dataset encoded in our proposed format. our format is designed to allow the interaction of lms with multiple external systems, such as a calculator. ## creation of calc-x collection out of the datasets reviewed in section 2, we create the first version of calc-x collection from these datasets: gsm8k, aqua-rat, mathqa, ape210k, mawps, svamp and asdiv. our selection considers the datasets' size, primary focus on arithmetics, and parseability of the tool calls. the resulting calc-x collection is designed to simplify the correct usability of the whole collection in both training and evaluation while persisting the maximum of datasets' original information. most importantly, the process includes (1) the unification of key attributes (i.e. inputs, labels and correct results) over all datasets, and (2) the elimination of data leakages between different (train/val/test) data splits throughout the whole collection. we perform the second step based on a lexical overlap between pairs of samples' input texts from different splits across all datasets. we consider a pair of train and test examples a leak if the jaccard similarity of their 1-gram and 2-gram representations is over 0.5. this results in data splits composed of subsets of the original datasets, but thanks to this step, the whole calc-x collection can be used to perform both validation and tests over all datasets when all datasets are also used in training. the remainder of this section describes the conversion process of each dataset currently included in the calc-x collection. ## experiments to explore the potential of our newly curated data collection, we train models of identical architecture and parametrization in two configurations: ## conclusion this paper introduces a calc-x dataset collection, transforming over 300,000 samples of arithmetic reasoning datasets into a unified chain-of-thought format with explicit annotation of interaction with a calculator. calc-x enables integration of a simple symbolic system in the reasoning chains of language models via traditional supervised learning, easily allowing the models to offload mathematical computation to an external tool. we support the correct use of the calc-x collection for both training and evaluation by unifying the format of all included datasets and eliminating the datasets' mutual data leakages, making calc-x a convenient default for any future research addressing models' arithmetic reasoning. finally, we demonstrate the potential of calc-x by utilizing the whole unified collection in training and adjusting the models' inference process for the use of the calculator. as a result, our calculatorassisted models reach an accuracy that approximately doubles the accuracy of the traditional generation and outperforms existing previous work. we make the calc-x collection and newly-created calculator-supported calcformer models publicly available to facilitate further research in the fastpaced area of tool-using language models.