A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers (2024)

Abstract: This paper proposes a methodology for generating and perturbing detailed derivations of equations at scale, aided by a symbolic engine, to evaluate the generalisability of Transformers to out-of-distribution mathematical reasoning problems. Instantiating the framework in the context of sequence classification tasks, we compare the capabilities of GPT-4, GPT-3.5, and a canon of fine-tuned BERT models, exploring the relationship between specific operators and generalisation failure via the perturbation of reasoning aspects such as symmetry and variable surface forms. Surprisingly, our empirical evaluation reveals that the average in-distribution performance of fine-tuned models surpasses GPT-3.5, and rivals GPT-4. However, perturbations to input reasoning can reduce their performance by up to 80 F1 points. Overall, the results suggest that the in-distribution performance of smaller open-source models may potentially rival GPT by incorporating appropriately structured derivation dependencies during training, and highlight a shared weakness between BERT and GPT involving a relative inability to decode indirect references to mathematical entities. We release the full codebase, constructed datasets, and fine-tuned models to encourage future progress in the field.

## generating and perturbing derivations with symbolic engines the data generation approach outputs and perturbs multi-step reasoning involving step annotations and equations. a model learns mathematical reasoning in the context of a given task, is evaluated on an in-distribution test set, then each element of that set is symbolically perturbed and the difference in model inference due to the perturbation contributes to a generalisability measure for the model (fig. 1 ). to outline this process, given a vocabulary of symbols v and set of computer algebra operations r, each set is sampled from to ultimately generate an ordered list of steps s @xmath0, where d represents the output derivation. an initial reasoning step s 1 = (premise, annotation) is generated such that @xmath1. an operation @xmath2, which in its most general form accepts two operands (arity 2). the first operand is an equation s j,1 from tuple s @xmath3. a suitable secondary variable (∈ v), expression, or equation operand m is extracted from d, and the next equation is generated by applying operation r through s i+1,1 = r(s j,1 , m). the annotation s i+1,2 is also a list containing (most generally) the name of the operation, the equation index, and secondary operand, such that s i+1,2 = [r, j, m ′ ] (where m ′ is a variable/expression string or equation index representing operand m). therefore, step s i+1 = (r(s j,1 , m), [r, j, m ′ ]). if @xmath4, then @xmath5, and the derivation updates such that @xmath6, s 2 ]. this process repeats until the derivation reaches a target length. given a specific task (e.g., sequence classification), a derivation d is adapted to form model input d ′ , such that a static test set is sampled from the same distribution as the training set or in-context examples. this static set x contains task examples d ′ i and labels. for all d ′ @xmath7, a perturbation p is applied to corresponding initial derivation d i to form a perturbed derivation p(d i ), which is similarly adapted to form an out-of-distribution perturbed task example p(d i ) ′ ∈ p , where p is the out-of-distribution test set corresponding to static set x, such that p : @xmath8. now that we have static (in-distribution) and perturbed (out-of-distribution) reasoning pairs (x i , p i ), for a given model m we can evaluate its generalisability by comparing its static and perturbed predictions, respectively m(x i ) and m(p i ). together with the respective labels, these outputs allow comparisons between aggregate scores (tab. 1 and 2 ), but also a pair-wise analysis involving more sophisticated logic involving m(x i ) and m(p i ), which may include perturbations other than p (tab. 3 ). this provides a highly controllable symbolic framework for evaluating mathematical generalisation capabilities of models. the approach produces mathematical reasoning at scale, and can both improve the depth of mathematical corpora through the generation of underrepresented reasoning, and serve as the backbone for testing model generalisability in numerous settings. ## evaluation in this section, we discuss how the scores obtained by the bert and gpt models reflect their reasoning capabilities within the scope of the classification tasks described in the previous section. training and prompts. details related to finetuning bert and prompting gpt are given in appendix a. to summarise, for a single pre-trained bert model, five fine-tuned models are trained for each of the task variations. for instance, in derivation step classification, a model is trained per number of steps in the input derivations (i.e., 2, 3, and 4 steps). for calculus classification a model is trained per operation (i.e., differentiation and integration). the gpt models are given 4-shot prompts. while we acknowledge that chain-ofthought @xcite and tree-of-thought (yao bert-base-uncased (@xmath9.7 88.9 87.0 88.1 87.0 88.0 87.5 88.7 ----bert-base-uncased (@xmath10.9 78.7 71.9 71.0 69.1 66.0 53.7 50.6 68.4 69.0 --bert-base-uncased (@xmath11.8 63.6 55.0 60. further generalisation. in addition to the outof-distribution perturbations applied to input sequences for each task, a model that can sufficiently generalise the underlying reasoning should be able to solve (on average) mathematically less complex versions of problems encountered during training. in derivation step classification, we evaluate models trained on derivations with a fixed step count on a set of derivations composed of a lower number of steps. this is represented in the s -1 and s -2 columns in tab. 1 given initial step count, s. in calculus classification, where models are exposed to examples comprising at least two variables, (e.g., cos(ax)z) we generate a set of easier problems with 1.5k examples consisting of only one variable (e.g., cos(x)), corresponding to the easy column of tab. 2. these out-of-distribution datasets complement the perturbations in the following discussion. gpt-4 rivals in-distribution performance of fine-tuned bert-based models while demongpt-4 can fail to predict mathematical coherence from in-context examples alone. the calculus classification task includes minimalistic sequences without operation annotations. surprisingly, while gpt-4 achieves the best performance on derivation step classification, competitive performance is not observed in calculus classification despite its lower complexity. we attribute this to the fact that, unlike bert, gpt is not fine-tuned on a specific operation, and in-context examples alone might not contain enough information to consistently discriminate whether a particular sequence involves either differentiation or integration. this is evidenced by the fact that both gpt models score higher on the ec (equation conversion) set. the ec perturbation changes nothing about the operation being performed, but adds context by writing (e.g.) differentiated expressions as equations with a lhs that includes d dx . f1 scores in gpt models increase by up to 12 points in this case, while bert-based scores decrease by up to 80 points (tab. 2). to reinforce this, in derivation step classification, both gpt models obtain comparatively lower scores on the ar (annotation replacement) set. this is because sufficient context has been provided only for an operator that differs to the main annotation operator. gpt only learns the format of the sequences and the expected output for the task in this case. however, static performance is maximised by designing the prompt in this manner (tab. 4). gpt-3.5 cannot effectively classify mathematical reasoning. gpt-3.5 scores 15 less f1 points than the average encoder score of 80% on the indistribution set, and is notably outperformed by bert-based models on most test sets (particularly scibert). a notable exception are those that contain less steps (tab. 1), where performance gener-ally increases comparative to static in-distribution scores. this contrasts with the significant corresponding performance drops observed in the bert-based evaluation, indicating that gpt learns enough from in-context examples to generalise to derivations with less steps, and therefore has a deeper relative understanding of the underlying mathematics. encoder models fail to generalise. for derivation step classification, models average 80% f1 over all static derivation lengths, and decreases due to perturbations average 10% (vr), 11% (ee), and 16% (ar). this is at most 4% above f1 majority baseline. bert-uncased and scibert-cased finetuned on 2-step derivations are exceptions, but the 13 other encoder models are sensitive to at least one perturbation. all models tested do not generalise to less derivation steps, reaching as low as 11% f1. in calculus classification static scores average 90% and perturbations decrease this by 17% (vr) and 33% (ec). all fine-tuned models fail to generalise to perturbations and simpler examples, with 97% f1 scores repeatedly dropping below 17%. despite the in-distribution performance, this indicates their reliance on superficial patterns rather than the underlying rules of the operators. ## limitations overall ethical impact. this work explores a systematic way to elicit the mathematical/symbolic inference properties of transformer-based models in mathematical language processing tasks. as such, it contributes in the direction of a critique of the reasoning capabilities and the biases of these models.