Multi-Operational Mathematical Derivations in Latent Space (2024)

Abstract: This paper investigates the possibility of approximating multiple mathematical operations in latent space for expression derivation. To this end, we introduce different multi-operational representation paradigms, modelling mathematical operations as explicit geometric transformations. By leveraging a symbolic engine, we construct a large-scale dataset comprising 1.7M derivation steps stemming from 61K premises and 6 operators, analysing the properties of each paradigm when instantiated with state-of-the-art neural encoders.Specifically, we investigate how different encoding mechanisms can approximate expression manipulation in latent space, exploring the trade-off between learning different operators and specialising within single operations, as well as the ability to support multi-step derivations and out-of-distribution generalisation. Our empirical analysis reveals that the multi-operational paradigm is crucial for disentangling different operators, while discriminating the conclusions for a single operation is achievable in the original expression encoder. Moreover, we show that architectural choices can heavily affect the training dynamics, structural organisation, and generalisation of the latent space, resulting in significant variations across paradigms and classes of encoders.

## introduction to what extent are neural networks capable of mathematical reasoning? this question has led many researchers to propose various methods to train and test neural models on different math-related tasks, such as math word problems, theorem proving, and premise selection (lu et al., 2023; meadows and freitas, 2023; mishra et al., 2022a; ferreira et al., 2022; ferreira and freitas, 2020; welleck 1 code & data available at: https://github.com/ neuro-symbolic-ai/latent_mathematical_ reasoning figure 1 : can neural encoders learn to approximate multiple mathematical operators in latent space? given a premise x, we investigate the problem of applying a sequence of latent operations (t 1 , . . . , t n ) to derive valid mathematical expressions (y 1 , . . . , y n ). @xcite @xcite . these methods aim to investigate how neural architectures learn and generalise mathematical concepts and symbolic rules, and how they cope with characteristic challenges of mathematical inference, such as abstraction, compositionality, and systematicity @xcite . in general, a key challenge in neural mathematical reasoning is to represent expressions and formulae into a latent space to enable the application of multiple operations in specific orders under contextual constraints. existing methods, however, typically focus on single-operational inferencei.e., optimising a latent space to approximate a specific mathematical operation @xcite @xcite . encoding multiple operations in the same latent space, therefore, remains an unexplored challenge that will likely require the development of novel mechanisms and representational paradigms. to investigate this problem, this paper focuses on equational reasoning, intended as the derivation of expressions from premises via the sequential application of specialised mathematical operations (i.e., addition, subtraction, multiplication, division, integration, differentiation). as derivations represent the workhorse of applied mathematical reasoning (including derivations in physics and engineering), projecting expressions and operators into a well-organised geometric space can unveil a myriad of applications, unlocking the approximation of mathematical solutions that are multiple steps apart within the embedding space via distance metrics and vector operations. specifically, this paper posits the following overarching research questions: rq1:"how can different representational paradigms and encoding mechanisms support expression derivation in latent space?"; rq2:"what is the representational tradeoff between generalising across different mathematical operations and specialising within single operations?"; rq3:"to what extent can different encoding mechanisms enable multi-step derivations through the sequential application and functional composition of latent operators?"; rq4:"to what extent can different encoding mechanisms support out-of-distribution generalisation?" to answer these questions, we investigate jointembedding predictive architectures (lecun, 2022) by introducing different multi-operational representation paradigms (i.e., projection and translation) to model mathematical operations as explicit geometric transformations within the latent space. moreover, by leveraging a symbolic engine @xcite , we build a large-scale dataset containing 1.7m derivation steps which span diverse mathematical expressions and operations. to understand the impact of different encoding schemes on equational reasoning, we instantiate the proposed architectures with state-of-the-art neural encoders, including graph neural networks (gnns) @xcite , convolutional neural networks (cnns) @xcite , recurrent neural networks (rnns) @xcite , and transformers @xcite , analysing the properties of the latent spaces and the ability to support multi-step derivations and generalisation. our empirical evaluation reveals that the multioperational paradigm is crucial for disentangling different mathematical operators (i.e., crossoperational inference), while the discrimination of the conclusions for a single operation (i.e., intraoperational inference) is achievable in the original expression encoder. moreover, we show that architectural choices can heavily affect the training dynamics and the structural organisation of the latent space, resulting in significant variations across paradigms and classes of encoders. overall, we conclude that the translation paradigm can result in a more fine-grained and smoother optimisation of the latent space, which better supports cross-operational inference and enables a more balanced integration. regarding the encoders, we found that sequential models achieve more robust performance when tested on multistep derivations, while graph-based encoders, on the contrary, exhibit better generalisation to out-ofdistribution examples. ## multi-operational derivations given a premise x -i.e., a mathematical expression including variables and constants, and a set of operations @xmath0, t 2 , . . . t n } -e.g., addition, multiplication, differentiation, etc., we investigate the extent to which a neural encoder can approximate a mathematical function f (x, t i ; v ) = y t i that takes the premise x and any operation t @xmath1, and produces the set of valid expressions y t @xmath2, y 2 , . . . , y m } derivable from x via t i given @xmath3, where v is a predefined set of operands such that y @xmath4, v j ). in this work, we focus on atomic operations in which v includes symbols representing variables. for example, consider the following premise x: u + cos (log (-z + o)) if the set of operands is @xmath5, u}, and the operation t i is addition, then the application of t i to x should result in the set of expressions: ## training details as the models are trained to predict a target embedding, the main goal during optimisation is to avoid a representational collapse in the expression encoder. to this end, we opted for a multiple negatives ranking (mnr) loss with in-batch negative examples @xcite . this technique allows us to sidestep the explicit selection of the negative sample, enabling a smoother optimisation of the latent space. we trained the models on a total of 12.800 premise expressions with 24 positive examples each derived from the application of 6 operations (see section 2.2). this produces over 307.200 training instances composed of premise x, operation t, and conclusion y. the models are then trained for 32 epochs with a batch size of 64 (with in-batch random negatives). we found that the best results are obtained with a learning rate of 1e-5. ## related work the quest to understand whether neural architectures can perform mathematical reasoning has led researchers to investigate several tasks and evaluation methods @xcite @xcite @xcite @xcite @xcite . in this work, we focused on equational reasoning, a particular instance of mathematical reasoning involving the manipulation of expressions through the systematic application of specialised operations @xcite @xcite . in particular, our work is inspired by previous attempts to approximate mathematical reasoning entirely in latent space @xcite . differently from @xcite , we investigate the joint approximation of multiple mathematical operations for expression derivation @xcite explore exclusively the rewriting operation for theorem proving). moreover, while @xcite focus on the evaluation of graph neural networks @xcite )), we analyse the behaviour of a diverse set of representational paradigms and neural encoders. our data generation methodology is inspired by recent work leveraging symbolic engines and algorithms to build systematic benchmarks for neural models @xcite @xcite . however, to the best of our knowledge, we are the first to construct and release a synthetic dataset to investigate multi-step and multi-operational derivations in latent space. ## conclusion this paper focused on equational reasoning for expression derivation to investigate the possibility of approximating and composing multiple mathematical operations in a single latent space. specifically, we investigated different representational paradigms and encoding mechanisms, analysing the trade-off between encoding different mathematical operators and specialising within single operations, as well as the ability to support multistep derivations and out-of-distribution generalisation. moreover, we constructed and released a large-scale dataset comprising 1.7m derivation steps stemming from 61k premises and 6 operators, which we hope will encourage researchers to explore future work in the field.