A Seq2Seq approach to Symbolic Regression
Abstract
Deep neural networks have proved to be powerful function approximators.
The large hypothesis space they implicitly model allows them to ﬁt very complicated black-box functions to the training data.
However, often the data generating process is characterized by a concise and relatively simple functional form.
This is especially true in natural sciences, where elegant physical laws govern the behaviour of the quantities of interest.
In this work, we address this dichotomy from the perspective of Symbolic Regression (SR).
In particular, we apply a fully-convolutional seq2seq model to map numerical data to the corresponding symbolic equations.
We demonstrate the effectiveness of our approach on a large set of mathematical expressions by providing both a qualitative and a quantitative analysis of our results.
Additionally, we release our new equation-generator Python library in order to facilitate benchmarking and stimulate new research on SR 1 .
SR refers to the task of learning a mapping from a set of numerical input-output pairs { x i , y i } Mi =1 , where x i ∈ R d and y ∈ R q , to the corresponding symbolic expression f , implicitly deﬁned by the mathematical relation y =
f ( x ) .
While standard regression methods aim at ﬁtting the training data as closely as possible, SR targets the underlying symbolic expression.
Having access to such a compact representation of the data is of high interest for researchers in many scientiﬁc disciplines since it could lead to a deeper understanding of the physical system under consideration.
However, SR is hard.
The space of mathematical expressions grows exponentially with the length of the equation , allowing current search methods to retrieve only limited-sized expressions.
The strong combinatorial nature of SR has motivated the application of evolutionary algorithms and speciﬁcally genetic programming (GP)
[Koza 1993, Schmidt and Lipson, 2009] techniques.
Despite their relatively good performances in several settings, including SR, these methods do not scale well to larger problems and are very sensitive to hyperparameter choices.
A more recent line of research [Sahoo et al., 2018, Udrescu and Tegmark, 2020, Petersen, 2020] has started exploring the application of Deep Learning (DL) approaches to tackle the aforementioned combinatorial challenge.
Despite some encouraging results, the performance of these approaches cannot improve with more data and compute. Speciﬁcally, solving more equations at training time does not change how the model works at test time, as the algorithms are hand-designed to work on one equation at the time. Instead, in this work we propose a method that learns to do symbolic regression, and beneﬁts from both data and compute [Sutton, 2019].
We adapt the convolutional seq2seq model proposed by Gehring et al.
[2017] in the context of Natural Language Processing (NLP) to directly map numerical input-output pairs { x i , y i } Mi =1 to the corresponding symbolic expression f .
We qualitatively show that our model can recover the exact form of the symbolic expression describing the data and provides better generalization performances than a fully-connected deep neural network trained to map each x i into the corresponding y i .
Methodology
Our methodology is based on three core ideas.
Primarily, we want to create a model that captures correlations across different symbolic expressions and does not need to be retrained for each new
SR task.
To this extent, we create programmatically a dataset, D =
{ Y i f i } Ni =1 , comprising N pairs of symbolic functions, f i , and the corresponding numerical evaluations, Y i =
f i ( X i ) , on a ﬁxed grid of M support points, X i =
X = { x j } Mj =1 .
This method allows us to create arbitrarily large and diverse datasets to train and evaluate our model in a supervised fashion.
As a result, our model takes as input a set of M numerical points, Y i , and outputs a string corresponding to the mathematical expression generating the data.
The second idea stems from the observation that using a numerical loss, such as the RMSE, measuring the distance between the predicted equation and the ground truth one, often leads to numerical issues and overﬁtting.
Indeed, given a model with enough capacity and an arbitrary underlying equation, training data points can often be approximated by multiple functions, with a different structure than the true one, up to a very small error.
To cope with this aspect, we base our models on a loss function measuring how far our prediction stands from the true expression entirely at a symbolic level.
We use the Cross Entropy Loss between the predicted tokens and the ground-truth ones as our objective function.
The third idea, similarly to Petersen [2020] is to defer to a second independent step, the computation of the numerical constants.
This design choice is motivated by the consideration that using tokenized values for representing numerical values is not possible.
On the other end, it is simple to ﬁt numerical constants with non-linear optimizers (e.g.
BFGS Fletcher [1987]), given a symbolic skeleton.
An illustration of the aforementioned ideas and an overview of our approach are shown in Fig.
[Figure]
numerical data points, Y i
= f i ( X i ) , described by a mathematical expression, f , are passed through an encoder. The encoder is responsible for processing the data and passing the resulting representation to the decoder. Finally, the decoder generates a sequence of integer tokens, each corresponding with a different symbol, making up the ﬁnal output expression.
In a second and independent step, we ﬁt the numerical constants with a non-linear optimizer (BFGS). The ideas exposed above present a number of advantages over existing SR approaches.
First and foremost, it is possible to generate an arbitrarily large dataset of equations to train our model.
This is in contrast with previous methods that have to be retrained from scratch for each new expression.
Furthermore, it is very easy to impose speciﬁc inductive biases into the learning process by opportunely customizing the dictionary used for generating the symbolic equations.
In contrast, previous works had to introduce speciﬁc design choices to make learning certain types of equations possible.
Sahoo et al.
[2018], for instance, had to modify their end-to-end architecture to include division activation functions and adapting the training strategy accordingly to avoid numerical instabilities.
Udrescu and Tegmark [2020] instead, manually check whether a set of data is characterized by multiplicative separability in order to recursively restrict the search space for their algorithm.
Our approach also differs from the technique proposed by Petersen [2020], since in their case, the network is not conditoned on the numerical input data and it is trained indirectly with the REINFORCE algorithm
[Williams, 1992] using the RMSE loss as the reward function.
Seq2Seq Modelling.
We choose to implement our approach by opportunely adapting the architecture proposed by Gehring et al.
[2017] to the SR setting.
This model is fully-convolutional and incorporates gated recurrent units, residual connections and attention.
CNNs are easier to parallelize during training than RNN-based models, thus allowing full exploitation of GPU hardware, resulting in signiﬁcant run-time gains.
Experimental Results
We evaluate our approach on a dataset, D = { Y i f i } Ni =1 , of N = 50 .
000 equations created with our library.
Each function, f i contains randomly generated numerical constants and we instantiate the input of our model, Y i , by evaluating f i on a ﬁxed set of M = 30 equally-spaced support points,
X , into the interval [0 . 1 , 3 . 1] .
Our seq2seq model is trained to map each Y i into an expression, g i , which has the same skeleton as f i but does not contain any numerical constants.
The ﬁnal function f i can then be retrieved by running the BFGS optimizer on the family of equations spanned by g i (see Fig. 1).
Since every function contains different constants, all the elements of the dataset are characterized by different numerical inputs Y i .
On the other hand, multiple instances of the same symbolic skeleton might be present.
Further details about the dataset can be found in the Appendix.
Visualizations.
Fig.
2 shows a comparison between our model and a three-layer fully-connected neural network with 100 units per layer, trained to approximate three different nonlinear functions.
For each plot, the green region indicates the training domain, whereas the red area shows the extrapolation region.
This example shows that our network (orange) successfully retrieves the underlying symbolic equation (green), while the fully-connected neural network (blue) fails to extrapolate outside the interval.
Note that the equations shown in this example appear in the training set with different internal and external constants than those used here.
Fig.
3 illustrates the behaviour of our model when equations comprising out-of-dictionary symbols are used as input.
These equations include tokens that our network has never seen during training (e.g. “ cos ”, “ cosh ” and the division operator “ / ”).
Our model outputs expressions with known symbols whose numerical values resemble those associated with the ground-truth equations.
In the left panel, the output of our model is “ y = sin( x 2 + 1) ”, which captures the sinusoidal nature of the ground-truth function.
In the middle, our network outputs “ y = log( x ) 2 + 1 ”, whereas for the equation to the right, its output is “ y = log( x ) 4 + 1 ”.
In both cases, the network’s output diverges at x = 0 , in accordance with the ground-truth.
In one case (middle), the prediction follows the same trend as the ground truth as x → ∞ , while for the other (right), the network fails to model the decreasing trend of the underlying function in the extrapolation region.
We argue that this behaviour results from the lack of training examples characterized by such decreasing trend and could thus be resolved by enlarging and diversifying our training set.
Quantitative Analysis.
Here, we propose a novel approach to evaluate SR models, with a particular emphasis on generalization.
We consider two test sets of 1.000 equations each.
The ﬁrst test set includes skeleton expressions appearing in the training set with different numerical constants.
The second dataset includes more complex novel symbolic expressions that our model has never seen during training.
For each function f i , we feed the network with the corresponding evaluation points f ( X ) =
Y i , where X is a vector of 30 equally spaced points into the interval I =
Then, the resulting output expression is evaluated on a set of 30 points into the extrapolation region
E = [3 . 1 , 6 . 1] .
Finally, the RMSE is calculated between prediction and ground-truth on E .
We compare its performances with a 3-layer feed-forward neural network (FFNN3 for short) trained on
I and tested on E .
Fig.
4 illustrates the extrapolation performance of the two methods.
As shown above, our model outperforms the neural network on both datasets.
The conventional feed-forward model must be trained separately for each new expression disregarding information of previously seen equations.
Our model can instead leverage knowledge from the entire training set.
Discussion
This work proposes a novel approach to perform SR tasks in an end-to-end manner, i.e., from numbers directly to symbolic expressions.
The results show that our seq2seq model, originally proposed in the context of NLP applications, can also be effective in the domain of SR.
We also introduce a new library to programmatically generate datasets of equations and facilitate benchmarking.
Future work will be mainly focused on four research lines: 1 ) Extending our method to handle multivariate equations and not relying on ﬁxed support points introduces scalability issues.
To cope with them, a possible approach could be to cast our method into a set-to-sequence framework and leverage recent advances in this domain [Choy et al., 2019, 2020].
2 ) Recursively reﬁning the network’s output according to the signal obtained from a suitable combination of symbolic and numerical losses could improve predictive performances.
A Reinforcement Learning approach could be helpful to this extent; 3) Leveraging more recent advances in the NLP literature, such as Transformers [Vaswani et al., 2017], could also provide additional beneﬁts.
Exploiting our freedom to generate arbitrarily large datasets will likely improve the performance of our method.
Appendix 4.1 Expression generation
By the term “expression”, we refer to a ﬁnite sequence of symbols with a well-deﬁned meaning from a mathematical point of view.
Syntactically meaningful expressions are generated programmatically by our developed library.
Tables 1 shows all the symbols making up the vocabulary we used in our experiments.
[Table]
It is important to note that there might be multiple ways to write the same mathematical expression.
Consider, for example, “ sin( x )+cos( x ) 2 +cos( x ) ”: one can reverse the order of the terms in the sum (e.g. “ cos( x ) 2 + sin( x ) + cos( x ) ”) or group some terms together (e.g. “ sin( x ) + cos( x )(cos( x ) + 1) ”) and the resulting equations will have exactly the same mathematical information content.
Furthermore, one can exploit trigonometric identities and rewrite the original equation as “ sin( x ) + 1 − sin( x ) 2 + cos( x ) ”.
These one-to-many relationships hinder our neural network performance by introducing ambiguities into the mapping between numbers and symbols.
To counteract this issue, when generating a batch of expressions used for training, we adopt the following additional rules:
We generate expressions that cannot be simpliﬁed by summation (i
e. library won’t generate “ 2 sin( x ) + 3 sin( x ) ” , while it may “ 5 sin( x ) ”).
We generate exclusively fully-expanded equations, where no elements are collected (e
g. “ sin( x )(sin( x ) + 1) ” is not generated, while “ sin( x ) 2 + sin( x ) ” is).
We keep order consistent by alphabetically sorting operands of summations and multiplications.
4.2 Training dataset
[Figure]
In this section, we provide some additional details about our training dataset.
Table 2 shows four data instances extracted form the dataset.
We generate the data starting from an expression, f i , containing numerical constants.
We evaluate such expression on a ﬁxed set of numerical support points, X , to obtain the input of our model, Y i .
The goal of the network is to predict the symbolic skeleton, g i , associated with the original expression f i .
Note that there might be several instances of the same symbolic skeletons, each associated with different evaluation points.
This aspect is illustrated in Fig.
5, were multiple instances of the same symbolic skeleton ( sin( x 3 ) ) are shown.
Overall, our training set contains 50.000 numerically different inputs Y i and ∼ 17 .
000 unique symbolic skeletons, g i .
[Table]
4.3 Additional Results
In this section, we report further experimental results not included in the main body of the manuscript due to space constraints.
First, we show how the BFGS optmizer acts in combination with our seq2seq neural network to successfully retrieve the constants present in the original expression.
Then, we report additional results related to the quantitative analysis performed in section 2.
Visualizations Fig.
6 provides three examples where the combination of our model and the BFGS optimizer manages to retrieve the exact expression of three equations with internal and external constants.
In all the above examples, the neural network outputs the correct symbolic skeleton and the BFGS optimizer successfully ﬁnds the numerical constants.
Quantitative Analysis Table 3 shows the percentage of expressions for which the RMSE extrapolation error is below three threshold levels for our seq2seq model and FFNN3.
[Table]
As expected, for both models the percentages increase as the threshold grows.
However, the seq2seq model consistently outperforms FFNN3 on both datasets.