R ETHINKING S YMBOLIC R EGRESSION : M ORPHOLOGY & A DAPTABILITY FOR E VOLUTIONARY A LGORITHMS
Anonymous authors
Paper under double-blind review
A BSTRACT
Symbolic Regression (SR) is the well-studied problem of finding closed-form analytical expressions that describe the relationship between variables in a measurement dataset.
In this paper, we rethink SR from 2 perspectives: morphology and adaptability.
Morphology: Current SR algorithms typically use several manmade heuristics to influence the morphology (or structure) of the expressions in the search space.
These man-made heuristics may introduce unintentional bias and data leakage, especially with the relatively few equation-recovery benchmark problems available for evaluating SR approaches.
To address this, we formulate a novel minimalistic approach, based on constructing a depth-aware mathematical language model trained on terminal walks of expression trees, as a replacement to these heuristics.
Adaptability: Current SR algorithms tend to select expressions based on only a single fitness function (e g., MSE on the training set).
We promote the use of an adaptability framework in evolutionary SR which uses fitness functions that alternate across generations.
This leads to robust expressions that perform well on the training set and are close to the true functional form.
We demonstrate this by alternating fitness functions that quantify faithfulness to values (via
MSE) and empirical derivatives (via a novel theoretically justified fitness metric coined MSEDI).
Proof-of-concept: We combine these ideas into a minimalistic evolutionary SR algorithm that outperforms a suite of benchmark and state of-theart SR algorithms in problems with unknown constants added , which we claim are more reflective of SR performance for real-world applications.
Our claim is then strengthened by reproducing the superior performance on real-world regression datasets from SRBench.
For researchers interested in equation-recovery problems, we also propose a set of conventions that can be used to promote fairness in comparison across SR methods and to reduce unintentional bias
I NTRODUCTION
Important discoveries rarely come in the form of large black-box models; they often appear as simple, elegant, and concise expressions.
The field of applying machine learning to generate such mathematical expressions is known as Symbolic Regression (SR).
The expressions obtained from
SR come in a compact and human-readable form that has fewer parameters than black-box models.
These expressions allow for useful scientific insights by mere inspection.
This property has led SR to be gradually recognized as a first-class algorithm in various scientific fields, including Physics (Udrescu & Tegmark, 2020), Material Sciences (Wang et al., 2019; Sun et al., 2019) and Knowledge
Engineering (Martinez-Gil & Chaves-Gonzalez, 2020) in recent years.
The most common technique used in SR is genetic programming (GP) (Koza, 1992).
GP generates populations of candidate expressions and evolves the best expressions (selected via fitness function) across generations through evolutionary operations such as selection, crossover, and mutation.
In this paper, we rethink GP-SR from 2 perspectives: morphology and adaptability.
Morphology.
SR algorithms have traditionally used several man-made heuristics to influence the morphology of expressions.
One method is to introduce rules (Worm & Chiu, 2013), constraints (Petersen et al., 2019; Bladek & Krawiec, 2019) and rule-based simplifications (Zhang et al., 2006), with the objective of removing redundant operations and author-defined senseless expression.
An example is disallowing nested trigonometric functions (e.g. sin (1 + cos ( x )) ).
Another method is to assign complexity scores to each elementary operations (Loftis et al., 2020; Korns, 2013), intending to suppress the appearance of rare operations that are given a high author-assigned complexity score.
However, these man-made heuristics may introduce unintentional bias and data leakage, exacerbated by the small quantity of benchmark problems in SR (Orzechowski et al., 2018).
With the success of deep learning and its applications to SR, there exists substantial motivation to utilize deep learning to generate potential morphologies of candidate expressions in SR (Petersen et al., 2019; Mundhenk et al., 2021).
Such a technique also comes with the benefit of being easily transferable to a problem with a different set of elementary operations.
In this regard, we first show how current SR methods are reliant on these man-made heuristics and highlight the potential drawbacks.
Then, in this paper, we show how using our neural network pre-trained on Physics equations (which we later introduce as TW-MLM) improves the performance of GP-SR even in the absence of such man-made heuristics.
Adaptability.
SR algorithms tend to evaluate a candidate expression based on its faithfulness to empirical values.
Some common fitness functions are Mean Absolute Error (MAE), Mean Squared
Error (MSE) and Normalized Root Mean Square Error (NRMSE) (Mundhenk et al., 2021), among other measurements.
We propose that SR should create a variety of characterization measures beyond the existing ones that measure faithfulness to empirical values.
While previous work have suggested alternative characteristics in dealing with time-series data (Schmidt & Lipson, 2010; 2009), it is not easily transferable to SR in general and was not theoretically derived.
In this paper, we propose to quantify the faithfulness of a candidate expression’s empirical derivative to the ground truth.
Motivated by evolutionary-theory (Bateson, 2017), we adopt an adaptability framework that changes the fitness functions across generations.
This process makes it harder for pseudo-equations to survive through the different fitness functions and easier for a ground truth equation to survive across the generations.
The additional benefit of such a method lies in the increased utility of the eventual expression.
If the intention of the user is to take the derivative of the eventual equation, then a measure of faithfulness to empirical derivatives as fitness would assist that objective.
In this paper, we alternate between different fitness functions to improve performance of GP-SR.
Specifically, we alternate between MSE and a newly defined fitness function we term MSEDI.
Proof-of-concept.
We combine these ideas to demonstrate a proof-of-concept (foreshadowed in
Figure 1) through a minimalistic evolutionary SR algorithm that outperforms all methods (including state-of-the-art SR) in problems with unknown constants (i.e., Jin* (Jin et al., 2019)) and outperforms many benchmark models in problems without unknown constants (i.e., Nguyen* (Uy et al., 2011) and R* (Krawiec & Pawlak, 2013)).
We contend that performance on datasets with unknown constants are more indicative of SR performance in real-world applications that including naturally occurring processes such as scaling.
Our claim is then strengthened by reproducing this superior performance on real-world regression datasets from SRBench (La Cava et al., 2021).
To accommodate future research in SR using real-life datasets, we propose extra SR conventions for synthetic datasets in line with the ”relevancy” criteria for results on benchmarks problem to correlate well with results on realworld applications (McDermott et al., 2012).
The remaining parts of this paper are organized as follows:
Section 2 explains the related work in SR and mechanisms to improve SR, with focus on recent deep learning work.
Section 3 describes our proposed methodology and mechanisms.
Section 4 combines all our proposed mechanisms with the vanilla GP approach and compares the results obtained with state-of-the-art, commercial and traditional SR approaches.
The conclusion and future work are given in Section 5.
The following summarizes the main contributions of this paper:
Proposed a set of conventions and best practices that reduce the risk of human bias in the context of using SR on real-life datasets.
Developed a predictive model called TW-MLM that learns the morphology of expressions through a mathematical language model, which is then used to generate candidate expressions
for SR.
TW-MLM serves as an
alternative to man-made heuristics that is less prone to human bias and easily transferable to new problems.
Proposed a method for alternating fitness functions inspired by adaptability in evolution theory to tackle the problem of multi-objective optimization in SR.
We do this by alternating between MSE and our novel theoretically justified metric called MSEDI.
Developed an integrated a proof-of-concept by combining our ideas and performed extensive testing on both synthetic and real-world datasets (from SRBench).
R ELATED W ORK
Deep Learning for SR.
Deep learning has seen multiple success in SR.
Recent successes include
Deep Symbolic Regression (DSR) (Petersen et al., 2019) which uses reinforcement learning to train a recurrent neural network (RNN) to complete an expression tree.
The laboratory also published an improved method, Deep Symbolic Optimization Neural-Guided Genetic Programming (DSONGGP), utilizing a hybrid approach that combines a similar RNN trained by reinforcement learning coupled with GP (Mundhenk et al., 2021).
The authors suggest that GP helps to produce large variations in the population of candidate equations when stuck at a local optima.
However, these methods are used in conjunction with several man-made heuristics to be effective.
Other works include AI-Feynman, which utilizes neural networks to discover simplifying properties in data such as symmetry and separability to reduce the problems to fewer variables (Udrescu & Tegmark, 2020).
Mathematical Language Model.
Symbolic mathematics has been successfully addressed as both a machine translation problem and a next word prediction problem (Lample & Charton, 2019;
Kim et al., 2021).
In particular, it has been reported that by adding a pre-trained seq2seq model as a supplementary mechanism to neural-guided SR, performance has improved (Kim et al., 2021).
The consensus is that mathematics should be viewed as a language with complex rules.
Most useful mathematical expressions are not only short and concise, they also tend to obey a variety of hidden rules, such as avoiding nested trigonometric functions (e.g. sin (1 + cos ( x )) .
In this context, a mathematical language model can be developed to learn such rules implicitly.
Derivatives in SR.
The usage of derivatives in SR has been hinted in previous works for timeseries data (Schmidt & Lipson, 2010; 2009).
In their work, given N data samples from two timedependent variables, x ( t ) and y ( t ) , the system’s derivatives are ∆ x/ ∆ y = x ′ /y ′ , where x ′ and y ′ represent the empirical derivatives of x and y with respect to time.
These values are then compared against the derivative obtained from the candidate expressions, δx i /δy
i , through a mean logarithmic error: − 1 N (cid:80) Ni
=1 log(1 + | ∆ x i / ∆ y
i − δx i /δy
i | ) .
However, the theoretical derivation is not explored in their paper.
Recent works in SR also include model discrimination by incorporating prior information on the sign of the first derivative based on physical knowledge (Engle & Sahinidis, 2021).
In our paper, we develop a theoretical basis in line with existing assumptions for modelling errors to quantify faithfulness to derivatives for general equations which do not necessitate time-dependency.
Multi-Objective Genetic Programming (MOGP).
Several general approaches exist for MOGP (Konak et al., 2006).
The first is to combine the multiple objectives into a single weighted linear combination.
However, it is difficult to optimize the weights and small changes in weights have been shown to lead to drastically different solutions.
The second approach is to set all but one objective as constraints and then optimize the remaining objective.
However, the process of selecting values for constraints and which objectives to set as constraints is arbitrary.
The last method is to output a Pareto optimal set instead of a single individual expression to reflect the trade-offs between the different objectives.
However, in the context of SR, the ground truth expression would be the best performer for both objectives, rather than forming a trade-off.
Benchmarks Methods.
For comparison, we include traditional SR, state-of-the-art (SOTA), commercial algorithms, and random search.
The methods selected report the lifetime population size of expressions to enable fair comparison of recovery rates .
The benchmark methods are: (i) DSR (Petersen et al., 2019)
Previous method that pioneered usage of reinforcement learning in SR (ii DSO NGGP (Mundhenk et al., 2021): Current method in SR is a hybrid of DSR and
GP (Stephens, 2016) standard
Koza, 1992) has seen wide usage Pankratius et al Ferreira et al (TuringBot, 2020) simulated which has been shown to be competitive
Ashok et al
M ETHODOLOGY AND M ECHANISMS
Here, we first propose revised conventions to be used for SR experiments targeted at resolving the flaws and criticism of current SR metrics.
We then introduce our novel methods of controlling morphology of expressions and promoting adaptability in evolution, and justify these methods individually through preliminary results and ablation studies.
3. P ROPOSED C ONVENTIONS AND B EST P RACTICES
We propose and justify conventions for SR experiments, adhering to the criteria discussed in the call for better benchmarking to be done so that SR experiments can correlate better with results on realworld applications (McDermott et al., 2012).
In many recent papers, recovery rate has been utilized as the primary metric for evaluating SR and is defined as ”the fraction of independent training runs in which an algorithm discovers an expression that is symbolically equivalent to the ground truth expression within a maximum of 2 million candidate expressions” (Mundhenk et al., 2021; Petersen et al., 2019; Kim et al., 2021; Larma et al., 2021).
To this end, the conventions we propose will be focused on improving the effectiveness of recovery rate as an evaluation metric for SR experiments.
Fixed set of primitive functions.
In contrast to previous SR experiments that use a varying primitive function set depending on the equation, we propose to use a fixed set of primitive functions across all datasets and all methods.
This is a necessary step towards using SR on real-life datasets since we are blind to the underlying primitive functions in real-world scenarios.
In our paper, we use a fixed primitive function set { add , mul , sub , div , sin , cos , arcsin , log , exp , pow } for all datasets and methods, selected from the dataset used to train our mathematical language model.
Top-1 Approximate Recovery.
We also propose a new measure, top-1 approximate recovery rate , that is more reflective of performance on real-life datasets compared to the exact recovery rate defined in the first paragraph of Section 3.1.
Top-1 means that we only take the best scoring expression in the strictest sense, and do not consider the other equations generated in the lifetime of the method as consistent with how SR is used for real-life data (Abdellaoui & Mehrkanoon, 2021; Phukoetphim et al., 2016;
Barmpalexis et al., 2011).
In other words, we only assess one best equation per experimental run.
We define an approximate recovery to be when the r-squared value of an expression (touted as the best error measure for SR (Keijzer, 2004)) over the entire sampling domain of the dataset is more than 99%.
This is consistent with calls from the GP community to discourage measuring exact recovery on synthetic datasets as they usually do not correlate well with performance on real-life applications (McDermott et al., 2012).
The most obvious drawback of an exact recovery rate is that for real-life datasets, it is impossible to measure the true recovery rate by checking mathematical equivalence since there will not be any accompanying ground truth equation for comparison.
In our paper, we present top-1 approximate recovery rate as our primary metric and include the results for top-1 exact recovery rate in brackets.
Selecting appropriate lifetime population size.
To guard against setting a lifetime population that is too high, we also propose to benchmark against a random method (generate the entire lifetime population in one generation without any genetic operation) and to ensure the metric across all methods is not saturated at the selected population size.
This is done to elicit meaningful conclusions from the results.
Previous works have used a lifetime population size of 2 million (Mundhenk et al., 2021; Petersen et al., 2019;
Kim et al., 2021; Larma et al., 2021), but it is difficult to compare results across the different methods since many equations have near 100% recovery, even for the worst-performing methods.
To this end, we reduce the lifetime population size to 10000 following 2 observations we make.
First, at the new size, the top-1 approximate recovery rate is never saturated across all methods, allowing us to compare the performance of each method across various equations.
Second, we evaluate a fully random search, which is implemented in practice by setting
GP with only 1 generation at full population size, and find that the performance across most of the equations is near 0.
This gives us higher confidence to report that the methods recover equations not by chance.
Datasets with unknown constants are more relevant.
We also recommend testing on datasets which include unknown constants, such as Jin* dataset, since we find that the performance of methods varies drastically with and without unknown constants.
In addition, it is of practical interest to consider datasets with unknown constants since it is common in real-life relations between variables, such as feature scaling (Udrescu & Tegmark, 2020).
For example, we have f = e − θ 2 / 2 / √ 2 π a real-world physics equation from AI-Feynman database (see Appendix Table 7).
[Table]
In our experimental results recorded in Table
[Table]
we observe that between TuringBot and GPLearn,
TuringBot has the poorer performance in Jin* dataset that has unknown constants.
However, TuringBot has the superior performance in Nguyen* dataset that does not contain unknown constants.
A similar observation can be made between GPLearn and DSO-NGGP where the relative performance is reversed depending on the presence of unknown constants.
This phenomenon is due to the difference in the frequency of appearance of constants and the presence and extent of constants optimization in each method.
For datasets with no unknown constants, the more the algorithm utilizes and optimizes constants, the more likely that equations with morphology that are dissimilar to the true equation can become top candidates.
This can be seen as inhibiting the evolutionary process, where the spots for expressions to undergo evolution are instead taken up by these psuedo-equations (expressions that perform well on the training set in terms of MSE but are not close to the ground truth equation in its functional form.).
In the context of real-life datasets, we argue that it is justifiable to assume that unknown constants will appear frequently in naturally occurring processes such as scaling.
The performance of methods on datasets without unknown constants would then be less reflective of the performance on real-life datasets.
We thus recommend that SR experiments favor datasets with unknown constants and assert that results obtained from such datasets are more reflective of real-life application.
3. M ORPHOLOGY OF E XPRESSIONS
Previous works have been done to influence the morphology of candidate expressions using deep learning methods (Petersen et al., 2019; Bladek & Krawiec, 2019; Udrescu & Tegmark, 2020).
However, these methods have never been fully independent, and have instead been an addition to another method or utilize man-made heuristics such as those mentioned in earlier sections.
We also find that these methods can be heavily reliant on the heuristics.
For instance, DSR performance drops sharply with the removal of in-situ constraints and complexity scores, with top-1 approximate recovery rate decreasing to 33% the original value and top-1 exact recovery rate decreasing to 10% the original value.
Furthermore, these heuristics increase the likelihood for the algorithm to be biased towards certain form of expressions, allowing for an implicit data leakage.
Finally, it is an extremely difficult task to form such man-made rules, and it is made even harder when the discovery of such rules needs to be repeated from scratch when the primitive function space is distinctively different, such as when changing from ordinary algebra to boolean algebra.
Thus, we aim to propose a method that is free of such man-made heuristics.
Here, we outline a standalone method of generating candidate expressions that can be used independently.
Terminal walks representation.
Instead of using the prefix representation of expressions as input to a seq2seq model as done in other SR methods, we take inspiration from random walks used in node2vec (Grover & Leskovec, 2016) and generate terminal walks from expression trees to reflect the hierarchical nature of expressions.
A single terminal walk refers to the collection of nodes traversed from the root of the expression tree to either a variable node or a constant node.
These terminal walks will then be treated as sentences.
The benefit of such a method over prefix representation is that the distances between operations in terminal walks are reflective of the distances between operations in expression tree.
On the other hand, in prefix representation, the operations that appear to be faraway may instead be near in the expression tree.
For example, for the equation sin (1 + cos ( x )) , the nested cos is 2 tokens away in both the terminal walk { sin , add , cos , x } and in the expression tree form.
However, it is 3 tokens away in the prefix notation.
[Table]
Building language model to replace heuristics.
We then treat the collection of all terminal walks as a corpus of sentences to train a RNN as commonly practiced in next-word-prediction natural language processing tasks (Barman & Boruah, 2018).
For our paper, we used an embedding layer, a long short-term memory layer and a dense layer sequentially to train a lightweight RNN.
This RNN will be our mathematical language model for our GP algorithm, which we coin as terminal walks mathematical language model (TW-MLM).
Candidate expressions are then generated by randomly selecting an operation from a uniform distribution, then for every incomplete link in the tree, an incomplete terminal walk is generated and fed into the TW-MLM.
The TW-MLM then outputs a probability distribution that is used to select the next node to complete the tree.
This process repeats until the tree has no incomplete links.
Ablation: TW-MLM to improve recovery rates.
In this ablation experiment, we use the baseline
GP algorithm (Koza, 1992) implemented in GPLearn for comparison since the other competitive SR methods impose man-made heuristics which would complicate the insights drawn from experimental results.
For training the TW-MLM, we use the set of 2023 terminal walks generated from
Physics equations (Udrescu & Tegmark, 2020).
These equations are suitable as they contain widely accepted expressions used in real-life scenarios, in contrast to the other SR datasets.
When tokenizing the equations, variables are represented as a single “variable token”.
When TW-MLM generates a new equation for a numerical dataset, this token is replaced by a randomly selected variable.
Throughout this paper, we use a lightweight RNN, comprising of a single embedding layer, a single long short-term memory layer and a single dense layer.
We experiment with replacing the RNN with a transformer, which we find to perform worse.
As seen in Table 2, our experiments show that with just the sole usage of our TW-MLM to generate candidate expressions, we can drastically improve both top-1 exact recovery rate and top-1 approximate recovery rate across all 3 datasets.
The intuition for the improvement of results is that the TW-MLM learns the intrinsic patterns which makes equations human-readable, which promotes the GP algorithm to explore a search space with a high concentration of human-readable equations.
3. A DAPTABILITY IN E VOLUTION
Borrowing the idea of adaptability from the theory of evolution (Bateson, 2017), we argue that a candidate expression close to the ground truth will survive through a multitude of fitness functions, whereas a pseudo-expression may perform deceptively well for one but perform worse for the other fitness measures.
In this context, the challenge for SR is to find suitable secondary fitness functions that measure characteristics beyond the primary fitness function, i.e., faithfulness to empirical values via the MSE.
One must also optimize for both fitness functions and we do this by alternating between the primary and secondary fitness functions.
Additionally, we note that one benefit of the simple closed-form analytical expression found by SR is that it allows the user to apply traditional mathematical tools on the expression, such as derivatives.
In this paper, we explore the derivative as part of the secondary fitness functions, e.g., faithfulness to empirical derivatives.
We describe two natural approaches to doing this, mean squared error of derivatives and mean squared error of difference, and argue that the former is superior to the latter for our application.
Mean Squared Error (MSE) of Derivative (MSEDE).
MSE is commonly used as a fitness and optimisation function, and yields the Maximum Likelihood Estimate (MLE): θ MSE = arg min θ (cid:80)
[Table]
Likewise, we can reward expressions for being faithful to empirical derivatives.
We can compute the MSE of the empirical derivatives from the dataset and candidate expressions.
Consider N pairs of values ( x i , y i ) , where i = 1 , 2 , . .
, N , sorted based in ascending values of x .
We define the empirical derivative as ∆ y i ∆ x i =
y
i +
− y i x
i +1 − x i .
Then, θ MSEDE = arg min θ (cid:80) N − 1 i =1 ( ∆ y i ∆ x i − ∆ˆ y i ∆ x i ) 2 .
Mean Squared Error of Difference (MSEDI).
Here, we develop a new fitness measure derived from a theoretical basis that is consistent with the traditional error modelling framework used in
MSE:
Given measurements y = f ( x ) + C + ϵ , where C is a an arbitrary constant and ϵ ∼ N (0 , σ ) , we aim to find ˆ y =
g θ ( x ) , such that g θ ( x ) is a close approximation to f ( x ) .
We derive the MLE of parameters θ by considering empirical derivatives.
When
[Equation]
We note that the difference between derivatives obtained from dataset and candidate expression follows a Gaussian distribution as well, which are independent to each other under the set of odd-valued or even-valued i .
The total log likelihood across both sets should be similar by symmetry.
We choose to evaluate the total log likelihood across all i instead of picking one set.
By letting σ ′ i = √ 2 σ ∆ x i , the log likelihood of a particular [Equation] y i ∆ x i is, ln
[Equation]
The total log likelihood across all i is thus,
[Equation]
The parameters θ of g θ ( x ) are obtained by maximizing the total log likelihood in (1) to obtain:
[Equation]
MSEDI more relevant for real-world compared to MSEDE.
However, we discover that MSEDE is a potentially harmful fitness function when dealing real-life datasets since it overfits to a selected set of noisy derivative values.
Our experiments that include noise yield that using MSEDE led to no recovery across 100 repetitions of experimentation, while MSEDI was successful.
We observe that random error, ϵ , contributes to large noise in empirical derivatives, especially when ∆ x i is small.
In other words, MSEDE can be viewed as a weighted version of MSEDI, with the weights
being(∆ x i ) 2 .
The MSEDE function thus encourages over-fitting to values which are heavily weighted.
Additionally, MSEDE relies on the value 1∆ x i , which is problematic when ∆ x i is 0, which is common in real-world datasets with duplicated x values.
Thus MSEDI better fulfills the criteria of relevancy to real-life problem (White et al., 2013), providing an additional justification to choose our theoretically derived MSEDI.
Multi-objective optimization using MSEDI and MSE.
As discussed in Section 2, it is difficult to create a general method to optimize for both fitness functions using traditional methods.
For instance, using a weighted combination do not work for some equations as one fitness will dominate, effectively only optimizing for one fitness in those cases.
Instead, using the idea of adaptability, we utilize MSEDI as a secondary test with the intention to filter away pseudo-expressions as described earlier in Section 3.3.
Our experiments show the addition of this adaptability mechanism (using
MSEDI as fitness function) once every 5 generations improves the exact recovery by more than double while maintaining a similar approximate recovery rate.
Ablation:
Escaping local optima with and without MSEDI.
In addition to acting as a secondary fitness function to remove pseudo-expressions, we also find that MSEDI functions by helping GP escape poor local optima.
Among the experiments that do not recover the equation, we save the state of the GP and run it for an additional 5 generations of MSE.
If the top-1 equation remains the same (which occurred for 72.82% of experiments), we rollback the state of the GP and run justgeneration of MSEDI.
The percentage of experiments in which this 1 generation of MSEDI helps to change the top equation are 19.82%, 15.58%, 44.33% for the Nguyen* , R* and Jin* datasets respectively.
Thus, from the results we know that MSEDI helps to escape poor local optima which would have otherwise stagnated.
I NTEGRATED P ROOF -O F -C ONCEPT
Combining the ideas described above, we added TW-MLM and the adaptability mechanism, that alternates MSE and MSEDI as fitness function, to GP with constant optimization and using the primitive function set { add , mul , sub , div , sin , cos , arcsin , log , exp , pow } as discussed in Section 3.1.
The interfacing of each component can be visualized in a simple schematic (Figure 1).
while current generation < max generations do
Fill up population of equations by generating equations from TW-MLM; if current generation % 5 then
Evaluate fitness of equations using
MSEDI else
Evaluate fitness of equations using MSE end
Evolve and select equations based on evolutionary operations in GP (e.g., crossover, subtree mutation, hoist mutation, point mutation, reproduction) end
Algorithm 1 outlines our proof-of-concept.
TW-MLM is trained once on a MLM dataset and utilized to generate candidate expressions at the start of every generation of GP.
GP then evaluates and filters from this set of expressions together with candidates derived from the previous generation through a fitness functions that alternates between generation.
In our method, we use MSE as our base fitness function and switch to MSEDI once every 5 generations (decided based on hyper-parameter tuning on a smaller-sized experiment).
We then conduct a series of experiments to test the proposed approach on both synthetic and real-world datasets .
Synthetic Datasets Experiments.
We compare with the benchmarks methods and random search outlined in Section 2.
We choose synthetic datasets with unknown constants (i.e., Jin* ) and without unknown constants (i.e., Nguyen* and R* ).
Each equation in Table 6 (Appendix) is used to conduct 100 experiments per method.
In
Table 3 and Table 4, we tabulate the performance on the three synthetic datasets (which containequations to be recovered).
Synthetic Datasets Performance Comparison.
Our method shows overall competitive performance with state-of-the-art.
On equations with unknown constants, shown in the Jin* dataset in
Table 3, our method outperforms all other methods by a large margin, in both top-1 approximate recovery rate and top-1 exact recovery rate .
On equations without unknown constants (e
[Table]
g., Nguyen* ), our method outperforms all except DSO-NGGP for almost all of the equations, as shown in Table 4.
Since Jin* included unknown constants that allows for relations that are reflective of real-life
[Table] [Table] [Table] variables such as feature scaling (Udrescu & Tegmark, 2020), it can be argued that the results from
Jin* hold more practical value.
In this sense, our model clearly outperformed all models as seen in
Table 3.
Real-World Datasets (SRBench) Evaluation.
We then evaluated our method against DSO-NGGP on 6 real-world datasets from SRBench (La Cava et al., 2021), with the results tabulated in Table 5.
Unlike traditional SR datasets, real-world datasets do not have an accompanying ground truth equation.
We present the percentage of experiments in which our method outperformed DSO-NGGP in terms of r-squared value.
Our method consistently outperforms DSO-NGGP, further corroborating our previous results and strengthening the claim that SR datasets with unknown constants are more reflective of real-world dataset performance.
Intuition and Ablation.
The function of TW-MLM in our method is to implicitly learn the rules and constraints about the morphology of human-readable equations that are previously developed based on human judgment.
This increases the likelihood of our method to venture into the search space containing expressions that are consistent with existing widely-accepted human-readable equations.
In this sense, TW-MLM acts as a guide in the vast search space of possible equations.
Though manmade heuristics such as rules and constraints exist to fulfill these roles, it is difficult to find, difficult to express and not easily transferable to new problems.
On the other hand, TW-MLM finds these rules and constraints implicitly and the method can be easily transferred to a new problem.
The adaptability mechanism (by alternating between MSE and MSEDI fitness function) then synergizes with TW-MLM and GP by increasing the difficulty for pseudo-equations to survive each generation, as shown in the ablation study in Section 3.2.
We also observed throught the ablation study in
Section 3.3 how MSEDI helps to escape from poor optima points in GP.
These mechanisms added to GP allow us to create a model that performs competitively with state-of-the-art.
R EFLECTIONS
Summary.
In this paper, we demonstrate an efficient proof-of-concept that incorporates 2 new independent mechanisms into genetic programming-based SR: (a) Terminal walks mathematical language model (TW-MLM) and (b) Adaptability via alternating fitness functions (i.e., between MSE and our novel theoretically justified metric called MSEDI).
Through these simple modifications, we are able to obtain competitive results with respect to a diverse range of methods, outperforming all, including state-of-the-art, when datasets with unknown constants are involved.
We then reproduce this outperformance on real-world datasets.
We also state and justify the conventions we use that can promote consistency, comparability and relevance to real-world problems.
Ultimately, we hope that
SR can demonstrate more competitive real-world results among other machine learning methods, given it’s natural advantage in both interpretability and explainability.
Limitations.
Our method performs much better on equations with unknown constants ( Jin* ), in contrast to equations without unknown constants ( Nguyen* ).
The reason behind this observation can be attributed to the difference in frequency of unknown constants during the generation of candidate expressions.
Our TW-MLM is trained on 100 AI-Feynman equations which has the frequent use of constants.
This means that the TW-MLM has to learn to include constants into candidate expressions frequently, increasing the chances of over-optimization of constants in candidate expressions.
Future Work.
(i)
The performance of our method on datasets without unknown constants may suffer due to over-optimization of constants in candidate expressions.
To reduce the chances of this, SR can alternate between randomly generating constants and optimizing for constants.
This is similar to the way we alternate between fitness functions in this paper.
(ii)
The search space of
SR grows rapidly with an increasing number of variables, making real-world datasets with many variables a difficult problem for SR.
We are currently working on an iterative greedy approach to multi-variable SR, dealing with one additional variable per SR-run to address the issues that SR faces with time complexity.
A A PPENDIX
[Table] [Table]
rF sin θ
I.18.
L = mrv sin θ
I.24.
E = 14 m (cid:0) ω
2 + ω 20 (cid:1) x
I.25.
V e = qC
I.26.
θ 1 = arcsin ( n sin θ 2 )
I.27.
f f = 1 1 d 1 + nd
I.29.
k = ωc
I.29.
x = (cid:112) x 21 + x 22 − 2 x 1 x 2 cos ( θ 1 − θ 2 )
I.30.
I ∗ =
I ∗ 0 sin 2 ( nθ/ 2) sin 2 ( θ/ 2)
I.30.
θ = arcsin (cid:0) λnd (cid:1)
I.32.
P = q 2 a 2 6 πϵc
I.32.
P = (cid:16) 12 ϵcE 2 f (cid:17) (cid:0) 8 πr 2 / 3 (cid:1) (cid:16) ω
4 / (cid:0) ω 2 − ω
20 (cid:1) 2 (cid:17)
I.34.
ω = qvBp
I.34.
ω = ω 0 1 − v/c
I.34.
ω
= 1+ v/c √ 1 − v 2 /c 2 ω
I.34.
E = ℏ ω
I.37.
I ∗ =
I
+ I 2 + 2 √
I 1 I 2 cos δ
I.38.
r = 4 πϵ ℏ 2 mq
I.39.
E = 32 p F V
I.39.
E = 1 γ − 1 p F V
I.39.
P F = nk b T V
I.40.
n = n 0 e − mgx k b T
[Table] [Table]
ρ c = 1 √ 1 − v 2 /c
II.13.
j = ρ c 0 v √ 1 − v 2 /c
II.15.
E = − µ M B cos θ
II.15.
E = − p d E f cos θ
II.21.
V e = q 4 πϵr (1 − v/c )
II.24.
k = (cid:113) ω
2 c 2 − π 2 d
II.27.
F E = ϵcE 2 f
II.27.
E den =
ϵE 2 f
II.34.2a
I = qv
2 πr
II.34.
µ M = qvr
II.34.
ω =
g − qB 2 m
II.34.29a µ M = qh 4 πm
II.34.29b
E = g − µ M BJ z ℏ
II.35.
n = n 0 exp( µ m B/ ( k b T ))
+exp( − µ m B/ ( k b T ))
II.35.
M = n ρ µ M tanh (cid:16) µ M B k b T (cid:17)
II.36.
f
= µ m B k b T + µ m αM ϵc 2 k b T
II.37.
E = µ M (1 + χ ) B
II.38.
F = Y Axd
II.38.
µ S =
Y 2(1+ σ )
III.4.
n = e ℏ ω kbT −
[Table]