A Unified Framework for Deep Symbolic Regression
Anonymous Author(s)
AffiliationAddress email
Abstract
The last few years have witnessed a surge in methods for symbolic regression,from advances in traditional evolutionary methods to novel deep learning-basedmethods.
Individual works typically focus on advancing the state-of-the-art forone particular class of solution methods, and there have been few attempts toinvestigate the benefits of hybridizing or integrating multiple methods.
In thiswork, we identify five standalone symbolic regression methods whose individualcapabilities—spanning neural-guided search, genetic programming, and linearregression—provide broad coverage of the overall space of existing approaches,and we propose a strategy to hybridize them into a single modular, unified symbolic regression framework.
Based on empirical evaluation using SRBench, anew community tool for benchmarking symbolic regression methods, our unifiedframework achieves state-of-the-art performance in its ability to (1) symbolicallyrecover analytical expressions, (2) fit datasets with high accuracy, and (3) balanceaccuracy-complexity trade-offs, across 252 ground-truth and black-box benchmarkproblems, in both noiseless settings and across various noise levels.
Finally, we provide practical use case-based guidance for constructing hybrid symbolic regressionalgorithms, supported by extensive, combinatorial ablation studies.
Introduction
Symbolic regression (SR) aims to identify a tractable mathematical expression to best fit a dataset.
Specifically, given the set of pairs { (cid:0) x i =
( x i, 1 , . . .
, x i,d ) , y i (cid:1) } ni =1 , SR seeks a mathematical functionf : R d → R such that the residual ∥ f ( X ) − y ∥ 2 is minimized, where the matrix X ∈ R n × d hasentries X i,j = x i,j , the vector y ∈ R n has entries y
i , and we assume point-wise application of realvalued functions over vectors and matrices, i.e., f ( X ) =
( f ( x i ))
n i =1 ∈ R n (we keep this conventionthroughout the paper for convenience).
[Equation]
[Table]
Solve
[Table] [Table]
+uxXx+C0iIK73EUQytgvVB0BWdopLZ95CE8oArSu0gOgWZewLDPmUwrWVu07YJTdCagi8SdkQKZodK2v7xOxJMAQuSSad10nRhbKVMouIQs7yUaYsYHrAdNQ0MWgG6lky8yemKUDu1GylSIdKL+nkhZoPUo8E3n+Eg9743F/7xmgt3LVirCOEEI+XRRN5EUIzqOhHaEAo5yZAjjSphbKe8zxTia4PImBHf+5UVSOyu658XSbalQvprFkSOH5JicEpdckDK5IRVSJZw8kmfySt6sJ+vFerc+pq1L1mzmgPyB9fkDeA2ZMw==</latexit> Solve
P i <latexit sha1_base64="5rhwB217eyFohz9OpeQcIUADdQs=">AAACBXicbVA9SwNBEN3zM8avqKUWi0GwCncS1DJoYxnRfEASwt5mkizZvTt254LhuMbGv2JjoYit/8HOf+Pmo9DEBwOP92aYmedHUhh03W9naXlldW09s5Hd3Nre2c3t7VdNGGsOFR7KUNd9ZkCKACooUEI90sCUL6HmD67Hfm0I2ogwuMdRBC3FeoHoCs7QSu3cURPhAbVK7kI5BJo2FcM+ZzIpp23VzuXdgjsBXSTejOTJDOV27qvZCXmsIEAumTENz42wlTCNgktIs83YQMT4gPWgYWnAFJhWMvkipSdW6dBuqG0FSCfq74mEKWNGyred4yPNvDcW//MaMXYvW4kIohgh4NNF3VhSDOk4EtoRGjjKkSWMa2FvpbzPNONog8vaELz5lxdJ9azgnReKt8V86WoWR4YckmNySjxyQUrkhpRJhXDySJ7JK3lznpwX5935mLYuObOZA/IHzucPfh2ZNw==</latexit> Solve
P m <latexit sha1_base64="jFyyjNrYHkgFkiRbmN/bX65uemI=">AAAB7HicbVBNS8NAFHypX7V+VT16WSyCp5KIqMeiF48VTFtoQ9lsN+3SzSbsvggl9Dd48aCIV3+QN/+N2zYHbR1YGGbesO9NmEph0HW/ndLa+sbmVnm7srO7t39QPTxqmSTTjPsskYnuhNRwKRT3UaDknVRzGoeSt8Px3cxvP3FtRKIecZLyIKZDJSLBKFrJ7w0SNP1qza27c5BV4hWkBgWa/eqXzbEs5gqZpMZ0PTfFIKcaBZN8WullhqeUjemQdy1VNOYmyOfLTsmZVQYkSrR9Cslc/Z3IaWzMJA7tZExxZJa9mfif180wuglyodIMuWKLj6JMEkzI7HIyEJozlBNLKNPC7krYiGrK0PZTsSV4yyevktZF3buqXz5c1hq3RR1lOIFTOAcPrqEB99AEHxgIeIZXeHOU8+K8Ox+L0ZJTZI7hD5zPH/RWjss=</latexit> ...
<latexit sha1_base64="zYA27x8OG2VzuukSUYFEBJW/jwk=">AAACynicbVHLatwwFNU4faTT1yRZZmM6BFIogx3SpsvQbrroIoVOEpiZDrJ87YiRJSNdJzFCi0L
[Table]
[Equation]
[Table]
RNN [ ⌧ GP
[Table]
+S9mXNu6rVH+qVxm0eRxFO4BSq4ME1NOAemtACAhN4hld4c2LnxXl3PhatBSefOYY/cD5/AOrWjqc=</latexit> ( X, y )<latexit sha1_base64="6MwE3OiTgq8CZRxn7pm7IihbgUs=">AAAB7nicbVBNSwMxEJ2tX7V+VT16CRahgpRdKeqx6MVjBfsB7VKyabYNTbJLkhWWpT/CiwdFvPp7vPlvTNs9aOuDgcd7M8zMC2LOtHHdb6ewtr6xuVXcLu3s7u0flA+P2jpKFKEtEvFIdQOsKWeStgwznHZjRbEIOO0Ek7uZ33miSrNIPpo0pr7AI8lCRrCxUqfavUjPB2xQrrg1dw60SrycVCBHc1D+6g8jkggqDeFY657nxsbPsDKMcDot9RNNY0wmeER7lkosqPaz+blTdGaVIQojZUsaNFd/T2RYaJ2KwHYKbMZ62ZuJ/3m9xIQ3fsZknBgqyWJRmHBkIjT7HQ2ZosTw1BJMFLO3IjLGChNjEyrZELzll1dJ+7LmXdXqD/VK4zaPowgncApV8OAaGnAPTWgBgQk8wyu8ObHz4rw7H4vWgpPPHMMfOJ8/P8WO3w==</latexit> ( X, y )
i <latexit sha1_base64="wn1Bga/NikvB5dfqkdlmgY6qcYs=">AAAC0HicbVFNb9QwEPWGj5blawtHLoFVJQ5olVRAOVZw4VgQ21baXa1sZ5Ja69iRPSm7sizECYk
[Table]
[Figure]
Since the introduction of SR asan application of genetic programming (GP) (Koza, 1994), many different classes of methods—especiallymachine learning-based methods inrecent years—have been proposed.
These methods include recursive problem simplification (Udrescu et al.
,2020), neural-guided search (Petersenet al., 2021a), large scale encodertraining for generalization (Biggioet al., 2021), sparse linear representation using basis functions (Bruntonet al., 2016), and combination of evo- lutionary search with learning (Mundhenk et al., 2021).
Each of these independent advances exploitsdifferent information (e.g., graph modularity or constraints) to offer key capabilities for variousaspects of the SR problem (e.g., search or generalization), but the specificity of each work resulted indesign choices that pose limitations in other facets of the problem.
This motivates an investigationof whether disparate methods are complementary to one another and how they can be effectivelycombined.
In this paper, we combine the five aforementioned methods into a unified SR frameworkand show that they are indeed complementary.
Based on the SRBench pipeline for benchmarking SR algorithms ( https://github.com/cavalab/srbench ), our unified deep symbolic regression (uDSR) framework achieves a new state-of-the-artin SR: for ground-truth problems, uDSR ranks highest in both symbolic-based and accuracy-basedsolution rates for both noiseless settings and for multiple noise levels; for black-box problems, uDSRfalls on the accuracy-complexity Pareto frontier alongside three other non-dominated algorithms.
Our main contributions in this work include: (1) a modular, unified framework for SR that integratesfive disparate SR methods (illustrated in Figure 1); (2) a novel LINEAR token that identifies sparsecoefficients of a linear combination of basis functions and can be used by almost any SR algorithm;(3) achieving new state-of-the-art for SR on SRBench in its metrics for symbolic recovery, solutionaccuracy, and accuracy vs complexity trade-offs; and (4) practical recommendations for designing
SR algorithms, driven by extensive and combinatorial ablation studies that show the complementarybenefits of these previously disjoint SR methods.
Background
We describe each of the five methods our framework unifies.
For each method, we focus on a keycapability and a key limitation 1 , summarized in Table 1, which motivates our unification that capturesall capabilities and mitigates all limitations via the inherent synergy among methods.
We encouragereaders to read the original works for additional background.
AI Feynman.
AI Feynman 2.0 (AIF) exploits knowledge of physics and the given training data( X, y ) , by identifying simplifying properties (e.g. multiplicative separability) of the underlyingfunctional form (Udrescu et al., 2020).
Applying a simplification yields sub-problems (or one subproblem) of lower dimensionality.
Simplifications are applied recursively until all sub-problems(i.e.
leaf nodes of the recursion tree) are 1-dimensional.
For each sub-problem, the paper employs acombination of brute force search plus polynomial fit (BF/PF).
Finally, solutions to the sub-problemsare then combined back into a final solution to the original problem.
While AIF positions itselfas a standalone SR algorithm, it is more appropriately understood as a SR problem simplificationalgorithm , as pointed out in prior work (Petersen et al., 2021a; Biggio et al., 2021).
From this generalperspective, the use of BF/PF is simply a design choice, orthogonal to the core methodology, andmay be enhanced by integration with more advanced search methods for solving sub-problems.
Deep symbolic regression.
Deep symbolic regression (DSR) leverages the effectiveness of deepreinforcement learning for combinatorial search in SR (Petersen et al., 2021a).
DSR employs a neuralnetwork controller with parameters θ to represent a distribution p ( τ | θ ) over mathematical expressionsτ =
[ τ 1 , . .
, τ T ] , represented by the sequence of “tokens” τ i (mathematical operators, input variables,or constants) in the pre-order (depth-first then left-to-right) traversal of the corresponding expressiontree (see Koza (1994) for details).
For instance, τ = [ + , SIN , x 1 , EXP , x 2 ] corresponds to expressionf ( x ) = sin( x 1 ) + exp( x 2 ) .
Expressions are sampled from the controller autoregressively, i.e.p ( τ | θ ) =
(cid:81)
Ti =1 p ( τ i | τ 1: i − 1 , θ ) , and evaluated based on fitness R ( τ, ( X, y )) to the ( X, y ) dataset.
The controller is then trained to maximize the reinforcement learning objective:
[Equation]
1) using a risk-seeking policy gradient that aims to maximize best-case performance.
Autoregressivesampling affords the ability to easily incorporate prior knowledge by directly modifying likelihoodsof individual tokens p ( τ i | τ 1: i − 1 , θ ) in situ , for example by precluding expressions with nested 85 trigonometric operators.
This dramatically reduces the search space, which has been shown to greatly improve the ability to recover ground-truth expressions (Petersen et al., 2021b).
Whereas DSRfocuses on discrete search and pruning the combinatorial search space, it does not exploit the ( X, y )data.
In fact, the ( X, y ) data itself only shows up in the reward computation in (1), which is a weak,aggregate learning signal.
Thus, DSR trains from scratch for each new SR problem and cannotgeneralize to new problems.
Neural Symbolic Regression that Scales (NeSymReS).
NeSymReS (Biggio et al., 2021) tackles theproblem of generalization in SR: it seeks a single model that, once trained, produces a distributionover solutions to any SR problem.
To accomplish this, the architecture uses a set transformer (Leeet al., 2019) component to learn an encoding of the ( X, y ) data.
A second transformer network thenmaps the encoding to a distribution over expressions using autoregressive sampling.
This model istrained end-to-end via supervised learning, using a large dataset of SR problems (input data ( X, y )and corresponding ground truth label τ ) produced by a programmatic expression generator (Lampleand Charton, 2019).
Given ( X, y ) at test time, beam search is used to find high-likelihood expressions.
Treating test time as an inference problem reinforces the authors’ claim that pre-training alone issufficient to solve many SR problems.
However, the lack of fine-tuning and the use of beam searchmeans that NeSymReS may not be optimized for each test problem.
Genetic programming.
Genetic programming (GP) refers to a well-established class of evolutionaryalgorithms commonly applied to SR in which a set or “population” of mathematical expressionsis maintained and “evolved” over time using genetic operators such as mutation, crossover, andselection.
We refer the reader to Langdon and Poli (2013) for an overview of GP.
In this work, weconsider only standard GP (as defined in Chapter 7 of Bäck et al.
; however, our integration of
GP into the unified framework applies to virtually any GP method.
GP excels at exploring the searchspace to find complex expressions, but does not employ a neural network or other parameterizedmodel to learn over time.
This limitation was addressed in Mundhenk et al.
(2021), where GP wascombined with a learned expression generator, but not in concert with other methods considered here.
SINDy.
SINDy is a simple, computationally expedient, and popular method for SR, particularlyfor learning systems of differential equations (Brunton et al., 2016).
SINDy learns functions ofthe form g = (cid:80) ki
=1 β i ϕ i
=
Φ β T , where Φ :=
( ϕ 1 , . . .
, ϕ k ) comprise a library of k user-definedbasis functions, each ϕ
i : R n → R , and β = ( β 1 , . . . , β k ) is a vector of learned scalar coefficients.
While SINDy employs LASSO (Tibshirani, 1996) to learn the values of β i , we consider this a designchoice for any arbitrary sparse linear regression method.
While effective at quickly learning sparseexpressions, restricting to a linear combination of pre-defined basis functions results in a massivelyreduced search space, precluding such functional forms as rationals and compositions.
In our unifiedframework, we abstract the entire process of learning a linear model into one single discrete token,thereby restoring the nonlinear, combinatorial search space while retaining the ability to quickly learnsparse linear sub-components.
[Table]
Quickly learns
[Table]
coefficients of a
[Table]
combination
[Table]
basis functions
Operates on a dramatically reduced search space.
Related Work
In addition to AIF, DSR, NeSymReS, GP, and SINDy described above, we describe several otherrelated SR methods that are not part of our unified framework.
EQL ÷ presents a unique deeplearning framework for SR that employs neural networks whose activation functions are elementaryoperators, enabling end-to-end differentiable training (Sahoo et al., 2018).
GrammarVAE encodes from and decodes to parse trees that explicitly enforces syntactic correctness of generated expressions(Kusner et al., 2017).
As end-to-end architectures, EQL ÷ and GrammarVAE would not facilitatestraightforward integration of any of the five methods except AIF.
The idea of pre-training networksfor use in neural-guided search, used by NeSymReS and our unified framework, is inspired by Belloet al.
(2017), who apply a similar method to the traveling salesman problem.
Previous work on hybrid approaches to SR mainly focus on improving a single aspect of one approach,such as alleviating the challenge of high dimensional data by extracting features for GP (Icke and
Bongard, 2013), or improving the starting population of GP iterations (Mundhenk et al., 2021).
Often,the motivation for combining separate methods comes from specific problem settings, such as thecase of datasets with missing values (Al-Helali et al., 2018) or noisy high-dimensional measurementsin systems with known physical constraints (Reinbold et al., 2021).
In contrast, the unification in ourwork serves to enhance multiple general capabilities of SR, such as generalization, combinatorialsearch and optimization, and single-token abstraction of function spaces.
Our empirical analysis heavily leverages SRBench, a reproducible and open-source platform forbenchmarking new SR methods against 14 established methods on accuracy and complexity for122 real-world datasets and exact symbolic correctness for 130 ground-truth problems with varyingnoise levels (La Cava et al., 2021).
SRBench has seen rapid adoption by recent work that use it toevaluate improvements in accuracy, exact solution rate, and solution complexity (Kamienny et al.
,2022; Virgolin and Bosman, 2022; Zhang et al., 2022).
Lastly, we note a similarity between this work and Rainbow, a work that combines several deepreinforcement learning methods into a unified algorithm and performs extensive ablation studies(Hessel et al., 2018).
Analogous to Rainbow’s use of Deep Q Networks (DQN) as a base frameworkinto which it integrates several improvements, this work uses DSR as a base framework into whichwe integrate AIF, NeSymReS, GP, and SINDy.
Methods
Our overall unification strategy is to employ individual methods as connected but non-overlappingmodules in an algorithmic framework that leverages the key capabilities of each method.
This isvisualized in Figure 1.
In an initial offline stage, pre-training a parametric controller model on alarge dataset facilitates generalization to test problem instances (NeSymReS).
In the online stage,given a specific problem instance, a recursive decomposition module (AIF) exploits modularity ofmathematical expressions to produce sub-problems of lower dimensionality for the main trunk ofuDSR.
Within the trunk, an RNN continually learns over many iterations (DSR) and provides goodcandidates to seed the starting population for genetic programming (GP); high fitness populationsproduced by GP are combined with elite candidates for controller parameter updates via risk-seekingpolicy gradient.
Finally, the RNN is permitted the use of a powerful symbolic token that encapsulatesan entire space spanned by basis functions (SINDy).
Below we describe the integration of each module in detail.
Algorithm 1 provides high-levelpseudocode for uDSR.
More detailed pseudocode is available in Appendix A Algorithm 2.
[Table]
✓
[Table]
✓
[Table]
✓ ✓
[Table]
✓
[Table]
✓ ✓ ✓ ✓ ✓ ✓ ✓
Beginning with DSR.
For DSR,we begin with the method proposed by Petersen et al.
(2021a),including two improvements toexploration (soft length priorand hierarchical entropy regularizer) introduced in followup works (Landajuela et al.
,2021a,b).
DSR includes a number of in situ priors and constraints to the search space,which help bias and/or prune thesearch, and have been shown togreatly improve performance (Landajuela et al., 2021b).
We provide a complete list of DSR priorsand constraints in Appendix B, including several new constraints required to unify all five methods
(described below).
DSR also includes a CONST token that represents an arbitrary floating-pointconstant, whose value is optimized using a nonlinear optimizer, e.g. L-BFGS-B (Zhu et al., 1997).
Integrating AIF via recursive simplifications.
Our key insight to integrate AIF (Udrescu et al.
,2020) is that its recursive problem simplification steps can be viewed as orthogonal to the underlying
SR algorithm.
While AIF resorts to BF/PF, we propose using a hybrid of the other four integratedalgorithms (which we continue to describe in subsequent sections).
To integrate AIF, we consider thestarting SR problem as P 1 , a d -dimensional SR problem.
One application of AIF’s simplification stepproduces either one sub-problem P 2 , with dimensionality d 2 < d , or two sub-problems, P 2 and P 3 , 189 with dimensionalities d 2 and d 3 such that d 2 + d 3 = d , as well as metadata M containing information 190 required to recombine sub-solutions back to to the full solution of
P 1 .
Recursive simplifications 191 are applied until sub-problems at the leaf nodes of the recursion tree are each 1-dimensional.
Foreach sub-problem P i (including the starting problem, P 1 ), we apply our hybrid SR algorithm to itscorresponding dataset, ( X, y ) i .
Finally, using the metadata from each simplification step, solutionsfrom all sub-problems are recombined to form a single, final solution to the original problem, P 1 .
Integrating SINDy via LINEAR token.
Our design goal for integrating SINDy (Brunton et al., 2016)is to retain the ability to quickly learn linear sub-components of an expression, without sacrificing thecombinatorial search space.
That is, while SINDy finds expressions of the form g ( x ) =
Φ( x ) β T , weseek to search the more generic space f
( x ) =
F ( x, g ( x )) , where F is a nonlinear function subject tocertain constraints (see below), while still using linear regression to determine the coefficients of g ( x ) .
We accomplish this by proposing to represent g ( x ) as a special type of terminal token, which we call
LINEAR .
LINEAR serves as a placeholder for a SINDy-like solution, Φ( x ) β T , where the values ofthe coefficients β are determined only when a traversal has terminated.
Thus, we abstract the entireprocess of learning a linear model into a single discrete token.
In our approach, the function F ( x, w )must admit an expression tree with a single argument token for w and there must exist G ( x, v ) suchthat y = F ( x, G ( x, y )) , ∀ x, y ∈ R d × R .
Given an expression f ( x ) =
F ( x, Φ( x ) β T ) , with F as described above, let f (Φ β T ) denote theview of f as a function of β only.
Naively, one could solve a nonlinear optimization problem todetermine the values of β , just as DSR and NeSymReS optimize CONST tokens.
However, nonlinearoptimization is computationally expensive, scales poorly with more coefficients, and is challengingto induce sparsity.
Instead, we consider the following convex optimization problem, for which thereare efficient sparse linear solvers:
[Equation]
We attain this formulation by solving the equation y = F ( X, LINEAR ) for LINEAR , which gives
LINEAR =
G ( X, y ) .
Since this requires taking the inverse of all ancestor operators of LINEAR —binary operators are reduced to unary operators by lexical closure, elaborated more programmaticallybelow—we use f − 1 ( y ) :
=
G ( X, y ) to denote the view of G as a function of y only.
Equation (2) exhibits an attractive property that enables LINEAR to exactly recover non-linearexpressions with linear sub-components.
When LINEAR is chosen at the correct position in a traversal,in the sense that the true computational graph contains a linear combination of basis functions in thecorresponding subgraph, we have that f − 1 ( y ) exactly equals that linear combination.
For example,given ( X, y ) data generated from a ground-truth expression f ⋆ ( x ) = sin( x 1 )+exp(1 .
2 x 31 +3 .
4 x 1 x 2 ) ,and a basis containing all monomials up to degree 3, the traversal τ =
[ + , SIN , x 1 , EXP , LINEAR ]yields f − 1 ( y )
=
log ( y − sin( x 1 )) , and the corresponding solution to (2) exactly recovers f ⋆ .
Programmatically, we solve (2) using the expression tree for τ and the following procedure (illustratedin Appendix C Figure 5; pseudocode is provided in Appendix A Algorithm 2): First, partially executethe tree by evaluating all nodes that are not direct ancestors of LINEAR .
Second, redefine binaryoperators as unary operators by fixing (or “binding”)
the one pre-computed argument (this is calledpartial application —a type of lexical closure —in computer science).
Third, successively apply theroot node’s inverse operation on the y data until LINEAR is the only token in the tree, resulting inf − 1 ( y ) .
Finally, perform linear fit on (2) using the transformed data, ( X, f − 1 ( y )) .
This procedure for abstracting a linear model into a single operator exhibits several complications:(1) it only works if there is exactly one LINEAR token, (2) it does not work in the presence of otherplaceholder tokens, e.g. the CONST token used in DSR, and (3) it does not work if LINEAR has non-invertible ancestors (e.g. SIN ), as y = F ( X, LINEAR ) might not have a unique solution on
LINEAR .
Fortunately, seamlessly incorporating these restrictions into the search space is preciselywhat the DSR algorithm excels at.
To this end, we enable LINEAR into our framework by introducingthree constraints to the search space (further detailed in Appendix B): (1) there can be at most one
LINEAR token; (2) the LINEAR token and CONST token are mutually exclusive—one cannot appear ifthe other is present; and (3) the LINEAR token cannot be the descendant of a non-invertible unarytoken, e.g. SIN .
In this work, we consider the set of basis functions Φ given by the monomials ofdegree up to D and solve (2) using a sparsity-inducing least square method (see Appendix C fordetails).
Lastly, we observe that the original SINDy method is restored when F ( x, w ) = w .
In that case, werecover the function f ( x ) =
Φ( x ) β T with trivial traversal τ =
[ LINEAR ] .
Thus, our method expandsthe search space of the original SINDy method by allowing nonlinear function compositions.
Forinstance, F ( x, w ) = 1 w leads to f ( x ) =
1 Φ( x ) β T , which is outside SINDy’s search space.
Integrating GP via population seeding.
To our knowledge, hybridization of DSR and GP is theonly combination of the five methods that has been explored to date.
To summarize, we hybridize
DSR with GP by abstracting the GP algorithm as an inner optimization loop within DSR.
Each time
DSR samples a batch of expressions, we use this batch as the starting population for an inner-loop
GP algorithm.
The top samples at the end of GP are combined with the DSR batch for training theneural network.
This process leverages GP’s ability to rapidly explore the search space, while using
DSR’s persistent neural network to mitigate the disadvantages of GP’s non-parametric nature.
Integrating NeSymReS via large-scale pre-training.
We integrate NeSymReS by merging the
RNN used in DSR with a set transformer in an encoder-decoder fashion.
The set transformer encodesa given dataset ( X, y ) into a latent representation h 0 .
This representation is passed as initial state forthe RNN, which decodes it into an expression τ .
The resulting architecture models the distributionp ( τ | θ, ψ, ( X, y )) , where θ and ψ are the parameters of the RNN and set transformer, respectively.
We pre-train the model end-to-end on many SR problems, using ground truth expressions, resultingin parameters θ ⋆ , ψ ⋆ .
Given a new dataset ( X ′ , y ′ ) , we propose to search for a symbolic expressionby fine-tuning p ( τ | θ, ψ ⋆ , ( X ′ , y ′ )) using reinforcement learning, with initial conditions θ 0 = θ ⋆ .
The pre-training phase requires a large dataset comprising SR problems, each defined by a set ofpoints ( X, y ) along with true expression τ .
To scale to arbitrary dataset size, we seek a probleminstance generator p G ( τ, ( X, y ))
=
p G (( X, y ) | τ ) p G ( τ ) , from which we can sample expressions τand corresponding ( X, y ) data.
Full details of the sampling process used by p G are provided in
Appendix D. Briefly, we leverage the fact that the distribution induced by DSR includes a prior(Petersen et al., 2021b).
To generate expressions, p G ( τ )
= p ( τ | θ = 0 ) directly samples from thisprior, ignoring emissions from the RNN.
Thus, the algorithm bootstraps or “imagines” expressionsthat it then uses for training data, rather than relying on an external expression generator as in
NeSymReS. After sampling τ , p G (( X, y ) | τ ) is obtained following the strategy in Biggio et al.
[Figure]
In this work, we explore two different objectives for model pre-training.
First, following Biggio et al.
(2021), we consider the supervised learning objective:
[Equation]
During fine-tuning, ground-truth labels will not be available.
To mitigate an objective mismatch between pre-training and fine-tuning objectives, we alternatively explore pre-training à la reinforcementlearning Bello et al.
Specifically, we consider:
[Equation]
In practice, this objective is maximized using a novel formulation of the risk-seeking policy gradientfor multi-task learning (see Appendix D for details).
Experiments
Pre-training.
We pre-train four models:
two trained using SL and two trained using RL, each withand without the LINEAR token.
Details of the pre-training setup can be found in Appendix D.
Experimental setup using SRBench.
We empirically assess uDSR using SRBench (La Cava et al.,2021), an open-source and reproducible pipeline for benchmarking SR algorithms.
SRBench features130 problems with hidden ground-truth analytic solutions and 122 real-world datasets with no knownanalytic model (“black-box” problems) from the PMLB database (Olson et al., 2017).
By runninguDSR through the SRBench pipeline, we enable direct comparison to its curated results of 14contemporary SR methods.
The 14 baselines are described in Appendix E.
For all experiments, we use a minimal set of tokens: + , − , × , ÷ , SIN , COS , EXP , LOG , SQRT , 1.0,
CONST , and (except for the appropriate ablations) LINEAR .
For simplicity, our choice of Φ (basisfunctions for LINEAR ) includes only polynomial terms up to degree 3.
Hyperparameters are listedin Appendix F Table 3.
When possible, we reused the published hyperparameters from the fiveintegrated methods.
We ran uDSR via SRBench on the 252 datasets with 10 random seeds each.
Each run was performedfor up to 2,000,000 expression evaluations per sub-problem, or until a 24 hr walltime limit wasreached.
For ground-truth problems, SRBench computes two metrics, each averaged across the 130problems: (1) symbolic solution rate, based on symbolic equivalence using SymPy (Meurer et al.,2017) and (2) accuracy solution rate, defined as achieving an R 2 > 0 .
999 on held-out test data.
Forblack-box problems, SRBench reports accuracy ( R 2 on held-out test data) vs complexity (number ofnodes in the SymPy-parsed expression tree).
[Figure] [Figure]
MRGP
NeSymReS
Operon
RandomForest
SBP
[Figure]
SINDy
[Figure]
Performance results.
Figure 2 showsthat uDSR outperforms all other 14 benchmarked methods both in symbolic solution (by a large margin) and accuracy solution rates.
Figure 3 shows that uDSR fallson the Pareto frontier, alongside Operon(Kommenda et al., 2020) (at higher accuracy and complexity), and DSR and
SINDy (at lower accuracy and complexity).
Notably, the previously published SRBench results had no clear winner acrossthe three categories of (1) symbolic solution, (2) accuracy solution, and (3) Paretoefficiency.
The top performers for symbolic solution rate (namely, AIF) differed from those for accuracy solutionrate (namely, Operon and MRGP).
Only
Operon appeared as a top performer intwo categories.
However, in this work,we see that uDSR is highest in all threecategories.
Ablations and Discussion
Ablations.
Since uDSR integrates many disparate components, it is critical to assess the relativecontributions of each component in various contexts.
To identify synergies, anti-synergies, anddiminishing returns among the integrated methods, we perform a large-scale study of all combinationsof the five integrated methods.
(Standard “leave-one-out” style ablations in which we begin with thefull uDSR and ablate one component at a time do not provide this full picture.)
There are 48 totalcombinations: AIF ∈ [on, off] × GP ∈ [on, off] × LINEAR ∈ [on, off] × pre-training ∈ [SL, RL, 325 off] × DSR ∈ [on, off].
We define ablating each components as follows:
For AIF, “off” means thatwe do not perform recursive problem-simplification; only the root problem is considered.
For GP,“off” means we do not perform inner-loop GP algorithms between batches.
For LINEAR , “off” meanswe exclude the LINEAR token from the library.
For pre-training, SL and RL refer to whether we usethe model pre-trained using SL or RL; “off” means we do not use any pre-trained model, and thecontroller architecture does not include the set transformer component.
For DSR, “off” means thelearning rate for the neural network is set to zero; notably, the controller is still used for sampling,including all priors and constraints.
For each ablation, we run on the 130 ground-truth SRBenchproblems with 10 random seeds each ( 48 × 130 × 10 = 62 , 400 total runs), using a maximum of500,000 expression evaluations per sub-problem.
Figure 4 shows the symbolic solution rate and accuracy solution rate computed by SRBench, withlabels for select ablations we refer to in the subsequent discussion.
Generally, performance increaseswith the number of enabled components, N .
For a fixed N , GP and SINDy tend to yield the largestmarginal performance gains.
Refer to Appendix G Figure 9 for a visualization of the marginalperformance gains yielded by each component.
Recommendations.
The full uDSR algorithm, using all components and SL pre-training, achievesthe best symbolic and accuracy success on SRBench.
Nonetheless, we believe the right choice of
SR algorithm should be guided by use case.
Thus, based on the extensive ablation studies presentedabove, we provide our recommendations for the five integrated methods.
AIF’s recursive simplifications have the attractive property of providing strict improvement over thebase SR algorithm, since the root sub-problem in the sub-problem tree is simply the original problem;sub-problems further down the tree only have the ability to improve the end result.
This propertycomes at the expense of incurring a roughly m -fold increase in computational cost, where m is thenumber of identified sub-problems.
Empirically, our ablations confirm the strict improvement (e.g.,compare ‘A’ with ‘DSR’ in Figure 4), though performance gains exhibit diminishing returns as the
[Figure]
number of integrated methods increases.
Thus, we suggest inclusion of AIF for use cases in whichcomputational cost is not an important bottleneck.
Finally, we note that we only considered AIFwithout its dimensionality analysis component, as this is really a pre-processing step orthogonal tothe SR algorithm.
For real-world problems with physical units, we highly recommend that users firstperform dimensional analysis on their SR problem.
The best results were obtained using SL pre-training with all other components enabled.
However, ingeneral, pre-training across the ablations yielded mixed results.
For example, the only significant antisynergy among all components occurs between SL pre-training and GP (compare ‘B1’ and ‘B2’ in
Figure 4).
To gain further insight into which use cases may benefit from pre-training, we inspected theinitial rewards for each pre-training modality before fine-tuning.
As shown in Appendix D Figure 6,both SL and RL pre-trained models yield higher reward than randomly initialized models.
This isespecially the case for RL pre-training, where the traces of the initial expressions (see Appendix D
Figure 8) appear very close to the real data.
Thus, pre-training can be desirable for use cases withvery low budgets (e.g. zero-shot learning), a point reinforced empirically by Biggio et al.
However, as shown in Appendix D Figure 7, pre-trained models may have tendencies to become too“sharp” (i.e., exhibiting low entropy), resulting in reduced exploration that can hinder performanceduring fine-tuning.
We conclude that to be effective in combination with other solution strategies in
SR, pre-trained models must strike the delicate balance between high initial reward and sufficientinitial diversity.
Both the LINEAR token and the inclusion of our GP component almost always provide large improvements in both symbolic solution and accuracy solution (e.g., compare ‘C’ and ‘D’ with ‘DSR’ in
Figure 4).
They also do not incur significant additional computational cost given a fixed budget ofexpressions (see Appendix H Table 4).
Thus, we highly recommend their inclusion for most usecases.
A limitation of the LINEAR token, and to a lesser extent the use of GP, is that it is inefficient inimproving accuracy relative to its complexity, which we show in Appendix I Figure 10.
Thus, usersmay not want LINEAR or GP for use cases in which they seek a low-complexity expression.
Conclusion
We introduce a modular framework for SR in which we carefully integrate five standalone SRalgorithms in an attempt to maximize the benefits of each and cover each other’s weaknesses.
Ourunification strategy focuses on abstraction of the constituent methods: using DSR as a baseline, weabstract AIF as an algorithmic wrapper, NeSymReS as a pre-training step, GP as an inner-optimizationloop, and SINDy as a single optimizable token.
Using the SRBench pipeline for benchmarking
SR algorithms, uDSR demonstrates state-of-the-art performance across 252 benchmark problems.
For future work, we leave open the questions of using more advanced modules for GP, such as
GP-GOMEA (Virgolin et al., 2017), or controller architectures (such as transformers as alternatives tothe RNN), as well as the possible benefits from inter-algorithm hyperparameter tuning.
We hope thatfuture work in SR considers additional strategies for hybridizing and/or abstracting SR algorithms.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Several limitations are discussed
in Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] To preserve
anonymity, code is included for reviewers as supplemental material; if accepted,
this will be replaced with a public URL.
Code includes SRBench launch scripts
used to generate results.(b)
Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Hyperparameters used and hyperparameter selection are
described in Section 5 and Appendix F.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All figures and tables with results include error
bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Computational resources used
are described in Appendix J.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Software and dataset
assets are described and cited in Appendix J. Our benchmarks use the PMLB
database, cited in Section 5.
(b) Did you mention the license of the assets?
[Yes] Asset licences are described in
Appendix J
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix
A Expanded pseudocode for uDSR
In Algorithm 2, we provide pseudocode for uDSR in more detail than Algorithm 1.
Note that CONSTand LINEAR optimization is only described in Lines 22-25 and Lines 26-31, respecitvely, but thesealso occur implicitly within GP operators (Line 19) and during RL pre-training (Line 10).
[Table]
[Table]
GP ← T RNN ▷ Seed GP with RNN batch 18: for s = 1 , . .
. , S do 19:
T ( s ) GP ← Γ (cid:16) T ( s − 1) GP (cid:17)
▷ Apply GP operator 20:
T GP ← top k from T (1) GP ∪ · · · ∪ T ( S ) GP ▷ Filter best k samples from GP 21:
T ← T RNN ∪ T GP ▷ Join RNN and best GP samples 22: for j = 1 , . . .
, N do 23: if CONST ∈ τ ( j ) then 24: ξ ⋆ ← arg max ξ R ( τ ( j ) ; ξ ) ▷ Maximize R w.r.t.
ξ , e.g. with BFGS 25: τ ( j ) ← ReplaceConstants ( τ ( j ) , ξ ⋆ )
▷ Replace placeholder constants 26: if LINEAR ∈ τ ( j ) then 27:
Let y =
f ( X, LINEAR )
▷ Instantiate expression 28:
Find G such that LINEAR =
G ( X, y )
▷ Solve equation for LINEAR 29:
Compute y ′
=
G ( X, y )
▷ Evaluate G on ( X, y )
i data 30: β ⋆ ← arg min β ∥ (cid:80) i β i
ϕ i ( X ) − y ′
∥▷ Optimize β , e.g. using LASSO 31: τ ( j ) ← ReplaceCoefficients ( τ ( j ) , β ⋆ )
▷ Replace LINEAR coefficients 32: θ ← train θ (with ψ fixed) on T ▷ Train using fine-tuning objective 33: τ ( i ) ← store best expression for problem P i 34:
τ ⋆ ← Combine ( τ (1) , . . .
, τ ( m ) )
▷ Use AIF to combine m solutions into final solution 35: return τ ⋆
B List of in situ priors and constraints
Autoregressive sampling within DSR affords the opportunity to apply user-specified priors andconstraints to the search space in situ , i.e., during the sampling process.
Each step of autoregressivesampling, the DSR controller emits logits ℓ θ .
Before these logits are passed through a softmax, DSRcomputes a prior denoted ℓ ◦ , which is an additional set of logits that are computed using all availableinformation so far (namely, the partial traversal and the ( X, y ) data).
The prior logits are added tothe controller emissions before a token is sampled via
τ i ∼ Categorical ( Softmax ( ℓ θ + ℓ ◦ )) .
Priorsallow users to incorporate a priori knowledge during the search.
Notably, when a prior includes−∞ entries, this constrains particular tokens from being sampled, thereby pruning the combinatorialsearch space.
Such priors have been used by several works using DSR (Petersen et al., 2021a,b;
Landajuela et al., 2021b; Mundhenk et al., 2021; Kim et al., 2021) as well as other non-SR worksusing similar autoregressive sampling procedures (Popova et al., 2019; Landajuela et al., 2021a).
These priors are enabled during pre-training, both by the problem instance generator and the controller,as well as during fine-tuning.
When integrated with GP, constraints are applied post hoc .
Specifically,if a genetic operator would result in an expression tree that violates any constraint, that geneticoperator is reverted, and instead the parent expression tree is returned.
We first summarize priors that have already been included in works utilizing DSR:
• Length constraint.
Following Petersen et al.
(2021a), traversals are constrained to have a
length no less than 4 and no more than 64.
The minimum length constraint is implemented
by constraining terminal tokens when selecting would would end the traversal before the
minimum length.
For example, given the partial traversal τ =
[ SIN ] , terminal tokens are
constrained because choosing one would end the traversal with length 2.
The maximum
length constraint is implemented by constraining unary and/or binary tokens when selecting
one, followed by selection of only terminal tokens, would result in a traversal that exceeds
the maximum length.
For example, given a partial traversal consisting of 31 instances of + ,
binary tokens would be constrained because choosing one, followed by 32 terminal tokens,
would not complete the end the traversal by length 64.
• Trigonometry constraint.
Following Petersen et al.
(2021a), trigonometric tokens that
would be descendants of other trigonometric tokens are constrained.
For example, given the
partial traversal τ =
[ + , x 1 , COS , ÷ , 1.0 ] , SIN and COS are constrained because
they would
be descendants of COS .
• Constant constraint.
Following Petersen et al.
(2021a), we constrain that constant tokens
(namely, CONST and 1.0) cannot be the only unique child of a token, because the result
would simply be another constant.
For example, given the partial traversal τ =
[ + , CONST ]
or τ = [ SIN ] , CONST and 1.0 are constrained.
• CONST repeat constraint.
Following Petersen et al.
(2021a), we limit the maximum number
of CONST tokens
to 3 per expression, since nonlinear optimization of CONST tokens can be
computationally expensive.
• Soft length prior.
Following Landajuela et al.
(2021b), we include a soft length prior that
biases expressions to have length somewhere between the minimum and maximum.
See
Landajuela et al.
(2021b)
for details.
• Uniform arity prior.
Following Landajuela et al.
(2021b), we include a prior that biases the
controller to have an equal likelihood of selecting a binary, unary, or terminal token.
See
Landajuela et al.
(2021b)
for details.
Next, we describe priors that are newly introduced in this work:
• Available input variables constraint.
Our architecture used for pre-training assumes a
maximum of d = 6 input dimensions.
However, not all problems include all 6 dimensions.
Since the dimensionality of an SR problem is known at runtime, we constrain input variables
beyond the dimensionality of the given problem.
For example, x 5 and x 6 are always
constrained for 4-dimensional problems.
This constraint is enabled when using NeSymReS.
• Constraints for LINEAR .
As discussed in Section 4, the LINEAR token requires threeconstraints.
First, CONST and LINEAR are mutually exclusive.
For example, given the
partial traversal τ =
[ + , CONST ] , LINEAR cannot be included anywhere in this traversal.
Second, LINEAR cannot be a direct ancestor of a non-invertible token (namely, SIN , COS , or
SQRT ).
For example, given the partial traversal τ =
[ SQRT , + , x 1 ] , LINEAR cannot be chosen
because SQRT is a direct ancestor.
Third, LINEAR can only be selected at most once per
traversal.
These constraints are enabled whenever LINEAR is in the library.
• Use each input variable constraint.
We include a constraint that each input variable,x 1 , . . .
, x d for a d -dimensional sub-problem, must appear in the traversal at least once.
This
constraint is implemented by preventing an expression from ending if it would end withoutincluding all input variables (or the LINEAR token, which may arbitrarily include all inputvariables).
For example, given the partial traversal τ =
[ × , + , x 1 , x 3 ] for a 3-dimensional
problem, terminals other than x 2 or LINEAR are constrained because x 2 has not yet beenchosen.
This constraint is enabled for all sub-problems up to 10 dimensions.
C Additional details on integrating SINDyC.1 Efficient sparse linear regression
From the perspective of the LINEAR token, the linear regression problem to solve can be described as:
[Equation]
[Equation]
) where Φ X ∈ R m × k is a Vandermonde matrix (data) computed by applying basis functions Φ( X )to input data X , β ∈ R k is the vector of coefficients (unknown), and
y ′ ∈
R m (denoted f − 1 ( y ) in
Section 4) is the independent variable (data) computed by solving for the LINEAR token as describedin Section 4 and Algorithm 2.
In addition, we would like β to be sparse, with at most l ≪ k non-zeroentries, to avoid over-fitting in cases where the functional form of the expression is not correct.
Underthis constraint, we have (4):
[Equation]
In our application, we need to solve (5) many thousands of times (each time LINEAR is selected), foridentical matrices Φ X (coming from independent variables in the given SR problem) and varyingvalues of y ′ (coming from solving different candidate expression trees for the LINEAR token).
Problem (5) is non-convex.
It can be re-formulated as an mixed-integer linear problem, but solving itwith mixed-integer programming techniques would be prohibitively expensive.
LASSO regression(Tibshirani, 1996), as used by SINDy (Brunton et al., 2016), is the standard approach to deal withsparse regression, but presents two issues for our application: setting the weights of the L 1 penalty isnon-trivial, and it is still one or two orders of magnitude more expensive than regular least squares.
We propose a heuristic approach to solve (5) based on standard least squares, statistical indicators,and constrained quadratic programming.
We start by computing the minimizer of the unconstrainedproblem in (4), β LS :=
Φ † X y ′ , where Φ † X =
(Φ T X Φ X ) − 1 Φ T X is the Moore-Penrose pseudo-inverse ofΦ X , as well as the corresponding p -values of each coefficient β LS for the null hypothesis β LS i
= 0for i
= 1 , . .
k ( t -test).
We then find the k − l coefficients with the largest p -values among β LS andstore their indices in I 0 ∈ N k − l .
Finally, we solve the following constrained quadratic programmingproblem:
[Equation] [Equation]
where C ∈ { 0 , 1 } ( k − l ) × k , with C i,I
i
= 1 for all i = 1 , . .
, k − l , and 0 otherwise, and z ∈ R k − lare dual variables.
The constraints in (6) ensure that all entries at I 0 positions zero in its minimizerβ ⋆ , which can be found by solving the following linear system:
[Equation]
The linear system in (7) has significant structure, which we exploit as follows.
Observe that β ⋆ =β LS − (Φ T X Φ X ) − 1 C T z ⋆ , thus C (Φ T X Φ X ) − 1 C T z ⋆ = Cβ LS .
Because Φ X is a Vandermonde matrix,
Original expression tree y × sin x
[Equation]
sqrt
LINEAR log x
Partially evaluated tree y × sin(x2) + sqrt
LINEAR log(x1)
Partial application y × (sin( x 2) , · ) +(log( x 1) , · ) sqrt
LINEAR for LINEAR
LINEAR square − ÷ y sin(x2) log(x1)
[Figure]
Φ T X Φ X is symmetric positive definite, and so is C (Φ T X Φ X ) − 1 C T .
Furthermore, since C is anelementary matrix we can form C (Φ T X Φ X ) − 1 C T very efficiently whenever (Φ T X Φ X ) − 1 is available.
Thus, we solve the linear system in (7) by first solving C (Φ T X Φ X ) − 1 C T z ⋆ =
Cβ LS using Choleskydecomposition, then computing β ⋆ = β LS − (Φ T X Φ X ) − 1 C T z ⋆ .
To efficiently carry out the above procedure repeatedly with Φ X constant for a given SR problem, wecompute Φ † X and (Φ T X Φ X ) − 1 only once and store the results for all future solutions to the LINEARtoken with the same data (i.e., for the duration of the given SR problem).
Thus, after the first solutionto LINEAR , subsequent solutions do not require recomputing Φ † X or (Φ T X Φ X ) − 1 .
C.2 Example illustration of solving LINEAR token
In Figure 5, we show an example of how to obtain the target data programmatically solve for targetdata f − 1 ( y ) =
G ( x, y ) given F ( x, LINEAR ) , which is then used to compute the sparse coefficientsof the LINEAR token according to (6).
D Additional details on integrating NeSymReSD.1 Pre-training architecture
Following Biggio et al.
(2021), we use a set transformer (ST) (Lee et al., 2019) as the encoder for eachinput dataset ( X, y ) .
This choice of architecture is motivated by its permutation invariant property,i.e., the output of the model does not change under any permutation of the rows of ( X, y ) , and by itsability to process input sets of variable size.
We use the ST architecture introduced in Lee et al.
(2019)with the following hyperparameter (we refer to the paper for their precise definitions): embeddingdimensions for ST encoder and ST decoder is 20, number of attention heads in ST encoder and STdecoder is 2 and number of seed vectors in Pooling Multihead Attention Block is 2.
To interface with the remainder of the RNN-based controller, the encoding of ( X, y ) is used toproduce the initial cell and hidden state of the RNN (illustrated in Figure 1), which in this way acts asa decoder.
For our RNN architecture (consisting of 1 layer of 32 LSTM cells), the decoder outputsh 0 ∈ R 64 ; the 32 first entries are used for the initial RNN cell state and the last 32 for the initial
RNN hidden state.
This way of merging RNN within encoder-decoder architectures is inspired byimage-to-caption systems (Karpathy and Fei-Fei, 2015).
( R , 2 ) ( R , 3 ) ( R , 4 ) ( R , 5 ) ( R , 6 ) ( s l , 2 ) ( s l , 3 ) ( s l , 4 ) ( s l , 5 ) ( s l , 6 ) ( r l , 2 ) ( r l , 3 ) ( r l , 4 ) ( r l , 5 ) ( r l , 6 ) 0.
[Figure]
[Figure] [Figure]
[Figure]
Following Biggio et al.
(2021), to allow the same model to treat problems of different dimensionality(i.e.
, problems with different number of columns in X ), we keep the number of input columns dconstant, and implement a zero-padding strategy in which unused columns of X are zero-padded upto d columns.
We use d = 6 for all experiments.
During pre-training, randomly generated probleminstances are thus limited to 6 dimensions.
At test time, when problems can yield arbitrary dimension, we disable the encoder architecturewhenever dimensionality exceeds 6, using zero-valued initial state and randomly initialized controllerweights (just as in ablations with NeSymReS “off”).
For ablations including AIF, this “triage” isapplied per sub-problem.
For example, given a 10-dimensional problem, the root node will notleverage the pre-trained model, but sub-problems up to 6 dimensions will use the pre-trained model.
This flexibility maximizes the use of the pre-trained models, and is made available by virtue ofcombining NeSymReS, AIF, and DSR to enable pre-trained models, enable sub-problems withsufficiently low dimensionality, and enable training without a pre-trained model, respectively.
This isdifferent than Biggio et al.
(2021), which does not work for problems greater than d = 3 dimensions.
The whole encoder-decoder model is trained on millions of expressions and corresponding ( X, y )datasets that are randomly generated on-the-fly during pre-training.
Problem instance generation isdescribed below.
To enable all possible ablation experiments, we pre-trained four models: two trainedusing SL and two trained using RL, each with and without the LINEAR token.
Each pre-trained modelwas trained for 8 days on a single GPU, seeing a total of ∼ 15M different problem instances each.
D.2 Characterization of pre-trained models
In this section, we characterize the pre-trained models p ( τ | θ ⋆ , ψ ⋆ , ( X, y )) where θ ⋆ , ψ ⋆ are pretrained using either SL or RL objectives.
As a reference, we also provide results obtained withp ( τ | θ R , ψ R , ( X, y )) where ( θ R , ψ R ) are randomly initialized.
In Figures 6 and 7, we show the average reward and entropy, respectively, over 1000 expressionssampled from each pre-trained model, computed across the 116 Feynman benchmark SR problems(which all have dimensionality at most 6) stratified by problem dimensionality.
The entropy iscomputed column-wise over the batch of traversals, i.e., 1 L (cid:80) Li =1 ¯ H [ { τ ( j ) i } 1000 j =1 ] , where ¯ H is empiricalentropy and L is the maximum traversal length in the batch.
Figure 8 shows traces of the top 5% of the 1000 samples from randomly initialized, SL pre-trained,or RL pre-trained models, over a series of one-dimensional problems.
By visual inspection, both SLand RL pre-trained models produce traces that are clearly qualitatively closer to the real data thanthe randomly initialized model.
Comparing traces between SL and RL pre-trained models, RL isclearly closer to the real data than SL.
This is not surprising because the RL objective directly aims toreduce residuals (i.e., generate traces close to the real data).
In contrast, the SL objective only learnsa mapping, though in the SR search space, a small deviation in traversal space may result in a largereduction in fitness to the data.
( R , 2 ) ( R , 3 ) ( R , 4 ) ( R , 5 ) ( R , 6 ) ( s l , 2 ) ( s l , 3 ) ( s l , 4 ) ( s l , 5 ) ( s l , 6 ) ( r l , 2 ) ( r l , 3 ) ( r l , 4 ) ( r l , 5 ) ( r l , 6 ) 0.
[Figure] [Figure]
E n t
[Figure]
y
[Figure]
D.3 Problem instance generation
Here we describe additional details for the SR problem instance generator, used to generate on-the-flyexpressions τ and corresponding datasets ( X, y ) during SL and RL pre-training.
Pseudocode for thisprocess is given in Algorithm
To sample pre-training problem instancess, we leverage the fact that the distribution induced by
DSR includes a prior (Petersen et al., 2021b).
Specifically, DSR samples each token
τ i accordingto τ
i ∼ Categorical ( Softmax ( ℓ θ + ℓ ◦ )) , where ℓ θ are logits emitted by the RNN under currentparameters θ , and ℓ ◦ are the logits defined by the prior (see Appendix B and Landajuela et al.
(2021b)for details).
To generate training problems, we ignore emissions ℓ θ and only sample from the prior,ℓ ◦ .
Since softmax is defined only up to a constant, this process is equivalent to sampling from aprior-informed uniform distribution, i.e. ℓ θ ≡ constant.
After sampling an expression τ , we generate ( X, y ) data from it similarly to Biggio et al.
First, if there is a LINEAR token in τ , we sample linear coefficient values using a spike-and-slabdistribution.
Similarly, if there are any CONST tokens in τ , we sample their values according to
U ( − 5 , 5) .
Next, we generate 500 data points using the expression induced by τ .
To do so, for eachdimension of X , we sample its domain extrema a, b according to U ( − 10 , 10) , then sample each Xcoordinate according to U (min( a, b ) , max( a, b )) .
Finally, we instantiate τ as an expression f andcompute y values as y =
f
( X ) .
To ensure numerical stability of τ and the corresponding ( X, y )data, we re-sample coordinates if | y | > 100 or is undefined until obtaining 500 points.
If 500 pointscannot be obtained within 5000 attempts, (e.g. given a, b < 0 and f ( X ) = log( X ) ), τ is discardedand re-sampled.
Algorithm 3 Symbolic regression pre-training problem instance generator input Expression generator p G ( τ ) output Symbolic regression problem instance ( X, y ) 1: τ ∼ p G ( · ) ▷ Sample expression 2: m ← dim ( τ ) ▷ Get dimensionality of τ 3: if LINEAR ∈ τ then 4: c i ∼ 0 .
5 U ( − 10 , 10) + 0 .
5 δ (0)
▷ Sample linear coefficient values 5: for CONST ∈ τ do 6: c ∼ U ( − 5 , 5)
▷ Sample constant value 7: for i ∈ 1 , . . .
, m do 8: a, b ∼ U ( − 10 , 10) ▷ Sample domain extrema 9: for j ∈ 1 , . . .
, 500 do 10:
X i,j ∼ U (min( a, b ) , max( a, b ))
▷ Sample X coordinate 11: y = f
( X ) ▷ Compute y values 12: return ( X, y )
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
D.4 RL pre-training using risk-seeking policy gradients
Direct optimization of (3) would maximize expected reward of sampled expressions for a givens =
( X, y ) (to simplify notation, we use s to refer to a dataset ( X, y ) in this section).
However, in
SR we are generally interested in best case performance, i.e.,
[Equation]
As as surrogate (relaxed) version of (8), we consider the following dual formulation of the problem:
[Equation]
where R ε ( θ, ψ, s g ) is the (1 − ε ) -quantile of the random variable R ( τ, s g ) , τ ∼ p ( τ | θ, ψ, s g ) .
While(8) is not suitable for gradient-based optimization (as R ( τ, s ) is not continuous), we can compute thegradient of J riskRL ( θ, ψ ) using the risk-seeking policy gradient introduced in Petersen et al.
(2021a),giving:
[Equation]
A Monte Carlo estimate of the above is given by:
[Equation]
θ,ψ J riskRL
[Equation]
θ,
[Equation]
≈
[Equation] [Equation]
with 1 C the characteristic function for set C , s 1 , . .
, s B ∼ p G ,  τ (1)1 , . . .
τ ( N B )∼ p ( τ | θ, ψ, s 1 ) .
τ (1) B , . . .
τ ( N B ) B ∼ p ( τ | θ, ψ, s B ) , and
[Equation]
where Q α ( x ) is the empirical α quantile of vector x .
This motivates a procedure to compute anapproximation of the gradient ∇ θ,ψ J riskRL ( θ, ψ ) : first, sample B problem instances s i =
( X, y ) ifrom p G , then, sample N B expressions τ ( j ) i per problem instance s i from p ( τ | θ, ψ, s i ) , then compute(9) using their rewards R ( τ ( j ) i , s i ) and the empirical (1 − ε )
-quantile.
E Short descriptions of the 14 original SRBench baselines
Below we provide short descriptions of the 14 original SR baseline methods used by SRBench in
Figures 2 and 3.
We refer readers to the original papers for additional details.
• Age-fitness Pareto optimization (AFP): Schmidt and Lipson (2011) propose a GP approach
that avoids premature convergence by adding a multi-dimensional optimization objective
that evaluates solutions based on fitness and age.
• AFP with co-evolved fitness estimates (AFP-FE): Schmidt and Lipson (2009) extend
standard AFP with a new method for fitness estimation (Schmidt and Lipson, 2008).
• AIF: The AIF algorithm (Udrescu et al., 2020) used in this work, using BF/PF as its
underlying search algorithm.
• Bayesian symbolic regression (BSR): Jin et al.
(2019) propose a method that incorporatesprior knowledge and samples expression trees from a posterior distribution using an efficientMarkov Chain Monte Carlo algorithm that is robust to hyperparameter settings and findsmore concise solutions than purely GP-based approaches.
[Table]
• Deep symbolic regression (DSR): The DSR algorithm (Petersen et al., 2021a) used inthis work.
Notably, this uses the predecessor to the version of DSR used in this work, anddoes not include improvements introduced in Landajuela et al.
(2021b) or Mundhenk
et al.(2021).• ε -lexicase selection (EPLEX): La Cava et al.
(2019) propose a method that improves theparent selection process in GP by considering performance on data samples individuallyinstead of in aggregate or average, rewarding expressions that perform well on harder partsof the problem.
[Table]
• Feature engineering automation tool (FEAT): La Cava et al.
(2018) propose a method thataims at finding simple solutions that generalize well by maintaining an archive of solutionswith accuracy-complexity trade-offs to improve generalization and interpretation.
[Table]
• Fast function extraction (FFX): McConaghy (2011) propose a non-evolutionary techniquefor SR based on pathwise learning (Friedman et al., 2010) that generates a set of solutionsthat trade off error versus complexity, while being orders of magnitude faster than GP andproviding deterministic convergence.
[Table]
• GP version of the gene-pool optimal mixing evolutionary algorithm
(GP-GOMEA):Virgolin et al.
(2021) propose to combine GP with linkage learning, an approach that learnsa model of interdependencies to estimate what patterns to propagate and proposes small andinterpretable solutions.
[Table]
• GP (gplearn): Basic, Koza-style GP (Koza, 1994) for SR using the open software library
[Table]
gplearn (https://github.com/trevorstephens/gplearn).
This implementation is very similar to
[Table]
the GP component used in uDSR.
• Interaction-transformation evolutionary algorithm (ITEA): de França and Aldeia (2021)introduce a mutation-based evolutionary algorithm based on six mutation heuristics thatallows both learning high-performing solutions and extracting the importance of eachoriginal feature of a data set as an analytical function.• Multiple regression genetic programming (MRGP): Arnaldo et al.
(2014) propose amethod that decouples and linearly combines sub-expressions via multiple regression on thetarget variable as a cost-neutral extension to basic GP.• SR with Non-linear least squares (Operon): Kommenda et al.
(2020) incorporate nonlinear least squares optimization into GP as a local search mechanism for offspring selectionto improve generalization.• Semantic backpropagation genetic programming (SBP-GP): Virgolin et al.
(2019) modify the random desired operator algorithm
(Pawlak et al., 2014), a GP approach usingsemantic backpropagation, by incorporating the principles of linear scaling, making thealgorithm much more effective despite being computationally more expensive.
F Hyperparameter selection
Hyperparameters for uDSR are listed in Table 3.
In general, we used default hyperparameters fromof each constituent method, with a few exceptions we justify below.
For DSR, we used defaulthyperparameters given by Mundhenk et al.
(2021), with the exception of increasing maximumexpression length from 30 to 64 to accommodate more challenging problems that those attempted in
Mundhenk et al.
Note that, following Mundhenk et al.
(2021), these hyperparameters usepriority queue training (Abolafia et al., 2018) during fine-tuning as an alternative to the risk-seekingpolicy gradient.
For GP, we used all default hyperparameters given by Mundhenk et al. (2021).
For AIF, we used all default hyperparameters given by Udrescu et al.
For NeSymReS, weused all default hyperparameters for the set encoder given by Lee et al.
For pre-training,we increased the maximum input dimension of the set transformer from 3 (in Biggio et al.
(2021))to 6 to accommodate higher dimensional problems.
For RL pre-training, since this has never beenperformed for SR to date, we estimated the learning rate and batch size, and tuned the entropy weight and risk factor via a small grid search of entropy weight ∈ { 0 , 0 . 3 } and risk factor in ∈ { 0 . 05 , 0 . 2 } .
817 For SINDy, since we used a novel formulation of sparse linear regression instead of LASSO, weset the maximum number of terms to 10.
For simplicity, we considered basis functions comprisingmonomials up to degree 3, e.g. x 21 x 3 , x 32 , or x 1 x 2 .
[Table]
S
Mundhenk et al.
[Table]
Mundhenk et al.
[Table]
Mundhenk et al.
[Table]
Mundhenk et al.
Samples used to train controller k
Mundhenk et al.
AIF
Learning rate – 0.
Udrescu et al.
Training epochs –
Udrescu et al.
Hidden layer sizes – [128, 128, 64, 64]
Udrescu et al.
Brute force try time –
Udrescu et al.
Polynomial fit degree –
Udrescu et al.
NeSymReS
Maximum input dimension d
Estimated
ST embedding size –
Lee et al.
ST attention heads –
Lee et al.
ST seed vectors –
Lee et al.
Learning rate (SL) – 0.
Estimated
Problem instances per batch (SL)
B
Estimated
Learning rate (RL) – 0.
Estimated
Problem instances per batch (RL)
B
Estimated
Samples per problem instance (RL only)
N B
Estimated
Entropy weight (RL only) –
Tuned { 0 , 0 .
3 } Risk factor (RL only) ε 0.
Tuned { 0 . 05 , 0 . 2 }
SINDy
Basis functions Φ
Monomials
Estimated
Maximum monomial degree
D
Estimated
Maximum non-zero terms –
Estimated
G Marginal gains
The modularity of uDSR allows one to activate each individual component independently of other components.
We measured the improvement r ( c i, on , c − i ) − r
( c i, off , c − i ) for r = symbolic solution rate 823 and r = accuracy solution rate, for each component c , for all possible combinations of settings forother components
c − i (NeSymReS has two “on” states, denoted “RL” and “SL”).
Figure 9 shows thatindividually enabling AIF, GP, SINDy (i.e., LINEAR token), and DSR leads to positive improvementin symbolic solution rate and accuracy solution rate, averaged over all possible configurations ofother components.
Notably, SINDy almost always improves the symbolic solution rate, AIF alwaysimproves the accuracy solution rate, and DSR provides the highest worst case improvement.
Pretraining with NeSymReS is negative on average, but it provides positive improvement in certaincases; specifically, the highest-performing combination uses NeSymReS (SL) (see Figure 4).
H Runtimes
We evaluate the runtime of uDSR and its constituent components in two ways: (1) runtime perexpression evaluation and (2) runtime per SR problem.
For runtime per expression evaluation, we consider the components GP, SINDy, and DSR.
NeSymReSdoes not significantly affect runtime per expression because the embedding at test time is computedonly once, and the subsequent architecture and algorithm is otherwise the same.
While AIF affectsoverall runtime by generating sub-problems, it does not affect runtime per expression.
We showaverage runtime per expression evaluation in Table 4.
The effect of GP ranges from no effect to a2-fold increase in runtime.
SINDy actually decreases runtime in many cases.
This can occur because
LINEAR “competes” with the CONST token, as both are terminal tokens and the tokens are mutuallyexclusive (see Appendix B).
Since (non-linear) CONST optimization is typically much slower than
LINEAR optimization, this results into an overall reduction in runtime.
Lastly, disabling DSR (i.e.,
[Figure]
not training the controller) has little to no effect on runtime, demonstrating that backpropagation isnot a computational bottleneck for uDSR at test time.
Average runtime per SR problem is given in Table 5 for all uDSR ablations.
Because uDSR usesearly stopping, these values are largely dominated by how early and how often each ablation achievesthe early stopping criterion (defined as in Petersen et al.
(2021a)
as achieving a normalized MSE lessthan 10 − 16 ), as opposed to reaching all 500,000 expression evaluations per problem-instance or the24 hr walltime limit.
The fold-increase in computational cost incurred by AIF is also evident in theseresults.
[Table]
I Accuracy vs complexity for GP and SINDy components
In our ablations on ground-truth SR problems, both GP and the LINEAR token (SINDy) yieldconsiderable gains in symbolic solution rate and accuracy solution rate.
However, GP can leadto longer expressions due to crossover operators (referred to as “bloat”), while LINEAR can resultin longer expressions by adding many linear terms.
To assess the trade-offs between accuracyand complexity, we performed a small ablation study comprising four configurations: GP ∈ [on,off] × SINDy ∈ [on, off], with all other components enabled.
In Figure 10, we see that the fourconfigurations form a Pareto front in terms of accuracy ( R 2 on held-out test data) vs complexity(expression length, computed as number of nodes in the SymPy-parsed expression tree).
Thisreinforces how the modularity of uDSR allows it to be configured based on use case.
[Table]
J Software, data, and computational resources
Our implementation of uDSR combines several open-source software projects.
For DSR we use the
Deep Symbolic Optimization software framework
[Table]
which includes the base implementation of DSR and facilities to integrate GP using the Distributed Evolutionary Algorithms in Python software
(Fortin et al., 2012).
We integrate AIF using its open-source implementation 4 .
We implement the
LINEAR token using standard numerical libraries.
The set transformer used for pre-training is basedon an open-source TensorFlow implementation 5 .
Our empirical evaluation uses SRBench 6 .
The
SRBench benchmark datasets are taken from the open-source PMLB database 7 (Olson et al., 2017).
Pre-training experiments were performed on a single NVIDIA TitanX
GPU.
All other experimentswere performed on a single TOSS 3 2.4 GHz CPU.
A total of 73,700 independent trials of uDSR wererun across all experiments.
Approximately 1% of runs were stopped early at the 24 hr walltime limit.
K Originally published methods versus corresponding uDSR componentsK.1 Methodological differences
uDSR integrates the key capabilities (listed in Table 1) of each of the five integrated algorithms; thealgorithms are not simply reused in their original, published form (with the exception of DSR, asdiscussed below).
While Section 4 details the methodological differences between the originally published algorithms and their corresponding uDSR component, for convenience, below we summarizethese key differences for each algorithm.
AIF.
The key difference between the published, standalone AIF algorithm (Udrescu et al., 2020) andour use of recursive simplification is that the original AIF obtains expressions using BF/PF as itsunderlying SR algorithm, whereas our module obtains expressions sampled from a parameterizeddistribution over expressions.
Another detail is that the original AIF used a slightly different set oftokens that were tailored to the Feynman benchmark problems, whereas our component used a moregeneric set of tokens.
NeSymReS. The key difference between the published, standalone NeSymReS and our pre-trainingcomponent lies in how expressions are sampled at test time.
Standalone NeSymReS uses beam searchat test time to identify high-likelihood expressions under the model conditioned on the test problem.
In contrast, our NeSymReS module fine-tunes the controller (conditioned on the test problem) bycontinuing to train non-encoder parameters at test time.
Differences during pre-training are thatstandalone NeSymReS generates problems of up to 3 input variables, whereas our module generatesproblems of up to 6, and that NeSymReS trains using SL, whereas our module can alternative trainusing RL using a modified version of the risk-seeking policy gradient.
GP.
There are two key differences between the published, standalone GP algorithm
(Koza, 1994) andour GP component.
(1) In standalone GP, starting populations are sampled using the traditional “halfand half” method, in which half the starting population is sampled using the “full” protocol and theother half is sampled using the “grow” protocol, as defined in Luke and Panait (2001).
In contrast,uDSR’s GP starting populations are sampled from the controller.
In standalone GP, there is only asingle starting population per run, and all generations are performed until the algorithm completes.
Inour module, a relatively small number of generations are performed before a new starting populationis sampled from the controller.
Thus, GP module operates in a random restart-like fashion, restartingwith new starting populations as the controller learns.
SINDy.
The key difference between the published, standalone SINDy algorithm (Brunton et al.
, 2016)and our LINEAR token is that the original SINDy uses a fixed functional form (given basis functionsΦ ), whereas LINEAR occurs within a functional form “wrapper” sampled from the distribution overexpressions.
Essentially, the original SINDy algorithm computes a single functional form withtraversal τ =
[ LINEAR ] , whereas even the SINDy-only uDSR ablation includes LINEAR withinnonlinear functional forms.
Two minor differences are that: (1) The original SINDy used LASSO tolearn sparse coefficients, whereas our module uses a modified version of least squares, and (2) Theoriginal SINDy includes both linear and nonlinear basis functions in Φ , e.g. sin( x 1 ) , whereas weused a simpler set of monomials up to degree 3.
DSR.
Unlike the other four components, our DSR module does not exhibit any algorithmic changesrelative to the published DSR in Landajuela et al.
(2021b)
, as DSR was used as a starting point intowhich we integrated the other four methods.
However, note that in Figures 2 and 3, the SRBench
GP
AIF
DSR
SINDy NeSymReS 0.
S y m b o li c s o l u t i o n r a t e
SL
RL
Standalone algorithm uDSR ablation
GP
AIF
DSR
SINDy NeSymReS 0.
A cc u r a c y s o l u t i o n r a t e
SL
RL baseline used the original DSR version in Petersen et al.
(2021a)
without the improvements introducedin Landajuela et al.
(2021b).
K.2 Performance differences
The methodological differences described above may result in differences in performance.
In Figure11, we compare each original, standalone method to the corresponding uDSR ablation, in whichall other components are disabled.
Note that Figure 11 uses the same raw data as in Figures 2 and4, but is reformatted here to enable facile comparison.
Also note that, besides the methodologicaldifferences listed above, there are additional differences in the experimental setting used betweenthe original papers and this work.
For example, AIF uses a different token set than the AIF-onlyuDSR ablation.
For simplicity, for the original methods, we use the experimental workflow used inthe original papers.
AIF is the only standalone algorithm that outperforms the corresponding AIF-only uDSR ablation.
This may be due to AIF using a different token set tailored to solving the Feynman benchmarkproblems, and/or because BF/PF outperforms sampling from a randomly-initialized controller.
The
GP-only uDSR ablation greatly outperforms standalone GP.
This is consistent with the finding by
Mundhenk et al.
(2021) that standard GP excels in a random restart-like fashion.
The two NeSymReSonly uDSR ablations (particularly with RL pre-training) outperform standalone NeSymReS, whichmay be attributed to the fact that standalone NeSymReS was only trained on SR problems of upto 3 input variables, whereas most SRBench problems have > 3 input variables.
(For problemswith > 3 input variables, it cannot select input variables beyond x 3 .)
The DSR-only uDSR ablationoutperforms the standalone DSR used by SRBench (Petersen et al., 2021a)
; however, when using theupdated version of DSR given by Landajuela et al. (2021b), performance is identical to the DSR-onlyuDSR ablation, by construction.
The SINDy-only uDSR ablation greatly outperforms standalone
SINDy.
This is because standalone SINDy only optimizes a single linear solution,
whereas the
SINDy-only uDSR ablation is still able to embed the LINEAR token into highly non-linear functionalforms (in particular, rational expressions).
L Illustrative example of solution formation
By tracing the formation of an expression solution, it is possible to identify which tokens in thesolution τ (i.e. which parts of the expression) can be attributed to which individual uDSR components.
In Figure 12, we provide an illustrative example of this process, which highlights how the five uDSRcomponents work together to find a final solution.
In this illustrative example: (1) AIF first identifies
÷ cos 𝑥 !
LINEAR ×
AIFidentifies multiplicative separability , simplifying 𝒫 into 𝒫 ! and 𝒫 "
The final LINEAR token
[Table]
Solution to 𝒫
S o l u t i o n t
o 𝒫 !
S o l u t i o n t
o 𝒫 " 𝑥 " sin
DSR, conditioned on 𝒫 " data 𝑋,𝑦 " via
[Table]
candidate solutions to 𝒫 "
[Table]
operator
[Table]
candidates
[Table]
form 𝒫 " ÷ cos 𝑥 !
𝑥 " log + exp
LINEAR
Final expression:
[Equation] [Equation]
Final traversal:
[Equation]
[Table]
𝑋, 𝑦
[Table]
[Figure]
6OyqI99bb21leaMJxkkPsIWiOpnhcaQDqri9RZ79m
[Figure]
DbIuJ402flKrb7JLGy6zpevIdXpNmNdceexCn2RyP7eH4PxgmLwbHn
[Figure]
GBx
[Figure]
/zULiTQ
[Figure]
</latexit
[Figure]
multiplicative separability , which simplifies the original problem P into two sub-problems, P 1 and
P 2 .
The solution to P is then the product of the solutions to P 1 and P 2 ; thus, AIF essentially learned 947 that the root node of the expression tree is × .
The simple solution to P 1 is found quickly by DSR.
948 (3) DSR, conditioning the controller on
P 1 data ( X, y ) 2 using the NeSymReS encoder, samples 949 candidate solutions to
P 2 .
The GP crossover operator recombines two candidate solutions into thesolution to P 2 .
(5) Lastly, since the solution to P 2 includes a LINEAR token, the linear combinationof basis functions is optimized according to the ( X, y ) data of P 2 . M Additional details M
1 Behavior of LINEAR for non-finite target data.
When solving for target data f − 1 ( y ) given F ( x, LINEAR ) (by performing successive application ofthe root node’s inverse operations), it may be the case that infinite or undefined values arise.
Forexample, given the traversal τ = [ EXP , LINEAR ] and negative values in y , the function for targetdata f − 1 ( y ) =
log( y ) produces undefined values.
In these cases, the LINEAR token returns a defaultvalue of 1 , i.e. coefficients β are all zero except for the constant This process is similar to the CONSTtoken defaulting to a value of 1 when L-BFGS-B encounters undefined values (Petersen et al., 2021a).
Note that if the candidate functional form is correct (e.g. the traversal τ = [exp, LINEAR] when the( X, y ) data is generated from ground-truth expression e x 21 +0 .
5 x 2 ), then, by construction, it will neverbe the case that ¯ f − 1 ( y ) yields undefined values.
M.2 Additional details for SRBench
SRBench includes metrics for symbolic-based performance, accuracy-based performance, and solution complexity.
“Symbolic solution rate” is computed by comparing the ground-truth SymPy expression to the SymPy-parsed expression returned by the algorithm.
Floating-point values arefirst rounded to 3 decimals.
Expressions are then deemed symbolically equivalent if their symbolicdifference is “0” or their symbolic quotient is “1.”
“Accuracy solution rate” is based on the coefficientof determination ( R 2 ) on a held-out test dataset exceeding 0.999.
R 2 is a useful metric because itis normalized across datasets, allowing averaging performance metrics across multiple problems.
“Complexity” is defined by the number of nodes in the SymPy-parsed expression tree.
Most of the baselines in Figures 2 and 3 report results directly curated by SRBench.
(SRBenchenables reproducibility, including across random number seeds, making such direct comparisonsappropriate.)
The original SRBench study (La Cava et al., 2021) already included baselines forstandalone GP, DSR, and AIF.
Since standalone SINDy and NeSymReS were not included in theoriginal SRBench study, we executed these ourselves using the SRBench pipeline.
For NeSymReS(Biggio et al., 2021), since it was trained on only up to 3 input variables, it ignores additional inputvariables if present.
M.3 Extrapolation and recommendations for NeSymReS
A key aspect regarding recommended use of NeSymReS is knowledge of how well the distributionover ( X, y ) datasets sampled by problem instance generator p G is thought to be representative ofthe user’s problems of interest.
In particular, a key characteristic of p G is the extrema consideredfor the domain of input variables ( [ − 10 , 10] in this study).
A test time problem outside this domainconstitutes an extrapolation problem.
While uDSR has mechanisms to escape such extrapolationcases, either by fine-tuning or via components that do not depend on input variable domains (e.g.
AIF or GP), including the pre-trained network may be unnecessary or harmful.
We investigate this in
Table 6, in which we show the performance of uDSR and baselines against the six SRBench groundtruth problems whose input data exceed that considered during pre-training, [ − 10 , 10] .
Notably,standalone NeSymReS (which does not fine-tune) yields 0% in both symbolic and accuracy solutionrates, suggesting that pre-training without fine-tuning on problems outside the pre-training inputranges can result in catastrophic failure.
On the other hand, uDSR yields 100% in both symbolicand accuracy solution rates over these problems, suggesting that pre-training was not harmful, andthat the algorithm’s fine-tuning and/or other components can render uDSR robust to this type ofextrapolation.
[Table]
In Figure
[Table]
we also show how the minimum and maximum input variables of the Feynman, Strogatz,and black-box SRBench problems compare to those of the pre-training dataset generator.
[Figure]