A Unified Framework for Deep Symbolic Regression
Abstract
The last few years have witnessed a surge in methods for symbolic regression, from advances in traditional evolutionary approaches to novel deep learning-based systems.
Individual works typically focus on advancing the state-of-the-art for one particular class of solution strategies, and there have been few attempts to investigate the benefits of hybridizing or integrating multiple strategies.
In this work, we identify five classes of symbolic regression solution strategies—recursive problem simplification, neural-guided search, large-scale pre-training, genetic programming, and linear models—and propose a strategy to hybridize them into a single modular, unified symbolic regression framework.
Based on empirical evaluation using SRBench, a new community tool for benchmarking symbolic regression methods, our unified framework achieves state-of-the-art performance in its ability to (1) symbolically recover analytical expressions, (2) fit datasets with high accuracy, and (3) balance accuracy-complexity trade-offs, acrossground-truth and black-box benchmark problems, in both noiseless settings and across various noise levels.
Finally, we provide practical use case-based guidance for constructing hybrid symbolic regression algorithms, supported by extensive, combinatorial ablation studies.
Introduction
[Equation]
[Table]
+5UVSOyu658XSbalQvprFkSOH5JicEpdckDK5IRVSJZw8kmfySt6sJ+vFerc+pq1L1mzmgPyB9fkDeA2ZMw==</latexit> Solve
P i <latexit sha1_base64="5rhwB217eyFohz9OpeQcIUADdQs=">AAACBXicbVA9SwNBEN3zM8avqKUWi0GwCncS1DJoYxnRfEASwt5mkizZvTt254LhuMbGv2JjoYit/8HOf+Pmo9DEBwOP92aYmedHUhh03W9naXlldW09s5Hd3Nre2c3t7VdNGGsOFR7KUNd9ZkCKACooUEI90sCUL6HmD67Hfm0I2ogwuMdRBC3FeoHoCs7QSu3cURPhAbVK7kI5BJo2FcM+ZzIpp23VzuXdgjsBXSTejOTJDOV27qvZCXmsIEAumTENz42wlTCNgktIs83YQMT4gPWgYWnAFJhWMvkipSdW6dBuqG0FSCfq74mEKWNGyred4yPNvDcW//MaMXYvW4kIohgh4NNF3VhSDOk4EtoRGjjKkSWMa2FvpbzPNONog8vaELz5lxdJ9azgnReKt8V86WoWR4YckmNySjxyQUrkhpRJhXDySJ7JK3lznpwX5935mLYuObOZA/IHzucPfh2ZNw==</latexit> Solve
P m <latexit sha1_base64="jFyyjNrYHkgFkiRbmN/bX65uemI=">AAAB7HicbVBNS8NAFHypX7V+VT16WSyCp5KIqMeiF48VTFtoQ9lsN+3SzSbsvggl9Dd48aCIV3+QN/+N2zYHbR1YGGbesO9NmEph0HW/ndLa+sbmVnm7srO7t39QPTxqmSTTjPsskYnuhNRwKRT3UaDknVRzGoeSt8Px3cxvP3FtRKIecZLyIKZDJSLBKFrJ7w0SNP1qza27c5BV4hWkBgWa/eqXzbEs5gqZpMZ0PTfFIKcaBZN8WullhqeUjemQdy1VNOYmyOfLTsmZVQYkSrR9Cslc/Z3IaWzMJA7tZExxZJa9mfif180wuglyodIMuWKLj6JMEkzI7HIyEJozlBNLKNPC7krYiGrK0PZTsSV4yyevktZF3buqXz5c1hq3RR1lOIFTOAcPrqEB99AEHxgIeIZXeHOU8+K8Ox+L0ZJTZI7hD5zPH/RWjss=</latexit> ...
<latexit sha1_base64="zYA27x8OG2VzuukSUYFEBJW/jwk=">AAACynicbVHLatwwFNU4faTT1yRZZmM6BFIogx3SpsvQbrroIoVOEpiZDrJ87YiRJSNdJzFCi0L/oZ/Qbfo5/ZvKHhecpBckDuce6V7OSUrBDUbRn0Gw8eDho8ebT4ZPnz1/8XK0tX1qVKUZTJkSSp8n1IDgEqbIUcB5qYEWiYCzZPWx6Z9dgjZcya9Yl7AoaC55xhlFTy1Hu3OEa2z
[Table]
+q/21ms2qhQyn8HayZTqVa4BpLM6T5z1pr3xzr1trsjb0KoEoM1LZxNRQZ9rY9CQ9jlptPFKra5knzZcprXrjev1UqOdveTKYzf0UcZ3g7sPTg8m8bvJ4ZfD8fGHLtRNsktekX0SkyNyTD6REzIljHwnv8gN+R18DnRQB3YtDQbdmx1yq4IffwE2MeHc</latexit> [Table] m [Table]
<latexit sha1_base64="mjKeKSvXZ0NJIuT+Y+xUYMoxadA=">AAACt3icbVFNb9QwEPWmQMvy1ZYjl4hVJQ5olaAWyq2iF45FYtuK3WU1cSZZax072JPCysq/qMQJ/hf/BicbpLRlJFvPb95oxm+SUgpLUfRnEGzdu/9ge+fh8NHjJ0+f7e7tn1tdGY4TrqU2lwlYlELhhARJvCwNQpFIvEhWp03+4gqNFVp9pnWJ8wJyJTLBgTz1ZUZQfZ1ZArPYHUXjqI3wLog7MGJdnC32Bj9nqeZVgYq4BGuncVTS3IEhwSXWw1llsQS+ghynHioo0M5dO3IdHngmDTNt/FEUtmy/wkFh7bpIvLIAWtrbuYb8X25aUXY8d0KVFaHim0ZZJUPSYfP/MBUGOcm1B8CN8LOGfAkGOHmXhjfa
/Ohm/RcHmzEbVYqZN7x9uhTMKjeIqnYmT2rnTXvtnTtqrsjb0KokksvL2iWywj4HIvNlmPY5ZY31SqO/qz5thUrXda9dL5daU7sroT2uh36V8e3F3QXnb8bx2/Hhp8PRyYduqTvsBXvJXrGYvWMn7CM7YxPGmWLX7Bf7HbwPFkEWLDfSYNDVPGc3Ivj2F5nE2gE=</latexit> ⌧ ? <latexit sha1_base64="TtVCgxlJ9zWo1aHp8nqd6xiGjtY
[Table]
/cD5/AOrWjqc=</latexit> ( X, y )<latexit sha1_base64="6MwE3OiTgq8CZRxn7pm7IihbgUs=">AAAB7nicbVBNSwMxEJ2tX7V+VT16CRahgpRdKeqx6MVjBfsB7VKyabYNTbJLkhWWpT/CiwdFvPp7vPlvTNs9aOuDgcd7M8zMC2LOtHHdb6ewtr6xuVXcLu3s7u0flA+P2jpKFKEtEvFIdQOsKWeStgwznHZjRbEIOO0Ek7uZ33miSrNIPpo0pr7AI8lCRrCxUqfavUjPB2xQrrg1dw60SrycVCBHc1D+6g8jkggqDeFY657nxsbPsDKMcDot9RNNY0wmeER7lkosqPaz+blTdGaVIQojZUsaNFd/T2RYaJ2KwHYKbMZ62ZuJ/3m9xIQ3fsZknBgqyWJRmHBkIjT7HQ2ZosTw1BJMFLO3IjLGChNjEyrZELzll1dJ+7LmXdXqD/VK4zaPowgncApV8OAaGnAPTWgBgQk8wyu8ObHz4rw7H4vWgpPPHMMfOJ8/P8WO3w==</latexit> ( X, y )
i <latexit sha1_base64="wn1Bga/NikvB5dfqkdlmgY6qcYs=">AAAC0HicbVFNb9QwEPWGj5blawtHLoFVJQ5olVRAOVZw4VgQ21baXa1sZ5Ja69iRPSm7sizECYk/wK/gCr+Ff4OTDVLaMpKt5zdvPKM3rJLCYpL8GUQ3bt66vbN7Z3j33v0HD0d7j06srg2HKddSmzNGLUihYIoCJZxVBmjJJJyy1bsmf3oBxgqtPuGmgkVJCyVywSkGajl6OkdYY/uPoyL3rn2z3HFdsvCn98vROJkkbcTXQdqBMenieLk3+DHPNK9LUMgltXaWJhUuHDUouAQ
[Table]
[Figure]
Symbolic regression (SR) aims to identify a tractable mathematical expression to best fit a dataset.
Specifically, given the set of pairs { (cid:0) x i =
( x i, 1 , . . .
, x i,d ) , y i (cid:1) } ni =1 , SR seeks a mathematical function f : R d → R such that the residual ∥ f ( X ) − y ∥ 2 is minimized, where the matrix X ∈ R n × d has entries X i,j = x i,j , the vector y ∈ R n has entries
y
i , and we assume point-wise application of real-valued functions over vectors and matrices, i.e., f ( X ) =
( f ( x i )) ni
=1 ∈ R n (we keep this convention throughout the paper for convenience).
Since the introduction of SR as an application of genetic programming (GP) (Koza, 1994), many classes of methods—especially machine learning-based methods in recent years—have been proposed.
These methods include recursive problem simplification (Udrescu et al., 2020), neural-guided search (Petersen et al., 2021a), large-scale pre-training for problem generalization (Biggio et al., 2021;
Kamienny et al., 2022; Vastl et al., 2022), sparse linear regression using nonlinear basis functions (Brunton et al., 2016), and combination of evolutionary search with learning (Mundhenk et al., 2021).
Each of these independent advances exploits different information (e.g., graph modularity or constraints) to offer key capabilities for various aspects of the SR problem (e.g., search or generalization), but the specificity of each work resulted in design choices that pose limitations in other facets of the problem.
This motivates an investigation of whether disparate solution strategies are complementary to one another and how they can be effectively combined.
In this paper, we carefully rewire the five aforementioned solution strategies into a unified SR framework and show that they are indeed complementary.
Based on the SRBench pipeline for benchmarking SR algorithms ( https://github.com/cavalab/ srbench ), our unified deep symbolic regression (uDSR) framework achieves a new state-of-the-art in SR: for ground-truth problems, uDSR ranks highest in both symbolic-based and accuracy-based solution rates for both noiseless settings and for multiple noise levels; for black-box problems, uDSR falls on the accuracy-complexity Pareto frontier alongside three other non-dominated algorithms.
Our main contributions in this work include: (1) a modular, unified framework for SR that integrates five disparate SR solution strategies (illustrated in Figure 1); (2) a novel LINEAR token that identifies sparse coefficients of a linear combination of basis functions and can be used by almost any SR algorithm; (3) achieving new state-of-the-art for SR on SRBench in its metrics for symbolic recovery, solution accuracy, and accuracy vs complexity trade-offs; and (4) practical recommendations for designing SR algorithms, driven by extensive and combinatorial ablation studies that show the complementary benefits of these previously disjoint SR solution strategies.
Background
We describe each of the five solution strategies our framework unifies.
For each strategy, we focus on a key capability and a key limitation 2 , summarized in Table 1, which motivates our unification that captures all capabilities and mitigates all limitations via the inherent synergy among strategies.
We encourage readers to read the original works for additional background.
AI Feynman.
AI Feynman 2.0 (AIF) (Udrescu et al., 2020) approaches SR via recursive problem simplification ; it is the only known algorithm to date to employ this solution strategy.
Specifically,
AIF exploits knowledge of physics and the given training data, ( X, y ) , by identifying simplifying properties (e.g., multiplicative separability) of the underlying functional form.
Applying a simplification yields sub-problems (or one sub-problem) of lower dimensionality.
Simplifications are applied recursively until all sub-problems (i.e., leaf nodes of the recursion tree) are 1-dimensional.
For each sub-problem, the paper employs a combination of brute force search plus polynomial fit (BF/PF).
Finally, solutions to the sub-problems are then combined back into a final solution to the original problem.
While AIF positions itself as a standalone SR algorithm, it is more appropriately understood as a SR problem simplification algorithm , as pointed out in prior work (Petersen et al., 2021a; Biggio et al., 2021).
From this general perspective, the use of BF/PF is simply a design choice, orthogonal to the core methodology, and may be enhanced by integration with more advanced search methods for solving sub-problems.
Deep symbolic regression.
Deep symbolic regression (DSR) (Petersen et al., 2021a) leverages the effectiveness of deep reinforcement learning for combinatorial search in SR.
DSR employs a neural network controller with parameters θ to represent a distribution p ( τ | θ ) over mathematical expressions τ = [ τ 1 , . . , τ T ] , represented by the sequence of “tokens” τ i (mathematical operators, input variables, or constants) in the pre-order (depth-first then left-to-right) traversal of the corresponding expression tree (see Koza (1994) for details).
For instance, τ = [ + , SIN , x 1 , EXP , x 2 ] corresponds to expression f ( x ) = sin( x 1 ) + exp( x 2 ) .
Expressions are sampled from the controller autoregressively, i.e. p ( τ | θ ) =
(cid:81)
Ti =1 p ( τ i | τ 1: i − 1 , θ ) , and evaluated based on fitness R ( τ, ( X, y )) to the ( X, y ) dataset.
The controller is then trained to maximize the reinforcement learning objective:
[Equation]
using a risk-seeking policy gradient that aims to maximize best-case performance.
Autoregressive sampling affords the ability to easily incorporate prior knowledge by directly modifying likelihoods of individual tokens p
( τ i | τ 1: i − 1 , θ ) in situ , for example by precluding expressions with nested trigonometric operators.
This dramatically reduces the search space, which has been shown to greatly improve the ability to recover ground-truth expressions (Petersen et al., 2021b).
Whereas DSR focuses on discrete search and pruning the combinatorial search space, it does not exploit the ( X, y ) data.
In fact, the ( X, y ) data itself only shows up in the reward computation in (1), which is a weak, aggregate learning signal.
Thus, DSR trains from scratch for each new SR problem and cannot generalize to new problems.
Large-scale pre
-training.
Large-scale pre-training (LSPT) algorithms (Biggio et al., 2021;
Kamienny et al., 2022; Vastl et al., 2022) tackle the problem of generalization in SR: they seek a single model that, once trained, produces a distribution over solutions to any SR problem.
To accomplish this, the algorithm “Neural Symbolic Regression that Scales” (NeSymReS) (Biggio et al., 2021) uses a set transformer (Lee et al., 2019) to learn an encoding of the ( X, y ) data.
A second transformer network then maps the encoding to a distribution over expressions using autoregressive sampling.
The model is trained end-to-end via supervised learning, using a large dataset of SR problems (input data ( X, y ) and corresponding ground truth label τ ) produced by a programmatic expression generator (Lample and Charton, 2019).
Given ( X, y ) at test time, beam search is used to find high-likelihood expressions.
LSPT approaches claim that pre-training alone is sufficient to solve many SR problems.
However, the lack of fine-tuning during testing means that LSPT algorithms may not be optimized for new out-of-distribution SR problems.
Genetic programming.
Genetic programming (GP) refers to a well-established class of evolutionary algorithms commonly applied to SR in which a set or “population” of mathematical expressions is maintained and “evolved” over time using genetic operators such as mutation, crossover, and selection.
We refer the reader to Langdon and Poli (2013) for an overview of GP.
In this work, we consider only standard GP (as defined in Chapter 7 of Bäck et al.
; however, our integration of
GP into the unified framework applies to virtually any GP method.
GP excels at exploring the search space to find complex expressions, but does not employ a neural network or other parameterized model to learn over time.
This limitation was addressed in Mundhenk et al.
(2021), where GP was combined with a learned expression generator, but not in concert with other methods considered here.
Linear models.
Linear models (LM) are a mainstay of statistics and machine learning (Murphy, 2022).
LM methods seek expressions of the form g = (cid:80) ki
=1 β i ϕ i
=
Φ β T , where Φ :=
( ϕ 1 , . . .
, ϕ k ) comprises a library of k user-defined basis functions ϕ
i : R n → R , and β = ( β 1 , . . . , β k ) is a vector of scalar coefficients that is often learned using least squares minimization.
To avoid over-fitting, the vector β can be alternatively optimized using sparsity inducing optimizers.
As an example, the algorithm “Sparse Identification of Nonlinear Dynamical systems” (SINDy) (Brunton et al., 2016) is a simple, computationally expedient, and popular LM method for SR, particularly for learning systems of differential equations.
SINDy identifies sparse representations by using LASSO (Tibshirani, 1996).
While effective at quickly learning sparse expressions, the restriction to a linear combination of pre-defined basis functions results in a massively reduced search space, precluding such functional forms as rationals and compositions.
In our unified framework, we abstract the entire process of learning a linear model into one single discrete token, thereby restoring the nonlinear, combinatorial search space while retaining the ability to quickly learn sparse linear subcomponents.
Related Work
In addition to AIF, DSR, LSPT, GP, and LM described above, we describe several other related SR solution strategies that are not part of our unified framework.
EQL ÷ presents a unique deep-learning framework for SR that employs neural networks whose activation functions are elementary operators, enabling end-to-end differentiable training (Sahoo et al., 2018).
GrammarVAE learns to encode expression trees in a continuous space and uses Bayesian optimization to optimize them in latent
[Table]
space (Kusner et al., 2017).
Both EQL
÷ and GrammarVAE impose specific representations of the expressions (either by constraining the search space or by projecting it into a latent space) and thus do not facilitate straightforward integration with the solution strategies in our framework.
The idea of pre-training models for use in neural-guided search, used by LSPT approaches and our unified framework, is inspired by Bello et al.
[Table]
who apply a similar method to the traveling salesman problem.
Previous work on hybrid approaches to SR mainly focus on improving a single aspect of one approach, such as alleviating the challenge of high dimensional data by extracting features for GP (Icke and
Bongard, 2013), or improving the starting population of GP iterations (Mundhenk et al., 2021).
Often, the motivation for combining separate methods comes from specific problem settings, such as the case of datasets with missing values (Al-Helali et al., 2018) or noisy high-dimensional measurements in systems with known physical constraints (Reinbold et al., 2021).
In contrast, the unification in our work serves to enhance multiple general capabilities of SR, such as generalization, combinatorial search and optimization, and single-token abstraction of function spaces.
Our empirical analysis heavily leverages SRBench, a reproducible and open-source platform for benchmarking new SR methods against 14 established methods on accuracy and complexity for 122 real-world datasets and exact symbolic correctness for 130 ground-truth problems with varying noise levels (La Cava et al., 2021).
SRBench has seen rapid adoption by recent works that use it to evaluate improvements in accuracy, exact solution rate, and solution complexity (Kamienny et al., 2022; Virgolin and Bosman, 2022; Zhang et al., 2022).
Lastly, we note a similarity between this work and Rainbow, a work that combines several deep reinforcement learning methods into a unified algorithm and performs extensive ablation studies (Hessel et al., 2018).
Analogous to Rainbow’s use of Deep Q Networks (DQN) as a base framework into which it integrates several improvements, this work uses DSR as a base framework into which we integrate AIF, LSPT, GP, and LM.
Methods
Our overall unification strategy is to carefully rewire the solution strategies identified above as connected but non-overlapping modules in an algorithmic framework that leverages their key capabilities.
This is visualized in Figure 1 (see also Figure 12 in Appendix L for an illustrative example of expression discovery under uDSR).
In an initial offline stage, pre-training a parametric controller model on a large dataset facilitates generalization to test problem instances (LSPT).
In the online stage, given a specific problem instance, a recursive problem simplification module (AIF) exploits modularity of mathematical expressions to produce sub-problems of lower dimensionality for the main trunk of uDSR.
Within the trunk, a Recurrent Neural Network (RNN) continually learns over many iterations (DSR) and provides good candidates to seed the starting population for genetic programming (GP); high fitness populations produced by GP are combined with elite candidates for controller parameter updates via risk-seeking policy gradient.
Finally, the RNN is permitted the use of a powerful symbolic token that encapsulates an entire space spanned by basis functions (LM).
Below, we describe the integration of each module in detail.
Algorithm 1 provides high-level pseudocode for uDSR.
More detailed pseudocode is available in Algorithm 2 of Appendix A.
[Table]
Beginning with DSR.
For DSR, we begin with the method proposed by Petersen et al.
(2021a), including two improvements to exploration (soft length prior and hierarchical entropy regularizer) introduced in follow-up works (Landajuela et al., 2021a,b).
DSR includes a number of in situ priors and constraints to the search space, which help bias and/or prune the search, and have been shown to greatly improve performance (Landajuela et al., 2021b).
We provide a complete list of DSR priors and constraints in Appendix B, including several new constraints required to unify all five methods (described below).
DSR also includes a CONST token that represents an arbitrary floating-point constant, whose value is optimized using a nonlinear optimizer, e.g., L-BFGS-B (Zhu et al., 1997).
Integrating AIF via recursive simplifications.
Our key insight to integrate AIF (Udrescu et al., 2020) is that its recursive problem simplification steps can be viewed as orthogonal to the underlying
SR algorithm.
While AIF resorts to BF/PF, we propose using a hybrid of the other four integrated algorithms (which we continue to describe in subsequent paragraphs).
To integrate AIF, we consider the starting SR problem as P 1 , a d -dimensional SR problem.
One application of AIF’s simplification step produces either one sub-problem P 2 , with dimensionality d 2 < d , or two sub-problems, P 2 and
P 3 , with dimensions d 2 and d 3 such that d 2 + d 3 = d , as well as metadata M containing information required to recombine sub-solutions back to the full solution of
P 1 .
Recursive simplifications are applied until sub-problems at the leaf nodes of the recursion tree are each 1-dimensional.
For each sub-problem P i (including the starting problem, P 1 ), we apply our hybrid SR algorithm to its corresponding dataset, ( X, y ) i .
Finally, using the metadata from each simplification step, solutions from all sub-problems are recombined to form a single, final solution to the original problem, P 1 .
Integrating LM via the novel LINEAR token.
Our design goal for integrating LM is to retain the ability to quickly learn linear subcomponents of an expression, while still operating in a general nonlinear functional space.
That is, while LM finds expressions of the form g ( x ) =
Φ( x ) β T , we seek to search more generic expressions of the form f ( x ) =
F ( x, g ( x )) , where F is a nonlinear function subject to certain constraints (see below), while still using linear methods to determine the coefficients of g ( x ) .
We accomplish this by proposing to represent g ( x ) as a special type of terminal token, which we call LINEAR .
LINEAR serves as a placeholder for a LM-like solution, Φ( x ) β T , where the values of the coefficients β are determined only when a traversal has terminated.
Thus, we abstract the entire process of learning a linear model into a single discrete token.
In our approach, the function F ( x, w ) must admit an expression tree with a single argument token for w and there must exist G ( x, v ) such that y = F ( x, G ( x, y )) , ∀ x, y ∈ R d × R .
Given an expression f ( x ) =
F ( x, Φ( x ) β T ) , with F as described above, let f (Φ β T ) denote the view of f as a function of β only.
Naively, one could solve a nonlinear optimization problem to determine the values of β , just as DSR and Biggio et al.
(2021) optimize CONST tokens.
However, nonlinear optimization is computationally expensive, scales poorly with more coefficients, and is challenging to induce sparsity.
Instead, we consider the following convex optimization problem, for which there are efficient sparse linear solvers:
[Equation]
We attain this formulation by solving the equation y = F ( X, LINEAR ) for LINEAR , which gives
LINEAR =
G ( X, y ) .
Since this requires taking the inverse of all ancestor operators of LINEAR — binary operators are reduced to unary operators by lexical closure, elaborated more programmatically below—we use f − 1 ( y ) :
=
G ( X, y ) to denote the view of G as a function of y only.
Equation (2) exhibits an attractive property that enables LINEAR to exactly recover non-linear expressions with linear subcomponents.
When LINEAR is chosen at the correct position in a traversal, in the sense that the true computational graph contains a linear combination of basis functions in the corresponding subgraph, we have that f − 1 ( y ) exactly equals that linear combination.
For example, given ( X, y ) data generated from a ground-truth expression f ⋆ ( x ) = sin( x 1 )+exp(1 .
2 x 31 +3 .
4 x 1 x 2 ) , and a basis containing all monomials up to degree 3, the traversal τ =
[ + , SIN , x 1 , EXP , LINEAR ] yields f − 1 ( y )
=
log ( y − sin( x 1 )) , and the corresponding solution to (2) exactly recovers f ⋆ .
Programmatically, we solve (2) using the expression tree for τ and the following procedure (illustrated in Figure 5 of Appendix C; pseudocode is provided in Algorithm 2 of Appendix A): First, partially execute the tree by evaluating all nodes that are not direct ancestors of LINEAR .
Second, redefine binary operators as unary operators by fixing (or “binding”)
the one pre-computed argument (this is called partial application —a type of lexical closure —in computer science).
Third, successively apply the root node’s inverse operation on the y data until LINEAR is the only token in the tree, resulting in f − 1 ( y ) .
Finally, perform a linear fit on (2) using the transformed data, ( X, f − 1 ( y )) .
This procedure for abstracting a linear model into a single operator exhibits several complications: (1) it only works if there is exactly one LINEAR token, (2) it does not work in the presence of other placeholder tokens, e.g., the CONST token used in DSR, and (3) it does not work if LINEAR has non-invertible ancestors (e.g., SIN ), as y = F ( X, LINEAR ) might not have a unique solution on
LINEAR .
Fortunately, seamlessly incorporating these restrictions into the search space is precisely what the DSR algorithm excels at.
To this end, we enable LINEAR into our framework by introducing three constraints to the search space (further detailed in Appendix B): (1) there can be at most one
LINEAR token; (2) the LINEAR token and CONST token are mutually exclusive—one cannot appear if the other is present; and (3) the LINEAR token cannot be the descendant of a non-invertible unary token, e.g., SIN .
In this work, we consider the set of basis functions Φ given by the monomials of degree up to D and solve (2) using a sparsity-inducing least square method (see Appendix C for details).
Lastly, we observe that standard LM regression is restored when F ( x, w ) =
w .
In that case, we recover the function f ( x ) =
Φ( x ) β T with trivial traversal τ =
[ LINEAR ] .
Thus, our method expands the search space of LM by allowing nonlinear function compositions.
For instance, F ( x, w ) = 1 w leads to f ( x ) =
1 Φ( x ) β T , which is outside the search space in standard LM regression.
Integrating GP via population seeding.
To our knowledge, hybridization of DSR and GP is the only combination of the five methods that has been explored to date.
To summarize, as in Mundhenk et al.
, we hybridize DSR with GP by abstracting the GP algorithm as an inner optimization loop within DSR.
Each time DSR samples a batch of expressions, we use this batch as the starting population for an inner-loop GP algorithm.
The top samples at the end of GP are combined with the
DSR batch for training the neural network.
This process leverages GP’s ability to rapidly explore the search space, while using DSR’s persistent neural network to mitigate the disadvantages of GP’s non-parametric nature (Mundhenk et al., 2021).
Integrating LSPT via supervised or reinforcement learning.
We integrate LSPT by merging the
RNN used in DSR with a set transformer in an encoder-decoder fashion.
Our encoder architecture closely mimics that of NeSymReS (Biggio et al., 2021).
The set transformer encodes a given dataset ( X, y ) into a latent representation h 0 .
This representation is passed as initial state to the RNN, which decodes it into an expression τ .
The resulting architecture models the distribution p ( τ | θ, ψ, ( X, y )) , where θ and ψ are the parameters of the RNN and set transformer, respectively (see Figure 1).
As in Biggio et al.
(2021), we pre-train the model end-to-end on many SR problems using supervised learning, using ground truth expressions as labels, resulting in parameters θ ⋆ , ψ ⋆ .
However, different from existing LSPT approaches, given a new dataset ( X ′ , y ′ ) , we propose to search for a symbolic expression by fine-tuning p ( τ | θ, ψ ⋆ , ( X ′ , y ′ )) using reinforcement learning, with initial conditions θ
= θ ⋆ .
The pre-training phase requires a large dataset comprising SR problems, each defined by a set of points ( X, y ) along with true expression τ .
To scale to arbitrary dataset size, we seek a problem instance generator p G ( τ, ( X, y ))
=
p G (( X, y ) | τ ) p G ( τ ) , from which we can sample expressions τ and corresponding ( X, y ) data.
Full details of the sampling process used by p
G are provided in Appendix D. Briefly, we leverage the fact that the distribution induced by DSR includes a prior (Petersen et al., 2021b).
To generate expressions, p G ( τ )
= p ( τ | θ = 0 ) directly samples from this prior, ignoring emissions from the RNN.
Thus, the algorithm bootstraps or “imagines” expressions that it then uses for training data, rather than relying on an external expression generator as in Biggio et al.
; Kamienny et al.
; Vastl et al.
After sampling τ , p G (( X, y ) | τ ) is obtained following the strategy in Biggio et al.
In this work, we explore two different objectives for model pre-training.
First, following Biggio et al.
(2021), we consider the supervised learning (SL) objective:
[Equation]
During fine-tuning, ground-truth labels will not be available.
To mitigate an objective mismatch between pre-training and fine-tuning objectives, we alternatively explore pre-training à la reinforcement learning (RL) (Bello et al., 2017).
Specifically, we consider:
[Equation]
In practice, this objective is maximized using a novel formulation of the risk-seeking policy gradient for multitask learning (see Appendix D for details).
Experiments
Pre-training.
We pre-train four models:
two trained using SL and two trained using RL, each with and without the LINEAR token.
Details of the pre-training setup can be found in Appendix D.
Experimental setup using SRBench.
We empirically assess uDSR using SRBench (La Cava et al., 2021), an open-source and reproducible pipeline for benchmarking SR algorithms.
SRBench features 130 problems with hidden ground-truth analytic solutions and 122 real-world datasets with no known analytic model (“black-box” problems) from the PMLB database (Olson et al., 2017).
By running uDSR through the SRBench pipeline, we enable direct comparison to its curated results ofcontemporary SR methods.
The 14 baselines are described in Appendix E.
For all experiments, we use a minimal set of tokens: + , − , × , ÷ , SIN , COS , EXP , LOG , SQRT , 1.0,
CONST , and (except for the appropriate ablations) LINEAR .
For simplicity, our choice of Φ (basis functions for LINEAR ) includes only polynomial terms up to degree 3.
Hyperparameters are listed
[Figure]
in Table 3 of Appendix F.
When possible, we reused the hyperparameters of the published methods most similar to each uDSR component.
We ran uDSR via SRBench on the 252 datasets with 10 random seeds each.
Each run was performed for up to 2,000,000 expression evaluations per sub-problem
[Figure]
or until a 24 hr walltime limit was reached.
For ground-truth problems, SRBench computes two metrics, each averaged across theproblems: (1) symbolic solution rate, based on symbolic equivalence using SymPy (Meurer et al., 2017) and (2) accuracy solution rate, defined as achieving an R 2 > 0 .
999 on held-out test data.
For black-box problems, SRBench reports accuracy ( R 2 on held-out test data) vs complexity (number of nodes in the SymPy-parsed expression tree)
[Figure]
Performance results.
Figure 2 shows that uDSR outperforms all other 14 benchmarked methods both in symbolic solution (by a large margin) and accuracy solution rates.
Figure 3 shows that uDSR falls on the Pareto frontier, alongside Operon (Kommenda et al., 2020)
(at higher accuracy and complexity), and DSR and
SINDy (at lower accuracy and complexity).
Notably, the previously published SRBench results had no clear winner across the three categories of (1) symbolic solution, (2) accuracy solution, and (3) Pareto efficiency.
The top performers for symbolic solution rate (namely, AIF) differed from those for accuracy solution rate (namely, Operon and MRGP).
Only
Operon appeared as a top performer in two categories.
However, in this work, we see that uDSR is highest in all three categories.
Ablations and Discussion
Ablations.
Since uDSR integrates many disparate components, it is critical to assess the relative contributions of each component in various contexts.
To identify synergies, anti-synergies, and
[Figure]
diminishing returns among the integrated methods, we perform a large-scale study of all combinations of the five integrated methods (standard “leave-one-out” style ablations in which we begin with the full uDSR and ablate one component at a time do not provide this full picture).
There are 48 total combinations: AIF ∈ [on, off] × GP ∈ [on, off] × LINEAR ∈ [on, off] × pre-training ∈ [SL, RL, off] × DSR ∈ [on, off].
We define ablating each component as follows:
For AIF, “off” means that we do not perform recursive problem-simplification; only the root problem is considered.
For GP, “off” means we do not perform inner-loop GP algorithms between batches.
For LINEAR , “off” means we exclude the LINEAR token from the library.
For pre-training, SL and RL refer to whether we use the model pre-trained using SL or RL; “off” means we do not use any pre-trained model, and the controller architecture does not include the set transformer component.
For DSR, “off” means the learning rate for the neural network is set to zero; notably, the controller is still used for sampling, including all priors and constraints.
For each ablation, we run on the 130 ground-truth SRBench problems with 10 random seeds each ( 48 × 130 × 10 = 62 , 400 total runs), using a maximum of 500,000 expression evaluations per sub-problem.
Figure 4 shows the symbolic solution rate and accuracy solution rate computed by SRBench, with labels for select ablations we refer to in the subsequent discussion.
Generally, performance increases with the number of enabled components, N .
For a fixed N , GP and LM tend to yield the largest marginal performance gains.
Refer to Figure 9 in Appendix G for a visualization of the marginal performance gains yielded by each component.
Recommendations.
The full uDSR algorithm, using all components and RL pre-training, achieves the best symbolic and accuracy success on SRBench.
Nonetheless, we believe that the choice of SR algorithm should be guided by use case.
Thus, based on the extensive ablation studies presented above, we provide our recommendations for the five integrated solution strategies.
AIF’s recursive simplifications have the attractive property of providing strict improvement over the base SR algorithm, since the root sub-problem in the sub-problem tree is simply the original problem; sub-problems further down the tree only have the ability to improve the end result.
This property comes at the expense of incurring a roughly m -fold increase in computational cost, where m is the number of identified sub-problems.
Empirically, our ablations confirm the strict improvement (e.g., compare ‘A’ with ‘DSR’ in Figure 4), though performance gains exhibit diminishing returns as the number of integrated methods increases.
Thus, we suggest inclusion of AIF for use cases in which computational cost is not an important bottleneck.
Finally, we note that we only considered AIF without its dimensionality analysis component, as this is really a pre-processing step orthogonal to the SR algorithm.
For real-world problems with physical units, we highly recommend that users first perform dimensional analysis on their SR problem.
The best results were obtained using RL pre-training with all other components enabled.
However, in general, LSPT across the ablations yielded mixed results.
For example, the only significant anti-synergy among all components occurs between SL pre-training and GP (compare ‘B1’ and ‘B2’ in Figure 4).
To gain further insight into which use cases may benefit from LSPT, we inspected the initial rewards for each pre-training modality before fine-tuning.
As shown in Figure 6 of Appendix D, both SL and RL pre-trained models yield higher reward than randomly initialized models.
This is especially the case for RL pre-training, where the traces of the initial expressions (see Figure 8 of
Appendix D) appear very close to the real data.
Thus, pre-training can be desirable for use cases with very low budgets (e.g., zero-shot learning), a point reinforced empirically by Biggio et al.
However, as shown in Figure 7 of Appendix D, pre-trained models may have tendencies to become too “sharp” (i.e., exhibiting low entropy), resulting in reduced exploration that can hinder performance during fine-tuning.
We conclude that to be effective in combination with other solution strategies in
SR, pre-trained models must strike the delicate balance between high initial reward and sufficient initial diversity.
Both the LINEAR token and the inclusion of our GP component almost always provide large improvements in both symbolic solution and accuracy solution (e.g., compare ‘C’ and ‘D’ with ‘DSR’ in
Figure 4).
They also do not incur significant additional computational cost given a fixed budget of expressions (see Table 4 of Appendix H).
Thus, we highly recommend their inclusion for most use cases.
A limitation of the LINEAR token, and to a lesser extent the use of GP, is that it is inefficient in improving accuracy relative to its complexity, which we show in Figure 10 of
Appendix I.
Thus, users may not want LINEAR or GP for use cases in which they seek a low-complexity expression.
Conclusion
We introduce a modular framework for SR, in which we carefully integrate five different SR solution strategies in an attempt to maximize the benefits of each and cover each other’s weaknesses.
Our unification strategy focuses on abstraction of the constituent strategies: using DSR as a baseline, we abstract AIF as an algorithmic wrapper, LSPT as a pre-training step, GP as an inner-optimization loop, and LM as a single optimizable token.
Using the SRBench pipeline for benchmarking SR algorithms, uDSR demonstrates state-of-the-art performance across 252 benchmark problems.
For future work, we leave open the questions of using more advanced modules for GP, such as GP-GOMEA (Virgolin et al., 2017), or controller architectures (such as transformers as alternatives to the RNN), as well as the possible benefits from hyperparameter tuning individual modules.
We hope that future work in
SR considers additional strategies for hybridizing and/or abstracting SR solution strategies
Acknowledgments and Disclosure of Funding
We thank Livermore Computing at Lawrence Livermore National Laboratory (LLNL) for the computational resources that enabled this work.
Funding was provided by the LLNL Laboratory Directed
Research and Development project 19-DR-003.
We thank the Computational Engineering Directorate and the Data Science Institute at LLNL for additional support.
This work was performed under the auspices of the U.S. Department of Energy by LLNL under contract DE-AC52-07NA27344.
LLNL-CONF-835375.
Appendix A Expanded pseudocode for uDSR
In Algorithm 2, we provide pseudocode for uDSR in more detail than Algorithm 1.
Note that CONST and LINEAR optimization is only described in lines 22-25 and Lines 26-31, respectively, but these also occur implicitly within GP operators (Line 19) and during RL pre-training (Line 10).
[Table] [Table]
[Table] [Table]
18: for s = 1 , . . .
, S do 19:
T ( s ) GP ← Γ (cid:16) T ( s − 1) GP (cid:17)
▷ Apply GP operator 20:
T GP ← top k from T (1) GP ∪ · · · ∪ T ( S ) GP ▷ Filter best k samples from GP 21:
T ← T RNN ∪ T GP ▷ Join RNN and best GP samples 22: for j = 1 , . . .
, N do 23: if CONST ∈ τ ( j ) then 24: ξ ⋆ ← arg max ξ R ( τ ( j ) ; ξ ) ▷ Maximize R w.r.t.
ξ , e.g. with BFGS 25: τ ( j ) ← ReplaceConstants ( τ ( j ) , ξ ⋆ )
▷ Replace placeholder constants 26: if LINEAR ∈ τ ( j ) then 27:
Let y =
f ( X, LINEAR )
▷ Instantiate expression 28:
Find G such that LINEAR =
G ( X, y )
▷ Solve equation for LINEAR 29:
Compute y ′
=
G ( X, y )
▷ Evaluate G on ( X, y )
i data 30: β
▷ Optimize β , e.g. using LASSO 31: τ ( j ) ← ReplaceCoefficients ( τ ( j ) , β ⋆ )
▷ Replace LINEAR coefficients 32: θ ← train θ (with ψ fixed) on T ▷ Train using fine-tuning objective 33: τ ( i ) ← store best expression for problem P i 34:
τ ⋆ ← Combine ( τ (1) , . . .
, τ ( m ) )
▷ Use AIF to combine m solutions into final solution 35: return τ ⋆
B List of in situ priors and constraints
Autoregressive sampling within DSR affords the opportunity to apply user-specified priors and constraints to the search space in situ , i.e., during the sampling process.
In each step of autoregressive sampling, the DSR controller emits logits ℓ θ .
Before these logits are passed through a softmax, DSR computes a prior denoted ℓ ◦ , which is an additional set of logits that are computed using all available information so far (namely, the partial traversal and the ( X, y ) data).
The prior logits are added to the controller emissions before a token is sampled via
τ i ∼ Categorical ( Softmax ( ℓ θ + ℓ ◦ )) .
Priors allow users to incorporate a priori knowledge during the search.
Notably, when a prior includes −∞ entries, this constrains particular tokens from being sampled, thereby pruning the combinatorial search space.
Such priors have been used by several works using DSR (Petersen et al., 2021a,b;
Landajuela et al., 2021b; Mundhenk et al., 2021; Kim et al., 2021), as well as other non-SR works using similar autoregressive sampling procedures (Popova et al., 2019; Landajuela et al., 2021a).
These priors are enabled during pre-training, both by the problem instance generator and the controller, as well as during fine-tuning.
When integrated with GP, constraints are applied post hoc .
Specifically, if a genetic operator would result in an expression tree that violates any constraint, that genetic operator is reverted, and instead the parent expression tree is returned.
We first summarize priors that have already been included in works utilizing DSR:
• Length constraint.
Following Petersen et al.
(2021a), traversals are constrained to have a length no less than 4 and no more than 64.
The minimum length constraint is implemented by constraining terminal tokens when selecting would end the traversal before the minimum length.
For example, given the partial traversal τ =
[ SIN ] , terminal tokens are constrained because choosing one would end the traversal with length 2.
The maximum length constraint is implemented by constraining unary and/or binary tokens when selecting one, followed by selection of only terminal tokens, would result in a traversal that exceeds the maximum length.
For example, given a partial traversal consisting of 31 instances of + , binary tokens would be constrained because choosing one, followed by 32 terminal tokens, would not complete the end of the traversal by length 64.
• Trigonometry constraint.
Following Petersen et al.
(2021a), trigonometric tokens that would be descendants of other trigonometric tokens are constrained.
For example, given the partial traversal τ =
[ + , x 1 , COS , ÷ , 1.0 ] , SIN and COS are constrained because they would be descendants of COS . • Constant constraint.
Following Petersen et al.
(2021a), we constrain that constant tokens (namely, CONST and 1.0) cannot be the only unique child of a token, because the result would simply be another constant.
For example, given the partial traversal τ =
[ + , CONST ] or τ = [ SIN ] , CONST and 1.0 are constrained.
• CONST repeat constraint.
Following Petersen et al.
(2021a), we limit the maximum number of CONST tokens to 3 per expression, since nonlinear optimization of CONST tokens can be computationally expensive.
• Soft length prior.
Following Landajuela et al.
(2021b), we include a soft length prior that biases expressions to have length somewhere between the minimum and maximum.
See Landajuela et al.
(2021b)
for details.
• Uniform arity prior.
Following Landajuela et al.
(2021b), we include a prior that biases the controller to have an equal likelihood of selecting a binary, unary, or terminal token.
See Landajuela et al.
(2021b)
for details.
Next we describe priors that are newly introduced in this work:
• Available input variables constraint.
Our architecture used for pre-training assumes a maximum of d = 6 input dimensions.
However, not all problems include all 6 dimensions.
Since the dimensionality of an SR problem is known at runtime, we constrain input variables beyond the dimensionality of the given problem.
For example, x 5 and x 6 are always constrained for 4-dimensional problems.
constraint is
when using
• Constraints for LINEAR .
As discussed in Section 4, the LINEAR token requires three constraints.
First, CONST and LINEAR are mutually exclusive.
For example, given the
partial traversal τ =
[ + , CONST ] , LINEAR cannot be included anywhere in this traversal.
Second, LINEAR cannot be a direct ancestor of a non-invertible token (namely, SIN , COS , or SQRT ).
For example, given the partial traversal τ =
[ SQRT , + , x 1 ] , LINEAR cannot be chosen because SQRT is a direct ancestor.
Third, LINEAR can only be selected at most once per traversal.
These constraints are enabled whenever LINEAR is in the library.
• Use each input variable constraint.
We include a constraint that each input variable, x 1 , . . .
, x d for a d -dimensional sub-problem, must appear in the traversal at least once.
This constraint is implemented by preventing an expression from ending if it would end without including all input variables (or the LINEAR token, which may arbitrarily include all input variables).
For example, given the partial traversal τ =
[ × , + , x 1 , x 3 ] for a 3-dimensional problem, terminals other than x 2 or LINEAR are constrained because x 2 has not yet been chosen.
This constraint is enabled for all sub-problems up to 10 dimensions.
C Additional details on integrating LM C.1 Efficient sparse linear regression
From the perspective of the LINEAR token, the linear regression problem to solve can be described as:
[Equation]
where Φ X ∈ R m × k is a Vandermonde matrix (data) computed by applying basis functions Φ( X ) to input data X ,
β ∈ R
k is the vector of coefficients (unknown), and
y ′ ∈
R m (denoted f − 1 ( y ) in
Section 4) is the independent variable (data) computed by solving for the LINEAR token as described in Section 4 and Algorithm 2.
In addition, we would like β to be sparse, with at most l ≪ k non-zero entries, to avoid over-fitting in cases where the functional form of the expression is not correct.
Under this constraint, we have (4):
[Equation]
[Equation]
In our application, we need to solve (5) many thousands of times (each time LINEAR is selected), for identical matrices Φ X (coming from independent variables in the given SR problem) and varying values of y ′ (coming from solving different candidate expression trees for the LINEAR token).
Problem (5) is non-convex.
It can be re-formulated as a mixed-integer linear problem, but solving it with mixed-integer programming techniques would be prohibitively expensive.
LASSO regression (Tibshirani, 1996), as used by (Brunton et al., 2016), is the standard approach to deal with sparse regression, but presents two issues for our application: setting the weights of the L 1 penalty is non-trivial, and it is still one or two orders of magnitude more expensive than regular least squares.
We propose a heuristic approach to solve (5) based on standard least squares, statistical indicators, and constrained quadratic programming.
We start by computing the minimizer of the unconstrained problem in (4), β LS :=
Φ † X y ′ , where Φ † X =
(Φ T X Φ X ) − 1 Φ T X is the Moore-Penrose pseudo-inverse of Φ X , as well as the corresponding p -values of each coefficient β LS for the null hypothesis β LS i
=for i = 1 , . . .
, k ( t -test).
We then find the k − l coefficients with the largest p -values among β LS and store their indices in I 0 ∈ N k − l .
Finally, we solve the following constrained quadratic programming problem:
[Equation]
where C ∈ { 0 , 1 } ( k − l ) × k , with C i,I
i
= 1 for all i = 1 , . .
, k − l , and 0 otherwise, and z ∈ R k − l are dual variables.
The constraints in (6) ensure that all entries at I 0 positions zero in its minimizer β ⋆ , which can be found by solving the following linear system:
[Equation]
The linear system in (7) has significant structure, which we exploit as follows.
Observe that β ⋆ = β LS − (Φ T X Φ X ) − 1 C T z ⋆ , thus C (Φ T X Φ X ) − 1 C T z ⋆ = Cβ LS .
Because Φ X is a Vandermonde matrix,
Original expression tree y ×
sin x + sqrt
LINEAR log x
Partially evaluated tree y × sin(x2) + sqrt
LINEAR log(x1)
Partial application y × (sin( x 2) , · ) +(log( x 1) , · ) sqrt
LINEAR
LINEAR square − ÷ y sin(x2) log(x1)
[Figure]
Φ T X Φ X is symmetric positive definite, and so is C (Φ T X Φ X ) − 1 C T .
Furthermore, since C is an elementary matrix, we can form C (Φ T X Φ X ) − 1 C T very efficiently whenever (Φ T X Φ X ) − 1 is available.
Thus, we solve the linear system in (7) by first solving C (Φ T X Φ X ) − 1 C T z ⋆ =
Cβ LS using Cholesky decomposition, then computing β ⋆ = β LS − (Φ T X Φ X ) − 1 C T z ⋆ .
To efficiently carry out the above procedure repeatedly with Φ X constant for a given SR problem, we compute Φ † X and (Φ T X Φ X ) − 1 only once and store the results for all future solutions to the LINEAR token with the same data (i.e., for the duration of the given SR problem).
Thus, after the first solution to LINEAR , subsequent solutions do not require recomputing Φ † X or (Φ T X Φ X ) − 1 .
C.2 Example illustration of solving LINEAR token
In Figure 5, we show an example of how to obtain the target data programmatically solve for target data f − 1 ( y ) =
G ( x, y ) given F ( x, LINEAR ) , which is then used to compute the sparse coefficients of the LINEAR token according to (6).
D
Additional details on integrating LSPT D.1 Pre-training architecture
Following Biggio et al.
(2021), we use a set transformer (ST) (Lee et al., 2019) as the encoder for each input dataset ( X, y ) .
This architecture is chosen because it is permutation invariant, i.e., the output of the model does not change under any permutation of the rows of ( X, y ) , and because it can process input sets of variable size.
We use the ST architecture introduced in Lee et al.
with the following hyperparameters (we refer to the paper for their precise definitions): embedding dimensions for ST encoder and ST decoder is 20, number of attention heads in ST encoder and ST decoder is 2 and number of seed vectors in Pooling Multihead Attention Block is 2.
To interface with the remainder of the RNN-based controller, the encoding of ( X, y ) is used to produce the initial cell and hidden state of the RNN (illustrated in Figure 1), which in this way acts as a decoder.
For our RNN architecture (consisting of 1 layer of 32 LSTM cells), the decoder outputs h 0 ∈ R 64 ; the 32 first entries are used for the initial RNN cell state and the last 32 for the initial
RNN hidden state.
This way of using an RNN as a decoder is inspired by image-to-caption systems (Karpathy and Fei-Fei, 2015).
Following Biggio et al.
(2021), to allow the same model to treat problems of different dimensionality (i.e., problems with different number of columns in X ), we keep the number of input columns d constant, and implement a zero-padding strategy in which unused columns of X are zero-padded up to d columns.
We use d = 6 for all experiments.
During pre-training, randomly generated problem instances are thus limited to 6 dimensions.
At test time, when problems can yield arbitrary dimension, we disable the encoder architecture whenever dimensionality exceeds 6, using zero-valued initial state and randomly initialized controller weights (just as in ablations with LSPT “off”).
For ablations including AIF, this “triage” is applied per sub-problem.
For example, given a 10-dimensional problem, the root node will not leverage the pre-trained model, but sub-problems up to 6 dimensions will use the pre-trained model.
This flexibility maximizes the use of the pre-trained models, and is made available by virtue of combining LSPT,
AIF, and DSR to enable pre-trained models, enable sub-problems with sufficiently low dimensionality, and enable training without a pre-trained model, respectively.
This is different from the published algorithm in Biggio et al.
(2021), which is restricted to problems with d ≤ 3 dimensions.
The whole encoder-decoder model is trained on millions of expressions and corresponding ( X, y ) datasets that are randomly generated on-the-fly during pre-training.
Problem instance generation is described below.
To enable all possible ablation experiments, we pre-trained four models: two trained using SL and two trained using RL, each with and without the LINEAR token.
Each pre-trained model was trained for 8 days on a single GPU, seeing a total of ∼ 15M different problem instances each.
D.2 Characterization of pre-trained models
In this section, we characterize the pre-trained models p ( τ | θ ⋆ , ψ ⋆ , ( X, y )) where θ ⋆ , ψ ⋆ are pretrained using either SL or RL objectives.
As a reference, we also provide results obtained with p ( τ | θ R , ψ R , ( X, y )) where ( θ R , ψ R ) are randomly initialized.
In Figures 6 and 7, we show the average reward and entropy, respectively, over 1000 expressions sampled from each pre-trained model, computed across the 116 Feynman benchmark SR problems (which all have dimensionality at most 6) stratified by problem dimensionality.
The entropy is computed column-wise over the batch of traversals, i.e., 1 L (cid:80)
Li =1 ¯ H [ { τ ( j ) i } 1000 j =1 ] , where ¯ H is empirical entropy and L is the maximum traversal length in the batch.
Figure 8 shows traces of the top 5% of the 1000 samples from randomly initialized, SL pre-trained, or
RL pre-trained models, over a series of one-dimensional problems.
By visual inspection, both SL and
RL pre-trained models produce traces that are qualitatively closer to the real data than the randomly initialized model.
Comparing traces between SL and RL pre-trained models, RL is closer to the real data than SL.
This is not surprising because the RL objective directly aims to reduce residuals (i.e., generate traces close to the real data).
In contrast, the SL objective only learns a mapping, though in the SR search space, a small deviation in traversal space may result in a large reduction in fitness to the data.
D.3 Problem instance generation
Here, we describe additional details for the SR problem instance generator, used to generate on-the-fly expressions τ and corresponding datasets ( X, y ) during SL and RL pre-training.
Pseudocode for this process is given in Algorithm 3.
To sample pre-training problem instances, we leverage the fact that the distribution induced by
DSR includes a prior (Petersen et al., 2021b).
Specifically, DSR samples each token τ i according to τ i ∼ Categorical ( Softmax ( ℓ θ + ℓ ◦ )) , where ℓ θ are logits emitted by the RNN under current parameters θ , and ℓ ◦ are the logits defined by the prior (see Appendix B and Landajuela et al.
(2021b) for details).
To generate training problems, we ignore emissions ℓ θ and only sample from the prior, ℓ ◦ .
Since softmax is defined only up to a constant, this process is equivalent to sampling from a prior-informed uniform distribution, i.e. ℓ θ ≡ constant.
After sampling an expression τ , we generate ( X, y ) data from it similarly to Biggio et al.
First, if there is a LINEAR token in τ , we sample linear coefficient values using a spike-and-slab distribution.
Similarly, if there are any CONST tokens in τ , we sample their values according to
U ( − 5 , 5) .
Next, we generate 500 data points using the expression induced by τ .
To do so, for each dimension of X , we sample its domain extrema a, b according to U ( − 10 , 10) , then sample each X
[Figure] [Figure]
[Table]
coordinate according to U (min( a, b ) , max( a, b )) .
Finally, we instantiate τ as an expression f and compute y values as y =
f
( X ) .
To ensure numerical stability of τ and the corresponding ( X, y ) data, we re-sample coordinates if | y | > 100 or is undefined until obtaining 500 points.
If 500 points cannot be obtained within 5000 attempts, (e.g., given a, b < 0 and f ( X ) = log( X ) ), τ is discarded and re-sampled.
D.4 RL pre-training using risk-seeking policy gradients
Direct optimization of (3) would maximize expected reward of sampled expressions for a given s =
( X, y ) (to simplify notation, we use s to refer to a dataset ( X, y ) in this section).
However, in
SR we are generally interested in best case performance, i.e.,
[Equation]
As a surrogate (relaxed) version of (8), we consider the following dual formulation of the problem:
[Equation]
where R ε ( θ, ψ, s g ) is the (1 − ε ) -quantile of the random variable R ( τ, s g ) , τ ∼ p ( τ | θ, ψ, s g ) .
While (8) is not suitable for gradient-based optimization (as R ( τ, s ) is not continuous), we can compute the gradient of J riskRL ( θ, ψ ) using the risk-seeking policy gradient introduced in Petersen et al.
(2021a), giving:
[Equation]
A Monte Carlo estimate of the above is given by:
[Equation]
Algorithm input Expression generator p G ( τ ) output Symbolic regression problem instance ( X, y ) 1: τ ∼ p G ( · ) ▷ Sample expression 2: m ← dim ( τ ) ▷ Get dimensionality of τ 3: if LINEAR ∈ τ then 4: c i ∼ 0 .
5 U ( − 10 , 10) + 0 .
5 δ (0)
▷ Sample linear coefficient values 5: for CONST ∈ τ do 6: c ∼ U ( − 5 , 5)
▷ Sample constant value 7: for i ∈ 1 , . . .
, m do 8: a, b ∼ U ( − 10 , 10) ▷ Sample domain extrema 9: for j ∈ 1 , . . .
, 500 do 10:
X i,j ∼ U (min( a, b ) , max( a, b ))
▷ Sample X coordinate 11: y = f
( X ) ▷ Compute y values 12: return ( X, y ) with 1 C the characteristic function for set C , s 1 , . . .
, s B ∼ p G ,   τ (
[Equation]
[Equation] [Equation]
[Equation] [Equation]
∼
[Equation]
and 
[Equation]
where Q α ( x ) is the empirical α quantile of the vector x .
This motivates a procedure to compute an approximation of the gradient ∇ θ,ψ J riskRL ( θ, ψ ) : first, sample B problem instances s i =
( X, y ) i from p G , then, sample N B expressions τ ( j ) i per problem instance s i from p ( τ | θ, ψ, s i ) , then compute (9) using their rewards R ( τ ( j ) i , s i ) and the empirical (1 − ε ) -quantile.
E Short descriptions of the 14 original SRBench baselines
Below, we provide short descriptions of the 14 original SR baseline methods used by SRBench in
Figures 2 and 3.
We refer readers to the original papers for additional details.
• Age-fitness Pareto optimization (AFP): Schmidt and Lipson (2011) propose a GP approach that avoids premature convergence by adding a multidimensional optimization objective that evaluates solutions based on fitness and age.
• AFP with co-evolved fitness estimates (AFP-FE): Schmidt and Lipson (2009) extend standard AFP with a new method for fitness estimation (Schmidt and Lipson, 2008).
• AIF: The AIF algorithm (Udrescu et al., 2020) used in this work, using BF/PF as its underlying search algorithm.
• Bayesian symbolic regression (BSR): Jin et al.
(2019) propose a method that incorporates prior knowledge and samples expression trees from a posterior distribution using an efficient Markov Chain Monte Carlo algorithm that is robust to hyperparameter settings and finds more concise solutions than purely GP-based approaches.
• Deep symbolic regression (DSR): The DSR algorithm (Petersen et al., 2021a) used in this work.
Notably, this uses the predecessor to the version of DSR used in this work, and does not include improvements introduced in Landajuela et al.
(2021b) or Mundhenk et al.
• ε -lexicase selection (EPLEX): La Cava et al.
(2019) propose a method that improves the parent selection process in GP by considering performance on data samples individually instead of in aggregate or average, rewarding expressions that perform well on harder parts of the problem.
• Feature engineering automation tool (FEAT): La Cava et al.
(2018) propose a method that aims at finding simple solutions that generalize well by maintaining an archive of solutions with accuracy-complexity trade-offs to improve generalization and interpretation.
• Fast function extraction (FFX): McConaghy (2011) propose a non-evolutionary technique for SR based on pathwise learning (Friedman et al., 2010) that generates a set of solutions that trade off error versus complexity, while being orders of magnitude faster than GP and providing deterministic convergence.
• GP version of the gene-pool optimal mixing evolutionary algorithm
(GP-GOMEA): Virgolin et al.
(2021) propose to combine GP with linkage learning, an approach that learns a model of interdependencies to estimate what patterns to propagate and proposes small and interpretable solutions.
• GP (gplearn): Basic, Koza-style GP (Koza, 1994) for SR using the open software library gplearn (https://github.com/trevorstephens/gplearn).
This implementation is very similar to the GP component used in uDSR.
• Interaction-transformation evolutionary algorithm (ITEA): de França and Aldeia (2021) introduce a mutation-based evolutionary algorithm based on six mutation heuristics that allows both learning high-performing solutions and extracting the importance of each original feature of a data set as an analytical function.
• Multiple regression genetic programming (MRGP): Arnaldo et al.
(2014) propose a method that decouples and linearly combines sub-expressions via multiple regression on the target variable as a cost-neutral extension to basic GP.
• SR with Non-linear least squares (Operon): Kommenda et al.
(2020) incorporate nonlinear least squares optimization into GP as a local search mechanism for offspring selection to improve generalization.
• Semantic backpropagation genetic programming (SBP-GP): Virgolin et al.
(2019) modify the random desired operator algorithm
(Pawlak et al., 2014), a GP approach using semantic backpropagation, by incorporating the principles of linear scaling, making the algorithm much more effective despite being computationally more expensive
F Hyperparameter selection
Hyperparameters for uDSR are listed in Table 3.
In general, we used default hyperparameters from of each constituent method, with a few exceptions we justify below.
For DSR, we used default hyperparameters given by Mundhenk et al.
(2021), with the exception of increasing maximum expression length from 30 to 64 to accommodate more challenging problems than those attempted in
Mundhenk et al.
Note that, following Mundhenk et al.
(2021), these hyperparameters use priority queue training (Abolafia et al., 2018) during fine-tuning as an alternative to the risk-seeking policy gradient.
For GP, we used all default hyperparameters given by Mundhenk et al. (2021).
For AIF, we used all default hyperparameters given by Udrescu et al.
For LSPT, we used all default hyperparameters for the set encoder given by Lee et al.
For pre-training, we increased the maximum input dimension of the set transformer from 3 (in Biggio et al.
to 6 to accommodate higher dimensional problems.
For RL pre-training, since this has never been performed for SR to date, we estimated the learning rate and batch size, and tuned the entropy weight and risk factor via a small grid search of entropy weight ∈ { 0 , 0 . 3 } and risk factor in ∈ { 0 . 05 , 0 . 2 } .
For LM, since we used a novel formulation of sparse linear regression instead of LASSO, we set the maximum number of terms to 10.
For simplicity, we considered basis functions comprising monomials up to degree 3, e.g. x 21 x 3 , x 32 , or x 1 x 2 .
G Marginal gains
The modularity of uDSR allows one to activate each individual component independently of other components.
We measured the improvement r ( c i, on , c − i ) − r
( c i, off , c − i ) for r = symbolic solution rate and r = accuracy solution rate, for each component c , for all possible combinations of settings for other components
c − i
(LSPT has two “on” states, denoted “RL” and “SL”).
Figure 9 shows that individually enabling AIF, GP, LM (i.e., LINEAR token), and DSR leads to positive improvement in symbolic solution rate and accuracy solution rate, averaged over all possible configurations of other components.
Notably, LM almost always improves the symbolic solution rate, AIF always improves the accuracy solution rate, and DSR provides the highest worst case improvement.
Pre-training with
LSPT is negative on average, but it provides positive improvement in certain cases; specifically, the highest-performing combination uses LSPT (RL) (see Figure 4).
H Runtimes
We evaluate the runtime of uDSR and its constituent components in two ways: (1) runtime per expression evaluation and (2) runtime per SR problem.
For runtime per expression evaluation, we consider the components GP, SINDy, and DSR.
LSPT does not significantly affect runtime per expression because the embedding at test time is computed only once, and the subsequent architecture and algorithm is otherwise the same.
While AIF affects overall runtime by generating sub-problems, it does not affect runtime per expression.
We show average runtime per expression evaluation in Table 4.
The effect of GP ranges from no effect to a
[Table]
N
Mundhenk
[Table]
al.
[Table]
ε
[Table]
Mundhenk
[Table]
al.
[Table]
Mundhenk
[Table]
al.
[Table] [Table]
Mundhenk
[Table]
al.
[Table] [Table]
Landajuela
[Table]
al.
[Table] [Table]
[Table]
Landajuela et al.
(2021b)
[Table]
R
Inverse NRMSE
Landajuela et al.
(2021b)
[Table]
Landajuela et al.
(2021b)
[Table]
Landajuela et al.
(2021b)
[Table]
Estimated
[Table]
Landajuela et al.
(2021b)
[Table]
Landajuela et al.
(2021b)
[Table]
Landajuela et al.
(2021b)
[Table]
S
Mundhenk et al.
[Table]
Mundhenk et al.
[Table]
Mundhenk et al.
[Table]
Mundhenk et al.
[Table]
k
Mundhenk et al.
[Table]
Udrescu et al.
[Table]
Udrescu et al.
[Table]
Udrescu et al.
[Table]
force try time –
Udrescu et al.
Polynomial fit degree –
Udrescu et al.
LSPT
Maximum input dimension d
Estimated
ST embedding size –
Lee et al.
ST attention heads –
Lee et al.
ST seed vectors –
Lee et al.
Learning rate (SL) – 0.
Estimated
Problem instances per batch (SL)
B
Estimated
Learning rate (RL) – 0.
Estimated
Problem instances per batch (RL)
B
Estimated
Samples per problem instance (RL only)
N B
Estimated
Entropy weight (RL only) – 0.
Tuned { 0 , 0 .
3 } Risk factor (RL only) ε 0.
Tuned { 0 . 05 , 0 . 2 }
LM
Basis functions Φ
Monomials
Estimated
Maximum monomial degree
D
Estimated
Maximum non-zero terms –
Estimated
2-fold increase in runtime.
LM actually decreases runtime in many cases.
This can occur because
LINEAR “competes” with the CONST token, as both are terminal tokens and the tokens are mutually exclusive (see Appendix B).
Since (non-linear) CONST optimization is typically much slower than
LINEAR optimization, this results into an overall reduction in runtime.
Lastly, disabling DSR (i.e., not training the controller) has little to no effect on runtime, demonstrating that backpropagation is not a computational bottleneck for uDSR at test time.
Average runtime per SR problem is given in Table 5 for all uDSR ablations.
Because uDSR uses early stopping, these values are largely dominated by how early and how often each ablation achieves the early stopping criterion (defined as in Petersen et al.
(2021a)
as achieving a normalized MSE less than 10 − 16 ), as opposed to reaching all 500,000 expression evaluations per problem-instance or the 24 hr walltime limit.
The fold-increase in computational cost incurred by AIF is also evident in these results.
[Table] [Table]
I Accuracy vs complexity for GP and LM components
In our ablations on ground-truth SR problems, both GP and the LINEAR token (LM) yield considerable gains in symbolic solution rate and accuracy solution rate.
However, GP can lead to longer expressions due to crossover operators (referred to as “bloat”)
[Table]
while LINEAR can result in longer expressions by adding many linear terms.
To assess the trade-offs between accuracy and complexity
[Table]
we performed a small ablation study comprising four configurations: GP ∈ [on, off] × LM ∈ [on, off], with all other components enabled.
In Figure
[Table]
we see that the four configurations form a Pareto front in terms of accuracy ( R 2 on held-out test data) vs complexity (expression length, computed as number of nodes
[Figure]
in the SymPy-parsed expression tree).
This reinforces how the modularity of uDSR allows it to be configured based on use case.
J Software, data, and computational resources
Our implementation of uDSR combines several open-source software projects.
For DSR, we use the Deep Symbolic Optimization software framework 3 , which includes the base implementation of
DSR and facilities to integrate GP using the Distributed Evolutionary Algorithms in Python software(Fortin et al., 2012).
We integrate AIF using its open-source implementation 5 .
We implement the
LINEAR token using standard numerical libraries.
The set transformer used for pre-training is based on an open-source TensorFlow implementation 6 .
Our empirical evaluation uses SRBench 7 .
The
SRBench benchmark datasets are taken from the open-source PMLB database 8 (Olson et al., 2017).
Pre-training experiments were performed on a single NVIDIA TitanX
GPU.
All other experiments were performed on a single TOSS 3 2.4 GHz CPU.
A total of 73,700 independent trials of uDSR were run across all experiments.
Approximately 1% of runs were stopped early at the 24 hr walltime limit.
K Originally published methods versus corresponding uDSR components K.1 Methodological differences
uDSR integrates the key capabilities (listed in Table 1) of each of the five integrated solution strategies.
In this section, we identify the closest standalone SR algorithm in the literature corresponding to each single-component uDSR ablation and summarize the key methodological differences.
AIF.
The key difference between the published, standalone AIF algorithm (Udrescu et al., 2020) and our use of recursive simplification is that the original AIF obtains expressions using BF/PF as its underlying SR algorithm, whereas our module obtains expressions sampled from a parameterized distribution over expressions.
Another detail is that the original AIF used a slightly different set of tokens that were tailored to the Feynman benchmark problems, whereas our component used a more generic set of tokens.
LSPT.
The closest standalone algorithm to our LSPT component is NeSymReS (Biggio et al., 2021).
The key difference between NeSymReS (Biggio et al., 2021) and our LSPT component lies in how expressions are sampled at test time.
NeSymReS uses beam search at test time to identify high-likelihood expressions under the model conditioned on the test problem.
In contrast, our LSPT module fine-tunes the controller (conditioned on the test problem) by continuing to train non-encoder parameters at test time.
Differences during pre-training are that NeSymReS generates problems of up to 3 input variables, whereas our module generates problems of up to 6, and that NeSymReS trains using SL, whereas our module can alternative train using RL using a modified version of the risk-seeking policy gradient.
GP.
Our GP component is most similar to a vanilla, “Koza-style” GP algorithm
(Koza, 1994).
There are two key differences between the published, standalone GP algorithm
(Koza, 1994) and our GP component.
(1) In standalone GP, starting populations are sampled using the traditional “half-and-half” method, in which half the starting population is sampled using the “full” protocol and the other half is sampled using the “grow” protocol, as defined in Luke and Panait (2001).
In contrast, uDSR’s
GP starting populations are sampled from the controller.
In standalone GP, there is only a single starting population per run, and all generations are performed until the algorithm completes.
In our module, a relatively small number of generations are performed before a new starting population is sampled from the controller.
Thus, GP module operates in a random restart-like fashion, restarting with new starting populations as the controller learns.
LM.
Our LM component is most similar to the SINDy
algorithm (Brunton et al., 2016).
The key difference between the published, standalone SINDy algorithm (Brunton et al., 2016) and our
LINEAR token is that the original SINDy algorithm finds expressions of fixed functional form (given basis functions Φ ), whereas LINEAR occurs within a functional form “wrapper” sampled from the distribution over expressions.
Essentially, the original SINDy algorithm computes a single functional form with traversal τ =
[ LINEAR ] , whereas even the LM-only uDSR ablation includes LINEAR within nonlinear functional forms.
Two minor differences are that: (1) The original SINDy algorithm used
LASSO to learn sparse coefficients, whereas our module uses a modified version of least squares, and (2) The original SINDy algorithm chose both linear and nonlinear basis functions in Φ , e.g. sin( x 1 ) , whereas we used a simpler set of monomials up to degree 3.
DSR.
Unlike the other four components, our DSR module does not exhibit any algorithmic changes relative to the published DSR in Landajuela et al.
(2021b), as DSR was used as a starting point into which we integrated the other four solution strategies.
However, note that in Figures 2 and 3, the
SRBench baseline used the original DSR version in Petersen et al.
(2021a) without the improvements introduced in Landajuela et al.
(2021b).
K.2 Performance differences
The methodological differences described above may result in differences in performance.
In Figure 11, we compare each single-component uDSR ablation to the corresponding “closest” standalone SR method.
Note that Figure 11 uses the same raw data as in Figures 2 and 4, but is reformatted here to enable facile comparison.
Also note that, besides the methodological differences listed above, there are additional differences in the experimental setting used between the original papers and this work.
For example, AIF uses a different token set than the AIF-only uDSR ablation.
For simplicity, for the original methods, we use the experimental workflow used in the original papers.
AIF is the only standalone algorithm that outperforms the corresponding AIF-only uDSR ablation.
This may be due to AIF using a different token set tailored to solving the Feynman benchmark problems, and/or because BF/PF outperforms sampling from a randomly-initialized controller.
The
GP-only uDSR ablation greatly outperforms standalone GP.
This is consistent with the finding by
Mundhenk et al.
(2021) that standard GP excels in a random restart-like fashion.
The two LSPT-only uDSR ablations (particularly with RL pre-training) outperform standalone NeSymReS, which may be attributed to the fact that standalone NeSymReS was only trained on SR problems of up to 3 input variables, whereas most SRBench problems have > 3 input variables.
(For problems with > 3 input variables, it cannot select input variables beyond x 3 .)
The DSR-only uDSR ablation outperforms the standalone DSR used by SRBench (Petersen et al., 2021a); however, when using the updated version
GP
AIF
DSR
LM
LSPT 0.
S y m b o li c s o l u t i o n r a t e
SL
RL
Closest standalone algorithm uDSR ablation
GP
AIF
DSR
LM
LSPT 0.
A cc u r a c y s o l u t i o n r a t e
SL
RL of DSR given by Landajuela et al.
(2021b), performance is identical to the DSR-only uDSR ablation, by construction.
The LM-only uDSR ablation greatly outperforms standalone SINDy.
This is because
SINDy only optimizes a single linear solution, whereas the LM-only uDSR ablation is still able to embed the LINEAR token into highly non-linear functional forms (in particular, rational expressions).
L Illustrative example of solution discovery
By tracing the formation of an expression solution, it is possible to identify which tokens in the solution τ (i.e. which parts of the expression) can be attributed to which individual uDSR components.
In Figure 12, we provide an illustrative example of this process, which highlights how the five uDSR components work together to find a final solution.
In this illustrative example: (1) AIF first identifies multiplicative separability , which simplifies the original problem P into two sub-problems, P 1 and
P 2 .
The solution to P is then the product of the solutions to P 1 and P 2 ; thus, AIF essentially learned that the root node of the expression tree is × .
The simple solution to P 1 is found quickly by DSR.
(3) DSR, conditioning the controller on
P 1 data ( X, y ) 2 using the LSPT encoder, samples candidate solutions to P 2 .
The GP crossover operator recombines two candidate solutions into the solution to P 2 .
(5) Lastly, since the solution to P 2 includes a LINEAR token, the linear combination of basis functions is optimized according to the ( X, y ) data of P 2 .
M Additional details
M.1 Behavior of LINEAR for non-finite target data.
When solving for target data f − 1 ( y ) given F ( x, LINEAR ) (by performing successive application of the root node’s inverse operations), it may be the case that infinite or undefined values arise.
For example, given the traversal τ = [ EXP , LINEAR ] and negative values in y , the function for target data f − 1 ( y ) =
log( y ) produces undefined values.
In these cases, the LINEAR token returns a default value of 1 , i.e., coefficients β are all zero except for the constant.
This process is similar to the CONST token defaulting to a value of 1 when L-BFGS-B encounters undefined values (Petersen et al., 2021a).
Note that if the candidate functional form is correct (e.g., the traversal τ = [exp, LINEAR] when the ( X, y ) data is generated from ground-truth expression e x 21 +0 .
5 x 2 ), then, by construction, it will never be the case that ¯ f − 1 ( y ) yields undefined values.
[Table]
AIF separability simplifying P into
P
[Table]
and
[Table]
latexit> GP crossover operatorrecombinesthecandidates to form P<latexit sha1_base64="ikd208yzkkB4VGDVSPYns6aeJ60=">AAACoXicbVHJjtNAEG2bbQhbgCOXFglSRmIiO2I7jkBikTiEJTMjxSEqt8tJK+1uq7sdjdXyf/Br/AYfgGg7OTAz1OmpXi2vXqWl4MZG0a8gvHb9xs1bB7d7d+7eu/+g//DRiVGVZjhjSih9loJBwSXOLLcCz0qNUKQCT9PNu5Y/3aI2XMnvti5xUcBK8pwzsD617P9MpOIyQ2npNyW2SO0aqVGiamlqFU2S3jApwK4ZCDdtlnQypLnSXZ1LOgHuMy/wg0aUDU0snlvDXKsHdNP4ERuU7ZQRGMolfc9XlUb68vA5rTmKjMvVbkcK2uXND3cUN6P6cLjsD6Jx1AW9CuI9GJB9TJf930mmWFX4U5gAY+ZxVNqFA205E9j0kspgCWwDK5x7KKFAs3DdAQ195jNZd1euvBVd9t8OB4UxdZH6ytYLc5lrk//j5pXN3ywcl2VlUbLdorwSrbPtO2jGNTIrag+Aae61UrYGDcz6p12YlG15afaqz3eye96k+LIlV8HJZBy/Gr/4Mhkcv93bdUCekKdkRGLymhyTj2RKZoSRP8EwOArG4SD8FE7Dr7vSMNj3PCYXIpz/BYamzks=</latexit>
Solve the solution to
P 2 for the linear token (as in Figure 5), yielding
[Equation]
<
[Equation]
="
[Equation]
AAACRnicbVA9TxtBEJ1zPiDOlxPKNKuYSKmsO4QCTSREmpSgxIDks9Dc3pxZsbd72tmzsCx+DX8G2qTKn0iBhGhZn68IkFc9vZm38/ZllVbs4/hP1Hny9NnzldUX3ZevXr9523v3/oBt7SQNpdXWHWXIpJWhoVde01HlCMtM02F2+m0xP5ySY2XNTz+raFzixKhCSfRBOu59TY1VJifjxQ+rpyQK68R6mpHHdVGzMhPBFTomkabdxRF0wtHEEXPj78eDuIF4TJKW9KHF3nHvOs2trMtwT2pkHiVx5cdzdF5JTefdtGaqUJ7ihEaBGiyJx/Pmm+fiU1DyJmBhQ95G/dcxx5J5VmZhs0R/wg9nC/F/s1Hti+3xXJmq9mTk8lBRa+GtWHQmcuVIej0LBKVTIauQJ+hQ+tDsvZfyqaq4TX22jN0NJSUPK3lMDjYGyZfB5v5Gf2e3rWsVPsBH
[Equation]
AwJbMEOfIc9GIKEC7iCX
[Equation]
A7uoz
[Equation]
RjfR7XK1E7WeNbiHDtwBqzqyMg==</latexit>
Solve for  using sparse linear regression <
[Equation]
Zpgok8Ki
[Equation]
NjuHHy6szo3HAZcS20eImZBCgUDFCjhITPA0kjCfTS7XM3v52Cs0OoWlxmMUjZRIhGcoaPGnatQaaFiUEhvp0AToZikRVg9Lq5ECt8NgCppiLBAy4uVDjNlGYYt1DNQVFg6EXMHouX5uNP1e35VdBsENeiSuq7Hnb9hrHmeOnkumbXDwM9wVDCDgksoW2FuIWN8xiYwdFCxFOyoqMyV9ItjYppo49rZr9jnFwVLrV2mkdtMGU7ty9mKfG02zDE5GxVCZTmC4muhJJcUNV1FSGNhgKNcOsC4Ec4r5VNmGEcX9ManeC4yW7terG23XEjBy0i2wV
[Equation]
k43y2v8BMke3bQ==</latexit>
The ﬁnal linear token is given by: <latexit sha1_base
[Equation]
i/aoiJxNMUoD4tEetTNMw9p4EjU=
[Table]
[Figure]
M.2 Additional details for SRBench
SRBench includes metrics for symbolic-based performance, accuracy-based performance, and solution complexity.
“Symbolic solution rate” is computed by comparing the ground-truth SymPy expression to the SymPy-parsed expression returned by the algorithm.
Floating-point values are first rounded to 3 decimals.
Expressions are then deemed symbolically equivalent if their symbolic difference is “0” or their symbolic quotient is “1.”
“Accuracy solution rate” is based on the coefficient of determination ( R 2 ) on a held-out test dataset exceeding 0.999.
R 2 is a useful metric because it is normalized across datasets, allowing averaging performance metrics across multiple problems.
“Complexity” is defined by the number of nodes in the SymPy-parsed expression tree.
Most of the baselines in Figures 2 and 3 report results directly curated by SRBench.
(SRBench enables reproducibility, including across random number seeds, making such direct comparisons appropriate.)
The original SRBench study (La Cava et al., 2021) already included baselines for standalone GP, DSR, and AIF.
Since SINDy and NeSymReS (the standalone SR methods closest to our LM and LSPT components, respectively) were not included in the original SRBench study,
we
[Table]
executed these ourselves using the SRBench pipeline.
For NeSymReS (Biggio et al., 2021)
[Table]
since it was trained on only up to 3 input variables, it ignores additional input variables if present
[Table]
M.3 Extrapolation and recommendations for LSPT
A key aspect regarding recommended use of LSPT is knowledge of how well the distribution over ( X, y ) datasets sampled by problem instance generator p G is thought to be representative of the user’s problems of interest.
In particular, a key characteristic of p G is the extrema considered for the domain of input variables ( [ − 10 , 10] in this study).
A test time problem outside this domain constitutes an extrapolation problem.
While uDSR has mechanisms to escape such extrapolation cases, either by fine-tuning or via components that do not depend on input variable domains (e
[Table]
g.
AIF or GP), including the pre-trained network may be unnecessary or harmful.
We investigate this in
Table
[Table]
in which we show the performance of uDSR and baselines against the six SRBench groundtruth problems whose input data exceed that considered during pre-training, [ − 10 , 10] .
Notably,
NeSymReS (which does not fine-tune) yields 0% in both symbolic and accuracy solution rates, suggesting that pre-training without fine-tuning on problems outside the pre-training input ranges can result in catastrophic failure.
On the other hand, uDSR yields 100% in both symbolic and accuracy solution rates over these problems, suggesting that pre-training was not harmful, and that the algorithm’s fine-tuning and/or other components can render uDSR robust to this type of extrapolation.
In Figure 13, we also show how the minimum and maximum input variables of the Feynman, Strogatz, and black-box SRBench problems compare to those of the pre-training dataset generator.
[Figure]