D EEP SYMBOLIC REGRESSION : R ECOVERING MATHEMATICAL EXPRESSIONS FROM DATA VIA RISKSEEKING POLICY
GRADIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Discovering the underlying mathematical expressions describing a dataset is a core challenge for artiﬁcial intelligence.
This is the problem of symbolic regression .
Despite recent advances in training neural networks to solve complex tasks, deep learning approaches to symbolic regression are underexplored.
We propose a framework that leverages deep learning for symbolic regression via a simple idea: use a large model to search the space of small models.
Speciﬁcally, we use a recurrent neural network to emit a distribution over tractable mathematical expressions and employ a novel risk-seeking policy gradient to train the network to generate better-ﬁtting expressions.
Our algorithm outperforms several baseline methods (including Eureqa, the gold standard for symbolic regression) in its ability to exactly recover symbolic expressions on a series of benchmark problems, both with and without added noise.
More broadly, our contributions include a framework that can be applied to optimize hierarchical, variable-length objects under a blackbox performance metric, with the ability to incorporate constraints in situ, and a risk-seeking policy gradient formulation that optimizes for best-case performance instead of expected performance.
I NTRODUCTION
Understanding the mathematical relationships among variables in a physical system is an integral component of the scientiﬁc process.
Symbolic regression aims to identify these relationships by searching over the space of tractable (i.e. concise, closed-form) mathematical expressions to best ﬁt a dataset.
Speciﬁcally, given a dataset ( X, y ) , where each point X i ∈ R n and y i ∈ R , symbolic regression aims to identify a function f : R n → R that best ﬁts the dataset, where the functional form of f is a short mathematical expression.
The resulting expression can be readily interpreted and/or provide useful scientiﬁc insights simply by inspection.
In contrast, conventional regression imposes a single model structure that is ﬁxed during training, often chosen to be expressive (e.g. a neural network) at the expense of being easily interpretable.
Symbolic regression exhibits several unique features that make it an excellent test problem for benchmarking automated machine learning (AutoML) and program synthesis methods: (1) there exist well-established, challenging benchmark problems with stringent success criteria (White et al., 2013); (2) there exist well-established baseline methods (most notably, the Eureqa algorithm (Schmidt & Lipson, 2009)); and (3) the reward function is computationally expedient, allowing sufﬁcient experiment replicates to achieve statistical signiﬁcance.
Most other AutoML tasks, e.g. neural architecture search (NAS), do not exhibit these features; in fact, even simply evaluating the efﬁciency of the discrete search itself is a known challenge within NAS (Yu et al., 2019).
The space of mathematical expressions is discrete (in model structure) and continuous (in model parameters), growing exponentially with the length of the expression, rendering symbolic regression a challenging machine learning problem—thought to be NP-hard (Lu et al., 2016).
Given this large, combinatorial search space, traditional approaches to symbolic regression typically utilize evolutionary algorithms, especially genetic programming (GP) (Koza, 1992; Schmidt & Lipson, 2009;
B¨ack et al., 2018).
In GP-based symbolic regression, a population of mathematical expressions is “evolved” using evolutionary operations like selection, crossover, and mutation to improve a ﬁtness function.
While GP can be effective, it is also known to scale poorly to larger problems and to exhibit high sensitivity to hyperparameters.
Deep learning has permeated almost all areas of artiﬁcial intelligence, from computer vision (Krizhevsky et al., 2012) to optimal control (Mnih et al., 2015).
However, deep learning may seem incongruous with or even antithetical toward symbolic regression, given that neural networks are typically highly complex, difﬁcult to interpret, and rely on gradient information.
We propose a framework that resolves this incongruity by tying deep learning and symbolic regression together with a simple idea: use a large model (i.e. neural network) to search the space of small models (i.e. symbolic expressions).
This framework leverages the representational capacity of neural networks to generate interpretable expressions, while entirely bypassing the need to interpret the network itself.
We present deep symbolic regression (DSR), a gradient-based approach for symbolic regression based on reinforcement learning.
In DSR, a recurrent neural network (RNN) emits a distribution over mathematical expressions.
Expressions are sampled from the distribution, instantiated, and evaluated based on their ﬁtness to the dataset.
This ﬁtness is used as the reward signal to train the RNN using a novel risk-seeking policy gradient algorithm.
As training proceeds, the RNN adjusts the likelihood of an expression relative to its reward, assigning higher probabilities to better expressions.
We demonstrate that DSR outperforms several baseline methods, including two commercial software algorithms.
We summarize our contributions as follows: (1) a novel method for symbolic regression that outperforms several baselines on a set of benchmark problems, (2) an autoregressive generative modeling framework for optimizing hierarchical, variable-length objects that accommodates in situ constraints, and (3) a novel risk-seeking policy gradient objective and accompanying Monte Carlo estimation procedure that optimizes for best-case performance instead of average performance.
R ELATED W ORK
Deep learning for symbolic regression.
Several recent approaches leverage deep learning for symbolic regression.
AI Feynman (Udrescu & Tegmark, 2020) propose a problem-simpliﬁcation tool for symbolic regression.
They use neural networks to identify simplifying properties in a dataset (e.g. multiplicative separability, translational symmetry), which they exploit to recursively deﬁne simpliﬁed sub-problems that can then be tackled using any symbolic regression algorithm.
In
GrammarVAE, Kusner et al.
(2017) develop a generative model for discrete objects that adhere to a pre-speciﬁed grammar, then optimize them in latent space.
They demonstrate this can be used for symbolic regression; however, the method struggles to exactly recover expressions, and the generated expressions are not always syntactically valid.
Sahoo et al.
(2018) develop a symbolic regression framework using neural networks whose activation functions are symbolic operators.
While this approach enables an end-to-end differentiable system, backpropagation through activation functions like division or logarithm requires the authors to make several simpliﬁcations to the search space, ultimately precluding learning certain simple classes of expressions like √ x or sin( x/y ) .
We address and/or directly compare to these works in Appendices C and E.
AutoML and program synthesis.
Symbolic regression is related to both automated machine learning (AutoML) and program synthesis, in that they all involve a search for an executable program (i.e. expression) to solve a particular task (i.e. to ﬁt data) (Abolaﬁa et al., 2018;
Devlin et al., 2017;
Riedel et al., 2016).
More speciﬁcally, our framework has many parallels to a body of works within AutoML that use an autoregressive RNN to deﬁne a distribution over discrete objects and use reinforcement learning to optimize this distribution under a black-box performance metric (Zoph & Le, 2017; Ramachandran et al., 2017; Bello et al., 2017;
Abolaﬁa et al., 2018).
For example, in neural architecture search (Zoph & Le, 2017), an RNN searches the space of neural network architectures, encoded by a sequence of discrete “tokens” specifying architectural properties (e.g. number of neurons) of each layer.
The length of the sequence is ﬁxed or scheduled during training; in contrast, our framework deﬁnes a search space that is both inherently hierarchical and variable length.
Ramachandran et al.
(2017) search the space of neural network activation functions.
While this space is hierarchical in nature, the authors (rightfully) constrain it substantially by positing a functional unit that is repeated sequentially, thus restricting their search space back to a ﬁxed-length sequence.
However, a repeating-unit constraint is not practical for symbolic regression because the ground truth expression may have arbitrary structure.
Autoregressive models.
The RNN-based distribution over expressions used in DSR is autoregressive, meaning each token is conditioned on the previously sampled tokens.
Autoregressive models have proven to be useful for audio and image data (Oord et al., 2016a;b) in addition to the AutoML works discussed above; we further demonstrate their efﬁcacy for hierarchical expressions.
GraphRNN deﬁnes a distribution over graphs that generates an adjacency matrix one column at a time in autoregressive fashion (You et al., 2018).
In principle, GraphRNN could be constrained to deﬁne a distribution over expressions, since trees are a special case of graphs.
However, GraphRNN constructs graphs breadth-ﬁrst, whereas expressions are more naturally represented using depth-ﬁrst traversals (Li et al., 2005).
Further, DSR exploits the hierarchical nature of trees by providing the parent and sibling as inputs to the RNN, and leverages the additional structure of expression trees that a node’s value determines its number of children (e.g. cosine is a unary operator and thus has one child).
Risk-aware reinforcement learning.
Many of the AutoML methods discussed above suffer from what we call the “expectation problem.”
That is, policy gradient methods are fundamentally suited for optimizing expectations ; however, domains like neural architecture search and symbolic regression are evaluated by the few or single best-performing samples.
Thus, there is a mismatch between the training objective function and the true desired objective: to maximize best-case performance.
Abolaﬁa et al.
(2018) address the expectation problem by maintaining a priority queue of the best seen samples and using supervised learning to increase the likelihood of those top samples.
Similarly,
Liang et al.
(2018) use a memory buffer to augment a policy gradient with off-policy training.
These methods, however, only apply in the context of reinforcement learning environments with both deterministic transition dynamics and deterministic rewards.
In contrast, the risk-seeking policy gradient introduced here is general, applying to any reinforcement learning environment and any stochastic policy gradient algorithm trained using batches, e.g. on Atari using proximal policy optimization (Schulman et al., 2017).
Lastly, our risk-seeking policy gradient is closely related to the
EPOpt-
ε algorithm used for robust reinforcement learning (Rajeswaran et al., 2016), which is based on a risk-averse policy gradient formulation (Tamar et al., 2014).
M ETHODS
Our overall approach involves representing mathematical expressions as sequences, developing an autoregressive model to generate expressions under a pre-speciﬁed set of constraints, and developing a risk-seeking policy gradient to train the model to generate better-ﬁtting expressions.
3. G ENERATING EXPRESSIONS WITH A RECURRENT NEURAL NETWORK
We leverage the fact that mathematical expressions can be represented using symbolic expression trees .
Expression trees are a type of binary tree in which internal nodes are mathematical operators and terminal nodes are input variables or constants.
Operators may be unary (i.e. one argument, such as sine) or binary (i.e. two arguments, such as multiply).
Further, we can represent an expression tree as a sequence of node values or “tokens” by using its pre-order traversal (i.e. by visiting each node depth-ﬁrst, then left-to-right).
This allows us to generate an expression tree sequentially while still maintaining a one-to-one correspondence between the tree and its traversal.
Thus, we represent an expression τ by the pre-order traversal of its corresponding expression tree.
We denote the i th token of the traversal as τ i and the length of the traversal as | τ | = T .
Each token has a value within a given library L of possible tokens, e.g. { + , − , × , ÷ , sin , x } .
We generate expressions one token at a time along the pre-order traversal (from τ 1 to τ T ).
A categorical distribution with parameters ψ deﬁnes the probabilities of selecting each token from L .
To capture the “context” of the expression as it is being generated, we condition this probability upon the selections of all previous tokens in that traversal.
This conditional dependence can be achieved very generally using an RNN with parameters θ that emits a probability vector ψ in an autoregressive manner.
Speciﬁcally, the i th output of the RNN passes through a softmax layer (with shared weights across time steps) to produce vector ψ ( i ) , which deﬁnes the probability distribution for selecting the i th token
τ i , conditioned on the previously selected tokens τ 1:( i − 1) .
That is, p ( τ i | τ 1:( i − 1) ; θ )
=
ψ ( i ) L ( τ i ) , where L ( τ i ) is the index in L corresponding to τ i .
The likelihood of the entire sampled expression is simply the product of the likelihoods of its tokens: p ( τ | θ ) =
(cid:81) | τ | i =1 p ( τ i | τ 1:( i − 1) ; θ ) = (cid:81) | τ | i =1 ψ ( i ) L ( τ i ) .
Library + 𝐬𝐢𝐧 𝐜𝐨𝐧𝐬𝐭 − ×
÷ 𝐜𝐨𝐬 𝐥𝐨𝐠 𝐞𝐱𝐩 𝒚 𝒙
Sampled token:
Pre-order traversal
Parent:
Sibling:
RNN:
Categorical distribution: ÷ 𝐬𝐢𝐧 × 𝐜𝐨𝐧𝐬𝐭 𝒙 𝐥𝐨𝐠 𝒚
÷ 𝐬𝐢𝐧 ×
× ÷ 𝐥𝐨𝐠 𝐜𝐨𝐧𝐬𝐭
𝐬𝐢𝐧 ÷ 𝐬𝐢𝐧 × 𝐜𝐨𝐧𝐬𝐭 𝒙 𝐥𝐨𝐠 𝒚      1 Expression tree
A
B
C
An example of the sampling process is illustrated in Figure 1; pseudocode is provided in Algorithm 2 in Appendix A. Note that different samples from the distribution have different tree structures of different size; thus, the search space is inherently both hierarchical and variable length.
Providing hierarchical inputs to the RNN.
Conventionally, the input to the RNN when sampling a token would be a representation of the previously sampled token.
However, the search space for symbolic regression is inherently hierarchical, and the previously sampled token may actually be very distant from the next token to be sampled in the expression tree.
For example, the ﬁfth and sixth tokens sampled in Figure 1 are adjacent nodes in the traversal but are four edges apart in the expression tree.
To better capture hierarchical information, we provide as inputs to the RNN a representation of the parent and sibling nodes of the token being sampled.
We introduce an empty token for cases in which a node does not have a parent or sibling.
Pseudocode for identifying the parent and sibling nodes given a partial traversal is provided in Subroutine 1 in Appendix A.
Constraining the search space.
Under our framework, it is straightforward to apply a priori constraints to reduce the search space.
To demonstrate, we impose several simple, domain-agnostic constraints: (1) Expressions are limited to a pre-speciﬁed minimum and maximum length.
We selected minimum length of 4 to prevent trivial expressions and a maximum length of 30 to ensure expressions are tractable.
The children of an operator should not all be constants, as the result would simply be a different constant.
The child of a unary operator should not be the inverse of that operator, e.g. log(exp( x )) is not allowed.
(4) Descendants of trigonometric operators should not be trigonometric operators, e.g. sin( x + cos( x )) is not allowed.
While still semantically meaningful, such composed trigonometric operators do not appear in virtually any scientiﬁc domain.
We apply these constraints in situ (i.e. concurrently with autoregressive sampling) by zeroing out the probabilities of selecting tokens that would violate a constraint.
Pseudocode for this process is provided in Subroutine 2 in Appendix A.
This process ensures that samples always adhere to all constraints, without rejecting samples post hoc.
In contrast, imposing constraints with GP requires rejecting evolutionary operations post hoc (Fortin et al., 2012), which can be problematic (Craenen et al., 2001), and as we show in our experiments, can actually reduce performance.
Reward function.
Once a pre-order traversal is sampled, we instantiate the corresponding symbolic expression and evaluate it with a reward function.
A standard ﬁtness measure in GP-based symbolic regression is normalized root-mean-square error (NRMSE), the root-mean-square error normalized by the standard deviation of the target values, σ y .
That is, given a dataset ( X, y ) of size n and candidate expression f , NRMSE = 1 σ y (cid:113) 1 n (cid:80) ni =
( y i − f ( X i ))
To bound the reward function, we apply a squashing function: R ( τ ) = 1 / (1 + NRMSE ) .
Constant optimization.
If the library L includes the constant token, sampled expressions may include several constant placeholders.
These can be viewed as parameters ξ of the symbolic expression, which we optimize by maximizing the reward function: ξ (cid:63) =
arg max ξ R ( τ ; ξ ) , using a nonlinear optimization algorithm, e.g. BFGS (Fletcher, 2013).
We perform this inner optimization loop for each sampled expression as part of the reward computation before performing each training step.
3. T RAINING THE RNN USING POLICY GRADIENTS
Standard policy gradient.
Now that we have a distribution over mathematical expressions p ( τ | θ ) , we ﬁrst consider the standard policy gradient objective to maximize J std ( θ ) , deﬁned as the expectation of a reward function R ( τ ) under expressions from the distribution: J std ( θ ) .
=
E τ ∼ p ( τ | θ )
[ R ( τ )] .
The standard REINFORCE policy gradient (Williams, 1992) can be used to maximize this expectation via gradient ascent:
[Equation]
This result allows one to estimate the expectation using samples from the distribution.
Speciﬁcally, an unbiased estimate of ∇ θ J std ( θ ) can be obtained by computing the sample mean over a batch of N sampled expressions T
= { τ ( i ) } Ni =1 :
[Equation]
This is an unbiased gradient estimate, but in practice it has high variance.
To reduce variance, it is common to subtract a baseline function b from the reward.
As long as the baseline is not a function of the current batch of expressions, the gradient estimate is still unbiased.
Common choices of baseline functions are a moving average of rewards or an estimate of the value function.
Risk-seeking policy gradient.
The standard policy gradient objective, J std ( θ ) , is deﬁned as an expectation .
This is the desired objective for control problems in which one seeks to optimize the average performance of a policy.
However, in domains like symbolic regression, program synthesis, or neural architecture search, the ﬁnal performance is measured by the single or few best-performing samples found during training.
Similarly, one might be interested in a policy that achieves a “high score” in a control environment (e.g. Atari).
For such problems, J std ( θ ) is not an appropriate objective, as there is a mismatch between the objective being optimized and the ﬁnal performance metric; this is the “expectation problem.”
To address this disconnect, we propose an alternative objective that focuses learning only on maximizing best-case performance .
We ﬁrst deﬁne R ε ( θ ) as the (1 − ε )
-quantile of the distribution of rewards under the current policy.
We then propose a new learning objective, J risk ( θ ; ε ) , parameterized by ε :
[Equation]
This objective aims to increase the reward of the top ε fraction of samples from the distribution, without regard for samples below that threshold.
This objective bears close resemblance with εconditional value at risk (CVaR), for which the “ ≤ ” symbol is used instead of “ ≥ ” and the ε -quantile of rewards is used instead of the (1 − ε )
-quantile.
Optimizing CVaR is a form of risk-averse learning that results in a policy that is robust against catastrophic outcomes (Tamar et al., 2014; Rajeswaran et al., 2016).
In contrast, optimizing J risk ( θ ; ε ) yields a risk-seeking policy gradient that aims to increase best-case performance at the expense of lower worst-case and average performances.
Next, we show the analogous policy gradient of J risk ( θ ; ε ) and how to estimate it via Monte Carlo sampling.
Proposition 1.
Let J risk ( θ ; ε ) denote the conditional expectation of rewards above the (1 − ε )
-quantile, as in Equation (1).
Then the gradient of J risk ( θ ; ε ) is given by:
[Equation]
The proof and assumptions are provided in Appendix B, and are adapted from the policy gradient derivation for the CVaR objective (Tamar et al., 2014).
This result suggests a simple Monte Carlo estimate of the gradient from a batch of N samples:
[Equation]
input learning rate α ; entropy coefﬁcient λ H ; risk factor ε ; batch size N ; reward function R output
Best ﬁtting expression τ (cid:63) 1: Initialize RNN with parameters θ , deﬁning distribution over expressions p ( ·| θ ) 2: repeat 3:
T ← { τ ( i ) ∼ p ( ·| θ ) } Ni =
(cid:46)
Sample N expressions (Alg. 2 in Appendix A) 4:
T ← { OptimizeConstants ( τ ( i ) , R ) } Ni =
(cid:46)
Optimize constants w.r.t.
reward function 5:
R ← { R ( τ ( i ) ) } Ni = (cid:46) Compute rewards 6:
R ε ← (1 − ε )
-quantile of R (cid:46) Compute reward threshold 7:
T ← { τ ( i ) : R ( τ ( i ) ) ≥ R ε } (cid:46) Select subset of expressions above threshold 8:
R ← { R ( τ ( i ) ) : R ( τ ( i ) ) ≥ R ε } (cid:46) Select corresponding subset of rewards 9: ˆ g 1 ← ReduceMean (( R − R ε ) ∇ θ log p
( T | θ ))
(cid:46) Compute risk-seeking policy gradient 10: ˆ g 2 ← ReduceMean ( − λ H ∇ θ
H ( T | θ ))
(cid:46) Compute entropy gradient 11: θ ← θ + α ( ˆ g 1 + ˆ g 2 ) (cid:46) Apply gradients 12: if max R > R ( τ (cid:63) )
then τ (cid:63) ← τ (argmax R ) (cid:46) Update best expression 13: return τ (cid:63) where ˜ R ε ( θ ) is the empirical (1 − ε ) -quantile of the batch of rewards, and 1 x returns 1 if condition x is true and 0 otherwise.
Essentially, this is the standard REINFORCE Monte Carlo estimate with two differences: (1) theory suggests a speciﬁc baseline, R ε ( θ ) , whereas the baseline for standard policy gradients is non-speciﬁc, chosen by the user; and (2) effectively, only the top ε fraction of samples from each batch are used in the gradient computation.
This process is essentially the opposite of the approach used to optimize CVaR (Tamar et al., 2014; Rajeswaran et al., 2016) for risk-averse reinforcement learning, in which only the bottom ε fraction of samples from each batch are used.
Lastly, in accordance with the maximum entropy reinforcement learning framework (Haarnoja et al., 2018), we provide a bonus to the loss function proportional to the entropy of the sampled expressions.
Pseudocode for DSR is shown in Algorithm 1.
Source code is made available at { anonymous-URL } .
R ESULTS AND D ISCUSSION
Evaluating DSR.
We evaluated DSR on the Nguyen symbolic regression benchmark suite (Uy et al., 2011), a set of 12 commonly used benchmark expressions developed and vetted by the symbolic regression community (White et al., 2013).
Each benchmark is deﬁned by a ground truth expression, a training and test dataset, and a set of allowed operators, described in Table 2 in Appendix D.
The training data is used to compute the reward for each candidate expression, the test data is used to evaluate the best found candidate expression at the end of training, and the ground truth expression is used to determine whether the best found candidate expression was correctly recovered.
We compared DSR against ﬁve strong symbolic regression baselines:
(1) PQT : an implementation of our framework trained using priority queue training (Abolaﬁa et al., 2018) in place of the risk-seeking policy gradient, (2) VPG : a “vanilla” implementation of our framework using the standard policy gradient (with baseline) in place of the risk-seeking policy gradient, (3) GP : a standard GP-based symbolic regression implementation (Fortin et al., 2012), (4) Eureqa : popular commercial softwarebased on Schmidt & Lipson (2009), the gold standard for symbolic regression, and (5) Wolfram : commercial software 2 based on Markov chain Monte Carlo and nonlinear regression (Fortuna, 2015).
Each baseline is further detailed in Appendix C.
Notably, the two RNN-based baselines (PQT and VPG) differ from DSR only via their training objective; all other aspects of our framework (in situ constraints, hierarchical RNN inputs, constant optimization) are also included.
Each experiment consists of 2 million expression evaluations for DSR, PQT, VPG, and GP, by which point training curves have levelled off.
Hyperparameters were tuned on benchmarks Nguyenand Nguyen-10 using a grid search comprising 800 hyperparameter combinations for GP andcombinations for each of DSR, PQT, and VPG (see Appendix D for details).
Commercial software algorithms (Eureqa and Wolfram) do not expose hyperparameters and were run until completion.
[Table]
All experiments were replicated with 100 different random seeds for each benchmark expression.
Wolfram results are only presented for one-dimensional benchmarks because the method is not applicable to higher dimensions.
Additional experiment details are provided in Appendix D. Training curves are provided in Appendix F.
In Table 1, we report the recovery rate for each benchmark.
We use the strictest deﬁnition of recovery: exact symbolic equivalence, as determined using a computer algebra system, e.g. SymPy (Meurer et al., 2017).
In Table 9 in Appendix F, we report recovery on several additional variants of Nguyen benchmarks in which we introduced real-valued constants (to demonstrate the constant optimizer) or altered the functional form to make the problems more challenging.
DSR signiﬁcantly outperforms all ﬁve baselines in its ability to exactly recover benchmark expressions.
In Tables 5–8 in Appendix
E, we compare DSR to literature-reported values from four additional baseline methods (Huynh et al., 2016; Kusner et al., 2017; Jin et al., 2019;
Trujillo et al., 2016) by carefully recapitulating their experimental setup.
DSR greatly outperforms each study’s published results.
Characterizing the risk-seeking policy gradient.
The intuition behind the risk-seeking policy gradient is that it explicitly optimizes for best-case performance, possibly at the expense of average performance.
We demonstrate this visually in Figure 2 by comparing the empirical reward distributions when trained with either the risk-seeking or standard policy gradient for Nguyen-8.
(Analogous plots for all Nguyen benchmarks are provided in Appendix F.)
Interestingly, at the end of training, the mean reward over the full batch (an estimate of J std ( θ ) ) is larger when training with the standard policy gradient, even though the risk-seeking policy gradient produces larger mean over the top ε fraction of the batch (an estimate of J risk ( θ ; ε ) ) and a superior best expression.
This is consistent with the intuition of maximizing best-case performance at the expense of average performance.
In contrast, the best-case performance of the standard policy gradient plateaus early in training (Figure 2E, dashed blue curve), whereas the risk-seeking policy gradient continues to increase until the end of training (Figure 2E, dashed
Ablation studies.
Algorithm 1 includes several additional components relative to a “vanilla” policy gradient search.
We performed a series of ablation studies to quantify the effect of each of these components, along with the effects of the various constraints on the search space.
In Figure 3, we show recovery rate for DSR on the Nguyen benchmarks for each ablation.
While no single ablation leads to catastrophic failure, combinations of ablations can cause large degradation in performance.
Noisy data and amount of data.
We evaluated the robustness of DSR to noisy data by adding independent Gaussian noise to the dependent variable, with mean zero and standard deviation proportional to the root-mean-square of the dependent variable in the training data.
In Figure 4, we varied the proportionality constant from 0 (noiseless) to 10 − 1 and evaluated each algorithm (except Wolfram, which catastrophically fails for even the smallest noise level) across all Nguyen benchmarks.
Because expressions can overﬁt to the noise in these experiments, we deﬁned recovery as exact symbolic equivalence on any expression along the reward-complexity Pareto front at the end of training (see Appendix D for details).
Increasing the dataset size may help prevent overﬁtting by
[Figure] [Figure]
smoothing the reward function.
Thus, we repeated the noise experiments with 10-fold larger training datasets (Figure 4, dashed lines).
For each dataset size, DSR consistently outperforms all baselines.
Performance under reward noise.
The risk-seeking policy gradient is derived from a conditional expectation, and is thus well-justiﬁed for tasks with stochastic rewards.
In contrast, PQT assumes deterministic rewards.
Since symbolic regression is a deterministic task, we emulated a stochastic reward function by adding independent Gaussian noise directly to the reward function: R (cid:48) ( τ ) =
R ( τ ) + N (0 , σ ) .
In Figure 5, we compare DSR and PQT performance under increasing reward noise on benchmark Nguyen-4, a task that is an easier exploitation problem but difﬁcult exploration problem.
As reward noise increases, recovery for PQT heavily relies on exploration and luck, whereas
DSR recovery remains stable even for high reward noise.
C ONCLUSION AND F UTURE W ORK
We introduce a reinforcement learning approach to symbolic regression that outperforms state-ofthe-art baselines in its ability to recover exact expressions on benchmark tasks.
Our framework is easily extensible to other domains, which we save for future work; for example, searching the space of expressions to be used as control policies in reinforcement learning environments, or searching the space of organic molecular structures for high binding afﬁnity to a reference compound.
Our riskseeking policy gradient formulation can also be applied to more traditional reinforcement learning domains; for example, optimizing for a high score (instead of average score) in Atari video games.