Original: D EEP SYMBOLIC REGRESSION
Revised: D EEP SYMBOLIC REGRESSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Discovering the underlying mathematical expressions describing a dataset is a core challenge for artiﬁcial intelligence.
Revised: Discovering the underlying mathematical expressions describing a dataset is a core challenge for artiﬁcial intelligence.

Original: This is the problem of symbolic regression .
Revised: This is the problem of symbolic regression .

Original: Despite recent advances in training neural networks to solve complex tasks, deep learning approaches to symbolic regression are lacking.
Revised: Despite recent advances in training neural networks to solve complex tasks, deep learning approaches to symbolic regression are lacking.

Original: We propose a framework that combines deep learning with symbolic regression via a simple idea: use a large model to search the space of small models.
Revised: We propose a framework that combines deep learning with symbolic regression via a simple idea: use a large model to search the space of small models.

Original: More speciﬁcally, we use a recurrent neural network to emit a distribution over tractable mathematical expressions, and employ reinforcement learning to train the network to generate better-ﬁtting expressions.
Revised: More speciﬁcally, we use a recurrent neural network to emit a distribution over tractable mathematical expressions, and employ reinforcement learning to train the network to generate better-ﬁtting expressions.

Original: Our algorithm signiﬁcantly outperforms standard genetic programming-based symbolic regression in its ability to exactly recover symbolic expressions on a series of benchmark problems, both with and without added noise.
Revised: Our algorithm signiﬁcantly outperforms standard genetic programming-based symbolic regression in its ability to exactly recover symbolic expressions on a series of benchmark problems, both with and without added noise.

Original: More broadly, our contributions include a framework that can be applied to optimize hierarchical, variable-length objects under a black-box performance metric, with the ability to incorporate a priori constraints in situ.
Revised: More broadly, our contributions include a framework that can be applied to optimize hierarchical, variable-length objects under a black-box performance metric, with the ability to incorporate a priori constraints in situ.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Understanding the mathematical relationships among variables in a physical system is an integral component of the scientiﬁc process.
Revised: Understanding the mathematical relationships among variables in a physical system is an integral component of the scientiﬁc process.

Original: Symbolic regression aims to identify these relationships by searching over the space of tractable mathematical expressions to best ﬁt a dataset.
Revised: Symbolic regression aims to identify these relationships by searching over the space of tractable mathematical expressions to best ﬁt a dataset.

Original: Speciﬁcally, given a dataset of ( X, y ) pairs, where
Revised: Speciﬁcally, given a dataset of ( X, y ) pairs, where

Original: X ∈ R n and y ∈ R , symbolic regression aims to identify a function f ( X ) :
Revised: X ∈ R n and y ∈ R , symbolic regression aims to identify a function f ( X ) :

Original: R n → R that minimizes a distance metric D ( y, f ( X )) between real and predicted values.
Revised: R n → R that minimizes a distance metric D ( y, f ( X )) between real and predicted values.

Original: That is, symbolic regression seeks to ﬁnd the optimal f (cid:63) =
Revised: That is, symbolic regression seeks to ﬁnd the optimal f (cid:63) =

Original: arg min f D ( y, f ( X )) , where the functional form of f is a tractable expression.
Revised: arg min f D ( y, f ( X )) , where the functional form of f is a tractable expression.

Original: The resulting expression f (cid:63) may be readily interpretable and/or provide useful scientiﬁc insights simply by inspection.
Revised: The resulting expression f (cid:63) may be readily interpretable and/or provide useful scientiﬁc insights simply by inspection.

Original: In contrast, conventional regression imposes a single model structure that is ﬁxed during training, often chosen to be expressive (e.g. a neural network) at the expense of being easily interpretable.
Revised: In contrast, conventional regression imposes a single model structure that is ﬁxed during training, often chosen to be expressive (e.g. a neural network) at the expense of being easily interpretable.

Original: However, the space of mathematical expressions is discrete (in model structure) and continuous (in model parameters), growing exponentially with the length of the expression, rendering symbolic regression an extremely challenging machine learning problem.
Revised: However, the space of mathematical expressions is discrete (in model structure) and continuous (in model parameters), growing exponentially with the length of the expression, rendering symbolic regression an extremely challenging machine learning problem.

Original: Given the large and combinatorial search space, traditional approaches to symbolic regression typically utilize evolutionary algorithms, especially genetic programming (GP) (Koza, 1992; B¨ack et al., 2018).
Revised: Given the large and combinatorial search space, traditional approaches to symbolic regression typically utilize evolutionary algorithms, especially genetic programming (GP) (Koza, 1992; B¨ack et al., 2018).

Original: In
Revised: In

Original: GP-based symbolic regression, a population of mathematical expressions is “evolved” using evolutionary operations like selection, crossover, and mutation to improve a ﬁtness function.
Revised: GP-based symbolic regression, a population of mathematical expressions is “evolved” using evolutionary operations like selection, crossover, and mutation to improve a ﬁtness function.

Original: While GP can be effective, it is also known to scale poorly to larger problems and to exhibit high sensitivity to hyperparameters.
Revised: While GP can be effective, it is also known to scale poorly to larger problems and to exhibit high sensitivity to hyperparameters.

Original: Deep learning has permeated almost all areas of artiﬁcial intelligence, from computer vision (Krizhevsky et al., 2012) to optimal control (Mnih et al., 2015).
Revised: Deep learning has permeated almost all areas of artiﬁcial intelligence, from computer vision (Krizhevsky et al., 2012) to optimal control (Mnih et al., 2015).

Original: However, deep learning may seem incongruous with or even antithetical toward symbolic regression, given that neural networks are typically highly complex, difﬁcult to interpret, and rely on gradient information.
Revised: However, deep learning may seem incongruous with or even antithetical toward symbolic regression, given that neural networks are typically highly complex, difﬁcult to interpret, and rely on gradient information.

Original: We propose a framework that resolves this incongruity by tying deep learning and symbolic regression together with a simple idea: use a large model (i.e. neural network) to search the space of small models (i.e. symbolic expressions).
Revised: We propose a framework that resolves this incongruity by tying deep learning and symbolic regression together with a simple idea: use a large model (i.e. neural network) to search the space of small models (i.e. symbolic expressions).

Original: This framework leverages the representational capacity of neural networks while entirely bypassing the need to interpret a network.
Revised: This framework leverages the representational capacity of neural networks while entirely bypassing the need to interpret a network.

Original: We present deep symbolic regression (DSR), a gradient-based approach for symbolic regression based on reinforcement learning.
Revised: We present deep symbolic regression (DSR), a gradient-based approach for symbolic regression based on reinforcement learning.

Original: In DSR, a recurrent neural network (RNN) emits a distribution over mathematical expressions.
Revised: In DSR, a recurrent neural network (RNN) emits a distribution over mathematical expressions.

Original: Expressions are sampled from the distribution, instantiated, and evaluated based on their ﬁtness to the dataset.
Revised: Expressions are sampled from the distribution, instantiated, and evaluated based on their ﬁtness to the dataset.

Original: This ﬁtness is used as the reward signal to train the RNN parameters using a policy gradient algorithm.
Revised: This ﬁtness is used as the reward signal to train the RNN parameters using a policy gradient algorithm.

Original: As training proceeds, the RNN adjusts the likelihood of an expression relative to its reward, assigning higher probabilities to better ﬁtting expressions.
Revised: As training proceeds, the RNN adjusts the likelihood of an expression relative to its reward, assigning higher probabilities to better ﬁtting expressions.

Original: We demonstrate that DSR outperforms a standard GP implementation in its ability to recover exact symbolic expressions from data, both with and without added noise.
Revised: We demonstrate that DSR outperforms a standard GP implementation in its ability to recover exact symbolic expressions from data, both with and without added noise.

Original: We summarize our contributions as follows: 1) a novel method for solving symbolic regression that outperforms standard GP, 2) an autoregressive generative modeling framework for optimizing hierarchical, variable-length objects, 3) a framework that accommodates in situ constraints, and 4) a novel risk-seeking strategy that optimizes for best-case performance.
Revised: We summarize our contributions as follows: 1) a novel method for solving symbolic regression that outperforms standard GP, 2) an autoregressive generative modeling framework for optimizing hierarchical, variable-length objects, 3) a framework that accommodates in situ constraints, and 4) a novel risk-seeking strategy that optimizes for best-case performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Symbolic regression.
Revised: Symbolic regression.

Original: Symbolic regression has a long history of evolutionary strategies, especially GP (Koza, 1992; B ¨ ack et al., 2018;
Revised: Symbolic regression has a long history of evolutionary strategies, especially GP (Koza, 1992; B ¨ ack et al., 2018;

Original: Uy et al., 2011).
Revised: Uy et al., 2011).

Original: Among non-evolutionary approaches, the recent AI Feynman algorithm (Udrescu & Tegmark, 2019) is a multi-staged approach to symbolic regression leveraging the observation that physical equations often exhibit simplifying properties like multiplicative separability and translational symmetry.
Revised: Among non-evolutionary approaches, the recent AI Feynman algorithm (Udrescu & Tegmark, 2019) is a multi-staged approach to symbolic regression leveraging the observation that physical equations often exhibit simplifying properties like multiplicative separability and translational symmetry.

Original: The algorithm identiﬁes and exploits such properties to recursively deﬁne simpliﬁed sub-problems that can eventually be solved using simple techniques like a polynomial ﬁt or small brute force search.
Revised: The algorithm identiﬁes and exploits such properties to recursively deﬁne simpliﬁed sub-problems that can eventually be solved using simple techniques like a polynomial ﬁt or small brute force search.

Original: Brunton et al.
Revised: Brunton et al.

Original: (2016) develop a sparse regression approach to recover nonlinear dynamics equations from data; however, their search space is limited to linear combinations of a library of basis functions.
Revised: (2016) develop a sparse regression approach to recover nonlinear dynamics equations from data; however, their search space is limited to linear combinations of a library of basis functions.

Original: AutoML.
Revised: AutoML.

Original: Our framework has many parallels to a body of works within automated machine learning (AutoML) that use an autoregressive RNN to deﬁne a distribution over discrete objects and use reinforcement learning to optimize this distribution under a black-box performance metric (Zoph & Le, 2017; Ramachandran et al., 2017; Bello et al., 2017).
Revised: Our framework has many parallels to a body of works within automated machine learning (AutoML) that use an autoregressive RNN to deﬁne a distribution over discrete objects and use reinforcement learning to optimize this distribution under a black-box performance metric (Zoph & Le, 2017; Ramachandran et al., 2017; Bello et al., 2017).

Original: The key methodological difference to our framework is that these works optimize objects that are both sequential and ﬁxed length .
Revised: The key methodological difference to our framework is that these works optimize objects that are both sequential and ﬁxed length .

Original: For example, in neural architecture search (Zoph & Le, 2017), an RNN searches the space of neural network architectures, which are encoded by a sequence of discrete “tokens” specifying architectural properties (e.g. number of neurons) of each layer.
Revised: For example, in neural architecture search (Zoph & Le, 2017), an RNN searches the space of neural network architectures, which are encoded by a sequence of discrete “tokens” specifying architectural properties (e.g. number of neurons) of each layer.

Original: The length of the sequence is ﬁxed or scheduled during training.
Revised: The length of the sequence is ﬁxed or scheduled during training.

Original: In contrast, a major contribution of our framework is deﬁning a search space that is both inherently hierarchical and variable length.
Revised: In contrast, a major contribution of our framework is deﬁning a search space that is both inherently hierarchical and variable length.

Original: The most similar AutoML work searches for neural network activation functions (Ramachandran et al., 2017).
Revised: The most similar AutoML work searches for neural network activation functions (Ramachandran et al., 2017).

Original: While the space of activation functions is hierarchical in nature, the authors (rightfully) constrain this space substantially by positing a functional unit that is repeated sequentially, thus restricting their search space back to a ﬁxed-length sequence.
Revised: While the space of activation functions is hierarchical in nature, the authors (rightfully) constrain this space substantially by positing a functional unit that is repeated sequentially, thus restricting their search space back to a ﬁxed-length sequence.

Original: This constraint is well-justiﬁed for learning activation functions, which tend to exhibit similar hierarchical structures.
Revised: This constraint is well-justiﬁed for learning activation functions, which tend to exhibit similar hierarchical structures.

Original: However, a repeating-unit constraint is not practical for symbolic regression because the ground truth expression may have arbitrary structure.
Revised: However, a repeating-unit constraint is not practical for symbolic regression because the ground truth expression may have arbitrary structure.

Original: Autoregressive models.
Revised: Autoregressive models.

Original: The RNN-based distribution over expressions used in DSR is autoregressive, meaning each token is conditioned on the previously sampled tokens.
Revised: The RNN-based distribution over expressions used in DSR is autoregressive, meaning each token is conditioned on the previously sampled tokens.

Original: Autoregressive models have proven to be useful for audio and image data (Oord et al., 2016a;b) in addition to the AutoML works discussed above; we further demonstrate their efﬁcacy for hierarchical expressions.
Revised: Autoregressive models have proven to be useful for audio and image data (Oord et al., 2016a;b) in addition to the AutoML works discussed above; we further demonstrate their efﬁcacy for hierarchical expressions.

Original: GraphRNN deﬁnes a distribution over graphs that generates an adjacency matrix one column at a time in autoregressive fashion (You et al., 2018).
Revised: GraphRNN deﬁnes a distribution over graphs that generates an adjacency matrix one column at a time in autoregressive fashion (You et al., 2018).

Original: In principle, we could have constrained GraphRNN to deﬁne the distribution over expressions, since trees are a special case of graphs.
Revised: In principle, we could have constrained GraphRNN to deﬁne the distribution over expressions, since trees are a special case of graphs.

Original: However, GraphRNN constructs graphs breadth-ﬁrst, whereas expressions are more naturally represented using depth-ﬁrst traversals (Li et al., 2005).
Revised: However, GraphRNN constructs graphs breadth-ﬁrst, whereas expressions are more naturally represented using depth-ﬁrst traversals (Li et al., 2005).

Original: Further, DSR exploits the hierarchical nature of trees by providing the parent and sibling as inputs to the RNN, and leverages the additional structure of expression trees that a node’s value determines its number of children (e.g. cosine is a unary node).
Revised: Further, DSR exploits the hierarchical nature of trees by providing the parent and sibling as inputs to the RNN, and leverages the additional structure of expression trees that a node’s value determines its number of children (e.g. cosine is a unary node).

Original:  M ETHODS
Revised:  M ETHODS

Original: Our overall approach involves representing mathematical expressions by the pre-order traversals of their corresponding symbolic expression trees, developing an autoregressive model to generate expression trees under a pre-speciﬁed set of constraints, and using reinforcement learning to train the model to generate better-ﬁtting expressions.
Revised: Our overall approach involves representing mathematical expressions by the pre-order traversals of their corresponding symbolic expression trees, developing an autoregressive model to generate expression trees under a pre-speciﬁed set of constraints, and using reinforcement learning to train the model to generate better-ﬁtting expressions.

Original: 3. G ENERATING EXPRESSIONS WITH A RECURRENT NEURAL NETWORK
Revised: 3. G ENERATING EXPRESSIONS WITH A RECURRENT NEURAL NETWORK

Original: We leverage the fact that algebraic expressions can be represented using symbolic expression trees, a type of binary tree in which nodes map to mathematical operators, input variables, or constants.
Revised: We leverage the fact that algebraic expressions can be represented using symbolic expression trees, a type of binary tree in which nodes map to mathematical operators, input variables, or constants.

Original: Operators are internal nodes and may be unary (e.g. sine) or binary (e.g. multiply).
Revised: Operators are internal nodes and may be unary (e.g. sine) or binary (e.g. multiply).

Original: Input variables and constants are terminal nodes.
Revised: Input variables and constants are terminal nodes.

Original: We encode an expression τ by the pre-order traversal (i.e. depth-ﬁrst, then left-to-right) of its corresponding expression tree.
Revised: We encode an expression τ by the pre-order traversal (i.e. depth-ﬁrst, then left-to-right) of its corresponding expression tree.

Original: 1 We denote the i th node in the traversal as τ i and the length of the traversal as | τ | = T .
Revised: 1 We denote the i th node in the traversal as τ i and the length of the traversal as | τ | = T .

Original: Each node has a value within a given library L of possible node values or “tokens,” e.g.
Revised: Each node has a value within a given library L of possible node values or “tokens,” e.g.

Original: { + , − , × , ÷ , sin , cos , x } .
Revised: { + , − , × , ÷ , sin , cos , x } .

Original: Expressions are generated one node at a time along the pre-order traversal (from τ 1 to τ T ).
Revised: Expressions are generated one node at a time along the pre-order traversal (from τ 1 to τ T ).

Original: For each node, a categorical distribution with parameters ψ deﬁnes the probabilities of selecting each node value from L .
Revised: For each node, a categorical distribution with parameters ψ deﬁnes the probabilities of selecting each node value from L .

Original: To capture the “context” of the expression as it is being generated, we condition this probability upon the selections of all previous nodes in that traversal.
Revised: To capture the “context” of the expression as it is being generated, we condition this probability upon the selections of all previous nodes in that traversal.

Original: This conditional dependence can be achieved very generally using an RNN with parameters θ that outputs a probability vector ψ in autoregressive manner.
Revised: This conditional dependence can be achieved very generally using an RNN with parameters θ that outputs a probability vector ψ in autoregressive manner.

Original: Speciﬁcally, the i th output vector ψ ( i ) of the RNN deﬁnes the probability distribution for selecting the i th node value τ i , conditioned on the previously selected node values τ 1:(
Revised: Speciﬁcally, the i th output vector ψ ( i ) of the RNN deﬁnes the probability distribution for selecting the i th node value τ i , conditioned on the previously selected node values τ 1:(

Original: i − 1) :
Revised: i − 1) :

Original: [Equation]
Revised: [Equation]

Original: where L ( τ i ) is the index in L corresponding to node value τ i .
Revised: where L ( τ i ) is the index in L corresponding to node value τ i .

Original: The likelihood of the sampled expression is computed using the chain rule of conditional probability:
Revised: The likelihood of the sampled expression is computed using the chain rule of conditional probability:

Original: [Equation]
Revised: [Equation]

Original: The sampling process is illustrated in Figure 1 and described in Algorithm 1.
Revised: The sampling process is illustrated in Figure 1 and described in Algorithm 1.

Original: Additional algorithmic details of the sampling process are described in Subroutines 1 and 2 in Appendix A.
Revised: Additional algorithmic details of the sampling process are described in Subroutines 1 and 2 in Appendix A.

Original: Starting at the root node, a node value is sampled according to ψ (1) .
Revised: Starting at the root node, a node value is sampled according to ψ (1) .

Original: Subsequent node values are sampled autoregressively in a depth-ﬁrst, left-to-right manner until the tree is complete (i.e. all tree branches reach terminal nodes).
Revised: Subsequent node values are sampled autoregressively in a depth-ﬁrst, left-to-right manner until the tree is complete (i.e. all tree branches reach terminal nodes).

Original: The resulting sequence of node values is the tree’s pre-order traversal, which can be used to reconstruct the tree 2 and its
Revised: The resulting sequence of node values is the tree’s pre-order traversal, which can be used to reconstruct the tree 2 and its

Original: Library + 𝐬𝐢𝐧 𝐜𝐨𝐧𝐬𝐭 − ×
Revised: Library + 𝐬𝐢𝐧 𝐜𝐨𝐧𝐬𝐭 − ×

Original: ÷ 𝐜𝐨𝐬 𝐥𝐨𝐠 𝐞𝐱𝐩 𝒚 𝒙
Revised: ÷ 𝐜𝐨𝐬 𝐥𝐨𝐠 𝐞𝐱𝐩 𝒚 𝒙

Original: Sampled token:
Revised: Sampled token:

Original: Pre-order traversal
Revised: Pre-order traversal

Original: Parent:
Revised: Parent:

Original: Sibling:
Revised: Sibling:

Original: RNN:
Revised: RNN:

Original: Categorical distribution: ÷ 𝐬𝐢𝐧 × 𝐜𝐨𝐧𝐬𝐭 𝒙 𝐥𝐨𝐠 𝒚
Revised: Categorical distribution: ÷ 𝐬𝐢𝐧 × 𝐜𝐨𝐧𝐬𝐭 𝒙 𝐥𝐨𝐠 𝒚

Original: ÷ 𝐬𝐢𝐧 ×
Revised: ÷ 𝐬𝐢𝐧 ×

Original: × ÷ 𝐥𝐨𝐠 𝐜𝐨𝐧𝐬𝐭
Revised: × ÷ 𝐥𝐨𝐠 𝐜𝐨𝐧𝐬𝐭

Original: 𝐬𝐢𝐧 ÷ 𝐬𝐢𝐧 × 𝐜𝐨𝐧𝐬𝐭 𝒙 𝐥𝐨𝐠 𝒚      1 Expression tree
Revised: 𝐬𝐢𝐧 ÷ 𝐬𝐢𝐧 × 𝐜𝐨𝐧𝐬𝐭 𝒙 𝐥𝐨𝐠 𝒚      1 Expression tree

Original: A
Revised: A

Original: B
Revised: B

Original: C     

Revised: C

Original: The library of tokens.
Revised: The library of tokens.

Original: C .
Revised: C .

Original: The expression tree sampled in A . corresponding expression.
Revised: The expression tree sampled in A . corresponding expression.

Original: Note that different samples of the distribution have different tree structures of different size.
Revised: Note that different samples of the distribution have different tree structures of different size.

Original: Thus, the search space is inherently both hierarchical and variable length.
Revised: Thus, the search space is inherently both hierarchical and variable length.

Original: Providing hierarchical inputs to the RNN.
Revised: Providing hierarchical inputs to the RNN.

Original: Naively, the input to the RNN when sampling τ i would be a representation (i.e. embedding or one-hot encoding) of the previously sampled token, τ i − 1 .
Revised: Naively, the input to the RNN when sampling τ i would be a representation (i.e. embedding or one-hot encoding) of the previously sampled token, τ i − 1 .

Original: Indeed, this is typical in related autoregressive models, e.g. when generating sentences (Vaswani et al., 2017) or for neural architecture search (Zoph & Le, 2017).
Revised: Indeed, this is typical in related autoregressive models, e.g. when generating sentences (Vaswani et al., 2017) or for neural architecture search (Zoph & Le, 2017).

Original: However, the search space for symbolic regression is inherently hierarchical, and the previously sampled token may actually be very distant from the next token to be sampled in the expression tree.
Revised: However, the search space for symbolic regression is inherently hierarchical, and the previously sampled token may actually be very distant from the next token to be sampled in the expression tree.

Original: For example, the ﬁfth and sixth tokens sampled in Figure 1 are adjacent nodes in the traversal but are four edges apart in the expression tree.
Revised: For example, the ﬁfth and sixth tokens sampled in Figure 1 are adjacent nodes in the traversal but are four edges apart in the expression tree.

Original: To better capture hierarchical information, we provide as inputs to the RNN a representation of the parent and sibling node of the token being sampled.
Revised: To better capture hierarchical information, we provide as inputs to the RNN a representation of the parent and sibling node of the token being sampled.

Original: We introduce an empty token for cases in which a node does not have a parent or sibling.
Revised: We introduce an empty token for cases in which a node does not have a parent or sibling.

Original: Pseudocode for identifying the parent and sibling nodes given a partial traversal is provided in Subroutine 2 in Appendix A.
Revised: Pseudocode for identifying the parent and sibling nodes given a partial traversal is provided in Subroutine 2 in Appendix A.

Original: Constraining the search space.
Revised: Constraining the search space.

Original: Under our framework, it is straightforward to apply a priori constraints to reduce the search space.
Revised: Under our framework, it is straightforward to apply a priori constraints to reduce the search space.

Original: To demonstrate, we impose several simple, domain-agnostic constraints: (1)
Revised: To demonstrate, we impose several simple, domain-agnostic constraints: (1)

Original: Expressions are limited to a pre-speciﬁed minimum and maximum length.
Revised: Expressions are limited to a pre-speciﬁed minimum and maximum length.

Original: We selected minimum length of 2 to prevent trivial expressions and a maximum length of 30 to ensure expressions are tractable.
Revised: We selected minimum length of 2 to prevent trivial expressions and a maximum length of 30 to ensure expressions are tractable.

Original: The children of an operator should not all be constants, as the result would simply be a different constant.
Revised: The children of an operator should not all be constants, as the result would simply be a different constant.

Original: The child of a unary operator should not be the inverse of that operator, e.g. log(exp( x )) is not allowed.
Revised: The child of a unary operator should not be the inverse of that operator, e.g. log(exp( x )) is not allowed.

Original: Direct descendants of trigonometric operators should not be trigonometric operators, e.g. sin( x + cos( x )) is not allowed because cosine is a descendant of sine.
Revised: Direct descendants of trigonometric operators should not be trigonometric operators, e.g. sin( x + cos( x )) is not allowed because cosine is a descendant of sine.

Original: While still semantically meaningful, such composed trigonometric operators do not appear in virtually any scientiﬁc domain.
Revised: While still semantically meaningful, such composed trigonometric operators do not appear in virtually any scientiﬁc domain.

Original: We apply these constraints in situ (concurrently with autoregressive sampling) by zeroing out the probabilities of selecting tokens that would violate a constraint.
Revised: We apply these constraints in situ (concurrently with autoregressive sampling) by zeroing out the probabilities of selecting tokens that would violate a constraint.

Original: Pseudocode for this process is provided in Subroutine
Revised: Pseudocode for this process is provided in Subroutine

Original: 1 function SampleExpression ( θ, L ) input
Revised: 1 function SampleExpression ( θ, L ) input

Original: τ =
Revised: τ =

Original: ] // Empty list x = empty //
Revised: ] // Empty list x = empty //

Original: Initial RNN input is empty parent and sibling h 

Revised: Initial RNN input is empty parent and sibling h

Original: = (cid:126)// Initialize RNN cell state to zero for i
Revised: = (cid:126)// Initialize RNN cell state to zero for i

Original: = 1 , . .
Revised: = 1 , . .

Original: , T do ( ψ ( i ) , h i ) =
Revised: , T do ( ψ ( i ) , h i ) =

Original: RNN ( x, h i − 1 ; θ ) ψ ( i ) ← ApplyConstraints ( ψ ( i ) , L , τ ) //
Revised: RNN ( x, h i − 1 ; θ ) ψ ( i ) ← ApplyConstraints ( ψ ( i ) , L , τ ) //

Original: Adjust probabilities τ i =
Revised: Adjust probabilities τ i =

Original: Categorical ( ψ ( i ) ) //
Revised: Categorical ( ψ ( i ) ) //

Original: Sample next token τ ← τ (cid:107) τ i //
Revised: Sample next token τ ← τ (cid:107) τ i //

Original: Append token to traversal if ExpressionComplete ( τ ) then return τ x ← ParentSibling ( τ ) //
Revised: Append token to traversal if ExpressionComplete ( τ ) then return τ x ← ParentSibling ( τ ) //

Original: Compute next parent and sibling end return τ
Revised: Compute next parent and sibling end return τ

Original: 1 function DSR ( α, N, L , X, y ) input
Revised: 1 function DSR ( α, N, L , X, y ) input

Original: Initialize RNN with parameters θ , deﬁning distribution over expressions p ( ·| θ ) τ (cid:63) =
Revised: Initialize RNN with parameters θ , deﬁning distribution over expressions p ( ·| θ ) τ (cid:63) =

Original: null b = repeat
Revised: null b = repeat

Original: T = { τ ( i ) ∼ p ( ·| θ ) } i =1: N // Sample expressions (Algorithm 1)
Revised: T = { τ ( i ) ∼ p ( ·| θ ) } i =1: N // Sample expressions (Algorithm 1)

Original: T ← { OptimizeConstants ( τ ( i ) , X, y ) } i =1: N // Optimize constants
Revised: T ← { OptimizeConstants ( τ ( i ) , X, y ) } i =1: N // Optimize constants

Original: R =
Revised: R =

Original: { R ( τ ( i ) )
Revised: { R ( τ ( i ) )

Original: − λ C C ( τ ( i ) )
Revised: − λ C C ( τ ( i ) )

Original: } i =1: N // Compute rewards ˆ g = 1 N (cid:80)
Revised: } i =1: N // Compute rewards ˆ g = 1 N (cid:80)

Original: Ni =1 R ( τ ( i ) ) ∇ θ log p
Revised: Ni =1 R ( τ ( i ) ) ∇ θ log p

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: [Equation]
Revised: [Equation]

Original: Compute policy gradient
Revised: Compute policy gradient

Original: // Apply gradients
Revised: // Apply gradients

Original: // Update best expression
Revised: // Update best expression

Original: 3. T RAINING THE RNN USING POLICY GRADIENTS
Revised: 3. T RAINING THE RNN USING POLICY GRADIENTS

Original: Optimizing the parameters of the sampled expressions.
Revised: Optimizing the parameters of the sampled expressions.

Original: Once a pre-order traversal is sampled, we instantiate the corresponding symbolic expression.
Revised: Once a pre-order traversal is sampled, we instantiate the corresponding symbolic expression.

Original: The expression may have several constant tokens, which can be viewed as model parameters.
Revised: The expression may have several constant tokens, which can be viewed as model parameters.

Original: We train these model parameters by minimizing the mean-squared error with respect to an input dataset using a nonlinear optimization algorithm, e.g. BFGS (Fletcher, 2013).
Revised: We train these model parameters by minimizing the mean-squared error with respect to an input dataset using a nonlinear optimization algorithm, e.g. BFGS (Fletcher, 2013).

Original: We perform this inner optimization loop for each sampled expression before training the RNN.
Revised: We perform this inner optimization loop for each sampled expression before training the RNN.

Original: Training the RNN using policy gradients.
Revised: Training the RNN using policy gradients.

Original: Given a distribution over mathematical expressions p ( τ | θ ) and a measure of performance of an expression R ( τ ) , we consider the objective to maximize J ( θ ) , deﬁned as the expectation of R under expressions sampled from the distribution:
Revised: Given a distribution over mathematical expressions p ( τ | θ ) and a measure of performance of an expression R ( τ ) , we consider the objective to maximize J ( θ ) , deﬁned as the expectation of R under expressions sampled from the distribution:

Original: [Equation]
Revised: [Equation]

Original: We use REINFORCE (Williams, 1992) to maximize this expectation via gradient ascent:
Revised: We use REINFORCE (Williams, 1992) to maximize this expectation via gradient ascent:

Original: [Equation]
Revised: [Equation]

Original: This result allows us to estimate the expectation using samples from the distribution.
Revised: This result allows us to estimate the expectation using samples from the distribution.

Original: Speciﬁcally, we can obtain an unbiased estimate of ∇ θ J ( θ ) by computing the sample mean over a batch of N sampled expressions
Revised: Speciﬁcally, we can obtain an unbiased estimate of ∇ θ J ( θ ) by computing the sample mean over a batch of N sampled expressions

Original: T = { τ ( i ) } i =1: N :
Revised: T = { τ ( i ) } i =1: N :

Original: [Equation]
Revised: [Equation]

Original: Reward function.
Revised: Reward function.

Original: A standard ﬁtness measure in GP-based symbolic regression is normalized root-meansquare error (NRMSE), the root-mean-square error normalized by the standard deviation of the target values, σ y .
Revised: A standard ﬁtness measure in GP-based symbolic regression is normalized root-meansquare error (NRMSE), the root-mean-square error normalized by the standard deviation of the target values, σ y .

Original: That is, given a dataset of n number of ( X, y ) pairs, NRMSE = 1 σ y (cid:113) 1 n (cid:80) ni
Revised: That is, given a dataset of n number of ( X, y ) pairs, NRMSE = 1 σ y (cid:113) 1 n (cid:80) ni

Original: =

Revised: =

Original: ( y i − ˆ y i ) 2 , where ˆ y = f
Revised: ( y i − ˆ y i ) 2 , where ˆ y = f

Original: ( X ) are the predicted values computed using the candidate expression f .
Revised: ( X ) are the predicted values computed using the candidate expression f .

Original: Normalization by σ y makes the metric commensurate across different datasets with potentially different ranges.
Revised: Normalization by σ y makes the metric commensurate across different datasets with potentially different ranges.

Original: However, metrics based on mean-square error exhibit extraordinarily large values for some expressions, e.g. an expression that incorrectly divides by an input variable with values near zero.
Revised: However, metrics based on mean-square error exhibit extraordinarily large values for some expressions, e.g. an expression that incorrectly divides by an input variable with values near zero.

Original: For a gradient-based approach like DSR, this results in the gradient being dominated by the worst expressions, which can lead to instability.
Revised: For a gradient-based approach like DSR, this results in the gradient being dominated by the worst expressions, which can lead to instability.

Original: We found that a bounded reward function is more stable; thus, we applied a squashing function, yielding the reward function R ( τ ) = 1 / (1 + NRMSE ) .
Revised: We found that a bounded reward function is more stable; thus, we applied a squashing function, yielding the reward function R ( τ ) = 1 / (1 + NRMSE ) .

Original: We introduce the “vanilla” version of DSR in Algorithm 2.
Revised: We introduce the “vanilla” version of DSR in Algorithm 2.

Original: Below we describe several simple extensions.
Revised: Below we describe several simple extensions.

Original: Reward baseline.
Revised: Reward baseline.

Original: The above approximation to ∇ θ J ( θ ) is an unbiased gradient estimate, but in practice has high variance.
Revised: The above approximation to ∇ θ J ( θ ) is an unbiased gradient estimate, but in practice has high variance.

Original: To reduce variance, we include a baseline function b :
Revised: To reduce variance, we include a baseline function b :

Original: [Equation]
Revised: [Equation]

Original: As long as the baseline is not a function of the current batch of expressions, the gradient estimate is still unbiased.
Revised: As long as the baseline is not a function of the current batch of expressions, the gradient estimate is still unbiased.

Original: We deﬁne the baseline function as an exponentially-weighted moving average of batches of rewards.
Revised: We deﬁne the baseline function as an exponentially-weighted moving average of batches of rewards.

Original: Intuitively, the gradient step increases the likelihood of expressions above the baseline and decreases the likelihood of expressions below the baseline.
Revised: Intuitively, the gradient step increases the likelihood of expressions above the baseline and decreases the likelihood of expressions below the baseline.

Original: Complexity penalty.
Revised: Complexity penalty.

Original: We include an optional complexity penalty that is added to the reward function.
Revised: We include an optional complexity penalty that is added to the reward function.

Original: For simplicity, we consider the complexity metric | τ | , i.e. the number of nodes in the expression tree.
Revised: For simplicity, we consider the complexity metric | τ | , i.e. the number of nodes in the expression tree.

Original: More complicated metrics have been proposed that capture hierarchical features of the tree and/or deduced properties of the resulting et al., 2008).
Revised: More complicated metrics have been proposed that capture hierarchical features of the tree and/or deduced properties of the resulting et al., 2008).

Original: 1 function DSR ( α, β, λ C , λ H , (cid:15), N, L , X, y )
Revised: 1 function DSR ( α, β, λ C , λ H , (cid:15), N, L , X, y )

Original: Initialize RNN with parameters θ , deﬁning distribution over expressions p ( ·| θ ) τ (cid:63) =
Revised: Initialize RNN with parameters θ , deﬁning distribution over expressions p ( ·| θ ) τ (cid:63) =

Original: null b = repeat
Revised: null b = repeat

Original: T = { τ ( i ) ∼ p ( ·| θ ) } i =1: N // Sample expressions (Algorithm 1)
Revised: T = { τ ( i ) ∼ p ( ·| θ ) } i =1: N // Sample expressions (Algorithm 1)

Original: T ← { OptimizeConstants ( τ ( i ) , X, y ) } i =1: N // Optimize constants
Revised: T ← { OptimizeConstants ( τ ( i ) , X, y ) } i =1: N // Optimize constants

Original: R =
Revised: R =

Original: { R ( τ ( i ) )
Revised: { R ( τ ( i ) )

Original: − λ C C ( τ ( i ) )
Revised: − λ C C ( τ ( i ) )

Original: } i =1: N // Compute rewards
Revised: } i =1: N // Compute rewards

Original: R (cid:15) =
Revised: R (cid:15) =

Original: (1 − (cid:15) ) percentile of R //
Revised: (1 − (cid:15) ) percentile of R //

Original: Compute threshold
Revised: Compute threshold

Original: T ← { τ ( i ) : R ( τ ( i ) ) ≥ R (cid:15) } //
Revised: T ← { τ ( i ) : R ( τ ( i ) ) ≥ R (cid:15) } //

Original: Select subset of expressions
Revised: Select subset of expressions

Original: R
Revised: R

Original: { R ( τ ( i ) ) : R ( τ ( i ) ) ≥ R (cid:15) } //
Revised: { R ( τ ( i ) ) : R ( τ ( i ) ) ≥ R (cid:15) } //

Original: Select subset of rewards ˆ g
Revised: Select subset of rewards ˆ g

Original: =
Revised: =

Original: (( R − b ) ∇ θ log p
Revised: (( R − b ) ∇ θ log p

Original: ( T | θ )) //
Revised: ( T | θ )) //

Original: Compute policy gradient
Revised: Compute policy gradient

Original: =
Revised: =

Original: ( λ H ∇
Revised: ( λ H ∇

Original: H ( T | θ )) //
Revised: H ( T | θ )) //

Original: Compute entropy gradient θ θ + α
Revised: Compute entropy gradient θ θ + α

Original: ( ˆ g 1 + ˆ g 2 ) //
Revised: ( ˆ g 1 + ˆ g 2 ) //

Original: Apply gradients b
Revised: Apply gradients b

Original: · ReduceMean ( R ) + (1 − β ) b // Update baseline
Revised: · ReduceMean ( R ) + (1 − β ) b // Update baseline

Original: (cid:63)
Revised: (cid:63)

Original: [Table]
Revised: [Table]

Original: // Update best expression
Revised: // Update best expression

Original: [Table]
Revised: [Table]

Original: Entropy bonus.
Revised: Entropy bonus.

Original: We provide a bonus to the loss function proportional to the entropy of the sampled expressions.
Revised: We provide a bonus to the loss function proportional to the entropy of the sampled expressions.

Original: In accordance with the maximum entropy reinforcement learning framework (Haarnoja et al., 2018), this bonus serves two purposes.
Revised: In accordance with the maximum entropy reinforcement learning framework (Haarnoja et al., 2018), this bonus serves two purposes.

Original: First, it encourages the RNN to explore more expressions, preventing premature convergence to a local optimum.
Revised: First, it encourages the RNN to explore more expressions, preventing premature convergence to a local optimum.

Original: In practice, this often leads to a better end result.
Revised: In practice, this often leads to a better end result.

Original: Second, it encourages the RNN to assign equal likelihood to different expressions that have equal ﬁtness.
Revised: Second, it encourages the RNN to assign equal likelihood to different expressions that have equal ﬁtness.

Original: Risk-seeking.
Revised: Risk-seeking.

Original: The policy performance, J , is deﬁned as an expectation .
Revised: The policy performance, J , is deﬁned as an expectation .

Original: However, in practice, the performance of symbolic regression is measured by the single or few best expressions.
Revised: However, in practice, the performance of symbolic regression is measured by the single or few best expressions.

Original: Thus, we employ a novel risk-seeking technique in which only the top (cid:15) percentile samples from each batch are used in the gradient computation.
Revised: Thus, we employ a novel risk-seeking technique in which only the top (cid:15) percentile samples from each batch are used in the gradient computation.

Original: This has the effect of increasing best-case performance at the expense of lower worst-case and average performances.
Revised: This has the effect of increasing best-case performance at the expense of lower worst-case and average performances.

Original: This process is essentially the opposite of the EpOpt technique (Rajeswaran et al., 2016) used for risk-averse reinforcement learning, in which only the bottom (cid:15) percentile samples from each batch are used.
Revised: This process is essentially the opposite of the EpOpt technique (Rajeswaran et al., 2016) used for risk-averse reinforcement learning, in which only the bottom (cid:15) percentile samples from each batch are used.

Original: The complete algorithm, including reward baseline, complexity penalty, entropy bonus, and risk-seeking, is shown in Algorithm 3.
Revised: The complete algorithm, including reward baseline, complexity penalty, entropy bonus, and risk-seeking, is shown in Algorithm 3.

Original:  R ESULTS AND D ISCUSSION
Revised:  R ESULTS AND D ISCUSSION

Original: Evaluating DSR.
Revised: Evaluating DSR.

Original: We evaluated DSR on a set of 12 commonly used symbolic regression benchmarks (Uy et al., 2011), as well as 4 additional variants in which we introduced real-valued constants to demonstrate the inner optimization loop.
Revised: We evaluated DSR on a set of 12 commonly used symbolic regression benchmarks (Uy et al., 2011), as well as 4 additional variants in which we introduced real-valued constants to demonstrate the inner optimization loop.

Original: Each benchmark is deﬁned by a ground truth expression, a training and testing dataset, and set of allowed operators, described in Table 2 in Appendix A.
Revised: Each benchmark is deﬁned by a ground truth expression, a training and testing dataset, and set of allowed operators, described in Table 2 in Appendix A.

Original: The training data is used to compute the reward for each candidate expression, the test data is used to evaluate the best found candidate
Revised: The training data is used to compute the reward for each candidate expression, the test data is used to evaluate the best found candidate

Original: [Table]
Revised: [Table]

Original: expression at the end of training, and the ground truth function is used to determine whether the best found candidate expression was correctly recovered.
Revised: expression at the end of training, and the ground truth function is used to determine whether the best found candidate expression was correctly recovered.

Original: As a baseline
Revised: As a baseline

Original: [Table]
Revised: [Table]

Original: we compared against standard GP-based symbolic regression.
Revised: we compared against standard GP-based symbolic regression.

Original: To ensure fair comparison
Revised: To ensure fair comparison

Original: [Table]
Revised: [Table]

Original: the same constant optimizer (BFGS) was used for both methods.
Revised: the same constant optimizer (BFGS) was used for both methods.

Original: We ran independent training runs for GP and
Revised: We ran independent training runs for GP and

Original: DSR for each benchmark expression ( n = 100 for benchmarks without constants; n = 10 for benchmarks with constants).
Revised: DSR for each benchmark expression ( n = 100 for benchmarks without constants; n = 10 for benchmarks with constants).

Original: For each experiment, we generated 1,000 candidate expressions per generation/iteration for 1,000 generations/iterations, resulting in 1,000,000 total expressions.
Revised: For each experiment, we generated 1,000 candidate expressions per generation/iteration for 1,000 generations/iterations, resulting in 1,000,000 total expressions.

Original: For each training run, the expression with the best reward is selected and we record the NRMSE on the test data.
Revised: For each training run, the expression with the best reward is selected and we record the NRMSE on the test data.

Original: For GP, we used the open-source software package “deap” (Fortin et al., 2012).
Revised: For GP, we used the open-source software package “deap” (Fortin et al., 2012).

Original: For DSR, the RNN comprised a single-layer LSTM of 32 hidden units.
Revised: For DSR, the RNN comprised a single-layer LSTM of 32 hidden units.

Original: Additional hyperparameters and experiment details are provided in Appendix A.
Revised: Additional hyperparameters and experiment details are provided in Appendix A.

Original: In Table 1, we report the percentage of runs that correctly recover the expression and NRMSE on the test data for each benchmark.
Revised: In Table 1, we report the percentage of runs that correctly recover the expression and NRMSE on the test data for each benchmark.

Original: DSR signiﬁcantly outperforms GP in its ability to exactly recover benchmark expressions.
Revised: DSR signiﬁcantly outperforms GP in its ability to exactly recover benchmark expressions.

Original: DSR also outperforms GP in the average NRMSE across all expressions; however, we observe that for the few expressions with low or zero recovery rate (e.g. Nguyen-7, Nguyen-8, and Nguyen-12), GP sometimes exhibits lower NRMSE.
Revised: DSR also outperforms GP in the average NRMSE across all expressions; however, we observe that for the few expressions with low or zero recovery rate (e.g. Nguyen-7, Nguyen-8, and Nguyen-12), GP sometimes exhibits lower NRMSE.

Original: One explanation is that GP is more prone to overﬁtting the expression to the dataset.
Revised: One explanation is that GP is more prone to overﬁtting the expression to the dataset.

Original: As an evolutionary approach, GP directly modiﬁes the previous generation’s expressions, allowing it to make small “corrections” that decrease error each generation even if the functional form is far from correct.
Revised: As an evolutionary approach, GP directly modiﬁes the previous generation’s expressions, allowing it to make small “corrections” that decrease error each generation even if the functional form is far from correct.

Original: In contrast, in DSR the RNN “rewrites” each expression from scratch each iteration after learning from a gradient update, making it less prone to overﬁtting.
Revised: In contrast, in DSR the RNN “rewrites” each expression from scratch each iteration after learning from a gradient update, making it less prone to overﬁtting.

Original: N o a b l a t i o ns
Revised: [Figure] a [Figure] e n

Original: P a r e n t / s i b li n g
Revised: [Figure] n [Figure]

Original: R i s k - s ee k
Revised: R i s k [Figure] ee k

Revised: [Figure]

Original: i n g
Revised: n [Figure]

Original: E n t r o py b
Revised: E n t r o p

Original: o nu s
Revised: [Figure]

Original: R e w a r d b a s e li n eA ll i m p r o v e
Revised: R e w a r d b a s e li n e

Original: m e n t s

Original: C o n s t r a i n i n v e r s e

Original: C o n s t r a i n m i n / m a x C o n s t r a i n t r i g A ll c o n s t r a i n t s All c o n s t
Revised: A

Original: [Figure]
Revised: [Figure]

Original: a i n t
Revised: i m p r o

Original: [Figure]
Revised: [Figure]

Original: p
Revised: e

Original: [Figure]
Revised: [Figure]

Original: o
Revised: n

Original: [Figure]
Revised: [Figure]

Original: e
Revised: C o n

Original: [Figure]
Revised: [Figure]

Revised: i n

Revised: [Figure]

Original: N o a
Revised: e r s

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: n s
Revised: C

Original: P a r e nt /
Revised: [Figure] n

Original: [Figure]
Revised: [Figure]

Original: li n g
Revised: n v e r s

Original: R i s k - s ee k i n g E n t r o p y b
Revised: [Figure]

Original:  o nu s
Revised: C o n

Original: R e w a r d b a s e li n e A ll i m p
Revised: [Figure]

Original: r
Revised: t

Original: o v e m e n t s C o n s t r ai n i n v e r s e
Revised: [Figure] a

Original: C o n s t r a i n m i n / m a x
Revised: [Figure]

Original: C o n s t r a i n t r i g
Revised: m [Figure] n [Figure]

Original: A ll c o n s t r a i n t s
Revised: m [Figure] C o n s [Figure]

Original: A ll c o n s t r a i n t s & i m p r
Revised: t [Figure] g A

Original:  ov e
Revised: [Figure] o

Original: m e n ts G
Revised: [Figure] n t

Original: [Figure]
Revised: [Figure]

Revised: A

Revised: [Figure]

Revised: c o n

Revised: [Figure]

Revised: p

Original: [Figure]
Revised: [Figure]

Original:  Surprisingly, DSR consistently performed best without a complexity penalty, i.e. λ C = 0 .
Revised: o [Figure] [Figure] Surprisingly, DSR consistently performed best without a complexity penalty, i.e. λ C = 0 .

Original: Due to the autoregressive nature of the RNN, shorter expressions tend to exhibit higher likelihood than longer ones.
Revised: Due to the autoregressive nature of the RNN, shorter expressions tend to exhibit higher likelihood than longer ones.

Original: We postulate that this property produces a self-regularization effect that precludes the need for an explicit complexity penalty.
Revised: We postulate that this property produces a self-regularization effect that precludes the need for an explicit complexity penalty.

Original: Ablation studies.
Revised: Ablation studies.

Original: Algorithm 3 includes several additional components relative to the “vanilla” Algorithm 2.
Revised: Algorithm 3 includes several additional components relative to the “vanilla” Algorithm 2.

Original: We performed a series of ablation studies to quantify the effect of each of these components, along with the effects of the various constraints on the search space, and including the parent and sibling as input to the RNN instead of the previous node value.
Revised: We performed a series of ablation studies to quantify the effect of each of these components, along with the effects of the various constraints on the search space, and including the parent and sibling as input to the RNN instead of the previous node value.

Original: In Figure 2, we performed DSR on the set of 12 Nguyen benchmarks for each ablation.
Revised: In Figure 2, we performed DSR on the set of 12 Nguyen benchmarks for each ablation.

Original: DSR is still competitive with GP even when removing all improvements and all constraints.
Revised: DSR is still competitive with GP even when removing all improvements and all constraints.

Original: Noisy data and amount of data.
Revised: Noisy data and amount of data.

Original: We evaluated the robustness of DSR to noisy data by adding independent Gaussian noise to the dependent variable, with mean zero and standard deviation proportional to the root-mean-square of the dependent variable in the training data.
Revised: We evaluated the robustness of DSR to noisy data by adding independent Gaussian noise to the dependent variable, with mean zero and standard deviation proportional to the root-mean-square of the dependent variable in the training data.

Original: In Figure 3, we varied the proportionality constant from 0 (noiseless) to 10 − 1 and compared the performance of GP and DSR across the set of
Revised: In Figure 3, we varied the proportionality constant from 0 (noiseless) to 10 − 1 and compared the performance of GP and DSR across the set of

Original: Nguyen benchmarks.
Revised: Nguyen benchmarks.

Original: DSR still outperforms GP in both recovery rate and NRMSE across noise levels.
Revised: DSR still outperforms GP in both recovery rate and NRMSE across noise levels.

Original: Symbolic regression excels in the low-data setting when data is noiseless, hence, the benchmark expressions included herein include only 20 data points (see Table 2).
Revised: Symbolic regression excels in the low-data setting when data is noiseless, hence, the benchmark expressions included herein include only 20 data points (see Table 2).

Original: With added noise, increasing the amount of data smooths the reward function and may help prevent overﬁtting.
Revised: With added noise, increasing the amount of data smooths the reward function and may help prevent overﬁtting.

Original: Thus, we repeated the noise experiments using the same benchmarks but with 10-fold larger training datasets (200 points data points).
Revised: Thus, we repeated the noise experiments using the same benchmarks but with 10-fold larger training datasets (200 points data points).

Original: As expected, recovery rates tend to increase for both methods; however, DSR maintains a much larger improvement than
Revised: As expected, recovery rates tend to increase for both methods; however, DSR maintains a much larger improvement than

Original: GP at higher noise levels.
Revised: GP at higher noise levels.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce an unconventional approach to symbolic regression based on reinforcement learning that outperforms a standard GP-based method on recovering exact expressions on benchmark problems, both with and without added noise.
Revised: We introduce an unconventional approach to symbolic regression based on reinforcement learning that outperforms a standard GP-based method on recovering exact expressions on benchmark problems, both with and without added noise.

Original: Since both DSR and GP generate expression trees, there are many opportunities for hybrid methods, for example including several generations of evolutionary operations in the inner optimization loop.
Revised: Since both DSR and GP generate expression trees, there are many opportunities for hybrid methods, for example including several generations of evolutionary operations in the inner optimization loop.

Original: From the perspective of AutoML, the main contributions are deﬁning a ﬂexible distribution over hierarchical, variable-length objects that allows imposing in situ constraints, and using risk-seeking training to optimize best-case performance.
Revised: From the perspective of AutoML, the main contributions are deﬁning a ﬂexible distribution over hierarchical, variable-length objects that allows imposing in situ constraints, and using risk-seeking training to optimize best-case performance.

Original: Thus, we note that our framework is easily extensible to domains outside symbolic regression, which we save for future work; for example, searching the space of organic molecular structures for high binding afﬁnity to a reference compound.
Revised: Thus, we note that our framework is easily extensible to domains outside symbolic regression, which we save for future work; for example, searching the space of organic molecular structures for high binding afﬁnity to a reference compound.

Original: We chose symbolic regression to demonstrate our framework in part because of the large search space, broad applicability, computationally expedient inner optimization loop (sub-second), and availability of vetted benchmark problems and baseline methods.
Revised: We chose symbolic regression to demonstrate our framework in part because of the large search space, broad applicability, computationally expedient inner optimization loop (sub-second), and availability of vetted benchmark problems and baseline methods.

Original: A PPENDIX A
Revised: A PPENDIX A

Original: Hyperparameters.
Revised: Hyperparameters.

Original: DSR hyperparameters are listed in Table 3.
Revised: DSR hyperparameters are listed in Table 3.

Original: GP hyperparameters are listed in Table 4.
Revised: GP hyperparameters are listed in Table 4.

Original: The same hyperparameters were used for all experiments and all benchmark expressions.
Revised: The same hyperparameters were used for all experiments and all benchmark expressions.

Original: Additional details for performance comparison experiments.
Revised: Additional details for performance comparison experiments.

Original: Details of the benchmark symbolic regression problems are shown in Table 2.
Revised: Details of the benchmark symbolic regression problems are shown in Table 2.

Original: All benchmarks use the function set { + , − , × , ÷ , sin , cos , exp , log } .
Revised: All benchmarks use the function set { + , − , × , ÷ , sin , cos , exp , log } .

Original: To ensure closures, we use protected versions of operators: log returns the logarithm of the absolute value of its argument, and ÷ , exp , and log return 1 for arguments that would cause overﬂow or other errors.
Revised: To ensure closures, we use protected versions of operators: log returns the logarithm of the absolute value of its argument, and ÷ , exp , and log return 1 for arguments that would cause overﬂow or other errors.

Original: Benchmarks without constants can be recovered exactly, thus recovery is deﬁned by exact correctness (modulo ﬂoating point precision error).
Revised: Benchmarks without constants can be recovered exactly, thus recovery is deﬁned by exact correctness (modulo ﬂoating point precision error).

Original: Note Nguyen-8 can be recovered via exp( xx + x log( x )) and Nguyen-11 can be recovered via exp( y log( x )) .
Revised: Note Nguyen-8 can be recovered via exp( xx + x log( x )) and Nguyen-11 can be recovered via exp( y log( x )) .

Original: [Table]
Revised: [Table]

Original: No
Revised: No

Original: [Table] [Table]
Revised: [Table] [Table]

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: U ( −
Revised: U ( −

Original: [Table]
Revised: [Table]

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: 2 +
Revised: 2 +

Original: [Table]
Revised: [Table]

Original: U ( − 1 , 1 , 20)
Revised: U ( − 1 , 1 , 20)

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: sin
Revised: sin

Original: [Table] [Table]
Revised: [Table] [Table]

Original: cos
Revised: cos

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: U ( − 1 , 1 , 20)
Revised: U ( − 1 , 1 , 20)

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: sin( x ) + sin( x + x 2 )
Revised: sin( x ) + sin( x + x 2 )

Original: U ( − 1 , 1 , 20)
Revised: U ( − 1 , 1 , 20)

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: log( x + 1) + log( x 2 + 1)
Revised: log( x + 1) + log( x 2 + 1)

Original: U (0 , 2 , 20)
Revised: U (0 , 2 , 20)

Original: No
Revised: No

Original: Nguyen-  √ x
Revised: Nguyen-  √ x

Original: U (0 , 4 , 20)
Revised: U (0 , 4 , 20)

Original: No
Revised: No

Original: Nguyensin( x ) + sin( y 2 )
Revised: Nguyensin( x ) + sin( y 2 )

Original: U (0 , 1 , 20)
Revised: U (0 , 1 , 20)

Original: No
Revised: No

Original: Nguyen2 sin( x ) cos( y )
Revised: Nguyen2 sin( x ) cos( y )

Original: U (0 , 1 , 20)
Revised: U (0 , 1 , 20)

Original: No
Revised: No

Original: Nguyenx y
Revised: Nguyenx y

Original: U (0 , 1 , 20)
Revised: U (0 , 1 , 20)

Original: No
Revised: No

Original: Nguyenx 4 − x 3 + 12 y 2 − y
Revised: Nguyenx 4 − x 3 + 12 y 2 − y

Original: U (0 , 1 , 20)
Revised: U (0 , 1 , 20)

Original: No
Revised: No

Original: Constant3 .
Revised: Constant3 .

Original: 39 x 3 + 2 .
Revised: 39 x 3 + 2 .

Original: 12 x 2 + 1 .
Revised: 12 x 2 + 1 .

Original: 78 x
Revised: 78 x

Original: U ( − 1 , 1 , 20)
Revised: U ( − 1 , 1 , 20)

Original: Yes
Revised: Yes

Original: Constantsin( x 2 ) cos( x ) − 0 .
Revised: Constantsin( x 2 ) cos( x ) − 0 .

Original: U ( − 1 , 1 , 20)
Revised: U ( − 1 , 1 , 20)

Original: Yes
Revised: Yes

Original: Constantsin(1 .
Revised: Constantsin(1 .

Original: 5 x ) cos(0 .
Revised: 5 x ) cos(0 .

Original: 5 y )
Revised: 5 y )

Original: U (0 , 1 , 20)
Revised: U (0 , 1 , 20)

Original: Yes
Revised: Yes

Original: Constant2 .
Revised: Constant2 .

Original: 7 x y
Revised: 7 x y

Original: U (0 , 1 , 20)
Revised: U (0 , 1 , 20)

Original: Yes
Revised: Yes

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For benchmarks with constants, constants are optimized using BFGS with an initial guess of 1.0 for each constant.
Revised: For benchmarks with constants, constants are optimized using BFGS with an initial guess of 1.0 for each constant.

Original: We ensured that all benchmarks with constants do not get stuck in a poor local optimum when optimizing with BFGS and the candidate functional form is correct.
Revised: We ensured that all benchmarks with constants do not get stuck in a poor local optimum when optimizing with BFGS and the candidate functional form is correct.

Original: Since ﬂoating point constants cannot be recovered exactly, for benchmarks with constants we manually determined correctness of the functional form by inspection.
Revised: Since ﬂoating point constants cannot be recovered exactly, for benchmarks with constants we manually determined correctness of the functional form by inspection.

Original: Since constant optimization is a computational bottleneck, we limited each expression to three constants for both DSR and GP experiments.
Revised: Since constant optimization is a computational bottleneck, we limited each expression to three constants for both DSR and GP experiments.

Original: For GP, the initial population of expressions is generated using the “full” method (Koza, 1992) with depth randomly selected between d min and d max .
Revised: For GP, the initial population of expressions is generated using the “full” method (Koza, 1992) with depth randomly selected between d min and d max .

Original: The selection operator is deﬁned by deterministic tournament selection, in which the expression with the best ﬁtness among k randomly selected expressions is chosen.
Revised: The selection operator is deﬁned by deterministic tournament selection, in which the expression with the best ﬁtness among k randomly selected expressions is chosen.

Original: The crossover operator is deﬁned by swapping random subtrees between two expressions.
Revised: The crossover operator is deﬁned by swapping random subtrees between two expressions.

Original: The point mutation operator is deﬁned by replacing a random subtree with a new subtree initialized using the “full” method with depth randomly selected between d min and d max .
Revised: The point mutation operator is deﬁned by replacing a random subtree with a new subtree initialized using the “full” method with depth randomly selected between d min and d max .

Original: Additional details for ablation studies.
Revised: Additional details for ablation studies.

Original: In Figure 2, “Parent/sibling” denotes that the previous node of the traversal is provided as input to the RNN, rather than the parent and sibling node.
Revised: In Figure 2, “Parent/sibling” denotes that the previous node of the traversal is provided as input to the RNN, rather than the parent and sibling node.

Original: “Risk-seeking” denotes no risk-seeking, equivalent to (cid:15) = 1 .
Revised: “Risk-seeking” denotes no risk-seeking, equivalent to (cid:15) = 1 .

Original: “Entropy bonus” denotes no entropy bonus, equivalent to λ H = 0 .
Revised: “Entropy bonus” denotes no entropy bonus, equivalent to λ H = 0 .

Original: “Reward baseline” denotes no reward baseline, equivalent to β = 0 .
Revised: “Reward baseline” denotes no reward baseline, equivalent to β = 0 .

Original: “All improvements” denotes combining ablations for Parent/sibling, Risk, Entropy, and Baseline.
Revised: “All improvements” denotes combining ablations for Parent/sibling, Risk, Entropy, and Baseline.

Original: “Constrain trig” denotes no constraint precluding nested trigonometric operators.
Revised: “Constrain trig” denotes no constraint precluding nested trigonometric operators.

Original: “Constrain inverse” denotes no constraint precluding inverse unary operators.
Revised: “Constrain inverse” denotes no constraint precluding inverse unary operators.

Original: “Constrain min/max” denotes no constraint precluding minimum or maximum length.
Revised: “Constrain min/max” denotes no constraint precluding minimum or maximum length.

Original: (If the maximum length is reached, the expression is appended with x until complete.)
Revised: (If the maximum length is reached, the expression is appended with x until complete.)

Original: “Constraints” denotes combining ablations for Trig, Inverse, and Min/max.
Revised: “Constraints” denotes combining ablations for Trig, Inverse, and Min/max.

Original: “All constraints & improvements” denotes combining all ablations.
Revised: “All constraints & improvements” denotes combining all ablations.

Original: Training curves.
Revised: Training curves.

Original: Figures 4 and 5 show the reward (1 / (1 + NRMSE )) and recovery rate, respectively, as a function of training step (DSR iteration or GP generation).
Revised: Figures 4 and 5 show the reward (1 / (1 + NRMSE )) and recovery rate, respectively, as a function of training step (DSR iteration or GP generation).

Original: For benchmarks with constants, the constant optimizer can allow both algorithms to quickly reach reward near 1.0.
Revised: For benchmarks with constants, the constant optimizer can allow both algorithms to quickly reach reward near 1.0.

Original: For these benchmarks, we provide zoomed inset plots demonstrate if and when all independent training runs correctly recovery the expression (Figure 4: Constant-1, Constant-2, and Constant-3).
Revised: For these benchmarks, we provide zoomed inset plots demonstrate if and when all independent training runs correctly recovery the expression (Figure 4: Constant-1, Constant-2, and Constant-3).

Original: Note that the NRMSE and recovery values in Tablecorrespond to the ﬁnal point on each curve in Figures 4 and 5, respectively.
Revised: Note that the NRMSE and recovery values in Tablecorrespond to the ﬁnal point on each curve in Figures 4 and 5, respectively.

Original: Additional subroutines.
Revised: Additional subroutines.

Original: DSR includes several subroutines used when sampling an expression from the
Revised: DSR includes several subroutines used when sampling an expression from the

Original: RNN and during training.
Revised: RNN and during training.

Original: In Subroutine 1, we describe the function ApplyConstraints ( ψ, L , τ ) used in
Revised: In Subroutine 1, we describe the function ApplyConstraints ( ψ, L , τ ) used in

Original: Algorithm 1, which zeros out the probabilities of tokens that would violate any given constraints.
Revised: Algorithm 1, which zeros out the probabilities of tokens that would violate any given constraints.

Original: Within this subroutine, the user-speciﬁc function ViolatesConstraint ( τ, L i ) returns TRUE if adding the i th token of the library L to the partial traversal τ would violate any user-speciﬁed constraints, and FALSE otherwise.
Revised: Within this subroutine, the user-speciﬁc function ViolatesConstraint ( τ, L i ) returns TRUE if adding the i th token of the library L to the partial traversal τ would violate any user-speciﬁed constraints, and FALSE otherwise.

Original: In Subroutine 2, we describe the function ParentSibling ( τ ) used in Algorithm 1, which computes the parent and sibling of the next token to be sampled.
Revised: In Subroutine 2, we describe the function ParentSibling ( τ ) used in Algorithm 1, which computes the parent and sibling of the next token to be sampled.

Original: This subroutine uses the following logic.
Revised: This subroutine uses the following logic.

Original: If the ﬁnal node in the partial traversal is a unary or binary operator, then that node is the parent and there is no sibling.
Revised: If the ﬁnal node in the partial traversal is a unary or binary operator, then that node is the parent and there is no sibling.

Original: Otherwise, the subroutine iterates backward through the traversal until ﬁnding a node with an unselected child node.
Revised: Otherwise, the subroutine iterates backward through the traversal until ﬁnding a node with an unselected child node.

Original: That node is the parent and the subsequent node is the child.
Revised: That node is the parent and the subsequent node is the child.

Original: Within this subroutine, the function Arity ( τ i ) simply returns the arity (number of arguments) of token τ i , i.e. two for binary operators, one for unary operators, or zero for input variables and constants.
Revised: Within this subroutine, the function Arity ( τ i ) simply returns the arity (number of arguments) of token τ i , i.e. two for binary operators, one for unary operators, or zero for input variables and constants.

Original: In Subroutine 3, we describe the function
Revised: In Subroutine 3, we describe the function

Original: OptimizeConstants ( τ, X, y ) used in Algorithms 2 and 3, which optimizes the placeholder constants c of an expression with respect to input dataset ( X, y ) using a black-box optimizer, e.g. BFGS.
Revised: OptimizeConstants ( τ, X, y ) used in Algorithms 2 and 3, which optimizes the placeholder constants c of an expression with respect to input dataset ( X, y ) using a black-box optimizer, e.g. BFGS.

Original: Within this subroutine, the function Instantiate ( τ ) instantiates the symbolic expression as a function f
Revised: Within this subroutine, the function Instantiate ( τ ) instantiates the symbolic expression as a function f

Original: ( X ; c ) with inputs X and parameters (constants) c , and the function ReplaceConstants ( τ, c (cid:63) ) replaces the placeholder constants in the expression with the optimized constants.
Revised: ( X ; c ) with inputs X and parameters (constants) c , and the function ReplaceConstants ( τ, c (cid:63) ) replaces the placeholder constants in the expression with the optimized constants.

Original: [Figure]
Revised: [Figure]

Original: Nguyen
Revised: Nguyen

Original: [Figure]
Revised: [Figure]

Original: NguyenNguyen-
Revised: NguyenNguyen-

Original: [Figure]
Revised: [Figure]

Original: Nguyen-
Revised: Nguyen-

Original: [Figure]
Revised: [Figure]

Original: NguyenNguyen-
Revised: NguyenNguyen-

Original: [Figure]
Revised: [Figure]

Original: Nguyen
Revised: Nguyen

Original: [Figure]
Revised: [Figure]

Original: Nguyen-
Revised: Nguyen-

Original: [Figure]
Revised: [Figure]

Original: Nguyen-
Revised: Nguyen-

Original: [Figure]
Revised: [Figure]

Original: Nguyen-
Revised: Nguyen-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Nguyen
Revised: Nguyen

Original: [Figure]
Revised: [Figure]

Original: Nguyen
Revised: Nguyen

Original: [Figure]
Revised: [Figure]

Original: v e r y r
Revised: v e r y r

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: Nguyen
Revised: Nguyen

Original: [Figure]
Revised: [Figure]

Original: Nguyen-
Revised: Nguyen-

Original: [Figure]
Revised: [Figure]

Original: r y r a t
Revised: r y r a t

Original: [Figure]
Revised: [Figure]

Original: Nguyen
Revised: Nguyen

Original: [Figure]
Revised: [Figure]

Original: Nguyen
Revised: Nguyen

Original: [Figure]
Revised: [Figure]

Original: Nguyen
Revised: Nguyen

Original: [Figure]
Revised: [Figure]

Original: Subroutine 1: Apply generic constraints in situ when sampling from the RNN 1 function ApplyConstraints ( ψ, L , τ ) input : Categorical probabilities ψ ; corresponding library of tokens L ; partially sampled traversal τ output: Adjusted categorical probabilities ψ
Revised: Subroutine 1: Apply generic constraints in situ when sampling from the RNN 1 function ApplyConstraints ( ψ, L , τ ) input : Categorical probabilities ψ ; corresponding library of tokens L ; partially sampled traversal τ output: Adjusted categorical probabilities ψ

Original: L = | ψ | // Length of library for i = 1 , . . .
Revised: L = | ψ | // Length of library for i = 1 , . . .

Original: , L do if ViolatesConstraint ( τ, L i ) then ψ i ←// Constrain that token end ψ ← ψ (cid:80) i ψ
Revised: , L do if ViolatesConstraint ( τ, L i ) then ψ i ←// Constrain that token end ψ ← ψ (cid:80) i ψ

Original: i // Normalize back to return ψ
Revised: i // Normalize back to return ψ

Original: Subroutine 2: Computing parent and sibling inputs to the RNN 1 function ParentSibling ( τ ) input : Partially sampled traversal τ output: Parent and sibling tokens of the next token to be sampled
Revised: Subroutine 2: Computing parent and sibling inputs to the RNN 1 function ParentSibling ( τ ) input : Partially sampled traversal τ output: Parent and sibling tokens of the next token to be sampled

Original: L = | τ | // Length of partial traversal counter =// Counter for number of unselected nodes if Arity( τ L ) > 0 then parent =
Revised: L = | τ | // Length of partial traversal counter =// Counter for number of unselected nodes if Arity( τ L ) > 0 then parent =

Original: τ L sibling =
Revised: τ L sibling =

Original: empty return parent (cid:107) sibling for i =
Revised: empty return parent (cid:107) sibling for i =

Original: L, . .
Revised: L, . .

Original: . , 1 do //
Revised: . , 1 do //

Original: Iterate backward counter ← counter + Arity ( τ i ) − if counter = 

Revised: Iterate backward counter ← counter + Arity ( τ i ) − if counter =

Original: then parent =
Revised: then parent =

Original: τ i sibling =
Revised: τ i sibling =

Original: τ i +  return parent (cid:107) sibling end
Revised: τ i +  return parent (cid:107) sibling end

Original: Subroutine 3: Optimize the constants of an expression (inner optimization loop) 1 function OptimizeConstants ( τ, X, y ) input : Expression τ with placeholder constants c ; input dataset ( X, y ) output: Expression τ (cid:63) with optimized constants c (cid:63) f
Revised: Subroutine 3: Optimize the constants of an expression (inner optimization loop) 1 function OptimizeConstants ( τ, X, y ) input : Expression τ with placeholder constants c ; input dataset ( X, y ) output: Expression τ (cid:63) with optimized constants c (cid:63) f

Original: ( X, c )
Revised: ( X, c )

Original: =
Revised: =

Original: Instantiate ( τ ) //
Revised: Instantiate ( τ ) //

Original: Instantiate the symbolic expression c (cid:63) =
Revised: Instantiate the symbolic expression c (cid:63) =

Original: arg min c (cid:107) y − f
Revised: arg min c (cid:107) y − f

Original: ( X, c ) (cid:107)// Minimize error (e.g. with BFGS) τ
Revised: ( X, c ) (cid:107)// Minimize error (e.g. with BFGS) τ