N EURAL -G UIDED S YMBOLIC R EGRESSION WITH A SYMPTOTIC C ONSTRAINTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Symbolic regression is a type of discrete optimization problem that involves searching expressions that ﬁt given data points.
In many cases, other mathematical constraints about the unknown expression not only provide more information beyond just values at some inputs, but also effectively constrain the search space.
We identify the asymptotic constraints of leading polynomial powers as the function approaches 0 and ∞ as useful constraints and create a system to use them for symbolic regression.
The ﬁrst part of the system is a conditional production rule generating neural network which preferentially generates production rules to construct expressions with the desired leading powers, producing novel expressions outside the training domain.
The second part, which we call Neural-Guided Monte Carlo Tree Search, uses the network during a search to ﬁnd an expression that conforms to a set of data points and desired leading powers.
Lastly, we provide an extensive experimental validation on thousands of target expressions showing the efﬁcacy of our system compared to exiting methods for ﬁnding unknown functions outside of the training set.
I NTRODUCTION
The long standing problem of symbolic regression tries to search expressions in large space that ﬁt given data points (Koza & Koza, 1992; Schmidt & Lipson, 2009).
These mathematical expressions are much more like discovered mathematical laws that have been an essential part of the natural sciences for centuries.
Since the size of the search space increases exponentially with the length of expressions, current search methods can only scale to ﬁnd expressions of limited length.
Moreover, current symbolic regression techniques fail to exploit a key value of mathematical expressions that has traditionally been well used by natural scientists.
Symbolically derivable properties such as bounds, limits, and derivatives can provide signiﬁcant guidance to ﬁnding an appropriate expression.
In this work, we consider one such property corresponding to the behavior of the unknown function as it approaches 0 and ∞ .
Many expressions have a deﬁned leading polynomial power in these limits.
For examples when x → ∞ , 2 x 2 + 5 x has a leading power of 2 (because the expression behaves like x 2 ) and 1 /x
+ 1 /x has a leading power of − 1 .
We call these properties “asymptotic constraints” because this kind of property is known a priori for some physical systems before the detailed law is derived.
For example, most materials have a heat capacity proportional to T 3 at low temperatures T and the gravitational ﬁeld of planets (at distance r ) should behave as 1 /r as r → ∞ .
Asymptotic constraints not only provide more information about the expression, leading to better extrapolation, but also constrain the search in the desired semantic subspace, making the search more tractable in much larger space.
These constraints can not be simply incorporated using syntactic restrictions over the grammar of expressions.
We present a system to effectively use asymptotic constraints for symbolic regression, which has two main parts.
The ﬁrst is a conditional production rule generating neural network (NN) of the desired polynomial leading powers that generates production rules to construct novel expressions (both syntactically and semantically) and, more surprisingly, generalize to leading powers not in the training set.
The second part is a NeuralGuided Monte Carlo Tree Search (NG-MCTS) that uses this NN to probabilistically guide the search at every step to ﬁnd expressions that ﬁt a set of data points.
Finally, we provide an extensive empirical evaluation of the system compared to several strong baseline techniques.
We examine both the NG-MCTS and conditional production rule generating NN alone.
In sharp contrast to almost all previous symbolic regression work, we evaluate our technique on thousands of target expressions and show that NG-MCTS can successfully ﬁnd the target expressions in a much larger fraction of cases ( 71% ) than other methods ( 23% ) with search space sizes of more than 10 50 expressions.
In summary, this paper makes the following key contributions: 1) We identify asymptotic constraints as important additional information for symbolic regression tasks.
We develop a conditional production rule generating NN to learn a distribution over (syntactically-valid) production rules conditioned on the asymptotic constraints.
We develop the NG-MCTS algorithm that uses the conditional production rule generating NN to efﬁciently guide the MCTS in large space.
We extensively evaluate our production rule generating NN to demonstrate generalization for leading powers, and show that the NG-MCTS algorithm signiﬁcantly outperforms previous techniques on thousands of tasks.
P ROBLEM D EFINITION
In order to demonstrate how prior knowledge can be incorporated into symbolic regression, we construct a symbolic space using a context-free grammar G :
[Equation]
This expression space covers a rich family of rational expressions, and the size of the space can be further parameterized by a bound on the maximum sequence length.
For an expression f ( x ) , the leading power at x 0 is deﬁned as P x → x 0 [ f ] =
p s.t.
lim x → x 0 f ( x ) /x
p = non-zero constant .
In this paper, we consider the leading powers at x 0 ∈ { 0 , ∞} as additional speciﬁcation.
Let S ( G, k ) denote the space of all expressions in the Grammar G with a maximum sequence length k .
Conventional symbolic regression searches for a desired expression f ( x ) in the space of expressions S ( G, k )
that conforms to a set of data points { ( x, f ( x ))
| x ∈ D train } , i.e. ﬁnd a g ( x ) ∈ S ( G, k ) : φ ( g ( x ) , D train ) , where φ denotes the acceptance criterion, usually root mean square error (RMSE).
With the additional speciﬁcation of leading powers c (0) and c ( ∞ ) at 0 and ∞ , the problem becomes: ﬁnd a g ( x ) ∈ S ( G, k ) : φ ( g ( x ) , D train )
∧ ( P x →
[ g ] = c (0) )
∧ ( P x →∞ [ g ] =
c ( ∞ ) ) .
C ONDITIONAL P RODUCTION R ULE G ENERATING N EURAL N ETWORK
It is difﬁcult to directly incorporate the asymptotic constraints as syntactic restrictions over the grammar.
We, therefore, develop a neural architecture that learns to generate production rule in the grammar conditioned on the given asymptotic constraints. [Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
[Equation]
Selection S / T S + T T + T S + T + T Q + U(p( ・ |s t , 0, -1))
Q + U(p( ・ |s t , 0, -1))
Q + U(p( ・ |s t+1 , 0, -1))
Q + U(p( ・ |s t+1 , 0, -1))
S
Expansion
S / T S + T
T + T S + T + T
S
Evaluation
S / T S + T
T + T S + T + T 1 / x +
S
Backpropagation
S / T S + T
T + T S + T + T 1 / x +
Q
Q ~ p( ・ | ・ , 0, -1)
f
Figure 1(a) and (c) show an example of how an expression is parsed as a parse tree by the grammar deﬁned in Eq.
The parse tree in Figure 1(c) can be serialized into a production rule sequence r 1 , . . .
, r L by a preorder traversal (Figure 1(d)), where L denotes the length of the production rule sequence.
Figure 1(b) shows the leading powers of the exemplary expression in Figure 1(a).
The conditional distribution of an expression is parameterized as a sequential model
[Equation]
We build a NN (as shown in Figure 1(e)) to predict the next production rule r t +1 from a partial sequence r 1 , . . .
, r t and conditions c (0) , c ( ∞ ) .
During training, each expression in the training set is ﬁrst parsed as a production rule sequence.
Then a partial sequence of length t ∈ { 1 , . . .
, L − 1 } is sampled randomly as the input and the ( t + 1) -th production rule is selected as the output (see blue and orange text in Figure 1(d)).
Each production rule of the partial sequence is represented as an embedding vector of size 10.
The conditions are concatenated with each embedding vector.
This sequence of embedding vectors are fed into a bidirectional Gated Recurrent Units (GRU) (Cho et al., 2014) with 1000 units.
A softmax layer is applied to the ﬁnal output of GRU to obtain the raw probability distribution over the next production rules in Eq.
Note that not all the production rules are grammatically valid as the next production rule for a given partial sequence.
The partial sequence is equivalent to a partial parse tree.
The next production rule expands the leftmost non-terminal symbol in the partial parse tree.
For the partial sequence colored in blue in Figure 1(d), the next production rule expands non-terminal symbol T , which constrains the next production rule to only those with left-hand-side symbol T .
We use a stack to keep track of non-terminal symbols in a partial sequence as described in GVAE (Kusner et al., 2017).
A mask of valid production rules is computed from the input partial sequence.
This mask is applied to the raw probability distribution and the result is normalized to 1 as the output probability distribution.
The training loss is calculated as the cross entropy between the output probability distribution and the next target production rule.
It is trained from partial sequences sampled from expressions in the training set using validation loss for early stopping.
N EURAL -G UIDED M ONTE C ARLO T REE S EARCH
We now brieﬂy describe the NG-MCTS algorithm that uses the conditional production rule generating
NN to guide the symbolic regression search.
The discrepancy between the best found expression g ( x ) and the desired f ( x ) is evaluated on data points and leading powers.
The error on data points is measured by RMSE ∆ g {·} = (cid:113)(cid:80) x ∈D {·} ( f ( x ) − g ( x )) 2 / |D {·} | on training points D train : { 1 . 2 , 1 . 6 , 2 . 0 , 2 . 4 , 2 . 8 } , points in interpolation region D interpolation : { 1 . 4 , 1 . 8 , 2 . 2 , 2 . 6 } and points in extrapolation region D extrapolation : { 5 , 6 , 7 , 8 , 9 } .
The error on leading powers is measured by sum of absolute errors at 0 and ∞ , ∆ P [ g ] =
| P x → 0 [ f ] − P x → 0 [ g ] | + | P x →∞ [ f ] − P x →∞ [ g ] | .
The default choice of objective function for symbolic regression algorithms is ∆ g train alone.
With additional leading powers constraint, the objective function can be deﬁned as ∆ g train +∆ P [ g ] , which minimizes both the RMSE on the training points and the absolute difference of the leading powers.
Most symbolic regression algorithms are based on EA (Schmidt & Lipson, 2009), where it is nontrivial to incorporate our conditional production rule generating NN to guide the generation strategy in a step-by-step manner, as the mutation and cross-over operators perform transformations on fully completed expressions.
However, it is possible to incorporate a probability distribution over expressions in many heuristic search algorithms such as Monte Carlo Tree Search (MCTS).
MCTS is a heuristic search algorithm that has been shown to perform exceedingly well in problems with large combinatorial space, such as mastering the game of Go (Silver et al., 2016) and planning chemical syntheses (Segler et al., 2018).
In MCTS for symbolic regression, a partial parse tree sequence r 1 , . . .
, r t can be deﬁned as a state s t and the next production rule is a set of actions { a } .
In the selection step, we use a variant of the PUCT algorithm (Silver et al., 2016; Rosin, 2011) for exploration.
For MCTS, the prior probability distribution p ( a i | s t ) is uniform among all valid actions.
We develop NG-MCTS by incorporating the conditional production rule generating NN into MCTS for symbolic regression.
Figure 1(f) presents a visual overview of NG-MCTS.
In particular, the prior 1 Model implemented in TensorFlow (Abadi et al., 2016) and available in submitted materials.
[Table] [Table]
probability distribution p ( a i | s t , c (0) , c ( ∞ ) ) is computed by our conditional production rule generating
NN on the partial sequence and the desired conditions.
We run MCTS for 500 simulations for each desired expression f ( x ) .
The exploration strength is set to 50 and the production rule sequence length limit is set to 100.
The total number of expressions in this combinatorial space is 3 × 10 93 .
We run evolutionary algorithm (EA) (Appendix G) and grammar variational autoencoder (GVAE) (Kusner et al., 2017)
(Appendix H) with comparable computational setup for comparison.
E VALUATION
Dataset and Search Space: We denote the leading power constraint as a pair of integers ( P x → 0 [ f ] , P x →∞ [ f ]) and deﬁne the complexity of a condition by M [ f ] = | P x → 0 [ f ] | + | P x →∞ [ f ] | .
Obviously, expressions with M [ f ] = 4 are more complicated to construct than those with M [ f ] = 0 .
We create a dataset balanced on each condition, as described in Appendix A.
The conditional production rule generating NN is trained on 28837 expressions and validated on 4095 expressions, both with
M [ f ] ≤ 4 .
The training expressions are sampled sparsely , which are only 10 − 23 of the expressions within 31 production rules.
Symbolic regression tasks are evaluated on 4250 expressions in holdout sets with M [ f ] ≤ 4 , = 5 , = 6 .
The challenges are: 1) conditions M [ f ] = 5 , 6 do not exist in training set; 2) the search spaces of M [ f ] = 5 , 6 are 10 7 and 10 11 times larger than the training set.
5. E VALUATION OF S YMBOLIC R EGRESSION T ASKS
We now present the evaluation of our NG-MCTS method, where each step in the search is guided by the conditional production rule generating NN.
Recent developments of symbolic regression methods (Kusner et al., 2017; Sahoo et al., 2018) compare methods on only a few expressions, which may cause the performance to depend on random seed for initialization and delicate tuning.
To mitigate this, we apply different symbolic regression methods to search for expressions in holdout sets with thousands of expressions and compare their results in Table 1.
Additional comparison on a subset of holdout sets is reported in Appendix H.
Each expression is a point, where log 10 ∆ g ext .
obtained by NG-MCTS is on x-axis and those obtained by other three methods are on the y-axis.
We ﬁrst discuss the results for holdout set M [ f ] ≤ 4 .
Conventional symbolic regression only ﬁts on the data points D train .
EA solves 12 .
83% expressions, while MCTS only solves 0 .
54% expressions.
This suggests that compared to EA, MCTS is not efﬁcient in searching a large space with limited number of simulations.
The median errors ∆ P [ g ] are both 3 for hard expressions (expressions unsolved by all methods), which are large as maximum M [ f ] for this set is 4.
In order to examine the effect of leading powers, we use leading powers alone in MCTS (PW-ONLY).
The median of ∆ P [ g ] for hard expressions is reduced to 1 but the medians of ∆ g int .
and ∆ g ext . are signiﬁcantly higher.
We then add leading powers to the objective function together with data points.
MCTS + PW does not have a notable difference to MCTS.
However, EA + PW improves solved expressions to 23 .
37% and ∆ P [ g ] of hard expressions is 0.
This indicates adding leading power constraints in the objective function is helpful for symbolic regression.
Most importantly, we observe step-wise guidance of NN conditioned on leading powers can lead to even more signiﬁcant improvements compared to adding them in the objective function.
NG-MCTS solves 71 .
22% expressions in the holdout set, three times over the best EA + PW.
Note that both MCTS + PW and
EA + PW have access to the same input information.
Although EA has the lowest medians of ∆ g train and ∆ g int . , NG-MCTS is only slightly worse.
On the other hand, NG-MCTS outperforms on ∆ g ext .
and ∆ P [ g ] , which indicates that step-wise guidance of leading powers helps to generalize better in extrapolation than all other methods.
We also apply the aforementioned methods to search expressions in holdout sets M [ f ] = 5 and
M [ f ] = 6 .
The percentage of solved expressions decreases as M [ f ] increases as larger M [ f ] requires learning expressions with more complex syntactic structure.
The median of ∆ P [ g ] also increases with larger M [ f ] for the other methods, but the value for NG-MCTS is always zero.
This demonstrates that our NN model is able to successfully guide the NG-MCTS even for leading powers not appearing in the training set.
Due to the restriction on the computational time of Bayesian optimization for
GVAE, we evaluate GVAE + PW on a subset of holdout sets (Appendix H).
GVAE+ PW fails in holdout sets M [ f ] = 5 , 6 .
Overall, NG-MCTS still signiﬁcantly outperforms other methods in solved percentage and extrapolation.
Figure 2 compares ∆ g ext .
for each expression among different methods in holdout set M [ f ] = 5 .
The upper right cluster in each plot represents expressions unsolved by both methods.
Most of the plotted points are above the 1:1 line (dashed), which shows that NG-MCTS outperforms the others for most unsolved expressions in extrapolation.
Examples of expressions solved by NG-MCTS but unsolved by EA + PW and vice versa are presented in Appendix J. We also perform similar experiments with Gaussian noise on D train and NG-MCTS still outperforms all other methods ( Appendix K).
5. C ASE S TUDY : F ORCE F IELD P OTENTIAL
Molecular dynamics simulations (Alder & Wainwright, 1959) study the dynamic evolution of physical systems, with extensive applications in physics, quantum chemistry, biology and material science.
The interaction of atoms or coarse-grained particles (Kmiecik et al., 2016) is described by potential energy function called force ﬁeld, which is derived from experiments or computations of quantum mechanics algorithms.
Typically, researchers know the interactions in short and long ranges, which are examples
[Table]
of asymptotic constraints.
We propose a force ﬁeld potential U ( x )
= 1 /x + x + ( x − 1) 2 with
Coulomb interaction, uniform electric ﬁeld and harmonic interaction.
Assuming the true potential is unknown, the goal is to discover this expression.
As a physical potential, besides values at
D train : { 1 . 2 , 1 . 6 , 2 . 0 , 2 . 4 , 2 . 8 } , researchers also know the short ( x → 0 ) and long range ( x → ∞ ) behaviors as leading powers.
Table 2 shows the expressions found by NG-MCTS, GVAE, GVAE +
PW, EA and EA + PW, which are plotted in Figure 3.
NG-MCTS can ﬁnd the desired expression.
The second best method is GVAE + PW, differing by a constant of 1 from the true expression.
5. E VALUATION OF C ONDITIONAL P RODUCTION R ULE G ENERATING NN
NG-MCTS signiﬁcantly outperforms other methods on searching expressions in large space.
To better understand the effective guidance from NN, we demonstrate its ability to generate syntactically and semantically novel expressions given desired conditions.
In order to examine the NN alone, we directly sample from the model by Eq.
(2) instead of using MCTS.
The model predicts the probability distribution over the next production rules from the starting rule r 1 : O → S and desired condition ( c (0) , c ( ∞ ) ).
The next production rule is sampled from distribution p θ ( r 2 | r 1 , c (0) , c ( ∞ ) ) and then appended to r 1 .
Then r 3 is sampled from p θ ( r 3 | r 1 , r 2 , c (0) , c ( ∞ ) ) and appended to [ r 1 , r 2 ] .
This procedure is repeated until [ r 1 , . . .
, r L ] form a parse tree where all the leaf nodes are terminal, or the length of generated sequence reaches the prespeciﬁed limit, which is set to 100 for our experiments.
Baseline Models We compare NN with a number of baseline models that provide a probability distribution over the next production rules.
All these distributions are masked by the valid production rules computed from the partial sequence before sampling.
For each desired condition within | P x → 0 [ f ] | ≤ 9 and | P x →∞ [ f ] | ≤ 9 , k = 100 expressions are generated.
We consider the following baseline models: i) Neural Network No Condition (NNNC) : same setup as NN except no conditioning on leading powers, ii)
Random : uniform distribution over valid next production rules, iii) Full History (FH) : using full partial sequence and conditions to sample next production rule from its empirical distribution, iv)
Full History No Condition (FHNC) , v) Limited
History (LH) ( l ) : sampling next production rule from its empirical distribution given only the last l rules in the partial sequence, and vi) Limited History No Condition (LHNC) ( l ) .
Note that the aforementioned empirical distributions are derived from the training set { f } .
For limited history models, if l exceeds the length of the partial sequence, we instead take the full partial sequence.
More details about the baseline models can be found in Appendix B.
Metrics We propose four metrics to evaluate the performance.
For each condition ( c (0) , c ( ∞ ) ) , k expressions { g i } are generated from model p θ ( f | c (0) , c ( ∞ ) ) .
i)
Success Rate : proportion of generated expressions with leading powers ( P x →
[ g i ] , P x →∞ [ g i ]) that match the desired condition, ii) Mean
L1-distance : the mean L1-distance between ( P x →
[ g i ] , P x →∞ [ g i ]) and ( c (0) , c ( ∞ ) ) , iii) Syntactic
Novelty Rate : proportion of generated expressions that satisfy the condition and are syntactically novel (no syntactic duplicate of generated expression in the training set), and iv) Semantic Novelty
Rate : proportion of expressions satisfying the condition and are semantically novel (the expression obtained after normalizing the generated expression does not exist in the training set).
For example, expressions x + 1 and (1) + x are syntactically different, but semantically duplicate.
We use simplify function in SymPy (Meurer et al., 2017) to normalize expressions.
To avoid inﬂating the rates of syntactic and semantic novelty, we only count the number of unique syntactic and semantic novelties in terms of their expressions and simpliﬁed expressions, respectively.
Quantitative Evaluation Table 3 compares the model performance of baseline and NN models measured by different metrics.
We deﬁne M [ f ] ≤ 4 as in-sample condition region and M [ f ] > 4 as out-of-sample condition region .
In both regions, the generalization ability of the model is reﬂected by the number of syntactic and semantic novelties it generates, not just the number of successes.
For
[Table]
7.6 LHNC (16) 0.
example, FH behaves as a look-up table based method (i.e., sampling from the training set) so it has 100% success rate in in-sample condition region.
However, it is not able to generate any novel expressions.
NN has the best performance on the syntactic and semantic novelty rates in both the insample ( 35% and 2 .
7% ) and out-of-sample ( 1416 and 1084 ) condition regions by a signiﬁcant margin.
This indicates the generalization ability of the model to generate unseen expressions matching a desired condition.
It is worth pointing out that NNNC performs much worse than NN, which indicates that NN is not only learning the distribution of expressions in the dataset, but instead is also learning a conditional distribution to map leading powers to the corresponding expression distributions.
Furthermore, the L1-distance measures the deviation from the desired condition when not matching exactly.
NN has the least mean L1-distance in the out-of-sample condition region.
This suggests that for the unmatched expressions, NN prefers expressions with leading powers closer to the desired condition than all other models.
NN outperforms the other models not only on the metrics aggregated over all conditions, but also for individual conditions.
Figure 4 shows the metrics for NN and LHNC (8) on each condition.
NN performs better in the in-sample region (inside the red boundary) and also generalizes to more conditions in the out-of-sample region (outside the red boundary).
Qualitative Evaluation To better comprehend the learned NN model and its generative behavior, we also perform a task of expression completion given a structure template of the form 1 / (cid:3) − (cid:3) and a variety of desired conditions in Table 4.
For each condition, 1000 expressions are generated by
NN and the probability of each syntactically unique expression is computed from its occurrence.
We ﬁrst start with c (0) = 0 , c ( ∞ )
= 1 .
The completed expression g ( x ) is required to be a nonzero constant as x → 0 and g ( x ) → x as x → ∞ .
The top three probabilities are close since this task is relatively easy to complete.
We then repeat the task on c (0)
= − 1 , c ( ∞ ) =
1 and c (0) =
− 2 , c ( ∞ ) = 2 , which are still in the in-sample condition region and the model can still complete expressions that match the desired conditions.
We also show examples of c (0) = − 3 , c ( ∞ )
= 2 , which is in the out-of-sample condition region.
Note that to match condition c ( ∞ ) = − 3 , more complicated completion such as 1 / ( x ∗ ( x ∗ x )) has to be constructed by the model.
Even in this challenging case, the model can still generate some expressions that match the desired condition.
We also show examples of the syntactic novelties learned by model in Appendix L.
R ELATED W ORK
[Table]
Symbolic Regression: Schmidt & Lipson (2009) present a symbolic regression technique to learn natural laws from experimental data.
The symbolic space is deﬁned by operators + , − , ∗ , / , sin , cos , constants, and variables.
An expression is represented as a graph, where intermediate nodes represent operators and leaves represent coefﬁcients and variables.
The EA varies the structures to search new expressions using a score that accounts for both accuracy and the complexity of the expression.
This approach has been further used to get empirical expressions in electronic engineering (Ceperic et al., 2014), water resources (Klotz et al., 2017), and social science (Truscott & Korns, 2014).
GVAE (Kusner et al., 2017) was recently proposed to learn a generative model of structured arithmetic expressions and molecules, where the latent representation captures the underlying structure.
This model was further shown to improve a
Bayesian optimization based method for symbolic regression.
Similar to these approaches, most other approaches search for expressions from scratch using only data points (Schmidt & Lipson, 2009;
Ramachandran et al., 2017; Ouyang et al., 2018) without other symbolic constraints about the desired expression.
Abu-Mostafa (1994) suggests incorporating prior knowledge of a similar form to our property constraints, but actually implements those priors by adding additional data points and terms in the loss function.
Neural Program Synthesis: Program synthesis is the task of learning programs in a domain-speciﬁc language (DSL) that satisfy a given speciﬁcation (Gulwani et al., 2017).
It is closely related to symbolic regression, where the DSL can be considered as a grammar deﬁning the space of expressions.
Some recent works use neural networks for learning programs (Devlin et al., 2017; Balog et al., 2016;
Parisotto et al., 2017; Vijayakumar et al., 2018).
RobustFill
(Devlin et al., 2017) trains an encoderdecoder model that learns to decode programs as a sequence of tokens given a set of input-output examples.
For more complex DSL grammars such as Karel that consists of nested control-ﬂow, an additional grammar mask is used to ensure syntactic validity of the decoded programs (Bunel et al., 2018).
However, these approaches only use examples as speciﬁcation.
Our technique can be useful to guide search for programs where the program space is deﬁned using grammars such as SyGuS with additional semantic constraints other than only examples (Alur et al., 2013; Si et al., 2018).
C ONCLUSION
We present a two-step framework of ﬁrst learning a neural network of the relation between syntactic structure and leading powers and then using that to guide MCTS for efﬁcient search.
This framework is evaluated in the context of symbolic regression and focused on the leading power properties on thousands of desired expressions, a much larger evaluation set than benchmarks considered in existing literature.
We plan to further extend the applicability of this framework to cover other symbolically derivable properties of expressions. Similar modeling ideas could be equally useful in general program synthesis settings, where other properties such as the desired time complexity or maximum control ﬂow nesting could be used as constraints.
A D ATASET
To create an initial dataset, we ﬁrst enumerate all possible parse trees from
[Equation]
within ten production rules.
Then we repeat the following downsampling and augmentation operations for four times to expand the dataset for longer expressions and diverse conditions.
Downsampling Expressions are grouped by their simpliﬁed expressions computed by
SymPy (Meurer et al., 2017).
We select 20 shortest expressions in terms of string length from each group.
These expressions are kept to ensure syntactical diversity and avoid having too long expressions.
Augmentation For each kept expression, ﬁve new expressions are created by randomly replacing one of the 1 or x symbols by (1 /x ) , ( x/ (1 + x )) , ( x/ (1 − x )) , (1 / (1 + x )) , (1 / (1 − x )) , (1 − x ) , (1 + x ) , ( x ∗ x ) , ( x ∗ (1 + x )) , ( x ∗ (1 − x )) .
These ﬁve newly created expressions are added back to the pool of kept expressions to form an expanded set of expressions.
After repeating the above operations for four times, we apply the downsampling step again in the end.
To make the dataset balanced on each condition, we keep 1000 shortest expressions in terms of string length for each condition.
In this way, we efﬁciently create an expanded dataset which is not only balanced on each condition but also contains a large variety of expressions with much longer length than the initial dataset.
Compared to enumerating all possible expressions given the maximum length, the created dataset is much sparser and smaller.
For each pair of integer leading powers satisfying M [ f ] ≤ 4 , 1000 shortest expressions are selected to obtain 41000 expressions in total.
They are randomly split into three sets.
The ﬁrst two are training (28837) and validation (4095) sets.
For the remaining expressions, 50 expressions with unique simpliﬁed expressions are sampled from each condition for M [ f ] ≤ 4 , to form a holdout set with 2050 expressions.
In the same way, we also create a holdout set of 1000 expressions for M [ f ] =and 1200 expressions for M [ f ] = 6 .
These conditions do not exist in training and validation sets.
[Table]
Figure A.1 shows the histogram of lengths of the production rule sequence for expressions in the training set and holdout sets.
Table A.1 shows the minimum, median and maximum of the lengths of the production rule sequence.
The last column, space size, shows the number of possible expressions within the maximum length of the production rule sequence (including those nonterminal expressions).
It is computed recursively as follows.
Let N ∗ ,i denote the number of possible expressions with length ≤ i and whose production rule sequences start with symbol ∗ , where ∗ can be O, S and T .
Then we have
[Equation]
and the initial condition is N S, 0 = N T,
= 1 .
We can obtain N S,i using the recursive formula.
The quantity of interest N O,i = N S,i − 1 given that the ﬁrst production rule is pre-deﬁned as O → S .
Holdout sets M [ f ] = 5 , 6 not only contain expressions of higher leading powers but also of longer length, which
B D ETAILS OF THE B ASELINE M ODELS
We provide more details of the baseline models we proposed to be compared with our NN model.
Using the same notation as in Eq.
(2), the conditional distribution of the next production rule given the partial sequence and the desired condition is denoted by
[Equation]
The baseline models are essentially different ways to approximate the conditional distribution using empirical distributions.
Limited History (LH) ( l )
The conditional distribution is approximated by the empirical conditional distribution given at most the last l production rules of the partial sequence and the desired condition.
We derive the empirical conditional distribution from the training set by ﬁrst ﬁnding all the partial sequences therein that match the given partial sequence and desired condition.
Then we compute the proportion of each production rule that appears as the next production rule of the found partial sequences.
The proportional is therefore the empirical conditional distribution.
To avoid introducing an invalid next production rule, the empirical conditional distribution is multiplied by the production rule mask of valid next production rules, and renormalized.
[Equation]
Full History (FH)
The conditional distribution is approximated by the empirical conditional distribution given the full partial sequence and the desired condition.
The empirical conditional distribution is derived from the training set similarly as the LH model.
[Equation]
Limited History No Condition (LHNC)
( l ) The conditional distribution is approximated by the empirical conditional distribution given at most the last l production rules of the partial sequence only, where the desired condition is ignored.
The empirical conditional distribution is derived from the training set similarly as the LH model.
[Equation]
Full History No Condition (FHNC)
The conditional distribution is approximated by the empirical conditional distribution given the full partial sequence only, where the desired condition is ignored.
The empirical conditional distribution is derived from the training set similarly as the LH model.
p FHNC =
ˆ p ( r t +1 | r 1 , . . .
, r t ) .
C D ETAILS OF A GGREGATING M ETRICS ON D IFFERENT L EVELS
The metrics are aggregated on different levels.
We compute the average of success rates, syntactic novelty rates and semantic novelty rates over all the conditions in the in-sample condition region.
The out-of-sample condition region is not bounded, and hence we consider the region within | P x → 0 [ f ] | ≤ 9 and | P x →∞ [ f ] | ≤ 9 .
Since the average can be arbitrarily small if the boundary is arbitrarily large, instead we compute the total number of success, syntactic novelty and semantic novelty expressions over all the conditions in the out-of-sample region.
The L1-distance is not well deﬁned for an expression which is non-terminal or ∞ .
For both cases, we specify the L1-distance as 18, which is the L1-distance between conditions (0 , 0) and (9 , 9) .
D D IVERSITY OF G ENERATED E XPRESSIONS
A model that generates expressions with a high success rate (i.e., satisfying the desired condition most of the times) but lacking of diversity is problematic.
To demonstrate the diversity of expressions generated by our NN model, we generate 1000 expressions on each desired condition, and compute the cumulative counts of unique expressions and unique simpliﬁed expressions that satisfy the desired condition among the ﬁrst number of expressions of the 1000 generated expressions, respectively.
Figure D.1 shows the cumulative counts on three typical desired conditions ( c (0) , c ( ∞ ) )
= (0 , 0) , (5 , 5) , ( − 2 , − 3) .
We can observe that the counts steadily increase as the number of generated expressions under consideration increases.
Even with 1000 expressions, the counts have not been saturated.
E A DDITIONAL P LOTS OF M ETRICS FOR P RODUCTION R ULE G ENERATING NN AND B ASELINE M ODELS
Due to the space limit of the paper, Figure 4 in the main text only contains the plots of metrics on each condition for LHNC (8) and NN.
Additional plots of all the baseline models in Table 3 are presented in this section.
Figure E.1 contains NN, NNNC and Random.
Figure E.2 contains FH and
FHNC.
Figure E.3 contains LH (2), LH (4), LH (8) and LH (16).
Figure E.4 contains LHNC (2),
LHNC (4), LHNC (8) and LHNC (16).
F S ELECTION S TEP IN M ONTE C ARLO T REE S EARCH
In the selection step, we use a variant of the PUCT algorithm (Silver et al., 2016; Rosin, 2011) for exploration,
[Equation]
where N ( s t , a ) is the number of visits to the current node, (cid:80) b N ( s t , b ) is the number of visits to the parent of the current node, c puct controls the strength of exploration and P ( s t , a ) is the prior probability of action a .
This strategy initially prefers an action with high prior probability and low visits for similar tree node quality Q ( s t , a ) .
G E VOLUTIONARY A LGORITHM
We implemented the conventional symbolic regression approach with EA using DEAP (Fortin et al., 2012), a popular package for symbolic regression research (Quade et al., 2016; Claveria et al., 2016).
We deﬁne a set of required primitives: + , − , ∗ , / , x and 1 .
An expression is represented as a tree where all the primitives are nodes.
We start with a population of 10 individual trees with the maximum tree height set to 50.
The probability of mating two individuals is 0 . 1 , and of mutating an individual is 0 .
(chosen based on a hyperparameter search, see Appendix I).
The limit of number of evaluations for a new offspring is set to 500 so that it is comparable to the number of simulations in MCTS.
H G RAMMAR V ARIATIONAL A UTOENCODER
We would like to point out that the dataset in this paper is more challenging for GVAE than the dataset used in Grammar Variational Autoencoder (GVAE) paper (Kusner et al., 2017), although it is constructed with fewer production rules.
First, the maximum length of production rule sequences in
GVAE paper is 15, while the maximum length is 31 in our training set and 43 in holdout set.
The
RNN decoder usually has difﬁculties in learning longer sequence due to e.g. exposure bias (Bengio et al., 2015).
Second, while our maximum length is more than doubled, our training set only contains 28837 expressions compared to 100000 in GVAE paper.
Samples in our training set are sparser in the syntactic space than those of the GVAE paper.
We trained a GVAE using the the open-sourced code 2 with the following modiﬁcations: 1) using context-free grammar in our paper 2) setting the max length to 100 so it is comparable with MCTS and EA experiments in our paper.
The GVAE paper reported reconstruction accuracy 0.537.
On our validation set, the reconstruction accuracy is 0.102.
During each iteration in Bayesian optimization, 500 data in the training set are randomly selected to model the posterior distribution and the model will suggest 50 new expressions.
These 50 new expressions will be evaluated and concatenated into the training set.
For each symbolic regression task, we ran 5 iterations.
The only difference from the setting in Kusner et al.
(2017) is that instead of averaging over 10 repetitions, we average over 2 repetitions so the total number of evaluation is 2 × 5 × 50 = 500 for searching each expression, which is comparable to the setting of MCTS and
EA.
Due to the restriction on the computational time of Bayesian optimization for GVAE, we evaluate
GVAE on a subset of holdout sets.
30 expressions are randomly selected from each of the holdout sets ( M < = 4 , = 5 , = 6 ) in Section 5 and symbolic regression tasks are performed on theseexpressions.
We report the results of NG-MCTS, GVAE and GVAE+PW (including the error of leading powers to the score function in Bayesian optimization) evaluated on median extrapolation
RMSE and absolute error of leading powers in Table H.1.
GVAE + PW has better ∆ P [ g ] comparing to GVAE.
NG-MCTS signiﬁcantly outperforms GVAE and GVAE + PW on solved percentage and extrapolation error ∆ g ext . .
I C HOICE OF H YPERPARAMETERS
NN is trained with batch size 256 for 10 7 steps.
The initial learning rate is 0 .
It decays exponentially every 10 5 steps with a base of 0.99.
The hyperparameters of MCTS (exploration strength = 50) and EA (the probability of mating two individuals = 0.1, the probability of mutating an individual = 0.5) are selected by hyperparameter searching to maximize the solved percentage in holdout set M [ f ] ≤ 4 with both D train and P x → 0 [ f ] ,
P x →∞ [ f ] provided.
[Table]
J E XAMPLES OF S YMBOLIC R EGRESSION R ESULTS
In this section, we select expressions solved by NG-MCTS but unsolved by EA + PW in Table 1 as examples of symbolic regression results.
Figure J.1, Figure J.2 and Figure J.3 show eight expressions with M [ f ] ≤ 4 , M [ f ] = 5 and M [ f ] = 6 , respectively.
The symbolic expressions, leading powers, interpolation errors and extrapolation errors of these 24 desired expressions f ( x ) , as well as their corresponding best expressions found by NG-MCTS, denoted by g NG − MCTS ( x ) , and by EA + PW, denoted by g EA+PW ( x ) , are listed in Table J.1.
We also select expressions solved by EA + PW but unsolved by NG-MCTS in Table 1 as examples of symbolic regression results.
Figure J.4, Figure J.5 and Figure J.6 show eight expressions with
M [ f ] ≤ 4 , M [ f ] = 5 and M [ f ] = 6 , respectively.
The symbolic expressions, leading powers, interpolation errors and extrapolation errors of these 24 desired expressions f ( x ) , as well as their corresponding best expressions found by NG-MCTS and EA + PW are listed in Table J.2.
K S YMBOLIC R EGRESSION WITH N OISE
In the main text, the training points D train from the desired expression is noise-free.
However, in realistic applications of symbolic regression, measurement noise usually exists.
We add a random
Gaussian noise with standard deviation 0 .
5 to expression evaluations on D train and compute ∆ g int .
and ∆ g ext .
using evaluations without noise.
The results are summarized in Table K.1.
The performance of all the methods is worse than that in the noise-free experiments, but the relative relationship still remains.
NG-MCTS solves the most expressions than all the other methods.
It has the lowest medians of ∆ g ext .
and ∆ P [ g ] , suggesting good generalization in extrapolation even with noise on training points.
L S YNTACTIC N OVELTY E XAMPLES
For ease of presentation, we show syntactic novelties generated by NN that only have one semantically identical expression (i.e., the expression that shares the same simpliﬁed expression) in the training set.
By comparing each syntactic novelty and its semantically identical expression in the training set (shown in Table L.1), we can observe that the model generates some nontrivial syntactic novelties.
For
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
)) / x
[Table]
x
[Table]
[Table]
x ) / x / ( x
[Table]
x )
[Table]
1 / 1 ) / x
[Table]
x /
[Table]
P
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
(((∗ ( 1 + x )) ∗ ( x / 1 )) ∗ x ) ∗ ( x / 1 )) − 1 )))) − ( x / 1 )) ∗ (+ x )) ∗ 1 ) ∗ x ) ∗ ( x / 1 )) ∗ x )) − ( x / x ))
[Table]
F i g J .
1 ( c ) f
( x ) ((+ ( 1 / x ))) /
(( 1 − x ) − x )
[Table]
g N G − M CT
S ( x ) (+ ( 1 / x )) / (− x − x )
[Table]
g E A + P W ( x ) ((( x / 1 ) ∗ 1 ) /
((( 1 − x ) − (( x − x ) + 1 )) ∗ x ))
[Table]
F i g J .
1 ( d ) f
( x ) ((/ x ) + 1 ) / x / ( 1 + (
[Table]
x ))
[Table]
g N G − M CT
S ( x ) (+ ( 1 / x )) / ( x − ( x ∗ x ) + x )
[Table]
g E A + P W ( x ) ( 1 / (( x ∗ x ) / (( 1 / 1 ) + (( x + ((+ 1 ) + ((+ 1 ) − (( 1 ∗ ( 1 + x )) − ( 1 + 1 ))))) /
(( x ∗ x ) + 1 )))))
[Table]
F i g J .
1 ( e ) f
( x ) ((+ x ) ∗ x − 1 ) −
[Table]
g N G − M CT
S ( x ) x + x ∗ x − 1 −
[Table]
g E A + P W ( x ) (((( x / 1 ) − ( 1 ∗ 1 ))
∗ ( x + ( x − ( 1 + 1 )) ))
+ x )
[Table]
F i g J .
( f ) f ( x ) (+ x ) ∗ x + x − ( 1 / x ) / x
[Table]
g N G − M CT
S ( x ) x − 1 / x / x + ( x ∗ ( 1 + x ))
[Table]
g E A + P W ( x ) (( 1 + 1 ) + (((( x / x ) ∗ ( x / 1 )) ∗ (( 1 ∗ (( 1 / (( 1 ∗ 1 ) − (+ x )))
∗ (((− x ) / ( x ∗)) ∗ ( 1 / x ))) )
+ (( x
[Table]
1 ) + ( x − 1 ))))
− (− x )))
[Table]
F i g J .
1 ( g ) f
( x ) (+ x ) ∗ x ∗ ( ( 1 − x ) + 1 )
[Table]
−
[Table]
CT
S ( x ) (+ ( 1 − x )) ∗ ( 1 + x ) ∗ x
[Table]
g E A + P W ( x ) ( x − ( x ∗ ( x ∗ ((( x / ( 1 + x )) ∗ (( x − 1 ) / ( x / 1 )))
∗ x ))) )
[Table]
F i g J .
1 ( h ) f
( x ) ((/ x ) + x ) / x − x ∗ x + x
[Table]
g N G − M CT
S ( x ) 1 + x ∗ (− x ) + ( 1 / x ) / x
[Table]
g E A + P W ( x ) ((((( 1 + 1 ) / x ) / x ) − x ) + ( 1 / (∗ x )))
[Table] [Table]
F i g J .
2 ( a ) f
( x ) ((− x ) + ( x / ( 1 − x ))) / ( x ∗ x ) / x
[Table]
g N G − M CT
S ( x ) (+ (( x ∗ x ) / ( 1 − x ))) /
x / x / 1 / x
[Table]
g E A + P W ( x ) (((((((∗ (( x / 1 ) / (− x ))) − (∗ (( 1 ∗ 1 ) + (( x − x ) ∗ ( 1 + 1 ))))) /
( 1 + 1 ))
− ( 1 ∗ ( 1 + x ))) / 1 ) / x ) / ( x / (( x / x ) /
(( 1 ∗ x ) /
( (( x − x ) + x ) / x )))))
[Table]
i g J .
2 ( b ) f
( x ) ((/ (( 1 + x ) + x )) − x ) / x / ( x ∗ x )
[Table]
g N G
[Table]
M CT
S ( x ) (((/ (( 1 + x ) + x )) − x ) / x ) / x / x
[Table]
E A
[Table]
P W ( x ) (((((( x − x ) ∗ ( 1 − 1 )) − ( 1 + x )) /
( 1 + 1 )) /
(( x ∗ x ) ∗ (( 1 ∗) / 1 ))) / (( x / x ) ∗ x ))
[Table]
F i g J .
2 ( c ) f ( x ) ((/ ( 1 + x ))
+ x ) /
(( 1 / x ) + x ) − x
[Table]
g N G − M CT
S ( x ) ( x ∗ ( 1 − x )) / ( x − ( 1 / x ) / ( x ∗ x ))
[Table]
g E A + P W ( x ) ((( x / 1 ) − x ) − (((( x + 1 ) − ( 1 ∗ 1 )) ∗ ( (( x − ( 1 ∗ 1 )) ∗ x ) / ( 1 + ((( 1 ∗ x ) − ( 1 / x )) ∗ 1 )))) /
( 1 + ((( x − 1 ) − ( 1 / x )) ∗ 1 ))))
[Table]
F i g J .
2 ( d ) f
( x ) ((/ ( 1 + x )) / ( 1 − x )) /
( x ∗ ( x ∗ x ) − 1 )
[Table]
g N G − M CT
S ( x ) (/ ( 1 − ( x ∗ ( x ∗ x )))) /
( x ∗ x − 1 )
[Table]
g E A + P W ( x ) (( 1 − (((( ( x + 1 )
[Table]
( x − x )) − x ) + 1 ) + 1 ))
/ ( x ∗ x ))
[Table]
F i g J .
2 ( e ) f
( x ) (((+ x ) / ( 1 − x )) − 1 ) ∗ x ∗ x
[Table]
g
[Table]
G − M CT
S ( x ) ( x + x ) ∗ x ∗ ( x ∗ ( 1 / ( 1 − x )))
[Table]
g E A + P W ( x ) ((( 1 − ( x + 1 )) − (( ( 1 + 1 )
+ ((( 1 + ( x / x )) ∗ 1 )
+ (( 1 + 1 ) + x ))) /
( 1 / x ) )) / 1 )
[Table]
F i g J .
( f ) f ( x ) (− ( 1 / x ) / x ) − ( x ∗ ( x ∗ x )) -
[Table]
g N G − M CT
S ( x ) 1 − 1 / ( x ∗ x ) − ( x ∗ x ) ∗ x
[Table]
g E A + P W ( x ) (( 1 − x ) ∗ ((((((((( x − x ) − ( 1 / x )) − 1 )
+ ( x + x ))
− 1 ) / x ) − 1 )
+ ( x + x )) + ( x + x )))
[Table]
F i g J . 2 ( g ) f ( x ) ( x ∗ x ) ∗ x − 1 / ( x ∗ x ) + ( 1 + x )
[Table]
g N G − M CT
S ( x ) x ∗ ( x ∗ x ) + x − (( 1 / x ) / x ) +
[Table]
g E A + P W ( x ) (( x ∗ ( x + ( x ∗ (( x − 1 ) + 1 ))))
− ((((( x ∗ ((( 1 ∗ (((( x / x ) − ( 1 − x )) / 1 ) + ( x − (( 1 / 1 ) / ( x / 1 )))))
− x ) + (( x − ( 1 + 1 ))
+ x )))
∗ (((( x / x ) − 1 ) / 1 ) + ( x − 1 ))) − x ) + ( 1 + ((( x ∗ 1 ) ∗ ( x − x )) ∗ ( x + ( x ∗ (( x − 1 ) + 1 )))))) /
( x ∗ x ))) -
[Table]
F i g J .
2 ( h ) f
( x ) ( x ∗ x ) ∗ x − ( 1 + x ) / ( x ∗ x ) −-
[Table]
g N G − M CT
S ( x ) x ∗ ( x ∗ x ) − ( 1 + ( 1 / x )) / x −
[Table]
[Table]
g E A + P W ( x ) (( x / ( 1 ∗ ( 1 − x )))
+ ( x + (( x ∗ ( x − (( 1 + x ) / (( x ∗ x ) ∗ x ))))
+ (((( x ∗ x ) / ( 1 + 1 ))
+ x ) + x )))) -
[Table]
F i g J .
3 ( a ) f
( x ) ((/ ( 1 − x )) / ( 1 − x )) /
(( 1 / x ) / ( x ∗ x ) − x )
[Table]
– g N G − M CT
S ( x ) (( x / ( 1 − x )) / ( 1 − x )) /
( 1 / ( x ∗ x ) − ( x ∗ x ))
[Table]
g E A + P W ( x ) ( x / (((( x + x ) ∗ ( 1 ∗ 1 )) / ( x / (∗ 1 ))) − ( ( x ∗ x ) ∗ ((( x − 1 ) ∗ x ) + x ))))
[Table] [Table]
F i g J .
3 ( b ) f
( x ) ((/ x ) + ( 1 − x )) /
x / x / ( x ∗ x ) − x
[Table]
[Table] [Table]
g N G − M CT
S ( x ) (/ x ) / x / x / ( x / (( 1 / x ) + ( 1 − x ))) − x
[Table]
g E A + P W ( x ) (( x − ((( 1 / ( x ∗ x )) / (( x ∗ ((( x + ( x / (( 1 ∗ x ) / ( x / x ))))
+ (( 1 − x ) + ( x ∗ x )))
∗ ( x − 1 ))) /
( 1 / x ))) /
(( x − x ) − ( x + x )))) − ( x + ( x / ( 1 /))))
[Table]
F i g J .
3 ( c ) f
( x ) ((/ x ) + ( 1 / (( x ∗ x ) + x )) )
x / x
[Table]
[Table]
g N G − M CT
S ( x ) ((/ ( 1 + x ))
+ 1 / 1 ) /
(( x ∗ x ) ∗ x )
[Table]
g E A + P W ( x ) (( 1 / ((( x / 1 ) ∗ ( x ∗ 1 ) ) − ( x − x ))) / x )
[Table] [Table]
F i g J .
3 ( d ) f
( x ) (((/ x ) / x ) / x ) / ( x + ( 1 / x ) + x )
[Table]
[Table]
g N G − M CT
S ( x ) ((/ x ) / ( x ∗ x )) /
( x + x + ( 1 / x ))
[Table]
[Table]
g E A + P W ( x ) (((( 1 − 1 ) / 1 ) − ( x / (( x + x ) ∗ (( x + 1 ) ∗ ( x / 1 ))))) /
(( 1 / (((((∗ 1 ) + ( x /)) /
( 1 ∗ x )) ∗ 1 ) ∗ (( 1 + x ) ∗ ( 1 ∗ 1 ) )))
+ ( x ∗ ( x / ((( 1 ∗ x ) / x ) /
((( x / 1 ) − ( 1 + x )) / 1 ))))))
[Table]
F i g J .
3 ( e ) f
( x ) ((+ x ) / x ) / ( x ∗ x ) + 1 + ( x ∗ x ) ∗ x
[Table]
g N G − M CT
S ( x ) 1 + x ∗ ( x ∗ x ) + (( 1 + x ) / ( x ∗ x )) / x
[Table]
g E A + P W ( x ) ((((( 1 ∗ x ) + ( 1 + ( x − ( 1 + 1 ))))
∗ x )
+ (((( x / x ) / x ) − ((((((/ x ) / x ) − 1 ) + x ) / 1 ) / ( 1 ∗ x ))) + (( x ∗ x ) + ( x + ( x − (( 1 / x ) + ( 1 + x ))))))) / 1 )
[Table]
F i g J .
( f ) f ( x ) ( x / ( 1 − ( 1 / x ))) ∗ ( x + ( x ∗ x ))
[Table]
g N G − M CT
S ( x ) ( x + x ∗ x ) ∗ ( x / ( 1 − ( 1 / x )))
[Table]
g E A + P W ( x ) ((( 1 + 1 ) + (((( 1 ∗ ((( x + ( x / x ))
+ x ) /
(∗ x )))
+ 1 ) ∗ x )
+ (( x − x ) /
( 1 / ((/ 1 ) ∗ ( 1 ∗ 1 )))))) ∗ ( x + ( x / x )))
[Table]
F i g J .
3 ( g ) f
( x ) 1 + 1 / ( x ∗ x ) − (( x ∗ x ) ∗ x ) ∗ x
[Table]
g N G − M CT
S ( x ) 1 + ( 1 / x ) / x − ( x ∗ ( x ∗ x )) ∗ x
[Table]
g E A + P W ( x ) ((( 1 / ( x / 1 )) − x ) ∗ (((( 1 / x ) + ( x − 1 ))
+ x ) ∗ (( x + ((( 1 ∗ x ) − x ) − (− x )))
+ (( x − x ) ∗ (((/ x ) + ( x + x )) / x )))))
[Table]
F i g J .
3 ( h ) f
( x ) ( x ) ∗ ( x / ( 1 − x )) ∗ ( x / (− x )) ∗ x
[Table]
[Table]
g N G − M CT
S ( x ) ( x ∗ ( x ∗ x )) /
( x + ( 1 / ( x / ( 1 − x ))) − 1 )
[Table]
g E A + P W ( x ) (((( 1 / 1 ) + ( 1 / 1 )) ∗ ((( x / 1 ) ∗ ( x − 1 ))
+ x ))
+ ((( x ∗ 1 ) / ( x − 1 )) ∗ ((( x / ( x − 1 ))
∗ x ) / 1 )))
2 .46 .humans to propose such syntactic novelties, they would need to know and apply the corresponding nontrivial mathematical rules (shown in the ﬁrst column of Table L.1) to derive the expressions from those already known in the training set.
On the contrary, NN generates the syntactic novelties such as
1 / ( A/B ) =
B/A , A/ ( B ∗ C )
=
A/B/C , etc, without being explicitly taught these mathematical rules.
[Table]
[Table] [Table]
[Table]
x ) ∗ (∗ 1 )) ∗ x ))) / 1 ) − ( 1 / x ))
− x ) − ( 1 /
[Table] [Table]
(( x / 1 ) / x )) − ( x /
[Table]
x ∗ 1 ))))
[Table]
f
[Table] [Table]
[Table] [Table]
[Table] [Table]
( x / (
[Table]
x ) )
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
x ))
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
x )
[Table]
x ) / (
[Table]
[Table] [Table]
x ))
[Table]
g E A + P W ( x ) (((((((( x − 1 ) + 1 ) ∗ 1 )
+ ( 1 / (∗ 1 ))) ∗ 1 )
+ ( x / 1 ))
∗ ( x ∗ ((( x / x )
+ (( 1 / ( x − 1 )) / ( 1 ∗ 1 ))) ∗ 1 ))) / 1 )
[Table]
) f ( x ) ((+ x ) ∗ x ) ∗ x
[Table]
x
[Table] [Table]
x
[Table]
– – g N G − M CT
S ( x ) (
[Table]
x ) ∗ x
[Table]
x ) ∗
[Table]
) ((((((( x ∗ x )
[Table]
((( x
[Table]
[Table] [Table]
∗ 1 ))
[Table] [Table]
[Table] [Table]
[Table] [Table]
1 ) ∗ (
[Table] [Table]
[Table] [Table]
∗ 1 ))
[Table] [Table]
[Table] [Table]
[Table] [Table]
∗
[Table] [Table]
−
[Table]
CT
S ( x ) ( x + x ) ∗ ( x ∗ x )
[Table]
x
[Table]
P W
[Table]
) ((((( x ∗ ( x / 1 )) − ( 1 ∗ 1 ))
+ ( x ∗ x ))
[Table]
x ) ∗ ( x / 1 ))
[Table]
) f ( x ) (− x ) ∗ x ∗
[Table]
− x − (+ x )
[Table]
N
[Table]
−
[Table]
CT
S ( x )
[Table]
( x − (
[Table]
∗ x ) + ( 1 /
[Table]
)))) / x )))) / x ) /
[Table]
[Table] [Table]
[Table]
) f ( x ) (( x / (
[Table]
x )) −
[Table]
x )) /
x / (
[Table]
∗ x )
[Table]
N
[Table]
−
[Table]
S
[Table] [Table]
[Table]
∗
[Table]
x /
[Table] [Table]
[Table] [Table]
) (((( x − 1 ) /
[Table]
[Table] [Table]
+
[Table] [Table]
[Table]
∗
[Table]
+
[Table] [Table]
[Table]
+ (
[Table] [Table]
[Table]
x ) /
[Table]
∗
[Table] [Table]
−
[Table]
S
[Table] [Table]
[Table]
1 + x ))
[Table] [Table]
[Table] [Table]
[Table] [Table]
) (((((( 1 / x ) ∗ ((
[Table]
−
[Table]
) + ( 1 / x )))
∗ (( 1 / x ) ∗ x )) /
( x ∗))
[Table]
1 ) / x )
[Table]
[Table] [Table]
[Table]
x ) ∗ ( x ∗
[Table]
1 + x ) ) ∗
[Table]
N
[Table]
−
[Table]
S
[Table] [Table]
∗
[Table]
∗
[Table]
∗ x )
[Table]
∗ x )
[Table]
A
[Table]
P W ( x ) (( 1 − ( x ∗ (( x ∗ x ) − ( 1 − x )))) ∗ x )
[Table]
(f) f ( x ) x − x ∗ (+ x ) ∗ ( 1 + ( x ∗ x )) − x
[Table]
– – g N G − M CT
S ( x ) x − x ∗ ( x ∗ x + x ) ∗ x − ( x ∗ x )
[Table]
g E A + P W ( x ) ( x ∗ (((− 1 ) − ( 1 ∗ 1 )) − (( x ∗ 1 ) ∗ (((((∗ 1 ) ∗ 1 ) + ( x ∗ ( x / 1 ))) ∗
[Table]
+ (
[Table] [Table]
) f ( x ) x ∗ (( x / (− x ))
+ x ) ∗ ( 1 + x )
[Table]
−
[Table]
S ( x ) ( x / ( 1 − x )) − x ∗ x ∗ (− x ) /
[Table]
E A + P W ( x ) ((((( 1 − x ) + ( 1 + x ) ) − x ) ∗ ( x + ( x ∗ x ))) /
((( 1 / x ) − (((( 1 /) + (( x / x ) + ( x / x ))) ∗ ( x − x )) ∗ 1 )) − 1 ))
[Table]
( h ) f
( x ) ((/ ( 1 − ( 1 / x )))
∗ x ) ∗ ( x + x + x )
[Table]
g N G − M CT
S ( x ) (/ ( 1 − ( 1 / x ))) ∗ ( 1 + x ) ∗ ( x ∗ x )
[Table]
g E A + P W ( x ) (( x + ((∗ x ) + (( x + (( x ∗ 1 ) ∗ (( 1 / x ) ∗ (+ 1 )))) /
(( x − ( x / x )) / x ) )))
∗ x )
[Table]
( a ) f ( x ) (/ (( x ∗ x ) + ( x ∗ x ) )) /
( x + x + x )
[Table]
g N G − M CT
S ( x ) (− ( 1 / x )) /
( x ∗ (( 1 + x ) + x ))
/ x
[Table]
g E A + P W ( x ) (((( 1 / ((( 1 + x ) − ( x − 1 )) + 1 )) /
( x + x )) /
x ) / x )
[Table]
[Table]
( b ) f ( x ) ( x ∗ x ) ∗ x − 1 + ((( 1 / x ) / x ) / x ) −
[Table]
g N G − M CT
S ( x ) x ∗ ( x ∗ x ) − (( 1 / x ) / x ) / x − x
[Table]
g E A + P W ( x ) (((( 1 / (( x ∗ x ) ∗ ( x / 1 ) )) − 1 )
+ ( 1 − ( 1 + 1 )))
+ ((∗ (( 1 ∗ x ) ∗ ( x ∗ 1 )))
∗ x ))
[Table]
( c ) f
( x ) (/ ( 1 + x )) − ( 1 / x ) /
( x ∗ x ) / ( x ∗ x ) + x
[Table]
g N G − M CT
S ( x ) x − (( 1 / x ) − x ) / x / ( x ∗ x ) / x
[Table]
g E A + P W ( x ) ( x − ((((/ ( x + ( x ∗ ((( x ∗ x ) / ( 1 ∗ ( 1 / x ))) − 1 ))))
− 1 )
+ ( 1 / (( x ∗ 1 ) + ( 1 ∗ ( x / x ))))) / x ))
[Table]
) f ( x ) ((/ (( x ∗ x ) ∗ x )) / x ) / ( x + x ) + x
[Table]
g N G − M CT
S ( x ) ((/ x ) / ( 1 + x )) /
( x ∗ x ) / x / x +
[Table]
g E A + P W ( x ) ((((( x / x ) / (( x ∗ ((/ 1 ) ∗ ( x / 1 ) ))
∗ x )) /
( x / ((( x / (( 1 / 1 ) ∗ ( x / 1 ))) /
( 1 + 1 ))
− (( x / 1 ) − x ))))
∗ ((/ x ) − ( 1 − 1 )))
+ ( x / 1 ))
[Table]
f ( x ) x ∗ ( 1 + (+ x )) ∗ x ∗ x
+ ( x ∗ x )
[Table]
g N G − M CT
S ( x ) ( x + x ∗ (( x + x ∗ x ) + x ) ∗ x )
[Table]
E A + P W ( x ) (( x − (( x ∗ x ) − ( x ∗ (( x + (( x ∗ x )
+ ((( x ∗ x ) ∗ x ) + (((((( x + 1 ) − (( 1 + 1 ) + 1 ))
+ 1 ) − 1 ) ∗ x ) − (( 1 ∗ 1 ) − ( 1 ∗ x ))))) )
+ (( x ∗ 1 ) + x ))))) / 1 )
[Table]
(f) f ( x ) 1 / x / x − x ∗ ( x ∗ ((+ x ) + ( x ∗ x )))
[Table]
[Table]
– – g N G − M CT
S ( x ) 1 / ( x ∗ x ) − ( x ∗ x ) ∗ x ∗ ( 1 + x )
[Table]
g E A + P W ( x ) ((( 1 + x ) / ( x ∗ (( x ∗ x ) + ( x /)))) − (( x ∗ (( x ∗ x ) + ( 1 − ( x / x ))))
+ ((( x ∗ x ) − ( 1 / (( x − 1 ) − x ) ))
∗ ( x ∗ x )))) -
[Table]
( g ) f
( x ) (( x ∗ x ) ∗ (( x ∗ x ) − x )) /
( x + x + 1 )  – – g N G − M CT
S ( x ) x ∗ x / (((+ x ) / x ) + ( 1 / ( 1 + x )))
[Table]
g E A + P W ( x ) ((((( x − 1 ) ∗ x ) /
( x + (( x ∗ 1 ) + ( 1 + (− 1 )))))
∗ ( ( 1 ∗ x ) ∗ ( x / 1 ))) / 1 )
[Table]
( h ) f
( x ) ( x − ( x ∗ x )) ∗ x ∗ ( 1 − x )  – – g N G − M CT
S ( x ) x ∗ 1 / (( 1 / x ) + x ) + x ∗ ( x ∗ x )
[Table]
g E A + P W ( x ) (( x − 1 ) ∗ ( x ∗ (((( x / 1 ) ∗ ( x − 1 )) / 1 )
+ (( 1 ∗ ( 1 − x ))
+ ( x − 1 )))))
[Table] [Table]
set.
M ATHEMATICAL R ULE
S YNTACTIC N OVELTY S
EM -I DENTICAL E XPRESSION 1 / ( A/B ) =
B/A 1 + 1 / (1 + ( x/ (1 + x )))
+ (1 + x ) /
((1 + x ) + x )
A/ ( B + B ) =
A/B/1 − x/ ((1 + x ) + (1 + x )) 1 − ( x/ (1 + x )) / (1 + 1)
A + B = B + A x/ (( x/ (1 + x ))
+ x + x ) x/ ( x + ( x/ (1 + x ))
+ x )
A/ ( B ∗ C ) =
A/B/C (1 − ( x/ (1 − x ))) /
( x + x ∗ x ) (1 − ( x/ (1 − x ))) /
( x + 1) /x
A/C + B/C =
( A + B ) /C
(1 + (1 /x )) / ( x − ( x ∗ x ) − 1) ((1 + x ) /x ) / ( x − 1 − ( x ∗ x ))