Original: Privileged Deep Symbolic Regression
Revised: Privileged Deep Symbolic Regression

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Symbolic regression is the process of finding an analytical expression that fitsexperimental data with the least amount of operators, variables and constantssymbols.
Revised: Symbolic regression is the process of finding an analytical expression that fits experimental data with the least amount of operators, variables and constants symbols.

Original: Given the huge combinatorial space of possible expressions, evolutionaryalgorithms struggle to find expressions that meet these criteria in a reasonableamount of time.
Revised: Given the huge combinatorial space of possible expressions, evolutionary algorithms struggle to find expressions that meet these criteria in a reasonable amount of time.

Original: To efficiently reduce the search space, neural symbolic regressionalgorithms have recently been proposed for their ability to identify patterns in thedata and output analytical expressions in a single forward-pass.
Revised: To efficiently reduce the search space, neural symbolic regression algorithms have recently been proposed for their ability to identify patterns in the data and output analytical expressions in a single forward-pass.

Original: However, thesenew approaches to symbolic regression do not allow for the direct encoding of userdefined prior knowledge, a common scenario in natural sciences and engineering.
Revised: However, these new approaches to symbolic regression do not allow for the direct encoding of userdefined prior knowledge, a common scenario in natural sciences and engineering.

Original: Inthis work, we propose the first neural symbolic regression method that allows usersto explicitly bias predictions towards expressions that satisfy a set of assumptions onthe expected structure of the ground-truth expression.
Revised: In this work, we propose the first neural symbolic regression method that allows users to explicitly bias predictions towards expressions that satisfy a set of assumptions on the expected structure of the ground-truth expression.

Original: Our experiments show thatour conditioned deep learning model outperforms its unconditioned counterparts interms of accuracy while achieving control over the predicted expression structure.
Revised: Our experiments show that our conditioned deep learning model outperforms its unconditioned counterpart in terms of accuracy while achieving control over the predicted expression structure.

Original:  Introduction1.1 Symbolic regression and problem-specific prior knowledge
Revised:  Introduction 1.1 Symbolic regression and problem-specific prior knowledge

Original: Symbolic Regression (SR) is a technique that searches  the space of analytical expressions E to fitexperimental data while trading off between minimizing expression complexity and maximizingaccuracy.
Revised: Symbolic Regression (SR) is a technique that searches over the space of analytical expressions E to fit experimental data while trading off between minimizing expression complexity and maximizing accuracy.

Original: As opposed to over-parametrized approaches, e.g. decision trees and neural networks,
Revised: As opposed to over-parametrized approaches, e g. decision trees and neural networks,

Original: SR can produce human-readable expressions that are particularly useful in Natural Sciences, e.g.materials sciences Wang et al. ; Kabliman et al. ; Ma et al.
Revised: SR can produce human-readable expressions that are particularly useful in Natural Sciences, e g. materials sciences (Wang et al., 2019; Kabliman et al., 2021; Ma et al., 2022), physics (Schmidt and

Original: physics Schmidtand Lipson (2009); Vaddireddy et al. (2020); Sun et al. ; Cranmer et al. Hernandezet al.
Revised: Lipson, 2009; Vaddireddy et al., 2020; Sun et al., 2022; Cranmer et al., 2020; Hernandez et al., 2019;

Original: (2019); Udrescu and Tegmark (2020), to interpret physical phenomena.
Revised: Udrescu and Tegmark, 2020), to interpret physical phenomena.

Original: Rather than perfectlyfitting the data, which are often noisy because of inevitable measurement errors, SR aims at acquiringmeaningful insights on the inner mechanisms governing physical systems via compact, or equivalentlylow-complexity, expressions.
Revised: Rather than perfectly fitting the data, which are often noisy because of inevitable measurement errors, SR aims at acquiring meaningful insights on the inner mechanisms governing physical systems via compact or equivalently lowcomplexity, expressions.

Original: La Cava et al.
Revised: La Cava et al.

Original: (2021) recent benchmarking effort has shown thatsymbolic regressors can additionally outperform their over-parametrized counterparts on a set ofblack-box and synthetic datasets.
Revised: (2021) recent benchmarking effort has shown that symbolic regressors can additionally outperform their over-parametrized counterparts on a set of black-box and synthetic datasets.

Original: In Natural Sciences, researchers generally have some insights on the datasets they study, usinganalogies to problems where a solution is known.
Revised: In Natural Sciences, researchers generally have some insights on the datasets they study, using analogies to problems where a solution is known.

Original: For instance, when studying a system with differentelements interacting with each other, a good assumption is that the laws of the system involveterms similar to Gravitational/Coulomb forces, e.g. ∝ 1 r 2 with r = (cid:112) x 2 + y 2 .
Revised: For instance, when studying a system with different elements interacting with each other, a good assumption is that the laws of the system involve terms similar to Gravitational/Coulomb forces, e.g. ∝ 1 r 2 with r = (cid:112) x 2 + y 2 .

Original: Access to a partof the underlying ground-truthsystem equation is also a common assumption done in the system identification literature where the physical laws are known up to a few parameters Brunton et al.(2016); Kaheman et al.
Revised: Access to a part of the underlying ground-truth equation is also a common assumption done in the system identification literature where the physical laws are known up to a few parameters (Brunton et al., 2016; Kaheman et al., 2020).

Original: In our work, we will name hypotheses the assumptions formulated by the SR practitioner, potentiallyincomplete or partially incorrect, about the underlying ground-truth expression that can be leveragedin any form to restrict the search space.
Revised: In our work, we will name hypotheses the assumptions formulated by the SR practitioner, potentially incomplete or partially incorrect, about the underlying ground-truth expression that can be leveraged in any form to restrict the search space.

Original: If an hypothesis is true, we will name it privileged information .
Revised: If an hypothesis is true, we will name it privileged information .

Original: 1.2 Related work
Revised: 1.2 Related work

Original: Searching for a satisfactory analytical expression is a difficult optimization problem, mostly tackledusing genetic programming (GP) algorithms.
Revised: Searching for a satisfactory analytical expression is a difficult optimization problem, mostly tackled using genetic programming (GP) algorithms.

Original: They work by i) defining a class of programs, represented in SR as tree structures where nodes are unary, e.g. cos , exp or binary operators add , div andleaves are variables and constants and ii) evolving a population of analytical expressions by the meansof selection, mutation and crossovers.
Revised: They work by i) defining a class of programs, represented in SR as tree structures where nodes are unary, e.g. cos , exp or binary operators add , div and leaves are variables and constants and ii) evolving a population of analytical expressions by means of selection, mutation and crossovers.

Original: Beinga greedy search approach, GPs are prone to falling inlocal minima, and extensive exploration leads to relatively large run-times.
Revised: Being greedy search approaches, GP algorithms are prone to fall in local minima, and extensive exploration leads to relatively large run-times.

Original: In practice with timelimits, e.g. 24 hours in La Cava et al. (2021), the most accurate GP methods provide expressions withoverly large complexity thus preventing deriving physical insights; on the Feynman datasets Udrescuand Tegmark (2020), whose functions have averaged complexity 20 as defined in La Cava et al.(2021), the current state-of-the-art Burlacu et al. (2020) predicts functions with averaged complexity100 .
Revised: In practice with time limits, e.g. 24 hours in (La Cava et al., 2021), the most accurate GP methods provide expressions with overly large complexity, thus preventing deriving physical insights; on the Feynman datasets (Udrescu and Tegmark, 2020), whose functions have averaged complexity 20 as defined in (La Cava et al., 2021), the current state-of-the-art (Burlacu et al., 2020) predicts functions with averaged complexity 100 .

Original: Up to our knowledge, injection of prior information in GPs can only be done by filtering duringselection, e.g. using properties like function positivity or convexity Kronberger et al. ; Haideret al. (2022).
Revised: Up to our knowledge, injection of prior information in GPs can only be done by filtering during selection, e.g. using properties like function positivity or convexity (Kronberger et al., 2022; Haider et al., 2022).

Original: This strategy is greedy in essence, therefore leading to premature convergence to localminima expressions.
Revised: This strategy is greedy in essence, therefore leading to premature convergence to local minima expressions.

Original: Other forms of high level prior information available to the user, e.g. complexityof the expected expression, can hardly be incorportated in GPs.
Revised: Other forms of high-level prior information available to the user, e.g. complexity of the expected expression, can hardly be incorporated in GP.

Original: GPs recently adopted neural networksfor their ability to detect qualitative patterns in the data to reduce search space Petersen et al.
Revised: GP recently adopted neural networks for their ability to detect qualitative patterns in the data to reduce search space (Petersen et al., 2019;

Original:  Our contribution draws inspiration from the works of Udrescu and Tegmark (2020); Udrescu et al.(2020) where  search is restricted to expressions characterized by some specific properties, likecompositionality, additivity and generalized symmetry.
Revised: Mundhenk et al., 2021). Our contribution draws inspiration from the works of Udrescu and Tegmark (2020) and Udrescu et al. (2020) where the search is restricted to expressions characterized by some specific properties, like compositionality, additivity and generalized symmetry.

Original: Making use of these properties effectivelysimplify the task of SR by exploiting the modularity of the resulting expression trees.
Revised: Making use of these properties effectively simplifies the task of SR by exploiting the modularity of the resulting expression trees.

Original: Despite beingsimilar in spirit to our work, their method requires fitting a new neural network on each new inputdataset and then probing the trained network in search of the aforementioned properties, a processresulting in an inevitably slow approach.
Revised: Despite being similar in spirit to our work, their method requires fitting a new neural network on each new input dataset and then probing the trained network in search of the aforementioned properties, a process resulting in an inevitably slow approach.

Original: Inspired by recent advances in language models, a line of work named neural symbolic regression(NSR), tackles SR as a natural language processing task Biggio et al.
Revised: Inspired by recent advances in language models, a line of work named neural symbolic regression (NSR), tackles SR as a natural language processing task (Biggio et al., 2020, 2021;

Original: (2020, 2021); Valipour et al.(2021); d’Ascoli et al. ; Kamienny et al. ; Vastl et al. ; Li et al.
Revised: Valipour et al., 2021; d’Ascoli et al., 2022; Kamienny et al., 2022; Vastl et al., 2022; Li et al., 2022).

Original: First, largesynthetic datasets are generated by i) sampling expressions from a prior distribution p θ ( E ) where θ isa parametrization induced by Lample and Charton (2019) generator, ii) evaluating these expressionson a set of points x ∈ R d where d is the feature dimension, e.g. sampled from a uniform distribution.
Revised: First, large synthetic datasets are generated by i) sampling expressions from a prior distribution p θ ( E ) where θ is a parametrization induced by Lample and Charton (2019) generator, ii) evaluating these expressions on a set of points x ∈ R d where d is the feature dimension, e.g. sampled from a uniform distribution.

Original: Secondly, a generative model g ϕ ( E|D ) , practically a Transformer Vaswani et al.
Revised: Secondly, a generative model g ϕ ( E|D ) , practically a Transformer Vaswani et al.

Original: (2017) parametrizedby weights ϕ , that is conditioned on input points D
Revised: (2017) parametrized by weights ϕ , that is conditioned on input points D

Original: = ( x , y ) , is trained on the task of next-tokenprediction with target the Polish notation of the expression.
Revised: = ( x , y ) , is trained on the task of next-token prediction with target the Polish notation of the expression.

Original: NSR predicts expressions that shareproperties of their implicitly biased synthetic generator p θ ( E ) .
Revised: NSR predicts expressions that share properties of their implicitly biased synthetic generator p θ ( E ) .

Original: Control over the shape of the predictedfunctions, e.g complexity or sub-expression terms, boils down to a sound design of the generatorand the pipeline introduced in Lample and Charton (2019) allows only limited degrees of freedom:operators, variables or constants probability and tree depth.
Revised: Control over the shape of the predicted functions, e.g complexity or sub-expression terms, boils down to a sound design of the generator and the pipeline introduced in (Lample and Charton, 2019) allows only limited degrees of freedom: operators, variables or constants’ probability and tree depth.

Original: Similarly to querying a text-to-image generative model Ramesh et al. ; Saharia et al. (2022)with a prompt, the SR practitioner might want to restrict the class of predicted expressions to be insubclass h ( E ) ⊂
Revised: Similarly to querying a text-to-image generative model (Ramesh et al., 2022; Saharia et al., 2022) with a prompt, the SR practitioner might want to restrict the class of predicted expressions to be in subclass h ( E ) ⊂

Original: E using privileged information, e.g. h ( E )  the class of expressions with complexitybetween 10 and 20 , or that have a specific sub-expression like e − √ x 21 + x 22 in them.
Revised: E using privileged information, e.g. h ( E ) can be the class of expressions with complexity between 10 and 20 , or that have a specific sub-expression like e − √ x 21 + x 22 in them.

Original: However, a trained NSR model g ϕ ( E|D ) , can only be adapted to h ( E ) by i) relying on rejection sampling, i.e. samplingfrom g ϕ ( E|D ) and discarding all expressions that do not satisfy its criterion, a time-inefficienttechnique that assumes that the model was trained on a very large class of analytical expressions,e.g. diverse set of operators/complexity, and ii) designing a new generator with desired properties, atedious task, and fine-tuning the model on the produced datasets.
Revised: However, a trained NSR model g ϕ ( E|D ) , can only be adapted to h ( E ) by i) relying on rejection sampling, i.e. sampling from g ϕ ( E|D ) and discarding all expressions that do not satisfy its criterion, a time-inefficient technique that assumes that the model was trained on a very large class of analytical expressions, e.g. diverse set of operators/complexity, and ii) designing a new generator with desired properties, a tedious task, and fine-tuning the model on the produced datasets.

Original: 1.3 Contributions
Revised: 1.3 Contributions

Original: In this work, we propose a solution to the above limitations of NSR algorithms, named Neural
Revised: In this work, we propose a solution to the above limitations of NSR algorithms, named Neural

Original: Symbolic Regression with Hypotheses (NSRwH).
Revised: Symbolic Regression with Hypotheses (NSRwH).

Original: NSRwH efficiently restricts the class of predictedexpressions of NSR models during inference, if provided privileged information D PI as described in1.1, with a simple modification to both the model architecture and the training data generation: withthe training set of expressions from p θ ( E ) , we produce descriptions D PI , e.g. appearing operators orcomplexity, and feed this meta-data into the Transformer model as an extra input, i.e. g ϕ ( S |D , D PI ) .
Revised: NSRwH efficiently restricts the class of predicted expressions of NSR models during inference, if provided privileged information D PI as described in 1.1, with a simple modification to both the model architecture and the training data generation: with the training set of expressions from p θ ( E ) , we produce descriptions D PI , e.g. appearing operators or complexity, and feed this meta-data into the Transformer model as an extra input, i.e. g ϕ ( S |D , D PI ) .

Original: During training, we use a masking strategy to avoid our model to consider sub-classes of functionswhen no privileged information is provided.
Revised: During training, we use a masking strategy to avoid our model considering sub-classes of functions when no privileged information is provided.

Original: We consider expressions with real constants with feature space R 5 dimensions, already a challengingsetting as the induced search space is already extremely large Lample and Charton (2019).
Revised: We consider expressions with real constants with feature space R 5 , already a challenging setting as the induced search space is already extremely large (Lample and Charton, 2019).

Original: To the best of our knowledge, the present is the first work that explores  injection of such priorknowledge into SR.
Revised: To the best of our knowledge, the present is the first work that explores the injection of such prior knowledge into SR.

Original: We insist on the fact that naive adaptation to GPs via filtering in the selection stepis prone to premature convergence to local mimina as mutations and crossovers break satisfaction ofthe properties we consider.
Revised: We insist on the fact that naive adaptation of GP algorithms via filtering in the selection step is prone to premature convergence to local minina as mutations and crossovers break satisfaction of the properties we consider.

Original: We show that our model manifests the following desirable characteristics:
Revised: We show that our model manifests the following desirable characteristics:

Original: Similarly to expression derivation and integration Lample and Charton (2019), we show that Transformers are able to capture non-trivial high-level symbolic expression properties such as complexity, an arguably harder task.
Revised: Similarly to expression derivation and integration (Lample and Charton, 2019), we show that Transformers are able to capture non-trivial high-level symbolic expression properties such as complexity, an arguably harder task.

Original: When conditioned on specific user-determined privileged information on the sought for expression, the model outputs expressions that closely adhere with the provided prior knowledge.
Revised: When conditioned on specific user-determined privileged information on the sought for expression, the model outputs expressions that closely adhere to the provided prior knowledge.

Original: This feature highlights that the proposed model is effectively controllable and its output reflects the expectations of the user in terms of the specified high-level properties.
Revised: This feature highlights that the proposed model is effectively controllable and its output reflects the expectations of the user in terms of the specified high-level properties.

Original: This is in stark contrast with prior work both in the NSR and GP literature, where steering symbolic regressors towards specific properties required either training from scratch or resorting to inefficient post-hoc greed search routines.
Revised: This is in stark contrast with prior work both in the NSR and GP literature, where steering symbolic regressors towards specific properties required either training from scratch or resorting to inefficient post-hoc greedy search routines.

Original: Our conditioned model outperforms its unconditioned counterpart both in the presence or not of output noise, showing that prior knowledge about the sought for expression helps inalleviating the effect of noise, making SR feasible also in this unfavourable and previouslyunaddressed setting.
Revised: Our conditioned model outperforms its unconditioned counterpart both in the presence or not of output noise, showing that prior knowledge about the sought for expression helps in alleviating the effect of noise, making SR feasible also in this unfavourable and previously unaddressed setting.

Original: Overall, the proposed framework is general and provides a simple and general recipe to easilycondition neural symbolic regressors on any sort of prior information the user deems relevant torestrict the search space over mathematical expressions.
Revised: Overall, the proposed framework is general and provides a simple and general recipe to easily condition neural symbolic regressors on any sort of prior information the user deems relevant to restrict the search space over mathematical expressions.

Original:  Method 
Revised:  Method 2.1 Notation and Background

Original: 2.1 Notation and Background

Original: A symbolic regressor is an algorithm that takes as input a dataset D of n features-value pairs( x i , y i ) ∼ R d × R , with d is the feature dimension, and returns a symbolic expression e ∼ E suchthat ∀ ( x i , y i ) ∈ D , e ( x i ) =
Revised: A symbolic regressor is an algorithm that takes as input a dataset D of n features-value pairs ( x i , y i ) ∼ R d × R , with d is the feature dimension, and returns a symbolic expression e ∼ E such that ∀ ( x i , y i ) ∈ D , e ( x i ) =

Original: ˜
Revised: ˜

Original: y
Revised: y

Original: i ≈
Revised: i ≈

Original: y i .
Revised: y i .

Original: SR literature usually considers the R 2 score, defined as below,as the metric to quantify satisfactory fitting levels:
Revised: SR literature usually considers the R 2 score, defined as below, as the metric to quantify satisfactory fitting levels:

Original: [Equation]
Revised: [Equation]

Original: Neural symbolic regression is a class of SR algorithms that learns a distribution model g ϕ ( E | D ) ,parametrized by a neural network with weights ϕ , over symbolic expressions conditioned on aninput dataset D .
Revised: Neural symbolic regression is a class of SR algorithms that learns a distribution model g ϕ ( E | D ) , parametrized by a neural network with weights ϕ , over symbolic expressions conditioned on an input dataset D .

Original: In this work, we introduce NSRwH, a new subclass of neural symbolic regressors,that allow for conditioning their predictions with user-specified prior knowledge about the output expression.
Revised: In this work, we introduce NSRwH, a new subclass of neural symbolic regressors, that allows for conditioning their predictions with user-specified prior knowledge about the output expression.

Original: More concretely, given a set of privileged information D PI , NSRwH approaches aretrained to model the conditional distribution g ϕ ( E | D , D PI ) .
Revised: More concretely, given a set of privileged information D PI , NSRwH approaches are trained to model the conditional distribution g ϕ ( E | D , D PI ) .

Original: 2.2 Data generation
Revised: 2.2 Data generation

Original: In our framework, a synthetic training sample is defined as a tuple ( e, D , D PI ) where each element isproduced as explained below.
Revised: In our framework, a synthetic training sample is defined as a tuple ( e, D , D PI ) where each element is produced as explained below.

Original: Generating expressions.
Revised: Generating expressions.

Original: As in other NSR works Biggio et al. ; Kamienny et al. (2022),we sample analytical expressions e using the strategy introduced by Lample and Charton (2019):random unary-binary trees with depth between 1 and 5 are generated, then internal nodes are assignedeither unary or binary operators described in Table 3 in the Appendix B.1 according to their arity, andleaves are assigned variables { x d } d ≤ 5 .
Revised: As in other NSR works (Biggio et al., 2021; Kamienny et al., 2022), we sample analytical expressions e using the strategy introduced by Lample and Charton (2019): random unary-binary trees with depth between 1 and 5 are generated, then internal nodes are assigned either unary or binary operators described in Table 3 in the Appendix B.1 according to their arity, and leaves are assigned variables { x d } d ≤ 5 .

Original: Generating D .
Revised: Generating D .

Original: For each expression from e , we sample a support of n points x i ∈ R d whereeach coordinate is sampled independently from others within uniform distribution U whose boundsare sampled in [ − 10 , 10] as in Biggio et al.
Revised: For each expression from e , we sample a support of n points x i ∈ R d where each coordinate is sampled independently from others within uniform distribution U whose bounds are sampled in [ − 10 , 10] as in (Biggio et al., 2021).

Original: Next,  expression value y i is obtained via theevaluation of the expression e on the previously sampled support.
Revised: Next, the expression value y i is obtained via the evaluation of the expression e on the previously sampled support.

Original: Expressions with more than 10% of
Revised: Expressions with more than 10% of

Original: NaNs are discarded from the training set.
Revised: NaNs are discarded from the training set.

Original: We train our model with output noise drawn from centered
Revised: We train our model with output noise drawn from centered

Original: Gaussian distribution with deviation
Revised: Gaussian distribution with deviation

Original: [Equation]
Revised: [Equation]

Original: with γ ∈ { 0 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 } as prescribed in La Cava et al.
Revised: with γ ∈ { 0 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 } as prescribed in (La Cava et al., 2021)

Original: Generating D PI .
Revised: Generating D PI .

Original: Privileged information D PI is composed of properties .
Revised: Privileged information D PI is composed of properties .

Original: From an expression e , weconsider the following  1 :
Revised: From an expression e , we consider the following properties 1 :

Original: complexity bins : We use the definition of complexity provided by Kommenda et al.  that assigns a score that reflects semantic information of the expression and better aligns with the human intuition of what a complex expression looks like, contrarily to more standard metrics measuring complexity based on the length of the expression tree.
Revised: complexity bins : We use the definition of complexity provided by Kommenda et al. (2015) that assigns a score that reflects semantic information of the expression and better aligns with the human intuition of what a complex expression looks like, contrarily to more standard metrics measuring complexity based on the length of the expression tree.

Original: Since the SR practioner generally has only a vague idea of the desired function, we bin the complexity into intervals, i.e. [0 , 5] , . . .
Revised: Since the SR practioner generally has only a vague idea of the desired function, we bin the complexity into intervals, i.e. [0 , 5] , . . .

Original: , [1000000 , ∞ ] , resulting in a total of 6 complexity levels.
Revised: , [1000000 , ∞ ] , resulting in a total of 6 complexity levels.

Original: Note that the bins delimiters where chosen to have approximately the same counts on our training set of expressions.
Revised: Note that the bins delimiters where chosen to have approximately the same counts on our training set of expressions.

Original: symmetries : We use the definition of generalized symmetry proposed in Udrescu et al.(2020): f has generalized symmetry if the d components of the vector x ∈ R d can be splitinto groups of k and d − k components (which we denote by the vectors x 
Revised: symmetries : We use the definition of generalized symmetry proposed in (Udrescu et al., 2020): f has generalized symmetry if the d components of the vector x ∈ R d can be split into groups of k and d − k components (which we denote by the vectors x ′ ∈ R k and x ′′ ∈ R d − k ) such that f ( x ) =

Original: ′ ∈ R k andx ′′ ∈ R d − k ) such that f ( x ) = f
Revised: f

Original: ( x ′ , x  ) =
Revised: ( x ′ , x ′′ ) =

Original: g [ h ( x ′ ) , x ′′ ] for some unknown function g .
Revised: g [ h ( x ′ ) , x ′′ ] for some unknown function g .

Original: Udrescu et al.
Revised: Udrescu et al.

Original: (2020) provides an efficient recipe based on ∇ f to assess if f is characterized by a generalized symmetry in every subset of its independent variables.
Revised: (2020) provides an efficient recipe based on ∇ f to assess if f is characterized by a generalized symmetry in every subset of its independent variables.

Original: We feed the network with a tensor of 2 d = 32 components, equal to the number of all possible subset of variables, each indicating whether that subset possesses a generalized symmetry or not.
Revised: We feed the network with a tensor of 2 d = 32 components, equal to the number of all possible subset of variables, each indicating whether that subset possesses a generalized symmetry or not.

Original: Appearing branches: We consider the set of all the branches that appear in the pre- fix tree of the generating expression.
Revised: Appearing branches: We consider the set of all the branches that appear in the prefix tree of the generating expression.

Original: For instance for x 1 + sin x 2 this set would be [+ , x 1 , + x 1 , sin , sin x 2 , x 2 , + sin , + sin x 2 ] .
Revised: For instance, for x 1 + sin x 2 this set would be [+ , x 1 , + x 1 , sin , sin x 2 , x 2 , + sin , + sin x 2 ] .

Original: We then sample between 0 and 4 items from this set, with a weighted probability inversely proportional to the length of them.
Revised: We then sample between 0 and 4 items from this set, with a weighted probability inversely proportional to their length.

Original: Absent branches: We sample from 0 up to 4 branches which do not appear in the generating expression, drawn from the space of all branches that appear in expressions in our dataset.
Revised: Absent branches: We sample from 0 up to 4 branches which do not appear in the generating expression, drawn from the space of all branches that appear in expressions in our dataset.

Original: As for the previous property, we sample them with a probability inversely proportional to their length.
Revised: As for the previous property, we sample them with a probability inversely proportional to their length.

Original: Hypotheses 1 44 52 52 ... 45 46 53 47 48 31 49 50 23 51symmetry complexity + subtrees - subtrees

Original: In our experiments, we explore whether our NSRwH model is able to capture the meaning of suchproperties andto generate expressions that adhere with them.
Revised: In our experiments we explore whether our NSRwH model is able to capture the meaning of such properties and generate expressions that adhere to them.

Original: We give more details on the exactcomputation of each properties in the Appendix A.
Revised: We give more details on the exact computation of each property in the Appendix A.

Revised: Hypotheses 1 44 52 52 ... 45 46 53 47 48 31 49 50 23 51symmetry complexity + subtrees - subtrees

Original: 2.3 Model
Revised: 2.3 Model

Original: In this section, we report the details of the architecture of our model as well as its training procedureand adaptation at test time.
Revised: In this section, we report the details of the architecture of our model as well as its training procedure and adaptation at test time.

Original: Architecture.
Revised: Architecture.

Original: We use Nesymres Biggio et al. as our base neural symbolic regressor for itssimplicity and explain how we incorporate the description D PI as an input to the model g ϕ ( e |D , D PI ) .
Revised: We use Nesymres (Biggio et al., 2021) as our base neural symbolic regressor for its simplicity and explain how we incorporate the description D PI as an input to the model g ϕ ( e |D , D PI ) .

Original: Note that the very same conditioning strategy can easily be applied to other more advanced NSRarchitectures, such as the one introduced in Valipour et al. ; Kamienny et al.
Revised: Note that the very same conditioning strategy can easily be applied to other more advanced NSR architectures, such as the one introduced in (Valipour et al., 2021; Kamienny et al., 2022).

Original: Floatnumbers are tokenized using a multi-hot bit representation according to the half-precision IEEE-754standard and expressions are tokenized usingthe their polish notation.
Revised: Float numbers are tokenized using a multi-hot bit representation according to the half-precision IEEEstandard and expressions are tokenized using their polish notation.

Original: The input dataset D of size n is encoded in z D ∈ R n × d emb by i) tokenizing floats and ii) passingthese tokens through a set encoder Lee et al. a variation of Vaswani et al. (2017) with betterinference time and less memory requirement.
Revised: The input dataset D of size n is encoded in z D ∈ R n × d emb by i) tokenizing floats and ii) passing these tokens through a set encoder (Lee et al., 2019), a variation of (Vaswani et al., 2017) with better inference time and less memory requirement.

Original: Positional embeddings are not included to obtain setpermutation-invariant representations.
Revised: Positional embeddings are not included to obtain set permutation-invariant representations.

Original: We use an additional  self-attention encoder Vaswani et al.(2017), which we call privileged encoder , to embed privileged information D PI into a sequencez D PI ∈ R m × d emb , where m ̸ = n.
Revised: We use an additional standard self-attention encoder (Vaswani et al., 2017), which we call privileged encoder , to embed privileged information D PI into a sequence z [Equation] where m ̸ = n

Original: All properties are first independently tokenized using propertytokens, e.g. complexity bin [0 , 5] , or polish notation for subtree expressions.
Revised: All properties are first independently tokenized using property tokens, e.g. complexity bin [0 , 5] , or polish notation for subtree expressions.

Original: Then, we concatenatethem using property separators, e.g. < symmetries > or < complexity > , into a large sequence.
Revised: Then, we concatenate them using property separators, e.g. < symmetries > or < complexity > , into a large sequence.

Original: The decoder uses self-attention as well as cross-attention on the tensor z ∈ R ( m + n ) × d emb obtainedby the concatenation of z D and z D PI and predicts the expression ˜ e in an auto-regressive fashionfollowing g ϕ (˜ e t +1 | ˜ e 1: t , D , D PI ) .
Revised: The decoder uses self-attention as well as cross-attention on the tensor z ∈ R ( m + n ) × d emb obtained by the concatenation of z D and z D PI and predicts the expression ˜ e in an auto-regressive fashion following g ϕ (˜ e t +1 | ˜ e 1: t , D , D PI ) .

Original: Training.
Revised: Training.

Original: As done in all NSR approaches, we use the cross-entropy loss on next-token predictionusing teacher-forcing Sutskever et al. (2014), i.e. conditioning g ϕ
Revised: As done in all NSR approaches, we use the cross-entropy loss on next-token prediction using teacher-forcing (Sutskever et al., 2014), i.e. conditioning g ϕ

Original: (˜ e t +1 | e 1: t , D , D PI ) on the groundtruth e first tokens.
Revised: (˜ e t +1 | e 1: t , D , D PI ) on the groundtruth e first tokens.

Original: As for Nesymres, we “skeletonize” target expressions by replacing constants by aconstant token ⋄ .
Revised: As for Nesymres, we “skeletonize” target expressions by replacing constants by a constant token ⋄ .

Original: To avoid our model not be dependent on privileged information at test time, we include trainingexamples with partial privileged information D PI .
Revised: To avoid our model being dependent on privileged information at test time, we include training examples with partial privileged information D PI .

Original: Practically, we mask property tokens by usingdropout Srivastava et al. (2014): with probability p , a property changes its value to the < PAD > token (also used for padding sequences of different lengths for batching).
Revised: Practically, we mask property tokens by using dropout (Srivastava et al., 2014): with probability p , a property changes its value to the < PAD > token (also used for padding sequences of different lengths for batching).

Original: More details on the traininghyperparameters can be found in Appendix B.
Revised: More details on the training hyperparameters can be found in Appendix B.

Original: Testing.
Revised: Testing.

Original: At test time, as for Nesymres, we use beam search to produce a set of predicted expressions,then we apply Broyden–Fletcher–Goldfarb–Shanno algorithm (BFGS) to recover the values of theconstants by minimizing the squared loss between the original outputs and the output from thepredicted expressions.
Revised: At test time, as for Nesymres, we use beam search to produce a set of predicted expressions, then we apply Broyden–Fletcher–Goldfarb–Shanno algorithm (BFGS) to recover the values of the constants by minimizing the squared loss between the original outputs and the output from the predicted expressions.

Original: The test-time procedure is summarized on Fig.
Revised: The test-time procedure is summarized on Fig.

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate NSRwH and compare to Biggio et al. (2021), which we name NSR forthe sake of clarity.
Revised: In this section, we evaluate NSRwH and compare to (Biggio et al., 2021), which we name NSR for the sake of clarity.

Original: First, we analyze whether transformers can capture the properties introducedin Sec. 2.
Revised: First, we analyze whether transformers can capture the properties introduced in Sec. 2.

Original: In addition, we describe a number of simple examples to illustrate how NSRwH can bepractically used to discover expressions given a set of prior assumptions by enhancing user interactionand facilitating hypothesis testing.
Revised: In addition, we describe a number of simple examples to illustrate how NSRwH can be practically used to discover expressions given a set of prior assumptions by enhancing user interaction and facilitating hypothesis testing.

Original: Finally, we assess the extent to which extra conditioning helps toimprove the test performance.
Revised: Finally, we assess the extent to which extra conditioning helps to improve the test performance.

Original: Following the procedure described in , we generated a set of 20 M training expressions withoutduplicates using  Sympy Meurer et al. (2017) simplification function.
Revised: Following the procedure described in 2.2, we generated a set of 20 M training expressions without duplicates using the Sympy (Meurer et al., 2017) simplification function.

Original: We trained our model on thetrain set using the procedure described in 2.3 on 8 P100 GPU for 2 days.
Revised: We trained our model on the training set using the procedure described in 2.3 on 8 P100 GPU for 2 days.

Original: 3.1 Can transformers efficiently restrict the inference space using descriptions?
Revised: 3.1 Can transformers efficiently restrict the inference space using descriptions?

Original: We investigate the ability of transformers to “understand” the properties of analytical expressionsintroduced in 2.
Revised: We investigate the ability of transformers to “understand” the properties of analytical expressions introduced in 2.

Original: We give as an input to the NSRwH model a single hypothesis (i.e. K = 1 on
Revised: We give as an input to the NSRwH model a single hypothesis (i.e. K = 1 on

Original: Fig. 1) which is the privileged one.
Revised: Fig. 1) which is the privileged one.

Original: Our test set consists of 100 randomly-generated mathematicalexpressions.
Revised: Our test set consists of 100 randomly-generated mathematical expressions.

Original: 3.1.1 Quantitative results
Revised: 3.1.1 Quantitative results

Original: To validate that transformers can restrict the class of predicted expressions according to a particularinput property, we introduce a metric is _ satisfied ( g ϕ , D , property _ type ) that measures the percentage of properties satisfied in the predicted expression for a given property type (averaged over10 samples from g ϕ ).
Revised: To validate that transformers can restrict the class of predicted expressions according to a particular input property, we introduce a metric is _ satisfied ( g ϕ , D , property _ type ) that measures the percentage of properties satisfied in the predicted expression for a given property type (averaged over 10 samples from g ϕ ).

Original: For instance, for the property type of appearing subtrees x 1 , sin , x 21 + x 22 ,x 1 /x 2 hypotheses, in predicted expression f ( x 1 , x 2 ) =
Revised: For instance, for the property type of appearing subtrees x 1 , sin , x 21 + x 22 , x 1 /x 2 hypotheses, in predicted expression f ( x 1 , x 2 ) =

Original: x 1 − sin ( x 21 + x 22 ) , only 75% properties arerespected since x 1 /x 2 does not appear in the ground truth expression.
Revised: x 1 − sin ( x 21 + x 22 ) , only 75% properties are respected since x 1 /x 2 does not appear in the ground truth expression.

Original: From the test set { ( e, D , D PI ) } j ≤ 100 , we derive a dataset of descriptions and measure is _ satisfiedaveraged over all 100 examples for each property type.
Revised: From the test set { ( e, D , D PI ) } j ≤ 100 , we derive a dataset of descriptions and measure is _ satisfied averaged over all 100 examples for each property type.

Original: To make the task more challenging, wecorrupt the numerical data with zero-mean gaussian noise with standard deviation as in 2, withγ = 10 − 2 .
Revised: To make the task more challenging, we corrupt the numerical data with zero-mean gaussian noise with standard deviation as in 2, with γ = 10 − 2 .

Original: This test is meant to assess the reliance of our conditioned model on the input descriptionregardless of the noise in the input numerical data.
Revised: This test is meant to assess the reliance of our conditioned model on the input description regardless of the noise in the input numerical data.

Original: We compare our model with NSR, i.e. a modelreceiving as input only numerical information, D .
Revised: We compare our model with NSR, i.e. a model receiving as input only numerical information, D .

Original: [Table]
Revised: [Table]

Original: Overall, as shown in Table.
Revised: Overall, as shown in Table.

Original: 1,the proposed NSRwH efficiently reduces the inference space whenconditioning on a set of hypotheses and outputs predictions that are well-aligned with expectations.
Revised: 1, the proposed NSRwH efficiently reduces the inference space when conditioning on a set of hypotheses and outputs predictions that are well-aligned with expectations.

Original: We conclude that, not only does the model “understands” the meaning of the input conditioning, but it does so also in the presence of a significant amount of noise in the input numerical data, hencemanifesting a high-level of controllability.
Revised: We conclude that not only does the model “understands” the meaning of the input conditioning, but it does so also in the presence of a significant amount of noise in the input numerical data, hence manifesting a high-level of controllability.

Original: 3.1.2 Qualitative results
Revised: 3.1.2 Qualitative results

Original: In this section, we qualitatively investigate the behaviour of our conditioned model NSRwH via anumber of examples where it is provided with different information about the ground-truth expression.
Revised: In this section, we qualitatively investigate the behaviour of our conditioned model NSRwH via several examples where it is provided with different information about the ground-truth expression.

Original: Role of Generalized Symmetry.
Revised: Role of Generalized Symmetry.

Original: Consider the following expression of five independent variables:
Revised: Consider the following expression of five independent variables:

Original: [Equation]
Revised: [Equation]

Original: When all the privileged information is masked, the model is not able to infer the ground truthexpression up to a beam-size of 50, despite capturing its sinusoidal nature in all the output predictions.
Revised: When all the privileged information is masked, the model is not able to infer the ground truth expression up to a beam-size of 50, despite capturing its sinusoidal nature in all the output predictions.

Original: When five variables are considered the search space is inevitably very large and any sort of priorknowledge can be precious to restrict it to a smaller subset.
Revised: When five variables are considered the search space is inevitably very large and any sort of prior knowledge can be precious to restrict it to a smaller subset.

Original: It is in this high-dimensional regimethat information about generalized symmetry is important.
Revised: It is in this high-dimensional regime that information about generalized symmetry is important.

Original: We note that 3 possesses non-trivialgeneralized symmetries in the variable tuples ( x 2 , x 3 ) and ( x 4 , x 5 ) , ( x 1 , x 2 , x 3 ) and ( x 1 , x 4 , x 5 ) .
Revised: We note that 3 possesses non-trivial generalized symmetries in the variable tuples ( x 2 , x 3 ) and ( x 4 , x 5 ) , ( x 1 , x 2 , x 3 ) and ( x 1 , x 4 , x 5 ) .

Original: Given only this high-level information as input, the model is able to recover the true expressionamong the first 20 output expressions.
Revised: Given only this high-level information as input, the model is able to recover the true expression among the first 20 output expressions.

Original: Role of complexity.
Revised: Role of complexity.

Original: Let’s consider the following expression of three independent variables:
Revised: Let’s consider the following expression of three independent variables:

Original: [Equation]
Revised: [Equation]

Original: Given numerical inputs only, NSRwH does not recover the true expression up to a beam-size of 50,despite capturing its sinusoidal nature and the linear dependence on x 3 .
Revised: Given numerical inputs only, NSRwH does not recover the true expression up to a beam-size of 50, despite capturing its sinusoidal nature and the linear dependence on x 3 .

Original: What makes the task hardis the complex compositional nature of the first term involving both sin and cos operators.
Revised: What makes the task hard is the complex compositional nature of the first term involving both sin and cos operators.

Original: In orderto bias the model towards more complex formulas, we condition it on a relatively high complexitybin, namely the fourth one.
Revised: In order to bias the model towards more complex formulas, we condition it on a relatively high complexity bin, namely the fourth one.

Original: By giving the model this additional information, the model’s predictionsstrictly adhere with the conditioning, giving the correct formula at beam 26.
Revised: By giving the model this additional information, the model’s predictions strictly adhere with the conditioning, giving the correct formula at beam 26.

Original: Role of appearing subtrees.
Revised: Role of appearing subtrees.

Original: Let’s consider the following expression of three independent variables:
Revised: Let’s consider the following expression of three independent variables:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: 3 tan( x 3 ) (5)

Original: The second term in the expression above is very difficult to detect precisely, due to the smallmultiplying factor making its contribution weak.
Revised: The second term in the expression above is very difficult to detect precisely, due to the small multiplying factor making its contribution weak.

Original: The fully-masked model does not recover the trueexpression, yet catching the first term consistently.
Revised: The fully-masked model does not recover the true expression, yet catching the first term consistently.

Original: By providing the model with the additionalinformation of the presence of the unary operator tan , NSRwH finds the true expression at beam 20.
Revised: By providing the model with the additional information of the presence of the unary operator tan , NSRwH finds the true expression at beam 20.

Original: 3.2 Does extra-conditioning help improve test accuracy?
Revised: 3.2 Does extra-conditioning help improve test accuracy?

Revised: Model g ϕ

Revised: Mask probability p

Revised: Noise level γ γ

Revised: =γ = 1 e −γ = 1 e −γ =

Revised: NSRwH 1.0 (NSR) 0.

Original: [Table]
Revised: [Table]

Original: We investigate if extra-conditioning in the form of privileged information results in improved accuracyand robustness to noise.
Revised: We investigate if extra-conditioning in the form of privileged information results in improved accuracy and robustness to noise.

Original: To this purpose, we utilize a set of 100 randomly-generated expressions, wecalculate the R 2 score, as defined in 1, on a set of support points sampled outside the distribution ofthe input ones, i.e. in the extrapolation regime.
Revised: To this purpose, we utilize a set of 100 randomly-generated expressions, we calculate the R 2 score, as defined in 1, on a set of support points sampled outside the distribution of the input ones, i.e. in the extrapolation regime.

Original: An equation is correctly classified if R 2 > 0 . 99 .
Revised: An equation is correctly classified if R 2 > 0 . 99 .

Original: As it is possible to see in Table 3.2 the model’s performance improves when leveraging the prompt information.
Revised: As shown in Table 3.2 the model’s performance improves when leveraging privileged information.

Original: In particular, we make the following observations: 1) The smaller the masking probability,the larger the accuracy 2 2) Conditioning is especially beneficial in the cases where noise is applied,leading to substantial improvements with respect to the fully-masked model.
Revised: In particular, we make the following observations: 1) The smaller the masking probability, the larger the accuracy 2 2) Conditioning is especially beneficial in the cases where noise is applied, leading to substantial improvements with respect to the fully-masked model.

Original:  Discussion
Revised:  Discussion

Original: Conclusive remarks.
Revised: Conclusive remarks.

Original: This work presents a novel approach for symbolic regression that enables theexplicit incorporation of inductive biases reflecting prior knowledge on the problem at-hand.
Revised: This work presents a novel approach for symbolic regression that enables the explicit incorporation of inductive biases reflecting prior knowledge on the problem at-hand.

Original: In starkcontrast to previous works, this can be effectively done at test time, drastically reducing computationaloverhead.
Revised: In stark contrast to previous works, this can be effectively done at test time, drastically reducing the computational overhead.

Original: Thanks to this property, our model better lends itself to online and interactive applicationsof symbolic regression, thus enabling fast hypothesis testing, an highly desirable feature for scientificdiscovery applications.
Revised: Thanks to this property, our model better lends itself to online and interactive applications of symbolic regression, thus enabling fast hypothesis testing, an highly desirable feature for scientific discovery applications.

Original: We demonstrate the value of this approach with a number of examples andpreliminary experiments where numerical data is sparse or affected by noise.
Revised: We demonstrate the value of this approach with a number of examples and preliminary experiments where numerical data is sparse or affected by noise.

Original: Future work and outlook.
Revised: Future work and outlook.

Original: We plan to extend the present research by further evaluating NSRwHon more datasets and benchmarks.
Revised: We plan to extend the present research by further evaluating NSRwH on more datasets and benchmarks.

Original: We emphasize that the current version of our model can beimproved from multiple angles.
Revised: We emphasize that the current version of our model can be improved from multiple angles.

Original: First and foremost, due to limited computational resources, NSRwHis not comparable with modern language models in terms of size.
Revised: First and foremost, due to limited computational resources, NSRwH is not comparable with modern language models in terms of size.

Original: Scaling NSRwH to such regimesrepresents an exciting avenue for future research and for fully unveiling the real capabilities of NSR.
Revised: Scaling NSRwH to such regimes represents an exciting avenue for future research and for fully unveiling the real capabilities of NSR.

Original: Second, additional properties can possibly be included in the privileged information conditioningaccording to the user’s prior knowledge of the data under consideration.
Revised: Second, additional properties can possibly be included in the privileged information conditioning according to the user’s prior knowledge of the data under consideration.

Original: Third, our framework canbe easily incorporated into more advanced NSR architectures, like Kamienny et al.
Revised: Third, our framework can be easily incorporated into more advanced NSR architectures, like Kamienny et al.

Original: , likelyresulting in further performance improvements.
Revised: (2022), likely resulting in further performance improvements.

Original: In conclusion, we believe that the method presentedin this work will enable a wider application of symbolic regression, resulting in more effective anduser-centric methods.
Revised: In conclusion, we believe that the method presented in this work will enable a wider application of symbolic regression, resulting in more effective and user-centric methods.

Original: Impact statement
Revised: Impact statement

Original: Symbolic Regression is a growing field and its combination with neural networks has witnessedsignificant interest in the last few years.
Revised: Symbolic Regression is a growing field and its combination with neural networks has witnessed significant interest in the last few years.

Original: We believe that this new direction represents a significantstep towards the adoption of symbolic regression methods as useful tools to support the scientificdiscovery process.
Revised: We believe that this new direction represents a significant step towards the adoption of symbolic regression methods as useful tools to support the scientific discovery process.

Original: We do not foresee any ethical concerns with this work.
Revised: We do not foresee any ethical concerns with this work.

Original: A Details on propertiesA.1 Generalized symmetry
Revised: A Details on properties A.1 Generalized symmetry

Original: We use the definition of generalized symmetry proposed in Udrescu et al.
Revised: We use the definition of generalized symmetry proposed in Udrescu et al.

Original: (2020): f has generalizedsymmetry if the d components of the vector x ∈ R d can be split into groups of k and d − k components(which we denote by the vectors x ′ ∈ R k and x ′′ ∈ R d − k ) such that f ( x ) = f
Revised: (2020): f has generalized symmetry if the d components of the vector x ∈ R d can be split into groups of k and d − k components (which we denote by the vectors x ′ ∈ R k and x ′′ ∈ R d − k ) such that f ( x ) = f

Original: ( x ′ , x ′′ ) =
Revised: ( x ′ , x ′′ ) =

Original: g [ h ( x ′ ) , x ′′ ] for some unknown function g .
Revised: g [ h ( x ′ ) , x ′′ ] for some unknown function g .

Original: As explained in Udrescu et al.
Revised: As explained in Udrescu et al.

Original: (2020), in order tocheck the presence of generalized symmetry in the set of variables x ′ , it is sufficient to checkwhether the normalized gradient of f with respect to x ′ is independent on x ′′ , i.e. ∇ x ′ f
Revised: (2020), in order to check the presence of generalized symmetry in the set of variables x ′ , it is sufficient to check whether the normalized gradient of f with respect to x ′ is independent on x ′′ , i.e. ∇ x ′ f

Original: ( x ′ , x ′′ ) |∇ x ′ f ( x ′ , x ′′ ) | isx ′′ -independent.
Revised: ( x ′ , x ′′ ) |∇ x ′ f ( x ′ , x ′′ ) | is x ′′ -independent.

Original: Thanks to this property, given a symbolic expression, we can rapidly extract itsgeneralized symmetries on every subset of variables.
Revised: Thanks to this property, given a symbolic expression, we can rapidly extract its generalized symmetries on every subset of variables.

Original: Since in this work we limit ourselves to 5independent variables, for each expression we have a 32 dimensional tensor encoding whether asymmetry is present for every subset of variables.
Revised: Since in this work we limit ourselves toindependent variables, for each expression we have a 32 dimensional tensor encoding whether a symmetry is present for every subset of variables.

Original: Ultimately, we exclude from this vector sometrivial and non-informative components, i.e. subset of only one-variable, the full-variable subset andthe empty set, resulting in a 25-dimensional tensor.
Revised: Ultimately, we exclude from this vector some trivial and non-informative components, i.e. subset of only one-variable, the full-variable subset and the empty set, resulting in a 25-dimensional tensor.

Original: A.2 Appearing / absent branches
Revised: A.2 Appearing / absent branches

Original: In order to sample branches from a prefix tree, we adopt the following procedure: first, we enumerateall possible branches of the tree using a Depth-First-Search (DFS) algorithm.
Revised: In order to sample branches from a prefix tree, we adopt the following procedure: first, we enumerate all possible branches of the tree using a Depth-First-Search (DFS) algorithm.

Original: We pre-compute the setof all possible branches for absent trees offline from a subset of the training data.
Revised: We pre-compute the set of all possible branches for absent trees offline from a subset of the training data.

Original: In particular, forthe experiments, we use a subset of 10,000 trees.
Revised: In particular, for the experiments, we use a subset of 10,000 trees.

Original: Then we compute the disjoint set between this setand the current prefix tree and sample from there.
Revised: Then we compute the disjoint set between this set and the current prefix tree and sample from there.

Original: For both appearing and absent subtrees, branchelements are sampled with a probability inverse to their length during both training and experimentalevaluation.
Revised: For both appearing and absent subtrees, branch elements are sampled with a probability inverse to their length during both training and experimental evaluation.

Original: Once sampled, the elements are sorted alphanumerically so that their order is consistentduring training and evaluation.
Revised: Once sampled, the elements are sorted alphanumerically so that their order is consistent during training and evaluation.

Original: A.3 Complexity
Revised: A.3 Complexity

Original: We follow the definition of complexity introduced in Kommenda et al.
Revised: We follow the definition of complexity introduced in Kommenda et al.

Original: (2015), with only some slightmodifications.
Revised: (2015), with only some slight modifications.

Original: Starting at the root node, the complexity of an expression tree can be calculated byapplying the equation below recursively:
Revised: Starting at the root node, the complexity of an expression tree can be calculated by applying the equation below recursively:

Original: [Equation]
Revised: [Equation]

Original: if n ∈
Revised: After obtaining the complexity score associated with an expression

Original: [Equation]
Revised: [Equation]

Original: sin [Equation] [Equation] , exp , log
Revised: we assign it a bin level according with the following rule:

Original: [Equation]
Revised: [Equation]

Original: After obtaining the complexity score associated with an expression, we assign it a bin level accordingwith the following rule:

Original: Complexity_level ( n

Original: [Equation]
Revised: [Equation]

Original: =    if Complexity ( n ) ∈ [0 , 5] if Complexity ( n ) ∈ (5 , 100] if Complexity ( n ) ∈ (100 , 1000] if Complexity ( n ) ∈ (1000 , 10000] if Complexity ( n ) ∈ (10000 , 1000000] if Complexity ( n ) >

Original: B Training detailsB.1 Dataset generation
Revised: B Training details B.1 Dataset generation

Original: We sample analytical expressions e using the strategy introduced by Lample and Charton (2019):random unary-binary trees with depth between 1 and 5 are generated, then internal nodes are assignedeither unary or binary operators described in Table 3{+
+} according to their arity, and leaves are assignedvariables { x d } d ≤[-
-]
Revised: We sample analytical expressions e using the strategy introduced by Lample and Charton (2019): random unary-binary trees with depth between 1 and 5 are generated, then internal nodes are assigned either unary or binary operators described in Table {+
+} according to their arity, and leaves are assigned variables { x d } d ≤[-
-]

Original: [Table]
Revised: [Table]

Original: B.2 Training hyper-parameters
Revised: B.2 Training hyper-parameters

Original: [Table]

Original: Set encoder.
Revised: Set encoder.

Original: We embed input datasets with set encoder Lee et al.
Revised: We embed input datasets with set encoder Lee et al.

Original: (2019) using 5 Induced Set
Revised: (2019) using 5 Induced Set

Original: Attention Blocks (ISAB) with 512 hidden dimensions, 8 heads, 10 Pooling by Multihead Attention(PMA) features and 50 inducing points.
Revised: Attention Blocks (ISAB) with 512 hidden dimensions, 8 heads, 10 Pooling by Multihead Attention (PMA) features and 50 inducing points.

Original: Privileged encoder.
Revised: Privileged encoder.

Original: We use the classic transformer encoder Vaswani et al.
Revised: We use the classic transformer encoder Vaswani et al.

Original: (2017) using 5 layerswith 512 hidden dimensions, 8 heads and embedding dimensions 32 .
Revised: (2017) using 5 layers with 512 hidden dimensions, 8 heads and embedding dimensions 32 .

Original: Decoder.
Revised: Decoder.

Original: We use the classic transformer decoder using 5 layers with 512 hidden dimensions, 8heads and embedding dimensions 32 .
Revised: We use the classic transformer decoder using 5 layers with 512 hidden dimensions,heads and embedding dimensions 32 .

Original: NN Optimizer.
Revised: NN Optimizer.

Original: We use Adam with batch size 32 , learning rate 10 − 4 and weight decay 10 − 3 .
Revised: We use Adam with batch size 32 , learning rate 10 − 4 and weight decay 10 − 3 .

Original: Numerical constant optimizer.
Revised: Numerical constant optimizer.

Original: In all quantitative experiments, we use the BFGS optimizer withbatch size of 32 and 2 restarts.
Revised: In all quantitative experiments, we use the BFGS optimizer with batch size of 32 and 2 restarts.