Original: T RANSFORMER - BASED MODEL FOR SYMBOLIC REGRESSION VIA JOINT SUPERVISED LEARNING
Revised: T RANSFORMER - BASED MODEL FOR SYMBOLIC REGRESSION VIA JOINT SUPERVISED LEARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Symbolic regression (SR) is an important technique for discovering hidden mathematical expressions from observed data.
Revised: Symbolic regression (SR) is an important technique for discovering hidden mathematical expressions from observed data.

Original: Transformer-based approaches have been widely used for machine translation due to their high performance, and are recently highly expected to be used for SR.
Revised: Transformer-based approaches have been widely used for machine translation due to their high performance, and are recently highly expected to be used for SR.

Original: They input the data points, then output the expression skeleton, and finally optimize the coefficients.
Revised: They input the data points, then output the expression skeleton, and finally optimize the coefficients.

Original: However, recent transformer-based methods for SR focus more attention on large scale training data and ignore the ill-posed problem: the lack of sufficient supervision, i.e. expressions that may be completely different have the same supervision because of their same skeleton, which makes it challenging to deal with data that may be from the same expression skeleton but with different coefficients.
Revised: However, recent transformer-based methods for SR focus more attention on large scale training data and ignore the ill-posed problem: the lack of sufficient supervision, i.e., expressions that may be completely different have the same supervision because of their same skeleton, which makes it challenging to deal with data that may be from the same expression skeleton but with different coefficients.

Original: Therefore, we present a transformer-based model for SR with the ability to alleviate this problem.
Revised: Therefore, we present a transformer-based model for SR with the ability to alleviate this problem.

Original: Specifically, we leverage a feature extractor based on pure residual MLP networks to obtain more information about data points.
Revised: Specifically, we leverage a feature extractor based on pure residual MLP networks to obtain more information about data points.

Original: Furthermore, the core idea is that we propose a joint learning mechanism combining supervised contrastive learning, which makes features of data points from expressions with the same skeleton more similar so as to effectively alleviates the ill-posed problem.
Revised: Furthermore, the core idea is that we propose a joint learning mechanism combining supervised contrastive learning, which makes features of data points from expressions with the same skeleton more similar so as to effectively alleviates the ill-posed problem.

Original: The benchmark results show that the proposed method is up to 25% higher with respect to the recovery rate of skeletons than typical transformer-based methods.
Revised: The benchmark results show that the proposed method is up to 25% higher with respect to the recovery rate of skeletons than typical transformer-based methods.

Original: Moreover, our method outperforms state-of-the-art SR methods based on reinforcement learning and genetic programming in terms of the coefficient of determination ( R 2 ).
Revised: Moreover, our method outperforms state-of-the-art SR methods based on reinforcement learning and genetic programming in terms of the coefficient of determination ( R 2 ).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Exploring mathematical expressions that can be fitted to real-world observed data is the core of expressing scientific discoveries.
Revised: Exploring mathematical expressions that can be fitted to real-world observed data is the core of expressing scientific discoveries.

Original: The correct expression would not only provide us with useful scientific insights simply by inspection but would also allow us to forecast how the process will change in the future.
Revised: The correct expression would not only provide us with useful scientific insights simply by inspection but would also allow us to forecast how the process will change in the future.

Original: The task of finding such an interpretable mathematical expression from observed data is called symbolic regression .
Revised: The task of finding such an interpretable mathematical expression from observed data is called symbolic regression .

Original: More specifically, given a dataset ( X, y ) , where each feature X i ∈ R n and target y i ∈ R , the goal of symbolic regression is to identify a function f (i.e. y ≈ f ( X ) : R n → R ) that best fits the dataset.
Revised: More specifically, given a dataset ( X, y ) , where each feature X i ∈ R n and target y i ∈ R , the goal of symbolic regression is to identify a function f (i.e., y ≈ f ( X ) : R n → R ) that best fits the dataset.

Original: Symbolic regression is NP-hard because the search space of an expression grows exponentially with the length of the expression, and the presence of numeric constants further exacerbates its difficulty (Lu et al., 2016).
Revised: Symbolic regression is NP-hard because the search space of an expression grows exponentially with the length of the expression, and the presence of numeric constants further exacerbates its difficulty (Lu et al., 2016).

Original: Considering this issue, genetic programming (GP) as the most common approach is leveraged to tackle the symbolic regression problems (Forrest, 1993; Koza, 1994;
Revised: Considering this issue, genetic programming (GP) as the most common approach is leveraged to tackle the symbolic regression problems (Forrest, 1993; Koza, 1994;

Original: Schmidt & Lipson, 2009; Staelens et al., 2013; Arnaldo et al., 2015; Bładek & Krawiec, 2019).
Revised: Schmidt & Lipson, 2009; Staelens et al., 2013; Arnaldo et al., 2015; Bładek & Krawiec, 2019).

Original: GP-based methods iteratively “evolves” each generation of mathematical expressions through selection, crossover, and mutation.
Revised: GP-based methods iteratively “evolves” each generation of mathematical expressions through selection, crossover, and mutation.

Original: Although this approach can be effective, the expression it yields is complex, and it is also known to be computationally expensive and to exhibit high sensitivity to hyperparameters.
Revised: Although this approach can be effective, the expression it yields is complex, and it is also known to be computationally expensive and to exhibit high sensitivity to hyperparameters.

Original: A more recent line of research has made use of the neural network to tackle the aforementioned shortcomings.
Revised: A more recent line of research has made use of the neural network to tackle the aforementioned shortcomings.

Original: Martius & Lampert (2016) propose a simple fully-connected neural network called “EQL”, where elementary functions ( sin , cos , + , ...) are used as activation functions.
Revised: Martius & Lampert (2016) propose a simple fully-connected neural network called “EQL”, where elementary functions ( sin , cos , + , ...) are used as activation functions.

Original: The limitation of EQL is the existence of vanishing gradient and exploding gradient, and the depth of the network limits the complexity of the predicted equation.
Revised: The limitation of EQL is the existence of vanishing gradient and exploding gradient, and the depth of the network limits the complexity of the predicted equation.

Original: More recently, deep symbolic optimization (DSO) (Petersen et al., 2021) trains the RNN using the reinforcement learning algorithm based on a risk-seeking policy gradient to generate expressions.
Revised: More recently, deep symbolic optimization (DSO) (Petersen et al., 2021) trains the RNN using the reinforcement learning algorithm based on a risk-seeking policy gradient to generate expressions.

Original: They take the output from
Revised: They take the output from

Original: RNN as an initial population for a genetic algorithm to find the target expression.
Revised: RNN as an initial population for a genetic algorithm to find the target expression.

Original: Albeit the above two approaches show promising results, they still handle symbolic regression as an instance-based problem, training a model from scratch on each new input dataset for a regression task.
Revised: Albeit the above two approaches show promising results, they still handle symbolic regression as an instance-based problem, training a model from scratch on each new input dataset for a regression task.

Original: Inspired by the successes of large scale pre-training, recent efforts in symbolic regression have focused on using the transformer-based model and training with a large amount of data (Valipour et al., 2021; Biggio et al., 2021).
Revised: Inspired by the successes of large scale pre-training, recent efforts in symbolic regression have focused on using the transformer-based model and training with a large amount of data (Valipour et al., 2021; Biggio et al., 2021).

Original: They all approach the symbolic regression problem as a machine translation problem, mapping the input data to latent representations via encoders, and then outputting the skeleton of expressions without constants by decoders.
Revised: They all approach the symbolic regression problem as a machine translation problem, mapping the input data to latent representations via encoders, and then outputting the skeleton of expressions without constants by decoders.

Original: Finally, they all use the nonlinear optimization algorithm, e.g. BFGS (Fletcher, 1984) to recover the values of the constants, by minimizing the squared loss between the original outputs and the output from the predicted equations.

Original: These transformer-based methods (Valipour et al., 2021; Biggio et al., 2021) for symbolic regression exists two main drawbacks: (i)
Revised: These transformer-based methods (Valipour et al., 2021; Biggio et al., 2021) for symbolic regression exists two main drawbacks: (i)

Original: A natural question is what architecture of the encoder is optimally suited for symbolic regression.
Revised: A natural question is what architecture of the encoder is optimally suited for symbolic regression.

Original: It is clear that the decoder’s ability to sample expressions efficiently is severely constrained by the encoder’s ability to extract the features of the input data.
Revised: It is clear that the decoder’s ability to sample expressions efficiently is severely constrained by the encoder’s ability to extract the features of the input data.

Original: The idea is that the encoder should not just encode the points, but also represent the expression on a high level such that the decoder only prints the representation as a sequence of symbols.
Revised: The idea is that the encoder should not just encode the points, but also represent the expression on a high level such that the decoder only prints the representation as a sequence of symbols.

Original: (ii)
Revised: (ii)

Original: They use the single character of the expression’s string (Valipour et al., 2021) and the pre-order traversal of the expression tree (Biggio et al., 2021) as supervision information, respectively, which is an ill-posed problem that does not provide sufficient supervision: different instances of the same skeleton can have very different shapes, and instances of very different skeletons can be very close.
Revised: They use the single character of the expression’s string (Valipour et al., 2021) and the pre-order traversal of the expression tree (Biggio et al., 2021) as supervision information, respectively, which is an ill-posed problem that does not provide sufficient supervision: different instances of the same skeleton can have very different shapes, and instances of very different skeletons can be very close.

Original: To alleviate these issues, we proposed a transformer-based method for symbolic regression using a new feature extractor and a joint supervised learning mechanism.
Revised: To alleviate these issues, we proposed a transformerbased method for symbolic regression using a new feature extractor and a joint supervised learning mechanism.

Original: In summary, we introduce the main contributions in this study as follows:
Revised: In summary, we introduce the main contributions in this study as follows:

Original: • We leverage a pure residual MLP feature extractor for extracting the local and global features of observed data targeting symbolic regression tasks, which aids the expression generator in producing more correct expression skeletons.
Revised: • We leverage a pure residual MLP feature extractor for extracting the local and global features of observed data targeting symbolic regression tasks, which aids the expression generator in producing more correct expression skeletons.

Original: • We propose a joint learning mechanism combining supervised contrastive learning that combines the supervision of the whole expression skeleton with the supervision of the preorder traversal of its expression tree, which alleviates the ill-posed problem effectively.
Revised: • We propose a joint learning mechanism combining supervised contrastive learning that combines the supervision of the whole expression skeleton with the supervision of the preorder traversal of its expression tree, which alleviates the ill-posed problem effectively.

Original: • Empirically, the proposed method is up to 25% better than recent transformer-based methods with respect to the recovery rate of expression skeleton.
Revised: • Empirically, the proposed method is up to 25% better than recent transformer-based methods with respect to the recovery rate of expression skeleton.

Original: Moreover, our method outperforms several strong baseline methods in terms of R 2 .
Revised: Moreover, our method outperforms several strong baseline methods in terms of R 2 .

Original: • The source code and the model checkpoints are publicly available.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Genetic Programming (GP) for Symbolic Regression.
Revised: Genetic programming (GP) for symbolic regression.

Original: Traditionally, the approaches to symbolic regression are based on genetic algorithms (Forrest, 1993).
Revised: Traditionally, the approaches to symbolic regression are based on genetic algorithms (Forrest, 1993).

Original: Later, the symbolic regression task is seen as an optimization problem for the search space (Koza, 1994).
Revised: Later, the symbolic regression task is seen as an optimization problem for the search space (Koza, 1994).

Original: By far the most popular commercial software Eureqa (Dubˇc´akov´a, 2011) is the most successful application based on GP methods.
Revised: By far the most popular commercial software Eureqa (Dubˇc´akov´a, 2011) is the most successful application based on GP methods.

Original: A limitation of the genetic algorithms-based methods is that they need to train for each equation from scratch, which is slow, computationally expensive and highly randomized.
Revised: A limitation of the genetic algorithms-based methods is that they need to train for each equation from scratch, which is slow, computationally expensive and highly randomized.

Original: The models tend to generate more complex equations and they are sensitive to the choice of hyperparameters (Petersen et al., 2021).
Revised: The models tend to generate more complex equations and they are sensitive to the choice of hyperparameters (Petersen et al., 2021).

Original: Neural Network for Symbolic Regression Symbolic regression based on neural network approaches can be broadly classified into three categories.
Revised: Neural network for symbolic regression. Symbolic regression based on neural network approaches can be broadly classified into three categories.

Original: First, the methods based on equation learner (EQL) (Martius & Lampert, 2016; Sahoo et al., 2018; Werner et al., 2021) are trained by replacing the activation function of the neural network with arithmetic operators, which inherits the ability of neural networks to deal with high-dimensional data and scales well with the number of input-output pairs (Biggio et al., 2021).
Revised: First, the methods based on equation learner (EQL) (Martius & Lampert, 2016; Sahoo et al., 2018; Werner et al., 2021) are trained by replacing the activation function of the neural network with arithmetic operators, which inherits the ability of neural networks to deal with high-dimensional data and scales well with the number of input-output pairs (Biggio et al., 2021).

Original: Nevertheless, the existence of exponential and logarithmic activationfunctions leads to gradient instability.
Revised: Nevertheless, the existence of exponential and logarithmic activation

Original: Also, the complexity of predicted expression depends on the depth of the EQL network. Reinforcement learning for symbolic regression. The second approach is the autoregressive model based on reinforcement learning.
Revised: [Figure]

Original: Petersen et al.
Revised: Max [Figure]

Original: (2021) uses reinforcement learning based on a risk-seeking policy gradient to train a RNN to generate a probability distribution over the
Revised: Residual

Original: [Figure]
Revised: [Figure]

Original: Embedding
Revised: Residual Point Block ··· ··· ×𝑁

Original: [Figure]
Revised: [Figure]

Original: Embedding
Revised: ×𝑁 ! Geometric Affine Module 𝑓(𝑥,𝑦) = 2𝑥 "

Original: [Figure]
Revised: [Figure]

Original: Add
Revised: Points Feature Vector Positional Embedding Token

Original: [Figure]
Revised: Embedding Masked Multi-Head Attention

Original:  Norm[Figure]
Revised: Add & Norm

Original:  Add [Figure] Norm
Revised: Feed Forward Add & Norm

Original: Softmax (1 − 𝜆)ℒ !"
Revised: Softmax (1 − 𝜆)ℒ !"

Original: #$$%&’("#)*
Revised: #$$%&’("#)*

Original: [Figure] [Figure] [Figure] [Figure]
Revised: 𝑋 #

Original: [Figure]
Revised: [Figure]

Original: ×𝑁
Revised: 𝜆ℒ !

Revised: #’("+$(,-& +

Revised: Pre-order of expression tree ×𝑁 &

Original: [Figure]
Revised: [Figure]

Original:  space of mathematical expressions.
Revised: Reinforcement learning for symbolic regression. The second approach is the autoregressive model based on reinforcement learning. Petersen et al. (2021) uses reinforcement learning based on a risk-seeking policy gradient to train a RNN to generate a probability distribution over the space of mathematical expressions.

Original: For such symbolic regression tasks, they proposed a new objective function based on risk-seeking policy gradients, that focuses on learning only on maximizing best-case performance rather than the average performance of a policy.
Revised: For such symbolic regression tasks, they proposed a new objective function based on risk-seeking policy gradients, that focuses on learning only on maximizing best-case performance rather than the average performance of a policy.

Original: Genetic programming and neural-guided search are mechanistically dissimilar, yet both have proven to be effective solutions to symbolic regression.
Revised: Genetic programming and neural-guided search are mechanistically dissimilar, yet both have proven to be effective solutions to symbolic regression.

Original: Mundhenk et al.
Revised: Mundhenk et al.

Original: (2021) proposed a more novel approach, combining the two approaches to leverage each of their strengths.
Revised: (2021) proposed a more novel approach, combining the two approaches to leverage each of their strengths.

Original: They take the output from the RNN as an initial population for a genetic algorithm.
Revised: They take the output from the RNN as an initial population for a genetic algorithm.

Original: The method represents a significant step forward in the application of deep learning to symbolic regression (Biggio et al., 2021).
Revised: The method represents a significant step forward in the application of deep learning to symbolic regression (Biggio et al., 2021).

Original: The promising results make it the currently recognized state-of-the-art approach to symbolic regression tasks.
Revised: The promising results make it the currently recognized state-of-the-art approach to symbolic regression tasks.

Original: Nevertheless, the limitations of this method are obvious, namely, the network has to be retrained from scratch for each new equation and the RNN is never directly conditioned on the data it is trained to model (Biggio et al., 2021).
Revised: Nevertheless, the limitations of this method are obvious, namely, the network has to be retrained from scratch for each new equation and the RNN is never directly conditioned on the data it is trained to model (Biggio et al., 2021).

Original: Large scale transformer-based models for symbolic regression.
Revised: Large scale transformer-based models for symbolic regression.

Original: The third approach is to train a large scale transformer-based model by using a large amount of data.
Revised: The third approach is to train a large scale transformer-based model by using a large amount of data.

Original: More recently, SymbolicGPT (Valipour et al., 2021) trained a GPT (Radford et al., 2019) model to construct a mapping of pairs of points and symbolic output.
Revised: More recently, SymbolicGPT (Valipour et al., 2021) trained a GPT (Radford et al., 2019) model to construct a mapping of pairs of points and symbolic output.

Original: They first input the data points into T-net (Qi et al., 2017) to get a potential representation of the data points and then input it to the GPT
Revised: They first input the data points into T-net (Qi et al., 2017) to get a potential representation of the data points and then input it to the GPT

Original: (Radford et al., 2019) for generating expression strings.
Revised: (Radford et al., 2019) for generating expression strings.

Original: They generate the expression at the character level and finally concatenate it into an expression.
Revised: They generate the expression at the character level and finally concatenate it into an expression.

Original: In general, they explore an alternative approach to symbolic regression by considering it as a task in language modeling.
Revised: In general, they explore an alternative approach to symbolic regression by considering it as a task in language modeling.

Original: Symbolic mathematics behaves as a language in its own right, with well-formed mathematical expressions treated as valid “sentences” in this language (Valipour et al., 2021).
Revised: Symbolic mathematics behaves as a language in its own right, with well-formed mathematical expressions treated as valid “sentences” in this language (Valipour et al., 2021).

Original: Furthermore, NeSymReS (Biggio et al., 2021) proposed a similar method, where they use the encoder from the Set transformer (Lee et al., 2019) and the decoder from the original transformer architecture (Vaswani et al., 2017).
Revised: Furthermore, NeSymReS (Biggio et al., 2021) proposed a similar method, where they use the encoder from the Set transformer (Lee et al., 2019) and the decoder from the original transformer architecture (Vaswani et al., 2017).

Original: Their greatest contribution is to show that their approach is able to improve performance as the size of the dataset increases.
Revised: Their greatest contribution is to show that their approach is able to improve performance as the size of the dataset increases.

Original: Different from SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021), we explore the feature extraction of data points and leverage the pure residual MLP network as the feature extractor.

Original: More importantly, we train a transformer-based model with the joint learning mechanism combining supervised contrastive learning.

Original: It integrates the supervision of the whole expression skeleton with the supervision of the pre-order traversal of its expression tree.

Original:  M ETHODS
Revised:  M ETHODS

Original: The proposed joint learning mechanism is shown in Figure 1.
Revised: The proposed joint learning mechanism is shown in Figure 1.

Original: First, we leverage a permutationinvariant feature extractor based on residual MLP networks to obtain feature vectors of data points.
Revised: First, we leverage a permutationinvariant feature extractor based on residual MLP networks to obtain feature vectors of data points.

Original: Then, given the feature vectors, the expression generator autoregressively generates individual math- ematical symbols until we obtain the entire skeleton of the expression.
Revised: Then, given the feature vectors, the expression generator autoregressively generates individual math- ematical symbols until we obtain the entire skeleton of the expression.

Original: In the forward propagation, we compute the contrastive loss with respect to the expression skeleton categories and the crossentropy loss with respect to the mathematical symbol categories separately.
Revised: In the forward propagation, we compute the contrastive loss with respect to the expression skeleton categories and the crossentropy loss with respect to the mathematical symbol categories separately.

Original: The parameters of the network are jointly updated in backpropagation.
Revised: The parameters of the network are jointly updated in backpropagation.

Original: Training can efficiently process each sequence in a single forward pass of the network thanks to the masked attention and teacher forcing (Vaswani et al., 2017).
Revised: Training can efficiently process each sequence in a single forward pass of the network thanks to the masked attention and teacher forcing (Vaswani et al., 2017).

Original: During inference, multiple predictions are sampled from the model using the beam search strategy and we select the prediction with the lowest error.
Revised: During inference, multiple predictions are sampled from the model using the beam search strategy and we select the prediction with the lowest error.

Original: 3. E XTRACTING EFFECTIVE FEATURE OF DATA POINTS
Revised: 3. E XTRACTING EFFECTIVE FEATURE OF DATA POINTS

Original: As mentioned in section 1, the feature extracted from data points affects generating expressions through the decoder.
Revised: As mentioned in section 1, the feature extracted from data points affects generating expressions through the decoder.

Original: Valipour et al.
Revised: Valipour et al.

Original: (2021) obtain the latent representation of data using the Tnet (Qi et al., 2017).
Revised: (2021) obtain the latent representation of data using the Tnet (Qi et al., 2017).

Original: Albeit efficient, local feature loss caused by non-locality and non-hierarchy degrades the representational quality of details for point cloud (Ma et al., 2022).
Revised: Albeit efficient, local feature loss caused by non-locality and non-hierarchy degrades the representational quality of details for point cloud (Ma et al., 2022).

Original: As a similar work,
Revised: As a similar work,

Original: NeSymReS (Biggio et al., 2021) use the Set Transformer (Lee et al., 2019) based on self-attention to extract data points features.
Revised: NeSymReS (Biggio et al., 2021) use the Set Transformer (Lee et al., 2019) based on self-attention to extract data points features.

Original: However, it focuses too much on local feature extraction and lacks global information, which is not suitable for expression data points.
Revised: However, it focuses too much on local feature extraction and lacks global information, which is not suitable for expression data points.

Original: We explore and visualize the similarity of the features extracted from data points by these methods and compare the expression skeletons generated by the decoder, which confirms the above problems.
Revised: We explore and visualize the similarity of the features extracted from data points by these methods and compare the expression skeletons generated by the decoder, which confirms the above problems.

Original: The result is shown in section 4.
Revised: The result is shown in section 4.

Original: The performance of the feature extractor has to be considered, therefore, we opt for the framework of PointMLP (Ma et al., 2022) based on pure residual MLP as our feature extractor in order to obtain the local and overall information of the data points.
Revised: The performance of the feature extractor has to be considered, therefore, we opt for the framework of PointMLP (Ma et al., 2022) based on pure residual MLP as our feature extractor in order to obtain the local and overall information of the data points.

Original: Given a set of data points D
Revised: Given a set of data points D

Original: = { ( x i , y i ) } ni =1 ∈ R n × ( d +1) , where n indicates the number of points and d denotes the dimension of the variable.
Revised: = { ( x i , y i ) } ni =1 ∈ R n × ( d +1) , where n indicates the number of points and d denotes the dimension of the variable.

Original: PointMLP learns hierarchical features of data points by stacking multiple learning stages.
Revised: PointMLP learns hierarchical features of data points by stacking multiple learning stages.

Original: In each stage, N s points are re-sampled by the farthest point sampling (FPS) algorithm, where s indexes the stage and K neighbors are employed for each sampled point (Ma et al., 2022).
Revised: In each stage, N s points are re-sampled by the farthest point sampling (FPS) algorithm, where s indexes the stage and K neighbors are employed for each sampled point (Ma et al., 2022).

Original: Conceptually, the kernel operation of PointMLP can be formulated as:
Revised: Conceptually, the kernel operation of PointMLP can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where f i,j is the j -th neighbor point feature of i -th sampled point.
Revised: where f i,j is the j -th neighbor point feature of i -th sampled point.

Original: POS ( · ) and PRE ( · ) are residual point MLP blocks: the shared PRE ( · ) is designed to learn shared weights from a local region while the POS ( · ) is leveraged to extract deep aggregated features.
Revised: POS ( · ) and PRE ( · ) are residual point MLP blocks: the shared PRE ( · ) is designed to learn shared weights from a local region while the POS ( · ) is leveraged to extract deep aggregated features.

Original: POS ( · ) and PRE ( · ) consist of several residual MLP blocks:
Revised: POS ( · ) and PRE ( · ) consist of several residual MLP blocks:

Original: MLP ( x ) + x .
Revised: MLP ( x ) + x .

Original: We use the max pooling layer to aggregate global features.
Revised: We use the max pooling layer to aggregate global features.

Original: After MLP blocks, we add a dropout layer (Srivastava et al., 2014).
Revised: After MLP blocks, we add a dropout layer (Srivastava et al., 2014).

Original: Following (Ma et al., 2022), we also leverage a lightweight geometric affine module to tackle the problem that is less robust and caused by the sparse and irregular geometric structures in local regions.
Revised: Following (Ma et al., 2022), we also leverage a lightweight geometric affine module to tackle the problem that is less robust and caused by the sparse and irregular geometric structures in local regions.

Original: Let { f i,j } j =1 , ··· ,
Revised: Let { f i,j } j =1 , ··· ,

Original: k ∈ R k × d be the grouped local neighbors of f i ∈ R d containing k points, and each neighbor point f i,j is a d -dimensional vector.
Revised: k ∈ R k × d be the grouped local neighbors of f i ∈ R d containing k points, and each neighbor point f i,j is a d -dimensional vector.

Original: We transform the local neighbor points by the following formulation:
Revised: We transform the local neighbor points by the following formulation:

Original: [Equation]
Revised: [Equation]

Original: where α ∈ R d and β ∈ R
Revised: where α ∈ R d and β ∈ R

Original: d are learnable parameters, ⊙ indicates Hadamard production, and ϵ = 1 e − 5 is a small number for numerical stability.
Revised: d are learnable parameters, ⊙ indicates Hadamard production, and ϵ = 1 e − 5 is a small number for numerical stability.

Original: Note that σ is a scalar that describes the feature deviation across all local groups and channels.
Revised: Note that σ is a scalar that describes the feature deviation across all local groups and channels.

Original: By doing so, we transform the local points to a normal distribution while maintaining original geometric properties.
Revised: By doing so, we transform the local points to a normal distribution while maintaining original geometric properties.

Original: 3. T RAINING WITH JOINT SUPERVISION INFORMATION
Revised: 3. T RAINING WITH JOINT SUPERVISION INFORMATION

Original: Recent transformer-based face an ill-posed problem because use insufficient supervision information: they use the single character of the expression’s string (Valipour et al., 2021) and the pre-order traversal of the expression (Biggio et al., 2021) as supervision information, respectively, and train the model by minimizing the cross-entropy loss.
Revised: Recent transformer-based face an ill-posed problem because use insufficient supervision information: they use the single character of the expression’s string (Valipour et al., 2021) and the pre-order traversal of the expression (Biggio et al., 2021) as supervision information, respectively, and train the model by minimizing the cross-entropy loss.

Original: In this work, we propose a joint objective function that combines cross-entropy (CE) loss and supervised contrastive learning (CL) loss.
Revised: In this work, we propose a joint objective function that combines cross-entropy (CE) loss and supervised contrastive learning (CL) loss.

Original: On the basis of using symbol labels as supervision, we treat the skeleton of expressions as category labels to enrich the supervisory information.
Revised: On the basis of using symbol labels as supervision, we treat the skeleton of expressions as category labels to enrich the supervisory information.

Original: The CL loss with respect to the expression skeleton categories and the
Revised: The CL loss with respect to the expression skeleton categories and the

Original: CE loss with respect to the mathematical symbol categories are separately calculated in the forward propagation.
Revised: CE loss with respect to the mathematical symbol categories are separately calculated in the forward propagation.

Original: In backpropagation, the network’s parameters are concurrently updated.
Revised: In backpropagation, the network’s parameters are concurrently updated.

Original: As the auxiliary loss, the CL loss is meant to capture the similarities of feature vectors between expressions with the same skeletons and contrast them with others.
Revised: As the auxiliary loss, the CL loss is meant to capture the similarities of feature vectors between expressions with the same skeletons and contrast them with others.

Original: The promising results are described in section 4.
Revised: The promising results are described in section 4.

Original: The overall loss is a weighted average of CE loss and proposed CL loss, as given in equation (1).
Revised: The overall loss is a weighted average of CE loss and proposed CL loss, as given in equation (1).

Original: The canonical definition of the CE loss that we use is given in equation (2).
Revised: The canonical definition of the CE loss that we use is given in equation (2).

Original: The noval CL loss is given in equation (3).
Revised: The novel CL loss is given in equation (3).

Original: The overall loss is then given in the following:
Revised: The overall loss is then given in the following:

Original: [Equation]
Revised: [Equation]

Original: Here, λ is a scalar weighting hyperparameter that we tune for the training stage; N represents the mini-batch size; C denotes the size of the pre-specified tokens library; y i,c denotes the symbol label and ˆ y i,c denotes the expression generator output for the probability of the ith token belonging to the class c ; N l i is the total number of examples in the batch that have the same skeleton label as ℓ i ; ϵ is a very small scalar preventing devision by zero; 1 i ̸
Revised: Here, λ is a scalar weighting hyperparameter that we tune for the training stage; N represents the mini-batch size; C denotes the size of the pre-specified tokens library; y i,c denotes the symbol label and ˆ y i,c denotes the expression generator output for the probability of the ith token belonging to the class c ; N l i is the total number of examples in the batch that have the same skeleton label as ℓ i ; ϵ is a very small scalar preventing devision by zero; 1 i ̸

Original: = j , 1 ℓ i = ℓ j and 1 ℓ i ̸ =
Revised: = j , 1 ℓ i = ℓ j and 1 ℓ i ̸ =

Original: ℓ k are similar indicator functions; s i,j = v i · v j / ∥ v i ∥
Revised: ℓ k are similar indicator functions; s i,j = v i · v j / ∥ v i ∥

Original: ∥ v j ∥ denotes the cosine similarity between the sample i and the sample j , where v i and v j represent the high-level feature vectors of the sample i and the sample j respectively from the feature extractor; τ > 0 is an adjustable scalar temperature parameter that controls the separation of classes.
Revised: ∥ v j ∥ denotes the cosine similarity between the sample i and the sample j , where v i and v j represent the high-level feature vectors of the sample i and the sample j respectively from the feature extractor; τ > 0 is an adjustable scalar temperature parameter that controls the separation of classes.

Original: 3. G ENERATING EXPRESSIONS WITH A TRANSFORMER - BASED MODEL
Revised: 3. G ENERATING EXPRESSIONS WITH A TRANSFORMER - BASED MODEL

Original: We leverage a framework of the GPT language model (Radford et al., 2019) as the expression generator, which is an autoregressive language model based on the decoder of transformer (Vaswani et al., 2017).
Revised: We leverage a framework of the GPT language model (Radford et al., 2019) as the expression generator, which is an autoregressive language model based on the decoder of transformer (Vaswani et al., 2017).

Original: During inference, we generate an expression τ one token at a time along the pre-order traversal.
Revised: During inference, we generate an expression τ one token at a time along the pre-order traversal.

Original: For example, the expression f ( x 1 , x 2 ) =
Revised: For example, the expression f ( x 1 , x 2 ) =

Original: sin ( x 1 ) + log ( x 2 ) is encoded as [+ , sin , x 1 , log , x 2 ] .
Revised: sin ( x 1 ) + log ( x 2 ) is encoded as [+ , sin , x 1 , log , x 2 ] .

Original: We denote the i th token of the traversal as τ i and each token has a value within a given library L of possible tokens, e.g. { + , − , × , ÷ , sin , cos , log , x 1 , x 2 } .
Revised: We denote the i th token of the traversal as τ i and each token has a value within a given library L of possible tokens, e.g., { + , − , × , ÷ , sin , cos , log , x 1 , x 2 } .

Original: Specifically, the i th output of the generator with parameters θ and feature vector v extracted from data D pass through a softmax layer to produce vector ψ ( i ) , which defines the probability distribution for selecting the i th token τ i .
Revised: Specifically, the i th output of the generator with parameters θ and feature vector v extracted from data D pass through a softmax layer to produce vector ψ ( i ) , which defines the probability distribution for selecting the i th token τ i .

Original: The likelihood of the entire generated expression is simply the product of the likelihoods of its tokens: p ( τ | θ, v ) =
Revised: The likelihood of the entire generated expression is simply the product of the likelihoods of its tokens: p ( τ | θ, v ) =

Original: (cid:81) | τ | i =1 p ( τ i | τ 1:( i − 1) ; θ, v ) =
Revised: (cid:81) | τ | i =1 p ( τ i | τ 1:( i − 1) ; θ, v ) =

Original: (cid:81) | τ | i =1 ψ ( i ) L ( τ i ) .
Revised: (cid:81) | τ | i =1 ψ ( i ) L ( τ i ) .

Original: Note that the output sequence from the generator does not contain any numerical constants.
Revised: Note that the output sequence from the generator does not contain any numerical constants.

Original: Instead, we use a special placeholder ⟨ C ⟩ denoting the presence of a constant that will be optimized at a later stage.
Revised: Instead, we use a special placeholder ⟨ C ⟩ denoting the presence of a constant that will be optimized at a later stage.

Original: 3. L EARNING C ONSTANTS USING BFGS BY RESTARTING MULTIPLE TIMES
Revised: 3. L EARNING C ONSTANTS USING BFGS BY RESTARTING MULTIPLE TIMES

Original: At inference time, Mundhenk et al.
Revised: At inference time, Mundhenk et al.

Original: ; Valipour et al.
Revised: ; Valipour et al.

Original: ; Biggio et al.
Revised: ; Biggio et al.

Original: all use BFGS optimization algorithm
Revised: all use BFGS optimization algorithm

Original: (Fletcher, 1984) on the mean squared error to fit the constants.
Revised: (Fletcher, 1984) on the mean squared error to fit the constants.

Original: BFGS is a quasi-newton method for solving unconstrained nonlinear convex optimization problems.
Revised: BFGS is a quasi-newton method for solving unconstrained nonlinear convex optimization problems.

Original: However, for the symbolic regression task, the loss function minimized by BFGS may be highly non-convex, which is likely to be falling into several local minima: even when the skeleton is perfectly predicted, the correct constants are not guaranteed to be found.
Revised: However, for the symbolic regression task, the loss function minimized by BFGS may be highly non-convex, which is likely to be falling into several local minima: even when the skeleton is perfectly predicted, the correct constants are not guaranteed to be found.

Original: To ameliorate this issue, we use a simple and hardly any space cost method: restart the BFGS algorithm by initializing the different starting points multiple times, and the global optimal point is achieved as much as possible.
Revised: To ameliorate this issue, we use a simple and hardly any space cost method: restart the BFGS algorithm by initializing the different starting points multiple times, and the global optimal point is achieved as much as possible.

Original: The result is shown in section 4.
Revised: The result is shown in section 4.

Original:  E XPERIMENTS AND R ESULTS 4. G ENERATING DATASETS
Revised:  E XPERIMENTS AND R ESULTS 4. G ENERATING DATASETS

Original: We generate mathematical expressions following the framework as described by (Lample & Charton, 2019).
Revised: We generate mathematical expressions following the framework as described by (Lample & Charton, 2019).

Original: The framework starts by generating a random unary-binary tree and filling the nodes with operators and the leaves with independent variables or constants.
Revised: The framework starts by generating a random unary-binary tree and filling the nodes with operators and the leaves with independent variables or constants.

Original: The unnormalized probabilities of each operation and operator are given in Appendix A.
Revised: The unnormalized probabilities of each operation and operator are given in Appendix A.

Original: We generate the training set containing approximately 100 , 000 unique expression skeletons.
Revised: We generate the training set containing approximately 100 , 000 unique expression skeletons.

Original: For each expression, we re-sample its constant values for 10 , 20 , 30 , 40  and 50 times.
Revised: For each expression, we re-sample its constant values for 10 , 20 , 30 , 40 , and 50 times.

Original: Once an expression tree has been generated, we can represent the expression using a pre-order traversal of the tree.
Revised: Once an expression tree has been generated, we can represent the expression using a pre-order traversal of the tree.

Original: We opt for scalar functions of up to two independent variables (i.e. d = 2 and y = f
Revised: We opt for scalar functions of up to two independent variables (i.e., d = 2 and y = f

Original: ( x 1 , x 2 ) ) and three numerical constants each.
Revised: ( x 1 , x 2 ) ) and three numerical constants each.

Original: Specifically, each generated expression’s constant values are replaced with the constant placeholder ⟨ C ⟩ .
Revised: Specifically, each generated expression’s constant values are replaced with the constant placeholder ⟨ C ⟩ .

Original: Then, the additive and multiplicative constant values are sampled uniformly from the interval [ − 2 , 2] to fill in the placeholder.
Revised: Then, the additive and multiplicative constant values are sampled uniformly from the interval [ − 2 , 2] to fill in the placeholder.

Original: After this, the entire equation will be simplified using the rules built in the symbolic manipulation library SymPy (Meurer et al., 2017).
Revised: After this, the entire equation will be simplified using the rules built in the symbolic manipulation library SymPy (Meurer et al., 2017).

Original: Finally, we sample uniformly at random 100 input points X
Revised: Finally, we sample uniformly at random 100 input points X

Original: = { x i } ni
Revised: = { x i } ni

Original: =1 from the interval [ − 10 , 10] and evaluate the expressions on the X to obtain the corresponding outputs Y .
Revised: =1 from the interval [ − 10 , 10] and evaluate the expressions on the X to obtain the corresponding outputs Y .

Original: X will be re-sampled if produce non-finite values (NaN or ±∞ ) and we discard the expression that cannot be sampled completely in 30 seconds.
Revised: X will be re-sampled if produce non-finite values (NaN or ±∞ ) and we discard the expression that cannot be sampled completely in 30 seconds.

Original: Creating the SSDNC benchmark.
Revised: Creating the SSDNC benchmark.

Original: We generate a more challenging test set to discover the performance of transformer-based methods, which includes approximately 100 unique expression skeletons and 10 re-sampled numerical constants for each skeleton.
Revised: We generate a more challenging test set to discover the performance of transformer-based methods, which includes approximately 100 unique expression skeletons and 10 re-sampled numerical constants for each skeleton.

Original: We sample random support ofpoints from the uniform distribution U ( − 10 , 10) , for each independent variable.
Revised: We sample random support ofpoints from the uniform distribution U ( − 10 , 10) , for each independent variable.

Original: We call it SSDNC, for the same skeletons with different numerical coefficients.
Revised: We call it SSDNC, for the same skeletons with different numerical coefficients.

Original: 4. T RAINING AND I NFERENCE
Revised: 4. T RAINING AND I NFERENCE

Original: We train the feature extractor and expression generator jointly to minimize the objective loss combined with cross-entropy loss and supervised contrastive loss.
Revised: We train the feature extractor and expression generator jointly to minimize the objective loss combined with cross-entropy loss and supervised contrastive loss.

Original: More specifically, we train the model using the Adam optimizer (Kingma & Ba, 2014) on 4 NVIDIA V100 GPUs.
Revised: More specifically, we train the model using the Adam optimizer (Kingma & Ba, 2014) on 4 NVIDIA V100 GPUs.

Original: More detailed hyperparameters are reported in Appendix 5, which were found empirically and not fine-tuned for maximum performance.
Revised: More detailed hyperparameters are reported in Appendix 5, which were found empirically and not fine-tuned for maximum performance.

Original: Note that we use the same dataset for training to facilitate quantitative benchmarking when evaluating the feature extraction and other capabilities with SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021).
Revised: Note that we use the same dataset for training to facilitate quantitative benchmarking when evaluating the feature extraction and other capabilities with SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021).

Original: At inference, most of the expressions of benchmark testsets are not seen during training, and we resampled all data points to avoid  possible overfitting problem.
Revised: At inference, most of the expressions of benchmark test sets are not seen during training, and we resampled all data points to avoid the possible overfitting problem.

Original: 4. M ETRICS
Revised: 4. M ETRICS

Original: We have selected the coefficient of determination ( R 2 ) to assess the quality of our method.
Revised: We have selected the coefficient of determination ( R 2 ) to assess the quality of our method.

Original: The R(Glantz & Slinker, 2001) is defined as follows:
Revised: The R(Glantz & Slinker, 2001) is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where y i and ˆ y
Revised: where y i and ˆ y

Original: i are the ground-truth and predicted values for point i , respectively.
Revised: i are the ground-truth and predicted values for point i , respectively.

Original: ¯ y is the average of y i over all the points.
Revised: ¯ y is the average of y i over all the points.

Original: k is the number of test points.
Revised: k is the number of test points.

Original: The advantage of using R 2 is its nice interpretation.
Revised: The advantage of using R 2 is its nice interpretation.

Original: if R 2 > 0 , then means the prediction is better than predicting just the average value, and if R 2 = 1 , then we get a perfect prediction.
Revised: if R 2 > 0 , then means the prediction is better than predicting just the average value, and if R 2 = 1 , then we get a perfect prediction.

Original: However, due to the presence of the max operator,
Revised: However, due to the presence of the max operator,

Original: R 2 is sensitive to outliers, and hence to the number of points considered at test time (more points entail a higher risk of an outlier).
Revised: R 2 is sensitive to outliers, and hence to the number of points considered at test time (more points entail a higher risk of an outlier).

Original: To circumvent this, we discard the 5% worst predictions for all methods used, following (Biggio et al., 2021).
Revised: To circumvent this, we discard the 5% worst predictions for all methods used, following (Biggio et al., 2021).

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We compare the performance of our method with four strong symbolic regression baselines:
Revised: We compare the performance of our method with four strong symbolic regression baselines:

Original: • SymbolicGPT
Revised: • SymbolicGPT

Original: (Valipour et al., 2021)
Revised: (Valipour et al., 2021)

Original: A recent novel transformer-based language model for symbolic regression.
Revised: A recent novel transformer-based language model for symbolic regression.

Original: We use the open-source implementation provided by the authors 1 . • Neuaral Symbolic Regression that Scales (NeSymReS) (Biggio et al., 2021)
Revised: We use the open-source implementation provided by the authors.• Neuaral Symbolic Regression that Scales (NeSymReS) (Biggio et al., 2021)

Original: Recently proposed transformer-based symbolic regression model on the large training data.
Revised: Recently proposed transformer-based symbolic regression model on the large training data.

Original: We use the open-source implementation provided by the authors 2 . 1 https://github.com/mojivalipour/symbolicgpthttps://
Revised: We use the open-source implementation provided by the authors.1 https://github.com/mojivalipour/symbolicgpthttps:

Original: 0 5 10 15 20 25 30 35 40 45 Expression id 

Revised: 0 5 10 15 20 25 30 35 40 45 Expression id

Original: E xp r e ss i on i d
Revised: E xp r e ss i on i d

Original: (a) SymbolicGPT 0 5 10 15 20 25 30 35 40 45 Expression id 

Revised: (a) SymbolicGPT 0 5 10 15 20 25 30 35 40 45 Expression id

Original: E xp r e ss i on i d (b) NeSymReS 0 5 10 15 20 25 30 35 40 45 Expression id 

Revised: E xp r e ss i on i d (b) NeSymReS 0 5 10 15 20 25 30 35 40 45 Expression id

Original: E xp r e ss i on i d (c) Ours w/o CL 

Revised: E xp r e ss i on i d (c) Ours w/o CL

Original: 5 10 15 20 25 30 35 40 45 Expression id 

Revised: 5 10 15 20 25 30 35 40 45 Expression id

Original: E xp r e ss i on i d (d) Ours w/ CL
Revised: E xp r e ss i on i d (d) Ours w/ CL

Original: • Deep Symbolic Optimization (DSO).
Revised: • Deep Symbolic Optimization (DSO).

Original: A symbolic regression method based on RNN and reinforcement learning search strategy (Mundhenk et al., 2021).
Revised: A symbolic regression method based on RNN and reinforcement learning search strategy (Mundhenk et al., 2021).

Original: We use the open-source implementation provided by the authors 3 . • Genetic Programming.
Revised: We use the open-source implementation provided by the authors.• Genetic Programming.

Original: Standard GP-based symbolic regression (Koza, 1994) based on the open-source Python library gplearn4 .
Revised: Standard GP-based symbolic regression (Koza, 1994) based on the open-source Python library gplearn .

Original: All details for baselines are reported in Appendix D.
Revised: All details for baselines are reported in Appendix D.

Original: 4. F EATURE E XTRACTION P ERFORMANCE
Revised: 4. F EATURE E XTRACTION P ERFORMANCE

Original: For transformer-based methods in symbolic regression, we empirically demonstrate that the performance of the feature extractor plays a critical role in the overall training and evaluation stages.
Revised: For transformer-based methods in symbolic regression, we empirically demonstrate that the performance of the feature extractor plays a critical role in the overall training and evaluation stages.

Original: We evaluate the feature extraction performance of recent transformer-based methods, i.e. SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021), and our method with/without
Revised: We evaluate the feature extraction performance of recent transformer-based methods, i.e., SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021), and our method with/without

Original: CL on SSDNC benchmark.
Revised: CL on SSDNC benchmark.

Original: After training, we input the data points corresponding to the specific expression skeleton with different constants into the feature extractor, then we compute the cosine similarity between the different feature vectors.
Revised: After training, we input the data points corresponding to the specific expression skeleton with different constants into the feature extractor, then we compute the cosine similarity between the different feature vectors.

Original: As shown in Figure 2, SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021) all produce dissimilar feature vectors for the data points of different expressions belonging to the same skeleton.
Revised: As shown in Figure 2, SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021) all produce dissimilar feature vectors for the data points of different expressions belonging to the same skeleton.

Original: The reason is that the feature extractor they used focuses more on local features and loses key information, which is not appropriate for the data points of symbolic regression.
Revised: The reason is that the feature extractor they used focuses more on local features and loses key information, which is not appropriate for the data points of symbolic regression.

Original: By manual inspection, we find that this problem can adversely affect the expression generator to produce an expression skeleton.
Revised: By manual inspection, we find that this problem can adversely affect the expression generator to produce an expression skeleton.

Original: The results in section 4. illustrate the high correlation between the feature extractor and expression generating.
Revised: The results in section 4. illustrate the high correlation between the feature extractor and expression generating.

Original: Benefiting from using the architecture of pure residual MLP , our feature extractor is able to obtain more similar feature vectors when facing the same skeleton, even without CL.
Revised: Benefiting from using the architecture of pure residual MLP , our feature extractor is able to obtain more similar feature vectors when facing the same skeleton, even without CL.

Original: After training jointly, we make the feature vectors of data points from the same skeleton more similar, which effectively alleviates the ill-posed problem and improves the performance of generating expressions with the same skeletons.
Revised: After training jointly, we make the feature vectors of data points from the same skeleton more similar, which effectively alleviates the ill-posed problem and improves the performance of generating expressions with the same skeletons.

Original: 4. E VALUATION ON BENCHMARKS
Revised: 4. E VALUATION ON BENCHMARKS

Original: [Table]
Revised: [Table]

Original: Recovery rate of skeletons.
Revised: Recovery rate of skeletons.

Original: In inference, since transformer-based methods first generate expression skeletons based on features of data points, we first compare the performance of these methods in recovering expression skeletons.
Revised: In inference, since transformer-based methods first generate expression skeletons based on features of data points, we first compare the performance of these methods in recovering expression skeletons.

Original: we evaluate the recovery rate of expression skeletons on the SSDNC benchmark.
Revised: we evaluate the recovery rate of expression skeletons on the SSDNC benchmark.

Original: As shown in Table 1, our method with the more effective feature extractor and the joint leaning mechanism can better guide the generator in generating expressions, thus outperforming other transformer-based methods.
Revised: As shown in Table 1, our method with the more effective feature extractor and the joint leaning mechanism can better guide the generator in generating expressions, thus outperforming other transformer-based methods.

Original: Generating the right expression skeleton is crucial to the final
Revised: Generating the right expression skeleton is crucial to the final

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: result because optimizing constants is relatively simple.
Revised: result because optimizing constants is relatively simple.

Original: This is reflected in the comparison for Rin the following results.
Revised: This is reflected in the comparison for Rin the following results.

Original: Statistics of fitting accuracy We evaluate our method and current state-of-the-art approaches on the widely used public benchmarks, i.e. the Nguyen benchmark (Uy et al., 2011), Constant, Keijzer (Keijzer, 2003), R rationals (Krawiec & Pawlak, 2013), AI-Feynman database (Udrescu &
Revised: Statistics of fitting accuracy We evaluate our method and current state-of-the-art approaches on the widely used public benchmarks, i.e., the Nguyen benchmark (Uy et al., 2011), Constant, Keijzer (Keijzer, 2003), R rationals (Krawiec & Pawlak, 2013), AI-Feynman database (Udrescu &

Original: Tegmark, 2020) and our SSDNC test set.
Revised: Tegmark, 2020) and our SSDNC test set.

Original: Nguyen was the main benchmark used in Petersen et al.
Revised: Nguyen was the main benchmark used in Petersen et al.

Original: There are terms that appear in three ground truth equations that are not included in the set of equations that our model can fit, specifically x 6 and x y , but we can find approximate expressions.
Revised: There are terms that appear in three ground truth equations that are not included in the set of equations that our model can fit, specifically x 6 and x y , but we can find approximate expressions.

Original: AI-Feynman database is extracted from the popular Feynman Lectures on Physics series and contains expressions with up to nine variables.
Revised: AI-Feynman database is extracted from the popular Feynman Lectures on Physics series and contains expressions with up to nine variables.

Original: In our study, we consider all the expressions with up to two variables.
Revised: In our study, we consider all the expressions with up to two variables.

Original: The complete benchmark functions are given in Appendix 4.
Revised: The complete benchmark functions are given in Appendix 4.

Original: From the results in
Revised: From the results in

Original: Table 2, our method outperforms all baseline methods in terms of average R 2 on six benchmarks.
Revised: Table 2, our method outperforms all baseline methods in terms of average R 2 on six benchmarks.

Original: Performance under different training sets sizes.
Revised: Performance under different training sets sizes.

Original: Following subsection 4.1, we generate the different size of data set that contains the various number of expressions with different constants and the same skeletons.
Revised: Following subsection 4.1, we generate the different size of data set that contains the various number of expressions with different constants and the same skeletons.

Original: Since DSO (Mundhenk et al., 2021) and GP-based methods are trained from scratch for specific problems, they are not included in this comparison.
Revised: Since DSO (Mundhenk et al., 2021) and GP-based methods are trained from scratch for specific problems, they are not included in this comparison.

Original: We train our model, SymbolicGPT and NeSymReS on our training sets separately.
Revised: We train our model, SymbolicGPT and NeSymReS on our training sets separately.

Original: As shown in Figure 3, our method with/without CL all outperforms these two baseline methods in terms of R 2 on the SSDNC benchmark.
Revised: As shown in Figure 3, our method with/without CL all outperforms these two baseline methods in terms of R 2 on the SSDNC benchmark.

Original: SymbolicGPT and NeSymReS can also improve performance simply by increasing the size of the dataset, which shows that the data-driven approach for SR can continuously improve performance as the size of the dataset increases.
Revised: SymbolicGPT and NeSymReS can also improve performance simply by increasing the size of the dataset, which shows that the data-driven approach for SR can continuously improve performance as the size of the dataset increases.

Original: Performance under noisy data.
Revised: Performance under noisy data.

Original: We evaluated the robustness of our and baseline methods to noisy data by adding independent Gaussian noise to the dependent variable, with mean zero and standard deviation proportional to the root-mean-square of the dependent variable in the training data (Mundhenk et al., 2021).
Revised: We evaluated the robustness of our and baseline methods to noisy data by adding independent Gaussian noise to the dependent variable, with mean zero and standard deviation proportional to the root-mean-square of the dependent variable in the training data (Mundhenk et al., 2021).

Original: In Figure 4, we varied the proportionality constant from 0 (noiseless) to 10 − 1 , following (Mundhenk et al., 2021), and evaluated each algorithm across the SSDNC benchmark.
Revised: In Figure 4, we varied the proportionality constant from 0 (noiseless) to 10 − 1 , following (Mundhenk et al., 2021), and evaluated each algorithm across the SSDNC benchmark.

Original: [Figure]
Revised: [Figure]

Original: Our method with CL is competitive compared with DSO based on reinforcement learning, not overfitting the noise when adding a small amount of noise.
Revised: Our method with CL is competitive compared with DSO based on reinforcement learning, not overfitting the noise when adding a small amount of noise.

Original: Optimizing constants via restarting BFGS multiple times.
Revised: Optimizing constants via restarting BFGS multiple times.

Original: We try to reach the global optimum rather than the local optimum by restarting BFGS several times, each time using a different initialization.
Revised: We try to reach the global optimum rather than the local optimum by restarting BFGS several times, each time using a different initialization.

Original: We set the restart times to 5 , 10 , 15 and 20 . 0 means that the optimization only once.
Revised: We set the restart times to 5 , 10 , 15 and 20 . 0 means that the optimization only once.

Original: Figureshows that R 2 improves as the number of BFGS restarts increases.
Revised: Figureshows that R 2 improves as the number of BFGS restarts increases.

Original: We can conclude that multiple initializations can effectively jump out of local optimum in the constant optimization process so as to achieve better fitting accuracy.
Revised: We can conclude that multiple initializations can effectively jump out of local optimum in the constant optimization process so as to achieve better fitting accuracy.

Original: Finding mathematically equivalent expressions.
Revised: Finding mathematically equivalent expressions.

Original: By manual observation, we find that our method can generate more expressions with the same symbol as the target expression, which benefits from the high skeleton recovery rate.
Revised: By manual observation, we find that our method can generate more expressions with the same symbol as the target expression, which benefits from the high skeleton recovery rate.

Original: Interestingly, we also find that the model sometimes predicts a more simple expression compared to the ground truth with fairly high fitting accuracy.
Revised: Interestingly, we also find that the model sometimes predicts a more simple expression compared to the ground truth with fairly high fitting accuracy.

Original: For example, for the Keijzer-4 expression x 3 · exp( − x ) · cos( x ) · sin( x ) · (sin( x ) 2 · cos( x ) − 1) , our method predicts − 0 .
Revised: For example, for the Keijzer-4 expression x 3 · exp( − x ) · cos( x ) · sin( x ) · (sin( x ) 2 · cos( x ) − 1) , our method predicts − 0 .

Original: 634193499960418 x 3 · exp( − x ) · sin(1 .
Revised: 634193499960418 x 3 · exp( − x ) · sin(1 .

Original: 83246993155237 x +10 . 9750345372947) , which is simpler in the terms of skeleton and achieving a high R 2 through the constant optimization.
Revised: 83246993155237 x +10 . 9750345372947) , which is simpler in the terms of skeleton and achieving a high R 2 through the constant optimization.

Original: Additionally, the model can learn some transformation relations of trigonometric functions, e.g., for
Revised: Additionally, the model can learn some transformation relations of trigonometric functions, e.g., for

Original: Constant-7 expression 2 sin(1 .
Revised: Constant-7 expression 2 sin(1 .

Original: 3 x 1 ) · cos( x 2 ) , our method predicts 2 cos(1 .
Revised: 3 x 1 ) · cos( x 2 ) , our method predicts 2 cos(1 .

Original: 3 x 1 − π 2 ) · cos( x 2 ) , which implicates the transformation relation, i.e. sin ( x ) =
Revised: 3 x 1 − π 2 ) · cos( x 2 ) , which implicates the transformation relation, i.e., sin ( x ) =

Original: cos ( x − π 2 ) .
Revised: cos ( x − π 2 ) .

Original: Out-of domain performance.
Revised: Out-of domain performance.

Original: To test the out-of domain performance of our method, we first run the inference on the points sampled from the training range and then evaluate these predicted functions on points outside the sampling range.
Revised: To test the out-of domain performance of our method, we first run the inference on the points sampled from the training range and then evaluate these predicted functions on points outside the sampling range.

Original: In Figure 6, we visualized some of the model predictions about unary and binary expressions.
Revised: In Figure 6, we visualized some of the model predictions about unary and binary expressions.

Original: The experimental results show that our model can have good generalization performance outside the sampling domain when predicting more complex expressions, without overfitting the sampled data.
Revised: The experimental results show that our model can have good generalization performance outside the sampling domain when predicting more complex expressions, without overfitting the sampled data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a transformer-based method for symbolic regression using a new feature extractor and a joint supervised learning mechanism.
Revised: We propose a transformer-based method for symbolic regression using a new feature extractor and a joint supervised learning mechanism.

Original: Specifically, we leverage a pure residual MLP feature extractor to obtain more valuable features on input data points.
Revised: Specifically, we leverage a pure residual MLP feature extractor to obtain more valuable features on input data points.

Original: In order to fundamentally alleviate the ill-posed problem, we propose a joint supervised learning mechanism combining supervised contrastive learning, which strengthens the similarity of feature vectors from the same skeleton.
Revised: In order to fundamentally alleviate the ill-posed problem, we propose a joint supervised learning mechanism combining supervised contrastive learning, which strengthens the similarity of feature vectors from the same skeleton.

Original: The expression skeleton recovery rate of the proposed method is up to 25% higher than that of recent transformer-based methods.
Revised: The expression skeleton recovery rate of the proposed method is up to 25% higher than that of recent transformer-based methods.

Original: Evaluated on six benchmarks, the results show that our method outperforms current state-of-the-art methods based on reinforcement learning and genetic programming in terms of R 2 .
Revised: Evaluated on six benchmarks, the results show that our method outperforms current state-of-the-art methods based on reinforcement learning and genetic programming in terms of R 2 .

Original: It is worth noting that our method is competitive with the reinforcement learning-based method in terms of robustness.
Revised: It is worth noting that our method is competitive with the reinforcement learning-based method in terms of robustness.

Original: Finally, by evaluating the performance of our method outside the sampling range, we showed that it has good extrapolation capabilities.
Revised: Finally, by evaluating the performance of our method outside the sampling range, we showed that it has good extrapolation capabilities.

Revised: A CKNOWLEDGMENTS

Revised: This work was supported by the Key Research Program of the Chinese Academy of Sciences,

Revised: Grant NO.XDPB22, and Key-Area Research and Development Program of Guangdong Province (No.2019B010107001).

Original: A D ETAILS FOR DATASET GENERATION
Revised: A D ETAILS FOR DATASET GENERATION

Original: In this section, we describe the configurations used to generate the training set, validation set, and
Revised: In this section [Table] we describe the configurations used to generate the training set, validation set [Table]

Revised: and

Original: SSDNC benchmark.
Revised: SSDNC benchmark.

Original: We sample each non-leaf node following the unnormalized weighted distribution shown in Table 

Revised: We sample each non-leaf node following the unnormalized weighted distribution shown in Table

Original: Our base data set approximately contains 100,000 unique expressions’ skeletons.
Revised: Our base data set approximately contains 100,000 unique expressions’ skeletons.

Original: Then the different training sets are generated according to the number of expressions with different coefficients.
Revised: Then the different training sets are generated according to the number of expressions with different coefficients.

Original: We set this number to 10 , 20 , 30 , 40 and 50 .
Revised: We set this number to 10 , 20 , 30 , 40 and 50 .

Original: The validation set contains 1K skeletons that are randomly sampled from the base data set and assigned constants that differ from the training set.
Revised: The validation set contains 1K skeletons that are randomly sampled from the base data set and assigned constants that differ from the training set.

Original: We generate the more challenging test benchmark SSDNC, which includes approximately 100 unique expression skeletons and 10 re-sampled numerical constants for each skeleton.
Revised: We generate the more challenging test benchmark SSDNC, which includes approximately 100 unique expression skeletons and 10 re-sampled numerical constants for each skeleton.

Original: Source code is available at https://anonymous.4open.science/anonymize/Joint_
Revised: Source code is available at https://github.com/AILWQ/Joint_Supervised_

Original: Supervised_Learning_for_SR .
Revised: Learning_for_SR .

Original: [Table]
Revised: [Table]

Original: B B ECHMARK FUNCTIONS
Revised: B B ECHMARK FUNCTIONS

Original: This section describes the exact functions used to compare our method with the current state-of-theart methods.
Revised: This section describes the exact functions used to compare our method with the current state-of-theart methods.

Original: In Table 4, we show the name of the benchmark and corresponding expressions.
Revised: In Table [Table] we show the name of the benchmark and corresponding expressions.

Original: Input variables are denoted by x and/or y .
Revised: Input variables are denoted by x and/or y .

Original: We have restricted ourselves only to the univariate and bivariate functions.
Revised: We have restricted ourselves only to the univariate and bivariate functions.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  x2 + x Nguyenx 6 + x 5 + x 4 + x 3 + x 2 + x
Revised: [Table] x 6 + x 5 + x 4 + x 3 + x 2 + x

Original: Nguyensin( x 2 ) cos( x ) −
Revised: Nguyensin( x 2 ) cos( x ) −

Original: Nguyensin( x ) + sin( x + x 2 )
Revised: Nguyensin( x ) + sin( x + x 2 )

Original: Nguyenln( x + 1) + ln( x 2 + 1)
Revised: Nguyenln( x + 1) + ln( x 2 + 1)

Original: Nguyen- √ x
Revised: Nguyen- √ x

Original: Nguyensin( x ) + sin( y 2 )
Revised: Nguyensin( x ) + sin( y 2 )

Original: Nguyen2 sin( x ) cos( y )
Revised: Nguyen2 sin( x ) cos( y )

Original: Nguyenx y
Revised: Nguyenx y

Original: Nguyenx 4 − x 3 + 12 y 2 − y
Revised: Nguyenx 4 − x 3 + 12 y 2 − y

Original: Constant3 .
Revised: Constant3 .

Original: 39 x 3 + 2 .
Revised: 39 x 3 + 2 .

Original: 12 x 2 + 1 .
Revised: 12 x 2 + 1 .

Original: 78 x
Revised: 78 x

Original: Constantsin x 2 · cos x − 0 .
Revised: Constantsin x 2 · cos x − 0 .

Original: Constantsin (1 . 5 x ) · cos (0 . 5 y ))
Revised: Constantsin (1 . 5 x ) · cos (0 . 5 y ))

Original: Constant2 .
Revised: Constant2 .

Original: 7 x y
Revised: 7 x y

Original: [Table]
Revised: [Table]

Original:  2Feynmanx · yFeynmanx 2 · (1+ y ) Feynmanx · y 2 · π
Revised:  Feynmanx · yFeynmanx 2 · (1+ y ) Feynmanx · y 2 · π

Original: C M ODEL H YPERPARAMETERS
Revised: C M ODEL H YPERPARAMETERS

Original: In this section, we give more details about the hyperparameters of our models with/without CL.
Revised: In this section, we give more details about the hyperparameters of our models with/without CL.

Original: The full set of hyperparameters can be seen in Table 
Revised: The full set of hyperparameters can be seen in Table 5.

Original: D B ASELINES DETAILS
Revised: D B ASELINES DETAILS

Original: Transformer-based methods.
Revised: Transformer-based methods.

Original: For SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021), we use the standard hyperparameters provided in the open-source implementation of these methods 56 .
Revised: For SymbolicGPT (Valipour et al., 2021) and NeSymReS (Biggio et al., 2021), we use the standard hyperparameters provided in the open-source implementation of these methods 56 .

Original: Some parameters such as the dimension of input, vocabulary size, and so on, are manually adjusted as the dataset changes.
Revised: Some parameters such as the dimension of input, vocabulary size, and so on, are manually adjusted as the dataset changes.

Original: Note that we do not change model hyperparameters that may affect performance.
Revised: Note that we do not change model hyperparameters that may affect performance.

Original: 5 https://github.com
Revised: 5 https://github.com

Original: https
Revised: https

Original: [Table]
Revised: [Table]

Original: Deep symbolic Optimization (DSO).
Revised: Deep symbolic Optimization (DSO).

Original: For DSO (Mundhenk et al., 2021), we use the standard parameter settings in the open-source implementation 7 .
Revised: For DSO (Mundhenk et al., 2021), we use the standard parameter settings in the open-source implementation 7 .

Original: DSO depends on two main hyper-parameters namely the entropy coefficient λ H and the risk factor ϵ
Revised: DSO depends on two main hyper-parameters namely the entropy coefficient λ H and the risk factor ϵ

Original: [Table]
Revised: [Table]

Original: and hyperparameters related to genetic programming hybrid methods.
Revised: and hyperparameters related to genetic programming hybrid methods.

Original: The λ H is used to weight a bonus proportional to the entropy of the sampled expression which is added to the main objective.
Revised: The λ H is used to weight a bonus proportional to the entropy of the sampled expression which is added to the main objective.

Original: The intervention in the definition of the final objectivewith depends on the (1 − ϵ ) quantile of the distribution of rewards under the current policy.
Revised: The intervention in the definition of the final objective depends on the (1 − ϵ ) quantile of the distribution of rewards under the current policy.

Original: According to the open-source implementation, the chosen hyperparameters are listed in Table 6.
Revised: According to the open-source implementation [Table] the chosen hyperparameters are listed in Table 6.

Original: Genetic
Revised: Genetic

Original: [Table]
Revised: Programming (GP). For

Original: GP
Revised: GP-based methods

Original: [Table]
Revised: [Table]

Revised: we opt for the function

Original: SymbolicRegressor of open-source Python library gplearn 8 .
Revised: SymbolicRegressor of open-source Python library gplearn 8 .

Original: Our choices for the hyperparameters are mostly the default values indicated in the library documentation.
Revised: Our choices for the hyperparameters are mostly the default values indicated in the library documentation.

Original: The detailed settings are reported in Table7.
Revised: The detailed settings are reported in Table

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: E E XPLORING PHYSIC LAWS
Revised: E E XPLORING PHYSIC LAWS

Original: Symbolic regression is used by many research communities to advance the study of numerous scientific fields, e.g., Physics (Wu & Tegmark, 2019; Udrescu & Tegmark, 2020), Chemistry (Batra et al., 2021), and Materials (Sun et al., 2019;
Revised: Symbolic regression is used by many research communities to advance the study of numerous scientific fields, e.g., Physics (Wu & Tegmark, 2019; Udrescu & Tegmark, 2020), Chemistry (Batra et al., 2021), and Materials (Sun et al., 2019;

Original: Wang et al., 2019; Weng et al., 2020; Loftis et al., 2020).
Revised: Wang et al., 2019; Weng et al., 2020; Loftis et al., 2020).

Original: Our method shows great potential in recovering some of the laws of physics.
Revised: Our method shows great potential in recovering some of the laws of physics.

Original: As reported in Table 8, we successfully recover all the expressions of two variables in Feynman benchmark test sets.
Revised: As reported in Table 8, we successfully recover all the expressions of two variables in Feynman benchmark test sets.

Original: Since π is not included in the dictionary during training, the corresponding value in the recovered expression is predicted to be a decimal.
Revised: Since π is not included in the dictionary during training, the corresponding value in the recovered expression is predicted to be a decimal.

Original: Symbolic regression algorithms are getting better.
Revised: Symbolic regression algorithms are getting better.

Original: Our work will be useful for future data-driven symbolic regression methods.
Revised: Our work will be useful for future data-driven symbolic regression methods.

Original: We look forward to the day when a computer helps physicists discover the basic laws of physics, even just like Kepler, discovers a useful and hitherto unknown physics expression through symbolic regression.
Revised: We look forward to the day when a computer helps physicists discover the basic laws of physics, even just like Kepler, discovers a useful and hitherto unknown physics expression through symbolic regression.

Original: [Table]
Revised: [Table]

Original: f = 
Revised: f = exp(

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: ) √ (
Revised: ) √ (

Original: [Table]
Revised: [Table]

Original: π 
Revised: π ) σ π

Original: U (1 , 3 , 100)
Revised: U (1 , 3 , 100)

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: I.12.
Revised: I.12.

Original: F = µN n
Revised: F = µN n

Original: None
Revised: None

Original: U (1 , 5 , 100) µN n 1.
Revised: U (1 , 5 , 100) µN n 1.

Original: I.12.
Revised: I.12.

Original: F = q 2 E f
Revised: F = q 2 E f

Original: None
Revised: None

Original: U (1 , 5 , 100) q 2 E f 1.
Revised: U (1 , 5 , 100) q 2 E f 1.

Original: I.14.
Revised: I.14.

Original: U =
Revised: U =

Original: k spring x 

Revised: k spring x

Original: None
Revised: None

Original: U (1 , 5 , 100) k spring x 21.
Revised: U (1 , 5 , 100) k spring x 21.

Original: I.25.
Revised: I.25.

Original: V = qC
Revised: V = qC

Original: None
Revised: None

Original: U (1 , 5 , 100) qC 1.
Revised: U (1 , 5 , 100) qC 1.

Original: I.26.
Revised: I.26.

Original: n = sin( θ 1 ) sin( θ 2 )
Revised: n = sin( θ 1 ) sin( θ 2 )

Original: None
Revised: None

Original: U (0 , π 2 , 100) sin( θ 1 ) sin( θ 2 ) 1.
Revised: U (0 , π 2 , 100) sin( θ 1 ) sin( θ 2 ) 1.

Original: I.29.
Revised: I.29.

Original: k
Revised: k

Original: = ωc c
Revised: = ωc c

Original: U (1 , 10 , 100) ωc 1.
Revised: U (1 , 10 , 100) ωc 1.

Original: I.34.
Revised: I.34.

Original: W =
Revised: W =

Original: h 2 π ω π , h
Revised: h 2 π ω π , h

Original: U (1 , 5 , 100) 0 .
Revised: U (1 , 5 , 100) 0 .

Original: hω 0.
Revised: hω 0.

Original: I.39.
Revised: I.39.

Original: U = 32 p r V
Revised: U = 32 p r V

Original: None
Revised: None

Original: U (1 , 5 , 100) 32 p r V 1.
Revised: U (1 , 5 , 100) 32 p r V 1.

Original: II.3.
Revised: II.3.

Original: h =
Revised: h =

Original: P wr 4 πrπ
Revised: P wr 4 πrπ

Original: U (1 , 5 , 100) 0 .
Revised: U (1 , 5 , 100) 0 .

Original: P wr r1.
Revised: P wr r1.

Original: II.8.
Revised: II.8.

Original: u = ϵE 2 fϵ
Revised: u = ϵE 2 fϵ

Original: U (1 , 5 , 100) ϵE 2 f1.
Revised: U (1 , 5 , 100) ϵE 2 f1.

Original: II.27.
Revised: II.27.

Original: u
Revised: u

Original: = ϵEϵ
Revised: = ϵEϵ

Original: U (1 , 5 , 100) ϵE1.
Revised: U (1 , 5 , 100) ϵE1.

Original: II.38.
Revised: II.38.

Original: µ =
Revised: µ =

Original: Y 2(1+ σ )
Revised: Y 2(1+ σ )

Original: None
Revised: None

Original: U (1 , 5 , 100) 0 .
Revised: U (1 , 5 , 100) 0 .

Original: 5 Y 1+ σ 1.
Revised: 5 Y 1+ σ 1.

Original: III.12.
Revised: III.12.

Original: J = mh 

Revised: J = mh

Original: π π , h
Revised: π π , h

Original: U (1 , 5 , 100) 0 . 15923 mh 0.
Revised: U (1 , 5 , 100) 0 . 15923 mh 0.