From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: All reviewers agree that this work is well-written and easy to follow with clear motivation and contributions. The experimental experiments support the effectiveness of the proposed method.
justification_for_why_not_higher_score: The contribution of this work is a solid accept but has not reached the bar for spotlight.
justification_for_why_not_lower_score: All reviewers agree to accept the paper.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper6520/Authors
title: Thank you!
comment: Dear Reviewer dp8W:

We sincerely thank you for your careful and detailed reading of our paper and responses. We are glad that our responses address your concern. Your suggestions have been very helpful for us to improve the paper. Thank you again for strongly supporting our paper!

Best regards,

Authors of Paper6520

From: ICLR.cc/2023/Conference/Paper6520/Authors
title: Thank you!
comment: Dear Reviewer eWWS:
 
We sincerely thank you for your careful and detailed reading of our paper and responses. We are happy that our responses are convincing you.  Thank you again for strongly supporting our paper!

Best regards,

Authors of Paper6520

From: ICLR.cc/2023/Conference/Paper6520/Reviewer_dp8W
title: Thanks for your response
comment: Thank you for your reply. The responses have addressed my concern. I will keep my score and recommend accepting this paper.

From: ICLR.cc/2023/Conference/Paper6520/Reviewer_eWWS
title: Thank you for your response
comment: Thank you for your reply. The responses are convincing me. I will keep my score and recommend accepting this paper.

From: ICLR.cc/2023/Conference/Paper6520/Authors
title: Looking forward to your reply
comment: Dear Reviewer dp8W:

We would like to thank you for your positive and constructive comments, which are very helpful and make our paper stronger! We have posted point-to-point replies to each question/comment raised by you and uploaded the revised version of our paper (with track changes marked in blue). We sincerely hope that we could address your concerns. We would appreciate it if you would like to consider updating the score and increasing your confidence. We will also be happy to answer any further questions you may have during the discussion.

Best regards, 

Authors of Paper6520

From: ICLR.cc/2023/Conference/Paper6520/Authors
title: Common Response
comment: Dear Reviewers, Area Chairs, and Program Chairs,

We thank all reviewers for their positive feedback and suggestions. We are pleased that all reviewers have acknowledged the contributions of our work and recognized its value. Besides, we are glad to see that the paper is considered to have the following strengths:

- well-written and easy to follow (Reviewer eWWS, dp8W, jvHV)
- clear motivation and contributions (Reviewer eWWS, jvHV)
- comprehensive experimental evaluation and good performance (Reviewer eWWS, dp8W, jvHV)
- the method is novel (Reviewer eWWS)
- the problem is interesting (Reviewer dp8W)
- cover a wide range of related work (Reviewer jvHV)

We again thank all the reviewers for their positive comments. Here, we summarize the motivation, novelty, and contributions of this work again.

- **Motivation**. In recent years, data-driven symbolic regression (SR) methods have shown promising results in terms of inference speed compared to traditional GP-based methods. However, these methods focus more attention on using large scale training data and ignore a key problem: the ill-posed problem, i.e. expressions that may be completely different have the same supervision because of their same skeleton, which makes it challenging to deal with data that may be from the same expression skeleton but with different coefficients. Specifically, previous methods only use the single traversal of the binary tree of the expression skeleton (coefficients are replaced by the constant placeholder) as supervision information and only use the cross-entropy as the loss function for training. This supervision is evidently far from adequate for mathematical expressions of which symbols and coefficients are strongly coupled. The experimental results show that the ill-posed problem seriously affects the accuracy of the model in reasoning about the expressions from observed data points, so we explore a more efficient data-driven SR method.

- **Novelty and Contributions**. Please note that we propose a new framework that aims to efficiently alleviate the crucial ill-posed problem existing in data-driven SR methods to promote the development of SR. The encoder of the data-driven SR model should not only be able to **(a)** extract more efficient features of the data points, but also **(b)** represent the expression on a high level such that the transformer decoder only prints the representation as a sequence of symbols. For **(a)**, we leverage a point cloud-like feature extractor based on pure residual MLP networks to extract more efficient features of the observed data points. For **(b)**, we cleverly combine contrastive learning (CL) techniques to achieve this. We propose a joint supervised learning mechanism that is our core idea. Specifically, we use whole expression skeletons as complementary supervision and increase the inter-class distance (for expressions with different skeletons) and decrease the intra-class distance (for expressions with the same skeleton but different coefficients), which makes the output feature of the feature extractor more efficient in the representation of expressions. We demonstrate the effectiveness of our method through extensive experiments of our model with/without CL. To our knowledge, we are the first to raise this problem and solve it by combining contrastive learning. Our method achieves up to 25% higher than other state-of-the-art methods in terms of the recovery rate of the expression skeleton, and achieves the highest accuracy on both the five popular public datasets and the synthetic SSDNC dataset.

We responded to the individual comments of each reviewer. Additionally, we highlight the key updates of the revised version marked in blue for the convenience of the reviewers (we will remove the coloring in the final version). The updates are listed below:

- **In Section 4.2**, we have clarified that the test set is different from the training set. (Reviewer eWWS, dp8W)
- **In the caption of Figure 1**, we have corrected a clerical error. (Reviewer eWWS)
- **In Appendix E**, we have cited several current successful applications of symbolic regression in different fields, as well as added the details of our method to recover the Feynman physical equations so as to clarify the potential applications of the proposed technique and method design. (Reviewer dp8W)
---
Finally, we thank all reviewers again for their thoughtful reviews and for spending time writing the reviews. Please let us know if have any recommendations for further improving the paper and strengthening support.

Best regards, Authors of Paper6520

From: ICLR.cc/2023/Conference/Paper6520/Authors
title: Response to Reviewer jvHV
comment: Dear reviewer jvHV,

Many thanks for your valuable time to review our paper! We're happy that you appreciate our idea and acknowledge the significant improvement we achieved compared with other baseline methods.

We agree with you. A good algorithm should be uncomplicated and effective. Our proposed joint supervised learning mechanism effectively alleviates the ill-posed problem existing in other all data-driven symbolic regression methods. To our knowledge, we are the first to address this problem by combining contrastive learning. We believe that our proposed joint supervised learning mechanism is beneficial for future data-driven symbolic regression methods.

Thank you again for your positive feedback!

Best regards, Authors of Paper6520

From: ICLR.cc/2023/Conference/Paper6520/Authors
title: Response to Reviewer dp8W
comment: Dear Reviewer dp8W,

Thank you for your positive feedback! We are glad that you acknowledge that the problem is interesting and the framework we propose performs well. We will elaborately address your concerns and questions as follows.

>W1: It would be better if the authors could further clarify the potential application of symbolic regression.

Thank you for your valuable suggestion! We accept your suggestion and it will make our paper better. Our method shows great potential in recovering some of the laws of physics. As reported in section4.6, our model performs well in recovering Feynman physics equations. Due to the limitation of the main text, **in Appendix E we have added a description of the current successful application of symbolic regression and the details of recovering the Feynman equations in our method.**

>W2: It is unclear to me why the authors adopt a point cloud-like feature extractor (like PointMLP). It seems that the tackled problem is a special regression problem, so why not use 1D convolution or directly use a transformer to handle it? Also, there are some works tackling tabular data, like TabNet. The authors should further clarify the motivation of network design.

Thank you for your questions. Let me clarify the motivation for our network design step by step.
- Symbolic regression aims to discover the underlying mathematical expression that **contains symbols and coefficients** and best fits the observed data, rather than a simple regression problem such as least squares or polynomial fitting. In our work, we regard the discovery process as a translation process, exploring the best mapping from data points' space to mathematical expressions' space.
- **We believe that learning a good representation of the expression is the key to data-driven symbolic regression methods.** Given the input data points $\mathcal{D}=\{ (\pmb{x_i}, y_i) \}_{i=1}^n \in\mathbb{R}^{n\times(d+1)}$, we use the point cloud-like feature extractor for two main reasons: (i) the conversation from the data point $\mathcal{D}$ to the latent representation should not be sensitive to the order in which the points of the dataset are given, because the optimal equation should not depend on the order in which the points are listed. Note that the input to a symbolic regression instance is a set of data points, rather than a sequence. Thus, **the feature of the data points should be similarly order-invariant**. (ii) the conversation should not strictly depend on the number of points in the input data. The reason is that the data points provided may have varying sizes, thus we do not want our method to be restricted to cases with a ﬁxed number of input points. **The above two crucial properties coincide with the point cloud**. Therefore, we treat data points sampled from the mathematical equation as the point cloud and use the point cloud-like feature extractor (i.e. PointMLP) to extract the features of data points. As described in Section4.5, the PointMLP-based feature extractor we used performs better than other methods.
- The 1D convolution and transformer you mentioned have shown promising results on sequence feature extraction. TabNet is used to process tabular data by using sequential attention to choose data features. The reason we do not use these is that our input is not an ordered sequence or tabular data but a set of points like the point cloud. In our method, given the feature extracted from PointMLP, we use the GPT based on the transformer decoder to autoregressively generate expressions.

>W3: Since the proposed framework was trained and evaluated on the same synthetic dataset. There may be overfitting problems. The authors should discuss this point.

We are sorry that we did not make this point clear in the paper, which caused you this concern. **The test set we use is different from the training set.** Most benchmark test sets for evaluation are not seen during training. At test time, the inputs to the model, i.e., the data points from the expression are resampled to avoid potential overfitting problems, which is different from the inputs the model sees during training. We have clearly clarified this point in Section4.2 of the revised version and highlighted the changes in blue.

---
Again, thank you for your suggestions and comments. We hope our responses have provided you with a clearer picture of our work and have addressed your concerns. If you still need more clarifications to increase the scores, please let us know.

Best regards, Authors of Paper6520

From: ICLR.cc/2023/Conference/Paper6520/Authors
title: Response to Reviewer eWWS
comment: Dear Reviewer eWWS,

Many thanks for your positive feedback! We are pleased that you found our paper novel, well-written, and clear. Here we will carefully address your concerns and questions in detail.

>weakness: In the experiment, the same dataset is used for training and testing. I am not confident whether such a setting is reasonable. From the machine learning perspective, the obtained model should be evaluated on test datasets.

We are sorry that we did not make this point clear in the paper. **The test set we use is different from the training set.** We use a total of six benchmarks for evaluation, most of which were not seen during training. To avoid potential overfitting problems, at test time, the inputs to the model, i.e., the data points from the expression, are resampled, which is different from the inputs the model sees during training. **We have clearly clarified this point in Section4.2 of the revised version and highlighted it in blue.**

>C1: It would be nice if the proposed method was evaluated using recent benchmark datasets such as SRBench (https://github.com/cavalab/srbench).

Thank you for your suggestion！SRBench[1] is a good living benchmark for symbolic regression. It mainly contains three benchmarks: ODE-Strogatz[2] problem set, Feynman[3] datasets, and black-box regression datasets from PMLB[4], among which Feynman datasets have been included in our test.  SRbench's other expressions are mostly high-dimensional, up to 1000 variables, and are therefore currently more suitable for comparing non-data-driven methods, such as GP[5] and DSO[6], which are processed or trained from scratch for each new dataset. In our method, because the training set we used contains expressions of up to two variables, our method and other data-driven baseline methods we chose (i.e. SymbolicGPT and NeSymReS) do not yet support testing expressions of higher dimensions. In future work, we can train with data of high-dimensional expressions to enable our method to infer expressions of more variables, but not the focus of our work now. We have used five popular public benchmarks in our work and a more challenging test set SSDNC for data-driven methods, so we believe that the combined result on the total six datasets is strong enough to support all the major claims. Thank you again for your suggestion and we will consider it in our future work.

>C2: In Figure 2, the redder color might indicate the cosine similarity is closer to 0.

Thank you for your careful review. We have corrected this clerical error in the revised version.

>C3: How is the computational cost of the proposed method compared to other SR methods?

Thank you for your question. In general, our method is comparable in computational cost to other data-driven SR methods, and they are all smaller than GP and DSO. More importantly, our and other data-driven SR methods have an obvious advantage over methods like GP and DSO in inference speed. GP and DSO need to train a model from scratch on each new dataset whereas our model only needs to be trained once. We evaluate our method, GP, and DSO in terms of computational time at inference. As shown in the below table, we report the average runtimes (on the same device) across the Nguyen benchmark, with early stopping if the benchmark expression is recovered.

| **Method** | **Runtime (sec)** |
|-----------------|-------------------|
| Ours        | 25.39             |
| DSO        | 324.65            |
| GP           | 475.63            |
---
Thanks again for your appreciation and constructive comments on our work! 

Best regards, Authors of Paper6520

**References:**

[1] William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabricio Olivetti de Franca, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason H Moore. Contemporary symbolic regression methods and their relative performance. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.

[2] Steven H Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018.

[3] Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020.

[4] Joseph D Romano, Trang T Le, William La Cava, John T Gregg, Daniel J Goldberg, Praneel Chakraborty, Natasha L Ray, Daniel Himmelstein, Weixuan Fu, and Jason H Moore. Pmlb v1. 0: an open-source dataset collection for benchmarking machine learning methods. Bioinformatics, 38 (3):878–880, 2022.

[5] John R Koza. Genetic programming as a means for programming computers by natural selection.
Statistics and computing, 4(2):87–112, 1994.

[6] T. N. Mundhenk, M. Landajuela, R. Glatt, C. P. Santiago, D. M. Faissol, and B. K. Petersen. Symbolic regression via neural-guided genetic programming population seeding. 2021.

From: ICLR.cc/2023/Conference/Paper6520/Reviewer_eWWS
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a novel transformer-based model for symbolic regression (SR), which produces mathematical expression skeletons from data points. In the proposed method, a feature extractor using pointMLP is added and jointly trained using contrastive loss to realize efficient training. The experimental evaluation using benchmark functions demonstrates that the proposed method exhibits higher recovery rates and R2 scores compared to existing SR methods.
strength_and_weaknesses: **Strengths**
* Introducing the idea of feature extractor and contrastive learning into the transformer-based symbolic regression model is convincing. Although each component of the proposed method was previously proposed in the context of other tasks, incorporating such methods into the SR model seems to be novel.
* The numerical experiment supports the effectiveness of the proposed method.

**Weaknesses**
* In the experiment, the same dataset is used for training and testing. I am not confident whether such a setting is reasonable. From the machine learning perspective, the obtained model should be evaluated on test datasets.

**Comments**
* It would be nice if the proposed method was evaluated using recent benchmark datasets such as SRBench (https://github.com/cavalab/srbench).
* In Figure 2, the redder color might indicate the cosine similarity is closer to **0**.
* How is the computational cost of the proposed method compared to other SR methods?
clarity,_quality,_novelty_and_reproducibility: * This paper is well-written. The motivation and contribution are easily understood.
* Although the components of the proposed method are existing ideas, introducing such promising techniques into the transformer-based SR model is novel.
* The code is provided and the detailed setting of the experiment is reported.
summary_of_the_review: Basically, this paper is well-written, and the motivation and contribution are clear. The effectiveness of the proposed method is supported by the numerical evaluation.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper6520/Reviewer_dp8W
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents a transformer-based method for symbolic regression using a new feature extractor and a joint supervised learning mechanism.  They evaluated the framework on synthetic datasets and compared it with several baseline methods. Experimental results show good performance of the proposed framework. 
strength_and_weaknesses: Strength: 

1. The problem is interesting. 
2. The proposed framework achieves good results. 

Weakness:

1. It would be better if the authors could further clarify the potential application of symbolic regression. 
2. It is unclear to me why the authors adopt a point cloud-like feature extractor (like PointMLP). It seems that the tackled problem is a special regression problem, so why not use 1D convolution or directly use a transformer to handle it? Also, there are some works tackling tabular data, like TabNet. The authors should further clarify the motivation of network design. 
3. Since the proposed framework was trained and evaluated on the same synthetic dataset. There may be overfitting problems. The authors should discuss this point. 
clarity,_quality,_novelty_and_reproducibility: The whole paper is well-written and proposes a new solution for an interesting problem.
summary_of_the_review: This is a good paper to tackle an interesting problem. While the potential application of the proposed technique and method design should be further clarified. The technical novelty of this paper seems limited.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6520/Reviewer_jvHV
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents a symbolic regression algorithm that uses a transformer backbone to learn a new feature extractor via joint supervised learning mechanism. The ill-posed problem mentioned in the paper was addressed by using the joint supervised learning mechanism combining supervised contrastive learning, which strengthens the similarity of feature vectors from the same skeleton. Experiments were done to prove the efficacy of the method.

strength_and_weaknesses: This paper is motivated by a practical issue. I like the idea of using the supervision of the whole expression skeleton and the supervision of the preorder traversal of its expression tree in contrastive learning (maybe with data augmentation) to deal with the case that different instances of the same skeleton can have very different shapes, and instances of very different skeletons can be very close.
The authors did extensive experiments, the presented algorithm performs very well on the benchmark dataset.
The paper has covered a wide range of related work.
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and straightforward. The novelty is not significant as it is a basic combination of contrastive learning and Transformers. The implementation is not hard but it works well.
summary_of_the_review: This paper proposes a good approach to tackle the symbolic regression problem. Although the presented algorithm seems to be a combination of contrastive learning and Transformers, its performance on benchmark data sets looks good. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: NA
recommendation: 6: marginally above the acceptance threshold