Paper: Symbolic equation solving via reinforcement learning (ICLR-2024)

Reviewer comment (Novelty):
The problem is interesting but the restriction to just linear equations is quite severe.

Author response:
A similar remark was made by Reviewer hHWo. We agree that the problem of solving linear equations is relatively simple. Yet we do believe that the addition of symbolic coefficients, in particular, adds some complexity to the problem, and we are working on extensions to other domains. The advantage of linear equations, and one reason why we chose them, is that this is a problem that is well understood by humans and that it is relatively straightforward to follow and analyze the RL agent's solution strategy, which we deemed helpful given that this is the first exploration of our approach/idea. Furthermore, the problem complexity we can handle is ultimately limited by the computing resources available to us.

We have adjusted the discussion/conclusions to work out more clearly why we think that our results are nontrivial.

Reviewer comment (Novelty):
I am not sure if this is a widely different approach from Computer Algebra System (CAS), as in the end CASs also implement a search strategy in a space of possible actions. This is not completely a criticism, as it could be exciting to extend CASs with RL for efficiency. But I am not sure this paper provides many novel ideas in that direction. I believe the paper can be a stepping stone to a real proof-of-concept for RL applications to CAS, but in its present form it is too limited.

Author response:
We agree with this interpretation of the RL agent performing a search for solution strategies in the space of equation transformations. In fact, we see it as one of our main contributions that we found a way to make such problems requiring exact mathematical transformations amenable to machine-learning/reinforcement-learning methods. We rephrased parts of the conclusions to highlight this better.

Reviewer comment (Novelty):
Have you looked into expanding the existing open-source CAS (such as Sympy) systems with RL?

Author response:
Yes, we would even argue that our current implementation already is an "expansion" of SymPy in some sense. Essentially, we replaced the "equation solving" module of SymPy with our RL agent, but the term handling is left to SymPy. This is briefly explained at the end of Sec. 2.3 with details in Appendix A.7. So our approach is already integrated with SymPy and can easily be adapted to replace or extend other SymPy functionality in principle.

We have added a comment at the end of Sec. 2.3. to highlight this integration with SymPy.

Reviewer comment (Experiments):
Why have you not experimented with quadratic equations? Does the space of actions significantly explode in that case?

Author response:
We are currently experimenting with quadratic equations as a natural next step. Quadratic equations come with additional complications like multiple solutions and rational powers (square roots) for inverse operations, which can lead to non-polynomial forms of equations during the RL agent's explorations, for example. We do not include results in this paper because we are still investigating the best strategy and are not satisfied with the success rates yet.

Reviewer comment (Writing):
In what case, if ever, is exponential (^) used?

Author response:
In the present context of linear equations, the exponential (^) operation is used/required only to invert products. For example, to solve the equation 2\*x = 4, the agent will want to multiply both sides by 1/2, and the typical steps to prepare this transformation will be as follows: push the coefficient in front of x (here 2) to the stack, push the constant -1 to the stack, apply ^ on the stack (resulting in 2^(-1) = 1/2 as the top element of the stack), apply \* on the equation. This is also exemplified in Fig. 2d, see the second-to-last action in particular. Of course, the agent explores other uses of ^ during training, but usually learns quickly that these are not helpful for the problems at hand.

Reviewer comment (Novelty):
There is no clear motivation why the proposed approach is a good idea. The authors state that:  
" Evidently, this process could benefit greatly from techniques that enable computers to discover and implement transformation rules on their own. Moreover, finding viable approaches to do so will eventually help to make machine-learning models more adept at mathematics and problems requiring exact solutions in general."

Author response:
The pain point, in our opinion, is that humans are relatively slow at discovering and implementing mathematical relations and rules. Current CAS build on thousands of years of mathematical research and the major ones have been under development for over 30 years to "teach" computers what humans know. Of course, we are well aware that, presently, our approach cannot compete with these established software packages and does not discover anything unknown to humans. However, we still believe that it is a first step towards automated ways of mathematical research and that such an autonomous "RL researcher" could greatly enhance the power of CAS as well as the human mathematical knowledge base. We emphasize that it will still build on established human insights (like our present approach does, too), but it should be capable of exploring new domains that it had not been explicitly taught about in an independent, autonomous way.

We have adjusted the introduction to explain more clearly in what sense we aim to enable computers to discover and implement transformation rules on their own.

Reviewer comment (Novelty):
It is unclear why it is a disadvantage that current automatic equation solvers integrate human expert knowledge, and what the pain point of the current solutions is.

Author response:
The pain point, in our opinion, is that humans are relatively slow at discovering and implementing mathematical relations and rules. Current CAS build on thousands of years of mathematical research and the major ones have been under development for over 30 years to "teach" computers what humans know. Of course, we are well aware that, presently, our approach cannot compete with these established software packages and does not discover anything unknown to humans. However, we still believe that it is a first step towards automated ways of mathematical research and that such an autonomous "RL researcher" could greatly enhance the power of CAS as well as the human mathematical knowledge base. We emphasize that it will still build on established human insights (like our present approach does, too), but it should be capable of exploring new domains that it had not been explicitly taught about in an independent, autonomous way.

We have adjusted the introduction to explain more clearly in what sense we aim to enable computers to discover and implement transformation rules on their own.

Reviewer comment (Experiments):
It is unclear how much faster/slower and more/less accurate the proposed method works compared to established methods from Mathematica, Maple, Matlab, or SymPy.

Author response:
We did not carry out such a comparison because this is not the aim of our study as outlined above. All of the established CAS will achieve success rates of essentially 100 % for the problem of linear equation solving. They will most probably run faster than our method, too, which has not been optimized for speed yet. It will not matter for the human user who is interested in solving a particular equation because they won't experience a noticeable delay, but it might become relevant if one wishes to solve a batch of several hundreds of equations at a time.

Reviewer comment (Experiments):
The impacts of the hyper-parameters on the performance of the solution are not discussed. For example, it is unclear how relevant the discount factor or the network complexity are. While deep RL approaches have many hyper-parameters, it is still important to identify the most sensitive ones and perform inspection and analysis regarding the robustness of the approach.

Author response:
A qualitative overview of the influence of the various hyperparameters can be found in Appendix A.8. As explained there, we did not carry out a systematic grid search due to the large number of hyperparameters, but we explored several different variations for each type of problem. In the revised manuscript, we have added Tables 5-16, which list the success rates for hyperparameter variations of each of the models from Figs. 2-4 to provide a quantitative account of the hyperparameter influences.

Reviewer comment (Experiments):
The impacts of the RL approach (double Q learning) on the solutions are not analyzed. It is also unclear how other methods, such as PPO, would compare.

Author response:
At present, we have not explored other RL approaches on this type of problem. Exploring different RL approaches like PPO is certainly an interesting suggestion and may become helpful when extending to other problem domains. Given the discrete nature of the action space, we did not see any obvious reasons why other methods should be preferred in our present setting, and evidently Q learning worked. We _did_ experiment with different variants of Q learning in the beginning (e.g., single vs. double Q learning; experience prioritization; loss function amendments like convergent DQN or weight decay; multistep action lookahead). Compared to the implementation detailed in Appendix A, those alternatives did not yield any significant advantages, but required longer training times and sometimes performed worse.

Reviewer comment (Experiments):
Compared to related work, no related method is used for comparison.

Author response:
The most closely related works that we are aware of have been discussed in Sec. 4.1, but none of them address the exact same problem, so we don't see a reasonable way to compare them quantitatively to our results.

Reviewer comment (Evaluation):
After training, the success rate compared to established solvers is not provided.

Author response:
See reply to item 2.1 above.

Reviewer comment (Evaluation):
The wall-clock time in inference compared to established solvers is not provided.

Author response:
See reply to item 2.1 above.