Paper: Learning to Compute Gröbner Bases (ICLR-2024)

Reviewer comment (Evaluation):
The authors acknowledge that transformers may only perform well for in-distribution samples, citing Dziri et al 2023. They dismiss this as "a fundamental challenge of transformers, and outside the score," but nevertheless this limits the applicability of their algorithm.

Author response:
Global response mostly replies to this comment. We admit that our current results are not very practical; however, we'd like to emphasize that this is because the learning of Gröbner basis computation is extremely challenging compared to the math problems addressed in the prior math transformers, requiring many machine learning and mathematical problems to be resolved. Besides, we consider that our task is based more on practical motivation. There is no scalable algorithm for computing Gröbner bases, while there are for the tasks in the related studies. The only exception is the line of works for attacking LWE using Transformers, which is also currently restricted to small problems and not considering generalization.

> a) The authors acknowledge that transformers may only perform well for in-distribution samples, ... nevertheless this limits the applicability of their algorithm. 

Yes, and this is actually why prior studies of math transformers explored better training distributions and tested generalization liability. However, this is strongly based on the fact that for their tasks, the random generation of samples is trivial and in-distribution accuracy is already high enough. We found that this is not the case for the Gröbner basis computation and thus focus on this step.

Reviewer comment (Theory):
Due to the limitations of transformers, they can learn only with instances of bounded size. It is unclear whether this limitation also applies to testing, but regardless it means that in-distribution is only of limited size. As a side note, the reviewer wonders whether the problem of finding the Grobner basis is also NP-hard for bounded size instances.

Author response:
Thank you for your suggestion. Yes, it's limited for now because the maximum sequence length of positional encoding was set based on the maximum sequence length of training samples. We retrained a few models with extended limits for $n=2, 3$. For $n=2$, we generated a new test test for which we set the maximum number of terms of polynomials in $U_1, U_2$ to 4 (previously 2). For $n=3$, we set the density $\sigma=1.0$. We observed a large accuracy drop for such out-distribution samples~(approximately 20-30% decrease). Note that the results relate to out-distribution accuracy, which is beyond the scope of current work (see global response). However, it may be true that the results will serve as a baseline for future work. 

We are not very sure what is meant by "bounded size of instances." An instance size of an NP-hard problem is always bounded. For reference, "computing a Gröbner basis from a given $F$ of finite size" is NP-hard. For example, the polynomials in MQ challenge, a post-cryptography challenge, are degree-2 polynomials. The difficulty of solving degree-2 polynomial systems is the foundation of the multi-variate cryptosystem.

Reviewer comment (Experiments):
The authors impose additional restrictions on instances in the training set, and they claim these are realistic.

Author response:
We consider that our setup is reasonable and oriented for practical scenario. Refer to the global response.

Reviewer comment (Theory):
Limited theory — although there are some theorems, they are simple and their proofs seem straightforward. The contribution from the mathematical side seems very limited.

Author response:
We have to admit the current theorems and algorithms are simple. This is partially because that the efficiency matters in the large-scale dataset generation. We may design a sophisticated method based on the necessary and sufficient condition (i.e., Theorem 4.7 (3)). Particularly, we may utilize the sygyzy module of $G$, the representatives of which can be efficiently computed as we have an access to a Gröbner basis $G$ (Schreyer's Theorem). We believe that this approach can give a more "uniform" sampling, but it needs a few more steps and is left to our future work.

Reviewer comment (Theory):
Contrary to what the authors claim, no real light is shed on the algebraic problems themselves.

Author response:
Our work introduces a motivation for unexplored algebraic problems and also presents an algorithm and theories for a base case. The intention of this comment is unclear to us, and we would appreciate it if we could have more elaboration.

Reviewer comment (Novelty):
The work will interest only specialists working on computational algebra.

Author response:
We believe that this work may attract interdisciplinary interests. Our new problems should encourage computational algebraists to work on a machine-learning topic. We showcase an example of math problems with difficulty in dataset generation, which should be of interest to the math transformer community. As you've pointed out, our algorithm and theory are designed simply, which may allow readers with limited knowledge of algebra to follow it. The limited in-distribution accuracy at Gröbner basis computation also provides an empirical limit of the Transformer's learnablity, which may interest the researchers in developping a better model. Futher, our work has a potential impact on the cyptography as many cryptosystems are in the scope of *algebraic attack*, which use Gröbner basis computation.

Reviewer comment (Experiments):
The reviewer understands that learning can only be done on limited size instances, but asks whether this limitation also applies to testing. If not, why were no such experiments reported?

Author response:
Thank you for your suggestion. Yes, it's limited for now because the maximum sequence length of positional encoding was set based on the maximum sequence length of training samples. We retrained a few models with extended limits for $n=2, 3$. For $n=2$, we generated a new test test for which we set the maximum number of terms of polynomials in $U_1, U_2$ to 4 (previously 2). For $n=3$, we set the density $\sigma=1.0$. We observed a large accuracy drop for such out-distribution samples~(approximately 20-30% decrease). Note that the results relate to out-distribution accuracy, which is beyond the scope of current work (see global response). However, it may be true that the results will serve as a baseline for future work.

Reviewer comment (Reproducibility):
On page 6, where you say that you sample O(s^2) polynomials, how are these sampled? What distribution?

Author response:
All the polynomials in this paper are sampled uniformly from the polynomials with a bounded total degree and a bounded number of terms. This is implemented by `random_element` function in SageMath.

Reviewer comment (Experiments):
The experimental evaluation is unconvincing because the test set seems to be randomly generated instances rather than problems arising out of real applications, and they are generated from the same distribution as the training set. Although the authors mention the problem of out-of-distribution generalisation, it is not addressed at all.

Author response:
**Weakness 1 : Out-distribution case**
> The test set seems to be randomly generated instances rather than problems arising out of some real applications. ... however it [the problem of out-of-distribution generalization] is not addressed ... .

This question is addressed in the global response. In short, the out-distribution case is beyond the scope of this study because, unlike the prior studies, the dataset generation itself is non-trivial, and in-distribution accuracy is not high enough.

Reviewer comment (Experiments):
It is usual for solvers to test their performance on some standard datasets (for example, the SAT competition for SAT solvers), and random instances are usually considered irrelevant. It is unclear whether the state-of-the-art Grobner basis tools have been optimised for random instances or practical instances, hence the comparison may not be fair.

Author response:
**Weakness 2: Standard datasets**
> It is usual for solvers to test their performance on some standard datasets (for ex. the SAT competition for SAT solvers), and random instances are usually considered irrelevant. 

This strongly relates to our main claim --- preparing many (non-Gröbner set, Gröbner set) pairs is non-trivial. To our knowledge, there are some classical families of polynomials for benchmarking Gröbner basis computation algorithms (or polynomial system solving algorithms), but these are very far from standard datasets. One may consult [this websight](https://www-sop.inria.fr/coprin/logiciels/ALIAS/Benches/node1.html). The listed examples are artificial, not necessarily zero-dimensional. They are empirically found difficult and/or designed for easily generating variations in the number of variables (e.g., testing for polynomial system Katsura-4 in 4 variables, then Katsura-5 in 5 variables, and so on). Classical benchmarks are useful for math algorithms, i.e., the algorithms proved to work for all the cases (i.e., 100 % "in/out-distribution" samples) except for the timeout, but not for our current work because they are out-distribution samples. 

Another benchmark can be found in cryptography challenges (e.g., [MQ Challenge](https://www.mqchallenge.org)). As we have discussed above, the polynomials appearing there are also strongly biased.

> It is not clear whether the state of the art Grobner basis tools have been optimised for random instances or practical instances, hence the comparision may not be fair.

The general purpose algorithms are Buchberger's algorithm and F4 algorithm (SoTA). We tested both of them in our experiments. Thus, the comparison must be fair as the focus of our study is oriented to the general case. Those specialized for particular "practical instances" are not discussed in this paper.

Reviewer comment (Novelty):
The use of transformers for this problem is not motivated well enough. While the paper mentions that transformers have been used for other math problems, including a step in the related Buchberger algorithm, it is not clear why they should be used here instead of something more basic.

Author response:
**Weakness 3: Motivation of using Transformer**

In the prior studies, math transformers were studied for testing the learnability of Transformers, rather than using them as practical solvers because we already have well-developed scalable algorithms for the adopted moderate-level math problems.

In contrast, the use of Transformers here is strongly motivated by the fact that there are no scalable mathematical algorithms for computing Gröbner bases. Unlike math algorithms, the runtime of the Transformer does not depend on the difficulty of the problem but on the sequence length. For a large-scale instance on which math algorithms cannot run, Transformers can return an answer with some probability or at least give a hint of the solutions (see discussion on the support accuracy of Table 2). We admit that the current results do not achieve this ultimate goal; resolving this challenge has a long way to go both in machine learning and computational algebra, and this study has initiated an essential step.

> The paper does mention that transformers have been used for other math problems, including for a step in the related Buchberger algorithm;

Quick correction. Transformers have not been used for Gröbner basis computation before. The technique applied to the Buchberger algorithm is simple reinforcement learning. Further, this algorithm assumes binomials and does not work for general polynomials.