Paper: Learning Polynomial Problems with $SL(2, \mathbb{R})$-Equivariance (ICLR-2024)

Reviewer comment (Theory):
I have some issues re the technical details of the main theorem and the premises of the method, see below. If these issues prove to be non-issues I will raise the score.

Author response:
We clarify the reviewer’s points in the sections below. Indeed, we believe these issues are non-issues, but the reviewer should please feel free to follow up if anything remains unclear (and we very much thank the reviewer for their attention to detail).

Reviewer comment (Novelty):
The architecture that actually works is rather basic: MLPs with augmentations. On the other hand one could credit the paper in finding the equivariant structure and hence what the relevant augmentations are.

Author response:
Indeed, the message of our empirical findings is that a simple method (data augmentation) works best, although as noted in Section 5, some care must be taken to augment with reasonably well-conditioned elements of $SL(2,\mathbb{R})$. This could save practitioners time, in not designing a complex polynomial-based equivariant architecture; it could also prompt the development of different equivariant paradigms.

Reviewer comment (Theory):
The formulation of finding the positive-definite witness with maximal determinant assumes that there are many such witnesses. Are there many witnesses? For example, when we discuss polynomials of degree 2 and the monomial vector is (x,y) I think that a symmetric matrix Q uniquely defines a quadratic polynomial (x,y)Q(x,y). When we discuss polynomials of higher degree there are ambiguities that come from the fact that, say, (x^2)(y^2) = (xy)(xy). But this can be dealt with directly by adding more symmetry constraints into the matrix. In other words, the matrix should be a moment matrix as defined in [Lasserre 2001]. Once these constraints are added I believe that there will be no more ambiguities. Do you agree? If so wouldn't it make sense to incorporate the symmetries and forget about optimizing over logdet?

Author response:
As you pointed out, for the degree 2 case there is a unique feasible point. However, as soon as the polynomial is degree 4, there is a degree of freedom. For example, consider $p = x^4  - x^2y^2 + y^4$. For any $z$, a matrix of the form $$Q =
\begin{pmatrix} 1 & 0 &      z\\\  0 &-1-2z& 0\\\ z &0  &     1
\end{pmatrix}$$
is a witness for the positivity of $p$. Restricting that the main antidiagonal has equal entries (like in a moment matrix) would give the matrix
$\begin{pmatrix}1&   0&    -\frac13\\\ 0 &   -\frac13&  0\\\ -\frac13 & 0  &  1\end{pmatrix}$,
which does not have a positive determinant. On the other hand, 
$\begin{pmatrix}1  &    0  & -.75 \\\ 0  & .5 & 0\\\ -.75 &  0 & 1\end{pmatrix}$
is also a witness (with $z = -.75$), and the eigenvalues are positive. This is just one example of why additional restrictions will sometimes prohibit you from finding a PSD certificate. (See Example 4.1 of [Parrilo] for another example of how playing with the free parameter may uncover a PSD certificate). We selected the certificate of maximal determinant because it is the analytic center of all such possible witnesses.

To be clear, we do require $(xy)(xy) = x^2  y^2$. For example, consider the following bilinear form:
$$ \begin{pmatrix} x_1^2 \\\ x_1y_1 \\\ y_1^2 \end{pmatrix}^T \begin{pmatrix}1 & 2& 3\\\ 2 & 4& 5\\\ 3& 5& 6\end{pmatrix}  \begin{pmatrix} x_2^2 \\\ x_2y_2 \\\ y_2^2 \end{pmatrix} =\cdots + 3y_1^2x_2^2 + 4x_1y_1x_2y_2+3x_1^2y_2^2 + \cdots$$
Because we require $x_1 = x_2$ and $y_1 = y_2$, these 3 terms collapse to $10 x^2y^2$. However, there is no reason to require the coefficient 3 to equal the coefficient 4.

Reviewer comment (Writing):
A further issue with the non-universality of the SL(2) network is not correctness but exposition. You prove that the function f you're interested in cannot be approximated by SL(2) equivariant polynomials. However, f is not defined on all of the vector space: f(p) is only defined if p is indeed positive, and moreover there exists a *strictly* positive definite matrix verifying this. So f is defined on some subset of your vector space. The universality results in [Bogatskiy] pertain to the complex SL_2 and to functions continuous on the whole domain, which may be a more substantial difference. Similarly, [Villar et al.] show that all continuous functions invariant with respect to the non-compact Lorenz group action are approximated by polynomials, but those functions are defined on all of the domain. For non-compact groups, often distinct orbits cannot be separated by continuous functions. For example, the action of SL_d on d×d matrices by multiplication from the right means that a matrix without full rank is not in the same orbit as the zero matrix, but its orbit can contain matrices arbitrarily close to zero, forcing any continuous invariant function to take the same value. For more on this see [Dym and Gortler], especially the paragraph titled "algebraic separation vs. orbit separation". To be concrete, I think you should mention in the paper that the function f is not defined everywhere, and suggest changing the paragraph "why is SL(2,R) different" and related discussions to note that this might also explain the difference between other universality results and your non-universality result.

Author response:
The proof of Bogatsky [2] cites the theorems in Yarotsky [3], which are of the form “for every compact subset…”. The theorems only imply that the functions are approximated on compact subsets. Therefore, one could take a compact subset including $x^8 + y^8$, and then this theorem would fail. Hence, we do not think the issue is because the domain of $f$ is not the whole space, but we agree mentioning it is important because it is yet another difference between our case and the existing theorems (and have added this to the draft).

We thank the reviewer for pointing out this interesting fact about continuous, $SL(2,\mathbb{R})$-invariant functions. We view this as an interesting structural fact about what continuous invariant functions exist, independently of how well they may be represented. One can still ask how well such a function can be approximated by polynomials -- since polynomials are continuous too, they will also exhibit this property, in which the function must take the same value on the orbit of the 0 matrix and the orbit of $\text{diag}(0,1,...,1)$. However, our impossibility result involves a particular continuous, equivariant function $f$ that already exists (indeed, it is the exact function that we care about). We then demonstrate that this function cannot be approximated by any equivariant polynomial.

Reviewer comment (Writing):
Somewhere in the paper, explain why you decided to restrict yourselves to polynomials of two variables.

Author response:
We restricted ourselves to polynomials of two variables for simplicity, but polynomials in more than two variables are a natural next step. We will note this in the paper.

Reviewer comment (Theory):
In your discussion of Schur's Lemma on page 6: the lemma applies to complex representations and not real. Do you address this? If not, maybe just add a disclaimer.

Author response:
We have mentioned that we need a real version of Schur’s lemma, which is less specific about the form of possible intertwiners.

Reviewer comment (Theory):
On page 4, when you introduce the function f, discuss its domain. Mention that in its domain the function is well-defined since the optimization problem has a unique maximizer.

Author response:
Page 4:  Thank you for this suggestion; we have added a mention of the domain after the definition.

Reviewer comment (Writing):
On page 6, I did not understand your explanation of the last layer.

Author response:
Page 6: We have modified the description of the last layer, moving the technical details of how $L$ is computed to the appendix. Is it any more clear now?

Reviewer comment (Writing):
On page 9 you reference the wrong paper by Puny. You meant [Puny 2021] not [Puny 2023].

Author response:
Page 9: Thank you for pointing out the reference mistake. We have corrected it.

Reviewer comment (Reproducibility):
Section 4.2 is extremely vague. The pseudocode is almost unreadable because it is calling functions (in lines 8 and 10) which have never been defined.

Author response:
Thank you for pointing out the ambiguity in the functions in the pseudocode in Section 4.2. We have corrected this in the updated draft.

We have also rewritten the description of the last layer in Section 4.2, deferring the exact details of the mathematical derivation (which may have been confusing) to the appendix.

Reviewer comment (Theory):
The entire motivation of Section 4.2 seems unclear, even if assuming the correctness of Lemma 1. How is this related to the training problem that eventually seems to be the target?

Author response:
Positivity verification, as described in Section 3, is an equivariant learning task. Therefore, one might naturally design an equivariant architecture for this problem, as we do in Sections 4.1-4.2. The surprising, and central, theoretical contribution of this paper is that an equivariant network ostensibly designed for the very purpose of positivity verification, is in fact unable to approximate the function of interest for positivity verification. Therefore, any network designed to solve this problem must be able to output more than just equivariant polynomials, which is a significant restriction on the architectural design space relative to prior work! To the best of our knowledge, such a finding is also without precedent in the equivariance literature. It also provides some motivation for using data augmentation, rather than an equivariant architecture. This is the content of Section 4.3. Corollary 1 implies that it doesn’t matter what loss function we use -- no equivariant polynomial can approximate the function of interest under any standard loss function. In the paper, we used the normalized mean squared error loss (with a small additive stabilization term in the denominator), as noted in the caption of Figure 2.

Reviewer comment (Experiments):
It would have been much better to use the space in Section 4.3 to explain the experimental setup. For example, it seems critical to understand what is the author’s idea of a “natural” polynomial, and these details are missing from the main paper.

Author response:
By “natural” polynomials, we just meant that they come from a naturally occurring, application-specific distribution, such that it is empirically possible to well-approximate an NP-hard optimization like polynomial positivity verification over this distribution. (Intuitively, we would expect such a distribution to have low entropy -- it should differ significantly from a uniform distribution over polynomials in a norm-bounded ball, e.g.) For example, one of our experiments is on the polynomials that arise from spherical code bounds; this provides one example of a “natural” set of polynomials, but there is no single definition of such a distribution. This point should not be critical to our main arguments.

Reviewer comment (Reproducibility):
The loss functions used in the experiment in Section 4.3 also seem to be not clearly specified, making it further challenging to understand what is happening.

Author response:
In the paper, we used the normalized mean squared error loss (with a small additive stabilization term in the denominator), as noted in the caption of Figure 2.

The loss function was normalized mean squared error (as noted in Figure 2), computed as the Frobenius norm of the error between f(p) and the network output (which are matrices of equal size). In other words, if the output matrix is $M$ and the correct matrix $f(p)$ is $N$, the loss function is $\frac{\sum_{i,j}(M_{ij} - N_{ij})^2}{1+\sum_{i,j}N_{ij}^2}$ (where we add $1$ to the denominator for stability), averaged over all data points. We have added the explicit equation to the caption of Figure 2.

Reviewer comment (Evaluation):
What is the training time for the nets involved in Table 2? The impression is that the reported “MLP times” are just the inference times, while the timings specified for the other methods are probably the “total” time they take to run, with no other time costs reported.

Author response:
Indeed, the MLP times are for inference, while the Mosek times are full solves. The motivation of this work is to speed up the *online* determination of nonnegativity, which is possible with just MLP inference. Amortizing the runtime of a traditional solver over many instances is not possible, hence why we compare to the full solve time. For reference, the training time for the MLP was on the order of 90 minutes, which even included many different loss evaluations (e.g. evaluating the loss under random $SL(2,\mathbb{R})$-augmentations).