Paper: How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computation (ICLR-2024)

Reviewer comment (Writing):
Page 2: Could you please explain the column-wise Kronecker product of V_1 and V_2 here as well?

Author response:
Q: Could you please explain the column-wise Kronecker product of V_1 and V_2 here as well?

A: Yes, thanks.

Reviewer comment (Theory):
Page 3: Please elaborate on SETH. I know it is standard, but perhaps too strong? :)

Author response:
Q: Please elaborate on SETH. I know it is standard, but perhaps too strong? :)

A: SETH roughly says that one cannot improve too much on our current best algorithms for solving CNF-SAT. This is one of the most well-studied algorithms problems, yet improvements have been rare and very small. Although the word “strong” appears in its name, the majority of complexity theorists believe it is true, and much of the area of fine-grained complexity theory is built off of it.

Reviewer comment (Writing):
Page 5: I do not understand "Approximating A."

Author response:
Q: I do not understand "Approximating A."

A: The matrix $\widetilde{A}$ is an ell_infinity approximation of A.

Reviewer comment (Theory):
Page 6: Could you please sketch an short example for the reduction from GapMaxIP to ATAttc?

Author response:
Q: Could you please sketch an short example for the reduction from GapMaxIP to ATAttc?

A: The reduction is depicted at the bottom of page 22; we will refer to it above.

Reviewer comment (Theory):
Page 8: Could you please add some discussion about Theorem 4.7?

Author response:
Q: Could you please add some discussion about Theorem 4.7?

A: Yes, this is the result we discuss in section 3.2 (“Hardness of Gap-MaxIP”).

Reviewer comment (Theory):
Page 9: Line 12 from bottom: Why is there such a \mu?

Author response:
Q: Line 12 from bottom: Why is there such a $\mu$?

A: Sorry, there is a typo here. If the three are _not_ orthogonal then there is no such $\mu$, and if they are orthogonal, then there is such a $\mu$. The message $\mu$ is the one Merlin would send to get the players to accept that the vectors are orthogonal.

Reviewer comment (Writing):
In Section 3.2 hardness, “3-Max IP” and “Gap-MaxIP” seem to refer to the same problem. It is confusing to give two names to the same problem.

Author response:
Q: In Section 3.2 hardness, “3-Max IP” and “Gap-MaxIP” seem to refer to the same problem. It is confusing to give two names to the same problem.  

A: Yes, our new problem “Gap-MaxIP” is a promise version of the more common “3-Max IP” problem from the literature. We’ve changed the heading of the paragraph to no longer say “3-MaxIP”, and discuss it in the text instead.

Reviewer comment (Presentation):
It might be clearer to present the upper bound (UB) and the lower bound (LB) in two separate sections, instead of giving an overview of the UB + LB, and then elaborate on the LB.

Author response:
Q: It might be clearer to present the upper bound (UB) and the lower bound (LB) in two separate sections, instead of giving an overview of the UB + LB, and then elaborate on the LB.  

A: Thank you. We arranged the paper as we did to fit in the page limit, but will try a clearer arrangement for the camera ready version.

Reviewer comment (Reproducibility):
Is it possible to give a few sentences of description of the mysterious $U_1, U_2, U_3$ matrices and how to compute them in Section 3.1?

Author response:
Q: Is it possible to give a few sentences of description of the mysterious $U_1, U_2, U_3$ matrices and how to compute them in Section 3.1?  

A: Yes we can elaborate on this. Just as showing a matrix is low-rank is equivalent to showing it is the product of two rectangular matrices, showing that a tensor is low-rank is equivalent to showing that it is the “product” of three rectangular matrices; $U_1, U_2, U_3$ are those three matrices.

Reviewer comment (Writing):
Minor issue: the last paragraph in page 5 states “showing that the same line of attack”? “attack” here means “techniques”, I assume?

Author response:
Q: Minor issue: the last paragraph in page 5 states “showing that the same line of attack”? “attack” here means “techniques”, I assume?  

A: Yes, by “line of attack” we mean “sequence of techniques”.

Reviewer comment (Writing):
In Section 3.1, does “$\widetilde{D} \approx D$” mean $\tilde{D}$ and $D$ close in the $\ell_{\infty}$ norm? (and so does “$\widetilde{A} \approx A$”?)

Author response:
Q: In Section 3.1, does “$\widetilde{D} \approx D$” mean $\widetilde{D}$ and $D$ close in the  norm? (and so does $\widetilde{A}$ and $A$?)  

A: Yes.

Reviewer comment (Theory):
In Section 4, what is the major challenge of extending the three-party communication protocol to a four-party communication protocol in Section 4.2? Why does one need to use the algebraic geometry code?

Author response:
Q: In Section 4, what is the major challenge of extending the three-party communication protocol to a four-party communication protocol in Section 4.2? Why does one need to use the algebraic geometry code?  

A: The challenging aspect is to determine what the correct problem is that the four players should be solving. Prior work ultimately showed hardness of an “approximate closest pair” problem, which does not easily generalize to yield hardness for our tensor setting. Indeed, natural options like “approximate closest triple” aren’t easy to reduce to attention. Once the correct problem definitions are in place, the actual protocol is not so difficult given prior work. AG codes are important here to save log factors compared to simpler constructions like Reed-Solomon codes. Saving the log factor is important since it ultimately appears in the exponent of the running time of our reduction in section E.

Reviewer comment (Theory):
In Section 4, where does $B = O(\sqrt[3]{\log n})$ pop up in the LB proof?

Author response:
Q: In Section 4, where does $B = O(\sqrt[3]{\log n})$  pop up in the LB proof?  

A: The bound B on the entries in attention arises in section E in the appendix, where we reduce from Gap-MaxIP to attention and complete the hardness proof. Section 4 proves hardness of Gap-MaxIP, and doesn’t yet involve the entry bound.