Paper: $\gamma$-Orthogonalized Tensor Deflation: Towards Robust \& Interpretable Tensor Decomposition in the Presence of Correlated Components (ICLR-2024)

Reviewer comment (Writing):
The paper is widely cited following paper (Seddik et al, 2021), so it is difficult to understand what the current paper is about without reading the cited paper.

Author response:
**Outline**. We have significantly revisited our manuscript, as outlined in our general comment. 
- (Seddik et al, 2021) lays the ground to a new angle to approach spectral tensor decomposition problems. Our paper follows the same path to construct our algorithm. While the reader is referred to check that paper for further details about random tensor theory based spectral tensor decomposition (proof of Theorem 1 in Section 3.2.1 of our paper for instance), our paper outlines the detailed construction of all our results in Section 3 of the main paper and provides a self-contained review of concepts of random matrix/tensor theory (RMT/RTT) in Appendix B to guide the unfamiliar reader.

Reviewer comment (Experiments):
The most important weakness, in my opinion, is too few experiments that have been conducted only on synthetic data. In general, experimental section 4 is very short, subsections 4.2 and 4.3 consist of only figures. Thus, it is not clear at all the relevance of this paper, where practically the presented results can be applied.

Author response:
**Experiments.** We update our experiments section, as outlined in our general comment. This further addresses the reviewer's specific questions. We list the relevant section again here for their convenience: 

- We now focus our empirical evaluation on two representative experiments and provide detailed explanations of both. In particular, we outline how our algorithmic approach performs compared to state-of-the-art and how our developed theory matches our predictions in practice with extensive experiments. In addition, we have also added separate visualizations for our theorems and assumptions throughout Section 3 and provided a detailed set of supplementary experiments in Appendix E.

Reviewer comment (Experiments):
Neither is there any comparison with other existing methods.

Author response:
**Questions.** 
- We thank the reviewer for their great question! We have further benchmarked our proposed algorithm against prevalent spectral tensor decomposition methods that are widely used in practice, CANDECOMP/PARAFAC (CP) and Tucker. We show that the latter collapse in the presence of higher levels of correlation $\alpha$ and fail to recover the signal of interest. We outline the details of that in Figure 5 and its subsequent explanation in Section 4/Benchmarks. Initially, we have not considered this benchmark highly relevant, as we only considered spectral decomposition algorithms.

Reviewer comment (Experiments):
Only rank-2 order-3 model is considered, which significantly reduces the breadth of application of the method and, as a consequence, its practical value.

Author response:
**Application.** We agree with the reviewer that the rank-2 order-3 model limits the breadth of applications of the considered model in general. However, this work only presents the rank-2 order-3 model for its accessibility and to show the potential of the considered analysis. For higher ranks/orders, a similar construction can be done (hints on that in Appendix G), which was previously developed for a similar model- Hotelling-type tensor deflation (See Theorem 3.4 in [1])

Reviewer comment (Experiments):
Weighted sum of 1-rank tensors is called CANDECOMP/PARAFAC tensor Decomposition (CP) and there is quite a large theoretical basis for it. Have you used or compared with other existing approaches for approximate tensor construction in CP?

Author response:
**Questions.** 
- We thank the reviewer for their great question! We have further benchmarked our proposed algorithm against prevalent spectral tensor decomposition methods that are widely used in practice, CANDECOMP/PARAFAC (CP) and Tucker. We show that the latter collapse in the presence of higher levels of correlation $\alpha$ and fail to recover the signal of interest. We outline the details of that in Figure 5 and its subsequent explanation in Section 4/Benchmarks. Initially, we have not considered this benchmark highly relevant, as we only considered spectral decomposition algorithms.

Reviewer comment (Theory):
Can your work be extended to other tensor decompositions such as Tensor Train (TT), Tucker and others?

Author response:
**Questions.** 
- Our work, for now, only considers spectral decomposition algorithms given the rich theoretical basis in RMT/RTT. Tensor Train (TT) and Tucker are, however, compression-based decomposition algorithms.

Reviewer comment (Writing):
Why did you not consider a (simpler) symmetric case where u = v = w for simplifying notations and exposing the main results more easily?

Author response:
For the symmetric case where $u=v=w$, this indeed seems a simpler case to consider. However, when considering such assumption, we impose further constraints on the structure of the noise tensor $\mathcal{W}$. The noise entries $\mathcal{W}_{i,j,k}$ are not i.i.d anymore and further steps would be needed to estimate the variance given a certain entry.

Reviewer comment (Theory):
Did you try to express the optimization problem in gamma as an objective function minimization problem? Can the problem benefit a joint optimization in both singular value / vectors and gamma?

Author response:
We thank the reviewer for bringing this point to our attention. Indeed, at the second deflation step of the $\gamma$-orthogonalized tensor deflation algorithm (in Appendix F, Algorithm 2), we are jointly optimizing $\gamma$ and the alignments. Furthermore, we empirically show that the different alignments are concave functions of $\gamma$ in Appendix E.4 and emphasize that in Section 3.4, which further explains the importance of our approach and provides theoretical guarantees on the quality of our recovery given $\gamma^*$.

Reviewer comment (Reproducibility):
According to the presented analysis, expanding this method to rank-$r$ and higher-order situations seems to be very cumbersome. Although the authors state that symbolic solvers can be used to solve such higher-order and higher-rank problems, it could be beneficial to provide some more details for the readers to follow.

Author response:
**Extension to Higher-Rank and Higher-Order Problems.** In addition to the Generalized Deflation Step Procedure in Section 3.1, we now motivate every step in our developed theory/algorithm in detail in the main text. We have considered the extension to higher ranks and/or orders in more detail in Appendix G, where we also list some of the key lemmas to producing our results. In addition, we point the reader to Theorem 3.4 in [1], where the extension follows a similar approach to ours and is done in full and in a high level of detail for the Hotelling-type deflation.

Reviewer comment (Presentation):
There are many figures in the text, but there are few related explanations. More explanations should be added to make such figures to be more meaningful.

Author response:
**Experiments Section.** We focused our experimental evaluation on 2 representative experiments and provide detailed explanations of both. In particular, we outline how our algorithmic approach performs compared to state-of-the-art and how our developed theory matches our predictions in practice with extensive experiments. In addition, we have also added separate visualizations for our theorems and assumptions throughout Section 3 and provided a detailed set of supplementary experiments in Appendix E.

Reviewer comment (Presentation):
The authors put too much stuff in the supplementary materials, which is not conducive for the readers' reading. For example, the proposed algorithms should be placed in the main text.

Author response:
**Outline of the Paper.** We substantially revisited our manuscript, as outlined as part of our general comment. The proposed algorithm is constructed and explained in a high level of detail in Section 3.

Reviewer comment (Experiments):
The introduction of the numerical experiments should be more detailed, such as how the simulated data is generated.

Author response:
**Experimental Setup.** We provided more details of our experimental setup in Section 4 (Experimental Setup). In addition, an easy-to-follow code sample is provided in Appendix E.1 to show exactly how our data is generated. A code repository will also be provided to ensure reproducibility.