Paper: A New Tensor Network: Tubal Tensor Train Network and its Applications (ICLR-2024)

Reviewer comment (Novelty):
Apart from the TTT model, nothing is new compared with the original TT. It would be good if the author could highlight the uniqueness from the existing models.

Author response:
Thank you for the critical comments. We agree that the TTT and TT models show similarities but the TTT model significantly differs from the TT decomposition since it uses a different operation (T-Product) to contract the core tensors. Furthermore, we want to highlight that the T-SVD for higher order tensors such as fourth-order color videos, fourth-order hyper-spectral videos, fifth-order light-field images, and sixth-order bidirectional texture functions, suffers from the curse of dimensionality problem. This limits its application in these important domains. For example, in the literature, the T-SVD was used to deal with such data tensors [1] but the TTT model is more efficient and attractive on a practical point of view since it mitigates the curse of limitation of the T-SVD in an efficient way. Moreover, the tensor train (TT) with convolution contraction was investigated in [2] and competitive results were reported for learning spatio-temporal features to train an LSTM neural network. Motivated by that work and the results achieved, we proposed to replace the classical tensor contraction with the T-Product. We believe these issues and the simulations done on a variety of applications show the efficiency of the proposed tensor model and are unique contributions of the paper.

Reviewer comment (Writing):
The writing should be carefully improved. For example, Definitions 3 and 4 define the identity and orthogonal tensors in the t-product context. But it should be clarified the ambiguity from the conventionally defined identity and orthogonal tensors.

Author response:
Thank you for the constructive comments. We have revised the paper to make it easier to follow. We added new appendices to explain in detail the tensor concepts such as T-Product. Also, we clarified the mentioned ambiguities. Please check the revised version of Section 2  (preliminaries).

Reviewer comment (Novelty):
The motivation of the proposed model is not well presented. The authors stated TTT addresses the curse of dimensionality issue of T-SVD. But what is the advantage compared to TT and other tensor network structures?

Author response:
Response. We thank you for the critical comments. Yes, we agree that the TTT model was developed mainly motivated by the limitation of the T-SVD model in handling the curse of dimensionality. However, in contrast to the TT or TC models, the core tensors in the TTT model are contracted via the T-Product. The simulation results on image compression and completion clearly show better performance of the proposed TTT model compared to the T-SVD and the TT decomposition.  Moreover, it is worth mentioning that the TT model with convolution contraction was investigated in [2] and competitive results were reported for learning spatio-temporal features to train an LSTM neural network. Motivated by that work and the results achieved we proposed to replace the classical tensor contraction with the T-Product. We believe these issues and the simulations done on a variety of applications show the efficiency of the proposed tensor model compared to the TT model and the T-SVD are unique contributions of the paper.

Reviewer comment (Writing):
The overall presentation is not clear enough. The preliminaries might be vague for readers not familiar with tensor decompositions. Moreover, the definition of the proposed model is not explicitly expressed. Besides, the review of related work is missing.

Author response:
Response. Thank you for the constructive comments. We tried to improve the presentation and make it easier for readers unfamiliar with tensor concepts. In particular, we added a new appendix A.1 to explain the T-Product more clearly and how it is related to the DFT.

Reviewer comment (Experiments):
For empirical evaluations, the authors only compare with TT and T-SVD, both of which are very old and classical models. It should be encouraged to choose more recent baselines.

Author response:
Response. Thank you for the critical comment. We compared our proposed model with the Tensor Chain (TC) which is known to be better than the TT model. The results still show better results of the proposed model than the TC model. Please check the Appendix A.3.

Reviewer comment (Reproducibility):
The authors proposed two algorithms, TTT-SVD and TACTU, but they did not claim which one was used for experiments. Moreover, they did not mention which algorithm was used for TT.

Author response:
Response. Thank you for your insightful feedback. We appreciate your observation. In our experiments, both TTT-SVD and TACTU algorithms were implemented. However, we found that TACTU consistently yields TTT models with significantly lower TTT-ranks compared to the TTT-SVD algorithm. Our simulation results demonstrated that, similar to the TT-SVD algorithm, TTT-SVD does not guarantee the production of a tensor with a minimum total TTT-rank or the minimum number of parameters. It often results in models with unbalanced ranks. Consequently, we exclusively reported results obtained using the TACTU algorithm, given its more favorable outcomes. Regarding the comparison with the TT model, we utilized the ACTU algorithm [put reference here], which demonstrated superior performance compared to the TT-SVD algorithm. These details, along with the reasons for our choices, have been explicitly outlined in the paper to enhance transparency and clarity in our methodology.

Reviewer comment (Theory):
What is the complexity of the proposed algorithm and comparison with previous algorithms?

Author response:
Response. The TTT model scales well with the order of the input tensor since it does not suffer from the curse of dimensionality in contrast to the T-SVD. For an N-th order tensor $X$ of size $I \times I \times … \times I$, and the TTT ranks $(R,R,...,R)$ the complexity of the algorithm is dominated by the DFT of the tubes which is $O(I^{N-1} \log(I))$ and the T-SVD of a third order tensors which is $O(RI^N)$. Therefore, the complexity of the algorithm is $O(I^{N-1} \log(I)+RI^N)$. Notably, the computation can be accelerated using the randomization framework or cross approximation (CUR) as previously considered  in [4] for the TT decomposition. The memory complexity of the model is $O((N-3) I^2 R^2)$ to store $(N-3)$ core tensors of size $I \times R \times I \times R$. The computational complexities of the TT decomposition and the T-SVD are $O(NRI^{N})$ and $O((N-2)I^{N-1} \log(I)+RI^N)$, respectively. The memory complexity of the TT decomposition T-SVD are also $O((N-2) R^2 I)$ and $O(I^{N-1} R)$. So, the TTT model has lower computational complexity and its memory complexity is much better than the T-SVD while it is a bit higher than the TT decomposition. Please note we have used the randomized version of all algorithms in our implementations.

Reviewer comment (Evaluation):
In Figure 4, it seems that TTT discards color information for Kodim 15, Barbara and Airplane images. Can the authors give some insights on this?

Author response:
Response. Agreed, we acknowledge the observation. We believe that the loss of color information in specific images can be traced back to: 
1. the reshaping process: using different reshaping strategies may affect in practice the accuracy of the approximation w.r.t. different features such as the texture, the shapes, the gradients and the colors. 
2. As we pointed out in our responses to comments from Reviewer aedc, the differences in color information preservation probably stem from the contrasting approaches of TTT and TT when applied to color images. We think that these variations are attributed to the influence of the Discrete Fourier Transform (DFT). Indeed, in the Fourier domain, TTT tends to focus on higher-energy bands (higher magnitude coefficients), mainly within lower frequency ranges, while TT achieves a more uniform approximation of image features across various frequency bands. This emphasis on frequency components likely contributes to the divergent characteristics in feature approximation between TTT and TT. 

We're exploring modifications to address this discrepancy in future iterations of the TTT model, particularly focusing on adapting the methodology to better retain color information during the approximation process. One potential approach includes integrating adaptive weights tailored to different frequency bands.

Reviewer comment (Writing):
The notations or preliminaries are not adequately introduced, especially for readers who are not familiar with tensors. I suppose the authors use $\ast$ to denote the T-product. However, they did not introduce this notation explicitly.

Author response:
Response. Thank you for the critical comments. We agree with this comment and added more necessary information about the T-Product. More details were also added in the appendix A.1. We also tried to improve the presentation and made it easier for readers unfamiliar with tensors.

Reviewer comment (Writing):
The definition of frontal slices are not introduced in Definition 5.

Author response:
Response. Thank you for the useful comment. We modified the definition.

Reviewer comment (Writing):
In the first Equation of Section 3.1, the middle index of $X_{N-1}$ might be $i_{N-1}$.

Author response:
Response. Thank you for this comment. Yes, this is true and we modified the formula.

Reviewer comment (Presentation):
The presentation can be improved. The paper might be hard to follow for non experts.

Author response:
Thank you for the critical comment. We revised te paper carefully and tried to improve the presentation and make it easier for non expert readers. In particular, we added a new appendix to explain in detail the tensor concepts such as T-Product so that the readers can understand the topic easily, please check the appendix A. 1.

Reviewer comment (Reproducibility):
The computational cost and scalability of the proposed method, and few other details are not clear.

Author response:
Thank you for the useful comment. The TTT model scales well with the order because it breaks the curse of dimensionality in contrast to the T-SVD. For an N-th order tensor $X$ of size $I \times I \times … \times I$, and the TTT ranks $(R,R,...,R)$, the complexity of the algorithm is dominated by the DFT of the tubes which is $O(I^{N-1} \log(I))$ and the T-SVD of a third order tensors which is $O(RI^N)$. SO the complexity of the algorithm is $O(I^{N-1} \log(I)+RI^N)$. This computation can be accelerated using the randomization framework or cross approximation (CUR) as was done for the TT decomposition in [4]. The memory complexity of the model is $O((N-3) I^2 R^2)$ so store $ (N-3)$ core tensors of size $I \times R \times I \times R$.

Reviewer comment (Presentation):
The presentation is difficult to follow and certain details about the method are not clear.

Author response:
Thank you for the critical comment. We tried to improve the presentation and explain the proposed model more clearly.

Reviewer comment (Writing):
The main section 4 will be difficult to follow for readers unfamiliar with tensor methods, particularly the T-product. It is not clear how the factor tensors interact with each other to form the full N-order tensor. The T-product between a 4th-order tensor and the method of multiplying a third tensor with a fourth-order tensor contracted using T-product is not obvious.

Author response:
Thank you for the critical comment. We tried to improve the presentation and provide more information about the T-Product, please check the appendix A.1 and the revised version of the preliminary section. Regarding the second comment, please note that we don’t multiply a third order tensor with a fourth order tensor. Indeed, we multiply a third order tensor with a sub-tensor of order three a fourth order tensor.

Reviewer comment (Theory):
It is unclear how to choose the TTT-rank vector for a given N-order tensor. Are these ranks unique for a given tensor? Are they related to the tubal rank corresponding to T-SVD?

Author response:
Thank you for your insightful comment. Much like the TT-rank or the Tucker rank, determining the optimal TTT-rank poses a challenge. Similar to TT-ranks, TTT-ranks are not unique. It's worth noting that the TTT-rank is associated with the tubal rank, and the elements of TTT-ranks represent the tubal rank of the reshaped form of the underlying tensors, as illustrated in Figure 13 for further clarification.

Reviewer comment (Theory):
It is unclear how to show that the space of all tensors of TTT-rank no higher than a given r_k is closed.

Author response:
It is known that the collection of tensors of TT-rank no higher than r_k forms a closed subset [1]. The tensors in the TTT format have a similar structure as the TT decomposition where instead of the SVD, the T-SVD is used. On the other hand, the T-SVD and SVD have similar optimality properties. So from these two points, it can be easily proved that the space of tensors of TTT-rank no higher than a given r_k is closed.

Reviewer comment (Reproducibility):
The meaning of "best low TTT rank" is unclear, as well as how to compute this for a given tensor.

Author response:
The best low TTT rank is defined as the following optimization problem 

$\min   ||X-Y||_F$    subject to  TTT-rank(Y)=R

where $Y$ is in the TTT format. This minimization problem can be solved via Alternating Least Squares (ALS) or the density matrix renormalization group (DMRG) technique.

Reviewer comment (Reproducibility):
In Algorithm 1, the output says that an approximation with error tolerance \(\epsilon\) is returned. But \(\epsilon\) is not an input parameter. It is unclear how to control the error, and this might be a typo.

Author response:
We thank you for this comment. Yes, we agree, this is a typo. The $\epsilon$ will be removed.

Reviewer comment (Experiments):
The computational cost and scalability of the proposed method is not clear.

Author response:
In our simulation results, the runtime of the TTT model was observed to be less than that of the TT model. This discrepancy can be attributed to the fact that each iteration of the TTT model necessitated a comparatively lower TTT rank than the TT model. The reduced complexity in determining the TTT rank contributes to the overall shorter runtime in our specific simulations.

Reviewer comment (Evaluation):
It is unclear why the runtime of TTT is less than the TT method; one might expect T-SVD computation to be more expensive than SVD.

Author response:
In our simulation results, the runtime of the TTT model was observed to be less than that of the TT model. This discrepancy can be attributed to the fact that each iteration of the TTT model necessitated a comparatively lower TTT rank than the TT model. The reduced complexity in determining the TTT rank contributes to the overall shorter runtime in our specific simulations.

Reviewer comment (Experiments):
The advantage of TTT in the applications considered is not really clear and seems a bit forced. Reshaping the images into a 10th-order tensor seems strange, and the choice of the rank values in the TTT-rank vector seems arbitrary. It is questioned whether considering the images (in all four examples) as 3rd-order tensors and using T-SVD would be sufficient. A comparison with this approach could be interesting to show that TTT achieves something different or better.

Author response:
Thank you for your insightful comment. Preserving the multidimensional structures of images and videos is paramount. The reshaping into 10th order tensors aims not to underscore the value of reshaping alone but to simulate real-life scenarios involving high-order input tensors. This allows us to assess tensor networks' performance in handling complex data structures. We acknowledge the suggestion to compare TTT with T-SVD on the natural 3rd order form. This comparison will be incorporated into our analysis to provide a more comprehensive understanding of the differences between TTT and T-SVD in these applications.
Furthermore, the usefulness of reshaping in achieving higher compression ratios has been established in prior research [2,3]. Notably, our primary focus is not on the reshaping technique, leveraging well-known efficient methods to achieve this.
We aim to delve into the contrasting approximation approaches of TTT and TT when applied to color images, attributing these differences to the influence of the Discrete Fourier Transform (DFT). In the Fourier domain, the TTT method notably accentuates and approximates higher-energy bands, predominantly situated within lower frequency ranges. Conversely, TT achieves a more uniform approximation of image features across various frequency bands. This distinct emphasis on frequency components likely underlies the divergent characteristics in feature approximation between TTT and TT. Our future works will involve proposing modified versions of the TTT model to address this discrepancy. One potential approach involves integrating adaptive weights tailored to different frequency bands.

Reviewer comment (Experiments):
It is unclear why a 10-order tensor is considered and what happens to the performance if a lower order is chosen.

Author response:
We have considered a tensor of high order to better show the performance of the TTT model, which is well suited for tensors of high order. We should emphasize that for tensors of lower order the performance of the TTT model compared to the T-SVD is reduced.

Reviewer comment (Writing):
In Section 3.2, "middle tensor if" should be corrected to "middle tensor is".

Author response:
Thank you for finding this typo. We corrected it.

Reviewer comment (Experiments):
Recently, \(\star_M\) product, a generalization of the T-product where FFT is replaced by a general invertible matrix M, has been proposed. Perhaps TTT can be extended using this general version of the tensor product. For image applications, it has been shown that using the DCT matrix as M performs better than FFT.

Author response:
Thank you for the useful suggestions. We will incorporate the generalized T-product and DCT operations in our future works while comments on these issues will be added in the paper.

Reviewer comment (Writing):
It is better to show an intuition behind TTT decomposition for better understanding.

Author response:
Thank you for the insightful comment. 
The main intuition behind the TTT decomposition is to express a high-dimensional data tensor as a sequence of convolution-like products of lower-dimensional core tensors. The core tensors are often of order 2, 3, or 4, and they capture the essential features and interactions of the data. The convolution-like operator used in the TTT decomposition is the tubal product, which is a generalization of the circular convolution to tensors.
To illustrate the TTT decomposition, we first introduce some new concepts and notations. We define a matrix of size $I \times T$ as a hyper-vector of length $I$, i.e., its elements are vectors (tubes) of length $T$. We call this a hierarchical vector, because it has two levels of structure: the vector level and the tube level. We also define the tubal-outer product of hyper-vectors, $a_n$ of length $I_n$ and tube length $T$, which yields a hyper-tensor, $Y$ of size $I_1 \times I_2 \times \cdots \times I_N$
$$Y = a_1 \ast a_2 \ast \cdots \ast a_N$$
where the elements of $Y$ are tubes of length $T$ given by
$$Y(i_1,i_2, \ldots, i_N) = a_1(i_1) \circledast a_2(i_2) \circledast \cdots \circledast a_N(i_N)$$
where $\circledast$ denotes the modulo-$T$ circular convolution of tubes.
The hyper-tensor $Y = a_1 \ast a_2 \ast \cdots \ast a_N$ represents a rank-1 tubal tensor, which is the simplest form of a tubal tensor. The TTT decomposition represents a general tubal tensor, $Y$, as a sum of rank-1 tubal tensors constructed from core hyper-tensors, $A_n$, of size $R_{n} \times I_n \times R_{n+1}$, where $R_1 = R_{N+1} = 1$
$$Y(i_1,i_2, \ldots, i_N) = \sum_{r_1 = 1}^{R_1} \sum_{r_2 = 1}^{R_2} \cdots \sum_{r_{N+1} = 1}^{R_{N+1}} A_1(r_1,i_1,r_2) \ast A_2(r_2,i_2,r_3) \ast \cdots \ast A_N(r_{N},i_N,r_{N+1})$$
or equivalently, as the tubal product of hyper-matrices $A_n(:,i_n,:)$:
$$Y(i_1,i_2, \ldots, i_N) =  A_1(1,i_1,:) \ast A_2(:,i_2,:) \ast \cdots \ast A_N(:,i_N,1)$$
In this sense, the TTT decomposition is interpreted as a Tubal-Matrix Product State, a tensor decomposition that operates on the tubal product. The TTT decomposition can reduce the storage and computational complexity of the data tensor as the TT model, while exploiting the convolution properties in one dimension of the data, revealing its latent structure and patterns.

Reviewer comment (Theory):
In theory, why can TTT decomposition surpass TT decomposition? The theoretical analysis is insufficient.

Author response:
The TTT decomposition can surpass the TT decomposition when the data tensor has a convolutive structure in one of its modes, which means that the latent features are mixed by the circular convolution instead of the linear combination. The TTT decomposition preserves the mode where the circular convolution operates, and decomposes the other modes into lower-dimensional core tensors. This way, the TTT decomposition can capture the convolutive feature patterns of the data tensor more effectively, and provide better reconstruction and compression.
The TT decomposition, on the other hand, decomposes all the modes of the data tensor into lower-dimensional core tensors, regardless of the structure of the data. This can result in a loss of information and accuracy, especially when the data tensor has a high order or a large rank. The TT decomposition assumes that the latent features are linearly combined in all the modes, which may not be true for some types of data.
Therefore, the TTT decomposition can surpass the TT decomposition when the data tensor admits the TTT model, which is more suitable for data with a convolutive structure in one of its modes.

Reviewer comment (Evaluation):
It is better to report the error bar for the experimental results.

Author response:
Thank you for the suggestion. The Mean Squared Errors were reported for the image compression task (Example). We will report the errors in our experimental results. However, the relative errors are reported below fo images.As can be seen, in all cases the proposed TTT model achieves a lower relative error.

               Airplane     Barbara      Kodim02     Kodim03      Kodim04     Kodim15      Kodim20      Kodim23    
              1.082e-01    1.496e-01    1.500e-01    1.466e-01    1.460e-01   1.485e-01    1.012e-01    1.497e-01 (TTT)
              1.499e-01    1.500e-01    1.500e-01    1.510e-01    1.510e-01   1.499e-01    1.497e-01    1.500e-01 (TT)

Reviewer comment (Writing):
$I_N$ lost $I$ in the last few lines of Section 3.2.

Author response:
Thank you for finding this inconsistency. We solved this problem.

Reviewer comment (Evaluation):
Why the compression ratio of the proposed method is lower than that of TT-based model in “News qcif” dataset?

Author response:
While the proposed model may not consistently outperform the TT model, as demonstrated in this specific instance where the TT model yielded superior results, it's central to highlight that the compression ratio of different methods can vary for video compression. Several factors, including codec variations, video content type, and quality settings, contribute to these variations. The TT model and the TTT decomposition employ distinct contraction operators between core tensors, leading to different compression approaches. It raises an intriguing question: what factors contribute to the TTT model's lower compression ratio for the New QCIF model? Investigating this aspect could be a valuable avenue for future research, albeit one that demands more time and effort.

Reviewer comment (Experiments):
You claim that “The key difference between the TT-SVD and the TTT-SVD is the first works on unfolded matrices, while the latter deals with reshaped form of the underlying tensors, which are of order three”. Thus, can a decomposition deals with reshaped form of the underlying tensors with order greater than three achieve even better performance?

Author response:
Yes, we agree that the TT-SVD and TTT-SVD are closely related to each other where they are working on different reshaped forms of the underlying tensors. However, it is important to recall that the contract operation between core tensors in the T-SVD and the TT model are different. The simulation results show that it is possible to achieve better results than the TT model.