Paper: A-Loc: Efficient Alternating Iterative Methods for Locating the $k$ Largest/Smallest Elements in a Factorized Tensor (ICLR-2024)

Reviewer comment (Novelty):
The paper takes known algorithms, and only connects them together in the special case of the problem of finding the extremum of CP-tensors.

Author response:
One of the contributions of this work is to transform the problem of finding the largest element of a factorized tensor into a continuous optimization problem, i.e., Theorem 1, which makes it possible to design algorithms for finding the largest element of a factorized tensor from a continuous optimization perspective. More importantly, our proposed continuous optimization model is applicable to other formats such as Tucker and tensor-train. For example, if the tensor is given in Tucker format, i.e., $\mathcal{A}=\mathcal{G}\times_{1}U^{(1)}\times_{2}U^{(2)}\cdots\times_{N}U^{(N)}$, then the corresponding optimization problem can be written as 
$$\max\limits_{x^{(1)},x^{(2)},\cdots,x^{(N)}}\mathcal{G}\times_{1}(x^{(1)T}(U^{(1)}\odot x^{(1)}))\times_2(x^{(2)T}(U^{(2)}\odot x^{(2)}))\cdots\times_{N}(x^{(N)T}(U^{(N)}\odot x^{(N)})),\ \text{subject to}\ ||x^{(n)}||_2=1\ \text{for all}\ n=1,2,\cdots,N.$$

Reviewer comment (Experiments):
Comparisons in numerical examples are given with only two (and in the case of finding the minimum, only one) other method, which is clearly insufficient.

Author response:
We have listed and compared the algorithms in the existing works. If there are any omissions, please point them out.

Reviewer comment (Experiments):
Table 1, which compares methods for random vectors, is not entirely honest. Indeed, the *star sampling* method can only find maximum (modulo) elements of CP-tensor. But once one is found, let its values be \( m \), we can do a transformation \( m \mathcal E - \mathcal A \) (very similar to the one described on page 2 in section 2 of the paper), which increases the CP-rank by only 1. Then the problem of finding the minimum in the initial tensor \( \mathcal A \) reduces to finding the maximum in this tensor. Unfortunately, I haven't found open source code for the *star sampling* method, but I suspect it would work (with this trick) just as well as the presented method. And in the problem of maximum finding the *star sampling* algorithm, as the authors honestly write, already wins, so the contribution made by this new approach may be considered negligible in this case.

Author response:
In the star sampling method, the top-k elements are obtained from the samples directly, but it may have too few effective samples that only include the largest element. Therefore, for a fair comparison, we use the same way to find the top-k elements in our experiments. Although the star sampling method is more accurate for random tensors, it requires many samples and is less efficient. Additionally, experiments in Sec. 5.2 and 5.3 show that the star sampling method is not always accurate.

Reviewer comment (Experiments):
As for finding optima of the Rastrigin and Schwefel model functions, this and many other functions were used to test a similar algorithm optima_tt, which is cited in this paper as Chertkov et al. (2022). If we take the *optima_tt* code, which is open source, and apply it to TT-tensors that exactly describe the two functions (with exact TT-ranks of 2 each) with the same parameters (n=4096, d=10), we get the following results for the minimum and for the maximum (I took the hyperparameter \( k=5 \)):  
Rastrigin: min: 0.0014192582... max: 403.5327475...  
Schwefel min: 0.000725485.... max: 8379.65727....  
Thus, for these two functions the combination of using CP-compression and the newly presented method is meaningless.

Author response:
The CP formats of Rastrigin and Schwefel can be directly obtained by Eq. (5.1), and their CP-ranks are both $d+1$. Since CP and TT formats can be converted to each other, they can also be expressed in TT format. As mentioned before, our proposed algorithms are also applicable to the tensor given in TT format due to the generality of the proposed continuous optimization model.

Reviewer comment (Theory):
In the proof of Theorem 1, in the last equation in (A.1) it is not clear what "\( x^{(n)}=1 \)" means. If it is a constant vector, how can we search for \( \max \) w.r.t. it? If it \( \||x^{(n)}\||_2=1 \) then (A.1) is not just equivalent (as the proof of the theorem states), but coincides with problem (2.3). But Theorem 1 asserts more than that. It asserts that for continuous values of \( x^{(n)} \), i.e., \( x^{(n)} \in \mathbb{R}^{I_n} \), the solution of the maximization problem (2.3) coincides with the maximum of CP-Tensor element, i.e., with the case where each vector \( x^{(n)} \) is binary and represents (different) rows of the unit matrix. In this form, Theorem 1 has never been proved in the paper.

Author response:
The key to Theorem 1 is that we observe the tensor corresponding to the solution is rank one, thus the problem (2.2) can be further rewritten as (2.3).

Reviewer comment (Experiments):
Question — Have you tried scaling the method to significantly higher dimensions (\( N=100,1000 \)), which overcome the curse of dimensionality?

Author response:
The goal of experiments in Sec. 5.3 is to verify the feasibility of our proposed algorithms, thus we use the data that can be stored in a laptop. However, for the higher-order tensor in practical applications, it can not be explicitly stored, and our proposed algorithms still work due to the fact that the time and memory cost of each iteration of them are $\mathcal{O}(RI_n)$, which is linear with the order $N$.

Reviewer comment (Theory):
The symmetric tensor eigenvalue formulation of the problem is used, but different factor matrices are optimized by alternating optimization. In the symmetric tensor eigenvalue problem, each component of the rank-1 factorization is typically set to be the same. Symmetry is mentioned in the problem definition in Section II, but never thereafter.

Author response:
We think it necessary to state that we consider the problem of finding the largest/smallest elements of a factorized tensor in this work, which can be equivalently transformed into a symmetric eigenvalue problem, where the symmetric matrix $A\in\mathbb{R}^{n_1n_2\cdots n_d\times n_1 n_2\cdots n_d}$ is a diagonal matrix consisting of the entries of the tensor $\mathcal{A}$. Based on this, we further combine the rank-one structure of the eigenvector to propose a continuous optimization model and use an alternating iterative method to solve it.

Reviewer comment (Theory):
The proof of theorem 1 does not appear complete or correct. Convergence is considered based only on the objective function, which is convergent for any method that has a nondecreasing and bounded objective. Hence, the contrast to convergence of standard alternating least squares does not make sense. See Uschmajew, A. (2012). Local convergence of the alternating least squares algorithm for canonical tensor approximation. SIAM Journal on Matrix Analysis and Applications, 33(2), 639-652.

Author response:
The goal of Theorem 1 is to show the equivalence of finding the largest element of $\mathcal{A}$ and the proposed continuous optimization problem, which can be easily proved due to the rank-one structure of the eigenvector. Our convergence analysis involves two parts. The first one is global convergence, i.e., Theorem 2, which shows that Algorithm 1 converges to a stationary point, which matches the theory established in B. Chen, et al., SIAM Journal on Optimization, 22(1):87–107, 2012. The second one is local convergence, which requires a very careful analysis of the property of the objective function at the stationary point, just like A. Uschmajew analyzes the convergence of alternating least squares algorithm for canonical tensor approximation.

Reviewer comment (Theory):
Theorem 1 formulates an equivalent optimization problem for finding the largest element in a tensor, which is a crucial step in the authors' method. However, the paper does not provide an in-depth analysis of this equivalent problem. For example, it would be beneficial to understand the number of minimizers that this problem may have. Such analysis could provide insights into the complexity and potential pitfalls of the optimization problem, and could be critical for the algorithm's performance.

Author response:
Theorem 1 is one of the main contributions of this work, which makes it possible to design algorithms for finding the largest element of a factorized tensor from a continuous optimization perspective. Analysis of the number of minimizers is equivalent to the analysis of the convergence speed, which is very challenging for the alternating iterative algorithm in the field of optimization. Even for the alternating iterative algorithm for linear problems, i.e., Gauss-Seidel, the estimation of its convergence rate is also very difficult.

Reviewer comment (Theory):
Theorem 2 shows that the proposed algorithm's subsequence converges to a stationary point of the optimization problem. However, it's unclear whether the entire sequence converges, and more importantly, whether it converges to the optimal solution. The absence of these guarantees may limit the algorithm's reliability and effectiveness in practice.

Author response:
Since the corresponding objective function sequence is monotonic, if there is a subsequence converges to a stationary point of the optimization problem, the objective function sequence converges to the value corresponding to the stationary point. Theorem 2 is a global convergence theory, we cannot guarantee that the alternating iterative method converges to the global optimum, which matches the theory established in B. Chen, et al., SIAM Journal on Optimization, 22(1):87–107, 2012. If the initial point is sufficiently close to the optimal solution, then the stationary point that the alternating iterative method converges to is the optimal point, see Theorem 3.

Reviewer comment (Theory):
While Theorem 3 establishes a linear convergence rate to the optimal solution, it does so under the assumption that the initial point is sufficiently close to the optimal solution. Regrettably, the paper falls short in providing a discussion on how to obtain such an initial point. Furthermore, the paper does not explore the potential existence of multiple optimal solutions, a factor that could significantly affect the algorithm's convergence behavior.

Author response:
Since the corresponding objective function sequence is monotonic, if there is a subsequence converges to a stationary point of the optimization problem, the objective function sequence converges to the value corresponding to the stationary point. Theorem 2 is a global convergence theory, we cannot guarantee that the alternating iterative method converges to the global optimum, which matches the theory established in B. Chen, et al., SIAM Journal on Optimization, 22(1):87–107, 2012. If the initial point is sufficiently close to the optimal solution, then the stationary point that the alternating iterative method converges to is the optimal point, see Theorem 3.