Fused Orthogonal Alternating Least Squares for Tensor Clustering
Abstract
We introduce a multi-mode tensor clustering method that implements a fused version of the alternating least squares algorithm (Fused-Orth-ALS) for simultaneous tensor factorization and clustering.
The statistical convergence rates of recovery and clustering are established when the data are a noise contaminated tensor with a latent low rank CP decomposition structure.
Specifically, we show that a modified alternating least squares algorithm can provably recover the true latent low rank factorization structure when the data form an asymmetric tensor with perturbation.
Clustering consistency is also established.
Finally, we illustrate the accuracy and computational efficient implementation of the Fused-Orth-ALS algorithm by using both simulations and real datasets
Introduction
Tensors, or multidimensional arrays, have played an essential role in a wide range of scientific and business applications, including neuroscience [9, 16], social networks [18, 4] and recommendation systems [3, 28].
As higher order tensors have been extensively used as a framework for storing and organizing massive data, the associated need to develop methods for tensor multi-mode clustering has sharply increased.
Several methods have been proposed for matrix biclustering for columns and rows simultaneously [6, 5, 7, 22].
However, those methods cannot be directly applied to tensors due to the complex higher order generalizations of the matrix singular value decomposition (SVD) and principal component analysis (PCA).
Motivated by the methodology gap between low and high dimensional arrays, we investigate here the multi-mode clustering problem for tensors.
We propose the Fused Orthogonal Alternating Least Squares (Fused-Orth-ALS) algorithm for uncovering co-clustering structures in high order tensors.
Clustering along a specific mode is equivalent to clustering the slice matrices for that mode as shown in Figure 1.
The core idea is to model true cluster means as having a rank-1 matrix basis.
The method is designed to systematically investigate and achieve the following.
First, under mild assumptions on the tensor decomposition structure, our approach encourages smoothness in the decomposed components, leading to clustering performance with higher accuracy.
Second, orthogonality is pursued for the decomposed components, avoiding the problem of local-minima convergence of the classical ALS tensor decomposition algorithm, preventing multiple recoveries of the same factors, and achieving faster convergence rates compared to popular tensor decomposition methods.
Third, theoretical guarantees are provided for the recovery and clustering consistency with even a single tensor sample, which is difficult to achieve in vector or matrix clustering.
The results in synthetic and real data demonstrate the robustness of our approach under mild model misspecification and assumption violations, and the effectiveness under high dimensional settings for extracting underlying clustering structures.
Even though we focus on order three tensors, it is straightforward to derive analogous results for higher order tensors.
Related work.
Multiple papers are closely related to, but also clearly distinct from, our proposed methodology for tensor multi-mode clustering.
Tensor clustering utilizing tensor block model appears in Chi et al.
[8] and is further developed in Wang and Zeng [26].
A popular alternative to the tensor block model is the CANDECOMP/PARAFAC (CP) decomposition that captures structure using the sum of rank-1 tensors.
Our paper adopts the CP decomposition because it handles heterogeneity in each mode, learns the clustering patterns across different modes of data in a more independent way, and provides flexibility for clustering a certain mode of the tensor without being affected by correlation with other modes.
Our method is similar to those in a recent series of papers [27, 21] that use the CP decomposition structure.
Note that their estimation algorithms use the framework of tensor power method [1].
In contrast, our algorithm utilizes Alternating Least Square (ALS), a widely employed decomposition algorithm
The ALS algorithm estimates the CP decomposition by specifying factor matrices simultaneously, instead of the column-by-column recovery for each factor matrices implemented in tensor power methods.
The ALS algorithm has been proved to be robust and computational efficient [10, 12, 15].
In particular, compared to Wang et al.
[27] which performs clustering over estimated factor matrices, our method encourages smoothness in the factor matrices by imposing a generalized LASSO penalty, resulting in apparent clustering structures in factor matrices since data from same cluster will possess the same value by choosing appropriate tuning parameters.
In addition, one critical assumption in Sun and Li [21] is that samples from the same cluster are ordered consecutively, an assumption that is rarely valid in applications.
Our approach employs a generalized LASSO regularization that is based on a graphical distance intuition learned from initial observations.
We demonstrate that the generalized LASSO penalty achieves better clustering performance when clustering assignments are randomly distributed.
Even though FusedOrth-ALS is motivated by the Orth-ALS algorithm [19], we make several advances.
First, Orth-ALS only considers tensor decomposition performance under the noiseless situation, and we successfully provide statistical guarantees for Fused-Orth-ALS under any error model, with a mild constraint on error tensor spectral norm.
Second, theoretical properties for Orth-ALS are established only for symmetric tensors, while Fused-Orth-ALS can achieve both recovery and clustering consistency for any asymmetric tensors.
Notations.
Calligraphic letters such as Y ∈ R d 1 × d 2 × d 3 are used to denote order three ( d 1 , d 2 , d 3 ) - dimensional tensors.
By convention, we use a colon to indicate all elements of a mode.
Thus, mode-1, mode-2, mode-3 slices of Y can be represented as Y i :: , Y : j : , Y :: k .
Bold uppercase and lowercase letters are used to represent matrices and vectors respectively, such as A ∈ R d 1 × d 2 and u ∈ R d 1 =
[ u 1 , ..., u d 1 ] ⊤ .
The i th row and j th column of A are denoted as A i : and A : j .
For notational convenience, we define the multilinear combination of tensor with three vectors u 1 , u 2 , uas Y ( u 1 , u 2 , u 3 ) =
(cid:80) i,j,k u 1 ,i u 2 ,j u 3 ,k Y ijk .
In particular, the spectral norm of an order three tensor
Y is defined by ∥Y∥ = max ∥ u 1 ∥ 2 = ∥
u 2 ∥ 2 = ∥ u 3 ∥ 2 =1 (cid:80) i,j,k u 1 ,i u 2 ,j u 3 ,k Y ijk .
The Frobenious norm of Y is the natural generalization of the Frobenius norm of a matrix, ∥Y∥ F = (cid:113)(cid:80) i,j,k Y 2 ijk .
Lastly, [ K ] := { 1 , 2 , ..., K } stands for the whole index set and we use ◦ , ⊙ to represent outer product and
Khatri–Rao product.
For two positive sequences { a n } , { b n } , a n ≲ b n means a n ≤ Cb n for some constant C > 0 independent of n .
Clustering model via tensor decomposition with regularization
Given observed tensor Y ∈ R d 1 × d 2 × d 3 that is a noisy version of a tensor of interest Y ∗ , i.e. Y =
Y ∗ + E , our goal is to uncover the clustering structures for the three modes of Y ∗ .
We assume that Y ∗ is a tensor with a rank K CP decomposition structure Y ∗ = (cid:80)
i ∈ [ K ] w i A : i ◦ B : i ◦ C : i , where A ∈ R d 1 × K , B ∈ R d 2 × K , C ∈ R d 3 × K are factor matrices with columnwise unit norm, i.e. ∥ A : i ∥ 2 = ∥
B : i ∥ 2 = ∥
C : i ∥ 2 = 1 , ∀ i ∈
[ K ] , and w =
[ w 1 , ..., w K ] ⊤ ∈ R K captures the weights of the factor matrices.
The construction of Y ∗ leads to dependence of clustering structures on the factor matrices along each mode.
Specifically, suppose there are s 1 clusters for the first mode, and each row in A can be expressed as A i : =
(cid:80) s 1 j =1 µ ⊤ 1 ,
j 1 i ∈
A ∗ j , where µ 1 ,j = [ µ (1 ,j ) , 1 , ..., µ (1 ,j ) ,K ] ⊤ is the mean value for rows in j th cluster and A ∗ j contains all the rows indices that belong to j th cluster over first mode.
Consequently, the true cluster means ¯ A 1 , ..., ¯ A s 1 for slices along first mode {Y 1:: , ..., Y d 1 :: } , as indicated in Figure 1, can be written as
[Equation]
This reveals the core idea of clustering via CP decomposition: all the true cluster means share the same K rank-1 matrix basis B : i ◦ C :
i , ∀ i ∈
[ K ] .
Thus, for every i , we would expect there exists s ∈ [ s 1 ] such that i ∈
A ∗ s and as a result, the clustering label for i th element along the first mode will be s .
As shown in Sun and Li [21], the tensor Gaussian mixture model can be viewed as a special case of this clustering structure.
However, finding the partitions A ∗ i , ∀
i ∈
[ s 1 ] is a combinatorial hard problem and could be computational intractable.
Inspired by regularization term proposed in Chi et al.
[8], we impose a full pairwise difference operator over rows of A to yield a weighted penalty on local differences,
[Equation]
where S :
= { ( i 1 , i 2 ) | i 1 < i 2 , i 1 ∈ [ d 1 ] , i 2 ∈
[ d 2 ] } is a set containing pairs of different row indices and parameters γ 1 = { γ 1 i 1 ,i 2 | ( i 1 , i 2 ) ∈ S} are non-negative weights that are critical for controlling the penalty imposed on pairwise row differences for A ; we will explain in detail how to choose them later.
Analogously, µ 2 ,j , B ∗ j , 2 ∆ , γ 2 and µ 3 ,j , C ∗ j , 3 ∆ , γ 3 can be defined to characterize the clustering structure and regularization along the second and third modes.
In summary, we propose the following penalized constrained optimization as an approach to reveal the latent clustering structure,
[Equation]
[Equation]
In particular, as λ increases, A i : will shrink towards each other, indicating pairwise differences of rows in A will become increasingly sparse.
Sparsity in pairwise row differences naturally leads to a partitioning assignment A ∗ j , ∀ j ∈ [ s 1 ] .
For simplicity, we choose λ to be the same over three modes.
We would like to draw the attention to the difference between the regularization term in (1) and that in Chi et al.
We use an ℓ 1 norm penalty which will yield pairwise difference equal todue to feature selection for ℓ 1 regularization, while Chi et al.
[8] adopts the Frobenious norm which only shrinks the pairwise difference to 0 gradually.
As mentioned, dynamic tensor clustering [21] used a fused LASSO penalty.
However, this simple fusion structure only works if samples from same cluster have consecutive indices.
Obviously, our weighted pairwise difference operator i ∆ is a generalization, and later we will show how our method outperforms dynamic tensor clustering when the cluster labels are randomly assigned.
Implementation: fused orthogonal alternating least squares algorithm 3.1 Algorithm
Starting from the alternating least square (ALS) algorithm
[14], we propose the fused orthogonal alternating least squares (Fused-Orth-ALS) method described in Algorithm 1 by adding two additional steps that are designed for multi-mode clustering.
First, the orthogonalization step is performed before each iteration of ALS, which allows for the avoidance of local optima and more rapid convergence to the true factors.
Moreover, a ’Fuse’ operator with generalized LASSO regularization ∆ is employed on each column of ALS estimates which in general is defined as
[Equation]
Algorithm
Input: Tensor Y ,
tensor CP rank K
Output: ˆ Y = (cid:80) Ki
=
ˆ w i
ˆ
A t : i ◦ ˆ B t : i ◦ ˆ C t : i
Initialize ˆ A 0 , ˆ B 0 , ˆ C 0 with columns randomly from the unit sphere and iteration index t = 0 ; while termination condition is not satisfied do t ← t + 1 ;
Find QR decomposition of ˆ A t − 1 , set ˆ A t − 1 = Q .
Orthogonalize ˆ B t − 1 and ˆ C t − 1 similarly;
X t ← Y (1) ( ˆ C t − 1 ⊙ ˆ B t − 1 ) , ˜ X t :=
[ ˜ X t : i ] Ki =1 with columns ˜
X t : i ← Fuse ( X t : i , 1 ∆ , λ ) ;
Y t ← Y (2) ( ˆ C t − 1 ⊙ ˆ
A t − 1 ) , ˜ Y t :=
[ ˜ Y t : i ] K i =1 with columns ˜
Y t : i ← Fuse ( Y t : i , 2 ∆ , λ ) ;
Z t ← Y (3) ( ˆ B t − 1 ⊙ ˆ A t − 1 ) , ˜ Z t :=
[˜ Z t : i ] Ki =1 , with columns ˜
Z t : i ← Fuse ( Z t : i , 3 ∆ , λ ) ;
Normalize ˜ X t , ˜
Y t , ˜ Z t with columnwise norm unit 1 and store the results as ˆ A t , ˆ B t , ˆ C t ; end
Estimate weights ˆ w i
=
Y ( ˆ A t : i , ˆ B t : i , ˆ C t : i ) , ∀ i ∈
[ K ]
We use the ’Fuse’ operator since the regularization term in (1) can be viewed as the sum of ’Fuse’ regularizations on each column of factor matrices, e.g., ∥ 1 ∆A ∥ 1 = (cid:80)
Ki =1 ∥ 1 ∆A : i ∥ 1 , and it can be solved efficiently via existing methods [2, 29].
After obtaining estimates ˆ A , ˆ B , ˆ C , clustering algorithm such as k -means or hierarchical clustering are used to obtain cluster assignment ˆ
A i , ˆ B j , ˆ C k , ∀ i ∈ [ s 1 ] , j ∈ [ s 2 ] , k ∈ [ s 3 ] .
From a computational complexity perspective, orthogonalization, ALS updates, and Fuse operations in
Algorithm 1 take O ( K 2 ( d 1 + d 2 + d 3 )) , O ( Kd 1 d
2 d 3 ) and O ( K (
d 31 + d 32 + d 33 )) number of operations respectively.
Thus, in a general case, the total computational complexity of Fused-Orth-ALS algorithm on a ( d 1 , d 2 , ..., d D ) -dimensional tensor is O
( K (max ( K (cid:80)
Dj =1 d j , D (cid:81)
Dj =
1 d j , (cid:80) Dj =1 d 3 j ))) .
When d j are of the same order d and tensor rank K
= o
( Dd D − 1 ) , the total complexity can be simplified to O ( KDd D ) which is the same order as classical ALS algorithm, meaning that extra orthogonalization and ’Fuse’ steps do not increase the computational complexity.
The proposed clustering model is clearly distinct from dynamic tensor clustering (DTC) [21], in terms of implementation of generalized LASSO regularization, ALS updates and added orthogonalization.
Our method has the same computational complexity as DTC but orthogonality leads to significant speedups in the iterations required for recovery convergence, which will be discussed in detail in section 5 later.
3.2 Regularization weights and tuning parameters
In practice, the choice of appropriate regularization weights γ 1 , γ 2 , γ 3 is critical since it affects clustering accuracy and computational efficiency.
For instance, γ 1 i 1 ,i 2 characterize the shrinkage of row difference between A i : and A j : .
From a graph perspective, each row of A can be treated as a node in an undirected graph and nodes i 1 , i 2 are connected by an edge with weight γ 1 i 1 ,i 2 .
All the edges compose the edge set S .
Clearly, large value of γ 1 i 1 ,i 2 implies high similarity between A i 1 : and A i 2 : , resulting in clustering nodes i 1 , i 2 into the same group.
In particular, our adopted 1 ∆ reduces to fused LASSO penalty utilized in DTC [21] in the case of a chain graph where
S = { i =
( i, i + 1) | i = 1 , 2 , ..., d 1 − 1 } .
A well-developed strategy [20, 5, 6] is espoused for choosing regularization weights: we perform a rank- K CP decomposition approximation to Y , utilize the estimated factor matrix ˆ A to quantify the ’distance’ among different nodes, and then calculate the regularization weights through
[Equation]
Here, ι ki 1 ,i 2 is an indicator function specifying whether ˆ A i 2 : is among k -nearest neighbors of ˆ A i 1 : that controls density of graph.
The smallest k that ensures the graph is connected is the default choice.
The remaining component in (2) is a Gaussian kernel with ν as a measure of scale chosen as median
Euclidean distance between the i 1 th and i 2 th rows that are k -nearest neighbors of each other.
Lastly, we normalize (cid:80) ( i 1 ,i 2 ) ∈S γ 1 i 1 ,i
= d 1 d 1 + d 2 + d 3 to ensure the penalty over three modes are on the same scale.
Similar formulas are used for γ 2 and γ 3 .
We choose λ through the use of the extended Bayesian Information Criterion [21, 26, 8] by minimizing
[Equation]
where df λ is the number of non-zero elements in estimated factor matrices, characterizing degrees of freedom.
Moreover, rank K and clusters s 1 , s 2 , s 3 are usually unknown and need to be estimated from data.
We use the ’elbow point’ method to choose rank K by plotting the recovery error, and determine s 1 , s 2 , s 3 by computing gap statistics [23].
Cluster recovery and convergence
This section provides a convergence analysis for the Fused-Orth-ALS algorithm.
For simplicity, we set d 1 = d 2 = d 3 = d and choose equal regularization weights over all modes.
Without loss of generality, we assume w max = w 1 ≥ w 2 ... ≥ w K = w min > 0 .
The technical assumptions needed for our analysis are as follows:
A1:
Define incoherence ρ = max i ̸ = j {|⟨ A : i , A : j ⟩| , |⟨ B : i , B : j ⟩| , |⟨ C : i , C : j ⟩|} ≤ α/ √ d for some α = polylog ( d ) and K =
o ( d 1 /
3 / polylog ( d ) 2 / 3 ) .
The spectral norm of Y ∗ satisfies ∥Y ∗ ∥ ≤ w max α .
A2:
Define the initialization error as ϵ 0 = max i {∥
ˆ A 0: i − A : i ∥ 2 , ∥ ˆ B 0: i − B : i ∥ 2 } which satisfies ϵ [Equation] ≤ min (cid:110)
[Equation]
with ξ is a constant upper bounded by 10 Kw max α/ ( w min √ d ) .
A
Denote spectral norm of E by ψ such that ψ ≤
Denote max j 3 ∆ 2 and define ϑ = max i ∥ [-
-]3 ∆C
i ∥ 1 that satisfies
[Equation]
Assumption A1 relaxes the requirements on the orthogonality of columns in factor matrices and imposes a weaker condition on the rank K than Orth-ALS algorithm [19].
Assumption A2 specifies initialization criteria that are needed only for computational issues.
The bounded perturbation is specified in A3.
In particular, it can be proved with high probability if elements in E are i.i.d from a sub-Gaussian distribution as stated in Tomioka and Suzuki [24].
The bounded fusion assumption, A4, restricts clustering complexity on factor matrices.
To illustrate its meaning, consider a simple scenario when all clusters along the third mode have the same size d/s 3 .
For C i : belonging to the same cluster C ∗ j , they should take the same value µ 3 ,j under our model assumption. Consequently,
A4 can be rephrased as ϑ = O ( d 1 . 5 (1 − 1 /s 3 )) ; in the special case when C has only one cluster, e.g. s [-
-]3 = 1 , A4 reduces to ∥ 3 ∆C ∥ 1 = 0 .
Thus, bounding ϑ is equivalent to bounding cluster size swhen sample size over third mode is kept fixed.
We can now derive a recovery error bound for the Fused-Orth-ALS algorithm.
Theorem 1 Assuming assumptions A1-A4 hold, factor matrix estimate
ˆ C : i , ∀ i ∈
[ K ] of Algorithmsatisfies the following error bound with high probability
[Equation]
when choosing appropriate λ .
Similar error bounds hold for the other two factor matrices estimates ˆ
A , ˆ B .
The error bound derived in Theorem 1 reveals how the weight w i , incoherence parameter ρ , rank K and perturbation level ψ interact with each other on affecting the convergence behavior of FusedOrth-ALS algorithm.
Lower perturbation level ψ , lower incoherence parameter ρ , rank K and lower signal ratio of w max /w min all result in lower error bounds.
Clearly, the error bound for each column in factor matrices has two parts: one is related to perturbation level ψ , the other is dependent on the underlying CP decomposition structure, signal ratio w max /w min , rank K and dimension d along each mode.
Thus, under high dimensional settings when d → ∞ , i.e., ρ 2 ≤ α 2 /d → 0 , the second part will dominate the error bound while if tensor data is almost noiseless, e.g. ψ ≈ 0 , the first part will be the main source of error.
The following corollary analyzes the relationship between these two parts under a special case when the error tensor follows a sub-Gaussian distribution.
Corollary 1 Assume assumptions A1-A4 hold.
If we further assume each element in error tensor,
E ijk , is independent, with zero mean and satisfies E [ e t E ijk ] ≤ e σ 2 t 2 2 , w max /w min ≤ C where C is positive constant and the minimal weight satisfies
[Equation]
then ˆ C : i , ∀ i ∈
[ K ] from Algorithm 1 satisfies the following error bound,
[Equation]
d with probability at least 1 − δ .
Same error bounds hold for the other two factor estimates ˆ A , ˆ B .
Thus, by imposing the sub-Gaussian distribution assumption on error tensor elements, recovery consistency for factor matrices estimated from Fused-Orth-ALS algorithm can be established.
Next, we demonstrate cluster consistency for the clustering algorithm performed on factor matrices recovered by Fused-Orth-ALS algorithm.
Since clustering algorithm is performed on rows of ˆ A , ˆ B , ˆ C , clustering error is quantified through the true mean value and its estimate ∥ µ 1 ,j 1 − ˆ µ 1 ,j 1 ∥ 2 , ∥ µ 2 ,j 2 − ˆ µ 2 ,j 2 ∥ 2 , ∥ µ 3 ,j 3 − ˆ µ 3 ,j 3 ∥ 2 .
Theorem 2 Assume assumptions in Corollary 1 hold, we have
[Equation]
hold with probability at least 1 − δ .
Furthermore, if min j 1 ∈ A ∗ m ,j ′ 1 ∈ A ∗ m ′ ,m ̸
= m ′ ∥ µ 1 ,j 1 − µ 1 ,j ′ 1 ∥ 2 ≳
K 1 .
5 /d , min j 2 ∈ B ∗ n ,j ′ 2 ∈ B ∗ n ′ ,n ̸ = n ′ ∥ µ 2 ,j 2 −
[Equation]
2 ,j ′ 2 ∥
[Equation]
µ 3 ,j
[Equation]
3 ∥ 2 ≳ K 1 .
5 /d , we have ˆ A m =
A ∗ m , ˆ B n = B ∗ n , ˆ C l = C ∗ l hold with probability at least 1 − δ .
This theorem shows that clustering consistency holds as long as K 1 .
5 /d → 0 , which allows rank K increase with the dimension d .
At first look, the conclusion in Theorem 2 seems to indicate that the cluster mean error bound does not depend on number of clusters over each mode, s 1 , s 2 , s 3 .
However, as we stated before, assumption A4 employs an "invisible" bound on the number of clusters which is closely related with the convergence rate derived in Theorem 2.
We provide a detailed explanation in supplementary material stating cluster mean error bound K 1 .
5 /d increases with the number of clusters s i .
Note that Theorem 2 assumes that true rank K is known; we leave the effect of an estimated rank ˆ K on clustering consistency for further study.
Numerical experiments 5.1 Synthetic datasets
In this section, we investigate the performance of the Fused-Orth-ALS algorithm on small samples and compare the recovery and clustering error with alternative tensor-based clustering methods.
Recovery error is evaluated through ∥ ˆ Y − Y ∗ ∥ F / ∥Y ∗ ∥ F .
Clustering error measures the frequency of mismatches between the estimated clustering assignment ˆ M and the true M over samples x 1 , ..., x n , which is defined as
[Equation]
and can be easily computed after obtaining the cluster assignment ˆ A m , ˆ B n , ˆ C l , ∀ m ∈
[ s 1 ] , n ∈ [ s 2 ] , l ∈ [ s 3 ] .
Note that the clustering error is similar to two other commonly used metrics: adjusted
Rand index [11] and variation of information [17].
For the simulations below, elements in the error tensor E are generated independently from a Gaussian distribution with mean 0 and variance σ 2 .
We compare Fused-Orth-ALS Algorithm with three methods: dynamic tensor clustering (DTC) [21],
CP-Kmeans algorithm which performs a rank- K CP decomposition on the tensor observations first and then independently applies k -means algorithm clustering to the rows of derived factor matrices, and multiway clustering for tensor block models (TBM)[26].
5.1.1 Single-mode clustering
We first generate an order three tensor with CP-rank 2 for which we would like to do single-mode clustering along the third mode.
The dimension of the first and second modes are set the same ( d 1 = d
2 = d ) and their unnormalized columns are
[Equation]
The third factor matrix with unnormalized & unshuffled columns is generated using
[Equation]
Then, the rows of C are shuffled randomly.
There are four clusters over the third mode with cluster means ( µ, − µ ) , ( µ, µ ) , ( − µ, µ ) , ( − µ, − µ ) respectively.
After normalizing the columns of A , B , C , we calculate the weights w i .
Even though this example is inspired from Sun and Li [21], shuffling the rows of C leads to additional clustering challenges since a more sophisticated regularization is required to capture the clustering features.
µ reflects cluster difficulty as it determines the distance between different clusters.
A smaller µ leads to a more challenging clustering problem.
For simplicity, the factor matrices used in this example, A , B , C , are orthogonal, satisfying assumption A1.
We set d = 20 , d 3 = 48 , µ = 1 , vary σ ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
and compare the performance of the four algorithms by reporting recovery and clustering errors.
As summarized in Figure 2 (a) and (b), Fused-Orth-ALS outperforms the other three methods under different values of perturbation, achieving the best recovery error and clustering error.
Not surprisingly, TBM performs worst due to model misspecification: the formulation of simulated factor matrices leads to an underlying cluster structure for mode three instead of a multiway block structure for all three modes.
Figure 2 (c) and (d) show that average running times are comparable for Fused-Orth-ALS and DTC where the orthogonalization step added in Fused-Orth-ALS still decrease number of iterations required for convergence.
Furthermore, to illustrate the advantage of generalized LASSO regularization imposed on Fused-Orth-ALS over fused LASSO penalty adopted in DTC, we vary the value for µ ∈ { 0 . 1 , 0 . 7 } to mimic different levels of clustering complexity and set d = 8 , σ = 0 .
001 , d 3 = 20 .
It is obvious from Table 1 that our method outperforms DTC especially as the ’signal’ µ becomes weak.
[Table]
[Figure]
To validate the performance on multi-mode clustering, we extend the simulation settings above by imposing clustering structures on all three factor matrices A , B , C , similarly to (3).
The corresponding choice of cluster means are: ( µ, 0 .
5 µ ) , ( − 0 .
5 µ, µ ) , (0 , − µ ) , ( − µ, 0) for mode 1, (0 , µ ) , ( − µ, 0) , (0 . 1 µ, − µ ) , ( − µ, − 0 .
1 µ ) for mode 2 and ( µ, µ ) , ( − µ, µ ) , ( µ, − µ ) , ( − µ, − µ ) for mode 3.
Accordingly, each mode has four clusters, i.e., s
= s
= s 3 = 4 of same size: ⌊ d 1 / 4 ⌋ , ⌊ d 2 / 4 ⌋ , ⌊ d 3 / 4 ⌋ respectively.
For example, for the first mode, there are ⌊ d 1 / 4 ⌋ rows of the factor matrix taking value ( µ, 0 .
5 µ ) , ⌊ d 1 / 4 ⌋ rows taking value ( − 0 .
5 µ, µ ) , ⌊ d 1 / 4 ⌋ rows taking value (0 , − µ ) and the rest rows taking value ( µ, 0) .
We fix d 1 = d 2 = 20 , d 3 = 40 , µ = 1 and vary σ ∈ { 1 , 2 , 3 , 4 , 5 } .
Since the detailed comparison between Fused-Orth-ALS and DTC has been illustrated above, we omit DTC (originally designed for dynamic single mode clustering of tensors).
Comparison results can be found in Figure 3.
It is clear that Fused-Orth-ALS algorithm outperforms the other two methods, especially under the noisy case when σ is large.
[Figure]
5.2 Real
Experiments are conducted on two real datasets: the brain node structural connectivity from Human
Connectome Project (HCP) 1 [25] and political relationships between nations 2 [13].
Details for choosing rank K and number of clusters are provided in supplementary material.
The HCP dataset is formatted as Y ∈ R 68 × 68 × 136 , and consists of brain connectivity among 68 brain nodes for 136 individuals.
Each entry takes on ordinal value { 0 , 1 , 2 } which indicates the strength level of connectivity { low, moderate, high } between different brain nodes.
Clustering results in Table 2 capture spatial connectivity between hemispheres of brain (the first character in the node name indicates the left or right hemisphere and the number in the parenthesis indicates the node count with same name).
Cluster I and II mainly capture connectivity in either left or right hemisphere while cluster III represents the cross-section connection between left and right hemisphere.
r.supramarginal and l.supramarginal are picked as smaller clusters - those regions are known to play a critical role in visual word recognition and reading.
[Table]
The Nations dataset is formatted as Y ∈ R 14 × 14 × 56 consisting of 56 political relationships ofcountries between 1950-1965.
Each entry represents the presence or absence of a political action, such as ’treaties’, ’send tourists to ’ between different nations.
Since 78.9% entries are zero in this dataset, we include an ℓ 0 penalized tensor block model (denoted as ’TBM-Sparse’) for comparison.
The clustering result in Table 3 for Fused-Orth-ALS algorithm assigned the nations into three clusters, one representing western-bloc countries (Cluster I), one communist bloc (Cluster III), one neutral bloc (Cluster II).
This result is consistent with the structure of political environment after world war
II.
In addition, the goodness-of-fit (proportion of variance explained) for five methods is provided in
Table 4.
For the nations dataset, Fused-Orth-ALS algorithm shows the highest variance explained.
[Table] [Table]
Conclusions
This paper studies the properties and convergence rates for a novel multiway tensor clustering algo- rithm, Fused-Orth-ALS.
The underlying CP decomposition structure imposes low-rank constraints on the tensor observation, and graph similarity based fusion regularization encourages smoothness on factor matrices, thus leading to automatic clustering.
The Fused-Orth-ALS algorithm can handle sparse and dense data tensors, and achieves clustering consistency even under model misspecification.
An open problem is developing theory for rank selection consistency.
Another interesting topic is on clustering consistency if there are missing entries in the data.
We leave these directions for further study.
There are no foreseeable negative social impacts of this work.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See supplementary materials.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Sectionand supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5 and supplementary material (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[Yes] See Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See Section 5.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]