Fused Orthogonal Alternating Least Squares for Tensor Clustering
Anonymous Author(s)
AffiliationAddress email
Abstract
We introduce a multi-mode tensor clustering method that implements a fused version of the alternating least squares algorithm (Fused-Orth-ALS) for simultaneoustensor factorization and clustering.
The statistical convergence rates of recoveryand clustering are established when the data are a noise contaminated tensor with alatent low rank CP decomposition structure.
Furthermore, we show that a modifiedalternating least squares algorithm can provably recover the true latent low rankfactorization structure when the data form an asymmetric tensor with perturbation.
Clustering consistency is also established.
Finally, we illustrate the accuracy andcomputational efficient implementation of the Fused-Orth-ALS algorithm by usingboth simulations and real datasets
Introduction
Tensors, or multidimensional arrays, have played an essential role in a wide range of scientific andbusiness applications, including neuroscience [9, 16], social networks [18, 4] and recommendationsystems [3, 28].
As higher order tensors have been extensively used as a framework for storing andorganizing massive data, the associated need to develop methods for tensor multi-mode clustering hassharply increased.
Several methods have been proposed for matrix biclustering for columns and rowssimultaneously [6, 5, 7, 22].
However, those methods cannot be directly applied to tensors due to thecomplex higher order generalizations of the matrix singular value decomposition (SVD) and principalcomponent analysis (PCA).
Motivated by the methodology gap between low and high dimensionalarrays, we investigate here the multi-mode clustering problem for tensors.
We propose the Fused Orthogonal Alternating Least Squares (Fused-Orth-ALS) algorithm for uncovering co-clustering structures in high order tensors.
Clustering along a specific mode is equivalentto clustering the slice matrices for that mode as shown in Figure 1.
The core idea is to model truecluster means as having a rank-1 matrix basis.
The method is designed to systematically investigateand achieve the following.
First, under mild assumptions on the tensor decomposition structure, ourapproach encourages smoothness in the decomposed components, leading to clustering performancewith higher accuracy.
Second, orthogonality is pursued for the decomposed components, avoidingthe problem of local-minima convergence of the classical ALS tensor decomposition algorithm,preventing multiple recoveries of the same factors, and achieving faster convergence rates comparedto popular tensor decomposition methods.
Third, theoretical guarantees are provided for the recoveryand clustering consistency with even a single tensor sample, which is difficult to achieve in vector ormatrix clustering.
The results in synthetic and real data demonstrate the robustness of our approachunder mild model misspecification and assumption violations, and the effectiveness under highdimensional settings for extracting underlying clustering structures.
Even though we focus on orderthree tensors, it is straightforward to derive analogous results for higher order tensors.
Related work.
Multiple papers are closely related to, but also clearly distinct from, our proposedmethodology for tensor multi-mode clustering.
Tensor clustering utilizing tensor block model appearsin Chi et al.
[8] and is further developed in Wang and Zeng [26].
A popular alternative to the tensorblock model is the CANDECOMP/PARAFAC (CP) decomposition that captures structure using thesum of rank-1 tensors.
Our paper adopts the CP decomposition because it handles heterogeneityin each mode, learns the clustering patterns across different modes of data in a more independentway, and provides flexibility for clustering a certain mode of the tensor without being affected bycorrelation with other modes.
Our method is similar to those in a recent series of papers [27, 21]that use the CP decomposition structure.
Note that their estimation algorithms use the frameworkof tensor power method [1].
In contrast, our algorithm utilizes Alternating Least Square (ALS), awidely employed decomposition algorithm
The ALS algorithm estimates the CP decompositionby specifying factor matrices simultaneously, instead of the column-by-column recovery for eachfactor matrices implemented in tensor power methods.
The ALS algorithm has been proved to berobust and computational efficient [10, 12, 15].
In particular, compared to Wang et al.
[27] whichperforms clustering over estimated factor matrices, our method encourages smoothness in the factormatrices by imposing a generalized LASSO penalty, resulting in apparent clustering structures infactor matrices since data from same cluster will possess the same value by choosing appropriatetuning parameters.
In addition, one critical assumption in Sun and Li [21] is that samples fromthe same cluster are ordered consecutively, an assumption that is rarely valid in applications.
Ourapproach employs a generalized LASSO regularization that is based on a graphical distance intuitionlearned from initial observations.
We demonstrate that the generalized LASSO penalty achieves betterclustering performance when clustering assignments are randomly distributed.
Even though FusedOrth-ALS is motivated by the Orth-ALS algorithm [19], we make several advances.
First, Orth-ALSonly considers tensor decomposition performance under the noiseless situation, and we successfullyprovide statistical guarantees for Fused-Orth-ALS under any error model, with a mild constrainton error tensor spectral norm.
Second, theoretical properties for Orth-ALS are established only forsymmetric tensors, while Fused-Orth-ALS can achieve both recovery and clustering consistency forany asymmetric tensors.
Notations.
Calligraphic letters such as Y ∈ R d 1 × d 2 × d 3 are used to denote order three ( d 1 , d 2 , d 3 ) -dimensional tensors.
By convention, we use a colon to indicate all elements of a mode.
Thus,mode-1, mode-2, mode-3 slices of Y can be represented as Y i :: , Y : j : , Y :: k .
Bold uppercase andlowercase letters are used to represent matrices and vectors respectively, such as A ∈ R d 1 × d
2 andu ∈ R
d 1 =
[ u 1 , ..., u d 1 ] ⊤ .
The i th row and j th column of A are denoted as A i : and A : j .
Fornotational convenience, we define the multilinear combination of tensor with three vectors u 1 , u 2 , u 3as Y ( u 1 , u 2 , u 3 ) =
(cid:80) i,j,k u 1 ,i u 2 ,j u 3 ,k Y ijk .
In particular, the spectral norm of an order three tensor
Y is defined by ∥Y∥ = min ∥ u 1 ∥ 2 = ∥
u 2 ∥ 2 = ∥ u 3 ∥ 2 =1 =
(cid:80) i,j,k u 1 ,i u 2 ,j u 3 ,k Y ijk .
The Frobenious normof Y is the natural generalization of the Frobenius norm of a matrix, ∥Y∥ F = (cid:113)(cid:80) i,j,k Y 2 ijk .
Lastly,[ K ] := { 1 , 2 , ..., K } stands for the whole index set and we use ◦ , ⊙ to represent outer product and
Khatri–Rao product.
For two positive sequences { a n } , { b n } , a n ≲ b n means a n ≤ Cb n for someconstant C > 0 independent of n .
through detecting cluster structures
Clustering model via tensor decomposition with regularization
Given observed tensor Y ∈ R d 1 × d 2 × d 3 that is a noisy version of a tensor of interest Y ∗ , i.e. Y =
Y ∗ + E , our goal is to uncover the clustering structures for the three modes of Y ∗ .
We assumethat Y ∗ is a tensor with a rank K CP decomposition structure Y ∗ = (cid:80) i ∈
[ K ] w i A : i ◦ B : i ◦ C : i ) ,with A ∈ R d 1 × K , B ∈ R d 2 × K , C ∈ R d 3 × K factor matrices with columnwise unit one norm, i.e.∥ A : i ∥ 2 = ∥
B : i ∥ 2 = ∥
C : i ∥ 2 = 1 , ∀ i ∈
[ K ] , and w =
[ w 1 , ..., w K ] ⊤ ∈ R K capturing the weights ofthe factor matrices.
The construction of Y ∗ leads to dependence of clustering structures on the factormatrices along each mode.
Specifically, suppose there are s 1 clusters for the first mode, and each rowin A can be expressed as A i : =
(cid:80) s 1 j =1 µ ⊤ 1 ,
j 1 i ∈
A ∗ j , where µ 1 ,j = [ µ (1 ,j ) , 1 , ..., µ (1 ,j ) ,K ] ⊤ is the meanvalue for rows in j th cluster and A ∗ j contains all the rows indices that belongs to j th cluster over firstmode.
Consequently, the true cluster means ¯ A 1 , ..., ¯ A s 1 for slices along first mode {Y 1:: , ..., Y d 1 :: } ,as indicated in Figure 1, can be written as
[Equation]
This reveals the core idea of clustering via CP decomposition: all the true cluster means share thesame K rank-1 matrix basis B : i ◦ C :
i , ∀ i ∈
[ K ] .
Thus, for every i , we would expect there existss ∈
[ s 1 ] such that i ∈
A ∗ s and as a result, the clustering label for i th element along the first mode willbe s .
As shown in Sun and Li [21], the tensor Gaussian mixture model can be viewed as a specialcase of this clustering structure.
However, finding the partitions A ∗ i , ∀
i ∈
[ s 1 ] is a combinatoricalhard problem and could be computational intractable.
Inspired by regularization term proposed in Chiet al.
[8], we impose a full pairwise difference operator over rows of A to yield a weighted penaltyon local differences,
[Equation]
where S :
= { ( i 1 , i 2 ) | i 1 < i 2 , i 1 ∈ [ d 1 ] , i 2 ∈
[ d 1 ] } is a set containing pairs of different row indicesand parameters γ 1 = { γ 1 i 1 ,i 2 | ( i 1 , i 2 ) ∈ S} are non-negative weights that are critical for controllingthe penalty imposed on pairwise row differences for A ; we will explain in detail how to choosethem later.
Analogously, µ 2 ,j , B ∗ j , 2 ∆ , γ 2 and µ 3 ,j , C ∗ j , 3 ∆ , γ 3 can be defined to characterize theclustering structure and regularization along the second and third modes.
In summary, we propose the following penalized constrained optimization as an approach to revealthe latent clustering structure,
[Equation]
In particular, as λ increases, A i : will shrink towards each other, indicating pairwise differences ofrows in A will become increasingly sparse.
Sparsity in pairwise row differences naturally leadsto a partitioning assignment A ∗ j , ∀ j ∈ [ s 1 ] .
For simplicity, we choose λ to be the same over threemodes.
We would like to draw the attention to the difference between the regularization term in(1) and that in Chi et al.
We use an ℓ 1 norm penalty which will yield pairwise difference equalto 0 due to feature selection for ℓ 1 regularization, while Chi et al.
[8] adopts the Frobenious normwhich only shrink the pairwise difference to 0 gradually.
As mentioned, dynamic tensor clustering[21] used a fused LASSO penalty.
However, this simple fusion structure only works if samples fromsame cluster have consecutive indices.
Obviously, our weighted pairwise difference operator i ∆ is ageneralization, and later we will show how our method outperforms dynamic tensor clustering whenthe cluster labels are randomly assigned.
Implementation: fused orthogonal alternating least squares algorithm3.1 Algorithm
Starting from the alternating least square (ALS) algorithm
[14], we propose the fused orthogonalalternating least squares (Fused-Orth-ALS) method described in Algorithm 1 by adding two additional steps that are designed for multi-mode clustering.
First, the orthogonalization step is performed beforeeach iteration of ALS, which allows for the avoidance of local optima and more rapid convergence tothe true factors.
Moreover, a ’Fuse’ operator with generalized LASSO regularization ∆ is employedon each column of ALS estimates; it is defined as
[Equation]
Input: Tensor Y ,
tensor CP rank K
Output: ˆ Y = (cid:80) Ki
=
ˆ w i
ˆ
A t : i ◦ ˆ B t : i ◦ ˆ C t : i
Initialize ˆ A 0 , ˆ B 0 , ˆ C 0 with columns randomly from the unit sphere and iteration index t = 0 ; while termination condition is not satisfied do t ← t + 1 ;
Find QR decomposition of ˆ A t − 1 , set ˆ A t − 1 = Q .
Orthogonalize ˆ B t − 1 and ˆ C t − 1 similarly;
X t ← Y (1) ( ˆ C t − 1 ⊙ ˆ B t − 1 ) , ˜ X t :=
[ ˜ X t : i ] Ki =1 with columns ˜
X t : i ← Fuse ( X t : i , 1 ∆ , λ ) ;
Y t ← Y (2) ( ˆ C t − 1 ⊙ ˆ
A t − 1 ) , ˜ Y t :=
[ ˜ Y t : i ] K i =1 with columns ˜
Y t : i ← Fuse ( Y t : i , 2 ∆ , λ ) ;
Z t ← Y (3) ( ˆ B t − 1 ⊙ ˆ A t − 1 ) , ˜ Z t :=
[˜ Z t : i ] Ki =1 , with columns ˜
Z t : i ← Fuse ( Z t : i , 3 ∆ , λ ) ;
Normalize ˜ X t , ˜
Y t , ˜ Z t with columnwise norm unit 1 and store the results as ˆ A t , ˆ B t , ˆ C t ; end
Estimate weights ˆ w i
=
Y ( ˆ A t : i , ˆ B t : i , ˆ C t : i ) , ∀ i ∈
[ K ]
We use the ’Fuse’ operator since the regularization term in (1) can be viewed as the sum of ’Fuse’regularizations on each column of factor matrices, e.g., ∥ 1 ∆A ∥ 1 = (cid:80)
Ki =1 ∥ 1 ∆A : i ∥ 1 , and itcan be solved efficiently via existing methods [2, 29].
After obtaining estimates ˆ A , ˆ B , ˆ C , clustering algorithm such as k -means or hierarchical clustering are used to obtain cluster assignmentˆ
A i , ˆ B j , ˆ C k , ∀ i ∈ [ s 1 ] , j ∈ [ s 2 ] , k ∈ [ s 3 ] .
From a computational complexity perspective, orthogonalization, ALS updates, and Fuse operations in
Algorithm 1 take O ( K 2 ( d 1 + d 2 + d 3 )) , O ( Kd 1 d
2 d 3 ) and O ( K (
d 31 + d 32 + d 33 )) number of operationsrespectively.
Thus, in a general case, the total computational complexity of Fused-Orth-ALS algorithm on a ( d 1 , d 2 , ..., d D ) -dimensional tensor is O
( K (max ( K (cid:80)
Dj =1 d j , D (cid:81)
Dj =
1 d j , (cid:80) Dj =1 d 3 j ))) .
When d j are of the same order d and tensor rank K
= o
( Dd D − 1 ) , the total complexity can besimplified to O ( KDd D ) which is the same order as classical ALS algorithm, meaning that extraorthogonalization and ’Fuse’ steps do not increase the computational complexity.
The proposedclustering model is clearly distinct from dynamic tensor clustering (DTC) [21], in terms of implementation of generalized LASSO regularization, ALS updates and added orthogonalization.
Our methodhas the same computational complexity as DTC but orthogonality leads to significant speedups in theiterations required for recovery convergence, which will be discussed in detail in section 5 later.
3.2 Regularization weights and tuning parameters
In practice, the choice of appropriate regularization weights γ 1 , γ 2 , γ 3 is critical since it affectsclustering accuracy and computational efficiency.
For instance, γ 1 i 1 ,i 2 characterize the shrinkageof pairwise row difference between A i : and A j : .
From a graph perspective, each row of A canbe treated as a node in an undirected graph and nodes i 1 , i 2 are connected by an edge with weightγ 1 i 1 ,i 2 .
All the edges compose the edge set S .
Clearly, large value of γ 1 i 1 ,i 2 implies high similarity 144 between A i 1 : and A i 2 : , resulting in clustering nodes i 1 , i 2 into the same group.
In particular, ouradopted 1 ∆ reduces to fused LASSO penalty utilized in DTC [21] in the case of a chain graphwhere S
= { i
=
[ i, i + 1] | i = 1 , 2 , ..., d 1 − 1 } .
A well-developed strategy [20, 5, 6] is espoused forchoosing regularization weights: we perform a rank- K CP decomposition approximation to Y , utilizethe estimated factor matrix ˆ A to quantify the ’distance’ among different nodes, and then calculate theregularization weights through
[Equation]
Here, ι ki 1 ,i 2 is an indicator function specifying whether ˆ A i 2 : is among k -nearest neighbors of ˆ A i 1 :that controls density of graph.
The smallest k that ensures the graph is connected is the default choice.
The remaining component in (2) is a Gaussian kernel with τ as a measure of scale chosen as median
Euclidean distance between the i 1 th and i 2 th rows that are k -nearest neighbors of each other.
Lastly,we normalize (cid:80)
[ i 1 ,i 2 ] ∈S γ 1 i 1 ,i
= d 1 d 1 + d 2 + d 3 to ensure the penalty over three modes are on the samescale.
Similar formulas are used for γ 2 and γ 3 .
We choose λ through the use of the extended Bayesian Information Criterion [21, 26, 8] by minimizing
[Equation]
where df λ is the number of non-zero elements in estimated factor matrices, characterizing degrees offreedom.
Moreover, rank K and clusters s 1 , s 2 , s 3 are usually unknown and need to be estimatedfrom data.
We use the ’elbow point’ method to choose rank K by plotting the recovery error, anddetermine s 1 , s 2 , s 3 by computing gap statistics [23].
Cluster recovery and convergence
This section provides a convergence analysis for the Fused-Orth-ALS algorithm.
For simplicity, weset d 1 = d 2 = d 3 =
d
and choose equal regularization weights over all modes.
Without loss ofgenerality, we assume w max = w 1 ≥ w 2 ... ≥ w K = w min > 0 .
The technical assumptions needed for our analysis are given as follows:
[Equation]
someα = polylog ( d ) and Kρ 2 =
o (1) .
The spectral norm of Y satisfies ∥Y∥ ≤ w max α .
A2:
Define the initialization error as ϵ 0 = max i {∥
ˆ A 0: i − A : i ∥ 2 , ∥ ˆ B 0: i − B : i ∥ 2 } which satisfies ϵ 0 ≤ min (cid:110) 1 6 γ − ρ 2 ( K − 1) ,
[Equation]
with γ = w max /w min .
A3:
Denote the spectral norm of E by ψ such that ψ ≤ min (cid:110) w min / 6 , w max K/d (cid:111) .
A4:
Define ζ = max i ∥
∆C : i ∥ 1 that satisfies
[Equation]
Assumption A1 relaxes the requirements on the orthogonality of columns in factor matrices andimposes a weaker condition on rank K than Orth-ALS algorithm [19].
A2 specifies initializationcriteria that are needed only for computational issues.
The bounded perturbation is specified in A3.
In particular, it can be proved with high probability if elements in E are i.i.d from a sub-Gaussiandistribution as stated in Tomioka and Suzuki [24].
A4, a bounded fusion assumption, restrictsclustering complexity on factor matrices.
To illustrate its meaning, consider a simple scenario whenall clusters along the third mode have the same size d/s 3 .
For C i : belonging to the same cluster
C j , they should take the same value µ 3 ,j under our model assumption.
Consequently, A4 can berephrased as ζ = O ( d 1 . 5 (1 − 1 /s 3 )) ; in the special case when C has only one cluster, e.g. s
= 1 ,
A4 reduces to ∥ 3 ∆C ∥ 1 = 0 .
Thus, bounding ζ is equivalent to bounding cluster size s 3 when samplesize over third mode is kept fixed.
We can now derive a recovery error bound for the Fused-Orth-ALS algorithm.
Theorem 1 Assuming assumptions A1-A4 hold, factor matrix estimate
ˆ C : i , ∀ i ∈
[ K ] of Algorithm 1satisfies the following error bound with high probability
[Equation]
when choosing λ ≥ 2 M ( w max ( ϵ 20 α + 2 ϵ 0 ρ ( K − 1) +
ρ 2 ( K − 1)) + ψ ) /
[ w i (1 − ϵ 20 ) − w max ( ϵ 20 α +2 ϵ
ρ ( K − 1) + ρ 2 ( K − 1)) − ψ ] , where M = max j ∥ 3 ∆ † j ∥ 2 .
Similar error bounds hold for theother two factor matrices estimates ˆ A , ˆ B .
The error bound derived in Theorem 1 reveals how the signal w i , incoherence parameter ρ , rank Kand perturbation level ψ interact with each other on affecting the convergence behavior of FusedOrth-ALS algorithm.
Lower perturbation level ψ , lower incoherence parameter ρ , rank K and largesignal ratio of w max /w min all result in lower error bounds.
Clearly, the error bound for each columnin factor matrices has two parts: one is related to perturbation level ψ , the other is dependent on theunderlying CP decomposition structure, weights ratio γ , rank K and dimension d along each mode.
Thus, under high dimensional settings when d → ∞ , i.e., ρ 2 ≤ α 2 /d → 0 , the second part willdominate the error bound while if tensor data is almost noiseless, e.g. ψ ≈ 0 , the first part will bethe main source of error.
The following corollary analyzes the relationship between these two partsunder a special case when the error tensor follows a sub-Gaussian distribution.
Corollary 1 Assume assumptions A1-A4 hold.
If we further assume each element in error tensor,
E ijk , is independent, with zero-mean and satisfies E [ e t E ijk ] ≤ e σ 2 t 2 2 , γ ≤ C where C is positiveconstant and the minimal weight satisfies
[Equation]
then ˆ C : i , ∀ i ∈
[ K ] from Algorithm 1 satisfies the following error bound,
[Equation]
/d with probability at least 1 − δ .
Same error bounds hold for the other two factor estimates ˆ A , ˆ B .
Thus, by imposing the sub-Gaussian distribution assumption on error tensor elements, recoveryconsistency for factor matrices estimated from Fused-Orth-ALS algorithm can be established.
Next, we demonstrate cluster consistency for the clustering algorithm performed on factor matricesrecovered by Fused-Orth-ALS algorithm.
Since clustering algorithm is performed on rows of ˆ A , ˆ B , ˆ C ,clustering error is quantified through the true mean value and its estimate ∥ µ 1 ,j 1 − ˆ µ 1 ,j 1 ∥ 2 , ∥ µ 2 ,j 2 −ˆ µ 2 ,j 2 ∥ 2 , ∥ µ 3 ,j 3 − ˆ µ 3 ,j 3 ∥ 2 .
Theorem 2 Assume assumptions in Corollary 1 hold, we have
[Equation]
hold with probability at least 1 − δ .
Furthermore, if min j 1 ∈ A ∗ m ,j ′ 1 ∈ A ∗ m ′ ,m ̸
= m ′ ∥ µ ∗
[Equation]
1 − µ ∗ 1 ,j
[Equation]
1 ∥ 2 ≳
K 1 .
5 /d , min j 2 ∈ B
[Equation]
n ,j [Equation] ∥ µ
[Equation] [Equation]
j 3 −µ ∗ 3 ,j ′ 3 ∥ 2 ≳ K 1 .
5 /d , we have ˆ A m =
A ∗ m , ˆ B n = B ∗ n , ˆ C l = C ∗ l hold with probability at least1 − δ .
This theorem shows that clustering consistency holds as long as K 1 .
5 /d → 0 , which allows rank Kincrease with the dimension d .
At first look, the conclusion in Theorem 2 seems to indicate that thecluster mean error bound does not depend on number of clusters over each mode, s 1 , s 2 , s 3 .
However,as we stated before, assumption A4 employs an ’invisible’ bound on the number of clusters which isclosely related with the convergence rate derived in Theorem 2.
We provide a detailed explanationin supplementary material stating cluster mean error bound K 1 .
5 /d increases with the number ofclusters s i .
Note that Theorem 2 assumes that true rank K is known; we leave the effect of anestimated rank ˆ K on clustering consistency for further study.
Numerical experiments5.1 Synthetic datasets
In this section, we investigate the performance of the Fused-Orth-ALS algorithm on small samplesand compare the recovery and clustering error with alternative tensor-based clustering methods.
Recovery error is evaluated through ∥ ˆ Y − Y ∗ ∥ F / ∥Y ∗ ∥ F .
Clustering error measures the frequency ofmismatches between the estimated clustering assignment ˆ M and the true M over samples x 1 , ..., x n ,is defined as
[Equation]
and can be easily computed after obtaining the cluster assignment ˆ A m , ˆ B n , ˆ C l , ∀ m ∈
[ s 1 ] , n ∈[ s 2 ] , l ∈ [ s 3 ] .
Note that the clustering error is similar to two other commonly used metrics: adjusted
Rand index [11] and variation of information [17].
For the simulations below, elements in the errortensor E are generated independently from a Gaussian distribution with mean 0 and variance σ 2 .
Wecompare Fused-Orth-ALS Algorithm with three methods: dynamic tensor clustering (DTC) [21],
CP-Kmeans algorithm which performs a rank- K CP decomposition on the tensor observations
firstand then independently applies k -means algorithm clustering to the rows of derived factor matrices,and multiway clustering for tensor block models (TBM)[26].
5.1.1 Single-mode clustering
We first generate an order three tensor with CP-rank 2 for which would like to do single-modeclustering along the third mode.
The dimension of the first and second modes are set the same( d 1 = d
2 = d ) and their unnormalized columns are
[Equation]
(cid:124)
[Equation]
) ⊤
The third factor matrix with unnormalized & unshuffled columns is generated using
[Equation]
Then, the rows of C are shuffled randomly.
There are four clusters over the third mode with clustermeans ( µ, − µ ) , ( µ, µ ) , ( − µ, µ ) , ( − µ, − µ ) respectively.
After normalizing the columns of A , B , C ,we calculate the weights w i .
Even though this example is inspired from Sun and Li [21], shufflingthe rows of C leads to additional clustering challenges since a more sophisticated regularization isrequired to capture the clustering features.
µ reflects cluster difficulty as it determines the distancebetween different clusters.
A smaller µ leads to a more challenging clustering problem.
For simplicity,the factor matrices used in this example, A , B , C , are orthogonal, satisfying assumption A1.
We set d = 20 , d 3 = 48 , µ = 1 , vary σ ∈ { 0 , 0 . 25 , 0 . 5 , 0 .
and compare the performance ofthe four algorithms by reporting recovery and clustering errors.
As summarized in Figure 2 (a) and(b), Fused-Orth-ALS outperforms the other three methods under different values of perturbation,achieving the best recovery error and clustering error.
Not surprisingly, TBM performs worst due tomodel misspecification: the formulation of simulated factor matrices leads to an underlying clusterstructure for mode three instead of a multiway block structure for all three modes.
Figure 2 (c) and(d) show that average running times are comparable for Fused-Orth-ALS and DTC, addressing theconcern that the orthogonalization step added in Fused-Orth-ALS may increase dramatically runningtime and number of iterations required for convergence.
Furthermore, to illustrate the advantage ofgeneralized LASSO regularization imposed on Fused-Orth-ALS over fused LASSO penalty adoptedin DTC, we vary the value for µ ∈ { 0 . 1 , 0 . 7 } to mimic different levels of clustering complexityand set d = 8 , σ = 0 .
001 , d 3 = 20 .
It is obvious from Table 1 that our method outperforms DTCespecially as the ’signal’ µ becomes weak.
[Table]
[Figure]
To validate the performance on multi-mode clustering, we extend the simulation settings aboveby imposing clustering structures on all three factor matrices A , B , C , similarly to (3).
Thecorresponding choice of cluster means are: ( µ, 0 .
5 µ ) , ( − 0 .
5 µ, µ ) , (0 , − µ ) , ( − µ, 0) for mode 1,(0 , µ ) , ( − µ, 0) , (0 . 1 µ, − µ ) , ( − µ, − 0 .
1 µ ) for mode 2 and ( µ, µ ) , ( − µ, µ ) , ( µ, − µ ) , ( − µ, − µ ) formode 3.
Accordingly, each mode has four clusters, i.e., s
= s
= s 3 = 4 of same size:⌊ d 1 / 4 ⌋ , ⌊ d 2 / 4 ⌋ ,
⌊ d 3 / 4 ⌋ respectively.
For example, for the first mode, there are ⌊ d 1 / 4 ⌋ rows ofthe factor matrix taking value ( µ, 0 .
5 µ ) , ⌊ d 1 / 4 ⌋ rows taking value ( − 0 .
5 µ, µ ) , ⌊ d 1 / 4 ⌋ rows takingvalue (0 , − µ ) and the rest rows taking value ( µ, 0) .
We fix d 1 = d 2 = 20 , d 3 = 40 , µ = 1 andvary σ ∈ { 1 , 2 , 3 , 4 , 5 } .
Since the detailed comparison between Fused-Orth-ALS and DTC has beenillustrated above, we omit DTC (originally designed for dynamic single mode clustering of tensors).
Comparison results can be found in Figure 3.
It is clear that Fused-Orth-ALS algorithm outperformsthe other two methods, especially under the noisy case when σ is large.
[Figure]
Experiments are conducted on two real datasets: the brain node structural connectivity from Human
Connectome Project (HCP) 1 [25] and political relationships between nations 2 [13].
Details forchoosing rank K and number of clusters are provided in supplementary material.
The HCP dataset is formatted as Y ∈ R 68 × 68 × 136 , and consists of brain connectivity among 68 brainnodes for 136 individuals.
Each entry takes on ordinal value { 0 , 1 , 2 } which indicates the strengthlevel of connectivity { low, moderate, high } between different brain nodes.
Clustering results in Table2 capture spatial connectivity between hemispheres of brain (the first character in the node nameindicates the left or right hemisphere and the number in the parenthesis indicates the node count withsame name).
Cluster I and II mainly capture connectivity in either left or right hemisphere while cluster III represents the cross-section connection between left and right hemisphere.
r.supramarginaland l.supramarginal are picked as smaller clusters - those regions are known to play a critical role invisual word recognition and reading.
[Table]
The Nations dataset is formatted as Y ∈ R 14 × 14 × 56 consisting of 56 political relationships of 14countries between 1950-1965.
Each entry represents the presence or absence of a political action,such as ’treaties’, ’send tourists to ’ between different nations.
Since 78.9% entries are zero in this dataset, we include an ℓ 0 penalized tensor block model (denoted as ’TBM-Sparse’) for comparison.
The clustering result in Table 3 for Fused-Orth-ALS algorithm assigned the nations into three clusters,one representing western-bloc countries (Cluster I), one communist bloc (Cluster III), one neutralbloc (Cluster II).
This result is consistent with the structure of political environment after world war
II.
In addition, the goodness-of-fit (proportion of variance explained) for five methods is provided in
Table 4.
For the nations dataset, Fused-Orth-ALS algorithm shows the highest variance explained.
[Table] [Table]
Conclusions
This paper studies the properties and convergence rates for a novel multiway tensor clustering algo- 298 rithm, Fused-Orth-ALS.
The underlying CP decomposition structure imposes low-rank constraintson the tensor observation, and graph similarity based fusion regularization encourages smoothness 300 on factor matrices, thus leading to automatic clustering.
The Fused-Orth-ALS algorithm can handlesparse and dense data tensors, and achieves clustering consistency even under model misspecification.
An open problem is developing theory for rank selection consistency.
Another interesting topic is onclustering consistency if there are missing entries in the data.
We leave these directions for further 304 study.
There are no foreseeable negative social impacts of this work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See supplementarymaterials.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section
and supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 5 and supplementary material
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See Section 5.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[Yes] See Section 5.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]See supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Section 5.
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A] [Table]
(b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A] [Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]