Multi-view Subspace Clustering on Topological Manifold
Anonymous Author(s)
AffiliationAddress
Abstract
Multi-view subspace clustering aims to exploit a common affinity representationby means of self-expression.
Plenty of works have been presented to boost theclustering performance, yet seldom considering the topological structure in data,which is crucial for clustering data on manifold.
Orthogonal to existing works, inthis paper, we argue that it is beneficial to explore the implied data manifold bylearning the topological relationship between data points.
Our model seamlessly integrates multiple affinity graphs into a consensus one with the topological relevanceconsidered.
Meanwhile, we manipulate the consensus graph by a connectivityconstraint such that the connected components precisely indicate different clusters.
Hence our model is able to directly obtain the final clustering result without relianceon any label discretization strategy as previous methods do.
Experimental resultson several benchmark datasets illustrate the effectiveness of the proposed model,compared to the state-of-the-art competitors over the clustering performance
Introduction
With the evolution of multimedia, now large amounts of data are represented by multiple types (i.e.,views).
A document, for example, can be presented by images, audio, text.
Likewise, a text can beexpressed in different languages [1].
In order to integrate the complementary information amongmultiple views, numerous sophisticated multi-view clustering algorithms have been proposed [2, 3],which typically produce better results than single-view clustering [4, 5].
Over the past years, self-representation subspace clustering methods, which assume that each datapoint can be reconstructed by a linear combination of other data points, have gained broad attention.
The self-representation property is able to capture the heterogeneous relationships between data andtherefore produces excellent clustering results [6].
Meanwhile, self-representation property basedmulti-view subspace clustering algorithms have been developed up to now [7].
For instance, diversityinduced multi-view subspace clustering (DiMSC) [8] adopts the Hilbert-Schmidt independencecriterion (HSIC) as a diversity term to measure the dependence of different views.
By minimizing thediversity term, the subspace representations are compelled to be distinct from each other.
Insteadof integrating multi-view graphs into a consensus one, [9] involves the spectral clustering into theirobject function to derive a unified indicator matrix.
Therefore, the consistent clustering results ofdata points in different views are ensured.
[10] separates the subspace representations of differentviews by a shared consistent representation and a set of specific representations, which is expected tobetter accommodate real-world data.
To eliminate the redundant information of data observations orkernel matrices, [11] utilizes eigendecomposition to get the robust data representations, which helpsto obtain the consensus representations of low redundancy and better clustering results.
Lately, anumber of anchor-based multi-view subspace clustering methods [12–14] draw significant attentionbecause they can achieve promising performance with a large reduction in storage and computational time
Despite the significant progress made by the aforementioned methods, there are still drawbacksthat can be modified.
On one hand, these methods do not consider the manifold topological structure.
Considering that real-world datasets are usually sampled from a nonlinear low-dimensionalmanifold [15–17], it is essential to explicitly explore the topological relationship for clustering dataon manifold.
On the other hand, existing multi-view clustering methods usually adopt predefinedsimilarity graphs.
That is, the graph learning and subsequent multi-view clustering in these methodsare separated.
Thus the constructed graph may not be suitable, let alone optimal, for the subsequentclustering.
It is preferred to automatically learn the similarity information between data points andinvolve graph learning as a part of the optimization procedure.
Regrading the deficiencies mentioned above, in this paper we argue that it is beneficial to explore theimplied data manifold by learning the topological relationship between data points.
To do so, wepropose to seamlessly integrate multiple affinity graphs into a consensus one with the topologicalrelevance considered.
Besides, we manipulate the consensus graph by a connectivity constraint suchthat the connected components precisely indicate different clusters.
Hence our model is able to directlyobtain the final clustering result without reliance on any label discretization step.
By leveraging thesubtasks of affinity graph constructing, topological relevance learning, and discrete label partitioninginto a unified framework, each subtask can be enhanced in a mutual reinforcement manner.
Analternating iterative algorithm is carefully designed to solve the optimization problem of the proposedmodel.
Experimental results on several benchmark datasets demonstrate the effectiveness of ourmethod.
Notations.
We use boldface uppercase letter, e.g., M , to denote the matrix.
M ij represents the ij -thelement of M .
∥·∥ F represents the Frobenius norm of a matrix and 1 is a column vector with all itselements equal to 1.
I denotes the identity matrix with proper size.
Preliminary
It is well-known that real-world datasets are often sampled from a nonlinear low-dimensional manifoldwhich is embedded in the high dimensional ambient space [15, 18–20].
Hence it is necessary toexploit the manifold structure implied within the original data.
Recently, [17] pointed out that it is preferred to boost the learning performance by making use ofthe manifold topological structure than the Euclidean structure.
It is based on a simple yet intuitiveassumption that the topological connectivities between individuals could be propagated from near tofar.
In other words, the spatial similarity between two individuals may be small, but their topologicalrelevance to each other would be high if they are linked by consecutive neighbors.
A visualizedexample is given in Figure 1.
For better illustration, we visualize the topological relevance in botha 2D version (left subfigure) as well as a 3D version (right subfigure).
As we see, although thegreen and red points are with low similarity in terms of spatial location and velocity, they are closelyconnected to each other considering the high topological relevance between them.
[Figure]
Instead of relying on the Euclidean structure, it is naturally expected to learn a more suitable manifoldtopological structure such that the intrinsic similarities can be explicitly uncovered.
Given a predefined similarity graph Z ∈ R n × n , where n is the number of data points.
Considering that data points witha high similarity would share similar topological relevance, [17] investigated the topological structureof data by solving
[Equation]
where β is a trade-off parameter.
i , j , and k are data point indexes.
S represents the target topologicalrelationship matrix, and S ij denotes the data point j ’s topological relevance to i .
Note that thefirst term in Eq.
(1) is essentially a smoothness constraint that fits the above assumption, i.e., itguarantees the data points j and k share a similar topological relationship with data point i if they aresimilar.
And the second term is incorporated to prevent the trivial solution.
According to Eq.
(1), thetopological consistency can be propagated through neighbors with high similarities, and the distantdata points will keep a close relationship if they are linked by consecutive neighbors.
Notwithstanding, Eq.
(1) is fed with a predefined graph, which would lead to performance degradationas the predefined graph Z might not be optimal for subsequent learning procedures.
Hence it ispreferred to automatically learn similarity graphs from data.
Self-expression has been widelyemployed to recover the similarity graph in the form of subspace representation [6].
Given adataset X =
[ x 1 , x 2 , · · · , x n ] ∈ R m × n with n data points and m features, the self-expression basedclustering problem can be defined as
[Equation]
where α is a trade-off parameter and diag ( Z ) represents the vector consists of diagonal elementsof Z .
It is obvious that Eq.
(2) learns the neighboring data points of a data point and the corresponding weights by the sparse reconstruction from other data points.
Moreover, the learned sparserepresentation enjoys several nice properties, e.g., the datum-adaptive ability and robustness tonoise [21, 22].
Based on Eq.
(2), we can easily extend it to a multi-view formulation when dealing with the multi-viewdata { X (1) , X (2) , · · · , X ( m ) } , which can be formed as
[Equation]
where m is the number of views, and Z ( v ) (1 ≤ v ≤ m ) denotes the affinity representation thatautomatically learned from the v -th view.
The Proposed Methodology
To explicitly incorporate the manifold topological structure into multi-view clustering, we propose tolearn a topological relation graph based on Eq.
With the subspace representation automaticallyobtained in Eq.
(3), we arrive atmin Z ( v ) , S m (cid:88)
[Equation]
[Equation]
where the consensus graph S reveals the topological relationship across multiple views and is furtherconstrained to be non-negative and the sum of each row is 1 .
w v is the weight for the v -th views,which can be considered as a constant within each iteration and will be introduced later.
Nevertheless, the learned S in Eq.
(4) does not contain explicit cluster structures.
It is expected thatthe consensus graph contains exactly c connected components where c is the number of clusters.
It sounds unrealistic that such a clear structured S could be achieved out of thin air.
Fortunately, we cansolve this problem with a useful connectivity constraint.
Moreover, in Eq.
(4), we see that if the j -thdata point is connected with many similar neighbors, it will largely affect the objective value.
Thuswe tend to normalize the objective function so that each point can be treated equally.
Consideringthe above concerns, we propose a normalized version of Eq.
(4) with an additional connectivityconstraint, which can be finally formulated as
[Equation]
where D ( v ) is the degree matrix of Z ( v ) , L S is the Laplacian matrix of S , and rank ( L S )
= n − c isa rank constraint that manipulates the target graph S containing exactly c connected components.
Inthis way, the subsequent label discretization step can be avoided since each connected componentprecisely indicates an individual cluster.
As for the weight w v of each view, we set it in a self-tunedway inspired by [23, 24]:
[Equation]
It is worth mentioning the salient contributions of our model formulated in Eq.
(5) as follows:
• Orthogonal to other multi-view subspace clustering approaches, our model explicitly ex-
plores the implied data manifold by learning the topological relationship across multiple
views.
To the best of our knowledge, this is the first work that delicately incorporates the
topological structure in multi-view subspace clustering.•
Our model flawlessly integrates the subtasks including affinity graph constructing, manifold
topological structure learning, and discrete label partitioning into a unified framework.
Hence our model is an end-to-end single-stage learning paradigm.
• An alternating iterative algorithm is introduced to solve the optimization problem.
Exper-
iments on several benchmark datasets illustrate the effectiveness of the proposed model,
compared to the state-of-the-art competitors over the clustering performance.
3.1 Algorithm Derivation
In order to solve the problem in Eq.
(5), we derive an optimization algorithm to optimize the objectivefunction.
Since the corresponding optimization problem is not jointly convex in all variables, wechoose to solve it by updating one variable while fixing other variables.
3.1.1 Update Z ( v ) for Each View
For Z ( v ) , the corresponding problem ismin Z
[Equation]
v )
[Equation]
v
[Equation]
w v n (cid:88)
[Equation]
Eq.
(7) can be rewritten as
[Equation]
where H ( v ) =
(cid:0) X ( v ) − X
( v ) Z ( v ) (cid:1) (cid:0) X
( v ) − X
( v ) Z ( v ) (cid:1) T .
Taking the derivative of Eq.
(8) w.r.t Z ( v )and setting the derivative to zero, we get following solution
[Equation]
where G = D − 12 S , and K ( v )
=
X ( v ) T X ( v ) can be treated as a liner kernel.
Note that the nonlinearkernel can also be applied in our model.
3.1.2 Update S
Dropping the unrelated terms of Eq.
(5) w.r.t.
S , thus we need to solve
[Equation]
We see that Eq.
(10) is difficult to solve due to the rank constraint.
Let σ i ( L S ) be the i -th smallesteigenvalue of L S .
The constraint rank ( L S )
= n − c would be satisfied if (cid:80) ci =1 σ
i
L S )
= 0 since
L S is a positive semidefinite matrix.
According to Ky Fan’s Theorem [25]:
[Equation] [Equation]
where F ∈ R n × c denotes the cluster indicator matrix.
We incorporate the constraint term 148 (cid:80) ki
=1 σ i ( L S ) , i.e., min F Tr (cid:0) F T L S F (cid:1) , into the cost function, thus Eq.
(10) can be rewritten as
[Equation]
where λ is a self-tuned parameter.
Based on Eq.
(12), first we search the optimal solution of F , which can be obtained by solving
[Equation]
[Equation]
which is essentially a spectral problem and F can be achieved by calculating the c eigenvectors of L Scorresponding to the c smallest eigenvalues.
Then we search the optimal solution of S .
It is clear that Eq.
(12) w.r.t.
S can be formulated in avector form as
[Equation]
Note that Eq.
(14) is independent for different i , thus we have the following compact formulation
[Equation]
where e i denotes the i -th column vector of identity matrix I , and u i represents a vector with its j -thelement u ij = λ
∥ f i − f j ∥ 22 .
Denote A = (cid:80) mv =1 w v (cid:16) I − D ( v )
− 12 Z ( v ) D
( v ) − 12 (cid:17) + β I and b = 2 β
e
i − u
i , Eq.
(15) is equivalentto optimizing
[Equation]
We see Eq.
(16) is essentially a quadratic convex optimization problem, hence it can be solvedwith the classical augmented Lagrangian multiplier (ALM) method [26].
Specifically, we can solve
Eq.
(16) by tackling its counterpart
[Equation]
Based on Eq.
(17), the corresponding augmented Lagrangian function can be defined as
[Equation]
where the second term in Eq.
(18) is employed to guarantee that p
= s i , η and q are the penaltycoefficient and parameter, respectively.
It is clear that we can solve p and s i in an iterative updating way:
1) Update p with fixed s i .
The corresponding cost function w.r.t.
p is
[Equation]
Setting ∂ L p ∂ p = 0 , we obtain
[Equation]
2) Update s i with fixed p .
The corresponding cost function w.r.t.
s i can be reformulated as
[Equation]
which leads to a closed-form solution and can be directly obtained by [21]
According to the
ALM principles [26], η can be exaggerated increasingly during each iteration, and q is updated byq ← q + η
( s i − p ) .
Finally, we can obtain a optimal solution for S based on this effective ALMstrategy.
The detailed algorithm to solve Eq.
(18) is outlined in Algorithm 2.
Up to now, with the variable Z ( v ) updated by Eq.
(9), S updated by Eq.
(18), F updated by Eq.
(13),and w v updated by Eq.
(6), the overall algorithm to solve the objective in Eq.
(5) can be summarizedin Algorithm 2.
Owing to space limitation, we give the convergence analysis and the time complexityof Algorithm 2 in the Appendix B and Appendix C, respectively.
[Table]
Experiments
We evaluate the proposed method by comparing it with following state-of-the-art competitors:
Diversity-induced Multi-view Subspace Clustering ( DiMSC ) [8], Auto-weighted Multiple Graph
Learning ( AMGL ) [27], Multi-view Clustering with Graph Learning ( MVGL ) [28], Weighted Multiview Spectral Clustering( WMSC ) [29], Consistent and Specific Multi-view Subspace Clustering( CSMSC )
[10], Graph-based Multi-view Clustering ( GMC ) [24], Large-scale Multi-View Subspace
Clustering in linear time ( LMVSC ) [12], Scalable Multi-View Subspace Clustering with unified anchors ( SMVSC ) [13], and Multi-View Subspace Clustering via Co-training robust data representation( CoMSC ) [11].
The classic graph-based algorithm, spectral clustering (SC) [18], is also included asa baseline.
We apply SC by making use of the most informative view, i.e., one that achieves the bestperformance (denoted by SC best ).
The experiments are conducted on several benchmark datasets, namely, 3Sources, MSRC, 100Leaves,
COIL-20, Caltech-7, Caltech-20, and MNIST.
The detailed information of all datasets is given in
Appendix A. The specific characteristics of these datasets are also summarized in Table 1.
Theparameters for comparison algorithms are set according to the recommendations in their correspondingpaper.
The parameter settings of our model will be introduced later.
We repeat all algorithms 10 timesto eliminate the effect of random factors, and report the average scores.
Four widely-used metrics,i.e., clustering accuracy (ACC), Normalized Mutual Information (NMI), Purity, and F-score, are usedto achieve a comprehensive evaluation.
4.1 Results and Analysis
The clustering performance of all methods on four metrics is shown in Table 2.
As we see, ourproposed method achieves the best performance in the majority of cases, which validates the effectiveness of our method.
Moreover, it is obvious that the improvement is remarkable.
On dataset MSRC,for instance, our method even achieves substantial improvements around 4.7%, 4.2%, 4.3%, and10.5% over the most competitive methods in terms of ACC, NMI, Purity, and F-score, respectively.
Note that DiMSC, CSMSC, LMVSC, SMVSC, and CoMSC are all multi-view subspace clusteringmethods that adopt the Euclidean structure.
From the results, it is obvious our method that employsmanifold topological structure is much better than that in terms of clustering performance, whichverifies our assumption that manifold topological structure is more suitable to explicitly uncoverthe intrinsic similarities.
Taking the dataset 100Leaves as an example, we visualize the target graphlearned by different methods.
For better visualization, we plot the first seven categories of dataset100Leaves in Figure 2.
As can be seen, DiMSC cannot even find the block diagonal structure ofthe target graph.
MVGL, CSMSC, and CoMSC are able to search the block diagonal structure but
[Table]
91.90 Caltech28.
[Table] [Table]
[Table]
100Leaves 86.
[Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
MNIST 47.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Purity 3Sources
[Table] [Table]
[Table] [Table]
MSRC
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
60 100Leaves 72.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
.02 MNIST 56.
87.44 F-score 3Sources 48.
75.25 MSRC 43.
75.29 COIL69.
82.29 Caltech40.
79.77 100Leaves 61.
69.29 Caltech33.
53.13 MNIST 41.
[Figure]
contain lots of noise obviously.
On the contrary, our model almost achieves a pure structured graphwith a much clear clustering structure.
4.2 Parameter Analysis
This section investigates the sensitivity of our method with respect to different parameter settings.
Note that λ involved in Eq.
(12) is a self-tuned parameter, which can be set in a heuristic way.
Thatis, we can initialize λ to a random positive value (e.g., λ = 10 ), if the connected components of Sis greater than c in current iteration, we set λ ← λ 2 , and if it is less than c , we set λ ← 2 λ .
Thisstrategy guarantees S contains a clear clustering structure with exact c connected components.
Hencewe only need to set the parameters α and β properly.
Here we empirically search them in the range[0.1,0.5,1,5,10,50] for simplicity.
The clustering performance of three datasets is shown in Figure 3(owing to space limitation, the results of other datasets are attached in Appendix D), we can findthat the clustering results of our method are relatively stable for different parameter settings, whichdemonstrates the robustness of our model.
As introduced before, the target consensus graph S canbe treated as an indicator matrix, where the points from the same cluster are connected to the samecomponent.
Once we obtain S , the cluster label of each data point can be directly assigned without
[Figure]
any postprocessing.
Hence our method is very stable.
In general, we could obtain a promisingclustering performance by setting α = β = 10 in practical applications.
[Figure]
4.3 Convergence Study
Here we verify the convergence property of the proposed algorithm.
The convergence curves alongwith the corresponding performance of our algorithm on 3Sources, MSRC, and COIL-20 datasets arerecorded in Figure 4 (the convergence curves of other datasets are attached in Appendix D), wherethe red line denotes the NMI of our method and the grey line indicates the objective value.
As wecan see, the proposed optimization algorithm is efficient and converges very fast, usually within 10iterations, which empirically illustrates the efficiency of the proposed algorithm.
Conclusion
In this paper, we propose to explore the implied data manifold by learning the topological relationshipbetween data points.
To do so, we seamlessly integrate multiple affinity graphs into a consensusone with the topological relevance considered.
Besides, we manipulate the consensus graph bya connectivity constraint such that the connected components precisely indicate different clusters.
Hence our model can directly obtain the discrete result without any postprocessing.
An alternatingiterative algorithm is carefully designed to solve the optimization problem of the proposed model.
Experimental results have shown that (1) manifold topological structure is suitable to explicitlyuncover the intrinsic similarities, thus is beneficial for multi-view subspace clustering task; (2) ourmodel is quite robust with respect to different parameter settings, which demonstrates its stability;(3)
the proposed optimization algorithm is very efficient and converges fast.
However, note thatour model cannot deal with the nonlinear data, which can be considered in future works.
Asmentioned in Subsection 3.1.1, the kernel strategy would be an option to explore the nonlinearstructure.
Furthermore, we are also interested in extending the proposed framework to other machinelearning frameworks, such as semi
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section [-
-]5(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section B.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] We provide the
code and data, and instructions in the supplementary materials.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] The important details are provided in Section 4.2 and the code in
the supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
For fair comparison, we repeat all algorithms 10 times.
Considering the appearance of the Table 2, we did not report the standard deviation.
We provide the code and data in the supplementary materials.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Appendix A.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include the proposed algorithm in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All assets used are open-source.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Appendix:
A Datasets
Several benchmark multi-view datasets are adopted in our experiments.
3Sources 1 is collected fromthree news sources, i.e., Reuters, BBC, and The Guardian.
There are 948 news articles covering416 different news stories.
Among them, 169 news were reported in all three sources and eachnews was annotated with one of six topical labels: business, health, politics, entertainment, sport,and technology.
MSRC is comprised of 240 images in eight classes.
We select seven classeswith each class containing 30 images.
For each image, five visual features are extracted for acomprehensive description.
Caltech101 is an object recognition dataset with 101 categories.
Thisdataset is represented by 6 types of features.
Following [23], we select 1474 images within 7 classes( Caltech-7 ) and 2386 images within 20 classes ( Caltech-20 ).
100Leaves 2 is the one-hundred plantleaves data set.
It contains 1600 data samples and each of which is described by fine-scale margin,shape descriptor, and texture histogram.
COIL-20 is a subset of a object database 3 includes 100categories.
Images of each object were taken 5-degree apart as the object is rotated on a turntableand each object has 72 gray images.
Each image is described by three types of features.
MNISTis a handwritten digits database 4 that is represented by three views.
We select 10000 data samplesincludes 10 categories in this paper, and there are 1000 data samples per category.
The specific characteristics of the datasets are given in Table 1.
B Convergence Analysis
In this subsection, we prove the convergence of problem (10) and Alg.
We will show that ouralgorithm can find a local optimal solution.
Before we prove its convergence, first we introduce animportant lemma as follows [30]:
Lemma
For any positive real number q and t , the following inequality holds:
[Equation]
Proof 1 It is obvious that inequality ( √ q − √ t ) 2 ≥ 0 , thus we have
[Equation]
which completes the proof.
Theorem 1 In each iteration, the updated S will monotonically decrease the objective in Eq.
(10),which generally makes the solution converge to the local optimum of Eq.
Proof 2 Suppose the alternatively updated S is (cid:101) S in each iteration.
By solving Eq.
(18), we get
[Equation]
According to Eq.
(6), i.e., w v =(cid:44)
[Equation]
, we get
[Equation]
S
[Equation]
where G ( S ) =
β ∥ S − I ∥ 2 F .
Based on Lemma 1, we have
[Figure]
[Equation]
[Table]
[Equation]
By summing over the above two equations in the two sides, we obtain
[Equation]
which completes the prove.
That is, the objective function value will monotonically decreases in eachiteration of updating S .
In Alg. 2, we can obtain the closed-form solutions with respect to Z ( v ) , w v , and F , as described in themain paper.
ALM optimization theory [31] guarantees that the iterations will make the optimizationprocess converge.
In a word, the updating of all variables with iterative optimization steps willmonotonically decrease toward the lower bound of the objective function in (5).
C Time complexity Analysis
As shown in Alg. 2, there are four steps that mainly determine the complexity of our model.
Recallthat n , m and c denote the number of data points, views, and clusters, respectively.
We summarizethe computational complexity of each step in Table 3.
In practical, we have m ≪ n and c ≪ n , thus the overall complexity is O (cid:0) n 2 (cid:1) , which is in the sameline with the classical graph-based methods and hence is acceptable.
Steps
Calculation
Complexity
[Table]
D Additional ExperimentsD.1 Computational performance
Given the computational complexity of our algorithm theoretical analyzed above, here we empiricallycompare the computational speed of our method with other multi-view subspace clustering approaches.
The execution time of all algorithms on a machine with 2.60GHz Intel Xeon Gold 6240 CPU and256GB RAM is shown in Table 4.
We see that LMVSC and SMVSC are the two timesaving-mostalgorithms, especially on large dataset MNIST.
On the other hand, DiMSC, CoMSC, and MVGL arethe three time-consuming-most algorithms, which is slower than SMVSC by nearly two orders ofmagnitude.
Generally speaking, our algorithm is faster than DiMSC, CoMSC, and MVGL, slowerthan LMVSC and SMVSC but in line with other methods, which validates the efficiency of theproposed algorithm.
[Table]
Caltech-
[Table]
[Figure]
D.2 Parameter Analysis
This section investigates the influence of different parameter settings to clustering performance.
Asmentioned in the main paper, λ involved in Eq.
(12) is a self-tuned parameter, and can be set ina heuristic way.
That is, we can initialize λ to a random positive value (e.g., λ = 10 ), then weautomatically halve (i.e., λ = λ/ 2 ) or double (i.e., λ = λ ∗ 2 ) it when the number of connectedcomponents is greater or smaller than the cluster number c in each iteration.
As a result, this strategyguarantees that the obtained graph contains a clear clustering structure.
As a result, we only need to setthe parameters α and β properly.
We empirically search both α and β in the range [0.1,0.5,1,5,10,50]for simplicity.
As shown in Figure 5, we can find that the clustering results of our method arerelatively stable with respect to different parameter settings, which demonstrates the robustness ofour model.
In general, we could obtain a promising clustering performance by setting α = β = 10 inpractical applications.
D.3 Convergence Study
Besides, as shown in the main paper, our empirical experiments also validate the convergence of theproposed method.
The convergence curves along with the corresponding performance of our algorithmon different datasets are recorded in Figure 6, where the red line denotes the NMI of our methodand the grey line indicates the objective value.
As we can see, the proposed optimization algorithmis efficient and converges very fast, usually within 10 iterations, which empirically illustrates theefficiency of the proposed algorithm.
[Figure]
D.4 Manifold Visualization
In this section, we show the manifold structure visualization of the data obtained by the proposedmethod.
As can be seen in Figure 7, all the benchmark datasets naturally exhibit a manifoldtopological structure.
Our empirical studies also corroborate our assumption and demonstrate that it isbeneficial to explore the topological structure for better clustering performance.
Besides, we presentthe visualization of the clustering results of all the datasets with t-SNE [32] in Figure 8.
As can beseen, the learned consensus graph generally contains a compact and accurate class cluster, whichagain showcases the effectiveness of our method by considering the manifold topological structure.
[Figure] [Figure]