Multi-view Subspace Clustering on Topological Manifold
Abstract
Multi-view subspace clustering aims to exploit a common affinity representation by means of self-expression.
Plenty of works have been presented to boost the clustering performance, yet seldom considering the topological structure in data, which is crucial for clustering data on manifold.
Orthogonal to existing works, in this paper, we argue that it is beneficial to explore the implied data manifold by learning the topological relationship between data points.
Our model seamlessly integrates multiple affinity graphs into a consensus one with the topological relevance considered.
Meanwhile, we manipulate the consensus graph by a connectivity constraint such that the connected components precisely indicate different clusters.
Hence our model is able to directly obtain the final clustering result without reliance on any label discretization strategy as previous methods do.
Experimental results on several benchmark datasets illustrate the effectiveness of the proposed model, compared to the state-of-the-art competitors over the clustering performance.
Introduction
With the evolution of multimedia, now large amounts of data are represented by multiple types (i e., views).
A document, for example, can be presented by images, audio, text.
Likewise, a text can be expressed in different languages [1].
In order to integrate the complementary information among multiple views, numerous sophisticated multi-view clustering algorithms have been proposed [2, 3] which typically produce better results than single-view clustering [4, 5]
Over the past years self-representation subspace clustering methods, which assume that each data point can be reconstructed by a linear combination of other data points, have gained broad attention.
The self-representation property is able to capture the heterogeneous relationships between data and therefore produces excellent clustering results [6].
Meanwhile, self-representation property based multi-view subspace clustering algorithms have been developed up to now [7].
For instance, diversityinduced multi-view subspace clustering (DiMSC) [8] adopts the Hilbert-Schmidt independence criterion (HSIC) as a diversity term to measure the dependence of different views.
By minimizing the diversity term, the subspace representations are compelled to be distinct from each other.
Instead of integrating multi-view graphs into a consensus one, [9] involves the spectral clustering into their object function to derive a unified indicator matrix.
Therefore, the consistent clustering results of data points in different views are ensured.
[10] separates the subspace representations of different views by a shared consistent representation and a set of specific representations, which is expected to better accommodate real-world data.
To eliminate the redundant information of data observations or kernel matrices, [11] utilizes eigendecomposition to get the robust data representations, which helps to obtain the consensus representations of low redundancy and better clustering results.
Lately, a number of anchor-based multi-view subspace clustering methods [12–14] draw significant attention because they can achieve promising performance with a large reduction in storage and computational time.
Despite the significant progress made by the aforementioned methods, there are still drawbacks that can be modified.
On one hand, these methods do not consider the manifold topological structure.
Considering that real-world datasets are usually sampled from a nonlinear low-dimensional manifold [15–17], it is essential to explicitly explore the topological relationship for clustering data on manifold.
On the other hand, existing multi-view clustering methods usually adopt predefined similarity graphs.
That is, the graph learning and subsequent multi-view clustering in these methods are separated.
Thus the constructed graph may not be suitable, let alone optimal, for the subsequent clustering.
It is preferred to automatically learn the similarity information between data points and involve graph learning as a part of the optimization procedure.
Regrading the deficiencies mentioned above, in this paper we argue that it is beneficial to explore the implied data manifold by learning the topological relationship between data points.
To do so, we propose to seamlessly integrate multiple affinity graphs into a consensus one with the topological relevance considered.
Besides, we manipulate the consensus graph by a connectivity constraint such that the connected components precisely indicate different clusters.
Hence our model is able to directly obtain the final clustering result without reliance on any label discretization step.
By leveraging the subtasks of affinity graph constructing, topological relevance learning, and discrete label partitioning into a unified framework, each subtask can be enhanced in a mutual reinforcement manner.
An alternating iterative algorithm is carefully designed to solve the optimization problem of the proposed model.
Experimental results on several benchmark datasets demonstrate the effectiveness of our method.
Notations.
We use boldface uppercase letter, e.g., M , to denote the matrix.
M ij represents the ij -th element of M .
∥·∥ F represents the Frobenius norm of a matrix and 1 is a column vector with all its elements equal to 1.
I denotes the identity matrix with proper size.
Preliminary
It is well-known that real-world datasets are often sampled from a nonlinear low-dimensional manifold which is embedded in the high dimensional ambient space [15, 18–20].
Hence it is necessary to exploit the manifold structure implied within the original data.
Recently, [17] pointed out that it is preferred to boost the learning performance by making use of the manifold topological structure than the Euclidean structure.
It is based on a simple yet intuitive assumption that the topological connectivities between individuals could be propagated from near to far.
In other words, the spatial similarity between two individuals may be small, but their topological relevance to each other would be high if they are linked by consecutive neighbors.
A visualized example is given in Figure 1.
For better illustration, we visualize the topological relevance in both a 2D version (left subfigure) as well as a 3D version (right subfigure).
Taking the left subfigure as an example, although the dark blue and light grey points are with low similarity in terms of spatial location and velocity, they are closely connected to each other considering the high topological relevance between them.
Instead of relying on the Euclidean structure, it is naturally expected to learn a more suitable manifold topological structure such that the intrinsic similarities can be explicitly uncovered.
Considering that data points with a high similarity would share similar topological relevance, given a predefined similarity graph Z ∈ R n × n , where n is the number of data points, [17] investigated the topological structure of data by solving
[Equation]
[Figure]
where β is a trade-off parameter.
i , j , and k are data point indexes.
S represents the target topological relationship matrix, and S ij denotes the data point j ’s topological relevance to i .
Note that the first term in Eq.
(1) is essentially a smoothness constraint that fits the above assumption, i.e., it guarantees the data points j and k share a similar topological relationship with data point i if they are similar.
And the second term is incorporated to prevent the trivial solution.
According to Eq.
(1), the topological consistency can be propagated through neighbors with high similarities, and the distant data points will keep a close relationship if they are linked by consecutive neighbors.
Notwithstanding, Eq.
(1) is fed with a predefined graph, which would lead to performance degradation as the predefined graph Z might not be optimal for subsequent learning procedures.
Hence it is preferred to automatically learn similarity graphs from data.
Self-expression has been widely employed to recover the similarity graph in the form of subspace representation [6].
Given a dataset X =
[ x 1 , x 2 , · · · , x n ] ∈ R m × n with n data points and m features, the self-expression based clustering problem can be defined as
[Equation] [Equation]
where α is a trade-off parameter and diag ( Z ) represents the vector consists of diagonal elements of Z .
It is obvious that Eq.
(2) learns the neighboring data points of a data point and the corresponding weights by the sparse reconstruction from other data points.
Moreover, the learned sparse representation enjoys several nice properties, e.g., the datum-adaptive ability and robustness to noise [21, 22].
Based on Eq.
(2), we can easily extend it to a multi-view formulation when dealing with the multi-view data { X (1) , X (2) , · · · , X ( m ) } , which can be formed as
[Equation]
where m is the number of views, and Z ( v ) (1 ≤ v ≤ m ) denotes the affinity representation that automatically learned from the v -th view.
The Proposed Methodology
To explicitly incorporate the manifold topological structure into multi-view clustering, we propose to learn a topological relation graph based on Eq.
With the subspace representation automatically obtained in Eq.
(3), we arrive at min Z ( v ) , S m
[Equation]
where the consensus graph S reveals the topological relationship across multiple views and is further constrained to be non-negative and the sum of each row is 1 .
w v is the weight for the v -th views, which can be considered as a constant within each iteration and will be introduced later.
Nevertheless, the learned S in Eq.
(4) does not contain explicit cluster structures.
It is expected that the consensus graph contains exactly c connected components where c is the number of clusters.
It sounds unrealistic that such a clear structured S could be achieved out of thin air.
Fortunately, we can solve this problem with a useful connectivity constraint.
Moreover, in Eq.
(4), we see that if the j -th data point is connected with many similar neighbors, it will largely affect the objective value.
Thus we tend to normalize the objective function so that each point can be treated equally.
Considering the above concerns, we propose a normalized version of Eq.
(4) with an additional connectivity constraint, which can be finally formulated as
[Equation]
where D ( v ) is the degree matrix of Z ( v ) , L S is the Laplacian matrix of S , and rank ( L S )
= n − c is a rank constraint that manipulates the target graph S containing exactly c connected components.
In this way, the subsequent label discretization step can be avoided since each connected component precisely indicates an individual cluster.
As for the weight w v of each view, we set it in a self-tuned way inspired by [23, 24]:
[Equation] [Equation]
It is worth mentioning the salient contributions of our model formulated in Eq.
(5) as follows:
• Orthogonal to other multi-view subspace clustering approaches, our model explicitly explores the implied data manifold by learning the topological relationship across multiple views.
To the best of our knowledge, this is the first work that delicately incorporates the topological structure in multi-view subspace clustering.
• Our model flawlessly integrates the subtasks including affinity graph constructing, manifold topological structure learning, and discrete label partitioning into a unified framework.
Hence our model is an end-to-end single-stage learning paradigm.
• An alternating iterative algorithm is introduced to solve the optimization problem.
Experiments on several benchmark datasets illustrate the effectiveness of the proposed model, compared to the state-of-the-art competitors over the clustering performance.
3.1 Algorithm Derivation
In order to solve the problem in Eq.
(5), we derive an optimization algorithm to optimize the objective function.
Since the corresponding optimization problem is not jointly convex in all variables, we choose to solve it by updating one variable while fixing other variables.
3.1.1 Update Z ( v ) for Each View
For Z ( v ) , the corresponding problem is
[Equation]
Eq.
(7) can be rewritten as min Z
[Equation] [Equation]
where H ( v ) =
(cid:0) X ( v ) − X
( v ) Z ( v ) (cid:1) (cid:0) X
( v ) − X
( v ) Z ( v ) (cid:1) T .
Taking the derivative of Eq.
(8) w.r.t Z ( v ) and setting the derivative to zero, we get following solution
[Equation]
where G = D − 12 S , and K ( v )
=
X ( v ) T X ( v ) can be treated as a liner kernel.
Note that the nonlinear kernel can also be applied in our model.
3.1.2 Update S
Dropping the unrelated terms of Eq.
(5) w.r.t.
S , thus we need to solve
[Equation]
We see that Eq.
(10) is difficult to solve due to the rank constraint.
Let σ i ( L S ) be the i -th smallest eigenvalue of L S .
The constraint rank ( L S )
= n − c would be satisfied if (cid:80) ci =1 σ
i
L S )
= 0 since
L S is a positive semidefinite matrix.
According to Ky Fan’s Theorem [25]:
[Equation]
where F ∈ R n × c denotes the cluster indicator matrix.
We incorporate the constraint term (cid:80) ki =
σ i ( L S ) , i.e., min F Tr (cid:0) F T L S F (cid:1) , into the cost function, thus Eq.
(10) can be rewritten as min S , F  m (cid:88) v = w v n (cid:88) i,
[Equation]
where λ is a self-tuned parameter.
Based on Eq.
(12), first we search the optimal solution of F , which can be obtained by solving
[Equation]
which is essentially a spectral problem and F can be achieved by calculating the c eigenvectors of L S corresponding to the c smallest eigenvalues.
Then we search the optimal solution of S .
It is clear that Eq.
(12) w.r.t.
S can be formulated in a vector form as min S n (cid:88)
[Equation] [Equation]
−
[Equation]
D
[Equation]
v )
[Equation]
+
[Equation]
(cid:88)
j =
[Equation] [Equation]
− I ij )
[Equation]
+ λ n (cid:88)
j = ∥
f i − f j
[Equation] [Equation]
ij (cid:41) s .
t .
s
Ti 1 = 1 , s
[Equation]
Note that Eq.
(14) is independent for different i , thus we have the following compact formulation
[Equation]
where e i denotes the i -th column vector of identity matrix I , and u i represents a vector with its j -th element u ij = λ
∥ f i − f j ∥ 22 .
Denote A = (cid:80) mv =1 w v (cid:16) I − D ( v )
− 12 Z ( v ) D
( v ) − 12 (cid:17) + β I and b = 2 β
e
i − u
i , Eq.
(15) is equivalent to optimizing
[Equation]
We see Eq.
(16) is essentially a quadratic convex optimization problem, hence it can be solved with the classical augmented Lagrangian multiplier (ALM) method [26].
Specifically, we can solve
Eq.
(16) by tackling its counterpart
[Equation]
Based on Eq.
(17), the corresponding augmented Lagrangian function can be defined as
[Equation]
where the second term in Eq.
(18) is employed to guarantee that p
= s i , η and q are the penalty coefficient and parameter, respectively.
It is clear that we can solve p and s i in an iterative updating way: 1) Update p with fixed s i .
The corresponding cost function w.r.t.
p is
[Equation]
Setting ∂ L p ∂ p = 0 , we obtain
[Equation]
2) Update s i with fixed p .
The corresponding cost function w.r.t.
s i can be reformulated as
[Equation]
which leads to a closed-form solution and can be directly obtained by [21]
According to the
ALM principles [26], η can be exaggerated increasingly during each iteration, and q is updated by q ← q + η ( s i − p ) .
Finally, we can obtain a optimal solution for S based on this effective ALM strategy.
The detailed algorithm to solve Eq.
(18) is outlined in Algorithm 2.
Up to now, with the variable Z ( v ) updated by Eq.
(9), S updated by Eq.
(18), F updated by Eq.
(13), and w v updated by Eq.
(6), the overall algorithm to solve the objective in Eq.
(5) can be summarized in Algorithm 2.
Owing to space limitation, we give the convergence analysis and the time complexity of Algorithm 2 in the Appendix B and Appendix C, respectively.
Experiments
We evaluate the proposed method by comparing it with following state-of-the-art competitors:
Diversity-induced Multi-view Subspace Clustering ( DiMSC ) [8], Auto-weighted Multiple Graph
Learning ( AMGL ) [27], Multi-view Clustering with Graph Learning ( MVGL ) [28], Weighted Multiview Spectral Clustering( WMSC ) [29], Consistent and Specific Multi-view Subspace Clustering ( CSMSC )
[10], Graph-based Multi-view Clustering ( GMC ) [24], Large-scale Multi-View Subspace
Clustering in linear time ( LMVSC ) [12], Scalable Multi-View Subspace Clustering with unified anchors ( SMVSC ) [13], and Multi-View Subspace Clustering via Co-training robust data representation ( CoMSC ) [11].
The classic graph-based algorithm, spectral clustering (SC) [18], is also included as a baseline.
We apply SC by making use of the most informative view, i.e., one that achieves the best performance (denoted by SC best ).
[Table]
The experiments are conducted on several benchmark datasets, namely, 3Sources, MSRC, 100Leaves,
COIL-20, Caltech-7, Caltech-20, and MNIST.
The detailed information of all datasets is given in
Appendix A. The specific characteristics of these datasets are also summarized in Table 1.
The parameters for comparison algorithms are set according to the recommendations in their corresponding paper.
The parameter settings of our model will be introduced later.
We repeat all algorithms 10 times to eliminate the effect of random factors, and report the average scores.
Four widely-used metrics, i.e., clustering accuracy (ACC), Normalized Mutual Information (NMI), Purity, and F-score, are used to achieve a comprehensive evaluation.
[Table]
91.90 Caltech28.
[Table] [Table]
[Table]
100Leaves 86.
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
MNIST 47.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] Purity 3Sources
[Table] [Table] [Table]
[Table]
MSRC
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
60 100Leaves 72.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
.02 MNIST 56.
87.44 F-score 3Sources 48.
75.25 MSRC 43.
75.29 COIL69.
82.29 Caltech40.
79.77 100Leaves 61.
69.29 Caltech33.
53.13 MNIST 41.
[Figure]
4.1 Results and Analysis
The clustering performance of all methods on four metrics is shown in Table 2.
As we see, our proposed method achieves the best performance in the majority of cases, which validates the effectiveness of our method.
Moreover, it is obvious that the improvement is remarkable.
On dataset MSRC, for instance, our method even achieves substantial improvements around 4.7%, 4.2%, 4.3%, and 10.5% over the most competitive methods in terms of ACC, NMI, Purity, and F-score, respectively.
Note that DiMSC, CSMSC, LMVSC, SMVSC, and CoMSC are all multi-view subspace clustering methods that adopt the Euclidean structure.
From the results, it is obvious our method that employs manifold topological structure is much better than that in terms of clustering performance, which verifies our assumption that manifold topological structure is more suitable to explicitly uncover the intrinsic similarities.
Taking the dataset 100Leaves as an example, we visualize the target graph learned by different methods.
For better visualization, we plot the first seven categories of dataset 100Leaves in Figure 2.
As can be seen, DiMSC cannot even find the block diagonal structure of the target graph.
MVGL, CSMSC, and CoMSC are able to search the block diagonal structure but contain lots of noise obviously.
On the contrary, our model almost achieves a pure structured graph with a much clear clustering structure.
[Figure]
4.2 Parameter Analysis
This section investigates the sensitivity of our method with respect to different parameter settings.
Note that λ involved in Eq.
(12) is a self-tuned parameter, which can be set in a heuristic way.
That is, we can initialize λ to a random positive value (e.g., λ = 10 ), if the connected components of S is greater than c in current iteration, we set λ ← λ 2 , and if it is less than c , we set λ ← 2 λ .
This strategy guarantees S contains a clear clustering structure with exact c connected components.
Hence we only need to set the parameters α and β properly.
Here we empirically search them in the range [0.1,0.5,1,5,10,50] for simplicity.
The clustering performance of three datasets is shown in Figure(owing to space limitation, the results of other datasets are attached in Appendix D), we can find that the clustering results of our method are relatively stable for different parameter settings, which demonstrates the robustness of our model.
As introduced before, the target consensus graph S can be treated as an indicator matrix, where the points from the same cluster are connected to the same component.
Once we obtain S , the cluster label of each data point can be directly assigned without any postprocessing.
Hence our method is very stable.
In general, we could obtain a promising clustering performance by setting α = β = 10 in practical applications.
[Figure]
4.3 Convergence Study
Here we verify the convergence property of the proposed algorithm.
The convergence curves along with the corresponding performance of our algorithm on 3Sources, MSRC, and COIL-20 datasets are recorded in Figure 4 (the convergence curves of other datasets are attached in Appendix D), where the red line denotes the NMI of our method and the grey line indicates the objective value.
As we can see, the proposed optimization algorithm is efficient and converges very fast, usually withiniterations, which empirically illustrates the efficiency of the proposed algorithm.
Conclusion
In this paper, we propose to explore the implied data manifold by learning the topological relationship between data points.
To do so, we seamlessly integrate multiple affinity graphs into a consensus one with the topological relevance considered.
Besides, we manipulate the consensus graph by a connectivity constraint such that the connected components precisely indicate different clusters.
Hence our model can directly obtain the discrete result without any postprocessing.
An alternating iterative algorithm is carefully designed to solve the optimization problem of the proposed model.
Experimental results have shown that (1) manifold topological structure is suitable to explicitly uncover the intrinsic similarities, thus beneficial for multi-view subspace clustering tasks; (2) our model is quite robust with respect to different parameter settings, which demonstrates its stability; (3) the proposed optimization algorithm is very efficient and converges fast.
However, note that our model cannot deal with the nonlinear data, which can be considered in future works.
As mentioned in Subsection 3.1.1, the kernel strategy would be an option to explore nonlinear structure.
Furthermore, we are also interested in extending the proposed framework to other machine learning frameworks, such as semi-supervised learning and deep learning.
Acknowledgments
We thank the anonymous reviewers for their helpful comments and suggestions.
This work is supported by the Key Program of National Science Foundation of China (Grant No. 61836006), partially supported by the National Science Foundation of China under Grant 62106164 and 62106161, the 111 Project under Grant B21044, and the Sichuan Science and Technology Program under Grants 2021ZDZX0011 and 2022YFG0188.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide them as a URL.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The important details are provided in Section 4.2 and in the code.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
For fair comparison, we repeat all algorithms 10 times.
Considering the appearance of the Table we did not report the standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix D.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See Appendix A. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We include the proposed algorithm in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] All assets used are open-source.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]