G RAPH C LUSTERING WITH THE W EAK S UPERVISION FROM G RAPH L ABELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Clustering, which clusters the nodes of a graph given its collection of node features and edge connections in an unsupervised manner, has long been researched in graph learning and is essential in certain applications.
While this task is common, more complex cases arise in practice—can we cluster nodes better with some graph-level side information or in a weakly supervised manner as, for example, identifying potential fraud users in a social network given additional labels of fraud communities.
This triggers an interesting problem which we deﬁne as Weakly Supervised Graph Clustering (WSGC).
In this paper, we ﬁrstly discuss the various possible settings of WSGC, formally.
Upon such discussion, we investigate a particular task of weakly supervised graph clustering by making use of the graph labels and node features, with the assistance of a hierarchical graph that further characterizes the connections between different graphs.
To address this task, we propose Gaussian Mixture Graph Convolutional Network (GMGCN), a simple yet effective framework for learning node representations under the supervision of graph labels guided by a proposed consensus loss and then inferring the category of each node via a Gaussian Mixture Layer (GML).
Extensive experiments are conducted to test the rationality of the formulation of weakly supervised graph clustering.
The experimental results show that, with the assistance of graph labels, the weakly supervised graph clustering method has a great improvement over the traditional graph clustering method.
I NTRODUCTION
With the development of deep graph learning, graph clustering has achieved signiﬁcant improvement in community detection (Jin et al., 2019), text clustering (Aggarwal & Zhai, 2012) and other applications (Yang et al., 2010).
Such applications all assume that only node/edge information and connections are accessible to assist with graph clustering in an unsupervised manner.
However, it is possible that more graph-level side information, such as graph labels, may also be accessible and helpful for clustering the nodes in graphs.
Or namely, can we use the global label of the graph to help improve the effectiveness of graph clustering?
This problem is interesting and very common in the real world.
For example, point cloud segmentation, which could be re-considered as clustering of nodes in point cloud if we only know the objects contained in each point cloud but not all the node labels in the training process instead of predictions for nodes with known node labels that are solved by supervised graph learning methods (Ye et al., 2018; Li et al., 2019).
Moreover, in social media, thinking of hot events that are widely discussed in the form of graphs where users are involved as nodes and the mutual following relationship among users as edges, it is interesting to think how to pick out the malicious users from the mass of users with the labels of these event topics such as the authenticity of each topic (Zhou & Zafarani, 2018; Cao et al., 2018).
Actually, this problem is more like graph clustering in a weakly supervised manner (Zhou, 2018), or named Weakly Supervised Graph Clustering (WSGC), where the available training labels are much coarser-grained than the node labels we want to ﬁt.
Unlike traditional graph clustering (Girvan & Newman, 2002) which only conducts clustering based on node features or graph structures, WSGC is better guaranteed by the given label information of graphs, which, we will demonstrate, closely inﬂuences the clustering results.
Note that WSGC is different from traditional weakly supervised learning problems.
Generally, weakly supervised learning is considered to consist of three types of weak supervision problems: incomplete supervision, inaccurate supervision and inexact supervision (Zhou, 2018).
WSGC is different from incomplete supervision where a subset of training nodes is given with labels and inaccurate supervision where the given node labels are not always ground-truth, since WSGC does not contain any node labels for training.
It is also different from inexact supervision, whose representative method as Multiple-Instance Learning (MIL)
(Carbonneau et al., 2018) adopts global labels of bags to identify the labels of local instances.
Because MIL assumes that each instance is i.i.d.
and there are no edge connections involved.
As far as we know, there are no studies on WSGC even though WSGC is very useful to formulate some real-world problems.
Therefore, it is interesting and important to study the WSGC as a new challenge and seek new solutions for it.
In this paper, we formally deﬁne Weakly Supervised Graph Clustering (WSGC), i.e. a graph clustering task by making use of graph labels, graph structure and node features.
To address this particular task, we propose a novel model based on Gaussian mixture model (GMM) and graph convolutional network (GCN) named as Gaussian Mixture Graph Convolutional Network (GMGCN).
The proposed model includes a Gaussian mixture layer (GML) and a new hierarchical GAT (hierGAT) to update the node and graph hidden representations, respectively.
Speciﬁcally, we design a consensus loss that plays a key role in improving node clustering with the assistant of graph labels in the training process.
Finally, a graph clustering is carried out according to the parameters of GML.
The main contributions are as follows:
New problem: We introduce a new problem,
i
e. Weakly Supervised Graph Clustering (WSGC), which tries to identify the nodes in graphs based on the labels of graphs, and we take a formal discussion of the variants of this problem to attract more research attention on this
problem. New solution: We propose an effective model called GMGCN based on GMM and GCN to solve WSGC.
To the best of our knowledge, this is the ﬁrst work to achieve graph clustering in graph structure by integrating GMM into GCN.
New loss: We propose a consensus loss function to boost the model training through the principle of ”same attraction, opposite repulsion”.
Experiments validate the effectiveness of this consensus loss.
We validate the signiﬁcant improvement of WSGC over traditional clustering methods on various synthetic and real-world datasets.
Compared to the best performing baseline method, GMGCN improves Normalized Mutual Information 1 (NMI) by an average of 13% on 5 synthetic datasets, by more than 13% on PHEME social media datasets, and improves mean intersection-over-union(mIoU) by more than 18% on point cloud segmentation datasets.
R ELATED W ORKS
Graph Learning.
Recently, there has been an increasing interest in the graph learning domain.
Among all the existing works, GCN is one of the most effective convolution models.
A typical GCN model is the message passing neural network (MPNN) proposed by Gilmer et al .
(Gilmer et al., 2017) which re-generalizes several neural network and graph convolutional network approaches as a general ”message-passing” architecture.
Many kinds of GCN (Bruna et al., 2014; Defferrard et al., 2016; Kipf & Welling, 2017; Chang et al., 2020) deliver different message propagation functions for
GCN.
Among them, Graph Attention Networks (GAT) (Veliˇckovi´c et al., 2017)
ﬁrst leverages learnable self-attentional layers to aggregate weighted neighbors’ information.
Besides these methods to obtain the appropriate node representation, pooling strategies are proposed to integrate information over the node representations (Wu et al., 2020), such as max/min pooling (Defferrard et al., 2016),
SortPooling (Zhang et al., 2018), and so on.
In addition, Lin et al .
(Lin et al., 2017) propose a learnable attentive pooling for weighted averaging of node representations.
Despite extensive research on node-level and graph-level graph learning tasks, few studies have been conducted on WSGC.
Graph Clustering.
Graph clustering is a fundamental data analysis task that aims to group similar nodes into the same category.
Many real-world applications are cast as graph clustering (Shi & Ma- lik, 2000; Hastings, 2006).
The major strategy of graph clustering is to perform traditional clustering algorithms such as K -means (Jain, 2010) or GMM (McLachlan & Basford, 1988) on the node features.
With the great achievements of deep learning, more graph clustering studies have resorted to graph learning to learn embedding that captures both node features and structural relationships (Wu et al., 2020).
Researchers employ the stacked sparse autoencoder (Tian et al., 2014), the variational autoencoder (Kipf & Welling, 2016), or the combination of both autoencoder and GCN (Bo et al., 2020) to obtain graph representations for clustering.
Nevertheless, these graph clustering methods do not perform better than WSGC methods due to the lack of attention to graph labels.
Multiple-Instance Learning.
Another task that has a similar deﬁnition to WSGC is MultipleInstance Learning (MIL).
MIL is a variant of inductive machine learning, where each learning example consists of a bag of instances instead of a single feature vector (Foulds & Frank, 2010).
When obtaining local instances annotations is costly or not possible, but global labels for bags are available, MIL is utilized to train classiﬁers using weakly labeled data.
It has received a considerable amount of attention due to its applicability to real-world problems (Andrews et al., 2002a; Cheplygina et al., 2019).
Although MIL learns a classiﬁer from the labels of bags to classify instance, it ignore structural information between instances and are therefore not suitable for WSGC.
Gaussian Mixture Model.
As a core part of the proposed GMGCN model, GMM is a parametric probability density function for a weighted sum of Gaussian component densities (McLachlan & Basford, 1988).
It is commonly used to ﬁnd underlying clusters in data samples (Bishop, 2006).
Generally, the GMM parameters are estimated using the iterative Expectation-Maximization (EM) (Moon, 1996) algorithm from training data.
In this paper, we integrate the GMM into GCN to establish a solution for WSGC and update the GMM parameters using stochastic gradient descent.
Point Cloud Semantic Segmentation.
Semantic segmentation of 3D point clouds has attracted more researchers’ attentions due to the developments of 3D scanners, such as Light Detection and Ranging (LIDAR), structure-from-motion (SFM) techniques, etc.
To operate directly on these unstructured point clouds, PointNet (Qi et al., 2017a) is proposed to subdivide the input points into a grid of blocks and process each such block individually, but limited neighborhood context of each point is taken into consideration.
To overcome this drawback, many approaches (Engelmann et al., 2017; Ye et al., 2018; Qi et al., 2017b) are built upon PointNet and better capture local spatial structures as well as long dependency context.
Recently, some latest works (Qi et al., 2017c; Landrieu & Simonovsky, 2018; Wang et al., 2019) re-consider semantic segmentation of unstructured 3D point clouds as predictions for nodes on graph data and incorporate contextual information and topological information on top of 3D points.
However, these supervised graph learning methods are not suitable for weakly supervised point cloud semantic segmentation datasets if the point labels are unknown.
N OTATIONS AND P ROBLEM S TATEMENT 3. N OTATIONS
We denote a set of graph instances as G =
{ ( G (1) , y (1) ) , . . .
, ( G ( N ) , y ( N ) ) } with N graphs, where
G ( n ) refers to the n -th graph instance and y ( n ) ∈ { 0 , 1 , . .
, C graph − 1 } is the corresponding graph label of C graph different categories.
We denote by G ( n ) =
( V ( n ) , E ( n ) )
the n -th graph instance of size M n with nodes v ( n ) i ∈ V and edges ( v ( n ) i , v ( n ) j ) ∈ E ( n ) , by X ( n )
=
{ x ( n ) 1 , . . .
, x ( n ) M n } ∈ R M n × d the feature matrix of nodes V ( n ) where d denotes the number of dimensions of node original features, and by A ( n ) ∈ { 0 , 1 } M n × M n the adjacency matrix which associate edge ( v ( n ) i , v ( n ) j ) with A ( n ) i,j .
We denote by { z ( n ) 1 , . . .
, z ( n ) M n } ∈ { 0 , 1 , . .
, C node − 1 } the potential labels of nodes V ( n ) where C node is the expected number of node categories.
In addition, we denote by A hier ∈ { 0 , 1 } N × N the adjacency matrix of the links among graphs if the graphs in G contain interconnections.
3. P ROBLEM S TATEMENT
The Weakly Supervised Graph Clustering (WSGC) problem is deﬁned as: Given a set of graph-label pairs G and node features X , how to infer the label of the i -th node in the n -th graph, z ( n ) i , where n ∈ { 1 , . . .
, N } and i ∈ { 1 , . . .
, M n } ?
Under the deﬁnition of WSGC, there are some different cases that could be extended from the problem as new tasks.
Case 1 (Infer Original Nodes)
Given the whole G and X as training data, how to infer all the node labels z ( n ) i in the G ( n ) s?
Case 2 (Infer New Nodes)
Given the portion of nodes V ( n ) train = { v ( n ) 1 , . . .
, v ( n ) M n − s } in each G ( n ) as training data, how to infer the labels of the rest of s nodes { v ( n ) M n − s +1 , . .
. , v ( n ) M n } in the G ( n ) s?
Case 3 (Infer New Graphs)
Given the portion of graphs
G train =
{ ( G (1) , y (1) ) , . . .
, ( G ( N − S ) , y ( N − S ) ) } as training data, how to infer the node labels z ( n ) i , n ∈ { N − S + 1 , . . .
, N } in the rest of S graphs, {G ( N − S +1) , . .
, G ( N ) } ?
Case 4 (With/Without Hierarchical Graph) Given the interconnections among graphs as A hier , how to infer the node labels?
And if there are no interconnections, how to infer the node labels (i.e.
A hier = I )?
P ROPOSED M ETHOD
In this section, we ﬁrst introduce the preliminaries related to the proposed model, then we discuss details of the proposed method.
4. P RELIMINARIES
Graph Attention Networks.
Graph Attention Networks (GAT) (Veliˇckovi´c et al., 2017) has been widely adopted in the ﬁeld of graph convolution due to the learnable attentions to neighbors during the message passing process.
A multi-head GAT Convolutional layer (GATConv) is formulated as:
[Equation]
j
[Equation]
[Equation] α [Equation] (1) where (cid:126)h i and (cid:126)x i refer to the hidden feature representation for the next layer and the current hidden feature vector of node i in the graph, respectively.
(cid:107) refers to a concatenation operation for K -head attention, and σ ( · ) refers to the nonlinear activation function.
Θ is the weight matrix and N ( i ) is the set of neighbors of node i .
α ki,j is the attention coefﬁcient computed by the k -th attention mechanism as follows:
[Equation]
where attention mechanism is a single-layer feedforward neural network parametrized by a weight vector a and applying the LeakyReLU nonlinearity.
Graph Attentive Pooling.
Lin et al .
(Lin et al., 2017) employ a self-attentive mechanism to learn the importance of each node and then transform a variable number of node representations into a ﬁxed-length graph representation:
[Equation]
where Θ s 1 and Θ s 2 refer to trainable parameters.
The uniﬁed graph representation of the n -th graph h ( n ) g is obtained by multiplying the node attention scores with H ( n ) .
4. O VERALL F RAMEWORK
Fig.
1 gives an overall framework of the proposed method to infer the categories of the nodes in the graphs.
It consists of ﬁve processes: node-level modeling, node feature redistribution, graph instance modeling, hierarchical graph modeling and consensus loss calculation.
The node-level modeling consists of 2-layer Multi-Head GAT and the graph instance modeling is accomplished by aggregating graph-level representations from the node-level GML via graph attentive pooling.
In this section, we will introduce the rest of the three key components of the proposed framework. [Figure] D C
C o n s e n s u s L o ss
Gaussian Mixture Layer (GML).
As the key component of the GMGCN, GML plays a crucial role in the process of node feature redistribution.
It is described as (cid:126)h
[Figure]
[Equation]
where w c ( · ) represents the Gaussian weight function as follows:
[Equation]
(4) where (cid:126)σ c and (cid:126)µ c are learnable standard deviation vector and mean vector, respectively, and Θ GML refers to the learning parameters.
Based on the idea of GMM, GML has the ability to effectively distinguish different types of nodes according to the input features.
However, it could not resolve two major challenges of WSGC: the node-level modeling and the graph instance modeling that affects the ﬁnal node inference.
Therefore, we employ the node representations from the 2-layer GAT as inputs of GML.
The node representations provide local structural information of the nodes to GML.
Meanwhile, the outputs (cid:126)h (cid:48) i of
GML are aggregated by attentive pooling as h ( n ) g , and further fed into a hierarchical GAT.
Hierarchical GAT (hierGAT).
As mentioned in Case 4 , the information of interconnections among graphs may infer the node labels.
To model such hierarchical structure information, we employ a 3-layer hierGAT to update the feature representations in the hierarchical graph as Eq.
[Equation]
where l ∈ { 0 , 1 , 2 } refers to the layer number of the 3-layer hierGAT, α hier refers to the hierarchical attention coefﬁcients, and H
g = H g , which is the hierarchical feature matrix that consists of the feature representations of graphs, h ( n ) g .
Each hierarchical attention coefﬁcient α hierij in α hier is caluculated as Eq.
We deﬁne the ﬁnal output of the 3-layer hierGAT
H 3 g as H (cid:48) g .
There are no training parameters that causes the spatial transformation in Eq.
(5) so that the node embeddings H (cid:48) ( n ) and graph embeddings H (cid:48) g are still in the same hidden space.
Note that, the hierGAT is reduced to a version without using the hierarchical graph when β = 0 .
Consensus Loss.
The core idea of consensus loss is to make the graph representations with the same labels closer, and those with different labels farther away in the hidden space.
Therefore, before computing the consensus loss, we ﬁrst compute the similarity matrix S ∈ R N × C graph of the graph representations, which is obtained by the following steps:
First, we compute the distance between the graph representation (cid:126)h (cid:48)
( n ) g obtained via hierGAT and the mean vector (cid:126)µ c of c -th Gaussian weight function as follows: [Equation] n ) g −
[Equation]
Second, we enhance the distance of the graph representations to the true mean vector on d nc by:
[Equation]
where δ is a discount hyperparameter, y ( n ) is the true label of graph G ( n ) , and d ny ( n ) represents the distance between the graph representation (cid:126)h (cid:48)
( n ) g and the mean vector (cid:126)µ y ( n ) .
Then the similarity matrix S is formulated as S = softmax( − D enhance ) , where D enhance consists of ( d nc ) enhance .
Finally, the consensus loss is formulated as l con := cross entropy ( S , y ) .
Model training Guidance.
As shown in Fig. 1(b), the consensus loss makes the graph representation of each graph (e.g. h g ) closer to the mean vector of the Gaussian function corresponding to its graph label (e.g. the yellow arrow), so as to categorize the graphs by gathering those graphs with the same labels together and isolating those graphs with different labels (e.g. the blue and green arrow) away in the graph updating process.
Meanwhile, node representations (e.g. h (cid:48) ) will be closer to the mean vector of their potential corresponding Gaussian functions through node updating process.
4. I NFERENCE OF N ODES
We infer the categories of the nodes in the graph based on the distance between the node representation (cid:126)h (cid:48)
i obtained from GML and the cluster centers, i.e. mean vector (cid:126)µ of Gaussian functions.
We use C node to denote the expected number of node clusters, as in traditional clustering methods this is a hyperparameter that needs to be set by the user.
When C node < C graph , clustering algorithm such as K-Means (Ding & He, 2002) is used to cluster C graph mean vectors of Gaussian functions to get C node cluster centers.
Then, the category ˆ
z i of the i -th node is determined as follows:
[Equation]
When C node > C graph , which is relatively rare in practice, clustering algorithm is used to cluster node representations (cid:126)h (cid:48) to C node clusters directly.
E XPERIMENTS
In this section, to validate the effectiveness of GMGCN under different cases, we ﬁrst conduct the experiments on several synthetic datasets.
Then we constructed two real-world datasets, the number of graph categories and node categories in these two real-world datasets are both 2.
5. B ASELINES & E XPERIMENTAL S ETTINGS
Since WSGC is a new challenge without any baseline work yet, we compare the proposed method with three types of most similar methods: Feature Clustering methods without structure information (Jain, 2010; McLachlan & Basford, 1988; Hinton & Salakhutdinov, 2006), Graph Clustering methods (Kipf & Welling, 2016;
Bo et al., 2020) and MIL (Ray & Craven, 2005; Andrews et al., 2002b) methods.
We also examine the variant of the proposed method as ATTGCN : GMGCN without GML, we replace node inference with multi-head attention mechanisms in the attentive pooling layer, then distinguish the categories of the nodes by comparing attention scores.
We set β = 0 .
1 in the hierGAT, the experimental results for β equals other values are shown in Appendix C.3.2.
The parameters are updated using stochastic gradient descent via Adam algorithm (Kingma & Ba, 2014).
The training process is iterated upon 800 epochs for GMGCN.
We apply Normalized Mutual Information (NMI) (Strehl & Ghosh, 2002) and Adjusted Rand Index (ARI) (Hubert & Arabie, 1985) to evaluate the graph clustering results on Synthetic Datasets and PHEME Datasets, apply the intersection-over-union (IoU) (Jaccard, 1912), as our evaluation metric on S3DIS Datasets.
We run all methods 10 times to avoid extremes and report the average value with a standard deviation.
We only report the best results of each type of method in the sequel due to the page limitations.
More detailed baseline descriptions and results are available in Appendix C.
5. GMGCN ON S YNTHETIC D ATA
Synthetic Data Statistics.
We generate several synthetic datasets to examine the proposed WSGC problem with various cases.
We name each dataset by the number of graph categories and the [Table] [Table] number of nodes, for example, we use GC4NC4 to denote a synthetic dataset with the number of graph categories as 4 and the number of node categories as 4.
More detailed data statistics and data generation process of these synthetic datasets are detailed in Appendix C.3.1.
Case 1: Transductive learning.
All graphs are available in the training.
Tables 1 and 2 show that GMGCN outperforms all the baselines.
In particular, it improves NMI and ARI by an average of more than 13% and 15%, respectively, compared to the best performing baseline.
This indicates a signiﬁcant improvement of the graph clustering guided by graph labels.
Since MIL cannot handle an unequal number of graph categories and node categories, it has no results on GC4NC2 datasets.
Case 2 & Case 3: Inductive learning.
Portions of graphs or nodes are available in the training.
For the testing of new nodes and new graphs, we sample 20% of the nodes from each graph and 20% of the graphs in GC2NC2 as testing, respectively.
GMGCN also outperforms the other baselines as shown in Table 1.
Therefore, GMGCN works for both transductive and inductive settings.
Case 4: With hierarchical graph.
We regard the inclusion of a hierarchical graph structure as a special case (Case 4) of WSGC.
As shown in Table 1 and Table 2, the hierarchical graph structure promotes both the performance of ATTGCN and GMGCN.
This indicates that the structure information between graphs is also beneﬁcial to graph clustering.
Attention vs. GML.
We compare the ATTGCN with GMGCN to verify the importance of GML.
The results show that GML greatly promotes GMGCN to identify nodes, while the attention mechanism fails.
In addition, a more detailed ablation study is employed to illustrate the importance of each component of GMGCN, due to page limitation we present the experimental results in C.3.3.
Visualization.
We also visualize the node representations of 4 graphs randomly sampled from
GC4NC4 in a 2D space by t-SNE (Van Der Maaten, 2014).
As shown in Fig. 2(a)-2(d), GMGCN clearly distinguishes these 4 node categories but the others fail.
This indicates that the node representations learned by GMGCN are reasonable for WSGC.
Consequently, the out-performance of GMGCN for all cases imply that the proposed method is a perfect solution for WSGC.
[Figure]
5. GMGCN ON S3DIS D ATA
Data Description.
S3DIS (Armeni et al., 2016) is composed of 6 large-scale indoor areas, mainly conference rooms, ofﬁces and open spaces.
Each point is labeled with one of the semantic labels from 13 categories (ceiling, ﬂoor, wall, etc. plus clutter).
Based on the S3DIS dataset, we construct 13 binary segmentation datasets according to different categories.
For each category, the points in the category are annotated with 1, and the rest points are annotated with 0.
We treat each point as a node in a graph and apply k -NN to construct the edges between points.
Each point is represented by a 9-dim vector of each point’s 3D spatial coordinates and auxiliary features such as color and normalized location.
Each graph is annotated with the label that over 50% of the nodes in the graph belong to.
Because there is no potential correlation between each graph, this is a case without a hierarchical graph (Case 4).
A detailed data description is presented in Appendix C.4.1.
[Table]
Segmentation Results.
We compare the results of each method on the training data and the test data of the S3DIS dataset. As shown in Table [Table]
the graph clustering methods and GMGCN outperform feature clustering methods and MIL, which demonstrates that structural information plays an essential role in graph clustering.
Furthermore, GMGCN outperforms all the baselines in 12 out of 13 datasets, which explicitly reveals the effectiveness of using graph labels to enhance the training of the graph clustering models.
In addition, GMGCN outperforms the best-performing baselines by 18% on the training data and by 13% on the test data in terms of mIoU, which further implies a strong inductive learning ability of GMGCN.
However, GMGCN does not perform well on the Sofa dataset because the dataset containing only 4 graphs in the training is too small for the proposed method to learn enough information from these graphs with the help of graph labels.
5. GMGCN ON PHEME D ATA
Data Description.
PHEME consists of 5 real-life tweets sets, where each set is related to a piece of breaking news (Zubiaga et al., 2016).
Every breaking news includes a lot of rumor and non-rumor topics.
A detailed data description is presented in Appendix C.5.1.
Based on this dataset, we assume each topic as a graph, where links between users are formed according to their follows/retweets. The link between two graphs is constructed if they have common users.
We label the users appearing in more than M rumor topics as abnormal users and the rest as normal users.
When M = 2 , 3 , 4 , the proposed method all achieves the best results, the detailed results are shown in Appendix C.5.2.
In this section, we only show the results when M = 4 . [Table]
Clustering Results.
As shown in Table [Table] GMGCN without hierarchical structure (denoted by GMGCN w/o hier) perfectly resolves WSGC with an average 13% improvement on NMI and a 15% improvement on ARI on these ﬁve PHEME datasets compared with the best-performing baseline methods.
Furthermore, our method with hierarchical structure (denoted by GMGCN w hier) performs best, which indicates that the interconnections among graphs are helpful for inferring the categories of nodes in the graphs.
This signiﬁcant improvement implies that there exist certain applications in the real world that fulﬁll the statement of WSGC.
In contrast, the other methods perform poorly because WSGC is a totally new clustering problem that differs from feature clustering, graph clustering, or MIL.
It should be noted that the best performing feature clustering method in the PHEME dataset, DBSCAN(Ester et al., 1996), is deterministic and no variance is caused, so the standard deviation of feature clustering is 0 shown in Table 4.
Data Validity.
To show that the artiﬁcially constructed data is valid, we conduct a comparative experiment on PHEME datasets by augmenting the node features with a vector, each element of which counts how many graphs of a particular category this node belongs to.
In Table 4, Method+feat aug means the node features augmented with the vector.
Although most clustering methods have been improved by augmenting the node features, they are still generally inferior to GMGCN.
This is because although we take into account the number of rumor graphs that each node has appeared when labeling the node, these counts do not participate in the supervised learning, so even if we augment the node features with these counts, the traditional methods still cannot identify the node labels.
The experimental results demonstrate that our proposed method can deduce the correct results, and it is valuable to propose a GNN-based method to solve such WSGC problem.
D ISCUSSION AND F UTURE W ORK
In this work, we explore an interesting and novel problem: Weakly Supervised Graph Clustering (WSGC).
In one word, can we identify nodes given the labels of graphs?
The proposed WSGC problem has different notions and statements from any of the feature clustering, graph clustering, or multiple-instance learning tasks.
To address this new issue, we propose GMGCN with the integration of GMM and GNNs to resolve certain cases of WSGC problem.
Experimental results on
Synthetic datasets and real-world datasets reveal the need for formulating WSGC and the advantage of the proposed GMGCN over other related baselines.
Nevertheless, our study is just an initial step towards the weakly supervised graph learning problems while a variety of extensions are still potential.
For example, can we infer the labels of edges or sub-graphs instead of nodes?
Can we detect anomaly nodes other than conducting graph clustering?
We hope our study will open up a new vein of graph learning and encourage more speciﬁcations, solutions and developments for weakly supervised graph learning A A LGORITHM OF T HE P ROPOSED M ETHOD
Algorithm 1 shows the pseudo-codes of GMGCN.
Input: graph instances set G ; feature matrix of nodes X ; number of graph categories C graph ; hyper-parameter: C node Output: { ˆ z (1)1 , . . . , ˆ z (1) M n ; . . . ; ˆ z ( n ) 1 , . . . , ˆ z ( n ) M n ; . . .
; ˆ z ( N ) 1 , . . .
, ˆ z ( N ) M n } ∈ { 0 , 1 , . . .
, C node − 1 } 1: while GMGCN not converge do 2:
N = number of graphs in G 3: (cid:126)µ c = mean vector of c -th Gaussian weight function 4: for n = 1 , . . .
, N do 5:
G ( n ) = graph structure of n -th graph in G 6: y ( n ) = graph label of n -th graph in G 7:
M n = number of nodes in G ( n ) 8: for i = 1 , . .
, M n do 9:
Update node representation (cid:126)h i ← GAT ( x ( n ) i ) 10:
Update node representation (cid:126)h (cid:48)
i ← 1 C graph (cid:80) C graph c = w c ( (cid:126)h i , (cid:126)µ c ) 11: end for 12:
Calculate graph representation h ( n ) g ← Attn( H ( n ) ) 13:
Update graph representation h (cid:48)
( n ) g ← hierGAT
( H g ) 14:
Calculate consensus loss L ← f ( (cid:126)µ c , h (cid:48) ( n ) g , y ( n ) ) 15: end for 16:
Update parameters 17: end while 18: Cluster C graph mean vectors into C node mean vectors 19: for n = 1 , . .
. , N do 20: for i = 1 , . .
B D ISCUSSION AND P OTENTIAL I MPACT
Since the proposed Weakly Supervised Graph Clustering (WSGC) is a new weakly supervised learning task, we used two applications as examples, but in fact this task is also widely existed in drug discovery, 3D vision and natural language processing, etc.
Some potential examples are described below.
In drug discovery, thinking of molecules as graphs in which atoms as nodes (the functional groups can also be treated as super-nodes) and chemical bonds as edges (Gilmer et al., 2017).
By applying our method, we can identify the roles of certain nodes, sub-graphs, or equivalently functional groups, in each molecular given its chemical or physical properties.
In the ﬁeld of computer vision, weakly supervised semantic segmentation and object detection are very popular research areas.
In many practical scenarios such as autonomous driving (Arnold et al., 2019; El Madawi et al., 2019) we need to convert the images information into 3D point clouds, which can be further processed with a graph learning based approach.
Therefore our proposed method has a lot of space for exploration in such 3D vision scenarios.
In the ﬁeld of natural language processing, there is a lot of research nowadays that is combined with graph neural networks (Zhang et al., 2019;
Linmei et al., 2019).
For example, treating words, sentences or paragraphs as nodes and articles as graphs, if we have information about the sentimentality of each article, we can apply our weakly supervised approach to identify the sentiment expressed in each words, sentences or paragraphs.
In this section, we present the detailed experimental results corresponding to the Experiment section.
First of all, we introduce the baseline methods mentioned in the Experiment section.
Then, we present the detailed results on Synthetic, S3DIS and PHEHE datasets.
C. B ASELINES
Feature Clustering Methods:
• K -means (Jain, 2010):
A classical clustering method that aims to partition data points into K clusters in which each data point belongs to the cluster with the nearest cluster centroid.
• GMM (McLachlan & Basford, 1988):
A probabilistic model that assumes all the data points are generated from a mixture of a ﬁnite number of Gaussian distributions with unknown parameters.
• DBSCAN (Ester et al., 1996):
A density-based algorithm for discovering clusters in large spatial databases with noise.
• AE (Hinton & Salakhutdinov, 2006):
A two-stage deep clustering algorithm.
We perform GMM on the representations learned by autoencoder.
Graph Clustering Methods:
• GAE & VGAE (Kipf & Welling, 2016):
A structural deep clustering model that combines GCN with the (variational) autoencoder to learn representations.
we perform GMM on the representations learned by graph autoencoder.
• SDCN (Bo et al., 2020):
A structural deep clustering network model that integrates the structural information into deep clustering.
Multiple Instance Learning:
• SIL (Ray & Craven, 2005):
Single-Instance Learning (SIL) is a MIL approach that assigns each instance the label of its bag, creating a supervised learning problem but mislabeling negative instances in positive bags.
• miSVM (Andrews et al., 2002b):
A SVM-based MIL method.
After instance labels have been initialized, an SVM classiﬁer is trained and used to update the label assignation.
These two steps are performed iteratively until the label assignation remains unchanged.
We implement K -means, GMM and DBSCAN with scikit-learn 3 ; AE, GAE, VGAE, SDCN with
Pytorch 4 ; SIL, miSVM with a Python implementation created by (Doran & Ray, 2014).
To be consistent with related works (Bo et al., 2020), the dimension of AE is set to d -500-500-2000-10, where d is the dimension of the input data, and the dimension of GAE and VGAE is set to d -256-16.
For GMGCN, we adopt 2 independent attention mechanisms for the ﬁrst GATConv layer and useattention mechanism for the second GATConv layer. C. M ETRICS NMI. NMI (Normalized Mutual Information) is a normalization of the Mutual Information score to scale the clustering results between 0 (no mutualinformation) and 1 (perfect correlation).
It is calculated as follows:
[Equation]
where Y denotes the class labels, C denotes the cluster labels, H ( · ) means entropy calculated by
H ( Y ) =
− (cid:80) | Y | i =1 P ( i ) log ( P ( i )) , I ( Y, C ) means mutual information between Y and C is calculated by:
[Equation]
ARI The Rand Index (RI) computes a similarity measure between two clusterings by considering all pairs of samples and counting pairs that are assigned in the same or different clusters in the predicted and true clusterings.
It is calculated as follows:
[Equation]
The ARI score using the following scheme: ARI =
( RI − Expected RI ) /
( max ( RI ) −
Expected RI ) .
The adjusted Rand index is thus ensured to have a value close to 0.0 for random labeling independently of the number of clusters and samples and exactly 1.0 when the clusterings are identical (up to a permutation).
mIoU IoU is a commonly used metric in semantic segmentation measures similarity between ﬁnite sample sets,and is deﬁned as the size of the intersection divided by the size of the union of the sample sets, i.e., IoU = true positive/ ( true positive + false positive + false negative ) .
mIoU (mean
Intersection-over-Union) is a common evaluation metric for semantic image segmentation, which ﬁrst computes the IoU for each semantic class and then computes the average over classes.
C. GMGCN ON S YNTHETIC D ATA C.3. D ATA G ENERATION & D ATA S TATISTICS
First, according to the number of node categories C node , we generate C node synthetic user groups.
The edge relationships between users in same user groups are generated by the Barab ´ asi–Albert graph model (Barab ´ asi & Albert, 1999).
We use Gaussian distribution to randomly generate user features.
The mean and standard deviation of different types of users are randomly sampled from [- 5, 5] and [1, 10], respectively.
Second, we randomly connect users between user groups to simulate the relationship between different types of users.
Third, graph structures composed of different types of users is constructed by sampling users in different user groups.
In the case that the number of node categories is equal to the number of graph categories, the label of each graph is determined by the major labels of the users in the graph.
In other cases, we annotate the same label to graphs with the same percentage of users in each category.
Finally, we construct the connections between two graphs if they have more than 5 common users.
The data statistics of each Synthetic Dataset are shown in the Table C1.
[Table]
D ETAILED R ESULTS
Table C2 and Table C3 show the detailed experimental results of Table 1 and Table 2 in the Experiment section.
According to the clustering effectiveness, K -Means is selected as the representative of the feature clustering methods, SDCN is selected as the representative of graph clustering methods and SIL is selected as the representative of the MIL methods to be displayed in the Experiment section. [Table] [Table]
Fig.
C1 shows the results of GMGCN with hierarchical graph when β equals different values.
As shown in Fig.
C [Table] when β is greater than 0 and less than 0.5, GMGCN is more effective.
We choose β to be equal to 0.1 in the Experiment section. [Table] [Table] [Table]
[Table] [Table] [Figure] t
[Figure]
[Figure]
) β
Original Nodes NMI
New Nodes NMI
New Graphs NMI
Original Nodes ARI
New Nodes ARI
New Graphs ARI
[Table]
C.
[Table]
A BLATION S TUDY .
To verify the importance of each component of GMGCN
[Table]
we replace each one of the components separately to evaluate its performance and compare it with GMGCN.
The results are shown in Table
C
[Table]
where ’w/o 2-layer GAT’ denotes 2-layer GAT is replaced by 2-layer GCN; ’w/o GML’ denotes
GML-based architecture is replaced by attention-based architecture
i
[Table]
e., ATTGCN; ’w/o Attentive
Pooling’ denotes attentive pooling is repalced by mean pooling; ’w/o hierGAT’ denotes hierGAT is repalced by fully conected layers; ’w/o consensus loss’ denotes consensus loss is repalced by cross entropy loss for graph classiﬁcation.
The best performance of GMGCN indicates that each component of the proposed model plays an important role in predicting the node labels.
Speciﬁcally, the proposed GML greatly promotes
GMGCN to identify nodes, while the attention mechanism fails.
When hierGAT is replaced with fully connected layers, the experimental results became very unstable with large standard deviations, illustrating the importance of hierGAT.
Consensus loss also greatly improves the model effectiveness because it effectively guides the training of the mean vector of Gaussian functions.
Distributions of training data and test data in S3DIS datasets are shown in Table C5.
In order to maintain the balance of training data, we randomly sample some graphs annotated with 0, and ﬁnally make the number of graphs of different categories in the training set equal.
Based on this rule, we can see that objects such as Wall , Ceiling and Floor have sufﬁcient graphs for training, whereas objects such as Sofa and Table have very little graphs for training.
C.4. D ETAILED R ESULTS
Table C6 shows the detailed experimental results of Table 4 in the Experiment section.
Since Sofa dataset containing only 4 graphs in the training is too small for the proposed method to learn enough information, GMGCN does not perform well on the dataset.
Due to the large number of instances in [Table] each bag, miSVM requires an extremely large amount of memory for training, so we cannot show its results.
[Table]
C. GMGCN ON PHEME D ATA C.5. D ATA D ESCRIPTION
PHEME dataset 5 consists of ﬁve different news events, all of which attracted substantial attention in the social media and were rife with rumors.
The whole dataset is a collection of 5802 tweets, of which 1972 were deemed rumors and 3830 were deemed non-rumors.
These tweets are distributed differently across the ﬁve events, as shown in Table C7.
We also present the distribution of abnormal users and normal users across ﬁve events when M = 2 , 3 , 4 in Table C8.
The features adopted in
PHEME dataset are listed in Table C9.
C.5. D ETAILED R ESULTS
In PHEME datasets [Table] we label the users appearing in more than M rumor topics as abnormal users, and the rest as normal users.
In this part, we present the detailed results on PHEME datasets when
M = 2 , 3 , 4 .
As shown in Table C10 and C11, it is obvious that our proposed method can achieve the best effectiveness on PHEME datasets with different M values.
According to the clustering effectiveness, DBSCAN is selected as the representative of the feature clustering methods, GAE is selected as the representative of graph clustering methods and SIL is selected as the representative of the MIL methods to be displayed in the Experiment section
[Table] [Table]
[Table] [Table]
[Table] [Table]