B AYESIAN R OBUST G RAPH C ONTRASTIVE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Neural Networks (GNNs) have been widely used to learn node representations and with outstanding performance on various tasks such as node classification.
However, noise, which inevitably exists in real-world graph data, would considerably degrade the performance of GNNs revealed by recent studies.
In this work, we propose a novel and robust method, Bayesian Robust Graph Contrastive Learning (BRGCL), which trains a GNN encoder to learn robust node representations.
The
BRGCL encoder is a completely unsupervised encoder.
Two steps are iteratively executed at each epoch of training the BRGCL encoder: (1) estimating confident nodes and computing robust cluster prototypes of node representations through a novel Bayesian nonparametric method; (2) prototypical contrastive learning between the node representations and the robust cluster prototypes.
Experiments on public benchmarks demonstrate the superior performance of BRGCL and the robustness of the learned node representations.
The code of BRGCL is available at https://anonymous.4open.science/r/BRGCL-code-2FD9/ .
I NTRODUCTION
Graph Neural Networks (GNNs) have become popular tools for node representation learning in recent years (Kipf & Welling, 2017; Bruna et al., 2014; Hamilton et al., 2017; Xu et al., 2019).
Most prevailing GNNs (Kipf & Welling, 2017; Zhu & Koniusz, 2020) leverage the graph structure and obtain the representation of nodes in a graph by utilizing the features of their connected nodes.
Benefiting from such propagation mechanism, node representations obtained by GNN encoders have demonstrated superior performance on various downstream tasks such as semi-supervised node classification and node clustering.
Although GNNs have achieved great success in node representation learning, current GNN approaches do not consider the noise in the input graph.
However, noise inherently exists in the graph data for many real-world applications.
Such noise may be present in node attributes or node labels, which forms two types of noise, attribute noise and label noise.
Recent works, such as (Patrini et al., 2017), have evidenced that noisy inputs hurt the generalization capability of neural networks.
Moreover, noise in a subset of the graph data can easily propagate through the graph topology to corrupt the remaining nodes in the graph data.
Nodes that are corrupted by noise or falsely labeled would adversely affect the representation learning of themselves and their neighbors.
While manual data cleaning and labeling could be remedies to the consequence of noise, they are expensive processes and difficult to scale, thus not able to handle an2 almost infinite amount of noisy data online.
Therefore, it is crucial to design a robust GNN encoder that could make use of noisy training data while circumventing the adverse effect of noise.
In this paper, we propose a novel and robust method termed Bayesian Robust Graph Contrastive Learning (BRGCL) to improve the robustness of node representations for GNNs.
Our key observation is that there exist a subset of nodes which are confident in their class/cluster labels.
Usually, such confident nodes are far away from the class/cluster boundaries, so these confident nodes are trustworthy, and noise in these nodes would not degrade the value of these nodes in training a GNN encoder.
To infer such confident nodes, we propose a novel algorithm named Bayesian nonparametric Estimation of Confidence (BEC).
Since the BRGCL encoder is completely unsupervised, it first infers pseudo labels of all the nodes with a Bayesian nonparametric method only based on the input node attributes, without knowing the ground truth labels or the ground truth class number in the training data.
Then, BEC is used to estimate the confident nodes based on the pseudo labels and the graph structure.
The robust prototype representations, as the cluster centers of the confident nodes, are computed and used to train the BRGCL encoder with a loss function for prototypical contrastive learning.
The confident nodes are updated during each epoch of the training of the BRGCL encoder, so the robust prototype representations are also updated accordingly.
1. C ONTRIBUTIONS
Our contributions are as follows.
First, we propose Bayesian Robust Graph Contrastive Learning (BRGCL), where a fully unsupervised encoder is trained on noisy graph data.
The fully unsupervised BRGCL encoder is trained only on the input node attributes without ground truth labels or even the ground truth class number in the training data.
GRGCL leverages confident nodes, which are estimated by a novel algorithm termed
Bayesian nonparametric Estimation of Confidence (BEC), to harvest noisy graph data without being compromised by the noise.
Experimental results on popular graph datasets evidence the advantage of BRGCL over competing GNN methods for node classification and node clustering on noisy graph data.
The significance of the improvement of BRGCL is evidenced by p -values of t-test.
Second, our study reveals the importance of confident nodes in training GNN encoders on noisy graph data, which opens the door for future research in this direction.
The visualization results in Section 5.3 show that the confident nodes estimated by BEC are usually far away from the class/cluster boundaries, and so are the robust prototype representations.
As a result, the BRGCL encoder trained with such robust prototypes is not vulnerable to noise, and it even outperforms GNNs trained with ground truth labels.
The better spectrum of the Neural Tangent Kernel (Jacot et al., 2018) of BRGCL is also demonstrated against its baseline in Section 5.3, explaining BRGCL’s better generalization capability from a perspective of spectral analysis of neural networks.
R ELATED W ORKS
Graph Neural Networks.
Graph neural networks (GNNs) have become popular tools for node representation learning.
They have shown superior performance in various graph learning tasks, such as node classification, node clustering, and graph classification.
Given the difference in the convolution domain, current GNNs fall into two classes.
The first class features spectral convolution (Bruna et al., 2014; Kipf & Welling, 2017), and the second class (Hamilton et al., 2017;
Veli ˇ ckovi ´ c et al., 2017; Xu et al., 2019) generates node representations by sampling and propagating features from their neighborhood.
GNNs such as ChebNet (Bruna et al., 2014) perform convolution on the graph Fourier transforms termed spectral convolution.
Graph Convolutional Network (GCN) (Kipf & Welling, 2017) further simplifies the spectral convolution (Bruna et al., 2014) by its first-order approximation.
GNNs such as GraphSAGE (Hamilton et al., 2017) propose to learn a function that generates node representations by sampling and propagating features from a node’s connected neighborhood to itself.
Various designs of the propagation function have been proposed.
For instance,
Graph Attention Network (GAT) (Veli ˇ ckovi ´ c et al., 2017) proposes to learn masked self-attention layers that enable nodes to attend over their neighborhoods’ features.
Different from GNNs based on spectral convolution, such methods could be trained on mini-batches (Hamilton et al., 2017; Xu et al., 2019), so they are more scalable to large graphs.
However, as pointed out by (Dai et al., 2021), the performance of GNNs can be easily degraded by noisy training data (NT et al., 2019).
Moreover, the adverse effects of noise in a subset of nodes can be exaggerated by being propagated to the remaining nodes through the network structure, exacerbating the negative impact of noise.
Existing Methods Handing Noisy Data.
Previous works (Zhang et al., 2021) have shown that deep neural networks usually generalize badly when trained on input with noise.
Existing literature on robust learning with noisy inputs mostly focuses on image or text domain.
Such robust learning methods fall into two categories.
The first category (Patrini et al., 2017; Goldberger & Ben-Reuven, 2016) mitigates the effects of noisy inputs by correcting the computation of loss function, known as loss corruption.
The second category aims to select clean samples from noisy inputs for the training (Malach & Shalev-Shwartz, 2017; Jiang et al., 2018;
Yu et al., 2019; Li et al., 2020; Han et al., 2018), known as sample selection.
For example, (Goldberger & Ben-Reuven, 2016) corrects the predicted probabilities with a corruption matrix computed on a clean set of inputs.
On the other hand, recent sample selection methods usually select a subset of training data to perform robust learning.
[Figure]
Confidence
[Figure]
Among the existing loss correction and sample selection methods, Co-teaching (Han et al., 2018) is promising, which trains two deep neural networks and performs sample selection in a training batch by comparing predictions from the two networks.
However, such sample selection strategy does not generalize well in graph domain (Dai et al., 2021) due to the extraordinarily small size of labeled nodes.
More details are to be introduced in Section 4.2.
Self-Training (Li et al., 2018) finds nodes with the most confident pseudo labels, and it augments the labeled training data by incorporating confident nodes with their pseudo labels into the existing training data.
In addition to the above two categories of robust learning methods, recent studies (Kang et al., 2020; Zhong et al., 2021; Wang et al., 2021) show that decoupling the feature representation learning and the training of the classifier can also improve the robustness of the learned feature representation.
P ROBLEM S ETUP
An attributed graph consisting of N nodes is formally represented by G =
( V , E , X ) , where V = { v 1 , v 2 , . .
, v N } and E ⊆
V × V denote the set of nodes and edges respectively.
X ∈ R N × D are the node attributes, and the attributes of each node is in R d .
Let A ∈ { 0 , 1 } N × N be the adjacency matrix of graph G , with A ij =
if and only if ( v i , v j ) ∈ E .
˜
A =
A + I denotes the adjacency matrix for a graph with self-loops added.
˜ D denotes the diagonal degree matrix of ˜ A .
Let V L and V U denote the set of labeled nodes and unlabeled nodes, respectively.
Noise usually exists in the input node attributes or labels of real-world graphs, which degrades the quality of the node representation obtained by common GCL encoders and affects the performance of the classifier trained on such representations.
We aim to obtain node representations robust to noise in two cases, where noise is present in either the labels of V L or in the input node attributes X .
That is, we consider either noisy label or noisy input node attributes.
The goal of BRGCL is to learn a node representation H =
g
( X , A ) , such that the node representations { h i } Ni =1 are robust to noise in the above two cases, where g ( · ) is the BRGCL encoder.
In our work, we use a two-layer GCN as our encoder.
Thus g ( X , A )
=
σ ( ˆ A σ ( ˆ AXW (0) )
W (1) ) , where ˆ A = ˜
D − 1 / 2 ˜
A ˜ D − 1 / 2 .
W (0) and W (1) are the weight matrices.
σ is the activation function ReLu.
h i is the i -th row of H .
To evaluate the performance of the robust node representations by BRGCL, the node representations { h i } Ni =1 are used for the following two tasks.
(1) Semi-supervised node classification, where a classifier is trained on V L , and then the classifier predicts the labels of the remaining unlabeled nodes, V U .
(2) Node clustering, where K-means clustering is performed on the node representations { h i } Ni =1 to obtain node clusters.
Notations.
Throughout this paper, we use ∥ · ∥ 2 to denote the Euclidean norm of a vector and [ n ] to denote all the natural numbers between 1 and n inclusively.
B AYESIAN R OBUST G RAPH C ONTRASTIVE L EARNING
We propose Bayesian Robust Graph Contrastive Learning (BRGCL) in this section to improve the robustness of node representations.
First, we review the preliminaries of graph contrastive learning.
Next, we propose Bayesian nonparametric Estimation of Confidence (BEC) algorithm to estimate robust nodes and prototypes.
Then, we show details of node classification and node clustering.
At last, we propose a decoupled training pipeline of BRGCL for semi-supervised node classification.
Figure 1 illustrates the overall framework of our proposed BRGCL.
4. P RELIMINARY OF G RAPH C ONTRASTIVE L EARNING
The general node representation learning aims to train an encoder g ( · ) , which is a two-layer Graph
Convolution Neural Network (GCN) (Kipf & Welling, 2017), to generate discriminative node representations.
In our work, we adopt contrastive learning to train the BRGCL encoder g ( · ) .
To perform contrastive learning, two different views, denoted as G 1 =
( X 1 , A 1 ) and G 2 =
( X 2 , A 2 ) are generated by node dropping, edge perturbation, and attribute masking.
The representation of two generated views are denoted as H 1 =
g
( X 1 , A 1 ) and H 2 = g
( X 2 , A 2 ) , with h 1 i and
h 2 i being the i -th row of H 1 and H 2 , respectively.
It is preferred that the mutual information between H 1 and
H 2 is maximized.
For computational reason, its lower bound is usually used as the objective for contrastive learning.
We use InfoNCE (Li et al., 2021a) as our node-wise contrastive loss, that is, L node =
(cid:80) Ni =1 − log s ( h 1 i , h 2 i ) s ( h 1 i , h 2 i ) + (cid:80) Nj =
1 s ( h 1 i , h 2 j ) , where s ( h 1 i , h 2 i ) =
| ⟨ h 1 i , h 2 i ⟩ | ∥ h 1 i ∥ 2 ∥ h 2 i ∥ 2 is the cosine similarity between two node representations, h 1 i and h 2 i .
In addition to the node-wise contrastive learning, we also adopt prototypical contrastive learning (Li et al., 2021a) to capture semantic information in the node representations, which can be interpreted as maximizing the mutual information between node representation and a set of estimated cluster prototypes { c 1 , ..., c K } .
Here K is the number of cluster prototypes.
The loss function for prototypical contrastive learning is L proto = − 1 N [Equation] log
[Equation]
BRGCL aims to improve the robustness of node representations by prototypical contrastive learning.
Our key observation is that there exists a subset of nodes that are confident about their class/cluster labels because they are faraway from class/cluster boundaries.
We propose an effective method to infer such confident nodes.
Because the BRGCL encoder is completely unsupervised, it does not have access to the ground truth label or ground truth class/cluster number.
Therefore, our algorithm for selection of confident nodes is based on Bayesian non-parameter style inference, and the algorithm is termed Bayesian nonparametric Estimation of Confidence (BEC) to be introduced next.
4. B AYESIAN NONPARAMETRIC E STIMATION OF C ONFIDENCE (BEC)
The key idea of Bayesian nonparametric Estimation of Confidence (BEC) is to estimate robust nodes by the confidence of nodes in their labels.
Intuitively, nodes more confident in their labels are less likely to be adversely affected by noise.
Because BRGCL is unsupervised, pseudo labels are used as the labels for such estimation.
We propose Bayesian nonparametric Prototype Learning (BPL) to infer the pseudo labels of nodes.
BPL, as a Bayesian nonparametric algorithm, infers the cluster prototypes by the Dirichlet Process
Mixture Model (DPMM) under the assumption that the distribution of node representations is a mixture of Gaussians.
The Gaussians share the same fixed covariance matrix σ I , and each Gaussian is used to model a cluster.
The DPMM model is specified by
[Equation]
where G is a Gaussian distribution draw from the Dirichlet process DP ( G 0 , α ) , and α is the concentration parameter for DP ( G 0 , α ) .
ϕ
i is the mean of the Gaussian sampled for generating the node representation h
i .
G 0 is the prior over means of the Gaussians.
G 0 is set to a zero-mean Gaussian
N ( 0 , ρ I ) for ρ > 0 .
A collapsed Gibbs sampler (Resnik & Hardisty, 2010) is used to infer the components of the GMM with the DPMM.
The Gibbs sampler iteratively samples pseudo labels for the nodes given the means of the Gaussian components, and samples the means of the Gaussian components given the pseudo labels of the nodes.
Following (Kulis & Jordan, 2011), such a process is almost equivalent to K-means when σ , the variance of the Gaussians, goes to 0 .
The almost zero variance eliminates the need to estimate the variance σ , making the inference efficient.
Let ˜ K denote the number of inferred prototypes at the current iteration,
the pseudo label z i of node v i is then calculated by
[Equation]
where the Euclidean distance { d ik } is used to determine the pseudo labels of the node representation h i .
ξ is the margin to initialize a new prototype.
In practice, we choose the value of ξ by performing cross-validation on each dataset with details in Section A in the supplementary.
After obtaining the pseudo labels of nodes by BPL with K being the inferred number of prototypes, we estimate the confidence of the nodes based on their pseudo labels and the graph structure.
We first select nodes confident in their labels, also referred to as confident nodes, by considering the label information from the neighborhood of each node specified by the adjacency matrix.
Let z i denote the one-hot pseudo label of node v i estimated by the Bayesian method.
Label propagation (Zhang &
Chen, 2018) is applied based on the adjacency matrix to get a soft pseudo label for each node.
Let
Z ∈ R N × K be the matrix of pseudo labels with z
i being the i -th row of Z .
The label propagation runs the following update for T steps,
[Equation]
where T is the number of propagation steps, α is the teleport probability, which are set to the suggested values in (Zhang & Chen, 2018).
Let ˜ Z
= Z ( T ) be the soft labels obtained by the label propagation with ˜
z i being the i -th row of ˜ Z .
Following (Han et al., 2018), we use the cross-entropy between z i and ˜
z
i , denoted by ϕ
( z i , ˜ z i ) , to identify confident nodes.
Intuitively, smaller cross-entropy ϕ
( z i , ˜ z
i ) of a node v i leads to a larger probability of the pseudo label, so node v i is more confident about its pseudo label ˜
z
i .
As a result, we denote the set of confident nodes assigned to the k -th cluster as
[Equation]
where γ k is a threshold for the k -th class.
Figure 3 illustrates the cross-entropy values of all the nodes for the case that different levels of noise are present in the input node attributes, where the heat value indicates the corresponding cross-entropy value for every node.
The confident nodes with less cross-entropy values, which are marked in more red, are far away from cluster boundaries, so that noise on these nodes is more unlikely to affect their classification/clustering labels.
These confident nodes are the robust nodes leveraged by BRGCL to fight against noise.
The threshold γ k is dynamically set by
[Equation]
where t is the current epoch number and t m is a preset number of epochs.
In Co-teaching (Han et al., 2018), a similar threshold is used to select a ratio of data for training.
However, due to the limited size of training data in graph domain, training with only a subset of nodes usually leads to degraded performance.
For example, with 5% of nodes labeled on Cora dataset, only 1% of nodes will be used for training if the threshold is set to 20% by Co-teaching.
In contrast, BEC selects confident nodes by a dynamic threshold on the confidence of nodes in their labels given the labels from their neighbors.
The selected confident nodes are only used to obtain the robust prototype representations, and BRGCL is trained with such robust prototypes to obtain robust representations for all the nodes of the graph.
γ is an annealing factor.
In practice, the value of γ 0 is decided by cross-validation for each dataset, with details in Section A of the supplementary.
Previous methods such as (Li et al., 2021a) estimate each prototype as the mean of node representations assigned to that prototype.
After acquiring the confident nodes {T k } Kk =1 , the prototype representations are updated by c k = 1 |T k | (cid:80) h i ∈T k h
i for each k ∈
[ K ] .
With the updated cluster prototypes { c k } Kk =1 in the prototypical contrastive learning loss L proto , we train the encoder g ( · ) with the following overall loss function,
[Equation]
Training BRGCL with the loss function L rep does not require any information about the ground truth labels.
We summarize the training algorithm for the BRGCL encoder in Algorithm 1.
It is noted that confident nodes and robust prototypes are estimated at each epoch by BEC.
4. D ECOUPLED T RAINING
The typical pipeline for semi-supervised node classification is to jointly train the classifier and the encoder.
However, the noise in the training data would degrade the performance of the classifier.
To alleviate this issue, we decouple the representation learning for the nodes from the classification of nodes to mitigate the effect of noise, which consists of two steps.
In the first step, the BRGCL encoder g ( · ) is trained by Algorithm 1.
In the second step, with the node representation H from the trained BRGCL encoder, the classifier f ( · ) is trained by optimizing the loss function L cls .
In
Section B.5 of the supplementary, we show the advantage of such decoupled learning pipeline over the conventional joint training of encoder and classifier.
E XPERIMENTS
In this section, we evaluate the performance of BRGCL on five public benchmarks, with details deferred to Section A.1 of the supplementary.
For semi-supervised node classification, the performance of BRGCL is evaluated with noisy label or noisy input node attributes.
For node clustering, only noisy input node attributes are considered because there are no ground truth labels given for clustering purposes.
The implementation details about node classification are deferred to Section A. of the supplementary.
5. E XPERIMENTAL S ETTINGS
Due to the fact that most public benchmark graph datasets do not come with corrupted labels or attribute noise, we manually inject noise into public datasets to evaluate our algorithm.
We follow the commonly used label noise generation methods from the existing work (Han et al., 2020) to inject label noise.
We generate noisy labels over all classes according to a noise transition matrix
Q K × K , where Q ij is the probability of nodes from class i being flipped to class j .
We consider two types of noise: (1) Symmetric , where nodes from each class can be flipped to other classes with a uniform random probability,
s.t. Q ij =
Q ji ; (2) Asymmetric , where mislabeling only occurs between similar classes.
The percentage of nodes with flipped labels is defined as the label noise level in our experiments.
To evaluate the performance of our method with attribute noise, we randomly [Figure] shuffle a certain percentage of input attributes for each node following (Ding et al., 2022).
The percentage of shuffled attributes is defined as the attribute noise level in our experiments.
We compare BRGCL against semi-supervised node representation learning methods GCN (Kipf &
Welling, 2017), GCE (Zhang & Sabuncu, 2018), S 2 GC (Zhu & Koniusz, 2020), UnionNet (Li et al., 2021b), NRGNN (Dai et al., 2021).
In addition, we also compare BRGCL against state-of-the-art
GCL methods, including GraphCL (You et al., 2020), MVGRL (Hassani & Khasahmadi, 2020), MERIT (Jin et al., 2021), and SUGRL (Mo et al., 2022).
The training settings for different baselines are categorized into two setups: (1) Unsupervised Setup , where the training of the encoder does not use the ground truth label information.
The node representations obtained by the encoder are then used for downstream tasks, which are node classification and node clustering; (2) Supervised Setup , where the training of the encoder uses the ground truth label information.
Our proposed BRGCL follows the unsupervised setup in all our experiments, and every baseline follows its corresponding setup by its nature.
5. E VALUATION R ESULTS
Semi-supervised Node Classification with Label Noise.
We compare BRGCL against competing methods for semi-supervised node classification on input with two types of label noise.
To show the robustness of BRGCL against label noise, we perform the experiments on graphs injected with different levels of label noise ranging from 40% to 80% with a step of 20% .
The classification follows the widely used semi-supervised setting (Kipf & Welling, 2017).
Note the labels are only used for the training of the classifier.
The BRGCL encoder generates node representations, and the classifier for node classification is trained on these node representations.
In our experiment, a two-layer MLP whose hidden dimension is 128 is used as the classifier.
The results of different methods with respect to different levels of symmetric and asymmetric label noise on ogbn-arxiv are illustrated in Figure 2.
Detailed results on PubMed and ogbn-arxiv are shown in Table 1, where we report the means of the accuracy of 10 runs and the standard deviation. We also show the results on Cora, Citeseer, and Coauthor CS in Section B.1 in the supplementary.
It is observed from the results that BRGCL outperforms all the baselines, including the methods using ground truth labels to train their encoders.
By selecting confident nodes and computing robust prototypes using BEC, BRGCL outperforms all the baselines by an even larger margin with a larger label noise level.
To verify the statistical significance of improvements, we show the p -values of t-test between BRGCL and the second best baseline in Section B.2 in the supplementary.
The p -values for all datasets with all noise levels for both symmetric label noise and asymmetric label noise are less than 0.05, suggesting the statistically significant improvement of BRGCL over baseline methods.
Semi-supervised Node Classification with Attribute Noise.
We compare BRGCL with baselines for noisy input with attribute noise levels ranging from 40% to 80% with a step of 20% .
The results on ogbn-arxiv are illustrated in Figure 2.
Detailed results on PubMed, and ogbn-arxiv are shown in Table 1, where we report the means of the accuracy of 10 runs and the standard deviation. The results on Cora, Citeseer, and Coauthor CS are deferred to Section B.1 in the supplementary.
The results clearly show that BRGCL is more robust to attribute noise compared to all the baselines for different [Table] noise levels.
To verify the statistical significance of improvements, we show the p -values of t-test between BRGCL and the second best baseline in Section B.2 in the supplementary.
The p -values for all datasets with all levels of attribute noise are less than 0.05, suggesting the statistically significant improvement of BRGCL over baseline methods.
Node Clustering with Attribute Noise.
To further evaluate the robustness of node representation learned by BRGCL, we perform experiments on node clustering with attribute noise injected.
We follow the same evaluation protocol as that in (Hassani & Khasahmadi, 2020).
K-means is applied on the learned node representations to obtain clustering results.
We use accuracy (ACC) and normalized mutual information (NMI) as the performance metrics for clustering. The node clustering results for inputs with 60% attribute noise are shown in Table 2.
We report the averaged clustering results and standard deviations over 20 times of execution. Results on clean benchmark datasets are deferred to Section B.3 in the supplementary.
It is observed that node representation obtained by BRGCL is more robust to attribute noise for node clustering.
To show the statistical significance of improvements, we also calculate p -values of the t-test between BRGCL and the second best baseline for each result.
It is observed that BRGCL significantly improves the performance of node clustering as the p -values for ACC and NMI are less than 0 .
05 on all datasets.
[Table]
Comparison to Existing Sample Selection.
We also compare our BRGCL to the representative sample selection methods for node classification, including Co-teaching (Han et al., 2018), in
Section B.4 of the supplementary.
It is observed that BRGCL outperforms these competing methods by a noticeable margin.
5. V ISUALIZATION OF C ONFIDENCE S CORE AND S PECTRUM OF BRGCL
We visualize the confident nodes selected by BEC in the embedding space of the learned node representations in Figure 3.
The node representations are visualized by the t-SNE figure.
Each mark in t-SNE represents the representation of a node, and the color of the mark denotes the confidence of that node.
The results are shown for different levels of attribute noise.
It can be observed from
Figure 3 that confident nodes, which are redder in Figure 3, are well separated in the embedding space.
With a higher level of attribute noise, the bluer nodes from different clusters blended around the cluster boundaries.
In contrast, the redder nodes are still well separated and far away from cluster boundaries, which leads to more robustness and better performance in downstream tasks.
[Figure]
In order to understand why BRGCL generalizes better than prior SOTA on noisy data, we plot the eigenvalues of the sample Neural Tangent Kernel (Jacot et al., 2018) in Figure 4.
Previous studies such as (Rahaman et al., 2019) indicate that neural networks empirically generalize well if the target function lies in low-frequency directions, or the subspace spanned by eigenfunctions corresponding to high eigenvalues of NTK.
It can be observed that BRGCL usually has larger eigenvalues than the top baseline, MERIT (Jin et al., 2021), which explains its better generalization from the perspective of spectral analysis of neural networks.
[Figure]
C ONCLUSIONS
In this paper [Figure] we propose a novel node representation learning method termed Bayesian Robust Graph
Contrastive Learning (BRGCL) that aims to improve the robustness of node representations by a novel Bayesian nonparametric algorithm, Bayesian nonparametric Estimation of Confidence (BEC).
We evaluate the performance of BRGCL with comparison to competing baselines on semi-supervised node classification and node clustering, where graph data are corrupted with noise in either the labels for the node attributes.
Experimental results demonstrate that BRGCL generates more robust node representations with better performance than the current state-of-the-art node representation learning methods.
A I MPLEMENTATION D ETAILS A. D ATASETS
We evaluate BRGCL on five public benchmarks that are widely used for node representation learning, namely Cora, Citeseer, PubMed (Sen et al., 2008), Coauthor CS, and ogbn-arxiv (Hu et al., 2020).
Cora, Citeseer and PubMed are three most widely used citation networks.
Coauthor CS is coauthorship graph.
The ogan-arxiv is a directed citation graph.
We summarize the statistics of the datasets in Table 3.
Among the five benchmarks, ogbn-arxiv is known for its larger scale, and is more challenging to deal with.
For all our experiments, we follow the default separation of training, validation, and test sets on each benchmark.
[Table]
A. M ORE D ETAILS ABOUT N ODE C LASSIFICATION
The robust node representations are used to perform node classification and node clustering mentioned in Section 3 of the main paper.
More details about node classification are introduced in this subsection.
As the connected neighbors in a graph usually show similar semantic information, we generate soft labels of nodes via label propagation on the graph to take the advantage of the information from the neighborhood.
The classifier for node classification is trained with soft labels instead of hard labels.
First, we define the one-hot hard label matrix Y ∈ R N × K , where Y ij = 1 if
and only if node v i is in class j for i ∈
[ N ] and j ∈ [ K ] .
If a node v i ∈ V L , then Y ij = 1 if the ground truth label of v i is j .
If v i / ∈ V L , we initialize Y ij = 0 for all j ∈ [ K ] .
Then the soft labels of all the nodes are generated by graph label propagation.
Similar to (3), after T aggregation steps of label propagation, we have Y ( t +1) =
(1 − β ) ˜
AY ( t ) + β Y (0) , t = 1 , ..., T − 1 , where Y (0) =
Y , β is the teleport probability.
The soft labels are then obtained by ˜ Y = softmax ( Y ( T ) ) .
We denote the i -th row of ˜ Y by ˜
y i , which is the soft label of node v i .
f ( · ) is a classifier built by a two-layer MLP followed by a softmax function, which is trained by minimizing the standard loss function for classification,
L cls = 1 |V L | (cid:80) v i ∈V L H (˜ y
i , f ( h i )) , where H is the cross-entropy function.
A. T UNING H YPER -P ARAMETERS BY C ROSS -V ALIDATION
In this section, we show the tuning procedures on the hyper-parameters ξ from Equation (2) and γfrom Equation (5).
We perform cross-validations on 20% of training data to decide the value of ξ and γ 0 .
The value of ξ is selected from { 0 . 1 , 0 . 15 , 0 . 2 , 0 . 25 , 0 . 3 , 0 . 35 , 0 . 4 , 0 . 45 , 0 , 5 } .
The value of γis selected from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
The selected values for ξ and γ 0 on each dataset are shown in Table 4.
[Table]
B A DDITIONAL E XPERIMENT R ESULTS B. A DDITIONAL N ODE C LASSIFICATION R ESULTS WITH L ABEL N OISE AND A TTRIBUTE N OISE .
The results for node classification with symmetric label noise and asymmetric label noise on Cora,
Citeseer
[Table]
and Coauthor CS are shown in Table 5.
In addition, the results for node classification with attribute noise on Cora, Citeseer, and Coauthor CS are shown in Table 6.
It is observed that BRGCL outperforms all the baselines for node classification with both label noise and attribute noise on all three benchmark datasets.
[Table] [Table] B. S TATISTICAL SIGNIFICANCE OF IMPROVEMENTS FOR NODE CLASSIFICATION WITH LABEL NOISE .
To validate the statistical significance of the improvements of BRGCL over competing methods [Table] we further calculate the p -values of t-test between BRGCL and the second best baseline for each noise level and dataset for symmetric label noise, asymmetric label noise [Table]
and attribute label noise.
The p -values for node classification with label noise are shown in Table 7.
The p -values for node classification with attribute noise are shown in Table 8.
We run all the experiments for 10 times with random initialization and injected symmetric and asymmetric label noise.
The p -values for all datasets with all noise levels are less than 0.05, suggesting the statistically significant improvement of BRGCL over baseline methods.
[Table]
[Table]
B. N ODE C LUSTERING WITH I NPUT A TTRIBUTE N OISE
To further validate the performance of the node representation learned by BRGCL [Table] we perform node clustering on clean benchmark datasets.
We follow the same evaluation protocol as that in Section 5.2.
K-means is then applied to the learned node representations to obtain the clustering results.
It can be observed from Table 9 that BRGCL still outperforms all baseline methods for node clustering [Table] [Table]
B. C OMPARISONS TO E XISTING S AMPLE S ELECTION M ETHODS
In this subsection
[Table]
we compare BRGCL against previous sample selection methods, including Coteaching (Han et al., 2018) and Self-Training (Li et al., 2018) for node classification with symmetric label noise.
Co-teaching maintains two networks to select clean samples for each other.
Self-Training finds nodes with the most confident pseudo labels, and it augmented the labeled training data by incorporating confident nodes with their pseudo labels into the existing training data.
The results are shown in Table 10.
We can clearly see that BRGCL greatly outperforms competing sample selection methods.
[Table]
B. J OINT T RAINING VS . D ECOUPLED T RAINING .
We study the effectiveness of our decoupled training framework compared with jointly training the encoder and the classifier.
We compare the performance on node classification with 20% label noise level.
The results are shown in Table 11.
It can be observed that decoupling the training of classifier and encoder can mitigate the effects of label noise.
[Table]
N UMBER OF C ONFIDENT P ROTOTYPES .
To further study the behavior of BRGCL
[Table]
we show the number of robust prototypes estimated by
BPL in Table 12.
It can be observed from the results that the estimated number of robust prototypes is usually very close to the ground truth number of classes for different datasets, justifying the effectiveness of BPL.
Because BEC is based on the pseudo labels estimated by BPL, the success of
BPL leads to trustworthy estimation of confident nodes and robust prototypes by BEC.
[Table]