T OWARDS R ELIABLE L INK P REDICTION WITH R OBUST G RAPH I NFORMATION B OTTLENECK
Anonymous authors
Paper under double-blind review
A BSTRACT
Link prediction on graphs has achieved great success with the rise of deep graph learning.
However, the potential robustness under the edge noise is less investigated.
We reveal that the inherent edge noise that naturally perturbs both input topology and target label leads to severe performance degradation and representation collapse.
In this work, we propose an information-theory-guided principle, Robust Graph
Information Bottleneck (RGIB), to extract reliable supervision signals and avoid representation collapse.
Different from the general information bottleneck, RGIB decouples and balances the mutual dependence among graph topology, target labels, and representation, building new learning objectives toward robust representation.
We also provide two instantiations, RGIB-SSL and RGIB-REP, which beneﬁt from different methodologies, i.e., self-supervised learning and data reparametrization, for implicit and explicit data denoising, respectively.
Extensive experiments onbenchmarks of various scenarios verify the effectiveness of the proposed RGIB.
I NTRODUCTION
As a fundamental problem in graph learning, link prediction (Liben-Nowell & Kleinberg, 2007) has attracted growing interest in real-world applications like drug discovery (Ioannidis et al., 2020), knowledge graph completion (Bordes et al., 2013), and question answering (Huang et al., 2019).
Recent advances from heuristic designs (Katz, 1953; Page et al., 1999) to graph neural networks (GNNs) (Kipf & Welling, 2016a; Gilmer et al., 2017; Kipf & Welling, 2016b; Zhang & Chen, 2018; Zhu et al., 2021) have achieved superior performances.
Nevertheless, the poor robustness in imperfect scenarios with the inherent edge noise is still a practical bottleneck to the current deep graph models (Gallagher et al., 2008; Ferrara et al., 2016; Wu et al., 2022a; Dai et al., 2022).
Early explorations improve the robustness of GNNs for node classiﬁcation under label noise (Dai et al., 2021;
Li et al., 2021) through the smoothing effect of neighboring nodes.
Other methods achieve a similar goal via randomly removing edges (Rong et al., 2020) or actively selecting the informative nodes or edges and pruning the task-irrelevant ones (Zheng et al., 2020; Luo et al., 2021).
However, when applying these noise-robust methods to the link prediction with noise, only marginal improvements are achieved (see Section 5).
The attribution is that the edge noise can naturally deteriorate both the input topology and the target labels (Figure 1(a)).
Previous works that consider the noise either in input space or label space cannot effectively deal with such a coupled scenario.
Therefore, it raises a new challenge to understand and tackle the edge noise for robust link prediction.
In this paper, we dive into the inherent edge noise and empirically show the signiﬁcantly degraded performances it leads to (Section 3.1).
Then, we reveal the negative effect of the edge noise through carefully inspecting the distribution of learned representations, and discover that graph representation is severely collapsed, which is reﬂected by much lower alignment and poorer uniformity (Section 3.2).
To solve this challenging problem, we propose the Robust Graph Information Bottleneck (RGIB) principle based on the basic GIB for adversarial robustness (Wu et al., 2020) (Section 4.1).
Conceptually, the RGIB principle is with new learning objectives that decouple the mutual information (MI) among noisy inputs ˜ A , noisy labels ˜ Y , and the representation H .
As illustrated in Figure 1(b),
RGIB generalizes the basic GIB to learn a robust representation that is resistant to the edge noise.
Technically, we provide two instantiations of RGIB based on different methodologies, i.e., RGIBSSL and RGIB-REP: (1) the former utilizes contrastive pairs with automatically augmented views to form the informative regularization in a self-supervised learning manner (Section 4.2); and (2) the latter explicitly puriﬁes the graph topology and supervision targets with the reparameterization mechanism (Section 4.3).
Both instantiations are equipped with adaptive designs, aiming to effectively estimate and balance the corresponding informative terms in a tractable manner.
For example, the hybrid augmentation algorithm and self-adversarial alignment loss for RGIB-SSL, and the relaxed information constraints on topology space as well as label space for RGIB-REP.
Empirically, we show that these two instantiations work effectively under extensive noisy scenarios and can be seamlessly integrated with various existing GNNs (Section 5).
Our main contributions are summarized as follows
• To our best knowledge, we are the ﬁrst to study the robustness problem of link prediction under the inherent edge noise.
We reveal that the inherent noise can bring a severe representation collapse and performance degradation, and such negative impacts are general to common datasets and GNNs.
• We propose a general learning framework, RGIB, with reﬁned representation learning objectives to promote the robustness of GNNs.
Two instantiations, RGIB-SSL and RGIB-REP, are proposed upon different methodologies that are equipped with adaptive designs and theoretical guarantees.
• Without modifying the GNN architectures, the RGIB
achieves state-of-the-art results on 3 GNNs and 6 datasets under various noisy scenarios, obtaining up to 12 .
9% AUC promotion.
The distribution of learned representations is notably recovered and more robust to the inherent noise.
P RELIMINARIES
Notation.
We denote V
= { v i } Ni =1 as the set of nodes and E = { e ij } Mij =1 as the set of edges.
With adjacent matrix A and node features X , an undirected graph is denoted as G =( A, X ) , where A ij = means there is an edge e ij between v i and v j . X
[ i, :] ∈
R D is the D -dimension node feature of v i .
Link prediction is to indicate the existence of query edges with labels Y that are not observed in A .
GNNs for Link Prediction.
We follow the common link prediction framework, i.e., graph autoencoders (Kipf & Welling, 2016b), where the GNN architecture can be GCN (Kipf & Welling, 2016a),
GAT (Veliˇckovi´c et al., 2018), or SAGE (Hamilton et al., 2017).
Given a L -layer GNN, the graph representations H ∈ R |V|× D (cid:48) for each node v i ∈ V are obtained by a L -layer message propagation as the encoding process.
For decoding, logits φ e ij of each query edge e ij are computed with a readout function, e.g., the dot product φ e ij = h (cid:62) i h j .
Finally, the optimization objective is to minimize the binary classiﬁcation loss, i.e., min L cls =
(cid:80) e ij ∈E train − y ij log (cid:0) σ ( φ e ij ) (cid:1) − (1 − y ij ) log (cid:0) 1 − σ ( φ e ij ) (cid:1) , where σ ( · ) is the sigmoid function, and y ij =1 for positive edges while y ij =0 for negative ones.
Topological denoising approaches.
A natural way to tackle the input edge noise is to directly clean the noisy graph.
Sampling-based methods, such as DropEdge (Rong et al., 2020), NeuralSparse (Zheng et al., 2020), and PTDNet (Luo et al., 2021), are proposed to remove the taskirrelevant edges.
Besides, as GNNs can be easily fooled by adversarial network with only a few perturbed edges (Chen et al., 2018; Zhu et al., 2019; Entezari et al., 2020), defending methods like
GCN-jaccard (Wu et al., 2019) and GIB (Wu et al., 2020) are designed for pruning adversarial edges.
Label-noise-resistant techniques.
For tackling the general problem of noisy label, Co-teaching (Han et al., 2018) lets two neural networks teach each other with small-loss samples based on the memorization effect (Arpit et al., 2017).
Besides, peer loss function (Liu & Guo, 2020) pairs independent peer examples for supervision and works within the standard empirical risk minimization framework.
As for the graph domain, label propagation techniques proposed by pioneer works (Dai et al., 2021;
Li et al., 2021) propagate the reliable signals from clean nodes to noisy ones, which are nonetheless entangled with the node annotations and node classiﬁcation task that cannot be directly applied here.
A N EMPIRICAL STUDY OF THE INHERENT EDGE NOISE
In this section, we attempt to ﬁgure out how GNNs behave when learning with the edge noise and what are the latent mechanisms behind it.
We ﬁrst present an empirical study in Section 3.1 and then investigate the negative impact of noise through the lens of representation distribution in Section 3.2.
3. H OW DO GNN S PERFORM UNDER THE INHERENT EDGE NOISE ?
Since existing benchmarks are usually well-annotated and clean, there is a need to simulate the inherent edge noise properly to investigate the impact of noise.
Note the data split manner adopt by most relevant works (Kipf & Welling, 2016b; Zhang & Chen, 2018; Zhu et al., 2021) randomly divides partial edges as observations and the others as prediction targets.
The inherent edge noise, if exists, should be false positive samples and uniformly distributed to both input observations and output labels.
Thus, the training data can be with noisy adjacence ˜ A and noisy labels ˜
Y , i.e., the input noise and label noise.
We elaborate the formal simulation of such an additive edge noise as follows.
Deﬁnition 3.1 (Additive edge noise) .
Given a clean training data, i.e., observed graph G =
( A, X ) and labels Y ∈ { 0 , 1 } of query edges, the noisy adjacence ˜
A is generated by only adding edges to the original adjacent matrix A while keeping the node features X unchanged.
The noisy labels ˜
Y are generated by only adding false-positive edges to the labels Y .
Speciﬁcally, given a noise ratio ε a , the added noisy edges A (cid:48) ( ˜ A =
A + A (cid:48) ) are randomly generated with the zero elements in A as candidates.
It satisﬁes that A (cid:48) (cid:12) A
= O and ε
a = | nonzero ( ˜ A ) |−| nonzero ( A ) | / | nonzero ( A ) | .
Similarly, noisy labels are generated and added to original labels, where ε y =
| nonzero (˜ Y )
|−| nonzero ( Y ) | / | nonzero ( Y ) | .
[Figure]
With the simulated noise, an empirical study is then performed with various GNNs and datasets.
As shown in Figure 2, the edge noise causes a signiﬁcant drop in performance, and a larger noise ratio generally leads to greater degradation.
It means that these common
GNNs normally trained with the stochastic gradient descent are vulnerable to the inherent edge noise, yielding a severe robustness problem.
However, none of the existing defending methods are designed for such a coupled noise, which is practical for real-world graph data that can be extremely noisy (Gallagher et al., 2008; Wu et al., 2022a).
Thus, it is urgent to understand the noise effect and devise the corresponding robust method.
3. U NDERSTANDING THE I MPACT OF N OISE V IA R EPRESENTATION D ISTRIBUTION
Denote a GNN as f w ( · ) with learnable weights w .
Node representations are extracted by a forward pass as H = f w ( A, X ) while the backward propagation with stochastic gradient descent optimizes the GNN as by contrast When encountering noise within ˜ A and ˜ Y , the representation H can be directly inﬂuenced, since the training neglects the adverse effect of data corruption.
Besides, the
GNN readouts the edge logit φ e ij based on top-layer node representations h i and h j , which are possibly degraded or even collapsed under the edge noise (Graf et al., 2021; Nguyen et al., 2022).
For quantitive and qualitative analysis of H under edge noise, we introduce two concepts (Wang & Isola, 2020) here, i.e., alignment and uniformity .
Speciﬁcally, alignment is computed as the distance of representations between two randomly augmented graphs. It quantiﬁes the stability of GNN when encountering edge noise in the testing phase. A higher alignment means the GNN is more resistant and invariant to input perturbations.
Uniformity, on the other hand, qualitatively measures the denoising effects of GNN when learning with edge noise in the training phase.
A greater uniformity implies that the learned representations of various samples are more uniformly distributed on the unit hypersphere, preserving as much information about the original data as possible.
As shown in Table 1 and Figure 3, a poorer alignment and a worse uniformity are brought by a severer edge noise.
The learned GNN f w ( · ) is more sensitive to input perturbations as the alignment values are sharply increased, and the learned edge representations tend to be less uniformly distributed and gradually collapse to individual points as the noise ratio increases.
Thereby, it is discovered that the graph representation is severely collapsed under the inherent edge noise, resulting in the nondiscriminative property.
This is usually undesirable and potentially risky for downstream applications.
{+two ran domly perturbed graphs i.e., 1 N (cid:80) Ni = || H i 
+}
− H {+i 
+}
|| 2 . where H i 1 = f w ( A i 1 , X ) and H i 2 = f w ( A i 2 , X ) .
Cora Citeseer clean .
ε =20% .
ε =40% .
ε =60% .
[-32 64 96     positive samples 32 64 
-]
[Table]
[Table]
[Table]
positive samples 32 64 96     negative samples (c) ε = 40% 32 64 96     positive samples 32 [-
-]64 96     negative samples (d) ε
= 60%
O N TACKLING THE REPRESENTATION COLLAPSE VIA R OBUST GIB
As aforementioned, the inherent edge noise brings unique challenges to link-level graph learning.
Without prior knowledge like the noise ratio, without the assistance of auxiliary datasets, and even without modifying GNN architectures, how can learned representations be resistant to the inherent noise?
Here, we formally build a method to address this problem from the perspective of graph information bottleneck (Section 4.1) and design its two practical instantiations (Section 4.2 and 4.3).
4. T HE P RINCIPLE OF R OBUST GIB
Recall in Section 3.2, the graph representation H is severely degraded due to incorporating noisy signals in ˜ A and ˜ Y .
To robustify H , one can naturally utilize the information constraint based on the basic graph information bottleneck (GIB) principle (Wu et al., 2020; Yu et al., 2020), i.e., solving
[Equation]
(1) where the hyper-parameter γ constrains the MI
I ( H ; ˜ A ) to avoid H from capturing excess taskirrelevant features from ˜ A .
The basic GIB (Eqn. 1) can effectively withstand the input perturbation (Wu et al., 2020).
However, it is intrinsically susceptive to label noise since it entirely preserves the label supervision with maximizing I ( H ; ˜ Y ) .
Empirical results in Section 5 show that it becomes ineffective when learning with inherent edge noise simulated as Deﬁnition 3.1.
The basic GIB decreases I ( H ; ˜
A | ˜ Y ) by directly constraining I ( H ; ˜ A ) to handle the input noise, as illustrated in Figure 1(b).
Symmetrically, the label noise can be hidden in I ( H ; ˜ Y | ˜ A ) , but trivially constraining I ( H ; ˜ Y ) to regularize I ( H ; ˜ Y | ˜ A ) is not ideal, since it will conﬂict with Eqn. 1 and also cannot tackle the noise within I ( ˜ A ; ˜ Y ) .
Thus, it is crucial to further decouple the dependences among ˜ A , ˜ Y , and H , while the noise can exist in areas of I ( H ; ˜ Y |
˜ A ) , I ( H ; ˜ A | ˜ Y ) , and I ( ˜ A ; ˜ Y | H ) .
[Figure]
Analytically, I ( ˜ A ; ˜ Y [Equation] H )
=
I
[Equation] A [Equation] ˜ Y )
+ I
( H [Equation] ˜ Y |
[Equation] A )
+ I ( H ; ˜
A | ˜ Y )
− H
( H )
+ H
( H | ˜ A, ˜ Y ) , where I ( ˜ A ; ˜ Y ) is a constant and redundancy H ( H | ˜ A, ˜ Y ) can be easily minimized.
Thus, the I ( ˜ A ; ˜ Y | H ) can be approximated by the other three terms, i.e., H ( H ) , I ( H ; ˜ Y |
˜ A ) and I ( H ; ˜
A | ˜ Y ) .
Since the two later terms are also with noise, a balance of these three informative terms can be a solution to the problem.
Based on the above analysis, we propose the Robust Graph Information Bottleneck (RGIB) , a new learning objective to balance informative signals regarding H , as illustrated in Figure 4(a), i.e., [Equation]
[Equation]
where constraints on H ( H ) encourage a diverse H to prevent representation collapse ( >γ − H ) and also limit its capacity ( <γ + H ) to avoid over-ﬁtting.
Another two symmetric terms, I ( H ; ˜ Y | ˜ A ) and
I ( H ; ˜ A | ˜ Y ) , mutually regularize posteriors to mitigate the negative impact of inherent noise on H .
Note that MI terms like I ( H ; ˜
A | ˜ Y ) are usually intractable.
Therefore, we introduce two practical implementations of RGIB, i.e., RGIB-SSL and RGIB-REP, based on different methodologies.
The former explicitly optimizes the representation H with the self-supervised regularization, while the latter implicitly optimizes H by purifying the noisy ˜ A and ˜ Y with the reparameterization mechanism.
4. I NSTANTIATING R OBUST GIB VIA S ELF -S UPERVISED L EARNING
Recall that the graph representation is deteriorated with the supervised learning paradigm.
Naturally, we modify it into a self-supervised counterpart by explicitly regularizing the representation H (shown in Figure 4(b)) to avoid collapse and to implicitly capture reliable relations among noisy edges, i.e.,
[Equation]
where margins λ s , λ u , λ a balance one supervised and two self-supervised regularization terms. When λ s ≡ 1 and λ u ≡ 0 , the RGIB-SSL can be degenerated to the basic GIB.
Note that contrastive learning is a prevalent technique in the self-supervised area to learn robust representation (Chen et al., 2020), where contrasting pair of samples with data augmentation plays an essential role.
In practice, we follow such a manner and calculate the supervision term by E [ I ( H ; ˜ Y | ˜ A )] ≤
E
[ I ( H ; ˜ Y )]= E ˜ A s ∼ P ( ˜ A )
[ I ( H s ; ˜ Y )]
≈ 1 / 2 ( I ( H 1 ; ˜ Y )
+ I
( H 2 ; ˜ Y ))
= 1 / 2 ( L cls ( H 1 ; ˜ Y )
+ L cls ( H 2 ; ˜ Y )) , where
˜ A 1 , ˜ A 2 are sampled from P ( ˜ A ) .
Similarly, we have E [ H ( H )]
≈ 1 / 2 ( H ( H 1 )
+ H ( H 2 )) , and a higher uniformity leads to a high entropy H ( H ) as proved in Proposition 4.1.
As for I ( H ; ˜
A | ˜ Y ) , we approximate it by the alignment term
I ( H 1 ; H 2 ) with the derived upper bound in Proposition 4.2.
Proposition 4.1.
A higher entropy H ( H ) leads to a higher uniformity.
Proof.
See Appendix C.1.
Proposition 4.2.
The alignment I
( H 1 ; H 2 ) is the upper bound of I ( H ; ˜
A | ˜ Y ) .
Since I ( H ; ˜ A | ˜ Y ) ≤
I ( H ; ˜ A ) ≤ 1 / 2 (cid:0) I ( H 1 ; H 2 )
+ I
( ˜ A 1 ;
˜ A 2 ) (cid:1) = 1 / 2 (cid:0) I ( H 1 ; H 2 )+ c (cid:1) , a constrained alignment estimated by MI I
( H 1 ; H 2 ) is capable to bound a lower I ( H ; ˜
A | ˜ Y ) and I ( H ;
˜ A ) .
Proof.
See Appendix C.2.
However, directly applying existing contrastive methods like (Chen et al., 2020; Khosla et al., 2020) can be suboptimal, since they are not originally designed for graph data and neglect the internal correlation of topology
˜ A and target ˜ Y .
Two following designs are proposed to avoid trivial solutions.
Hybrid graph augmentation.
To encourage more diverse views with lower I ( A 1 ; A 2 ) and to avoid manual selection of augmentation operations, we propose a hybrid augmentation method with four augmentation operations as predeﬁned candidates and ranges of their corresponding hyper-parameters.
In each training iteration, two augmentation operators, T 1 ( · ) and T 2 ( · ) , and their hyper-parameters θ
and θ 2 are automatically sampled from the search space.
Then, two augmented graphs are obtained by applying the two operators on the original graph G .
Namely, G 1 = T 1 ( G| θ 1 ) and G 2 =
T 2 ( G| θ 2 ) .
Self-adversarial alignment & uniformity loss.
With representations H 1 and H 2 from two augmented views, we build the alignment objective by minimizing the representation similarity of the positive pairs ( h 1 ij , h 2 ij ) and maximizing that of the randomly-sampled negative pairs ( h 1 ij , h 2 mn ) , e ij (cid:54) = e mn .
The proposed self-adversarial alignment loss is R align
= (cid:80) Ni =1 R posi + R negi , where
R posi = p pos ( h 1 ij , h 2 ij ) · (cid:13)(cid:13) h 1 ij − h 2 ij (cid:13)(cid:13) 2 2 and R negi =
p neg ( h 1 ij , h 2 mn ) · ( γ − (cid:13)(cid:13)
h 1 ij − h
mn (cid:13)(cid:13) 2 2 ) .
Importantly, softmax functions p pos ( · ) and p neg ( · ) aim to mitigate the inefﬁciency problem (Sun et al., 2019) that aligned pairs are not informative.
Besides, with Gaussian potential kernel, the uniformity loss is as R unif =
(cid:80) Kij,mn e − (cid:107)
h 1 ij − h 1 mn (cid:107) 2
+ e − (cid:107) h 2 ij − h 2 mn (cid:107) 2 2 , where e ij ,e mn are randomly sampled.
Optimization.
As Eqn. 3, the overall loss function of RGIB-SSL is L = λ
s L cls + λ a R align + λ u R unif .
Remark 4.1.
The inherent noise leads to class collapse, i.e., samples from the same class have the same representation.
It comes from trivially minimizing the noisy supervision
I ( H ; ˜ Y ) and results in degraded representations shown in Section 3.2.
Fortunately, the alignment and uniformity terms regularizing representations can alleviate such noise effects and avoid collapse (see Section 5).
4. I NSTANTIATING R OBUST GIB VIA D ATA R EPARAMETERIZATION
Another realization is by reparameterizing the graph data on both topology space and label space jointly to preserve clean information and discard noise.
We propose RGIB-REP that explicitly models the reliability of ˜ A and ˜ Y via latent variables Z to learn a noise-resistant H (as Figure 4(c)), namely,
[Equation]
where latent variables Z Y and Z A are clean signals extracted from noisy ˜ Y and ˜ A .
Their complementary parts Z Y (cid:48) and Z A (cid:48) are considered as noise, satisfying ˜ Y
=
Z Y + Z Y (cid:48) and ˜ A =
Z A + Z A (cid:48) .
When
Z Y ≡ ˜ Y and Z A ≡ ˜ A , the RGIB-REP can be degenerated to the basic GIB. Here, the I ( H ; Z Y ) measures the supervised signals with selected samples Z Y , where the classiﬁer takes Z A (a subgraph of ˜ A ) as input instead of the original ˜ A , i.e., H = f w ( Z A , X ) .
Constraints I
( Z A ; ˜ A ) and I ( Z Y ; ˜ Y ) aim to select the cleanest and most task-relevant information from ˜ A and ˜ Y .
Instantiation.
For deriving a tractable objective regarding Z A and Z Y , a parameterized sampler f φ ( · ) sharing the same architecture and weights as f w ( · ) is adopted here.
f φ ( · ) generates the probabilistic distribution of edges that include both ˜ A and ˜ Y by P = σ ( H φ H (cid:62) φ ) ∈ (0 , 1) |V|×|V| , where hidden representations H φ = f φ ( ˜ A, X ) .
Then, the Bernoulli sampling is used to obtain edges of high conﬁdences, i.e., Z A =
SAMP ( P | ˜ A ) , Z Y =
SAMP ( P | ˜ Y ) where | Z A |≤| ˜
A | and | Z Y |≤| ˜ Y | .
Optimization.
A relaxation is then conducted on the three MI terms in Eqn. 4.
With derived bounds in Proposition 4.3 (i.e., R A and R Y ) and Proposition 4.4 (i.e., L cls ), the ﬁnal objective is formed as
L = λ
s L cls + λ A R A
+ λ Y R Y , and the corresponding guarantee is discussed in Theorem 4.5.
Proposition 4.3.
Given the edge number n of ˜ A , the marginal distribution of Z A is Q ( Z A ) = (cid:80) n P φ ( P | n ) P ( ˜ A = n ) =
P ( n ) (cid:81) n ˜
A ij =1 P ij .
Then, we derive the upper bound
I ( Z A ; ˜ A ) ≤
E [ KL ( P φ ( Z A | A ) || Q
( Z A ))]
= (cid:80) e
ij ∈ ˜
A P ij log P ij τ +(1 − P ij ) log 1 − P ij 1 − τ =
R A , where τ is a constant.
Similarly, the label constraint is bounded as I ( Z Y ; ˜ Y ) ≤ R Y , Proof.
See Appendix C.3.
Proposition 4.4.
Since I ( H ; Z Y ) ≥ E Z Y , Z A [log P w ( Z Y | Z A )]
≈ −L cls ( f w ( Z A ) , Z Y ) , the supervision term in Eqn.
4 can be empirically reduced to the classiﬁcation loss.
Proof.
See Appendix C.4.
Theorem 4.5.
Assume the noisy training data D train =
( ˜ A, X, ˜ Y ) contains a potentially clean subset D sub =( Z ∗ A , X, Z ∗ Y ) and Z ∗ Y ≈ Y , based on which a trained GNN predictor f w ( · ) satisﬁes f w ( Z ∗ A , X )
=
Z ∗ Y + (cid:15) .
The error (cid:15) is independent from D sub and (cid:15) → 0 .
Then, for any λ s , λ A , λ Y ∈ [0 , 1] , Z A = Z ∗ A and Z Y =
Z ∗ Y minimizes the RGIB-REP (Eqn. 4). Proof. See Appendix C.5. Speciﬁcally
[Equation] [Equation] [Equation] [Equation]
[Table] [Table]
E XPERIMENTS
Setup.
In this section, we empirically verify the effectiveness of the proposed RGIB framework.popular datasets and 3 GNNs are taken in the experiments.
The inherent edge noise is generated based on Deﬁnition.
3.1 after the commonly used data split where 85% edges are randomly selected for training, 5% as the validation set
[Table]
and 10% for testing.
The AUC is used as the evaluation metric as in (Zhang & Chen, 2018; Zhu et al., 2021).
The software framework is the Pytorch (Paszke et al., 2017)
[Table]
while the hardware platform is a single NVIDIA RTX 3090 GPU.
We repeat all the experiments 10 times to obtain evaluation results with mean values and standard deviations.
Baselines.
As existing robust methods separately deal with input noise or label noise, both kinds of methods should be considered as baselines.
For input noise, three sampling-based approaches are used for comparison, i.e., DropEdge (Rong et al., 2020), NeuralSparse (Zheng et al., 2020), and PTDNet (Luo et al., 2021).
Jaccard (Wu et al., 2019), GIB (Wu et al., 2020), VIB (Sun et al., 2022), and PRI (Yu et al., 2022), which are designed for pruning adversarial edges, are also included.
Two generic methods are selected for label noise, i.e., Co-teaching (Han et al., 2018) and Peer loss (Liu & Guo, 2020).
In addition, two contrastive learning methods are taken into comparison with adaptation to the link prediction task, including SupCon (Khosla et al., 2020) utilizing the full labels for supervision and GRACE (Zhu et al., 2020) that are optimized in a self-supervised manner without labels.
All the above baseline methods are evaluated w.r.t.
their original implementations
5. M AIN RESULTS
Performance comparison.
As shown in Table 2, the RGIB achieves the best results in all 6 datasets under the inherent edge noise with various noise ratios, especially on the more challenging datasets, i.e., Cora and Citeseer, where a 12 . 9% AUC promotion can be gained compared with the second-best methods.
When it comes to decoupled noise settings shown in Table 3, RGIB also surpasses all the baselines ad hoc for input noise or label noise by a large margin.
Remark 5.1.
The two instantiations of RGIB can be generalized to different scenarios with their own priority w.r.t.
intrinsic graph properties that can be complementary to each other with ﬂexible options in practice.
Basically, the RGIB-SSL is more adaptive to sparser graphs, e.g., Cora and Citeseer, where the inherent edge noise presents a considerable challenge and results in greater performance degradation.
The RGIB-REP can be more suitable for denser graphs, e.g., Facebook and Chameleon.
Meanwhile, two instantiations also work effectively on heterogeneous graphs with low homophily.
The learned representation distribution.
Next, we justify that the proposed method can effectively alleviate the representation collapse.
Compared with the standard training, both RGIB-REP and
RGIB-SSL bring signiﬁcant improvements for the alignment with much lower values, as in Table 4.
At the same time, the uniformity of learned representation is also enhanced: it can be seen from
Figure 5 that the various query edges tend to be more uniformly distributed on the unit circle.
As
RGIB-SSL explicitly constrains the representation, its recovery power on representation distribution is naturally stronger than RGIB-REP, resulting in comparably better alignment and uniformity measures.
[Table]
[Table]
std. is short for standard training , and SSL/REP are short for RGIBSSL/RGIB-REP, respectively.
dataset
Cora
Citeseer method std.
REP SSL std.
REP SSL clean .616 .524 .475 .445 .439 .
ε =20% .687 .642 .543 .586 .533 .
ε =40% .695 .679 .578 .689 .623 .
ε =60% .732 .704 .615 .696 .647 .
[Figure] [Figure]
Optimization schedulers.
To reduce the search cost of coefﬁcients in objectives of RGIB-SSL and RGIB-REP, we setup a uniﬁed optimization framework that can be formed as L = α L cls + (1 − α ) R 1 + (1 − α ) R 2 .
Here, we attempt 5 different schedulers to tune the only hyper-parameter α ∈ [0 , 1] , including (1) constant, α ≡ c ; (2) linear, α t = k · t , where t is the normalized time step; (3) sine, α t = sin( t · π / 2 ) ; (4) cosine, α t = cos( t · π / 2 ) ; and (5) exponential, α t =
e k · t .
As empirical results summarized in Table 5, the selection of optimization schedulers greatly inﬂuences the ﬁnal results.
Although there is no gold scheduler that consistently performs the best, the constant and sine schedulers are generally better than others among the 5 above candidates. A further [Table]
5. A BLATION STUDY
In this part [Table] we conduct a thorough ablation study for the RGIB framework.
As shown in Table [Table] each component contributes to the ﬁnal performance.
For RGIB-SSL, we have the following analysis:
• Hybrid augmentation.
RGIB-SSL beneﬁts from the hybrid augmentation algorithm that automatically generates graphs of high diversity for contrast.
Compared with ﬁxed augmentation, the hybrid augmentation brings consistent improvements with a 3 . 0% average AUC promotion on Cora.
• Self-adversarial alignment loss.
Randomly-sampled pairs are with hierarchical information to be learned from, and the proposed re-weighting technique further enhances high-quality pairs and decreases low-quality counterparts.
It enables to discriminate the more informative contrasting pairs and thus reﬁnes the alignment signal for optimization, bringing to up 2 .
1% AUC promotion.
• Information constraints.
Label supervision contributes the most among the three informative terms, even with label noise.
Degenerating RGIB-SSL to a pure self-supervised manner without supervision (i.e., λ s =0 ) leads to an average 11 .
9% AUC drop.
Meanwhile, we show that three regularization terms can be jointly optimized while another two terms are also of signiﬁcant values.
As for RGIB-REP, its sample selection mechanism and corresponding constraints are also essential:
• Edge / label selection.
Two sample selection methods are near equally important for learning with the coupled inherent noise that both informative sources are required to be puriﬁed.
Besides, the edge selection is more important for tackling the decoupled input noise, as a greater drop will come when removed.
Similarly, the label selection plays a dominant role in handling the label noise.
• Topological / label constraint.
Table 6 also shows that the selection mechanism should be regularized by the related constraint; and otherwise, sub-optimal solutions will be achieved.
Besides, the topological constraint is generally more sensitive than the label constraint for RGIB-REP.
C ONCLUSION
In this work, we study the problem of link prediction with the inherent edge noise and reveal that the graph representation is severely collapsed under such a coupled noise.
Based on the observation, we introduce the Robust Graph Information Bottleneck (RGIB) principle, aiming to extract reliable signals via decoupling and balancing the mutual information among inputs, labels, and representation to enhance the robustness and avoid collapse.
Regarding the instantiation of RGIB, the self-supervised learning technique and data reparametrization mechanism are utilized to establish the RGIB-SSL and RGIB-REP, respectively.
Empirical studies on 6 datasets and 3 GNNs verify the denoising effect of the proposed RGIB under different noisy scenarios.
In future work, we will generalize RGIB to more scenarios in the link prediction domain, e.g., multi-hop reasoning on knowledge graphs with multi-relational edges that characterize more diverse and complex patterns.
R EPRODUCIBILITY STATEMENT
To ensure the reproducibility of the empirical result in this work, we have stated all the details about our experiments in corresponding contents of Section 3 and Section 5.
The implementation details are further elaborated in Appendix E. Besides, we will provide an anonymous Github repository for the source codes during the discussion phase for the reviewers of ICLR 2023.
Appendix Table of Contents
A Broader impact
B Notations C Theoretical justiﬁcation
C.1 Proof for Proposition 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Proof for Proposition 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Proof for Proposition 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.4 Proof for Proposition 4.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.5 Proof for Theorem 4.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D A Further Empirical Study of Noise Effects
D.1 Full evaluation results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
{+D.
+} Further investigation on the decoupled noise . . . . . . . . . . . . . . . . . . . D.3 Decoupling input noise and label noise . . D.4 A further discussion on the inherent edge noise . . . . . . . . . . . . . . . . . .
E Implementation details
E.1 GNNs for link prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . .[-D.
-]
E.2 Implementation details of RGIB-SSL . . . . . . . . . . . . . . . . . . . . . . .
E.3 Baseline implementations . . . . λ . . . . . . . . . . . . . . .
A B ROADER IMPACT
In this work, we propose the RGIB framework for denoising the inherent edge noise in graphs and for improving the robustness of link prediction.
We conceptually derive the RGIB and empirically justify that it enables the various GNNs to learn the input-invariant and label-invariant graph representations, preventing representation collapse and obtaining superior performances against edge noise.
Our objective is to protect and enhance the current graph models, and we do not think our work would have negative societal impacts.
A similar problem has been noticed in an early work (Gallagher et al., 2008), where robots tend to build connections with normal users to spread misinformation on social networks, yielding the degeneration of GNNs for robot detection.
In addition, as pointed out by a recent survey (Wu et al., 2022b), the inherent noise can be produced in the data generation process that requires manually annotating the data.
As introduced in (Wu et al., 2022b), the inherent noise here refers to irreducible noises in graph structures, attributes, and labels.
In short, the studied edge noise in our work is in line with the generic concept of graph inherent noise in the literature, which is recognized as a common problem in practice but also an under-explored challenge in both academic and industrial scenes.
However, several existing benchmarks, e.g., Cora and Citeseer, are generally clean and without annotated noisy edges.
Inevitably, there is a gap when one would like to study the inﬂuence of inherent noise on these common benchmarks. Thus, to ﬁll this gap, it is necessary to simulate the inherent edge noise properly to investigate the impact of noise. If the inherent edge noise exists, it should be false positive samples, while the false negative samples are often intractable to be collected. Thus, we focus on the investigation of the false positive edges as the inherent noise and design the simulation approach in Deﬁnition 3.1 as it is more close to the real-world scenarios.
Besides, to our best knowledge, we are the ﬁrst to study the robustness problem of link prediction under the inherent edge noise. One of our major contributions to this research problem is that we reveal that the inherent noise can bring a severe representation collapse and performance degradation, and such negative impacts are general to common datasets and GNNs. What’s more, it is also possible that new instantiations based on other kinds of methodology are inspired by the robust GIB principle. We believe that such a bidirectional information bottleneck that strictly treats the information source on both input side and label side is helpful in practice, especially for extremely noisy scenarios.
We summarize the frequently used notations in Table
[Table] [Table]
C T HEORETICAL JUSTIFICATION C. P ROOF FOR P ROPOSITION 4.
Proof.
As introduced in Section 4.2, the uniformity loss on two augmented graphs is formed as
R unif =
[Equation]
e
[Equation]
2 mn (cid:107) 2 2 . For simplicity, we can reduce it to one graph, i.e., R unif =
E ij,mn ∼ p E [ e −(cid:107) h ij − h mn (cid:107) 22 ] .
Next, we obtain its upper bound by the following derivation.
[Equation]
Here, ˆ p vMF-KDE ( · ) is a von Mises-Fisher (vMF) kernel density estimation based on N training samples, and Z vMF is the vMF normalization constant.
ˆ H ( · ) is the resubstitution entropy estimator of
H = f w ( · ) (Ahmad & Lin, 1976).
As the uniformity loss L unif can be approximated by entropy ˆ H ( H ) , a higher entropy H ( H ) indicates a lower uniformity loss, i.e., a higher uniformity.
C.
P ROOF FOR P ROPOSITION 4.
Proof.
Recall the hybrid graph augmentation technique is adopted by RGIB-SSL for contrasting, we can approximate the expectation E
[ I ( H ; ˜
A | ˜ Y )] by 1 / N (cid:80)
Ni
=
I ( H i ; ˜
A i | ˜ Y ) with N augmented graphs.
Based on this, we drive the upper bound of I ( H ; ˜
A | ˜ Y ) as follows.
[Equation]
Thus, a lower I ( H 1 ; H 2 ) can upper bounded as lower I ( H ; ˜ A ) and I ( H ;
˜
A | ˜ Y ) .
C. P ROOF FOR P ROPOSITION 4.
Proof.
With the marginal distribution Q ( Z A )= (cid:80) n P φ ( P | n ) P ( ˜ A = n )= P ( n ) (cid:81) n ˜
A ij =1 P ij , we drive the upper bound of MI
I ( Z A ; ˜ A ) as:
[Equation]
where the KL divergence on two given distribution P ( x ) and Q ( x ) is deﬁned as KL ( P ( x ) || Q ( x ))
= (cid:80) x P ( x ) log P ( x ) / Q ( x ) .
Thus, we obtain the upper bound of I ( Z A ; ˜ A ) as Eqn. 7.
Similarly, the label constraint is bounded as I ( Z Y ; ˜ Y ) ≤ (cid:80) e ij ∈ ˜ Y P ij log P ij τ +(1 − P ij ) log 1 − P ij 1 − τ = R Y .
C. P ROOF FOR P ROPOSITION 4.
Proof.
We derive the lower bound of I ( H ; Z Y ) as follows.
[Equation]
Thus, the MI I ( H ; Z Y ) can be lower bounded by the classiﬁcation loss, and min − λ s I
( H ; Z Y ) in
RGIB-REP (Eqn. 4) is upper bounded by min λ s / | Z Y | (cid:80) e ij ∈ Z Y L cls ( f w ( Z A ) , Z Y ) as Eqn. 8.
C. P ROOF FOR T HEOREM 4.
Proof.
With the relaxation via parametrization in C.4, we ﬁrst relax the standard RGIB-REP to its upper bound as follows.
[Equation]
As Z ∗ Y ≈ Y , Eqn.
9 can be reduced to min
− λ
s
I
( Z A ; Y )
+
λ A I
( Z A ; ˜ A )
+ λ Y H
( Y ) , where
I ( Y ; ˜ Y ) =
H ( Y ) as Y ⊆
˜ Y .
Removing the ﬁnal term with constant H ( Y ) , it can be further reduced to min − I
( Z A ; Y )
+ λI ( Z A ; ˜ A ) , where λ =
λ A / λ s .
Since to minimize the − I
( Z A ; Y )
+ λI ( Z A ; ˜ A ) is equal to maximize the I ( Z A ; Y ) − λI
( Z A ; ˜ A ) , next, we conduct the following derivation:
[Equation]
Since MI I ( ˜ A ; Y | Z A ) ≥
and I ( Z A ; ˜
A | Y ) ≥ 0 are always true, the optimal Z ∗ A should makes − (1 − λ )
I ( ˜ A ; Y | Z A )
− λI ( Z A ; ˜
A | Y )
=
to reach the optimal case.
Thus, it should satisfy I ( ˜ A ; Y | Z A )
=
and I ( Z A ; ˜ A | Y ) = 0 simultaneously.
Therefore, Z A
= Z ∗ A maximizes
I
( Z A ; Y )
− λI
( Z A ; ˜ A ) , which is equal to minimize − I ( Z A ; Y )
+ λI ( Z A ; ˜ A ) , i.e., the
RGIB-REP.
Symmetrically, when Z A ≡ A , Z Y =
Z ∗ Y maximizes I
( A ; Z Y ) − λI ( Z Y ; ˜ Y ) as
I ( A ; Z Y ) − λI ( Z Y ; ˜ Y ) =
(1 − λ )
I ( ˜ Y ; A ) − (1 − λ ) I ( ˜ Y ;
A | Z Y ) − λI
( Z Y ; ˜ Y | A ) .
D A F URTHER E MPIRICAL S TUDY OF N OISE E FFECTS
This section is the extension of Section 3 in the main content.
D. F ULL EVALUATION RESULTS
Evaluation settings.
In this part, we provide a thorough empirical study traversing all combinations of following settings from 5 different aspects.
• 3 GNN architectures: GCN, GAT, SAGE.
• 3 numbers of layers: 2 , 4 , 6 .
• 4 noise types: clean, mixed noise, input noise, label noise.
• 3 noise ratios: 20% , 40% , 60% .
• 6 datasets: Cora, CiteSeer, PubMed, Facebook, Chameleon, Squirrel.
Then, we summarize the entire evaluation results of three kinds of GNNs as follows.
As can be seen, all three common GNNs, including GCN, GAT, and SAGE, are vulnerable to the inherent edge noise.
• Table 13: full evaluation results with GCN.
• Table 14: full evaluation results with GAT.
• Table 15: full evaluation results with SAGE.
Data statistics.
The statistics of the 6 datasets in our experiments are shown in Table 8.
As can be seen, 4 homogeneous graphs (Cora, CiteSeer, PubMed, and Facebook) are with the much higher homophily values than the other 2 heterogeneous graphs (Chameleon and Squirrel).
[Table] [Table]
Homophily 0.
D. F URTHER INVESTIGATION ON THE DECOUPLED NOISE
Loss distribution We visualize the loss distribution under different scenarios to further investigate the memorization effect of GNNs.
As shown in Figure 6, two clusters are gradually separated apart with clean data
[Figure]
but such a learning process can be slowed down when training with the noisy-input data, which confuses the model and leads to overlapped distributions.
As for the label noise, the model cannot distinguish the noisy samples with a clear decision boundary to separate the clean and noisy samples apart.
Besides
[Figure]
it is found that the model can gradually memorize the noisy edges according to the decreasing trend of the corresponding losses, where the loss distribution of noisy edges is minimized and progressively moving towards that of the clean ones
[Figure]
[Figure]
Visualization of edge representations As shown in Figure 7, the dimensionality reduction technique,
T-SNE, is utilized here to measure the similarity of the learned edge representations h ij =
[ h i , h j ] with edges e ij in the test set E test .
Compared with the representation distribution of clean data (Figure 7(a)) [Figure] which present comparatively distinct cluster boundaries, both the input noise (Figure 7(b)) and the label noise (Figure 7(c)) are with overlapped clusters of the positive and negative samples.
Namely, the GCN model is confused by the two patterns of noise
[Figure]
and fails to give a discriminate the true or false edges with their representations
[Figure]
D. D ECOUPLING INPUT NOISE AND LABEL NOISE
For a further and deeper study of the coupled noise
[Figure]
we use the edge homophily metric to quantify the distribution of edges. Speciﬁcally, the homophily value h homoij of the edge e ij is computed as the cosine similarity of the node feature x i and x j , i.e., h homoij = cos ( x i , x j ) . As shown in ﬁgure 8, the distributions of edge homophily are nearly the same for label noise and structure noise, where the envelope of the two distributions are almost overlapping.
Here, we justify that the randomly split label noise and structure noise are indeed coupled together.
[Figure]
Based on the measurement of edge homophily, we then decouple these two kinds of noise in simulation.
Here, we consider two cases that separate these two kinds of noise apart, i.e., (case 1) input noise with high homophily and label noise with low homophily shown in Figure 9, and (case 2) input noise with low homophily and label noise with high homophily shown in Figure 10.
[Figure] [Figure] D. A FURTHER DISCUSSION ON THE INHERENT EDGE NOISE
Basically, the edge noise can exist in the form of false positive edges or false negative noise.
Speciﬁcally, the false positive edges are treated as existing edges with label 1, but in fact, such edges do not exist.
On the other hand, the false negative edges are treated as non-existing edges with label 0 that the predictive probabilities of such edges will be minimized.
Here, we would highlight that our work focuses on the false positive edges as it is more practical and common in real-world scenarios since the data annotating procedure can produce such a kind of noise (Wu et al., 2022b).
Thus, if the inherent edge noise exists, it is more likely to be false positive samples, while the false negative samples are often intractable to be collected and annotated in practice.
Actually, investigating the inﬂuences of the false negative samples is another line of research, such as (Yang et al., 2020; Kamigaito & Hayashi, 2022), which is orthogonal and complementary to our work.
Learning from a clean graph can also encounter the problem of false negative samples, which is usually due to the random sampling of negative nodes/edges.
Note that tackling the false negative samples is a well-known problem in the area of link prediction while handling the false positive samples is also valuable but still remains under-explored.
The input noise and label noise does look quite similar and coupled.
From the perspective of data processing, the collected noisy edges in the training set can be randomly split into the observed graph (i.e., the input of GNN) or the predictive graph (i.e., the query edges for the GNN to predict).
Considering the noisy edges might come from similar sources, e.g., biases from human annotation, the corresponding noise patterns can also be similar between these two kinds of noise that are naturally coupled together to some extent.
However, we would claim and justify that the two kinds of noise are fundamentally different.
Although both noise can bring severe degradation on empirical performances, they actually act in different places from the model perspective.
As the learnable weights are updated as w := w − η ∇ w L
( H , ˜Y ) , the label noise ˜ Y acting on the backward propagation can directly inﬂuence the model.
By contrast, the input noise indirectly acts on the learned weights as it appears in the front end of the forward inference, i.e., H = f w ( A , X ) .
Empirically, as results shown in Table 3 and Table 4, the standard GNN (without any defenses) performs quite differently under the same proportion of input noise and label noise.
Such a phenomenon can inspire one to understand the intrinsic denoising mechanism or memorization effects of the GNN, and we would leave that as further work.
More importantly, from the perspective of defending, it could be easy and trivial to defend if these two kinds of noise are the same.
However, none of the existing robust methods can effectively defend such an inherently coupled noise.
As can be seen from Table 2, only marginal improvements are achieved when applying the existing robust methods to the coupled noise.
While in Table 3 and Table 4, these robust methods work effectively in handling the decoupled noise, i.e., only the structure noise or label noise exists.
The reason is that, the properties of coupled noise are much more complex than the single decoupled noise.
Both sides of the information sources, i.e., ˜ A and ˜ Y , should be considered noisy, based on which the defending mechanism could be devised.
We provide a detailed introduction forward propagation and backward update of GNNs in this part.
Formally, let (cid:96) = 1 . . .
L denote the layer index, h (cid:96)i is the representation of the node i , MESS ( · ) is a learnable mapping function to transformer the input feature, AGGREGATE ( · ) captures the 1 -hop information from neighborhood N ( v ) in the graph, and COMBINE ( · ) is ﬁnal combination between neighbour features and the node itself.
Then, the l -layer operation of GNNs can be formulated as m (cid:96)v = AGGREGATE (cid:96) ( { MESS ( h (cid:96) − 1 u , h (cid:96) − 1 v , e uv ): u ∈ N ( v ) } ) , where the representation of node v is h (cid:96)v = COMBINE (cid:96) ( h (cid:96) − 1 v , m (cid:96)v ) .
After L -layer propagation, the ﬁnal node representations h Le of each e ∈ V are obtained.
Then, for each query edge e ij ∈E train unseen from the input graph, the logit φ e ij is computed with the node representations h Li and h Lj 4 with the readout function, i.e., φ e ij = READOUT ( h Li , h Lj )
→ R .
Finally, the optimization objective can be deﬁned as minimizing the binary cross-entropy loss, i.e., min L cls =
(cid:80) e ij ∈E train − y ij log ( σ ( φ e ij )) − (1 − y ij ) log (1 − σ ( φ e ij )) where σ ( · ) is the sigmoid function, and y ij =1 for positive edges while y ij =0 for negative ones.
In addtion, we summarize the detailed architectures of different GNNs in the following Table 9.
[Table]
E. I MPLEMENTATION DETAILS OF RGIB-SSL
As introduced in Section 4.2, the graph augmentation technique is adopted here to generate the perpetuated graphs of various views.
To avoid manually selecting and tuning the augmentation operations, we propose the hybrid graph augmentation method with the four augmentation operations as predeﬁned candidates and the ranges of their corresponding hyper-parameters.
The search space is summarized in Table 10, where the candidate operations cover most augmentation approaches except for those operations modifying the number of nodes that are unsuitable for the link prediction task.
In each training iteration, two augmentation operators T 1 ( · ) and T 2 ( · ) and their hyper-parameters θ 1 and θ 2 are randomly sampled from the search space as elaborated in Algorithm.
The two operators will be performed on the observed graph G , obtaining the two augmented graphs, namely,
G 1 = T 1 ( G| θ 1 ) and G 2 =
T 2 ( G| θ 2 ) .
The edge representations are gained by h 1 ij = f
( G 1 , e ij | w ) =
h [-
-]1 i (cid:12) h 1 j , where the node representations h 1 i and h 2 i are generated by the GNN model f ( ·| w ) with learnable weights w , and so it is for h 2 ij with G 2 . [Table]
[Table]
Require: number of augmentation operators n Then, we learn the self-supervised edge representations by maximizing the edge-level agreement between the same query edge of different augmented graphs (positive pairs) and minimizing the agreement among different edges (negative pairs) with their representations as shown in Figure 11.
Note the h ij here is the edge representation.
Speciﬁcally, we minimize the representation similarity of the positive pairs ( h 1 ij , h 2 ij ) and maximize the representation similarity of the randomly-sampled negative pairs ( h 1 ij , h 2 mn ) , where e ij (cid:54) = e mn . [Figure]
ASELINE IMPLEMENTATIONS
All baselines compared in this paper are based on their own original implementations.
We list their source links here.
• DropEdge, https://github.com/DropEdge/DropEdge .
• NeuralSparse, https://github.com/flyingdoog/PTDNet .
• PTDNet, https://github.com/flyingdoog/PTDNet .
• GCN Jaccard, https://github.com/DSE-MSU/DeepRobust .
• GIB, https://github.com/snap-stanford/GIB . • VIB, https://github.com/RingBDStack/VIB-GSL . • PRI, https://github.com/SJYuCNEL/PRI-Graphs .
• Co-teaching, https://github.com/bhanML/Co-teaching .
• Peer loss functions, https://github.com/weijiaheng/Multi-class-Peer-Loss-functions .
• SupCon, https://github.com/HobbitLong/SupContrast .
• GRACE, https://github.com/CRIPAC-DIG/GRACE .
F F ULL EMPIRICAL RESULTS
In this section, we elaborate the full empirical study on inherent edge noise with various robust methods and GNN architectures. F. R OBUST METHODS COMPARISON WITH CLEAN DATA Here, we would like to ﬁgure out how the robust methods introduced in Section 5 behave when learning with clean data, i.e., no edge noise exists. As shown in Table 11, the proposed two instantiations of RGIB can also boost the predicting performance when learning on clean graphs, and outperforms other baselines in most cases.
F. F URTHER ABLATION STUDIES ON THE TRADE - OFF PARAMETERS λ
We conduct an ablation study with the grid search of several hyper-parameters λ in RGIB.
For simplicity, we ﬁx the weight of the supervision signal as one, i.e., λ s = 1 .
Then, the objective of
RGIB can be formed as L =
L cls + λ 1 R 1 + λ 2 R 2 , where the information regularization terms
R 1 / R 2 are alignment and uniformity for RGIB-SSL, while topology constraint and label constraint for RGIB-REP, respectively.
As the heatmaps illustrated in Figure 12 and Figure 13, the λ 1 , λ 2 are
[Table]
better in certain ranges.
Neither too large nor too small value is not guaranteed to ﬁnd a good solution.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
F. F URTHER CASE STUDIES
Alignment and uniformity of baseline methods.
The alignment of other methods is summarized in Table 12, while the uniformity is visualized in Figure 14 and Figure 15.
We have the following three observations.
First, the sampling-based methods, e.g., DropEdge, PTDNet, and NeuralSparse, can also promote alignment and uniformity due to their sampling mechanisms to defend the structural perturbations.
Second, the contrastive methods, e.g., SupCon and GRACE, are with much better alignment but much worse uniformity.
The reason is that the learned representations are severely collapsed, which can be degenerated to single points seen from the uniformity plots but stay nearly unchanged when encountering structural perturbations.
Third, the remaining methods are not observed with signiﬁcant improvements in alignment or uniformity.
When connecting the above observations with their empirical performances, we can draw a conclusion.
That is, both alignment and uniformity are important to evaluate the robust methods from the perspective of representation learning.
Besides, such a conclusion is in line with the previous study (Wang & Isola, 2020).
[Table]
Jaccard
[Table]
GRACE
[Table]
[Table]
Learning curves of RGIB.
We draw the learning curves of RGIB with constant schedulers in
Figure 16, Figure 17, Figure 18, and Figure 19.
We normalize the values of each plotted line to (0 , 1) for better visualization.
For RGIB-SSL, the uniformity term, i.e, H ( H ) , converges quickly and remains low after 200 epochs.
Similarly, the alignment term, i.e, I ( H 1 ; H 2 ) also converges in the early stages and keeps stable in the rest.
At the same time, the supervised signal, i.e, I ( H ; Y ) gradually and steadily decreases as the training time moves forward.
The learning processes are generally stable across different datasets.
As for RGIB-REP, we observe that the topology I ( Z A ; ˜A ) and label constraints I ( Z Y ; ˜Y ) can indeed adapt to noisy scenarios with different noise ratios.
As can be seen, these two regularizations converge more signiﬁcantly when learning on a more noisy case.
That is, when noise ratio (cid:15) increases from 0 to 60% , these two regularizations react adaptively to the noisy data ˜ A, ˜ Y .
Such a phenomenon shows that RGIB-REP with these two information constraints works as an effective information bottleneck to ﬁlter out the noisy signals.
positive samples          negative samples (a) standard          positive samples          negative samples (b)
Dropedge          positive samples          negative samples (c) Neural Sparse          positive samples          negative samples (d) PTDNet          positive samples          negative samples (e) co-teaching          positive samples          negative samples (f) peer loss          positive samples          negative samples (g) Jaccard          positive samples          negative samples (h) GIB          positive samples          negative samples (i) SupCon          positive samples          negative samples (j) GRACE          positive samples          negative samples (k) RGIB-REP          positive samples          negative samples (l) RGIB-SSL
positive samples          negative samples (a) standard          positive samples          negative samples (b)
Dropedge          positive samples          negative samples (c) Neural Sparse          positive samples          negative samples (d) PTDNet          positive samples          negative samples (e) co-teaching          positive samples          negative samples (f) peer loss          positive samples          negative samples (g) Jaccard          positive samples          negative samples (h) GIB          positive samples          negative samples (i) SupCon          positive samples          negative samples (j) GRACE          positive samples          negative samples (k) RGIB-REP          positive samples          negative samples (l) RGIB-SSL
[Figure]
[Figure]
[Figure] [Figure] F. F ULL BASELINE COMPARISON WITH DECOUPLED NOISE The entire evaluation with 10 baselines and two proposed methods are conducted keeping the same settings as in D.1.
Results on each dataset are summarized as follows.
• Table 16, Table 17, Table 18 : full results of GCN/GAT/SAGE on Cora dataset. • Table 19, Table 20, Table 21 : full results of GCN/GAT/SAGE on CiteSeer dataset.
• Table 22, Table 23, Table 24 : full results of GCN/GAT/SAGE on PubMed dataset. • Table 25, Table 26, Table 27 : full results of GCN/GAT/SAGE on Facebook dataset.
• Table 28, Table 29, Table 30 : full results of GCN/GAT/SAGE on Chameleon dataset. • Table 31, Table 32, Table 33 : full results of GCN/GAT/SAGE on Squirrel dataset.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]