C HASING A LL -R OUND G RAPH R EPRESENTATION R OBUSTNESS : M ODEL , T RAINING , AND O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Neural Networks (GNNs) have achieved state-of-the-art results on a variety of graph learning tasks, however, it has been demonstrated that they are vulnerable to adversarial attacks, raising serious security concerns.
A lot of studies have been developed to train GNNs in a noisy environment and increase their robustness against adversarial attacks.
However, existing methods have not uncovered a principled difficulty: the convoluted mixture distribution between clean and attacked data samples, which leads to sub-optimal model design and limits their frameworks’ robustness.
In this work, we first begin by identifying the root cause of mixture distribution, then, for tackling it, we propose a novel method
G AME - G raph A dversarial M ixture of E xperts to enlarge the model capacity and enrich the representation diversity of adversarial samples, from three perspectives of model, training, and optimization.
Specifically, we first propose a plug-andplay G AME layer that can be easily incorporated into any GNNs and enhance their adversarial learning capabilities.
Second , we design a decoupling-based graph adversarial training in which the component of the model used to generate adversarial graphs is separated from the component used to update weights.
Third , we introduce a graph diversity regularization that enables the model to learn diverse representation and further improves model performance.
Extensive experiments demonstrate the effectiveness and advantages of G AME over the state-of-the-art adversarial training methods across various datasets given different attacks.
I NTRODUCTION
Graph neural networks (GNNs) have been demonstrated to be effective at learning from graphs.
They explore a message-passing mechanism to update node representations by iteratively aggregating information from their neighbors, allowing GNNs to achieve state-of-the-art performance (Kipf & Welling, 2017;
Veli ˇ ckovi ´ c et al., 2018; Hamilton et al., 2017).
Many real-world applications are based on GNNs, such as modeling over social networks (McAuley et al., 2015; Hu et al., 2020) and biological molecules (Jin et al., 2018; Xu et al., 2019).
Nevertheless, despite their outstanding performance, GNNs are susceptible to adversarial attacks (Z¨ugner & G¨unnemann, 2019; Zheng et al., 2021), which necessitate techniques to leverage
GNN’s robustness against adversarial perturbations.
Attackers can downgrade the performance of
GNNs from multiple perspectives, such as adding or removing edges (Geisler et al., 2021), perturbing node properties (Z¨ugner & G¨unnemann, 2019; Sun et al., 2020), and injecting malicious nodes (Zou et al., 2021).
To enhance GNN’s robustness, multiple defense methods against graph attacks have been proposed (Jin et al., 2020; Entezari et al., 2020; Zhang & Zitnik, 2020).
However, most existing methods have not uncovered the principled difficulty (i.e., the convoluted mixture distribution between clean and attacked data samples), which results in sub-optimal model design, poor robustness, and limited performance.
In light of this, we study the robustness of GNNs from a more fundamental perspective by discovering the key pattern behind the adversarial attacks that jeopardizes the performance of GNNs.
We begin by comparing the statistical divergence between the latent representations of nodes on the clean graph and the adversarially generated graph, as shown in Figure 1.
We observe that the distributions of node representations for clean and adversarial graphs before the message passing are highly similar (i.e., Figure 1(a)).
However, as the model gets deeper, these two distributions get
[Figure]
increasingly distinct, as demonstrated by the progressively larger shift shown from Figure 1 (a) to (c).
This demonstrates that adversarial attacks imperil GNN’s performance by generating adversarial graphs belonging to a distribution different from the clean graph, and the GNN model fails to transfer the knowledge learned from the clean graph to the generated adversarial graph.
To address the above challenge, we propose G raph A dversarial M ixture of E xperts ( G AME ), a novel framework that enhances the robustness for GNN by expanding the model capacity and increasing the representation diversity for adversarial graphs.
Specifically, we design G AME from three perspectives: ( i ) To strengthen the model capacity , we propose a plug-and-play G AME layer to accommodate the adversarial graphs with diverse mixture distributions by dynamically routing multiple assembled expert networks.
( ii )
From the training perspective, we present a decoupling graph adversarial training strategy, namely D E C O G, where each expert network is trained by adversarial graphs generated by maximizing the gradient of other experts.
D E C O G enforces each expert to learn distinct distributions that all other experts under-perform at.
( iii )
From the optimization perspective, we incorporate a graph diversity regularization (G RA D IV ) to further enhance the diversity of knowledge learned via all expert networks such that G AME is capable of handling various adversarial graphs.
G AME is an all-round robust framework that not only improves GNN’s resilience to adversarial attacks, but also without too much extra cost compared with normal GNN, since G AME dynamically activates only one subset of the experts to participate in the computation.
The contributions of this paper can be summarized as follows:
• To the best of our knowledge, this is the first work to improve GNN’s robustness from the perspective of distribution divergence.
According to our empirical studies, existing GNNs fail to transfer the knowledge learned from one clean graph’s distribution to another generated adversarial counterpart, which results in vulnerabilities to adversarial attacks.
• To solve this challenge, we propose an all-round framework, namely G raph A dversarial M ixture of E xperts ( G AME ), from the perspectives of model design (i
e., G AME layer to bolster the model capacity), training (i.e., D E C O G to diversify the adversarial graphs), and optimization (i
e., G RA D IV to further diversify the experts’ knowledge).
• Comprehensive experiments are performed on multiple benchmark datasets across varying scales, demonstrating that the robustness contributed to our proposed all-around G AME .
The suggested method beats other common baselines in a variety of attack evaluations and natural evaluations, demonstrating that the all-around robust design of G AME handles intricate mixture distribution well and cleverly addresses a fundamental difficulty in graph adversarial training.
R ELATED W ORK
Graph Neural Networks.
Graph Neural Networks have recently attracted a great deal of interest due to their effectiveness in learning non-Euclidean data and their remarkable performance in a vast array of graph mining tasks (Hamilton et al., 2017; Battaglia et al., 2018; Wu et al., 2020).
Graph convolutional network (GCN) is proposed in the early stage of GNN research to apply the concept of convolution from image to graph data (Kipf & Welling, 2017; Gao et al., 2018; Wu et al., 2019).
Instead of simply averaging the features of neighboring nodes, graph attention networks (Veliˇckovi´c et al., 2018; Wang et al., 2019) use the attention module to value each neighboring node and learn more important nodes during message passing.
In addition, graph recurrent neural networks (Ruiz et al., 2020) adopts the gating design on information propagation among graph structure inspired by recurrent neural networks in order to overcome the limitations of GNN regarding its message passing over long distances on large graphs.
Simultaneously, skip connection is introduced to construct deeper GNNs and learn comprehensive representations in order to overcome the over-smoothing phenomenon (Li et al., 2019; 2021a;b).
Unlike prior works focusing on improving model’s standard accuracy, to the best of our knowledge, our G AME is the first attempt to improve the robustness of adversarial graphs by introducing the Mixture of Experts mechanism to strengthen GNN capacity.
Adversarial Learning on Graphs.
It is demonstrated that deep learning models are susceptible to inputs with small adversarial perturbations, and several methods are proposed to improve the model’s robustness (Goodfellow et al., 2015; Kurakin et al., 2017; Xie & Yuille, 2020).
Recent research has shown that GNNs are susceptible to adversarial attacks without exception (Zheng et al., 2021), highlighting the urgent need to improve their robustness.
Several methods are proposed for attacking graph data, including inserting or removing connections (Du et al., 2017; Chen et al., 2018;
Waniek et al., 2018), perturbing node features (Z¨ugner et al., 2018; Z¨ugner & G¨unnemann, 2019;
Sun et al., 2020), or adding virtual nodes (Wang et al., 2020; Zou et al., 2021).
In the meantime, numerous defense methods against graph attacks have been developed for learning robust GNNs (Zhu et al., 2019;
Feng et al., 2020; Jin et al., 2020) or removing the attacked input during preprocessing (Entezari et al., 2020; Zhang & Zitnik, 2020).
Go beyond prior works, this paper addresses a fundamental issue in graph adversarial learning (i.e., overly complex mixture distributions between clean and attacked nodes), subsequently improving the model capacity and representation diversity.
P RELIMINARIES
Mixture of Experts.
In deep learning, the Mixture of Experts (MoE) constructs big neural networks with a dynamic routing strategy, which facilitates superior model capacity and attains better data parallelism (Shazeer et al., 2017).
Given the input x ∈ R d , the current expert layer E =
{ E i ( · ) } ni = with n experts, and the gating network P ( · ) =
{ p i ( · ) } ni =1 , the output of MoE module can be formulated as follows:
[Equation]
where T represents the set of activated top- k expert indices.
In the above equation, the gating module P ( x ) makes the activated portion of the model the same size as the normal network, hence enabling the efficient training of a larger neural network.
Specifically, we calculate the gate-value p i ( x ) for i -th expert as follows:
[Equation]
where h ( x ) is a linear transformation to compute the logits of experts given input x , and h ( x ) i reflects the i -th value of the obtained logits, which weights the i -th expert in current layer.
Adversarial Training.
Given the data distribution D over inputs x ∈ R d and their labels y ∈ R c , a standard classifier ( e . g ., a neural network) f : R d → R c with parameter θ maps an input to labels for classification, utilizing empirical risk minimization:
[Equation]
where L ( · , · ) represents the cross-entropy loss.
Numerous strategies have been proposed to improve neural network adversarial robustness, with adversarial training-based methods being the most effective.
The majority of cutting-edge adversarial training algorithms optimize a hybrid loss consisting of a standard classification loss and an adversarial loss term:
[Equation]
where L cls denotes the classification loss over standard (or clean) inputs, while L adv is is the loss that encourages the model to learn from adversarial data samples, and B ( ϵ ) =
{ δ | ∥ δ ∥ ∞ ≤ ϵ } is the perturbation set.
Popular adversarial training methods such as PGD (Madry et al., 2018) employs the same L cls as in Equation 4, but substitute L adv with a soft logits-pairing term.
Thus, the overall goal of adversarial training is to minimize the following objective:
[Equation]
where λ is a fixed weight hyper-parameter.
To balance standard and robust accuracies, the hyperparameter λ must be set to a constant value during training for all of these contemporary adversarial learning works.
∗ + +
Mixture of Experts Gradient Update Adversarial Graph Generation maximizing ℒ !"
∗ ∇ $ ℒ !"
Activating Different Partial Experts ℒ !"
# ℒ %&'!
( ℛ )*!
+,# minimizing ℒ )-./ …
Gate × … % &
N Layers Σ 1 1 0 1 1 0 0 0" 1 1 1 0 0 1 0 1 1 1 00 0 1… … " ∗ (1) GAME layer (b) 1 1 1 0 0 1 01 1 0 0 0 0 11 1 1 0 0 1 11 1 1 0 0 0 0… … … … … … …
Gate × … % &
N Layers Σ …
Gate × … % &
N Layers Σ … (2) DeCoG (3) GraDiv ∇ - ℒ !"
# ℒ !"
# ℒ )-./
M ETHODOLOGY
In this section, motivated by the prior analysis of adversarial mixture distribution in Section 1, we present a novel framework G AME to increase the adversarial robustness of GNNs.
Figure 2 illustrates the overall framework of the proposed model.
Specifically, first, our model is developed using
G AME layers, which increases model capacity on adversarial mixture distribution by introducing the
MoE mechanism.
Second, based on the G AME layer, we design a novel D E C O G training strategy that augments more varied graphs to facilitate GNNs’ adversarial training.
Third, in order to classify adversarial samples from mixture distribution, we suggest graph diversity regularization for learning more distinguishable representations.
The details of different parts are described in the following.
4. E XPANDING M ODEL C APACITY : G AME L AYER
To improve the learning capacity of GNNs and account for the overly complex mixture distribution, each GNN layer in our model includes a G AME layer at the model level.
To update the feature of a target node, our model employs G AME layer to transform and aggregate the features of the neighboring nodes.
Subsequently, the G AME combines the feature of target node and neighboring nodes to formulate node representation in the graph:
[Equation]
where h ( l ) v denotes representation of node v at l -th layer; AGGR( · ) and COMB( · ) represent the neighbor aggregation and combination functions, respectively.
In Equation 6, the G AME layer are constructed by W =
{ W i ( · ) } ni =1 and P ( · ) , which represent the set of n expert networks and the gate module, respectively.
Then, the G AME layer is formulated as follows:
[Equation]
where T represents the set of activated top- k experts in each G AME layer.
In comparison to conventional GNNs, the proposed sparse G AME layer is able to build a wider model with low computational cost, resulting in a greater capacity to express mixed distributions, as shown in Figure 2 (1).
4. B OOSTING ADVERSARIAL GRAPHS : D E C O G
For boosting the diversity of attacked node attributes and attacked adjacency matrices, we present
Decoupling-based Graph Adversarial Training Strategy (D E C O G).
D E C O G aims to deliver more robust performance than traditional graph adversarial training by transferring adversarial features from all experts to each individual expert.
Specifically, given a clean graph G and the node labels y with data distribution D , we first calculate the loss function over both clean graph G and attacked graph G ∗ between input G ’s target nodes and their ground truth label y .
Then, we learn a robust
G AME model f θ with parameters θ :
[Equation]
where L cleance ( · , · ) , L advce ( · , · ) indicate the losses (e.g., cross-entropy for node classification) on clean graph and attacked graph, respectively, and λ regulates the weight of the adversarial loss.
The adversarial graph G ∗ is generated by G AME ’s customized PGD attack: during the learning procedure of
PGD, G AME activates partial experts to compute multiple iterations of adversarial gradients.
When the computation is finished, the gradients are added to the original graph as the final adversarial input for GNNs to minimize the adversarial learning loss.
Therefore, each expert acquires the adversarial features generated by others.
From a high-level vantage point, D E C O G enables our G AME layer to implicitly transfer the aggregated knowledge of all experts to each individual expert.
To formally describe the aforementioned pipeline, given the initial clean graph as G (0) , we first extract the initial adjacency matrix A (0) and node features X (0) from G (0) .
Then, we dynamically sample the index set of activated experts T 0 using { A (0) , X (0) } to determine the G AME model’s current activated part f θ ′ (0) | T 0 ( · ) , where θ ′ (0) | T 0 is a subset of the G AME model f θ with parameters θ at the current iteration.
To maximize L ce , f θ ′ (0) | T 0 ( · ) utilizes { A (0) , X (0) } as input.
The gradients computed for A (0) and X (0) are represented as ∇ A (0) L ce ( f θ ′ (0) ( G (0) ) and ∇ X (0)
L ce ( f θ ′ (0) ( G (0) ) , y ) , respectively.
Both types of gradients are considered as adversarial noises, which are later incorporated to the current input { A (0) , X (0) } to derive the perturbed adjacency matrix and node features { A (1) , X (1) } .
The procedure can be formulated as:
[Equation]
where B ( A, ϵ ) is the ℓ 0 ball with radius around clean A , B ( X, ϵ ) is the ℓ ∞ ball with radius around clean X , Π B ( A,ϵ ) means the nearest projection to B ( A, ϵ ) , Π B ( X,ϵ ) means the nearest projection to
B ( X, ϵ ) , and α is the step size.
With the aim of acquiring the adversarial graph G ∗ =
{ A ( ∗ ) , X ( ∗ ) } , we calculate the gradients T times as in Equation 9 and treat the final gradient as the perturbation.
Later, the adversarial graph G ∗ is utilized to train a different subset of activated G AME experts.
In general, D E C O G enables the dynamic activation of each expert in G AME and facilitates the computation of more diverse attacked graph adjacency matrix and node features, as depicted in
Figure 2 (2).
4. E NHANCING CLASSIFIABILITY IN ROBUST REPRESENTATION : G RA D IV
For empowering G AME in learning more distinguishable representation from the complex mixture distribution, we design Graph Diversity Regularization (G RA D IV ) from the optimization level.
This regularization term penalizes the model to maximize the distance between node embeddings and enforces the model to learn distinct representations.
The regularization is formulated as:
[Equation]
where S IM ( · , · ) calculates the cosine similarity between two node representation, and N denotes the number of nodes.
R G RA D IV increases the distance between any pairs of learned node embeddings, which offers explicit supervision signal to learn node representations with more variety, and hence improving the model learning capacity.
Finally, the overall learning objective function L G AME is defined as a weighted combination of L clean ce
[Equation]
Consequently, L G AME enables G AME to learn distinguishable node embeddings and preserve robustness to attacked node features, thereby improving the representation quality and downstream task performance.
E XPERIMENT
In this section, we perform comprehensive experiments on the graph robustness benchmarks to demonstrate the effectiveness of our proposed G AME model against adversarial graphs with complex distributions.
This section is guided by answers to the following five research questions: RQ-1 : Can
G AME achieve better robustness compared to other SOTAs?
RQ-2 : How does each component in our all-around framework contribute to the robustness improvement? RQ-3 : Can G AME separate the mixed distribution of clean and attacked nodes? RQ-4 :
Does G AME generate more diversified training adversarial graphs compared to baselines?
and RQ-5 : Can G AME learn distinguishable node representations?
5. E XPERIMENTAL S ETUP
Datasets .
We utilize Graph Robust Benchmark (Zheng et al., 2021) dataset to evaluate our model’s performance by graphs with varying scales, including grb-cora (small-scale), grb-citeseer (smallscale), grb-flickr (medium-scale), grb-reddit (large-scale), and grb-aminer (large-scale).
Baseline Methods .
We compare G AME with various baseline methods, spanning multiple perspectives.
For models that specifically focus on robustness, we explore R-GCN (Zhu et al., 2019), GNNSVD (Entezari et al., 2020), and GNNGuard (Zhang & Zitnik, 2020).
In addition, we incoporate general GNN models (i.e., GCN (Kipf & Welling, 2017), GAT
(Veli ˇ ckovi ´ c et al., 2018), GIN (Xu et al., 2019),
APPNP (Gasteiger et al., 2019), TAGCN (Du et al., 2017), GraphSAGE (Hamilton et al., 2017), SGCN (Wu et al., 2019)) with two generic defense approaches (i.e., layer normalization (Ba et al., 2016) (LN) and adversarial training (AT) (Madry et al., 2018)).
Attacking Strategies .
We explore five effective yet diverse graph attack methods to imperil the victim GNNs: RND (Zheng et al., 2021), FGSM (Goodfellow et al., 2015; Zheng et al., 2021),
PGD (Madry et al., 2018), SPEIT (Zheng et al., 2021), TDGIA (Zou et al., 2021).
These attack methods have been proven to deliver scalable and transferable attacks (Zheng et al., 2021).
Details of attacking strategies and adversarial training are described in Appendix B.1 and B.2, respectively.
In addtion, we also include the statistics of datasets in Appendix C.
5. O VERALL P ERFORMANCE ON G RAPH R OBUST B ENCHMARK
To answer RQ-1 , we conduct the experiments to evaluate the robustness and report the performance in Figure 3 (We run 10 times for mean results/standard deviation and the train:val:test split is 0.6:0.1:0.3. Due to the page limitation, we include the full results Table 2 with all numerical values in Appendix A ).
As shown in Figure 7, G AME comprehensively achieves better robust accuracy than other baselines under five distinct attack assessments.
For instance, on the small-scale grb-citeseer, the average accuracy of G AME outperforms the second-place R-GCN +AT by 2.96%; on the mediumscale grb-flickr, the average accuracy of our G AME outperforms the runner-up GAT +AT by 2.11%; on the large-scale grb-aminer, G AME outperforms the second-best GAT +AT by 1.32%.
These results demonstrate the outstanding effectiveness of G AME against the distribution divergence across five
GRB graphs with different scales.
Besides, under the w.o. attack setting, G AME significantly outperforms baselines by a large margin, proving that G AME equipped with multiple expert networks has stronger learning capability as well as higher accuracy on clean graphs.
Though more parametrized than other baselines, G AME still enjoys the efficiency like regular GNNs and retains remarkable robustness against adversarial attacks, thanks to our proposed dynamic routing strategy that only activates partial experts to approximate the forward and backward of a normal GNN.
5. C ONTRIBUTION OF I NDIVIDUAL C OMPONENT IN THE A LL - ROUND F RAMEWORK
G AME integrates three different components into a comprehensive graph robust learning framework, and to answer RQ-2 , we conduct experiments on the performance without one of the individual components in G AME , denoted as ( a ) w/o G AME layer, ( b ) w/o D E C O G, and ( c ) w/o G RA D IV , as shown in Table 1.
We observe that the performance in both adversarial and clean graphs decreases after removing each component, demonstrating the contribution of each design in increasing the model’s performance.
For ( a ) w/o G AME layer , removing G AME layer from our framework results in G AME being vanilla GCN on the model level.
Such a removal disables the G RA D IV training technique, as G RA D IV depends on the G AME layer.
Removing G AME layer decreases the performance on clean graphs (under w.o. attack) more than that on adversarial graphs.
Specifically, under wo attack, this removal causes the model lose 2.58% accuracy, which is higher than the accuracy losses under two representative attacks (0.98% of PGD-based and 0.98% of FDSM-based) on grb-citeseer, respectively
For ( b ) w/o D E C O G, disabling the D E C O G training strategy results in the generated adversarial graphs tightly coupled with the activated experts at the current iteration (i.e., the activated experts for adversarial gradient computation are identical to those updated by minimizing the loss).
As a result, this removal causes a more severe performance deterioration for adversarial graphs than it does for clean graphs.
Specifically, under PGD-based and FDSM-based attacks, this removal causes the model on grb-citeseer to lose 0.56% and 0.75% accuracy respectively, which is comparable to the removal of G AME layer.
However, the accuracy loss on wo attack is only 1.56%, which is significantly less than the accuracy loss caused by G AME layer.
It demonstrates that D E C O G improves the performance of GNNs against adversarial attacks by generating diverse yet effective adversarial graphs, which is also compatible with the subsequent additional studies in
Section 5.5.
And for ( c ) w/o G RA D IV , G AME without the use of R G RA D IV in Equation 11 leads to a scenario where actively diversifying node embeddings is no longer an explicit supervision signal during the optimization process.
As a result, this removal leads to performance downgrade on both clean graph and adversarial graph (0.65% on clean grb-citeseer and 0.42% on FGSM-attacked grbciteseer, respectively), showing that the effectiveness of G RA D IV in assisting G AME model to learn distinguishable representations on both clean and adversarial graphs.
5. P ERFORMANCE AGAINST D ISTRIBUTION D IVERGENCE
We evaluate the performance of G AME and the vanilla GCN against the attacks from graphs generated through the adversarial distributions that are extremely divergent from the distribution of clean graphs, as shown in Figure 4.
To answer RQ-3 , we visualize the distributions of node representations from GCN and G AME at three stages (i.e., input layer (left column) and after first and second message passing layers (middle and right column respectively)).
We observe that for the vanilla GCN, the distribution shift between the node representations in the clean and adversarial graphs still
[Table]
r
[Figure]
i n e r G
[Figure] [Table]
[Figure]
exists, even when the learning model forwards. In addition, the distribution difference between adversarial graphs and clean graphs is small. This phenomenon demonstrates that vanilla GNNs such as GCN suffer from mixture distributions between adversarial graphs and clean graphs, which prevents GCN from learning distinguishable representations.
Compared with the vanilla GCN, G AME can successfully distinguish the node representations of the adversarial graphs from those of the clean graphs.
Besides, as node representations pass through deeper layers, G AME still maintains the ability of discrimination, demonstrating G AME ’s outstanding effectiveness against adversarial graphs.
5. D IVERSITY OF A DVERSARIAL G RAPHS G ENERATED BY G AME
[Figure]
To answer RQ- we visualize the distributions of both clean and adversarial graphs generated by G AME and GCN in Figure 5.
We observe that the adversarial graphs generated from
GCN are similarly distributed to the clean graph.
On the contrary, G AME generates adversarial graphs whose distribution is statistically distinct from that of the clean graphs, further [Figure] demonstrating the effectiveness of D E C O G and G RA D IV .
Diverse experts enable G AME to learn distinguishable node representations for robust performance, which significantly mitigates the GNN’s training difficulties on the graphs with distinct distributions
. 5. D IVERSITY OF N ODE R EPRESENTATIONS BY G AME
[Figure]
To answer RQ-5 , on grb-cora dataset, we visualize the node representations generated by
G AME and a vanilla GCN, shown in Figure 6.
We observe that the node representations generated by GCN are generally entangled and intertwined with each other, while those generated by G AME exhibit a very well-clustered representation space with clear inter-cluster difference.
This phenomenon demonstrates that
G AME can distinguish the complex distributions and further learn distinguishable node representations, thanks to the all-around design of G AME .
5. A NALYSES ON OPTIMIZATION LANDSCAPE
To further validate the robustness of G AME , we analyze whether our all-round framework G AME reduces the difficulty of learning adversarial graphs by plotting its loss landscape (Li et al., 2018) w.r.t.
both input and weight spaces.
According to Figure 7, we observe that G AME leads to a flatter minima optimization landscape than the adversarially trained GCN on both clean graphs and adversarial graphs, indicating that the method advantageously alleviates the learning difficulty on the adversarial graph.
our G AME reduces the complexity of learning adversarial graphs, allowing adversarial training model weights to be as simple as in a standard environment rather than GCN.
C ONCLUSION
In this paper, we first identify the fundamental issue in adversarial graph learning: the mixture distribution between clean and attacked data samples.
Motivated by this problem, we propose G raph
A dversarial M ixture of E xperts ( G AME ), a novel method to improve the model capacity, augment adversarial graphs, and enrich the graph representation diversity.
For acquiring these triple improvements, G AME contains three innovative components, including a plug-and-play G AME layer, a decoupling graph adversarial training strategy D E C O G, and a graph diversity regularization strategy G RA D IV .
G AME outperforms other baselines when evaluated on several datasets given different attack methods.
Additional experimental analyses prove the effectiveness of G AME in handling the complex mixture distribution, generating distinct adversarial graphs, and learning distinguishable representations.
E THICS S TATEMENT
G AME enhances the robustness of GNN models against adversarial attacks, and hence we believe that no ethical issue can be entailed by our approach.
In general, we should be very careful when applying machine learning models to guarantee that no negative societal impact is cast.
R EPRODUCIBILITY S TATEMENT
To ensure the reproducibility of our experiments, we include the anonymous ink of the source code in Appendix B.2.
In addition, the hyper-parameters and other factors to reproduce our method are also provided in the Appendix B.
A F ULL R ESULTS OF P ERFORMANCE C OMPARISON
We conduct extensive experiments on all five datasets in Table 2.
Here we display the results of graph injection scenario with Top-5 attacks v.s. Top-10 defenses plus our G AME model.
Since we have chosen strong defense methods as baselines, it is generally hard for attacks to be all effectives.
The best performance is Bolded and the runner-up is underlined.
[Table]
B I MPLEMENTATION D ETAILS B. R EPRODUCIBILITY SETTINGS OF A TTACK M ETHODS
We evaluate all the methods using five most common graph attack methods provide by GRB benchmark (Zheng et al., 2021), including random, fast gradient sign method, projected gradient descent,
SPEIT, and topological defective graph injection attack.
The details of these five attack methods are as follows:
• RND (Random): a method that injects random noises generated by Gaussian distribution (Z¨ugner et al., 2018).
• FGSM (Fast Gradient Sign Method): a method that linearizes the loss function around the current value of parameters to get an optimal max-norm constrained perturbation (Goodfellow et al., 2015).
• PGD (Projected Gradient Descent): a first-order adversary method that generates strongest assault using local first-order information about the network (Madry et al., 2018).
• SPEIT : a winning solution of KDD-CUP 2020 Graph Adversarial Attack & Defense competition, which is a global black-box graph injection attack with adversarial adjacent matrix and feature gradient attacks (Qinkai et al., 2020).
• TDGIA
(Topological Defective Graph Injection Attack): a powerful graph injection attack that injects malicious nodes progressively around topologically vulnerable nodes in the graph (Zou et al., 2021).
Following the prior work (Zheng et al., 2021), using a vanilla GCN as the surrogate model brings more transferable and better black-box attack effects than other GNN models.
Therefore, we choose
GCN as the surrogate model for all attacks in our experiments.
B. H YPER P ARAMETERS AND A DVERSARIAL T RAINING D ETAILS
The hyper-parameters of G AME are shown in Table 3.
The Hyper-parameters for adversarial training used in D E C O G are included in Table 4.
We then show the the adversarial training (AT) procedure of G AME as follows:
Step ① Initialization: The warm-up step.
The training graph is utilized to train G AME model for a few iterations
Step ② PGD attack: The PGD attack is employed to inject the malicious nodes and edges that assault the training nodes by message passing and create an attacked graph.
Step ③ Update gradients :
The model parameters are updated based on the gradients that are calculated by training on the attacked graph and minimizing the node classification loss.
Step ④ Repetition: This adversarial training procedure is repeated until the training loss converges.
Finally, we can obtain a G AME model with better learning capability and outstanding robustness.
According to Table 2, we intriguingly discover that G AME with PGD can also defend against other types of attacks, demonstrating the superior generality and applicability of G AME .
Notably, we discover that the G AME layer’s favorable attribute, i.e., sparsely activating partial experts, can be employed to design D E C O G training strategy, which augments more diverse graphs.
Specifically, in the Step ② and Step ③ , we activate different portions of experts in G AME model, despite the fact that both steps are performed in the same epoch.
The code is provided this anonymous link 1 .
In addition, the suggested G AME is scalable to graphs of varying sizes.
We use mini-batch training via neighborhood sampling with batchsize 1k to train G AME on grb-reddit and grb-aminer datasets.
In the future, we will transfer the framework of G AME to other GNN models.
[Table]
[Table]
C S TATISTICS OF G RAPH R OBUST B ENCHMARK D ATASETS
[Table] We evaluate our proposed G AME as well as adversarial learning framework on five real-world GRB datasets (Zheng et al., 2021), spanning from small- to large-scales.
The data statistics are displayed in Table 5.
To utilize grb-cora, grb-citeseer, grb-flickr, grb-reddit, grb-aminer , we apply the tool provided by Graph Robustness Benchmark [Table]
D G RA D IV ’ S E FFECTIVENESS ON D IVERSITY OF N ODE R EPRESENTATIONS
To discuss G RA D IV ’s impact on nodes’ representation diversity, on grb-cora dataset, we visualize the node representations generated by G AME without G RA D IV and G AME , shown in Figure 8.
We observe that the node representations generated by G AME without G RA D IV are more entangled and intertwined with each other then G AME .
This phenomenon demonstrates that G RA D IV is able to learn more distinguishable node representations.
E A DVERSARIAL G RAPHS G ENERATED BY G AME ON GRB - FLICKR
We observe that the adversarial graphs generated from GCN are similarly distributed to the clean graph in Figure 9 (i.e.,the red and black dots in the lower left corner of the GCN figure overlap significantly, but in the GAME figure the overlap is minor).
On the contrary, G AME generates adversarial
[Figure]
graphs whose distribution is statistically distinct from that of the clean
, further the effectiveness of D E C O G and G RA D IV .
Diverse experts enable G AME to learn distinguishable node representations for robust performance, which significantly mitigates the GNN’s training difficulties on the graphs with distinct distributions.
F T HE Z OOMED VERSION OF F IGURE
Figure 10 is a zoomed version of Figure 5 for better clarity.
Because the features of the GCNgenerated adversarial graphs almost overlap with the normal graphs (e.g., the red and black dots in the lower left corner of the GCN figure overlap significantly, but in the GAME figure the overlap is minor), their distributions are too similar and not as diverse as those generated by GAME.
Because the adversarial samples created by our GAME have more different adversarial distributions than
GCN, our GAME have better graph adversarial training effects.
In summary, Figure 5 demonstrates that GCN’s adversarial graphs are of worse quality than GAME’s.
[Figure]