ASGNN: G RAPH N EURAL N ETWORKS WITH A DAPTIVE S TRUCTURE
Anonymous authors
Paper under double-blind review
A BSTRACT
The graph neural network (GNN) models have presented impressive achievements in numerous machine learning tasks.
However, many existing GNN models are shown to be vulnerable to adversarial attacks, which creates a stringent need to build robust GNN architectures.
In this work, we propose a novel interpretable message passing scheme with adaptive structure (ASMP) to defend against adversarial attacks on graph structure.
Layers in ASMP are derived based on optimization steps that minimize an objective function that learns the node feature and the graph structure simultaneously.
ASMP is adaptive in the sense that the message passing process in different layers is able to be carried out over dynamically adjusted graphs.
Such property allows more ﬁne-grained handling of the noisy (or perturbed) graph structure and hence improves the robustness.
Convergence properties of the ASMP scheme are theoretically established.
Integrating ASMP with neural networks can lead to a new family of GNN models with adaptive structure (ASGNN).
Extensive experiments on semi-supervised node classiﬁcation tasks demonstrate that the proposed ASGNN outperforms the state-of-the-art GNN architectures in terms of classiﬁcation performance under various adversarial attacks.
I NTRODUCTION
Graphs, or networks, are ubiquitous data structures in many ﬁelds of science and engineering (Newman, 2018), like molecular biology, computer vision, social science, ﬁnancial technology, etc.
In the past few years, due to its appealing capability of learning representations through message passing over the graph structure, graph neural network (GNN) models have become popular choices for processing graph-structured data and have achieved astonishing success in various applications (Kipf and Welling, 2017; Bronstein et al., 2017; Wu et al., 2020;
Zhou et al., 2020; Wu et al., 2022).
However, existing GNN backbones such as the graph convolutional network (GCN) (Kipf and
Welling, 2017) and the graph attention network (Veliˇckovi´c et al., 2018) are shown to be extremely vulnerable to carefully designed adversarial attacks on the graph structure (Sun et al., 2018; Jin et al., 2021; Günnemann, 2022).
With unnoticeable malicious manipulations of the graph, the performance of GNNs signiﬁcantly drops and may even be worse than the performance of a simple baseline that ignores all the relational information among data feature (Dai et al., 2018; Zügner et al., 2018; Zügner and Günnemann, 2019; Zhang and Zitnik, 2020).
With the increasing deployments of GNN models in various real-world applications, it is of vital importance to ensure their reliability and robustness, especially in scenarios, such as medical diagnosis and credit scoring, where a deﬂected model can lead to dramatic consequences (Günnemann, 2022).
To improve the robustness of GNNs with a potentially noisy graph structure input, a natural idea is to “purify” the given graph structure.
Existing work in this line can be roughly classiﬁed into two categories.
The ﬁrst category of robustifying GNNs can be viewed as a two-stage approach (Wu et al., 2019b; Entezari et al., 2020; Gu et al., 2021).
A puriﬁed graph is ﬁrstly obtained by “pre-processing” the input graph structure leveraging on information from the node feature.
Next, a GNN model is trained based on this puriﬁed graph.
For example, in the GNN-Jaccard method (Wu et al., 2019b), a new graph is obtained by removing the edges with small “Jaccard similarity.”
In Entezari et al.
(2020), observing that adversarial attacks can scale up the rank of the graph adjacency matrix, the authors propose to use a low-rank approximation version of the given graph adjacency matrix as a substitute.
In the second category, the graph adjacency matrix in a GNN model is treated as an unknown, a puriﬁed graph structure with a parameterized form will be “learned” through optimizing the supervised GNN training loss (Zhu et al., 2022).
For example, in Franceschi et al.
(2019), the graph adjacency matrix is directly learned with a GNN in a bilevel optimization way, where a full parametrization of the graph adjacency matrix is adopted.
Moreover, under this full parametrization setting, structural regularizers are adopted in Jin et al.
; Luo et al.
(2021) as augmentations on the training loss function to promote certain properties of the puriﬁed graph.
Besides the full parametrization approach, a multi-head weighted cosine similarity metric function (Chen et al., 2020) and a GNN model (Yu et al., 2020) have also been used to parameterize the graph adjacency matrix for structure learning.
Going beyond purifying the graph structures to robustify the GNN models, there are also efforts on designing robust GNN architectures via directly designing the feature aggregation schemes.
Under the observation that aggregation functions such as sum, weighted mean, or the max operations can be arbitrarily distorted by only a single outlier node, Geisler et al.
; Wang et al.
; Zhang and Lu (2020) try to design robust GNN models via designing robust aggregation functions.
Moreover, some works apply the attention mechanism (Veliˇckovi´c et al., 2018) to mitigate the inﬂuence of adversarial perturbations.
For example, Zhu et al.
(2019) consider the node feature following a Gaussian distribution and use the variance information to determine the attention scores. Tang et al.
(2020) use clean graph information and their adversarial counterparts to train an attention mechanism to learn to assign small attention scores to the perturbed edges. In Zhang and Zitnik (2020), the authors deﬁne an attention mechanism based on the similarity of neighboring nodes.
Different from existing approaches to robustify GNNs, in this work, we propose a novel robust and interpretable message passing scheme with adaptive structure (ASMP).
Based on ASMP, a family of GNN models with adaptive structure (ASGNN) can be designed.
Prior works have revealed that the message passing processes in a class of GNNs are actually (unrolled) gradient steps for solving a graph signal denoising (GSD) problem (Zhu et al., 2021; Ma et al., 2021; Zhang and Zhao, 2022). ASMP is actually generated by an alternating (proximal) gradient descent algorithm for simultaneously denoising the graph signal and the graph structure.
Designed in such a principled way, ASMP is not only friendly to back-propagation training but also achieves the desired structure adaptivity with a theoretical convergence guarantee. Once trained, ASMP can be naturally interpreted as a parameter-optimized iterative algorithm.
This work falls into the category of GNN architecture designs. Conceptually different from the existing robustiﬁed GNNs with ﬁxed graph structure, ASGNN interweaves the graph puriﬁcation process and the message passing process, which makes it possible to conduct message passing over different graph structures at different layers, i.e., in an adaptive graph structure fashion.
Thus, an edge might be excluded in some layers but included in other layers, depending on the dynamic structure learning process.
Such property allows more ﬁne-grained handling of perturbations than existing graph puriﬁcation methods that use a single graph in the entire GNN.
To be more speciﬁc, the major contributions of this work are highlighted in the following.
• We propose a novel message passing scheme over graphs called ASMP with convergence guarantee and speciﬁcations. To the best of our knowledge, ASMP is the ﬁrst message passing scheme with adaptive structure that is designed based on an optimization problem.
• Based on ASMP, a family of GNN models with adaptive structure, named ASGNN, are further introduced.
The adaptive structure in ASGNN allows more ﬁne-grained handling of noisy graph structures and strengthens the model robustness against adversarial attacks.
• Extensive experiments under various adversarial attack scenarios showcase the superiority of the proposed ASGNN.
The numerical results corroborate that the adaptive structure property inherited in ASGNN can help mitigate the impact of perturbed graph structure.
P RELIMINARIES AND B ACKGROUND
An unweighted graph with self-loops is denoted as G =
( V , E ) , where V and E denote the node set and the edge set, respectively.
The graph adjacency matrix is given by A ∈ R N × N .
We denote by 1 and I the all-one column vector and the identity matrix, respectively.
Given D =
Diag ( A1 ) ∈ R N × N as the diagonal degree matrix, the Laplacian matrix is deﬁned as L = D − A .
We denote by A rw =
D − [-
-]1 A the random walk (or row-wise) normalized adjacency matrix and by A sym
= D − 12 AD − 12 the symmetric normalized adjacency matrix.
Subsequently, the random walk normalized and symmetric normalized Laplacian matrices are deﬁned as L rw
=
I − D − 1 A and L sym
=
I − D − 12 AD − 12 , respectively.
X ∈ R N × M ( M is assumed to be the dimension of the node feature) is a node feature matrix or a graph signal, and its i -th row X i, : represents the feature vector at the i -th node with i = 1 , . . .
, N .
X ij (or [ X ] ij ) denotes the ( i, j ) -th element of X with i, j = 1 , . . .
, N .
For vector X i, : , X − 1 i, : represents its element-wise inverse.
2. GNN S AS G RAPH S IGNAL D ENOISING
In the literature (Yang et al., 2021; Pan et al., 2021; Zhu et al., 2021), it has been realized that the message passing layers for feature learning in many GNN models could be uniformly interpreted as gradient steps for minimizing certain energy functions, which carries a meaning of GSD (Ma et al., 2021). Recently, Zhang and Zhao (2022) further showed that some popular GNNs are neural networks induced from unrolling (proximal) gradient descent algorithms for solving speciﬁc GSD problems.
Taking the approximate personalized propagation of neural predictions (APPNP) model (Klicpera et al., 2019) as an example, the initial node feature matrix Z is ﬁrst pre-propcessed by a multilayer perceptron g θ ( · ) with model parameter θ producing an output X
= g θ ( Z ) , and then X is fed into a K -layer message passing scheme given as follows:
[Equation]
where H (0) denotes the input feature of the message passing process, H ( k ) represents the learned feature after the k -th layer, and α is the teleport probability.
Therefore, the message passing of an
APPNP model is fully speciﬁed by two parameters, namely, a graph structure matrix A sym and a parameter α , in which A sym assumes to be known beforehand and α is treated as a hyperparameter.
From an optimization perspective, the message passing process in Eq. (1) can be seen as executing
K steps of gradient descent to solve a GSD problem with initialization H (0) =
X and step size 0 .(Zhu
et al., 2021; Ma et al., 2021; Zhang and Zhao, 2022), which is given by
[Equation]
where X and α are given and share the same meaning as in Eq.
In Problem (2), the ﬁrst term is a ﬁdelity term forcing the recovered graph signal H to be as close as possible to a noisy graph signal X , and the second term is the symmetric normalized Laplacian smoothing term measuring the variation of the graph signal H , which can be explicitly expressed as
[Equation]
For more technical discussions on relationships between GNNs with iterative optimization algorithms for solving GSD problems, please refer to Ma et al.
; Zhang and Zhao (2022).
Apart from using the lens of optimization to interpret existing GNN models, there are also literature (Liu et al., 2021b;
Chen et al., 2021; Fu et al., 2022) working on building new GNN architectures based on designing novel optimization problems and the corresponding iterative algorithms (more discussions are provided in Appendix A).
2. G RAPH L EARNING WITH S TRUCTURAL R EGULARIZERS
Structural regularizers are commonly adopted to promote certain desirable properties when learning a graph (Kalofolias, 2016; Pu et al., 2021).
In the following, we discuss several widely used graph structural regularizers which will be incorporated into the design of ASMP.
We denote the learnable graph adjacency matrix as S satisfying S ∈ S , where
[Equation]
deﬁnes the class of adjacency matrices.
Under the assumption that node feature changes smoothly between adjacent nodes (Ortega et al., 2018), the Laplacian smoothing regularization term is commonly considered in graph structure learning. Eq.
(3) is the symmetric normalized Laplacian smoothing term, and a random walk normalized alternative can be similarly deﬁned by replacing L sym in Eq. (3) by L rw .
Real-world graphs are normally sparsely connected, which can be represented by sparse adjacency matrices.
Moreover, it is also observed that singular values of these adjacency matrices are commonly small (Zhou et al., 2013; Kumar et al., 2020).
However, a noisy adjacency matrix (e.g., one perturbed by adversarial attacks) tends to be dense and to gain singular values in larger magnitudes (Jin et al., 2020).
In view of this, graph structural regularizers for promoting sparsity and/or suppressing the singular values are widely adopted in the literature of graph learning (Kalofolias, 2016; Egilmez et al., 2017; Dong et al., 2019).
Speciﬁcally, the (cid:96) 1 -norm of the adjacency matrix is often used to promote sparsity, deﬁned as (cid:107) S (cid:107) 1 = (cid:80) Ni,j =1 | S ij | .
For penalizing the singular values, the (cid:96) 1 -norm and the (cid:96) 2 -norm on the singular value vector of the adjacency matrix S can help.
Equivalently, they can be translated to be the nuclear norm and the Frobenius norm on S , which are given by (cid:107) S (cid:107) ∗ =
(cid:80) Ni =
1 σ i ( S ) and (cid:107) S (cid:107) F = (cid:113)(cid:80) Ni =1 σ 2 i ( S ) , respectively, where σ 1 ( S ) ≥ · · · ≥ σ N ( S ) denote the ordered singular values of S .
These two regularizers both restrict the scale of the singular values while the nuclear norm also promotes low-rankness.
A recent study (Deng et al., 2022) points out that graph learning methods with low-rank promoting regularizers may lose a wide range of spectrum of the clean graph corresponding to important structure in the spatial domain.
Thus, the nuclear norm regularizer may impair the quality of the reconstructed graph and therefore limit the performance of GNNs.
Besides, the nuclear norm is not amicable for back-propagation and incurs high computational complexity (Luo et al., 2021).
Arguably, the Frobenius norm of S is a more suitable regularizer for graph structure learning in comparison with the nuclear norm.
T HE P ROPOSED G RAPH N EURAL N ETWORKS
In this section, we ﬁrst motivate the design principle based on jointly node feature learning and graph structure learning.
Then, we develop an efﬁcient optimization algorithm for solving this optimization problem, which eventually leads to a novel message passing scheme with adaptive structure (ASMP).
After that, we provide interpretations, convergence guarantees, and speciﬁcations of ASMP.
Finally, integrating ASMP with deep neural networks ends up with a new family of GNNs with adaptive structure, named ASGNNs.
3. A N OVEL D ESIGN P RINCIPLE WITH A DAPTIVE G RAPH S TRUCTURE
As discussed in Section 2.1, the message passing procedure in many popular GNNs can be viewed as performing graph signal denoising (or node feature learning)
(Zhu et al., 2021; Ma et al., 2021; Pan et al., 2021; Zhang and Zhao, 2022) over a preﬁxed graph.
Unfortunately, if some edges in the graph are task-irrelevant or even maliciously manipulated, the node feature learned may not be appropriate for the downstream tasks.
Motivated by this, we propose a new design principle for message passing, that is, to learn the node feature and the graph structure simultaneously. It enables learning an adaptive graph structure from the feature for the message passing procedure.
Hence, such a message passing scheme can potentially improve robustness against noisy input graph structure.
Speciﬁcally, we construct an optimization objective by augmenting the GSD objective in Eq. (2) (we have used a random walk normalized graph Laplacian smoothing term) with a structure ﬁdelity term (cid:107) S − A (cid:107) 2F , where A is the given initial graph adjacency matrix, and the structural regularizers (cid:107) S (cid:107)and (cid:107) S (cid:107) 2F .
Then we obtain the following optimization problem: minimize
[Equation]
[Figure]
[Equation]
∈S
[Equation] [Table] [Equation] where H is the feature variable, S is the structure variable, and γ , λ , µ 1 , and µ 2 are parameters balancing different terms.
To enable the interplay between feature learning and structure learning, the Laplacian smoothing term is concerned with S rather than A , i.e., L rw =
I − D − 1 S with
D = Diag ( S1 ) .
When adversarial attacks exist, a perturbed adjacency matrix A will be generated.
Since attacks are generally designed to be unnoticeable (Jin et al., 2021), the perturbed graph adjacency matrix is largely similar to the original graph matrix in value.
In view of this, we also [Figure] include a structural ﬁdelity term (cid:107) S − A (cid:107) 2F .
The motivation for introducing the last two regularizers has been elaborated in Section 2.2.
3. ASMP: M ESSAGE P ASSING WITH A DAPTIVE S TRUCTURE
Following the idea that the message passing of a GNN model can be derived based on the optimization of a GSD objective function (Ma et al., 2021; Zhang and Zhao, 2022), we can obtain a message passing scheme from Problem (4).
Different from the existing GSD problems for GNN model design with only the feature variable, Problem (4) is nonconvex and much more challenging.
To obtain an efﬁcient iterative algorithm that is friendly to back-propagation training, we propose to use the alternating (proximal) gradient descent method (Parikh and Boyd, 2014), i.e., alternatingly optimizing one variable by taking one (proximal) gradient step at a time with the other variable ﬁxed.
(Note that a joint optimization approach is also eligible, while it would lead to slower convergence than the alternating optimization approach. More details can be found in Appendix D.)
We denote by H ( k ) and S ( k ) the variables at the k -th iteration ( k = 0 , . . .
, K ).
In the following, the update rules for H and S will be discussed, respectively.
Updating node feature matrix H :
Given { H ( k ) , S ( k ) } , the subproblem with respect to feature matrix H is given by minimize
[Equation]
where L ( k ) rw
=
I − Diag (cid:0) S ( k ) 1 (cid:1) − 1 S ( k ) .
One gradient step for H is computed as
[Equation]
where η 1 denotes the step size.
Updating graph structure matrix S :
Given { H ( k +1) , S ( k ) } and Tr( H ( k +1) (cid:62) L rw H ( k +1) ) =
Tr( H ( k +1) (cid:62)
[Equation] k +1) ) [Equation] [Equation]
Diag [Equation]
[Equation]
( k +1) ) , the subproblem for S becomes minimize S ∈S
[Equation]
Due to the non-smoothness of the objective function, we apply one step of the proximal gradient descent (Parikh and Boyd, 2014) for this problem.
Deﬁne
[Equation]
One step of proximal gradient descent is given as follows (details are given in Appendix B):
[Equation]
where η 2 is the step size and I S ( S ) denotes the indicator function taking value
if S ∈ S and + ∞ otherwise.
Moreover, the proximal operator in Eq. (7) can be computed analytically as
[Equation]
where ReLU( X ) = max { 0 , X } .
In conclusion, the overall procedure of ASMP can be summarized as follows:
[Equation]
The ASMP can be interpreted as the standard message passing (i.e., the update step of H ) with extra operations that adaptively adjust the graph structure (i.e., the update step of S ).
Therefore, an edge of the graph included in some layers may be excluded or down-weighted in other layers.
A pictorial illustration of the ASMP procedure is provided in Figure 1.
A K -layer ASMP can be fully speciﬁed by parameters γ , λ , µ 1 , µ 2 , η 1 , and η 2 , which we generally denote as ASMP K (cid:0) X , A , γ, λ, µ 1 , µ 2 , η 1 , η 2 (cid:1) .
Note that ASMP is general enough to cover several existing propagation rules as special cases.
Remark 1 (Special cases) .
If we use a ﬁxed graph structure S (0) = · · · = S ( K )
=
A in ASMP, i.e., µ 1 = µ 2 = γ = 0 , the ASMP reduces to a classical message passing procedure that only performs feature learning.
Speciﬁcally, with η 1 = 1 2+2 λ and the symmetric normalized adjacency matrix, ASMP can be written as
[Equation]
Case I: when λ
=
α − 1 , the operation in Eq. (8) becomes the message passing rule of APPNP (Klicpera et al., 2019):
[Equation]
Case II: when λ = ∞ , the operation in Eq. (8) becomes the simple aggregation in many GNN models such as the GCN model (Kipf and Welling, 2017) and the simple graph convolution (SGC) model (Wu et al., 2019a):
[Equation]
Instead of updating both S and H once, we can also choose to update them for several steps.
The convergence of ASMP is guaranteed with proper selections of the step sizes as demonstrated in
Theorem 4.
Before proceeding to the convergence result, we ﬁrst introduce some standard assumptions on the node feature vectors and the degree matrices, which are widely adopted in the literature (Garg et al., 2020; Liao et al., 2021; Cong et al., 2021). Assumption 2. The energy of the node feature is uniformly upperbounded, i.e., || H ( k ) i, : || 2 ≤ B for i = 1 , . .
. , N and k = 0 , . . , K . Assumption 3. The diagonal elements of the degree matrix is lowerbounded by a positive constant, i.e., min i D ii = c > 0 for i = 1 , . . .
, N . Theorem 4.
Let H (0) =
X and S (0)
=
A . Under Assumption 2 and Assumption 3, the sequence { H ( k ) , S ( k ) } Kk =1 generated by (ASMP) with 0 < η 1 < 1 1+2 λ and 0 <
η 2 <γ + µ 2 +(1+ 1 c N √ N ) λc 2 N 2 B 2 converges to a ﬁrst-order stationary point of Problem (4) denoted as { H ∗ , S ∗ } with rate
[Equation]
where ρ is a constant depending on the step sizes and Lipschitz constants, and p ( H , S ) represents the objective of Problem (4) .
Proof.
The proof for Theorem 4 is in Appendix C. Note that if multiple updating steps are used for S and H in ASMP, this convergence result still holds (Bolte et al., 2014; Nikolova and Tan, 2017).
3. ASGNN: G RAPH N EURAL N ETWORKS WITH A DAPTIVE S TRUCTURE
In this section, we introduce a family of GNNs leveraging the ASMP scheme.
Integrating (ASMP) with a machine learning model g θ ( · ) (e.g., a multilayer perceptron) with H (0) = X
= g θ ( Z ) , a K -layer ASGNN model is deﬁned as follows:
[Equation]
In ASGNN, we have chosen a decoupled architecture similar to APPNP (Klicpera et al., 2019) and deep adaptive GNN (DAGNN) (Liu et al., 2021a).
Specially, in ASGNN, the model g θ will ﬁrst transform the initial node feature as X =
g θ ( Z ) , and then ASMP performs K steps of message passing with input g θ ( Z ) .
The parameters in ASMP, i.e., γ, λ, µ 1 , and µ 2 , are set to be weights to be learned from the downstream tasks.
For example, in semi-supervised node classiﬁcation tasks, the loss function is chosen as the cross-entropy classiﬁcation loss on the labeled nodes and the whole model is trained in an end-to-end way.
Since ASMP is derived from the alternating (proximal) gradient descent algorithm, a trained ASMP is naturally a parameter-optimized iterative algorithm.
The step sizes ηand η 2 in ASMP can be chosen according to the results in Theorem 4.
However, such choices seem to be too conservative in practice and may lead to slow convergence.
Thus, we may also consider the step sizes η 1 and η 2 as learnable parameters.
Convergence property of ASMP with learned step sizes will be showcased in the experiments.
In conclusion, there are in total six parameters in ASMP considered during the learning process.
In this paper, we have focused on problems in which there is an initial graph structure, while the use of ASGNN may also be extended to scenarios where the initial structure is not available.
In such case, we can ﬁrst create a k -nearest neighbor graph or use some optimization methods (Dong et al., 2016; Kalofolias, 2016; Kumar et al., 2020) to learn a graph structure based on the node feature.
Such extensions of ASGNN can be promising future research directions.
E XPERIMENTS
In this section, we conduct experiments to validate the effectiveness of the proposed ASGNN model.
First, we introduce the experimental settings.
Then, we assess the performance of ASGNN on semisupervised node classiﬁcations tasks and investigate the beneﬁts of introducing adaptive structure into GNNs against global attacks and targeted attacks.
Finally, we analyze the structure denoising ability and the convergence property of ASMP with the learned step sizes.
4. E XPERIMENT S ETTINGS
Datasets : We perform numerical experiments on 4 real-world citation graphs, i.e., Cora (Sen et al., 2008),
Citeseer (Sen et al., 2008), Cora-ML (Bojchevski and Günnemann, 2018), and ACM (Wang et al., 2019), and only consider the largest connected component in each dataset.
Baselines : To evaluate the effectiveness of ASGNN, we compare it with GCN and several benchmarks that are designed from different perspectives to robustify the GNNs, including GCN-Jaccard (Wu et al., 2019b) that pre-processes the graph by eliminating edges with low Jaccard similarity of node feature vectors, GCN-SVD (Entezari et al., 2020) that applies the low-rank approximation of the given graph adjacency matrix, GNNGuard (Zhang and Zitnik, 2020) that deﬁnes an attention mechanism based on the similarity of neighboring nodes, Pro-GNN (Jin et al., 2020) that jointly learns a graph structure and a GNN model guided by some predeﬁned structural priors, and Elastic GNN (Liu et al., 2021b) that utilizes trend ﬁltering instead of Laplacian smoothing to promote robustness.
The code is implemented based on PyTorch Geometric (Fey and Lenssen, 2019).
For GCN-Jaccard, GCN-SVD, and Pro-GNN, we use the implementation provided in DeepRobust (Li et al., 2020).
For GNNGuard and Elastic GNN, we follow the implementation provided in the original papers (Zhang and Zitnik, 2020; Liu et al., 2021b).
Parameter settings : For all the experimental results, we give the average performance and standard variance with 10 independent trials.
For each graph, we randomly select 10%/10%/80% of nodes for training, validation, and testing.
The Adam optimizer is used in all experiments.
The models’ hyperparameters are tuned based on the results of the validation set.
The search space of hyperparameters [Table] [Table]
are as follows: 1) learning rate: {0.005, 0.01, 0.05}; 2) weight decay: {0, 5e-5, 5e-4}; 3) dropout rate: {0.1, 0.5, 0.8}; 4) model depth: {2, 4, 8, 16}.
For GCN-Jaccard, the threshold of Jaccard similarity for removing dissimilar edges is chosen from {0.01, 0.02, 0.03, 0.04, 0.05, 0.1}.
For GCN-SVD, the reduced rank of the graph is tuned from {5, 10, 15, 50, 100, 200}.
For Elastic GNN, the regularization coefﬁcients are chosen from {3, 6, 9}.
For Pro-GNN, we adopt the hyperparameters provided in their paper (Jin et al., 2020).
4. P ERFORMANCE U NDER A DVERSARIAL A TTACK
The performance of the compared models is evaluated under the training-time adversarial attacks (Wang and Gong, 2019; Zügner and Günnemann, 2019), i.e., the graph is ﬁrst attacked, and then the
GNN models are trained on the perturbed graph.
In the following, we conduct experiments under both the global attack and the targeted attack.
Speciﬁcally, the global attack aims to reduce the overall performance of GNNs (Zügner and Günnemann, 2019) while the targeted attack aims to fool GNNs on some speciﬁc nodes (Zügner et al., 2018).
Global Attack : We ﬁrst test the node classiﬁcation performance of ASGNN and other baselines under global attack using a representative global attack method called meta-attack (Zügner and Günnemann, 2019).
We vary the perturbation rate, i.e., the ratio of changed edges, from 0% to 25% with an increasing step of 5%.
The results are reported in Table 1.
From the table, we observe that the proposed ASGNN model outperforms other methods in most cases.
For instance, ASGNN improves GCN over 30% on the Cora-ML dataset at a 20% perturbation rate and over 20% on the Cora dataset at a 25% perturbation rate.
On Cora, Citeseer, and ACM datasets, ASGNN beats other baselines at various perturbation rates by a large margin.
The GCN-Jaccard method and the GNNGuard slightly outperforms ASGNN on the Cora-ML dataset at a 15%-25% perturbation rate, while they performs poorly on other datasets.
Speciﬁcally, on the other three datasets under the 25% perturbation rate,
ASGNN outperforms GCN-Jaccard by 22%, 10%, and 10%, respectively.
Such inspiring results demonstrate that ASGNN can better resist global attack than other baseline methods.
Targeted Attack : For the targeted attack, we use a representative method called NETTACK (Zügner et al., 2018).
Following existing works (Zhu et al., 2019; Jin et al., 2020), we vary the perturbation number made on every node, i.e., the number of edge removals/additions, from 0 to 5 with an increasing step of 1.
The results are reported in Table 2.
We choose the nodes in the test set with degrees larger than 10 as targeted nodes and the reported classiﬁcation performance is evaluated on target nodes.
Thus, the results in Table 2 is not directly comparable with the results in Table 1.
From the table, we can see that the proposed ASGNN attains better performance than other baselines in most cases.
For instance, on the Citeseer dataset with 5 perturbations per targeted node, ASGNN
[Table]
improves GCN by 25% and outperforms other baselines by over 5%.
The reported results demonstrate that ASGNN can also effectively resist the targeted attack.
4. R OBUSTNESS OF ASGNN
The message passing scheme in ASGNN is designed based on jointly node feature learning and graph structure learning principle.
To validate that ASGNN can help purify (i.e., denoise) the structure, we evaluate the quality of the graphs generated by ASGNN via evaluating the performance of a GCN model trained on the generated graphs in the last layers of ASGNN (which we named as
GCN-AS). Under the assumption that the GCN trained on a purer graph will give better performance, the performance of GCN-AS can indicate the quality of the learned graph in ASGNN. We conduct experiments on different datasets under targeted attack with ﬁve perturbations per node and the results are in Table 3.
It can be seen that the performance of GCN-AS is much better than GCN.
Moreover, GCN-AS even outperforms other defense models on some datasets, e.g., GCN-AS outperforms all other baselines on ACM dataset.
These inspiring results indicate that ASGNN can mitigate the inﬂuence of adversarial attacks. [Table]
We provide some additional experiments in Appendix E, including the runtime analysis in Appendix E.1, the discussion of the leaned coeeﬁcients in ASGNN in Appendix E.2, the sparsity level of the graphs generated in ASGNN in Appendix E.3, and the convergence analysis of ASMP with learned step sizes in Appendix E.4.
C ONCLUSION
In this work, we have developed an interpretable robust message passing scheme named ASMP following the jointly node feature learning and graph structure learning principle.
ASMP is provably convergent and it has a clear interpretation as a standard message passing scheme with adaptive structure.
Integrating ASMP with neural network components, we have obtained a family of robust graph neural networks with adaptive structure.
Extensive experiments on real-world datasets with various adversarial attack settings corroborate the effectiveness and the robustness of the proposed graph neural network architecture.
A R ELATED W ORK ON O PTIMIZATION -I NDUCED G RAPH N EURAL N ETWORK D ESIGN
Since ASGNN proposed in this paper is induced from an optimization algorithm, in this section, we give more literatue review on optimization-induced GNN model design to supplement our discussion. The idea of optimization-induced GNN model design partly stems from the observation that many primitive handcrafted GNN models could be nicely interpreted as (unrolled) iterative algorithms for solving a GSD optimization problem (Ma et al., 2021; Zhu et al., 2021; Zhang and Zhao, 2022). Based on this observation, many papers aim at strengthening the capability of GNNs by carefully designing the underlying optimization problems and/or the iterative algorithms solving it.
For example, inspired by the idea of trend ﬁltering (Wang et al., 2015), Liu et al.
(2021b) replace the
Laplacian smoothing term (which is in the form of (cid:96) 2 -norm) in the GSD problem with an (cid:96) 2 , 1 -norm to promote robustness against abnormal edges.
Also for robustness pursuit, Yang et al.
(2021) replace the Laplacian smoothing term with nonlinear functions imposed over pairwise node distances.
Since the classical Laplacian smoothing term in GSD only promotes smoothness over connected nodes, the authors in Zhang et al.
Zhao and Akoglu (2020) further suggest promoting the non-smoothness over the disconnected nodes, which is achieved by deducting the sum of distances between disconnected pairs of nodes from the denoising objective.
In Jiang et al.
(2022), the authors augment the GSD objective with a fairness term to ﬁght against large topology bias.
Most recently, Fu et al.
(2022) propose a p -Laplacian message passing scheme and a p GNN model, which is capable of dealing with heterophilic graphs and is robust to adversarial perturbations.
Apart from that, Ahn et al.
(2022) designs a novel regularization term to build heterogeneous GNNs.
Although there is rich literature on optimization-induced GNN model design, all of them are focusing on learning the node feature matrix. The idea of this paper is similar to them in terms of the GNN design philosophy, however, we design an objective to jointly learn the node feature and the graph structure which was rarely covered in the literature.
B D ERIVATION OF T HE P ROXIMAL G RADIENT S TEP IN E Q . (7)
For the S -block optimization, i.e., Eq. (6), we deﬁne the objective function except the term µ 1 (cid:107) S (cid:107)as f S ( S ) , i.e., [Equation] where D = Diag ( S1 ) .
In this section, we ﬁrst derive the expression of ∇ f S ( S ) and then compute the proximal operator in Eq. B. O N C OMPUTATION OF ∇ f S ( S )
We ﬁrst focus on the gradient computation of the second term in Eq. For the graph degree matrix, we have
[Equation]
where (cid:12) denotes the Hadamard product.
Based on the rule of matrix calculus, the differential of the scalar function Tr( H (cid:62) D − 1 SH ) with respect to matrix variable S can be computed as follows:
[Equation]
For an invertible D (note that, in this paper, the graphs considered all have self loops, so D is always invertible), we have
[Equation]
− D − 1 d ( D ) D − 1 .
Thus, we can get d
[Equation]
SH
[Equation]
=
[Equation]
[Equation]
=
[Equation]
Since d (cid:0) Tr( H (cid:62) D − 1 SH ) (cid:1) = Tr (cid:16)(cid:0) dTr( H (cid:62) D
[Equation]
[Equation]
S (cid:1) (cid:62) d
[Equation]
S ) [-denotes the indicator function taking value 
 if S ∈ S and + ∞ otherwise. The proximal operator above can be computed analytically as follows:-](cid:17) , we have
[Equation]
For other terms in f S ( S ) , the gradients with respect to S can be easily computed. Finally, we obtain
[Equation]
Lemma 5. Given a matrix M ∈ R N × N , we have
[Equation] where ReLU( X ) = max { 0 , X } . Proof.
The proximal step in Eq.
(10) can be rewritten as the following optimization problem:
[Equation] [Equation] It is easy to observe that Problem (11) is decoupled over different elements in matrix S . Therefore, each S ij with i, j = 1 , . . .
, N can be optimized individually by solving the following optimization problem:
[Equation]
According to Eq. (10), we have [Equation] Then, Lemma 5 can be proved by showing that for i, j = 1 , . . .
N , S (cid:63)ij is the optimal solution for Problem (12). The optimality of S (cid:63)ij can be validated by verifying the optimality condition, i.e., there exists a subgradient ψ ∈ ∂h ij (cid:0) S (cid:63)ij (cid:1) such that
[Equation]
for all 0 ≤ S ij ≤ 1 .
Observe that the subdifferential of h ij ( S ij ) is computed as follows:
[Equation]
where (cid:15) can be any constant satisfying − 1 ≤ (cid:15) ≤ 1 . Then, the subdifferential ∂h (cid:0) S (cid:63)ij (cid:1) is given by
[Equation]
In the following, we will show that the optimality condition holds for each of the above cases.
For 1 + κ ≤ M ij , we have S (cid:63)ij = 1 and ψ = 1 − M ij + κ ≤ 0 .
Since S ij ≤ 1 , we can get ψ ( S ij − S (cid:63) ij ) ≥ 0 .
For κ ≤ M ij < 1 + κ , we have ψ = 0 and hence, ψ ( S ij − S (cid:63)ij )
=
for all 0 ≤ S ij ≤ 1 . 3.
For M ij < κ , we have S (cid:63)ij = 0 and ψ = − M ij + κ(cid:15) with (cid:15) being any constant satisfying − 1 ≤ (cid:15) ≤ 1 .
Thus, we can choose (cid:15) = 1 , which leads to ψ > 0 .
Since S ij ≥ 0 , we can get ψ ( S ij − S (cid:63) ij ) ≥ 0 .
In conclusion, there exists a subgradient ψ ∈ ∂h ij ( S (cid:63)ij ) such that ψ ( S ij − S (cid:63)ij ) ≥ 0 for all 0 ≤ S ij ≤ 1 , based on which the optimality of Eq. (10) is validated and the proof is completed.
Based on the result in Lemma 5, by choosing M = S ( k ) − η 2 ∇ f S ( S ( k ) ) and κ = η 2 µ 1 , we get the analytical expression for the proximal step in Eq. (7) as follows:
[Equation] which sufﬁces to perform the soft-thresholding operation and then project the solution onto the constraint set S .
C P ROOF OF T HEOREM 4 (C ONVERGENCE OF ASMP)
In this section, we will ﬁrst prove that the objective function at the H -block optimization problem and the smooth part of the objective function at the S -block optimization problem are L -smooth. Then we give the conditions to ensure convergence of ASMP.
Denote f H ( H ) as the objective function at the H -block optimization problem, i.e.,
[Equation]
The L -smoothness of f H ( H ) is demonstrated in the following lemma.
Lemma 6.
Function f H ( H ) is L -smooth with L H = 2 + 4 λ , i.e., for any H 1 , H 2 ∈ R N × M , the following inequality holds: [Equation] Proof.
First observe that
[Equation]
Lemma 7 (Chung, 1997) . The largest eigenvalue of a random walk normalized Laplacian matrix L rw is less than or equal to 2, i.e., || L rw || 2 ≤ 2 . Based on Lemma 7, we can conclude that
[Equation]
Therefore, function f H ( H ) is L -smooth with L H = [-
-]2 + 4 λ and the proof is completed With f S deﬁned in Eq. (9), the L -smoothness of f S is deomnstrated in the following lemma. Lemma 8.
Function f S ( S ) is L -smooth with L S = 2 γ + {+2 µ 
 + 2+} λ c 2 N 2 B 2 + 2 λ c 3 N 3 √ NB 2 , i.e., for any S 1 , S 2 ∈ R N × N , the following inequality holds: [Equation] Proof.
Denote D 1 = Diag ( S 1 1 ) and D 2 = Diag ( S 2 1 ) as two degree matrices corresponding to S 1 and S 2 .
We have
[Equation]
To derive the Lipschitz constant of ∇ f S ( S ) , we ﬁrst present several useful lemmas. Lemma 9. Under Assumption 2 that the norm of node feature vectors is upperbounded, i.e., (cid:13)(cid:13)
H i, : (cid:13)(cid:13) 2 ≤ B , we have [Equation]
Lemma 10. Given S ∈ S and D = Diag ( S1 ) , under Assumption 3 that the diagonal elements of D is lowerbounded by a positive constant, i.e., min i D ii = c > 0 for i = 1 , . . .
, N , we have
[Equation]
Lemma 11.
Given S 1 , S 2 ∈ S , D 1 =
Diag ( S 1 1 ) , and D 2 =
Diag ( S 2 1 ) , under Assumption 3 that the diagonal elements of the degree matrix is lowerbounded by a positive constant, i
[Equation]
e., min i D ii = c > 0 for i = 1 , . .
, N , we have
[Equation]
and
[Equation]
Proof.
It can be observed that
[Equation]
which proves Eq.
Based on Eq. [Equation]
we further have
[Equation]
through which the proof is completed.
Based on Lemma 9 and Lemma 11, the second term in Eq.
, i.e., λ (cid:13)(cid:13) HH [Equation] (cid:13)(cid:13) 2 (cid:13)(cid:13) D − 1 1 − D − 1 2 (cid:13)(cid:13) F , can be upperbounded as follows:
[Equation]
[Equation]
For the third term in Eq.
(13), we have
[Equation]
[Table]
Based on Lemma 9, Lemma 10, and Lemma 11, we can get the following result:
[Equation]
Substituting the results in Eq. (16) and Eq. (17) into Eq. (13) gives
[Equation]
Therefore, function f S ( S ) is L -smooth with L S = [Equation] + 2 µ [Equation]
+ [Equation] N 3 √ NB 2 and the proof is completed. Based on the results in Lemma 6 and Lemma 8, we can conclude that f H ( H ) and f S ( S ) are both
L -smooth. To ensure the monotonically decreasing property of (ASMP), the step sizes must satisfy (Parikh and Boyd, 2014): [Equation]
Under such condition, the convergence of (ASMP) to a ﬁrst-order stationary point of Problem (4) can be readily obtained based on the results for alternating proximal gradient descent method in Bolte et al. ; Nikolova and Tan (2017) with convergence rate [Equation] 18) where ρ = min (cid:8) 1 η 1 − L H 2 , 1 η 2 − L S 2 (cid:9) and p ( H , S ) represents the objective function in Eq.
D D ISCUSSION ON THE J OINT O PTIMIZATION A PPROACH
In this paper, we have used the alternating optimization approach to induce the ASMP scheme, while another natural idea is to apply a joint optimization approach for Problem (4).
In this section, we will show that the joint optimization approach actually is inferior compared to the alternating one, since joint optimization would lead to slower convergence, which motivates the use of alternating optimization in ASMP.
We deﬁne the smooth part of the objective in Problem (4) as
[Equation]
The L -smoothness of f ( H , S ) is deomnstrated in the following lemma. Lemma 12. Function f
( H , S ) is L -smooth with L = max (cid:110)(cid:113)
{+L 2 H + (1 + 1 c N √ N ) 2 4 λ 2 c 2 NB 2 , (cid:113) L 2 S + (1 + 1 c N 2 ) 2 
 λ 2 c 2 NB 2 (cid:111) , i+}
[Equation]
e., for any H 1 , H 2 ∈ R N × M and S 1 , S 2 ∈ R N × N , the following inequality holds: [Equation] Proof. Observe that
[Equation]
In the following, we will derive the upper bound for the third and the fourth term in Eq. Based on Lemma 11, the term (cid:107)∇ H f ( H 1 , S 1 ) − ∇ H f ( H 1 , S 2 ) (cid:107) F can be upperbounded as follows:
[Equation] Besides [Equation] for the forth term
[Equation]
we have [Equation] [Table] [Equation]
According to Lemma
[Equation]
we have [Equation] Then
[Equation] we can get [Equation] [Equation]
Substituting the results in Eq. (21) and Eq.
(22) into Eq.
(19) gives
[Equation]
Thus, function f ( H , S ) is L -smooth with
[Equation]
through which the proof is completed.
This result in Lemma 12 indicates that the Lipschitz constant L is larger than both L H and L S .
Practical graphs are commonly large graphs; i.e., the number of nodes N would dominate the other constants in L , i.e., c , λ , and B . Therefore, if we use the joint optimization approach, the Lipschitz constant is larger than L H and L S by a large margin.
After deriving the Lipschitz constant in joint optimization, we compare its convergence rate with the alternating optimization approach.
Denote { H ( k ) , S ( k ) {+} Kk =
+} as the sequence generated by the above joint optimization approach.
Following the results in Bolte et al.
; Nikolova and Tan (2017), the convergence property of the joint optimization approach is also given in Eq.
(18) with ρ = 1 η − L 2 .
Theoretically, to guarantee the sufﬁcient descent of the objective at each step, ρ can be chosen to be min { 1 η 1 − L H 2 , 1 η 2 − L S 2 } in the alternating optimization approach.
Due to the fact that L > max { L H , L S } according to Eq. (23), the alternating optimization approach is allowed to adopt a larger step size at each block than the joint optimization approach, resulting in a faster convergence behavior of the sequence.
Motivated by this fact, we develop ASMP based on the alternating procedure rather than the joint one so that the resulting message passing structure contains fewer layers to achieve the similar or even better numerical performance compared to the joint one.
E A DDITIONAL E XPERIMENTS E. R UNTIME A NALYSIS
The computation complexity of ASMP is in order O (cid:0) N 2 d (cid:1) , where N is the number of nodes and d is the feature dimension. This is larger than the complexity of simple graph convolution, which is in order O ( Ed ) with E being the number of edges. To better understand the complexity of ASGNN and other baselines, we provide a runtime comparison of different models.
Speciﬁcally, we train different models for 200 epochs on Cora dataset under nettack with one perturbation per target node and the averaged runtimes from ten trials are shown in the following table.
[Table]
From the table [Table] we can see that although ASGNN requires more computation time than GCN, GCN-Jaccard
[Table] and GCN-SVD [Table] it is much faster than Pro-GNN and the complexity of ASGNN is comparable to GNNGuard and Elastic GNN.
To better understand how different terms in the joitn node feature learning and graph structure learning objective [Table] we provide the learned ASGNN coefﬁcients on Cora dataset under targeted attack below.
[Table]
From the table, we can see that the value of λ is much larger than other coefﬁcients. Besides, the value of the coefﬁcients do not vary much under different perturbation numbers, indicating the importance of different terms are not affected by the perturbation number. E. S PARSITY L EVEL OF THE L EARNED G RAPHS IN ASGNN
The sparsity level of the graphs (percentage of zero elements in the learned adjacency matrix) generated in the last layer of ASGNN are provided below. Note that the sparsity level can be tuned by constraining the learning space of µ 1 . For example, we can ﬁx µ 1 to be a large value, which will lead to sparser graphs. [Table]
C ONVERGENCE P ROPERTY OF ASMP IN P RACTICE
According to Theorem 4, the convergence of ASMP is guaranteed with proper choices of step sizes.
Since the step sizes are set to be learnable parameters, we provide an additional experiment to evaluate the convergence of ASMP with learned step sizes empirically.
[Figure]
To evaluate the convergence property of ASMP with learned step sizes, we conduct experiments on
Cora, Citeseer, and Cora-ML datasets at a 25% perturbation rate under meta-attack.
Specially, we train a 4-layer ASGNN model and observe that the learned step sizes do not satisfy the condition in Theorem 4.
In the following, we investigate the empirical convergence property of ASMP.
Since we use a recurrent structure in ASGNN, i.e., the step sizes used in different layers are the same, we are able to extend the trained 4-layer ASGNN model to a deeper one.
The values of the objective function in Problem (4) in different layers are showcased in Figure 2, where the objective values are normalized by dividing the objective value in the ﬁrst layer.
From Figure 2, we can ﬁnd that ASMP with learned step sizes can monotonically decrease the objective function value during the message passing process.
Note that although the monotonic decreasing property does not hold in 16-18 layers on the Cora-ML dataset, this may be mainly due to the fact that the step sizes are learned only based on a 4-layer model.
The results indicate that although the learned step sizes do not satisfy the condition in Theorem 4, they still ensure the monotonic decrease of the objective function value in practice.