Certifying Robust Graph Classification under Orthogonal Gromov-Wasserstein Threats
Anonymous Author(s)
AffiliationAddress email
Abstract
Graph classifiers are vulnerable to topological attacks.
Although certificates ofrobustness have been recently developed, their threat model only counts local andglobal edge perturbations, which effectively ignores important graph structuressuch as isomorphism.
To address this issue, we propose measuring the perturbationwith the orthogonal Gromov-Wasserstein discrepancy, and building its Fenchelbiconjugate to facilitate convex optimization.
Our key insight is drawn from thematching loss whose root connects two variables via a monotone operator and ityields a tight outer convex approximation for resistance distance on graph nodes.
When applied to graph classification by graph convolutional networks, both ourcertificate and attack algorithm are demonstrated effective.
Introduction
State-of-the-art graph classifiers such as graph convolutional networks (GCNs) have been recentlyrevealed vulnerable to adversarial structural attacks, namely adding and removing edges [1, 2].
Dueto the hardness of finding the strongest attack, an alternative strategy aims to certify robustness,i.e. , proving that no perturbation in the threat model can successfully attack the classifier.
To thisend, [3, 4] extended randomized smoothing to discrete noises with applications in, e.g. , communitydetection.
Although our focus is on graph classification, certificates have been studied for nodeclassification using, e.g. , convex outer adversarial polytope [5, 6].
Most relevant to our method is [7],which constructed the tightest lower bound of the margin via Fenchel biconjugation.
However, all existing threat models only count the total edges removed or added (global budget) or thechange of each node’s degree (local budget), while overlooking the important notion of isomorphism .
In contrast, there has been a wealth of more refined discrepancy measures between graphs, includingkernels [8, 9] and GCNs [10, 11].
Recently, the Gromov-Wasserstein discrepancy [ GW , 12], whichextends the Gromov-Wasserstein distance [13], has emerged as an effective transportation distancebetween structured data, alleviating the incomparability issue between different structures by aligningthe intra -relational geometries.
Thanks to its favorable properties such as efficiency and awareness ofisomorphism, GW has been extensively applied to domain adaptation [14], word embedding [15],graph classification [16], metric alignment [17], generative modeling [18], and graph matching andnode embedding [19, 20].
Therefore, it is clearly natural to adopt such a measure in the certificationof graph robustness.
Analogously, Wasserstein distance has already been employed in distributionalrobustness [21] and in attacking images [22], where perturbations can be found that better reflect theimage content compared with the standard ℓ p attacks.
Unfortunately, GW is not tractable to evaluate, differing from the standard Wasserstein distancewhich is a linear program.
So all existing optimization techniques settle for local solutions, lackingan analyzable guarantee.
To develop certificates, tractable lower bounds of GW would be essential,but the Kantorovich dual no longer exists and the existing lower bounds [13, 23] are not tight [24].
Further, underlying the standard probability coupling is the metrics within each graph, e.g. , shortestpath distance [13, SP distance], which requires nontrivial discrete optimization.
This significantlycomplicates the entire certificate as the subject of perturbation is exactly the topological structure.
Inaddition, certificates based on randomized smoothing [4, 25], which adds noise to the input, are alsohindered by the significant challenge in designing the noise distribution that aligns with GW .
The goal of this work, therefore, is to construct the first robustness certificate for GW -style threatmodels, with applications in graph classification.
This is achieved by casting the entire computationprocedure of GW as a two-layer model, each of which is subsequently relaxed in a tractable fashion:graph structure
[Figure]
[Table]
[Figure]
[Table]
discrepancy btw graphs .
Firstly, we circumvent the discrete optimization in SP distance by resorting to the resistance distance[26], which, along with its scaled version known as commute time, is one of the most commonlyused distances in computer science, machine learning, and beyond [27].
Compared with SP distance,it admits a closed-form based on matrix inversion, which is a monotone operator on the positivesemi-definite cone.
This enables our first contribution, where a convex relaxation is designed forresistance distance in Section 3 based on the matching loss [28].
Our second contribution achieves convex relaxation of the GW -style discrepancy measure.
Ourseed inspiration is drawn from the latest orthogonal GW discrepancy ( OGW ) [24] reviewed in
Section 2.2—it preserves most of the desirable properties of GW while admitting a tight and efficientlower bound.
We next make the key observation in Section 2.3 that the lower bound, despite notconvex, admits a closed-form Fenchel dual, which in turn facilitates an efficient evaluation of the
Fenchel biconjugate.
Such a convex lower bound is applied to graph convolution networks in
Section 4, where both attacking algorithm and robustness certificates are developed.
Our experiments in Section 5 verify the effectiveness of our attacker and certificate, in that a largeproportion of the graphs can be proved either vulnerable or robust.
Related Work Certification or verification has been a longstanding pursuit in machine learning andcyber-physical systems.
Formally, it inquires if f ( x ) can be driven down to negative over x ∈ X ,where both f and X can be too complicated for global optimization.
So a lower bound of the optimalobjective is sought, which provides incomplete verification (some true properties are confirmed true).
A lower bound can also be useful in branch-and-bound algorithms [6].
A natural approach is the
Lagrange dual [29, 30], which minorizes the primal by the weak duality.
However, it can be loose,and a provably tighter lower bound is the convex envelope or Fenchel biconjugate [7].
But theythemselves are efficiently computable only under limited scenarios.
In general, the objective f is composed of several “layers”, i.e. , f = f 0 ◦ f 1 ◦ . .
, where f i is simplerbut no longer scalar-valued ( i ≥ 1 ).
So it is natural to relax their graphs.
For example, min f 0 ( f 1 ( x ))is equivalent to min x,y f 0 ( y ) s.t.
y
=
f 1 ( x ) .
So we can possibly use the convex envelop of f 0 (whichis simpler than f ) and employ the convex hull of { ( x, y ) :
y
= f 1 ( x ) } .
The latter is represented bythe convex envelope relaxation of the ReLU activation [31–35], which has also found applications ingraph neural networks [6, 36].
However, they have been so far limited to R → R nonlinear functionswhile a multivariate extension is not available yet, as the convex hull is much harder to compute.
Another commonly used approach is randomized smoothing, which adds noise to the input [25, 37–39].
Although it has also been shown effective in graph robustness [3, 4], the design of discretenoise can be quite intricate and so far no such methods exist for GW -style threat models.
Othermethods include semi-definite relaxation [40, 41] which is often loose, Lipschitz continuity analysis[42–44] where the local and global estimation of curvature is difficult in the discrete domain, andreformulation linearization technique [45] used by [5] for directed but not undirected graphs.
Gromov-Wasserstein Threats for Attacking Graph Classifiers
We consider attacking a graph classifier where the edge connectivity is perturbed.
Here both theoriginal graph D and the resulting graph C are assumed undirected , unweighted , and connected .
Letthe adjacency matrix of D be A ∈ { 0 , 1 } n × n , where A ii = 0 , and A ij ∈ { 0 , 1 } ( i ̸ = j ) indicatingwhether an edge exists between nodes i and j .
Here n is the order of the graph, i.e. , number of nodes.
Now suppose the graph’s edge connectivity is perturbed, leading to a new adjacency matrix ˜
A :=
A + X (corresponding to C ).
Obviously, X needs to be symmetric, zero diagonal, and X ij ∈
{− A ij , 1 − A ij } .
If we relax it to the continuous domain such as convex relaxation, X ij is naturallyrelaxed to [ − A ij , 1 − A ij ] .
The threat model governs additional budgets on X besides the aboveadmissible conditions.
Global change budget can be written as 1 ⊤ ( X ◦ S ) 1 ≤ δ g , where S = 1 − 2 Aand ◦ stands for the Hadamard product.
Local budget is ( X ◦ S ) 1 ≤ δ l 1 (elementwise).
If we onlyallow adding edges, then X ij ≥ 0 .
All these basic constraints are linear in X , and we will collectivelyrefer to them as X ∈ X , a convex set.
Suppose we have trained a graph classifier G ( e.g. , graph convolution network, GCN) which maps anadjacency matrix A to a K -dimensional logit vector G ( A ) corresponding to K classes.
Assume thetrue class is y and G predicts correctly, i.e. , y = arg max c G c
( A ) .
Then the margin of classification is
[Equation]
The attacker seeks a feasible perturbation X ∈ X that drives the classification margin to negative, i.e. ,a misclassification.
G is called robust on this graph if min X ∈X M ( A + X ) is positive, and is calledvulnerable if the minimal value is negative.
It is noteworthy that we employ graph classification and
GCN only as an example context of applying our proposed relaxation technique.
Other tasks such asnode classification are readily applicable too – simply replace the M function.
Although the global and local budgets are natural and reasonable characterizations of the perturbation,they ignore an important notion on graphs: isomorphism.
Exact isomorphism can be relaxed bydefining a distance between two graphs, which quantify their difference under the most favorablepermutation of nodes.
One of the most commonly used distance is the Gromov-Wasserstein distance[13], which has been extended to Gromov-Wasserstein discrepancy [ GW , 12].
So in addition to thestandard local and global budgets specified by X , it is natural to further constrain the perturbationin terms of the GW distance.
To conclude, the attack and certification tasks can be formalized asmin
X ∈X M ( A + X ) s.t. GW ( new graph A + X, old graph A ) ≤ δ Ω for some budget δ Ω > 0 .
2.1 Definition of Gromov-Wasserstein distance
Since our threat model will only add or remove edges while the nodes remain intact, we only need tocompare two graphs C and D with the same order.
Although GW takes a prior distribution on nodesthat represents their importance, it is often observed that a simple uniform distribution performsequally well or better [12, 16, 46], and many applications do not have such an external prior.
So wecan restrict GW to uniform distributions, leading to the following expression under ℓ 2 distance:
[Equation]
Here 1 is an all-one vector, and C ij is a distance measure between nodes i and j in C , e.g. , SPdistance.
We will denote the n × n matrices as C (for C ) and D (for D ).
We also follow the idea of[12], where C ij does not have to be a metric and ℓ 2 can be extended to asymmetric losses.
They call it
GW discrepancy, and we do not take the square root of the right-hand side of (2) when defining GW .
In the sequel, we will also write GW ( C, D ) instead of GW ( C , D ) whenever it causes no confusion.
2.2 Orthogonal GW and its upper and lower bounds
Unfortunately, GW relies on a nonconvex optimization, and the optimal P in (2) is intractable to find.
Although polytime lower bounds have been proposed by [13], they are shown quite loose [24].
Sinceour aim is to certify robustness under such a measure, a tight and efficient lower bound is in demand(see below).
To this end, we resort to the recently proposed orthogonal GW [ OGW , 24], which doesoffer such a lower bound.
OGW first rewrites the GW in the Koopmans-Beckmann form [47]:
[Equation]
where E
[Equation]
P
[Equation]
=
[Equation]
Let Π be the set of n × n permutation matrices.
The Birkhoff-von Neumann theorem asserts that thedomain of coupling ( E ∩ N ) is the convex hull of Π .
In addition, Π can be characterized byΠ = E ∩ N ∩ O n , where O n := { P ∈ R n × n : P ⊤ P = PP ⊤ = I } .
[Equation]
Since both O n and E are spectral constraints, [24] proposed using these two domains only:
[Equation] [Equation]
To summarize, the attacker tries to minimize the margin by solving the discrete problem:
[Equation]
where C A + X and C A = D are the base distance matrices for A + X and A , respectively.
Since OGW is still intractable to compute, we will next review how [24] proposed to approximate it.
Lower bound of OGW for certificates To certify the robustness, one only needs to optimize (7)with the discrete constraint relaxed into X ij ∈ [ − A ij , 1 − A ij ] .
If the optimal objective value remainsabove 0, then robustness is certified.
To tackle the intractability of OGW , a conservative approach(no false positive) is to relax the feasible domain by replacing OGW with an efficient lower bound .
Although the optimal P in (2) is still intractable to solve, OGW admits an efficient lower boundthat is inspired by the quadratic assignment literature [48].
To begin with, [48] noted that O n ∩ E = 138 (cid:8)n 11 ⊤ + V QV ⊤ : Q ∈ O n − 1 (cid:9) , where V is any n × ( n − 1) matrix satisfying V ⊤ 1 = 0 and
V ⊤ V
= I .
So denoting ˆ X :=
V ⊤ XV and s X :=
1 ⊤ X 1 for any compatible matrix X , we obtain
[Equation]
where Q ( C, D ) :=
[Equation]
So a lower bound of OGW (named OGW lb ) can be naturally derived by decoupling the two occurrences of Q in Q ( C, D ) :
[Equation]
where Q ub
[Equation]
C, D
[Equation]
One can derive that Q 2 is optimized at U E V ⊤ E where U E Λ E
V ⊤ E is the singular value decomposition(SVD) of ˆ E .
This results in tr( ˆ E ⊤ Q 2 ) =
∥ ˆ E ∥ ∗ , the trace norm, which is the sum of its singular values.
An optimal Q 1 is P 1 P ⊤ 2 , if ˆ C and ˆ D respectively admit eigen-decompositions ˆ C = P 1 diag( λ 1 ) P ⊤ 1and
ˆ D
= P 2 diag( λ 2 ) P ⊤ 2 [49, 50].
This yields an optimal value of tr( ˆ CQ 1 ˆ DQ ⊤ 1 ) as λ ⊤ 1 λ 2 .
Theoverall computation costs O ( n 3 ) and the gap arising from decoupling Q 1 and Q 2 is small because ingeneral, the matrix ˆ
E is much smaller than ˆ C and ˆ D in magnitude, as noted by both [48] and [24].
Upper bound of OGW for the attacker
A conservative estimate of whether the attack can besuccessful is obtainable by further restraining the feasible domain.
This can be served by replacing
OGW with its upper bound , which can be trivially achieved by locally optimizing Q in (9).
Wedenote it as OGW ub .
Locally optimizing over O n − 1 ( a.k.a.
Stiefel manifold) has been very wellstudied [51–53], and we adopt a straightforward approach of projected quasi-Newton, noting that theprojection of any matrix Q on O n − 1 is simply U Q V ⊤ Q , where the SVD of Q is U Q Λ Q V ⊤ Q .
As is shown by [24], both OGW and OGW lb are nonnegative, symmetric, and evaluates zero when Cand D are isomorphic.
Their square root also satisfies the triangle inequality.
Experiments on graphclassification and barycenter show that they well capture the structural dissimilarities between graphs.
Similarly to GW , we compute OGW in practice via OGW ub .
2.3 Convex lower bound of OGW
Although OGW lb minorizes OGW , it is still not convex in its input C , falling short of the requirementof certification.
We next derive a convex lower bound via Fenchel biconjugation.
In general,biconjugation can be computationally expensive, but interestingly, our case admits efficient recipes.
Recall that in the context of perturbation, we assume C and D correspond to the new and originalgraphs, respectively, and our goal is to find a convex lower bound for the part of OGW ( C, D ) thatdepends on C , i.e. , f ( C ) :
=
∥
C
∥ 2 F [Equation] 2 max P ∈E∩O tr( CPDP ⊤ ) .
We first compute its Fenchel dual:
[Equation] [Equation]
As f ≥ f ∗∗ ( = ( f ∗ ) ∗ , the biconjugation), OGW ( C, D ) as a function of C can be lower bounded by
[Equation]
The evaluation of Ω( C ) requires solving a convex optimization, utilizing the closed form for Q ubin (11).
In the optimization for the certificate below, we will avoid such an optimization over R byconsidering the dual of Ω( C ) , which can be easily read off from the objective of R in (17).
Convex Outer Approximation for Resistance Distance
So far, we have taken the distance matrix C as the source of variation.
In the certification problem,however, variations originate from the perturbation X on the graph structure.
So we will next relate
X to C A + X , and formulate a convex domain in X .
Unfortunately, this is difficult, so we resort tothe convex outer approximation of the graph of X (cid:55)→ C A + X .
Such an approach has been commonlyused in approximating the ReLU activation [31], and we extend it to a multivariate setting via a newapproach of matching loss [28].
Using resistance distance as the base measure
Although the SP distance has enjoyed significant popularity, its computation requires a nontrivialdiscrete optimization, obstructing a convex relaxation.
As such, we propose using resistance distanceas the base distance because it is a bona fide distance [26], admits a closed form, and costs O ( n 3 ) tocompute all-pair distance (same as SP distance).
In Section 5.1 and 5.2, we will show that it not onlyeases computation, but also performs competitively on machine learning tasks.
Let the Laplacian ofthe perturbed graph be ˜ L := diag
( ˜ A 1 ) − ˜ A .
Then the resistance distance between node i and j is
[Equation]
19) where ˜ L † is the pseudo-inverse of ˜ L .
Since we only consider connected undirected graphs, it follows˜ L † =
(˜ L + 1 n 11 ⊤ ) − 1 − 1 n 11 ⊤ .
A closely related distance measure is the commute time, whichmultiply the resistance distance by the volume π of the graph ( π := 1 ⊤ ˜
A 1 ).
Since π is a scalar andrelates closely with the global budget, it can be incorporated with ease as discussed in Appendix A.1.
In summary, letting Z = − (˜ L + 1 n 11 ⊤ ) − 1 , C is determined through X →
˜ A → ˜ L → Z → ˜ L † → C .
As both the first two steps and the last two steps are affine, we will denote them as ˜ L X and C Zrespectively.
Only the step of ˜ L → Z is nonlinear, posing the major challenge we will address next.
3.2 Convex outer approximation of matrix inversion
Let S + and S ++ be respectively the positive semi-definite and positive definite matrix cones sizedn -by- n .
We consider the set
[Equation]
The conditions on ˜ L are naturally motivated from graph Laplacian, and it follows implicitly that˜ L + 1 n 11 ⊤ ∈ S
++ .
A trivial choice of β ij is 0.
If we only allow adding edges, then β ij can be − A ij .
Given a strictly convex function F , the associated matching loss is defined as
[Equation]
The minimum value of ℓ is 0, attained if, and only if, Z = ∇ F (Φ) .
Now consider F (Φ)
=− log det(Φ) over Φ ∈ S
++ .
It is easy to show that F is strictly convex, ∇ F (Φ)
= − Φ − 1 ,and F ∗ ( Z )
= − n − log det( − Z ) over Z ∈ −S ++ .
∇ F ∗ ( Z ) = − Z − 1 .
Therefore, setting
Z = − (˜ L + 1 n 11 ⊤ ) − 1 is equivalent to enforcing
[Equation]
Input: global budget δ g and OGW budget δ Ω λ ← 1 , and compute X λ via Algorithm 2.
if Υ( A + X λ ) ≥ δ Ω then
Double λ until Υ( A + X λ ) ≤ δ Ω u ← λ (upper est.), l ← u 2 (lower est.) else
Halve λ until Υ( A + X λ ) ≥ δ Ω l ← λ , u ← 2 l while u − l > 0 .
01 (or any threshold) do λ ← ( u + l ) /
If Υ( A + X λ ) ≥ δ Ω then l ← λ else u ← λ
Return X λ
Algorithm
A 0 ← A for t = 0 , 1 , . . .
, δ g − 1 do C t ← cost matrix for A t .
Q t ← locally optimal Q in Q ( C t , C A ) for e ∈ all pairs of nodes do
If flipping edge e on A t violates any local budget then V e ← ∞ and continue
A ′ ← adjacency matrix flipping e on A t
Update C for A ′
V e ← M ( A ′ )
+ λ Υ( A ′ ) , where the Q in
Q ( C, C A ) is initialized by Q t
A t +1 ← flip on A t the edge arg min
e V e Return X λ :=
A δ g − A
This has significantly reduced the difficulty of relaxation because almost all the nonlinearities havebeen subsumed by the convex functions F ∗ and F .
The only remaining challenge is the bilinear termin trace, but it is much easier to relax than a general nonlinear function.
In particular, by the propertyof Laplacian, Z 1 = − 1 and Z ∈ −S ++ .
Let T ii = − Z ii and T ij =
2 Z ij − Z ii − Z jj for i ̸
= j .
Then T ≥ 0 elementwise.
So it follows that
[Equation] [Equation]
Let B be a symmetric matrix such that tr( BZ ) = 12 (cid:80) i ̸
= j β ij T ij = 12 (cid:80) i ̸ =
j β ij (2 Z ij − Z ii − Z jj ) .
Then the constraint set Γ is enclosed by a convex outer approximation asΓ out
[Equation] [Equation]
Overall, we can relax the budget on OGW as follows:
[Equation] [Equation]
Finally, we attain a convex domain.
The last two subsumptions summarize our key contributions ofconvex relaxation in Section 2.2 and 3, and we will experimentally demonstrate that they are tight.
Optimization of Attacks and Robustness Certificates with Relaxed OGW
Attacker algorithm
As discussed in Section 2.2, an attacker solves (7) by resorting to an upperbound of OGW , requiring OGW ub ≤ δ Ω .
Here OGW ub only needs an efficient local optimization.
To deal with the constraint, we resort to the Lagrange dual:
[Equation]
Given λ , X can be optimized greedily as shown in Algorithm 2.
To ease notation, we denote
OGW ub ( C A + X , C A ) as Υ( A + X ) .
So the problem becomes a simple search for the smallest λsuch that Υ( A + X ) ≤ δ Ω , and it can be solved by binary search as in Algorithm 1.
Due to the lackof strong duality, the success of the attack is checked by the sign of M ( A + X ) at the X found by
Algorithm 1, not by the sign of the final objective value in (30).
4.1 Certificate algorithm
When the GCN has a single hidden layer, [7] showed that M ∗ ( A ) can be computed in a closed formover the domain of A ij that is relaxed to [0 , 1] ( i ̸ = j ), intersected with local and global budgets.
As a result, M ∗∗ ( A ) can also be computed efficiently through a convex optimization.
Since M ∗∗minorizes M , we can confirm the robustness ( i.e. , (7) cannot be reduced to negative) if the lowestpossible value of M ∗∗ is nonnegative over an even larger domain of X than that in (7).
This isa sufficient but not necessary condition, and the tightness of the relaxation can be verified via thepercentage of graphs that can neither be certified as robust, nor successfully attacked by Algorithm 1.
Formalizing this idea, the certification algorithm needs to solve the following problem based on (29),
[Equation]
[Equation]
Plugging in the definition of Γ out and noting that the conditions on ˜ L are automatically satisfied by agraph Laplacian, we can explicitize the final convex optimization problem as
[Equation]
Optimization algorithm Solving this optimization efficiently requires appropriately leveragingthe structure in the problem.
Since both M ∗∗ and Ω involve an inner optimization, it will be veryinefficient if we nest their evaluation inside the overall procedure.
Noting that both M ∗ and Ω ∗ havea closed form, we will dualize these terms and introduce two Lagrange multipliers to enforce theinequalities Ω( C Z ) ≤ δ Ω and (34):
[Equation]
The introduction of Ψ makes the resulting objective concave in all the max variables ( α, γ, U, Ψ ),noting that α Ω ∗ (Ψ /α ) is the perspective function of Ω ∗ , hence convex.
This is not the case in (35)because α Ω ∗ ( S ) is not jointly convex.
Finally, swapping the min and max in (37), we obtain thedual objective.
Fixing the max variables, we can 1) minimize over X by projected quasi-Newtonbecause the objective is smooth and strongly convex, and the domain X is simple (Appendix A.2); 2)minimize over Z , which admits a closed-form solution.
We defer the details to Appendix A.3.
Finally,we maximize over ( α, γ, U, Ψ ) by L-BFGS-B [54], terminating once the dual objective gets positive.
Remark on connected graph .
As we set up above, the graphs under consideration are alwaysconnected.
This is the case for all the (original) graphs in our datasets.
If a perturbation makes agraph disconnected, then both Ω and OGW will be infinity.
Therefore the attacker in Algorithm 1will automatically avoid dropping an edge e that disconnects a graph, because the corresponding
V e in Algorithm 2 will be infinity.
As for the certificate, since F (Φ) = − log det(Φ) , the term
F (˜ L X + 1 n 11 ⊤ ) in (34) creates a barrier to ensure ˜ L X + 1 n 11 ⊤ is positive definite, hence connected.
Experimental Results
The goal of our experiments lies in two folds: i) verify that resistance distance (RD)
well characterizesgraph structures and performs better than or as well as SP distance in classification and barycenterproblems, when applied under OGW and GW ; ii) demonstrate the effectiveness of our attack andcertificate algorithms on real datasets, thereby confirming the tightness of our relaxations.
The codeis available on an anonymous Dropbox folder at [55], with no tracking.
Datasets.
We experimented on four graph datasets whose statistics are given in Table 1 [56].
Theycontain a collection of molecules where the vertices represent atoms and edges are chemical bonds.
The class label represents certain property of the molecules, e.g. , mutagenic effect on a specificbacterium (MUTAG) and carcinogenicity of compounds for male rats (PTC-MR).
BZR and COX2consist of ligands for the benzodiazepine receptor and cyclooxygenase-2 inhibitors, respectively.
5.1 Effectiveness of resistance distance in classification
To verify the effectiveness of resistance distance, we first followed [16] to study the graph classification accuracy of an SVM, whose kernel k ( C , D ) is defined as exp ( − γ · OGW ( C , D )) and the base
[Table] [Table]
[Figure]
measure for OGW is RD or SP distance.
We split the dataset into 75% / 25% for training / testing,and tuned the γ and regularizer weight in SVM by 5-fold cross validation on the training set.
The accuracy achieved by SVM based on GW and OGW is presented in Table 2.
In addition, wealso present graph kernels including graphlet sampling [57] and Weisfeiler-Lehman method [9] asthe baselines.
Clearly, taking resistance distance as the base measure produces similar or betterperformance than SP distance under both GW and OGW .
5.2 Effectiveness of resistance distance in barycenter
Furthermore, we also validated the resistance distance in the barycenter problem.
Given a set ofstructured data represented as graphs {D i : i = 1 , 2 , . . .
, S } , it finds the Fréchet mean defined asarg min C (cid:80) S i =
λ i d
( C , D i ) , where d can be GW and OGW , and uniform weights λ
i are endowedon all the examples as in [12].
We generated 8 cycle-like graphs with structured noises from order15 to 25 (Figure 1), and constructed the barycenters of 20 nodes with respect to GW and OGWdiscrepancies under the base measure of SP and RD.
Following Section 3 of [24], we took blockcoordinate update between C and P i in (6), where the former has a closed form and the latter islocally optimized inside GW and OGW .
Figure 2 shows the results of constructed barycenter with different discrepancies and base measures.
Clearly, both SP and RD capture the key structure property (cycle graph) with some additionalstructural noise under GW and OGW .
It is hard to differentiate which structure is better in thecontext of noised samples.
5.3 Performance of certificates and attacks
We next measure the effectiveness of our robustness certificate and attack algorithm jointly, and thecriterion is the fraction of graphs that are neither certified as robust nor vulnerable ( i.e. , successfullyattacked).
Due to space limitation, we only report the result of MUTAG in the main paper, and deferthe result of the other three datasets to Appendix B. To the best of our knowledge, there is no existing
[Figure]
[Figure] [Figure]
certification algorithm that addresses GW or OGW budgets, except generic global optimizationtechniques which do not scale well when nested inside the overall problem (7).
Settings.
Following [7], we split each dataset into 30%, 20%, and 50% for training, validation, andtesting, respectively.
A GCN model was then learned using a single linear convolutional layer with64 hidden nodes, followed by average pooling.
[7] also constructed M ∗∗ with ReLU activation, butshowed larger gap than linear activation.
Since our focus here is on the tightness of certificate andattack under OGW , we would like to insulate the complication from ReLU and hence focused onlinear activation.
Following [5, 36], the GCN is trained with a hinge loss that promotes large marginfrom (1) for robustness: (cid:80) c
̸ =
y max { 0 , 1 + max A {G c ( A ) − G y
( A ) }} , where A is optimized underthe budget of δ l = 1 and δ g = 10 .
Results.
To start with, we empirically checked the tightness of convexified OGW .
We took the firstgraph in the MUTAG dataset, and randomly perturbed its structure (adding or deleting edges) for 20times with the global budget δ g varied from 1 to 10.
Figure 3 shows the average values of OGW andΩ with resistance distance used as the base measure.
As δ g increases, both OGW and Ω grow, butnot monotonically.
This is consistent with the fact that perturbing multiple edges may just lead tosmall changes in GW and OGW due to isomorphism.
Moreover, the gap between OGW and Ω getswider with higher δ g , but their relative difference remains almost intact.
Figure 5 shows the fraction of certifiably robust graphs in the lower green area, and 100 minus thefraction of certified vulnerable graphs in the upper red area.
This leaves the white area showing thegap of undetermined graphs, and a narrow white area indicates the effectiveness of both the certificateand the attacker.
Here, we fixed δ l = 1 , which is sufficient for the global budget δ g on MUTAG.
Sub-plots (a) and (b) present the result for δ Ω = 1 and 10 , respectively.
The gap grows with δ g andstays below 20% of the graphs.
Sub-plots (c) and (d) provide a clearer comparison, between differentvalues of δ Ω , over the performance of certificate and attack.
Increasing the value of δ Ω enlarges thefeasible domain in (33), allowing more (less) graphs to be certified vulnerable (robust).
Figure 4 shows, for graph No. 125 in MUTAG, the value of OGW and M as δ g is increased andδ Ω = 1 .
Interestingly, the increase of δ g does not monotonically consume more budget in Ω , which isconsistent with the property of Ω .
Conclusion.
We designed a new convex lower bound for the orthogonal Gromov-Wassersteindistance based on Fenchel biconjugation.
Combined with a convex relaxation of the resistancedistance using matching loss, it provides a tight certificate of robustness for graph classificationwith GCNs.
Future work can extend the approach to certifying distributional robustness, and furtherleverage the closed form of resistance distance for provable optimization on graphs.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A
crowdsourcing
or
research
human
subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]