Certifying Robust Graph Classification under Orthogonal Gromov-Wasserstein Threats
Abstract
Graph classifiers are vulnerable to topological attacks.
Although certificates of robustness have been recently developed, their threat model only counts local and global edge perturbations, which effectively ignores important graph structures such as isomorphism.
To address this issue, we propose measuring the perturbation with the orthogonal Gromov-Wasserstein discrepancy, and building its Fenchel biconjugate to facilitate convex optimization.
Our key insight is drawn from the matching loss whose root connects two variables via a monotone operator, and it yields a tight outer convex approximation for resistance distance on graph nodes.
When applied to graph classification by graph convolutional networks, both our certificate and attack algorithm are demonstrated effective.
Introduction
State-of-the-art graph classifiers such as graph convolutional networks (GCNs) have been recently revealed vulnerable to adversarial structural attacks, namely adding and removing edges [1, 2].
Due to the hardness of finding the strongest attack, an alternative strategy aims to certify robustness, i.e. , proving that no perturbation in the threat model can successfully attack the classifier.
To this end, [3, 4] extended randomized smoothing to discrete noises with applications in, e.g. , community detection.
Although our focus is on graph classification, certificates have been studied for node classification using, e.g. , convex outer adversarial polytope [5, 6].
Most relevant to our method is [7], which constructed the tightest lower bound of the margin via Fenchel biconjugation.
However, all existing threat models only count the total edges removed or added (global budget) or the change of each node’s degree (local budget), while overlooking the important notion of isomorphism .
In contrast, there has been a wealth of more refined discrepancy measures between graphs, including kernels [8, 9] and GCNs [10, 11].
Recently, the Gromov-Wasserstein discrepancy [ GW , 12], which extends the Gromov-Wasserstein distance [13], has emerged as an effective transportation distance between structured data, alleviating the incomparability issue between different structures by aligning the intra -relational geometries.
Thanks to its favorable properties such as efficiency and awareness of isomorphism, GW has been extensively applied to domain adaptation [14], word embedding [15], graph classification [16], metric alignment [17], generative modeling [18], and graph matching and node embedding [19, 20].
Therefore, it is clearly natural to adopt such a measure in the certification of graph robustness.
Analogously, Wasserstein distance has already been employed in distributional robustness [21] and in attacking images [22], where perturbations can be found that better reflect the image content compared with the standard ℓ p attacks.
Unfortunately, GW is not tractable to evaluate, differing from the standard Wasserstein distance which is a linear program.
So all existing optimization techniques settle for local solutions, lacking an analyzable guarantee.
To develop certificates, tractable lower bounds of GW would be essential, but the Kantorovich dual no longer exists and the existing lower bounds [13, 23] are not tight [24].
Further, underlying the standard probability coupling is the metrics within each graph, e.g. , shortestpath distance [13, SP distance], which requires nontrivial discrete optimization.
This significantly complicates the entire certificate as the subject of perturbation is exactly the topological structure.
In addition, certificates based on randomized smoothing [4, 25], which adds noise to the input, are also hindered by the significant challenge in designing the noise distribution that aligns with GW .
The goal of this work, therefore, is to construct the first robustness certificate for GW -style threat models, with applications in graph classification.
This is achieved by casting the entire computation procedure of GW as a two-layer model, each of which is subsequently relaxed in a tractable fashion: graph structure
[Figure]
btw
[Figure]
discrepancy btw graphs .
Firstly, we circumvent the discrete optimization in SP distance by resorting to the resistance distance [26], which, along with its scaled version known as commute time, is one of the most commonly used distances in computer science, machine learning, and beyond [27].
Compared with SP distance, it admits a closed-form based on matrix inversion, which is a monotone operator on the positive semi-definite cone.
This enables our first contribution, where a convex relaxation is designed for resistance distance in Section 3 based on the matching loss [28].
Our second contribution achieves convex relaxation of the GW -style discrepancy measure.
Our seed inspiration is drawn from the latest orthogonal GW discrepancy ( OGW ) [24] reviewed in
Section 2.2—it preserves most of the desirable properties of GW while admitting a tight and efficient lower bound.
We next make the key observation in Section 2.3 that the lower bound, despite not convex, admits a closed-form Fenchel dual, which in turn facilitates an efficient evaluation of the
Fenchel biconjugate.
Such a convex lower bound is applied to graph convolution networks in
Section 4, where both attacking algorithm and robustness certificates are developed.
Our experiments in Section 5 verify the effectiveness of our attacker and certificate, in that a large proportion of the graphs can be proved either vulnerable or robust.
Related Work Certification or verification has been a longstanding pursuit in machine learning and cyber-physical systems.
Formally, it inquires if f ( x ) can be driven down to negative over x ∈ X , where both f and X can be too complicated for global optimization.
So a lower bound of the optimal objective is sought, which provides incomplete verification (some true properties are confirmed true).
A lower bound can also be useful in branch-and-bound algorithms [6].
A natural approach is the
Lagrange dual [29, 30], which minorizes the primal by the weak duality.
However, it can be loose, and a provably tighter lower bound is the convex envelope or Fenchel biconjugate [7].
But they themselves are efficiently computable only under limited scenarios.
In general, the objective f is composed of several “layers”, i.e. , f = f 0 ◦ f 1 ◦ . .
, where f i is simpler but no longer scalar-valued ( i ≥ 1 ).
So it is natural to relax their graphs.
For example, min f 0 ( f 1 ( x )) is equivalent to min x,y f 0 ( y ) s.t.
y
=
f 1 ( x ) .
So we can possibly use the convex envelop of f 0 (which is simpler than f ) and employ the convex hull of { ( x, y ) :
y
= f 1 ( x ) } .
The latter is represented by the convex envelope relaxation of the ReLU activation [31–35], which has also found applications in graph neural networks [6, 36].
However, they have been so far limited to R → R nonlinear functions while a multivariate extension is not available yet, as the convex hull is much harder to compute.
Another commonly used approach is randomized smoothing, which adds noise to the input [25, 37– 39].
Although it has also been shown effective in graph robustness [3, 4], the design of discrete noise can be quite intricate and so far no such methods exist for GW -style threat models.
Other methods include semi-definite relaxation [40, 41] which is often loose, Lipschitz continuity analysis [42–44] where the local and global estimation of curvature is difficult in the discrete domain, and reformulation linearization technique [45] used by [5] for directed but not undirected graphs.
Gromov-Wasserstein Threats for Attacking Graph Classifiers
We consider attacking a graph classifier where the edge connectivity is perturbed.
Here both the original graph D and the resulting graph C are assumed undirected , unweighted , and connected .
Let the adjacency matrix of D be A ∈ { 0 , 1 } n × n , where A ii = 0 , and A ij ∈ { 0 , 1 } ( i ̸ =
j ) indicating whether an edge exists between nodes i and j .
Here n is the order of the graph, i.e. , number of nodes.
Now suppose the graph’s edge connectivity is perturbed, leading to a new adjacency matrix ˜
A :=
A + X (corresponding to C ).
Obviously, X needs to be symmetric, zero diagonal, and X ij ∈ {− A ij , 1 − A ij } .
If we relax it to the continuous domain such as convex relaxation, X ij is naturally relaxed to [ − A ij , 1 − A ij ] .
The threat model governs additional budgets on X besides the above admissible conditions.
Global change budget can be written as 1 ⊤ ( X ◦ S ) 1 ≤ δ g , where S = 1 − 2 A and ◦ stands for the Hadamard product.
Local budget is ( X ◦ S ) 1 ≤ δ l 1 (elementwise).
If we only allow adding edges, then X ij ≥ 0 .
All these basic constraints are linear in X , and we will collectively refer to them as X ∈ X , a convex set.
Suppose we have trained a graph classifier G ( e.g. , graph convolution network, GCN) which maps an adjacency matrix A to a K -dimensional logit vector G ( A ) corresponding to K classes.
Assume the true class is y and G predicts correctly,
i.e. , y = arg max c G c ( A ) .
Then the margin of classification is
[Equation]
The attacker seeks a feasible perturbation X ∈ X that drives the classification margin to negative, i.e. , a misclassification.
G is called robust on this graph if min X ∈X M ( A + X ) is positive, and is called vulnerable if the minimal value is negative.
It is noteworthy that we employ graph classification and
GCN only as an example context of applying our proposed relaxation technique.
Other tasks such as node classification are readily applicable too – simply replace the M function.
Although the global and local budgets are natural and reasonable characterizations of the perturbation, they ignore an important notion on graphs: isomorphism.
Exact isomorphism can be relaxed by defining a distance between two graphs, which quantify their difference under the most favorable permutation of nodes.
One of the most commonly used distance is the Gromov-Wasserstein distance [13], which has been extended to Gromov-Wasserstein discrepancy [ GW , 12].
So in addition to the standard local and global budgets specified by X , it is natural to further constrain the perturbation in terms of the GW distance.
To conclude, the attack and certification tasks can be formalized as min X ∈X M ( A + X ) s.t. GW ( new graph A + X, old graph A ) ≤ δ Ω for some budget δ Ω > 0 .
2.1 Definition of Gromov-Wasserstein distance
Since our threat model will only add or remove edges while the nodes remain intact, we only need to compare two graphs C and D with the same order.
Although GW takes a prior distribution on nodes that represents their importance, it is often observed that a simple uniform distribution performs equally well or better [12, 16, 46], and many applications do not have such an external prior.
So we can restrict GW to uniform distributions, leading to the following expression under ℓ 2 distance:
[Equation]
Here 1 is an all-one vector, and C ij is a distance measure between nodes i and j in C , e.g. , SP distance.
We will denote the n × n matrices as C (for C ) and D (for D ).
We also follow the idea of [12], where C ij does not have to be a metric and ℓ 2 can be extended to asymmetric losses.
They call it
GW discrepancy, and we do not take the square root of the right-hand side of (2) when defining GW .
In the sequel, we will also write GW ( C, D ) instead of GW ( C , D ) whenever it causes no confusion.
2.2 Orthogonal GW and its upper and lower bounds
Unfortunately, GW relies on a nonconvex optimization, and the optimal P in (2) is intractable to find.
Although polytime lower bounds have been proposed by [13], they are shown quite loose [24].
Since our aim is to certify robustness under such a measure, a tight and efficient lower bound is in demand (see below).
To this end, we resort to the recently proposed orthogonal GW [ OGW , 24], which does offer such a lower bound.
OGW first rewrites the GW in the Koopmans-Beckmann form [47]:
[Equation]
where E := { P
[Equation]
Let Π be the set of n × n permutation matrices.
The Birkhoff-von Neumann theorem asserts that the domain of coupling ( E ∩ N ) is the convex hull of Π .
In addition, Π can be characterized by Π = E ∩ N ∩ O n , where O n := { P ∈ R n × n : P ⊤ P = PP ⊤ = I } .
Since both O n and E are spectral constraints, [24] proposed using these two domains only:
[Equation]
To summarize, the attacker tries to minimize the margin by solving the discrete problem:
[Equation]
where C A + X and C A
= D are the base distance matrices for A + X and A , respectively.
Since
OGW is still intractable to compute, we will next review how [24] proposed to approximate it.
Lower bound of OGW for certificates To certify the robustness, one only needs to optimize (7) with the discrete constraint relaxed into X ij ∈ [ − A ij , 1 − A ij ] .
If the optimal objective value remains above 0, then robustness is certified.
To tackle the intractability of OGW , a conservative approach (no false positive) is to relax the feasible domain by replacing OGW with an efficient lower bound .
Although the optimal P in (2) is still intractable to solve, OGW admits an efficient lower bound that is inspired by the quadratic assignment literature [48].
To begin with, [48] noted that O n ∩ E = (cid:8)n 11 ⊤ + V QV ⊤ : Q ∈ O n − 1 (cid:9) , where V is any n × ( n − 1) matrix satisfying V ⊤ 1 = 0 and
V ⊤ V
= I .
So denoting ˆ X :=
V ⊤ XV and s X :=
1 ⊤ X 1 for any compatible matrix X , we obtain
[Equation]
where Q
[Equation]
So a lower bound of OGW (named OGW lb ) can be naturally derived by decoupling the two occurrences of Q in Q ( C, D ) :
[Equation]
where Q ub ( C, D ) :
=
[Equation]
One can derive that Q 2 is optimized at U E V ⊤ E where U E Λ E
V ⊤ E is the singular value decomposition (SVD) of ˆ E .
This results in tr( ˆ E ⊤ Q 2 ) =
∥ ˆ E ∥ ∗ , the trace norm, which is the sum of its singular values.
An optimal Q 1 is P 1 P ⊤ 2 , if ˆ C and ˆ D respectively admit eigen-decompositions ˆ C = P 1 diag( λ 1 ) P ⊤and
ˆ D = P 2 diag( λ 2 ) P ⊤ 2 [49, 50].
This yields an optimal value of tr( ˆ CQ 1 ˆ DQ ⊤ 1 ) as λ ⊤ 1 λ 2 .
The overall computation costs O ( n 3 ) and the gap arising from decoupling Q 1 and Q 2 is small because in general, the matrix
ˆ E is much smaller than ˆ C and ˆ D in magnitude, as noted by both [48] and [24].
Upper bound of OGW for the attacker
A conservative estimate of whether the attack can be successful is obtainable by further restraining the feasible domain.
This can be served by replacing
OGW with its upper bound , which can be trivially achieved by locally optimizing Q in (9).
We denote it as OGW ub .
Locally optimizing over O n − 1 ( a.k.a.
Stiefel manifold) has been very well studied [51–53], and we adopt a straightforward approach of projected quasi-Newton, noting that the projection of any matrix Q on O n − 1 is simply U Q V ⊤ Q , where the SVD of Q is U Q Λ Q V ⊤ Q .
As is shown by [24], both OGW and OGW lb are nonnegative, symmetric, and evaluates zero when C and D are isomorphic.
Their square root also satisfies the triangle inequality.
Experiments on graph classification and barycenter show that they well capture the structural dissimilarities between graphs.
Similarly to GW , we compute OGW in practice via OGW ub .
2.3 Convex lower bound of OGW
Although OGW lb minorizes OGW , it is still not convex in its input C , falling short of the requirement of certification.
We next derive a convex lower bound via Fenchel biconjugation.
In general, biconjugation can be computationally expensive, but interestingly, our case admits efficient recipes.
Recall that in the context of perturbation, we assume C and D correspond to the new and original graphs, respectively, and our goal is to find a convex lower bound for the part of OGW ( C, D ) that depends on C , i.e. , f ( C ):= ∥
C ∥ 2 F − 2 max P ∈E∩O n tr( CPDP ⊤ ) .
We first compute its Fenchel dual:
[Equation]
As f ≥ f ∗∗ ( = ( f ∗ ) ∗
[Equation]
the biconjugation)
[Equation]
OGW ( C, D ) as a function of C can be lower bounded by
[Equation]
The evaluation of Ω( C ) requires solving a convex optimization, utilizing the closed form for Q ub in (11).
In the optimization for the certificate below, we will avoid such an optimization over R by considering the dual of Ω( C ) , which can be easily read off from the objective of R in (17).
Convex Outer Approximation for Resistance Distance
So far, we have taken the distance matrix C as the source of variation.
In the certification problem, however, variations originate from the perturbation X on the graph structure.
So we will next relate
X to C A + X , and formulate a convex domain in X .
Unfortunately, this is difficult, so we resort to the convex outer approximation of the graph of X (cid:55)→ C A + X .
Such an approach has been commonly used in approximating the ReLU activation [31], and we extend it to a multivariate setting via a new approach of matching loss [28].
3.1 Using resistance distance as the base measure
Although the SP distance has enjoyed significant popularity, its computation requires a nontrivial discrete optimization, obstructing a convex relaxation.
As such, we propose using resistance distance as the base distance because it is a bona fide metric [26], admits a closed form, and costs O ( n 3 ) to compute all-pair distance (same as SP distance).
In Section 5.1 and 5.2, we will show that it not only eases computation, but also performs competitively on machine learning tasks.
Let the Laplacian of the perturbed graph be ˜ L :=
diag
( ˜ A 1 ) − ˜ A .
Then the resistance distance between node i and j is
[Equation]
(19) where ˜ L † is the pseudo-inverse of ˜ L .
Since we only consider connected undirected graphs, it follows ˜
L † =
( ˜ L + 1 n 11 ⊤ ) − 1 − 1 n 11 ⊤ .
A closely related distance measure is the commute time, which multiply the resistance distance by the volume of the graph ( 1 ⊤ ˜
A 1 ).
Since the volume is a scalar and relates closely with the global budget, it can be incorporated with ease as discussed in Appendix A.1.
In summary, letting Z = − (˜ L + 1 n 11 ⊤ ) − 1 , C is determined through X →
˜ A → ˜ L → Z → ˜ L † → C .
As both the first two steps and the last two steps are affine, we will denote them as ˜ L X and C Z respectively.
Only the step of ˜ L → Z is nonlinear, posing the major challenge we will address next.
3.2 Convex outer approximation of matrix inversion
Let S + and S ++ be respectively the positive semi-definite and positive definite matrix cones sized n -by- n .
We consider the set
[Equation]
The conditions on ˜ L are naturally motivated from graph Laplacian, and it follows implicitly that ˜
L + 1 n 11 ⊤ ∈ S
++ .
A trivial choice of β ij is 0.
If we only allow adding edges, then β ij can be − A ij .
Given a strictly convex function F , the associated matching loss is defined as [28]
[Equation]
and it is well known that the minimum value of ℓ is 0, attained if, and only if, Z = ∇ F (Φ) .
Now consider F (Φ)
= − log det(Φ) over Φ ∈ S
++ .
It is easy to show that F is strictly convex, ∇ F (Φ)
= − Φ − 1 , and F ∗ ( Z )
= − n − log det( − Z ) over Z ∈ −S ++ .
∇ F ∗ ( Z ) = − Z − 1 .
Therefore, setting Z = − (˜ L + 1 n 11 ⊤ ) − 1 is equivalent to enforcing
[Equation]
This has significantly reduced the difficulty of relaxation because almost all the nonlinearities have been subsumed by the convex functions F ∗ and F .
The only remaining challenge is the bilinear term in trace, but it is much easier to relax than a general nonlinear function.
In particular, by the property of Laplacian, Z 1 = − 1 and Z ∈ −S ++ .
Let T ii = − Z ii and T ij =
2 Z ij − Z ii − Z jj for i ̸
= j .
Then T ≥ 0 elementwise.
So it follows that
[Equation]
Let B be a symmetric matrix such that tr( BZ
[Equation]
= [Equation] Then the constraint set F is enclosed by a convex outer approximation as
F out
[Equation]
Overall, we can relax the budget on OGW as follows:
[Equation]
Finally, we attain a convex domain.
The last two subsumptions summarize our key contributions of convex relaxation in Section 2.2 and 3, and we will experimentally demonstrate that they are tight.
Optimization of Attacks and Robustness Certificates with Relaxed OGW
Attacker algorithm
As discussed in Section 2.2, an attacker solves (7) by resorting to an upper bound of OGW , requiring OGW ub ≤ δ Ω .
Here OGW ub only needs an efficient local optimization.
To deal with the constraint, we resort to the Lagrange dual: [Equation]
[Equation]
Given λ , X can be optimized greedily as shown in Algorithm 2.
To ease notation, we denote
OGW ub ( C A + X , C A ) as Υ( A + X ) .
So the problem becomes a simple search for the smallest λ such that Υ( A + X ) ≤ δ Ω , and it can be solved by binary search as in Algorithm 1.
Due to the lack of strong duality, the success of the attack is checked by the sign of M ( A + X ) at the X found by
Algorithm 1, not by the sign of the final objective value in (30).
4.1 Certificate algorithm
When the GCN has a single hidden layer, [7] showed that M ∗ ( A ) can be computed in a closed form over the domain of A ij that is relaxed to [0 , 1] ( i ̸ = j ), intersected with local and global budgets.
As a result, M ∗∗ ( A ) can also be computed efficiently through a convex optimization.
Since M ∗∗ minorizes M , we can confirm the robustness ( i.e. , (7) cannot be reduced to negative) if the lowest possible value of M ∗∗ is nonnegative over an even larger domain of X than that in (7).
This is
Input: global budget δ g and OGW budget δ Ω λ ← 1 , and compute X λ via Algorithm 2.
if Υ( A + X λ ) ≥ δ Ω then
Double λ until Υ( A + X λ ) ≤ δ Ω u ← λ (upper est.), l ← u 2 (lower est.) else
Halve λ until Υ( A + X λ ) ≥ δ Ω l ← λ , u ← 2 l while u − l > 0 .
01 (or any threshold) do λ ← ( u + l ) /
If Υ( A + X λ ) ≥ δ Ω then l ← λ else u ← λ
Return X λ
A 0 ← A for t = 0 , 1 , . . .
, δ g − 1 do C t ← cost matrix for A t .
Q t ← locally optimal Q in Q ( C t , C A ) for e ∈ all pairs of nodes do
If flipping edge e on A t violates any local budget then V e ← ∞ and continue
A ′ ← adjacency matrix flipping e on A t
Update C for A ′
V e ← M ( A ′ )
+ λ Υ( A ′ ) , where the Q in
Q ( C, C A ) is initialized by Q t
A t +1 ← flip on A t the edge arg min
e V e Return X λ :=
A δ g − A a sufficient but not necessary condition, and the tightness of the relaxation can be verified via the percentage of graphs that can neither be certified as robust, nor successfully attacked by Algorithm 1.
Formalizing this idea, the certification algorithm needs to solve the following problem based on (29),
[Equation]
Plugging in the definition of F out and noting that the conditions on ˜ L are automatically satisfied by a graph Laplacian, we can explicitize the final convex optimization problem as
[Equation]
[Equation]
Optimization algorithm Solving this optimization efficiently requires appropriately leveraging the structure in the problem.
Since both M ∗∗ and Ω involve an inner optimization
[Equation]
it will be very inefficient if we nest their evaluation inside the overall procedure.
Noting that both M ∗ and Ω ∗ have a closed form
[Equation]
we will dualize these terms and introduce two Lagrange multipliers to enforce the inequalities Ω( C Z ) ≤ δ Ω and (34):
[Equation]
The introduction of Ψ makes the resulting objective concave in all the max variables ( α, γ, U, Ψ ), noting that α Ω ∗ (Ψ /α ) is the perspective function of Ω ∗ , hence convex.
This is not the case in (35) because α Ω ∗ ( S ) is not jointly convex.
Finally, swapping the min and max in (37), we obtain the dual objective.
Fixing the max variables, we can 1) minimize over X by projected quasi-Newton because the objective is smooth and strongly convex, and the domain X is simple (Appendix A.2); 2) minimize over Z , which admits a closed-form solution.
We defer the details to Appendix A.3.
Finally, we maximize over ( α, γ, U, Ψ ) by L-BFGS-B [54], terminating once the dual objective gets positive.
Remark on connected graph .
As we set up above, the graphs under consideration are always connected.
This is the case for all the (original) graphs in our datasets.
If a perturbation makes a graph disconnected, then both Ω and OGW will be infinity.
Therefore the attacker in Algorithmwill automatically avoid dropping an edge e that disconnects a graph, because the corresponding
V e in Algorithm 2 will be infinity.
As for the certificate, since F (Φ) = − log det(Φ) , the term
F (˜ L X + 1 n 11 ⊤ ) in (34) creates a barrier to ensure ˜ L X + 1 n 11 ⊤ is positive definite, hence connected.
Experimental Results
The goal of our experiments lies in two folds: i) verify that resistance distance (RD)
well characterizes graph structures and performs better than or as well as SP distance in classification and barycenter
[Table]
[Table]
problems, when applied under OGW and GW ; ii) demonstrate the effectiveness of our attack and certificate algorithms on real datasets, thereby confirming the tightness of our relaxations.
The code is available at [55].
Datasets.
We experimented on four graph datasets whose statistics are given in Table 1 [56].
They contain a collection of molecules where the vertices represent atoms and edges are chemical bonds.
The class label represents certain property of the molecules, e.g. , mutagenic effect on a specific bacterium (MUTAG) and carcinogenicity of compounds for male rats (PTC-MR).
BZR and COX consist of ligands for the benzodiazepine receptor and cyclooxygenase-2 inhibitors, respectively.
5.1 Effectiveness of resistance distance in classification
To verify the effectiveness of resistance distance, we first followed [16] to study the graph classification accuracy of an SVM, whose kernel k ( C , D ) is defined as exp ( − γ · OGW ( C , D )) and the base measure for OGW is RD or SP distance.
We split the dataset into 75% / 25% for training / testing, and tuned the γ and regularizer weight in SVM by 5-fold cross validation on the training set.
The accuracy achieved by SVM based on GW and OGW is presented in Table 2.
In addition, we also present graph kernels including graphlet sampling [57] and Weisfeiler-Lehman method [9] as the baselines.
Clearly, taking resistance distance as the base measure produces similar or better performance than SP distance under both GW and OGW .
5.2 Effectiveness of resistance distance in barycenter
Furthermore, we also validated the resistance distance in the barycenter problem.
Given a set of structured data represented as graphs {D i : i = 1 , 2 , . . .
, S } , it finds the Fréchet mean defined as arg min C (cid:80) Si
=
λ i d
( C , D i ) , where d can be GW and OGW , and uniform weights λ
i are endowed on all the examples as in [12].
We generated 8 cycle-like graphs with structured noises from order 15 to 25 (Figure 1), and constructed the barycenters of 20 nodes with respect to GW and OGW discrepancies under the base measure of SP and RD.
Following Section 3 of [24], we took block coordinate update between C and P i in (6), where the former has a closed form and the latter is locally optimized inside GW and OGW .
Figure 2 shows the results of constructed barycenter with different discrepancies and base measures.
Clearly, both SP and RD capture the key structure property (cycle graph) with some additional structural noise under GW and OGW .
It is hard to differentiate which structure is better in the context of noised samples.
5.3 Performance of certificates and attacks
We next measure the effectiveness of our robustness certificate and attack algorithm jointly, and the criterion is the fraction of graphs that are neither certified as robust nor vulnerable ( i.e. , successfully attacked).
Due to space limitation, we only report the result of MUTAG in the main paper, and defer the result of the other three datasets to Appendix B. To the best of our knowledge, there is no existing
[Figure] [Figure]
[Figure] [Figure]
Settings.
Following [7], we split each dataset into 30%, 20%, and 50% for training, validation
[Figure]
and testing, respectively.
A GCN model was then learned using a single linear convolutional layer with 64 hidden nodes, followed by average pooling.
[7] also constructed M ∗∗ with ReLU activation
[Figure]
but showed larger gap than linear activation.
Since our focus here is on the tightness of certificate and attack under OGW , we would like to insulate the complication from ReLU and hence focused on linear activation.
Following [5, 36], the GCN is trained with a hinge loss that promotes large margin from (1) for robustness: (cid:80) c
̸ =
y max { 0 , 1 + max A {G c ( A ) − G y
( A ) }} , where A is optimized under the budget of δ l = 1 and δ g = 10 .
Results.
To start with, we empirically checked the tightness of convexified OGW .
We took the first graph in the MUTAG dataset, and randomly perturbed its structure (adding or deleting edges) fortimes with the global budget δ g varied from 1 to 10.
Figure 3 shows the average values of OGW and Ω with resistance distance used as the base measure.
As δ g increases, both OGW and Ω grow, but not monotonically.
This is consistent with the fact that perturbing multiple edges may just lead to small changes in GW and OGW due to isomorphism.
Moreover, the gap between OGW and Ω gets wider with higher δ g , but their relative difference remains almost intact.
Figure 5 shows the fraction of certifiably robust graphs in the lower green area, and 100 minus the fraction of certified vulnerable graphs in the upper red area.
This leaves the white area showing the gap of undetermined graphs, and a narrow white area indicates the effectiveness of both the certificate and the attacker.
Here, we fixed δ l = 1 , which is sufficient for the global budget δ g on MUTAG.
Sub-plots (a) and (b) present the result for δ Ω = 1 and 10 , respectively.
The gap grows with δ g and stays below 20% of the graphs.
Sub-plots (c) and (d) provide a clearer comparison, between different values of δ Ω , over the performance of certificate and attack.
Increasing the value of δ Ω enlarges the feasible domain in (33), allowing more (less) graphs to be certified vulnerable (robust).
Figure 4 shows, for graph No. 125 in MUTAG, the value of OGW and M as δ g is increased and δ Ω = 1 .
Interestingly, the increase of δ g does not monotonically consume more budget in Ω , which is consistent with the property of Ω .
We additionally compared the performance of our vanilla and robust one-layer GCN model with an
MLP model with node feature only, and two other state-of-the-art models, namely MemGNN [58] and FactorGCN [59].
The results are deferred to Appendix E.
Conclusion
We designed a new convex lower bound for the orthogonal Gromov-Wasserstein distance based on Fenchel biconjugation.
Combined with a convex relaxation of the resistance distance using matching loss, it provides a tight certificate of robustness for graph classification with GCNs.
Future work can extend the approach to certifying distributional robustness, and further leverage the closed form of resistance distance for provable optimization on graphs.
It is noteworthy that our relaxation of OGW with resistance distance is orthogonal to the graph classifier itself.
When the classifier is changed, one will only need to re-derive the expression of M ∗∗ for the risk or margin.
The constraints in (30) and (32), however, remain intact, and that part is our contribution.
It will also be interesting to extend GCN to multiple layers GCN, by, e.g. , leveraging the convex envelop discussed in Appendix D of [7].
Acknowledgments and Disclosure of Funding
This material is based upon work supported by the National Science Foundation under Grant No. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]