Can Hybrid Geometric Scattering Networks Help Solve the Maximal Clique Problem?
Anonymous Author(s)
AffiliationAddress email
Abstract
We propose a geometric scattering-based graph neural network (GNN) for approximating solutions of the NP-hard maximal clique (MC) problem.
We construct aloss function with two terms, one which encourages the network to find a large setof nodes and the other which acts as a surrogate for the constraint that the nodesform a clique.
We then use this loss to train a novel GNN architecture that outputsa vector representing the probability for each node to be part of the MC and apply arule-based decoder to make our final prediction.
The incorporation of the scatteringtransform alleviates the so-called oversmoothing problem that is often encounteredin GNNs and would degrade the performance of our proposed setup.
Our empiricalresults demonstrate that our method outperforms representative GNN baselines interms of solution accuracy and inference speed as well as conventional solvers like GUROBI with limited time budgets.
Introduction
The success of Graph Neural Networks (GNNs) for a variety of machine learning tasks [Scarselliet al., 2008, Gori et al., 2005, Kipf and Welling, 2016, Gilmer et al., 2017] has sparked interests inusing GNNs to solve graph combinatorial optimization (CO) problems [Li et al., 2018a, Karalias and
Loukas, 2020].
For example, Joshi et al.
[2019] introduces a GNN-based method for approximatelysolving the travelling salesman problem (TSP), while Karalias and Loukas [2020] approximatesolutions of the maximal clique (MC) problem.
Solving such problems in an end-to-end fashionvia GNNs is challenging for several reasons.
First of all, many CO problems are provably either
NP-Hard or NP-complete.
Therefore, learning CO solvers in a supervised or semi-supervised fashioncan be computationally infeasible since the cost of generating the truth labels grows exponentiallywith the problem size.
Second, the solution to the CO problems must satisfy a number of constraintsor limitations (e.g. belonging to a clique).
Although these constraints can be imposed during trainingtime, there is still no guarantee that these constraints are still satisfied at test time.
Furthermore,solving graph combinatorial problems largely depends on the expressive power of GNNs.
Many
GNNs aggregate information via local averaging, which can be interpreted as a smoothing operation.
This degrades the expressive power of GNNs and results in the so-called oversmoothing problem [Liet al., 2018b] where neighboring nodes have similar representations and are difficult to distinguishfrom each other.
This is problematic if a node that is not in the solution set borders many points thatare in the solution set.
The purpose of this paper is to use a geometric scattering-based GNN to approximate the solution ofthe maximal clique (MC) problem, that is to find the largest complete subgraph contained within alarge graph G .
The use of geometric scattering, rather than a more traditional GNN is motivated byrecent work [Wenkel et al., 2022] showing that the geometric scattering transform can help overcomethe oversmoothing problem via the use of bandpass wavelet filters in conjunction with GCN-type
[Figure]
filters.
This is particularly important in the context of the MC problem because it is critical todistinguish a point which connects to many members of the clique from an actual member of theclique.
As we shall show, our method, which utilizes bandpass wavelet filters is able to better detectthe border between the MC and the rest of the graph.
This parallels the traditional use of wavelets asedge detectors in image processing[Grossmann, 1988].
Our method uses a two-phase strategy shown in Figure 1.
We first use an unsupervised learningmodel to generate an efficient representation of how likely each node is to be part of the maximalclique.
Then, this representations is fed into a constraint-preserving decoder to generate the maximalclique.
Ideally, given a graph G with n nodes, we would build a representation p ∈ [0 , 1] n wherethe nodes in the maximal clique are assigned the value 1 (True) while all other nodes are assigned 0(False), so that we can ‘cut’ the MC from the graph via O ( n ) operations.
In practice, we will build arepresentation which assigns large probabilities to the maximal clique nodes and small probabilitiesto the other nodes.
Figure 2 (a) presents a situation where oversmoothing does not occur.
The signalp is much larger on the nodes (1 , 2 , 3 , 4 , 6) , which form the maximal clique, than it is on the othernodes (5 , 7 , 8) .
Therefore, the decoder can easily find the MC by, e.g., setting a threshold.
Figure2 (b), on the other hand, illustrates a situation in which oversmoothing does occur and p has lessvariability.
Indeed, there are no blue colored nodes at all.
In this setting, it is nearly impossible todistinguish node 7 , which does not belong to the maximal clique, from the nodes 1 , 6 , and thereforeextracting the MC from p is difficult if not impossible.
[Figure]
Background and Related Work
2.1 Graph Signal Processing
Consider a graph G =
( V, E ) with a set of nodes (or vertices) V :=
{ v 1 , . .
. , v n } and a set of undirected edges E ⊂ V × V .
For a function x : V → R defined on V , we will, in minor abuse of notation identify x with the vector where x [ i ] =
x ( v i ) .
We let W ∈ R n × n denote the adjacency matrix of G and define the symmetric normalized grah Laplacian by
[Equation] where d i := (cid:80) nj =1 W [ v i , v j ] is the degree of the node v i and D :=
diag( d 1 , ..., d n ) ∈ R n × n is the degree matrix.
We will let q i
and λ i denote the (normalized) eigenvectors and eigenvalues of L with
L q i =
λ i q i , and make use of the eigendecomposition L = Q Λ Q ⊤ , where Q is the orthogonal matrix whose i -th column is the normalized eigenvector q i , and Λ :=diag( λ 1 , ..., λ n ) .
Notably, the eigenvectors q 1 , . . .
q n can be seen as a generalization of Fouriermodes in the context of graph domains with the corresponding eigenvalues representing the increasingfrequencies 0 ≤ λ 1 ≤ · · · ≤ λ n ≤ 2 .
This link may be established by viewing frequency as ameasure of variation, in particular the variation of (degree-normalized)
Fourier modes across edgesis λ i =
q ⊤ i
L q i = (cid:80) { u,v }∈ E (˜ q i [ u ] − ˜ q
i [ v ]) 2 for ˜ q i :=
D − 1 / 2 q i .
The graph Fourier transformof a signal vector x is then defined as by ˆ x [ i ] = ⟨ x , q i ⟩ and the inverse Fourier is given byx = (cid:80) ni =
ˆ x [ i ] q i .
Compactly, we write ˆ x = Q ⊤ x and x = Q
ˆ x .
In the Euclidean setting, it is known that convolution in the spatial domain corresponds to multiplication in the Fourier domain.
This fact has motivated work such as Shuman et al.
[2016], which defines the convolution of a signal x with a filter g to be the unique vector verifying ( (cid:91) g ⋆ x )
[ i ]
= ˆ g
[ i ] ˆ x
[ i ] , which implies that
[-=
 ˆ g-][Equation]
where (cid:98) G :=
diag( ˆ g ) =
diag( ˆ g [1] , . .
, ˆ g [ n ]) .
When incorporating this notion of convolution into a graph neural network, a common choice [e.g., Defferrard et al., 2016] is to require the coefficients ˆ g [ i ] to be polynomials of the eigenvalues
λ
i , i ∈ [ n ] , i.e., ˆ g [ i ] := (cid:80) k γ k λ ki in which case we have (cid:98) G = (cid:80) k γ k Λ k .
This allows one to implement convolution in the spatial domain by verifying that
[Equation]
In particular, the entire method may be implemented without the need to diagonalize a matrix whichis extremely expensive for large graphs.
2.2 Graph Convolutional Networks
In Kipf and Welling [2016], the authors set ˆ g [ i ] :
= θ (2 − λ i ) , where θ is a real number, which yields
[Equation]
[Equation] As the eigenvalues of the above convolutional filter lie in [0,2] and could lead to vanishing or exploding gradients, Kipf and Welling [2016] apply a renormalization trick which replaces
I n + D − 1 / 2 W D − 1 / 2 with
[Equation]
The layer-wise propagation rule of GCN is then defined by x ℓj = σ (cid:0) (cid:80) N ℓ − 1 i =1 θ ℓij A x ℓ − 1 i (cid:1) , wherex ℓ − 1 i ∈ R n is the i -th feature vector in layer ℓ − 1 , θ ℓij is a trainable parameter and σ ( . ) is a nonlinearactivation function, x ℓj is the j -th feature activation vector in layer ℓ .
We can further write thiscompactly in matrix notation, this gives
[Equation]
where Θ ℓ ∈ R N ℓ − 1 × N ℓ is the weight-matrix of layer ℓ and X ℓ ∈ R n × N ℓ contains the activationsoutput by layer ℓ .
The GCN model updates node features at every node by averaging over the nodefeatures of the node itself and its neighbors, which enforces similarity throughout node neighborhoods.
This makes nodes increasingly difficult to discriminate for ‘deeper’ models and is widely referredto as the oversmoothing problem [Li et al., 2018b].
From a graph spectral theory perspective,oversmoothing is related to lowpass filtering as the filter in Eq.
1 puts larger weights on the lowfrequency spectrum (as 0 ≤ λ i ≤ 2 ).
This motivates the idea of using GCN-type lowpass filters inconjunction with bandpass filters, that can be implemented, for example, using graph scattering [Minet al., 2020], which we discuss in the following.
2.3 Graph Scattering
The graph scattering transform [Gama et al., 2019, Zou and Lerman, 2019, Gao et al., 2019] is a wavelet-based model for machine learning on graphs.
Unlike the one-hop localized low-pass filters used in GCN, which aim to promote smoothness between neighboring nodes, these wavelets are band-pass filters, which in turn incorporate long-range dependencies through the large spatial support of the used aggregations.
The scattering transform is based upon raising the lazy random walk matrix
[Equation] to different powers in order to capture the diffusion geometry of the graph G at various time scales.
In particular, subtracting such powers allows us to detect changes in these diffusion geometries.
Following the lead of Coifman and Maggioni [2006], for k ∈ N 0 , we define a wavelet matrixΨ k ∈ R n × n at scale 2 k by
[Equation]
Intuitively, at every node, these diffusion wavelets can be seen as a comparison operation thatcalculates the difference of the averaged features of two neighborhoods of different sizes (namelysizes 2 k − 1 and 2 k ).
The geometric scattering transform is constructed through an alternating cascadeon wavelet filterings and pointwise nonlinearities.
Towards this end, we let σ : R → R be a nonlinearfunction which is nonexpansive in the sense that | σ ( x ) − σ ( y ) | ≤ | x − y | , and for a scattering pathp :
= ( k 1 , . . .
, k m ) , we define
[Equation]
Model
Our method is centered around three main components, (i) a hybrid scattering-GCN model M (Sec. 3.1-3.3) that transforms a small set of simple node-level statistics represented by the matrix
X ∈ R n × d to a probability vector p ∈ [0 , 1] n representing the probabilities that each node is part of the maximal clique, (ii) an easy-to-optimize unsupervised loss function L (Sec. 3.4) that is small for “good” probability vectors p ⋆ ; and (iii) a rule-based decoder D (Sec. 3.5) that maps the probability vector to the maximal clique C ⋆ ⊂ V of G .
[Equation]
3.1 Embedding Module
The input of our model is a node feature matrix X ∈ R n × d , containing d features for each node.
In our experiments, we set d = 3 and let the features be the eccentricity, the clustering coefficient, and the logarithm of the degree of each vertex.
The encoder transforms the node features to d h -dimensional embeddings H 0 using a multi-layer perceptrone (MLP) m
emb : R d → R d h , i.e.,
H 0 :
= m emb ( X ) .
We will use H 0 as the input to the diffusion model introduced in the next subsection.
3.2 Diffusion Module
The aggregation module consists of a cascade of K ∈ N aggregation (or diffusion) layers withoperations that are chosen for each node via an attention mechanism.
Our network is based uponthe framework introduced in Wenkel et al.
However, our method differs from Wenkel et al.[2022] by storing every intermediate node representation H ℓ , 0 ≤ ℓ ≤ K , in a list of readouts .
In each layer ℓ , every node has access to node representations from set a of filters F that contains aselection of lowpass and bandpass filters.
Similar to Wenkel et al.
[2022], our bandpass filters f low ,rare modified GCN filters that have the form
[Equation]
and are parameterized by the power r ≥ 1 of the matrix A .
Similarly, we define the scattering filterf band ,k of order k according to
[Equation]
Next, we want to obtain data-driven scores s f ( v ) that determine the importance of filter f ∈ F fornode v ∈ V .
We set H ℓf := f
( H ℓ − 1 ) and calculate the scores via an attention mechanism, setting
[Equation]
which, at every node, calculates the dot product of the learnable attention vector a ℓ ∈ R 2 d h with thefiltered node representation of the node concatenated to its previous representation.
We normalizethe scores across the filters using the softmax function, i.e., α f ( v ) =
softmax F ( s f ( v )) , and use thecorresponding attention vectors α ℓf ∈ R n , f ∈ F to update node representations via
[Equation]
(cid:88) [Equation]
Notably, the softmax function enforces that every node is likely to focus on one single filter.
We notethat the method is compatible with multi-head attention but omit further details as it is not necessaryfor the present experiments.
Finally, we transform the aggregated node representations via an MLPm ℓ : R d h → R d h , i.e., H ℓ := m ℓ ( H ℓ agg ) .
3.3 Output Module
In the output module, we combine the information from the readouts H ℓ , 0 ≤ ℓ ≤ K .
We first concatenate the readouts horizontally to
H
[Equation]
and then transform H cat to a vector h ∈ R n using an MLP, i.e., h :
= m out ( H cat ) .
Finally, to obtain a probability vector, we apply min-max normalization, i.e., p :=
( h − min( h ) · 1 n ) / (max( h ) − min( h )) , and interpret p [ i ] as the probability that the i -th node is part of the maximal clique.
3.4 Training Loss
We now derive an unsupervised training loss for the maximal clique problem.
Letting Ω denote the set of cliques, we note that the task of finding the maximal clique is equivalent to identifying the clique C ∈ Ω that contains the most edges, i.e., maximizes the objective function
[Equation]
Our approach relies on producing a vector p such that p v ≈ 1 if v is in the maximal clique andp v ≈ 0 otherwise.
We consider the vector x ∈ R n where x v = 1 if v is in the maximal clique andx v = {+
+} otherwise.
We then model x v as a Bernoulli random variable where P ( x v = 1) =
p v .
Ourgoal then becomes to maximize the expectation
[Equation]
Maximizing (9) encourages p to be large on the support of the largest clique.
However, we also thesupport of p to be concentrated within a clique.
This naturally motivates the constrained optimizationproblem of minimizing
[Equation]
subject to: supp( p ) is contained in a clique .
[Equation]
[Equation]
However, the constraint that p is contained in a clique is hard to enforce directly.
In order toconstruct a surrogate, we consider the complement graph G =
( V , E ) .
The vertices of G aretaken to be the same as G , i.e., V = V , while the edges are defined by the rule that for u ̸ = vthere is an edge between u and v in G if and only if there is not an edge between them in G , i.e.,
E = { ( u, v ) ∈
V × V : u ̸ = v }\ E .
We denote the adjacency matrix of G by W = ( w u,v ) u,v ∈ V ,and note that [Equation]
In particular, W has zeros on the diagonal and can be computed from W via W = 1 n × n − ( I + W ) .
The following Lemma indicates that a second loss term L 2 ( p ) :
= p ⊤ W p can be used to ensure thatmass is primarily concentrated in a set of nodes that form a clique.
Lemma 1. Consider a graph G =
( V, E ) , a signal p ≥ 0 and define supp( p ) :
= { v ∈ V : p v > 0 } .
Then, L 2 ( p ) =
p ⊤ W p =
if and only if there exists a clique C ⊂ V such that the support of p iscontained in C .
Proof.
We first assume p ⊤ W p = 0 and suppose for the sake of a contradiction that there exists no clique C ⊂ V with supp( p ) ⊂ C .
Then, there exist at least two nodes u 0 , v 0 ∈ supp( p ) with { u 0 , v 0 } ̸∈ E .
Now, [Equation]
which is a contradiction.
For the opposite direction, we assume that there exists a clique C ⊂ V such that supp( p )
⊂
C .
Hence, every pair of distinct nodes in supp( p ) is connected by an edge and consequently w u,v =for all u, v ∈ supp( p ) .
Thus,
[Equation]
In light of (9) and Lemma 1, we now define our training loss as
[Equation] [Equation]
We note that the loss function decomposes in two terms.
The first term, L 1 , encourages the allocationof mass at highly connected nodes, while the second term L 2 encourages most of the mass of p to becontained in a clique.
β is a hyper-parameter which is used to balance the contributions of the twoterms.
3.5 Decoder
In order to extract the MC from p , we use a greedy decoder detailed in in Alg. 1.
Our algorithm creates a set Ω
= { ˆ C j } κj =1 of κ cliques and then selects the ˆ C j with largest cardinality.
To construct the ˆ C j , we first create a sorting function π : V → { 1 , . .
, | V |} which puts the nodes in descending order, so
[Equation]
Each ˆ C j starts as a single vertex π − 1 ( j ) .
We then consider τ − 1 , additional candidate, π − 1 ( j + 1) , . . .
, π − 1 ( τ ) and accept each candidate vertex v canditate if ˆ C j ∪ { v candidate } forms a clique.
This is tested by verifying that x ⊤ W x = 0 for
[Equation]
We remark that in order for our decoder to predict true maximal clique exactly, we must choose theparameters τ and κ such that there exists a j , 1 ≤ j ≤ κ , such that j ≤ π − 1 ( i ) ≤ j + τ for allv i in the MC
On simple datasets, such as IMDB, it suffices to set κ = 1 .
More complex datasets
Order nodes via permutation π : V → [ n ] such that p π −
≥ p such as Twitter require larger values of κ .
Importantly, we note that the ˆ C j can be computed inparallel, so increasing the value of κ does not create major scalability issues.
We note that setting thehyperparameters of the decoder requires some prior knowledge of the largest sizes of the maximalcliques present in the data, which we tune using the validation set.
Results
We empirically experiment on three popular graph learning baselines, namely IMDB, COLLAB [Yanardag and Vishwanathan, 2015] and TWITTER [Yan et al., 2008].
We compare both against otherneural network based methods and against an integer-programming-based solver.
For the the neuralnetworks, Erdös’ GNN [Karalias and Loukas, 2020] and RUN-CSP
[Toenshoff et al., 2019], theauthors provide two implementations, one optimized towards computational speed (fast), and theother optimized for best approximation score.
The integer-programming-based solver, GUROBI9.0[Gurobi Optimization, LLC, 2022], is in general more accurate than neural network based methods.
However, it is slower on larger graphs.
In order to highlight the contribution of our hybrid, scatteringmodel, we also provide results for a pure lowpass model, which only uses GCN-type layers [Kipfand Welling, 2016] followed by our decoder module.
We provide our results in Tab. 1, where wetrack each model’s test approximation score and the average time needed to classify one graph ofthe dataset. Dataset statistics can be found in Tab. 2.
For a given graph, the approximation score iscomputed by dividing the size of the clique found by the algorithm by the true MC.
So if the MC of agraph has size 10 and the algorithm returns a clique of size 9, it would be given a score of 90%.
Wethen average these scores over all graphs in each dataset.
For the IMDB dataset, our hybrid approach and most other methods (except RUN-CSP) solve the MCproblem with a perfect (100%) approximation score.
However, our model significantly outperformsthe other neural network-based models (Erdös’ GNN and RUN-CSP) with respect to computationtime (although the highly optimized GUROBI solver is slightly faster than our method on this datasetdue to the small graph size).
On COLLAB, our model almost reaches perfect performance (99.8%) in0.035 S/G, outperforming the other neural network based methods in both speed and approximationscore.
Some versions of the Gurobi solver are able to achieve 99.9% and 100% approximation scores,however, they are slower than our method on this dataset.
Lastly, we experimented on the TWITTERdataset, the most challenging of the datasets considered here.
Among neural networks, RUN-CSP(accurate) does exceptionally well at 98.7% approximation score our method is second best with 95%,while running faster than all compared neural networks, in particular, 3 times faster than RUN-CSP.
Figure 3 illustrates the differences between our hybrid scattering model and the low-pass GCN-basedmodel[Kipf and Welling, 2016] and helps explain why the scattering-based model performs betteracross all three datasets.
It shows that the probability vector p generated by our hybrid scatteringmodel is a more discriminative representation than the one produced by the low-pass model.
Whenusing the low-pass model
cannot tell which nodes have higher priorities given such a smooth output.
This leads to accepting nodes which are not part of the MC.
If the new clique is not a sub-clique of the MC, [Table] IMDB [Table]
TWITTER
[Table]
brackets) on test graphs for the different baseline methods compared on across datasets.
We set the number of samplers, κ , equal to 1, 3 and 10 on IMDB, COLLAB and TWITTER, respectively.
We mark the two best neural network models with highest prediction accuracy bold and underlined and the best one.
In case of the same performance, we use lower computation time as a tiebreaker.
We provide results for GUROBI with 4 different time budgets.
[Table]
in turn can lead to the decoder rejecting nodes which are part of the MC.
The scattering model onthe other hand produces a less smooth representation.
The nodes outside the MC are assigned lowerprobabilities so that our decoder can successfully find the MC.
This is consistent with previous work[Wenkel et al., 2022] showing that the graph scattering transform helps overcome the oversmoothnessproblem in node classification.
We emphasize that both of these models are trained with the same lossfunction and the same rule-based decoder to predict the maximal clique.
Therefore, the differencesillustrated in Figure 3 are the direct result using scattering.
More comparison between the scatteringmodel and low-pass model can be found in the appendix.
D ATASET
[Table]
To further explore how our model is compares to the current state-of-the-art commercial solver
GUROBI 9.0, we introduce a new dataset that contains three classes of graphs with different difficulties (easy, medium and hard) for the maximal clique prediction.
The dataset generation is based onthe approach in Xu et al.
[2005], where we set the hardness parameter to 0.2, 0.5 and 0.8, respectively,to generate the easy, medium and hard cases, as shown in Table 4.
Each hardness class case contain1,000 graphs.
We notice that our model outperforms GUROBI under limited time budget.
Theresults are shown in Table 3.
Notably, the scattering model is very efficient when dealing withmedium hardness cases, where our model is 20 times faster than GUROBI when achieving similarapproximation score.
[Table]
Overall, we remark that our hybrid model provides considerable speedup compared to both theother neural networks and to GUROBI for large graphs.
We also note that our model takes fewerparameters than the Erdös’ GNN model, which uses MLPs with 512 hidden units for each layer,while in our model, whereas the MLPs used in our method have either 8, 16, or 20 hidden units on
IMDB, COLLAB, and TWITTER, respectively.
This in turn suggests that the use of both GCN andscattering filters leads to a powerful feature extractors that help learn informative node representationswith relatively few parameters.
Conclusion
In this paper, we show the expressive power of GNNs can be critical for solving graph CO problems.
Our results suggest that low-pass models that generally enforce smoothness over graph neighborhoods due to oversmoothing, make the nodes indistinguishable, which leads to unfavorable noderepresentations for the MC problem.
Inspired by the geometric scattering transform, we proposea novel scattering-based GNN to overcome the oversmoothing problem.
We further construct anovel two-term loss function which encourages our network both to find a large set of nodes and tofind a set of nodes which are contained within a clique, which, in conjunction mimic the otherwisehard to optimize problem of finding maximal cliques.
Our scattering model is able to generateefficient representations that enable us to learn probabilities for each node to be part of the maximalclique.
Based on these node representations we approximate the correct maximal cliques using aconstraint-preserving greedy decoder.
Overall, our performance is competitive with other neuralnetworks-based methods and commercial solvers in terms of time and accuracy.
A Checklist
Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Yes2.
Have you read the ethics review guidelines and ensured that your paper conforms to them?
Yes3.
Did you discuss any potential negative societal impacts of your work?
No4.
Did you describe the limitations of your work?
No
Did you state the full set of assumptions of all theoretical results
Yes
Did you include complete proofs of all theoretical results?
Yes
Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
We will include a link to an
anonymous github repository in the supplementary material.
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Yes
Did you report error bars (e g., with respect to the random seed after running experiments
multiple times)?
Yes, we report the standard deviation of our model.
Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
No