F EW -S HOT LEARNING ON GRAPHS VIA SUPERCLASSES BASED ON GRAPH SPECTRAL MEASURES
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose to study the problem of few-shot graph classiﬁcation in graph neural networks (GNNs) to recognize unseen classes, given limited labeled graph examples.
Despite several interesting GNN variants being proposed recently for node and graph classiﬁcation tasks, when faced with scarce labeled examples in the few-shot setting, these GNNs exhibit signiﬁcant loss in classiﬁcation performance.
Here, we present an approach where a probability measure is assigned to each graph based on the spectrum of the graph’s normalized Laplacian.
This enables us to accordingly cluster the graph base-labels associated with each graph into super-classes, where the L p Wasserstein distance serves as our underlying distance metric.
Subsequently, a super-graph constructed based on the super-classes is then fed to our proposed GNN framework which exploits the latent inter-class relationships made explicit by the super-graph to achieve better class label separation among the graphs.
We conduct exhaustive empirical evaluations of our proposed method and show that it outperforms both the adaptation of state-ofthe-art graph classiﬁcation methods to few-shot scenario and our naive baseline
GNNs.
Additionally, we also extend and study the behavior of our method to semi-supervised and active learning scenarios.
I NTRODUCTION
The need to analyze graph structured data coupled with the ubiquitous nature of graphs (Borgwardt et al., 2005; Duvenaud et al., 2015; Backstrom & Leskovec, 2010; Chau et al., 2011), has given greater impetus to research interest in developing graph neural networks (GNNs) (Defferrard et al., 2016; Kipf & Welling, 2016; Hamilton et al., 2017; Velikovi et al., 2018) for learning tasks on such graphs.
The overarching theme in GNNs is for each node’s feature vector to be generated by passing, transforming, and recursively aggregating feature information from a given k -hop neighborhood surrounding the node.
However, GNNs still fall short in the ”few-shot” learning setting, where the classiﬁer must generalize well after seeing abundant base-class samples (while training) and very few (or even zero) samples from a novel class (while testing).
Given the scarcity and difﬁculty involved with generation of labeled graph samples, it becomes all the more important to solve the problem of graph classiﬁcation in the few-shot setting.
Limitations and challenges: Recent work by Xu et.
al.
(Xu et al., 2019) indicated that most recently proposed GNNs were designed based on empirical intuition and heuristic approaches.
They studied the representational power of these GNNs and identiﬁed that most neighborhood aggregation and graph-pooling schemes had diminished discriminative power.
They rectiﬁed this problem with the introduction of a novel injective neighborhood aggregation scheme, making it as strong as the
Weisfeiler-Lehman (WL) graph isomorphism test (Weisfeiler & Leman, 1968).
Nevertheless, the problem posed by extremely scarce novel-class samples in the few-shot setting remains to persist as a formidable challenge, as it requires more rounds of aggregation to affect larger neighborhoods and hence necessitate greater depth in the GNN.
However, when it comes to
GNNs, experimental studies have shown that an increase in the number of layers results in dramatic performance drops in GNNs (Wu et al., 2019; Li et al., 2018b).
Our work: Motivated by the aforementioned observations and challenges, our method does the following.
We begin with a once-off preprocessing step.
We assign a probability measure to each graph, which we refer to as a graph spectral measure (similar to (Gu et al., 2015)), based on the spectrum of the graph’s normalized Laplacian matrix representation.
Given this metric space of graph spectral measures and the underlying distance as the L p Wasserstein distance, we compute
Wasserstein barycenters (Agueh & Carlier, 2011) for each set of graphs speciﬁc to a base class and term these barycenters as prototype graphs .
With this set of prototype graphs for each base class label, we cluster the spectral measures associated with each prototype graph in Wasserstein space to create a super-class label.
Utilizing this super-class information, we then build a graph of graphs called a super-graph .
The intuition behind this is to exploit the non-explicit and latent inter-class relationships between graphs via their spectral measures and use a GNN on this to also introduce a relational inductive bias (Battaglia et al., 2018), which in turn affords us an improved sample complexity and hence better combinatorial generalization given such few samples to begin with.
Given, the super-classes and the super-graph, we train our proposed GNN model for few-shot learning on graphs.
Our GNN consists of a graph isomorphism network (GIN)
Xu et al.
(2019) as a feature extractor F θ ( . ) to generate graph embeddings; on which subsequently acts our classiﬁer
C ( . )
comprising of two components: (i) C sup : a MLP layer to learn and predict the super class associated to a graph, and (ii) C GAT : a graph attention network (GAT) to predict the actual class label of a graph.
The overall loss function is a sum of the cross-entropy losses associated with C sup and
C GAT .
We follow initialization based strategy (Chen et al., 2019), with a training and ﬁne-tuning phase, so that in the ﬁne-tuning phase, the pre-trained parameters associated with F θ ( . ) and C sup are frozen, and the few novel labeled graph samples are used to update the weights and attention learned by C GAT .
Our contributions: To the best of our knowledge, we are the ﬁrst to introduce few shot learning on graphs for graph classiﬁcation.
Next, we propose an architecture that makes use of the graph’s spectral measures to generate a set of super-classes and a super-graph to better model the latent relations between classes, followed by our GNN trained using an initialization method.
Finally, we conduct extensive experiments to gain insight into our method.
For example, in the 20 -shot setting on the TRIANGLES dataset, our method shows a substantial improvement of nearly 7% and 20% over DL-based and unsupervised baselines, respectively.
R ELATED W ORK
Few-shot learning in the computer vision community was ﬁrst introduced by (Fei-Fei et al., 2006) with the intuition that learning the underlying properties of the base classes given abundant samples can help generalize better to unseen classes with few-labeled samples available.
Various learning algorithms have been proposed in the image domain , among which a broad category of initialization based methods aim to learn transferable knowledge from training classes, so that the model can be adapted to unseen classes with limited labeled examples (Finn et al., 2017); (Rusu et al., 2018); (Nichol et al., 2018).
Recently proposed and widely accepted Initialization based methods can broadly be classiﬁed into: (i) methods that learn good model parameters with limited labeled examples and a small number of gradient update steps (Finn et al., 2017) and (ii) methods that learn an optimizer (Ravi & Larochelle, 2017).
We refer the interested reader to Chen et.
al.
(Chen et al., 2019) for more examples of few-shot learning methods in vision.
Graph neural networks (GNNs) were ﬁrst introduced in (Gori et al., 2005); (Scarselli et al., 2009) as recurrent message passing algorithms .
Subsequent work (Bruna et al., 2014); (Henaff et al., 2015) proposed to learn smooth spectral multipliers of the graph Laplacian, but incurred higher computational cost.
This computational bottleneck was later resolved (Defferrard et al., 2016); (Kipf & Welling, 2016) by learning polynomials of the graph Laplacian.
GNNs are a natural extension to
Convolutional neural networks (CNNs) on non-Euclidean data.
Recent work (Velikovi et al., 2018) introduced the concept of self-attention in GNNs, which allows each node to provide attention to the enclosing neighborhood resulting in improved learning.
We refer the reader to (Bronstein et al., 2016) for detailed information on GNNs.
Despite all the success of GNNs, few-shot classiﬁcation remains an under-addressed problem.
Some recent attempts have focused on solving the few-shot learning on graph data where GNNs are either trained via co-training and self-training (Li et al., 2018a), or extended by stacking transposed graph convolutional layers imputing a structural regularizer (Zhang et al., 2019) - however, both these works focus only on the node classiﬁcation task.
To the best of our knowledge, there does not exist any work pertaining few-shot learning on graphs focusing on the graph classiﬁcation task, thus providing the motivation for this work.
Comparison to few-shot learning on images: Few shot learning (FSL) has gained wide-spread traction in the image domain in recent years. However the success of FSL in images is not easily translated to the graph domain for the following reasons: (a) Images are typically represented in Euclidean space and thus can easily be manipulated and handled using well-known metrics like cosine similarity, L p norms etc. However, graphs come from non-Euclidean domains and exhibit much more complex relationships and interdependency between objects. Furthermore, the notion of a distance between graphs is also not straightforward and requires construction of graph kernels or the use of standard metrics on graph embeddings (Kriege et al., 2019).
Additionally, such graph kernels dont capture higher order relations very well.
(b) In the FSL setting on images, the number of training samples from various classes is also abundantly more than what is available for graph datasets.
The image domain allows training generative models to learn the task distribution and can further be used to generate samples for data augmentation, which act as very good priors.
In contrast, graph generative models are still in their infancy and work in very restricted settings.
Furthermore, methods like cropping and rotation to improve the models can’t be used for graphs given the permutation invariant nature of graphs.
Additionally, removal of any component from the graph can adversely affect its structural properties, such as in biological datasets.
(c)
The image domain has very well-known regularization methods (e.g. Tikhonov, Lasso) that help generalize much better to novel datasets.
Although, they dont bring any extra supervised information and hence cannot fully address the problem of FSL in the image domain.
To the best of our knowledge, this is still an open research problem in the image domain.
On the other hand, in the graph domain, our work would be a ﬁrst step towards graph classiﬁcation in an FSL setting, which would then hopefully pave the path for better FSL graph regularizers.
(d) Transfer learning has led to substantial improvements on various image related tasks due to the high degree of transferability of feature extractors.
Thus, downstream tasks like few-shot learning can be performed well with high quality feature extractor models, such as Resnet variants trained on Imagenet.
Transfer learning, or for that matter even good feature extractors, remains a daunting challenge in the graph domain.
For graphs, there neither exists a dataset which can serve as a pivot for high quality feature learning, nor does there exist a Graph
NN which can capture the higher order relations between various categories of graphs, thus making this a highly challenging problem.
P RELIMINARIES
In this section, we introduce our notation and provide the necessary background for our few-shot learning setup on graphs.
We begin by describing the various data sample types, followed by our learning procedure, in order to formally deﬁne few-shot learning on graphs.
Finally, we deﬁne the graph spectral distance between a pair of graphs.
Data sample sets: Let G denote a set of undirected unweighted graphs and Y be the set of associated class labels.
We consider two disjoint populations of labeled graphs consisting of i.i.d.
graph samples, the set of base class labeled graphs G B = { ( g ( B ) i , y ( B ) i ) } ni =1 and the set of novel class labeled graphs G N = { ( g ( N ) i , y ( N ) i ) } mi =1 , where g ( B ) i , g ( N ) i ∈ G , y ( B ) i ∈ Y ( B ) , and y ( N ) i ∈ Y ( N ) .
Here, the set of base and novel class labels are denoted by Y ( B ) = { 1 , . . .
, K } and Y ( N ) =
{ K +1 , . . .
, K (cid:48) } , respectively, where K (cid:48) >
K .
Both Y ( B ) and Y ( N ) are disjoint subsets of Y , so, Y ( B ) ∩ Y ( N ) = ∅ .
Note that m (cid:28) n , i.e., there are far fewer novel class labeled graphs compared to the base class labeled ones.
Besides G B and G N , we consider a set of t unlabeled unseen graphs G U :=
{ g ( U ) 1 , . . .
, g ( U ) t | g ( U ) i ∈
π 1 ( G N ) , i = 1 . . .
t } , for testing 1 .
Learning procedure: Inspired by the initialization based methods , we similarly follow a two-stage approach of training followed by ﬁne-tuning .
During training, we train a graph feature extractor F θ ( G B ) with network parameters θ followed by a classiﬁer C ( G B ) on graphs from G B , where the loss function is the standard cross-entropy loss
L c .
In order to better recognize and generalize well on samples from novel classes, in the ﬁnetuning phase, the pre-trained feature extractor F θ ( . ) along with its trained parameters is ﬁxed and the classiﬁer C ( G N ) is trained on the novel class labeled graph samples from G N , with the same loss L c .
Now, given the classiﬁcation of data samples and the two-stage learning method, our problem of few-shot classiﬁcation on graphs can be deﬁned as follows.
Problem deﬁnition:
Given n base-class labeled graphs from G B during the training phase and m novel-class labeled graphs from G N during the ﬁne-tuning phase, where m (cid:28) n , the objective of few-shot graph classiﬁcation is to classify t unseen test graph samples from G U .
Moreover, if m = qT , where T = K (cid:48) − K , i.e., each novel class label appears exactly q times in G N , then this setting is referred to as the q -shot, T -way learning.
Graph spectral distance: Let us consider the graphs in G .
The normalized Laplacian of a graph g ∈ G is deﬁned as ∆ g
=
I − D − 1 / 2 AD 1 / 2 , where A and D are the adjacency and the degree matrices of graph g , respectively.
The set of eigenvalues of ∆ g given by { λ i } | V | i =1 is called the spectrum of ∆ g and is denoted by σ ( g ) .
It is well known that the spectrum σ ( g ) of a normalized
Laplacian matrix is contained in interval [0 , 2] .
We assign a Dirac mass δ λ i concentrated on each λ i ∈
σ ( g ) , thus associating a probability measure to σ ( g ) supported on [0 , 2] , called the graph spectral measure µ σ ( g ) .
Furthermore, let P ([0 , 2]) be the set of probability measures on interval [0 , 2] .
We now deﬁne the p -th Wasserstein distance between probability measures, which we later use to deﬁne the spectral distance between a pair of graphs.
Deﬁnition 1 Let p ∈
[1 , ∞ )
and let c : [0 , 2] × [0 , 2] → [0 , + ∞ ] be the cost function between the probability measures µ, ν ∈ P ([0 , 2]) .
Then the p -th Wasserstein distance between measures µ and ν is given by
[Equation]
where Π( µ, ν ) is the set of transport plans , i.e., the collection of all measures on [0 , 2] × [0 , 2] with marginals µ and ν .
Given the general deﬁnition of the p -th Wasserstein distance between probability measures and the graph spectral measure , we can now deﬁne the spectral distance between a pair of graphs in G .
Deﬁnition
Given two graphs g, g (cid:48) ∈ G , the spectral distance between them is deﬁned as
[Equation]
In words, W p ( g, g (cid:48) ) is the optimal cost of moving mass from the graph spectral measure of graph g to that of graph g (cid:48) , where the cost of moving unit mass is proportional to the p -th power of the difference of real-eigenvalues in interval [0 , 2] 2 .
O UR M ETHOD
We present our proposed approach here.
First, given abundant base-class labels, we cluster them into super-classes by computing prototype graphs from each class, followed by clustering the prototype graphs based on their spectral properties.
This clustering of prototype graphs induces a natural clustering on their corresponding class labels, resulting in super-classes (as outlined in Section 4.1).
These super-classes are then used in the creation of a super-graph used further down by our GNN.
Note that the creation of super-classes, followed by building a super-graph are a once-off process. [Figure]
The prototype graphs as well as the super-classes for the base classes can be stored in memory for further use.
Next, we explain our graph neural network’s architecture which comprises of a feature extractor
F θ ( . ) and a classiﬁer C ( . ) , described in Section 4.2.
The classiﬁer C ( . ) is further subdivided into a classiﬁer C sup that predicts the superclass of a graph feature vector and a graph attention network (GAT) C GAT to predict the graph’s class label.
Figure 1 illustrates the training and ﬁne-tuning phases of our GNN.
4. C OMPUTING S UPER CLASSES
In order to exploit inter-class relationships between base-class labels, we cluster them in the following manner.
First, we partition the set G B into class-speciﬁc sets G ( i ) , for i = 1 . . .
K , where G ( i ) is the set of graphs with base-class label i .
Thus, G B = (cid:70) Ki =1 G ( i ) .
Then, we compute class prototype graphs for each class-speciﬁc set.
The class prototype graph for class i represented by p i is given by
[Equation]
Essentially, the class prototype graph p i for the i -th class is the graph with the least average spectral distance to the rest of the graphs in the same class.
Given these K prototypes, we cluster them using
Lloyd’s method (also known as k -means) 3 .
Clustering prototype graphs: Given K unlabeled prototypes p 1 , . .
, p K ∈ π 1 ( G B ) and their associated spectral measures µ σ ( p 1 ) , . .
. , µ σ ( p K ) ∈ P ([0 , 2]) .
We rename the spectral measures as s 1 , . . .
, s K to ease notation.
Thus, our goal is to associate these spectral measures to at most k clusters, where k ≥ 1 is a user deﬁned parameter.
The k -means problem ﬁnds a k -partition C
= { C 1 , . .
, C k } that minimizes the following objective that represents the overall distortion error of the clustering
[Equation]
where s i is a prototype graph in cluster C i and B ( C i ) is the Wasserstein barycenter of the cluster
C i .
The barycenter is computed as
[Equation]
where s ( i, j ) denotes the j -th spectral measure in the i -th cluster C i . [Figure]
Lloyd’s algorithm:
Given an initial set of Wasserstein barycenters B (1) ( C 1 ) , . . .
, B (1) ( C k ) of spectral measures at step t = 1 , one uses the standard Lloyd’s algorithm to ﬁnd the solution by alternating between the assignment (Equation 4) and update (Equation 5) steps
[Equation]
Lloyd’s algorithm is known to converge to a local minimum (except in pathological cases, where it can oscillate between equivalent solutions).
The ﬁnal output is a grouping of the prototype graphs into k groups, which also induces a grouping of the corresponding base classes.
We denote these class groups as super-classes and denote the set of super-classes as Y sup .
4. O UR GRAPH NEURAL NETWORK
Feature extractor: To apply standard neural network architectures for downstream tasks we must embed the graphs in a ﬁnite dimensional vector space.
We consider graph neural networks (GNNs) that employ the following message-passing architecture
[Equation]
where H ( j ) ∈ R | V |× d are the node embeddings (i.e., messages) computed after j steps of the GNN and M is the message propagation function which depends on the adjacency matrix of the graph
A , the trainable parameters of the j th layer θ ( j ) , and node embeddings H ( j − 1) generated from the previous step.
A recently proposed GNN called the graph isomorphism network (GIN) by Xu et al.
(2019) was shown to be stronger than several popular GNN variants like GCN Kipf & Welling (2016) and
GraphSAGE Hamilton et al.
What makes GIN so powerful and sets it apart from the other
GNN variants is its injective neighborhood aggregation scheme which allows it to be as powerful as the Weisfeiler-Lehman (WL) graph isomorphism test.
Motivated by this ﬁnding, we chose GIN as our graph feature extractor.
The message propagation scheme in GIN is given by
[Equation]
Here, (cid:15) is a layer-wise learnable scalar parameter and MLP represents a multi-layer perceptron with layer-wise non-linearities for more expressive representations.
The full GIN model run R iterations of Equation 6 to generate ﬁnal node embeddings which we represent by H ( R ) .
As features from earlier iterations can also be helpful in achieving higher discriminative power, embeddings H ( j ) from all R iterations are concatenated as
[Equation]
Here, H ( j ) =
(cid:80) v ∈ V H ( j ) v , where H ( j ) i represents the i -th node’s embedding in the j -th iteration and (cid:107) denotes a concatenation operator.
H g now contains the graph embedding of a graph g and is passed on to the classiﬁer.
Classiﬁer: Here, our objective is to improve the class separation produced by the graph embeddings of the feature extractor F θ ( . ) and we do this by building a “graph of graph embeddings”, called a super-graph g sup , where each node is a graph feature vector.
We then employ our classiﬁer C ( . ) on this super-graph to achieve better separation among the graph classes in the embedding space.
During training, we ﬁrst build the super-graph g sup on a batch of base-labeled graphs as a collection of k -NN graphs, where each constituent k -NN graph is built on the graphs belonging to the same super-class.
g sup is then passed through a multi-layered graph attention network C GAT to learn the associated class probabilities.
The features extracted from F θ ( . ) are passed into the MLP network
C sup to learn the associated super-class labels.
C sup and C GAT combine to form our classiﬁer
C ( . ) .
The cross-entropy losses associated with C sup and C GAT are added to give the overall loss for C ( . ) .
The intuition behind the construction of g sup to train C GAT on was to further improve the existing cluster separation based on graph spectral measures by introducing a relational inductive bias (Battaglia et al., 2018) that is inherent to the GNN C GAT .
Recall that we adopt an initialization method (described in 3).
In our ﬁne-tuning stage, novel class labeled graphs from G N are input to the network.
The pre-trained parameters learned by the feature extractor F θ ( . ) are ﬁxed and C sup is used to infer the novel graph’s super-class label, followed by creation of super-graph on the novel graph samples and ﬁnally updating the parameters in C GAT through the loss. Finally the evaluation is performed on the samples from the unseen test set G U .
Discussion: We make the assumption that the novel test classes belong to the same set of super-classes from the training graphs.
The reason being that the novel class labeled samples are so much fewer than the base class labeled samples, that the resulting super-graph ends up being extremely sparse and deviates a lot from the shape of the super-graph from the base classes; therefore it severely hinders C GAT s ability to effectively aggregate information from the embeddings of the novel class labeled graphs.
Instead, we pass the novel graph samples through our trained C sup and infer its super-class label and this works very effectively for us, as is evidenced by our empirical results.
E XPERIMENTAL R ESULTS 5. B ASELINES AND D ATASETS
The standard graph classiﬁcation datasets do not adequately satisfy the requirements for few-shot learning due to the dearth of unique class labels.
Hence, we pick four new classiﬁcation datasets, namely, Letter-High , TRIANGLES , Reddit-12K , and ENZYMES .
The details and statistics for these datasets are given in Appendix A.1.
As there do not exist any standard state-of-the-art methods for few-shot graph classiﬁcation, we chose existing baselines for standard graph classiﬁcation from both supervised and unsupervised methods.
For supervised deep learning baselines, we chose - GIN (Xu et al.
, CapsGNN (Xinyi & Chen (2019)), and Diffpool (Lee et al.
We ran these methods with similar settings as ours, i.e., by partitioning the main model into feature extraction and classiﬁer sub-models to compare them in a fair and informative manner.
From the unsupervised category, we consider 4 powerful SOTA methods - AWE (Ivanov & Burnaev (2018)), Graph2Vec (Narayanan et al.
, WeisfeilerLehman subtree Kernel (Shervashidze et al.
(2011)), and Graphlet count kernel (Shervashidze et al.
Since we want to analyze the few-shot classiﬁcation abilities of these models, we essentially want to ﬁnd out how well these algorithms can achieve class separation.
We use k -NN search on the output embeddings of these algorithms.
Further conﬁguration and implementation details for the baselines can be found in Appendix A.2.
We also emphasize the beneﬁt of using a GNN as a classiﬁer by showing the adaptation of our model to semi-supervised ﬁne-tuning (in Appendix A.5) and active learning (in Appendix A.6) settings.
[Table]
5. F EW - SHOT R ESULTS
We consider two variants of our model as naive baselines .
In the ﬁrst variant, we replace our GAT classiﬁer with GCN Kipf & Welling (2016).
We call this model OurMethod-GCN .
This variant is used to justify the choice of GAT over GCN.
In the second variant [Table] we replace the entire classiﬁer with the k -NN algorithm over the features extracted from various layers of the feature extractor .
We call this variant GIN- k -NN and this is introduced to emphasize the signiﬁcance of building a super-graph and using a GAT on it as a classiﬁer to exploit the relational inductive bias.
The results for all the datasets in various q -shot scenarios, where q ∈ { 5 , 10 , 20 } are given in Table 1.
We run each model 50 times and report averaged results.
In every run, we select a different novel labeled subset G N for ﬁne-tuning the classiﬁers of the models.
The evaluation for all models is done by randomly selecting a subset of 500 samples from the testing set G U for Letter-High and TRIANGLES , whereas over 150 for ENZYMES and 300 for Reddit dataset and averaging oversuch random selections.
[Table] The results clearly show that our proposed method and its GCN variant (i [Table]
e., OurMethod-GCN ) outperform the baselines.
GIN- k -NN shows signiﬁcant degradation in results on nearly all the three paradigms for all datasets with exceptions on 5 -shot and 10 -shot on ENZYMES dataset, thus strongly indicating that the improvements of our method can primarily be attributed to our GNN classiﬁer fed with the super-graph constructed from our proposed method.
The improvements in results are higher on TRIANGLES and Reddit datasets in contrast to Letter-High and ENZYMES , which can be attributed to the smaller size of the graphs in Letter-High making it difﬁcult to distinguish based on graph spectra alone, whereas the complex and highly inter-related structure of enzymes makes it difﬁcult for the DL based feature extractors as well as the graph kernel methods to segregate the classes in the feature and graph space respectively.
GIN and WL show much better results as compared to other baselines for all the q -shot scenarios, whereas AWE and Graphlet Kernel show signiﬁcantly low results, unable to capture the properties of the graphs well.
The DL baselines apart from GIN on the other hand show improvements on the TRIANGLES dataset performing close to GIN, where [Figure] the unsupervised methods fails to capture the local node properties, however still perform poorly on other datasets.
For the 20 -shot scenario on TRIANGLES , our GAT variant shows an improvement of around 7% over DL baselines and more than 20% when compared to unsupervised methods. The substantial improvements of around 4% on Reddit dataset shows the superiority of our model for both the variants - GAT and GCN.
Furthermore, the t -SNE plots in Figure 3 show a substantial and interesting separation of class labels which strongly indicate that a good feature extractor in conjunction with a GNN perform well as a combination. The t-SNE plots for ENZYMES , Reddit , and
Letter-High are shown in ﬁgures 4, 5 and 6 respectively.
5. A BLATION S TUDY ON N UMBER OF S UPER -C LASSES
Here, we study the behavior of our proposed network model without the super-class classiﬁer C sup .
In Table 3 ( 10 and 20 -shot setting), we observe a marked increase with the addition of our classiﬁer which uses the super-class information and the super-graph based on spectral measures to guide
C GAT towards improving the class separation of the graphs during both the training and ﬁne-tuning stages.
Using super-classes help in reducing the sample complexity of the large Hypothesis space and makes tuning of the model parameters easier during ﬁne-tuning stage with less samples and few iterations.
Negligible differences are observed on ENZYMES dataset since both the number of training classes as well as test classes are low, thus, the model performs equally well on removing super-classes.
This is because of the latent inter-class representations can still be captured between few classes especially during the ﬁne-tuning phase.
[Table] 5. S ENSITIVITY A NALYSIS OF V ARIOUS A TTRIBUTES
Our proposed method contains two crucial attributes.
We analyze our model by varying: (i) the number of super-classes and (ii) the k -value in super-graph construction.
The effect of varying these attributes on model accuracy are shown in Tables 4 and [Table] respectively.
For TRIANGLES and LetterHigh datasets [Table] as we increase the number of super-classes, we observe the accuracy improving steadily up to 3 super-classes and then dropping from there onwards.
For super-classes less than 3 , we observe that the k -NN graph does not respect the class boundaries that are already imposed by the graph spectral measures [Table] thus connecting more arbitrary classes. For Reddit we observe the performance is slightly better on using 2 super-classes and for ENZYMES similar performances
[Table] [Table] are observed for both 1 and 2 super-classes as described in the ablation study. On the other hand, increasing the number of super-classes past 3 , makes each super-class cluster very sparse with few graph classes within, leading to an underﬂow of information between the graph classes.
The same effect is observed for all datasets. The k -value or the number of neighbors of each node belonging to the same connected component in the super-graph (i.e., belonging to the same super-class) is another salient parameter upon which hinges the information ﬂow (via message passing) between the graphs of the same super-class.
We analyze our model with k values in the set { 2 , 4 , 6 , 8 } and a commonly used heuristic method, whereby each graph is connected to √ b s nearest neighboring graphs based on the Euclidean similarity of their feature representations, where b s is the number of samples in the mini-batch corresponding to super-classes s .
We achieve best results with 2 -NN graphs per super-class and increasing k beyond it leads to denser graphs with unnecessary connections between classes belonging to the same super-class.
C ONCLUSION
In this paper, we investigated the problem of few-shot learning on graphs for the graph classiﬁcation task.
We explicitly created a super-graph on the base-labeled graphs and then grouped / clustered their associated class labels into super-classes , based on the graph spectral measures attributed to each graph and the L p -Wasserstein distances between them.
We found that training our GNN on the super-graph along with the auxiliary super-classes resulted in a marked improvement over stateof-the-art GNNs.
A promising future work is to propose new GNN models that break away from current neighborhood aggregation schemes to speciﬁcally overcome the obstacle posed by few-shot learning on graphs.
Our source-code and dataset splits have been made public in an attempt to attract more attention to the context of few-shot learning on graphs.
A A PPENDIX A. D ATASET DETAILS
We use 4 different datasets namely - Reddit-12K , ENZYMES , Letter-High and TRIANGLES to perform exhaustive empirical evaluation of our model on various real-world datasets varying from small average graph size on Letter-High to large graphs like Reddit-12K .
These datasets can be downloaded here 4 .
The dataset statistics are provided in Table 6, while the split statistics are provided in Table
[Table]
Dataset Description: Reddit-12K datasets contains 11929 graphs where each graph corresponds to a thread in which each node represents a user and each edge represents that one user has responded to a comment from some other user.
There are 11 different types of discussion forums corresponding to each of the 11 classes.
ENZYMES is a dataset of protein tertiary structures consisting of 600 enzymes from the BRENDA enzyme database.
The dataset contains 6 different graph categories corresponding to each different top-level EC enzyme.
TRIANGLES dataset contain 10 different classes where the classes are numbered from 1 tocorresponding to the number of triangles/3-cliques in each graph of the dataset.
Letter-High dataset contains graphs which represent distorted letter drawings from the english alphabets - A, E, F, H, I, K, L, M, N, T, V, W, X, Y, Z .
Each graph is a prototype manual construction of the alphabets.
[Table]
The validation graphs are used to assess model performance on training classes itself to check overﬁtting as well as for grid-search over hyperparameters.
The actual train-testing class splits used for this paper are provided with the code.
Since the TRIANGLES dataset has a large number of samples, this makes it infeasible to run many baselines including DL and non-DL methods.
Hence, we sample 200 graphs from each class, making the total sample size 2000 .
Similarly we downsample the number of graphs from 11929 to 1111 (nearly 101 graphs per class).
Downsampling is performed for Reddit-12K given extremely large graph sizes which makes the graph kernels as well as some deep learning baselines extremely slow.
A. B ASELINE D ETAILS
This section details the implementation of the baseline methods.
Since, DL-based methods - GIN,
CapsGNN and DIFFPOOL have not been previously run on these datasets, we select the crucial hyper-parameters - such as number of layers heuristically based on the results of standard graph classiﬁcation datasets on the best performing variants of these models.
For these three methods we take the novel layers proposed in the corresponding papers as their feature extractors, while downstream MLP layers are chosen as the classiﬁer.
The training and evaluation strategies are similar to our model, i.e., the models are ﬁrst trained in an end-to-end fashion on the training dataset G B until convergence with learning rate decay on loss plateau and then the classiﬁer layers are ﬁne-tuned over G N , keeping the parameters of the feature extractor layers ﬁxed.
For the unsupervised models - WL subtree kernel, Graphlet Count kernel, AWE and Graph2Vec, the evaluation is done using k -NN search to assess the clustering quality of these models in our few-shot scenario.
We refrain from using high-level classiﬁer models such as SVM or MLPs, since training these classiﬁers on few-shot regime will not properly assess the abilities of these models to cluster together graphs of similar class labels.
We empirically found that using high level classiﬁers resulted in higher deviations and lower mean accuracies.
We choose the hyper-parameters for these models using grid-search, since they are signiﬁcantly faster and each one of these models have few highly sensitive parameters which affect the model signiﬁcantly.
For these models, we perform a grid search for selection of k in the k -NN algorithm from the set { 1 , 2 , 3 , 4 , 5 } for the 5 -shot scenario, of which k = 1 was found to perform the best.
For higher shot scenario, the search was performed over the set { 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 } , where we again found k = 1 to be the best.
The validation set is used to check overﬁtting and hyper-parameter selection on the baseline methods.
A. O UR M ODEL D ETAILS
This section provides the implementation details of our proposed model.
Since, our feature extractor model is GIN, we maintain similar parameter settings as recommended by their paper.
As mentioned in section 4.2, using embeddings from all iterations of the message passing network helps achieve better discriminative power and improved gradient ﬂow, therefore we employ the same strategy in our feature extractor.
The number of super-classes are selected from the set { 1 , 2 , 3 , 4 , 5 } using grid-search.
The k -value for construction of super-graph was selected from the set { 2 , 4 , 6 , 8 } .
The feature extractor model uses batch-normalization between subsequent message passing layers.
We use dropout of 0 .
5 in the C sup layers.
The C GAT layers undergo normalization of inputs between subsequent layers along with a dropout of 0 .
5 , however, the normalization mechanism in classiﬁer layers is different from batch-norm.
We normalize each feature embedding to have Euclidean norm with value 1 .
Essentially,
[Equation]
where x j +1 input is the input of j + 1 th layer of classiﬁer, x jout is the output of the j th layer.
The inputs of the ﬁrst layer of C GAT also undergo the same transformation over the outputs of the feature extractor model.
We train our models with Adam (Kingma & Ba (2014)) with an initial learning rate of 10 − 3 for 50 epochs.
Each epoch has 10 iterations, where we randomly select a mini-batch from the training data G B .
The ﬁne-tuning stage consists of 20 epochs with 10 iterations per epoch.
We use a two-layer MLP over the ﬁnal attention layer of C GAT for classiﬁcation.
The attention layers use multi-head attention with 2 heads and leaky ReLU slope of 0 . 1 .
The embeddings from both the attention heads are concatenated.
For 20 -shot, we set k to 2 , number of super-classes to 3 and batch size to 128 on the Letter-High dataset, while k is set to 2 and batch size 64 on Reddit , ENZYMES and TRIANGLES datasets.
The number of super-classes for Reddit are set to 2, for ENZYMES it is set to 1 and for TRIANGLES are 3 . For ENZYMES , there are negligible differences on usingand 2 super-classes as shown in table 4.
We used Python Optimal Transport (POT) library 5 for implementation of the p -th Wasserstein distance. [Figure] [Figure]
[Figure]
A. S ILHOUETTE S CORES
To assess the clustering abilities of the models we analyze the silhouette scores of the test embeddings produced by the GAT variant of our method, GIN and WL Kernel.
Silhouette coefﬁcient essentially measures the ratio of intra-class versus inter-class distance.
The Silhouette Coefﬁcient is calculated using the mean intra-cluster distance (a) and the mean nearest-cluster distance (b) for each sample.
The Silhouette Coefﬁcient for a sample is given by ( b − a ) max ( a,b ) ,where b is the distance between a sample and the nearest cluster that the sample is not a part of.
The results for mean silhouette coefﬁcient over the test samples averaged over multiple runs are shown in Table 8.
We normalize the embeddings before calculating the silhouette coefﬁcient.
We can clearly see that our model creates better clusters with low intra-cluster distance as well as high inter-cluster distance.
Note that the coefﬁcient value for WL remains the same for all scenarios since it computes ﬁxed embeddings attributed to absence of any DL component.
[Table] [Table]
A. S EMI -S UPERVISED F INE - TUNING
In many real-world learning scenarios, it is quite common to ﬁnd abundant unlabelled data.
Since our model uses a GNN classiﬁer, this makes it possible to use unlabelled data while learning through message passing, where the ﬁne tuning stage of our method is performed in semi-supervised settings.
Essentially, while ﬁne tuning the model, i.e., only training the classiﬁer C GAT on G N , we additionally use p more graphs along with G N , whose labels are unknown.
The learning objective for ﬁne tuning stage doesn’t change since the gradients are back-propagated from the labeled samples only.
In this setting, each node in the attention classiﬁer can aggregate information from unlabelled samples as well, thus allowing improved learning of the graphs features in C GAT .
We show the results for p values 25 and 50 on Letter-High and TRIANGLES datasets, whereas for p values 10 and 20 on Reddit and ENZYMES datasets.
The results are shown in Table 9.
We observe an increase in the accuracy with increase in number of unlabeled samples during ﬁne-tuning phase.
A. A DAPTATION TO A CTIVE -L EARNING
In this section, we show the adaptation of our model to highly practical active learning scenario.
In many real world applications, we might start with few samples per class, however as the number of samples to classify from these classes increase over time, some of these samples can be used by the model to adaptively learn and improve with very less human intervention, since the number of number of samples to be queried for theirs label can always be controlled.
To perform active-learning, we ﬁrst select a random subset of size 100 for Letter-High and TRIANGLES datasets as well as a random subset of size 40 for Reddit and ENZYMES datasets, which we term as G random , then ﬁne tune the model on G N and further evaluate the model on G random . the number samples l , added to G N for second ﬁne-tuning step, where “No AL” represents the model evaluation without additional labeled samples [Table]
Thereafter, l relatively important samples are chosen from G random and added to G N for another step of ﬁne-tuning.
There can be multiple strategies for deﬁning relative importance of a sample.
For our purpose, we deﬁne a sample’s relative importance via its predicted class probability distribution.
We sort these samples in increasing order of the difference between their highest and second highest predicted class probabilities and choose the ﬁrst l samples from this sorted ranking.
We call this importance relative, since each sample is evaluated with respect to the set G N and thus, there is transductive ﬂow of information among the samples, hence deﬁning the relative embeddings in the space.
Intuitively speaking, we have chosen the samples lying closer to separation boundary with respect to G N .
The results for various values of l are shown in Table 10.
The evaluation is done as mentioned earlier on the unseen set G U .
We observe signiﬁcant improvement for all the datasets.
This shows our model is capable of selecting important samples with respect to the few existing samples and learn actively. A. P ERFORMANCE OF M ODEL WITH 1 S UPER -C LASS
From table 4 in correspondence to tables 1 and 2, one can observe that the results obtained by using only 1 super-class which is equivalent to removing the super-classes are still better in comparison with many GNN and graph kernel baselines.
By removing the super-classes and thus forming the super-graph solely based on k-nearest neighbor heuristic the GAT still learns latent inter class connections via information ﬂow better than the GNNs which use MLP as their classiﬁer.
The super graph constructed in such scenario will have arbitrary connections between classes in the beginning, however as the GIN feature extractor learns over time the segregation in the feature space increases leading to better inter as well intra-class connections.
Despite this, the performance with superclasses is better as this inductive bias allows the model to initiate with a better alignment in the feature space.
The silhouette score comparison for OurMethod-GAT with 1 super-class and with the best performing number of super-classes to GIN and WL clearly indicates the multifold beneﬁts of using GNNs
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Figure]