G RAPH ENS:N EIGHBOR -A WARE E GO N ETWORK S YNTHESIS FOR C LASS -I MBALANCED N ODE C LASSIFICATION
A BSTRACT
In many real-world node classiﬁcation scenarios, nodes are highly classimbalanced, where graph neural networks (GNNs) can be readily biased to major class instances.
Albeit existing class imbalance approaches in other domains can alleviate this issue to some extent, they do not consider the impact of message passing between nodes.
In this paper, we hypothesize that overﬁtting to the neighbor sets of minor class due to message passing is a major challenge for class-imbalanced node classiﬁcation.
To tackle this issue, we propose GraphENS , a novel augmentation method that synthesizes the whole ego network for minor class (minor node and its one-hop neighbors) by combining two different ego networks based on their similarity.
Additionally, we introduce a saliency-based node mixing method to exploit the abundant class-generic attributes of other nodes while blocking the injection of class-speciﬁc features.
Our approach consistently outperforms the baselines over multiple node classiﬁcation benchmark datasets.
I NTRODUCTION
Node classiﬁcation on graphs has attracted signiﬁcant attention as the importance of large-scale graphs analysis increases in various domains such as bioinformatics and commercial graphs to name a few (Perozzi et al., 2016; Hamilton et al., 2017;
Ying et al., 2018; Mohammadrezaei et al., 2018).
For example, in retail services, acquiring the qualitative node representations for items or customers is critical for improving the quality of recommendation systems (Perozzi et al., 2016;
Ying et al., 2018).
Detecting abnormal users in social networks, as another example, is also closely related to classifying the property of each node (Mohammadrezaei et al., 2018).
Recently, Graph Neural
Networks (GNNs) have demonstrated their effectiveness on learning node representations (Hamilton et al., 2017; Kipf & Welling, 2017; Velickovic et al.).
However, the nodes in many real-world graphs are inherently class-imbalanced (Mohammadrezaei et al., 2018; Wang et al., 2020a), hence GNNs are prone to be biased toward major classes, as in general class-imbalance tasks.
This bias forces networks to poorly classify the nodes of minor classes, resulting in destructive impacts and a large cost to their services.
While the peculiar characteristics of imbalanced node classiﬁcation and specialized solutions suitable for it have hardly been investigated, applying generic imbalance handling methods (Chawla et al., 2002;
Cui et al., 2019; Cao et al., 2019; Kang et al., 2020; Menon et al., 2021) directly to the graph domain has several non-trivial challenges.
One of the distinct natures of graph data is that adjacent nodes are involved in constructing the representation of each node, which makes the model more confused to learn the unbiased representation of minor class nodes.
Here, we hypothesize that it is in fact more serious to overﬁtting to neighbors of minor nodes than to overﬁtting to the node feature itself.
This ‘neighbor memorization’ is the critical obstacle to naively adopt the class-imbalance approaches of other domains such as re-weighting and re-sampling used in image classiﬁcation.
Speciﬁcally, re-weighting approaches (Cui et al., 2019; Tan et al., 2020; Cao et al., 2019; Menon et al., 2021; Hong et al., 2021), applying penalties according to the number of data, simply assign large weight to minor nodes, hence there is no change in neighbor sets of minor nodes observed during training.
Re-sampling methods (Chawla et al., 2002; Kang et al., 2020; Zhang et al., 2021; Wang et al., 2021), sampling data to balance the number of data for each class, are also vulnerable to overﬁt on minor nodes with their neighbors.
Another challenge of the re-sampling method, especially for oversampling variants, is determining how to connect the newly sampled nodes to the original graph.
For example, simply connecting an oversampled node with all neighbors of original node will change the edges of neighbor nodes as well and hence signiﬁcantly alters the message passing to the neighbors, which might impair their class semantics.
To mitigate this issue,
GraphSMOTE (Zhao et al., 2021) exploits edge predictor to decide the connectivity with the neighbors of two minor nodes used in oversampling.
Nevertheless, GNNs trained with GraphSMOTE still suffer from neighbor memorization when the number of minor nodes is limited.
In this paper, we propose GraphENS , a novel augmentation approach that synthesizes the whole ego network for minor class (minor node and its one-hop neighbors) by interpolating two different ego networks in the data; to enlarge the limited neighbor views of minor instances in the data, our method combines the ego network of anchoring minor node with that of randomly selected node from all classes, where it interpolates ego networks based on KL-divergence between model predictions of ego networks in order to keep the semantics of minor classes.
Synthesized ego networks are attached to the original graph to construct a class-balanced graph, and GNNs are trained with the enlarged graph.
GraphENS enables the model to learn the minor class nodes with feasible neighbors by generating the virtual ego networks.
To further prevent the synthesis of deleterious ego networks, we introduce a saliency-based node mixing approach to generate the central node of ego network.
Our method separates class-generic node features from class-speciﬁc node features by using saliency information of each feature, and exploits only class-generic attributes to combine with node feature of anchoring minor node.
We validate our method on various benchmark datasets including citation networks (Sen et al., 2008), and Amazon product co-purchasing networks (Shchur et al., 2018) with diverse architectures such as GCN (Kipf & Welling, 2017), GAT (Velickovic et al.), and GraphSAGE (Hamilton et al., 2017), and conﬁrm that our approach consistently outperforms baseline methods over various settings.
In summary, our contribution is threefold:
• We demonstrate that in class-imbalanced node classiﬁcation, GNNs severely overﬁt to neighbor sets of minor class nodes, rather than to minor nodes themselves.
This ‘neighbor memorization’ problem becomes severe when the number of minor nodes is extremely small.
• Our method effectively alleviates the neighbor memorization problem in class-imbalanced graphs by synthesizing feasible ego networks based on the similarity between source ego networks.
We also block the injection of harmful features in generating the mixed nodes using node feature saliency.
• Through extensive experiments, we show that our approach consistently outperforms baselines on multiple benchmark datasets including real-world imbalanced datasets.
Even in highly imbalanced synthetic graphs, our method exhibits superior performance.
R ELATED W ORK AND P RELIMINARY 2. C LASS I MBALANCE P ROBLEM
The goal of class-imbalance handling in classiﬁcation is to construct an unbiased classiﬁer to the label distribution of the training set.
There are three main streams: loss modiﬁcation, post-hoc correction, and re-sampling approaches.
Loss modiﬁcation methods alter the objective function to assign more weights (Japkowicz & Stephen, 2002; Cui et al., 2019) or margins (Tan et al., 2020; Cao et al., 2019; Menon et al., 2021) on minor classes.
Post-hoc correction strategies (Kang et al., 2020;
Tian et al., 2020; Menon et al., 2021;
Hong et al., 2021) remedy logits to compensate minor classes in the inference.
Re-sampling approaches augment minor class data by sampling strategies (Kang et al., 2020; Liu et al., 2020;
Ren et al., 2020) or generation (Chawla et al., 2002;
Kim et al., 2020a;
Chu et al., 2020; Zhang et al., 2021; Wang et al., 2021).
Among minor class generation approaches,
SMOTE (Chawla et al., 2002) is a widely used method to mix minor data with the nearest data of the identical class.
Synthesizing minor class data from data of other classes (Kim et al., 2020a; Chu et al., 2020; Wang et al., 2021) is introduced to exploit the rich information of other classes.
Kim et al.
(2020a) produces new minor class data by taking gradient steps to translate major class data into minor class data.
Wang et al.
(2021) synthesizes minor class data by combining features of minor class data with feature displacements of other data.
To extend these approaches to the graph domain, structural aspects of graph have to be considered when generating minor instances.
In node classiﬁcation, imbalance handling works (Zhou et al., 2018; Wang et al., 2020b; Shi et al., 2020; Zhao et al., 2021; Qu et al., 2021) are proposed to exploit structural information in graphs.
DR-GCN (Shi et al., 2020) produces the virtual minor nodes generated by additional conditional
GAN and regularizes the features of virtual nodes close to adjacent nodes.
GraphSMOTE (Zhao et al., 2021) generates synthetic minor nodes by interpolating two minor class nodes and a (pretrained) edge predictor determines the connectivity of synthetized nodes between synthetized nodes and neighbors of two source minor nodes.
ImGAGN
Qu et al., 2021) synthesizes minor nodes by interpolating features among whole minor nodes with the generated weight matrix.
Then the synthesized nodes are connected to the original minor nodes if weights in the matrix are larger than a ﬁxed threshold.
As GraphSMOTE and ImGAGN only utilize nodes of the identical minor class to generate minor nodes, the sample diversity of synthesized nodes would be signiﬁcantly constrained.
Moreover, ImGAGN mainly targets binary classiﬁcation and its extension to multi-class classiﬁcation is non-trivial since an independent generator is required per each class.
Compared to these approaches, our GraphENS utilizes whole nodes to synthesize minor nodes, thus our method outperforms baselines when the number of minor classes is low in Section 5.3 (Table 4).
2. G RAPH N EURAL N ETWORKS FOR N ODE C LASSIFICATION
We brieﬂy introduce GNNs for node classiﬁcation tasks.
Let us ﬁrst deﬁne graph G ( V, E ) where
V is the set of nodes and E is the set of edges between two nodes.
Let X ∈ R | V |× d be the node features whose the i -th row represents the d -dimensional feature of the i -th node.
N ( v ) is the set of adjacent nodes that are directly connected to v : { u ∈ V |{ u, v } ∈ E } .
In node classiﬁcation, each node in the graph corresponds to a class y ∈ { 1 , . . .
, C } where C is the number of classes.
In this paper, we consider several variants of GNNs that consists of three differentiable functions: 1) message function m l , 2) permutation invariant message aggregation function φ l , and 3) node update function h l .
Let x ( l ) v be the latent vector of node v at layer l .
To simplify notations for the recursive deﬁnition of GNNs, we use x (0) v to denote the input node feature.
At each GNN layer, node features are updated as x ( l +1) v = h l ( x ( l ) v , φ l ( { m l ( x ( l ) v , x ( l ) u , e v,u ) | u ∈ N ( v ) } ) to consider adjacent node embeddings.
By passing these aggregated node features to a linear classiﬁer as input, we obtain the prediction ˆ y =
f
( x v ) for node v where ˆ y c
=
P
( y = c | x v ) .
As a representative example, a layer of Graph Convolutional Network (GCN) (Kipf & Welling, 2017) is deﬁned as x ( l +1) v = Θ (cid:80)
u ∈N ( v ) ∪{ v } e v,u √ ˆ
d
u ˆ d v x ( l ) u where ˆ d v = 1 + (cid:80) u ∈N ( v ) e v,u with e v,u is the edge weight of edge { u, v } ∈ E and Θ is a ﬁlter parameters.
There are several variants depending on how these key components are designed, but they are beyond scope of our work.
N EIGHBOR M EMORIZATION P ROBLEM
In this section, we deﬁne and investigate the neighbor memorization problem uniquely appearing in the node classiﬁcation task of a class-imbalanced graph.
Conventional imbalance handling approaches such as re-weighting and oversampling enable the model to train in a class-balanced manner.
However, these methods are vulnerable to overﬁt toward minor classes (Zhou et al., 2020) since they solely assign large weights on limited minor instances in the training phase.
In the node classiﬁcation task, the unique characteristics of this task, which have not been considered until now, should be considered: not only overﬁtting to the minor node feature v minor , but also overﬁtting to
N ( v minor ) can be an issue.
That is, the model is exposed to highly restricted neighboring structures for minor classes.
Here, since GNN models are based on message passing that receives information from (a number of) neighboring nodes, we hypothesize that the overﬁtting and memorizing
N ( v minor ) can be more serious than overﬁtting to v minor , and we experimentally verify that below.
Experimental setup We ﬁrst conduct experiments to observe overﬁtting to minor classes on classimbalanced graphs.
GNNs are trained with two widely used methods - re-weighting and over- [Figure] sampling on a node classiﬁcation benchmark dataset, PubMed (Sen et al., 2008).
We process the
PubMed dataset to follow a long-tailed distribution by well-known construction rule in the vision domain (Cui et al., 2019).
The imbalance ratio between the numbers of the most frequent class and the least frequent class is set as 100.
We ﬁx architecture as the 2-layer GraphSAGE havinghidden dimension and train models for 2000 epochs.
We select the model by the validation accuracy.
Overﬁtting to minor classes
The learning curves of imbalance handling approaches are presented in Figure 1 (a) and (b).
The solid lines and the dash lines are the curves of minor class accuracy and overall accuracy, respectively.
We observe that the test accuracy of the minor class is much lower than the overall accuracy while the train accuracy of the minor class is extremely high.
This result implies that the existing approaches are prone to memorize minor class nodes.
Note that we only present the learning curves for the ﬁrst 200 epochs to show the tendency of the early phase more clearly.
We conﬁrm the test accuracy differences between minor classes and all classes are almost maintained until the last epoch for both methods.
The learning curve for entire epochs is in
Appendix A.2 (Figure 4).
In the following paragraph, we investigate more speciﬁcally whether this overﬁtting is mainly due to memorizing node features or their neighbor structures .
Neighbor memorization problem To investigate how seriously GNN is prone to be overﬁtting to self node features or aggregated features from neighbors, we design two ‘replacing’ experiments.
In the ﬁrst replacing experiment, we scrutinize ‘node memorization’ where GNNs memorize seen minor node features excessively.
We intentionally replace a seen minor node with an unseen minor node given a ﬁxed seen neighbors (node-replacing experiment).
Toward this, we randomly sample three nodes from the identical minor class: we sample v anchor (recipient), v seen (donator) from the training set and v unseen (donator) from the test set.
We then replace the feature of v anchor with that of v unseen in the original graph, and evaluate the classiﬁcation accuracy of replaced v anchor .
That is, the replaced feature of v anchor is not exposed during training, but its neighbor view does.
For comparison, we replace the feature of v anchor with of v seen as a baseline; it is the case of seen node features and neighbors but it has the artifact of grafting.
We measure the accuracy for all minor test nodes and experiments are repeated 50 times.
In Figure 1(c), the blue and red bars represent the results of v anchor with seen node features and v anchor with unseen node features, respectively.
Now, we conduct a similar experiment for ‘neighbor memorization’.
We would like to estimate how much GNN relies on seen topological structures.
To this end, we measure the performance degradation when the seen neighbor set is replaced by an unseen neighbor set given a ﬁxed seen selfnode feature (neighbor-replacing experiment).
We sample v seen (donator) and v anchor (recipient) from the training set and v unseen (recipient) from the test set as before, but v anchor now has a different role compared to the node-replacing experiment: it serves as providing a seen neighbor set.
Speciﬁcally, we replace the feature of v unseen with that of v seen , which represents the situation where a seen node is embedded into unseen neighbors.
Then, we evaluate the classiﬁcation accuracy of replaced v unseen .
For a baseline, the node feature of v anchor is replaced by that of v seen and measures the accuracy of replaced v anchor ; it is the case of seen node feature with a seen neighbor set.
In Figure 1(d), the blue bar denotes the performance of seen node feature with seen neighbor set
[Figure]
and the red bar represents the performance of seen node feature with an unseen neighbor set.
The overall procedure is described by Figure 8 in Appendix D.
We demonstrate that conventional imbalance handling approaches severely suffer from a neighbor memorization problem in Figure 1(c) and (d) * .
The performance drop of conventional algorithms in the neighbor-replacing experiment is steeper than in the node-replacing setting.
These results imply that the neighbor memorization problem is a critical obstacle in properly handling the classimbalanced problem in node classiﬁcation tasks.
From the learning curves and replacing experiments, we ﬁnd that conventional approaches to handle imbalanced data largely rely on the neighbor information.
This neighbor memorization problem results in poor generalization performance on the minor class, so it must be appropriately addressed in class-imbalanced node classiﬁcation.
Recent work, GraphSMOTE (Zhao et al., 2021), connects a mixed node to neighbors of both the source node and target node in the same class, and then removes the unreliable edges by the feature similarity-based edge predictor.
However, since GraphSMOTE only utilizes the neighbors of the seed nodes in the identical class, it still prone to memorize the neighbors of minor classes when the number of minor class nodes is extremely limited.
G RAPH ENS
We now describe our novel class-imbalance handling strategy for node classiﬁcation, GraphENS that synthesizes an ego network for generated minor nodes considering the prediction similarity of the ego network instances, in order to alleviate the neighbor memorization described above.
GraphENS is mainly composed of two core components.
Both components exploit the features from two nodes: the minor class node v minor and the target node v target from entire classes * to synthesize an ego network of minor class.
The ﬁrst component is to augment the neighbor set by combining neighbors of a minor node v minor with those of the target node v target .
To prevent the generation of harmful ego networks, GraphENS regulates the combination ratio depending on the KL divergence between ego networks centered on v minor and v target , respectively (Section 4.1).
The second component of GraphENS deﬁnes the node feature saliency for GNNs that represents the importance of each node feature in classifying the node property.
Equipped with this node feature saliency, GraphENS mixes v minor and v target to generate a central node and combine with the ego network synthesized above (Section 4.2).
This feature saliency information is utilized to ﬁlter out the class-speciﬁc node features in the node feature mixing process to preserve the semantics of the minor class.
The overall procedure of GraphENS is described in Figure 2.
Our full algorithm is also provided in the
Appendix E (Algorithm 1).
We discuss the details of each component in the following subsections
4. N EIGHBOR S AMPLING
Neighbor sampling is a method to determine which nodes to be connected to a new mixed node, v mixed (how to synthesize a mixed node is described in Section 4.2).
The goal of neighbor sampling is to expose minor nodes to various environments to mitigate neighbor memorization problem while preventing the generation of harmful connections.
To provide diverse neighbor sets, we construct ‘adjacent node distribution’ for a mixed node from two ego networks of v minor and v target and stochastically sample the neighbors from this distribution.
At the same time, regarding the avoidance of detrimental connectivity, we devise a scheme that reﬂects N ( v target ) more in constructing the adjacent node distribution when the ego network of v target is more similar to that of v minor .
Note that the ego network G v of node v consists of {N v ∪ v } nodes and edges directly connected to the central node v .
The overall scheme of neighbor sampling is in Figure 2 (a).
Constructing adjacent node distribution Now, we build the similarity-aware adjacent node distribution for v mixed by mixing adjacent node distributions of v minor and v target .
Here, the adjacency node distribution for a real node v , p ( u | v ) , is deﬁned as p ( u | v ) =
1 |N ( v ) | if u ∈ V, { u [Equation] v } ∈ E , and p ( u | v ) =
otherwise.
To construct a ‘probable’ adjacent node distribution, the mixing ratio is determined based on the similarity between the ego networks of G v minor and G v target .
Toward this, we utilize logits o v = f ( G ) v for a node v given the full graph G as input of GNN f since
GNN aggregates neighbor information with self node feature.
It is worth noting that the logits can be obtained without additional cost (we can bring the model conﬁdences at the previous epoch).
To further reﬂect the neighbor information explicitly and avoid the overconﬁdence problem, we aggregate adjacent logits and utilize these logits to compute the similarity.
Speciﬁcally, the aggregated logit of node v is computed as ˆ o v =
[Equation]
| +1 (cid:80) u ∈ ( N ( v ) ∪ v ) o u .
Then, we calculate the
KL-divergence between ˆ o minor and ˆ o target to estimate the distance between G v minor and G v target : φ = KL ( σ (ˆ o minor ) (cid:107) σ (ˆ o target )) where σ is a softmax function.
Now, the normalized distance is used as the mixing ratio for the minor adjacent node distribution, and it is calculated as ˆ φ = 1 1+ e − φ .
Finally, the adjacent node distribution for the synthesized v mixed is deﬁned as
[Equation]
Note here that the mixing ratio of the target node decreases as the distance from the minor node increases and the mixing ratio of anchoring minor node is guaranteed to be at least 0 .
Sampling neighbors Given the adjacent node distribution p ( u | v mixed ) in Equation 1, we sample neighbors from this distribution without replacement.
The number of neighbors is sampled from the degree distribution of a given graph to keep degree statistics with the original graph.
Not only that, we make the message passing work only on the incoming edges to v mixed since undirected edges affect message passing in the original graph.
4. S ALIENCY - BASED N ODE M IXING
We now describe how our method synthesizes new central node v mixed of G v mixed by interpolating two nodes v minor and v target .
Note that our method does not restrict the target node to be from minor classes to utilize the abundant class-generic information of other classes.
To extract only generic attributes independent of class, we ﬁrst deﬁne the node feature saliency to identify the importance of each node feature.
Armed with this node feature saliency, GraphENS synthesizes new minor node v mixed using minor node features and generic features of target node via convex combination.
Feature saliency Albeit GraphENS does not depend on speciﬁc ways of calculating the importance of each node attribute, we simply adopt an approach that evaluates the importance of node features based on gradients.
In particular, motivated by the works using the gradient of the loss function with respect to input features in image classiﬁcations (Simonyan et al., 2013; Shrikumar et al., 2017), we deﬁne the feature saliency as follows.
We ﬁrst compute the gradient of classiﬁcation loss L ( G, y ) with respect to the input node feature matrix ∂ L ∂X ∈ R | V |× d via backpropagation.
Then, the saliency value of i -th feature s ( v,i ) is obtained by the magnitude of the gradient regarding i -th attribute in node v as s ( v,i ) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:104) ∂
L ( G, y ) ∂X (cid:105) ( v,i ) (cid:12)(cid:12)(cid:12)(cid:12) .
The feature saliency vector S v =
[ s ( v, 1) , ..., s ( v,d ) ]
T is derived without any modiﬁcation to the GNN architecture.
Our gradient-based feature saliency can be derived without additional backpropagation as we simply exploit the gradient values computed from the previous training iteration.
Node mixup We introduce our simple mixup-like augmentation to oversample the minor nodes without label mixing.
Since we select target nodes v target from all classes unlike SMOTE (Chawla et al., 2002), ﬁltering (or masking) out class-speciﬁc attributes of the target node is required to prevent the generation of noisy samples.
To this end, we explicitly utilize the feature saliency information of the target node.
First, we determine the masking ratio K % using a normalized distance ˆ φ (from Section 4.1).
As ˆ φ implies the distance between two ego networks centralized by v minor and v major , we adaptively assign K % proportional to normalized distance ˆ φ ; K =
k ˆ φ where k is a hyperparameter.
The intuition behind here is if target node v target is signiﬁcantly differ from minor node v minor , we mask more salient features of v target .
Speciﬁcally, given the feature saliency vector S v target , a binary mask M K ∈ R d , which masks the K % of node attributes to 0, is sampled from multinomial distribution by applying softmax to S v target .
Our ﬁnal synthesized minor nodes v mixed can be formulated as follows given a random mixing ratio λ ∼ Beta ( α, α ) :
[Equation]
We also evaluate our method but without using saliency on multiple datasets in Section 5.4 (Table 3).
E XPERIMENT 5. E XPERIMENT SETUP
Datasets We validate GraphENS on ﬁve benchmark datasets: Cora, CiteSeer, PubMed for citation networks (Sen et al., 2008), AmazonPhoto and AmazonComputers for co-purchase graphs (Shchur et al., 2018).
For citation networks, we follow splits in Yang et al.
In supervised learning setting, we construct long-tailed citation networks following Cui et al.
to validate models at a high imbalance ratio, which denotes the ratio between the most frequent class and the least frequent class.
Nodes are removed until the class distribution follows a long-tailed distribution with keeping the connection in graphs at most.
We sort the classes from major class to minor class and then remove nodes for each class started from major classes.
When eliminating nodes, we remove nodes having low degrees and the corresponding edges of those nodes.
We also evaluate imbanlance handling methods on natural imbalanced datasets such as AmazonPhoto and AmazonComputers.
To make validation/test sets balanced, we sample the same number of nodes from each class for validation/test sets.
Then, the remaining nodes are assigned to the training set.
We ﬁx the least number of train nodes for all classes as 20.
In semi-supervised setting, we set the imbalance ratio as 10.
The detailed setup is provided in Appendix F.5.
Baselines We test our methods over three architectures as GCN (Kipf & Welling, 2017),
GAT (Velickovic et al.), and GraphSAGE (Hamilton et al., 2017).
We adopt re-weight as costsensitive loss (Japkowicz & Stephen, 2002) inversely proportional to the number of class data.
Oversampling is the approach that samples each class node until the number of each class data reaches the maximum number of class data.
For oversampling, we duplicate the edges of the original node when adding an oversampled node to the original graph.
cRT (Kang et al., 2020) and
PC Softmax (Hong et al., 2021) are recent effective baselines for decoupling and post-hoc correction approaches, respectively.
DR-GCN generates virtual minor nodes and forces virtual nodes to be similar to the neighbors of a source node.
Although DR-GCN has an additional component to exploit entire unlabeled nodes for semi-supervised learning, we do not consider this component for a fair comparison.
GraphSMOTE (Zhao et al., 2021) has two versions, depending on whether the predicted edges are discrete or continuous values.
We adopt a discrete version of GraphSMOTE since it exhibits superior performance on multiple benchmark datasets.
Implementation details of ours and description of evaluation protocol are deferred to Appendix F.
[Table] [Table]
[Table]
5. T ARGET N ODE S ELECTION
[Figure]
Our method utilizes the target node v target from all classes, not only from the identical minor class of v minor .
The intention of constructing the target node pool from entire classes is to guarantee sufﬁcient diversity of synthesized ego networks.
If the target nodes are restricted to a minor class at highly imbalanced scenarios, identical neighbor nodes would be redundantly duplicated.
This issue makes it hard to mitigate the neighbor memorization problem.
To demonstrate this hypothesis, we conduct an experiment and compare our design choice (all classes) to selecting a target from the identical class as the minor node v minor .
In the Figure 3, we conﬁrm that exploiting entire classes for the target node achieves superior performances.
From this design choice of the target node, our method can utilize the manifold assumption that ‘similar predictions of neural networks indicate the close proximity in the manifold’, which are commonly utilized in semi-supervised learning (Van Engelen & Hoos, 2020).
We aim to enlarge and smooth the decision boundary of the minor class by interpolating the minor nodes and the target node of the entire classes (but excluding target-speciﬁc attributes using saliency) as investigated in Verma et al.
Moreover, our method considers the prediction of the ego network (not a single node) to reﬂect the structural aspects of graph.
To select a target node, we sample a target class from the multinomial distribution with ( logN 1 , logN 2 , . . .
, logN C ) where N i is the number of the i -th class data and C is the number of classes.
Then, we randomly select a target node from the training nodes of the sampled class. [Table]
[Table]
M AIN R ESULTS
Neighbor sampling mitigates neighbor memorization
We investigate neighbor memorization over conventional algorithms in Section 3.
We validate ours with respect to overﬁtting to minor classes, node memorization, and neighbor memorization.
Although our training accuracy of the minor class is similar to re-weighting and oversampling, the test accuracy of our method is much higher than baselines, which indicates that our approach mitigates overﬁtting to minor classes in
Figure 1 (a) and (b).
Our method signiﬁcantly decreases the performance degradation when seen neighbor set is replaced by an unseen neighbor set, implying that our approach can generalize well for unseen neighbor views compared to baselines (Figure 1 (d)).
Neighbor sampling thus effectively alleviates the neighbor memorization by exposing minor nodes to diverse circumstances.
Supervised Learning We evaluate GraphENS on highly class-imbalanced situations as most realworld graphs.
First, we test our method on three long-tailed citation networks: Cora- LT , CiteSeerLT , and PubMed- LT .
In Table 1, our method brings better imbalance handling performance in most cases compared to other baselines.
To verify the effectiveness of our method on real-world imbalanced graphs, we also test GNNs trained with GraphENS on naturally class-imbalanced benchmark datasets without any modiﬁcations in graphs.
Our method outperforms other contenders by significant margins in AmazonComputers (Table 2).
Note that our method consistently exhibits higher accuracy and F1 score in other architectures (Table 6 in Appendix B.1).
Semi-supervised learning To validate our approaches when labeled nodes are extremely restricted, we conduct experiments in a semi-supervised setting on citation networks (Table 4).
Our approach signiﬁcantly outperforms baselines for all datasets.
Ours also exhibits consistent superior performances in other GNN architectures (Table 7).
Note that since GraphENS exploits entire classes to synthesize minor nodes, our method can avoid the generation of redundant nodes when the number of minor nodes is extremely low.
Hence, our method signiﬁcantly improves the performance against existing oversampling based approaches- GraphSMOTE, Oversampling.
Ours brings further performance gain by utilizing unlabeled nodes to generate ego networks (Appendix B.2).
5. A BLATION S TUDY
We verify two key components of our method: ‘prediction similarity’ on neighbor sampling and ‘saliency masking’ on node mixing.
To this end, we introduce baselines: 1) ‘ GraphENS w/o PS’ determines prediction similarity uniformly random, 2) ‘w/o SM’ mixes nodes without saliency masking ( M K = 1 ), and 3) ‘w/o SM,NS’ generates mixed minor nodes without masking and connecting to neighbors of the target .
We ﬁnd that removing each module drops performance (Table 3).
Thus, we believe each module effectively contributes to restrain the generation of harmful ego networks.
C ONCLUSION
We investigated that existing imbalance handling algorithms suffer from neighbor memorization problem in class-imbalanced node classiﬁcation.
Thus, we proposed a novel augmentation method,
GraphENS , to synthesize an ego network for minor classes with neighbor sampling and saliencybased node mixing.
We veriﬁed that our method effectively mitigates neighbor memorization by synthesizing diverse but probable minor ego networks.
GraphENS demonstrated its effectiveness in that it outperforms baselines over multiple benchmark datasets with various GNN architectures.
A CKNOWLEDGEMENT
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No.2019-0-00075, Artiﬁcial Intelligence Graduate School Program (KAIST), No.2019-0-01371, Development of brain-inspired AI with human-like intelligence,
No.2021-0-02068, Artiﬁcial Intelligence Innovation Hub) and the National Research Foundation of Korea (NRF) grants (No.2018R1A5A1059921) funded by the Korea government (MSIT). This work was also supported by Samsung Electronics Co., Ltd (No.IO201214-08133-01) A A UXILIARY R ESULTS FOR N EIGHBOR M EMORIZATION P ROBLEM
In this section, we present experimental results which are not reported in the Section 3 due to the space constraints.
A. F URTHER I NVESTIGATION FOR N EIGHBOR M EMORIZATION
In Section [Table] we verify our hypothesis - easy to overﬁt to neighbor structures - on the basis of the most simple imbalance handling approaches. To further strengthen our claim
[Table]
we intensively examine the neighbor memorization problem with other recent baselines over ﬁve benchmark datasets.
As can be seen from Table [Table]
we observe that baselines consistently suffer from neighbor memorization issues in most cases. In addition, our approach shows signiﬁcantly superior generalization performance than baselines on both node and neighbor replacing experiments except for node-replacing experiments on PubMed- LT .
[Table]
We conducted experiments with 2-layer GCN having 256 hidden dimensions and adopted PC Softmax and GraphSMOTE as recent algorithms since PC Softmax and GraphSMOTE outperform cRT and DR-GCN in the main experiments, respectively. Note that for these experiments we group classes with less than 100 training nodes as minor classes.
A. L EARNING C URVE OF THE T RAINING P HASE
To show the early trend of the training process, we only show the learning curve for the ﬁrstepochs in our main paper.
We observe that the gap between training and test accuracy is almost maintained over entire epochs in Table 4. [Figure] cc (
[Figure]
B A DDITIONAL E XPERIMENTS ON M ULTIPLE S ETTINGS B. R ESULTS OF A MAZON P HOTO AND A MAZON C OMPUTERS
Due to space constraints
[Table]
we only present the experiment with GraphSAGE in the main paper, which exhibits the best vanilla performances, on two co-purchase graph datasets - AmazonPhoto,
AmazonComputers.
We also test our method with two different GNN architectures - GCN, GAT.
As shown in Table [Table] GraphENS shows the best performances over baselines in most cases.
Detailed experimental settings are described in Section F.
Interestingly
[Table]
we observe that cRT and PC Softmax show inferior performance than other algorithms on AmazonComputers (Table 6).
We conjecture that these approaches fail to learn discriminative representations of minor classes due to message-passing in a highly imbalanced graph.
Decoupling methods such as cRT and PC Softmax are proposed based on the observation that retraining the classiﬁer with class-balanced sampling after training on imbalanced datasets show comparable performance with baselines (Kang et al., 2020).
In other words, learning distinctive representation is not signiﬁcantly affected by class imbalance.
However, oversmoothing caused by message passing impedes learning distinctive representation of minor classes in highly imbalanced datasets.
It is consistent with the observation that decoupling methods fail at the extremely high imbalance ratio (Ren et al., 2020) [Table]
[Table]
B. R ESULTS OF S EMI -S UPERVISED L EARNING
In Section 5.
[Table]
we report the results of GCN on three citation networks - Cora, CiteSeer, PubMed.
We here additionally offer the result table of two different GNN architectures - GAT, GraphSAGE.
GraphENS outperforms baselines in most cases (Table 7).
To ensure that our method brings improvement for any data split (other than public split), we assess
GraphENS on randomly selected train nodes while keeping the public split of validation/test nodes unmodiﬁed (Table 8).
We conﬁrm that the improvements of our method is not depending on a particular data split.
We also attempt to extend GraphENS to utilize unlabeled nodes and validate this extended method.
First, we regard unlabeled data as the ( C + 1) -th class ( C is the number of classes) and this class could be sampled when selecting a target node.
Since the number of unlabeled data is large, the labeled class could be sampled less frequently if we utilize the number of unlabeled data in the multinomial distribution in Section 5.2.
Thus, we treat the unlabeled class as the class having the same number of data with the least frequent class and sample a target class from the distribution with ( logN 1 , logN 2 , . . .
logN C , logN C +1 ) where N i is the number of the i -th class data.
Note that we utilize model prediction on unlabeled nodes for neighbor sampling and saliency maps of source nodes for saliency masking.
To keep class-speciﬁc information of source nodes, [Table] exploitation of saliency maps of source nodes is required.
In Table 8, the adoption of unlabeled data in GraphENS improves the performance signiﬁcantly in most cases.
We believe that our approach could exploit rich information even in unlabeled nodes to produce safe and diverse ego networks [Table] [Table]
C I N - DEPTH A NALYSIS C. C LASS - WISE P ERFORMANCE A NALYSIS
To clearly present alleviation of overﬁtting to minor classes
[Figure]
we provide the test accuracy curves for each class.
Note that PubMed- LT has three classes and the number of nodes for each class is as follows: ( Minor 1: 72, Minor 2: 726, Major: 7260 ).
The experiment settings are identical to Section 3.
In Figure 5 (a)
[Figure]
while the training accuracies for minor classes ( Minor 1, Minor 2 ) are highly similar over entire methods
[Figure]
GraphENS exhibits signiﬁcantly superior test accuracies for minor classes (Figure 5 (b,c)).
Albeit our method slightly sacriﬁces the test accuracy of the major class
[Figure]
we conﬁrmed that our approach substantially mitigates overﬁtting to the minor classes
[Figure]
C. V ISUALIZATION OF THE HIDDEN REPRESENTATIONS
In this subsection
[Figure]
we shed light on why our algorithm works.
Our method aims to enlarge and smooth the decision boundary of the minor class.
First, to extend the boundary of the minor class,
GraphENS utilizes the manifold assumption that “similar predictions of neural networks indicate the close proximity in the manifold” (Van Engelen & Hoos, 2020; Iscen et al., 2019) as a key inductive bias.
By synthesizing the minor ego network using the target node of the entire class, our method widens the decision boundary effectively in that we interpolate the ego networks based on prediction similarity.
In the vision domain, M2m (Kim et al., 2020b), one of the imbalance-handling methods, also translates major class data into minor class to prevent overﬁtting on minor classes exploiting our assumption implicitly.
This extrapolation scheme for the minor class becomes more signiﬁcant when the number of data is highly scarce.
[Figure]
Another advantage of our method for node classiﬁcation is that the generalization performance of
GNNs can be improved by smoothing the decision boundary.
As investigated in Verma et al.
(2019), convex combinations of node features encourage the smoothness of the boundary.
To verify the smoothness and enlargement of minor classes, we visualize the decision boundaries of Re-weighting and our method by using 2D bottleneck hidden representations of GNN in Figure 6.
Note that we conduct experiments on Cora-LT with 2 layers of GraphSAGE.
Compared to Re-weighting, we observe that our boundaries are smoother and the fewest class occupies more region with signiﬁcantly large margin.
Therefore, along with the main results, this result implies that our method could improve generalization performance by enlarging the decision boundary of minor classes smoothly.
C. O VERSAMPLING R ATIO
We investigate the performance tendency according to oversampling scale, which determines the number of generated minor nodes.
Speciﬁcally, the number of oversampled nodes is computed as oversampling ratio * (the number of nodes in the largest class - the number of nodes in the minor class)).
In Figure 7, we observe that the performance decreases as the scale is (1) under 0.6 or (2) over 1.0.
We conjecture that the induced bias of major classes is not sufﬁciently corrected at low oversampling scales (1).
In the regime of (2), a high oversampling scale impedes the learning of major class representations, resulting in performance degradation.
Thus, the oversampling scale between 0.8 and 1.0 is a preferable option and we adopted 1.0 as the scale in our main experiments, following conventional oversampling methods.
[Figure]
GraphSMOTE
Oversampling
D I LLUSTRATIONS OF REPLACING EXPERIMENTS
To facilitate understanding of node/neighbor replacing experiments, we describe the experiments with illustrations.
In Figure 8, to observe node memorization, we replace the node feature of v anchor with v seen in (a) and v anchor with v unseen in (b).
The performance gap between v anchor in (a) and (b) is computed.
Similar to node-replacing experiments, to see neighbor memorization, we replace the node feature of v seen with v anchor in (c) and v seen with v unseen in (d).
The accuracy difference between v seen in (c) and v unseen in (d) is calculated.
The results of replacing experiments are in Figure 1.
E M AIN A LGORITHM
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
: Model f
[Table] [Table]
t
[Table] [Table] [Table]
[Table]
do 4:
[Table] [Table]
V , E 5: for c
[Table] [Table]
do 6: for
[Table] [Table] [Table]
[Table]
[Table]
) 11: if t
[Table]
10 then
[Table]
Warmup 12:
[Table] [Table]
[Table]
15: else 16:
[Table]
[Table]
18: 19: (cid:46)
[Table] [Table]
Compute M
[Table]
from multinomial distribution using
[Table]
, K 22:
[Table]
[Table] [Table]
sampling 26:
[Table]
27: r ∼
[Table]
degree
[Table]
D ) 28: {
[Table]
n 1 , v n 2 , · · · , v n r } ∼ p ( u | v mixed ) 29: ˆ E new ← {{ v n 1 , v mixed } , { v n 2 , v mixed } , · · · , { v n r , v mixed }} 30: end if 31: V new ← V new ∪ { v mixed } 32:
E new ← E new ∪ ˆ E new 33: end for 34: end for 35: o ← f θ ( V new , E new ) 36: L new ← CrossEntropy ( o , y ) 37: θ ← θ − η ( ∇ θ L new ) 38: for v ∈ V do 39: ˆ o tv ← 1 |N ( v ) | +
[Table]
u
[Table]
[Table]
v ) ∪ v ) Softmax ( o u /τ )
(cid:46)
Conﬁdence aggregation 40: end for 41:
S
42: end for 43: Output: f
F D ETAILED E XPERIMENT S ETTINGS
In this section
[Table]
we address all the details of experiments we conducted.
F. D ATA STATISTICS
[Table]
F. A RCHITECTURE
We evaluate our method with three representative GNN architectures - GCN (Kipf & Welling, 2017),
GAT (Velickovic et al.)
[Table]
and GraphSAGE (Hamilton et al., 2017).
In this section, we describe the detailed architecture of each GNN.
For GCN, our model consists of l GCN layers with ReLU activation, followed by dropout
Srivastava et al.
(2014) with 0.5 dropping rate and a linear classiﬁer.
For GAT, our model comprises l GAT layers, which involves an ELU activation Clevert et al.
(2016), followed by dropout with 0.5 dropping rate and a linear classiﬁer.
We also adopt multi-head attention with 8 heads and apply dropout with 0.6 dropping rate to its attention.
GraphSAGE is composed of l SAGE layers with ReLU activation, followed by dropout with 0.5 dropping rate and a linear classiﬁer.
We search the architecture to maximize the validation accuracy for all algorithms.
The search spaces of reported results are the number of layers l ∈ { 1 , 2 , 3 } and hidden dimension d ∈ { 64 , 128 , 256 } .
F. E VALUATION P ROTOCOL
We adopt Adam (Kingma & Ba, 2015) optimizer with the initial learning rate as 0.01 and train GNNs for 2000 epochs.
The best models are selected with validation accuracy.
We design a scheduler as a learning rate is halved if there is no improvement on validation loss for 100 iterations.
We choose weight decay to all convolutional layers except for a linear classiﬁer with 0 . 0005 .
We select the best model using validation accuracy over the training phase.
F. I MPLEMENTATION D ETAILS
In this subsection, we explain the hyperparameters of our experiments.
For all datasets, we use
Beta (2 , 2) distribution to sample λ .
Feature masking hyperparameter k and temperature τ are tuned among { 1 , 5 , 10 } and { 1 , 2 } , respectively.
We employ warmup since our approach utilizes model conﬁdence, so models need to be trained to some extent.
The number of epochs for warmup is tuned among { 1 , 5 } .
F. T HE DETAILED SETUP FOR SEMI - SUPERVISED LEARNING
We validate imbalance handling approaches on the citation networks such as Cora, CiteSeer, and
PubMed.
We follow the split in Yang et al.
and set the imbalance ratio as 10.
Speciﬁcally, for Cora and CiteSeer, three classes have only two labeled nodes while other classes have 20 labeled nodes since the number of labeled nodes in each class is 20 in Yang et al.
For PubMed, only two classes have two labeled nodes as PubMed consists of three classes.