I NTRUSION -F REE G RAPH M IXUP
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a simple and yet effective interpolation-based regularization technique to improve the generalization of Graph Neural Networks (GNNs).
We leverage the recent advances in Mixup regularizer for vision and text, where random sample pairs and their labels are interpolated to create synthetic samples for training.
Unlike images or natural sentences, which embrace a grid or linear sequence format, graphs have arbitrary structure and topology, which play a vital role on the semantic information of a graph.
Consequently, even simply deleting or adding one edge from a graph can dramatically change its semantic meanings.
This makes interpolating graph inputs very challenging because mixing random graph pairs may naturally create graphs with identical structure but with different labels, causing the manifold intrusion issue.
To cope with this obstacle, we propose the ﬁrst input mixing schema for Mixup on graph.
We theoretically prove that our mixing strategy can recover the source graphs from the mixed graph, and guarantees that the mixed graphs are manifold intrusion free.
We also empirically show that our method can effectively regularize the graph classiﬁcation learning, resulting in superior predictive accuracy over popular graph augmentation baselines.
I NTRODUCTION
Graph Neural Networks (GNNs) (Kipf & Welling, 2017; Veli ˇ ckovi ´ c et al., 2018; Xu et al., 2019) have recently shown its profound successes in many challenging applications, including predicting molecule properties for drug and material discovery (Gilmer et al., 2017; Wu et al., 2018), forecasting protein functions for biological networks (Alvarez & Yan, 2012; Jiang et al., 2017), and estimating circuit functionality in modern circuit design (Zhang et al., 2019).
Nevertheless, like other successfully deployed deep networks such as those for image classiﬁcation (Krizhevsky et al., 2012), speech recognition (Graves et al., 2013) and machine translation (Sutskever et al., 2014; Bahdanau et al., 2014), GNNs are also suffering from the data-hungry issue due to their high modeling freedom.
Consequently, researchers have been actively seeking effective regularization techniques for GNNs, aiming to power their learning but to avoid over-smoothing (Li et al., 2018; Wu et al., 2019) and over-ﬁtting (Goodfellow et al., 2016; Zhang et al., 2021) for better model generalization.
To this end, data augmentation schemes for regularizing GNNs mostly involve edge and node manipulation (e.g., deletion and addition) on a single graph (Rong et al., 2020;
Zhou et al., 2020; You et al., 2020).
[Figure]
In this paper, we look into a very successful pairwise data augmentation technique for image recognition (Zhang et al., 2018a;
Verma et al., 2018; Guo et al., 2019a;
Kim et al., 2020) and natural text classiﬁcation (Guo et al., 2019b; Guo, 2020; Jindal et al., 2020), called Mixup.
Mixup was originally introduced by Zhang et al.
(2018a)
as an interpolation-based regularizer for image classiﬁcation.
It regularizes the learning of deep classiﬁcation models by training with synthetic samples, which are created by linearly interpolating a pair of randomly selected training samples as well as their modeling targets.
Nevertheless, unlike images or natural sentences, which embrace a grid or linear sequence format, graph data have arbitrary structure and topology, which play a critical role on the semantics of a graph, and consequently even simply removing or adding one edge can dramatically change the semantic meaning of a graph.
As a result, mixing a graph pair is challenging and may naturally cause the manifold intrusion issue in Mixup as identiﬁed by Guo et al. (2019a).
Manifold intrusion results from conﬂicts between the synthetic labels of the mixed-up samples or between the synthetic labels and the labels of the original training data.
For example, consider the graph on the left of Figure 1, and its application of the popular graph perturbation action of node and edge deletion (gray lines in the ﬁgure).
In this case, the resulting two graphs in the middle will have the same structure as the two on the right from the original training set, but with different labels (indicated by the color bars under the graphs).
Note: an illustration of manifold intrusion when mixing two graphs is presented in A.5.
As discussed in (Guo et al., 2019a), when such intrusions occur, regularization using these synthetic graphs will contradict with the original training data.
This essentially induces a form of under-ﬁtting, resulting in the degradation of the model performance.
[Figure]
To address the aforementioned challenge, we propose the ﬁrst input mixing schema for Mixup on graph learning, coined ifMixup (intrusion-free Mixup).
As illustrated in Figure 2, ifMixup ﬁrst samples a random graph pair (top subﬁgure) from the training data.
For a given mixing ratio λ sampled from a Beta distribution, ifMixup then creates a synthetic graph (bottom subﬁgure), for each sample pair.
ifMixup treats a graph pair as two sets of ordered nodes in the same size (middle subﬁgure, where the unﬁlled node depicts an added dummy node).
As a result, it can linearly interpolate the node features and the edges of the input pair.
The newly generated graphs, which have a much larger number of graphs with changing local neighborhood properties than the original training dataset, are then used for training to regularize the GNNs learning.
Theoretically, we prove that our mixing strategy can recover the source graphs from the mixed graph, and such invertibility property guarantees that the mixed graphs are intrusion free.
That is, our method eliminates the possibility for the graphs resulting from mixing to coincide with any other graph in the training set or with a mixed graph from any other graph pairs.
Experimentally, we show, using eight benchmarking tasks from different domains, that our strategy effectively regularizes the graph classiﬁcation to improve its predictive performance, outperforming popular graph augmentation approaches and existing pair-wise graph mixing methods 1 .
Our key contributions are as follows: 1) To the best of our knowledge, we are the ﬁrst to propose an input Mixup for graph classiﬁcation.
We prove that our Mixup schema is free of manifold intrusion, which can signiﬁcantly degrade a Mixup-like model’s predictive accuracy.
We obtain the SOTA performance among popular graph perturbation baselines.
R ELATED W ORK
GNNs have been shown to be very effective for graph classiﬁcation in a variety of domains (Kipf & Welling, 2017;
Ying et al., 2018; Veliˇckovi´c et al., 2018; Klicpera et al., 2019; Xu et al., 2019;
Bianchi et al., 2020).
One of the key challenges of these successes is to leverage strong regularization techniques such as data augmentation to regularize those GNNs models with high modeling freedom.
Nonetheless, data augmentation is still a less touched area in graph data due to the arbitrary structure and topology.
Most of the graph augmentation strategies are for node classiﬁcation tasks, and heavily focus on perturbing nodes and edges in one given graph (Hamilton et al., 2017; Zhang et al., 2018b; Rong et al., 2020;
Chen et al., 2020;
Zhou et al., 2020; Qiu et al., 2020; You et al., 2020;
Wang et al.
; Fu et al., 2020; Wang et al., 2020; Song et al., 2021;
Zhao et al., 2021).
For example,
DropEdge (Rong et al., 2020) randomly removes a set of edges of a given graph.
GAUG (Zhao et al., 2021) learns to perturb graph edges for node classiﬁcation.
DropNode, representing node sampling based methods (Hamilton et al., 2017; Chen et al., 2018; Huang et al., 2018), samples a set of nodes from a given graph.
Unlike these approaches, our proposed strategy leverages a pair of graphs, instead of one graph, to augment the learning of graph level classiﬁcation.
Despite its great success in augmenting data for image recognition and text processing (Zhang et al., 2018a; Verma et al., 2018; Guo et al., 2019a; Guo, 2020; Jindal et al., 2020; Kim et al., 2020), Mixup has been less explored for graph learning.
To the best of our knowledge, there are two methods that apply Mixup to GNNs.
GraphMix (Verma et al., 2019) leverages the idea of mixing on the embedding layer, with an additional fully-connected network to share parameters with the GNNs, for graph node classiﬁcation in semi-supervised learning.
MixupGraph (Wang et al., 2021) also leverages a simple way to avoid dealing with the arbitrary structure in the input space for mixing a graph pair, through mixing the graph representation resulting from passing the graph through the GNNs.
Our paper here introduces the ﬁrst input mixing method for Mixup to augment training data for graph classiﬁcation.
Furthermore, our method guarantees that the mixed graphs are manifold intrusion free for Mixup.
I NTRUSION -F REE G RAPH M IXING 3. G RAPH C LASSIFICATION AND M IXUP I NTERPOLATION
Graph Classiﬁcation In the graph classiﬁcation problem we consider, the input G is a graph labelled with node features, or a “node-featured graph”.
Speciﬁcally, assume that the graph has n nodes, each identiﬁed with an integer ID in [ n ] := { 1 , 2 , . . . , n } .
The set of edges E of the graph is a collection of unordered pairs ( i, j ) ’s of node-IDs, as the current paper considers only undirected graphs (although there is no difﬁculty to extend the setting to directed graphs).
Associated with node i , there is a feature vector v ( i ) of dimension d .
We will use v to denote the collection of all feature vectors and one may simply regards v as a matrix of dimension n × d .
Thus, each input node featured graph G is essentially speciﬁed via the pair ( v, E ) .
The output y is a class label in ﬁnite set Y :
= { 1 , 2 , . .
, C } , which will be expressed as a (one-hot) probability vector over the label set Y .
The classiﬁcation problem is to ﬁnd a mapping that predicts the label y for a node-featured graph G .
The training data
G of this learning task is a collection of such ( G, y ) pairs .
Modern GNNs use the graph structure and node features to learn a distributed vector to represent a graph.
The learning follows the “message passing” mechanism for neighborhood aggregation.
It iteratively updates the embedding of a node h v by aggregating representations/embeddings of its neighbors.
The entire graph representation h G is then obtained through a READOUT function, which aggregates embeddings from all nodes of the graph.
Formally, representation h ki of node i at the k -th layer of a GNN is deﬁned as:
[Equation]
where W k denotes the trainable weights at layer k , N ( i ) denotes the set of all nodes adjacent to i , and AGGREGATE is an aggregation function implemented by the speciﬁc GNN model (popular ones include Max, Mean, Summation pooling operations), and h
i is typically initialized as the input node feature v ( i ) .
The graph representation h G aggregates node representations h v using the READOUT graph pooling function:
[Equation]
This graph representation is then mapped to label y using a standard classiﬁcation network (for example, a softmax layer).
Mixup Interpolation Mixup was introduced by (Zhang et al., 2018a) as an interpolation-based regularizer for image classiﬁcation.
It regularizes the learning of deep classiﬁcation models by training with synthetic samples, which are created by linearly interpolating a pair of randomly selected training samples as well as their modeling targets.
In detail, let ( x A , y A ) and ( x B , y B ) be two training instances, in which x A and x B refer to the input images and y A and y B refer to their corresponding labels.
For a randomly chosen such training pair, Mixup generates a synthetic sample as follows.
[Equation]
where λ is a scalar mixing ratio, sampled from a Beta( α, β ) distribution with hyper-parameters α and β .
Such synthetic instances ( (cid:101) x, (cid:101) y ) ’s are then used for training.
Motivated by the effectiveness of Mixup in regularizing image classiﬁcation models, we are naturally motivated to design a similar “Mixup” scheme for graph data, in particular, the node-featured graphs, as are the interest of this paper.
When this is possible, we may use the synthetic instances ( (cid:101) G, (cid:101) y ) ’s to learn the modele parameter θ by minimizing the loss L :
[Equation]
To mix ( G A , y A ) and ( G B , y B ) , it is straight-forward to apply Equation (4) to obtain the mixed label (cid:101) y .
The key question of investigation is how to mix G A and G B to obtain (cid:101) G .
It is worth noting that, unlike images or natural sentences which embrace a rigid structure of spatial coordinates or time axis, the underlying “coordinate system” of graph data may have different and arbitrary topology across different instances.
Consequently even simply deleting or adding one edge can invalid the semantic meaning of a graph.
One simple way to avoid dealing with the arbitrary structure in the input space for mixing a graph pair is to mix their ﬁxed-size graph representation that results from the READOUT function as depicted in Equation 2, namely mixing the two graphs by (cid:101) G = λh G A
+ (1 − λ ) h G B , as proposed by Wang et al.
We here propose to directly mix the graph inputs with arbitrary sizes for Mixup.
Our mixing strategy can recover the source graphs from the mixed graph, and such invertibility guarantees that the mixed graphs are free of manifold intrusion (Guo et al., 2019a), which can cause severe underﬁtting for Mixup learning.
3. I NVERTIBLE G RAPH M IXING S CHEMA
Now we propose a simple approach, ifMixup, for generating mixed node-featured graph (cid:101) G from a pair of such graphs G A and G B .
In the nutshell, ifMixup simply adopts a different representation for each node-featured graph.
Speciﬁcally, given a node featured graph G = ( v, E ) , we represent E as a binary matrix e with n rows and n columns, in which e ( i, j ) =
if ( i, j ) ∈ E , and e ( i, j ) = 0 otherwise.
Thus intead of expressing G as ( v, E ) we express it as ( v, e ) .
The mixing of G A
= ( v A , e A ) with G B = ( v B , e B ) to obtain (cid:101) G =
( (cid:101) v, (cid:101) e ) , can simply be done as follow,
[Equation]
In order for the above mixing rule to be well deﬁned, we need the two graphs to have the same number of nodes.
For this purpose we deﬁne n = max( n A , n B ) , where n A and n B are the number of nodes in instances A and B respectively.
If G A or G B has less than n nodes, we simply introduce dummy node to the graph and make them disconnected from the existing nodes.
The feature vectors for the dummy nodes are set to the all-zero vector.
This mixing process is illustrated in Figure 2, where the top subﬁgure is the source graph pair, and the middle depicts the added dummy node (i.e., the node with unﬁlled circle).
The bottom is the resulting mixed graph, where a mixed color indicates that the resulting node or edge is mixed by existing nodes or edges from both source graphs, and the blue color denotes a node that is mixed with a dummy node or an edge that is mixed with a zero-weighted edge.
It is worth noting that the resulting mixed graphs, through Equations 6 and 7, contain edges with weights between [0, 1].
As a result, during training, this will require the GNN networks be able to take the edge weights into account for message passing.
Below we will discuss how the two popular
GNN networks, namely GCN (Kipf & Welling, 2017) and GIN (Xu et al., 2019), cope with the weighted edges in graphs, namely how they implement Equation 1 to generate node representations.
GCN handles edge weights naturally by enabling adjacency matrix to have values between zero and one (Kipf & Welling, 2017), instead of either zero or one, representing edge weights:
[Equation]
where ˆ d i = 1 + (cid:80) j ∈N ( i ) e ( i, j ) ; W k stands for the trainable weights at layer k ; σ denotes the non-linearity transformation, i.e. the ReLu function.
To enable GIN to handle soft edge weight, we replace the sum operation of the isomorphism operator in GIN with a weighted sum calculation.
That is, the GIN updates node representations as:
[Equation]
where (cid:15) k is a learnable parameter.
3. I NVERTIBILITY AND I NTRUSION -F REENESS
We now show that such a simple mixing scheme in fact makes the original two node-featured graphs
G A and G B recoverable from the mixed graph (cid:101)
G under a mild assumption and hence avoids manifold intrusion.
To see this, we ﬁrst show that the graph topology and node features of the two original instances can both be recovered from the mixed instance.
Lemma 1 (Edge Invertibility) Let (cid:101) e be constructed using Equation 6 with λ (cid:54)
= 0 .
Consider equation
[Equation]
with unknowns s , e and e (cid:48) , where s is a scalar and e and e (cid:48) are binary (i.e., { 0 , 1 } -valued) n × n matrices.
There are exactly two solutions to this equation:
[Equation]
Note: the proof of this lemma is in Section A.1.
By this lemma, we see that if the mixing coefﬁcient λ (cid:54) = 0 .
5 , from the mixed edge representation (cid:101) e , we can always recover e A and e B (and hence E A and E B ) and their corresponding weights used for mixing.
Note that if λ is drawn from a continuous distribution over (0 , 1) , the probability it takes value 0 .
5 is zero.
That is, the connectivity of original two graphs can be perfectly recovered from the mixed edge representation (cid:101) e .
Lemma 2 (Node Feature Invertibility) Suppose that the node feature vectors for all instances in the task take values from a ﬁnite set V ⊂ R d and that V is linearly independent.
Let (cid:101) v be cosntructed using Equation 7.
Let V ∗ =
V ∪ { 0 } , where 0 denotes the zero vector in R d .
Consider equation
[Equation]
in which n × d matricess v and v (cid:48) are unknowns with rows taking value in V ∗ .
For any ﬁxed s ∈ (0 , 1) , there is exactly one solution of ( v, v (cid:48) ) for this equation.
Note: the proof of this lemma is in Section A.2.
The node feature invertibility in this lemma requires that the node feature vectors are linear independent.
Note that if the feature dimension d is larger than the size | V | of V and for each vector in
V , its elements are drawn at random, then the linear independence property of V is satisﬁed with high probability.
Thus, if we have the modeling freedom in designing the dimension d of the feature vectors (for example, in designing the embedding dimension), choosing a large d will make the linear independence condition of V satisﬁed.
There are however cases in which feature vectors are given and d < | V | .
In this case, we establish the following result which requires a much weaker condition.
To that end, suppose that the span SPAN( V ) of V is an m − dimensional space and m < d .
Let B be an m × d matrix whose rows form a basis of SPAN( V ) .
Any node feature matrix v can then be expressed as
[Equation]
for some matrix T with size n × m .
In this case, we may identify a node-featured graph as ( TB, e ) .
Let T denote the collection of all T matrices for all instances in the training set.
That is,
[Equation]
Lemma 3 (Node Feature Invertibility) Let (cid:101) v be constructed using Equation 7 and suppose that T is linearly independent.
Consider equation
[Equation]
in which n × d matrices v and v (cid:48) are unknowns where v = TB and v (cid:48)
=
T (cid:48) B for some T and T (cid:48) in
T .
For any ﬁxed s ∈ (0 , 1) , there is exactly one solution of ( v, v (cid:48) ) for this equation.
Note: the proof of this lemma is in Section A.3.
Note that since each T has size n × m , usually a large number, it is much easier for the linear independence condition of T to get satisﬁed in practice.
Theorem 1 (Intrusion-Freeness) Suppose that λ (cid:54) = 1 / 2 and that either the condition for Lemma 2 is satisﬁed or the condition for Lemma 3 is satisﬁed.
Then for any mixed node-featured graph (cid:101) G =
( (cid:101) v, (cid:101) e ) constructed using Equations 6 and 7, the two original node-feature graph G A and G B can be uniquely recovered.
Proof: Since λ (cid:54) = 0 .
5 , we can recover e A , e B and λ from (cid:101) e .
Given λ and either the condition for
Lemma 2 or the condition for Lemma 3, we can recover v A and v B from (cid:101) v .
(cid:3)
By this theorem, there is no other pair ( G (cid:48)
A , G (cid:48) B ) from the training set C that can be mixed into (cid:101)
G using any λ .
Thus, manifold intrusion does not occur under the mild condition of the theorem.
We note the intrusion-freeness of the proposed ifMixup scheme relies on the fact that input graphs do not have soft (weighted) edges.
We believe however that there is a potential to extend the scheme to graphs with weighted edges.
Promising directions include a combination of the following techniques.
First, instead of recovering edges and node features in tandem (as shown in the proof Theorem 1), we may consider jointly recover both.
Second we may quantize the edge weights to a set of discrete values and consider a judiciously designed distribution for the mixing coefﬁcient λ .
Third, we may insist the ordering of nodes in a graph to reﬂect certain semantics or graph topology of instance, whereby only allowing a restricted family of alignment schemes of the two graphs before mixing them.
It is our interest to investigate in these directions further.
E XPERIMENTS 4. S ETTINGS
Datasets We evaluate our method with eight graph classiﬁcation tasks from the graph benchmark datasets collection TUDatasets (Morris et al., 2020): PTC MR, NCI109, NCI1, and MUTAG for small molecule classiﬁcation, ENZYMES and PROTEINS for protein categorization, and IMDB-M and
IMDB-B for social networks classiﬁcation.
These datasets have been widely used for benchmarking such as in Xu et al.
and can be downloaded directly using PyTorch Geometric (Fey & Lenssen, 2019)
’s build-in function online 2 .
The social networks datasets IMDB-M and IMDB-B have no node features, and we use the node degrees as feature as in (Xu et al., 2019).
Data statistics of these datasets are shown in Table 3, including the number graphs, the average node number per graph, the average edge number per graph, the number of node features, and the number of classes.
Comparison Baselines We compare our method with four baselines:
MixupGraph (Wang et al., 2021), DropEdge (Rong et al., 2020), DropNode (Hamilton et al., 2017; Chen et al., 2018; Huang et al., 2018), and Baseline.
For the Baseline model, we use two popular GNNs network architectures:
GCN (Kipf & Welling, 2017) and GIN (Xu et al., 2019).
MixupGraph is the only available approach for applying Mixup on graph classiﬁcation.
It leverages a simple way to avoid dealing with the arbitrary structure for mixing a graph pair, through mixing the entire graph representation resulting from the READOUT function of the GNNs.
DropEdge and
DropNode are two widely used graph perturbation strategies for graph augmentation.
DropEdge randomly removes a set of existing edges from a given graph.
DropNode randomly deletes a portion of nodes and their connected edges.
GCN and GIN are two popular GNN architectures and have been widely adopted for graph classiﬁcation.
GCN leverages spectral-based convolutional operation to learn spectral features of graph through aggregation, beneﬁting from a normalized adjacency matrix, while GIN leverages the nodes’ spatial relations to aggregate neighborhood features, representing the state-of-the-art GNN network architecture.
We use their implementations in the PyTorch Geometric platform 3 .
Note that, for the
GCN, we use the GCN with Skip Connection (He et al., 2016) as that in (Li et al., 2019), This Skip
Connection powers the GCN to beneﬁt from deeper layers in GNN networks.
Detail Settings We follow the evaluation protocol and hyperparameters search of GIN (Xu et al., 2019) and DropEdge (Rong et al., 2020).
We evaluate the models using 10-fold cross validation, and report the mean and standard deviation of three runs on a NVidia V100 GPU with 32 GB memory.
Each fold is trained with 350 epochs with AdamW optimizer (Kingma & Ba, 2015), and the initial learning rate is reduced by half every 50 epochs.
The hyper-parameters we search for all models on each dataset are as follows: (1) initial learning rate ∈ { 0.01, 0.0005 } ; (2) hidden unit of size ∈ { 64, 128 } ; (3) batch size ∈ { 32, 128 } ; (4) dropout ratio after the dense layer ∈ { 0, 0.5 } ; (5) DropNode and
DropEdge drop ratio ∈ { 20%, 40% } ; (6) number of layers in GNNs ∈ { 5, 8 } ; (7) Beta distribution for ifMixup, MixupGraph and Manifold Mixup ∈ { Beta(1, 1), Beta(2, 2), Beta(20, 1) } .
Following
GIN (Xu et al., 2019) and DropEdge (Rong et al., 2020), we report the case giving the best 10-fold average cross-validation accuracy.
4. R ESULTS OF USING GCN AS BASELINE
The accuracy obtained by the GCN (Kipf & Welling, 2017) baseline, ifMixup, MixupGraph, DropEdge, and DropNode with GCN on the eight datasets are presented in Table 1 (best results in Bold )
[Table]
with GCN networks as baseline.
We report mean accuracy over 3 runs of 10-fold cross validation with standard deviations (denoted ± ).
The relative improvement of ifMixup over the baseline GCN is provided in the last row of the table.
Best results are in Bold .
Results in Table 1 show that ifMixup outperformed all the four comparison models against all the eight datasets.
For example, when comparing with the GCN baseline, ifMixup obtained a relative accuracy improvement of 5.36%, 5.31%, and 3.41% on the ENZYMES, PTC MR, and MUTAG datasets, respectively.
When considering the comparison with the Mixup-like approach MixupGraph, ifMixup also obtained superior accuracy on all the eight datasets.
For example, ifMixup was able to improve the accuracy over MixupGraph from 80.1%, 80.8%, 63.3%, and 51.3% to 82.0%, 81.9%, 65.4%, and 52.3% on the NCI109, NCI1, PTC MR and IMDB-M datasets, respectively.
Furthermore, as shown Table 1, unlike all the other augmentation methods (namely MixupGraph,
DropEdge, and DropNode), which can degrade the predictive performance of the baseline models, our method never degraded the baseline models’ predictive accuracy.
4.2. M ANIFOLD I NTRUSION : M IXING R ATIOS FOR G RAPH P AIRS
[Figure]
In this ablation study, we evaluate the sensitivity of the graph mixing ratio to the two Mixup-like approaches: ifMixup and MixupGraph.
We present the accuracy obtained by these two methods with Beta distribution as Beta(1, 1), Beta(2, 2), Beta(5, 1), Beta(10, 1) and
Beta(20, 1) on the ﬁrst six datasets of Table 3.
Results are presented in Figure 4.
Note: the density functions of these ﬁve Beta distributions are depicted in Figure 3.
Results in Figure 4 show that both MixupGraph and ifMixup obtained superior results on the six testing datasets with Beta(20, 1).
Nevertheless, MixupGraph seemed to very sensitive to the mixing ratio distribution.
For example, when Beta distributions were (1, 1) and (2, 2) (ﬁrst two bars in Figure 4), MixupGraph signiﬁcantly degraded its accuracy on all the six tasks (except for PTC MR).
In contrast, ifMixup was robust to the ﬁve Beta distributions we tested.
[Figure]
We here conjecture that, the decrease of MixupGraph’s accuracy obtained with Beta(1, 1) and
Beta(2, 2) was due to the manifold intrusion issue.
The mixing ratios sampled from Beta(1, 1) follow an uniform distribution between [0, 1], and those sampled from Beta(2, 2) follow a Bell-Shaped distribution between [0, 1] (see Figure 3).
Those mixing ratios have a wide range, and thus may aggravate the creation of mixed embeddings with conﬂict labels for MixupGraph.
On the other hand, ratios being sampled from Beta(5, 1), Beta(10, 1) and Beta(20, 1) mostly fall in the range of [0.8, 1].
Such conservative mixing ratios may alleviate creating conﬂict training samples for MixupGraph.
Promisingly, due to the intrusion-free nature, ifMixup did not suffer from the manifold intrusion problem, showing less sensitivity to the mixing ratios as in Figure 4.
4.2. O VER -S MOOTHING : I MPACT OF GNN S LAYERS
In this ablation study, we also evaluate the accuracy obtained by GCN, ifMixup and MixupGraph on the ﬁrst six datasets of Table 3, when varying the number of layers of the GCN networks.
The results for all the six datasets are depicted in Figure 5.
Results in this ﬁgure show that when increasing the GCN networks from 5 layers (blue bars) to 8 layers (red bars), both GCN and
MixupGraph seemed to degrade its performance on all the six datasets.
For example, for the NCI and NCI1 datasets, MixupGraph resulted in about 10% of accuracy drop when increasing the number of layers in GCNs (with Skip Connection) from 5 to 8.
On the contrary, ifMixup was able to increase the accuracy on all the six test datasets.
[Figure]
Baseline ifMixup
MixupGraph
ENZYMES
Layer5 Layer
[Figure]
Baseline ifMixup
MixupGraph
PROTEIN
Layer5 Layer
Such decrease of accuracy obtained by GCN and MixupGraph may due to the over-smoothing problem (Li et al., 2018; Wu et al., 2019).
That is, with deeper networks architectures (i.e., more layers), the representations of all nodes of a graph may converge to a subspace that makes these representations unrelated to the input.
This negative effect is mainly caused by the fact that the message passing between adjacent nodes is conducted along edge paths in GCNs.
That is, each graph convolutional layer in the GCNs keeps pushing the representations of adjacent nodes to blend with each other, based on the ﬁxed connections between nodes.
Through generating new adjacency matrices for each training step by randomly deleting a portion of edges of the input graphs, DropEdge (Rong et al., 2020) has show its effectiveness on mitigating over-smoothing.
Similar to DropEdge, ifMixup also creates graphs with changing node connections as inputs to the GCNs in each training step.
That is, each mixed graph in ifMixup will provide a new adjacency matrix to the networks, making node connections very random and diverse as that in DropEdge.
These changing local neighborhood properties in the mixed graphs thus help ifMixup alleviate the over-smoothing problem when GCNs goes deeper.
4.2. O VER -F ITTING : R EGULARIZATION E FFECT
In this ablation study, we evaluate the over-ﬁtting effect of our method.
We plot the training loss and validation accuracy of ifMixup, GCN, and MixupGraph methods across the 350 training epochs on both the NCI109 and NCI1 datasets in Figure 6.
[Figure]
Figure 6 shows that the training loss of ifMixup (left subﬁgures) maintained a relatively high level, when compared to GCN and MixupGraph.
For GCN, the training loss reduced to near zero after 300 training epochs.
Compared to GCN, MixupGraph and ifMixup remained higher training loss, although the loss of MixupGraph kept decreasing after 300 epochs.
The relative high loss here allows the models to keep tuning.
Such high loss is due to the much larger space of the synthetic graphs as random and diverse inputs to the networks, thus preventing the model from being over-ﬁtted by limited number of graph samples in the original training set.
As a result, as shown in the right subﬁgure, even training for a long time, the ifMixup models were not overﬁtting.
Note: a 2D visualization of the learned representations of the training graphs is presented in A.6.
4. R ESULTS OF USING GIN AS BASELINE
We also evaluate our method using the GIN (Xu et al., 2019) network architecture.
The accuracy obtained by the GIN baseline, ifMixup, MixupGraph, DropEdge
[Table]
and DropNode using GIN as baseline on the eight test datasets are presented in Table 2, where best results are in Bold .
[Table]
Results in Table 2 show that, similar to the GCN case, the ifMixup with GIN as baseline outperformed all the four comparison models against all the eight datasets.
For example, when comparing with GIN, ifMixup obtained a relative accuracy improvement of 4.35%, 3.23%, and 2.57% on the PTC MR,
ENZYMES, and NCI1 datasets, respectively.
When comparing with the Mixup-like approach
MixupGraph, ifMixup also obtained higher accuracy on all the eight datasets.
For example, ifMixup was able to improve the accuracy over MixupGraph from 82.2%, 82.2%, 63.1%, and 51.8% to 83.7%, 83.9%, 67.2%, and 53.2% on the NCI109, NCI1, PTC MR, and IMDB-M datasets, respectively.
C ONCLUSIONS AND F UTURE W ORK
We proposed the ﬁrst input mixing schema for Mixup on graph classiﬁcation.
We proved that our mixing strategy can recover the source graphs from the mixed graph, and such invertibility in turn guarantees that the mixed graphs are free of manifold intrusion, a form of under-ﬁtting which can signiﬁcantly degrade a Mixup-like model’s predictive accuracy.
We showed, using eight benchmark graph classiﬁcation tasks from different domains, that our strategy obtained superior predictive accuracy over popular graph augmentation approaches and existing pair-wise graph mixing methods.
In the future, we will extend our method for node classiﬁcation in graph.
Also, we will study the potential of our graph mixing schema on semantic-persevering graph mutation.
R EPRODUCIBILITY S TATEMENT
Our results are easily reproducible by following the experimental settings in Section 4.1 since our implementations used the PyTorch Geometric platform and the standard datasets from TUDatasets.
Also, we will make our PyTorch code publicly available upon the acceptance of the paper.
A A PPENDIX A. P ROOF OF L EMMA
Proof: First note that the values in matrix (cid:101) e can only take values in { 0 , λ, 1 − λ, 1 } .
The set [ n ] × [ n ] of all node pairs can be partitioned into four sets:
[Equation]
[Equation]
It is clear that
[Equation]
Let e, e (cid:48) and s be the solution of the equation in the lemma.
On M 00 ∪M 11 , we must have e =
e
(cid:48)
= (cid:101) e .
We only need to determine e and e (cid:48) on M 01 and M 10 .
When λ (cid:54) = 0 .
5 , we must have either s = λ, e
( i, j ) =
[Equation]
1 , ( i, j ) ∈ M0 , ( i, j
) ∈
Mand e (cid:48)
( i, j ) =
(cid:26) 0 , ( i, j ) ∈
M 10 1 , ( i, j
) ∈
Mor s = 1 − λ, e ( i, j ) =
(cid:26) 0 , ( i, j ) ∈ M1 , ( i, j
) ∈
Mand e (cid:48)
( i, j ) =
(cid:26) 1 , ( i, j ) ∈ M 10 0 , ( i, j ) ∈ M
Comparing such solutions with e A and e B , we prove the lemma.
(cid:3)
A. P ROOF OF L EMMA
Proof: We will prove the lemma by showing that for any i ∈ [ n ] , based on v ( i ) , we can uniquely recover v ( i ) and v (cid:48)
( i ) .
Case 1: (cid:101) v ( i ) = 0 .
It is obvious v ( i ) = v (cid:48)
( i )
= 0 .
Case 2: (cid:101) v ( i ) / ∈ V but (cid:101) v = cu for some u ∈ V and some scalar c .
In this case, c must be either s or 1 − s .
If c = s , then v ( i ) =
u, v (cid:48)
( i ) = 0 .
If c = 1 − s , then v ( i ) = 0 , v (cid:48)
( i ) =
u .
Case 3: (cid:101) v ( i ) / ∈ V and (cid:101) v (cid:54) =
cu for any u ∈ V and any scalar c (cid:54) = 0 .
For any two u, u (cid:48) ∈ V , let
SPAN( u, u (cid:48) ) denote the vector space spanned u and u (cid:48) .
Since V is a linearly independent set, it is clear every choice of ( u, u (cid:48) ) gives a different space SPAN( u, u (cid:48) ) , and (cid:101) v ( i ) must live in one and only one such space.
After identifying this space, we can identify ( u, u (cid:48) ) .
With the knowledge of s , we can precisely correspond u and u (cid:48) with v ( i ) and v (cid:48)
( i ) since either u = v ( i ) and u (cid:48)
= v (cid:48)
( i ) are true, or u = v (cid:48)
( i ) and u
= v ( i ) are true, but both can not be true at the same time.
Thus we have enumerated all possible cases, and in each case, there is a unique solution to the equation of interest.
[Equation]
A. P ROOF OF L EMMA
Proof : Since the rows of B are linearly independent, there is a unique (cid:101) T for which
[Equation]
We can recover (cid:101) T by projecting the rows of (cid:101) v on the basis B .
It is clear (cid:101) T =
sT +(1 − s ) T (cid:48) .
Then we only need to recover T and T (cid:48) from (cid:101) T .
But since T is linearly independent and T, T (cid:48) ∈ T , following a similar argument as in Case 3 of the proof for Lemma 2, we see that T and T (cid:48) can be uniquely recovered.
[Equation]
A. S TATISTICS OF BENCHMARK DATASETS
Table 3 details the statistics of the 8 benchmark datasets used in the paper.
[Table]
A. I LLUSTRATION OF M ANIFOLD I NTRUSION FROM M IXING G RAPH P AIRS
[Figure]
Figure 7 depicts an intrusion caused by connecting a graph pair.
The synthetic graph in the middle is created by connecting the two graphs from the left, but assigning a soft label (i.e., 50% of class and 50% of class 2).
This synthetic graph has the same structure as the right graph from the original training set, with an one-hot label (i.e., 100% of class 3).
A. V ISUALIZATION OF THE L EARNED R EPRESENTATIONS
In Figure 8, we also visualize the ﬁnal-layer representations formed by the GCN baseline, MixupGraph, and ifMixup on the original training graphs of the NCI109 and NCI1 datasets.
We project these embeddings to 2D using t-SNE (van der Maaten & Hinton, 2008).
The upper row of Figure 8 shows that for NCI109, both GCN and MixupGraph were not able to separate the two classes, while ifMixup completely separated the training graphs with different labels.
Similarly, when considering the bottom row of Figure 8 as for NCI1, both GCN and MixupGraph did not completely divide the two classes, while ifMixup attained a perfect separation for the two.
A. U NEXPECTED R ESULTS
We here also report an unexpected result.
We randomly shufﬂe the node order of one of the graphs in the graph pair before mixing for ifMixup.
Such shufﬂing is able to signiﬁcantly increase the number of synthetic graphs used for training for ifMixup, and we expect this would further improve the model’s predicative accuracy.
The accuracy obtained over the ﬁrst six datasets of Table 3 obtained by ifMixup with GCNs as baseline is presented in Table
[Table] Unexpectedly, results in Table 4 show that leveraging node order shufﬂing to increase the training data size did not help in terms of accuracy obtained.
We hypothesis that this is may due to the modeling capability of the GCN networks.
We will further investigate this hypothesis in our future work.
[Table]
A. A V ARIANT OF M IXUP G RAPH
MixupGraph
[Table]
For GIN, the ﬁnal-layer representation of a graph is the concatenation of all the representations of each layer of the networks.
In this sense, we can implement the idea of mixing on a random embedding layer as that in the Manifold Mixup (Verma et al., 2018) for vision.
We compare Manifold Mixup with the MixupGraph, and the results are shown in Table 5.
Results in the table show that MixupGraph and Manifold Mixup obtained similar accuracy on all the eight datasets.
For example, for the PTC MR and IMDB-M datasets, Manifold Mixup obtained higher accuracy, while on the ENZYMES and NCI109 datasets Manifold
Mixup obtained lower accuracy than MixupGraph.
For the other four datasets, the accuracy obtained by the two methods are comparable.
A. S HALLOW GCN AND GIN
We also tested the performance of a 3-layer GCN and a 3-layer GIN baseline models.
Results are presented in Table 6.
As can be seen in the table, both GCN and GIN baselines obtained inferior
[Table]
accuracy than a deeper GNN networks, namely 5 or 8 layers as used in the experiments in the main paper.
A. I LLUSTRATION OF M IXING AND G RAPH S TRUCTURE R ECOVERING
In Figure
[Table]
we illustrate how the mixed graph structure is formed and how the structures of the two source graphs can be recovered from the mixed graph
[Table] [Figure]