D YNAMIC S ELF - TRAINING F RAMEWORK FOR G RAPH C ONVOLUTIONAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph neural networks (GNN) such as GCN, GAT, MoNet have achieved stateof-the-art results on semi-supervised learning on graphs.
However, when the number of labeled nodes is very small, the performances of GNNs downgrade dramatically.
Self-training has proved to be effective for resolving this issue, however, the performance of self-trained GCN is still inferior to that of G2G and
DGI for many settings.
Moreover, additional model complexity make it more difﬁcult to tune the hyper-parameters and do model selection.
We argue that the power of self-training is still not fully explored for the node classiﬁcation task.
In this paper, we propose a uniﬁed end-to-end self-training framework called Dynamic
Self-traning , which generalizes and simpliﬁes prior work.
A simple instantiation of the framework based on GCN is provided and empirical results show that our framework outperforms all previous methods including GNNs, embedding based method and self-trained GCNs by a noticeable margin.
Moreover, compared with standard self-training, hyper-parameter tuning for our framework is easier.
I NTRODUCTION
Graphs or networks can be used to model any interactions between entities such as social interactions (Facebook, Twitter), biological networks (protein-protein interaction), and citation networks.
There has been an increasing research interest in deep learning on graph structured data, e.g., (Bruna et al., 2014; Defferrard et al., 2016;
Monti et al., 2017; Kipf & Welling, 2017; Hamilton et al., 2017;
Velickovic et al., 2018; Tang et al., 2015; Perozzi et al., 2014).
Semi-supervised node classiﬁcation on graphs is a fundamental learning task with many applications.
Classic methods rely on some underly diffusion process to propagate label information.
Recently, network embedding approaches have demonstrate outstanding performance on node classiﬁcation (Tang et al., 2015; Grover & Leskovec, 2016; Bojchevski & Günnemann, 2018).
This approach ﬁrst learns a lower-dimensional embedding for each node in an unsupervised manner, and then the embeddings are used to train a supervised classiﬁer for node classiﬁcation, e.g., logistic regression or multi-layer perceptron (MLP).
Graph neural networks (GNN) are semi-supervised models and have achieved state-of-the-art performance on many benchmark data sets (Monti et al., 2017; Kipf &
Welling, 2017; Velickovic et al., 2018).
GNNs generalize convolution to graph structured data and typically have a clear advantage when the number of training examples is reasonably large.
However, when there are very few labeled nodes, GNNs is outperformed by embedding based method (as shown by our experimental results), e.g., G2G from (Bojchevski & Günnemann, 2018) and DGI from (Veliˇckovi´c et al., 2019).
To overcome this limitation of GCNs (Kipf & Welling, 2017), Li et al.
(Li et al., 2018) propose to apply self-training and co-training techniques (Scudder, 1965).
The idea of these techniques is to augment the original training set by adding in some unlabeled examples together with their label predictions.
Such “pseudo-label” information is either from the base model trained on the original training set (self-training) or another learning algorithm (co-training).
The results from (Li et al., 2018) demonstrate the effectiveness of co-training and self-training.
However, among the four variants implemented in (Li et al., 2018), there is not a single one that achieves the best performance across different settings; and from our experiments, G2G and DGI outperforms all the four variants when the number of labels from each class is less than 10 .
There are clear restrictions in prior self-training approaches.
First, the pseudo-label set is incremental only, i.e., after an unlabeled example is added to the training set, it will never be deleted and
its pseudo-label will never change even if its prediction and/or the corresponding margin has changed drastically.
Secondly, all the pseudo-labels are considered equal, although they may have very different classiﬁcation margins.
Furthermore, it introduces extra hyper-parameters such as the number of unlabeled nodes to be added into the training set and the total number of self-training iterations.
The performance gain is sensitive to such parameters and their optimal values may differ for different data sets and label rates (Buchnik & Cohen, 2018).
To fully understand and explore the power of self-training on the node classiﬁcation task, we propose a novel self-training framework, named Dynamic Self-training , which is general, ﬂexible, and easy to use.
We provide a simple instantiation of the framework based on GCN (Kipf & Welling, 2017) and empirically show that it outperforms state-of-art methods including GNNs, self-trained GCN (Li et al., 2018), and embedding based methods.
Our framework has the following distinguishing features compared with (Li et al., 2018; Buchnik & Cohen, 2018).
We augment the training set and recalculate the pseudo-labels after each epoch .
So the number self-training iterations is the same as the number of epochs and the pseudo-label assigned to an unlabeled example may change during the training process.
In stead of inserting a ﬁxed number of new pseudo-labels with highest margin in each iteration, we use a threshold-based rule, i
e., insert an unlabeled node if and only if its classiﬁcation margin is above the threshold.
The pseudo-label set is dynamic.
When the margin of an unlabeled node is above the threshold, we activate it by adding it to the loss function, but if the margin of this node becomes lower than the threshold in a later epoch, we will deactivate it.
We assign a (dynamic) personalized weight to each active pseudo-label proportional to its current classiﬁcation margin.
The total pseudo-label loss is thus the
weighted
sum of
losses
corresponds to all
pseudo labels.
P RELIMINARIES 2. G RAPH N OTATION AND P ROBLEM D EFINITION
In the problem, we are given an undirected graph with node attributes G =
( V, E, X ) , where V is the vertex set, E is the edge set.
Here, X is the feature matrix, the i -th row of which, denoted as x i , is the feature vector of node i .
We assume each node belongs to exactly one class and use y i to denote the class label of the i -th node.
The aim is to design learning algorithms to predict the labels of all nodes based on the labels of a small set of training nodes provided in the beginning.
We use
N k
( i ) to denote the set of nodes whose distance to node i is at most k .
L ⊂ V is the set of labeled nodes and U =
V \ L is the set of unlabeled nodes.
2. G RAPH C ONVOLUTIONAL N ETWORKS
GCN introduced in (Kipf & Welling, 2017) is a graph neural network model for semi-supervised classiﬁcation.
GCN learns the representations of each node by iteratively aggregating the embeddings of its neighbors.
Speciﬁcally, GCN consists of L > 0 layers each with the same propagation rule deﬁned as follows.
In the l -th layer, the hidden representations H ( l − 1) are averaged among one-hop neighbors as:
[Equation]
Here, ˜ A =
A + I n is the adjacency matrix of G after adding self-loops ( I n is the identity matrix), ˜ D is a diagonal matrix with ˜ D ii = (cid:80)
j ˜
A ij , W ( l ) is a trainable weight matrix of the l -th layer, and σ is a nonlinear activation function;
H ( l ) ∈ R n × d l denotes hidden feature matrix of the l -th layer and
H (0) =
X and f i
= H ( L ) i represents the output of i -th node.
We use l ( y i , f i ) to denote the classiﬁcation loss of node i , which is typically the cross entropy function.
Thus, loss function used by GCN is of the form:
[Equation]
For a k -layer GCN, the receptive ﬁeld of each training example is its order- k neighborhood.
When there are only few training samples, we need to increase the number of layers in order to cover most of the unlabeled nodes.
However, deeper GCN will cause the problem of over-smoothing , i.e., critical features of the vertices may be smoothed through the iterative averaging process, which makes nodes from different class indistinguishable (Xu et al., 2018;
Li et al., 2018).
2. S ELF T RAINING
Recently (Li et al., 2018) apply self-training to overcome these limitations of GCNs.
Self-training is a natural and general approach to semi-supervised learning, which is particularly well-motivated in the context of node classiﬁcation (Buchnik & Cohen, 2018; Li et al., 2018).
Assume we have a base model/algorithm for the learning problem, which takes as input a set of labeled examples and makes predictions for other examples.
Typically, for each unlabeled node, the base algorithm will also return an associated margin or conﬁdence score.
The self-training framework trains and applies the base model in rounds, where at the end of each round, the highest-conﬁdence predictions are converted to become new labeled examples in the next round of training and prediction.
Thus, the receptive ﬁelds of all the labeled nodes increases and will eventually cover the entire graph, which resolve the issue of GCNs without adding more layers.
O UR M ETHOD 3. A G ENERALIZED S ELF - TRAINING F RAMEWORK
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Break end 9 end
Sun et al.
(Sun et al., 2019) proposed Multi-stage Training Framework as generalization for selftraining method in (Li et al., 2018).
Inspired by this, we propose a more generalized end-to-end self-training framework named Dynamic Self-training Framework shown in algorithm 1.
Instead of operating on data split, we maintain a conﬁdence score in each iteration.
There is no speciﬁed training stages here, but we update the conﬁdence value for each unlabeled node after every epoch.
Consider the original model f ( · , · ) as a forward predicting function with backward trainable parameters.
The graph data G and the trainable parameters θ t is the input of this function, and the output of this model is collected into f V ∈ R n × C , where f v denotes the output vector (before assigned with label) of node v ∈ V , and
C =
d L is the number of classes.
Then we construct the conﬁdence score vector S V ∈ R n from the model output f v using a function UC , which can be instantiated in many forms.
For example, Algorithm 2 illustrates how standard multi-stage self-training GCN implement this part.
Finally we update the model parameters using a speciﬁed algorithm such as gradient descent, where the conﬁdence score vector plays a role.
The conﬁdence score participates in the parameter updating process in an end-to-end manner.
An example of this part can be seen in section 3.3.
3. P SEUDO L ABEL M ETHOD
Deﬁne the pseudo label ˜
y i ∈
R d L of i -th node which satisﬁes :
[Equation]
(Lee, 2013) introduced a pseudo label version of semi-supervised losses:
[Equation]
i
[Equation]
l (
[Equation] [Equation]
(4) where λ
= nn (cid:48) γ , n = |L| , n (cid:48) = |U| , γ ∈ R is a hyper-parameter and the additive term (cid:80) i ∈U l (˜ y
i , f i ) is the pseudo label loss.
Here, λ measures how much the pseudo label term inﬂuence the training process.
This is equivalent to Entropy Regularization for classiﬁcation problems (Lee, 2013).
3. S OFT L ABEL C ONFIDENCE
In standard multi-stage self-training methods, a node just has two states: in the training set or not, which corresponds to binary-valued conﬁdences { 0 , 1 }; and in most cases, if a node is added in training set, it will be kept there.
This simple setting hinders learning in some cases.
For instance, if the classiﬁer puts a wrongly labeled node into the training set, which is of high possibility in preliminary training epochs, it will persistently learn wrong knowledge from this node.
Worse still, another wrongly adding is more possible.
This negative feedback loop may contribute to a extremely poor classiﬁer.
Moreover, original labeled nodes and added nodes in the training are treated equally, which is too restricted and may harm the learning; explicitly distinguishing them in the training process could be beneﬁcial.
To resolve these problems, we introduce a mechanism named Soft Label
Conﬁdence as the conﬁdence updating component in algorithm 1, which computes a personalized conﬁdence value for each node, and the training set is dynamically changing except the ground truth labels.
Based on the pseudo label loss (4), we propose the loss wrapped by soft label conﬁdence:
[Equation]
Here α is a function mapping from R d L to R , deﬁned as conﬁdence function .
While there are other possible choices for α , in our method we adopt a threshold based function:
[Equation]
Here β ∈ (0 , 1) is a hyper-parameter as threshold, n (cid:48)
c i denotes the number of nodes whose pseudo label belongs to class c
i , c
i is the class which i -th node’s pseudo label belongs to, and 1 is the allvector.
We introduce n (cid:48)
c i here to balance the categories of pseudo labels, because pseudo labels could be initially extremely unbalanced and lead to a poor classiﬁer in practice.
Although α ( f i ) depends on f i , and thus a function of network’s weights, we will block the ﬂow of gradient through α ( f i ) for the following reasons: Firstly, conﬁdence function is non-differentiable in most cases.
Secondly, if we allow the gradient to ﬂow through α ( f i ) , the optimizer may tend to ﬁnd a solution that satisﬁes max( f i ) < β, ∀ i ∈ V , since for such a solution, α ( f i )
=
for all i and the pseudo label loss is zero, which does no good to self-supervised learning.
So we use the following way to compute the gradient:
[Equation]
R ELATED W ORK
Convolutional Network
The work of GNNs seeks generalizations of the convolution operator to graph structured data.
One way to do this is to apply convolution in the spectral domain, where the eigenvectors of the graph Laplacian are considered as the Fourier basis (Bruna et al., 2014; Henaff et al., 2015; Defferrard et al., 2016; Kipf & Welling, 2017).
Such spectral methods learns hidden layer representations that encode both graph structure and node features simultaneously.
Kipf and Welling (Kipf & Welling, 2017) simplify previous spectral techniques by restricting the propagation to a 1 -hop neighborhood in each layer.
(Chen et al., 2018) propose fast GCNs, which improves the training speed of the original GCN.
GAT of (Velickovic et al., 2018) allows for assigning different importances to nodes of the same neighborhood via attention mechanisms.
(Xu et al., 2018) introduce JK networks, which adjust the inﬂuence radii of each node adaptively.
Another direction that generalizes convolutions to graph structured data, namely non-spectral approaches, deﬁne convolutions directly in the spatial domain (Duvenaud et al., 2015; Atwood & Towsley, 2016; Monti et al., 2017).
Such methods are easier to be adapted to do inductive learning (Hamilton et al., 2017;
Velickovic et al., 2018; Bojchevski & Günnemann, 2018).
However, few-shot learning remains a challenge for this class of methods.
Label Propagation
Unlike GNNs, which propagate node representations, the classic Label Propagation (LP) method (Zhu et al., 2003) iteratively propagates (soft) labels.
More speciﬁcally, in each iteration, each unlabeled node obtains a new soft label that is the aggregation of the soft labels from the previous iteration of its neighbors.
The key to LP is to design an effective propagation rule; for some propagation rules, the algorithm may not converge and/or the accuracy may not improve over iterations.
Thus, one often needs to specify a stopping criteria and a validation set for model selection.
LP can also be used as the base algorithm in the self-training framework.
Self-training Self-training is a natural and general approach to semi-supervised learning (Scudder, 1965) and has been widely used in the NLP literature.
Self-training is used by (Yarowsky, 1995;
Hearst, 1991) for word sense disambiguation.
(Riloff et al., 1999) used self-training in the form of bootstrapping for information extraction and later for learning subjective nouns.
(Riloff et al., 2003) with (Nigam et al., 2000) using EM for text classiﬁcation.
Self-training has been used for object recognition (Rosenberg et al., 2005;
Zhou et al., 2012).
(McClosky et al., 2006; 2008; Huang &
Harper, 2009; Sagae, 2010) shows how effective can self-training be in parsing.
(Wang et al., 2007;
Huang et al., 2009; Qi et al., 2009) introduce self-training techniques to part of speech tagging, and (Kozareva et al., 2005; Liu et al., 2013a) adopt self-training in named entity recognition.
(Van Asch &
Daelemans, 2016; Drury et al., 2011; Liu et al., 2013b) used self-training in sentiment classiﬁcation.
Recently, self-training has also been successfully applied on node classiﬁcation.
Li et al.
(Li et al., 2018) study self-training GCNs; Buchnik and Cohen (Buchnik & Cohen, 2018) mainly consider the effect self-training for diffusion-based techniques.
In pseudo-label method of (Lee, 2013), for unlabeled data, their pseudo-labels are recalculated every weights update.
However, they don’t assign weight to each unlabeled data.
As for the self-training algorithm itself, (Chen et al., 2011) shows that selecting highly conﬁdent instances with a pre-deﬁned threshold may not perform well.
(McClosky et al., 2006) produce a ranked list of n-best predicted parses and selected the best one.
(Rosenberg et al., 2005) shows that a training data selection metric that is deﬁned independently of the detector greatly outperforms a selection metric based on the detection conﬁdence generated by the detector.
(Zhou et al., 2012) suggests that selecting more informative unlabelled data using a guided search algorithm can signiﬁcantly improve performance over standard self-training framework.
Most recently, (Levati´c et al., 2017) proposed proposed an algorithm to automatically select appropriate threshold.
Network Embedding Node classiﬁcation is also one of the main applications of network embedding methods, which learns a lower-dimensional representation for each node in an unsupervised manner, followed by a supervised classiﬁer layer for node classiﬁcation (Perozzi et al., 2014; Tang et al., 2015; Grover & Leskovec, 2016; Wang et al., 2016; Bojchevski & Günnemann, 2018).
A recent work of (Bojchevski & Günnemann, 2018) proposes Graph2Gauss.
This method embeds each node as a Gaussian distribution according to a novel ranking similarity based on the shortest path distances between nodes.
A distribution embedding naturally captures the uncertainty about the representation.
DGI (Veliˇckovi´c et al., 2019) is an embedding method based on GCNs, the unsupervised objective of which is to maximize mutual information.
The work of Embedding approaches achieve competitive performance in node classiﬁcation tasks, while the learned representations also prove to be extremely useful for other downstream applications.
E VALUATION 5. D ATASET
We conduct the evaluation on four benchmark citation datasets: Cora, Citeseer, Pubmed (Sen et al., 2008), and Core-full (Bojchevski & Günnemann, 2018).
Each of these four datasets is undirected graph with node feature.
Each node is a document and the edges denote the citation relationship; the feature of a node is the bag-of-words representation of the document.
The number of layers in
GCN is two by default, and thus the receptive ﬁeld of each labeled node is its order-2 neighborhood.
We measure the fraction of nodes which is covered by the 2-hop neighbors of all labeled nodes, i.e., | (cid:83) s ∈S N 2 ( s ) | / | V | , where S is the set of labeled nodes randomly sampled from V .
Here we report the 2-hop coverage ratio on the four datasets when the label rates are 1% and 0.5% respectively.
We summarize the information of datasets in table 1.
[Table]
5. E XPERIMENT S ETTINGS
We evaluate models on semi-supervised node classiﬁcation tasks with varying label rates.
Instead of evaluating on a ﬁxed data split as in (Kipf & Welling, 2017; Velickovic et al., 2018), we mainly consider random splits as (Li et al., 2018) does.
In detail, for a given label rate, we randomly generate 100 different splits on each dataset.
In each split, there is a labeled set with prespeciﬁed size for training, and in this set each class contains the same number of labeled nodes.
As in (Li et al., 2018), we don’t use a validation set, and all the remaining nodes will be used for testing.
For simplicity, we will refer to a task in the form of dataset- l , where l is the number of labeled nodes per class.
For example, Cora-1 denotes the classiﬁcation task on dataset Cora with one seed per class.
5. I MPLEMENTATION D ETAILS
For all the models(Perozzi et al., 2014; Tang et al., 2015; Grover & Leskovec, 2016; Wang et al., 2016; Bojchevski & Günnemann, 2018; Velickovic et al., 2018; Monti et al., 2017) except for GCN based methods, settings of hyper-parameters are the same as suggested in original papers.
All GCN based methods including GCN, Self-training GCN, Co-training GCN, Intersection GCN, Union GCN, and DSGCN share the same setting of hyper-parameter following (Shchur et al., 2018): one hidden layer with 64 units, dropout rate 0.8, Adam optimizer (Kingma & Ba, 2015) with learning rate 10 − 2 , a L 2 regularization with weight 10 − 3 .
We train other GCN based methods for a ﬁxed epochs of 200, while DSGCN is trained for 600 epochs in few-label tasks such as 1, 3, 5, 10 tasks.
Because 20 orlabels per class implies ample supervised information, we train DSGCN for 200 epochs in these tasks.
The four variants of (Li et al., 2018): Self-training GCN, Co-training GCN, Intersection GCN and
Union GCN follow original self-training settings in (Li et al., 2018).
For DSGCN, we use a threshold of 0 .
6 when the number of labels per class is below 3 , and set the threshold to 0.75 for label rate above 3 but below 10 .
Otherwise, the threshold is 0.9 by default.
5. R ESULT A NALYSIS
The numerical results are summarized in table 2 and table 3.
The highest accuracy
[Table] [Table]
Comparison Between GNN Variants and Embedding Methods As unsupervised methods, G2G and DGI outperform all GNN variants in very few labels cases, e.g., 1 and 3 per class on both
Cora and Citeseer.
Observing that LP performs well in Cora-1 while other feature propagation methods not, we can naturally conclude that in dataset with graph structure, concentrating more on the unsupervised information (both strong manifold structure(Li et al., 2018) and feature patterns) will improve semi-supervised model compared to just utilizing supervised information, in the case of low label rate.
When label rate goes higher, all GNN variants enjoy better accuracies compared to unsupervised models.
Hence we empirically verify the strong generalization ability of GNNs when the supervised information is sufﬁcient.
Sun et al.
(Sun et al., 2019) has demonstrated the limitation of GCN in few labels case, and here we ﬁnd that these convolution based methods suffer from inefﬁcient propagation of label information as well, which can be seen as the intrinsic drawbacks of semi-supervised graph convolution based methods.
Comparison Between Self-training GCNs and All Other Models
In all few-label tasks, selftraining strategies improve over GCN by a remarkable margin.
Except for tasks with 50 labels per class, the best accuracy is always obtained by self-training GCN.
Even in extreme one-label case, where unsupervised information is more vital, DSGCN outperforms G2G by a margin of 6 . 2% in
Cora and 9 . 2% in Citeseer.
We conclude that self-training strategy is capable of utilizing unsupervised information more effectively.
Thus it signiﬁcantly helps classiﬁcation.
Additionally, four naive selftraining GCNs implemented in (Li et al., 2018) are worse than GCN when label rate goes higher, e.g.,
Cora-50 and Cora-full-5, which manifests that inappropriate self-training strategies will sometimes degrade the performance of the base model.
Hence there is a trade-off: capturing unsupervised signals, or learning supervised information well.
However, DSGCN holds a good balance here.
It doesn’t show much decrease compared to GCN even in the worst case task, Cora-full-50, where the accuracy only decreases by 0 .
6% ; in all other cases it is always better than GCN.
This demonstrates that the dynamic self-training framework not only helps the original model to capture unsupervised information, but also retains the learning ability when there are enough labels.
Comparison of Self-training GCNs By applying a simpler and more general self-training strategy,
DSGCN outperforms other self-training based GCNs with considerable margins in most cases.
In
Citeseer-1, the margin even reaches 14 .
1% compared with the best strategy among Co-training, Selftraining, Union and Intersection.
This empirically supports the advantage of DSGCN for tackling a wide range of classiﬁcation tasks over conventional self-training methods.
Effect of Threshold
Here we discuss how the important hyper-parameter β inﬂuence the performance of DSGCN.
We train DSGCN with different threshold: 0.45, 0.6, 0.75, 0.9, 1.0 forepochs on dataset Cora and Citeseer for the same split with the same initialized weights.
We conduct these experiments on tasks with different seed numbers, the results are presented in ﬁgure 1.
As shown in ﬁgure 1, when labels are very few, DSGCN with a relatively lower threshold β demonstrate a clear improvement in accuracy over the original GCN.
Besides, GCN’s accuracy curve erratically ﬂuctuates while the curve of DSGCNwith a low threshold does not.
Thus, we observe that the stability of the base model is also improved by wrapping it into the dynamic self-training framework.
When more labels are provided, all models tend to be stable and a low threshold could harm the training process.
C ONCLUSION
In this paper, we ﬁrstly introduce a novel self-training framework.
This framework generalizes and simpliﬁes prior work, providing customizable modules as extension for multi-stage self-training.
Then we instantiate this framework based on GCN and empirically compare this model with a number of methods on different dataset splits.
Result of experiments suggests that when labels are few, the proposed DSGCN not only outperform all previous models with noticeable margins in accuracy but also enjoy better stability in the training process.
Overall, the Dynamic Self-training Framework is powerful for few-label tasks on graph data, and provides a novel perspective on self-training techniques.
A A PPENDIX : A DDITIONAL E XPERIMENTS
We also test our self-training methods on other GNNs as well, e.g., SGC(Wu et al., 2019), GAT (Velickovic et al., 2018)
[Table]
and GraphSage (Hamilton et al., 2017).
For the three GNN models, settings of hyper-parameters are the same as suggested in original papers.
And our dynamic self-training framework share the same setting of hyper-parameter: one hidden layer with 32 units, dropout rate 0.7, Adam optimizer (Kingma & Ba, 2015), a L 2 regularization with weight 5 − 4 and set the threshold to 0.9.
Clearly, our dynamic self-training framework achieves similar improvements on all the three base models.
The numerical results are summarized in table 4.
We can see equipped with our DS framework, these models enjoys noticeable increase in performance.
[Table]
To evaluate the computation overhead introduced by dynamic self-training framework, we test the total training time for various models.
Intuitively the computational cost will only slightly increase.
The reason is that the computational cost of the original GCN model is dominated by previous layers, where the entire graph is included.
So even if all nodes become pseudo labels, the size of the entire network is increased by at most a factor of 2, and the number of parameters remains the same.
Therefore, the computational costs will increase by at most a small constant in theory.
We have also veriﬁed this empirically.
We record the training time of base models before and after applying our framework.
In the experiments, the training size is 20 per class, the number of epoch is 200, and the time is the average time (in seconds) of 25 runs.
The numerical results can be seen in Tabel 5.
[Table]