B RIDGING G RAPH N ETWORK TO L IFELONG L EARNING WITH F EATURE I NTERACTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph neural networks (GNN) are powerful models for many graph-structured tasks.
In this paper, we aim to bridge GNN to lifelong learning, which is to overcome the effect of “catastrophic forgetting” for continuously learning a sequence of graph-structured tasks.
Although many lifelong learning techniques for convolutional neural networks (CNN) have been developed, lifelong learning for GNN is still underexplored and suffers from incomplete graph structure during learning.
This is because in lifelong learning the nodes increase dynamically and can only be present to the model once, which makes many graph models and sampling strategies inapplicable.
To solve this problem, we propose a new graph topology based on feature interaction, called the feature graph.
It takes features as new nodes and turns nodes into independent graphs.
This successfully converts the original problem of node classiﬁcation to graph classiﬁcation, in which the increasing nodes are turned into training samples.
Therefore, the lifelong learning techniques developed for CNN become applicable to GNN for the ﬁrst time.
In the experiments, we demonstrate both the efﬁciency and effectiveness of feature graphs for lifelong learning tasks using a rehearsal method.
We expect that it will have broad potential applications for graph-structured tasks in lifelong learning.
I NTRODUCTION
Graph neural networks (GNN) have received increasing attention and proved useful for many tasks with graph-structured data, such as citation, social, and protein networks (Veli ˇ ckovi ´
c et al., 2018).
In this paper, we aim to extend graph neural networks to lifelong learning, which is also referred to as continual or incremental learning (Lopez-Paz & Ranzato, 2017).
Concretely, lifelong learning aims at breaking the barrier between training and testing, and the goal is to develop algorithms that are able to keep updating the model parameters over time (Aljundi et al., 2019a).
In other words, we expect the model to gradually accumulate knowledge based on a data (task) stream, where each sample can only be present to the model once.
Intuitively, it is easy to suffer from the problem of “catastrophic forgetting” if the model is simply updated for new samples via gradient descent by the back-propagation algorithm (Rumelhart et al., 1986).
Although some strategies have been developed to alleviate the forgetting problem for convolutional neural networks (CNN), lifelong learning is still very difﬁcult for graph networks.
This is because most of existing graph models such as graph convolutional networks (GCN) (Kipf & Welling, 2017) require the entire graph for training.
This is infeasible for lifelong learning, as the graph size can increase over time and we have to drop old data to learn new knowledge.
To reduce the memory consumption, some graph sampling strategies were proposed, but they are also difﬁcult to directly apply to lifelong learning (Hamilton et al., 2017; Chen et al., 2018b; Zeng et al., 2020).
Moreover, since the training samples are not identically and independently distributed ( iid ) and only available in a continuum (Zenke et al., 2017), the sampling strategies that require pre-processing of the entire graph become infeasible (Zeng et al., 2020).
As far as we know, lifelong learning for GNN is still an underexplored topic but has broad potential applications (Wu et al., 2020).
Recall that regular CNNs are trained in a mini-batch manner where the model can take samples as independent inputs (Krizhevsky et al., 2012).
Our question is that can we convert GNN into a traditional CNN-like classiﬁcation problem, so that (i) nodes can be predicted independently and (ii) the lifelong learning techniques developed for CNN can be easily adopted for GNN?
However, this
Graph 1.
Feature Graph 0. is not straightforward as node connections cannot be considered in a regular classiﬁcation model.
To solve this problem, we propose to construct a different graph topology, the feature graph in Figure 1, to bridge GNN to lifelong learning.
It takes the features as nodes and turns the nodes into graphs.
This converts the problem of node classiﬁcation to graph classiﬁcation.
In this way, the increasing nodes become training samples, thus the feature graphs can be trained in a native mini-batch manner.
Note that we do not focus on speciﬁc lifelong learning techniques or graph models, instead we try to overcome the difﬁculty of directly applying lifelong learning techniques to graph networks.
We will demonstrate its effectiveness and efﬁciency using a rehearsal method (Aljundi et al., 2019b).
A feature graph does not retain the intuitive meaning of a regular graph.
Take a social network for example, in a regular graph, each user is a node and user connections are edges.
In feature graphs, each user is a graph, the user features are nodes, including age, gender, etc.
Therefore, user classiﬁcation become a graph classiﬁcation problem and the increasing users become multiple training samples.
Since the feature number is more stable than the node number in a regular graph, thus the size of a feature graph is more stable than its regular graph, which makes classiﬁcation easier.
We list their detailed relationship in Table 1 and will further explain them in Section 4.
[Table]
The most difﬁcult and important step to create feature graphs is to establish the feature node connections (edges).
We observe that the node features of a regular graph are often propagated in an element-wise manner (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018; Klicpera et al., 2019; Zeng et al., 2020), hence better way of feature interaction is needed.
To solve this problem, we propose
FIX to deﬁne the feature adjacency matrix as features’ cross-correlation with neighbor nodes’ features, which allows the feature graph to learn feature interactions explicitly.
This is crucial for many graphstructured tasks, as useful information can be encoded in one’s neighbor features.
Take the citation
NEW graph as an example, a keyword in an article may have an impact on the importance of another keyword in an article citing the former one.
Feature graph is of constant computational complexity with the increased learning tasks.
We will demonstrate both the effectiveness and efﬁciency of the feature graphs.
To the best of our knowledge, it might the ﬁrst practical method, which overcomes the difﬁculty of directly applying lifelong learning techniques to graph networks.
R ELATED W ORK
The existing lifelong graph learning methods are very limited, thus we will present the related work on lifelong learning and graph neural networks, respectively.
2. L IFELONG L EARNING
Non-rehearsal methods Methods in the ﬁrst category do not preserve any old data.
To alleviate the forgetting problem, progressive neural networks (Rusu et al., 2016) leveraged prior knowledge via lateral connections to previously learned features.
Learning with forgetting (LwF) (Li & Hoiem, 2017) introduced a knowledge distillation loss (Hinton et al., 2015) to neural networks, which encouraged the network output for new classes to be close to the original outputs.
Distillation loss was also applied to learning object detectors incrementally (Shmelkov et al., 2017).
Learning without memorizing (LwM) (Dhar et al., 2019) extended LwF by adding an attention distillation term based on attention maps and demonstrated that it is helpful for retaining information of the old classes.
EWC (Kirkpatrick et al., 2017) remembered old tasks by slowing down learning on important weights.
RWalk (Chaudhry et al., 2018) generalized EWC and improved weight consolidation by adding a KL-divergence-based regularization.
Memory aware synapses (MAS) (Aljundi et al.,
FIX 2018) computed an importance value for each parameter in an unsupervised manner based on the sensitivity of output function to parameter changes.
Wei et al.
(2019) presented an embedding frameNEW work for dynamic attributed network based on parameter regularization.
A sparse writing protocol is introduced to a memory module (Wang et al., 2020), ensuring that only a few memory spaces can be affected during training.
Rehearsal methods Rehearsal methods can be roughly divided into rehearsal with synthetic data or exemplars from old data (Robins, 1995).
For example, to ensure that the loss of exemplars does not increase, gradient episodic memory (GEM) (Lopez-Paz & Ranzato, 2017) introduced orientation constraints during gradient updates.
Inspired by GEM, Aljundi et al.
(2019b) selected exemplars with a maximal cosine similarity of the gradient orientation.
iCaRL (Rebufﬁ et al., 2017) preserved a subset of images with a herding algorithm
(Welling, 2009) and included the subset when updating the network for new classes.
EEIL (Castro et al., 2018) extended iCaRL by learning the classiﬁer in an end-to-end manner.
Wu et al.
(2019) further extended iCaRL and discovered that updating the model with class-balanced exemplars further improved the performance.
Similarly, Hou et al.
(2019); Belouadah & Popescu (2019) further added constraints to the loss function to mitigate the effect of imbalance.
To reduce the memory consumption of exemplars, Iscen et al.
(2020) applied the distillation loss to feature space without having to access to the corresponding images.
Several exemplar selection methods are tested in (Zhou et al., 2020) for graph networks, while it has a different problem formulation with this paper.
Rehearsal approaches with synthetic data based on generative adversary networks (GAN) were used to reduce the dependence of the exemplars on old data (Shin et al., 2017; Wu et al., 2018;
He et al., 2018; Xiang et al., 2019).
2. G RAPH N EURAL N ETWORKS
Graph neural networks have been widely used to solve problems about graph-structured data (Zhou et al., 2018).
The spectral network was the ﬁrst to extend convolution to the graph domain (Bruna et al., 2014).
Later the graph convolutional network (GCN) (Kipf & Welling, 2017) followed this idea and alleviated the problem of overﬁtting on local neighborhoods via ﬁrst order Chebyshev expansion.
This obtained a good generalization ability and inspired more recent work.
For example, to identify the importance of neighborhood features, graph attention network (GAT) (Veliˇckovi´c et al., 2018) added an attention mechanism into GCN, which further improved the performance on citation networks and the protein-protein interaction dataset.
However, GCN and its variants require the entire graph during training, thus they cannot scale to large graphs.
To solve this problem and train GNN with mini-batches, a sampling method, GraphFIX
SAGE (Hamilton et al., 2017) is introduced to learn a function to generate node embedding by sampling and aggregating neighborhood features.
JK-Net (Xu et al., 2018) followed the same sampling
FIX strategy and demonstrated a signiﬁcant accuracy improvement on GCN with jumping connections.
DiffPool (Ying et al., 2018b) learned a differentiable soft cluster assignment to map nodes to a set of clusters, which then formed a coarsened input for the next layer.
Ying et al.
(2018a) designed a training strategy that relied on harder-and-harder training examples to improve the robustness and convergence speed of the model.
S-GCN (Chen et al., 2018a) developed control variate-based algorithms to sample small neighbor sizes and used historical activation in the previous layer to avoid redundant re-evaluation.
FastGCN (Chen et al., 2018b) applied importance sampling to reduce variance and performed node sampling for each layer independently, which resulted in a constant sample size in all layers.
Huang et al.
(2018) sampled the lower layer conditioned on the top one ensuring higher accuracy and ﬁxed-size sampling.
Subgraph sampling techniques were also developed to reduce memory consumption.
ClusterGCN (Chiang et al., 2019) sampled a block of nodes in a dense subgraph identiﬁed by a clustering algorithm and restricted the neighborhood search within the subgraph.
GraphSiant
(Zeng et al., 2020) constructed mini-batches by sampling the training graph.
A complete GCN with a ﬁxed number of well-connected nodes is built from the properly sampled subgraph during each iteration.
Nevertheless, most of the sampling techniques still require a pre-processing of the entire graph to determine the graph sampling process or require a complete graph structure, which makes those algorithms not directly applicable to lifelong learning.
In this paper, we hypothesize that a different graph structure is required for lifelong learning, and the new structure is not necessary to maintain the original/intuitive meaning of the nodes and edges.
P ROBLEM F ORMULATION
We start from deﬁning regular graph learning before lifelong graph learning for coherence.
An attribute graph is deﬁned as G =
( V , E ) , where V is a set of nodes and E ⊆ (cid:8) { v a , v b } | ( v a , v b )
∈ V 2 (cid:9) is a set of edges.
Each node v ∈ V is associated with a target vector z v ∈ Z and a multi-channel feature vector x v ∈ X ⊂ R F × C .
Each edge e ∈ E is associated with a weight vector w e ∈ W ⊂ R W .
In regular graph learning (Kipf & Welling, 2017), we aim to learn a predictor f to predict the target vector z v associated with a node x v , v ∈ V (cid:48) , given the graph G , nodes features X , weights vectors
W , and part of the targets z v , v ∈ V \ V (cid:48) .
In lifelong graph learning, we have the same objective with the regular graph learning, but we are only able to obtain the graph-structured data as a data continuum G L = { ( x i , t i , z i , N k
=1: K ( x i ) ,
W k =1: K ( x i ))
i =1: N } , where each item is formed by a node feature x i ∈ X , a task descriptor t i ∈ T , a target vector z
i ∈
Z t i , a k -hop neighbor set N k =1: K ( x i ) , and a weight vector set W k =1: K ( x i ) associated with the k -hop neighbors.
For simplicity, we will use the symbol N ( x i ) to denote the available neighbor set and their associated weights.
We assume that every item ( x i , N ( x i ) , t i , z i ) satisﬁes ( x i , N ( x i ) , z i ) ∼ P t i ( X , N ( X ) , Z ) , where P t i is a probability distribution and describes a single learning task.
In lifelong graph learning, we will observe, example by example, the continuum of the graph-structured data as
[Equation]
While observing (1), our goal is to learn a predictor f L to associate a test sample ( x , N ( x ) , t ) with a target z such that ( x , N ( x ) , z ) ∼ P t .
Such test sample can belong to a task that we have observed in the past, the current task, or a task that we will observe (or not) in the future.
Note that examples are not drawn locally identically and independently distributed ( iid ) from a ﬁxed probability distribution, since we don’t know the task boundary and a long sequence from the current task may be observed before switching to the next task.
In the continuum (1), we only know the label of x i , but have no information about the labels of its neighbors N ( x i ) .
In the experiments, the edge
FIX weight is a scalar and k = 1 .
Note that the task descriptors deﬁned here is for compatibility with the lifelong learning algorithms that requires such information but it is not used in the experiments.
F EATURE G RAPH N ETWORK
To better show the relationship with a regular graph, we ﬁrst review the graph convolutional network (GCN) (Kipf & Welling, 2017).
Given an attribute graph G =
( V , E ) described in Section 3, the stacked node features can be written as X =
[ x 1 , x 2 , · · · , x N ] T ∈ R N × FC , where x i ∈ R FC is a vectorized node feature.
The GCN takes feature channel as C = 1 , so that x i ∈ R F , X ∈ R N × F .
In this sense, the l -th graph convolutional layer in GCN can be deﬁned as
[Equation]
where σ ( · ) is a non-linear activation function, W ∈ R F ( l ) × F ( l +1) is a learnable parameter, and ˆ A ∈ R N × N is the symmetrically normalized adjacency matrix A with self-loops.
[Equation]
For unweighted graph, A [ i,j ] = 1 for ( i, j ) ∈ E and A
[ i,j ] =
otherwise.
The node features x i =1: N share the same learnable weights W , thus it can be regarded as a speciﬁc convolution.
It doesn’t change the number of nodes (rows of X ), but is able to change the feature dimension F ( l ) to F ( l +1) .
GCN has been applied to many graph-structured tasks due to its simplicity and good generalization ability.
However, the problems of GCN are also obvious.
The node features in the next layer is a linear combination of the node features in the current layer, thus its graph propagation cannot directly model the feature interaction.
Moreover, to propagate node features across layers, a full adjacency matrix A is required, which means that the entire graph topology has to present to the model.
This limits its usage to large graphs and makes it inapplicable to lifelong learning either.
It becomes more severe when the neighbor features are concatenated and propagated to the next layer iteratively (Xu et al., 2019), resulting in the effect of “neighbor explosion.”
As mentioned in
Section 2, to scale GCN to large graphs, many sampling techniques are proposed, however, most of them are still difﬁcult to directly apply to lifelong learning.
Therefore, we argue that such difﬁculty is caused by the inappropriate graph topology and propose to establish the feature graphs.
4. F EATURE N ODES
Recall that each node v in a regular graph G =
( V , E ) is associated with a multi-channel feature vector x = (cid:104) x
T[1 , :] , · · · , x T[ F, :] (cid:105) ∈ R F × C , where x [ i, :] is the i -th feature (row) of x .
An attribute feature graph takes the features of a regular graph as nodes.
It can be deﬁned as G F =
( V F , E F ) , where each node v F ∈ V F is associated with a feature x T
[ i, :] and will be denoted as x F i .
Intuitively, the number of nodes in the feature graph is the feature dimension in the regular graph, i.e. , (cid:12)(cid:12) V F (cid:12)(cid:12) =
F , and the feature dimension in feature graph is the feature channel in a regular graph, i.e. , x F i ∈ R C .
Therefore, we can deﬁne the feature nodes for feature graph as
[Equation]
In this way, for each node v ∈ V , we can construct an associated feature graph G F .
We next establish their associated feature adjacency matrices via feature interaction.
4. F EATURE A DJACENCY M ATRIX
For each item in continuum (1), the edges between x and its neighbors N ( x ) imply the relationship of their features.
Therefore, feature interaction can be modeled as the cross-correlation matrix of the feature vectors, so that the k -hop c -th channel feature adjacency matrix can be deﬁned as:
[Equation]
where sgnroot( x ) = sign( x ) (cid:112) ( | x | ) and w x,y is the associated edge weight.
Note that x [: ,c ] in (5) is the c -th column (channel) of x , while the feature node x
[ i, :] is the i -th row of x .
Intuitively, we have A F k,c ( x ) ∈ R F × F , where F (cid:28) N due to the settings of lifelong learning.
We can calculate multiple feature adjacency matrices if the edge weight is a high dimensional vector.
In practice, the expectation in (5) is approximated by averaging over the observed neighbors.
[Equation] [Equation]
where we we need to change x
[: ,c ] y T
[: ,c ] to (cid:16) x [: ,c ] y T
[: ,c ] + y [: ,c ] x T[: ,c ] (cid:17) for undirected graphs.
Therefore, the feature adjacency matrix can be constructed online via (6), together with the availability of new samples in the continuum.
In this way, the continuum (1) is converted to
[Equation]
where G F i = (cid:0) V F , A F (cid:1) .
This means that our objective becomes learning a graph predictor
[Equation]
) to associate a test sample ( G F , t ) to target vector z such that ( G F , z ) ∼ P F t .
Note that the samples in the new continuum (7) are still non- iid .
In this way, a lifelong graph learning problem becomes a regular lifelong learning problem similar to (Lopez-Paz & Ranzato, 2017) and the problem of
NEW increasing nodes in graph learning can be solved by applying lifelong learning to the continuum.
4. F EATURE B ROADCAST L AYER
Given the feature graph G F , the l -th feature broadcast layer can be deﬁned as
[Equation]
9) where σ ( · ) is a non-linear activation function, ˆ A F k is the associated normalized feature adjacency matrix, and W ∈ R C ( l ) × C ( l +1) is a learnable parameter.
Normally, the lifelong graph learning
NEW requires that the number of graph layers L is larger than the availability of K -hop neighbors, i.e.,
L > K .
It is worth noting that, although the deﬁnition of (9) appears similar to (2), they have totally different meanings.
For simplicity, the channel c is left out in (9) and the layer channel is NEW broadcasted independently.
The node feature x F in (9) can be a row of X in (2) if C = 1 .
In this sense, the increasing nodes in a regular graph becomes feature graphs with different adjacency matrices.
Similar to (2), the feature broadcast layer doesn’t change the number of feature nodes.
However, this is not always necessary for feature graphs, as our objective has been turned into graph classiﬁcation.
Therefore, we also deﬁne a feature transform layer to change the number of feature nodes and present it in Appendix E due to limited pages.
Note that we can also construct other types of layers based on the topology G F , e.g. , a graph attention layer (Veli ˇ ckovi ´
c et al., 2018) or extend convolution to kervolution (Wang et al., 2019).
However, they are out of the scope of this paper, we will demonstrate the effectiveness of the proposed feature graph using the simple feature broadcast layer and feature transform layer in the experiments.
4. A SSUMPTION , A NALYSIS , AND C OMPUTATIONAL C OMPLEXITY
As mentioned before, we assume that some useful information of a regular node is encoded in its neighbors’ features.
Therefore, we expect that our feature graph can perform better because of the explicit modeling of feature interaction.
Intuitively, a typical regular graph is propagated by calculating ˆ AX =
ˆ A [ x 1 , x 2 , · · · , x N ] T , which means that the node features in the next layer are element-wise weighted average of the current node features.
As a comparison, a feature of feature graphs are able to interact with another by calculating ˆ A F x F , where the feature adjacency matrix A F is a function of cross-correlation matrix of features.
This can be crucial for many graphstructured tasks that node features or part of the features have a strong relationship with each other.
Feature graphs have a low computational complexity.
Intuitively, the complexity for calculating the feature adjacency matrix is O ( nF 2( l ) ) , where n is the expected number of neighbors of each node in the continuum.
Regardless of the number of layers which is a constant for speciﬁc models, the complexity of graph propagation for a feature broadcast layer and a feature transform layer is of
O ( F 2( l ) C ( l ) C ( l +1) ) and
O ( F ( l ) F ( l +1) C 2( l ) ) , respectively.
If we take E ( l ) = F ( l ) C ( l ) as the number of elements of the features, the complexity for calculating each new item in the continuum is roughly of O ( E 2 + nF 2 ) , where we leave out the layer index for simplicity.
Note that they are irrelevant to the task number, thus feature graphs have constant complexity with the increased learning tasks.
4. L IFELONG L EARNING
Since our objective has become a regular lifelong learning problem by deﬁning the feature graph continuum (7), our feature graph models can inherit the achievements in lifelong learning developed for CNN, such as the weight consolidation (Kirkpatrick et al., 2017) or knowledge distillation (Rebufﬁ et al., 2017).
This further veriﬁes the generalization ability of feature graphs.
As we only aim to demonstrate its effectiveness in lifelong learning, we will adopt a simple yet effective rehearsal method with random sample selection (Aljundi et al., 2019b).
As reported in (Aljundi et al., 2019b), a uniform sampling tends to produce lower selection probability for earlier items.
To compensate for such effect, we set the probability that the n -th observed item is selected at time t as
[Equation]
where M denotes the memory size.
We present the derivation of (10) in the Appendix A.
E XPERIMENTS
We perform comprehensive experiments using the feature broadcast layer on three popular citation graph datasets including Cora, Citeseer, and Pubmed (Sen et al., 2008), whose detailed statistics is listed in Appendix B.
We construct two different continuum for each dataset, i.e. , data-incremental and class-incremental.
For data-incremental tasks, all samples are randomly observed, while in class-incremental tasks, i.e. , each task contains one class and all samples from one task are observed before switching to the next task.
For both tasks, each node can only be present to the model once, i.e. , a sample cannot update model parameters again once it is dropped.
More implementation details are presented in Appendix C. We adopt the same evaluation metric used in (Aljundi et al., 2019b), which is the overall test accuracy after learning all tasks.
We also test the feature transform layer on a social network dataset Flickr, which is presented in Appendix F due to limited pages.
As the existing graph models are difﬁcult to directly apply to lifelong learning, we lack off-the-shelf methods to compare.
Therefore, we slightly modify one of the state-of-the-art methods, GraphSAGE (Hamilton et al., 2017) to meet the lifelong learning settings.
GraphSAGE needs to traverse the entire graph for every layer which is impossible for lifelong learning.
To be fair, we set the neighbor sampling of GraphSAGE to available neighbors in the continuum.
We also adopt the same lifelong learning method in Section 4 to alleviate its forgetting problem.
Other graph models, such as GCN (Kipf & Welling, 2017), GAT (Veli ˇ ckovi ´
c et al., 2018), and PPNP (Klicpera et al.,
) require a full adjacency matrix, GraphSiant (Zeng et al., 2020) requires a pre-processing of the entire dataset to determine the sampling process, which cannot satisfy the requirements of lifelong learning.
Intuitively, lifelong learning has a performance upper bound given by non-lifelong learning settings.
GraphSAGE achieves an accuracy of 0 . 850 , 0 .
768 , and 0 .
858 on the three datasets, respectively, while feature graph achieves a comparable accuracy of 0 .
788 , and 0 .
872 , respectively.
It will be shown that feature graph in lifelong learning is approaching this upper bound and has a much better performance than GraphSAGE, which veriﬁes its effectiveness.
5. D ATA -I NCREMENTAL T ASKS
The overall performance on data-incremental tasks is reported in Table 2, where all performance is an average of ten runs.
For a fair comparison, we use a memory size of 100 for both methods.
It can be seen that the proposed feature graph achieves an average of 5 .
5% , 1 . 9% , and 5 .
6% higher accuracy than GraphSAGE, respectively.
It is also worth noting that both methods have a low standard deviation, which means that their performance is stable and not sensitive to the data sequence.
[Table]
5. C LASS -I NCREMENTAL T ASKS
Samples in the class-incremental tasks are non-
iid
and we have no prior knowledge on the task boundary, thus it is more difﬁcult than data-incremental tasks.
We use a memory size of 200 and list their overall test accuracy in Table 3.
It can be seen that the proposed feature graph achieves an average of 5 .
8% higher accuracy on the three datasets.
This is a large margin and we expect that other customized layers as well as extensive hyper-parameter searches can further improve the performance.
We also perform more tests with larger memory in Appendix D
[Figure]
[Table]
5. A BLATION S TUDY
As reported in (Aljundi et al., 2019b)
[Figure]
the number of exemplars from old data, which is also referred to as memory size M
[Figure]
may have a signiﬁcant impact on lifelong learning.
This section explores its inﬂuence on the feature graphs.
We present the performance of both data-incremental and classincremental tasks for the Cora dataset in Figure
[Figure]
which shows the averaged performance over ten runs.
It can be seen in Figure 2a that even without any exemplars, the feature graph achieves nearly the same average precision compared to a setup with a larger memory size.
This means that the feature graph can keep most of the knowledge for iid samples.
Since the samples are non- iid in class-incremental learning, larger memory size will produce a better accuracy as in Figure 2b.
[Figure]
tasks.
5. E FFICIENCY
As in Table 4, our feature graph requires only 1 / 10 parameters compared with GraphSAGE on the
Cora dataset, which demonstrates its efﬁciency.
As a comparison, we also list the number of parameters of GCN, which is the fastest method as reported in (Fey & Lenssen, 2019).
[Table]
C ONCLUSION
To solve the problem of lifelong graph learning, this paper introduces the feature graph as a new graph topology, which can be trained via mini-batches naturally.
It takes features of a regular graph as nodes and takes the original nodes as independent graphs.
This successfully converts the original node classiﬁcation to graph classiﬁcation and turns the problem from lifelong graph learning into regular lifelong learning.
To construct the feature adjacency matrices, we accumulate the crosscorrelation matrices of the connected feature vectors to model feature interactions.
The comprehensive experiments show that the feature graph achieves much higher accuracy than the state-of-the-art methods in both data-incremental and class-incremental tasks.
This might be one of the ﬁrst practical models that bridges graph networks to lifelong learning.
We note that this topic is still far from well developed, and we expect further improved performance via more customized layers and model architectures.
We will also perform more experiments on other types of datasets in the future
A S AMPLE S ELECTION IN L IFELONG L EARNING
In the experiments we adopt the simple rehearsal method with random sampling described in (Aljundi et al., 2019b)
Let P be the probability that the observed items are selected at time t , thus the probability that one item is still kept in the memory after k selection is P k .
This means that earlier items have lower probability to be kept in the memory and this phenomenon was observed in (Aljundi et al., 2019b)
(Sec. 4.2).
To compensate for such effect and
Proposition A.1.
To ensure that all items in the continuum have the same probability to be kept in the memory at any time t we can set the probability that the n -th item is selected at time t as
[Equation]
where M denotes the memory size.
Proof.
The proof is obvious for t (cid:54) M as we only need to keep all items in the continuum.
For t > M , the probability that the n -th item is still kept in the memory at time t is
[Equation]
This means the probability P n,t is irrelevant to n and all items in the continuum share the same probability.
In practice, we keep M items and sample balanced items across different classes.
B S TATISTICS OF C ITATION D ATASETS
We perform comprehensive experiments on popular citation graph datasets including Cora, Citeseer, and Pubmed (Sen et al., 2008).
The nodes in a citation graph are articles and edges are references among the articles.
The dataset statistics are listed in Table 5.
[Table]
C I MPLEMENTATION D ETAILS
In all tasks, we adopt a feature graph model consisting of two feature broadcast layers and one fully connected layer.
For hidden layers, feature channel is set as C (1) = 1 and C (2) =
We take the
NEW one-hot vector as the target
vector
z i , adopt the cross-entropy loss, and use the softsign function σ ( x ) =
x / (1+ | x | ) as the non-linear activation (Elliott, 1993).
During training, we update the model parameters for 5 times using each newly observed samples and apply the stochastic gradient descent (SGD) (Kiefer et al., 1952) for optimization, where a mini-batch size of 10 and a learning rate of 0.01 are adopted.
Our implementation is based on PyTorch (Paszke et al., 2017) and tested on a
Nvidia GPU of GeForce GTX 1080Ti.
D P ERFORMANCE ON C ITATION G RAPH
Due to space limitation, we only show the performance using a memory size of 100 and 200 in
Section 5, respectively.
In this section, we further show that with a larger memory, feature networks are able to achieve much higher performance than GraphSAGE.
Table 6 and Table 7 present the average performance with 10 runs on the three citation datasets with a memory size of 500 for dataincremental and class-incremental tasks, respectively.
It can be seen that the feature graph achieves an average of 3 . 1% and 4 .
8% improvements on the two tasks, respectively.
This further demonstrates the effectiveness of the proposed feature graph.
[Table] [Table]
[Figure]
NEW
We next show the per-class precision.
After each class is learned, we measure the model precision for all classes, resulting in a per-class precision matrix P ∈ R T × T , where T is the total number of classes.
Each element P i,j is the model precision on the i -th class after the j -th class is learned.
Figure 3a shows the difference of precision matrix with GraphSAGE on the Cora dataset, where the upper triangular matrix is zero due to the class is not learned at that time.
It can be seen that our feature graph achieves a much higher per-class precision than GraphSAGE.
[Figure]
The feature graph also has a lower forgetting effect.
To show this, we deﬁne the per-class forgetting rate as F ∈ R T , where each element
[Equation]
We show the difference of forgetting rate with GraphSAGE in Figure 3b, where the last class forgetting difference is zero because that class is just learned.
Figure 3b demonstrated that the proposed feature graph largely alleviates the issue of catastrophic forgetting in lifelong graph learning.
E F EATURE T RANSFORM L AYER
Similar to graph convolutional layer (2), the feature broadcast layer deﬁned in Section 4.3 is able to retain the number of feature nodes in hidden layers.
This is a requirement for node classiﬁcation but may not be necessary for graph classiﬁcation.
In this section, we further deﬁne a feature transform layer which can change the number of feature nodes and will be helpful for reducing the number
NEW of learnable parameters.
Different from the feature broadcast layer, we also need to re-calculate the feature adjacency matrices from transformed neighbors.
Therefore, given the feature graph G F , the l -th feature transform layer can be deﬁned as
[Equation]
where W F ∈ R F ( l +1)
× F ( l ) is a learnable parameter, σ ( · ) is a non-linear activation function, and ˆ A F ( l ) ( x ) ∈ R F ( l ) × F ( l ) is the normalized feature adjacency matrix of A F ( l ) ( x ) , which is calculated from transformed feature and neighbors following (5).
F P ERFORMANCE OF F EATURE T RANSFORM L AYER
In this section, we further evaluate the feature graph using the feature transform layer.
We construct the network by concatenating two feature transform layers with one fully connected layer.
We test the network on larger datasets, ogbn-arXiv (Mikolov et al., 2013) and Flickr (McAuley & Leskovec, 2012), which are listed in Table 8.
Ogbn-arXiv is a citation dataset, while Flickr is built by forming links between images sharing common metadata.
Edges are formed between images from the same location, submitted to the same gallery, group, or set, etc. [Table]
A memory size of 500, a mini-batch size of 64, and a learning rate of 0.01 are adopted for obgnarXiv. The model is updated by Adam (Kingma & Ba, 2015) optimizer for 5 times using each new item in the continuum.
Table 9 shows the performance of our method together with the GraphSAGE.
Feature graph achieve a lower performance in the non-lifelong setting, while a higher performance in lifelong learning.
This further demonstrate the effectiveness of feature graphs in lifelong learning.
[Table]
We test more memory sizes using the ﬂickr dataset and update the model once using each newly available samples.
We adopt the Adam optimizer, where a mini-batch size of 10 and a learning rate of 0.001 are adopted.
It is worth noting that the non-lifelong learning method GraphSIANT (Zeng et al., 2020) achieves an upper bound accuracy of 0 . 51 , which indicates that Flickr is very difﬁcult.
Nevertheless, it can be seen in Table 10 that our feature graph achieves an average accuracy of 0 .
470 and 0 .
468 even with a zero memory size for the data-incremental and class-incremental tasks, respectively.
This further demonstrates the effectiveness of the proposed feature graph topology.
We observe an overﬁtting for larger memory on the Flickr dataset.
This is because we adopt a simple updating strategy using the memory samples and larger memory results in more update steps.
We believe that a well-designed updating strategy is able to further improve the performance, but it is out the scope of this paper.
[Table]