O N M AKING G RAPH C ONTINUAL L EARNING E ASY , F OOL -P ROOF , AND E XTENSIVE : A B ENCHMARK F RAMEWORK AND S CENARIOS
Anonymous authors
Paper under double-blind review
A BSTRACT
Continual Learning (CL) is the process of learning ceaselessly a sequence of tasks.
Most existing CL methods deal with independent data (e.g., images and text) for which many benchmark frameworks and results under standard experimental settings are available.
CL methods for graph data, however, are surprisingly underexplored because of (a) the lack of standard experimental settings, especially regarding how to deal with the dependency between instances, (b) the lack of benchmark datasets and scenarios, and (c) high complexity in implementation and evaluation due to the dependency.
In this paper, regarding (a), we define four standard incremental settings (task-, class-, domain-, and time-incremental settings) for graph data, which are naturally applied to many node-, link-, and graph-level problems.
Regarding (b), we provide 23 benchmark scenarios based on 14 real-world graphs.
Regarding (c), we develop B E G IN , an easy and fool-proof framework for graph
CL.
B E G IN is easily extended since it is modularized with reusable modules for data processing, algorithm design, and evaluation.
Especially, the evaluation module is completely separated from user code to eliminate potential mistakes in evaluation.
Using all above, we report extensive benchmark results of seven graph
CL methods.
Compared to the latest benchmark for graph CL, using B E G IN , we cover 3 × more combinations of incremental settings and levels of problems.
I NTRODUCTION
Continual Learning (CL), which is also known as lifelong learning and incremental learning, is the process of learning continuously a sequence of tasks.
CL aims to retain knowledge from previous tasks (i.e., knowledge consolidation) to overcome the degradation of performance (i.e., catastrophic forgetting) on previous tasks.
Recently, CL has received considerable attention because of its similarity to the development of human intelligence.
Most of the existing works for CL deal with independent data, such as images and text.
For example, Shin et al.
(2017) aim to learn a sequence of image-classification tasks where the domains of images vary with tasks (i.e., under a domain-incremental setting), and Ermis et al.
(2022) aim to learn a sequence of text-classification tasks where the possible classes of text grow over tasks (i.e., under a class-incremental setting).
For CL with independent data, many datasets (Hsu et al., 2018;
Goodfellow et al., 2013; Lomonaco & Maltoni, 2017) and benchmarks (Lomonaco et al., 2021; Lin et al., 2021; Pham et al., 2021; He & Sick, 2021; Shin et al., 2017) have been provided.
CL is naturally beneficial to graph data in the real world, which grows in size and diversity while being applied to more and more domains.
Especially, since real-world graphs are massive and changing ceaselessly, it is computationally inefficient to train a new model per task or change, and
CL is a promising remedy.
However, compared to CL with independent data, CL with graph data (graph CL)
(Febrinanto et al., 2022; Galke et al., 2021; Zhou & Cao, 2021) has been relatively underexplored, mainly due to the complexity caused by the dependency between instances.
For example, in node classification, the class of a node is correlated not only with its features but also with its connections to other nodes, which may belong to other tasks, and their features.
In addition to the complexity, major reasons of relative unpopularity of graph CL include (a) the lack of standard experimental settings, especially regarding how to deal with changes in various dimensions (e.g.,
[Table]
domains and classes) and the dependency between instances, and (b) the lack of benchmarking datasets and scenarios.
In this paper, we focus on resolving these issues.
Our first contribution is to define four incremental settings for graph data.
To this end, we identify and decouple four possible dimensions of changes, which are tasks, classes, domains, and time.
Each of the settings is defined so that (a) the dependency between instances, which is a unique property of graph data, can be utilized, and at the same time, (b) catastrophic forgetting may happen without careful consideration of it.
Then, we show that the settings can be applied to node-, link-, and graph-level problems, including node classification, link prediction, and graph classification.
After that, we provide 23 benchmark scenarios for graph from 14 real-world datasets, which cover 12 combinations of the incremental settings and the levels of problems, as summarized in Table 1.
Our second contribution is B E G IN ( Be nchmarking G raph Cont in ual Learning), which is a framework for implementation and evaluation of graph CL methods.
Evaluation is complicated for graph
CL due to additional dependency between instances.
Specifically, instances of one task are often used also for other tasks to exploit the dependency, and thus without careful consideration, information that should not be shared among tasks can be leaked among tasks.
In order to eliminate potential mistakes in evaluation, B E G IN is fool-proof by completely separating the evaluation module from the learning part, where users implement their own graph CL methods.
The learning part only has to answer queries provided by the evaluation module after each task is processed.
B E G IN is easy-to-use .
It is easily extended since it is modularized with reusable modules for data processing, algorithm design, validation, and evaluation.
Our last contribution is extensive benchmark results of seven graph CL methods based on our proposed scenarios and framework.
We use four metrics for evaluation from various perspectives.
For reproducibility, we provide all source code required for reproducing the benchmark results and documents at https://anonymous.4open.science/r/BeGin-1C33/ .
R ELATED W ORKS
Continual Learning with Independent Data.
Continual learning (CL) methods have been developed mostly for independent data (e.g., images and text), and they are mainly categorized into regularization-, replay-, and parameter-isolation-based methods.
Regularization-based methods (Kirkpatrick et al., 2017; Aljundi et al., 2018; Li & Hoiem, 2017) seek to consolidate knowledge from previous tasks by introducing regularization terms in the loss function.
For example, Elastic
Weight Consolidation (EWC) (Kirkpatrick et al., 2017) weights to parameters according to the diagonal of the Fisher information matrix, and Memory Aware Synapses (MAS)
(Aljundi et al., 2018) computes the importance of parameters according to how sensitive the parameters are.
Learning without Forgetting (LwF) (Li & Hoiem, 2017) minimizes the difference between outputs of a previous model and a new model.
Replay-based methods store a sample of data for previous tasks (Chaudhry et al., 2019; Isele & Cosgun, 2018;
Rolnick et al., 2019; Rebuffi et al., 2017).
Then, they re-use the data while learning a new task to mitigate forgetting.
For example, Gradient Episodic
Memory (GEM) (Lopez-Paz & Ranzato, 2017) stores data from previous tasks and prevents the increase of losses on them while learning a new task.
Parameter-isolation-based methods (Aljundi et al., 2017; Fernando et al., 2017; Mallya & Lazebnik, 2018;
Serra et al., 2018) allocate different parameters to each task to prevent forgetting.
For example, Progressive Neural Network (Rusu et al., 2016) freezes parameters for previous tasks while learning a new task.
There exist a number of frameworks (Lomonaco et al., 2021; Lin et al., 2021; Pham et al., 2021; He & Sick, 2021; Shin et al., 2017) for implementation and evaluation of continual learning methods for independent data.
However, none of them currently supports CL with graph data.
Continual Learning with Graph Data.
Due to their expressiveness, graphs are widely used to model various types of data, and thus considerable attention has been paid to machine learning with graph-structured data.
Since many such graphs (e.g., online social networks) evolve over time, continual learning is desirable for them, and thus several CL methods for graph-structured data have been developed (Febrinanto et al., 2022;
Wang et al., 2020a; Liu et al., 2021; Zhou & Cao, 2021;
Galke et al., 2021; Wang et al., 2022).
For example, ER-GNN (Zhou & Cao, 2021) carefully samples nodes and uses them for re-training, and TWP (Liu et al., 2021) stabilizes parameters important in topological aggregation by graph neural networks through regularization.
The replayand regularization-based approaches are combined in CGNN (Wang et al., 2020a).
Despite these efforts, CL with graph data (graph CL) is still largely underexplored, especially compared with CL with independent data, and the lack of benchmark frameworks and scenarios is one major reason, as discussed in Section 1.
To the best of our knowledge, there exist only two benchmarks for graph
CL (Carta et al., 2021; Zhang et al., 2022).
However, Carta et al.
(2021) supports only graph-level tasks under one incremental setting, and Zhang et al.
(2022) supports node- and graph-level tasks under only two incremental settings, as further described in Section 3.2.
Compared to them, our benchmark and framework are more extensive, as summarized in Table 1.
Comparison with Other Graph Learning Methods.
In CL problems, distribution shifts may occur over tasks, and such shifts can be observed from training data for each task.
There also exist works that deal with unobservable shifts between training and test distributions Li et al.
; Wu et al.
; Baek et al.
For instance, Baek et al.
(2020) use meta-learning to address fewshot out-of-graph link prediction.
Moreover, transfer learning (TL) on graph data has been studied for various downstream tasks (e.g., link prediction Tang et al.
While the goal of CL is to train a single model for a sequence of tasks, TL aims to adapt a separate model to a new task.
In addition, many works on incremental/dynamic graph learning (Rossi et al., 2020; You et al., 2022) focus on the latest snapshot of a dynamic graph to maximize performance on it.
On the other hand,
Graph CL, especially, under Time-IL settings, aims not only for the current snapshot but also to preserve performance on past snapshots, which can be especially useful when seasonality is present.
B ENCHMARK S CENARIOS : P ROBLEMS , S ETTINGS , AND D ATASETS
We introduce our benchmark scenarios by describing the considered graph-learning problems, incremental settings, and real-world datasets.
Common Notations.
We denote each i -th task in a sequence by T i .
We use G =
( V , E , X ) to denote a graph that consists of a set of nodes V , a set of edges E , and node features X :
V → R d , where d is the number of node features.
In some of the considered datasets, edge features are given in addition to or instead of node features, and they can be treated similarly to node features.
Lastly, we use Q to indicate the set of queries used for evaluation.
3. G RAPH P ROBLEMS OF T HREE L EVELS
Our benchmark scenarios are based on various node-, link-, graph-level problems.
Below, we describe node classification, link prediction, and graph classification, as examples.
Node Classification (NC).
For a node classification (NC) task T i , the input consists of (a) a graph
G i =
( V i , E i , X i ) , (b) a labelled set V ′
i
∈
V i of nodes, (c) a set of classes C i , and (d) the class f ( v ) ∈ C
i for each node v ∈ V
i .
A query q on a NC task T i is a node v q / ∈ V ′ i where f ( v q ) ∈ C i , and its ground-truth answer is f ( v q ) .
Link Prediction (LP).
For a link prediction (LP) task T i , the input consists of a graph G i
=
( V i , E i \
E ′
i , X i ) , where E i is the ground-truth set of edges, and E ′ i ∈
E i is the set of missing edges among them.
A query on a LP task T i is a node pair ( u, v ) / ∈ ( E i \ E
′ i ) , and its ground-truth answer is 1 (( u, v ) ∈
E ′ i ) , i.e., whether there exist a missing edge between u and v or not.
Graph Classification (GC).
For a graph classification (GC) task T i , the input consists of (a) a labelled set of graphs S i , (b) a set of classes C i , and (c) the class f ( G )
∈
C
i for each graph G ∈ S i .
A query q on a GC task T i is a graph G q / ∈ S i
where f ( G q ) ∈ C i , and its ground-truth answer is f ( G q ) .
The problem definition of Link Classification (LC) problem, which is also used for our benchmark, is extended straightforwardly from that of node classification.
3. F OUR I NCREMENTAL S ETTINGS
We introduce four incremental settings for continual learning with graph data and describe how they can be applied to the above three problems.
When designing them, we aim to (a) decouple changes in different dimensions (tasks, classes, domains, and time), (b) make the dependency between instances (e.g., connections between nodes) exploitable, and (c) make the input for a task (partially) lost in later tasks so that catastrophic forgetting may happen without careful attention to it.
Task-Incremental (Task-IL).
In this incremental setting, the set of classes varies with tasks (i.e., ∀ i ̸
= j, C i ̸ = C j ), and they are often disjoint (i.e., ∀ i ̸
= j, C i ∩ C j = ∅ ).
In addition, for each query in Q , the corresponding task, which we denote by T i , is provided, and thus its answer is predicted among C i .
This setting is applied to NC and GC tasks, where the sets of classes can vary with tasks, and for NC tasks, the input graph is fixed (i.e., ∀ i ̸
= j , G i =
G j ).
Class-Incremental (Class-IL).
In this incremental setting, the set of classes grows over tasks (i.e., ∀ i <
j, C i ⊊ C j ).
In addition, for each query in Q , the corresponding task is NOT provided, and thus its answer is predicted among all classes seen so far (i.e., (cid:83) j ≤ i C j for a current task T i ).
This setting is applied to NC and GC tasks, where the sets of classes can vary with tasks, and for NC tasks, the input graph is fixed (i.e., ∀ i ̸
= j , G i =
G j ).
Domain-Incremental (Domain-IL).
In this incremental setting, we divided entities (i.e., nodes, edges, and graphs) over tasks according to their domains, which are additionally given (see Section 3.3 for real-world examples of domains).
The details for each problem is as follows:
• NC: The labelled nodes of the input graph are divided into NC tasks according to their domains.
The input graph is fixed (i.e., ∀ i ̸
=
j , G i =
G j ) for all NC tasks.
• LP: The ground-truth edges are partitioned into (a) the set ¯ E of base edges and (b) the set ˜ E of additional edges.
The base edges are provided commonly for all LP tasks, and they are especially useful when answering queries on past tasks.
The additional edges ˜
E are divided further into LP tasks according to their domains.
For each LP task T i , ¯ E ∪ ˜ E i , where ˜ E i is the additional edges assigned to T i , is used as the ground-truth edges (i.e., E i = ¯ E ∪ ˜ E i ), and the missing ground-truth edges are chosen among ˜ E
i , i.e., E ′ i ⊂ ˜
E i .
GC: The labelled graphs are divided into GC tasks, according to their domains.
Note that, as domains, we use information not directly related to labels, which we aim to infer.
Time-Incremental (Time-IL).
In this incremental setting, we consider a dynamic graph evolving over time.
We denote its i -th snapshot by G ( i ) =
( V ( i ) , E ( i ) , X ( i ) ) .
The set of classes may or may not vary across tasks, and the details for each problem is as follows:
• NC: The input graph for each NC task T i is the i -th snapshot G ( i ) of the dynamic graph, and labelled nodes are given among new nodes added to the snapshot (i.e., V ′ i
⊂ V ( i ) \ V ( i − 1) , where V (0) = ∅ ).
The label and features of each node are fixed over time and thus for all tasks.
• LP: As in the Domain-IL setting, base edges are used.
For each LP task T i , the set ¯ E i of base edges so far and the new edges added to the i -th snapshot G ( i ) of the dynamic graph are used as the ground-truth edges (i.e., E i =
¯ E i ∪E ( i ) \E ( i − 1) , where E (0) = ∅ ).
After each task is processed, a subset of E ( i ) \ E ( i − 1) \ E
′ i (i.e., new edges that are not used as missing edges) are added as base edges.
The features of each node are fixed over time and thus for all tasks.
• GC: The snapshots of the dynamic graph are grouped and assigned to tasks in chronological order.
Specifically, for any i and j where i < j , every snapshot in the labelled set S i of the GC task T i is older than every snapshot in S j of T j .
The features of nodes may change over time.
The above settings can also be applied to the link classification (LC) problem straightforwardly, as they are applied to the NC problem.
Remarks : The above Task/Class-IL settings are different from those in (Zhang et al., 2022), where the edges between labelled nodes are added to the input graph together with the labelled nodes in [Table]
each task. Thus, the dynamics of the input graph, which we independently consider in the Time-IL setting, are coupled with changes in tasks or classes, without any good reason.
3. 14 R EAL - WORLD DATASETS AND 23 B ENCHMARK S CENARIOS
We describe 14 real-world datasets and 23 benchmark scenarios based on them under various incremental settings.
Datasets for Node-Level Problems.
• Cora , Citeseer (Sen et al., 2008) , and CoraFull (Bojchevski & G ¨ unnemann, 2018) are citation networks.
Each node is a scientific publication, and its class is the field of the publication.
For Cora and Citeseer , based on six classes in each dataset, we formulate three binary classification tasks for Task-IL and three tasks with 2 , 4 , and 6 classes for Class-IL. Similarly, for CoraFull , we formulate 35 binary classification tasks for Task-IL. Note that, one class in Cora is not used.
• Nodes in ogbn - proteins (Hu et al., 2020; Szklarczyk et al., 2019) are proteins, and edges indicate meaningful associations between proteins.
For each protein, 112 binary classes, which indicate the presence of 112 functions, are available.
Each protein belongs to one among 8 species, which are used as domains in Domain-IL.
Each of the 8 task consists of 112 binary-classification problems.
• ogbn - arxiv
(Hu et al., 2020; Wang et al., 2020b) is a citation network, where each node is a research paper, and its class belongs to 40 subject areas, which are divided into 8 groups for TaskIL.
Similarly, the number of classes increase by 5 in each task in Class-IL.
Publication years are used for the Time-IL setting.
• ogbn - products (Hu et al., 2020; Chiang et al., 2019) is a co-purchase network, where each node is a product, and its class belongs to 47 categories, which are divided into 9 groups for Class-IL.
The number of classes increase by 5 in each task, and two categories are not used.
Dataset for Link-Level Problems.
• Bitcoin - OTC (Kumar et al., 2016; 2018) is a who-trust-whom network, where nodes are users of a bitcoin-trading platform.
Each directed edge has an integer rating between − 10 to 10 and a timestamp.
The ratings are divided into 6 groups.
Two of them are used separately for Task-IL and accumulated for Class-IL.
For Time-IL, we formulate 7 tasks using the timestamps, where the signs of the ratings are used as binary classes.
Since there is no external node feature, we use inand out-degrees as node features, as in (Errica et al., 2020).
• Wiki - CS (Mernyei & Cangea, 2020) is a hyperlink network between computer science articles.
Each article has a label indicating one of the 10 subfields that it belongs to.
For Domain-IL, the node labels are used as domains, and specifically, the edges are divide into 10 groups, according to the labels of their endpoints.
If the domains of its two endpoints are different, the domain considered in a later task is assigned to the edge.
• ogbl - collab (Hu et al., 2020; Wang et al., 2020b) is a co-authorship network, where nodes are authors.
We use publication years for the Time-IL setting.
Datasets for Graph-Level Problems.
[Figure]
[Figure] of B E G IN . • Images in MNIST and CIFAR10 (Dwivedi et al., 2020; Achanta et al., 2012) are converted to graphs of super-pixels.
There are 10 classes of graphs, and they are partitioned into 5 groups, which are used separately for Task-IL and accumulated for Class-IL.
• Graphs in ogbg - molhiv (Hu et al., 2020; Wu et al., 2018; Landrum et al., 2006) are molecules consisting of atoms and their chemical bonds.
The binary class of each graph indicates whether the molecule inhibits HIV virus replication or not.
For Domain-IL, we divide molecules intogroups based on structural similarity by the scaffold splitting procedure (Landrum et al., 2006).
• Graphs in Aromaticity (Xiong et al., 2019; Wu et al., 2018) are also molecules consisting of atoms and their chemical bonds.
The original dataset contains labels representing the number of aromatic atoms in each molecule.
In this paper, we divide molecules into 30 groups based on the labels and formulate Task-IL and Class-IL settings with 10 tasks.
• Each graph in NYC - Taxi 2 shows the amount of taxi traffic between locations in New York City during an hour in 2021 .
Specifically, nodes are locations, and there exist a directed edge between two nodes if there existed a taxi customer between them during an hour.
The number of such customers is used as the edge weight.
The date and time of the corresponding taxi traffic are used to partition the graphs into 12 groups for Time-IL.
The binary class of each graph indicates whether it indicates taxi traffic on weekdays (Mon.-Fri.) or weekends (Sat.-Sun.).
B E G IN : A P ROPOSED B ENCHMARK F RAMEWORK
In this section we present B E G IN ( Be nchmarking G raph Cont in ual Learning), our proposed benchmark framework for making graph continual learning (graph CL) (a) Easy : assisting users so that they can implement new graph CL methods with little effort, (b) Fool-proof: preventing potential mistakes of users in evaluation, which is complicated for graph CL and (c) Extensive : supporting various benchmark scenarios, including those described in Section 3.
The modularized structure of
B E G IN is shown in Figure 1, and below, we focus on its three components.
4. S CENARIO L OADER (L OADER )
The ScenarioLoader (loader in short) of B E G IN is responsible for communicating with user code (i.e., the training part) to perform a benchmark under a desired incremental setting.
First, the loader receives the entire dataset and the desired incremental setting as inputs.
Then, according to the inputs, it processes the dataset into a sequence of tasks, as described in Section 3.
Before each task starts, the loader provides (a) the input for the task and (b) the set Q of queries for evaluation to the user code.
Once the user code is done with the current task, the loader receives the predicted answers for the queries in Q .
Lastly, if there is no more task to be performed, the loader returns the evaluation results, which are computed by the evaluator module, to the user code.
In Figure 2, we provide an example of such communications between the loader and user code.
It should be noticed that the evaluation part, including the ground-truth answers of the queries, are never revealed to the user code even after all tasks are processed, in order to prevent potential mistakes and misuses by users.
Also note that all queries in Q are asked by the loader and answered by the user code after every task is processed, even when some of the queries are on unseen tasks.
Otherwise, information about the tasks that queries are on can be revealed to the user code and exploited for answering the questions, which is prohibited in Class-IL settings.
4. E VALUATOR
B E G IN provides the evaluator, which computes basic metrics (specifically, accuracy, AUROC, and
HITS@K) based on the ground-truth and predicted answers for the queries in Q provided by the loader after each task is processed.
The basic evaluator can easily be extended by users for additional basic metrics.
The basic metrics are sent to the loader, and for each basic metric, the basic performance matrix M ∈ R N × N , whre N is the number of tasks, is computed.
Each ( i, j ) -th entry
M i,j indicates the performance on on j -th task T j after the i -th task T i is processed.
Based on M , the following evaluation metrics are computed when k is greater than or equal to i .
• Average Performance (AP): Average performance on k tasks after learning the k -th task T k .
• Average Forgetting (AF): Average forgetting on ( k − 1 ) tasks after learning the k -th task T k .
We measure the forgetting on T i by the difference between the performance on T i after learning T k and the performance on T
i right after learning T i .
It is valid for 2 ≤ k ≤ N .
• Intransigence (INT)
(Chaudhry et al., 2018): Average intransigence on k tasks.
We measure the intransigence on T i by the difference between the performances of the Joint model (see Section 5.1) and the target model on T i after learning T i .
• Forward Transfer (FWT) (Lopez-Paz & Ranzato, 2017): Average forward transfer on ( k − 1 ) tasks.
We measure the forward transfer on T i by the difference between the performance on T i after learning T i − 1 and the performance on T i without any learning.
It is valid for 2 ≤ k ≤ N .
Formally, the evaluation metrics are defined as follows:
[Equation]
where M Joint is a basic performance matrix of the Joint model, and r i denotes the performance of a randomly initialized model on T i .
Note that AF quantifies forgetting of previous tasks, and INT measures performance on the current task.
FWT measures performance on future tasks, to quantify generalizable knowledge retained from previous tasks.
4. T RAINER
For usability, B E G IN provides the trainer, which users can extend when implementing and benchmarking new methods.
It manages the overall training procedure, including preparing the dataloader, training, and validation, so that users only have to implement novel parts of their methods.
As in (Lomonaco et al., 2021), the trainer divides the training procedure of continual learning as a series of events.
For example, the subprocesses in the training procedure where the trainer (a) receives the input for the current task, (b) trains the model for one iteration for the current task, and (c) handles the necessary tasks before and after the training are events.
Each event is modularized as a function, which users can fill out, and the trainer proceeds the training procedure with the event functions.
One thing we need to consider is that there can be cases where intermediate results generated in each event must be stored to be used in other events.
For example, in the EWC method, a penalty term for preventing catastrophic forgetting should be additionally considered to compute the training loss.
To compute the term, the learned parameters and the weights computed from the fisher information matrix on the previous tasks are needed, but they cannot be obtained on the current task.
In order to resolve this issue, the trainer provides a dictionary where intermediate results can
[Figure]
be stored and shared by events.
For the aforementioned EWC method, the learned parameters and computed weights on a task are stored in the dictionary and used for computing the training loss on the following tasks.
Figure 3 shows how the EWC method for node classification can be implemented with B E G IN .
In this case, users only need to fill out (a) init training states to initialize the dictionary for storing training states, (b) process train iteration to additionally consider the penalty term in the loss function, and (c) process after training , which is executed after training is done, to store learned weights and compute the weights of the parameters for the penalty term.
The usability of B E G IN is compared numerically with (Zhang et al., 2022) in Table 10.
B ENCHMARK R ESULTS
In this section, we provide benchmark results of seven graph continual learning methods based on the proposed scenarios and framework.
5. E XPERIMENTAL S ETTINGS
Machines.
We performed all experiments on a Linux server with Quadro RTX 8000 GPUs.
Models.
For all experiments, we used the Graph Convolutional Network (GCN) (Kipf & Welling, 2017), as the backbone model to compute node embeddings, and the Adam (Kingma & Ba, 2015) optimizer to train the model.
For NC, we used a fully-connected layer right after the backbone model to compute the final output.
For LC and LP, we additionally used a 3-layer MLP that receives a pair of node embeddings and outputs the final embeddings of the pair.
For GC, we used mean pooling to obtain the graph-level embedding from the output of the backbone model, and feed the computed embedding to a 3-layer MLP.
For NC and LC, we set the number of layers to 3 and the hidden dimension to 256.
For GC, we followed the settings in (Dwivedi et al., 2020) by setting the number of layers to 4 and the hidden dimension to 146.
Training Protocol.
We set the number of training epochs to 1 , 000 for the Cora , Citeseer , ogbnarxiv , and Bitcoin - OTC datasets, 200 for the ogbn - proteins , Wiki - CS , and ogbl - collab datasets, and 100 for ogbn - products and GC datasets.
For all datasets except for ogbn - products , we performed full-batch training. For ogbn - products , we trained GNNs with the neighborhood sampler provided by DGL for mini-batch training.
For all experiments, we used early stopping.
Specifically, for NC and LC, we reduced the learning rate by a factor of 10 , if the performance did not improve afterepochs, and stopped the experiment if the learning rate became 1 , 000 times smaller than the initial learning rate.
For Citeseer , we set the patience to 50 epochs and used the stopping criteria.
For LP and GC, we reduced the learning rate by a factor of 10 , if performance did not improve afterepochs, and stopped the experiment if the learning rate became 100 times ( 1 , 000 times in Wiki - CS ) smaller the initial learning rate.
For the hyperparmeters, we provided the details in Appendix B.
CL Methods.
Among general CL methods, we used (1) LwF (Li & Hoiem, 2017), (2) EWC (Kirkpatrick et al., 2017), (3) MAS
(Aljundi et al., 2018), and (4) GEM (Lopez-Paz & Ranzato, 2017).
For CL methods designed for graph-structured data, we used (5) TWP (Liu et al., 2021), (6) ERGNN (Zhou & Cao, 2021), and (7) CGNN (Wang et al., 2020a). Note that ERGNN and CGNN, which [Table] [Table]
N/A
N/A -0.
[Table] N/A N/A N/A
N/A
N/A N/A N/A were designed for node-level problems, were not applied to link- and graph-level problems. For the baseline methods without CL techniques, we used the Bare and Joint models used in (Zhang et al., 2022).
The Bare model follows the incremental learning schemes, but no CL technique is applied to the model.
Lastly, the Joint model trains the backbone model directly on the entire dataset, ignoring the CL procure with a sequence of tasks.
5. A VERAGE P ERFORMANCE & A VERAGE F ORGETTING
Our benchmark results in terms of final 3 AP and AF are shown in Table 3 and 4, respectively.
In Task-IL, the replay-based methods show poor results in general.
Specifically, the average rankings of GEM and ERGNN are 5 . 5 and 6 in terms of AP, and 6 and 6 in terms of AF, respectively.
We compute the average rankings over all problems considered for the results in the tables.
Interestingly, regularization-based methods are highly-ranked, outperforming CGNN, which combines replay- and regularization-based methods.
For node-level problems in Class-IL, most of the graph CL methods perform better than the Bare model, and CGNN achieves the best performance in both terms of both
AP and AF.
However, the graph-level problems in Class-IL are challenging.
We see that the most of graph CL methods show poor results in terms of both AP and AF.
In Domain-IL, the replay-based method (i.e., GEM) outperforms the regularization-based methods.
Specifically, on average, GEM is the best and the second-best among the five methods in terms of AP and AF, respectively.
In TimeIL, only LwF outperforms the Bare model on all datasets in terms of AP.
That is, it is challenging for current graph CL methods to deal with dynamic graphs evolving over time.
We also provide and discuss the benchmark results in terms of INT and FWT in Appendix A.2.
C ONCLUSION
In this work, we define four incremental settings for evaluating continual learning methods for graph data (graph CL) by identifying and decoupling four possible dimensions of changes, which are tasks, classes, domains, and time.
Then, we apply the settings to node-, link-, and graph-level learning problems, and as a result, we provide 23 benchmark scenarios from 14 real-world datasets, which cover 12 combinations of the four incremental settings and the three levels of problems.
In addition, we propose B E G IN , a fool-proof and easy-to-use benchmark framework for implementation and evaluation of graph CL methods.
We support numerically that our benchmark and framework are more extensive, rigorous, and usable than the previous ones for the same purpose.
For reproducibility, we provide all source code required for reproducing the benchmark results and documents at https://anonymous. .science/r
A F ULL E XPERIMENT R ESULTS A. A VERAGE P ERFORMANCE & A VERAGE F ORGETTING
In Table 5 and Table
[Table]
we report the results of average performance (AP) and average forgetting (AF) for all 23 benchmark scenarios
. A. I NTRANSIGENCE & F ORWARD T RANSFER
We report the results of Intransigence (INT) and Forward transfer (FWT) in Table 7 and 8, respectively.
In terms of INT, we can see that many methods are outperformed by the Bare model, which only focuses on learning the current task.
In terms of FWT, EWC performs best for NC and GC problems, and TWP performs best for LP problems.
Note that we are able to compute FWT only in Domain-IL since we cannot infer classes among C i after learning on T ( i − 1) in Task-IL and Class-IL, and we cannot get the i -th task (spec.,
G ( i ) and S ( i ) ) in advance in Time-IL.
B H YPERPARAMETER S ETTINGS
We performed a grid search to find the best hyperparameter settings for the backbone model (e.g., the learning rate and the dropout ratio) for each method.
Specifically, we chose the setting where AP on the validation dataset was maximized and reported the results on the test dataset in the selected setting.
For evaluation, we conducted ten experiments with different random seeds and averaged the performance over the ten trials.
For the CL methods except for TWP, we set the initial learning rate [Table] [Table]
[Table] among { 1e-3, 5e-3, 1e-3 } , the dropout ratio among { 0, 0.25, 0.5 } , and the weight decay coefficient among { 0, 5e-4 } . For TWP, we set the initial learning rate among { 1e-3, 5e-3 } , the dropout ratio among { 0, 0.25, 0.5 } , and the weight decay coefficient to 0. For the experiments on ogbn - proteins and ogbn - products , we fixed the learning rate to 1e-3.
For Citeseer , we additionally considered the learning rate 5e-4.
For ogbn - products , we set the dropout ratio to { 0 , 0 . 25 } , and we restricted the number of maximum neighbors to receive messages to 5 , 10 , and 10 on the first layer, the second layer, and the third layer, respectively, by using the sampler.
For replay-based methods (e.g., GEM, ERGNN, and CGNN), we set the maximum size of memory the same for a fair comparison.
Specifically, we set it to 12 for (a) all experiments on Cora and
Citeseer , (b) 210 for CoraFull , (c) 2 , 000 for ogbn - arxiv and ogbn - proteins , (d) 25 , 000 for ogbnproducts , (e) 500 for MNIST , CIFAR10 , and ogbg - molhiv , {+(f) 50 for Aromaticity , (g) 180 for NYCTaxi , (h) 4 , 
+}
for Wiki - CS , and (i) 20 , 000 for ogbl - collab .
Following the original papers, we set the margin for quadratic programming to 0 .
5 for GEM, and we used the Coverage Maximization (CM) sampler and set the distance threshold to 0 .
5 for ERGNN.
For regularization methods (e.g., LwF, EWC, MAS, TWP, and CGNN), we chose the regularization coefficient λ among { 0 . 1 , 1 . 0 } , { 100 , 10000 } , { 0 . 1 , 1 . 0 } , and { 80 } for LwF, EWC, MAS, and
CGNN, respectively.
For TWP, we set beta to 0 . 01 and λ l to 10 , 000 , and we chose λ t among { 100 , 1000 } .
Note that CGNN combines regularization- and replay-based approaches, and thus we need to consider both the maximum size of memory and the regularization coefficients. C E FFECTS OF THE NUMBER OF TOTAL TASKS
We conducted additional experiments to investigate the effect of the number of total tasks N on the performance of graph CL methods.
Accordingly, we changed the number of classes considered (additionally) in each task proportionally to the reciprocal of the number of tasks.
Under Taskand Class-IL settings on ogbn - arxiv , we measured how the performance changes depending on the [Table] [Table] [Figure]
[Table]
number of total tasks from 5 to 20 .
In Table 9, we report the benchmark final (i.e., when k is equal to N ) Performance in terms of AP, AF, and INT, respectively.
Under Task-IL, most graph CL methods perform better as the number of tasks increases in terms of all 3 metrics, due to the decrease in the number of tasks considered in each task.
On the contrary, under Class-IL, the performance tends to degrade in terms of AP and AF as the number of tasks increases since the distribution shift, which each model needs to adapt to, occurs more frequently.
Interestingly, we find that INT tends to change positively as the number of tasks increases because
INT does not consider the forgetting of previous tasks.
D P ERFORMANCE C URVE (E FFECTS OF THE NUMBER OF LEARNED TASKS )
In Figure 4, we report the performance curves, which show how the average performance (AP) (when k is equal to n ) changes depending on the number of learned tasks n .
Note that most considered methods, including the Joint model, tend to perform worse, as the number of learned tasks increases, since they need to retain more previous knowledge.
However, opposite trends are observed on ogbn - arxiv , ogbg - molhiv , and NYC - Taxi , which we guess have slighter or easier-to-adapt distribution shifts.
Also, on Bitcoin - OTC under Task-IL, the performance of the Joint model and that of graph CL methods show different tendencies.
We provide the per-
[Table]
For ERGNN, we only considered the code for classification with the
[Table]
formance curves under all considered scenarios in the online appendix, which is available at https://anonymous.4open.science/r/BeGin-1C33/ .
E U SABILITY OF B E G IN
In Table 10, we report the lines of Python code for the CL methods commonly implemented in
B E G IN and CGLB (Zhang et al., 2022).
For each CL method, we calculated the sum of the lines of the Python code for node classification and graph classification (if available) in Task-IL and Class-IL settings.
When counting the number of lines, we ignored comments, empty lines, and unnecessary imports in the codes.
It should be noticed that, with B E G IN , we can implement the same graph CL method with about 30% fewer lines of code, on average.
[Figure]
A v [Figure] r a g
[Figure] P [Figure] r f o r m a n c
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
LP
[Table]
[Figure]
A v e r a g e P e
r f o r m a n c e MNIST (Task-IL) (i)
MNIST (GC, Task-IL)
[Figure]
m
[Figure]
n
[Figure]
e
[Figure]
(j)
Aromaticity (GC, Class-IL)
[Figure]
[Table]
[Figure]
[Table]
[Figure]
(k) ogbg - molhiv (GC, Domain-IL)
[Figure]