From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to post-rebuttal response (1)
comment: We appreciate your response! 

Our responses are as follows:

>Q1-1. About the settings claimed as contributions of this work, the authors agree that they already exist. The authors then said that these settings have different definitions in different works and different settings are often mixed. However, besides this vague statement, no explanation was given on how the settings are different or mixed in any existing work. No explanation was given on how the settings in this paper is better either, and the definition in this paper seems no different from other works. Since these settings are commonly adopted, I don't think the statement saying that existing works have problematic settings is convincing.
>>A1-1. Our response is three-fold:
>>- For example, in [1], the change in the tasks and the change in the input graph are coupled in a specific way, when they can be independent without necessarily being coupled.   
>>- We agree that the proposed settings are partially considered in previous stuides. Please note that we would like to claim that our scenarios are more extensive and exhaustive rather than claiming that all previous ones are problematic.
>>- We will clarify how our scenarios are related to previously considered ones in the final version of the paper.

[1] Zhang, Xikun, Dongjin Song, and Dacheng Tao. "CGLB: Benchmark Tasks for Continual Graph Learning." Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022.


>Q1-2. About the seperation of the evaluation and training parts, the explanation is mostly simply repeating the sentences in the paper, and some part is still confusing. For example, what are the dependency among the tasks and the information leak? What is the relation between this information leak and the separation of the evaluation module? What the 'evaluation module' refers to becomes confusing now, I though it is only about the implementation details, but now it seems to be related to the experimental settings (the information leak issue).
>>A1-2. Our response is three-fold:
>>- For example, for a node classification task, not only instances (i.e., nodes) assigned to the current task but also those assigned to previous or future tasks are used (e.g., for graph convolution) because they are connected by edges (this is an example of dependency between tasks). Due to this fact, if users are not careful enough, it is easy for some information that should be unavailable during the current task (e.g., labels of the nodes assigned to previous or future tasks) to be unintentionally leaked.
>>- In order to prevent such mistakes of users, we do not provide the test label information to users at all by separating user code and the evaluation module, where the test labels are used. In addition, our framework restricts the information given for each task to prevent potential information leak.
>>- Note that our design objective is to reduce potential mistakes (i.e., unintentional information leak) but not to prevent adversarial users who cheat by any means.


>Q1-3. About the differential between CL and graph CL, I could agree that the graph CL methods may consider some graph related issues like the connections between nodes. However, from the experimental results (e.g. Table 3,4) the best performance are mostly obtained by the non-graph CL methods, therefore there seems no unique challenge on graph CL and I still think the application of CL to graph CL is straightforward and the differention is unclear (i.e. why studyinf this problem is important).
>>A1-3. Our response is three-fold:
>>- The recent “explosion” of the studies of graph learning proves that utilizing graph topological information (in addition to feature information) is critical and at least beneficial in many problems and applications, and we believe that CL will be one of them.
>>- In our opinion, the results can be interpreted in the oppositie direction. Graph CL (i.e., utilizing graph topological information for CL) is challenging, and that is why existing graph CL methods have a large room for improvement (even being underperformed by CL methods not specialized to graphs). We believe our proposed benchmark scenarios and frameworks will help speed up the improvement.
>>- In the introduction section of the revised paper, we discussed some challenges and importance of graph CL.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to post-rebuttal response (2)
comment: >Q2-1. About the extremely small number of tasks. The authors agree with this and included a new dataset with 35 tasks. This is much better, but for a benchmark paper, this is far from enough because most of the other datasets are 3 or 5 tasks, which means the majority datasets used in this paper are not qualified.
>>A2-1. Our response is two-fold:
>>- In addition to CoraFull dataset (w. 35 tasks), we set the number of tasks to 20 in the ogbn-arxiv dataset, and the results can be found in Table 9 of the revised paper.
>>- Downward trends in the performance curves in Figure 4 of Appendix D show that most considered CL methods suffer from catastrophic forgetting, at least to some degree, in our evaluation scenarios. This shows that our evaluation scenarios perform as intended.
>>- We are using classes disjointly in tasks, and that is a reason why the number of tasks can be limited. In the final version of the paper, we will include the results when we increase the number of tasks by using classes jointly across tasks.


>Q2-2. About the time-IL setting. The explanaton on this point is not convincing. First, splitting the graph in chronological order is realistic but it may not be meaningful for continual learning setting, since the distribution shift is not guaranteed. Second, low performance may not result from the distribution shift. Since the AF is also low, the low performance actually is caused by the difficulty of each single task. In other words, if a model cannot perform well on each task, even if the distribution is same across all tasks, the performance is still low.
>>A2-2. Our response is two-fold:
>>- According to the performance curves in Figure 4 of Appendix D, distribution shifts in at least some datasets (e.g., ogbl-collab) under Time-IL are large enough to cause catastropic forgetting, at least to some degree. 
>>- We believe that an ideal CL method should perform well under various degrees of distributional shifts, and it can be meaningful to test CL methods not only under substantial distributional shifts but also under mild distributional shifts, especially when actual distributional shifts in real-world datasets are mild.

From: ICLR.cc/2023/Conference/Paper5922/Reviewer_TvQ3
title: post-rebuttal response
comment: Thanks for the detailed responses from the authors. I have checked every point carefully. Overall, some of the concerns are resolved and revisions are made to the paper, but many major concerns are not resolved. The authors also acknowledge the existence of these problems. Let me breifly summarize the major unresolved points.

1. Questionable contributions. 

About the settings claimed as contributions of this work, the authors agree that they already exist. The authors then said that these settings have different definitions in different works and different settings are often mixed. However, besides this vague statement, no explanation was given on how the settings are different or mixed in any existing work. No explanation was given on how the settings in this paper is better either, and the definition in this paper seems no different from other works. Since these settings are commonly adopted, I don't think the statement saying that existing works have problematic settings is convincing. 

About the seperation of the evaluation and training parts, the explanation is mostly simply repeating the sentences in the paper, and some part is still confusing. For example, what are the dependency among the tasks and the information leak? What is the relation between this information leak and the separation of the evaluation module? What the 'evaluation module' refers to becomes confusing now, I though it is only about the implementation details, but now it seems to be related to the experimental settings (the information leak issue).

About the differential between CL and graph CL, I could agree that the graph CL methods may consider some graph related issues like the connections between nodes. However, from the experimental results (e.g. Table 3,4) the best performance are mostly obtained by the non-graph CL methods, therefore there seems no unique challenge on graph CL and I still think the application of CL to graph CL is straightforward and the differention is unclear (i.e. why studyinf this problem is important). 

2. Problems on the experimental settings. 

About the extremely small number of tasks. The authors agree with this and included a new dataset with 35 tasks. This is much better, but for a benchmark paper, this is far from enough because most of the other datasets are 3 or 5 tasks, which means the majority datasets used in this paper are not qualified.

About the time-IL setting. The explanaton on this point is not convincing. First, splitting the graph in chronological order is realistic but it may not be meaningful for continual learning setting, since the distribution shift is not guaranteed. Second, low performance may not result from the distribution shift. Since the AF is also low, the low performance actually is caused by the difficulty of each single task. In other words, if a model cannot perform well on each task, even if the distribution is same across all tasks, the performance is still low.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to TvQ3 (4) 
comment: We added the results of performance changes in Section D of the Appendix in the revised paper.
Also, we uploaded full results of performance changes via Anonymous Github.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to Reviewer ycgz (3) 
comment: We added results and discussions according to the number of total tasks in Section C and to the number of learned tasks in Section D of the Appendix in the revised paper.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Common Response (2)
comment: We re-uploaded the revised paper containing the materials below:
- Link prediction results (Wiki-CS, ogbl-collab)(Section 5)
- Increasing the number of tasks (CoraFull dataset with 35 tasks) (Section A of Appendix)
- Real-world graph classification (Aromaticity) (Section A of Appendix)
- More detail about hyperparameter settings (Section B of Appendix)
- Effects of the number of total tasks (Section C of Appendix)
- Performance curve (effects of the number of learned tasks) (Section D of Appendix)

In the revised paper, we colored red the parts that we newly added or significantly changed during the rebuttal period.

From: ICLR.cc/2023/Conference/Paper5922/Reviewer_rhFV
title: Post Rebuttal Comment
comment: Dear authors,

Thank you for your response. After reading your response and the rest of the reviews, I maintain my rating. I think this is a good research direction but it needs significantly more effort to rise to a contribution to a top-tier conference. Simply proposing an evaluation protocol is not a significant contribution.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to rhFV
comment: We appreciate your helpful suggestion.

>W1. “The settings are contrived without much connection to reality.”

>W2. “Algorithmic novelty of the paper is incremental.”

>W3 (in Clarity, Quality, Novelty And Reproducibility). “However, the paper is not novel in terms of introducing an algorithm. For ICLR, algorithmic novelty is more important and hence, I think the paper is not novel. Code for the experiments is provided and seems well documented.”

>W4 (in Summary). “This is a benchmark dataset without much algorithmic novelty. The paper is not introducing a dataset based on new measurements, rather it builds the benchmark using existing datasets. However, The datasets that are used in this work lack a real notion of time dependence, and the tasks are generated somewhat synthetically. I was wondering why datasets such as GDELT are not used that have a natural temporal axis that can be used to define tasks that are naturally dependent. I think this paper is more suitable for NeurIPS Dataset and Benchmarks track because it is not a strong paper in terms of the code proposed idea.”

>A.Our response is four-fold:
>>- We admit that not all our proposed settings have strong connection to reality. However, we would like to emphasize that the limitation is not unique in the graph domain. We do not think that our benchmark settings are less realistic than most of the widely-used continual benchmark datasets in other domains, most of which are also not based a real notion of time dependence. Despite the limitation, existing benchmark datasets in other domains are useful enough to evaluate CL methods, and thus they are playing a key role in the rapid advancement of continual learning. We believe ours will play a similarly important role in the advancement of graph continual learning.
>>- Moreover, for all scenarios in time-IL settings in the paper, we used the datasets with a natural temporal axis. For example, the ogbn-arxiv and ogbl-collab datasets contain publication years. In the bitcoin-otc dataset, a timestamp of rating is attached to every edge. Lastly, we used the NYC-Taxi dataset and formulated the time-IL setting so that transportation data collected in different months are used for different tasks.
>>- We were unaware of the GDELT dataset, which can be used for a time-IL LP scenario. We will include the dataset for future releases of our benchmark. Thanks for the suggestion.
>>- Thanks for the suggestion regarding submission tracks. We agree that the suggested track can be a better fit. However, when considering the growing interest in graph continual learning, our benchmark settings and our framework are pivotal, we thought it may be better not to wait until the next NeurIPS deadline. In addition, according to the submission system, ICLR has an area named “Infrastructure (eg, datasets, competitions, implementations, libraries)”, which our work clearly falls into.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to TvQ3 (1)
comment: Thank you for sharing your expertise.

Our responses are as follows:

**I. Questionable contributions**

>Q1-1-1. "The first is defining the four CL settings. However, three of them (task-IL, class-IL,domain-IL) already exist and are widely adopted in almost all CL works [1,2]. Therefore, this is simply adopting existing settings and is not a contribution. As for the time-IL setting, it is conceptually problematic, which is also reflected in experiments (this will be detailed below in the problems on the experimental settings)."
>>A1-1-1. As the reviewer said, Task-, Class-, and Domain-IL already exist in continual learning for independent data, and the settings have been considered for graph CL. However, their definitions vary in studies, where different settings are often mixed with no good reason. Thus, it is difficult to directly compare different methods, and this has hindered the advancement of graph CL. Therefore, to address these difficulties, we formally define and systematically organize the incremental settings for graph learning problems of three levels, and provide real-world benchmark datasets for each of the above settings. We believe that our work will be valuable for advancement of GCL, as many benchmark datasets from various domains have done.

>Q1-2-1. The first one is that the proposed framework is fool-proof because it separates the evaluation part and training part and can 'eliminate potential mistakes in evaluation'. But what are the potential mistakes in evaluation? This requires further clarification. As far as I know, this problem was never mentioned in any existing CL work. If this is a unique problem in graph CL, please add more explanations on it. 
>>A1-2-1. Evaluation is especially complicated for graph CL due to additional dependency (e.g., edges) between instances (nodes). Specifically, instances of one task are often used also for other tasks to exploit the dependency, and thus without careful consideration, information that should not be shared among tasks can be leaked among tasks. In order to eliminate potential mistakes in evaluation, our framework completely separates the evaluation module from the learning part, where users implement their own graph CL methods. The learning part only has to answer queries provided by the evaluation module after each task is processed. We added more explanations about fool-proofness in the introduction section of the revised paper.

>Q1-2-2. The second part is that the proposed framework is easy-to-use, which should be an important point for a benchmark. However, in the current version, the code are unfortunately incomplete and not well documented. This problem will be detailed later.
>>A1-2-2. We are sorry for the inconvenience. On the Anonymous Github repo, we uploaded the README file, a tutorial, extra documents, and source code with detailed comments. 

>Q1-3-1. The first is the benchmark results of seven graph CL methods. However, at least four benchmarked methods (LwF,EWC,MAS,GEM) are not graph CL methods but are methods for independent data. Since their implementations are publicly available, testing them on graph data can hardly be a contribution. Moreover, it seems that applying the CL methods for independent data to graph data is straightforward, making the diffrentiation between the graph CL and CL on independent data very unclear. 
>>A1-3-1. Our response is two-fold:
>>- Three benchmarked methods (ERGNN, CGNN, and TWP) are designed specifically for CL on graph data. The difference between such graph CL methods and CL on independent data is that graph CL methods (e.g., ERGNN, CGNN, and TWP) consider the dependency among instances. For instance, for node classification, graph CL methods consider not only features of a node but also its connections to other nodes, as described in detail in Section 2.
>>- We would like to emphasize that we considered EWC, MAS, GEM, and LwF as baseline methods against which users compare their own methods. In addition, due to their simplicity, they are used to give a tutorial of our framework.

>Q1-3-2. Second, the validation step is claimed as a contribution, because it is absent from previous benchmarks, which seems not true after checking the two previous graph CL benchmarks compared in the paper. 
>>A1-3-2. The validation step was not provided by CGLB at the time of submission. It seems that the validation step was later added. We excluded it from our contribution. Thanks for pointing this out.

>Q1-3-3. Finally, reporting four metrics is claimed as a contribution. But the adopted metrics are insufficient, because all of them are something averaged over all tasks. Continual learning works require the demonstration of the performance change. This will be detailed later.
>>Q1-3-3. Please refer to A2-3.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to TvQ3 (2)
comment: >Q1-4. Besides, using fewer lines of code is also strengthened as an advantage. However, fewer lines of code does not necessarily means higher efficiency or better readability, therefore this point is not a convincing advantage.
>>A1-4. We admit your concern. We excluded the comparison of lines of code both from the main paper and our contribution. Instead, let us elaborate on why our framework can be easier to use than CGLB (Zhang et al., 2022). CGLB simply provides the template based on which users should directly implement the procedure, while ours provides a modularized event function. With our framework, users can implement their desired CL algorithm while concerning relatively less about controlling the overall training procedure. For example, in order to implement a procedure before training on the current epoch, in CGLB, users have to manually check whether the current minibatch is the first minibatch of the current task. However, when using our framework, users do not need to consider it; they only need to implement the 'processBeforeTraining' event function.

**II. Problems on the experiment settings**

>Q2-1. A common problem for all scenarios is that the extremely small number of tasks, which is far from enough for continual learning. Most scenarios have only 3 or 5 tasks to learn. The scenario with most tasks only contains 12 tasks. Continual learning works typically considers at least tens of tasks [2,3,4,5], and some even consider thousands of tasks, e.g. [3].
>>A2-1. We appreciate the references. Our respond is three-fold:
>>- We are performing additional experiments with more tasks, and the results will be added to the paper shortly. Specifically, we will add the results of Cora-Full (w. # of tasks = 35) in Section 5 of the revised paper. 
>>- We would like to emphasize that it is not trivial to increase the number of tasks for real-world graph datasets, without artificial manipulation of the datasets. One major reason is that the numbers of classes in real-world graph datasets (spec., up to 40 in the considered datasets) tend to be much smaller than those in image datasets (e.g., CIFAR-100 with 100 classes, iILSVCRC2012 with 1000 classes) [2,5]. 
>>- Due to this difficulty, the numbers of tasks considered in previous studies of graph continual learning (ERGNN, CGNN, CGLB) are also similar to those considered in our work.

>Q2-2. Among the four graph prediction datasets, two (MNIST and CIFAR10) are image datasets. Is there any good reason to use these 'pseudo graphs' instead of the real graph datasets? For graph prediction, both the task- and class-IL settings are only based on the image datasets, making the benchmark results not convincing.
>>A2-2. We admit your concern. We are performing additional experiments with real graph datasets, and the results will be added to the paper shortly. Specifically, we will add the results on a real-world graph classification dataset "Aromaticity" under Task- and Class-IL in Section 5 of the revised paper.

>Q2-3. The four adopted evaluation metrics use single values as the average performance over all tasks. However, the dynamics of the performance (how performance changes after every task in the form of performance curve) is very important for CL, and are demonstrated in almost all existing CL works [2,3,4,5], including the other two graph CL benchmarks mentioned by the authors.
>>A2-3. We admit your concern. We are preparing plots that show the performance changes, and they will be added to the paper shortly, Specifically, we will provide the performance chagnes under all settings as an online supplementary material and report some representative ones in the appendix of the revised paper.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to TvQ3 (3)
comment: >Q2-4. The time-IL setting is not reasonable and potentially misleading. As mentioned by the authors, a key point in designing the tasks is 'the forgetting problem should happen without careful consideration'. In other words, the distribution of different tasks should be sufficiently different. However, in the proposed time-IL, there is no evidence that the adopted temporal splitting ensure a sufficient distribution shift. On the contrary, according to the experiments, the forgetting (AF) of time-IL is much lower than the other settings, which denotes that the tasks in time-IL indeed are not different enough for continual learning.
>>A2-4. Our response is two-fold:
>>- We think that Time-IL provides the most realistic distribution shift because we splited a graph in chronological order based on actual real timestamps. 
>>- We think that still many Time-IL settings provide enough distribution shifts for models to overcome. Please kindly note that the performance under Time-IL was worse than the performances under Task-IL for Link Classification in Table 5. Also, for Graph Classification, AF of LwF under Time-IL is similar to AF in LwF under Task-IL. Moreover, in most Time-IL settings, the performance gap between the best method and the others tends to be significant (e.g., see the results on NYC-Taxi), indicating that most methods fail to address the distribution shifts well under Time-IL settings.

>Q2-5. The hyper-parameter setting is problematic. It is stated that grid search is used to find the best hyperparameters. However, for CGNN, GEM, and ERGNN, only one candidate hyperparameter is used, isn't it weired to perform grid search over only one choice? For other methods, only two choices are used for grid search. Besides, for the replay-based methods (GEM, ERGNN,CGNN), the size of memory is arbitrarily given for different datasets without explanations.
>>A2-5. Thanks for pointing this out. We performed a grid search to find the best hyperparameter settings for the backbone model (e.g., the learning rate and the dropout ratio) for each method. We set hyperparameters specific for some methods (e.g., margin in GEM and the distance threshold in ERGNN) to the value reported by the authors of the paper. In all replay-based methods (GEM, ERGNN, CGNN), we set the number of replayed nodes the same for a fair comparison. We clarified this fact in Appendix B of the revised paper.

>Q2-6. According to the authors, link prediction is a straightforward extension from node prediction, then why is ERGNN applicable to Node classification but not to link prediction
>>A2-6. We replied assuming that you meant link classification, not link prediction. We meant that the problem definition of link classification is straightforwardly extended from that of node classification, and we did not mean that individual algorithms are. For example, ERGNN is one of the graph CL methods for node classification, and all its subprocesses are implemented based on nodes, including those that are not easily extended to the link level. This is the same for CGNNs. We added an explanation to “Models” of Sections 3.1 and 5.1 in the revised paper.

**III. Problems with the provided code and reproducibility**

>Q. “As a benchmark paper, the code repository should be well maintained and easy to use. Providing easy-to-use code is also claimed as a contribution of this paper. However, after checking the paper, supplementary materials, and the code repository, I found no instruction given on using the code. When trying to understand the code with the demo.py, I found that the folder 'src.benchmarks.cora' used in the demo does not even exist. Moreover, the joint baseline seems not included in the code. I didn't go further to check other problems, but it is obvious that the current code is not qualified. Besides, the code is not well commented, most of the code have no comment at all, making is hard to understand.”
>>A. We are sorry for the inconvenience. On the Anonymous Github repo, we uploaded the README file, a tutorial, extra documents, and source code with detailed comments, after fixing the error about 'src.benchmarks.cora'.. You can get the result of the joint model if you give “True” to “full_mode” when you initialize a trainer instance (e.g., NCClassILBareTrainer). Thank you for your constructive comments.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to QtYb
comment: We genuinely appreciate your constructive feedback!

Our responses are as follows:

>Q1. “The paper lacks the discussion about the real-world applications and significance of continual learning on graphs. Rationality behind the scenario designs is in need.”
>>A1. Many graph data in the real world grow in size and diversity while being applied to more and more domains. Thus, continual learning is naturally beneficial to graph data, as pointed out many previous studies of continual graph learning. One of such real-world graphs is a citation network (e.g., Cora and Citeseer), which grows with newly published papers (Time-IL), shifts in feature distributions (Time-IL), and new fields of studies (Class-IL). Especially, since real-world graphs (e.g., online social networks) are massive and changing ceaselessly, it is computationally inefficient to train a new model per task or change, and CL is a promising remedy Let us explain briefly the significance of continual learning on graphs. We discussed the significance of continual graph learning with extra references in the introduction section of the revised paper.

>Q2. “The paper also lacks more discussions on other areas related to continual learning on graphs. For example, there are some recent studies focusing on distribution shifts on graphs [1, 2, 3], e.g., graph data may contain the pieces of different domains/distributions, which are highly related to the continual learning setting. Besides, some graph-based extrapolation problems also intersect with the incremental learning paradigm, e.g., feature-space incremental learning [4], entity incremental learning on knowledge graphs [5]. I believe more discussions beyond the scope of the specific setting in the current paper would definitely help to increase the impact.”
>>A2. Thank you for the interesting connections. We would like to emphasize that goal of graph continual learning and those of [1-5] are not exactly the same. Specifically, in CL problems, distribution shifts may occur over tasks, and such shifts can be observed from training data for each task. However, [1-5] deal with shifts between training and test distributions that are unobservable during training. Specifically, [1-3] aims to generalize models handling unobservable distribution shifts, [4] proposes the new training paradigm for overcoming the incremental features, and [5] solves a few-shot out-of-graph link prediction they introduced by meta-learning. We briefly discussed the works and their difference from ours in Section 2 of the revised paper.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to Reviewer ycgz (1) 
comment: We appreciate your helpful comments.

Our responses are as follows:

>Q1. “One of the major issues is that it is not clear what’s the unique challenges for continual graph learning and how the proposed benchmark contribute to advancing the direction.”
>>A1. As stated in the abstract and introduction sections, unique challenges for continual graph learning that we identify are (a) the lack of standard experimental settings, especially regarding how to deal with the dependency between instances, (b) the lack of benchmark datasets and scenarios, and (c) high complexity in implementation and evaluation due to the dependency. Our proposed benchmark, BeGin, contributes to continual graph learning by (a) being first at formulating and systematically organizing four incremental settings (Task-, Class-, Domain-, and Time-) for graph learning problems of three levels, (b) providing real-world benchmark datasets for each of the above settings, and (c) making the implementation and evaluation of (new) graph CL methods easy. 

>Q2. “Another major concern is that for link prediction/classification tasks, the existing related works are neither discussed nor compared. For instance, traditional degree or triad-based features have shown good performance for domain adaptation/transfer learning. In addition, many network embedding works or GNN-based models have explored this direction, but none of them are mentioned.”
>>A2. Thanks for the suggestion. We agree that transfer learning on graph data (e.g., those for link prediction) is related to our work, and it is briefly discussed in Section 2 of the revised paper. However, we would like to emphasize that our work focuses on continual learning, which aims to train a single model for a sequence of tasks without catastrophic forgetting. Its goal is different from that of domain adaptation or transfer learning, where a separate model is trained or adapted for a new domain. 

>Q3. “Similar problem exists for dynamic graphs, and related works (spatial-temporal modeling) in this direction are not discussed or compared.”
>>A3. Thanks for the suggestion. We briefly discuss dynamic graph learning in Section 2 of the revised paper. As you pointed out, in each timestamp, the format of input (snapshot of dynamic graph) is the same on dynamic graph learning (DGL) and graph CL under Time-IL setting. However, we would like to emphasize that graph CL aims not only to learn well on the current snapshot but also to preserve performance on previous snapshots (i.e., avoiding catastrophic forgetting), while most studies of DGL only focus on the current snapshot. Preserving performance on previous snapshots can be especially useful when seasonality is present, as in the NYC-Taxi dataset. 

>Q4. “Based on my understanding, the proposed domain incremental setting can be treated as a special case of class incremental setting.”
>>A4. Sorry for the confusion. In Domain-IL, as the domains of entities (node, link, and graph), we use input information not directly related to the labels, which we aim to infer. However, in Class-IL, we split entities into groups by the labels of the entities. We clarified this difference in “Domain-Incremental” of Section 3.2 of the revised paper.

>Q5. “The paper provides two more metrics to evaluate continual graph learning, however, the authors fail to explain why those two metrics are necessary and what extra info they can bring.”
>>A5. Given a sequence of tasks, continual learning (CL) aims to (1) learn the current task well while (2) retaining knowledge from previous tasks (i.e., knowledge consolidation) and (3) overcoming the degradation of performance (i.e., catastrophic forgetting). In this perspective, AP and AF measure (3),  INT does (1), and FWT does (2). We added an explanation of each metric in Section 4.2 of the revised paper.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to Reviewer ycgz (2) 
comment: >Q6.”As a benchmark, in the code, the readme file does not provide any useful information (only dependencies are useful) for utilizing the benchmark. I roughly check the code but no comment is provided…”
>>A6. We are sorry for the inconvenience. We added a README file, a Tutorial, and additional documents on the Anonymous Github repo. Also, we added comments to our code. Thank you for your constructive comments.

>Q7. “Lines of code are not an effective criterion to compare different benchmarks. The implementation/running efficiency actually matters.”
>>A7. We admit your concern. We excluded the comparison of lines of code both from the main paper and our contribution. Instead, let us elaborate on why our framework can be easier to use than CGLB (Zhang et al., 2022). CGLB simply provides the template based on which users should directly implement the procedure, while ours provides a modularized event function. With our framework, users can implement their desired CL algorithm while concerning relatively less about controlling the overall training procedure. For example, in order to implement a procedure before training on the current epoch, in CGLB, users have to manually check whether the current minibatch is the first minibatch of the current task. However, when using our framework, users do not need to consider it; they only need to implement the 'processBeforeTraining' event function.

>Q8. “As a benchmark, the datasets used in the paper are all relatively small. 4 graphs used in the experiment only have thousands of nodes and the largest dataset only have around 160K nodes. In addition, the number of tasks is also relatively small and the detailed partition is not discussed or provided.”
>>A8. Thanks for your constructive review. We are performing additional experiments with larger datasets, and the results will be added to the paper shortly. Specifically, we will add the results on OGBN-Products (w. 2,449,029 nodes and 61,859,140 edges) and Cora-Full (w. # of tasks = 35) in Section 5 of the revised paper. Also, we will add the result and the discussion according to the number of tasks in Section E of the Appendix in the revised paper.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Response to Reviewer 4C6j
comment: Thank you for your thoughtful comments!

Our responses are as follows:

> Q1. "The presentation of source code needs to be improved significantly.
The first line in demo.sh cannot be run because cora directory does not exist.
When the third line of LwF for GC is run, an error is raised as No such file or directory: 'data/ogbg-molhiv_metadata_domainIL.pkl'. Is there any issue of file paths in the submitted implementation?
The library on Anonymous Github cannot be downloaded as a whole, which needs reviewers to download each file separately and then rebuild the file paths from scratch."
>> A1. We are sorry for the inconvenience. We fixed the typos that cause the mentioned error. We uploaded a zipped file On the Anonymous Github repository so that you can download the library as a whole. 

> Q2. “There is some ambiguity regarding link prediction and link classification. If I understand correctly, ''Link'' in Table 1 stands for link classification. But in Section 3.1, a lot more contexts are about link prediction, which turns out not included in BeGin. Is there any reason for not including LP?”
>>A2. Our response is two-fold:
>> - “Link” in Table 1 denotes the “link”-level problems, which include link classification (LC). As stated in Section 3.1, for link-level problems, we deal with not only link prediction (LP) but also LC. LC is extended straightforwardly from the node classification problem.
>> - In addition, we performed an additional experiment on LP in time-IL and domain-IL settings and updated the results in Table 3, 4 and Appendix A of the revised paper.

> Q3. “A small concern is that the slight difference between NC and LC needs to clarified in ''Models'' paragraph of Section 5.1, where now it ignores NC.”
>> A3. We provided additional details about the difference in Section 5.1 of the revised paper. Thank you for pointing it out.

> Q4. “Could you please reveal why BeGin is simpler than Zhang et al. in the context?”
>> A4. As our framework does, CGLB (Zhang et al., 2022) provides functions that can be used to implement new CL algorithms. CGLB simply provides the template based on which users should directly implement the procedure, while ours provides a modularized event function. With our framework, users can implement their desired CL algorithm while concerning relatively less about controlling the overall training procedure. For example, in order to implement a procedure before training on the current epoch, in CGLB, users have to manually check whether the current minibatch is the first minibatch of the current task. However, when using our framework, users do not need to consider it; they only need to implement the 'processBeforeTraining' event function.

From: ICLR.cc/2023/Conference/Paper5922/Authors
title: Our Common Response
comment: We thank the reviewers for their constructive and helpful comments. We will continue to revise the paper during the rebuttal period. In the revised paper, we colored red the parts that we newly added or significantly changed during the rebuttal period.

From: ICLR.cc/2023/Conference/Paper5922/Reviewer_4C6j
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposes a new benchmark for continual learning on graphs with a grid of different incremental settings and different task levels. Compared with previous works, it extends the settings to domain- and time-incremental, as well link-level tasks. It presents an introduction to elements in the framework with a slighter workload for users' programming. It also shows empirical results of current graph continual learning methods in the framework as baselines for future use.
strength_and_weaknesses: Strengths:
1. It is a natural motivation to extend settings and task levels of graph continual learning. The achieved extension against previous works is significant as shown in Table 1.
2. The workload for users' programming is better 

Concerns:
1. The presentation of source code needs to be improved significantly.
    * The first line in `demo.sh` cannot be run because cora directory does not exist.
    * When the third line of LwF for GC is run, an error is raised as `No such file or directory: 'data/ogbg-molhiv_metadata_domainIL.pkl'`. Is there any issue of file paths in the submitted implementation?
    * The library on Anonymous Github cannot be downloaded as a whole, which needs reviewers to download each file separately and then rebuild the file paths from scratch.
2. There is some ambiguity regarding link prediction and link classification. If I understand correctly, ''Link'' in Table 1 stands for link classification. But in Section 3.1, a lot more contexts are about link prediction, which turns out not included in BeGin. Is there any reason for not including LP?
3. A small concern is that the slight difference between NC and LC needs to clarified in ''Models'' paragraph of Section 5.1, where now it ignores NC.
4. Could you please reveal why BeGin is simpler than Zhang et al. in the context?
clarity,_quality,_novelty_and_reproducibility: The novelty is good, with significant extensions compared with previous works.

The clarify, quality and reproducibility need improvement, especially the reproducibility as discussed above.
summary_of_the_review: I would like to recommend below threshold due to issues of presentation and reproducibility. I am willing to raise my score if the above concerns are resolved.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
Yes, Discrimination / bias / fairness concerns
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5922/Reviewer_ycgz
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents a benchmark for continual graph learning. The authors discussed 4 settings over 3 different problems. From a technical perspective, the novelty is limited since similar continual graph learning benchmarks are available. In addition, several settings discussed in the paper are problematic or even misleading.
strength_and_weaknesses: Strengths:

1. Continual graph learning is an interesting problem to investigate.

2. 4 different settings are discussed over 3 different problems.

Weaknesses:

1. One of the major issues is that it is not clear what’s the unique challenges for continual graph learning and how the proposed benchmark contribute to advancing the direction. 

2. Another major concern is that for link prediction/classification tasks, the existing related works are neither discussed nor compared. For instance, traditional degree or triad-based features have shown good performance for domain adaptation/transfer learning. In addition, many network embedding works or GNN-based models have explored this direction, but none of them are mentioned. 

3. Similar problem exists for dynamic graphs, and related works (spatial-temporal modeling) in this direction are not discussed or compared.

4. Based on my understanding, the proposed domain incremental setting can be treated as a special case of class incremental setting. 

5. The paper provides two more metrics to evaluate continual graph learning, however, the authors fail to explain why those two metrics are necessary and what extra info they can bring.

6.  As a benchmark, in the code, the readme file does not provide any useful information (only dependencies are useful) for utilizing the benchmark. I roughly check the code but no comment is provided…

7.  Lines of code are not an effective criterion to compare different benchmarks. The implementation/running efficiency actually matters.

8. As a benchmark, the datasets used in the paper are all relatively small.  4 graphs used in the experiment only have thousands of nodes and the largest dataset only have around 160K nodes. In addition, the number of tasks is also relatively small and the detailed partition is not discussed or provided.

clarity,_quality,_novelty_and_reproducibility: Certain settings in the paper are not clear.
Technical novelty is limited. No new algorithm is presented.
Code is provided but the documentation is limited.
summary_of_the_review: Overall, I feel this paper is problematic or even misleading in many aspects. Therefore, I cannot recommend accepting it with the current status.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: No
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper5922/Reviewer_QtYb
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a benchmark on graph continual learning with three tasks and four incremental settings. The framework is comprehensive and easy-to-use. Authors are recommended to discuss more about the signifiance of this problem to the community, instead of combining the two areas of research directly.


strength_and_weaknesses: Pros:

The paper propose a continual learning framework that includes multiple tasks( node-, link-, and graph-level tasks) and multiple incremental scenarios. The framework is easy and foolproof, demonstrating a more comprehensive settings and fewer lines of code for implementation.

Cons:

1. The paper lacks the discussion about the real-world applications and significance of continual learning on graphs. Rationality behind the scenario designs is in need.

2. The paper also lacks more discussions on other areas related to continual learning on graphs. For example, there are some recent studies focusing on distribution shifts on graphs [1, 2, 3], e.g., graph data may contain the pieces of different domains/distributions, which are highly related to the continual learning setting. Besides, some graph-based extrapolation problems also intersect with the incremental learning paradigm, e.g., feature-space incremental learning [4], entity incremental learning on knowledge graphs [5]. I believe more discussions beyond the scope of the specific setting in the current paper would definitely help to increase the impact.

[1] Handling Distribution Shifts on Graphs: An Invariance Perspective, ICLR22

[2] OOD-GNN: Out-of-Distribution Generalized Graph Neural Network, TKDE21

[3] GOOD: A Graph Out-of-Distribution Benchmark, NeurIPS22

[4] Towards open-world feature extrapolation: an inductive graph learning approach, NeurIPS21

[5] Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Prediction, NeurIPS20
clarity,_quality,_novelty_and_reproducibility: Clarity
The paper is in general well-organized and easy to follow.

Quality:
The paper is self-contained and well-organized. The overall quality is high.

Novelty:
The paper proposes multiple continual learning scenarios that fit with the characteristics of the graph dataset. However, authors are encouraged to provide more rationality behind these scenario designs.

Reproducibility
The paper provides the code to reproduce the benchmark.
summary_of_the_review: Overall, I think this paper is well motivated and clearly written. I like the scope of this paper and appreciate the empirical contributions. However, I think more discussions on the significance and more justification for the specific designs/protocol are needed to strenghthen the paper. Furthermore, more discussions on the related areas could help to increase the impact of the paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5922/Reviewer_TvQ3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Targeting that existing CL (continual learning) methods mostly deal with independent data like images and text, this paper proposes a benchmark framework for CL methods on graph data, and reports the benchmark results of several CL methods with different settings. 

In the proposed framework, nine public datasets are divided into multiple tasks in different ways for different continual learning settings (i.e. task-,class-,domain-,time-incremental settings). Then, different CL methods are tested with these settings. For evaluation, four metrics (average performance, average forgetting, intransigence, and forward transfer) are adopted. A code repository of the implemented framework is also provided.

strength_and_weaknesses: **Strengths**

This paper has a clear logic on what they propose and is easy to follow. 

The literature review looks comprehensive. The related works cover both continual learning works for independent data and graph data. Two existing similar benchmark works are also mentioned and compared.

**Weakness**

A benchmark should be very carefully built in terms of the experimental settings, evaluatioon metrics, benchmark results, etc., because it may be the basis of many future works. Also, it should have clear contribution and differentiation compared to existing works. However, after carefully cheking this paper, its quality is not satisfying. The weakness concerns multiple aspects as listed below. 

**I. Questionable contributions**

This will be explained with repesct to each contribution claimed in the paper.

1. The first contribution consists of two parts. The first is defining the four CL settings. However, three of them (task-IL, class-IL,domain-IL) already exist and are widely adopted in almost all CL works [1,2]. Therefore, this is simply adopting existing settings and is not a contribution. As for the time-IL setting, it is conceptually problematic, which is also reflected in experiments (this will be detailed below in the problems on the experimental settings). Second, the authors shows that these settings can be applied to different graph related problems and provides 17 benchmark scenarios with 9 real-world datasets. Although the number of the benchmark scenarios is large, most of them are not carefully built and are not qualified enough to be benchmark for public use. This will be explained later in details.

2. The second contribution consists of two parts. The first one is that the proposed framework is fool-proof because it seperates the evaluation part and training part and can 'eliminate potential mistakes in evaluation'. But what are the potential mistakes in evaluation? This requires further clarification. As far as I know, this problem was never mentioned in any existing CL work. If this is a unique problem in graph CL, please add more explanations on it. The second part is that the proposed framework is easy-to-use, which should be an important point for a benchmark. However, in the current version, the code are unfortunately incomplete and not well documented. This problem will be detailed later.

3. The last contribution consists of three parts. The first is the benchmark results of seven graph CL methods. However, at least four benchmarked methods (LwF,EWC,MAS,GEM) are not graph CL methods but are methods for independent data. Since their implementations are publicly available, testing them on graph data can hardly be a contribution. Moreover, it seems that applying the CL methods for independent data to graph data is straightforward, making the diffrentiation between the graph CL and CL on independent data very unclear. Second, the validation step is claimed as a contribution, because it is absent from previous benchmarks, which seems not true after checking the two previous graph CL benchmarks compared in the paper. Finally, reporting four metrics is claimed as a contribution. But the adopted metrics are insufficient, because all of them are something averaged over all tasks. Continual learning works require the demonstration of the performance change. This will be detailed later.

Besides, using fewer lines of code is also strengthened as an advantage. However, fewer lines of code does not necessarily means higher efficiency or better readability, therefore this point is not a convincing advantage. 


**II. Problems on the experiment settings**

Although many benchmark scenarios are considered in this paper, most of them are not carefully built and are not qualified enough for benchmarking.

1. A common problem for all scenarios is that the extremely small number of tasks, which is far from enough for continual learning. Most scenarios have only 3 or 5 tasks to learn. The scenario with most tasks only contains 12 tasks. Continual learning works typically considers at least tens of tasks [2,3,4,5], and some even consider thousands of tasks, e.g. [3].

2. Among the four graph prediction datasets, two (MNIST and CIFAR10) are image datasets. Is there any good reason to use these 'pseudo graphs' instead of the real graph datasets? For graph prediction, both the task- and class-IL settings are only based on the image datasets, making the benchmark results not convincing. 

3. The four adopted evaluation metrics use single values as the average performance over all tasks. However, the dynamics of the performance (how performance changes after every task in the form of performance curve) is very important for CL, and are demonstrated in almost all existing CL works [2,3,4,5], including the other two graph CL benchmarks mentioned by the authors. 

4. The time-IL setting is not reasonable and potentially misleading. As mentioned by the authors, a key point in designing the tasks is 'the forgetting problem should happen without careful consideration'. In other words, the distribution of different tasks should be sufficiently different. However, in the proposed time-IL, there is no evidence that the adopted temporal splitting ensure a sufficient distribution shift. On the contrary, according to the experiments, the forgetting (AF) of time-IL is much lower than the other settings, which denotes that the tasks in time-IL indeed are not different enough for continual learning. 

5. The hyper-parameter setting is problematic. It is stated that grid search is used to find the best hyperparameters. However, for CGNN, GEM, and ERGNN, only one candidate hyperparameter is used, isn't it weired to perform grid search over only one choice? For other methods, only two choices are used for grid search. Besides, for the replay-based methods (GEM, ERGNN,CGNN), the size of memory is arbitrarily given for different datasets without explanations.

6. According to the authors, link prediction is a straightforward extension from node prediction, then why is ERGNN applicable to Node classification but not to link prediction?


**III. Problems with the provided code and reproducibility**

As a benchmark paper, the code repository should be well maintained and easy to use. Providing easy-to-use code is also claimed as a contribution of this paper. However, after checking the paper, supplementary materials, and the code repository, I found no instruction given on using the code. 

When trying to understand the code with the demo.py, I found that the folder 'src.benchmarks.cora' used in the demo does not even exist. Moreover, the joint baseline seems not included in the code. I didn't go further to check other problems, but it is obvious that the current code is not qualified. Besides, the code is not well commented, most of the code have no comment at all, making is hard to understand.

[1] Van de Ven, Gido M., and Andreas S. Tolias. "Three scenarios for continual learning." arXiv preprint arXiv:1904.07734 (2019).

[2] Rebuffi, Sylvestre-Alvise, et al. "icarl: Incremental classifier and representation learning." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017.

[3] Wortsman, Mitchell, et al. "Supermasks in superposition." Advances in Neural Information Processing Systems 33 (2020): 15173-15184.

[4] Yoon, Jaehong, et al. "Lifelong learning with dynamically expandable networks." arXiv preprint arXiv:1708.01547 (2017).

[5] Lopez-Paz, David, and Marc'Aurelio Ranzato. "Gradient episodic memory for continual learning." Advances in neural information processing systems 30 (2017).

clarity,_quality,_novelty_and_reproducibility: The clarity of this work is good, everything is explained clearly.

The quality of the current version is not enough. However, if the proposed problems can be properly addressed, this should be a meaningful work.

Novelty is limited.

Reproducibility is not likely to be guaranteed, since the code is incomplete.
summary_of_the_review: A benchmark paper would typically be the basis of many future works, and therefore should be rigourous in terms of both the experimental settings and the implementations. However, as explained above, this paper fails to fullfill the requirements. The problematic experiment settings and the resulting questionable benchmark results, as well as the badly maintained code repository are potentially misleading to future works. Therefore, I recommend to reject this paper. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5922/Reviewer_rhFV
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper explores continual learning of graph-structured data. The paper formulates four scenarios of continual learning for graph-structured data and provides suitable evaluation protocols for continual learning algorithms in these settings. The paper also proposed 
 BEGIN which is a continual learning benchmark for evaluating continual learning algorithms. Extensive experiments using existing algorithms are provided to demonstrate that BEGIN is an effective benchmark. 
strength_and_weaknesses: Strength 

1. The paper is well-written and easy to follow.

2. CL for graph-structured data is not well-explored and is an interesting research problem.


Weaknesses

1. The settings are contrived without much connection to reality.

2. Algorithmic novelty of the paper is incremental.
clarity,_quality,_novelty_and_reproducibility: The paper is written well and is easy to follow. The paper is novel in that it provides a benchmark and a centralized codebase that will make evaluating continual learning algorithms on graph-structured data straightforward. However, the paper is not novel in terms of introducing an algorithm. For ICLR, algorithmic novelty is more important and hence, I think the paper is not novel. Code for the experiments is provided and seems well documented.
summary_of_the_review: This is a benchmark dataset without much algorithmic novelty. The paper is not introducing a dataset based on new measurements, rather it builds the benchmark using existing datasets. However, The datasets that are used in this work lack a real notion of time dependence, and the tasks are generated somewhat synthetically. I was wondering why datasets such as GDELT are not used that have a natural temporal axis that can be used to define tasks that are naturally dependent. I think this paper is more suitable for NeurIPS Dataset and Benchmarks track because it is not a strong paper in terms of the code proposed idea.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough