Paper: ON LEARNABILITY AND EXPERIENCE REPLAY METHODS FOR GRAPH INCREMENTAL LEARNING ON EVOLVING GRAPHS (ICLR-2024)

Reviewer comment (Theory):
Theorem 2.3 seems problematic. See the counter-example below.

Author response:
Retraining from scratch is NOT a counterexample to Theorem 2.3, and we maintain that no essential assumptions have been overlooked for Theorem 2.3 to retain its validity.

- The implication of Theorem 2.3 lies in whether there exists a learning algorithm capable of training or updating the model in the prescribed incremental manner (as described in Section 2). Retraining from scratch stands apart as it does not align with the principles of incremental learning and, therefore, does not qualify as a counterexample to Theorem 2.3
- The objective of incremental learning is to devise an algorithm that achieves performance comparable to retraining from scratch, which is why we utilize it as a baseline for performance comparison in the empirical study.
- We intentionally omitted some minor technical details from Theorem 2.3, as indicated by our note that it is an informal version. The formal version is provided in the appendix. However, we do not believe we have overlooked any crucial assumptions for Theorem 2.3 to hold.

Reviewer comment (Theory):
It is not clear why the proposed method can mitigate the issue of “structural shift.”

Author response:
We can not control structural shifts within the data but we can manage and mitigate the impact of structural shifts on the model's performance.

- The key concept underpinning our proposed approach to mitigating the effects of structural shifts is our replay strategy (as detailed in Section 3.2). This strategy employs the principle of importance reweighting, which works to reduce the distance between sample representations in the latent space. This approach is specifically engineered to counteract the adverse effects of structural shifts.

Reviewer comment (Theory):
The impossibility result of NGIL in Theorem 2.3 is hard to understand, particularly regarding what the authors mean by a **good** classifier \( f \).

Author response:
In Theorem 2.3, the symbol $\mathcal{F}$, as defined in Section 2.1, represents the hypothesis space of the model, encompassing all possible parameters from the neural network. This is a standard notion employed in statistical learning theory.

Reviewer comment (Theory):
The exact setting of available training data information used at each time step in Theorem 2.3 is unclear; it must rule out retraining from scratch (Joint Training) for the theorem to make sense, but it is not evident where or how this is done.

Author response:
Retraining from scratch is NOT a counterexample to Theorem 2.3, and we maintain that no essential assumptions have been overlooked for Theorem 2.3 to retain its validity.

- The implication of Theorem 2.3 lies in whether there exists a learning algorithm capable of training or updating the model in the prescribed incremental manner (as described in Section 2). Retraining from scratch stands apart as it does not align with the principles of incremental learning and, therefore, does not qualify as a counterexample to Theorem 2.3
- The objective of incremental learning is to devise an algorithm that achieves performance comparable to retraining from scratch, which is why we utilize it as a baseline for performance comparison in the empirical study.
- We intentionally omitted some minor technical details from Theorem 2.3, as indicated by our note that it is an informal version. The formal version is provided in the appendix. However, we do not believe we have overlooked any crucial assumptions for Theorem 2.3 to hold.

Reviewer comment (Theory):
Even if Theorem 2.3 is correct, the main issue is that structural shift is uncontrolled. Since structural shift depends on how new nodes are added to the graph, which is determined by the data rather than algorithm design, it is unclear how the authors can claim their method mitigates this issue.

Author response:
We can not control structural shifts within the data but we can manage and mitigate the impact of structural shifts on the model's performance.

- The key concept underpinning our proposed approach to mitigating the effects of structural shifts is our replay strategy (as detailed in Section 3.2). This strategy employs the principle of importance reweighting, which works to reduce the distance between sample representations in the latent space. This approach is specifically engineered to counteract the adverse effects of structural shifts.

Reviewer comment (Theory):
The relationship between NGIL and graph unlearning is not discussed. Without privacy considerations, NGIL can be viewed as the reverse of graph unlearning, and techniques from [1,2] seem provably applicable for simple problems and models. This raises the question of whether Theorem 2.3 contradicts findings in the machine unlearning literature.

Author response:
Theorem 2.3 does not contradict the graph unlearning literature.

- Graph unlearning and graph incremental learning represent two distinct problems, despite the presence of some shared terminology. Both research areas have extensive bodies of existing work. The objective of graph incremental learning is to update or train the model with new data without "forgetting" past knowledge. In contrast, graph unlearning seeks to remove specific target information from the model.
- Both the form and the objective of the learning algorithms sought by these two problems are different, and therefore, Theorem 2.3 does not contradict the graph unlearning literature.

Reviewer comment (Novelty):
The problem setting is not novel. My first concern is the novelty of the problem setting. Node-wise Graph Incremental Learning has been extensively studied by previous works.

Author response:
**W1 & W2. The problem setting is not novel.**

The novelty of our works comes from the theoretical analysis of the learnability of NGIL and the proposed experience replay method.

- We discuss extensively in the introduction and related work sections that the NIGL problem has garnered increasing attention in the community and has been studied in other related works (see the related work section in the paper).
- Rather than diminishing the significance of our work, the increasing and existing attention to the NGIL problem should highlights the importance of our results. We provide the first impossibility result linking the learnability of NGIL and structural shift. Additionally, we introduce a novel experience replay method that outperforms other experience replay methods for NGIL under structural shift.

Reviewer comment (Novelty):
The technical contributions, while sound, seem limited. The method presented integrates an experience buffer and importance re-weighting to tackle challenges such as catastrophic forgetting and structural shifts. However, using an experience buffer for stream data isn't a new concept, and many works have already explored it. The reviewer finds the technical contributions of this section somewhat limited.

Author response:
**W2. The technical contributions, while sound, seem limited.** 

We respectfully disagree with the reviewer's assessment of the technical contributions of our study and would like to clarify a few points to assert that our contribution is not, in fact, limited.

- First, it's important to note that our paper has two main contributions: 1) the first impossibility result for NGIL (also see the response to W1) and 2) a novel experience replay method.
- Experience replay is a widely used framework for incremental learning, and there is a substantial body of work dedicated to exploring different experience replay strategies (as discussed in the related works and baselines of our empirical study).
- The experience replay method proposed in the paper is novel. It is designed with the insight from the theoretical analysis and tailored for NGIL with structural shift. It demonstrates state-of-the-art performance over other experience replay methods in the NGIL problem.

In conclusion, considering the heightened attention directed towards NGIL (as indicated by the reviewer), and considering the theoretical and methodological contributions of this paper to NGIL, we assert that the technical contribution of our paper is sufficient.

Reviewer comment (Novelty):
The theoretical innovation appears to be minimal. Once the input graph is broken down into a series of ego-graphs, the definitions, formulations, and theoretical underpinnings seem like straightforward adaptations from their IID data counterparts.

Author response:
**W3. Theoretical setting and  IID data counterparts**

The theoretical setting and problem formulation do not reduce to the I.I.D data counterparts

- Formulating the learning problem from the perspective of ego graphs is a common approach for the theoretical analysis of graph neural networks (e.g., [1-3]). In this formulation, the dependency among vertex is manifested by the shared underlying graph structure. Therefore, this can not be deemed as an I.I.D data setting.
- The ``NON. I.I.D ness'' of the setting is also reflected by the analysis of the structural shift, which is the dependency of data sampling across different tasks/training sessions. Such dependency does not occur in the I.I.D data setting (e.g., incremental learning in CV) and is unique to the NGIL problem.

Reviewer comment (Writing):
The writing requires refinement. The use of "bf" before "Node-wise Graph Incremental Learning" appears to be a formatting mistake.

Author response:
**W4. use of ``\bf'' to highlight and definition of distortion**

- use of ``\bf''
    - The use of **`\bf`** was a deliberate choice to emphasize and draw the reader's attention to important results or concepts.
    - This formatting convention has been widely adopted and accepted in machine-learning conference papers to enhance readability.
- Definition of distortion
    - The definition and description of distortion presented in Section 3.3 are complete.
    - The definition of distortion was originally presented under the definition of environment and was later changed due to space constraints. This alteration resulted in some cross-referencing issues in the appendix. We appreciate your observation, and we have corrected this in the revised version of the paper.

Reviewer comment (Writing):
The definition of distortion, as presented in Definition ??, is incomplete or missing.

Author response:
**W4. use of ``\bf'' to highlight and definition of distortion**

- use of ``\bf''
    - The use of **`\bf`** was a deliberate choice to emphasize and draw the reader's attention to important results or concepts.
    - This formatting convention has been widely adopted and accepted in machine-learning conference papers to enhance readability.
- Definition of distortion
    - The definition and description of distortion presented in Section 3.3 are complete.
    - The definition of distortion was originally presented under the definition of environment and was later changed due to space constraints. This alteration resulted in some cross-referencing issues in the appendix. We appreciate your observation, and we have corrected this in the revised version of the paper.

Reviewer comment (Writing):
The paper is written with unnecessary formalism in some cases which makes it harder to read.

Author response:
**W1 & W2 Unnecessary formalism and straightforward presentation**

We respectfully disagree that the formalism presented in the paper is unnecessary.

- A key motivation and contribution of this paper is to formally formulate the NGIL problem within the statistical learning framework. This formalism is essential for two reasons:
    - Given the inherent complexity and vague terminology associated with NGIL, formalism is crucial for precisely defining the problem and distinguishing it from similar settings, such as unlearning.
    - The formalism, rooted in elements from statistical learning, is a necessary foundation for formally deriving the learnability of the problem.
- We have made an effort to simplify the formalism for readability and believe its current presentation is essential.

We would appreciate it if the reviewer could specify which aspects of the formalism (definition or setting) could be considered unnecessary for our results, as this would help us make targeted improvements.

Reviewer comment (Writing):
The main idea and the rational could have been presented in a more straightforward manner.

Author response:
**W1 & W2 Unnecessary formalism and straightforward presentation**

We respectfully disagree that the formalism presented in the paper is unnecessary.

- A key motivation and contribution of this paper is to formally formulate the NGIL problem within the statistical learning framework. This formalism is essential for two reasons:
    - Given the inherent complexity and vague terminology associated with NGIL, formalism is crucial for precisely defining the problem and distinguishing it from similar settings, such as unlearning.
    - The formalism, rooted in elements from statistical learning, is a necessary foundation for formally deriving the learnability of the problem.
- We have made an effort to simplify the formalism for readability and believe its current presentation is essential.

We would appreciate it if the reviewer could specify which aspects of the formalism (definition or setting) could be considered unnecessary for our results, as this would help us make targeted improvements.

Reviewer comment (Experiments):
The main concern is using some samples again and again. Although it may appear to be replay or revision, it has to be critically analysed how revisiting some samples is justified.

Author response:
**W3. The main concern is using some samples again and again. Although it may appear to be replay or revision but it has to be critically analysed how revisiting some samples is justified.**

- Experience replay (revisiting or replaying a small set of important samples) is one of the most effective and standard frameworks for tackling catastrophic forgetting in incremental learning [2] (also see a detailed discussion in the related work section). Instead of retraining from scratch, experience replay only needs to replay or revisit a small set of samples for consolidating knowledge and information, achieving comparable performance to retraining the model with complete data. This is a huge gain in efficiency.
- Experience replay was originally inspired by human learning theory, which uses repeat visits to important samples to consolidate knowledge and information. The idea of experience replay has been widely adopted in incremental learning[2](also see a detailed discussion in the related work section for its application to incremental learning in CV and NLP). Many different experience methods have been developed to tailor incremental learning with different settings.

Reviewer comment (Evaluation):
It is not clear if there is some unlearning and relearning effect or not.

Author response:
**W4.  It is not clear if there is some unlearning and relearning effect is there or not.**

- Catastrophic forgetting (or ``unlearning'') is a well-documented phenomenon in incremental learning studies, including NGIL contexts [1, 2]. This is evidenced in Table 1 by comparing the difference between the Bare model and Joint Training, where the Bare model shows significant performance degradation due to catastrophic forgetting.
- Our proposed method's effectiveness (the ``relearning effect'') is demonstrated by performance retention in the NGIL setting (as defined in Eq. (1)) and is measured using standard NGIL evaluation metrics (described in Sec. 4).  The performance of our proposed method, along with other experience replay methods, is also presented in Table 1 and is reflected by the improvement over the Bare model.

The above responses are based on our interpretation of what the reviewer means by "unlearning and relearning effect," as we have not used these terms anywhere in the paper. Please let us know if "unlearning and relearning effect" has a different interpretation.

Reviewer comment (Theory):
It is not clear how such replays deviate from the overall learning objective.

Author response:
**W5. It is also not clear how such replays deviate the overall learning objective.**

- As discussed in Section 3.2, existing experience replay methods for NGIL typically employ Eq. (4) as their learning objective. However, based on insights into structural shifts, we propose a novel replay strategy (learning objective) that employs importance re-weighting and is manifested in Eq. (5).

The above responses are based on our interpretation of what the reviewer means by the "overall learning objective." Please clarify if the "overall learning objective" has a different meaning.

Reviewer comment (Evaluation):
How does the objective or the learning path change with respect to the order of increments of the graphs?

Author response:
**W6. How does the objective or the learning path change with respect to the order of increments of the graphs ?**

- The primary metric used to evaluate NGIL is the average performance of the model on all tasks after all training sessions. In theory, while the order of graph increments may affect the performance of individual tasks, it should not have a significant impact on the overall performance metric. In our empirical study, we followed the recent benchmark paper to create the NGIL setting[2].
- In response to this, we have included additional experiments to showcase the learning dynamics under different permutations of graph increments in Appendix D. The results align with the theoretical predictions above.