Paper: Forward Explanation : Why Catastrophic Forgetting Occurs (ICLR-2024)

Reviewer comment (Theory):
The paper presents unsubstantiated claims and contents. For example, in Section 2.2: “This indirectly suggests that the model training is equivalent to training an encoder. However, we have not provided an explanation for these phenomenon. On one hand, we do not have the time to provide a comprehensive mathematical proof,”.

Author response:
(1) There may be a bias in the expression. We simply aim to convey that training is primarily to enable the encoder to acquire the ability to differentiate inputs.  
(2) This part does lack additional experiments, but it is indeed the case.  
(3) If other optimization algorithms are used here, it is true that it won't be as straightforward as in Section 2.4. However, the change will be more complex, but it still occurs along the memory trace. If forgetting occurs, similar representations will be generated, and if normal training takes place, dissimilar representations will be produced.

Reviewer comment (Theory):
The paper presents unsubstantiated claims and contents. For example, in Section 2.4: “In the model initialization phase, all parameters are randomly initialized. At this point, regardless of the input received by the encoder, after undergoing multiple rounds of random parameter adjustment, they will almost all be mapped to similar representations.”

Author response:
Thank you for your information.

Reviewer comment (Theory):
The paper presents unsubstantiated claims and contents. For example, in Section 2.4: “The paper assumes default training using the stochastic gradient descent (SGD) algorithm. If other optimization algorithms are used, there may be differences in the formulation processes, but it does not affect the conclusion.”

Author response:
Thank you for your feedback.

Reviewer comment (Writing):
Is the memory trace just the collection of all layer activations?

Author response:
Response to the question: Yes , it comes in two forms: one as a input vector and the other as traces embedded within the parameters.

Reviewer comment (Experiments):
The authors only give intuitions about the main concepts without any experimental validation: for instance, in the Task Representation Convergence section, the authors claim that *"the parameters of the decoder converge to the expectation of the representation"*. What does this mean? All over the paper text there are similar issues.

Author response:
Part of this is indeed missing in Section 2.2, as reflected in Figure 6. If we use the same dataset for testing in the experiments of Equation 3 as in the pre-training dataset, the accuracy curve will stay around 100%. This is why we claim that "the parameters of the decoder converge to the expectation of the representation"

Reviewer comment (Writing):
The procedure of training the decoder using the *expected representation* is only mentioned without a proper definition. The authors should clarify all the definitions and better formalize them.

Author response:
The implementation details of the expectation of representation are mentioned in the final part of Section 2.1.

Reviewer comment (Writing):
Writing can be improved—Text is copied from some thesis, as it talks about chapter, conclusion 1, 2 etc., and none of those are mentioned in the main paper.

Author response:
Response to Question 2: This is a writing issue and specifically refers to Sections 2.1 and 2.2.

Reviewer comment (Experiments):
The test set is shuffled:  
testloader = data.DataLoader(DATASET(train=False,task=task),batch_size=64,shuffle=True,num_workers=4), why would you shuffle the test set?

Author response:
Response to Weak6: The dataset content is controlled by the parameter [task] here. For instance, [0] indicates only dataset 0, while [0,1,2] includes datasets 0, 1, and 2. Therefore, the use of SHUFFLE has no impact. When we specifically desire a certain dataset, it is controlled through [task].

Reviewer comment (Novelty):
There is a large body of theoretical work neglected in this work. As the issue of catastrophic forgetting is well studied theoretically [1-8], and the issues of neural cross-talk across tasks are studied empirically [9-11], the reviewer is not sure about the novelty of this work, as almost all concepts covered have already been proven in prior works. What is the novelty?

Author response:
Response to Question 1: We have provided an explanation of the relationship between representations, parameters, and catastrophic forgetting. Then we try to reproduce how catastrophic forgetting manifests in parameter changes during the training process. This is distinct from [1-11]. The contribution lies in a negative assessment, asserting that without altering the training method, it would be impossible to overcome  catastrophic forgetting at the cost of retraining, as referenced in Section 3, paragraph 3. Simultaneously, it suggests directions for improvement. To effectively overcome catastrophic forgetting at a reasonable cost, one must utilize the manifestation of historical memory trace in network parameters, aiming to achieve effects similar to those described in Section 2.4.

Reviewer comment (Writing):
The current manuscript appears to be borrowed from a thesis, as section 2.3 says, “In the previous chapter”—What is the chapter here?

Author response:
Response to Question 2: This is a writing issue and specifically refers to Sections 2.1 and 2.2.

Reviewer comment (Writing):
What are Conclusion One and Conclusion 2? Why are they not mentioned in the text? On page 3, the author talks about conclusion one; however, it is not mentioned anywhere in the manuscript what the conclusion means.

Author response:
Response to Question 4: Regarding the experiments in Section 2.2, we have provided the source code. For Conclusion One, parameters can be freely adjusted, while for Conclusion Two, there is no training process involved as it is computed directly. As for the experiments in Section 3, we utilized a third-party visualization library, so the source code is not included.

Reviewer comment (Experiments):
Experiment details are missing; nothing is mentioned to reproduce the results, no ablation study, and no discussion on hyper-parameter optimization.

Author response:
Response to Question 4: Regarding the experiments in Section 2.2, we have provided the source code. For Conclusion One, parameters can be freely adjusted, while for Conclusion Two, there is no training process involved as it is computed directly. As for the experiments in Section 3, we utilized a third-party visualization library, so the source code is not included.

Reviewer comment (Evaluation):
Forgetting in these systems can also be understood by forgetting ratio, backward transfer, and forward transfer (for instance, see the gradient episodic memory paper).

Author response:
Responses to Questions 5-7: We aim to establish a simpler logical chain to illustrate how catastrophic forgetting specifically occurs in engineering.