Paper: Catastrophic Negative Transfer: An Overlooked Problem in Continual Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The problem of negative transfer between RL tasks has been well known for a long time -- though not as extensively explored, indeed.

Author response:
- As we already mentioned in the Introduction section, we agree that the negative transfer has been studied in a similar way like loss of plasticity or capacity loss. However, our experiments have shown that a proper solution for resolving the negative transfer in CRL setting has not been proposed in other works. Either CReLU or InFeR cannot solve this problem, and we stress that this problem cannot be explained via plasticity or capacity loss. Furthermore, as reviewer p3gB mentioned, this problem is not extensively explored in previous methods, and we think that we extensively explored the negative transfer problem in Meta-World environment.
- For the more in-depth analysis about the distinction between negative transfer and capacity/plasticity loss, please refer to the global comment.

Reviewer comment (Theory):
The proposed solution is underwhelming -- in order to avoid negative transfer, the approach also rids itself of the possibility of achieving positive transfer.

Author response:
As mentioned in the conclusion, we agree that our method is not capable of positive transfer. However, addressing negative transfer is a prerequisite before contemplating positive transfer. Despite previous studies, even in the case of P&C, attempting to achieve positive transfer, our experiments revealed that such efforts did not effectively tackle negative transfer. We have demonstrated that our method outperforms existing baselines by solely addressing negative transfer, without explicitly incorporating positive transfer in a straightforward manner. Hence, we assert that our contribution lies in this aspect.

Reviewer comment (Evaluation):
The most closely related approach, Progress & Compress (P&C) was apparently misunderstood by the authors, leading to a misleading evaluation.

Author response:
We acknowledge that we were not aware of the reference to a reset in P&C. We have further experimented with the implementation of P&C. These results are available for review in the Supplementary Materials of the revised manuscript. Our conclusion is that we cannot agree that P&C solves the negative transfer issue. In our experimental setup, we implemented the algorithm proposed by P&C as it is, attaching an adaptor to the policy and receiving features from the knowledge base policy(`policy_kb`). When learning a new task, we reset the policy and the Q function (or the value function), and proceeded with learning again. Note that when learning second task, the adaptor is trained from randomly initialized network, and the third task is learned with pre-trained adaptor from the second task. In the result, we can see that the task is not learned at all. We think this is due to the part where the information from `policy_kb` is passed through the adaptor. It might seem that applying resetting to P&C would solve the negative transfer, but in our experiments, we found that it did not solve the negative transfer at all. Therefore, we can see that P&C is not able to maintain positive transfers and is also adversely affected by negative transfers.

Reviewer comment (Presentation):
There is no distinct related work section.

Author response:
Our intention was to have the Introduction section serve as an introduction to the related work, but this seems to have been confusing. We will add a new Related Work section to describe more about works such as transfer learning later on.

Reviewer comment (Novelty):
It is unclear what is the novelty of the setting studied in this work. The problem of negative transfer in RL is well known. Indeed, in the continual setting it has been somewhat tied with plasticity/capacity (probably incorrectly). The authors' discussion of how it differs from these other issues is an interesting first step, but fails to provide any clear insight. Questions such as what is the reason for this negative transfer, when does it happen, and how can we avoid it, remain unanswered.

Author response:
- Firstly, the results in Figure 1 are from conducting only finetuning. We are highlighting the issue of negative transfer persisting even in the simple finetuning scenarios. Similar concerns were previously raised in the context of capacity/plasticity loss.
- Secondly, I think what you're saying is somewhat similar to what we're trying to say. This is because as the agent learns, it gets better at some tasks and worse at others as it fits into one task.  In this case, from the network's point of view, the only difference from learning from scratch is the parameter initialization. This is the reason we directed our attention towards the re-initialization of the network.

Reviewer comment (Novelty):
The authors fail to reference a recent treatment of a very similar issue in Kessler et al. (2022), in which the authors explicitly study one particular setting where negative transfer occurs: when two tasks require executing different actions in the same state.

Author response:
- [1] discusses the difficulty of learning two tasks with the same state space and opposing behaviors. While there may be some similarity in addressing dissimilarity between tasks, there are significant differences in our work.
- To simplify, let's consider the scenario of sequentially learning two tasks, A and B, with the same state space and opposing target behaviors in the order of A→B. [1] does not address the situation where learning B is hindered by A. Instead, it focuses on situations where, as B is well-learned, A is forgotten since it requires opposite actions. In contrast, the negative transfer proposed in our work deals with a scenario where learning B becomes impossible due to prior learning of A. This is not simply because A and B target opposite behaviors; our experiments demonstrated the inability to learn B even when all information about A was lost during fine-tuning without any regularization.
- Such a problem cannot be resolved by methods like OWL mentioned in [1]. OWL, in brief, attaches heads corresponding to each task to the network and applies EWC or other continual learning methods to the shared parameters. In fact, all of our experiments use multi-head agents, but negative transfer is still not solved.
- Additionally, interference due to opposite behavior, as mentioned in [1], intuitively seems plausible but somewhat contradicts our experimental results. For example, when sequentially learning 'window-close' and 'window-open,' which have the exactly same states and opposite goals, we observed successful learning without negative transfer.

Reviewer comment (Evaluation):
It is unclear why the results of Figure 2 cannot be explained via loss of capacity/plasticity. While the reviewer agrees that this is probably not the cause (because finetuning doesn't restrict capacity or plasticity), there is no clear enough argument in the result to clearly establish that claim.

Author response:
- Firstly, the results in Figure 1 are from conducting only finetuning. We are highlighting the issue of negative transfer persisting even in the simple finetuning scenarios. Similar concerns were previously raised in the context of capacity/plasticity loss.
- Secondly, I think what you're saying is somewhat similar to what we're trying to say. This is because as the agent learns, it gets better at some tasks and worse at others as it fits into one task.  In this case, from the network's point of view, the only difference from learning from scratch is the parameter initialization. This is the reason we directed our attention towards the re-initialization of the network.

Reviewer comment (Reproducibility):
There is a critical flaw in the authors' implementation of P&C. The authors claim that P&C is restricted to "one active column... without any re-initialization." However, the original P&C paper states that re-initialisation of these parameters is recommended for more diverse tasks to make learning more successful.

Author response:
We acknowledge that we were not aware of the reference to a reset in P&C. We have further experimented with the implementation of P&C. These results are available for review in the Supplementary Materials of the revised manuscript. Our conclusion is that we cannot agree that P&C solves the negative transfer issue. In our experimental setup, we implemented the algorithm proposed by P&C as it is, attaching an adaptor to the policy and receiving features from the knowledge base policy(`policy_kb`). When learning a new task, we reset the policy and the Q function (or the value function), and proceeded with learning again. Note that when learning second task, the adaptor is trained from randomly initialized network, and the third task is learned with pre-trained adaptor from the second task. In the result, we can see that the task is not learned at all. We think this is due to the part where the information from `policy_kb` is passed through the adaptor. It might seem that applying resetting to P&C would solve the negative transfer, but in our experiments, we found that it did not solve the negative transfer at all. Therefore, we can see that P&C is not able to maintain positive transfers and is also adversely affected by negative transfers.

Reviewer comment (Experiments):
The choice of tasks that can be trained within 3 million steps is especially well suited for an algorithm that achieves no forward transfer. It is trained only on the easy tasks, which can be learned by definition with no forward transfer.

Author response:
We focused our experiments on the occurrence and resolution of negative transfers. We excluded tasks that were not learned within 3 minutes of the CRL because they would not allow us to see the extent of negative transfer. Though it would be better to consider more hard and complex tasks to show the forward transfer in our experiment, we think that resolving the negative transfer is our major challenge, and achieving the forward transfer remained as future work.

Reviewer comment (Experiments):
For the experiments with CReLU or InFeR, why is R&D not compared? It would help bolster your method if in both scenarios R&D shows benefit. If not, it would be helpful to state why R&D is not included here. I can understand from logic, you state that it is not helpful. Yet, you used R&D in a separate application, so it feels like there is a missing component since you used R&D in a different application.

Author response:
Thank you for the valuable suggestion. We included the results from R&D in the Supplementary Materials. In this result, R&D effectively resolves the negatvie transfer. Though some tasks (e.g. the tasks after learning the tasks in 'Sweep group') are still hard to be learned in SAC, now the tasks in 'Sweep' group can be learned when those tasks lie in the second task. Furthermore, many tasks do not suffer from negative transfer in PPO.

Reviewer comment (Presentation):
For Figures 3 and 5, it may benefit to have each SAC and PPO separate. I understand that the paper limit is a hard problem. Consider in the supplementary material, to have each SAC and PPO plots in there so it makes it easier to see the separation.

Author response:
- Firstly, apologies for the confusion caused by the typo. We also used 10 random seeds in the experiments for Figure 2.  
- We will take into account the other suggestions later on. Thank you for the feedback.

Reviewer comment (Reproducibility):
In the second paragraph you mention that you ran 10 different random seeds. How is that different from Figure 2 where you mentioned 3 seeds? Please provide more details on this.

Author response:
- Firstly, apologies for the confusion caused by the typo. We also used 10 random seeds in the experiments for Figure 2.  
- We will take into account the other suggestions later on. Thank you for the feedback.

Reviewer comment (Evaluation):
In Section 7.3 you mention statistical significance but where are the p-values?

Author response:
- While we did not directly provide p-values in the Supplementary Materials, we have provided enough information through standard deviation for some inference.  
- It seems that the term "statistical significance" may have caused some misunderstanding. We will make corrections to clarify this.

Reviewer comment (Novelty):
Other works show that resetting works for continual learning. It is not new to add Resetting and KL divergence to the loss function to mitigate forgetting.

Author response:
We appreciate the comments and would like to respond; however, we are facing challenges due to the limited availability of specific details. As far as we know, ClonEx introduced behavioral cloning (KL divergence) to prevent forgetting, but it does not address negative transfers. If there are other works with parameter resetting and KD that we missed, we would really appreciate it if you could let us know.

Reviewer comment (Evaluation):
In the section 5.1, it compares fig 2 with fig 4 roughly. However it is not enough for the claim. It is needed to compare precisely the numbers in the two figures.

Author response:
We believe you had this question based on the following statement in Section 5.2 of our manuscript.

> The findings reveal that fine-tuning with CReLU and InFeR yields similar success rates when compared to the results presented in Figure 2.

What we wanted to emphasize with Figure 4 is that the solutions designed to address traditional capacity/plasticity loss, such as InFeR and CReLU, do not address negative transfer. For this purpose, we believed it was more important to compare the results with learning a single task from scratch rather than with finetuning (Figure 2). We have recognized that the wording in the manuscript could be potentially misleading, so we will fix it.

Reviewer comment (Theory):
There is not evidence or proof for remark 1 to say the reason is reward signal. There are different reasons for loss of plasticity.

Author response:
Please refer to the global comment for a response to Remark 1.

Reviewer comment (Reproducibility):
How is the offline actor trained?

Author response:
First, a randomly initialized network (online actor) is used to learn each task. We use a known RL algorithm (SAC and PPO in our case) for training. In the process of training the online actor, there will be a replay buffer stored. If you use an on-policy algorithm such as PPO, you need to save the replay buffer after training. The states and actions in this stored replay buffer contain the knowledge of the online actor. By distilling this knowledge to the offline actor, the offline actor learns.

Reviewer comment (Experiments):
What is the beneficial use of R&D vs just training each task from scratch like just doing resetting?

Author response:
If we train each task with a re-initialized network, it will forget everything learned in the previous tasks, even though it can learn the current task well. Continual learning is not just about effectively learning the current task; it is crucial not to forget the content of previously learned tasks. Simple resetting alone cannot preserve performance on tasks learned earlier, so additional mechanisms are necessary. We solved the forgetting problem by adding behavioral cloning of the previous tasks to the knowledge distillation for the current task.

Reviewer comment (Evaluation):
The results in table 1 are very close together. It is not sufficient to compare just the numbers. It needs to clarify if they are significantly different.

Author response:
Additional explanations about the interpretation of the results of Table 1 have been provided in the global comments. So please refer to them for further clarification.

Reviewer comment (Writing):
What is exactly the expert buffer?

Author response:
Let's consider learning tasks A→B sequentially. The policy immediately after learning task A can be seen as an expert for task A. To retain the learned information at this point, behavioral cloning in continual learning involves storing the actions of the expert for states in task A in a buffer. Later, when learning task B, regularization is performed using the state-action pairs in this buffer. In summary, the buffer containing state-action pairs for the previously learned task is referred to as the expert buffer.

Reviewer comment (Experiments):
In the baselines, it would be worth to compare with just resetting agent or training an agent from scratch.

Author response:
As mentioned earlier, simply resetting the agent for each task is not effective in continual learning. While this method may lead to effective learning of the current task, it results in a significant loss of information for previously learned tasks, ultimately leading to very poor performance.

Reviewer comment (Theory):
What is the evidence for Remark 1? any proof?

Author response:
Please refer to the global comment for a response to Remark 1.

Reviewer comment (Novelty):
The proposed algorithm is straightforward and lacks technical contribution, although it is effective and technically sound in the reviewer’s opinion.

Author response:
The main contribution of our method is to reset the network before learning each task to avoid negative transfer. We also applied knowledge distillation to transfer this information to the continuous learner. It may seem similar to behavioral cloning, but it is only used as a means to an end. What we want to emphasize is that even this simple method can effectively alleviate negative transfer. Also, although our method takes two processes to learn the current task, it is not a big problem because the process of distilling the knowledge of online actors (learners from scratch) to offline actor (continual learner) requires very little time compared to learning online actors.

Reviewer comment (Theory):
The authors claimed a new phenomenon is overlooked in continual RL, but the reviewer finds the resulting conclusion counterintuitive, skeptical, and potentially unreliable without further investigations. Based on knowledge of Q learning in the tabular setting, the convergence of Q learning does not rely on initialization, as supported by the well-accepted proof in [1], especially Theorem 1. This implies that in the tabular setting, even with poor initialization, Q learning can always converge to an optimal policy under mild conditions, which conflicts with the current manuscript’s claim. Although the paper targets deep RL, the reviewer believes it is too early to conclude that catastrophic negative transfer commonly exists, and reasons for its occurrence in deep RL should be carefully investigated and discussed. The reviewer suggests it would be more reasonable to conclude the phenomenon is very unlikely in tabular settings but more likely in deep RL due to optimization/plasticity challenges under various initializations. The current manuscript fails to address this, and thus the reviewer is not convinced by the conclusions and strongly suggests more investigation and discussion.

Author response:
First, there is a difference between tabular setting and deep RL initialization. While tabular initializes the output of the Q function directly, initialization of the Q network in deep RL means initialization of the network parameters.  
- This difference is even more pronounced for Q function updates. In the tabular setting, we directly change the output of the Q function according to the Bellman Equation, but we define a loss function and update the Q function by gradient descent in deep RL.  
- As far as we know, convergence in deep RL is not as well understood as in tabular settings. Therefore, a direct comparison between tabular RL and deep RL is somewhat inappropriate. For this reason, we did not address the theoretical convergence in deep RL. However, we have shown through extensive experiments that learning in deep RL can be affected by initialization, and it is not simply due to the loss of plasticity or capacity of the network.

Reviewer comment (Evaluation):
The reason proposed by the paper for the phenomenon is also unconvincing to the reviewer. While the paper hypothesizes that the reward in current tasks is insufficient to guarantee convergence — which the reviewer mostly agrees with — they believe deeper reasons may lie in detrimental effects from previous task initializations, especially in deep RL. Another possibility is that training time is insufficient, or the algorithms used (SAC, PPO) are not advanced enough to handle varied initializations. The reviewer also suggests that acceleration techniques in RL could mitigate the phenomenon, leading them to believe the issue studied may not be critical.

Author response:
Please refer to the global comment for a response to Remark 1.  
- We also think that training time can be an important point. In many works, such as [1], the number of frames used per task in the Meta-World is set to 1M. In order to exclude the situation of running out of frames as much as possible, we set the number of frames per task to 3M, and then conducted the experiment by selecting only the tasks that can be learned well within 3M from scratch with 10 random seeds. Given that these tasks are effectively trained within 3M steps using well-known RL algorithms such as SAC or PPO, it is difficult for us to regard this as an issue.  
- Regarding the acceleration technique, we first thought that the policy trained on the previous task is not good at exploration, so we tried to train it by adding an auxiliary loss that forces it to some extent. However, this method cannot solve negative transfer.  
- Also, the ClonEx[1] that we adopted as a baseline not only utilizes behavioral cloning to prevent forgetting but also applies several exploration techniques to enhance learning efficiency. However, as evident from our experimental results, these methods did not effectively address negative transfer.  
- Thus, in our opinion, currently known acceleration techniques are not designed with negative transfer in mind, so it is unlikely that they can fundamentally solve it.

Reviewer comment (Novelty):
The proposed algorithm is considered too straightforward. Although well-motivated, effective, and technically sound, it is highly based on behavior cloning and appears equivalent to behavior cloning that additionally considers the current task’s memory buffer. This approach nearly doubles training costs due to the need to train an extra policy, thereby lacking technical contribution.

Author response:
The main contribution of our method is to reset the network before learning each task to avoid negative transfer. We also applied knowledge distillation to transfer this information to the continuous learner. It may seem similar to behavioral cloning, but it is only used as a means to an end. What we want to emphasize is that even this simple method can effectively alleviate negative transfer. Also, although our method takes two processes to learn the current task, it is not a big problem because the process of distilling the knowledge of online actors (learners from scratch) to offline actor (continual learner) requires very little time compared to learning online actors.

Reviewer comment (Evaluation):
The reviewer asks how many seeds are used in Table 1, noting that R&D appears competitive with ClonEX without significant improvement.

Author response:
The results in Table 1 were negative transfer and forgetting measures computed based on the results shown in Figure 3 and 5. Therefore, a total of 10 random seeds were employed. Additional explanations regarding the interpretation of the results have been provided in the global comments; please refer to them for further clarification.

Reviewer comment (Experiments):
The considered baselines are limited, and the straightforward use of behavior cloning in the proposed algorithm may not be superior to other state-of-the-art continual RL baselines.

Author response:
Existing continual RL works primarily focus on addressing catastrophic forgetting. To the best of our knowledge, ClonEx[1] is currently one of the best performing continual RL algorithms, and it effectively addresses the forgetting problem, as shown in our experimental results. Forgetting is minimal in the results when ClonEx is applied, making the need for additional baselines to compare forgetting unnecessary.  
- Furthermore, ClonEx considers not only behavioral cloning but also incorporates several exploration techniques to account for forward transfer. However, we deemed this alone insufficient as a baseline for addressing negative transfer. Therefore, we augmented ClonEx with solutions proposed in InFeR and CReLU, addressing capacity/plasticity loss, as additional baselines. While InFeR and CReLU contribute to improving ClonEx's performance to some extent, as observed in Figure 5 and Table 1, they do not provide a fundamental solution.  
- As mentioned earlier, the most significant contribution of our method is the reset of parameters before learning each task to avoid negative transfer. Our method does not simply apply behavioral cloning, and it can effectively address both forgetting and negative transfer.