Paper: Divide and not forget: Ensemble of  selectively trained experts  in Continual Learning (ICLR-2024)

Reviewer comment (Novelty):
Lack of novelty: selecting the next task based on largest KL divergence in continual learning is not a novel strategy. So is the expandable MoE architecture and the inference method.

Author response:
**Novelty concern related to the next task selection**

> *Selecting the next task based on largest KL divergence in continual learning is not a novel strategy. So is the expandable MoE architecture and the inference method.*

We thank the reviewer for that comment, yet we believe that there may be some misunderstanding of our method. Please note that we do not select tasks. We use KL-divergency to assign a single, existing expert to be trained on a given task. However, during the inference time, we use an ensemble of all available $K$ experts whose predictions are aggregated, and this approach is different from conventional MoE architectures. As far as the novelty of our approach is concerned, we believe that our work is the first to show how allocating experts to be trained on tasks can alleviate catastrophic forgetting in continual learning. We would appreciate if the Reviewer could link references to the mentioned relevant existing continual learning strategies so we can refer to them in our work.

Reviewer comment (Reproducibility):
Overhead growth: one biggest disadvantage of architecture expansion is that the memory overhead grows linearly with respect to the number of tasks. That is, it needs K experts for K tasks with nothing reusable. SEED omits the selection procedure to decide the number of experts needed even if the number of tasks is large. This is not scalable in continual learning, where the number of tasks can be unbounded.

Author response:
**Concern of the overhead growth**

As stated in Method section 3 ```Our approach, presented in Fig.2, consists of $K$ deep network experts```. Therefore, K is the number of experts, which is equal to 3 or 5 in our experimental section regardless of the number of tasks |T|. We do not grow the network architecture above that number, e.g., in Fig. 4 and Tab. 1 we present experiments with |T|=50, where K=5 experts are used.

Reviewer comment (Experiments):
In table 1 and 2, although it is clear that SEED performs better than the baselines, different methods have different memory footprint (model parameters and other components) or training complexities. Thus, it is difficult to judge if the gains come from the proposed method or from the additional complexities.

Author response:
**Do the gains come from the proposed method or from the additional complexities?**

We agree that the question whether the gains come from the proposed method or the additional complexities is important. Therefore, we run the experiments with results presented in Tab. 3 and Tab. 4 of our submission. More specifically, in Tab. 4, we show that our method of ensembling experts in CIL outperforms existing SoTA ensembling methods; all require the same amount of parameters. In Tab. 3, our method achieves better results than competitors while using fewer parameters. We have also performed another experiment presented in Tab. 5 (and below) where we evaluate methods with regard to the number of theirs parameters during inference. We have added it to the revised version. The above observations lead us to the conclusion that the gain of our method does not result only from the additional complexities.  

We affirm that various CL methods present different memory/computational footprints. Especially for ensemble methods, it is inherent that few experts have to be stored in memory. Evaluating the complexity of some methods from Tab. 1 & 2 is not trivial, e.g., SSRE uses a complex few-phase training with expansion and pruning phases. However, pruning can also be applied to every other method from these tables. We try to be clear about SEED, and therefore we present and discuss: (1) the parameter efficiency in Tab. 3 compared to other MoE and ensemble architectures, (2) memory footprint in Appendix A.3, (3) we have also added a tradeoff between number of models' parameters vs accuracy in Appendix A.4 (revision version) to better address this concern.

Reviewer comment (Theory):
SEED implies that the number of experts should be smaller than the number of tasks and there is only one expert activated per task. Thus, it is inevitable that some experts will be reused in the future, leading to catastrophic forgetting especially when the tasks are conflicting.

Author response:
**Avoiding catastrophic forgetting in SEED**

One of the main goals of our method is to limit catastrophic forgetting. For that reason, we proposed our expert selection strategy to alleviate forgetting. We demonstrate the improvements in Tab. 4 and Fig. 6. Please note that we train only one expert in each task and that knowledge distillation regularization is applied only to it. If an expert is inactive (not trained on tasks), the parameters of its feature extractor do not change. Thus, it's not prone to catastrophic forgetting.

Reviewer comment (Theory):
The use of LwF regularization might not be helpful if the expert stays inactive for a long period.

Author response:
**Avoiding catastrophic forgetting in SEED**

One of the main goals of our method is to limit catastrophic forgetting. For that reason, we proposed our expert selection strategy to alleviate forgetting. We demonstrate the improvements in Tab. 4 and Fig. 6. Please note that we train only one expert in each task and that knowledge distillation regularization is applied only to it. If an expert is inactive (not trained on tasks), the parameters of its feature extractor do not change. Thus, it's not prone to catastrophic forgetting.

Reviewer comment (Novelty):
A recent related method [A] that seems to outperform SEED should be discussed.

Author response:
**Comparing VariGrow to SEED**

The method cited by the Reviewer performs experiments with exemplars and auxilary datasets (4.1 Implementation details: ```We also use these exemplars along with OoD datasets```), while we propose an exemplar-free method for more challenging scenarios and we do not use exemplars or auxilary data in our experiments.

Reviewer comment (Writing):
There are two different references for CoSCL in the Introduction.

Author response:
**Two different references for CoSCL in the Introduction.**

We have fixed this in the revised version. Thank you.

Reviewer comment (Theory):
In the final version, the authors are strongly encouraged to clearly state the assumptions and scenarios that the proposed method will work well on.

Author response:
**CTrL benchmark evaluation**

We thank the Reviewer for pointing out this benchmark. Although, as rightfully mentioned by the Reviewer, time does not permit us to run the evaluation on this framework within the rebuttal period, we acknowledge the need to state the assumptions of SEED clearly and reference CTrL work as a relevant part of the literature. We have done so, revising the manuscript as described below.  

In Section 2 we have referred to CTrL after introducing FeTrIL work: 
> However, freezing the backbone can limit the plasticity and not be sufficient for more complex settings, e.g., when tasks are unrelated, like in CTrL (Veniat et al.,2020).  

In Section 5 we have added the new limitation:
> Firstly, SEED may be not feasible for scenarios where tasks are completely unrelated and the number of parameters is limited, as in that case sharing initial parameters between experts may lead to a poor performance.

Reviewer comment (Experiments):
Performance comparison under a fair setting.

Author response:
**Performance comparison under a limited number of parameters.**

To address this concern, we have added Tab. 5 to the appendix (and below). SEED achieves better accuracy than the best competitor (FeTriL) from Tables 1&2 while using less parameters. Here we consider number of parameters required by the method to perform the inference. For SEED these paramters come from feature extractors of experts (here they share first 17 layers) and Gaussian distributions.

| Method | Network    | Network weights |Total parameters| \|T\|=10 | \|T\|=20 | \|T\|=50 |
| -------- | -------- |-------- |-------- | -------- | -------- | -------- |
| EWC     |ResNet32   |  473K   |  473K  | 24.5     | 21.2 | 15.9
| LWF     |ResNet32   |  473K   |  473K  | 45.9     | 27.4 | 20.1
| FeTrIL  |ResNet32   |  467K   |  473K  | 46.3     | 38.7 | 27.0
| SEED    |ResNet20   |  339K   | 460K   | 54.7     | 48.6 | 33.1

Reviewer comment (Experiments):
How could SEED avoid forgetting when the number of tasks is significantly larger than the number of experts.

Author response:
**How could SEED avoid forgetting when the number of tasks is significantly larger than the number of experts?**

In that case SEED avoids forgetting in two ways. First, in each task SEED trains only one expert, other experts (K-1) remain unchanged so that they avoid forgetting. Please note, that layers shared by experts are frozen after the first task. Second, we utilize feature distillation for trained expert to alleviate its forgetting - we made it more clear in Section 3 in the revised version.

Reviewer comment (Experiments):
No major weaknesses. Perhaps just the fact that several models (experts) have to be trained and stored (but this is inherent in ensemble methods).

Author response:
**The need to train and store a number of experts**

As the Reviewer stated, this is inherent in ensemble methods. However, we discuss the number of parameters in Tab.3 and Appendix A.3. We have additionally run experiments to examine the trade-off between the number of parameters and accuracy of SEED in Appendix 4. We show that the total number of parameters can be lowered, and SEED can still achieve satisfactory results.

Reviewer comment (Presentation):
In Figure 3, it's hard to see that task 3 overlap least with the second expert. Perhaps make this a bit clearer.

Author response:
>_In Figure 3, it's hard to see that task 3 overlap least with the second expert. Perhaps make this a bit clearer._

We appreciate that suggestion. We updated the figure in the new revision.

One more time, we express our gratitude towards the Reviewer for improving the quality of our work and considering our work to be worth publication. We have improved Fig. 3 in the newest revision.

Reviewer comment (Presentation):
In Page 4, perhaps the inference and training algorithms would be easier to read in an "Algorithm" structure rather than simple paragraphs.

Author response:
>_In Page 4, perhaps the inference and training algorithms would be easier to read in an "Algorithm" structure rather than simple paragraphs._
    
>_In Page 5, just before Section 4.1., the task incremental scenario is mentioned but is not really defined properly in my opinion.

We have addressed both comments in the revised version. Thank you.

Reviewer comment (Writing):
In Page 5, just before Section 4.1., the task incremental scenario is mentioned but is not really defined properly in my opinion.

Author response:
>_In Page 4, perhaps the inference and training algorithms would be easier to read in an "Algorithm" structure rather than simple paragraphs._
    
>_In Page 5, just before Section 4.1., the task incremental scenario is mentioned but is not really defined properly in my opinion.

We have addressed both comments in the revised version. Thank you.

Reviewer comment (Writing):
In Page 8, it's written that *"SEED uses a regularization method known from LwF"*. Are you actually using LwF for each expert? I don't recall reading it in other places in the paper. Please clarify.

Author response:
>_In Page 8, it's written that "SEED uses a regularization method known from LwF". Are you actually using LwF for each expert? I don't recall reading it in other places in the paper. Please clarify._

SEED uses feature distillation for each expert, similar to LwF (logit distillation) but performed in the latent space. It's mentioned in Sec. 3, Training (page 4/5). We clarified this part of our method description in the new revision.

Reviewer comment (Experiments):
How are the hyperparameters of EWC and LwF set in the experiments? For instance, Figure 7 shows $\lambda\in [100,10000]$ without any justification.

Author response:
>_How are the hyperparameters of EWC and LwF set in the experiments ? For instance, Figure 7 shows without any justification._

We set default parameters of the methods as reported in the original articles. For avoidance of doubt, we also stated it in Appendix A.1: ```For baseline methods, we set default hyperparameters```. For Fig. 7 hyperparameters were manually chosen to exhibit the tradeoff between intransigence and forgetting, and we list all these parameters in Appendix A.1 as well.

Reviewer comment (Writing):
In Page 1, the authors mention *"The trend is evident... results steadily improve over time"*. What time?

Author response:
>_In Page 1, the authors mention "The trend is evident... results steadily improve over time". What time?_

In this case, we meant the dates of publication of the corresponding articles. The methods presented in Fig. 1 are positioned according to the order of their publication dates. We have labeled the publication year axis x in Fig. 1 to better visualize that in the revised version.

Reviewer comment (Reproducibility):
In Table 1, how many repetitions are performed per experiment?

Author response:
>_In Table 1, how many repetitions are performed per experiment?_

We perform three repetitions per experiment, as stated in Appendix A.1: ```All experiments are the average over three runs```.

Reviewer comment (Evaluation):
In Figure 5, the presented metric is "relative accuracy". Relative to what?

Author response:
>_In Figure 5, the presented metric is "relative accuracy". Relative to what?_

We present the metric relative to the mean achieved by all experts. We have clarified that in the caption of Fig. 5 in the updated version: ```It is calculated by subtracting the accuracy of each expert from the averaged accuracy of all experts.```