Paper: Generate to Discriminate: Expert Routing for Continual Learning (ICLR-2024)

Reviewer comment (Experiments):
Why are different methods compared for different (vision) datasets?

Author response:
Thank you very much for your time and thoughtful feedback! We have now made the following changes which we believe address all your concerns and strengthen the paper
* Added additional results on ER (50/class) for DomainNet and DermCL (see Tables 1,3)
* Added additional results on CaSSLe for CORe50 and DermCL (see Tables 2,3)
* Added clarification to Table 1 caption on the difference between G2D vs. G2D (Full FT) 

> In tab 1: where is ER? Whats CaSSLe?...Why are different methods compared for different (vision) datasets

Thank you for bringing this to our attention. As per your suggestion for completeness, we have run ER and CaSSLe numbers for all remaining baselines and have updated Tables 1,2,3 accordingly, which now unifies all baselines assessed for the vision datasets. TLDR: Our method outbeats both baselines across all datasets by a large margin. We have included a summary of the new results in the table below (*denotes results obtained from [1]):

| Method                                | DomainNet         | CORe50           | DermCL           |
|---------------------------------------|-------------------|------------------|------------------|
| ER (50/class)                         | 52.79 ± 0.03      | 80.10 ± 0.56*   | 84.94 ± 0.69     |
| Supervised Contrastive - CaSSLe       | 50.90*           | 75.68 ± 0.60     | 73.45 ± 0.70     |
| G2D                                   | 58.45 ± 0.56      | 89.11 ± 0.30     | 89.14 ± 2.47     |

For further clarification: In our earlier version, these two baselines were only evaluated on selected datasets because previous works [1] [2] included ER and CaSSLe results only on selected datasets and we directly obtained those results. CaSSLe [2] is a method for continual self-supervised learning and ER cannot be performed in our setup due to data sharing constraints; and thus, we had not focused too much on evaluation of these two baselines in our earlier draft. As noted, we have now updated our draft to address your concern.

Reviewer comment (Reproducibility):
The paper does not provide enough experimental details to properly assess whether the comparison between G2D and Generative Replay is performed in a fair manner, making it unclear whether Generative Replay has been implemented in an optimal manner.

Author response:
Thank you very much for your time and thoughtful feedback! We have now made the following changes which we believe address all your questions and strengthen the paper

* Added a section on implementation details of Generative Replay in Appendix C.3
* Added a summary of these clarifications to Section 5.2 with a reference to Appendix C.3
* Updated section 4.2 to correct description about ER (see below) 

 > ... does not provide enough experimental details to properly assess whether the comparison between G2D and Generative Replay is performed in a fair manner…it is unclear to me whether Generative Replay has been implemented in an optimal manner

We have updated the draft with the following clarifications (see Section 5.2  and Appendix C.3). Further, we plan to release all code regarding our implementation of both G2D and Generative Replay, for the community. 

We sought to make the comparison fair by (a) using the same replay buffer (i.e., fixing to the same set of synthetic data) for both methods and (b) using the *same* fine-tuning approach for classifier models (see below):

Reviewer comment (Reproducibility):
How is / are the classifier model(s) finetuned? In section 5.4. it is stated “we fine-tune only 1.04 ~ 2.5% of trainable parameters”. How was this percentage decided? How is it decided which parameters are fine-tuned? Is this approach of fine-tuning the same for the classifier models of G2D and the classifier model of Generative Replay?

Author response:
> How is / are the classifier model(s) finetuned?...Is this approach of fine-tuning the same for the classifier models of G2D and the classifier model of Generative Replay?

To be explicit, we will answer this question for (1) G2D (Full FT) vs. Generative Replay and (2) G2D vs. Generative Replay, 

1. Comparison of G2D (Full FT) vs. Generative Replay:

Yes, the approach remains the same. *Both* approaches (for fine-tuning classifier models of G2D (Full FT) and the classifier for Generative Replay) are done by:
* Sequential fine-tuning from the previous domain checkpoint
* Hyperparameter tuning is done over the same set of hyperparameters (learning rates, batch size, etc.) and chosen based on performance on the held-out validation set of the first domain. Hyperparameters remain fixed throughout the domain sequence, for both approaches (see Appendix C.1 and C.2 for hyperparameter details).

2. Comparison of G2D vs. Generative Replay:

The fine-tuning procedure (above two bullet points) remains the same. There is just one difference: For G2D, we employed parameter-efficient fine-tuning techniques to improve the parameter efficiency aspect of our method:
* G2D uses a parameter-efficient fine-tuning method (i.e., LoRA) 
* Generative Replay uses the same full fine-tuning method as G2D (Full FT). 

For generative replay, we opted against employing parameter-efficient fine-tuning (LoRA) techniques due to its inferior performance compared to full fine-tuning. The drawback arises from the sequential learning of multiple domains using limited trainable parameters in LoRA, which adversely impacts performance. Therefore, we presented the *optimal performance of Generative Replay*, achieved through the full fine-tuning procedure, mirroring the approach used in G2D (Full FT). We apologize that this might have not been clear. We have updated the draft to clarify this point (see Section 5.2 and Appendix C.3). Thank you for raising this clarification point!

>  In section 5.4. it is stated “we fine-tune only 1.04 ~ 2.5% of trainable parameters”. How was this percentage decided? How is it decided which parameters are fine-tuned? 

The number of trainable parameters (so-called “update matrices”) and which of them are fine-tuned is controlled by (1) the dimension used by the LoRA update matrices (i.e., r) and (2) the scaling factor (i.e., lora alpha), which gives the flexibility to balance a trade-off between end performance and parameter efficiency [2]. For our vision experiments, we set the default values for both hyperparameters (r = 16, lora alpha = 16). For our text experiments, we set the rank and scaling factor to 32 to match the performance of parameter-efficient finetuning with full finetuning on a single domain. These parameter choices result in 1.04 ~ 2.5% of trainable parameters.

Reviewer comment (Theory):
On p5 towards the bottom, the authors claim that ER with a limited buffer size is an upper bound for generative replay. This does not seem correct.

Author response:
>  On p5 towards the bottom the authors claim that ER with a limited buffer size is an upper bound for generative replay. This does not seem correct.

Thanks for catching this. We meant to clarify that Experience Replay (ER) with a lot of samples (not “limited buffer size”) is the upper bound. Indeed, Experience Replay (ER) with “limited buffer size” is not the upper bound. We have made this revision in the draft (see Section 4.2).

Reviewer comment (Reproducibility):
Most importantly, the authors should provide full details regarding how the generative replay experiments were implemented in order for the reviewers to be able to judge whether the key comparison of this paper was performed in a fair manner.

Author response:
Thank you very much for your time and thoughtful feedback! We have now made the following changes which we believe address all your questions and strengthen the paper

* Added a section on implementation details of Generative Replay in Appendix C.3
* Added a summary of these clarifications to Section 5.2 with a reference to Appendix C.3
* Updated section 4.2 to correct description about ER (see below) 

 > ... does not provide enough experimental details to properly assess whether the comparison between G2D and Generative Replay is performed in a fair manner…it is unclear to me whether Generative Replay has been implemented in an optimal manner

We have updated the draft with the following clarifications (see Section 5.2  and Appendix C.3). Further, we plan to release all code regarding our implementation of both G2D and Generative Replay, for the community. 

We sought to make the comparison fair by (a) using the same replay buffer (i.e., fixing to the same set of synthetic data) for both methods and (b) using the *same* fine-tuning approach for classifier models (see below):

Reviewer comment (Experiments):
Could the authors explain why they took the S-iPrompts results from the Wang et al. (2022a) paper, but not the S-liPrompts results? (Repeated from Weaknesses section)

Author response:
> Could the authors explain why they took the S-iPrompts results from the Wang et al (2022a) paper, but not the S-liPrompts results?

As noted in Section 4.2 (end of third paragraph), for a fair comparison, we fix the model backbone of the downstream classifiers as ViT-B16 pretrained with ImageNet checkpoint. This is the same across all baselines and our methods. S-liPrompts uses a different pretrained backbone in the CLIP family [3]. Thus, we compare with the S-Prompts method (S-iPrompts), which uses the same pre-trained architecture.

Reviewer comment (Experiments):
How does this approach scale? My understanding is that it does so poorly if the task discriminator is not trained continually.

Author response:
As per your concern, we performed an additional experiment (added to Appendix B, Table 5 with a reference in the main text in section 3.2), where we train the task discriminator in a continual manner. Note that this turns the learning of domain discriminator into a class-incremental continual learning problem, introducing a new challenge on top of our original domain incremental learning problem.

We find that the performance impact of continual training of the discriminator is minimal, implying that we *can continually train our discriminator model*. To assess the continuous training of the discriminator, we selected the dataset with the most number of domains, which is the CORe50 benchmark with a total of 8 different domains. This results in the performance of 88.13 ± 0.01 (averaged over 5 random seeds), while our original implementation results in 89.11 ± 0.30 (see Table 5). Thus. this continual fine-tuning of the domain discriminator results in a performance drop of less than 1 point. Therefore, our conclusions remain the same in that we *still* outperform existing state-of-the-art baselines by a substantial margin.

Reviewer comment (Theory):
More generally, it seems that the authors don't quite understand the computational efficiency related to PEFT approaches; taking LoRA for example, the computational cost saved from not performing a gradient update step on the full parameters is quite small compared to the cost of having to compute forward and backward passes in the model. The "gains" from PEFT are really in parameter efficiency and serving of these models.

Author response:
Our focus in elaborating on the LoRA fine-tuning techniques was to touch on parameter efficiency and not computational efficiency. While we have been careful to describe these methods as “parameter-efficient”, we realize there is a reference to “computational efficiency” in section 4.3 which can potentially confuse the reader. We apologize for this phrasing. To address this, we have updated the draft.

Reviewer comment (Reproducibility):
T5 is an encoder-decoder model, thus enabling conditional generation. How are you generating synthetic data from this model, i.e. where is the data fed to the encoder coming from? Do you have a separate generator for this?

Author response:
In Appendix C.2, we have listed implementation details about the T5 generator model. For our generator, we use the prompt tuning (Lester et al., 2021) to learn the parameter-efficient models. We use the pre-trained T5-Large v1.1 checkpoint adapted for prompt tuning as the backbone (Raffel et al., 2020), and the prompt embeddings are initialized randomly. We set the prompt length to 400 tokens which accounts for 819K trainable parameters, i.e., around 0.1% in comparison to 784M frozen T5-Large parameters. During training, we input a special token into the model and conditionally generate a document content, question, and answer, all separated by the special tokens. Our maximum sequence length is set to 512, and we truncate the document content after tokenizing the question-answer pair. During the generation process, we provide multiple text prompts along with the special input token and learned prompt embeddings. We do not use any generator to sample our input text prompts and consider the following text prompts to conditional generate synthetic samples – “Generate article, question and answer.”, “Generate context, question and answer.”, “Generate answers by copying from the generated article.”, “Generate factual questions from the generated article.”. During generation, we use ancestral sampling, which selects the next token randomly based on the model’s probability distribution over the entire vocabulary, thereby reducing the risk of repetition. We generate samples with a minimum length of 50 tokens and a maximum of 1,000 tokens, retaining only those samples that contain exactly one question-answer pair with the answer included in the generated document content.

Reviewer comment (Experiments):
Is the classifier at task t fine-tuned from task t - 1? Or fine-tuned from the pretrained model?

Author response:
The classifier is continually fine-tuned from task t-1.