Paper: IBCL: Zero-shot Model Generation for Task Trade-offs in Continual Learning (ICLR-2024)

Reviewer comment (Presentation):
The paper itself is too dense - a lot of the crucial intuition and motivation is moved to the appendix, which makes the main paper very difficult to parse. For example: separation and placement in literature of the proposed method IBCL and MAML/BMAML (App. A), the reason for working in a bayesian CL setting (App. B), the importance of the particular task similarity assumption (App. E), limited discussion on how preferences are formalized (App. G), and very basic experimental details on the CL experiments (App. J). Without continuously looking at the supplementary, understanding both key elements of the method and its motivation become in parts near impossible. This specifically refers to the reason behind bayesian CL, and details regarding the made assumptions. It would be great to see that changed.

Author response:
We appreciate your thoughtful feedback, and we invite you to read our revised manuscript, which significantly improves readability and addresses your concerns. 

“The paper itself is too dense.”

We have improved readability in our revised manuscript. Please also refer to our comment to everyone for a list of our modifications.

Reviewer comment (Novelty):
The proposed setup seems contrived - in particular the continual aspect, with both examples provided in the introduction primarily highlighting the multi-task nature of the problem. The continual aspect seems to be mostly a sidenote (e.g. "preference may even change over time"). Similarly, the authors only provide limited references for the relevance of the described problem scenario where a large number of preferences has to be accounted for continually over time. It would be great if the authors could offer some more clarity here.

Author response:
“The proposed setup seems contrived.”

Our setup is simplified and clarified in the revised manuscript. Please read our revised version of the Introduction for a better structured story.

Reviewer comment (Experiments):
The authors list poorly performing models to also be sampled from the HDRs, and require separate evaluation on a withheld validation set. This seems like a rather crucial point to elaborate on - what exactly is the rate of poorly performing models, and how dependent is ICBL on selection using validation metrics?

Author response:
“Poorly performing models can also be sampled from HDRs.”

We are able to completely avoid sampling these models by modifying the hyperparameter alpha. Please refer to our ablation studies in Appendix H.3 of the revised manuscript, where we are able to always sample near Pareto-optimal models with a small alpha (0.01). In fact, avoiding poor models does not depend on validation at all, and we made this correction in our latest manuscript.

Reviewer comment (Experiments):
The experimental studies are limited, and only compare to a single continual learning method, while not providing any context as to why GEM was selected in particular. Are other methods not suitable for this scenario? Similarly, can ICBL not be deployed on standard, single-preference continual learning benchmarks? This would be great to understand, and if not, why.

Author response:
“The experimental studies are limited.”

We have added experiments based on the suggestions of reviewer 79Li. As in our revised manuscript, GEM and A-GEM are selected because they are rehearsal-based, and there is an established method to specify task preferences on rehearsal-based continual learning – that is, using preferences as weights on loss functions. The other methods, such as prompt-based continual learning, have no standard way to specify a task preference so far. In our revised experiments, we made an attempt to use a preference-weighted sum of prompts in L2P, but it ended up with poor performance except for the CelebA benchmark.

“Can IBCL be deployed to single-preference benchmarks?”

Yes. We can request IBCL to generate models for any number of preferences (including one preference only), and the total training overhead (measured in number of batch updates) remains constant, regardless of the number of preferences requested.

Reviewer comment (Theory):
What drives the definition of the Assumption 1 for Task Similarity? In particular, how close it is to a realistic assumption (in particular F being a convex subset of \(\Delta_{xy}\))? Conceptual motivation for such a key assumption would make it easier to grasp the proposed approach quicker.

Author response:
“What drives the assumption of task similarity?”

Task similarity assumption is needed to mitigate the possible model misspecification, which in turn could lead to catastrophic forgetting even when Bayesian inference is carried out exactly. This is now explicitly mentioned after presenting Assumption 1, and discussed in more depth in Appendix E.

Reviewer comment (Theory):
Intuitively, it seems like the diameter of the convex subsets F (r) could connect to the expected continuous distribution shifts that can be handled. Is that right? And generally, how does the choice of r drive/change the behaviour/applicability of ICBL?

Author response:
“Does the diameter of F(r) connect to the expected continuous distribution shifts that can be handled? How does the choice of r affect the applicability of IBCL?”

Yes: assuming a bounded diameter implies that the tasks are somehow close to each other. In other words, the distribution shift between the distributions for task i and j, p_i and p_j respectively, must not be too pronounced. If we were to allow arbitrary differences between the tasks the agent needs to accomplish, as a consequence of IBCL being embedded in a Bayesian framework, we may incur catastrophic forgetting. As pointed out in Appendix E, in the future, we will generalize IBCL to ProtoCL instead of Bayesian learning to let go of Assumption 1. Having a bounded diameter does not limit the applicability of IBCL, since in the majority of applications the tasks we want to accomplish are reasonably similar to each other. This is for example the case of classification, where Assumption 1 should always be satisfied, with a reasonably small diameter r.

Reviewer comment (Experiments):
Since the proposed method argues the benefit of zero-shot, it needs to compare with the models that have zero-shot capabilities. CLIP [1] is recommended as a baseline. If CLIP is considered unfair for comparison, it is also acceptable to compare with traditional zero-shot learning techniques.

Author response:
We indeed believe CLIP [3] is a well-established zero-shot approach. However, it requires natural languages as prompts, which is unavailable in our settings. Therefore, we choose L2P, where prompts are trainable parameters.

Reviewer comment (Experiments):
There are several continual learning papers with low training cost. Many such papers have been published recently [2, 3, 4], and including them as baselines would help validate the effectiveness of the proposed method.

Author response:
We appreciate your thoughtful comments, and we invite you to read our revised manuscript, which significantly improves readability and compares to two more baselines: A-GEM [1] and L2P [2]. Due to time constraints, we are unable to finish working on the other baselines you suggested, but we will keep working on them.

Our new experiments show that IBCL is able to obtain high continual learning performance, thanks to its probabilistic Pareto-optimality. Moreover, it is able to maintain a constant training overhead per task, regardless of how many preferences are requested. Compared to rehearsal-based baselines, like GEM and A-GEM, the performance is not worsened while the training overhead is significantly reduced. 

Although prompt-based baselines, like L2P, also achieves constant training overhead per task, IBCL has a much higher performance overall. This is due to the fact that there is no standard way of specifying preferences over tasks to prompt-based methods, and we made an attempt to train one prompt per task, and use a preference-weighted sum of the learned prompts to specify a preference. The resulting performance is poor except for on the CelebA benchmark. As a consequence, we point out how specifying task preferences in prompt-based continual learning can be a future research direction.

Again, we appreciate your comments that helped us improve the overall quality of the manuscript, and we sincerely hope you can update the rating accordingly.

Reviewer comment (Experiments):
There is advanced work after the publication of GEM. Since the authors selected GEM as a baseline, it would be better to also compare the method with A-GEM [5].

Author response:
We appreciate your thoughtful comments, and we invite you to read our revised manuscript, which significantly improves readability and compares to two more baselines: A-GEM [1] and L2P [2]. Due to time constraints, we are unable to finish working on the other baselines you suggested, but we will keep working on them.

Our new experiments show that IBCL is able to obtain high continual learning performance, thanks to its probabilistic Pareto-optimality. Moreover, it is able to maintain a constant training overhead per task, regardless of how many preferences are requested. Compared to rehearsal-based baselines, like GEM and A-GEM, the performance is not worsened while the training overhead is significantly reduced. 

Although prompt-based baselines, like L2P, also achieves constant training overhead per task, IBCL has a much higher performance overall. This is due to the fact that there is no standard way of specifying preferences over tasks to prompt-based methods, and we made an attempt to train one prompt per task, and use a preference-weighted sum of the learned prompts to specify a preference. The resulting performance is poor except for on the CelebA benchmark. As a consequence, we point out how specifying task preferences in prompt-based continual learning can be a future research direction.

Again, we appreciate your comments that helped us improve the overall quality of the manuscript, and we sincerely hope you can update the rating accordingly.

Reviewer comment (Writing):
The HDR concept (highlighted in purple in Fig.1) is not immediately clear. It would be more helpful to use its full name, 'high density region'. The term 'finitely generated credal set' presents a similar issue, needing a more explicit definition.

Author response:
“The HDR concept is not immediately clear.”

We modified Figure 1 as well as the paragraph right after it. In Figure 1, the full names of FGCS and HDR are provided. In the paragraph, we give a more detailed description about what FGCS and HDR are. We also added a reference to Section 2 at the end of the paragraph, to show where formal definitions can be found.

Reviewer comment (Novelty):
While the paper mentions several preference-conditioned Pareto models [1, 2] in the appendix, a more detailed explanation of how IBCL differs from these models would be beneficial.

Author response:
“A more detailed explanation of how IBCL differed from [1] [2] would be beneficial.”

We detailed this explanation in the last two paragraphs before “Contributions” in the Introduction. That is, compared to existing efficient knowledge transfer methods like [1], IBCL enables learning models under specified task trade-off preferences. Compared to existing methods that enable specified preferences like [2], IBCL is not only much more efficient by maintaining a constant training overhead per task, but also guarantees probabilistic Pareto optimality. Moreover, IBCL is a Bayesian CL method, hence it mitigates catastrophic forgetting by design. We prove this and discuss Bayesian CL at length in Appendix A.

Reviewer comment (Experiments):
In the experiments, it appears that a smaller $\alpha$ value is preferable. Why not choose an even smaller $\alpha$ (e.g., 0.001)?

Author response:
“Why not choose a smaller alpha, e.g. 0.001?”

Indeed, a smaller alpha is more preferable. However, when alpha becomes very small, like 0.01, we already have a very high probability of sampling the Pareto-optimal model, and this probability can be hardly improved by using an even smaller alpha. For example, in Figure 2, the shaded blue area that illustrates IBCL’s performance range has already converged to a single curve. This means that the worst sampled model and the best sampled model (Pareto-optimal model) have a negligible difference. Therefore, there is no need to keep shrinking down this difference by using a smaller alpha.

Reviewer comment (Experiments):
The paper is motivated as we have a large number of users, and the model is scalable for the larger number, but the results are shown only for the 5/10 task, which is small and does not align with the motivation.

Author response:
**2. “The paper is motivated as we have a large number of users, and the model is scalable for the larger number, but the results are shown only for the 5/10 task, which is small and does not align with the motivation.”**

The number of tasks and the number of preferences are two different concepts. We can have a small number of tasks, but with each task we have a large number of preferences. These two numbers are distinguished by $n_{pref}$ and $i$ in our Table 1.

Reviewer comment (Experiments):
The baseline papers are outdated; the recent work shows much better results even without replay samples.

Author response:
**3. “The baseline papers are outdated.”**

We have explained the baseline selection in Section 5.1, as well as an explanation in the last paragraph before Figure 1 in Section 1 (“However, we identify two major disadvantages …”). That is, so far, people can only specify preferences over tasks on rehearsal-based continual learning algorithms, by using a preference-weighted sum on the losses of the rehearsal memory, making them comparable to IBCL. The "new" and “efficient” algorithms, such as model-based or prompt-based algorithms, do not enable an input of preferences. However, we still made an attempt to input preferences to L2P after the discussion with reviewer 79Li. This is a relatively new prompt-based method for comparison.

Reviewer comment (Experiments):
The ablations are convincing, but the results are insufficient to evaluate the model. The used datasets are limited, and the training procedure is not clear.

Author response:
**4. “The ablation results are insufficient to evaluate the model.”**

We would appreciate the reviewer to point out how to make the ablation studies sufficient. We conduct these ablations by using different hyperparameters $d$ and $\alpha$ and the results already show a clear trend of how the performance can be affected. The training procedure is exactly the same as in the main experiments, which is detailed in Section 5.1 and Appendix H, just with different hyperparameters given.

Reviewer comment (Experiments):
The claim in the motivation is not supported in the experiments, lack of motivation about Pareto-optimality, weak/unfair baselines, and the computation of the HDR are not discussed, which is key in the paper.

Author response:
**5. “The claim in the motivation is not supported in the experiments.”**

Our experiments already show that IBCL can achieve high continual learning performance metrics, supporting the claim of probabilistic Pareto-optimality. We have also analyzed its overhead in Table 1, supporting the claim of zero-shot model generation under preferences. We are not sure why the reviewer thinks the claim is not supported.

Reviewer comment (Experiments):
In Algorithm-1 paper shows the FGCS Knowledge Base Update, which is based on some distance, mostly selecting the samples that have maximum diversity. There are many similar works based on entropy, loss, and other metrics (please refer to [b]). What advantages do they have over the other? Most of the earlier work used fixed/constant size buffers, which is better than sublinear growth. When the results are evaluated, IBCL has sublinear growth; however, the compared method GEM/A-GEM uses a fixed-size buffer. In this scenario, how do the authors ensure a fair comparison? Also, the L2P is a replay-free model, which is not a fair comparison since the model used the replay buffer. What is the buffer growth rate, and how does the performance change with the sublinear growth?

Author response:
**8. "What advantages does one distance metric over another for similarity among distributions?"**

We have discussed the advantages of choosing the 2-Wasserstein metric in Appendix D. That is, we can obtain this distance efficiently by computing the distance among two distributions’ parameters, when the two distributions are Gaussian.

**9. "How do the authors ensure a fair comparison between rehearsal-based methods and IBCL (in terms of buffer growth)?"**

We would like to correct the reviewer that rehearsal-based methods, like GEM and A-GEM, do not have fixed-size buffers. Instead, they use **fixed-size buffers per task**, i.e., they cache a number of data points per task, and the total buffer size grows along with the number of tasks. This can be found in Section 3 of the GEM paper (Lopez-Paz and Ranzato 2017, as cited in our paper). Therefore, both rehearsal-based and IBCL grow their buffer along the number of tasks, leading to a fair comparison.

**10. “ The L2P is a replay-free model, which is not a fair comparison since the model used the replay buffer.”**

Please refer to our discussion with reviewer 79Li. We added this new baseline in our revised manuscript because we have been criticized for not comparing IBCL to “more efficient methods”, such as prompt-based methods. Indeed, there is no established way to specify a preference over task to L2P and other prompt-based methods, and we only made an attempt to do so, as explained in Section 5.1. The buffer growth rate is linear for L2P, since it trains more prompts as we have more number of tasks, but this coefficient of linear growth is very small, because each prompt only takes a very small memory. We do not quite understand “how does the performance change with the sublinear growth”, because L2P does not have a sublinear growth. We will be grateful if the reviewer can clarify this question.

Reviewer comment (Experiments):
The baseline papers are outdated; the recent work shows much better results even without replay samples. Please include the recent replay-based model in the baseline. Also, the L2P (is only a recent model) is a replay-free prompting-based model and there are many updated prompting-based approaches [a, c], etc. which should be included in the baseline.

Author response:
1. More rehearsal-based baselines with fixed buffer size need to be compared.

We thank the reviewer for pointing these papers out. A fixed buffer size is not comparable to our approach, because there is no established methods to specify preferences on these methods. When data from different tasks are all cached, established methods (like Lin et al. 2019) specify a preference by using a weighted sum of losses on these data, where the weight being the preference. However, with fixed buffer size that does not remember which task each data point belongs to, there is no established method to specify this weight. The goal of our experiment that compares to GEM, A-GEM and VCL is to show that IBCL has advantages over **methods with established ways to specify preferences**. 

We are willing to make some attempts to specify preferences to the new baselines you recommended, like what we have done to L2P. Although we are afraid the design and experiment cannot be finished before December 1, we are willing to try.

Reviewer comment (Theory):
The motivation behind the probabilistic Pareto-optimality is not clear. Why is it important for continual learning?

Author response:
**6. “ Lack of motivation about Pareto-optimality.”**

Pareto-optimality is a common goal in multi-objective optimization, which is the nature of multi-task and continual learning. This motivation is already well supported by the cited previous work. Please refer to Kendall et al. 2018, Sener and Koltun 2018, Lin et al. 2019 and Lin et al. 2020, cited in our introduction section.

2. The Pareto-optimality motivation for CL needs to be clarified. How does Pareto-optimality in continual learning differ from multi-task learning in general? 

The motivation for Pareto-optimality in CL is the same as for generic multi-task learning. As explained in our section 1 and appendix C, CL is only a special case of multi-task learning, and therefore it has a performance trade-offs among tasks. The motivation for Pareto-optimality is already well-established in the cited papers, such as Lin et al. 2019. Consider there exist multiple tasks, we certainly don't want our learned model is strictly worse than any other model on all tasks, i.e., we want Pareto-optimality. This is the same motivation for optimality in single-task learning.

Reviewer comment (Reproducibility):
There are no clear descriptions about the HDR computation, i.e., how Algo-2, line-5 computes the HDR? It looks like Preference HDR Computation is expensive, and as the task grows, the complexity increases. Please discuss its computation method and complexity.

Author response:
**7. “The computation of the HDR are not discussed.”**

The computation of HDR is an established standard procedure, which we explain right after Theorem 4 and cite Juan et al. 2022 for an R package that implements it. This is also not our focus in this paper. In practice, we use Gaussians to model the parameter distributions, and we select an interval, with the lower and upper bounds symmetrically on the two sides of the mean, such that the enclosed probability density function has an integral = 1 - $\alpha$. The two bounds are computed as the inverse cumulative density function (icdf) of a Gaussian at the quantiles of 0.5 - (1 - $\alpha$) / 2 and 0.5 + (1 - $\alpha$) / 2. A Gaussian’s icdf can be very fast computed by tools like scipy.

Reviewer comment (Experiments):
The ablations are convincing, but the results are insufficient to evaluate the model. The used datasets are limited, and the training procedure is not clear.

Author response:
**4. “The ablation results are insufficient to evaluate the model.”**

We would appreciate the reviewer to point out how to make the ablation studies sufficient. We conduct these ablations by using different hyperparameters $d$ and $\alpha$ and the results already show a clear trend of how the performance can be affected. The training procedure is exactly the same as in the main experiments, which is detailed in Section 5.1 and Appendix H, just with different hyperparameters given.