Paper: A Probabilistic Framework for Modular Continual Learning (ICLR-2024)

Reviewer comment (Novelty):
LMC also introduced a benchmark for compositional generalization based on colored-mnist that is not mentioned when introducing BELL.

Author response:
Thank you for pointing this out, we will change the text to mention the LMC sequence for compositional generalization.

LMC introduced a sequence of tasks for evaluating compositional generalization. In it, each problem is derived from MNIST and is defined by one background-foreground color selection and one MNIST label pair. This leads to a limited variety between problems -- input domains are always somewhat similar as they all use MNIST digits, and tasks are restricted to be distinguishing between MNIST classes.

This sequence appears to be the most similar to BELL’s S^few which evaluates a model’s ability to achieve few-shot transfer by recomposing previous knowledge, with S^few involving disparate input domains and output tasks.

Reviewer comment (Presentation):
This work introduces a method and a benchmark, however most information about the benchmark is left in the appendix. I suggest providing simpler versions of the Algorithms in the main text, and to use the space to make the paper more self-contained (less dependent on the Appendix).

Author response:
Thank you for the suggestion, we will add more information and motivation about the benchmark suite to the main text.

Reviewer comment (Novelty):
There exists a vast number of continual learning benchmarks. Although they enrich the field, they also dilute the efforts of the research community. Thus I suggest the authors to include some more motivation on why BELL is needed and why researchers should use it rather than other benchmarks or tasks.

Author response:
Other CL benchmarks, such as split-CIFAR and CLEAR[2], can differ by the continual learning setup they assume (e.g. class-incremental, domain-incremental etc.). As a result, their sequences implicitly simultaneously evaluate a subset of CL properties, e.g. plasticity, stability and perceptual transfer, which are all summarized by a single number --- the final average accuracy. This makes it difficult to determine which of these properties is lacking from a CL algorithm.

It is important that the benchmarks which we use be able to measure whether algorithms have specific CL properties, like those we list in Table 4. This would guide future research in the field by making the desiderata explicit and facilitating the comparison of different CL algorithms against it.

CTrL [1] is a step in this direction. However, it evaluates perceptual transfer between two problems by shuffling the original problem’s labels, and it evaluates latent transfer between MNIST images with two different background colors. As such, CTrL cannot evaluate perceptual and latent transfer between disparate tasks and input domains. \
BELL is a further step in that direction, which addresses CTrL’s shortcomings by creating sequences of compositional problems. The problems’ compositionality allows us to create sequences which evaluate perceptual and latent transfer across disparate tasks and input domains. It also allows us to create more challenging sequences for evaluating perceptual transfer --- S^out* and S^out**. BELL also introduces new sequences which evaluate new CL properties --- S^few which evaluates few-shot transfer and S^sp which evaluates latent transfer across different input spaces.

Overall, BELL should be used by researchers in order to diagnose the CL properties of a CL algorithm, which would improve our ability to compare different approaches.

Reviewer comment (Experiments):
Would it be possible to run PICLE on compositional color MNIST to be able to compare with LMC (see LMC paper)? (It is ok if you do not have enough time / compute resources to do it).

Author response:
We agree that these results would be nice to have. It's unlikely that we will have time to get these results by the end of the discussion period, but we will try and report back if we manage.

Reviewer comment (Theory):
Why $\{\pi}^*$ is not used in Algorithm 2?

Author response:
In Algorithm 2, $\mathbf{\pi}^\*$ is the given list of previous solutions. Our search through PT paths does not need it, which is reflected in it being unused in Algorithm 2. We currently have $\mathbf{\pi}^\*$ as an input in order to keep the inputs to Algorithms 2 and 3 the same. However, we now appreciate that this can cause confusion and will remove $\mathbf{\pi}^\*$ from the inputs of Algorithm 2.

Reviewer comment (Evaluation):
On page 3, the authors mention their strategy is based on a generative model of the input x. How is the generative quality of the proposed method quantitatively? Some further evaluation of the proposed method using metrics like ECE can thus be more insightful.

Author response:
**q1: Quantitative evaluation of the generative quality of the proposed method.**

For our purposes, the employed generative model needs to be useful for distinguishing between inputs sampled from different distributions. As a result, we have not explored the generative quality of the samples. Given our use of efficient Gaussian approximations, we expect that it is poor. Instead, in Appendix I, we have provided a quantitative evaluation of our method’s ability to distinguish between different input domains. The results indicate that using random projections improves the discriminative capabilities of our generative model.

Reviewer comment (Experiments):
While I am not very familiar with the up-to-date modular continual learning literature, the baselines in Tables 1-2 look classic to me. Can the authors comment on comparing with more recent works?

Author response:
**q2: Baselines in Tables 1-2  appear classic. Can the authors comment on comparing with more recent works?**

Our work aims to improve modular CL methods. Therefore, we compare to the latest state-of-the-art modular CL algorithms, namely: MNTDP-D [1] (2020) and LMC [2] (2021). We also compare to an additional modular CL algorithm -- HOUDINI [3].
For completeness, as done in previous modular CL work [1, 2], we use O-EWC [4, 5] as a classic regularization-based CL algorithm, and ER [6] as a replay-based method, which was shown to have competitive performance despite its simplicity.

Tables 1-2 do not report the performance of LMC since it performed poorly, despite our efforts to adjust it to the BELL benchmark suite.

Reviewer comment (Evaluation):
Can the authors compare the computational overhead of their method against the baselines?

Author response:
**q3: Can the authors compare the computational overhead of their method against the baselines?**

Figure 3a of our submission compares the computation demand of PICLE to those of the baselines, evaluated in FLOPs. It can be seen that since PICLE uses slightly more resources than MNTDP-D since it considers a larger search space. However, it can also be observed that PICLE’s computational requirements scale well with the number of solve problems.

Reviewer comment (Evaluation):
The table in the paper lacks uncertainty metrics such as standard deviation. This omission should be addressed to enhance the study's reliability. Additionally, for readability purposes, it would be better to show the percentages only up to three digits instead of four (e.g., XX.XX% should be changed to XX.X%).

Author response:
**w1a: Lack of uncertainty metrics in the table.**

As the performance of the different algorithms on the short sequences of the CTrL benchmark is similar, we added the standard deviations of the results shown in Table 3. This can be found in Table 6, Appendix K.1. Does this address your concern?

**w1b: Show percentages for up to 1 decimal point.**

We agree that this would improve the paper’s readability and will make the changes.

Reviewer comment (Evaluation):
The paper should clearly state that the PICLE method is task-aware, which is an important limitation. Ideally, there would be a column in Table 4 that discusses task-agnosticism, a feature that another algorithm, LMC, is capable of.

Author response:
**w2: The paper should clearly state that the PICLE method is task-aware**

We agree that this is an important property, which provides an interesting distinction between the algorithms. Therefore, we will mention it and add it as a row in Table 4.

Reviewer comment (Novelty):
The concept of using a generative model to approximate latent activations and assume local independence from one layer to the next, as introduced in Section 4, was initially proposed in the LMC paper. The authors should give credit to this paper and clarify how their methods differ from those originally proposed in the LMC paper.

Author response:
**w3: Credit to and better distinction from LMC.**

We have credited LMC with a similar approach for perceptual and few-shot transfer in the Related Work section, while also outlining the differences. However, we agree with the wording proposed by the reviewer, and will also add it to Section 4 for better visibility. Moreover, we acknowledge that the conceptual differences can be better presented and will amend the text accordingly.

The main conceptual difference is that while LMC also uses a module-specific generative model, PICLE unites the generative model for each layer using a single probabilistic model, which brings numerous advantages. First, our probabilistic model allows us to define a prior over the choice of modules. Second, by changing the probabilistic model, we can incorporate further assumptions in a principled way, for example, we can put a prior over the choice of pre-trained modules for multiple layers, rather than layer-specific priors.
Third, the probabilistic model allows us to compute the posterior over the choice of pre-trained module for each layer collectively, rather than individually.

The main implementation difference is that PICLE utilizes a different approximation of a module’s input distribution, using orders of magnitude fewer extra parameters per pre-trained module than LMC.

Reviewer comment (Evaluation):
The quantitative advantage over MNTDP-D in accuracy is marginal on average ~1-3%.

Author response:
**w1: The quantitative advantage over MNTDP-D in accuracy is marginal on average ~1-3% \
q1: “Why are the numbers for MNTDP-D and PICL forward transfer so similar?”**

Since PICLE and MNTDP-D share many CL properties, apart from few-shot transfer and latent transfer, their performance --- including accuracy and forward transfer --- on the short sequences is similar, apart from on S^few, S^in, S^sp.

In Table 1, the accuracy is averaged over mostly shared CL properties (apart from few-shot transfer). PICLE’s advantage is only apparent by its superior performance on the last problem of S^few and S^out**. As a result, the average accuracy in Table 1 averages over 42 problems, while PICLE achieves significant improvement over MNTDP-D on 2 of them. This leads to the similar average accuracy which we observe. Similarly, the short sequences of CTrL mostly evaluate CL properties which are shared between MNTDP-D and PICLE, leading to PICLE outperforming MNTDP-D only on the last problem of S^in, and in turn leading similar average accuracies in Table 3.

PICLE’s advantage over MNTDP-D is that it can achieve few-shot and latent transfer. Overall, PICLE achieves much higher transfer on the last problem than MNTDP-D on: perceptual transfer (S^out**, Table 1, **+10.3**), few-shot transfer (S^few, Table 1, **+34.65**), latent transfer (S^in, Table 2, **+12.58**), (S^sp, Table2, **+23,65**), (S^in, Table 3, **+16,89**).

Reviewer comment (Evaluation):
The method requires slightly more FLOPs than MNTDP-D (Figure 3 (a)).

Author response:
**w2: The method requires slightly more FLOPs than MNTDP-D (Figure 3 (a))**

This is correct. Figure 3(a) shows that PICLE requires slightly more FLOPs, while considering a much bigger bigger search space. The figure also shows that the computational demands of both PICLE and MNTDP-D scale well with the number of problems.

Reviewer comment (Evaluation):
Table 1: Why are the numbers for MNTDP-D and PICL forward transfer so similar?

Author response:
**w1: The quantitative advantage over MNTDP-D in accuracy is marginal on average ~1-3% \
q1: “Why are the numbers for MNTDP-D and PICL forward transfer so similar?”**

Since PICLE and MNTDP-D share many CL properties, apart from few-shot transfer and latent transfer, their performance --- including accuracy and forward transfer --- on the short sequences is similar, apart from on S^few, S^in, S^sp.

In Table 1, the accuracy is averaged over mostly shared CL properties (apart from few-shot transfer). PICLE’s advantage is only apparent by its superior performance on the last problem of S^few and S^out**. As a result, the average accuracy in Table 1 averages over 42 problems, while PICLE achieves significant improvement over MNTDP-D on 2 of them. This leads to the similar average accuracy which we observe. Similarly, the short sequences of CTrL mostly evaluate CL properties which are shared between MNTDP-D and PICLE, leading to PICLE outperforming MNTDP-D only on the last problem of S^in, and in turn leading similar average accuracies in Table 3.

PICLE’s advantage over MNTDP-D is that it can achieve few-shot and latent transfer. Overall, PICLE achieves much higher transfer on the last problem than MNTDP-D on: perceptual transfer (S^out**, Table 1, **+10.3**), few-shot transfer (S^few, Table 1, **+34.65**), latent transfer (S^in, Table 2, **+12.58**), (S^sp, Table2, **+23,65**), (S^in, Table 3, **+16,89**).

Reviewer comment (Writing):
Typo: “The CL algorithm should be able to ”remember” previous problems“ (backwards quotation).

Author response:
We thank the reviewer for their feedback. We will correct the reported typos.

Reviewer comment (Writing):
“This search results in the most relevant previous solution $\pi'$. Finally, in lines 11-14, we evaluate NT paths created by transferring a different number of the last $\ell \in \\{\ell_{min} + 1, ..., L − 1\\}$ layers of $\pi'$, to see if re-using more layers leads to further improvement”. (do you mean $\ell \in \\{2, \ldots L - 1\\}$? That's what the code appears to be searching over).

Author response:
**q2: “Algorithm 3 - Do you mean $\ell \in \{ 2, ..., L-1 \} $?”**

The $\ell$ defined on line 11 of Algorithm 3 refers to the index of the first pre-trained module. On line 12 we use it to extract the pre-trained modules $\pi’[\ell : L]$. For $\ell = \ell_{min}$ , the length of the pre-trained suffix $\pi’[\ell_{min} : L]$ is $\ell_{min} + 1$. And for $\ell = 2$, the length of the pre-trained suffix $\pi’[2 : L]$ is $L-1$. Therefore, we indeed attempt to transfer ${\ell_{min}+1, ..., L-1}$ pre-trained layers.
However, thanks to your comment, we now see that it is confusing to use $\ell$ inside the text to denote length while simultaneously using it in Algorithm 3 to denote an index. We will adjust the text accordingly.