Paper: Sequential Bayesian Continual Learning with Meta-Learned Neural Networks (ICLR-2024)

Reviewer comment (Theory):
One of the main ideas of the paper is the use of distributions from the exponential family to accumulate knowledge from the past without increasing the parameters needed. The Fisher-Darmois-Koopman-Pitman theorem implies that distributions from the non-exponential family need sufficient statistics of unbounded dimensions to fully capture the information of the data. While the authors acknowledge that models only based on exponential family update are often too simple, they do not address directly what are the consequences of this simplification in their proposal. Probably the use of a strongly nonlinear model (as a neural network) is sufficient to obtain the same learning power as a model that is not limited to exponential family distribution. In any case it would be interesting to directly address this point.

Author response:
## The Price of Lossless CL with Exponential Family Posteriors

Compliant with the "no free lunch" theorem, we do pay a price for harnessing the power of exponential family posteriors: the necessity of meta-training. Due to the simplicity of the exponential family, meta-learned neural networks are vital for handling complex and high-dimensional problems. On the other hand, SGD can be applied to standard CL scenarios, although it requires additional techniques to mitigate forgetting. We made this clear in the second paragraph of section 3.3 in the updated draft.

However, as we mentioned in the introduction, MCL is a promising direction toward solving CL. Once we embrace the MCL setting, SB-MCL arises as an attractive solution with minimal downsides. In theory, limiting the inner loop updates to the exponential family posterior should not have a significant impact on the representational capacity, as long as the meta-learned neural networks fulfill their duty as universal function approximators. Our experiments also empirically verify that the meta-learned neural networks are sufficient for SB-MCL to outperform SGD-based MCL approaches.


### The Capacity of Exponential Family Posteriors

We find multiple reviewers are concerned about the capacity of exponential family posteriors. The question "Are we losing something by choosing this family to accumulate past knowledge?" seems to be in line with such concern. Despite the simplicity of the exponential family, a high-dimensional exponential family latent variable can contain a huge amount of information since the capacity increases exponentially to the number of dimensions. Therefore, even a single 512-dimensional Gaussian variable can hold a sufficient amount of information to solve our benchmarks.

VAE [1] is a good example that demonstrates the capability of an exponential family latent variable. Another example is GAN [2]. Both of them map a simple Gaussian random variable to real-world images.

While we mentioned the necessity of meta-training as the price to pay, we currently do not see any significant downside of SB-MCL when compared with SGD-based MCL approaches. We think it can be an interesting research topic to find and analyze weaknesses of SB-MCL.

Reviewer comment (Theory):
The sequential Bayesian update with the exponential family has results completely equivalent to having all examples available at the same time. This would solve “by definition” the problem of forgetting. The idea is very interesting, but it should be specified that this is possible due to a simplification assumption on the distribution, and exploring what the consequences of this assumption are. Claiming that this method transforms a CL problem in an underfitting problem should be verified empirically by varying the number of parameters of the neural networks and/or the dimension of the latent space, showing how the memorization ability changes. More importantly, this claim is somehow in contradiction with the “losslessness” of the exponential family sequential update, implying that, with the increase of tasks, the number of parameters should increase.

Author response:
## Underfitting and the Lossless Sequential Update

As neatly summarized in the review, our framework "by definition" produces the same results regardless of how the training data is provided. Our claim about forgetting vs. underfitting was to highlight this fundamental property of our approach, not as an empirical observation. Even in standard multi-task learning settings where all tasks are available throughout training, the performance can degrade as the number of tasks increases, which is often described as underfitting. Since SB-MCL’s performance drop in a CL setting is exactly the same as the performance drop in the corresponding multi-task learning setting with the same tasks, we referred to it as underfitting. More specifically, the sequential Bayesian update is indeed lossless; however, information loss can occur (i) when the learner transforms the raw data to the variational likelihood and (ii) when the model forwards $x$ and $z$. Therefore, underfitting and lossless sequential updates do not contradict each other.

From this perspective, it becomes evident that there would be a positive correlation between the model size and the performance up to a certain point. If we reduce the number of parameters, the performance will surely drop. However, naively increasing the number of parameters does not necessarily lead to a better performance, which is why people do not use a gigantic multi-layer perceptron to solve all kinds of problems. We do need a clever architectural design to further improve the performance, and it would be an exciting topic for future research.


### The Diminishing Returns of Increasing the Model Size

Naively increasing the neural network size or the number of dimensions of the latent variable has diminishing returns. It is effective until a certain point, but after that, adding more parameters or dimensions does not lead to a meaningful improvement in performance. For instance, we chose the 512D latent variable because we did not see any significant performance gain beyond that (actually, even 256D was enough).

This is why we emphasized architectural improvement in our initial response and the paper. Simply increasing the number of parameters or latent dimensions would **not** solve CL. We can indeed show that decreasing the number of dimensions (e.g., to 64D or 32D) harms performance. But we are not sure if that would be meaningful since such a trend is universal in all kinds of architecture. If the reviewer still thinks adding such a graph would be helpful, we will certainly add it to the appendix.

Reviewer comment (Theory):
On a higher level, the approach (like any other meta-continual learning framework, in fairness) shifts the forgetting problem from the task to the meta task. Else said, the approach assumes that all meta non-stationarities are represented at meta-training time. If the meta-test phase characterizes by substantial non-stationarities the method is bound to fail as it cannot adapt (without incurring once again in catastrophic forgetting at meta level).

Author response:
## MCL as a Data-Driven Approach to CL

It is true that MCL frameworks transform the challenge of designing a good CL algorithm into the challenge of building a good meta-training set. The meta-training set should be large and diverse enough to cover the meta-test distribution. But is this really a weakness of MCL?

As we mentioned in the introduction, it has been proved that CL is fundamentally an NP-hard problem (Knoblauch et al. 2020). This theoretical result entails a profound message to the CL literature: no matter how meticulously a CL algorithm is designed, there are CL problems that it cannot solve efficiently. This is why we think MCL is a promising direction. Instead of manually designing domain-specific CL algorithms, we can design a general MCL algorithm and collect meta-training datasets for target domains. In this sense, MCL is also better aligned with [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) by Rich Sutton. It is an argument that general learning algorithms, which leverage more computation and data, have always won over complicated algorithms relying on domain-specific human knowledge in the long run. Therefore, we believe the data-driven aspect of MCL can be considered a strength, rather than a weakness.

Reviewer comment (Experiments):
While the experiments cover a wide array of tasks, it is not clear why the proposed model is not tested in all cases. It would be useful for additional clarity to have the same models tested in all possible cases. If the claim is that the proposed model is the generator of a family of specific models known in the literature, it should be tested anyway, showing if its generality can make an improvement. Is there any technical reason why the proposed general model is not tested in all scenarios?

Author response:
## Improved Experiments Section

We admit that the description of the experimental settings and the presentation of the results were not entirely clear. We have largely improved the overall experiments section in the updated draft, reflecting the comments of reviewer DFZf as well.

Since SB-MCL is proposed as a generic framework, there can be multiple instantiations of it depending on the formats of input and output. For image classification and simple regression, GeMCL/PN and ALPaCA are the corresponding instantiations. In these cases, the output formats are simple enough to be handled by GMMs or linear models, and the posterior predictive distribution can be computed analytically. Therefore, there would be no benefit of adding an additional neural output module that introduces additional approximations. Our generic supervised/unsupervised SB-MCL architectures are useful when statistical models are unsuitable for the output, which is the case in the completion, rotation, VAE, and DDPM experiments. In the updated draft, we improve the description of the compared methods, and the results of GeMCL and ALPaCA are also presented under the name SB-MCL to reduce confusion.

Reviewer comment (Experiments):
**Motivation:** I am having a hard time finding a realistic scenario to deploy such a framework. For instance, it seems too restrictive to train VAEs continually while only seeing the data once. At least in the context of generative models, we usually do not operate in such restrictive scenarios.

Author response:
### Potential Applications of MCL with Generative Models

The integration of MCL with deep generative models holds tremendous potential, unlocking possibilities for various applications that are currently beyond reach. Consider, for instance, text-to-image applications like OpenAI's DALL-E, Adobe's Firefly, or Midjourney that are based on the diffusion model. They significantly expedite the artistic design processes and are already creating substantial economic values. Combining MCL can take this further; artists can supply a few examples of desired results to get a rapidly adapted personalized model. Since the design process often involves gradually adding new assets (e.g., characters, scenes, and objects when creating an animation) built upon the existing assets, continually and rapidly adapting the model to the current project would be especially beneficial.

### Motivation

It is unfortunate that the reviewer did not find our example convincing enough. Here, we provide a more general argument, instead of a specific example, about this concern:
> I am having a hard time finding a realistic scenario to deploy such a framework.

Since our SB-MCL is proposed as a generic MCL framework, it can be applied to almost any MCL setting and model architecture. Meanwhile, MCL can be regarded as a data-driven approach to CL problems that require additional meta-training data. Therefore, if there is a CL scenario that the reviewer considers realistic, our framework can be a potential solution.

Reviewer comment (Experiments):
**Evaluations:** This is tightly related to the first issue. The meta-training tasks are very similar to the meta-testing ones. I believe it is necessary to evaluate the expressiveness of the Bayesian learning approach on more diverse tasks.

Author response:
### Meta-Learning Evaluations

As the review pointed out, our meta-training tasks are similar to the meta-testing ones. However, it is one of the most fundamental assumptions of meta-learning [1, 5, 7, 9] and MCL [2, 3, 4], not confined to our approach. In essence, it represents a broader principle applicable to all machine learning paradigms: the training set should cover the test distribution. Even in standard learning scenarios, we generally do not expect a learned model to magically generalize beyond its training set; for example, if we train an image classifier on the images of cats and dogs, it would not perform well on the images of cars or airplanes. This challenge may be related to other topics like open-set classification or out-of-distribution generalization, which are orthogonal to meta-learning and MCL.

Reviewer comment (Presentation):
**plots:** There are too many plots in Figure 3. Not all of them are informative and some should definitely be moved to the appendix. The message would be better conveyed if we had more tables in the main body instead of a large number of plots in just one figure. The legends are better to be close to the actual plot. It is not easy to match the colors in the top rows to the legend in the bottom row. Also, some of the gray trends (denoted by "standard" in the legend), are out of the plot frame, especially the one in the bottom right.

Author response:
### Updated Presentation of the Results

We greatly appreciate the feedback on the plots. The original plot was indeed overly crowded, as we crammed too many results into a small area. We largely updated the experiments section as suggested in the review.

- We added a table in the main text that summarizes the key results.
- The number of plots is significantly reduced.
- For each plot, we compare only three methods, which are necessary to deliver our key message: Standard, OML, and SB-MCL.
- To reduce confusion, the special cases of SB-MCL (PN, GeMCL, and ALPaCA) are now compared under the name SB-MCL, along with the generic SB-MCL architectures for supervised and unsupervised settings.

We believe the updated section is far more concise and clearly conveys the essence of the results.

Reviewer comment (Experiments):
**Q1:** Have the authors tried deploying their framework on more diverse sets of tasks? I am curious to see the limits of expressivity of the Bayesian methods to add plasticity.

Author response:
### The Capacity of High-Dimensional Exponential Family Posterior

Since the reviewer’s major concern is about the plasticity and expressivity of exponential family posteriors, we would like to note that the dimensionality of the posterior should be taken into consideration. Even exponential family posteriors can hold a huge amount of information if it has hundreds of dimensions, as the capacity increases exponentially to the number of dimensions. While its form is simple, its capacity to hold information should not be underestimated.

Moreover, the capability of an exponential family latent variable has been extensively proven in VAEs [18] and GANs [19] where a simple Gaussian variable is mapped to complex images. We hope these concrete examples clear up the concerns about the exponential family.