Paper: BOWLL: A DECEPTIVELY SIMPLE OPEN WORLD LIFELONG LEARNER (ICLR-2024)

Reviewer comment (Reproducibility):
Although the paper provides a comprehensive explanation of the methodology, further technical insights regarding the implementation and each module within the BOWLL method would be beneficial.

Author response:
**W1 & W4: “further technical insights regarding the implementation and novelty”**

We agree that more details were necessary. Whereas earlier we have just had a technical description of the OoD module (A.2), added appendices A.3 and A.4 now provide the full details for the active query and Deep Inversion. With respect to novelty, we emphasize that there exists no “GDUMB equivalent” as the default in open world learning. BOWLL provides this first baseline with a single easy to use common component, by identifying the three-fold use of batch-normalization. Both the OoD detector and the active query are novel by themselves.

Reviewer comment (Theory):
The computational complexity of the BOWLL algorithm, especially for those selection and replacement strategies, which could be a concern for large-scale datasets or practical applications, is not discussed in the manuscript.

Author response:
**W3: “computational complexity of BOWLL”**

We now provide an intuitive walkthrough of the computational aspect of BOWLL’s components at the end of new appendix A.4.. In summary, the main critical component is Deep Inversion, not the active query step, as DI requires many updates on a synthetic image before converging. However, we also alleviate this concern by proceeding to discuss the trade-off in light of our findings of figure 3, where BOWLL* (without Deep Inversion) shows only around a 3% accuracy decrease. We agree that it was helpful to expose this to prospective readers in the pdf.

Reviewer comment (Novelty):
Although the paper employs sound-good methodology and achieves competitive performance, further efforts regarding the technical innovation and methodological novelty would be beneficial.

Author response:
**W1 & W4: “further technical insights regarding the implementation and novelty”**

We agree that more details were necessary. Whereas earlier we have just had a technical description of the OoD module (A.2), added appendices A.3 and A.4 now provide the full details for the active query and Deep Inversion. With respect to novelty, we emphasize that there exists no “GDUMB equivalent” as the default in open world learning. BOWLL provides this first baseline with a single easy to use common component, by identifying the three-fold use of batch-normalization. Both the OoD detector and the active query are novel by themselves.

Reviewer comment (Reproducibility):
The manuscript could delve deeper into the Continual Train Step, a factor which could be pivotal for understanding the pipeline of open-world lifelong learning.

Author response:
**W5: “”delve deeper into the continual learning step**

In addition to the new appendix sections to provide more details for the individual components, we have revised the main body’s sections 3.3 and 3.4, where we now more precisely describe the use of the memory and the update step. This should improve clarity and remove potential prior ambiguity.

Reviewer comment (Reproducibility):
How does the method balance the data in the memory buffer and pseudo-images?

Author response:
**C1: “Balance data in memory buffer and pseudo-images”**

We now describe that we only generate as many pseudo-images as in the memory buffer, a 1:1 balance that is technically a prospective hyper-parameter, discussed in appendix A.4 together with computational trade-offs.

Reviewer comment (Writing):
What is the formulation of $R_{TV}()$ and $R_{l_2}()$ respectively in Eq. (7)?

Author response:
**C2: ”Formulation of equation 7 - priors”**

We now state the intuition behind the priors in the main body and provide exhaustive definitions  in appendix A.4. In short, The l2 term ensures that the range of values of the synthetic data remains in a reasonable range and the total variation (TV) prior ensures that pixels in a vicinity are related (to form entities in the specific case of images) - as motivated by cited prior work.

Reviewer comment (Experiments):
Haven't the model used those discarded data?

Author response:
**C4: “Hasn’t the model used those discarded data”**

We have revised the text to more precisely point out that data that has been rejected in the OoD step is never queried by the active learner and that data that is not in the memory buffer, is not included in optimization steps. As such, all data may have been passed through the model to figure out whether to include it, but only data that makes it into the memory buffer is ultimately trained on. Table 2 shows the number of data points that are being used in optimization.

Reviewer comment (Writing):
What is the relationship between open-world learning and open-world lifelong learning?

Author response:
**C5: ”Relationship between open world learning and open world lifelong learning”**

We realize we have not pointed this out precisely before, but open world learning only  mentions that data is included or rejected, then queried/labelled and an incremental learning step is conducted. Often this leads to an interpretation that novel identified data is simply concatenated to existing datasets over time (inspired by active learning). To disambiguate this from a continual learning step (where we do not store old data), we have included the term lifelong. We have included a statement in the related work section, pointing to a new appendix A.1 with prior work’s definitions.

Reviewer comment (Experiments):
It seems to focus solely on the class-incremental learning scenario in continual learning, despite claiming to address various types of continual learning settings. How about, for example, Task-incremental learning [1]?

Author response:
**“Overstated contributions: paper focuses only on class-incremental learning in continual learning”**

We believe there is a misconception of what the paper is doing and trying to achieve, but we taken the point as feedback to improve and have extended the writing to avoid this in the future. 
To first clarify, the paper is not tackling a purely continual learning problem. In fact, the experiments follow a logical sequence (see next point) to step by step lead the reader to open world learning, where the data does not only change over time, but may also include both irrelevant or corrupt data. Open world learning is a very challenging realistic scenario, that requires the model to predict whether a data point should be rejected/accepted, how informative it is, and how to continue training. Traditional continual learning only focuses on the last aspect. We are not investigating task-incremental learning because it assumes one knows “which task to predict” during inference through a provided label. This is a step back in the opposite direction of open world learning, where we don’t even assume that the data must be related at all to any of the tasks. 
As mentioned, we have however taken the feedback to improve the paper and now include a) a detailed introduction and definition to open world learning in appendix A.1; b) an improved experimental setup section to clarify our experiments (see also below) c) a more detailed description of the data and training details in appendix A.7; d) extended motivation and discussion of the active query’s formulation in appendix A.3 (active query).

Reviewer comment (Experiments):
The paper claims that BOWLL can achieve OOD detection, active learning, and continual learning, but I only see a comparison in final and LCA performance in table 2. This falls short of adequately demonstrating the model's superiority in all three objectives.

Author response:
**“Paper claims can achieve OOD detection, active learning, continual learning but doesn’t adequately demonstrate the model’s superiority”**

We believe the paper shows this already, as the experiments are meant to showcase individual components in a cascade of insights, first on learning speed, then on forgetting, and ultimately in the full open world learning experiment (third experiment). Especially the last, realistic scenario without curated data shows that BOWLL is the only method to succeed here, making it a very reasonable future baseline. However, we have taken the reviewer’s feedback to improve presentation and more rigorously highlight the individual aspects. Primarily we have improved the paper in the following: 

* Each experimental subsection has been reworded to be more precise on the experimental take-away and now clearly states which component is supported by the found experimental evidence. In short:
  1. Table 2 focuses on highlighting that BOWLL achieves the same performance in the particular domain incremental setting as GDUMB, but does so with less data and massively faster learning speed (LCA) - highlighting the “active” part. 
  2. Figures 2 and 3 support that BOWLL features less catastrophic forgetting through the way it maintains its memory - highlighting the “continual” part.
  3. Figures 4 highlights how BOWLL’s OoD detector is crucial to perform meaningful in the open scenario, where the scenario actually contains non-curated data. 
* Based also on reviewer BmgJ’s feedback, we have included two additional baselines in the second and third experiment: experience replay (which visits all task’s data first and then keeps a memory as it proceeds) and Softmax based OoD detection, to highlight that BOWLL is a very meaningful baseline beyond GDUMB alone. The discussion and plots are updated respectively.

Reviewer comment (Experiments):
The main weakness that I see in the paper is the limitation of the experiments. I would have expected more robust experiments in more varied datasets, and a larger number of datasets and tasks.

Author response:
**“The main weakness that I see in the paper is the limitation of the experiments”**

We have provided a more detailed description of our experiments and have clarified their purpose more thoroughly in the main body and appendix. In addition, we agree that there may not exist any SOTA methods designed specifically for open world learning, but that some other methods are interesting for readers to see in this context. To this end, we have run and added two further popular baselines to our experiments: experience replay (ER) and Softmax based OoD detection (according to the published works of Rollick et al and Hendrycks et al). Whereas GDUMB actively fills a memory buffer at random and thus is the closest in spirit to BOWLL, ER is a more intuitive comparison in experiment 2’s focus on forgetting, as ER first trains on all data before maintaining a random memory buffer over time. In figure 3, we can see that this training on all data does not lead to ER outperforming BOWLL. Naturally ER fails in the experiment with corrupted and OoD data. Here, Softmax based OoD detection provides a baseline for the open world learning scenario of the third experiment. We further note that appendix A.8 contains an ablation study, demonstrating the meaningfulness of BOWLL’s components and showing how BOWLL performs when individual parts are stripped away.

Reviewer comment (Experiments):
Similarly, I would have expected more comparisons with other SOTA methods that, although not originally open-world learning, perhaps could be slightly modified for the sake of comparison.

Author response:
**“The main weakness that I see in the paper is the limitation of the experiments”**

We have provided a more detailed description of our experiments and have clarified their purpose more thoroughly in the main body and appendix. In addition, we agree that there may not exist any SOTA methods designed specifically for open world learning, but that some other methods are interesting for readers to see in this context. To this end, we have run and added two further popular baselines to our experiments: experience replay (ER) and Softmax based OoD detection (according to the published works of Rollick et al and Hendrycks et al). Whereas GDUMB actively fills a memory buffer at random and thus is the closest in spirit to BOWLL, ER is a more intuitive comparison in experiment 2’s focus on forgetting, as ER first trains on all data before maintaining a random memory buffer over time. In figure 3, we can see that this training on all data does not lead to ER outperforming BOWLL. Naturally ER fails in the experiment with corrupted and OoD data. Here, Softmax based OoD detection provides a baseline for the open world learning scenario of the third experiment. We further note that appendix A.8 contains an ablation study, demonstrating the meaningfulness of BOWLL’s components and showing how BOWLL performs when individual parts are stripped away.

Reviewer comment (Evaluation):
Table 2 shows quite a remarkable good performance of the proposed method in the case of backward transfer, which is a very challenging problem in continual learning, and is difficult to achieve. Could you provide more insights as to why this would be the case?

Author response:
**Question: could you provide more insights in the case of backward transfer**

We agree with the reviewer that observing positive backward transfer is remarkable and interesting. In general, and particularly in the mentioned specific scenario of the sequence of multiple digit based datasets (table 2), we attribute the high performance to the fact that BOWLL contains an active learning query that balances data novelty with similarity (rather than for instance querying based on high entropy alone or doing so at random as in GDUMB). As such, new data that features some form of similarity is prioritized and thus contains partial information to further improve retrospectively. In line with the question of reviewer ELi2, we understand that the precise benefit of the exact active query formulation may not have been fully clear enough in the present manuscript. We have made the wording more precise and now provide an extended discussion of how and why the active query has been conceived in appendix A.3.

Reviewer comment (Experiments):
The image synthesis method based on Deep Inversion seems interesting. All it's doing is to generate class-conditioned pseudo-images using past representations (the running mean and variance from the batch normalization layers). How much cost will such image synthesis incur? How faithful are the generated images? Why not opt for feature synthesis which seems natural and efficient given the maintained feature mean and variance?

Author response:
**“How faithful are the generated images of Deep Inversion (a); Why not opt for feature synthesis, which seems natural and efficient given the maintained feature mean and variance (b). ”**

a) We remark that the generated images do not necessarily need to be faithful in the sense of the original data, as the intend is to mitigate catastrophic forgetting only and not train on them from scratch or reconstructing natural images. However, the included L2 and TV priors (now explained better in the main body and in much more detail in appendix A.4) regulate the image values to stay within a meaningful range and ensure that pixels in a local vicinity retain similarity, in order to ensure that entities are formed rather than just producing any (adversarial) noise that fits the statistics. 

b) We agree that feature synthesis is a highly intriguing direction, and have included a short discussion and motivation for our choice in the new appendix A.4. In essence, we believe that a strong baseline needs to both be performant but also simple to implement. Using synthesized data points from Deep Inversion matches this requirement, as it allows us to straightforward interleave these instances with the real memory buffer, populated by the active learner, in optimization. It is thus trivial to code and employ in any system that already contains batch-norm, the majority of current NNs. We do however agree that feature synthesis is intriguing and now explicitly point to it, referring also to works such as Pellegrini et al (IROS 2020) that have already demonstrated the efficacy of “latent replay” techniques. (We are happy to include more references here, if the reviewer has explicit pointers). We are excited for future work to pick up on this direction and showcase how such a more involved approach could beat the current BOWLL baseline, in similar spirit to how future work may extend the use of diagonal covariance towards full measures.