Paper: Continual Learning in Open-vocabulary Classification with Complementary Memory Systems (ICLR-2024)

Reviewer comment (Experiments):
There are various setting in the continual learning (task incremental, class incremental, data incremental etc.) and zero-shot learning (generalized/non-generalised setting). The exact experimental setting, evaluation strategy, and the motivation of each setting are not clear. It’s difficult to follow the section 4.2, there should me better illustration and discrimination between the various evaluation scenarios.

Author response:
**Experimental settings**: We agree to clarify and add detail in the main paper (Sec 4.1, 4.2) and supplemental (S-3) about experimental settings. We also clarify below and in our response to Reviewer ShHL. 
- Our experimental settings are designed to represent different practical requirements of continual learning. In each case, labeled training examples are received in batches.
- Data incremental (Fig 2a): the batches are uniformly randomly sampled, reflecting a case when data is received, e.g. from users, and labeled.  
- Class incremental (Fig 2b): each batch contains all samples of a particular category, reflecting a case where the developer intentionally finds and labels examples of new classes to increase capability.  
- For data- and class-incremental, experiments are done individually for each task, and accuracy is averaged across tasks. The candidate label set for a task is constant, so accuracy is measured over all classes, even for classes that do not yet have training data. 
- Task incremental (Fig 2c): A single system is trained for each task sequentially, and test accuracy is averaged across all tasks at each stage. 
- Flexible inference (Fig 2d): In the task incremental setting, we also measure performance when the candidate label set does not match provided examples (zero-shot, averaging over three held-out tasks), or is the union of all tasks’ candidate labels and labels from zero-shot tasks (union + zero-shot), or is a subset of all candidate labels and labels from zero-shot tasks (mix+zero-shot). These are highly challenging scenarios that require retaining the ability to predict over arbitrary label sets, as well as the label sets that correspond to training examples.

Reviewer comment (Experiments):
There are few recent works [1,2] follow the similar setting. These works can be considered as baseline along with the CLIP zero-shot.

Author response:
**References**: Thank you for these references, which we will incorporate.  Although [1,2] have similar goals, they use attribute-based representations, while we build on language-image representations. As mentioned to Reviewer ShHL, reported results in those papers and the Radford et al. paper show that CLIP achieves much higher zero-shot accuracy than these methods reach even after training. We confirmed this in early testing as well. This is also posted to Reviewer ShHL: In our own test, under the generalized zero-shot learning setting which requires mapping learned representations to a union of seen and unseen labels, CLIP obtains the seen and unseen accuracy of 91.7 and 94.7 on the AwA2 dataset, respectively, largely surpassing the results (seen: 60.2, unseen: 77.1) obtained by [B] after training.

Reviewer comment (Presentation):
I believe that adding the problem setting before the section-3 (Method) with the proper notation will increase the readability.

Author response:
**Paper organization**: We will consider moving the problem definition before Sec 3, as suggested.

Reviewer comment (Experiments):
The recent prompting based continual learning approach [3,4] leverages the strong pretrained model and shows the promising result for the continual learning without complementary memory system. Instead of examplar storage, if model leverages promoting based approach, how the model behaves?

Author response:
**Discussion to prompt-based methods**: We agree that prompt-based methods are an important direction to explore in future work. Currently, such methods are suited mainly for task-incremental due to the large computation required.  Generally, our AIM approach to combine systems can apply to any specialized model, so larger and more tuned models such as the prompt-based models will predictably improve accuracy (at cost to slower training). Table 2 shows that using a large CLIP model, for instance, increases target task accuracy from 80.5 to 90.2.

Reviewer comment (Writing):
The paper is hard to follow. Specifically, the proposed setup is vaguely described in the paper. For example, in the literature of open-vocabulary learning, there are only base and target classes [A, B]. However, the paper keeps mentioning about zero-shot and target tasks without clear explanation on what is zero-shot class (in open-vocabulary learning).

Author response:
**Experimental settings**: We agree to clarify and add detail in the main paper (Sec 4.1, 4.2) and supplemental (S-3) about experimental settings.  There are two branches of "open vocabulary" in the literature, one based on language embeddings and one based on attribute-category relationships.  We clarify our meaning below and in response to Reviewer aTSX.

- Task: A task is defined by an image distribution and a set of candidate labels. The model’s input is an image and a set of candidate labels (without an explicit task identifier), and the output is one of the candidate labels.
- Target task: A task for which at least some training examples are received. We aim to improve performance on the original CLIP model for these tasks.
- Zero-shot task: A task for which no training examples are received. Prediction on zero-shot tasks is enabled using vision-language embeddings. We aim to retain performance from the original CLIP model on these tasks.

Reviewer comment (Novelty):
The proposed idea is highly similar to continual zero-shot learning work where the goal is also to update model with new samples while maintaining the zero-shot performance [B]. It seems that the main difference is the use of CLIP encoder which boosts the model zero-shot capability but the core idea of continuously updating the zero-shot model is similar. However, there is no discussion or comparison with these prior works.

Author response:
**References**: Thank you for these references. We'll incorporate them and others to improve our related work section.  A major difference between many of the “continual zero-shot learning” or “generalized zero-shot learning” is that these methods, such as [B], rely on attribute-based representations, while we use image and language embeddings to predict novel/unseen labels. In reported results and our own tests, CLIP has much higher zero-shot accuracy (e.g. for the aPY and SUN datasets) than the continual attribute-based methods achieve even after training, supporting the idea of building on vision-language models for open vocabulary image classification. In our own test, under the generalized zero-shot learning setting which requires mapping learned representations to a union of seen and unseen labels, CLIP obtains the seen and unseen accuracy of 91.7 and 94.7 on the AwA2 dataset, respectively, largely surpassing the results (seen: 60.2, unseen: 77.1) obtained by [B] after training.

Reviewer comment (Evaluation):
The reviewer has doubts on the effectiveness of the proposed method as on average task performance the model only improves 0.5% compared to the strong baseline ZSCL (as reported in table 1 in the main paper). Moreover, based on table 4, it appears that the proposed method doesn't perform well on fine-grained classification tasks of Flowers, Cars and EuroSat. Thus, the reviewer is not confident on whether the proposed method advances the continual open-vocabulary classification task.

Author response:
**Comparison to ZSCL**: We consider our performance with respect to the most closely related SotA ZSCL to be highly encouraging. Here’s why:

1. Shown in Table 3, our linear probe outperforms by 1.2\% in Transfer, 1.7\% in Avg, and 2.4\% in Last performance. Our TreeProbe method outperforms by 1.2\% in Transfer, 0.5\% in Avg, and 1.9\% in Last. 
2. The Transfer performance reflects nearly perfect retention of zero-shot capability, which is also demonstrated in other experiments, while the forgetting experienced by ZSCL and WiSE-FT will tend to grow as more is learned due to their weight tuning strategies.
3. Simplicity/robustness: Our method outperforms ZSCL on their own test setup, despite that:
   - Their method fine-tunes; ours does not fine-tune.
   - They tune learning rate parameters for each task individually; we do no hyperparameter tuning. 
   - Their method requires a large auxiliary dataset (ImageNet + ConceptualCaptions); we require no additional data. 

**Performance on some tasks**: For particular tasks that especially benefit from fine-tuning, our system may underperform, but this can likely be mitigated in future work that explores developing stronger exemplar-based models while still limiting the computation required to incorporate new examples.

Reviewer comment (Writing):
Can the author verify the continual learning setup as well as the terminology (zero-shot tasks) clearly in the manuscript? This would significantly improve the paper readability.

Author response:
**Experimental settings**: We agree to clarify and add detail in the main paper (Sec 4.1, 4.2) and supplemental (S-3) about experimental settings.  There are two branches of "open vocabulary" in the literature, one based on language embeddings and one based on attribute-category relationships.  We clarify our meaning below and in response to Reviewer aTSX.

- Task: A task is defined by an image distribution and a set of candidate labels. The model’s input is an image and a set of candidate labels (without an explicit task identifier), and the output is one of the candidate labels.
- Target task: A task for which at least some training examples are received. We aim to improve performance on the original CLIP model for these tasks.
- Zero-shot task: A task for which no training examples are received. Prediction on zero-shot tasks is enabled using vision-language embeddings. We aim to retain performance from the original CLIP model on these tasks.

Reviewer comment (Novelty):
Sufficient discussion should be made between the proposed work and continual zero-shot learning literature.

Author response:
**References**: Thank you for these references. We'll incorporate them and others to improve our related work section.  A major difference between many of the “continual zero-shot learning” or “generalized zero-shot learning” is that these methods, such as [B], rely on attribute-based representations, while we use image and language embeddings to predict novel/unseen labels. In reported results and our own tests, CLIP has much higher zero-shot accuracy (e.g. for the aPY and SUN datasets) than the continual attribute-based methods achieve even after training, supporting the idea of building on vision-language models for open vocabulary image classification. In our own test, under the generalized zero-shot learning setting which requires mapping learned representations to a union of seen and unseen labels, CLIP obtains the seen and unseen accuracy of 91.7 and 94.7 on the AwA2 dataset, respectively, largely surpassing the results (seen: 60.2, unseen: 77.1) obtained by [B] after training.

Reviewer comment (Evaluation):
Can the author justify the modest improvement of 0.5% compared to SOTA?

Author response:
**Comparison to ZSCL**: We consider our performance with respect to the most closely related SotA ZSCL to be highly encouraging. Here’s why:

1. Shown in Table 3, our linear probe outperforms by 1.2\% in Transfer, 1.7\% in Avg, and 2.4\% in Last performance. Our TreeProbe method outperforms by 1.2\% in Transfer, 0.5\% in Avg, and 1.9\% in Last. 
2. The Transfer performance reflects nearly perfect retention of zero-shot capability, which is also demonstrated in other experiments, while the forgetting experienced by ZSCL and WiSE-FT will tend to grow as more is learned due to their weight tuning strategies.
3. Simplicity/robustness: Our method outperforms ZSCL on their own test setup, despite that:
   - Their method fine-tunes; ours does not fine-tune.
   - They tune learning rate parameters for each task individually; we do no hyperparameter tuning. 
   - Their method requires a large auxiliary dataset (ImageNet + ConceptualCaptions); we require no additional data. 

**Performance on some tasks**: For particular tasks that especially benefit from fine-tuning, our system may underperform, but this can likely be mitigated in future work that explores developing stronger exemplar-based models while still limiting the computation required to incorporate new examples.