Paper: Transforming Transformers for Resilient Lifelong Learning (ICLR-2024)

Reviewer comment (Experiments):
The method does not scale well as the number of tasks increases, especially for scenarios where multiple small tasks exist.

Author response:
We agree and have acknowledged both of these limitations in the submission. However, we would like to point out that our proposed exploration-exploitation sampling scheme can perform well with just 50 epochs of supernet training, whereas pure exploration cannot match this accuracy even with 300 epochs. This shows that our method can scale better than pure exploration. However, very few benchmarks containing a larger number of tasks exist that can tackle our goal of testing the behavior of prompt-based methods and our proposed parameter-based method on diverse tasks on our computational budget (with SKILL benchmark [4] being a very recent development). We will address this issue in future work.

Reviewer comment (Novelty):
Missing related Transformer-based continual learning methods, e.g., [1] Continual Learning with Transformers for Image Classification. CVPR 2022 CLVision workshop, [2] Continual Learning with Lifelong Vision Transformer. CVPR 2022, [3] D3Former: Debiased Dual Distilled Transformer for Incremental Learning. CVPR 2023.

Author response:
Thank you for pointing put these methods to us. We will cite these and discuss the differences in the Appendix of the revised submission due to space limitations. Please note that we have cited [1] (last reference on page 11 in the initial submission)

> [1] Continual Learning with Transformers for Image Classification. CVPR 2022 CLVision workshop

> [2] Continual Learning with Lifelong Vision Transformer. CVPR 2022

> [3] D3Former: Debiased Dual Distilled Transformer for Incremental Learning. CVPR 2023

> [4] Lightweight Learner for Shared Knowledge Lifelong Learning. TMLR 2023

Reviewer comment (Presentation):
Why Tokenized Data is used as input data in Figure 1, is there any special meaning?

Author response:
Figure 1 shows the typical lifelong learning methods that use the Transformer architecture. Since Transformers handle the input data by the means of tokenization (e.g. text tokens of image tokens), we represent the data in the form of tokens, and make a distinction between data tokens and prompt tokens in Figure 1a.

Reviewer comment (Novelty):
Please discuss the differences and similarities between this paper and other Transformer-based CL methods.

Author response:
Thank you for pointing put these methods to us. We will cite these and discuss the differences in the Appendix of the revised submission due to space limitations. Please note that we have cited [1] (last reference on page 11 in the initial submission)

> [1] Continual Learning with Transformers for Image Classification. CVPR 2022 CLVision workshop

> [2] Continual Learning with Lifelong Vision Transformer. CVPR 2022

> [3] D3Former: Debiased Dual Distilled Transformer for Incremental Learning. CVPR 2023

> [4] Lightweight Learner for Shared Knowledge Lifelong Learning. TMLR 2023

Reviewer comment (Theory):
The design choices are mostly experience-guided: e.g. identifying the projection layer as the ArtiHippo, using the mean class tokens to measure task similarity, and four operations to grow the ArtiHippo. More discussions on principles and analysis would make the paper more solid.

Author response:
Principles and Analysis: We agree that principles and analysis would be beneficial, we would like to note that a theoretical analysis of Transformer models is largely an open area of research. [1] provide some theoretical analysis of the role played by each component of transformers, but are restricted to toy problems. Moreover, such a theoretical analysis of Transformers applied to images is lacking. We agree that this is certainly an important area that needs to be investigated, but leave it for future work for a comprehensive treatment.

Reviewer comment (Evaluation):
Could some evaluation on the soundness of measuring task-similarity with the normalized cosine similarity between the mean class tokens be provided?

Author response:
Our choice of the use of class token is similar to Learn to Prompt, which uses the class token from the last layer to measure the similarity between the keys of the prompt pool and the current image. Following Learn to Prompt, we too use the cosine similarity to measure task similarity.

Reviewer comment (Experiments):
Would the same conclusion hold for stronger/larger ViT models?

Author response:
Due to resource constraints, we have left this for future work. However, some work from Parameter Efficient Fine-Tuning [2] suggests that our conclusions might hold for larger models as well. In particular, Figure 2 in [2] suggests that methods that tune the parameters (Compacter, Compacter++ [3], IA3 [2], LoRA [4], etc.) perform better than prompt based approaches. We could expect the same trend to follow here as well.

Reviewer comment (Novelty):
Refinement Rather Than Revolution: The integration of ArtiHippo into Vision Transformers, though presented as a novel idea, is actually a clever twist on the established "learning to grow" concept. It's a smart update, but it falls short of being a game-changer. It feels like we're seeing a refinement of existing ideas rather than a bold reimagining of lifelong learning.

Author response:
While our method is built on Learn to Grow and SPOS, we make the following novel contributions, and also show the limitations of the two methods:

1. We show that the choice of where to apply NAS matters for achieving good forward transfer. We propose to use the final Linear projection layer in the MHSA block, which is a very lightweight component. Through ablation studies, we validate that this choice indeed results in the best average accuracy. Please see Table 6 in the Appendix for a comparison with other components (Value, Query, Key, and FFN). While it has been shown that finetuning the entire MHSA block achieves good downstream performance [1], we show that using only the projection layer is an equally viable (and more efficient) option.
2. We show that the original Learn to Grow which uses DARTS on ConvNets cannot perform well when applied to ViTs, even when an advanced version of DARTS ($\beta$-DARTS) is used (Rows 4 and 5 in Table 2).
3. We show that using the original SPOS formulation, which samples all the operators uniformly, is not sufficient to achieve better performance than DARTS (Row 6 in Table 2).
4. We propose a novel algorithm to convert task similarities (as measured by the normalized cosine similarity between the mean class tokens of each task) into a prior distribution over the NAS operations (reuse, adapt, new, skip). Intuitively, this translates to assigning higher probability to the reuse operation if the mean class token calculated using the data of task is similar to the mean class token calculated using the data of task .
5. We propose an exploration-exploitation driven sampling strategy, which is designed to enhance the prior. Here, exploration refers to sampling from a uniform distribution over the operations, and exploitation refers to sampling from the prior distribution calculated in Eqn.3. We show that this strategy significantly outperforms the origin Learn to Grow, as well as popular prompt based approaches.
6. We further show that our strategy is complementary to the prompt based approaches, and combining them leads to even higher performance.

> [1] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, Hervé Jégou: Three Things Everyone Should Know About Vision Transformers. ECCV (24) 2022: 497-515

Reviewer comment (Novelty):
A Safe Bet Over a Leap of Faith: Employing Reuse, New, Adapt, and Skip operations within Transformers comes across as a safe, almost expected move. It's as if the paper takes a well-trodden path, applying tried-and-tested strategies to new territory, rather than venturing into unexplored innovative realms.

Author response:
While the use of 4 growing operations Reuse, New, Adapt, and Skip has been explored before, we are the first to apply this paradigm to Vision Transformers. However, ViTs are computationally expensive class of models, and applying the operations uniformly to all the layers will be extremely costly. We propose, and empirically validate the choice of the linear projection layer in the MHSA block to apply the 4 operations effectively.

Venturing into unexplored innovative realms is certainly important, but applying existing techniques to new problems is also equally important. As we show in Table 2 in the main text, applying existing techniques to new problems is not straightforward, as evidenced by the low average accuracy of Learn to Grow in its original form (rows 4 and 5), and SOPS in its original form (row 6). Hence, our contribution also lies in addressing the weaknesses of the existing methods when applied to ViTs, and proposing novel techniques to overcome them (please see our response to (2)).

Reviewer comment (Novelty):
The major concern is the lack of novelty for the paper to be published in ICLR. Although the paper is well thought out and demonstrated with extensive experiments, the novelty is rather incremental. See the question section.

Author response:
While our method is built on Learn to Grow and SPOS, we make the following novel contributions, and also show the limitations of the two methods:
1. We show that the choice of where to apply NAS matters for achieving good forward transfer. We propose to use the final Linear projection layer in the MHSA block, which is a very lightweight component. Through ablation studies, we validate that this choice indeed results in the best average accuracy. Please see **Table 6 in the Appendix** for a comparison with other components (Value, Query, Key, and FFN). While it has been shown that finetuning the entire MHSA block achieves good downstream performance [1], we show that using only the projection layer is an equally viable (and more efficient) option.
2. We show that the original Learn to Grow which uses DARTS on ConvNets cannot perform well when applied to ViTs, even when an advanced version of DARTS ($\beta$-DARTS) is used (**Rows 4 and 5 in Table 2**).
3. We show that using the original SPOS formulation, which samples all the operators uniformly, is not sufficient to achieve better performance than DARTS (**Row 6 in Table 2**).
4. We propose a novel algorithm to convert task similarities (as measured by the normalized cosine similarity between the mean class tokens of each task) into a prior distribution over the NAS operations (reuse, adapt, new, skip). Intuitively, this translates to assigning higher probability to the reuse operation if the mean class token calculated using the data of task $t-1$ is similar to the mean class token calculated using the data of task $t$.
5. We propose an exploration-exploitation driven sampling strategy, which is designed to enhance the prior. Here, **exploration** refers to sampling from a uniform distribution over the operations, and **exploitation** refers to sampling from the prior distribution calculated in Eqn.3. We show that this strategy significantly outperforms the origin Learn to Grow, as well as popular prompt based approaches.
6. We further show that our strategy is complementary to the prompt based approaches, and combining them leads to even higher performance.

> [1] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, Hervé Jégou: Three Things Everyone Should Know About Vision Transformers. ECCV (24) 2022: 497-515

Reviewer comment (Novelty):
Another issue is the research problem “task incremental learning,” which seems not very practical in many vision applications. As the authors discussed, “class incremental” is more demanded since the task information is not always (unlikely) available.

Author response:
We acknowledge that the task incremental setting is a limitation of our method.  We note that the class incremental setting has its own limitations, e.g., it often requires the number of classes between different tasks is the same, which is not applicable in the VDD benchmark we tested. In the meanwhile,  the task incremental  setting remains a commonly used setting in studying lifelong learning, where  the focus is how to achieve maximal forward transfer across tasks that may have different output space  by reusing the relevant parameters learned for all the previous tasks and adding minimal new parameters, which is also an equally important problem.

Reviewer comment (Experiments):
Some comparisons in experiments seem unfair and may not be able to reflect the true performance of each framework.

Author response:
Please note that we modify all the methods to work in a task incremental setting for a fair comparison with our method. We request you to elaborate which comparisons seem unfair, so that we can address your concerns in a comprehensive manner.

Reviewer comment (Novelty):
The key methodology of this paper is mainly built upon several existing works. The four operations are adapted from “learning-to-grow,” and the construction of supernet is based on SPOS. Although finding the right place to place ArtiHippo is tricky, it has been common sense that classification information is mostly likely located on the upper level of the network.

Author response:
We respectfully disagree with this statement. [2] show that which layer to finetune depends on the nature of the data, and surgically choosing which layer to finetune outperforms tuning the last layer. Consequently, which layer to reuse, adapt, renew or skip in the learn to grow will depend on the nature of the current task, and our method provides a framework to derive a prior distribution on the four operators tased on task similarities, rather than manually inspecting which layer to tune for each incoming task. 

> [2] Yoonho Lee, Annie S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, Chelsea Finn: Surgical Fine-Tuning Improves Adaptation to Distribution Shifts. ICLR 2023

Reviewer comment (Novelty):
In addition, the method leverages the recent “prompt” based method such as S-Prompts to further improve the performance. This makes the neat “ArtiHippo” based algorithm a bit over-complicated and more engineering-driven.

Author response:
We would like to clarify that integration with S-Prompts does not form the core component of our proposed method, but rather an initial exploration of integration of the two approaches due to their complementary behavior. As shown in Table 2, rows 7 and 8, our proposed method without S-Prompt performs better than Learn to Prompt$^\dagger$, S-Prompts$^\dagger$ and Learn to Grow. The integration with S-Prompt, although very simple, **further** boosts the average accuracy. We leave a more comprehensive (and possibly simpler) integration for future work.

Reviewer comment (Experiments):
It seems all methods always use pre-trained strong ViT backbone for incremental tasks. This does cover all the cases in lifelong learning, e.g., starting from a moderated size model, or from zero knowledge. The current methodology is still like finetuning a strong baseline on several small datasets. The problem discussed and experiments are not typical lifelong learning scenarios.

Author response:
We agree that this is not a typical lifelong learning scenario, but a different and equally important scenario of continually improving a pretrained model. This methodology has been used in all the works we compare with, and is the fundamental idea behind prompt based methods, i.e., Learning to Prompt (CVPR22), S-Prompts (NeurIPS22), as well as Efficient Feature Transformation (CVPR21), and Lightweight Learner (TMLR23). Hence, the comparisons with these methods are fair. We show that adding plasticity through model parameters achieves better forward transfer than all these methods (as evidenced by the average accuracy Table 2 and Table 4). Please see the advantages of our method in Sections 4.1 and 4.4.