S-Prompts Learning with Pre-trained Transformers: An Occam’s Razor for Domain Incremental Learning
Abstract
State-of-the-art deep neural networks are still struggling to address the catastrophic forgetting problem in continual learning.
In this paper, we propose one simple paradigm (named as S-Prompting) and two concrete approaches to highly reduce the forgetting degree in one of the most typical continual learning scenarios, i.e., domain increment learning (DIL).
The key idea of the paradigm is to learn prompts independently across domains with pre-trained transformers, avoiding the use of exemplars that commonly appear in conventional methods.
This results in a win-win game where the prompting can achieve the best for each domain.
The independent prompting across domains only requests one single cross-entropy loss for training and one simple K-NN operation as a domain identifier for inference.
The learning paradigm derives an image prompt learning approach and a novel language-image prompt learning approach.
Owning an excellent scalability (0.03% parameter increase per domain), the best of our approaches achieves a remarkable relative improvement (an average of about 30%) over the best of the state-of-the-art exemplar-free methods for three standard DIL tasks, and even surpasses the best of them relatively by about 6% in average when they use exemplars.
Source code is available at https://github.com/iamwangyabin/S-Prompts .
Introduction
State-of-the-art deep neural networks are capable of performing impressively on a wide variety of individual machine learning tasks.
However, learning multiple tasks in sequence, generally called continual learning , remains a substantial challenge for deep learning.
When trained on a new task, standard neural networks typically forget most of the knowledge related to previously learned tasks.
This is a well-known phenomenon named as catastrophic forgetting .
There are three common continual learning scenarios.
Task-incremental learning (TIL) is provided with task indexes for inference and thus using task-specific neural networks could overcome forgetting.
Class-incremental learning (CIL) increases classes in sequence with task indexes being unknown for inference.
Nevertheless, the classes are generally from the same domain, which more or less decreases the challenge.
In this paper, we focus on exploring domain-incremental learning (DIL), where classes keep the same but the involved domains commonly vary a lot in sequence, with task indexes being not provided for inference.
The larger domain gaps result in more severe forgetting.
[Figure]
To alleviate catastrophic forgetting, numerous methods have been proposed for continual learning.
One of the most effective methods is to leverage a buffer of exemplars from old tasks to perform either a rehearsal or a distillation when tuning the whole network to learn new tasks (see Non-Prompting
Methods in Fig.1).
Both the rehearsal and the distillation typically decrease the forgetting degree.
However, it is often more desired that the exemplars of old tasks are not stored for better data security and privacy.
Moreover, storing a large amount of exemplars might run out available memories.
Therefore, our focus is shifted to the challenging exemplar-free DIL in this paper.
One of the most promising solutions to the exemplar-free DIL problem is to learn a set of promptsover transformers that achieve the state-of-the-arts in a wide range of areas.
In this solution, the domain-specific knowledge is stored by a prompt pool, and hence a rehearsal buffer is no longer mandatory to mitigate forgetting.
For instance, two recent prompting methods [14, 66] 5 aim at learning task/domain-specific prompts dependently across domains in an incremental update fashion.
In [14], the prompts are added task by task, and the newly added prompts share the same taskattention learning with the old prompts.
This requests a joint tuning on all the prompts and the entire transformer using a distillation loss to keep a balance between the new prompting and the old ones.
In [66], a pool of prompts is initialized at the right beginning, and a set of the prompts from the pool are selected for each instance.
The instance-level prompts generally share task-level knowledge among any similar instances, by jointly tuning the selected prompts and the whole classifier, with the pre-trained transformer being fixed.
While using different prompting approaches, [14, 66] both follow a commonly-respected principle that the prompting should keep sharing across tasks/domains.
However, the sharing-driven dependent prompt tuning paradigm as well as many of the non-prompting paradigms generally result in a tug-of-war (or a zero-sum game) , in which one side’s gain is equivalent to the other’s loss, as studied in [56, 33].
In other words, these paradigms generally accumulate new knowledge in the same feature space, which is very likely to mix up the subspaces of old/new knowledge resulting in less separable classes (more forgetting or worse transferring) (Fig.1).
In this paper, we explore a rule-breaking idea to instead play a win-win game, i.e., learning the prompts independently across domains so that the prompting can achieve the best for each domain.
The excellent generalization capability of the transformers and the strong transfer learning ability of the recently appeared prompting mechanisms allow for the realization of this idea.
In particular, we introduce a new learning paradigm that learns the prompts independently domain by domain, and incrementally inserts the learned prompts in a pool.
As shown in Fig.1, compared to the other prompting and non-prompting methods, the new paradigm suggests merely tuning the current domain- related prompts and classifiers with all the rest unrelated network components being fixed.
The frozen pre-trained transformer is capable of extracting general features, and the independent prompting reaches the best fit on each domain.
The independent prompting is able to generate a feature space where each subspace is spanned by one single domain, and all the subspaces are less overlapped (see
Feature Space in Fig.1).
This could lead to no forgetting across domains.
The learning process only requests the most naive cross-entropy loss for supervision.
The inference is also simple and efficient.
It simply uses K-NN to search for the nearest domain center generated by K-Means on the training data to the features of the given test samples, followed by prepending the learned domain-associated prompts with the image tokens to the transformer for the final classification.
As S domain-related prompts will be finally learned independently, where S is the total number of domains, we name the proposed paradigm as S-Prompts or S-Prompting for simplicity throughout the paper.
We hope the proposed S-Prompting becomes an Occam’s Razor (i.e., a simple and elegant principle) for DIL.
The S-Prompting paradigm can be applied to any pre-trained transformers.
In this paper, we exploit two approaches of S-Prompts learning.
One is based on vision transformer (ViT) [13], and the other is based on Contrastive Language-Image Pre-Training transformer (CLIP) [54].
Particularly, we suggest the CLIP-based approach that exploits a new language-image prompting scheme.
The key idea is to prompt the pair of language-image transformers synchronously with a pair of learnable language-image prompts at the two ends.
This highly enhances CLIP’s transfer learning ability on a variety of new domains.
As a result, this approach owns a great ability to reach a favorable feature space where the subspaces of different domains are pushed far away from each other and the classes are separated clearly, leading to highly impressive DIL performances.
To the best of our knowledge, this is the first time to introduce the paired language-image prompting scheme in the literature.
The contributions of this paper can be summarized as follows:
• We introduce a simple and effective S-Prompts learning paradigm with pre-trained transformers for the DIL problem.
This paradigm breaks the commonly-respected principle in continual learning.
• We exploit an Image S-Prompts (S-iPrompts) learning approach and a novel Language-Image S-Prompts (S-liPrompts) learning approach, based on two different pre-trained transformers.
• The best S-Prompts surpasses the best of the state-of-the-art exemplar-free methods significantly (an average of 30% relative improvement) for three standard DIL benchmark datasets, and even outperforms them relatively by an average of 6% when they use exemplars.
S-Prompts can scale to a large number of domains with a tiny parameter increase, e.g., 0.03% per domain in S-liPrompts.
Related Work
Continual Learning.
Numerous methods have been exploited to address catastrophic forgetting.
For instance, memory-based methods alleviate forgetting by either replaying a small set of examples from past tasks saved in a memory [57, 55, 46, 7, 23, 28, 9] or replaying synthesized data from previous tasks using generative models [60, 63].
Distillation-based methods (e.g., [40, 55, 22, 5, 67, 70, 62, 44, 49, 11]) apply the technique of knowledge distillation [21] to mitigate catastrophic forgetting.
In addition, there exist other kinds of methods like gradient-based methods (e.g., [56, 73, 16, 58, 61, 64]), regularization-based ones (e.g. [32, 74, 2, 1]), and expansion-based ones (e.g. [69, 34, 25]).
Recently, some of work like [48, 31, 29, 36, 66, 18, 78] adapts the CIL methods to the DIL problems such as continual deepfake detection, where each domain distinguishes between deepfakes and reals.
Following them, one of our approaches uses the common classifiers from the CIL methods.
While each domain share the same classes, the incremental classifiers simply treat them as difference classes.
The finer-grained task could make networks more powerful [48].
Prompt Learning.
The general idea of prompting is to learn a function to modify the input texts or images, such that the language or image model obtains additional information about the task [66].
There have been emerging a bunch of prompting methods, such as [35, 38, 26, 76, 77, 3, 24, 66, 14], to name a few 6 .
Nevertheless, these methods merely learn either image-end prompts or language-end prompts (at most using conditions from image-end), while ours explicitly learns joint language-image prompting.
Recently, L2P [66] and DyTox [14] exploit dependent prompting methods for continual learning, which leads to a tug-of-war issue, as discussed in Sec.1.
Our suggested S-Prompting paradigm instead learns task/domain associated prompts independently for a win-win gain.
[Figure]
Proposed Approach 3.1 Problem Formulation and S-Prompts Learning
We focus on the problem of exemplar-free domain incremental learning (DIL).
In a common and practical DIL scenario, the model is expected to learn knowledge of various domains in sequence to finally become a universal expert for all the domains.
Formally, at domain s , the model accesses to the incoming data D s = { x si , y si }
N s i =1 where x si is the i -th input image from domain s and y si is the corresponding label and the total number of samples at domain s is N s .
In the entire incremental learning stream, S domains are presented sequentially and each domain s has data D s from different or even highly heterogeneous domains (Fig.1).
The total class categories the model observed is C , and in DIL all categories are identified from the beginning and will not increase during the incremental learning stream.
The exemplar-free DIL requests zero exemplar from the learned sessions during the training and the inference for better data security, privacy and cheaper memory consumption.
To address the exemplar-free DIL problem, we propose an S-Prompts learning paradigm that is simple and effective.
The key idea is to learn prompts domain by domain independently with pre-trained transformers.
The pre-trained transformers are fixed to extract highly expressive low-level features, and the prompts are tuned to encode the high-level domain knowledge.
In this case, each domain knowledge will be encoded by one prompt or one set of prompts.
This not only avoids the use of exemplars but also highly reduces the catastrophic forgetting.
Accordingly, our focus is on learning the prompts of S domains or so-called S-Prompts .
As shown in Fig.1, the suggested S-Prompting merely tunes the current domain related prompts and classifier with all the rest neural network components, including the transformer, the current session unrelated prompts and classifiers, being fixed.
In this case, each set of domain-specific prompts are learned independently from the other domains related prompts.
This requests a domain identifier for inference.
In particular, we apply
K-Means to store the centriods for each domain during training, and K-NN to search for the nearest centroid of the given test image feature to identify its corresponding domain during inference.
Note that both K-Means and K-NN are performed on the feature space of the fixed pre-trained transformer without using any prompts.
Finally, we prepend the domain associated prompts and the image tokens to the transformer for image classification.
The inference steps are detailed in the caption of Fig.2.
Since DIL’s domain shifts are generally explicit and the state-of-the-art transformers are commonly good at image classification, the suggested domain identifier performs well in standard DIL tasks.
Due to the strong generalization capability of transformers, the suggested S-Prompting still performs well for those cases with incorrect domain identification or those on unseen domains.
Thanks to this mechanism, the suggested S-Prompting can achieve the best transferring for each domain, leading to the least forgetting.
Moreover, the proposed S-Prompting only requests the most naive cross-entropy loss for the guidance on classification, which further enhances the paradigm’s simplicity.
In this paper, we mainly study two concrete S-Prompting approaches: 1) Image S-Prompts (SiPrompts) learning based on the pre-trained vision transformer (ViT) [13], and 2) Language-Image SPrompts (S-liPrompts) based on the pre-trained Contrastive Language-Image Pre-Training transformer (CLIP) [54].
In essential, the main differences between these two approaches are two-folds: 1) prompt design, and 2) classifier design.
We detail them below for these two approaches.
3.2 Image S-Prompts (S-iPrompts) Learning Approach
Prompt Design.
In the approach of S-iPrompts, for one domain s , we suggest using an independent set of continuous learnable parameters (i.e., one image prompt) P is ∈ R L i × D i as a part of inputs to the pre-trained ViT, where L i ∈ R and D i ∈ R indicate the prompt’s length and embedding dimension respectively.
As shown in Fig.
2, the whole embedding of any single image from domain s follows the format of x = [ x img , P is , x cls ] , where x img denotes the image tokens and x cls represents the pre-trained class tokens of ViT.
The extended image embedding is used as an input of the transformer blocks as normal ViT does.
When trained on a new domain s + 1 , a new independent set of images prompts P is +1 is added.
Hence, learning all the domains sequentially results in a pool of domain-wise prompt pool.
The image prompt pool can be defined as P i = (cid:8) P i 1 , P i 2 , ..., P iS (cid:9) , where P is ∈ R L i ×
D i is a single set of prompts belonging to domain s , and S is the total number of domains.
Classifier Design.
The classifier of S-iPrompts is a normal fully connected (FC) layer, which is made of a linear projection parameterized by [ W s , b s ] , where W s ∈ R C × D i , b s ∈ R C , D i , C indicate the embedding dimension and the total class number respectively, and s is the domain index.
Formally, the classifier output is calculated by
[Equation]
where f ( x ) indicates the output image feature of the ViT encoder f , and σ is the softmax activation operation, σ ( z j ) =
e zj (cid:80) Ck =
e zk , for j = 1 , 2 , . . .
, C ( C indicates the number of classes).
Every learning session owns such one classifier, and thus we use a classifier pool
P fc = { [ W 1 , b 1 ] , [ W 2 , b 2 ] , ..., [ W S , b S ] } to store all these classifiers that are independent to each other.
During the inference, once the domain center is selected, the associated prompts and the corresponding classifier will be chosen for image classification.
3.3 Language-Image S-Prompts (S-liPrompts) Learning Approach
The S-liPrompts approach aims at exploiting an effective prompting scheme on CLIP [54] so as to transfer the pre-trained CLIP to a variety of domains.
As illustrated in Fig.2, the CLIP model consists of two transformers for image and text encoding respectively.
The CLIP model was trained on an enormous amount of image-text pairs in a contrastive learning manner, which aligns the pair of language-image transformers very tightly.
Hence, it is essential to prompt the pair of CLIP transformers synchronously for an effective transfer learning.
To this end, we follow the paradigm of
S-Prompts to exploit a joint language-image prompting scheme.
The scheme learns a pair of an imageend prompt and a language-end prompt domain by domain, with the pre-trained CLIP always being fixed.
On one hand, fixing CLIP enables the prompting model to inherit the extraordinary capability of CLIP in terms of zero-shot learning and few-shot learning [76].
On the other hand, the joint prompting can make the image-end transformer has an excellent fit into any upcoming domains that might be highly different from the pre-training data.
This is mainly because synchronously prompting the language-end transformer could get more semantic information and more domain-aware (Fig.4).
Prompt Design.
The design of image-end prompts for S-liPrompts is the same as that of S-iPrompts.
For the design of language-end prompts, we suggest using a learnable context prompt to replace the artificially designed prompts used by the original CLIP.
This design can better adapt pre-trained visionlanguage models to downstream tasks, especially for few-shot learning.
In particular, for the learning domain s , we use M learnable vectors v as the context prompt P ls = { v 1 , v 2 , ..., v M } ∈ R L l × D l , where L l , D l are the prompt’s length and embedding dimension respectively.
We regard it as text prompts corresponding to image prompts used in ViT. For the j -th class, the whole input t j of the text encoder is of the format t j = (cid:8) P ls , c j (cid:9) , where c j is the word embeddings of the j -th class names [CLASS] and has the same dimension of D l .
The text prompts are also associated with one single domain, and they are independent to those from the other domains.
After learning all the sequential domains, a pool of text prompts is formed as P l = (cid:8) P l 1 , P l 2 , ..., P lS (cid:9) , where S is the domain number.
The learnable context prompt can better adopt the pre-trained CLIP to various domains, which is an essential property for incremental learning.
Classifier Design.
The text encoder g of CLIP receives the input t j defined above, and produces a vectorized representation g ( t j ) , where j = 1 , 2 , . .
, C , and C denotes the class number.
The contrastive language-image pre-training of CLIP uses a contrastive loss to learn a joint embedding space for vision and language models.
One of its objectives is to maximize the cosine similarity of each image with its matched text and minimize the cosine similarities with all other unmatched texts.
The other objective is computed in a similar fashion for each text.
Let f ( x ) be the image feature generated by the image encoder f , and { g ( t j ) }
Cj is a set of weight vectors produced by the text encoder g ( . ) , each g ( t j ) representing the text feature of j -class.
The prediction probability is computed as
[Equation]
where ⟨ .,
⟩ is the cosine similarity, and ⟨ x, y ⟩ = x · y | x || y | .
We conduct the softmax activation operation σ over the result of Eq.2 to obtain the final logits for image classification.
Like S-iPrompts, S-liPrompts also grows the pool of domain-based classifiers (Fig.2).
Compared with the FC-based classifiers in S-iPrompts, there are two main advantages using the CLIP-based classifier in S-liPrompts.
Firstly, the language model provides more domain information using the domain-specific text prompts for DIL, because it generally leads to more domain-separable features (see Fig.4).
Secondly, in the incremental process, only one domain context needs to be stored for each session, while the class names are shared, which highly reduces the memory consumption.
Experiments
Datasets.
We perform experiments on three standard DIL benchmark datasets.
They are CDDB [36],
CORe50 [45], and DomainNet [52].
CDDB [36] is a dataset for continual deepfake detection, which is a challenging DIL task.
It designs Easy, Long, and Hard tracks.
We select the most challenging track (i.e., the Hard track).
The track requests learning on 5 sequential deepfake detection domains (about 27,000 images), which are GauGAN, BigGAN, WildDeepfake, WhichFaceReal, and SAN respectively.
CORe50 [45] is a widely used dataset for continual object recognition.
It hascategories from 11 distinct domains.
The continual learning setting uses 8 domains (120,000 images) for incremental training and the data from the rest (i.e., unseen) domains as test set.
DomainNet [52] is a dataset for domain adaptation and DIL, which has 345 categories and roughly 600,000 images.
Images from DomainNet is split into 6 domains.
The DIL setup on DomainNet is the same as that of CaSSLe [18].
We follow [36] to report the average forward detection accuracy and the average forgetting degree on CDDB-Hard.
For CORe50 and DomainNet, we follow [66] and [18] to present the average forward classification accuracy.
Methods.
We compare the proposed S-Prompts methods (S-iPrompts, S-liPrompts) against the stateof-the-art CIL/DIL methods.
They include non-prompting methods EWC [32], LwF [40] ER [8],
GDumb [53], BiC [67], DER++ [4] and Co2L [6], as well as the prompting methods DyTox [14] and L2P [66] and self-supervised learning method CaSSLe [18].
Among them, since EWC [32],
LwF [40], L2P [66], CaSSLe [18] can work for examplar-free DIL, we regard them as our real competitors.
For DyTox and L2P, we use their official implementations by tuning their parameters for better performances.
To compare fairly, we use the same ViT model (i.e., ViT-B/16 [13]) for most of the real competitors 7 as well as our S-iPrompts/S-liPrompts.
The proposed S-liPrompts’s text-end encoder is the same as that of CLIP [54].
For more implementations details and studies, please refer to Appendix.
Overheads.
For each domain, our proposed S-Prompting methods additionally use a set of prompts as well as a set of centroids from K-Means.
For S-liPrompts, the overhead in memory is merely of +0 .
03% per domain.
Compared to the other prompting methods DyTox and L2P, S-liPrompts enjoys much less memory overhead of the growing pool of domain-based classifiers due to its CLIP based classifier design (i.e., extremely cheap prompts and shared class names) when the class number promising
[Table]
[Figure]
[Table]
is large (e.g., 345 classes in DomainNet).
Accordingly, the proposed methods own an excellent scalability.
On the other hand, S-liPrompts takes 11.84 millisecond (ms) in average to classify an image on a single RTX-3090, while the pre-trained CLIP model takes an average of 9.37ms.
This shows that the additional domain identification merely takes around 2.5ms, which is affordable for the real-world scenarios.
The appendix presents more details about the overheads.
Results.
The results in Table 1, Table 2 and Table 3 demonstrate that the proposed S-iPrompts and S-liPrompts significantly outperform the other exemplar-free methods including the two recent prompting methods L2P and DyTox.
We can compute that the proposed S-liPrompts obtains a considerable relative improvement (an average of roughly 30%) over the best of these methods in terms of forward classification accuracy.
From Table 1 and Fig.3, we could also find the proposed
S-Prompting methods’ forgetting degrees are much less than those of the others.
On the other hand, compared to those methods using exemplars of old tasks, our S-liPrompting still enjoys the clear superiority (an average of about 6% relative improvement) over the best of them.
Besides, it is fairly surprising that our S-liPrompts even surpasses its upper-bound (joint training version) on
DomainNet that contains highly heterogeneous domains.
This is very likely that the joint learning cannot handle the large domain shifts well, and our independent prompting avoids this issue.
Another interesting phenomena is that our S-Prompts still enjoy the extraordinary generalization capability (both S-iPrompts and S-liPrompts clearly surpass others) on the unseen domains on CORe50 where all the tested domains do not appear in the incremental training.
The inferior performance of L2P on DomianNet might be due to the less transfer learning capability of its dependent instance-level prompting (prompt sharing) on heterogeneous domains.
While DyTox reaches closer to the proposed
S-liPrompts on DomainNet, it requests a large memory overhead (i.e., 50 × 345=17,250, where 50,are the exemplar number per/class and the class number respectively) for the storage of exemplars 8 .
Ablations.
The studies on the cases of fixing either FC length/weights or Prompt length/weights in Table 4 justifies the necessity of using the proposed growing classifier pool and independent
[Table]
[Table]
image/language prompting for S-iPrompts/S-liPrompts.
We also study the case that the domains are randomly selected for the inference.
It is surprising that the accuracy (80.12%) does not drop a lot and is still better than those of the state-of-the-art methods.
In fact, the domain identification accuracies are not high using either ViT-based or CLIP-based features, as shown in Table 5.
Both of these two results show that the proposed S-Promptings are not fundamentally influenced by the domain identification accuracy.
As shown in Fig.4, the intrinsic reason might be that the proposed
S-iPrompts and S-liPrompts actually result in a clearer domain separation compared to the pre-trained
ViT and CLIP, and also they inherit the excellent generalization capabilities of the used transformers.
This is justified by the result of S-liPrompts (K=5 & Zero-shot) that is merely trained on the first domain and tested on all the domains.
Through the study on different K values in K-Means, we find that the proposed S-liPrompts works very stably for different K values.
Hence, we consistently use
K=5 for all the experiments on CDDB-Hard, CORe50 and DomainNet.
Apart from the number of centers K for K-Means (studied in the Table 4), there are three other key hyper-parameters for the proposed S-Prompts, including the length of an image prompt L i , the length of a language prompt L l (only for S-liPrompts), and the value of K in K-NN selection algorithm.
We perform further ablations on the proposed S-liPrompts by keeping one of these hyper-parameters varied and fixing the rest hyper-parameters on CDDB-Hard.
Fig.5 (Top Left) illustrates the accuracy curve of using different image prompt lengths.
We can see that too small L i has negative impact, which means that the prompts have not much ability to transfer the pre-trained model to downstream tasks.
However, too large L i can not help the model to get better performance.
Fig.5 (Top Right) illustrates the accuracy curve of using different language prompt length.
Compared with the study [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
CDDB-Hard.
GauGAN
BigGAN
WildDeepfake
WhichFaceReal
[Figure]
on image prompts, language prompts also favor a sufficient length (the longest one is 16 in our case).
Fig.5 (Bottom Left) studies the influence of using different K values in K-NN on the domain incremental learning accuracy.
From the results, we can find that 1-NN works clearly better than the other cases.
The main reason is that the other K-NN cases result in a clear performance drop for domain identification (Fig.5 (Bottom Right)).
Therefore, we suggest using 1-NN that is the simplest and the most effective for the proposed S-Prompts.
TIL vs. DIL.
Table 6 shows the results of the proposed S-liPrompts in the scenarios of taskincremental learning (TIL) and domain-incremental learning (DIL).
TIL offers domain indexes for the inference, while DIL does not provide the information.
We report task-wise average accuracy (AA) that computes the average on all the task-based accuracies, and also we present task-agnostic
AA that computes the average on all the test images without considering the task indexes.
Therefore,
[Table] [Table]
the task-agnostic AA is actually more in line with the real-world settings.
The task-agnostic AAs show that S-liPrompts almost closes the gap between DIL and TIL.
Generalization.
We test the generalization capability of the proposed S-liPrompts on unseen domains i.e., out-of-domain data (OOD).
In detail, we apply the pre-trained S-liPrompts on CDDB-Hard’sdomains to 3 OOD data, which are FaceForensic++, Glow, StarGAN used in CDDB [36].
Tablereports the results of the OOD experiment.
The accuracies on the three OODs increase clearly as
S-liPrompts is trained on more seen data (S1-S5).
We report the accuracies in both task-wise and task-agnostic cases.
The results show that our S-liPrompts has a better generalization ability to these
OODs than DyTox [14], which is the runner-up on CDDB-Hard.
Discussion and Conclusion
Continual learning studies the problem of learning from a data stream from changing domains/tasks.
The general principle of conventional methods is to adapt to new domains/tasks by accumulating previously acquired knowledge, while protecting previous learning from being erased.
Beyond this commonly-respected principle, this paper proposes a rule-breaking paradigm, which is simple but effective, for the domain incremental learning problem.
In particular, the proposed paradigm performs incremental learning without using any expert knowledge from previously learned domains except for general knowledge from pre-trained transformers.
Instead, it merely learns the expert knowledge independently domain by domain, through simply prompting the state-of-the-art transformers.
The independent knowledge learning mechanism avoids the traditional tug-of-war among different learning sessions and instead plays a win-win game to achieve the best for each learning session.
The learned expert knowledge for each domain is finally gathered in a pool for a general inference on varying domains.
Our comprehensive empirical study shows that the proposed learning paradigm enjoys a groundbreaking gain (about 30% relative improvement) over the state-of-the-art competitors as well as a great scalability to a large number of incrementally appearing domains.
There remain limitations on the suggested paradigm.
For example, we do not obtain appealing results when applying it directly to the class-incremental learning (CIL) tasks.
This is because these tasks are more sensitive to the incorrect domain/task identification.
Nevertheless, the proposed paradigm still has a high potential to inspire a wide range of research areas.
For instance, the suggested prompting can be applied directly to any transfer learning problems, as it essentially learns to prompt the transformers to any upcoming new domains.
In addition, it provides an inspiration to zero-shot and few-shot learning, since the proposed S-liPrompts shows superior generalization capability on unseen domains i.e., out-of-domain data (OOD).
Acknowledgments.
This work is funded by the Singapore Ministry of Education
(MOE) Academic Research
Fund (AcRF) Tier 1 grant (MSS21C002).
This work is also supported by the National Natural Science Foundation of China (62076195) and the Fundamental Research Funds for the Central Universities (AUGA5710011522).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider
Yes
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
In the appendix, we additionally present detailed algorithms of the proposed S-liPrompts, thorough experimental settings of the competing methods and the proposed ones, more ablation experiments, as well as more discussions on related works.
6.1 Algorithm Details
Inference Pipeline.
Fig.6 illustrates the inference pipeline of the proposed language-image S-Prompts (S- liPrompts).
For the inference phase, we suggest the following steps: 1) feeding the tokens of a given test image into the image transformer to obtain the image feature, 2) utilizing K-NN to search for the nearest centroids obtained by K-Means on training data to the given test image, 3) feeding the image token and the nearest centroids associated image prompts into the image transformer, with the centroid related language prompts and class tokens being fed into the text transformer, 4) calculating the language-image prediction probabilities for the final classification.
Note that K-NN is conducted on the image feature space of the fixed pre-trained image transformer without using any prompts.
Algorithm Details.
Alg.1 and Alg.2 show the training/testing process details of the proposed S-liPrompts.
[Figure]
6.2 Experimental Details
Implementation details.
We implement the proposed S-Prompts in PyTorch with NVIDIA RTX 3090 GPUs.
We use the same image encoder architecture of ViT-B/16 [13] for S-iPrompts and S-liPrompts across all
1: Initialization: Training data: D s = { x si , y si } N s i =1 , s = 1 , 2 , ..., S ; Pre-trained image model: f ;
Pre-trained language model: g ;
Class embeddings: c j , j = 1 , 2 , ..., C ;
Image Prompt Pool: P i = ∅ ; Language Prompt Pool: P l = ∅ ; Domain centroids: M = ∅ ;
Maximum Iteration: T max ; Batch size: N bs ; Number of centroids per session: N K ; 2: for s = 1 , 2 , ..., S do 3:
Initialize image prompt P is for domain s 4:
Initialize language prompt P ls for domain s 5: for iter = 1 , 2 , ..., T max do 6:
Extract a mini-batch samples { x si , y si } N b i =1 from Traning Data D s ; 7:
Prepend x si with corresponding prompts P is by [ x si , P is , x cls ] ; 8:
Prepare input for text encoder by t j = (cid:8) P ls , c j (cid:9) , j = 1 , 2 , ..., C ; 9:
Generate class representation by g ( t j ) ; 10:
Compute the prediction probability by p = exp( ⟨ f ([ x si ,P is ,x cls ]) ,g ( t j ) ⟩ ) (cid:80)
Ck =1 exp( ⟨ f ([ x si ,P is ,x cls ]) ,g ( t k ) ⟩ ) ; 11:
Calculate Cross-Entropy loss by L B =
L ( σ ( p ) , y si ) ; 12:
Update θ P is =
θ P is − η ▽ P is θ P is ; 13:
Update θ P ls =
θ P ls − η ▽ P ls θ P ls ; 14: end for 15:
Initialize a temporal memory buffer B to store features 16: for i do 17:
Extract
D
B ← f ( x si , cls ]) 19: end for 20:
Run K-Means on B to get N K domain centroids { m
M ← { m
P l ← P
P i ← P is 24: end for
1: Initialization:
Test
Pre trained
Pre trained language model: g ;
Class embeddings c j , j = 1 , 2 , ..., C ; Image Prompt Pool: P i ; Language Prompt Pool: P l ;
Domain centroids
M domain
N M
N do
Calculate
13: Compute exp( ⟨ f ([ x,P is ,x cls ]) ,g ( t j ) ⟩ ) (cid:80)
Ck = exp( ⟨ f ([ x,P is ,x cls ]) ,g ( t k ) ⟩ ) classification
[Table] [Table]
tasks/domains.
The text encoder of S-liPrompts is the same as that of CLIP [54].
In our experiments, the total class number C is 2 for CDDB [36], 50 for CORe50 [45], and 345 for DomainNet [52] respectively.
For all the experiments in the main paper, the length of a single image prompt L i is 10 , and the length of one language prompt L l is 16 .
The embedding dimension D i is 768 for both of the used ViT and CLIP.
The dimension of word embedding c i and the embedding dimension of text transformer D l are both 512 .
The implementation of K-Means and K-nearest neighbors (KNN) are both based on scikit-learn [50]. The distance measure is the L distance.
Training details.
Our S-Prompts is insensitive to the setting of hyper-parameters and we do not tune much.
For all benchmarks, we adopt SGD optimizer with a momentum of 0 .
9 , an initial learning rate of 0 .
1 , a batch size of 128 , and a cosine annealing scheduler [47].
The number of epochs is different for different datasets, which should be large enough to fit the training sets.
Specifically, we use 50 epochs for CDDB, 10 epochs for DomainNet and CORe50.
All the training images are resized to 224 × 224 .
We only adopt simple data augmentation as other methods [48, 22], e.g., horizontal flip and random crop.
Memory overhead.
The number of context vectors for CLIP is 16 and the dimension is 512 , and thus the number of parameters of language prompts for each session is 34 , 816 .
The length of image prompts is 10 and the dimension is 768 , which has 33 , 792 parameters.
The dimension of K-Means cluster centers is 512 for
S-liPrompts and 768 for S-iPrompts.
In our experiments, we use 5 centers for all the three used benchmark datasets, and hence it adds 12 , 288 to 18 , 432 parameters per session.
Accordingly, S-iPrompts only adds 52 , 224 parameters to the original pre-trained ViT-B/16 model, leading to a paltry 0 .
05% total parameter increase.
Similarly, S-liPrompts adds 80 , 896 parameters per session.
Compared with pre-trained CLIP (with ViT-B/16 image encoder), it has a total parameter increase of 0 .
Baseline methods.
We implement EWC [32] and LwF [40] with pre-trained ViT-B/16 using the pyciltoolbox [75].
The hyper-parameters remain the same as those in their original paper.
DyTox [14] and L2P [76] are two main baselines in this paper, and we list the parameters below for reference.
We use the official implementation 10 of DyTox for its evaluation on CDDB, CORe50 and DomainNet.
The original backbone of
DyTox is a shallow ConViT [15] ( 5 self-attention blocks and a task-attention block) without pre-training.
For a fair comparison with other methods, we use a pre-trained base-ConViT 11 with first 5 layers as the backbone.
The model size of 5 layers base-ConViT (78M) is almost the same as the base-ViT (86M) used in other methods including L2P and ours.
We use the learning rate of 0 . 0001 and 50 epochs for CDDB to get the proper performance.
For DomainNet and CORe50, the learning rate is 0 .
001 and the number of epoch is 20 .
All the other parts of DyTox
DyTox [14] originally suggests training base-ConViT from scratch.
But this case’s performance (about 56%) is much lower than that (86.21%) of training from the pre-trained base-ConViT (78M), which is also higher than that (84.20%) of training from the pre-trained base-ViT (86M), on the CDDB-Hard dataset.
We to adapt it to the cases using exemplars but they generally work badly, e g., about 59% on CDDB-Hard.
[Figure]
6.3 More Experiments
Random Domain Selection vs. Proposed Domain Selection.
In addition to the comparison on CDDB (Table 4 in the main paper), we further compare the case of performing random domain selection against that of using our proposed domain selection on DomainNet.
Table 8 reports the results on these two datasets.
It shows the consistently remarkable relative improvements (by more than 8% on CDDB and about 18% on
DomainNet) of the proposed domain selection over the random domain selection.
This considerable improvement justifies the necessity of the proposed domain selection, which attributes to the significance of the proposed independent prompting paradigm.
The out-of-distribution (OOD) experiment in Table 7 verifies that the random domain selection (in Table 8) has promising results mainly due to the strong generalization of the proposed language-image prompt learning scheme over the CLIP model.
Nevertheless, the results can be further improved significantly using the proposed domain selection 8, showing the clear superiority of the proposed independent prompting paradigm.
Domain Ensemble vs. Proposed Domain Selection.
We further study one of the most popular ensemble methods (i.e., voting) on the proposed S-liPrompts.
Table 9 summarizes the comparison of the proposed
S-liPrompts against the voting-based version in terms of average accuracies on CDDB for deepfake domain incremental learning.
The significant relative increases (about 23% on CDDB, 9% on DomainNet) show that the proposed S-liPrompts method favors the proposed domain selection method that chooses the most expertised (i.e., the most related domain-based) prompting model for the inference on each test sample.
The excellent performance of the proposed domain-based prompting models verifies the significance of the proposed independent prompting scheme.
For the voting-based inference on each test sample, we first feed all the learned domain-based prompts to the CLIP models one by one resulting in multiple predictions, and we then use the majority voting strategy on the individual results to get the final prediction.
The clearly superior performance of the proposed S-liPrompts with domain selection mainly stems from the most expertised model for the given test sample.
Except for this model, the rest ones are less expertised for the given test sample, and they might be dominant to corrupt the final prediction when doing the majority voting.
In this case, the voting scheme could lead to clear performance degradation.
Moreover, it requests running all the learned CLIP-based prompting models for the voting on each test sample, and thus it is much more time-consuming than the proposed inference scheme.
Test on Unseen Domains.
Based on Table 7 in the paper, Fig.7 additionally shows the detection accuracy curves of the proposed S-liPrompts when applied to three unseen domains (FaceForensic++, Glow and StarGAN).
The
S-liPrompts models were trained on the five seen domains (GauGAN, BigGAN, WildDeepfake, WhichFaceReal and SAN) from CDDB-Hard sequentially.
As studied in the main paper, the results in Fig.7 demonstrate that the proposed S-liPrompts owns an excellent generalization ability on the unseen domains.
Successes and Failures.
Fig.8 lists some success cases which are predicted to incorrect domains but are classified correctly in the end using S-liPrompts on CDDB-Hard.
It is interesting to see that most of them are predicted to a semantically close domain.
For example, as shown in the first two rows in Fig.8, some faces from
[Figure]
Domain 1, 2, 4, 5 are predicted to either Domain 3 (WildDeepfaces) or Domain 4 (WhichFaceReal), where all the images are of faces as shown by the exemplars of Domain 3 and Domain 4 (in row 4 and row 5 of Fig.8).
In this case, they are very likely to be classified correctly, due to the excellent learning power of S-liPrompts on Domain 3 (WildDeepfaces) and Domain 4 (WhichFaceReal).
The same observation can be derived on those images at row 3 in Fig.8.
As their architecture details are very similar to those in the exemplars of Domain 5 (SAN), they are assigned to Domain 5 that still leads to the right classification.
Fig.9 shows some classical failure
[Figure]
cases. As we can see in Table 6, the challenging cases are mostly on the domains of WildDeepfake and SAN.
For the SAN cases, we can see that some irregular-looking real buildings (e.g., wired structures) are misclassified as fake, while those fake buildings with realistic-looking structures are classified as real, as displayed in Fig.9.
For the WildDeepfake domain, there exist a large number of real faces that are often of blurry or occluded or low contrast or non-frontal.
These real faces are generally hard for real/fake detection.
On the other hand, the fake faces are often of the same quality.
This highly increases the for the
Feature Spaces of Competing Methods.
Fig.10 uses t-SNE to visualize the resulting feature spaces of pre- trained ViT, CLIP, the two most competing methods DyTox [14] and L2P [76] as well as our S-iPrompts and
GauGAN/real
BigGAN/real
WildDeepfake/real
WhichFaceReal/real
SAN/real
GauGAN/fake BigGAN/fake
WildDeepfake/fake
WhichFaceReal/fake
SAN/fake
Pre-trained CLIP Proposed S-iPrompts
Proposed S-liPrompts Pre-trained ViT
DyTox L2P
GauGAN/real
BigGAN/real
WildDeepfake/real
WhichFaceReal/real
SAN/real
GauGAN/fake BigGAN/fake
WildDeepfake/fake
WhichFaceReal/fake
SAN/fake
Pre-trained CLIP Proposed S-iPrompts
Proposed S-liPrompts Pre-trained ViT
DyTox L2P
GauGAN/real
GauGAN/fake
BigGAN/real
BigGAN/fake
WildDeepfake/real
WildDeepfake/fake
WhichFaceReal/real
WhichFaceReal/fake
SAN/real
SAN/fake
GauGAN
BigGAN
WildDeepfake
WhichFaceReal
SAN
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
80.73 DyTox 48.
[Table]
S-liPrompts on CDDB-Hard.
As we discussed in the main paper, dependent learning across domains is very likely to mix up the subspaces of old/new domains.
The t-SNE results of L2P and DyTox can verify this to some extent.
In contrast, our proposed S-Prompts (especially S-liPrompts) is capable of producing much more separable subspaces.
This demonstrates that our independent learning could reach a more favorable feature space where subspaces of different domains are pushed far away from each other.
This property could lead to better performances that were reported in the main paper, showing the clear superiority of the proposed independent prompting paradigm over the commonly-respected dependent learning principle.
[Table] [Table]
Metrics to Measure Separation Degree.
We used t-SNE as it is a popular choice for visualization.
Though the significant superiority of our proposed S-liPrompts is shown clearly via t-SNE, it is mainly in terms of domain separation rather than class separation.
This phenomenon is mainly from the fact that t-SNE visualization is limited to the 2-dim projection of the original high-dimensional data.
Therefore, apart from using t-SNE visualization, we additionally use quantitative metrics to measure the degree of class separation domain by domain.
Accordingly, we (re-)collected the classification/detection accuracies in Tables 1 & 2 (main paper), which reflects the average domain-wise accuracies and thus can be a favorable metric for this purpose.
Following the related paper [36], we additionally introduce a precision based metric to measure the domain-wise class separation degree.
The results in Table 10 and Table 11 demonstrate the consistent superiority of the proposed S-iPrompts/S-liPrompts methods in terms of the class separation using such various metrics.
Memory Consumption Comparison.
We further report the memory overheads of the proposed S-iPrompts/S- liPrompts methods (ViT/CLIP-based), the ViT-based prompting methods (DyTox and L2P) as well as those
ViT-based non-prompting methods (without expansion).
As the non-prompting methods do not expand their architectures for the increasing tasks, their model parameters keep the same for the learning process.
Therefore, the main comparison is on the prompting based methods including ours.
As shown in Table 12, the proposed S-iPrompts increases the model parameters for the domain-specific prompts that are of 52.22K (0.05% relative increase) for each domain (session), and the proposed S-liPrompts’s increase is of 80.89K (0.03% relative increase) per session.
By comparison, DyTox needs to add one task-attention block over the ViT-based model and dynamically expands the domain-specific tokens (prompts) every session.
L2P is like ours directly using ViT-based model without adding any neural network blocks, and it assigns a certain memory to save the pool of used prompts once for the initialization.
From the results, we can see that DyTox’s increase is the most, and that of L2P is the least.
The relative increase ratio of the proposed S-liPrompts/S-iPrompts is very close to that of
L2P, while obtaining clearly better performances than the other competing methods.
Backbones Used by Competing Methods.
As presented in the main paper, to compare fairly, we used the same backbone (i.e., base-ViT [13]) for all the real competitors (except for DyTox that used a more advanced ViT model, i.e., ConViT [15]) as well as the proposed methods (S-iPrompts, S-liPrompts) for all the experiments (Table 1, 2, 3).
In Footnote 3 of the supp.
material, we additionally state that we reported the results of DyTox with base-ConViT [15] rather than base-ViT [13], since the performance (86.21%) of the former model is better than that (84.20%) of the latter model on CDDB.
Hence, we follow the suggestion of the original DyTox paper to use a more advanced ViT model (ConViT) for DyTox in the main paper.
Besides, DyTox with base-ViT performs very badly on CDDB with the average accuracy being 48.89%, which is even worse than that of the random guess on CDDB.
We further evaluate DyTox with base-ViT on the other two datasets (CORe50 and DomainNet), and we find that the ViT-based exemplar-free DyTox still fails on these two datasets as discovered on the
ConViT-based DyTox in the main paper.
This is mainly because DyTox requests a distillation on exemplars, which are more demanded to balance the multiple classes from new/old domains on CORe50 and DomainNet, for a promising prompt learning (Footnote 7 in the main paper).
Therefore, we report the results of the better DyTox (DyTox with base-ConViT) in the main paper, and we here merely report its results using examplars on the three benchmark datasets.
The results of Table 13 show that DyTox with base-ConViT generally outperforms (or at least comparable with) DyTox using base-ViT on the three used datasets.
Besides, CaSSLe is a self-supervised incremental learning method, and replacing supervised trained weights trained on ImageNet is not suitable.
Hence, we use its officially released backbone.
[Table] [Table]
[Table]
[Table]
Contribution of Independent Image-end Prompting.
Table 14 reports the results that are copied from Table 1, 2, 3 of the main paper.
The considerable relative improvement (13% on CDDB, 5% on CORe50, 10% on
DomainNet) of the proposed S-iPrompts (independent prompting) over the two main competitors L2P and DyTox (dependent prompting) justifies the significant superiority of the contribution of independent image prompting.
We conduct this comparison fairly in the scenario of exemplar-free domain incremental learning (DIL), which is the main aim of our paper,
i
[Table]
e., better data security, privacy and less memory consumption.
In the context of exemplar-free DIL, DyTox collapses on CORe50 and DomainNet, because it requires a distillation on selected exemplars for a better balance among multiple classes from new/old domains on these two datasets, which seems essential for a promising dependent prompting [14].
In addition, we also compared the proposed S-iPrompts against those state-of-the-art exemplar-based methods for a reference.
We are delighted to find that S-iPrompts (exemplar-free) outperforms the exemplar-based DyTox clearly (about 4% increase) and the exemplar-based L2P (about 2% improvement) on CORe50, although it is outperformed by the exemplar-based DyTox on CDDB and DomainNet using a large number of exemplars (e.g., 17,250 exemplars on DomainNet).
The cost of using large exemplar data is generally not favorable in the real-world scenarios.
Contribution of Independent Language-Image Prompting.
Table 15 presents the results that are mainly from Table 4 in the main paper, with additional notes in the second and third columns for further clarification.
As shown in Table 15, performing CLIP or CLIP+S-liPrompts directly cannot achieve improved performance over S-liPrompts using ViT as a backbone.
In contrast, applying the proposed language-image prompting scheme on the CLIP model makes remarkable improvement.
From the comparison between S-liPrompts (final, K=5) and S-liPrompts (fixed Language Prompt weights), we can find the contribution of language-image prompting brings an significant improvement of about 16% over the proposed S-iPrompts that is based on the contribution of independent image-end prompting.
By comparing S-liPrompts (fixed Image Language Prompt Length) against S-liPrompts (fixed Language Prompt weights), we can see the considerable superiority (about 8% relative improvement) of S-iPrompts (independent image-end prompting) over L2P-like learning paradigm (dependent image-end prompting), which verifies the significant contribution of independent image-end prompting again.
Additionally, the results in Table 1, 2, 3 of the main paper show that our best model S-liPrompts supasses the best of the state-of-the-art exemplar-free methods significantly (30% relative improvement on average) for the three standard DIL benchmark datasets, and even outperforms them clearly (an average increase of 6%) when they use exemplars.
In addition, we evaluate the ablation on CORe50 and DomainNet.
Table 16 shows the results.
The observation on these two datasets is consistent with that on CDDB.
In particular, for DomainNet, the considerable improvement of the final S-liPrompts model demonstrates its clear superiority over the ablated model.
In comparison, the improvement on CORe50 is clear but relatively smaller for the CLIP case, while the increase is remarkable when comparing our ViT-based method (S-iPrompts) against ViT-based L2P on CORe50.
This might be because CORe50’s domain gaps are not as large as those of CDDB and DomainNet (the CORe50 domains are captured by the same Kinect sensor under different backgrounds and lighting [45]).
On CORe50, we find most of the evaluated methods have much less forgetting compared to those on CDDB and DomainNet due to the smaller domain gaps of CORe50.
In this case, the more powerful language-image prompting scheme might share beneficial knowledge from those similar domains, which does not harm the performance a lot.
In contrast, the relative improvements are very remarkable on CDDB (Table 15) and DomainNet (Table 16), showing the significant superiority of the proposed prompting method in the cases with large domain gaps.
[Table]
[Table]
6.4 More Discussions on Related Works
As presented in the main paper
[Table]
there exist some expansion-based continual learning methods like [69, 34, 17, 59, 25, 19, 39].
They either dynamically expand the network architectures or reshape their internal structures task by task.
This results in the following three issues: 1) Though they generally separate the structure and parameter learning significantly, they still request effective sharing of parameters among tasks.
The sharing-driven network tuning inevitably results in a non-trivial trade-off (or so-called tug-of-war) between old task learning and new task learning [56, 33].
They generally require that the task indexes should be provided for the inference.
They often result in a significantly growing number of model parameters for a large number of tasks.
To overcome the second issue of the expansion-based methods, [68, 41] suggest learning a single classifier on a concatenation of all resulting feature maps from different subsets of parameters, which connects all the tasks/domains.
Nevertheless, it suffers from the dramatically increased memory overhead issue when handling a long task sequence, which thus requires a complex post-processing for pruning.
To overcome this large memory consumption (i.e, the third issue mentioned above), [14] suggests expanding prompts instead over a transformer network.
The prompt expansion leads to a small amount of parameter increase.
Yet, as discussed in the main paper, like many of traditional methods, the dependent learning across tasks always goes for a tug-of-war (i.e., the first issue mentioned above), where it can only achieve a trade-off between transferring and forgetting and thus it is almost impossible to achieve the best for every task/domain.
By comparison, our paper breaks the commonly-respected learning principle and instead establishes a new independent prompting paradigm to play a win-win game, where it can achieve the best for each domain.
In addition, our proposed S-Prompts can address another two issues well with a rough domain identifier and a negligible parameter increase through the proposed prompting paradigm.
There have been also emerging some other transformer-based incremental learning methods like [71, 27, 37], and other incremental prompting methods like [42, 65] 14 .
The transformer-based methods without prompting demand much more tuning on the whole network than the prompting methods like [76] and ours.
Also, they keep applying traditional dependent learning paradigm to play the tug-of-war game.
The prompting method [42] suggests learning episodic memory prompts to explicitly carry previously learned knowledge to subsequent tasks for class-incremental event detection.
However, it still follows the conventional learning paradigm to perform experience replay and knowledge distillation for the trade-off between transfer and interference.