S-Prompts Learning with Pre-trained Transformers: An Occam’s Razor for Domain Incremental Learning
Abstract
State-of-the-art deep neural networks are still struggling to address the catastrophicforgetting problem in continual learning.
In this paper, we propose one simpleparadigm (named as S-Prompting) and two concrete approaches to highly reducethe forgetting degree in one of the most typical continual learning scenarios, i.e.,domain increment learning (DIL).
The key idea of the paradigm is to learn promptsindependently across domains with pre-trained transformers, avoiding the useof exemplars that commonly appear in conventional methods.
This results in awin-win game where the prompting can achieve the best for each domain.
Theindependent prompting across domains only requests one single cross-entropy lossfor training and one simple K-NN operation as a domain identifier for inference.
The learning paradigm derives an image prompt learning approach and a brand-newlanguage-image prompt learning approach.
Owning an excellent scalability (0.05%parameter increase per domain), the best of our approaches achieves a remarkablerelative improvement (an average of about 30%) over the best of the state-of-the-artexemplar-free methods for three standard DIL tasks, and even surpasses the best ofthem relatively by about 6% in average when they use exemplars.
Introduction
State-of-the-art deep neural networks are capable of performing impressively on a wide variety ofindividual machine learning tasks.
However, learning multiple tasks in sequence, generally called ascontinual learning , remains a substantial challenge for deep learning.
When trained on a new task,standard neural networks typically forget most of the knowledge related to previously learned tasks.
This is a well-known phenomenon named as catastrophic forgetting .
There are three common continual learning scenarios.
Task-incremental learning (TIL) is providedwith task indexes for inference and thus using task-specific neural networks could overcome forgetting.
Class-incremental learning (CIL) increases classes in sequence with task indexes being unknownfor inference.
Nevertheless, the classes are generally from the same domain, which more or lessdecreases the challenge.
In this paper, we focus on exploring domain-incremental learning (DIL),where classes keep the same but the involved domains commonly vary a lot in sequence, with taskindexes being not provided for inference.
The larger domain gaps result in more severe forgetting.
To alleviate catastrophic forgetting, numerous methods have been proposed for continual learning.
One of the most effective methods is to leverage a buffer of exemplars from old tasks to perform eithera rehearsal or a distillation when tuning the whole network to learn new tasks (See Non-Prompting
Methods in Fig.1).
Both the rehearsal and the distillation typically decrease the forgetting degree.
[Figure]
However, it is often more desired that the exemplars of old tasks are not stored for better data securityand privacy.
Moreover, storing a large amount of exemplars might run out available memories.
Therefore, our focus is moved on to the challenging exemplar-free DIL in this paper.
One of the most promising solutions to the exemplar-free DIL problem is to learn a set of prompts 1over transformers that achieve the state-of-the-arts in a wide range of areas.
In this solution, thedomain-specific knowledge is stored by a prompt pool, hence a rehearsal buffer is no longer mandatoryto mitigate forgetting.
For instance, two recent prompting methods [14, 54] 2 aim at learningtask/domain-specific prompts dependently across domains in an incremental update fashion.
In [14],the prompts are added task by task, and the newly added prompt shares the same task-attentionlearning with the old prompts.
This requests a joint tuning on all the prompts and the entiretransformer using a distillation loss to keep a balance between the new prompting and the old ones.
In [54], a pool of prompts is initialized at the right beginning, and a set of the prompts from thepool are selected for each instance.
The instance-level prompts generally share task-level knowledgeamong any similar instances, by jointly tuning the selected prompts and the whole classifier, withthe pre-trained transformer being fixed.
While using different prompting approaches, [14, 54] bothfollow a commonly-respected principle that the prompting should keep sharing across tasks/domains.
However, the sharing-driven dependent prompt tuning paradigm as well as many of the non-promptingparadigms generally result in a tug-of-war (or a zero-sum game) , in which one side’s gain is equivalentto the other’s loss, as studied in [46, 29].
In other words, these paradigms generally accumulatenew knowledge in the same feature space, which is very likely to mix up the subspaces of old/newknowledge resulting in less separable classes (more forgetting or worse transferring) (Fig.1).
In this paper, we explore a rule-breaking idea to instead play a win-win game, i.e., learning theprompts independently across domains so that the prompting can achieve the best for each domain.
The excellent generalization capability of the transformers and the strong transfer learning abilityof the recently appeared prompting mechanisms allow for the realization of this idea.
In particular,we introduce a new prompting paradigm that learns the prompts independently domain by domain,and incrementally inserts the learned prompts in a pool.
As shown in Fig.1, compared to the otherprompting and non-prompting methods, the new paradigm suggests merely tuning the current domainrelated prompts and classifiers with all the rest unrelated network components being fixed.
The frozenpre-trained transformer is capable of extracting general features, and the independent promptingreaches the best fit on each domain.
The independent prompting is able to generate a feature spacewhere each subspace is spanned by one single domain, and all the subspaces are less overlapped (
Feature Space in Fig.1).
This could lead to no forgetting across domains.
The learning process onlyrequests the most naive cross-entropy loss for supervision.
The inference is also simple and efficient.
It simply uses K-NN to search for the nearest domain center generated by K-Means on the trainingdata to the features of the given test sample, followed by prepending the learned domain-associatedprompts with the image tokens to the transformer for the final classification.
As S domain-relatedprompts will be finally learned independently, where S is the total number of domains, we name theproposed paradigm as S-Prompts or S-Prompting for simplicity throughout the paper.
We hope theproposed S-Prompting becomes an Occam’s Razor (i.e., a simple and elegant principle) for DIL.
The S-Prompting paradigm can be applied to any pre-trained transformers.
In this paper, we exploittwo approaches of S-Prompts learning.
One is based on vision transformer (ViT) [13], and the otheris based on Contrastive Language-Image Pre-Training transformer (CLIP) [44].
Particularly, wesuggest the CLIP-based approach that exploits a brand-new language-image prompting scheme.
Thekey idea is to prompt the pair of language-image transformers synchronously with a pair of learnablelanguage-image prompts at the two ends.
This highly enhances CLIP’s transfer learning ability on avariety of new domains.
As a result, this approach owns a great ability to reach a favorable featurespace where the subspaces of different domains are pushed far away from each other and the classesare separated clearly, leading to highly impressive DIL performances.
To the best of our knowledge,this is the first time to introduce the joint language-image prompting scheme in the literature.
The contributions of this paper can be summarized as follows:
• We introduce a simple and effective S-Prompts learning paradigm with pre-trained transformers forthe DIL problem.
This paradigm breaks the commonly-respected principle in continual learning.• We exploit an Image S-Prompts (S-iPrompts) learning approach and a brand-new Language-ImageS-Prompts (S-liPrompts) learning approach, based on two different pre-trained transformers.
• The best S-Prompts surpasses the best of the state-of-the-art exemplar-free methods significantly(an average of 30% relative improvement) for three standard DIL benchmark datasets, and evenoutperforms them relatively by an average of 6% when they use exemplars.
S-Prompts can scale toa large number of domains with a tiny parameter increase, e.g., 0.05% per domain in S-liPrompts.
Related Work
Continual Learning.
Numerous methods have been exploited to address catastrophic forgetting.
Forinstance, memory-based methods alleviate forgetting by either replaying a small set of examples frompast tasks saved in a memory [47, 45, 38, 7, 20, 24, 9] or replaying synthesized data from previoustasks using generative models [49, 52].
Distillation-based methods (e.g., [34, 45, 19, 5, 55, 57, 51,36 , 40, 11]) apply the technique of knowledge distillation [18] to mitigate catastrophic forgetting.
Inaddition, there exist other kinds of methods like gradient-based methods (e.g., [46, 59, 15, 48, 50, 53]),regularization-based ones (e.g. [28, 60, 2, 1]), and expansion-based ones (e.g. [56, 30, 22]).
Recently, some of work like [39, 27, 25, 32, 54, 16, 63] adapts the CIL methods to the DIL problemssuch as continual deepfake detection, where each domain distinguishes between deepfakes and reals.
Following them, one of our approaches uses the common classifiers from the CIL methods.
Whileeach domain share the same classes, the incremental classifiers simply treat them as difference classes.
The finer-grained task could make networks more powerful [39].
Prompt Learning.
The general idea of prompting is to learn a function to modify the input texts orimages, such that the language or image model obtains additional information about the task [54].
There have been emerging a bunch of prompting methods, such as [31, 33, 23, 61, 62, 3, 21, 54, 14],to name a few 3 .
Nevertheless, these methods merely learn either image-end prompts or language-endprompts (at most using conditions from image-end), while ours explicitly learns joint language-imageprompting.
Recently, L2P [54] and DyTox [14] exploit dependent prompting methods for continuallearning, which leads to a tug-of-war issue, as discussed in Sec.1.
Our suggested S-Promptingparadigm instead learns task/domain associated prompts independently for a win-win gain
[Figure]
Proposed Approach
3.1 Problem Formulation and S-Prompts Learning
We focus on the problem of exemplar-free domain incremental learning (DIL).
In a common andpractical DIL scenario, the model is expected to learn knowledge of various domains in sequence tofinally become a universal expert for all the domains.
Formally, at domain s , the model accesses tothe incoming data D s = { x si , y si }
N s i =1 where x si is the i -th input image from domain s and y si is thecorresponding label and the total number of samples at domain s is N s .
In the entire incrementallearning stream, S domains are presented sequentially, and each domain s has data D s from differentor even highly heterogeneous domains (Fig.1).
The total class categories the model observed is C , andin DIL all categories are identified from the beginning and will not increase during the incrementallearning stream.
The exemplar-free DIL requests zero exemplar from the learned sessions during thetraining and the inference for better data security, privacy and cheaper memory consumption.
To address the exemplar-free DIL problem, we propose an S-Prompts learning paradigm that is simpleand effective.
The key idea is to learn prompts domain by domain independently with pre-trainedtransformers.
The pre-trained transformers are fixed to extract highly expressive low-level features,and the prompts are tuned to encode the high-level domain knowledge.
In this case, each domainknowledge will be encoded by one prompt or one set of prompts.
This not only avoids the use ofexemplars but also highly reduces the catastrophic forgetting.
Accordingly, our focus is on learningthe prompts of S domains or so-called S-Prompts .
As shown in Fig.1, the suggested S-Promptingmerely tunes the current domain related prompts and classifier with all the rest neural networkcomponents, including the transformer, the current session unrelated prompts and classifiers, beingfixed.
In this case, each set of domain-specific prompts are learned independently from the otherdomains related prompts.
This requests a domain identifier for inference.
In particular, we apply
K-Means to store the centriods for each domain during training, and K-NN to search for the nearestcentroids of the given test image feature to identify its corresponding domain during inference.
Notethat both K-Means and K-NN are performed on the feature space of the fixed pre-trained transformerwithout using any prompts.
Finally, we prepend the domain associated prompts and the image tokensto the transformer for image classification.
The inference steps are detailed in the caption of Fig.2.
Since DIL’s domain shifts are generally explicit and the state-of-the-art transformers are commonlygood at image classification, the suggested domain identifier performs well in standard DIL tasks.
Due to the strong generalization capability of transformers, the suggested S-Prompting still performswell for those cases with wrong domain identification or those on unseen domains.
Thanks to thismechanism, the suggested S-Prompting can achieve the best transferring for each domain, leading to the least forgetting.
Moreover, the proposed S-Prompting only requests the most naive cross-entropyloss for the guidance on classification, which further enhances the paradigm’s simplicity.
In this paper, we mainly study two concrete S-Prompting approaches: 1) Image S-Prompts (SiPrompts) learning based on the pre-trained vision transformer (ViT) [13], and 2) Language-Image SPrompts (S-liPrompts) based on the pre-trained Contrastive Language-Image Pre-Training transformer(CLIP)
In essential, the main differences between these two approaches are two-folds: 1) promptdesign, and 2) classifier design.
We detail them below for these two approaches.
3.2 Image S-Prompts (S-iPrompts) Learning Approach
Prompt Design.
In the approach of S-iPrompts, for one domain s , we suggest using an independentset of continuous learnable parameters (i.e., one image prompt) P is ∈ R L i × D i as a part of inputs to thepre-trained ViT, where L i ∈ R and D i ∈ R indicate the prompt’s length and embedding dimensionrespectively.
As shown in Fig.
2, the whole embedding of any single image from domain s followsthe format of x = [ x img , P is , x cls ] , where x img denotes the image tokens and x cls represents thepre-trained class tokens of ViT.
The extended image embedding is used as an input of the transformerblocks as normal ViT does.
When trained on a new domain s + 1 , a new independent set of imagesprompts P
i s +1 is added.
Hence, learning all the domains sequentially results in a pool of domain-wiseprompt pool.
The image prompt pool can be defined as P i = (cid:8) P i 1 , P i 2 , ..., P iS (cid:9) , where P is ∈ R L i ×
D iis a single set of prompts belonging to domain s , and S is the total number of domains.
Classifier Design.
The classifier of S-iPrompts is a normal fully connected (FC) layer, which is madeof a linear projection parameterized by [ W s , b s ] , where W s ∈ R C × D i , b s ∈ R C , D i , C indicate theembedding dimension and the total class number respectively, and s is the domain index.
Formally,the classifier output is calculated by
[Equation]
where f ( x ) indicates the output image feature of the ViT encoder f , and σ is the softmaxactivation operation, σ ( z j ) =
e zj (cid:80) Ck =
e zk , for j = 1 , 2 , . . .
, C ( C indicates the number ofclasses).
Every learning session owns such one classifier, and thus we use a classifier pool
P fc = { [ W 1 , b 1 ] , [ W 2 , b 2 ] , ..., [ W S , b S ] } to store all these classifiers that are independent to eachother.
During the inference, when the domain center is selected the associated prompts and thecorresponding classifier is also selected for image classification.
3.3 Language-Image S-Prompts (S-liPrompts) Learning Approach
The S-liPrompts approach aims at exploiting an effective prompting scheme on CLIP [44] so asto transfer the pre-trained CLIP to a variety of domains.
As illustrated in Fig.2, the CLIP modelconsists of two transformers for image and text encoding respectively.
The CLIP model was trainedon an enormous amount of image-text pairs in a contrastive learning manner, which aligns thepair of language-image transformers very tightly.
Hence, it is essential to prompt the pair of CLIPtransformers synchronously for an effective transfer learning.
To this end, we follow the paradigm of
S-Prompts to exploit a joint language-image prompting scheme.
The scheme learns a pair of an imageend prompt and a language-end prompt domain by domain, with the pre-trained CLIP always beingfixed.
On one hand, fixing CLIP enables the prompting model to inherit the extraordinary capabilityof CLIP in terms of zero-shot learning and few-shot learning [61].
On the other hand, the jointprompting can make the image-end transformer has an excellent fit into any upcoming domains thatmight be highly different from the pre-training data.
This is mainly because synchronously promptingthe language-end transformer could get more semantic information and more domain-aware (Fig.4).
Prompt Design.
The design of image-end prompts for S-liPrompts is the same as that of S-iPrompts.
For the design of language-end prompts, we suggest using a learnable context prompt to replace theartificially designed prompts used by the original CLIP.
This design can better adapt pre-trained visionlanguage models to downstream tasks, especially for few-shot learning.
In particular, for the learningdomain s , we use M learnable vectors v as the context prompt P ls = { v 1 , v 2 , ..., v M } ∈ R L l × D l ,where L l , D l are the prompt’s length and embedding dimension respectively.
We regard it as text prompts corresponding to image prompts used in ViT. For the j -th class, the whole input t j of thetext encoder is of the format t
j = (cid:8) P ls , c j (cid:9) , where c j is the word embeddings of the j -th class names[CLASS] and has the same dimension of D l .
The text prompts are also associated with one singledomain, and they are independent to those from the other domains.
After learning all the sequentialdomains, a pool of text prompts is formed as P l = (cid:8) P l 1 , P l 2 , ..., P lS (cid:9) , where S is the domain number.
The learnable context prompt can better adopt the pre-trained CLIP to various domains, which is anessential property for incremental learning.
Classifier Design.
The text encoder g of CLIP receives the input t j defined above, and producesa vectorized representation g ( t j ) , where j = 1 , 2 , . .
, C and C denotes the class number.
Thecontrastive language-image pre-training of CLIP uses a contrastive loss to learn a joint embeddingspace for vision and language models.
One of its objectives is to maximize the cosine similarityof each image with its matched text and minimize the cosine similarities with all other unmatchedtexts.
The other objective is computed in a similar fashion for each text.
Let f ( x ) be the imagefeature generated by the image encoder f , and { g ( t j ) }
Cj is a set of weight vectors produced by thetext encoder g ( . ) , each g ( t j ) representing the text feature of j -class.
The prediction probability iscomputed as
[Equation]
where ⟨ .,
⟩ is the cosine similarity, and ⟨ x, y ⟩ = x · y | x || y | .
We conduct the softmax activation operationσ over the result of Eq.2 to obtain the final logits for image classification.
Compared with the traditional FC-based classifiers, there are two main advantages using the languagemodel for the computation of classification weights.
Firstly, the language model can provide moredomain information using the domain-specific text prompts for DIL, because it generally leads tomore domain-separable features (see Fig.4).
Secondly, in the incremental process, only one domaincontext needs to be stored for each incremental session, while the class names are shared, whichhighly reduces the consumption of memory.
Experiments
Datasets.
We perform experiments on three standard DIL benchmark datasets.
They are CDDB [32],
CORe50 [37], and DomainNet [42].
CDDB [32] is a dataset for continual deepfake detection, whichis a challenging DIL task.
It designs Easy, Long, and Hard tracks.
We select the most challengingtrack (i.e., the Hard track).
The track requests learning on 5 sequential deepfake detection domains(about 27,000 images), which are GauGAN, BigGAN, WildDeepfake, WhichFaceReal, and SANrespectively.
CORe50 [37] is a widely used dataset for continual object recognition.
It has 50categories from 11 distinct domains.
The continual learning setting uses 8 domains (120,000 images)for incremental training and the data from the rest (i.e., unseen) domains as test set.
DomainNet [42]is a dataset for domain adaptation and DIL, which has 345 categories and roughly 600,000 images.
Images from DomainNet is split into 6 domains.
The DIL setup on DomainNet is the same as thatof CaSSLe [16].
We follow [32] to report the average forward detection accuracy and the averageforgetting degree on CDDB-Hard.
For CORe50 and DomainNet, we follow [54] and [16] to presentthe average forward classification accuracy.
Methods.
We compare the proposed S-Prompts methods (S-iPrompts, S-liPrompts) against the stateof-the-art CIL/DIL methods.
They include non-prompting methods EWC [28], LwF [34] ER [8],
GDumb [43], BiC [55], DER++ [4] and Co2L [6], as well as the recently published promptingmethods DyTox [14] and L2P [54] and self-supervised learning method CaSSLe [16].
Among them,since EWC [28], LwF [34], L2P [54], CaSSLe [16] can work for examplar-free DIL, we regard themas our real competitors.
For DyTox and L2P, we use their official implementations by tuning theirparameters for the best performances.
To compare fairly, we use almost the same ViT models (i.e., ViT-B/16 [13]) for all the real competitors as well as our S-iPrompts/S-liPrompts.
The proposed S-liPrompts’s text-end encoder is the same as that of CLIP [44].
For more implementations details,please refer to the supplementary material.
[Table]
[Figure]
Overheads.
For each domain, our proposed S-Prompting methods additionally use a set of promptsas well as a set of centroids from K-Means.
For S-liPrompts, the overhead in memory is merely of+0 .
05% per domain.
Compared to the other prompting methods DyTox and L2P, S-liPrompts enjoysmuch less memory overhead of the dynamic classifiers due to its language model based classifierdesign (i
[Table]
e., extremely cheep prompts and shared class names) when the class number is large (e.g.,345 classes in DomainNet).
Accordingly, the proposed methods own an excellent scalability.
Onthe other hand, S-liPrompts takes 11.84 millisecond (ms) in average to classify an image on a single RTX-3090, while the pre-trained CLIP model takes an average of 9.37ms.
This shows that theadditional domain identification merely takes around 2.5ms, which is affordable for the real-worldscenarios.
The supplementary material presents more details about the overheads.
Results.
The results in Table 1, Table 2 and Table 3 demonstrate that the proposed S-iPromptsand S-liPrompts significantly outperform the other exemplar-free methods including the two recentprompting methods L2P and DyTox.
We can compute that the proposed S-liPrompts obtains aconsiderable relative improvement (an average of roughly 30%) over the best of these methods interms of forward classification accuracy.
From Table 1 and Fig.3, we could also find the proposed
S-Prompting methods’ forgetting degrees are much less than those of the others.
On the other hand,compared to those methods using exemplars of old tasks, our S-liPrompting still enjoys the clearsuperiority (an average of about 6% relative improvement) over the best of them.
Besides, it isfairly surprising that our S-liPrompts even surpasses its upper-bound (joint training version) on
DomainNet that contains highly heterogeneous domains.
This is very likely that the joint learningcannot handle the large domain shifts well, and our independent prompting avoids this issue.
Anotherinteresting phenomena is that our S-Prompts still enjoy the extraordinary generalization capability(both S-iPrompts and S-liPrompts clearly surpass others) on the unseen domains on CORe50 whereall the tested domains do not appear in the incremental training.
The inferior performance of L2Pon DomianNet might be due to the less transfer learning capability of its dependent instance-level
[Table] [Table]
prompting (prompt sharing) on heterogeneous domains.
While DyTox reaches closer to the proposed
S-liPrompts on DomainNet, it requests a large memory overhead (i.e., 50 × 345=17,250, where 50, 345are the exemplar number per/class and the class number respectively) for the storage of exemplars 4 .
Ablations.
The studies on the cases of fixing either FC length/weights or Prompt length/weights in Table 4 justifies the necessity of using the proposed dynamic FC and prompts in the final S-iPromptsand S-liPrompts approaches.
We also study the case that the domains are randomly selected (i.e.,without using K-NN to identify the related domains) for the inference.
It is surprising that the accuracy(80.12%) does not drop a lot and is still better than those of the state-of-the-art methods.
In fact, thedomain identification accuracies are not high using either ViT-based or CLIP-based features, as shownin Table 5.
Both of these two results show that the proposed S-Promptings are not fundamentallyinfluenced by the domain identification accuracy.
As shown in Fig.4, the intrinsic reason might bethat the proposed S-iPrompts and S-liPrompts actually result in a clearer domain separation comparedto the pre-trained ViT and CLIP, and also they inherit the excellent generalization capabilities of theused transformers.
This is justified by the result of S-liPrompts (K=5 & Zero-shot) that is merelytrained on the first domain and tested on all the domains.
Through the study on different K values in K-Means, we find that the proposed S-liPrompts works very stably for different K values.
Hence, weconsistently use K=5 for all the experiments on CDDB-Hard, CORe50 and DomainNet.
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
Discussion and Conclusion
[Table]
Continual learning studies the problem of learning from a data stream from changing domains/tasks.
The general principal of conventional methods is to adapt to new domains/tasks by accumulatingpreviously acquired knowledge, while protecting previous learning from being erased.
Beyond thiscommonly-respected principle, this paper proposes a rule-breaking paradigm, which is simple buteffective, for the domain incremental learning problem.
It does not require the accumulation of theknowledge from previously learned domains.
Instead, it merely learns the knowledge independentlydomain by domain, through simply prompting the state-of-the-art transformers.
Though it additionallyrequires a domain identifier for the inference, the empirical studies show that the suggested promptingparadigm is fairly robust to the incorrect domain identification or even unseen domains.
The increasedprompting enjoys a great scalability to a large number of domains.
Hence, following the suggestednew learning rule, we recommend just learning a set of prompts independently on each new domain,and perform the inference with a rough domain identification.
There remain limitations on the suggested paradigm.
For example, we do not obtain appealing resultswhen applying it directly to the class-incremental learning tasks.
This is because these tasks are moresensitive to the incorrect domain/task identification.
Nevertheless, the proposed paradigm still has ahigh potential to inspire a wide range of research areas.
For instance, the suggested prompting can beapplied directly to any transfer learning problems, as it essentially learns to prompt the transformersto any upcoming new domains.
Besides, it might provide an inspiration to zero-shot and few-shotlearning, since we discover that the joint language-image prompting S-liPrompts (66.57%) worksbetter than the language prompting method CoOp [61] (65.75%) in the zero-shot scenario of CDDB.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]