Learning Efﬁcient Vision Transformers via Fine-Grained Manifold Distillation
Author(s)
AfﬁliationAddress email
Abstract
In the past few years, transformers have achieved promising performance on variouscomputer vision tasks.
Unfortunately, the immense inference overhead of mostexisting vision transformers withholds them from being deployed on edge devicessuch as cell phones and smart watches.
Knowledge distillation is a widely usedparadigm for compressing cumbersome architectures into compact students viatransferring information.
However, most of them are designed for convolutionalneural networks (CNNs), which do not fully investigate the character of visiontransformer (ViT).
In this paper, we utilize the patch-level information and proposea ﬁne-grained manifold distillation method.
Speciﬁcally, we train a tiny studentmodel to match a pre-trained teacher model in the patch-level manifold space.
Then,we decouple the manifold matching loss into three terms with careful design tofurther reduce the computational costs for the patch relationship.
Equipped with theproposed method, a DeiT-Tiny model containing 5M parameters achieves 76.5%top-1 accuracy on ImageNet-1k, which is +2.0% higher than previous distillationapproaches.
Transfer learning results on other classiﬁcation benchmarks anddownstream vision tasks also demonstrate the superiority of our method over thestate-of-the-art algorithms.
Introduction
The past decade has witnessed the rise of attention-based models in the ﬁeld of natural languageprocessing (NLP) [1, 2].
Such models belonging to the transformer family [3] can effortlessly buildlong-range dependencies and have achieved remarkable performance.
Inspired by the success in
NLP, researchers have made great efforts introducing the transformer-based architectures to visiondomain and achieved promising results.
In an early attempt, Dosovitskiy et al.
[4] proposed atransformer-based vision model termed ViT which takes split image patches as the input.
ViTobtains comparable performance when compared to the state-of-the-art convolutional neural networks(CNNs), demonstrating the immense potential of applying transformers to vision tasks.
Inspired bythe design of splitting the whole image into patches as input, various vision transformers have beenproposed, including Swin [5], T2T [6], Twins [7], and TNT [8].
Although many transformers have shown the excellent capability for various vision tasks, they oftenrequire large amounts of parameters, resulting in heavy computational burden.
For example, ViTB [4] with 86M parameters pretrained on JFT-300M can only achieve comparable performance with
CNN-based EfﬁcientNet [9] while the latter is trained on ImageNet and contains only 5M parameters.
The requirement for inordinate computing resource and storage prevents them from being deployedon memory-bounded edge devices such as cell phones and smart watches.
To facilitate the challengesabove, a series of methods have been proposed to investigate compact deep neural networks, such asnetwork pruning [10, 11], low-bit quantization [12] and knowledge distillation [13].
[Figure]
The patch-level manifold space containing ﬁne-grained information facilitates knowledge transfer.
However, smaller models usually lead to performance degradation.
Knowledge distillation (KD)
[13]is a promising approach for inheriting information from a high-performance teacher to a compactstudent and maintaining the strong performance.
Touvron et al.
[14] ﬁrst proposed a KD-based compression approach for vision transformers.
Theytrained the student transformer to match hard labels provided by a pre-trained CNN teacher.
Althoughthis approach obtains satisfying results, they ignore the intermediate-layer information inheritedin vision transformers.
There have been works [15, 16] proving the effectiveness of learning fromintermediate layers for transformers in NLP, but these methods require teachers and students to haveexactly the same embedding dimension at corresponding layers, which is a fairly tight constraint andcannot always be satisﬁed.
Manifold learning-based KD [17, 18] can support layers with mismatchingdimensions and make use of inter-sample information concurrently.
However, existing manifolddistillation approaches are designed for CNNs and cannot utilize the patch-level information of visiontransformers.
As shown in Figure 1, patches depict a manifold space in a more ﬁne-grained way.
Such information can facilitate knowledge transfer remarkably.
Based on this consideration, we propose a ﬁne-grained patch-level manifold distillation method.
Inparticular, we regard vision transformers as projectors mapping inputs into multiple manifold spaceslayer by layer.
At each layer, we collect embeddings of patches to build their manifold relation mapand train the student to match the relation map of the teacher.
Since the computational complexity ishigh, we decouple the relation maps into three parts, which reduces the complexity by two orders ofmagnitude approximately.
We evaluate the proposed method on the ImageNet-1k image classiﬁcationtask.
The proposed manifold KD outperforms the distillation method in [14] by +2.0% top-1 accuracyon DeiT-Tiny.
We also conduct transfer learning experiments on CIFAR-10/100 and evaluate ourmethod on downstream tasks such as object detection and semantic segmentation.
The proposedmethod outperforms its counterparts on both tasks.
Our contributions are summarized as follows:
• We propose a ﬁne-grained manifold distillation method, which transfers patch-level manifoldinformation between vision transformers.
• We use three decoupled terms to describe the manifold space and simplify the computationalcomplexity signiﬁcantly.•
We conduct extensive experiments to verify the effectiveness of the proposed method.
The
results also demonstrate the importance of soft-label distillation and ﬁxed-depth students.
Related works
Vision transformer Transformer is originally designed for NLP tasks [19].
Inspired by its remarkableperformance, researchers have made efforts to adopt transformer-based models in CV tasks [20, 21].
Among them, Dosovitskiy et al.
[4] proposed to divide an image into patches and used embeddingsof the patches as model input.
Based on their proposed input processing scheme for images, manyvariants of vision transformers have been proposed.
For example, Han et al.
[8] proposed TNT tomodel in-patch attention, Zhou et al.
[22] and Touvron et al.
[23] built deeper vision transformers, andsome researchers [24, 5] adopted the experience in CNN to guide the design of vision transformers.
However, Most well-performed vision transformers are extremely resource-consuming and should becompressed for deployment.
Knowledge distillation KD is a model compression method proposed by Hinton et al.
[13], whichtrains a lightweight student model to match soft labels given by a large pre-trained teacher model[25].
Moreover, there are also works using structure of a information ﬂow in the teacher model asknowledge [26, 27], combining adversarial training with KD [28], or training a student with only afew samples [29].
Due to its excellent performance, KD has been adopted in various research ﬁelds,such as CV [29, 28], NLP [15], and recommendation systems [30].
To compress vision transformers,
Touvron et al.
[14] proposed a KD-based method termed DeiT.
They added a distillation token intothe student and trained the student with hard labels provided by the teacher.
Although DeiT hasachieved remarkable performance, KD for vision transformers has not been well explored yet.
Manifold learning Manifold learning is an approach for non-linear dimensionality reduction.
Itlearns a smooth manifold embedded in the original feature space to construct low-dimensionalfeatures [31, 32].
Recently, several works introduce manifold learning to KD [17, 18].
These methodstrain the student to preserve the relationships among samples learned by the teacher.
For visiontransformers, these primary attempts are coarse and can be further improved because the basic inputelements are patches not images.
Method3.1 Preliminaries
Vision transformer.
Vision transformers are attention-based models inherited from NLP, and eachlayer of transformer consists of a multi-head self-attention (MSA) block and a multi-layer perceptron(MLP) block.
These models take split images as input, i.e. , the patches.
In particular, assuming thepatch is of size P × P , an input image x ∈ R H × W × C of size H × W and channel number C isreshaped into ﬂattened patches x p ∈ R N ×
( P 2 · C ) for processing, where N = HW/P 2 .
The patchesare then projected into a D -dimension embedding space and added to positional embeddings.
Wedenote the summation result as x e ∈ R N × D .
Each model layer works as follows:
[Equation]
where LN denotes the layer normalization operation.
Knowledge Distillation.
KD is a widely used model compression method, where we use predictionsof a large pre-trained teacher model as the learning target of a tiny student model.
Given a sample xcorresponding to a label y , representing the prediction of the student and the teacher as f s ( x ) andf t ( x ) , respectively, the loss function of KD can be formulated as:
[Equation]
where H CE is the cross-entropy function, H KL is the Kullback-Leibler divergence function, τ is alabel smoothing hyper-parameter termed temperature, and λ is a balancing hyper-parameter.
Sometimes the intermediate features of the teacher can also be used for knowledge transfer.
Forexample, Romero et al.
[33] trained the student to output features similar to the teacher at intermediatelayers.
However, such methods require the teacher and the student to have the same embeddingdimension.
Otherwise, additional mapping layers are required for aligning, making the distillationprocess non-transparent.
Manifold learning-based KD methods [18, 17] support mismatched embedding dimensions.
They train the student to learn sample relationships predicted by the teacher butignore patch-level information in vision transformers.
3.2 Fine-grained manifold distillation
To utilize the batch-level and patch-level information, we propose a ﬁne-grained manifold distillationmethod.
Unlike existing KD methods [14] for vision transformer only distilling with logits, ourmethod distills batch and patch level manifolds at intermediate layers.
Figure 2 provides an overviewof the proposed method.
Image
Patches
P a t c h
E m b e dd i ng
B l o c k
B l o c k 𝑛
B l o c k 𝑁
C l a ss i f
i e r ⋯ ⋯
Teacher(fixed)
P a t c h
E m b e dd i ng
B l o c k
B l o c k 𝑚 B l o c k 𝑀
C l a ss i f
i e r ⋯ ⋯
Student
KDloss
Teacher Block 𝑛
MHA &
LN
MLP&
LN
MHA &
LN
MLP&
LN
Student Block 𝑚
Matching in Manifold Space
MD loss
[Figure]
In the ﬁne-grained manifold distillation method, we regard a vision transformer as a feature projectorthat embeds image patches into a series of smooth manifold space layer by layer.
At each pair ofmanually selected teacher-student layers, we aim to teach the student layer to output features havingthe same patch-level manifold structure as the teacher layer.
In particular, given samples of batchsize B , we denote the feature of the student layer and the teacher layer as F S ∈ R B × N × D S and
F T ∈ R B × N × D T , respectively, where D S and D T are embedding dimensions.
We ﬁrst normalizethe feature at the last dimension and then compute the manifold structure, or manifold relation map,as follows:
[Equation]
where ψ : R D 1 × D 2 × D 3 → R D 1 D 2 × D 3 is a tensor reshape operation.
The manifold relation map of
F T can be obtained in a similar way.
After that, we train the student to minimize the gap between
M ( ψ ( F T )) and M ( ψ ( F S )) with the following loss:
[Equation]
However, the computation of manifold relation maps is resource-consuming.
Its computationalcomplexity is O
( B 2 N 2 D ) and a memory space of B 2 N 2 size is required to save the result.
Takingthe settings B = 128 , N = 196 and D = 192 in the DeiT-Tiny model [14] as an example, it requiresmore than 240GFLOPs to compute and 2.5GB of memory space to save a single manifold relationmap.
The remarkable resource consumption limits the ﬁne-grained manifold distillation methodscaling up to multiple layers.
Hence we must simplify the computation.
Inspired by the orthogonal decomposition of matrices, we decouple a manifold relation map into threeparts: an intra-image relation map, an inter-image relation map, and a randomly sampled relation map.
Figure 3 illustrates the decoupling.
We compute the intra-image patch-level manifold distillation lossas follows:
[Equation] [Equation]
Similarly, the inter-image patch-level manifold distillation loss is computed by:
[Equation] [Equation]
Moreover, to relieve the information loss caused by the decoupling, we relate the intra-image and theinter-image manifolds via a relation map computed across randomly sampled patches.
Speciﬁcally,
[Figure]
we sample K rows in the reshaped feature ψ ( F ) to obtain F r ∈ R K × D , and compute the randomlysampled patch-level manifold distillation loss as follows:
[Equation]
Hence, the overall loss function of our proposed method is:
[Equation]
where α , β , and γ are hyper-parameters.
The summation over l means that the manifold relation mapmatching is performed on multiple pairs of teacher-student layers.
We will discuss the layer selectingscheme in Section 4.3.
Complexity analysis.
The computational complexity of the decoupled manifold relation map is
O ( BN 2 D + B 2 ND + K 2 D ) , and the memory space requirement is BN 2 + NB 2 + K 2 , where eachone is reduced by nearly BN/ ( B + N ) times when ignoring the lower order terms.
Still taking DeiTTiny as an example, the ﬂoating-point operations and memory space requirement reduce to 3GFLOPsand 32MB, respectively.
The sampling number K is set to 192, the same as our experiments.
Withthe decoupling scheme, ﬁne-grained manifold distillation across multiple layers becomes feasible.
3.3 Optimization
Patch merging.
Although the decoupling reduces the computation and memory space remarkably,the computing and storing overhead is still unaffordable when the patch size is too small.
For instance,the patch size at the ﬁrst stage of SwinTransformer [5] is 4 , indicating that the total patch number
N is 3136 under the input size of 224 × 224.
Such a large patch number signiﬁcantly increases thecomputational complexity and memory space requirement of the intra-image patch-level manifoldloss L intra .
To remedy this drawback, we merge adjacent patches and view them as a single patch tofurther simplify the computation.
In particular, given a feature map F ∈ R N × D , we ﬁrst reshape itinto F r ∈ R H
× W × D , where the height H and the width W are obtained following the original patchsplitting operation.
Then we adopt a merging setting ( H (cid:48) , W (cid:48) ) to merge every ( H/H (cid:48) )
× ( W/W (cid:48) )
adjacent patches in a non-overlapping manner.
Zero-padding is adopted when H/H (cid:48) or W/W (cid:48) is notan integer.
After merging, the feature map becomes F rm ∈ R H (cid:48)
× W (cid:48) × ( HWD/H (cid:48) W (cid:48) ) and is ﬁnallyreshaped into F m ∈ R ( H (cid:48) W (cid:48) )
× ( HWD/H (cid:48) W (cid:48) ) .
By adjusting the merging setting ( H (cid:48) , W (cid:48) ) , we caneasily strike the trade-off between the complexity and the granularity of manifold relation maps.
Soft distillation.
Previous work [14] adds an additional distillation token into the student, and usesthis token to learn hard labels provided by a CNN teacher.
However, when teacher and student areboth vision transformers, we propose that it is better to teach the student only with soft labels of theteacher.
This design is based on the assumption that a larger model can learn more knowledge than asmaller one, and models of the same family share the same knowledge pattern, i.e. , a student canlearn most knowledge of a teacher.
In our work, unless the student is larger than the teacher, we setthe hyper-parameter λ in Equation 3 to 1.
[Table] [Table]
Fixed depth.
Stochastic depth [34] is a regularization method that has becomes an infrastructure intraining vision transformers [14, 5].
We propose that the student should not adopt this regularizationwhen the teacher is trained with stochastic depth, since the soft labels already contain knowledgeabout this regularization.
Otherwise, the repeated regularizations may harm the student performance.
Hence, we adopt a ﬁx-depth student in our method, i.e. , the stochastic depth regularization is not usedfor training the student.
Experiments
We evaluate our ﬁne-grained manifold distillation method on ImageNet-1k [36] classiﬁcation task,
CIFAR-10/100 [37] transfer learning task, COCO [38] object detection task, and ADE20K [39]semantic segmentation task.
4.1 Setup
Datasets.
We evaluate the proposed method mainly on the ImageNet-1k image classiﬁcation task.
ImageNet-1k is a subset of the ImageNet dataset [36], which consists of more than 1.2M trainingimages and 50K validation images from 1000 classes.
To test the generalization performance ofstudent models trained with the proposed method, we conduct transfer learning experiments on
CIFAR-10 and CIFAR-100 datasets [37].
The two datasets both contain 50K training images and10K testing images, which are categorized into 10 classes and 100 classes, respectively.
Moreover,we conduct experiments on the object detection downstream task with COCO dataset [38].
We usethe COCO 2017 split, which consists of 118K training images and 5K validation images containingobjects from 80 categories.
Baselines and models.
We compare the proposed method with DeiT [14], which adds an additionaldistillation token in the student model to learn hard labels from the teacher.
SwinTransformer studentscontaining no distillation token, so we compare with the original KD method [13] on these models.
Table 1 summarizes the used models in our experiments.
4.2 Distillation results on ImageNet-1k
Implementation details.
On the ImageNet-1k image classiﬁcation task, we train DeiT students with
CaiT teachers and SwinTransformer students with SwinTransformer teachers.
We slightly modifythe architecture of DeiT students by removing the distillation token and only using the class token.
The hyper-parameter λ in the KD loss is set to 1, i.e. , the real label is not used to train the student.
When the teacher is smaller than the student, to prevent the performance degradation caused by theweak teacher, we set λ to 0.5.
In the ﬁne-grained manifold distillation loss, hyper-parameters α ,β , and γ are set to 4, 0.1, and 0.2, respectively.
The sampling number K in loss term L random isset to 192.
We set the above hyper-parameters one by one with the grid search method, indicatingthat their combination may not be optimal.
We select the ﬁrst 4 layers and the last 4 layers of thestudent and the teacher to conduct manifold distillation.
Note that the class token in DeiT is ignoredwhen computing manifold relation maps.
Moreover, to train SwinTransformer students efﬁciently,we adopt a patch merging setting of (14 , 14) .
Other training settings follow those in DeiT [14] and
SwinTransformer [5], except the stochastic depth regularization, which is not used in our experiments.
Each student is trained for 300 epochs with 8 Tesla-V100 GPUs.
The “Hard" logits label is the hard-label distillation scheme in
DeiT. We adopt a CaiT-S24 teacher and a DeiT- student.
[Table]
Hard (cid:88) 75.9 (cid:88)
Soft (cid:88) 76.
Results.
Table 2 presents classiﬁcation results on ImageNet1k.
When compared with thehard logits distillation proposedin DeiT [14], our manifoldmethod achieves remarkable performance improvements.
Forexample, Deit-Tiny distilled viamanifold method outperforms
Deit by +2 .
0% top-1 accuracy,and Deit-Small outperforms Deitby +0 .
9% with the CaiT-S24teacher.
Although the “soft logitsdistillation" can boost the result by 0.4% compared to the “hard logits distillation", our proposedmanifold still obtains better performance by +0 . 6% .
When the teacher is CaiT-XXS24, a muchweaker architecture, the corresponding improvements are +1 .
6% on DeiT-Tiny and +1 .
2% on DeiTSmall, respectively.
Note that we report the RegNetY-16GF teacher results in DeiT for comparison,but do not conduct ﬁne-grained manifold distillation experiments with this CNN model because theproposed method is designed for distillation between vision transformers.
Moreover, we conduct experiments based on Swin Transformer, one of the state-of-the-art visiontransformer architecture.
When the teacher is Swin-Small and the student is Swin-Tiny, our proposedmethod surpasses the original student by +1 .
and achieves +0 .
5% performance improvementcompared with the original soft logits label based KD method, demonstrating the effectiveness of ourﬁne-grained manifold distillation.
4.3 Ablation and parameter comparison
[Table]
Ablation of main components.
We design experiments to verifythe effectiveness of our proposed ﬁne-grained manifold distillation method, the ﬁxed student depth, together with the “hard" and“soft" logits label in previous distillation methods.
As shown in Table 3, soft logits label can bring +0.5% top-1 accuracy compared tohard logits label, and ﬁxed depth can improve the baseline by +0.81.0%, which is a practical strategy to distill transformer.
Moreover,when combined with the ﬁne-grained manifold distillation, studentperformance can be further improved by +0.7%.
Layers for ﬁne-grained manifold distillation.
To study the impact of different layer selections in manifold relation map matching, we evaluate 5 layer selecting schemes and compare theirperformance.
In particular, we take CaiT-S24 as the teacher modeland DeiT-Tiny as the student model.
The number of selected layers is set to 6.
From the results reported in Table 5, conducting ﬁne-grained manifold distillation at
[Table] [Table]
DeiT-Tiny.
[Table]
the head and the tail of student models are both crucial.
Hence, we speculate that the “Shallow/Deep”layer selecting scheme is the best because of its outstanding performance and ease of use.
Followingthis selecting strategy, we further ablate the inﬂuence of the number of selected layers.
We denote theindices of layers selected from a L -layer model as { 1 , 2 , ..., k, L − k + 1 , ..., L − 1 , L } , and conductexperiments with the CaiT-small teacher ( L = 24 ) and the DeiT-Tiny ( L = 12 ) student to comparedifferent k setting.
The corresponding are shown in Table 4.
We can ﬁnd that our method is robustwith various layer numbers, selecting 4 layers ( k = 2 ) obtains 76.4 top-1 accuracy, and selecting 8( k = 4 ) and 12 ( k = 6 ) layers achieve 76.5 top-1 accuracy, respectively.
Ablation of decoupled manifold loss terms.
The decoupled ﬁne-grained manifold distillation lossconsists of three terms.
We study their effectiveness and report the results in Table 6.
The resultsshow that each component in the decoupled ﬁne-grained manifold distillation loss contributes toimproving the ﬁnal performance.
In particular, only transferring inter-image relation maps leads to anunstable training process (mixed precision training will corrupt the student model, leading to a NANloss), we speculate that these relations are highly related to the randomly sampled image batches.
When equipped with intra-image and random-image maps, the inter-image relation maps can furtherimprove the accuracy by +0.1%.
[Table]
Parameters in decoupled manifold distillation loss.
There are 4 hyper-parameters in decoupled manifold distillation loss: the weightof inter-image loss α , the weight of intraimage loss β , the weight of randomly sampled loss γ , and the sampling number K .
Wecompare different settings of these parametersand report the results in Table 7.
Our defaultsetting outperforms those either increasing orreducing one of the parameters.
However, dueto our coarse parameter searching scheme, webelieve that the performance can be further improved by setting these hyper-parameters more carefully.
4.4 Transfer learning
To measure the generalization ability of the proposed method, we conduct transfer learning experiments.
We ﬁne-tune a DeiT-Tiny student trained with a CaiT-S24 teacher on CIFAR-10 and
CIFAR-100 datasets for 1000 epochs.
We adopt a batch size of 768 and an AdamW optimizer with alearning rate of 5 × 10 − 6 .
Other settings follow DeiT [14].
Table 8 reports the transfer learning results.
Students trained with the ﬁne-grained manifold distillation method generalize better than others (+0.25% on CIFAR-10 and +0.71% on CIFAR-100),demonstrating a favorable generalization ability of the proposed method.
4.5 Downstream task
To further evaluate the effectiveness of our proposed method, we adopt the ﬁne-grained manifolddistillation to train object detection models on COCO 2017 dataset and semantic segmentation modelson ADE20K dataset.
[Table]
Implementation de- [Table] tails on COCO.
Weadopt Mask R-CNN[40] as the detectionframework.
Theteacher backbone is
Swin-Small and thestudent backbone is
Swin-Tiny.
All used backbones are pretrined on ImageNet-1k.
The teacher is trained for 36 epochsand each student is trained for 12 epochs.
When training students with ﬁne-grained manifolddistillation, we adopt the distillation loss on outputs of the last two backbone stages and outputs ofthe feature pyramid network neck [41].
Results on COCO.
Table 9 presents the detection results.
Our manifold distilled student outperformsthe student trained without distillation (+1.0% box AP), demonstrating that the proposed methodbeneﬁts the object detection downstream task.
[Table]
Implementation details on ADE20K.
We adopt UPerNet [42] as the segmentation framework.
The student is trainedfor 160K iterations, and other settingsare the same as experiments on COCO.
Results on ADE20K. Table 10 presentsthe segmentation results.
The proposedmethod outperforms the ditect trainingmethod and FitNet [43], an intermediate feature distilling approach, indicating that the manifolddistillation approach can also help train a semantic segmentation model.
Conclusion
In this paper, we propose a ﬁne-grained manifold distillation method for vision transformers.
Wematch batch and patch level features of student and teacher in a manifold space, and decouplethe matching loss into three terms to reduce the computational complexity.
Moreover, we adopta patch merging scheme to further simplify the computation.
Different from previous works, wedistill the student with soft labels and ﬁxed depth.
We conduct experiments on ImageNet-1k imageclassiﬁcation, CIFAR-10/100 transfer learning, COCO object detection, and ADE20K semanticsegmentation. The corresponding results demonstrate that our method consistently outperformsexisting methods.
We also conduct substantial ablation studies to show the superiority of our ﬁnegrained manifold distillation method.
The large search space of hyper-parameters is one of themost serious drawbacks of the proposed method.
In the future, we will study to further simplify theproposed method and help it be more parameter insensitive.
Societal impact.
Our work does not present any foreseeable societal consequence.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyper-parameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]