Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets
Author(s)
AffiliationAddress
Abstract
There still remains extreme performance gap between Vision Transformers (ViTs)and Convolutional Neural Networks (CNNs) when training from scratch on smalldatasets, which is concluded to the lack of inductive bias.
In this paper, we furtherconsider this problem and point out two weakness of ViTs in inductive biases, thatis, the spatial relevance and diverse channel representation .
First, on spatialaspect, objects are locally compact and relevant, thus fine-grained feature needsto be extracted from a token and its neighbours.
While the lack of data hinders
ViTs to attend the spatial relevance.
Second, on channel aspect, representationexhibits diversity on different channels.
But the scarce data can not enable ViTsto learn strong enough representation for accurate recognition.
To this end, wepropose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhancethe two inductive biases.
On spatial aspect, we adopt a hybrid structure, in whichconvolution is integrated into patch embedding and multi-layer perceptron module,forcing the model to capture the token features as well as theirs neighbouringfeatures.
On channel aspect, we introduce a dynamic feature aggregation module in
MLP and a brand new "head token" design in multi-head self-attention module tohelp re-calibrate channel representation and make different channel group representation interacts with each other.
The fusion of weak channel representation forms astrong enough representation for classification.
With this design, we successfullyeliminate the performance gap between CNNs and ViTs, and our DHVT achieves aseries of state-of-the-art performance with a lightweight model, 85.68 % on CIFAR100 with 22.8M parameters, 82.3 % on ImageNet-1K with 24.0M parameters.
Codewill be released if accepted.
Introduction
After a long-term domination by Convolutional Neural Networks (CNNs) in Computer Vision (CV)field, these years have witnessed the rapid growth of another promising alternative architectureparadigm, Vision Transformers (ViTs).
They have already exhibited great performance in manyvision tasks, such as image classification [1, 2, 3, 4, 5], object detection [6, 7, 8], segmentation [9, 10]and image generation [11, 12].
ViT [1] is the pioneering model that brings Transformer architecture [13] from Natural Language
Processing (NLP) into CV.
It has higher performance upper bound than standard CNNs, while itis at the cost of expensive computation and extremely huge amount of training data.
The vanilla
ViT needs to be firstly pre-trained on the huge dataset JFT-300M [1] and then fine-tuned on thecommon dataset ImageNet-1K [14].
Under this experimental setting, it shows higher performancethan standard CNNs.
However, when training from scratch on ImageNet-1K only, the accuracy ismuch lower.
From the practical perspective, most of the datasets are even smaller than ImageNet-1K,and not all the researchers can hold the burden of pre-training their own model on large datasets and then fine-tune on the target small datasets.
Thus, an effective architecture for training ViTs fromscratch on small datasets are demanded.
Recent works [15, 16, 17] explore the reasons for the difference in data-efficiency between ViT and
CNNs, and draw a conclusion to the lack of inductive bias.
In [15], it points out that with not enoughdata, ViT does not learn to attend locally in earlier layers.
And in [16], it says that the strongerthe inductive biases, the stronger the representations.
Large datasets tend to help ViT learn strongrepresentations.
Locality constraints improve the performance of ViT. Meanwhile in recent work[17], it demonstrates that convolutional constraints can enable strongly sample-efficient training inthe small-data regime.
The insufficient training data makes ViT hard to derive the inductive biasof attending locality, thus many recent works strive to introduce local inductive bias by integratingconvolution into ViTs [5, 2, 18, 19, 20] and modify it to hierarchical structure [21, 22, 3, 4, 23], making
ViTs more like traditional CNNs.
This style of hybrid structure shows comparable performance withstrong CNNs when training from scratch on medium dataset ImageNet-1K only.
But the performancegap on much smaller datasets still remains.
Here, we consider that the scarce training data weakens the inductive biases in ViTs.
Two kinds ofinductive bias need to be enhanced and better exploited to improve the data-efficiency, that is, thespatial relevance and diverse channel representation .
On spatial aspect , tokens are relevant andobject are locally compact.
Important fine-grained low-level feature needs to be extracted from thetoken and its neighbours at the earlier layers.
Rethinking the feature extraction framework in ViTs,the module for feature representation is the multi-layer perceptron (MLP) and its receptive field canbe seen as only itself.
So ViTs depend on the multi-head self-attention (MHSA) module to model andcapture the relation between tokens.
As is pointed out in work [15], with less training data, lowerattention layers do not learn to attend locally.
In other words, they do not focus on neighbouringtokens and aggregate local information in early stage.
As is known, capturing local features in lowerlayers facilitates the whole representation pipeline.
The deep layers sequentially process the low-leveltexture feature into high-level semantic features for final recognition.
Thus ViTs have an extremeperformance gap compared with CNNs when training from scratch on small datasets.
On channelaspect , feature representation exhibits diversity in different channels.
And ViT has its own inductivebias that different channel group encodes different feature representation of the object, and the wholetoken vector forms the representation of the object.
As is pointed out in work [16], large datasetstend to help ViT learn strong representation.
The insufficient data can not enable ViTs to learn strongenough representation, thus the whole representation is poor for accurate classification.
In this paper, we solve the performance gap of training from scratch on small datasets between
CNNs and ViTs and provide a hybrid architecture called Dynamic Hybrid Vision Transformer(DHVT) for substitute.
We first introduce a hybrid model to address the issue on spatial aspect .
The proposed hybrid model integrates a sequence of convolution layers in patch embedding stageto eliminate non-overlapping problem, preserving fine-grained low-level feature, and it involvesdepth-wise convolution [24] in MLP for local feature extraction.
In addition, we design two modulesfor making feature representation stronger to solve the problem on channel view .
To be specific, in
MLP, depth-wise convolution is adopted for the patch tokens, and the class token is identically passedthrough without any computation.
We then leverage the output patch tokens to produce channelweight like Squeeze-Excitation (SE) [25] for the class token.
This operation helps re-calibrate eachchannel for the class token to reinforce its feature representation.
Moreover, in order to enhanceinteraction among different semantic representation of different channel group and owing to thevariable length of token sequence in vision transformer structure, we devise a brand new tokenmechanism called "head token".
The number of head tokens is the same as the number of attentionheads in MHSA.
Head tokens are generated by segmenting and projecting input tokens along channel.
The head tokens will be concatenated with all other tokens to pass through the MHSA.
Each channelgroup in corresponding attention head in the MHSA now is able to interact with others.
Thoughmaybe the representation in each channel and channel group is poor for classification on account ofinsufficient training data, the head tokens help re-calibrate each learned feature pattern and enable astronger integral representation of the object, which is beneficial to final recognition.
We conduct experiments of training from scratch on various small datasets, the common dataset
CIFAR-100 and small domain datasets Clipart, Painting, Sketch from DomainNet [26] to examinethe performance of our model.
On CIFAR-100, our proposed models show significant performancemargin with strong CNNs like ResNeXt, DenseNet and Res2Net.
The Tiny model achieves 83.54 %with only 5.8M parameters, and our Small model reaches the state-of-the-art 85.68 % accuracy with only 22.8M parameters, outperforming a series of strong CNNs.
Therefore, we eliminate the gapbetween CNNs and ViTs, providing an alternative architecture which can train from scratch on smalldatasets.
We also evaluate the performance of DHVT when training from scratch on ImageNet-1K.
Our proposed DHVT-S achieves competitive 82.3 % accuracy with only 24.0M parameters, which isthe state-of-the-art non-hierarchical vision transformer structure as far as we know, demonstrating theeffectiveness of our model on larger datasets.
In summary, our main contributions are:
We conclude that the data-efficiency on small datasets can be addressed by strengthening two
inductive biases in ViTs, which are spatial relevance and diverse channel representation.
On spatial aspect, we adopt a hybrid model integrated with convolution, preserving fine-grainedlow-level feature at earlier stage and forcing the model to extract tokens feature and correspondingneighbour feature.
On channel aspect, we leverage the output patch tokens to re-calibrate class token channel-wise,producing better feature representation.
We further introduce "head token", a novel design whichhelps fusing diverse feature representation encoded in different channel group into a stronger integralrepresentation.
Related Work
Vision Transformers.
Convolutional Neural Networks [27, 28, 29, 30, 31, 32] dominated thecomputer vision fields in the past decade, with its intrinsic inductive biases designed for imagerecognition.
The past two years witnessed the rise of Vision Transformer models in various visiontasks [33, 11, 9, 6, 34, 35].
Although there exist previous works introducing attention mechanisminto CNNs [25, 36, 37], the pioneering full transformer architecture in computer vision are iGPT [38]and ViT [1].
ViT is widely adopted as the architecture paradigm for vision tasks especially imagerecognition.
It processes image as token sequence and exploits relation among tokens.
It uses "classtoken" like BERT [39] to exchange information every layer and for final classification.
It performswell when pre-trained on huge datasets.
But when training from scratch on ImageNet-1K only, itunderperforms ResNets, demonstrating a data-hungry problem.
Data-efficient ViTs.
Many of the subsequent modifications on ViT strive for a more data-efficientarchitecture which can perform well without pre-training on larger datasets.
The methods canbe divided into different groups.
[33, 40] use knowledge distillation strategy and stronger dataaugmentation methods to enable training from scratch.
[41] points out that using convolution in thepatch embedding stage greatly benefits ViTs training.
[42, 2, 5, 43, 44] leverage convolution for patchembedding to eliminate the discontinuity brought by non-overlapping patch embedding in vanilla
ViT, and such design becomes a paradigm in subsequent works.
To further introduce inductive biasinto ViT, [2, 18, 22, 45, 46] integrate depth-wise convolution into feed forward network, resultinga hybrid architecture combining the self-attention and convolution.
To make ViTs more similarto standard CNNs, [3, 44, 4, 22, 21, 23, 47, 20] re-design the spatial and channel dimension ofvanilla ViT, producing a series of hierarchical style vision transformer.
[19, 48, 49] design anotherparallel convolution branch and enable the interaction with self-attention branch, making the twobranch complements each other.
The above architectures introduce strong inductive bias and becomedata-efficient when training from scratch on ImageNet-1K.
In addition, works like [50, 51, 52]investigate channel-wise representation through conducting self-attention channel-wise, while weenhance channel representation by dynamically aggregating patch token features to enhance classtoken channel-wise and compatibly involve channel group-wise head tokens into vanilla self-attention.
Finally, works like [53, 54, 55], suggesting that the number of tokens can be variable.
ViTs for small datasets .
There exists several works on solving the training from scratch problemon small datasets.
Though the above modified vision transformers perform well when trained on
ImageNet-1K, they fail to compete with standard CNNs when training on much smaller datasetslike CIFAR-100.
Work [56] introduces a self-supervised style training strategy and a loss functionto help training ViTs on small datasets.
CCT [57] adopts a convolutional tokenization module andreplaces the class token with final sequence pooling operation.
SL-ViT [58] adopts shifted patchtokenization module and modifies self-attention to make it focus more locally.
Though the previousworks reduce the performance gap between standard CNNs ResNets[29], they fail to be sub-optimalwhen compared with strong CNNs.
Our proposed method leverage local constraints and enhancerepresentation interaction, successfully bridging the performance gap on small datasets.
[Figure]
Methods
3.1 Overview of DHVT
As shown in Fig.
1, the framework of our proposed DHVT is similar to vanilla ViT. We choosenon-hierarchical structure, where every encoder block shares the same parameter setting, processingthe same shape of features.
Under this structure, we can deal with variable length of token sequence.
We keep the design of using class token to interact with all the patch tokens and for final prediction.
In the patch embedding module, input image will be split into patches first.
Given the input imagewith resolution H × W and the target patch size P , the resulting length of patch token sequence willbe N = HW/P 2 .
Our modified patch embedding is called Sequential Overlapping Patch Embedding(SOPE), which contains several successive convolution layers of 3 × 3 convolution with stride s = 2 ,
Batch Normalization and GELU [59] activation.
The relation between the number of convolutionlayer and the patch size is P = 2 k .
SOPE is able to eliminate the discontinuity brought by vanillapatch embedding module, preserving important low-level features.
It is able to provide positioninformation to some extent.
We also adopt two affine transformations before and after the series ofconvolution layers.
This operation rescales and shifts the input feature, and it acts like normalization,making the training performance more stable on small datasets.
The whole process of SOPE can beformulated as follows.
[Equation] [Equation]
[Equation]
In Eq.1, α and β are learnable parameters, and initialized as 1 and 0 respectively.
After the sequenceof convolution layers, the feature maps are then reshaped as patch tokens and concatenated witha class token.
Then the sequence of token will be fed into encoder layers.
After SOPE, tokensequence will pass through layers of encoder, where each encoder contains Layer Normalization [60],multi-head self-attention and feed forward network.
Here we modified the MHSA as Head-Interacted
Multi-Head Self-Attention (HI-MHSA) and feed forward network as Dynamic Aggregation Feed
Forward (DAFF).
We will introduce them in the following sections.
After the final encoder layer, theoutput class token will be fed into linear head for final prediction.
3.2 Dynamic Aggregation Feed Forward
The vanilla feed forward network (FFN) in ViT is formed by two fully-connected layers and GELUactivation.
All the tokens, either patch tokens or class token, will be processed by FFN.
Here
[Figure]
we integrate depth-wise convolution [24] (DWConv) in FFN and resulting a hybrid model.
Suchhybrid model is similar to standard CNNs because it can be seen as using convolution to do featurerepresentation.
With the inductive bias brought by depth-wise convolution, the model is forced tocapture neighbouring feature, solving the problem on spatial view.
It greatly reduces the performancegap when training from scratch on small datasets, and converges faster than standard CNNs.
However,such structure still performs worse than stronger CNNs.
More investigations are required to solve theproblem on channel aspect.
We propose two methods that make the whole model more dynamic and learn stronger featurerepresentation under insufficient data.
The first proposed module is Dynamic Aggregation Feed
Forward (DAFF).
We aggregate the feature of patch tokens into class token in an channel attentionway, similar to Squeeze-Excitation operation in SENet [25], as is shown in Fig.
Class token issplit before the projection layers.
Then the patch tokens will go through a depth-wise integratedmulti-layer perceptron with shortcut inside.
The output patch tokens will then be averaged into aweight vector W .
After the squeeze-excitation operation, the output weight vector will be multipliedwith class token channel-wise.
Then the re-calibrated class token will be concatenated with outputpatch tokens to restore the token sequence.
We use X c , X p to denote class token and patch tokensrespectively.
The process can be formulated as:
[Equation] [Equation]
3.3 Head Token
The second design to enhance feature representation is "head token", which is a brand new mechanismas far as we know.
There are two reasons why we introduce head token here.
First, in the original
MHSA module, each attention head is not interacted with others, which means each head only focuson itself to calculate attention.
Second, channel groups in different head are responsible for differentfeature representation, which is the inductive bias of ViTs.
And as we pointed out above, the lackof training data can not enable models to learn strong representation.
Under this circumstance, therepresentation in each channel group is too weak for recognition.
After introducing head tokens intoattention calculation, the channel group in each head are able to interact with those in other heads,and different representation can be fused into an integral representation of the object.
Representationlearned by insufficient data may be poor in each channel, but their combination will produce an strongenough representation.
The structure of vision transformer also guarantees this mechanism becausethe length of input tokens is variable, except for the hierarchical structure vision transformer withwindow attention such as[4, 23].
The process of generating head tokens are shown as Fig.
3 (a).
We denote the number of patch tokensas N , so the length of input sequence is N + 1 .
According to the pre-defined number of heads h ,each D -dimensional token, including class token, will be reshaped into h parts.
Each part containsd channels, where D = d × h .
We average all the separated tokens in their own parts.
Thus weget totally h tokens and each one is d -dimensional.
All such intermediate tokens will be projectedinto D -dimension again, resulting h head tokens in total.
The head tokens will be added with headembedding, which provides positional information for head tokens.
Head embedding is a group oflearnable parameters, just like positional embedding.
Finally, they are concatenated with patch tokensand class token, forming the token sequence for standard MHSA, as Eq. 7, in which X H denotes
N+
D d d d d
N+ h
d d d dh d h
D h
D =
d × h Reshape
N+1+h
D
Average
Reshape input tokens c headtokens
Multi-Head Self-Attention
N
D
D h outputtokens s (a) Generate Head Tokens (b) Multi-Head Self-Attention cls token patch token s split c concat head token
Average
D c
N+
D head embedding
FC
GELU head tokens.
We do not change the attention calculation in MHSA.
Head tokens will also be linearlyprojected into query, key and value, and they will interacted with all other tokens.
After MHSA, thehead tokens will be averaged and added to class token, just as Fig.
3 (b) shows.
Head tokens can bederived as Eq.
6 shows.
We use E head to denote head embedding.
[Equation] [Equation]
Experiments
All the experiments presented in our paper are based on image classification.
We do not conductexperiments on downstream tasks.
We first introduce the training datasets and experimental settingsin Section 4.1.
The performance comparisons are shown in Section 4.2.
We also show the result ofablation study in Section 4.3.
And finally we present an example of visualization in 4.4.
4.1 Datasets and Experimental Settings
Datasets.
Our main focus is training from scratch on small datasets.
There are two factors to considerwhether a dataset is small: the total number of training data in the dataset and the average numberof training data for each class.
Some datasets are small on the first factor, but large on the second.
The example is CIFAR-10 [61], with 50000 training data in total for 10 classes, has an average of5000 instances in each class.
Considering this, we do not choose CIFAR-10 as our target dataset here.
We choose 5 different datasets here.
The main performance comparisons are on CIFAR-100 [61].
And we choose three datasets from DomainNet [26], a benchmark commonly for domain adaptationtasks.
They have a large domain-shift from common medium dataset ImageNet-1K [14], making thefine-tunning experiments non-trivial, as pointed in [56].
Finally, we also choose ImageNet-1K to testthe performance of our proposed model.
The details of the datasets are shown in Table 1.
Model Variants.
We propose two architecture variants.
• DHVT-T: 12 encoder layers, embedding dimension of 192, MLP ratios of 4, attention heads
of 4 on CIFAR-100 and DomainNet, and 3 on ImageNet-1K.
• DHVT-S: 12 encoder layers, embedding dimension of 384, MLP ratios of 4, attention headsof 8 on CIFAR-100, 6 on DomainNet and ImageNet-1K.
[Table]
Implementation Details.
When training our DHVT, we keep the image size in CIFAR-100 as itsoriginal resolution 32 × 32 , and patch size is set to 4 or 2.
For ImageNet-1K, ClipArt, Painting and
Sketch, we adopt resolution 224 × 224 , and the patch size comes to 16.
All the data-augmentationsare the same as those in DeiT [33].
We do not tune data-augmentation hyperparameters for betterperformance.
On all of the datasets, we train our network from random initialization with the AdamW[62] optimizer with a cosine decay learning-rate scheduler.
We set batch size of 512 and 256 for
DHVT-T and DHVT-S
when training on CIFAR-100, an initial learning rate of 0.001, and a weightdecay of 0.05, warm-up epoch of 5.
When on ClipArt, Sketch and Painting, we use batch size of 256and 128 respectively for DHVT-T and DHVT-S, the initial learning rate of 0.001, warm-up epoch of20 and weight decay of 0.05.
For ImageNet-1K, we use the batch size of 512 for both models andinitial learning rate of 0.0005 and weight decay of 0.05, warm-up epoch of 10.
All of the trainingdevices are Nvidia 3090 GPUs.
We use Pytorch tools and our code is modified from timm 1 .
[Table]
4.2 Performance Comparisons
Results on CIFAR-100.
We mainly compare the performance of our proposed model on CIFAR-100.
Patch size set to 1 means taking raw pixel input.
For comparison of other methods, we directylycite the results reported in the corresponding paper.
The results of our model are the best out of fiveruns with different random seed.
As is shown in Table 2, our model DHVT-T reaches 83.54 withonly 5.8M parameters.
and DHVT-S reaches 85.68 with only 22.8M parameters.
With much less parameters, our model has a much higher performance against other ViT based models and strong
CNNs ResNeXt, SENet, SKNet, DenseNet and Res2Net.
We not only bridge the performance gapbetween CNNs and ViTs, but also push the state-of-the-art result to a higher level.
[Table] [Table]
Results on DomainNet.
We also conduct experiments on other small datasets.
Here we choose threedatasets from DomainNet as our target.
We use the implementation of ResNet-50 in Pytorch officialcode for performance comparison.
All of the data-augmentations, such as Mixup [65] and CutMix[66] and AutoAugment [67], are also adopted for training ResNet-50 from scratch on these datasets.
All of the results reported are the best out of four runs.
As is shown in Table 3, our model showsbetter results than standard ResNet-50, demonstrating its performance across different small datasets.
Results on ImageNet-1K To test the train-from-scratch performance of our model on commonmedium size dataset ImageNet-1K, we also conduct experiments on it.
We follow the same experimental settings as in DeiT [33].
The results are shown in the Table 4.
Surprisingly, our DHVT-Treaches 76.47 accuracy and our DHVT-S reaches 82.3 accuracy.
As far as we know, this is thebest performance under such non-hierarchical vision transformer structure with class token.
Andour model outperforms many of the state-of-the-art methods with comparable parameters.
Thisexperiment shows that our model not only behaves well on small datasets, but also exhibits powerfulperformance on larger datasets.
We will show the performance comparison with other methodstraining from scrach on ImageNet-1K in the supplementary materials.
4.3 Ablation Studies
All the results in the ablation study are the average over four runs with different random seed.
Themodel for ablation study is DHVT-T, with patch size of 4 and training from scratch on CIFAR-100with same data-augmentation as in Section 4.2.
Here DHVT-T is trained with learning rate of 0.001,warm-up epoch of 10 and batch size of 512, total epoch of 300.
The baseline is DeiT-T with 4 headsand the patch size is set to 4.
The results are shown in the following tables.
[Table]
[Table]
(cid:37) (cid:33) 69.
(+1.51) (cid:37) (cid:33) (cid:33) 80.
(+13.26)
The importance of positional information.
We have baseline performance 67.59 from DeiT-T with4 heads, training from scratch with 300 epochs.
When removing absolute positional embedding,the performance drops drastically to 58.72, demonstrating the importance of position informationin vision transformers.
SOPE is able to provide positional information to some extent because suchabsolute positional information can be derived from the zero padding.
As is shown in Table 5,when adopting SOPE and removing absolute position embedding, the performance does not drop sodrastically.
But only depending on SOPE to provide position information is not enough.
The role of DAFF.
When adopting DAFF, the performance gain increases greatly to 79.47, because
DAFF solve the problem on both spatial and channel aspect, introducing strong local constraints and re-calibrating channel feature representation.
It is sensible to see that removing absolute positionembedding can increase performance.
The positional information have been encode into tokensthrough the depth-wise convolution in DAFF, and the absolute position embedding will breaktranslation invariance.
When both SOPE and DAFF are adopted, the positional information will beencoded comprehensively, and SOPE will also help address non-overlapping problem here, preservingfine-grained low-level feature in early stage.
The role of head tokens.
From Table 6, we can also see the stable performance gain brought by headtokens across different model structure.
When introducing head tokens into DeiT-T, the performancegets a +1.51 gain, demonstrating its effectiveness.
As we said before, head tokens guarantee theinteraction among different channel groups, better fusing the diverse representation.
The resultingintegral representation is now strong enough for classification.
When adopting all three modifications,we get +13.26 accuracy gain, successfully bridging the performance gap with CNNs.
[Figure] [Figure]
[Figure]
We visualize the attention maps of head tokens to patch tokens in Fig.
Each row represents oneimage.
The results are samples in the second encoder layer.
We can see that different head tokenactivates on different patch tokens, exhibiting its diverse representation.
On such low layer, low-levelfine-grained feature is able to be captured in our model.
More visualization results are shown in thesupplementary materials.
[Figure]
4.5 Limitation
Though we achieve a much higher performance than existing methods, such performance gain comesat the expense of computation.
The performance when patch size set to 2 boosts higher than usingpatch size of 4.
But the computation expense rises quadratically.
In practical usage, we suggestchoose a good patch size for better trade-off between performance and computation.
We will showthe FLOPs and throughput in the supplementary materials.
Conclusion
In this paper, we present an alternative vision transformer architecture DHVT, which can trainfrom scratch on small datasets and reach state-of-the-art performance on series of datasets.
Theweak inductive biases of spatial relevance and diverse channel representation brought by insufficienttraining data is strengthened in our model.
The highlighted head token design is able to transferred tovariants of ViT model to enable better feature representation.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes]
[Table]
) Did you describe any potential participant risks, with
[Table]
to Institutional Review
[Table]
(IRB) approvals, if applicable?
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[Yes
[Table]
A Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
This section will often be part of the supplemental material.