Adaptive Attention Link-based Regularization for Vision Transformers
Anonymous Author(s)
AffiliationAddress email
Abstract
Although transformer networks are recently employed in the various vision taskswith the outperforming performance, large training data and a lengthy trainingtime are required to train a model to disregard an inductive bias.
Using trainablelinks between the channel-wise spatial attention of a pre-trained Convolutional
Neural Network (CNN) and the attention head of Vision Transformers (ViT), wepresent a regularization technique to improve the training efficiency of ViT. Thetrainable links are referred to as the attention augmentation module, which istrained simultaneously with ViT, boosting the training of ViT and allowing it toavoid the overfitting issue caused by a lack of data.
From the trained attentionaugmentation module, we can extract the relevant relationship between each CNNactivation map and each ViT attention head, and based on this, we also propose anadvanced attention augmentation module.
Consequently, even with a small amountof data, the suggested method considerably improves the performance of ViT whileachieving faster convergence during training.
Introduction
Convolutional Neural Networks (CNN) have become standard for solving image-related tasks usingdeep neural networks since the advent of large publicly available datasets [1, 2].
Recently, modelswith attention mechanisms mainly adopted in the area of natural language processing are becomingto take part in solving image-based tasks, which is called the Vision Transformer (ViT) [3, 4].
ViT isa transformer-based neural network fed by the patches of images with class-token for classification,replacing its input of the embedded words in natural language processing.
Although ViT showsimpressive accuracy compared to modern CNNs by ignoring the inductive bias of locality, a significantamount of data is required to train the model to achieve satisfactory performance without overfittingissues.
Furthermore, most researchers with the limited computing hardware are not affordable to trainthe ViT due to its lengthy training time.
The overfitting and lengthy training issues must be solved to broaden the usability of ViT, so manyrecent studies have tried to solve the problem [5, 6, 7, 8, 9, 10].
We can divide the studies by threecategories: the advanced architecture-based method [4, 8, 11, 12], the parameter compression-basedmethod [13, 14], and the knowledge distillation-based method [6, 15].
The advanced architecturebased methods manipulate the architecture of ViT to achieve improved training efficiency andgeneralized prediction even with the small dataset.
On the other hand, the parameter compressionbased methods focus on a low-rank approximation of the transformer encoder in ViT, which resultsin the boosted training speed and the suppressed overfitting issue.
The knowledge distillation-basedmethods utilize the prediction of additional CNN models to avoid the overfitting problem and achieverapid training convergence.
The previous studies have shown the meaningful development of ViT forthe small dataset and the reduced training time.
However, the previous studies have the remaining limitations where either the architectural manipulation or the initial model fully trained by a target dataset is essential.
The architectural manipulation ofinitial ViT [3] cannot be applied to different versions of ViTs, hence limiting the handling of new ViT models.
Even though the knowledge distillation-based methods can be employed with the smallmanipulation of a model such as the knowledge distillation token, it should be assured to have theinitial models trained by a target dataset, which takes the additional costs for the acquisition of initialmodels before the training of main ViT model.
In this paper, we propose a novel regularization method to reduce the convergence time and avoid theoverfitting problem on a small dataset, simultaneously.
The proposed method utilizes an attentionaugmentation module containing multiple trainable weights that estimate the affinity between thechannel-wise activation map of CNN and the head-wise attention map of ViT. Since the attentionaugmentation module is located to regularize the attention of ViT heads, the architecture of ViT canbe perfectly preserved, which lets us enable to employ the proposed algorithm in ViT variants basedon the attention maps.
In addition, since the attention map can be obtained even when the task ofthe pre-trained CNN is not equivalent to the target task, we can employ our method without thepre-trained CNN model with the same target dataset.
We validate our regularization method by using
ImageNet and CIFAR10 datasets with various scenarios, which show the outperforming accuracyand the reduction of epochs required for its training convergence.
Furthermore, we investigate theimportant factors for ViT to avoid the overfitting issue by analyzing the trained weights of the attentionaugmentation module, and through the investigation, we present the dissimilarity of the deep layers’roles between CNN and ViT.
We can summarize our contributions as following:
• We propose a novel regularization method to resolve the issues of overfitting and lengthytraining time of ViT through the trainable attention links between the ViT attention mapsand CNN activation maps.•
The proposed scheme preserves the original architecture of ViT, which results in its generalemployment regardless of the architecture of ViT. • Through the proposed algorithm, the performance of ViT can be dramatically improved withthe limited size of dataset, and the training time is reduced without the loss of performancein various scenarios.
• The relationship between ViT and CNN is analyzed in terms of attentional regions, whichvalidates the analysis from the previous studies.
Related Work2.1 Transformers in Vision
Transformer models introduced by [16] are neural networks that purely utilize the attention mechanism.
While they have been used broadly in the field of natural language processing, Vision
Transformer (ViT) [3] adapted them in the domain of computer vision with minimal modification toits architecture.
ViT showed comparable performance to CNN in the condition of large pre-training.
For the advanced optimization of ViT, CaiT [4] used layer normalization in ViT layers and changedthe input location of class token to prevent saturation of performance in deep layers.
Swin Transformers [12] adopted a hierarchical transformer that computes shifted windows to make it suitable for thevision domain.
PiT [10] introduced the concept of pooling in ViT from CNN, improving the generalization of ViT. T2T-ViT [8] enhanced sample efficiency by reshaping input tokens and changingthe backbone of networks motivated by several CNN architectures.
Raghu et al.
[17] measured thesimilarity of representations between specific layers of CNN and ViT using centered kernel alignment.
With additional relative positional encoding, Cordonnier et al.
[18] proved attention mechanisms in
ViTs can perform as convolution layers in CNNs and showed their functional similarity.
From theinvestigation, ConViT [7] was motivated to use relative positional encoding to give locality – theinductive bias of CNNs – to ViT.
The research was extended to [19], reparameterizing pre-trained convolutional layers as a format of
ConViT. Refiner [11] tackled the over-smoothing problem between tokens in deep layers of ViT, andrelieve it by projecting attention heads into the higher dimensions and applying convolution directlyto attention maps to learn local relationship among the tokens.
Those variants of ViT improvedthe optimization and data efficiency of the initial ViT model by modifying the architecture itself.
[Figure]
However, our method does not touch any part of ViT modules, but only connects attention between
ViT and CNN, transferring attention through links to give ViT a learning signal from the teacher CNN.
2.2 Knowledge Distillation:
In knowledge distillation, a student model leverages a pre-trained teacher model’s soft predictiondivided by the same temperature values [20].
The softened predictions can be regarded as labelsmoothing, and by using them, the student model can achieve the data augmentation effect.
Distillationbetween different types of neural architectures has also been proposed, DistilBERT [21] showed theeffectiveness of a distilled knowledge from BERT [22] into LSTM [23].
DeiT [6] distilled knowledgefrom CNN to ViT, which seems similar to our work.
However, in contrast to the previous study usingthe prediction for the knowledge distillation, our framework transfers the knowledge based on thesimilarity of the latent feature maps.
As a result, we can extend the range of teacher models to coverthe models in which the prediction vectors differ from the prediction of the target task.
On the other hand, we can transfer latent representations of teacher models to those of student models.
FitNets [24] improved the stability of deep network training by guiding the latent layers to theteacher’s well-trained latent representation.
Zagoruyko et al.
[25] considered attention as projectedactivation maps of CNN into a spatial dimension, which could be regarded as spatial attention.
Theyshowed that spatial attention contains valuable information that is useful to improve the performanceof the student network.
Kim et al.
[26] used a paraphraser to extract and pass the teacher’s knowledgeto the student’s translator to learn its representation.
Meanwhile, Heo et al.
[5] demonstrated thatthe knowledge transfer based on the neurons’ activation is a more classification-friendly approachthan the direct transfer using output values.
Attention-based feature distillation [27] measured thesimilarities between teacher and student features through attention, which determines the importanceof knowledge to transfer.
Attention Link-based ViT Regularization
In this section, we first explain the backgrounds of the self-attention mechanism and ViT.
Then,we explain the method to extract the attention maps from ViT, followed by the description of the architecture and the training method of the augmented attention module is described.
The overallframework is depicted in Fig.
3.1 Background of ViT
We first explain the self-attention mechanism and the original ViT model referred to by [3].
Theself-attention mechanism mimics the human cognition system making the attention to the externalstimulus, which is designed by a transformer-based model with the attention matrix estimated bypairs of key and query.
Self-attention Mechanism: We define the input sequence by X ∈ R L × D in where L is the lengthof the sequence and D in means the dimension of one sequential element in the sequence.
Then, wecan estimate the elements of the attention mechanism composed of key, query, and value vectors bylinearly projecting the input sequence by the corresponding embedding weights W k , W q , and W v ,respectively.
Thus, when we define the key, query, and value vectors by Q , K , and V , respectively,we obtain the vectors as following:
[Equation]
where W ∈ R D in × D head from W ∈ { W k , W q , W v } and D head is the dimension of the headembedding.
Then, the self-attention of the head can be estimated by:
[Equation]
where s ( Z ) is a function to transfer each row vector of input matrix Z by its softmax probability.
According to the derivation, self-attention can consider the semantic dependency among sequentialinputs.
Many transformer-based models are based on the architecture stacked by the Multi-Head SelfAttention layers (MHSA) containing multiple self-attention heads with independent embeddingweights.
For the given input X , we define the output of m -th self-attention head at n -th level depthby f ( m,n ) ( X ) .
Then, we denote the corresponding key, query, and value vectors by K ( m,n ) , Q ( m,n ) , 140 and V ( m,n ) , respectively.
ViT Framework: The original ViT model directly employed the conventional transformer-basedmodel built for the natural language processing of the visual classification task.
We can summarizethe inference process of the original ViT as following.
At first, we divide an input image by P 2patches with the same size and sequentially order the patches after their vectorization.
Since thetransformer network is invariant to the order of the sequential data, ViT concatenates positionalembedding vectors to the input patches to represent the original position of the patch.
We define the sequential data obtained from one image by X 0 ∈ R P 2 × D im , where D im is thesize of the vector linearly projected from the vectorized image patch and the positional embeddingvector.
Before we feed X 0 into the transformer modules, the trainable class token sized by R D im issequentially connected ahead of X ′ 0 , which results in X 0 ∈ R
( P 2 +1)
× D im .
The transformer-based encoders of ViT are the modules containing the series of a layer normalization,a self-attention multi-head module, a fully connected layer, and a layer normalization, where everynormalization layer has a residual connection.
We define the serial process by a function of X n +1 =
g ( X n ) ∈ R
( P 2 +1)
× D where D is the size of latent vectors.
When N modules are stacked in thetransformer-based encoder, the class-wise score is estimated by linearly projecting the final output ofthe class token as following: p ( cls | X ) = softmax ( FC ( X N )) .
For a detailed explanation of ViT,you can be referred to [3].
3.2 Attention Map Extraction
We need to compare the ViT attention map and the CNN activation map for our regularization-basedalgorithm.
Instead of the relative positional embedding [7] or the attention bias [11], we preserve theoriginal architecture of ViT to generalize the usability of our framework to cover the various ViTvariants.
To obtain the attention map from the original ViT, we utilize the attention value between the classtoken and the image patch.
The class token takes a key role to determine the final prediction, sowe can assume that the attention to the class token may represent the importance of image patchesfor the classification result.
Thus, when feeding the class token into the transformer module as itsquery vector, we obtain the attention value by estimating the dot product between the embeddingvectors of the class token and the corresponding image patch.
Then, for m -th head in n -th multi-headself-attention layer, we can estimate the attention value as:
[Equation]
where Rec is a function to reconstruct the rectangular matrix of P × P from its input vector of P ∈according to the order of the sequential patches, and Q ( m,n ) [0] and K ( m,n ) [1 :] represent the firstquery vector of the class token and the key vectors of the image patches, respectively.
In the case of the CNN activation map, we extract the activation maps after the normalization of everyconvolution block.
Instead of integrating the channel-wise activation maps, we consider the separatedactivation maps independently to improve the degree of freedom of our attention augmentationmodule.
In contrast to the constant resolution of ViT attention maps, the resolution of the CNNactivation maps decreases with deep layers by pooling layers and strides of convolution layers.
Thus,to preserve the resolution of every activation map, we resize all the CNN activation maps to have thesame size as the ViT attention maps.
We define the c -th resized CNN activation map by B c , wherec ∈ { 1 , ..., C } and C is the number of entire CNN activation maps.
3.3 Attention Augmentation Module
3.3.1 Module Architecture
Even though both the CNN activation and ViT attention maps represent the key parts of the targetobject for the prediction, their distribution such as a center point and a variance would be differentfrom each other due to the dissimilarities of their operations.
For example, while the ViT attention mapis distributed between 0 and 1 because of the softmax estimation, the values in the CNN activationmap are normalized by a batch normalization, which can contain negative values.
Furthermore, ingeneral, the number of CNN activation maps is much larger than the number of ViT attention mapsdue to the large channel-wise depth of CNNs.
As a result, it is impossible to directly compare each ofthe CNN activation maps with the ViT attention maps.
The attention augmentation module is designed to solve the problems of different distributions and avarying number of maps.
We design the attention augmentation module to contain multiple attentionlinks which are the trainable weight parameters to scale the ViT attention maps.
By estimating theweighted summation of ViT attention maps with the attention links, we can obtain the augmentedattention maps where the number is equivalent to the number of CNN activation maps.
Thus, we canestimate the augmented attention maps as following:
[Equation]
where w ( m,n ) c and b c are the attention link and a trainable bias for c -th augmented attention map( c ∈ { 1 , ..., C } ), respectively.
Note that the weight of attention link w ( m,n ) c is used to analyze thestrength of connectivity for each CNN/ViT layer in section 4.1.
We implement the augmented attention module by a 1 × 1 convolution layer generating C augmentedattention maps from a tensor of R P 2 × MN where the ViT attention maps are stacked.
Because weonly use the augmented attention maps only for the training loss, the attention augmentation modulehas no role in the inference, which can be removed after the training of ViT.
3.3.2 Module Training
By using the augmented attention module, we can obtain the same number of augmented attentionmaps A + c with the CNN attention maps B c .
To ignore the remaining scale gap between the twomaps, we first apply the l 2 normalization, and then the mean squared error is estimated to build the [Table] 1 ∼ [Table]
4 ∼4 ∼6 ∼7 ∼
[Table]
1 ∼1 ∼3 ∼4 ∼4 ∼6 ∼
[Table]
∼
[Table]
attention-based regularization loss as:
[Equation]
Then, we integrate the attention-based regularization loss L att with the cross-entropy loss L CE oforiginal ViT as:
[Equation]
where λ is a scaling factor to control the effect of our regularization.
Since the regularization loss canwork as an obstacle to ignoring the inductive bias, referred by [7], we suppress the value of λ at thespecified epochs to increase the effectiveness of the cross-entropy loss.
We exponentially decay thevalue of λ by multiplying a decay constant between 1 and 0 at every epoch.
Link Selection for Advanced Regularization
In this section, we build the advanced architecture of the attention augmentation module based on theanalysis of the fully-trained attention links.
After showing the resultant attention links, we explain theadvanced link designed by considering the relation between the CNN activation and the ViT attentionmaps.
4.1 Analysis of Resultant Attention Links
We visualize heat maps to show the scale distribution of the attention links after their training.
To compare the relationship between the ViT attention and the CNN activation maps, we onlyconsider the magnitude of the weight parameters in the attention links.
As shown in Fig.
2, we obtainmultiple heat maps by using three datasets including ImageNet [28], a 5% subset of ImageNet , and
Cifar-10 [29].
As analyzed in many previous studies [18, 17], the ViT attention maps are highly related to the CNNactivation maps located at a similar level.
The results validate that the self-attention multi-heads of
ViT can train the hierarchical information by the stacked architecture, which is similar to the trainingmechanism of CNN.
Thus, the CNN activation maps would be helpful to regularize the ViT attentionmaps if we can find their level similarity.
In addition to the well-known hypothesis, the heat maps from the attention links show the interestingcharacteristic where the high-level heads are not regularized when the large dataset is given.
Comparedto the small datasets including Cifar-10 and the subset of ImageNet , we could see the suppressedmagnitudes of the attention links at high-level heads when large dataset (i.e. ImageNet ) is given.
Thediscovery exploits the high-level heads should not be trained by the regularization of the high-level CNN layers, which verifies the high-level heads can represent more complicated semantic informationthan CNN layers.
In other words, the representation can be seen out of the inductive bias of CNNs, sowe can show the inductive bias is hard to be trained without a large dataset.
DeiT [6] and Swin Transformers [12] showed that the employment of the inductive bias of CNN iseffective for the training of ViT. At the end of the training, we observed that high-level heads aredisconnected from augmented attention maps, which means they are no more regularized by CNNactivation maps.
This indicates that high-level heads would escape the inductive bias of locality, andthey are trained by a long-range dependency that cannot be acquired by CNNs.
We can summarize the two hypotheses from the analysis as following: [Figure] Methods DeiT-S
ConViT-S
Proposed(Full-link)
Proposed( [Figure] Epochs • The ViT attention and the CNN activation maps are similar to each other at a similar level.
• The high-level ViT heads can present the semantic information that cannot be representedby the CNN layers, but training the semantic information requires a large dataset.
4.2 Selective Attention Link
Based on the analysis, we additionally propose the selective attention link to improve the trainingefficiency of the attention augmentation module.
Instead of the full link in the original attentionaugmentation module, only a part of the attention links are utilized to obtain the augmented attentionmaps.
Based on the two hypothesis from our analysis, the augmented attention map is generatedonly by the ViT attention maps with the similar levels, and no link is connected to the high-level ViTattention maps when the training dataset is sufficiently large.
Accordingly, we build two types of selective attention link, which are denoted by α -link and β -link.α -link connects the ViT attention maps to only the augmented attention maps at a similar level.
β -linkis similar to the α -link but the links to the high-level heads are entirely disconnected.
The detailedconnections are given in Table 1.
To show that our selective attention link-based transfer efficiently matches ViT attention maps with
CNN activation maps, we compared two different settings on the attention augmentation module. One fully connects each ViT attention map to produce augmented attention maps that match CNNactivation maps channel-wise as one-to-one correspondence. In the case of the full ImageNet dataset,β -link was used for the selective link, while the α -link was utilized for the other small datasets. Asshown in Table 2, while attention transfer with a fully connected attention link has already shownremarkable results, using a selective attention link has brought even higher final test classificationaccuracy when given limited samples.
For selective links in Table 2, β -link is used for the ImageNetfull dataset, and α -link is used for partial datasets.
In experiments, we showed that transferring attention from pre-trained CNN models to ViTs can inject
CNN’s inductive bias (i.e locality) naturally in standard self-attention layers, without the necessityof additional modules extending the self-attention network.
Thus, we examine how efficiently themethod helps ViT to be converged for optimal performance, especially showing a large gap in a smalldata regime.
Training Details: The computing resource used in our experiments is Nvidia A100.
If not mentioned otherwise, the student ViT model used for experiments is DeiT-S(distilled version) and used EfficientNet-B3 [30] as the teacher CNN model.
To validate the effectiveness of the proposedframework in the limited data and computational cost, we used the same architecture setting as
DeiT-S which used less number of attention heads but the same dimension on each head.
Note
[Table] [Table]
that EfficientNet-B3 shows lower test accuracy on ImageNet than the teacher model of DeiT [6], RegNetY-16GF(84M parameters)
We were able to apply the proposed method to the distilledversion of DeiT-S since our method can be jointly applied with class prediction-based distillation.
Weadd results with different teacher/student models to show our method is also applicable to the otherscenario in table 5.
We validate that DeiT-B of our student ViT model can overwhelm the teachermodel of EfficientNet-B3.
When we reduce the setting by DeiT-S and ResNet34 as student and teachermodels, respectively, we can verify our method show the synergy consistently.
For a fair comparison,we based our hyperparameters and training strategies from DeiT [6].
For our experiments, the decayconstant for λ is set to 0.99 for the first 200 epochs and 0.98 for the last 100 epochs.
Faster training with better data efficiency: To demonstrate that our method relieves data dependencyof ViT training, we compare it with another distillation method DeiT which also targeted to train
ViT more data efficiently using CNN teacher’s output.
When small training data is given, a largemargin of improvement was achieved by using the selective attention link. As shown in Tables 7, 4,even with the same number of parameters as DeiT, our method outperforms DeiT and ConViT-S,which had 5M more parameters.
Our additional trainable module, which is the attention augmentationmodule, includes only a single 1x1 Conv layer which augments the attention maps of the student ViT.
In our default settings, the number of the parameter is 0.068M, which is quite small compared to
DeiT-S of 22M parameters.
As we mentioned, the lengthy training time of ViT is a critical drawbackespecially when the computational resources are limited.
While higher test accuracy with limitedtraining epochs infers better data efficiency when data augmentation is used, we compared othermethods that also focused on sample efficiency of training ViT in Table 3. We also experimented on CIFAR10 dataset, another small-sized but traditional image classificationdataset. The result is shown in Table 7.
In the DeiT paper, 7200 training epochs were needed toachieve 75.0% top-1 test accuracy when training from scratch using the DeiT-B model which hasmore attention heads than DeiT-S. On the other hand, our method only needed to train 300 trainingepochs to reach the same test accuracy while using the DeiT-S model. Also, since our method doesnot need any class prediction about the target dataset, we could use a teacher network that is trainedon the ImageNet dataset. We also compared ConViT with our method, with the same training strategy.
Ablation studies: For additional verification of our knowledge transfer method, we trained on CIFAR10 with different scenarios.
To sternly check the performance difference of knowledge distillationeffect from each method, we used much weaker data augmentation than the setting used in otherexperiments, with only simple techniques such as random-crop and horizontal flips.
This allows us toconfirm the data efficiency in a low data regime.
As presented in Table 6, we compared our methodto other knowledge distillation methods introduced by DeiT with a teacher model pre-trained on the Cifar10 dataset.
For both soft label distillation and hard label distillation, our method outperformedthe class prediction-based distillation method.
From this result, we could infer that directly transferring attention gives a better learning signal than giving the teacher model’s output predictions.
Furthermore, We could confirm that knowledge earned from a larger dataset can give a better learningsignal.
Using the teacher model pre-trained on ImageNet showed better test accuracy than using the
[Table]
[Table]
[Table] [Table] teacher model trained on CIFAR10. Note that the results of the former settings are shown at Table 7and the latter at Table 6.
This could be another advantage since teacher models in DeiT are restrictedto be trained on the target dataset to give proper output prediction.
Lastly, our method showed synergywith DeiT. When both signals from attention and output vector were used during training, it showedeven better test accuracy than only using attention transfer.
Since both output vector and attention inintermediate layers could be fetched by a single forward propagation, an increase in computation costis marginal. Versus label-based distillation & fine-tuning : Our method can be jointly applied with classprediction-based distillation. Thus, both Table 4 and Table 7 verify that our method performs betterthan the distillation method proposed in [6]. We perform the additional comparison to fine-tuningwith self-supervised learning(SSL) for ViT[32], and SSL with linear classifier and SSL with k-NNclassifier show 77% and 74.5% for ImageNet top-1 test accuracy, respectively.
Both results are lowerthan our accuracy of 81.0% with the same ViT model.
Architectural changes are not prohibitive: We show that applying our method is not only limitedto standard ViT in Table 8.
We employed our method to Pooling based ViT [10]. We observed thesample efficiency of the model increased by a large margin using our method.
Robustness to Random Initialization: We perform several trials with different random seeds asshown in Table 9.
Our algorithm shows consistency even with the various initial parameters.
Due tolimited computing, we run 240 epochs of training, unlike 300 epochs of training in other experiments.
Conclusion
In this paper, we have introduced a novel method of transferring knowledge from CNN to ViT. Byaccessing attention of CNNs and adaptively adopting them, student ViT was able to earn high qualityof learning signal with CNN’s inductive bias.
By applying our method, we could train ViT in lesstraining epochs without overfitting even with the small dataset or limited labeled data.
Also, werevealed relations between intermediate representations from those different types of neural networks,which varied due to the training dataset.
Furthermore, by analyzing those relationship with trainedattention links, we could take advantage of more efficient connection between networks.
We leavewider application of our methods to new ViT architectures in future works.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 422 answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You arestrongly encouraged to include a justification to your answer , either by referencing the appropriate section of 424 your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data are proprietary.•
Did you include the license to the code and datasets?
[N/A]Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist 429 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.1.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Section 1(b)
Did you describe the limitations of your work?
[Yes] See Section 6(c)
Did you discuss any potential negative societal impacts of your work?
[No] Our research do nothave any potential negative societal impacts(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 43.
If you ran experiments...(
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section
(c) Did you report error bars (e.g., with respect to the random seed after running experiments [Table] multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Section
[Table]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section [Table]
(b) Did you mention the license of the assets?
[Yes] See Section
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] There are
[Table]
new assets [Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
[Table]
ing/curating?
[No] I only use open sources
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- [Table]
tion or offensive content?
[No]
[Table]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
[Table] approvals, if applicable?
[No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on [Table] participant compensation?
[No]