Logit Margin Matters: Improving Transferable Targeted Adversarial Attack by Logit Calibration
Abstract
Previous works have extensively studied the transferability of adversarial samplesin untargeted black-box scenarios.
However, it still remains challenging to craftthe targeted adversarial examples with higher transferability than non-targetedones.
Recent studies reveal that the traditional Cross-Entropy (CE) loss function isinsufficient to learn transferable targeted perturbations due to the issue of vanishinggradient.
In this work, we provide a comprehensive investigation of the CE functionand find that the logit margin between the targeted and non-targeted classes willquickly obtain saturated in CE, which largely limits the transferability.
Therefore,in this paper, we devote to the goal of enlarging logit margins and propose twosimple and effective logit calibration methods, which are achieved by downscalethe logits with a temperature factor and an adaptive margin, respectively.
Both ofthem can effectively encourage the optimization to produce larger logit margins andlead to higher transferability.
Besides, we show that minimizing the cosine distancebetween the adversarial examples and the targeted classifier can further improvethe transferability, which is benefited from downscale logits via L2-normalization.
Experiments conducted on the ImageNet dataset validate the effectiveness of theproposed methods, which outperforms the state-of-the-art methods in black-boxtargeted attacks.
The source code of our method is available at Link.
Introduction
In the past decade, deep neural networks (DNNs) have achieved remarkable success in variousfields, e.g. , image classification [24], image segmentation [19] and object detection [23].
However,
Goodfellow et al.
[5] reveal that the DNNs are vulnerable to adversarial attacks, in which addingimperceptible disturbances to the input can lead the DNNs to make an incorrect prediction.
Manyfollowing approaches [3, 4, 1, 27, 29] have been proposed to construct more destructive adversarialsamples for investigating the vulnerability of the DNNs.
[5, 18] also show that the adversarial samplesare transferable across different networks, raising a more critical robustness threat under the black-boxscenarios.
Therefore, it is vital to explore the vulnerability of the DNNs, which is very useful fordesigning robust DNNs.
Currently, most of the works [3, 29, 16, 10, 28, 6] have been devoted to the untargeted black-boxattack, in which adversarial examples are crafted to fool unknown CNN models to predict unspecifiedincorrect labels.
For example, [3, 29] leveraged input-level transformation or augmentation toimprove the non-targeted transferability.
[10] proposed a powerful intermediate feature-level attack.[28, 6] demonstrated that backpropagating more gradients through the skip-connections can increasethe transferability.
Despite the success in non-targeted cases, the targeted transferability remainschallenging, which requires eliciting the black-box models into a pre-defined target category label. 300 Iterations L o g i t s
Top-1Top-2Top-3Margin (a) Cross-Entropy     300 Iterations
L o g i t s
{+Top-1Top-2Top-3Margin (b) Po+Trip     300 Iterations
+}
L o g i t s Top-1Top-2Top-3Margin (c) Logit For learning the transferable adversarial samples in untargeted cases, most methods have leveragedthe Cross-Entropy (CE) as the loss function.
However, [15, 30] recently showed that the CE loss isinsufficient for learning the adversarial perturbation in the targeted case due to the issue of vanishinggradient.
To deal with this issue, [15] adopt the Poincaré distance to increase the gradient magnitudeduring the optimization adaptively.
[30] demonstrated that an effortless logit loss equal to the negativevalue of the targeted logits could alleviate the gradient issue and achieve surprisingly strong targetedtransferability.
Besides, [30] also showed that optimizing with more iterations can significantlyincrease the targeted transferability.
Although [30] demonstrated that continually enlarging the logitsof the targeted class can improve the transferability of adversarial samples, it still does not thoroughlyanalyze the insufficient issue in the CE loss function.
In this study, we take a closer look at the vanishing gradient issue in the CE and find that thelogit margin between the targeted and non-targeted classes will quickly get saturated during theoptimization (as shown in Fig. 1(a)).
Moreover, this issue will influence the performance of theperturbations and thus essentially limit the transferability.
Specifically, along with the trainingiterations in CE, we observe that the logits of the targeted and non-targeted classes increase rapidly inthe first few iterations.
However, after reaching the peak, the logit margin between the targeted andnon-targeted classes will get saturated, and further training will decrease the logits simultaneouslyto maintain this margin.
This phenomenon is mainly due to the fact that the softmax function in CE will approximately output the probability of the target class to 1 when reaching the saturatedmargin ( e.g. , 10).
Thus, it raises the problem that the transferability will not be further increased evenoptimized with more iterations.
While in practice, we are encouraged to increase the transferabilityby maximizing both the logit for the targeted class and its margin against other non-targeted classesto cross the decision boundaries of other black-box models.
In this paper, we devote to enlarging logit margins to alleviate the above saturation issue in CE.
Inspired by the temperature-scaling used in the knowledge distillation [8], a higher temperature Twill produce a softer probability distribution over different classes.
We firstly leverage this scalingtechnique into the targeted adversarial attack to calibrate the logits.
Then the logits margin betweenthe targeted and non-targeted classes will not be saturated after only a few iterations and will keepimproving the transferability.
On the other aspect, instead of using a constant T , we further exploredan adaptive margin-based calibration by scaling the logits based on the logit margin of the targetclass and the highest non-target class.
In addition, we also investigate the effectiveness of calibratingthe targeted logit into the unit length feature space by L2-normalization, which is equivalent tominimizing the angle between the adversarial examples and the targeted classifier.
Finally, we conduct experiments on the ImageNet dataset to validate the effectiveness of the logitscalibration for crafting transferable targeted adversarial examples.
Experimental results demonstratedthat the calibration of the logits helps achieve a higher attack success rate than other state-of-the-artmethods.
Additionally, we tested the logit calibration in Generative Adversarial Networks (GANs)-based TTP method [21] to train the target-class-specific generators, which is also beneficial forincreasing the transferability in the resource-intensive method.
Related Works
In this section, we give a brief introduction of the related works from the following two aspects:untargeted black-box attacks and targeted attacks .
2.1 Untargetd Black-box Attacks
After [25] exposed the vulnerability of deep neural networks, many attack methods [29, 4] have beenproposed to craft highly transferable adversaries in the non-targeted scenario.
We first review severalgradient-based attack methods that focus on enhancing the transferability against black-box models.
Iterative-Fast Gradient Sign Method (I-FGSM)
[14] is an iterative version of FGSM [5], whichadds a small perturbation with a small step size α in the gradient direction iteratively:
[Equation]
where ˆ x ′ i denotes the adversarial image in the i th iteration, α = ϵ/T ensures the adversaries beconstrained within an upper-bound perturbation ϵ through the l p -norm when optimized by T iterations.
Following the seminal I-FGSM [14], a series of methods have been proposed to improve the transferability of attacking black-box models from different aspects, e.g. , gradient-based, input augmentationbased.
For example, the Momentum Iterative-FGSM (MI-FGSM)
[3] introduces a momentumterm to compute the gradient of the I-FGSM, encouraging the perturbation is updated in a stabledirection.
The Translation Invariant-FGSM (TI-FGSM) [4] adopts a predefined kernel W to convolve the gradient ∇ ˆ x J (ˆ x ′ i , y ) at each iteration t , which can approximated the average gradient overmultiple randomly translated images of the input ˆ x t .
On the other aspects, the Diverse Input-FGSM(DI-FGSM) leveraged the random resizing and padding to augmentation the input ˆ x t at each iteration.
Currently, most targeted attack methods [15, 30, 21] simultaneously use the MI, TI and DI to form astrong baseline with better transferability.
2.2 Targeted Attacks
Targeted attacks are different from non-targeted attacks, which need to change the decision to aspecific target class.
[13] integrates the above non-targeted attack methods into targeted attacks tocraft targeted adversarial examples.
However, the performance is limited because it is insufficient tofool the black-box model only by maximizing the probability of the target class with the CE loss.
Po+Trip [15] found the insufficiency is mainly due to vanishing gradient issue in CE.
Then, [15]leverage dthe Poincaré space as the metric space and further utilized Triplet loss to improve targetedtransferability by forcing adversarial example toward the target label and away from the ground-truthlabel.
To further address this gradient issue, Logits [30] adopts a simple and straightforward idea bydirectly maximizing the target logit to pull the adversarial examples close to the target class, whichcan be expressed as:
[Equation]
where z t ( · ) is the output logits of the target class.
On the other hand, many studies employ resource-intensive approaches to achieve targeted attack,which train target class-specific models (auxiliary classifiers or generative models) on additionallarge-scale data.
For example, the FDA methods [12, 11] used the intermediate feature distributions of CNNs to boost the targeted transferability by training class-specific auxiliary classifiers to model layerwise feature distributions.
The GAP [22] trained a generative model for crafting targeted adversarialexamples.
Subsequently, [20] adopted a relativistic training objective to train the generative modelfor improving attack performance and cross-domain transferability.
Recently, the TTP [21] utilizedthe global and local distribution matching for training target class-specific generators for obtaininghigh targeted transferability.
However, the TTP requires actual data samples from the target classand brings expensive training costs.
Different from the above methods, we introduce three simpleand effective logit calibration methods into the CE loss function, which can achieve competitiveperformance without additional data and training.
Method
Problem Definition Given a white-box surrogate model F s and an input x not from the targetedclass t , our primary goal is to learn an imperceptible perturbation δ that can fool the F s to output thetarget t for ˆ x = x + δ .
Besides, the prediction of ˆ x will also be t when feeding to other unknownblack-box surrogate models.
The l ∞ -norm is usually used to constrained the δ within an upper-boundϵ , denoted as || δ || ∞ ≤ ϵ .
For the surrogate model F s , we denoted the feature for the final classification layer of the input x asϕ ( x ) .
The logit z i of a category i is computed by z i
=
W Ti ϕ
( x ) + b i , the W i and b
i are the classifierweights and bias.
The corresponding probability p i after the softmax is p i =
e zi (cid:80) e zj .
3.1 Logit Margin
When successfully attacked the F s , the logit z t of the target class will be higher than the logits z nt ofany other non-target class in the classification task.
Their logit margins can be computed by,
[Equation]
[15, 30] showed that it is insufficient to obtain transferable targeted adversarial samples that are onlyclose to the target class while not away from true class and other non-targeted classes.
Based on thisproperty, it encourages us to continually enlarge this logit margin to increase the separation betweenthe targeted and other non-targeted classes.
To have a better understanding of the relationship between the logit margins and the targetedtransferability, we visualize the average Top-3 logits (1 targeted class and other two non-targetedclasses) of 50 random adversarial samples trained for crafted ResNet50 by the CE, Po+Trip, and
the
Logit loss functions with MI, DI and TI following [30].
We also compute the average logit marginof the targeted class against the Top-20 non-targeted classes.
The logit and margin are shown in
Figure 1, and the transferability from ResNet50 to VGG16 is plot in Figure 2.
[Figure]
From Figure 1, we can observe that the logits of the targeted class and the Top-2 non-targeted classes increaserapidly in the first few iterations for the CE and Po+Triploss, as well as their logit margins.
When reaching thepeak, the margin is saturated, and the logits start to decrease simultaneously to maintain the saturated margin.
By comparing the CE and Po+Trip, the Po+Trip needsfewer more iterations to reach the saturated status andthus shows a marginal better transferability than CE, asshown in Figure 2.
In comparison, the Logit loss functionwill keep increasing the logits of the targeted categoryand the logit margin.
Thus, the Logit loss function showsa much better targeted-attack success rate than CE and
Po+Trip.
On the other hand, the Logit loss also significantly increases the logits for other non-targeted classes.
To further analyze why the CE loss function saturatedto this logit margin and explore the effectiveness of increasing the margin during training, in the following sections, we will revisit the cross-entropy lossfunction and introduce the logit calibration to achieve this goal.
3.2 Revisiting the Cross-Entropy Loss
Firstly, our objective is to maximize the logit margin in Eq.
After computing the gradient w.r.t.
toϕ (ˆ x ) , we can get
[Equation]
This gradient indicates that the adversarial feature ϕ (ˆ x ) needs to move towards the target class whileapart from those non-target classes.
Next, we compute the gradient w.r.t.
to ϕ
(ˆ x ) in the Cross-Entropyloss function
[Equation]
and get the gradient ∂L ce ∂ϕ (ˆ x ) as
[Equation]
From Eq.
6, we actually can find the CE loss function is designed to adaptively optimize the ϕ (ˆ x )towards W t and away from other W i .
However, after being optimized for several iterations, the p i ofthe non-targeted class will quickly approximate to 0 and then significantly vanish the W t − W i .
Let’s consider the case only with 2 classes ( t and nt ), we have the probabilities p t and p nt as:
[Equation] [Equation]
As shown in Figure 3, the p t will get close to 1 when z t − z nt > 6 ( e.g. , p nt ≈ 2 e − 9 whenz t − z nt = 20 ).
In such a context, the gradient will significantly vanish.
Recall that, in the CEloss function (Figure 1 (a)), the logit margin between Top-1 and Top-2 logits first increases rapidlybut will reach saturated status when approaching a certain value.
This further indicates that theoptimization of the CE loss function is largely restrained when the logit margin reaches a certain value.     {+10 z t − z nt 0 .0 .0 .
+}
.p t To this end, we raise the question if we explicitly enforce theoptimization to enlarge the logit margin ( z t − z nt ), could we getbetter transferable targeted adversarial samples?
To answer this, we propose to downscale the z t − z nt by a factors in the CE and extent the informative optimization for moreiterations.
Since in such circumstance, z t − z nt will be enlargerby the factor s .
Specifically, suppose that the optimization will besaturated when z t − z nt reaches a certain value v .
Using z t − z nt 186 and z t − z nt s in the CE will both approach the saturated value of v .
Then, it is easy to infer that, for the latter case, z t − z nt will bev × s .
3.3 Calibrating the Logits
To downscale the z t − z nt during the optimization, we investigate three different types of logitcalibrations in this study, i.e. , Temperature-based, Margin-based, and Angle-based. 3.3.1 Temperature-based
Inspired by the Temperature-scaling used in the Knowledge distillation [8], our first logit calibrationdirectly downscale the logits by a constant temperature factor T ,
[Equation]
After introducing the T , the probability distribution p will be more softer over different classes.
Thecorresponding gradient can be compute by:
[Equation]
The ˆ p i will not quickly approach to 0 after only a few iterations.
In Figure 4 (a)(b), we visualized the logits of using T = 5 and T = 20 .
We can find that targetedlogits and the logit margin will keep increasing as the same as the Logit in Figure 1.
Meanwhile, thetrend of T = 20 is very similar with the Logit [30] and we show that the Logit loss function can beconsidered as a special case of calibrating the logits with a large T in the supplementary. 300 Iterations L o g i t s
{+Top-1Top-2Top-3Margin (a) T=      300 Iterations
+}
L o g i t s
{+Top-1Top-2Top-3Margin (b) T=      300 Iterations
+}
L o g i t s Top-1Top-2Top-3Margin (c) Margin 3.3.2 Margin-based
The previous Temperature-based logit calibration contains a hype-parameter T , which could bedifferent for different surrogate model F s .
To migrate this issue, we further introduce an adaptivemargin-based logit calibration.
Specifically, we calibrate the logits by using the margin between the
Top-2 logits in each iteration, denoted as:
[Equation]
(11) where ˆ z 1 and ˆ z 2 are the Top-1 and the Top-2 logit, respectively.
In this Margin-based logit calibration, we will enforce the p t and p ˆ1 of the Top-1 non-target class ateach iteration meet the following constraints:
[Equation]
[Equation]
[Equation]
Then, it can adaptively deal with the vanishing gradient issue in the original CE loss function.
Thelogits and the margin is shown in Figure 4 (c).
3.3.3 Angle-based
On the other aspect, different W t usually has a different norm.
To further alleviate the influence ofvarious norms, we calibrate the logit into the feature space with unit length by L2-normalization,
W Ti ϕ (ˆ x )+ b i
||
W i |||| ϕ ( x ) || .
If omit the b i , this calibration is compute the cos ( θ ) between ϕ (ˆ x ) and W i , and we termit as angle-based calibration.
Since, this angle-based calibration will bound each logit smaller thanone.
Instead of using the CE loss function, we directly minimize the angle between the ϕ (ˆ x ) and thetargeted W t .
The optimization loss function is:
[Equation]
The angle-based classifiers have been widely using in Face-Recognition task [17, 2].
In the experiments, we evaluate the performance of using different logit calibrations and their mutual benefits.
Experiments
Experimental Setup
In this section, we evaluate the effectiveness of logit calibration for improvingtransferable targeted adversarial attack.
Following the recent study [30], we conduct the experimentson the difficult ImageNet-Compatible Dataset 1 .
This dataset contains 1,000 images with 1,000 uniqueclass labels corresponded to the ImageNet dataset.
We implement our methods based on the source [Table] 2.5/7.7/ 9.4 Angle 38.9/72.5/77. [Table]
[Table]
CE
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Logit 3.4/
[Table]
[Table]
[Table]
[Table]
[Table]
0.6/1.5/2.8 T
[Table]
0.4/1.6/2.6 T=3.6/9.0/9.
[Table]
code 2 provided by the Logit [30].
The same four diverse CNN models are used for evaluation, i.e. ,
ResNet-50 [7], DenseNet-121 [9], VGG-16 with Batch Normalization [24] and Inception-v3 [26].
The perturbation is bounded by L ∞ ≤ 16 .
The TI [4], MI [3] and DI [29] were used for all attacks,and || W || 1 = 5 is set for TI.
The I-FSGM is adopted for optimization with the α = 2 .
The attacks 231 are trained for 300 iterations on a NVIDIA-2080 Ti GPU.
We run all the experiments for 5 times, andreport the average targeted transfer success rates (%). More experimental results can be found in thesupplementary.
4.1 Comparison with Other Methods in Single-Model Transfer
We first compare the proposed (temperature-based, margin-based and angle-based) logit calibrationswith the original CE, Po+Trip [15], and Logit [30] in the single-model transfer task.
In this task, wetake one surrogate model for training, and test the targeted transferability in attacking other 3 models.
As shown in Table 1, the original CE loss function produces a worst performance than the Po+Tripand Logit.
But after performing the logit calibration in the CE loss function, we can find a significantperformance boost compared with the original CE.
All the calibration methods can outperform the
Logit, especially when using the ResNet50 and Dense121 as the surrogate.
These results indicate thatthe logit margin can significantly influence the performance of the targeted transferability.
On theother aspect, we find that T
= 10 has better performance than T = 5 on the VGG-16, suggestingthat different models may need different T .
Instead of finding the best T for a different model, the
Margin-based calibration can solve the issue and reach the overall best transferability in all fourmodels.
However, we find that the Angle-based calibration is not working on the VGG16, whichneeds further investigation.
4.2 The Influence of Different T in CE
In this section, we evaluate the influence of using different T in the CE loss function.
The results arereported in Table 2.
From the Table, we can have the following observations.
The scaling factor
T has a significant influence on the targeted transferability.
Specifically, there is a large decrease inperformance when using a small T = 0 .
After increasing the T , we can observe the number ofsuccessfully attacked samples will increase.
The optimal T for different model is different.
Forexample, T = 5 can produce the overall best performance for ResNet50, Dense121, and Inception v3,while the VGG16 with fewer convolutional layers requires a large T to obtain better transferability.3)
The performance are comparable when using T = 5 and T = 10 for ResNet50, Dense121, and Inception v3.
This is because that we use I-FSGM for optimization, which only considers the sign ofthe gradients.
4) Using a larger T , the performance will be similar to the Logit loss function (see Table 1).
We provide a deep analysis of this phenomenon in the supplementary material.
[Table] [Table] 4.3 The Targeted Success rates for Transfer with Varied Targets
In Table 3, we report the result of a worse-case transfer scenario by gradually varying the targetclass from the highest-ranked to the lowest one, and have the following findings: 1) The threetypes of logit calibration methods can improve the targeted transfer success rate over the original
CE.
The angle-based calibration has the best performance. But, we notice that the margin-basedcalibration doesn’t work well in this setting.
The Temperature-based (T=5/10) and the Angle-basedcalibrations can outperform the Logit loss by a large margin, especially the Angle-based calibration. 4.4 The Mutual Benefits of Different Calibration Methods
[Table]
In this part, we evaluate the mutual benefits ofcombining different calibrations and can have thefollowing findings.
Combining the T=5/10/20and Margin, there is no increase in performancecompared with using one of them.
This is becausethat the gradient directions of these two methodsare very similar. 2)
Combining the T=5 and Angle, we can observe a further improvement whenusing ResNet50 and Dense121 as the surrogatemodel, e.g. , the transferable rate of “ResNet50 → Dense121” is increased to 82.4% with 300 iterations.
Since the Angle obtains poor performance on VGG16, the transferable rates of corresponding combinations are also low in T=5/10+Angle, but
T=20+Angle can deal with this issue.
Combining the Margin and Angle, there are only slightimprovements on ResNet50 and Dense121, while it can alleviate the negative effects caused by theangle-based calibration.
{+Finally, by jointly considering the results in Table 1, 2 and 4, we suggestusing T=
+}
+ Angle for CNNs with more layers and the single Margin-based calibration for CNNswith fewer layers to achieve better targeted transfer attack.
4.5 Comparison with The TTP Method
In this section, we further evaluate the proposed temperature-based logit calibration in the GAN-basedtargeted attacks.
Following the setting in TTP [21], we sampled 50K images from the ImageNettraining set and 50K images from the Painting dataset 3 , which are used to train the targeted generatorsfrom different source domains.
Instead of using the distribution matching and neighborhood similaritymatching loss [21], we only use the cross-entropy function for training the targeted generators whilekeeping other settings identical.
More training and evaluation details used by TTP can be referred to[21].
We used the ResNet50 as the surrogate model and reported the results in Table 5. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
From Table [Table] we make the following findings. By using ImageNet as the training dataset, the TTPshows better transferability than the CE in attacking other black-box models.
The average targetedaccuracy of TTP is around 3% higher than that of CE.
After downscale the logit by 5 in the CE lossfunction (CE (T=5)), we can observe a significant boost of the Top-1 targeted accuracy for all models,reaching the average targeted accuracy of 91.98% (ImageNet).
For both ImageNet and Paintingas the training source, the CE (T=5) can surpass the TTP by a large margin (91.98% vs. 83.12% &89.70% vs. 81.33%).
These experimental results demonstrate that the proposed temperate-based logitcalibration is also effective in training generator-based targeted attackers.
Note that, compared to
TPP, our logit calibration has the benefit of without using any data from the target class.
Conclusion
In this study, we analyzed the logit margin in different loss functions for the transferable targetedattack, and find that the margin will quickly get saturated in the CE loss and thus limited thetransferablity.
To deal with this issue, we introduce to use logit calibrations in the CE loss function,including Temperature-based, Margin-based, and Angle-based.
Experimental results verified theeffectiveness of using the logit calibration in the CE loss function for crafting transferable targetedadversarial samples.
The proposed logit calibration methods are simple and easy to implement, whichcan achieve state-of-the-art performance in transferable targeted attack.
Potential Social Impact .
Our findings in targeted transfer attacks can potentially motivate the AIcommunity to design more robust defenses against transferable attacks.
In the long run, it may alsobe directly used for suitable social applications, such as protecting privacy.
Contrariwise, someapplications may use targeted transferable attacks in a harmful manner to damage the outcome of AIsystems, especially in scenarios of speech recognition and facial verification systems.
Finally, wefirmly believe that our investigation in this study can provide valuable insight for future researchersby using the logit calibration for both adversarial attack and defense.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[No] No negative
effects
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See abstract.
(b) Did you specify all the training details ( e
g. , data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars ( e
g. , with respect to the random seed after running experiments multiple times)?
[No] We report the average of running experiments 5times.
(d) Did you include the total amount of compute and the type of resources used ( e
g. , typeof GPUs, internal cluster, or cloud provider)?
[Yes] See URL in abstract
If you are using existing assets ( e
g. , code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No](d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] The code and data are public
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[No] No use of personally identifiable information orobjectionable content.5.
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[N/A] [Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]