Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal
(s
Abstract
Vision transformers (ViTs) have demonstrated impressive performance and strongeradversarial robustness compared to Convolutional Neural Networks (CNNs).
Onthe one hand, ViTs’ focus on global interaction between individual patches reducesthe local noise sensitivity of images.
On the other hand, the neglect of noisesensitivity differences between image regions by existing decision-based attacksfurther compromises the efﬁciency of noise compression, especially for ViTs.
Therefore, validating the black-box adversarial robustness of ViTs when the targetmodel can only be queried still remains a challenging problem.
In this paper,we theoretically analyze the limitations of existing decision-based attacks fromthe perspective of noise sensitivity difference between regions of the image, andpropose a new decision-based black-box attack against ViTs, termed Patch-wise
Adversarial Removal (PAR).
PAR divides images into patches through a coarseto-ﬁne search process and compresses the noise on each patch separately.
PARrecords the noise magnitude and noise sensitivity of each patch and selects thepatch with the highest query value for noise compression.
In addition, PAR can beused as a noise initialization method for other decision-based attacks to improvethe noise compression efﬁciency on both ViTs and CNNs without introducingadditional calculations.
Extensive experiments on three datasets demonstrate that
PAR achieves a much lower noise magnitude with the same number of queries.
Introduction
Vision transformers (ViTs)[1] not only achieve signiﬁcant performance improvement in a wide rangeof computer vision tasks [2, 3, 4], but also show stronger robustness against adversarial examplesgenerated by different attack methods [5, 6, 7].
The adversarial examples are generated by attackersto fool the target model by adding imperceptible noises to original data [8, 9, 10].
The characteristicof using non-overlapping patches in ViTs reduces the inﬂuence of adversarial examples with the samenoise magnitude on the overall results [5].
According to the amount of information the attacker can obtain, adversarial attacks can be dividedinto white-box attacks and black-box attacks [11].
In the black-box scenario where the attacker canonly obtain the hard labels output by the target model, adversarial attacks can be further dividedinto transfer-based methods [12, 13] and decision-based methods [14].
Transfer-based methods usea substitute model to generate adversarial examples and transfer them to the target model takingadvantage of the transferability [15, 16, 17].
Compared with transfer-based attacks, decision-basedattacks face a more practical setting where a substitute model is not available.
The only source ofinformation for decision-based attacks is hard label obtained by querying the target model.
In theimage classiﬁcation task, the decision-based attacks [14, 18, 19] start from a random noise witha large noise magnitude, randomly sample in the image input space, and gradually compress theadversarial noise under the premise of ensuring misclassiﬁcation.
The existing adversarial attacks
[Figure]
against transformers are only white-box attacks [20, 5] and transfer-based black-box attacks [21, 22].
The characteristics of ViTs’ patch-wise splitting of images reduce the impact of adversarial noise,leaving decision-based black-box attacks against ViTs an open problem [5].
The challenge of attacking ViTs with decision-based methods comes from their properties in noisesensitivity, which derive from the structural characteristics of ViTs.
On the one hand, ViTs learn fewerlow-level features and more transferable features than CNNs, resulting a much more noise neededto attack ViTs [20].
In other words, the overall noise sensitivity of ViTs is low.
Decision-basedattacks against ViTs need to add random noise with much larger noise magnitude to ﬁnd the initialadversarial examples.
Larger initial noise makes it more difﬁcult for the decision-based attacksto compress, i.e., to ﬁnd the smallest adversarial noise under the same number of queries.
On theother hand, ViTs split the image into multiple non-overlapping patches, which reduces the impact ofnoises on one single patch to the ﬁnal classiﬁcation results [5].
This leads to a notable differencein ViTs’ noise sensitivity between different regions of an image, which is rarely considered byexisting decision-based methods.
For example, the noise compression process of Boundary Attack[14] treats all pixels equally regardless of their noise sensitivity, as demonstrated in Fig.
1, whichseverely hinders the efﬁciency of noise compression.
These two properties in noise sensitivity makeit extremely difﬁcult for existing decision-based attacks to ﬁnd adversarial examples with small noisemagnitude against ViTs.
Noise sensitivity directly reﬂects the black-box adversarial robustness of
ViTs, which has not been well studied and unable to shed light on the mechanism of improving noisecompression efﬁciency.
Therefore, designing decision-based attacks against ViTs according to theirnoise sensitivity properties is an essential problem.
In this paper, we verify that noise sensitivity of ViTs varies signiﬁcantly between different imageregions.
The limitation about the compression process of decision-based attacks represented by
Boundary Attack is theoretically analyzed.
Based on the relationship between ViTs’ patch-wisesensitivity and noise compression success rate, we propose a new decision-based attack method
Patch-wise Adversarial Removal (PAR).
PAR splits the adversarial example into multiple patchesand perform coarse-to-ﬁne noise removal.
Speciﬁcally, PAR maintains two masks to record the noisesensitivity and noise magnitude of each patch, respectively.
Before querying the target ViTs, PARlocates the patch with the highest query value based on these two masks.
As the search progresses,the size of each patch gets smaller while the measure of the noise sensitivity of the ViTs becomesmore accurate.
PAR achieves a signiﬁcant noise compression under a small number of queries, whichcan be used as an initialization for other decision-based attacks without additional computation.
We validate the effectiveness of PAR on three datasets: ImageNet-21k [23], ILSVRC-2012 [24],and Tiny-Imagenet [25].
We compare PAR with 7 state-of-the-art decision-based attacks against18 different target models, including 8 CNNs and 10 ViTs or hybrid models.
Beneﬁting from thepowerful redundant noise compression capability, the noise magnitude of all decision-based
Related Work2.1 Robustness of Vision Transformer
The ViTs show stronger adversarial robustnes[26].
Not only does fooling ViTs in the white-box scenario require larger noise magnitude [20], but it is also difﬁcult for existing transfer-based black- box attacks to transfer adversarial examples from CNNs to ViTs [21].
Existing research mainlyfocuses on the white-box attacks and transfer-based black-box attacks against ViTs.
However, thispaper explores the decision-based attack against black-box ViTs without substitute model.
2.2 Decision-based Attack
Decision-based attacks do not rely on substitute models, but require an initial adversarial example thathas already been misclassiﬁed as starting point.
Boundary Attack [14] starts from an gaussian noiseand searches along two directions simultaneously, namely source direction and spherical direction:
[Equation]
where x ∗ is the adversarial example with smallest noise that already been found.
η and ( x − x ∗ ) referto the directions of spherical and source direction, respectively.
δ and ε are stepsizes of sphericaland source direction.
The Biased Boundary Attack [19] concentrates on low-frequency domain ofinput space to make the adversarial example more ‘natural’.
The Evolutionary Attack [18] reducesthe dimension of sampling space by bilinear interpolation.
Evolutionary attack performs better intasks involving strong prior knowledge such as face recognition.
HopSkipJumpAttack
[27] estimatesthe gradient direction using binary information at the decision boundary.
Customized Adversarial
Boundary (CAB) [28] uses current noise to select the sensitive regions of images and customizessampling distribution.
SurFree Attack [29] is based on the geometrical mechanism to get the biggestdistortion decrease for a given direction to be explored.
Sign-OPT attack [30] uses a zeroth orderoracle to compute the sign of directional derivative of the attack objective.
Proposed Method
3.1 Notation
Suppose F is the target model to be attacked: F : X N → Y C , where X represents the input space,
N is the dimension ( N = Width ×
Height × Channel for image data) and Y represents theclassiﬁcation space with C categories.
The goal of decision-based attack can be expressed as:
[Equation]
where x represents the original image, x (cid:48) refers to adversarial example, y is the label of x , S Q is theset of all adversarial examples generated to query the target model, T is the limit of query number.
vrefers to the norm used to measure the noise magnitude including (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ norm.
In this paperthe (cid:96) 2 distance is calculated.
In the process of decision-based attack, the attacker can only obtain thehard label F ( x (cid:48) )
output by the target model.
3.2 Noise Sensitivity of ViTs
Here we measure the noise sensitivity Sens of one patch in the image according to the maximumratio the noise can be compressed under the current adversarial example on ViT:
Deﬁnition 1.
Let x (cid:48) be an adversarial example of ViT model F on the original image x , i.e.,
F ( x (cid:48) )
(cid:54) =
F ( x ) , and z be the current adversarial noise z = x (cid:48) − x .
Let ˜ z be a new adversarial noisecompressed from z in a rectangle patch with width of w , height of h , top left corner of sr, sc :
[Equation] [Equation]
[Equation] [Equation]
else,
[Equation]
where r and c refer to the row and column index of one pixel in noise z , respectively.
κ ∈ [0 , 1]denotes the noise compression ratio.
Deﬁne the noise sensitivity of a rectangle patch as the minimumnoise compression ratio κ min when F misclassiﬁes x + ˜ z :
Sens ( F, x, x (cid:48) , sr, sc, h, w ) = κ min , s.t. F ( x + ˜ z ( sr, sc, h, w, κ min )) (cid:54) =
F ( x ) and ∀ κ (cid:48) < κ min , F ( x + ˜ z ( sr, sc, h, w, κ (cid:48) ))
=
F ( x ) .
[Figure]
Sens measures the minimum amount of noise that is required for an adversarial example.
A smaller
Sens means that more noise can be removed without changing the misclassiﬁcation results, i.e.,adding noise in this patch has less impact on classiﬁcation results.
When h = w = 1 , Sens measuresthe pixel-level noise sensitivity.
We use binary search to show the patch-wise noise sensitivity ofvit-tiny-patch16 [31] with patch size of 16 trained on ILSVRC-2012 [24].
The detail of evaluation isprovided in the Supplementary Material A.1.
The heat map in Fig.
2 shows the Sens on all 14 × 14patches.
In the heat map, a lighter color means lower Sens on one patch.
1 indicates that any smallnoise compression on this patch makes the target model outputs correct label.
0 indicates that theadversarial example remains misclassiﬁed even all noises on this patch are removed.
Consideringthe attack process of Boundary Attack, the patch-wise sensitivity of adversarial examples and thecompression success probability within a patch of one single query have the following relationship:
Proposition 1.
Assume x (cid:48) is an initial adversarial example generated by Boundary Attack against ViT
F starting from original image x , F ( x ) (cid:54) =
F ( x (cid:48) ) .
For any 0 < r 1 , r 2 , h ≤ Height , 0 < c 1 , c 2 , w ≤
Width , if Sens ( F, x, x (cid:48) , r 1 , c 1 , h, w ) <
Sens ( F, x, x (cid:48) , r 2 , c 2 , h, w ) , and the new noise added by onestep by Boundary Attack is z (cid:48) , then P ( F ( x (cid:48)
+ z (cid:48) 1 ) (cid:54) =
F ( x ) | F ( x (cid:48)
+ z (cid:48) )
=
F ( x )) <
P ( F ( x (cid:48)
+ z (cid:48) 2 ) (cid:54) =
F ( x ) | F ( x (cid:48)
+ z (cid:48) )
=
F ( x )) , where for ι = 1 , 2z (cid:48)
ι,r,c = (cid:40) 0 , if r ι ≤ r < r ι + h and c ι ≤ c <
c ι + w, z (cid:48) r,c , else, (5)
Proof Idea.
The expectation of each pixel on the initial noise x (cid:48) generated by Boundary Attack isequal, and the noise compression ratio after one-step Boundary Attack for each pixel is i.i.d.
Thepossibility that the noise compression ratio on at least one pixel exceeds κ is also the same for anypixel.
Since the probability that noise compression ratio on at least one pixel exceeds Sens increasesmonotonically w.r.t.
the noise sensitivity on the whole patch, the noise removal of high Sens patch ismore likely to be the cause of misclassiﬁcation failures.
(cid:3)
The detailed proof of Proposition 1 is provided in the Supplementary Material A.2.
Proposition 1indicates that during the noise compression process of Boundary Attack, patches with higher Sensare more likely to be the cause of query failure than regions with lower Sens .
Obviously, the noisesensitivity of ViTs varies greatly in different regions of the image.
In the right part of Fig. 2, wecompare the patch-wise sensitivity of original image (b) and the probability that noise compressionratio exceeding Sens on each patch P ( F ( x (cid:48)
+ z (cid:48) b ) (cid:54) =
F ( x ) | F ( x (cid:48)
+ z (cid:48) )
=
F ( x )) caused by one stepof Boundary Attack.
As can be seen that these two heat maps are basically consistent on patches,which veriﬁes Proposition 1.
It can be seen from Fig.
2 that completely removing noises of manypatches from x init does not affect the misclassiﬁcation.
However, the uniform compression of
Boundary Attack usually keeps these redundant noises to the end.
The magnitude of the whole blockof redundant noise is considerable, especially for ViTs which require a larger initial noise.
3.3 Patch-wise Adversarial Removal
According to Proposition 1, the Boundary Attack compresses the overall noise of x (cid:48) together, whosenoise compression rate depends on those patches with the highest Sens .
Ideally, a decision-basedattack should ﬁrstly compresses regions with low Sens and high noise magnitude.
In this way, boththe success rate of query and the magnitude of one-step noise compression can be guaranteed, and theefﬁciency of noise compression can be maximized with limited query numbers.
On the one hand, the
Black-Box Vision
Transformer (cid:28564)
Query
Success
Fail
Initial Adversarial
Example (cid:2206) (cid:2191)(cid:2196)(cid:2191)(cid:2202)
Initial Noise (cid:2208) (cid:2191)(cid:2196)(cid:2191)(cid:2202)
Update
Current Noise (cid:2208) (cid:1499)
Current Adversarial
Example (cid:2206) (cid:1499)
Noise Magnitude Mask (cid:2169) (cid:2170)
[Figure]
(cid:2169) (cid:2175)
Update
Record Record
[Figure]
Initialize Initialize
Update noise sensitivity of the target model to different regions of the initial noise cannot be directly obtained.
On the other hand, using binary search similar in Fig. 2 for one patch and perform grid search for allthe patches consumes large quantity of queries.
Therefore, we propose a new decision-based attack
Patch-wise Adversarial Removal (PAR).
PAR divides the initial noise into patches, probes their noisesensitivity and compresses the noise in a patch-wise manner.
As illustrated in Fig.
3, PAR guides the probe process by maintaining two masks that record the noisesensitivity of the target model and the noise magnitude of each mask, respectively.
Since the detail ofthe ViTs is not available in the black-box attack setting, PAR does not assume the patch size of ViTs,but starts from a large patch size to conduct multiple rounds of coarse-to-ﬁne search.
Firstly, PAR initializes the noise sensitivity mask M S and noise magnitude mask M N .
The shape ofthe two masks is PS 0 × PS 0 , where PS 0 is a hyper parameter denotes the initial patch size of PAR.
We use the initial noise magnitude in (cid:96) 2 of each patch in x init to initialize M N :
[Equation]
where row and col indicate the row index and column index of M N , row, col ∈ [1 , PS 0 ] .
The noisesensitivity mask is binary.
1 in M S indicates that the corresponding patch in the adversarial noiseremains a low noise sensitivity state, which may not have been queried or noise removal has beensuccessfully carried out.
in M S indicates that the previous noise compression process is failed.
Theinitial value for each element in M S is 1.
Before each query to the target model, we use element-wiseproduct to obtain the query value mask M Q :
[Equation]
where J is a unit matrix of all-ones.
If a patch does not contain noise or previous query is failed, there is no query value.
We sort thevalues in M Q in descending order, and remove the noise in the patch corresponding to the highestvalue in M Q .
We input the updated adversarial examples x query into the target model to obtain thequery result.
If x query still misclassiﬁes the target model, it indicates that the noise sensitivity of thispatch is low.
In this case, we set x ∗ as x query and update M N .
Otherwise, the noise sensitivity of thepatch is high, and the corresponding element in noise sensitivity mask M S is set to 0.
Input: Target model F ( x ) , noise magnitude limit τ , original image x and label y
Max querying number T , initial variance of gaussian var
Identity matrix I of the same dimension as x , initial and minimum patch size PS 0 and PS min
Output: Adversarial example x ∗ with compressed noise while F ( x init ) =
y do ξ init ∼ N (0 , var 2 I ) , x init ← Clip x,τ { x + ξ Gau} , var ← var ∗ 2 ,
T ← T −
Initialize M N and M S according to Eqn.
(6) and Eqn.
PS ← PS 0 , x ∗ ← x init while T > 0 do
M Q ← M N (cid:12) M S if (cid:80) M Q
=
then
PS ← PS/ 2 , initialize M N and M S according to Eqn.
(6) and Eqn.
(7) if PS ≤ PS min then break // Locate the highest value in query value mask row ∗ , col ∗ ← argmax ( M Q )
z query ← x ∗ − x, z queryrow ∗ ∗ PS +1:( row ∗ +1) ∗ PS,col ∗ ∗ PS +1:( col ∗ +1)
∗ PS ← 0 x query ← Clip x,τ { z query + x } if F ( x query ) (cid:54) =
y then x ∗ ← x query , update M N else M Srow ∗ ,col ∗ ←
T ← T −
If the sum of M Q is 0, all patches under current patch size PS either have no noise, or the query hasalready been made.
In this case, we halve the patch size and reinitialize M N and M S according to
Eqn.
(6) and Eqn.
The next round will conduct more ﬁne-grained query on the patches wherethere are still noises.
Since the noise on some patches have been removed in the previous rounds, thesearch process of PAR with gradually reduced patch size is much more efﬁcient in terms of queriesthan using very small patch size at the beginning.
There are two exit conditions for the search process of PAR, either the max query number T isreached or the minimum patch size PS min has been reached.
The PS min is set for the efﬁciency ofone-step noise compression.
When the patch size is too small, the compressed noise magnitude afterone step is not worthwhile even if the subsequent query succeeds.
Algorithm 1 details PAR.
3.4 PAR as Noise Initialization Method
As an query-efﬁcient decision-based attack, PAR can also be used as a noise initialization method forother decision-based methods.
PAR removes all possible blocks of noise larger than the minimumpatch size PS min , leaving the remaining regions to be noise-sensitive for ViTs.
In this way, PARgreatly reduces the search space for subsequent noise compression.
After initializing noises with PAR,decision-based attacks may concentrate each sampling in the regions with higher noise sensitivity.
Experiments
4.1 Experiment Settings
We conduct experiments on three image classiﬁcation datasets: ImageNet-21k [23], ILSVRC-2012[24], and Tiny-Imagenet [25].
We pick 10000 images from the validation sets of ImageNet-21k and
ILSVRC-2012 that can be correctly classiﬁed by all target models for test.
As for Tiny-Imagenet with200 image categories, we choose 2000 images, 10 images for each category.
10 vision transformermodels with different structures [31] are compared: vit-s32, vit-b16, vit-b32, r50-l32, r50-s32,vit-large-patch16-224, vit-tiny-patch16-224, vit-small-r26-s32-224, vit-tiny-patch16-224, vit-smallpatch16-224.
We also include 8 CNNs as target models: resnet-18 [32], resnet-101, inception v3 [33],
[Table]
[Table]
PAR+BBA 3.
Evo 8.
3.924 PAR+Evo 4.
Boundary 11.
3.876 PAR+Boundary 4.
SurFree 6.
2.25 PAR+SurFree 4.
CAB 4.
2.084 PAR+CAB 1. Sign-OPT 30.581 36.
12.083 PAR+Sign-OPT 4.
[Table] [Table]
[Table] [Table] [Table] [Table]
[-We also include 8 CNNs as target models: resnet-18 [35], resnet-101, inception v
-][Table] [Table]
inception-resnet v2 [34], nasnet [35], densenet-161 [36], vgg19-bn [37], senet-154 [38].
4 RTX
GPU cards are used for calculation.
We compare 7 decision-based attacks with our PAR under the black-box setting with limited querynumber: Boundary Attack [14], Biased Boundary Attack (BBA) [19], Evolutionary Attack (Evo)
HSJA [27], CAB [28], Sign-OPT [30], and SurFree [29].
Stepsizes of spherical direction and sourcedirection are δ
= 0 .
1 , ε 0 = 0 .
003 for Boundary, BBA, Evo and CAB.
For BBA [19], we use theversion that does not incorporate information from a substitute model at each step for fair comparison.
Noise magnitude limit τ = [0 , 255] .
The initial and minimum patch size for ImageNet-21k and
ILSVRC-2012 is set to 56 and 7, respectively.
For tiny-imagenet, we set the initial and minimumpatch size as 16 and 2, respectively.
All datasets are under BSD 3-Clause License.
For evaluation criterion, we choose the median and average size of adversarial perturbation, as appliedin NIPS 2018 Adversarial Vision Challenge [25]:
[Equation]
where n data is the number of images in a dataset, x is an original image in the dataset X .
x (cid:48) is theadversarial example found that is closest to x .
A smaller (cid:96) 2 distance indicates a better adversarial
[Table]
[Figure] [Figure] [Table] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Table]
distance of adversarial noise under different query number T
example.
It is worth noting that adversarial examples are rounded before being input to the targetmodel for a more realistic black-box attack setting.
4.2 Experimental Results
To verify the advantage of PAR over existing decision-based attacks on ViTs and CNNs, we report themedian and average adversarial perturbation on ImageNet-21k, ILSVRC-2012, and Tiny-Imagenet in
Table 2, Table 3, and Table 1, respectively.
The ﬁrst row of three tables represents target models withdifferent structures.
We compare the average (Avg) and median (Mid) noise magnitude generated by
PAR and other 6 attacks on different target models.
We also use PAR as the noise initialization forother decision-based attacks.
The noise compressed by PAR is handed over to other decision-basedattacks for further compression.
It can be seen that when PAR is used to initialize adversarial noise,the average and median noise magnitude drops signiﬁcantly compared with only using the originaldecision-based attack.
This veriﬁes the strong noise compression ability of PAR.
We also combine
PAR with other decision-based attacks, and compare the query efﬁciency under total query numbersof 3000 in Fig. 4.
The target models are notated under each subﬁgure.
There is a noticeable drop innoise magnitude when initializing the noise with PAR.
Table.
4 compares the noise compression efﬁciency and the average query number of PAR underdifferent initial patch sizes and minimum patch sizes.
The compressed noise is handed over to
Boundary Attack for further compress until 1000 queries.
It can be seen that when the initial patchsize is small, the average query number will be large, resulting in low query efﬁciency and less query
[Table]
[Figure]
[Table]
[Figure]
number for subsequent decision-based attacks.
A more reasonable strategy is to use a large initialpatch size and stop at a small minimum patch size.
Fig.
5 compares adversarial noises generated against vit-small-r26-s32-224 by seven different attackson ILSVRC-2012.
The ﬁrst row shows the original images.
The second to eighth images of each roware the noises generated by each attack.
PAR stops when the exit condition is met.
All other attacksperform 1000 queries on the target model for each adversarial example.
The noises of PAR mainlyconcentrate on a few patches instead of spreading over the entire image.
The noise magnitude ofother decision-based attacks decrease signiﬁcantly when PAR is used for noise initialization.
PARproposed in this paper is only used for the study of adversarial machine learning and the robustness of
ViTs, and does not target any real system.
There is no potential negative impact.
More experimentalresults on different target ViT and CNN structures are provided in the Supplementary Material A.3.
Conclusion
In this paper, we explore decision-based adversarial attacks against Vision Transformers.
In viewof the huge difference in the noise sensitivity between patches of ViTs, we propose Patch-wise
Adversarial Removal to achieve query-efﬁcient noise compression.
PAR maintains noise magnitudeand noise sensitivity masks to probe and compress adversarial noises in a patch-wise manner, andimprove query efﬁciency through a coarse-to-ﬁne search process on the patch size.
Experiments onthree image classiﬁcation datasets verify the feasibility and generalizability of PAR to improve thequery efﬁciency with limited query number.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes
](c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A](
d ) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Yes
If you are using existing assets (e.
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table] you include the full text of instructions given to participants and screenshots, if [Table]
applicable
[Table]
you describe participant
to Institutional Review [Table] Board
[Table]
compensation
[Table]