Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal
Abstract
Vision transformers (ViTs) have demonstrated impressive performance and stronger adversarial robustness compared to Convolutional Neural Networks (CNNs).
On the one hand, ViTs’ focus on global interaction between individual patches reduces the local noise sensitivity of images.
On the other hand, the neglect of noise sensitivity differences between image regions by existing decision-based attacks further compromises the efﬁciency of noise compression, especially for ViTs.
Therefore, validating the black-box adversarial robustness of ViTs when the target model can only be queried still remains a challenging problem.
In this paper, we theoretically analyze the limitations of existing decision-based attacks from the perspective of noise sensitivity difference between regions of the image, and propose a new decision-based black-box attack against ViTs, termed Patch-wise
Adversarial Removal (PAR).
PAR divides images into patches through a coarseto-ﬁne search process and compresses the noise on each patch separately.
PAR records the noise magnitude and noise sensitivity of each patch and selects the patch with the highest query value for noise compression.
In addition, PAR can be used as a noise initialization method for other decision-based attacks to improve the noise compression efﬁciency on both ViTs and CNNs without introducing additional calculations.
Extensive experiments on three datasets demonstrate that
PAR achieves a much lower noise magnitude with the same number of queries.
Introduction
Vision transformers (ViTs)[1] not only achieve signiﬁcant performance improvement in a wide range of computer vision tasks [2, 3, 4], but also show stronger robustness against adversarial examples generated by different attack methods [5, 6, 7, 8].
The adversarial examples are generated by attackers to fool the target model by adding imperceptible noises to original data [9, 10, 11].
The characteristic of using non-overlapping patches in ViTs reduces the inﬂuence of adversarial examples with the same noise magnitude on the overall results [5]. [Figure]
According to the amount of information the attacker can obtain, adversarial attacks can be divided into white-box attacks and black-box attacks [12].
In the black-box scenario where the attacker can only obtain the hard labels output by the target model, adversarial attacks can be further divided into transfer-based methods [13, 14] and decision-based methods [15].
Transfer-based methods use a substitute model to generate adversarial examples and transfer them to the target model taking advantage of the transferability [16, 17, 18].
Compared with transfer-based attacks, decision-based attacks face a more practical setting where a substitute model is not available.
The only source of information for decision-based attacks is hard label obtained by querying the target model.
In the image classiﬁcation task, the decision-based attacks [15, 19, 20] start from a random noise with a large noise magnitude, randomly sample in the image input space, and gradually compress the adversarial noise under the premise of ensuring misclassiﬁcation.
The existing adversarial attacks against transformers are only white-box attacks [21, 5, 22, 23] and transfer-based black-box attacks [24, 25].
The characteristics of ViTs’ patch-wise splitting of images reduce the impact of adversarial noise, leaving decision-based black-box attacks against ViTs an open problem [5].
The challenge of attacking ViTs with decision-based methods comes from their properties in noise sensitivity, which derive from the structural characteristics of ViTs.
On the one hand, ViTs learn fewer low-level features and more transferable features than CNNs, resulting a much more noise needed to attack ViTs [21].
In other words, the overall noise sensitivity of ViTs is low.
Decision-based attacks against ViTs need to add random noise with much larger noise magnitude to ﬁnd the initial adversarial examples.
Larger initial noise makes it more difﬁcult for the decision-based attacks to compress, i.e., to ﬁnd the smallest adversarial noise under the same number of queries.
On the other hand, ViTs split the image into multiple non-overlapping patches, which reduces the impact of noises on one single patch to the ﬁnal classiﬁcation results [5].
This leads to a notable difference in ViTs’ noise sensitivity between different regions of an image, which is rarely considered by existing decision-based methods.
For example, the noise compression process of Boundary Attack [15] treats all pixels equally regardless of their noise sensitivity, as demonstrated in Fig.
1, which severely hinders the efﬁciency of noise compression.
These two properties in noise sensitivity make it extremely difﬁcult for existing decision-based attacks to ﬁnd adversarial examples with small noise magnitude against ViTs.
Noise sensitivity directly reﬂects the black-box adversarial robustness of
ViTs, which has not been well studied and unable to shed light on the mechanism of improving noise compression efﬁciency.
Therefore, designing decision-based attacks against ViTs according to their noise sensitivity properties is an essential problem.
In this paper, we verify that noise sensitivity of ViTs varies signiﬁcantly between different image regions.
The limitation about the compression process of decision-based attacks represented by
Boundary Attack is theoretically analyzed.
Based on the relationship between ViTs’ patch-wise sensitivity and noise compression success rate, we propose a new decision-based attack method
Patch-wise Adversarial Removal (PAR).
PAR splits the adversarial example into multiple patches and perform coarse-to-ﬁne noise removal.
Speciﬁcally, PAR maintains two masks to record the noise sensitivity and noise magnitude of each patch, respectively.
Before querying the target ViTs, PAR locates the patch with the highest query value based on these two masks.
As the search progresses, the size of each patch gets smaller while the measure of the noise sensitivity of the ViTs becomes more accurate.
PAR achieves a signiﬁcant noise compression under a small number of queries, which can be used as an initialization for other decision-based attacks without additional computation.
We validate the effectiveness of PAR on three datasets: ImageNet-21k [26], ILSVRC-2012 [27], and Tiny-Imagenet [28].
We compare PAR with 7 state-of-the-art decision-based attacks against 18 different target models, including 8 CNNs and 10 ViTs or hybrid models.
Beneﬁting from the powerful redundant noise compression capability, the noise magnitude of all decision-based attacks has been notably reduced after using PAR for noise initialization without increasing the query number.
Related Work 2.1 Robustness of Vision Transformer
The ViTs show stronger adversarial robustnes[29].
Not only does fooling ViTs in the white-box scenario require larger noise magnitude [21], but it is also difﬁcult for existing transfer-based blackbox attacks to transfer adversarial examples from CNNs to ViTs [24].
Existing research mainly focuses on the white-box attacks and transfer-based black-box attacks against ViTs.
However, this paper explores the decision-based attack against black-box ViTs without substitute model.
2.2 Decision-based Attack
Decision-based attacks do not rely on substitute models, but require an initial adversarial example that has already been misclassiﬁed as starting point.
Boundary Attack [15] starts from an gaussian noise and searches along two directions simultaneously, namely source direction and spherical direction:
[Equation]
where x ∗ is the adversarial example with smallest noise that already been found.
η and ( x − x ∗ ) refer to the directions of spherical and source direction, respectively.
δ and ε are stepsizes of spherical and source direction.
The Biased Boundary Attack [20] concentrates on low-frequency domain of input space to make the adversarial example more ‘natural’.
The Evolutionary Attack [19] reduces the dimension of sampling space by bilinear interpolation.
Evolutionary attack performs better in tasks involving strong prior knowledge such as face recognition.
HopSkipJumpAttack
[30] estimates the gradient direction using binary information at the decision boundary.
Customized Adversarial
Boundary (CAB) [31] uses current noise to select the sensitive regions of images and customizes sampling distribution.
SurFree Attack [32] is based on the geometrical mechanism to get the biggest distortion decrease for a given direction to be explored.
Sign-OPT attack [33] uses a zeroth order oracle to compute the sign of directional derivative of the attack objective.
Boundary Attack maintains two search directions according to Eqn.
The source direction is pointed to the original image x with a small stepsize, which is responsible for noise compression.
The spherical direction is a random direction with a large stepsize, which is used to perturb in the neighborhood of current adversarial example x ∗ and ensure that the updated adversarial example is still misclassiﬁed by the target model. It can be seen from Eqn. (1) that in the one-step update of the current adversarial example, the spherical direction η follows the standard Gaussian distribution with the same dimensions as the original image. The source direction points to the opposite direction of the current adversarial noise.
Both search directions show no preference for any region or pixel in the image.
In other words, under a uniform initial random adversarial noise, the Boundary Attack performs noise compression with basically the same magnitude for all pixels. In fact, the Boundary Attack does not make any distinction between different pixels in the image, and the noise compression of each pixel is proportional to its initial noise magnitude [31].  Proposed Method 3.1 Notation
Suppose F is the target model to be attacked: F : X N → Y C , where X represents the input space,
N is the dimension ( N = Width ×
Height × Channel for image data) and Y represents the classiﬁcation space with C categories.
The goal of decision-based attack can be expressed as:
[Equation]
[Figure] where x represents the original image, x (cid:48) refers to adversarial example, y is the label of x , S Q is the set of all adversarial examples generated to query the target model, T is the limit of query number.
v refers to the norm used to measure the noise magnitude including (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ norm.
In this paper the (cid:96) 2 distance is calculated.
In the process of decision-based attack, the attacker can only obtain the hard label F ( x (cid:48) )
output by the target model.
3.2 Noise Sensitivity of ViTs
Here we measure the noise sensitivity Sens of one patch in the image according to the maximum ratio the noise can be compressed under the current adversarial example on ViT:
Deﬁnition 1.
Let x (cid:48) be an adversarial example of ViT model F on the original image x , i.e.,
F ( x (cid:48) )
(cid:54) =
F ( x ) , and z be the current adversarial noise z = x (cid:48) − x .
Let ˜ z be a new adversarial noise compressed from z in a rectangle patch with width of w , height of h , top left corner of sr
[Equation]
where r and c refer to the row and column index of one pixel in noise z , respectively.
κ ∈ [0 , 1] denotes the noise compression ratio.
Deﬁne the noise sensitivity of a rectangle patch as the minimum noise compression ratio κ min when F misclassiﬁes x + ˜ z : Sens
[Equation] F [Equation] x
[Equation] x (cid:48) , sr, sc [Equation] h
[Equation] w [Equation] = κ min , s.t.
[Equation] ∀ [Equation] x
[Equation]
Sens measures the minimum amount of noise that is required for an adversarial example.
A smaller
Sens means that more noise can be removed without changing the misclassiﬁcation results, i.e., adding noise in this patch has less impact on classiﬁcation results.
When h = w = 1 , Sens measures the pixel-level noise sensitivity.
We use a vision transformer vit-tiny-patch16 [34] with patch size of 16 trained on ILSVRC- [27] as the target model. To demonstrate the difference of patch-wise noise sensitivity in Fig.
2, we add initial Gaussian noise to the original images until they are misclassiﬁed. After getting the initial noise, we try to reduce the noise on each patch to evaluate the patch-wise noise sensitivity of the images on the target model. Since the size of the original image is 224 × 224 × 3 , there are 14 ×patches. We use binary search to evaluate Sens on each patch:
[Equation]
where row, col ∈ [1 , 14] refer to the row index and column index of one patch in the image, respectively.
The heat map in Fig.
2 shows the Sens on all 14 × 14 patches.
In the heat map, a lighter color means lower Sens on one patch.
1 indicates that any small noise compression on this patch makes the target model outputs correct label.
0 indicates that the adversarial example remains misclassiﬁed even all noises on this patch are removed.
Considering the attack process of Boundary Attack, the patch-wise sensitivity of adversarial examples and the compression success probability within a patch of one single query have the following relationship: Black-Box Vision
Transformer (cid:28564)
Query
Success
Fail
Initial Adversarial Example (cid:2206) (cid:2191)(cid:2196)(cid:2191)(cid:2202) Initial Noise (cid:2208) (cid:2191)(cid:2196)(cid:2191)(cid:2202) Update
Current Noise (cid:2208) (cid:1499)
Current Adversarial
Example (cid:2206) (cid:1499)
Noise Magnitude Mask (cid:2169) (cid:2170)
[Figure]
(cid:2169) (cid:2175)
Update
Record
Record
[Figure]
Initialize
Initialize Update Proposition 1. Assume x (cid:48) is an initial adversarial example generated by Boundary Attack against ViT
F starting from original image x , F ( x ) (cid:54) =
F ( x (cid:48) ) .
For any 0 < r 1 , r 2 , h ≤ Height , 0 < c 1 , c 2 , w ≤
Width , if Sens ( F, x, x (cid:48) , r 1 , c 1 , h, w ) <
Sens ( F, x, x (cid:48) , r 2 , c 2 , h, w ) , and the new noise added by one step by Boundary Attack is z (cid:48) , then P ( F ( x (cid:48)
+ z [Equation] 1 ) (cid:54) =
F ( x ) | F ( x [Equation]
+ z (cid:48) )
=
F ( x )) <
P ( F ( x (cid:48)
+ z (cid:48) 2 ) (cid:54) =
F ( x ) | F ( x (cid:48)
+ z (cid:48) )
=
F ( x )) , where for ι = 1 ,z (cid:48)
[Equation] r [Equation] , if
[Equation] else [Equation]
Proof Idea.
The expectation of each pixel on the initial noise x (cid:48) generated by Boundary Attack is equal, and the noise compression ratio after one-step Boundary Attack for each pixel is i.i.d.
The possibility that the noise compression ratio on at least one pixel exceeds κ is also the same for any patch.
Since the probability that noise compression ratio on at least one pixel exceeds Sens increases monotonically w.r.t.
the noise sensitivity on the whole patch, the noise removal of high Sens patch is more likely to be the cause of misclassiﬁcation failures.
(cid:3)
The detailed proof of Proposition 1 is provided in the Supplementary Material A.2.
Propositionindicates that during the noise compression process of Boundary Attack, patches with higher Sens are more likely to be the cause of query failure than regions with lower Sens .
Obviously, the noise sensitivity of ViTs varies greatly in different regions of the image.
In the right part of Fig. 2, we compare the patch-wise sensitivity of original image (b) and the probability that noise compression ratio exceeding Sens on each patch P ( F ( x (cid:48)
+ z (cid:48) b ) (cid:54) =
F ( x ) | F ( x (cid:48)
+ z (cid:48) )
=
F ( x )) caused by one step of Boundary Attack.
As can be seen that these two heat maps are basically consistent on patches, which veriﬁes Proposition 1.
It can be seen from Fig.
2 that completely removing noises of many patches from x init does not affect the misclassiﬁcation.
However, the uniform compression of
Boundary Attack usually keeps these redundant noises to the end.
The magnitude of the whole block of redundant noise is considerable, especially for ViTs which require a larger initial noise.
3.3 Patch-wise Adversarial Removal
According to Proposition 1, the Boundary Attack compresses the overall noise of x (cid:48) together, whose noise compression rate depends on those patches with the highest Sens .
Ideally, a decision-based attack should ﬁrstly compresses regions with low Sens and high noise magnitude.
In this way, both
Input: Target model F ( x ) , noise magnitude limit τ , original image x and label y
Max querying number T , initial variance of gaussian distribution var
Identity matrix I of the same dimension as x , initial and minimum patch size PS 0 and PS min
Output: Adversarial example x ∗ with compressed noise while F ( x init ) = y do ξ init ∼ N (0 , var 2 I ) , x init ← Clip x,τ { x + ξ Gau} , var ← var ∗ 2 ,
T ← T −
Initialize M N and M S according to Eqn. (8) and Eqn.
PS ← PS 0 , x ∗ ← x init while T > 0 do
M Q ← M N (cid:12) M S if (cid:80) M Q
{+= 
+}
then
PS ← PS/ 2 , initialize M N and M S according to Eqn.
(8) and Eqn.
(9) if PS ≤ PS min then break // Locate the highest value in query value mask row ∗ , col ∗ ← argmax ( M Q )
z query ← x ∗ − x, z queryrow ∗ ∗ PS +1:( row ∗ +1) ∗ PS,col ∗ ∗ PS +1:( col ∗ +1) ∗ PS ← 0 x query ← Clip x,τ { z query + x } if F ( x query ) (cid:54) = y then x ∗ ← x query , update M N else M Srow ∗ ,col ∗ ←
T ← T −the success rate of query and the magnitude of one-step noise compression can be guaranteed, and the efﬁciency of noise compression can be maximized with limited query numbers.
On the one hand, the noise sensitivity of the target model to different regions of the initial noise cannot be directly obtained.
On the other hand, using binary search similar in Fig. 2 for one patch and perform grid search for all the patches consumes large quantity of queries.
Therefore, we propose a new decision-based attack
Patch-wise Adversarial Removal (PAR).
PAR divides the initial noise into patches, probes their noise sensitivity and compresses the noise in a patch-wise manner.
As illustrated in Fig.
3, PAR guides the probe process by maintaining two masks that record the noise sensitivity of the target model and the noise magnitude of each mask, respectively.
Since the detail of the ViTs is not available in the black-box attack setting, PAR does not assume the patch size of ViTs, but starts from a large patch size to conduct multiple rounds of coarse-to-ﬁne search.
Firstly, PAR initializes the noise sensitivity mask M S and noise magnitude mask M N .
The shape of the two masks is PS 0 × PS 0 , where PS 0 is a hyper parameter denotes the initial patch size of PAR.
We use the initial noise magnitude in (cid:96) 2 of each patch in x init to initialize M N :
[Equation]
where row and col indicate the row index and column index of M N , row, col ∈ [1 , PS 0 ] .
The noise sensitivity mask is binary.
1 in M S indicates that the corresponding patch in the adversarial noise remains a low noise sensitivity state, which may not have been queried or noise removal has been successfully carried out.
in M S indicates that the previous noise compression process is failed.
The initial value for each element in M S is 1.
Before each query to the target model, we use element-wise product to obtain the query value mask M Q :
[Equation]
where J is a unit matrix of all-ones.
If a patch does not contain noise or previous query is failed, there is no query value.
We sort the values in M Q in descending order, and remove the noise in the patch corresponding to the highest value in M Q .
We input the updated adversarial examples x query into the target model to obtain the
[Table]
query result.
If x [-init while T > 0 do M Q ← M N (cid:12) M S if (cid:80) M Q = 
-]query still misclassiﬁes the target model, it indicates that the noise sensitivity of this patch is low.
In this case, we set x ∗ as x query and update M N .
Otherwise, the noise sensitivity of the patch is high, and the corresponding element in noise sensitivity mask M S is set to 0.
If the sum of M Q is 0, all patches under current patch size PS either have no noise, or the query has already been made.
In this case, we halve the patch size and reinitialize M N and M S according to
Eqn.
(8) and Eqn.
The next round will conduct more ﬁne-grained query on the patches where there are still noises.
Since the noise on some patches have been removed in the previous rounds, the search process of PAR with gradually reduced patch size is much more efﬁcient in terms of queries than using very small patch size at the beginning.
There are two exit conditions for the search process of PAR, either the max query number T is reached or the minimum patch size PS min has been reached.
The PS min is set for the efﬁciency of one-step noise compression.
When the patch size is too small, the compressed noise magnitude after one step is not worthwhile even if the subsequent query succeeds.
Algorithm 1 details PAR.
3.4 PAR as Noise Initialization Method
As an query-efﬁcient decision-based attack, PAR can also be used as a noise initialization method for other decision-based methods.
PAR removes all possible blocks of noise larger than the minimum patch size PS min , leaving the remaining regions to be noise-sensitive for ViTs.
In this way, PAR greatly reduces the search space for subsequent noise compression.
After initializing noises with PAR, decision-based attacks may concentrate each sampling in the regions with higher noise sensitivity.
Experiments 4.1 Experiment Settings
We conduct experiments on three image classiﬁcation datasets: ImageNet-21k [26], ILSVRC- [27], and Tiny-Imagenet [28].
We pick 10000 images from the validation sets of ImageNet-21k and
ILSVRC-2012 that can be correctly classiﬁed by all target models for test.
As for Tiny-Imagenet with 200 image categories, we choose 2000 images, 10 images for each category.
10 vision transformer models with different structures [34] are compared: vit-s32, vit-b16, vit-b32, r50-l32, r50-s32, vit-large-patch16-224, vit-tiny-patch16-224, vit-small-r26-s32-224, vit-tiny-patch16-224, vit-smallpatch16-224.
We also include 8 CNNs as target models: resnet-18 [35], resnet-101, inception [-v
 [36],inception-resnet-]v3 [36], inception-resnet v2 [37], nasnet [38], densenet-161 [39], vgg19-bn [40], senet-154 [41].
4 RTX[-
-]
GPU cards are used for calculation.
We compare 7 decision-based attacks with our PAR under the black-box setting with limited query number: Boundary Attack [15], Biased Boundary Attack (BBA) [20], Evolutionary Attack (Evo) [19],
[Table] [Table] HSJA [30], CAB [31], Sign SurFree [32].
Stepsizes of spherical direction and [-sourcedirection are δ 
-]source direction
0 =
Boundary BBA CAB.
BBA [20] we use the version that does not incorporate information from a substitute model at step for comparison.
Noise magnitude limit τ = [0 , 255] .
The initial and minimum patch size for ImageNet-21k and
ILSVRC-2012 is set to 56 and 7, respectively.
For Tiny-Imagenet, we set the initial and minimum patch size as 16 and 2, respectively.
All datasets are under BSD 3-Clause License.
For evaluation criterion, we choose the median and average size of adversarial perturbation, as applied in NIPS 2018 Adversarial Vision Challenge [28]:
[Equation]
where n data is the number of images in a dataset, x is an original image in the dataset X .
x (cid:48) is the adversarial example found that is closest to x .
A smaller (cid:96) 2 distance indicates a better adversarial example.
It is worth noting that adversarial examples are rounded before being input to the target model for a more realistic black-box attack setting.
[Figure] [Figure]
[Table]
[Figure]
4.2 Experimental Results
To verify the advantage of PAR over existing decision-based attacks on ViTs and CNNs [Figure] we report the median and average adversarial perturbation on ImageNet-21k, ILSVRC-2012, and Tiny-Imagenet in
Table 2, Table [Figure] and Table 1, respectively.
The ﬁrst row of three tables represents target models with different structures.
We compare the average (Avg) and median (Mid) noise magnitude generated by
PAR and other 6 attacks on different target models.
We also use PAR as the noise initialization for other decision-based attacks.
The noise compressed by PAR is handed over to other decision-based attacks for further compression.
It can be seen that when PAR is used to initialize adversarial noise, the average and median noise magnitude drops signiﬁcantly compared with only using the original decision-based attack.
This veriﬁes the strong noise compression ability of PAR.
We also combine
PAR with other decision-based attacks, and compare the query efﬁciency under total query numbers of 3000 in Fig.
The target models are notated under each subﬁgure.
There is a noticeable drop in noise magnitude when initializing the noise with PAR.
Table 5 compares the noise compression efﬁciency and the average query number of PAR under different initial patch sizes and minimum patch sizes.
The compressed noise is handed over to
Boundary Attack for further compress until 1000 queries.
It can be seen that when the initial patch size is small, the average query number will be large, resulting in low query efﬁciency and less query number for subsequent decision-based attacks.
A more reasonable strategy is to use a large initial patch size and stop at a small minimum patch size. We also extend PAR to targeted attack. We randomly choose an image of target class as starting point and keep the adversarial examples in target class.
The target model is vit-small-r26-s32.
Targeted results are shown in Table 4.
The targeted noise of PAR is still signiﬁcantly smaller than others.
Fig.
5 compares adversarial noises generated against vit-small-r26-s32-224 by seven different attacks on ILSVRC-2012.
The ﬁrst row shows the original images.
The second to eighth images of each row are the noises generated by each attack.
PAR stops when the exit condition is met.
All other attacks perform 1000 queries on the target model for each adversarial example.
The noises of PAR mainly concentrate on a few patches instead of spreading over the entire image.
The noise magnitude of other decision-based attacks decrease signiﬁcantly when PAR is used for noise initialization. PAR [Table]
[Figure] proposed in this paper is only used for the study of adversarial machine learning and the robustness of
ViTs, and does not target any real system.
There is no potential negative impact.
More experimental results on different target ViT and CNN structures are provided in the Supplementary Material A.3.
Conclusion
In this paper, we explore decision-based adversarial attacks against Vision Transformers.
In view of the huge difference in the noise sensitivity between patches of ViTs, we propose Patch-wise
Adversarial Removal to achieve query-efﬁcient noise compression.
PAR maintains noise magnitude and noise sensitivity masks to probe and compress adversarial noises in a patch-wise manner, and improve query efﬁciency through a coarse-to-ﬁne search process on the patch size.
Experiments on three image classiﬁcation datasets verify the feasibility and generalizability of PAR to improve the query efﬁciency with limited query number. Acknowledgements
This work is supported by the National Natural Science Foundation of China (NSFC) (under Grant 61876130, 61932009) and the Open Foundation of Henan Key Laboratory of Cyberspace Situation
Awareness (No HNTS2022028).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]