Original: H ARD - LABEL M ANIFOLDS : U NEXPECTED ADVANTAGES OF QUERY EFFICIENCY FOR FINDING ONMANIFOLD ADVERSARIAL EXAMPLES
Revised: H ARD - LABEL M ANIFOLDS : U NEXPECTED ADVANTAGES OF QUERY EFFICIENCY FOR FINDING ONMANIFOLD ADVERSARIAL EXAMPLES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Designing deep networks robust to adversarial examples remains an open problem.
Revised: Designing deep networks robust to adversarial examples remains an open problem.

Original: Likewise, recent zeroth order hard-label attacks on image classiﬁcation tasks have shown comparable performance to their ﬁrst-order alternatives.
Revised: Likewise, recent zeroth order hard-label attacks on image classiﬁcation tasks have shown comparable performance to their ﬁrst-order alternatives.

Original: It is well known that in this setting, the adversary must search for the nearest decision boundary in a query-efﬁcient manner.
Revised: It is well known that in this setting, the adversary must search for the nearest decision boundary in a query-efﬁcient manner.

Original: State-of-the-art (SotA) attacks rely on the concept of pixel grouping, or super-pixels, to perform efﬁcient boundary search.
Revised: State-of-the-art (SotA) attacks rely on the concept of pixel grouping, or super-pixels, to perform efﬁcient boundary search.

Original: It was recently shown in the ﬁrst-order setting, that regular adversarial examples leave the data manifold, and on-manifold examples are generalization errors (Stutz et al., 2019).
Revised: It was recently shown in the ﬁrst-order setting, that regular adversarial examples leave the data manifold, and on-manifold examples are generalization errors (Stutz et al., 2019).

Original: In this paper, we argue that query efﬁciency in the zeroth-order setting is connected to the adversary’s traversal through the data manifold.
Revised: In this paper, we argue that query efﬁciency in the zeroth-order setting is connected to the adversary’s traversal through the data manifold.

Original: In particular, query-efﬁcient hard-label attacks have the unexpected advantage of ﬁnding adversarial examples close to the data manifold.
Revised: In particular, query-efﬁcient hard-label attacks have the unexpected advantage of ﬁnding adversarial examples close to the data manifold.

Original: We empirically demonstrate that against both natural and robustly trained models, an efﬁcient zeroth-order attack produces samples with a progressively smaller manifold distance measure.
Revised: We empirically demonstrate that against both natural and robustly trained models, an efﬁcient zeroth-order attack produces samples with a progressively smaller manifold distance measure.

Original: Further, when a normal zeroth-order attack is made query-efﬁcient through the use of pixel grouping, it can make up to a two-fold increase in query efﬁciency, and in some cases, reduce a sample’s distance to the manifold by an order of magnitude.
Revised: Further, when a normal zeroth-order attack is made query-efﬁcient through the use of pixel grouping, it can make up to a two-fold increase in query efﬁciency, and in some cases, reduce a sample’s distance to the manifold by an order of magnitude.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Adversarial examples in the context of deep learning models were originally investigated as blind spots in classiﬁcation (Szegedy et al., 2013; Goodfellow et al., 2014).
Revised: Adversarial examples in the context of deep learning models were originally investigated as blind spots in classiﬁcation (Szegedy et al., 2013; Goodfellow et al., 2014).

Original: Formalized methods for discovering these blind spots emerged, referred to as gradient-level attacks, and became the ﬁrst style to reach widespread attention by the deep learning community (Papernot et al., 2016; MoosaviDezfooli et al., 2015; Carlini & Wagner, 2016; 2017).
Revised: Formalized methods for discovering these blind spots emerged, referred to as gradient-level attacks, and became the ﬁrst style to reach widespread attention by the deep learning community (Papernot et al., 2016; MoosaviDezfooli et al., 2015; Carlini & Wagner, 2016; 2017).

Original: In order to compute the necessary gradient information, such techniques required access to model parameters and a sizeable query budget, needing surrogate information to be competitive (Papernot et al., 2017).
Revised: In order to compute the necessary gradient information, such techniques required access to model parameters and a sizeable query budget, needing surrogate information to be competitive (Papernot et al., 2017).

Original: This naturally led to the creation of score-level attacks, which only require the conﬁdence values output by machine learning models (Fredrikson et al., 2015; Tram`er et al., 2016; Chen et al., 2017; Ilyas et al., 2018).
Revised: This naturally led to the creation of score-level attacks, which only require the conﬁdence values output by machine learning models (Fredrikson et al., 2015; Tram`er et al., 2016; Chen et al., 2017; Ilyas et al., 2018).

Original: However, the grand prize of adversarial ML (AML), hard-label attacks, have been proposed in very recent years.
Revised: However, the grand prize of adversarial ML (AML), hard-label attacks, have been proposed in very recent years.

Original: These methods, which originated from a random-walk on the decision boundary (Brendel et al., 2017), have been carefully reﬁned to offer convergence guarantees (Cheng et al., 2019), query efﬁciency (Chen et al., 2019; Cheng et al., 2020), and simplicity through super-pixel grouping (Chen & Gu, 2020), without ever sacriﬁcing earlier advances.
Revised: These methods, which originated from a random-walk on the decision boundary (Brendel et al., 2017), have been carefully reﬁned to offer convergence guarantees (Cheng et al., 2019), query efﬁciency (Chen et al., 2019; Cheng et al., 2020), and simplicity through super-pixel grouping (Chen & Gu, 2020), without ever sacriﬁcing earlier advances.

Original: Despite the steady improvements of hard-label attacks, open questions persist about their behavior, and AML attacks at large.
Revised: Despite the steady improvements of hard-label attacks, open questions persist about their behavior, and AML attacks at large.

Original: The existence of adversarial samples were originally assumed to lie in rare pockets of the input space (Goodfellow et al., 2014), but this was later challenged by the boundary tilting assumption (Tanay & Grifﬁn, 2016;
Revised: The existence of adversarial samples were originally assumed to lie in rare pockets of the input space (Goodfellow et al., 2014), but this was later challenged by the boundary tilting assumption (Tanay & Grifﬁn, 2016;

Original: Gilmer et al., 2018), which adopts a “data-geometric” view of the input space living on a manifold.
Revised: Gilmer et al., 2018), which adopts a “data-geometric” view of the input space living on a manifold.

Original: This is supported by Stutz et al.
Revised: This is supported by Stutz et al.

Original: (2019), who suggest that regular adversarial examples leave the data manifold, while on-manifold adversarial examples are generalization errors.
Revised: (2019), who suggest that regular adversarial examples leave the data manifold, while on-manifold adversarial examples are generalization errors.

Original: In this paper, we adopt the boundary-tilting assumption and demonstrate an unexpected beneﬁt of query-efﬁcient zeroth order attacks; such attacks are more likely to discover on-manifold examples, and it is primarily enabled by the use of down-scaling techniques, such as super-pixel grouping.
Revised: In this paper, we adopt the boundary-tilting assumption and demonstrate an unexpected beneﬁt of query-efﬁcient zeroth order attacks; such attacks are more likely to discover on-manifold examples, and it is primarily enabled by the use of down-scaling techniques, such as super-pixel grouping.

Original: This is initially counter-intuitive, since down-scaling techniques reduce the search dimension, which artiﬁcially limits the search space, and can lead to worse (farther-away) adversarial examples.
Revised: This is initially counter-intuitive, since down-scaling techniques reduce the search dimension, which artiﬁcially limits the search space, and can lead to worse (farther-away) adversarial examples.

Original: Our results suggest, however, that super-pixels help eliminate the search space of off-manifold adversarial examples, leading to examples which are truly generalization errors .
Revised: Our results suggest, however, that super-pixels help eliminate the search space of off-manifold adversarial examples, leading to examples which are truly generalization errors .

Original: With this knowledge, it is possible to rethink the design of hard-label attacks, towards those which resemble attack (b) in
Revised: With this knowledge, it is possible to rethink the design of hard-label attacks, towards those which resemble attack (b) in

Original: Figure 1, rather than (a) or (c).
Revised: Figure 1, rather than (a) or (c).

Original: [Figure]
Revised: [Figure]

Original: Our speciﬁc contributions are as follows:
Revised: Our speciﬁc contributions are as follows:

Original: • Reveal a novel nuance of zeroth order attacks.
Revised: • Reveal new insights of manifold feedback during query-efﬁcient zeroth-order search.

Original: We describe an approach for extending dimension-reduction technqiues in the score-level setting (Tu et al., 2019) to hard-label attacks.
Revised: We describe an approach for extending dimension-reduction technqiues in the score-level setting (Tu et al., 2019) to hardlabel attacks.

Original: Afterwards, we propose the use of FID score (Heusel et al., 2018) as an L p -agnostic means for estimating distance to the sampled submanifold.
Revised: Afterwards, we propose the use of FID score (Heusel et al., 2018) as an L p -agnostic means for estimating distance to the sampled submanifold.

Original: This measure allows to empirically demonstrate the connection between query efﬁciency and on-manifold adversarial examples, beyond the known convergence rates tied to dimensionality (Nesterov & Spokoiny, 2017). 
Revised: This measure allows to empirically demonstrate the connection between query efﬁciency and manifold feedback from the model, beyond the known convergence rates tied to dimensionality (Nesterov & Spokoiny, 2017). We later tie this result to known behavior in the gradientlevel setting (Engstrom et al., 2019), which shows that manifold information is leaked from models in the hardlabel setting, regardless of their robustness.

Original: • Attack-agnostic method for super-pixel grouping.
Revised: • Attack-agnostic method for super-pixel grouping.

Original: We show that bilinear down-scaling of the input space act as a form of super-pixel grouping, which yields up to 140% and 210% query efﬁciency gain for previouslyproposed HSJA (Chen et al., 2019) and Sign-OPT attack (Cheng et al., 2020), respectively. 
Revised: We show that bilinear down-scaling of the input space act as a form of super-pixel grouping, which yields up to 140% and 210% query efﬁciency gain for previouslyproposed HSJA (Chen et al., 2019) and Sign-OPT attack (Cheng et al., 2020), respectively. These results align with previously observed behavior in the scorelevel setting (Tu et al., 2019).

Original: •  Our analysis covers a comprehensive array of datasets, model regularization methods, and L p -norm settings from the literature.
Revised: • Introduction of manifold distance oracle. Our analysis covers a comprehensive array of datasets, model regularization methods, and L p -norm settings from the literature.

Original: Regardless of the setting, we observe a consistent behavior in query-efﬁcient attacks.
Revised: Regardless of the setting, we observe a consistent behavior of leveraging manifold information during query-efﬁcient attacks.

Original: Thus we propose an information-theoretical formulation and interpretation of the noisy manifold distance oracle , which enables zeroth-order attacks to craft onmanifold examples.
Revised: Thus we propose an information-theoretical formulation and interpretation of the noisy manifold distance oracle , which enables zeroth-order attacks to craft on-manifold examples.

Original: Studying this problem may assist in understanding the fundamental limits and utility of hard-label attacks
Revised: Studying this problem may assist in understanding the fundamental limits and utility of hard-label attacks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Since the original discovery of adversarial samples (Szegedy et al., 2013; Goodfellow et al., 2014) and later formulations based on optimization (Carlini & Wagner, 2016; Moosavi-Dezfooli et al., 2015), the prevailing question was why such examples existed.
Revised: Since the original discovery of adversarial samples (Szegedy et al., 2013; Goodfellow et al., 2014) and later formulations based on optimization (Carlini & Wagner, 2016; Moosavi-Dezfooli et al., 2015), the prevailing question was why such examples existed.

Original: The original assumption was that adversarial examples lived in low-probability pockets of the input space, and were thus never encountered during parameter optimization (Szegedy et al., 2013).
Revised: The original assumption was that adversarial examples lived in low-probability pockets of the input space, and were thus never encountered during parameter optimization (Szegedy et al., 2013).

Original: This effect was believed to be ampliﬁed by the linearity of weight activations in the presence of small perturbations (Goodfellow et al., 2014).
Revised: This effect was believed to be ampliﬁed by the linearity of weight activations in the presence of small perturbations (Goodfellow et al., 2014).

Original: These assumptions were later challenged by the manifold assumption, which in summary claims that 1) the train and test sets of a model only occupy a sub-manifold of the true data, while the decision boundary lies close to samples on and beyond the sub-manifold (Tanay & Grifﬁn, 2016), and 2) the “data geometric“ view, where high-dimensional geometry of the true data manifold enables a low-probability error set to exist (Gilmer et al., 2018).
Revised: These assumptions were later challenged by the manifold assumption, which in summary claims that 1) the train and test sets of a model only occupy a sub-manifold of the true data, while the decision boundary lies close to samples on and beyond the sub-manifold (Tanay & Grifﬁn, 2016), and 2) the “data geometric“ view, where high-dimensional geometry of the true data manifold enables a low-probability error set to exist (Gilmer et al., 2018).

Original: Likewise the manifold assumption describes adversarial samples as leaving the manifold, which has inspired many defenses based on projecting such samples back to the data manifold (Jalal et al., 2019; Samangouei et al., 2018), and adaptive attacks for foiling these defenses (Carlini et al., 2019; Carlini & Wagner, 2017; Tramer et al., 2020).
Revised: Likewise the manifold assumption describes adversarial samples as leaving the manifold, which has inspired many defenses based on projecting such samples back to the data manifold (Jalal et al., 2019; Samangouei et al., 2018a), and adaptive attacks for foiling these defenses (Carlini et al., 2019; Carlini & Wagner, 2017; Tramer et al., 2020).

Original: We investigate the scenario where an adversary uses zeroth-order information to either estimate the gradient direction, or ﬁnd the closest decision boundary, related to previous work in the gradient-level setting by Stutz et al.
Revised: We investigate the scenario where an adversary uses zeroth-order information to either estimate the gradient direction, or ﬁnd the closest decision boundary, related to previous work in the gradient-level setting by Stutz et al.

Original: In our setting, the adversary uses the top-1 label feedback from the model to reach their goal.
Revised: In our setting, the adversary uses the top-1 label feedback from the model to reach their goal.

Original: They can also use a low-dimensional approximation of the data manifold to encourage query-efﬁciency.
Revised: They can also use a low-dimensional approximation of the data manifold to encourage query-efﬁciency.

Original: However, to date it is not completely understood how this affects the traversal through the data manifold, particularly in the zeroth-order setting.
Revised: However, to date it is not completely understood how this affects the traversal through the data manifold, particularly in the zeroth-order setting.

Original:  Z EROTH - ORDER SEARCH THROUGH THE MANIFOLD
Revised:  Z EROTH - ORDER SEARCH THROUGH THE MANIFOLD

Original: Our primary motivation is to characterize recent zeroth-order attacks as they relate to ideas of manifold traversal (Chen et al., 2019; Cheng et al., 2020; Chen & Gu, 2020).
Revised: Our primary motivation is to characterize recent zeroth-order attacks as they relate to ideas of manifold traversal (Chen et al., 2019; Cheng et al., 2020; Chen & Gu, 2020).

Original: In the most common problem setting, the adversary is interested in attacking a K -way multi-class classiﬁcation model f : R d → { 1 , . .
Revised: In the most common problem setting, the adversary is interested in attacking a K -way multi-class classiﬁcation model f : R d → { 1 , . .

Original: . , K } .
Revised: . , K } .

Original: Given an original example x 0 , the goal is to generate adversarial example x such that x is close to x 0 and f ( x ) (cid:54) =
Revised: Given an original example x 0 , the goal is to generate adversarial example x such that x is close to x 0 and f ( x ) (cid:54) =

Original: f
Revised: f

Original: ( x 0 ) , where closeness is often approximated by the L p -norm of x − x 0 .
Revised: ( x 0 ) , where closeness is often approximated by the L p -norm of x − x 0 .

Original: The value of this approximation is debated in the literature (Heusel et al., 2018;
Revised: The value of this approximation is debated in the literature (Heusel et al., 2018;

Original: Tsipras et al., 2018; Engstrom et al., 2019).
Revised: Tsipras et al., 2018; Engstrom et al., 2019).

Original: Likewise we turn to alternative methods shown later for measuring closeness.
Revised: Likewise we turn to alternative methods shown later for measuring closeness.

Original: First we step through the formulation for contemporary hard-label attacks, then show how dimension-reduced score-level attacks are extended to the hard-label setting, which enables analysis of zeroth-order decision boundary search.
Revised: First we step through the formulation for contemporary hard-label attacks, then show how dimension-reduced score-level attacks are extended to the hard-label setting, which enables analysis of zeroth-order decision boundary search.

Original: 3. G RADIENT - LEVEL FORMULATION
Revised: 3. G RADIENT - LEVEL FORMULATION

Original: For gradient-level attacks, the goal is satisﬁed by ﬁrst assuming that f ( x ) = argmax i ( Z ( x ) i ) , where
Revised: For gradient-level attacks, the goal is satisﬁed by ﬁrst assuming that f ( x ) = argmax i ( Z ( x ) i ) , where

Original: Z ( x ) ∈ R K is the ﬁnal (logit) layer output, and Z ( x ) i is the prediction score for the i -th class, the stated goal is satisﬁed by the optimization problem,
Revised: Z ( x ) ∈ R K is the ﬁnal (logit) layer output, and Z ( x ) i is the prediction score for the i -th class, the stated goal is satisﬁed by the optimization problem,

Original: [Equation]
Revised: [Equation]

Original: for the Euclidean L p -norm || · || p , L ( · ) is the loss function corresponding to the goal of the attack, and c is a regularization parameter.
Revised: for the Euclidean L p -norm || · || p , L ( · ) is the loss function corresponding to the goal of the attack, and c is a regularization parameter.

Original: A popular choice of loss function is the Carlini & Wagner (2016) loss function.
Revised: A popular choice of loss function is the Carlini & Wagner (2016) loss function.

Original: 3. S CORE - LEVEL AND HARD - LABEL ATTACKS
Revised: 3. S CORE - LEVEL AND HARD - LABEL ATTACKS

Original: In the gradient-level setting, we require the gradient ∇ f ( · ) .
Revised: In the gradient-level setting, we require the gradient ∇ f ( · ) .

Original: However, in the score-level setting we are forced to estimate ∂f ( x ) ∂ x without access to ∇ f ( · ) , only evaluations of Z ( · ) .
Revised: However, in the score-level setting we are forced to estimate ∂f ( x ) ∂ x without access to ∇ f ( · ) , only evaluations of Z ( · ) .

Original: Tu et al.
Revised: Tu et al.

Original: (2019) reformulate the previous problem to a version relying instead on the ranking of class predictions from
Revised: (2019) reformulate the previous problem to a version relying instead on the ranking of class predictions from

Original: Z .
Revised: Z .

Original: In practical scenarios, the estimate is found using random gradient-free method (RGF), a scaled random full gradient estimator of ∇ f ( x ) , over q random directions { u j } qj =1 .
Revised: In practical scenarios, the estimate is found using random gradient-free method (RGF), a scaled random full gradient estimator of ∇ f ( x ) , over q random directions { u i } q i =1 .

Original: The score-level setting was extended to several renditions of the hard-label setting, which we clarify below.
Revised: The score-level setting was extended to several renditions of the hard-label setting, which we clarify below.

Original: In each case the goal is to approximate the gradient by ˆ g .
Revised: In each case the goal is to approximate the gradient by ˆ g .

Original: OPT-Attack For given example x 0 , true label y 0 , and hard-label black-box function f : R d → { 1 , . . .
Revised: OPT-Attack For given example x 0 , true label y 0 , and hard-label black-box function f : R d → { 1 , . . .

Original: , K } , Cheng et al.
Revised: , K } , Cheng et al.

Original: (2019) deﬁne the objective function g : R d → R as a function of search direction θ , where g ( θ ∗ ) is the minimum distance from x 0 to the nearest adversarial example along the direction θ .
Revised: (2019) deﬁne the objective function g : R d → R as a function of search direction θ , where g ( θ ∗ ) is the minimum distance from x 0 to the nearest adversarial example along the direction θ .

Original: For the untargeted attack, g ( θ ) corresponds to the distance to the decision boundary along direction θ , and allows for estimating the gradient as,
Revised: For the untargeted attack, g ( θ ) corresponds to the distance to the decision boundary along direction θ , and allows for estimating the gradient as,

Original: [Equation]
Revised: [Equation]

Original:  Notably, g ( θ ) is continuous even if f is a non-continuous step function.
Revised: where β is a small smoothing parameter. Notably, g ( θ ) is continuous even if f is a non-continuous step function.

Original: Sign-OPT Cheng et al.
Revised: Sign-OPT Cheng et al.

Original: (2020) later improved the query efﬁciency by only considering the sign of the gradient estimate,
Revised: (2020) later improved the query efﬁciency by only considering the sign of the gradient estimate,

Original: [Equation]
Revised: [Equation]

Original: We focus on the Sign-OPT variant, as the ﬁndings are more relevant to the current SotA.
Revised: We focus on the Sign-OPT variant, as the ﬁndings are more relevant to the current SotA.

Original: HopSkipJumpAttack
Revised: HopSkipJumpAttack

Original: Similar to Sign-OPT, HopSkipJumpAttack (HSJA)
Revised: Similar to Sign-OPT, HopSkipJumpAttack (HSJA)

Original: (Chen et al., 2019) uses a zeroth-order sign oracle to improve Boundary Attack proposed by Brendel et al.
Revised: (Chen et al., 2019) uses a zeroth-order sign oracle to improve Boundary Attack proposed by Brendel et al.

Original: HSJA lacks the convergence analysis of OPT Attack/Sign-OPT and relies on one-point gradient estimate.
Revised: HSJA lacks the convergence analysis of OPT Attack/Sign-OPT and relies on one-point gradient estimate.

Original: Regardless, HSJA is competitive with Sign-OPT for SotA in the L 2 setting.
Revised: Regardless, HSJA is competitive with Sign-OPT for SotA in the L 2 setting.

Original: RayS Chen & Gu (2020) propose an alternative method which is to search for the minimum decision boundary radius r from a sample x 0 , along a ray direction θ .
Revised: RayS Chen & Gu (2020) propose an alternative method which is to search for the minimum decision boundary radius r from a sample x 0 , along a ray direction θ .

Original: Instead of searching over R d to minimize g ( θ ) , Chen et al.
Revised: Instead of searching over R d to minimize g ( θ ) , Chen et al.

Original: propose to perform ray search over directions θ ∈ {− 1 , 1 } d , resulting in 2 d maximum possible directions.
Revised: propose to perform ray search over directions θ ∈ {− 1 , 1 } d , resulting in 2 d maximum possible directions.

Original: This reduction of the search resolution enables SotA query efﬁciency in the L ∞ setting with proof of convergence.
Revised: This reduction of the search resolution enables SotA query efﬁciency in the L ∞ setting with proof of convergence.

Original: The search resolution is further reduced by the hierarchical variant of RayS, which performs on-the-ﬂy upscaling of image super-pixels.
Revised: The search resolution is further reduced by the hierarchical variant of RayS, which performs on-the-ﬂy upscaling of image super-pixels.

Original: 3. D IMENSION - REDUCED ZEROTH - ORDER SEARCH
Revised: 3. D IMENSION - REDUCED ZEROTH - ORDER SEARCH

Original: Instead of performing search over directions in the input space θ ∈ R d , Tu et al. (2019) experiment in the score-level setting with a search over reduced-dimension directions θ (cid:48)
Revised: The attacks described so far each represent an improvement in query efﬁciency under different

Original: ∈ R d (cid:48) .
Revised: L p -norm scenarios.

Original: The motivation for this stems from the known convergence rate of zeroth ordergradient descent, which is tied to  a dimensionality d of the vectorized input. The convergence rate is O ( (cid:112) d/T ) for number of iterations T , on both convex (Nesterov & Spokoiny, 2017; Liu et al., 2018) and non-convex loss functions (Ghadimi & Lan, 2013).
Revised: The difﬁculty in performing a holistic analysis of their behavior lies in each attack’s unique design. In order to characterize query-efﬁcient attacks in a consistent way, we rely on a hard-label version of the reduced-dimension scheme proposed by Tu et al. This scheme can allow to dynamically scale the expected query-efﬁciency up or down in a controlled manner.

Original: The reduced-dimension search is feasible since it was shown that the intrinsic dimensionality of data is often lower than the true dimension (Amsaleg et al., 2017).
Revised: The reduced-dimension search is feasible since it was shown that the intrinsic dimensionality of data is often lower than the true dimension (Amsaleg et al., 2017).

Revised: Likewise, zeroth-order attacks can exploit the known convergence rate of zeroth order gradient descent, which is tied to a dimensionality d of the vectorized input (Nesterov & Spokoiny, 2017; Liu et al., 2018).

Original: In practice this reduction is implemented through an encoding map E : R d → R d (cid:48)  and decoding map
Revised: In practice this reduction is implemented through an encoding map E : R d → R d (cid:48) for reduced dimension d (cid:48) and decoding map

Original: D : R d (cid:48)
Revised: D : R d (cid:48)

Original: → R d .
Revised: → R d .

Original: The adversarial sample is created by x = x 0 + δ , where δ = D ( θ (cid:48) ) with θ (cid:48) ∈ R d (cid:48)  .
Revised: In general the adversarial sample is created by [Equation] ) where θ (cid:48) ∈ R d (cid:48) and is optimized depending on the respective attack (e.g., Sign-OPT and HSJA), and as before, g is a measure of distance to the decision boundary in direction D ( θ (cid:48) ) .

Original: The mapping functions can be initialized with either an autoencoder (AE), or a pair of channel-wise bilinear transform functions (henceforth referred to as BiLIN) which simply scale the input up and down.
Revised: The mapping functions can be initialized with either an autoencoder (AE), or a pair of channel-wise bilinear transform functions (henceforth referred to as BiLN) which simply scales the input up or down depending on a ﬁxed scaling factor. These choices were previously investigated by Tu et al.

Original: In the case of the autoencoder, it is tuned to minimize reconstruction error ofinput images using disjoint data from the victim.
Revised: as a way to improve query efﬁciency in score-level attacks, and ultimately performed similarly with respect to query efﬁciency. For our purposes, these functions represent two distinct methods of synthesizing adversarial samples, which either rely on an approximate description of the manifold (AE), or instead use a deterministic rescaling to achieve efﬁciency (BiLN).

Original: In the case of BiLIN, no additonal training is required.
Revised: The inclusion of BiLN is important, because it allows to measure the scenario where the adversary has no explicit knowledge of the manifold, and only relies on the feedback from the model. Under the AE scenario, the AE is tuned to minimize reconstruction error of input images. Due to this dependence on labeled data, the output quality of the AE is dependent on the adversary’s ability to collect data, which is a realistic consideration. We model the scenario where the adversary only has access to the test set, which is often considerably less informative than the training set.

Original: We note that the  the output of the AE-initialized decoder can be used in different ways, and leave this implementation detail for Section A.1 of the Appendix.
Revised: This manifests as an extra distortion in addition to the adversarial noise. Thus the output of the AE-initialized decoder can be used in different ways, which we discuss brieﬂy in Section A.3 of the Appendix.

Original: Given this seemingly straightforward gain in query efﬁciency, we are still interested in controlling the error of the boundary search in zeroth order methods. In the lower dimension case, the search takes place in a lower ﬁdelity version of the true data manifold, since the true intrinsic manifold is not known. In order to characterize hard-label methods, we rely on a hard-label version of the reduced-dimension search proposed by Tu et al.
Revised: In the case of BiLN, no additional training is required, which means it synthesizes search direction independent of the adversary’s manifold description (i.e., possible extracted knowledge about test samples). This can manifest as a lower overall distortion, in the case where only a crude manifold description can be extracted from the test set.

Original: (2019), which later allows to unify the concept of query efﬁciency with search ﬁdelity.
Revised: Next we describe the exact usage of the mapping functions for each attack scheme.

Original: Sign-OPT & HSJA.
Revised: Sign-OPT & HSJA.

Original: In general for the attacks which rely on the Cheng et al.
Revised: In general for the attacks which rely on the Cheng et al.

Original: (2019) formulation, the update in Equation 2 becomes
Revised: (2019) formulation, the update in Equation 2 becomes

Original: [Equation]
Revised: [Equation]

Original: for the reduced-dimension Gaussian vectors { u (cid:48) i ∈ R d (cid:48) } qi =0 and direction θ (cid:48) ∈ R d (cid:48) for integer d (cid:48) <
Revised: for the reduced-dimension Gaussian vectors { u (cid:48) i ∈ R d [Equation] } qi =0 and direction θ (cid:48) ∈ R d (cid:48) for integer d (cid:48) <

Original: d .
Revised: d .

Original: The reduced-dimension direction θ (cid:48) is found withE ( θ ) and vice versa with D ( θ (cid:48) ) .
Revised: The reduced-dimension direction θ (cid:48) is initialized randomly with θ (cid:48) ∼ N (0 , 1) for the untargeted case, or for the targeted case as θ (cid:48) =

Revised: E ( x t ) , where x t is a test sample correctly classiﬁed as target class t by the victim model.

Original: Notably this is also applicable for HSJA, since HSJA performs a single-point sign estimate.
Revised: This scheme also applies to HSJA, since HSJA performs a single-point sign estimate.

Revised: As in the normal variants, ˆ g is used to update θ (cid:48) .

Original: RayS.
Revised: RayS.

Original: The intuition behind RayS attack is to perform a discrete search in at most 2 d directions.
Revised: The intuition behind RayS attack is to perform a discrete search in at most 2 d directions.

Original: Chen et al.
Revised: Chen et al.

Original: also perform a hierarchical search over progressively larger super-pixels of the image.
Revised: also perform a hierarchical search over progressively larger super-pixels of the image.

Original: This has the effect of already upscaling on-the-ﬂy (Chen & Gu, 2020).
Revised: This has the effect of already upscaling on-the-ﬂy (Chen & Gu, 2020).

Revised: The inclusion of RayS in our analysis is important, since it has the unique behavior of performing a discrete search for the decision boundary, rather than an explicit gradient estimate.

Original: To achieve an appropriate reduced-dimension version of RayS, and test our hypothesis, we modify the calculation of s in 
Revised: To achieve an appropriate reduced-dimension version of RayS, and test our hypothesis, we modify the calculation of s in Algorithm 3 of Chen &

Original: Algorithm 3, which either speeds up upscaling by a factor a (i.e., s = s + a ), or extends the search through a speciﬁc block index by a factor b (increase block level at k = 2 s b instead of k = 2 s ).
Revised: Gu (2020), which either speeds up upscaling by a factor a (i.e., s = s + a ), or extends the search through a speciﬁc block index by a factor b (increase block level at k = 2 s b instead of k = 2 s ).

Original: 3. C APTURING ZEROTH - ORDER SEARCH DEVIATION
Revised: 3. C APTURING ZEROTH - ORDER SEARCH DEVIATION

Original: Our analysis follows in the wake of ﬁndings presented by (Stutz et al., 2019).
Revised: Our analysis follows in the wake of ﬁndings presented by (Stutz et al., 2019).

Original: It is shown that “regular” gradient-level adversarial examples leave the data manifold i.e., the sample’s distance to the manifold is larger than with an “on-manifold” gradient-level adversarial example.
Revised: It is shown that “regular” gradient-level adversarial examples leave the data manifold i.e., the sample’s distance to the manifold is larger than with an “on-manifold” gradient-level adversarial example.

Original: In the score-level and hard-label settings, the manifold can be used to guide the search for the boundary.
Revised: In the score-level and hard-label settings, the manifold can be used to guide the search for the boundary.

Original: The beneﬁts of this scenario can be observed in score-level attacks by Tu et al.
Revised: The beneﬁts of this scenario can be observed in score-level attacks by Tu et al.

Original: Similarly, hard-label attacks can leverage the concept of super-pixels to achieve gains in performance (Chen & Gu, 2020).
Revised: Similarly, hard-label attacks can leverage the concept of super-pixels to achieve gains in performance (Chen & Gu, 2020).

Original: However, searching over a reduced-dimension imposes a reduced search ﬁdelity, which can limit the  search space tofarther-away adversarial examples. Thus, we are interested in the tradeoff between query efﬁciency and search ﬁdelity.
Revised: However, to date the connection between dimension manipulation and manifold traversal has not been investigated. The reduced search ﬁdelity naturally limits the resolution of search direction, which is the ﬁnal noise vector applied to the original sample, as evidenced by Equation 3.

Original: This introduces our central research question: How does searching over reduced-dimension increase efﬁciency, if the search resolution is decreased as a side-effect?
Revised: This introduces our central research question: How does searching over reduced-dimension increase efﬁciency, if the search resolution is decreased as a side-effect?

Original: To approach this question, we  rely on a sample’s distance to the manifold as a measure of deviation during the search, similar to the work by Stutz et al.
Revised: To approach this question, we observe that certain gradient-level and score-level attacks can leverage a prior manifold description to attack more efﬁciently (Stutz et al., 2019; Tu et al., 2019). We can rely on a sample’s distance to the manifold as a measure of deviation during the search, similar to the work by Stutz et al.

Original: Hereafter, we refer to this distance w.l.o.g as the manifold distance.
Revised: Hereafter, we refer to this distance w.l.o.g as the manifold distance.

Original: Our choice is motivated by the fact that from a data-geometric perspective, manifold distance describes the amount of semantic features preserved during the attack process (Engstrom et al., 2019).
Revised: Our choice is motivated by the fact that from a data-geometric perspective, manifold distance describes the amount of semantic features preserved during the attack process (Engstrom et al., 2019).

Original: Likewise the manifold distance can communicatemore information about attack behavior  than L p distortion measurements, which are the common choice in existing zeroth-order attack literature (Cheng et al., 2020; Chen et al., 2019; Chen & Gu, 2020).
Revised: Likewise the manifold distance can communicate information about attack behavior better than L p distortion measurements, which are the common choice in existing zeroth-order attack literature (Cheng et al., 2020; Chen et al., 2019; Chen & Gu, 2020).

Original: Second, the manifold approximation technique by Stutz et al.
Revised: Second, the manifold approximation technique by Stutz et al.

Original: (2019) is mainly suited for L 2 -norm, whereas hard-label attacks exist for both L 2 and L ∞ -norm. 
Revised: (2019) is mainly suited for L 2 -norm, whereas hard-label attacks exist for both L 2 and L ∞ -norm. Unfortunately, the real data manifold is difﬁcult to describe.

Revised: This is an open problem in the study of Generative Adversarial Networks (GANs), as designers must ensure that generator images are on-manifold using a continuous function.

Original: Thus we rely on the recently proposed Frechet 
Revised: This has motivated the recently proposed Frechet Inception

Original: Inception Distance (FID) as a surrogate measure of the manifold distance (Heusel et al., 2018).
Revised: Distance (FID), which acts as a surrogate measure of the manifold distance (Heusel et al., 2018).

Original:  FID is viable as it computes Frechet distance of candidate images with respect to images sampled from the true data sub-manifold.
Revised: We can leverage FID by treating the adversarial samples as synthetically generated images. FID is viable as it computes Frechet distance of candidate images with respect to images sampled from the true data sub-manifold.

Original: Since FID uses an Inception-V3 coding layer to encode images, this distance will correlate with distortion of high-level features, i.e., features found by searching near the data manifold, which we are primarily interested in.
Revised: Since FID uses an Inception-V3 coding layer to encode images, this distance will correlate with distortion of high-level features, which will be a result of sampling farther from the data manifold. We do not target the Inception-V3 network in any of our experiments, so the FID metric will not rely on any internal aspects of the victim models.

Original:  R ESULTS 4. M ETHODOLOGY
Revised:  R ESULTS 4. M ETHODOLOGY

Original: Our experimental analysis addresses the following three research questions about zeroth-order attacks:
Revised: Our experimental analysis addresses the following three research questions about zeroth-order attacks:

Original: Q1.
Revised: Q1.

Original: What is the trade-off between query efﬁciency and reduced search resolution in the zeroth order setting, against both natural and “robust” models?
Revised: What is the trade-off between query efﬁciency and reduced search resolution in the zeroth order setting, against both natural and “robust” models?

Original: Q2.
Revised: Q2.

Original: In addition to reducing queries, are there unexpected beneﬁts for performing a query-efﬁcient zeroth order attack?
Revised: In addition to reducing queries, are there unexpected beneﬁts for performing a query-efﬁcient zeroth order attack?

Original: Q3.
Revised: Q3.

Original: Compared to the  score-level setting, do dimension-reduced zeroth order attacks produce a similar amount of reduction in query usage?
Revised: Compared to the previous results in the score-level setting (Tu et al., 2019), do dimensionreduced hard-label attacks produce a similar amount of reduction in query usage?

Original: We answer these questions by comparing three hard-label attacks against their dimension-reduced variants.
Revised: We answer these questions by comparing three hard-label attacks against their dimension-reduced variants.

Original: Some variants are not shown due to incompatibility with the base attack.
Revised: Some variants are not shown due to incompatibility with the base attack.

Original: For example,
Revised: For example,

Original: AE+HSJA is not implemented as it relies on only a single-point estimate, thereby only allowing to attack on the manifold directly.
Revised: AE+HSJA is not implemented as it relies on only a single-point estimate, thereby only allowing to attack on the manifold directly.

Original: This is not practical due to induced distortion discussed previously in
Revised: This is not practical due to induced distortion discussed previously in

Original: Section 3.
Revised: Section 3.

Original: RayS can perform two-point search, but it assumes codependence of input features, which may not be the case for well-deﬁned latent space of an autoencoder.
Revised: RayS can perform two-point search, but it assumes codependence of input features, which may not be the case for well-deﬁned latent space of an autoencoder.

Original: Thus for the AE variant, we rely on Sign-OPT as it can perform two-point estimate and does not rely on codependence of features.
Revised: Thus for the AE variant, we rely on Sign-OPT as it can perform two-point estimate and does not rely on codependence of features.

Original: In the BiLN cases, the implementations follow the discussion in Section 3.3.
Revised: In the BiLN cases, the implementations follow the discussion in Section 3.3.

Original: Experimental Highlights.
Revised: Experimental Highlights.

Original: Our experiments show that query-efﬁcient attacks exhibit unexpected behaviors and beneﬁts, with answers summarized below:
Revised: Our experiments show that query-efﬁcient attacks exhibit unexpected behaviors and beneﬁts, with explanations summarized below:

Original: A1.
Revised: A1.

Original: Reduced search resolution incurs a cost relative to the situation.
Revised: Reduced search resolution increases search efﬁciency by allowing the discovery of samples not encountered during training.

Original: BiLN variants can leverage the reduced resolution to more efﬁciently attack both natural and robust models, at the cost of less ﬁne-tuned distortion, seen best with BiLN+HSJA and BiLN+Sign-OPT.
Revised: This is especially true for robust models, which we show have a tendency to overﬁt on the ﬁrst-order noise characteristics.

Original: A2.
Revised: A2.

Original: Surprisingly, query-efﬁcient attacks search closer to the manifold than non-efﬁcient attacks, thus are more likely to produce on-manifold examples. 
Revised: Surprisingly, query-efﬁcient attacks search closer to the manifold than non-efﬁcient attacks, thus are more likely to produce on-manifold examples. This occurs because query-efﬁcient search acts as indirect manifold feedback.

Revised: By reducing the search ﬁdelity, we are more likely to modify high-level features.

Original: This occurs in the absence of direct feedback from the model about manifold distance, and regardless of model regularization.
Revised: This effect is magniﬁed on robust models, which are already known to leak manifold information in gradient-level settings (Engstrom et al

Original: A3.
Revised: A3.

Original: Dimension-reduced attacks are capable of SotA query-efﬁciency gains for HSJA, and a two-fold improvement for Sign-OPT.
Revised: Dimension-reduced attacks are capable of SotA query-efﬁciency gains for HSJA, and a two-fold improvement for Sign-OPT

Original: Setup.
Revised: Setup.

Original: Samples are chosen using the same technique as in Chen et al.
Revised: Samples are chosen using the same technique as in Chen et al.

Original: (2019), ten random classes chosen with ten random samples taken from each (100 samples total).
Revised: (2019), ten random classes chosen with ten random samples taken from each (100 samples total).

Original: All attacks run for 25k queries without early stopping.
Revised: All attacks run for 25k queries without early stopping.

Original: For brevity, we only show results for the untargeted case.
Revised: For brevity, we only show results for the untargeted case.

Original: FID score is calculated using the 64-dimensional max pooling layer of the Inception-V3 deep network for coding (denoted as FID-64 in ﬁgures), taken from an open-source implementation.
Revised: FID score is calculated using the 64-dimensional max pooling layer of the Inception-V3 deep network for coding (denoted as FID-64 in ﬁgures), taken from an open-source implementation.

Original: This allows to calculate  FID without the full 2048 sample count of original FID.
Revised: The choice of the 64-dimensional feature layer allows to calculate full-rank FID without the full 2,048 sample count of original FID, which is prohibitive given the high time complexity of chosen robust models for analysis (in particular, certiﬁed smoothing ImageNet).

Original: This incurs the cost of losing some high-level features, but due to the position of the chosen coding layer in the network, it is still valuable for direct comparison of manifold distance between attacks.
Revised: This incurs the cost of losing some high-level features, but due to the position of the chosen coding layer in the network, it is still valuable for direct comparison of manifold distance between attacks.

Original: Since the coding layer differs from the original implementation, the magnitudes will differ from those published by Heusel et al.
Revised: Since the coding layer differs from the original implementation, the magnitudes will differ from those published by Heusel et al.

Original: Image data consists of the CIFAR-10 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) classiﬁcation datasets.
Revised: Image data consists of the CIFAR-10 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) classiﬁcation datasets.

Original: This selection allows to study attack behavior on both small and highresolution image data.
Revised: This selection allows to study attack behavior on both small and highresolution image data.

Original: We provide further implementation details in Section A.1 of the Appendix.
Revised: We provide further implementation details in Section A.3 of the Appendix.

Original: In addition to natural images, we are interested in the attack behavior for a model regularized with some variant of ﬁrst-order noise.
Revised: In addition to natural images, we are interested in the attack behavior for a model regularized with some variant of ﬁrst-order noise.

Original: For CIFAR-10, we choose the representative adversarial training technique proposed by Madry et al.
Revised: For CIFAR-10, we choose the representative adversarial training technique proposed by Madry et al.

Original: For completeness, we also include ablation results on more recent regularization techniques in Section A.3 of the Appendix.
Revised: For completeness, we also include ablation results on more recent regularization techniques in Section A.5 of the Appendix.

Original: For ImageNet, we compare  against the SotA at time of writing, randomized smoothing proposed by Cohen et al.
Revised: For ImageNet, we compare -fid [Figure] against the SotA at time of writing, randomized smoothing proposed by Cohen et al.

Original: We use the pre-trained Resnet50 weights and implementation provided by Cohen et al., corresponding to smoothing parameter σ = 0 .
Revised: We use the pre-trained Resnet50 weights and implementation provided by Cohen et al., corresponding to smoothing parameter σ = 0 .

Original: 5 and (cid:15) (cid:39) 1 . 0 .
Revised: 5 and (cid:15) (cid:39) 1 . 0 .

Original: 4. E XPERIMENTAL D ETAILS
Revised: 4. E XPERIMENTAL D ETAILS

Original: On each dataset, we target the L p -norm which the robust models were regularized or certiﬁed under.
Revised: On each dataset, we target the L p -norm which the robust models were regularized or certiﬁed under.

Original: Likewise we use the L ∞ versions of each attack for CIFAR-10 and L 2 versions for ImageNet.
Revised: Likewise we use the L ∞ versions of each attack for CIFAR-10 and L 2 versions for ImageNet.

Original: CIFAR-10 case study ( L ∞ ) .
Revised: CIFAR-10 case study ( L ∞ ) .

Original: We start by measuring the distortion against remaining query budget of the adversary in Figure 2a.
Revised: We start by measuring the distortion against remaining query budget of the adversary in Figure 2a.

Original: In general, the normal variants of each attack align with the published results.
Revised: In general, the normal variants of each attack align with the published results.

Original: The BiLN variants of RayS each have minimal effect on overall query efﬁciency.
Revised: The BiLN variants of RayS each have minimal effect on overall query efﬁciency (Insets 2a.i and 2a.iii).

Original:  The only notable improvement is with BiLN+HSJA against the Madry  model, with average distortion at 4k queries decreasing from 0.09 to 0.07, and corresponding success rate increasing from 15% to 25%.
Revised: This is a result of RayS not relying on explicit gradient estimation. The main improvement is with BiLN+HSJA against the Madry adversarial training model, with average distortion at 4k queries decreasing from 0.09 to 0.07, and corresponding success rate increasing from 15% to 25%.

Original: This improvement aligns with the result of Tu et al.
Revised: This improvement aligns with the result of Tu et al.

Original:  AE+Sign-OPT outperforms against regular Sign-OPT and BiLN+Sign-OPT on the robust model, since searching along the manifold can grant distortion which was not encountered during ﬁrst-order regularization (Stutz et al.,  2019).
Revised: (2019), and is contrary to the minimal effect on the natural model (Inset 2a.ii). AE+Sign-OPT outperforms against regular Sign-OPT and BiLN+SignOPT on the robust model, since searching along the manifold can grant distortion which was not encountered during ﬁrst-order regularization (Stutz et al., 2019; Jalal et al., 2019).

Original: However, the success with AE+Sign-OPT tends to be situational, whereas HSJA and RayS outperform in either scenario.
Revised: However, the success with AE+Sign-OPT tends to be situational, whereas HSJA and RayS outperform in either scenario.

Revised: Mainly the low quality of the adversary’s AE does not permit ﬁne-grained adjustment in latent space that RayS and HSJA can provide in image space.

Original: To understand why, we next focus on Figure 2b, which shows the FID score’s trajectory as the search progresses.
Revised: We next focus on Figure 2b, which shows the FID score’s trajectory as the search progresses.

Original: Every trajectory will begin at a zero value, since there is an expected score of zero for identical images, and then peaks as the attack initialization is performed.
Revised: Every trajectory will begin at a zero value, since there is an expected score of zero for identical images, and then peaks as the attack initialization is performed.

Original: In this scenario, our main observation is the similarity of FID trajectory between AE+Sign-OPT,
Revised: In this scenario, our main observation is the similarity of FID trajectory between RayS,

Original: RayS, and HSJA,  despite the method of model regularization. For example, the magnitudes for
Revised: HSJA, and dimension-reduced Sign-OPT, despite the method of model regularization, as shown in

Original:  AE+Sign-OPT and RayS peak to 0.28 and 0.04,respectively, then fall (and stay) at values near 0.004 (refer to inset plots for detail).
Revised: Insets 2b.i, 2b.ii, and 2b.iii, respectively. For example, the magnitudes for AE+Sign-OPT and RayS peak to 0.28 and 0.04, then fall (and stay) at values near 0.004.

Original: BiLN+HSJA and BiLN+Sign-OPT both exhibit lower FID scores than their normal variants, as much as two orders of magnitude less in the case of Sign-OPT and BiLN+Sign-OPT.
Revised: BiLN+HSJA and BiLN+Sign-OPT both exhibit lower FID scores than their normal variants, as much as two orders of magnitude less in the case of Sign-OPT and BiLN+Sign-OPT.

Original: For a detailed comparison of distortion and FID values for CIFAR-10, we offer Table 1 in Section A.3 of the Appendix. 
Revised: For a detailed comparison of distortion and FID values for CIFAR-10, we offer Table 1 in Section A.5 of the Appendix. The dimension-reduced variants of

Revised: RayS do not have a large variation between them (Inset 2b.iv), a side-effect of the adaptive super-pixel

Original:  At a high level, we observe that BiLN+HSJA offers comparable performance to regular RayS, while in the robust case, BiLN and AE variants tend towards lower FID scores than regular variants.
Revised: [Figure] search, which can automatically scale the super-pixel size as the search progresses. At a high level, we observe that BiLN+HSJA offers comparable performance to regular RayS, while in the robust case, BiLN and AE variants tend towards lower FID scores than regular variants.

Original: This can be viewed as the model leaking manifold information through the decision, as was shown for the ﬁrst-order gradient by Engstrom et al.
Revised: This can be viewed as the model leaking manifold information through the decision, as was shown for the ﬁrst-order gradient by Engstrom et al.

Original: ImageNet case study ( L 2 ).
Revised: ImageNet case study ( L 2 ).

Original: We attack ImageNet in the L 2 -norm setting so that we may compare against the certiﬁed smoothing technique proposed by Cohen et al.
Revised: We attack ImageNet in the L 2 -norm setting to compare against the certiﬁed smoothing technique proposed by Cohen et al.

Original: The label output comes[Figure] [Figure] from a smooth classiﬁer, approximated by many rounds of Monte Carlo search, which uses the regular model regularized by Gaussian noise.
Revised: The label output comes from a smooth classiﬁer, approximated by many rounds of Monte Carlo search, which uses the regular model regularized by Gaussian noise.

Original: Thus hard-label attacks are well positioned for attacking the smoothing technique.
Revised: Thus hard-label attacks are well positioned for attacking the smoothing technique.

Original: The  results of these attacks are shown in Figure 3a.
Revised: The distortion results of these attacks are shown in Figure 3a.

Original: Dimension reduction has a larger impact when coupled with the large ImageNet resolution.
Revised: Dimension reduction has a larger impact when coupled with the large ImageNet resolution.

Original: Particularly the BiLN+HSJA and BiLN+Sign-OPT attacks proﬁt the most.
Revised: Particularly the BiLN+HSJA and BiLN+Sign-OPT attacks proﬁt the most.

Original: At 8k queries, success rate increases 1.4x and 2.1x for HSJA and Sign-OPT, respectively.
Revised: At 8k queries, success rate increases 1.4x and 2.1x for HSJA and Sign-OPT, respectively.

Original: As before with CIFAR-10, the RayS dimension reduction is saturated on ImageNet.
Revised: As before with CIFAR-10, the RayS dimension reduction is saturated on ImageNet (Insets 3a.i and 3a.ii).

Revised: Since RayS does not explicitly rely on gradient estimation, it beneﬁts the least from dimension-reduction techniques.

Original: Apart from RayS, BiLN variants outperform both AE and regular variants in every scenario, despite the reduced search ﬁdelity.
Revised: Apart from RayS, BiLN variants outperform both AE and regular variants in every scenario, despite the reduced search ﬁdelity.

Revised: The improvement is largest on the smoothing technique, particularly with HSJA and Sign-OPT (second column of Figure 3b).

Original: This islikely due to 1) the ImageNet test set being insufﬁcient toapproximate the true submanifold, and 2) BiLN allowing to exploit generalization error in both regular and smoothed scenarios.
Revised: This is due to 1) the adversary’s reduced description of the manifold, by only having access to the test set, and 2) BiLN allowing to search closer to the original sample, since it is a deterministic function independent of the adversary’s knowledge.

Original: The FID scores in Figure 3b paint a more comprehensive picture.
Revised: The FID scores in Figure 3b paint a more comprehensive picture.

Original: BiLN variants produce adversarial examples closer to the manifold than either regular or AE variants.
Revised: BiLN variants produce adversarial examples closer to the manifold than either regular or AE variants, highlighted with HSJA+BiLN in Inset 3b.ii.

Original: In fact, this is the case for RayS  as well, despite displaying nearly identical performance in the context of distortion.
Revised: RayS saw no such improvement, as it does not perform an explicit gradient estimate (Inset 3b.i).

Original: We interpret this as follows:
Revised: We interpret this as follows:

Original: BiLN variants  leverage reduced ﬁdelity to increase the probability of ﬁnding the decision boundary, and 1) produce a smoother noise distribution, resulting in more spatially correlated distortion, which as a result 2) produces adversarial examples closer to the manifold. 
Revised: BiLN variants on HSJA and Sign-OPT leverage reduced ﬁdelity to increase the probability of ﬁnding the decision boundary, and 1) produce a smoother noise distribution, resulting in more spatially correlated distortion, which as a result 2) produces adversarial examples closer to the manifold. Since the manifold description of ImageNet is crude, BiLN variants can excel, since they search independent of this description.

Original: Another key observation is the ﬂuctuation of LID score towards the end of Sign-OPT and AE+Sign-OPT, which are not present for HSJA or RayS.
Revised: Another key observation is the ﬂuctuation of LID score towards the end of Sign-OPT and AE+Sign-OPT, which are not present for HSJA or RayS (ﬁrst column of Figure 3b).

Original: Notably there is no direct signal of manifold distance in any attack shown.
Revised: Notably there is no direct signal of manifold distance for the normal or BiLN variants.

Original: This indicates thatperhaps, query-efﬁcient attacks do a better job of capturing implicit manifold distance feedback from the model.
Revised: This indicates that query-efﬁcient attacks do a better job of capturing implicit manifold distance feedback from the model.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: The noisy manifold distance oracle.
Revised: The noisy manifold distance oracle.

Original: In Section 4 we observed that query-efﬁcient attacks, i.e., those which leverage the concept of super-pixels to reduce search ﬁdelity, are more likely to produce samples close to the manifold. 
Revised: In Section 4 we observed that query-efﬁcient attacks, i.e., those which leverage the concept of super-pixels to reduce search ﬁdelity, are more likely to produce samples close to the manifold. This generates samples that are unseen by robust models during their ﬁrst-order adversarial training.

Original: However, this takes place without any direct feedback about manifold distance.
Revised: However, this takes place without any direct feedback about manifold distance.

Original: To approach this, we ﬁrst consider that the model relies on a subsampled manifold of the image space.
Revised: To approach this, we ﬁrst consider that the model relies on a subsampled manifold of the image space.

Original: This manifold can be leaked by the loss landscape of the model, as shown by Engstrom et al.
Revised: This manifold can be leaked by the loss landscape of the model, as shown by Engstrom et al.

Original: From an information-theoretical perspective, the zeroth-order adversary observes the noisy gradient , which is leaked as side information by each model decision.
Revised: From an information-theoretical perspective, the zeroth-order adversary observes the noisy gradient , which is leaked as side information by each model decision.

Original: Under this explanation, the decision feedback by the model is viewed as a noisy manifold distance (NMD) oracle.
Revised: Under this explanation, the decision feedback by the model is viewed as a noisy manifold distance (NMD) oracle.

Original: The improvement of AE+Sign-OPT on robust CIFAR-10 can be argued as a result of the NMD oracle improving as well.
Revised: The improvement of AE+Sign-OPT on robust CIFAR-10 can be argued as a result of the NMD oracle improving as well.

Original: This can be shown using the data processing inequality (Beaudry & Renner, 2012): if MI ( M , g ) increases, then MI ( M , ¨ g ) also increases, where MI is mutual information, M is the manifold, and ¨ g is the noisy gradient.
Revised: This can be shown using the data processing inequality (Beaudry & Renner, 2012): if I ( M , g ) increases, then I ( M , ¨ g ) also increases, where I is mutual information, M is the manifold, and ¨ g is the noisy gradient.

Original: In words, the quality of the noisy gradient depends on the quality of the model’s loss landscape, which can more closely resemble the manifold under robust regularization.
Revised: In words, the quality of the noisy gradient depends on the quality of the model’s loss landscape, which can more closely resemble the manifold under robust regularization.

Original: This means a higher quality loss landscape leads to a higher quality zeroth-order attack.
Revised: This means a higher quality loss landscape leads to a higher quality zeroth-order attack.

Original: Qualitative evidence of this effect can be observed in Section A.2 of the Appendix.
Revised: Qualitative evidence of this effect can be observed in Section A.4 of the Appendix.

Original: “Topology” of hard-label settings.
Revised: “Topology” of hard-label settings.

Original: We can view zeroth-order attacks as following a topological hierarchy.
Revised: We can view zeroth-order attacks as following a topological hierarchy that is a function of the original data dimension.

Original: A very simple version of this idea is illustrated in Figure 1.
Revised: A very simple version of this idea is illustrated in Figure 1.

Original: Each technique illustrated in Figure 1 offers a different traversal distance both along the manifold, and away from it.
Revised: Each technique illustrated in Figure 1 offers a different traversal distance both along the manifold, and away from it.

Original: Efﬁcient attacks represented by (b) can combine elements of staying near the manifold, and traversing it.
Revised: Efﬁcient attacks represented by (b) can combine elements of staying near the manifold, and traversing it.

Original: This is best seen by the results of BiLN variants and regular RayS. 
Revised: This is best seen by the results of BiLN variants and regular RayS. RayS succeeds because it assumes spatial correlation, which can be considered a loose description of the manifold, without making further assumptions of the data. Thus we can expect this behavior from future attacks which do not perform explicit gradient estimation.

Original: Purely traversing close to an ideal manifold description, as in (c), may not be advantageous, because the crude manifold approximation induces its own error.
Revised: Similarly, purely traversing close to an ideal manifold description, as in (c), may not be advantageous, because a crude manifold approximation induces its own error.

Original: Further, the nearest boundary on the manifold could be far away.
Revised: Further, the nearest boundary on the manifold could be far away.

Original: Thus we can consider an attack which leverages an ideal “manifold description” (e.g., through an autoencoder), but leverages the description as a method for selecting progressively smaller super-pixel groupings.
Revised: Thus we can consider an attack which leverages an ideal “manifold description” (e.g., through an autoencoder), but leverages the description as a method for selecting progressively smaller super-pixel groupings.

Original: For instance, RayS selects super-pixels as blocks, which are bifurcated until the number of super-pixels equals the original dimension.
Revised: For instance, RayS selects super-pixels as blocks, which are bifurcated until the number of super-pixels equals the original dimension.

Revised: Notably, RayS displayed little to no beneﬁt from dimension-reduction, which is a byproduct of not relying on explicit gradient estimation.

Original: A technique which leverages the manifold description  could yield better performance, both in terms of distortion and FID score.
Revised: A technique which leverages the manifold description to select super-pixel groups could yield better performance, both in terms of distortion and FID score.

Original: To this end, we feel the FID score offers a valuable measure of manifold distance, which can inform the topological behavior, and the quality of future hard-label attacks.
Revised: To this end, we feel the FID score offers a valuable measure of manifold distance, which can inform the topological behavior, and the quality of future hard-label attacks.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Despite the recent progress in zeroth-order attack methods, open questions remain about their precise behavior.
Revised: Despite the recent progress in zeroth-order attack methods, open questions remain about their precise behavior.

Original: We shed light on an unexpected nuance, which is their ability to produce on-manifold adversarial examples.
Revised: We shed light on an unexpected nuance, which is their ability to produce on-manifold adversarial examples.

Original: This is despite the absence of manifold distance information, which motivates the proposal of the noisy manifold distance oracle.
Revised: This is despite the absence of manifold distance information, which motivates the proposal of the noisy manifold distance oracle.

Original: Future work could create a formal deﬁnition of this oracle, and attempt to bound the information revealed by the oracle.
Revised: Future work could create a formal deﬁnition of this oracle, and attempt to bound the information revealed by the oracle.

Original: On the other hand, with knowledge of the oracle’s existence, it is possible to further reﬁne hard-label attacks, so they continue to reveal insights into the weaknesses of learning systems. 
Revised: On the other hand, with knowledge of the oracle’s existence, it is possible to further reﬁne hard-label attacks, so they continue to reveal insights into the weaknesses of learning systems. A A PPENDIX A. MNIST ABLATION STUDY [Figure]

Original: A A PPENDIX A. I MPLEMENTATION DETAILS
Revised: Manifold projection was originally proposed by Samangouei et al. (2018b) as a scheme to defend models in the gradient-level setting, and was partially broken by Athalye et al. (2018) due to the imperfect projection of the defender decoder.

Revised: Jalal et al.

Revised: (2019) later showed it could be circumvented completely, by searching for latent sample pairs which are close to each other in latent space, but far apart on the model loss landscape (Jalal et al., 2019).

Revised: These sample pairs are projected to input space and used for adversarial training, which yields the Robust Manifold defense.

Revised: In direct comparisons under the ﬁrst-order setting, Jalal et al.

Revised: (2019) show that the Robust Manifold scheme defends better than previous baselines, Madry adversarial training (Madry et al., 2017) and TRADES (Zhang &

Revised: Wang, 2019), which do not leverage the manifold projection.

Revised: We conduct experiments to directly compare this behavior in the hard-label setting, shown in Figure 4.

Revised: The hard-label attacks consistently outperform against the Robust Manifold defense, despite this defense outperforming both the Madry and TRADES defenses under gradient-level attacks.

Revised: This can be considered a result of the Robust

Revised: Manifold defense overﬁtting to the manifold projection of the ﬁxed generator (or “spanner” in the vocabulary of Jalal et al.).

Revised: Further, dimension-reduced variants yield better exploration, shown by consistently better performance from 0-5000 queries.

Revised: This supports the results of the main paper, which show that these defenses can overﬁt to ﬁrst-order information.

Revised: The results on the MNIST ablation also support the notion of a dynamic topological hierarchy, which is dependent on the original data dimension d .

Revised: In the case of HSJA (second row of Figure 4), query-efﬁcient attacks do not beneﬁt the adversary, because MNIST is already sufﬁciently lowdimension to yield samples near the manifold with regular hard-label attacks.

Revised: We can validate this ﬁnding by comparing a notion of manifold distance between the attacks.

Revised: FID is not suitable for

Revised: MNIST as it requires re-training of the Inception-V3 network and calibration for use as a metric, which is outside the scope of this work.

Revised: Instead we project the adversarial samples back to the manifold, using the Robust Manifold defender’s variational autoencoder (VAE), and measure the pairwise distances between original and adversarial projections.

Revised: This distance is described formally as ||D ( E ( x 0 )) − D

Revised: ( E ( x ))

Revised: || 2 , for each adversarial sample x and original sample x 0 .

Revised: We use the checkpoint provided by Jalal et al.

Revised: Notably the defender network has the best description of

Revised: [Figure]

Revised: the manifold, since it has access to the full training set.

Revised: The distribution of pair-wise distances for each attack and defense combination is shown in Figure 5.

Revised: In these plots, we expect the reconstruction distance to be smaller for adversarial samples closer to the manifold, since the VAE will capture high-level semantic information of the dataset.

Revised: Under our hypothesis, successful hard-label attacks should have a smaller distance on natural models than regular white-box attacks.

Revised: Thus we also compare against samples generated with Projected

Revised: Gradient Descent (PGD) in red.

Revised: The results in Figure 5 support this hypothesis, as both Sign-OPT and HSJA have smaller distance than the PGD samples.

Revised: This is exaggerated with the AE+SignOPT variant, due to the manifold approximation of the adversary.

Revised: On robust models (second to last columns), PGD acts as a best-case for the adversary, since it is known from previous results by Engstrom et al.

Revised: (2019) that gradient-level attacks on robust models leak manifold information.

Revised: As discussed in the main text, hard-label attacks receive a noisy version of this manifold feedback, which is sufﬁcient to create samples near the manifold.

Revised: This is supported by the pairwise distances on robust models, since PGD samples are consistently closer to regular samples.

Revised: Due to the dynamics of the topological hierarchy discussed before, the dimension-reduction is less effective on MNIST.

Revised: Notably the regular hard-label attack variants (shaded blue) can be closer to the original samples than their dimension-reduced variants.

Revised: RayS (top row) is consistently farther from the PGD samples, as it does not explicitly rely on gradient estimation like Sign-OPT and HSJA.

Revised: We further point out that despite the variability of distortion in image space between regular and BiLN variants in Figure 4, these samples still lie near each other when projected to the manifold in Figure 5.

Revised: On the Robust Manifold defense, HSJA and Sign-OPT attacks, regardless of variant, will lie closer to the PGD samples.

Revised: This supports our main result, that query-efﬁcient hard-label attacks leverage manifold information to produce samples, particularly against robust models.

Revised: A. S UCCESS R ATE P LOTS

Revised: In Figure 6 we provide query vs. success rate to accompany the results in the main text.

Original:  We are primarily interested in the effect of reduced search resolution on attack behavior.
Revised: [Figure] A. I MPLEMENTATION DETAILS We are primarily interested in the effect of reduced search resolution on attack behavior.

Original: Thus in this work, given a candidate direction θ (cid:48) and magnitude (or radius) r , the adversarial sample in the AE case is the blending (1 − r ) x 0 + r D (cid:0) E ( x 0 ) + θ (cid:48) (cid:1) .
Revised: Thus in this work, given a candidate direction θ (cid:48) and magnitude (or radius) r , the adversarial sample in the AE case is the blending (1 − r ) x 0 + r D (cid:0) E ( x 0 ) + θ (cid:48) (cid:1) .

Original: Natural victim architectures consist of a deep convolutional neural network for CIFAR-10, and a
Revised: Natural victim architectures consist of a deep convolutional neural network for CIFAR-10, and a

Original: Resnet50 network for ImageNet.
Revised: Resnet50 network for ImageNet.

Original: The CIFAR-10 network is the same implementation open-sourced by Cheng et al.
Revised: The CIFAR-10 network is the same implementation open-sourced by Cheng et al.

Original: (2020), while the Resnet50 network is taken from the PyTorch Torchvision library, including pre-trained weights.
Revised: (2020), while the Resnet50 network is taken from the PyTorch Torchvision library, including pre-trained weights.

Original: 4 For AE attack variants, we implement the same architecture described by Tu et al.
Revised: 4 For AE attack variants, we implement the same architecture described by Tu et al.

Original: Speciﬁcally it leverages a fully convolutional network for the encoder and decoder.
Revised: Speciﬁcally it leverages a fully convolutional network for the encoder and decoder.

Original: ImageNet samples are downsized to 128x128 before passing to the encoder, and the output of the decoder is scaled back to 224x224, as described by Tu et al.
Revised: ImageNet samples are downsized to 128x128 before passing to the encoder, and the output of the decoder is scaled back to 224x224, as described by Tu et al.

Original: Every AE is trained using the held out test set, as we assume disjoint data between adversary and victim.
Revised: Every AE is trained using the held out test set, as we assume disjoint data between adversary and victim.

Original: To avoid ambiguity, we label each BiLN variant with the spatial dimension after performing the bilinear transformation, and keep this variable ﬁxed for simplicity.
Revised: To avoid ambiguity, we label each BiLN variant with the spatial dimension after performing the bilinear transformation, and keep this variable ﬁxed for simplicity.

Original: A. V ISUAL RESULTS - CIFAR 
Revised: A. V ISUAL RESULTS - CIFAR

Original: We provide visual qualitative results for each attack on CIFAR-10 in Figure 4.
Revised: We provide visual qualitative results for each attack on CIFAR-10 in Figure 7. [Figure]

Original: A. S UPPLEMENTAL R ESULTS A.3. E XTRA EXPERIMENTS ON ROBUST CIFAR- 
Revised: A. S UPPLEMENTAL R ESULTS A.5. E XTRA EXPERIMENTS ON ROBUST CIFAR-

Original: In Figures 5 and 6, we provide results on four extra model regularization methods from the literature:
Revised: In Figures 8 and [Figure] we provide results on four extra model regularization methods from the literature:

Original: TRADES (Zhang et al., 2019), adversarial interpolation learning (Zhang & Xu, 2020), feature scattering (Zhang & Wang, 2019), and SENSE (Jungeum & Wang, 2020).
Revised: TRADES (Zhang et al., 2019), adversarial interpolation learning (Zhang & Xu, 2020), feature scattering (Zhang & Wang, 2019) [Figure] and SENSE (Jungeum & Wang, 2020) [Figure]

Original: [Table]
Revised: [Figure] + Feat. Scattering [Figure]

Revised: + SENSE

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: RayS 4,

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: RayS
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: BiLN+RayS a= 4, 0.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: BiLN+RayS b= 4, 0.
Revised: [Table] [Table] [Table] [Table]

Original: BiLN+RayS b= 4, 0.
Revised: [Table] [Table] [Table] [Table]

Original: HSJA 4, 0. BiLN+HSJA4, 0. Sign-OPT 4, 0.
Revised: [Table] [Table]

Original: BiLN+Sign-OPT4, 0.
Revised: BiLN

Original: AE+Sign-OPT 4, 0.[Figure]
Revised: [Table] RayS b= 4, 0.

Original: HSJA on CIFAR [Figure] CIFAR [Figure]
Revised: HSJA 4, 0.

Original: [Figure] + Feat.
Revised: BiLN+HSJA4, 0.

Original: Scattering [Figure]
Revised: Sign-OPT 4, 0.

Original: + SENSE [Figure] [Figure]
Revised: BiLN+Sign-OPT4, 0. AE+Sign-OPT 4, 0.

Original: [Table]
Revised: [Table]

Original: SR
Revised: SR

Original: [Table]
Revised: [Table]

Original: SR
Revised: SR

Original: [Table]
Revised: [Table]

Original: RayS
Revised: RayS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: RayS
Revised: RayS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: BiLN+HSJA4, 2.
Revised: BiLN+HSJA4, 2.

Original: Sign-OPT 4, 28.
Revised: Sign-OPT 4, 28.

Original: BiLN+Sign-OPT4, 5.
Revised: BiLN+Sign-OPT4, 5.

Original: AE+Sign-OPT 4, 34.
Revised: AE+Sign-OPT 4, 34.