Less is More: Dimension Reduction Finds On-Manifold Adversarial Examples in Hard-Label Attacks
Abstract —Designing deep networks robust to adversarial examples remains an open problem.
Recently, it was shown that adversaries relying on only top-1 feedback (i.e., the hard-label) from an image classification model can arbitrarily shift an image towards an intended target prediction. Likewise, these hardlabel adversaries enjoy performance comparable to first-order adversaries relying on the full model gradient.
It was also shown in the gradient-level setting that regular adversarial examples leave the data manifold, while their on-manifold counterparts are in fact generalization errors.
In this paper, we argue that query efficiency in the hard-label setting is also connected to an adversary’s traversal through the data manifold.
To explain this behavior, we propose an information-theoretic argument based on a noisy manifold distance oracle , which leaks manifold information through the adversary’s distribution of gradient estimates.
Through numerical experiments of manifold-gradient mutual information, we show this behavior acts as a function of the effective problem dimensionality.
On high-dimensional realworld datasets and multiple hard-label attacks using dimension reduction, we observe the same behavior to produce samples closer to the data manifold.
This can result in up to 10x decrease in the manifold distance measure, regardless of the model robustness.
Our results suggest that our variant of hard- label attack can find a higher concentration of generalization errors than previous techniques, leading to improved worst-case analysis for model designers.
I. I NTRODUCTION
Adversarial examples against deep learning models have become a persistent topic of investigation in recent years, as they offer a principled approach to studying worst-case behavior in machine learning systems [1], [2].
Formal methods for discovering adversarial examples were originally conceived by assuming gradient-level (i.e., first-order) access to machine learning models and these became the first techniques to reach widespread attention within the deep learning community [3]–[7].
In order to compute the necessary gradient information, gradient-level techniques required access to the sensitive model parameters and a sizeable query budget.
These shortcomings were later addressed by the creation of scorelevel attacks, which only required the confidence values output by the deep learning models [8]–[11].
However, score-level attacks still rely on models to divulge information that would be impractical to receive in real-world systems.
By contrast, hard-label attacks make no assumptions about available model information, and instead rely only on the top-1 predicted class, thus providing the weakest, yet most realistic adversarial
[Figure] threat model to date. Hard-label attacks have traditionally been formulated by leveraging a framework known as zeroth-order optimization, which allows to estimate the input gradient of a sample with respect to the adversary’s desired label [12]– [14].
These methods have been carefully refined to offer convergence guarantees [15], query efficiency [16], [17], and powerful capabilities in the physical world [18].
Despite the steady improvements of hard-label attacks, open questions persist about their behavior and adversarial machine learning (AML) attacks at large, particularly in the task of image classification (the focus of this work).
Adversarial examples were originally assumed to lie in rare pockets of the input space [2], but this conventional wisdom was later challenged by the boundary tilting assumption [19], [21], which adopts a “data-geometric” view that observable inputs concentrate on a lower-dimensional connected region, called a manifold.
Through this lens, Stutz et al.
[20] suggest that adversarial examples can be categorized based on their distance to the manifold. Adversaries may add unnatural highfrequency distortion (e.g., noise), causing the image to leave the data manifold to cross the model’s decision boundary (blue orb in Figure 1). Another option is to only add natural distortions that alter the image without changing the true label (e.g., color shifting or distorting entire pixel groups). This latter option produces on-manifold adversarial examples, which are essentially generalization errors, i.e., examples not captured by the model’s approximation of the manifold (green orb in Figure 1).
One can also produce arbitrarily nearmanifold examples, which can be considered on-manifold for brevity (yellow orbs in Figure 1).
As a direct consequence, an adversarial example’s distance to the manifold describes the feasibility for such an example to generate naturally from the true data distribution.
This makes it advantageous to produce on-manifold adversarial examples, since the adversary can exploit the inherent generalization error of the model, while producing samples that are perceptually similar for humans.
Discovery of generalization errors are also motivated from the perspective of adversarial training, since it was observed that models adversarially trained with traditional off-manifold examples can be easily bypassed by perceptual distance attacks in the gradient-level setting [22].
Unfortunately, the true data manifold is either difficult or impossible to describe, and relying solely on approximations of the manifold can lead to the creation of crude, high distortion examples [20].
In this paper, we adopt the boundary-tilting assumption and demonstrate an unexpected benefit of dimension-reduced zeroth-order attacks.
These attacks are more likely to discover on-manifold examples, which we theoretically demonstrate is the result of manifold-gradient mutual information.
Our results suggest that this quantity can increase when data dimensionality is reduced, allowing an attacker to learn useful semantic variations of the data from the distribution of gradient estimates alone.
With this knowledge, we empirically demonstrate how to improve hard-label worst-case analyses in a generic yet principled way, by enabling the discovery of new generalization errors within “robust” models that may be useful for adversarial training in future work.
Due to insights from our experiments, we provide a geometric interpretation of hard-label attack behavior, summarized in Figure 1.
Our specific contributions are as follows, with key contributions and flow of arguments illustrated in Figure 2:
• Introduction of manifold distance oracle.
To create onmanifold examples, the adversary must leverage manifold information during the attack phase.
We thus propose an information-theoretic formulation of the noisy manifold distance (NMD) oracle, which can explain how zerothorder attacks craft on-manifold examples.
We theoretically demonstrate on a Gaussian data model that manifoldgradient mutual information can increase as a function of data dimensionality .
We empirically show this is true even on large-scale image datasets such as CIFAR-10 and ImageNet.
This finding relates to known behavior in the gradient-level setting, where useful variations of the data (e.g., shapes and textures) can be leaked from the model gradient [23].
• Reveal new insights of manifold feedback during queryefficient zeroth-order search.
In practice, the data manifold is difficult to characterize.
We propose the use of two proxies for manifold distance, which all show consistent results in terms of an adversary’s ability to search near the manifold.
This methodology allows us to empirically demonstrate the connection between dimension reduction, model robustness, and manifold feedback from the model, beyond the known convergence rates tied to dimensionality [24].
Our findings inform how to search closer to the manifold (Table I and Tables II-V), improve query efficiency (Figure 6), and reduce gradient deviation (Table VI) in a simple and generic way for hard-label attacks.
• Attack-agnostic method for semantic super-pixel construction.
We show that spatial dimension reduction of a decision-based gradient estimate acts as an attack- and knowledge-agnostic method for learning the most important super-pixels of an image.
More importantly, this allows an attacker to synthesize the semantic search directions for a sample using only the knowledge gathered during hardlabel boundary traversal.
Our attack formulation leads to a 2x success rate improvement for state-of-the-art hardlabel attacks such as HSJA [16], accompanied by up to 10x lower manifold distance, well below the manifold distance of previously-proposed RayS [25], the current state-of-theart for crafting on-manifold examples.
II. R ELATED W ORK
We investigate the scenario where an adversary uses toplabel feedback to estimate an input input gradient direction with respect to a desired label, showcased by attacks such as Sign-OPT [17] and HopSkipJumpAttack [16].
These contemporary attacks are variants of random gradient-free method (RGF) [24] and aim to approximate the true input gradients G with respect to the adversarial label through zeroth-order estimates ˜ G .
The core idea is to convert the top-1 (hard) label, which is a step function, into a continuous real-valued function g : R d → R , which takes search direction θ ∈ R d and outputs the distance to the nearest adversarial example [13].
The gradient estimates ˜ G are conceived as a function of the local search direction gradient ∇ g , and can be estimated with either two samples of information (Sign-OPT) [17], or a single point (HopSkipJumpAttack) [16].
Details of specific formulations for each attack are provided in Appendix B.
The theory of gradient estimation error and convergence provides some clues to improve efficiency, such as the fact that the estimation cost is polynomial in d , the dimension of the optimized variable, thus motivating standard dimensionreduction techniques based on autoencoding [26].
Alternatively, it is possible to change entire groups of pixels at a time, a technique known as super-pixel grouping, which is effective in attacks that leverage heuristic-based search rather than gradient estimation, such as RayS [25].
However, it is not completely understood how dimensionality-reduction relates to the adversary’s probing of the data manifold.
In the scorelevel setting, it was shown that better query efficiency can be achieved by leveraging time- and data-scale dependencies within the distribution of gradient estimates [27].
We examine analagous dependencies in service of generating on-manifold
[Figure]
[Figure]
[Equation] [Figure] [Figure] adversarial examples, since to date it is unclear how such dependencies interact with the model’s encoded manifold, particularly in the hard-label setting.
III. N OISY M ANIFOLD D ISTANCE O RACLE
Main idea:
Let F be a classifier that returns the top1 predicted label for some query, G the distribution of true input gradients with respect to an adversary’s desired label, and ˜ G the distribution of respective hard-label gradient estimates.
By the Data Processing Inequality (Definition III.1) we posit that their mutual information ( I ) has the relation
I ( F ; G ) ≥ I ( F ; ˜ G ) .
We use I ( F ; ˜ G ) as a measure for how much information F leaks in the gradient estimates ˜ G .
The main hypothesis is that as data dimension d increases, so does
I ( F ; G ) , but at a lower rate when scaling for d in the Schmidt et al.
[28] data model, giving higher I ( F ; G ) at lower d .
A. Preliminaries
A common observation is that due to the low probability of encountering “interesting” images in an entire input space (e.g., images of numbers or animals), the data of interest highly concentrates along a lower-dimension region of connected points, referred to as a manifold and denoted M [19], [29], [30].
The manifold is considered lower-dimension because there are generally only a few valid variations for interesting inputs, such as translating or rotating objects in images, and connected because traversal along the manifold yields new valid inputs (i.e., rotating the object in an image does not change the object’s true label).
Likewise, one can consider a subset of the manifold dimensions, which is analagous to considering a subset of useful variations.
It is also known that deep learning models encode information about the data manifold [31].
Thus we denote the model’s learned manifold as M ( F ) and use M for brevity.
B. Approach
Due to the fact that observable data generates from a random process, any function of the observed data (and corresponding manifold) is random, and thus the model point estimate is also a random variable.
As described in Section II, hardlabel attacks assume the existence of true input gradients G (a function of the model’s point estimate) which points in the direction of a desired adversarial label.
The task of hardlabel attacks is to synthesize an estimate of the true input gradients, ˜ G , using only the top-1 feedback from the model.
Since the victim model’s loss cannot be observed, hard-label attacks usually synthesize a continuous surrogate loss for estimation, e.g., distance to nearest adversarial sample, meaning the adversary relies on data-driven feedback.
We assume that a Markov chain maps the hard-label attack pipeline originating from the data manifold: M → G → ˜ G , where G ( ˜ G ) describes the true (estimated) input gradient distributions.
From the hardlabel attacker’s perspective, the Markov chain is only partially observable due to having access to top-1 feedback alone.
We posit that the adversary may be able to increase their available information by leveraging a basic result from data processing.
Definition III.1 (Data Processing Inequality (DPI) [32]) .
If three random variables form the
Markov chain
X → Y → Z , then their mutual information (I) has the relation I ( X ; Z ) ⩽ I ( X ; Y ) .
In the context of Definition III.1 and our hard-label attack pipeline ( M → G → ˜ G ), I ( M ; G ) acts as an upper bound on I ( M ; ˜ G ) .
In the information-theoretic sense, increasing the upper bound allows an optimal hard-label adversary to further maximize I ( M ; ˜ G ) , i.e., make the adversarial sample look as if it generated from the true data distribution, which is desireable for exploiting the generalization error of the model as discussed by [20].
Ideally, one would also like to show the lower bound of I ( M ; ˜ G ) will also increase with the upper bound if the adversary is stronger.
However, to the best of our knowledge such an ideal lower bound (e.g., some constant c < 1 such that c · I ( M ; G ) is a lower bound of I ( M ; ˜ G ) ) does not exist in general, unless one makes further assumptions (e.g., the true distribution is a known
Gaussian)
Leveraging DPI is a natural interpretation of attack behavior, since maximum likelihood estimation (MLE) can be interpreted as minimizing the dissimilarity between the empirical data distribution (defined by the data manifold) and the learned model distribution (defined by the point estimate).
In practice the dissimilarity is quantified by measures of KL-divergence (e.g., binary cross entropy loss between distributions of interest) [29], [33].
Although DPI only offers an upper bound, rather than a guaranteed increase, it allows us to operationalize the approach to hard-label attacks by attempting to increase I ( M ; G ) .
Since the states are only partially observable by the adversary, how could they increase the right-hand term I ( M ; G ) ?
Recent work by [28] prove that adversarial robustness requires a significantly larger number of data samples as a function of data dimensionality (e.g., √ d larger sample complexity for Gaussian data models), which can be argued as an effect of the curse of dimensionality.
In a similar twist, [21] showed that the volume of an error region along a subset of “adversarial directions” becomes exceedingly small with high data dimensionality.
Under the data model of [28], we expect the data categories to become more separated with higher dimensionality, effectively concentrating in smaller regions.
This leads us to our main hypothesis.
Hypothesis 1.
Assume the manifold-gradient Markov chain M → G → ˜ G exists. Then if we decrease the data dimension d by considering only a subset data dimensions, or equivalently, considering lower-dimension versions of the manifold distribution, we can yield higher mutual information I ( M ; G ) in the Schmidt et al.
[28] data model.
In order to empirically verify Hypothesis 1, we perform synthetic experiments quantifying I ( M ; G ) under different dimensionality values.
In particular, we leverage the Gaussian data model and results from [28] to derive an analytical solution for I ( M ; G ) .
Definition III.2 (Data model and optimal weights [28]) .
Let µ ∈ R
d be the per-class centers (means) and let σ > 0 be the variance parameter.
Then the ( µ , σI ) -Gaussian model is defined by the following distribution over ( x , y ) ∈ R d ×{± 1 } :
First, draw a label y ∈ {± 1 } uniformly at random.
Then sample the data point x ∈ R d from N ( y · µ , σI ) .
Definition III.3 (Optimal classification weight [28]) .
Fix σ ≤ c 1 d 14 for the universal constant c 1 , and samples ( x 1 , y 1 ) , · · · , ( x n , y n ) drawn i.i.d from the ( µ , σI ) -Gaussian model with || µ || = √ d (i.e., µ k = 1 for all dimensions k ∈ { 0 , . . .
, d } ).
Schmidt et al.
[28] prove the weight setting (cid:98) w = 1 n (cid:80) ni
=
y i x
i yields an l ϵ ∞ -robust classification error of at most 1% for the linear classifier f (cid:98) w : R d → {± 1 } instantiated as f (cid:98) w ( x ) =
sgn ( (cid:98) w T x ) if n ≥ (cid:40) 1 , for ϵ ≤ 14 d −c 2 ϵ 2 √ d, for 14 d − 14 ≤ ϵ ≤ 14 , (1) for a universal constant c 2 .
Note that the instantiation of (cid:98) w must change with choice of ϵ and d .
We can leverage the weight settings as a function of n and d to give a definition of manifold-gradient mutual information.
C. Manifold-Gradient Mutual Information
Notice the classifier sgn ( · ) in Definition III.3 is discontinuous at x k = 0 for any dimension k .
Instead we consider the sub-gradient of the classifier at x k < 0 and x k > 0 .
In either case (non-robust or robust), the input sub
gradient for f (cid:98) w ( x ′ k ) is defined dimension-wise for our isotropic Gaussian as ∇ x ′ k f (cid:99) w k = sgn ( w k ) .
Since the weight of each dimension is Gaussian distributed with (cid:99) w k ∼
N ( µ k , σ 2 ) , we can define the distribution of gradients as
G ∼ Rademacher ( P (cid:99) w k ∼N [ (cid:99) w k ≥ 0]) .
Using this fact, we define manifold-gradient mutual information in three parts: (1) defining the manifold-gradient point-wise joint probabilities between g k and x k at each dimension k for the sub-gradient cases where x k > 0 and x k < 0 , (2) defining the manifoldgradient marginal probability under the gradient, and (3) defining the marginal probability under the manifold.
The complete derivation of the joint and marginal probabilities can be found in Appendix A.
The three parts are used in the standard definition of mutual information [33].
Notation.
Fix σ
= c 1 d 14 for both cases.
We denote the submanifold sampled from the positive ( y = 1 ) and negative ( y = − 1 ) classes as M + and M − , respectively.
For brevity we label x k > 0 as x + and x k < 0 as x − .
Definition III.4 (Manifold-Gradient Mutual Information) .
We define the manifold-gradient mutual information, based on the standard definition of mutual information from information theory [33], as
[Equation]
with the total unnormalized mutual information defined as the summation over dimensions (due to dimension coindependence)
[Equation]
D. Mutual information as a function of dimensionality
To provide numerical support for Hypothesis 1, we run experiments using the Riemann approximation of Equation 2, provided in the Appendix as Equation 15.
We estimate the average per-dimension mutual information, I ( M ; G ) ϵ,k =
I ( M ; G ) ϵ d , for the case where x ∈ R d while varying the dimensionality term d ∈ [5 , 2000) with ϵ ∈ { 0 . 000 , 0 . 180 , 0 . 250 } .
We target an error within 10 − 1 (i.e., 0 .
9 ≤ p G (1)+ p G ( − 1) ≤ 1 . 0 ) by setting the multiplicative factor c 2 = 10 2 , and 101010d 0 .0 .0 .0 .0 . I ( M , G ) ε , k ( b it s ) multiplying each branch of Equation 1 by a large constant ( 10 4 ).
We run the approximation over ten different random seeds and show the average with standard error shaded.
The estimation result is shown in Figure 3 with log-scale x-axis.
Regardless of ϵ , lower values of the dimensionality evidence a higher mutual information.
Since the curve is normalized with respect to d , it shows that although mutual information scales with d , the rate of increase for mutual information is slower than the rate of increase for d .
Observation 1.
Given reduced data dimensionality, an adversary could increase I ( M ; G ) ϵ,k and lead to leaking better search direction through the gradient.
This supports Hypothesis 1.
An implication of Observation 1 is that better search directions come from a better encoding of the data manifold.
This same behavior was observed empirically by [23] and [30], who demonstrated that the input gradient from a “robustified” model leads to synthesizing images with higher visual alignment to the training data, which are essentially on-manifold synthetic images. In this capacity, it is useful to describe any sample’s distance to the true manifold w.l.o.g as manifold distance , often quantified by the L p -norm of the distance between a synthetic sample (e.g., machine-generated or adversarial sample) and the original when projected onto an approximate manifold representation [20], [31]. Given Observation 1 and the previous empirical results by Santurkar et al.
[30], we denote the true input gradient distribution G as a manifold distance oracle , because it offers useful variations of the input data for decreasing manifold distance.
In the hard-label setting, the data manifold, true gradient, and model parameters are not accessible, but they may be connected due to the Markov chain assumption of Hypothesis 1.
Thus we propose the noisy manifold distance (NMD) oracle, an abstraction of the information captured by the distribution of gradient estimates ˜ G . From the security perspective, the NMD oracle acts as a side channel leaking information (e.g., useful semantic directions) as a factor of the data dimensionality.
IV. Z EROTH - ORDER S EARCH T
THE M ANIFOLD D ISTANCE O RACLE
According to Observation 1, dimensionality reduction could increase the upper bound on I ( M , ˜ G ) .
Under our Markov chain assumption, this means the distribution of gradient estimates from an attack algorithm can act as a noisy manifold distance oracle.
However, Observation 1 mainly applies to a
Gaussian data model with a simple manifold, thus it does not necessarily lift into real-world datasets.
Likewise, to our best knowledge, previous work have yet to decompose the hard-label attack process into the Markov chain described in Section III.
Since Observation 1 relies on its existence, we proceed by first formulating dimension-reduced versions of existing hard-label attacks, then demonstrating through an empirical study that in practice, the Markov chain assumption is reasonable due to the learnability of semantic directions in the neighborhood around a hard-label sample.
We subsequently show that lower search dimensions lead to a higher quality model of the adversarial sample’s semantic directions, which in turn enables a lower manifold distance measured by standard metrics such as LPIPS and FID, introduced later.
A. Dimension-reduced zeroth-order search
We modify existing hard-label attacks to produce dimension-reduced variants.
In practice we modify attacks so they generate dimension-reduced gradient estimates to update their candidate search direction θ ′ ∈ R d ′ for reduced image dimension d ′ , where d ′ <
d and d is the original data dimension.
To update an adversarial example and query the victim model, the candidate direction is upsampled using a decoding map D : R d ′ → R d .
In general the adversarial sample is created by x = x 0 + g (cid:0) D ( θ ′ ) (cid:1) D ( θ ′ ) ||D ( θ ′ )
|| , where g : R d → R is a regularizing function dependent on the attack formulation.
For Sign-OPT attack, g is the distance to the decision boundary in direction D ( θ ′ ) , while for HSJA, g represents the optimal step-size in direction D ( θ ′ ) .
For our purposes, the decoder function D is initialized with a bilinear upsampler (henceforth referred to as BiLN followed by d ′ ) which simply scales the height and width dimensions of candidate directions (shaded middle section of Figure 2).
It is possible to instantiate more complex mapping functions, for example the use of autoencoders [20], [26].
We forego analysis of autoencoder variants, since hard-label adversaries may not have access to the full training or test set necessary to create such an approximation in the first place.
B. Attack-agnostic model of the manifold-gradient Markov chain
Due to the hard-label threat model, an adversary does not have access to the true manifold distribution (e.g., training samples)
M or the distribution of true input gradients G .
The distribution of gradient estimates ˜ G is updated over the course of a hard-label attack, and in the score-level setting, is known to possess useful time-scale dependencies [27]. Unfortunately, current hard-label attacks immediately discard the distribution after each attack iteration [16], [17].
Due to the complexity of real-world data, even if each of the global distributions was known, it would be intractable for designers, much less adversaries, to analytically model every possible relationship between manifold points and gradient estimates.
Instead, we can study the local Markov chain at a single point on the manifold (e.g., starting from a singular clean image). If an adversary can learn the semantic features of an image using only previous adversarial attempts, it may validate the Markov chain assumption.
It was previously shown that a designer can find a sample’s semantic directions using only a linear model of their classifier’s local decision boundary, e.g., the LIME technique proposed by Ribeiro et al. The linear model (standard
Ridge regression) is trained using uniform-random perturbations around a sample.
The linear model coefficients are interpreted as per-pixel feature importance, the quality of which is measured by the linear model’s R 2 score.
Rather than use uniform-randomly perturbed samples, we propose to leverage samples collected during hard-label attack initialization and gradient estimation to train the LIME model.
If the hard-label adversary’s LIME model yields a high quality (i.e., high R 2 ) set of coefficients, it shows that the local manifold-gradient Markov chain can be directly modeled over the course of an attack.
In fact, the R 2 score is a weighted estimate based on an exponential kernel, which in our implementation is chosen to correspond with a local measure of mutual information (i.e,.
binary cross entropy between original sample and hardlabel attempts).
We can later use the R 2 score to measure the quality of the linear model after applying different dimensionreduction schemes, providing additional empirical validation for Observation 1.
In practice, LIME was shown to have unstable behavior due to the difficulty in selecting a kernel width for the weighted estimate of coefficients [35].
Instead we leverage
OptiLIME, which factorizes individual training runs of LIME as Gaussian processes, and maximizes R 2 by trying different kernel width selections at each run through Bayesian optimization [35].
Similarly, we decompose the typical hard-label attack into independent Gaussian processes corresponding to common parameter-independent attack stages, e.g., sample initialization, gradient approximation, and binary search for nearest boundary [13], [16], [17].
A stage can be called
Input: Benign sample x 0 , Hard-label attack initialization ( init ) and gradient approximation ( approximate_gradient ),
OptiLIME bayesian optimization routine for kernel width ( OptiLIME ), num. iterations n
Output: Hard-label attack sample x , whole-attack feature coefficients { W init , W ˆ g 1 , . .
, W ˆ g n } and their quality scores { R 2 init , R 2ˆ g 1 , . . .
, R 2ˆ g n } 1 GP :=
, x 0 ) 2 kernel width k ← OptiLIME ( MC_step , GP ) 3 /* Initialize through MC_step */ 4 W init , R 2 , x ← MC_step ( GP , k ) 5 for i := 1 to n do Hard-label attack loop
GP ← ( approximate_gradient , x ) k ← OptiLIME ( MC_step , GP , x ′ ) /
* Approximate through MC_step * /
W ˆ g i , R 2 , θ ← MC_step ( GP , k )
Update x θ using attack formulation 11 end 12 return x , { W init , W ˆ g 1 , . .
, W ˆ g n } , { R 2 init , R 2ˆ g 1 , . .
, R 2ˆ g n } multiple times to collect independent sets of samples, allowing to train OptiLIME’s Bayesian classifier and obtain optimal coefficients for that stage. We implement this logic for wholeattack Markov chain (MC) analysis across two algorithms, illustrated visually in Figure 2.
The first algorithm, described in Algorithm 1 ( MC_step ), takes as input an attack stage Gaussian process (denoted GP ) and acts as the vehicle for collecting a neighborhood of hard-label samples around the attacker’s current sample.
LIME is used in this algorithm alongside a candidate kernel width k to train a linear model f , parameterized by model coefficients W ∈ R d , which outputs an R 2 score that acts as an objective function of the candidate kernel width.
Thus MC_step is used as the input to OptiLIME to conduct independent trials for kernel width optimization at each attack stage. At each attack stage, MC_step is instantiated with the stage’s Gaussian process and given to OptiLIME in Algorithm 2 (Lines 2 and 9), which handles the collection of feature coefficients and linear model scores at different attack stages for a generic hard-label attack.
For simplicity, we only consider attack initialization (Line 1-3) and subsequent gradient estimation attempts (Lines 7-9).
Although the OptiLIME Bayesian optimization requires several trials to choose an optimal kernel width, in practice OptiLIME can obtain reasonable R 2 scores from only twenty LIME training runs (each requiring 10-200 hard-label queries, depending on the attack and particular stage).
The final output of Algorithmis the feature coefficients W and corresponding R 2 score at each attack stage from initialization ( Init ) through to the
Input
BiLN+HSJA
R 2 = 0.
Init.
R 2 = 1.
ˆ g
R 2 = 1.
ˆ g
R 2 = 0.
ˆ g
R 2 = 0.
ˆ g
Input
BiLN+HSJA
R 2 = 0.
Init.
R 2 = 1.
ˆ g
R 2 = 0.
ˆ g
R 2 = 0.
ˆ g
R 2 = 0.
ˆ g
Input
BiLN+HSJA
R 2 = 0.
Init.
R 2 = 1.
ˆ g
R 2 = 0.
ˆ g
R 2 = 0.
ˆ g
R 2 = 0.
ˆ g
Input
HSJA
R 2 = 0.
Init.
R 2 = 0.
ˆ g
R 2 = 0.
ˆ g
R 2 = 0.
ˆ g
R 2 = 0.
ˆ g i -th gradient estimation step ( ˆ g i ). We emphasize that Algorithm 2 does not interfere with the initialization or gradient approximation of the attack, and merely serves to model the manifold-gradient Markov-chain in generic hard-label attacks.
An example of this process is illustrated in Figure 4 on a single sample from CIFAR-10.
Each column after the input represents the activated feature coefficients at the specified attack stage, visualized by white pixels (positive coefficients) and black pixels (negative coefficients).
Notably, the hardlabel adversary’s initialization ( Init. ) is enough to learn a coarse spatial representation of the subject in the image, seen as the rough outlines of the cruise ship in the second column.
Counter-intuitively, the subject can be isolated with better detail using dimension-reduced variants of HSJA (e.g.,
BiLN 4 in row one and BiLN 8 in row two)
compared to regular HSJA in the final row.
The increased semantic quality of the dimension-reduced variants is reflected in the higher
R 2 score of the learned feature coefficients (e.g., R 2 = 0 .of
BiLN+HSJA 8 compared to R 2 = 0 .
24 of regular HSJA).
Likewise, we see that the feature importance of subsequent gradient estimation steps is scattered throughout the spatial dimension of the image, although larger patches of the image are affected in lower-dimension variants, which retain a higher
R 2 score for more iterations.
To gain a holistic view of the phenomenon observed in
Figure 4, we attack 500 samples on a standard CIFARmodel with variants of HSJA in the l ∞ setting and obtain the mean R 2 score at each stage using Algorithm 2, shown in [Figure] [Figure]
Figure 5.
As before, we observe that on average, each HSJA variant can learn a reasonable representation of the semantic features in the image, evidenced by high R 2 scores up to the second gradient estimation stage, thereafter decreasing quickly depending on the use of dimension reduction.
Surprisingly, dimension-reduced variants can retain high semantic quality of the learned feature coefficients for more iterations, as evidenced by the ability for BiLN+HSJA 4 to retain an R 2 ≥ 0 .on average through every stage.
C. Estimating manifold distance
Given the ability for the hard-label adversary to model coarse semantic features of the input sample, this may indicate that the manifold-gradient Markov chain is a reasonable assumption.
However, the result in Observation 1 implies that the dimension-reduced attack variants should lead to adversarial samples lying closer to the original concentration of training points (i.e., closer to the manifold).
To connect Observationto the findings in Figures 4 and 5, we leverage an estimate of manifold distance to determine if higher R 2 score does in fact lead to samples lying closer to the data manifold.
The Learned
Perceptual Image Patch Similarity (LPIPS) acts as such a measure for manifold distance, since it was previously shown to correlate with human visual perception [22], [31].
We use the same LPIPS code and checkpoint provided by the authors.
In addition to LPIPS, we rely on the popular Fr´echet Inception
Distance (FID) to measure the semantic drift between sets of original samples and their adversarial counterparts [36].
Although FID lacks human studies, Heusel et al.
showed that the internal representations of Inception network capture the visual semantic quality of synthetically generated images [36].
Our attack variants do not target the Inception network directly, meaning the results should not be affected by the addition of low-magnitude adversarial noise.
When measuring either LPIPS or FID, we use the first successful adversarial sample which was below the noise threshold ϵ = 0 .
031 studied by previous hard-label attacks in the l ∞ setting [16], [25].
In this way, all comparison samples for the purposes of measuring [Table] FID and LPIPS are “clipped” at the same distortion threshold, so as to not bias the computation of either score.
We measure FID and the average LPIPS of the samesamples generated during the Markov-chain attack probe described in Algorithm 2.
As observed in Table I, the dimensionreduced variants create samples with consistently lower FID, which is strongly negatively correlated with the average wholeattack R 2 score from these same variants (denoted ¯ R 2 ).
Shown in bold, the BiLN 4 variant of HSJA achieves over ten times lower FID score on an adversarially trained model [37] (denoted Madry CIFAR-10) compared to the baseline attack.
The same trend holds on the natural model without adversarial training, where the FID is instead twenty times lower.
The trade-off in the natural case is the success rate; as is observed in the fourth column, success rate after forty thousand queries ( SR@40k ) is consistently lower as the dimension is reduced, but the opposite is true for the adversarially trained model.
Instead, we see that the BiLN 16 variant (bold) outperforms the standard HSJA attack on adversarially trained models.
From the perspective of human alignment, we see that the average
LPIPS score for each attack variant is within the standard deviation of the baseline attack’s LPIPS score, meaning these variants are likely not discernible by the human eye.
We summarize our findings so far.
Observation 2.
Motivated by Hypothesis 1 and Observation 1, we proposed a probing algorithm to investigate if the manifoldgradient Markov chain assumption is reasonable on real-world datasets.
Based on the holistic comparison in Figure 5 and
Table I, we show it is possible to model the Markov chain (i.e., the noisy manifold distance oracle) from boundary traversals alone, leading to discovery of the semantic directions which correspond to manifold traversal, as illustrated by the feature importance masks of Figure 4.
The lower FID in Table I for dimension-reduced variants offer further empirical support for
Hypothesis 1.
V. I NFORMING P RACTICE
Our findings so far offer empirical support for Hypothesis 1 through the existence of a usable manifold-gradient
Markov chain, which could serve as a passive noisy manifold distance (NMD) oracle in dimension-reduced attacks.
In this sense, we are left with key research questions concerning the practical application of the increased manifold-gradient mutual information that the NMD oracle provides, beyond the contemporary interest of query efficiency.
We outline our key research questions as follows.
Q1.
Although dimension-reduction offers lower manifold distance, it may not be clear how many dimensions to choose from the outset of an attack.
From the active learning sense, how can Algorithm 2 be modified so that an adversary actively maximizes the manifold-gradient mutual information (by way of automatically maximizing their whole-attack R 2 score)?
Q2.
Does the phenomenon of manifold-gradient mutual information hold across different attacks, datasets, and levels of model robustness?
Q3.
Is there a connection between gradient estimation attacks (Sign-OPT [17], HSJA [16]) and combinatorial search attacks (RayS [25]) in terms of the hard-label adversary’s manifold traversal?
We subsequently answer each research question through our experimental design, enabling the following experimental highlights.
A1.
We propose two variants of existing gradient estimationbased hard-label attacks by modifying our Markov chian probe described in Algorithm 2: (1) Markov chain (MC) variant and (2) Dynamic R 2 -based upsampling attack variant (denoted DynBiLN), based on the MC variant, which can automatically upsample the attack search dimension based on the quality of the learned feature importance W (described fully in Algorithm 3).
A2.
By conducting experiments over three state-of-the-art attacks, two large-scale image datasets, and both natural and adversarially trained models, we show a consistent trend for dimension-reduced attacks to achieve lower
FID, while remaining visually imperceptible to humans according to LPIPS standard deviation.
A3.
Through dimension-reduction techniques and attack variants proposed in A1, we show it is possible for theoretically-grounded gradient estimation attacks such as Sign-OPT and HSJA to achieve up to 39x lower FID than the state-of-the-art combinatorial search-based RayS attack.
Experiment setup.
To provide widespread empirical evidence for the NMD oracle and address the research questions, we compare between three state-of-the-art attacks which adopt unique approaches to the hard-label attack problem.
1) Sign-OPT [17] is a variant of random gradient-free (RGF) method [24] and enjoys both query efficiency (due to its sample update using the sign of the gradient estimate) and convergence guarantees grounded in the theory of zeroth-order optimization [38].
2) HSJA [16] is another variant of RGF, which like SignOPT, only uses the sign of the gradient estimate to update the attack sample.
HSJA enjoys a bounded estimation error and design optimized for l ∞ setting, which enables better query efficiency than Sign-OPT, but lacks the respective convergence guarantees.
3) RayS [25] does not employ gradient estimation, and instead performs a combinatorial ray search in image space using progressively larger search dimensions.
RayS can create semantically similar adversarial samples in the least amount of queries, but lacks the theoretical grounding of Sign-OPT and HSJA.
We perform experiments with the above attacks (and our variants) using CIFAR-10 [39] and ImageNet [40] as input image data.
The natural CIFAR-10 network is the same implementation open-sourced by Cheng et al.
The architecture (and accompanying pre-trained weights) for natural ImageNet are taken from the ResNet-50 network implementation in the PyTorch Torchvision library.
In addition, we leverage the representative adversarial training technique proposed by
Madry et al.
[37] (and their ϵ = 8255 =
031 checkpoints for l ∞ setting) as the robust models for CIFAR-10 and ImageNet.
In all experiments, our BiLN variants rescale the attack search directions to the dimension shown next to their denomination (e.g., BiLN 4 scales attack spatial dimensions for θ ′ to 4 × 4 ).
We use l ∞ -norm versions of attacks for all experiments, and the same ϵ = 0 .
031 distortion threshold to measure success for both natural and robust models (hereafter referred to as Madry
CIFAR-10 and Madry ImageNet).
All attacks run for 25k queries without early stopping on correctly classified samples.
For brevity, we only show results for the untargeted case.
Additional implementation details, such as hyperparameters and hardware used, can be found in Appendix C. Code for experiments will be released publicly to encourage reproducibility.
A. Actively exploiting the NMD oracle (Q1)
Motivated by the results in Section IV, we propose an attack algorithm which can leverage the learned feature importance coefficients W from the LIME Ridge regression model to inform the initial attack search directions.
This leads to our
Markov chain (MC) variant, which uses the learned W from trials of attack initialization to mask the search directions for a “final” attack initialization step (i.e., only perform Line 7 in
Algorithm 3).
This MC variant acts as a baseline for the case where W informs only the attack sample initialization.
Further motivated by the negative correlation between FID and Rscore in Table I, we propose a dynamic attack variant which in addition to using W for attack initialization, also maximizes the average whole-attack R 2 score.
Our proposed dynamic
R 2 -based upsampling attack variant, denoted DynBiLN and described in Algorithm 3, automatically upsamples the attack search dimension d ′ based on the quality of the learned feature importance coefficients.
This is achieved by recording the best R 2 score at the current reduced attack dimension d ′ (starting on Line 5, denoted R 2 d ′ ) and updating each time the upsampling occurs (Line 13).
Upsampling is performed when the current linear model R 2 score drops below a small
Input: Benign sample x 0 , original attack dimension d , intial reduced attack dimension d ′ << d ,
DynBiLN differential coefficient η
Output: Hard-label attack sample x 1 GP :=
( init , x 0 ) 2 kernel width k ← OptiLIME ( MC_step , GP ) 3 W, R 2 , x ← MC_step ( GP , k ) 4 /* Set best score at current d ′ */ 5 R 2 d ′ ← R6 /* Use W to mask initialization */ 7 x ← init ( x 0 , W ) 8 for i := 1 to n do Hard-label attack loop
GP ← ( approximate_gradient , x ) /* Re-use initial kernel width */
W, R 2 , θ ′ ← MC_step ( GP , k ) if R 2 d ′ − R 2 > η R 2 d ′ then d ′ ← min (2 d ′ , d )
R 2 d ′ ← R end
Update x from θ ′ using attack formulation 17 end 18 return x differential factor η of R 2 d ′ (Lines 12-14). In this way, we can gain the query efficiency of low search dimension, without incurring the damaging estimation bias during later stages (which lead to low SR AUC in Table I). To avoid the query overhead of Bayesian optimization for kernel width at every attack iteration, the adversary re-uses the kernel width obtained from initialization (Line 2), as in practice, this choice offers reasonable linear model quality for subsequent iterations.
Since DynBiLN does not make assumptions about the underlying intitialization or gradient approximation routines, it can be adapted to any gradient estimate-based hard-label attack that uses sample initialization [13], [16], [17].
Although
DynBiLN depends on the differential parameter η , we found through grid search experiments any choice of η ∈ [0 . 2 , 0 . 5] provides adequate scaling performance for both CIFAR-10 and
ImageNet.
For the sake of comparison, we choose η = 0 .
3 for
CIFAR-10 and η = 0 .
5 for ImageNet.
The full grid search for η on CIFAR-10 is made available in Appendix D.
We demonstrate the effectiveness of our MC and DynBiLN variants by implementing MC HSJA, MC Sign-OPT, and
DynBiLN HSJA variants to compare against baseline versions.
The average success rate across 200 samples is plotted against 25k queries in Figure 6 (and the corresponding distortion plots with error regions are made available in Appendix D).
The
MC HSJA (dashed blue lines) and MC Sign-OPT (dashed green lines) are able to closely match the SR curve of their baseline versions, but can subsequently lower the FID as shown in MC HSJA and MC Sign-OPT rows of Tables II-V.
The same is true for DynBiLN HSJA (cyan lines), which can [Figure] [Table] automatically strike a balance between the query efficiency of down-sampling attacks (orange lines) and simplicity of the baseline HSJA attack (blue lines), meanwhile achieving lower FID compared to HSJA.
In fact, on large scale data such as ImageNet (Figure 6b)
[Table]
DynBiLN enables a gradient estimate-based attack, such as HSJA, to gain enough query efficiency to be competitive against RayS (red lines), the current state-of-the-art which does not use gradient estimation (and likewise cannot enjoy the theoretical guarantees from zeroth-order optimization).
B. NMD oracle across attacks, datasets, and robustness (Q2)
We summarize results on key attack implementations in Figure [Table] and provide quantitative analysis in Tables II-V through
FID, LPIPS, and normalized area-under-curve (AUC) of the average success rate curves for all attacks across 25k queries [Table] and 200 samples.
On natural models (Tables II and III)
[Table]
we observe the consistent trend for BiLN HSJA variants to reduce the FID compared to baseline (downward green arrows).
This trend is only present for Sign-OPT on large-scale ImageNet data, although in both cases of CIFAR-10 and ImageNet,
BiLN Sign-OPT variants enjoy the lowest FID score across all attacks (FID of 0.001 for Sign-OPT BiLN 16 and MC
Sign-OPT BiLN 8, respectively).
Although RayS retains the best query efficiency (SR AUC of 1 . 0 ), BiLN variants have an inverse effect on the query efficiency depending on original data dimension.
On natural CIFAR-10 (Table II), SR AUC is lowered alongside BiLN dimension, whereas for natural
ImageNet (Table III), SR AUC is consistently higher for lower
BiLN dimension on HSJA (upward green arrows), and similar or higher on Sign-OPT.
In both cases, LPIPS of attack variants [Table]
[Table]
remains within the margin of error for their baseline versions.
To summarize, the effectiveness of dimension reduction may depend on the attack difficulty.
We can track this idea by investigating the HSJA attack performance on adversariallytrained models (Tables IV and V).
We omit Sign-OPT due to achieving insufficient samples within the adversarial radius ϵ = 0 .
031 for FID and LPIPS calculation.
As in the case of Natural ImageNet, regardless of dataset or baseline attack choice, dimension-reduced variants showcase the consistent trend of lower FID (downward green arrows) and increased
SR AUC (upward green arrows).
We can likewise observe that the DynBiLN HSJA variant from Q1 offers a balance between low FID and higher SR AUC, evidenced by the performance on Madry ImageNet (doubling SR AUC and halving FID compared to HSJA in Table V).
C. Connection to combinatorial search attacks (Q3)
Dimension-reduced attacks have so far exhibited the ability to achieve lower FID compared to their baseline versions, in some cases reducing FID by 10x on HSJA (e.g., MC
HSJA+BiLN 4 in Table V) and 87x on Sign-OPT (MC SignOPT+BiLN 8 in Table III).
However, there is a persistent need for query efficiency in the design of hard-label attacks.
As evidenced in Tables II-V, RayS achieves the best query efficiency, but is often out-performed on FID compared to dimension-reduced variants of gradient estimation attacks (and in some cases, regular Sign-OPT as well).
In this sense, our results show that the ability to generate adversarial samples closer to the original model observations (i.e., the data manifold) stem from the ability to abstract away unnecessary dimensions in the attack search space, rather than the specific attack formulation or design (e.g., using gradient estimation or combinatorial search techniques).
RayS upsamples the attack search space dynamically, thereby eliminating unnecessary search directions early on, similar to our DynBiLN variant, but certain BiLN variants show there is untapped potential in simply searching a single attack dimensionality over the entire attack.
In the case of Madry CIFAR-10 (Table IV), the MC HSJA+BiLN 4 variant achieves less than half the
FID that was possible using RayS. Dynamic subsampling is not unwarranted, since in comparison, DynBiLN on the same model achieves a similar FID ( 0 . 030 ) with double the
SR AUC ( 0 . 607 ).
Our empirical results highlight a limitation of Observation 1; although higher manifold-gradient mutual information may lower the manifold distance, the search directions near the manifold yield a lower concentration of adversarial samples on average.
This presents an interesting trade-off which builds on previous work investigating the data geometric properties of deep learning models.
Stutz et al.
[20] used autoencoders to synthesize on-manifold examples which are far from a sample’s nearest decision boundary (giving the lowest success rate), while Chen et al.
[25] showcased the ability for RayS to find the nearest decision boundary (giving the highest success rate).
In this sense, we find that on-manifold generalization errors are more common than originally captured by Stutz et al.
[20]; some can be found near the decision boundary, showcased by the low FID of RayS, but they also become more common as we drift away from the boundary, as we obtain the lowest FID using the lowestdimension BiLN attacks, at the cost of lower SR AUC.
VI. D ISCUSSION
Through our theoretical results in Section III, we evidenced the ability for manifold-gradient mutual information to increase with lower data dimension.
This motivated empirical experiments on real-world data to support our Markov chain assumption.
By borrowing techniques from the interpretable
ML literature, we showed the gradient estimate distribution can be leveraged to model the Markov chain, by way of learning the feature importance coefficients in image space that correspond to manifold traversal.
By reducing the attack dimensionality, we showed that an attacker increases the quality of these coefficients (represented by R 2 score), and counter-intuitively, better inform the possible search directions during the course of an attack.
As a result, we propose a novel attack-agnostic variant (Algorithm 3) which can initialize attack samples in the most semantically plausible direction.
By tracking the decrease of R 2 during each attack iteration, an attacker can also automatically increase the attack search space dimensionality to ensure better search resolution in later attack stages.
In the end, the reduced-dimension attacker can achieve lower FID than was previously possible with RayS, and an attack variant based on our proposed DynBiLN can automatically select the ideal search dimension with minimal tuning.
Geometric interpretation of hard-label attacks.
Based on our results, we can view zeroth-order attacks as following a geometric hierarchy that reveals the inherent concentration of adversarial samples.
Our interpretation is illustrated in
Figure 1.
Each technique offers a traversal direction which is either away from the manifold (towards x a ), arbitrarily near the manifold ( x b ), or along the manifold ( x c ).
Efficient attacks create samples such as x b , by finding the direction of the nearest decision boundary which also points to the data manifold.
This is representative of RayS, which can find adversarial samples closer to the manifold through elimination of off-manifold directions.
The smaller points below x b are representative of BiLN and DynBiLN variants, which can get closer to the manifold, despite this region having a lower concentration of adversarial samples.
In contrast, traversing close to an approximate manifold description leads to x c , which may not find adversarial samples inside the ϵ radius.
As shown by Stutz et al.
[20], strict manifold traversal leads to a lower concentration of adversarial samples and high average distortion.
To this end, the R 2 score of local semantic features can inform the geometric behavior of hard-label attacks which resemble the smaller yellow points near x b , rather than x a or x c .
This ultimately enables a better evaluation of model robustness in future work, as we are able to find errors which are closer to the manifold.
Implications for robust models.
As discussed in Section V-C, our results highlight specific nuances for designing dimension-reduced hard-label attacks against robust models.
When measuring FID as a proxy of manifold distance in Tables II-V, we observed that dimensionality-reduction favored “harder” learning problems such as Madry CIFAR-10 and
ImageNet.
We know from empirical results by Santurkar et al.
[30] that gradients of adversarially trained models contain better semantic alignment with the original image.
From the gradient estimation perspective, the gradient estimates could leverage the upper-bounded noisy mutual information during attacks (Hypothesis 1).
This would manifest in a lower gradient deviation, or in other words, a lower distance between the true gradient from a robust model and the gradient estimate at the first attack step.
Since adversarial training effectively smooths the sampled data manifold by augmenting perturbed data samples during training, robust models have a welldefined boundary that aligns with salient input changes [30].
We posit that adversarial trained models lower the variance of the gradient estimate compared to natural models.
We test this idea by calculating per-pixel gradient deviation ||
g − ˆ g
|| 2 H × W for true gradient g (in the direction of the adversarial label) from an adversarially trained model, first gradient estimate
ˆ g , estimate height H
[Table]
and estimate width W . When taking the true input gradient in the direction of the adversarial label, we use the robust model’s original criterion to calculate the gradient, which was cross-entropy for all models in
[Table]
our evaluation. The results of this experiment are shown in Table VI, which exhibits a consistent trend for dimensionreduced attacks to reduce the gradient deviation, particularly for HSJA (which is known to have a higher gradient estimation error due to relying on a single point estimate [14]).
In this sense, dimension-reduced attacks can turn the strength of adversarial training into a weakness, since the robust model now leaks more semantically meaningful decision boundaries.
This reveals an interesting direction for future work, as it can lead to creation of hard-label attacks specifically formulated for adversarial training schemes, which can heal the inherent generalization errors due to adversarial training on narrow threat models [22].
On supplemental experiments in the Appendix, we show that our hard-label attacks are in fact capable of lower manifold distance than strong white-box attacks such as AutoAttack [41].
Although it remains outside the scope of this work, incorporating the hard-label threat model into adversarial training provides a ripe direction for future work.
VII. C ONCLUSION
Despite the recent progress in zeroth-order attack methods, open questions remain about their precise behavior.
We develop an information-theoretic analysis that sheds light on their ability to produce on-manifold adversarial examples.
Through experiments on real-world datasets, we show an over two-fold increase in attack success rates by leveraging new findings about manifold distance and gradient deviation.
With knowledge of the manifold-gradient relationship, it is possible to further refine hard-label attacks, and inform a better evaluation of model robustness.
Given the availability of larger datasets in the future, our method may turn the strength of deep learning, which is efficiently extracting patterns in large-scale data, into a weakness
A PPENDIX A. Derivation of Manifold-Gradient Mutual Information (MI)
We define the manifold-gradient point-wise joint probability in a case-wise manner, for the respective values under g ∈ {− 1 , 1 } d and x ∈ R d .
We are concerned with the sub-gradient cases where x > 0 (denoted x + ) and x < 0 (denoted x − ) which correspond to fixed values of g based on class means y · µ with y ∈ {− 1 , 1 } .
This gives for each dimension k ,
[Equation]
g k = 1 , x
[Equation]
[Equation]
Since the Schmidt et al.
Gaussian mixture is created symmetrically (the probability mass is evenly split between the two classes i.e.
[Equation]
the mixture comprises one Gaussian offset by µ k and mirrored at x k = 0 ) we can simplify to
[Equation]
where x ∼ N ( y · µ , σI ) .
In words, Equation 6 is the symmetrical tail of the Gaussian mixture marginal while
Equation 5 is the remainder of the mixture.
Similarly, a point-wise gradient is given as the Rademacher outcome g k ∈ {± 1 } .
The choice of ϵ directly influences the marginal probability over the manifold.
The marginal probability over the manifold can be given as the Riemann approximations
[Equation]
and
[Equation]
with ∆ i = x + i,k − x + i − 1 ,k for arbitrary x ∗ i,k ∈ [ x + i − 1 ,k , x + i,k ] , and n is controlled by the hyper-parameter ϵ .
The marginal for the manifold under the gradient is given similarly as
[Equation]
where x + k > 0 for all dimensions k .
Denote the submanifold sampled from the positive ( y = 1 ) and negative ( y = − 1 ) classes as M + and M − , respectively.
Our definition for manifold-gradient mutual information is based on the standard definition of mutual information from information theory [33],
[Equation]
where ϵ is treated as a hyper-parameter controlling the value of n in p G ( g k ) .
By substitution into Equation 10 we have
[Equation]
This is split further similar to true positive, true negative, false positive
[Equation]
and false negative, as
[Equation]
and simplified due to symmetry at 0 as
[Equation]
The total un-normalized mutual information is given by the summation over dimensions I ( M ; G ) ϵ
= (cid:80) dk
=
I
M ; G ) ϵ,k .
Notably the cases for each possible scenario under detection theory are represented.
Each case is bounded by the results of [28].
By substitution from each marginal and joint probability in Equations 7, 3, and 4 respectively, we have the closed form solution for mutual information.
This leads to the Riemann approximation of Equation 2,
I ( M ; G ) ϵ,k =
[Equation]
with ∆ i = x + i,k − x + i − 1 ,k for arbitrary positive x ∗ i,k ∈ [ x + i − 1 ,k , x
+ i,k ] .
Since x + is a standard multi-variate Gaussian [33], the final mutual information is the summation over each dimension,
[Equation]
B. Hard-label attack formulation
Contemporary hard-label attacks are variants of random gradient-free method (RGF) [24], a gradient estimator which yields the estimate ˆ g over q random directions { u i } qi =1 .
OPT-Attack For benign example x 0 , true label y 0 , and hardlabel black-box function f : R d → { 1 , . .
, K } , [15] define the objective function g : R d → R as a function of search direction θ , where the optimal solution is g ( θ ∗ ) , the minimum distance from x 0 to the nearest adversarial example along the direction θ ∗ .
For the untargeted attack, g ( θ ) is the distance to any decision boundary along direction θ , and allows for estimating the gradient as
[Equation]
where β is a small smoothing parameter.
Notably, g ( θ ) is continuous even if f is a non-continuous step function.
Sign-OPT [17] later improved the query efficiency by only considering the sign of the gradient estimate,
[Equation]
We focus on the Sign-OPT variant, since the findings are more relevant to the current state-of-the-art.
HopSkipJumpAttack
Similar to
Sign-OPT,
HopSkipJumpAttack (HSJA) [16] uses a zeroth-order sign oracle to improve Boundary Attack [42].
HSJA lacks the convergence analysis of Sign-OPT and relies on one-point gradient estimate.
Regardless, HSJA is competitive and can excel in the L ∞ setting.
Dimension-reduced Sign-OPT & HSJA.
In general, for attacks relying on the [15] formulation, the update in Equation 16 becomes
[Equation]
for the reduced-dimension Gaussian vectors { u ′ i ∈ R d ′ } qi
= for integer d ′ <
d and direction θ ′ ∈ R d ′ .
The reduceddimension direction θ ′ is initialized randomly with θ ′ ∼
N (0 , 1) for the untargeted case, or for the targeted case as θ ′ =
E ( x t ) , where x t is a test sample correctly classified as target class t by the victim model.
This scheme also applies to HSJA, since HSJA performs a single-point sign estimate.
As in the normal variants, ˆ g is used to update θ ′ .
C. Implementation details
1) Hardware and Attack Hyperparameters: All experiments in the main paper were performed on an internal highperformance compute cluster equipped with NVIDIA Tesla
V100 Tensor Core GPUs and high-speed non-volatile flash storage.
In total 16 GPUs, 1TB main system memory, and 40 Intel Xeon CPU cores were used to run experiments completely.
Depending on dataset dimension, HSJA requires tuning of parameter γ for best performance.
On CIFAR-10 we used γ = 10 . 0 .
For ImageNet, it was necessary to set γ ≥ 1000 .
0 to re-create the published results of the regular variant [16].
Due to similar performance we use γ = 1000 . 0 for regular and dimension-reduced variants.
We note that the dimensionreduced variants like HSJA+BiLN were less sensitive to γ , performing similarly regardless of the setting.
2) Data sampling: Original samples are chosen from the test set using a technique similar to Chen et al.
[16]: on
CIFAR-10, twenty random samples are taken from each of the ten chosen classes (200 total samples).
On the ImageNet dataset, twenty classes are uniform-randomly chosen and ten random samples taken from each (200 total samples). [Figure] [Figure]
D. Supplemental Results
1) Query vs. Distortion Plots: We show the model queries against attack distortion measurement in Figure 7 to accompany the results in the main paper.
2) Grid search for DynBiLN Differential Factor: We performed a simple grid search to find an ideal differential factor η for our proposed DynBiLN attack variant.
The result on
CIFAR-10 is shown in Figure 8, averaged over 40 samples.
Comparison to strong white-box attacks: Recent work found that traditional adversarial training relies on narrow threat models which do not accurately capture the generalization error of a model, or equivalently, the distinction between off- and on-manifold examples [22].
Laidlaw et al.
proposed to instead formulate LPA and PPGD within the neural perceptual threat model (NPTM), directly optimizing for lower LPIPS, since it acts as a reliable estimator of perceptual distance (i.e., manifold distance) from adversarial sample to original.
In this way LPA and PPGD act as a best-case scenario for the creation of on-manifold adversarial examples in the gradientlevel setting.
We examine the relationship of these idealized gradient-level attacks to hard-label variants discussed in the
[Table] [Table]
main text, by attacking 500 samples with each attack on an adversarially trained model (equivalent to Madry CIFARin the main text).
We show in Table VII that on successful samples within the ϵ -ball [Table] our hard-label attack variants can achieve a similar decrease in LPIPS and FID, undercutting the narrow threat model and gradient-level baseline of l ∞ AutoAttack.
Notably our attacks can generate adversarial examples closer to the manifold, without having to directly optimize LPIPS (e.g., LPA and PPGD)
[Table]
The success rate gap between hard-label and gradient-level attacks in Table VII evidences the need for further improvements in hard-label attack efficiency.
To this end, it is notable that LPA and PPGD generate adversarial samples outside the ϵ -ball, which would be considered invalid in our threat model.
However, Laidlaw et al.
demonstrate that incorporating these “invalid” samples into adversarial training leads to models that are holistically more robust against unseen adversaries compared to previous adversarial training schemes.
If we relax the requirement for hard-label attacks to land inside the traditional l ∞ ϵ -ball, and instead adopt the NPTM, can hardlabel attacks still find on-manifold examples?
We perform this comparison in Table VIII by examining the “unconstrained” performance of the same hard-label attack variants.
We provide results within the radius using AutoAttack as a comparison in the last row.
Under the NPTM, we can see that hardlabel success rate is much higher due to finding samples arbitrarily close to the ϵ = 0 . 031 radius (a known behavior of adversarially trained models [5], [43]), at the cost of higher LPIPS and FID, since our hard-label variants still rely on l ∞ distance for their objectives (e.g., formulation of g for gradient estimation is optimal l ∞ step size in HSJA). LPA and
PPGD, which are gradient-level attacks, have the advantage of directly optimizing LPIPS instead of l ∞ , resulting in lower
FID and LPIPS.
It can be observed that FID is still consistently lower for dimension-reduced variants (e.g., 0.305/1.715 for
BiLN+HSJA/HSJA).
This motivates an exciting new research direction for hard-label attacks which formulate their objective surrogates under NPTM, rather than traditional l p -based metrics.
The reviews from ICML 2022 are given below verbatim, followed by a description of changes.
Reviewer
1) Summarize the contributions made in the paper with your own words: This paper focuses on the zeroth-order attack in adversarial attacking.
The paper discovers that the dimension reduction in attacks increases the information about data manifold in the gradient.
2) Novelty, relevance, significance:
The novelty of the paper is limited.
Many definitions and metrics in the paper are proposed in previous studies, e.g., Definition 3.1-3.3.
3) Soundness: Actually, the poor writing of this paper makes it hard to evaluate its soundness.
To my understanding, some claims in the paper are not well justified.
Specifically, there are many hypotheses but without proofs.
The hypotheses are only verified on observations, which are obtained from limited experiments.
4) Quality of writing/presentation:
The presentation of the paper is poor, which is the main shorting coming of the paper.
First, the motivation of the paper is not clearly stated.
Second, the structure of the paper is not wellorganized.
In Section 3 and Section 4, there are many definitions, hypothesis and observations, but the relationship or linkage between these hypotheses and observations is unclear.
Besides, the verifications for hypotheses are usually far away from the hypothesis, which makes it hard to follow.
Third, many technical terms are used without
The presentation of the paper is poor, which is the main shortcoming of the paper.
First, the motivation of the paper is not clearly stated.
Second, the structure of the paper is not well-organized.
There are many definitions, hypotheses, and observations in Section 3 and Section 4, but the relationship or linkage between these hypotheses and observations is unclear.
Besides, the verifications for hypotheses are usually far away from the hypothesis, which makes it hard to follow.
Third, many technical terms are used without a rigorous definition.
For example, the “model point estimate” and “visual alignment” are confusing.a rigorous definition.
For example, the “model point estimate” and
“visual alignment”
are confusing
5) Literature: Seems good. 6) Basis of review (how much of the paper did you read)?
I read the main paper
but do not
read the appendix
7) Summary:
Pros:Thepaper discovers an interesting phenomenon that reducing dimensions in zeroth-order hard label attacking may increase the information about data manifold in the gradient.
Cons:
1.Mylargest concern is the poor writing of the paper.
As aforementioned, the motivation, structure, and many terms in the paper are not clear.
The poor presentation of the paper hinders my understanding of the contribution of the paper.
The Markov chain M − > G − > ˜ G seems problematic.
The estimated gradient ˜ G is not obtained from or based on the true gradient G , though they are relevant to some extent.
Instead, both the true gradient and the estimated gradient ˜ G come from the model.
Therefore, I suppose the Markov chain should be look like M − > model − > G and M − > model − > ˜ G . 3.
The dimension-reduction technique in attacking is a main focus of the paper, but it is not formally introduced.
I suggest the authors first introduce the standard formulation for the dimension-reduction technique before hypothesis 1. 4.
All hypotheses in the paper are proposed based on heuristics and proven by observations, but without solid theoretical proofs.
Meanwhile, some observations are obtained on simple examples (e.g., observation 1 is based on simple assumptions for the data distribution).
It is claimed in hypothesis 2 that higher a upper bound on I ( M, ˜ G ) caused a smaller manifold distance.
However, the theoretical relationship between I ( M, ˜ G ) and manifold distance is unclear.
Experimental results do not support main claims in the paper.
For example, in Table 1, the LPIPS scores are not reduced when using dimension-reduction techniques in most cases.
In Table 2, results on HSJA and results on Sign-OPT conflict with each other.
Reviewer
1) Summarize the contributions made in the paper with your own words: The paper aims to provide better understanding of the hard-label adversarial attack from information
- theoretic and the manifold perspectives.
The paper is then evaluated on real-world datasets and with state-of-the-art hard label attacks (HSJA and Sign-OPT).
2) Novelty, relevance, significance: The paper is about hard label adversarial attack against machine learning models, thus it is relevant to machine learning community.
I’m not familiar with the manifold theory.
Based on my confidence (which is low), the paper is to some extent novel because of new perspectives.
The paper has not proposed novel attack but provided only the understanding of existing attacks, thus the significance is low.
3) Soundness: Since I’m not familiar with the manifold theory, I think the paper is sound because the results seem to be corresponding the claimed observation.
Quality of writing/presentation: The paper is not easy to follow and the writing can be improved.
5) Literature: Even though the authors provide related works in appendix, I think they should be moved to main text for completeness.
6) Basis of review (how much of the paper did you read)?
I have read the full main text, except the appendix. 7)
Summary: Strength: 1.Thetopic of hard-label adversarial attacks is important
for developing
robust models.
The perspective is interesting and some experimental results are provided
Weakness:
1.Thepaper is not easy to read and can be improved by more careful revision.
It is not clear how claimed observations can help build more robust neural networks.
Description of changes.
The primary issues from the previous review cycle were concerned with the paper presentation and assumptions around the Markov chain behavior.
In particular the Gaussian data distribution from Section 3 was considered too simple for the claims made in the previous draft.
Additionally, it wasn’t clear to reviewers how the adversary might take advantage of the Markov chain through the gradient estimates (or whether it existed at all on practical datasets).
To address these issues, we entirely re-wrote the evaluation sections (Section IV and onwards) and performed major revision of all sections before, shifting the posture of the paper to take a more empirical stance.
The new changes include the following:
To motivate the Markov chain behavior on practical datasets
, we
designed a probing algorithm based on interpretable ML techniques that verifies the learnability of an image’s local semantic superpixels from hard-label boundary traversals alone
Thanks to the new
probing
algorithm,
we performed an analysis against dimension-reduced variants, and found the quality of learned semantic superpixels
improves when
dimension
reduction is utilized (represented by higher R score , particularly across
common attack stages 3) We included several new visualizations (Figure 4), thanks in part to the Markov chain probing algorithm, which helps clarify the specific mechanism that hard-label adversaries use to obtain lower manifold distance.
To further aid clarity, we included Figure 1 to summarize the geometric interpretation of our results.
We also made sure to summarize key findings at the end of each section to enable better understanding.
Based on the new experiments described above, we launched a new large-scale experimental evaluation which included more baselines, ablation studies for dimensionreduced variants, and larger sample sizes to reduce margins of error.
To address concerns with the selection of dimensionality from previous reviews, we proposed the dynamic scaling variant (DynBiLN) which uses the Rscore as a metric for the quality of search directions.
5) To summarize the key findings, we re-organized the experimental evaluation through three research questions, and included a discussion section to summarize the takeaways of our paper, limitations, and future work.