L OWER B OUNDS ON THE R OBUSTNESS OF F IXED F EATURE E XTRACTORS TO T EST - TIME A DVERSARIES
Anonymous authors
Paper under double-blind review
A BSTRACT
Understanding the robustness of machine learning models to adversarial examples generated by test-time adversaries is a problem of great interest.
Recent theoretical work has derived lower bounds on how robust any model can be, when a data distribution and attacker constraints are speciﬁed.
However, these bounds only apply to arbitrary classiﬁcation functions and do not account for speciﬁc architectures and models used in practice, such as neural networks.
In this paper, we develop a methodology to analyze the robustness of ﬁxed feature extractors, which in turn provides bounds on the robustness of any classiﬁer trained on top of it.
The tightness of these bounds relies on the effectiveness of the method used to ﬁnd collisions between pairs of perturbed examples at deeper layers.
For linear feature extractors, we provide closed-form expressions for collision ﬁnding while for arbitrary feature extractors, we propose a bespoke algorithm based on the iterative solution of a convex program that provably ﬁnds collisions.
We utilize our bounds to identify the layers of robustly trained models that contribute the most to a lack of robustness, as well as compare the same layer across different training methods to provide a quantitative comparison of their relative robustness
I NTRODUCTION
The robustness of machine learning models to test-time (evasion) attacks, particularly via adversarial examples, has been studied extensively from an empirical perspective (Croce et al., 2020; Papernot et al., 2016; Li et al., 2020; Biggio & Roli, 2017).
A plethora of attacks (Szegedy et al., 2013;
Carlini & Wagner, 2017; Bhagoji et al., 2018; Croce & Hein, 2020) and defenses (Madry et al., 2018; Zhang et al., 2019) has been proposed.
The resulting arms race between proposed attacks and defenses, has led recent theoretical work (Dohmatob, 2019; Cullina et al., 2018;
Mahloujifar et al., 2019;
Bhagoji et al., 2019; 2021; Montasser et al., 2019) to focus on understanding the fundamental bounds of learning in the presence of adversarially perturbed inputs, in terms of both the existence of robust classiﬁers and the sample complexity required to learn them.
A line of research on this front has investigated lower bounds on the robustness attainable by any classiﬁer when classifying data from distributions modiﬁed by adversarial perturbations (Pydi &
Jog, 2020; Bhagoji et al., 2019; 2021).
In particular, information-theoretic lower bounds on the minimum possible robust 0 − 1 and cross-entropy losses have been derived for arbitrary discrete distributions and very general attack models.
However, the bounds from these papers are only applicable when considering the set of possible classiﬁers to be all measurable functions , which does not translate to practice.
The classiﬁer architecture and even some portion of the parameters are ﬁxed (such as in transfer learning) in practical approaches to training machine learning models.
In this paper, we leverage this line of work to ﬁnd lower bounds on the robustness of commonly used, ﬁxed feature extractors.
This provides a principled method to compare the robustness of feature extractors obtained by different training methods to arbitrary test-time adversaries.
The key question we answer is:
What is the minimum robust loss incurred by any classiﬁer trained on top of a ﬁxed feature extractor?
Two challenges for training robust machine learning models motivate this question. First, since transfer learning is now common practice, it is incumbent upon the model trainer to choose a robust pre-trained feature extractor. Our work offers a principled and quantitative method to choose among feature extractors obtained from different layers and training methods.
Second, we are able to shed light on the impact of common architectural choices, such as activation functions and dimensionality reduction on robustness. This arises from our study of how the lower bound evolves from layer to layer within deep neural networks.
Importance of lower bounds on robustness: To determine classiﬁer-agnostic lower bounds on robustness, earlier work focused on the interaction between points from different classes in the input space when perturbed through the construction of a conﬂict graph .
Minimizing an appropriately deﬁned loss function over this graph determines an information-theoretic lower bound on the loss.
Intuitively, the denser the graph is, the higher the lower bound.
These classiﬁer-agnostic bounds are able to determine the plausibility of robust classiﬁcation for a given adversary and dataset, and highlight practically relevant perturbation budget.
1. C ONTRIBUTIONS
Lower bounds on robustness for ﬁxed feature extractors: We signiﬁcantly extend prior work by deriving lower bounds that depend on the architecture and weights of a ﬁxed feature extractors.
We construct a distance function over the input space that depends on the feature extractor in use.
Our results have implications for both analyzing the robustness of already trained classiﬁers as well as determining feature extractors to use for applications such as transfer learning .
Bespoke algorithms for collision ﬁnding:
For linear feature extractors such as the ﬁrst layer of a neural network, we determine exact, closed form expressions to ﬁnd collisions.
Collision ﬁnding for non-linear feature extractors is a non-convex optimization problem, for which we propose a custom algorithm.
We approach the collision-ﬁnding problem with a descent algorithm that solves a sequence of convex optimization problems over polytopes.
This algorithm has the structural advantage of maintaining feasibility (for some adversarial budget constraint) at each step.
Despite the fact that it does not ﬁnd a global minimum, the solutions found are useful for obtaining not just approximations but actual lower bounds on optimal adversarial classiﬁcation loss.
Empirical ﬁndings: We utilize our method to ﬁnd numerical lower bounds on the robustness of ﬁxed feature extractors trained on two different datasets with both fully-connected and convolutional architectures.
Our results indicate that the use of dimensionality-reducing linear layers as well as the effective compression induced by ReLU activations lead to signiﬁcantly less robust networks.
Further, we conﬁrm the oft-cited observation that mismatches between the adversary’s budget at train and test time impacts robustness negatively.
Taken together, these ﬁndings point towards future design considerations for robust models.
D ERIVING LOWER BOUNDS FOR FIXED FEATURE EXTRACTORS
In this section, we develop a method for evaluating the robustness of a feature extractor for classiﬁcation in the presence of a test-time adversary.
We characterize the optimal adversarial loss achievable by any classiﬁer that uses the ﬁxed feature extractor as its initial layer.
This characterization is based on a conﬂict graph derived from a discrete data distribution and the constraints of the adversary.
Examples and labels: We consider a supervised classiﬁcation problem with a test-time adversary.
We have an example space X and a label space Y = {− 1 , 1 } .
Labeled examples are sampled from a joint probability distribution P over X × Y .
Test-time adversary:
The test-time adversary modiﬁes a natural example subject to some constraints to generate an adversarial example that will be classiﬁed Goodfellow et al.
; Szegedy et al.
; Carlini & Wagner (2017).
Formally, the adversary samples a labeled example ( x, y ) from P and selects ˜ x ∈ N ( x ) , where N : X → 2 ˜ X is the neighborhood function encoding the constraints on the adversary and ˜ X is the space of adversarial examples.
For all x , N ( x ) must be nonempty.
This deﬁnition encompasses the (cid:96) p family of constraints widely used in previous work.
Measuring adversarial loss: We consider ‘soft’ classiﬁcation functions (or classiﬁers) that map examples to probability distributions over the classes.
These are h : X → ∆ Y , where ∆ Y = { p ∈ R Y : p ≥ 0 , (cid:80)
y ∈Y p
y
= 1 } .
We measure classiﬁcation performance with a loss function (cid:96) :
∆ Y × Y → R , so the expected performance of a classiﬁer h is E [
sup ˜ x ∈ N ( x ) (cid:96) ( h (˜ x ) , y )] , where ( x, y ) ∼ P .
In the two class setting, h (˜ x ) − 1 = 1 − h (˜ x ) 1 , so any loss function that treats the classes symmetrically can be expressed as (cid:96) ( p, y ) =
(cid:96) (cid:48) ( p y ) .
Additionally, (cid:96) (cid:48) should be decreasing.
Together, these allow the optimization over adversarial examples to be moved inside the loss function, giving
E (cid:104) (cid:96) (
cid:16) inf ˜ x ∈ N ( x ) h (˜ x ) y , y (cid:17)(cid:105) . Thus, the adversarial optimization can be analyzed by itself.
Deﬁnition 1.
For a soft classiﬁer h , the correct-classiﬁcation probability q v that it achieves on an example v = ( x, y ) in the presence of an adversary is q v = inf ˜ x ∈ N ( x ) h (˜ x ) y .
The space of achievable correct classiﬁcation probabilities is
[Equation]
Bhagoji et al.
characterized P V ,N, H in the case that H is the class of measurable functions
X → ∆ Y .
For a data distribution P that is discrete with ﬁnite support V ⊆
X × Y , this allows the minimum adversarial loss achievable to be expressed as an optimization over P V ,N, H : [Equation]
We extend their approach to analyze feature extractors as follows.
Given a feature space Z and a ﬁxed, measurable feature extractor f : ˜ X → Z , deﬁne H f = { h ∈ H : h = g ◦ f } : an element of H f is some measureable g : Z → ∆ Y composed with f .
Our aim is to characterize P V ,N, H f so that we can optimize loss functions over it to evaluate the suitability of f .
Conﬂict graph: In Bhagoji et al.
(2021), the notion of a conﬂict graph was used to record neighborhood intersections for pairs of points from different classes.
When such an intersection exists, it is impossible for any classiﬁer to correctly classify both of those points in the adversarial setting.
We extend this notion to apply to the family of classiﬁers using a ﬁxed feature extractor f .
In our setting, a conﬂict exists between a pair of points when each of them has a neighbor that is mapped to the same point in the feature space.
We call the conﬂict graph G V ,N, H f , where V ⊆
X × Y .
When we apply G V ,N,f in order to understand classiﬁcation of labeled examples with distribution P , we take V to be the support of P .
The graph is bipartite: the vertex set is partitioned into parts V c = V ∩ ( X × { c } ) .
The edge set is
E V ,N,f = { (( u, 1) , ( v, − 1))
∈ V 1 × V − 1 : ∃ ˜ u ∈ N ( u ) , ˜ v ∈ N ( v ) such that f (˜ u ) =
f
(˜ v ) } .
Lemma 1 (Feasible output probabilities) .
The set of correct classiﬁcation probability vectors for support points V , adversarial constraint N , and hypothesis class H f is
[Equation]
where B ∈ R E×V is the edge incidence matrix of the conﬂict graph G V ,N,f .
The proof is given in Appendix A and follows the structure of the proof in Bhagoji et al.
Approximating G V ,N,f and P V ,N, H f : If instead of knowing the true conﬂict graph G V ,N,f , we have some subgraph, then we can ﬁnd a polytope that is a superset of the true P V ,N, H f .
If we minimize some expected loss over this proxy polytope, we obtain a lower bound on the optimal loss over
H f .
Because subgraphs of the conﬂict graph lead to valid lower bounds on optimal classiﬁcation performance, we can use this method to evaluate the quality of a feature extractor f even if exact computation of the conﬂict graph is computationally intractable.
2. D ISTANCE INTERPRETATION
The construction of a conﬂict graph and characterization of the feasible correct classiﬁcation probabilities from the previous section apply to any feature extractor and neighborhood constraint. In [Figure]
the most commonly studied settings, the neighborhood constraint arises from a distance function:
N ε ( x ) =
{ ˜ x ∈ ˜ X : d ( x, ˜ x ) ≤ ε } .
This parameter ε is the adversarial budget constraint.
For any two examples u and v , a natural quantity to consider is the smallest adversarial budget that would cause the edge ( u, v ) be appear in the conﬂict graph:
[Equation]
We will call this the distance on X induced by d .
When X = ˜ X
= R d and d ( x, x (cid:48) )
= (cid:107) x − x (cid:48) (cid:107) p , the minimal adversarial budget is simply 12 (cid:107) u − v (cid:107) p .
Because the relationship to the distance used in the adversarial budget constraint is so simple, this quantity is not usually discussed independently.
This deﬁnition generalizes easily to the setting of classiﬁcation with a particular feature extractor, but the resulting quantity is much more interesting.
Deﬁnition 2.
The distance induced on X by d and f is the minimum adversarial budget required to create a conﬂict between u and v after a feature extractor f :
[Equation]
This reduces to ε ∗ ( u, v ) when f is the identity function, or more generally any injective function.
Observe that any choice of ˜ u and ˜ v in (2) provide an upper bound on ε ∗ f ( u, v ) , which is useful for ﬁnding a subgraph of G V ,N,f and lower bounds on optimal classiﬁcation performance in H f .
Figure 1 illustrates the computation of ε ∗ f for a simple f that is related to the ReLU function.
Induced distance is not a distance on the feature space:
The fact ε ∗ f is deﬁned on X is essential: it is not possible to interpret it as a distance on Z .
For a full explanation of this point, see Appendix B.
D ISTANCE COMPUTATIONS FOR PRACTICAL FEATURE EXTRACTORS
Throughout the remainder of the paper, we will work in a more concrete setting.
We assume that our example space is a real vector space and that adversarial examples are from the same space: X = ˜ X
= R n 1 .
Let B ⊆
X be a nonempty, closed, convex, origin-symmetric set.
These conditions imply the zero vector in contained in B .
We take the neighborhood of any point to be a scaled, shifted version of this ball: N ε ( x ) : x + ε B .
Neighborhood constraints derived from (cid:96) p norms ﬁt into this class: we encode them by deﬁning B = { δ ∈ R n 1 : (cid:107) δ (cid:107) p ≤ 1 } , which results in
N ε ( x ) =
{ ˜ x ∈ R n 1 : (cid:107) ˜ x − x (cid:107) p ≤ ε } .
We will focus on (cid:96) 2 -based neighborhoods, but our algorithms could be adapted to any choice of B over which efﬁcient optimization is possible.
3. L INEAR FEATURE EXTRACTORS
Suppose that our feature extractor is an afﬁne linear function of the input example: f ( x ) =
Lx + k for some matrix L ∈ R n 2 × n 1 and vector k ∈ R n 2 .
Then the distance ε f ( u, v ) becomes inf
[Equation]
[Equation] = ∅ [Equation] Because { ( ε, δ ) ∈ R 1+ n 1 : δ ∈ ε B} is closed convex cone, ε f ( u, v ) can be expressed as the following convex optimization problem:
[Equation]
Because B is closed and the linear subspace is always nonempty, the inﬁmum is achieved.
Also, it is sufﬁcient to consider ( δ, δ (cid:48) ) satisfying δ (cid:48)
= − δ : for any feasible ( ε, δ, δ (cid:48) ) , the point ( ε, ( δ − δ (cid:48) ) / 2 , ( δ (cid:48) − δ ) / 2) is also feasible, has the same value, and satisﬁes the additional constraint.
The feasibility of the symmetrized point uses the origin-symmetry of B .
The simpliﬁed program is min ε subject to δ ∈ ε B ,
Lδ = L ( v − u ) / 2 .
Thus the optimal adversarial strategy for creating conﬂict between u and v is intuitive: produce examples with the same features as the midpoint ( u + v ) / 2 .
If B is the unit (cid:96) 2 ball, there are further simpliﬁcations.
Consider the singular value decomposition
L = U Σ V T where we do not include zero singular values in Σ .
Then the linear map given by U Σ is injective and can be canceled from the linear constraint on δ .
The resulting program is min ε subject to (cid:107) δ (cid:107) 2 ≤
[Equation]
[Equation]
the optimal choice of δ is δ = 12 V V T ( v − u ) , and ε f ( u, v )
=
12 (cid:107) V T ( v − u ) (cid:107) 2 .
Observe that this is the norm of a vector in R n 2 , i.e. the feature space, contrasting with our discussion in Section B.
However, this is essentially the only case ε f ( u, v ) simpliﬁes into a feature space distance.
F ULLY CONNECTED NEURAL NETWORKS WITH R E LU ACTIVATIONS
For this feature extractor architecture, we have f = f
( (cid:96) ) ◦ . . .
◦ f (1) where the layer i function is f ( i ) :
R n i → R n i +1 , f ( i ) ( z ) =
( k ( i ) + L ( i ) z )
+ .
Here z + represents the component-wise positive part of the vector z .
Then ε f ( u, v ) is the value of the following optimization problem: min ε subject to δ ∈ ε B ,
[Equation]
As in the linear case, the minimum exists because the cones are closed and the equality constraint is feasible.
In contrast with the linear case, the equality constraint is nonconvex.
The local linear approximation to f ( i ) ( z ) around a point z (cid:48) is diag( s ( i,z (cid:48) ) )
( k ( i ) + L ( i ) z ) , where s ( i,z (cid:48) ) is a zero-one vector that depends on the sign pattern of k ( i )
+ L ( i ) z (cid:48) :
s ( i,z (cid:48) )
j = 1 (( k ( i ) + L ( i ) z (cid:48) )
j > 0) .
In other words, s ( i,z (cid:48) ) is the ReLU activation pattern at layer
i when z (cid:48) is the input to that layer.
Using these linear approximations, the feasible set of ( δ, δ (cid:48) )
∈ R n 1 + n 1 satisfying the constraint f ( u + δ ) =
f ( v + δ (cid:48) ) can be decomposed as a union of polytopes: each activation pattern deﬁnes a linear subspace and there are some linear inequalities specifying the region where that activation pattern actually occurs.
For a one-layer network, the linear piece for pattern s is
[Equation]
Thus one of the polytopes composing the feasible set is ( δ, δ (cid:48) )
∈ R n 1 + n 1 satisfying
[Equation]
In the one-layer case, the whole feasible region is covered by polytopes where s = s (cid:48) .
Observe that the dimension of the subspace satisfying the linear equality varies with s .
When f contains multiple layers, each polytope in the feasible set is deﬁned by a linear equality constraint involving feature vectors together with a linear inequality for the sign of each ReLU input.
We optimize over this feasible set with a descent algorithm:
Algorithm 2 details the version for single-layer networks. The method generalizes to multiple layers and is used to obtain our results in
Section 4.2).
A full description of the general version is in Appendix C. We initialize our search in a polytope that we know to be nonempty because it contains the feature space collision induced by the midpoint of the two examples.
Within each polytope, we can minimize the objective exactly by solving a linear cone program.
We examine the dual variables associated with the linear inequality constraints to determine whether an adjacent polytope exists in which we can continue the search.
In the one layer case, the cone program that we solve, which depends on ( L, k, u, v, s ) , is min ε subject to ( ε, δ, δ (cid:48) )
∈ R 1+ n 1 + n 1 , δ ∈ ε B , δ (cid:48) ∈ ε B , diag( s ) L ( u + δ ) =
diag( s ) L ( v + δ (cid:48) ) , (2 diag( s ) − I ) ( k + L ( u + δ )) ≥ 0 , (2 diag( s ) − I ) ( k + L ( v + δ (cid:48) )) ≥ 0 .
Input: u, v ∈ R n 1 , L ∈ R n 2 × n 1 , k ∈ R n
Output: ε ∈ R , δ, δ (cid:48)
∈ R n1: z ← k + 12 L ( u + v ) , ε ← 12 (cid:107) u − v (cid:107) 2: for 0 ≤ j < n 2 do 3: s j ← 1 ( z j > 0) , 4: end for 5: repeat 6: ε old ← ε 7: ( ε, δ, δ (cid:48) , z, z (cid:48) )
← ConeLP ( L, k, u, v, s ) 8: s old ← s 9: for 0 ≤ j < n 2 do 10: if z j > 0 and z (cid:48)
j > 0 then 11: s j ← 1 − s j 12: end if 13: end for 14: until s old = s or ε old =
ε
We need not just the values of the primal solution, but the values of dual variables associated with the linear inequalities in the solution to the dual problem.
These are called z and z (cid:48) in Algorithm 2.
When both z j > 0 and z (cid:48) j > 0 , the input to ReLU j is zero when both f ( u + δ ) and f ( v + δ (cid:48) ) are computed, and the objective function could be decreased further by allowing the input to switch signs.
In this case, we move to another polytope based on the new ReLU states and search there.
When we fail to ﬁnd such pairs of dual variables, either the minimum is in the interior of the current polytope (and thus is supported only by cone constraints), or the minimum is on the boundary of the current polytope but there is no adjacent polytope in which we could continue the search.
Thus we end the search.
Algorithm termination, convergence, and complexity: The descent algorithm will terminate in a ﬁnite number of iterations and will ﬁnd a local minimum of the distance function. Since the feasible space is non-convex, any local descent procedure is not guaranteed to ﬁnd the global minimum. The number of variables in the cone program is proportional to the number of ReLUs in the feature extractor plus the dimension of the example space. The time-complexity of a single iteration of the search is polynomial in the input dimension and number of ReLUs.
The feasible region is the union of a ﬁnite but exponentially large number of convex polytopes.
Due to the requirement that each iteration make progress, it is impossible to ever revisit a polytope.
Thus, termination is guaranteed , but no polynomial bound on the number of iterations is available.
We suspect that as in the case of the simplex algorithm for linear programming, input data resulting in an extremely large number of iterations may exist, but would have a delicate structure that is unlikely to arise in practice.
Convolution: It is straightforward to represent a convolutional layer as a linear layer by constructing the appropriate Topelitz matrix, and thus our method applies directly. Batch-normalization: Batch norm layers are simply afﬁne functions of their inputs at test-time, and the transformations they induce can be easily included in a linear layer. Our method thus applies.
Max pooling: Max-pool layers, like ReLUs, are piecewise linear functions of their inputs, so constraints coming from the equality of max-pool outputs also lead to feasible regions that are the union of polytopes.
This is a simple extension left for future work due to a lack of space. Other activation functions and architectures: Injective activation functions such as Leaky ReLU,
ELU and sigmoid will not lead to additional collisions. Further, since they are not piecewise, a different descent algorithm would be needed. We note that our framework cannot ﬁnd collisions in networks with both forward and backward ﬂows, such as those with attention.
E VALUATION
In this section, we demonstrate how the methodology outlined in the previous sections for determining the robustness of a ﬁxed feature extractor can be used in practice.
Our detailed results provide insights on how the overall robustness of a neural network evolves through its different layers and is impacted by the training procedure used.
4. F ROM COLLISION - FINDING TO LOWER BOUNDS
We ﬁnd lower bounds on robust loss over the training set by minimizing a loss function over G V ,N,f . Vertices V from training data: The vertex set V is a representation of the training data.
We use 2 -class problems derived from MNIST (LeCun & Cortes, 1998) or Fashion MNIST (Xiao et al., 2017) as in previous work (Pydi & Jog, 2020; Bhagoji et al., 2019; 2021). Neighborhood function N : We use the common (cid:96) 2 -norm ball constraint (Madry et al., 2018), in which the adversary’s strength is parametrized by the radius (cid:15) of the ball 2 .
Fixed feature extractors f : We use the composition of the layers of convolutional and fullyconnected DNNs as our ﬁxed feature extractors (details in Appendix E).
The ﬁrst layer of any of these networks (before a non-linear activation is applied) behaves as a linear feature extractor.
Subsequent layers behave as non-linear feature extractors.
These networks are trained using one of standard cross-entropy loss minimization or robust training with either PGD-based adversarial training (Madry et al., 2018) or the TRADES loss (Zhang et al., 2019).
Edge set E from collision ﬁnding:
Algorithm 2 provides a greedy, iterative method to ﬁnd collisions between feature representations of pairs of points from different classes.
Each successful collision is an edge in the bipartite conﬂict graph.
Each iteration of this algorithm can be cast as a convex program in the form of a linear cone (as detailed in Appendix D).
We solve this linear cone program using CVXOPT (Andersen et al., 2013).
Since we are working with an (cid:96) 2 -norm constrained adversary, we can speed up computation by projecting the inputs onto the space spanned by the right singular vectors of the ﬁrst linear layer.
For linear convolutional layers, we cast the convolution operation as a matrix multiplication to enable the use of the closed form derived in Section 3.1.
We also experimented with a modiﬁcation of the Auto-PGD (Croce & Hein, 2020) algorithm to ﬁnd collisions at deeper layers with ﬁxed budgets as done in Engstrom et al.
However, this was less effective and more expensive at ﬁnding collisions, so all further results use Algorithm 2.
Computing the lower bound from the conﬂict graph: The conﬂict graph determines the set of possible output probabilities for each vertex (training data point) for the optimal classiﬁer. Minimizing the 0 − 1 and cross-entropy losses over this graph thus results in a lower bound over these losses since any classiﬁer must incur a larger loss than the optimal classiﬁer, by deﬁnition. We use the method from Bhagoji et al.
(2021) over the conﬂict graphs we derive from deeper layer representations.
Results in the main body are for the cross-entropy loss (others in Appendix F).
4. R ESULTS
Interpreting lower bound values: The lower bound on the input space representations is computed by considering all possible classiﬁers and is thus the best possible , while bounds for other representations are computed by restricting the space of possible classiﬁers.
Intuitively, the latter will always be larger than or equal to the former.
The key metric to understanding the robustness of feature extractors is to check how much larger the corresponding lower bounds are.
The magnitude of this difference measures the contribution of that layer to the overall network’s lack of robustness.
Robustness of linear layer representations: Using the procedure outlined in Section 3.1, we ﬁnd collisions after the ﬁrst linear layer and construct a conﬂict graph for several models (Fig. 2) for
[Figure]
[Figure]
both the MNIST and Fashion MNIST datasets.
As expected, the linear layers of models with benign training are the least robust, with robustness increasing with the budget (cid:15) train used during training.
Somewhat surprisingly, the marginal beneﬁts of using a larger budget reduce as (cid:15) train increases.
How does robustness evolve across layers?
Having looked at the how robust linear layer representations are, it is pertinent to consider other representations derived from other layers deeper in the network.
Of particular interest are post-ReLU representations for models with benign (Fig. 3) and robust training (Fig. 4).
We ﬁnd that the ﬁrst ReLU activation layer contributes signiﬁcantly to an increase in the lower bound for both benign and robustly trained networks.
We observe that postReLU representations tend to be sparse, leading to an easier search problem and a larger number of collisions.
The second linear layer, on the other hand, does not lead to much additional increase in the loss.
This is a property of the particular architecture we use, and a smaller linear layer is likely to lead to larger increases in loss.
Finally, the second set of ReLU activations does have a measurable impact on robustness, particularly for the benign network.
How does the parametrization of robust training impact layer-wise robustness?
As expected, layers extracted from robustly trained network are more robust than their benign counterparts for corresponding values of (cid:15) .
We ﬁnd a signiﬁcant difference between PGD-based adversarial training and TRADES in terms of the robustness of their ﬁrst linear layers (Fig. 2), but this largely disappears by the second ReLU activation layer (Fig. 5).
Interestingly, we observe a phenomenon where layers of a network robustly trained using higher values of (cid:15) train can be less robust than those using a lower value, when the value of (cid:15) at which evaluation is performed is less than the higher (cid:15) train .
layers on robustness: As discussed in Section 3.3, the ﬁrst convolutional layer can be thought as simply a matrix multiplication, although the size of the resulting matrix can be large.
We ﬁnd that since the effective dimension of the resulting features is much larger than the input dimension for the datasets we consider, the linear convolutional layer does not lead to an increase in the lower bound (Fig. 11 in Appendix).
R ELATED W ORK AND D ISCUSSION 5. R ELATED WORK
Lower bounds on robustness: For cases when the data distribution is speciﬁed, Dohmatob (2019) and Mahloujifar et al.
(2019) use the ‘blowup’ property to determine bounds on the robust loss, given some level of loss on benign data.
Bhagoji et al.
(2019), Pydi & Jog (2020) and Bhagoji et al.
[Figure] [Figure] provide lower bounds on robust loss when the set of classiﬁers under consideration is all measurable functions.
These bounds are distribution-agnostic and do not depend on the loss on benign data.
Certiﬁcation and Veriﬁcation: Work on certiﬁed robustness has considered techniques for training neural networks such that the resulting models are provably robust to perturbations upper bounded by a given budget (Kolter & Wong, 2018; Raghunathan et al., 2018;
Cohen et al., 2019; Li et al., 2020).
Typically, these models can only be certiﬁed to be robust to small budgets.
In contrast, our work provides lower bounds on the robustness of partial models which are applicable across a large range of budgets.
Approaches to verifying the robustness of neural networks (Bunel et al., 2017;
Tjeng et al., 2019; Gowal et al., 2018) are closely related to our work, but differ in that they consider ﬁxed end-to-end networks while we focus on a layer-by-layer analysis, allowing us to argue about the robustness of classiﬁers trained on top of given feature extractors.
5. D ISCUSSION
Implications for training robust models: Our results indicate that layers in the network that reduce the effective dimension of their incoming inputs have the largest negative impact on robustness (see further results in Appendix F.1).
Two prominent examples of this are ReLU layers that reduce effective dimension by only considering non-negative outputs and fully connected layers with fewer outputs than inputs.
On the other hand, linear convolutional layers do not have a negative impact on robustness.
This indicates that not reducing the effective dimension of any deeper feature to be lower than that of the input data is likely to beneﬁt robustness.
Further, our results conﬁrm that the use of larger values of (cid:15) train does not necessarily translate to higher robustness at lower budgets.
This points towards the need for algorithms that can effectively train a network to be robust to more than a single adversary at a time.
Finally, we ﬁnd a qualitative difference in the layers learned using
PGD-training and TRADES, implying interesting learning dynamics with different robust losses.
Extending to state-of-the-art models and datasets: All of our experiments in this paper are on simple models and datasets which demonstrate the feasibility and use of our method. However, state-of-the art feature extractors for datasets such as Imagenet are far deeper than those considered in this paper.
Thus, our algorithm would need to be made considerably faster to handle these cases. While our framework, can handle skip connections, networks with attention are beyond its scope. Nevertheless, the feature extractors we consider are robust for the tasks we evaluate them on, making our results and conclusions representative.
A P ROOFS
Proof of Lemma 1.
Suppose that e = (( u, 1) , ( v, − 1)) ∈ E .
Then, there are some ˜ z ∈ Z , ˜ u ∈ N ( u ) , and ˜ v ∈ N ( v ) such that f (˜ u ) =
f (˜ v ) =
˜ z .
We have q u ≤ h ( f (˜ u )) 1 = h
(˜ z ) 1 , q v ≤ h ( f (˜ v ))
− 1 =
h
(˜ z ) − 1 , and h (˜ z )
+ h (˜ z ) − 1 = 1 .
Combining these gives the constraint ( Bq ) e ≤ 1 , which appears in (1).
Now, we will show that each vector q in the polytope is achievable by some h .
The construction is simple: at each point in the feature space, assign the largest possible probability to class 1: let h (˜ z ) 1 = sup w :˜
z ∈ f ( N ( w ))
q w and h (˜ z ) − 1 = 1 − h
(˜ z ) 1 .
This achieves the desired performance for examples from class 1 :
[Equation]
For an example is v in class − 1 we have
[Equation]
The ﬁnal inequality uses the fact that q satisﬁes Bq ≤ 1 .
B I NDUCED DISTANCE IS NOT A DISTANCE ON Z
The fact ε ∗ f is deﬁned on X is essential.
It is not possible to interpret the induced distance as a distance in the feature space Z .
This is because f ( N ε ( x )) , the set of features of points near x , cannot be derived from f ( N 0 ( x )) , the set of features of the uncorrupted version of x .
This is because distinct choices of x may lead to the same features f ( N 0 ( x )) , but f may vary more in the neighborhood of one choice of x that the other.
An example is helpful to illustrate this point.
Let X = ˜
X
= R 2 , let Z
= R , let N ε ( x ) be the closed (cid:96) 2 ball of radius ε around x , and let f ( x 0 , x 1 ) =
arctan( x 1 /x 0 ) for x (cid:54) =
(the value that we pick for f (0 , x 1 ) is irrelevant).
In other words, f ﬁnds the angle between the horizontal axis and the line containing x .
The range of values that the adversary can cause f (˜ x ) to take depends on (cid:107) x (cid:107) .
If (cid:107) x (cid:107) 2 < ε , f (˜ x ) can be any angle from 0 to π , and if (cid:107) x (cid:107) 2 ≥ ε , | f
(˜ x ) − f ( x ) | ≤ arcsin( ε/ (cid:107) x (cid:107) 2 ) .
This example also illustrates that ε f is not necessarily a metric, even in cases where d is a metric.
Given u, v ∈ R 2 , we have ε f ( u, αu ) = 0 , ε f ( u, αu ) = 0 , and ε f ( αu, αv ) can be made arbitrarily small by selecting α to be small.
Despite this, ε ( u, v ) will be on the order of min( (cid:107) u (cid:107) , (cid:107) v (cid:107) ) .
An alternative approach to studying the induced distance is to deﬁne an adversarial classiﬁcation problem on Z by taking N Z ε ( z ) =
f ( N ε ( f − 1 ( { z } ))) .
Note that this construction only makes sense when X =
˜
X and thus the domain of f is X .
This is more conservative: for any feature point z it considers the worst-case x with feature z : the x in whose neighborhood f varies the most.
C D ESCENT ALGORITHM FOR MULTIPLE LAYER NETWORKS
We start by repeating the notion used in Section 3.2: f = f ( (cid:96) ) ◦ . . .
◦ f (1) where the layer i function is f ( i ) :
R n i → R n i +1 , f ( i ) ( z ) =
( k ( i ) + L ( i ) z )
+ .
Then ε f ( u, v ) is the value of the following optimization problem: min ε subject to δ ∈ ε B ,
[Equation]
Using the local linear approximation, the equality constraint becomes
[Equation]
[Equation]
must hold in order for the linear approximation to be valid.
Any collision must be in a polytope with s ( (cid:96) )
=
s (cid:48) ( (cid:96) ) , which is why we only needed one set of s variables in the single layer case.
However,
ReLU activation variables for earlier layers cannot be merged.
The main modiﬁcation to algorithm is to the process of changing the s variables after each iteration.
The variables for all but the ﬁnal layer are allowed to change independently, while the variables in the ﬁnal layer are changed together following the same rule as in the single layer algorithm.
Input: u, v ∈ R n 1 , L ∈ R n 2 × n 1 , k ∈ R n
Output: ε ∈ R , δ, δ (cid:48)
∈ R n1: z ← k + 12 L ( u + v ) , ε ← 12 (cid:107) u − v (cid:107) 2: for 0 ≤ j < n 2 do 3: s j ← 1 ( z j > 0) , 4: end for 5: repeat 6: ε old ← ε 7: ( ε, δ, δ (cid:48) , z, z (cid:48) )
← ConeLP ( L (1) . . .
L ( (cid:96) ) , k (1) . . .
k ( (cid:96) ) , u, v, s (1) . . .
s ( (cid:96) ) , s (cid:48) (1) . . .
s (cid:48) ( (cid:96) − 1) ) 8: s old ← s 9: for 1 ≤ i ≤ (cid:96) − 1 do 10: for 0 ≤ j < n i +1 do 11: if z ( i ) j > 0 then 12: s ( i )
j ← 1 − s ( i ) j 13: end if 14: if z (cid:48)
( i ) j > 0 then 15: s (cid:48)
( i )
j ← 1 − s (cid:48)
( i ) j 16: end if 17: end for 18: end for 19: for 0 ≤ j < n (cid:96) +1 do 20: if z ( (cid:96) )
j > 0 and z (cid:48) ( (cid:96) )
j > 0 then 21: s ( (cid:96) )
j ← 1 − s ( (cid:96) )
j 22: end if 23: end for 24: until s old = s or ε old = ε
D C ONVERTING COLLISION FINDING TO A LINEAR CONE PROGRAM
In this section, we demonstrate how the collision ﬁnding problem after one linear and one ReLU layer can be cast as a linear cone program.
We have a network with ﬁrst layer v (cid:55)→
Lv + k where L ∈ R n 1 × n 0 and k ∈ R n 1 .
Given a pair of points ( v (cid:48) , v (cid:48)(cid:48) )
∈ R 2 n 0 we would like to search over the space of ( δ (cid:48) , δ (cid:48)(cid:48) )
∈ R 2 nsuch that ( L ( v (cid:48)
+ δ (cid:48) )
− k )
+ =
( L ( v (cid:48)(cid:48)
+ δ (cid:48)(cid:48) )
− k )
+ .
This space is always nonempty because we can take v (cid:48)
+ δ (cid:48)
= v (cid:48)(cid:48)
+ δ (cid:48)(cid:48)
= ( v (cid:48)
+ v (cid:48)(cid:48) ) / 2 .
Let S ⊆
[ n 1 ] be the subset of active ReLUs.
Let F ∈ R | S |× n 1 be the inclusion indicator matrix for the subset: F i,j = 1 if
and only if j ∈ S and | S ∩
[ j ] | =
i (there are exactly i elements of S strictly smaller than j , so j is the i + 1 st element of S ).
Let D be the diagonal matrix with D j,j = 1 for j ∈ S and D j,j = − 1 for j (cid:54)∈ S .
For j ∈ S (the active ReLUs), we need the constraint ( L ( v (cid:48)
+ δ (cid:48) )
+ k )
j = ( L ( v (cid:48)
+ δ (cid:48) )
+ k ) j .
In matrix form, this is FL ( v (cid:48)
+ δ (cid:48) )
=
FL ( v (cid:48)(cid:48)
+ δ (cid:48)(cid:48) ) ,
For j ∈ S , we need ( L ( v (cid:48)
+ δ (cid:48) )
+ k ) j ≥ 0 , ( L ( v (cid:48)(cid:48)
+ δ (cid:48)(cid:48) )
+ k ) j ≥ 0 , and for j (cid:54)∈ S , we need ( L ( v (cid:48)
+ δ (cid:48) )
+ k ) j ≤ 0 , ( L ( v (cid:48)(cid:48)
+ δ (cid:48)(cid:48) )
+ k ) j ≤ 0 .
In matrix form, these are D ( L ( v (cid:48)
+ δ (cid:48) )
+ k ) ≥ 0 ,
D ( L ( v (cid:48)(cid:48)
+ δ (cid:48)(cid:48) )
+ k ) ≥ 0 .
Our objective is max( (cid:107) δ (cid:48) (cid:107) 2 , (cid:107) δ (cid:48)(cid:48)
(cid:107) 2 ) .
We will replace with a linear objective by adding two additional variables ( t (cid:48) , t (cid:48)(cid:48) ) satisfying t (cid:48) ≥ (cid:107) δ (cid:48) (cid:107) 2 and t (cid:48)(cid:48) ≥ (cid:107) δ (cid:48)(cid:48) (cid:107) 2 .
The cvx-opt system is minimize c T x subject to Gx + s = h
Ax = b s (cid:23)
The cones involved are a nonnegative orthant of dimension 2 n 1 and two second order cones, each of dimension n 0 + 1 .
Thus s =
( D ( L ( v (cid:48) + δ (cid:48) )
+ k )
, D ( L ( v (cid:48)(cid:48)
+ δ (cid:48)(cid:48) )
+ k ) , t (cid:48) , δ (cid:48) , t (cid:48)(cid:48) , δ (cid:48)(cid:48) ) .
We then let x =
( t, δ (cid:48) , δ (cid:48)(cid:48) ) .
The matrix relating s and x is G ∈ R ( n 1 + n 1 +1+ n 0 +1+ n 0 ) × (1+ n 0 + n 0 ) and s = h − Gx .
The block structure is
G =   − DL   − DL −   − I −    − I 
 h =  
D ( Lv (cid:48) + k ) D ( Lv (cid:48)(cid:48) + k )  
We use Ax = b to encode FL ( δ (cid:48) − δ (cid:48)(cid:48) )
= − FL
( v (cid:48) − v (cid:48)(cid:48) ) , so A ∈ R | S |× (1+ n 0 + n 0 ) , b ∈ R | S | , and
A = (0 FL − FL ) b
= − FL ( v (cid:48) − v (cid:48)(cid:48) ) .
Finally, c = (cid:32) (cid:33) .
D. F ACTORIZED L
If n 1 < n 0 , then we can take advantage of the fact that the rank of L is at most n 1 .
Let L = U Σ V T .
Replace L ( v + δ ) with U Σ( V T v + (cid:15) ) , n 0 with size of Σ .
New G and A :
[Equation]
Also, the length of c is reduced.
The constant vectors h and b are unchanged.
E F URTHER EXPERIMENTAL DETAILS
We consider the following two architectures for our experiments:
1. 3-layer fully connected neural network (FCNN): 300 FC-ReLU-200 FC-ReLU-2 FC 2.
4-layer convolutional neural network (CNN): 20 × 5 × 5 conv.-BN-ReLU- 2 × 2 MaxPool50 × 5 × 5 conv.-BN-ReLU- 2 × 2 MaxPool- 500 FC - 2 FC
We also construct a ‘Small’ and ‘Smaller’ version of the FCNN with layers that are 2 × and 4 × narrower respectively.
[Figure]
[Figure]
In Figure 6, we compare the robustness of representations obtained from fully connected networks with decreasing layer sizes.
The ‘Regular’ network is the one used throughout, while the ‘Small’ and ‘Smaller’ networks have corresponding layers that are 2 × and 4 × narrower respectively. We can clearly see that as the width of the feature extractor decreases, so does its robustness. F. 0 − 1 L OSS RESULTS
In Figures 7, 8, 9 and 10 we provide lower bounds on the 0 − 1 loss in the same settings as those considered in the main body of the paper.
We note that the results and conclusions remain the same qualitatively.
F. L INEAR CONVOLUTIONAL LAYERS
In Figure 11, we can see that the representations extracted from the ﬁrst linear layer of a convolutional network do not have any negative impact on the robustness of the overall model.
[Figure]
[Figure] [Figure] [Figure]
the ﬁrst linear layer of a 4-layer CNN using different training procedures.