Rethinking Lipschitz Neural Networks for Certified L-infinity Robustness
Anonymous Author(s)
AffiliationAddress email
Abstract
Designing neural networks with bounded Lipschitz constant is a promising wayto obtain certifiably robust classifiers against adversarial examples.
However, therelevant progress for the important ℓ ∞ perturbation setting is rather limited, and aprincipled understanding of how to design expressive ℓ ∞ Lipschitz networks is stilllacking.
In this paper, we bridge the gap by studying certified ℓ ∞ robustness froma novel perspective of representing Boolean functions.
We derive two fundamentalimpossibility results that hold for any standard Lipschitz network: one for robustclassification on finite datasets, and the other for Lipschitz function approximation.
These results identify that networks built upon norm-bounded affine layers and Lipschitz activations intrinsically lose expressive power even in the two-dimensionalcase, and shed light on how recently proposed Lipschitz networks (e.g., GroupSortand ℓ ∞ -distance nets) bypass these impossibilities by leveraging order statisticfunctions .
Finally, based on these insights, we develop a unified Lipschitz networkthat generalizes prior works, and design a practical version that can be efficientlytrained (making certified robust training free).
Extensive experiments show that ourapproach is scalable, efficient, and consistently yields better certified robustnessacross multiple datasets and perturbation radii than prior Lipschitz networks.
Introduction
Modern neural networks, despite their great success in various applications [22, 13], typically sufferfrom a severe drawback of lacking adversarial robustness.
In classification tasks, given an image xcorrectly classified by a neural network, there often exists a small adversarial perturbation δ , makingthe perturbed image x + δ look indistinguishable to humans but fool the network to predict a wrongclass with high confidence [56, 4].
It is well-known that the adversarial robustness of a neural network is closely related to its Lipschitzcontinuity [8, 60] (see Section 3.1).
Accordingly, training neural networks with bounded Lipschitzconstant has been considered a promising way to address the problem.
A variety of works studied
Lipschitz architectures for the ordinary Euclidean norm [60, 59, 36, 53], and recent works evenestablished state-of-the-art (deterministic) certified ℓ 2 robustness [54, 40].
However, when it comesto the more critical (realistic) ℓ ∞ perturbation setting, the progress seems to be rather limited.
In fact,standard Lipschitz ReLU networks have been shown to perform poorly in terms of ℓ ∞ robustness[60, 24, 2].
While other more advanced Lipschitz networks have been proposed [2, 10], the achievedresults are still not satisfactory even on simple datasets like MNIST.
Until recently, Zhang et al.[72, 73] designed a quite unusual Lipschitz network based on a heuristic choice of the ℓ ∞ -distancefunction , which surprisingly established state-of-the-art certified ℓ ∞ robustness on multiple datasetsover prior works.
Yet, it remains unclear why previous Lipschitz networks typically failed, and whatis the essential reason behind the success of this particular ℓ ∞ -distance network structure.
Theoretical contributions .
In this work, we systematically investigate how to design expressive
Lipschitz neural networks (w.r.t.
ℓ ∞ -norm) through the novel lens of representing discrete Booleanfunctions, which provides a deep understanding on the aforementioned problems.
Specifically, wefirst figure out a fundamental limitation of standard Lipschitz networks in representing a class oflogical operations called symmetric Boolean functions (SBF), which comprises the basic logical
AND/OR as special cases.
We prove that for any non-constant SBF of d variables, there exists a finitedataset of size O ( d ) such that the certified ℓ ∞ robust radius must vanish as O (1 /d ) for any classifierinduced by a standard Lipschitz network.
Remarkably, since logical AND/OR operations correspondto perhaps the most basic classifiers, our result indicates an intrinsic difficulty of such networks infitting high-dimensional real-world datasets with guaranteed certified ℓ ∞ robustness.
Our analysis can be readily extended into the Lipschitz function approximation setting.
We point outthe relationship between monotonic SBF and the order statistics (which are 1-Lipschitz functions),and then prove that any d -dimensional order statistic (including the max/min function) on a compactdomain cannot be approximated by standard Lipschitz networks with error O (1 − 1 /d ) , regardlessof the network size.
This impossibility result is significant in that: (i) it applies to all Lipschitzactivations (thus extending prior works [2, 24]), (ii) it resolves an open problem raised recently in[43], and (iii) a quantitative lower bound of approximation error is established.
Equipped by the above impossibility results, we proceed to examine two advanced Lipschitz architectures: the GroupSort network [2] and the recently proposed ℓ ∞ -distance net [72, 73].
We find thatbesides the linear operation, both networks incorporate other Lipschitz aggregation operations intothe neuron design, especially the order statistic functions, thus shedding light on how they work.
However, for the MaxMin network [2] — a computationally efficient version of the GroupSort networkimplemented in practice, representing Boolean functions / order statistics is possible only when thenetwork is very deep .
In particular, we prove that representing certain d -dimensional Boolean functions requires a depth of Ω( d ) , implying that shallow MaxMin networks are not Lipschitz-universalfunction approximators.
In contrast, we show a two-layer ℓ ∞ -distance net suffices to represent anyorder statistic function on a compact domain or even all Boolean functions.
This strongly justifies theempirical success of ℓ ∞ -distance net over GroupSort (MaxMin) networks.
Practical contributions .
Our theoretical insights can also guide in designing better Lipschitz networkarchitectures.
Inspired by the importance of order statistics, we propose a general form of Lipschitznetwork, called SortNet, that extends both GroupSort and ℓ ∞ distance networks and incorporatesthem into a unified framework.
Yet, the full-sort operation is computationally expensive and leads tooptimization difficulties (as with the GroupSort network).
We further propose a specialized SortNetthat can be efficiently trained, by assigning each weight vector w using geometric series , i.e. w iproportional to ρ i for some 0 ≤ ρ < 1 .
This leads to a restricted version of SortNet but still coversℓ ∞ -distance net as a special case.
For this particular SortNet, we skillfully derive a stochasticestimation that gives an unbiased approximation of the neuron output without performing sortingoperations explicitly.
This eventually yields an efficient training strategy with similar cost as trainingstandard networks, thus making certified robust training free.
Extensive experiments demonstrate thatthe proposed SortNet is scalable, efficient, and consistently achieves better certified robustness thanprior Lipschitz networks across multiple datasets and perturbation radii.
In particular, our approacheven scales on a variant of ImageNet, and surpasses the best-known result [68] with a 22-fold decreasein training time thanks to our “free” certified training approach.
The contribution and organization of this paper can be summarized as follows:
• We develop a systematic study for the expressive power of Lipschitz neural networks usingthe tools of Boolean function theory.
We prove the impossibility results of standard Lipschitznetworks in two settings: a) certified ℓ ∞ robustness on discrete datasets (Section 3.2); b)Lipschitz function approximation (Section 3.3).• We provide insights into how recently proposed networks can bypass the impossibility results.
In particular, we show that a two-layer ℓ ∞ -distance net can precisely represent any Booleanfunctions, while shallow GroupSort networks cannot (Section 3.4).• We propose SortNet, a Lipschitz network that generalizes GroupSort and ℓ ∞ -distance net.
Fora special type of SortNet, we derive a stochastic training approach that bypasses the difficultiesin calculating sorting operations explicitly and makes certified training free (Section 4).• Extensive experiments demonstrate that SortNet exhibits better certified robustness on severalbenchmark datasets over baseline methods with high training efficiency (Section 5).
Related Work
Extensive studies have been devoted to developing neural networks with certified robustness guarantees.
Existing approaches can be mainly divided into the following three categories.
Certified defenses for standard networks .
A variety of works focus on establishing certifiedrobustness for standard neural networks.
However, exactly calculating the certified radius of astandard ReLU network is known to be NP-hard [28].
Researchers thus developed a class ofrelaxation-based approaches that provide a tight lower bound estimate of the certified robustnessefficiently.
These approaches typically use convex relaxation to calculate a bound of the neuronoutputs under input perturbations layer by layer [64, 65, 63, 52, 42, 19, 61, 75].
See also [3, 15,16, 47, 48, 67, 11, 35, 62, 23] for more advanced approaches.
However, most of these works sufferfrom high computational costs and are hard to scale up to large datasets.
Currently, the only scalableconvex relaxation approach is based on interval bound propagation (IBP) [42, 21, 74, 68, 51], butthe produced bound is known to be loose [50], and a recent study showed that IBP cannot achieveenough certified robustness on simple datasets for any standard ReLU network [41].
Certified defenses using Lipschitz networks .
On the other hand, Lipschitz networks inherentlyimply certified robustness, resulting in a much simpler certification process based on the output margin(see Proposition 3.1).
Yet, most prior works can only handle the ℓ 2 -norm Lipschitz situation byleveraging specific mathematical properties such as the spectral norm [8, 70, 20, 60, 17, 46, 2, 36, 40]or orthogonality of weight matrices [39, 59, 53, 54].
For the ℓ ∞ -norm, standard Lipschitz networkswere shown to give only a vanishingly small certified radius [60].
Huster et al.
[24] found thatstandard Lipschitz ReLU networks cannot represent certain simple functions such as the absolutevalue, which inspired the first expressive Lipschitz architecture called the GroupSort network [2].
Since then, GroupSort has been extensively investigated [10, 57], but its performance is still muchworse than the above relaxation-based approaches even on MNIST.
Recently, Zhang et al.
[72, 73]first proposed a practical 1-Lipschitz architecture w.r.t.
ℓ ∞ -norm based on a special neuron called theℓ ∞ -distance neuron, which can scale to TinyImageNet with state-of-the-art certified robustness overrelaxation-based approaches.
However, despite its practical success, it is rather puzzling how such asimple architecture can work while prior approaches all failed.
Answering this question may requirean in-depth re-examination of Lipschitz networks (w.r.t.
ℓ ∞ -norm), which is the focus of this paper.
Certified defenses via randomized smoothing .
As a rather different and parallel research line,randomized smoothing typically provides probabilistic certified ℓ 2 robustness guarantees.
Due tothe wide applicability, randomized smoothing has been scaled up to ImageNet and achieves state-ofthe-art certified accuracy for ℓ 2 perturbations [34, 38, 9, 49, 71, 27].
However, certifying robustnesswith high probability requires sampling a large number of noisy inputs (e.g., 10 5 ) for a single image,leading to a high computational cost at inference.
Moreover, theoretical results pointed out that itcannot achieve non-trivial certified ℓ ∞ robustness if the perturbation radius is larger than Ω( d − 1 / 2 )where d is the input dimension [69, 5, 30, 66].
The Expressive Power of Lipschitz Neural Networks
3.1 Preliminaries
Notations .
We use boldface letters to denote vectors (e.g., x ) or vector functions (e.g., f ), and usex i (or f i ) to denote its i -th element.
For a unary function σ , σ ( x ) applies σ ( · ) element-wise onvector x .
The ℓ p -norm ( p ≥ 1 ) and ℓ ∞ -norm of a vector x are defined as ∥ x
∥ p = ( (cid:80) i | x i | p ) 1 /p and∥ x ∥ ∞ =
max i | x i | , respectively.
The matrix ∞ -norm is defined as ∥ W ∥ ∞ =
max i
∥ W i, : ∥ 1 where
W i, : is the i -th row of the matrix W .
The k -th largest element of a vector x is denoted as x ( k ) .
Weuse [ n ] to denote the set { 1 , · · · , n } , and use e i to denote the unit vector with the i -th element beingone.
We adopt the big O notations by using O ( · ) , Ω( · ) , and Θ( · ) to hide universal constants.
Lipschitzness .
A mapping f : R n → R m is said to be L -Lipschitz continuous w.r.t.
norm ∥ · ∥ if forany pair of inputs x 1 , x 2 ∈ R n ,
[Equation]
If the mapping f represented by a neural network has a small Lipschitz constant L , then (1) impliesthat the change of network output can be strictly controlled under input perturbations, resulting incertified robustness guarantees as shown in the following proposition.
Proposition 3.1.
(Certified robustness of Lipschitz networks)
For a neural network f with Lipschitzconstant L under ℓ p -norm ∥ · ∥ p , define the resulting classifier g as g ( x ) :
=
arg max k f k ( x ) for aninput x .
Then g is provably robust under perturbations ∥
δ ∥ p < cL margin( f ( x )) , i.e.g ( x + δ ) =
g ( x ) for all δ with ∥ δ ∥
p < c/L · margin( f ( x )) .
Here c = p √ 2 / 2 is a constant depending only on the norm ∥ · ∥ p , which is 1 / 2 for the ℓ ∞ -norm, andmargin( f ( x )) is the margin between the largest and second largest output logits.
The proof of Proposition 3.1 is simple and can be found in Appendix B.1 or [39, Appendix P].
It canbe seen that the robust radius is inversely proportional to the Lipschitz constant L .
Standard Lipschitz networks .
Throughout this paper, we refer to standard neural networks as neuralnetworks formed by affine layers (e.g., fully-connected or convolutional layers) and element-wiseactivation functions.
Based on the Lipschitz property of composite functions, most prior worksenforce the 1-Lipschitzness of a multi-layer neural network by constraining each layer to be a 1Lipschitz mapping.
For the ℓ ∞ -norm, it is further equivalent to constraining the weight matrices tohave bounded ∞ -norm, plus using Lipschitz activation functions [2], which can be formalized asx ( l )
= σ ( l ) ( W ( l ) x
( l − 1)
+ b ( l ) )
s.t.
∥ W ( l ) ∥ ∞ ≤ 1 and σ ( l ) being 1-Lipschitz , l ∈ [ M ] .
Here M is the number of layers and usually σ ( M ) ( x ) = x is the identity function.
The network takesx (0) :
= x as the input and outputs x ( M ) .
For K -class classification problems, x ( M ) ∈ R K andthe network predicts the class g ( x ) :
= arg max k ∈
[ K ] x ( M ) k .
We refer to the resulting network as astandard Lipschitz network.
3.2 Certified robustness on discrete Boolean datasets
In this section, we will construct a class of counterexamples for which certified ℓ ∞ robustness can bearbitrarily poor using standard Lipschitz networks.
We focus on the Boolean dataset , a discrete datasetwhere both inputs and labels are Boolean-valued and the relationship between inputs and their labels( x ( i ) , y ( i ) )
∈ { 0 , 1 } d × { 0 , 1 } can be described using a Boolean function g B : { 0 , 1 } d → { 0 , 1 } .
Thekey motivation lies in the finding that Boolean vectors correspond to the vertices of a d -dimensionalhypercube, and thus are geometrically related to the ℓ ∞ -distance metric.
In particular, the ℓ ∞ -distancebetween any two different data points in a Boolean dataset is always 1, which means that the datasetis well-separated .
This yields the following proposition, stating that it is always possible to achieveoptimal certified ℓ ∞ robustness on Boolean datasets by using Lipschitz classifiers.
Proposition 3.2.
For any Boolean dataset D = { ( x ( i ) , y ( i ) ) } ni =1 , there exists a classifier ˆ g : R d →{ 0 , 1 } induced by a 1-Lipschitz mapping ˆ f : R d → R 2 , such that ˆ g can fit the whole dataset withmargin( ˆ f ( x ( i ) ))
= 1 ∀ i ∈ [ n ] , thus achieving a certified ℓ ∞ radius of 1 / 2 by Proposition 3.1.
The key observation in the proof (Appendix B.2) is that one can construct a so-called nearest neighborclassifier that achieves a large margin on the whole dataset and is 1-Lipschitz .
Based on Proposition3.2, it is natural to ask whether standard Lipschitz networks of the form (3) can perform well on
Boolean datasets.
Unfortunately, we show it is not the case, even on a class of simple datasetsconstructed using symmetric Boolean functions.
Definition 3.3.
A Boolean function g B : { 0 , 1 } d → { 0 , 1 } is symmetric if it is invariant under inputpermutation, i.e. g B ( x 1 , · · · , x d ) =
g B ( x π (1) , · · · , x π ( d ) ) for any x ∈ { 0 , 1 } d and π ∈ S d .
Example 3.4.
Two of the most basic operations in Boolean algebra are the logical AND/OR, bothof which belong to the class of symmetric Boolean functions.
Other important examples includethe exclusive-or (XOR, also called the parity function), NAND, NOR, and the majority function (orgenerally, the threshold functions).
See Appendix A.2 for a detailed description of these examples.
Theorem 3.5.
For any non-constant symmetric Boolean function g B : { 0 , 1 } d → { 0 , 1 } , there existsa Boolean dataset with labels y ( i ) =
g B ( x ( i ) ) , such that no standard Lipschitz network can achieve acertified ℓ ∞ robust radius larger than 1 / 2 d on the dataset.
Implications .
Theorem 3.5 shows that the certified radius of standard Lipschitz networks must vanishas dimension d grows, which is in stark contrast to the constant radius given by Proposition 3.2.
Also, note that the logical AND/OR functions are perhaps the most basic classifiers (which simplymake predictions based on the existence of binary input features).
It is thus not surprising to see thatstandard Lipschitz networks perform poorly on real-world datasets (e.g., even the simple MNISTdataset where input pixels are almost Boolean-valued (black/white) [60]).
We give an elegant proof of Theorem 3.5 in Appendix B.3, where we also prove that the bound of1 / 2 d is tight in Proposition B.8.
Moreover, we discuss the sample complexity by proving that adataset of size O ( d ) already suffices to give O (1 /d ) certified radius (Corollary B.10).
To the best ofour knowledge, Theorem 3.5 is the first impossibility result that targets the certified ℓ ∞ robustnessof standard Lipschitz networks with an quantitative upper bound on the certified radius.
In the nextsection, we will extend our analysis to the function approximation setting and make discussions withliterature results [24, 2].
3.3 Lipschitz function approximation
Classic approximation theory has shown that standard neural networks are universal function approximators [12, 37], in that they can approximate any continuous function on a compact domain arbitrarilywell.
For 1-Lipschitz neural networks, an analogous question is whether they can approximate all1-Lipschitz functions accordingly.
Unfortunately, the result in Section 3.2 already implies a negativeanswer.
Indeed, by combining Proposition 3.2 and Theorem 3.5, ˆ f is clearly a 1-Lipschitz functionthat cannot be approximated by any standard Lipschitz network.
To gain further insights into the structure of unrepresentable 1-Lipschitz functions, let us consider thecontinuousization of discrete Boolean functions.
For the symmetric case, one needs to find a class of1-Lipschitz continuous functions that are also invariant under permutations.
It can be found that asimple class of candidates is the order statistics , i.e. the k -th largest element of a vector.
One cancheck that the k -th order statistic x ( k ) is indeed 1-Lipschitz and is precisely the continuousization ofthe k -threshold Boolean function defined as g B ,k ( x ) :
= I ( (cid:80) i x
i ≥ k ) .
In particular, x (1) =
max i x iand x ( d ) =
min
i x i corresponds to the logical OR/AND functions, respectively.
Importantly, note thatany Boolean function that is both symmetric and monotonic is a k -threshold function, and vice versa(Appendix A.2).
Therefore, k -threshold functions can be regarded as the most elementary Booleanfunctions, suggesting that the ability to express order statistics is necessary for neural networks.
Unfortunately, using a similar analysis as the previous section, we have the following theorem:
Theorem 3.6.
Any standard Lipschitz network f : R d → R cannot approximate the simple 1Lipschitz function x → x ( k ) for arbitrary k ∈ [ d ] on a bounded domain K
= [0 , 1] d if d ≥ 2 .
Moreover, there exists a point (cid:98) x ∈ K , such that
[Equation]
We give a proof in Appendix B.4.
The above theorem indicates that order statistics cannot be uniformlyapproximated using any standard Lipschitz network regardless of the network size.
Moreover, notethat the trivial constant function ˜ f
( x ) = 1 / 2 already achieves an approximation error of 1 / 2uniformly on K , implying that standard Lipschitz networks can hardly improve upon trivial solutions.
Remark 3.7.
Theorem 3.6 can be easily generalized to weaker forms of non-uniform approximation,e.g., using the ℓ p -norm as distance metrics [45], by proving that there exists a hypercube B ∞ ( (cid:98) x )centered at (cid:98) x with length Θ(1) , such that | f (˜ x ) − ˜ x
( k ) | ≥ Θ(1) holds for all ˜ x ∈ B ∞ ( (cid:98) x ) whend ≥ 2 .
See Corollary B.13 for details.
Discussion with prior works [2, 24, 43].
The work of Anil et al.
also gave negative results on theexpressive power of standard Lipschitz networks 1 [2, Theorem 1].
They proved a different (weaker)version of Theorem 3.6, showing that if the activation function σ is monotonic , the network cannotprecisely represent non-linear 1-Lipschitz functions whose gradient norm is 1 almost everywhere (e.g.,the absolute value function proved before by [24]).
They did not give a quantitative approximationerror.
The intuition is that any monotonic non-linear 1-Lipschitz activation (e.g., ReLU) must haveregions with slopes less than 1, leading to gradient attenuation during backpropagation.
The authorsthus attributed the reason to the activation function, which is not gradient-norm preserving (GNP).
However, such an explanation is still not fully satisfactory, as GNP can be simply achieved using anon-monotonic activation (e.g., σ ( x ) =
| x | ).
Consequently, one may expect that a standard Lipschitznetwork build on a suitable (non-monotonic) activation function can have sufficient expressivepower.
Such an idea is recently explored in [43], where the authors proved that using a general1-Lipschitz piecewise linear activation with 3 linear regions, the corresponding network achievesthe maximum expressive power compared with other Lipschitz activations and can approximate anyone-dimensional 1-Lipschitz function.
They pose the high dimension setting as an open problem.
Unfortunately, Theorem 3.6 addressed the open problem with negative answer, stating that suchnetworks are not expressive even for the two-dimensional setting.
It also implies that GNP is notsufficient to explain the failure of standard Lipschitz networks.
Instead, we draw a rather differentconclusion, arguing that the lack of expressiveness is due to the inability of weight-constrained affinetransformations to perform basic Boolean operations (even in two dimensions).
A further justificationis given in Section 3.4.
Finally, compared with Anil et al.
[2], the form of Theorem 3.6 is morefundamental, in the sense that it does not make assumptions on the activation function, and it gives aquantitative error bound on the approximation that is arbitrarily close to the plain fit f ( x ) = 1 / 2 .
3.4 Investigating more advanced Lipschitz networks
Seeing the above impossibility results, we then examine two representative works of (non-standard)
Lipschitz networks in literature: the GroupSort network [2] and the recently proposed ℓ ∞ -distancenet [72, 73].
Notably, both networks are Lipschitz-universal function approximators and thus fullyexpressive.
The GroupSort network makes minimum changes to standard Lipschitz networks (3), byreplacing element-wise activation σ with GroupSort layers.
GroupSort partitions the input vectorinto groups, sorts the sub-vector of each group in descending order, and finally concatenates theresulting sub-vectors.
Since sorting is computationally expensive, the authors considered a practicalversion of GroupSort with a group size of 2, called MaxMin, which simply calculates the maximumand minimum pair by pair [6].
ℓ ∞ -distance net, on the other hand, is fundamentally different fromstandard Lipschitz networks.
Each neuron in an ℓ ∞ -distance net is designed based on the ℓ ∞ -distancefunction y =
∥ x − w ∥ ∞ + b (with parameters { w , b } ).
Despite the somewhat unusual structure,ℓ ∞ -distance net has been shown to substantially outperform GroupSort (MaxMin) in terms of certifiedℓ ∞ robustness according to [72], a puzzling thing to be understood.
We provide an possible explanation for this.
We find that both networks incorporate order statisticsinto the neuron design, either explicitly (GroupSort) or implicitly ( ℓ ∞ -distance net), thus bypassingthe impossibility result in Theorem 3.6.
Indeed, the sorting operations in GroupSort explicitlycalculate order statistics.
As for ℓ ∞ -distance net, we show its basic neuron can implicitly representthe max function on a bounded domain, by assigning the weight w = − c 1 and the bias b
= − c witha sufficiently large constant c :
[Equation]
and thus can represent the logical OR operation.
In general, we have the following theorem:
Theorem 3.8.
A two-layer ℓ ∞ -distance net can exactly represent the following functions: (i) anydiscrete Boolean function; (ii) any continuous order-statistic function on a compact domain.
We give a proof in Appendix B.5.
Our proof uses the fundamental result in Boolean algebra thatany Boolean function can be written in its disjunctive normal form (DNF, see Appendix A.1), whichcan be further reduced to using only the composition of logical OR operations of literals and thusbe realized by a two-layer ℓ ∞ -distance net.
To represent order statistics, we formulate them asnested max-min functions, which can also be realized by a two-layer ℓ ∞ -distance net.
Therefore, theconstruction in our proof provides a novel understanding of the mechanism behind the success ofℓ ∞ -distance nets, since each ℓ ∞ -distance neuron can be regarded as a basic “logical gate” and thewhole network can simulate any Boolean circuit .
For GroupSort networks with a group size G ≥ d , a similar result holds.
However, it is not the casefor practically used MaxMin networks ( G = 2 ), where we have the following impossibility results:
Theorem 3.9.
An M -layer MaxMin network f : R d → R cannot approximate any k -th order statisticfunction on a bounded domain K = [0 , 1] d if M ≤ ⌈ log 2 d ⌉ (no matter how wide the network is).
Moreover, there exists a point (cid:98) x ∈ K , such that
[Equation]
Theorem 3.10.
Let M d be the minimum depth such that an M d -layer MaxMin network can representany (discrete) d -dimensional Boolean function.
Then M d = Ω( d ) .
The above theorems show that MaxMin networks must be very deep in order to represent Booleanfunctions and order statistics, which is in stark contrast to Theorem 3.8, as a constant depth issufficient for ℓ ∞ -distance nets.
Based on Theorem 3.10, we directly have the following corollary:
Corollary 3.11.
The function class induced by M d -layer MaxMin networks is not a universalapproximator to the d -dimensional 1-Lipschitz functions if M d = o ( d ) .
The proofs of Theorems 3.9 and 3.10 are deferred to Appendix B.6 and B.7, respectively.
In particular,the proof of Theorem 3.10 is non-trivial and makes elegant use of Boolean circuit theory, so wepresent a proof sketch here.
The key insight is that for any Boolean function, if it can be expressedby some MaxMin network f , then it can be expressed by a special MaxMin network with the sametopology as f such that all the weight vectors w are sparse with at most one non-zero element, either1 or − 1 (Corollary B.23).
This implies that weight vectors have no use in representing Booleanfunctions and thus MaxMin networks reduce to 2-ary Boolean circuits , i.e. directed acyclic graphswhose internal nodes are logical gates including NOT and the 2-ary AND/OR.
Note that for a 2-ary
Boolean circuit that has M layers and outputs a scalar, the number of nodes will not exceed 2 M +1 − 1(achieved by a complete binary tree).
However, the classic result in Boolean circuit theory (Shannon1942) showed that for most Boolean functions of d variables, a lower bound on the minimum size of2-ary Boolean circuits is Ω(2 d /d ) , which thus yields M = Ω( d ) and concludes the proof.
In Appendix, we also discuss the tightness of the above theorems.
We prove that a depth of O (log 2 d )is sufficient to represent any order statistic function using Boolean circuit theory (Theorem B.19),and a straightforward construction using DNF shows that a depth of O ( d ) is sufficient to representany Boolean functions (Proposition B.20).
Thus both theorems are tight.
Unfortunately, training deep MaxMin networks is known to be challenging due to optimizationdifficulties [10].
Consequently, prior works only use a shallow MaxMin network with no morethan 4 layers [2, 10], which severely lacks expressive power.
One possible solution is to increasethe group size, and several works explored this aspect using toy examples and observed significantbenefits empirically [10, 57].
However, a large group size involves computationally expensive sortingoperations and makes the network hard to train [2], limiting its value in practice.
A Unified Framework of Lipschitz Neural Networks
The above theoretical results have justified order statistics as a crucial component in representinga class of Boolean functions, shedding light on how GroupSort and ℓ ∞ -distance net work.
Basedon these insights, in this section, we will propose a unified framework of Lipschitz networks thattake the respective advantage of prior Lipschitz architectures, and then give a practical (specialized)version that enables efficient training.
Consider a general Lipschitz network constructed using the following three basic types of 1-Lipschitzoperations: (i) norm-bounded affine transformations, e.g. y
= w T x ( ∥ w ∥ 1 ≤ 1 ) and y
= x + b ;(ii) 1-Lipschitz unary activation functions σ ; (iii) order statistics.
The first two types are extensivelyused in standard Lipschitz networks, while the last type is motivated by Section 3 and is of crucialimportance.
We propose the following network which naturally combines the above components:
Definition 4.1.
(SortNet)
Define an M -layer fully-connected SortNet f as follows.
The networktakes x = x (0) as input, and the k -th unit in the l -th hidden layer x
( l ) k is computed by
[Equation]
where d l is the size of the l -th layer, and sort( x ) :
= ( x (1) , · · · , x ( d ) )
T calculates all order statistics ofx ∈ R d .
The network outputs f ( x ) =
x ( M ) + b out .
Here { w ( l,k ) } , { b ( l,k ) } and b out are parameters.
It is easy to see that SortNet is 1-Lipschitz w.r.t.
ℓ ∞ -norm.
We now show that SortNet is a generalarchitecture that extends both GroupSort and ℓ ∞ -distance networks into a unified framework.
Proposition 4.2.
Any GroupSort network with an arbitrary group size on a compact input domaincan be represented by a SortNet with the same topological structure using activation σ ( x ) = x .
Proposition 4.3.
Any ℓ ∞ -distance net can be represented by a SortNet with the same topologicalstructure by fixing the weights w ( l,k ) =
e 1 and using the absolute-value activation σ ( x ) =
| x | .
See Appendix C for a proof.
As a result, SortNet can exploit the respective advantage of these
Lipschitz networks.
Compared with GroupSort, SortNet can freely use activation functions such asthe absolute value, thus easily addressing the problem claimed in [24, 2].
Moreover, unlike GroupSort,the bias vector b ( l,k ) in SortNet (7) can be assigned diversely for different neurons in the same layer.
In this way, one can control the sorting behavior of each neuron individually by varying the biasvalue without disturbing the output of other neurons, which is very beneficial (see Appendix C.1 fordetails).
Compared with ℓ ∞ -distance net, SortNet adds linear transformation and incorporates allorder statistics (rather than only the maximum), thus can represent certain functions more effectively.
A practical version of SortNet .
As with GroupSort networks, we also design a practical (specialized)version of SortNet which enjoys efficient training.
But different from the MaxMin network thatreduces the group size, we keep the full-dimensional order statistics as they are crucial for theexpressive power (Section 3.4).
The key observation is that in (7), the only required computationis the linear combination of order statistics (i.e. w T sort( · ) ), rather than the entire sorting results(i.e.
sort( · ) ).
We find that for certain carefully designed choices of the weight vector w , there existefficient approximation algorithms that can give a good estimation of w T sort( · ) .
In particular, wepropose an assignment of the weight vector that follows geometric series, i.e. w i proportional to ρ i ,in which case we have the following result:
Proposition 4.4.
Let w ∈ R
d be a vector satisfying w i
=
(1 − ρ )
ρ i − 1 , i ∈ [ d ] for some 0 ≤ ρ < 1 .
Then for any vector x ∈ R d + with non-negative elements,
[Equation]
Here s is a random vector following independent Bernoulli distribution with probability 1 − ρ being1 and ρ being 0.
Proof.
Without loss of generality, assume x 1 , · · · , x d are different from each other.
Denote j 1 , · · · , j das the sorting indices such that sort( x ) =
( x j 1 , · · · , x j d ) .
Then
[Equation]
) =
w T sort( x ) .
It is easy to check that the weight w in the above proposition satisfies ∥ w ∥ 1 ≤ 1 , which guaranteesthe Lipschitzness.
The non-negative condition on x in Proposition 4.4 holds when using a suitableactivation function in neuron (7), such as the absolute value function.
Proposition 4.4 suggests that one can use max i s i x i to give an unbiased estimation of w T sort( x ) .
Inthis way, the expensive sorting operation is avoided and replaced by a max operation, thus significantlyreducing the computational cost in training.
Note that s is a random Bernoulli vector, so the abovecalculation is similar to applying a mask on the input of each neuron, like dropout [55].
It means thatthe introduced stochasticity may further prevent overfitting and benefit generalization performance.
Regarding the value of ρ .
When ρ = 0 , only the maximum value is taken into the computation andthe resulting network can recover the ℓ ∞ -distance net by choosing the activation function σ ( x ) =
| x | .
This means the specialized SortNet still extends ℓ ∞ -distance net and thus has sufficient expressivepower.
When ρ > 0 , all order statistics become utilized.
A simple way of selecting ρ is to regard it asa hyper-parameter and set its value by cross-validation, which is adopted in our experiments.
One canalso consider treating ρ as learnable parameters for each neuron that participate in the optimizationprocess, but this involves calculating the gradient of ρ which may be complicated due to the stochasticsampling procedure (8).
We will leave the study as future work.
Experiments
In this section, we perform extensive empirical evaluations of the proposed SortNet architecture aswell as various prior works in the certified ℓ ∞ robustness area.
To show the scalability of differentapproaches, we consider a variety of benchmark datasets, including MNIST [33], CIFAR-10 [29],
TinyImageNet [32], and ImageNet ( 64 × 64 ) [7].
Due to space limitations, a complete training recipeis given in Appendix E.
Our code and trained models will be released.
SortNet model configuration .
Since SortNet generalizes the ℓ ∞ -distance net, we simply followthe same model configurations as [72] and consider two types of models.
The first one is a simple
SortNet consisting of M fully-connected layers with a hidden size of 5120, which is used in MNISTand CIFAR-10.
Like [72], we choose M = 5 for MNIST and M = 6 for CIFAR-10.
Since SortNet is
Lipschitz, we directly apply the margin-based certification method to calculate the certified accuracy(Proposition 3.1).
To achieve the best results on ImageNet-like datasets, in our second type of modelwe consider using a composite architecture consisting of a base SortNet backbone and a predictionhead (denoted as SortNet+MLP).
Following [72], the SortNet backbone has 5 layers with a width of5120 neurons, which serves as a robust feature extractor.
The top prediction head is a lightweight2-layer perceptron with 512 hidden neurons (or 2048 for ImageNet), which takes the robust features as input to give classification results.
We also try a larger SortNet backbone, denoted as SortNet+MLP(2x), that has roughly four times the training cost (see Appendix E.2 for architectural details).
Weuse the same approach as [72] to train and certify these models, i.e. by combining margin-basedcertification for the SortNet backbone and interval bound propagation for the top MLP [21].
Baseline methods and metrics .
We compare SortNet with representative literature approachesincluding relaxation-based certification (for standard networks), margin-based certification (using
Lipschitz networks), and mixed-integer linear programming (MILP) [58].
In Appendix G, we alsodiscuss randomized smoothing approaches [9, 49], which provide probabilistic guarantees rather thandeterministic ones.
For each method in these tables, we report five metrics: (i) training efficiency,measured by the wall-clock time per training epoch; (ii) certification efficiency, measured by thetime needed to calculate the certified accuracy on the test dataset; (iii) the clean test accuracy withoutperturbation (denoted as Clean); (iv) the robust test accuracy under 100-step PGD attack (denotedas PGD); (v) the certified robust test accuracy (denoted as Certified).
For a fair comparison, wereproduce most of baseline methods using the official codes and report the wall-clock time under thesame NVIDIA-RTX 3090 GPU.
These results are presented in Tables 1, 2 and 3.
Performance on MNIST.
The results are presented in Table 1.
Following the common practice, weconsider both a small perturbation radius ϵ = 0 .
1 and a larger one ϵ = 0 .
It can be seen that the
SortNet models can achieve 98.14% ( ϵ = 0 . 1 ) and 93.40% ( ϵ = 0 . 3 ) certified accuracy, respectively,both of which outperform all previous baseline methods.
In contrast, the GroupSort network can onlyachieve a trivial certified accuracy for ϵ = 0 .
This matches our theory in Section 3.4, indicatingthat the expressive power of shallow MaxMin networks is insufficient in real-world applications.
Performance on CIFAR-10.
The results are presented in Table 2.
Following the common practice,we consider two perturbation radii: ϵ = 2 / 255 and ϵ = 8 / 255 .
Our models can achieve 56.94%( ϵ = 2 / 255 ) and 40.39% ( ϵ = 8 / 255 ) certified accuracy, respectively.
Moreover, the trainingapproach proposed in Section 4 is very efficient, e.g., with a training time of 13 ∼ 14 seconds perepoch.
For both radii, our models perform the best among all existing approaches that can be certifiedin a reasonable time.
Compared with relaxation-based methods, the certified accuracy of SortNetmodels is much higher (typically +3 ∼ +6 point for both radii), despite our training speed beingseveral times faster.
Such results may indicate that certified ℓ ∞ robustness can be better achieved bydesigning suitable Lipschitz models than by devising relaxation procedures for non-Lipschitz models.
Performance on TinyImageNet and ImageNet.
To demonstrate the scalability of SortNet models,we finally run experiments on two large-scale datasets: Tiny-ImageNet and ImageNet ( 64 × 64 ).
Notably, the ImageNet dataset has 1000 classes and contains 1.28 million images for training and50,000 images for testing.
Due to both the large size and the huge number of classes, achievingcertified ℓ ∞ robustness on the ImageNet level has long been a challenging task.
Table 3 presents our results along with existing baselines.
Among them, we achieve 18.18%certified accuracy on TinyImageNet and achieve 9.54% certified accuracy on ImageNet, both ofwhich establish state-of-the-art results.
The gap is most prominent on ImageNet, where our small
SortNet+MLP model already outperforms the largest model of [68] while being 22 times faster totrain.
Even for the largest model (SortNet+MLP 2x), the training is still 7 times faster, resulting in atraining overhead of 4 days using two GPUs.
We suspect that continuing to increase the model sizewill yield better results, given the noticeable improvement of the larger model over the smaller one.
Comparing with ℓ ∞ -distance net .
As can be seen, SortNet models consistently achieve bettercertified accuracy than ℓ ∞ -distance nets for all different datasets and perturbation levels, and theperformance gap is quite prominent compared with the original work [72].
Very recently, a follow-uppaper [73] significantly improved the performance of ℓ ∞ -distance net using a carefully designedtraining strategy, creating a strong baseline on CIFAR-10.
However, the authors did not present theresults on large-scale datasets, and we find their approach does not suit the ImageNet-like datasetswhen the number of classes is large (see Appendix E.6).
In contrast, SortNet models enjoy greatscalability ranging from MNIST to ImageNet and consistently outperform [73].
The improvement isalso remarkable for ϵ = 2 / 255 on CIFAR-10 ( +7 .
and +2 .
82% in clean / certified accuracy).
In Appendix F, we conduct ablation studies on CIFAR-10 by varying the value of ρ and comparing
SortNet models ( ρ > 0 ) with ℓ ∞ -distance net ( ρ = 0 ), under the same training strategy in thispaper without additional tricks.
We observe a large gain in certified accuracy when switching fromℓ ∞ -distance net to general SortNet.
This empirically indicates that incorporating other order statisticshas extra benefits in certified ℓ ∞ robustness than using only the maximum (the first order statistic).
[Table]
[Table] [Table]
Conclusion
In this paper, we study certified ℓ ∞ robustness from the novel perspective of representing Booleanfunctions.
Our analysis points out an inherent problem in the expressive power of standard Lipschitznetworks, and provides novel insights on how recently proposed Lipschitz networks resolve theproblem.
We also answer several previous open problems, such as (i) the expressive power ofstandard Lipschitz networks with general activations [43] and (ii) the lower bound on the depth of
MaxMin networks to become universal approximators [57, 43].
Finally, guided by the theoreticalresults, we design a new Lipschitz network with better empirical performance than prior works.
Limitations and open problems .
One major limitation of this work is that we only focus on ℓ ∞robustness.
While such results may shed light on general ℓ p -norm settings when p is large, it does notapply to the standard ℓ 2 -norm.
In particular, in this case MaxMin is equivalent to the absolute valuefunction in terms of expressive power [2], and it remains a fantastic open problem whether certain
Lipschitz networks with bounded matrix 2-norm are universal approximators.
For the ℓ ∞ -norm case,providing a tight upper bound on the depth of MaxMin networks to achieve universal approximationis also open.
Another limitation is that our results apply only for standard Lipschitz networks.
Whenthe Lipschitz constant is constrained using carefully designed bounding methods [47, 18, 31] (ratherthan a simple stacking of 1-Lipschitz layers), the robustness certification will be less efficient, but theresulting networks are likely to bypass the impossibility results in this paper.
On the application side,it is interesting to study how to design efficient training approaches of the general SortNet modelswith learnable weights or learnable ρ .
Another meaningful question is how to encode inductive biasesinto these Lipschitz networks (e.g., designing convolutional architectures) to better suit image tasks.
Broader Impact
Interestingly, our theoretical results point out a surprising connection between MaxMin/ ℓ ∞ -distancenetworks and Boolean circuits.
We believe the value of this paper may go beyond the certifiedrobustness community and link to the field of theoretical computer science.
Acknowledgement
We sincerely thank all the anonymous reviewers for the very careful and detailed reviews as well asthe valuable suggestions.
Their help has further enhanced our work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We have discussed the limitationsof this work in detail.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our
work is only for academic research purpose, without any foreseeable negative societal
impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs are
given either in the main text or in Appendix B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] We givecomplete implementation details in Appendix E.
Our code and models will be released once the paper is published.
We use public benchmark datasets which can be
downloaded freely.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix E.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix F.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A] We use public benchmark datasets which can be downloadedfreely.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A] We use public benchmark datasets which canbe downloaded freely.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A] [Table]