Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks
Abstract
Lipschitz constrained networks have gathered considerable attention in the deep learning community, with usages ranging from Wasserstein distance estimation to the training of certiﬁably robust classiﬁers.
However they remain commonly considered as less accurate, and their properties in learning are still not fully understood.
In this paper we clarify the matter: when it comes to classiﬁcation 1-Lipschitz neural networks enjoy several advantages over their unconstrained counterpart.
First, we show that these networks are as accurate as classical ones, and can ﬁt arbitrarily difﬁcult boundaries.
Then, relying on a robustness metric that reﬂects operational needs we characterize the most robust classiﬁer: the WGAN discriminator.
Next, we show that 1-Lipschitz neural networks generalize well under milder assumptions.
Finally, we show that hyper-parameters of the loss are crucial for controlling the accuracy-robustness trade-off.
We conclude that they exhibit appealing properties to pave the way toward provably accurate, and provably robust neural networks.
Introduction
1-Lipschitz neural networks have drawn great attention in the last decade, with motivation ranging from adversarial robustness to Wasserstein distance computation.
In the following, we denote by
LipNet1 the class of 1-Lipschitz neural networks, by AllNet the class of neural networks without constraints on their Lipschitz constant, i.e conventional neural networks.
Roughly speaking the Lipschitz constant of neural networks quantiﬁes how much their outputs can change when inputs are perturbed.
When this constant is high as it is often the case for neural networks of AllNet , they become vulnerable to adversarial attacks (see [1, 2] and references therein): a carefully chosen small noise added to the inputs, usually imperceptible, can change the class prediction.
One possible defense against adversarial attacks is to constrain the network to be 1-Lipschitz (in LipNet1 )
[3], which provides provable robustness guarantees, together with an improvement of generalization [4] and interpretability of the model [5].
LipNet1 networks are also used to estimate Wasserstein distance, thanks to Kantorovich-Rubinstein duality in the seminal work of WGAN [6].
Despite their competitiveness with networks of AllNet on medium scale problems [7, 8], they still suffer from misconceptions.
A belief commonly invoked against networks of LipNet1 is that they are less expressive: “Lipschitz-based approaches suffer from some representational limitations that may 36th Conference on Neural Information Processing Systems (NeurIPS 2022). prevent them from achieving higher levels of performance and being applicable to more complicated problems” [9].
Although this claim seems rational at ﬁrst glance, the link between Lipschitz constant and expressiveness is not trivial.
While there is an obvious lack of expressiveness for regression tasks, this intuition fades when it comes to classiﬁcation.
Indeed, every AllNet network g : R n → R K is L -Lipschitz for some (generally unknown) L > 0 .
Then f = 1 L g is a 1-Lipschitz neural network with the same decision boundary, since prediction arg max k g k is invariant by positive rescaling of the logits.
In particular, f has the same accuracy and also the same robustness to adversarial attacks as g .
We illustrate this empirically by training a LipNet1 network until it reaches 99.96% accuracy on
CIFAR-100 with random labels (see Appendix I).
We demonstrate that LipNet1 networks are theoretically better grounded than AllNet networks when it comes to classiﬁcation, through our threefold contribution on Expressiveness (Section 3), Robustness (Section 4) and Generalization (Section 5).
First, in Section 3 we conﬁrm that LipNet1 are as expressive as AllNet networks for classiﬁcation, and can learn arbitrary complex decision boundary.
We show that hyper-parameters of the loss are of crucial importance, and control the ability to ﬁt properly the train set.
Then, in Section 4 we show that accuracy and robustness are often antipodal objectives.
We characterize the robustness of the highest accuracy LipNet1 classiﬁer: it is achieved by the Signed
Distance Function (Deﬁnition 6 in Appendix A).
We also characterize the classiﬁer of highest certiﬁable robustness, and we show it corresponds to the dual potential of Wasserstein-1 distance (i.e the discriminator of a WGAN [6]).
Finally, in Section 5 we show that LipNet1 beneﬁt from several generalization guarantees.
They are consistent estimators: contrary to AllNet , we prove that their train loss will converge to test loss as the size of the train set increases.
Moreover, we show that LipNet1 classiﬁers with margin are
PAC-learnable [10]: it provides bounds on the number of train examples required to reach a targeted test accuracy.
Interestingly, this bound is independent of the architecture size, which allows to train enormous LipNet1 networks without risking overﬁtting.
Notations and experimental setting
The core of the paper mainly deal with binary classiﬁcation over R n with label set Y = {− 1 , +1 } .
Let ( X, Y ) be a random variable taking values on X ×Y , where X ⊂ R n is assumed to be a compact set.
Such a pair follows the joint distribution P XY , deﬁned on the space of probability measures
P ( X × Y ) .
The marginal distribution of X is denoted by P X ∈ P ( X ) and its support by supp P X .
We suppose the observation of a sample ( x 1 , y 1 ) , . . .
, ( x p , y p ) i.i.d.
with common law P XY , and the goal is to learn a classiﬁer c : X → Y modeling the optimal Bayes classiﬁer arg max y ∈Y P Y | X ( y | x ) .
P (resp.
Q ) denotes the input distribution of label +1 (resp.
− 1 ).
The Lipschitz constant Lip ( f ) of a function f : R n → R K is deﬁned as the smallest L ≥ 0 such that for all x, z ∈ R n we have (cid:107) f
( x ) − f ( z ) (cid:107) ≤ L (cid:107) x − z (cid:107) .
In the rest of the paper, we focus on euclidean norm (cid:107) · (cid:107) for vectors and spectral norm (cid:107) · (cid:107) 2 for matrices.
The set of L -Lipschitz functions over
X ⊂ R n with image in R K is denoted Lip L ( X , R K ) .
Deﬁnition 1 (Class of AllNet networks)
AllNet denotes the set of unconstrained neural networks.
It includes any feed-forward network of ﬁxed depth (without recurrent mechanisms) using afﬁne layers (including convolutions and batch normalization) with weight matrices W 1 , W 2 , . . .
W d and Lipschitz activation function σ (such as
ReLU, sigmoid, tanh, etc).
No constraint is enforced on their Lipschitz constant during training.
Deﬁnition 2 (Class of LipNet1 networks)
LipNet1 denotes the set of feed-forward neural networks f deﬁned as in Theorem 3 of Anil et al.
[11]: (cid:107) W 1 (cid:107) 2 →∞ ≤ 1 (see [12] for details on the mixed norm (cid:107) · (cid:107) 2 →∞ ) and (cid:107) W i (cid:107) ∞ ≤ 1 for i ≥ 2 , and
GroupSort2 activation function.
They fulﬁll Lip ( f ) ≤ 1 .
Remark.
AllNet networks beneﬁt from universal approximation theorem in C ( X , R K ) , a classical result of literature [13].
LipNet1 networks also beneﬁt from an universal approximation theorem in Lip 1 ( X , R ) with respect to uniform convergence [11].
Note that Lip L ( X , R K ) =
{ Lf | f ∈
Lip 1 ( X , R K ) } so LipNet1 can be used to approximate functions in Lip L ( X , R K ) .
In practice authors of [11] noticed that using orthogonal weight matrices (i.e W Ti W = I ) yielded the best results.
All our experiments use the Deel.
Lip 1 library [8], following ideas of [11].
The networks use 1) orthogonal weight matrices and 2) GroupSort2 activations [11].
Orthogonalization is enforced using Spectral normalization [14] and Björck algorithm
These networks belong to LipNet1 by construction (see Appendix D for our choice of architecture and relevant related work).
LipNet1 networks provide robustness radius certiﬁcates against adversarial attacks [16].
Computing these certiﬁcates is straightforward and does not increase runtime, contrary to methods based on bounding boxes or abstract interpretation [17, 18, 19, 20].
There is no need for adversarial training [21] that fails to produce guarantees, or for randomized smoothing [22] which is costly.
Confusingly, any network of AllNet has a ﬁnite Lipschitz constant, but computing it is NP-hard [23].
Only a loose upper bound can be cheaply estimated: Lip ( f ) ≤ Lip ( σ ) d Π di =1 (cid:107) W
i (cid:107) 2 using the property that Lip ( f d ◦ f d − 1 ◦ . . .
◦ f 1 ) ≤ Π di =1 Lip ( f i ) .
In practice, this bound is often too high to provide meaningful certiﬁcates and besides, AllNet networks have usually very small robustness radius [1].
Deﬁnition 3 (Adversarial Attack)
For any classiﬁer c : X → Y , any x ∈ R n , consider the following optimization problem:
[Equation]
such that c ( x + δ ) (cid:54) =
c ( x ) .
[Equation] δ is an adversarial attack, x + δ is an adversarial example, and (cid:15) is the robustness radius of x .
Property 1 (Local Robustness Certiﬁcates [16]) .
For any f ∈ LipNet1 the robustness radius (cid:15) of binary classiﬁer sign ◦ f at example x veriﬁes (cid:15) ≥ | f ( x ) | .
Losses: The Binary Cross-Entropy (BCE) loss (also called logloss) is among the most popular choices of loss within the deep learning community.
Let f : R n → R a neural network.
For an example x ∈ R n with label y ∈ Y , and σ ( x ) = 1 1+exp( − x ) the logistic function mapping logits to probabilities, the BCE is written L bceτ ( f ( x ) , y ) =
− log σ ( yτf ( x )) , with temperature scaling parameter τ > 0 .
This hyper-parameter of the loss defaults to τ = 1 in most frameworks such as
Tensorﬂow or Pytorch.
Note that L bceτ ( f ( x ) , y )
=
L bce 1 ( τf ( x ) , y ) so we can equivalently tune τ or the
Lipschitz constant L .
We show in Section 5.1 that for LipNet1 the temperature τ allow to control the generalization gap .
We also consider the Hinge loss L Hm ( f ( x ) , y )
= max (0 , m − yf ( x )) with margin m > 0 , as used in [3] for LipNet1 networks training.
We focus on binary classiﬁcation for readability and clarity purposes; however, we prove in Appendices A.2 and E that the following theoretical results generalize to the multi-class case , as done in experiments.
The proofs of all propositions can be found in the appendix.
1-Lipschitz classiﬁers are expressive
In this section, we show that LipNet1 are as powerful as any other classiﬁer, like their unconstrained counterpart.
In particular, when classes are separable they can achieve 100% accuracy.
3.1 Boundary decision ﬁtting
Proposition 1.
Lipschitz Binary classiﬁcation.
For any binary classiﬁer c : X → Y with closed pre-images ( c − 1 ( { y } ) is a closed set) there exists a 1-Lipschitz function f : R n → R such that sign ( f ( x ))
=
c ( x ) on X and such that (cid:107)∇ x f (cid:107) =
almost everywhere (w.r.t Lebesgue measure).
The level-sets of a Lip 1 ( X , R K ) functions (and especially the decision boundary) can be arbitrarily complex: restraining classiﬁers to Lip 1 ( X , R ) does not affect the classiﬁcation power.
Deﬁnition 4 ( (cid:15) -separated distributions )
Distributions P and Q are (cid:15) -separated
if the distance between supp P and supp Q exceeds (cid:15) > 0 .
Corollary 1.
Separable classes implies zero error .
If P and Q are (cid:15) -separated, then there exists a network f ∈ LipNet1 such that error E ( sign ◦ f ) :
=
E ( x,y ) ∼ P XY [ 1 { sign ( f ( x )) (cid:54) =
y } ]
= 0 .
The class of LipNet1 networks does not suffer from bias for classiﬁcation tasks.
Some empirical studies show that indeed most datasets classes are separable [24] such as CIFAR10 or MNIST.
Furthermore, even if the classes are not separable, functions of LipNet1 can nonetheless approximate the optimal Bayes classiﬁer.
Lipschitz constraint is not a constraint on the shape of the boundary (Figure 1a), but on the slope of the landscape of f .
3.2 Understanding why LipNet1 are often perceived as not expressive
LipNet1 networks cannot reach zero loss with BCE: this may explain why they are perceived as not expressive enough.
Yet the minimizer of BCE exists and is well deﬁned.
Proposition 2.
BCE minimization for 1-Lipschitz functions .
Let X ⊂ R n be a compact and τ > 0 .
Then the inﬁmum in Equation 2 is a minimum, denoted f τ ∈ Lip 1 ( X , R ) :
[Equation]
Moreover, the LipNet1 networks will not suffer of vanishing gradient of the loss (see Appendix F).
Machine learning practitioners are mostly interested in maximizing accuracy.
However, the minimizer of BCE is not necessarily a minimizer of the error (see Figure 1b).
Yet, BCE is notoriously a differentiable proxy of the error E ( sign ◦ f ) , and as τ → ∞ we get asymptotically closer to maximum empirical accuracy.
Bigger value for τ might ultimately lead to overﬁtting, playing the same role as the Lipschitz constant L (see Figure 1b).
The implicit parameter τ = 1 of the loss is partially responsible of the poor accuracy of
LipNet1 networks in literature , and not by any means the hypothesis space LipNet1 itself.
This can be observed in practice : when temperature τ (resp.
margin m ) of cross-entropy (resp.
hinge loss) is correctly adjusted a small LipNet1 CNN can reach a competitive 88.2% validation accuracy on the CIFAR-10 dataset (results synthetized and discussed in Figure 3) without residual connections, batch normalization or dropout.
Conversely, AllNet networks are roughly equivalent to learning a
LipNet1 network with τ → ∞ : without regularization or data augmentation, such a network can always reach 100% train accuracy without generalization guarantees. The leftmost network is the most accurate at robustness radius ≤ maximizes the
MCR at the cost of low clean accuracy.
The center network corresponds to a compromise
1-Lipschitz classiﬁers are certiﬁably robust
Is there a trade-off between accuracy and robustness?
Although the existence of a trade-off between accuracy and robustness is commonly admitted, some works argue that “Robustness is not inherently at odds with accuracy”[24].
We propose a uniﬁed consideration by stating that for a given train accuracy, robustness can be maximized up to a certain point, but allowing a lower train accuracy helps achieving a higher robustness.
Finally one must keep in mind that this trade-off lives in the shade of generalization (see Section 5).
4.1 Improving the robustness of the maximally accurate classiﬁer
The Signed Distance Function [25] (SDF) (see Deﬁnition 6 in Appendix A) associated to the frontier ∂ of Bayes classiﬁer b is the 1-Lipschitz function that provides the largest certiﬁcates among the classiﬁers of maximum accuracy.
Moreover, those certiﬁcates are exactly equal to the distance of adversarial samples.
Iterative gradient based attacks (see [26] and references therein) can succeed in one step: far from being a weakness, this may improve the interpretability of the model [27, 28, 29].
Corollary 2.
For the SDF( b ), the bound of Property 1 is tight: (cid:15) =
| f
( x ) | .
In particular δ = − f ( x ) ∇ x f ( x ) is guaranteed to be an adversarial attack.
The risk is the smallest possible.
There is no classiﬁer with the same risk and better certiﬁcates.
Said otherwise the SDF( b ) is the solution to:
[Equation]
under the constraint f ∈ arg min g ∈ Lip 1 ( R n , R ) E ( sign ◦ g )
[Equation]
The SDF( b ) cannot be explicitly constructed since it relies on the (unknown) optimal Bayes classiﬁer.
4.2 Improving the accuracy of the maximally robust classiﬁer
On the opposite side, we exhibit a family of classiﬁers with lower accuracy but with higher certiﬁable robustness.
We insist that the quantity of interest is the certiﬁable robustness | f ( x ) | and not the true empirical robustness (cid:15) (which can be higher).
The former is computed exactly and freely, while the latter is a difﬁcult problem for which only upper bounds returned by attacks are available.
In the literature, the robustness is only evaluated on well classiﬁed examples.
The certiﬁcate can be both interpreted as a form of “conﬁdence” of the network, and as the minimal perturbations required to switch the class.
Hence, we shall weight negatively this certiﬁcate for the examples that are misclassiﬁed since conﬁdence in presence of errors is worse.
For this reason, we propose in
Deﬁnition 5 a new metric called the Mean Certiﬁable Robustness (MCR).
Deﬁnition 5 ( Mean Certiﬁable Robustness – MCR )
For any function f : X → R ∈
LipNet1 we deﬁne its weighted mean certiﬁable robustness R ( P,y )
( f ) on class P with label y as:
[Equation]
We can readily see from the deﬁnition that the classiﬁer with highest MCR for class P is the constant classiﬁer f = y × ∞ .
The interest of this notion arises when we consider minimizing the loss function L W ( f ( x ) , y ) :
= − yf ( x ) , i.e when looking for classiﬁer with the highest MCR.
Property 2.
Wasserstein classiﬁers (i.e WGAN discriminators) are optimally robust .
The minimum of L W ( f ( x ) , y ) over P and Q is the Wasserstein-1 distance [30] between P and Q according to the Kantorovich-Rubinstein duality:
[Equation]
Even though the minimizer of L W ( f ( x ) , y ) can have low accuracy, it has the highest MCR.
Interestingly, the minimizer f ∗ of equation 5 is invariant by translation: f ∗ − T is also a minimizer for any T ∈ R .
When T → ∞ (resp.
−∞ ) the classiﬁer has 100% recall on Q (resp. P ), and 0% on P (resp. Q ).
Does it always exist T ∗ with 100% accuracy overall?
Sadly, even when the P and Q have disjoint support, the answer is no.
We precise this empirical observation of [8] in Proposition 3.
Proposition 3.
WGAN discriminators are weak classiﬁers .
For every 12 ≥ (cid:15) > 0 there exist distributions P and Q with disjoint supports in R such that for any optimum f of equation 5, the error of classiﬁer sign ◦ f is superior to 12 − (cid:15) .
Note that this minimum also invariant by dilatation: any ﬁnite upper bound L can be chosen for
Equation 5 (see Appendix G).
4.3 Controlling the accuracy/robustness tradeoff with loss parameters
Now that the extrema of the accuracy robustness tradeoff were characterized in 4.1 and 4.2, is yet to be answered if it is possible to control this tradeoff using conventional loss (and its parameters, as introduced in 3.2).
Interestingly, observe that L bceτ
[Equation]
when τ → 0 we get:
[Equation]
In the limit of small temperatures, the BCE minimizer essentially behaves like the classiﬁer of the highest MCR (see Figure 3 and Appendix H).
Similarly, the HKR loss L hkr introduced in [8] for
LipNet1 training allows ﬁne grained control of the accuracy-robustness tradeoff:
[Equation]
We recover W 1 behavior for α = 0 , and hinge L H m behavior for α → ∞ , in a fashion that reminds the role of τ for L bce .
A key takeaway is that BCE, HKR and hinge loss have parameters that allow to control the accuracy robustness tradeoff, reaching on one side the maximum robustness of MCR, and the accuracy of unconstrained networks on the other.
Empirically this tradeoff is observed as a Pareto front with accuracy on one axis, and robustness on the other.
Figure 3 shows this on the CIFAR10 dataset using the (cid:15) robustness as robustness measures (other robustness measure yield similar observations, see ﬁg 10a and 10b).
In conclusion, these last two sections demonstrate that restraining networks to be in LipNet1 does not impact the classiﬁcation capabilities while providing certiﬁcates of robustness; however, for these networks the loss parameters play an important role in this trade-off.
1-Lipschitz classiﬁers have generalization guarantees
In this section, we explore the statistical and optimization properties of LipNet1 networks, and we prove the assumption of [31] that “adjusting the Lipschitz constant of a feed-forward neural network controls how well the model will generalise to new data”. Entropy (CCE) hyper-parameters.
Overall, for a given network architecture, a Pareto front appears between clean accuracy and robust accuracy.
We move along it by tuning the parameters of each loss.
We trained small LipNet1 CNNs (0.4M params) with basic data augmentation (see appendix K for detailed experimental setting).
5.1 Consistency of LipNet1 class
LipNet1 class enjoys another remarkable property since it is a Glivenko-Cantelli class: minimizers of
Lipschitz losses are consistent estimators.
In other words, as the size of the training set increases, the training loss becomes a proxy for the test loss: LipNet1 neural networks will not overﬁt in the limit of (very) large sample sizes.
Proposition 4.
Train Loss is a proxy of Test Loss .
Let P XY a probability measure on X × Y where X ⊂ R n is a bounded set.
Let ( x i , y i ) 1 ≤ i ≤ p be a sample of p iid random variables with law P
XY .
Let L be a Lipschitz loss function over R × Y .
We deﬁne:
[Equation]
E
[Equation]
Then the empirical loss E p ( f ) converges to the test loss E ∞ ( f ) (taking the limit p → ∞ ):
[Equation]
It is another ﬂavor of the bias-variance trade-off in learning.
Thanks to Corollary 1 we know the
LipNet1 class does not suffer of bias, while the generalization gap (i.e the variance) can be made as small as we want by increasing the size of the training set (see Figure 4).
The number of examples required to close the generalization gap is dataset speciﬁc in general, however it seems that with low τ fewer examples are required.
This result may seem obvious, but we emphasize this property is not shared by AllNet networks (see Proposition 9 in Appendix C.2).
Nonetheless, most practitioners take for granted that bigger training sets ensure generalization for AllNet networks.
5.2 Understanding why unconstrained networks are prone to overﬁtting
Surprisingly, on AllNet networks, minimization of BCE leads to uncontrolled growth of Lipschitz constant and saturation of the predicted probabilities.
This is an impediment to generalization results.
Proposition 5.
Optimizing BCE over AllNet leads to divergence .
Let f t be a sequence of neural networks, that minimizes the BCE over a non-trivial training set (at least two different examples with different labels) of size p , i.e assume that:
[Equation]
Let L t be the Lipschitz constant of f t .
Then lim t →∞ L t = + ∞ .
There is at least one weight matrix
W such that lim t →∞ [Equation] W t (cid:107) =
+ ∞ .
Furthermore, the predicted probabilities are saturated:
[Equation]
This issue is especially important since Lipschitz constant and adversarial vulnerabilities are related [32].
The predicted probability σ ( f ( x )) will either be 0 or 1 (regardless of the train set), which do not carry any useful information on the true conﬁdence of the classiﬁer
Example 1.
Consider a classiﬁcation task on R with linearly separable inputs {− 1 , 1 } and labels {− 1 , 1 } .
We use an afﬁne model f ( x ) =
Wx + b for the logits (with W ∈ R and b ∈ R ) (one-layer neural network).
It exists ¯ W, ¯ b such that f achieves 100% accuracy.
However, as noticed in [33] (Section 4.3.2) the BCE loss will not be zero.
The minimization occurs only with the diverging sequence of parameters ( λ ¯ W, λ ¯ b ) as λ → ∞ .
It turns out the inﬁmum is not a minimum!
Even on toy example 1 with a trivial model, the minimization problem is ill-deﬁned.
Without weight regularization, the minimizer can not be attained.
This is compliant with the high Lipschitz constant of AllNet networks that have been observed in practice [23], and is conﬁrmed by our experiment on
MNIST with a ConvNet (see Figure 12).
The spectral norm of the weights is multiplied by 5 over the course of 25 epochs, whereas the validation accuracy remains the same (around 99%).
Furthermore, there is an issue of vanishing gradients with BCE : ﬁrst order methods struggle to saturate the logits of AllNet networks, whereas second order methods in ﬂoat64 diverge as expected.
The poor properties of the optimizer, and the rounding errors in 32 bits ﬂoating point arithmetic, have greatly contributed to the caveat of BCE minimization remaining mostly unnoticed by the community.
5.3 Lipschitz classiﬁers are PAC learnable
Hinge loss L Hm and HKR loss L hkr beneﬁt from Proposition 4.
The certiﬁcate | f ( x ) | can be understood as conﬁdence.
Hence, we are interested in a classiﬁer that makes a decision only if the prediction is above some threshold m > 0 , while | f ( x ) | < m can be understood as examples x for which the classiﬁer is unsure: the label may be ﬂipped using attacks of norm (cid:15) ≤ m . In this setting, we fall back to PAC learnability [10]: this theory gives bounds on the number of train samples [Table] (Proposition 4, Figures 1b,
[Table]
(Figure 3)
[Table] (Figure [Table] required to guarantee that the test error will fall below some threshold 0 ≤ e < 12 with probability at least 1 > β ≥ 0 , through the use of Vapnik Chervonenkis (VC) dimension bounds [34].
Proposition 6. 1-Lipschitz Functions with margin are PAC learnable .
Assume P and Q have bounded support X .
Let m > 0 the margin.
Let C m ( X ) =
{ c mf : X → {− 1 , ⊥ , +1 } , f ∈
Lip 1 ( X , R ) } be the hypothesis class deﬁned as follow.
[Equation]
otherwise, meaning “ f doesn’t feel conﬁdent”.
Let B be the unit ball.
Then the VC dimension of C m is ﬁnite:
[Equation]
Interestingly if the classes are (cid:15) separable ( (cid:15) > 0 ), choosing m = (cid:15) guarantees that 100% accuracy is reachable.
Prior over the separability of the input space is turned into VC bounds over the space of hypothesis.
When m =
the VC dimension of space C m ( X ) becomes inﬁnite and the class is not
PAC learnable anymore: the training error will not converge to test error in general, regardless of the size of the training set.
It is not a contradiction with Proposition 4: error E ( c mf ( x )) lacks continuity w.r.t f
( x ) so it is not a consistent estimator.
This VC bound is architecture independent which contrasts with the rest of literature on
AllNet networks.
Practically, it means that the LipNet1 network architecture can be chosen as big as we want without risking overﬁtting, as long as the margin m is chosen appropriately.
Propositionalso provides an architecture dependant bound for LipNet1 networks.
Proposition 7.
VC dimension of LipNet1 neural networks .
Let f θ : R n → R a LipNet1 neural network with parameters θ ∈ Θ , with GroupSort2 activation functions, and a total of W neurons.
Let H =
{ sign f θ | θ ∈ Θ } the hypothesis class spanned by this architecture.
Then we have:
[Equation]
In literature, tighter VC dimension bounds for neural networks exist, but they assume element-wise activation function [35].
This hypothesis does not apply to GroupSort2 which is known to be more expressive [36], however we believe that this preliminary result can be strengthened.
Related work
LipNet1 networks parametrization beneﬁt from a rich literature (see Appendix D) to enforce the Lipschitz constraint in various layers [37, 38, 6, 39, 40, 14, 41, 42, 43, 44] such as activation functions, afﬁne layers, attention layers or recurrent units.
Residual connections are also Lipschitz (see Appendix F).
Gradient Norm Preserving networks avoid the vanishing gradients [3, 45] phenomenon to which the LipNet1 networks are prone, by using orthogonal matrices in afﬁne layers.
This justiﬁes the “orthogonal neural network” terminology [46, 47, 48].
ReLU based Lipschitz networks suffer from expressiveness issues [11], and activation functions like GroupSort [11, 36] (a special case of Householder reﬂection [49, 50]) have been proposed in replacement.
Orthogonal kernels are still an active research area [51, 52, 53, 54, 3, 55, 56, 57].
They are used in normalizing ﬂows [58], ensemble methods [59], reinforcement learning [60] or graph neural networks [61].
The optimization over the group of orthogonal matrices (known as Stiefel manifold) has been extensively studied in [62], and algorithms suitable for deep learning are detailed in [63, 64, 64, 65, 66, 67, 68, 69].
Generalization bounds for general Lipschitz classiﬁers are given in [70, 71, 72].
Links between adversarial robustness, large margins classiﬁers and optimization bias are studied in [73, 74, 16, 75].
The importance of the loss in adversarial robustness is studied in [76].
See Appendix C.5.
Conclusions
In this paper, we challenged the common belief that constraining Lipschitz constant degrades the classiﬁcation performance of neural networks.
We proved that LipNet1 networks exhibit numerous attractive properties (see Table 1 in summary): they provide robustness radius certiﬁcates without restrictions on their expressive power.
They beneﬁt from generalization guarantees.
We showed that the hidden parameters of the loss allow to control the generalization gap and certiﬁable robustness.
While the question of the LipNet1 architecture is often in the spotlight, the loss is overlooked. We pointed out that Cross-Entropy is not necessarily the best choice, margin-based losses, such as hinge or its variant HKR, have appealing properties (table 1).  Perspectives
This paper aims to be at the intersection between theoretical ML and (empirical) deep learning.
Lipschitz constrained networks allow to directly put in perspective mathematical proofs and we are conﬁdent that this theory can be veriﬁed empirically on very large-scale vision datasets (such as Imagenet [77]). This paper also provides a toolbox of results and experiments to serve as a basis for future works. We aim to open new research directions, including outside the ﬁeld of robust learning.
AllNet networks could beneﬁt from LipNet1 literature: the absence of control over the Lipschitz constant of AllNet is mitigated in practice by elements such as mixup or weight decay. Such elements would be better understood by looking at how they affect the (uncontrolled) Lipschitz constant of AllNet . The efﬁcient training over LipNet1 is still an active research area. Moreover, AllNet networks beneﬁts from architectural elements such as skip connections and batch normalization (see appendix F).
As
LipNet1 networks get more mature, empirical results will improve, matching theory even more (explaining the emphasis on the theoretical proofs instead of the design of LipNet1 depicted in appendix D). Many practices in deep learning entangle the questions of architecture, of generalization and of optimization. However, these elements usually have unexpected consequences on the nature of the optimum and the optimization process. Our work is a ﬁrst step toward a better separation of these components and their role.
We thank Sébastien Gerchinovitz for critical proof checking, Jean-Michel Loubes for useful discussions, and Etienne de Montbrun, Thomas Fel and Antonin Poché for their read-checking. A special thank to Agustin Picard for his useful advice and thorough reading of the paper. This work has beneﬁted from the AI Interdisciplinary Institute ANITI, which is funded by the French “Investing for the Future – PIA3” program under the Grant agreement ANR-19-P3IA-0004.
The authors gratefully acknowledge the support of the DEEL project.
Proofs of Section
C.1 Consistency of Lipschitz estimators
C.2 Proofs of Section 5. C.3 VC dimension for Lipschitz classiﬁers with margin
C.4 VC dimension for GroupSort networks
C.5 Generalization bounds literature survey
D Deel. Lip networks
E Multiclass Hinge Kantorovich Rubinstein
F Gradient Norm Preserving (GNP) networks[-
-]
G Wasserstein discriminator does not depend of the Lipschitz constant[-
-]
H BCE through the lens of OT
I
Fitting CIFAR100 with random labels
J 1-Lipschitz estimators are consistent (experimental protocol of ﬁgure 4)[-
-]
K Controlling the accuracy/robustness tradeoff (experimental protocol of ﬁgure 3)[-
-]
L Hardware
M Divergence of the weights on AllNet networks
N
Stability
of
training
of LipNet[-
-]
N.1 Moving along Pareto front by tuning temperature . . . . . . . . . . . . . . . . . .
N.2 Shifting Pareto front by tuning architecture . . . . . . . . . . . . . . . . . . . . .
A Proofs of SectionA.1 Single class case
The properties of Lebesgue measure over euclidean space R n are recalled in [78].
Importantly, the Lebesgue measure µ is translation invariant and measure the volume of hyperboxes: µ ([ a 1 , b 1 ] , [ a 2 , b 2 ] , . . .
[ a n , b n ]) =
Π ni =
( b i − a i ) .
Proposition 1.
Lipschitz Binary classiﬁcation.
For any binary classiﬁer c : X → Y with closed pre-images ( c − 1 ( { y } ) is a closed set) there exists a 1-Lipschitz function f : R n → R such that sign ( f ( x ))
=
c ( x ) on X and such that (cid:107)∇ x f (cid:107) =
almost everywhere (w.r.t Lebesgue measure).
The proof of Proposition 1 is constructive, we need to introduce the Signed Distance Function, already popularized in shape processing [25].
Deﬁnition[-
-]
Signed Distance Function associated to decision boundary.
Let c : X → {− 1 , +1 } be any classiﬁer with closed pre-images.
Let ¯
A = { x ∈ R n | c ( x ) = +1 } and ¯ B = { x ∈ R n | c ( x ) = − 1 } =
X \ ¯ A .
Let d ( x, y )
= (cid:107) x − y (cid:107) and d ( x, S ) =
min y ∈ S d ( x, y ) be the distance to a closed set S .
Let ∂ =
{ x ∈ R n | d ( x, ¯ A ) =
d ( x, ¯ B ) } .
We deﬁne f : R n → R as follow:
[Equation]
We denote by SDF( c ) the function f .
The signed distance function f previously deﬁned veriﬁes all the properties, as a special case of
Eikonal equation.
We give the full proof here for completeness.
Proof.
We start by proving that f is 1-Lipschitz.
The reasoning applies more broadly to arbitrary
Banach space (topological normed vector space), not only ( R n , (cid:107) · (cid:107) 2 ) .
First, consider the case d ( x, ¯ B ) ≥ d ( x, ¯ A ) and d ( y, ¯ B ) ≥ d ( y, ¯ A ) .
Then we have | f ( x ) − f ( y ) | =
| d ( x, ∂ ) − d ( y, ∂ ) | .
Assume without loss of generality that d ( x, ∂ ) ≥ d ( y, ∂ ) .
Let z ∈ ∂ be such that d ( y, ∂ )
=
d ( y, z ) (it is guaranteed to exist since ∂ is a closed set).
Then by deﬁnition of d ( x, ∂ ) we have d ( x, z ) ≥ d ( x, ∂ ) .
So:
[Equation]
[Equation]
The cases d ( x, ¯ B ) <
d ( x, ¯ A ) and d ( y, ¯ B ) <
d ( y, ¯ A ) are identical.
Now consider the case d ( x, ¯ B ) <
d ( x, ¯ A ) and d ( y, ¯ B ) ≥ d ( y, ¯ A ) .
Then we have | f ( x ) − f ( y ) |
=
d ( x, ∂ ) + d ( y, ∂ ) .
We will proceed by contradiction.
Such complicated reasoning is superﬂuous for ( R n , (cid:107) · (cid:107) 2 ) , but has the appealing property to generalize to any Banach space.
Assume that d ( x, ∂ ) + d ( y, ∂ ) >
d ( x, y ) .
Let R > 0 be such that R <
d ( x, ∂ ) and R + d ( y, ∂ ) >
d ( x, y ) .
Let:
[Equation]
Then we have d ( x, z ) = (cid:107) R d ( x,y )
( x − y ) (cid:107) =
R d ( x,y ) d ( x, y )
=
R <
d ( x, ∂ ) .
So by deﬁnition of ∂ we have d ( z, ¯ B ) <
d ( z, ¯ A ) .
But we also have:
[Equation]
| using the hypothesis on R.
[Equation]
[Equation]
So we have d ( z, ¯ B ) ≥ d ( z, ¯ A ) which is a contradiction.
Consequently, we must have d ( x, ∂ ) + d ( y, ∂ ) ≤ d ( x, y ) .
The function f is indeed 1-Lipschitz.
Now, we will prove that (cid:107)∇ x f (cid:107) =
1 everywhere it is deﬁned.
Let x be such that y ∈ arg min y ∈ ∂
d ( x, y ) is unique.
Consider h = (cid:15) ( y − x ) (cid:107) y − x (cid:107) with 1 ≥ (cid:15) > 0 a small positive real.
We have d ( x, x + h ) =
(cid:15) , it follows by triangular inequality that d ( x + h, ∂ ) =
d ( x, ∂ ) − (cid:15) .
We see that:
[Equation]
The vector u = −∇ x f is the (unique) vector for which (cid:104) u, f ( x + h ) − f ( x ) (cid:107) h (cid:107) (cid:105) is minimal.
Knowing that f is 1-Lipschitz yields that (cid:107)∇ x f (cid:107) = 1 .
For points x for which arg min y ∈ ∂
d ( x, y ) is not unique, the gradient is not deﬁned because different directions minimize (cid:104) u, f ( x + h ) − f ( x ) (cid:107) h (cid:107) (cid:105) which contradicts the uniqueness of gradient vector.
The number of points for which y ∈ arg min y ∈
∂
d ( x, y ) is not unique must have null measure, since Lipschitz functions are almost everywhere differentiable (by
Rademacher’s Theorem).
Finally, note that sign f ( x ) =
c ( x ) on ¯ A and ¯ B .
Indeed, in this case either d ( x, ¯ B ) <
d ( x, ¯ A ) either d ( x, ¯ B ) >
d ( x, ¯ A ) and the result is straightforward.
A.2 Multiclass case
The label set is now Y = { 1 , 2 , . .
. , K } .
In practice we use one-hot encoded vectors to compute the loss, by taking the arg max k over a vector of R K .
Proposition 8 (Lipschitz Multiclass classiﬁcation) .
For any multiclass classiﬁer c : X → Y with closed pre-images there exists a 1-Lipschitz function f : R n → R K such that arg max k f k ( x ) =
c ( x ) on X and such that (cid:107)
J x f (cid:107) =
1 almost everywhere (w.r.t Lebesgue measure).
For the case K > 2 we must slightly change the deﬁnition to prove Proposition 8 .
Deﬁnition 7 (Multiclass Signed Distance Function)
Let c : X → { 1 , 2 , . . .
K } be any classiﬁer with closed pre-images.
Let ¯
A k = c − 1 ( { k } ) .
Let ∂ =
{ x ∈ R n |∃ k (cid:54)
=
l, d ( x, ¯ A k ) = d ( x, ¯ A l ) =
arg min m d ( x, ¯ A m ) } .
We deﬁne f : R n → R k as follow:
[Equation]
In overall the proof remains the same.
Proof.
We start by proving that f is 1-Lipschitz.
We need to prove that (cid:107) f
( x ) − f ( y ) (cid:107) p ≤ (cid:107) x − y (cid:107) for any p -norm on R K with p ≥ 1 .
First, consider the case f k ( x ) =
f k ( y ) (cid:54) = 0 .
Then (cid:107) f
( x ) − f ( y ) (cid:107) p =
| f k ( x ) − f k ( y ) | =
| d ( x, ∂ ) − d ( y, ∂ ) | ≤ (cid:107) x − y (cid:107) using the proof of Proposition 1.
Now, consider the case f k ( x ) > 0 , f l ( y ) > 0 and k (cid:54)
= l . Then: (cid:107) f
( x ) − f ( y ) (cid:107) p = p (cid:113) f
pk ( x ) + f pl ( y ) ≤ | f k ( x ) | + | f l ( y ) |
=
d ( x, ∂ ) + d ( y, ∂ ) .
[Equation]
Using the same technique as in the previous proof, if we assume d ( x, ∂ ) + d ( y, ∂ ) >
d ( x, y ) then we can construct z verifying both f k ( z ) <
f l ( z ) and f l ( z ) < f k ( z ) which is a contradiction.
Consequently d ( x, ∂ ) + d ( y, ∂ ) ≤ d ( x, y ) .
Each row of J x f is either full of zeros, or the gradient of some f k on which the reasoning of the case K = 2 applies (like in the previous proof).
In this case, the spectral norm J x f is equal to the norm of the gradient of the non zero row.
We conclude similarly that (cid:107)
J x f (cid:107) =
1 everywhere it is deﬁned.
Finally, note that arg max k f k ( x ) is equal to c ( x ) everywhere c is deﬁned, which concludes the proof.
With this proposition in mind, we can deduce Corollary 3.
Corollary 3 (LipNet1 is as powerful as AllNet for classiﬁcation) .
For any neural network f : R n →
R there exists 1-Lipschitz neural network ˜ f
: R n → R such that sign ( f ( x ))
=
sign ( ˜ f ( x )) .
Proof.
The proof sketched in Introduction is sufﬁcient to show that LipNet1 networks and unconstrained ones have the same decision frontiers.
We could have also taken a more convoluted path: take the classiﬁer c associated to an AllNet network, consider the restriction to a subset X of the input space making the pre-images separated.
Then we can apply Proposition 1 to get a 1-Lipschitz function with the same classiﬁcation power, and ﬁnally approximate those functions (in the sense of uniform convergence) with LipNet1 network.
Corollary 1.
Separable classes implies zero error .
If P and Q are (cid:15) -separated, then there exists a network f ∈ LipNet1 such that error E ( sign ◦ [Equation]
:=
[Equation]
∼ [Equation]
Proof.
If classes are separable the optimal Bayes classiﬁer b achieves zero error.
Moreover, the topological closure b − 1 ( { y } ) , y ∈ Y yields a set of closed sets that are all disjoints (since (cid:15) > 0 ) and on which Proposition 8 can be applied, yielding a LipNet1 neural network with the wanted properties.
Bonus: non separable case.
We can also handle the case of non separable classes by imitating the optimal Baye classiﬁer c .
We take X a subset of the input space on which the pre-images of c are closed.
The application of Proposition 1 for optimal Bayes classiﬁer gives us a 1-Lipschitz function f with same decision frontier as c .
Finally, we can use the universal approximation theorem of Anil and all.
in [11] to conclude there exists LipNet1 network that can approximate arbitrary well the function f , and hence approximate arbitrarily well the classiﬁer c on X .
Outside X , the error is not controlled but depends of the volume of the set ( supp P X ) /
X whose Lebesgue measure can be made arbitrary small (by taking X big enough).
As P X admits a pdf w.r.t Lebesgue measure, then P
X (( supp P X ) / X ) can be made arbitrary small, and consequently the risk as well.
Example 1a.
We plot the level set of the network f trained from the discretized ground truth (in 400 × 400 pixels) of the Signed distance function.
The distance to the frontier ∂ is easily computed since the frontier ∂ is a ﬁnite collection of segments (fourth iteration of Von Koch snowﬂake fractal).
We train a 128 (cid:41) 128 (cid:41) 128 (cid:41) 128 (cid:41) 128 LipNet1 network.
The network is trained with Mean Square
Error (MSE) and the criterion used to stop training is the Mean Absolute Error (MAE).
A.3 Proofs of Section [-3.
-]3.
Proposition 2.
BCE minimization for 1-Lipschitz functions .
Let X ⊂ R n be a compact and τ > 0 .
Then the inﬁmum in Equation 2 is a minimum, denoted f τ ∈ Lip 1 ( X [Equation]
[Equation]
Moreover, the LipNet1 networks will not suffer of vanishing gradient of the loss (see Appendix F).
The proof of Proposition 2 is an application of Arzelà–Ascoli theorem.
Proof.
Let E ( f ) =
E ( x,y ) ∼ P XY [ L ( f ( x ) , y )] .
Consider a sequence of functions f t in Lip L ( X , R ) such that lim t →∞ E ( f t ) = inf f ∈ Lip L ( X , R ) E
[Equation]
f [Equation] = E ∗ . Consider the sequence u t = (cid:107) f t (cid:107) ∞ .
We want to prove that ( u t ) t ∈ N is bounded.
Proceed by contradiction and observe that if lim sup t →∞ u t = + ∞ then lim sup t →∞ E ( f t ) = + ∞ .
Indeed, for (cid:107) f t (cid:107) ∞ ≥ 2 L diam X we can guarantee that sign f t is constant over X and in this case one of the two classes y is misclassiﬁed, knowing that lim f ( x ) →∞ L ( − yf ( x ) , y )
=
O ( f ( x ))
→ + ∞ yields the desired result.
But if lim sup t →∞ E ( f t ) =
+ ∞ , then E ( f t ) cannot not converges to E ∗ .
Consequently, u t must be upper bounded by some M .
Hence the sequence f t is uniformly bounded.
Moreover each function f t is L -Lipschitz so the sequence f t is uniformly equicontinuous.
By applying Arzelà–Ascoli theorem we deduce that it exists a subsequence f φ ( t ) (where φ : N → N is strictly increasing) that converges uniformly to some f ∗ , and f ∗ ∈ Lip L ( X , R ) .
As E ( f ∗ ) =
E ∗ , the inﬁmum is indeed a minimum.
The upper bound on Lip( f ) is turned into a lower bound on (cid:107)∇ θ L ( f θ ∗ L ( x ) , y ) (cid:107) (no element-wise vanishing gradient), but its average (cid:107)∇ θ E ( x,y ) ∼ P XY [ L (( f θ ∗ L ( x ) , y )] (cid:107) = 0 is null (see Appendix F).
B Proofs of Section
We recall below the deﬁnition of the Signed Distance Function (SDF) associated to a classiﬁer.
Deﬁnition[-
-]
Signed Distance Function associated to decision boundary.
Let c : X → {− 1 , +1 } be any classiﬁer with closed pre-images.
Let ¯
A = { x ∈ R n | c ( x ) = +1 } and ¯ B = { x ∈ R n | c ( x ) = − 1 } =
X \ ¯ A .
Let d ( x, y )
= (cid:107) x − y (cid:107) and d ( x, S ) =
min y ∈ S d ( x, y ) be the distance to a closed set S .
Let ∂ =
{ x ∈ R n | d ( x, ¯ A ) =
d ( x, ¯ B ) } .
We deﬁne f : R n → R as follow:
[Equation]
We denote by SDF( c ) the function f .
In proof of Corollary 2 we use the Bayes classiﬁer b : X → Y associated to the classiﬁcation task between P and Q .
Corollary 2.
For the SDF( b ), the bound of Property 1 is tight: (cid:15) =
| f
( x ) | .
In particular δ = − f ( x ) ∇ x f ( x ) is guaranteed to be an adversarial attack.
The risk is the smallest possible.
There is no classiﬁer with the same risk and better certiﬁcates.
Said otherwise the SDF( b ) is the solution to:
[Equation]
under the constraint f ∈ arg
[Equation]
Proof.
Those properties hold by construction.
The risk R ( sign ( f )) is minimal since f is build with the optimal Bayes classiﬁer.
Note that, in general, for any classiﬁer c : X → Y the bound of
Property 1 is tight by construction for SDF( c ).
Indeed f ( x ) is the distance to the frontier, and the direction is given by ∇ x f ( x ) .
For the proof of Proposition 2 we recall below the deﬁnition of Wasserstein-1 distance, as found in [30] (Deﬁnition 6.1).
Deﬁnition 8 (Wasserstein-1 distance)
Let d : R n × R n → R be a metric.
For any two measures P and Q on R n the Wasserstein-1 distance is deﬁned by the following formula:
[Equation]
where Π( P, Q ) denote the set of measures on R n × R n whose marginals are P and Q respectively.
Equivalently we can write:
[Equation]
In our case we are working with neural networks that are Lipschitz with respect to l 2 distance, so we have d ( x, y ) :
= (cid:107) x − y (cid:107) 2 .
Property 2.
Wasserstein classiﬁers (i.e WGAN discriminators) are optimally robust .
The minimum of L W ( f ( x ) , y ) over P and Q is the Wasserstein-1 distance [30] between P and Q according to the Kantorovich-Rubinstein duality:
[Equation]
Proof.
The result is straightforward by writing the dual formulation (following KantorovichRubinstein) of Wasserstein W 1 metric.
By Remark 6.3 of [30] the Wasserstein-1 distance is the Kantorovich-Rubinstein distance:
[Equation]
We see that:
[Equation]
By Kirszbraun’s theorem the optimum of Equation 19 can be extended into a 1-Lipschitz function over R n .
This function can, in turn, be approximated by a LipNet1 network over the domain of interest.
Proposition 3.
WGAN discriminators are weak classiﬁers .
For every 12 ≥ (cid:15) > 0 there exist distributions P and Q with disjoint supports in R such that for any optimum f of equation 5, the error of classiﬁer sign ◦ f is superior to 12 − (cid:15) .
Proof.
We will build P and Q as a ﬁnite collection of Diracs.
Let P = 1 n (cid:80) ni
=1 δ 4( i − 1) and
Q = 1 n (cid:80) ni
=1 δ 4 i − 1 for some n ∈ N , where δ x denotes the Dirac distribution in x ∈ R .
A example is depicted in Figure 6 for n = 20 .
In dimension one, the optimal transportation plan is easy to compute: each atom of mass from P at position i is matched with the corresponding one in Q to its immediate right.
Consequently we must have f (4 i − 1) =
f (4( i − 1))
+ 3 .
The function f is not uniquely deﬁned on segments [4 i − 1 , 4 i ] but it does not matter: since f is 1-Lipschitz we must have | f (4 i − 1) − f (4 i ) | ≤ 1 .
Consequently in every case for i <
j we must have f (4( i − 1)) <
f (4( j − 1)) and f (4 i − 1) < f
(4 j − 1) .
Said otherwise, f is strictly increasing on supp P and supp Q .
The solutions of the problems are invariant by translations: if f is the solution, then f − T with T ∈ R is also a solution.
Let’s take a look at classiﬁer c ( x ) = sign ( f ( x ) − T ) .
If T is chosen such that f (4( i − 1)) − T < 0 and f (4 i − 1) − T > 0 for some 1 ≤ i ≤ n then ( n − 1) + 2 = n +points are correctly classiﬁed on a total of 2 n points.
It corresponds to an error of n − 1 2 n = 12 − 12 n .
Take n = (cid:100) 12 (cid:15) (cid:101) to conclude.
C Proofs of SectionC.1 Consistency of Lipschitz estimators
Proposition 4.
Train Loss is a proxy of Test Loss .
Let P XY a probability measure on X × Y where X ⊂ R n is a bounded set.
Let ( x i , y i ) 1 ≤ i ≤ p be a sample of p iid random variables with law P
XY .
Let L be a Lipschitz loss function over R × Y .
We deﬁne:
[Equation]
Then the empirical loss E p ( f ) converges to the test loss E ∞ ( f ) (taking the limit p → ∞ ):
[Equation]
Proof of Theorem 4 is an application of Glivenko-Cantelli theorem.
Proof.
We proved in Proposition 2 that the minimum of equation 2 is attained, so we replace inf by min for the Lipschitz loss function L .
We restrict ourselves to a subset of Lip L ( X , R ) on which (cid:107) f (cid:107) ∞ ≤ 2 L diam X because the minimum lies in this subspace.
We have:
[Equation]
Let g y ( x ) =
L ( f ( x ) , y ) .
Note that g is also Lipschitz and bounded on X .
The entropy with bracket (see [79], Chapter 2.1) of the class of functions G = { g y
=
L ◦ f | f ∈ Lip L ( X , R ) , y ∈
Y , X bounded and (cid:107) f (cid:107) ∞ ≤ 2 L diam X} is ﬁnite (see [79], Chapter 3.2).
Consequently G is
Glivenko-Cantelli.
Finally max f |E p ( f ) − E ∞ ( f ) | a.s −−→ 0 which concludes the proof.
Results of Table 1 .
Loss L hkrm,λ still belong to Glivenko-Cantelli classes as sum of functions L W and
L Hm from Glivenko-Cantelli classes (on same distribution P X ).
C.2 Proofs of Section
[-of Section 5.
-]
Proposition 5.
Optimizing BCE over AllNet leads to divergence .
Let f t be a sequence of neural networks, that minimizes the BCE over a non-trivial training set (at least two different examples with different labels) of size p , i.e assume that:
[Equation]
Let L t be the Lipschitz constant of f t .
Then lim t →∞ L t = + ∞ .
There is at least one weight matrix
W such that lim t →∞ (cid:107) W t
[Equation]
∞ .
Furthermore, the predicted probabilities are saturated:
[Equation]
The proof of Proposition 5 only requires to take a look at the logits of two examples having different labels.
Proof.
Let t ∈ N .
For the pair i, j , as y i (cid:54) =
y j , by positivity of L we must have:
[Equation]
As the right hand side has limit zero, we have:
[Equation] [Equation]
Consequently lim t →∞ | f t ( x i ) − f t ( x j ) | = + ∞ .
By deﬁnition L t ≥ | f t ( x i ) − f t ( x j ) | (cid:107) x
i − x j (cid:107)
[Equation]
= + ∞ .
We can always ﬁnd a network reaching arbitrary small loss on the train set, and arbitrary high loss on the test set.
Hence, minimization of train loss does not guarantee generalization.
Proposition 9 (AllNet networks can always overﬁt) .
Assume that distributions P and Q admit a pdf.
Let n ∈ N , M > 0 and (cid:15) > 0 .
Let ( x i , y i ) 1 ≤ i ≤ p be a sample of p iid random variables with law P
XY with x i (cid:54) =
x j for all i (cid:54) =
j .
Then there exists f ∗ ∈ AllNet such that:
[Equation]
and
[Equation]
Proof.
The proof follows the strategy of Proposition 5.
Let d = min 1 ≤ i,j ≤ n i (cid:54) =
j (cid:107) x
i − x j (cid:107) the minimum distance between dataset points.
We extend the dataset with a new point ( x n +1 , y = 1)) chosen such that (cid:107) x
j − x n +1 (cid:107) ≥ δ 2 for all 1 ≤ j ≤ n .
Then we transform this collections of n +1 Diracs functions (cid:80) n +1 i = 1 n +1 δ x i into a a distribution P that admits a pdf by replacing each Dirac with the uniform distribution over the ball of radius r = d 6 which yields P = (cid:80) n +1 i = 1 n +1 U ( B ( x i , r )) .
All the balls are disjoint so it exists f ∈ AllNet such that sign f ( x i ) =
y i for all 1 ≤ i ≤ n and sign f ( x n +1 ) =
− 1 .
Now let | f ( x i ) | → ∞ to guarantee that E p ( f ) → 0 and E ∞ ( f ) → ∞ .
Fortunately, as soon as the deep learning practitioner restricts itself to a subset of architectures of bounded size, the Proposition 9 is no longer relevant.
However, this theorem suggests that if one wants to beneﬁt from useful generalization guarantees, one must keep the architecture of the network ﬁxed once for all while increasing the training set size.
This contradicts the trend in deep learning community to use bigger and bigger models when more data becomes available (Resnet-152,
GPT3).
In the light of this observation, the existence of adversarial attacks should be an expected phenomenon.
Lipschitz networks, on the other side, beneﬁt from Proposition 4: minimization of train loss implies minimization of test loss.
Conversely, if the test loss is high and the sample size huge, it means that the train loss is high too.
C.3 VC dimension for Lipschitz classiﬁers with margin
We recall below the deﬁnition of the Vapnik-Chervonenkis dimension [34] of a class of hypothesis, that build upon shattered sets.
Deﬁnition 9 ( Set shattered by an hypothesis class )
Let Y =
− 1 , +1 } .
Let H be a class of hypothesis - that is, a set of functions X
→ Y .
The set of points ( x i ) 1 ≤ i ≤ N ∈ X N is said to be shattered by H if for every sequence of labels ( y i ) 1 ≤ i ≤ N ∈ Y N , there exists an hypothesis h ∈ H such that for every 1 ≤ i ≤ N we have h ( x i ) =
y i .
Deﬁnition 10 ( Vapnik-Chervonenkis dimension )
The VC dimension of H , denoted V C dim ( H ) , is the greatest integer N ∈ N such that it exists a sequence of points ( x i ) 1 ≤ i ≤ N ∈ X N shattered by H .
Roughly speaking, the VC dimension of H is the size of the biggest set of points such that H agrees with any label assignment on this set of points.
It measures the capacity of a set of classiﬁers H to separate some sets of points.
The interest of VC dimension introduced in Deﬁnition 10 is its link with Probably Approximately Correct (PAC) learning [10].
Deﬁnition 11 ( Agnostic Probably Approximately Correct (PAC) learnability )
An hypothesis class H of functions X
→ Y is PAC learnable if there exists a function m H : (0 , 1) 2 → N and a learning algorithm D (cid:55)→ h m such that for every ( e, β ) ∈ (0 , 1) 2 , for any distribution P XY on X × Y , for any dataset D
=
(( x 1 , y 1 ) , ( x 2 , y 2 ) , . .
, ( x m , y m ))
iid ∼ P XY of size m ≥ m H
( e, β ) , we have:
[Equation]
We denote by E P XY ( h ) :
=
E ( x,y ) ∼ P XY [ 1 { h ( x ) (cid:54) =
y } ] the empirical risk: the expectation of error function over P XY .
Roughly speaking, for an agnostic PAC learnable class, the probability to pick the best hypothesis h ∗ ∈ H up to error e > 0 happens with probability at least 1 − β > 0 over datasets of size at least m H ( e, β ) sampled from distribution P XY .
This deﬁnition captures the hypothesis classes that are “small enough” such that a reasonably high number of samples allows you to pick the best hypothesis by high probability.
The implication “ﬁnite VC dimension” =
⇒ “agnostic PAC learnable” is a classical result from [80].
This motivates to compute the VC dimension of Lipschitz classiﬁers: it yields PAC learnability results.
Proposition 6. 1-Lipschitz Functions with margin are PAC learnable .
Assume P and Q have bounded support X .
Let m > 0 the margin.
Let C m ( X ) =
{ c mf : X → {− 1 , ⊥ , +1 } , f ∈
Lip 1 ( X , R ) } be the hypothesis class deﬁned as follow. [Equation] ⊥ [Equation]
, meaning “ f doesn’t feel conﬁdent”.
[Equation]
Let B be the unit ball.
Then the VC dimension of C m is ﬁnite:
[Equation]
Proof.
This approach with margins m yields objects known in the literature as m -fat shattering sets [71].
The VC dimension of C m ( X ) is the maximum size of a set shattered by C m ( X ) .
As the functions f are 1-Lipschitz, if c mf ( x ) = − c mf ( y ) then f ( x ) ≥ m , f ( y ) ≤ m and (cid:107) x − y (cid:107) ≥ 2 m .
Consequently, a ﬁnite set X ⊂ X n is shattered by C m ( X ) if and only if for all x,
y ∈ X we have B ( x, m ) ∩
B ( y, m )
= ∅ where B ( x, m ) is the open ball of center x and radius m .
The maximum number of disjoint balls of radius m that ﬁt inside X is known as the packing number of X with radius m .
X is bounded, hence its packing number is ﬁnite.
The bounds on the packing number are a direct application of [81] (Lemma 1).
C.4 VC dimension for GroupSort networks
With GroupSort2 activation functions (as in the work of [36]) we get the following rough upper bound:
Proposition 7.
VC dimension of LipNet1 neural networks .
Let f θ : R n → R a LipNet1 neural network with parameters θ ∈ Θ , with GroupSort2 activation functions, and a total of W neurons.
Let H =
{ sign f θ | θ ∈ Θ } the hypothesis class spanned by this architecture.
Then we have:
[Equation]
From Proposition 7 we can derive generalization bounds using PAC theory.
Note that most results on
VC dimension of neural network use the hypothesis that the activation function is applied elementwise (such as in [35]) and get asymptotically tighter lower bounds for ReLU case.
Such hypothesis does not apply anymore here, however we believe that this preliminary result can be strengthened.
Our result is actually a bit more general and applies more broadly to activation functions that piece-wise linear and partition the input space into convex sets.
The proof of Proposition 7 uses the number of afﬁne pieces generated by GroupSort2 activation function, and the VC dimension of piecewise afﬁne classiﬁers with convex regions.
Proof.
First, we need the following lemma.
Lemma 1 (Piecewise afﬁne function) .
Let H a class of classiﬁers that are piecewise afﬁne, such that the pieces form a convex partition of R n with B pieces (each piece of the partition is a convex set).
Then we have:
[Equation]
The proof of Lemma 1 is detailed below.
Let G ( N ) be the growth function [82] of H .
According to Sauer’s lemma [82] if it grows polynomially with the number of points, then the degree of the polynomial is an upper bound on the VC dimension.
We will show that is indeed the case by computing a crude upper bound of the degree.
Assume that we are given N points, and N big enough such that Sauer’s lemma can be applied.
Assume that we can choose freely the convex partition, and then only the afﬁne classiﬁer inside each piece.
In general for neural networks that might not be the case (the boundary between partitions depends of the afﬁne functions inside it, since neural networks are continuous); however, we are only interested in an upper bound so we can consider this generalization.
Each piece of the partition is a polytope [83].
Each polytope is characterized by a set of exactly
B − 1 afﬁne inequalities since each polytope is the intersection of B − 1 halfspaces [83].
The whole partition is characterized by B ( B − 1)afﬁne inequalities.
We divide by two because of the symmetry.
Hence there exists an injective map from the set of convex partitions with B pieces into ( R n +1 ) B ( B − 1).
It is not a bijective map in general, since different systems might describe the same partition, and some degenerate systems do not correspond to partitions at all.
We split the problem and consider each one of the B ( B − 1)inequalities independently.
According to
Sauer’s lemma, there is O ( N n +1 ) ways to place the ﬁrst hyperplane characterizing the ﬁrst halfspace.
Idem for the second hyperplane, and so on.
Hence, there is at most O (( N n +1 ) B ( B − 1)) ways to assign the N points to the B convex bodies.
Each convex body (among the B of them) contains atmost N points, on which (still according to
Sauer’s lemma) there is at most O ( N n +1 ) way to assign them labels +1 or − 1 , since the classiﬁer is piecewise afﬁne .
Consequently, we have G ( N ) =
O (( N n +1 ) B ( B − 1)( N n +1 ) B ) =
O (( N n +1 ) B ( B
[Equation]
) =
O (( N n +1 ) B 2 ) .
Sauer’s lemma allows us to conclude:
[Equation]
Proof of the result.
Now, we need to prove that f is piecewise afﬁne and the number of such pieces is not greater than (cid:81)
ki
=1 2 w i 2 = √ 2 W , where w i is the number of neurons in layer i .
We proceed by induction on the depth of the neural network.
For depth K =
we have an afﬁne function R n → R which contains only one afﬁne piece by deﬁnition (the whole domain), so the result is true.
Now assume that a neural network R w 1 → R of depth K with widths w 2 w 3 . . .
w k has S k afﬁne pieces.
The enumeration starting at w 2 is not a mistake: we pursue the induction for a neural network R n → R of depth K + 1 and widths w 1 w 2 . . .
w k .
The composition of afﬁne function is afﬁne, hence applying an afﬁne transformation R n → R w 1 preserves the number of pieces.
The analysis falls back to the number of distinct afﬁne pieces created by GroupSort2 activation function.
If such activation function creates S pieces then we have the immediate bound S K +1 ≤ SS k .
Let ( Jf )( x ) ∈ R w 1 × w 1 be the Jacobian of the GroupSort2 operation evaluated in x .
The cardinal |{ ( Jf )( x ) , x ∈ R w 1 }| is the number of distinct afﬁne pieces.
For GroupSort2 we have combinations of w i 2 MinMax gates.
Each MinMax gate is deﬁned on R 2 and contains two pieces: one on which the gate behaves like identity and the other one on which the gate behaves like a transposition.
Consequently we have S k +1 ≤ 2 wk 2 S k and unrolling the recurrence yields the desired result.
Finally, we just need to apply the Lemma 1 with B = √ 2 W .
C.5 Generalization bounds literature survey
In [70] a link is established between Lipschitz classiﬁers and linear large-margin classiﬁers.
Generalization bounds for large class of Lipschitz classiﬁers are provided by the work of [71] using
Vapnik–Chervonenkis theory.
Other generalization bounds related to spectral normalization can be found in [72].
Links between adversarial robustness, large margins classiﬁers and optimization bias are studied in [75, 73, 74].
The importance of the loss in adversarial robustness is studied in [76].
In [16], the control of Lipschitz constant and margins is used to guarantee robustness against attacks.
A link between classiﬁcation and optimal transport is established in [8] by considering a hinge regularized version of the Kantorovich-Rubinstein dual objective.
D Deel.Lip networks
The theorem 3 of [11] bound the (cid:107) · (cid:107) 2 (cid:41) ∞ [12] and (cid:107) · (cid:107) ∞ norms of weight matrices to obtain universal approximation in Lip 1 ( X , R ) .
In practice, they reported that bounding spectral norm (cid:107) · (cid:107) 2 and enforcing orthogonality of rows/columns of weight matrices yielded the best empirical results, because it turned the network into a Gradient Norm Preserving network.
Unfortunately, this last construction still lacks universal approximation results.
Nonetheless, neither [11] nor ourselves encountered (so far) a function that couldn’t be approximated by those GNP networks.
All the experiments done in the paper use the Deel-Lip 3 library [8], following ideas of [11].
The networks use 1) orthogonal matrices and 2) GroupSort2 activation.
Orthogonalization is enforced using Spectral normalization [14] and Björck algorithm
We have for all i :
[Equation]
The networks parameterized by this library are GNP and belong to LipNet1 by construction.
The implementation of Lipschitz neural networks beneﬁts from a rich literature.
We outline below the most signiﬁcant results and contributions of literature, that motivated us to use Deel-Lip library.
LipNet1 networks parametrization.
The Lipschitz constant of afﬁne layers can be constrained with a Gradient penalty [37] (WGAN) or spectral regularization [38], without formal guarantee, only a very crude upper bound.
Weight clipping [6] (WGAN), Frobenius normalization [39] and spectral normalization [14] lead to a tighter upper bound.
However, naively stacking such layers leads to vanishing gradients.
Most activation functions are Lipschitz, the popular including ReLU, sigmoid, tanh, softplus; layers such as Attention are not Lipschitz [40].
Lipschitz recurrent units have been proposed in [41, 42].
Residual connections are Lipschitz but prone to vanishing gradients (see
Appendix F).
Gradient Norm Preserving networks.
In [37], authors show that the potential f of the
Kantorovich-Rubinstein dual transport problem veriﬁes (cid:107)∇ x f ( x ) (cid:107) =
1 almost everywhere on the support of the distributions P XY .LipNet
networks fulﬁlling (cid:107)∇ x f ( x ) (cid:107) =[-
-]
1 almost everywhere wrt any intermediate activation x are said to be Gradient Norm Preserving (GNP), and elegantly avoids the vanishing gradients phenomenon [3, 45].
This is typically achieved in afﬁne layers with orthogonal matrices, which justify the “orthogonal neural network” terminology [46, 47].
[11] establish that GNP networks with ReLU are exactly afﬁne functions.
They proposed Sorting activation functions to circumvent this expressiveness issue.
In particular GroupSort2 revealed to be an efﬁcient alternative [36] to ReLU, and can be seen as a particular case of Householder reﬂections [49, 50].
Other authors tried to ﬁx ReLU itself [43].
Orthogonal kernels are of special interest in the context of normalizing ﬂows [58], ensemble methods [59], reinforcement learning [60] or graph neural networks [61].
The optimization over the orthogonal group (known as Stiefel manifold) has been extensively studied in [62], while [63, 64, 64, 65, 66] focus on neural networks retractions like Cayley transform or exponential map; more recently [67] proposed a landing algorithm, [68] proposed an algorithm inspired by quantum computing, and [69] proposed an approach based on graph matching.
Orthogonal convolutions are still an active research area: the constraint is enforced by using appropriate regularization [51, 52], by expressing convolutions in Fourrier space [53, 54], or by optimizing over the set of orthogonal convolutions directly [3, 55, 56].
In order to build CNN we used the convolution layers already provided in Deel-Lip.
One limitation of these layers is that it uses the Reshaped Kernel Orthogonalization (RKO) [3] method which, although it ensures Lipschitz bounds, does not guarantee exact orthogonality.
We also attempted to use Skew Orthogonal Convolutions (SOC, as described in algorithm 1 of [56]).
However, when we performed a sanity-check with the power iteration method, we obtained convolutions with Lipschitz constant greater than 1 .
As the method used to build 1-Lipschitz networks does not affect the conclusions of our paper, we decided to stick with Lipschitz constant provably smaller than one.
We did not observe improvements by using Householder activation functions, so we used GroupSort2 activation functions instead (which are computationally cheaper).
E Multiclass Hinge Kantorovich Rubinstein
The loss HKR proposed by [8] was originally designed for binary classiﬁcation.
There are several ways to adapt it to the multi-class K > 2 setting.
The most obvious one would be a one-versus-all scheme.
However, in multiclass classiﬁcation the prediction is given by arg max f k and not by sign ◦ f , so f − 1 ( { 0 } ) is not longer the frontier.
Consequently, this approach fails to yield meaningful certiﬁcates.
Instead the construction of HKR loss should once again rely on Multiclass Mean Certiﬁcate Robustness (see Deﬁnition 12).
Indeed, the robustness radius δ for class k veriﬁes:
[Equation]
The 12 comes from the fact that each f i is 1-Lipschitz, so their difference is 2 -Lipschitz at most.
This deﬁnition is coherent with the one of multiclass hinge loss found in most frameworks.
We compare the logits of the true class with the ones of the closest other class to weight the certiﬁcate positively or negatively according to the true label.
Deﬁnition 12 (Multiclass Mean Certiﬁable Robustness (MMCR))
For any function f : X → R ∈
LipNet1 we deﬁne its weighted multiclass mean certiﬁable robustness
R ( P,y ) ( f ) on class P with label k as:
[Equation]
Note that f k ( x ) − arg max i (cid:54) =
k f i ( x ) is either positive or negative, according to the prediction.
Then we deﬁne the Multiclass HKR:
Deﬁnition 13 (Multiclass HKR)
For class an example x of label k let:
[Equation]
We deﬁne the multiclass HKR as:
[Equation]
For K = 2 we recover the binary case on the function ˆ f = f 1 − f 2 .
Experiments showed that the one versus all approach was outperformed by the multiclass HKR, in both robust accuracy and training time.
All f k functions can be learned independently, however in practice they share the same Lipschitz backbone and only differ in the last layer, as early experiments showed that it did not impact negatively the results.
Using the same arguments as Proposition 2 based on Arzelà-Ascoli theorem we show that the minimum of L Mλ is well deﬁned and attained for each f k .
F Gradient Norm Preserving (GNP) networks
Vanishing and Exploding gradients have been a long-time issue in the training of neural networks.
The latter is usually avoided by regularizing the weights of the networks and using bounded losses, while the former can be avoided using residual connections (such ideas can found on LSTM [84] or
ResNet [85]).
On Gradient Norm Preserving (GNP) networks (orthogonal networks with GroupSort activation such as the ones of Deel.lip library), we can guarantee the absence of exploding gradient:
Proposition 10 (No exploding gradients [3]) .
Assume that f = h M ◦ h M − 1 ◦ . . .
◦
h 2 ◦ h 1 is a feedforward neural network and that each layer h i is 1-Lipschitz, where h i is either a 1-Lipschitz afﬁne transformation
h i ( x ) =
W i x + B i either a 1-Lipschitz activation function.
Let L : R k × Y → R the loss function.
Let ˜ y
= f ( x ) , H i
=
h
i ◦ h
i − 1 ◦ . . .
◦
h 2 ◦ h 1 and H 0 ( x ) = x .
Then we have:
[Equation]
To prove Proposition 10 we just need to write the chain rule.
Proof.
The gradient is computed using chain rule.
Let θ be any parameter of layer
h
i .
Let h j ⊥ be a dummy variable corresponding to the input of layer h j , which is also the output of layer
h j − 1 .
Then we have:
[Equation]
with M = (cid:16)(cid:81) i +1 j
= M J h
j ⊥ h j ( H j − 1 ( x )) (cid:17) .
As the layers of the neural network are all 1-Lipschitz, we have:
[Equation]
Hence we get the following inequality:
[Equation]
Finally, for h i ( H i − 1 ( x ))
=
W i H
i − 1 ( x ) + B
i we replace θ by the appropriate parameter which yields the desired result.
There is still a risk of vanishing gradient, which strongly depends of the loss L .
For Lipschitz neural networks, BCE L T does not suffer from vanishing gradient.
Proposition 11 (No vanishing BCE gradients) .
Let ( x i , y i ) 1 ≤ i ≤ p be a non trivial training set (i.e with more than one class) such that x i ∈ X , X a bounded subset of R n .
Then there exists a constant
K > 0 such that, for every minimizer f ∗ L of BCE (known to exist thanks to Proposition 2) we have:
[Equation]
And such that for every 1 ≤ i ≤ p we have the following:
[Equation]
Note that K only depends of the training set, not f ∗ L .
Proof.
Note that it exists K (cid:48) > 0 such that | f ∗ L ( x i ) | ≤ K (cid:48) for all x i and all minimizers f ∗ L , just like in the proof of Proposition 2, because otherwise we could exhibit a sequence of minimizers ( f ∗ L ) t not uniformly bounded, which is a contradiction.
Consequently | ∂∂ ˜
[Equation]
It means that a non-null gradient will remain for each training example taken independently, but their mean over the train set after convergence will be the null vector.
Consequently, we must expect high variance in gradients and oscillations when we get closer to the minimum.
We used VGG-like architectures instead of Resnet because GNP property makes residuals connections less useful overall (no need for shortcuts when gradient is preserved), and because those residual connections can actually be harmful:
Remark (Residual connections) .
If f veriﬁes (cid:107)∇ x f ( x ) (cid:107) = 1 almost everywhere, and if g veriﬁes (cid:107)∇ x g ( x ) (cid:107) = 1 almost everywhere, then (cid:107)∇ x ( 12 f ( x ) + 12 g ( x )) (cid:107) < 1 in general, unless ∇ x f ( x ) = ∇ x g ( x ) .
Taking f ( x ) =
x we end up with residual connections, for which ensuring (cid:107)∇ x ( 12 f ( x ) + 12 g ( x )) (cid:107)
=
1 almost everywhere is not possible unless f = g .
Remark essentially shows that the set of GNP layers is not stable by sum or other common operations.
This makes their practical implementation and the demonstration of universal approximation theorems trickier.
G Wasserstein discriminator does not depend of the Lipschitz constant
The dual problem can be reformulated by swapping the objective and the constraint:
[Equation]
(29) (cid:15) can be seen as re-scaling (change of units in physicist vocabulary).
This makes more clear the fact that changing the Lipschitz constant is just changing the units used to measure distance.
The invariance by dilation mentioned in Section 4 must be understood in this sense: any constant L can be chosen for the computation of W 1 as long as this constant is chosen in advance and bounded throughout the optimization process.
H BCE through the lens of OT
In the following, we try to draw links between BCE minimization and optimal transport.
Since the objective function is optimized with gradient descent, the gradients of the loss is the object of interest.
We re-introduce f θ as a function parameterized by θ , mapping the input to the logits.
Let g pθ ( x ) =
σ ( f θ ( x )) and g qθ ( x ) = 1 − σ ( f θ ( x )) .
g pθ ( x ) (resp.
g qθ ( x ) ) are the predicted probabilities of class +1 (resp.
Now deﬁne Z pθ =
E x ∼ P [ g qθ ( x )] and Z qθ =
E x ∼ Q [ g pθ ( x )] .
Z pθ can be seen as the weighted rate of false negatives.
That is, the average mass of probability given to class − 1 by f θ when examples are sampled from class +1 .
Similarly, Z qθ can be seen as the rate of false positives.
Let:
[Equation]
Consequently, P θ (resp.
Q θ ) is a valid probability distribution on R n corresponding to the probability of an example x to be incorrectly classiﬁed in class − 1 (resp.
+1 ).
With these notations, the full expression of the gradient takes a simple form.
Behold the minus sign: it is a gradient descent and not a gradient ascent .
[Equation]
We apply a bias term T ∈ R to classify with f θ − T instead.
For a well-chosen T we can enforce
Z pθ =
Z qθ , and such T can be found using the bisection method.
The optimization is performed over the set of 1-Lipschitz functions.
We end up with:
[Equation]
This is the gradient for the computation of Wasserstein metric W between P θ and Q θ , using
Rubinstein-Kantorovich dual formulation.
Hence, binary cross-entropy minimization is similar to the computation of a transportation plan between errors distributions P θ and Q θ .
Note that P θ and
Q θ depend of the current classiﬁer f θ − T , so the problem is not stationary.
Finally, observe that L bceτ ( f ( x ) , y ) =
log 2 − yτf ( x ) [Equation]
τ
[Equation]
f
[Equation]
x
[Equation]
so when τ → 0 we get:
[Equation]
[Table]
[Figure]
[Table]
[Figure] [Table]
In the limit of small temperatures, the Binary Cross-Entropy is essentially equivalent to Wasserstein.
In AllNet networks, as the training proceeds, the Lipschitz constant increases (equivalently increasing τ ) and the loss self-correct with P θ and Q θ to improve accuracy.
I Fitting CIFAR100 with random labels
This experiment illustrates that constraining the Lipschitz of a network does not affect its expressive power.
To show this we train a constrained network on the CIFAR100 dataset where all labels have been replaced with random labels, this task is now a widely recognized benchmark to evaluate the expressiveness of an architecture [86].
The architecture of this network is as simple as possible: two orthogonal dense layers withneurons are followed by a dense layer which is normalized but not orthogonal.
The GroupSort activation function is used and biases are enabled.
Hyper parameters for this experiment are listed in table 7a, and results are reported in table 7b.
At ﬁrst glance it’s might seem surprising to see both high accuracy and high provable robustness on a dataset with random labels.
This is compliant with the idea expresses by the authors of [24]: for a given accuracy one can increase the robustness radius around a sample x 1 up to the value (cid:107) x 1 − x 2(cid:107) where x 2 is the closest sample with a different label.
The decision frontier is close to the decision frontier of the 1-nearest neighbor based on the trained set.
This illustrates that constraining the Lipschitz constant does not necessarily decrease accuracy and does not necessarily increase robustness.
Also, it shows that there is no trivial link between robustness and generalization.
J 1-Lipschitz estimators are consistent (experimental protocol of ﬁgure 4)
Lipschitz classiﬁers are consistent: as the size of the training set increases, the training loss becomes a proxy for the test loss.
However, we do not give convergence speed bounds: we do not know how many samples are needed for a given task to observe the convergence between train and test losses.
Moreover, the losses are parametrized (e.g by τ, α, m ) so we expect to have different convergence rates, depending on those parameters.
In order to observe this empirically on the CIFAR10 dataset, the same architecture (described in ﬁg 9a) was trained successively on 2%, 5%, 10%, 25%, 50% and 100% of the dataset.
The sub-sampling was performed with a different seed each time, showing that, for this range of τ training is still stable.
Similarly, this procedure has been repeated with different values for τ .
Hyper-parameters used for learning are reported in ﬁg 9b.
Learning results are reported in Figure 8.
It also shows certiﬁable accuracy and empirical accuracy, that were not displayed in the ﬁg 4 from the main paper.
We see that lower values for τ yield tighter robustness certiﬁcates (certiﬁcate value is close to the distance found by L2-PGD).
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
K Controlling the accuracy/robustness tradeoff (experimental protocol of ﬁgure 3)
A small CNN architecture, described more precisely in Figure 11a was trained multiple times with different losses and different loss parameters.
Besides the learning rate, other parameters were left unchanged, and are depicted in Figure 11b.
The learning rate is chosen depending on the loss parameters: when trained with CCE, changing τ implicitly changes the norm of the gradient, thus when doubling τ one must divide the learning rate by a factor of two.
The same phenomenon occurs with the α parameter of HKR.
We kept m = 20 ﬁxed for HKR, and we tuned α .
For each run the ﬁnal validation accuracy is reported on x -axis while the provable accuracy at (cid:15) =is reported on y axis.
The choice of the robustness metric was set to the robust accuracy at (cid:15) =
because of its wide use in the community.
However, other robustness metrics also yield a Pareto front: two examples are shown in Figures 10a and 10b.
In those examples, we also test different combinations of values for m in HKR.
Note that this Pareto front can also be inﬂuenced by other factors: training larger architectures could improve both accuracy and robustness.
Similarly, data augmentation has an impact on accuracy and robustness, but studying the phenomenon is out of the scope of this paper.
Finally, it is important to note that, the comparison between two architectures (or two robustness methods) cannot be done properly with a single training (and ﬁxed hyper-parameters): comparing their Pareto front is more relevant.
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure]
L Hardware
Toy experiments depicted in Fig.
1b, Fig.
1a, Fig. 2 and example 1 were run on a personal workstation with NVIDIA Geforce 1080 GTX and 8GB VRAM, 16 cores Xeon and 32GB RAM.
Large scales experiments depicted in Fig.
4, Fig.3 were run on Google Cloud with TPU v2-8.
For reference, the experiments with CNNs on CIFAR10 (appendix K), took 4.9s per epoch on average.
Tensorﬂow framework was used in every experiment but the one of Example 1, where Jax was used instead (because order2 and ﬂoat64 experiments are easier to write in this library).
M Divergence of the weights on AllNet networks
In this example, we illustrate that example1 behavior can be observed at larger scale on MNIST with a ConvNet of AllNet .
We used 3 × 3 convolution ﬁlters of widths 32 (cid:41) 64 with MaxPool and ReLU , followed by a ﬂattening operation and densely connected layers of widths 256 (cid:41) 10 .
Newton’s method cannot be used due to its memory requirements on ConvNet.
We tested SGD with learning rate η = 0 .
1 and momentum m
= 0 .
9 , and Adam with learning rate η = 1 e − 3 and other default parameters.
Experiments were run both in ﬂoat32 and ﬂoat64 precision.
We monitor the maximum spectral norm of the weights of the network throughout training for each epoch t ∈ N :
[Equation]
We report M t as function of epoch t in Figure 12.
The validation accuracy is above 98% after the ﬁrst epoch, and ﬂuctuates between 98.5% and 99.5% during the following epochs (in either cases).
Similarly the validation loss ﬂuctuates between 1 e − 1 and 1 e − 3 .
We see that on this simple task the spectral norm of weight matrices continues to grow indeﬁnitely, even though the classiﬁer is almost perfect after one epoch.
Interestingly, on this experiment the vanishing gradient phenomenon cannot be observed after 25 epochs and the results are robust with respect to the precision of the ﬂoating point arithmetic.
This is compliant with the observations made in the literature about the high Lipschitz constant of
AllNet networks [23].
We observe that Adam makes the problem worse, even if its learning rate is smaller.
This may explain why many practitioners reported that Adam was more susceptible to overﬁt than SGD with a carefully tuned learning rate scheduling.
N Stability of training of LipNet
To check this, we trained different LipNet1 networks, either by tuning the value of temperature τ , or by tuning the number of ﬁlters in convolutional layers.
We used Fashion-Mnist dataset.
N.1 Moving along Pareto front by tuning temperature
In this experiment, we explore the stability of training with respect to the loss parameter (here we use CCE with τ ).
To do so we perform a scheduling on τ : each network of the experiment is trained with a ﬁxed τ for 200 epochs, then τ is increased/decreased linearly by a factor of 2 for 200 more epochs (see 13. We train a total of 12 LipNet1 networks with the same architecture: three blocks of two convolutions, bias and group sort are followed by a Pooling layer (L2NormPooling) ﬁnally followed by a ﬂatten and a Dense layer (architecture synthesized as c32-c32-P-c64-c64-P-c128-c128-D256 ). Each training is performed with the same optimizer (Adam) and the same learning rate (0.001).
Each dot in the graph of Figure 16 corresponds to the metrics of a network after one epoch.
The validation accuracy can be found on x-axis and MCR metric on y-axis
During the ﬁrst epochs, the dots can be found inside the region delimited by the Pareto front: the network has not converged yet, and both Mean Certiﬁable Robustness and validation accuracy are low.
After few epochs, the dots start to accumulate on the Pareto front.
Then, the value of temperature τ is tuned during the training, from initial τ start to τ ﬁnal .
Each of the color corresponds to a different value of Tau.
We see that the temperature can be modiﬁed during training to move along the Pareto front.
We can get a closer look at the trajectory of two networks, which are reported in ﬁg 15 to better illustrate this phenomenon.
Despite their starting point being different, they end up on a minimum with the same MCR/accuracy tradeoff.
It seems that the position on the Pareto front only depends on the value of τ ﬁnal .
Not only the functions are similar at a global scale, but it is also valid at a local scale: while the two nets are only 77% accurate, they agree on 93% of the validation samples (ie. they make the same error on the same sample).
[Figure]
as the is the same at the
Although these only have 69% accuracy their
Observe that all dots tend to accumulate on the Pareto front, even though they aredifferent networks being trained.
It suggests that this method is stable with respect to the input seed.
Some of the networks trained with high τ start (for example τ start = 8 and τ ﬁnal = 16 ) seem to “lag behind”: empirically we observe that more epochs are required to make the network converge.
Hence, the speed at which τ is modiﬁed must be scaled appropriately to ensure that the best Pareto front is recovered.
This experiment also suggests that a curriculum can be a satisfying approach to tune τ : an expensive grid search over τ could be replaced by a single training with a scheduler placed on τ .
N.2 Shifting Pareto front by tuning architecture
In this experiment we explore an important question: what happens when the architecture is changed?
To explore this we perform the same experiment as N, but with smaller and larger architectures: the architectures are denoted by the number of ﬁlters in their ﬁrst convolutions (ﬁlter of other block are adjusted accordingly by doubling the number of ﬁlter of the previous block).
It shows how the expressiveness of the architectures affects the Pareto front.
We report the results in Figure 17.
The validation accuracy can be found on x-axis and the MCR on y-axis .
dot corresponds to an epoch/a network.
Different colors correspond to different architecture widths.
As expected, larger networks are more expressive, and as a result, the Pareto front is shifted toward higher accuracy and higher robustness.
This observation holds for every scheduling τ .
The
MCR/accuracy is also architecture dependent.