I MPROVED ROBUSTNESS TO ADVERSARIAL EXAMPLES USING L IPSCHITZ REGULARIZATION OF THE LOSS
Anonymous authors
Paper under double-blind review
A BSTRACT
We augment adversarial training (AT) with worst case adversarial training (WCAT) which improves adversarial robustness by 11% over the current stateof-the-art result in the (cid:96) 2 norm on CIFAR-10.
We interpret adversarial training as Total Variation Regularization, which is a fundamental tool in mathematical image processing, and WCAT as Lipschitz regularization , which appears in Image
Inpainting.
We obtain veriﬁable worst and average case robustness guarantees, based on the expected and maximum values of the norm of the gradient of the loss.
I NTRODUCTION
We augment adversarial training (AT) with worst case adversarial training (WCAT) which improves adversarial robustness by 11% over the current state-of-the-art result (Qian & Wegman, 2018) in the (cid:96) 2 norm.
The method also achieves results comparable to the state-of-the-art results of Madry et al. (2017) in the (cid:96) ∞ norm. Moreover, our adversarial training step uses only one gradient evaluation compared to seven steps in the Madry et al.
(2017) work.
The worst case adversarial training method is described as follows. During adversarial training, the gradient of the loss is computed for each perturbed image.
WCAT records the largest of the gradients norms, and adds a penalty to the loss proportional to this term.
In § 4 we show that the norm of the gradient of the loss of the model is a measure of the robustness of a model to adversarial examples. We obtain veriﬁable worst and average case robustness guarantees, based on the expected and maximum values of the norm of the gradient of the loss.
We then compute these quantities empirically on trained models, and demonstrate that improving these quantities leads to proportional improvements in adversarial robustness. Next, we interpret adversarial training as Total Variation Regularization (Rudin et al., 1992), which is a fundamental tool in mathematical image processing, and WCAT as Lipschitz regularization , which appears in Image Inpainting (Bertalmio et al., 2000).
Write (cid:96) ( x ) = (cid:96) ◦
f ( x ) for the loss of the model. Thus with AT and WCAT, the training loss 1 is replaced with
[Equation]
where (cid:15) is the size of the adversarial training perturbation, and λ is the WCAT multiplier.
The dual norm (cid:107) · (cid:107) ∗ is (cid:107) · (cid:107) 1 for attacks measured in (cid:96) ∞ and (cid:107) · (cid:107) 2 for attacks measured in (cid:96) 2 , see § 2.1.
We also show that the third term in (1), corresponding to WCAT, is an approximation of the Lipschitz constant of the loss of the model,
[Equation]
This inequality is based on Rachemacher’s Theorem, and is derived in § 3.1.
A DVERSARIAL ATTACKS , TRAINING , AND REGULARIZATION
The earliest and most successful defense is adversarial training (Szegedy et al.
; Goodfellow et al.
(2014); Tram`er et al.
; Madry et al.
Top entries in a recent adversarial defence competition (Kurakin et al.
(2017)) used Ensemble Adversarial Training (Tram`er et al.
(2018)), where a model is adversarially trained with inputs generated by an ensemble of other models.
For more background, refer to the recent review Goodfellow et al.
(2018) which discusses defences against adversarial attacks and their limitations.
Adversarial attacks seek to ﬁnd the minimum norm vector which leads to a misclassiﬁcation by the network.
Finding the optimal attack is intractable (Athalye et al., 2018).
An alternative which permits loss gradients to be used is to consider the attack vector measured in a given norm which most increases the loss,
[Equation]
2. AT CORRESPONDS TO TV REGULARIZATION
The following result shows that Signed Gradient (Goodfellow et al., 2014) attack, and the normalized gradient attack are nearly optimal in the following sense.
Lemma 2.1.
Write (cid:96) ( x ) =
(cid:96) ( f ( x ) , y ) . The attack vector directions
[Equation]
, and
[Equation]
are nearly optimal the (cid:96) ∞ and (cid:96) 2 norm, respectively, in the sense that
[Equation]
respectively and [-the solution of (5) is given by a (cid:96) 
-]moreover,
[Equation]
The proof of Lemma 2.1 is in § B.
Adversarial training improves robustness to adversarial attacks by solving the minimax problem
[Equation]
In practice, (5) is approximated by using an attack vector, a , in place of the optimal δ ,
[Equation]
Using the one-step Signed Gradient attack vector given by (3) during adversarial training can be interpreted as regularization by the average of the 1-norm of the gradient of the loss over the data.
This regularization is called Total Variation regularization, and it has been used effectively in image denoising (Rudin et al., 1992).
The next result comes from taking expectations in (4).
Lemma 2.2.
Adversarial training using the ε -scaled attack vector (3) can be interpreted, up to terms of order ε 2 , as training with the augmented loss function
[Equation]
where the (cid:107) · (cid:107) ∗ is the dual norm.
L IPSCHITZ R EGULARIZATION
In this section we interpret WCAT as Lipschitz regularization, which leads to a much more accurate and efﬁcient method for estimating the Lipschitz constant in a deep neural network, compared to other recent methods.
These methods are based on the product of weight matrix norms, whose error grows exponentially in the number of layers, as explained below.
3. WCAT CORRESPONDS TO L IPSCHITZ REGULARIZATION
The basis for this result is Rademacher’s Theorem (Evans, 2018, § 3.1), which states that if a function g ( x ) is Lipschitz continuous then it is differentiable almost everywhere and
[Equation]
We obtain an underestimate of Lip( g ) by sampling the norm of the gradient on a subset of points.
Deﬁnition 3.1 (Method for estimating the Lipschitz constant) .
Let g be a Lipschitz continuous function.
Then
[Equation]
We will apply (6) to (cid:96) ( x ) , the loss of the model, over the full data set when estimating the Lipschitz constant, and with a mini-batch during training.
Remark 3.2 .
Regularization of the loss corresponds to partially regularizing the model, but at a much lower cost.
Since the loss is a scalar, regularizing by the Lipschitz constant of the loss is equivalent to regularization of the model f in one direction.
By the chain rule,
[Equation]
For example, when (cid:96) is the KL divergence, and when f = softmax( z ( x )) then
[Equation]
Thus, in this case, regularizing (cid:96) ( x ) corresponds to regularization of z ( x ) in the direction f ( x ) − y .
3. P REVIOUS METHODS FOR ESTIMATING THE L IPSCHITZ CONSTANT OF A MODEL
It is well known that data independent upper bounds on the Lipschitz constant of the model are available (Bartlett, 1996) using the product of the norm of the weight matrices.
Let W k be the weight matrix of the k -th layer of a network f comprised of N layers, and suppose all non linearities of a network are at most 1-Lipschitz.
Then
[Equation]
with p 0 = p and p n = q .
Certain conditions on the p k ’s must be met.
For a proof with p, q =see Tsuzuku et al. However, for deep models, we have found that this bound is much too large: on the models we considered, this estimate was no smaller than 10 12 and as large as 10 23 . In contrast our empirical results, which are a lower bound (see § D), give values less than 10 . Stability bounds suggest that the true value is closer to 10 than 10 12 .
Recent works use the logarithm of (7): Ciss´e et al.
(2017) uses the distance to the nearest orthonormal matrix;
Miyato et al.
(2018) uses the 2-norm; Gouk et al. (2018) uses either the 1-norm or ∞ -norm; and Qian & Wegman (2018) uses the ∞ -norm. In these works, the estimate is used to penalize the Lipschitz constant of the model during training. Of these four papers, only the implementation of Gouk et al.
accounts for batch normalization.
Since batch normalization multiplies each weight matrix by a diagonal scaling matrix, the other works are missing important terms in their implementations.
A different method for estimating the Lipschitz constant of a pre-trained model was presented in
Weng et al.
(2018), using statistical techniques from extreme value theory. This estimate accurately captures the minimum adversarial distance needed to successfully misclassify an image.
However it requires at a minimum many tens of model evaluations for each image, and so is not tractable as a Lipschitz penalty during training. Lipschitz regularization leads to a proof of model generalization, as recently shown in Oberman & Calder (2018).
Weng et al. and Hein & Andriushchenko (2017) showed that the Lipschitz constant of the network can be used to establish rigorous worst-case bounds on adversarial robustness. In particular Weng et al.
(2018) show that the Lipschitz constant of the model gives a certiﬁable minimum adversarial distance: a successful attack on image x will have adversarial distance at least
[Equation] where L f is the Lipschitz constant of the model, f , and i ∗ is the correct label of x . Thus training models to have small Lipschitz constant should improve adversarial robustness (Hein & Andriushchenko (2017); Tsuzuku et al. Here we prove that the Lipschitz constant of the loss of the network also provides such bounds: the second and third terms in (1) estimate the average and worst case robustness.
The following Lemma shows that the Lipschitz constant of the loss function gives a robustness guarantee for the loss incurred by an adversarial perturbation of norm ε . This is analogous to (8), giving the corresponding robustness result with the Lipschitz constant of the model.
Recall that we write (cid:96) ( x ) = (cid:96) ( f ( x ) , y ) for the composed loss function. Lemma 4.1 (Worst-case stability of network) . Suppose (cid:96) ( x ) is L -Lipschitz continuous.
Let a be any adversarial perturbation of norm ε . Then [Equation] [Equation]
Proof. By Lipschitz continuity of (cid:96) [Equation] There are two cases for the left-hand side, depending on the sign.
In both cases we obtain (9).
We can also see how the adversarial training interpretation of the term in Lemma 2.2 corresponds to expected stability of the network. Lemma 4.2 (Expected stability of network) . Suppose that [Equation]
Let a = a ( x ) be any adversarial perturbation of norm (cid:107) a ( x ) (cid:107) ≤ ε .
Then
[Equation]
Proof.
Using Taylor’s theorem
[Equation]
by dual norm § 2.
[Equation] Taking expectations we obtain (10). Remark 4.3 . The previous two lemmas are useful in the following context.
Suppose we wish to compare two trained models. We can estimate the values R and L for each model.
We expect the ratio of either of these values between models to be a effective prediction of the relative robustness of each model, measured in the average or worst case. For example we expect that the relative robustness of two models, using median adversarial distance, should fall in between the average and the worst case bounds.
Indeed we observe this empirically. For example, after training, we measured the terms in (1) and compared these values for a regularized and an undefended model.
The median adversarial distance on the regularized model trained with (1) improves by a factor of 5 compared to the undefended model.
The bounds given in this section show an improvement of the average robustness by a factor of 3.3, and in the worst case by a factor of 9, consistent with the improvement in median adversarial distance. See § 5 for more details.
[Figure]
In theory, it is possible to extend the data with a function whose Lipschitz constant matches that of the data (or is only slightly larger, depending on the norms used), see § C for more details.
We argue that a model which better approximates the Lipschitz constant of the data brings us closer to the ground truth, in addition to improving robustness.
Lipschitz regularization brings the (estimated)
Lipschitz constant of a network close to the Lipschitz constant of the data on which it was trained.
For example on CIFAR-10, the Lipschitz constants are: 0.36 for the dataset, 1.32 for a regularized model (ResNeXt-34, see § 5), and 13.70 for the undefended model (the latter two measured on the test/validation set).
E MPIRICAL RESULTS
We studied image classiﬁcation on the CIFAR-10 and CIFAR-100 datasets (Krizhevsky & Hinton (2009)).
We tested our methods on three networks, chosen to represent a broad range of architectures: AllCNN-C (Springenberg et al.
(2014)), having nine layers; a 34 layer ResNet (He et al.
(2016)); and a 34 layer ResNeXt (Xie et al.
Training and model details are provided in Appendix A.
In § 5.1 we deﬁne error curves, an error metric which allows for easy comparisons of model robustness and attack strength across a full range of attack norms. Then in § 5.2 we use these curves to rank common attack methods. The curves illustrate a clear ranking of attack methods across models. The change in attack ranking is based on the norm used: Iterative FGSM ( (cid:96) ∞ PGD) is the most effective attack when distance is measured in the (cid:96) ∞ norm; (cid:96) 2 PGD is the most effective attack when distance is measured in the (cid:96) 2 norm.
The attack ranking makes defence robustness easier to assess, since we only need to check one attack per norm for each model. In § 5.3 we compare the different choices of adversarial defences discussed in this paper, which amounts to a combination of adversarial training and Lipschitz regularization in the corresponding norm. Finally, in § 5.4 we compare our results to other works. 5. E RROR CURVES FOR MODELS AND ROBUSTNESS METRICS
The error curve of a model for a given attack measures the robustness of the model as a function of the norm of the attack.
Deﬁnition 5.1.
The error curve C err ( ε ) of the model f for the attack a is the probability over the test data D that an attack of size ε leads to a misclassiﬁcation
C
[Equation]
x ) | for an attack
[Equation]
X ≤ ε } where c ( x ) is the model’s label at x and c ∗ ( x ) is the true label.
See Figure 1 for error curves for an undefended model over a variety of attacks.
[Figure]
We can compare how models or attacks perform using the model error curve.
We report common adversarial statistics, which can be read off the curve, in Table 1. For example the percent misclassiﬁcation at adversarial distance ε = 0 and 0 .
1 (in the 2-norm) are easily read off the error curve.
These values of ε correspond to the test error, and noise which is slightly smaller than a human perceptible perturbation.
We also report the median (cid:96) 2 distance which corresponds to the x -intercept of 50% error on the curve.
5. A TTACK EVALUATION
We attacked models on the test/validation set using seven untargeted attack methods: gradient attack; projected gradient descent (constrained in (cid:96) 2 ); the Fast Gradient Sign Method (FGSM) (Goodfellow et al.
; Iterative FGSM (I-FGSM) (Kurakin et al. (2016)), or projected gradient descent in (cid:96) ∞ ; DeepFool (Moosavi-Dezfooli et al. (2016)); the Carlini-Wagner attack (Carlini & Wagner, 2017); and the Boundary attack (Brendel et al., 2018). The ﬁrst six methods are ﬁrst order gradient based white-box attacks, while the last is a black-box attack.
I-FGSM and the (cid:96) 2 projected gradient attack are iterative gradient methods, whereas FGSM and the gradient attack are single step.
All attacks were implemented with Foolbox (Rauber et al.
Hyperparameters were set to Foolbox defaults, except for the Boundary attack 2 .
For each image and attack method, we calculated the adversarial distance in (cid:96) 2 and (cid:96) ∞ .
On each model, dataset, and regularization method, we tested all seven attack methods on the entire test/validation set.
We compared attack methods using the attack error curve.
For example see
Figure 1, where we plot attack error curves for each attack method on an undefended model.
We plot the attack error curves in both Euclidean and (cid:96) ∞ distances.
When attack norms are measured in (cid:96) ∞ , the most effective attack is Iterative FGSM (PGD in (cid:96) ∞ ). Similarly, when attack norms are measured in (cid:96) 2 , (cid:96) 2 projected gradient descent is the strongest attack.
See Figure 1 for an illustration.
We observed the same ranking of attacks on all models and defences studied.
For this reason in what follows, we only report model statistics using projected gradient descent corresponding to the distance metric used (either (cid:96) 2 or (cid:96) ∞ ).
5. E VALUATION OF DEFENCE METHODS
We tested many combinations of defence methods. We use the superscipts 0 , 1 , 2 to indicate the type of adversarial training used: 0 for none, 1 for one step FGSM, and 2 for one step gradient ascent. [Table]
We used the superscript Lip for Lipschitz regularization, which measured the gradient in the same norm as was used for adversarial training. For example J 2 − Lip indicates adversarial training and Lipschitz regularization in the (cid:96) 2 norm. We also considered adding a ﬁnal sigmoid layer to the network, prior to the softmax .
The choice of sigmoid we choose is tanh , and is inspired by (but not equivalent to)
tanh -estimators used in classical statistics as a robust estimator (Hampel et al., 2011, Chapter 2).
The intuition behind this choice is to normalize the logit scores of the model, which we believe should improve robustness to outliers.
Outside of deep learning, tanh -estimators have been successfully used to normalize scores and improve robustness, for example in machine learning biometrics (Jain et al.
See Appendix A.1 for layer details.
Model robustness is evaluated on the entire test/validation set using the median adversarial distance in both (cid:96) 2 and (cid:96) ∞ , and the percent misclassiﬁed at adversarial (cid:96) 2 distance ε = 0 .
We chose ε = 0 .because at this magnitude attacks are still imperceptible to the human eye.
For ease of comparison with other works, we also report percent misclassiﬁed at (cid:96) 2 distance ε = 1 . 5 , and (on CIFAR-10) (cid:96) ∞ distance ε = 8255 .
In addition we plot the attack error curve for each model.
Table 1 and Figure 2 present a summary of results for ResNeXt-34.
The best statistics are in bold.
Complete results, for all models, datasets, and adversarial defences, are presented in Appendix D. Here we summarize our results for ResNeXt-34, the model studied with the greatest capacity using the most effect attack in the (cid:96) 2 norm, (cid:96) 2 PGD. Refer also to Table 1 for a summary. Without adversarial perturbations, all ResNeXt-34 models achieve roughly 4% test error on CIFAR-10.
However, the undefended (baseline, J 0 ) model achieves 54% test error at adversarial (cid:96) 2 distance ε = 0 .
Adversarial training via FGSM ( J 1 ) reduces test error to 24.6%, whereas (cid:96) 2 adversarial training ( J 2 ) reduces test error to 13.5%.
A combination of all defences ( J 2 − Lip with tanh ) further reduces test error to 12.1%.
The models are ranked in the same order when instead measured with median (cid:96) 2 adversarial distance.
The model with all defences has median adversarial distance six times that of the undefended model.
FGSM ( J 1 ) only doubles the median adversarial distance relative to the baseline undefended model.
Figure 2a illustrates that this ranking of defences holds over all distances of adversarial perturbations.
We observe a similar ranking on CIFAR-100.
See for Figure 2b.
Unperturbed, all models achieve between 21% and 22% test error.
Without adversarial defences, ResNeXt-34 (4x32d) has a test error of 74% at adversarial (cid:96) 2 distance ε = 0 .
Adversarial training alone brings the test error down to 56.3% and 53.7%, with respectively FGSM ( J 1 ) and (cid:96) {+
 ( J+} 2 ) adversarial training.
A combination of all defences further reduces test error to 42.6%.
Median (cid:96) 2 adversarial distance increased from 0. on the undefended model to 0.14 on the model with all defences.
In Table 1 we also report statistics measuring the model’s Lipschitz constant.
For vector valued functions, f , in order to measure the Lipschitz constant the induced matrix norm 3 is used, coming from the norms in the source and target spaces.
Further statistics for all adversarial defences and models are deferred to Appendix D.
The columns (cid:107)∇ l (cid:107) 2 and (cid:107)∇ f (cid:107) 2 , ∞ give the maximum of these norms over the test/validation set.
Employing all defences signiﬁcantly decreases the norm of the model Jacobian on the test data, and hence improves model robustness.
On CIFAR-10 the model with all defences has Jacobian norm 10 times smaller than the undefended model, whereas adversarial training only improves the Jacobian norm by a factor of three at most.
Similarly on
CIFAR-100, adversarial training alone improves the norm of the Jacobian by a factor of no more than three.
However a combination of all defences decreases the norm of the model Jacobian by a factor of six.
These statistics indicate that 2-Lipschitz regularization of the loss, combined with (cid:96)adversarial training, dramatically reduce the Lipschitz constant of a network measured in the 2 , ∞ norm.
Interestingly, 2-Lipschitz regularization of the loss also decreases the 1 , 1 Jacobian norm of the model (see Tables 4 and 6).
Thus 2-Lipschitz regularization improves robustness to (cid:96) ∞ attacks as well.
In Appendix D we report results for all models and combinations of defence methods.
Of the individual defences by themselves, adversarial training ( J 1 or J 2 ) improves model robustness the most.
We ﬁnd (cid:96) 2 adversarial training ( J 2 ) to be more effective than FGSM ( J 1 ) when attacks are measured in (cid:96) 2 .
We observe the same ranking of defence methods for AllCNN and ResNet-34.
Adversarial training improves model robustness.
However model robustness is further improved by adding Lipschitz regularization of the loss, which empirically decreases the Jacobian norm of the model on the test data.
In terms of training time, both adversarial training and Lipschitz regularization increase training time by a factor of no more than four.
In contrast, adding a ﬁnal tanh layer to normalize the logits is nearly free, and consistently improves model robustness by itself.
5. C OMPARISON WITH OTHER WORK
On the CIFAR-10 dataset (as of revision) the current state-of-the-art (cid:96) 2 robustness is Qian & Wegman (2018).
Without adversarial training, their method achieves 89.9% error at ε = 1 . 5 . In contrast, without adversarial training, Lipschitz regularization as implemented here achieves 78.42% error at ε = 1 . 5 , an improvement of over 11%.
With adversarial training, our results are comparable to Qian & Wegman (2018): they achieved 79.6% error at adversarial perturbation ε = 1 . 5 , whereas our method achieves 78.59%, an improvement of 1%. A direct comparison is difﬁcult, since Qian & Wegman used the Carlini-Wagner attack, which (with Foolbox defaults) is a weaker (cid:96) 2 attack than the attack used here ( (cid:96) 2 PGD), so our results should be adjusted accordingly.
(We applied the
Carlini-Wagner attack
but we only recorded the results for the strongest attack). After revision we ran comparisons for attacks measured in (cid:96) ∞ .
We only used one step adversarial training. The current state-of-the-art on CIFAR-10 is Madry et al.
(2017) which used seven steps of (cid:96) ∞ PGD. The strongest attack achieves 54.2% error at ε = 8255 .
Our best model, trained with J 2 − Lip regularization, is slightly worse, with 62.4% error at ε = 8 255 . We expect that multi-step adversarial training would improve our results in this setting.
A M ODEL AND TRAINING DETAILS
We used standard data augmentation for the CIFAR dataset, comprising of horizontal ﬂips and random crops of padded images, four pixels per side.
Images are scaled to have pixel values in [0 , 1] .
We used square cutout (Devries & Taylor (2017)) of width 16 on CIFAR-10, and width 8 on CIFAR-100, but no dropout.
Batch normalization was used after every convolution layer.
We used SGD with an initial learning rate of 0.1, momentum set to 0.9, and a batch size of 128.
CIFARwas trained for 200 epochs, dropping the learning rate by a factor of ﬁve after epochs 60, 120, and 180.
On CIFAR-100, networks were trained for 300 epochs, and the learning rate was dropped by a factor of 10 after epochs 150 and 225.
For CIFAR-10 weight decay (Tikhonov/ (cid:96) 2 regularization) was set to 5e − 4 ; on CIFAR-100 it was 1e − 4 .
For networks with Lipschitz regularization in the 2-norm, the Lagrange multiplier λ of the excess Lipschitz term was set to λ = 0 .
With 1-norm Lipschitz regularization, the Lagrange multiplier was scaled down by a factor of √ n from the 2-Lipschitz Lagrange multiplier, where n is the input dimension.
Adversarially trained models were trained with images perturbed to an (cid:96) 2 distance of ε = 0 .
We did not tune either of these hyperparameters.
[Table] For CIFAR-10, the ResNeXt architecture we used had a depth of 34 layers, cardinality 2 and width 32, with a basic residual block rather than a bottleneck.
The branches (convolution groups) of the blocks were aggregated via a mean, rather than using a fully connected layer.
For CIFAR-100 the architecture was the same, but had cardinality 4.
A. P RE - softmax SIGMOID LAYER
Prior to the ﬁnal softmax layer, we found inserting a sigmoid activation function improved model robustness.
In this case, the sigmoid layer comprised of ﬁrst batch normalization (without learnable parameters), followed by the activation function t tanh( xt ) , where t is a single learnable parameter, common across all layer inputs.
Other possible defences discussed in Goodfellow et al.
(2018) include input validation and preprocessing, which would potentially allow adversarial samples to be recognized before being input to the model, and architecture modiﬁcations designed to improve robustness to adversarial samples.
For more information we refer to the review (Goodfellow et al.
and the discussion of attack methods in (Brendel et al.
B P ROOOF OF L EMMA 2.
Proof.
Use the Taylor expansion [Equation] Thus, the optimal attack vector deﬁned by (2) in a generic norm (cid:107) · (cid:107) can be approximated to order ε 2 by solving the problem [Equation]
According to (Boyd & Vandenberghe, 2004, A.1.6), the equation [Equation] where the norm on the right hand size is the dual norm. Thus
[Equation]
respectively
[Equation]
C L IPSCHITZ CONSTANT OF DATA AND OPTIMAL EXTENSIONS
Deﬁne the Lipschitz constant of the data (in the 2 , ∞ norm) to be
[Equation]
Table 2 lists the Lipschitz constant of the training data for common datasets, which are all small: all but one are below 1 in the 2 , ∞ norm.
The Lipschitz extension theorem (Valentine, 1945) says that given function values { f ( x ) } x ∈D , there exists an extension f ext which perfectly ﬁts the data, and has the same Lipschitz constant, provided the appropriate norm are used on the X and Y spaces.
This can be done using, for example, the 2norm for X and the ∞ norm on the label space.
In other norms, we can also make an extension, but the Lipschitz constant may increase (Johnson & Lindenstrauss, 1984).
Of course, such a function may not be consistent with a given architecture.
See Table 2, where we present the Lipschitz constant of common datasets.
[Table]
4e
−68.
J 2 − Lip 2 & tanh 5.
8e −73.
ResNet
J6.
3e −
J
[Table]
Lip
[Table]
2e −
J 0 − Lip5.
8e −
J
[Table]
& tanh 5.
7e −89.
J 0 − Lip 1 & tanh 7.
7 . 3e −88.
J 0 − Lip
[Table]
& tanh 6.
7e −88.
J5.
4e −97.
J 1 − Lip
[Table]
9e −95.
J 1 & tanh 5.
4e −92.
J 1 − Lip 1 & tanh 6.
8 . 4e −88. J5. 7 . 4e −99. J 2 − Lip5.
7 . 9e −87.
J
[Table]
& tanh 5. 1e −71. J 2 − Lip [Table]
& tanh 5. 3e −77.
ResNeXt (2x32d) J4.
7e − J 0 − Lip5. 5 . 9e −99.
J 0 − Lip4. 8e −99. J 0 & tanh 4.
4e −73.
J 0 − Lip 1 & tanh 4. 5e −78. J 0 − Lip 2 & tanh 4. 1e −72.
J3. 6e −92. J 1 − Lip5. 6 . 93e −91.
J 1 & tanh 4. 8e −75. J 1 − Lip 1 & tanh 4. 8e −81.
J3. 9 . 0e −98. J 2 − Lip4.
1e −98. J 2 & tanh 3. 2e −62. J 2 − Lip 2 & tanh 4.
4e −63. D F URTHER EXPERIMENTAL RESULTS Here we present complete results for all regularization types studied, on all models and datasets considered. Adversarial distances reported in (cid:96) 2 were generated using (cid:96)
[Table]
J 0 − Lip 1 & tanh 9. J 0 − Lip 2 & tanh 7. J32. J 1 − Lip12.
J 1 & tanh 22. J 1 − Lip 1 & tanh 8. J18. J 2 − Lip12.
J 2 & tanh 14.
J
[Table]
Lip 2 & tanh 7. ResNeXt (2x32d) J85. J 0 − Lip15.
J 0 − Lip21.
J 0 & tanh 43.
J 0 − Lip 1 & tanh 10.
J 0 − Lip 2 & tanh 8.
J35.
J 1 − Lip14.
J 1 & tanh 24.
J 1 − Lip 1 & tanh 9.
J32.
J 2 − Lip14.
[Table]
[Table]
[Table]
J
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
7e −53.
5e −61.
ResNet
J27.
6e
−90.
0 . 07e
−94. J 0 − Lip29. 8e −62. 7e −74.
J 0 − Lip28.
8e −70.
4e −81.
J 0 & tanh 40.
5e −81.
05e −82.
J 0 − Lip 1 & tanh 29.
6 . 9e −60.
7e −71.
J 0 − Lip 2 & tanh 38.
6e −68.
1 . 0e −74.
J28.
8e −66.
4e −78.
J 1 − Lip29.
7 . 9e −56.
0e −68.
J 1 & tanh 29.
5 . 9e −62.
1 . 6e −71.
J 1 − Lip 1 & tanh 30.
7 . 9e −55.
1e −64.
J28.
6e −66.
5e −75.
J 2 − Lip28.
6 . 9e −66.
8e −70.
J 2 & tanh 29.
6e −64.
7e −71.
J 2 − Lip 2 & tanh 28.
1e −58. 1e −66. ResNeXt (4x32d) J21.
7e −74.
4e −81.
J 0 − Lip23.
9 . 9e −51.
6e −62.
J 0 − Lip21.
8e −47.
2e −55.
J 0 & tanh 21.
4e −52.
1 . 3e −74.
J 0 − Lip 1 & tanh 23.
7e −46.
3 . 2e −56.
J 0 − Lip 2 & tanh 21.
4e −52.
7e −60.
J22.
8e −53.
5e −63.
J 1 − Lip23.
8e −47.
3 . 1e −56.
J 1 & tanh 22.
7e −53.
5e −62.
J 1 − Lip 1 & tanh 23.
7e −43.
7e −51.
J21.
6e −53.
6e −61.
J 2 − Lip21.
2e −46.
2e −55.
J 2 & tanh 21.
9 . 9e −50.
1e −55.
J 2 − Lip 2 & tanh 21.
6e −42.
0e −48.
[Table]
J 0 − Lip 1 & tanh 15. J 0 − Lip 2 & tanh 7. J25. J 1 − Lip15.
J 1 & tanh 24. J 1 − Lip 1 & tanh 13. J31. J 2 − Lip18.
J 2 & tanh 27. J 2 − Lip 2 & tanh 16. ResNeXt (4x32d) J93.
J 0 − Lip22.
J 0 − Lip25.
J 0 & tanh 50. J 0 − Lip 1 & tanh 19. J 0 − Lip 2 & tanh 9. J34.
J 1 − Lip21.
J 1 & tanh 41. J 1 − Lip 1 & tanh 15. J44. [Table]