B OOSTING THE C ERTIFIED R OBUSTNESS OF L- INFINITY D ISTANCE N ETS
A BSTRACT
Recently, Zhang et al.
(2021) developed a new neural network architecture based on (cid:96) ∞ -distance functions, which naturally possesses certiﬁed (cid:96) ∞ robustness by its construction.
Despite rigorous theoretical guarantees, the model so far can only achieve comparable performance to conventional networks.
In this paper, we make the following two contributions: (i) We demonstrate that (cid:96) ∞ -distance nets enjoy a fundamental advantage in certiﬁed robustness over conventional networks (under typical certiﬁcation approaches); (ii) With an improved training process we are able to signiﬁcantly boost the certiﬁed accuracy of (cid:96) ∞ -distance nets.
Our training approach largely alleviates the optimization problem that arose in the previous training scheme, in particular, the unexpected large Lipschitz constant due to the use of a crucial trick called (cid:96) p -relaxation .
The core of our training approach is a novel objective function that combines scaled cross-entropy loss and clipped hinge loss with a decaying mixing coefﬁcient.
Experiments show that using the proposed training strategy, the certiﬁed accuracy of (cid:96) ∞ -distance net can be dramatically improved from 33.30% to 40.06% on CIFAR-10 ( (cid:15)
= 8 / 255 ) , meanwhile outperforming other approaches in this area by a large margin.
Our results clearly demonstrate the effectiveness and potential of (cid:96) ∞ -distance net for certiﬁed robustness.
Codes are available at https://github.com/zbh2047/L inf-dist-net-v2.
I NTRODUCTION
Modern neural networks, while achieving high accuracy on various tasks, are found to be vulnerable to small, adversarially-chosen perturbations of the inputs (Szegedy et al., 2013; Biggio et al., 2013).
Given an image x correctly classiﬁed by a neural network, there often exists a small adversarial perturbation δ , such that the perturbed image x + δ looks indistinguishable to x , but fools the network to predict an incorrect class with high conﬁdence.
Such vulnerability creates security concerns in many real-world applications.
A large body of works has been developed to obtain robust classiﬁers.
One line of works proposed heuristic approaches that are empirically robust to particular attack methods, among which adversarial training is the most successful approach (Goodfellow et al., 2014; Madry et al., 2017; Zhang et al., 2019a).
However, a variety of these heuristics have been subsequently broken by stronger and adaptive attacks (Carlini & Wagner, 2017; Athalye et al., 2018; Uesato et al., 2018; Tramer et al., 2020; Croce & Hein, 2020), and there are no formal guarantees whether the resulting model is truly robust.
This motivates another line of works that seeks certiﬁably robust classiﬁers whose prediction is guaranteed to remain the same under all allowed perturbations.
Representatives of this ﬁeld use convex relaxation (Wong & Kolter, 2018; Mirman et al., 2018;
Gowal et al., 2018; Zhang et al., 2020b) or randomized smoothing (Cohen et al., 2019;
Salman et al., 2019a; Zhai et al., 2020; Yang et al., 2020a).
However, these approaches typically suffer from high computational cost, yet still cannot achieve satisfactory results for commonly used (cid:96) ∞ -norm perturbation scenario.
Recently, Zhang et al.
(2021) proposed a fundamentally different approach by designing a new network architecture called (cid:96) ∞ -distance net, a name coming from its construction that the basic neuron is deﬁned as the (cid:96) ∞ -distance function.
Using the fact that any (cid:96) ∞ -distance net is inherently a 1Lipschitz mapping, one can easily check whether the prediction is certiﬁably robust for a given data point according to the output margin.
The whole procedure only requires a forward pass without any additional computation.
The authors further showed that the model family has strong expressive power, e.g., a large enough (cid:96) ∞ -distance net can approximate any 1-Lipschitz function on a bounded domain.
Unfortunately, however, the empirical model performance did not well reﬂect the theoretical advantages.
As shown in Zhang et al.
(2021), it is necessary to use a conventional multi-layer perception (MLP) 1 on top of an (cid:96) ∞ -distance net backbone to achieve better performance compared to the baseline methods.
It makes both the training and the certiﬁcation procedure complicated.
More importantly, it calls into question whether the (cid:96) ∞ -distance net is really a better model conﬁguration than conventional architectures in the regime of certiﬁed robustness.
In this paper, we give an afﬁrmative answer by showing that (cid:96) ∞ -distance net itself sufﬁces for good performance and can be well learned using an improved training strategy.
We ﬁrst mathematically prove that under mild assumptions of the dataset, there exists an (cid:96) ∞ -distance net with reasonable size by construction that achieves perfect certiﬁed robustness.
This result indicates the strong expressive power of (cid:96) ∞ -distance nets in robustness certiﬁcation, and shows a fundamental advantage over conventional networks with typical certiﬁcation approaches (Mirman et al., 2021).
However, it seems to contradict the previous empirical observations, suggesting that the model may fail to ﬁnd an optimal solution and further motivating us to revisit the optimization process designed in Zhang et al.
Due to the non-smoothness of the (cid:96) ∞ -distance function, Zhang et al.
(2021) developed several training tricks to overcome the optimization difﬁculty.
A notable trick is called the (cid:96) p -relaxation, in which (cid:96) p -distance neurons are used during optimization to give a smooth approximation of (cid:96) ∞ -distance.
However, we ﬁnd that the relaxation on neurons unexpectedly relaxes the Lipschitz constant of the network to an exponentially large value, making the objective function no longer maximize the robust accuracy and leading to sub-optimal solutions.
We develop a novel modiﬁcation of the objective function to bypass the problem mentioned above.
The objective function is a linear combination of a scaled cross-entropy term and a modiﬁed clipped hinge term.
The cross-entropy loss maximizes the output margin despite the model’s
Lipschitzness and makes optimization sufﬁcient at the early training stage when p is small.
The clipped hinge loss then focuses on robustness for correctly classiﬁed samples at the late training phase when p approaches inﬁnity.
The switch from cross-entropy loss to clipped hinge loss is reﬂected in the mixing coefﬁcient, which decays to zero as p grows to inﬁnity throughout the training procedure.
Despite its simplicity, our experimental results show signiﬁcant performance gains on various datasets.
In particular, an (cid:96) ∞ -distance net backbone can achieve 40.06% certiﬁed robust accuracy on CIFAR-10 ( (cid:15) = 8 / 255 ).
This goes far beyond the previous results, which achieved 33.30% certiﬁed accuracy on CIFAR-10 using the same architecture (Zhang et al., 2021).
Besides, it surpasses the relaxation-based certiﬁcation approaches by at least 5 points (Shi et al., 2021;
Lyu et al., 2021), establishing a new state-of-the-art result.
To summarize, both the theoretical ﬁnding and empirical results in this paper demonstrate the merit of (cid:96) ∞ -distance net for certiﬁed robustness.
Considering the simplicity of the architecture and training strategy used in this paper, we believe there are still many potentials for future research of (cid:96) ∞ -distance nets, and more generally, the class of Lipschitz architectures.
P RELIMINARY
In this section, we brieﬂy introduce the (cid:96) ∞ -distance net and its training strategy.
An (cid:96) ∞ -distance net is constructed using (cid:96) ∞ -distance neurons as the basic component.
The (cid:96) ∞ -distance neuron u takes vector x as the input and calculates the (cid:96) ∞ -norm distance between x and parameter w with a bias term b .
The neuron can be written as
[Equation]
Based on the neuron deﬁnition, a fully connected (cid:96) ∞ -distance net can then be constructed.
Formally, an L layer network g takes x (0) =
x as the input, and the l th layer x ( l ) is calculated by
[Equation]
Here n l is the number of neurons in the l th layer.
For K -class classiﬁcation problems, n L = K .
The network outputs g ( x ) = x ( L ) as logits and predicts the class arg max i ∈
[ K ] [ g ( x )] i .
An important property of (cid:96) ∞ -distance net is its Lipschitz continuity, as is stated below.
Deﬁnition 2.1.
A mapping f ( z ) : R m → R n is called λ -Lipschitz with respect to (cid:96) p -norm (cid:107) · (cid:107) p , if for any z 1 , z 2 , the following holds:
[Equation]
Proposition 2.2.
The mapping of an (cid:96) ∞ -distance layer is 1-Lipschitz with respect to (cid:96) ∞ -norm.
Thus by composition, any (cid:96) ∞ -distance net g ( · ) is 1-Lipschitz with respect to (cid:96) ∞ -norm.
(cid:96) ∞ -distance nets naturally possess certiﬁed robustness using the Lipschitz property.
In detail, for any data point x with label y , denote the output margin of network g as
[Equation]
If x is correctly classiﬁed by g , then the prediction of a perturbed input x + δ will remain the same as x if (cid:107) δ (cid:107) ∞ < margin ( x , y ; g ) / 2 .
In other words, we can obtain the certiﬁed robustness for a given perturbation level (cid:15) according to I ( margin ( x , y ; g ) / 2 > (cid:15) ) , where I ( · ) is the indicator function.
We call this margin-based certiﬁcation .
Given this certiﬁcation approach, a corresponding training approach can then be developed, where one simply learns a large margin classiﬁer using standard loss functions, e.g., hinge loss, without using adversarial training.
Therefore the whole training procedure is as efﬁcient as training standard networks with no additional cost.
Zhang et al.
(2021) further show that (cid:96) ∞ -distance nets are Lipschitz-universal approximators.
In detail, a large enough (cid:96) ∞ -distance net can approximate any 1-Lipschitz function with respect to (cid:96) ∞ -norm on a bounded domain arbitrarily well.
Training (cid:96) ∞ -distance nets.
One major challenge in training (cid:96) ∞ -distance net is that the (cid:96) ∞ -distance operation is highly non-smooth, and the gradients (i.e. ∇ x (cid:107) x − w (cid:107) ∞ and ∇ w (cid:107) x − w (cid:107) ∞ ) are sparse.
To mitigate the problem, Zhang et al.
(2021) used (cid:96) p -distance neurons instead of (cid:96) ∞ -distance ones during training, resulting in approximate and non-sparse gradients.
Typically p is set to a small value (e.g., 8) in the beginning and increases throughout training until it reaches a large number (e.g., 1000).
The authors also designed several other tricks to further address the optimization difﬁculty.
However, even with the help of tricks, (cid:96) ∞ -distance nets only perform competitively to previous works.
The authors thus considered using a hybrid model architecture, in which the (cid:96) ∞ -distance net serves as a robust feature extractor, and an additional conventional multi-layer perceptron is used as the prediction head.
This architecture achieves the best performance, but both the training and the certiﬁcation approach become complicated again due to the presence of non-Lipschitz MLP layers.
E XPRESSIVE P OWER OF (cid:96) ∞ - DISTANCE N ETS IN R OBUST C LASSIFICATION
In this section, we challenge the conclusion of previous work by proving that simple (cid:96) ∞ -distance nets (without the top MLP) sufﬁce for achieving perfect certiﬁed robustness in classiﬁcation.
Recall that Zhang et al.
(2021) already provides a universal approximation theorem, showing the expressive power of (cid:96) ∞ -distance nets to represent Lipschitz functions.
However, their result focuses on realvalued function approximation and is not directly helpful for certiﬁed robustness in classiﬁcation.
One may ask: Does a certiﬁably robust (cid:96) ∞ -distance net exist given a dataset ? If so, how large does the network need to be?
We will answer these questions and show that one can explicitly construct an (cid:96) ∞ -distance net that achieves perfect certiﬁed robustness as long as the dataset satisﬁes the following (weak) condition called r -separation
(Yang et al., 2020b).
Deﬁnition 3.1.
( r -separation) Consider a labeled dataset D = { ( x i , y i ) } where y i ∈
[ K ] is the label of x i .
We say D is r -separated with respect to (cid:96) p -norm if for any pair of samples ( x i , y i ) , ( x j , y j ) , as long as y i (cid:54) =
y j , we have (cid:107) x
i − x j (cid:107) p > 2 r . [Table]
It is easy to see that r -separation is a necessary condition for robustness under (cid:96) p -norm perturbation (cid:15)
=
r .
In fact, the condition holds for all commonly used datasets (e.g., MNIST, CIFAR-10): the value of r in each dataset is much greater than the allowed perturbation level (cid:15) as is demonstrated in Yang et al.
(2020b) (see Table 1 above).
The authors took a further step and showed there always exists a classiﬁer that achieves perfect robust accuracy if the condition holds.
We now prove that even if we restrict the classiﬁer to be the network function class represented by (cid:96) ∞ -distance nets, the conclusion is still correct: a simple two-layer (cid:96) ∞ -distance net with hidden size O ( n ) can already achieve perfect robustness for r -separated datasets.
Theorem 3.2.
Let D be a dataset with n elements satisfying the r -separation condition with respect to (cid:96) ∞ -norm.
Then there exists a two-layer (cid:96) ∞ -distance net with hidden size n , such that when using margin-based certiﬁcation, the certiﬁed (cid:96) ∞ robust accuracy is 100% on D under perturbation (cid:15)
=
r .
Proof sketch.
Consider a two layer (cid:96) ∞ -distance net g deﬁned in Equation (2).
Let its parameters be assigned by
[Equation]
for [Equation] [Equation] where C = 4 max i ∈ [ n ] (cid:107) x i (cid:107) ∞ is a constant, and I ( · ) is the indicator function.
For the above assignment, it can be proved that the network outputs
[Equation]
From Equation (5) the network g can represent a nearest neighbor classiﬁer, in that it outputs the negative of the nearest neighbor distance between input x and the samples of each class.
Therefore, given data x = x i in dataset D , the output margin of g ( x ) is at least 2 r due to the r -separation condition.
In other words, g achieves 100% certiﬁed robust accuracy on D . Remark 3.3.
The above result can be extended to multi-layer networks.
In general, we can prove the existence of such networks with L layers and no more than O ( n/L + K + d ) hidden neurons for each hidden layer where d is the input dimension.
See Appendix A for details of the proof.
The signiﬁcance of Theorem 3.2 can be reﬂected in the following two aspects.
Firstly, our result explicitly shows the strong expressive power of (cid:96) ∞ -distance nets in robust classiﬁcation , which complements the universal approximation theorem in Zhang et al.
Moreover, Theorem 3. gives an upper bound of O ( n ) on the required network size which is close to practical applications.
It is much smaller than the size needed for function approximation ( O (1 /ε d ) under approximation error ε , proved in Zhang et al. (2021)), which scales exponentially in the input dimension d .
Secondly, our result justiﬁes that for well-designed architectures , using only the global Lipschitz property is sufﬁcient for robustness certiﬁcation.
It contrasts to the prior view that suggests leveraging the local Lipschitz constant is necessary (Huster et al., 2018), which typically needs sophisticated calculations (Wong et al., 2018; Zhang et al., 2018; 2020b).
More importantly, as a comparison, Mirman et al. (2021) very recently proved that for any conventional network, the commonlyused interval bound propagation (IBP) (Mirman et al., 2018; Gowal et al., 2018) intrinsically cannot achieve perfect certiﬁed robustness on a simple r -separation dataset containing only three data points (under (cid:15) = r ). In other words, (cid:96) ∞ -distance nets certiﬁed using the global Lipschitz property have a fundamental advantage over conventional networks certiﬁed using interval bound propagation.  I NVESTIGATING THE T RAINING OF (cid:96) ∞ - DISTANCE N ETS
Since robust (cid:96) ∞ -distance nets exist in principle, the remaining thing is understanding why the current training strategy cannot ﬁnd a robust solution.
In this section, we ﬁrst provide evidence that the training method in Zhang et al.
(2021) is indeed problematic and cannot achieve good certiﬁed robustness, then provide a novel way to address the issue.
4. P ROBLEMS OF THE C URRENT T RAINING S TRATEGY
As shown in Section 2, given any perturbation level (cid:15) , the certiﬁed accuracy of data point x can be calculated according to I ( margin ( x , y ; g ) / 2 ≤ (cid:15) ) for 1-Lipschitz functions.
Then the hinge loss becomes standard to learn a robust (cid:96) ∞ -distance net:
[Equation]
[Figure]
p e r c e n t a g e ( % ) p=8 p= p= p=      12 hinge threshold ( / )     c e r t i f i e d a cc u r a c y ( % ) train test
[Figure]
where θ is the hinge threshold which should be larger than 2 (cid:15) .
Hinge loss aims at making the output margin for any sample greater than θ , and L hinge ( g , D ; θ )
=
if and only if the network g achieves perfect certiﬁed robustness under perturbation (cid:15) = θ/ 2 .
Hinge loss fails to learn robust classiﬁers.
We ﬁrst start with some empirical observations.
Consider a plain (cid:96) ∞ -distance net trained on CIFAR-10 dataset using the same approach and hyperparameters provided in Zhang et al.
When the training ﬁnishes, we count the percentage of training samples whose margin is greater than θ , i.e., achieving zero loss.
We expect the value to be large if the optimization is successful.
However, the result surprisingly reveals that only 1.62% of the training samples are classiﬁed correctly with a margin greater than θ .
Even if we use much longer training epochs (e.g. 4000 epochs on CIFAR-10), the percentage is still below 2.5% (see Figure 1(a)).
Thus we conclude that hinge loss fails to optimize well for most of the training samples.
Since the output margins of the vast majority of training samples are less than θ , the loss approximately degenerates to a linear function without the maximum operation:
[Equation]
where θ becomes irrelevant, and training becomes to optimize the average margin over the dataset regardless of the allowed perturbation (cid:15) which is deﬁnitely problematic.
After checking the hyper-parameters used in Zhang et al.
(2021), we ﬁnd that the hinge threshold θ is set to 80/255, which is much larger than the perturbation level (cid:15) = 8 / 255 .
This partly explains the above degeneration phenomenon, but it is still unclear why such a large value has to be taken.
We then conduct experiments to see the performance with different chosen hinge thresholds θ .
The results are illustrated in Figure 1(b).
As we can see, a smaller hinge threshold not only reduces certiﬁed test accuracy but even makes training worse.
Why does this happen?
We ﬁnd that the reason for the loss degeneration stems from the (cid:96) prelaxation used in training.
While (cid:96) p -relaxation alleviates the sparse gradient problem, it destroys the Lipschitz property of the (cid:96) ∞ -distance neuron, as stated in Proposition 4.1:
Proposition 4.1.
A layer constructed using (cid:96) p -distance neurons
[Equation]
is d 1 /p Lipschitz with respect to (cid:96) ∞ -norm, where d is the dimension of x .
Thus by composition, an
L layer (cid:96) p -distance net is d L/p Lipschitz with respect to (cid:96) ∞ -norm.
Zhang et al.
(2021) uses a small p in the beginning and increases its value gradually during training.
From Proposition 4.1, the Lipschitz constant of the smoothed network can be signiﬁcantly large 2 at the early training stage.
Note that the robustness certiﬁcation I ( margin ( x , y ; g ) / 2 ≤ (cid:15) ) holds for 1-Lipschitz functions.
When the Lipschitz constant is large, even if the margin of a data point passes the threshold, the data point can still lie near to classiﬁcation boundary.
This makes the training using hinge loss ineffective at the early stage and converge to a wrong solution far from the real optima.
We argue that the early-stage training is important as when p rises to a large value, the optimization becomes intrinsically difﬁcult to push the parameters back to the correct solution due to sparse gradients.
Such argument can be veriﬁed from Figure 1(a), in which we plot the percentage of training samples with margins greater than θ throughout a long training process.
After p starts to rise, the margin decreases drastically, and the percentage sharply drops and never increases again during the whole (cid:96) p -relaxation procedure.
It is also clear why the value of θ must be chosen to be much larger than 2 (cid:15) .
For small θ , the margin optimization becomes insufﬁcient at early training stages when the Lipschitz constant is exponentially large, leading to worse performance even on the training dataset .
4. O UR S OLUTION
In the previous section, one can see that the hinge loss and the (cid:96) p -relaxation are incompatible.
As the (cid:96) p -relaxation is essential to overcome the sparse gradient problem, we focus on developing better loss functions.
We show in this section that a simple change of the objective function can address the above problem, leading to non-trivial improvements.
We approach the issue by investigating the commonly used cross-entropy loss.
For conventional non-Lipschitz networks, cross-entropy loss aims at increasing the logit of the true class while decreasing the other logits as much as possible, therefore enlarges the output margin without a threshold constraint.
This makes the optimization of output margin sufﬁcient, which largely alleviates the problem in Section 4.1 for training (cid:96) p -distance nets with small p .
Such ﬁndings thus motivate us to replace hinge loss with cross-entropy.
However, on the other hand, cross-entropy loss only coarsely enlarges the margin, rather than exactly optimizing the surrogate of certiﬁed accuracy
I ( margin ( x , y ; g ) / 2 ≤ (cid:15) ) that depends on (cid:15) .
When the model is almost 1-Lipschitz (i.e. p approaches inﬁnity), hinge loss can still be better than cross-entropy.
In other words, cross-entropy loss and hinge loss are complementary.
Based on the above argument, we propose to simply combine cross-entropy loss and hinge loss to obtain the following objective function: [Equation]
[Equation]
where (cid:96) CE
( z , y ) =
log( (cid:80) i exp( z i ))
− z y and (cid:96) hinge ( z , y )
=
max { max i (cid:54) =
y z i − z
y + 1 , 0 } .
We now make detailed explanations about each term in Equation (9).
Scaled cross-entropy loss (cid:96) CE ( s · g ( x i ) , y i ) .
Cross-entropy loss deals with the optimization issue when p is small.
Here a slight difference is the introduced scaling s as is explained below.
We know cross-entropy loss is invariant to the shift operation (adding a constant to each output logit) but not scaling (multiplying a constant).
For conventional networks, the output logits are produced through the last linear layer, and the scaling factor can be implicitly learned in the parameters of the linear layer to match the cross-entropy loss.
However, (cid:96) ∞ -distance net is strictly 1-Lipschitz and does not have any scaling operation.
We thus introduce a learnable scalar (temperature) s that multiplies the network output g ( x i ) before taking cross-entropy loss.
We simply initialize it to be 1.
Note that s does not inﬂuence the classiﬁcation results and can be removed once the training ﬁnishes.
Clipped hinge loss min { (cid:96) hinge ( g ( x i ) /θ, y i ) , 1 } .
Clipped hinge loss is designed to achieve robustness when p approaches inﬁnity.
Unlike the standard hinge loss, the clipped version plateaus if the output margin is negative (i.e., misclassiﬁed).
In other words, clipped hinge loss is equivalent to applying hinge loss only on correctly-classiﬁed samples.
The reason for applying such a clipping is three-fold.
(i)
Scaled cross-entropy loss already focuses on learning a model with high (clean) accuracy, thus there is no need to optimize on wrongly-classiﬁed samples duplicatively using hinge loss.
Moreover, cross-entropy is better than hinge loss when used in classiﬁcation, as the gradient of hinge loss makes optimization harder 3 .
(ii) In the late training phase, the optimization becomes intrinsically difﬁcult ( p approaching inﬁnity). As a consequence, wrongly classiﬁed samples may have little chance to be robust.
Clipped hinge loss thus ignores these samples and concentrates on easier ones to increase their potential to be robust after training.
(iii)
The clipped hinge loss is a better surrogate for 0-1 robust error I ( margin ( x , y ; g ) / 2 ≤ (cid:15) ) .
Compared with hinge loss, the clipped version is closer to our goal and more likely to achieve better certiﬁed accuracy.
Finally, due to the presence of cross-entropy loss, we will show in Section 5.3 that the hinge threshold θ can be set to a much smaller value unlike Figure 1(b), which thus avoids the loss degeneration problem.
The mixing coefﬁcient λ .
The coefﬁcient λ in loss (9) plays a role in the trade-off between crossentropy and hinge loss.
Based on the above motivation, we use a decaying λ that attenuates from λ 0 to zero throughout the process of (cid:96) p -relaxation ( λ 0 is a hyper-parameter).
When p is small at the early training stage, we focus more on cross-entropy loss.
After p grows large, λ vanishes, and a surrogate of 0-1 robust error is optimized.
We point out that objective functions similar to (9) also appeared in previous literature.
In particular, the TRADES loss (Zhang et al., 2019a) and MMA loss (Ding et al., 2020) are both composed of a mixture of the cross-entropy loss and a form of robust loss.
Nevertheless, the motivations of these methods are quite different.
For example, TRADES was proposed based on the theoretical results suggesting robustness may be at odds with accuracy (Tsipras et al., 2019), while our training approach is mainly motivated by the optimization issue.
Furthermore, the implementations of these methods also vary a lot.
We use clipped hinge loss to achieve robustness due to its simplicity, and uses a decaying λ correlate to p in (cid:96) p -relaxation due to the optimization problem in Section 4.1.
E XPERIMENTS 5. E XPERIMENTAL S ETTING
In this section, we evaluate the proposed training strategy on benchmark datasets MNIST and
CIFAR-10 to show the effectiveness of (cid:96) ∞ -distance net.
Model details.
We use exactly the same model as Zhang et al.
(2021) for a fair comparison.
Concretely, we consider the simple fully-connect (cid:96) ∞ -distance nets deﬁned in Equation (2).
All hidden layers have 5120 neurons.
We use a 5-layer network for MNIST and a 6-layer one for CIFAR-10.
Training details.
In all experiments, we choose the Adam optimizer with a batch size of 512.
The learning rate is set to 0.03 initially and decayed using a simple cosine annealing throughout the whole training process.
We use padding and random crop data augmentation for MNIST and
CIFAR-10, and also use random horizontal ﬂip for CIFAR-10.
The (cid:96) p -relaxation starts at p = 8 and ends at p = 1000 with p increasing exponentially.
The mixing coefﬁcient λ decays exponentially during the (cid:96) p -relaxation process from λ 0 to a vanishing value λ end .
We do not use further tricks that are used in Zhang et al.
(2021), e.g. the (cid:96) p weight decay or a warmup over perturbation (cid:15) , to keep our training strategy clean and simple.
The dataset dependent hyper-parameters, including θ , λ 0 , λ end and the number of epochs T , can be found in Appendix B. All experiments are run for 8 times on a single NVIDIA Tesla-V100 GPU, and the median of the performance number is reported.
Evaluation.
We test the robustness of the trained models under (cid:15) -bounded (cid:96) ∞ -norm perturbations.
Following the common practice (Madry et al., 2017), we mainly use (cid:15) = 0 .
3 for MNIST dataset and 8 / 255 for CIFAR-10 dataset.
We also provide results under other perturbation magnitudes, e.g.
(cid:15)
= 0 .
1 for MNIST and (cid:15) = 2 / 255 , (cid:15) = 16 / 255 for CIFAR-10.
We ﬁrst evaluate the robust test accuracy under the Projected Gradient Descent (PGD) attack (Madry et al., 2017).
The number of iterations of the PGD attack is set to a large number of 100.
We then calculate the certiﬁed robust accuracy based on the output margin.
5. E XPERIMENTAL R ESULTS
Results are presented in Table 2.
For each method in the table, we report the clean test accuracy without perturbation (denoted as Clean), the robust test accuracy under PGD attack (denoted as
PGD), and the certiﬁed robust test accuracy (denoted as Certiﬁed).
We also compare with randomized smoothing (see Appendix D), despite these methods provides probabilistic certiﬁed guarantee and usually take thousands of times more time than other approaches for robustness certiﬁcation. [Table] Shi et al., 2021) 66.
- 52.85 COLT (Balunovic & Vechev, 2020) 78.
- 60.5 (cid:107)
[Table]
Smoothing (Blum et al., 2020) 78. - 62.6 §(cid:107) (cid:96) [Table] † (Zhang et al., 2021) 60.
51.45 ‡ 50.94 (cid:96)
[Table]
Net
This paper 60.
IBP ∗ (Gowal et al., 2018) 50.
[Table]
IBP (Zhang et al., 2020b) 45.
33.06 CROWN
[Table]
IBP (Xu et al., 2020) 46.
33.38 IBP (Shi et al., 2021) 48.
[Table]
(Lyu et al., 2021) 48.
34.92 COLT (Balunovic & Vechev, 2020) 51.
- 27.5 (cid:107)
[Table]
(Salman et al., 2019a) 53.
- 24.0 §(cid:107)
[Table]
(Jeong & Shin, 2020) 52.
- 25.2 §(cid:107) (cid:96)
[Table]
+MLP (Zhang et al., 2021) 50.
37.06 ‡ 35.42 (cid:96)
[Table]
Net (Zhang et al., 2021) 56.
37.46 ‡ 33.30 (cid:96)
[Table]
[Table]
Net
This paper 54.
IBP ∗ (Gowal et al., 2018) 31.
[Table]
IBP (Zhang et al., 2020b) 33.
23.20 IBP (Shi et al., 2021) 36.
[Table]
Comparing with Zhang et al.
It can be seen that for all perturbation levels (cid:15) and datasets, our proposed training strategy improves the performance of (cid:96) ∞ -distance nets.
In particular, we boost the certiﬁed accuracy on CIFAR-10 from 33.30% to 40.06% under (cid:15) = 8 / 255 , and from 19.28% to 29.04% under a larger (cid:15) = 16 / 255 .
Note that we use exactly the same architecture as
Zhang et al.
(2021), and a larger network with better architecture may further improve the results.
Another observation from Table 2 is that the improvement of our proposed training strategy gets more prominent with the increase of (cid:15) .
This is consistent with our ﬁnding in Section 4.1, in that the optimization is particularly insufﬁcient for large (cid:15) using hinge loss, and in this case our proposed objective function can signiﬁcantly alleviate such problem.
Comparing with other certiﬁcation methods.
For most settings in Table 2, our results establish new state-of-the-arts over previous baselines, despite we use the margin-based certiﬁcation which is much simpler .
The gap is most noticeable for (cid:15)
= 8 / 255 on CIFAR-10, where we surpass recent relaxation-based approaches by more than 5 points (Shi et al., 2021;
Lyu et al., 2021).
It can also be observed that (cid:96) ∞ -distance net is most suitable for the case when (cid:96) ∞ perturbation is relatively large.
This is not surprising since Lipschitz property is well exhibited in this case.
If (cid:15) is vanishing small, the advantage of the Lipschitz property will not be well-exploited and (cid:96) ∞ -distance net will face more optimization and generalization problems compared with conventional networks. [Figure] [Figure] [Figure]
[Figure]
[Figure]
5. I NVESTIGATING THE P ROPOSED T RAINING S TRATEGY
We ﬁnally demonstrate by experiments that the proposed training strategy indeed addresses the optimization problem in Section 4.1.
We ﬁrst trace the output margin of the training samples throughout training on CIFAR-10 dataset ( (cid:15) = 8 / 255 ), and plot the percentage of samples with a margin greater than θ , less than 0, or between 0 and θ , shown in Figure 2(a).
In contrast to Figure 1(a), it can be seen that the loss does not degenerate in the whole training process.
We then demonstrate in Figure 2(b) that the accuracy curve regarding different choices of hinge threshold θ is well-behaved compared with Figure 1(b).
In particular, the best θ that maximizes the certiﬁed accuracy on training dataset approaches 2 (cid:15) (while for original hinge loss the value is 6 (cid:15) ).
The peak certiﬁed accuracy on training dataset also improves by 10 points (see blue lines in the two ﬁgures).
Such evidence clearly demonstrates the effectiveness of the proposed training strategy.
Sensitivity analysis .
We perform sensitivity analysis on CIFAR-10 dataset ( (cid:15) = 8 / 255 ) over hyperparameters including the hinge threshold θ and the mixing coefﬁcient λ . Results are shown in Figure 2(b) and 2(c).
It can be seen that (i) the certiﬁed accuracy is above 39% for a wide range of θ (between 5 (cid:15) and 8 (cid:15) ); (ii) among the 20 hyper-parameter combinations of ( λ 0 , λ end ) , all certiﬁed accuracy results surpass 39%, and half of the results can achieve a certiﬁed accuracy of more than 39.75%. In summary, the performance is not sensitive to the choice of the hyper-parameters θ and λ . See Appendix F for more details on other hyper-parameters. Ablation studies .
We also conduct ablation experiments for the proposed loss function on CIFAR10 dataset ( (cid:15) = 8 / 255 ). Due to the space limit, we put results in Appendix E. In summary, both the cross-entropy loss and clipped hinge loss are crucial to boost the certiﬁed accuracy. Using a decaying mixing coefﬁcient λ can further improve the performance and stabilize the training.
R ELATED W ORK
In recent years substantial efforts have been taken to obtain robust classiﬁers.
Existing approaches mainly fall into two categories: adversarial training and certiﬁed defenses.
Adversarial training.
Adversarial training methods ﬁrst leverage attack algorithms to generate adversarial examples of the inputs on the ﬂy, then update the model’s parameters using these perturbed inputs together with the original labels (Goodfellow et al., 2014; Kurakin et al., 2016; Madry et al., 2017).
In particular, Madry et al.
(2017) suggested using Projected Gradient Descent (PGD) as the universal attacker to ﬁnd a perturbation that maximizes the standard cross-entropy loss, which achieves decent empirical robustness.
Some recent works considered other training objectives that combine cross-entropy loss and a carefully designed robust surrogate loss (Zhang et al., 2019a; Ding et al., 2020; Wang et al., 2020), which show similarities to this paper.
However, all methods above are evaluated empirically using ﬁrst-order attacks such as PGD, and there is no formal guarantee whether the learned model is truly robust.
This motivates researchers to study a new type of method called certiﬁed defenses, in which the prediction is guaranteed to remain the same under all allowed perturbations, thus provably resists against all potential attacks.
Relaxation-based certiﬁed defenses.
These methods adopt convex relaxation to calculate a convex region containing all possible network outputs for a given input under perturbation (Wong & Kolter, 2018;
Wong et al., 2018; Dvijotham et al., 2018; 2020; Raghunathan et al., 2018a;b; Weng et al., 2018; Singh et al., 2018; Mirman et al., 2018; Gehr et al., 2018; Wang et al., 2018; 2021; Gowal et al., 2018; Zhang et al., 2018; 2020b; Xiao et al., 2019; Croce et al., 2019; Balunovic & Vechev, 2020; Lee et al., 2020; Dathathri et al., 2020; Xu et al., 2020; Lyu et al., 2021; Shi et al., 2021).
If all points in this region correspond to the correct prediction, then the network is provably robust.
However, the relaxation procedure is usually complicated and computationally expensive. Furthermore,
Salman et al. (2019b) ; Mirman et al. (2021) indicated that there might be an inherent barrier to tight relaxation for a large class of convex relaxation approaches.
This is also reﬂected in experiments, where the trained model often suffers from severe accuracy drop even on training data.
Randomized smoothing for certiﬁed robustness .
Randomized smoothing provides another way to calculate a probabilistic certiﬁcation under (cid:96) 2 perturbations (Lecuyer et al., 2019; Li et al., 2019a;
Cohen et al., 2019;
Salman et al., 2019a; Zhai et al., 2020; Jeong & Shin, 2020; Zhang et al., 2020a).
For any classiﬁer, if a Gaussian random noise is added to the input, the resulting “smoothed” classiﬁer then possesses a certiﬁed guarantee under (cid:96) 2 perturbations.
Randomized smoothing has been scaled up to ImageNet and achieves state-of-the-art certiﬁed accuracy for (cid:96) 2 perturbations.
However, recent studies imply that it cannot achieve nontrivial certiﬁed accuracy for (cid:96) p perturbations under (cid:15) =
Ω( d 1 /p − 1 / 2 ) when p > 2 which depends on the input dimension d (Yang et al., 2020a;
Blum et al., 2020; Kumar et al., 2020; Wu et al., 2021).
Therefore it is not suitable for (cid:96) ∞ perturbation scenario if (cid:15) is not very small.
Lipschitz networks.
An even simpler way for certiﬁed robustness is to use Lipschitz networks, which directly possess margin-based certiﬁcation.
Earlier works in this area mainly regard the Lipschitz property as a kind of regularization and penalize (or constrain) the Lipschitz constant of a conventional ReLU network based on the spectral norms of its weight matrices (Cisse et al., 2017;
Yoshida & Miyato, 2017; Gouk et al., 2018; Tsuzuku et al., 2018; Farnia et al., 2019; Qian & Wegman, 2019; Pauli et al., 2021).
However, these methods either can not provide certiﬁed guarantees or provide a vanishing small certiﬁed radius.
Anil et al.
(2019) ﬁgured out that current Lipschitz networks intrinsically lack expressivity to some simple Lipschitz functions, and designed the ﬁrst
Lipschitz-universal approximator called GroupSort network. Li et al.
(2019b)
; Singla & Feizi (2021) studied Lipschitz networks for convolutional architectures.
Leino et al.
(2021) achieved the stateof-the-art certiﬁed robustness using GroupSort network under (cid:96) 2 perturbations.
However, none of these approaches above can provide good certiﬁed results for (cid:96) ∞ robustness.
The most relevant work to this paper is Zhang et al.
(2021), in which the author designed a novel Lipschitz network with respect to (cid:96) ∞ -norm.
We show such architecture can establish new state-of-the-art results in the (cid:96) ∞ perturbation scenario.
C ONCLUSION
In this paper, we demonstrate that a simple (cid:96) ∞ -distance net sufﬁces for good certiﬁed robustness from both theoretical and experimental perspectives. Theoretically, we prove the strong expressive power of (cid:96) ∞ -distance nets in robust classiﬁcation.
Combining with Mirman et al.
(2021), this result may indicate that (cid:96) ∞ -distance nets have inherent advantages over conventional networks for certiﬁed robustness. Experimentally, despite simplicity, our approach yields a large gain over previous (possibly more complicated) certiﬁcation approaches and the trained models establish new state-of-the-art certiﬁed robustness. Despite these promising results, there are still many aspects that remain unexplored. Firstly, in the case of very small (cid:15) setting (cid:96) ∞ -distance nets may have a lot of room for improvement.
Secondly, it is important to design better architectures suitable for image classiﬁcation tasks than the simple fully connected network used in this paper.
Finally, it might be interesting to design better optimization algorithms for (cid:96) ∞ -distance nets to further handle the model’s non-smoothness and gradient sparsity.
We hope this work can make promising the study of (cid:96) ∞ -distance nets, and more generally, the global Lipschitz architectures for certiﬁed robustness.
A P ROOF OF T HEOREM 3.2 AND B EYOND
Theorem A.1.
Let D be a dataset with n elements satisfying the r -separation condition with respect to (cid:96) ∞ -norm.
Then there exists a two-layer (cid:96) ∞ -distance net with hidden size n , such that the certiﬁed (cid:96) ∞
robust accuracy is 100% on D under perturbation (cid:15)
=
r .
Proof.
Consider a two layer (cid:96) ∞ -distance net g deﬁned in Equation (2).
Let its parameters be assigned by
[Equation]
where C = 4 max i ∈ [ n ] (cid:107) x i (cid:107) ∞ is a constant, and I ( · ) is the indicator function.
The chosen of C is large enough so that the following holds:
[Equation]
For the above assignment, the ﬁrst layer simply calculates the (cid:96) ∞ -distance between x and each sample in dataset D .
We now derive the output of the second layer.
We have
[Equation]
Here a core step is Equation (13) which follows by using Inequality (11) when the image x is in dataset D .
From Equation (14) the network g can represent a nearest neighbor classiﬁer, in that it outputs the negative of the nearest neighbor distance between input x and the samples of each class.
Therefore, given data x = x i in dataset D , the output [ g ( x )]
j is either 0 or less than − 2 r depending on whether j = y i , due to the r -separation condition.
Therefore the output margin is at least 2 r .
In other words, g achieves 100% certiﬁed robust accuracy on D .
We now give a general result which shows that any L layer ( L ≥ 2 ) (cid:96) ∞ -distance net with hidden size
O ( n/L + K + d ) can achieve perfect certiﬁed robustness.
In this general setting, the total number of neurons in the network is thus O ( n + KL + dL ) which is still close to real practice.
Theorem A.2.
Let D be a dataset with n elements satisfying the r -separation condition with respect to (cid:96) ∞ -norm.
Then there exists an L -layer (cid:96) ∞ -distance net with hidden size no more than (cid:100) nL − 1 (cid:101) +
K + 2 d where d is the input dimension, such that the certiﬁed (cid:96) ∞
robust accuracy is 100% on D under perturbation (cid:15)
=
r .
Proof.
The basic idea is to rearrange the computation process of the two-layer network in the above proof by order so as to satisfy the width constraint.
To formulate the proof below, we ﬁrst deﬁne some notations.
Deﬁne K preﬁx arrays h j ( j ∈ [ K ]) as follows:
[Equation]
Note that we want the network output to be the negative of the nearest neighbor distance of all samples in a class j , i.e. − min i ∈ [ n ] ,y i =
j (cid:107) x − x i (cid:107) ∞ , which corresponds to h j,n .
For any hidden layer x ( l ) ( l ∈ [ L − 1] ), we separate it into four sets: I ( l ) = { x ( l ) i : i ∈ [ d ] } , (cid:101)
I
( l ) =
{ x ( l ) i : d < i ≤
2 d } , O ( l ) =
{ x ( l ) i : 2 d < i ≤ 2 d + K } and S ( l ) containing the rest (cid:100) nL − 1 (cid:101) neurons.
We also denote
O ( L ) =
{ x ( L ) i : i ∈
[ K ] } for ease of presentation.
We ﬁrst make a construction in which the neurons of I ( l ) in each layer exactly represent the input x , e.g. x ( l ) i
= x
i ( 1 ≤ i ≤ d ).
This is feasible since an (cid:96) ∞ -distance neuron can represent the operation that fetches an element of the neuron input on a bounded domain, e.g. the following operation
[Equation]
if we assign w j = − C, b = − C, w k =
( k (cid:54) =
j ) where C is larger than twice the diameter of domain K .
In this way, all hidden neurons in the network can have access to the network input x .
We similarly let the neurons of (cid:101) I ( l ) represent the input x again, e.g. x ( l ) i + d = x i ( d < i ≤ 2 d ).
Next, we aim at designing the following computation pattern for O ( l ) :
[Equation]
In this way O ( L ) =
{ h j,n : j ∈ [ K ] } , and the network exactly represents a nearest neighbor classiﬁer which is desired.
To represent O ( l +1) in (17), we use the following recursive relation
[Equation]
Note that h j, (cid:100) n ( l − 1) L − 1 (cid:101) =
x ( l ) 2 d
+ j is already calculated in O ( l ) in the previous layer.
The left thing is to calculate (cid:107) x − x i (cid:107) ∞ for all i ∈ (cid:110) (cid:100) n ( l − 1)
L − 1 (cid:101) + 1 , · · · , (cid:100) nlL − 1 (cid:101) (cid:111) , which can be done by the neurons of the set S ( l ) in the previous layer (which will be proven later).
Assume S ( l ) represents
[Equation]
19) then the neuron x ( l +1) 2 d + j merges the information of neuron x ( l ) 2 d + j and part of neurons x ( l ) [-
-]2 d + K
+ i in S ( l ) depending on whether y i =
j , using the construction similar to (10).
In detail,
[Equation]
holds by assigning
[Equation]
where C is a sufﬁciently large constant.
Now it remains to represent S ( l ) in (19).
We ﬁrst consider the simplest case when l = 1 .
In this case we can directly calculate x (1)2 d + K + i = (cid:107) x − x i (cid:107) ∞ by assigning proper weights and zero bias.
Now assume l ≥ 2 .
In this case, we cannot calculate the (cid:96) ∞ -distance directly since the previous layer has irrelavant neurons, e.g. the neurons in sets O ( l − 1) and S ( l − 1) .
We want to only use the sets of neurons I ( l − 1) and (cid:101) I ( l − 1) in the previous layer.
Suppose the objective is to represent (cid:107) x − x i (cid:107) ∞ for some i .
Note that
[Equation]
We assign the parameters of the (cid:96) ∞ -distance neuron x ( l ) j = (cid:107) x
( l − 1) − w ( l,j ) (cid:107) ∞ + b ( l ) j for some j as follows:
[Equation]
=
[Table]
nL
[Table]
b
[Table]
l ) j = − C where C is a sufﬁciently large constant.
In this way x (
[Equation]
which is desired.
Proof completes.
B A DDITIONAL E XPERIMENTAL D ETAILS AND H YPER - PARAMETERS
Our experiments are implemented using the Pytorch framework.
We run all experiments in this paper using a single NVIDIA Tesla-V100 GPU.
The CUDA version is 11.2.
The learnable scalar in Equation (9) is initialized to be one and trained using a smaller learning rate that is one-ﬁfth of the base learning rate.
This is mainly to make training stable as suggested in Zhang et al.
(2019b) since the scalar scales the whole network output.
The ﬁnal performance is not sensitive to the scalar learning rate as long as it is set to a small value.
For random crop data augmentation, we use padding = 1 for MNIST and padding = 3 for CIFAR-10.
The model is initialized using identity-map initialization (see Section 5.3 in Zhang et al.
(2021)), and mean-shift batch normalization is used for all intermediate layers.
The training procedure is as follows:
• In the ﬁrst e 1 epochs, we set p = 8 in (cid:96) p -relaxation and use λ = λ 0 as the mixing coefﬁcient; •
In the next e 2 epochs, p exponentially increases from 8 to 1000.
Accordingly, λ exponentially decreases from λ 0 to a vanishing small value λ end ; •
In the ﬁnal e 3 epochs, p is set to inﬁnity and λ is set to 0.
All hyper-parameters are provided in Table 3.
Most hyper-parameters are directly borrow from
Zhang et al.
(2021), e
[Table]
g. hyper-parameters of the optimizer, the batch size
[Table]
and the value p in (cid:96) prelaxation.
The only searched hyper-parameters are the hinge threshold θ and the mixing coefﬁcient λ 0 , λ end .
These hyper-parameters are obtained using a course grid search
[Table] [Table]
We also run additional experiments using the training strategy in Zhang et al.
(2021) for performance comparison when the original paper does not present the corresponding results.
This mainly includes the case (cid:15) = 0 .
1 on MNIST and (cid:15)
= 2 / 255 , (cid:15)
= 16 / 255 on CIFAR-10, as shown in Table 2.
We use the same hyper-parameters in Zhang et al.
(2021), except for the hinge threshold θ where we perform a careful grid search.
The choice of θ is listed in Table 4.
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
C T HE L IPSCHITZ C ONSTANT OF (cid:96) p - DISTANCE N ET
We have shown in Section 4.1 that an L layer (cid:96) p -distance net with d neurons in each hidden layer is d L/p Lipschitz with respect to (cid:96) ∞ -norm.
The value becomes quite large if p is small.
For example,
Zhang et al.
(2021) uses a 6-layer (cid:96) p -dist net with d = 5120 .
This gives a Lipschitz constant of approximate 568 when p = 8 at the beginning of training.
One may ask whether such upper bound of Lipschitz constant (Proposition 4.1) is tight and reﬂects the true Lipschitz property in practice.
To validate the tightness of the bound, we run the following experiments.
Consider the (cid:96) ∞ -distance net used in Zhang et al.
We train this architecture following the training strategy either in Zhang et al.
(2021) or in this paper.
After training ﬁnishes, we then set p = 8 without changing the model parameters.
We approximate the Lipschitz constant of the model using Projected Gradient Descent (PGD), which provides a lower bound estimate.
In detail, for each image x in the test dataset, we estimate the quantity
[Equation]
21) where g is the network and (cid:15) is a small constant taken to be 1/255.
The expression (21) is clearly a lower bound of the Lipschitz constant (can be seen as the “local Lipschitz constant” near point x ).
It can be further lower bounded by using the PGD solution δ = δ PGD .
We run PGD for each target label j and optimize
[Equation]
using 20 PGD steps with step size (cid:15)/ 4 .
Results are shown in Figure 3.
It can be observed that the “local Lipschitz constant” around real data points is indeed far larger than one.
The average value exceeds 100 which is close to the theoretical upper bound.
D R ANDOMIZED S MOOTHING FOR (cid:96) ∞ P ERTURBATIONS
Randomized smoothing approaches typically provide probabilistic certiﬁed guarantees for (cid:96) 2 perturbations.
To apply these methods in the (cid:96) ∞ perturbation scenario, most of works convert the result of (cid:96) 2 perturbation into (cid:96) ∞ perturbation using norm inequalities (Salman et al., 2019a; Blum et al., 2020).
Speciﬁcally, to certify the robustness under (cid:15) -bounded (cid:96) ∞ perturbations, one can certify the robustness under ( (cid:15) √ d ) -bounded (cid:96) 2 perturbations to obtain a lower bound estimate where d is the input dimension.
On CIFAR-10 dataset the input d
This radius
For the case (cid:15) = 2 / 255 , Blum et al.
(2020) directly reported a certiﬁed accuracy of 62.6% using randomized smoothing which is currently state-of-the-art.
For the case (cid:15) = 8 / 255 , there are no literature results that directly report (cid:96) ∞ robustness, so we use the results of (cid:96) 2 robustness from representative papers (Salman et al., 2019a; Jeong & Shin, 2020).
Salman et al. (2019a) reported a certiﬁed accuracy of 24% and a clean accuracy of 53% under (cid:96) 2 perturbation (cid:15)
= 1 .
75 (Table 17 in their paper).
Jeong & Shin (2020) reported a certiﬁed accuracy of 25.2% and a clean accuracy of 52.3% under (cid:96) 2 perturbation (cid:15)
= 1 .
75 (Table 1 in their paper, σ = 0 .
For the case (cid:15) = 16 / 255 , all randomized smoothing methods fail and only achieve a trivial certiﬁed accuracy of 10%.
E A BLATION S TUDIES
In this section we conduct ablation experiments to the proposed loss.
Let a training sample be ( x , y ) where y is the label of x , and denote g ( x ) as the output of an (cid:96) ∞ -distance net for input x .
Let (cid:96) hinge ( z , y )
=
max { max i (cid:54) =
y z i − z
y + 1 , 0 } and (cid:96) CE
( z , y ) =
log( (cid:80) i exp( z i ))
− z
y represent the hinge loss and cross-entropy loss, respectively.
We would like to justify that (i) Cross-entropy loss can alleviate the optimization issue in (cid:96) p -relaxation (which is a better substitute over hinge loss), but the threshold of hinge loss is also crucial as it explicitly optimizes the certiﬁed accuracy; (ii)
Combining cross-entropy loss and clipped hinge loss leads to a much better performance; (ii) Using a decaying mixing coefﬁcient λ can further boost the performance and stabilize the training.
We consider the following objective functions:
(1) The baseline hinge loss: (cid:96) hinge ( g ( x ) /θ, y ) with hinge threshold θ .
This loss is used in Zhang et al.
The cross-entropy loss: (cid:96) CE ( s · g ( x ) , y ) where s is a scalar (temperature).
Note that the information of the allowed perturbation radius (cid:15) is not encoded in the loss, and the loss only coarsely enlarges the output margin (see Section 4.2).
Therefore it may not achieve desired certiﬁed robustness.
A variant of cross-entropy loss with threshold: (cid:96) CE ( s · g ( x − θ 1 y ) , y ) where s is a scalar (temperature), θ is the threshold hyper-parameter and 1 y is the one-hot vector with the y th element being one.
Intuitively speaking, we subtract the y th output logit by θ before taking cross-entropy loss.
Compared to the above loss (2), now the information (cid:15) is encoded in the threshold hyper-parameter θ .
We point out that this loss can be seen as a smooth approximation of the hinge loss.
The combination of cross-entropy loss and clipped hinge loss:
λ(cid:96) CE ( s · g ( x ) , y )
+ min( (cid:96) hinge ( g ( x ) /θ, y ) , 1) with a ﬁxed mixing coefﬁcient λ .
The combination of cross-entropy loss and clipped hinge loss:
λ(cid:96) CE ( s · g ( x ) , y )
+ min( (cid:96) hinge ( g ( x ) /θ, y ) , 1) with a decaying λ .
The loss is used in this paper.
We keep the training procedure the same for the different objective functions above.
The hyperparameters such as θ and λ are independently tuned for each objective function to achieve the best certiﬁed accuracy.
The scalar s is a learnable parameter in each loss except for objective function (2) where we tune the value of s .
For other hyper-parameters, we use the values in Table 3.
We independently run 5 experiments for each setting and the median of the performance is reported.
Results are listed in Table 5, and the bracket in Table 5(b) shows the standard deviation over 5 runs.
[Table]
We can draw the following conclusions from Table 5:
• Hinge loss and cross-entropy loss are complementary.
Cross-entropy is better in the early training phase when the Lipschitz constant is large, while hinge loss is better for certiﬁed robustness when the model is almost 1-Lipschitz in the later training phase.
This can be seen from the results of objective functions (1-3) in Table 5(a), where (3) incorporates cross-entropy loss and the threshold in hinge loss, and outperforms both (1) and (2) by a comparable margin.
• Combining cross-entropy loss and clipped hinge loss leads to much better performance.
This can be seen from the result of the objective function (4), which signiﬁcantly outperforms (1-3).
However, this loss is very sensitive to the hyper-parameter λ as is demonstrated in Table 5(b).
If λ is too large, the certiﬁed accuracy gets worse.
If λ is too small, the training becomes unstable and the clean accuracy drops signiﬁcantly.
In the extreme case when λ = 0 , the loss (4) reduces to the clipped hinge loss and the optimization fails because clipped hinge loss does not optimize for wrongly-classiﬁed samples.
• Using a decaying mixing coefﬁcient λ can further boost the performance and stabilize the training.
In contrast to the loss (4), we will show in Appendix F that the proposed objective function (5) in this paper is not sensitive to hyper-parameter λ .
F S ENSITIVITY A NALYSIS
In this section we provide sensitive analysis of the proposed objective function (9) with respect to hyper-parameters.
We consider the setting (cid:15) = 8 / 255 on CIFAR-10 dataset.
For each choice of hyper-parameters, we independently run 3 experiments and the median of the certiﬁed accuracy is reported.
The hinge threshold θ . The results are already plotted in Figure 2(b).
We list the concrete numbers below.
[Table]
The mixing coefﬁcients λ 0 and λ end . The results are already shown in Figure 2(c).
The number of epochs . Our best result reported in Table 2 is trained for 1300 epochs, which is longer than Zhang et al. We also consider using the same training budget by setting e 1 = 100 , e 2 = 650 , e 3 = 50 in Table 3.
This yields a total of 800 training epochs. In this way we can achieve 54.52 clean accuracy and 39.61 certiﬁed accuracy. Adam hyper-parameters . While we use the same Adam hyper-parameters as Zhang et al.
(2021), note that the values are different from the default numbers in Pytorch (e,g. β 2 = 0 .
999 and (cid:15) =
10 − 8 ).
Instead, we use β 2 = 0 .
99 and (cid:15) =
10 − 10 in all experiments.
For (cid:15) , we ﬁnd the value is essential to be small, because for the (cid:96) p -distance function, the gradients of most of the elements are close to zero when p is large.
For Adam optimizer (Kingma & Ba, 2015), the update is written as
[Equation]
Then a large (cid:15) will dominate the denominator in Adam if v t is close to zero, which severely weakens the parameters’ update and leads to worse performance. We ﬁnd the value of (cid:15) is not sensitive if it is small enough, i.e. we can obtain almost identical performance for (cid:15) = 10 − 10 , 10 − 11 or 10 − 12 . The use of a smaller β 2 is mainly because of the (cid:96) p relaxation.
Since p increases exponentially during training, the network function changes through time, therefore the long-time-ago historical gradient information will become meaningless and even do harm to the training.
This is why a smaller β 2 is used, so that the second-order momentum (the term v t ) in Adam only depends on recent gradient information.
We ﬁnd it is also OK to choice β 2 = 0.98 or 0.995, and the certiﬁed accuracy can reach 39.5%.
However, the default value of 0.999 is too large, as it corresponds totimes more historical information than the value of 0.99.