I MPROVED DETERMINISTIC l 2 ROBUSTNESS ON CIFAR-10 AND CIFAR-
A BSTRACT
Training convolutional neural networks (CNNs) with a strict Lipschitz constraint under the l 2 norm is useful for provable adversarial robustness, interpretable gradients and stable training.
While 1 -Lipschitz CNNs can be designed by enforcing a 1 -Lipschitz constraint on each layer, training such networks requires each layer to have an orthogonal Jacobian matrix (for all inputs) to prevent the gradients from vanishing during backpropagation.
A layer with this property is said to be Gradient Norm Preserving (GNP).
In this work, we introduce a procedure to certify the robustness of 1 -Lipschitz CNNs by relaxing the orthogonalization of the last linear layer of the network that signiﬁcantly advances the state of the art for both standard and provable robust accuracies on CIFAR-100 (gains of 4 .
80% and 4 .
71% , respectively).
We further boost their robustness by introducing (i) a novel Gradient Norm preserving activation function called the Householder activation function (that includes every GroupSort activation) and (ii) a certiﬁcate regularization.
On CIFAR-10, we achieve signiﬁcant improvements over prior works in provable robust accuracy ( 5 . 81% ) with only a minor drop in standard accuracy ( − 0 .
Code for reproducing all experiments in the paper is available at https://github.com/singlasahil14/SOC .
I NTRODUCTION
Given a neural network f : R d → R k , the Lipschitz constant 1 Lip( f ) enforces an upper bound on how much the output is allowed to change in proportion to a change in the input.
Previous work has demonstrated that a small Lipschitz constant is useful for improved adversarial robustness (Szegedy et al., 2014; Cissé et al., 2017), generalization bounds (Bartlett et al., 2017; Long & Sedghi, 2020), interpretable gradients (Tsipras et al., 2018) and Wasserstein distance estimation (Villani, 2008).
Lip( f ) also upper bounds the increase in the norm of gradient during backpropagation and can thus prevent gradient explosion during training, enabling us to train very deep networks (Xiao et al., 2018).
While heuristic methods to enforce Lipschitz constraints (Miyato et al., 2018; Gulrajani et al., 2017) have achieved much practical success, they do not provably enforce a bound on Lip( f ) globally and it remains challenging to achieve similar results when Lip( f ) is provably bounded.
Using the property: Lip( g ◦ h ) ≤
Lip( g ) Lip( h ) , the Lipschitz constant of the neural network can be bounded by the product of the Lipschitz constant of all layers.
While this allows us to construct 1 -Lipschitz neural networks by constraining each layer to be 1 -Lipschitz, Anil et al.
identiﬁed a key difﬁculty with this approach.
Because a 1 -Lipschitz layer can only reduce the norm of gradient during backpropagation, backprop through each layer reduces the gradient norm, resulting in small gradient values for layers closer to the input, making training slow and difﬁcult.
To address this problem, they introduce Gradient Norm Preserving (GNP) architectures where each layer preserves the gradient norm during backpropagation.
This involves constraining the Jacobian of each linear layer to be an orthogonal matrix and using a GNP activation function called GroupSort .
GroupSort activation function (Anil et al., 2018)
ﬁrst separates the vector of preactivations z ∈ R m into groups of pre-speciﬁed sizes, sorts each group in the descending order and then concatenates these sorted groups.
When the group size is 2 , the resulting activation function is called MaxMin .
For 1 -Lipschitz CNNs, the robustness certiﬁcate for a sample x from class l is computed as
M f ( x ) / √ 2 where M f ( x ) =
f l ( x ) − max i (cid:54) =
l f i ( x ) .
Naturally, larger values of f l ( x ) and smaller values of max j (cid:54) =
l f j ( x ) will lead to larger certiﬁcates.
This requires the last weight matrix in the network, denoted by W ∈ R k × m
( k is the number of classes, m is the dimension of the penultimate layer, m > k ), to enforce the following constraints throughout training:
[Equation]
where W i, : denotes the i th row of W .
Now suppose that for some input image with label l , we want to update W to increase the logit for the l th class.
Since (cid:107) W l, : (cid:107) 2 is constrained to be 1 , the logit can only be increased by changing the direction of the vector W l, : .
Because the other rows { W i, : , i (cid:54) =
l } are constrained to be orthogonal to W l, : , this further requires an update for all the rows of W .
Thus during training, any update made to learn some class must necessitate the forgetting of information relevant for the other classes.
This can be particularly problematic when the number of classes k and thus the number of orthogonality constraints per row (i.e., k − 1 ) is large (such as in CIFAR-100).
To address this limitation, we propose to keep the last weight layer of the network unchanged.
But then the resulting function is no longer 1 -Lipschitz and the certiﬁcate M f ( x ) / √ 2 is not valid.
Thus, we introduce a new certiﬁcation procedure that does not require the last weight layer of the network
W to be orthogonal.
Our certiﬁcate is then given by the following equation:
[Equation]
However, a limitation of using the above certiﬁcate is that because the weight layers are completely unconstrained, larger norms of rows (i.e., (cid:107) W i, : (cid:107) ) can result in larger values of (cid:107) W l, : − W i, : (cid:107) 2 and thus smaller certiﬁcate values.
To address this limitation, we normalize all rows to be of unit norm before computing the logits.
While this still requires all the rows to be of unit norm, their directions are now allowed to change freely thus preventing the need to update other rows and forgetting of learned information.
We show that this provides signiﬁcant improvements when the number of classes is large.
We call this procedure Last Layer Normalization (abbreviated as LLN ).
On CIFAR-100, this signiﬁcantly improves both the standard ( > 3% ) and provable robust accuracy ( > 4% at ρ = 36 / 255 ) across multiple 1 -Lipschitz CNN architectures (Table 1).
Here, ρ is the l 2 attack radius.
Another limitation of existing 1 -Lipschitz CNNs (Li et al., 2019b; Trockman & Kolter, 2021; Singla & Feizi, 2021) is that their robustness guarantees do not scale properly with the l 2 radius ρ .
For example, the provable robust accuracy of (Singla & Feizi, 2021) drops ∼ 30% at ρ = 108 /compared to 36 / 255 on CIFAR-10 (Table 2).
To address this limitation, we introduce a certiﬁcate regularization denoted by CR (Section 5) that when used along with the Householder activation results in signiﬁcantly improved provable robust accuracy at larger radius values with minimal loss in standard accuracy.
On the CIFAR-10 dataset, we achieve signiﬁcant improvements in the provable robust accuracy for large ρ = 108 / 255 (min gain of +4 .
96% ) across different architectures with minimal loss in the standard accuracy (max drop of − 0 .
Results are in Table 2.
Additionally, we characterize the MaxMin activation function as a special case of the more general
Householder ( HH ) activations.
Recall that given z ∈ R m , the HH transformation is a linear function reﬂecting z about the hyperplane v T x =
( (cid:107) v (cid:107) 2 = 1 ), given by ( I − 2 vv T )
z where I − 2 vv T is orthogonal because (cid:107) v (cid:107) 2 = 1 .
The Householder activation function σ v is deﬁned below:
[Equation]
First, note that since z =
( I − 2 vv T )
z along v T z = 0 , σ v is continuous.
Moreover, the Jacobian ∇ z σ v is either I or I − 2 vv T (both orthogonal) implying σ v is GNP.
Since these properties hold ∀ v : (cid:107) v (cid:107) 2 = 1 , v can be learned during the training.
In fact, we prove that any GNP piecewise linear function that changes from Q 1 z to Q 2 z ( Q 1 , Q 2 are square orthogonal matrices) along v T z =must satisfy Q 2 = Q 1 (cid:0) I − 2 vv T (cid:1) to be continuous (Theorem 1).
Thus, this characterization proves that every GroupSort activation is a special case of the more general Householder activation function (example in Figure 1, discussion in Section 6).
In summary, in this paper, we make the following contributions:
• We introduce a certiﬁcation procedure without orthogonalizing the last linear layer called Last Layer Normalization .
This procedure signiﬁcantly enhances the standard and provable robust accuracy when the number of classes is large.
Using the LipConvnet-15 network on CIFAR-100, our modiﬁcation achieves a gain of +4 .
71% in provable robust accuracy (at ρ = 36 / 255 ) with a gain of +4 .
80% in standard accuracy (Table 1).
• We introduce a Certiﬁcate Regularizer that signiﬁcantly advances the provable robust accuracy with a small reduction in standard accuracy.
Using LipConvnet-15 network on CIFAR-10, we achieve +5 .
81% improvement in provable robust accuracy (at ρ = 108 / 255 ) with only a − 0 .
29% drop in standard accuracy over the existing methods (Table 2).
• We introduce a class of piecewise linear GNP activation functions called Householder or HH activations.
We show that the MaxMin activation is a special case of the HH activation for certain settings.
We prove that Householder transformations are necessary for any GNP piecewise linear function to be continuous (Theorem 1).
R ELATED WORK
Provably Lipschitz convolutional neural networks : The class of fully connected neural networks (FCNs) which are Gradient Norm Preserving (GNP) and 1-Lipschitz were ﬁrst introduced by Anil et al.
They orthogonalize weight matrices and use GroupSort as the activation function to design each layer to be GNP.
While there have been numerous works on enforcing Lipschitz constraints on convolution layers (Cissé et al., 2017; Tsuzuku et al., 2018; Qian & Wegman, 2019;
Gouk et al., 2020; Sedghi et al., 2019), they either enforce loose Lipschitz bounds or are not scalable to large networks.
To ensure that the Lipschitz constraint on convolutional layers is tight, multiple recent works try to construct convolution layers with an orthogonal Jacobian matrix (Li et al., 2019b;
Trockman & Kolter, 2021; Singla & Feizi, 2021).
These approaches avoid the aforementioned issues and allow the training of large, provably 1 -Lipschitz CNNs while achieving impressive results.
Provable defenses against adversarial examples : A provably robust classiﬁer is one for which we can guarantee that the classiﬁer’s prediction remains constant within some region around the input.
Most of the existing methods for provable robustness either bound the Lipschitz constant of the neural network or the individual layers (Weng et al., 2018; Zhang et al., 2019; 2018; Wong et al., 2018;
Wong & Kolter, 2018; Raghunathan et al., 2018; Croce et al., 2019; Singh et al., 2018; Singla & Feizi, 2020; Zhang et al., 2021).
However, these methods do not scale to large and practical networks on the
ImageNet dataset (Deng et al., 2009).
To scale to such large networks, randomized smoothing (Liu et al., 2018; Cao & Gong, 2017; Lécuyer et al., 2018; Li et al., 2019a;
Cohen et al., 2019;
Salman et al., 2019; Levine et al., 2019; Kumar et al., 2020a;b) has been proposed as a probabilistically certiﬁed defense .
However, certifying robustness with high probability requires generating a large number of noisy samples leading to high inference-time computation.
In contrast, the defense we propose is deterministic and hence not comparable to randomized smoothing.
While Levine & Feizi (2021) provide deterministic robustness certiﬁcates using randomized smoothing, their certiﬁcates are in the l 1 norm and not directly applicable for the l 2 threat model studied in this work.
We discuss the differences between l 1 and l 2 certiﬁcates in Appendix Section C.
P ROBLEM SETUP AND N OTATION
For a vector v , v j denotes its j th element.
For a matrix A , A j, : and A : ,k denote the j th row and k th column respectively.
Both A j, : and A : ,k are assumed to be column vectors (thus A j, : is the transpose of j th row of A ).
A j,k denotes the element in j th row and k th column of A .
A : j, : k denotes the matrix containing the ﬁrst j rows and k columns of A .
The same rules are directly extended to higher order tensors.
I denotes the identity matrix, R to denote the ﬁeld of real numbers.
For θ ∈ R , J + ( θ ) and J − ( θ ) denote the orthogonal matrices with determinants +1 and − 1 deﬁned as follows:
[Equation]
We construct a 1 -Lipschitz neural network, f : R d → R k ( d is the input dimension, k is the number of classes) by composing 1 -Lipschitz convolution layers and GNP activation functions.
To certify robustness for some input x with prediction l , we ﬁrst deﬁne the margin of prediction:
M f ( x ) = max (0 , f l ( x ) − max
i (cid:54) =
l f i ( x )) where f i ( x ) is the logit for class i and l is the correct label.
Using Theorem 7 in Li et al. (2019b), we can derive the robustness certiﬁcate (in the l 2 norm) as M f ( x ) / √ 2 .
Thus, the l 2 distance of x to the decision boundary is lower bounded by M f ( x ) / √ 2 :
[Equation]
We often use the abbreviation f i − f j : R D
→ R to denote the function so that:
[Equation]
Our goal is to train the neural network f to achieve the maximum possible provably robust accuracy while also simultaneously improving (or maintaining) standard accuracy.
L AST L AYER N ORMALIZATION
To ensure that the network is 1 -Lipschitz so that the certiﬁcate in equation (3) is valid, existing 1 -Lipschitz neural networks require the weight matrices of all the linear layers of the network to be orthogonal.
For the weight matrix in the last layer of the network (that maps the penultimate layer neurons to the logits), W ∈ R k × m
( k is the number of classes, m is the dimension of the penultimate layer, m > k ), this enforces the following constraints on each row W i, :
∈ R m :
[Equation]
Now suppose that for some input x with label l , we want to update W to increase the logit for the l th class.
Since (cid:107) W l, : (cid:107) 2 is constrained to be 1 , the gradient update can only change the direction of the vector W l, : .
But now, because the other rows { W i, : , i (cid:54) = l } are constrained to be orthogonal to
W l, : , this further requires an update for all the rows of W .
This has the negative effect that during training, any update made to learn some class must necessitate the forgetting of information relevant for the other classes.
This can be particularly problematic when the number of classes k is large (such as in CIFAR-100) and thus the number of orthogonality constraints per row i.e. k − 1 is large.
To address this limitation, ﬁrst observe that the neural network from the input layer to the penultimate layer (i.e excluding the last linear layer) is 1 -Lipschitz.
Let g : R d → R m be this function so that f ( x ) =
W g ( x ) + b .
This equation suggests that even if W is not orthogonal, the Lipschitz constant of the function f l − f i , can be computed by multiplying the Lipschitz constant of g (which is 1 ) and that of ( W l, : − W i, : ) (which is (cid:107) W l, : − W i, : (cid:107) 2 ).
The robustness certiﬁcate can then be computed as M f ( x ) / (cid:107) W l, : − W i, : (cid:107) 2 .
This procedure leads to the following proposition:
Proposition 1.
Given 1 -Lipschitz continuous function g : R d → R m and W ∈ R k × m , b ∈ R k ( k is the number of classes), construct a new function f : R d → R k deﬁned as: f ( x ) =
W g ( x ) + b .
Let f l ( x ) > max i (cid:54)
=
l f i ( x ) .
The robustness certiﬁcate (under the l 2 norm) is given by:
[Equation]
Proof is in Appendix Section A.1.
However, in our experiments, we found that using this procedure directly i.e without any constraint on the weight matrix W often results in large norms of row vectors (cid:107) W i, : (cid:107) 2 , and thus large values of (cid:107) W l, : − W i, : (cid:107) 2 and smaller certiﬁcates (Theorem 1).
To address this problem, we normalize all rows of the matrix to be of unit norm before computing the logits so that for the input x , the logit g i ( x ) can be computed as follows:
[Equation]
The robustness certiﬁcate can then be computed as follows:
[Equation]
where ∀ j, W ( n ) j, : =
W j
[Equation]
[Equation]
W j, : (cid:107)
[Equation]
g j
[Equation]
x
[Equation]
W
[Equation]
f
[Equation] x [Equation] While each row W i, : is still constrained to be of unit norm, unlike with orthogonality constraints (equation (4)), their directions are allowed to change freely.
This provides signiﬁcant improvements when the number of classes is large.
We call this procedure Last Layer Normalization ( LLN ). [Figure] θ
− θ/ orthogonal implying σ θ
C ERTIFICATE R EGULARIZATION
A limitation of using cross entropy loss for training 1 -Lipschitz CNNs is that it is not explicitly designed to maximize the margin M f ( x ) and thus, the robustness certiﬁcate.
That is, once the cross entropy loss becomes small, the gradients will no longer try to further increase the margin M f ( x ) even though the network may have the capacity to learn bigger margins.
To address this limitation, we can simply subtract the certiﬁcate i.e −M f ( x ) / √ 2 from the usual cross entropy loss function during training.
Observe that we subtract the certiﬁcate because we want to maximize the certiﬁcate values while minimizing the cross entropy loss.
However, in our experiments we found that this regularization term excessively penalizes the network for the misclassiﬁed examples and as a result, the certiﬁcate values for the correctly classiﬁed inputs are not large.
Thus, we propose to use the following regularized loss function during training:
[Equation]
[Equation]
In the above equation, f Ω denotes the 1 -Lipschitz neural network parametrized by Ω , f Ω ( x ) denotes the logits for the input x , (cid:96) ( f Ω ( x ) , l ) is the cross entropy loss for input x with label l and γ > 0 is the regularization coefﬁcient for maximizing the certiﬁcate.
We have the minus sign in front of the regularization term γ relu ( M f ( x ) / √ 2) because we want to maximize the certiﬁcate while minimizing the cross entropy loss.
For wrongly classiﬁed inputs, M f ( x ) / √ 2 < 0 = ⇒ relu ( M f ( x ) / √ 2)
= 0 .
This ensures that the optimization tries to increase the certiﬁcates only for the correctly classiﬁed inputs.
We call the above mentioned procedure Certiﬁcate Regularization (abbreviated as CR ).
H OUSEHOLDER A CTIVATION F UNCTIONS
Recall that given z ∈ R m , the Householder ( HH ) transformation reﬂects z about the hyperplane v T x = 0 where (cid:107) v (cid:107) 2 = 1 .
The linear transformation is given by the equation ( I − 2 vv T )
z where
I − 2 vv T is orthogonal because (cid:107) v (cid:107) 2 = 1 .
Now, consider the nonlinear function σ v deﬁned below:
Deﬁnition 1.
( Householder Activation of Order 1 )
The activation function σ v : R m → R m , applied on z ∈ R m , is called the m -dimensional Householder Activation of Order 1 :
[Equation]
Since σ v is linear when v T z > 0 or v T z < 0 , it is also continuous in both cases.
At the hyperplane separating the two cases (i.e., v T z = 0 ) we have: ( I − 2 vv T ) z
= z − 2( v T z ) v =
z (both linear functions are equal).
Thus, σ v is continuous ∀
z ∈ R m .
Moreover, the Jacobian is either I or
I − 2 vv T which are both square orthogonal matrices.
Thus, σ v is also GNP and 1 -Lipschitz.
Since these properties hold for all v satisfying (cid:107) v (cid:107) 2 = 1 , v can be made a learnable parameter.
While the above arguments suggest that HH transformations are sufﬁcient to ensure such functions are continuous, we also prove that they are necessary .
That is, we prove that if a GNP piecewise linear function g : R m → R m transitions between different linear functions Q 1 z and Q 2 z (in an open set S ⊂ R m ) along a hyperplane v T z =
(where (cid:107) v (cid:107) 2 = 1 ), then g is continuous in S if and only if Q
= Q 1 ( I − 2 vv T ) .
This theoretical result provides a general principle for designing piecewise linear GNP activation functions.
The formal result is stated in the following Theorem:
Theorem 1.
Given an open set S ⊂ R m , orthogonal square matrices Q 1 (cid:54) =
Q 2 , and vector v ∈ R
m ( (cid:107) v (cid:107) 2 = 1) such that S ∩ { z : v T z = 0 } (cid:54) = ∅ , the function g deﬁned as follows:
[Equation]
is continuous in S if and only if Q 2 = Q 1 ( I − 2 vv T ) .
Proof of Theorem 1 is in Appendix A.2.
Note that since the matrix I − 2 vv T has determinant − 1 , the above theorem necessitates that det( Q 1 ) =
− det( Q 2 )
i.e the determinant of the Jacobian must change sign whenever the Jacobian of a piecewise linear GNP activation function changes.
Recall that for the MaxMin activation function, MaxMin( z 1 , z 2 ) =
( z 1 , z 2 ) if z 1 > z 2 and ( z 2 , z 1 ) otherwise.
Thus, the Jacobian of MaxMin for z 1 > z 2 case is I =
J + (0) while for z 1 ≤ z 2 is
J − ( π/ 2) .
Using Theorem 1, we can easily prove that MaxMin is a special case of the more general
Householder activation functions where the Jacobian J − ( π/ 2) is replaced with J − ( θ ) and the conditions z 1 > z 2 are replaced with z 1 sin( θ/ 2) >
z 2 cos( θ/ 2) (similarly for ≤ ).
The construction of Householder activations in 2 dimensions, denoted by σ θ , is given in the following corollary:
Corollary 1.
The function σ θ : R 2 → R 2 deﬁned as
[Equation]
is continuous and is called 2D Householder Activation of Order 1 .
The two cases are demonstrated in Figure 1a and Figure 1b, respectively.
Since σ θ is continuous, GNP and 1 -Lipschitz ∀ θ ∈ R , θ is a learnable parameter.
For θ = π/ 2 in equation (8), σ θ is equivalent to
MaxMin .
Thus, σ θ is at least as expressive as MaxMin .
A major limitation of using σ v (equation (6)) directly is that it has only 2 linear regions and is thus limited in its expressive power.
In contrast, MaxMin ﬁrst divides the preactivation z ∈ R m (assuming m is divisible by 2 ) into m/ 2 groups of size 2 each.
Since each group has 2 linear regions, we get 2 m/ 2 linear regions from the m/ 2 groups.
Thus, to increase the expressive power, we similarly divide z into m/ 2 groups of size 2 each and apply the 2 -dimensional Householder activation function of
Order 1 ( σ θ ) to each group resulting in 2 m/ 2 linear regions (same as MaxMin ).
To apply σ θ to the output of a convolution layer z ∈ R m × n × n ( m is the number of channels and n × n is the spatial size), we ﬁrst split z into 2 tensors along the channel dimension giving the tensors:
z : m/ 2 , : , : and z m/ 2: , : , : .
Each of these tensors is of size m/ 2 × n × n giving n 2 m/ 2 groups.
We use the same θ for each pair of channels (irrespective of spatial location) resulting in m/ 2 learnable parameters.
We initialize each θ = π/ 2 so that σ θ is equivalent to MaxMin at initialization.
E XPERIMENTS
Our goal is to evaluate the effectiveness of the three changes proposed in this work: (a) Last Layer
Normalization, (b) Certiﬁcate regularization and (c) Householder activation functions.
We perform experiments under the setting of provably robust image classiﬁcation on CIFAR-10 and CIFARdatasets using the same 1 -Lipschitz CNN architectures used by Singla & Feizi (2021) (LipConvnet-5, 10, 15, . . .
, 40) due to their superior performance over the prior works.
We compare with the three orthogonal convolution layers in the literature: SOC (Singla & Feizi, 2021), BCOP (Li et al., 2019b) and Cayley (Trockman & Kolter, 2021) using MaxMin as the activation function.
We use SOC with MaxMin as the primary baseline for comparison in the maintext due to their superior performance over the prior works (BCOP, Cayley).
Results using BCOP and Cayley convolutions are given in Appendix Sections H and I for completeness.
We use the same implementations for these convolution layers as given in their respective github repositories.
We compare the provable robust accuracy using 3 different values of the l 2 perturbation radius ρ = 36 / 255 , 72 / 255 , 108 / 255 .
In both Tables 1 and 2, for all networks, we use SOC as the convolution layer.
Using SOC, we achieve the same bound on the approximation error of an orthogonal Jacobian as achieved in Singla & Feizi (2021) i.e. 2 .
42 × 10 − 6 across all convolution layers in all networks.
Thus, even for the network with 40 layers, this results in maximum Lipschitz constant of (1 + 2 .
42 × 10 − 6 ) 40 = 1 . 000097 ≤ 1 . 0001 .
Thus, the Lipschitz constant across all our networks is bounded by 1 . 0001 .
The symbol HH (in
Tables 1, 2) is for the 2D Householder Activation of order 1 or σ θ (deﬁned in equation (8)).
All experiments were performed using 1 NVIDIA GeForce RTX 2080 Ti GPU.
All networks were trained for 200 epochs with initial learning rate of 0.1, dropped by a factor of 0.1 after 100 andepochs.
For Certiﬁcate Regularization (or CR), we set the parameter γ = 0 .
7. R ESULTS ON CIFAR-
In Table 1, for each architecture, the row "SOC + MaxMin " uses the MaxMin activation, "+ LLN" adds Last Layer Normalization (uses MaxMin ), "+ HH" replaces MaxMin with σ θ (also uses LLN) , "+ CR" also adds Certiﬁcate Regularization with γ = 0 .
1 (uses both σ θ and LLN).
The column, "Increase (Standard)" denotes the increase in standard accuracy relative to "SOC + MaxMin ".
By adding LLN (the row "+ LLN"), we observe gains in standard (min gain of 1 . 10% ) and provable robust accuracy (min gain of 1 . 71% at ρ = 36 / 255 ) across all the LipConvnet architectures (gains relative to "SOC + MaxMin ").
These gains are smallest for the LipConvnet- 40 network with the maximum depth.
However, replacing MaxMin with the σ θ activation further improves the standard (min gain of 3 .
65% ) and provable robust accuracy (min gain of 4 .
46% at ρ = 36 / 255 ) across all networks (again relative to "SOC + MaxMin ").
We observe that replacing MaxMin with σ θ signiﬁcantly improves the performance of the deeper LipConvnet- 35 , 40 networks.
Adding CR further improves the provable robust accuracy while only slightly reducing the standard accuracy.
Because LLN signiﬁcantly improves the standard accuracy, we compare the standard accuracy numbers between rows "+ CR" and "+ LLN" to evaluate the drop due to CR.
We observe a small drop in standard accuracy ( − 0 .
04% , − 0 .
11% ) only for LipConvnet-5 and LipConvnetnetworks.
For the other networks, the standard accuracy actually increases .
7. R ESULTS ON CIFAR-
In Table 2, for each architecture, the row "SOC + MaxMin " uses the MaxMin activation, the row "+
HH" uses σ θ activation (replacing MaxMin ) and the row "+ CR" also adds Certiﬁcate Regularization with γ = 0 .
1 (again using σ θ as the activation).
Due to the small number of classes in CIFAR-10, we do not observe signiﬁcant gains using Last Layer Normalization or LLN (Appendix Table 10).
Thus, we do not include LLN for any of the results in Table 2.
The column, "Increase (108 / 255) " denotes the increase in provable robust accuracy with ρ = 108 / 255 relative to "SOC + MaxMin ".
For LipConvnet- 25 , 30 , 35 , 40 architectures, we observe gains in both the standard and provable robust accuracy by replacing MaxMin with the HH activation (i.e σ θ ).
Similar to what we observe for CIFAR-100 in Table 1, the gains in provable robust accuracy ( ρ = 108 / 255 ) are signiﬁcantly higher for deeper networks: LipConvnet-35 ( 3 . 65% ) and LipConvnet-40 ( 4 . 35% ) with decent gains in standard accuracy ( 1 . 71 and 1 .
61% respectively).
Again similar to CIFAR-100, adding CR further boosts the provable robust accuracy while slightly reducing the standard accuracy.
Comparing "+ CR" with "SOC + MaxMin ", we observe small drops in standard accuracy for LipConvnet- 5 , 10 , . .
. , 30 networks (max.
drop of − 0 .
56% ), and gains for LipConvnet-35 ( +0 . 52% ) and LipConvnet-40 ( +0 . 96% ).
For provable robust accuracy ( ρ = 108 / 255 ), we observe very signiﬁcant gains of > 4 .
96% for all networks and > 8% for the deeper LipConvnet-35, 40 networks.
[Table] % + CR 47.84% 34.77% 23.70% 15.84% +4.78% LipConvnetSOC + MaxMin 43.37% 28.59% 18.18% 10.85% _ + LLN 46.32% 32.87% 21.53% 13.86% +2.95% + HH 47.70% 34.00% 22.67% 14.57% +4.33% + CR 46.87% 34.09% 23.41% 15.61% +3.50%
LipConvnetSOC + MaxMin 42.87% 28.74% 18.47% 11.21% _ + LLN 46.18% 32.82% 21.52% 13.52% +3.31% + HH 46.80% 33.72% 22.70% 14.31%
+3.93% + CR 46.92% 34.17% 23.21% 15.84%
+4.05%
LipConvnetSOC + MaxMin 42.42% 28.34% 18.10% 10.96% _ + LLN 45.22% 32.10% 21.28% 13.25%
+2.80%
+ HH 46.21% 32.80% 21.55% 14.13%
+3.79% + CR 46.88% 33.64% 23.34% 15.73%
+4.46%
LipConvnetSOC + MaxMin 41.84% 28.00% 17.40% 10.28% _ + LLN 42.94% 29.71% 19.30% 11.99%
+1.10%
+ HH 45.84% 32.79% 21.98% 14.07% +4.00% + CR 45.03% 32.57% 22.
[Table]
Results with BCOP and Cayley convolutions are in Appendix Tables 13 and 14.
C ONCLUSION
In this work, we introduce a procedure to certify robustness of 1 -Lipschitz convolutional neural networks without orthogonalizing of the last linear layer of the network.
Additionally, we introduce a certiﬁcate regularization that signiﬁcantly improves the provable robust accuracy for these models at higher l 2 radii.
Finally, we introduce a class of GNP activation functions called Householder (or HH ) activations and prove that the Jacobian of any Gradient Norm Preserving (GNP) piecewise linear function is only allowed to change via Householder transformations for the function to be continuous [Table] which provides a general principle for designing piecewise linear GNP functions.
These ideas lead to improved deterministic (cid:96) 2 robustness certiﬁcates CIFAR- and A CKNOWLEDGEMENTS
This project was supported in part by NSF CAREER AWARD 1942230, a grant from NIST 60NANB20D134, HR001119S0026-GARD-FP-052, HR00112090132, ONR YIP award N00014-221-2271, Army Grant W911NF2120076.
R EPRODUCIBILITY
The code for reproducing all experiments in the paper is available at https://github.com/ singlasahil14/SOC .
E THICS S TATEMENT
This paper introduces novel set of techniques for improving the provable robustness of 1 -Lipschitz
CNNs.
We do not see any foreseeable negative consequences associated with our work.
A P ROOFS A. P ROOF OF P ROPOSITION
Proof.
We proceed by computing the Lipschitz constant of the function f l − f i .
The gradient of the function: f l − f i at x can be computed using the chain rule:
[Equation]
Since g is given to be 1 -Lipschitz, the Lipschitz constant of f l − f i can be computed using the above equation as follows:
[Equation]
Thus
[Equation]
the distance of x to the decision boundary f l − f i = 0 , is lower bounded by:
[Equation]
Thus
[Equation]
the distance to decision boundary across all classes
i (cid:54) =
l is lower bounded by:
[Equation]
A. P ROOF OF T HEOREM
Proof.
We ﬁrst prove that if Q 2 = ( I − 2 vv T ) Q 1 , then the function g is continuous.
First, observe that for v T z > 0 , g ( z ) =
Q 1 z which is continuous.
Similarly, for v T z < 0 , g ( z ) =
Q 2 z which is again continuous.
This proves that the function g is continuous when v T z > 0 or v T z < 0 .
Thus, to prove continuity ∀ z ∈ S , we must prove that:
[Equation]
Since Q 2 = Q 1 ( I − 2 vv T ) , we have:
[Equation]
The above equation directly proves (9).
Now, we prove the other direction i.e if g is continuous in S then, Q 2 = Q 1 ( I − 2 vv T ) .
Since g is continuous for all z : v T z = 0 , we have:
[Equation]
Since z ∈ R m , we know that the set of vectors: { z : v T z = 0 } spans a m − 1 dimensional subspace.
Thus, the null space of Q 2 − Q 1 is of size m − 1 .
This in turn implies that Q 2 − Q 1 is a rank one matrix given by the following equation:
[Equation]
where the vector u remains to be determined.
Since Q 1 and Q 2 are orthogonal matrices, we have the following set of equations:
[Equation]
[Equation]
We ﬁrst simplify equation (11):
[Equation]
Right multiplying both sides by v and using (cid:107) v (cid:107) 2 = 1 , we get:
[Equation]
=
λ v u = λ Q 1 v , where λ = − (cid:0)
u T u + u T Q 1 v (cid:1)
[Equation]
Substituting u using equation (13) in equation (10), we get:
[Equation]
Since Q T 2 Q 2 =
I
[Equation]
we have:
[Equation]
Since λ = 0 would imply Q 1 = Q 2 which is not allowed by the assumption of the Theorem that
Q 1 (cid:54) =
Q 2 .
λ = − 2 is the only possibility allowed.
This proves the other direction i.e:
[Equation]
A. P ROOF OF C OROLLARY
Proof.
Subsitute Q 1 , v as follows in Theorem 1.
[Equation]
Theorem 1 then directly implies the corollary.
A. P ROOF OF T HEOREM
Theorem 2.
Given: 0 ≤ θ 0 < θ 1 · · · < θ 2 n = 2 π + θ 0 such that (cid:80) n − 1 i =
( θ 2 i +1 − θ 2 i ) =
π and α i = 2 (cid:80) ij =
θ i − j ( − 1) j , The function σ Θ : R 2 → R 2 is continuous, GNP and 1 -Lipschitz where Θ = [ θ 0 , θ 1 , . . .
, θ 2 n − 1 ] (also called 2D Householder Activation of order n ):
[Equation]
where ϕ ∈ [ θ 0 , θ 2 n = 2 π + θ 0 ) and cos( ϕ ) =
z 1 / (cid:112) z 21 + z 22 , sin( ϕ ) [Equation] z
[Equation]
(cid:112) z [Equation] z 22 .
Proof.
We are given the following:
[Equation]
Note that by deﬁnition (equation (14)), the function is linear for θ i < ϕ <
θ i +1 and hence continuous.
Furthermore, since ϕ ∈ [ θ 0 , θ 2 n ) , we proceed to prove continuity for the following two cases:
[Equation]
Proof for Case 1 :
Using equation (14), we know that σ Θ realizes different linear functions for θ i − (cid:15) <
ϕ < θ i and θ i <
ϕ < θ i + (cid:15) .
Thus, for σ Θ to be continuous, we require that the two linear functions be the same at the boundary i.e ϕ
= θ i .
We ﬁrst write the input ( z 1 , z 2 ) in terms of shifted polar coordinates i.e: ( r cos( ϕ ) , r sin( ϕ )) where r = (cid:112) z
21 + z 22 and cos
[Equation]
=
[Equation]
/ (cid:112) [Equation]
[Equation]
Thus, the function for θ i − (cid:15) <
ϕ < θ i is given by:
[Equation]
Similarly, the function for θ i < ϕ
< θ i + (cid:15) is given by:
[Equation]
The difference in the function values at the boundary i.e ϕ
=
θ
i , obtained by subtracting equations (17) and (16) is given as follows:
[Equation]
Using sum-to-product trigonometric identities
[Equation]
the above equals: = r
[Equation]
Using equation (15), we directly have: α i =
2 θ
i − α
i − 1 .
Thus, the above equation reduces to:
[Equation]
Hence, the linear functions given by equations (16) and (17) are equal at ϕ
= θ i .
This proves that the function is continuous for Case 1.
Proof for Case 2 :
Using equation (14), we know that σ Θ realizes different linear functions for θ 0 < ϕ <
θ 0 + (cid:15) and θ 2 n − (cid:15) <
ϕ < θ 2 n .
Thus, for σ Θ to be continuous, we require that the two linear functions be the same at the boundary i.e ϕ = θ 0 .
As before, we ﬁrst write the input ( z 1 , z 2 ) in terms of shifted polar coordinates i.e: ( r cos( ϕ ) , r sin( ϕ )) .
Thus, the function for θ 0 < ϕ <
θ
+ (cid:15) is given by:
[Equation]
Similarly, the function for θ 2 n − (cid:15) <
φ < θ 2 n is given by:
[Equation]
Using equation (15), α 2 n − 1 is given as follows:
[Equation]
Thus, equation (19) reduces to:
[Equation]
The difference in the function values at the boundary i.e ϕ = θ 0 , obtained by subtracting equations (20) and (18) is given as follows:
[Equation]
Using the trigonometric identities: 1 − cos( θ ) = 2 sin 2 ( θ/ 2) , 1 + cos( θ ) =
cos 2 ( θ/ 2) and sin( θ ) = 2 sin( θ/ 2) cos( θ/ 2) , we have:
[Equation]
Using equation (15), we have: α
= 2 θ 0 .
Thus, the above equation reduces to:
[Equation]
Hence, the linear functions given by equations (18) and (20) are equal at ϕ = θ 0 .
This proves that the function is continuous for Case 2.
B S ELECTION OF γ USING CROSS VALIDATION
Using 5000 held out samples from CIFAR-10, we tested 6 different values of γ shown in Table 3 and selected γ = 0 .
5 because it resulted in less than 0 .
5% decrease in standard accuracy while 4 .
96% increase in provably robust accuracy.
We used the LipConvnet- 5 network with the 2 D Householder activation function of order 2 i.e σ θ .
[Table]
C D IFFERENCES BETWEEN l 1 AND l 2 CERTIFICATE
By the equivalence of norms, we have the following result:
[Equation]
Let us assume that we have an l 1 certiﬁcate for the input x so that the prediction of a neural network remains constant in a region of l 1 radius ρ 1 around the input x i.e:
[Equation]
We want to compute the l 2 certiﬁcate implied by the certiﬁcate given in equation (22).
Let ρ 2 be the l 2 certiﬁcate so that:
[Equation]
Using equation (21), we have the following set of equations:
[Equation]
Using equation (22), we have the following:
[Equation]
Hence, the l 2 norm certiﬁcate induced by the l 1 norm certiﬁcate can be signiﬁcantly smaller for high dimensional inputs.
Using the l 1 certiﬁcate used in Levine & Feizi (2021)
i.e ρ 1 = 4 and d = 32 √for CIFAR-10 and CIFAR-100, we get an implied certiﬁcate of ρ 2 = 4 / (32 √ 3) = 0 .
In contrast, in this work we study l 2 robustness for much higher certiﬁcate values i.e 36 / 255 = 0 . 14118 .
D R ESULTS USING REVISED L IPSCHITZ CONSTANTS
In Tables 4 and 5, we show results where the Lipschitz constant was computed using product of
Lipschitz constant of all layers.
The Lipschitz constant of each layer was computed using direct power iteration on the linear layer (using 50 iterations) and not using the approximation bound provided in
Singla & Feizi (2021).
[Table]
% + CR 47.84% 34.77% 23.70% 15.83% +4.78%
LipConvnetSOC + MaxMin 43.37% 28.59% 18.17% 10.85% _ + LLN 46.32% 32.87% 21.53% 13.86% +2.95%
+ HH 47.70% 34.00% 22.67% 14.57% +4.33% + CR 46.87% 34.09% 23.41% 15.61% +3.50%
LipConvnetSOC + MaxMin 42.87% 28.74% 18.47% 11.20% _ + LLN 46.18% 32.82% 21.52% 13.52% +3.31% + HH 46.80% 33.72% 22.70% 14.31%
+3.93% + CR 46.92% 34.17% 23.21% 15.84%
+4.05%
LipConvnetSOC + MaxMin 42.42% 28.34% 18.10% 10.96% _ + LLN 45.22% 32.10% 21.28% 13.25%
+2.80%
+ HH 46.21% 32.80% 21.55% 14.13%
+3.79% + CR 46.88% 33.64% 23.34% 15.73%
+4.46%
LipConvnetSOC + MaxMin 41.84% 28.00% 17.40% 10.28% _ + LLN 42.94% 29.71% 19.30% 11.99%
+1.10%
+ HH 45.84% 32.79% 21.98% 14.07% +4.00% + CR 45.03% 32.56% 22.
[Table] [Table]
(a) The output of σ θ always lies in the pink region.
Applying σ φ on this where φ (cid:54)
= θ + 2 nπ, n
∈ Z further divides this region into two linear regions (light and original pink).
(b) In each colored region, the function is linear with the Jacobian mentioned.
For the function to be continuous, we must have θ 3 − θ 2 + θ 1 − θ
= π .
Thus, any 3 of { θ 0 , θ 1 , θ 2 , θ 3 } can be chosen as learnable parameters.
E V ERIFICATION THAT σ θ ( z 1 , z 2 ) ALWAYS LIES ON ONE SIDE OF THE HYPERPLANE
Consider the case: z 1 sin ( θ/ 2) − z 2 cos ( θ/ 2) >
In this case σ θ ( z 1 , z 2 )
= ( z 1 , z 2 ) and the result follows directly.
Consider the other case: z 1 sin ( θ/ 2) − z 2 cos ( θ/ 2) ≤(cid:20) aa(cid:21) = (cid:20) cos θ sin θ sin θ
[Equation]
a 1 sin ( θ/ 2) − a 2 cos ( θ/ 2) =
( z 1 cos θ + z 2 sin θ ) sin ( θ/ 2) − ( z 1 sin θ − z 2 cos θ ) cos ( θ/ 2) =
z 1 (cos θ sin ( θ/ 2) − sin θ cos ( θ/ 2))
+ z 2 (sin θ sin ( θ/ 2)
+ cos θ cos ( θ/ 2))
= − z 1 sin ( θ/ 2) + z 2 cos ( θ/ 2)
Since z 1 sin ( θ/ 2) − z 2 cos ( θ/ 2) ≤ 0 , we have − z 1 sin ( θ/ 2) + z 2 cos ( θ/ 2) ≥ 0 .
F H IGHER ORDER H OUSEHOLDER ACTIVATION FUNCTIONS
We know that MaxMin( z 1 , z 2 ) =
(max( z 1 , z 2 ) , min( z 1 , z 2 )) where z 1 , z 2 ∈ R .
Because max( z 1 , z 2 ) > min( z 1 , z 2 ) , applying MaxMin again gives the same result i.e MaxMin ◦ MaxMin =
MaxMin .
Now consider the function σ θ (discussed in the maintext, given below for convenience):
[Equation]
From Figure 1, we observe that if ( u 1 , u 2 ) = σ θ ( z 1 , z 2 ) , then ( u 1 , u 2 ) always lies on the right side of the hyperplane (pink colored region).
In other words, u 1 sin ( θ/ 2) − u 2 cos ( θ/ 2) > 0 ∀ z 1 , z(proof in Appendix E).
This further implies that σ θ ◦ σ θ = σ θ .
However from Figure 2a in the maintext, we observe that if we use a different angle φ where φ (cid:54)
= θ + 2 nπ for some n ∈ Z , then σ φ ( u 1 , u 2 ) (cid:54) =
( u 1 , u 2 ) for all ( u 1 , u 2 ) in the pink colored region ( u 1 sin ( θ/ 2) − u 2 cos ( θ/ 2) > 0 ).
This motivates us to construct the function σ ( n ) : R 2 → Rdeﬁned as follows:
[Equation]
Clearly, σ ( n ) has a larger number of linear regions than σ θ and thus expected to have more expressive power.
However, a drawback of using σ ( n ) is that it requires a sequential application of σ θ which can be expensive if the number of iterations n is large.
To address this limitation, ﬁrst observe that σ θ realizes the same linear function for ( z 1 , z 2 ) and ( cz 1 , cz 2 ) when c > 0 i.e ∇ ( z 1 ,z 2 ) σ θ = ∇ ( cz 1 ,cz 2 ) σ θ .
Since σ θ is piecewise linear, σ θ ( cz 1 , cz 2 ) = cσ θ
( z 1 , z 2 ) .
Thus, the input of the next function in the iteration is scaled by c as well and its linear function (or the Jacobian) remains unchanged.
By induction, same holds for all the subsequent iterations.
By chain rule, the Jacobian of composition of functions is equal to the product of Jacobian of each individual function.
Since the Jacobian of each function is unchanged on scaling by c > 0 , the Jacobian ∇ ( z 1 ,z 2 ) σ ( n ) also remains unchanged: ∇ ( z 1 ,z 2 ) σ ( n ) = ∇ ( cz 1 ,cz 2 ) σ ( n ) .
This suggests that it is possible to determine the Jacobian ∇ ( z 1 ,z 2 ) σ ( n ) for the input ( z 1 , z 2 ) by ﬁrst converting to the polar coordinates ( (cid:112) z
21 + z 22 , ϕ ) and then using the phase angle ϕ alone (where cos( ϕ ) =
z 1 / (cid:112) z 21 + z 22 , sin( ϕ ) =
z 2 / (cid:112) z 21 + z 22 ).
This motivates us to construct another GNP piecewise linear activation that only depends on the phase of the input but unlike σ θ , it is allowed to have more than 2 linear regions without requiring a sequential application.
This construction is given in the following theorem (example in Figure 2b):
Theorem 2.
Given: 0 ≤ θ 0 < θ 1 · · · < θ 2 n = 2 π + θ 0 such that (cid:80) n − 1 i =
( θ 2 i +1 − θ 2 i ) =
π and α i = 2 (cid:80) ij =
θ i − j ( − 1) j , The function σ Θ : R 2 → R 2 is continuous, GNP and 1 -Lipschitz where Θ = [ θ 0 , θ 1 , . . .
, θ 2 n − 1 ] (also called Householder Activation of order n in 2 dimensions):
[Equation]
where ϕ ∈ [ θ 0 , θ 2 n = 2 π + θ 0 ) and cos( ϕ ) =
z 1 / (cid:112) z 21 + z 22 , sin( ϕ ) =
z 2 / (cid:112) z 21 + z 22 .
Using the deﬁnition of α i , α 2 n − 1 can be computed as follows:
[Equation]
By continuity, σ Θ ( z 1 , z 2 ) =
( z 1 , z 2 ) for ϕ
= θ 0 .
Thus if we set θ 0 = 0 , σ Θ is ﬁxed to be identity when ϕ
= 0 (or z 2 = 0 ).
However, a learnable θ 0 offers the ﬂexibility of choosing arbitrary intervals around the ϕ
= θ 0 to be the identity function (while of course allowing θ 0 = 0 ).
Since we can choose any interval of 2 π for the phase angle, we choose ϕ ∈
[ θ 0 , θ 2 n = θ 0 + 2 π ) instead of the usual [0 , 2 π ) to allow this possibility.
We call ( (cid:112) z
21 + z 22 , ϕ ) , the shifted polar coordinates .
Additionally, we make the following observations about Theorem 2.
First, by construction σ Θ has 2 n linear regions.
Second, since (cid:80) n − 1 i =
( θ 2 i +1 − θ 2 i ) =
π , the sum of angles subtended by linear regions with determinant − 1 equals π .
This in turn implies that sum of angles subtended by linear regions with determinant +1 must also equal π .
Third, again using (cid:80) n − 1 j =0 ( θ 2 j +1 − θ 2 j ) =
π , we know that only 2 n − 1 of the 2 n parameters in [ θ 0 , θ 1 , θ 2 , . . .
, θ 2 n − 1 ] can be chosen independently implying that σ Θ has 2 n − 1 learnable parameters.
In contrast, σ ( n ) has only n learnable parameters.
Fourth, when n = 1 , σ Θ reduces to the original σ θ activation.
Because every function of the form σ ( n ) (equation (23)) can have potentially 2 n linear regions, while σ Θ has only 2 n linear regions, σ Θ cannot express every function of the form σ ( n ) .
The primary beneﬁt of using σ Θ is that it can be easily applied by ﬁrst determining the angle ϕ (using shifted polar coordinates), the region [ θ i , θ i +1 ) to which ϕ belongs and the Jacobian for this region.
This requires 1 multiplication with the Jacobian instead of n required for σ ( n ) .
G E XTENSION TO HIGHER DIMENSIONS
We introduced Householder activation function of Order 1 in m dimensions in maintext Deﬁnition 1.
However, it suffers from the limitation that it has only 2 linear regions thus limiting its expressive power.
The construction given in Appendix Section F allows more than 2 linear regions but is valid only for 2 dimensional inputs.
This motivates us to construct Householder activations that depend on all the m components of input z ∈ R m , ( m ≥ 3) while allowing
A straightforward way of constructing such an activation function is to apply an orthogonal matrix
Q ∈ R m × m , followed by dividing the output Qz into groups of size 2 each and then applying σ θ to each group.
However, since 1 -Lipschitz neural networks involve multiplication with an orthogonal weight matrix followed by GNP activation anyway, this construction is trivial because it does not lead to additional gains in expressive power over using 2 dimensional σ Θ activation functions.
Recall that the function σ v is given by the following equation:
[Equation]
By a similar analysis as for the 2 -dimensional case (Figure 2 in maintext), a repeated application of σ v leads to increased number of linear regions and thus higher expressive power.
This motivates us to construct the function σ ( m,n ) :
R m → R m by applying the function σ v (equation (24)) n times iteratively with different learnable parameter v at each iteration:
[Equation]
Since σ v realizes the same linear function for both the inputs z and c z i.e ∇ z σ v = ∇ c
z σ v when c > 0 , σ ( m,n ) satisﬁes this property as well.
This suggests that it is possible to determine the Jacobian of σ ( m,n ) for the given input z by projecting z onto a unit sphere: z / (cid:107) z (cid:107) 2 .
Moreover, we want our constructed function to have at least 2 n linear regions while requiring k iterations of σ v where k is independent of n .
This motivates the following open question:
Open Problem.
Can non-trivial order- n ( n > 1 ) householder activation functions with 2 n linear regions be constructed for m dimensional input ( m > 2 ) using k iterations of σ v where k is independent of n (but may depend on m )? [Equation]
DDITIONAL RESULTS ON CIFAR-
The rows "BCOP", "Cayley" and "SOC (baseline)" all use the MaxMin activation function.
"SOC + HH " replaces MaxMin with 2D Householder activation of order 1 ( σ θ ), "+ CR" adds Certiﬁcate
Regularization (CR) with γ = 0 .
1 (while using σ θ as the activation function).
In Table 9, the row "SOC + HH (2) " uses Householder activation of order 2 ( σ Θ deﬁned in Theorem 2), "+ CR" adds Certiﬁcate Regularization (CR) with γ = 0 .
1 (while using the HH activation of order 2 i.e σ Θ as the activation function).
None of the results in Tables 7, 8 and 9 include Last Layer Normalization (LLN).
[Table] [Table]
[Table] [Table]
None of these results include Last
[Table]
I A DDITIONAL R ESULTS ON CIFAR-
All results in Tables 13, 14 and 15 include Last Layer Normalization (LLN).
The rows "BCOP", "Cayley" and "SOC (baseline)" all use the MaxMin activation function.
"SOC + HH " replaces MaxMin with 2D Householder activation of order 1 ( σ θ ), "+ CR" adds Certiﬁcate
Regularization (CR) with γ = 0 .
1 (while using σ θ as the activation function).
In Table 15, the row "SOC + HH (2) " uses Householder activation of order 2 ( σ Θ deﬁned in Theorem 2), "+ CR" adds Certiﬁcate Regularization (CR) with γ = 0 .
1 (while using the HH activation of order 2 i.e σ Θ as the activation function).
[Table]
StandardAccuracy
[Table]
Increase [Table] [Table]
[Table]
[Table]
LipConvnet
[Table]
[Table]
_ + LLN
[Table]
+ HH
[Table]
[Table] [Table]
% +
[Table]
% + CR
[Table]
% +
[Table]
LipConvnet-
[Table]
_ + LLN [Table] % +
[Table]
+ HH
[Table]
[Table]
% +
[Table]
% + CR
[Table]
[Table]
[Table]
% +
[Table]
[Table]
[Table]
[Table]
[Table]
+ CR
[Table]
+ HH ( 2 )
[Table]
+ CR
[Table]
All of these results include Last Layer Normalization (
[Table]