O RTHOGONALIZING C ONVOLUTIONAL L AYERS WITH THE C AYLEY T RANSFORM
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work has highlighted several advantages of enforcing orthogonality in the weight layers of deep networks, such as maintaining the stability of activations, preserving gradient norms, and enhancing adversarial robustness by enforcing low
Lipschitz constants.
Although numerous methods exist for enforcing the orthogonality of fully-connected layers, those for convolutional layers are more heuristic in nature, often focusing on penalty methods or limited classes of convolutions.
In this work, we propose and evaluate an alternative approach to directly parameterize convolutional layers that are constrained to be orthogonal.
Speciﬁcally we propose to apply the Cayley transform to a skew-symmetric convolution in the Fourier domain so that the inverse convolution needed by the Cayley transform can be computed efﬁciently.
We compare our method to previous Lipschitzconstrained and orthogonal convolutional layers and show that it indeed preserves orthogonality to a high degree even for large convolutions.
Applied to the problem of certiﬁed adversarial robustness
, we show that networks incorporating the layer outperform existing deterministic methods for certiﬁed defense against
(cid:96) 2norm-bounded adversaries
, while scaling to larger architectures than previously investigated.
I NTRODUCTION
Encouraging orthogonality in neural networks has proven to yield several compelling beneﬁts.
For example, orthogonal initializations allow extremely deep vanilla convolutional neural networks to be trained quickly and stably (Xiao et al., 2018;
Saxe et al., 2013).
And, initializations that remain closer to orthogonality throughout training seem to learn faster and generalize better (Pennington et al., 2017).
Unlike Lipschitz-constrained layers, orthogonal layers are gradient-normpreserving (Anil et al., 2019), discouraging vanishing and exploding gradients and stabilizing activations (Rodr ´ ıguez et al., 2017).
Orthogonality is thus a potential alternative to batch normalization in
CNNs, and can help to remember long-term dependencies in RNNs (Arjovsky et al., 2016; Vorontsov et al., 2017).
Constraints and penalty terms encouraging orthogonality can improve generalization in practice (Bansal et al., 2018; Sedghi et al., 2018), improve adversarial robustness by enforcing low Lipschitz constants, and allow deterministic certiﬁcates of robustness (Tsuzuku et al., 2018).
Despite evidence for the beneﬁts of orthogonality constraints, and while there are many methods to orthogonalize fully-connected layers, the orthogonalization of convolutions has posed challenges.
More broadly, current Lipschitz-constrained convolutions rely on spectral normalization and kernel reshaping methods (Tsuzuku et al., 2018), which only allow loose bounds and can cause vanishing gradients.
Sedghi et al.
(2018) showed how to clip the singular values of convolutions and thus enforce orthogonality, but relied on costly alternating projections to achieve tight constraints.
Most recently, Li et al.
(2019) introduced the Block Convolution Orthogonal Parameterization (BCOP), which cannot express the full space of orthogonal convolutions.
In contrast to previous work, we provide a direct , more expressive, and scalable parameterization of orthogonal convolutions.
Our method relies on the Cayley transform, which is well-known for parameterizing orthogonal matrices in terms of skew-symmetric matrices, and can be easily extended to non-square weight matrices.
The transform requires efﬁciently computing the inverse of a particular convolution in the Fourier domain, which we show works well in practice.
We demonstrate that our Cayley layer is indeed orthogonal in practice when implemented in 32-bit precision, irrespective of the number of channels.
Further, we compare it to alternative convolutional and Lipschitz-constrained layers: we include them in several architectures and evaluate their deterministic certiﬁable robustness against an (cid:96) 2 -norm-bounded adversary.
Our layer provides stateof-the-art results on this task.
We also demonstrate that the layers empirically endow a considerable degree of robustness without adversarial training.
Our layer generally outperforms the alternatives, particularly for larger architectures.
R ELATED W ORK
Orthogonality in neural networks.
The beneﬁts of orthogonal weight initializations for dynamical isometry , i.e. , ensuring signals propagate through deep networks, are explained by Saxe et al.
and Pennington et al.
(2017), with limited theoretical guarantees investigated by Hu et al.
Xiao et al.
(2018) provided a method to initialize orthogonal convolutions, and demonstrated that it allows the training of extremely deep CNNs without batch normalization or residual connections.
The signal-preserving properties of orthogonality can help with remembering long-term dependencies in RNNs, on which there has been much work (Helfrich et al., 2018; Arjovsky et al., 2016).
One way to orthogonalize weight matrices is with the Cayley transform , which is often used in Riemannian optimization (Absil et al., 2009).
Helfrich et al.
(2018) and Maduranga et al.
(2019) avoid vanishing/exploding gradients in RNNs using the scaled Cayley transform.
Similarly, LezcanoCasado & Mart ´ ınez-Rubio (2019) use the exponential map, of which the Cayley transform is an approximant.
Li et al.
(2020) derive an iterative approximation of the Cayley transform for orthogonally-constrained optimizers and show it speeds the convergence of CNNs and RNNs.
However, they merely orthogonalize a matrix obtained by reshaping the kernel, which is not the same as an orthogonal convolution (Sedghi et al., 2018).
Our contribution is unique here in that we parameterize orthogonal convolutions directly , as opposed to reshaping kernels.
Bounding neural network Lipschitzness.
Orthogonality imposes a strict constraint on the Lipschitz constant, which itself comes with many beneﬁts: Lower Lipschitz constants are associated with improved robustness (Yang et al., 2020) and better generalization bounds (Bartlett et al., 2017).
Tsuzuku et al.
(2018) showed that neural network classiﬁcations can be certiﬁed as robust to (cid:96) 2 -norm-bounded perturbations given a Lipschitz bound and sufﬁciently conﬁdent classiﬁcations.
Along with Szegedy et al.
, they noted that the Lipschitz constant of neural networks can be bounded if the constants of the layers are known.
Thus, there is substantial work on Lipschitzconstrained and regularized layers, which we review in Sec. 5.
However, Anil et al.
(2019) realized that mere Lipschitz constraints can attenuate gradients, unlike orthogonal layers.
There have been other ideas for calculating and controlling the minimal Lipschitzness of neural networks, e.g. , using semi-deﬁnite programs or extreme value theory (Fazlyab et al., 2019; Hein & Andriushchenko, 2017; Weng et al., 2018), but constructing bounds from Lipschitz-constrained layers is more scalable and efﬁcient.
Besides Tsuzuku et al.
(2018)’s strategy for deterministic certiﬁable robustness, there are many approaches to deterministically verifying neural network defenses using SMT solvers (Huang et al., 2017; Ehlers, 2017; Carlini & Wagner, 2017), integer programming approaches (Lomuscio & Maganti, 2017; Tjeng & Tedrake, 2017; Cheng et al., 2017), or semideﬁnite programming (Raghunathan et al., 2018).
Wong et al.
(2018)’s approach to minimize
an
LP-based bound on the robust loss is more scalable, but networks made from Lipschitz-constrained components can be more efﬁcient still, as shown by Li et al.
(2019) who outperform their approach.
However, none of these methods yet perform as well as probabilistic methods (Cohen et al., 2019).
Consequently, orthogonal layers appear to be an important component to enhance the convergence of deep networks while encouraging robustness and generalization.
B ACKGROUND
Orthogonality.
Since we are concerned with orthogonal convolutions, we review orthogonal matrices: A matrix Q ∈ R n × n is orthogonal if Q T Q = QQ T
=
I .
However, in building neural networks, layers do not always have equal input and output dimensions: more generally, a matrix
U ∈ R m × n is semi-orthogonal if U T U
=
I or UU T
= I .
Importantly, if m ≥ n , then U is also norm-preserving:
(cid:107)
Ux (cid:107)
= (cid:107) x (cid:107) 2 for all x ∈ R n .
If m < n , then the mapping is merely non-expansive (a contraction), i.e., (cid:107) Ux (cid:107) 2 ≤ (cid:107) x (cid:107) 2 .
A matrix having all singular values equal to 1 is orthogonal, and vice versa.
Lipschitzness under the (cid:96) 2 norm.
Hence, a consequence of orthogonality is 1-Lipschitzness.
A function f : R n → R m is L -Lipschitz with respect to the (cid:96) 2 norm iff (cid:107) f
( x ) − f ( y ) (cid:107) 2 ≤ L (cid:107) x − y (cid:107)for all x, y ∈ R n .
If L is the smallest such constant for f , then it’s called the Lipschitz constant of f , denoted by Lip ( f ) .
An useful property for certiﬁable robustness is that the Lipschitz constant of the composition of f and g is upper-bounded by the product of their constants: Lip ( f ◦ g ) ≤
Lip ( f ) Lip ( g ) .
Since simple neural networks are fundamentally just composed functions, this allows us to bound their Lipschitz constants, albeit loosely.
We can extend this idea to residual networks using the fact that Lip ( f + g ) ≤ Lip ( f ) + Lip ( g ) , which motivates using a convex combination in residual connections.
More details can be found in Li et al.
; Szegedy et al.
Lipschitz bounds for provable robustness.
If we know the Lipschitz constant of the neural network, we can certify that a classiﬁcation with sufﬁciently a large margin is robust to (cid:96) 2 perturbations below a certain magnitude.
Speciﬁcally, denote the margin of a classiﬁcation with label t as
[Equation]
) which can be interpreted as the distance between the correct logit and the next largest logit.
Then if the logit function f has Lipschitz constant L , and M f ( x ) > √ 2 L(cid:15) , then f ( x ) is certiﬁably robust to perturbations { δ : (cid:107) δ (cid:107) 2 ≤ (cid:15) } .
Tsuzuku et al.
(2018) and Li et al.
(2019) provide proofs.
T HE C AYLEY TRANSFORM OF A C ONVOLUTION
Before describing our method, we ﬁrst review discrete convolutions and the Cayley transform; then, we show the need for inverse convolutions and how to compute them efﬁciently in the Fourier domain, which lets us parameterize orthogonal convolutions via the Cayley transform.
An unstrided convolutional layer with c in input channels and c out output channels has a weight tensor
W of shape R c out × c in × n × n , and takes an input X of shape R c in × n × n to produce an output Y of shape
R c out × n × n .
We refer to a pixel as a c in or c out -dimensional slice of such a tensor, like X [: , i, j ] .
Each c out -dimensional pixel of the output is computed by centering W [ c, : , :] on the ( i, j ) th pixel of the input X and computing a dot product.
Typically, W is zero except for a k × k region, which we call the kernel or the receptive ﬁeld .
Typically, convolutional layers have small kernels, e.g. , k = 3 .
The Cayley transform is a bijection between skew-symmetric matrices A and special orthogonal matrices Q without a − 1 eigenvalue (we will look at this caveat later):
[Equation]
A matrix is skew-symmetric if A = − A T , and we can skew-symmetrize any square matrix W as
A = W − W T .
We can also apply the Cayley transform to convolutions, noting they are also linear transformations.
While it is straightforward to skew-symmetrize a convolution with weight W , e.g. , conv W ( x ) − conv W T ( x ) , efﬁciently ﬁnding the inverse convolution conv − 1 I + A ( x ) is more difﬁcult:
A convolution does not generally even have the same kernel size as its inverse convolution.
This motivates us to work in the Fourier domain instead, where we can ﬁnd the inverse efﬁciently.
It is easiest to analyze convolutions when they are circular : if the kernel goes out of bounds of X , it wraps around to the other side—this operation can be carried out efﬁciently in the Fourier domain.
According to the convolution theorem , the circular convolution of W and X in the Fourier domain is simply their elementwise product: a W with a small receptive ﬁeld only modiﬁes high frequencies, and larger receptive ﬁelds can modify low frequencies.
We denote the 2D Discrete (Fast)
Fourier
Transform with FFT : R n × n → C n × n , which maps spatial information in R n × n to frequency components in C n × n .
It is deﬁned by FFT ( X ) =
F n XF ∗ n , where F n [ i, j ] = 1 √ n exp( − 2 πın ) ( i − 1)( j − 1) .
The c th output channel in the Fourier domain can be expressed as the sum of elementwise products of the c in input and kernel channels, or equivalently, the c th channel of each ( i, j ) th output pixel is the dot product of the ( i, j )
th input pixel and ( i, j ) th receptive ﬁeld pixel:
FFT ( conv W ( x ))
[: , i, j ] =
FFT ( W )[: , : , i, j ] FFT ( X )
[: , i, j ] .
˜ j ] − ˜ i j ] T
˜ j ]
1 ˜
[: , i, j ] j ] − j ]
From this, we can see that the ( i, j )
th Fourier domain output pixel is given by the matrix-vector product FFT ( W )
[: , : , i, j ] FFT ( X )[: , i, j ] .
This interpretation gives a way to compute the inverse convolution as required for the Cayley transform, assuming c in = c out :
[Equation]
Given this method to compute inverse convolutions, we can now parameterize an orthogonal convolution with a skew-symmetric convolution through the Cayley transform, highlighted in Algorithm 1:
In line 1, we use the Fast Fourier Transform on the weight and input tensors.
In line 4, we compute the Fourier domain weights for the skew-symmetric convolution.
Next, in lines 4–5 we compute the inverses required for FFT ( conv − 1 I + A ( x )) and use them to compute the Cayley transform written as ( I + A ) − 1 − A ( I + A ) − 1 in line 6.
Finally, we get our spatial domain result with the inverse FFT.
In practice, we found that learning was easier if we instead parameterized W in Algorithm 1 by
W = gV/ (cid:107) V (cid:107) 2 for a learnable scalar g and tensor V , i.e. , as in weight normalization (Salimans &
Kingma, 2016).
4. P ROPERTIES OF OUR APPROACH
It is important to note that the inverse in the Cayley transform always exists : Because A is skew symmetric, it has all eigenvalues either zero or imaginary, so I + A has all nonzero eigenvalues and is thus nonsingular.
Since only square matrices can be skew-symmetrized and inverted, Algorithmonly works for square matrices, but can be extended to the rectangular case where c out ≥ c in by padding the matrix with zeros and then projecting out the ﬁrst c in columns after the transform, resulting in a norm-preserving semi-orthogonal matrix; the case c in ≥ c out follows similarly, leading to a contraction.
With efﬁcient implementation in terms of the Schur complement (Appendix B), this only requires inverting a square matrix of order min( c in , c out ) .
While the Block Convolution Orthogonal Parameterization (BCOP) (Li et al., 2019) only expresses the limited subset of convolutions with k × k -sized kernels, our parameterization does not have that restriction, and does not rely on costly projections like (Sedghi et al., 2018).
However, our parameterization as presented is limited to special orthogonal convolutions, or those withouteigenvalues.
However, it can be extended to all orthogonal convolutions by multiplying by a diagonal matrix with ± 1 entries (Gallier, 2006; Helfrich et al., 2018).
In our experiments, we did not ﬁnd improvements from this.
Runtime comparison Our Cayley layer does c in c out FFTs on n × n matrices ( i.e. , the kernels padded to the input size), and c in FFTs for each n × n input.
These have complexity O
( c in c out n 2 log n ) and O ( c out n 2 log n ) respectively.
The most expensive step is computing the inverse of n 2 square matrices of order c = min( c in , c out ) , with complexity O ( n 2 c 3 ) , similarly to the method of Sedghi et al.
We note like the authors that parallelization could effectively make this O ( n 2 log n + c 3 ) , and it is quite feasible in practice.
As in Li et al.
(2020), the inverse could be replaced with an iterative approximation, but we did not ﬁnd it necessary for our relatively small architectures.
For comparison, the related layers BCOP and RKO (Sec. 5) take only O ( c 3 ) to orthogonalize the convolution, and OSSN takes O ( n 2 c 3 )
(Li et al., 2019).
In practice, we found our Cayley layer takes anywhere from 1 / 2 × to 4 × as long as BCOP, depending on the architecture (see Appendix C).
E XPERIMENTS
Our experiments have two goals: First, we show that our layer remains orthogonal in practice.
Second, we compare the performance of our layer versus alternatives (particularly BCOP) on two adversarial robustness tasks on CIFAR-10:
We investigate the certiﬁable robustness against an (cid:96) 2 -norm-bounded adversary using the idea of Lipschitz Margin Training (Tsuzuku et al., 2018), and then we look at robustness in practice against a powerful adversary.
We ﬁnd that our layer is always orthogonal and performs relatively well in the robustness tasks.
For alternative layers, we adopt the naming scheme for previous work on Lipschitz-constrained convolutions from Li et al.
and we compare directly against their implementations.
We outline the methods below.
RKO.
A convolution can be represented as a matrix-vector product, e.g. , using a doubly blockcirculant matrix and the unrolled input.
Alternatively, one could stack each k × k receptive ﬁeld, and multiply by the c out × k 2 c in reshaped kernel matrix (Cisse et al., 2017).
The spectral norm of this reshaped matrix is bounded by the convolution’s true spectral norm (Tsuzuku et al., 2018).
Consequently, reshaped kernel methods orthogonalize this reshaped matrix, upper-bounding the singular values of the convolution by 1.
Cisse et al.
(2017) created a penalty term based on this matrix; instead, like Li et al.
(2019), we orthogonalize the reshaped matrix directly, called reshaped kernel orthogonalization (RKO) .
They used an iterative algorithm for orthogonalization (Bj ¨ orck & Bowie, 1971); for comparison, we implement RKO using the Cayley transform, called CRKO .
OSSN.
A prevalent idea to constrain the Lipschitz constants of convolutions is to approximate the maximum singular value and normalize it out: Miyato et al.
(2018) used the power method on the matrix W associated with the convolution, i.e. , s i +1 :
= W T Ws i , and σ max ≈ (cid:107)
Ws n (cid:107) / (cid:107) s n (cid:107) .
Gouk et al.
(2018) improved upon this idea by applying the power method directly to convolutions, using the transposed convolution for W T .
However, this one-sided spectral normalization is quite restrictive; dividing out σ max can make other singular values vanishingly small.
SVCM.
Sedghi et al.
(2018) showed how to exactly compute the singular values of convolutional layers using the Fourier transform before the SVD, and proposed a singular value clipping method .
However, the clipped convolution can have an arbitrarily large kernel size, so they resorted to alternating projections between orthogonal convolutions and k × k -kernel convolutions, which can be expensive.
Like Li et al.
(2019), we found that ≈ 50 projections are needed for orthogonalization.
BCOP.
The Block Convolution Orthogonal Parameterization extends the orthogonal initialization method of Xiao et al.
It differentiably parameterizes k × k orthogonal convolutions with an orthogonal matrix and 2( k − 1) symmetric projection matrices.
The method only parameterizes the subspace of orthogonal convolutions with k × k -sized kernels, but is quite expressive empirically.
Internally, orthogonalization is done with the method by Bj ¨ orck & Bowie (1971).
5. T RAINING AND A RCHITECTURAL D ETAILS
Training details.
For all experiments, we used CIFAR-10 with standard augmentation, i.e. , random cropping and ﬂipping.
Inputs to the model are always in the range [0 , 1] ; we implement normalization as a layer for compatibility with AutoAttack.
For each architecture/convolution pair, we tried learning rates in { 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 } , choosing the one with the best test accuracy.
Most often, 0 . 001 is appropriate.
We found that a piecewise triangular learning rate, as used in top performers in the DAWNBench competition (Coleman et al., 2017), performed best.
Adam (Kingma & Ba, 2014) showed a signiﬁcant improvement over plain SGD, and we used it for all experiments.
Loss function.
Inspired by Tsuzuku et al.
(2018), Anil et al.
and
Li et al.
(2019) used multiclass hinge loss where the margin is the robustness certiﬁcate √ 2 L(cid:15) 0 .
We corroborate their ﬁnding that this works better than cross-entropy, and similarly use (cid:15)
= 0 .
Varying (cid:15) 0 controls a tradeoff between accuracy and robustness (see Fig. 5).
Initialization.
We found that the standard uniform initialization in PyTorch performed well for our layer.
We adjusted the variance, but signiﬁcant differences required order-of-magnitude changes.
For residual networks, we tried Fixup initialization (Zhang et al., 2019), but saw no signiﬁcant improvement.
We hypothesize this is due to (1) the learnable scaling parameter inside the Cayley
[Figure]
Architecture considerations.
For fair comparison with previous work, we use the “large” network from Li et al.
(2019), which was ﬁrst implemented in Kolter & Wong (2017)’s work on certiﬁable robustness.
We also compare the performance of the different layers in a 1-Lipschitz-constrained version of ResNet9 (He et al., 2016) and WideResNet10-10 (Zagoruyko & Komodakis, 2016).
The architectures we could investigate were limited by compute and memory, as all the layers compared are relatively expensive.
For RKO, OSSN, SVCM and BCOP, we use Bj¨orck orthogonalization
Bj ¨ orck & Bowie (1971) for fully-connected layers, as reported in Li et al.
Anil et al.
For our Cayley layer and CRKO, we use the Cayley transform for consistency.
We found the gradient-norm-preserving GroupSort activation function from Anil et al.
(2019) to be more effective than ReLU, and we used a group size of 2, i.e. , MaxMin .
Strided convolutions.
For the KWLarge network, we used “invertible downsampling”, which emulates striding by rearranging the inputs to have 4 × more channels while halving the two spatial dimensions and reducing the kernel size to (cid:98) k/ 2 (cid:99) (Jacobsen et al., 2018).
For the residual networks, we simply used a version of pooling, noting that average pooling is still non-expansive when multiplied by its kernel size, which allows us to use more of the network’s capacity.
We also halved the kernel size of the last pooling layer, instead adding another fully-connected layer; empirically, this used resulted in higher local Lipschitz constants.
Ensuring Lipschitz constraints.
Batch normalization layers scale their output, so they can’t be included in our 1-Lipschitz-constrained architecture; the gradient-norm-preserving properties of our layers compensate for this.
We ensure residual connections are non-expansive by making them a convex combination with a new learnable parameter α , i.e. , g ( x ) =
αf ( x )+(1 − α )
x , for α ∈ [0 , 1] .
To ensure the latter constraint, we use sigmoid ( α ) .
We can tune the overall Lipschitz bound to a given L using the Lipschitz composition property, multiplying each of the m layers by L 1 /m .
5. A DVERSARIAL R OBUSTNESS
For certiﬁable robustness, we report the fraction of certiﬁable test points: i.e. , those with classiﬁcation margin M f ( x ) greater than √ 2 L(cid:15) , where (cid:15) = 36 / 255 .
For empirical defense, we use both vanilla projected gradient descent and AutoAttack by Croce & Hein (2020).
For PGD, we use α = (cid:15)/ 4 . 0 with 10 iterations.
Within AutoAttack, we use both APGD-CE and APGD-DLR, ﬁnding the decision-based attacks provided no improvements.
We report on (cid:15) = 36 / 255 for consistency with Li et al.
and previous work on deterministic certiﬁable robustness (Wong et al., 2018).
Additionally, we found it useful to report on empirical local Lipschitz constants throughout training using the PGD-like method from Yang et al.
5. R ESULTS
Practical orthogonality.
We show that our layer remains very close to orthogonality in practice, both before and after learning.
We investigated Cayley layers from one of our ResNet9 architectures, running them on random tensors to see if their norm is preserved, which is equivalent to orthogonality.
We found that (cid:107) Conv ( x ) (cid:107) / (cid:107) x (cid:107) , the extent to which our layer is gradient norm preserving, is always extremely close to 1.
We illustrate the small discrepancies, easily bounded between 0 .
[Table]
Trained for provable robustness
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
and 1 . 00001 , in Figure 1.
Cayley layers which do not change or increase the number of channels are guaranteed to be orthogonal, which we see in practice for graphs (b, c, d, e).
Those which decrease the number of channels can only be non-expansive, and in fact the layer seems to become slightly more norm-preserving after training (a).
In short, our Cayley layer can capture the full beneﬁts of orthogonality.
Certiﬁable robustness.
We use our layer and alternatives within the KWLarge architecture for a more direct comparison to previous work on deterministic certifable robustness (Li et al., 2019;
Wong et al., 2018).
As in (Li et al., 2019), we got the best performance without normalizing inputs, and can thus say that all networks compared here are at most 1-Lipschitz.
Our layer outperforms BCOP on this task (see Table 1), and is thus state-of-the-art, getting on average 75 .
33% clean test accuracy and 59 .
16% certiﬁable robust accuracy against adversarial perturbations with norm less than (cid:15) = 36 / 255 .
In contrast, BCOP gets 75 .
11% test accuracy and 58 .
29% certiﬁable robust accuracy.
The reshaped kernel methods perform only a percent or two worse on this task, while the spectral normalization and clipping methods lag behind.
We assumed that a layer is only meaningfully better than the other if both the test and robust accuracy are improved; otherwise, the methods may simply occupy different parts of the tradeoff curve.
Since reshaped kernel methods can encourage smaller Lipschitz constants than orthogonal layers (Sedghi et al., 2018), we investigated the clean vs. certiﬁable robust accuracy tradeoff enabled by scaling the
Lipschitz upper bound of the network, visualized in Figure 2.
To that end, in light of the competitiveness of RKO, we chose a Lipschitz upper-bound of 0 . 85 which gave our Cayley layer similar test accuracy; this allowed for even higher certiﬁable robustness of 59 .
99% , but lower test accuracy of 74 .
Overall, we were surprised by the similarity between the four top-performing methods after scaling Lipschitz constants.
We were not able to improve certiﬁable accuracy with ResNets.
However, it was useful to increase the kernel size: we found 5 was an improvement in accuracy, while 7 and 9 were slightly worse.
(Since our method operates in the Fourier domain, increases in kernel size incur no extra cost.)
We also saw an improvement from scaling up the width of each layer of KWLarge, and our Cayley layer was substantially faster than BCOP as the width of KWLarge increased (see Appendix C).
Multiplying the width by 3 and increasing the kernel size to 5, we were able to get 61 .
13% certiﬁed robust accuracy with our layer, and 60 .
55% with BCOP.
[Table] [Table]
[Figure]
Empirical robustness.
Previous work has shown that adversarial robustness is associated with lower
Lipschitz constants.
Thus, we investigated the robustness endowed by our layer against (cid:96) 2 gradientbased adversaries.
Here, we found better accuracy with the standard practice of normalizing inputs.
Our layer outperformed the others in ResNet9 and WideResNet10-10 architectures; results were less decisive for KWLarge (see Appendix A).
For the WideResNet, we got 82 .
99% clean accuracy and 73 .
16% robust accuracy for (cid:15) = 36 / 255 .
For comparison, the state-of-the-art achieves 91 .
08% clean accuracy and 72 .
91% robust accuracy for a much larger (cid:15)
= 0 .
5 using a ResNet50 with adversarial training and additional unlabeled data (Augustin et al., 2020).
We visualize the tradeoffs for our residual networks in Figure 3, noting that they empirically have smaller local Lipschitz constants than KWLarge.
While our layer outperforms others for the default Lipschitz bound of 1, and consistently slightly better than BCOP, RKO can perform similarly well for larger bounds.
This provides some support for studies showing that hard constraints such as ours may not allow for the performance of softer constraints, such as RKO and penalty terms (Bansal et al., 2018; Vorontsov et al., 2017).
C ONCLUSION
In this paper, we presented a new parameterization of orthogonal convolutions using the Cayley transform, which is more expressive than previous work.
Unlike previous approaches to Lipschitzconstrained convolutions, ours allows deep networks to reap the full beneﬁts of orthogonality, such as gradient norm preservation.
We showed empirically that our method indeed maintains a high degree of orthogonality both before and after learning, and also scales better to some architectures than previous approaches.
Using our layer, we were able to improve upon the state-of-the-art in deterministic certiﬁable robustness against an (cid:96) 2 -norm-bounded adversary, and also demonstrated that it endows networks with considerable inherent robustness empirically.
While our layer offers beneﬁts theoretically, we observed that heuristics involving orthogonalizing reshaped kernels were also quite effective for empirical robustness.
Orthogonal convolutions may only show their true advantage in gradient norm preservation for deeper networks than we investigated.
In light of our experiments in scaling the Lipschitz bound, we hypothesize that not orthogonality, but insead the ability of layers such as ours to exert control over the Lipschitz constant, may be best for the robustness/accuracy tradeoff.
Future work may avoid expensive inverses using approximations or the exponential map, or compare various orthogonal and Lipschitz-constrained layers in the context of very deep networks.
A E MPIRICAL ROBUSTNESS ON KWL ARGE ARCHITECTURE
[Table]
increases the Lipschitz
[Table]
[Figure]
For KWLarge, our results on empirical robustness were mixed: while our Cayley layer outperforms
BCOP in robust accuracy, the RKO methods are overall more robust by around 2% , for only a marginal decrease in clean accuracy.
We note the lower empirical local Lipschitzness of RKO methods, which may explain their higher robustness: Figure 4 shows that the best choice of Lipschitz upper-bound for Cayley and BCOP layers may be less than 1 for this architecture.
[Figure]
B E FFICIENT NON - SQUARE C AYLEY TRANSFORM
For c in ≤ c out , this can be done efﬁciently by ﬁnding the inverse in terms of the Schur complement of the bottom-right block of the padded matrix.
For example, if W ∈ C c out × c in , we can split W into square and rectangular components T ∈ C c in × c in , P ∈ C c out − c in × c in respectively, pad with zeros, ﬁnd the skew-Hermitian part, take the Cayley transform, and project:
[Equation]
which has an efﬁciently-computable expression in terms of the Schur complement as
Q =
[Equation]
The case where c out < c in follows similarly, but results in a contraction.
The result is a “more general” version of the Cayley transform that can be applied to any matrix.
C E MPIRICAL RUNTIMES
KWLarge Empirical Runtimes ↓ Layer
Width Mult.
→
Cayley
Test Acc.
Certiﬁed 59.
Avg.
sec/Epoch 17.
BCOP
Test Acc.
Certiﬁed 58.
Avg.
sec/Epoch 22.
RKO
Test Acc.
Certiﬁed 57.
Avg.
sec/Epoch 18.
[Table]
[Figure]
(see (f) ).
For very large spatial size (image width and height) (see (d) ), or the combination of relatively large spatial size and many channels (see (b) ),
BCOP (Li et al., 2019) tends to be more efﬁcient.
Since convolutional layers in neural networks tend to decrease the spatial dimensionality while increasing the number of channels, and also often have unequal numbers of input and output channels, our Cayley layer is often more efﬁcient in practice.
In all cases, orthogonal convolutional layers are signiﬁcantly slower than plain convolutional layers.
Each runtime was recorded using the autograd proﬁler in PyTorch (Paszke et al., 2019) by summing the CUDA execution times.
The batch size was ﬁxed at 128 for all graphs, and each data point was averaged over 32 iterations.
We used a Nvidia Quadro RTX 8000.