S CALING - UP D IVERSE O RTHOGONAL C ONVOLUTIONAL N ETWORKS BY A P ARAUNITARY F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Enforcing orthogonality in convolutional neural networks is an antidote for gradient vanishing/exploding problems, sensitivity to perturbation and bounding generalization errors.
Many previous approaches for orthogonal convolutions enforce orthogonality on its ﬂattened kernel, which, however, do not lead to the orthogonality of the operation.
Some recent approaches consider orthogonality for standard convolutional layers and propose speciﬁc classes of their realizations.
In this work, we propose a theoretical framework that establishes the equivalence between diverse orthogonal convolutional layers in the spatial domain and the paraunitary systems in the spectral domain.
Since 1D paraunitary systems admit a complete factorization, we can parameterize any separable orthogonal convolution as a composition of spatial ﬁlters.
As a result, our framework endows high expressive power to various convolutional layers while maintaining their exact orthogonality.
Furthermore, our layers are memory and computationally efﬁcient for deep networks compared to previous designs.
Our versatile framework, for the ﬁrst time, enables the study of architectural designs for deep orthogonal networks, such as choices of skip connection, initialization, stride, and dilation.
Consequently, we scale up orthogonal networks to deep architectures, including ResNet and ShufﬂeNet, substantially outperforming their shallower counterparts.
Finally, we show how to construct residual ﬂows, a ﬂow-based generative model that requires strict Lipschitzness, using our orthogonal networks.
I NTRODUCTION
Convolutional neural networks (CNNs), whose deployment has witnessed extensive empirical success, still exhibit a range of limitations that are not thoroughly studied.
Firstly, deep convolutional networks are in general difﬁcult to learn, and their high performance heavily relies on techniques that are not fully understood, such as skip-connections (He et al., 2016a), batch normalization (Ioffe & Szegedy, 2015), specialized initialization (Glorot & Bengio, 2010).
Secondly, they are notoriously sensitive to imperceptible perturbations, including adversarial attacks (Goodfellow et al., 2014) or geometric transformations (Azulay & Weiss, 2019).
Finally, a precise characterization of their generalization property is still under active investigation (Neyshabur et al., 2018; Jia et al., 2019).
Orthogonal networks, which have a “ﬂat” spectrum with all singular values of the layer being 1 (each layer’s output norm (cid:107) y
(cid:107) equals its input norm (cid:107) x (cid:107) , ∀ x ), alleviate all problems above.
As shown in recent works, by enforcing orthogonality in neural nets, we obtain (1) easier optimization (Zhang et al., 2018a;
Qi et al., 2020): since each orthogonal layer preserves the gradient norm during backpropagation, an orthogonal network is free from gradient vanishing/ exploding problems; (2) robustness against adversarial perturbation (Anil et al., 2019; Li et al., 2019b; Trockman & Kolter, 2021): since each orthogonal layer is 1 -Lipschitz, an orthogonal network can not amplify any perturbation to the input to ﬂip the output prediction; (3) better generalizability as proved in (Jia et al., 2019): a network’s generalization error is positively related to the standard deviation of each linear layer’s singular values, thus encouraging orthogonality in the network lowers its generalization error.
Despite the beneﬁts, enforcing orthogonality in convolutional networks is challenging.
To avoid the strict constraint, orthogonal initialization (dynamical isometry) (Pennington et al., 2017; Xiao et al., 2018) and orthogonal regularization (Wang et al., 2019; Qi et al., 2020) are adopted to address the gradient vanishing/exploding problems.
However, these methods are not suitable for applications that require strict Lipschitzness, such as adversarial robustness (Anil et al., 2019) and residual ﬂows (Chen et al., 2019), as they do not enforce strict orthogonality (and Lipschitzness) during training.
Our goal is to enforce exact orthogonality in state-of-the-art convolutional networks without expensive computations.
We identify three main challenges in doing so.
Challenge I: Achieving exact orthogonality throughout the training process.
Prior works such as orthogonal regularization (Wang et al., 2019) and reshaped kernel orthogonality (Jia et al., 2017; Cisse et al., 2017), while enjoying algorithmic simplicity, do not meet the requirement of exact orthogonality.
Note that enforcing orthogonality during training is necessary as a post-training orthogonalization of the weights can ruin the performance.
Challenge II:
Avoiding expensive computations.
An efﬁcient algorithm is crucial for scalability to large networks.
Existing work based on projected gradient descent (Sedghi et al., 2019), however, requires expensive projection after each update.
For instance, the projection step in Sedghi et al.
(2019) computes an SVD and ﬂattens the spectrum to enforce orthogonality, which costs O (size(feature) · channels 3 ) for a convolutional layer.
Challenge III: Scaling-up to state-of-the-art deep convolutional networks.
There are many variants to the standard convolutional layer, including dilated, strided, group convolutions, which are essential for state-of-the-art deep convolutional networks.
However, none of the existing methods proposes mechanisms to orthogonalize these variants.
The lack of techniques, as a result, limits the broad applications of orthogonal convolutional layers to state-of-the-art deeper convolutional networks.
We resolve challenges I, II, and III by proposing complete parameterizations for orthogonal 1Dconvolutions and separable orthogonal 2D-convolutions .
First, using the convolution theorem (Oppenheim et al., 1996) (convolution in the spatial domain is equivalent to multiplication in the spectral domain), we reduce the problem of designing orthogonal convolutions to constructing unitary matrices for all frequencies, i.e., a paraunitary system (Vaidyanathan, 1993).
Therefore, we obtain a parameterization for all separable orthogonal 2D-convolutions, guaranteeing exact orthogonality and high expressive power.
Note that there is no previous approach that achieves exact orthogonality (up to machine precision).
For the ﬁrst time, our versatile framework enables the study in the designs of deep orthogonal networks, such as skip connection, initialization, stride, and dilation.
Consequently, we scale orthogonal networks to deep architectures, including ResNet and ShufﬂeNet, substantially outperforming their shallower counterparts.
Our proposed method serves as gaining insight into the trade-off between orthogonality and expressiveness.
We observe that exact orthogonality is essential in these deeper architectures with more than 10 layers (detailed discussion of exact orthogonality in
Appendix E.3). Finally, we show how to deploy our orthogonal networks in Residual Flow (Chen et al., 2019), a ﬂow-based generative model that requires strict Lipschitzness (Appendix F).
Summary of Contributions:
We establish the equivalence between orthogonal convolutions in the spatial domain and paraunitary systems in the spectral domain.
Consequently, we can interpret existing approaches for orthogonal convolutions as implicit designs of paraunitary systems.
Based on a complete factorization of 1D paraunitary systems, we propose the ﬁrst exact and complete design for orthogonal 1D-convolutions as well as separable orthogonal 2D-convolutions, ensuring exact orthogonality and high expressive power.
None of the existing works achieve a complete parameterization of all orthogonal 2D-convolutions.
We prove that orthogonality for various convolutional layers (strided, dilated, group) are also entirely characterized by paraunitary systems.
Consequently, our design easily extends to these variants, ensuring both completeness and exactness of the orthogonal convolutions.
We study the design considerations for orthogonal networks (choices of skip connection, initialization, depth, width, and kernel size), and show that orthogonal networks can scale to deep architectures including ResNet, WideResNet, and ShufﬂeNet.
O RTHOGONAL C ONVOLUTIONS VIA P ARAUNITARY S YSTEMS
Designing orthogonal convolutional layer { h t,s : y t
= h t,s ∗ x s } T,St =1 ,s =1 ( s, t index input/output channels) in the spatial domain is challenging.
When a convolution layer is written as a matrix-vector product, where the matrix is block-circulant and its ( t, s ) th takes a circulant form Cir ( h t,s ) as:
Cir ( h t,s )
=
  h t [Equation] s [1] h [Equation]
s [ N ] · [Equation] h [Equation]
h t
[Equation]
[Equation]
] h
[Equation]
N ] ·
[Equation]
h
[Equation]
[ N ]
[Equation]
[Equation] ]  [Equation] Therefore, the layer is orthogonal if the block-circulant matrix [ Cir ( h t,s )] T,St =1 ,s =1 is orthogonal.
However, it is not obvious how to enforce orthogonality in a block-circulant matrix.
2. A CHIEVING O RTHOGONAL C ONVOLUTIONS BY P ARAUNITARY S YSTEMS
We propose a novel design of orthogonal convolutions from a spectral domain perspective, motivated by the convolution theorem (Theorem 2.1).
For simplicity, we group the entries at the same locations into a vector/matrix, e.g., we denote { x s [ n ] } Ss =1 as x [ n ] ∈ R S and { h t,s [ n ] } T,St =1 ,s =1 as h [ n ] ∈ R T × S .
Theorem 2.1 (Convolution theorem (Oppenheim et al., 1996)) .
For a standard convolution layer h : y
[ i ] = (cid:80) n h [ n ] x [ i − n ] , the convolution in the spatial domain is equivalent to a matrixvector multiplication in the spectral domain, i.e., Y ( z )
=
H ( z ) X
( z ) , ∀ z ∈ C .
Here, X ( z ) =
(cid:80) N − 1 n =0 x
[ n ] z − n , Y ( z ) = (cid:80) N − 1 n =
y [ n ] z − n , H ( z ) =
(cid:80) Ln = − L h [ n ] z − n denote the z -transforms of input, output, kernel respectively, where N is the length of x , y and [ − L, L ] is the span of the ﬁlter h .
The convolution theorem states that a standard convolution layer is a matrix-vector multiplication in the spectral domain.
As long as the transfer matrix H ( z ) is unitary at z = e j ω for all frequencies
∀ ω ∈ R
( j is the imaginary unit), the convolutional layer h is orthogonal.
Therefore, as a major novelty of our paper, we design orthogonal convolutions through designing unitary transfer matrix H ( e j ω ) at all frequencies ω ∈ R , which is known as a paraunitary systems (Vaidyanathan, 1993; Strang & Nguyen, 1996).
We prove in Theorem B.5 that a convolutional layer is orthogonal in the spatial domain if and only if it is paraunitary in the spectral domain.
Beneﬁts through paraunitary systems.
The spectral representation simpliﬁes the designs of orthogonal convolutions and avoids the analysis of block-circulant structure.
(2) Since paraunitary systems are necessary and sufﬁcient for orthogonal convolutions, it is impossible to ﬁnd an orthogonal convolution whose transfer matrix is not paraunitary.
(3) There exists a complete factorization of paraunitary systems: any paraunitary H ( z ) can be realized through multiplications in the spectral domain, as will show in Equation (2.2a).
(4) Since multiplications in spectral domain correspond to convolutions in spatial domain, any orthogonal convolution can be realized as cascaded convolutions of multiple sub-layers, each parameterized by an orthogonal matrix.
(5) There are mature methods that represent orthogonal matrices via unconstrained parameters.
Consequently, we can learn orthogonal convolutions using standard optimizers on a model parameterized via our design.
Interpretation of existing methods.
Since paraunitary system is a necessary and sufﬁcient condition for orthogonal convolution, all existing approaches, including singular value clipping and masking (SVCM) (Sedghi et al., 2019), block convolution orthogonal parameterization (BCOP) (Li et al., 2019b), Cayley Convolution (CayleyConv) (Trockman & Kolter, 2021), skew orthogonal convolution (SOC) design paraunitary systems implicitly.
We discuss these interpretations in Appendix C.3.3, and show that our SC-Fac has the lowest computational complexity among all approaches.
2. R EALIZING P ARAUNITARY S YSTEMS VIA R E - PARAMETERIZATION
After reducing the problem of orthogonal convolutions to paraunitary systems, we are left with the question of how to realize paraunitary systems.
We use a complete factorization form of paraunitary systems to realize any paraunitary systems.
According to Theorem C.7, any paraunitary system H ( z ) can be written as the form in Equation (2.2a) and any H ( z ) in this form is a paraunitary system:
[Equation]
where V
[Equation]
Here Q is an orthogonal matrix and each U ( (cid:96) ) is a column-orthogonal matrix whose number of columns is sampled uniformly from { 1 , · · · , T } .
As multiplications in the spectral domain are equivalent to convolutions in the spatial domain, this complete spectral factorization of paraunitary systems in Equation (2.2a) allows us to parameterize any orthogonal convolution in the spatial domain as cascaded convolutions of V ( z ; U ( (cid:96) ) )
’s spatial counterparts and the orthogonal matrix Q .
Model design in the spatial domain.
We obtain a complete design of orthogonal 1D-convolutions: using learnable (column)-orthogonal matrices ( { U ( (cid:96) ) } − 1 (cid:96) =
− L , Q , { U ( (cid:96) ) } L(cid:96)
=1 ) , we parameterize a size ( L + L + 1) convolution as cascaded convolutions of the following ﬁlters in the spatial domain
[Equation]
[Table]
(cid:96)
[Table]
Figure 1 provides a visualization of our proposed design of orthogonal convolution layers; each block denotes a convolution and the form of the ﬁlter is displayed in each of the block.
In practice, we compose all ( L + L + 1) ﬁlters into one for orthogonal convolution, which not only increases the computational parallelism but also avoids storing intermediate outputs between ﬁlters.
[Equation]
[Figure]
Using this 1D-convolution, we obtain a complete design for separable orthogonal 2D-convolutions , which can be represented as a convolution of two orthogonal 1D-convolutional layers.
With separability, we obtain a design of orthogonal 2D-convolutions by composing two complete designs of orthogonal 1D-convolutions.
As a result, we can parameterize separable orthogonal 2D-convolution with ﬁlter size ( L 1 + L 1 +1) ×
( L 2 + L 2 +1) as a convolution of two orthogonal 1D-convolutions with learnable (column-)orthogonal matrices ( { U ( (cid:96) ) 1 } − 1 (cid:96) = − L 1 , Q 1 , { U ( (cid:96) ) 1 } L 1 (cid:96) =1 ) and ( { U ( (cid:96) ) 2 } − 1 (cid:96) =
− L 2 , Q 2 , { U ( (cid:96) ) 2 } L 2 (cid:96) =1 ) .
With a complete factorization of paraunitary systems (1D and separable 2D), we reduce the problem of designing orthogonal convolutions to the one for orthogonal matrices.
Parameterization for orthogonal matrices.
In Appendix C.4, we perform a comparative study on different parameterizations of orthogonal matrices, including the Björck orthogonalization (Anil et al., 2019; Li et al., 2019b), the Cayley transform (Helfrich et al., 2018; Maduranga et al., 2019), and the exponential map (Lezcano-Casado & Martínez-Rubio, 2019).
In our implementation, we adopt a modiﬁed exponential map due to its efﬁciency, exactness, and completeness.
The exponential map is a surjective mapping from a skew-symmetry matrix A to a special orthogonal matrix U (i.e., det( U ) = 1 ) with U = exp
( A ) =
I + A + A 2 / 2 + · · · , where the inﬁnite series is computed up to machine-precision (Higham, 2009).
To parameterize all orthogonal matrices, we introduce an additional orthogonal matrix Q in U =
Q exp( A ) , where Q is (randomly) generated at initialization and ﬁxed during training (Lezcano Casado, 2019) .
Now we have an end-to-end pipeline for orthogonal convolutions as shown in Figure 2 (See Algorithm 1 in Appendix E for a pseudo implementation).
Since our method relies on separability and complete factorization for 1D paraunitary systems, we call it Separable Complete Factorization (SC-Fac) .
[Figure]
[Equation]
SC-Fac: A pipeline for designing orthogonal convolutional layer.
(1) An orthogonal convolution h [ n ] is equivalent a paraunitary system H ( z ) in the spectral domain (Theorem 2.1).
The paraunitary system H ( z ) is multiplications of factors characterized by (column-)orthogonal matrices ( { U ( (cid:96) ) } − 1 (cid:96) =
− L , Q , { U ( (cid:96) ) } L(cid:96)
=1 ) (Equation (2.2a), Theorem B.5).
(3) These orthogonal matrices are parameterized by skew-symmetric matrices using exponential map.
U NIFYING O RTHOGONAL C ONVOLUTIONS V ARIANTS
Various convolutional layers (strided, dilated, and group convolution) are widely used in neural networks.
However, it is not apparent how to enforce their orthogonality, as the convolution theorem (Theorem 2.1) only holds for standard convolutions.
Previous approaches only deal with standard convolutions (Sedghi et al., 2019; Li et al., 2019b; Trockman & Kolter, 2021), thus orthogonality for state-of-the-art architectures has never been studied before.
We address this limitation by modifying convolution theorem for each variant of convolution layer, which allows us to design these variants using paraunitary systems.
Theorem 3.1 (Convolution and paraunitary theorems for various convolutions) .
Strided, dilated, and group convolutions can be uniﬁed in the spectral domain as Y ( z ) =
H ( z ) X
( z ) , where Y ( z ) ,
H ( z ) , X ( z ) are modiﬁed Z-transforms of y , h , x . We instantiate H ( z ) for strided convolutions in Proposition C.4, dilated convolution in Proposition C.5, and group convolution in Proposition C.6.
Furthermore, a convolution is orthogonal if and only if H ( z ) is paraunitary.
0 1 2 3 4…… 0 1 2 3 4 5 6 7 8 9x [ n ] x ↑ 2 [ n ] n n (a) Up-sample 0 1 2 3 40 1 2x [ n ] x 0 ∣ 2 [ n ] n n 6…… 0 1 2x 1 ∣ 2 [ n ] n … … (b) Down-sample
In Table 1, we formulate strided, dilated, and group convolutions in the spatial domain, interpreting them as up-sampled or down-sampled variants of a standard convolution.
Now, we introduce the concept of up-sampling and down-sampling precisely below.
Given a sequence x , we introduce its up-sampled sequence x ↑ R with sampling rate R as x ↑ R [ n ] (cid:44) x
[ n/R ] for n ≡ 0 (mod R ) .
On the other hand, its ( r , R )-polyphase component x r | R indicates the r -th down-sampled sequence with sampling rate R , deﬁned as x r | R [ n ] (cid:44) x
[ nR + r ] .
We illustrated an example of x ↑ R and x r | R in Figure 3 when sampling rate R = 2 .
The Z-transforms of x ↑ R , x r | R are denoted as X ↑ R ( z ) , X r | R ( z ) respectively.
Their relations to X ( z ) are studied in Appendix C.1.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
z
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Now we are ready to interpret convolution variants.
(1) Strided convolution is used to adjust the feature resolution: a strided convolution ( ↓ R -strided) decreases the resolution by down-sampling after a standard convolution [Table] while a transposed strided convolutional layer ( ↑ R -strided) increases the resolution by up-sampling before a standard convolution.
(2) Dilated convolution increases the receptive ﬁeld of a convolution without extra parameters: an R -dilated convolution up-samples its ﬁlters before convolution with the input.
(3) Group convolution reduces the parameters and computations, thus widely used by efﬁcient architectures: a G -group convolution divides the input/output channels into G groups and restricts the connections within each group.
In Appendix C.2, we prove that a convolution is orthogonal if and only if its modiﬁed Z-transform H ( z ) is paraunitary.
L EARNING D EEP O RTHOGONAL N ETWORKS WITH L IPSCHITZ B OUNDS
In this section, we switch our focus from layer design to network design.
In particular, we aim to study how to scale-up deep orthogonal networks with Lipschitz bounds.
Lipschitz networks (Anil et al., 2019; Li et al., 2019b; Trockman & Kolter, 2021), whose Lipschitz bounds are imposed by their architectures, are proposed as competitive candidates to guarantee robustness in deep learning.
A Lipschitz network consists of orthogonal layers and GroupSort activations (Anil et al., 2019) — both are 1 -Lipschitz and gradient norm preserving. See Appendix D for more discussions on the properties of GroupSort and Lipschitz networks.
Given a Lipschitz constant L , a Lipschitz network f can compute a certiﬁed radius for each input from its output margin.
Formally, denote the output margin of an input x with label c as
[Equation]
(4.1) i.e., the difference between the correct logit and the second largest logit.
Then the output is robust to perturbation such that f ( x + (cid:15)
[Equation]
/ √ 2 L .
Despite the beneﬁt, existing architectures for Lipschitz networks remain simple and shallow, and a
Lipschitz network is typically an interleaving cascade of orthogonal layers and GroupSort activations (Li et al., 2019b).
More advanced architectures, such as ResNet and ShufﬂeNet, are still out of reach.
While orthogonal layers supposedly substitute the role of batch normalization (Pennington et al., 2017; Xiao et al., 2018; Qi et al., 2020), other critical factors, including skip-connections (He et al., 2016a;b) and proper initialization (Glorot & Bengio, 2010) are lacking.
In this section, we explore skip-connections and initialization methods toward addressing this problem.
Skip-connections.
Two general types of skip-connections are widely used in deep networks, one based on addition and another on concatenation .
The addition-based connection is proposed in
ResNet (He et al., 2016a), and adopted in SE-Net (Hu et al., 2018) and EffcientNet (Tan & Le, 2019).
The concatenation-based connection is proposed in ﬂow-based generative models (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018), and adopted in DenseNet (Huang et al., 2017) and ShufﬂeNet (Zhang et al., 2018b; Ma et al., 2018).
In what follows, we propose Lipschitz skip-connections with these two mechanisms, illustrated in Figure 6 (in Appendix D).
Proposition 4.1 (Lipschitzness of residual blocks) .
Suppose f 1 , f 2 are L -Lipschitz (for residual/shortcut branches) and α ∈ [0 , 1] is a learnable scalar, then an additive residual block f : f ( x ) (cid:44) αf 1 ( x )+(1 − α )
f 2 ( x ) is L -Lipschitz.
Alternatively, suppose g 1 , g 2 are L -Lipschitz and
P is a channel permutation, then a concatenative residual block g : g ( x ) (cid:44) P (cid:2) g 1 ( x 1 ); g 2 ( x 2 ) (cid:3) is LLipschitz, where [ · ; · ] denotes channel concatenation, and x is split into x 1 and x 2 , i.e., x = [ x 1 , x 2 ] .
Initialization.
Proper initialization is crucial in training deep networks (Glorot & Bengio, 2010;
He et al., 2016a).
Various methods are proposed to initialize orthogonal matrices, including the identical/permutation and torus initialization in the context of orthogonal RNNs (Henaff et al., 2016;
Helfrich et al., 2018; Lezcano-Casado & Martínez-Rubio, 2019).
However, initialization of orthogonal convolutions was not systematically studied, and all previous approaches inherit the initialization from the underlying parameterization (Li et al., 2019b; Trockman & Kolter, 2021).
In
Proposition D.1, we study the condition when a paraunitary system (represented in Equation (2.2a)) reduces to an orthogonal matrix.
This reduction allows us to apply the initialization methods for orthogonal matrices (e.g., uniform, torus) to orthogonal convolutions.
In the experiments, we will evaluate the impact of different choices of skip-connections and initialization methods to the performance of deep Lipschitz networks.
R ELATED W ORK
Dynamical isometry (Pennington et al., 2017; 2018; Chen et al., 2018; Pennington et al., 2018) aims to address the gradient vanishing/exploding problems in deep vanilla networks with orthogonal initialization .
These works focus on understanding the interplay between initialization and various nonlinear activations.
However, these approaches do not guarantee Lipschitzness after training, thus are unsuitable for applications that require strict characterization of Lipschitz constants, such as adversarial robustness (Anil et al., 2019) and residual ﬂows (Behrmann et al., 2019).
Learning orthogonality has three typical approaches: regularization , parameterization (representing the feasible set with unconstrained parameters), and projected gradient descent (PGD) /
Riemannian gradient descent (RGD). While regularization is approximate, the latter two learn exact orthogonality.
(1) For orthogonal matrices , various regularizations are proposed in Xie et al.
Bansal et al. Alternatively, numerous parameterizations exist, including Householder reﬂections (Mhammedi et al., 2017), Given rotations (Dorobantu et al., 2016), Cayley transform (Helfrich et al., 2018), matrix exponential (Lezcano-Casado & Martínez-Rubio, 2019), and algorithmic unrolling (Anil et al., 2019; Huang et al., 2020).
Lastly, Jia et al.
(2017) propose PGD via singular value clipping , and Vorontsov et al.
; Li et al.
(2019a) consider RGD.
(2) For orthogonal convolutions , some existing works learn orthogonality for the ﬂattened matrix (Jia et al., 2017; Cisse et al., 2017; Bansal et al., 2018) or each output channel (Liu et al., 2021). However, these methods do not lead to orthogonality (norm preserving) of the operation.
Sedghi et al. (2019) propose to use PGD via singular value clipping and masking , which is expensive and can result in inexact orthogonality.
Recent works adopt parameterizations, using block convolutions (Li et al., 2019b), Cayley transform (Trockman & Kolter, 2021), or convolution exponential (Singla & Feizi, 2021).
Note that network deconvolution (Ye et al., 2020) aims to whiten the activations (i.e., orthogonalize their distribution), but the added whitening operations do not necessarily preserve orthogonality.
Paraunitary systems are extensively studied in ﬁlter banks and wavelets (Vaidyanathan, 1993; Strang & Nguyen, 1996; Lin & Vaidyanathan, 1996).
Classic theory shows that 1D-paraunitary systems are fully characterized by a spectral factorization (see Chapter 14 of Vaidyanathan (1993)), but not all MD-paraunitary systems admit a factorized form (see Chapter 8 of Lin & Vaidyanathan (1996)).
While the complete characterization of MD-paraunitary systems is known in theory (which incurs a difﬁcult problem of solving a system of nonlinear equations) (Venkataraman & Levy, 1995; Zhou, 2005), most practical constructions use separable paraunitary systems (Lin & Vaidyanathan, 1996) and special classes of non-separable paraunitary systems (Hurley & Hurley, 2012).
The equivalence between orthogonal convolutions and paraunitary systems thus opens the opportunities to apply these classic theories in designing orthogonal convolutions.
E XPERIMENTS
In the experiments, we achieve the following goals.
(1) We demonstrate in Section 6.1 that our separable complete factorization (SC-Fac) achieves precise orthogonality (up to machine-precision), resulting in more accurate orthogonal designs than previous ones (Sedghi et al., 2019; Li et al., 2019b;
Trockman & Kolter, 2021).
Despite the differences in preciseness, we show in Section 6.2 that different realizations of paraunitary systems only have a minor impact on the adversarial robustness of Lipschitz networks.
Due to the versatility of our convolutional layers and architectures, in
Section 6.3, we explore the best strategy to scale Lipschitz networks to wider/deeper architectures.
(4) In Appendix F, we further demonstrate in a successful application of orthogonal convolutions in residual ﬂows (Chen et al., 2019).
Training details are provided in Appendix E.1.
6. E XACT O RTHOGONALITY
We evaluate the orthogonality of our SC-Fac layer verse previous approaches, including CayleyConv (Trockman & Kolter, 2021), BCOP (Li et al., 2019b), SVCM (Sedghi et al., 2019), RKO (Cisse et al., 2017), OSSN (Miyato et al., 2018).
Our experiments are based on a convolutional layer withinput channels and 16 × 16 input size.
We orthogonalize the layer using each approach, and evaluate it with Gaussian inputs.
For our SC-Fac layer, We initialize all orthogonal matrices uniformly, while we use built-in initialization for others.
We evaluate the difference between 1 and the of the norm to the input norm — a layer is exactly orthogonal if the number is close to 0 .
[Table]
(1) Standard convolution.
We show in Table 2 (Left) that our SC-Fac is orders of magnitude more precise than all other approaches.
The SC-Fac layer is in fact exactly orthogonal up to machine epsilon, which is 2 − 24 ≈ 5 .
96 × 10 − 8 for 32 -bits ﬂoats.
While RKO and OSSN are known not to be orthogonal, we surprisingly ﬁnd that SVCM is far from orthogonal due to its masking step.
(2) Convolutions variants.
In Section 3, we show that various orthogonal convolutions can be constructed using paraunitary systems.
We verify our theory in Table 2 (Right): our SC-Fac layers are exactly orthogonal (up to machine precision) for various convolutions.
6. A DVERSARIAL R OBUSTNESS
In this subsection, we evaluate the adversarial robustness of Lipschitz networks.
Following the setup in (Trockman & Kolter, 2021), we adopt KW-Large, ResNet9, WideResNet10-10 as the backbone architectures, and evaluate their robust accuracy on CIFAR-10 with different designs of orthogonal convolutions.
We extensively perform a hyper-parameter search and choose the best hyper-parameters for each approach based on the robust accuracy.
The details of the hyper-parameter search is in
Appendix E. We run each model with 5 different seeds and report the best accuracy.
(1) Certiﬁed robustness.
Following Li et al.
(2019b), we use the raw images (without normalization) for network input to achieve the best certiﬁed accuracy.
As shown in Table 3 (Top), different realizations of paraunitary systems, SC-Fac, CayleyConv and BCOP achieve comparable performance — CayleyConv performs < 1% better in clean accuracy, but the difference in robust accuracy are negligible.
(2) Practical robustness.
Trockman & Kolter (2021) shows that the certiﬁed accuracy is too conservative, and it is possible to increase the practical robustness (against PGD attacks) with a standard input normalization.
Notice that the normalization increases the Lipschitz bound, thus lower the certiﬁed accuracy.
Our experiments in Table 3 (Bottom) are based on ResNet9, WideResNet1010 (Trockman & Kolter, 2021) and a deeper WideResNet22.
For the shallow architectures (ResNet9,
WideResNet10-10), our SC-Fac, CayleyConv, and BCOP again achieve comparable performance —
CayleyConv is slightly ahead in robust accuracy.
For the deeper architecture, our SC-Fac has a clear advantage in both clean and robustness accuracy , and the clean accuracy to only 5% lower than a traditional ResNet 32 trained with batch normalization.
Surprisingly, we ﬁnd that RKO also performs well in robust accuracy while not exactly orthogonal.
In summary, our experiments show that various paraunitary realizations provide different impacts on certiﬁed and practical robustness.
While exact orthogonality provides tight Lipschitz bound, there is a trade-off between the exact orthogonality and the practical robustness (especially with the shallow architectures).
[Table]
6. S CALING - UP D EEP O RTHOGONAL N ETWORKS WITH L IPSCHITZ B OUNDS
All previously proposed Lipschitz networks (Li et al., 2019b; Trockman & Kolter, 2021) consider only shallow architectures ( ≤ 10 layers).
In this part, we investigate various factors to scale Lipschitz networks to deeper architectures: skip-connection, depth-width, receptive ﬁeld [Table] and down-sampling.
(1) Skip-connections.
Conventional wisdom suggests that skip-connections mainly address the gradient vanishing/exploding problem; thus, they are not needed for orthogonal networks.
To understand their role, we perform an experiment that trains deep Lipschitz networks without skipconnection and with skip-connections based on addition/concatenation (see Section 4).
As shown in
Table 4 (left), the network with additive skip-connection substantially outperforms the other two, and the one without skip-connections performs the worst.
Therefore, we empirically show that (additive) skip-connection is crucial in deep Lipschitz networks.
(2) Depth and width.
Exact orthogonality is criticized for harming the expressive power of neural networks.
We show that the decrease of expressive power can be alleviated by increasing the network depth/width.
In Table 3 (Bottom) and Table 7 (Appendix E), we observe that deeper/wider architectures increase both the clean and robust accuracy.
(3) Initialization methods.
We try different initialization methods, including identical, permutation, uniform, and torus (Henaff et al., 2016;
Helfrich et al., 2018).
We ﬁnd that identical initialization works the best for deep Lipschitz networks ( > 10 layers), while all methods perform similarly in shallow networks as shown in Table 6 (Appendix E).
(4) Receptive ﬁeld and down-sampling.
Previous works (Li et al., 2019b; Trockman & Kolter, 2021) use larger kernel [Table] size and no stride for Lipschitz networks.
In Table 4 (Right)
[Table]
we perform a study on the effects of kernel/dilation size and down-sampling types for the orthogonal convolutions.
We ﬁnd that an average pooling as down-sampling consistently outperforms strided convolutions.
Furthermore, a larger kernel size helps to boost the performance.
(5) Run-time and memory comparison.
We ﬁnd that previously proposed orthogonal convolutions such as CayleyConv, BCOP, and RKO require more GPU memory and computation time than SC-Fac.
Therefore, we could not to scale them due to memory constraints (for 22 and 32 layers using Tesla V100 32G).
In order to scale up Lipschitz networks, economical implementation of orthogonal convolution is crucial.
As shown in Figure 4, for deep and wide architectures, our SC-Fac is the most computationally and memory efﬁcient method and the only method that scales to a width increase of 10 on WideResNet22.
Missing numbers in
Figure 4 and Table 7 (Appendix E) are due to the large memory requirement.
[Figure]
In summary, additive skip-connections are still essential for learning deep orthogonal networks.
Due to the orthogonal constraints, it is helpful to increase the depth/width of the network.
However, this signiﬁcantly increases the memory requirement; thus, a cheap implementation (like SC-Fac) is desirable.
Finally, we ﬁnd that a larger kernel size and down-sampling based on average pooling is helpful, unlike standard practices in deep networks [Figure]
C ONCLUSION
In this paper, we present a paraunitary framework for orthogonal convolutions.
Speciﬁcally, we establish the equivalence between orthogonal convolutions in the spatial domain and paraunitary systems in the spectral domain.
Therefore, any design for orthogonal convolutions is implicitly constructing paraunitary systems.
We further show that the orthogonality for variants of convolution (strided, dilated, and group convolutions) is also fully characterized by paraunitary systems.
In summary, paraunitary systems are all we need to ensure orthogonality for diverse types of convolutions.
Based on the complete factorization of 1D paraunitary systems, we develop the ﬁrst exact and complete design of separable orthogonal 2D-convolutions.
Our versatile design allows us to study the design principles for orthogonal convolutional networks.
Consequently, we scale orthogonal networks to deeper architectures, substantially outperforming their shallower counterparts. In our experiments, we observe that exact orthogonality plays a crucial role in learning deep Lipschitz networks. In the future, we plan to investigate other use cases that exact orthogonality is essential.
E THICS S TATEMENT
Our work lies in the foundational research on neural information processing.
Speciﬁcally, we establish the equivalence between orthogonal convolutions in neural networks and paraunitary systems in signal processing.
Furthermore, our presented orthogonal convolutional layers are plug-and-play modules that can replace various convolutional layers in neural networks.
Consequently, our modules are applicable in Lipschitz networks for adversarial robustness, recurrent networks for learning long-term dependency, or ﬂow-based networks for effortless reversibility.
The vulnerability of neural networks raises concerns about their deployment in security-sensitive scenarios, such as healthcare systems or self-driving cars.
In our experiment, we demonstrate a successful application of orthogonal convolutions in learning robust networks.
Furthermore, these networks achieve higher robust accuracy without additional techniques such as adversarial training or randomized smoothing.
Therefore, our research contributes to the robust learning of neural networks and potentially leads to their broader deployment.
Furthermore, we reduce the inference cost of our layer to be the same as a traditional convolution layer, signiﬁcantly lowering the expense for deployment compared with other methods for orthogonal networks.
However, our layers are memory and computationally more expensive during training than traditional layers.
The overhead to the already expensive training cost exacerbates the concerns on the efﬁcacy of learning neural networks.
Therefore, balancing between robustness and efﬁciency is an important research topic that requires more research in the future.
We develop more efﬁcient implementation than previous approaches, narrowing the gap between these two conﬂicting goals.
R EPRODUCIBILITY S TATEMENT
We provide detailed proofs to our theoretical claims in Appendices B to D. We describe experimental setups and training details of our models in Appendix E. We include our code for experiments in the supplementary material.
A P SEUDO C ODE FOR O UR SC-F AC A LGORITHM
We include the pseudo-code for separable complete factorization (Section 2) in Algorithm 1 and diverse orthogonal convolutions (Section 3) in Algorithm 2. The pseudo-code in Algorithm 1 consists of three parts: (1) First, we obtain orthogonal matrices from skew-symmetric matrices using matrix exponential. We use GeoTorch library (Lezcano Casado, 2019) for the function matrix _ exp in our implementation; (2) Subsequently, we construct two 1D paraunitary systems using these orthogonal matrices; (3) Lastly, we compose two 1D paraunitary systems to obtain one 2D paraunitary systems The pseudo-code in Algorithm 2 consists of two parts: (1) First, we reshape each paraunitary system into an orthogonal convolution depending on the stride; and (5) second, we concatenate the orthogonal kernels for different groups and return the output.
B O RTHOGONAL C ONVOLUTIONS VIA P ARAUNITARY S YSTEMS
In this section, we prove the convolution theorem and Parseval’s theorem for standard convolutional layers.
Subsequently, we prove the paraunitary theorem which establishes the equivalence between orthogonal convolutional layers and paraunitary systems.
B. S PECTRAL A NALYSIS OF S TANDARD C ONVOLUTION L AYERS
Standard convolutional layers are the default building blocks for convolutional neural networks.
One such layer consists of a ﬁlter bank with T × S ﬁlters h
= { h ts [ n ] , n ∈ Z } t ∈
[ T ] ,s ∈
[ S ] , where
S , T are the number of input and output channels respectively.
The layer maps an S -channel input x = { x s [ i ] ,
i ∈ Z } s ∈
[ S ] to a T -channel output y =
{ y t [ i ] , i ∈ Z } t ∈
[ T ] according to
[Equation]
where i indexes the output location to be computed, and n indexes the ﬁlter coefﬁcients.
Alternatively, we can rewrite Equation (B.1) in matrix-vector form as
[Equation]
where each h [ n ] ∈ R T × S is a matrix, and each x [ i − n ] ∈ R S or y
[ i ] ∈ R T is a vector.
Notice that in Equation (B.2), we group entries from all channels into a vector or matrix (e.g., from { x 0 [ n ] } s ∈
[ S ] to x [ n ] ), different from a common notation that groups entries from all locations into a vector or matrix (e.g., from { x s [ n ] , n ∈ Z } into x s ).
In the matrix-vector form, a standard convolutional layer computes a vector sequence y =
{ y [ i ] ∈ R T , i ∈
Z } with a convolution between a matrix sequence h
= { h [ n ] ∈ R T × S , n ∈ Z } and a vector sequence x = { x [ i ] ∈ R S , i ∈
Z } .
/* Iterate for vertical/horizontal dimensions */ for d = 1 to 2 do /* 1) Compute orthogonal matrices from
*/ /* Iterate for filter locations */ for (cid:96) =
{+− L to L do if (cid:96) = 
+}
then
Q d ← matrix _ exp ( A (0) d ) // use matrix _ exp () in GeoTorch (Lezcano Casado, 2019) else U ( (cid:96) ) d ← select ( matrix _ exp ( A ( (cid:96) ) d ) , cols =
N ( (cid:96) )
d ) // selects the first cols columns of the matrix end if end for /* 2)
Compose 1D paraunitary systems from orthogonal matrices */
H d ← Q d for (cid:96) = 1 to L do
H d ← conv1d ( H d , (cid:20)
U ( (cid:96) )
d U ( (cid:96) )
d (cid:62) , I − U
( (cid:96) )
d U ( (cid:96) )
d (cid:62) (cid:21) )
H d ← conv1d ( (cid:20)
I − U ( − (cid:96) )
d
U ( − (cid:96) )
d (cid:62) , U ( − (cid:96) )
d
U ( − (cid:96)
Algorithm
Input: Number of base channels C , kernel size K = R
(2 L + 1) , stride R , dilation D , number of groups G
Output: An orthogonal kernel W ∈ R T × S × K × K
Set K (cid:48) ← K/R , number of input channels S ← GC/R 2 and output channels T ← GC for g = 0 to G − 1 do /* 1) from systems */
Initialize skew-symmetric matrices {{ A ( (cid:96),g ) d } L(cid:96)
=
− L } 2 d =1 for the current g
H g ← Algorithm 1: SC-Fac ( C, K (cid:48) , {{ A ( (cid:96),g ) d } L(cid:96) =
− L } 2 d =1 )
H g ← reshape ( H g , ( C, C, K (cid:48) , K (cid:48) )
→ ( C/R 2 , C, K, K )) end for /* groups */
W ← concatenate ( { H g } G − 1 g =0 , dim = 0) return W (where the ﬁlter for input channel s and output channel t is W t,s, : , : ∈ R K × K )
Let us ﬁrst deﬁne the Z-transform and various types of Fourier transform in Deﬁnition B.1 before proving the convolution theorem (Theorem 2.1).
Deﬁnition B.1 (Z-transform and Fourier transforms) .
For a sequence (of scalars, vectors, or matrices) x = { x [ n ] , n ∈ Z } , its Z-transform X ( z ) is deﬁned as
[Equation]
where z ∈ C is a complex number such that the inﬁnite sum is convergent.
If z is restricted to the unit circle z = e j ω (i.e., | z | = 1 ), the z-transform X ( z ) reduces to a discrete-time Fourier transform (DTFT)
X ( e j ω ) .
If ω is further restricted to a ﬁnite set ω ∈ { 2 πk/N, k ∈
[ N ] } , the DTFT X ( e j ω ) reduces to an N -points discrete
Fourier transform (DFT) X
( e j 2 πk/N ) .
The celebrated convolution theorem states that the convolution in spatial domain (Equation (B.2)) is equivalent to a multiplication in the spectral domain, i.e., Y ( z )
=
H ( z ) X
( z ) , ∀ z ∈ C .
Proof of Theorem 2.1.
The proof follows directly from the deﬁnitions of standard convolution (Equation (B.2)) and Z-transform (Equation (B.3)).
[Equation]
where Equations (B.4) and (B.8) use the deﬁnition of Z-transform, Equation (B.5) uses the deﬁnition of convolution, and Equation (B.7) makes a change of variable k
= i − n .
Next, we introduce the concepts of inner product and Frobenius norm for sequences.
We then prove Parseval’s theorem , which allows us to compute the sequence norm in the spectral domain.
Deﬁnition B.2 (Inner product and norm for sequences) .
Given two sequences x = { x [ n ] , n ∈ Z } and y = { y [ n ] , n ∈ Z } with x [ n ] , y [ n ] having the same dimension for all n , the inner product of these two sequences is deﬁned as
[Equation]
n ]
[Equation]
n ] [Equation] [Equation] where (cid:104) x [ n ] , y [ n ] (cid:105) denotes the Frobenius inner product between x [ n ] and y [ n ] .
Subsequently, we can deﬁne the Frobenius norm of a sequence using inner product as
[Equation]
Theorem B.3 (Parsavel’s theorem) .
Given a sequence x = { x [ n ] , n ∈ Z } , its sequence norm (cid:107) x (cid:107) can be computed by X ( e j ω ) in the spectral domain as
[Equation]
) where (cid:107) X ( e j ω ) (cid:107)
= X ( e j ω ) † X
e j ω ) is an inner product between two complex arrays.
Proof of Theorem B.3.
The theorem follows from the deﬁnitions of convolution and discrete-time
Fourier transform (DTFT).
[Equation]
where Equation (B.14) is due to the bi-linearity of inner products, and Equation (B.15) makes uses of the fact that (cid:82) π − π e − j ωk dω = 0 for k (cid:54) =[-
-]
0 and (cid:82) π − π e − j
ωk dω = (cid:82) π − π dω = 2 π for k = 0 .
B. E QUIVALENCE BETWEEN O RTHOGONAL C ONVOLUTIONS AND P ARAUNITARY S YSTEMS
With the sequence norm introduced earlier, we formally deﬁne orthogonality for convolutional layers.
Deﬁnition B.4 (Orthogonal convolutional layer) .
A convolution layer is orthogonal if the input norm (cid:107) x (cid:107) is equal to the output norm (cid:107) y (cid:107) for arbitrary input x , that is
[Equation]
where (cid:107) x (cid:107) (or (cid:107) y (cid:107) ) is deﬁned as the squared root of (cid:80) n ∈ Z (cid:107) x
[ n ] (cid:107) 2 (or (cid:80) n ∈ Z (cid:107) y
[ n ] (cid:107) 2 ).
This deﬁnition of orthogonality not only applies to standard convolutions in Equation (B.2) but also variants of convolutions in Appendix C.3.
In this section, however, we ﬁrst establish the equivalence between orthogonality for standard convolutions and paraunitary systems .
Theorem B.5 (Paraunitary theorem) .
A standard convolutional layer (in Equation (B.2) ) is orthogonal (by Deﬁnition B.4)
if and only if its transfer matrix H ( z ) is paraunitary, i.e.,
[Equation]
[Equation]
In other words, the transfer matrix H ( e j ω ) is unitary for all frequencies ω ∈ R .
Proof of Theorem B.5.
We ﬁrst prove that a convolutional layer is orthogonal if its transfer matrix
H ( z ) is paraunitary (i.e., H ( e j ω ) is unitary for any frequency ω ∈ R ).
[Equation]
where Equations (B.19) and (B.22) are due to Parseval’s theorem (Theorem B.3), Equations (B.19) and (B.20) follows from the convolution theorem (Theorem 2.1), and Equation (B.21) utilizes that
H ( e jω ) is unitary for any frequency ω ∈ R (thus (cid:107) H ( e jω ) X ( e jω ) (cid:107) =
(cid:107) X
( e jω ) (cid:107) for any X ( e jω ) ).
The ‘only if’ part also holds in practice.
We here prove by contradiction using periodic inputs (e.g., ﬁnite inputs with circular padding).
Suppose there exists a frequency ω and H ( e j ω ) is not unitary.
Since H ( e j ω ) is continuous (due to h ∈ L 2 and dominated convergence theorem), there exist integers
N , k , such that ω ≈ 2 kπ/N and H ( e j 2 πk/N ) is also not unitary.
As a result, there exists a complex vector u such that v = H
( e j 2 πk/N ) u while (cid:107) v (cid:107) (cid:54) =
(cid:107) u (cid:107) .
Therefore, we can construct two periodic sequences x = { x [ n ] , n ∈ [ N ] } and y = { y [ n ] , n ∈ [ N ] } such that
[Equation]
Now the input norm (cid:107) x (cid:107) = (cid:113)(cid:80) n ∈ [ N ] (cid:107) x
[ n ] (cid:107) 2 = √ N (cid:107) u
(cid:107) is not equal to the output norm (cid:107) y (cid:107)
=
(cid:113)(cid:80) n ∈
[ N ] (cid:107) y
[ n ] (cid:107) 2 = √ N (cid:107) v (cid:107) , i.e., the layer is not orthogonal, which leads to a contradiction.
C A P ARAUNITARY F RAMEWORK FOR O RTHOGONAL C ONVOLUTIONS
[Figure] B. Section B.
[Figure]
[Figure] show that their designs reduce to constructing paraunitary systems. In Appendix C.3, we show that a paraunitary system can be constructed with different approaches: our approach and BCOP (Li et al., 2019b) represent the paraunitary using orthogonal matrices, while CayleyConv (Trockman & Kolter, 2021) and SOC (Singla & Feizi, 2021) directly parameterizes it using unconstrained parameters. In Appendix C.4, we investigate various parameterizations for orthogonal matrices, such as matrix exponential, Cayley transform, and Björck orthogonalization.
C. M ULTI - RESOLUTION A NALYSIS
Multi-resolution operations are essential in various convolutional layers, in particular strided and dilated convolutions.
In order to deﬁne and analyze these convolutions rigorously, we ﬁrst review the concepts of up-sampling , down-sampling , and polyphase components .
(1) Up-sampling.
Given a sequence (of scalars, vectors, matrices) x = { x [ n ] , n ∈ Z } , its upsampled sequence x ↑ R = { x ↑ R [ n ] , n ∈ Z } is deﬁned as
[Equation]
where R ∈ Z + is the up-sampling rate.
Accordingly, we denote the Z-transform of x ↑ R as
[Equation]
The following proposition shows that X ↑ R ( z ) is easily computed from X ( z ) .
Proposition C.1 (Z-transform of up-sampled sequence) .
Given a sequence x and its up-sampled sequence x ↑ R , their Z-transforms X ( z ) and X ↑ R ( z ) are related by
[Equation]
Proof of Proposition C.1.
The proof makes use of the deﬁnition of Z-transform (Equation (B.3)).
[Equation]
where Equation (C.5) makes a change of variables m = n/R since x ↑ R [ n ] = 0 , ∀ n (cid:54)
=
mR .
(2) Down-sampling and polyphase components.
Different from the up-sampled sequence, there exist multiple down-sampled sequences, depending on the phase of down-sampling.
These sequences are known as the polyphase components .
Speciﬁcally, given a sequence (of scalars, vectors, or matrices) x
= { x [ n ] , n ∈ Z } , its r th polyphase component x r | R = { x r | R [ n ] , n ∈ Z } is deﬁned as
[Equation]
where R ∈ Z + is the down-sampling rate.
We further denote the Z-transform of x r | R as
[Equation]
Note that r ∈ Z is an arbitrary integer, which does not necessarily take values from [ R ] .
In fact, we have x ( r + kR ) | R [ n ] = x r | R [ n + k ] and X r + kR
| R ( z ) =
z k X r | R ( z ) .
In Proposition C.2, we establish the relation between H ( z ) and { H r | R ( z ) } r ∈ [ R ] , i.e., to represent H ( z ) in terms of { H r | R ( z ) } r ∈
[ R ] .
Proposition C.2 (Polyphase decomposition) .
Given a sequence x and its polyphase components x r | R ’s, the Z-transform X ( z ) can be represented by { X r | R ( z ) }
r ∈
[ R ] as
[Equation]
Proof of Proposition C.2.
We start with X ( z ) , and try to decompose it into its polyphase components
X 0 | R ( z ) , · · · , X R − 1 | R ( z ) .
[Equation]
where Equation (C.11) makes a change of variables n = mR
+ r , and Equation (C.13) is the deﬁnition of polyphase components x r | R [ m ] =
x [ mR + r ] .
For simplicity, we stack R consecutive polyphase components into polyphase matrices as
[Equation]
The following proposition extends the Parseval’s theorem in Theorem B.3 and shows that the sequence norm (cid:107) x (cid:107) can also be computed in terms of the polyphase matrix
X
[ R ] ( z ) (or (cid:102) X
[ R ] ( z ) ).
Proposition C.3 (Parseval’s theorem for polyphase matrices) .
Given a sequence x , its sequence norm (cid:107) x (cid:107) can be computed by X
[ R ] ( e j ω ) (or (cid:102) X
[ R ] ( e j ω ) ) in the spectral domain as
[Equation]
Proof of Proposition C.3.
The proof follows the standard Parseval’s theorem in Theorem B.3.
We only prove the ﬁrst part of the proposition (using X
[ R ] ( e j ω ) ) as follows.
[Equation]
where Equation (C.17) follows the deﬁnition of sequence norm, Equation (C.18) changes variables as n = mR + r , Equation (C.19) is the deﬁnition of polyphase components, and Equation (C.20) applies Parseval’s theorem to x r | R ’s.
The second part (using (cid:102) X
[ R ] ( e j ω ) ) can be proved similarly.
C. U NIFYING V ARIOUS C ONVOLUTIONAL L AYERS IN THE S PECTRAL D OMAIN
In Appendix B, the convolution theorem states that a standard convolutional layer is a matrix-vector product Y ( z )
=
H ( z ) X
( z ) in the spectral domain, and the layer is orthogonal if and only if H ( z ) is paraunitary (Theorem B.5).
However, the canonical convolution theorem does not hold for variants of convolutions, thus enforcing a paraunitary H ( z ) may not lead to orthogonal convolution .
In this subsection, we address this limitation by showing that various convolutions can be uniformly written as Y ( z ) =
H ( z ) X
( z ) , where Y ( z ) , H ( z ) , X ( z ) are some spectral representations of y , h , x .
Subsequently, we prove that any of these layers is orthogonal if and only if its H ( z ) is paraunitary.
(1) Strided convolutional layers are widely used in neural networks to adjust the feature resolution: a strided convolution layer decreases the resolution by down-sampling after a standard convolution, while a transposed convolution increases the resolution by up-sampling before a standard convolution.
Formally, a strided convolutional layer with stride R (abbrev.
as ↓ R -strided convolution) computes its output following
[Equation]
In contrast, a transposed strided convolutional layer with stride R (abbrev.
as ↑ R -strided convolution) computes its output according to
[Equation]
Proposition C.4 (Orthogonality of strided convolutional layers) .
For a ↓ R -strided convolution, the spatial convolution in Equation (C.22) leads to the following spectral representation:
[Equation]
And for an ↑ R -strided convolution, the spatial convolution is represented in spectral domain as:
[Equation]
Furthermore, a ↓ R -strided convolution is orthogonal if and only if (cid:102) H
[ R ] ( z ) is paraunitary, and an ↑ R -strided convolution is orthogonal if and only if H [ R ] [ z ] is paraunitary.
Proof of Proposition C.4.
(1a) ↓ R -strided convolutions.
We ﬁrst prove the spectral representation of ↓ R -strided convolution in Equation (C.24).
[Equation]
where Equation (C.26) follows from the deﬁnitions of the ↓ R -strided convolution (Equation (C.22)) and the Z-transform (Equation (B.3)), Equation (C.27) makes a change of variables n = mR − r ,
Equation (C.29) further changes i (cid:48)
=
i − m , and Equation (C.30) is due to the deﬁnition of polyphase components (Equation (C.7)).
Now We rewrite the last equation concisely as
[Equation]
[Equation] which is the spectral representation of ↓ R -strided convolutions in Equation (C.24)
[Equation]
Now we prove the orthogonality condition for ↓ R -strided convolutions.
[Equation]
where Equations (C.32) and (C.35) are due to Parseval’s theorems (Theorem B.3 and Proposition C.3), Equation (C.33) follows from the spectral representation of the ↓ R -strided convolution (Equation (C.24)), and Equation (C.34) utilizes that the transfer matrix is unitary at each frequency.
The “only if” part can be proved by contradiction similar to Theorem B.5.
(1b) ↑ R -strided convolutions.
According to
Proposition C.1, the Z-transform of x ↑ R is X ( z R ) .
Therefore, an application of the convolution theorem (Theorem 2.1) on Equation (C.23) leads us to
[Equation]
Expanding Y ( z ) and H ( z ) using polyphase decomposition (Proposition C.2), we have
[Equation]
where Equation (C.39) is due to the uniqueness of Z-transform, and Equation (C.40) changes the variables from z R to z .
Again„we can rewrite the last equation in concisely as
[Equation]
which is the spectral representation of ↑ R -strided convolutions in Equation (C.25).
Lastly, we prove the orthogonality condition for ↑ R -strided convolutions.
[Equation]
where Equations (C.42) and (C.45) are due to Parseval’s theorems (Theorem B.3 and Proposition C.3), Equation (C.43) follows from the spectral representation of the ↑ R -strided convolution (Equation (C.25)), and Equation (C.44) uses the fact that the transfer matrix is unitary for each frequency.
The “only if” part can be proved by contradiction similar to Theorem B.5.
(2) Dilated convolutional layer is proposed to increase the receptive ﬁeld of a convolutional layer without extra parameters and computation.
The layer up-samples its ﬁlter bank before convolution with the input.
R -dilated convolutional layer) computes its output with the following equation:
[Equation]
Proposition C.5 (Orthogonality of dilated convolutional layer) .
For an R -dilated convolution, the spatial convolution in Equation (C.46) leads to a spectral representation as
[Equation]
Furthermore, an R -dilated convolutional layer is orthogonal if and only if H ( z R ) is paraunitary.
Proof of Proposition C.5.
According to Proposition C.1, the Z-transform of h ↑ R is H ( z R ) .
Therefore, the “if” part follows directly from the convolution theorem.
The “only if” part can be proved by constructing a counterexample similar to Theorem B.5.
Notice that H ( z R ) is paraunitary if and only if H ( e j ω ) is unitary for all frequency ω ∈ R , which is the same as X Hz being paraunitary.
In other words, any ﬁlter bank that is orthogonal for a standard convolution is also orthogonal for a dilated convolution and vice versa.
(3) Group convolutional layer is proposed to reduce the parameters and computations and used in many efﬁcient architectures, including MobileNet, ShufﬂeNet.
The layer divides both input/output channels into multiple groups and restricts the connections within each group.
Formally, a group convolutional layer with G groups (abbrev.
as G -group convolutions) is parameterized by G ﬁlter banks { h g } g ∈ [ G ] , each consists of ( T/G ) ×
( S/G ) ﬁlters.
The layer maps an S channels input x to a T channels output y according to
[Equation]
where blkdiag ( {·} ) computes a block diagonal matrix from a set of matrices.
Proposition C.6 (Orthogonality of group convolutional layer) .
For a G -group convolution, the spatial convolution in Equation (C.48) leads a spectral representation as , their z-transforms satisfy
[Equation]
Furthermore, a G -group convolutional layer is orthogonal if and only if the block diagonal matrix is paraunitary, i [Equation] e., each h g ( z ) is paraunitary.
Proof of Proposition C
[Equation] Due to the convolution theorem [Equation] it sufﬁces to prove that the Z-transform of a sequence of block diagonal matrices is also block diagonal in the spectral domain.
[Equation]
As a result [Equation] we can write the orthogonality condition as
[Equation]
The equation implies H g ( z ) † H g
( z ) =
I , ∀| z | = 1 , ∀ g ∈ [ G ] , i.e., each H g ( z ) is paraunitary.
C. R EALIZATIONS OF P ARAUNITARY S YSTEMS
In this subsection, we ﬁrst prove that all ﬁnite-length 1D-paraunitary systems can be represented in a factorized form.
Next, we show how we can construct MD-paraunitary systems using 1D systems.
Lastly, we study the relationship of existing approaches to paraunitary systems.
C.3. C OMPLETE F ACTORIZATION OF 1D-P ARAUNITARY S YSTEMS
The classic theorem for spectral factorization of paraunitary systems is traditionally developed for causal systems (Vaidyanathan, 1993; Kautsky & Turcajová, 1994).
Given a causal paraunitary system of length L (i.e., polynomial in z − 1 ), there always exists a factorization such that
[Equation]
where Q is an orthogonal matrix, U ( (cid:96) ) is a column-orthogonal matrix, and V ( z ; U ) is deﬁned as
[Equation]
In Theorem C.7, we extends this theorem from causal systems to ﬁnite-length (but non-causal) ones.
Theorem C.7 (Complete factorization for 1D-paraunitary systems) .
Suppose that a paraunitary system H ( z ) is ﬁnite-length, i.e., it can be written as (cid:80) n h [ n ] z − n for some sequence { h [ n ] , n ∈ [ − L , L ] } , then it can be factorized in the following form: [Equation]
[Equation]
where Q is an orthogonal matrix, U ( (cid:96) ) is a column-orthogonal matrix, and V ( z ; U ) is deﬁned in Equation (C.54) .
Consequently, the paraunitary system H ( z ) is parameterized by L + L +(column-)orthogonal matrices Q and U ( (cid:96) ) ’
s.
Proof for Theorem C.7.
Given a non-causal paraunitary system H ( z ) , we can always ﬁnd a causal counterpart ˆ H ( z ) such that H ( z ) =
z
L ˆ H ( z ) (This can be done by shifting the causal system backward by L steps, which is equivalent to multiplying z L in the spectral domain).
Since the causal system ˆ H ( z ) admits a factorization in Equation (C.55), we can write the non-causal system H ( z ) as
[Equation]
Therefore, it sufﬁces to show that for an orthogonal matrix Q and any column-orthogonal matrix ˆ U , we can always ﬁnd another column-orthogonal matrix U such that
[Equation]
If the equation above is true, we can set U ( (cid:96) )
=
ˆ U ( (cid:96) − 1 − L ) for (cid:96) < 0 and U ( (cid:96) )
=
ˆ U ( (cid:96) − L ) for (cid:96) > 0 , which will convert Equation (C.56) into Equation (C.55).
Now we start to prove Equation (C.57).
Note that any column-orthogonal ˆ U has a complement ¯ U such that [ ˆ U , ¯ U ] is orthogonal and I
=
ˆ U ˆ U (cid:62) + ¯ U ¯ U (cid:62) .
We then rewrite Equation (C.57) as
[Equation]
where in Equation (C.61) we set U
=
Q ¯ U .
This completes the proof.
C.3. M ULTI - DIMENSIONAL (MD) P ARAUNITARY S YSTEMS
If the data are multi-dimensional (MD), we will need MD-convolutional layers in neural networks.
Analogously, we can prove the equivalence between orthogonal MD-convolutions in the spatial domain and MD-paraunitary systems in the spectral domain, i.e.,
[Equation]
where D is the data dimension.
In this work, we adopt a parameterization based on separable systems.
Deﬁnition C.8 (Separable MD-paraunitary system) .
A MD-paraunitary system H ( z ) is separable if there exists D 1D-paraunitary systems H 1 ( z 1 ) , · · · , H D ( z D ) such that
[Equation]
Therefore, we can construct an MD-paraunitary system with D number of 1D-paraunitary systems, each of which is represented in Equation (C.55).
Notice that not all MD-paraunitary systems are separable, thus the parameterization in Equation (C.64) is not complete (see Section 5 for a discussion).
However, we can guarantee that our parameterization realizes all separable MD-paraunitary systems — each separable paraunitary system admits a factorization in Equation (C.64), where each 1D-system admits a factorization in Equation (C.55).
C.3. I NTERPRETATIONS OF P REVIOUS A PPROACHES
In Theorem B.5, we have shown that a paraunitary transfer matrix is both necessary and sufﬁcient for a convolution to be orthogonal.
Therefore, we can interpret all approaches for orthogonal convolutions as implicit constructions of paraunitary systems, including singular value clipping and masking (SVCM) (Sedghi et al., 2019), block convolution orthogonal parameterization (BCOP) (Li et al., 2019b), Cayley convolution (CayleyConv) (Trockman & Kolter, 2021), skew orthogonal convolution (SOC) (Singla & Feizi, 2021).
Furthermore, we prove how orthogonal regularization (Wang et al., 2019; Qi et al., 2020) encourages the transfer matrix to be unitary for all frequencies.
(1) Singular value clipping and masking (SVCM) (Sedghi et al., 2019) clips all singular values of H ( e j ω ) to ones for each frequency ω after gradient update.
Since the clipping step can arbitrarily enlarge the ﬁlter length, SVCM subsequently masks out the coefﬁcients outside the ﬁlter length.
However, the masking step breaks the orthogonality, as we have seen in the experiments (Section 6.1).
(2) Block convolution orthogonal parameterization (BCOP) (Li et al., 2019b) tries to generalize the spectral factorization of 1D-paraunitary systems in Equation (C.55) to 2D-paraunitary systems.
[Equation]
where V ( z 1 , z {+
+} ; U ( (cid:96) ) 1 , U ( (cid:96) ) 2 ) =
V ( z 1 ; U ( (cid:96) ) 1 ) V ( z 2 ; U ( (cid:96) ) 2 ) .
In other words, this approach makes each V -block, instead of the whole paraunitary system, separable.
This factorization in BCOP is incomplete for 2D-paraunitary system — unlike 1D-paraunitary systems, not every 2D-paraunitary system admits a factorized form (Lin & Vaidyanathan, 1996).
(2) Calyey convolution (CayleyConv) (Trockman & Kolter, 2021) aims to generalize the Cayley transform for orthogonal matrices in Equation (C.88) to 2D-paraunitary systems H ( z 1 , z 2 ) :
[Equation]
where A ( z 1 , z 2 ) is a skew-Hermitian matrix for | z 1 | = 1 , | z 2 | = 1 (i.e., A ( e j ω 1 , e j ω 2 ) † =
A ( e j ω 1 , e j ω 2 ) for any ω 1 , ω 2 ).
Since Cayley transform cannot parameterize a matrix with singular value − 1 for any frequency, the CayleyConv is not a complete parameterization.
(4) Skew orthogonal convolution (SOC) (Singla & Feizi, 2021) aims to generalize the matrix exponential for orthogonal matrices (Equation (C.90)) to convolution exponential for 2D-paraunitary systems H ( z 1 , z 2 ) :
[Equation]
where A ( z 1 , z 2 ) is skew-Hermitian matrix for any matrix for | z 1 | = 1 , | z 2 | = 1 (in other words,
A ( e j ω 1 , e j ω 2 ) † =
A ( e j ω 1 , e j ω 2 ) for any ω 1 , ω 2 ).
It is not resolved whether all 2D-paraunitary systems can be represented in terms of convolution exponential.
(5) Orthogonal regularization (Ortho-Reg) (Wang et al., 2019; Qi et al., 2020) is developed to encourage orthogonality in convolutional layers.
We show that such orthogonal regularization is equivalent to a unitary regularization of the paraunitary system with uniform weights on all frequencies.
[Equation]
where δ [0]
= I is an identity matrix, δ [ n ] = 0 is a zero matrix for n (cid:54)
= 0 .
We prove the equivalence more generally in Proposition C.9.
However, this approach cannot enforce exact orthogonality and in practice requires hyperparameter search for a proper regularizer coefﬁcient.
Proposition C.9 (Parseval’s theorem for ridge regularization) .
Given a sequence of matrices h
= { h [ n ]
[Equation]
n ∈ Z }
[Equation]
the following four expressions are equivalent:
[Equation]
Proof of Proposition C.9.
We ﬁrst prove the equivalence between Equations (C.69a) and (C.69c)
[Equation]
Equations (C.70) and (C.74) make use of (cid:107) A (cid:107) 2 = tr ( A † A ) , Equations (C.71) and (C.73) are due to the linearity of tr ( · ) , and Equation (C.72) utilizes tr ( AB ) =
tr ( BA ) .
Next, we prove the equivalence between Equations (C.69a) and (C.69b).
[Equation]
where Equation (C.75) follows from the deﬁnition of polyphase matrix in Equation (C.15).
Equation (C.76) uses a number of properties of Fourier transform: a Hermitian in the spectral domain is a transposed reﬂection in the spatial domain, a frequency-wise multiplication in the spectral domain is a convolution in the spatial domain, and an identical mapping in the spectral domain is an impulse sequence in the spatial domain.
Equation (C.77) follows from the deﬁnition of polyphase components in Equation (C.7), and Equation (C.79) makes a change of variables n =
Rm + r .
In summary, we show that the LHS (denoted D ( e j ω ) ) is a Fourier transform of the RHS (denoted as d [ i ] ):
[Equation]
Applying Parseval’s theorem (Theorem B.3) to the sequence d = { d [ i ] , i ∈
Z } , we have
[Equation]
which proves the equivalence between Equations (C.69a) and (C.69b).
With almost identical arguments, we can prove the equivalence between Equations (C.69c) and (C.69d), that is
[Equation]
which completes the proof
[Equation] [Table] In Table 5, we compare the computational complexities (forward pass) of orthogonal convolutions against normal convolution.
For simplicity, we assume the feature maps have size N × N , the convolution ﬁlters have size L × L , and the maximum of input/output channels is C .
We use K to denote the number of iterations for Björck’s algorithm in BCOP, or Taylor’s series order in SOC.
• For SC-Fac, BCOP, SVCM, or Ortho-Reg, the ﬁrst term O
( L 2 N 2 C 2 ) is the base cost of a normal convolution, and the second term is the overhead for reconstruction, projection, or regularization.
Note that the overhead in SC-Fac is comparable to Ortho-Reg, which is lower than SVCM or BCOP.
If C < N 2 , the overhead in SC-Fac is negligible compared to the base cost.
• For CayleyConv, the ﬁrst term O
( N 2 log( N ) C 2 ) is the cost for fast Fourier transform (FFT) and the second term O ( N 2 C 3 ) is the cost for matrix inversion for all frequencies.
The cost of SOC is exactly K times as a normal convolution.
The computational complexities of CayleyConv and SOC are signiﬁcantly higher than the one of normal convolution.
• For those approaches whose ﬁlters can be explicitly obtained (SC-Fac, BCOP, SVCM, and OrthoReg), the inference time of an orthogonal convolution is no different from a normal convolution, i
e., O ( L 2 N 2 C 2 ) .
On the other hand, for those approaches whose ﬁlters are implicitly deﬁned (CayleyConv and SOC), the inference time is the same as the forward pass in training.
C. C ONSTRAINED O PTIMIZATION OVER O RTHOGONAL M ATRICES
In Theorem C.7, we have shown how orthogonal matrices characterize paraunitary systems.
Our remaining goal, therefore, is to parameterize orthogonal matrices using unconstrained parameters.
In this part, we review popular parameterization methods: Householder reﬂections (Mhammedi et al., 2017; Mathiasen et al., 2020), Givens rotations (Dorobantu et al., 2016;
Jing et al., 2017), Björck orthogonalization (Anil et al., 2019), Cayley transform (Helfrich et al., 2018; Maduranga et al., 2019), and exponential map (Lezcano-Casado & Martínez-Rubio, 2019; Lezcano Casado, 2019).
(1) Householder reﬂections (Mhammedi et al., 2017; Mathiasen et al., 2020).
The parameterization represents an orthogonal matrix using a product of Householder matrices.
Given N ∈ N and n ∈ { 1 , · · · , N } , we deﬁne H ( n ) as a mapping from a vector v ∈ R n (a scalar v ∈ R for n = 1 ) to a block-diagonal matrix H ( n ) ( v ) ∈ R N × N : [Equation] v [Equation]
C
[Equation]
) where I n ∈ R n × n denotes an identity matrix.
For n ≥ 2 , H ( n ) ( v ) is the Householder matrix that represents the reﬂection with respect to the hyperplane orthogonal to concat ( 0 N − n , v ) ∈
R N and passing through the origin.
For n = 1 , the scalar v takes values {− 1 , +1 } , which makes
H (1) ( v ) either an identity matrix (for v = 1 ) or a Householder matrix (for v = − 1 ).
This method parameterizes an orthogonal matrix as a product of H ( n ) ’s:
[Equation]
where each v ( n ) ∈ R n is a learnable vector for n ≥ 2 , and v (1) is a ﬁxed constant that is generated at initialization and ﬁxed afterward.
Observing that Equation (C.84) is serial (unfriendly to parallel computing)
, Mathiasen et al.
(2020) proposes to increase its parallelism using WY transform .
(2) Givens rotations (Dorobantu et al., 2016; Jing et al., 2017).
The parameterization represents a special orthogonal matrix using a product of Given rotations matrices.
Given N ∈ N and i, j ∈ { 1 , · · · , N } with i (cid:54) =
j , we deﬁne G ( i,j ) as a mapping from an angle θ ∈ R to the corresponding rotation matrix G ( i,j
) ∈ R N × N :
G ( i,j ) ( [Equation] ) =   1 · · · · · · · · ·...
0 · · · cos θ · · · − sin θ · · ·... 0 · · · sin θ · · · cos θ · · ·...
0 · · · · · · · · ·  .
(C.85)
This method parameterizes a special orthogonal matrix U as a product of G ( i,j ) ’s:
[Equation]
(C.86) where θ ( i,j ) ’s are N ( N − 1) unconstrained parameters.
Since the determinant of each rotation matrix is +1 , their product U also has +1 determinant, thus is a special orthogonal matrix.
Again,
Equation (C.86) is highly serial, thus both Dorobantu et al.
and Jing et al.
(2017) propose to group N/ 2 Given rotations into a packed rotation to increase the parallelism.
(2) Björck orthogonalization (Anil et al., 2019).
The algorithm was ﬁrst introduced in Björck &
Bowie (1971) to compute the closest orthogonal matrix of a given matrix.
Given an initial matrix U 0 , this algorithm iteratively approaches its closest orthogonal matrix as:
[Equation]
where K is the iterative steps,
P k
= I − U (cid:62) k
U k , and p controls the trade-off between efﬁciency and accuracy at each step.
When the algorithm is used for parameterization, it maps an unconstrained matrix U 0 to an approximately orthogonal matrix U K in K steps.
Although Björck parameterization is complete (since any orthogonal matrix Q can be represented by U 0 = Q ), it is inexact due to the iterative approximation.
(3) Cayley transform (Helfrich et al., 2018; Maduranga et al., 2019).
The transform provides a bijective parameterization of orthogonal matrices without − 1 eigenvalue with skew-symmetric matrices (i.e., A (cid:62) = −
[Equation]
where the skew-symmetric matrix A is represented by its upper-triangle entries.
Since orthogonal matrices with − 1 eigenvalue are out of consideration, the parameterization is incomplete.
Helfrich et al.
; Maduranga et al.
(2019) overcome this difﬁculty by a scaled Cayley transform :
[Equation]
where D is a diagonal matrix with ± 1 non-zero entries, which is (randomly) generated at initialization and ﬁxed during training.
(4) Exponential map (Lezcano-Casado & Martínez-Rubio, 2019; Lezcano Casado, 2019).
The mapping provides a surjective parameterization of all special orthogonal matrices (with +1 determinant) with using skew-symmetry matrices (i.e., A (cid:62) [Equation] A ).
[Equation]
where the inﬁnite sum can be computed exactly up to machine-precision (Higham, 2009).
LezcanoCasado & Martínez-Rubio (2019) derives an efﬁcient backpropagation algorithm for the mapping, making it an exact and efﬁcient parameterization in neural networks.
To support a complete parameterization for all orthogonal matrices, Lezcano Casado (2019) extends the mapping as:
[Equation]
where Q is an orthogonal matrix, which is generated at initialization and ﬁxed during training.
In principle, we can use any of these approaches to parameterize the orthogonal matrices.
In this work, we choose exponential map due to its exactness, efﬁciency, and completeness.
D L EARNING D EEP O RTHOGONAL N ETWORKS WITH L IPSCHITZ B OUNDS
In this section, we ﬁrst discuss the properties of GroupSort and Lipschitz networks. Subsequently, we prove Proposition 4.1, which exhibits two approaches to construct Lipschitz residual blocks.
Lastly, we prove in Proposition D.1 when a paraunitary system (represented by a complete factorization as in Theorem C.7) reduces to an orthogonal matrix.
The reduction allows us to apply the initialization methods for orthogonal matrices to paraunitary systems.
GroupSort and orthogonality.
The GroupSort activation separates inputs into groups and sorts each group into ascending order (Anil et al., 2019).
It guarantees two properties: (1) The activation is norm preserving in the forward pass — a sorting function does not change the norm of any input vector.
The activation is gradient norm preserving in the backward pass — a sorting function acts as a permutation, which is orthogonal.
GroupSort with the group size of two is a spacial case that we use in the paper followed by (Chernodub & Nowicki, 2016; Anil et al., 2019). A Lipschitz network with orthogonal layers and GroupSort activations is locally orthogonal . Given any input x to the network, there exists a neighborhood N ( x, r ) with radius r such that the sorting order in each activation does not change. In this neighborhood N ( x, r ) , each GroupSort layer operates as a constant permutation (thus orthogonal); consequently, the whole network is locally orthogonal.
Lipschitz residual blocks.
In Proposition 4.1, we prove the Lipschitzness of two types of residual blocks, one based on additive skip-connection and another based on concatenative one (See Figure 6).
Proof for Proposition 4.1.
We ﬁrst prove the Lipschitzness for the additive residual block f : f ( x ) (cid:44) αf 1 ( x ) + (1 − α ) f 2 ( x ) .
Let x , x (cid:48) be two inputs to f and f ( x ) , f ( x (cid:48) ) be their outputs, we have (cid:107) f ( x (cid:48)
[Equation]
where Equation (D.3) makes uses of the triangle inequality, and Equation (D.4) is due to the LLipschitzness of both f 1 , f 2 .
Therefore, we have shown that (cid:107) f
( x (cid:48) )
− f ( x ) (cid:107) ≤ L (cid:107) x (cid:48)
− x (cid:107) .
Similarly, we prove the Lipschitzness for the concatenative residual block g : g ( x ) (cid:44)
P (cid:2) g 1 ( x 1 ); g 2 ( x 2 ) (cid:3) .
Let x , x (cid:48) be two inputs to g and g ( x ) , g ( x (cid:48) ) be their outputs, we have [Equation] [Equation] D
[Equation]
D.
[Equation] D [Equation] where Equation (D.7) utilizes (cid:107) P x (cid:107)
=
(cid:107) x (cid:107) , ∀ x , and Equation (D.9) is due to the L -Lipschitzness of g 1 , g 2 .
The equations above implies that (cid:107) g
( x (cid:48) )
− g ( x ) (cid:107) ≤ L (cid:107) x (cid:48) − x (cid:107) .
Reduction of a paraunitary system to an orthogonal matrix.
In Proposition D.1, we prove a special case when a paraunitary system reduces to an orthogonal matrix. The reduction allows us to apply the initialization methods for orthogonal matrices to paraunitary systems.
Proposition D.1 (Reduction of a paraunitary matrix to an orthogonal matrix) .
Suppose a paraunitary system H ( z ) takes the complete factorization in Equation (C.55) , and assume
L =
L with U ( − (cid:96) ) =
QU ( (cid:96) ) for all (cid:96) , then the paraunitary matrix H ( z ) reduces to an orthogonal matrix Q ,
[Equation]
(D.12)
[Figure]
Proof for Proposition D.1.
In order to prove Equation (D.12), it sufﬁce to show that
[Equation]
and Equation (D.12) will reduce recursively to the orthogonal matrix Q .
For simplicity, we rewrite
U ( − (cid:96) ) as L and U ( (cid:96) ) as R , by which we have L = QR (or R = Q (cid:62) L ) and we aim to prove
V ( z ; L ) QV ( z − 1 ; R ) =
Q .
By the deﬁnition of V ( z ; · ) in Equation (C.54), we expand it as
[Equation]
Therefore, we will need to show that c [ − 1] = 0 , c [1] = 0 and c [0] =
Q .
We ﬁrst show that both c [ − 1] for z and c [1] for z − 1 are zero matrices.
[Equation]
QRR (cid:62) (D.21) = ( QR ) R (cid:62) − LL (cid:62) ( QR ) R (cid:62) (D.22) = LR (cid:62) − L ( L (cid:62) L ) R (cid:62) (D.23) = LR (cid:62) − LR (cid:62) = 0 . (D.24)
[Table]
Lastly, we show that the constant coefﬁcient c [0] is equal to Q .
[Equation]
which completes the proof.
E S UPPLEMENTARY M ATERIALS FOR E XPERIMENTS E. E XPERIMENTAL S ETUP
Network architectures.
For fair comparisons, we follow the architectures by Trockman & Kolter (2021) for KW-Large, ResNet9, WideResNet10-10 (i.e., shallow networks). We set the group size for
GroupSort activations as 2 in all experiments.
For networks deeper than 10 layers, we implement their architectures modifying from the Pytorch ofﬁcial implementation of ResNet.
It is crucial to replace the global pooling before fully-connected layers with an average pooling with a window size of 4 .
For the average pooling, we multiply the output with the window size to maintain its 1 -Lipschitzness.
Other architectures, including ShufﬂeNet and plain convolutional network (ConvNet), are further modiﬁed from the ResNet, where only the skip-connections are changed or removed.
We use the widen factor to indicate the channel number: we set the number of channels at each layer as base channels multiplied by the widen factor.
The base channels are 16 , 32 , 64 for three groups of residual blocks.
More details of the ResNet architecture can be found in the ofﬁcial Pytorch implementation.
Learning strategies.
We use the CIFAR-10 dataset for all our experiments.
We normalize all input images to [0 , 1] followed by standard augmentation, including random cropping and horizontal ﬂipping.
We use the Adam optimizer with a maximum learning rate of 10 − 2 coupled with a piece-wise triangular learning rate scheduler.
We initialize all our SC-Fac layers as permutation matrices: (1) we select the number of columns for each pair U ( (cid:96) ) , U ( − (cid:96) ) uniformly from { 1 , · · · , T } at initialization (the number is ﬁxed during training); (2) for (cid:96) > 0 , we sample the entries in U ( (cid:96) ) uniformly with respect to the Haar measure; (3) for (cid:96) < 0 , we set U ( − (cid:96) )
=
QU ( (cid:96) )
according to Proposition D.1.
E. A DDITIONAL E MPIRICAL R ESULTS ON H YPER - PARAMETERS S ELECTION
Multi-class hinge loss.
Following previous works on Lipschitz networks (Anil et al., 2019; Li et al., 2019b; Trockman & Kolter, 2021), we adopt the multi-class hinge loss in training.
For each model, we perform a grid search on different margins (cid:15) 0 ∈ { 1 × 10 − 3 , 2 × 10 − 3 , 5 × 10 − 3 , 1 × 10 − 2 , 2 × 10 − 2 , 5 × 10 − 2 , 0 . 1 , 0 . 2 , 0 . 5 } and report the best performance in terms of robust accuracy.
Notice that the margin (cid:15) 0 controls the trade-off between clean and robust accuracy, as shown in Figure 7.
Initialization methods.
In Proposition D.1, we have shown how to initialize our orthogonal convolutional layers as orthogonal matrices.
In Table 6, we perform a study on different initialization methods, including identical, permutation, uniform, and torus (Henaff et al., 2016;
Helfrich et al., 2018).
We ﬁnd that permutation works the best for WideResNet22-10, while all methods are similar in shallower
WideResNet10-10.
Therefore, we use permutation initialization for all other experiments. [Figure] [Table]
Network depth and width.
Exact orthogonality is criticized for harming the expressive power of neural networks, and we ﬁnd that increasing network depth/width can partially compensate for such loss.
In Table 7, we perform a study on the impact of network depth/width on the predictive performance.
As shown, deeper/wider architectures consistently improve both the clean and robust accuracy for our implementation.
However, the best robust accuracy is achieved by a 22 -layer network since we can afford a wide architecture for 34 -layer architecture.
Comparison against normal convolutional networks.
In Table 8, We perform a comparison between our orthogonal networks and normal convolutional networks.
Their architecture are identical except for the activation function (GroupSort for ours and ReLU for normal convolutional networks).
Since batch normalization is common in normal convolutional networks but not in Lipschitz networks,
[Table]
[Table]
[-Normal (with BN) 16.56 34.16 87.
-]
[Table]
Ours 76.
[Table] we provide both results for normal convolutional networks with or without batch normalization. In the table, we report the clean/robust accuracy, train time for epoch, and inference time for the test set.
Robustness against (cid:96) ∞ attacks using adversarial training.
Since orthogonality only guarantees (cid:96)
Lipschitzness, Lipschitz networks with orthogonal layers are not naturally robust to (cid:96) ∞ perturbations.
To further guard Lipschitz networks against (cid:96) ∞ attacks, we follow the approach of adversarial training in Wong et al. For training, we use a FGSM variant with step size 10 / 255 ; for evaluation, we use 50 PGD iterations with step size 2 / 255 and 10 random restarts. We report the experimental results in Table 9. We observe that different orthogonal convolution methods achieve similar (cid:96) ∞ robustness on WideResNet-22.
Furthermore, the Lipschitz networks with WideResNet22 architecture is consistently better than ResNet-18, which is previously used in Singla & Feizi (2021).
E. O N THE N ECESSITY OF E XACT O RTHOGONALITY
Due to the beneﬁts like generalizability and robustness, achieving exact orthogonality in convolutions is the primary goal of a current research line (Sedghi et al., 2019; Li et al., 2019b; Trockman & Kolter, 2021; Singla & Feizi, 2021).
However, until our work, no previous approach achieves orthogonality up to machine precision.
Therefore, our proposed method serves as an extreme case in gaining insight into the trade-off between orthogonality and expressiveness.
In Section 6, we have seen that exact orthogonality is not critical in shallow Lipschitz networks for robustness, and various orthogonal convolutions (with different precision) achieve comparable results.
However, our method is more favorable in deeper networks (with more than 10 layers) — we show the results in Table 7.
It indicates that exact orthogonality is crucial in learning deep Lipschitz networks.
However, without our implementation of exact orthogonality (which does not exist before), it is unclear whether exact orthogonality up to machine precision is needed in Lipschitz networks.
Moreover, exact orthogonality is essential for other important and timely applications.
For example, reversible networks/normalizing ﬂows (Kingma & Dhariwal, 2018; Van Den Berg et al., 2018) require exact orthogonality to compute inverse transform and determinants accurately.
F O RTHOGONAL C ONVOLUTIONS FOR R ESIDUAL F LOWS
In this section, we ﬁrst review the class of ﬂow-based generative models (Papamakarios et al., 2019;
Kobyzev et al., 2020).
We focus on invertible residual network (Behrmann et al., 2019), a ﬂow-based model that relies on Lipschitz residual block, and its extended version Residual Flow (Chen et al., 2019).
We then show how to construct improved Residual Flow using our orthogonal convolutions.
Flow-based models.
Given an observable vector x ∈ R D and a latent vector z ∈ R D , we deﬁne a bijective mapping f : R D
→ R D from the latent vector z to an observation x = f ( z ) .
We further deﬁne the inverse of f as F = f − 1 , with which we represent the likelihood of x by the one of z as:
[Equation]
where p X is the data distribution, p Z is the base distribution (usually a normal distribution), and
J F ( x ) is the Jacobian of F at x .
In practice, the bijective mapping f is composed by a sequence of K bijective mapping such that f = f K ◦ · · · ◦ f 1 , where each f k is named as a ﬂow .
Since the inverse mapping F = F 1 ◦ · · · F K transforms the data distribution p X into a normal distribution p Z , ﬂow-based models are also known as normalizing ﬂows .
Accordingly, we rewrite Equation (F.1) as:
[Equation]
In a practical ﬂow-based model, we require efﬁcient computations of (a) each bijective mapping f k , (b) its inverse mapping F k
= f − 1 k , and (c) the corresponding log-determinant ln | det J F ( · ) | .
Invertible residual networks (i-ResNets).
Behrmann et al.
(2019) proposes a ﬂow-based model based on residual network (ResNet).
Note that a block in ResNet is deﬁned as F ( x ) =
x + g ( x ) , where g is a convolutional network.
In (Behrmann et al., 2019), the authors prove that F is a bijective mapping if g is 1 -Lipschitz, and its inverse mapping can be computed by ﬁxed-point iterations :
[Equation]
where y = g ( x ) is the output of F and the initialization of the iterative algorithm is x 0 :
=
y .
From the Banach ﬁxed-point theorem, we have
[Equation]
i.e., the convergence rate is exponential in the number of iterations and smaller Lipschitz constant will yield faster convergence.
Furthermore, the log-determinant can be computed as:
[Equation]
where the inﬁnite sum is approximated by truncation and the trace is efﬁciently estimated using the
Hutchinson trace estimator tr ( A ) =
E v ∼N ( 0 , I ) [ v (cid:62) Av ] .
To constrain the Lipschitz constant, i-ResNet uses spectral normalization on each linear layer in the block.
Moreover, to improve optimization stability, i-ResNet changes the activation function from
ReLU to ELU, ensuring nonlinear activations have continous derivatives.
As summarized in the conclusion of Behrmann et al.
(2019), there are two remaining problems in this model: (1) The estimator of the log-determinant is biased and inefﬁcient; (2) Designing and learning networks with a Lipschitz constraint are challenging — one needs to constrain each linear layer in the block instead of being able to control the Lipschitz constant of a block.
Residual Flow.
Chen et al.
(2019) address problem {(1) by proposing an unbiased Russian roulette estimator for Equation (F.6):
[Equation]
[Table]
where n ∼ p ( N ) and v ∼ N ( 0 , I ) .
Residual Flow further changes the activation function from ELU to LipSwish.
The LipSwich activation avoids derivative saturation, which occurs when the second derivative is zero in a large region.
However, problem (2) remains unresolved.
Residual ﬂows with orthogonal convolutions.
We propose to address problem (2) by replacing the spectral normalized layers by our orthogonal convolutional layers (SC-Fac).
Note that orthogonal convolutions directly control the Lipschitz constant of a ResNet block.
We keep all other components unchanged — in particular, we use LipSwish activation instead of GroupSort, as GroupSort suffers from derivative saturation.
We experiment our model on MNIST dataset.
As shown in Table 10, our model substantially improve the performance over the original Residual Flow.
We display some images generated by our model in Figure 8.