Original: C ONSTRUCTING O RTHOGONAL C ONVOLUTIONS IN AN EXPLICIT MANNER
Revised: C ONSTRUCTING O RTHOGONAL C ONVOLUTIONS IN AN EXPLICIT MANNER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The convolution with orthogonal input-output Jacobian matrix, i.e. , orthogonal convolution, has attracted substantial attention due to their excellent properties.
Revised: The convolution with orthogonal input-output Jacobian matrix, i.e. , orthogonal convolution, has attracted substantial attention due to their excellent properties.

Original: A convolution layer with an orthogonal Jacobian matrix is 1-Lipschitz in the 2-norm, making the output robust to the perturbation in input.
Revised: A convolution layer with an orthogonal Jacobian matrix is 1-Lipschitz in the 2-norm, making the output robust to the perturbation in input.

Original: Meanwhile, an orthogonal
Revised: Meanwhile, an orthogonal

Original: Jacobian matrix preserves the gradient norm in back-propagation, which is critical for stable training deep networks.
Revised: Jacobian matrix preserves the gradient norm in back-propagation, which is critical for stable training deep networks.

Original: Nevertheless, existing orthogonal convolutions are burdened by high computational costs for preserving orthogonality.
Revised: Nevertheless, existing orthogonal convolutions are burdened by high computational costs for preserving orthogonality.

Original: In this work, we exploit the relation between the singular values of the convolution layer’s
Revised: In this work, we exploit the relation between the singular values of the convolution layer’s

Original: Jacobian and the structure of the convolution kernel.
Revised: Jacobian and the structure of the convolution kernel.

Original: To achieve the orthogonality, we explicitly construct the convolution kernel for enforcing all singular values of the convolution layer’s Jacobian to be 1 s. After training, the explicitly constructed orthogonal (ECO) convolution can be constructed only once, and their weights are stored.
Revised: To achieve the orthogonality, we explicitly construct the convolution kernel for enforcing all singular values of the convolution layer’s Jacobian to be 1 s. After training, the explicitly constructed orthogonal (ECO) convolution can be constructed only once, and their weights are stored.

Original: Then, in evaluation, we only need to load the stored weights of the trained
Revised: Then, in evaluation, we only need to load the stored weights of the trained

Original: ECO convolution, and the computational cost of ECO convolution is the same as the standard dilated convolution.
Revised: ECO convolution, and the computational cost of ECO convolution is the same as the standard dilated convolution.

Original: It is signiﬁcantly more efﬁcient than the recent state-of-the-art approach, skew orthogonal convolution (SOC) in evaluation.
Revised: It is signiﬁcantly more efﬁcient than the recent state-of-the-art approach, skew orthogonal convolution (SOC) in evaluation.

Original: Our extensive experiments on CIFAR-10 and CIFAR-100 datasets demonstrate that the proposed ECO convolution is faster than SOC in evaluation while leading to competitive standard and certiﬁed robust accuracies.
Revised: Our extensive experiments on CIFAR-10 and CIFAR-100 datasets demonstrate that the proposed ECO convolution is faster than SOC in evaluation while leading to competitive standard and certiﬁed robust accuracies.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A layer with an orthogonal input-output Jacobian matrix is 1-Lipschitz in the 2-norm, robust to the perturbation in input.
Revised: A layer with an orthogonal input-output Jacobian matrix is 1-Lipschitz in the 2-norm, robust to the perturbation in input.

Original: Meanwhile, it preserves the gradient norm in back-propagating the gradient, which effectively overcomes the gradient explosion and attenuation issues in training deep neural networks.
Revised: Meanwhile, it preserves the gradient norm in back-propagating the gradient, which effectively overcomes the gradient explosion and attenuation issues in training deep neural networks.

Original: In the past years, many studies have shown that exploiting orthogonality of the Jacobian matrix of layers in neural networks can achieve provable robustness to adversarial attacks (Li et al., 2019), stabler and faster training (Arjovsky et al., 2016; Xiao et al., 2018), and improved generalization (Cogswell et al., 2016; Bansal et al., 2018; Sedghi et al., 2019).
Revised: In the past years, many studies have shown that exploiting orthogonality of the Jacobian matrix of layers in neural networks can achieve provable robustness to adversarial attacks (Li et al., 2019), stabler and faster training (Arjovsky et al., 2016; Xiao et al., 2018), and improved generalization (Cogswell et al., 2016; Bansal et al., 2018; Sedghi et al., 2019).

Original: In a fully-connected layer y
Revised: In a fully-connected layer y

Original: =
Revised: =

Original: Wx where W ∈ R c out × c in is the weight matrix, the layer’s inputoutput Jacobian matrix J = ∂ y ∂ x is just W .
Revised: Wx where W ∈ R c out × c in is the weight matrix, the layer’s inputoutput Jacobian matrix J = ∂ y ∂ x is just W .

Original: Thus, preserving the orthogonality of J can be accomplished through imposing the orthogonal constraint on W , which has been extensively studied in previous works (Mhammedi et al., 2017; Ciss´e et al., 2017; Anil et al., 2019).
Revised: Thus, preserving the orthogonality of J can be accomplished through imposing the orthogonal constraint on W , which has been extensively studied in previous works (Mhammedi et al., 2017; Ciss´e et al., 2017; Anil et al., 2019).

Original: In contrast, in a convolution layer, the Jacobian matrix is no longer the weight matrix (convolution kernel).
Revised: In contrast, in a convolution layer, the Jacobian matrix is no longer the weight matrix (convolution kernel).

Original: Instead, it is the circulant matrix composed of convolution kernel (Karami et al., 2019; Sedghi et al., 2019).
Revised: Instead, it is the circulant matrix composed of convolution kernel (Karami et al., 2019; Sedghi et al., 2019).

Original: Thus, generally, simply constructing an orthogonal convolution kernel cannot achieve an orthogonal convolution.
Revised: Thus, generally, simply constructing an orthogonal convolution kernel cannot achieve an orthogonal convolution.

Original: Achieving an orthogonal Jacobian matrix in the convolution layer is more challenging than that in a fully-connected layer.
Revised: Achieving an orthogonal Jacobian matrix in the convolution layer is more challenging than that in a fully-connected layer.

Original: It is plausibly straightforward to expand the convolution kernel to the doubly block circulant Jacobian matrix and directly impose the orthogonal constraint on the
Revised: It is plausibly straightforward to expand the convolution kernel to the doubly block circulant Jacobian matrix and directly impose the orthogonal constraint on the

Original: Jacobian matrix.
Revised: Jacobian matrix.

Original: Nevertheless, it is extremely difﬁcult to construct a Jacobian matrix that is both doubly block-circulant and orthogonal.
Revised: Nevertheless, it is extremely difﬁcult to construct a Jacobian matrix that is both doubly block-circulant and orthogonal.

Original: Block Convolutional Orthogonal parameterization (BCOP) (Li et al., 2019) is one of the pioneering works for constructing the orthogonal convolution neural networks.
Revised: Block Convolutional Orthogonal parameterization (BCOP) (Li et al., 2019) is one of the pioneering works for constructing the orthogonal convolution neural networks.

Original: It adopts the construction algorithm (Xiao et al., 2018) which decomposes a 2-D convolution into a stack of 1-D convolutions and a channel-wise orthogonal transformation.
Revised: It adopts the construction algorithm (Xiao et al., 2018) which decomposes a 2-D convolution into a stack of 1-D convolutions and a channel-wise orthogonal transformation.

Original: Trockman & Kolter (2021) maps the convolution kernel and the feature tensor into the frequency domain using Fast Fourier Transform (FFT), and achieves the orthogonality of the Jacobian matrix through Cayley transform on the weight matrix in the frequency domain.
Revised: Trockman & Kolter (2021) maps the convolution kernel and the feature tensor into the frequency domain using Fast Fourier Transform (FFT), and achieves the orthogonality of the Jacobian matrix through Cayley transform on the weight matrix in the frequency domain.

Original: They devise the convolution kernel of the same size as the input feature map, which takes more parameters than standard convolution layers with a local reception ﬁeld.
Revised: They devise the convolution kernel of the same size as the input feature map, which takes more parameters than standard convolution layers with a local reception ﬁeld.

Original: Meanwhile,
Revised: Meanwhile,

Original: FFT maps the real-value feature map and the weight matrix into matrices of complex values, increasing the computational cost to 4 times its counterpart in the real-value domain.
Revised: FFT maps the real-value feature map and the weight matrix into matrices of complex values, increasing the computational cost to 4 times its counterpart in the real-value domain.

Original: Meanwhile, the
Revised: Meanwhile, the

Original: Cayley transform requires computing the matrix inverse, which is not friendly for GPU computation.
Revised: Cayley transform requires computing the matrix inverse, which is not friendly for GPU computation.

Original: Recently, Skew Orthogonal Convolutions (SOC) (Singla & Feizi, 2021b) devises skew-symmetric ﬁlter and exploits matrix exponential (Hoogeboom et al., 2020) to attain the orthogonality of the
Revised: Recently, Skew Orthogonal Convolutions (SOC) (Singla & Feizi, 2021b) devises skew-symmetric ﬁlter and exploits matrix exponential (Hoogeboom et al., 2020) to attain the orthogonality of the

Original: Jacobian matrix.
Revised: Jacobian matrix.

Original: But SOC is slow in evaluation since it needs to apply a convolution ﬁlter multiple times sequentially on the feature map to obtain the Taylor expansion.
Revised: But SOC is slow in evaluation since it needs to apply a convolution ﬁlter multiple times sequentially on the feature map to obtain the Taylor expansion.

Original: Observing the efﬁciency limitations of the existing methods, in this work, we propose an explicitly constructed orthogonal (ECO) convolution, which is fast in both training and evaluation.
Revised: Observing the efﬁciency limitations of the existing methods, in this work, we propose an explicitly constructed orthogonal (ECO) convolution, which is fast in both training and evaluation.

Original: It is well known that the Jacobian matrix of a layer is orthogonal if and only if each of its singular values is 1 .
Revised: It is well known that the Jacobian matrix of a layer is orthogonal if and only if each of its singular values is 1 .

Original: Thus we convert the problem of ensuring the orthogonality of the Jacobian matrix into making every singular value as 1 .
Revised: Thus we convert the problem of ensuring the orthogonality of the Jacobian matrix into making every singular value as 1 .

Original: Based on the relation between the singular values of the
Revised: Based on the relation between the singular values of the

Original: Jacobian matrix for the convolution layer and the weight matrix of the convolution layer discovered by Sedghi et al.
Revised: Jacobian matrix for the convolution layer and the weight matrix of the convolution layer discovered by Sedghi et al.

Original: (2019), we construct the convolution kernel so that it ensures every singular value of the Jacobian matrix to be 1 .
Revised: (2019), we construct the convolution kernel so that it ensures every singular value of the Jacobian matrix to be 1 .

Original: Compared with the recent state-of-the-art method, SOC (Singla & Feizi, 2021b) implicitly approximating orthogonal convolution by multiple times convolution operations, ours explicitly builds the orthogonal convolution.
Revised: Compared with the recent state-of-the-art method, SOC (Singla & Feizi, 2021b) implicitly approximating orthogonal convolution by multiple times convolution operations, ours explicitly builds the orthogonal convolution.

Original: Thus, we can directly deploy the constructed orthogonal convolution in evaluation, taking the same computational cost as the standard convolution.
Revised: Thus, we can directly deploy the constructed orthogonal convolution in evaluation, taking the same computational cost as the standard convolution.

Original: It is more efﬁcient than SOC with multiple times convolution operations in evaluation.
Revised: It is more efﬁcient than SOC with multiple times convolution operations in evaluation.

Original: Experiments on CIFAR10 and CIFAR100 show that, in evaluation, taking less time, ours achieves competitive standard and robust accuracy compared with SOC.
Revised: Experiments on CIFAR10 and CIFAR100 show that, in evaluation, taking less time, ours achieves competitive standard and robust accuracy compared with SOC.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Weight orthogonalization.
Revised: Weight orthogonalization.

Original: In a fully-connected layer, the input-output Jacobian matrix is the transform itself.
Revised: In a fully-connected layer, the input-output Jacobian matrix is the transform itself.

Original: Thus many efforts are devoted to orthogonalizing the weights.
Revised: Thus many efforts are devoted to orthogonalizing the weights.

Original: Some early works exploit an orthogonal weight initialization to speed up the training (Saxe et al., 2014; Pennington et al., 2017) and build extremely deep neural networks (Xiao et al., 2018).
Revised: Some early works exploit an orthogonal weight initialization to speed up the training (Saxe et al., 2014; Pennington et al., 2017) and build extremely deep neural networks (Xiao et al., 2018).

Original: Recently, more efforts are devoted to exploiting the orthogonality not only in initialization but also in the whole training process.
Revised: Recently, more efforts are devoted to exploiting the orthogonality not only in initialization but also in the whole training process.

Original: Some approaches propose “soft” constraints on the weight matrix.
Revised: Some approaches propose “soft” constraints on the weight matrix.

Original: For example, Bansal et al.
Revised: For example, Bansal et al.

Original: ; Xiong et al.
Revised: ; Xiong et al.

Original: (2016) introduce mutual coherence and spectral restricted isometry as orthogonal regularization on the weight matrix.
Revised: (2016) introduce mutual coherence and spectral restricted isometry as orthogonal regularization on the weight matrix.

Original: Parseval Networks (Ciss´e et al., 2017) adapts a regularizer to encourage the orthogonality of the weight matrices.
Revised: Parseval Networks (Ciss´e et al., 2017) adapts a regularizer to encourage the orthogonality of the weight matrices.

Original: However, these methods can not guarantee the exact orthogonality of the weight matrix.
Revised: However, these methods can not guarantee the exact orthogonality of the weight matrix.

Original: Some approaches orthogonalize features during the forward pass.
Revised: Some approaches orthogonalize features during the forward pass.

Original: For example, Huang et al.
Revised: For example, Huang et al.

Original: (2018b) extends Batch Normalization (Ioffe & Szegedy, 2015) with ZCA; (Huang et al., 2018a) solves a suboptimization problem to decorrelate deep features.
Revised: (2018b) extends Batch Normalization (Ioffe & Szegedy, 2015) with ZCA; (Huang et al., 2018a) solves a suboptimization problem to decorrelate deep features.

Original: These methods can not avoid computationally expensive operations like SVD, thus slow in practice.
Revised: These methods can not avoid computationally expensive operations like SVD, thus slow in practice.

Original: Some approaches enforce orthogonality by using the Riemannian optimization on the
Revised: Some approaches enforce orthogonality by using the Riemannian optimization on the

Original: Stiefel manifold.
Revised: Stiefel manifold.

Original: For example, Casado & Mart ´ ınez-Rubio (2019) uses Pade approximation as an alternative to matrix exponential mapping to update the weight matrix on the Stifel manifold.
Revised: For example, Casado & Mart ´ ınez-Rubio (2019) uses Pade approximation as an alternative to matrix exponential mapping to update the weight matrix on the Stifel manifold.

Original: Li et al.
Revised: Li et al.

Original: (2020) uses an iterative Cayley transform to enforce weight matrix orthonormal.
Revised: (2020) uses an iterative Cayley transform to enforce weight matrix orthonormal.

Original: Anil et al.
Revised: Anil et al.

Original: (2019) also attains orthogonal weights by orthogonal parameterization based on Bj ¨o rck orthonormalization (BjoRick & Bowie, 1971).
Revised: (2019) also attains orthogonal weights by orthogonal parameterization based on Bj ¨o rck orthonormalization (BjoRick & Bowie, 1971).

Original: Lastly, some approaches incorporate orthogonal constraint into their network architecture.
Revised: Lastly, some approaches incorporate orthogonal constraint into their network architecture.

Original: For example, Mhammedi et al.
Revised: For example, Mhammedi et al.

Original: (2017) builds orthgonal layers in
Revised: (2017) builds orthgonal layers in

Original: RNNs with householder reﬂections.
Revised: RNNs with householder reﬂections.

Original: Orthogonal convolution.
Revised: Orthogonal convolution.

Original: In a convolution layer, the Jacobian matrix is no longer the weight matrix.
Revised: In a convolution layer, the Jacobian matrix is no longer the weight matrix.

Original: Thus, the above-mentioned weight orthogonalization methods cannot be trivially applied in convolution layers.
Revised: Thus, the above-mentioned weight orthogonalization methods cannot be trivially applied in convolution layers.

Original: Wang et al.
Revised: Wang et al.

Original: ; Qi et al.
Revised: ; Qi et al.

Original: (2020) encourage the orthogonality of the Jacobian matrix of convolution layers through a regularizer, but they cannot achieve the strict orthogonality and cannot attain the provable robustness.
Revised: (2020) encourage the orthogonality of the Jacobian matrix of convolution layers through a regularizer, but they cannot achieve the strict orthogonality and cannot attain the provable robustness.

Original: Block Convolutional Orthogonal parameterization (BCOP) (Li et al., 2019) is a pioneering work for enforcing the orthogonality of the Jacobian matrix of a convolution layer.
Revised: Block Convolutional Orthogonal parameterization (BCOP) (Li et al., 2019) is a pioneering work for enforcing the orthogonality of the Jacobian matrix of a convolution layer.

Original: It conducts parameterization of orthogonal convolutions by adapting the construction algorithm proposed by Xiao et al.
Revised: It conducts parameterization of orthogonal convolutions by adapting the construction algorithm proposed by Xiao et al.

Original: But BCOP is slow in training.
Revised: But BCOP is slow in training.

Original: Trockman & Kolter (2021) applies the Cayley transform to a skew-symmetric convolution weight in the Fourier domain so that the convolution recovered from the Fourier domain has an orthogonal Jacobian matrix.
Revised: Trockman & Kolter (2021) applies the Cayley transform to a skew-symmetric convolution weight in the Fourier domain so that the convolution recovered from the Fourier domain has an orthogonal Jacobian matrix.

Original: Though it achieves higher efﬁciency in some applications than BCOP, it is still signiﬁcantly more costly than standard convolutional layers.
Revised: Though it achieves higher efﬁciency in some applications than BCOP, it is still signiﬁcantly more costly than standard convolutional layers.

Original: Skew Orthogonal Convolution (Singla & Feizi, 2021b) achieves orthogonal convolution through Taylor expansion of the matrix exponential.
Revised: Skew Orthogonal Convolution (Singla & Feizi, 2021b) achieves orthogonal convolution through Taylor expansion of the matrix exponential.

Original: It is faster than BCOP in training but slower in evaluation.
Revised: It is faster than BCOP in training but slower in evaluation.

Original: In contrast, our method is efﬁcient in both training and evaluation.
Revised: In contrast, our method is efﬁcient in both training and evaluation.

Original: Recently, Su et al.
Revised: Recently, Su et al.

Original: (2021) achieves orthogonal convolutions via paraunitary systems in an elegant manner.
Revised: (2021) achieves orthogonal convolutions via paraunitary systems in an elegant manner.

Original: Nevertheless, it also suffers from inefﬁciency issue.
Revised: Nevertheless, it also suffers from inefﬁciency issue.

Original:  P RELIMINARY
Revised:  P RELIMINARY

Original: Notation.
Revised: Notation.

Original: For a matrix M , M [ i, j ] denotes the element in the i -th row and the j -th column, M [ i, :] denotes the i -th row and M [: , j ] denotes the j -th column.
Revised: For a matrix M , M [ i, j ] denotes the element in the i -th row and the j -th column, M [ i, :] denotes the i -th row and M [: , j ] denotes the j -th column.

Original: We use the similar notation for indexing tensors.
Revised: We use the similar notation for indexing tensors.

Original: For n ∈ N , [ n ] denotes the set of n numbers from 0 to n − 1 , that is, { 0 , 1 , · · · , n − 1 } .
Revised: For n ∈ N , [ n ] denotes the set of n numbers from 0 to n − 1 , that is, { 0 , 1 , · · · , n − 1 } .

Original: We denote the function to obtain the singular values by σ ( · ) .
Revised: We denote the function to obtain the singular values by σ ( · ) .

Original: Let vec( · ) denotes the function which unfolds a matrix or higher-order tensor into a vector.
Revised: Let vec( · ) denotes the function which unfolds a matrix or higher-order tensor into a vector.

Original: Let ω
Revised: Let ω

Original: k
Revised: k

Original: = exp(2 πi/k ) where i = √− 1 .
Revised: = exp(2 πi/k ) where i = √− 1 .

Original: Let
Revised: Let

Original: F k ∈ R k × k be matrix for the discrete Fourier transform (DFT) for k -element vectors and each entry of F k is deﬁned as F k [ p, q ]
Revised: F k ∈ R k × k be matrix for the discrete Fourier transform (DFT) for k -element vectors and each entry of F k is deﬁned as F k [ p, q ]

Original: = ω pqk .
Revised: = ω pqk .

Original: 3. R ELATION BETWEEN THE J ACOBIAN MATRIX AND THE WEIGHT MATRIX
Revised: 3. R ELATION BETWEEN THE J ACOBIAN MATRIX AND THE WEIGHT MATRIX

Original: We denote the input tensor of a convolution layer by X ∈ R c × n × n , where n is the spatial size, and c is the number of input channels.
Revised: We denote the input tensor of a convolution layer by X ∈ R c × n × n , where n is the spatial size, and c is the number of input channels.

Original: We denote the convolution operation on X by conv ˆ W ( X ) , where ˆ W ∈ R k × k × c × c size where k is the size of the reception ﬁeld and k is normally smaller than size of the input image, n .
Revised: We denote the convolution operation on X by conv ˆ W ( X ) , where ˆ W ∈ R k × k × c × c size where k is the size of the reception ﬁeld and k is normally smaller than size of the input image, n .

Original: In this deﬁnition, we set the number of output channels identical to that of input channels.
Revised: In this deﬁnition, we set the number of output channels identical to that of input channels.

Original: Meanwhile, we set the convolution stride as 1 and adapt the cyclic padding to make the output of the same size as the input.
Revised: Meanwhile, we set the convolution stride as 1 and adapt the cyclic padding to make the output of the same size as the input.

Original: We denote the output tensor by Y ∈ R c × n × n and each element of Y is obtained by
Revised: We denote the output tensor by Y ∈ R c × n × n and each element of Y is obtained by

Original: [Equation]
Revised: [Equation]

Original: Since the convolution operation is a linear transformation, there exists a linear transformation M ∈
Revised: Since the convolution operation is a linear transformation, there exists a linear transformation M ∈

Original: R n 2 c × n 2 c which satisﬁes:
Revised: R n 2 c × n 2 c which satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: To simplify illustration, ˆ W ∈ R k × k × c × c is normally expanded into W ∈ R n × n × c × c through zeropadding.
Revised: To simplify illustration, ˆ W ∈ R k × k × c × c is normally expanded into W ∈ R n × n × c × c through zeropadding.

Original: We term W as the expanded convolution kernel.
Revised: We term W as the expanded convolution kernel.

Original: In W , only k 2 c 2 elements are non-zero.
Revised: In W , only k 2 c 2 elements are non-zero.

Original: Based on the above deﬁnition, we review the following theorem in Sedghi et al.
Revised: Based on the above deﬁnition, we review the following theorem in Sedghi et al.

Original: Theorem 1 ( see Sedghi et al.
Revised: Theorem 1 ( see Sedghi et al.

Original: (2019) Section 2.2 ) .
Revised: (2019) Section 2.2 ) .

Original: For any expanded convolution kernel W ∈ R n × n × c × c , let M is the matrix encoding the linear transformation of a convolution with W .
Revised: For any expanded convolution kernel W ∈ R n × n × c × c , let M is the matrix encoding the linear transformation of a convolution with W .

Original: For each p, q ∈ [ n ] × [ n ] , let P ( p,q ) be the c × c matrix computed by
Revised: For each p, q ∈ [ n ] × [ n ] , let P ( p,q ) be the c × c matrix computed by

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The above theorem gives the relation between the singular values of the transformation matrix M and the expanded convolution W .
Revised: The above theorem gives the relation between the singular values of the transformation matrix M and the expanded convolution W .

Original: Theorem 2.
Revised: Theorem 2.

Original: For any W ∈ R n × n × c × c , the Jacobian matrix of the convolution layer, J , is orthogonal if and only if ∀ p, q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p,q ) is 1 .
Revised: For any W ∈ R n × n × c × c , the Jacobian matrix of the convolution layer, J , is orthogonal if and only if ∀ p, q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p,q ) is 1 .

Original: Proof.
Revised: Proof.

Original: A real matrix is orthogonal if and only if each singular value is 1 .
Revised: A real matrix is orthogonal if and only if each singular value is 1 .

Original: Meanwhile, the Jacobian matrix J = ∂ vec( Y ) ∂ vec( X ) =
Revised: Meanwhile, the Jacobian matrix J = ∂ vec( Y ) ∂ vec( X ) =

Original: M .
Revised: M .

Original: Based on Theorem 1, each singular value of M is 1 if and only if ∀ p, q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p,q ) is 1 , thus we complete the proof.
Revised: Based on Theorem 1, each singular value of M is 1 if and only if ∀ p, q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p,q ) is 1 , thus we complete the proof.

Original: 3. M ATRIX O RTHOGONALIZATION
Revised: 3. M ATRIX O RTHOGONALIZATION

Original: Carley transform.
Revised: Carley transform.

Original: Orthogonal parameterization has been extensively studied in the previous works.
Revised: Orthogonal parameterization has been extensively studied in the previous works.

Original: Many works (Absil et al., 2008; Helfrich et al., 2018; Maduranga et al., 2019) utilize Cayley transform on skew-symmetric matrices ( A (cid:62) = - A ) to obtain the orthogonal weight matrices.
Revised: Many works (Absil et al., 2008; Helfrich et al., 2018; Maduranga et al., 2019) utilize Cayley transform on skew-symmetric matrices ( A (cid:62) = - A ) to obtain the orthogonal weight matrices.

Original: Cayley transform is a bijection between skew-symmetric matrices and orthogonal matrices without −eigenvalues.
Revised: Cayley transform is a bijection between skew-symmetric matrices and orthogonal matrices without −eigenvalues.

Original: Speciﬁcally, it maps a skew-symmetric matrix A to an orthogonal matrix Q by
Revised: Speciﬁcally, it maps a skew-symmetric matrix A to an orthogonal matrix Q by

Original: [Equation]
Revised: [Equation]

Original: Li et al.
Revised: Li et al.

Original: (2020) proposes an iterative approximation of the Cayley transform for orthogonallyconstrained optimizers and achieves higher speed for training CNNs and RNNs.
Revised: (2020) proposes an iterative approximation of the Cayley transform for orthogonallyconstrained optimizers and achieves higher speed for training CNNs and RNNs.

Original: Matrix exponential.
Revised: Matrix exponential.

Original: Casado & Mart´ınez-Rubio (2019) exploits the matrix exponential of the skewsymmetric matrix for orthogonal parameterization.
Revised: Casado & Mart´ınez-Rubio (2019) exploits the matrix exponential of the skewsymmetric matrix for orthogonal parameterization.

Original: To be speciﬁc, if A is a skew-symmetric matrix, exp( A ) is orthogonal.
Revised: To be speciﬁc, if A is a skew-symmetric matrix, exp( A ) is orthogonal.

Original: Skew Orthogonal Convolutions (SOC) (Singla & Feizi, 2021b) approximates exp( A ) through Taylor series exp( A )
Revised: Skew Orthogonal Convolutions (SOC) (Singla & Feizi, 2021b) approximates exp( A ) through Taylor series exp( A )

Original: ≈ S T ( A ) =
Revised: ≈ S T ( A ) =

Original: (cid:80)
Revised: (cid:80)

Original: Ti =

Revised: Ti =

Original: A i
Revised: A i

Original: i ! , and derives a bound on the approximation error:
Revised: i ! , and derives a bound on the approximation error:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) · (cid:107) 2 denotes the spectral norm, which is the maximum singular value of a matrix.
Revised: where (cid:107) · (cid:107) 2 denotes the spectral norm, which is the maximum singular value of a matrix.

Original: When (cid:107) A (cid:107) 2 is small, the approximation can achieve high precision, thus SOC further adopts the spectral normalization on A to make (cid:107) A (cid:107) 2 small.
Revised: When (cid:107) A (cid:107) 2 is small, the approximation can achieve high precision, thus SOC further adopts the spectral normalization on A to make (cid:107) A (cid:107) 2 small.

Original: Orthogonality of convolution.
Revised: Orthogonality of convolution.

Original: In a fully-connected layer, its Jacobian matrix is just the transpose of the weight matrix.
Revised: In a fully-connected layer, its Jacobian matrix is just the transpose of the weight matrix.

Original: Thus we can easily achieve the orthogonality of the Jacobian matrix through orthogonal parameterization on the weight matrix through the well-studied orthogonal parameterization approaches such as Cayley transform or matrix exponential of the skew-symmetric matrix.
Revised: Thus we can easily achieve the orthogonality of the Jacobian matrix through orthogonal parameterization on the weight matrix through the well-studied orthogonal parameterization approaches such as Cayley transform or matrix exponential of the skew-symmetric matrix.

Original: Nevertheless, in a convolution layer, the Jacobian matrix is no longer the transpose of the weight matrix.
Revised: Nevertheless, in a convolution layer, the Jacobian matrix is no longer the transpose of the weight matrix.

Original: Thus we can not trivially apply the orthogonal parameterization approaches on the weight matrix to achieve the orthogonality of the Jacobian matrix.
Revised: Thus we can not trivially apply the orthogonal parameterization approaches on the weight matrix to achieve the orthogonality of the Jacobian matrix.

Original: 3. L IPSCHITZNESS AND P ROVABLY R OBUSTNESS
Revised: 3. L IPSCHITZNESS AND P ROVABLY R OBUSTNESS

Original: Lipschitz constant.
Revised: Lipschitz constant.

Original: A function f : R m → R n is L -Lipschitz under the (cid:96)
Revised: A function f : R m → R n is L -Lipschitz under the (cid:96)

Original: 2 norm iff (cid:107) f
Revised: 2 norm iff (cid:107) f

Original: ( x ) − f ( y ) (cid:107) 2 ≤ L (cid:107) x − y (cid:107) 2 , ∀ ( x
Revised: ( x ) − f ( y ) (cid:107) 2 ≤ L (cid:107) x − y (cid:107) 2 , ∀ ( x

Original: , y ) ∈
Revised: , y ) ∈

Original: R m × R m , where L is a positive constant.
Revised: R m × R m , where L is a positive constant.

Original: We deﬁne the smallest value of L satisfying L -Lipschitz for f as the Lipschitz constant of f , denoted by Lip( f ) .
Revised: We deﬁne the smallest value of L satisfying L -Lipschitz for f as the Lipschitz constant of f , denoted by Lip( f ) .

Original: Given two Lipschitz continuous functions f
Revised: Given two Lipschitz continuous functions f

Original: and g , the Lipschitz constant of the a composition of them,
Revised: and g , the Lipschitz constant of the a composition of them,

Original: Lip( f ◦ g ) , is upper-bounded by the product of the Lipschitz constants of f and g (Weaver, 2018):
Revised: Lip( f ◦ g ) , is upper-bounded by the product of the Lipschitz constants of f and g (Weaver, 2018):

Original: [Equation]
Revised: [Equation]

Original: Thus, given a network n ( · ) consisting of L layers, if the Lipschitz constant of each layer is 1 , the
Revised: Thus, given a network n ( · ) consisting of L layers, if the Lipschitz constant of each layer is 1 , the

Original: Lipschitz constant of the whole network, Lip( n ) is less than 1 .
Revised: Lipschitz constant of the whole network, Lip( n ) is less than 1 .

Original: Provably robustness.
Revised: Provably robustness.

Original: Considering a neural network n ( · ) for c -class classiﬁcation.
Revised: Considering a neural network n ( · ) for c -class classiﬁcation.

Original: Given an input x with the ground-truth label y ∈ [ c ] , the network generates a c -dimensional prediction vector n ( x ) =
Revised: Given an input x with the ground-truth label y ∈ [ c ] , the network generates a c -dimensional prediction vector n ( x ) =

Original: [ s 1 , · · · , s c ] where the i -th item in n ( x ) denotes the conﬁdence score for the class i .
Revised: [ s 1 , · · · , s c ] where the i -th item in n ( x ) denotes the conﬁdence score for the class i .

Original: The margin of prediction for x by n ( · ) is deﬁned as
Revised: The margin of prediction for x by n ( · ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: As proved by Tsuzuku et al.
Revised: As proved by Tsuzuku et al.

Original: (2018) and Li et al.
Revised: (2018) and Li et al.

Original: (2019), for a network n ( · ) with a Lipschitz constant of L , x is provably robustly classiﬁed by n ( · ) under perturbation with a (cid:96) 2 -norm of M n ( x ) √ 2 L , that is,
Revised: (2019), for a network n ( · ) with a Lipschitz constant of L , x is provably robustly classiﬁed by n ( · ) under perturbation with a (cid:96) 2 -norm of M n ( x ) √ 2 L , that is,

Original: [Equation]
Revised: [Equation]

Original:  M ETHOD
Revised:  M ETHOD

Original: Based on Theorem 2, to achieve an orthogonal Jacobian matrix J for a convolution layer, we need to ensure that, ∀ p, q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p,q ) is 1 .
Revised: Based on Theorem 2, to achieve an orthogonal Jacobian matrix J for a convolution layer, we need to ensure that, ∀ p, q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p,q ) is 1 .

Original: When P ( p,q ) is a real matrix, this condition is equivalent to the orthogonality of P ( p,q ) .
Revised: When P ( p,q ) is a real matrix, this condition is equivalent to the orthogonality of P ( p,q ) .

Original: To simplify the notation, we assemble the matrices P ( p,q ) ( p, q ∈ [ n ] × [ n ] ) into a tensor P ∈ R n × n × c
Revised: To simplify the notation, we assemble the matrices P ( p,q ) ( p, q ∈ [ n ] × [ n ] ) into a tensor P ∈ R n × n × c

Original: ×
Revised: ×

Original: c
Revised: c

Original: such that P [ p, q, : , :] =
Revised: such that P [ p, q, : , :] =

Original: P ( p,q ) .
Revised: P ( p,q ) .

Original: Based on the deﬁnition, we rewrite the transformation from W to P ( p,q ) in Eq.
Revised: Based on the deﬁnition, we rewrite the transformation from W to P ( p,q ) in Eq.

Original: (3) into
Revised: (3) into

Original: [Equation]
Revised: [Equation]

Original: Since F n is the full-rank square matrix, the mapping from W to P is invertible.
Revised: Since F n is the full-rank square matrix, the mapping from W to P is invertible.

Original: Thus, to achieve an orthogonal Jacobian matrix, we can directly construct orthogonal P [ p, q, : , :] ( p, q ∈ [ n ] × [ n ] ) using
Revised: Thus, to achieve an orthogonal Jacobian matrix, we can directly construct orthogonal P [ p, q, : , :] ( p, q ∈ [ n ] × [ n ] ) using

Original: Carley transform or matrix exponential of skew-symmetric matrices (introduced in Section 3.2) to make each singular value of them identical to 1 .
Revised: Carley transform or matrix exponential of skew-symmetric matrices (introduced in Section 3.2) to make each singular value of them identical to 1 .

Original: Then the convolution kernel weight W can be determined uniquely from the P through the inverse transformation in Eq.
Revised: Then the convolution kernel weight W can be determined uniquely from the P through the inverse transformation in Eq.

Original: Since F n is the
Revised: Since F n is the

Original: Discrete Fourier Transform (DFT) matrix, F (cid:62) n W [: , : , s, t ] F n in Eq.
Revised: Discrete Fourier Transform (DFT) matrix, F (cid:62) n W [: , : , s, t ] F n in Eq.

Original: (10) is equivalent to conducting a 2-dimensional DFT on W [: , : , s, t ] .
Revised: (10) is equivalent to conducting a 2-dimensional DFT on W [: , : , s, t ] .

Original: Thus, we can recover W [: , : , s, t ] through the inverse 2-dimensional DFT (IDFT 2D ) on the constructed orthogonal matrix P
Revised: Thus, we can recover W [: , : , s, t ] through the inverse 2-dimensional DFT (IDFT 2D ) on the constructed orthogonal matrix P

Original: [: , : , s, t ] .
Revised: [: , : , s, t ] .

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: Based on Theorem 2, the Jacobian matrix of the convolution layer with the kernel W is orthogonal.
Revised: Based on Theorem 2, the Jacobian matrix of the convolution layer with the kernel W is orthogonal.

Original: Nevertheless, the above approach has many drawbacks.
Revised: Nevertheless, the above approach has many drawbacks.

Original: First of all, for a generally constructed
Revised: First of all, for a generally constructed

Original: P , the recovered expanded convolution kernel W computed based on the inverse 2-dimensional
Revised: P , the recovered expanded convolution kernel W computed based on the inverse 2-dimensional

Original: DFT might contain n 2 c 2 non-zeros elements.
Revised: DFT might contain n 2 c 2 non-zeros elements.

Original: It signiﬁcantly increases the number of parameters in the convolution layer from k 2 c 2 to n 2 c 2 .
Revised: It signiﬁcantly increases the number of parameters in the convolution layer from k 2 c 2 to n 2 c 2 .

Original: Meanwhile, the convolution with kernel W has a global reception ﬁeld, which is also slower than the original convolution kernel with a local reception ﬁeld.
Revised: Meanwhile, the convolution with kernel W has a global reception ﬁeld, which is also slower than the original convolution kernel with a local reception ﬁeld.

Original: Besides, the above approach needs to compute n 2 times matrix orthogonalization (Carley transform or matrix exponential) on P [ p, q, : , :] ( p, q ∈ [ n ] × [ n ] ).
Revised: Besides, the above approach needs to compute n 2 times matrix orthogonalization (Carley transform or matrix exponential) on P [ p, q, : , :] ( p, q ∈ [ n ] × [ n ] ).

Original: Considering the spatial size of the input feature tensor, n and the input/out channel number c are not a small value, n 2 times orthogonalization on c × c matrices might be computationally expensive.
Revised: Considering the spatial size of the input feature tensor, n and the input/out channel number c are not a small value, n 2 times orthogonalization on c × c matrices might be computationally expensive.

Original: Thus, to achieve a high efﬁcient orthogonal convolution, we devise a 2-dimensional periodic P .
Revised: Thus, to achieve a high efﬁcient orthogonal convolution, we devise a 2-dimensional periodic P .

Original: We set the period as k and assume n is divisible by k .
Revised: We set the period as k and assume n is divisible by k .

Original: Speciﬁcally, P satisﬁes
Revised: Speciﬁcally, P satisﬁes

Original: [Equation]
Revised: [Equation]

Original: We deﬁne P 

Revised: We deﬁne P

Original: = P [0: k, 0: k, : , :] ∈
Revised: = P [0: k, 0: k, : , :] ∈

Original: R k × k × c × c .
Revised: R k × k × c × c .

Original: That is, P can be obtained by repeating P 0 by n/k × n/k times in the ﬁrst two dimensions.
Revised: That is, P can be obtained by repeating P 0 by n/k × n/k times in the ﬁrst two dimensions.

Original: It has many beneﬁts by the 2-dimensional periodic conﬁguration.
Revised: It has many beneﬁts by the 2-dimensional periodic conﬁguration.

Original: First of all, based on the property of DFT, the convolution kernel W [: , : , s, t ] obtained from an inverse 2D DFT on the 2-dimensional periodic P [: , : , s, t ] on contains only k 2 non-zero elements.
Revised: First of all, based on the property of DFT, the convolution kernel W [: , : , s, t ] obtained from an inverse 2D DFT on the 2-dimensional periodic P [: , : , s, t ] on contains only k 2 non-zero elements.

Original: That is, the number of non-zero elements in the recovered W is only k 2 c 2 , which is the same as the number of elements in original convolution kernel ˆ W with a local reception ﬁeld of k × k size.
Revised: That is, the number of non-zero elements in the recovered W is only k 2 c 2 , which is the same as the number of elements in original convolution kernel ˆ W with a local reception ﬁeld of k × k size.

Original: Besides, the non-zero elements in W can be efﬁciently obtained.
Revised: Besides, the non-zero elements in W can be efﬁciently obtained.

Original: To be speciﬁc, we deﬁne W 0 ∈
Revised: To be speciﬁc, we deﬁne W 0 ∈

Original: R k × k × c × c obtained from W 

Revised: R k × k × c × c obtained from W

Original: [: , : , s, t ] = IDFT 2D ( P 

Revised: [: , : , s, t ] = IDFT 2D ( P

Original: [: , : , s, t ]) .
Revised: [: , : , s, t ]) .

Original: Based on the property of DFT (Nussbaumer, 1981), W and W 0 have the following relation:
Revised: Based on the property of DFT (Nussbaumer, 1981), W and W 0 have the following relation:

Original: [Equation]
Revised: [Equation]

Original: That is, we can obtain the non-zero elements of W ∈ R n × n through computing W 0 ∈ R k × k , which just takes only k 2 times 2D FFT on c × c matrices and is efﬁcient.
Revised: That is, we can obtain the non-zero elements of W ∈ R n × n through computing W 0 ∈ R k × k , which just takes only k 2 times 2D FFT on c × c matrices and is efﬁcient.

Original: Moreover, since P is periodic along the ﬁrst two dimensions, to achieve an orthogonal Jacobian matrix based on Theorem 2, we only need to explicitly construct k 2 orthogonal matrices P 

Revised: Moreover, since P is periodic along the ﬁrst two dimensions, to achieve an orthogonal Jacobian matrix based on Theorem 2, we only need to explicitly construct k 2 orthogonal matrices P

Original: [ p, q, : , :] (( p, q ) ∈
Revised: [ p, q, : , :] (( p, q ) ∈

Original: [ k ] × [ k ] ) to ensure that n 2 matrices P [ p, q, : , :] ( ( p, q ) ∈ [ n ] × [ n ] ) are orthogonal, which is also efﬁcient.
Revised: [ k ] × [ k ] ) to ensure that n 2 matrices P [ p, q, : , :] ( ( p, q ) ∈ [ n ] × [ n ] ) are orthogonal, which is also efﬁcient.

Original: Moreover, as the recovered W [ i, j, : , :] is non-zero only if i % nk = 

Revised: Moreover, as the recovered W [ i, j, : , :] is non-zero only if i % nk =

Original: and j % nk = 0 .
Revised: and j % nk = 0 .

Original: This property corresponds to the dilated convolution (Yu & Koltun, 2016).
Revised: This property corresponds to the dilated convolution (Yu & Koltun, 2016).

Original: Speciﬁcally, using the relation between W and Win Eq.
Revised: Speciﬁcally, using the relation between W and Win Eq.

Original: (13), the following equation holds:
Revised: (13), the following equation holds:

Original: [Equation]
Revised: [Equation]

Original: That is, we can achieve the orthogonal convolution through a dilated convolution with the kernel Wand the dilation n/k .
Revised: That is, we can achieve the orthogonal convolution through a dilated convolution with the kernel Wand the dilation n/k .

Original: The dilated convolution has been well supported by mainstream deep learning frameworks like Pytorch and Tensorﬂow.
Revised: The dilated convolution has been well supported by mainstream deep learning frameworks like Pytorch and Tensorﬂow.

Original: Based on above analysis, the process of achieving the orthogonal convolution is decomposed into three steps:
Revised: Based on above analysis, the process of achieving the orthogonal convolution is decomposed into three steps:

Original: Construct the orthogonal matrices P 

Revised: Construct the orthogonal matrices P

Original: [ p, q, : , :] , ∀ ( p, q ) ∈
Revised: [ p, q, : , :] , ∀ ( p, q ) ∈

Original: [ k ] × [ k ] .
Revised: [ k ] × [ k ] .

Original: Recover W 

Revised: Recover W

Original: [: , : , s, t ] from P 

Revised: [: , : , s, t ] from P

Original: [: , : , s, t ] using inverse 2D DFT, ∀ ( s, t ) ∈
Revised: [: , : , s, t ] using inverse 2D DFT, ∀ ( s, t ) ∈

Original: [ c ] × [ c ] .
Revised: [ c ] × [ c ] .

Original: Conduct the dilated convolution using the recovered kernel W 0 and the dilation n/k .
Revised: Conduct the dilated convolution using the recovered kernel W 0 and the dilation n/k .

Original: Below we introduce the details in these three steps individually.
Revised: Below we introduce the details in these three steps individually.

Original: Orthogonal parameterization for P 0 [ p, q, : , :] .
Revised: Orthogonal parameterization for P 0 [ p, q, : , :] .

Original: This step can be implemented by any matrix orthogonalization approach such as Carley Transform on skew-symmetric matrix or matrix exponential of the skew-symmetric matrix.
Revised: This step can be implemented by any matrix orthogonalization approach such as Carley Transform on skew-symmetric matrix or matrix exponential of the skew-symmetric matrix.

Original: By default, we adopt the Taylor expansion for the exponential of the skew-symmetric matrix proposed in SOC (Singla & Feizi, 2021b) in this step considering its simplicity and efﬁciency.
Revised: By default, we adopt the Taylor expansion for the exponential of the skew-symmetric matrix proposed in SOC (Singla & Feizi, 2021b) in this step considering its simplicity and efﬁciency.

Original: To be speciﬁc, we start from a parameter tensor P 0 ∈ R k × k × c × c .
Revised: To be speciﬁc, we start from a parameter tensor P 0 ∈ R k × k × c × c .

Original: ∀ ( p, q ) ∈ [ k ] × [ k ] , we compute
Revised: ∀ ( p, q ) ∈ [ k ] × [ k ] , we compute

Original: [Equation]
Revised: [Equation]

Original: Recall from Eq.
Revised: Recall from Eq.

Original: (6) that the approximation error between exp( A ) and the Taylor expansion S k ( A ) , (cid:107) exp
Revised: (6) that the approximation error between exp( A ) and the Taylor expansion S k ( A ) , (cid:107) exp

Original: ( A ) − S k
Revised: ( A ) − S k

Original: ( A ) (cid:107) 2 , is bounded by (cid:107) A (cid:107) k 2 k ! .
Revised: ( A ) (cid:107) 2 , is bounded by (cid:107) A (cid:107) k 2 k ! .

Original: That is, when (cid:107) A (cid:107) 2 < 1 , it can achieve a good approximation.
Revised: That is, when (cid:107) A (cid:107) 2 < 1 , it can achieve a good approximation.

Original: Thus, following SOC (Singla & Feizi, 2021b), we conduct spectral normalization on
Revised: Thus, following SOC (Singla & Feizi, 2021b), we conduct spectral normalization on

Original: P 

Revised: P

Original: [ p, q, : , :] so that (cid:107) exp
Revised: [ p, q, : , :] so that (cid:107) exp

Original: ( P 0 [ p, q, : , :]) − (cid:80) Ti =0 P 

Revised: ( P 0 [ p, q, : , :]) − (cid:80) Ti =0 P

Original: [ p,q, : , :] i i !
Revised: [ p,q, : , :] i i !

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Recover W 0 from P 0 .
Revised: Recover W 0 from P 0 .

Original: ∀ ( s, t ) ∈ [ c ] × [ c ] , inverse 2D DFT is on P 

Revised: ∀ ( s, t ) ∈ [ c ] × [ c ] , inverse 2D DFT is on P

Original: [: , : , s, t ] to obtain W 

Revised: [: , : , s, t ] to obtain W

Original: [: , : , s, t ] :
Revised: [: , : , s, t ] :

Original: W
Revised: W

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: : , s, t ] =
Revised: : , s, t ] =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As W 0 is the convolution kernel for the further convolution, to make the convolution compatible with mainstream real-value convolutional neural networks, we should keep W 0 real.
Revised: As W 0 is the convolution kernel for the further convolution, to make the convolution compatible with mainstream real-value convolutional neural networks, we should keep W 0 real.

Original: Based on the property of DFT, for inverse 2D discrete Fourier transform, the output Y ∈ R k
Revised: Based on the property of DFT, for inverse 2D discrete Fourier transform, the output Y ∈ R k

Original: ×
Revised: ×

Original: k
Revised: k

Original: if purely real if and only if the input X is conjugate-symmetry, that is, X
Revised: if purely real if and only if the input X is conjugate-symmetry, that is, X

Original: [ i, j ] = X ∗ [( k − i )
Revised: [ i, j ] = X ∗ [( k − i )

Original: k, ( k − j )% k ] for each ( i, j ) ∈
Revised: k, ( k − j )% k ] for each ( i, j ) ∈

Original: [ k ] ×
Revised: [ k ] ×

Original: [ k ] .
Revised: [ k ] .

Original: Thus, to make W 0 real, we need set P 

Revised: Thus, to make W 0 real, we need set P

Original: [ i, j, : , :] :=
Revised: [ i, j, : , :] :=

Original: P ∗ 

Revised: P ∗

Original: [( k − i )
Revised: [( k − i )

Original: k, ( k − j )% k, : , :] for each ( i, j ) ∈
Revised: k, ( k − j )% k, : , :] for each ( i, j ) ∈

Original: [ k ] ×
Revised: [ k ] ×

Original: [ k ] .
Revised: [ k ] .

Original: Since P 0 is real-value, we just need to set P 

Revised: Since P 0 is real-value, we just need to set P

Original: [ i, j, : , :] and
Revised: [ i, j, : , :] and

Original: P 

Revised: P

Original: [( k − i )
Revised: [( k − i )

Original: k, ( k − j )% k, : , :] identical.
Revised: k, ( k − j )% k, : , :] identical.

Original: Due to the conjugate-symmetry settings, among these kmatrices { P 

Revised: Due to the conjugate-symmetry settings, among these kmatrices { P

Original: [( i, j, : , :] } k,ki =1 ,j =1 , there are only L different matrices.
Revised: [( i, j, : , :] } k,ki =1 ,j =1 , there are only L different matrices.

Original: L is ( k 2 + 1) / 2 when k is odd and ( k 2 +4) / 2 when k is even.
Revised: L is ( k 2 + 1) / 2 when k is odd and ( k 2 +4) / 2 when k is even.

Original: In the implementation, we encode all the information of P 0 through a L × c × c tensor ˆ P 0 .
Revised: In the implementation, we encode all the information of P 0 through a L × c × c tensor ˆ P 0 .

Original: Then we build P 0 from ˆ P 0 by a mapping matrix
Revised: Then we build P 0 from ˆ P 0 by a mapping matrix

Original: I ∈ R k × k :
Revised: I ∈ R k × k :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: I
Revised: I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , : , :] , ∀ ( i,
Revised: , : , :] , ∀ ( i,

Original: [Equation]
Revised: [Equation]

Original: ∈ [ k ] ×
Revised: ∈ [ k ] ×

Original: [ k ] , (17) where I is a mapping matrix, which maps a pair ( i, j ) ∈ [ k ] ×
Revised: [ k ] , (17) where I is a mapping matrix, which maps a pair ( i, j ) ∈ [ k ] ×

Original: [ k ] to a scalar l ∈ [1 , L ] and it satisﬁes
Revised: [ k ] to a scalar l ∈ [1 , L ] and it satisﬁes

Original: [Equation]
Revised: [Equation]

Original: The details of constructing the mapping matrix I is provided in Algorithm 2 of Appendix A.
Revised: The details of constructing the mapping matrix I is provided in Algorithm 2 of Appendix A.

Original: Dilation convolution.
Revised: Dilation convolution.

Original: In this step, we conduct the standard dilated convolution using the recovered kernel W 0 obtained in Eq.
Revised: In this step, we conduct the standard dilated convolution using the recovered kernel W 0 obtained in Eq.

Original: (16) with a dilation n/k on the input X and obtain the output Y .
Revised: (16) with a dilation n/k on the input X and obtain the output Y .

Original: We summarize the proposed model in Algorithm 1.
Revised: We summarize the proposed model in Algorithm 1.

Original: It is worth noting that, in each training iteration, given a batch of B samples, the lines 1-13 are only conducted once, and the obtained W 0 is shared across B samples.
Revised: It is worth noting that, in each training iteration, given a batch of B samples, the lines 1-13 are only conducted once, and the obtained W 0 is shared across B samples.

Original: When the training ﬁnishes, we store the computed W 0 in line 12 in the model.
Revised: When the training ﬁnishes, we store the computed W 0 in line 12 in the model.

Original: In evaluation, we directly load the stored W 0 and only need to compute the operation in line 14, which is just vanilla dilated convolution and thus is very efﬁcient.
Revised: In evaluation, we directly load the stored W 0 and only need to compute the operation in line 14, which is just vanilla dilated convolution and thus is very efﬁcient.

Original: In the current conﬁguration of Algorithm 1, we set P 

Revised: In the current conﬁguration of Algorithm 1, we set P

Original: [ p, q, : , :] ∈ R c
Revised: [ p, q, : , :] ∈ R c

Original: × c to be real matrix.
Revised: × c to be real matrix.

Original: In fact, P 0 [ p, q, : , :] can also be complex matrix.
Revised: In fact, P 0 [ p, q, : , :] can also be complex matrix.

Original: To make the output of the proposed orthogonal convolution compatible with existing mainstream real-value neural network, the only constraint is to make W 

Revised: To make the output of the proposed orthogonal convolution compatible with existing mainstream real-value neural network, the only constraint is to make W

Original: [: , : , i, j ] = IDFT 2D P 

Revised: [: , : , i, j ] = IDFT 2D P

Original: [: , : , i, j ] a real matrix.
Revised: [: , : , i, j ] a real matrix.

Original: That is, the only constraint is to set
Revised: That is, the only constraint is to set

Original: P 

Revised: P

Original: [: , : , i, j ] to be conjugate symmetric and thus P 

Revised: [: , : , i, j ] to be conjugate symmetric and thus P

Original: [: , : , i, j ] is unnecessarily real.
Revised: [: , : , i, j ] is unnecessarily real.

Original: When considering
Revised: When considering

Original: P 

Revised: P

Original: [ p, q, : , :] as a complex matrix, it makes the constructed orthogonal convolution kernel W 0 more complete.
Revised: [ p, q, : , :] as a complex matrix, it makes the constructed orthogonal convolution kernel W 0 more complete.

Original: Meanwhile, it brings more computational cost in training.
Revised: Meanwhile, it brings more computational cost in training.

Original: In Appendix B, we compare with using a complex P 

Revised: In Appendix B, we compare with using a complex P

Original: [ p, q, : , :] .
Revised: [ p, q, : , :] .

Original: [Table]
Revised: [Table]

Original:  I MPLEMENTATION
Revised:  I MPLEMENTATION

Original: Spectral normalization.
Revised: Spectral normalization.

Original: For a small approximation error from Taylor expansion, we conduct spectral normalization on P 

Revised: For a small approximation error from Taylor expansion, we conduct spectral normalization on P

Original: [ p, q, : , :] to make (cid:107) P 

Revised: [ p, q, : , :] to make (cid:107) P

Original: [ p, q, : , :] (cid:107) 2 ≤ 1 .
Revised: [ p, q, : , :] (cid:107) 2 ≤ 1 .

Original: Following Miyato et al.
Revised: Following Miyato et al.

Original: Singla & Feizi (2021a;b), we use the power method to estimate the approximated spectral norm and then divide P 

Revised: Singla & Feizi (2021a;b), we use the power method to estimate the approximated spectral norm and then divide P

Original: [ p, q, : , :] by the estimated spectral norm.
Revised: [ p, q, : , :] by the estimated spectral norm.

Original: We use the mix-precision training supported by the APEX package for faster training and less GPU memory cost.
Revised: We use the mix-precision training supported by the APEX package for faster training and less GPU memory cost.

Original: Padding.
Revised: Padding.

Original: Following SOC (Singla & Feizi, 2021b), we use cyclic padding to substitute zeropadding since zero-padded orthogonal convolutions must be size 1 (see proof in Appendix D.2 of
Revised: Following SOC (Singla & Feizi, 2021b), we use cyclic padding to substitute zeropadding since zero-padded orthogonal convolutions must be size 1 (see proof in Appendix D.2 of

Original: SOC (Singla & Feizi, 2021b)).
Revised: SOC (Singla & Feizi, 2021b)).

Original: To be speciﬁc, for a dilated convolution of k × k size with a dilation d , we add d ( k − 1) / 2 paddings on each side in a cyclic manner.
Revised: To be speciﬁc, for a dilated convolution of k × k size with a dilation d , we add d ( k − 1) / 2 paddings on each side in a cyclic manner.

Original: Input and output channels.
Revised: Input and output channels.

Original: The proposed orthogonal convolution is devised under the condition that the number of channels in input is equal to that in output, that is, c in = c out = c .
Revised: The proposed orthogonal convolution is devised under the condition that the number of channels in input is equal to that in output, that is, c in = c out = c .

Original: When c in (cid:54) =
Revised: When c in (cid:54) =

Original: c out , following SOC (Singla & Feizi, 2021b), we additionally conduct the following operations:
Revised: c out , following SOC (Singla & Feizi, 2021b), we additionally conduct the following operations:

Original: if c in > c out , we apply our orthogonal convolution with a kernel size of k × k × c in × c in , generating the output with c in channels.
Revised: if c in > c out , we apply our orthogonal convolution with a kernel size of k × k × c in × c in , generating the output with c in channels.

Original: We only select the ﬁrst c out channels in the output.
Revised: We only select the ﬁrst c out channels in the output.

Original: if c in < c out , we use zero pad the input with c out − c in channels.
Revised: if c in < c out , we use zero pad the input with c out − c in channels.

Original: Then we apply our orthogonal convolution with a kernel size of k × k × c out × c out .
Revised: Then we apply our orthogonal convolution with a kernel size of k × k × c out × c out .

Original: Strided convolution.
Revised: Strided convolution.

Original: Given an input X ∈ R n × n × c in , a convolution with stride s generates the output Y ∈ R n s × n s × c out .
Revised: Given an input X ∈ R n × n × c in , a convolution with stride s generates the output Y ∈ R n s × n s × c out .

Original: The proposed orthogonal convolution is designed for 1-stride convolution.
Revised: The proposed orthogonal convolution is designed for 1-stride convolution.

Original: For a more general s -stride convolution, we decompose it into a down-sampling module and an orthogonal convolution, following BCOP (Li et al., 2019) and SOC (Singla & Feizi, 2021b).
Revised: For a more general s -stride convolution, we decompose it into a down-sampling module and an orthogonal convolution, following BCOP (Li et al., 2019) and SOC (Singla & Feizi, 2021b).

Original: Specifically, the down-sampling module adapts the invertible down-sampling (Maduranga et al., 2019), which simply reshapes the input feature map X from R n × n × c in to R ns × ns × c in s 2 .
Revised: Specifically, the down-sampling module adapts the invertible down-sampling (Maduranga et al., 2019), which simply reshapes the input feature map X from R n × n × c in to R ns × ns × c in s 2 .

Original: The invertible down-sampling decreases the spatial size of the feature map from k × k to k/s × k/s and meanwhile strictly preserves the gradient norm.
Revised: The invertible down-sampling decreases the spatial size of the feature map from k × k to k/s × k/s and meanwhile strictly preserves the gradient norm.

Original: Then we conduct the proposed 1-stride orthogonal convolution on the reshaped X .
Revised: Then we conduct the proposed 1-stride orthogonal convolution on the reshaped X .

Original: 1-Lipschitz architecture.
Revised: 1-Lipschitz architecture.

Original: Following SOC (Singla & Feizi, 2021b), we build a provably 1-Lipschitz architecture based on the proposed orthogonal convolution.
Revised: Following SOC (Singla & Feizi, 2021b), we build a provably 1-Lipschitz architecture based on the proposed orthogonal convolution.

Original: For a fair comparison with SOC, our structure is the same as SOC, and we only replace the orthogonal convolution in SOC with the proposed orthogonal convolution.
Revised: For a fair comparison with SOC, our structure is the same as SOC, and we only replace the orthogonal convolution in SOC with the proposed orthogonal convolution.

Original: The details of the architecture are given in Table 1.
Revised: The details of the architecture are given in Table 1.

Original: It consists of
Revised: It consists of

Original: L convolution layers in 5 blocks, and each block contains L/ 5 layers.
Revised: L convolution layers in 5 blocks, and each block contains L/ 5 layers.

Original: Each block consists of two groups of convolutions layers.
Revised: Each block consists of two groups of convolutions layers.

Original: The ﬁrst group repeats L/ 5 − 1 convolutions layers with stride 1 , and the second group contains a single convolution layer with stride 2 , and the second group contains a single convolution layer with stride 2 .
Revised: The ﬁrst group repeats L/ 5 − 1 convolutions layers with stride 1 , and the second group contains a single convolution layer with stride 2 , and the second group contains a single convolution layer with stride 2 .

Original: Following SOC, a MaxMin activation function (Anil et al., 2019) is added after each convolution layer.
Revised: Following SOC, a MaxMin activation function (Anil et al., 2019) is added after each convolution layer.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Settings.
Revised: Settings.

Original: The training is conducted on a single NVIDIA V100 GPU with 32G memory.
Revised: The training is conducted on a single NVIDIA V100 GPU with 32G memory.

Original: Following Singla & Feizi (2021b), the whole training process takes 200 epochs for all models.
Revised: Following Singla & Feizi (2021b), the whole training process takes 200 epochs for all models.

Original: The initial learning rate is 0 .
Revised: The initial learning rate is 0 .

Original: 1 when the number of convolution layers is larger than 25 and 0 .
Revised: 1 when the number of convolution layers is larger than 25 and 0 .

Original: otherwise, and it decreases by a factor of 0.1 at the 50th and the 150th epoch.
Revised: otherwise, and it decreases by a factor of 0.1 at the 50th and the 150th epoch.

Original: We set the weight decay as 0 when training BCOP (Li et al., 2019) and 5e-4 when training SOC (Singla & Feizi, 2021b) and our ECO.
Revised: We set the weight decay as 0 when training BCOP (Li et al., 2019) and 5e-4 when training SOC (Singla & Feizi, 2021b) and our ECO.

Original: By default, we set the batch size as 256 on all models in training.
Revised: By default, we set the batch size as 256 on all models in training.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We evaluate the performance proposed orthogonal convolution by two metrics: standard recognition accuracy and the robust recognition accuracy.
Revised: We evaluate the performance proposed orthogonal convolution by two metrics: standard recognition accuracy and the robust recognition accuracy.

Original: The standard recognition
Revised: The standard recognition

Original: Model
Revised: Model

Original: [Table]
Revised: [Table]

Original: LipConvnet
Revised: LipConvnet

Original: [Table] [Table]
Revised: [Table] [Table]

Original: LipConvnet
Revised: LipConvnet

Original: [Table] [Table]
Revised: [Table] [Table]

Original: LipConvnet
Revised: LipConvnet

Original: [Table] [Table]
Revised: [Table] [Table]

Original: accuracy is the same as that used in classic image classiﬁcation.
Revised: accuracy is the same as that used in classic image classiﬁcation.

Original: In contrast, the robust recognition accuracy is the accuracy under an l 2 perturbation.
Revised: In contrast, the robust recognition accuracy is the accuracy under an l 2 perturbation.

Original: As shown in Section 3.3, L -Lipschitz network has the robustness certiﬁcate when the (cid:96) 2 -norm perturbation (cid:15) is less than M n ( x ) / √ 2 L .
Revised: As shown in Section 3.3, L -Lipschitz network has the robustness certiﬁcate when the (cid:96) 2 -norm perturbation (cid:15) is less than M n ( x ) / √ 2 L .

Original: Thus, for a 1 -Lipschitz we built based on the proposed ECO convolution has the robustness certiﬁcate as
Revised: Thus, for a 1 -Lipschitz we built based on the proposed ECO convolution has the robustness certiﬁcate as

Original: M n ( x ) / √ 2 ≥ (cid:15) .
Revised: M n ( x ) / √ 2 ≥ (cid:15) .

Original: Following Li et al.
Revised: Following Li et al.

Original: and Singla & Feizi (2021b), when testing the robust accuracy, we set the default norm of perturbation, (cid:15) , as 36 / 255 .
Revised: and Singla & Feizi (2021b), when testing the robust accuracy, we set the default norm of perturbation, (cid:15) , as 36 / 255 .

Original: In this case, the robust accuracy is the ratio of samples satisfying M n ( x ) / √ 2 ≥ 36 / 255 .
Revised: In this case, the robust accuracy is the ratio of samples satisfying M n ( x ) / √ 2 ≥ 36 / 255 .

Original: Standard and robust recognition accuracy.
Revised: Standard and robust recognition accuracy.

Original: Table 2 compares the recognition accuracy of the proposed ECO convolution with BCOP (Li et al., 2019) and SOC (Singla & Feizi, 2021b) on CIFAR and CIFAR100 datasets.
Revised: Table 2 compares the recognition accuracy of the proposed ECO convolution with BCOP (Li et al., 2019) and SOC (Singla & Feizi, 2021b) on CIFAR and CIFAR100 datasets.

Original: As we can observe from the table that our ECO convolution achieves competitive performance compared with BCOP and SOC in both standard and robust recognition accuracy.
Revised: As we can observe from the table that our ECO convolution achieves competitive performance compared with BCOP and SOC in both standard and robust recognition accuracy.

Original: Speciﬁcally, on CIFAR-10 dataset, when the number of convolution layers, L , is larger than 25 , ours outperforms both SOC and BCOP in standard and robust recognition accuracy.
Revised: Speciﬁcally, on CIFAR-10 dataset, when the number of convolution layers, L , is larger than 25 , ours outperforms both SOC and BCOP in standard and robust recognition accuracy.

Original: Meanwhile, on CIFAR-100 dataset, our ECO convolution achieves higher robust accuracy than SOC in all models from LipConvnet-5 to LipConvnet-40.
Revised: Meanwhile, on CIFAR-100 dataset, our ECO convolution achieves higher robust accuracy than SOC in all models from LipConvnet-5 to LipConvnet-40.

Original: [Table]
Revised: [Table]

Original: Training and evaluation time.
Revised: Training and evaluation time.

Original: As shown in the last column of Table 2, SOC is slightly more efﬁcient than our ECO convolution in training.
Revised: As shown in the last column of Table 2, SOC is slightly more efﬁcient than our ECO convolution in training.

Original: It is because the dilation convolution in our ECO adds more padding, making the input feature map larger than that in SOC.
Revised: It is because the dilation convolution in our ECO adds more padding, making the input feature map larger than that in SOC.

Original: Meanwhile, as shown in Table 3, our ECO convolution is signiﬁcantly faster than SOC in evaluation.
Revised: Meanwhile, as shown in Table 3, our ECO convolution is signiﬁcantly faster than SOC in evaluation.

Original: It is due to that we have already constructed the ECO convolution before evaluation for once.
Revised: It is due to that we have already constructed the ECO convolution before evaluation for once.

Original: Thus, in evaluation, the computational cost of our ECO convolution is the same as standard convolution.
Revised: Thus, in evaluation, the computational cost of our ECO convolution is the same as standard convolution.

Original: In contrast, SOC needs to take T times convolution operations per convolution layer to approximate the orthogonal convolution, which takes T times the computational cost as standard convolution.
Revised: In contrast, SOC needs to take T times convolution operations per convolution layer to approximate the orthogonal convolution, which takes T times the computational cost as standard convolution.

Original: By default, SOC sets T = 5 in training and T = 10 in evaluation.
Revised: By default, SOC sets T = 5 in training and T = 10 in evaluation.

Original: As shown in Table 3, in LipConvnet-40, ours ECO convolution achieves an around 6 × speed-up ratio over SOC.
Revised: As shown in Table 3, in LipConvnet-40, ours ECO convolution achieves an around 6 × speed-up ratio over SOC.

Original: Robust accuracy with respect to distortion.
Revised: Robust accuracy with respect to distortion.

Original: As mentioned, the 1-Lipschitz network has the robustness certiﬁcate as M n ( x ) / √ 2 ≤ (cid:15) and the testing accuracy is the ratio of samples satisfying
Revised: As mentioned, the 1-Lipschitz network has the robustness certiﬁcate as M n ( x ) / √ 2 ≤ (cid:15) and the testing accuracy is the ratio of samples satisfying

Original: M n ( x ) / √ 2 ≤ (cid:15) .
Revised: M n ( x ) / √ 2 ≤ (cid:15) .

Original: We evaluate the robust accuracy of LipConvnet-25 when the distortion norm (cid:15) adapts different values.
Revised: We evaluate the robust accuracy of LipConvnet-25 when the distortion norm (cid:15) adapts different values.

Original: We compare with SOC and the robust accuracies of SOC under different (cid:15) are copied from Singla et al.
Revised: We compare with SOC and the robust accuracies of SOC under different (cid:15) are copied from Singla et al.

Original: (2021) using the same MaxMin activation function.
Revised: (2021) using the same MaxMin activation function.

Original: As shown in
Revised: As shown in

Original: Table 4, our ECO convolution consistently outperforms ECO under different (cid:15) and the advantage is more signiﬁcant for a larger (cid:15) .
Revised: Table 4, our ECO convolution consistently outperforms ECO under different (cid:15) and the advantage is more signiﬁcant for a larger (cid:15) .

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work investigates building the convolution layer with an orthogonal Jacobian matrix, which is 1 -Lipschitz in the 2-norm and preserves the gradient.
Revised: This work investigates building the convolution layer with an orthogonal Jacobian matrix, which is 1 -Lipschitz in the 2-norm and preserves the gradient.

Original: By exploiting the relation between the convolution kernel and the Jacobian matrix, we explicitly construct the convolution kernel to achieve the orthogonal convolution.
Revised: By exploiting the relation between the convolution kernel and the Jacobian matrix, we explicitly construct the convolution kernel to achieve the orthogonal convolution.

Original: The explicitly constructed orthogonal convolution (ECO) takes the exact computational cost as a standard dilated convolution in evaluation, which is more efﬁcient than the existing state-of-the-art method, SOC, approximating the orthogonal convolution through multiple times convolution operations.
Revised: The explicitly constructed orthogonal convolution (ECO) takes the exact computational cost as a standard dilated convolution in evaluation, which is more efﬁcient than the existing state-of-the-art method, SOC, approximating the orthogonal convolution through multiple times convolution operations.

Original: Our experiments on CIFAR-10 and CIFAR-100 show that our ECO achieves competitive standard and robust accuracy as SOC using much less evaluation time.
Revised: Our experiments on CIFAR-10 and CIFAR-100 show that our ECO achieves competitive standard and robust accuracy as SOC using much less evaluation time.

Original: A C ONSTRUCTING THE MAPPING MATRIX I
Revised: A C ONSTRUCTING THE MAPPING MATRIX I

Original: In Algorithm 2, we give the details of constructing the mapping matrix I .
Revised: In Algorithm 2, we give the details of constructing the mapping matrix I .

Original: B C OMPARISONS WITH THE COMPLEX SETTINGS
Revised: B C OMPARISONS WITH THE COMPLEX SETTINGS

Original: As we mentioned in Section 4, to obtain a real-value convolution kernel W 0 , the entries in P 0 are unnecessarily real.
Revised: As we mentioned in Section 4, to obtain a real-value convolution kernel W 0 , the entries in P 0 are unnecessarily real.

Original: We compare the performance when P 0 is constrained to be real as Algorithmand that when P 0 is a complex tensor.
Revised: We compare the performance when P 0 is constrained to be real as Algorithmand that when P 0 is a complex tensor.

Original: As shown in Table 5, on CIFAR-10 benchmark, using complex P 0 , it consistently outperforms its real counterpart on both standard accuracy and robust accuracy.
Revised: As shown in Table 5, on CIFAR-10 benchmark, using complex P 0 , it consistently outperforms its real counterpart on both standard accuracy and robust accuracy.

Original: The accuracy gap between the complex P 0 and its real counterpart is large for shallow networks and turns smaller as the network goes deeper.
Revised: The accuracy gap between the complex P 0 and its real counterpart is large for shallow networks and turns smaller as the network goes deeper.

Original: It is worth noting that the computational cost of multiplication between complex matrices is four times as that between real matrices of the same sizes.
Revised: It is worth noting that the computational cost of multiplication between complex matrices is four times as that between real matrices of the same sizes.

Original: But the complex P 0 is only used for constructing the real orthogonal kernel W 0 .
Revised: But the complex P 0 is only used for constructing the real orthogonal kernel W 0 .

Original: After kernel construction, the computation cost of convolution is the same for the layer with complex Pand that with real P 0 .
Revised: After kernel construction, the computation cost of convolution is the same for the layer with complex Pand that with real P 0 .

Original: Thus, as shown in Table 5, the training time per epoch when using complex
Revised: Thus, as shown in Table 5, the training time per epoch when using complex

Original: P 0 is around 2 to 3 times as that using real P 0 .
Revised: P 0 is around 2 to 3 times as that using real P 0 .

Original: In evaluation, since we have constructed the real
Revised: In evaluation, since we have constructed the real

Original: W 0 from a real or complex P 0 before evaluation, the computational cost is the same no matter Pis complex and real.
Revised: W 0 from a real or complex P 0 before evaluation, the computational cost is the same no matter Pis complex and real.

Original: [Table]
Revised: [Table]

Original: C C OMPARISONS WITH THE STANDARD CONVOLUTION
Revised: C C OMPARISONS WITH THE STANDARD CONVOLUTION

Original: Here, we compare the proposed orthogonal convolution with the standard convolution without orthogonal constraint.
Revised: Here, we compare the proposed orthogonal convolution with the standard convolution without orthogonal constraint.

Original: To make a fair comparison, we only replace the orthogonal convolution layers of LipConvnet-5 by the standard convolution and keep the other layers unﬁxed.
Revised: To make a fair comparison, we only replace the orthogonal convolution layers of LipConvnet-5 by the standard convolution and keep the other layers unﬁxed.

Original: Since the network built upon the standard convolution cannot preserve the 1-Lipschitz property, we cannot directly obtain the robust accuracy based on M n ( x ) as the original LipConvnet-5 network.
Revised: Since the network built upon the standard convolution cannot preserve the 1-Lipschitz property, we cannot directly obtain the robust accuracy based on M n ( x ) as the original LipConvnet-5 network.

Original: Thus, we utilize
Revised: Thus, we utilize

Original: Projected Gradient Descent (PGD) attack (Madry et al., 2017) to obtain the robust accuracy upper bound under the distortion with the (cid:96) 2 norm less than 36255 .
Revised: Projected Gradient Descent (PGD) attack (Madry et al., 2017) to obtain the robust accuracy upper bound under the distortion with the (cid:96) 2 norm less than 36255 .

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: As shown in the table, on both CIFAR10 and CIFAR10 datasets, the network based on the standard con- volution achieves higher standard accuracy than the network based on our ECO convolution.
Revised: As shown in the table, on both CIFAR10 and CIFAR10 datasets, the network based on the standard con- volution achieves higher standard accuracy than the network based on our ECO convolution.

Original: In the meanwhile, ours achieves higher robust accuracy on both datasets under the same scale of distortion.
Revised: In the meanwhile, ours achieves higher robust accuracy on both datasets under the same scale of distortion.

Original: [Table]
Revised: [Table]

Original: D S TANDARD T RAINING ON R ES N ET
Revised: D S TANDARD T RAINING ON R ES N ET

Original: We conduct experiments based on the ResNet-18, which is one of mainstream backbones in computer vision.
Revised: We conduct experiments based on the ResNet-18, which is one of mainstream backbones in computer vision.

Original: To be speciﬁc, we replace the standard convolution in ResNet-18 with the orthogonal convolutions based on BCOP, SOC and ECO.
Revised: To be speciﬁc, we replace the standard convolution in ResNet-18 with the orthogonal convolutions based on BCOP, SOC and ECO.

Original: As shown in Table 7, the orthogonal convolution can not achieve the high recognition accuracy as its standard counterpart.
Revised: As shown in Table 7, the orthogonal convolution can not achieve the high recognition accuracy as its standard counterpart.

Original: But the advantage of orthogonal convolution layers is their provable robustness in LipConvNet, making them still attractive in scenarios when the recognition robustness is important.
Revised: But the advantage of orthogonal convolution layers is their provable robustness in LipConvNet, making them still attractive in scenarios when the recognition robustness is important.

Original: [Table]
Revised: [Table]

Original: E T RUNCATED T AYLOR E XPANSION
Revised: E T RUNCATED T AYLOR E XPANSION

Original: Note that, the exact matrix exponential (Casado & Mart´ınez-Rubio, 2019)
Revised: Note that, the exact matrix exponential (Casado & Mart´ınez-Rubio, 2019)

Original: exp( A ) relies on singular value decomposition (SVD), which is not friendly for GPU computing platform.
Revised: exp( A ) relies on singular value decomposition (SVD), which is not friendly for GPU computing platform.

Original: Thus, we adopt the truncated Taylor Expansion (Singla & Feizi, 2021b)
Revised: Thus, we adopt the truncated Taylor Expansion (Singla & Feizi, 2021b), S T ( A ) =

Revised: (cid:80)

Original: S T ({+Ti =
+} A ) to approximate the exact matrix exponential exp( A ) .
Revised: {+Ti =
+} A i i ! , to approximate the exact matrix exponential exp( A ) .

Original: The truncation error R = exp( A ) − S T ( A ) satisﬁes 
Revised: The truncation error R = exp( A ) − S T ( A ) satisﬁes (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: Below we ﬁrst compare the time cost in GPU platform using the exact matrix exponential exp( A ) and that based on the truncated Taylor Expansion S T ( A ) .

Revised: To be speciﬁc, we implement exp( A ) simply based on the torch.matrix exp.

Revised: We measure the time cost on GPU using the function torch.cuda.

Revised: Event and we set T = 10 in truncated Taylor Expansion S T ( A ) .

Revised: As shown in Table 8, the latency based on the truncated Taylor Expansion is signiﬁcantly less than that based on the exact matrix exponential.

Revised: [Table]

Original: Below we show the scale of the average factual truncated error R with respect to T .
Revised: Then we show the scale of the average factual truncated error R with respect to T .

Original: It is based on the trained LipConvnet-5 network built upon the proposed ECO convolution.
Revised: It is based on the trained LipConvnet-5 network built upon the proposed ECO convolution.

Original: By default, we set T = 5 in training and T = 10 in testing.
Revised: By default, we set T = 5 in training and T = 10 in testing.

Original: As shown in Table 8, the scale of the truncation error is relatively small when T is 5 or 10 .
Revised: As shown in Table 9, the scale of the truncation error is relatively small when T is 5 or 10 .

Revised: Meanwhile, we also show the inﬂuence of the truncation error on the orthogonality of the convolution.

Revised: Following Trockman & Kolter (2021), we measure the orthogonality of a convolution layer through (cid:107) vec

Revised: [Equation]

Revised: − 1 , and a strict orthogonal convolution layer should satisfy

Original: [Table]
Revised: [Equation]

Revised: We estimate the deviation of the proposed ECO convolution from the strict orthogonal convolution through this measurement.

Revised: As shown in Table 9, when T is 5 or 10 , the deviation of our ECO convolution from the strict orthogonal convolution is relative marginal.

Revised: [Table]