Towards Practical Control of Singular Values of Convolutional Layers
Abstract
In general, convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control.
Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties and offered several methods for controlling them.
Nevertheless, these methods present an intractable computational challenge or resort to coarse approximations.
In this paper, we offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity.
Our method is based on the tensor-train decomposition; it retains control over the actual singular values of convolutional mappings while providing structurally sparse and hardware-friendly representation.
We demonstrate the improved properties of modern CNNs with our method and analyze its impact on the model performance, calibration, and adversarial robustness.
The source code is available at: https://github.com/WhiteTeaDragon/practical_svd_conv
Introduction
Over the past decade, empirical advances in Deep Learning have made machine learning ubiquitous to researchers and practitioners from various fields of science and industry.
However, the theory of
Deep Learning lags behind its practical applications, resulting in unexpected outcomes or lack of explainability of the models.
These adverse effects are becoming more prominent with many models put into customer-facing products, such as perception systems of self-driving cars, chatbots and other applications.
The demand for tighter control over deployed models has given rise to several subfields of Deep Learning, such as the study of adversarial robustness and model calibration, to name a few.
Convolutional neural networks (CNNs) and, in particular, residual CNNs have become a benchmark for various computer vision tasks.
The key component of CNNs is convolutional layers, representing linear transformations that account for the image data structure.
The singular values of these linear transformations are key to the properties of the whole network, such as the Lipschitz constant and, hence, generalization error and robustness to adversarial examples.
Moreover, bounding the Lipschitz constant of convolutional layers can also improve training, as it prevents gradients from exploding.
Nevertheless, finding and controlling singular values of convolutional layers is challenging.
Indeed, computing and then clipping exact singular values of a layer [Sedghi et al., 2019], given by a kernel tensor of the size k × k × c in × c out , has time and space complexities of O ( n 2 c 2 ( c + log n )) and
O ( n 2 c 2 ) respectively, where n × n is the input image size, c in and c out are the numbers of input and output channels respectively, c
= max { c in , c out } , and k × k is the filter size.
Since typically k 2 ≪ c , computing singular values requires many more operations than computing a single convolution with its asymptotic complexity O ( n 2 c 2 k 2 ) .
When computing singular values, the storage of an arising kernel tensor of the size n × n × c in × c out (padded to image dimensions) can also be an issue for larger networks and inputs.
This effect becomes even more pronounced in three-dimensional convolutions used for volumetric data processing.
Alternative methods that parametrize the convolutional layers are also computationally demanding [Singla and Feizi, 2021b, Li et al., 2019].
In this paper, we propose a practical approach to constraining the singular values of the convolutional layers based on intractable techniques from the prior art.
Our approach relies on the assumption that modern over-parameterized neural networks can be made sparse using tensor decompositions, incurring insignificant degradation of the downstream task performance.
We investigate the impact of using our approach on model performance, calibration, and adversarial robustness.
More specifically, our contributions are as follows:
We propose a new framework for reducing the computational complexity of controlling singular values of a convolutional layer by imposing the Tensor-Train (TT) decomposition constraint on a kernel tensor.
It allows for substituting the computation of singular values of the original layer by a smaller layer.
As a result, it reduces the complexity of singular values control by using a method of choice.
We extend the formula of exact singular values of a convolutional layer [Sedghi et al., 2019] to the case of strided convolutions, which is ubiquitous in CNN architectures.
We apply the proposed framework to different methods of controlling singular values and test them on several CNN architectures.
Contrary to measuring just the downstream task performance as in the prior art, we additionally demonstrate improvements in adversarial robustness and model calibration of the networks.
The paper is organized as follows:
Sec. 2 provides an overview of prior art on singular values and
Lipschitz constant estimation, related methods employing these techniques, and assumptions used to develop our approach; Sec.
3 revisits computation of singular values of convolutional layers and describes our approach to dealing with computational complexity; Sec. 4 walks through the steps of our algorithm to computing singular values of compressed convolutional layers; Sec. 5 contains the empirical study of our method; Sec.
6 wraps up the paper.
Related Work 2.1 Computing Singular Values of Convolutions
Traditional discrete convolutional layers used in signal processing and computer vision [LeCun et al., 1989] can be seen from two perspectives.
On the one hand, a d -dimensional convolution linearly maps (potentially overlapping) windows of a c -channel input signal with side k into vectors of c output values ( window map ).
This perspective is related to the one implementation of convolutional layers, decomposing the operation into window extraction termed im2col [Chellapilla et al., 2006], followed by matrix multiplication via GEMM [Blackford et al., 2002].
From this perspective, the transformation is defined by the convolution weight tensor unfolded into a matrix of size k 2 c in × c out ( Conv2D case); its singular values can be computed and controlled through SVD-based schemes over the said unfolding matrix.
For example, Yoshida and Miyato [2017] proposed a method of
Spectral Normalization to stabilize GAN training [Goodfellow et al., 2020].
Their method effectively estimates the largest singular value of the map via the power method.
In a similar vein, Liu et al.
[2019], Obukhov et al.
[2021] explicitly parametrized the SVD decomposition of the mapping to control multiple singular values during training.
Nevertheless, the above perspective lacks the generality of treating the input signal as a whole, which is especially important when chaining maps, such as seen in deep CNNs.
To this end, a discrete convolutional layer can be viewed as a mapping from and onto the space defined by the signal shape ( signal map ).
The map is defined by an N -ly block-circulant matrix, where N is the number of spatial dimensions of the convolution.
A straightforward way to compute singular values of such a matrix by applying a full SVD algorithm is prohibitively expensive, even for small numbers of channels, as well as signal and window sizes.
To overcome this issue, Sedghi et al.
[2019] proposed a method based on the Fast Fourier Transform (FFT) for computing exact singular values of convolutional layers that has a much better complexity than the naive approach.
Nevertheless, it is still quite demanding in space and time complexities, as it deals with a padded kernel of the size n × n × c × c .
Alternatively, Singla and Feizi [2021b] directly impose nearly orthogonal constraints on convolutional layers using Taylor expansion of a matrix exponential of a skew-Hermitian matrix, which appears to be more efficient than a Block Convolution Orthogonal Parameterization approach of Li et al.
[2019] proposed earlier.
Singla and Feizi [2021a] drew connections between both perspectives and proved that the Lipschitz constant of the window map could serve as a genuine upper bound of the Lipschitz constant of the signal map if multiplied by a certain constant.
2.2 Effects of Singular Values
The study of singular values and the closely related Lipschitzness of convolutional neural networks impacts many domains of Deep Learning research.
One essential expectation about neural networks is to generalize input data instead of memorizing it.
Bartlett et al.
[2017] present the generalization bound that depends on the product of the largest singular values of layers’ Jacobians.
Gouk et al.
[2021] performed a detailed empirical study of the influence of bounding individual Lipschitz constants of each layer and its effect on the generalization error of the whole CNN.
Singular values of Jacobians greater or smaller than 1 .
can also be responsible for the growth or decay of gradients.
Therefore, controlling all singular values can also be helpful to avoid exploding or vanishing gradients.
The decay of singular values in layers also plays a role in the performance of GAN generators [Liu et al., 2019].
Apart from boosting the accuracy of predictions, researchers have been working on improving the robustness of neural networks to adversarial attacks, which is also affected by the Lipschitz constant [Cisse et al., 2017].
Adversarial attacks aim to find a negligible (to human perception) perturbation of input data that would sabotage the predictions of a model.
In this paper, we use the
AutoAttack Robust Accuracy module [Croce and Hein, 2020] to evaluate results.
We additionally use a metric of classification model calibration (Expected Calibration Error, ECE)
[Naeini et al., 2015].
Connections between model Lipschitz constraints, calibration, and out-of-distribution (OOD) model performance have been drawn in recent literature [Postels et al., 2022].
Various training techniques can also affect model properties [Kodryan et al., 2022] and efficiency of training [Gusak et al., 2022].
2.3 Tensor Decompositions in Deep Learning
Our proposed framework for CNN weights reparameterization relies on the well-studied property of neural network overparameterization [Frankle and Carbin, 2019].
Many network compression approaches [Anwar et al., 2017, Lee et al., 2019] agree that it is often possible to approach the uncompressed model performance by imposing some form of sparsity on the network weights.
Tensor decompositions have been used previously for compression [Novikov et al., 2015, Garipov et al., 2016, Wang et al., 2018, Obukhov et al., 2020], multitask learning [Kanakis et al., 2020], neural rendering [Obukhov et al., 2022] and reinforcement learning [Sozykin et al., 2022].
The idea of this paper is to utilize tensor decomposition closely related to the SVD decomposition and to take advantage of its sparsity to reduce the complexity of controlling singular values.
For that, we resort to the TT decomposition [Oseledets, 2011], which in the outlined context is also equivalent to the
Tucker-2 decomposition [Kim et al., 2016].
As opposed to alternative tensor decompositions, this one allows us to naturally access singular values of the convolution layer with any method of choice.
Method-independent Complexity Reduction
This section explains how to reduce the complexity of calculating the singular values of a convolutional layer (in the signal map sense explained in Sec. 2.1), regardless of the chosen method.
For the sake of generality, we consider layers acting on ( d +1)
-dimensional input arrays (tensors) with d spatial dimensions and 1 channel dimension.
Of particular interest are cases d = 2 and d = 3 , corresponding to regular images and volumetric data.
We bootstrap our approach based on the recent research on neural network sparsity by compressing convolutional layers with the TT decomposition.
We focus on dealing with the channel-wise complexity of the methods employed.
In what follows, Sec.
3.1 revisits the convolutional operator in neural networks; Sec.
3.2 introduces our principled approach to compressing convolutional layers and reducing the complexity of the problem by a significant margin.
3.1 Regular Convolutional Layers
To introduce a convolutional layer formally, let K ∈ R k ×···× k × c in × c out be a ( d +2) -dimensional kernel tensor, where k is a filter size and c in , c out are the numbers of input and output channels, respectively.
A convolutional layer is given by a linear map C K – multichannel convolution with the kernel tensor K such that C K : R c in × n ×···× n → R c out × f ( n,k ) ×···× f ( n,k ) and
[Equation]
where f ( n, k ) is an integer, depending on the type of the convolution ′ ∗ ′
used and convolution parameters, e.g., strides.
The key assumption needed for our derivations is the bilinearity of ′ ∗ ′ , which covers different convolution types, e.g., linear, periodic, and correlation.
For example, a widely used correlation-type convolution with strides equal to one, reads
[Equation]
for all j = 0 , . . .
, c out − 1 and q α = 0 , . . .
, f ( n, k ) − 1 , α = 1 , . . .
, d , where f ( n, k ) =
n − k + 1 .
In what follows in this section, we write
[Equation]
implying one of the convolution types mentioned above.
Using the linearity of C K , we can rewrite (2) as a matrix-vector product:
[Equation]
where vec is a row-major reshaping of a multidimensional array into a column vector.
In turn,
T K ∈ R c out n d × c in n d is a matrix with an additional block structure: its n d × n d blocks are d -level
Toeplitz matrices (see Sedghi et al.
[2019] for d = 2 ).
Representation (3) allows us to replace singular values of a linear map C K with singular values of a matrix T K , which is handy for analysis.
In turn, access to singular values allows for controlling the Lipschitz constant of C K in terms of the largest singular value of T K , denoted as σ 1 ( T K ) .
Indeed, since σ 1 ( T K ) =
∥ T K ∥ 2 , we have ∥
[Equation]
3.2 Compressed Convolutional Layers
Recall that even if d = 2 , finding the SVD of T K requires O ( n 2 ( c 2 + log n )) FLOPs, which is too much for practical use with large networks.
To reduce the computational cost of SVD of T K , we propose a low-rank compressed layer representation based on the following tensor decomposition:
[Equation] [Equation]
where K (1) ∈ R c in × r 1 , K (3) ∈ R r 2 × c out and K (2) ∈ R k ×···× k × r 1 × r 2 are some small tensors and integers r 1 , r 2 : 1 ≤ r 1 ≤ c in , 1 ≤ r 2 ≤ c out are called ranks.
This decomposition is essentially the
TT decomposition [Oseledets, 2011] of the kernel tensor with filter modes p 1 , . .
. , p d merged into a single index.
We also note that it is equivalent to the so-called Tucker-2 decomposition, successfully used by Kim et al.
[2016] to compress large convolutional networks.
The proposed decomposition is visualized in Fig.
We note that we chose the TT decomposition as it gives us access to the SVD decomposition of the convolution (Theorem 1).
It is, however, unclear how to obtain similar results with other decompositions, such as tensor-ring or canonical tensor decomposition, as they do not admit SVD-like form, see, e.g., [Grasedyck et al., 2013] for more details of these formats.
By substituting (4) into (1), using bilinearity of ′ ∗ ′ and omitting summation limits for clarity:
[Equation]
we obtain that the convolution with kernel given by (4) is equivalent to a sequence of three convolutions without nonlinear activation functions between them:
1. 1 × · · · × 1 convolution with c in input and r 1 output channels (kernel tensor – K (1) ); 2.
k × · · · × k convolution with r 1 input and r 2 output channels (kernel tensor – K (2) ); 3. 1 × · · · × 1 convolution with r 2 input and c out output channels (kernel tensor – K (3) );
or equivalently,
[Equation]
where A ◦ B denotes the composition of two maps A and B ; T K ( i ) denotes 1 × · · · × 1 convolution map given by a convolution kernel K ( i ) .
In the rest of this section, we show that singular values of the original convolutional layer with kernel
K correspond to singular values of the compressed kernel K (2) .
As a first step, we demonstrate that we can impose orthogonality constraints on K (1) and K (3) without loss of expressivity.
Lemma 1.
Let K ∈ R k ×···× k × c in × c out be given by (4) .
Then there exist matrices Q (1) ∈
R c in × r 1 , Q (3) ∈ R r 2 × c out satisfying Q (1) ⊤ Q (1) =
I r 1 , Q (3) Q (3) ⊤ =
I r 2 and a tensor Q (2) ∈
R k ×···× k × r 1 × r 2 , such that
[Equation]
Proof.
See Sec.
A.1 of the Appendix.
Now we have all preliminaries in place to formulate the key result that our framework is based on.
[Equation]
of (4) for d = 3 with the constraints (7) as in Theorem 1.
A node with D “legs” represents a D -dimensional tensor; individual legs represent indices; connected legs represent summation over the respective indices; halffilled nodes are orthogonal matrices.
Theorem 1.
Let K ∈ R k ×···× k × c in × c out be given by (4) .
Let also K (1) ∈
[Equation]
Then the multiset of nonzero singular values of C K defined in (1) equals the multiset of nonzero singular values of C K (2) .
Proof.
See Sec.
A.2 of the Appendix.
Theorem 1 suggests that after forcing the orthogonality of K (1) and K (3) , one only needs to find singular values of the layer corresponding to the smaller kernel tensor K (2) of the size n × n × r 1 × r 2 instead of the original kernel tensor K of the size n × n × r 1 × r 2 .
At the same time, thanks to Lemma 1, one may always set K (1) and K (3) to be orthogonal without additionally losing the expressivity of C K .
Fig.
1 demonstrates the resulting decomposition using tensor diagram notation.
Application of the Proposed Framework
This section discusses how to apply the proposed framework to various methods.
We start with the explicit formula for singular values computation and then discuss how to apply the framework when the convolutions are parametrized, for example, as in [Singla and Feizi, 2021b, Li et al., 2019].
Input:
K (1) ∈ R c in × r 1 – left core tensor of
K (2) ∈
R k × k ×
r 1 × r 2 – middle core tensor of the
-compressed kernel
K (3) ∈
R r 2 × c in – right core tensor of the TT-compressed kernel K , n – input image size, s – stride of the defined by kernel K s.
n ≡ 0 (mod s ) .
Output:
Non-zero singular values of the convolutional layer, defined by the kernel tensor K .
1: Q (1) , R (1) ← QR (cid:0) K (1) (cid:1)
▷ Perform QR decomposition of the left core 2: Q (3) , R (3) ← QR (cid:0) K (3) ⊤ (cid:1)
▷ Perform QR decomposition of the right core 3: (cid:98) K (2) p 1 ,p 2 , : , : ← R (1) K (2) p 1 ,p 2 , : , : R (3) ⊤ , ∀ p 1 , p 2 ▷ Absorb non-orthogonal factors in the middle core 4: (cid:98) K (2) ← pad_zeros (cid:16) (cid:98) K (2) , shape= ( n × n × r 1 × r 2 ) (cid:17)
▷ Pad middle core to the image size 5: R ← reshape (cid:16) (cid:98) K (2) , shape= (8) (cid:17) ▷ Reshape middle core as in (8).
See also Sec.
B.1.
6: P (: , :) ij ← fft2 (cid:0) R j mod s 2 , : , : ,i, ⌊ j/s 2 ⌋ (cid:1) , ∀ i, j ▷ Perform 2 -dimensional FFT of slices of R 7: return (cid:83)
[Equation]
▷ Return a union of all singular values of P ( · , · )
4.1 Explicit Formulas for Strided Convolutions
Sedghi et al.
[2019] consider periodic convolutions, allowing them to express singular values through discrete Fourier transforms and several SVDs.
We extend the main result of their work to the case of strided convolutions and summarize it in the following theorem.
Theorem 2.
Let (cid:98) K ∈ R n × n × c in × c out be a kernel K ∈ R k × k × c in × c out of a periodic convolution, padded with zeros along the filter modes.
Let s be a stride of the convolution C K : n ≡ 0 (mod s ) and let the reshaped kernel R ∈ R s 2 × ns × ns × c in × c out be such that:
[Equation]
Let us consider (cid:0) ns (cid:1) 2 matrices P ( p 1 ,p 2 )
∈ R c in × s 2 c out with entries:
[Equation]
where F is an n × n Fourier matrix.
Then the multiset of singular values of T K is as follows:
[Equation]
Proof.
See Sec.
B.2 of the Appendix.
The algorithm for finding the singular values of a TT-compressed convolutional layer based on
Theorem 2 is summarized in Alg. 1.
This algorithm consists of several major parts.
We start with a layer with the imposed TT structure as in (4).
We reduce this decomposition to the form with orthogonality conditions (7) by using the QR decomposition (steps 1–3).
The second part is the application of Theorem 2 to the smaller kernel tensor (cid:98) K (2) (steps 4–7).
For detailed pseudocode of applying Theorem 2, see Sec.
B.1.
Computing singular values using (9) and (10) has the complexity O ( n 2 c 2 ( cs 2 + log ns )) ,
c = max { c in , c out } , which depends cubically on c .
Thus, reducing c can significantly reduce the computational cost of finding the singular values.
Let us now use the TT-representation (4) for kernel tensor with the ranks r 1 = r 2 = r .
Theorem 1 suggests that we only need to apply Theorem 2 to the core K (
2) ∈
R k × k × r 1 ×
r 2 , which leads to the complexity O ( n 2 r 2 ( r + log ns )) .
For example, r = c/ 2 provides a theoretical speedup of up to 8 times (see Fig.
2 for practical evaluation).
Memory consumption can also be high with the method of Sedghi et al.
Processing the padded kernel tensor of the size n × n × c × c with our method requires 4 times less memory for r =
c/ 2 andtimes less for r =
c/ 3 (see also Fig. 2).
[Figure]
= c/
[Figure]
c/
[Figure]
y , G
[Figure]
r = c/2 r =
c/ r = c/4 full
4.2 Orthogonality Constraints
To ensure that the frame matrices K (1) and K (3) are orthogonal, we apply QR decomposition to the kernel in Alg. 1.
However, we can use other methods to maintain the orthogonality of frame matrices; for example, we can impose regularization on the layers.
Regularization is quite helpful in case K (2) is not a standard convolution but has a special complex structure, such as the SOC layer from [Singla and Feizi, 2021b].
For that, let n l denote the number of TT-decomposed layers in a network, K (1) i and K (3) i – left and right TT-cores of the i th layer, and r 1 ,i , r 2 ,i – layer ranks.
Then, the orthogonal loss reads as:
[Equation]
i =
r 21 ,i + r 22 ,i (cid:33) .
In the case of applying this regularization, we sum this loss with cross-entropy: loss = loss CE + λ ort loss ort .
The value of loss ort is the MSE measure of non-orthogonality of all K (1) i and K (3) i , so the coefficient λ ort tends to be large.
We set it to 1 e 5 for experiments with WideResNets.
Experiments
To study the effects of the proposed regularization on the performance of CNNs, we combined it with various methods of controlling singular values and applied it during training on the image classification task on the CIFAR-10 and CIFAR-100 datasets [Krizhevsky et al., 2009].
The code was implemented in PyTorch [Paszke et al., 2019] and executed on a NVIDIA Tesla V100, 32GB.
In our experiments, we used the LipConvNet architecture [Singla and Feizi, 2021b], a WideResNet [Zagoruyko and
Komodakis, 2016], and VGG-19 [Simonyan and Zisserman, 2015].
Furthermore, the ranks are tested in the range [ c/ 4 , c/ 2] , where c is the largest number of channels among all convolutions (except for 1 × 1 ), which we found to give the better performance/speedup trade-off.
Evaluation Metrics We consider four main metrics: standard accuracy on the test split, accuracy on the CIFAR-C dataset (CC)
[Hendrycks and Dietterich, 2019] 1 , accuracy after applying the AutoAttack module (AA) [Croce and Hein, 2020], and the Expected Calibration Error (ECE)
[Guo et al., 2017].
The last columns of tables with the results of our experiments contain compression ratios of all layers in the network.
In addition, we reported a dedicated metric for the LipConvNet architecture: p.r. – provable bound-norm robustness, introduced by Li et al.
This metric shows the percentage of input images guaranteed to be predicted correctly despite any perturbation within radius 36 / 255 .
[Table]
1.0 76.0 ± 0.
32.6 ± 0.
68.0 ± 0.
5.0 ± 0.
61.7 ± 0.
3.5 77.9 ± 0.
34.7 ± 0.
69.7 ± 0.
4.8 ± 0.
64.8 ± 0.
5.1 Experiments on LipConvNet
Singla and Feizi [2021b] introduced a new architecture called Lipschitz Convolutional Networks, or LipConvNet for short.
LipConvNet- N consists of N convolutional layers and activations.
This architecture is provably 1 -Lipschitz by contrast to popular residual network architectures.
In Singla and Feizi [2021b], the authors also propose an orthogonal layer called Skew Orthogonal Convolution (SOC) and applied it in LipConvNet.
[Figure]
We modify the SOC layer in correspondence with the proposed framework by adding a 1 × 1 × c in × r convolution before applying it, and finally applying the last convolution of size 1 × 1 × r × c out , thus reducing the number of channels to r × r in the bottleneck.
To maintain the orthogonality of this new layer that we call SOC-TT, we add orthogonal loss (Sec. 4.2) to keep these two frame matrices orthogonal.
Tab. 1 demonstrates metrics improvement after replacing SOC with SOC-TT.
For rank 256, we performed a grid search of the λ ort coefficient.
The training protocol is the same as in Singla and Feizi [2021b] except for the learning rate of LipConvNet-20 with the SOC-TT layer, which we set to 0.05 instead of 0.1.
To investigate the observed increase in metrics, we present histograms of empirically estimated Lischitz constants by using adversarial attacks in Fig.
3 (see
Sec. C.1) with optimally chosen λ ort from (12).
We observed that increasing λ ort forces the frame matrices in the TT decomposition to be closer to orthogonal and, as a result, the histogram converges to that of the original SOC method.
At the same time, decreasing λ ort may lead to networks with constants larger than 1 but with better metrics.
5.2 Experiments with WideResNet-16-
Next, we consider WRN-16-10 [Zagoruyko and Komodakis, 2016] with almost 17M parameters.
The results of our experiments on CIFAR-10 are summarised in Tab. 2.
Additional experiments on CIFAR100 are presented in Sec.
C.3.
The experimental setup and training schedule are taken from Gouk et al.
We train our models with batches of 128 elements for 200 epochs, using SGD optimizer and a learning rate of 0.1, multiplied by 0.1 after 60, 120, and 160 epochs.
The hyperparameters for
Both clipping and division combined with TT layers lead to an increase in metrics (except for ECE), with 256 being the best overall rank.
Clipping tends to be a better option in terms of metrics but is substantially slower than the division approach.
Moreover, clipping the singular values to 2 results in better performance than clipping to 1.
“Speedup” is a speedup of the overhead resulting from singular values control in all the layers in the network.
Clipping the whole network w/o decomposition takes 6.2 min while applying division – 0.6 sec.
“Comp.”
(compression) is the ratio between the number of parameters of convolutional layers in the
and decomposed networks
[Table]
the experiments with division were taken from Gouk et al.
The orthogonal loss (11) was used in all experiments with decomposed networks.
Clipping Following Sedghi et al.
[2019], we apply the so-called clipping of singular values after computing the singular values of a layer using Theorem 2.
In particular, we first fix a threshold parameter δ , chosen to be 1 or 2 in numerical experiments.
Then all singular values greater than δ are replaced with δ .
This procedure allows for maintaining maximal singular values at the desired level.
After the clipping, we reconstruct the expanded kernel (cid:98) K δ of the size n × n × c in × c out , which is no longer sparse.
Finally, we revert to its original shape by setting K = (cid:98) K δ (1: k, 1: k, : , :) .
This operation is performed every 100 training iterations; we provide the code for computing singular values in
Sec.
B.1 of the Appendix.
The results suggest that applying TT decomposition not only compresses the network by decreasing the number of parameters of its convolutional layers, but also speeds up the clipping operation.
Network robustness increased when training decomposed layers with clipping, which is confirmed by the results of the AutoAttack module.
Division Even though we can bound the Lipschitz constants of all convolutional layers, the Lipschitz constant of the whole network can be arbitrary.
In particular, for a WideResNet architecture, the
Lipschitz constant would still be unconstrained due to residual connections, fully connected layers, batch normalization, and average pooling layers of the network.
Gouk et al.
[2021] derived a formula for the Lipschitz constants of batch normalization layers and proposed regularizing them too.
After each training step, they set the desired values for Lipschitz constants of convolutional, fully connected, and batch normalization layers.
To control the Lipschitz constant of convolutional and fully connected layers, they first compute the estimate of the largest singular value of a layer via power iteration.
Then they divide the layer by the estimate and multiply it by the desired Lipschitz constant.
A similar operation is performed for batch normalization layers, but the Lipschitz constant is computed directly.
In the original paper, this approach proved to be successful for WideResNet-16-10.
The method itself already increases the robust metrics and accuracy substantially.
The pattern remains intact after applying our decomposition to the layers: the robust accuracy of the AutoAttack module improves.
Overall, the TT decomposition, applied without any other regularization, performs poorly.
However,
C
I F A RAcc.
AA
Cifar-C
ECE
TT TT+clipping TT+division 128 192 full
C
I F
A R128 192 full   128 192 full    128 192 full     when used to facilitate the application of a particular method of control over the singular values, the decomposition only slightly decreases the accuracy, which might still exceed that of the baseline.
5.3 Experiments with VGG-
This section presents the results of controlling singular values on the VGG-19 network and two datasets: CIFAR-10 and CIFAR-100.
As previously, we consider the clipping method with the exact computation of singular values [Sedghi et al., 2019] and the application of one iteration of the power method [Gouk et al., 2021].
The clipping parameter is set to 0 .
The results are presented in Fig.
Firstly, we observe that all the considered robust metrics benefit from controlling singular values.
This effect is more pronounced than for the WRN-16-10 and was also observed in [Gouk et al., 2021].
We conjecture that this happens due to the lack of residual connections in VGG-type architectures, which makes them more sensitive to singular values of convolutional layers; in residual networks, convolutional layers are only corrections to the signal.
In almost all scenarios, we observed that the clipping method performed better than the division with the power method.
Conclusion
We proposed a principled sparsity-driven approach to controlling the singular values of convolutional layers.
We investigated two different families of CNNs and analyzed their performance and robustness under the proposed singular values constraint.
The limitations of our approach are partly inherited from the encapsulated methods of computing singular values (e.g., the usage of periodic convolutions) and partly stem from the core assumption of CNN layer sparsity.
Overall, our approach has proven effective for large-scale networks with hundreds of input and output channels, making it stand out compared to intractable methods from the prior art.
Societal Impact As our work is concerned with improving the robustness of neural networks, it positively impacts their reliability.
Nevertheless, due to the generality of our approach, it can be used in various application domains, including malicious purposes.
Acknowledgments
The publication was supported by the grant for research centers in the field of AI provided by the
Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE University № 70-2021-00139.
The calculations were performed in part through the computational resources of HPC facilities at HSE University [Kostenetskiy et al., 2021].
A Compressed Convolutional Layers A.1 Proof of Lemma
The result essentially follows from different ways to represent the TT decomposition [Holtz et al., 2012].
By applying the QR decomposition to K (1) and K (3) ⊤ , we obtain:
[Equation]
where R (1) , R (3) are upper triangular.
Substituting these formulas into (4) yields (6) with
[Equation]
which completes the proof.
A.2 Proof of Theorem
Using (5), we have
[Equation]
Let us first show that given K (1) ⊤ K (1)
=
I r 1 , the matrix T K (1) has orthonormal rows, i.e., it satisfies
[Equation]
To do so, let us find T K (1) in terms of K (1) .
For any X ∈ R c in × n ×···× n and its row-major reshaping into a matrix X ∈ R c in × n d , we have
[Equation]
(cid:17) vec ( X ) .
Therefore, T K (1) =
K (1) ⊗ I n d , where ⊗ denotes the Kronecker product of matrices.
Using basic
Kronecker product properties and the orthogonality of K (1) , we arrive at (13).
Analogously, we may obtain T K (3) =
K (3) ⊤ ⊗ I f ( n,k ) d and
[Equation]
Finally, using SVD of T K (2) : T K (2) =
U Σ V ⊤ , we get:
[Equation]
where (cid:101) U, (cid:101) V have orthonormal columns as a product of matrices with orthonormal columns.
Hence,
T K = (cid:101) U Σ (cid:101) V ⊤ is in the compact SVD form, which completes the proof.
B Periodic Strided Convolutions B.1 Code for Computing Singular Values of a Strided Convolution
According to Theorem 2, the following code computes the singular values of a transform encoded by a strided convolution.
Note that neither the full SVD nor the clipping operation is included in the code.
The code for computing the singular vectors and the new kernel with constrained singular values can be found in the source code repository.
def SingularValues(kernel, input_shape, stride): kernel_tr = np.transpose(kernel, axes=[2, 3, 0, 1]) d1 = input_shape[0] - kernel_tr.shape[2] d2 = input_shape[1] - kernel_tr.shape[3] kernel_pad =
np.pad(kernel_tr, ((0, 0), (0, 0), (0, d1), (0, d2))) str_shape
=
input_shape // stride r1, r2 = kernel pad.shape[:2] transforms = np
, str
B.2 Proof of Theorem
In this section, we prove Theorem 2 from Sec. 4.1.
Firstly, we analyze the structure of the matrix corresponding to a strided convolution.
Secondly, we show that the columns of this matrix can be permuted to make matrix structure similar to that of a non-strided convolution.
Therefore, the new theorem can be reduced to the already proven theorem.
Let us denote a circulant with each row shifted by the value of stride as a “strided circulant”.
The shape of such a strided circulant is ns × n , where n is the number of elements in the first row.
Here is an example of a strided circulant with n = 4 , s = 2 : (cid:18)
a b c d c d a b (cid:19) .
One can think of this strided circulant as a block-circulant matrix with block sizes 1 × s .
At the same time, slicing this matrix by taking columns with a step equal to the stride, e.g., columns (cid:0) 0 , s, 2 · s, . . .
(cid:0) ns − 1 (cid:1) · s (cid:1) , gives a standard circulant matrix.
This fact can be stated as
[Equation]
where ⊗ denotes the Kronecker product, A is a strided circulant, e
i ∈
R s is the
i -th standard basis vector in R s , and A i is a circulant obtained by slicing columns of A .
The regular 2D convolutional transform matrix with a single input and single output channels has a doubly block-circulant structure (see Section 5.5 in Jain [1989]).
However, for strided convolutions, the structure is different.
For a fixed pair of input and output channels, the output of a strided convolution is a submatrix of an output of a convolution with the same kernel but without the stride.
More specifically, it is a slice with the stride s by both dimensions (in Python, it would be written as [:: s , :: s ] ).
For the matrix encoding the transformation, it means that only every s -th block row (simulating the slice by the first dimension) and every s -th regular row of a block (simulating the slice by the second dimension) are considered.
This means that the doubly block-circulant structure of the initial matrix turns into a doubly block-strided circulant structure of shape (cid:0) ns (cid:1) 2 × n 2 , where n is the size of a kernel.
Each block of this matrix is a strided circulant, and the block structure is that of a strided circulant as well.
The matrix consists of ns × n blocks, and each of them has the shape ns × n .
If we denote a strided circulant of a row vector a as circ s ( a ) , then the matrix of the transform is as follows:
[Equation]
It can be noted that, in the same way as in strided circulants, we can take block columns of the matrix with a step equal to the stride and get block circulants (where each block is a strided circulant).
This block structure can be described as follows:
[Equation]
Here B is a doubly block-strided circulant matrix, and B i ∈ R ( ns ) 2 × n 2 s is a block-circulant matrix.
This formula is similar to our previous sum representation for a strided circulant; however, the dimensions of the right term of each element are larger to account for the block structure of the left term.
Note that this term is needed to describe how the columns of B i are positioned in the matrix B , similar to the 1D case.
Let us consider B i .
It consists of block columns (cid:0) i, s
[Equation]
i
[Equation]

As a block-circulant with n s × n s blocks of n s × n , it can be expanded as follows:
[Equation]
where C ik ∈ R ns × n is a strided circulant block, C ik =
circ s
( K i − sk, : ) and P is a permutation matrix:
P =   0 . . .
1
[Equation]
We can expand C ik as a strided circulant:
[Equation]
where A ikj ∈ R ns × ns is a regular circulant matrix that can be acquired as circ 1 ( K i − sk,j :: s ) , i.e. as a circulant built from the slice of a string K i − sk , taken with a step s starting from the index j .
Finally,
B i =
[Equation]
This reformulation helps us see that the slices of B i by columns with step s are, in fact, doubly blockcirculant matrices defined by spatial slices of the kernel.
There are s matrices B i , each containing s column slices.
The j th column slice of B i is defined by A i, : ,j , which, in turn, is defined by the kernel slice K i :: s,j :: s .
In order to reduce the task of computing singular values of the matrix B , corresponding to the strided convolution, to the simpler task of computing singular values of a matrix corresponding to a regular convolution, let us permute the columns of B i :
[Equation]
This matrix consists of s consecutive doubly block-circulant matrices circ 2 ( K i :: s,j :: s ) .
The first dimension of B i is the same as the first dimension of B .
Therefore, for B , this is also just a permutation of the columns.
The next step is to permute the block columns of B :
[Equation]
After this permutation, matrix B consists of s 2 consecutive doubly block-circulant matrices.
Note that the particular order of these blocks is not important, as it is just a matter of column order.
Finally, let us look at the matrix associated with the multiple-channel convolution.
The matrix of the convolutional transform with c in input channels and c out output channels is as follows (equivalent to the structure described in Sedghi et al.
[Equation]
where
[Equation]
However, now we know that we can permute the columns of this matrix to get a matrix comprised of doubly block-circulant blocks.
After the permutation described above
[Equation]
we get a matrix
[Equation]
where
[Equation]
We chose a range of lambda values to allow us to keep the Lipschitz constant under 1 .
The
[Table]
i, j – some integer indices from 0 to s − 1 .
The exact relationship between i, j and c, d is not important, as it depends only on the order of the columns.
The only important thing is that it has to be the same in all the rows.
We choose the following functional form:
[Equation]
This matrix M ′ can be perceived as the matrix of convolution for a new kernel K ′ ∈
R ns × ns × c in × c out s 2 , defined by this equation:
[Equation]
Alternatively, to make things simpler, we can use the additional tensor R , defined in (8).
It is easier to use this tensor for implementing the formula in Python.
To conclude, we reduced the task of computing the singular values of M to the task of computing the singular values of the M ′ , solved by Sedghi et al.
The reduction is made possible via columns permutation, meaning that the singular values of the matrix did not change.
C Additional Empirical Studies C.1 Plotting Empirical Lipschitz Constant
In order to estimate the Lipschitz constants of LipConvNet networks and their TT-compressed counterparts, we plot histograms of empirical Lipschitz constants inspired by Sanyal et al.
The Lipschitz constants are evaluated by attacking each image from the test set using the FGSM attack of a fixed radius (0.5 in our experiments)
[Goodfellow et al., 2015].
We compute a ratio that is upper-bounded by the true Lipschitz constant L :
[Equation]
(14) for each image and plot the results as a histogram.
Here, f ( X ) is the output of a model f for an input image X ; X attacked is X perturbed, as described in the first paragraph.
Then, by computing
[Figure]
LipConv
[Figure] SOC-TT [Figure] L estimated ( X ) for different X from the dataset, we plot the histograms (Fig. 5).
These histograms give us insights into the true Lipschitz constant of the network.
If all network layers are 1 -Lipschitz, then we expect each L estimated ( X ) on the histogram to be strictly bounded by 1 .
However, for the LipConvNet architectures, the bound on the Lipschitz constant might not be exact in practice due to truncating Taylor expansion.
In our case, the frame matrices K (1) and K (3) are trained with the regularization loss, so they are not precisely orthogonal either.
By increasing the coefficient λ ort of this loss component, though, we can force the matrices to become orthogonal by the end of the training process.
To support this claim, we provide the distributions of the empirical Lipschitz constants for several
LipConvNet architectures trained with different λ ort .
The trained models are presented in Tab. 3.
The last metric, “Lip.”, or Maximum Empirical Lipschitz Constant, is a lower bound on the real Lipschitz constant of the corresponding model.
It is obtained as the maximum value of all calculated empirical
Lipschitz constants.
As we can see, by increasing λ ort , we can balance between the 1 -lipschitzness and the quality of metrics: the higher λ ort , the lower the metrics are and the more constrained the
Lipschitz constant is.
Numbers selected with bold correspond to models that were selected as best for the corresponding LipConvNet- N architectures: they demonstrate high performance in terms of accuracy and robust metrics while at the same time maintaining the Maximum Empirical Lipschitz
Constant that does not exceed 1.
We also note that LipConvNet architectures do not converge if the convolutional layers are too far from being 1-Lipschitz.
For example, this effect was observed on
LipConvNet-5 with λ ort = 4 e 3 , where the λ ort turns out to be not big enough.
Fig.
5 demonstrates the shifts in distribution depending on λ ort .
We can observe ranges of empirical
Lipschitz constants on the x -axis.
Maximum values for each model, denoted with dotted vertical lines, are the discussed Maximum Empirical Lipschitz Constants in Tab. 3.
C.2 Inference Time of a SOC-TT Layer
[Table]
[Figure]
In this section, we present the inference time of our framework when applied
In particular, we consider the application of a single SOC layer with various numbers of channels.
Fig.
6 illustrates speedups when a single
SOC layer is accelerated using the proposed method with different rank values.
The figure suggests that for larger residual networks that contain layers with a number of channels up to 1000 , the speedup can be up to ≈ 6 times for c/ 4 and up to ≈ 3 times for c/ 2 .
However, for networks where the number of channels is less than 500 , the speedup is less than 2 .
C.3 WideResNet16-10 on CIFAR-
In addition to evaluating our framework on CIFAR-10, we conducted experiments on another classic vision dataset, CIFAR-100.
We trained WideResNet-16-10 on CIFAR-100 with the same experimental setup as in Gouk et al.
[2021] and compared clipping and division as in the main body of the paper.
The results are presented in Tab. 4.
Compared with CIFAR-10 experiments from 2, we observe that both clipping and division do not give gain on CIFAR-C (CC).
The other metrics tend to improve with the control of singular values.
Except for clipping- 1 , the TT-compressed versions with singular value control outperform the baseline accuracy in most cases.
The AA metric of the baseline is outperformed when using clipping to 2 and r = 256 .