“Lossless” Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach
Anonymous Author(s)
AffiliationAddress email
Abstract
Modern deep neural networks (DNNs) are extremely powerful; however, this comesat the price of increased depth and having more parameters per layer, making theirtraining and inference more computationally challenging.
In an attempt to addressthis key limitation, efforts have been devoted to the compression (e.g., sparsificationand/or quantization) of these large-scale machine learning models, so that theycan be deployed on low-power IoT devices.
In this paper, building upon recentresearch advances in neural tangent kernel (NTK) and random matrix theory, weprovide a novel compression approach to wide and fully-connected deep neuralnets.
Specifically, we demonstrate that in the high-dimensional regime wherethe number of data points n and their dimension p are both large, and under a
Gaussian mixture model for the data, there exists asymptotic spectral equivalencebetween the NTK matrices for a large family of DNN models.
This theoreticalresult enables “lossless” compression of a given DNN to be performed, in the sensethat the compressed network yields asymptotically the same NTK as the original(dense and unquantized) network, with its weights and activations taking valuesonly in { 0 , ± 1 } up to a scaling.
Experiments on both synthetic and real-world dataare conducted to support the numerical advantages of the proposed method.
Introduction
Modern deep neural networks (DNNs) are becoming increasingly over-parameterized, having moreparameters than required to fit the also increasingly large, complex, and high-dimensional data.
While the list of successful applications of these large-scale machine learning (ML) models is rapidlygrowing, the energy consumption of these models is also increasing, making them more challengingto deploy on close-to-user and low-power devices.
To address this issue, compression techniqueshave been proposed that prune, sparsify and/or quantize DNN models [13, 22], thereby yielding
DNNs of a much smaller size that can still achieve satisfactory performance on a given ML task.
As an illustrative example, it has been recently demonstrated that at least 90% of the weights in popular
DNN models such as VGG19 and ResNet32 can be removed with virtually no performance loss [52].
Despite the remarkable progress achieved by various DNN model compression techniques, due to the nonlinear and highly non-convex nature of DNNs, our theoretical understanding of these large-scale
ML models, as well as of their compression schemes, is progressing at a more modest pace.
For example, it is unclear how much a given DNN model can be compressed without severe performance degradation; perhaps more importantly, on the degree to which such a trade-off between performance and complexity depends on the ML problem and the data also remains unknown
In this respect, neural tangent kernels (NTKs) [27], provide a powerful tool for use in assessing the convergence and generalization properties of very wide (sometimes unrealistically so) DNNs by studying their corresponding NTK eigenspectra, which are solely dependent on the input data, thenetwork activation function, and the random weights distribution.
In this paper, building upon recent advances in random matrix theory (RMT) and high-dimensionalstatistics, we demonstrate that for data x 1 , . . .
, x n ∈ R p drawn from a K -class Gaussian mixturemodel (GMM), in a high-dimensional and non-trivial classification regime where the input datadimension p and their size n are both large and comparable, the eigenspectra of both the NTK andthe closely related conjugate kernel (CK) matrices at any layer ℓ ∈ { 1 , . . .
, L } are independent of thedistribution of the i.i.d.
entries of the (random) weight matrix W ℓ , provided that they are normalizedto have zero mean and unit variance, and only depend on the activation function σ ℓ ( · ) via four scalarparameters.
In a sense, we establish, at least for GMM data, the asymptotic spectral equivalencebetween the NTK matrices of the corresponding network layers, and consequently of the wholenetwork, for a large family of DNN models with possibly very different weights and activations,given that they have normalized weight entries and share the same few activation parameters.
Since the convergence and generalization properties of ultra-wide DNNs depend only on the eigenspectra (i.e., eigenvalue-eigenvector pairs; see also Remark 2 below) of the corresponding NTKmatrices [27, 17], in the sense that, e.g., the time evolutions (when trained with gradient descent usinga sufficiently small step size) of both the residual error and the in-sample prediction of the (sufficiently wide) network can be expressed as explicit functions of the NTK eigenvalues and eigenvectors[17, 1, 25]. We further exploit the above theoretical results to propose a novel NTK-based approachthat allows for the “lossless” compression of a given fully-connected DNN model, by designing asparse and quantized DNN that (i) has asymptotically the same NTK eigenspectra as the original“dense and full precision” network, and (ii) has both weights and activations taking values in the set{− 1 , 0 , +1 } before scaling, and can thus be stored and computed much more efficiently.
Despite being derived here for Gaussian mixture data, an unexpected close match is observed betweenour theory and the empirical results on real-world datasets, suggesting possibly wider applicabilityfor the proposed NTK-based Lossless Compression (NTK-LC) approach.
Looking forward, weexpect that our analysis will open the door to improved analysis based on RMT and high-dimensionalstatistics, which will demystify the seemingly striking empirical observations in modern DNNs.
1.1 Our contributions
Our main results can be summarized as follows:
We provide, in Theorems 1 and 2 respectively, for GMM data and in the high-dimensional regime (Assumption 1), precise spectral characterizations of the CK and NTK matrices of fully-connectednetworks; furthermore, we show that the CK and NTK eigenspectra do not depend on the distributionof i.i
d. weights, and depend solely on the activation function via a few scalar parameters.
In Corollary 1 and Algorithm 1, we exploit these results to propose a novel DNN compression scheme, named NTK-based Lossless Compression (NTK-LC), with sparsified and ternarized weightsand activations, which nevertheless do not affect (the spectral behavior of) the NTK matrices.3.
In Section 4, we present empirical evidence on (not so) wide DNNs trained with both syntheticGaussian and popular real-world data such as MNIST [30] and CIFAR10 [29], and show a factor of10 3 less memory is needed with the proposed NTK-LC approach, with virtually no performance loss
1.2 Related work
Neural network model compression.
The study of NN compression dates back to early 1990[31], at which point, in the absence of the (possibly more than) sufficient computational power thatwe have today, compression techniques allowed neural networks to be empirically evaluated oncomputers with limited computational and/or storage resources [48].
Alongside the rapid growthof increasingly powerful computing devices, the development of more efficient NN architecturesand training/inference protocols, and the need to implement NNs on mobile and low-power devices,(D)NN model compression has become an active research topic and many elegant and efficientcompression approaches have been proposed over the years [21, 24, 26, 22].
However, due to the nonlinear and highly non-convex nature of DNNs, our theoretical understanding of these largescale ML models, as well as of (e.g., the fundamental “performance and complexity” trade-off of)compressed DNNs, is somewhat limited [22].
Neural tangent kernel.
Neural tangent kernel (NTK) theory recently proposed in [27], by consideringthe limit of infinitely wide DNNs, characterizes the convergence and generalization properties ofvery wide DNNs when trained using gradient descent with small steps.
Initially proposed for fullyconnected nets, the NTK framework has been subsequently extended to convolutional [4], graph [15],and recurrent [2] settings.
The NTK theory, while having the advantage of being mathematicallymore tractable (via, e.g., the characterization of the associated reproducing kernel Hilbert space [7]),seems to diverge from the regime on which modern (and not so wide) DNNs operate, see [9, 37, 17].
Random matrix theory and neural networks.
Random matrix theory (RMT), a powerful andflexible tool for assessing the behavior of large-scale systems with a large “degree of freedom,” isgaining popularity in the field of NN analysis [43, 44], in both shallow [47, 35, 36] and deep [6, 17, 46]settings, and considers both homogeneous (e.g., standard normal) [47, 45] and mixture-type data[35, 3].
From a technical perspective, the most relevant paper is [3] , in which the authors proposed a RMT-inspired NN compression scheme, albeit only in the single-hidden-layer setting.
This paperextends the analysis in [3] to multi-layer fully-connected DNNs by focusing on the associated NTKmatrices, and proposes a novel sparsification and quantization scheme for fully-connected DNNs(which is in spirit similar to, although formally different from, that proposed in [3]).
1.3 Notations and organization of the paper
We denote scalars by lowercase letters, vectors by bold lowercase, and matrices by bold uppercase.
We denote the transpose operator by ( · ) T , and use ∥ · ∥ to denote the Euclidean norm for vectors andthe spectral/operator norm for matrices.
For a random variable z , E [ z ] denotes the expectation of z .
We use 1 p and I p to represent an all-ones vector of dimension p and the identity matrix of size p × p .
The remainder of this article is structured as follows.
In Section 2, we present the DNN model understudy, together with our working assumptions.
Section 3 contains our main technical results on theeigenspectra of the conjugate kernel K CK and NTK matrix K NTK , along with an account of how theyapply to the compression of fully-connected deep neural nets with the proposed NTK-based Lossless
Compression (NTK-LC) approach.
Empirical evidence is provided in Section 4 to demonstrate thesignificant computation and storage savings, with virtually no performance degradation, that can beobtained using NTK-LC.
Conclusion and future perspectives are placed in Section 5.
Preliminaries
Let x 1 , . . .
, x n ∈ R p be n data vectors independently drawn from one of the K -class Gaussianmixtures C 1 , . .
, C K , and let X =
[ x 1 , . . .
, x n ] ∈ R p × n , with class C a having cardinality n a ; that is,
[Equation]
for the mean vector µ a ∈ R p and covariance matrix C a ∈ R p × p associated with class C a .
In the high-dimensional scenario where n, p are both large and comparable, we position ourselvesin the following non-trivial classification setting, so that the K -class classification above is neithertrivially easy nor impossible; see also [12] and [8, Section 2].
Assumption 1 (High-dimensional asymptotics) .
As n → ∞ , we have, for a ∈ { 1 , . .
, K }
that (i)p/n → c ∈ (0 , ∞ ) and n a /n
→ c a ∈ [0 , 1) ; (ii) ∥ µ a ∥ =
O (1) ; (iii) for C ◦ ≡ (cid:80) Ka
=1 n a n C aand C ◦ a ≡ C a − C ◦ , we have ∥ C a ∥ =
O (1) , tr C ◦ a
=
O
√ p )
and tr( C a C b ) =
O ( p ) fora, b ∈ { 1 , . .
, K } ; and (iv) τ 0 ≡ (cid:112)
tr C ◦ /p converges in (0 , ∞ ) .
We consider using a fully-connected neural network model of depth L for the classification of theabove K -class Gaussian mixture.
Such a network can be parameterized by a sequence of weightmatrices W
1 ∈ R d 1 × d 0 , . . .
, W L ∈ R d
L × d L − 1 (with d 0 = p ), and nonlinear activation functionsσ 1 , . . .
, σ L that apply entry-wise, so that the network output is given by the following:
[Equation]
for an input data vector x ∈ R p and output vector w ∈ R d L .
We denote Σ ℓ ∈ R d ℓ × n therepresentations of the data matrix X ∈ R p × n at layer ℓ ∈ { 1 , . . .
, L } defined as
[Equation]
The normalization by √ d ℓ follows from the NTK literature and ensures the consistent asymptoticbehavior of the network in the high-dimensional setting in Assumption 1 and 2; see also [27, 7, 17].
The training and generalization performance of the neural network model defined in (2) are closelyrelated to two types of kernel matrices: the Conjugate Kernel (CK) matrix and Neural Tangent Kernel(NTK) matrix, defined respectively for ℓ ∈ { 1 , . . .
, L } as follows:
[Equation]
with expectation taken with respect to the weights and Σ ℓ ∈ R d ℓ × n the data representation at theoutput of layer ℓ defined in (3). In particular, CKs are known to satisfy the following recursiverelation [27, 7]
[Equation]
while for the NTK matrix K NTK ,ℓ ∈ R n × n of layer ℓ , we have:
[Equation]
where ‘ A ◦ B ’ denotes the Hadamard product between two matrices A , B of the same size, and
K ′ CK ,ℓ denotes the CK matrix with nonlinear function σ ′ ℓ instead of σ ℓ as for K CK ,ℓ defined in(5); that is, [ K ′ CK ,ℓ ] ij = E u,v [ σ ′ ℓ ( u ) σ ′ ℓ ( v )] .
Note in particular that for a given DNN model, thecorresponding CK and NTK matrices depend only on the network structure (i.e., the number oflayers and the activation function in each layer), the distribution of the random (initializations of the)weights to be integrated over (e.g., in the expectation in equation (4)), and the input data.
It has been shown in a series of previous efforts [27, 17, 25] that for very (and sometimes unrealistically) wide DNNs trained using gradient descent with a small step size, the time evolution of theresidual errors and in-sample predictions of a given DNN are explicit functionals of the corresponding
K NTK involving its eigenvalues and eigenvectors.
In this respect, the NTK theory provides, via theeigenspectral behavior of K NTK , precise characterizations of the convergence and generalizationproperties of DNNs [27, 7], by focusing on the impact of the network structure (e.g., the number oflayers and the choice of activation functions), the input data, and the weight initialization schemes.
In this paper, we focus on fully-connected nets under the following assumption regarding the weights.
Assumption 2 (On random weights) .
The weight matrices W 1 ∈ R d 1 × p , . . .
, W L ∈ R d
L × d L − 1 areindependent and have i.i.d.
entries of zero mean, unit variance, and finite fourth-order moment.
Assumption 2, together with the √ d ℓ normalization, is compatible with fully-connected DNNs in (2),which are admittedly less interesting, from a practical perspective, compared to their convolutionalcounterparts.
The proposed framework is envisioned to be extendable to a convolutional [4, 7] andmore involved setting (e.g., graph NNs [15]) by considering (e.g., Toeplitz-type) structures on W s.
Unlike most existing NTK literature [27, 7, 16], we do not assume the Gaussianity of the entries of
W ℓ s, but only that they are i.i.d.
and “normalized” to have zero mean and unit variance.
As it turnsout, this assumption together with a (Lyapunov-type) central limit theorem argument, is sufficient toestablish most existing results on the convergence and generalization of DNNs; see for example [33].
Assumption 3 (On activation functions) .
The activation functions σ 1 ( · ) , . . .
, σ L ( · ) in each layerare at least four-times differentiable with respect to standard normal measure, in the sense thatmax k ∈{ 0 , 1 , 2 , 3 , 4 } {| E [ σ ( k ) ℓ ( ξ )] |} is finite for ξ ∼ N (0 , 1) and ℓ ∈ { 1 , . . .
, L } .
Using the Gaussian integration by parts formula, one has E [ σ ′ ( ξ )]
=
E [ ξσ ( ξ )] for ξ ∼ N (0 , 1) , aslong as the right side expectation exists.
As a result, even for non-differentiable functions, it sufficesto have | σ ℓ | upper-bounded by some (high-degree) polynomial function for Assumption 3 to hold.
With these preliminaries, we can move on to present our main technical results on the spectralbehavior of the CK and NTK matrices for a large family of fully-connected DNNs.
Main results
For a fully-connected DNN defined in (2), our first result is on the eigenspectral behavior of thecorresponding CK matrices K CK defined in (4).
More specifically, we show for Gaussian mixture datain (1) and in the high-dimensional setting of Assumption 1, that the K CK ,ℓ of layer ℓ ∈ { 1 , . . .
, L } isasymptotically spectrally equivalent to another random matrix ˜ K CK ,ℓ , in the sense that their spectralnorm difference ∥ K CK ,ℓ − ˜ K CK ,ℓ ∥ vanishes as n, p → ∞ .
This result is stated as follows, the proofof which is based on an induction on ℓ and is given in Section A.1 of the appendix.
Theorem 1 (Asymptotic equivalents for CK matrices) .
Let Assumptions 1–3 hold, and letτ 0 , τ 1 , . .
, τ L ≥ 0 be a sequence of non-negative numbers satisfying the following recursion:
[Equation]
Further assume that the activation functions σ ℓ ( · ) s are “centered,” such that E [ σ ℓ ( τ ℓ − 1 ξ )] = 0 .
Then,for the CK matrix K CK ,ℓ of layer ℓ ∈ { 1 , . . .
, L } defined in (4) , as n, p → ∞ , one has that:
[Equation]
almost surely, with
[Equation]
, (9) for class label vectors J = [ j 1 , . . , j K ] ∈ R n × K , “second-order” data fluctuation vector ψ ={∥ x i − µ a / √ p ∥ 2 − E [ ∥ x i − µ a / √ p ∥ 2 ] } ni =1 ∈ R n , second-order discriminative statistics t
={ tr C ◦ a / √ p } Ka =1 ∈ R K and T = { tr C a C b /p } Ka,b =1 ∈ R K × K of the Gaussian mixture in (1) , aswell as non-negative scalars α ℓ, 1 , α ℓ, 2 , α ℓ, 3 ≥ 0 satisfying
[Equation]
(11) with α ℓ, 4 = α ℓ − 1 , 4 E (cid:2) ( σ ′ ℓ ( τ ℓ − 1 ξ )) 2 + σ ℓ ( τ ℓ − 1 ξ ) σ ′′ ℓ ( τ ℓ − 1 ξ ) (cid:3) for ξ ∼ N (0 , 1) .
A few remarks on Theorem 1 are in order.
The first remark is on the condition E [ σ ℓ ( τ ℓ − 1 ξ )]
= 0 .
Remark 1 (On activation centering) .
The condition E [ σ ℓ ( τ ℓ − 1 ξ )] = 0 , seemingly restrictive at firstsight, in fact only subtracts an identical constant from all entries of the data representation Σ ℓ at layerℓ as defined in (3) , and should not restrict the expressive power of the network, nor its performanceon downstream ML tasks.
For a given DNN model of interest, it suffices to “center” the output ofeach layer by subtracting a constant to satisfy Assumption 3, and to further apply our Theorem 1.
Theorem 1 unveils the (possibly surprising) fact that, for the high-dimensional and non-trivial
Gaussian mixture classification of (1), the spectral behavior of ˜ K CK ,ℓ (and thus that of the CKmatrix K CK ,ℓ ) is (i) independent of the distribution of the (entries of the) weights W ℓ when they are“normalized” to have zero mean and unit variance, as demanded in Assumption 2, and (ii) dependson the activation function σ ℓ only via four 2 scalar parameters α ℓ, 1 , α ℓ, 2 , α ℓ, 3 and τ ℓ : such universalresults have been previously observed in random matrix theory and high-dimensional statisticsliterature (see for example [11, 5, 55]) and indicate the wide applicability of our theoretical result.
On closer inspection of Theorem 1, we further observe that:(i) for a given DNN, Theorem 1 characterizes, via the form of ˜ K CK ,ℓ in (8) and the recursions in(10) and (11), how the linear (via α ℓ, 1 , which is multiplied by X T X ) and nonlinear (via α ℓ, 2 and α ℓ, 3in A ℓ , which respectively weight the second-order data statistics t and T ) data features “propagate,”in a layer-by-layer fashion, as ℓ increases, as quantitatively measured by the corresponding α ℓ s; and (ii) for two DNNs with the same number of layers, but possibly different weights and activations,given the same input data X (so that the two nets have the same K CK , 0 ), if they have asymptoticallyequivalent CK matrices K CK ,ℓ − 1 at layer ℓ − 1 with the same α ℓ − 1 , 1 , α ℓ − 1 , 2 and α ℓ − 1 , 3 , then havingthe same E [ σ ′ ℓ ( τ ℓ − 1 ξ )] 2 , E [ σ ′′ ℓ ( τ ℓ − 1 ξ )] 2 , and E [( σ 2 ℓ ( τ ℓ − 1 ξ )) ′′ ] (which only depends on the activationσ ℓ of layer ℓ and τ ℓ − 1 ) will suffice for the two nets to have equivalent K CK ,ℓ at layer ℓ .
It follows from the above item (ii) that for a given DNN of depth L , it is possible to design a novel
DNN model that “matches” the original one – in the sense that both models will have asymptoticallyequivalent CK matrices at each layer , by using the following layer-by-layer matching strategy:
Starting from the same K CK ,
= X T X , one chooses the weights W 1 of the novel DNN accordingto Assumption 2, and then select the first-layer activation σ 1 in such a way that the novel net has thesame parameters
α 1 , 1 , α 1 , 2 and α 1 , 3 as the original one, so that the first-layer CK matrices K CK , 1of the two nets are spectrally matched as per Theorem 1; one then proceeds similarly to match thesecond, the third, etc., and eventually the L th layer of the two nets.
As we shall see below, thislayer-by-layer matching strategy facilitates the “lossless” compression of a given DNN.
Using the relation in (6), a similar result (as in Theorem 1 for CKs) can be established for NTKmatrices, as shown in the following theorem.
The proof is provided in Section A.2 of the appendix.
Theorem 2 (Asymptotic equivalent for NTK matrices) .
Let Assumptions 1–3 hold, let σ ℓ ( · ) s becentered so
that E [ σ ℓ ( τ ℓ − 1 ξ )] = 0 for τ ℓ s defined in (7) , and let κ 1 , . .
, κ L ≥ 0 be a sequence ofnon-negative numbers satisfying the following recursion:
[Equation]
Then, for the NTK matrix K NTK ,ℓ of layer ℓ defined in (6) , as n, p → ∞ one has that
[Equation]
almost surely, with V ∈ R n × ( K +1) , t ∈ R K , T ∈ R K × K as defined in Theorem 1, and
[Equation]
as well as the non-negative constants β ℓ, 1 , β ℓ, 2 , β ℓ, 3 , β ℓ, 4 ≥ 0 , such that
[Equation]
and β ℓ, 4 satisfies the recursion β ℓ, 4 = E [ σ ′′ ℓ ( τ ℓ − 1 ξ )] 2 β ℓ − 1 , 4 .
Roughly speaking, Theorem 2 shows that a similar eigenspectral behavior established in Theorem 1for CK matrices also holds for NTK matrices if we change the associated coefficients α ℓ s to β ℓ s. Theremarks after Theorem 1 thus remain valid, at least in spirit, for NTK matrices.
Remark 2 (On spectral norm characterization) .
Note that the characterizations in Theorem 1 and 2for CK and NTK matrices are provided in a spectral norm sense.
It then follows from Weyl’s inequality[23] and the Davis–Kahan theorem [59] that the difference between the eigenvalues (e.g., whenlisted in a decreasing order) and the associated eigenvectors (when the eigenvalues under study are“isolated”) of K NTK and ˜ K NTK vanish asymptotically as n, p → ∞ .
As such, the spectral normguarantees in Theorem 1 and 2 provide more tractable access to the convergence and generalizationproperties of wide DNNs, at least for GMM data, via the spectral study of ˜ K NTK [27, 17].
Despite being derived here for the Gaussian mixture model in (1), we conjecture that the resultsin Theorem 1 and 2 hold beyond the Gaussian setting and can be extended, for example, to thefamily of concentrated random vectors [49, 32].
As previously discussed after Assumption 2 forthe distribution of W , universality commonly arises in random matrix theory and high-dimensionalstatistics [11, 5, 55]; we refer interested readers to Remark 4 in Appendix A for further discussions.
From a technical perspective, the results in Theorem 1 and 2 extend the single-hidden-layer CKanalysis in [3, 35] to both CK and NTK matrices of fully-connected DNNs with an arbitrary numberof layers.
In particular, taking ℓ = 1 in Theorem 1, one obtains [3, Theorem 1] as a special case. 3 r 1 r 2 s 1s 2 r 3 ra b 2 b [Figure] E
[Figure] τξ ) [Figure] a
[Figure]
(cid:18) erf (cid:18) [Figure] √ 2 τ (cid:19) − erf (cid:18) [Figure]
√ [Figure] a
[Figure] Remark 3 (On the equivalent CK and NTK matrices for two neural nets) .
It follows from Theorem 1that for a given DNN model, it suffices to “match” the coefficients α ℓ, 1 , α ℓ, 2 and α ℓ, 3 in a layer-bylayer manner to design a novel DNN with asymptotically equivalent CK.
In doing so, one can choosethe activation σ ℓ at layer ℓ to satisfy a system of equations involving E [ σ ′ ℓ ( τ ℓ − 1 ξ )] 2 , E [ σ ′′ ℓ ( τ ℓ − 1 ξ )] 2 ,and E (cid:2) ( σ ′ ℓ ( τ ℓ − 1 ξ )) [-
-]2 + σ ℓ ( τ ℓ − 1 ξ ) σ ′′ ℓ ( τ ℓ − 1 ξ ) (cid:3) .
Interestingly, it can then be checked from Theorem 2that, by performing the above matching, if one has equivalent NTK matrices K NTK ,ℓ − 1 at layerℓ − 1 for the two nets under consideration, then the corresponding K NTK ,ℓ at layer ℓ are also“matched” in a spectral sense, since the (additional) parameter E [ σ ′′ ℓ ( τ ℓ − 1 ξ )] 2 in the recursion of β ℓ, 4in Theorem 2 has already been matched.
Also, similarly to Footnote 2 for CK, matching (or nor) theparameter κ ℓ only affects the level of regularization by “shifting” all eigenvalues of K NTK .
In the following corollary, we present a concrete example of how to apply the results in Theorem 1 and 2 in the design of a novel computationally and storage efficient DNN, that shares the same
CK and NTK eigenspectra with any given fully-connected neural net having centered activation.
Corollary 1 (Sparse and quantized DNNs) .
For a given fully-connected DNN (referred to as DNN1 )of depth L with centered activation such that E [ σ ℓ ( τ ℓ − 1 ξ )] = 0 for ξ ∼ N (0 , 1) , one is able toconstruct (again in a layer-by-layer manner) a sparse and quantized “equivalent” DNN model, ofdepth L and referred to as DNN2 , such that the two nets have asymptotically the same eigenspectrafor their CK (and thus NTK, as per Remark 3) matrices, by using the following ternary weights:
[Equation]
as well as quantized activations (as visually displayed in Figure 1):
[Equation]
We refer readers to Section A.4 in the appendix for the proof and discussions of Corollary 1, aswell as the detailed expressions of E [ σ ′ ( τξ )] , E [ σ ′′ ( τξ )] , and E [( σ 2 ( τξ )) ′′ ] , of direct algorithmicuse for both σ T and σ Q as functions of a, s 1 , s 2 and b 1 , b 2 , r 1 , r 2 , r 3 , r 4 .
The proposed NTK-LCcompression approach based on Corollary 1, is described in detail below.
Note that, the “sign” of activations does not really matter in the design of computationally efficient
DNNs described in Algorithm 1, in the sense that the key parameters α ℓ, 1 , α ℓ, 2 and α ℓ, 3 for CKs, aswell as β ℓ, 1 , β ℓ, 2 and β ℓ, 3 for NTKs, remain unchanged when − σ ℓ ( t ) is used instead of σ ℓ ( t ) . Before embarking on the detailed numerical experiments in Section 4, we would like to bring thereaders’ attention to the recent line of works [34, 52, 56, 53, 18] showing that for wide and deep NNmodels, very efficient sparse sub-networks can be found that almost match the performance of theoriginal dense nets with little or even no training , for instance by uniformly pruning the networkweights [52]. To develop a theoretical grasp of these (extremely counterintuitive) empirical successes,a few attempts have been made, for example, to carefully prune the network weights to retain thesame (limiting) NTK [39], or to show that randomly pruned sparse nets have the same (limiting) NTK as the original net up to a scaling factor [58].
Instead, our work, by considering the statisticalstructure of the input data, leverages tools from RMT to “compress” both the weights and activations(per Theorem 2 and Corollary 1) without affecting the NTK eigenstructure.
Algorithm
Input: Input data x 1 , . . .
, x n , sparsity level ε ∈ [0 , 1) , and DNN1 with σ 1 , . . .
, σ L activations
Output: Sparse and quantized DNN2 model with weights W ℓ s and activations ˜ σ ℓ s.
Estimate τ 0 from input data as τ 0 = (cid:113) 1 n (cid:80) ni
=
∥ x i ∥ 2 .
Set τ = τ 0 , ˜ τ
= τ 0 .
for ℓ = 1 , . . .
, L − 1 do
Compute E [ σ ′ ℓ ( τξ )] 2 , E [ σ ′′ ℓ ( τξ )] 2 , and E [( σ 2 ℓ ( τξ )) ′′ ] , and use them to solve for the coefficients a, s 1 , s 2 in the novel activation ˜ σ ℓ , with ˜ τ , Corollary 1, and expressions in Appendix A.4.
Set τ = (cid:112) E [ σ 2 ℓ ( τξ )] , ˜ τ
= (cid:112) E [ σ 2 ℓ (˜ τξ )] .
end for
For layer ℓ =
L , compute (cid:112) E
[ σ 2 L ( τξ )] , E [ σ ′ L ( τξ )] 2 , E [ σ ′′ L ( τξ )] 2 , and E [( σ 2 L ( τξ )) ′′ ] , and use them to solve for the coefficients b 1 , b 2 , r 1 , r 2 , r 3 , r 4 in the novel activation ˜ σ L , with ˜ τ , Corollaryand detailed expressions in Appendix A.4.
Draw independently the i.i.d.
entries of W 1 , . .
, W L according to (16) with sparsity level ε .
return DNN2 model with weights W ℓ and activations ˜ σ ℓ , ℓ = 1 , . . .
, L .
Numerical experiments
In this section, we provide numerical experiments to (i) validate the asymptotic characterizations in
Theorem 1 and 2, on both synthetic GMM and real-world data (such as MNIST and CIFAR10) of (infact not so ) large sizes and dimensions; and to (ii) show how these results can be used to sparsifyand quantize fully-connected DNNs, leading to huge savings in computational and storage resources(up to a factor of 10 3 in memory and a level of sparsity ε − 90% ) without significant performancedegradation.
We refer readers to Section B in the appendix for further experiments and discussions.
[Figure]
Eigenvalue C oun t
Eigs of K
[Figure]
Eigenvalue C oun t Eigs of ˜ K
[Figure]
Eigenvalue
C oun t Eigs of K  {+30   · 
+}
Eigenvalue C oun t
Eigs of K
[Figure]
Eigenvalue
C oun t Eigs of ˜ K  {+30   · 
 Eigenvalue C oun+}
[Figure]
Eigs of ˜ K
[Figure]
Figure 2 compares the eigenvalues and dominant eigenvectors of the CK matrices K CK definedin (4) versus those of their asymptotic approximations ˜ K CK given in Theorem 1, in the case offully-connected DNNs having three hidden layers (of width d 1 = 2 000 , d 2 = 2 000 , d 3 = 1 000 ).
For different types of activations ( poly( t ) = 0 .
2 t 2 + t and ReLU( t ) =
max( t, 0) ), different weightdistributions (Gaussian and symmetric Bernoulli) and on synthetic GMM as well as MNIST data,we consistantly observe a close match between the eigenvalues and the dominant eigenvectors of
K CK and ˜ K CK , as a consequence of the spectral norm convergence in Theorem 1 (and Remark 2),suggesting a possibly wider applicability of the proposed results beyond GMM data.
[Figure] Figure 3 depicts the test accuracies of (i) the original dense and unquantized network with threefully-connected layers, (ii) the proposed NTK-LC approach described in Corollary 1 and Algorithm 1,as well as its variant having ternarized weights but dense and unquantized activations, (iii) thepopular magnitude-based pruning approach as proposed in [19], together with (iiii) two “heuristic”compression approaches: (iiii-i) sparsification by uniformly zeroing out 80% of the weights (wecannot do more, as the resultant performance is too poor to be compared with other curves) , and(iiii-ii) binarization using σ ( t )
= 1 t< − 1 + 1 t> 1 , for different choices of width per layer, and theten-class classification problems of MNIST and CIFAR10.
Specifically, Neural networks beforeand after compression both have three fully-connected layers, and the original network uses ReLUactivation each layer.
Classification is performed on a concatenated trainable fully-connected layer.
Raw data are taken as network input for MNIST datasets; flattened output of the 16th convolutionallayer of VGG19 [51] are taken as net input for CIFAR10 dataset.
In Figure 3, we observe that the proposed NTK-LC approach occupies (up to) a factor of 10 3 lessmemory, and produces significantly sparser networks (up to 90% of weights set to zero) with minimalperformance loss, when compared to the original or the “heuristically” compressed nets. In addition,when compared with its ternary weights variant (with unquantized activations), the proposed sparseand quantized NTK-LC trades limited performance drop for 32 times less memory, and achieveshigher accuracies than the popular magnitude-based pruning under the same memory budget.
Wealso see that the sparsity level ε has limited impact on the classification accuracy, which is in linewith our theory.
These experimental results show that the proposed NTK-LC approach achieves abetter performance-complexity trade-off than commonly used heuristic DNN compression methods.
Conclusion and perspectives
In this paper, built upon recent advances in random matrix theory and high-dimensional statistics,we provide precise characterizations of the eigenspectra of both conjugate kernel and neural tangentkernel matrices, for high-dimensional Gaussian mixture data and fully-connected multi-layer neuralnets.
These results further allows us to sparsify and quantize fully-connected deep nets, resulting in afactor of 10 3 less memory consumption with virtually no performance degradation.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately
reflect the
paper’scontributions and
scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[No]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]