“Lossless” Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach
Abstract
Modern deep neural networks (DNNs) are extremely powerful; however, this comes at the price of increased depth and having more parameters per layer, making their training and inference more computationally challenging.
In an attempt to address this key limitation, efforts have been devoted to the compression (e.g., sparsiﬁcation and/or quantization) of these large-scale machine learning models, so that they can be deployed on low-power IoT devices.
In this paper, building upon recent advances in neural tangent kernel (NTK) and random matrix theory (RMT), we provide a novel compression approach to wide and fully-connected deep neural nets.
Speciﬁcally, we demonstrate that in the high-dimensional regime where the number of data points n and their dimension p are both large, and under a
Gaussian mixture model for the data, there exists asymptotic spectral equivalence between the NTK matrices for a large family of DNN models.
This theoretical result enables “lossless” compression of a given DNN to be performed, in the sense that the compressed network yields asymptotically the same NTK as the original (dense and unquantized) network, with its weights and activations taking values only in { 0 , ± 1 } up to a scaling.
Experiments on both synthetic and real- world data are conducted to support the advantages of the proposed compression scheme, with code available at https://github.com/Model-Compression/
Lossless_Compression .
Introduction
Modern deep neural networks (DNNs) are becoming increasingly over-parameterized, having more parameters than required to ﬁt the also increasingly large, complex, and high-dimensional data.
While the list of successful applications of these large-scale machine learning (ML) models is rapidly growing, the energy consumption of these models is also increasing, making them more challenging to deploy on close-to-user and low-power devices.
To address this issue, compression techniques have been proposed that prune, sparsify, and/or quantize DNN models [14, 23], thereby yielding
DNNs of a much smaller size that can still achieve satisfactory performance on a given ML task.
As an illustrative example, it has been recently shown that at least 90% of the weights in popular DNN models such as VGG19 and ResNet32 can be removed with virtually no performance loss [53].
Despite the remarkable progress achieved by various DNN model compression techniques, due to the nonlinear and highly non-convex nature of DNNs, our theoretical understanding of these large-scale
ML models, as well as of their compression schemes, is progressing at a more modest pace.
For example, it remains unclear how much a given DNN model can be compressed without severe performance degradation; perhaps more importantly, on the degree to which such a performance and complexity trade-off depends on the ML task and the data also remains unknown.
In this respect, neural tangent kernels (NTKs) [28], provide a powerful tool for use in assessing the convergence and generalization properties of very wide (sometimes unrealistically so) DNNs by studying their corresponding NTK eigenspectra, which are solely dependent on the input data, the network architecture and activation function, as well as the random weights distribution.
In this paper, building upon recent advances in random matrix theory (RMT) and high-dimensional statistics, we demonstrate that for data x 1 , . . .
, x n 2 R p drawn from a K -class Gaussian mixture model (GMM), in a high-dimensional and non-trivial classiﬁcation regime where the input data dimension p and their size n are both large and comparable, the eigenspectra of both the NTK and the closely related conjugate kernel (CK) matrices at any layer ` 2 { 1 , . . .
, L } are independent of the distribution of the i.i.d.
entries of the (random) weight matrix W ` , provided that they are “normalized” to have zero mean and unit variance, and only depend on the activation function  ` ( · ) via four scalar parameters.
In a sense, we establish, at least for GMM data, the asymptotic spectral equivalence between the CK and NTK matrices of the corresponding network layers, and consequently of the whole network, for a large family of DNN models with possibly very different weights and activations, given that they have normalized weights and share the same few activation-related parameters.
Since the convergence and generalization properties of wide DNNs depend only on the eigenspectra (i.e., eigenvalue-eigenvector pairs; see also Remark 2 below) of the corresponding NTK matrices [28, 18], in the sense that, e.g., the time evolutions (when trained with gradient descent using a sufﬁciently small step size) of both the residual error and the in-sample prediction of the network can be expressed as explicit functions of NTK eigenvalues and eigenvectors [18, 1, 26], we further exploit the above theoretical results to propose a novel “lossless” compression approach for fullyconnected DNN models, by designing a sparse and quantized DNN that (i) has asymptotically the same NTK eigenspectra as the original “dense and full precision” network, and (ii) has both weights and activations taking values in the set {  1 , 0 , +1 } before scaling, and can thus be stored and computed much more efﬁciently.
Despite being derived here for Gaussian mixture data, an unexpected close match is observed between our theory and the empirical results on real-world datasets, suggesting possibly wider applicability for the proposed “lossless” compression approach.
Looking forward, we expect that our analysis will open the door to improved analysis of involved ML methods based on RMT and high-dimensional statistics, which will demystify the seemingly striking empirical observations in, say, modern DNNs.
1.1 Our contributions
Our main results can be summarized as follows:
We provide, in Theorems 1 and 2 respectively, for GMM data and in the high-dimensional regime of Assumption 1, precise eigenspectral characterizations of CK and NTK matrices of fully-connected DNNs; we particularly show that the CK and NTK eigenspectra do not depend on the distribution of i.i.d.
network weights, and depend solely on the activation function of each layer via a few scalar parameters.
In Corollary 1 and Algorithm 1, we exploit these results to propose a novel DNN compression scheme, with sparsiﬁed and ternarized weights and activations, without affecting the NTK spectral behavior, and thus the convergence and generalization properties of the network.
We provide empirical evidence on (not so) wide DNNs trained on both synthetic Gaussian and real-world datasets such as MNIST [31] and CIFAR10 [30], and show a factor of 10less memory is needed with the proposed DNN compression approach, with virtually no performance loss.
1.2 Related work
Neural network model compression.
The study of NN compression dates back to early[32], at which point, in the absence of the (possibly more than) sufﬁcient computational power that we have today, compression techniques allowed neural networks to be empirically evaluated on computers with limited computational and/or storage resources [49].
Alongside the rapid growth of increasingly powerful computing devices, the development of more efﬁcient NN architectures and training/inference protocols, and the need to implement NNs on mobile and low-power devices, (D)NN model compression has become an active research topic and many elegant and efﬁcient compression approaches have been proposed over the years [22, 25, 27, 23].
However, due to the nonlinear and highly non-convex nature of DNNs, our theoretical understanding of these largescale ML models, as well as of (e.g., the fundamental “performance and complexity” trade-off of) compressed DNNs, is somewhat limited [23].
Neural tangent kernel.
Neural tangent kernel (NTK) theory proposed in [28], by considering the limit of inﬁnitely wide DNNs, characterizes the convergence and generalization properties of very wide DNNs when trained using gradient descent with small steps.
Initially proposed for fullyconnected nets, the NTK framework has been subsequently extended to convolutional [5], graph [16], and recurrent [3] settings.
The NTK theory, while having the advantage of being mathematically more tractable (via, e.g., the characterization of the associated reproducing kernel Hilbert space [8]), seems to diverge from the regime on which modern (and not so wide) DNNs operate, see [10, 38, 18].
Random matrix theory and neural networks.
Random matrix theory (RMT), a powerful and ﬂexible tool for assessing the behavior of large-scale systems with a large “degree of freedom”, is gaining popularity in the ﬁeld of NN analysis [44, 45], in both shallow [48, 36, 37] and deep [7, 18, 47] settings, and for both homogeneous (e.g., standard normal) [48, 46] and mixture-type data [36, 4].
From a technical perspective, the most relevant paper is [4] , in which the authors proposed a
RMT-inspired NN compression scheme, albeit only in the single-hidden-layer setting.
This paper extends the analysis in [4] to multi-layer fully-connected DNNs, by focusing on both the CK and
NTK matrices, and proposes a novel sparsiﬁcation and quantization scheme for fully-connected
DNNs (which is in spirit similar to, although formally different from, that in [4]).
1.3 Notations and organization of the paper
We denote scalars by lowercase letters, vectors by bold lowercase, and matrices by bold uppercase.
We denote the transpose operator by ( · ) T , and use k · k to denote the Euclidean norm for vectors and the spectral/operator norm for matrices.
For a random variable z , E [ z ] denotes the expectation of z .
We use 1 p and I p to represent an all-ones vector of dimension p and the identity matrix of size p ⇥ p .
The remainder of this article is structured as follows.
In Section 2, we present the fully-connected
DNN model under study, together with our working assumptions.
Section 3 contains our main technical results on the eigenspectra of the conjugate kernel K CK and NTK matrix K NTK , along with an account of how they apply to the compression of fully-connected DNNs with the proposed “lossless” compression scheme.
Empirical evidence is provided in Section 4 to demonstrate the signiﬁcant computation and storage savings, with minimal performance degradation, that can be obtained using the proposed approach.
Conclusion and future perspectives are placed in Section 5.
Preliminaries
Let x 1 , . . .
, x n 2 R p be n data vectors independently drawn from one of the K -class Gaussian mixtures C 1 , . . .
, C K , and let X =
[ x 1 , . . .
, x n ] 2 R p ⇥ n , with class C a having cardinality n a ; that is,
[Equation]
(1) for mean vector µ a 2 R p and covariance matrix C a 2 R p ⇥ p associated with class C a .
In the high-dimensional scenario where n, p are both large and comparable, we position ourselves in the following non-trivial classiﬁcation setting, so that the classiﬁcation of the K -class mixture is neither trivially easy nor impossible; see also [13] and [9, Section 2].
Assumption 1 (High-dimensional asymptotics) .
As n ! 1 , we have, for a 2 { 1 , . .
, K } that (i) p/n ! c 2 (0 , 1 ) and n a /n !
c a 2 [0 , 1) ; (ii) k µ a k
= O (1) ; (iii) for C  ⌘ P
Ka =1 n a n C a and C  a ⌘ C a  C  , we have k C a k
= O (1) , tr C
a = O ( p p ) and tr( C a C b ) =
O ( p ) for a, b 2 { 1 , . . .
, K } ; and (iv) ⌧ 0 ⌘ p tr C  /p converges in (0 , 1 ) .
We consider using a fully-connected DNN model of depth L for the classiﬁcation of the above
K -class Gaussian mixture.
Such a network can be parameterized by a sequence of weight matrices
W 1 2 R d 1 ⇥ d 0 , . . .
, W L 2 R d L ⇥ d L  1 (with the convention d 0 = p ), and nonlinear activation functions  1 , . .
,  L that apply entry-wise, so that the network output f ( x ) 2 R is given by:
[Equation]
for an input data vector x 2 R p and output vector w 2 R d L .
We denote ⌃ ` 2 R d ` ⇥ n the representations of the data matrix X 2 R p ⇥ n at layer ` 2 { 1 , . . .
, L } deﬁned as
[Equation]
The normalization by 1 / p d ` follows from the NTK literature and ensures the consistent asymptotic behavior of the network in the high-dimensional setting in Assumption 1 and 2; see also [28, 8, 18].
The training and generalization performance of the NN model deﬁned in (2) are closely related to two types of kernel matrices: the Conjugate Kernel (CK) matrix and Neural Tangent Kernel (NTK) matrix, deﬁned respectively for ` 2 { 1 , . . .
, L } as follows:
[Equation]
with expectation taken with respect to the random weights W 1 , . . .
, W ` and ⌃ ` 2 R d ` ⇥ n the data representation at the output of layer ` deﬁned in (3).
In particular, CK matrices are known to satisfy the following recursive relation [28, 8]
[Equation]
while for the NTK matrix K NTK , ` 2 R n ⇥ n of layer ` , we have:
[Equation]
where ‘ A  B ’ denotes the Hadamard product between two matrices A , B of the same size, and
K 0 CK , ` denotes the CK matrix with nonlinear function  0 ` instead of  ` as for K CK , ` deﬁned in (5); that is, [ K 0 CK , ` ] ij = E u,v [  0 ` ( u )  0 ` ( v )] .
Note in particular that for a given DNN model, the corresponding CK and NTK matrices depend only on the network structure (i.e., the number of layers and the activation function in each layer), the distribution of the random (initializations of the) weights to be integrated over (e.g., in the expectation in equation (4)), and the input data.
It has been shown in a series of previous efforts [28, 18, 26] that for very (and sometimes unrealistically) wide DNNs trained using gradient descent with a small step size, the time evolution of the residual errors and in-sample predictions of a given DNN are explicit functionals of the corresponding
K NTK involving its eigenvalues and eigenvectors.
In this respect, the NTK theory provides, via the eigenspectral behavior of K NTK , precise characterizations of the convergence and generalization properties of DNNs [28, 8], by focusing on the impact of the network structure, the input data, and the weight initialization schemes.
In this paper, we focus on fully-connected nets under the following assumption regarding the weights.
Assumption 2 (On weight initializations) .
The random weights W 1 2 R d 1 ⇥ p , . . .
, W L 2 R d L ⇥ d L are independent and have i.i.d.
entries of zero mean, unit variance, and ﬁnite fourth-order moment.
Assumption 2, together with the 1 / p d ` normalization, is compatible with fully-connected DNNs in (2), which are admittedly less interesting, from a practical perspective, compared to their convolutional counterparts.
The proposed framework is envisioned to be extendable to a convolutional [5, 8] and more involved setting (e.g., graph NNs [16]) by considering (e.g., Toeplitz-type) structures on W s.
Unlike most existing NTK literature [28, 8, 17], we do not assume the Gaussianity of the entries of
W ` s, but only that they are i.i.d.
and “normalized” to have zero mean and unit variance.
As it turns out, this assumption together with a (Lyapunov-type) central limit theorem argument, is sufﬁcient to establish most existing results on the convergence and generalization of DNNs; see for example [34].
We also need the following assumption on the activation functions in each layer.
Assumption 3 (On activation functions) .
The activations  1 , . .
,  L are at least four-times differentiable with respect to standard normal measure, in the sense that max k 2 { 0 , 1 , 2 , 3 , 4 } {| E [  ( k ) ` ( ⇠ )] |} < C for some universal constant C > 0 , ⇠ ⇠ N (0 , 1) , and ` 2 { 1 , . . .
, L } .
Using the Gaussian integration by parts formula, one has E [  0 ( ⇠ )]
=
E [ ⇠ ( ⇠ )] for ⇠ ⇠ N (0 , 1) , as long as the right-hand side expectation exists.
As a result, for non-differentiable functions, it sufﬁces to have |  ` | upper-bounded by some (high-degree) polynomial function for Assumption 3 to hold.
With these preliminaries, we are now ready to present our main technical results on the eigenspectral behavior of the CK and NTK matrices for a large family of fully-connected DNN models.
Main results
For a fully-connected DNN deﬁned in (2), our ﬁrst result is on the eigenspectral behavior of the corresponding CK matrices K CK deﬁned in (4).
More speciﬁcally, we show for Gaussian mixture data in (1) and in the high-dimensional setting of Assumption 1, that the K CK , ` of layer ` 2 { 1 , . . .
, L } is asymptotically spectrally equivalent to another random matrix ˜ K CK , ` , in the sense that their spectral norm difference k K CK , `  ˜ K CK , ` k vanishes as n, p ! 1 .
This result is stated as follows, the proof of which is based on an induction on ` and is given in Section A.1 of the appendix.
Theorem 1 (Asymptotic spectral equivalents for CK matrices) .
Let Assumptions 1–3 hold, and let ⌧ 0 , ⌧ 1 , . .
⌧ L  0 be a sequence of non-negative numbers satisfying the following recursion:
[Equation]
Further assume that the activation functions  ` ( · ) s are “centered,” such that E [  ` ( ⌧ `  1 ⇠ )]
= 0 .
Then, for the CK matrix K CK , ` of layer ` 2 { 0 , 1 , . . .
, L } deﬁned in (4) , as n, p ! 1 , one has that:
[Equation]
almost surely, with
[Equation]
for class label vectors
J =
[ j 1 , . .
, j K ] 2 R n ⇥ K with [ j a ] =
x i 2 C a , second-order data ﬂuctuation vector
[Equation]
, second-order discriminative statistics t = { tr C  a / p p } Ka = [Equation]
R K and T = { tr C a C b /p } Ka,b =1 2 R K ⇥ K of the Gaussian mixture in (1) , as well as non-negative scalars ↵ ` , 1 , ↵ ` , 2 , ↵ ` , 3  0 satisfying the following recursions:
[Equation]
, 2 = E [  0 ` ( ⌧ `  1 ⇠ )] 2 ↵ `  1 , 2 + 1 4 E
{+[  00 ` ( ⌧ `  1 ⇠ )] 2 ↵ 2 `  1 , 4 , (10) ↵ ` , 
+}
= E [  0 ` ( ⌧ `  1 ⇠ )] 2 ↵ `  1 , 3 + 1 2 E [  00 ` ( ⌧ `  1 ⇠ )] 2 ↵ 2 `  1 , 1 , (11) with ↵ ` , 4 = ↵ `  1 , 4 E ⇥ (  0 ` ( ⌧ `  1 ⇠ )) {+
+} +  ` ( ⌧ `  1 ⇠ )  00 ` ( ⌧ `  1 ⇠ ) ⇤ for ⇠ ⇠ N (0 , 1) .
A few remarks on Theorem 1 are in order.
The ﬁrst remark is on the assumption E
[  ` ( ⌧ `  1 ⇠ )] = 0 .
Remark 1 (On activation centering) .
The condition E [  ` ( ⌧ `  1 ⇠ )] = 0 , seemingly restrictive at ﬁrst sight, in fact only subtracts an identical constant from all entries of the data representation ⌃ ` at layer ` , and should therefore not restrict the expressive power of the network, nor its performance on downstream ML tasks.
For a given DNN model of interest, it sufﬁces to “center” the output of each layer by subtracting a constant to satisfy Assumption 3, and to further apply our Theorem 1.
Theorem 1 unveils the (possibly surprising) fact that, for the high-dimensional and non-trivial
Gaussian mixture classiﬁcation in (1), the spectral behavior of ˜ K CK , ` , and thus that of the CK matrix K CK , ` , is (i) independent of the distribution of the (entries of the) weights W ` when they are
“normalized” to have zero mean and unit variance, as demanded in Assumption 2, and (ii) depends on the activation function  ` only via four 2 scalar parameters ↵ ` , 1 , ↵ ` , 2 , ↵ ` , 3 and ⌧ ` : such universal results have been previously observed in random matrix theory and high-dimensional statistics literature (see for example [12, 6, 56]) and indicate some kind of universality of DNN models.
On closer inspection of Theorem 1, we further observe that:
(i) for a given DNN, Theorem 1 characterizes, via the form of ˜ K CK , ` in (8) and the recursions in (10) and (11), how the linear (via ↵ ` , 1 , which is multiplied by X T X ) and nonlinear (via ↵ ` , 2 and ↵ ` , 3 in A ` , which respectively weight the second-order data statistics t and T ) data features “propagate” in a DNN, in a layer-by-layer fashion, as ` increases, as quantitatively measured by the corresponding ↵ ` s; and (ii) for two DNNs with the same number of layers, but possibly different weights and activations, given the same input data X (so that the two nets have the same K CK , 0 ), if they have asymptotically equivalent CK matrices K CK , `  1 at layer `  1 with the same ↵ `  1 , 1 , ↵ `  1 ,and ↵ `  1 , 3 , then its follows from Equation (10) and (11) that having the same E [  0 ` ( ⌧ `  1 ⇠ )] 2 , E [  00 ` ( ⌧ `  1 ⇠ )] 2 , and E [(  2 ` ( ⌧ `  1 ⇠ )) 00 ] (which only depends on the activation  ` of layer ` and ⌧ `  1 ) sufﬁces for the two nets to have asymptotically equivalent K CK , ` at layer ` .
It follows from the above item (ii) that for a given DNN of depth L , it is possible to design a novel
DNN model that “matches” the original one – in the sense that both models will have asymptotically equivalent CK matrices at each layer , by using the following layer-by-layer matching strategy:
Starting from the same K CK ,
= X T X , one chooses the ﬁrst-layer weights W 1 of the novel DNN according to Assumption 2, and then select the ﬁrst-layer activation  1 in such a way that the novel net has the same parameters ↵ 1 , 1 , ↵ 1 , 2 and ↵ 1 , 3 as the original one, so that the ﬁrst-layer CK matrices
K CK , 1 of the two nets are spectrally matched as per Theorem 1; one then proceeds similarly to match the second, the third, etc., and eventually the L th layer of the two nets.
As we shall see below, this layer-by-layer matching strategy facilitates the “lossless” compression of a given DNN.
Using the relation in (6), a similar result (as in Theorem 1 for CKs) can be established for NTK matrices, as shown in the following theorem.
The proof is also based on an induction on ` and is provided in Section A.2 of the appendix.
Theorem 2 (Asymptotic spectral equivalent for NTK matrices) .
Let Assumptions 1–3 hold, let the activations  ` ( · ) s be centered so that E [  ` ( ⌧ `  1 ⇠ )]
= 0 for ⌧ ` s deﬁned in (7) , let ˙ ⌧ 0 , ˙ ⌧ 1 , . . .
, ˙ ⌧ L be a sequence of non-negative numbers satisfying the following recursion:
[Equation]
with ˙ ⌧ 0 = ⌧ 0 (which is similar to the ⌧ ` s deﬁned in (7) , but on the derivative  0 ` instead of  ` itself), and let  1 , . . . ,  L  0 be a sequence of non-negative numbers satisfying
[Equation]
with  0 = ⌧ 0 .
Then, for the NTK matrix K NTK , ` of layer ` deﬁned in (6) , as n, p ! 1 , one has that k
[Equation]
almost surely, with V 2 R n ⇥ ( K +1) , t 2 R K , T 2 R K ⇥ K as deﬁned in Theorem 1, and
[Equation]
as well as scalars  ` , 1 ,  ` , 2 ,  ` , 3  0 , and ˙ d ` , ˙ ↵ ` , 1 such that
[Equation]
and
[Equation]
with ˙  `  1 = ˙
⌧ 2 `  1  ˙ d [Equation]  1 ˙ ⌧ [Equation]
for independent ⇠ 1 , ⇠ 2 ⇠ N (0 , 1) .
Roughly speaking, Theorem 2 shows that the eigenspectral behavior established in Theorem 1 for
CK matrices also holds for NTK matrices, up to a change of the associated coefﬁcients ↵ ` s to  ` s.
The remarks after Theorem 1 thus remain valid, at least in spirit, for NTK matrices.
Remark 2 (On spectral norm characterization) .
Note that the characterizations in Theorem 1 andfor CK and NTK matrices are provided in a spectral norm sense.
It then follows from Weyl’s inequality [24] and the Davis–Kahan theorem [61] that the difference between the eigenvalues (e.g., when listed in a decreasing order) and the associated eigenvectors (when the eigenvalues under study are “isolated”) of K NTK and ˜ K NTK vanish asymptotically as n, p ! 1 .
As such, the spectral norm guarantees in Theorem 1 and 2 provide more tractable access to the convergence and generalization properties of wide DNNs, at least for GMM data, via the spectral study of ˜ K NTK [28, 18].
Despite being derived here for the Gaussian mixture model in (1), we conjecture that the results in Theorem 1 and 2 hold beyond the Gaussian setting and extend, for example, to the family of concentrated random vectors [50, 33].
As discussed after Assumption 2 for the distribution of
W , such universality commonly arises in random matrix theory and high-dimensional statistics [12, 6, 56]; we refer interested readers to Remark 4 in Appendix A for further discussions.
From a technical perspective, the results in Theorem 1 and 2 extend the single-hidden-layer CK analysis in [4, 36] to both CK and NTK matrices of fully-connected DNNs with an arbitrary number of layers.
In particular, taking ` = 1 in Theorem 1, one obtains [4, Theorem 1] as a special case.
Remark 3 (On CK and NTK matrices) .
It follows from Theorem 1 that for a given DNN model, it sufﬁces to match the coefﬁcients ↵ ` , 1 , ↵ ` , 2 and ↵ ` , 3 in a layer-by-layer manner to propose a novel
DNN with asymptotically equivalent CK matrices.
Similarly, matching the key coefﬁcients  ` , 1 ,  ` , 2 , and  ` , 3 leads to a DNN model with asymptotically equivalent NTK matrices.
Moreover, for two nets with the same ↵ ` , 1 , ↵ ` , 2 , ↵ ` , 3 , and therefore the same CKs, it sufﬁces to match the additional ˙ d ` , ˙ ↵ ` to render the two nets asymptotically equivalent in both CK and NTK senses.
In the following corollary, we present a concrete example of how to apply the results in Theorem 1 and 2 in the design of a novel computationally and storage efﬁcient DNN, that shares the same
CK and NTK eigenspectra with any given fully-connected neural net having centered activation.
Corollary 1 (Sparse and quantized DNNs) .
For a given fully-connected DNN (referred to as DNN1 ) of depth L with centered activation such that E [  ` ( ⌧ `  1 ⇠ )] = 0 for ⇠ ⇠ N (0 , 1) , one is able to construct, say in a layer-by-layer manner, a sparse and quantized “equivalent” DNN model, of depth
L and referred to as DNN2 , such that the two nets have asymptotically the same eigenspectra for their CK (and NTK similarly, as per Remark 3) matrices, by using the following ternary weights:
[Equation]
as well as quantized activations (as visually displayed in Figure 1):
[Equation]
We refer readers to Appendix A.4 for the proof and discussions of Corollary 1, as well as the detailed expressions of E [  0 ( ⌧⇠ )] , E [  00 ( ⌧⇠ )] , and E [(  2 ( ⌧⇠ )) 00 ] , of direct algorithmic use for both  T and  Q as functions of the parameters a, s 1 , s 2 and b 1 , b 2 , r 1 , r 2 , r 3 , r 4 .
Built upon Corollary 1, we propose a DNN “lossless” compression scheme with equivalent CKs, as summarized in Algorithm 1 below.
[Figure]
Solve, with Corollary 1 and the detailed expressions in Appendix A.4, the system of equations ( ↵ ` , 1 , ↵ ` , 2 , ↵ ` , 3 ) =
(˜ ↵ ` , 1 , ˜ ↵ ` , 2 , ˜ ↵ s s
˜
` of
DNN compute ↵ L
↵ L
↵ L
⌧ L
˜ ↵ L,
˜ ↵ L, ˜ ↵ L, 3 , ˜ ⌧ L solve for parameters
b 1 , b 2 , r 1 , r 2 , r 3 , r 4 to obtain the activation ˜  L of DNN2 at layer ` i i.d entries
As a side remark, note that the “sign” of activations does not matter in Corollary 1 or Algorithm 1, in the sense that the key parameters ↵ ` , 1 , ↵ ` , 2 , and ↵ ` , 3 for CKs, as well as  ` , 1 ,  ` , 2 , and  ` , 3 for
NTKs, remain unchanged when   ` ( t ) is used instead of  ` ( t ) .
Before embarking on the detailed numerical experiments in Section 4, we would like to bring the readers’ attention to the recent line of works [35, 53, 57, 54, 19] showing that for wide and deep NN models, very efﬁcient sparse sub-networks can be found that almost match the performance of the original dense nets with little or even no training , for instance by uniformly pruning the network weights [53].
To develop a theoretical grasp of these (extremely counterintuitive) empirical successes, a few attempts have been made, for example, to carefully prune the network weights to retain the same (limiting) NTK [40], or to show that randomly pruned sparse nets have the same (limiting)
NTK as the original net up to a scaling factor [60].
Instead, our work, by considering the statistical structure of the input data, leverages tools from RMT to “compress” both the weights and activations (per Theorem 2 and Corollary 1) without affecting the NTK eigenstructure.
Numerical experiments
In this section, we provide numerical experiments to (i) validate the asymptotic characterizations in
Theorem 1 and 2, on both synthetic GMM and real-world data (such as MNIST and CIFAR10) of (in fact not so) large sizes and dimensions; and to (ii) show how these results can be used to sparsify and quantize fully-connected DNNs, leading to huge savings in computational and storage resources (up to a factor of 10 3 in memory and a level of sparsity " = 90% ) without signiﬁcant performance degradation.
We refer readers to Section B in the appendix for further experiments and discussions.
[Figure]
Eigenvalue
C oun t
Eigs of K
[Figure]
Eigenvalue
C oun t
Eigs of ˜ K
[Figure]
Eigenvalue
C oun t
Eigs of K      6 ·
Eigenvalue
C oun t
Eigs of K
[Figure]
Eigenvalue
C oun t
Eigs of ˜ K[-30  6 · 
-]
[Figure]
Eigenvalue
C oun t
Eigs of ˜ K
[Figure]
Figure 2 compares the eigenvalues and dominant eigenvectors of the CK matrices K CK deﬁned in (4) versus those of their asymptotic approximations ˜ K CK given in Theorem 1, in the case of fully-connected DNNs having three hidden layers (of width d 1 = 2 000 , d 2 = 2 000 , d 3 = 1 000 ).
For different types of activations: poly( t ) = 0 .
2 t 2 + t and ReLU( t ) =
max( t, 0) , different weight distributions (Gaussian and symmetric Bernoulli), and on synthetic GMM as well as MNIST data, we consistently observe a close match between the eigenvalues and dominant eigenvectors of K CK and ˜
K CK , as a consequence of the spectral norm convergence in Theorem 1 (and Remark 2), suggesting a possibly wider applicability of the proposed results beyond GMM data.
Following the idea of CK matching in Figure 2, Figure 3 depicts the test accuracies of (i) the original dense and unquantized network with three fully-connected layers, (ii) the proposed “lossless” compression scheme described in Corollary 1 and Algorithm 1 via CK matching, as well as its variant having ternarized weights but dense and unquantized activations, (iii) the popular magnitudebased pruning approach as in [20], together with (iv) two “heuristic” compression approaches: (iv-i) sparsiﬁcation by uniformly zeroing out 80% of the weights (we cannot do more, as the resultant performance is too poor to be visually compared with other curves) , and (iv-ii) binarization using  ( t ) = 1 t<  1 +1 t> 1 , for different choices of width per layer, and the ten-class classiﬁcation problems of MNIST and CIFAR10.
Speciﬁcally, neural networks before and after compression have three fully-connected layers, and the original network uses ReLU activation in each layer.
Classiﬁcation is performed on a concatenated and trainable fully-connected layer.
For MNIST datasets, raw data are taken as the network input; for CIFAR10 dataset, ﬂattened output of the 16th convolutional layer of
VGG19 [52] are taken as the network input.
We observe from Figure 3 that the proposed “lossless” compression scheme occupies (up to) a factor of 10 3 less memory, and produces signiﬁcantly sparser networks (up to 90% of weights set to zero) with minimal performance loss, when compared to the original or the heuristically compressed nets.
In addition, when compared with its ternary weights variant (with unquantized activations), the proposed sparse and quantized DNN model trades limited performance drop for inference accelerating, and achieves higher accuracies than the popular magnitude-based pruning under the same memory budget.
We also see that the sparsity level " has limited impact on the classiﬁcation accuracy, as in line with our theory.
These experimental results show that the proposed approach achieves a better performance-complexity trade-off than commonly used heuristic DNN compression methods.
[Figure]
original [Figure] quantized
[Figure]
sparse mag-based pruning ternary
[Figure]
In this paper built upon recent advances in random matrix theory and high-dimensional statistics, we provide precise characterizations of the eigenspectra of both conjugate kernel and neural tangent kernel matrices, for high-dimensional Gaussian mixture data and fully-connected multi-layer neural nets.
These results further allows us to sparsify and quantize fully-connected deep nets, resulting in a factor of 10 3 less memory consumption with virtually no performance degradation.
Extending the present theoretical framework to more involved settings such as convolutional nets requires reﬁned analysis on the block Toeplitz weights W and on their connection to the corresponding
CK and NTK matrices [5, 59].
Also, since the NTK eigenspectra determine the gradient descent dynamics of ultra-wide deep nets, the asymptotic characterizations in Theorem 2 can be applied to assess the learning dynamics [37, 2, 2] of fully-connected DNN models, in and possibly beyond the inﬁnitely wide NTK regime.
Acknowledgments and Disclosure of Funding
ZL would like to acknowledge the CCF-Hikvision Open Fund (20210008), the National Natural
Science Foundation of China (via fund NSFC-62206101 and NSFC-12141107), the Fundamental
Research Funds for the Central Universities of China (2021XXJS110), the Key Research and Development Program of Hubei (2021BAA037), and the Key Research and Development Program of
Guangxi (GuiKe-AB21196034) for providing partial support.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[-. , n } that, Putting everything together in matrix form, we obtain, as in the proof of Theoremin Appendix A.
-]
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[-covariance-oriented activations with α 1 = 
-]
[-mean-oriented activations with α 2 = 0 and α 3 = 
-]