R ANDOM M ATRIX T HEORY P ROVES THAT D EEP L EARNING R EPRESENTATIONS OF GAN- DATA B EHAVE AS G AUSSIAN M IXTURES
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called concentrated random vectors.
Further exploiting the fact that Gram matrices, of the type G = X (cid:124)
X with X =
[ x 1 , . . .
, x n ] ∈ R p × n and x i independent concentrated random vectors from a mixture model, behave asymptotically (as n, p → ∞ ) as if the x i were drawn from a Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their ﬁrst two statistical moments for a wide range of standard classiﬁers.
Our theoretical ﬁndings are validated by generating images with the BigGAN model and across different popular deep representation networks
I NTRODUCTION
The performance of machine learning methods depends strongly on the choice of the data representation (or features) on which they are applied.
This data representation should ideally contain relevant information about the learning task in order to achieve learning with simple models and small amount of samples.
Deep neural networks (Rumelhart et al., 1988) have particularly shown impressive results by automatically learning representations from raw data ( e.g. , images).
However, due to the complex structure of deep learning models, the characterization of their hidden representations is still an open problem (Bengio et al., 2009).
Speciﬁcally, quantifying what makes a given deep learning representation better than another is a fundamental question in the ﬁeld of Representation Learning (Bengio et al., 2013).
Relying on (Montavon et al., 2011) a data representation is said to be good when it is possible to build simple models on top of it that are accurate for the given learning problem.
Montavon et al.
(2011) have notably quantiﬁed the layer-wise evolution of the representation in deep networks by computing the principal components of the Gram matrix G (cid:96) =
{ φ (cid:96) ( x i ) (cid:124) φ (cid:96) ( x j ) } n i,j =1 at each layer for n input data x 1 , . . .
, x n , where φ (cid:96) ( x ) is the representation of x at layer (cid:96) of the given DL model, and the number of components controls the model simplicity.
In their study, the impact of the representation at each layer is quantiﬁed through the prediction error of a linear predictor trained on the principal subspace of G (cid:96) .
Pursuing on this idea, given a certain representation model x (cid:55)→ φ ( x ) , we aim in this article at theoretically studying the large dimensional behavior, and in particular the spectral information ( i.e. , eigenvalues and dominant eigenvectors), of the corresponding Gram matrix G = { φ ( x i ) (cid:124) φ ( x j ) } ni,j =1 in order to determine the information encoded ( i.e. , the sufﬁcient statistics) by the representation model on a set of real data x 1 , . . .
, x n .
Indeed, standard classiﬁcation and regression algorithms –along with the last layer of a neural network (Yeh et al., 2018)– retrieve the data information directly from functionals or the eigenspectrum of G 1 .
To this end, though, one needs a statistical model for the representations given the distribution of the raw data ( e.g. , images) which is generally unknown.
Yet, due to recent advances in generative models since the advent of
Generative Adversarial Nets (Goodfellow et al., 2014), it is now possible to generate complex data structures by applying successive Lipschitz operations to Gaussian random vectors.
In particular,
GAN-data are used in practice as substitutes of real data for data augmentation (Antoniou et al., 2017).
On the other hand, the fundamental concentration of measure phenomenon (Ledoux, 2005) tells us that Lipschitz-ally transformed Gaussian vectors satisfy a concentration property.
Precisely, deﬁning the class of concentrated vectors x ∈ E through concentration inequalities of f ( x ) , for any real Lipschitz observation f : E → R , implies that deep learning representations of GAN-data fall within this class of random vectors, since the mapping x
(cid:55)→ φ ( x ) is Lipschitz.
Thus, GAN-data are concentrated random vectors and thus an appropriate statistical model of realistic data.
Targeting classiﬁcation applications by assuming a mixture of concentrated random vectors model, this article studies the spectral behavior of Gram matrices G in the large n, p regime.
Precisely, we show that these matrices have asymptotically (as n, p → ∞ with p/n → c < ∞ ) the same ﬁrstorder behavior as for a Gaussian Mixture Model (GMM).
As a result, by generating images using the
BigGAN model (Brock et al., 2018) and considering different commonly used deep representation models, we show that the spectral behavior of the Gram matrix computed on these representations is the same as on a GMM model with the same p -dimensional means and covariances.
A surprising consequence is that, for GAN data, the aforementioned sufﬁcient statistics to characterize the quality of a given representation network are only the ﬁrst and second order statistics of the representations.
This behavior is shown by simulations to extend beyond random GAN-data to real images from the
Imagenet dataset (Deng et al., 2009).
The rest of the paper is organized as follows.
In Section 2, we introduce the notion of concentrated vectors and their main properties.
Our main theoretical results are then provided in Section 3.
In
Section 4 we present experimental results.
Section 5 concludes the article.
Notation: In the following, we use the notation from (Goodfellow et al., 2016).
[ n ] denotes the set { 1 , . . . , n } .
Given a vector x ∈ R n , the (cid:96) 2 -norm of x is given as (cid:107) x (cid:107) 2 = (cid:80) ni
=1 x 2 i .
Given a p × n matrix M , its Frobenius norm is deﬁned as (cid:107) M (cid:107) 2 F = (cid:80) pi =1 (cid:80) nj =1 M 2 ij and its spectral norm as (cid:107) M (cid:107) =
sup (cid:107) x (cid:107) =
1 (cid:107) Mx (cid:107) .
(cid:12) for the Hadamard product.
An application F : E → F is said to be (cid:107)F(cid:107) lip -Lipschitz, if ∀ ( x , y ) ∈
E 2 , (cid:107)F ( x ) − F ( y ) (cid:107) F ≤ (cid:107)F(cid:107) lip · (cid:107) x − y (cid:107) E and (cid:107)F(cid:107) lip is ﬁnite.
B ASIC NOTIONS OF CONCENTRATED VECTORS
Being the central tool of our study, we start by introducing the notion of concentrated vectors.
While advanced concentration notions have been recently developed in (Louart & Couillet, 2019) in order to speciﬁcally analyze the behavior of large dimensional sample covariance matrices, for simplicity, we restrict ourselves here to the sufﬁcient so-called q -exponentially concentrated random vectors.
Deﬁnition 2.1 ( q -exponential concentration) .
Given a normed space ( E, (cid:107) · (cid:107) E ) and a real q , a random vector x ∈ E is said to be q -exponentially concentrated if for any 1 -Lipschitz real function f : E → R , there exists C ≥ 0 independent of dim( E ) and σ > 0 such that for all t ≥
[Equation]
(1) which we denote x ∈ E q ( σ | E, (cid:107) · (cid:107) E ) .
We simply write x ∈ E q (1 | E, (cid:107) · (cid:107) E ) if the tail parameter σ does not depend on dim( E ) , and x ∈ E q (1) for x a scalar real random variable.
Therefore, concentrated vectors are deﬁned through the concentration of any 1 -Lipschitz real scalar “observation”.
One of the most important examples of concentrated vectors are standard Gaussian vectors.
Precisely, we have the following proposition.
See (Ledoux, 2005)) for more examples such as uniform and Gamma distribution.
Proposition 2.2 (Gaussian vectors (Ledoux, 2005)) .
Let d ∈ N and x ∼ N (0 , I d ) .
Then x is a 2 -exponentially concentrated vector independently on the dimension d , i.e. x ∈ E 2 (1 | R d , (cid:107) · (cid:107) ) .
Concentrated vectors have the interesting property of being stable by application of R d → R p vector-Lipschitz transformations.
Indeed, Lipschitz-ally transformed concentrated vectors remain concentrated according to the following proposition.
Proposition 2.3 (Lipschitz stability (Louart & Couillet, 2019)) .
Let x ∈ E q (1 | E, (cid:107) · (cid:107) E ) and
G : E → F a Lipschitz application with Lipschitz constant (cid:107)G(cid:107) lip which may depend on dim( F ) .
Then the concentration property on x is transferred to G ( x ) , precisely
[Equation]
Note importantly for the following that the Lipschitz constant of the transformation G must be controlled, in order to constrain the tail parameter of the obtained concentration.
In particular, we have the coming corollary to Proposition 2.3 of central importance in the following.
Corollary 2.4.
Let G 1 , . .
G n : R d → R p a set of n Lipschitz applications with Lipschitz constants (cid:107)G i (cid:107) lip .
Let G : R d × n → R p × n be deﬁned for each X ∈ R d × n as G ( X ) =
[ G 1 ( X : , 1 ) , . . . ,
G n ( X : ,n )] .
Then,
[Equation]
Proof.
This is a consequence of Proposition 2.3 since the map G is sup i (cid:107)G i (cid:107) lip -Lipschitz with respect to (w.r.t.)
the Frobenius norm.
Indeed, for X , H ∈ R d × n : (cid:107)G ( X + H ) − G ( X ) (cid:107) 2 F ≤ (cid:80) ni =1 (cid:107)G i (cid:107) 2 lip · (cid:107) H : ,i (cid:107) 2 ≤ sup i (cid:107)G i (cid:107) 2 lip · (cid:107) H (cid:107) 2 F .
M AIN R ESULTS 3. GAN DATA : A N E XAMPLE OF C ONCENTRATED V ECTORS
Concentrated random vectors are particularly interesting from a practical standpoint for real data modeling.
In fact, unlike simple Gaussian vectors, the former do not suffer from the constraint of having independent entries which is quite a restrictive assumption when modeling real data such as images or their non-linear features ( e.g. , DL representations).
The other modeling interest of concentrated vectors lies in their being already present in practice as alternatives to real data.
Indeed, adversarial neural networks (GANs) have the ability nowadays to generate random realistic data (for instance realistic images) by applying successive Lipschitz operations to standard Gaussian vectors (Goodfellow et al., 2014).
A GAN architecture involves two networks, a generator model which maps random Gaussian noise to new plausible synthetic data and a discriminator model which classiﬁes real data as real (from the dataset) or fake (for the generated data).
The discriminator is updated directly through a binary classiﬁcation problem, whereas the generator is updated through the discriminator.
As such, the two models are trained alternatively in an adversarial manner, where the generator seeks to better deceive the discriminator and the former seeks to better identify the fake data (Goodfellow et al., 2014).
In particular, once both models are trained (when they reach a Nash equilibrium), DL representations of GAN-data –and GAN-data themselves– are schematically constructed in practice as follows:
[Equation]
where d stands for the input dimension of the generator model, N the number of layers, and
the
F i ’s either Fully Connected Layers, Convolutional Layers, Pooling Layers, Up-sampling Layers and Activation Functions, Residual Layers or Batch Normalizations.
All these operations happen to be Lipschitz applications.
Precisely,
[Figure]
• Fully Connected Layers and Convolutional Layers: These are afﬁne operations which can be expressed as
[Equation]
, for W i the weight matrix and b i the bias vector.
Here the Lipschitz constant is the operator norm (the largest singular value) of the weight matrix W i , that is (cid:107)F i (cid:107) lip =
sup u (cid:54) =
(cid:107) W i u (cid:107) 2 (cid:107) u (cid:107) 2 .
• Pooling Layers and Activation Functions:
Most commonly used activation functions and pooling operations are
[Equation]
where S i ’s are patches ( i.e. , subsets of [dim( x )] ).
These are at most 1 -Lipschitz operations with respect to the Frobenius norm.
Speciﬁcally, the maximum absolute sub-gradient of the ReLU activation function is 1 , thus the ReLU operation has a Lipschitz constant of 1 .
Similarly, we can show that the Lipschitz constant of MaxPooling layers is also 1 .
• Residual Connections: Residual layers act the following way
[Equation]
where the F ( j ) i ’s are Fully Connected Layers or Convolutional Layers with Activation Functions, and which are Lipschitz operations.
Thus F i is a Lipschitz operation with Lipschitz constant bounded by 1 + (cid:81) (cid:96)j =1 (cid:107)F ( j ) i (cid:107) lip .
• Batch Normalization (BN) Layers: They consist in statistically standardizing (Ioffe & Szegedy, 2015) the vectors of a small batch B = { x i } bi =1 ⊂ R d as follows: for each x k ∈ B
[Equation]
a (cid:112) σ 2 B + (cid:15) (cid:33) ( x k − µ B 1 d ) + b where µ B = 1 db (cid:80) bk
=1 (cid:80) di
=1 [ x k ] i , σ 2 B = 1 db (cid:80) bk
=1 (cid:80) di
=1 ([ x k ] i − µ B ) 2 , a , b ∈ R d are parameters to be learned and diag ( v ) transforms a vector v to a diagonal matrix with its diagonal entries being those of v .
Thus BN is a Lipschitz transformation with Lipschitz constant (cid:107)F i (cid:107) lip = sup i | a i √ σ 2 B + (cid:15) | .
Therefore, as illustrated in Figure 1, since standard Gaussian vectors are concentrated vectors as mentioned in Proposition 2.2 and since the notion of concentrated vectors is stable by Lipschitz transformations thanks to Proposition 2.3, GAN-data (and their DL representations) are concentrated vectors by design given the construction in Equation (4).
Moreover, in order to generate data belonging to a speciﬁc class, Conditional GANs have been introduced (Mirza & Osindero, 2014); once again data generated by these models are concentrated vectors as a consequence of
Corollary 2.4.
Indeed, a generator of a Conditional GAN model can be seen as a set of multiple generators where each generates data of a speciﬁc class conditionally on the class label ( e.g. , BigGAN model (Brock et al., 2018)).
Yet, in order to ensure that the resulting Lipschitz constant of the combination of the above operations does not scale with the network or data size, so to maintain good concentration behaviors, a careful control of the learned network parameters is needed.
This control happens to be already considered in practice in order to ensure the stability of GANs during the learning phase, notably to generate realistic and high-resolution images (Roth et al., 2017; Brock et al., 2018).
The control of the Lipschitz constant of representation networks is also needed in practice in order to make them robust against adversarial examples (Szegedy et al., 2013; Gulrajani et al., 2017).
This control is particularly ensured through spectral normalization of the afﬁne layers (Brock et al., 2018), such as Fully Connected Layers, Convolutional Layers and Batch Normalization.
Indeed, spectral normalization (Miyato et al., 2018) consists in applying the operation W ← W /σ 1 ( W ) to the afﬁne layers at each backward iteration of the back-propagation algorithm, where σ 1 ( W ) stands for the largest singular value of the weight matrix W .
Brock et al.
(2018), have notably observed that, without spectral constraints, a subset of the generator layers grow throughout their GAN training and explode at collapse.
They thus suggested the following spectral normalization –which happens to be less restrictive than the standard spectral normalization W ← W /σ 1 ( W ) (Miyato et al., 2018)– to the afﬁne layers:
[Equation]
[Figure]
L a r g e s t
s
i ngu l a r v a l u e σ
WithoutSN WithSN
Theoreticalbound where u 1 ( W ) and v 1 ( W ) denote respectively the left and right largest singular vectors of W , and σ ∗ is an hyper-parameter ﬁxed during training.
To get an insight about the inﬂuence of this operation and to ensure that it controls the Lipschitz constant of the generator, the following proposition provides the dynamics of a random walk in the space of parameters along with the spectral normalization in Equation (5).
Indeed, since stochastic gradient descent (SGD) consists in estimating the gradient of the loss function on randomly selected batches of data, it can be assimilated to a random walk in the space of parameters (Antognini &
Sohl-Dickstein, 2018).
Proposition 3.1 (Lipschitz constant control) .
Let σ ∗ > 0 and G be a neural network composed of N afﬁne layers, each one of input dimension d i − 1 and output dimension d i for i ∈
[ N ] , with 1 -Lipschitz activation functions.
Assume that the weights of G at layer i + 1 are initialized as U ([ − 1 √ d
i , 1 √ d i ]) , and consider the following dynamics with learning rate η :
[Equation]
Then, ∀ ε > 0 , the Lipschitz constant of G is bounded at convergence with high probability as:
[Equation]
Proof.
The proof is provided in Appendix B.
Proposition 3.1 shows that the Lipschitz constant of a neural network is controlled when trained with the spectral normalization in Equation (5).
In particular, recalling the notations in Proposition 3.1, in the limit where d i → ∞ with d i d i − 1 → γ i ∈ (0 , ∞ ) for all i ∈ [ N ] and choosing the learning rate η
= O ( d − 1 0 ) , the Lipschitz constant of G is of order O (1) if it has ﬁnitely many layers N and σ ∗ is constant.
Therefore, with this spectral normalization, it can be assumed that (cid:107)G(cid:107) lip =
O (1) when dimensions grow.
Figure 2 depicts the behavior of the Lipschitz constant of a linear layer with and without spectral normalization in the setting of Proposition 3.1, which conﬁrms the obtained bound.
3. M IXTURE OF C ONCENTRATED V ECTORS
In this section, we assume data to be a mixture of concentrated random vectors with controlled O (1)
Lipschitz constant ( e.g. , DL representations of GAN-data as we discussed in the previous section).
Precisely, let x 1 , . . .
, x n be a set of mutually independent random vectors in R p .
We suppose that these vectors are distributed as one of k classes of distribution laws µ 1 , . . . , µ k with distinct means { m (cid:96) } k(cid:96)
=1 and “covariances” { C (cid:96) } k(cid:96)
=1 deﬁned receptively as
[Equation]
i ] .
For some q > 0 , we consider a q -exponential concentration property on the laws µ (cid:96) , in the sense that for any family of independent vectors y 1 , . . .
, y s sampled from µ (cid:96) , [ y 1 , . . .
, y s ] ∈ E q (1 | R p × s , (cid:107) · (cid:107) F ) .
Without loss of generality, we arrange the x i ’s in a data matrix X
=
[ x 1 , . . .
, x n ] such that, for each (cid:96) ∈ [ k ] , x 1+ (cid:80) (cid:96) − 1 j =1 n j , . . .
, x (cid:80) (cid:96)j =1 n j ∼ µ (cid:96) , where n (cid:96) stands for the number of x i ’s sampled from µ (cid:96) .
In particular, we have the concentration of X as
[Equation]
Such a data matrix X can be constructed through Lipschitz-ally transformed Gaussian vectors ( q = 2 ), with controlled Lipschitz constant, thanks to Corollary 2.4.
In particular, DL representations of GAN-data are constructed as such, as shown in Section 3.1.
We further introduce the following notations that will be used subsequently.
[Equation]
where j (cid:96) ∈ R n stands for the canonical vector selecting the x i ’s of distribution µ (cid:96) , deﬁned by ( j (cid:96) )
i =
1 x i ∼ µ (cid:96) , and the z i ’s are the centered versions of the x i ’s, i.e. z i
=
x
i − m (cid:96) for x i ∼ µ (cid:96) .
[Equation]
Now we study the behavior of the Gram matrix G = 1 p X (cid:124)
X in the large n, p limit and under the model of the previous section.
Indeed, G appears as a central component in many classiﬁcation, regression and clustering methods.
Precisely, a ﬁner description of the behavior of G provides access to the internal functioning and performance evaluation of a wide range of machine learning methods such as Least Squares SVMs (AK et al., 2002), Semi-supervised Learning (Chapelle et al., 2009) and Spectral Clustering (Ng et al., 2002).
Indeed, the performance evaluation of these methods has already been studied under GMM models in (Liao & Couillet, 2017; Mai & Couillet, 2017; Couillet & Benaych-Georges, 2016) through RMT.
On the other hand, analyzing the spectral behavior of
G for DL representations quantiﬁes their quality –through its principal subspace (Montavon et al., 2011)– as we have discussed in the introduction.
In particular, the Gram matrix decomposes as
[Equation]
Intuitively G decomposes as a low-rank informative matrix containing the class canonical vectors through J and a noise term represented by the other matrices and essentially Z (cid:124) Z .
Given the form of this decomposition, RMT predicts –through an analysis of the spectrum of G and under a GMM model (Benaych-Georges & Couillet, 2016)– the existence of a threshold ξ function of the ratio p/n and the data statistics for which the dominant eigenvectors of G contain information about the classes only when (cid:107) M
(cid:124) M (cid:107) ≥ ξ asymptotically ( i.e. , only when the means of the different classes are sufﬁciently distinct).
In order to characterize the spectral behavior ( i.e. , eigenvalues and leading eigenvectors) of G under the concentration assumption in Equation (9) on X , we will be interested in determining the spectral distribution L = 1 n (cid:80) ni =
δ λ i of G , with λ 1 , . . .
, λ n the eigenvalues of G , where δ x stands for the
Dirac measure at point x .
Essentially, to determine the limiting eigenvalue distribution as p, n → ∞ and p/n → c ∈ (0 , ∞ ) , a conventional approach in RMT consists in determining an estimate of the
Stieltjes transform (Silverstein & Choi, 1995) m L of L , which is deﬁned for some z ∈ C \ Supp( L )
[Equation]
Hence, quantifying the behavior of the resolvent of G deﬁned as R ( z ) =
( G + z I n ) − 1 determines the limiting measure of L through m L ( z ) .
Furthermore, since R ( z ) and G share the same eigenvectors with associated eigenvalues
[Equation]
, the projector matrix corresponding to the top m eigenvectors
U =
[ u 1 , . . . , u m ] of G can be calculated through a Cauchy integral UU (cid:124)
[Equation] [Equation]
(cid:72)
[Equation]
R ( − z ) dz where γ is an oriented complex contour surrounding the top m eigenvalues of G .
To study the behavior of R ( z ) , we look for a so-called deterministic equivalent (Hachem et al., 2007)
˜ R ( z ) for R ( z ) , which is a deterministic matrix that satisﬁes for all A ∈ R n × n and all u , v ∈
R n of respectively bounded spectral and Eucildean norms, 1 n tr( AR ( z )) − 1 n tr( A ˜ R ( z ))
→and u (cid:124)
( R ( z ) − ˜ R ( z )) v → 0 almost surely as n → ∞ .
In the following, we present our main result which gives such a deterministic equivalent under the concentration assumption on X in
Equation (9) and under the following assumptions.
Assumption 3.2.
As p → ∞ , 1.
p/n → c ∈ (0 , ∞ ) , 2.
The number of classes k is bounded,
[Equation]
Theorem 3.3 (Deterministic Equivalent for R ( z ) ) .
Under the model described in Section 3.2 and
Assumptions 3.2, we have R ( z ) ∈
E q ( p − 1 / 2 | R n × n , (cid:107) · (cid:107) F ) .
Furthermore,
[Equation]
) with Ω z = M (cid:124) ˜ Q ( z ) M
[Equation]
δ
[Equation]
∗
[Equation]
(cid:96) =1 and ˜
[Equation]
=
[Equation]
where δ ∗ ( z ) =
[ δ ∗ 1 ( z ) , . . .
, δ ∗ k
( z )] (cid:124) is the unique ﬁxed point of the system of equations
[Equation]
Sketch of proof.
The ﬁrst step of the proof is to show the concentration of R ( z ) .
This comes from the fact that the application X (cid:55)→ R ( z ) is 2 z − 3 / 2 p − 1 / 2 -Lipschitz w.r.t.
the Frobenius norm, thus we have by Proposition 2.
that R ( z ) ∈
E q ( p − 1 / 2 | R n × n , (cid:107) · (cid:107) F ) .
The second step consists in estimating E R ( z ) through a deterministic matrix ˜ R ( z ) .
Indeed, R ( z ) can be expressed as a function of Q ( z ) =
( XX (cid:124) /p
+ z I p ) − 1 as R ( z ) =
z − 1 ( I n − X (cid:124) Q ( z ) X /p ) , and exploiting the result of (Louart & Couillet, 2019) which shows that E Q ( z ) can be estimated through ˜ Q ( z ) , we obtain the estimator ˜ R ( z ) for E R ( z ) .
A more detailed proof is provided in Section A.3 of the Appendix.
This result allows speciﬁcally to (i) describe the limiting eigenvalues distribution of G , (ii) determine the spectral detectability threshold mentioned above, (iii) evaluate the asymptotic “content” of the leading eigenvectors of G and, much more fundamentally, (iv) infer the asymptotic performances of machine learning algorithms that are based on simple functionals of G ( e.g. , LS-SVM, spectral clustering etc.).
Looking carefully at Theorem 3.3 we see that the spectral behavior of the Gram matrix G computed on concentrated vectors only depends on the ﬁrst and second order statistics of the laws µ (cid:96) (their means m (cid:96) and “covariances” C (cid:96) ).
This suggests the surprising result that G has the same behavior as when the data follow a GMM model with the same means and covariances.
The asymptotic spectral behavior of G is therefore universal with respect to the data distribution laws which satisfy the aforementioned concentration properties (for instance DL representations of
GAN-data).
We illustrate this universality result in the next section by considering data as CNN representations of GAN generated images.
[Figure]
G A
N
I m a g e s
R e a l
I m a g e s
A PPLICATION TO CNN R EPRESENTATIONS OF GAN- GENERATED I MAGES
In this section, we consider n = 1500 data x 1 , . . .
, x n ∈ R p as CNN representations –across popular CNN architectures of different sizes p – of GAN-generated images using the generator of the
Big-GAN model (Brock et al., 2018).
We further use real images from the Imagenet dataset (Deng et al., 2009) for comparison.
In particular, we empirically compare the spectrum of the Gram matrix of this data with the Gram matrix of a GMM model with the same means and covariances.
We also consider the leading 2 -dimensional eigenspace of the Gram matrix which contains clustering information as detailed in the previous section.
Figure 3 depicts some images generated using the
Big-GAN model (Top) and the corresponding real class images from the Imagenet dataset (Bottom).
The Big-GAN model is visually able to generate highly realistic images which are by construction concentrated vectors, as discussed in Section 3.1.
Figure 4 depicts the spectrum and leading 2D eigenspace of the Gram matrix computed on CNN representations of GAN generated and real images (in gray), and the corresponding GMM model with same ﬁrst and second order statistics (in green).
The Gram matrix is seen to follow the same spectral behavior for GAN-data as for the GMM model which is a natural consequence of the universality result of Theorem 3.3 with respect to the data distribution.
Besides, and perhaps no longer surprisingly, we further observe that the spectral properties of G for real data (here CNN representations of real images) are conclusively matched by their Gaussian counterpart.
This both theoretically and empirically conﬁrms that the proposed random matrix framework is fully compliant with the theoretical analysis of real machine learning datasets.
C ONCLUSION
Leveraging on random matrix theory (RMT) and the concentration of measure phenomenon, we have shown through this paper that DL representations of GAN-data behave as Gaussian mixtures for linear classiﬁers, a fundamental universal property which is only valid in high-dimension of data.
This result constitutes a ﬁrst step towards the theoretical understanding of complex objects such as
DL representations as well as the understanding of the behavior of more elaborate machine learning algorithms for complex data structures.
In addition, the article explicitly demonstrated our ability, through RMT, to anticipate the behavior of a wide range of standard classiﬁers for data as complex as DL representations of the realistic and surprising images generated by GANs.
This opens the way to a more systematic analysis and improvement of machine learning algorithms on real datasets by means of large dimensional statistics
A P ROOF OF T HEOREM 3. A. S ETTING OF THE PROOF
For simplicity we will only suppose the case k =
and we consider the following notations that will be used subsequently.
[Equation]
Let
[Equation]
the matrix X with a vector of zeros at its i th column.
Denote the resolvents
[Equation]
And let
[Equation]
where δ is the solution to the ﬁxed point equation
[Equation]
A. B ASIC TOOLS
Lemma A.1 ((Ledoux, 2005)) .
Let z ∈ E q (1 | R p , (cid:107) · (cid:107) ) and M ∈ E q (1 | R p × n , (cid:107) · (cid:107) F ) .
Then, for some numerical constant C >
• E (cid:107) z (cid:107) ≤ (cid:107) E z (cid:107) + C √ p, E (cid:107) z (cid:107) ∞ ≤ (cid:107) E z (cid:107) ∞ + C √ log
p. •
E (cid:107) M (cid:107) ≤ (cid:107) E M (cid:107) + C √ p + n, E (cid:107) M (cid:107) F ≤ (cid:107) E M (cid:107) F + C √ pn.
Lemma A.2.
Denote Q ¯ x = (¯ x ¯ x (cid:124)
+ z I p ) − 1 , we have:
Q ¯ x ¯ x = ¯ x (cid:107) ¯ x (cid:107)
+ z and (cid:107) ˜ Q
¯ x (cid:107) , ¯ x ˜ Q ¯ x = O (1) .
Moreover, if (cid:107) ¯ x (cid:107) ≥ √ p , (cid:107) ˜
Q ¯ x (cid:107) =
O ( p − 1 / 2 ) .
Proof.
Since z Q ¯ x = I p − Q ¯ x ¯ x ¯ x (cid:124) :
z Q ¯ x ¯ x = ¯ x − (cid:107) ¯ x (cid:107) 2 Q ¯ x ¯ x , and we recover the ﬁrst identity of the Lemma.
And since the matrix C 0 is nonnegative symmetric, we have :
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation] + zc [Equation] 1 + δ ) .
Therefore, ¯ x ˜ Q ¯ x = c (1+ δ ) (cid:107) ¯ x (cid:107) 2 (cid:107) ¯ x (cid:107) 2 + zc (1+ δ ) =
O (1) and: (cid:107) ˜
Q ¯ x (cid:107) =
c (1 + δ ) (cid:107) ¯ x (cid:107) (cid:107) ¯ x (cid:107) 2 + zc (1 + δ ) ≤   (cid:107) ¯ x (cid:107) z
= O (1) if (cid:107) ¯ x (cid:107) ≤ 1 , c (1 + δ ) (cid:107) ¯ x (cid:107) =
O (1) if (cid:107) ¯ x (cid:107) ≥ 1 .
Proposition A.3.
¯ x (cid:124) E [ Q ]¯ x = [Equation] x
[Equation]
˜ Q ¯ x + O (cid:16)(cid:113) log p p (cid:17)
Proof.
Let us bound:
[Equation]
Now let us consider a supplementary random vector x n +1 following the same low as the x i ’s and independent of X .
We divide the set I =
[ n +1] into two sets I
and I 22 of same cardinality ( (cid:98) n +12 (cid:99) ≤ # I 12 , # I 2 2 ≤ (cid:100) n +1 2 (cid:101) ) , we note X 12 =[-( x i | i ∈ I 12 ) , X 
-]
( x i | i ∈ I 1 2 ) , X 2 2 = ( x i | i ∈ I [-12 (cid:17) , D = diag (cid:16) 
-]2 2 ) and we introduce the
[Equation]
We have the bound:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
thanks to Lemma A.1 and Lemma A.2 (the spectral norm of ∆ and D is just an inﬁnity norm if we see them as random vectors of R n ). We can bound 1 p (cid:12)(cid:12)(cid:12)
[Equation]
the same way to obtain the result of the proposition.
Proposition A.4.
[Equation]
Proof.
Considering u ∈ R n such that (cid:107) u (cid:107) = 1 :
[Equation]
[Table]
[Equation]
where the ﬁrst term is treated the same way as we did in the proof of Proposition A.3 and the second term is bounded thanks to Proposition A.
A. M AIN BODY OF THE PROOF
Proof of Theorem 3.3.
Recall the deﬁnition of the resolvents R and Q in Equation (13).
The ﬁrst step of the proof is to show the concentration of R .
This comes from the fact that the application Φ : X (cid:55)→ ( X (cid:124) X
+ z I n ) − 1 is 2 z − 3 / 2 -Lipschitz w.r.t.
the Frobenius norm.
Indeed, by the matrix identity A − B =
A ( B − 1 − A − 1 ) B , we have
[Equation]
And by the bounds (cid:107) AB (cid:107) F ≤ (cid:107) A (cid:107) · (cid:107) B (cid:107) F , (cid:107) Φ( X ) X (cid:124) (cid:107) ≤ z − 1 / 2 and (cid:107) Φ( X ) (cid:107) ≤ z − 1 , we have (cid:107) Φ( X + H ) − Φ( X ) (cid:107) F ≤ z 3 / 2 (cid:107) H (cid:107) F .
Therefore, given X ∈ E q (1 | R p × n , (cid:107) · (cid:107) F ) and since the application X (cid:55)→ R =
Φ( X / √ p ) is 2 z − 3 / 2 p − 1 / 2 -Lipschitz, we have by Proposition 2.3 that R ∈ E q ( p − 1 / 2 | R n × n , (cid:107) · (cid:107) F ) .
The second step consists in estimating E R ( z ) through a deterministic matrix ˜ R .
Indeed, by the identity ( M (cid:124) M
+ z I ) − 1 M (cid:124) =
M (cid:124) ( MM (cid:124)
+ z I ) − 1 , the resolvent R can be expressed in function of Q as follows
[Equation]
thus a deterministic equivalent for R can therefore be obtained through a deterministic equivalent of the matrix X (cid:124) QX .
However, as demonstrated in Louart & Couillet (2019), the matrix Q has as a deterministic equivalent the matrix ˜ Q deﬁned in equation 14.
In the following, we aim at deriving a deterministic equivalent for 1 p X (cid:124) QX in function of ˜ Q .
Let u and v be two unitary vectors in R n , and let us estimate
[Equation]
With the following matrix identities (to explore the independence of the columns of X ):
[Equation]
and the decomposition QXX (cid:124) = [Equation] i x [Equation]
i , we obtain:
[Equation]
We can show with Holder’s inequality and the concentration bounds (mainly the fact that 1 p x (cid:124)
i Q − i x i concentrates around δ ) developed in (Louart & Couillet, 2019), that most of the above quantities vanish asymptotically.
As a toy example
[Equation]
we consider the following term:
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] O (cid:18) [Table] p
[Equation]
Similarly, we can show that:
[Equation]
Finally, the remaining terms in ∆ can be estimated as follows:
[Equation]
Where the last equality is obtained through the following estimation:
[Equation]
With the following bound:
[Equation]
we have again with Holder’s inequality and Proposition A.4:
[Equation]
Now that we estimated ∆
[Equation]
it remains to estimate E [ 1 p X (cid:124) ˜ QX ] .
Indeed, given two unit norm vectors u, v ∈ R n we have:
[Equation]
since we have ¯ x (cid:124)
˜ Q ¯ x = O (1) by Lemma A.2; we introduced the matrix M 1 = 11 (cid:124) .
Therefore we have the following estimation:
[Equation]
where A = B + O (cid:107)·(cid:107) ( α ( p )) means that (cid:107) A − B (cid:107) =
O ( α ( p )) .
Finally, since R concentrates around its mean, we can then conclude:
[Equation]
B P ROOF OF P ROPOSITION 3.
Proof.
Since the Lipschitz constant of a composition of Lipschitz functions is bounded by the product of their Lipschitz constants, we consider the case N = 1 and a linear activation function.
In this case, the Lipschitz constant corresponds to the largest singular value of the weight matrix.
We consider the following notations for the proof
[Equation]
where ¯ σ 1 ,t = σ 1 ( ¯ W t ) , ¯ u 1 ,t = u 1 ( ¯ W t ) and ¯ v 1 ,t = v 1 ( ¯ W t ) .
The effect of spectral normalization is observed in the case where σ ∗ > ¯ σ 1 ,t , otherwise the Lipschitz constant is bounded by σ ∗ .
We therefore have
[Equation]
[Equation]
• If (cid:107)
W t +
[Equation]
F ≥ (cid:107) W t (cid:107) F , we have by equation 16 and equation
[Equation]
And
since (cid:107) W t +1 (cid:107) ≤ (cid:107) ¯ W t (cid:107) we have (cid:107) W t +
(cid:107) ≤ δ . Otherwise, if there exits τ such that (cid:107) W τ +1 (cid:107) F < (cid:107) W τ (cid:107) F , then for all ε > 0 there exists an iteration τ (cid:48) ≥ τ
such that (cid:107) W τ (cid:48) (cid:107) ≤ δ + ε .
Indeed, otherwise we denote ε t
= (cid:107) W t (cid:107) 2 − δand ε t > 0 for all t ≥
τ .
And if for all t ≥ τ , (cid:107) W t +1 (cid:107) F ≤ (cid:107) W t (cid:107) F , we have by equationand equation
[Equation]
Integrating the above expression from τ to T − 1 ≥ τ [Equation] we end up with
[Equation]
T − 1 (cid:88) t = τ ε t , therefore, when T → ∞ , ε t has to tend to 0 otherwise the right hand-side of the last inequality will tend to −∞ which is absurd.