MaGNET : U NIFORM S AMPLING FROM D EEP G ENERATIVE N ETWORK M ANIFOLDS W ITHOUT R ETRAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep Generative Networks (DGNs) are extensively employed in Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and their variants to approximate the data manifold and distribution.
However, training samples are often distributed in a non-uniform fashion on the manifold, due to costs or convenience of collection. For example, the CelebA dataset contains a large fraction of smiling faces.
These inconsistencies will be reproduced when sampling from the trained DGN, which is not always preferred, e.g., for fairness or data augmentation.
In response, we develop MaGNET , a novel and theoretically motivated latent space sampler for any pre-trained DGN, that produces samples uniformly distributed on the learned manifold.
We perform a range of experiments on various datasets and DGNs, e.g., for the state-of-the-art StyleGAN2 trained on FFHQ dataset, uniform sampling via MaGNET increases distribution precision and recall by 4.1% & 3.0% and decreases gender bias by 41.2%, without requiring labels or retraining. As uniform distribution does not imply uniform semantic distribution, we also explore separately how semantic attributes of generated samples vary under MaGNET sampling.
I NTRODUCTION
Deep Generative Networks (DGNs) are Deep Networks (DNs) trained to learn latent representations of datasets; such frameworks include Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), Variational Autoencoders (VAEs) (Kingma & Welling, 2013), ﬂow-based models such as
NICE (Dinh et al., 2014), and their variants (Dziugaite et al., 2015; Zhao et al., 2016; Durugkar et al., 2016;
Arjovsky et al., 2017; Mao et al., 2017; Yang et al., 2019; Fabius & van Amersfoort, 2014; van den Oord et al., 2017; Higgins et al., 2017; Tomczak & Welling, 2017; Davidson et al., 2018;
Dinh et al., 2016; Grathwohl et al., 2018; Kingma & Dhariwal, 2018).
A common assumption that we will carry through our study is that the datasets of interest are not uniformly distributed in their ambient space, but rather are concentrated on, or around, manifolds of lower intrinsic dimension, e.g., the manifold of natural images (Peyr´e, 2009).
Different DGN training methods have been developed and reﬁned to obtain models that approximate as closely as possible the training set distribution.
This becomes an Achilles heel when the training set, regardless of its size, is not representative of the true data distribution, i.e., when the training samples have been curated based on cost or availability that result in implicit/explicit biases.
In such scenarios, while the training samples will lie on the true data manifold, the density distribution of the training set will be different from the natural distribution of the data.
Deploying a DGN trained with a biased data distribution can be catastrophic , in particular, when employed for tasks such as data augmentation (Sandfort et al., 2019), controlled data generation for exploration/interpretation (Thirumuruganathan et al., 2020), or estimation of statistical quantities of the data geometry, such as the Lipschitz constant of the data manifold (Gulrajani et al., 2017; Scaman & Virmaux, 2018).
Biased data generation from DGNs due to skewed training distributions also raises serious concerns in terms of fair machine learning (Hwang et al., 2020; Tan et al., 2020).
While ensuring semantic uniformity in samples is an extremely challenging task, we take one step in the more reachable goal of controlling the DGN sampling distribution to be uniform in terms of the sample distribution on the data manifold . To that end, we propose MaGNET (for Ma ximum entropy G enerative NET work), a simple and efﬁcient modiﬁcation to any DGN that adapts its latent space distribution to provably produce samples uniformly spread on the DGN learned manifold .
Importantly, MaGNET can be employed on any pre-trained and differentiable DGN regardless of its training setting , reducing the requirement of ﬁne-tuning or retraining of the DGN.
This is crucial as many models, such as BigGAN (Brock et al., 2019) and StyleGAN (Karras et al., 2020), have signiﬁcant computational and energy requirements for training.
A plug-and-play method is thus greatly preferred to ease deployment in any already built/trained deep learning pipeline.
Previously, there has been rigorous work on DGNs aimed at improving the training stability of models, deriving theoretical approximation results, understanding the role of the DGN architectures, and numerical approximations to speed-up training and deployment of trained models (Mao et al., 2017; Chen et al., 2018; Arjovsky & Bottou; Miyato et al., 2018; Xu & Durrett, 2018; Liu et al., 2017; Zhang et al., 2017; Biau et al., 2018; Li et al., 2017; Kodali et al., 2017; Roy et al., 2018;
Andr ´ es-Terr ´ e & Li ´ o, 2019; Chen et al., 2018; Balestriero et al., 2020a; Tomczak & Welling, 2016;
Berg et al., 2018).
Existing methods (Metz et al., 2016; Tanaka, 2019;
Che et al., 2020) also try to tackle mode dropping by improving approximation of the data distribution, but this can potentially increase the bias learned implicitly by the DGNs.
We are the ﬁrst to consider the task of providing uniform sampling on the DGN underlying manifold, which has far-reaching consequences, ranging from producing DGNs stable to data curation and capable of handling inconsistencies such as repeated samples in the training set.
We provide a ﬁrst-of-its-kind provable uniform sampling on the data manifold that can be used to speed up estimation of various geometric quantities, such as estimation of the Lipschitz constant.
MaGNET applies to any (pretrained) DGN architecture (GAN, VAE, NF, etc.)
using continuous piecewise afﬁne (CPA) nonlinearities, such as the (leaky) ReLU; smooth nonlinearities can be dealt with via a ﬁrst-order Taylor approximation argument.
Our main contributions are as follows:
[C1]
We characterize the transformation incurred by a density distribution when composed with a CPA mapping (Sec. 3.1) and derive the analytical sampling strategy that enables one to obtain a uniform distribution on a manifold that is continuous and piecewise afﬁne (Sec 3.2).
[C2] We observe that current DGNs produce continuous piecewise afﬁne manifolds, and we demonstrate how to leverage [C1] to produce uniform sampling on the manifold of any DGN (Sec. 3.2).
[C3]
We conduct several carefully controlled experiments that validate the importance of uniform sampling
and showcase the performance of MaGNET on pretrained models such as BigGAN (Brock et al., 2019), StyleGAN2 (Karras et al., 2020), progGAN (Karras et al., 2017), and NVAE (Vahdat & Kautz, 2020), e.g., we show that MaGNET can be used to increase distribution precision and recall of StyleGAN2 by 4.1% & 3.0% and decrease gender bias by 41.2%, without requiring labels or retraining (Sec. 4.2 and Sec. 4.3).
Reproducible code for the various experiments and ﬁgures will be provided upon completion of the review process.
Computation and software details are provided in Appendix H, with the proofs of our results in Appendix I. Discussion on the settings in which MaGNET is desirable and on possible limitations is provided in (Sec. 5).
B ACKGROUND
Continuous Piecewise Afﬁne (CPA) Mappings.
A rich class of functions emerge from piecewise polynomials: spline operators.
In short, given a partition Ω of a domain R S , a spline of order k is a mapping deﬁned by a polynomial of order k on each region ω ∈ Ω with continuity constraints on the entire domain for the derivatives of order 0 ,..., k − 1 .
As we will focus entirely on afﬁne splines ( k = 1 ) we only deﬁne this case for concreteness.
An afﬁne spline S produces its output via
[Equation]
with input z and A ω , b ω the per-region slope and offset parameters respectively, with the key constraint that the entire mapping is continuous on the domain S ∈ C {+
+} ( R S ) .
Spline operators and especially afﬁne spline operators have been extensively used in function approximation theory (Cheney & Light, 2009), optimal control (Egerstedt & Martin, 2009), statistics (Fantuzzi et al., 2002) and related ﬁelds.
Deep Generative Networks. A deep generative network (DGN) is a (nonlinear) operator G Θ with parameters Θ mapping a latent input z ∈ R S to an observation x ∈ R D by composing L intermediate layer mappings.
The only assumption we require for our study is that the nonlinearities present in the DGN are continuous piecewise afﬁne as is the case with (leaky-)ReLU, absolute value, max-pooling. For smooth nonlinearities, our results hold from a ﬁrst-order Taylor approximation argument. Precise deﬁnitions of DGN operators can be found in Goodfellow et al. We will omit Θ from the G Θ operator for conciseness unless needed.
It is also common to refer to z as the latent representation , and x as the generated/observed data , e.g., a time-series or image. One property of DGNs that employ nonlinearities such as (leaky-)ReLU, max-pooling, and the likes, is that the entire input-output mapping becomes a CPA spline with z as both the argument and region parameter.  C ONTINUOUS P IECEWISE A FFINE M APPING OF A P ROBABILITY D ENSITY In this section, we study the properties of a probability density that is transformed by a CPA mapping.
Our goal is to derive the produced density and characterize its properties, such as how the per-region afﬁne mappings in Eq. 1 impact the density concentration.
We present some key results that serve as the backbone of our core result in the next section: how to sample uniformly from the manifold generated by DGNs.
3. D ENSITY ON THE G ENERATED M ANIFOLD
Consider an afﬁne spline operator S (Eq. 1) going from a space of dimension S to a space of dimension D with D ≥ S .
The image of this mapping is a CPA manifold of dimension at most S , the exact dimension is determined by the rank of the per-region slope matrices.
Formally, the image of S is given by
[Equation]
with Aff ( ω ; A ω , b ω ) =
{ A ω z + b ω : z ∈ ω } the afﬁne transformation of region ω by the per-region parameters A ω b ω .
From Eq.
2 we observe that the generated manifold i.e., surface is made of regions that are the afﬁne transformations of the latent space partition regions ω ∈ Ω based on the coordinate change induced by A ω and the shift induced by b ω .
We visualize this in Fig.
2 for a toy spline operator with 2dimensional latent space and 3 -dimensional ambient/output space.
In the remainder of our study we will denote for conciseness S ( ω ) (cid:44) Aff ( ω ; A ω , b ω ) .
When the input space is equipped with a density distribution, then this density is transformed by the mapping S and “lives” on the surface of the CPA manifold generated by S .
Given a distribution p z over the latent space, we can explicitly compute the output distribution after the application of S , which leads to an intuitive result exploiting the CPA property of the generator.
For this result, we require that operator S be bijective between its domain and its image.
That is, each slope matrix A ω , ∀ ω ∈ Ω should be full rank, and there should not be any folding of the generated CPA surface that intersects with itself, i.e., S ( ω ) ∩ S ( ω (cid:48) )
(cid:54) = {} ⇐⇒ ω = ω (cid:48) .
We now derive the key result of this section that characterizes the density distribution lying on the manifold.
Lemma 1.
The volume of a region ω ∈ Ω denoted by µ ( ω ) is related to the volume afﬁnely transformed region S ( ω ) by,
[Equation]
where µ ( S ( ω )) is using the measure on the S -dimensional afﬁne subspace spanned by the CPA mapping.
(Proof in Appendix I.1.)
Theorem 1.
The probability density p S ( x ) generated by S for latent space distribution p z is given by,
[Equation]
(Proof in Appendix I.2.)
In words, the distribution obtained in the output space naturally corresponds to a piecewise afﬁne transformation of the original latent space distribution, weighted by the change in volume of the perregion mappings from (Eq. 3).
For Gaussian and Uniform p z , we use the above results to obtain the analytical form of the density covering the output manifold, we have provided proof and differential entropy derivations in Appendix B.
3. M AKING THE D ENSITY ON THE M ANIFOLD U NIFORM
The goal of this section is to build on Thm.
1 to provide a novel latent space distribution such that the density distribution lying on the generated manifold is uniform.
One important point that we highlight is that having a Uniform density distribution in the latent space of the afﬁne spline is not enough to have a uniform density lying on the manifold; it would be if
[Equation]
[-det( A T ! 
 A ! 0 )-] , in words, the change in volume of the per region mapping is equal for all ω .
This is evident from Appendix B (Eq. 8).
So we propose here a novel latent space sampler with the purpose that once transformed by the afﬁne spline i.e., the DGN, that distribution becomes uniform on the DGN manifold.
We focus here on the technical aspect and provide precise motivations behind such construction in the next section that deals with practical applications.
To obtain K samples uniformly distributed on the output manifold of S :
For K MaGNET samples, sample N (cid:29) K (as large as possible) iid latent vectors with U being the latent space domain of S ( z 1 , . . .
, z N ) , with z i ∼ U ( U ) 2.
Compute the per-region slope matrices
A i (cid:44) J S ( z i ) (Eq. 1) and the change of volume scalar ( σ 1 , . . .
, σ N ) (cid:44) (cid:18)(cid:112) det( A T 1 A 1 ) , . . . , (cid:113) det( A TN A N ) (cid:19) , where A i =
A ω 1 { z i ∈ ω }
3. Sample (with replacement) K latent vectors ( z 1 , . . . , z K ) with probability ∝ ( σ 1 , . . . , σ N )
We discuss the possible choices of N and K in Appendix D, where we observe that, even for stateof-the-art models like StyleGAN2, N = 250,000 is sufﬁcient to provide a stable approximation of the true latent space target distribution. In practice, A i is simply obtained through backpropagation as it is the Jacobian matrix of the DGN at z i , as in A i = JS ( z i ) .
The above provides a Monte-Carlo approximation that does not require knowledge of the partition Ω nor the per-region slope matrices (Eq. 1).
Those are computed on-demand as z i are sampled. The above procedure produces uniform samples on the manifold learned by a DGN regardless of how it has been trained.
M A GNET: M AXIMUM E NTROPY G ENERATIVE N ETWORK S AMPLING
The goal of this section is to ﬁrst bridge current DGNs with afﬁne splines, then leverage Thm. 1 and
Sec. 3.2 in order to effectively produce uniform samples on the manifold of current Deep Generative Networks such as BigGAN, StyleGAN.
We build this afﬁne splines-DGN bridge and propose carefully motivation for uniform sampling in Sec.
4.1 and ﬁnally present various experiments across architectures in Sec.
4.2, 4.3 and 4.4.
4. U NIFORM S AMPLING ON THE D EEP G ENERATIVE N ETWORK M ANIFOLD
We proposed in the previous section a thorough study of afﬁne splines and how those mappings transform a given input distribution.
This now takes high relevance as per the following remark.
Remark 1.
Any DGN (or part of it) that employs CPA nonlinearities (as in Sec. 2) is itself a CPA; that is, the input-output mapping can be expressed as in (Eq. 1).
This observation in the context of classiﬁer DNs goes back to Montufar et al.
and has been further studied in Unser (2018); Balestriero & Baraniuk (2018b).
We also shall emphasize that operators such as Batch-Normalization (Ioffe & Szegedy, 2015) are not continuous piecewise afﬁne during training but become afﬁne operators during evaluation time.
For completeness, we also provide that analytical form of the per-region afﬁne mappings A ω , b ω of Eq.
1 in the case of DGNs in Sec.
C in the appendix.
The key for our method is thus to combine the above with the results from
Sec. 3.2 to obtain the following statement.
Theorem 2.
Consider a training set sampled from a manifold M and a (trained) CPA DGN S .
As long as M ⊂ Im ( S ) , sampling from S as per Sec. 3.2 produces uniform samples on M , regardless of the training set sampling.
(Proof in I.4.)
This result follows by leveraging the analytical DGN distribution from Thm. 1 and by replacing p z with the proposed one, leading to p S ( x )
∝ (cid:80) ω ∈ Ω 1 { x ∈ S ( ω ) } i.e. being uniform on the DGN manifold. By using the above one can take any (trained) DGN and produce uniform samples on the learned underlying manifold .
Hence, our solution allows to produce a generative process that becomes invariant to the training set distribution.
While this provides a theoretical guarantee for uniform sampling, it also highlight the main limitation of MaGNET: the uniform samples will be lying on a continuous piecewise afﬁne manifold .
That is, unless the true manifold M is also continuous, MaGNET will occasionally introduce abnormal samples that correspond to sampling from the regions of discontinuity of M .
We will see in the following sections how even on high-quality image datasets, MaGNET produces very few abnormal samples, one reason being that for complicated data manifold, state-of-the-art DGNs are often built with a (class) conditioning.
That is, the above continuity assumption on M lessens only to a within-class continuity assumption which is a much more realistic assumption.
Sampling uniformly on the DGN manifold has many important applications that are deferred to the following sections.
4. Q UANTITATIVE V ALIDATION : (cid:15) -B ALL C ONCENTRATION , GMM L IKELIHOOD AND F R ´ ECHET I NCEPTION D ISTANCE
We now report three controlled experiments to validate the applicability of the theoretical results from Sec. 3.2 for the MaGNET sampling procedure.
First, we consider MNIST and assume that the entire data manifold is approximately covered by the training samples.
Regardless of the training data distribution on the manifold (uniform or not), we can pick a datum at random, count how many generated samples ( η ) are within this datum (cid:15) -ball [Figure]
m [Figure] e r ( l og ) o
f t r a i n i
ng s a m
[Figure]
l e s w
[Figure] n [Figure] a t e
[Figure]
n e i ghbo r s
[Figure]
[Figure] neighborhood and repeat this process for 10,000 training samples.
If this count does not vary between training datum, then it strongly indicates that the generated samples are uniformly distributed on the manifold covered by the training data.
We perform this experiment using a pretrained stateof-the-art variational autoencoder NVAE (Vahdat & Kautz, 2020) to compare between standard and MaGNET sampling with the number of generated samples N ranging from 1,000 to 10,000.
We report the distribution of counts in Fig.
Again, uniform sampling is equivalent to having the same count for all training samples, i.e., a Dirac distribution in the reported histograms.
We can see that MaGNET sampling approaches that distribution while standard sampling has a heavy-tail count distribution, i.e., the generated digits have different concentration at different parts of the data manifold.
Another quantitative measure consists of ﬁtting a Gaussian Mixture Model (GMM) with varying number of clusters, on the generated data, and comparing the likelihood obtained for standard and MaGNET sampling.
As we know that in both cases the samples lie on the same manifold and domain, the sampling with lower likelihood will correspond to the one for which samples are spread more uniformly on the manifold.
We report this in Fig.
4, further conﬁrming the ability of MaGNET to produce uniformly spread samples.
We report the generated samples in Appendix E. Lastly, we compare the Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) between 50,000 generated samples and 70,000 training samples for StyleGAN2 trained on FFHQ. Since uniform sampling via MaGNET increases the diversity of generated samples, we see that MaGNET sampling improves the FID for truncation (Karras et al., 2019; Brock et al., 2019), ψ = { . 4 , .
7 } by 2 .
76 points on average (see Appendix F).
While for the aforementioned ψ MaGNET samples alone provide an improved FID, for higher ψ values, we introduce an increasing amount of MaGNET samples for FID calculation. We observe in Fig. 4 that by progressively increasing the percentage of MaGNET samples, we are able to exceed the state-of-the-art FID of 2 . 74 for StyleGAN2 ( ψ = 1 ), reaching an
FID of 2 . 66 with ∼ 4 % of MaGNET samples.
4. Q UALITATIVE V ALIDATION : H IGH -D IMENSIONAL S TATE - OF - THE - ART I MAGE G ENERATION
We now turn into the qualitative evaluation of MaGNET sampling, to do so we propose extensive experiments on various state-of-the-art image DGNs.
We also remind the reader that in all cases, standard and MaGNET sampling are performed on the same DGN (same weights) as discussed in
Sec. 3.2.
2 -Dimensional Dataset and Colored-MNIST.
The ﬁrst set of controlled experiments are designed such that the training set contains inconsistencies while it is known that the original distribution is uniform on the data manifold.
Such inconsistencies can occur in real datasets due to challenges related to dataset compilation.
We provide illustrative examples in Fig.
5, where we demonstrate that unless uniform sampling is employed, the trained DGN reproduces the inconsistencies present in the training set, as expected.
This toy dataset visualization validates our method from Sec. 3.2.
Going further, we take the MNIST dataset (in this case, only digit 8 samples) and apply imbalanced [Figure] hood [Figure]
[Figure] [Figure] Original dataset purposefully favored cyan coloring to represent training set inconsistency is able to approximately generate uniformly colored MNIST samples drop around represent regions where the DGN manifold approxi- mation was incorrect, hence uniform sampling can not recover those samples coloring based on the hue distribution provided in Appendix Fig. 12, that favors cyan color.
We train a β -VAE DGN (BVAE) on that cyan-inclined dataset, and present in Fig.
6 the hue distributions for samples obtained via standard sampling and MaGNET sampling.
We observe that MaGNET corrects the hue distribution back to uniformity.
Uniform Face Generation: CelebA-HQ and Flickr-Faces-HQ with progGAN and StyleGAN2.
Our ﬁrst experiment concerns sampling from the StyleGAN2 (Karras et al., 2020) model pretrained on the Flickr-Faces-HQ (FFHQ) dataset.
StyleGAN2 has two DGNs, one that maps to an intermediate latent space, termed style-space and another DGN that maps style-space vectors to the pixels-space (output of StyleGAN2).
Implementation details are contained in Appendix H.
We focus here on applying MaGNET onto the entire StyleGAN2 model (the composition of both DGNs), in Sec. 4.4 we discuss applying MaGNET to the style-space DGN.
In Fig.
1 we provide random samples from the same StyleGAN2 model obtained via standard and MaGNET sampling.
Upon qualitative evaluation it can be seen that the samples obtained via MaGNET (MaGNET StyleGAN2) have a signiﬁcantly larger variety of age distribution, background variations and wearable accessories compared to standard sampling.
For experiments with CelebA-HQ dataset, we adopt the DGN of Progressively Growing GAN (progGAN)
(Karras et al., 2017), trained on 1024 × 1024 resolution images.
In Fig.
9 & we provide random samples from standard and MaGNET sampling, the latter portraying more qualitative diversity.
We see that uniform manifold sampling via MaGNET recovers samples containing number of attributes that are generally underrepresented in the samples generated by vanilla progGAN.
(See
Appendix E) for larger batches and attribute distributions).
Notice that uniform sampling not only recovers under-represented groups e g., age < 30 , head-wear, and bald hair, it also increases the presence of neutral emotion and black hair.
One interesting observation is that MaGNET also increases the number of samples off the true data manifold (images that are not celebrity faces), exposing regions where the manifold is not well approximated by progGAN.
Conditionally Uniform Generation: ImageNet with BigGAN.
We present experiments on the state-of-the-art conditional generative model BigGAN (Brock et al., 2019) using MaGNET sampling.
In Fig.
7 we provide random samples from standard and MaGNET sampling.
More experiments on different classes are presented in
Appendix E.
We see that uniform sampling on the learned data manifold yields a large span of backgrounds and textures, including humans, while standard sampling produces examples closer to the modes of the training dataset.
This is quite understandable considering that ImageNET was curated using a large number of images scraped from the internet.
MaGNET therefore could possibly be used for data exploration/model interpretation and also as a diagnostic tool to assess the quality of the learned manifold a posteriori of training.
4. A PPLICATION : M ONTE -C ARLO E STIMATION AND A TTRIBUTE R EBALANCING
We conclude this section with two more practical aspects offered by MaGNET.
Reduced-Variance Monte-Carlo Estimator.
The ﬁrst is to speed-up (in term of number of required samples) basic Monte-Carlo estimation of arbitrary topological quantities of the generated manifold.
In fact, suppose one’s goal is to estimate the Lipschitz constant of a DGN.
A direct estimation method would use the known bound given by max z (cid:107) J S ( z ) (cid:107) F (Wood & Zhang, 1996).
This estimation can be done by repeatedly sampling latent vectors z from the same distribution that one used for training a DGN.
However this implies that the produced samples will not be uniformly distributed on the manifold in turn leading to slower convergence of the estimator.
Instead, we propose to use MaGNET, and report our ﬁndings in Fig.
More domains of application, where MaGNET can be used for estimator variance reduction, can be found in Baggenstoss (2017).
Style-space MaGNET sampling rebalances attributes.
When thinking of uniform sampling on a manifold, it might seem natural to expect fairness i.e., fair representation of different attributes such as equal representation of gender, ethnicity, hair color, etc.
However, this is not necessarily true in all cases.
In fact, it is trivial to show that each attribute category will be equally represented iff their support on the true data manifold is of equal volume (integrated with respect to the data manifold).
Fortunately, as we mentioned in Sec. 4.3, architectures such as StyleGAN2 have explicitly built a style-space, which is a latent space in which attributes are better organized along afﬁne subspaces and occupy nearly the same volume (Karras et al., 2019) i.e., MaGNET applied on the style-space DGN should improve fairness.
By applying MaGNET sampling on the style-space, we are able to reduce gender bias from 67–33% (female-male) in standard StyleGAN2 to 60–40%.
This simple result demonstrates the importance of our proposed sampling and how it can be used to increase fairness for DGNs trained on biased training sets.
MaGNET in the style-space also yields improvements in terms of recall and precision (Sajjadi et al., 2018).
Given a reference distribution (e.g., FFHQ dataset) and a learned distribution, precision measures the ﬁdelity of generated samples while recall measures diversity.
We compare the metrics for face images generated via latents z ∼ N (0 , aI ) where a ∈ 0 .
5 , 1 , 1 . 5 , 2 , z ∼ U [ − 2 , 2] , and MaGNET sampling on style-space.
For 70K samples generated for each case, MaGNET sampling obtains a recall and precision of (0 . 822 , 0 . 92) with a 4.12% relative increase in recall and 3.01% relative increase in precision compared to other latent m z sampling methods (metrics were averaged for 10 seeds).
C ONCLUSIONS , L IMITATIONS AND F UTURE W ORK
We have demonstrated how the afﬁne spline formulation of DGN offers a rich portal to obtain theoretical results to provably provide uniform sampling on the manifold learned by a DGN.
This allows to become robust to possibly incorrect training set distribution that any DGN would learn to replicate after during its training.
We provided various experiments using pretrained state-of-the-art generative models and demonstrated that uniform sampling on the manifold offers many beneﬁts from data exploration to statistical estimation.
Beyond the sole goal of uniform sampling on a manifold, MaGNET opens many avenues, yet MaGNET is not a “one size ﬁts all” solution.
When not to sample uniformly.
We can identify to general cases in which one should not employ uniform sampling of the DGN manifold.
The ﬁrst case occurs whenever the true manifold is known to be discontinuous and one imperatively needs to avoid sampling in those regions of discontinuities.
In fact, in the discontinuous case, DGN training, will adapt its weight to put zero (or near zero) density in those discontinuous regions preventing standard sampling to reach those regions (Balestriero et al., 2020b).
However, MaGNET will reverse this process and introduce samples back in those regions.
The second case occurs if one aims to produce samples from the exact same distribution as the training set distribution (assuming training of the DGN was successful).
In fact, in this scenario one should use the exact same latent distribution at evaluation time than the one used during training.
Future work. Currently, there are two main limitations of our MaGNET sampling strategy.
The ﬁrst one lies in the assumption that the trained DGN is able to learn a good enough approximation of the true underlying data manifold.
In future work, we plan to explore how MaGNET can be used to test such an assumption.
One potential directions is as follows.
Train a DGN using several different sub-sampled datasets (similar to bootstrap methods) and then study if MaGNET samples populate manifolds that all coincide between the different DGNs.
If training was successful, then those sampled manifolds should coincide.
The second one lies in understanding the relationship between uniform sampling and uniform attribute representation.
We demonstrated how uniform sampling in the style-space of StyleGAN2 ensures that relationship by construction.
However, doing MaGNET sampling arbitrarily on any DGN might not always produce better attribute distributions.
Hence, developing novel theoretical results to characterize those two effects is one crucial future direction to equip MaGNET with theoretical fairness guarantees.
R EPRODUCIBILITY STATEMENT
Reproducible code for the various experiments and ﬁgures will be provided upon completion of the review process.
Computation and software details are provided in Appendix H, with the proofs of our results in Appendix I. Discussion on the settings in which MaGNET is desirable and on possible limitations is provided in (Sec. 5).
S UPPLEMENTARY M ATERIAL
The following appendices support the main paper and are organized as follows: Appendix A addresses some details on the background of continuous piecewise afﬁne deep networks, that were omitted in the main paper.
Appendices E and H provide additional ﬁgures and training details for all the experiments that were studied in the main text, and Appendix I provides the proofs of all the theoretical results.
Due to size constraints, the high-quality batch of samples are provided in the supplementary ﬁles .
A B ACKGROUND ON C ONTINUOUS P IECEWISE A FFINE D EEP N ETWORKS
A max-afﬁne spline operator (MASO) concatenates independent max-afﬁne spline (MAS) functions, with each MAS formed from the point-wise maximum of R afﬁne mappings (Magnani & Boyd, 2009; Hannah & Dunson, 2013).
For our purpose each MASO will express a DN layer and is thus an operator producing a D (cid:96) dimensional vector from a D (cid:96) − 1 dimensional vector and is formally given by
[Equation]
) where A r ∈ R D (cid:96) × D (cid:96) − 1 are the slopes and b r ∈ R D (cid:96) are the offset/bias parameters and the maximum is taken coordinate-wise.
For example, a layer comprising a fully connected operator with weights W (cid:96) and biases b (cid:96) followed by a ReLU activation operator corresponds to a (single) MASO with R = 2 , A 1 = W (cid:96) ,
A 2 = 0 , b 1 = b (cid:96) , b 2 = 0 .
Note that a MASO is a continuous piecewiseafﬁne (CPA) operator (Wang & Sun, 2005).
The key background result for this paper is that the layers of DNs constructed from piecewise afﬁne operators (e.g., convolution, ReLU, and max-pooling) are MASOs (Balestriero & Baraniuk, 2018b [Equation]
[Equation]
making the entire DGN a composition of MASOs.
The CPA spline interpretation enabled from a MASO formulation of DGNs provides a powerful global geometric interpretation of the network mapping based on a partition of its input space R S into polyhedral regions and a per-region afﬁne transformation producing the network output.
The partition regions are built up over the layers via a subdivision process and are closely related to
Voronoi and power diagrams (Balestriero et al., 2019).
We now propose to greatly extend such insights to carefully characterize and understand DGNs as well as provide theoretical justiﬁcations to various observed behaviors e.g. mode collapse.
B U NIFORM AND G AUSSIAN M ANIFOLD D ISTRIBUTIONS
We now demonstrate the use of the above result by considering practical examples for which we are able to gain insights into the DGN data modeling and generation.
We consider the two most common cases: (i) the latent distribution is set as z ∼ N (0 , 1) and (ii) the latent distribution is set as z ∼ U (0 , 1) (on the hypercube of dimension S ).
We obtain the following result by direct application of Thm. 1.
Corollary 1.
The generated density distribution p S of the Gaussian and uniform densities are given by
[Equation]
The two above formulae provide a precise description of the produced density given that the latent space density is Gaussian or Uniform.
In the Gaussian case, the per-region slope matrices act upon the (cid:96) 2 distance by rescaling it from the coordinates of A ω and the per-region offset parameters b ω are the mean against which the input x is compared against.
In the Uniform case, the change of volume (recall Eq. 3) is the only quantity that impacts the produced density.
We will heavily rely on this observation for the next section where we study how to produce a uniform sampling onto the
CPA manifold of an afﬁne spline.
We derive the analytical form for the case of Gaussian and Uniform latent distribution in Appendix I.3.
From the analytical derivation of the generator density distribution, we obtain its differential entropy.
Corollary 2.
The differential Shannon entropy of the output distribution p G of the DGN is given by
[Equation]
As a result, the differential entropy of the output distribution p G corresponds to the differential entropy of the latent distribution p z plus a convex combination of the per-region volume changes.
It is thus possible to optimize the latent distribution p z to better ﬁt the target distribution entropy as in Ben-Yosef & Weinshall (2018) and whenever the prior distribution is ﬁxed, any gap between the latent and output distribution entropy imply the need for high change in volumes between ω and
G ( ω ) .
C P ER -R EGION A FFINE M APPINGS
For completeness we also provide that analytical form of the per-region afﬁne mappings
[Equation]
where ˙ σ (cid:96) ( z ) is the pointwise derivative of the activation function of layer (cid:96) based on its input
W (cid:96) z (cid:96) − 1 + b (cid:96) , which we note as a function of z directly.
For precise deﬁnitions of those operators see Balestriero & Baraniuk (2020).
The diag operator simply puts the given vector into a diagonal square matrix.
For convolutional layers (or else) one can simply replace the corresponding
W (cid:96) with the correct slope matrix parametrization as discussed in Sec. 2.
Notice that since the employed activation functions σ (cid:96) , ∀ (cid:96) ∈ { 1 , . . .
, L } are piecewise afﬁne, their derivative is piecewise constant, in particular with values [ ˙ σ (cid:96) ( z )] k ∈ { α, 1 } with α = 0 for ReLU, α = − 1 for absolute value, and in general with α > 0 for Leaky-ReLU for k ∈ { 1 , . . .
, D (cid:96) } .
D N UMBER OF S AMPLES AND U NIFORMITY
Exact uniformity is reached when the Monte Carlo samples have covered each region of the DGN partition boundary.
For large state-of-the-art models this condition requires sampling on the order of millions.
However, we conducted an experiment to see how the number of samples really impacted the uniformity of the generated manifold as follows.
We compute precision and recall metrics [4] for StyleGAN2 with K generated samples obtained from N Monte Carlo samples based on our sampling strategy by varying N .
We use K = 5000 and N ranging from 10 , 000 to 500 , 000 .
Based on the metrics, we identify that increasing beyond K = 250 , 000 no longer impacts the metrics, showing that this number of monte carlo samples is enough to converge (approximately) to the uniform sampling in that case, see Fig.
We report here the Jacobian computation times for Tensorﬂow 2.5 with CUDA 11 and Cudnn 8 on an NVIDIA Titan RTX GPU. For StyleGAN2 pixel space, 5.03s/it; StyleGAN2 style-space, 1.12s/it; BigGAN 5.95s/it; ProgGAN 3.02s/it.
For NVAE on Torch 1.6 it takes 20.3s/it. Singular value calculation for StyleGAN2 pixel space takes .005s/it, StyleGAN2 style space .008s /it, BigGAN .001s/ it, ProgGAN .004s/it and NVAE .02s
/it on NumPy.
E E XTRA F IGURES
This section contains samples from our proposed methods, more samples along with attribute data and pretrained weights are available at our project link . [Figure]
[Figure]
N u m b e r o f t r a i n i
ng s a m p l e
s h a v
i ng x ( x - a x i s ) s a m p l e s a t m
[Figure] t [Figure] y
[Figure]
N u m b e r o f t r a i n i
ng s a m p l e
s h a v
i ng x ( x - a x i s ) s a m p l e s a t m o s t (cid:15) a w a y
[Figure]
F E XTRA T ABLES
Truncation [Table] G A LGORITHMS
Algorithm
Input
Output: MaGNET Samples, { x
=
Initialize, Z , S ← [] , [] ; for n = 1 , . .
. , N do z ∼ U ( U ) ;
Get Slope Matrix, A
= J G ( z ) ;
Get volume scalar at z , σ z = (cid:112) det ( A T A ) ;
Z .
append( z ) ; S .
append( σ z ) endfor n = 1 , . .
K do i
Categorical prob softmax
; x i ← Z [ i ] end
Input
Latent
Generator
Output
MaGNET while True do
Sample z ∼ U ( U
Sample α ∼ U [0 , 1] ;
Get Slope Matrix, A
= J G ( z ) ;
{+Get volume scalar at z , σ z = (cid:112) det ( A T A ) ; if σ z σ z + (cid:80) N i =
+}
σ i ≥ α then x = G ( z ) ; break ; end end
H A RCHITECTURE , H ARDWARE AND I MPLEMENTATION D ETAILS
All the experiments were run on a Quadro RTX 8000 GPU, which has 48 GB of high-speed GDDR memory and 576 Tensor cores.
For the software details we refer the reader to the provided codebase.
In short, we employed TF2 (2.4 at the time of writing), all the usual Python scientiﬁc libraries such as NumPy and PyTorch.
We employed the ofﬁcial repositories of the various models we employed with ofﬁcial pre-trained weights.
As a note, most of the architectures can not be run on GPUs with less or equal to 12 GB of memory.
For StyleGAN2, we use the ofﬁcial conﬁg-e provided in the GitHub StyleGAN2 repo 1 , unless speciﬁed.
We use the recommended default of ψ = 0 . 5 as the interpolating stylespace truncation, to ensure generation quality of faces for the qualitative experiments.
For BigGAN we use the BigGANdeep architecture with no truncation, available on TFHub 2 .
We also use the NVAE 3 and ProgGANmodels and weights from their respective ofﬁcial implementations.
For the Jacobian determinant calculation of images w.r.t latents, we ﬁrst use a random orthogonal matrix to project generated images into a lower dimensional space, calculate the Jacobian of the projection w.r.t the latents and calculate the singular values of the jacobian to estimate the volume scalar. We use a projection of 256 dimensions for StyleGAN2-pixel, ProgGAN and BigGAN, and 128 dimensions for NVAE. To estimate the volume scalar we use the top 30, 20, 15 singular values for StyleGAN2 MaGNET pixel, ProgGAN and BigGAN; 40 for StyleGAN2 MaGNET style, and 30 for NVAE.
I P ROOFS I. P ROOF OF L EMMA
Proof.
In the special case of an afﬁne transform of the coordinate given by the matrix A ∈ R D × D the well known result from demonstrates that the change of volume is given by | det( A ) | (see Theorem 7.26 in Rudin (2006)).
However in our case the mapping is a rectangular matrix as we span an afﬁne subspace in the ambient space R D making | det( A ) | not deﬁned.
First, we shall note that in the case of a Riemannian manifold (as is the produced surface from the per-region afﬁne mapping) the volume form used in the usual change of variable formula can be deﬁned via the square root of the determinant of the metric tensor.
Now, for a surface of intrinsic dimension n embedded in Euclidean space of dimension m (in our case, the per-region afﬁne mapping produces an afﬁne subspace) parametrized by the mapping M : R n (cid:55)→ R m (in our case this mapping is simply the afﬁne mapping M ( z ) =
z ω z + b ω for each region)
the metric tensor is given by g = DM T DM with D the jacobian/differential operator (in our case g =
A Tω A ω for each region).
This result is also known as Sard’s theorem (Spivak, 2018).
We thus obtain that the change of volume from the region ω to the afﬁne subspace G ( ω ) is given by (cid:112) det( A T A ) which can also be written as follows with USV T the svd-decomposition of the matrix A :
[Equation]
leading to
[Equation]
I. P ROOF OF T HEOREM
Proof.
We will be doing the change of variables z
= ( A Tω A ω ) [- 
-]− 1 A
Tω ( x − b ω ) =
A + ω ( x − b ω ) , also notice that J G − 1 ( x ) =
A + .
First, we know that P G ( z ) ( x ∈ w ) =
P z ( z ∈ G − 1 ( w )) =
(cid:82)
G − 1 ( w ) p z ( z ) d z which is well deﬁned based on our full rank assumptions.
We then proceed by
[Equation]
Let now prove the Etape 1 step by proving that σ i ( A + ) =
( σ i ( A )) − 1 where we lighten notations as
A :
=
A ω and USV T is the svd-decomposition of A :
[Equation]
with the above it is direct to see that (cid:113) det(( A + ω ) T A + ω ) = [Equation] √ det( A Tω A ω ) as follows
[Equation]
which gives the desired result.
I. P ROOF OF C OROLLARY
Proof.
We now demonstrate the use of Thm.
1 where we consider that the latent distribution is set as z ∼ N (0 , 1) .
We obtain that
[Equation]
giving the desired result that is reminiscent of Kernel Density Estimation (KDE) (Rosenblatt, 1956) and in particular adaptive KDE (Breiman et al., 1977), where a partitioning of the data manifold is performed on each cell ( ω in our case) different kernel parameters are used.
Proof.
We now turn into the uniform latent distribution case on a bounded domain U in the DGN input space.
By employing again Thm. 1, the given formula one can directly obtain that the output density is given by
[Equation]
I. P ROOF OF T HM .
As we assume successful training, then regardless of the actual distribution p x , the DGN will learn the correct underlying manifold, and learn the best approximation to p x as possible onto this manifold.
Now, applying MaGNET sampling i.e. Sec. 3.2 is equivalent to sampling from a distribution
p m z such that after DGN mapping, that distribution is uniform on the learned manifold (see Thm. 1).
As we assumed that regardless of p x the DGN approximates correctly the true manifold, and as we then adapt the sampling distribution p m z to always obtain uniform sampling on that manifold, we see that this ﬁnal sampling becomes invariant upon the data distribution (on the manifold) leading to the desired result.