H YPER GAN: G ENERATING D IVERSE , P ERFORMANT N EURAL N ETWORKS WITH A GAN
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce HyperGAN, a generative network that learns to generate all the weight parameters of deep neural networks.
HyperGAN ﬁrst transforms low dimensional noise into a latent space, which can be sampled from to obtain diverse, performant sets of parameters for a target architecture.
We utilize an architecture that bears resemblance to generative adversarial networks, but we evaluate the likelihood of samples with a classiﬁcation loss.
This is equivalent to minimizing the KL-divergence between the generated network parameter distribution and an unknown true parameter distribution.
We apply HyperGAN to classiﬁcation, showing that HyperGAN can learn to generate parameters which solve the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning, while learning a rich distribution of effective parameters.
We also show that HyperGAN can also provide better uncertainty than standard ensembles.
This is evaluated by the ability of HyperGAN-generated ensembles to detect out of distribution data as well as adversarial examples.
We see that in addition to being highly accurate on inlier data, HyperGAN can provide reasonable uncertainty estimates.
I NTRODUCTION
Since the inception of deep neural networks, it has been found that it is possible to train from different random initializations and obtain networks that, albeit having quite different parameters, achieve quite similar accuracy (Freeman & Bruna, 2016).
It has further been found that ensembles of deep networks that are trained in such a way have signiﬁcant performance advantages over single models (Maclin & Opitz, 2011), similar to the classical bagging approach in statistics.
Ensemble models also have other beneﬁts, such as being robust to outliers and being able to provide variance or uncertainty estimates over their inputs (Lakshminarayanan et al., 2017).
In Bayesian deep learning, there is a signiﬁcant interest in having a probabilistic interpretation of network parameters and modeling a distribution over them.
Earlier approaches mostly utilize dropout as a Bayesian approximation, by randomly setting different parameters to zero and thus integrating over many possible networks.
(Gal & Ghahramani, 2016) showed that networks with dropout following each layer are equivalent to a deep Gaussian process (Damianou & Lawrence, 2013) marginalized over its covariance functions.
They proposed MCdropout as a simple way to estimate model uncertainty.
These approximations are not well aligned with current training patterns of neural networks.
Applying dropout to every layer results in over-regularization and underﬁtting of the target function.
Moreover, dropout does not integrate over the full variation of possible models, only those which may be reached from one (random) initialization.
As another interesting direction, hypernetworks (Ha et al., 2016) are neural networks which output parameters for a target neural network.
The hypernetwork and the target network together form a single model which is trained jointly.
The original hypernetwork produced the target weights as a deterministic function of its own weights, but Bayesian Hypernetworks (BHNs) (Krueger et al., 2017), and Multiplicative Normalizing Flows (MNF) (Louizos & Welling, 2016) generate model parameters by sampling a Gaussian prior.
However, these approaches use normalizing ﬂows to transform a simple prior into a sample of the more complicated posterior, which are composed of only bijective, invertible functions.
This limits their scalability and the variety of learnable functions.
In this paper we explore an approach which focuses on generating all the parameters of a neural network, without assuming any ﬁxed noise models on parameters.
To keep our method scalable, we avoid utilizing invertible functions as in normalizing ﬂow approaches, and instead utilize the ideas from generative adversarial networks (GANs).
We especially observe recent Wasserstein Autoencoder (Tolstikhin et al., 2017) approaches.
These approaches have demonstrated an impressive capability to model complicated, multimodal distributions.
In our approach, a random noise vector is ﬁrst encoded to a number of different random vectors, then each random vector is used to generate all parameters within one layer of a deep network.
The generator is then trained with conventional maximum likelihood (classiﬁcation/regression) on the parameters it generates, and an adversarial regularization keeps it from collapsing onto only one mode.
In this way, it is possible to generate much larger networks than the dimensionality of the latent code, making our approach capable of generating all the parameters of a deep network with a single GPU.
As an example, in our experiments on CIFAR-10 we start from a 256 -dimensional latent vector and generate all 50 , 000+ parameters in one pass, consuming only 4 GB GPU memory.
This shows that deep networks may indeed span a low-dimensional manifold, and could spur further thoughts and research.
1. S UMMARY OF C ONTRIBUTIONS
We propose HyperGAN, a novel approach for generating all the parameters for a target network architecture using a modiﬁed GAN, and we do so starting from a small Gaussian noise vector which scales well with the size of the output.
Our approach is different from strict Bayesian approaches since we do not attempt to model the entire posterior, i.e., from an observed network we cannot compute its probability of it being generated.
Moreover, we use deterministic generators, our only source of stochasticity is the initial prior sample.
After our GAN is trained, one can directly generate many diverse, well-trained deep models without needing to further train or ﬁne-tune them.
The diversity of the models we can generate is beyond just adding dropout or scaling factors, which is shown by the superior performance of ensembles of the generated networks.
We believe HyperGAN is widely applicable to a variety of tasks.
One area where populations of diverse networks show promise is in uncertainty estimation and anomaly detection.
We show through a variety of experiments that populations of networks sampled from HyperGAN are able to approximate the data distribution such that it can detect out of distribution samples.
We show that we can provide a reasonable measure of uncertainty by calculating the entropy within the predictive distribution of sampled networks.
Our method is straightforward, as well as easy to train and sample from.
We hope that we can inspire future work in estimation of the manifold of neural networks.
R ELATED W ORK
Generating parameters for neural networks has been framed in contexts other than the Bayesian approaches described above.
The hypernetwork framework (Ha et al., 2016) has come to describe models where one network directly supervises the weight updates of another network. (Pawlowski et al., 2017) uses hypernetworks to generate layer-wise weights for a target architecture given some auxiliary noise as input.
The auxiliary noise is noted to be independent between layers, while we impose structure in our inputs to create diversity in our generated samples.
Computer vision methods often use data driven approaches as seen in methods such as Spatial Transformer networks (Jaderberg et al., 2015), or Dynamic Filter networks (Jia et al., 2016).
In these methods the ﬁlter parameters of the main network are conditioned on the input data, receiving contextual scale and shift updates from an auxiliary network.
Our method, however, generates parameters for the whole network.
Furthermore our predicted parameters are highly nonlinear functions of the input, instead of simple afﬁne transformations based on the input examples.
Recently, (Lakshminarayanan et al., 2017) proposed Deep Ensembles, where adversarial training was applied to standard ensembles to smooth the predictive variance.
However, adversarial training is an expensive training process. Adversarial examples are generated for each batch of data seen.
We seek a method to learn a distribution over parameters which does not require adversarial training.
Meta learning approaches use different kinds of weights in order to increase the generalization ability of neural networks.
The ﬁrst proposed method is fast weights (Hinton & Plaut, 1987) which uses an auxiliary (slow) network to produce weight changes in the target (fast) network, acting as a memory
Meta Networks (Munkhdalai & Yu, 2017) build on this approach by using an external neural memory store in addition to multiple sets of fast and slow weights.
(Ba et al., 2016) augment recurrent networks with fast weights as a more biologically plausible memory system than Neural Turing Machines. Unfortunately, the generation of each predicting network requires querying the base (slow) learner many times.
Many of the methods presented there, along with hyperparameter learning (Lorraine & Duvenaud, 2018), and the original hypernetwork, propose learning target weights which are deterministic functions of the training data.
Our method instead captures a distribution over parameters, and provides a cheap way to directly sample full networks from a learned distribution.
H YPER GAN
Taking note from the original hypernetwork framework for generating neural networks (Ha et al., 2016), we coin our approach HyperGAN. Standard GAN training dictates that we train a generator
G to model the target distribution, aided by a discriminator network D . This entails acquiring a large dataset of trained neural network parameters, and providing samples to D to judge the output of the generator. However, a large collection of neural networks would be extremely costly to build. Instead, our data distribution is built online by evaluating the performance of our generated parameters at each step.
We begin with assuming that a neural network F ( x ; θ ) with input x and parameters θ consisting of a given architecture with N layers, and a training set M with inputs and targets ( X, Y ) =
{ x i , y i } Mi =1 .
The standard training regime consists of computing a loss function L ( F ( x ; θ ) , y ) and updating the parameters θ with backpropagation until L is minimized.
This works if we only want a point estimate of θ .
However, if we want to generate more than one non-trivial network, some diversity is needed.
Our approach to creating diversity is similar to a regular GAN architecture in that we start by drawing a random sample s ∼ S =
N ( 0 , I j ) , where 0 is an all-zero vector and I j is a j × j identity matrix, and generate parameters using a generator from s .
Figure 1 shows the HyperGAN architecture.
Distinct from the standard GAN, we use parallel, untied generators to form the parameters of each layer.
In order for that to work, we observe that parameters between network layers are not independent, but are strongly correlated. The parameters of each layer depend on the output and the parameters of the preceding layers.
Therefore, the generated parameters must be correlated to produce well-performing neural networks.
We propose to add a
Mixer Q which maps s ∼ S to the mixed latent space Z ∈ R Nd , d < j .
An Nd -dimensional
Q ( s ) is then partitioned into N layer embeddings [ q 1 , . . . , q N ] , each being a d -dimensional vector.
Finally, we use N parallel generators G = { G 1 ( q 1 ) . . .
G N ( q n ) } to generate the parameters θ G for all layers in F . After generation, we can evaluate the new model F ( x, θ G ) on the training set.
We deﬁne an objective which minimizes the error of generated parameters with respect to a task loss L :
[Equation]
At each training step we generate a different network G ( Q ( s )) from a random s ∼ S , and then evaluate the loss function on a mini-batch from the training set. The resulting loss is backpropagated through the generators until θ G minimizes the target loss L .
The formulation in (1) presents a problem: the codes sampled from Q ( s ) will certainly collapse to the maximum likelihood (ML) estimate (when L is a log-likelihood). This means that the generators will learn a very narrow approximation of Θ , and indeed we see this happen in tables 3 and 4. To assure that the parameters are well distributed, we added an adversarial constraint on the mixed latent space D ( Q ( s )) so that it should not deviate too much from a Gaussian prior P . This constraint makes it closer to the generated parameters and ensures that Q ( s ) itself does not collapse to always outputting the same latent code.
With this we arrive at the HyperGAN objective: [Equation] Where β is a hyperparameter, and D is the regularization term which penalizes the distance between the prior and the distribution of latent codes. In practice D could be any distance function between two distributions.
We choose to parameterize D as a discriminator network D that output probabilities, and use the adversarial loss (Goodfellow et al., 2014) to approximate D ( Q ( s ) , P ) .
Note that while P and S are both multivariate Gaussians, they are distinct distributions as seen in ﬁgure 1.
[Equation]
Note that we ﬁnd it difﬁcult to learn a discriminator in the output (parameter) space because the dimensionality is high and there is no structure in those parameters to be utilized as in images (where CNNs can be trained). Our experiments show that regularizing in the latent space works well. We hypothesize this is because of overparametrization in θ .
The latent space initializes with random projections, which have a restricted isometry property according to the Johnson-Lindenstrauss lemma, hence if the mixed latent factor q is Gaussian, a random projection from it preserves the distance and diversity from q to θ . If θ is indeed severely overparametrized, and it is possible to generate diverse parameters by maximizing the likelihood, then the generators would not collapse to a single θ , since that would require breaking the restricted isometry from the initialization. This framework is general and can be adapted to a variety of tasks and losses. In this work, we show that HyperGAN can operate in both classiﬁcation and regression settings.
For multi-class classiﬁcation, the generators and mixer are trained with the cross entropy loss function:
[Equation]
where θ = { G 1 ( q 1 ) , . .
. , G n ( q n ) } [Equation]
For regression tasks we replace the cross entropy term with the mean squared error function:
[Equation]
where θ = { G 1 ( q 1 ) , . . . , G n ( q n ) } [Equation]
In implicit generative models such as GAN or WAE (Tolstikhin et al., 2017), it is always necessary to have a collection of inlier data points to train with. Inlier samples come from the distribution that is being estimated, and learning without them is difﬁcult. HyperGAN does not have a given set of inlier points to train with.
Instead, HyperGAN uses the error on the target loss to estimate the distance between the generated samples θ G ∈ Θ G and the target samples θ ∈ Θ . We note that θ G represents the maximum likelihood estimate of F ( x ; θ ) . (6) shows that by minimizing the error of the ML estimate on the log-likelihood, we are in fact minimizing the KL divergence between the target distribution Θ and the generated samples θ G ∈ Θ G . [Equation]
Where the term log P ( x | θ ) is the entropy of the target distribution and does not contribute to the optimization problem.
The derived likelihood function is ﬂexible and used widely across different domains.
In our work it is represented by the cross entropy and MSE losses that we study.
Its useful to note here how HyperGAN is distinct from other density estimators such as GAN and
WAE.
These methods use a likelihood function, such as MSE, to estimate the quality of the samples (in GANs the likelihood function is learned via the discriminator).
The likelihood is given as the distance from generated samples, to given samples of the target distribution.
HyperGAN instead estimates both the target and the approximation.
We assume the target distribution exists, and update our approximation Θ G to better match Θ .
If we have samples of the target distribution, we can reduce HyperGAN to a GAN (by moving the discriminator to the output space).We thoroughly explore the connections and differences to GANs and WAE in A.5.
HyperGAN is different from fully probabilistic approaches such as MNF (Louizos & Welling, 2016) since one cannot compute the probability of the generated θ , nor can one encode a generated θ back to the latent spaces.
E XPERIMENTS 4. H IGH L EVEL D ESCRIPTION AND E XPERIMENTAL S ETUP
We conduct a variety of experiments to show HyperGAN’s ability to achieve both high accuracy and obtain accurate uncertainty estimates.
First we show classiﬁcation performance on both MNIST and CIFAR-10 datasets.
Next we examine HyperGAN’s ability to learn the variance of a simple 1D dataset.
We perform experiments on anomaly detection: testing HyperGAN on notMNIST, andclasses of CIFAR-10 which are hidden during training.
We also examine adversarial examples as extreme cases of off-manifold data, and test our robustness to them.
In our experiments we compare with (Louizos & Welling, 2016) (MNF) as well as standard ensembles.
In all experiments we report results with two HyperGANs, one trained on MNIST and another on CIFAR-10.
Both of our models take a 256 dimensional sample of S as input, but have different sized mixed latent spaces.
The HyperGAN for the MNIST experiments consists of three weight generators, each using a 128 dimensional latent point as input.
The target network for the MNIST experiments is a small two layer convolutional network, using leaky ReLU activations and 2x max pooling after each convolutional layer.
Our HyperGAN trained on CIFAR-10 used 5 weight generators and latent points with dimensonality 256.
The target architecture for CIFAR-10 tests consists of three convolutional layers, each followed by leaky ReLU and 2x2 max pooling. The exact architectures we used for the target networks is given in A.4. In our experiments we used the same network architecture across different methods. Note that our architecture is different from the
LeNet-5 used in MNF, yet we see similar results as reported in (Louizos & Welling, 2016).
H YPER GAN D ETAILS
The mixing network, generators, and discriminator are each parameterized by 2 layer MLPs with 512 units each and ReLU nonlinearities.
We found in a pilot study that larger networks offered little performance beneﬁt, and ultimately hurt scalability.
In all experiments, we pretrain the mixer so that the mean and covariance of Q ( q | s ) are close to P .
It should be noted that the architecture is ﬂexible.
The number and width of layers may be varied without harming HyperGANs ability to model the target distribution.
We trained our HyperGAN on MNIST using less than 1.5GB of memory on a single GPU, while CIFAR-10 used just 4GB, making HyperGAN surprisingly scalable.
In Table 1 we show some statistics of the networks generated by HyperGAN on MNIST.
We note that
HyperGAN can generate very diverse networks, as the variance of network parameters generated by the HyperGAN is signiﬁcantly higher than standard training from different random initializations.
4. C LASSIFICATION
First we evaluate the classiﬁcation accuracy of HyperGAN on MNIST and CIFAR-10.
Classiﬁcation serves as an entrance exam into our other experiments, as the distribution we want to learn is over parameters which can effectively solve the classiﬁcation task.
We test with both single network samples, and ensembles.
For our ensembles we average predictions from N sampled models with
[Table] the scoring rule p ( y | x ) =
1 N (cid:80) Nn [-=
-]=1 p n ( y | x, θ n ) .
It should be noted that we did not perform ﬁne tuning, or any additional training on the sampled networks.
The results are shown in Table 2.
We generate ensembles of different sizes and compare against both Bayesian (Louizos & Welling, 2016) (Krueger et al., 2017) and non-Bayesian (Lakshminarayanan et al., 2017) methods, as well as MC dropout ( Ghahramani We
[Table] MNIST and CIFAR-10. In order to compare against MNF and Deep Ensembles, we also train a HyperGAN on only the ﬁrst 5 classes of CIFAR10, which we denote as CIFAR-5. In addition we examine our generalization ability by training on only 5000 examples of MNIST and CIFAR-10 with a small target network.
We do not attempt to outperform state of the art, but we perform better than other probabilistic neural network approaches
4. 1-D T OY R EGRESSION T ASK
We next evaluate the ability of HyperGAN to ﬁt a simple 1D function from noisy samples.
This dataset was ﬁrst proposed by (Hern´andez-Lobato & Adams, 2015), and consists of a training set of 20 points drawn uniformly from the interval [ − 4 , 4] .
The targets are given by y =
x 3 + (cid:15) where (cid:15) ∼ N (0 , 3 2 ) .
We used the same target architecture as in (Hern ´ andez-Lobato & Adams, 2015)
Lakshminarayanan et al.
(2017) and (Louizos & Welling, 2016): a one layer neural network with 100 hidden units and ReLU nonlinearity.
For HyperGAN we use two layer generators, andhidden units across all networks.
Because this is a small task, we use only a 64 dimensional latent space. MSE loss is used as our target loss function to train HyperGAN.
Results in ﬁgure 2 show that HyperGAN clearly learns the target function and captures the variation in the data well.
In addition, it can be seen that sampling more networks to compose a larger ensemble improves predictive uncertainty as we sample farther from the mean of the training data.
4. A NOMALY D ETECTION
To test our uncertainty measurements, we perform the same experiments as (Louizos & Welling, 2016) (MNF), (Lakshminarayanan et al., 2017); we measure the total entropy in predictions from
HyperGAN-generated networks.
We compare our uncertainty measurements with those of MNF and standard ensembles. For MNIST experiments we train a HyperGAN on the MNIST dataset, and test on the notMNIST dataset, which consists of 28x28 binary images of alphabetic letters.
In this setting, we want the softmax probabilities on inlier MNIST examples to have maximum entropya single large activation close to 1.
On off-manifold data we want to have equal probability across predictions.
We test our CIFAR-10 model by training on the ﬁrst 5 classes, and we use the latter 5 classes as out of distribution examples.
To build an estimate of the predictive entropy we sample multiple networks from HyperGAN per example, and measure their predictive entropy..
In Fig.
3 we show that HyperGAN can separate CIFAR-10 inlier and outlier samples much better than MNF or standard ensembles.
HyperGAN is less certain about data it does not recognize, as the probability of a low entropy prediction is overall lower on outliers.
On notMNIST we also show separation, though HyperGAN is also overall less conﬁdant about inliers then MNF.
Conventionally trained ensembles without the HyperGAN, referred to as L2 networks in the ﬁgure, are highly overconﬁdent on outliers and cannot provide a notion of uncertainty.
to conventional ensembles
4. A DVERSARIAL D ETECTION
We employ the same experimental setup to the detection of adversarial examples, an extreme sort of off-manifold data.
Adversarial examples are often optimized to lie within a small neighborhood of a classiﬁer’s decision boundaries.
They are created by adding perturbations in the direction of the greatest loss with respect to the model’s parameters.
Because HyperGAN learns a distribution over parameters, it should be more robust to attacks.
We generate adversarial examples using the Fast
Gradient Sign method (FGSM) (Goodfellow et al., 2015) and Projected Gradient Descent (PGD) (Madry et al., 2017).
FGSM adds a small perturbation (cid:15) to the target image in the direction of greatest loss.
FGSM is known to underﬁt to the target model, hence it may transfer well across many similar models.
In contrast, PGD takes many steps in the direction of greatest loss, producing a stronger adversarial example, at the risk of overﬁtting to a single set of parameters.
This poses the following challenge: to detect attacks by FGSM and PGD, HyperGAN will need to generate diverse parameters to avoid both attacks.
To detect adversarial examples, we ﬁrst hypothesize that a single adversarial example will not fool the entire space of parameters learned by HyperGAN. If we then evaluate adversarial examples against many generated networks, then we should see a high entropy among predictions (softmax probabilities) for any individual class.
transfers better than PGD, though both attacks fail to cover the distribution learned by HyperGAN
Adversarial examples have been shown to successfully fool ensembles (Dong et al., 2017), but with
HyperGAN one can always generate signiﬁcantly more models that can be added to the ensemble for the cost of one forward pass, making it hard to attack against.
In Fig.
4 we test HyperGAN against adversarial examples generated to fool one network.
It is shown that while those examples can fool 50% − 70% of the networks generated by HyperGAN, they usually do not fool all of them.
We compare the performance of HyperGAN with ensembles of N ∈ { 5 , 10 } models trained on
MNIST with normal supervised training.
We fuse their logits (unnormalized log probabilities) together as l ( x ) =
(cid:80) Nn =1 w n l n ( x ) where w n is the n th model weighting, and l n is the logits of the n th model.
In all experiments we consider uniformly weighted ensembles.
For HyperGAN we simply sample the generators to create as many models as we need, then we fuse their logits together.
Speciﬁcally we test ensembles with N ∈ { 5 , 10 , 100 , 1000 } members each.
Adversarial examples are generated by attacking the ensemble directly until the generated image completely fools the ensemble.
For HyperGAN, we attack the ensemble, but test with a new ensemble of equal size. For MNF we follow their experimental setup and do not form an ensemble.
Rather we attack sampled networks and measure the entropy of their predictions. For the purposes of detection, we compute the entropy within the predictive of of the ensemble members to score the example on the likelihood that it was drawn from the training distribution.
Figure 5 shows that HyperGAN clearly identiﬁes adversarial examples as being out-ofdistribution, performing better than MNF as well as standard ensembles.
HyperGAN is especially suited to this task as adversarial examples are optimized against parameters - parameters which HyperGAN can change.
We ﬁnd that we can successfully detect over 97% of adversarial examples, with a low false positive rate for both attacks just by thresholding the entropy.
D ISCUSSION AND F UTURE D IRECTIONS
We have proposed a generative model for parameter selection which performs strongly on detecting out-of-distribution samples as well as classiﬁcation.
Training a GAN to learn a probability distribution over parameters allows us to non-deterministically sample diverse, performant networks which we can use to form ensembles that can give good uncertainty estimates.
Our method is ultimately scalable to any number of networks in the predicting ensemble, requiring just one forward pass to generate a new set of parameters and a low GPU memory footprint.
We showed that we can generate models with signiﬁcant variation over the learned distribution and thus provide uncertainty estimates on outlier data.
There is still much room for exploration, we believe that HyperGAN may be useful in a variety of domains including meta learning and reinforcement learning
A A PPENDIX A. G ENERATED F ILTER E XAMPLES
We show the ﬁrst ﬁlter in 25 different networks generated by the HyperGAN to illustrate their difference in Fig.
It can be seen that qualitatively HyperGAN learns to generate classiﬁers with a variety of ﬁlters.
[Figure]
A. O UTLIER E XAMPLES
In Figure 7 we show images of examples which do not behave like most of their respective distribution.
On top are MNIST images which HyperGAN networks predict to have high entropy.
We can see that they are generally ambiguous and do not ﬁt with the rest of the training data.
The bottom row shows notMNIST examples which score with low entropy according to HyperGAN.
It can be seen that these examples look like they could come from the MNIST training distribution, making HyperGAN’s predictions reasonable [Figure] A. D IVERSITY WITHOUT M IXER OR D ISCRIMINATOR
We run experiments on both MNIST and CIFAR-10 where we remove both the mixer and the discriminator.
Tables 3 and 4 show statistics of the networks generated by HyperGAN using only independent Gaussian samples to the generators.
In this setting, HyperGAN learns to generate only a very small distribution of parameters
[Table]
[Table]
H YPER GAN N ETWORK D ETAILS
In tables 5 and 6 we show how the latent points are transformed through the generators to become a full layer of parameters.
For a MNIST based HyperGAN we generate layers from small latent points of dimensionality 128 .
For CIFAR-10 based HyperGANs we use a larger dimensionality of 256 for the latent points.
[Table]
[Table]
A. C ONNECTIONS TO GAN S AND W ASSERSTEIN A UTO - ENCODERS
Despite being largely applied to the same task of learning probability distributions
[Table]
implicit generative models such as Generative Adversarial Networks (GAN) and Wasserstein Auto-encoders (WAE) are presented under different frameworks.
GANs are a likelihood free density estimator, specifying only a generator
[Table]
and do not have any notion of reconstruction.
WAEs ﬁnd an optimal transport map between target and model distributions by ﬁrst encoding samples to a latent space and reconstructing the original data samples using a generator.
Even with these differences
[Table]
many combinations of these models have been proposed.
Creating composite models and loss functions from the GAN and WAE building blocks is commonplace.
HyperGAN is also an implicit generative model, in that there is no explicit data distribution from which to sample from.
HyperGAN learns to estimate from the data distribution not from real samples
[Table]
but from the error of its estimates.
We believe that HyperGAN is not just a combination of existing generative models.
To show this we give an overview of both GANs and WAE, and contrast
HyperGAN to them.
G ENERATIVE A DVERSARIAL N ETWORKS
GANs in the original formulation (Goodfellow et al., 2014) play a minimax game between two neural networks.
A generator G : Z → X samples a prior distribution P Z on the latent space Z and attempts to produce outputs from the data distribution X .
A discriminator D evaluates samples from X and from G ( Z ) and outputs a score between [0 , 1] .
This score given by the discriminator corresponds to how likely the input is to have come from X , according to D .
This creates the following minimax game between the generator and the discriminator:
[Equation]
The goal of the generator is to learn to transform noise samples into samples which fool the discriminator, thus minimizing the loss on G .
While the discriminator tries to maximize its ability to tell real images from fake samples from the generator.
In contrast, HyperGAN is not a likelihood free model like a GAN.
While GANs do not specify an explicit evaluation criterion, HyperGAN minimizes the negative likelihood of the generated parameters with an explicit criterion: the loss on the target task.
HyperGAN then has the beneﬁt of a principled quantitative evaluation of its samples, and does not need to rely on metrics like the
Inception (Salimans et al., 2016) or FID scores (Heusel et al., 2017) commonly used with GANs.
W ASSERSTEIN A UTO - ENCODERS
Wasserstein Auto-encoders (WAE) (Tolstikhin et al., 2017) use a similar formulation under the optimal transport framework (Villani, 2008) to minimize the difference between generative model P G and the data distribution P X , where X ∈ X in this case are images.
In the WAE, A latent space Z and prior distribution P Z on Z are deﬁned as sample spaces.
In general, there is an optimal transport map between P X and P G if the data distribution P X is well distributed according to a prior distribution (in this case P Z ).
The problem is that the data distribution P X is not a well deﬁned distribution, and so computation of an OT mapping is intractable.
To remedy this, an encoder Q : X → Z is used to project points from the data distribution to the latent space.
WAE considers the marginal distribution Z ∼ Q ( Z | X ) .
The conditional Q ( Z | X ) represents the aggregated posterior Q Z : the space of Z which we can optimize over with an encoder.
The authors argue that this allows WAE to not have to search over all possible maps P X
→ P G , but only those originating from the better behaved Q Z .
The encoder’s aggregated posterior Q Z still needs to be distributed according to prior
P Z , so a penalty is imposed on Q Z to match the mean and covariance of P Z .
In principle the penalty term D Z ( P Z , Q Z ) could be any distance between probability distributions.
WAE uses either MMD or a discriminator trained with an adversarial loss.
[Equation]
Where c is any measurable cost function measuring the error between the data X and reconstruction
G ( Z ) , and λ is a hyperparameter.
While HyperGAN and WAE use a similar architecture, HyperGAN is not an auto-encoder.
As such, HyperGAN replaces the WAE reconstruction cost function c ( X, G ( Z )) with a term that does not depend on X .
Instead, X is hidden, and HyperGAN minimizes the KL divergence between estimates θ G ∈ Θ G and the true parameter distribution Θ .
Because the Wasserstein distance does not reduce to the KL divergence, we can not exactly compute the optimal transport map:
[Equation]
[Equation]
because we cannot compute the reconstruction cost c .
To compute c we need samples from the target distribution.
Instead, we can only approximate the optimal map by minimizing
[Equation]
Which nonetheless gives us well-behaved samples from a learned distribution of parameters.