O N I NCORPORATING I NDUCTIVE B IASES INTO VAE S
Anonymous authors
Paper under double-blind review
A BSTRACT
We explain why directly changing the prior can be a surprisingly ineffective mechanism for incorporating inductive biases into variational auto-encoders (VAEs), and introduce a simple and effective alternative approach: Intermediary Latent
Space VAEs (InteL-VAEs).
InteL-VAEs use an intermediary set of latent variables to control the stochasticity of the encoding process, before mapping these in turn to the latent representation using a parametric function that encapsulates our desired inductive bias(es).
This allows us to impose properties like sparsity or clustering on learned representations, and incorporate human knowledge into the generative model.
Whereas changing the prior only indirectly encourages behavior through regularizing the encoder, InteL-VAEs are able to directly enforce desired characteristics.
Moreover, they bypass the computation and encoder design issues caused by non-Gaussian priors, while allowing for additional flexibility through training of the parametric mapping function.
We show that these advantages, in turn, lead to both better generative models and better representations being learned.
I NTRODUCTION
VAEs provide a rich class of deep generative models (DGMs) with many variants (Kingma & Welling, 2014;
Rezende & Mohamed, 2015; Burda et al., 2016; Gulrajani et al., 2016; Vahdat & Kautz, 2020).
Based on an encoder-decoder structure, VAEs encode datapoints into latent embeddings before decoding them back to data space.
By parameterizing the encoder and decoder using expressive neural networks, VAEs provide a powerful basis for learning both generative models and representations.
The standard VAE framework assumes an isotropic Gaussian prior.
However, this can cause issues, such as when one desires the learned representations to exhibit some properties of interest, for example sparsity (Tonolini et al., 2020) or clustering (Dilokthanakul et al., 2016), or when the data distribution has very different topological properties from a Gaussian, for example multi-modality (Shi et al., 2020) or group structure (Falorsi et al., 2018).
Therefore, a variety of recent works have looked to use non-Gaussian priors (van den Oord et al., 2017; Tomczak & Welling, 2018; Casale et al., 2018;
Razavi et al., 2019; Bauer & Mnih, 2019), often with the motivation of adding inductive biases into the model (Davidson et al., 2018b; Mathieu et al., 2019b; Nagano et al., 2019; Skopek et al., 2019).
In this work, we argue that this approach of using non-Gaussian priors can be a problematic, and even ineffective, mechanism for adding inductive biases into VAEs.
Firstly, non-Gaussian priors will often necessitate complex encoder models to maintain consistency with the prior’s shape and dependency structure (Webb et al., 2018).
These will typically no longer permit simple parameterization, substantially complicating both model construction and training.
Secondly, the latent encodings are still not guaranteed to follow the desired structure: the ‘prior’ is not really a prior in a Bayesian sense and only appears in the training objective as a regularizer on the encoder; this regularization might not be sufficient to induce the desired behavior.
Indeed, Mathieu et al.
(2019b) find that changing the prior is typically insufficient in practice to learn the desired representations at a population level , with mismatches occurring between the data distribution and learned model.
To provide an alternative, more effective, approach that does not suffer from these pathologies, we introduce Intermediary Latent Space VAEs (InteL-VAEs), an extension to the standard VAE framework that allows a wide range of powerful inductive biases to be incorporated while maintaining an isotropic Gaussian prior.
This is achieved by introducing an intermediary set of latent variables that deal with the stochasticity of the encoding process before incorporating the desired inductive biases via a parametric function that maps these intermediary latents to the latent representation itself, with the decoder taking this final representation as input [Equation] 𝑥𝑥 [Equation]
[Figure]
The InteL-VAE framework provides a variety of advantages over directly replacing the prior.
Firstly, it directly enforces our inductive biases on the representations, rather than relying on the regularizing effect of the prior to encourage this implicitly.
Secondly, it provides a natural congruence between the generative and representational models via sharing of the mapping function, side-stepping the issues that non-Gaussian priors can cause for the inference model.
Finally, it allows for more general and more flexible inductive biases to be incorporated, by removing the need to express them with an explicit density function and allowing for parts of the mapping to be learned during training.
To further introduce a number of novel specific realizations of the InteL-VAE framework, showing how they can be used to incorporate various inductive biases, enforcing latent representations that are, for example, multiply connected, multi-modal, sparse, or hierarchical.
Experimental results show their superiority compared with baseline methods in both generation and feature quality, most notably providing state-of-the-art performance for learning sparse representations in the VAE framework.
To summarize, we a) highlight the need for inductive biases in VAEs and explain why directly changing the prior is a suboptimal means for incorporating them; b) propose InteL-VAEs as a simple but effective general framework to introduce inductive biases; and c) introduce specific InteL-VAE variants which can learn improved generative models and representations over existing baselines on a number of tasks.
Accompanying code is provided at github.com/djkdsjwkjerkjermf/InteL- [Table]
T HE N EED FOR I NDUCTIVE B IASES IN VAE S
Variational auto-encoders (VAEs) are deep stochastic auto-encoders that can be used for learning both deep generative models and low-dimensional representations of complex data.
Their key components are an encoder, q ϕ ( z | x ) , which probabilistically maps from data x ∈ X to latents z ∈ Z ; a decoder, p θ ( x | z ) , which probabilistically maps from latents to data; and a prior, p ( z ) , that completes the generative model, p ( z ) p θ ( x | z ) , and regularizes the encoder during training.
The encoder and decoder are parameterized by deep neural networks and are simultaneously trained using a dataset { x 1 , x 2 , ..., x N } and a variational lower bound on the log-likelihood, most commonly,
[Equation]
Namely, we optimize L ( θ, ϕ ) :
= E x ∼ p data (x) [ L ( x, θ, ϕ )] , where p data (x) represents the empirical data distribution.
Here the prior is typically fixed to a standard Gaussian, i.e. p ( z ) =
N ( z ; 0 , I ) .
While it is well documented that this standard VAE setup with a ‘Gaussian’ latent space can be suboptimal (Davidson et al., 2018a; Mathieu et al., 2019b; Tomczak & Welling, 2018; Bauer & Mnih, 2019; Tonolini et al., 2020), there is perhaps less of a unified high-level view on exactly when, why, and how one should change it to incorporate inductive biases.
Note here that the prior does not play the same role as in a Bayesian model: because the latents themselves are somewhat arbitrary and the model is learned from data, it does not encapsulate our initial beliefs in the way one might expect.
We argue that there are two core reasons why inductive biases can be important for VAEs: (a) standard VAEs can fail to encourage, and even prohibit, desired structure in the representations we learn; and (b) standard VAEs do not allow one to impart prior information or desired topological characteristic into the generative model .
Considering the former, one often has some a priori desired characteristics, or constraints, on the representations learned (Bengio et al., 2013).
For example, sparse features can be desirable because they can improve data efficiency (Yip & Sussman, 1997), and provide robustness to noise (Wright et al., 2009; Ahmad & Scheinkman, 2019) and attacks (Gopalakrishnan et al., 2018).
In other settings one might desire clustered (Jiang et al., 2017), disentangled (Ansari & Soh, 2019; Kim & Mnih, 2018;
Higgins et al., 2018) or hierarchical representations (Song & Li, 2013; Sønderby et al., 2016; Zhao et al., 2017).
The KL-divergence term in Eq.
(1) regularizes the encoding distribution towards the prior and, as a standard Gaussian distribution typically does not exhibit our desired characteristics, this regularization can significantly hinder our ability to learn representations with the desired properties.
Not only can this be problematic at an individual sample level, it can cause even more pronounced issues at the population level : desired structural characteristics of our representations often relate to the pushforward distribution of the data in the latent space, q ϕ ( z ) :=
E p data (x) [ q ϕ ( z | x )] , which is both difficult to control and only implicitly regularized to the prior (Hoffman & Johnson, 2016).
[Figure]
Inductive biases can also be essential to the generation quality of VAEs: because the generation process of standard VAEs is essentially pushing-forward the Gaussian prior on Z to data space X by a ‘smooth’ decoder, there is an underlying inductive bias that standard VAEs prefer sample distributions with similar topology structures to
Gaussians.
As a result, VAEs can perform poorly when the data manifold exhibits certain different topological properties (Caterini et al., 2020).
For example, they can struggle when data is clustered into unconnected components as shown in Fig.
2, or when data is not simply-connected.
This renders learning effective mappings using finite datasets and conventional architectures (potentially prohibitively) difficult.
In particular, it can necessitate large Lipschitz constants in the decoder, causing knock-on issues like unstable training and brittle models (Scaman & Virmaux, 2018), as well as posterior collapse (van den
Oord et al., 2017; Alemi et al., 2018).
In short, the Gaussian prior of a standard VAE can induce fundamental topological differences to the true data distribution (Falorsi et al., 2018; Shi et al., 2020).
S HORTFALLS OF VAE S WITH NON -G AUSSIAN P RIORS
[Figure]
Though directly replacing the Gaussian prior with a different prior sounds like a simple solution, effectively introducing inductive biases can, unfortunately, be more complicated.
Firstly, the only influence of the prior during training is as a regularizer on the encoder through the D KL ( q ϕ ( z | x ) ∥ p ( z )) term.
This regularization is always competing with the need for effective reconstructions and only has an indirect influence on q ϕ ( z ) .
As such, simply replacing the prior can be an ineffective way of inducing desired structure at the population level (Mathieu et al., 2019b), particularly if p ( z ) is a complex distribution that it is difficult to fit (see, e.g., Fig. 3a).
Mismatches between q ϕ ( z ) and p ( z ) can also have further deleterious effects on the learned generative model: the former represents the distribution of the data in latent space during training, while the latter is what is used by the learned generative model, leading to unrepresentative generations if there is mismatch.
Secondly, it can be extremely difficult to construct appropriate encoder mappings and distributions for non-Gaussian priors.
While the typical choice of a mean-field Gaussian for the encoder distribution is simple, easy to train, and often effective for Gaussian priors, it is often inappropriate for other choices of prior.
For example, in Fig. 3, we consider replacement with a sparse prior.
A VAE with a
Gaussian encoder struggles to encode points in a manner that even remotely matches the prior.
One might suggest replacing the encoder distribution as well, but this has its own issues, most notably that other distributions can be hard to effectively parameterize or train.
In particular, the form of the required encoding noise might become heavily spatially variant; in our sparse example, the noise must be elongated in a particular direction depending on where the mean embedding is.
If the prior has constraints or topological properties distinct from the data, it can even be difficult to learn a mean encoder mapping that respects these, due to the continuous nature of neural networks.
T HE I NTE L-VAE F RAMEWORK
To solve the issues highlighted in the previous section, and provide a principled and effective method for adding inductive biases to VAEs, we propose Intermediary Latent Space VAEs (InteL-VAEs).
The key idea behind InteL-VAEs is to introduce an intermediary set of latent variables y ∈ Y , used as a stepping stone in the construction of the representation z ∈ Z .
Data is initially encoded in Y using a conventional VAE encoder (e.g. a mean-field Gaussian) before being passed through a non-stochastic mapping g ψ : Y (cid:55)→ Z that incorporates our desired inductive biases and which can be trained, if needed, through its parameters ψ .
The prior is defined on Y and taken to be a standard Gaussian, p ( y ) =
N ( y ; 0 , I ) , while our representations, z
= g ψ ( y ) , correspond to a pushforward of y .
By first encoding datapoints to y , rather than z directly, we can deal with all the encoder and prior stochasticity in this first, well-behaved, latent space, while maintaining z as our representation and using it for the decoder p θ ( x | z ) .
In principle, g ψ can be any arbitrary parametric (or fixed) mapping, including non-differentiable or even discontinuous functions.
However, to allow for reparameterized gradient estimators (Kingma & Welling, 2014; Rezende & Mohamed, 2015), we will restrict ourselves to g ψ that are sub-differentiable (and thus continuous) with respect to both their inputs and parameters.
Note that setting g ψ to the identity mapping recovers a conventional VAE.
As shown in Fig.
1, the auto-encoding process is now X q ϕ −→ Y g ψ −−→ Z p θ −→ X .
This three-step process no longer unambiguously fits into the encoder-decoder terminology of the standard VAE and permits a variety of interpretations; for now we take the convention of calling q ϕ ( y | x ) the encoder and p θ ( x | z ) the decoder, but also discuss some alternative interpretations below.
We emphasize here that these no longer respectively match up with our representation model—which corresponds to passing an input into the encoder and then mapping the resulting encoding using g ψ —and our generative model—which corresponds to N ( y ; 0 , I ) p θ ( x | z = g ψ ( y )) , such that we sample a y from the prior and then pass this through through g ψ and the decoder in turn.
The mapping g ψ introduces inductive biases into both the generative model and our representations by imposing a particular form on z , such as the spherical structure enforced in Fig.
1 (see also Sec. 6).
It can be viewed as a shared module between them, ensuring congruence between the two.
This congruence allows us to more directly introduce inductive biases through careful construction of g ψ , without complicating the process of learning an effective inference network.
In particular, because Y is treated as our latent space for the purposes of training, we sidestep the inference issues that non-Gaussian priors usually cause.
Moreover, because all samples must explicitly pass through g ψ during both training and generation, we can more directly ensure the desired structure is enforced without causing a mismatch in the latent distribution between training and deployment.
Training As with standard VAEs, training of an InteL-VAE is done by maximizing a variational lower bound (ELBO) on the log evidence, which we denote L Y .
Most simply, we have
[Equation]
Note that the regularization is on y , but our representation corresponds to z = g ψ ( y ) .
Training corresponds to the optimization arg max θ,ϕ,ψ E x ∼ p data (x)
[ L Y ( x, θ, ϕ, ψ )] , which can be performed using stochastic gradient ascent with reparameterized gradients in the standard manner.
Although inductive biases are introduced, the calculation, and optimization, of L Y is thus equivalent to the standard ELBO.
In particular, parameterizing q ϕ ( y | x ) with a Gaussian distribution still yields an analytical D KL ( q ϕ ( y | x ) ∥
N ( y ; 0 , I )) term.
Alternative Interpretations It is interesting to note that our representation, g ψ ( y ) , only appears in the context of the decoder in this training objective. As such, we see that an important alternative interpretation of InteL-VAEs is to consider g ψ as being a customized first layer in the decoder, and our test–time representations as partial decodings of the latents y .
This viewpoint allows it to be applied with more general bounds and VAE variants (e.g. Burda et al.
; Le et al. Maddison et al. ; Naesseth et al. ; Zhao et al.
(2019)), as it requires only a carefully customized decoder architecture during training and an adjusted mechanism for constructing representations at test–time.
Yet another interpretation is to think about InteL-VAEs as implicitly defining a conventional VAE with latents z , but where both the non-Gaussian prior, p ψ ( z ) , and our encoder distribution, q ϕ,ψ ( z | x ) , are themselves defined implicitly as pushforwards along g ψ , which acts as a shared module that instills a natural compatibility between the two. Formally we have the following theorem.
Theorem 1.
Let p ψ ( z ) and q ϕ,ψ ( z | x ) represent the respective pushforward distributions of N (0 , I ) and q ϕ ( y | x ) induced by the mapping g ψ : Y (cid:55)→ Z .
The following holds for all measurable g ψ :
[Equation]
If g ψ is also an invertible function then the above becomes an equality and L Y equals the standard
ELBO on the space of Z as follows
[Equation]
The proof is given in Appendix A. Here, (3) shows that the divergence in our representation space
Z is never more than that in Y , or equivalently that the implied ELBO on the space of Z is always at least as tight as that on Y ; (4) shows they are exactly equal if g ψ is invertible.
As the magnitude of D KL ( q ϕ ( y | x ) ∥ N ( y ; 0 , I )) in an InteL-VAE will remain comparable to the KL divergence in a standard Gaussian prior VAE setup, this, in turn, ensures that D KL ( q ϕ,ψ ( z | x ) ∥ p ψ ( z )) does not become overly large.
This is in stark contrast to the conventional non-Gaussian prior setup, where it can be difficult to avoid D KL ( q ϕ ( z | x ) ∥ p ψ ( z )) exploding without undermining reconstruction (Mathieu et al., 2019b).
The intuition here is that having the stochasticity in the encoder before it is passed through g ψ ensures that the form of the noise in the embedding is inherently appropriate for the space: the same mapping is used to warp this noise as to define the generative model in the first place.
For example, when g ψ is a sparse mapping, the Gaussian noise in q ϕ ( y | x ) will be compressed to a sparse subspace by g ψ , leading to a sparse variational posterior q ϕ,ψ ( z | x ) as shown in Fig.
3b.
In particular, q ϕ ( y | x ) does not need to learn any complex spatial variations that result from properties of Z .
In turn, InteL-VAEs further alleviate issues of mismatch between p ψ ( z ) and q ϕ,ψ ( z ) .
Further Benefits A key benefit of InteL-VAEs is that the extracted features are guaranteed to have the desired structure.
Take the spherical case for example, all extracted features g ψ ( µ ϕ ( x )) lie within a small neighborhood of the unit sphere.
By comparison, methods based on training loss modifications, e.g. Mathieu et al.
(2019b), often fail to generate features with the targeted properties.
A more subtle advantage is that we do not need to explicitly specify p ψ ( z ) .
This can be extremely helpful when we want to specify complex inductive biases: designing a non-stochastic mapping is typically much easier than a density function, particularly for complex spaces.
Further, this can make it much easier to parameterize and learn aspects of p ψ ( z ) in a data-driven manner (see e.g. Sec. 6.3).
R ELATED WORK
Inductive biases There is much prior work on introducing human knowledge to deep learning models by structural design, such as CNNs (LeCun et al., 1989), RNNs (Hochreiter & Schmidhuber, 1997) and transformers (Vaswani et al., 2017).
However, most of these designs are on the sample level, utilizing low–level information such as transformation invariances or internal correlations in each sample.
By contrast, InteL-VAEs provide a convenient way to incorporate population level knowledge—information about the global properties of data distributions can be effectively utilized.
Non-Gaussian priors There is an abundance of prior work utilizing non-Gaussian priors to improve the fit and generation capabilities of VAEs, including MoG priors (Dilokthanakul et al., 2016; Shi et al., 2020), sparse priors (Mathieu et al., 2019b; Tonolini et al., 2020; Barello et al., 2018), Gaussianprocess priors (Casale et al., 2018) and autoregressive priors (Razavi et al., 2019; van den Oord et al., 2017).
However, these methods often require specialized algorithms to train and are primarily applicable only to specific kinds of data.
Moreover, as we have explained, changing the prior alone often provides insufficient pressure on its own to induce the desired characteristics.
Others have proposed non-Gaussian priors to reduce the prior-posterior gap, such as Vamp-VAE (Tomczak &
Welling, 2018) and LARS (Bauer & Mnih, 2019), but these are tangential to our inductive bias aims.
Non-Euclidean latents A related line of work has focused on non-Euclidean latent spaces.
For instance Davidson et al.
(2018a) leveraged a von Mises-Fisher distribution on a hyperspherical latent space, Falorsi et al.
(2018) endowed the latent space with a SO (3) group structure, and Mathieu et al.
(2019a)
; Ovinnikov (2019); Nagano et al.
with hyperbolic geometry.
Other spaces like product of constant curvature spaces (Skopek et al., 2019) and embedded manifolds (Rey et al., 2019) have also been considered.
However, these works generally require careful design and training.
Normalizing flows Our use of a non-stochastic mapping shares some interesting links to normalizing flows (NFs)
(Rezende & Mohamed, 2015; Papamakarios et al., 2019; Grathwohl et al., 2018; Dinh et al., 2017; Huang et al., 2018;
Papamakarios et al., 2018).
Indeed a NF would be a valid choice for g ψ , albeit an unlikely one due to their architectural constraints.
However, unlike previous use of NFs in VAEs, our g ψ is crucially shared between the generative and representational models, rather than just being used in the encoder, while the KL divergence in our framework is taken before, not after, the mapping.
Moreover, the underlying motivation, and type of mapping typically used, differs substantially: our mapping is used to introduce inductive biases, not purely to improve inference.
Our mapping is also more general than a NF (e.g. it need not be invertible) and does not introduce additional constraints or computational issues.
S PECIFIC R EALIZATIONS OF THE I NTE L-VAE F RAMEWORK
We now present several novel example InteL-VAEs, introducing various inductive biases through different choices of g ψ .
We will start with artificial, but surprisingly challenging, examples where some precise topological properties of the target distributions are known, incorporating them directly through a fixed g ψ .
We will then move onto experiments where we impose a fixed clustering inductive bias when training on image data, allowing us to learn InteL-VAEs that account effectively for multi-modality in the data distribution.
Finally, we consider the example of learning sparse representations of high–dimensional data.
Here we will see that it is imperative to exploit the ability of InteL-VAEs to learn aspects of g ψ during training, providing a flexible inductive bias framework, rather than a pre-fixed mapping.
By comparing InteL-VAEs with strong baselines, we show that InteL-VAEs are effective in introducing these desired inductive biases, and consequently both improve generation quality and learn better data representations for downstream tasks.
One note of particular importance is that we find that InteL-VAEs provide state-of-the-art performance for learning sparse
VAE representations.
A further example of using InteL-VAEs to learn hierarchical representations is presented in Appendix B, while full details on the various examples are given in Appendix C.
6. M ULTIPLE –C ONNECTIVITY
[Figure] InteL-VAE uses [Rows 1,2] circular prior with one hole, [Row 3] multiply-connected prior with two holes, and [Row 4] clustered prior. Vamp-VAE behaves similarly to a vanilla VAE; its results are presented in Fig. Data is often most naturally described on nonEuclidean spaces such as circles, e.g. wind directions (Mardia & Jupp, 2000), and other multiplyconnected shapes, e.g. holes in disease databases (Liu et al., 1997).
For reasons previously explained in Sec. 2, standard VAEs cannot practically model such topologies, which prevents them from learning generative models which match even the simplest data distributions with non-trivial topological structures, as shown in Fig.
4b.
Luckily, by designing g ψ to map the Gaussian prior to a simple representative distribution in a topological class, we can easily equip InteL-VAEs with the knowledge to approximate any data distributions with similar topological properties.
Specifically, by defining g ψ as the orthogonal projection to S 1 , g ψ ( z ) =
z/ ( || z
|| 2 + ϵ ) , we map the Gaussian prior approximately to a uniform distribution to S 1 , where ϵ is a small positive constant to ensure the continuity of g ψ near the origin.
From Rows 1 and 2 of Fig.
4, we find that this inductive bias gives InteL-VAEs the ability to learn various distributions with a hole. We can add further holes by simply ‘gluing’ point pairs.
For example, for two holes we can use
[Equation] which first map y to approximately S 1 , and then glues (0 , 1) and (0 , − 1) together to create new holes (see Fig. C.1 for an illustration). Furthermore, we can continue to glue points together to achieve a higher number of holes h , and thus more complex connectivity.
Row 3 of Fig.
4 gives an example of learning an infinity sign by introducing a ‘two-hole’ inductive bias.
Compared with vanilla-VAE and Vamp-VAE, which try to find a convex hull for real data distributions,
InteL-VAEs can deal with distributions with highly non-convex and very non-smooth supports (see
Fig. 4 and Appendix C.1).
We emphasize here that our inductive bias does not contain the information about the precise shape of the data, only the number of holes.
We thus see that InteL-VAEs can provide substantial improvements in performance by incorporating only basic prior information about the topological properties of the data, which point out a way to approximate distributions on more complex structures, such as linear groups (Gupta & Mishra, 2018).
6. M ULTI –M ODALITY
Many real-world datasets exhibit multi-modality.
For example, data with distinct classes are often naturally clustered into (nearly) disconnected components representing each class.
However, vanilla
VAEs generally fail to fit multi-modal data due to the topological issues explained in Sec. 2.
Previous work (Johnson et al., 2017; Mathieu et al., 2019b) has thus proposed the use of a multi-modal prior, such as a mixture of Gaussian (MoG) distribution, so as to capture all components of the data.
Nonetheless, VAEs with such priors often still struggle to model multi-modal data because of mismatch between q ϕ ( z ) and p ( z ) or training instability issues.
[Figure]
We tackle this problem by using a mapping g ψ which contains a clustering inductive bias.
The high-level idea is to design a mapping g ψ with a localized high Lipschitz constant that ‘splits’ the continuous Gaussian distribution into K disconnected parts and then pushes them away from each other. In particular, we split Y it into K equally sized sectors using its first two dimensions (noting it is not needed to split on all dimensions to form clusters), as shown in Fig.
For any point y , we can easily get the center direction r ( y ) of the sector that y belongs to and the distance dis ( y ) between y and the sector boundary.
Then we define g ψ ( y ) as:
[Equation]
where c 1 and c 2 are empirical constants. We can see that although g ψ has very different function on different sectors, it is still continuous on the whole plane with g ψ ( y ) = y on sector boundaries, which is desirable for gradient-based training. See Appendix C.2 for more details.
To assess the performance of our approach, we first consider a simple 2-component MoG synthetic dataset in the last row of Fig.
We see that the vanilla VAE fails to learn a clustered distribution that fits the data, while the InteL-VAE sorts this issue and fits the data well. [Table]
standard deviation To provide a more real-world example, we train an
InteL-VAE and a variety of baselines on the MNIST dataset, comparing the generation quality of the learned models using the FID score (Heusel et al., 2017) in Table 1.
We find that the GM-VAE (Dilokthanakul et al., 2016) and MoG-VAE (VAE with a fixed MoG prior) achieve performance gains by using non-Gaussian priors.
The Vamp-VAE (Tomczak & Welling, 2018) and a VAE with a Sylvester Normalizing Flow (Berg et al., 2018) encoder provide further gains by making the prior and encoder distributions more flexible respectively.
However, the InteL-VAE comfortably outperforms all of them.
[Table] [Figure]
To gain insight into how InteL-VAEs achieve superior generation quality, we perform analysis on a simplified setting where we select only the ‘0’ and ‘1’ digits from the MNIST dataset to form a strongly clustered dataset, MNIST-01 .
We further decrease the latent dimension to 1 to make the problem more challenging.
Fig.
6 shows that here the vanilla VAE generates some samples which look like interpolations between ’0’ and ’1’, meaning that it still tries to learn a connected distribution containing ’0’ and ’1’.
Further, the general generation quality is poor, with blurred images and a lack of diversity in generated samples (e.g. all the ‘1’s have the same slant).
Despite using a clustered prior, the MoG-VAE still produces unwanted interpolations between the classes.
By contrast, InteL-VAE generates digits that are unambiguous and crisper.
[Table]
[Table]
To quantify these results, we further train a logistic classifier on MNIST-01 and use it to classify images generated by each method.
For each method, we calculate the proportion of samples produced by the generative model that are assigned to each class by this pre-trained classifier, as well as the proportion of samples for which the classifier is uncertain.
From Table 2 we see that InteL-VAE significantly outperforms its competitors in the ability to generate balanced and unambiguous digits.
To extend this example further, and show the ability of InteL-VAEs to learn aspects of g ψ during training, we further consider parameterizing and then learning the relative size of the clusters.
Table 3 shows that this can be successfully learned by InteL-VAEs on MNIST-01 .
6. S PARSITY
Sparse features are often well-suited to data efficiency on downstream tasks (Huang & Aviyente, 2006), in addition to being naturally easier to visualize and manipulate than dense features (Ng et al., 2011).
However, existing VAE models for sparse representations trade off generation quality to achieve this sparsity (Mathieu et al., 2019b; Tonolini et al., 2020;
Barello et al., 2018).
Here, we show that InteL-VAEs can instead simultaneously increase feature sparsity and generation quality.
Moreover, they are able to achieve state-of-the-art scores on sparsity metrics.
Compared with our previous examples, the g ψ here needs to be more flexible so that it can learn to map points in a data-specific way and induce sparsity without unduly harming reconstruction.
To achieve this, we use the simple form for the mapping: g ψ ( y ) =
y ⊙ DS ψ ( y ) , where ⊙ is pointwise multiplication, and DS is a ‘dimension selector’ network that selects dimensions to deactivate given y .
DS outputs values between [0 , 1] for each dimension, with 0 being fully deactivated and 1 fully activated; the more dimensions we deactivate, the sparser the representation.
By learning DS during training, this setup allows us to learn a sparse representation in a data-driven manner.
To control the degree of sparsity, we add a sparsity regularizer, L sp , to the ELBO with weighting parameter γ (higher γ corresponds to more sparsity). Namely, we optimize L Y ( θ, ϕ, ψ ) + γ L sp ( ϕ, ψ ) , where [Equation] H (
[Equation]
= − (cid:80) i [Equation] v i
[Equation]
v ∥ 1 ) log ( v i / ∥ v ∥ 1 ) is the normalized entropy of an positive vector v , and the expectation is over drawing a minibatch of samples x 1 , . . .
, x M and then sampling each corresponding y i ∼ q ϕ ( ·| x = x i ) .
L sp encourages DS to deactivate more dimensions, while also encouraging diversity in which dimensions are activated for different data points, improving utilization of the latent space.
Please see Appendix C.3 for more details and intuitions.
Initial qualitative results are shown in Fig.
8, where we see that our InteL-VAE is able to learn sparse and intuitive representations.
[Figure] [Figure]
To quantitatively assess the ability of our approach to yield sparse representations and good quality generations we compare against vanilla VAEs, the specially customized sparse-VAE of Tonolini et al.
(2020), and the sparse version of Mathieu et al.
(2019b) (DD) on FashionMNIST (Xiao et al., 2017) and MNIST .
As shown in Fig.
7 ( left ), we find that InteLVAEs increase sparsity of the representations— measured by the Hoyer metric (Hurley & Rickard, 2009)—while increasing generative sample quality at the same time.
Indeed, the FID score obtained by InteL-VAE outperforms the vanilla VAE when γ < 3 . 0 , while the sparsity score substantially increases with γ , reaching extremely high levels.
By comparison, DD significantly degrades generation quality and only provides a more modest increase in sparsity, while its sparsity also drops if the regularization coefficient is set too high.
The level of sparsity achieved by sparse-VAEs was substantially less than both DD and InteL-VAEs.
To further evaluate the quality of the learned features for downstream tasks, we trained a classifier to predict class labels from the latent representations.
For this, we choose a random forest (Breiman, 2001) with maximum depth 4 as it is well-suited for sparse features.
We vary the size of training data given to the classifier to measure the data efficiency of each model.
Fig.
7 ( right ) shows that InteL-VAE typically outperforms other the models, especially in few-shot scenarios.
[Table]
Finally, to verify InteL-VAE’s effectiveness on larger and higher-resolution datasets, we also make comparisons on CelebA (Liu et al., 2015).
From Table 4, we can see that InteL-VAE increase sparse scores to 0.46 without sacrificing generation quality.
By comparison, the maximal sparse score that sparse-VAE gets is 0.30, with unacceptable sample quality.
Interestingly, InteL-VAEs with ly low regulation γ achieved particularly good generative sample quality, outperforming even the Vamp-VAE and a VAE with a Sylvester NF encoder.
Conclusions In this paper, we proposed InteL-VAEs, a general schema for incorporating inductive biases into VAEs.
Experiments show that InteL-VAEs can both provide representations with desired properties and improve generation quality, outperforming a variety of baselines such as directly changing the prior.
This is achieved while maintaining the simplicity and stability of standard VAEs.
E THICS S TATEMENT
We do not believe that there are direct ethical concerns regarding our paper: the datasets we consider are all already well established and do not contain sensitive information, while the methods and ideas we introduce have no clear direct potential negative societal impacts of their own.
From a bigger picture perspective, work like ours that looks to permit more effective incorporation of inductive biases into models can be thought of as allowing more direct human control on how models will behave after training.
While this will typically be a force for good, for example by encouraging model interpretability and providing mechanisms to try and induce positive characteristics like fairness, in rare circumstances there may also be the potential for this to be used nefariously by deliberately encouraging undesirable behavior.
However, we do not believe that our work is any more prone to such exploitation than existing methods or that the risk of it being used in such as a way is significant.
R EPRODUCIBILITY S TATEMENT
Full experimental details are given in Appendix C, while anonymized source code for reproducing all our experiments directly is provided at https://github.com/djkdsjwkjerkjermf/InteL-VAE.
Together these should make it straightforward for others to reproduce our empirical results.
We have been careful to provide quantitative metrics of performance whenever possible, rather than just relying on qualitative or anecdotal evidence.
Repeat runs and error bars are provided whenever this is feasible, with the level of variability always found to be sufficiently small to draw reliable and statistically sound conclusions.
In fact, the training stability and consistent performance of our general approach under retraining provides a clear advantage in itself compared to many of the baseline methods.
Full formal proof for our only theoretical result is given in Appendix A, while the assumptions it makes are clearly stated and easily verifiable. A PPENDIX A P ROOFS Theorem 1.
Let p ψ ( z ) and q ϕ,ψ ( z | x ) represent the respective pushforward distributions of N (0 , I ) and q ϕ ( y | x ) induced by the mapping g ψ : Y (cid:55)→ Z .
The following holds for all measurable g ψ :
[Equation]
If g ψ is also an invertible function then the above becomes an equality and L Y equals the standard
ELBO on the space of Z as follows
[Equation]
Proof.
We first prove the inequality from Eq.
(3), then we show that Eq.
(3) is actually an equality when g ψ is invertible, and finally we prove that the reconstruction term is unchanged by g ψ .
Let us denote by F and G the sigma-algebras of respectively Y and Z , and we have by construction a measurable map g ψ : ( Y , F ) →
( Z , G ) .
We can actually define the measurable space ( Z , G ) as the image of ( Y , F ) by g ψ , then g ψ is automatically both surjective and measurable.
1 We also assume that there exists a measure on Y , which we denote ξ , and denote with ν the corresponding pushforward measure by g ψ on Z .
We further have ν ( A ) =
ξ ( g − 1 ψ ( A )) for any A ∈ G .
We start by proving Eq.
(3), where the Kullback-Leibler (KL) divergence between the two pushforward measures 3 q ϕ,ψ ≜ q ϕ ◦ g − 1 ψ and p ψ ≜ p ◦ g − 1 ψ is upper bounded by D KL ( q ϕ ( y | x ) ∥ p ( y )) , where here we have p ( y ) =
N ( y ; 0 , I ) but we will use p as a convenient shorthand.
At a high-level, we essentially have that Eq.
(3) follows directly the data processing inequality (Sason, 2019) with a deterministic kernel
z = g ψ ( y ) .
Nonetheless, we develop in what follows a proof which additionally gives sufficient conditions for when this inequality becomes non-strict.
We can assume that
D KL ( q ϕ ( y | x ) ∥ N ( y ; 0 , I )) is finite, as otherwise the result is trivially true, which in turn implies q ϕ ≪ p .
For any A ∈ G , we have that if p ψ ( A ) =
p ◦ g − 1 ψ ( A ) =
p ( g − 1 ψ ( A ))
{+= 
+}
then this implies q ϕ ( g − 1 ψ ( A )) =
q ϕ ◦ g − 1 ψ ( A ) =
q ϕ,ψ ( A ) = 0 .
As such, we have that q ϕ,ψ ≪
p ψ
and so the
D KL ( q ϕ,ψ ( z | x ) ∥ p ψ ( z )) is also defined.
Our next significant step is to show that
[Equation]
where σ ( g ψ ) denotes the sigma-algebra generated by the function g ψ .
To do this, let h : ( Z , G ) →
( R + , B ( R + )) be a measurable function s.t.
E p ( y ) (cid:104) q ϕ p (cid:12)(cid:12)(cid:12) σ ( g ψ ) (cid:105) =
h ◦ g ψ .
To show this, we will demonstrate that they lead to equivalent measures when integrated over any arbitrary set A ∈ G : (cid:90)
[Equation]
We denote the pushforward of a probability measure χ along a map g by χ ◦ g − where we have leveraged the definition of pushforward measures in (a & e); the absolute continuity of q ϕ w.r.t.
p in (b); the conditional expectation definition in (c); and the definition of h in (d).
By equating terms, we have that q ϕ ◦ g − 1 ψ /p ◦
g − 1 ψ = h , almost-surely with respect to q ϕ ◦ g − 1 ψ and thus that Eq.
(A.1) is verified.
Let us define f : x (cid:55)→ x log( x ) , which is strictly convex on [0 , ∞ ) (as it can be prolonged with f (0) = 0 ).
We have the following
[Equation]
where we leveraged the definition of the KL divergence in (a & j); the absolute continuity of q ϕ w.r.t.
p in (b & i); the definition of f in (c & h); the definition of the pushforward measure in (d); Eq.
(A.1) in (e); the conditional Jensen inequality in (f) and the law of total expectation in (g).
Note that this proof not only holds for the KL divergence, but for any f-divergences as they are defined as in (b) with f convex.
To prove Eq.
(4), we now need to show that line (f) above becomes an equality when g ψ is invertible.
As f is strictly convex, this happens if and only if q ϕ p =
E p ( y ) (cid:104) q ϕ p (cid:12)(cid:12) (cid:12) σ ( g ψ ) (cid:105) .
A sufficient condition for this to be true is for q ϕ p to be measurable w.r.t.
σ ( g ψ ) which is satisfied when g ψ : Y (cid:55)→ Z is invertible as σ ( g ψ ) ⊇ F , as required.
We have thus shown that the KL divergences are equal when using an invertible g ψ .
For the reconstruction term, we instead have
[Equation]
Eq.
(4) now follows from the fact that both the reconstruction and KL terms are equal.
A PPENDIX B H IERARCHICAL R EPRESENTATIONS
The isotropic Gaussian prior in standard VAEs assumes that representations are independent across dimensions (Kumar et al., 2018).
However, this assumption is often unrealistic (Belghazi et al., 2018; Mathieu et al., 2019b).
For example, in Fashion-MNIST, high-level features such as object category, may affect low-level features such as shape or height.
Separately extracting such global and local information can be beneficial for visualization and data manipulation (Zhao et al., 2017).
To try and capture this, we introduce an inductive bias that is tailored to model and learn hierarchical features.
We note here that our aim is not to try and provide a state-of-the-art hierarchical VAE approach, as a wide variety of highly–customized and powerful approaches are already well–established, but to show how easily the InteL-VAE framework can be used to induce hierarchical representations in a simple, lightweight, manner.
Mapping design Following existing ideas from hierarchical VAEs (Sønderby et al., 2016; Zhao et al., 2017), we propose a hierarchical mapping g ψ .
As shown in Fig.
B.1, the intermediary Gaussian variable y is first split into a set of N layers [ y 0 , y 1 , ..., y N ] .
The mapping z = g ψ ( y ) is then recursively defined as z i =
NN i ( z i − 1 , y i ) , where NN i is a neural network combining information from higher-level feature z
i − 1 and new information from y i .
As a result, we get a hierarchical encoding z
=
[ z 0 , z 1 , ..., z N ] , where high-level features influence low-level ones but not vice-versa.
This g ψ thus endows InteL-VAEs with hierarchical representations.
(a
[Equation]
of a hierarchical InteL-VAE.
The features are split into 5 levels, with each of (a)
[highest] to (e) [lowest] corresponding to an example feature from each.
We see that high-level features control more complex properties, such as class label or topological structure, while low-level features control simpler details, (e.g. (d) controls collar shape).
Experiments While conventional hierarchical
VAEs, e.g. (Sønderby et al., 2016; Zhao et al., 2017; Vahdat & Kautz, 2020), use hierarchies to try and improve generation quality, our usage is explicitly from the representation perspective, with our experiments set up accordingly.
Fig.
B.2 shows some hierarchical features learned by InteL-VAE on Fashion-MNIST .
We observe that high-level information such as categories have indeed been learned in the top-level features, while low-level features control more detailed aspects.
To provide more quantitative investigation, we also consider the CelebA dataset (Liu et al., 2015) and investigate performance on downstream tasks, comparing to vanilla-VAEs with different latent dimensions.
For this, we train a linear classifier to predict all 40 binary labels from the learned features for each method.
In order to eliminate the effect of latent dimensions, we compare InteL-VAE (with fixed latent dimension 128 ) and vanilla VAE with different latent dimensions ( 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 ).
We show experiment results on some labels as well as the average accuracy on all labels in Table B.1 and Fig.
B.3.
We first find that the optimal latent dimension increases with the number of data points for the vanilla-VAEs, but is always worse than the InteL-VAE.
Notably, the accuracy with InteL-VAE is quite robust, even as the number of data points gets dramatically low, indicating high data efficiency.
To the best of our knowledge, this is the first result showing that a hierarchical inductive bias in VAE is beneficial to feature quality.
Related work Hierarchical VAEs (Vahdat & Kautz, 2020;
Ranganath et al., 2016;
Sønderby et al., 2016; Klushyn et al.
; Zhao et al., 2017) seek to improve the fit and generation quality of VAEs by recursively correcting the generative distributions.
However, they require careful design of neural
[Table]
layers, and the hierarchical KL divergence makes training deep hierarchical VAEs unstable (Vahdat &
Kautz, 2020).
In comparison, InteL-VAE with hierarchical mappings is extremely easy to implement without causing any computational instabilities, while its aims also differ noticeably: our approach successfully learns hierarchical representations —something that is rarely mentioned in prior works.
[Figure]
A PPENDIX C F ULL M ETHOD AND E XPERIMENT D ETAILS
In this section, we first provide complete details of the mapping designs used for our different InteLVAE realizations along with some additional experiments.
We then provide other general information about datasets, network structures, and experiment settings to facilitate results reproduction.
C. M ULTIPLE - CONNECTIVITY
Mapping design Full details for this mapping were given in the main paper.
Fig.
C.1 provides a further illustration of the gluing process.
Additional resulting including the Vamp-VAE are given in Fig.
[Figure]
C. M ULTI - MODALITY
Mapping design In Sec.
6.2, we see the general idea of designing clustered mappings.
In this part, we delve into the details of mapping design as well as extending it to 1 dimensional and highdimensional cases.
For simplicity’s sake let us temporarily assume that the dimension of Y is 2 .
Our approach is based on splitting the original space into K equally sized sectors, where K is the number of clusters we wish to create, as shown in Fig.
5b.
For any point y , we can get its component (sector) index ci ( y ) as well as its distance from the sector boundary dis ( y ) .
By further defining the radius direction for the k -th sector (cf Fig. 5c) as
[Equation]
we can in turn define g ( y ) as:
[Equation]
where c 1 and c 2 are constants, which are set to 5 and 0.2 in our experiments.
we make sure g still continuous by keeping g ( y ) =
y on boundaries.
When dimension of Y is greater than 2 , we have more diverse choice for g .
When K is decomposable, i.e., K = (cid:81)
{+i K i , we can separately cut the plane expanded by Y 2 i and Y 
+}
i +1 into K i sectors by the
Eq.
(C.1).
As a result, Y is split into K =
(cid:81)
i k
i clusters.
When K = 2 , we find that g only changes the 1 -st dimension of Y , so it can be applied to cases where latent dimension is 1 .
Learnable proportions We can also make the mapping more flexible by learning rather than assigning the cluster proportions.
To do so, we keep a learnable value u i for each cluster and set the angle of the i -th sector as 2 π Softmax ( u ) i .
Things are simpler for the 1-dimensional case where we can uniformly translate y by a learnable bias b before splitting the space from the origin.
C. S PARSITY
Relationship to soft attention We note that our setup for the sparsity mapping shares some similarities with a soft attention layer (Bahdanau et al., 2014).
However, there are also some
[Figure]
important points of difference.
Firstly, soft attention aims to find the weights to blend features from different time steps (for sequential data) or different positions (for image data).
In contrast, the dimension selector (DS) selects which dimensions to activate or deactivate for the same latent vector.
Secondly, the weights of features are usually calculated by inner products of features for soft attention, while DS relies on a network to directly output the logits.
Sparsity regularizer Our sparsity regularizer term, L sp , is used to encourage our dimensionality selector network (DS) to produce sparse mappings.
It is defined using a mini-batch of samples { y i } M i =1 drawn during training as per (7).
During training, the first term of L sp decreases the number of activated dimensions for each sample, while the second term prevents the samples from all using the same set of activated dimensions, which would cause the model to degenerate to a vanilla VAE with a lower latent dimensionality.
We note that L sp alone is not expected to induce sparsity without also using the carefully constructed g ψ of the suggested InteL-VAE.
We confirm this empirically by performing an ablation study on
MNIST where we apply this regularization directly to a vanilla VAE.
{+We find that even when using very large values of γ > 30 . 
+}
we can only slightly increase the sparsity score ( 0 . 230 → 0 . 235 ).
{+Moreover, unlikely for the InteL-VAE, this substantially deteriorates generation quality, with the FID score raising to more than 80 . 
+}
at the same time.
Sparse metric We use the Hoyer extrinsic metric (Hurley & Rickard, 2009) to measure the sparsity of representations.
For a representation z ∈ R D ,
[Equation]
Here, following Mathieu et al. (2019b), we crucially first normalized each dimension d of z to have standard deviation 1 , ˆ z d
= z d /σ d , to ensure that we only measure sparsity that varies between data
[Figure]
[Table]
points (as is desired), rather than any tendency to uniformly ‘switch off’ certain latent dimensions (which is tangential to our aims).
In other words, this normalization is necessary to avoid giving high scores to representations whose length scales vary between dimensions, but which are not really sparse.
By averaging Hoyer ( z ) over all representations, we can get the sparse score of a method.
For the sparsest case, where each representation has a single activated dimension, the sparse score is 1 .
And when the representations get denser, ||
ˆ z
|| 2 get smaller compared with ||
ˆ z
|| 1 , leading to smaller sparse scores.
Reproduction of Sparse-VAE We tried two different code bases for Sparse-VAE (Tonolini et al., 2020).
The official code base 5 gives higher sparse scores for MNIST and FashionMNIST (though still lower than InteL-VAE), but is very unstable during training, with runs regularly failing after diverging and producing NaNs.
This issue gets even more severe on CelebA which occurs after only a few training steps, undermining our ability to train anything meaningful at all.
To account for this, we switched to the codebase 6 from De la Fuente & Aduviri (2019) that looked to replicate the results of the original paper.
We report the results from this code base because it solves the instability issue and achieves reasonable results on CelebA. Interestingly, though its generation quality is good on
MNIST and Fashion-MNIST, it fails to achieve a sparse score significantly higher than vanilla-VAE.
As the original paper does not provide any quantitative evaluation of the achieved sparsity, it is difficult to know if this behavior is expected.
We note though that the qualitative results shown in the paper appear to be substantially less sparse than those we show for the InteL-VAE, cf their Figurecompared to the top row of our Fig. 8.
In particular, their representation seems to mostly ‘switch off’ some latents entirely, rather than having diversity between datapoints that is needed to score well under the Hoyer metric.
[Table]
[Table]
E
[Table]
D ETAILS
Datasets Both synthetic and real datasets are used in this paper.
All synthetic datasets (sphere, square, star, and mixture of Gaussian) are generated by generators provided in our codes.
For real datasets, We load MNIST, Fashion-MNIST, and CelebA directly from Tensorflow (Abadi et al., 2015), and we resize images from CelebA to 64 x 64 following Hou et al.
For experiments with a specified number of training samples, we randomly select a subset of the training data.
We use the same random seed for each model in the same experiment and different random seeds when repeating experiments.
Model structure For low-dimensional data, the encoder and decoder are both simple multilayer perceptrons with 3 hidden layers (10-10-10) and ReLU (Glorot et al., 2011) activation.
For MNIST and Fashion-MNIST, we use the same encoder and decoder as Mathieu et al.
(2019b).
For CelebA, the structure of convolutional networks are shown in Table C.2.
Experiment settings Other hyperparameters are shown in Table C.1.
All experiments are run on a
GTX-1080-Ti GPU.