I NFORMATION M AXIMIZATION A UTO -E NCODING
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose the Information Maximization Autoencoder (IMAE), an information theoretic approach to simultaneously learn continuous and discrete representations in an unsupervised setting.
Unlike the Variational Autoencoder framework, IMAE starts from a stochastic encoder that seeks to map each input data to a hybrid discrete and continuous representation with the objective of maximizing the mutual information between the data and their representations.
A decoder is included to approximate the posterior distribution of the data given their representations, where a high ﬁdelity approximation can be achieved by leveraging the informative representations.
We show that the proposed objective is theoretically valid and provides a principled framework for understanding the tradeoffs regarding informativeness of each representation factor, disentanglement of representations, and decoding quality.
I NTRODUCTION
A central tenet for designing and learning a model for data is that the resulting representation should be compact yet informative.
Therefore, the goal of learning can be formulated as ﬁnding informative representations about the data under proper constraints.
Generative latent variable models are a popular approach to this problem, where a model parameterized by θ of the form p θ ( x ) = (cid:82) p θ ( x
| z ) p ( z ) d z is used to represent the relationship between the data x and the low dimensional latent variable z .
The model is optimized by ﬁtting the generative data distribution p θ ( x ) to the training data distribution (cid:98) p ( x ) , which involves maximizing the likelihood for θ .
Typically, this model is intractable even for moderately complicated functions p θ ( x | z ) with continuous z .
To remedy this issue, variational autoencoder (VAE) (Kingma and Welling, 2013; Rezende et al., 2014) proposes to maximize the evidence lower bound (ELBO) of the marginal likelihood objective.
However, as was initially pointed out in (Hoffman and Johnson, 2016), maximizing ELBO also penalizes the mutual information between data and their representations.
This in turn makes the representation learning even harder.
Many recent efforts have focused on resolving this problem by revising ELBO.
Generally speaking, these works fall into two lines.
One of them targets “disentangled representations” by encouraging the statistical independence between representation components (Higgins et al., 2016; Kim and Mnih, 2018; Gao et al., 2018; Chen et al., 2018; Esmaeili et al., 2018), while the other line of work seeks to control or encourage the mutual information between data and their representations (Mary Phuong, 2018; Burgess et al., 2018;
Alemi et al., 2017; Dupont, 2018; Zhao et al., 2017).
However, these approaches either result in an invalid lower bound for the
VAE objective or cannot avoid sacriﬁcing the mutual information.
Instead of building upon the generative latent variable model, we start with a stochastic encoder p θ ( z | x ) and aim at maximizing the mutual information between the data x and its representations z .
In this setting, a reconstruction or generating phase can be obtained as the variational inference of the true posterior p θ ( x | z ) .
By explicitly seeking for informative representations, the proposed model yields better decoding quality.
Moreover, we show that the information maximization objective naturally induces a balance between the informativeness of each latent factor and the statistical independence between them, which gives a more principled way to learn semantically meaningful representations without invalidating ELBO or removing individual terms from it.
Another contribution of this work is proposing a framework for simultaneously learning continuous and discrete representations for categorical data.
Categorical data are ubiquitous in real-world tasks, where using a hybrid discrete and continuous representation to capture both categorical information and continuous variation in data is more consistent with the natural generation process.
In this work, we focus on categorical data that are similar in nature, i.e., where different categories still share similar variations (features).
We seek to learn semantically meaningful discrete representations while maintaining disentanglement of the continuous representations that capture the variations shared across categories.
We show that, compared to the VAE based approaches, our proposed objective gives a more natural yet effective way for learning these hybrid representations.
R ELATED W ORK
Recently, there has been a surge of interest in learning interpretable representations.
Among them, β -VAE (Higgins et al., 2016) is a popular method for learning disentangled representations, which modiﬁes ELBO by increasing the penalty on the KL divergence between the variational posterior and the factorized prior.
However, by using large weight for the KL divergence term, β -VAE also penalizes the mutual information between the data and the latent representations more than a standard
VAE does, resulting in more severe under utilization of the latent representation space.
Several follow up works propose different approaches to address the limitations of β -VAE.
(Dupont, 2018;
Alemi et al., 2017; Burgess et al., 2018; Mary Phuong, 2018) propose to constrain the mutual information between the representations and the data by pushing its upper bound, i.e., the KL divergence term in ELBO, towards a progressively increased target value.
However, specifying and tuning this target value can itself be very challenging, which makes this method less practical.
Moreover, this extra constraint results in an invalid lower bound for the VAE objective.
Alternatively, (Zhao et al., 2017) drops the mutual information term in ELBO.
By pushing only the aggregated posterior towards a factorial prior, they implicitly encourage independence across the dimensions of latent representations without sacriﬁcing the informativeness of the representations.
However, simply removing the mutual information term also violates the lower bound of the VAE objective.
Another relevant line of work (Gao et al., 2018; Kim and Mnih, 2018; Chen et al., 2018; Esmaeili et al., 2018) seek to learn disentangled representations by explicitly encouraging statistical independence between latent factors.
They all propose to minimize the total correlation term of the latent representations, either augmented as an extra term to ELBO or obtained by reinterpreting or re-weighting the terms in the VAE objective, as a way to encourage statistical independence between the representation components.
In contrast, we show that our information maximization objective inherently contains the total correlation term while simultaneously seeking to maximize the informativeness of each representation factor.
In this paper, we introduce a different perspective to the growing body of the VAE based approaches for unsupervised representation learning.
Starting by seeking informative representations for the data, we follow a more intuitive way to maximize the mutual information between the data and the representations.
Moreover, we augment the continuous representation with a discrete one, which allows more ﬂexibilities to model real world data that are generated from different categories.
We invoke the information maximization principle (Linsker, 1988; Bell and Sejnowski, 1995) with proper constraints implied by the objective itself to avoid degenerate solutions.
The proposed objective gives a theoretically elegant yet effective way to learn semantically meaningful representations.
I NFORMATION M AXIMIZATION R EPRESENTATION L EARNING
Given data x ∈ R d , we consider learning a hybrid continuous-discrete representation, denoted respectively with variables z ∈ R K 1 and y ∈ { 1 , . .
, K 2 } , using a stochastic encoder parameterized by θ , i.e., p θ ( y , z | x ) .
We seek to learn compact yet semantically meaningful representations in the sense that they should be low dimensional but informative enough about the data.
A natural approach is to maximize the mutual information (Cover and Thomas, 2012)
I θ ( x ; y , z ) between the data and its representations under the constraint K 1 , K 2 (cid:28) d .
Here the mutual information between two random variables, e.g., x and z , is deﬁned as I θ ( x ; z ) =
H θ ( z ) − H θ
( z | x ) , where
H θ ( z ) = − E p θ ( z ) [log p θ ( z )] is the entropy of z and H θ
( z | x ) = − E p θ ( x , z ) [log p θ ( z | x )] is the conditional entropy of z given x .
The mutual information can be interpreted as the decrease in uncertainty of one random variable given another random variable.
In other words, it quantiﬁes how much information one random variable has about the other.
A probabilistic decoder q φ ( x | y , z ) is adopted to approximate the true posterior p θ ( x | y , z ) , which can be hard to estimate or even intractable.
The dissimilarity between them is optimized by minimizing the KL divergence D KL ( p θ ( x | y , z )
||
q φ ( x | y , z )) .
In summary, IMAE considers the following,
[Equation]
Given that H ( x ) is independent of the optimization procedure, we can show that optimizing (1) is equivalent to optimize the following 1 ,
[Equation]
We set β > 0 to balance between maximizing the informativeness of latent representations and maintaining the decoding quality.
The second term is often interpreted as the “reconstruction error” which can be optimized using the reparameterization tricks proposed by (Kingma and Welling, 2013) and (Jang et al., 2016) for continuous representation z and discrete representation y respectively.
Now we introduce proper method to optimize the ﬁrst term I θ ( x ; y , z ) in (2).
3. S IMULTANEOUSLY SEEKING INFORMATIVENESS AND DISENTANGLEMENT
We ﬁrst show that I θ ( x ; y , z ) inherently involves two keys terms that quantify the informativeness of each representation factor and the statistical dependence between these factors.
Assuming the conditional distribution of the representation ( y , z ) given x is factorial, we also assume the marginal distribution of y and z are independent, i.e., p θ ( y , z )
= p θ ( y ) p θ ( z ) , then
[Equation]
The ﬁrst two terms of the RHS quantify how much information each latent factor, i.e., y or z k , carry about the data.
The last term is known as the total correlation of z (Watanabe, 1960), which quantiﬁes the statistical independence between the continuous latent factors and achieves the minimum if and only if they are independent of each other.
As is implied by (3), maximizing I θ ( x ; y , z ) can be conducted by maximizing informativeness of each latent factor while simultaneously promoting statistical independence between the continuous factors.
Various Monte Carlo based sampling strategies have been proposed to optimize the total correlation term (Chen et al., 2018; Esmaeili et al., 2018); in this work we follow this line (see
Appendix B).
Next we proceed by constructing tractable approximations for I θ ( x ; z k ) and I θ ( x ; y ) respectively.
3. I NFORMATIVE CONTINUOUS REPRESENTATIONS
Without any constraints, the mutual information I θ ( x ; z k ) between a continuous latent factor and data can be trivially maximized by severely fragmenting the latent space.
To be more precise, consider the following proposition.
While similar results have likely been established in the information theory literature, we include this proposition to motivate our objective design.
Proposition 1.
Suppose the conditional distribution p θ ( z | x ) is a factorial Gaussian distribution with mean µ ( x ) and covariance Σ( x ) .
Let σ ( x ) ∈ R K 1 denote the diagonal entries of Σ( x ) , then
I
[Equation]
The equality in (4) is attained if and only if z k is Gaussian distributed, given which we have
[Equation]
Note here both µ k ( x ) and σ k ( x ) are random variables.
The above result implies that z k is more informative about x if it has less uncertainty given x yet captures more variance in data, i.e., σ k ( x ) is small while µ k ( x ) disperses within a large space.
However, this can result in discontinuity of z k , where in the extreme case each data sample is associated with a delta distribution in the latent space.
In light of this, we can make what we described above more precise.
A vanishing variance of the conditional distribution p ( z k | x ) leads to a plain autoencoder that maps each data sample to a deterministic latent point, which can fragment the latent space in a way that each data sample corresponds with a delta distribution in the latent space p θ ( z k | x ( i ) )
=
δ ( z ( i ) k ) .
On the other hand, Propositionalso implies that controlling the variance σ k ( x ) to be ﬁnite, I θ ( x ; z k ) will be maximized by pushing µ k ( x ) towards two extremes ( ±∞ ).
To remedy this issue while achieving the upper bound, a natural resolution is to squeeze z k within the domain of a Gaussian distribution with ﬁnite mean and variance.
By doing so, we can avoid the degenerate solution while achieving a more reasonable trade-off between enlarging the spread of µ k ( x ) and maintaining the continuity of z .
Therefore, we consider the following as the surrogate for maximizing I
[Equation]
x
[Equation]
z k ) ,
[Equation]
Here r ( z k ) are i.i.d scaled normal distribution with ﬁnite variance.
That is, we push each p θ ( z k ) towards a Gaussian distribution r ( z k ) by minimizing the KL divergence between them.
3. I NFORMATIVE DISCRETE REPRESENTATIONS
Unlike the continuous representation, the mutual information I θ ( x ; y ) between a discrete representation and data can be well approximated, given the fact that the cardinality of the space of y is typically low.
To be more speciﬁc, given N i.i.d samples { x n } Nn =1 of the data, the empirical estimation of I θ ( x ; y ) under the conditional distribution p θ ( y | x n ) follows as
[Equation]
As shown in Proposition 2, with a suitably large batch of samples, the empirical mutual information (cid:98) I θ ( x ; y ) is a good approximation to I θ ( x ; y ) .
This enables us to optimize I θ ( x ; y ) in a theoretically justiﬁable way that is amenable to stochastic gradient descent with minibatches of data.
Proposition 2.
Let y be a discrete random variable that belongs to some categorical class C .
Assume the marginal probabilities of the true and the predicted labels are bounded below, i.e. p θ ( y ) ,
[Equation]
p θ
[Equation]
y ) ∈ [1 / ( CK 2 ) , 1] for all y ∈ C with some constant C > 1 .
Then for any δ ∈ (0 , 1) ,
[Equation]
Here N denotes the number of samples used to establish (cid:98) I θ ( x ; y ) according to Eq (7).
Therefore, to maximize the mutual information I θ ( x ; y ) , we consider the following:
[Equation]
Maximizing the the mutual information I θ ( x ; y ) provides a natural way to learn discrete categorical representations.
To see this, notice that I θ ( x ; y ) contains two fundamental quantities, the category balance term H θ ( y ) and the category separation term H
θ ( y | x ) .
In other words, maximizing I θ ( x ; y ) trades off uniformly assigning data over categories and seeking highly conﬁdent categorical identity for each sample x .
The maximum is achieved if p θ ( y | x ) is deterministic while the marginal distribution p θ ( y ) is uniform, that is H θ ( y | x ) = 0 and H θ ( y )
=
log K 2 .
Overall Objective As a summary of (3) (6) and (9), our overall objective is
[Equation]
The ﬁrst three terms associate with our information maximization objective, while the last one aims at better approximation of the posterior p θ ( x | y , z ) .
A better balance between these two targets can be achieved by weighting them differently.
One the other hand, the informativeness of each latent factor can be optimized through L θ ( z ) and L θ ( y ) , while statistically independent latent continuous factors can be promoted by minimizing the total correlation term D KL (cid:104)
p ( z )
||
Π K 1 k =1 p ( z k ) (cid:105) .
Therefore, trade-offs can be formalized regarding the informativeness of each latent factor, disen- tanglement of the representation, and better decoding quality.
This motivates us to consider the following objective, let β, γ > 0 ,
[Equation]
E XPERIMENTAL R ESULTS
We compare IMAE against various VAE based approaches that are summarized in Figure 1.
We would like to demonstrate that IMAE can (i) successfully learn a hybrid of continuous and discrete representations, with y matching the intrinsic categorical information y true well and z capturing the disentangled feature information shared across categories; (ii) outperform the VAE based models by achieving a better trade-off between representation interpretability and decoding quality.
We choose the priors r ( z ) and r ( y ) to be the isotropic Gaussian distribution and uniform distribution respectively.
Detailed experimental settings are provided in Appendix G
[Equation]
[Table]
[Equation]
[Table]
[Figure]
4. I NFORMATIVE REPRESENTATIONS YIELD BETTER INTERPRETABILITY
We ﬁrst qualitatively demonstrate that informative representations can yield better interpretability.
For the continuous representation, Figure 2 validates Proposition 1 by showing that, with roughly same amount of variance for each latent variable z k , those achieving high mutual information with the data have mean values µ k ( x ) of the conditional probability p ( z k | x ) disperse across data samples and variances σ k ( x ) decrease to small values for all data samples.
As a qualitative evaluation, we traverse latent dimensions corresponding with different levels of I ( x , z k ) .
As seen in Figure 2(b)-(d), informative variables in the continuous representation have uncovered intuitive continuous factors of the variation in the data, while the factor z 8 has no mutual information with the data and shows no variation.
We observe the same phenomenon for the discrete representation y in Figure 2(e)&(f), which were obtained with two different values of β and γ , where the more informative one discovers matches the natural labels better.
This provides further evidence for that interpretable latent factors can be attained by maximizing the mutual information between the representations and the data.
[Figure]
continuous representations z .
The rows are conditioned on the discrete representations y learnt by IMAE, and the initial value of z for each row is obtained by feeding the encoder with randomly selected data corresponds with y .
We then manipulate each selected z k within [ − 2 , 2] while keeping all other dimensions ﬁxed.
(e) & (f) Discrete representations learnt by IMAE with different β values.
We set γ = 2 β for IMAE.
For each β , we run each method over 10 random initializations.
Q UANTITATIVE COMPARISONS
In this section, we perform quantitative evaluations on MNIST (LeCun and Cortes, 2010), Fashion
MNIST (Xiao et al., 2017) and dSprites (Matthey et al., 2017).
We show that IMAE achieves better interpretability vs. decoding quality trade-off.
Unsupervised learning of discrete latent factor Before we present our main results, we ﬁrst describe an assumption that we make on the discrete representations.
For the discrete representation, a reasonable assumption is that the conditional distribution p ( y | x ) should be locally smooth so that the data samples that are close on their manifold should have high probability of being assigned to the same category (Agakov, 2005).
This assumption is crucial for using neural networks to learn discrete representations, since it’s easy for a high capacity model to learn a non-smooth function p ( y | x ) that can abruptly change its predictions without guaranteeing similar data samples will be mapped to similar y .
To remedy this issue, we adopt the virtual adversarial training (VAT) trick proposed by (Miyato et al., 2016) and augment L θ ( y ) as follows:
[Equation]
The second term of RHS regularizes p θ ( y | x ) to be consistent within the (cid:15) norm ball of each data sample so as to maintain the local smoothness of the prediction model.
For fair comparison, we augment all four methods with VAT.
As demonstrated in Appendix D, using VAT is essential for all of them except β -VAE to learn interpretable discrete representations.
4.2. MNIST AND F ASHION MNIST
We start by evaluating different methods on MNIST and Fashion MNIST, for which we train over a range of β values (we set γ = 2 β for IMAE).
Discrete representations For the discrete representations, by simply pushing the conditional distribution p ( y | x ) towards the uniform distribution r ( y ) , β -VAE sacriﬁces the mutual information
I ( x ; y ) and hence struggles in learning interpretable discrete representation even with VAT.
As a comparison, InfoVAE performs much better by dropping I ( x ; y ) from ELBO.
For data that are distinctive enough between categories (MNIST), with large β values InfoVAE performs well by uniformly distributing the whole data over categories through minimizing D KL ( p ( y )
||
r ( y )) while simultaneously encouraging local smoothness with VAT.
However, InfoVAE struggles with less distinctive data (Fashion-MNIST), where it cannot give fairly conﬁdent category separation by only
[Figure]
requiring local smoothness.
In contrast, IMAE achieves much better performance by explicitly encouraging conﬁdent category separation via minimizing the conditional entropy H ( y | x ) , while using
VAT to maintain local smoothness so as to prevent overﬁtting of neural network.
Although JointVAE performs much better than β -VAE by pushing the upper bound of I ( x ; y ) towards a progressively increasing target value C y , we found it can easily get stuck at some bad local optima where I ( x ; y ) is comparatively large while the accuracy is poor.
A heuristic is that once JointVAE enters the local region of a local optima, progressively increasing C y only induces oscillation within that region.
Informativeness, interpretability and decoding quality As illustrated in Figure 1, by using large β values, β -VAE sacriﬁces more mutual information between the data and its representations, which in turn (see Figure 3) results in less informative representations followed by poor decoding quality.
In contrast, the other three methods can remedy this issue to different degrees, and hence attains better trade-off regarding informativeness of latent representations and decoding quality.
Compared to JointVAE and InfoVAE, IMAE is more capable of learning discrete presentations over a wide range of β, γ values, which implies less overlap between the manifolds of different categories is induced.
As a result, IMAE is expected to yield better decoding quality for each category.
Although
InfoVAE and JointVAE can also learn comparatively good discrete representations when using large and small β values respectively, the corresponding results of these two regions associate with either poor decoding quality or much lower disentanglement score (see section 4.2.2).
In contrast, IMAE consistently performs well with different hyperparameters, especially in the region of interest where the decoding quality as well as the informativeness of latent representations are good enough.
4.2. 2D S HAPES
In this section, we quantitatively evaluate the disentanglement capability of IMAE on dSprites where the ground truth factors of both continuous and discrete representaions are available.
We use the disentanglement metric proposed by (Chen et al., 2018), which is deﬁned in terms of the gap between the top two empirical mutual information of each latent representation factor and a ground truth factor.
The disentanglement score is deﬁned as the weighted average of the gaps.
A high disentanglement score implies that each ground truth factor associates with one single representation factor that is more informative than the others, i.e., the learnt representation factors are more disentangled.
Figure 5 shows that, with large β values, β -VAE penalizes the mutual information too much and this degrades the usefulness of representations.
while all other three methods achieve higher disentanglement score with better decoding quality.
For JointVAE, higher β values push the upper bound of mutual information converges to the preﬁxed target value, it therefore can maintain more mutual
[Figure]
information between the data and the whole latent representations and give better decoding quality.
However, the disentanglement quality is poor in this region, which implies that simply restricting the overall capacity of the latent representations is not enough for learning disentangled representations.
While InfoVAE yields comparatively better disentanglement score by pushing the marginal joint distribution of the representations towards a factorial distribution harder with large values of β , the associated decoding quality and informativeness of latent representations are both poor.
In contrast, IMAE is capable of achieving better trade-off between the disentanglement score and the decoding quality in the region of interest where the decoding quality as well as the informativeness are fairly good.
We attribute this to the effect of explicitly seeking for statistically independent latent factors by minimizing the total correlation term in our objective.
C ONCLUSION
We have proposed IMAE, a novel approach for simultaneously learning the categorical information of data while uncovering latent continuous features shared across categories.
Different from VAE,
IMAE starts with a stochastic encoder that seeks to maximize the mutual information between data and their representations, where a decoder is used to approximate the true posterior distribution of the data given the representations.
This model targets at informative representations directly, which in turn naturally yields an objective that is capable of simultaneously inducing semantically meaningful representations and maintaining good decoding quality, which is further demonstrated by the numerical results.
Unsupervised joint learning of disentangled continuous and discrete representations is a challenging problem due to the lack of prior for semantic awareness and other inherent difﬁculties that arise in learning discrete representations.
This work takes a step towards achieving this goal.
A limitation of our model is that it pursues disentanglement by assuming or trying to encourage independent scalar latent factors, which may not always be sufﬁcient for representing the real data.
For example, data may exhibit category speciﬁc variation, or a subset of latent factors might be correlated.
This motivates us to explore more structured disentangled representations; one possible direction is to encourage group independence.
We leave this for future work.
A P ROOF OF S ECTION
Balance between posterior inference ﬁdelity and information maximization Notice that we can rewrite the mutual information between the data x and its representations as the following,
[Equation]
It then follows that,
[Equation]
Since H ( x ) is independent of the optimization procedure
[Equation]
we have the following,
[Equation]
where β trade-off the informativeness of the latent representation and generation ﬁdelity.
Decomposition of I θ ( x ; y , z ) Let b =
( z , y ) denote the joint random variable consisting of the continuous random variable b and discrete random variable y .
Note that I θ ( x
; y , z ) =
I θ ( x ; b ) can be written as:
[Equation]
The second term in Eq (15) has the form:
[Equation]
where ϑ 1 follows by the assumption that p θ ( b | x ) is factorial
[Equation]
For the ﬁrst term in Eq (15)
[Equation]
we have:
[Equation]
Substituting Eqs (16) & (17) into Eq (15) yields the result:
[Equation]
Since y and z are assumed to be marginally independent, i.e., p θ ( y ; z )
= p θ ( y ) p θ ( z ) , then
I θ
[Equation]
x
[Equation]
Proof of proposition
Proof.
We start with computing the expectation of z k :
[Equation]
Then the variance of z k followed as:
[Equation]
Note that
[Equation]
for which we have the following,
[Equation]
For the entropy of z k , we leverage the fact that H θ ( z k ) is upper bounded by the entropy of a
Gaussian distributed random variable with the same mean and variance, that is
[Equation]
Substituting Eqs (23) & (24) into Eq (22) completes the proof.
Proof of proposition
Proof.
Let (cid:98) p θ ( y )
= 1 N (cid:80) Nn
=1 p θ ( y | x n ) denote the Monte Carlo estimator of the true probability p θ ( y )
=
(cid:82) X p ( x ) p θ ( y | x ) d x = E x [ p θ ( y | x )] .
Note that p θ ( y | x ) ∈ [0 , 1] for all x ∈ X , then applying the Hoeffding’s inequality for bounded random variables [Theorem 2.2.6, (Vershynin, 2018)] yields,
[Equation]
Let δ (cid:48)
= 2 exp (cid:0) − 2 Nt 2 (cid:1) , it then follows,
[Equation]
[Equation]
Given Eq (26), we ﬁrst establish the concentration results of the entropy H (cid:98) p θ ( y ) with respect to the empirical distribution (cid:98) p θ ( y ) .
Assume For all y ∈ C , we have p θ ( y ) , (cid:98) p θ ( y ) bounded below by 1 / ( CK 2 ) for some ﬁxed constant C > 1 .
This assumption is practical since the distributions of true data and predicted data are approximately uniform and therefore p θ ( y ) , (cid:98) p θ ( y )
≈ 1 /K 2 for all y ∈ C .
Consider the function t log t , with derivative 1 + log t ∈ [1 − log CK 2 , 1] for t ∈ [1 / ( CK 2 ) , 1] ,
[Equation]
Summing over C gives
[Equation]
Let δ
= K 2 δ (cid:48) , then Eq (26) together with Eq (28) yield the following,
[Equation]
Next we are going to bound the divergence between (cid:98) H
θ ( y | x ) and H θ ( y | x ) which are deﬁned as,
[Equation]
Note that h log h ∈ [ − 1 /e, 0] for all h ∈ [0 , 1] , then again applying [Theorem 2.2.6, (Vershynin, 2018)] yields,
[Equation]
Following the similar arguments as before, let δ (cid:48)
= 2 exp (cid:0) − 2 t 2 e 2 N (cid:1) , then
P
[Equation]
N
[Equation] [Equation]
Now let δ = K 2 δ (cid:48) , then applying the union bound we have |
[Equation]
| x
[Equation]
− H
[Equation]
( y | x
[Equation]
| ≤
[Equation]
(32) hold with probability 1 − δ .
Conclude from Eqs (29) & (32), we have (cid:12)(cid:12)(cid:12) I θ ( x ; y ) − (cid:98)
I θ ( x ; y ) (cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12) H θ ( y ) − (cid:98) H
θ ( y ) (cid:12)(cid:12)(cid:12)
+ (cid:12)(cid:12)(cid:12) H
θ ( y | x ) − (cid:98) H θ ( y | x ) (cid:12)(cid:12)(cid:12) =
[Equation] [Equation]
(cid:114)
[Equation]
B A PPROXIMATION OF THE MARGINAL DISTRIBUTION
Computing the marginal distributions of the continuous representations z and z k requires the entire dataset, e.g., p θ ( z ) =
(cid:82) X p θ ( z , x ) d x ≈ 1 N (cid:80) Ni =1 p θ ( z | x ( i ) ) .
To scale up our method to large datasets, we propose to estimate based on the minibatch data, e.g., p θ ( z ) ≈ 1 B (cid:80) Bi
=1 p θ ( z | x ( i ) ) .
Now consider the entropy H ( z ) of z , which we approximate in the following way,
[Equation]
We estimate the integral of z by sampling z ∼ p θ ( z | x i ) and perform the Monte Carlo approximation.
Although we minimize the unbiased estimator of the lower bound of the KL divergence, the term inside the logarithm is a summation of probability densities of Gaussians.
In particular, we record the distribution of the variances output by our encoder and observe that the mean of the variances of the Gaussians is bounded between 0.2 and 2, which implies that the values of probability densities do not range in a large scale.
Since logarithm is locally afﬁne, we argue that our bound in (34) is tight.
Other quantities involved in our objective function (10) are estimated in a similar fashion.
C C ONNECTIONS TO VAE
In VAE, they assume a generative model speciﬁed by a stochastic decoder p θ ( x | z ) , taking the continuous representation as an example, and seek an encoder q φ ( z | x ) as a variational approximation of the true posterior p θ ( z | x ) .
The model is ﬁtted by maximizing the evidence lower bound (ELBO) of the marginal likelihood,
[Equation]
Here the KL divergence term can be further decomposed as (Hoffman and Johnson, 2016),
[Equation]
That is, minimizing the KL divergence also penalizes the mutual information I θ ( x ; z ) , thus reduces the amount of information z has about x .
This can make the inference task q φ ( z | x ) hard and lead to poor reconstructions of x as well.
Many recent efforts have been focused on resolving this problem by revising ELBO.
Although approaches differ, it can be summarized as either dropping the mutual information term in Eq (36), or encouraging statistical independence across the dimensions of z by increasing the penalty on the total correlation term extracted from the KL divergence
D KL ( q φ ( z ) ||
r ( z )) with respect to q φ ( z ) .
However, these approaches either result in an invalid lower bound for the VAE objective, or cannot avoid minimizing the mutual information I θ ( x ; z ) between the representation and the data.
In contrast, IMAE starts with a stochastic encoder p θ ( z | x ) and aims at maximizing the mutual information between the data x and the representations z from the very beginning.
By following the constraints which are naturally implied by the objective in order to avoid degenerated solutions,
IMAE targets at both informative and statistical independent representations.
On the other hand, in
IMAE the decoder q φ ( x | z ) serves as a variational approximation to the true posterior p θ ( x | z ) .
As we will show in Section 4, being able to learn more interpretable representations allows IMAE to reconstruct and generate data with better quality.
[Figure]
D VAT STABILIZES THE LEARNING OF CATEGORICAL REPRESENTATIONS
E D ISENTANGLEMENT QUALITY WITH RESPECT TO BOTH CONTINUOUS AND DISCRETE REPRESENTATIONS ON 2D SHAPES F M ORE RESULTS ON J OINT VAE
See ﬁgure 8.
G E XPERIMENTAL SETTINGS
[Table] [Table]
Training procedure:
• MNIST & Fashion MNIST:
We use momentum to train all models.
The initial learning rate is set as 1 e -3, and we decay the learning rate by 0 . 98 every epoch.
• dSprites: We use Adam to train all models.
The learning rate is set as 1 e -3.