Original: ISA-VAE: I NDEPENDENT S UBSPACE A NALYSIS WITH V ARIATIONAL A UTOENCODERS
Revised: ISA-VAE: I NDEPENDENT S UBSPACE A NALYSIS WITH V ARIATIONAL A UTOENCODERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We provide a natural approach to encourage interpretable representations  in variational autoencoders. Building on previous work on Independent Component
Revised: Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way. These methods have focussed largely on modifying the variational cost function to achieve this goal.

Original: Analysis (ICA) and Independent Subspace Analysis (ISA) we show that simply choosing an appropriate prior for the latent distribution breaks the degeneracy of the standard prior and leads to  a interpretable laten representations.
Revised: However, we show that methods like β -VAE simplify the tendency of variational inference to underﬁt causing pathological overpruning and over-orthogonalization of learned components. In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered. Speciﬁcally, the standard

Original: The proposed family of prior distributions leads to a factorized representation and its rotational asymmetry allows interpretability of individual latent dimensions.
Revised: VAE probabilistic model is unidentiﬁable: the likelihood of the parameters is invariant under rotations of the latent space. This means there is no pressure to identify each true factor of variation with a latent variable. We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry.

Original: In our experiments we show that the commonly used higher weight (e g. β -VAE) of the 
Revised: Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modiﬁed cost functions like β -VAE and

Original: Kullback-Leibler divergence term in the evidence lower bound (ELBO) ampliﬁes two biases of variational inference: over-pruning and over-orthogonalization.
Revised: TCVAE between reconstruction loss and disentanglement.

Original: Extensive quantitative experiments demonstrate the performance of the proposed approach for disentanglement in variational autoencoders
Revised: The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality signiﬁcantly over the state of the art

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently there has been an increased interest in unsupervised learning of disentangled representations.
Revised: Recently there has been an increased interest in unsupervised learning of disentangled representations.

Original: The term disentangled usually describes two main objectives: First,  afactorized latent representation, and second, interpretability of the latent factors. Schmidhuber (1992); Ridgeway (2016); Achille & Soatto (2017)
Revised: The term disentangled usually describes two main objectives: First, to identify each true factor of variation with a latent variable, and second, interpretability of these latent factors (Schmidhuber, 1992; Ridgeway, 2016; Achille & Soatto, 2017).

Original: Most of this recent work is inspired by the β -VAE concept introduced in Higgins et al.
Revised: Most of this recent work is inspired by the β -VAE concept introduced in Higgins et al.

Original: (2016), which proposes to re-weight different parts of the evidence lower bound (ELBO) objective.
Revised: (2016), which proposes to re-weight the terms in the evidence lower bound (ELBO) objective.

Original: In Higgins et al.
Revised: In Higgins et al.

Original: a higher weight for the Kullback-Leibler divergence (KL) term is proposed, and putative mechanistic explanations for the effects of this modiﬁcation are studied in Burgess et al.
Revised: a higher weight for the Kullback-Leibler divergence (KL) between approximate posterior and prior is proposed, and putative mechanistic explanations for the effects of this modiﬁcation are studied in Burgess et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: An alternative decomposition of the ELBO leads to the recent variant of β -VAE called β -TCVAE Chen et al.
Revised: An alternative decomposition of the ELBO leads to the recent variant of β -VAE called βTCVAE

Original: (2018), which shows the highest scores on recent disentanglement benchmarks.
Revised: (Chen et al., 2018), which shows the highest scores on recent disentanglement benchmarks.

Original: These modiﬁcations of the evidence lower bound however lead to a tradeoff between disentanglement and reconstruction loss and therefore the quality of the learned model.
Revised: These modiﬁcations of the evidence lower bound however lead to a trade-off between disentanglement and reconstruction loss and therefore the quality of the learned model.

Original: This tradeoff is direcly encoded in the modiﬁed objective: by increasing the β -weight of the KL-term, the relative weight of the reconstruction loss term is more and more decreased.
Revised: This trade-off is directly encoded in the modiﬁed objective: by increasing the β -weight of the KL-term, the relative weight of the reconstruction loss term is more and more decreased.

Original: Therefore, optimization of the modiﬁed
Revised: Therefore, optimization of the modiﬁed

Original: ELBO will lead to latent encodings which have a lower KL-divergence from the prior, but at the same time lead to a higher reconstruction loss.
Revised: ELBO will lead to latent encodings which have a lower KL-divergence from the prior, but at the same time lead to a higher reconstruction loss.

Original: Furthermore, we discuss in section 2.4 that using a higher weight for the KL-term ampliﬁes existing biases of variational inference.
Revised: Furthermore, we discuss in section 2.4 that using a higher weight for the KL-term ampliﬁes existing biases of variational inference, potentially to a catastrophic extent.

Original: There is a foundational contradiction in many approaches to disentangling deep generative models (DGMs): the standard model employed is not identiﬁable as it employs a standard normal prior which then undergoes a linear transformation.
Revised: There is a foundational contradiction in many approaches to disentangling deep generative models (DGMs): the standard model employed is not identiﬁable as it employs a standard normal prior which then undergoes a linear transformation.

Original: Any rotation of the latent space can be absorbed into the linear transform and is therefore statistically indistinguishable.
Revised: Any rotation of the latent space can be absorbed into the linear transform and is therefore statistically indistinguishable.

Original: If interpretability is desired, the modelling choices are setting us up to fail.
Revised: If interpretability is desired, the modelling choices are setting us up to fail.

Original: We make the following contributions:
Revised: We make the following contributions:

Original: • We show that the current state of the art approaches employ a tradeoff between reconstruction loss and disentanglement of the latent representation.
Revised: • We show that the current state of the art approaches employ a trade-off between reconstruction loss and disentanglement of the latent representation.

Original: • In section 2.3 we show that variational inference techniques are  biased towards estimating an over-orthogonal components andtowards over-pruning the variance of the latent distribution.
Revised: • In section 2.3 we show that variational inference techniques are biased: the estimated components are biased towards having orthogonal effects on the data and the number of components is underestimated.

Original: • We provide a novel description of the origin of disentanglement in β -VAE and demonstrate in section 2.4 that increasing the weight of the KL term increases the over-pruning bias of variational inference.
Revised: • We provide a novel description of the origin of disentanglement in β -VAE and demonstrate in section 2.4 that increasing the weight of the KL term increases the over-pruning bias of variational inference.

Original: • To mitigate these drawbacks of existing approaches, we propose a family of rotationally asymmetric distributions for the latent prior, which prevents the  ambiguity of the latent encoding for arbitrary rotations of the latent space.
Revised: • To mitigate these drawbacks of existing approaches, we propose a family of rotationally asymmetric distributions for the latent prior, which removes the rotational ambiguity from the model.

Original: This approach resembles independent component analysis (ICA) for variational autoencoders.
Revised: This approach resembles independent component analysis (ICA) for variational autoencoders.

Original: • We propose to use a prior which allows a decomposition of the latent space using independent subspace analysis (ISA) and demonstrate that this prior leads to disentangled representations even for the unmodiﬁed ELBO objective.
Revised: • We propose to use a prior which allows a decomposition of the latent space using independent subspace analysis (ISA) and demonstrate that this prior leads to disentangled representations even for the unmodiﬁed ELBO objective.

Original: This removes the tradeoff between disentanglement and reconstruction loss of existing approaches.
Revised: This removes the trade-off between disentanglement and reconstruction loss of existing approaches.

Original: • An even higher disentanglement of the latent space can be achieved by incorporating the proposed prior distribution into the existing approaches β -VAE and β -TCVAE, thus exploiting their orthogonalization bias.
Revised: • An even higher disentanglement of the latent space can be achieved by incorporating the proposed prior distribution into the existing approaches β -VAE and β -TCVAE.

Original: Because the prior distribution already favours a disentangled representation,a high disentanglement score with at the same time low reconstruction loss can be achieved.
Revised: Since the prior distribution already favours a disentangled representation, the new method dominates previous in terms of the trade-off between disentanglement and model quality.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We brieﬂy discuss previous work on variational inference in deep generative models and two modiﬁcations of the learning objective that have been proposed to learn a disentangled representation.
Revised: We brieﬂy discuss previous work on variational inference in deep generative models and two modiﬁcations of the learning objective that have been proposed to learn a disentangled representation.

Original: We discuss characteristic biases of variational inference and how the modiﬁcations of the learning objective actually accentuate these biases.
Revised: We discuss characteristic biases of variational inference and how the modiﬁcations of the learning objective actually accentuate these biases.

Original: 2. D ISENTANGLED R EPRESENTATION L EARNING
Revised: 2. D ISENTANGLED R EPRESENTATION L EARNING

Original: Variational Autoencoder Kingma & Welling (2014) introduce a latent variable model that combines a generative model, the decoder, with an inference network, the encoder.
Revised: Variational Autoencoder Kingma & Welling (2014) introduce a latent variable model that combines a generative model, the decoder, with an inference network, the encoder.

Original: Training is performed by optimizing the evidence lower bound (ELBO) averaged over the empirical distribution:
Revised: Training is performed by optimizing the evidence lower bound (ELBO) averaged over the empirical distribution:

Original: [Equation]
Revised: [Equation]

Original: 1) where the decoder p θ ( x | z ) and encoder q θ ( z | x ) are deep learning models with parameters θ and each z l is sampled from  z l ∼ q θ ( z | x )  .
Revised: where the decoder p θ ( x | z ) is a deep learning model with parameters θ and each z l is sampled from the encoder z l ∼ q φ ( z | x ) with variational parameters φ .

Original: When choosing appropriate families of distributions, gradients through the samples z l can be estimated using the reparameterization trick .
Revised: When choosing appropriate families of distributions, gradients through the samples z l can be estimated using the reparameterization trick .

Original: The approximate posterior q θ ( z | x ) is usually modelled as a multivariate Gaussian with diagonal covariance matrix and the prior p ( z ) is typically the standard normal distribution.
Revised: The approximate posterior q φ ( z | x ) is usually modelled as a multivariate Gaussian with diagonal covariance matrix and the prior p ( z ) is typically the standard normal distribution.

Original: β -VAE Higgins et al.
Revised: β -VAE Higgins et al.

Original: (2016) propose to modify the evidence lower bound objective and penalize the KL-divergence of the ELBO:
Revised: (2016) propose to modify the evidence lower bound objective and penalize the KL-divergence of the ELBO:

Original: [Equation]
Revised: [Equation]

Original: where β > 1 is a free parameter that should encourage a disentangled representation.
Revised: where β > 1 is a free parameter that should encourage a disentangled representation.

Original: In Burgess et al.
Revised: In Burgess et al.

Original: (2017) the authors provide further thoughts on the mechanism that leads to these disentangled representations.
Revised: (2017) the authors provide further thoughts on the mechanism that leads to these disentangled representations.

Original: However we will show in the following that this parameter introduces a trade-off between reconstruction loss and disentanglement.
Revised: However we will show in the following that this parameter introduces a trade-off between reconstruction loss and disentanglement.

Original: Furthermore, we show in section 2.4 that this parameter ampliﬁes biases of variational inference towards orthogonalization and pruning.
Revised: Furthermore, we show in section 2.4 that this parameter ampliﬁes biases of variational inference towards orthogonalization and pruning.

Original: β -TCVAE
Revised: β -TCVAE

Original: In Chen et al.
Revised: In Chen et al.

Original: the authors propose an alternative decomposition of the ELBO, tjat leads to the recent variant of β -VAE called β -TCVAE.
Revised: the authors propose an alternative decomposition of the ELBO, that leads to the recent variant of β -VAE called β -TCVAE.

Original: They demonstrate that β -TCVAE allows to learn representations with higher MIG score than β -VAEHiggins et al.
Revised: They demonstrate that β -TCVAE allows to learn representations with higher MIG score than β -VAE (Higgins et al., 2016),

Original: (2016), InfoGAN Chen et al. (2016) and FactorVAE Kim & Mnih (2018).
Revised: InfoGAN (Chen et al., 2016) and FactorVAE (Kim & Mnih, 2018).

Original: The authors propose to decompose the KL-term in the ELBO objective into three parts and to weight them independently:
Revised: The authors propose to decompose the KL-term in the ELBO objective into three parts and to weight them independently:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is the index-code mutual information, the second term is the total correlation and the third term the dimension-wise KL-divergence.
Revised: The ﬁrst term is the index-code mutual information, the second term is the total correlation and the third term the dimension-wise KL-divergence.

Original: Because the index-code mutual information can be viewed as an estimator for the mutual information between p  ( x ) and q  ( z ) , the authors propose to exclude this term when reweighting the KL-term with the β weight.
Revised: Because the index-code mutual information can be viewed as an estimator for the mutual information between p θ ( x ) and q φ ( z ) , the authors propose to exclude this term when reweighting the KL-term with the β weight.

Original: In addition to the improved objective, the authors propose a quantitative evaluation score for disentanglement, the mutual information gap (MIG).
Revised: In addition to the improved objective, the authors propose a quantitative evaluation score for disentanglement, the mutual information gap (MIG).

Original: They propose to ﬁrst estimate the mutual information between a latent factor and an underlying generative factor of the dataset.
Revised: They propose to ﬁrst estimate the mutual information between a latent factor and an underlying generative factor of the dataset.

Original: The mutual information gap is then deﬁned as the difference of the mutual information between the highest and second highest correlated underlying factor.
Revised: The mutual information gap is then deﬁned as the difference of the mutual information between the highest and second highest correlated underlying factor.

Original: 2. R ELATED W ORK
Revised: 2. R ELATED W ORK

Original: Recent work has shown an increased interest into learning of interpretable representations.
Revised: Recent work has shown an increased interest into learning of interpretable representations.

Original: In addition to therelated work mentioned already, we brieﬂy review some of the inﬂuential papers:
Revised: In addition to the work mentioned already, we brieﬂy review some of the inﬂuential papers:

Original: Chen et al.
Revised: Chen et al.

Original: (2016) present a variant of a GAN that encourages an interpretable latent representation by maximizing the mutual information between the observation and a small subset of latent variables.
Revised: (2016) present a variant of a GAN that encourages an interpretable latent representation by maximizing the mutual information between the observation and a small subset of latent variables.

Original: The approach relies on optimizing a lower bound of the intractable mutual information.
Revised: The approach relies on optimizing a lower bound of the intractable mutual information.

Original: Kim &
Revised: Kim &

Original: Mnih (2018) propose a learning objective equivalent to β -TCVAE, and train it with the density ration trick Sugiyama et al.
Revised: Mnih (2018) propose a learning objective equivalent to β -TCVAE, and train it with the density ratio trick (Sugiyama et al., 2012).

Original: Kumar et al.
Revised: Kumar et al.

Original: (2017) introduce a regularizer of the KL-divergence between the approximate posterior and the prior distribution.
Revised: (2017) introduce a regularizer of the KL-divergence between the approximate posterior and the prior distribution.

Original: A parallel line of research proposes not to train a perfect generative model but instead to ﬁnd a simpler representation of the data (Vedantam et al. ; Hinton et al. (2011b)).
Revised: A parallel line of research proposes not to train a perfect generative model but instead to ﬁnd a simpler representation of the data (Vedantam et al., 2017; Hinton et al., 2011b).

Original: A similar strategy is followed in semi-supervised approaches that require implicit or explicit knowledge about the true underlying factors of the data Kulkarni et al.
Revised: A similar strategy is followed in semi-supervised approaches that require implicit or explicit knowledge about the true underlying factors of the data (Kulkarni et al., 2015;

Original: ; Kingma et al.
Revised: Kingma et al., 2014;

Original: Reed et al.
Revised: Reed et al., 2014;

Original: Baydin et al.

Original: ; Hinton et al. (2011a); Zhu et al. ; Goroshin et al.
Revised: Baydin et al., 2017; Hinton et al., 2011a; Zhu et al., 2017; Goroshin et al., 2015;

Original: ; Hsu et al. ; Denton et al.
Revised: Hsu et al., 2017; Denton et al., 2017).

Original: 2. O RTHOGONALIZATION AND P RUNING IN V ARIATIONAL I NFERENCE
Revised: 2. O RTHOGONALIZATION AND P RUNING IN V ARIATIONAL I NFERENCE

Original: There have been several interpretations of the behaviour of the β -VAE (Chen et al., 2018; Burgess et al., 2017).
Revised: There have been several interpretations of the behaviour of the β -VAE (Chen et al., 2018; Burgess et al., 2017).

Original: Here we provide a complementary perspective: that it enhances well known statistical biases in VI (Turner & Sahani, 2011) to produce disentangled, but not necessarily useful, representations.
Revised: Here we provide a complementary perspective: that it enhances well known statistical biases in VI (Turner & Sahani, 2011) to produce disentangled, but not necessarily useful, representations.

Original: The form of these biases can be understood by considering the variational objective when written as an explicit lower-bound: the log-likelihood of the parameters minus the KL divergence between the approximate posterior and the true posterior
Revised: The form of these biases can be understood by considering the variational objective when written as an explicit lower-bound: the log-likelihood of the parameters minus the KL divergence between the approximate posterior and the true posterior

Original: [Equation]
Revised: [Equation]

Original: From this form it is clear that VI’s estimates of the parameters θ will be biased away from the maximum likelihood solution (the maximizer of the ﬁrst term) in a direction that reduces the KL between the approximate and true posteriors.
Revised: From this form it is clear that VI’s estimates of the parameters θ will be biased away from the maximum likelihood solution (the maximizer of the ﬁrst term) in a direction that reduces the KL between the approximate and true posteriors.

Original: When factorized approximating distributions are used,
Revised: When factorized approximating distributions are used,

Original: VI will therefore be biased towards settings of the parameters that reduce the  dependence of the latent variables in the posterior.
Revised: VI will therefore be biased towards settings of the parameters that reduce the statistical dependence between the latent variables in the posterior.

Original: For example, this will bias learned components towards orthogonal directions in the output space as this reduces explaining away (e.g. in the factor analysis model, VI breaks the degeneracy of the maximum-likelihood solution ﬁnding the orthogonal PCA directions, see appendix).
Revised: For example, this will bias learned components towards orthogonal directions in the output space as this reduces explaining away (e.g. in the factor analysis model, VI breaks the degeneracy of the maximum-likelihood solution ﬁnding the orthogonal PCA directions, see appendix B.8).

Original: Moreover, these biases often cause components to be pruned out (in the sense that they have no effect on the observed variables) since then their posterior sits at the prior, which is typically factorized (e.g. in an over-complete factor analysis model VI prunes out components to return a complete model, see appendix).
Revised: Moreover, these biases often cause components to be pruned out (in the sense that they have no effect on the observed variables) since then their posterior sits at the prior, which is typically factorized (e.g. in an over-complete factor analysis model VI prunes out components to return a complete model, see appendix B.8).

Original: For simple linear models these effects are not pathological: indeed VI is arguably selecting from amongst the degenerate maximum likelihood solutions in a sensible way. 
Revised: For simple linear models these effects are not pathological: indeed VI is arguably selecting from amongst the degenerate maximum likelihood solutions in a sensible way. However, for more complex models the biases are more v ₀ z ₂ z ₃ z ₁ v ₁ p ₀ p ₁ (a) Tree corresponding to Eq.v ₀ p ₀ v ₃ p ₃ v ₂ p ₂ v ₄ p ₄ z v

Original: However, for more complex models the biases are more severe: often the true posterior of the underlying model has signiﬁcant dependencies (e.g. due to explaining away) and the biases can prevent the discovery of some components.
Revised: ₁ p ₁ z ₁ n (b) Tree visualization of Eq. 9, an L p -nested ISA model. severe: often the true posterior of the underlying model has signiﬁcant dependencies (e.g. due to explaining away) and the biases can prevent the discovery of some components.

Original: For example, VAEs are known to over-prune (Burda et al., 2015; Cremer et al., 2018).
Revised: For example, VAEs are known to over-prune (Burda et al., 2015; Cremer et al., 2018).

Original: 2. β -VAE E MPHASIZES O RTHOGONALIZATION AND P RUNING
Revised: 2. β -VAE E MPHASIZES O RTHOGONALIZATION AND P RUNING

Original: What happens to these biases in the β -VAE generalization when β > 1 ?[Equation]
Revised: What happens to these biases in the β -VAE generalization when β > 1 ?

Original: The short answer is that they grow.
Revised: The short answer is that they grow.

Original: This can be understood by considering coordinate ascent of the modiﬁed objective.
Revised: This can be understood by considering coordinate ascent of the modiﬁed objective.

Original: With θ ﬁxed, optimising q ﬁnds a solution that is closer to the prior distributionv ₀ z ₂ z ₃ z ₁ v ₁ p ₀ p ₁ v ₀ p ₀ v ₃ p ₃ v ₂ p ₂ v ₄ p ₄ z v ₁ p ₁ z ₁ n than VI due to the upweighting of the KL term in 4.
Revised: With θ ﬁxed, optimising q ﬁnds a solution that is closer to the prior distribution than VI due to the upweighting of the KL term in 2.

Original: With q ﬁxed, optimization over θ returns the same solution as VI (since the prior does not depend on the parameters θ and so the value of β is irrelevant).
Revised: With q ﬁxed, optimization over θ returns the same solution as VI (since the prior does not depend on the parameters θ and so the value of β is irrelevant).

Original: However, since q is now closer to the prior than before, the KL bias in 5 will be greater.
Revised: However, since q is now closer to the prior than before, the KL bias in equation 2 will be greater.

Original: These effects are shown in the ICA examplebelow [or in the appendix].
Revised: These effects are shown in the ICA example in appendix B.8.

Original: VI ( beta = 1 ) learns components that are more orthogonal than the underlying ones, but β = 5 prunes out one component entirely and sets the other two to be orthogonal.
Revised: VI ( β = 1 ) learns components that are more orthogonal than the underlying ones, but β = 5 prunes out one component entirely and sets the other two to be orthogonal.

Original: This is disentangled, but arguably leads to incorrect interpretation of the data.
Revised: This is disentangled, but arguably leads to incorrect interpretation of the data.

Original: This happens even though both methods are initialised at the true model.
Revised: This happens even though both methods are initialised at the true model.

Original: Arguably, the β -VAE is enhancing a statistical bug in VI and leveraging this as a feature.
Revised: Arguably, the β -VAE is enhancing a statistical bug in VI and leveraging this as a feature.

Original: We believe that this can be dangerous, preventing the discovery of the underlying model. 
Revised: We believe that this can be dangerous, preventing the discovery of the underlying model. [Equation]

Original: L ATENT P RIOR D ISTRIBUTIONS FOR U NSUPERVISED F ACTORIZATION
Revised: D ISTRIBUTIONS FOR U NSUPERVISED F ACTORIZATION

Original: In this section we describe an approach for unsupervised learning of disentangled representations.
Revised: In this section we describe an approach for unsupervised learning of disentangled representations.

Original: Instead of modifying the ELBO-objective, we propose to use certain families of prior distributions p ( z ) , that lead to identiﬁable and interpretable models.
Revised: Instead of modifying the ELBO-objective, we propose to use certain families of prior distributions p ( z ) , that lead to identiﬁable and interpretable models.

Original: In contrast to the standard normal distribution, the proposed priors are not rotationally invariant, and therefore allow interpretability of the latent space.
Revised: In contrast to the standard normal distribution, the proposed priors are not rotationally invariant, and therefore allow interpretability of the latent space.

Original: 3. I NDEPENDENT C OMPONENT A NALYSIS
Revised: 3. I NDEPENDENT C OMPONENT A NALYSIS

Original: Independent Component Analysis (ICA) seeks to factorize a distribution into non-Gaussian factors.
Revised: Independent Component Analysis (ICA) seeks to factorize a distribution into non-Gaussian factors.

Original: In order avoid the ambiguities of latent space rotations, a non-Gaussian distribution (e.g. Laplace or
Revised: In order avoid the ambiguities of latent space rotations, a non-Gaussian distribution (e.g. Laplace or

Original: Student-t distribution) is used as prior for the latent variables.
Revised: Student-t distribution) is used as prior for the latent variables.

Original: Generalized Gaussian Distribution A generalized version of ICALee & Lewicki (2000); Zhang et al. (2004); Lewicki (2002); Sinz & Bethge (2010) uses a prior from the family of exponential power distributions of the form
Revised: Generalized Gaussian Distribution A generalized version of ICA (Lee & Lewicki, 2000; Zhang et al., 2004; Lewicki, 2002; Sinz & Bethge, 2010) uses a prior from the family of exponential power distributions of the form

Original: [Equation]
Revised: [Equation]

Original:  also called generalized Gaussian , generalized Laplacian or p -generalized normal distribution.
Revised: (5) also called generalized Gaussian , generalized Laplacian or p -generalized normal distribution.

Original: Using p = 2 / (1 + κ ) the parameter κ is a measure of kurtosis. Box & Tiao (1973)
Revised: Using p = 2 / (1 + κ ) the parameter κ is a measure of kurtosis (Box & Tiao, 1973).

Original: This family of distributions generalizes the normal ( κ = 0 ) and the Laplacian ( κ = 1 ) distribution.
Revised: This family of distributions generalizes the normal ( κ = 0 ) and the Laplacian ( κ = 1 ) distribution.

Original: In general we get for κ > 0 leptokurtic and for κ < 0 platykurtic distributions.
Revised: In general we get for κ > 0 leptokurtic and for κ < 0 platykurtic distributions.

Original: Obviously the normal distribution is a special instance of the class of L p -spherically symmetric distributions, and the normal distribution is the only L 2 -spherically symmetric distribution with independent marginals.
Revised: The choice of a leptokurtic or platykurtic distribution has a strong inﬂuence on how a generative factor of the data is represented by a latent dimension. Fig. 2 depicts two possible prior distributions over latents that represent the [Figure]

Original: Equivalently Sinz et al.{+al., 2017). The leptokurtic distribution expects most of the probability mass around 
 and therefore favours a projection of the x and y coordinates, which are distributed in a square, onto the diagonal. The platykurtic prior is closer to a uniform distribution and therefore encourages an axis-aligned representation.+}
Revised: (x,y) spatial location of a sprite in the dSprites dataset (Matthey et {+al., 2017). The leptokurtic distribution expects most of the probability mass around 
 and therefore favours a projection of the x and y coordinates, which are distributed in a square, onto the diagonal. The platykurtic prior is closer to a uniform distribution and therefore encourages an axis-aligned representation.+}

Revised: This example shows the choice of the prior will effect the latent representation.

Revised: Obviously the normal distribution is a special instance of the class of L p -spherically symmetric distributions, and the normal distribution is the only L 2 -spherically symmetric distribution with independent marginals.

Original: (2009a) showed that this also generalizes to arbitrary values of p .
Revised: Equivalently (Sinz et al., 2009a) showed that this also generalizes to arbitrary values of p .

Original: The marginals of the p -generalized normal distribution are independent, and it is the only factorial model in the class of L p -spherically symmetric distributions.
Revised: The marginals of the p -generalized normal distribution are independent, and it is the only factorial model in the class of L p -spherically symmetric distributions.

Original: We investigate the behaviour of L p -spherically symmetric distributions as prior distributions for p ( z ) in the experiments in section 4.
Revised: We investigate the behaviour of L p -spherically symmetric distributions as prior distributions for p ( z ) in the experiments in section 4.

Original: 3. I NDEPENDENT S UBSPACE A NALYSIS
Revised: 3. I NDEPENDENT S UBSPACE A NALYSIS

Original: ICA can be  generalized to include indepdence between subspaces, but dependencies within them, by using a more general prior, the family of L p -nested symmetric distributions.
Revised: ICA can be further generalized to include independence between subspaces, but dependencies within them, by using a more general prior, the family of L p -nested symmetric distributions (Hyv ¨ arinen &

Original:  Hyv¨arinen & Hoyer (2000); Hyv¨arinen & K¨oster (2007); Sinz et al. (2009b) ; Sinz & Bethge (2010) [Figure]
Revised: Hoyer, 2000; Hyv¨arinen & K¨oster, 2007; Sinz et al., 2009b; Sinz & Bethge, 2010).

Original:  To start, let’s consider functions of the form
Revised: L p -nested Function To start, let’s consider functions of the form

Original: [Equation]
Revised: [Equation]

Original: with p 0 , p 1 ∈ R .
Revised: with p 0 , p 1 ∈ R .

Original: This function is a cascade of two L p -norms.
Revised: This function is a cascade of two L p -norms.

Original: To give a better intuition we provide a visualization of this distribution in ﬁgure 1a, which depicts equation 10 as a tree that visualizes the nested structure of the norms.
Revised: To aid intuition we provide a visualization of this distribution in ﬁgure 1a, which depicts (6) as a tree that visualizes the nested structure of the norms.

Original: We call the class of functions which employ this structure L p -nested .
Revised: We call the class of functions which employ this structure L p -nested .

Original: L p -nested Distribution Given a L p -nested function f and a radial density φ : R (cid:55)→ R + we deﬁne the L p -nested symmetric distribution following Fernandez et al.
Revised: L p -nested Distribution Given an L p -nested function f and a radial density ψ 0 : R (cid:55)→ R + we deﬁne the L p -nested symmetric distribution following Fernandez et al.

Original: (1995) as
Revised: (1995) as

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where S f (1) is the surface area of the L p -nested sphere.
Revised: where S f (1) is the surface area of the L p -nested sphere.

Original: This surface area can be obtained by using the gamma function:
Revised: This surface area can be obtained by using the gamma function:

Original: [Equation] 
Revised: [Equation] [Equation]

Original:  where l i is the number of children of a node i , n i is the number of leaves in a subtree under the node i , and n i,k is the number of leaves in the subtree of the k -th children of node i .
Revised: ) where l i is the number of children of a node i , n i is the number of leaves in a subtree under the node i , and n i,k is the number of leaves in the subtree of the k -th children of node i .

Original: For further details we refer the reader to the excellent work of Sinz & Bethge (2010).
Revised: For further details we refer the reader to the excellent work of Sinz & Bethge (2010).

Original: Independent Subspace Analysis The family of L p -nested distibutions allow a generalization of ICA called independent subspace analysis (ISA).
Revised: Independent Subspace Analysis The family of L p -nested distributions allows a generalization of ICA called independent subspace analysis (ISA).

Original: ISA uses a subclass of L p -distributions, which are deﬁned by functions of the form
Revised: ISA uses a subclass of L p -nested distributions, which are deﬁned by functions of the form

Original: [Equation]
Revised: [Equation]

Original: and correspond to trees of depth two.
Revised: and correspond to trees of depth two.

Original: The tree structure of this subclass of functions is visualizaed in ﬁgure 1b.
Revised: The tree structure of this subclass of functions is visualized in ﬁgure 1b where each v i , i = 1 , . . .

Revised: , l 0 denotes the function value of the L p -norm evaluated over a node’s children.

Revised: The components z j of z that contribute to each v

Original:  Sinz & Bethge (2010) showed that the subspaces v 1 , . .
Revised: i form a subspace [Equation] Sinz & Bethge (2010) showed that the subspaces V 1 , . .

Original: . , v l 0 become independent when using the radial distribution
Revised: , V l 0 become independent when using the radial distribution

Original: [Equation]
Revised: [Equation]

Original: which we can interpret as a generalization of the Chi-distribution.
Revised: which we can interpret as a generalization of the Chi-distribution.

Original:  We propose to choose the latent prior p  ( z ) from this family of distributions, which allows us to deﬁne independent subspaces in the latent space.
Revised: ISA-VAE We propose to choose the latent prior p ISA ( z ) (Eq. 7) with f ( z ) from the family of ISA models of the form of Eq. 9, which allows us to deﬁne independent subspaces in the latent space.

Revised: The Kulback-Leibler divergence of the ELBO-objective can be estimated by Monte-Carlo sampling.

Revised: This leads to an ELBO-objective of the form

Revised: [Equation]

Revised: which only requires to compute the log-density of the prior that is readily accessible from the density deﬁned in Eq.

Revised: As discussed in Roeder et al.

Original: Experiments in the following section demonstrate that priors of this formsupport unsupervised learning ofdisentangled representation even for theunmodiﬁed ELBO objective ( β = 1 ).
Revised: (2017) this form of the ELBO even has potential advantages (variance reduction) in comparison to a closed form KL-divergence. ISA-TCVAE The proposed latent prior can also be combined with the β -TCVAE approach and we get the objective [Equation]

Revised: where I q denotes the index code mutual information.

Revised: To compute the terms in Eq. 13, Chen et al.

Revised: (2018) propose a Monte-Carlo sampling approach called minibatch-weighted sampling, which also only requires to compute the log density of the prior.

Revised: Sampling and the Reparameterization Trick

Revised: If we want to sample from the generative model we have to be able to sample from the prior distribution.

Revised: Sinz & Bethge (2010) describe an exact sampling approach to sample from an L p -nested distribution, which we reproduce as Algorithm 1 in the appendix.

Revised: Note that during training we only have to sample from the approximate posterior q φ , which we do not have to modify and which can remain a multivariate Gaussian distribution following the original VAE approach.

Revised: As a consequence, the reparameterization trick can be applied (Kingma & Welling, 2014).

Revised: Experiments in the following section demonstrate that the proposed prior supports unsupervised learning of disentangled representation even for the unmodiﬁed ELBO objective ( β = 1 ).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments, we evaluate the inﬂuence of the proposed prior distribution on disentanglement and on the quality of the reconstruction.
Revised: In our experiments, we evaluate the inﬂuence of the proposed prior distribution on disentanglement and on the quality of the reconstruction on the dSprites dataset (Matthey et al., 2017), which contains images of three different shapes undergoing transformations of their position, scale and rotation. y - po s s ca l e r o t a ti on r o t a ti on x - po s (a) ISA-VAE, β = 1 , MIG: 0 .r o t a ti on y - po s x - po s r o t a ti on s ca l e (b) ISA-TCVAE, β = 2 . 5 , MIG: 0 .

Original:  To provide a quantitative evaluation of the disentanglement we compute the disentanglement metric Mutual Information Gap (MIG) that was proposed in Chen et al.
Revised: Disentanglement Metrics To provide a quantitative evaluation of the disentanglement we compute the disentanglement metric Mutual Information Gap (MIG) that was proposed in Chen et al.

Revised: The MIG score measures how much mutual information a latent dimension shares with the underlying factor and how well this latent dimension is separated from the other latent factors.

Revised: Therefore the MIG measures the two desired properties usually referred to with the term disentanglement : a factorized latent representation, and interpretability of the latent factors.

Original: In this publication the authors demonstrate that β -TCVAE, a modiﬁcation of β -VAE, enables learning of representations with higher MIG score than β -VAEHiggins et al.
Revised: Chen et al. (2018) compare the MIG metric to existing disentanglement metrics (Higgins et al., 2016; Kim & Mnih, 2018) and demonstrate that the MIG is more effective and that the other metrics do not allow to capture both properties in a desirable way.

Original: (2016), InfoGAN Chen et al. 
Revised: Reconstruction Quality To quantify the reconstruction quality, we report the expected (log-)likelihood of the reconstructed data E q φ ( z | x ) [log p θ ( x | z )] . In our opinion this measure is more informative than the ELBO, frequently reported in existing work, e.g. (Chen et al., 2018), especially when varying the β parameter, the weighting of the KL term, which is part of the ELBO. Comparison Baselines Chen et al. (2018) demonstrate that β -TCVAE, a modiﬁcation of the βVAE, enables learning of representations with higher MIG score than β -VAE (Higgins et al., 2016),

Original: (2016) and FactorVAE Kim & Mnih (2018).
Revised: InfoGAN (Chen et al., 2016), and FactorVAE (Kim & Mnih, 2018).

Original: Therefore we choose  β -TCVAE and β -VAEfor comparison in our experiments.
Revised: Therefore we choose to compare against β -TCVAE and β -VAE in our experiments.

Revised: Architecture of the Encoder and Decoder To allow an accurate comparison we use the same architecture for the decoder and encoder as presented in Chen et al.

Revised: We reproduce the description of the encoder and decoder in appendix A.

Revised: Choosing the ISA-layout We follow the practice of Sinz et al.

Revised: (2009b)

Revised: and perform a search over different values for the parameters of the ISA model and choose the model with the highest disentanglement (MIG) score.

Original:  4. S UPPORT OF THE P RIOR TO L EARN D ISENTANGLED R EPRESENTATIONS
Revised: A comparison of different parameter settings and more details about this procedure are provided in appendix A. [Figure] = 2 . 2 . 4. S UPPORT OF THE P RIOR TO L EARN D ISENTANGLED R EPRESENTATIONS

Original: First, we investigate the ability of the prior to support unsupervised learning of disentangled representations for the unmodiﬁed ELBO-objective.
Revised: First, we investigate the ability of the prior to support unsupervised learning of disentangled representations for the unmodiﬁed ELBO-objective.

Original: We compare two DGMs after training with the unmodiﬁed ELBO-objective: the standard VAE with standard normal prior p ( z ) , and ISA-VAE, the combination of thestandard VAE with a prior p ( z ) from the family of L p -nested ISA distributions.
Revised: Figure 3a depicts the structure of the latent representation after training for ISA-VAE, a combination of the L p -nested ISA prior with the standard VAE approach.

Original: The combination of the L p -nested ISA prior with the standard VAE approach learns reasonably disentangled representations even for the unmodiﬁed ELBO objective( β = 1 ).
Revised: Because our prior allows independent subspaces the latent space becomes interpretable even when using the unmodiﬁed ELBO objective with β = 1 . The plots were produced with the reference implementation of Chen et al.

Original: Figure 4a depicts the structure of the latent representation for ISA-VAE after training.
Revised: (2018) for visualizing latent representations for the dSprites dataset.

Original: Even for the unmodiﬁed objective the latent space becomes interpretable, which we did not observe in our experiments for  the standard VAE with standard normal prior.
Revised: When combining the ISA-model with β -TCVAE and varying the β parameter, a model with a high disentanglement score of MIG = 0 . 54 can be reached. This is the so far highest score reported for dSprites in the literature.

Original: 4. Q UANTITATIVE C OMPARISONS OF THE D IFFERENT A PPROACHES
Revised: 4. Q UANTITATIVE C OMPARISON OF THE D IFFERENT A PPROACHES

Original: The proposed L p -nested prior distribution can be easily combined with the existing approaches for unsupervised disentangled representation learning β -VAE Higgins et al.
Revised: This beneﬁt of the proposed prior, that it encourages disentangled representations becomes even more obvious in our quantitative comparison. We compare the approaches ISA-VAE and ISATCVAE, that use the proposed L p -nested prior p ISA with their respective counterpart β -VAE (Higgins et al., 2016) and β -TCVAE

Original: (2016) and β -TCVAE Chen et al.
Revised: (Chen et al., 2018) that use the standard normal prior p N .

Revised: Because the amount of disentanglement depends on the choice of the parameter β , we vary β in the interval between 1 and 6 with a stepsize of 0 . 5 .

Original: We compare the performance of the four different approaches in ﬁgure 5: β -VAE and
Revised: We compare the performance of the four different approaches in ﬁgure 4 with 16 experiments for each value of β .

Original: ISA- β -VAE, a combination of β -VAE with the proposed L p -nested ISA prior, and equivalently βTCVAE and ISA- β -TCVAE. Clearly  both approaches beneﬁt from the proposed prior, and higher disentanglement scores can be achieved  with ISA- β -VAE and ISA- β -TCVAE in comparison to the original approaches. 
Revised: Clearly in both cases a higher disentanglement score can be achieved already for smaller values of β with ISA- β -VAE and ISA- β -TCVAE in comparison to the original approaches. Even when choosing the individually best value of β , that reaches the highest MIG score for each method, the poposed approaches reach a higher MIG score than their respective counterparts.

Original: [Figure]
Revised: Fig.

Original: [Figure]
Revised: 5 depicts the distribution of MIG scores for the individually best value of β : Both average and mean of the MIG scores are higher for the variants that use the ISA-model.

Original: 4. T RADE - OFF BETWEEN D ISENTANGLEMENT AND R ECONSTRUCTION L OSS
Revised: 4. T RADE - OFF BETWEEN D ISENTANGLEMENT AND R ECONSTRUCTION L OSS

Original: Since the  prior facilitates learning of disentangled representations, not only a higher disentanglement score can be reached, but also higher scores are reached for smaller values of β , when compared to the original approaches.
Revised: Since the proposed prior facilitates learning of disentangled representations, not only a higher disentanglement score can be reached, but also higher scores are reached for smaller values of β , when compared to the original approaches.

Original: This leads to a clear improvement of the trade-off between disentanglement and reconstruction loss.
Revised: This leads to a clear improvement of the trade-off between disentanglement and reconstruction loss.

Original: The improvement of this trade-off is demonstrated in ﬁgure 5, where we plot both the disentanglement score and the reconstruction loss for varying values of β .
Revised: The improvement of this trade-off is demonstrated in ﬁgure 4, where we plot both the disentanglement score and the reconstruction loss for varying values [Figure]

Original:  ISA- βVAE and ISA- β -TCVAE reach high values of the disentanglement score for smaller values of β which and at the same time preserves a higher quality of the reconstruction than the respective original approaches.
Revised: The proposed approaches ISA-VAE and ISA-TCVAE allow a better trade-off between disentanglement and the reconstruction loss, almost reaching the reconstruction quality of the standard, nondisentangling VAE. Number of experiments: Layout the model all of β . ISA- β -VAE and ISA- β -TCVAE reach high values of the disentanglement score for smaller values of β which and at the same time preserves a higher quality of the reconstruction than the respective original approaches.

Original: At the same time we observe that with the proposed prior the quality of the reconstruction decreases at a smaller rate than for the original approaches.
Revised: At the same time we observe that with the proposed prior the quality of the reconstruction decreases at a smaller rate than for the original approaches.

Original: C ONCLUSION
Revised: This improvement of the trade-off between disentanglement and the reconstruction loss becomes also obvious in ﬁgure 5b where we plot the MIG score with respect to the reconstruction loss for the individually best value of β . The proposed approaches ISA-VAE and ISA-TCVAE allow higher MIG scores than their respective base lines while at the same time providing a better reconstruction quality, almost reaching the reconstruction quality of the standard, non-disentangling VAE. Interestingly the plot also shows that the increase of the MIG score for the baseline method β -TCVAE comes at the cost of a much lower reconstruction quality.

Revised: This difference in the reconstruction quality becomes obvious in the quality of the reconstructed images.

Revised: Please refer to the appendix where we present latent traversals in appendix A.3 and image reconstruction experiments in appendix A.4.

Revised: With the proposed approach ISA-TCVAE the reconstruction quality can be increased signiﬁcantly while at the same time providing a higher disentanglement.

Revised:  C ONCLUSION

Original: We presented a structured prior for unsupervised learning of disentangled representations in deep generative models.
Revised: We presented a structured prior for unsupervised learning of disentangled representations in deep generative models.

Original: We choose the prior from the family of L p -nested symmetric distributions which enables deﬁning of a hierarchy of independent subspaces in the latent space.
Revised: We choose the prior from the family of L p -nested symmetric distributions which enables the deﬁnition of a hierarchy of independent subspaces in the latent space.

Original: In contrast to the standard normal prior that is often used in training of deep generative models the proposed prior is not rotationally invariant and therefore enhances the interpretability of the latent space.
Revised: In contrast to the standard normal prior that is often used in training of deep generative models the proposed prior is not rotationally invariant and therefore enhances the interpretability of the latent space.

Original: We demonstrate in our experiments, that a combination of the proposed prior with existing approaches for unsupervised learning of disentangled representations allows a signiﬁcant improvement of the trade-off between disentanglement and reconstruction loss
Revised: We demonstrate in our experiments, that a combination of the proposed prior with existing approaches for unsupervised learning of disentangled representations allows a signiﬁcant improvement of the trade-off between disentanglement and reconstruction loss. A A PPENDIX A. C HOOSING THE ISA- LAYOUT We vary l 0 between 4 and 10 and choose the same value for l 1 = l 2 ,...,l 0 between 2 and 10 .

Revised: We set the parameter range of the exponents p i to p i ∈ [0 . 9 , 2 . 4] with a discretization step size of 0 . 1 , which includes lepto- and platokyrtic distributions.

Original: Toy examples showing biases in VI and β -VI{+platykurtic distributions at the child subspaces lead to different representations of the x and y coordinate. Because the MIG metric evaluates axis-alignment of the latent dimensions to the underlying factors, here the x and y coordinate, platykurtic priors in general achieve a higher MIG score. The child subspaces share the same parameter p 1 = p 2 ,...,l 
+}
Revised: Fig. 2 depicts how lepto- and {+platykurtic distributions at the child subspaces lead to different representations of the x and y coordinate. Because the MIG metric evaluates axis-alignment of the latent dimensions to the underlying factors, here the x and y coordinate, platykurtic priors in general achieve a higher MIG score. The child subspaces share the same parameter p 1 = p 2 ,...,l 
+}

Revised: and we choose the exponent of the root node as p 0 (cid:54) =

Revised: p 1 to ensure independence of the subspaces.

Revised: To study the inﬂuence of the layout on the reconstruction quality and MIG score we compare the results for different values of p 0 , p 1 ,..., 5 and l 1 , and vary the value of β in the interval β ∈ [1 , 4] with a step size of 0 . 5 and repeat each experiment four times.

Revised: We compare four layouts with the highest MIG score for each subspace layout in Fig.

Revised: 6 where we plot the mean and standard error of MIG score and reconstruction loss.

Revised: [Figure]

Revised: A. S AMPLING FROM L p - NESTED S YMMETRIC D ISTRIBUTIONS

Revised: Sample v 0 from a beta distribution β [ n, 1] 2.

Revised: For each inner node i of the tree associated with f , sample the auxiliary variable s i from a Dirichlet distribution Dir (cid:104) n i, 1 p i , . .

Revised: . , n i,l 1 p i (cid:105) where n i,k are the number of leaves in the subtree under node i, k .

Revised: Obtain coordinates on the L p -nested sphere within the positive orthant by s i

Revised: (cid:55)→ s

Revised: (the exponentiation is taken component-wise) 3.

Revised: Transform these samples to Cartesian coordinates by v i · ˜ u

Revised: i = v i, 1: l i for each inner node, starting from the root node and descending to lower layers.

Revised: The components of v i, 1: l i constitute the radii for the layer direct below them.

Revised: If i = 0 , the radius had been sampled in step4.

Revised: Once the two previous steps have been repeated until no inner node is left, we have a sample x from the uniform distribution in the positive quadrant.

Revised: Normalize x to get a uniform sample from the sphere u = x f ( x ) 5.

Revised: Sample a new radius ˜ v 0 from the radial distribution of the target radial distribution ψ 0 and obtain the sample via ˜ x

Revised: = ˜ v 0 · u 6.

Revised: Multiply each entry x

Revised: i of ˜ x by and independent sample z i from the uniform distribution over {− 1 , 1 } .

Revised: A. D ISENTANGLEMENT R EPRESENTATIONS AND L ATENT T RAVERSALS

Revised: We follow standard practice established in Chen et al.

Revised: (2018) for visualizing latent representations and additionally show images generated by traversals of the latent along the respective axis.

Revised: The red and blue color coding in the ﬁrst

Revised: ISA-layouts:

Revised: ISA-VAE: l 0 = 5 , l 1 ,..., 5 = 5 , p 0 = 2 . 1 , p 1 ,..., 5 = 2 .

Revised: 2 ISA-TCVAE l 0 = 5 , l 1 ,..., 5 = 4 , p 0 = 2 . 1 , p 1 ,..., 5 = 2 .

Revised: r o t a ti on x - po s s ca l e y - po s

Revised: [Figure]

Revised: s ca l e y - po s r o t

Revised: a ti on x - po s

Revised: [Figure]

Revised: s ca l e y - po s r o t

Revised: a ti on x - po s r o t a ti on (

Revised: [Figure]

Revised: [Figure]

Revised: r o t a ti on y - po s x - po s r o t a ti on s

Revised: [Figure]

Revised: l e

Revised: [Figure]

Revised: r o t a ti on x

Revised: [Figure]

Revised: po s s ca l e y - po s

Revised: [Figure]

Revised: s

Revised: [Figure]

Revised: l e y - po s r o t a ti

Revised: [Figure]

Revised: x

Revised: [Figure]

Revised: po s

Revised: [Figure]

Revised: s ca l e y - po s r o t

Revised: a ti on x - po s r o t a ti on

Revised: [Figure]

Revised: [Figure]

Revised: r o t a ti on y - po s x - po s r o t a ti on s

Revised: [Figure]

Revised: l e

Revised: [Figure]

Revised: r o t a ti on x

Revised: [Figure]

Revised: po s s ca l e y - po s

Revised: [Figure]

Revised: s

Revised: [Figure]

Revised: l e y - po s r o t a ti

Revised: [Figure]

Revised: x

Revised: [Figure]

Revised: po s

Revised: [Figure]

Revised: s ca l e y - po s r o t

Revised: a ti on x - po s r o t a ti on

Revised: [Figure]

Revised: [Figure]

Revised: r o t a ti on y - po s x - po s r o t a ti on s

Revised: [Figure]

Revised: l e

Revised: [Figure]

Revised: [Figure]

Revised: i npu t β - VA E β - T C VA

Revised: E

Revised: I S A - VA

Revised: E

Revised: I S A - T C

Revised: VA E

Revised: A. M ODEL A RCHITECTURE (P Y T ORCH )

Revised: The models were trained with the optimization algorithm Adam (Kingma & Ba, 2015) using a learning rate parameter of 0.

Revised: All unmentioned hyperparameters are PyTorch v0.41 defaults.

Revised: c

Revised: l a s s MLPEncoder ( nn . Module ) :

Revised: def i n i t ( s e l f , output dim ) : super ( MLPEncoder , s e l f ) .

Revised: i n i t ( ) s e l f .

Revised: output dim = output dim s e l f .

Revised: fc1 = nn .

Revised: Linear (4096 , 1200)

Revised: s e l f .

Revised: fc2 =

Revised: nn .

Revised: Linear (1200 , 1200)

Revised: s e l f .

Original: {+fc
+}
Revised: {+fc
+}

Revised: = nn .

Revised: Linear (1200 , output dim ) s e l f .

Revised: conv z = nn . Conv2d (64 , output dim , 4 , 1 , 0)

Revised: # setup the non − l i n e a r

Revised: i t y s e l f .

Revised: a c t

Revised: = nn . ReLU( i n p l a c e =True ) def forward ( s e l f , x ) : h = x . view ( − 1 , 64 ∗ 64) h

Revised: = s e l f .

Revised: a c t ( s e l f .

Revised: fc1 ( h ) )

Revised: h

Revised: = s e l f .

Revised: a c t ( s e l f .

Revised: fc2 ( h ) )

Revised: h

Revised: = s e l f .

Revised: fc3 ( h ) z

Revised: = h . view ( x . s i

Revised: z e ( 0 ) , s e l f .

Revised: output dim ) r e t u r n z c

Revised: l

Revised: a s s MLPDecoder ( nn . Module ) :

Revised: def i n i t ( s e l f , input dim ) : super ( MLPDecoder , s e l f ) .

Revised: i n i t ( ) s e l f .

Revised: net = nn .

Revised: S e q u e n t i a l ( nn .

Revised: Linear ( input dim , 1200) , nn .

Revised: Tanh ( ) , nn .

Revised: Linear (1200 , 1200) , nn .

Revised: Tanh ( ) , nn .

Revised: Linear (1200 , 1200) , nn .

Revised: Tanh ( ) , nn .

Revised: Linear (1200 , 4096) )

Revised: h ) mu img =

Revised: . view z

Original:  This section provides the details of the toy examples that reveal the biases in variational methods.
Revised: [Table] [Table] B T OY EXAMPLES SHOWING BIASES IN VI AND β -VI This section provides the details of the toy examples that reveal the biases in variational methods.

Original: First we will consider the factor analysis model showing that VI breaks the degeneracy of the maximum-likelihood solution to 1) discover orthogonal weights that lie in the PCA directions, 2) prune out extra components in over-complete factor analysis models, even though there are solutions with the same likelihood that preserve all components.
Revised: First we will consider the factor analysis model showing that VI breaks the degeneracy of the maximum-likelihood solution to 1) discover orthogonal weights that lie in the PCA directions, 2) prune out extra components in over-complete factor analysis models, even though there are solutions with the same likelihood that preserve all components.

Original: We also show that in these examples the β -VI returns identical model ﬁts to VI regardless of the setting of β .
Revised: We also show that in these examples the β -VI returns identical model ﬁts to VI regardless of the setting of β .

Original: Second, we consider an over-complete ICA model and initialize using the true model.
Revised: Second, we consider an over-complete ICA model and initialize using the true model.

Original: We show that 1) VI is biased away from the true component directions towards more orthogonal directions, and 2) β -VI with a modest setting of β =prunes away one of the components and ﬁnds orthogonal directions for the other two.
Revised: We show that 1) VI is biased away from the true component directions towards more orthogonal directions, and 2) β -VI with a modest setting of β = 5 prunes away one of the components and ﬁnds orthogonal directions for the other two.

Original: That is, it ﬁnds a disentangled representation, but one which does not reﬂect the underlying components.
Revised: That is, it ﬁnds a disentangled representation, but one which does not reﬂect the underlying components.

Original: 1. Background
Revised: B. B ACKGROUND

Original: The β -VAE optimizes the modiﬁed free-energy, F β ( q ( z 1: N ) , θ ), with respect to the parameters θ and the variational approximation q ( z 1: N ),
Revised: The β -VAE optimizes the modiﬁed free-energy, F β ( q ( z 1: N ) , θ ) , with respect to the parameters θ and the variational approximation q ( z 1: N ) ,

Original: [Equation]
Revised: [Equation]

Original: Consider the case where M = 1 β is a positive integer, M ∈ N , we then haveF
Revised: Consider the case where M = 1 β is a positive integer, M ∈ N , we then have

Original: [Equation]
Revised: [Equation]

Original: z [Equation] In this case, the β -VAE can be thought of as attaching M replicated observations to each latent variable z n and then running standard variational inference on the new replicated dataset.
Revised: In this case, the β -VAE can be thought of as attaching M replicated observations to each latent variable z n and then running standard variational inference on the new replicated dataset.

Original: This can equivalently be thought of as raising each likelihood p ( x n | z n , θ ) to the power M .
Revised: This can equivalently be thought of as raising each likelihood p ( x n | z n , θ ) to the power M .

Original: Now in real applications β will be set to a value that is greater than one.
Revised: Now in real applications β will be set to a value that is greater than one.

Original: In this case, the eﬀect of β is the opposite: it is to reduce the number of eﬀective data points per latent variable to be less than one M < 1.
Revised: In this case, the effect of β is the opposite: it is to reduce the number of effective data points per latent variable to be less than one M < 1 .

Original: Or equivalently we raise each likelihood term to a power M that is less than one.
Revised: Or equivalently we raise each likelihood term to a power M that is less than one.

Original: Standard VI is then run on these modiﬁed data (e.g. via joint optimization of q and θ ).
Revised: Standard VI is then run on these modiﬁed data (e.g. via joint optimization of q and θ ).

Original: Although this view is mathematically straightforward, the perspective of the β -VAE i) modifying the dataset, and ii) applying standard VI, is useful as it will allow us to derive optimal solutions for the variational distribution q ( z ) in simple cases like the factor analysis model considered next.
Revised: Although this view is mathematically straightforward, the perspective of the β -VAE i) modifying the dataset, and ii) applying standard VI, is useful as it will allow us to derive optimal solutions for the variational distribution q ( z ) in simple cases like the factor analysis model considered next.

Original: 1. Factor analysis
Revised: B. F ACTOR ANALYSIS

Original: Consider the factor analysis generative model.
Revised: Consider the factor analysis generative model.

Original: Let x ∈ R L and z ∈ R K .
Revised: Let x ∈ R L and z ∈ R K .

Original: for n = 1 ...N
Revised: for n = 1 ...N

Original: [Equation]
Revised: [Equation]

Revised: D = diag ([

Original: [Equation] 
Revised: [Equation] [Equation]

Original: , D ) where D = [ σ 21 , ..., σ 2D ] (1) The true posterior is a Gaussian p ( z n | x n , θ ) = N ( z ; µ z | x , Σ z | x ) where µ z | x = Σ z | x W (cid:62) D − 1 x and Σ z | x = ( W (cid:62) D
Revised: σ 2D ])

Original: [Equation]
Revised: [Equation]

Original: 1 W + I) − 1 .

Original: The true log-likelihood of the parameters is log p ( x 1: N |  θ ) = 
Revised: The true posterior is a Gaussian p ( z n | x n , θ ) = N ( z ; µ z | x , Σ z | x ) where

Original: [Equation] 
Revised: [Equation] [Equation]

Original: (cid:88)
Revised: The true log-likelihood of the parameters is

Original: [Equation]
Revised: [Equation]

Original: Here we have deﬁned the empirical mean and covariance of the observations µ x = 1 N (cid:80)
Revised: Here we have deﬁned the empirical mean and covariance of the observations µ x = 1 N (cid:80)

Original: Nn =1 x n and Σ x = 1 N (cid:80)
Revised: Nn =1 x n and Σ x = 1 N (cid:80)

Original: Nn =1 ( x n − µ x )( x n − µ x ) (cid:62) i.e. the suﬃcient statistics.
Revised: Nn =1 ( x n − µ x )( x n − µ x ) (cid:62) i.e. the sufﬁcient statistics.

Original: Thetruelikelihood is invariant under orthogonal transformations of the latent variables: z (cid:48)
Revised: The true likelihood is invariant under orthogonal transformations of the latent variables: z (cid:48)

Original: =
Revised: =

Original: R z where RR (cid:62) =
Revised: R z where RR (cid:62) =

Original: I.
Revised: I .

Original: Interpreting β -VI as running VI in a modiﬁed generative model (see previous section)
Revised: Interpreting β -VI as running VI in a modiﬁed generative model (see previous section)

Original: we have the new generative process for n = 1 ...Nz n ∼ N
Revised: we have the new generative process for n = 1 ...N

Original: ( z ; 0 , I) , for m = 1 ...M ( β ) x n,m ∼ N (x ; W z  , D ) where D = [ σ 21 , ..., σ 2D ]
Revised: [Equation] , I) , for m = 1 ...M ( β ) x n,m ∼ N ( W z n , D ) where D = diag ([ σ 21 , ..., σ 2D ])

Original: We now observe data and set x n,m = x n .
Revised: We now observe data and set x n,m = x n .

Original: The posterior is again Gaussian p ( z n | x n , θ, M ( β ))
Revised: The posterior is again Gaussian p ( z n | x n , θ, M ( β ))

Original: =
Revised: =

Original: N ( z  ; ˜ µ z | x ( β ) , ˜Σ z | x ( β )) where
Revised: N ( z n ; ˜ µ z | x ( β, n ) , ˜Σ z | x ( β )) where

Original: [Equation]
Revised: [Equation]

Original: Here we have taken care to explicitly reveal all of the direct dependencies on β .
Revised: Here we have taken care to explicitly reveal all of the direct dependencies on β .

Original: Mean-ﬁeld variational inference, q ( z n ) =
Revised: Mean-ﬁeld variational inference, q ( z n ) =

Original: (cid:81) k q n,k
Revised: (cid:81) k q n,k

Original: ( z k,d ), will return a diagonal Gaussian approximation to the true posterior with the same mean and matching diagonal precision, 
Revised: ( z k,d ) , will return a diagonal Gaussian approximation to the true posterior with the same mean and matching diagonal precision, q ( z n | x n , θ, M ( β ))

Original: [Equation]
Revised: = N (cid:0) z n ; ˜ µ z | x ( β, n ) , Σ q ( β ) (cid:1) where Σ − 1 q ( β ) = diag (cid:16) ˜Σ − 1 z | x ( β ) (cid:17)

Original: We notice that the posterior mean is a linear combination of the observations ˜ µ z | x ( β ) =
Revised: We notice that the posterior mean is a linear combination of the observations ˜ µ z | x ( β, n ) =

Original: R ( β ) x  where R ( β ) =
Revised: R ( β ) x n where R ( β ) =

Original: ˜Σ z | x ( β ) M ( β ) W (cid:62) D − 1 are recognition weights.
Revised: ˜Σ z | x ( β ) M ( β ) W (cid:62) D − 1 are recognition weights.

Original: Notice that the recognition weights and the posterior variances are the same for all data points: they do not depend on n .
Revised: Notice that the recognition weights and the posterior variances are the same for all data points: they do not depend on n .

Original: The free-energy is then
Revised: The free-energy is then

Original: [Equation]
Revised: [Equation]

Original: with the reconstruction term being
Revised: with the reconstruction term being

Original: [Equation]
Revised: [Equation]

Original: and the KL or regularization term being
Revised: and the KL or regularization term being

Original: [Equation]
Revised: [Equation]

Original: We will now consider the objective functions and the posterior distributions in several cases to reason about the parameter estimates arising from the methods above.
Revised: We will now consider the objective functions and the posterior distributions in several cases to reason about the parameter estimates arising from the methods above.

Original: 1. Experiment 1: mean ﬁeld VI applied to factor analysis yields the PCA directions
Revised: B. E XPERIMENT 1: MEAN FIELD VI APPLIED TO FACTOR ANALYSIS YIELDS THE PCA DIRECTIONS

Original: Consider the situation where we know a maximum likelihood solution of the weights W ML .
Revised: Consider the situation where we know a maximum likelihood solution of the weights W ML .

Original: For simplicity we select the solution W ML which has orthogonal weights in the observation space.
Revised: For simplicity we select the solution W ML which has orthogonal weights in the observation space.

Original: We then rotate this solution by an amount θ so that W (cid:48)
Revised: We then rotate this solution by an amount θ so that W (cid:48)

Original: ML =
Revised: ML =

Original: R ( θ ) W ML .
Revised: R ( θ ) W ML .

Original: The resulting weights are no longer orthogonal (assuming the rotation is not an integer multiple of π/ 2).
Revised: The resulting weights are no longer orthogonal (assuming the rotation is not an integer multiple of π/ 2 ).

Original: We compute the loglikelihood (which will not change) and the free-energy (which will change) and plot the true and approximate posterior covariance (which does not depend on the datapoint value x n ).
Revised: We compute the loglikelihood (which will not change) and the free-energy (which will change) and plot the true and approximate posterior covariance (which does not depend on the datapoint value x n ).

Original: First here are the weights are aligned with the true ones.
Revised: First here are the weights are aligned with the true ones.

Original: The log-likelihood and the free-energy is -17.82 nats.
Revised: The log-likelihood and the free-energy take the same value of -17.82 nats.

Original: [Figure]
Revised: [Figure]

Original: Second, here are the weights rotated π/ 4 and the log-likelihood is -17.82 nats and the free-energy -57.16 nats.
Revised: Second, here are the weights rotated π/ 4 and the log-likelihood is -17.82 nats and the free-energy -57.16 nats.

Original: [Figure]
Revised: [Figure]

Original: When varying  θ  the plots above indicate that orthogonal settings of the weights ( θ = mπ/ 2 where m = 0 , 1 , 2 , ... ) lead to factorized posteriors.
Revised: When varying the rotation away fom the orthogonal setting, θ , the plots above indicate that orthogonal settings of the weights ( θ = mπ/ 2 where m = 0 , 1 , 2 , ... ) lead to factorized posteriors.

Original: In these cases the KL between the approximate posterior and the true posterior is zero and the free-energy is equal to the log-likelihood.
Revised: In these cases the KL between the approximate posterior and the true posterior is zero and the free-energy is equal to the log-likelihood.

Original: This will be the optimal free-energy for any weight setting (due to the fact that it is equal to the true log-likelihood which is maximal, and the free-energy is a lower bound of this quantity.)
Revised: This will be the optimal free-energy for any weight setting (due to the fact that it is equal to the true log-likelihood which is maximal, and the free-energy is a lower bound of this quantity.)

Original: For intermediate values of θ the posterior is correlated and the free-energy is not tight to the log likelihood.
Revised: For intermediate values of θ the posterior is correlated and the free-energy is not tight to the log likelihood.

Original: Now let’s plot the free-energy and the log-likelihood as θ is varied.
Revised: Now let’s plot the free-energy and the log-likelihood as θ is varied.

Original: This shows that the free-energy prefers orthogonal settings of the weights as this leads to factorized posteriors, even though the log-likelihood is insensitive to θ .
Revised: This shows that the free-energy prefers orthogonal settings of the weights as this leads to factorized posteriors, even though the loglikelihood is insensitive to θ .

Original: So, variational inference recovers the same weight directions as the PCA solution.
Revised: So, variational inference recovers the same weight directions as the PCA solution.

Original: e v i d e n c e dependence of free-energy on weight rotation
Revised: e v i d e n c e dependence of free-energy on weight rotation

Original: ELBO log-likelihood
Revised: ELBO log-likelihood

Original: The above shows that the bias inherent in variational methods will cause them to break the symmetry in the log-likelihood and ﬁnd orthorgonal latent components.
Revised: The above shows that the bias inherent in variational methods will cause them to break the symmetry in the log-likelihood and ﬁnd orthorgonal latent components.

Original: This occurs because orthoginal components result in posterior distributions that are factorized.
Revised: This occurs because orthoginal components result in posterior distributions that are factorized.

Original: These are then well-modelled by the variational approximation and result in a small KL between the approximate and true posteriors.
Revised: These are then well-modelled by the variational approximation and result in a small KL between the approximate and true posteriors.

Original: 1. Experiment 2: mean ﬁeld VI applied to over-complete factor analysis prunes out the additional latent dimensions
Revised: B. E XPERIMENT 2: MEAN FIELD VI APPLIED TO OVER - COMPLETE FACTOR ANALYSIS PRUNES OUT THE ADDITIONAL LATENT DIMENSIONS

Original: A similar eﬀect occurs if we model 2D data with a 3D latent space.
Revised: A similar effect occurs if we model 2D data with a 3D latent space.

Original: Many settings of the weights attain the maximum of the likelihood, including solutions which use all three latent variables.
Revised: Many settings of the weights attain the maximum of the likelihood, including solutions which use all three latent variables.

Original: However, the optimal solution for VI is to retain two orthogonal components and to set the magnitude of the third component to zero.
Revised: However, the optimal solution for VI is to retain two orthogonal components and to set the magnitude of the third component to zero.

Original: This solution a) has weights  maximise the likelihood, and b) has a factorised posterior distribution (the pruned component having a posterior equal to its prior) that therefore incurs no cost KL( q ( z )
Revised: This solution a) returns weights that maximise the likelihood, and b) has a factorised posterior distribution (the pruned component having a posterior equal to its prior) that therefore incurs no cost KL( q ( z )

Original: || p ( z | x , θ ))
Revised: || p ( z | x , θ ))

Original: = 0.
Revised: = 0 .

Original: In this way the bound becomes tight.
Revised: In this way the bound becomes tight.

Original: Here’s an example of this eﬀect.
Revised: Here’s an example of this effect.

Original: We consider a model of the form:
Revised: We consider a model of the form:

Original: [Equation]
Revised: [Equation]

Original: We set α 2 + β 2 = 1 so that all models imply the same covariance and set this to be the maximum likelihood covariance by construction.
Revised: We set α 2 + β 2 = 1 so that all models imply the same covariance and set this to be the maximum likelihood covariance by construction.

Original: We then consider varying α from 0 to 1 / 2.
Revised: We then consider varying α from 0 to 1 / 2 .

Original: The setting equal to 0 attains the maximum of the free-energy, even though it has the same likelihood as any other setting.
Revised: The setting equal to 0 attains the maximum of the free-energy, even though it has the same likelihood as any other setting.

Original: [Figure]
Revised: [Figure]

Original:  i [Figure] n c e
Revised: e v i d e n c e

Original: [Figure]
Revised: [Figure]

Original: Experiment 3: The β -VAE also yields the PCA components, changing β has no eﬀect on the direction of
Revised: E XPERIMENT 3: T HE β -VAE ALSO YIELDS THE PCA COMPONENTS , CHANGING β HAS NO EFFECT ON THE DIRECTION OF THE ESTIMATED COMPONENTS IN THE FA MODEL

Original: How diﬀerent
Revised: How does the setting of β change things? Here we rerun experiment 1 for different values of β .

Original: [Figure]
Revised: [Table] [Table]

Original: v i

Original: [Figure]
Revised: [Table]

Original: e  n c e
Revised: e v i d e n c e

Original: [Figure]
Revised: [Table]

Original: In this example, changing β in this example just reduces the amplitude of the ﬂuctuations in the free-energy, but it does not change the directions found.
Revised: In this example, changing β in this example just reduces the amplitude of the ﬂuctuations in the free-energy, but it does not change the directions found.

Original: A similar observation applies to the pruning experiment.
Revised: A similar observation applies to the pruning experiment.

Original: Increasing β will increase the uncertainty in the posterior as it is like reducing the number of observations (or increasing the observation noise, from the perspective of q ).
Revised: Increasing β will increase the uncertainty in the posterior as it is like reducing the number of observations (or increasing the observation noise, from the perspective of q ).

Original: 1. Summary of Factor Analysis Experiments
Revised: B. S UMMARY OF F ACTOR A NALYSIS E XPERIMENTS

Original: The behaviours introduced by the β -VAE appear relatively benign, and perhaps even helpful, in the linear case: VI is breaking the degeneracy of the maximum likelihood solution in a sensible way: selecting amongst the maximum likelihood solutions to ﬁnd those that have orthogonal components and removing spurious latent dimensions.
Revised: The behaviours introduced by the β -VAE appear relatively benign, and perhaps even helpful, in the linear case: VI is breaking the degeneracy of the maximum likelihood solution in a sensible way: selecting amongst the maximum likelihood solutions to ﬁnd those that have orthogonal components and removing spurious latent dimensions.

Original: This should be tempered by the fact that the β generalization recovered precisely the same solutions and so it was necessary to obtain the desired behaviour in the PCA case.
Revised: This should be tempered by the fact that the β generalization recovered precisely the same solutions and so it was necessary to obtain the desired behaviour in the PCA case.

Original: Similar eﬀects will occur in deep generative models, not least since these typically also have a Gaussian prior over latent variables, and these latents are initially linearly transformed, thereby resulting in a similar degeneracy to factor analysis.
Revised: Similar effects will occur in deep generative models, not least since these typically also have a Gaussian prior over latent variables, and these latents are initially linearly transformed, thereby resulting in a similar degeneracy to factor analysis.

Original: However, the behaviours above beneﬁted from the fact that maximum-likelihood solutions could be found in which the posterior distribution over latent variables factorized.
Revised: However, the behaviours above beneﬁted from the fact that maximum-likelihood solutions could be found in which the posterior distribution over latent variables factorized.

Original: In real world examples, for example in deep generative models, this will not be case.
Revised: In real world examples, for example in deep generative models, this will not be case.

Original: In such cases, these same eﬀects will cause the variational freeenergy and its β -generalization to **bias the estimated parameters far away from maximum-likelihood settings, toward those settings that imply factorized Gaussian posteriors over the latent variables**.
Revised: In such cases, these same effects will cause the variational free-energy and its β -generalization to bias the estimated parameters far away from maximum-likelihood settings, toward those settings that imply factorized Gaussian posteriors over the latent variables .

Original: 1. Independent Component Analysis
Revised: B. I NDEPENDENT C OMPONENT A NALYSIS

Original: We now apply VI and the β free-energy method to ICA.
Revised: We now apply VI and the β free-energy method to ICA.

Original: We’re interested the properties of the variational objective and the β -VI objective and so we 1.
Revised: We’re interested the properties of the variational objective and the β -VI objective and so we 1.

Original: ﬁt the data using the true generative model to investigate the biases in VI and β -VI 2.
Revised: ﬁt the data using the true generative model to investigate the biases in VI and β -VI 2.

Original: do not use amortized inference, just optimizing the approximating distributions for each data point (this is possible for these small examples)
Revised: do not use amortized inference, just optimizing the approximating distributions for each data point (this is possible for these small examples).

Original: The linear independent component analysis generative model we use is deﬁned as follows.
Revised: The linear independent component analysis generative model we use is deﬁned as follows.

Original: Let x ∈ R L and z ∈ R K .
Revised: Let x ∈ R L and z ∈ R K .

Original: for n = 1 ...N for k = 1 ...K z [Equation] [Equation] ,  v ) , x n ∼ N ( W z  , D ) where D 
Revised: for n = 1 ...N for k = 1 ...K z n,k ∼ Student-t (0 , σ, v ) , x n ∼ N ( W z n , D ) where D = diag ([ σ 21 , ..., σ 2D ])

Original: = [ σ 21 , ..., σ 2D ]

Original: We apply mean-ﬁeld variational inference, q ( z n ) =
Revised: We apply mean-ﬁeld variational inference, q ( z n ) =

Original: (cid:81) k q n,k
Revised: (cid:81) k q n,k

Original: ( z k,d ), and use Gaussian distributions for each factor q n,k ( z n,k ) =
Revised: ( z k,d ) , and use Gaussian distributions for each factor q n,k ( z n,k ) =

Original: N ( z n,k ; µ n,k , σ 2 n,k ).
Revised: N ( z n,k ; µ n,k , σ 2 n,k ) .

Original: The free-energy is computed as follows:
Revised: The free-energy is computed as follows:

Original: The reconstruction term is identical to PCA: an avergage of a quadratic form wrt to a Gaussian, which is analytic.
Revised: The reconstruction term is identical to PCA: an avergage of a quadratic form wrt to a Gaussian, which is analytic.

Original: The KL is broken down into the diﬀerential entropy of q which is also analytic and the cross-entropy with the prior which we evaluate by numerical integration (ﬁnite diﬀerences).
Revised: The KL is broken down into the differential entropy of q which is also analytic and the cross-entropy with the prior which we evaluate by numerical integration (ﬁnite differences).

Original: There is a cross-entropy term for each latent variable which is one reason why the code is slow (requiring N 1D numerical integrations).
Revised: There is a cross-entropy term for each latent variable which is one reason why the code is slow (requiring N 1D numerical integrations).

Original: The gradient of the free-energy wrt the parameters W and the means and variances of the Gaussian q distributions are computed using autograd.
Revised: The gradient of the free-energy wrt the parameters W and the means and variances of the Gaussian q distributions are computed using autograd.

Original: In order to be as certain as possible that we are ﬁnding a global maximum of the free-energies, all experiments initialise at the true value of the parameters and then ensure that each gradient step improves the free-energy.
Revised: In order to be as certain as possible that we are ﬁnding a global maximum of the free-energies, all experiments initialise at the true value of the parameters and then ensure that each gradient step improves the free-energy.

Original: Stochastic optimization or a procedure that accepted all steps regardless of the change in the objective would be faster, but they might also move us into the basis of attraction of a worse (local) optima.
Revised: Stochastic optimization or a procedure that accepted all steps regardless of the change in the objective would be faster, but they might also move us into the basis of attraction of a worse (local) optima.

Original: 1. Experiment 1: Learning in over-complete ICA
Revised: B. E XPERIMENT 1: L EARNING IN OVER - COMPLETE ICA

Original: Now we deﬁne the dataset.
Revised: Now we deﬁne the dataset.

Original: We use a very sparse Student’s t-distribtion with v = 3 . 5.
Revised: We use a very sparse Student’s t-distribtion with v = 3 . 5 .

Original: For v < 4 the the kurtosis is undeﬁned so the model is fairly simple to estimate model (it’s a long way away from the degenerate factor analysis case which is recovered in the limit v → ∞ ).
Revised: For v < 4 the the kurtosis is undeﬁned so the model is fairly simple to estimate model (it’s a long way away from the degenerate factor analysis case which is recovered in the limit v → ∞ ).

Original: We use three latent components and a two dimensional observed space.
Revised: We use three latent components and a two dimensional observed space.

Original: The directions of thethree weights are shown in blue below with data as blue circles.
Revised: The directions of the

Original: [Figure]
Revised: [Figure]

Original: First we run variational inference ﬁnding components (shown in red below) which are more orthogonal than the true directions.
Revised: First we run variational inference ﬁnding components (shown in red below) which are more orthogonal than the true directions.

Original: This bias is in this directions as this reduces the dependencies (explaining away) in the underlying posterior.
Revised: This bias is in this directions as this reduces the dependencies (explaining away) in the underlying posterior.

Original: [Figure]
Revised: [Figure]

Original: Second we run β -VI with β = 5.
Revised: Second we run β -VI with β = 5 .

Original: Two components are now found that are orthogonal with one component pruned from the solution.
Revised: Two components are now found that are orthogonal with one component pruned from the solution.

Original: [Figure]
Revised: [Figure]

Original: In this case the bias is so great that the true component directions are not discovered.
Revised: In this case the bias is so great that the true component directions are not discovered.

Original: Instead the components are forced into the orthogonal setting regardless of the structure in the data.
Revised: Instead the components are forced into the orthogonal setting regardless of the structure in the data.

Original: Summary of ICA experiment
Revised: B. S UMMARY OF I NDEPENDENT C OMPONENT A NALYSIS EXPERIMENT

Original: The ICA example illustrates that this approach – of relying on a bias inherent in VI to discover meaningful components – will sometimes return meaningful structure (e.g. in the PCA experiments above).
Revised: The ICA example illustrates that this approach – of relying on a bias inherent in VI to discover meaningful components – will sometimes return meaningful structure (e.g. in the PCA experiments above).

Original: However it does not seem to be a sensible way of doing so in general.
Revised: However it does not seem to be a sensible way of doing so in general.

Original: For example, explaining away often means that the true components will be entangled in the posterior, as is the case in the ICA example, and the variational bias will then move us away from this solution. 
Revised: For example, explaining away often means that the true components will be entangled in the posterior, as is the case in the ICA example, and the variational bias will then move us away from this solution. The β -VI generalisation only enhances this undesirable bias.