Original: D ISENTANGLEMENT BY N ONLINEAR ICA WITH G ENERAL I NCOMPRESSIBLE - FLOW N ETWORKS (GIN)
Revised: D ISENTANGLEMENT BY N ONLINEAR ICA WITH G ENERAL I NCOMPRESSIBLE - FLOW N ETWORKS (GIN)

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process.
Revised: A central question of representation learning asks under which conditions it is possible to reconstruct the true latent variables of an arbitrarily complex generative process.

Original: Recent breakthrough work by Khemakhem et al.
Revised: Recent breakthrough work by Khemakhem et al.

Original: on nonlinear
Revised: on nonlinear

Original: ICA has answered this question for a broad class of conditional generative processes.
Revised: ICA has answered this question for a broad class of conditional generative processes.

Original: We extend this important result in a direction relevant for application to real-world data.
Revised: We extend this important result in a direction relevant for application to real-world data.

Original: First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in some special (but not very restrictive) cases, informative latent variables will be automatically separated from noise by an estimating model.
Revised: First, we generalize the theory to the case of unknown intrinsic problem dimension and prove that in some special (but not very restrictive) cases, informative latent variables will be automatically separated from noise by an estimating model.

Original: Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating process, up to a trivial component-wise transformation.
Revised: Furthermore, the recovered informative latent variables will be in one-to-one correspondence with the true latent variables of the generating process, up to a trivial component-wise transformation.

Original: Second, we introduce a modiﬁcation of the RealNVP invertible neural network architecture (Dinh et al. (2016)) which is particularly suitable for this type of problem: the General Incompressibleﬂow Network (GIN).
Revised: Second, we introduce a modiﬁcation of the RealNVP invertible neural network architecture (Dinh et al., 2016) which is particularly suitable for this type of problem: the General Incompressibleﬂow Network (GIN).

Original: Experiments on artiﬁcial data and EMNIST demonstrate that theoretical predictions are indeed veriﬁed in practice.
Revised: Experiments on artiﬁcial data and EMNIST demonstrate that theoretical predictions are indeed veriﬁed in practice.

Original: In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.
Revised: In particular, we provide a detailed set of exactly 22 informative latent variables extracted from EMNIST.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep latent-variable models promise to unlock the key factors of variation within a dataset, opening a window to interpretation and granting the power to manipulate data in an intuitive fashion.
Revised: Deep latent-variable models promise to unlock the key factors of variation within a dataset, opening a window to interpretation and granting the power to manipulate data in an intuitive fashion.

Original: The theory of identiﬁability in linear independent component analysis (ICA) (Comon (1994)) tells us when this is possible, if we restrict the model to asimple linear transformation, but until recently there was no corresponding theory for the highly nonlinear models needed to manipulate complex data.
Revised: The theory of identiﬁability in linear independent component analysis (ICA) (Comon, 1994) tells us when this is possible, if we restrict the model to a linear transformation, but until recently there was no corresponding theory for the highly nonlinear models needed to manipulate complex data.

Original: We can often successfully approximate the marginal distribution of observed data with such models, but there were no theoretical guarantees of them also being able to approximate the distribution of the generating latent space.

Original: This changed with the recent breakthrough work by Khemakhem et al.
Revised: This changed with the recent breakthrough work by Khemakhem et al.

Original: (2019), which showed that under relatively mild conditions, it is possible to recover the joint data and latent space distribution, up to a simple transformation in the latent space.
Revised: (2019), which showed that under relatively mild conditions, it is possible to recover the joint data and latent space distribution, up to a simple transformation in the latent space.

Original: The key requirement is that the generating process is conditioned on a variable which is observed along with the data.
Revised: The key requirement is that the generating process is conditioned on a variable which is observed along with the data.

Original: This condition could be a class label, time index of a time series, or any other piece of information related to thegiven data.
Revised: This condition could be a class label, time index of a time series, or any other piece of information additional to the data.

Original: They interpret their theory as a nonlinear version of ICA.
Revised: They interpret their theory as a nonlinear version of ICA.

Original: This work extends this theory in a direction relevant for application to real-world data.
Revised: This work extends this theory in a direction relevant for application to real-world data.

Original: The existing theory assumes knowledge of the intrinsic problem dimension, but this is unrealistic for anything but artiﬁcially generated datasets.
Revised: The existing theory assumes knowledge of the intrinsic problem dimension, but this is unrealistic for anything but artiﬁcially generated datasets.

Original: Here, we show that in the special case of Gaussian latent space distributions, the intrinsic problem dimension can be discovered .
Revised: Here, we show that in the special case of Gaussian latent space distributions, the intrinsic problem dimension can be discovered .

Original: The important latent variables are organically separated from noise variables by the estimating model.
Revised: The important latent variables are organically separated from noise variables by the estimating model.

Original: Furthermore, the variables discovered correspond to the true generating latent variables, up  a trivial component-wise translation and scaling.
Revised: Furthermore, the variables discovered correspond to the true generating latent variables, up to a trivial component-wise translation and scaling.

Original: Very similar results exist for other members of the exponential family with two parameters, such as the beta and gamma distributions.
Revised: Very similar results exist for other members of the exponential family with two parameters, such as the beta and gamma distributions.

Original: We introduce a variant of the RealNVPinvertible neural network (Dinh et al.
Revised: We introduce a variant of the RealNVP (Dinh et al., 2016) invertible neural network: the General

Original: (2016)): the General Incompressible-ﬂow Network (GIN).
Revised: Incompressible-ﬂow Network (GIN).

Original: The ﬂow is called incompressible in reference to ﬂuid dynamics, since it preserves volumes: the Jacobian determinant is simply unity.
Revised: The ﬂow is called incompressible in reference to ﬂuid dynamics, since it preserves volumes: the Jacobian determinant is simply unity.

Original: We emphasise its generality and increased expressive power in comparison to previous volume-preserving ﬂows, such as NICE (Dinh et al.
Revised: We emphasise its generality and increased expressive power in comparison to previous volume-preserving ﬂows, such as NICE (Dinh et al., 2014).

Original: As already noted in Khemakhem et al.
Revised: As already noted in Khemakhem et al.

Original: (2019), ﬂow-based generative models are a natural ﬁt for the theory of nonlinear ICA, as are the variational autoencoders (VAEs) (Kingma & Welling (2013)) used in that work.
Revised: (2019), ﬂow-based generative models are a natural ﬁt for the theory of nonlinear ICA, as are the variational autoencoders (VAEs) (Kingma & Welling, 2013) used in that work.

Original: For us, major advantages ofan invertible architecture over VAEs are the ability to  directly optimize the likelihood, and freedom from the requirement to specify the dimension of the model’s latent space.
Revised: For us, major advantages of invertible architectures over VAEs are the ability to specify volume preservation and directly optimize the likelihood, and freedom from the requirement to specify the dimension of the model’s latent space.

Original: An INN always has a latent space of the same dimension as the data.
Revised: An INN always has a latent space of the same dimension as the data.

Original: In addition, the forward and backward models share parameters, saving the effort of learning separate, but complementary models for each direction.
Revised: In addition, the forward and backward models share parameters, saving the effort of learning separate models for each direction.

Original: In summary, our work makes the following contributions:
Revised: In summary, our work makes the following contributions:

Original: • We extend the theory of nonlinear ICA to allow for unknown intrinsic problem dimension.
Revised: • We extend the theory of nonlinear ICA to allow for unknown intrinsic problem dimension.

Original: Doing so, we ﬁnd that this dimension can be discovered and a one-to-one correspondence between generating and estimated latent variables established.
Revised: Doing so, we ﬁnd that this dimension can be discovered and a one-to-one correspondence between generating and estimated latent variables established.

Original: • We propose as an implementation an invertible neural network obtained by modifying the RealNVP architecture.
Revised: • We propose as an implementation an invertible neural network obtained by modifying the RealNVP architecture.

Original: We call our new architecture GIN: the General Incompressible-ﬂow Network.
Revised: We call our new architecture GIN: the General Incompressible-ﬂow Network.

Original: • We demonstrate the viability of the model on artiﬁcial data and the EMNIST dataset.
Revised: • We demonstrate the viability of the model on artiﬁcial data and the EMNIST dataset.

Original: We extractexactly 22 meaningful variables from EMNIST, encoding both global and local features.
Revised: We extract 22 meaningful variables from EMNIST, encoding both global and local features.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The basic goals of nonlinear ICA stem from the original work on linear ICA.
Revised: The basic goals of nonlinear ICA stem from the original work on linear ICA.

Original: An inﬂuential formulation, as well as the ﬁrst identiﬁability results, were given in Comon (1994).
Revised: An inﬂuential formulation, as well as the ﬁrst identiﬁability results, were given in Comon (1994).

Original: These stated the conditions which allow the generating latent variables to be discovered, when the mixing function is a linear transformation.
Revised: These stated the conditions which allow the generating latent variables to be discovered, when the mixing function is a linear transformation.

Original: However, it was shown in Hyv ¨ arinen & Pajunen (1999) that this  identiﬁability does not extend to general nonlinear functions.
Revised: However, it was shown in Hyv¨arinen & Pajunen (1999) that this approach to identiﬁability does not extend to general nonlinear functions.

Original: The ﬁrst identiﬁability results in nonlinear ICA came in Hyv ¨ arinen & Morioka (2016) and Hyv ¨ arinen & Morioka (2017), applied to time series, and implemented via a discriminative model and semisupervised learning.
Revised: The ﬁrst identiﬁability results in nonlinear ICA came in Hyv ¨ arinen & Morioka (2016) and Hyv ¨ arinen & Morioka (2017), applied to time series, and implemented via a discriminative model and semisupervised learning.

Original: A more general formulation, valid for other forms of data, was given in
Revised: A more general formulation, valid for other forms of data, was given in

Original: Hyv ¨ arinen et al.
Revised: Hyv¨arinen et al.

Original: and the theory was extended to generative models in Khemakhem et al.
Revised: and the theory was extended to generative models in Khemakhem et al.

Original: (2019), where experiments were implemented by a VAE.
Revised: (2019), where experiments were implemented by a VAE.

Original: Many authors have addressed the general problem of disentanglement , and proposed models to learn disentangled features.
Revised: Many authors have addressed the general problem of disentanglement , and proposed models to learn disentangled features.

Original: Prominent among these is β -VAE (Higgins et al. and its variations (e.g. Chen et al. (2018)) which augment the standard ELBO loss with tunable hyperparameters to encourage disentanglement.
Revised: Prominent among these is β -VAE (Higgins et al., 2017) and its variations (e.g. Chen et al., 2018) which augment the standard ELBO loss with tunable hyperparameters to encourage disentanglement.

Original: There are also attempts to modify the GAN framework (Goodfellow et al. such as InfoGAN (Chen et al. (2016)), which tries to maximize the mutual information between some dimensions of the latent space and the observed data.
Revised: There are also attempts to modify the GAN framework (Goodfellow et al., 2014) such as InfoGAN (Chen et al., 2016), which tries to maximize the mutual information between some dimensions of the latent space and the observed data.

Original: Many of these approaches are unsupervised.
Revised: Many of these approaches are unsupervised.

Original: However, as pointed out and empirically demonstrated in Locatello et al.
Revised: However, as pointed out and empirically demonstrated in Locatello et al.

Original: (2018), unsupervised models  are in general unidentiﬁable.
Revised: (2018), unsupervised models without conditioning in the latent space are in general unidentiﬁable.

Revised: Several unsupervised VAE models implement conditioning in the latent space by means of Gaussian mixtures (Johnson et al., 2016;

Revised: Dilokthanakul et al., 2016; Zhao et al., 2019).

Original: This is because they do not take into account conditioning in the generating latent space, made clear in Khemakhem et al. 
Revised: Our work differs mainly by (i) only considering supervised tasks, therefore being safely covered by the theory of Khemakhem et al. (2019), and (ii) enforcing volume-preservation, not possible in a VAE.

Original: The invertible neural networks in this work build upon the NICE framework (Dinh et al. and its extension in RealNVP (Dinh et al.
Revised: The invertible neural networks in this work build upon the NICE framework (Dinh et al., 2014) and its extension in RealNVP (Dinh et al., 2016).

Original: A similar network design to ours is Ardizzone et al.
Revised: A similar network design to ours is Ardizzone et al.

Original: This is a conditioned INN based on RealNVP, however the conditioning information is applied as a parameter of the network.
Revised: This is a conditioned INN based on RealNVP, however the conditioning information is applied as a parameter of the network.

Original: The authors ﬁnd in experiments with MNIST that meaningful variables are present in their latent space, but are rotated such that they are not aligned with the axes of the space.
Revised: The authors ﬁnd in experiments with MNIST that meaningful variables are present in their latent space, but are rotated such that they are not aligned with the axes of the space.

Original: In this work, the conditioning is only present as a parameter of the latent space distributions.
Revised: In this work, the conditioning is only present as a parameter of the latent space distributions.

Original: As a result, it is covered by the theory of Khemakhem et al.
Revised: As a result, it is covered by the theory of Khemakhem et al.

Original: (2019) and its extension here, which results in non-rotated, meaningful latent variables.
Revised: (2019) and its extension here, which results in non-rotated, meaningful latent variables.

Original:  T HEORY 3. E XISTING T HEORY OF N ONLINEAR ICA
Revised:  T HEORY 3. E XISTING T HEORY OF N ONLINEAR ICA

Original: This section reviews the main results from Khemakhem et al.
Revised: This section adapts theoretical results from Khemakhem et al.

Revised: to the context of invertible neural networks.

Original: Suppose the existence of the following three random variables: a latent generating variable z ∈ R n , a condition u ∈ R m and a data point x ∈ R d , where n ≤ d . 
Revised: Suppose the existence of the following three random variables: a latent generating variable z ∈ R n , a condition u ∈ R m and a data point x ∈ R d , where n ≤ d . If n < d , also suppose 𝐱 𝐮 𝒑 ( 𝐳 │ 𝐮 ) 𝒇 ( 𝐳 ) 𝐳 𝒈 –1 ( 𝐱 ; 𝛉 ) 𝐰 OBSERVEDDATA AND

Revised: CONDITION

Revised: GENERATINGMODEL

Original: The distribution of z is a factorial member of the exponential family with k sufﬁcient statistics, conditioned on u.
Revised: ESTIMATINGMODEL [Figure] DISTRIBUTION GENERATINGLATENTVARIABLE

Revised: ESTIMATEDLATENTVARIABLE 𝓛 ( 𝐰 , 𝐮 )

Revised: LOSS 𝛆 the existence of a noise variable ε ∈ R d − n .

Revised: The variables z and ε make up the generating latent space, and are not necessarily known, whereas u and x are observable and therefore always known.

Revised: The distribution of z is a factorial member of the exponential family with k sufﬁcient statistics, conditioned on u.

Original: In its most general form the distribution can be written as
Revised: In its most general form the distribution can be written as

Original: [Equation]
Revised: [Equation]

Original: where the T i,j are the sufﬁcient statistics, the λ i,j their coefﬁcients and Z i the normalizing constant.
Revised: where the T i,j are the sufﬁcient statistics, the λ i,j their coefﬁcients and Z i the normalizing constant.

Original: Q i is called the base measure, which in many cases is simply 1.
Revised: Q i is called the base measure, which in many cases is simply 1.

Original: Explicitly applying (1) to a Gaussian with diagonal covariance: [Equation]
Revised: The distribution of ε must not depend on u and its probability density function must be always ﬁnite.

Original: so we can write [Equation] The variable x is the result of an arbitrarily complex, injective transformation from the generating latent space to the data space: x = f ( z  ) where f is not necessarily deterministic.
Revised: The variable x is the result of an arbitrarily complex, invertible, deterministic transformation from the generating latent space to the data space: x = f ( z , ε ) . This can alternatively be formulated as an injective, non-deterministic transformation from the lower-dimensional z-space to the higherdimensional x-space: x = f ε ( z ) .

Original: In general, an observed dataset D will consist only of instances of x and u. 
Revised: In general, an observed dataset D will consist only of instances of x and u. The task of nonlinear

Original: The task of nonlinear ICA is tounmix or disentangle the data to recover the generating latent variables z, as well as the form of the function f and its inverse. 
Revised: ICA is to disentangle the data to recover the generating latent variables z, as well as the form of the function f and its inverse. We can try to achieve this with a sufﬁciently general, invertible function approximator g , which maps from a latent variable w ∈ R d to the data space: x = g ( w ; θ ) . Here θ denotes the parameters of g . Note that the dimensions of the latent space and data space are the same due to the invertibility of g .

Original:  In addition to the above speciﬁcation of z, u and x, some conditions are necessary to ensure the latent variables can be recovered.
Revised: We assume that w follows a conditionally independent exponential probability distribution, conditioned on the same condition u as z: [Equation] The coefﬁcients of the sufﬁcient statistics λ (cid:48) i,j ( u ) are not restricted, which means that a given variable i of the estimated latent space is allowed to lose its dependence on u. If this occurs, we will consider this variable to encode noise, playing a role equivalent to ε in the generating latent space. In addition to this speciﬁcation of the generative and estimating models, some conditions are necessary to ensure the latent variables can be recovered.

Original: The most important of these concerns the variability of λ i,j ( u ) under u.
Revised: The most important of these concerns the variability of λ i,j ( u ) under u.

Original: However, as long as the λ i,j are randomly and independently generated, and there are at least nk + 1 distinct conditions u, this condition is almost surely fulﬁlled.
Revised: However, as long as the λ i,j are randomly and independently generated, and there are at least nk + 1 distinct conditions u, this condition is almost surely fulﬁlled.

Original: More details can be found in Khemakhem et al.
Revised: See Appendix A.1 for further details.

Original: If the necessary conditions are met, the unmixing can be achieved with a sufﬁciently general, invertible function approximator g , for example a VAE or a ﬂow-based model. A variable w sampled from the latent space of this model must follow a conditionally independent exponential probability distribution, conditioned on the same condition u as z:
Revised: In the limit of inﬁnite data and perfect convergence, the estimating model will give the same conditional likelihood to all data points as the true generating model:

Original: [Equation]
Revised: [Equation]

Original: Note that k and n must be the same as for the unknown latent space.

Original: In the limit of inﬁnite data and perfect convergence, the estimating model will be able to exactly reproduce the distribution of x: p f, z ( x | u ) =

Original: p g, w ( x | u ) .

Original: In this  case, the vector of sufﬁcient statistics(cid:101) T from the generating latent space will be related to that of the estimating latent space by an afﬁne transformation: 
Revised: If this is the case, the vector of sufﬁcient statistics T from the generating latent space will be related to that of the estimated latent space by an afﬁne transformation: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where A is some constant, invertible, square matrix and c  some constant vector.
Revised: where A is some constant, full-rank, nk × dk matrix and c ∈ R nk some constant vector.

Original: The relationship holds for all values of z and w.  This  is the main theoretical result of  Khemakhem et al.(2019), which will be generalized in the following.
Revised: The relationship holds for all values of z and w. The proof of this result can be found in Appendix A. This relationship is a generalization of that derived in Khemakhem et al.

Original: The current theory of identiﬁability in nonlinear ICA assumes that the intrinsic dimension of  thegenerating latent space is known. The dimension of the  latent space of the estimating modelis then assumed to be the same. When dealing with areal-world dataset, the dimension of the latent space is in general unknown.
Revised: In that version, they assume knowledge of n , the dimension of the generating latent space, and give their estimating model a latent space of the same dimension.

Original: Here we show how to expand the theory to this case.

Original: Suppose the dimension of the generating latent space is n and the dimension of the latent space used by the estimating model is n (cid:48) , where n (cid:48) ≥ n .

Original: In this case, under the same conditions required by the theorem that results in equation (6), we can derive avery similar relationship betweenvariables in the generating and estimated latent spaces(see Appendix A). In brief, we maintain the relationship in equation (6), adjusting for the different dimensionality of z and w by making A into a rectangular matrix of size nk × n (cid:48) k . A is no longer necessarily squarebut maintains full rank (with rank nk ). The number of sufﬁcient statistics k per dimension must still be speciﬁed and cannot be inferred.
Revised: In this context the matrix A deﬁnes a relationship between latent spaces of the same dimension, making it square and invertible.

Original: 3. N ONLINEAR ICA WITH A G AUSSIAN LATENT SPACE
Revised: 3. N ONLINEAR ICA WITH A G AUSSIAN LATENT SPACE

Original: Armed with knowledge of the relationship between elements of the true generating latent space and the latent space estimated by the model (equation (6)), we can ask whether any stronger results hold for particular special cases.
Revised: Given the relationship in equation (4), we can ask whether any stronger results hold for particular special cases.

Original: We hope in particular to induce sparsity in the matrix A , so that the estimated latent space is related to the true one by as simple a transformation as possible.
Revised: We hope in particular to induce sparsity in the matrix A , so that the estimated latent space is related to the true one by as simple a transformation as possible.

Original: For the case of a Gaussian  latent space distribution, we do exactly that, demonstrating a result ofequivalent strength to linear ICA. Refer to Appendix B for the proof.
Revised: We show (proof in Appendix B) that when both the generating and estimated latent spaces follow a Gaussian distribution, the generating latent space variables are recovered up to a trivial translation and scaling. Furthermore, the dimension of the generating latent space is recovered.

Original: Our result is that each generating latent variable z i is related to exactly one estimated latent variable w j , for some j , as [Equation]
Revised: More precisely, each generating latent variable z i is related to exactly one estimated latent variable w j , for some j , as z i

Original: [Equation] ) for some constantscaling and translation a i and b i .
Revised: = a i w j + b i , for some constant a i and b i .

Original: Furthermore, each estimated latent variable w j is related to at most one z
Revised: Furthermore, each estimated latent variable w j is related to at most one z

Original: i .
Revised: i .

Original: If the estimating latent space has higher dimension than the dimension of the generating latent space, some estimating latent variables are not related to any generating latent variables and so must encode only noise.
Revised: If the estimating latent space has higher dimension than the dimension of the generating latent space, some estimating latent variables are not related to any generating latent variables and so must encode only noise.

Original: This acts as a dimension discovery mechanism, since the estimating latent space organically splits into informative and non-informative parts, with the dimension of the informative part equal to the unknown intrinsic dimension of the generating latent space.
Revised: This is the dimension discovery mechanism, since the estimated latent space organically splits into informative and non-informative parts, with the dimension of the informative part equal to the unknown intrinsic dimension of the generating latent space.

Original: Very similar results can be derived for all common continuous two-parameter members of the exponential family, such as the gamma and beta distributions.
Revised: Very similar results can be derived for all common continuous two-parameter members of the exponential family, including the gamma and beta distributions.

Original: See Appendix C.
Revised: See Appendix C.

Revised: 3. V OLUME P RESERVATION VIA I NCOMPRESSIBLE F LOW

Revised: In a multivariate Gaussian distribution with diagonal covariance, the standard deviations are directly proportional to the principal axes of the ellipsoid deﬁning a surface of constant probability.

Original: E XPERIMENTS
Revised: In classical PCA, this property is used to assign importance to each principal component based on its standard deviation. In PCA we can think of the rotation of the data into the basis deﬁned by the principal components as the latent space. In a deep latent variable model the transformation between the latent space and the data is signiﬁcantly more complex, but if this transformation preserves volumes like the rotation in PCA, it will retain the desirable correspondence between the standard deviation of a latent space variable and its importance in explaining the data. Because invertible neural networks (normalizing ﬂows) have a tractable Jacobian determinant, they open the possibility to constrain it to unity.

Revised: This is equivalent to volume preservation and is the rationale behind the

Original:  Experiments on artiﬁcial datasets conﬁrm the theory for a normally distributed latent space, as well as identifying potential causes of failure.
Revised: General Incompressible-ﬂow Network.  E XPERIMENTS Experiments on artiﬁcial datasets conﬁrm the theory for a normally distributed latent space, as well as identifying potential causes of failure.

Original: Experiments on the EMNIST dataset (Cohen et al., 2017) demonstrate the ability of GIN to estimate independent and interpretable latent variables from realworld data.
Revised: Experiments on the EMNIST dataset (Cohen et al., 2017) demonstrate the ability of GIN to estimate independent and interpretable latent variables from realworld data.

Original: 4. M ODEL D ESCRIPTION
Revised: 4. M ODEL D ESCRIPTION

Original: The GIN model is similar in form to RealNVP (Dinh et al. and shares its ﬂexibility, but retains the volume-preserving properties of the NICE framework (Dinh et al.
Revised: The GIN model is similar in form to RealNVP (Dinh et al., 2016) and shares its ﬂexibility, but retains the volume-preserving properties of the NICE framework (Dinh et al., 2014).

Original: Preserving volume creates a direct relationship between the standard deviation of the distribution of an estimated latent variable and its importance, as the standard deviation encodes the volume contributed by that variable, which is then preserved by the network.

Original: RealNVP coupling layers split D -dimensional input x into two parts, x 1: d and x d +1: D where d < D .
Revised: RealNVP coupling layers split D -dimensional input x into two parts, x 1: d and x d +1: D where d < D .

Original: The output of the layer is the concatentation of y 1: d and y d +1:
Revised: The output of the layer is the concatentation of y 1: d and y d +1:

Original: D with y 1: d = x 1: d (8) y d
Revised: D with y 1: d = x 1: d (5) y d

Original: +1: D = x
Revised: +1: D = x

Original: d +1: D (cid:12) exp( s ( x 1: d ))
Revised: d +1: D (cid:12) exp( s ( x 1: d ))

Original: + t ( x 1: d ) (9) where addition, multiplication and exponentiation are applied component-wise.
Revised: + t ( x 1: d ) (6) where addition, multiplication and exponentiation are applied component-wise.

Original: The logarithm of the Jacobian determinant of a coupling layer is simply the sum of the scaling function s ( x 1: d ) .
Revised: The logarithm of the Jacobian determinant of a coupling layer is simply the sum of the scaling function s ( x 1: d ) .

Original: Volume preservation is achieved by setting the ﬁnal component s ( x d )of s to the negative sum of the previous components. Therefore the total sum of s ( x 1: d ) is zero, and the logarithm of the Jacobian determinant is also zero.
Revised: Volume preservation is achieved by setting the ﬁnal component s ( x d ) to the negative sum of the previous components, so the total sum of s ( x 1: d ) is zero (in contrast, NICE sets s ( x 1: d ) ≡ 0 ).

Original: Hence the Jacobian determinant is unity and volume is preserved.
Revised: Hence the Jacobian determinant is unity and volume is preserved.

Original: The network is free to allow the volume contribution of  the outputs of any coupling layer to grow, but only by shrinking the other outputs in direct proportion.
Revised: The network is free to allow the volume contribution of some dimensions of the output of any coupling layer to grow, but only by shrinking the other dimensions of the output in direct proportion.

Original: As well as enforcing a strong correspondence between the importance of a latent variable and its standard deviation, we believe volume-preservation has a regularizing effect, since it is a very strong constraint, comparable to orthogonality in linear transformations.
Revised: As well as enforcing a strong correspondence between the importance of a latent variable and its standard deviation, we believe volume-preservation has a regularizing effect, since it is a very strong constraint, comparable to orthonormality in linear transformations.

Original: 4. O PTIMIZATIONM ETHOD
Revised: 4. O PTIMIZATION

Original: The experiments in this section deal with labeled data, where each data point belongs to one of
Revised: The experiments in this section deal with labeled data, where each data point belongs to one of

Original: M different classes.
Revised: M different classes.

Original: This class label is used as the condition u associated with each data point x. In the estimated latent space, all data instances with the same label should belong to the same
Revised: This class label is used as the condition u associated with each data point x. In the estimated latent space, all data instances with the same label should belong to the same

Original: Gaussian distribution.
Revised: Gaussian distribution.

Original: Hence we are learning a Gaussian mixture in the estimated latent space, with
Revised: Hence we are learning a Gaussian mixture in the estimated latent space, with

Original: M mixture components.
Revised: M mixture components.

Original: Since the distribution for each class in the estimated latent space is required to be factorial, the variance of each mixture component is diagonal, and we can write σ 2 i ( u ) for the variance in the i -th dimension.
Revised: Since the distribution for each class in the estimated latent space is required to be factorial, the variance of each mixture component is diagonal, and we can write σ 2 i ( u ) for the variance in the i -th dimension.

Original: Given a set of data, condition pairs D = { ( x (1) , u (1) ) , . . .
Revised: Given a set of data, condition pairs D = { ( x (1) , u (1) ) , . . .

Original: , ( x ( N ) , u ( N ) ) } and model g , parameterized by θ (where g maps from the latent space to the data space) we can construct a loss from the log-likelihood.
Revised: , ( x ( N ) , u ( N ) ) } and model g , parameterized by θ (where g maps from the latent space to the data space) we can construct a loss from the log-likelihood.

Original: Using the change of variables formula, we have log p ( x | u ) =
Revised: Using the change of variables formula, we have log p ( x | u ) =

Original: log p ( w | u ) , with no Jacobian term since the transformation w = g − 1 ( x ; θ ) is volume-preserving.
Revised: log p ( w | u ) , with no Jacobian term since the transformation w = g − 1 ( x ; θ ) is volume-preserving.

Original: To maximize the likelihood of D , we minimize the negative log-likelihood of w in the estimated latent space (refer to equation (2) for the likelihood of a Gaussian distribution):
Revised: To maximize the likelihood of D , we minimize the negative log-likelihood of w in the estimated latent space:

Original: [Equation]
Revised: [Equation]

Original: Take D to be a mini-batch of data.

Original: The mean and variance of a mixture component is updated at each iteration as the mean and variance of the transformations to latent space of all data points in D belonging to that mixture component:

Original: [Equation]

Original: [Equation]

Original: Hence the parameters of the mixture components change in each batch, according to the data present.

Original: The notation µ i ( u ; θ ) does not indicate that µ is directly parameterized by θ and learned, instead it indicates that µ is a function of g − 1 which is parameterized by θ .

Original: A change in θ will also change the output of µ , given the same mini-batch of data D .

Original: The same holds for σ i ( u ; θ ) .

Original: 4. A RTIFICIAL D ATA 4.3. E XPERIMENT 1: CONDITIONS OF THEORY FULFILLED
Revised: 4. A RTIFICIAL D ATA 4.3. E XPERIMENT 1: CONDITIONS OF THEORY FULFILLED

Original: The other eight are correctly identiﬁed as noise.

Original: The observed data is ten-dimensional (projection into two dimensions shown here).

Original: The spectrum shows the standard deviation of each variable of the reconstruction (in black, log scale) which quantiﬁes its importance.

Original: Ground truth is in gray.

Original: There is a clear distinction between the two informative dimensions and the noise dimensions, showing that

Original: GIN has correctly detected a two-dimensional manifold in the ten-dimensional data presented to it.

Original: Samples are generated in two dimensions, conditioned on ﬁve different cluster labels.
Revised: Samples are generated in two dimensions, conditioned on ﬁve different cluster labels, see Fig.

Original: The means of the clusters are chosen independently from a uniform distribution on [ − 5 , 5] and variances from a uniform distribution on [ . 5 , 3] .
Revised: The means of the clusters are chosen independently from a uniform distribution on [ − 5 , 5] and variances from a uniform distribution on [ . 5 , 3] .

Original: This data is then concatenated with independent Gaussian noise in eight dimensions to make a ten-dimensional generating latent space, where only the ﬁrst two variables are informative.
Revised: This data is then concatenated with independent Gaussian noise in eight dimensions to make a ten-dimensional generating latent space, where only the ﬁrst two variables are informative.

Original: The noise is scaled by 0.01 to be small in comparison to the informative dimensions.
Revised: The noise is scaled by 0.01 to be small in comparison to the informative dimensions.

Original: The latent space samples are then passed through a RealNVP network with 8 fully connected coupling blocks with randomly initialized weights to produce the observed data.
Revised: The latent space samples are then passed through a RealNVP network with 8 fully connected coupling blocks with randomly initialized weights to produce the observed data.

Original: This acts as a highly nonlinear mixing which can only be successfully treated with nonlinear methods.
Revised: This acts as a highly nonlinear mixing which can only be successfully treated with nonlinear methods.

Original: GIN is used as the estimating model, with 8 fully connected coupling blocks (full details in Appendix
Revised: GIN is used as the estimating model, with 8 fully connected coupling blocks (full details in Appendix

Original: D).
Revised: D).

Original: Training converges quickly and stably using the Adam optimizer (Kingma & Ba (2014)) with initial learning rate 10 − 2 and other values set to the usual recommendations.
Revised: Training converges quickly and stably using the Adam optimizer (Kingma & Ba, 2014) with initial learning rate 10 − 2 and other values set to the usual recommendations.

Original: Batch size is 1,000 and the data is augmented with Gaussian noise ( σ = 0 .
Revised: Batch size is 1,000 and the data is augmented with Gaussian noise ( σ = 0 .

Original: at each iteration.
Revised: at each iteration.

Original: After convergence of the loss, the learning rate is reduced by a factor of 10 and trained again until convergence.
Revised: After convergence of the loss, the learning rate is reduced by a factor of 10 and trained again until convergence.

Original: Over a number of experiments we made the following observations:
Revised: Over a number of experiments we made the following observations:

Original: • The model converges stably and gives importance (quantiﬁed by standard deviation) to only two variables in the estimated latent space, provided there is sufﬁcient overlap between the mixture components in the generating latent space.
Revised: • The model converges stably and gives importance (quantiﬁed by standard deviation) to only two variables in the estimated latent space, provided there is sufﬁcient overlap between the mixture components in the generating latent space.

Original: • Where there is not enough overlap in the generating latent space, the model cannot recognize common variables across all the different classes, and tends to split one genuine dimension of variation into two or more in its estimated latent space.
Revised: • Where there is not enough overlap in the generating latent space, the model cannot recognize common variables across all the different classes, and tends to split one genuine dimension of variation into two or more in its estimated latent space.

Original: This appears to be a problem of ﬁnite data.
Revised: This appears to be a problem of ﬁnite data.

Original: We have observed that when this behaviour occurs, and if the gap between mixture components is not too large, it can be prevented by increasing the number of samples so that the space between the mixture components is better ﬁlled (see Figuresand 7 in Appendix E).
Revised: We have observed that when this behaviour occurs, and if the gap between mixture components is not too large, it can be prevented by increasing the number of samples so that the space between the mixture components is better ﬁlled (see Fig. 6 and 7 in Appendix E).

Original: This is consistent with the theory, where equation (6) is true asymptotically.
Revised: This is consistent with the theory, where equation (4) is true asymptotically.

Original: Since the latent space distributions are members of the exponential family, they have support across the entire domain of the latent space, hence gaps can never remain in the limit of inﬁnite samples.
Revised: Since the latent space distributions are members of the exponential family, they have support across the entire domain of the latent space, hence gaps can never remain in the limit of inﬁnite samples.

Original: • Choice of learning rate is important
Revised: • Choice of learning rate is important.

Original: If the initial learning rate is too low, training gets stuck in bad local optima, where single true variables are split into several latent dimensions.
Revised: If the initial learning rate is too low, training gets stuck in bad local optima, where single true variables are split into several latent dimensions.

Original: 4.3. E XPERIMENT 2: CONDITIONS OF THEORY NOT FULFILLED
Revised: 4.3. E XPERIMENT 2: CONDITIONS OF THEORY NOT FULFILLED

Original: Samples are generated as in Experiment 1, but with only three mixture components.
Revised: Samples are generated as in Experiment 1, but with only three mixture components.

Original: Since there are two sufﬁcient statistics per dimension, and two dimensions of variation, according to the theory we need at least nk + 1 = 5 distinct conditions u for equation (6) to hold (see section 3.1).
Revised: Since there are two sufﬁcient statistics per dimension, and two dimensions of variation, according to the theory we need at least nk + 1 = 5 distinct conditions u for equation (4) to hold (see section 3.1).

Original: Therefore, we might not expect successful experiments.
Revised: Therefore, we might not expect successful experiments.

Original: Nonetheless, we observe essentially the same results as for the previous experiments (see Figure 8 in Appendix E), with the same caveats regarding gaps between the mixture components in the generating latent space.
Revised: Nonetheless, we observe essentially the same results as for the previous experiments (see Fig. 8 in Appendix E), with the same caveats regarding gaps between the mixture components in the generating latent space.

Original: This suggests that the conditions derived in Khemakhem et al.
Revised: This suggests that the conditions derived in Khemakhem et al.

Original: (2019), although sufﬁcient for disentanglement, are not necessary.
Revised: (2019), although sufﬁcient for disentanglement, are not necessary.

Original: 4. EMNIST 4.4. E XPERIMENT
Revised: [Figure]

Original: The data comes from the EMNIST Digits training set of 240,000 images of handwritten digits with labels (Cohen et al.
Revised: The data comes from the EMNIST Digits training set of 240,000 images of handwritten digits with labels (Cohen et al., 2017).

Original: EMNIST is a larger version of the well-known MNIST dataset, and also includes handwritten letters.
Revised: EMNIST is a larger version of the well-known MNIST dataset, and also includes handwritten letters.

Original: Here we use only the digits.
Revised: Here we use only the digits.

Original: The digit label is used as the condition u, hence we construct a Gaussian mixture in the estimated latent space with 10 mixture components.
Revised: The digit label is used as the condition u, hence we construct a Gaussian mixture in the estimated latent space with 10 mixture components.

Original: According to the theory (see 3.1), these are only enough conditions to guarantee identiﬁability if there are only four informative latent variables in the generating latent space.
Revised: According to the theory (see 3.1), these are only enough conditions to guarantee identiﬁability if there are only four informative latent variables in the generating latent space.

Original: We expect that the true number of informative variables is somewhat higher, so as in Experiment 2 above, we are operating outside of the guarantees of the theory.
Revised: We expect that the true number of informative variables is somewhat higher, so as in Experiment 2 above, we are operating outside of the guarantees of the theory.

Original: In addition, the true generative process of human handwriting may not exactly fulﬁll our method’s assumptions, and we might still lack data on rare or subtle variations, despite the large size of the dataset in comparison to MNIST.
Revised: In addition, the true generative process of human handwriting may not exactly fulﬁll our method’s assumptions, and we might still lack data on rare or subtle variations, despite the large size of the dataset in comparison to MNIST.

Original: The estimating model is a GIN which uses convolutional coupling blocks and fully connected coupling blocks to transform the data to the latent space (full details in Appendix D).
Revised: The estimating model is a GIN which uses convolutional coupling blocks and fully connected coupling blocks to transform the data to the latent space (full details in Appendix D).

Original: Optimization is with the Adam optimizer, with initial learning rate 3e-4.
Revised: Optimization is with the Adam optimizer, with initial learning rate 3e-4.

Original: Batch size is 240 and the data is augmented with Gaussian noise ( σ = 0 .
Revised: Batch size is 240 and the data is augmented with Gaussian noise ( σ = 0 .

Original: at each iteration.
Revised: at each iteration.

Original: The model is trained for 45 epochs, then for a further 50 epochs with the learning rate reduced by a factor of 
Revised: The model is trained for 45 epochs, then for a further 50 epochs with the learning rate reduced by a factor of 10.

Original: 4.4. R 
Revised: 4.4. R ESULTS

Original: [Figure]
Revised: [Figure]

Original:  Appendix F.
Revised: cases is somewhat entangled with slant. Variable 3 controls the height and variable 4 controls how bent the digit is. Full set of variables for all digits in Appendix F.

Original: [Figure]
Revised: [Figure]

Original: The model encodes information into 22 broadly interpretable latent space variables, with the remainder adding small amounts of noise.
Revised: The model encodes information into 22 broadly interpretable latent space variables.

Original: The effects of each of the interpretable variables can be seen in

Original: Appendix F. The eight variables with the highest standard deviation encode global information, such as slant, line thickness and width which affect all digits more or less equally.
Revised: The eight variables with the highest standard deviation encode global information, such as slant, line thickness and width which affect all digits more or less equally.

Original: The remaining  meaningful variables encode more local information, which does not affect all digits equally. 
Revised: The remaining 14 meaningful variables encode more local information, which does not affect all digits equally. We interpret all other variables as encoding noise. This is motivated mainly by our observation of an abrupt end to any observable change in the reconstructed digits when any variable after the ﬁrst 22 is changed. This can be seen clearly in Fig.

Revised: 15 in Appendix F.

Original: Some variables, particularly the global ones, are not entirely disentangled from another.
Revised: Some variables, particularly the global ones, are not entirely disentangled from another.

Original: Nevertheless, the results are compelling and suggest that the assumptions required by the theory are approximately met.
Revised: Nevertheless, the results are compelling and suggest that the assumptions required by the theory are approximately met.

Original: We observed some global features which are not usually seen in disentanglement experiments on
Revised: We observed some global features which are not usually seen in disentanglement experiments on

Original: MNIST (e.g. Chen et al. (2016), Dupont (2018)).
Revised: MNIST (e.g. Chen et al., 2016; Dupont, 2018).

Original: These experiments usually obtain digit slant, width and line thickness as the major global independent variables.
Revised: These experiments usually obtain digit slant, width and line thickness as the major global independent variables.

Original: We too observe slant and line thickness as independent variables (variables 2 and 5), but ﬁnd width to be split into two variables, one governing the width of the upper half of a digit (variable 1) and the other the lower half (variable 8).
Revised: We too observe slant and line thickness as independent variables (variables 2 and 5), but ﬁnd width to be split into two variables, one governing the width of the upper half of a digit (variable 1) and the other the lower half (variable 8).

Original: This makes sense, since these can in fact vary independently.
Revised: This makes sense, since these can in fact vary independently.

Original: We also observe the height of a digit (variable 3) (see Figure 4).
Revised: We also observe the height of a digit (variable 3) (see Fig. 4).

Original: This does not usually appear in disentanglement experiments, possibly because it is too subtle a variation for those experiments, but possibly because it is not present or not discoverable in the smaller MNIST dataset but is in EMNIST.
Revised: This does not usually appear in disentanglement experiments, possibly because it is too subtle a variation for those experiments, but possibly because it is not present or not discoverable in the smaller MNIST dataset but is in EMNIST.

Original: Local features are also usually not observed in such experiments, so the variables which control these are particularly interesting.
Revised: Local features are also usually not observed in such experiments, so the variables which control these are particularly interesting.

Original: These variables modify only a region of the digit, leaving the rest of the digit untouched.
Revised: These variables modify only a region of the digit, leaving the rest of the digit untouched.

Original: In addition, digits which do not have the feature which is being modiﬁed in that region are left alone.
Revised: In addition, digits which do not have the feature which is being modiﬁed in that region are left alone.

Original: Examples include variable 13, which changes the orientation of the top-left stroke in 2, 3 and 7, and variable 16, which modiﬁes only the lower-right stroke of a 2 (see Figure 5).
Revised: Examples include variable 13, which changes the orientation of the top-left stroke in 2, 3 and 7, and variable 16, which modiﬁes only the lower-right stroke of a 2 (see Fig. 5).

Original: The full set of local and global variables can be seen in  Appendix F.
Revised: The full set of local and global variables can be seen in Fig. 10 to Fig. 15 in Appendix F.

Original: We can compare the latent space representations of digits from different classes by normalizing the latent space values with the mean and variance of the class the digit belongs to. To do so, we calculate the mean and variance in the latent space of some set of digits which all belong to  the same class.
Revised: The concept of a “true generative process” rests on the assumption that conditioning variables u act as causes for the latent variables z, such that the elements of the latter become independent given the former. In the case of handwritten digits, u represents a person’s decision to write a particular digit, and z determines the hand motions to produce this digit.

Original: We used a subset of the EMNIST test dataset of 40,000 images for this purpose. We can then transfer a normalized value into the distribution ofanother digit as a method ofchanging the  conditioning while retaining information about style. Samples generated with different conditions (class labels) but a consistent style can be found in Appendix F.
Revised: Modelling u in terms of digit labels is a plausible proxy for the unknown causal variables in the human brain, and the success of the resulting model suggests that the assumptions are approximately fulﬁlled. Identifying promising conditioning variables u in less obvious situations is an important open problem for future work.

Original: 4.4. P OTENTIAL I MPROVEMENTS
Revised: 4.4. P OTENTIAL I MPROVEMENTS

Original: Increasing the number of conditioning variables would bring this work closer in line with the existing theory.
Revised: Increasing the number of conditioning variables would bring this work closer in line with the existing theory.

Original: At the current value of ten conditioning variables, the theory only applies if there are only four informative generating latent variables, which is almost certainly too low a number.
Revised: At the current value of ten, the theory only applies if there are at most four informative generating latent variables, which is almost certainly too low a number.

Original: One option to increase the number of conditions is to relax the labels from hard to soft, i.e. compute posterior probabilities of class membership given a data example.
Revised: One option to increase the number of conditions is to relax the labels from hard to soft, i.e. compute posterior probabilities of class membership given a data example.

Original: This could be achieved by information distillation as in Hinton et al.
Revised: This could be achieved by information distillation as in Hinton et al.

Original: By doing so, important variations within a class could be encoded, such as the crossbar on a 7.
Revised: Another possibility is to split existing labels into sub-labels, for example making a distinction between sevens with and without crossbars.

Original: Although there are many examples in the dataset of the digit 7 with a crossbar, GIN has apparently treated it as an anomalous feature. As a result, samples never show a 7 with a crossbar (see Appendix F). With a soft label, a 7 with a crossbar could be grouped with similar examples, allowing the crossbar to be expressed when the appropriate condition is selected. However, allowing class labels to become continuous requires non-trivial modiﬁcations to the optimization method used in this work (Sec. 4.2).
Revised: This may also aid the generation of more realistic samples, since we observed no sevens with crossbars in our generated samples, even though they make up approximately 5% of the sevens in the dataset.

Original:  C ONCLUSION AND O UTLOOK
Revised:  C ONCLUSION AND O UTLOOK

Original: We have expanded the theory of nonlinear ICA to cover problems with unknown intrinsic dimension and demonstrated an implementation with GIN, a new volume-preserving modiﬁcation of the
Revised: We have expanded the theory of nonlinear ICA to cover problems with unknown intrinsic dimension and demonstrated an implementation with GIN, a new volume-preserving modiﬁcation of the

Original: RealNVP invertible network architecture.
Revised: RealNVP invertible network architecture.

Original: The variables discovered by GIN in EMNIST are interpretable and detailed enough to suggest that the assumptions made about the generating process are approximately true.
Revised: The variables discovered by GIN in EMNIST are interpretable and detailed enough to suggest that the assumptions made about the generating process are approximately true.

Original: Furthermore, our experiments with EMNIST demonstrate the viability of applying models inspired by the new theory of nonlinear ICA to real-world data, even when not all of the conditions of the theory are met.
Revised: Furthermore, our experiments with EMNIST demonstrate the viability of applying models inspired by the new theory of nonlinear ICA to real-world data, even when not all of the conditions of the theory are met.

Original: It is not clear if the methods from this work will scale to larger problems.
Revised: It is not clear if the methods from this work will scale to larger problems.

Original: However, given the recent advances of similar ﬂow-based generative models in density estimation on larger datasets such as
Revised: However, given the recent advances of similar ﬂow-based generative models in density estimation on larger datasets such as

Original: CelebA and ImageNet (e.g. Kingma & Dhariwal (2018), Ho et al. , it is a plausible prospect.
Revised: CelebA and ImageNet (e.g. Kingma & Dhariwal, 2018; Ho et al., 2019), it is a plausible prospect.

Original: More investigation is necessary in this direction.
Revised: In addition, it is not clear whether the method can successfully be extended to the context of semisupervised learning, or ultimately, unsupervised learning. S UPPLEMENTARY M ATERIAL A P ROOF OF IDENTIFIABILITY This section reproduces a proof from Khemakhem et al. (2019) with some modiﬁcations to adapt it to the context of invertible neural networks (normalizing ﬂows).

Original: S UPPLEMENTARY M ATERIAL A I DENTIFIABILITY THEORY WITH LATENT SPACES OF DIFFERENT DIMENSION
Revised: Deﬁne the domain of f ε as Z = Z 1 × · · · × Z n . Deﬁne the vector of sufﬁcient statistics T ( z ) =

Original: We need to prove that
Revised: ( T 1 , 1 ( z 1 ) , . . . , T n,k ( z n )) and the vector of their coefﬁcients λ ( u ) = ( λ 1 , 1 ( u ) , . . . , λ n,k ( u )) .

Revised: Theorem 1 Assume we observe data distributed according to the generative model deﬁned in Sec.

Revised: Further suppose the following:

Revised: (i)

Revised: The sufﬁcient statistics T i,j ( z ) are differentiable almost everywhere and their derivatives d T i,j ( z ) /

Revised: d z are nonzero almost surely for all z ∈ Z i and all 1 ≤ i ≤ n and 1 ≤ j ≤ k .

Revised: (ii)

Revised: There exist nk + 1 distinct conditions u 0 , . . .

Revised: , u nk such that the matrix

Revised: [Equation]

Revised: of size

Revised: nk × nk is invertible.

Revised: Then the sufﬁcient statistics of the generating latent space are related to those of the estimated latent space by the following relationship:

Original: [Equation]
Revised: [Equation]

Original: where A ∈ R nk × n (cid:48) k is full rank (with rank nk  ) when the dimension n (cid:48) of the estimating latent space is greater or equal insize to the dimension n of the generating latent space.
Revised: where A is a constant, full-rank nk × dk matrix and c ∈ R nk a constant vector. Proof: The conditional probabilities p T , λ ,f,ε ( x | u ) and p T (cid:48) , λ (cid:48) ,g ( x | u ) are assumed to be the same in the limit of inﬁnite data.

Original: First note that
Revised: By expanding these expressions via the change of variables formula and taking the logarithm we ﬁnd [Equation] where J is the Jacobian matrix. Let u 0 , . . .

Original: [Equation]
Revised: [Equation]

Revised: u nk be the conditions from (ii) above.

Original:  for any matrices A and B . 
Revised: We can subtract this expression for u 0 from the expression for some condition u l . The Jacobian terms and the term involving ε will vanish, since they do not depend on u: [Equation] Since the conditional probabilities of z and w are exponential family members

Revised: [Equation]

Original: The proof is almost identical to that in Appendix B of Khemakhem et al.
Revised: we can write this expression as [Equation] where the base measures Q i have cancelled out [Equation]

Original: (2019), so only the points of difference will be noted.
Revised: since they do not depend on u. Deﬁning ¯ λ ( u ) = λ ( u ) − λ ( u 0 ) and writing the above in terms of inner products, we ﬁnd

Revised: [Equation]

Original: All reasoning is identical (except for replacing n with n (cid:48) on the right hand side of the equalities) until equation (24):
Revised: for 1 ≤ l ≤ nk . Combining the nk expressions into a single matrix equation we can write this in terms of L from condition (ii) above and an analogously deﬁned L (cid:48) ∈ R dk × nk :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: 14) where b l = (cid:80) i log Z (cid:48) i ( u 0 ) Z i ( u l ) Z i ( u 0 ) Z (cid:48) i ( u l ) . Since L T is invertible, we can multiply this expression by its inverse from the left to get

Revised: [Equation]

Revised: [Equation]

Original: where here Q (cid:48) ∈ R n (cid:48) 
Revised: where A = L − T L (cid:48) T and c = L − T b .

Revised: It now remains to show that A has full rank.

Revised: According to a lemma from Khemakhem et al.

Revised: (2019), there exist k distinct values z 1 i to z ki such that (cid:16) d T i

Original: k
Revised: d

Revised: z i ( z 1 i ) , . . .

Revised: , d T i

Revised: d z i

Original: × nk . 
Revised: ( z ki ) (cid:17) are linearly independent in R k , for all 1 ≤ i ≤ n . Deﬁne k vectors z l = ( z l 1 , . . . , z ln ) from the points given by this lemma. Take the derivative of equation (15) evaluated at each of these k vectors and concatentate the resulting Jacobians as Q =

Revised: [ J T ( z 1 ) , . . .

Revised: , J T ( z k )] .

Revised: Each Jacobian has size nk × n , hence Q has size nk × nk and is invertible by the lemma and the fact that each component of T is univariate.

Revised: We can construct a corresponding matrix Q (cid:48) made up of the Jacobians of T (cid:48)

Revised: ( g − 1 ◦ f ε ( z )) evaluated at the same points and write

Original: Q  is invertible, so has rank nk , which means that A and Q (cid:48) have rank at least nk , or else they would violate (14).
Revised: Q = AQ (cid:48) . Since Q is full rank, both A and Q (cid:48) must also be full rank. A. R EMARK REGARDING CONDITION ( II )

Original: The maximum rank A can have is  nk , hence A has full rank.
Revised: If the coefﬁcients of the sufﬁcient statistics λ are generated randomly and independently, then condition (ii) is almost surely fulﬁlled. In this case, we can ignore the dependence on u to consider the λ ( u l ) as independent random variables. Then condition (ii) states that instances of these random variables are in general position in R nk , which is true almost surely. A. R EMARK REGARDING NOISE VARIABLES IN THE ESTIMATED LATENT SPACE

Revised: Any variables in the estimated latent space whose distribution does not depend on the conditioning variable u is considered as encoding only noise.

Revised: Such variables will be cancelled out in equation (12) and the corresponding column of L (cid:48) T will contain only zeros.

Revised: This means the corresponding column of A will also contain only zeros, so any variation in such a noise variable has no effect on z.

Revised: The reverse is also true: if a column of A contains only zeros, the corresponding variable in the estimated latent space does not depend on u and must encode only noise.

Original: B S PARSITY IN UNMIXING MATRIX : G AUSSIAN DISTRIBUTION
Revised: B S PARSITY IN UNMIXING MATRIX : G AUSSIAN DISTRIBUTION

Original: Suppose samples z from the generating latent space follow a conditional Gaussian distribution (equation (2)).
Revised: Suppose samples z from the generating latent space follow a conditional Gaussian distribution.

Original: Suppose the estimating model g faithfully reproduces the observed conditional density p ( x | u ) and samples w from its latent space also follow a conditional Gaussian distribution.
Revised: Suppose the estimating model g faithfully reproduces the observed conditional density p ( x | u ) and samples w from its latent space also follow a conditional Gaussian distribution.

Original: Then we can apply equation (6) to relate the generating and estimating latent spaces.
Revised: Then we can apply equation (4) to relate the generating and estimating latent spaces.

Original: Suppose arbitrary values of n and n (cid:48) , with n (cid:48) ≥ n .

Original: The sufﬁcient statistics of a normal distribution with free mean and variance are z and z 2 (see equation (4)).
Revised: The sufﬁcient statistics of a normal distribution with free mean and variance are z and z 2 .

Original: Hence the relationship between the latent spaces becomes
Revised: Hence the relationship between the latent spaces becomes

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the squaring is applied element-wise.
Revised: where the squaring is applied element-wise.

Original: We can write A in block matrix form as
Revised: We can write A in block matrix form as

Original: [Equation]
Revised: [Equation]

Original:  and c as
Revised: (18) and c as

Original: [Equation]
Revised: [Equation]

Original: Then:
Revised: Then:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  so we can write for each dimension i of z
Revised: 21) so we can write for each dimension i of z

Original: [Equation]
Revised: [Equation]

Original: In order to compare the equations, we need to square (21).
Revised: In order to compare the equations, we need to square (22).

Original: To do so, we will have to square the second term on the right hand side, involving w 2 j .
Revised: To do so, we will have to square the second term on the right hand side, involving w 2 j .

Original: There is no matching term in (22), so we have to set all entries of A (2) to zero.
Revised: There is no matching term in (23), so we have to set all entries of A (2) to zero.

Original: In more detail:
Revised: In more detail:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term with w 4 j matches no term in (22), so we have to set A (2) ij = 0 for all i and j .
Revised: The ﬁrst term with w 4 j matches no term in (23), so we have to set A (2) ij = 0 for all i and j .

Original: This simpliﬁes the earlier equation:
Revised: This simpliﬁes the earlier equation:

Original: [Equation]
Revised: [Equation]

Original: The square of the ﬁrst term on the right hand side involves terms with w j w j (cid:48) cross terms:
Revised: The square of the ﬁrst term on the right hand side involves terms with w j w j (cid:48) cross terms:

Original: [Equation]
Revised: [Equation]

Original: so we have to set A (1) ij A (1) ij (cid:48)
Revised: so we have to set A (1) ij A (1) ij (cid:48)

Original: = 

Revised: =

Original: for all j (cid:54) =
Revised: for all j (cid:54) =

Original: j (cid:48) .
Revised: j (cid:48) .

Original: This means that the i -th row of A (1) can have at most one nonzero entry.
Revised: This means that the i -th row of A (1) can have at most one nonzero entry.

Original: It must also have at least one nonzero entry, since if the row were all zero, a row of A would be all zero (since A (2) = 0 ), but A has full rank.
Revised: It must also have at least one nonzero entry, since if the row were all zero, a row of A would be all zero (since A (2) = 0 ), but A has full rank.

Original: Since there are as many or fewer rows than columns ( n ≤ n (cid:48) ), each row of A is linearly independent, so it is not possible for one to be zero.
Revised: Since there are as many or fewer rows than columns ( n ≤ d ), each row of A is linearly independent, so it is not possible for one to be zero.

Original: Hence each row of A (1) has exactly one nonzero entry.
Revised: Hence each row of A (1) has exactly one nonzero entry.

Original: Moreover, no two rows of A (1) have their nonzero entries in the same column.
Revised: Moreover, no two rows of A (1) have their nonzero entries in the same column.

Original: If they did, the two rows would not be linearly independent, but they must be since A has full rank.
Revised: If they did, the two rows would not be linearly independent, but they must be since A has full rank.

Original: Therefore we can write
Revised: Therefore we can write

Original: [Equation]
Revised: [Equation]

Original: where a i =
Revised: where a i =

Original: A (1) ij and b i =
Revised: A (1) ij and b i =

Original: c
Revised: c

Original: (1) i .
Revised: (1) i .

Original: That is, the generating latent variable z i is linearly related to some latent variable of the estimating model w j .
Revised: That is, the generating latent variable z i is linearly related to some latent variable of the estimating model w j .

Original: This estimated latent variable is uniquely associated with z i and any estimated latent variables not associated with a generating latent variable
Revised: This estimated latent variable is uniquely associated with z i and any estimated latent variables not associated with a generating latent variable

Original: z i (in the case n (cid:48) > n ) encode no information about the generating latent space.
Revised: z i (in the case d > n ) encode no information about the generating latent space.

Original: So the model has decoded the original latent variables z up to an afﬁne transformation and permutation as a subset of variables in its estimated latent space and has encoded no information (only noise) into the remaining latent variables.
Revised: So the model has decoded the original latent variables z up to an afﬁne transformation and permutation as a subset of variables in its estimated latent space and has encoded no information (only noise) into the remaining latent variables.

Original: C S PARSITY IN UNMIXING MATRIX : TWO - PARAMETER EXPONENTIAL FAMILY MEMBERS
Revised: C S PARSITY IN UNMIXING MATRIX : TWO - PARAMETER EXPONENTIAL FAMILY MEMBERS

Original: The results of Appendix B can be extended to other members of the exponential family withparameters.
Revised: The results of Appendix B can be extended to other members of the exponential family withparameters.

Original: In the general case, writing T i, 1 = T 1 and T i, 2 = T 2 for all i , equations(21) and (22)  become
Revised: In the general case, writing T i, 1 = T 1 and T i, 2 = T 2 for all i , equations (22) and (23) become

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 29) which, with the deﬁnition t =
Revised: which, with the deﬁnition t [Equation]

Original: T 2 ◦ T − 1becomes
Revised: becomes

Original: [Equation]
Revised: [Equation]

Original: We can combine these equations to get 
Revised: We can combine these equations to get t 

Original: [Equation]
Revised: [Equation]

Original: This equation has many summed terms on the right.
Revised: This equation has many summed terms on the right.

Original: Suppose that t has a convergent Taylor expansion in some region of its domain.
Revised: Suppose that t has a convergent Taylor expansion in some region of its domain.

Original: We can take this expansion of the term on the left and compare coefﬁcients.
Revised: We can take this expansion of the term on the left and compare coefﬁcients.

Original: Since t cannot be linear (otherwise the two sufﬁcient statistics would be the same), there will be terms in the expansion of order two or higher.
Revised: Since t cannot be linear, there will be terms in the expansion of order two or higher.

Original: As in the Gaussian case, these polynomial terms create cross terms which are impossible to reconcile with those on the right hand side of the equation.
Revised: As in the Gaussian case, these polynomial terms create cross terms which are impossible to reconcile with those on the right hand side of the equation.

Original: The only consistent solutions can be found by setting all coefﬁcients except those of functions of w j (for some j ) to zero:
Revised: The only consistent solutions can be found by setting all coefﬁcients except those of functions of w j (for some j ) to zero:

Original: [Equation]
Revised: [Equation]

Original: We can further simplify this equation by examining higher-order terms, but need to know the form of t to do so.
Revised: We can further simplify this equation by examining higher-order terms, but need to know the form of t to do so.

Original: In any case, we can now write equation (28) as 
Revised: In any case, we can now write equation (29) as [Equation]

Original: [Equation]
Revised: [Equation]

Original: showing that each generating latent variable z
Revised: showing that each generating latent variable z

Original: i is related to exactly one estimated latent variable w j .
Revised: i is related to exactly one estimated latent variable w j .

Original: As in the Gaussian case, we can use the full rank property of A to see that each estimated latent variable is associated with at most one generating latent variable, and any estimated latent variables not associated with any generating latent variables must encode only noise.
Revised: As in the Gaussian case, we can use the full rank property of A to see that each estimated latent variable is associated with at most one generating latent variable, and any estimated latent variables not associated with any generating latent variables must encode only noise.

Original: The task now is to check the form of t for each two-parameter member of the exponential family, to see what further constraints we can derive from equation (33).
Revised: The task now is to check the form of t for each two-parameter member of the exponential family, to see what further constraints we can derive from equation (34).

Original: This will not be done in detail here, but the results are stated in Table 1.
Revised: The results are stated in Table 1.

Original: [Table]
Revised: [Table]

Original: ( z, z 2 ) xz 
Revised: ( z, z 2 ) xz i

Original: [Equation]

Original: =
Revised: =

Original: [Equation] j [Equation]
Revised: aw j + c

Original: [Table]
Revised: [Table]

Original: (log z, (log z ) 2 ) x
Revised: (log z, (log z ) 2 ) xlog

Original: [Equation]

Original:  log [Equation] [Table] z, 1 /
Revised: z i = a log w j + c

Original: [Table]
Revised: [Table]

Original:  ) 1 /x z [Equation]
Revised: ( z, 1 /z ) 1 /x z i

Original: =
Revised: =

Original: [Equation]

Original:  j or z i = [Equation] /w j
Revised: aw j or z i = a/w j

Original: [Table]
Revised: [Table]

Original: (log z, z ) exp( x ) z i
Revised: (log z, z ) exp( x ) z i

Original: =
Revised: =

Original: aw j
Revised: aw j

Original: [Table]
Revised: [Table]

Original: (log z, 1 /z ) exp( − x )
Revised: (log z, 1 /z ) exp( − x )

Original: z 
Revised: z i

Original: [Equation]
Revised: = aw j

Original: [Table] (log z, log(1 − z ))  log [Equation]
Revised: Beta (log z, log(1 − z )) log(1 − exp( x )) log z

Original: 1 − [Equation] x )) [Equation]
Revised: i = log w j or log z

Original: or
Revised: i

Original: [Equation]
Revised: = log(1 − w j )

Original: D N ETWORK A RCHITECTURE
Revised: D N ETWORK A RCHITECTURE

Original: The estimating model g is built in the reverse direction for practical purposes, so the models described here are g − 1 which maps from the data space to the latent space.
Revised: The estimating model g is built in the reverse direction for practical purposes, so the models described here are g − 1 which maps from the data space to the latent space.

Original: The type and number of coupling blocks for the different experiments are shown below.
Revised: The type and number of coupling blocks for the different experiments are shown below.

Original: The afﬁne coupling function is the concatentation of the scale function s and the translation function t , computed together for efﬁciency, as in Kingma & Dhariwal (2018).
Revised: The afﬁne coupling function is the concatentation of the scale function s and the translation function t , computed together for efﬁciency, as in Kingma & Dhariwal (2018).

Original: It is implemented as either a fully connected network (MLP) or convolutional network, with the speciﬁed layer widths and a ReLU activation after all but the ﬁnal layers.
Revised: It is implemented as either a fully connected network (MLP) or convolutional network, with the speciﬁed layer widths and a ReLU activation after all but the ﬁnal layers.

Original: For the convolutional coupling blocks, the splits are along the channel dimension.
Revised: For the convolutional coupling blocks, the splits are along the channel dimension.

Original: The scale function s is passed through a clamping function 2 tanh( s ) , which limits the output to the range (-2, 2), as in Ardizzone et al.
Revised: The scale function s is passed through a clamping function 2 tanh( s ) , which limits the output to the range (-2, 2), as in Ardizzone et al.

Original: Two afﬁne coupling functions are applied per block, as described in Dinh et al.
Revised: Two afﬁne coupling functions are applied per block, as described in Dinh et al.

Original: Downsampling increases the number of channels by a factor ofand decreases the image width and height by a factor of 2, done in a checkerboard-like manner, as described in Jacobsen et al.
Revised: Downsampling increases the number of channels by a factor ofand decreases the image width and height by a factor of 2, done in a checkerboard-like manner, as described in Jacobsen et al.

Original: The dimensions are permuted in a random but ﬁxed way before application of each fully connected coupling block and likewise the channels for the convolutional coupling blocks.
Revised: The dimensions are permuted in a random but ﬁxed way before application of each fully connected coupling block and likewise the channels for the convolutional coupling blocks.

Original: The network for the artiﬁcial data experiments has 4,480 learnable parameters and the network for the EMNIST experiments has 2,620,192 learnable parameters.
Revised: The network for the artiﬁcial data experiments has 4,480 learnable parameters and the network for the EMNIST experiments has 2,620,192 learnable parameters.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Convolutional Coupling (16, 7, 7) 8 → 32 → 32 →
Revised: Convolutional Coupling (16, 7, [Equation] [Equation]

Original: Flattening (16, 7, 7)
Revised: Flattening (16, 7, 7)

Original: Fully Connected Coupling 392 → 392 → 392 →
Revised: Fully Connected Coupling [Equation] D. N OTE ON O PTIMIZATION M ETHOD

Revised: In the experiments described in this paper, the mean and variance of a mixture component was updated at each iteration as the mean and variance of the transformations to latent space of all data points belonging to that mixture component in a minibatch of data D :

Original: E F
Revised: [Equation] [Equation] Hence the parameters of the mixture components would change in each batch, according to the data present. The notation µ i ( u ; θ ) does not indicate that µ is directly parameterized by θ and learned, instead it indicates that µ is a function of g − 1 which is parameterized by θ .

Revised: A change in θ will also change the output of µ , given the same mini-batch of data D .

Revised: The same holds for σ i ( u ; θ ) .

Revised: We expect that specifying the means and variances as learnable parameters updated in tandem with the model weights would have worked equally well.

Revised: E

Original: F
Revised: F

Original: EMNIST
Revised: EMNIST

Original: [Figure]
Revised: [Figure]

Original: IGURES
Revised: IGURES

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]