Original: I DENTIFYING I NFORMATIVE L ATENT V ARIABLES L EARNED BY GIN VIA M UTUAL I NFORMATION
Revised: I DENTIFYING I NFORMATIVE L ATENT V ARIABLES L EARNED BY GIN VIA M UTUAL I NFORMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: How to learn a good representation of data is one of the most important topics of machine learning.
Revised: How to learn a good representation of data is one of the most important topics of machine learning.

Original: Disentanglement of representations, though believed to be the core feature of good representations, has caused a lot of debates and discussions in recent.
Revised: Disentanglement of representations, though believed to be the core feature of good representations, has caused a lot of debates and discussions in recent.

Original: Sorrenson et al.
Revised: Sorrenson et al.

Original: (2020), using the techniques developed in nonlinear independent  analysis theory, show that general incompressible-ﬂow networks (GIN) can recover the underlying latent variables that generate the data, and thus can provide a compact and disentangled representation.
Revised: (2020), using the techniques developed in nonlinear independent component analysis theory, show that general incompressible-ﬂow networks (GIN) can recover the underlying latent variables that generate the data, and thus can provide a compact and disentangled representation.

Original: However, in this paper, we point out that the method taken by GIN for informative latent variables identiﬁcation is not theoretically supported and can be disproved by experiments.
Revised: However, in this paper, we point out that the method taken by GIN for informative latent variables selection is not theoretically supported and can be disproved by experiments.

Original: We propose to use the mutual information between  latent variables and the auxiliary variable to correctly identify informative latent variables.
Revised: We propose to use the mutual information between each learned latent variables and the auxiliary variable to correctly identify informative latent variables.

Original: We directly verify the improvement brought by our method in experiments on synthetic data.
Revised: We directly verify the improvement brought by our method in experiments on synthetic data.

Original: We further show the advantage of our method on various downstream tasks including classiﬁcation, outlier detection and adversarial attack defence.
Revised: We further show the advantage of our method on various downstream tasks including classiﬁcation, outlier detection and adversarial attack defence on both synthetic and real data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Representation learning is arguably one of the most important area in machine learning.
Revised: Representation learning is arguably one of the most important area in machine learning.

Original: Many researchers believe that being able to extract useful and interpretable features is a crucial advantage of deep networks over other learning models.
Revised: Many researchers believe that being able to extract useful and interpretable features is a crucial advantage of deep networks over other learning models.

Original: A data representation can be obtained either via a supervised learning task or an unsupervised learning task.
Revised: A data representation can be obtained either via a supervised learning task or an unsupervised learning task.

Original: The former one includes the popular ImageNet pretrained backbone in computer vision tasks, while the latter oneis mainly  generative models like variants of VAEs, GANs and ﬂow-based models.
Revised: The former one includes the popular ImageNet pretrained backbone in computer vision tasks, while the latter one mainly consists of generative models like variants of VAEs, GANs and ﬂow-based models.

Original: Among all the generative models, VAEs and ﬂowbased models can naturally output the representation of data and their density, which is convenient for the representation learning purpose.
Revised: Among all the generative models, VAEs and ﬂow-based models can naturally output the representation of data or even their density, which is convenient for the representation learning purpose.

Original: Moreover, supervision information of labels can be integrated into generative models to further improve their performance.
Revised: Moreover, supervision information of labels can be integrated into generative models to further improve their performance.

Original: General Incompressible-ﬂow 
Revised: General Incompressible-ﬂow Networks (GIN, Sorrenson et al.

Original: Networks (GIN), the model we considered in this paper, falls into this case.
Revised: , the model we considered in this paper, falls into this case.

Original: Disentanglement is a widely discussed concept by many representation learning works.
Revised: Disentanglement is a widely discussed concept by many representation learning works.

Original: However, to the best of our knowledge, it has not been given a widely accepted deﬁnition (Bengio et al., 2013;
Revised: However, to the best of our knowledge, it has not been given a widely accepted deﬁnition (Bengio et al., 2013;

Original: Higgins et al., 2018).
Revised: Higgins et al., 2018).

Original: Many disentangled representation learning algorithms focus on recovering the independent latent variables that generate the data (Burgess et al., 2018; Chen et al., 2018b).
Revised: Many disentangled representation learning algorithms focus on recovering the independent latent variables that generate the data (Burgess et al., 2018; Chen et al., 2018b).

Original: However, Locatello et al.
Revised: However, Locatello et al.

Original: (2018) show that without more assumptions than independence among latent variables, it is impossible to recover them from the observation of data.
Revised: (2018) show that without more assumptions than independence among latent variables, it is impossible to recover them from the observation of data.

Original: This result is equivalent to the non-identiﬁability of non-linear independent component analysis (ICA).
Revised: This result is equivalent to the non-identiﬁability of non-linear independent component analysis (ICA)

Original:  Actually, any assumptions solely on the latent variables’ distribution without referring to the observable data is not sufﬁcient for the identiﬁability (Hyvarinen and Morioka, 2016; Khemakhem et al., 2020).
Revised: (Comon, 1994). Actually, any assumptions solely on the latent variables’ distribution without referring to the observable data is not sufﬁcient for the identiﬁability (Hyvarinen and Morioka, 2016; Khemakhem et al., 2020).

Original: A set of sufﬁcient conditions is proposed under the framework of nonlinear ICA by Khemakhem et al.
Revised: A set of sufﬁcient conditions is proposed under the framework of nonlinear ICA by Khemakhem et al.

Original: The core condition requires that the data, denoted by x , are generated by latent vectors z through a generative model p ( x | z ) , and conditioned on an auxiliary variable u , the entries of z are independent and follow some exponential family distributions.
Revised: The core condition requires that the data, denoted by x , are generated by latent vectors z through a generative model p ( x | z ) , and conditioned on an auxiliary variable u , the entries of z are independent and follow some exponential family distributions.

Original: This can be expressed by the following formulas.
Revised: This can be expressed by the following formulas.

Original: [Equation]
Revised: [Equation]

Original: In Eq.
Revised: In Eq.

Original: 1 and 2, z ∈ R n and x ∈ R d .
Revised: 1 and 2, z ∈ R n and x ∈ R d .

Original: Usually people believe that n (cid:28) d ; that is, the data are distributed near a low dimensional manifold embedded in a very high dimensional space.
Revised: Usually people believe that n (cid:28) d ; that is, the data are distributed near a low dimensional manifold embedded in a very high dimensional space.

Original: With both u and x being observable, the identiﬁability of z can be established.
Revised: With both u and x being observable, the identiﬁability of z can be established.

Original: Moreover, Khemakhem et al.
Revised: Moreover, Khemakhem et al.

Original: also show that the latent variables and the generative function f can be learned by optimizing the
Revised: also show that the latent variables and the generative function f can be learned by optimizing the

Original: ELBO using a special VAE, called iVAE (i for identiﬁable).
Revised: ELBO using a special VAE, called iVAE (i for identiﬁable).

Original: However, iVAE assumes that all the latent variables are correlated with u , and it requirescorrectly setting the  number of latent variables for iVAE to work, which is usually not practical.
Revised: However, iVAE assumes that all the latent variables are correlated with u , and it requires the correct number of latent variables for iVAE to work, which is usually not known.

Original: Sorrenson et al.
Revised: Sorrenson et al.

Original: (2020) consider the generative model with Eq.
Revised: (2020) consider the generative model with Eq.

Original: 2 replaced by
Revised: 2 replaced by

Original: [Equation]
Revised: [Equation]

Original: where f is invertible, p T ,λ ( z | u ) is  Gaussian  supported on R n with free expectations and diagonal covariance, and p ( (cid:15) ) , supported on R d − n , is free from u .
Revised: where f is invertible, p T ,λ ( z | u ) is a Gaussian density supported on R n with free expectations and diagonal covariance, and p ( (cid:15) ) , supported on R d − n , is free from u .

Original: Entries of z are refered to as informative latent variables, while entries of (cid:15) are treated as noise.
Revised: Entries of z are refered to as informative latent variables, while entries of (cid:15) are treated as noise.

Original: They prove that z is identiﬁable up to a permutation and scaling.
Revised: They prove that z is identiﬁable up to a permutation and scaling.

Original: Based on this identiﬁability result, Sorrenson et al.
Revised: Based on this identiﬁability result, Sorrenson et al.

Original: propose to learn the informative latent variables z using a ﬂow-based model, called the General Incompressible-ﬂow
Revised: propose to learn the informative latent variables z using a ﬂow-based model, called the General Incompressible-ﬂow

Original: Networks (GIN).
Revised: Networks (GIN).

Original: Denote by w the representation learned by the model. Since the ﬂow-based model maps data to representation vectors of the same dimension, it requires a method to identify the entries of w that estimate z .
Revised: The output of a GIN model, w = g ( x ) , is a vector with its dimension equal to dim( z , (cid:15) ) , since both f and g are invertible. We, hence, need a method to identify the entries of w that estimate z .

Original: In Sorrenson et al.
Revised: Sorrenson et al.

Original: (2020), a variance based criterion is proposed.
Revised: propose a variance based criterion for the informative latent variables selection.

Original: Speciﬁcally,  w i ’s with large variancesare selected as estimates to z and  the remaining entriesare treated as  related to (cid:15) . 
Revised: Speciﬁcally, they select w i ’s with large variances as estimates to z and treat the remaining entries as solely related to (cid:15) . Sorrenson et al.

Original: Even though they show that this criterion correctly identiﬁes the ground truth informative latent variables in experiments on synthetic data, the noise latent variables they consider all have very small variances. Indeed, we ﬁnd that if we increase the variances of(cid:15) (but still less than the variances of z ), this criterion will fail.
Revised: do not clearly explain the logic behind this criterion. For the following reasons, we think this criterion is not guaranteed to work all the time: Note that a volume-preserving transform cannot preserve the variances of the input data, because volume-preserving constrains the determinant of the Jacobian instead of the size of each singular value.

Original: According to the identiﬁability theorem in Sorrenson et al.
Revised: According to the identiﬁability theorem in Sorrenson et al.

Original: (2020), each z i can only be estimated up to an afﬁne transform. In other words, in the learned representation, the variances of informative w i ’s can be smaller than the variances of non-informative ones.
Revised: (2020), each z i can only be estimated up to an afﬁne transform, which means that the variances of informative w i ’s can be smaller than the variances of non-informative ones.

Original: As a result, the inconsistent results caused purely by the noise level and the contradiction to the identiﬁcation theory indicate that the variance based criterion is not suitable for informative latent variables selection.
Revised: Nevertheless, they show that the variance based criterion can correctly identify the ground truth informative latent variables in experiments on synthetic data. However, in their experiments, the noise latent variables all have very small variances. When reproducing their experiments, we ﬁnd that if we increase the variances of (cid:15) , but which are still less than the variances of z ), this criterion will fail. On the other hand, the identiﬁability result in Sorrenson et al.

Revised: (2020) implies that the entries of w can be categorized into two classes, entries in one class are linearly related to entries of z , while entries in the other class have distributions that are irrelevant to u .

Original: This observation inspires us to scrutinise the notion of informative latent variables under the generative framework described by Eq. 3.
Revised: This observation inspires us to scrutinise the notion of informative latent variables under the generative framework described by Eq. 3.

Original: Even though we accept the assumption that n (cid:28) d , the dimension of data is still d due to the existence of noise (cid:15) .

Original: The key difference between (cid:15) and z is not their variances, but whether they are correlated with the auxiliary variable u .
Revised: The key difference between informative and non-informative latent variables is not their variances, but whether they are correlated with the auxiliary variable u .

Original: Following this new perspective, it is natural to choose the mutual information between w i ’s and u as the criterion for identifying informative latent variables, which we call it the mutual information (MI) criterion.
Revised: Following this new perspective, it is natural to choose the mutual information between w i ’s and u as the criterion for identifying informative latent variables, which we call it the mutual information (MI) criterion.

Original: In this paper, we compare the MI criterion with the previous criterion based on variance/standard deviation (VAR) using abundant experiments on various tasks, including disentanglement quality on synthetic data, classiﬁcation performance on real data, outlier detection capability, and robustness under adversarial attacks.
Revised: In this paper, we compare the MI criterion with the previous variance based criterion (VAR) using abundant experiments on various tasks, including disentanglement quality on synthetic data, classiﬁcation performance on real data, outlier detection capability, and robustness under adversarial attacks.

Original: In all these cases, the MI criterion shows superior performance over the VAR criterion.
Revised: In all these cases, the MI criterion shows superior performance over the VAR criterion.

Original: The main content of the paper is organised as follows.
Revised: The main content of the paper is organised as follows.

Original: In Section 2, we brieﬂy review the identiﬁability result of Sorrenson et al.
Revised: In Section 2, we brieﬂy review the identiﬁability result of Sorrenson et al.

Original: (2020), and show why the mutual information can work for informative latent variables selection.
Revised: (2020), and show why the mutual information can work for informative latent variables selection.

Original: In Section 3, we compare the performance of the MI criterion with that of the VAR criterion by experiments on synthetic data generated following the theoretical assumptions.
Revised: In Section 3, we compare the performance of the MI criterion with that of the VAR criterion by experiments on synthetic data generated following the theoretical assumptions.

Original: In Section 4, we demonstrate the impacts of different informative variables selection criteria on classiﬁcation, outlier detection, and defence against adversarial attacks by experiments on EMNIST (Cohen et al., 2017).
Revised: In Section 4, we demonstrate the impacts of different informative variables selection criteria on classiﬁcation, outlier detection, and defence against adversarial attacks by experiments on EMNIST (Cohen et al., 2017).

Original: In Section 5, we discuss some other interesting discoveries related to the representation learning under the framework of nonlinear ICA.
Revised: In Section 5, we discuss some other interesting discoveries related to the representation learning under the framework of nonlinear ICA.

Original: And ﬁnally, we conclude our results and summarise some future working directions in the last section.
Revised: And ﬁnally, we conclude our results and summarise some future working directions in the last section.

Original:  T HEORY OF N ONLINEAR ICA AND M UTUAL I NFORMATION C RITERION
Revised:  T HEORY OF N ONLINEAR ICA AND M UTUAL I NFORMATION C RITERION

Original: Since our work is heavily entangled with Sorrenson et al.
Revised: Since our work is an improvement based on GIN, we ﬁrst review the main result of Sorrenson et al.

Original: (2020), we ﬁrst review their main result with an emphasis on the interesting consequence of the identiﬁability result, and then make use of the subtle result to derive our mutual information criterion.
Revised: (2020) with a detailed elaboration on the relation between the learned representation w and the auxiliary variable u , and then make use of this property to derive our mutual information criterion.

Original: Due to the length restriction of the paper, we refer interested readers to Appendix A for more details.
Revised: Due to the length restriction of the paper, we refer interested readers to Appendix A for formal assumptions and statements.

Original: As shown in Eq 3, the noise in the data generation modelis deﬁned as part of latent variables. The proof of the identiﬁability theorem in Sorrenson et al.
Revised: With the data generation model deﬁned by Eq 1 and 3 The proof of the identiﬁability theorem in Sorrenson et al.

Original: (2020)  consists of the following three steps:
Revised: (2020) mainly consists of the following three steps:

Original: Start with p θ ( x | u ) =
Revised: Start with p θ ( x | u ) =

Original: p φ ( x | u ) , where the conditional marginal data distributions are determined by Eq. 1 and Eq.
Revised: p φ ( x | u ) , where the conditional marginal data distributions are determined by Eq. 1 and Eq.

Original: 3 with two possibly different transforms, latent variables and parameters; θ = ( f, T ( z , (cid:15) ) , λ ) in the generation process and φ = ( g, T ( w ) , λ (cid:48) )
Revised: 3 with two possibly different transforms, latent variables and parameters; θ = ( f, T ( z , (cid:15) ) , λ ) in the generation process and φ = ( g, T ( w ) , λ (cid:48) )

Original: in the estimation process.
Revised: in the estimation process.

Original: Then eliminate terms  irrelevant with the auxiliary variable u by the constraints of the exponential family.
Revised: Then eliminate terms in Eq. 1 that are irrelevant to the auxiliary variable u . 3.

Original: And ﬁnally obtain the relation between the statistics T ( z ) and T ( w ) ,
Revised: And ﬁnally obtain the relation between the statistics T ( z ) and T ( w ) ,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: (cid:48)

Original: [Equation]
Revised: [Table]

Original: [Equation] u 0 ) λ (cid:48)
Revised: n,

Original: [Equation]
Revised: [Table]

Original: 2 ( u 1 ) − λ (cid:48) [Equation] [Equation] 0 ) · · · λ (cid:48) d,k ( u 1 ) − λ (cid:48) d,k ( u 0 ) ...
Revised: [Table]

Original: [Equation]
Revised: [Table]

Original: (cid:48)

Original: [Equation] [Equation] [Equation] (cid:48)
Revised: 

Original: [Equation]

Original: [Equation]
Revised: [Table]

Original: (cid:48) [Equation] [Equation] (cid:48)
Revised:  

Original: [Equation] [Equation] [Equation] [Equation]
Revised: [Table]

Original: (cid:48)

Original: [Equation]
Revised: [Table]

Original:   (5)
Revised:  and entries of c are functions of u .

Original: Since the latent variables conditioned on u are assumed to be distributed as Gaussian, T ( z )
Revised: Since the latent variables conditioned on u are assumed to be distributed as Gaussian, T ( z )

Original: =
Revised: =

Original: ( z , z 2 ) (cid:62) .
Revised: ( z , z 2 ) (cid:62) .

Original: Their main theorem shows that after appropriate rearrangement of entries of z ,
Revised: Their main theorem shows that after appropriate rearrangement of entries of z ,

Original: [Equation]
Revised: [Equation]

Original: (6) where D is an n × n diagonal matrix with non-zero entries.
Revised: where D is an n × n diagonal matrix with non-zero entries.

Original: This means that after rearrangement, the ﬁrst n entries of w are just scaling of z .
Revised: This means that after rearrangement, the ﬁrst n entries of w are just scaling of z .

Original: For the rest d − n entries, denoted by w n +1: d , their connection with z cannot be justiﬁed by Eq.
Revised: For the rest d − n entries, denoted by w n +1: d , their connection with z cannot be justiﬁed by Eq.

Original: 6, since the corresponding coefﬁcients are zeros.
Revised: 4, since the corresponding coefﬁcients are zeros.

Original: Their relation with u , however, is clear.
Revised: Their relation with u , however, is clear.

Original: Since entries in L − 1 L (cid:48) corresponding to w n +1: d are all zero, we know that λ (cid:48) ij ( u l ) must equal to λ (cid:48) ij ( u 0 ) for i ∈ [ n + 1 , d ] , j = 1 , 2 and l ∈ [1 , nk ] .
Revised: Since entries in L − 1 L (cid:48) corresponding to w n +1: d are all zero, we know that λ (cid:48) ij ( u l ) must equal to λ (cid:48) ij ( u 0 ) for i ∈ [ n + 1 , d ] , j = 1 , 2 and l ∈ [1 , nk ] .

Revised: This means that the distribution of w i for i ∈ [ n + 1 , d ] remains unchanged no matter what is the value of u .

Original: Based on their result, we know that an informative w i equals to some z j after an afﬁne transform and thus has non-zero mutual information with u , while non-informative w i has zero mutual information with u .
Revised: Based on their result, we know that an informative w i equals to some z j after an afﬁne transform and thus has non-zero mutual information with u , while non-informative w i has zero mutual information with u .

Original: Therefore, the mutual information between each w i and u can be used as a reliable indicator for justifying whether w i is an informative latent variable.
Revised: Therefore, the mutual information between each w i and u can be used as a reliable indicator for justifying whether w i is an informative latent variable.

Original: On the other hand, as we explained in the introduction, the composite mapping g − 1 ◦ f does not necessarily preserve the scale of z and (cid:15) .
Revised: On the other hand, as we explained in the introduction, the composite mapping g − 1 ◦ f does not necessarily preserve the scale of z and (cid:15) .

Original: Even if (cid:15) have much smaller variances compared to z , it is not necessarily true that the variances of { w i } di = n +1 are less than those of { w i } ni =1 .
Revised: Even if (cid:15) have much smaller variances compared to z , it is not necessarily true that the variances of { w i } d i = n +1 are less than those of { w i } n i =1 .

Original: We demonstrate this by an experiment in the next section.
Revised: We demonstrate this by an experiment in the next section.

Original: Therefore using the size of variances of { w i } di =1 for informative latent variables selection can result in serious problems.
Revised: Therefore using the size of variances of { w i } di =1 for informative latent variables selection can result in serious problems.

Original: We summarise the algorithm for  MI criterion in Alg.1 and conclude this section with a discussion on computational complexity.
Revised: We summarise the algorithm for the MI criterion in Alg.

Original: There are two schemes training the GIN framework, one is setting the distribution parameters trainable and the other one is evaluating the distribution parameters empirically on the training batches. While using VAR criterion for the latter scheme involves extra computation, one can directly use the trained parameter from the ﬁrst scheme.
Revised: The mutual information between w i and u is computed based on the samples by directly calling the function provided in the python package sklearn .

Original: Besides, evaluating variances is normally more efﬁcient than evaluating mutual information on the same data batch. Nevertheless, since we only need to evaluate the mutual information once after training, the extra computation introduced by MI criterion would be negligible compared to the training time.
Revised: The computational cost of mutual information is usually huge for high dimensional random vectors, but for that between a continuous scalar random variable and a discrete random variable, the computational cost is negligible.

Original:  P ERFORMANCE ON S YNTHETIC D ATA
Revised:  P ERFORMANCE ON S YNTHETIC D ATA

Original: In this section, we demonstrate the performance of the MI criterion on synthetic data, compared to the VAR criterion.
Revised: In this section, we demonstrate the performance of the MI criterion on synthetic data, compared to the VAR criterion.

Original: The synthetic data are constructed similarly with the GIN paper.
Revised: The synthetic data are constructed similarly with the GIN paper.

Original: The data space is Rand so is the latent space.
Revised: The data space is Rand so is the latent space.

Original: Within the 10 latent variables, there are only 2 informative ones.
Revised: Within the 10 latent variables, there are only 2 informative ones.

Original: These two latent variables z 1 , z 2 are distributed as a Gaussian mixture with 5 components, each of which, labelled by the auxiliary variable u , has a different expectation and a diagonal covariance matrix.
Revised: These two latent variables z 1 , z 2 are distributed as a Gaussian mixture with 5 components, each of which, labelled by the auxiliary variable u , has a different expectation and a diagonal covariance matrix.

Original: The extra 8 latent variables (cid:15) are treated as noise, which follows independent Gaussian distributions.
Revised: The extra 8 latent variables (cid:15) are treated as noise, which follows independent Gaussian distributions.

Original: An invertible function f : R 10 → R 10 maps each latent vector sampled according to the distribution described above to a data point x .
Revised: An invertible function f : R 10 → R 10 maps each latent vector sampled according to the distribution described above to a data point x .

Original: In our experiment, the statistics of the Gaussian mixture are randomly generated (see Appendix D), the noise variables are distributed as the standard Gaussian, and the invertible function is simulated by a ﬂow model with random parameters.
Revised: In our experiment, the statistics of the Gaussian mixture are randomly generated (see Appendix D), the noise variables are distributed as the standard Gaussian, and the invertible function is simulated by a ﬂow model with random parameters.

Original: Note that in our data generating process, the variance of the noise is  ofthe same scale with informative variables.
Revised: Note that in our data generating process, the variance of the noise is still less than that of informative variables, but not as small as in Sorrenson et al.

Original: Figure 1 shows 4 examples of the distributions of the 10 dimensional latent vectors projected to different choices of variable pairs.
Revised: Figure 1 shows 4 examples of the distributions of thedimensional latent vectors projected to different choices of variable pairs.

Original: When they are projected to the correct informative dimensions, the illustration shows a typical Gaussian mixture shape, as shown in the left subﬁgure.
Revised: When they are projected to the correct informative dimensions, the illustration shows a typical Gaussian mixture shape, as shown in the left subﬁgure.

Original: The other three subﬁgures show the distribution of latent variables when only one or none of the projection directions is informative.
Revised: The other three subﬁgures show the distribution of latent variables when only one or none of the projection directions is informative.

Original: Figure 2 shows the 
Revised: Figure 2 shows the distribution of data x projected to different choices of dimensions.

Original: To learn the representation fromthe data, we train a GIN model by minimising the following objective:
Revised: To learn the representation from data, we train a GIN model by minimising the following objective:

Original: [Equation]
Revised: [Equation]

Original: Note that here both the weights of the ﬂow θ and the statistics of the Gaussian mixture ( µ , σ ) are tuned during training.
Revised: Note that here both the weights of the ﬂow θ and the statistics of the Gaussian mixture ( µ , σ ) are tuned during training.

Original: Another form of objective suggested for the GIN model in the original paper is to replace trainable ( µ , σ ) by batch statistics.
Revised: Another form of objective suggested for the GIN model in the original paper is to replace trainable ( µ , σ ) by batch statistics.

Original: Consistent with the report of the original paper, we ﬁnd that setting ( µ , σ ) to be trainable shows similar performance with equating them to batch statistics.
Revised: Consistent with the report of the original paper, we ﬁnd that setting ( µ , σ ) to be trainable shows similar performance with equating them to batch statistics.

Original: However, trainable ( µ , σ ) match the theoretical framework better, avoid large variation caused by small batch size, and save the computational cost of evaluating the statistics for density estimation during downstream tasks.
Revised: However, trainable ( µ , σ ) match the theoretical framework better, avoid large variation caused by small batch size, and save the computational cost of evaluating the statistics for density estimation during downstream tasks.

Original: Therefore, in the main content of the paper, we will always consider the objective given by Eq 7.
Revised: Therefore, in the main content of the paper, we will always consider the objective given by Eq 5.

Original: For the experiment results of the alternative way, see Appendix
Revised: For the experiment results of the alternative way, see Appendix

Original: C. Other speciﬁc choice of parameters for the training include: batch size at 240 and initial learning rate at 10 − 2 with multiplicative decay rate 0 . 1 .
Revised: C. Other speciﬁc choices of parameters for the training include: batch size at 240 and initial learning rate at 10 − 2 with multiplicative decay rate 0 . 1 .

Original: After training, we compute both the variances of the features in the representation w and their mutual information with the auxiliary variable u , and plot them in descending order; see Figure 3.
Revised: After training, we compute both the variances of the features in the representation w and their mutual information with the auxiliary variable u , and plot them in descending order; see Figure 3.

Original: If we select the two latent variables with the largest variances and plot them, we can see that their distribution is different from the informative latent variables as shown in Figure 1.
Revised: If we select the two latent variables with the largest variances and plot them, we can see that their distribution is different from the informative latent variables as shown in Figure 1.

Original: According to the
Revised: According to the

Original: MI criterion, we select the two variables with the largest mutual information with u .
Revised: MI criterion, we select the two variables with the largest mutual information with u .

Original: They almost perfectly reproduce the distribution of the 2 informative latent variables.
Revised: They almost perfectly reproduce the distribution of the 2 informative latent variables.

Original: If we do not know the correct number of informative variables, we can estimate by truncating the number of features in the representation according to the magnitude of mutual information values.
Revised: If we do not know the correct number of informative variables, we can estimate by truncating the number of features in the representation according to the magnitude of mutual information values.

Original: In Figure 3, we can see that the sharp drop occurs exactly between the second and the third variables from left.
Revised: In Figure 3, we can see that the sharp drop occurs exactly between the second and the third variables from left.

Original: This experiment shows that the GIN model can successfully learn the informative latent variables as indicated by the theory, but we need the MI criterion to correctly identify them.
Revised: This experiment shows that the GIN model can successfully learn the informative latent variables as indicated by the theory, but we need the MI criterion to correctly identify them.

Original: [Figure]
Revised: [Figure]

Original: The standard deviations of the 10 features (black) of w and the 10 true latent variables (grey) in descending order.

Original: The numbers are the indices of features for convenience of comparison

Original: The distribution of the 2 features with the largest mutual information with u .

Original: d)

Original: The mutual information of the 10 features (black) of w and the 10 true latent variables (grey) in descending order.

Original: We further demonstrate the importance of identifying the correct informative latent variables for the classiﬁcation and outlier detection tasks.
Revised: We further demonstrate the importance of identifying the correct informative latent variables for the classiﬁcation and outlier detection tasks.

Original: Note that in our experiments u is the label in the classiﬁcation task.
Revised: Note that in our experiments u is the label in the classiﬁcation task.

Original: For the classiﬁcation task, we evaluate the posterior probability p S ( u | x ) for each test example using the features of the representation selected into the set S according to either our MI criterion or the VAR criterion.
Revised: For the classiﬁcation task, we evaluate the posterior probability p S ( u | x ) for each test example using the features of the representation selected into the set S according to either our MI criterion or the VAR criterion.

Original: Speciﬁcally, thethe posterior distribution of label u is deﬁned as
Revised: Speciﬁcally, the posterior distribution of label u is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where w S is the vector of selected features, p ( w S | u ) is derived from p ( w | u ) that learned by the
Revised: where w S is the vector of selected features, p ( w S | u ) is derived from p ( w | u ) that learned by the

Original: GIN model and p ( u ) is the distribution of labels (uniform distribution of 5 classes).
Revised: GIN model and p ( u ) is the distribution of labels (uniform distribution of 5 classes).

Original: In Figure 4, we can see that using the two latent variables identiﬁed by our MI criterion, the classiﬁcation tasks can be solved almost perfectly, while the original VAR criterion is not competent at all.
Revised: In Figure 4, we can see that using the two latent variables identiﬁed by our MI criterion, the classiﬁcation tasks can be solved almost perfectly, while the original VAR criterion is not competent at all.

Original: For the outlier (or out-of-distribution samples) detection task, we sample outliers in the latent space from two Gaussians far apart from the high-density area of normal latent samples, as shown in
Revised: For the outlier (or out-of-distribution sample) detection task, we sample outliers in the latent space from two Gaussians far apart from the high-density area of normal latent samples, as shown in

Original: Figure5, and then send them through the generator to obtain the data outliers.
Revised: Figure and then send them through the generator to obtain the data outliers.

Original: The test set is constructed by mixing nearly equal number of normal samples and outlier samples.
Revised: The test set is constructed by mixing nearly equal number of normal samples and outlier samples.

Original: We choose the density evaluated on the chosen features of representations as the criterion for detection.
Revised: We choose the density evaluated on the chosen w i ’s as the criterion for detection.

Original: The values of area under receiver operating characteristic (AUROC) in Figure 5 reﬂect the effectiveness of the criterion in outlier detection (Hendrycks and Gimpel, 2016).
Revised: The values of area under receiver operating characteristic (AUROC) in Figure 5 reﬂect the effectiveness of the criterion in outlier detection (Hendrycks and Gimpel, 2016).

Original: The MI criterion again shows signiﬁcant advantages over the VAR criterion.
Revised: The MI criterion again shows signiﬁcant advantages over the VAR criterion.

Original: In both the classiﬁcation and the outlier detection tasks, it seems that involving all the latent variables can still work well, though we may pay a huge computational price when the dimension of data is high.
Revised: In both the classiﬁcation and the outlier detection tasks, it seems that involving all the latent variables can still work well, though we may pay a huge computational price when the dimension of data is high.

Original: The situation is different in practice.
Revised: The situation is different in practice.

Original: When too many noise variables are involved in the density evaluation, the signal provided by the informative latent variables may not be able to outstand.
Revised: When too many noise variables are involved in the density evaluation, the signal provided by the informative latent variables may not be able to outstand.

Original: This becomes clear in our experiments on EMNIST in the next section.
Revised: This becomes clear in our experiments on EMNIST in the next section.

Original:  P ERFORMANCE ON EMNIST
Revised:  P ERFORMANCE ON EMNIST

Original: In this section, we investigate the performance of the MI criterion on the EMNIST dataset.
Revised: In this section, we investigate the performance of the MI criterion on the EMNIST dataset.

Original: The models are trained on the EMNIST-digits dataset following the same network architecture adopted in Sorrenson et al.
Revised: The models are trained on the EMNIST-digits dataset with the same network architecture adopted by Sorrenson et al.

Original: and the objective function shown in Eq 7 with trainable Gaussian mixture distribution parameters.
Revised: and the objective function shown in Eq 5.

Original: For this experiment, the data x ∈ R 28 × 28 are the digits images,  auxiliary variable u will be the label of digits and  w ∈ R 784 will be the estimated representation output by the GIN model.
Revised: For this experiment, the data x ∈ R 28 × 28 are the digits images, the auxiliary variable u is the label of digits and the learned feature vector w is 784 dimensional.

Original: The real dataset by no means satisﬁes the exact assumptions required by the identiﬁability theorem.
Revised: The real dataset by no means satisﬁes the exact assumptions required by the identiﬁability theorem.

Original: And hence experiments on real datasets can help us understand how much of reality these assumptions capture.
Revised: And hence experiments on real datasets can help us understand how much of reality these assumptions capture.

Original: Since the underlying latent variables are not available for real datasets, we cannot check the performance of the representation learning and the informative latent variable selection algorithms by visualising their projected distribution.
Revised: Since the underlying latent variables are not available for real datasets, we cannot directly verify the performance of the representation learning and the informative latent variable selection algorithms by visualising their projected distribution.

Original: A common method is to visualise the sample generation quality by traversing the latent space to demonstrate the performance.
Revised: A common method is to visualise the sample generation quality by traversing the latent space to demonstrate the performance.

Original: But this is still very qualitative and arbitrary to subjective interpretation.
Revised: But this is still very qualitative and arbitrary to subjective interpretation.

Original: Instead of these visualisation methods, we choose three downstream tasks: classiﬁcation, outlier detection and adversarial defence to measure the performance of the algorithm.
Revised: Instead of these visualisation methods, we choose three downstream tasks: classiﬁcation, outlier detection and adversarial defence, to measure the performance of the algorithm.

Original: The classiﬁcation tasks are solved in a similar way with last section by computing the posterior probability of labels conditioned on the test samples using features selected by each criterion.
Revised: The classiﬁcation tasks are solved in a similar way as in last section by computing the posterior probability of labels conditioned on the test samples using features selected by each criterion.

Original: The result is shown in the left subﬁgure in Figure 6.
Revised: The result is shown in the left subﬁgure of Figure 6.

Original: With 30 latent variables identiﬁed by the MI criterion, the classiﬁcation accuracy can reach 94 .
Revised: With 30 latent variables identiﬁed by the MI criterion, the classiﬁcation accuracy can reach 94 .

Original: 15% , while using VAR criterion cannot gives us satisfactory accuracy.
Revised: 15% , while using VAR criterion cannot gives us satisfactory accuracy.

Original: Different from the phenomenon we observed on synthetic data, the accuracy degrades when we select too many features that have small mutual information with the auxiliary variable.
Revised: Different from the phenomenon we observed on synthetic data, the accuracy degrades when we select too many features that have small mutual information with the auxiliary variable.

Original: This indicates that the noise in the representation extracted from data may hurt the performance of downstream tasks, and thus accurately selecting the informative variables is crucial.
Revised: This indicates that the noise in the representation extracted from data may hurt the performance of downstream tasks, and thus accurately selecting the informative variables is crucial.

Original: From the right subﬁgure, we can see that the mutual information curve has a sharp decay point, which corresponds to the number of latent variables that gives the best classiﬁcation performance.
Revised: From the right subﬁgure, we can see that the mutual information curve has a sharp decay point, which corresponds to the number of latent variables that gives the best classiﬁcation performance.

Original: In contrast, recall that on synthetic data, involving noise variables does not hurt the classiﬁcation accuracy.
Revised: In contrast, recall that on synthetic data, involving noise variables does not hurt the classiﬁcation accuracy.

Original: Our conjecture is that the GIN model trained on the synthetic data almost perfectly disentangled the informative and noise latent variables, which can be seen from the sharp decay of mutual information in Figure 3.
Revised: Our conjecture is that the GIN model trained on the synthetic data almost perfectly disentangled the informative and noise latent variables, which can be seen from the sharp decay of mutual information in Figure 3.

Original: Since the noise features in the representation contain little information of u , they cannot impact the calculation of posterior probability p ( u | x ) .
Revised: Since the noise features in the representation contain little information of u , they cannot impact the calculation of posterior probability p ( u | x ) .

Original: Therefore, in this case, involving noise variables does not hurt the performance.
Revised: Therefore, in this case, involving noise variables does not hurt the performance.

Original: However, for the model trained on real data, the features it learned are not all informative at the same level.
Revised: However, for the model trained on real data, the features it learned are not all informative at the same level.

Original: We can see this from the right subﬁgure of Figure 6, where many latent variables have mutual information with u around the level of 0 . 1 .
Revised: We can see this from the right subﬁgure of Figure 6, where many latent variables have mutual information with u around the level of 0 . 1 .

Original: These variables may not be able to provide more information about the labels than  ﬁrst 30 variables, but they may affect the predicted labels.
Revised: These variables may not be able to provide more information about the labels than the ﬁrst 30 variables, but they may affect the predicted labels.

Original: In particular, if the marginal distributions of these variables have small variances; that is, all components are clustered close with each other, then a small value shift on these variables will cause a huge change of density, which can dominate the overall density and change the predicted label.
Revised: In particular, if the marginal distributions of these variables have small variances; that is, all components are clustered close with each other, then a small value shift on these variables will cause a huge change of density, which can dominate the overall density and change the predicted label.

Original: This also explains why the variance criterion works better when more than 200 variables are included: The variables with relatively large variances are not likely to strongly affect the overall density.
Revised: This also explains why the variance criterion works better when more than 200 variables are included: The variables with relatively large variances are not likely to strongly affect the overall density.

Original: This conjecture can be further investigated in the future.
Revised: This conjecture can be further investigated in the future.

Original: Nevertheless, it is worth noting that as the low mutual information entries are very likely encoding no additional information to the previous ones and mainly encode noise, we should not include them for classiﬁcation in principle.
Revised: Nevertheless, it is worth noting that as the low mutual information entries are very unlikely to encode additional information than those with high mutual information, and thus we should not include them for downstream tasks in principle.

Original: In practice, one only need to select a truncation level and use it for classiﬁcation.

Original: As a result, the quicker deterioration of larger truncation levels (by including more non-informative latents) of the MI criterion does not hurt its top performance for classiﬁcation purpose.

Original: To investigate the outliers detection capability, we use the model trained on EMNIST-digits to distinguish EMNIST-letters from EMNIST-digits samples.
Revised: To investigate the outlier detection capability, we use the model trained on EMNIST-digits to distinguish EMNIST-letters from EMNIST-digits samples.

Original: Using the density as the detection threshold, we plot the ROC curves for the 30 variables selected by the MI criterion, referred to as MI30, and the 300 variables selected by the VAR criterion, referred to as VAR300.
Revised: Using the density as the detection threshold, we plot the ROC curves for the 30 variables selected by the MI criterion, referred to as MI30, and the 300 variables selected by the VAR criterion, referred to as VAR300.

Original: The number of latent variables are chosen at the best classiﬁcation accuracy level.
Revised: The number of latent variables are chosen at the best classiﬁcation accuracy level.

Original: Figure 7 shows that MI30 dominates the
Revised: Figure 7 shows that MI30 dominates the

Original: VAR300.
Revised: VAR300.

Original: The AUROC value of the MI30 is 0 .
Revised: The AUROC value of the MI30 is 0 .

Original: 763 , while the value of VAR300 is 0 .
Revised: 763 , while the value of VAR300 is 0 .

Original: Adversarial examples are an extreme case of outliers.
Revised: Adversarial examples are an extreme case of outliers.

Original: These examples deviate from normal samples so little that human vision cannot identify the difference, but they can fool the classiﬁcation model and change its prediction.
Revised: These examples deviate from normal samples so little that human vision cannot identify the difference, but they can fool the classiﬁcation model and change its prediction.

Original: Li et al.
Revised: Li et al.

Original: (2019) shows that conditional variational autoencoders are more robust than discriminative models against adversarial attacks when used together with the Bayes rule for the classiﬁcation purpose and carefully designed defence mechanism.
Revised: (2019) shows that conditional variational autoencoders are more robust than discriminative models against adversarial attacks when used together with the Bayes rule for the classiﬁcation purpose and carefully designed defence mechanism.

Original: Here we are not trying to show if the GIN is a better model for adversarial robustness than other probabilistic models.
Revised: Here we are not trying to show if the GIN is a better model for adversarial robustness than other probabilistic models.

Original: Instead, we want to show that using the mutual information criterion can help GIN models achieve better adversarial robustness.
Revised: Instead, we want to show that using the mutual information criterion can help GIN models achieve better adversarial robustness.

Original: In the experiment, we attack the model with the widely-used fast gradient sign method (FGSM)
Revised: In the experiment, we attack the model with the widely-used fast gradient sign method (FGSM)

Original: (Goodfellow et al., 2015).
Revised: (Goodfellow et al., 2015).

Original: By attempting adversarial attack at different levels, we ﬁnd out that without applying defence, the classiﬁer based on p S ( u | x ) is not robust no matter for MI30 or VAR300.
Revised: By attempting adversarial attack at different levels, we ﬁnd out that without applying defence, the classiﬁer based on p S ( u | x ) is not robust no matter for MI30 or VAR300.

Original: For the model using MI30, the optimal attacking strength is η = 0 .
Revised: For the model using MI30, the optimal attacking strength is η = 0 .

Original: As shown in Table 1, at that level, the prediction of the model can be changed by nearly 99% adversarial examples.
Revised: As shown in Table 1, at that level, the prediction of the model can be changed by nearly 99% adversarial examples.

Original: At the same level, the model using VAR300 can only be successfully fooled by 56% examples.
Revised: At the same level, the model using VAR300 can only be successfully fooled by 56% examples.

Original: However, we cannot simply conclude that VAR300 should be preferred than MI30 because its clean accuracy is much lower and η = 0 .
Revised: However, we cannot simply conclude that VAR300 should be preferred than MI30 because its clean accuracy is much lower and η = 0 .

Original: 02 is not the optimal attacking strength for VAR300.
Revised: 02 is not the optimal attacking strength for VAR300.

Original: To defend the adversarial attack, we deﬁne the following rejection rule: If p S ( w ( x ) | u ) < p S ( µ y − k σ u | u ) (here k = 1 by our choice) for all label u , the criterion reports x as an adversarial example and refuses to classify it.
Revised: To defend the adversarial attack, we deﬁne the following rejection rule: If p S ( w ( x ) | u ) < p S ( µ y − k σ u | u ) (here k = 1 by our choice) for all label u , the criterion reports x as an adversarial example and refuses to classify it.

Original: Note that for MI30 and VAR300, their subset S of features are different.
Revised: Note that for MI30 and VAR300, their subset S of features are different.

Original: At the level η = 0 .
Revised: At the level η = 0 .

Original: 02 , this defence strategy on MI30 can recognise about 94% attacks and thus only 5 .
Revised: 02 , this defence strategy on MI30 can recognise about 94% attacks and thus only 5 .

Original: 8% of all attacks can succeed.
Revised: 8% of all attacks can succeed.

Original: If the attacker wants to bypass the defence, it has to choose a much weaker attacking strength and the overall success rate is still quite low.
Revised: If the attacker wants to bypass the defence, it has to choose a much weaker attacking strength and the overall success rate is still quite low.

Original: This is the case for MI30 when η = 0 .
Revised: This is the case for MI30 when η = 0 .

Original: 001 as shown in Table 1 .
Revised: 001 as shown in Table 1 .

Original: However, if we use VAR300, such a defence strategy has almost no effect at level η = 0 .
Revised: However, if we use VAR300, such a defence strategy has almost no effect at level η = 0 .

Original: 001 and very limited effect at level η = 0 .
Revised: 001 and very limited effect at level η = 0 .

Original: An interesting observation on the no attack case ( η = 0 ) is that the error rate after applying the defence module is slightly lower than the error rate without defence.
Revised: An interesting observation on the no attack case ( η = 0 ) is that the error rate after applying the defence module is slightly lower than the error rate without defence.

Original: Though it does not mean that the overall classiﬁcation accuracy is improved because around 1 − 0 .
Revised: Though it does not mean that the overall classiﬁcation accuracy is improved because around 1 − 0 .

Original: 916% = 0 .
Revised: 916% = 0 .

Original: 084% test samples are screened out and do not get their predicted labels, it does imply that these test samples are overall harder for the model, which is not conﬁdent in its predictions on them.
Revised: 084% test samples are screened out and do not get their predicted labels, it does imply that these test samples are overall harder for the model, which is not conﬁdent in its predictions on them.

Original: For VAR300, we did not observe such an effect.
Revised: For VAR300, we did not observe such an effect.

Original: All these results indicate that for defending adversarial attacks, features selected by the MI criterion are more reliable.
Revised: All these results indicate that for defending adversarial attacks, features selected by the MI criterion are more reliable.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In the experiments on synthetic data
Revised: In the experiments on synthetic data

Original: [Table]
Revised: [Table]

Original: we have two interesting discoveries.
Revised: we have two interesting discoveries.

Original: First, when we use GIN to learn the synthetic data, not only the informative variables can be successfully recovered as predicted by the theory, but also can the noise variables be learned.
Revised: First, when we use GIN to learn the synthetic data, not only the informative variables can be successfully recovered as predicted by the theory, but also can the noise variables be learned.

Original: Actually, the features in the learned representation w is very close to a permutation of the true latent variables ( z , (cid:15) ) .
Revised: Actually, the features in the learned representation w is very close to a permutation of the true latent variables ( z , (cid:15) ) .

Original: This is demonstrated in Figure 9 by plotting the correlation matrix between w and ( z , (cid:15) ) .
Revised: This is demonstrated in Figure 9 by plotting the correlation matrix between w and ( z , (cid:15) ) .

Original: This result indicates that the identiﬁability theorem established by the current nonlinear ICA theory may be extended to a stronger version.
Revised: This result indicates that the identiﬁability theorem established by the current nonlinear ICA theory may be extended to a stronger version.

Original: In the identiﬁability theorem of linear ICA (Comon, 1994), the transforms allowed are restricted to linear ones while distributions of data are only required not to be Gaussian.
Revised: In the identiﬁability theorem of linear ICA (Comon, 1994), the transforms allowed are restricted to linear ones while distributions of data are only required not to be Gaussian.

Original: Compared to the linear ICA theory, current nonlinear ICA theory requires a stronger assumption on latent distributions, which have to be in an exponential family, but has no constraints on the transforms.
Revised: Compared to the linear ICA theory, current nonlinear ICA theory requires a stronger assumption on latent distributions, which have to be in an exponential family, but has no constraints on the transforms.

Original: Note that in the synthetic data experiments, both the generating and the learning models are ﬂows.
Revised: Note that in the synthetic data experiments, both the generating and the learning models are ﬂows.

Original: If we restrict the nonlinear transform to certain types of neural networks, we may be able to drop the exponential family assumption on their latent variables but still get identiﬁability.
Revised: If we restrict the nonlinear transform to certain types of neural networks, we may be able to drop the exponential family assumption on their latent variables but still get identiﬁability.

Original: More than the correct recovery of noise variables, we even ﬁnd out that the variances of latent variables are approximately preserved, in particular when the variances of noise variables are much smaller ( 10 − 4 for the following example) than those of the informative variables; see Figure 8.
Revised: More than the correct recovery of noise variables, we even ﬁnd out that the variances of latent variables are approximately preserved, in particular when the variances of noise variables are much smaller ( 10 − 4 for the following example) than those of the informative variables; see Figure 8.

Original: This may be caused by the fact that the greatest and least singular values of the Jacobian of ﬂow-based models are both upper and lower bounded.
Revised: This may be caused by the fact that the greatest and least singular values of the Jacobian of ﬂow-based models are both upper and lower bounded.

Original: Because of this phenomena, the variance criteria can ﬁnd out the correct informative variables for low noise scenarios.
Revised: Because of this phenomena, the variance criteria can ﬁnd out the correct informative variables for low noise scenarios.

Original: Such a variance-preserving property is not stable when the variance of informative variables and noise have closer scales.
Revised: Such a variance-preserving property is not stable when the variance of informative variables and noise have closer scales.

Original: Therefore, the mutual information criterion has more reliable performance in informative latent variables selection.
Revised: Therefore, the mutual information criterion has more reliable performance in informative latent variables selection.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Based on our theoretical analysis and extensive experiments on synthetic and real data, we conclude that for any GIN type of models, the mutual information between representations and the auxiliary variable is a better criterion for informative latent variables selection than the previously proposed variance criterion.
Revised: Based on our theoretical analysis and extensive experiments on synthetic and real data, we conclude that for any GIN type of models, the mutual information between representations and the auxiliary variable is a better criterion for informative latent variables selection than the previously proposed variance criterion.

Original: We also show that by selecting the correct features in the representation, the GIN model can perform better on the classiﬁcation, outlier detection and adversarial defence tasks.
Revised: We also show that by selecting the correct features in the representation, the GIN model can perform better on the classiﬁcation, outlier detection and adversarial defence tasks.

Original: To further improve the performance on the real data, developing ﬁner and more efﬁcient informative variable selection method based on mutual information is an important future direction.

Original: The capability of GIN models perfectly recovering of all latent variables up to a permutation and a bounded scaling on synthetic data indicates the possibility of a stronger version of identiﬁability of nonlinear ICA using ﬂow-based models.
Revised: The capability of GIN models perfectly recovering of all latent variables up to a permutation and a bounded scaling on synthetic data indicates the possibility of a stronger version of identiﬁability of nonlinear ICA using ﬂow-based models.

Original: More importantly, the perspective that the auxiliary variable u provides a context for the informative latent variables selection eliminates the boundary between informative latent variables and noise.
Revised: More importantly, the perspective that the auxiliary variable u provides a context for the informative latent variables selection eliminates the boundary between informative latent variables and noise.

Original: It is thus possible that different auxiliary variables deﬁne different group of informative latent variables.
Revised: It is thus possible that different auxiliary variables deﬁne different group of informative latent variables.

Original: This will lead to representation of data with a ﬁner structure.
Revised: This will lead to representation of data with a ﬁner structure.

Original: A N ONLINEAR ICA
Revised: A N ONLINEAR ICA

Original: In this section, we brieﬂy review the framework of Nonlinear ICA  to give abetter context of discussions in the paper.
Revised: In this section, we provide the formal statements about the identiﬁability of Nonlinear ICA in Sorrenson et al. (2020) to give a context of discussions in the paper.

Original: In the problem setting of Nonlinear ICA, the data x ∈ R d is assumed to be generated by a latent variable z ∈ R n through a nonlinear function f with a noise variable (cid:15) .
Revised: In the problem setting of Nonlinear ICA, the data x ∈ R d is assumed to be generated by a latent variable z ∈ R n through a nonlinear function f with a noise variable (cid:15) .

Original: There are two different formulations about the generation process, where both formulations would normally allow n ≤ d .
Revised: There are two different formulations about the generation process, where both formulations would normally allow n ≤ d .

Original: By assuming n ≤ d , the framework is enabled to model data with intrinsic/informative support on a lower dimensional manifold.
Revised: By assuming n ≤ d , the framework is able to model data with intrinsic/informative support on a lower dimensional manifold.

Revised: • In Khemakhem et al.

Original: • In the ﬁrst setting, the nonlinear function f is assumed to be an injection from R n to R d and another noise variable (
Revised: (2020)’s setting, the nonlinear function f is assumed to be an injection from R n to R d and another noise variable (

Original: cid:15) ∈
Revised: cid:15) ∈

Original: R d is applied after f (Khemakhem et al., 2020), i.e.
Revised: R d is applied after f , i e.,

Original: [Equation]
Revised: [Equation]

Original: • In the second setting, the nonlinear function f is assumed to be an bijection from R d to R d and another noise variable (cid:15) ∈
Revised: • In Sorrenson et al. (2020)’s setting, the nonlinear function f is assumed to be a bijection from R d to R d and another noise variable (cid:15) ∈

Original: R d − n is concatenated to z before f (Sorrenson et al. [Equation] [Equation] e
Revised: R d − n is concatenated to z before f , i.e.,

Original: [Equation]
Revised: [Equation]

Original: In this work we are interested in Nonlinear ICA with ﬂow based models, which suits better with the second setting.
Revised: In this work we only discuss the second setting.

Original: Apart from f being injective in the ﬁrst setting and bijective in the second, normally we do not stipulate stronger assumptions on the generating function f .
Revised: Apart from f being injective in the ﬁrst setting and bijective in the second, normally we do not put more assumptions on the generating function f .

Original: As indicated by the name of ICA, we would require independence between entries in the latent variable z .
Revised: As indicated by the name of ICA, we would require independence among entries in the latent variable z .

Original: Although for linear ICA problems one usually assumes the mutual independence on entries in z , one would usually introduce an auxiliary variable u and assume the conditional independence z i | u ⊥ z
Revised: Although for linear ICA problems one usually assumes the mutual independence on entries in z , one would usually introduce an auxiliary variable u and assume the conditional independence z i | u ⊥ z

Original: j | u for any i (cid:54) =
Revised: j | u for any i (cid:54) =

Original: j due to the consideration of identiﬁability (Khemakhem et al., 2020).
Revised: j due to obtain identiﬁability.

Original: The identiﬁability, i.e., existence and uniqueness, of the solution is at thetheoretical core of independent component analysis.
Revised: The identiﬁability of the latent variables is at the core of independent component analysis theory.

Original: Without the existence of solutions, independent component analysis for variable x will not be well-deﬁned.

Original: Without the uniqueness of the solution, the latent variable z reconstructed by ICA can be very arbitrary and thus less meaningful.

Original: It is worth noting that simple transformations, e.g., scaling, shifting and permutation, usually do not hurt the interpretation of these independent components.
Revised: It is worth noting that simple transforms, e.g., scaling, shifting and permutation, of the underlying true latent variables do not hurt the indepedence/conditional independence among them.

Original: As a result, the uniqueness is actually in the quotient set where the equivalence relationshipis induced by these transformations.
Revised: As a result, the identiﬁability is actually considered under the equivalence relationship induced by these transforms.

Original: To provide a rigorous presentation of identiﬁability of Nonlinear ICA to readers, we recall the theory established by preceding work (Khemakhem et al., 2020; Sorrenson et al., 2020).

Original: More speciﬁcally, we cite the results from Sorrenson et al.
Revised: Next we review the results from Sorrenson et al.

Original: (2020), which also employs the second setting. We refer to theground truth relationship between the  latent  and data as  generation  and  the estimated relationship by nonlinear ICA algorithms as estimation.
Revised: We refer to the relationship between the ground truth latent variables and data as the generation process and that between the estimated latent variables and data as the estimation process.

Original: Notation 1.
Revised: Notation 1.

Original: (Generation) Assume n ≤ d
Revised: (Generation) Assume n ≤ d

Original: and we do not have (cid:15) for n = d .
Revised: and we do not have (cid:15) for n = d .

Original: Denote by x ∈ R
Revised: Denote by x ∈ R

Original: d
Revised: d

Original: the observable data.
Revised: the observable data.

Original: Denote by u the auxiliary variable which is also observable.
Revised: Denote by u the auxiliary variable which is also observable.

Original: u can be either a scalar or vector.
Revised: u can be either a scalar or vector.

Original: Denote by z ∈ R n := Z 1 × · · · × Z n the informative latent variable 4.
Revised: Denote by z ∈ R n := Z 1 × · · · × Z n the informative latent variable 4.

Original: Denote by (cid:15) ∈ R d − n the noise variable.
Revised: Denote by (cid:15) ∈ R d − n the noise variable.

Original: Denote by f a bijection from R d to R d .
Revised: Denote by f a bijection from R d to R d .

Original: Assumption 1.
Revised: Assumption 1.

Original: (Generation)
Revised: (Generation)

Original: 1. Assume the observable data x is generated as x = f ( z , (cid:15) ) .
Revised: 1. Assume the observable data x is generated as x = f ( z , (cid:15) ) .

Original: Assume the informative latent z conditioned on the auxiliary variable u admits an exponential family distribution
Revised: Assume the informative latent z conditioned on the auxiliary variable u admits an exponential family distribution

Original: [Equation]
Revised: [Equation]

Original: where T i,j ’s are the sufﬁcient statistics.
Revised: where T i,j ’s are the sufﬁcient statistics.

Original: λ i,j ’s are the coefﬁcients and Z i ’s are the normalising constants.
Revised: λ i,j ’s are the coefﬁcients and Z i ’s are the normalising constants.

Original: This assumption implies the conditional independence between entries in z .3.
Revised: This assumption implies the conditional independence between entries in z .

Original: Assume the distribution of (cid:15) must not dependent on u .
Revised: Assume the distribution of (cid:15) must not dependent on u .

Original: This assumption indicates that (cid:15) has zero mutual information with u by deﬁnition.
Revised: This assumption indicates that (cid:15) has zero mutual information with u by deﬁnition.

Original: Notation 2.
Revised: Notation 2.

Original: (Estimation)
Revised: (Estimation)

Original: Denote by w ∈ R d the estimated latent variable.
Revised: Denote by w ∈ R d the estimated latent variable.

Original: Denote by g θ : R d → R d a bijection from a parametric family with parameter θ ∈ Θ .
Revised: Denote by g θ : R d → R d a bijection from a parametric family with parameter θ ∈ Θ .

Original: We may omit the parameter θ for notation simplicity.
Revised: We may omit the parameter θ for notation simplicity.

Original: Assumption 2.
Revised: Assumption 2.

Original: (Estimation)
Revised: (Estimation)

Original: Assume there exists a θ † ∈ Θ such that x = g θ † ( w ) .
Revised: Assume there exists a θ † ∈ Θ such that x = g θ † ( w ) .

Original: Assume the estimated latent variable w conditioned on u also admits an exponential family distribution
Revised: Assume the estimated latent variable w conditioned on u also admits an exponential family distribution

Original: [Equation]
Revised: [Equation]

Original: where T (cid:48)
Revised: where T (cid:48)

Original: i,j ’s are the sufﬁcient statistics.
Revised: i,j ’s are the sufﬁcient statistics.

Original: λ (cid:48)
Revised: λ (cid:48)

Original: i,j ’s are the coefﬁcients and Z (cid:48)
Revised: i,j ’s are the coefﬁcients and Z (cid:48)

Original: i ’s are the normalising constants.
Revised: i ’s are the normalising constants.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Sorrenson et al., 2020)
Revised: (Sorrenson et al., 2020)

Original: In addition to Assumption 2 and 1, we further assume:
Revised: In addition to Assumption 2 and 1, we further assume:

Original: The sufﬁcient statistics T i,j ( z ) ’s are differentiable almost everywhere and their derivatives dT i, [Equation] are nonzero almost surely for all z ∈ Z i and all 1 ≤ i ≤ n and 1 ≤ j ≤ k . 2.
Revised: The sufﬁcient statistics T i,j ( z ) ’s are differentiable almost everywhere and their derivatives dT i,j ( z ) dz are nonzero almost surely for all z ∈ Z i and all 1 ≤ i ≤ n and 1 ≤ j ≤ k . 2.

Original: There exist nk + 1 different values of u , i.e., u 0 , . . .
Revised: There exist nk + 1 different values of u , i.e., u 0 , . . .

Original: , u nk such that the matrix
Revised: , u nk such that the matrix

Original: [Equation]
Revised: [Equation]

Original: of size nk × nk is invertible
Revised: where λ ( u [Equation] =

Original: [Equation]
Revised: [Equation]

Revised: is invertible.

Original: Then the sufﬁcient statistics of the generating latent space are related to those of the estimated latent space by the following relationship:
Revised: Then the sufﬁcient statistics of the generating latent space are related to those of the estimated latent space by the following equation:

Original: [Equation]
Revised: [Equation]

Original: where A is a constant, full row rank nk × dk matrix and c is a constant vector.
Revised: where A is a constant, full row rank nk × dk matrix and c is a constant vector.

Original: From Theorem 1, we can see thatthere exists an relationship from the estimated latent  to the ground truth latent and the relationship is unique.
Revised: From Theorem 1, we can see that the estimated latent variables are connected to the ground truth latent variables by the unique linear relationship.

Original: Since we do not have further knowledge about the matrix A , the information from z may be still entangled in w for general cases.
Revised: Without further knowledge about the matrix A , entries of z are still entangled in w in general.

Original: However, for two-parameter exponential family distributions, each row in the mixing matrix A only has one non-zero element (Sorrenson et al., 2020).
Revised: However, for two-parameter exponential family distributions, each row in the mixing matrix A only has one non-zero element (Sorrenson et al., 2020).

Original: Here we recall the results for Gaussian distributions and refer interested readers to Sorrenson et al.
Revised: Here we recall the results for Gaussian distributions and refer interested readers to Sorrenson et al.

Original: (2020) for more results.
Revised: (2020) for more results.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Sorrenson et al., 2020) Assume both p ( z | u ) and p ( w | u ) are Gaussian distributions, then each row in A only has one non-zero entry and each column has at most one non-zero entry.
Revised: (Sorrenson et al., 2020) Assume both p ( z | u ) and p ( w | u ) are Gaussian distributions, then each row in A only has one non-zero entry and each column has at most one non-zero entry.

Original: In other words, for each z i in z = [ z 1 , . . . , z n ] , there exist unique w j , a i (cid:54) = 0 and b i , such that
Revised: In other words, for each z i in z = [ z 1 , . . . , z n ] , there exist unique w j , a i (cid:54) = 0 and b i , such that

Original: [Equation]
Revised: [Equation]

Original: From Theorem 2, we can see that for each z i , there is only one w j estimating it up to an afﬁne transform.
Revised: From Theorem 2, we can see that for each z i , there is only one w j estimating it up to an afﬁne transform.

Original: It indicates that there are n entries in w that can disentangle attributes in z .

Original: B G ENERAL I NCOMPRESSIBLE - FLOW N ETWORK
Revised: B G ENERAL I NCOMPRESSIBLE - FLOW N ETWORK

Original: A General Incompressible-ﬂow Network (GIN) is a ﬂow-based model integrating the features of both RealNVP
Revised: A General Incompressible-ﬂow Network (GIN) is a ﬂow-based model integrating the features of both RealNVP

Original: (Dinh et al., 2017) and NICE (Dinh et al., 2015).
Revised: (Dinh et al., 2017) and NICE (Dinh et al., 2015).

Original: Flow based models, a.k.a.
Revised: Flow based models, a.k.a.

Original: normalising ﬂows, describe discrete dynamics for random variables and thus induce transports of probabilistic densities via change of variable.
Revised: normalising ﬂows, describe discrete dynamics for random variables and thus induce transports of probabilistic densities via change of variable.

Original: In the continuous time limit, neural networks can also be used to model the time derivatives and thus deﬁne neural ODEs (Chen et al., 2018a).
Revised: In the continuous time limit, neural networks can also be used to model the time derivatives and thus deﬁne neural ODEs (Chen et al., 2018a).

Original: The absolute value of the determinant of the Jacobian matrix in the formula of change of variable usually requires highly complex computation.
Revised: The absolute value of the determinant of the Jacobian matrix in the formula of change of variable usually requires highly complex computation.

Original: The design of ﬂows calls for the balance between sufﬁcient capacity of the model family for ﬂexible modelling and sophisticated structure of the models for efﬁcient computation.
Revised: The design of ﬂows calls for the balance between sufﬁcient capacity of the model family for ﬂexible modelling and sophisticated structure of the models for efﬁcient computation.

Original: RealNVP approaches the problem by stacking coupling layers with form
Revised: RealNVP approaches the problem by stacking coupling layers with form

Original: [Equation]
Revised: [Equation]

Original: where the input x and output y are respectively divided into two parts ( x 1 , x 2 ) and ( y 1 , y 2 ) .
Revised: where the input x and output y are respectively divided into two parts ( x 1 , x 2 ) and ( y 1 , y 2 ) .

Original: Due to the clever construction, evaluation of the Jacobian’s determinant is reduced to the summation of s ( x 1 ) .
Revised: Due to the clever construction, evaluation of the Jacobian’s determinant is reduced to the summation of s ( x 1 ) .

Original: The computation of the determinant can be completely eliminated by constructing s ( · ) to maintain a zero sum.
Revised: The computation of the determinant can be completely eliminated by constructing s ( · ) to maintain a zero sum.

Original: In the coupling structure of NICE, there is no elementwise multiplication on x 2 , which de facto deﬁnes a zero sum s ( · ) .
Revised: In the coupling structure of NICE, there is no elementwise multiplication on x 2 , which de facto deﬁnes a zero sum s ( · ) .

Original: The GIN paper proposed to output the last entry of s ( · ) as the negative sum of previous entries.
Revised: The GIN paper proposed to output the last entry of s ( · ) as the negative sum of previous entries.

Original: As summing over huge amount of dimensions might cause numerical issues in extreme cases, we replace it with a mean subtraction mechanism
Revised: As summing over huge amount of dimensions might cause numerical issues in extreme cases, we replace it with a mean subtraction mechanism

Original: [Equation]
Revised: [Equation]

Original: where s ( x 1 ) i is the i -th entry of the output vector of s ( x 1 ) and p is dimension of s ( x 1 ) .
Revised: where s ( x 1 ) i is the i -th entry of the output vector of s ( x 1 ) and p is dimension of s ( x 1 ) .

Original: A later release of the ofﬁcial GIN implementation also employs this mechanism independently.
Revised: A later release of the ofﬁcial GIN implementation also employs this mechanism independently.

Original: Note that by using zero-output-sum network, we could not only save the computation of the Jacobian’s determinant but also preserve the volume of the object passing the ﬂow.
Revised: Note that by using zero-output-sum network, we could not only save the computation of the Jacobian’s determinant but also preserve the volume of the object passing the ﬂow.

Original: C T RAINABLE STATISTICS VERSUS EMPIRICAL STATISTICS
Revised: C T RAINABLE STATISTICS VERSUS EMPIRICAL STATISTICS

Original: As we mentioned in Section 3, there are two different objectives proposed by GIN: trainable statistics versus batch evaluations.
Revised: As we mentioned in Section 3, there are two different objectives proposed by GIN: trainable statistics versus batch evaluations.

Original: To compare the performance of these two approaches, we also train the batch evaluation version of GIN on synthetic data for two different variance choices of (cid:15) : σ (cid:15)
Revised: To compare the performance of these two approaches, we also train the batch evaluation version of GIN on synthetic data for two different variance choices of (cid:15) : σ (cid:15)

Original: i
Revised: i

Original: =
Revised: =

Original: and σ (cid:15) i = 10 − 2 .
Revised: and σ (cid:15) i = 10 − 2 .

Original: The results are shown in Figure 10 and Figure 11, and they are very similar to the results of trainable statistics in Figure 3 and Figure 8.
Revised: The results are shown in Figure 10 and Figure 11, and they are very similar to the results of trainable statistics in Figure 3 and Figure 8.

Original: [Figure]
Revised: [Figure]

Original: D E XPERIMENTS C ONFIGURATIONS
Revised: D E XPERIMENTS C ONFIGURATIONS

Original: Since the main scope of our paper is to compare the mutual information criterion with the variance criterion for informative latent variable identiﬁcation and to show the beneﬁt for downstream[Figure] tasks, we adopt the same conﬁgurations of network structure and training methods with the original GIN paper.
Revised: Since the main scope of our paper is to compare the mutual information criterion with the variance criterion for informative latent variable identiﬁcation and to show the beneﬁt for downstream tasks, we adopt the same conﬁgurations of network structure and training methods with the original GIN paper.

Original: The networks are composed of serial coupling layers and dimension permutation layers.
Revised: The networks are composed of serial coupling layers and dimension permutation layers.

Original: While the random permutation could allow different segmentation of the variable x and add additional ﬂexibility on coupling layers, it will not affect the Jacobian determinant’s evaluation.
Revised: While the random permutation could allow different segmentation of the variable x and add additional ﬂexibility on coupling layers, it will not affect the Jacobian determinant’s evaluation.

Original: The permutations are applied on the length axis for synthetic data and the channel axis for the EMNIST data.
Revised: The permutations are applied on the length axis for synthetic data and the channel axis for the EMNIST data.

Original: For the EMNIST data, the coupling layers are mainly based on convolutional layers, while  for the synthetic data, the coupling layers are based on fully connected layers.
Revised: For the EMNIST data, the coupling layers are mainly based on convolutional layers, while [Figure] for the synthetic data, the coupling layers are based on fully connected layers.

Original: Besides, for EMNIST data, the model also adopts invertible downsampling layers proposed in Jacobsen et al.
Revised: Besides, for EMNIST data, the model also adopts invertible downsampling layers proposed in Jacobsen et al.

Original: (2018) and fully connected couplings after a ﬂattening layer.
Revised: (2018) and fully connected couplings after a ﬂattening layer.

Original: More details can be found in Appendix D of
Revised: More details can be found in Appendix D of

Original: Sorrenson et al.
Revised: Sorrenson et al.

Original: Here we provide more information about the data used in our experiments.
Revised: Here we provide more information about the data used in our experiments.

Original: For the synthetic data, the parameters of Gaussian mixtures are randomly generated from uniform distributions.
Revised: For the synthetic data, the parameters of Gaussian mixtures are randomly generated from uniform distributions.

Original: As with the GIN paper, we use interval [ − 5 , 5] to sample the expectations and [0 . 5 , 3] to sample the standard deviation of z 1 , z 2 .
Revised: As with the GIN paper, we use interval [ − 5 , 5] to sample the expectations and [0 . 5 , 3] to sample the standard deviation of z 1 , z 2 .

Original: For the one shown in Section 3, the statistics are  
Revised: For the one shown in Section 3, the statistics are  

Original: [Equation]
Revised: [Equation]

Revised: 

Original:   2 .4 .−3 . [Equation]
Revised:  =   2 .4 .−

Original:  3 .− 4 .4 .4 .2{+.
+} .4 .−
Revised: − 3 .− 4 .4 .4 {+.
+} .4 .−

Original: 4 .  and [Equation]  1 .1 .2 .2 .0 .2 .2 .1 .0 .{+.
+}
Revised: 4 .  and   σ 1 σσ 3 σσ  =   1 .1 .2 .2 .0 {+.
+}

Original:   (18)
Revised:   (16)

Original: And the distributions of OOD samples come from two Gaussian distributions with statistics ˆ µ 1 =
Revised: And the distributions of OOD samples come from two Gaussian distributions with statistics ˆ µ 1 =

Original: [ − 20 , 20] ˆ σ 1 =
Revised: [ − 20 , 20] ˆ σ 1 =

Original: [1 . 34 , 1 . 35] and 
Revised: [1 . 34 , 1 . 35] and ˆ µ 2 =

Original: [Equation]
Revised: ˆ σ 2 =

Original: Figure 12 and 13 respectively show all the possible 2D projections of the ground truth latent variable ( z , (cid:15) ) and observation x adopted in the results represented in Section 3.
Revised: Figure 12 and 13 respectively show all the possible 2D projections of the ground truth latent variable ( z , (cid:15) ) and observation x adopted in the results represented in Section 3.

Original: Figure 14 shows the estimated representation w by the trained GIN model shown in Section 3.
Revised: Figure 14 shows the estimated representation w by the trained GIN model shown in Section 3.

Original: While the correct informative latent dimensions are reconstructed within w , the mutual information criterion can correctly identify them but the variance criterion cannot.
Revised: While the correct informative latent dimensions are reconstructed within w , the mutual information criterion can correctly identify them but the variance criterion cannot.