Original: T OWARDS N ONLINEAR D ISENTANGLEMENT IN N ATURAL D ATA WITH T EMPORAL S PARSE C ODING
Revised: T OWARDS N ONLINEAR D ISENTANGLEMENT IN N ATURAL D ATA WITH T EMPORAL S PARSE C ODING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Disentangling the underlying generative factors from complex data has so far been limited to carefully constructed scenarios.
Revised: Disentangling the underlying generative factors from complex data has so far been limited to carefully constructed scenarios.

Original: We propose a path towardsevaluation on natural data by ﬁrst showing that the statistics of natural data provide enough structure to enable disentanglement, both theoretically and empirically.
Revised: We propose a path towards natural data by ﬁrst showing that the statistics of natural data provide enough structure to enable disentanglement, both theoretically and empirically.

Original: Speciﬁcally, we provide evidence that objects in natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution.
Revised: Speciﬁcally, we provide evidence that objects in natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution.

Original: To address this ﬁnding we provide a novel proof that relies on a sparse prior on temporally adjacent observations to recover the true latent variables up to permutations and sign ﬂips,directly providing a stronger result than previous work.
Revised: To address this ﬁnding we provide a novel proof that relies on a sparse prior on temporally adjacent observations to recover the true latent variables up to permutations and sign ﬂips, providing a stronger result than previous work.

Original: We show that equipping practical estimation methods with our prior often surpasses the current state-of-the-art on several established benchmark datasets without any impractical assumptions, such as knowledge of the number of changing generative factors.
Revised: We show that equipping practical estimation methods with our prior often surpasses the current state-of-the-art on several established benchmark datasets without any impractical assumptions, such as knowledge of the number of changing generative factors.

Original: Furthermore, we contribute two new benchmarks,
Revised: Furthermore, we contribute two new benchmarks,

Original: Natural Sprites and KITTI Masks, which integrate the measured natural dynamics to enable disentanglement evaluation with more realistic datasets.
Revised: Natural Sprites and KITTI Masks, which integrate the measured natural dynamics to enable disentanglement evaluation with more realistic datasets.

Original: We leverage these benchmarks to test our theory, demonstrating improved performance.
Revised: We leverage these benchmarks to test our theory, demonstrating improved performance.

Original: We also identify non-obvious challenges for current methods in scaling to more natural domains.
Revised: We also identify non-obvious challenges for current methods in scaling to more natural domains.

Original: Taken together our work addresses key issues in disentanglement research for moving towards more natural settings.
Revised: Taken together our work addresses key issues in disentanglement research for moving towards more natural settings.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural scene understanding can be achieved by decomposing the signal into its underlying factors of variation.
Revised: Natural scene understanding can be achieved by decomposing the signal into its underlying factors of variation.

Original: An intuitive approach for this problem(Bengio et al., 2013) assumes that a visual representation of the world can be constructed via a generative process that receives factors as input and produces natural signals as output.
Revised: An intuitive approach for this problem assumes that a visual representation of the world can be constructed via a generative process that receives factors as input and produces natural signals as output (Bengio et al., 2013).

Original: Although untrue in the literal sense, this point of view is justiﬁed by the fact that our world is composed of distinct entities that can vary independently, but with regularity imposed by physics.
Revised: This assumption is justiﬁed by the fact that our world is composed of distinct entities that can vary independently, but with regularity imposed by physics.

Revised: What makes this approach appealing is that it formalizes representation learning by directly comparing representations to underlying ground-truth state, as opposed to the indirect evaluation of benchmarking against heuristic downstream tasks (e.g. object recognition).

Original: A desired solution to this problem isto avoid non-identiﬁability , which means a set of possible solutions may all appear equally valid to the model, while only one identiﬁes the true generative factors.
Revised: However, the core issue with this approach is non-identiﬁability , which means a set of possible solutions may all appear equally valid to the model, while only one identiﬁes the true generative factors.

Original: Our work is motivated by the question of whether identiﬁability can be achieved using assumptions that are predicated on natural data.
Revised: Our work is motivated by the question of whether the statistics of natural data will allow for the formulation of an identiﬁable model.

Original: Our core observation that enables us to make progress in addressing this question is that  natural data have sparse transitions .
Revised: Our core observation that enables us to make progress in addressing this question is that generative factors of natural data have sparse transitions .

Original: To estimate the generative factors of natural data, we compute statistics on measured transitions of area and position for object masks from largescale, natural, unstructured videos.
Revised: To estimate these generative factors, we compute statistics on measured transitions of area and position for object masks from large-scale, natural, unstructured videos.

Original: Speciﬁcally, we extracted over 300,000  object segmentation mask transitions from YouTube-VOS (Xu et al., 2018;
Revised: Speciﬁcally, we extracted over 300,  object segmentation mask transitions from YouTube-VOS (Xu et al., 2018;

Original: Yang et al., 2019) and KITTI-MOTS (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) (discussed in detail in Appendix H).
Revised: Yang et al., 2019) and KITTI-MOTS (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) (discussed in detail in Appendix H).

Original: We ﬁt generalized Laplace distributions to the collected data (equation (2)), which we indicate with red lines in Fig.
Revised: We ﬁt generalized Laplace distributions to the collected data (Eq. 2), which we indicate with red lines in Fig.

Original: We see empirically that all  distributions of temporal transitions are highly sparse.
Revised: We see empirically that all marginal distributions of temporal transitions are highly sparse and there exists complex dependencies between natural factors (e.g. motion typically affects both position and apparent size).

Revised: In this study, we focus on the sparse marginals, which we believe constitutes an important advance that sets the stage for solving further issues and eventually applying the technology to real-world problems.

Original: With this information at hand, we are able to provide a  proof for capturing the underlying generative factors of the data up to permutations and sign ﬂipsin the case of sparse transitions that is not covered by previous work (Hyvärinen and Morioka, 2016).
Revised: With this information at hand, we are able to provide a stronger proof for capturing the underlying generative factors of the data up to permutations and sign ﬂips that is not covered by previous work (Hyvärinen and Morioka, 2016; 2017; Khemakhem et al., 2020a).

Revised: Thus, we present the ﬁrst work, to the best of our knowledge, which proposes a theoretically grounded solution that covers the statistics observed in real videos.

Original: In total, our contributions are: With measurements from unstructured natural video annotations we provide evidence that natural generative factors undergo sparse changes across time.
Revised: In total, our contributions are: With measurements from unstructured natural video annotations we provide evidence that natural generative factors undergo sparse changes across time.

Original: We provide a proof of identiﬁability that relies on the observed sparse innovations to identify nonlinearly mixed sources up to a permutation and sign-ﬂip, which we then validate with practical estimation methods for empirical comparisons.
Revised: We provide a proof of identiﬁability that relies on the observed sparse innovations to identify nonlinearly mixed sources up to a permutation and sign-ﬂip, which we then validate with practical estimation methods for empirical comparisons.

Original: We leverage the natural scene information to create novel datasets where the latent transitions between frames follow natural statistics, providing a benchmark to evaluate the practical capabilities of models in uncovering the true latent generative factors in the presence of realistic dynamics.
Revised: We leverage the natural scene information to create novel datasets where the latent transitions between frames follow natural statistics, providing a benchmark to evaluate the practical capabilities of models in uncovering the true latent generative factors in the presence of realistic dynamics.

Original: We demonstrate improved disentanglement over previous models on existing datasets and our contributed ones with quantitative metrics from both the disentanglement (Locatello et al., 2018) and the nonlinear ICA community (Hyvärinen and Morioka, 2016).
Revised: We demonstrate improved disentanglement over previous models on existing datasets and our contributed ones with quantitative metrics from both the disentanglement (Locatello et al., 2018) and the nonlinear ICA community (Hyvärinen and Morioka, 2016).

Original: We show via numerous visualization techniques that the learned representations for competing models have important differences, even when quantitative metrics suggest that they are performing equally well.
Revised: We show via numerous visualization techniques that the learned representations for competing models have important differences, even when quantitative metrics suggest that they are performing equally well.

Original:  R ELATED W ORK – D ISENTANGLEMENT AND N ONLINEAR ICA
Revised:  R ELATED W ORK – D ISENTANGLEMENT AND N ONLINEAR ICA

Original: Disentangled representation learning has its roots in blind source separation (Jutten and Herault, 1991) and shares goals  ﬁelds such as inverse graphics (Kulkarni et al., 2015;
Revised: Disentangled representation learning has its roots in blind source separation (Jutten and Herault, 1991) and shares goals with ﬁelds such as inverse graphics (Kulkarni et al., 2015;

Original: Yildirim et al., 2020) and developing models of invariant neural computation (Wiskott and Sejnowski, 2002; Hyvärinen and
Revised: Yildirim et al., 2020) and developing models of invariant neural computation (Wiskott and Sejnowski, 2002; Hyvärinen and

Original: Hoyer, 2000)(see Bengio et al., 2013, for a review).
Revised: Hoyer, 2000)(see Bengio et al., 2013, for a review).

Original: A disentangled representation would be valuable for a wide variety of machine learning applications, including sample efﬁciency for downstream tasks (Locatello et al., 2018; Gao et al., 2019), fairness (Locatello et al., 2019; Creager et al., 2019) and interpretability (Bengio et al., 2013; Higgins et al., 2017; Adel et al., 2018).
Revised: A disentangled representation would be valuable for a wide variety of machine learning applications, including sample efﬁciency for downstream tasks (Locatello et al., 2018; Gao et al., 2019), fairness (Locatello et al., 2019; Creager et al., 2019) and interpretability (Bengio et al., 2013; Higgins et al., 2017; Adel et al., 2018).

Original: Since there is no agreed upon deﬁnition of disentanglement in the literature, we adopt two common measurable criteria: i) each encoding element represents a single generative factor and ii) the values of generative factors are trivially decodable from the encoding (Ridgeway and Mozer, 2018;
Revised: Since there is no agreed upon deﬁnition of disentanglement in the literature, we adopt two common measurable criteria: i) each encoding element represents a single generative factor and ii) the values of generative factors are trivially decodable from the encoding (Ridgeway and Mozer, 2018;

Original: Eastwood and Williams, 2018).
Revised: Eastwood and Williams, 2018).

Original: Uncovering the underlying factors of variation has been a long-standing goal in independent component analysis (ICA)
Revised: Uncovering the underlying factors of variation has been a long-standing goal in independent component analysis (ICA)

Original: (Jutten and Herault, 1991; Bell and Sejnowski, 1995; Hyvärinen and Pajunen, 1999; Barron and Malik, 2012; Sohl-Dickstein et al., 2010).
Revised: (Jutten and Herault, 1991; Bell and Sejnowski, 1995; Hyvärinen and Pajunen, 1999; Barron and Malik, 2012; Sohl-Dickstein et al., 2010).

Original: ICA (Comon, 1994) provides an identiﬁable solution for disentangling data mixed via  linear generators receiving non-Gaussian factors as input.
Revised: ICA (Comon, 1994) provides an identiﬁable solution for disentangling data mixed via an invertible linear generator receiving at most one Gaussian factor as input.

Original: Recently, unsupervised approaches towards this goal have largely been based on Variational Autoencoders (VAEs) (Kingma and Welling, 2013) and have assumed that the data is independent and identically distributed ( i.i.d. )
Revised: Recently, unsupervised approaches for nonlinear generators have largely been based on Variational Autoencoders (VAEs) (Kingma and Welling, 2013) and have assumed that the data is independent and identically distributed ( i.i.d. )

Original: (Locatello et al., 2018) even though nonlinear methods that make this i.i.d.
Revised: (Locatello et al., 2018) even though nonlinear methods that make this i.i.d.

Original: assumption have been proven to be non-identiﬁable (Hyvärinen and Pajunen, 1999;
Revised: assumption have been proven to be non-identiﬁable (Hyvärinen and Pajunen, 1999;

Original: Locatello et al., 2018).
Revised: Locatello et al., 2018).

Original: Nonetheless, the bottom-up approach of starting with a nonlinear generator that produces well-controlled data has led to considerable achievements in understanding nonlinear disentanglement in VAEs (Higgins et al., 2017; Burgess et al., 2018; Rolinek et al., 2019; Chen et al., 2018), consolidating ideas from neural computation and machine learning (Khemakhem et al., 2020), and seeking a principled deﬁnition of disentanglement (Higgins et al., 2018; Ridgeway, 2016; Eastwood and Williams, 2018; Hyvärinen and Pajunen, 1999).
Revised: Nonetheless, the bottom-up approach of starting with a nonlinear generator that produces well-controlled data has led to considerable achievements in understanding nonlinear disentanglement in VAEs (Higgins et al., 2017; Burgess et al., 2018; Rolinek et al., 2019; Chen et al., 2018), consolidating ideas from neural computation and machine learning (Khemakhem et al., 2020a), and seeking a principled deﬁnition of disentanglement (Higgins et al., 2018; Ridgeway, 2016; Eastwood and Williams, 2018; Hyvärinen and Pajunen, 1999).

Original: Recently, Hyvärinen and colleagues (Hyvärinen and Morioka, 2016; 2017; Hyvärinen et al., 2018;
Revised: Recently, Hyvärinen and colleagues (Hyvärinen and Morioka, 2016; 2017; Hyvärinen et al., 2018;

Original: Khemakhem et al., 2020) showed that a solution to the non-identiﬁability of nonlinear ICA can be found by assuming that generative factors are conditioned on an additional observed variable, such as past states or the time index itself.
Revised: Khemakhem et al., 2020a) showed that a solution to identiﬁable nonlinear ICA can be found by assuming that generative factors are conditioned on an additional observed variable, such as past states or the time index itself.

Original: This contribution was generalized by Khemakhem et al.
Revised: This contribution was generalized by Khemakhem et al.

Original: (2020) past the nonlinear ICA domain to any consistent parameter estimation method for deep latent-variable models, including the VAE framework.
Revised: (2020a) past the nonlinear ICA domain to any consistent parameter estimation method for deep latent-variable models, including the VAE framework.

Original: However, the theoretical assumptions underlying this branch of work do not account for the sparse transitions we observe in the statistics of natural scenes, which we discuss in further detail in appendix G. Another branch of work requires some form of supervision (Szabó et al., 2017; Shu et al., 2019; Locatello et al., 2020), namely regarding the number of factors changed.
Revised: However, the theoretical assumptions underlying this branch of work do not account for the sparse transitions we observe in the statistics of natural scenes, which we discuss in further detail in appendix G.1.1. Another branch of work requires some form of supervision (Szabó et al., 2017; Shu et al., 2019; Locatello et al., 2020), namely regarding the number of factors changed.

Original: Locatello et al.
Revised: Locatello et al.

Original: (2020) proposed an approach to estimate this required knowledge at runtime, however their assumptions underlying identiﬁability may not be satisﬁed in practice.
Revised: (2020) proposed an approach to estimate this required knowledge at runtime, however their assumptions underlying identiﬁability may not be satisﬁed in practice.

Revised: We select two of the above approaches, that are both different in their formulation and state-of-the-art in their respective empirical settings, Hyvärinen and Morioka (2017);

Revised: Locatello et al.

Original:  The motivation of our method and dataset contributions is to address these limitations and to enable unsupervised disentanglement learning in more naturalistic scenarios 1 .
Revised: (2020), for our experiments below. The motivation of our method and dataset contributions is to address these limitations and to enable unsupervised disentanglement learning in more naturalistic scenarios 1 .

Original: The fact that physical processes bind generative factors in temporally adjacent natural video segments has been thoroughly explored for learning in neural networks (Hinton, 1990; Földiák, 1991; Mitchison, 1991; Wiskott and Sejnowski, 2002; Hyvärinen et al., 2003; Denton and Birodkar, 2017).
Revised: The fact that physical processes bind generative factors in temporally adjacent natural video segments has been thoroughly explored for learning in neural networks (Hinton, 1990; Földiák, 1991; Mitchison, 1991; Wiskott and Sejnowski, 2002; Hyvärinen et al., 2003; Denton and Birodkar, 2017).

Original: We propose a method that uses time information in the form of an L 1 - sparse temporal prior, which is motivated by the natural scene measurements presented above as well as by previous work (Simoncelli and
Revised: We propose a method that uses time information in the form of an L 1 - sparse temporal prior, which is motivated by the natural scene measurements presented above as well as by previous work (Simoncelli and

Original: Olshausen, 2001; Olshausen, 2003; Hyvärinen et al., 2003; Cadieu and Olshausen, 2012).
Revised: Olshausen, 2001; Olshausen, 2003; Hyvärinen et al., 2003; Cadieu and Olshausen, 2012).

Original: Such a prior would intuitively allow for sharp changes in some latent factors, while most other factors remain unchanged between adjacent time-points.
Revised: Such a prior would intuitively allow for sharp changes in some latent factors, while most other factors remain unchanged between adjacent time-points.

Original: Almost all similar methods are variants of slow feature analysis (SFA, Wiskott and Sejnowski, 2002), which measure slowness in terms of the Euclidean (i.e.
Revised: Almost all similar methods are variants of slow feature analysis (SFA, Wiskott and Sejnowski, 2002), which measure slowness in terms of the Euclidean (i.e.

Original: L 2 , or log Gaussian) distance between temporally adjacent encodings.
Revised: L 2 , or log Gaussian) distance between temporally adjacent encodings.

Original: Related to our approach, a probabilistic interpretation of SFA has been previously proposed (Turner and Sahani, 2007), as well as extensions to variational inference (Grathwohl and Wilson, 2016).
Revised: Related to our approach, a probabilistic interpretation of SFA has been previously proposed (Turner and Sahani, 2007), as well as extensions to variational inference (Grathwohl and Wilson, 2016).

Original: Additionally, Hashimoto (2003) suggested that a sparse (Cauchy) slowness prior improves correspondence to biological complex cells over the L 2 slowness prior in a two-layer model.
Revised: Additionally, Hashimoto (2003) suggested that a sparse (Cauchy) slowness prior improves correspondence to biological complex cells over the L 2 slowness prior in a two-layer model.

Original: However, to the best of our knowledge an
Revised: However, to the best of our knowledge, an

Original: L 1 temporal prior has previously only been used in deep auto-encoder frameworks when applied to semi-supervised tasks (Mobahi et al., 2009; Zou et al., 2012), and was mentioned in (Cadieu and
Revised: L 1 temporal prior has previously only been used in deep auto-encoder frameworks when applied to semi-supervised tasks (Mobahi et al., 2009; Zou et al., 2012), and was mentioned in Cadieu and

Original: Olshausen, 2012), which used an L 2 prior, but claimed that an L 1 prior performed similarly on their task.
Revised: Olshausen (2012), who used an L 2 prior, but claimed that an L 1 prior performed similarly on their task.

Original: Inspired by Hyvärinen et al.
Revised: Similar to Hyvärinen et al.

Original: (Hyvärinen and Morioka, 2016; Hyvärinen et al., 2018), we only assume that the latent factors are temporally dependent, thus avoiding assuming knowledge of the number of factors where the two observations differ (Shu et al., 2019),
Revised: (Hyvärinen and Morioka, 2016; Hyvärinen et al., 2018), we only assume that the latent factors are temporally dependent, thus avoiding assuming knowledge of the number of factors where the two observations differ (Shu et al., 2019;

Original: which Locatello et al.
Revised: Locatello et al., 2020).

Original: (2020) estimate during evaluation. Most of the standard datasets for disentanglement (dSprites (Matthey et al., 2017), Cars3D (Reed et al., 2015), SmallNORB (LeCun et al., 2004), Shapes3D (Kim and Mnih, 2018), MPI3D (Gondal et al., 2019)) have been compiled into a disentanglement library (DisLib) by Locatello et al.
Revised: Most of the standard datasets for disentanglement (dSprites (Matthey et al., 2017), Cars3D (Reed et al., 2015), SmallNORB (LeCun et al., 2004), Shapes3D (Kim and Mnih, 2018), MPI3D (Gondal et al., 2019)) have been compiled into a disentanglement library (DisLib) by Locatello et al.

Original: However, all of the DisLib datasets are limited in that the data generating process is independent and identically distributed ( i.i.d. ) and all generative factors are assumed to be discrete.
Revised: However, all of the DisLib datasets are limited in that the data generating process is independent and identically distributed ( i.i.d. ) and all generative factors are assumed to be discrete.

Original: In a follow-up study, Locatello et al.
Revised: In a follow-up study, Locatello et al.

Original: (2020) proposed combining pairs of images such that only k factors change, as this matches their modeling assumptions required to prove identiﬁability, where k ∈ U{ 1 , D − 1 } and
Revised: (2020) proposed combining pairs of images such that only k factors change, as this matches their modeling assumptions required to prove identiﬁability. Here, k ∈ U{ 1 , D − 1 } and

Original: D denotes the number of ground-truth factors, which are then sampled uniformly.
Revised: D denotes the number of ground-truth factors, which are then sampled uniformly.

Original: We additionally use the measurements from Fig.
Revised: We additionally use the measurements from Fig.

Original: 1 to construct datasetswhich can be used for evaluating disentanglement, but where the change in pairs of images directly corresponds to natural dynamics.
Revised: 1 to construct datasets for evaluating disentanglement that have time transitions which directly correspond to natural dynamics.

Original:  T HEORY 3. G ENERATIVE M ODEL
Revised:  T HEORY 3. G ENERATIVE M ODEL

Original: We have provided evidence to support the hypothesis that generative factors of natural videos have sparse temporal transitions (see Fig. 1).
Revised: We have provided evidence to support the hypothesis that generative factors of natural videos have sparse temporal transitions (see Fig. 1).

Original: To mimic this process, we assume temporally adjacent input pairs ( x t − 1 , x t ) coming from a nonlinear generator that maps factors to images x = g ( z ) , where pairs of generative factors form a Markov chain:
Revised: To model this process, we assume temporally adjacent input pairs ( x t − 1 , x t ) coming from a nonlinear generator that maps factors to images x = g ( z ) , where pairs of generative factors form a Markov chain:

Original: [Equation]
Revised: [Equation]

Original: Assume the observed data ( x t , x t − 1 ) comes from the following generative process:
Revised: Assume the observed data ( x t , x t − 1 ) comes from the following generative process, where the latent factors are assumed to be independent (cf. Appendix G.2):

Original: [Equation]
Revised: [Equation]

Original: where λ is the distribution rate, p ( z t − 1 ) is a factorized Gaussian prior N ( 0 , I ) (as in Kingma and
Revised: where λ is the distribution rate, p ( z t − 1 ) is a factorized Gaussian prior N ( 0 , I ) (as in Kingma and

Original: Welling, 2013) and p ( z t | z t − 1 ) is a factorized generalized Laplace distribution (Bethge, 2006; Sinz et al., 2009) with shape parameter α , which determines the shape and especially the kurtosis of the function 2 .
Revised: Welling, 2013) and p ( z t | z t − 1 ) is a factorized generalized Laplace distribution (Subbotin, 1923) with shape parameter α , which determines the shape and especially the kurtosis of the function 2 .

Original: Intuitively, smaller α implies larger kurtosis and sparser temporal transitions of the generative factors (special cases are Gaussian, α = 2 , and Laplacian, α = 1 ).
Revised: Intuitively, smaller α implies larger kurtosis and sparser temporal transitions of the generative factors (special cases are Gaussian, α = 2 , and Laplacian, α = 1 ).

Original: Critically, for our proof we assume α < 2 to ensure that temporal transitions are sparse.
Revised: Critically, for our proof we assume α < 2 to ensure that temporal transitions are sparse.

Original: Like Locatello et al. (2020), we assume that noise is modeled indirectly as a latent variable and is manifested through the generator g . The novelty of our approach lies in our explicit modeling of sparse transitions, which results in a stronger identiﬁability proof than previously achieved, a scalable implementation, and assurance that we cover the transition statistics of natural data (Hyvärinen and Morioka, 2017; Khemakhem et al., 2020) (see Appendix G for a more detailed comparison).
Revised: The novelty of our approach lies in our explicit modeling of sparse transitions that cover the statistics of natural data, which results in a stronger identiﬁability proof than previously achieved (see Appendix G.1.1 for a more detailed comparison with Hyvärinen and Morioka, 2017; Khemakhem et al., 2020a).

Original: 3. I DENTIFIABILITY P ROOF
Revised: 3. I DENTIFIABILITY P ROOF

Original: Theorem 1 For a ground-truth ( g ∗ , λ ∗ , α ∗ ) and a learned ( g, λ, α ) model as deﬁned in equation (2) , if the functions g ∗ and g are injective and differentiable almost everywhere, λ ∗ = λ , α ∗ =
Revised: Theorem 1 For a ground-truth ( g ∗ , λ ∗ , α ∗ ) and a learned ( g, λ, α ) model as deﬁned in Eq. (2), if the functions g ∗ and g are injective and differentiable almost everywhere, λ ∗ = λ , α ∗ =

Original: α < 2 (i.e. there is no model misspeciﬁcation) and the distributions of pairs of images generated from the priors z ∗ ∼ p ∗ ( z ) andz ∼ p ( z ) are matched ( g ∗ ( z ∗ t − 1 ) , g ∗ ( z ∗ t ))
Revised: α < 2 (i.e. there is no model misspeciﬁcation) and the distributions of pairs of images generated from the priors z ∗ ∼ p ∗ ( z ) and

Original: = ( g  ( z  t − 1 ) , g  ( z  t ))  almost everywhere, then g = g ∗ ◦ σ , where σ is a permutation with sign ﬂipping.
Revised: z ∼ p ( z ) generated as ( g ∗ ( z ∗ t − 1 ) , g ∗ ( z ∗ t )) and ( g ( z t − 1 ) , g ( z t )) , respectively, are matched almost everywhere, then g = g ∗ ◦ σ , where σ is a permutation with sign ﬂipping.

Original: The formal proof is provided in Appendix B.1.
Revised: The formal proof is provided in Appendix B.1.

Revised: Similar to linear ICA, but in the temporal domain, we have to assume that the transitions of generative factors across time be non-Gaussian.

Original: It implies that if the temporal changes of ground-truth factors are sparse, then the only generator consistent with the observations is the ground-truth one (up to a permutation and sign ﬂips).
Revised: Speciﬁcally, if the temporal changes of ground-truth factors are sparse, then the only generator consistent with the observations is the ground-truth one (up to a permutation and sign ﬂips).

Original: The main idea behind the proof is to represent g as g ∗ ◦ h and note that if h were not a permutation, then the distributions (( g ∗ ◦ h )( z t − 1 ) , ( g ∗ ◦ h )( z t )) and ( g ∗ ( z ∗ t − 1 ) , g ∗ ( z ∗ t )) would not match, due to the injectivity of g ∗ .
Revised: The main idea behind the proof is to represent g as g ∗ ◦ h and note that if h were not a permutation, then the distributions (( g ∗ ◦ h )( z t − 1 ) , ( g ∗ ◦ h )( z t )) and ( g ∗ ( z ∗ t − 1 ) , g ∗ ( z ∗ t )) would not match, due to the injectivity of g ∗ .

Original:  Fig.
Revised: Whether or not these distributions are the same is equivalent to whether or not the distributions of pairs ( z t − 1 , z t ) and ( h ( z t − 1 ) , h ( z t )) are the same. For these distributions to be the same, the function h must preserve the Gaussian marginal for the ﬁrst time step as well as the joint distribution, implying that it must preserve both the vector lengths and distances in the latent space. As we argue in the extended proof, this can only be the case if h is a composition of permutations and sign ﬂips. Intuition Fig.

Original: 2 illustrates why the model deﬁned in equation (2) has a unique generator.
Revised: 2 illustrates, by contradiction, why the model deﬁned in Eq. (2) is identiﬁable.

Original: The crucial point is that we consider  pairs of images, which differ effectively in only a few factors of variation due to the sparse  prior ( α < 2).
Revised: We consider temporal pairs of latents represented by connected points. A sparse transition prior encourages axis-alignment, as can be seen from the Laplace transition prior in the third image of Fig. 3.

Original: The 2D latent representations of such pairs are shown as blue points connected by lines.
Revised: This results in lines that are parallel with the axes in both the ground truth (left, blue, z ∗ ) and learned model (right, red, z ).

Original: If thefunction h were not a permutation, it would map these latent points into a conﬁguration in which many of the connecting lines are not parallel to one of the axes. However, the ground-truth generator, g ∗ , should still map this transformed latent space to the observed pairs of images differing in a few factors of variation, which contradicts it being injective.
Revised: In this example, z ∗ 0 corresponds to horizontal position, while z ∗ 1 corresponds to vertical position. The learned model must satisfy two criteria: (1) the latent factors should match the sparse prior (axis-aligned) and (2) the generated image pairs should match the ground-truth image pairs. If the learned latent factors were mismatched, for example by rotation, then the image pair distributions would not be matched.

Original: Moreover,

Original: Fig. 2 also illustrates that a rotationally symmetric Gaussian prior ( α = 2 ) on the transitions in either the generative or  the learned model would make the problem non-identiﬁable. Similar to linear ICA, but in  the temporal domain, we therefore have to assume that the transitions of generative factors across time be non-Gaussian.
Revised: In this example, the ground truth model would produce image pairs with typically vertical or horizontal transitions, while the learned model pairs result in mostly diagonal transitions. Thus, the learned model cannot satisfy both criteria without aligning the latent axes with the ground-truth axes.

Original: 3. S LOW V ARIATIONAL A UTOENCODER
Revised: 3. S LOW V ARIATIONAL A UTOENCODER

Original: In order to validate our proof, we must choose a probabilistic latent variable model for estimating the data density.
Revised: In order to validate our proof, we must choose a probabilistic latent variable model for estimating the data density.

Original: We chose to build upon the framework of VAEs because of their efﬁciency in estimating a variational approximation to the ground truth posterior of a deep latent variable model (Kingma and Welling, 2013). 
Revised: We chose to build upon the framework of VAEs because of their efﬁciency in estimating a variational approximation to the ground truth posterior of a deep latent variable model (Kingma and Welling, 2013). We will refer to this model as SlowVAE .

Original: In Appendix C we also note their shortcomings and discuss alternative  approaches.
Revised: In Appendix C we also note their shortcomings and test alternative ﬂow-based approaches.

Original: The standard VAE objective assumes i.i.d.
Revised: The standard VAE objective assumes i.i.d.

Original: data and a standard normal prior with diagonal covariance on the learned latent representations z ∼ N ( 0 , I ) .
Revised: data and a standard normal prior with diagonal covariance on the learned latent representations z ∼ N ( 0 , I ) .

Original: To extend this to sequences, we assume the same functional form for our model prior as in equation (1) and (2).
Revised: To extend this to sequences, we assume the same functional form for our model prior as in Eq. (1) and Eq.

Original: The posterior of our model is independent across time steps.
Revised: The posterior of our model is independent across time steps.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: where µ i ( x ) and σ 2 i ( x ) are the input-dependent mean and variance of our model’s posterior.
Revised: where µ i ( x ) and σ 2 i ( x ) are the input-dependent mean and variance of our model’s posterior.

Original: We visualize this combination of priors and posteriors in Fig.
Revised: We visualize this combination of priors and posteriors in Fig.

Original: For a given pair of inputs ( x t , x t − 1 ), the full ELBO (which we derive in  B.2) can be written as
Revised: For a given pair of inputs ( x t , x t − 1 ), the full ELBO (which we derive in Appendix B.2) can be written as

Original: [Equation]
Revised: [Equation]

Original: where γ is a regularization term for the sparsity prior, analogous to β in β -VAEs (Higgins et al., 2017).
Revised: where γ is a regularization term for the sparsity prior, analogous to β in β -VAEs (Higgins et al., 2017) (technically, Eq. 4 is only an ELBO with γ ≤ 1 ).

Original: The ﬁrst term on the right-hand side is the log-likelihood (i.e. the negative reconstruction error, with p ( x t , x t − 1 | z t , z t − 1 ) parameterized by the decoder of the VAE), the second term is the KL to a normal prior as in the standard VAE and the last term is an expectation of the KL between the posterior at time step t and the conditional prior p ( z t | z t − 1 ) .
Revised: The ﬁrst term on the right-hand side is the log-likelihood (i.e. the negative reconstruction error, with p ( x t , x t − 1 | z t , z t − 1 ) parameterized by the decoder of the VAE), the second term is the KL to a normal prior as in the standard VAE and the last term is an expectation of the KL between the posterior at time step t and the conditional prior p ( z t | z t − 1 ) .

Original: The expectation in the last term is taken over samples from the posterior at the previous time step q ( z t − 1 | x t − 1 ) .
Revised: The expectation in the last term is taken over samples from the posterior at the previous time step q ( z t − 1 | x t − 1 ) .

Original: We observe that taking the mean µ ( x t − 1 )  as a single sample is sufﬁcient (analogous to the log-likelihood that is typically evaluated at a single sample from the posterior).
Revised: We observed empirically that taking the mean, µ ( x t − 1 ) , as a single sample produces good results, analogous to the log-likelihood that is typically evaluated at a single sample from the posterior (see (Blei et al., 2017) for context).

Original: In practice, we need to choose α , λ , and γ .
Revised: In practice, we need to choose α , λ , and γ .

Original: For the latter two, we can perform a random search for hyperparameters, as we discuss below.
Revised: For the latter two, we can perform a random search for hyperparameters, as we discuss below.

Original: For the former, a parsimonious choice would be α = 1 , corresponding to a Laplace prior on the ﬁrst temporal derivative of the latent code. For natural data, a [-value of α = 
-]
Revised: For the former, any α < 2 would break the general rotation symmetry by having an optimum for axis-aligned representations, which theorem 1 includes as a [-value of α = 
-]requirement for identiﬁability.

Original: .would provide a better ﬁt to Fig.
Revised: As can be seen in Figs.

Original: 1, but a Laplace prior also achieves the goal of breaking the general rotation symmetry by having an optimum for axisaligned representations, which is a requirement for identiﬁability. Therefore, we chose α = 1  for this study. We derive a closed-form expression of the resulting KL in Appendix B.2.
Revised: 1 and 11, α ≈ 0 .provides the best ﬁt to the ground-truth marginals. However, we used α = 1 as a parsimonious choice for SlowVAE, since the Laplace is a well-understood distribution that allows us to derive a simple closedform solution for the ELBO in Eq. 4, which we derive in Appendix B.2.

Original: 3. T OWARDS AN A PPROXIMATE T HEORY OF D ISENTANGLEMENT
Revised: 3. T OWARDS AN A PPROXIMATE T HEORY OF D ISENTANGLEMENT

Original: A number of our theoretical assumptions are violated in practice: After non-convex optimization, on a ﬁnite data sample, the distributions p ( x t , x t − 1 ) and p ∗ ( x t , x t − 1 ) are probably not perfectly matched.
Revised: A number of our theoretical assumptions are violated in practice: After non-convex optimization, on a ﬁnite data sample, the distributions p ( x t , x t − 1 ) and p ∗ ( x t , x t − 1 ) are probably not perfectly matched.

Original: In addition, the model assumptions on p ( z t , z t − 1 ) likely do not fully match the distribution of the ground truth factors.
Revised: In addition, the model assumptions on p ( z t , z t − 1 ) likely do not fully match the distribution of the ground truth factors.

Original: For example, the model may be misspeciﬁed such that α (cid:54) =
Revised: For example, the model may be misspeciﬁed such that α (cid:54) =

Original: α ∗ or λ (cid:54) = λ ∗ , or the chosen family of distributions may be incorrect altogether.
Revised: α ∗ or λ (cid:54) = λ ∗ , or the chosen family of distributions may be incorrect altogether.

Original: In the following section we will present results on several datasets where the marginal distributions p ( z t − 1 ) are drawn from a Uniform (not
Revised: In the following section we will present results on several datasets where the marginal distributions p ( z t − 1 ) are drawn from a Uniform (not

Original: Normal) distribution, and some of them are over unordered sets (categories) or bounded periodic spaces (rotation).
Revised: Normal) distribution, and some of them are over unordered sets (categories) or bounded periodic spaces (rotation).

Original: Also, in practice the model latent space is usually chosen to have more dimensions than the ground truth generative model.
Revised: Also, in practice the model latent space is usually chosen to have more dimensions than the ground truth generative model.

Original: On real data, factors of variation may be dependent (c.f.
Revised: On real data, factors of variation may be dependent (cf.

Original: Träuble et al., 2020; Yang et al., 2020).
Revised: Träuble et al., 2020; Yang et al., 2020).

Revised: We show this is the case on YouTube-VOS and KITTI-MOTS in Appendix H and we provide evidence that breaking these dependencies has no clear consequence on disentanglement in Appendix G.2.

Original: Furthermore, the hypothesis class G of learnable functions in the VAE architecture may not contain the invertible ground truth generator g ∗ / ∈ G .
Revised: Furthermore, the hypothesis class G of learnable functions in the VAE architecture may not contain the invertible ground truth generator g ∗ / ∈ G .

Revised: Despite these violations, we consider it a strength of our method that the practical implementation still achieves improved disentanglement over previous approaches.

Original: We note understanding the impact of these violations as an important focus area for continued progress towards developing a practical yet theoretically supported method for disentanglement on natural scenes.
Revised: However, we note understanding the impact of these violations as an important focus area for continued progress towards developing a practical yet theoretically supported method for disentanglement on natural scenes.

Original:  D ATASETS WITH N ATURAL T RANSITIONS
Revised:  D ATASETS WITH N ATURAL T RANSITIONS

Original: While the standard datasets compiled by DisLib are an important step towards real-world applications, they still assume the data is i.i.d. , which is decidedly untrue for natural data. 
Revised: While the standard datasets compiled by DisLib are an important step towards real-world applications, they still assume the data is i.i.d. , which is decidedly untrue for natural data. As described in section 2,

Original: As described in section 2, (Locatello et al., 2020) proposed uniformly sampling the number of factors to be changed, k , and changing said factors by uniformly sampling over the possible set of values. We enforce that k factors are changed for each transition, and refer tothis data as “UNI”.
Revised: Locatello et al. (2020) proposed uniformly sampling the number of factors to be changed, k = Rnd, and changing said factors by uniformly sampling over the possible set of values. What we refer to as “UNI” is a dataset variant modeled after the described scheme (Locatello et al., 2020) (further details in Appendix F.2).

Original: Considering our natural data analysis presented in Figure 1, such transitions are certainly unnatural.
Revised: Considering our natural data analysis presented in Figure 1, such transitions are certainly unnatural.

Original: Given the current state of evaluation, we provide a set of incrementally more complex datasets that we propose  should be included in the standard benchmark suite to indicate how disentanglement approaches will scale towards  natural data.
Revised: Given the current state of evaluation, we provide a set of incrementally more natural datasets which are otherwise comparable to existing work. We propose that said datasets should be included in the standard benchmark suite to provide a step towards disentanglement in natural data.

Original: Speciﬁcally, we construct datasets with (1) imposed sparse transitions, (2) augmented with natural continuous generative factors using measurements from unstructured natural videos, as well as (3) data from unstructured natural videos themselves, but provided in segmentation mask form to ensure visual complexity manageable for current methods.

Original: For the provided datasets, the object categories never change across transitions – reﬂecting natural object permanence.

Original: We summarize our contributed progression of increasingly complex and more natural datasets below, for further details, please refer to Appendix F).

Original:  Laplace Transitions (LAP) is a procedure for constructing image pairs from DisLib datasets by sampling from a  conditional distribution shown to ﬁt natural transitions well.
Revised: (1) Laplace Transitions (LAP) is a procedure for constructing image pairs from DisLib datasets by sampling from a sparse conditional distribution.

Original: For each groundtruth factor, the ﬁrst value in the pair is chosen i.i.d.
Revised: For each ground-truth factor, the ﬁrst value in the pair is chosen i.i.d.

Original: from the dataset and the second is chosen by weighting nearby factor values using Laplace distributed probabilities.
Revised: from the dataset and the second is chosen by weighting nearby factor values using Laplace distributed probabilities.

Original: If all factors remain constant (no transition), the sample is rejected as the pair would not result in any temporal learning signal.
Revised: LAP is a step towards natural data that closely resembles previous extensions of DisLib datasets to the time domain, but in a way that matches the marginal distribution of natural transitions (see Appendix E.1 for more details).

Original:  Natural Sprites consists of pairs of  sprite images rendered from a distribution augmented by sampling from natural statistics.
Revised: (2) Natural Sprites consists of pairs of rendered sprite images with generative factors sampled from real YouTube-VOS transitions.

Original: For a given image pair, the position and scale of the sprites are set using measured values from adjacent time pointsfor unique entities present in YouTube-VOS.
Revised: For a given image pair, the position and scale of the sprites are set using measured values from adjacent time points in YouTube-VOS.

Original: The sprite shapes  are simple, like dSprites, and, along with orientation, are ﬁxed for a given pair.
Revised: The sprite shapes and orientations are simple, like dSprites, and are ﬁxed for a given pair.

Revised: While ﬁxing shape follows the natural transitions of objects, it is unclear how to accurately estimate object orientation from the masks, and thus we ﬁxed the factor

Revised: to avoid introducing artiﬁcial transitions.

Original: We additionally consider a version of the dataset discretized to the granularity of  dSprites, allowing us to leverage the standard DisLibmetrics for evaluation  (see Appendix F.3 for  details).
Revised: We additionally consider a version that is discretized to the same number of object states as dSprites, which i) allows us to leverage the standard DisLib evaluation metrics and ii) helps isolate the effect of including natural transitions from the effect of increasing data complexity (see Appendix E.3 for more details).

Original:  KITTI Masks is composed of pedestrian segmentation masks from an autonomous driving vision benchmark KITTI-MOTS, thus with natural shapes and continuous natural transitions in all underlying factors.
Revised: (3) KITTI Masks is composed of pedestrian segmentation masks from an autonomous driving vision benchmark KITTI-MOTS, thus with natural shapes and continuous natural transitions in all underlying factors.

Original: We consider adjacent frames ( ∆ t  = 1 ), as well as frames with a  temporal gap ( ∆ t  = 5 ).
Revised: We consider adjacent frames which correspond to mean (∆ t ) = 0 . 05 s in physical time (we report the mean because of variable sampling rates in the original data); as well as frames with a larger temporal gap of mean (∆ t ) = 0 .

Revised: 15 s , which corresponds to samples of pairs that are at most 5 frames apart.

Revised: We show in Appendix H.3 that SlowVAE disentanglement performance increases and then plateaus as we continue to increase mean (∆ t ) .

Revised: In summary, we construct datasets with (1) imposed sparse transitions, (2) augmented with natural continuous generative factors using measurements from unstructured natural videos, as well as (3) data from unstructured natural videos themselves, but provided in segmentation mask form to ensure visual complexity manageable for current methods.

Revised: For the provided datasets, the object categories never change across transitions – reﬂecting natural object permanence.

Revised: Finally, as (2) and (3) use factor transitions measured from natural videos, they exhibit any natural statistical structure present for those factors, such as natural dependencies (see Appendix G.2 for further discussion on dependencies and Appendix E for further discussion on the datasets themselves).

Original:  E XPERIMENTS 5. E MPIRICAL S TUDIES
Revised:  E XPERIMENTS 5. E MPIRICAL S TUDIES

Original: We evaluate models using the DisLib implementation for the following supervised metrics: BetaVAE (Higgins et al., 2017); FactorVAE (Kim and Mnih, 2018); Mutual Information Gap (MIG, (Chen et al., 2018)); Disentanglement, Compactness, and Informativeness (DCI [Disentanglement], (Eastwood and Williams, 2018)); Modularity (Ridgeway and Mozer, 2018); and Separated
Revised: We evaluate models using the DisLib implementation for the following supervised metrics: BetaVAE (Higgins et al., 2017); FactorVAE (Kim and Mnih, 2018); Mutual Information Gap (MIG, (Chen et al., 2018)); Disentanglement, Compactness, and Informativeness (DCI [Disentanglement], (Eastwood and Williams, 2018)); Modularity (Ridgeway and Mozer, 2018); and Separated

Original: Attribute Predictability (SAP, (Kumar et al., 2018))
Revised: Attribute Predictability (SAP, (Kumar et al., 2018))

Original: (see Appendix D for metric details).
Revised: (see Appendix D for metric details).

Original: None of the
Revised: None of the

Original: DisLib metrics support ground-truth labels with continuous variation, which is required for evaluation  on the continuous Natural Sprites and KITTI Masks datasets.
Revised: DisLib metrics support ground-truth labels with continuous variation, which is required for evaluation [Table] on the continuous Natural Sprites and KITTI Masks datasets.

Original: To reconcile this, we measure the Mean
Revised: To reconcile this, we measure the Mean

Original: Correlation Coefﬁcient (MCC), a standard metric in the ICA literature that is applicable to continuous variables.
Revised: Correlation Coefﬁcient (MCC), a standard metric in the ICA literature that is applicable to continuous variables.

Original: We report mean and standard deviation across 10 random seeds and bold-face signiﬁcant (independent T-test, p < 0 . 05 ) improvements between models.
Revised: We report mean and standard deviation across 10 random seeds.

Original: In order to select the conditional prior regularization and the prior rate in an unsupervised manner [Table] we perform a random search over γ ∈ [1 , 16] and λ ∈ [1 , 10] and compute the recently proposed unsupervised disentanglement ranking (UDR) scores (Duan et al., 2020).
Revised: In order to select the conditional prior regularization and the prior rate in an unsupervised manner, we perform a random search over γ ∈ [1 , 16] and λ ∈ [1 , 10] and compute the recently proposed unsupervised disentanglement ranking (UDR) scores (Duan et al., 2020).

Original: We notice that the optimal values are close to γ = 10 and λ = 6 on most datasets [Table] and thus use these values for all experiments.
Revised: We notice that the optimal values are close to γ = 10 and λ = 6 on most datasets, and thus use these values for all experiments.

Original: We leave ﬁnding optimal values for speciﬁc datasets to future work [Table] but note that it is a strong advantage of our approach that it works well with the same model speciﬁcation across 14 datasets (counting LAP and UNI for DisLib, optional discretization for Natural Sprites, as well as different values for ∆ t with KITTI Masks), addressing a concern posed in (Locatello et al., 2018).
Revised: We leave ﬁnding optimal values for speciﬁc datasets to future work, but note that it is a strong advantage of our approach that it works well with the same model speciﬁcation across 13 datasets (counting LAP and UNI for DisLib and optional discretization for Natural Sprites), addressing a concern posed in (Locatello et al., 2018).

Original: Additional details on model selection and training can be found in Appendix E.2.
Revised: Additional details on model selection and training can be found in Appendix F.1.

Revised: Although we train on image pairs, our model does not need paired data points at test time.

Original: For all visualizations, we pick the models with the highest average score across the DisLib metrics. 
Revised: For all visualizations, we pick the models with the highest average score across the DisLib metrics. To compare our model fairly against other methods that also take image pairs as inputs [Table] we took

Original: 5. R ESULTS ON D IS L IB AND N EW B ENCHMARKS
Revised: Permutation-Contrastive Learning from nonlinear ICA (PCL, Hyvärinen and Morioka, 2017) and AdaGVAE, the leading method in the study by (Locatello et al., 2020). We scaled up the implementation of PCL for evaluation on our high-dimensional pixel inputs [Table] and note this method does not have any hyperparameters.

Revised: For Ada-GVAE, following the paper’s recommendations, we select β (per dataset) using the considered parameter set [1 , 2 , 4 , 6 , 8 , 16]

Original: [Table]
Revised: [Table]

Revised: and use the weakly supervised reconstruction loss as the unsupervised model selection criterion (Locatello et al., 2020)

Original: [Table]
Revised: [Table]

Original: datasets.
Revised: [Table]

Original: See Table 1 for details.
Revised: See Table 1 for details.

Original: Our proposed model requires pairs for the training data provided in DisLib, although we show through evaluation with LAP and  UNI that the speciﬁed conditional does not need to exactly match our theoretical assumptions. However, we do not need paired data points at test time, and thus it is possible to evaluate our model against previous i.i.d.
Revised: In Table 1 we demonstrate favorable performance compared to PCL and Ada-GVAE across all applicable metrics for discrete ground-truth variable datasets. The relative improvement on UNI is particularly surprising given the drastic mismatch between UNI and SlowVAE’s assumptions. In Appendix H, we report results for the remaining DisLib datasets, where the observed dSprites results largely transfer.

Original: approaches. In Table 1 we demonstrate favorable performance when compared to state of the art methods from nonlinear ICA (Hyvärinen et al., 2018) and disentanglement (Locatello et al., 2020) which leverage non- i.i.d.
Revised: We also outperform PCL with a (ﬂow-based) exact likelihood implementation of our slow transition prior in Appendix G.1.1.

Original: nature of the data for identiﬁability.
Revised: In

Original: In appendix H, we report results from a large-scale empirical test across metrics, datasets, and models to demonstrate that our model is superior in almost all settings. [Table] [Table]
Revised: Appendix G.3, we show that a model with a transition L 2 ( α = 2 ) prior performs much worse, conﬁrming our theoretical prediction.

Original: [Table] [Table] [Table] [Table]
Revised: On the KITTI Masks dataset, one source of variation in the data is the average temporal separation within pairs of images mean (∆ t ) .

Original: [Table] [Table] [Table] DIP-VAE-
Revised: We present two settings (mean (∆ t )

Original: I
Revised: = 0 .

Original: MCC = [Table] [Table]
Revised: 05 s , mean (∆ t ) = 0 .

Original: [Table] [Table]
Revised: 15 s )

Original: [Figure]
Revised: [Figure]

Original: Speciﬁcally, SlowVAE trained on UNI outperforms the previous state of the art in a majority of metrics on four out of the ﬁve benchmark datasets, with large gains speciﬁcally on dSprites, Shapes3D, and MPI3D. As expected, SlowVAE also performs better with the LAP dataset variant, which is better matched to the model assumptions, than the UNI dataset variant on almost datasets and metrics.

Original: We outperform β -VAE, which was chosen as representative of previous i.i.d. approaches, on all contributed datasets and metrics ( β = 8 , selected via UDR). On the KITTI Masks dataset, one source of variation in the data source is the parameter that determines the temporal distance between sampled frames. We run two settings ( ∆ t = 1 , ∆ t = 5 ) and observe that the beneﬁt of temporal information is more apparent at ∆ t = 5 (Table 2).
Revised: S h a p e

Original: This links to recent work by Tschannen et al.
Revised: S c a l e - R o t x - P o s y - P o s and observe a comparative increase in MCC for the latter (Table 2).

Original: (2019), who show that temporal separation between frame embeddings inﬂuences the representation that is learned from videos.
Revised: Namely, the increase in performance for larger time gap is more pronounced with SlowVAE than the baselines, resulting in a statistically signiﬁcant MCC gain. We provide details on the settings and ablate over the mean (∆ t ) parameter in Appendix H.3, where we observe a positive trend between mean (∆ t ) and MCC (reﬂecting Table 2, in Oord et al., 2018). Finally, we also verify that the transition distributions remain sparse despite the increase in this parameter (Appendix H.3).

Original: In Fig.
Revised: In Fig.

Original: 5, we can see thatthe SlowVAE has learned latent dimensions which have correspondence with the estimated ground truth factors of x/y-position and scale.
Revised: 4, we can see that SlowVAE has learned latent dimensions which have correspondence with the estimated ground truth factors of x/y-position and scale.

Original: Locatello et al.
Revised: Locatello et al.

Original: (2018) showed that all i.i.d.
Revised: (2018) showed that all i.i.d.

Original: models performed similarly across the DisLib datasets and metrics when testing was carefully controlled.
Revised: models performed similarly across the DisLib datasets and metrics when testing was carefully controlled.

Original: However, in Fig.
Revised: However, in Fig.

Original: 4 we observe that the different modeling assumptions result in differences in representation quality.
Revised: 5 we observe that the different modeling assumptions result in differences in representation quality.

Original: To construct the visuals, we ﬁrst compute the sorted correlation coefﬁcient matrix between the latents and generative factors, which we visualize in the top row.
Revised: To construct the visuals, we ﬁrst compute the sorted correlation coefﬁcient matrix between the latents and generative factors, which we visualize in the top row.

Original: The correlation matrix is sorted via linear sum assignment such that each ground-truth factor is non-greedily associated with the latent variable with highest correlation.
Revised: The correlation matrix is sorted via linear sum assignment such that each ground-truth factor is non-greedily associated with the latent variable with highest correlation.

Original: Each row of  scatter plots corresponds to an individual generative factor.
Revised: Each row of the scatter plots corresponds to an individual generative factor.

Original: In each subplot, the horizontal axis indicates the ground truth value, the vertical axis indicates the corresponding latent value, and the colors indicate object shape.
Revised: In each subplot, the horizontal axis indicates the ground truth value, the vertical axis indicates the corresponding latent value, and the colors indicate object shape.

Revised: The models displayed are those with the maximum average score across evaluated metrics.

Original: These plots aid in understanding how eachground-truth factor is encoded inthe latent space in a way that is more informative than exclusively visualizing latent traversals or embeddings of pairs of latent units (Cheung et al., 2014; Chen et al., 2016; Szabó et al., 2017; Ma et al., 2018).
Revised: The latent space visualizations leverage the known ground-truth factors to aid in understanding how each factor is encoded in a way that is more informative than exclusively visualizing latent traversals or embeddings of pairs of latent units (Cheung et al., 2014; Chen et al., 2016; Szabó et al., 2017; Ma et al., 2018).

Original: For example, in the third row, we observe that SlowVAE trained on UNI shows three sinusoidal oscillations withequal phase, but frequencies ∼ ω, 2 ω, and 4 ω , which correspond to the three distinct rotational symmetries of the shapes: heart, ellipse and square.
Revised: For example, in the third row, we observe that several models have a sinusoidal variation with frequencies ∼ ω, 2 ω, and 4 ω , which correspond to the three distinct rotational symmetries of the shapes: heart, ellipse and square.

Original: We also note that the two datasets where we see the smallest performance differences (SmallNORB,
Revised: This directly impacts MCC performance (third row in the MCC matrix), which measures rank correlation between the matching latent factor (an angular variable) and the ground truth, which encodes the angles with monotonically increasing indices. Furthermore, the square has a four-fold rotational symmetry and repeats after 90 ◦ , but it is represented in a full 360 ◦ rotation in the DisLib ground truth encoding format, resulting in different ground truth labels for identical input images. A similar observation can be made with respect to the categorical factors, which are also represented as ordinal ground truth variables. For example, the PCL correlation score (top left element in the PCL

Original:  Cars3D) have signiﬁcantly more discrete categories ( 50 , 183 ) than the other datasets ( 3 − 6 ).
Revised: MCC matrix) is quite high, while the corresponding shape correlation score for SlowVAE is quite low. However, if we consider the shape scatter plots, we clearly see that SlowVAE separates the three shapes more distinctively than PCL, only in an order that differs from the ground truth. One solution is to modify MCC to report the maximum correlation over all permutations of the ground truth assignments, although brute force methods for this would scale poorly with the number of categories. We also note that datasets where we see small performance differences among models (e.g., Cars3D) have signiﬁcantly more discrete categories (e.g., 183 ) than the other datasets ( 3 − 6 ).

Original: This could also explain why all models  perform comparably on the Natural Sprites datasets, where unlike KITTI Masks the ground truth evaluation includes categorical and angular variables.
Revised: This could also explain why all models considered in Table 1 and 2 perform comparably on the Natural Sprites datasets, where unlike KITTI Masks the ground truth evaluation includes categorical and angular variables.

Original: As can be seen in Figure 4, such variables give all methods trouble, denoting an aspect of evaluation which future work can focus on improving performance with respect to. ) latent unit that corresponds to the corresponding row in the MCC matrix.
Revised: We note that properly evaluating disentanglement is an ongoing area of research, with notable results in recent work (Higgins et al., 2018; Duan et al., 2020; Anonymous, 2021a;b).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We provide evidence to support the hypothesis that natural scenes exhibit highly sparse  transition probabilities, which wethen leverage to contribute a novel nonlinear ICA framework that is provably identiﬁable up to permutations and sign-ﬂips – a stronger result than has been achieved previously.
Revised: We provide evidence to support the hypothesis that natural scenes exhibit highly sparse marginal transition probabilities. Leveraging this ﬁnding, we contribute a novel nonlinear ICA framework that is provably identiﬁable up to permutations and sign-ﬂips — a stronger result than has been achieved previously.

Original: With the SlowVAE model we provide a parsimonious implementation that is inspired by a long history of learning visual representations from temporal data (Hinton, 1990; Földiák, 1991).
Revised: With the SlowVAE model we provide a parsimonious implementation that is inspired by a long history of learning visual representations from temporal data (Sutton, 1988; Hinton, 1990; Földiák, 1991).

Original: We apply this model tothe current metric-based disentanglement benchmarks and it outperforms existing approaches (Locatello et al., 2020; Hyvärinen et al., 2018) on aggregateacross almost all metrics and datasets without any tuning ofits hyperparameters to individual datasets.
Revised: We apply this model to current metric-based disentanglement benchmarks to demonstrate that it outperforms existing approaches (Locatello et al., 2020; Hyvärinen and Morioka, 2017) on aggregate without any tuning of hyperparameters to individual datasets.

Original: Additionally, we providea benchmark of novel video datasets to guide disentanglement research towards more natural domains.
Revised: We also provide novel video dataset benchmarks to guide disentanglement research towards more natural domains.

Original: Going beyond KITTI Masks, which is constrained, YouTube-VOS is a large-scale, unstructured, natural dataset with a sufﬁcient number of samples toalleviate overﬁtting concerns. The Natural Sprites dataset serves as a tractable bridge to facilitate progress on the way to the more challenging
Revised: We observe that these datasets have complex dependencies that our theory will have to be extended to account for, although we demonstrate with empirical comparisons the efﬁcacy of our approach. In addition to Natural Sprites and KITTI Masks, we suggest that YouTube-VOS will be valuable as a large-scale dataset that is unconstrained by object type and scenario for more advanced models.

Original: YouTube-VOS.
Revised: Variance in such categorical factors is problematic for evaluation due to the cited drawbacks of existing quantitative metrics, which should be addressed in tandem with scaling to natural data.

Original: Taken together, our dataset and model proposals set the stage for utilizing knowledge of natural scene statistics to advance unsupervised disentangled representation learning.
Revised: Taken together, our dataset and model proposals set the stage for utilizing knowledge of natural scene statistics to advance unsupervised disentangled representation learning.

Original: In our experiments we see that approximate identiﬁcation as measured by the different disentanglement metrics increases despite violations of theoretical assumptions, which is in line with prior studies (Shu et al., 2019; Khemakhem et al., 2020; Locatello et al., 2020).
Revised: In our experiments we see that approximate identiﬁcation as measured by the different disentanglement metrics increases despite violations of theoretical assumptions, which is in line with prior studies (Shu et al., 2019; Khemakhem et al., 2020a; Locatello et al., 2020).

Original: Nevertheless, future work should address gaining a better understanding of the theoretical and empirical consequences of such model misspeciﬁcations, in order to make the theory of disentanglement more predictive about empirically found solutions.
Revised: Nevertheless, future work should address gaining a better understanding of the theoretical and empirical consequences of such model misspeciﬁcations, in order to make the theory of disentanglement more predictive about empirically found solutions.

Original: A PPENDIX A Broader Impact
Revised: A PPENDIX A Broader Impact

Original: B Formal Methods
Revised: B Formal Methods

Original: B.1 Proof of Identiﬁability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.1 Proof of Identiﬁability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2 Kullback Leibler Divergence of Slow Variational Autoencoder . . . . . . . . . . .
Revised: B.2 Kullback Leibler Divergence of Slow Variational Autoencoder . . . . . . . . . . .

Original: C Choosing a Latent 
Revised: C Choosing a Latent Variable Model

Original: Variable Model

Original: D Disentanglement Metrics
Revised: D Disentanglement Metrics

Original: D.1 Mean Correlation Coefﬁcient . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1 Mean Correlation Coefﬁcient . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 DisLib Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .E Estimation Methods E.1 Model Training and Selection . . E.2 Implementation Details . .
Revised: D.2 DisLib Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F Natural Datasets
Revised: E Natural Datasets

Original: F. Laplace Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .F.
Revised: E.1 Laplace Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original:  YouTube-VOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: E.2 YouTube-VOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.3 Natural Sprites . . E.4 KITTI MOTS Pedestrian Masks . .

Revised: F Implementation Details

Original: F.
Revised: F.

Original: Natural Sprites . .
Revised: Model Training and Selection . .

Original: F.
Revised: F.

Original: KITTI MOTS Pedestrian Masks . .
Revised: Dataset Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G Extended Comparison to Similar Methods
Revised: G Extended Comparisons and Controls

Original: G.1Theoretical Comparison  . . . . 
Revised: G.1 Comparison to Nonlinear ICA . . . . . . . . . . . . . . . . . . . . . . . . . . . . G. [Table] Joint

Revised: Factor Dependence Evaluation . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G
Revised: G

Original: .2 Empirical Comparison
Revised: [Table] Transition Prior Ablation . .

Original: H Additional Results
Revised: H Additional Results

Revised: H.

Revised: [Table]

Original: H.1 Extended Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: Extended Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H. [Table]

Original: H.2 All DisLib Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: All DisLib Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H. [Table]

Original: H.3 Latent Space Visualizations . . . .
Revised: Masks ∆ t Ablation . . H. [Table] Space Visualizations . . . .

Original: A B ROADER I MPACT
Revised: A B ROADER I MPACT

Original: Representation learning is at the heart of model building for cognition.
Revised: Representation learning is at the heart of model building for cognition.

Original: Our speciﬁc contribution is focused on core methods for modeling natural videos and the datasets used are more simplistic than real-world examples.
Revised: Our speciﬁc contribution is focused on core methods for modeling natural videos and the datasets used are more simplistic than real-world examples.

Original: However, foundational research on unsupervised representation learning has potentially large impact on AI for advancing the power of self-learning systems.
Revised: However, foundational research on unsupervised representation learning has potentially large impact on AI for advancing the power of self-learning systems.

Original: The broader ﬁeld of representation learning has a large number of focused research directions that span machine learning and computational neuroscience.
Revised: The broader ﬁeld of representation learning has a large number of focused research directions that span machine learning and computational neuroscience.

Original: As such, the application space for this work is vast.
Revised: As such, the application space for this work is vast.

Original: For example, applications in unsupervised analysis of complicated and unintuitive data, such as medical imaging and gene expression information, have great potential to solve fundamental problems in health sciences.
Revised: For example, applications in unsupervised analysis of complicated and unintuitive data, such as medical imaging and gene expression information, have great potential to solve fundamental problems in health sciences.

Original: A future iteration of our disentangling approach could be used to encode such complicated data into a lower-dimensional and more understandable space that might reveal important factors of variation to medical researchers.
Revised: A future iteration of our disentangling approach could be used to encode such complicated data into a lower-dimensional and more understandable space that might reveal important factors of variation to medical researchers.

Original: Another important and complex modeling space that could potentially be improved by this line of research is in environmental sciences and combating global climate change.
Revised: Another important and complex modeling space that could potentially be improved by this line of research is in environmental sciences and combating global climate change.

Original: Nonetheless, we acknowledge that any machine learning method can be used for nefarious purposes, which can be mitigated via effective, scientiﬁcally informed communication, outreach, and policy direction.
Revised: Nonetheless, we acknowledge that any machine learning method can be used for nefarious purposes, which can be mitigated via effective, scientiﬁcally informed communication, outreach, and policy direction.

Original: We unconditionally denounce the use of derivatives of our work for weaponized or wartime applications.
Revised: We unconditionally denounce the use of derivatives of our work for weaponized or wartime applications.

Original: Additionally, due to the lack of interpretability generally found in modern deep learning approaches, it is possible for practitioners to inadvertently introduce harmful biases or errors in machine learning applications.
Revised: Additionally, due to the lack of interpretability generally found in modern deep learning approaches, it is possible for practitioners to inadvertently introduce harmful biases or errors in machine learning applications.

Original: Although we certainly do not solve this problem, our focus on providing identiﬁable solutions to representation learning is likely beneﬁcial for both interpretability and fairness in machine learning.
Revised: Although we certainly do not solve this problem, our focus on providing identiﬁable solutions to representation learning is likely beneﬁcial for both interpretability and fairness in machine learning.

Original: B F ORMAL M ETHODS 
Revised: B F ORMAL M ETHODS Function / variable

Original: [Table]
Revised: [Table]

Original: B. P ROOF OF I DENTIFIABILITY
Revised: B. P ROOF OF I DENTIFIABILITY

Original: To study disentanglement, we assume that the generative factors z ∈ R D are mapped to images x ∈ R N (usually D << N ) by a nonlinear ground-truth generator g ∗ :
Revised: To study disentanglement, we assume that the generative factors z ∈ R D are mapped to images x ∈ R N (usually D (cid:28) N , but see section C) by a nonlinear ground-truth generator g ∗ :

Original: z (cid:55)→ x .
Revised: z (cid:55)→ x .

Original: We consider a representation with a learned generator g disentangled if it inverts the ground-truth generator in the sense that the composition σ = ( g ∗ ) − 1 ◦ g can be described by a simple permutation and sign ﬂips.

Original: Theorem 1 For a ground-truth ( g ∗ , λ ∗ , α ∗ ) anda learned ( g, λ, α )  generative model as deﬁned in equation (2), if the following assumptions are satisﬁed:
Revised: Theorem 1 Let ( g ∗ , λ ∗ , α ∗ ) and ( g, λ, α ) respectively be ground-truth and learned generative models as deﬁned in Eq. If the following conditions are satisﬁed : (i)

Original: • The generators g ∗ and g are deﬁned everywhere in the latent space.
Revised: The generators g ∗ and g are deﬁned everywhere in the latent space.

Original: Moreover, they are injective and differentiable almost everywhere, • There is no model misspeciﬁcation i.e. α = α ∗ and λ = λ ∗ , so z ∼ p ( z )
Revised: Moreover, they are injective and differentiable almost everywhere, (ii) There is no model misspeciﬁcation i.e. α = α ∗ and λ = λ ∗ , so z ∼ p ( z )

Original: =
Revised: =

Original: p ∗ ( z ) , • Pairs of images are generated as ( x ∗ t − 1 , x ∗ t )
Revised: p ∗ ( z ) , (iii) Pairs of images are generated as ( x ∗ t − 1 , x ∗ t )

Original: =
Revised: =

Original: ( g ∗ ( z t − 1 ) , g ∗ ( z t )) and ( x t − 1 , x t ) =
Revised: ( g ∗ ( z t − 1 ) , g ∗ ( z t )) and ( x t − 1 , x t ) =

Original: ( g ( z t − 1 ) , g ( z t )) , • The distributions of ( x ∗ t − 1 , x ∗ t ) and ( x t − 1 , x t ) are the same (i.e. the corresponding densities are equal almost everywhere: p ∗ ( x t − 1 , x t ) =
Revised: ( g ( z t − 1 ) , g ( z t )) , (iv) The distributions of ( x ∗ t − 1 , x ∗ t ) and ( x t − 1 , x t ) are the same (i e. the corresponding densities are equal almost everywhere: p ∗ ( x t − 1 , x t ) =

Original: p ( x t − 1 , x t ) ,
Revised: p ( x t − 1 , x t ) ,

Original: then g = g ∗ ◦ σ , where σ is a composition of a permutation and sign ﬂips.
Revised: then g = g ∗ ◦ σ , where σ is a composition of a permutation and sign ﬂips.

Original: Proof.
Revised: Proof.

Original: Since x = g ( z ) can be written as x = ( g ∗ ◦ ( g ∗ ) − 1 ◦ g )
Revised: Since x = g ( z ) can be written as x = ( g ∗ ◦ ( g ∗ ) − 1 ◦ g )

Original: ( z ) , we can assume that g = g ∗ ◦ h for some function h on the latent space.
Revised: ( z ) , we can assume that g = g ∗ ◦ h for some function h on the latent space.

Original: We ﬁrst show that the function h is a bijection on the latent space.
Revised: We ﬁrst show that the function h is a bijection on the latent space.

Original: It is injective, since both g and g ∗ are injective.
Revised: It is injective, since both g and g ∗ are injective.

Original: If it were not surjective, there would be a point ˜
Revised: If it were not surjective, there would be a point ˜

Original: z in the ground-truth latent space, which would not have a pre-image under h .
Revised: z in the ground-truth latent space that would not have a pre-image under h .

Original: Because of continuity of h , there would be some neighborhood U ˜ z of ˜ z which would not have a pre-image under h .
Revised: Because of continuity of h , there would be some neighborhood U ˜ z of ˜ z that would not have a pre-image under h .

Original: That would mean that images generated by g ∗ from U ˜ z would have zero density under the distribution of images generated by g (i.e. p ( g ∗ ( U ˜ z ))
Revised: This would mean that images generated by g ∗ from U ˜ z would have zero density under the distribution of images generated by g (i.e. p ( g ∗ ( U ˜ z ))

Original: = 0 ), while this density would be non-zero under the distribution of images directly generated by the ground-truth generator g ∗ (i.e. p ∗ ( g ∗ ( U ˜ z )) (cid:54) = 0 ), which contradicts the assumption that these distributions are equal.
Revised: = 0 ), while this density would be non-zero under the distribution of images directly generated by the ground-truth generator g ∗ (i.e. p ∗ ( g ∗ ( U ˜ z )) (cid:54) = 0 ), which contradicts the assumption that these distributions are equal.

Original: It follows that h is bijective.
Revised: It follows that h is bijective.

Original: In the next step, we show that the distribution of latent space pairs ( h ( z t − 1 ) , h ( z t )) matches the latent space prior distribution (i.e. h preserves the prior distribution in the latent space).
Revised: In the next step, we show that the distribution of latent space pairs ( h ( z t − 1 ) , h ( z t )) matches the latent space prior distribution (i.e. h preserves the prior distribution in the latent space).

Original: Indeed, using the assumption that the distributions of ( g ∗ ( z t − 1 ) , g ∗ ( z t )) and (( g ∗ ◦ h )( z t − 1 ) , ( g ∗ ◦ h )( z t )) are the same, we can write the following equality using the change of variables formula:
Revised: Indeed, using the assumption that the distributions of ( g ∗ ( z t − 1 ) , g ∗ ( z t )) and (( g ∗ ◦ h )( z t − 1 ) , ( g ∗ ◦ h )( z t )) are the same, we can write the following equality using the change of variables formula:

Original: [Equation]
Revised: [Equation]

Original: where p and p h are densities of ( z t − 1 , z t ) and ( h ( z t − 1 ) , h ( z t )) .
Revised: where p and p h are densities of ( z t − 1 , z t ) and ( h ( z t − 1 ) , h ( z t )) .

Original: Since the determinants above cancel [Equation] these densities are equal at the pre-image of any pair of images ( x t − 1 , x t ) .
Revised: Since the determinants above cancel, these densities are equal at the pre-image of any pair of images ( x t − 1 , x t ) .

Original: Because g ∗ is deﬁned everywhere in the latent space [Equation] p and p h are equal for any pair of latent space points.
Revised: Because g ∗ is deﬁned everywhere in the latent space, p and p h are equal for any pair of latent space points.

Original: Applying the change of variables formula again
Revised: Applying the change of variables formula again

Original: [Equation]
Revised: [Equation]

Original: we obtain the following equation:
Revised: we obtain the following equation:

Original: [Equation]
Revised: [Equation]

Original: Note that the probability measure p is the same before and after the change of variables, since we showed that the prior distribution in the latent space must be invariant under the function h .
Revised: Note that the probability measure p is the same before and after the change of variables, since we showed that the prior distribution in the latent space must be invariant under the function h .

Original: The same condition for the marginal p ( z t − 1 ) is as follows:
Revised: The same condition for the marginal p ( z t − 1 ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: Solving for the determinant of the Jacobian in (7) and plugging it into (6), we obtain
Revised: Solving for the determinant of the Jacobian in (7) and plugging it into (6), we obtain

Original: [Equation]
Revised: [Equation]

Original: Taking logs of both sides, we arrive at the following equation:
Revised: Taking logs of both sides, we arrive at the following equation:

Original: [Equation]
Revised: [Equation]

Original: where A and B are the constants appearing in the exponentials in p ( z t − 1 ) and p ( z t | z t − 1 ) .
Revised: where A and B are the constants appearing in the exponentials in p ( z t − 1 ) and p ( z t | z t − 1 ) .

Original: The logs of normalization constants cancel out.
Revised: The logs of normalization constants cancel out.

Original: For any z t we can choose z t − 1 =
Revised: For any z t we can choose z t − 1 =

Original: z t making the left hand side in (9) equal to zero.
Revised: z t making the left hand side in (9) equal to zero.

Original: This implies that || z t || 22 =
Revised: This implies that || z t || 22 =

Original: ||
Revised: ||

Original: h − 1 ( z t ) || 22 for any z t , i.e. function h − 1 preserves the 2-norm.
Revised: h − 1 ( z t ) || 22 for any z t , i.e. function h − 1 preserves the 2-norm.

Original: Moreover, the preservation of the 2-norm implies that p ( z t − 1 ) =
Revised: Moreover, the preservation of the 2-norm implies that p ( z t − 1 ) =

Original: p ( h − 1 ( z t − 1 ))
Revised: p ( h − 1 ( z t − 1 ))

Original: and therefore it follows from (7) that for any z
Revised: and therefore it follows from (7) that for any z

Original: [Equation]
Revised: [Equation]

Original: Thus, the left hand side of (9) can be re-written as
Revised: Thus, the left hand side of (9) can be re-written as

Original: [Equation]
Revised: [Equation]

Original: This means that h − 1 preserves the α -distances between points.
Revised: This means that h − 1 preserves the α -distances between points.

Original: Moreover, because h is bijective, the
Revised: Moreover, because h is bijective, the

Original: Mazur-Ulam theorem (Mazur and Ulam, 1932) tells us that h must be an afﬁne transform.
Revised: Mazur-Ulam theorem (Mazur and Ulam, 1932) tells us that h must be an afﬁne transform.

Original: In the next step, to prove that h must be a permutation and sign ﬂip, let us choose an arbitrary point z t − 1 and z t =
Revised: In the next step, to prove that h must be a permutation and sign ﬂip, let us choose an arbitrary point z t − 1 and z t =

Original: z t − 1 + ε e k
Revised: z t − 1 + ε e k

Original: =
Revised: =

Original: z 1 , 1 , . .
Revised: z 1 , 1 , . .

Original: , z 1 ,k
Revised: , z 1 ,k

Original: + ε, . .
Revised: + ε, . .

Original: , z 1 ,D ) .
Revised: , z 1 ,D ) .

Original: Using (11) and performing a Taylor expansion around z t − 1 , we obtain the following:
Revised: Using (11) and performing a Taylor expansion around z t − 1 , we obtain the following:

Original: [Equation]
Revised: [Equation]

Original: The higher-order terms O ( ε 2 ) are zero since h is afﬁne, therefore dividing both sides of the above equation by ε
Revised: The higher-order terms O ( ε 2 ) are zero since h is afﬁne, therefore dividing both sides of the above equation by ε

Original: α
Revised: α

Original: we ﬁnd that
Revised: we ﬁnd that

Original: [Equation]
Revised: [Equation]

Original: The vectors of k -th partial derivatives of components of h − 1 are columns of the Jacobian matrix (cid:16) d 
Revised: The vectors of k -th partial derivatives of components of h − 1 are columns of the Jacobian matrix (cid:16) d h [Equation] 1 ( z

Original: [Equation]
Revised: [Equation]

Original:  z (cid:17) .
Revised: d z (cid:17) .

Original: Using the fact that the determinant of that matrix is equal to one and applying Hadamard’s inequality
Revised: Using the fact that the determinant of that matrix is equal to one and applying Hadamard’s inequality

Original: [Equation]
Revised: [Equation]

Original: we obtain that
Revised: we obtain that

Original: [Equation]
Revised: [Equation]

Original: Since α <
Revised: Since α <

Original: [Equation]
Revised: [Equation]

Original: for any vector v it holds
Revised: for any vector v it holds

Original: that ||
Revised: that ||

Original: v || 2 ≤
Revised: v || 2 ≤

Original: ||
Revised: ||

Original: v || α , with equality only if at most one component of v is non-zero.
Revised: v || α , with equality only if at most one component of v is non-zero.

Original: This inequality implies that both (13) and (14) hold at the same time if and only if
Revised: This inequality implies that both (13) and (14) hold at the same time if and only if

Original: [Equation]
Revised: [Equation]

Original: meaning that only one element of these vectors of k -th partial derivatives is non-zero, and it is equal to 1 or -1.
Revised: meaning that only one element of these vectors of k -th partial derivatives is non-zero, and it is equal to 1 or -1.

Original: Thus, the function h is a composition of a permutation and sign ﬂips at every point.
Revised: Thus, the function h is a composition of a permutation and sign ﬂips at every point.

Original: Potentially, this permutation might be input-dependent, but we argued above that h is afﬁne, therefore the permutation must be the same for all points.
Revised: Potentially, this permutation might be input-dependent, but we argued above that h is afﬁne, therefore the permutation must be the same for all points.

Original: [Equation]
Revised: [Equation]

Original: B. K ULLBACK L EIBLER D IVERGENCE OF S LOW V ARIATIONAL A UTOENCODER
Revised: B. K ULLBACK L EIBLER D IVERGENCE OF S LOW V ARIATIONAL A UTOENCODER

Original: The VAE learns a variational approximation to the true posterior by maximizing a lower bound on the log-likelihood of the empirical data distribution DE
Revised: The VAE learns a variational approximation to the true posterior by maximizing a lower bound on the log-likelihood of the empirical data distribution D

Original: [Equation]
Revised: [Equation]

Original: For this, we need to compute the Kullback-Leibler divergence (KL) between the posterior q ( z t , z t − 1 | x t , x t − 1 ) and the prior p ( z t , z t − 1 ) .
Revised: For this, we need to compute the Kullback-Leibler divergence (KL) between the posterior q ( z t , z t − 1 | x t , x t − 1 ) and the prior p ( z t , z t − 1 ) .

Original: Since all of these distributions are per design factorial, we will, for simplicity, derive the KL below for scalar variables (log-probabilities will simply have to be summed to obtain the full expression).
Revised: Since all of these distributions are per design factorial, we will, for simplicity, derive the KL below for scalar variables (log-probabilities will simply have to be summed to obtain the full expression).

Original: Recall that the model prior and posterior factorize like
Revised: Recall that the model prior and posterior factorize like

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, given a pair of inputs ( x t − 1 , x t ) , the KL can be written
Revised: Then, given a pair of inputs ( x t − 1 , x t ) , the KL can be written

Original: [Equation]
Revised: [Equation]

Original: Where we use the fact that KL divergences decompose like D KL ( X, Y )
Revised: Where we use the fact that KL divergences decompose like D KL ( X, Y )

Original: =
Revised: =

Original: H ( X, Y ) − H
Revised: H ( X, Y ) − H

Original: ( X ) into (differential) cross-entropy H ( X, Y ) and entropy H ( X ) .
Revised: ( X ) into (differential) cross-entropy H ( X, Y ) and entropy H ( X ) .

Original: The ﬁrst term of the last line in (18) is the same KL divergence as in the standard VAE, namely between a Gaussian distribution q ( z t − 1 | x t − 1 ) with some µ ( x t − 1 ) and σ ( x t − 1 ) and a standard Normal distribution p ( z t − 1 ) .
Revised: The ﬁrst term of the last line in (18) is the same KL divergence as in the standard VAE, namely between a Gaussian distribution q ( z t − 1 | x t − 1 ) with some µ ( x t − 1 ) and σ ( x t − 1 ) and a standard Normal distribution p ( z t − 1 ) .

Original: The solution of the KL is given by D
Revised: The solution of the KL is given by D

Original: [Equation]
Revised: [Equation]

Original: (Bishop, 2006).
Revised: (Bishop, 2006).

Original: The second term on the RHS, i.e. the entropy of a Gaussian is simply given by
Revised: The second term on the RHS, i.e. the entropy of a Gaussian is simply given by

Original: [Equation]
Revised: [Equation]

Original: To compute the last term on the RHS
Revised: To compute the last term on the RHS

Original: [Equation]
Revised: [Equation]

Original: let us recall the Laplace form of the conditional prior
Revised: let us recall the Laplace form of the conditional prior

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus the cross-entropy becomes
Revised: Thus the cross-entropy becomes

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: if some random variable X ∼ N ( µ, σ 2 )
Revised: if some random variable X ∼ N ( µ, σ 2 )

Original: [Equation]
Revised: [Equation]

Original: then Y =
Revised: then Y =

Original: | X | follows a folded normal distribution , for which the mean is deﬁned as
Revised: | X | follows a folded normal distribution , for which the mean is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where Φ is the cumulative distribution function of a standard normal distribution (mean zero and variance one).
Revised: where Φ is the cumulative distribution function of a standard normal distribution (mean zero and variance one) [Equation]

Original: Thus, denoting µ ( x t , z t − 1 ) = µ ( x t ) − z t − 1 , we can rewrite further
Revised: z ∗ ∆ z ∗ x ∗ z ∆ z x µ σ 5. MCC loss 0 π 2 π 0 π 2 π [Figure] z ∗ ∆ z ∗ x ∗ z ∆ z 2.

Original: [Equation]
Revised: [Figure]

Original: C C HOOSING A L ATENT V ARIABLE M ODEL
Revised: C C HOOSING A L ATENT V ARIABLE M ODEL

Original: Our proposed method for disentanglement can be implemented in conjunction with different probabilistic latent variable models.
Revised: Our proposed method for disentanglement can be implemented in conjunction with different probabilistic latent variable models.

Original: In this section, we compare VAEs and normalizing ﬂows as possible candidates.
Revised: In this section, we compare VAEs and normalizing ﬂows as possible candidates.

Original: Variational Autoencoders (VAEs) (Kingma and Welling, 2013) are a widely used probabilistic latent variable model.
Revised: Variational Autoencoders (VAEs) (Kingma and Welling, 2013) are a widely used probabilistic latent variable model.

Original: Despite their simple structure and empirical success, VAEs can converge to a pathological solution called posterior collapse (Lucas et al., 2019; Bowman et al., 2016;
Revised: Despite their simple structure and empirical success, VAEs can converge to a pathological solution called posterior collapse (Lucas et al., 2019; Bowman et al., 2016;

Original: He et al., 
Revised: He et al., 2019).

Original: This solution results in the encoder’s variational posterior approximation matching the prior, which is typically chosen to be a multivariate standard normal qφ ( z | x∗ ) ≈ p ( z ) =
Revised: This solution results in the encoder’s variational posterior approximation matching the prior, which is typically chosen to be a multivariate standard normal q ( z | x ) ≈ p ( z ) =

Original: N ( 0 , I ) .
Revised: N ( 0 , I ) .

Original: This disconnects the encoder from the decoder, making them approximately independent, i.e. pθ ( x | z )
Revised: This disconnects the encoder from the decoder, making them approximately independent, i.e. p ( x | z )

Original: ≈ pθ ( x ) .
Revised: ≈ p ( x ) .

Original: The failure mode is often observed when the decoder architecture is overly expressive, i.e. with autoregressive models, or when the evidence p∗ ( x ) is easy to estimate.
Revised: The failure mode is often observed when the decoder architecture is overly expressive, i.e. with autoregressive models, or when the likelihood p ( x ) is easy to estimate.

Original: Approaches that alleviate this problem rely on modifying the ELBO training objective (Bowman et al., 2016; Kingma et al., 2016) or restricting the decoder structure (Dieng et al., 2019; Maaløe et al., 2019).
Revised: Approaches that alleviate this problem rely on modifying the ELBO training objective (Bowman et al., 2016; Kingma et al., 2016) or restricting the decoder structure (Dieng et al., 2019; Maaløe et al., 2019).

Original: However, these approaches come with various drawbacks, including optimization issues (Lucas et al., 2019).
Revised: However, these approaches come with various drawbacks, including optimization issues (Lucas et al., 2019).

Original: Another approach to estimate latent variables are normalizing ﬂows which describe a sequence of invertible mappings by iteratively applying the change of variables rule (Dinh et al., 2017b).
Revised: Another approach to estimate latent variables are normalizing ﬂows which describe a sequence of invertible mappings by iteratively applying the change of variables rule (Dinh et al., 2017b).

Original: Unlike VAEs, ﬂow based latent variable models allow for a direct optimization of the evidence p ∗ ( x ) (Dinh et al., 2017b).
Revised: Unlike VAEs, ﬂow based latent variable models allow for a direct optimization of the likelihood (Dinh et al., 2017b).

Original: Most normalizing ﬂow models rely on a fast and reliable calculation of the determinant of the Jacobian of the outputs with respect to the inputs, which constrains the architectural design and limits the capacity of the network (Tabak et al., 2010; Tabak and Turner, 2013;
Revised: Most normalizing ﬂow models rely on a fast and reliable calculation of the determinant of the Jacobian of the outputs with respect to the inputs, which constrains the architectural design and limits the capacity of the network (Tabak et al., 2010; Tabak and Turner, 2013;

Original: Dinh et al., 2017b).
Revised: Dinh et al., 2017b).

Original: Thus, competitive ﬂows require very deep architectures in practice (Kingma and Dhariwal, 2018).
Revised: Thus, competitive ﬂows require very deep architectures in practice (Kingma and Dhariwal, 2018).

Original: Furthermore, ﬂows are not directly suited for a scenario where the observation space is higher dimensional than the generating latent factors, dim ( z∗ ) << dim ( x∗ ) , as the computation of the determinant requires a square Jacobian matrix.
Revised: Furthermore, ﬂows are not directly suited for a scenario where the observation space is higher dimensional than the generating latent factors, dim ( z ) < dim ( x ) , as the computation of the determinant requires a square Jacobian matrix.

Original: We tried settingthe number of dimensions of the observation space equal to the number of dimensions in the latent space dim ( z )
Revised: We tried setting dim ( z )

Original: = dim ( x ) >> dim ( z ∗ ) , but observed instability while optimizing the objective deﬁned below.
Revised: = dim ( x ) > dim ( z ∗ ) , but observed instability while optimizing the objective deﬁned below.

Original: It is straight forward to derive a ﬂow based objective based on the assumptions in equation (2).
Revised: It is straightforward to derive a ﬂow-based objective based on the assumptions in Eq.

Original: We consider a normalizing ﬂow with with K blocks f ( x ) =
Revised: We consider a normalizing ﬂow with with K blocks f ( x ) =

Original: f K ◦ ... ◦
Revised: f K ◦ ... ◦

Original: f 1 : x
Revised: f 1 : x

Original: (cid:55)→ z .
Revised: (cid:55)→ z .

Original: The coupling blocks can refer to nonlinear mixing similar to Kingma and Dhariwal (2018), or in the linear case ( K = 1 ) to an invertible de-mixing matrix.
Revised: The coupling blocks can refer to nonlinear mixing similar to Kingma and Dhariwal (2018), or in the linear case ( K = 1 ) to an invertible de-mixing matrix.

Original: This leads to the following estimation of the data probability p
Revised: This leads to the following estimation of the likelihood p

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that p ( f ( x t − 1 )) is Gaussian and p ( f ( x t )
Revised: Note that p ( f ( x t − 1 )) is Gaussian and p ( f ( x t )

Original: | f
Revised: | f

Original: ( x t − 1 )) is a Laplacian, similar to equation (2).
Revised: ( x t − 1 )) is a Laplacian, similar to Eq.

Original: For notation convenience, we also introduced z 0 = x ∗ .

Original: During optimization we take the − log of both sides and minimize w.r.t.
Revised: During optimization we take the − log of both sides and minimize w.r.t.

Original: the parameters of f .
Revised: the parameters of f .

Original: We refer to this estimator as SlowFlow .
Revised: We refer to this estimator as SlowFlow .

Revised: Our SlowFlow model is very similar to the ﬂow described in (Pineau et al., 2020), who use a Gaussian transition prior and therefore would have weaker identiﬁability guarantees.

Original: Next, we compare SlowFlow and SlowVAE in the context of disentanglement.
Revised: Next, we compare SlowFlow and SlowVAE in the context of disentanglement.

Original: To demonstrate the posterior collapse in VAEs, we generate data points ( x t , x t − 1 ) according to equation (2) with a two dimensional latent space dim ( z ∗ ) = 2 .
Revised: To demonstrate the posterior collapse in VAEs, we generate data points ( x t , x t − 1 ) according to Eq. (2) with a two dimensional latent space dim ( z ∗ ) = 2 .

Original: We consider a trivial linear mixing of x ∗ =
Revised: We consider a trivial linear mixing of x ∗ =

Original: W ∗ z ∗ =
Revised: W ∗ z ∗ =

Original: g ∗ ( z ∗ ) withW ∗ = [Equation]
Revised: g ∗ ( z ∗ ) with

Original: [Equation]
Revised: [Equation]

Original: (24) and κ ∈ [0 . 1 , 1] .
Revised: and κ ∈ [0 . 1 , 1] .

Original: As can be seen by looking at the σ and µ outputs of the encoder in Fig 6, for κ < 0 .
Revised: As can be seen by looking at the σ and µ outputs of the encoder in Fig 6a, for κ < 0 .

Original: 4 , the encoder q φ ( z 0 | x ) collapses to the prior p ( z 0 ) .
Revised: 4 , the encoder for the minor axis collapses to the prior.

Original: The decoder then tries  minimize the reconstruction loss by solely covering the main principal component of the data, which is also described in Rolinek et al.
Revised: The decoder then tries to minimize the reconstruction loss by solely covering the ﬁrst principal component of the data, which is also described in Rolinek et al.

Original: Despite the collapse and decrease in MCC, the SlowVAE loss from equation (4) still improves during training.
Revised: Despite the collapse and decrease in MCC, the SlowVAE loss from Eq. (4) still improves during training.

Original: On the other hand, a simple linear SlowFlow model f ( x ) =
Revised: On the other hand, a simple linear SlowFlow model f ( x ) =

Original: W x , which directly optimizes the evidence p ∗ ( x ) , recovers the latents consistently as seen by the MCC measure.
Revised: W x , which directly optimizes the likelihood, recovers the latents consistently as seen by the MCC measure (Fig 6b).

Original: To show the strength of the VAE model we increase the complexity of the data-distribution by using a non-linear expanding decoder such that dim ( x∗ ) >> dim ( z ∗ ) .
Revised: To show the strength of the VAE model we increase the complexity of the data-distribution by using a non-linear expanding decoder such that dim ( x ) (cid:29) dim ( z ∗ ) .

Original: In Fig.
Revised: In Fig.

Original: 7 we observe that increasing the input dimensionality is sufﬁcient for SlowVAE to ﬁnd the corresponding latents and achieve high
Revised: 7 we observe that increasing the input dimensionality is sufﬁcient for SlowVAE to ﬁnd the corresponding latents and achieve high

Original: MCC with low loss.
Revised: MCC with low loss.

Revised: z ∗ ∆ z ∗ x ∗ z ∆ z x µ σ 34.

Revised: MCC loss

Original: Each estimation method is practically useful in different experimental settings.
Revised: [Figure] VAEs prefer data dimensions to be greater than latent dimensions. Individual subplots are as described in Fig. For all data in this experiment we used a 20-dimensional latent space, dim ( z ∗ ) = 20 .

Revised: Each row corresponds to the dimensionality of the x ∗ , with values of 20 , 200 , and 2000 .

Revised: The ﬁrst two dimensions of z ∗ are plotted as well as the two dimensions of z with the highest corresponding mean correlation coefﬁcient (MCC).

Revised: The x ∗ and x data are projected onto their ﬁrst two principal component axes before plotting.

Revised: A two-layer mixing matrix was used to transform data from Z gt to

Revised: X gt .

Revised: As one increases the data dimensionality, the SlowVAE network performs increasingly better in terms of MCC, although worse in terms of total training loss.

Original:  In the case when the mixing operation is trivially deﬁned (equation (24), or when the number of dimensions in Z ∗ match those in X ∗ , the VAE estimator tends to learn a pathological solution.
Revised: Each estimation method is practically useful in different experimental settings. In the case when the mixing operation is trivially deﬁned (Eq. (24), or when the number of dimensions in z ∗ match those in x ∗ ), the VAE estimator tends to learn a pathological solution.

Original: On the other hand, the normalizing ﬂow estimator does not scale well to high  datadimensions due to the requirement of computing the network Jacobian.
Revised: On the other hand, the normalizing ﬂow estimator does not scale well to high dimensional data due to the requirement of computing the network Jacobian.

Original: Additionally, the framework for constructing normalizing ﬂow estimators assumes the latent dimensionality is equal to the data dimensionality to allow for an invertible transform.
Revised: Additionally, the framework for constructing normalizing ﬂow estimators assumes the latent dimensionality is equal to the data dimensionality to allow for an invertible transform.

Original: Together these results lead us to choose an estimator based on the nature of the problem.
Revised: Together these results lead us to choose an estimator based on the nature of the problem.

Original: For our contributing datasets and the DisLib experiments we adopt the VAE framework.
Revised: For our contributed datasets and the DisLib experiments we adopt the VAE framework.

Original: However, if one aims to perform simpliﬁed experiments such as those typically conducted in the nonlinear ICA literature, itoften will  make practical sense to switch to a ﬂow-based estimator.
Revised: However, if one aims to perform simpliﬁed experiments such as those typically conducted in the nonlinear ICA literature, it will often make practical sense to switch to a ﬂow-based estimator.

Original: D D ISENTANGLEMENT M ETRICS
Revised: D D ISENTANGLEMENT M ETRICS

Original: Several recent studies have brought to light shortcomings in a number of proposed disentanglement metrics (Kim and Mnih, 2018; Eastwood and Williams, 2018;
Revised: Several recent studies have brought to light shortcomings in a number of proposed disentanglement metrics (Kim and Mnih, 2018; Eastwood and Williams, 2018;

Original: Chen et al., 2018; Higgins et al., 2018;
Revised: Chen et al., 2018; Higgins et al., 2018;

Original: Mathieu et al., 2019), many of which have been compiled in the DisLib benchmark.
Revised: Mathieu et al., 2019), many of which have been compiled in the DisLib benchmark.

Original: In addition to the concerns they raise, it is important to note
Revised: In addition to the concerns they raise, it is important to note

Original: that none of the supervised metrics implemented in
Revised: that none of the supervised metrics implemented in

Original: DisLib allow for continuous ground-truth factors, which is necessary for evaluating with the Natural
Revised: DisLib allow for continuous ground-truth factors, which is necessary for evaluating with the Natural

Original: Sprites and KITTI Masks datasets, as factors such as position and scale are effectively continuous in reality.
Revised: Sprites and KITTI Masks datasets, as factors such as position and scale are effectively continuous in reality.

Original: To rectify this issue without introducing novel metrics, we include the Mean Correlation
Revised: To rectify this issue without introducing novel metrics, we include the Mean Correlation

Original: Coefﬁcient (MCC) as implemented by Hyvärinen and Morioka (2016), which is described below.
Revised: Coefﬁcient (MCC) in our evaluations, using the implementation of Hyvärinen and Morioka (2016), which is described below.

Original: We measure all metrics presented below between 10 , 000 samples of latent factors z and the corresponding encoded means of our model µ ( g ∗ ( z )) .
Revised: We measure all metrics presented below between 10 , 000 samples of latent factors z and the corresponding encoded means of our model µ ( g ∗ ( z )) .

Original: We increase this sample size to 100 , 000 for
Revised: We increase this sample size to 100 , 000 for

Original: Modularity and MIG to stabilize the entropy estimates.
Revised: Modularity and MIG to stabilize the entropy estimates.

Original: D. M EAN C ORRELATION C OEFFICIENT
Revised: D. M EAN C ORRELATION C OEFFICIENT

Original: In addition to the DisLib metrics, we also compute the Mean Correlation Coefﬁcient (MCC) in order to perform quantitative evaluation with continuous variables.
Revised: In addition to the DisLib metrics, we also compute the Mean Correlation Coefﬁcient (MCC) in order to perform quantitative evaluation with continuous variables.

Original: Because of Theorem 1, perfect disentanglement in the noiseless case should always lead to a correlation coefﬁcient of 1 or − 1 , although note that we report 100 times the absolute value of the correlation coefﬁcient.
Revised: Because of Theorem 1, perfect disentanglement in the noiseless case should always lead to a correlation coefﬁcient of 1 or − 1 , although note that we report 100 times the absolute value of the correlation coefﬁcient.

Original: In our experiments, MCC is used without modiﬁcation from the authors’ open-sourced code (Morioka, 2018).
Revised: In our experiments, MCC is used without modiﬁcation from the authors’ open-sourced code (Morioka, 2018).

Original: The method ﬁrst measures correlation between the ground-truth factors and the encoded latent variables.
Revised: The method ﬁrst measures correlation between the ground-truth factors and the encoded latent variables.

Original: The initial correlation matrix is then used to match each latent unit with a preferred ground-truth factor.
Revised: The initial correlation matrix is then used to match each latent unit with a preferred groundtruth factor.

Original: This is an assignment problem that can be solved in polynomial time via the Munkres algorithm, as described in the code release (Morioka, 2018).
Revised: This is an assignment problem that can be solved in polynomial time via the Munkres algorithm, as described in the code release from Morioka (2018).

Original: After solving the assignment problem, the correlation coefﬁcients are computed again for the vector of ground-truth factors and the resulting permuted vector of latent encodings, where the output is a matrix of correlation coefﬁcients with D columns for each ground-truth factor and D (cid:48) rows for each latent variable.
Revised: After solving the assignment problem, the correlation coefﬁcients are computed again for the vector of ground-truth factors and the resulting permuted vector of latent encodings, where the output is a matrix of correlation coefﬁcients with D columns for each ground-truth factor and D (cid:48) rows for each latent variable.

Original: We use the (absolute value of the) Spearman coefﬁcient as our correlation measure which assumes a monotonic relationship between the ground-truth factors and latent encodings but tolerates deviations from a strictly linear correspondence.
Revised: We use the (absolute value of the) Spearman coefﬁcient as our correlation measure which assumes a monotonic relationship between the ground-truth factors and latent encodings but tolerates deviations from a strictly linear correspondence.

Original: In the existing implementation for MCC, the ground truth factors, latent encodings, and mixed signal inputs are assumed to have the same dimensionality, i.e. D = D (cid:48)
Revised: In the existing implementation for MCC, the ground truth factors, latent encodings, and mixed signal inputs are assumed to have the same dimensionality, i.e. D = D (cid:48)

Original: =
Revised: =

Original: N .
Revised: N .

Original: However, in our case, the ground-truth generating factors are much lower dimensional than the signal, N << D , and the latent encoding is higher dimensional than the ground-truth factors D (cid:48)
Revised: However, in our case, the ground-truth generating factors are much lower dimensional than the signal, N (cid:28) D , and the latent encoding is higher dimensional than the ground-truth factors D (cid:48)

Original: > D (see Appendix E.2 for details).
Revised: > D (see Appendix F.1 for details).

Original: To resolve this discrepancy, we add D (cid:48) − D standard Gaussian noise channels to the ground-truth factors.
Revised: To resolve this discrepancy, we add D (cid:48) − D standard Gaussian noise channels to the ground-truth factors.

Original: To compute the MCC score, we take the mean of the absolute value of the upper diagonal of the correlation matrix.
Revised: To compute the MCC score, we take the mean of the absolute value of the upper diagonal of the correlation matrix.

Original: The upper diagonal is the diagonal of the square matrix of D ground-truth factors by the top D most correlated latent dimensions after sorting.
Revised: The upper diagonal is the diagonal of the square matrix of D ground-truth factors by the top D most correlated latent dimensions after sorting.

Original: In this way, we obtain an MCC estimate which averages only over the D correlation coefﬁcients of the D ground truth factors with their corresponding best matching latent factors,
Revised: In this way, we obtain an MCC estimate which averages only over the D correlation coefﬁcients of the D ground truth factors with their corresponding best matching latent factors.

Original: [Equation] BetaVAE (Higgins et al., 2017)
Revised: D. D IS L IB M ETRICS BetaVAE (Higgins et al., 2017)

Original:  BetaVAE  uses a biased estimator with tunable hyperparameters, although we follow the convention established in (Locatello et al., 2018) of using the scikit-learn defaults.
Revised: The BetaVAE metric uses a biased estimator with tunable hyperparameters, although we follow the convention established in (Locatello et al., 2018) of using the scikit-learn defaults.

Original: For a sample in a batch, a pair of images, ( x 1 , x 2 ) , is generated by ﬁxing the value of one of the data generative factors while uniformly sampling the rest.
Revised: For a sample in a batch, a pair of images, ( x 1 , x 2 ) , is generated by ﬁxing the value of one of the data generative factors while uniformly sampling the rest.

Original: The absolute value of the difference between the latent codes produced from the image pairs is then taken, z diff = | z 1 − z 2 | .
Revised: The absolute value of the difference between the latent codes produced from the image pairs is then taken, z diff = | z 1 − z 2 | .

Original: A logistic classiﬁer is ﬁt with batches of z diff variables and the corresponding index of the ﬁxed ground-truth factor serves as the label.
Revised: A logistic classiﬁer is ﬁt with batches of z diff variables and the corresponding index of the ﬁxed ground-truth factor serves as the label.

Original: Once the classiﬁer is trained, the metric itself is the mean classiﬁer accuracy on a batch of held-out test data.
Revised: Once the classiﬁer is trained, the metric itself is the mean classiﬁer accuracy on a batch of held-out test data.

Original: The training minimizes the following loss: 
Revised: The training minimizes the following loss: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where w and c are the learnable weight matrix and bias, respectively, and y is the index of the ﬁxed ground-truth factor for the batch.
Revised: where w and c are the learnable weight matrix and bias, respectively, and y is the index of the ﬁxed ground-truth factor for the batch.

Original: The network is trained using the lbfgs optimizer (Byrd et al., 1995), which is implemented via the scikit-learn Python package (Pedregosa et al., 2011) in disentanglement_lib (Locatello et al., 2018).
Revised: The network is trained using the lbfgs optimizer (Byrd et al., 1995), which is implemented via the scikit-learn Python package (Pedregosa et al., 2011) in the Disentanglement Library (DisLib, Locatello et al., 2018).

Original: In the original work, the authors argue that their metric improves over a correlation metric such as the mean correlation coefﬁcient by additionally measuring interpretability, although the linear operation of z T diff ,i w + c can perform demixing, which means the measure gives no direct indication of identiﬁability and thus does not guarantee that the latent encodings are interpretable, especially in the case of dependent factors.
Revised: In the original work, the authors argue that their metric improves over a correlation metric such as the mean correlation coefﬁcient by additionally measuring interpretability. However, the linear operation of z T diff ,i w + c can perform demixing, which means the measure gives no direct indication of identiﬁability and thus does not guarantee that the latent encodings are interpretable, especially in the case of dependent factors.

Original: Additionally, as noted by Kim and Mnih (2018), BetaVAE can report perfect accuracy when all but one of the ground-truth factors are disentangled, since the classiﬁer can trivially attribute the remaining factor to the remaining latents.
Revised: Additionally, as noted by Kim and Mnih (2018), BetaVAE can report perfect accuracy when all but one of the ground-truth factors are disentangled, since the classiﬁer can trivially attribute the remaining factor to the remaining latents.

Original: FactorVAE (Kim and Mnih, 2018)
Revised: FactorVAE (Kim and Mnih, 2018)

Original: For FactorVAE, the variance of the latent encodings is computed for a large (10,000 in DisLib) batch of data where all factors could possibly be changing.
Revised: For the FactorVAE metric, the variance of the latent encodings is computed for a large (10,000 in DisLib) batch of data where all factors could possibly be changing.

Original: Latent dimensions with variance below some threshold ( 0 . 05 in DisLib) are rejected and not considered further.
Revised: Latent dimensions with variance below some threshold ( 0 . 05 in DisLib) are rejected and not considered further.

Original: Next, the encoding variance is computed again on a smaller batch ( 64 in DisLib) of data where one factor is ﬁxed during sampling.
Revised: Next, the encoding variance is computed again on a smaller batch ( 64 in DisLib) of data where one factor is ﬁxed during sampling.

Original: The quotient of these two quantities (with the larger batch variance as the denominator) is then taken to obtain a normalized variance estimate per latent factor.
Revised: The quotient of these two quantities (with the larger batch variance as the denominator) is then taken to obtain a normalized variance estimate per latent factor.

Original: Finally, a majority-vote classiﬁer is trained to predict the index of the ground-truth factor with the latent unit that has the lowest normalized variance.
Revised: Finally, a majority-vote classiﬁer is trained to predict the index of the ground-truth factor with the latent unit that has the lowest normalized variance.

Original: The FactorVAE score is the classiﬁcation accuracy for a batch of held-out data.
Revised: The FactorVAE score is the classiﬁcation accuracy for a batch of held-out data.

Original: Mutual Information Gap (Chen et al., 2018)
Revised: Mutual Information Gap (Chen et al., 2018)

Original: The Mutual Information Gap (MIG) metric was introduced as an alternative to the classiﬁer-based metrics that we discuss herein.
Revised: The Mutual Information Gap (MIG) metric was introduced as an alternative to the classiﬁer-based metrics.

Original: It provides a normalized measure of the mean difference in mutual information between each ground truth factor and the two latent codes that have the highest mutual information with the given ground truth factor.
Revised: It provides a normalized measure of the mean difference in mutual information between each ground truth factor and the two latent codes that have the highest mutual information with the given ground truth factor.

Original: As it is implemented in DisLib, MIG measures entropy by discretizing the model’s latent code using a histogram with 20 bins equally spaced between the representation minimum and maximum.
Revised: As it is implemented in DisLib, MIG measures entropy by discretizing the model’s latent code using a histogram with 20 bins equally spaced between the representation minimum and maximum.

Original: It then computes the discrete mutual information between the ground-truth values and the discretized latents using the scikit-learn metrics.mutual_info_score function (Pedregosa et al., 2011).
Revised: It then computes the discrete mutual information between the groundtruth values and the discretized latents using the scikit-learn metrics.mutual_info_score function (Pedregosa et al., 2011).

Original: For the normalization it divides this difference by the entropy of the discretized ground truth factors.
Revised: For the normalization it divides this difference by the entropy of the discretized ground truth factors.

Original: Modularity 
Revised: Modularity (Ridgeway and Mozer, 2018)

Original: (Ridgeway and Mozer, 2018)

Original: Ridgeway and Mozer (2018) measure disentanglement in terms of three factors: modularity, compactness, and explicitness.
Revised: Ridgeway and Mozer (2018) measure disentanglement in terms of three factors: modularity, compactness, and explicitness.

Original: For modularity, they ﬁrst measure the mutual information between the discretized latents and ground-truth factors using the same histogram procedure that was used for the MIG, resulting in a matrix, M ∈ R D (cid:48)
Revised: For modularity, they ﬁrst measure the mutual information between the discretized latents and ground-truth factors using the same histogram procedure that was used for the MIG, resulting in a matrix, M ∈ R D (cid:48)

Original: × D with entries for each mutual information pair.
Revised: × D with entries for each mutual information pair.

Original: Their measure of modularity is then
Revised: Their measure of modularity is then

Original: [Equation]
Revised: [Equation]

Original: where max ( M 2 i ) returns the maximum of the vector of squared mutual information measurements between ground truth i and each latent factor.
Revised: where max ( M 2 i ) returns the maximum of the vector of squared mutual information measurements between ground truth i and each latent factor.

Original: Additionally, Θ is a selection function that returns zero for any i where max ( M 2 i ) = 0 and otherwise acts as the identity function.
Revised: Additionally, Θ is a selection function that returns zero for any i where max ( M 2 i ) = 0 and otherwise acts as the identity function.

Original: DCI Disentanglement (Eastwood and Williams, 2018)
Revised: DCI Disentanglement (Eastwood and Williams, 2018)

Original: The DCI scores measure disentanglement, completeness, and informativeness, which have intuitive correspondence to the modularity, compactness, and explicitness of (Ridgeway and Mozer, 2018), respectively.
Revised: The DCI scores measure disentanglement, completeness, and informativeness, which have intuitive correspondence to the modularity, compactness, and explicitness of (Ridgeway and Mozer, 2018), respectively.

Original: To measure DCI Disentanglement, D regressors are trained to predict each ground truth factor state given the latent encoding.
Revised: To measure DCI Disentanglement, D regressors are trained to predict each ground truth factor state given the latent encoding.

Original: The DisLib implementation uses the ensemble.
Revised: The DisLib implementation uses the ensemble.

Original: GradientBoostingClassifier function from scikit-learn with default parameters, which trains D gradient boosted logistic regression tree classiﬁers.
Revised: GradientBoostingClassifier function from scikit-learn with default parameters, which trains D gradient boosted logistic regression tree classiﬁers.

Original: Importance is assigned to each latent factor using the built-in feature_importance_ property of the classiﬁer, which computes the normalized total reduction of the classiﬁer criterion loss contributed by each latent.
Revised: Importance is assigned to each latent factor using the built-in feature_importance_ property of the classiﬁer, which computes the normalized total reduction of the classiﬁer criterion loss contributed by each latent.

Original: Disentanglement is then measured as
Revised: Disentanglement is then measured as

Original: [Equation]
Revised: [Equation]

Original:  where H is the entropy computed with the stats.entropy function from scikit-learn , 
Revised: 27) where H is the entropy computed with the stats.entropy function from scikit-learn , I ∈

Original: I ∈ R D × D (cid:48) is a matrix of the absolute value of the feature importance between each factor and each ground truth, and ˜ I is a normalized version of the matrix
Revised: R D × D [Equation] is a matrix of the absolute value of the feature importance between each factor and each ground truth, and ˜ I is a normalized version of the matrix

Original: [Equation]
Revised: [Equation]

Original: SAP Score (Kumar et al., 2018)
Revised: SAP Score (Kumar et al., 2018)

Original: To compute the SAP score, Kumar et al.
Revised: To compute the SAP score, Kumar et al.

Original: ﬁrst train a linear support vector classiﬁer with squared hinge loss and L 2 penalty to predict each ground truth factor from each latent variable.
Revised: ﬁrst train a linear support vector classiﬁer with squared hinge loss and L 2 penalty to predict each ground truth factor from each latent variable.

Original: In DisLib this is implemented with the svm.
Revised: In DisLib this is implemented with the svm.

Original: LinearSVC function with default parameters from scikit-learn .
Revised: LinearSVC function with default parameters from scikit-learn .

Original: They construct a score matrix S ∈ R D (cid:48)
Revised: They construct a score matrix S ∈ R D (cid:48)

Original: × D , where each entry in the matrix is the batch-mean classiﬁer accuracy for predicting each ground truth given each individual latent encoding.
Revised: × D , where each entry in the matrix is the batch-mean classiﬁer accuracy for predicting each ground truth given each individual latent encoding.

Original: For each generative factor, they compute the difference between the top two most predictive latent dimensions, which are the two highest scores in a given column of S .
Revised: For each generative factor, they compute the difference between the top two most predictive latent dimensions, which are the two highest scores in a given column of S .

Original: The mean (across ground-truth factors) of these differences is the SAP score.
Revised: The mean (across ground-truth factors) of these differences is the SAP score.

Original: E E STIMATION M ETHODS E. M ODEL T RAINING AND S ELECTION
Revised: E N ATURAL D ATASETS

Original: We train all models on all datasets provided in DisLib with the UNI and LAP variants. We also train SlowVAE and β -VAE on Natural Sprites and KITTI Masks .
Revised: We introduce several datasets to investigate disentanglement in more natural scenarios. Here, we provide an overview on the motivation and design of each dataset.

Original: When available, we compare scores directly using models we train, otherwise we reprint scores reported in (Locatello et al., 2020). E. I MPLEMENTATION D ETAILS
Revised: In Appendix F.2 we give speciﬁc implementation details. An issue with evaluating disentanglement on natural datasets is the fact that the existing disentanglement metrics require knowledge of the underlying generative process of the given data.

Original: All models are implemented in PyTorch (Paszke et al., 2019). To facilitate comparison, the training parameters, e.g. optimizer, batch size, number of training steps, as well as the VAE encoder and decoder architecture are identical to those reported in (Locatello et al., 2018; 2020). We use this architecture for all datasets, only adjusting the number of input channels (greyscale for dSprites, smallNORB, and KITTI Masks; three color channels for all other datasets). The model formulation is agnostic to  the direction of time.
Revised: Although we can observe that the world is composed of distinct entities that vary according to rules imposed by physics, we are unable to determine the appropriate “factors” that generate such scenes.

Original: Therefore, to increase the temporal training signal at a ﬁxed computational cost for each batch of input pairs ( x 0 , x 1 ) , we optimize the model in both directions i.e. optimizing the model objective for both t 0 = 0 , t 1 = 1 as well ast 0 = 1 , t 1 = 0 . F N ATURAL D ATASETS We introduce several datasets to investigate disentanglement in more natural scenarios. Here, we provide a detailed description on the design of each dataset.
Revised: To mitigate this problem, we compile object measurements by calculating the x and y coordinates of the center of mass as well as the area of object masks in natural video frames.

Original: F. L APLACE T RANSITIONS
Revised: We use these measurements to a) inform new disentanglement benchmarks with natural transitions that have similar complexity to existing benchmarks (Natural Sprites) and b) evaluate the ability of algorithms to decode intrinsic object properties (KITTI Masks). We additionally propose a simple extension to the existing DisLib datasets in the form of collecting images into pairs that exhibit sparse (i.e. Laplace) transition probabilities. E. L APLACE T RANSITIONS

Original: For each of the datasets in DisLib, we collect pairs of images.
Revised: For each of the datasets in DisLib, we collect pairs of images.

Original: For each ground-truth factor, the ﬁrst value in the pair is chosen from a uniform distribution across all possible values in latent space, while the second is chosen by weighting nearby values in latent space using Laplace distributed probabilities (see equation 2).
Revised: For each ground-truth factor, the ﬁrst value in the pair is chosen from a uniform distribution across all possible values in latent space, while the second is chosen by weighting nearby values in latent space using Laplace distributed probabilities (see Eq. 2).

Original: Since we are restricted to the set of given factor values, the dataset has edge conditions where samples that would push a factor outside  the preset range are rejected.
Revised: We reject samples that would push a factor outside of the preset range provided by the dataset.

Original: This may, however, alleviate issues introduced by rotation symmetries since, under thisLaplace prior in the latent space, it is extremely unlikely to observe ’large’ transitions from orientations of θ = 2 π − (cid:15) to θ = (cid:15) . This dataset introduces a hyper-parameter λ that controls the rate of the Laplace sampling distribution, while the location is set by the initial factor value.
Revised: We call this the LAP DisLib extension.

Original: Effectively, when this rate is λ =most of the factors change most of the time, whereas for a rate of λ = 10 most ofthe factorswill not change most of the time (recall thatwe reject samples without any changes, so at least one factor will change in every valid sample).
Revised: Although the sparse prior indicates that any individual factor is more likely to remain constant, the number of factors that change in a given transition is still typically greater than one.

Original: To maximize comparability with (Locatello et al., 2020), for each pair of inputs we sample the rate of the Laplace prior from a uniform distribution λ ∼ U (1 , 10) .
Revised: To show this in Fig. 8, we sampled 10, transitions from each DisLib dataset with LAP transitions and computed the number of factors that had changed within a pair. This extension of the DisLib datasets provides a bridge from i.i.d.

Original: This correspond to  the setting ( k = rand) in the LOC data, which is reported throughout the paper.
Revised: data to natural data by explicitly modeling the observed sparse marginal transition distributions.

Original: F. Y OU T UBE -VOS
Revised: E. Y OU T UBE -VOS

Original: For the YouTube dataset, we download annotations from the 2019 version of the video instance segmentation (Youtube-VIS) dataset
Revised: For the YouTube dataset, we download annotations from the 2019 version of the video instance segmentation (Youtube-VIS) dataset

Original: (Yang et al., 2019) 3 , which is built on top of the video object segmentation (Youtube-VOS) dataset (Xu et al., 2018).
Revised: (Yang et al., 2019) 3 , which is built on top of the video object segmentation (Youtube-VOS) dataset (Xu et al., 2018).

Original: The dataset has  annotations for every ﬁve frames in a 30fps video, which results in a 6fps sampling rate.
Revised: The dataset has multi-object annotations for every ﬁve frames in a 30fps video, which results in a 6fps sampling rate.

Original: The authors believe that the temporal correlation between ﬁve consecutive frames is sufﬁciently strong that annotations can be omitted for intermediate frames to reduce the annotation efforts.
Revised: The authors state that the temporal correlation between ﬁve consecutive frames is sufﬁciently strong that annotations can be omitted for intermediate frames to reduce the annotation efforts.

Original: Such a skip-frame annotation strategy enables scaling up the number of videos and objects annotated under the same budget, yielding 131, annotations for 2,883 videos, with 4,883 unique video object instances.
Revised: Such a skip-frame annotation strategy enables scaling up the number of videos and objects annotated under the same budget, yielding 131,000 annotations for 2,883 videos, with 4,883 unique video object instances.

Original: The original image size of the dataset is 720 × 1280 . In order to preserve the statistics of the transitions, we choose not to directly downsample to 64 × 64 , but instead preserve the aspect ratio by downsampling to 64 × 128 . In order to minimize the bias yielded by the extraction method, noting the center bias typically present in human videos, we extract three equally spaced 64 × 64 pixel windows (with overlap) with a stride of 32 . For each resulting 64 × 64 × T sequence, where T denotes the number of time steps in the sequence, we ﬁlter out all pairs where the given object instance is not present in adjacent frames, resulting in 234,652 pairs.
Revised: Although we do not evaluate against YouTube-VOS in this study, we see it as the logical next step in transitioning to natural data.

Original: An issue with evaluating disentanglement on natural datasets is the fact that the existing disentanglement metrics require knowledge of the underlying generative process of the given data. Although we can observe that the world is composed of distinct entities that vary according to rules imposed by physics, we are unable to determine the precise “factors” that generate such scenes. To mitigate this problem, we compile object measurements by calculating the x and y coordinates of the center of mass as well as the area ofeach object in each frame. We use these measurements to a) augment existing disentanglement benchmarks with natural transitions (Natural Sprites) and b) evaluate the ability of algorithms to decode intrinsic object properties (KITTI Masks).
Revised: The large scale, lack of environmental constraints, and abundance of object types makes it the most challenging of the datasets considered herein.

Original: F. N ATURAL S PRITES
Revised: E. N ATURAL S PRITES

Original: Without a deﬁnition of disentanglement that can be applied to unknown data generating processes, we are limited to synthetic datasets with known ground-truth factors.
Revised: Without a deﬁnition of disentanglement that can be applied to unknown data generating processes, we are limited to synthetic datasets with known ground-truth factors.

Original: Let us take dSprites (Matthey et al., 2017) as an example.
Revised: Let us take dSprites (Matthey et al., 2017) as an example.

Original: The dataset consists of all combinations of a set of latent factor values, namely,
Revised: The dataset consists of all combinations of a set of latent factor values, namely,

Original: • Color: white • Shape: square, ellipse, heart • Scale: 6 values linearly spaced in [0 . 5 , 1] • Orientation: 40 values in [0 , 2 π ] • Position X : 32 values in [0 , 1] • Position Y : 32 values in [0 , 1]
Revised: • Color: white • Shape: square, ellipse, heart • Scale: 6 values linearly spaced in [0 . 5 , 1] • Orientation: 40 values in [0 , 2 π ] • Position X : 32 values in [0 , 1] • Position Y : 32 values in [0 , 1]

Original: Given the limited set of discrete values each factor can take on, all possible samples can be described by a tractable dataset, compiled and released to the public.
Revised: Given the limited set of discrete values each factor can take on, all possible samples can be described by a tractable dataset, compiled and released to the public.

Original: But, in reality, all of these factors should be continuous: a spectrum of possible colors, shapes, scales, orientations, and positions exist.
Revised: But, in reality, all of these factors should be continuous: a spectrum of possible colors, shapes, scales, orientations, and positions exist.

Original: We address this by constructing a dataset that is augmented with natural and continuous ground truth[Table] factors, using the mask properties measured from thelarge-scale, natural, unstructured YouTube (Xu et al., 2018; Yang et al., 2019) dataset.
Revised: We address this by constructing a dataset that is augmented with natural and continuous ground truth factors, using the mask properties measured from the YouTube dataset described in Appendix F.2.

Original: We can choose the complexity of the dataset by discretizing the 234,652 transition pairs of position and scale into an arbitrary number of bins.
Revised: We can choose the complexity of the dataset by discretizing the 234,652 transition pairs of position and scale into an arbitrary number of bins.

Original: In this study, we discretize to match  dSprites, which we present in Table 4.
Revised: In this study, we discretize to match the number of possible object states as dSprites, which we present in Table 4.

Revised: This helps isolate the effect of including natural transitions from the effect of increasing data complexity.

Original: We produce a pair by ﬁxing the color, shape, and orientation, but updating the position and scale with transitions sampled from the YouTube measurements.
Revised: We produce a pair by ﬁxing the color, shape, and orientation, but updating the position and scale with transitions sampled from the YouTube measurements.

Revised: We motivate ﬁxing shape and color by noting that this is consistent with object permanence in the real world.

Revised: We decided to ﬁx the orientation because we do not currently have a way to approximate it from object masks and we did not want to introduce artiﬁcial transition probabilities.

Original: To minimize the effect of extreme outliers, we ﬁlter out 10% of the data by removing frames if the mask area falls below the 5% or above the 95% quantiles, which reduces the number of pairs to 207,794.
Revised: To minimize the effect of extreme outliers, we ﬁlter out 10% of the data by removing frames if the mask area falls below the 5% or above the 95% quantiles, which reduces the number of pairs to 207,794.

Original: Finally, we use the Spriteworld (Watters et al., 2019) renderer to generate the images.
Revised: Finally, we use the Spriteworld (Watters et al., 2019) renderer to generate the images.

Revised: Spriteworld allows us to render entirely new sprite objects at the precise position and scale as was measured from YouTube.

Revised: For example, if one would want to apply YouTube-VOS transitions to MPI3D (Gondal et al., 2019), this option is unavailable without the associated renderer.

Original: The benchmark is available at https://zenodo.org/record/3948069 .
Revised: The benchmark is available at https://zenodo.org/record/3948069 .

Original: In relation to the Laplace transitions described in section F.1, this update a) produces pairs that correspond to transitions observed in real data and b) allows for smooth transitions by deﬁning the data generation process as opposed to being limited by the given collected dataset (e.g. dSprites).
Revised: In relation to the Laplace transitions described in section E.1, this update i) produces pairs that correspond to transitions observed in real data, ii) allows for smooth transitions by deﬁning the data generation process as opposed to being limited by the given collected dataset (e.g. dSprites), and iii) includes complex dependencies among factors that are present in natural data.

Original: We generate the data online, thus training the model to ﬁt the underlying distribution as opposed to a sampled ﬁnite dataset.
Revised: We generate the data online, thus training the model to ﬁt the underlying distribution as opposed to a sampled ﬁnite dataset.

Original: However, as noted previously, all supervised metrics aggregated in DisLib are inapplicable to continuous factors, which is problematic as the generating distribution is effectively continuous with respect to a subset of the factors.
Revised: However, as noted previously, all supervised metrics aggregated in DisLib are inapplicable to continuous factors, which is problematic as the generating distribution is effectively continuous with respect to a subset of the factors.

Original: Therefore, we limit our quantitative evaluation to MCC for continuous datasets.
Revised: Therefore, we limit our quantitative evaluation to MCC for continuous datasets.

Original: Additionally, as mentioned above, to evaluate disentanglement with the standard metrics, we bin the collected transition statistics to match the size of dSprites and use the set of bin centers as the new set of values. The details of which are in Table 4, note that the discretization corresponds to the scale of the dSprites factor set.
Revised: However, we are able to evaluate disentanglement with the standard metrics on the discretized version.

Original: F. KITTI MOTS P EDESTRIAN M ASKS
Revised: E. KITTI MOTS P EDESTRIAN M ASKS

Original: While Natural Sprites enables evaluation of disentanglement with natural transitions, we note that the disentanglement framework which requires knowledge of the underlying generative factors is unrealistic for real-world data.
Revised: While Natural Sprites enables evaluation of disentanglement with natural transitions, we note that any disentanglement framework that requires knowledge of the underlying generative factors is unrealistic [Table] [Table]

Original: As we state in the introduction, the assumption that a visual representation of the world can be constructed via a generative process that receives factors of variation as input and produces natural signals as output is not true in the literal sense.
Revised: [Figure]

Original:  Measurements such as scale and position correspond to object properties that are ecologically relevant to the observer and can serve as suitable alternatives to the typical generative factors.
Revised: corresponds to the area. for real-world data. Measurements such as scale and position correspond to object properties that are ecologically relevant to the observer and can serve as suitable alternatives to the typical generative factors.

Original: We directly test this using our KITTI Masks dataset.
Revised: We directly test this using our KITTI Masks dataset.

Original: To create the dataset, we download annotations from the Multi-Object Tracking and Segmentation (MOTS) Evaluation Benchmark (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016), which is split into KITTI MOTS and MOTSChallenge 4 .
Revised: To create the dataset, we download annotations from the Multi-Object Tracking and Segmentation (MOTS) Evaluation Benchmark (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016), which is split into KITTI MOTS and MOTSChallenge 4 .

Original: Both datasets contain sequences of pedestrians with their positions densely annotated in the time and pixel domains.
Revised: Both datasets contain sequences of pedestrians with their positions densely annotated in the time and pixel domains.

Revised: For simplicity, we only consider the instance segmentation masks for pedestrians and do not use the raw data.

Original: The original image sizes are 1080 × 1920 or 480 × 640 resolution for MOTSChallenge and between 370 and 374 pixels tall by 1224 and 1242 pixels wide for KITTI MOTS.
Revised: The resulting KITTI Masks dataset consists of 2,120 sequences of individual pedestrians with lengths between 2 and 710 frames each, resulting in a total of 84,626 individual frames. As we did with

Original: For simplicity, we only consider the instance segmentation masks for pedestrians and do not use the raw data.
Revised: YouTube-VOS, we estimate ground truth factors by calculating the x and y coordinates of the center of mass of each pedestrian mask in each frame. We deﬁne the object size as the area of the mask, i.e. the total number of pixels. We consider the disentanglement performance for different mean time gaps between image pairs in table 2 and Appendix H.3.

Original: We use nearest neighbor down-sampling for each frame such that the height was 64 pixels and the width is set to conserve the aspect ratio.
Revised: In Appendix F.2.4 we discuss how the variable framerate of the KITTI datasets leads to us reporting the mean time gap between frames for a given image pair. For samples and the corresponding ground truth factors see Fig. 9. The benchmark is available at https://zenodo.org/record/3931823 . F I MPLEMENTATION D ETAILS F. M ODEL T RAINING AND S ELECTION

Revised: We train all models on all datasets provided in DisLib with the UNI and LAP variants.

Revised: All models are implemented in PyTorch (Paszke et al., 2019).

Revised: To facilitate comparison, the training parameters, e.g. optimizer, batch size, number of training steps, as well as the VAE encoder and decoder architecture are identical to those reported in (Locatello et al., 2018; 2020).

Revised: We use this architecture for all datasets, only adjusting the number of input channels (greyscale for dSprites, smallNORB, and KITTI Masks; three color channels for all other datasets).

Revised: The model formulation is agnostic to the direction of time.

Revised: Therefore, to increase the temporal training signal at a ﬁxed computational cost for each batch of input pairs ( x 0 , x 1 ) , we optimize the model in both directions i.e. optimizing the model objective for both t 0 = 0 , t 1 = 1 as well as t 0 = 1 , t 1 = 0 .

Revised: F. D ATASET D ETAILS

Revised: We have chosen to work with pairs of inputs as minimal sequences because we are interested in the ﬁrst temporal derivative, more speciﬁcally in the sparsity of the transitions between pairs of images.

Revised: Other methods that look at the second temporal derivative, such as work from Hénaff et al.

Revised: (2019) on straightening, would require triplets as minimal sequences.

Revised: Extending our approach beyond this minimal requirement would be simple in terms of the resulting ELBO (which would still factorise like in Eq.

Revised: 4 because of the Markov property).

Revised: The only additional complexity would be in the data and loss handling.

Revised: We note that both YouTube-VOS (Xu et al., 2018; Yang et al., 2019) and KITTI-MOTS (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) are multi-object datasets, although we consider each unique object (mask) separately.

Revised: Multi-object representation learning and disentanglement are highly connected, in fact they have recently begun to be used interchangeably (Wulfmeier et al., 2020).

Revised: To brieﬂy comment on possible extensions in this direction, we see no reason why our prior would not be beneﬁcial to multi-object methods such as MONet (Burgess et al., 2019) and IODINE (Greff et al., 2019), or video extensions such as ViMON (Weis et al., 2020) and OP3 (Veerapaneni et al., 2019).

Revised: F.2. UNI

Revised: The key differences between our implementation of UNI , which we based on the description in the paper (Locatello et al., 2020), and the what was used by Locatello et al.

Revised: (2020) is: (i) their coderandomly (with 50% probability) sets k = 1 even in the k =

Revised: Rnd setting, and (ii) we ensure that exactly k factors change.

Revised: Though we consider this discrepancy minor, we nonetheless label all results from (Locatello et al., 2020) with “LOC”, as opposed to “UNI”, for clarity.

Revised: F.2. LAP

Revised: When training models on the LAP dataset it is possible to reject samples without transitions (i.e. all factors remain constant) since the pair would not result in any temporal learning signal.

Revised: However, it would arguably be more natural to leave these samples as they would more accurately reﬂect occurrences of stationary objects in real data.

Revised: We report the rejection setting in the main text, but found no signiﬁcant difference between the two settings (see Appendix H).

Revised: This dataset also introduces a hyper-parameter λ that controls the rate of the Laplace sampling distribution, while the location is set by the initial factor value.

Revised: Effectively, when this rate is λ =most of the factors change most of the time, whereas for a rate of λ = 10 most of the factors will not change most of the time.

Revised: Note that this means λ (inversely) changes the scale, which results in larger or smaller movements, but does not affect the distribution itself.

Revised: In other words, the sparsity is unchanged, as the sparsity is controlled by the shape α .

Revised: We ﬁx λ = 1 , which yields multiple changes, thus making this dataset fundamentally different both in spirit and in practice, from the UNI dataset modeled after Locatello et al.

Revised: F.2. Y OU T UBE -VOS

Revised: The original image size of the YouTube-VOS dataset is 720 × 1280 .

Revised: In order to preserve the statistics of the transitions, we choose not to directly downsample to 64 × 64 , but instead preserve the aspect ratio by downsampling to 64 × 128 .

Revised: In order to minimize the bias yielded by the extraction method, noting the center bias typically present in human videos, we extract three overlapping, equally spaced 64 × 64 pixel windows with a stride of 32 .

Revised: For each resulting 64 × 64 × T sequence, where T denotes the number of time steps in the sequence, we ﬁlter out all pairs where the given object instance is not present in adjacent frames, resulting in 234,652 pairs.

Revised: F.2. KITTI M ASKS

Revised: The original KITTI image sizes are 1080 × 1920 or 480 × 640 resolution for MOTSChallenge and between 370 and 374 pixels tall by 1224 and 1242 pixels wide for KITTI MOTS.

Original:  After down-sampling, we use a horizontal sliding window approach to extract six equally spaced windows of size 64 × 64 (with overlap) for each sequence in both datasets.
Revised: The frame rates of the videos vary from 14 to 30 fps, which can be seen in Table 2 of Milan et al. We use nearest neighbor down-sampling for each frame such that the height was 64 pixels and the width is set to conserve the aspect ratio. After down-sampling, we use a horizontal sliding window approach to extract six equally spaced windows of size 64 × 64 (with overlap) for each sequence in both datasets This results in a 64 × 64 × T sequence, where T denotes the number of time steps in the sequence.

Original: Note that here we make reasonable assumptions on horizontal translation and scale invariance of the dataset.
Revised: Note that here we make reasonable assumptions on horizontal translation and scale invariance of the dataset.

Original: We justify the assumed scale invariance by observing that the data is collected from a camera mounted onto a car which has varying distance to pedestrians.
Revised: We justify the assumed scale invariance by observing that the data is collected from a camera mounted onto a car which has varying distance to pedestrians.

Original: To conﬁrm the translation invariance, we show an ablation study on the number of horizontal images.
Revised: To conﬁrm the translation invariance, we performed an ablation study on the number of horizontal images.

Original: Instead of six horizontal, equally spaced sliding windows, we only use two which leads to differently placed windows.
Revised: Instead of six horizontal, equally spaced sliding windows, we only use two which leads to differently placed windows.

Original: We do not observe signiﬁcant changes in the reported data statistics (e.g. the kurtosis of the ﬁt stays within ± 10% of the previous value for ∆ x transitions).
Revised: We do not observe signiﬁcant changes in the reported data statistics (e.g. the kurtosis of the ﬁt stays within ± 10% of the previous value for ∆ x transitions).

Original: The values of ∆ y and ∆ area do not change signiﬁcantly compared to Table 6.
Revised: The values of ∆ y and ∆ area do not change signiﬁcantly compared to Table 7.

Original: For each resulting 64 × 64 × T sequence, where T denotes the number of time steps in the sequence, we extract all individual pedestrian masks based on their object instance identity and create a new sequence for each pedestrian such that each resulting sequence only contains a single pedestrian.
Revised: For each resulting 64 × 64 × T sequence, where T denotes the number of time steps in the sequence, we extract all individual pedestrian masks based on their object instance identity and create a new sequence for each pedestrian such that each resulting sequence only contains a single pedestrian.

Original: We ignore images with masks that have less than 30 pixels as they are too far away or occluded and  not recognizable by humans.
Revised: We ignore images with masks that have less than 30 pixels as they are too far away or occluded and were not recognizable by the authors.

Original: We keep all sequences of two or more frames, as the algorithm only requires pairs of frames for training.
Revised: We keep all sequences of two or more frames, as the algorithm only requires pairs of frames for training.

Original: The resulting KITTI Masks dataset consists of 2,120 sequences of individual pedestrians with lengths between 2 and 710 frames each, resulting in a total of 84,626 individual frames. We estimate ground truth factors by calculating the x and y coordinates of the center of mass of each pedestrian in each frame and the area (number of pixels in mask).
Revised: We leave the maximum distance between time frames within a pair, max (∆ frames ) , as a hyperparameter. For a given max (∆ frames ) , we report the mean change in physical time in seconds (denoted by mean (∆ t ) ). We test adjacent frames (max (∆ frames ) = 1 ), which corresponds to a mean (∆ t = 0 . 05) and max (∆ frames )

Original: We leave the ∆ t  between time frames within a pair as a hyperparameter, and test ∆ t = 1 and ∆ t = 5 as described in the main paper.
Revised: = 5 , which corresponds to a mean (∆ t = 0 . 15) . This procedure is motivated by the fact that different sequences were recorded with different frame rates and reporting the mean (∆ t ) in seconds allows for a physical interpretation. The relationship between max (∆ frames ) and mean (∆ t ) is in Fig.

Original:  During training, we augment the data by applying horizontal and vertical translations of ± 5 pixels and rotations of ± 2 ◦ degree.
Revised: We show results for testing additional values of mean (∆ t ) in Appendix H.3. During training, we augment the data by applying horizontal and vertical translations of ± 5 pixels and rotations of ± 2 ◦ degree.

Original: We apply the exact same data augmentation to both images within a pair to not change any transition statistics.
Revised: We apply the exact same data augmentation to both images within a pair to not change any transition statistics.

Original: For samples and the corresponding ground truth factors see Fig. 8.

Original: The benchmark is available at https://zenodo.org/record/3931823 .

Original: G E XTENDED C  OMPARISON TO S IMILAR M ETHODS G. T HEORETICAL C OMPARISON
Revised: G E XTENDED C OMPARISONS AND C ONTROLS G. C OMPARISON TO N ONLINEAR ICA G.1. T HEORETICAL C OMPARISON

Original: Nonlinear ICA has recently been advanced signiﬁcantly by several papers from Hyvärinen and colleagues.
Revised: Nonlinear ICA has recently been advanced signiﬁcantly by several papers from Hyvärinen and colleagues.

Original: Of these studies, the two that are most comparable to our work is Hyvärinen and Morioka (2017), which uses an unsupervised contrastive loss for nonlinear demixing and Khemakhem et al. 
Revised: Of these studies, the two that are most comparable to our work is Hyvärinen and Morioka (2017), which uses an unsupervised contrastive loss for nonlinear demixing and Khemakhem et al. [Figure]

Original: (2020), which extends the nonlinear ICA framework to include variational autoencoders (VAEs).
Revised: (2020a), which extends the nonlinear ICA framework to include variational autoencoders (VAEs).

Original: However, our theory covers an important class of transitions relevant for natural data that is not covered by the identiﬁability proofs of either of the aforementioned studies.
Revised: However, our theory covers an important class of transitions relevant for natural data that is not covered by the identiﬁability proofs of either of the aforementioned studies.

Original: Speciﬁcally, the nonGaussian autoregressive model  their identiﬁability proof rests upon (their equation (8)) assumes that the second derivative of the innovation probability density function is less than zero to satisfy uniform dependence , which is only met for α > 1 for generalized Laplace transitions.
Revised: As a speciﬁc comparison to the ﬁrst paper, the non-Gaussian autoregressive model that their identiﬁability proof rests upon (Eq. 8 in Hyvärinen and Morioka, 2017) assumes that the second derivative of the innovation probability density function is less than zero to satisfy uniform dependence , which is only met for α > 1 for generalized Laplace transition distributions.

Original: While they denote (footnote 3) that Laplace distributions ( α = 1 ) are not covered by their theory, they offer a suggestion for a smooth approximation.
Revised: While they denote (footnote 3) that Laplace distributions ( α = 1 ) are not covered by their theory, they offer a suggestion for a smooth approximation.

Original: However, they do not demonstrate that this approximation is useful in practice.
Revised: However, they do not demonstrate that this approximation is useful in practice, or offer a solution to a general class of sparse distributions for α ≤ 1 .

Original: We chose a generalized Laplacian to ﬁt our data and for our model assumption as it allows for simple parameterization of ﬁts to data (e.g. α = 0 .
Revised: We chose a generalized Laplacian to ﬁt our data and for our model assumption as it allows for simple parameterization of ﬁts to data (e.g. α = 0 .

Original: 5 for natural movie transitions), but is simultaneously quite expressive (Sinz et al., 2009).
Revised: 5 for natural movie transitions), but is simultaneously quite expressive (Sinz et al., 2009).

Original: Though we use α = 1 in practice for our estimation method, we prove identiﬁability up to permutations and sign ﬂips for any α < 2 , covering all sparse distributions under the expressive generalizedLaplacian model.
Revised: Though we use α = 1 in practice for our estimation method, we prove identiﬁability up to permutations and sign ﬂips for any α < 2 , covering all sparse distributions under the expressive generalized

Original: Hyvärinen and Morioka (2017) do not provide a class of smooth approximations for α < = 1 , and thus their theory does not achieve the same ends.
Revised: Laplacian model.

Original: In addition, we assume a Gaussian marginal distribution that allows us to derive a fundamentally stronger proof of identiﬁability – where we identify up to permutation and sign-ﬂips.
Revised: In addition, we assume a Gaussian marginal distribution that allows us to derive a fundamentally stronger proof of identiﬁability – where we identify up to permutation and sign-ﬂips.

Original: Hyvärinen and Morioka (2017) only identify the sources up to arbitrary non-linear element-wise transformations.
Revised: Hyvärinen and Morioka (2017) only identify the sources up to arbitrary non-linear element-wise transformations.

Original: Thus they require a subsequent step of ICA  recover the signal up to permutations and sign ﬂips for a class of distributions where it is unclear whether they account for temporal sparsity.
Revised: Thus they require a subsequent step of ICA (under the typical assumption that at most one marginal source distribution is Gaussian) to recover the signal up to permutations and sign ﬂips for a class of distributions where it is unclear whether they account for temporal sparsity.

Original: This difference isalso evident in the form of the conditional priors, p ( z t | z t − 1 ) , used by us and by Khemakhem et al.
Revised: The work of Khemakhem et al. (2020a) has a couple of differences from our own, most notable of which is the form of the conditional prior, p ( z t | z t − 1 ) .

Original: They assume that the conditional posterior is part of the exponential family, but this does not include Laplacian conditionals.
Revised: They assume that the conditional posterior is part of the exponential family, which does not include Laplacian conditionals.

Original: Even though the exponential family contains the Laplace distribution with ﬁxed mean as its member, this does not allow their approach to model sparse transitions.
Revised: Though the exponential family contains the Laplace distribution with ﬁxed mean as its member, it does not allow their approach to model sparse transitions.

Original: They assume that the natural parameters of the exponential family distribution are conditioned on z t − 1 , meaning that only the scale but not the mean of the Laplace prior for z t can be modulated by the previous time step, thus not allowing for sparse transition probabilities.
Revised: They assume that the natural parameters of the exponential family distribution are conditioned on z t − 1 , meaning that only the scale but not the mean of the Laplace prior for z t can be modulated by the previous time step, thus not allowing for sparse transition probabilities.

Original: Additionally, their implementation requires the number of classes (i.e. states of the conditioning variable) to equal the number of stationary segments, which is impractical for the datasets we consider.
Revised: Additionally, their implementation requires the number of classes (i.e. states of the conditioning variable) to equal the number of stationary segments, which is impractical for the datasets we consider.

Original: [Table] Thus, we provide a closer match to natural data transitions, with a stronger identiﬁability result, as well as an implementation capable of scale when compared to previous nonlinear ICA approaches.
Revised: Thus, we provide a closer match to natural data transitions, with a stronger identiﬁability result.

Original: We provide validation by performing an extensive evaluation leveraging our contributed datasets as well as the models, metrics, and datasets provided by the Disentanglement Library (discussed in section 4).
Revised: We provide validation by performing an extensive evaluation leveraging our contributed datasets as well as the models, metrics, and datasets provided by the Disentanglement Library (DisLib, discussed in section 4).

Original: We consider methodsfunctionally capable of processing transitions fromnot only the disentanglement literature (Locatello et al., 2020), but that of nonlinear ICA (Hyvärinen and Morioka, 2017).
Revised: We consider methods from the disentanglement literature (Locatello et al., 2020) as well as nonlinear ICA (Hyvärinen and Morioka, 2017), that are functionally capable of processing transitions.

Original: As (Hyvärinen and Morioka, 2017), unlike (Hyvärinen and Morioka, 2016; Khemakhem et al., 2020), contributes a ﬁtting model, we consider their simulation setting more carefully in the following section.

Original: G. E MPIRICAL C OMPARISON
Revised: G.1. E MPIRICAL C OMPARISON

Original: Hyvärinen and Morioka (2017) conducted a simulation where the sources in the nonlinear ICA model come from a linear autoregressive (AR) model with non-Gaussian innovations.
Revised: Hyvärinen and Morioka (2017) conducted a simulation where the sources in the nonlinear ICA model come from a linear autoregressive (AR) model with non-Gaussian innovations.

Original: Speciﬁcally, temporally dependent 20 -dimensional source signals were randomly generated according to log p ( s ( t ) | s ( t − 1)) =
Revised: Speciﬁcally, temporally dependent 20 -dimensional source signals were randomly generated according to log p ( s ( t ) | s ( t − 1)) =

Original: −| s ( t ) − 0 .
Revised: −| s ( t ) − 0 .

Original: 7 s ( t − 1) | .
Revised: 7 s ( t − 1) | .

Original: Though this generative process was noted to not be covered by the theory presented in (Hyvärinen and Morioka, 2017), the authors demonstrated that 
Revised: Though this generative process was noted to not be covered by the theory presented in (Hyvärinen and Morioka, 2017), the authors demonstrated that [Table]

Original: PCL could reconstruct the source signals reasonably well even for the nonlinear mixture case.
Revised: PCL could reconstruct the source signals reasonably well even for the nonlinear mixture case.

Original: Given our practical use of a Laplacian conditional, we evaluated our theory in this artiﬁcial setting.
Revised: Given our practical use of a Laplacian conditional, we found it a valuable comparison to evaluate our theory in this artiﬁcial setting.

Original: Given  C, we use SlowFlow for these experiments.
Revised: Given the discussion in Appendix C, we use SlowFlow for these experiments.

Original: For computational tractability in demixing highly nonlinear transformations, we consider normalizing ﬂows (Dinh et al., 2017a;b; Kingma and Dhariwal, 2018), namely volume-preserving ﬂows (Sorrenson et al., 2017), as we ﬁnd constraining the Jacobian determinant to stabilize learning.
Revised: For computational tractability in demixing highly nonlinear transformations, we consider normalizing ﬂows (Dinh et al., 2017a;b; Kingma and Dhariwal, 2018), namely volume-preserving ﬂows (Sorrenson et al., 2017), as we ﬁnd constraining the Jacobian determinant stabilizes learning.

Original: To ensure sufﬁcient expressivity, we consider 6 coupling blocks, each containing a 2 -layer MLP with 500 hidden units and ReLU nonlinearities.
Revised: To ensure sufﬁcient expressivity, we consider 6 coupling blocks, each containing a 2 -layer MLP with 500 hidden units and ReLU nonlinearities.

Original: We compare to the PCL implementation presented in (Hyvärinen et al., 2018), where an MLP with the same number of hidden layers as the mixing MLP was adopted, where we use 100 hidden units as we did not ﬁnd increasing the value improved performance.
Revised: We compare to the PCL implementation presented in (Hyvärinen and Morioka, 2017), where an MLP with the same number of hidden layers as the mixing MLP was adopted. We use 100 hidden units as we did not ﬁnd increasing the value improved performance.

Original: To account for the architectural difference serving as a possible confounder, we use the same  encoder for optimizing the PCL objective, which we term “PCL (NF)”.
Revised: To account for the architectural difference serving as a possible confounder, we use the same normalizing ﬂow encoder for optimizing the PCL objective, which we term “PCL (NF)”.

Original: While (Hyvärinen and Morioka, 2017) used leaky ReLU nonlinearities to make the mixing invertible, said mixing is non-differentiable.
Revised: While (Hyvärinen and Morioka, 2017) used leaky ReLU nonlinearities to make the mixing invertible, said mixing is non-differentiable.

Original: This is problematic for SlowFlow, as it involves gradient optimization of the Jacobian term, and more importantly, unlike PCL, aims to explicitly recover the mixing process.
Revised: This is problematic for SlowFlow, as it involves gradient optimization of the Jacobian term, and more importantly, unlike PCL, aims to explicitly recover the mixing process.

Original: We thus use a a smooth version of the leaky-ReLU activation function with a hyperparameter α (Gresele et al., 2020),
Revised: We thus use a a smooth version of the leaky-ReLU activation function with a hyperparameter α (Gresele et al., 2020),

Original: [Equation]
Revised: [Equation]

Original: By ensuring the mixing process is smooth, we ﬁnd that SlowFlow performs favorably relative to
Revised: By ensuring the mixing process is smooth, we ﬁnd that SlowFlow performs favorably relative to

Original: PCL 5 when evaluated in the same setting, converging to a better optimum at higher levels of mixing.
Revised: PCL (Table 5) when evaluated in the same setting, converging to a better optimum at higher levels of mixing.

Revised: G. J OINT F ACTOR D EPENDENCE E VALUATION

Original: [Table]
Revised: In order to consider joint dependencies among natural generative factors, we leverage Natural Sprites to construct modiﬁed datasets where time-pairs of factors are shufﬂed per-factor (e.g. combining the x transition from one clip with the y transition from a different clip). This destroys dependencies between the factors, while maintaining the sparse marginal distributions. In Fig. 11 (right), we show 2D marginals before (blue) and after (orange) this shufﬂing.

Revised: The additional density on the diagonals in the unshufﬂed data reveals dependencies between pairs of factors on both datasets.

Revised: As mentioned in section 3.4, the observed dependency is mismatched from the theoretical assumptions of our model.

Revised: We test how robust SlowVAE is to such a mismatch by training it on the permuted data and reevaluating disentanglement.

Original: [Table]
Revised: In Table 22, we highlight that the improvement of SlowVAE on the permuted (i.e. independent) continuous Natural Sprites is not signiﬁcant. In Table 21, we surprisingly ﬁnd an overall improved score with non-permuted transitions (i.e. with dependencies), with three out of seven metrics showing a signiﬁcant improvement. This is in line with Fig. 1f in Khemakhem et al.

Revised: (2020b), where, at least for simple mixing, a model (Khemakhem et al., 2020a) that does not account for dependencies performs as well as one that does (Khemakhem et al., 2020b).

Revised: We conclude that these preliminary results do not support the hypothesis that SlowVAE’s disentanglement is reliant upon the model assumption that the factors are independent, but do acknowledge that the empirical effect of statistical dependence in natural video warrants further exploration (Träuble et al., 2020;

Revised: Yang et al., 2020).

Original: H A DDITIONAL R ESULTS
Revised: G. T RANSITION P RIOR A BLATION

Revised: We consider an ablated model which minimizes a KL-divergence term between the posteriors at time-step t and time-step t − 1 .

Revised: This encourages the model to match the posteriors of both time points as closely as possible, and resembles a probabilistic variant of Slow Feature Analysis (Turner and

Revised: Sahani, 2007).

Revised: Speciﬁcally, we set p ( z t | z t − 1 ) =

Revised: q ( z t − 1 | x t − 1 ) , replacing the Laplace prior with the posterior of the previous time step.

Revised: This is equivalent to a Gaussian ( α = 2 ) transition prior, where the mean and variance are speciﬁed by the previous time step.

Revised: We ablate over the regularization parameter γ and provide results in Tables 14 and 15, although we note that we still use the same hyperparameter values for SlowVAE as in all other experiments.

Revised: As predicted by our theoretical result, α = 2 leads to entangled representations in aggregate across evaluated datasets and metrics, even when considering a spectrum of γ values, resulting in a drastic reduction in scores, particularly on dSprites and Natural Sprites.

Revised: H A DDITIONAL R ESULTS H. E XTENDED D ATA A NALYSIS

Original: [Figure]
Revised: [Table]

Original: We report the empirical estimates of Kurtosis in Table 6.
Revised: We report the empirical estimates of Kurtosis in Table 7.

Original: We report the log-likelihood scores for the ∆ area, ∆ x, ∆ y statistics in Tables7, 8,and 9,  respectively for a Normal, a Laplace and a generalized Laplace/Normal distribution.
Revised: We report the log-likelihood scores for the ∆ area, ∆ x, ∆ y statistics in Tables 8, 9, and 10, respectively for a Normal, a Laplace and a generalized Laplace/Normal distribution.

Original: For these distributions, we also report the ﬁt parameters for the ∆ area, ∆ x, ∆ y statistics in Tables10, 11,and 12,  respectively, where the shape parameter α of the generalized Laplacian is bolded.
Revised: For these distributions, we also report the ﬁt parameters for the ∆ area, ∆ x, ∆ y statistics in Tables 11, 12, and 13, respectively, where the shape parameter α of the generalized Laplacian is in bold face.

Original: As a higher likelihood indicates a better ﬁt, we can see further evidence that natural transitions are highly leptokurtic; a Laplace distribution ( α = 1 ) is a better ﬁt than a Gaussian ( α = 2 ), while the generalized Laplacian yields the highest likelihood consistently with α ≈ 0 .
Revised: As a higher likelihood indicates a better ﬁt, we can see further evidence that natural transitions are highly leptokurtic; a Laplace distribution ( α = 1 ) is a better ﬁt than a Gaussian ( α = 2 ), while the generalized Laplacian yields the highest likelihood consistently with α ≈ 0 .

Original: 5 for all measurements, as indicated in the main paper.
Revised: 5 for all measurements, as indicated in the main paper.

Original: H.
Revised: For the plots in Figs. 1 and 11, we set the standard deviation of each component to 1 and clipped the minimum ( − 5 ) and maximum ( 5 ) values. We note that while the marginal transitions appear sparse in metrics computed from the given object masks, since the object masks are projections of 3D objects onto 2D frames, there’s potential room for bias. This is a compelling question from a broader perspective of visual processing, but unfortunately,

Original: [Figure]
Revised: [Table] [Table]

Revised: the KITTI-MOTS masks (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) lack the associated depth data required to answer it.

Revised: Nonetheless, the natural scene statistics we compute are still relevant, given that most computer vision models and vision-based animals see the 3D world as projected onto their 2D receptor arrays.

Original:  LL D IS L IB R ESULTS
Revised: H. A LL D IS L IB R ESULTS

Original: We include results on all DisLib datasets, dSprites (Matthey et al., 2017), Cars3D (Reed et al., 2015),
Revised: We include results on all DisLib datasets, dSprites (Matthey et al., 2017), Cars3D (Reed et al., 2015),

Original: SmallNORB (LeCun et al., 2004), Shapes3D (Kim and Mnih, 2018), MPI3D (Gondal et al., 2019), in
Revised: SmallNORB (LeCun et al., 2004), Shapes3D (Kim and Mnih, 2018), MPI3D (Gondal et al., 2019), in

Original: Tables13, 14, 15, 16,and 17,  respectively.
Revised: Tables 16, 17, 18, 19, and 20, respectively.

Original: We report both median (a.d.) to compare to the previous median scores reported in (Locatello et al., 2020), as well as the the more common mean (s.d.) scores for future comparisons and straightforward statistical estimates of signiﬁcant differences between models. 
Revised: We report both median (a.d.) to compare to the previous median scores reported in (Locatello et al., 2020), as well as the the more common mean (s.d.) scores for future comparisons and straightforward statistical estimates of signiﬁcant differences between models. We also consider allowing for static transitions, which we denote with “NC”, e.g. “LAP-NC”, in the tabular results. H. KITTI M ASKS ∆ t A BLATION

Revised: As seen in the main text, considering image pairs separated further apart in time appears beneﬁcial.

Revised: Here we evaluate a wider range by taking frames which are further apart in a sequence.

Original: [Table]
Revised: max (∆ frames ) = N indicates that all pairs differ by at most N frames. We chose an upper bound of N , rather than sampling pairs with a ﬁxed separation, to account for the variable frame rates and sequence lengths in the original dataset (Milan et al., 2016), without introducing a confounding factor of varying dataset size. We report in Fig.

Revised: 10 how the max (∆ frames ) criterion corresponds to the mean time gap between image pairs (mean (∆ t ) ) in seconds.

Revised: For further details, we refer to

Revised: Appendix E.4.

Revised: In Fig.

Revised: 13 we visualize an ablation over mean (∆ t ) .

Revised: We ﬁnd that model performance increased initially with larger temporal separation between data points, then plateaued.

Revised: We also observe in Fig.

Revised: 14 that the measured factor marginals remain sparse, with α < 1 , for all tested settings of mean (∆ t ) .

Revised: While it is intuitive that increasing mean (∆ t ) leads to increased diversity, and thus learning signal, in the transitions, it is worth noting that since SlowVAE assumes α = 1 , an increase in α from increasing the temporal gap leads to a reduction in mismatch.

Revised: Our results on increasing the temporal difference within pairs of inputs links to recent work by

Revised: Oord et al.

Revised: (2018, Table 2), who show increased performance in representation learning for larger

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Table] [Table]
Revised: separation between positive samples in a contrastive objective function. It also links to recent work by Tschannen et al. (2019), who show that temporal separation between frame embeddings inﬂuences the representation that is learned from videos.

Original: [Table] [Table]
Revised: [Figure] [Table]

Original: [Table]
Revised: [Table]

Original: H. L ATENT S PACE V ISUALIZATIONS
Revised: H. L ATENT S PACE V ISUALIZATIONS

Original: We visualize differences in learned latent representations using image embedding in Figures 11- 24.
Revised: We visualize differences in learned latent representations using image embedding in Figures 15- 28.

Original: We show four different plots for each dataset considered and include all available models.
Revised: We show four different plots for each dataset considered and include all available models.

Original: Each ﬁgure corresponds to a different dataset.
Revised: Each ﬁgure corresponds to a different dataset.

Original: In Figures 11- 17 we display the mean correlation coefﬁcient matrix and the latent representations for each ground-truth, as described in the main text for Fig. 4.
Revised: In Figures 15- 21 we display the mean correlation coefﬁcient matrix and the latent representations for each ground-truth, as described in the main text for Fig. 5.

Original: The top row is the sorted absolute correlation coefﬁcient matrix between the latents (rows) and the ground truth generating factors (columns).
Revised: The top row is the sorted absolute correlation coefﬁcient matrix between the latents (rows) and the ground truth generating factors (columns).

Original: The latent dimensions are permuted such that the sum on the diagonal is maximal.
Revised: The latent dimensions are permuted such that the sum on the diagonal is maximal.

Original: This is achieved by an optimal, non-greedy matching process for each ground truth factor with its corresponding latent, as described in appendix D. As such, a more prevalent diagonal structure corresponds to a better mapping between the ground-truth factors and latent encoding.
Revised: This is achieved by an optimal, non-greedy matching process for each ground truth factor with its corresponding latent, as described in appendix D. As such, a more prevalent diagonal structure corresponds to a better mapping between the ground-truth factors and latent encoding.

Original: The middle set of plots are latent embeddings of random training data samples.
Revised: The middle set of plots are latent embeddings of random training data samples.

Original: The y-axis denotes the ground truth generating factor and the x-axis denotes the corresponding latent factor as matched according to the main diagonal of the correlation matrix.
Revised: The x-axis denotes the ground truth generating factor and the y-axis denotes the corresponding latent factor as matched according to the main diagonal of the correlation matrix.

Original: For each dataset, we further color-code the latents by a categorical variable as denoted in each ﬁgure.
Revised: For each dataset, we further color-code the latents by a categorical variable as denoted in each ﬁgure.

Original: The bottom set of plots show the ground truth encoding compared to the second best latent as opposed to the diagonally matched latent.
Revised: The bottom set of plots show the ground truth encoding compared to the second best latent as opposed to the diagonally matched latent.

Original: This plot can be used to judge how much the correspondence between latents is one-to-one or rather one-to-many.
Revised: This plot can be used to judge how much the correspondence between latents is one-to-one or rather one-to-many.

Original: To further investigate the latent representations, we show a scatter plot over the best and second best latents in ﬁgures 18-24.
Revised: To further investigate the latent representations, we show a scatter plot over the best and second best latents in ﬁgures 22-28.

Original: Here, the color-coding is matched by the ground truth factor denoted in each row.
Revised: Here, the color-coding is matched by the ground truth factor denoted in each row.

Original:  When comparing the correlation matrix with the corresponding scatter plots, one can see that embeddings with sinusoidal curves have low correlation, which illustrates a shortcoming of the metric.
Revised: [Table] [Table] When comparing the correlation matrix with the corresponding scatter plots, one can see that embeddings with sinusoidal curves have low correlation, which illustrates a shortcoming of the metric.

Original: Another limitation is that categorical variables which have no natural ordering have an[Table] [Table] order-dependent MCC score, indicating the permutation variance of MCC.
Revised: Another limitation is that categorical variables which have no natural ordering have an order-dependent MCC score, indicating the permutation variance of MCC.

Original: With SlowVAE, we can infer three different types of embeddings.
Revised: With SlowVAE, we can infer three different types of embeddings.

Original: First, we have simple ordered ground truth factors with noncircular boundary conditions.
Revised: First, we have simple ordered ground truth factors with noncircular boundary conditions.

Original: Here, SlowVAE models often show a clear one-to-one correspondence (e.g. Fig 18 scale, x-position and y-position; Fig 21 θ -rotation; Fig 22 Φ -rotation).
Revised: Here, SlowVAE models often show a clear one-to-one correspondence (e.g. Fig 22 scale, x-position and y-position; Fig 25 θ -rotation; Fig 26 Φ -rotation).

Original: Second, we observe circular embeddings due to boundary conditions for certain factors (e [Table] g. Fig 11, 18 3rd row;
Revised: Second, we observe circular embeddings due to boundary conditions for certain factors (e.g. Fig 15, 22 3rd row;

Original: Fig 12, 19 2nd row).
Revised: Fig 16, 23 2nd row).

Original: Note that not all datasets with orientations exhibit full rotations and thus do not have circular boundary conditions, e.g. smallNORB.
Revised: Note that not all datasets with orientations exhibit full rotations and thus do not have circular boundary conditions, e.g. smallNORB.

Original: Finally, we have categorical variables, where no order exists (e.g. Fig.
Revised: Finally, we have categorical variables, where no order exists (e.g. Fig.

Original: 12, 19 top row, Fig 13, 20 top row, Fig 14, 21 top row) resulting in separated but not necessarily ordered clusters.
Revised: 16, 23 top row, Fig 17, 24 top row, Fig 18, 25 top row) resulting in separated but not necessarily ordered clusters.

Original: [Table]
Revised: [Table]

Original: z ∗ ∆ z ∗ x ∗ z ∆ z x µ σ 5.

Original: MCC loss 0 π

Original: [Figure] 0 π 2 π [Figure] performance.
Revised: [Table]

Original: z ∗ ∆ z ∗ x ∗ z ∆ z 2.

Original: [Figure]
Revised: [Figure]

Original: z ∗ ∆ z ∗ x ∗ z ∆ z x µ σ 34.

Original: MCC loss

Original: [Figure] [Figure] [Table][Figure]
Revised: [Table]

Original: [Table][Figure] [Table] [Table]
Revised: [Table]

Original: [Figure]

Original: [Table][Figure] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: GT Factors
Revised: GT Factors

Original: L a t e n t s z
Revised: L a t e n t s z

Original: MCC = 74.8 86 141 5926 179 

Revised: MCC = 74.8 86 141 5926 179

Original: 45 9-VAE 0 

Revised: 45 9-VAE 0

Original: MCC = 76.0 

Revised: MCC = 76.0

Original: Ada-GVAE
Revised: Ada-GVAE

Original: MCC = 62.4 66 1530 6913 4812 503 263 3121 3334 1549 106 

Revised: MCC = 62.4 66 1530 6913 4812 503 263 3121 3334 1549 106

Original: PCL
Revised: PCL

Original: MCC = 90.6 95 210 817 550 179 1625 714 1110 711 159 

Revised: MCC = 90.6 95 210 817 550 179 1625 714 1110 711 159

Original: SlowVAE
Revised: SlowVAE

Original: [Table] [Figure]
Revised: [Table] [Figure]