Original: T OWARDS N ONLINEAR D ISENTANGLEMENT IN N ATURAL D ATA WITH T EMPORAL S PARSE C ODING
Revised: T OWARDS N ONLINEAR D ISENTANGLEMENT IN N ATURAL D ATA WITH T EMPORAL S PARSE C ODING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Disentangling the underlying generative factors fromcomplex data has so far been limited to carefully constructed scenarios.
Revised: Disentangling the underlying generative factors from data has so far been limited to carefully constructed scenarios.

Original: We propose a path towards natural data by ﬁrst showing that the statistics of natural data provide enough structure to enable disentanglement, both theoretically and empirically.
Revised: We propose a path towards natural data by ﬁrst showing that the statistics of natural data provide enough structure to enable disentanglement, both theoretically and empirically.

Original: Speciﬁcally, we provide evidence that objects in natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution.
Revised: Speciﬁcally, we provide evidence that objects in natural movies undergo transitions that are typically small in magnitude with occasional large jumps, which is characteristic of a temporally sparse distribution.

Original: To address this ﬁnding we provide a novel proof that relies on a sparse prior on temporally adjacent observations to recover the true latent variables up to permutations and sign ﬂips, providing a stronger result than previous work.
Revised: Leveraging this ﬁnding we provide a novel proof that relies on a sparse prior on temporally adjacent observations to recover the true latent variables up to permutations and sign ﬂips, providing a stronger result than previous work.

Original: We show that equipping practical estimation methods with our prior often surpasses the current state-of-the-art on several established benchmark datasets without any impractical assumptions, such as knowledge of the number of changing generative factors.
Revised: We show that equipping practical estimation methods with our prior often surpasses the current state-of-the-art on several established benchmark datasets without any impractical assumptions, such as knowledge of the number of changing generative factors.

Original: Furthermore, we contribute two new benchmarks, 
Revised: Furthermore, we contribute two new benchmarks, Natural

Original: Natural Sprites and KITTI Masks, which integrate the measured natural dynamics to enable disentanglement evaluation with more realistic datasets.
Revised: Sprites and KITTI Masks, which integrate the measured natural dynamics to enable disentanglement evaluation with more realistic datasets.

Original: Weleverage these benchmarks to test our theory, demonstrating improved performance.
Revised: We test our theory on these benchmarks and demonstrate improved performance.

Original: We also identify non-obvious challenges for current methods in scaling to more natural domains.
Revised: We also identify non-obvious challenges for current methods in scaling to more natural domains.

Original: Taken together our work addresses key issues in disentanglement research for moving towards more natural settings.
Revised: Taken together our work addresses key issues in disentanglement research for moving towards more natural settings

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural scene understanding can be achieved by decomposing the signal into its underlying factors of variation.
Revised: Natural scene understanding can be achieved by decomposing the signal into its underlying factors of variation.

Original: An intuitive approach for this problem assumes that a visual representation of the world can be constructed via a generative process that receives factors as input and produces natural signals as output (Bengio et al., 2013).
Revised: An intuitive approach for this problem assumes that a visual representation of the world can be constructed via a generative process that receives factors as input and produces natural signals as output (Bengio et al., 2013).

Original: This assumption is justiﬁed by the fact that our world is composed of distinct entities that can vary independently, but with regularity imposed by physics.
Revised: This analogy is justiﬁed by the fact that our world is composed of distinct entities that can vary independently, but with regularity imposed by physics.

Original: What makes this approach appealing is that it formalizes representation learning by directly comparing representations to underlying ground-truth state, as opposed to the indirect evaluation of benchmarking against heuristic downstream tasks (e.g. object recognition).
Revised: What makes the approach appealing is that it formalizes representation learning by directly comparing representations to underlying ground-truth states, as opposed to the indirect evaluation of benchmarking against heuristic downstream tasks (e.g. object recognition).

Original: However, the core issue with this approach is non-identiﬁability , which means a set of possible solutions may all appear equally valid to the model, while only one identiﬁes the true generative factors.
Revised: However, the core issue with this approach is non-identiﬁability , which means a set of possible solutions may all appear equally valid to the model, while only one identiﬁes the true generative factors.

Original: Our work is motivated by the question of whether the statistics of natural data will allow for the formulation of an identiﬁable model.
Revised: Our work is motivated by the question of whether the statistics of natural data will allow for the formulation of an identiﬁable model.

Original: Our core observation that enables us to make progress in  addressing this question is that generative factors of natural data have sparse transitions .
Revised: Our core observation that enables us to make progress in slow addressing this question is that generative factors of natural data have sparse transitions .

Original: To estimate these generative factors, we compute statistics on measured transitions of area and position for object masks from large-scale, natural, unstructured videos.
Revised: To estimate these generative factors, we compute statistics on measured transitions of area and position for object masks from large-scale, natural, unstructured videos.

Original: Speciﬁcally, we extracted over 300, object segmentation mask transitions from YouTube-VOS (Xu et al., 2018;
Revised: Speciﬁcally, we extracted over 300, object segmentation mask transitions from YouTube-VOS (Xu et al., 2018;

Original: Yang et al., 2019)
Revised: Yang et al., 2019)

Original: and
Revised: and

Original: KITTI-MOTS (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) (discussed in detail in
Revised: KITTI-MOTS (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) (discussed in detail in

Original: Appendix H).
Revised: Appendix D).

Original: We ﬁt generalized Laplace distributions to the collected data (Eq. 2), which we indicate with red lines in Fig.
Revised: We ﬁt generalized Laplace distributions to the collected data (Eq. 2), which we indicate with orange lines in Fig.

Original: We see empirically that all marginal distributions of temporal transitions are highly sparse and  there exists complex dependencies between natural factors (e.g. motion typically affects both position and apparent size).
Revised: We see empirically that all marginal distributions of temporal transitions are highly sparse and that there exist complex dependencies between natural factors (e.g. motion typically affects both position and apparent size).

Original: In this study, we focus on the sparse marginals, which we believe constitutes an important advance that sets the stage for solving further issues and eventually applying the technology to real-world problems.
Revised: In this study, we focus on the sparse marginals, which we believe constitutes an important advance that sets the stage for solving further issues and eventually applying the technology to real-world problems.

Original: With this information at hand, we are able to provide a stronger proof for capturing the underlying generative factors of the data up to permutations and sign ﬂips that is not covered by previous work (Hyvärinen and Morioka, 2016; 2017; Khemakhem et al., 2020a).
Revised: With this information at hand, we are able to provide a stronger proof for capturing the underlying generative factors of the data up to permutations and sign ﬂips that is not covered by previous work (Hyvärinen and Morioka, 2016; 2017; Khemakhem et al., 2020a).

Original: Thus, we present the ﬁrst work, to the best of our knowledge, which proposes a theoretically grounded solution that covers the statistics observed in real videos.
Revised: Thus, we present the ﬁrst work, to the best of our knowledge, which proposes a theoretically grounded solution that covers the statistics observed in real videos.

Original: In total, our contributions are: With measurements from unstructured natural video annotations we provide evidence that natural generative factors undergo sparse changes across time.
Revised: Our contributions are: With measurements from unstructured natural video annotations we provide evidence that natural generative factors undergo sparse changes across time.

Original: We provide a proof of identiﬁability that relies on the observed sparse innovations to identify nonlinearly mixed sources up to a permutation and sign-ﬂip, which we then validate with practical estimation methods for empirical comparisons.
Revised: We provide a proof of identiﬁability that relies on the observed sparse innovations to identify nonlinearly mixed sources up to a permutation and sign-ﬂips, which we then validate with practical estimation methods for empirical comparisons.

Original: We leverage the natural scene information to create novel datasets where the latent transitions between frames follow natural statistics, providing a benchmark to evaluate the practical capabilities of models in uncovering the true latent generative factors in the presence of realistic dynamics.
Revised: We leverage the natural scene information to create novel datasets where the latent transitions between frames follow natural statistics. These datasets provide a benchmark to evaluate how well models can uncover the true latent generative factors in the presence of realistic dynamics.

Original: We demonstrate improved disentanglement over previous models on existing datasets and our contributed ones with quantitative metrics from both the disentanglement (Locatello et al., 2018) and the nonlinear ICA community (Hyvärinen and Morioka, 2016).
Revised: We demonstrate improved disentanglement over previous models on existing datasets and our contributed ones with quantitative metrics from both the disentanglement (Locatello et al., 2018) and the nonlinear ICA community (Hyvärinen and Morioka, 2016).

Original: We show via numerous visualization techniques that the learned representations for competing models have important differences, even when quantitative metrics suggest that they are performing equally well.
Revised: We show via numerous visualization techniques that the learned representations for competing models have important differences, even when quantitative metrics suggest that they are performing equally well.

Original:  R ELATED W ORK – D ISENTANGLEMENT AND N ONLINEAR ICA
Revised:  R ELATED W ORK – D ISENTANGLEMENT AND N ONLINEAR ICA

Original: Disentangled representation learning has its roots in blind source separation (Jutten and Herault, 1991) and shares goals with ﬁelds such as inverse graphics (Kulkarni et al., 2015;
Revised: Disentangled representation learning has its roots in blind source separation (Cardoso, 1989; Jutten and Herault, 1991) and shares goals with ﬁelds such as inverse graphics (Kulkarni et al., 2015;

Original: Yildirim et al., 2020) and developing models of invariant neural computation (Wiskott and Sejnowski, 2002; Hyvärinen and Hoyer, 2000)(see Bengio et al., 2013, for a review).
Revised: Yildirim et al., 2020; Barron and Malik, 2012) and developing models of invariant neural computation (Hyvärinen and Hoyer, 2000; Wiskott and Sejnowski, 2002; Sohl-Dickstein et al., 2010) (see Bengio et al., 2013, for a review).

Original: A disentangled representation would be valuable for a wide variety of machine learning applications, including sample efﬁciency for downstream tasks (Locatello et al., 2018; Gao et al., 2019), fairness (Locatello et al., 2019; Creager et al., 2019) and interpretability (Bengio et al., 2013; Higgins et al., 2017; Adel et al., 2018).
Revised: A disentangled representation would be valuable for a wide variety of machine learning applications, including sample efﬁciency for downstream tasks (Locatello et al., 2018; Gao et al., 2019), fairness (Locatello et al., 2019; Creager et al., 2019) and interpretability (Bengio et al., 2013; Higgins et al., 2017; Adel et al., 2018).

Original: Since there is no agreed upon deﬁnition of disentanglement in the literature, we adopt two common measurable criteria: i) each encoding element represents a single generative factor and ii) the values of generative factors are trivially decodable from the encoding (Ridgeway and Mozer, 2018;
Revised: Since there is no agreed upon deﬁnition of disentanglement in the literature, we adopt two common measurable criteria: i) each encoding element represents a single generative factor and ii) the values of generative factors are trivially decodable from the encoding (Ridgeway and Mozer, 2018;

Original: Eastwood and Williams, 2018).
Revised: Eastwood and Williams, 2018).

Original: Uncovering the underlying factors of variation has been a long-standing goal in independent component analysis (ICA) (Jutten and Herault, 1991; Bell and Sejnowski, 1995; Hyvärinen and Pajunen, 1999; Barron and Malik, 2012; Sohl-Dickstein et al., 2010). ICA (Comon, 1994) provides an identiﬁable solution for disentangling data mixed via an invertible linear generator receiving at most one Gaussian factor as input.
Revised: Uncovering the underlying factors of variation has been a long-standing goal in independent component analysis (ICA) (Comon, 1994; Bell and Sejnowski, 1995), which provides an identiﬁable solution for disentangling data mixed via an invertible linear generator receiving at most one Gaussian factor as input.

Original: Recently, unsupervised approaches for nonlinear generators have largely been based on Variational Autoencoders (VAEs) (Kingma and Welling, 2013) and have assumed that the data is independent and identically distributed ( i.i.d. )
Revised: Recent unsupervised approaches for nonlinear generators have largely been based on Variational Autoencoders (VAEs) (Kingma and Welling, 2013) and have assumed that the data is independent and identically distributed ( i.i.d. )

Original: (Locatello et al., 2018) even though nonlinear methods that make this i.i.d.
Revised: (Locatello et al., 2018), even though nonlinear methods that make this i.i.d.

Original: assumption have been proven to be non-identiﬁable (Hyvärinen and Pajunen,1999;
Revised: assumption have been proven to be non-identiﬁable (Hyvärinen and Pajunen,

Original:  Locatello et al., 2018).
Revised: ; Locatello et al., 2018).

Original: Nonetheless, the bottom-up approach of starting with a nonlinear generator that produces well-controlled data has led to considerable achievements in understanding nonlinear disentanglement in VAEs (Higgins et al., 2017; Burgess et al., 2018; Rolinek et al., 2019; Chen et al., 2018), consolidating ideas from neural computation and machine learning (Khemakhem et al., 2020a), and seeking a principled deﬁnition of disentanglement (Higgins et al., 2018;Ridgeway, 2016; Eastwood and Williams, 2018; Hyvärinen and Pajunen, 1999).
Revised: Nonetheless, the bottom-up approach of starting with a nonlinear generator that produces well-controlled data has led to considerable achievements in understanding nonlinear disentanglement in VAEs (Higgins et al., 2017; Burgess et al., 2018; Rolinek et al., 2019; Chen et al., 2018), consolidating ideas from neural computation and machine learning (Khemakhem et al., 2020a), and seeking a principled deﬁnition of disentanglement (Ridgeway, 2016; Higgins et al., 2018; Eastwood and Williams, 2018).

Original: Recently, Hyvärinen and colleagues (Hyvärinen and Morioka, 2016; 2017; Hyvärinen et al., 2018; Khemakhem et al., 2020a) showed that a solution to identiﬁable nonlinear ICA can be found by assuming that generative factors are conditioned on an additional observed variable, such as past states or the time index itself.
Revised: Recently, Hyvärinen and colleagues (Hyvärinen and Morioka, 2016; 2017; Hyvärinen et al., 2018) showed that a solution to identiﬁable nonlinear ICA can be found by assuming that generative factors are conditioned on an additional observed variable, such as past states or the time index itself.

Original: This contribution was generalized by Khemakhem et al.
Revised: This contribution was generalized by Khemakhem et al.

Original: (2020a) past the nonlinear ICA domain to any consistent parameter estimation method for deep latent-variable models, including the VAE framework.
Revised: (2020a) past the nonlinear ICA domain to any consistent parameter estimation method for deep latent-variable models, including the VAE framework.

Original: However, the theoretical assumptions underlying this branch of work do not account for the sparse transitions we observe in the statistics of natural scenes, which we discuss in further detail in appendix G.1.1.
Revised: However, the theoretical assumptions underlying this branch of work do not account for the sparse transitions we observe in the statistics of natural scenes, which we discuss in further detail in appendix F.1.1.

Original: Another branch of work requires some form of supervision  (Szabó et al., 2017; Shu et al., 2019; Locatello et al., 2020), namely regarding the number of factors changed. Locatello et al.
Revised: Another branch of work requires some form of supervision to demonstrate disentanglement (Szabó et al., 2017; Shu et al., 2019; Locatello et al., 2020).

Original: (2020) proposed an approach to estimate this required knowledge at runtime, however their assumptions underlying identiﬁability may not be satisﬁed in practice.

Original: We select two of the above approaches, that are both different in their formulation and state-of-the-art in their respective empirical settings, Hyvärinen and Morioka (2017); Locatello et al.
Revised: We select two of the above approaches, that are both different in their formulation and state-of-the-art in their respective empirical settings, Hyvärinen and Morioka (2017) and Locatello et al.

Original: (2020), for our experiments below.
Revised: (2020), for our experiments below.

Original: The motivation of our method and dataset contributions is to address these limitations  and to enable unsupervised disentanglement learning in more naturalistic scenarios 1 .
Revised: The motivation of our method and dataset contributions is to address the limitations of previous approaches and to enable unsupervised disentanglement learning in more naturalistic scenarios.

Original: The fact that physical processes bind generative factors in temporally adjacent natural video segments has been thoroughly explored for learning in neural networks (Hinton, 1990; Földiák, 1991; Mitchison, 1991; Wiskott and Sejnowski, 2002;Hyvärinen et al., 2003; Denton and Birodkar, 2017).
Revised: The fact that physical processes bind generative factors in temporally adjacent natural video segments has been thoroughly explored for learning in neural networks (Hinton, 1990; Földiák, 1991; Mitchison, 1991; Wiskott and Sejnowski, 2002; Denton and Birodkar, 2017).

Original: We propose a method that uses time information in the form of an L 1 - sparse temporal prior, which is motivated by the natural scene measurements presented above as well as by previous work (Simoncelli and Olshausen, 2001; Olshausen, 2003; Hyvärinen et al., 2003; Cadieu and Olshausen, 2012).
Revised: We propose a method that uses time information in the form of an L 1 - sparse temporal prior, which is motivated by the natural scene measurements presented above as well as by previous work (Simoncelli and Olshausen, 2001; Olshausen, 2003; Hyvärinen et al., 2003; Cadieu and Olshausen, 2012).

Original: Such a prior would intuitively allow for sharp changes in some latent factors, while most other factors remain unchanged between adjacent time-points.
Revised: Such a prior would intuitively allow for sharp changes in some latent factors, while most other factors remain unchanged between adjacent time-points.

Original: Almost all similar methods are variants of slow feature analysis (SFA, Wiskott and Sejnowski, 2002), which measure slowness in terms of the Euclidean (i.e. 
Revised: Almost all similar methods are variants of slow feature analysis (SFA, Wiskott and Sejnowski, 2002), which measure slowness in terms of the Euclidean (i.e. L 2 , or log

Original: L 2 , or log Gaussian) distance between temporally adjacent encodings.
Revised: Gaussian) distance between temporally adjacent encodings.

Original: Related to our approach, a probabilistic interpretation of SFA has been previously proposed (Turner and Sahani, 2007), as well as extensions to variational inference (Grathwohl and Wilson, 2016).
Revised: Related to our approach, a probabilistic interpretation of SFA has been previously proposed (Turner and Sahani, 2007), as well as extensions to variational inference (Grathwohl and Wilson, 2016).

Original: Additionally, Hashimoto (2003) suggested that a sparse (Cauchy) slowness prior improves correspondence to biological complex cells over the L 2 slowness prior in a two-layer model.
Revised: Additionally, Hashimoto (2003) suggested that a sparse (Cauchy) slowness prior improves correspondence to biological complex cells over the L 2 slowness prior in a two-layer model.

Original: However, to the best of our knowledge, an L 1 temporal prior has previously only been used in deep auto-encoder frameworks when applied to semi-supervised tasks (Mobahi et al., 2009; Zou et al., 2012), and was mentioned in Cadieu and Olshausen (2012), who used an L 2 prior, but claimed that an L 1 prior performed similarly on their task.
Revised: However, to the best of our knowledge, an L 1 temporal prior has previously only been used in deep auto-encoder frameworks when applied to semi-supervised tasks (Mobahi et al., 2009; Zou et al., 2012), and was mentioned in Cadieu and Olshausen (2012), who used an L 2 prior, but claimed that an L 1 prior performed similarly on their task.

Original: Similar to Hyvärinen et al.
Revised: Similar to Hyvärinen et al.

Original: (Hyvärinen and Morioka, 2016; Hyvärinen et al., 2018), we only assume that the latent factors are temporally dependent, thus avoiding assuming knowledge of the number of factors where the two observations differ (Shu et al., 2019;
Revised: (Hyvärinen and Morioka, 2016; Hyvärinen et al., 2018), we only assume that the latent factors are temporally dependent, thus avoiding assuming knowledge of the number of factors where the two observations differ (Shu et al., 2019;

Original: Locatello et al., 2020).
Revised: Locatello et al., 2020).

Original: Most of the standard datasets for disentanglement (dSprites (Matthey et al., 2017), Cars3D (Reed et al., 2015), SmallNORB (LeCun et al., 2004), Shapes3D (Kim and Mnih, 2018), MPI3D (Gondal et al., 2019)) have been compiled into a disentanglement library (DisLib) by Locatello et al.
Revised: Most of the standard datasets for disentanglement (dSprites (Matthey et al., 2017), Cars3D (Reed et al., 2015), SmallNORB (LeCun et al., 2004), Shapes3D (Kim and Mnih, 2018), MPI3D (Gondal et al., 2019)) have been compiled into a disentanglement library (DisLib) by Locatello et al.

Original: However, all of the DisLib datasets are limited in that the data generating process is independent and identically distributed ( i.i.d. ) and all generative factors are assumed to be discrete.
Revised: However, all of the DisLib datasets are limited in that the data generating process is independent and identically distributed ( i.i.d. ) and all generative factors are assumed to be discrete.

Original: In a follow-up study, Locatello et al.
Revised: In a follow-up study, Locatello et al.

Original: (2020) proposed combining pairs of images such that only k factors change, as this matches their modeling assumptions required to prove identiﬁability.
Revised: (2020) proposed combining pairs of images such that only k factors change, as this matches their modeling assumptions required to prove identiﬁability.

Original: Here, k ∈ U{ 1 , D − 1 } and
Revised: Here, k ∈ U{ 1 , D − 1 } and

Original: D denotes the number of ground-truth factors, which are then sampled uniformly.
Revised: D denotes the number of ground-truth factors, which are then sampled uniformly.

Original: We additionally use the measurements from Fig.
Revised: We additionally use the measurements from Fig.

Original: 1 to construct datasets for evaluating disentanglement that have time transitions which directly correspond to natural dynamics.
Revised: 1 to construct datasets for evaluating disentanglement that have time transitions which directly correspond to natural dynamics.

Original:  T HEORY 3. G ENERATIVE M ODEL
Revised:  T HEORY 3. G ENERATIVE M ODEL

Original: We have provided evidence to support the hypothesis that generative factors of natural videos have sparse temporal transitions (see Fig. 1).
Revised: We have provided evidence to support the hypothesis that generative factors of natural videos have sparse temporal transitions (see Fig. 1).

Original: To model this process, we assume temporally adjacent input pairs ( x t − 1 , x t ) coming from a nonlinear generator that maps factors to images x = g ( z ) , wherepairs of generative factors form a Markov chain:
Revised: To model this process, we assume temporally adjacent input pairs ( x t − 1 , x t ) coming from a nonlinear generator that maps factors to images x = g ( z ) , where generative factors are dependent over time:

Original: [Equation]
Revised: [Equation]

Original: Assume the observed data ( x t , x t − 1 ) comes from the following generative process, where the latent factors are assumed to be independent (cf.
Revised: Assume the observed data ( x t , x t − 1 ) comes from the following generative process, where different latent factors are assumed to be independent (cf.

Original: Appendix G.2):
Revised: Appendix F.2):

Original: [Equation]
Revised: [Equation]

Original: where λ is the distribution rate, p ( z t − 1 ) is a factorized Gaussian prior N ( 0 , I ) (as in Kingma and
Revised: where λ is the distribution rate, p ( z t − 1 ) is a factorized Gaussian prior N ( 0 , I ) (as in Kingma and

Original: Welling, 2013) and p ( z t | z t − 1 ) is a factorized generalized Laplace distribution (Subbotin, 1923) with shape parameter α , which determines the shape and especially the kurtosis of the function 2 .
Revised: Welling, 2013) and p ( z t | z t − 1 ) is a factorized generalized Laplace distribution (Subbotin, 1923) with shape parameter α , which determines the shape and especially the kurtosis of the function.

Original: Intuitively, smaller α implies larger kurtosis and sparser temporal transitions of the generative factors (special cases are Gaussian, α = 2 , and Laplacian, α = 1 ).
Revised: Intuitively, smaller α implies larger kurtosis and sparser temporal transitions of the generative factors (special cases are Gaussian, α = 2 , and Laplacian, α = 1 ).

Original: Critically, for our proof we assume α < 2 to ensure that temporal transitions are sparse.
Revised: Critically, for our proof we assume α < 2 to ensure that temporal transitions are sparse.

Original: The novelty of our approach lies in our explicit modeling of sparse transitions that cover the statistics of natural data, which results in a stronger identiﬁability proof than previously achieved (see Appendix G.1.1 for a more detailed comparison with Hyvärinen and Morioka, 2017; Khemakhem et al., 2020a).
Revised: The novelty of our approach lies in our explicit modeling of sparse transitions that cover the statistics of natural data, which results in a stronger identiﬁability proof than previously achieved (see Appendix F.1.1 for a more detailed comparison with Hyvärinen and Morioka, 2017; Khemakhem et al., 2020a).

Original: 3. I DENTIFIABILITY P ROOF
Revised: 3. I DENTIFIABILITY P ROOF

Original: Theorem 1 For a ground-truth ( g ∗ , λ ∗ , α ∗ ) and a learned ( g, λ, α ) model as deﬁned in Eq.
Revised: Theorem 1 For a ground-truth ( g ∗ , λ ∗ , α ∗ ) and a learned ( g, λ, α ) model as deﬁned in Eq.

Original: (2), if the functions g ∗ and g are injective and differentiable almost everywhere, λ ∗ =
Revised: (2), if the functions g ∗ and g are injective and differentiable almost everywhere, λ ∗ =

Original: λ , α ∗ =
Revised: λ , α ∗ =

Original: α < 2 (i.e. there is no model misspeciﬁcation) and the distributions of pairs of images generated from the priors z ∗ ∼ p ∗ ( z ) and
Revised: α < 2 (i.e. there is no model misspeciﬁcation) and the distributions of pairs of images generated from the priors z ∗ ∼ p ∗ ( z ) and

Original: z ∼ p ( z ) generated as ( g ∗ ( z ∗ t − 1 ) , g ∗ ( z ∗ t )) and ( g ( z t − 1 ) , g ( z t )) , respectively, are matched almost everywhere, then g = g ∗ ◦ σ , where σ is  a permutation with sign ﬂipping.
Revised: z ∼ p ( z ) generated as ( g ∗ ( z ∗ t − 1 ) , g ∗ ( z ∗ t )) and ( g ( z t − 1 ) , g ( z t )) , respectively, are matched almost everywhere, then g = g ∗ ◦ σ , where σ is composed of a permutation and sign ﬂips.

Original: The formal proof is provided in Appendix B.1.
Revised: The formal proof is provided in Appendix A.1.

Original: Similar to linear ICA, but in the temporal domain, we have to assume that the transitions of generative factors across time be non-Gaussian.
Revised: Similar to linear ICA, but in the temporal domain, we have to assume that the transitions of generative factors across time be non-Gaussian.

Original: Speciﬁcally, if the temporal changes of ground-truth factors are sparse, then the only generator consistent with the observations is the ground-truth one (up to a permutation and sign ﬂips).
Revised: Speciﬁcally, if the temporal changes of ground-truth factors are sparse, then the only generator consistent with the observations is the ground-truth one (up to a permutation and sign ﬂips).

Original: The main idea behind the proof is to represent g as g ∗ ◦ h and note that if h were not a permutation, then the distributions (( g ∗ ◦ h )( z t − 1 ) , ( g ∗ ◦ h )( z t )) and ( g ∗ ( z ∗ t − 1 ) , g ∗ ( z ∗ t )) would not match, due to the injectivity of g ∗ .
Revised: The main idea behind the proof is to represent g as g ∗ ◦ h and note that if h were not a permutation, then the distributions (( g ∗ ◦ h )( z t − 1 ) , ( g ∗ ◦ h )( z t )) and ( g ∗ ( z ∗ t − 1 ) , g ∗ ( z ∗ t )) would not match, due to the injectivity of g ∗ .

Original: Whether or not these distributions are the same is equivalent to whether or not the distributions of pairs ( z t − 1 , z t ) and ( h ( z t − 1 ) , h ( z t )) are the same.
Revised: Whether or not these distributions are the same is equivalent to whether or not the distributions of pairs ( z t − 1 , z t ) and ( h ( z t − 1 ) , h ( z t )) are the same.

Original: For these distributions to be the same, the function h must preserve the Gaussian marginal for the ﬁrst time step as well as the joint distribution, implying that it must preserve both the vector lengths and distances in the latent space.
Revised: For these distributions to be the same, the function h must preserve the Gaussian marginal for the ﬁrst time step as well as the joint distribution, implying that it must preserve both the vector lengths and distances in the latent space.

Original: As we argue in the extended proof, this can only be the case if h is a composition of permutations and sign ﬂips.
Revised: As we argue in the extended proof, this can only be the case if h is a composition of permutations and sign ﬂips.

Original: Intuition Fig.
Revised: Intuition Fig.

Original: 2 illustrates, by contradiction, why the model deﬁned in Eq.
Revised: 2 illustrates, by contradiction, why the model deﬁned in Eq.

Original: (2) is identiﬁable.
Revised: (2) is identiﬁable.

Original: We consider temporal pairs of latents represented by connected points.
Revised: We consider temporal pairs of latents represented by connected points.

Original: A sparse transition prior encourages axis-alignment, as can be seen from the Laplace transition prior in the third image of Fig. 3.
Revised: A sparse transition prior encourages axis-alignment, as can be seen from the Laplace transition prior in the third image of Fig. 3.

Original: This results in lines that are parallel with the axes in both the ground truth (left, blue, z ∗ ) and learned model (right, red, z ).
Revised: This results in lines that are parallel with the axes in both the ground truth (left, blue, z ∗ ) and learned model (right, red, z ).

Original: In this example, z ∗ 0 corresponds to horizontal position, while z ∗ 1 corresponds to vertical position.
Revised: In this example, z ∗ 0 corresponds to horizontal position, while z ∗ 1 corresponds to vertical position.

Original: The learned model must satisfy two criteria: (1) the latent factors should match the sparse prior (axis-aligned) and (2) the generated image pairs should match the ground-truth image pairs.
Revised: The learned model must satisfy two criteria: (1) the latent factors should match the sparse prior (axis-aligned) and (2) the generated image pairs should match the ground-truth image pairs.

Original: If the learned latent factors were mismatched, for example by rotation, then the image pair distributions would not be matched.
Revised: If the learned latent factors were mismatched, for example by rotation, then the image pair distributions would not be matched.

Original: In this example, the ground truth model would produce image pairs with typically vertical or horizontal transitions, while the learned model pairs result in mostly diagonal transitions.
Revised: In this example, the ground truth model would produce image pairs with typically vertical or horizontal transitions, while the learned model pairs result in mostly diagonal transitions.

Original: Thus, the learned model cannot satisfy both criteria without aligning the latent axes with the ground-truth axes.
Revised: Thus, the learned model cannot satisfy both criteria without aligning the latent axes with the ground-truth axes.

Original: 3. S LOW V ARIATIONAL A UTOENCODER
Revised: 3. S LOW V ARIATIONAL A UTOENCODER

Original: In order to validate our proof, we must choose a probabilistic latent variable model for estimating the data density.
Revised: In order to validate our proof, we must choose a probabilistic latent variable model for estimating the data density.

Original: We chose to build upon the framework of VAEs because of their efﬁciency in estimating a variational approximation to the ground truth posterior of a deep latent variable model (Kingma and Welling, 2013).
Revised: We chose to build upon the framework of VAEs because of their efﬁciency in estimating a variational approximation to the ground truth posterior of a deep latent variable model (Kingma and Welling, 2013).

Original: We will refer to this model as SlowVAE .
Revised: We will refer to this model as SlowVAE .

Original: In Appendix C wealso notetheir shortcomings  and test  alternative ﬂow-based approaches.
Revised: In Appendix B we note shortcomings of such an approach and test an alternative ﬂow-based model.

Original: The standard VAE objective assumes i.i.d.
Revised: The standard VAE objective assumes i.i.d.

Original: data and a standard normal prior with diagonal covariance on the learned latent representations z ∼ N ( 0 , I ) .
Revised: data and a standard normal prior with diagonal covariance on the learned latent representations z ∼ N ( 0 , I ) .

Original: To extend this to sequences, we assume the same functional form for our model prior as in Eq.
Revised: To extend this to sequences, we assume the same functional form for our model prior as in Eq.

Original: (1) and Eq.
Revised: (1) and Eq.

Original: The posterior of our model is independent across time steps.
Revised: The posterior of our model is independent across time steps.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: where µ i ( x ) and σ 2 i ( x ) are the input-dependent mean and variance of our model’s posterior.
Revised: where µ i ( x ) and σ 2 i ( x ) are the input-dependent mean and variance of our model’s posterior.

Original: We visualize this combination of priors and posteriors in Fig.
Revised: We visualize this combination of priors and posteriors in Fig.

Original: For a given pair of inputs ( x t , x t − 1 ), the full ELBO (which we derive in Appendix B.2) can be written as
Revised: For a given pair of inputs ( x t , x t − 1 ), the full evidence lower bound (ELBO, which we derive in Appendix A.2) can be written as

Original: [Equation]
Revised: [Equation]

Original:  where γ is a regularization term for the sparsity prior, analogous to β in β -VAEs (Higgins et al., 2017) (technically, Eq. 4 is only an ELBO with γ ≤ 1 ).
Revised: [Equation] where γ is a regularization term for the sparsity prior, analogous to β in β -VAEs (Higgins et al., 2017) (technically, Eq. 4 is only an ELBO with γ ≤ 1 ).

Original: The ﬁrst term on the right-hand side is the log-likelihood (i.e. the negative reconstruction error, with p ( x t , x t − 1 | z t , z t − 1 ) parameterized by the decoder of the VAE), the second term is the KL to a normal prior as in the standard VAE and the last term is an expectation of the KL between the posterior at time step t and the conditional prior p ( z t | z t − 1 ) .
Revised: The ﬁrst term on the right-hand side is the log-likelihood (i.e. the negative reconstruction error, with p ( x t , x t − 1 | z t , z t − 1 ) parameterized by the decoder of the VAE), the second term is the KL to a normal prior as in the standard VAE and the last term is an expectation of the KL between the posterior at time step t and the conditional prior p ( z t | z t − 1 ) .

Original: The expectation in the last term is taken over samples from the posterior at the previous time step q ( z t − 1 | x t − 1 ) .
Revised: The expectation in the last term is taken over samples from the posterior at the previous time step q ( z t − 1 | x t − 1 ) .

Original: We observed empirically that taking the mean, µ ( x t − 1 ) , as a single sample produces good results, analogous to the log-likelihood that is typically evaluated at a single sample from the posterior (see (Blei et al., 2017) for context). 
Revised: We observed empirically that taking the mean, µ ( x t − 1 ) , as a single sample produces good results, analogous to the log-likelihood that is typically evaluated at a single sample from the posterior (see Blei et al. (2017) for context). of the posterior for t − 1 .

Original: In practice, we need to choose α , λ , and γ .
Revised: In practice, we need to choose α , λ , and γ .

Original: For the latter two, we can perform a random search for hyperparameters, as we discuss below.
Revised: For the latter two, we can perform a random search for hyperparameters, as we discuss below.

Original: For the former, any α < 2 would break the general rotation symmetry by having an optimum for axis-aligned representations, which theorem 1 includes as a requirement for identiﬁability.
Revised: For the former, any α < 2 would break the general rotation symmetry by having an optimum for axis-aligned representations, which theorem 1 includes as a requirement for identiﬁability.

Original: As can be seen in Figs.
Revised: As can be seen in Figs.

Original: 1 and 11, α ≈ 0 .provides the best ﬁt to the ground-truth marginals.
Revised: 1 and 11, α ≈ 0 .provides the best ﬁt to the ground-truth marginals.

Original: However, we used α = 1 as a parsimonious choice for SlowVAE, since the Laplace is a well-understood distribution that allows us to derive a simple closedform solution for the ELBO in Eq. 4, which we derive in Appendix B.2.
Revised: However, we used α = 1 as a parsimonious choice for SlowVAE, since the Laplace is a well-understood distribution that allows us to derive a simple closedform solution for the ELBO in Eq. 4, which we derive in Appendix A.2.

Original: 3. T OWARDS AN A PPROXIMATE T HEORY OF D ISENTANGLEMENT
Revised: 3. T OWARDS AN A PPROXIMATE T HEORY OF D ISENTANGLEMENT

Original: A number of our theoretical assumptions are violated in practice: After non-convex optimization, on a ﬁnite data sample, the distributions p ( x t , x t − 1 ) and p ∗ ( x t , x t − 1 ) are probably not perfectly matched.
Revised: A number of our theoretical assumptions are violated in practice: After non-convex optimization, on a ﬁnite data sample, the distributions p ( x t , x t − 1 ) and p ∗ ( x t , x t − 1 ) are probably not perfectly matched.

Original: In addition, the model assumptions on p ( z t , z t − 1 ) likely do not fully match the distribution of the ground truth factors.
Revised: In addition, the model assumptions on p ( z t , z t − 1 ) likely do not fully match the distribution of the ground truth factors.

Original: For example, the model may be misspeciﬁed such that α (cid:54) =
Revised: For example, the model may be misspeciﬁed such that α (cid:54) =

Original: α ∗ or λ (cid:54) = λ ∗ , or the chosen family of distributions may be incorrect altogether.
Revised: α ∗ or λ (cid:54) = λ ∗ , or the chosen family of distributions may be incorrect altogether.

Original: In the following section we will present results on several datasets where the marginal distributions p ( z t − 1 ) are drawn from a Uniform (not Normal) distribution, and some of them are over unordered sets (categories) or bounded periodic spaces (rotation).
Revised: In the following section we will present results on several datasets where the marginal distributions p ( z t − 1 ) are drawn from a Uniform (not Normal) distribution, and some of them are over unordered sets (categories) or bounded periodic spaces (rotation).

Original: Also, in practice the model latent space is usually chosen to have more dimensions than the ground truth generative model.
Revised: Also, in practice the model latent space is usually chosen to have more dimensions than the ground truth generative model.

Original: On real data, factors of variation may be dependent (cf. Träuble et al., 2020; Yang et al., 2020).
Revised: On real data, factors of variation may be dependent (Träuble et al., 2020; Yang et al., 2020).

Original: We show this is the case on YouTube-VOS and KITTI-MOTS in Appendix H and we provide evidence that breaking these dependencies has no clear consequence on disentanglement in Appendix G.2.
Revised: We show this is the case on YouTube-VOS and KITTI-MOTS in Appendix G and we provide evidence that breaking these dependencies has no clear consequence on disentanglement in Appendix F.2.

Revised: A more formal treatment of dependence is done by Khemakhem et al.

Original:  Furthermore, the hypothesis class G of learnable functions in the VAE architecture may not contain the invertible ground truth generator g ∗ / ∈ G .
Revised: (2020b) who relax the independence assumption of ICA to Independently Modulated Components Analysis (IMCA) and introduce a family of conditional energy-based models that are identiﬁable up to simple transformations. Furthermore, the hypothesis class G of learnable functions in the VAE architecture may not contain the invertible ground truth generator g ∗ / ∈ G , if it exists at all (e.g. occlusions may already lead to non-invertibility).

Original: Despite these violations, we consider it a strength of our method that the practical implementation still achieves improved disentanglement over previous approaches.
Revised: Despite these violations, we consider it a strength of our method that the practical implementation still achieves improved disentanglement over previous approaches.

Original: However, we note understanding the impact of these violations as an important focus area for continued progress towards developing a practical yet theoretically supported method for disentanglement on natural scenes.
Revised: However, we note understanding the impact of these violations as an important focus area for continued progress towards developing a practical yet theoretically supported method for disentanglement on natural scenes.

Original:  D ATASETS WITH N ATURAL T RANSITIONS
Revised:  D ATASETS WITH N ATURAL T RANSITIONS

Original: While the standard datasets compiled by DisLib are an important step towards real-world applications, they still assume the data is i.i.d. , which is decidedly untrue for natural data.
Revised: While the standard datasets compiled by DisLib are an important step towards real-world applications, they still assume the data is i.i.d. .

Original: As described in section 2, Locatello et al.
Revised: As described in section 2, Locatello et al.

Original: (2020) proposed uniformly sampling the number of factors to be changed, k = Rnd, and changing said factors by uniformly sampling over the possible set of values.
Revised: (2020) proposed uniformly sampling the number of factors to be changed, k = Rnd, and changing said factors by uniformly sampling over the possible set of values.

Original: What we refer to as “UNI” is a dataset variant modeled after the described scheme (Locatello et al., 2020) (further details in Appendix F.2).
Revised: What we refer to as “UNI” is a dataset variant modeled after the described scheme (Locatello et al., 2020) (further details in Appendix D).

Original: Considering our natural data analysis presented in Figure 1, such transitions are certainly unnatural.
Revised: Considering our natural data analysis presented in Figure 1, such transitions are certainly unnatural.

Original: Given the current state of evaluation, we provide a set of incrementally more natural datasets which are otherwise comparable to existing work.
Revised: Given the current state of evaluation, we provide a set of incrementally more natural datasets which are otherwise comparable to existing work.

Original: We propose that said datasets should be included in the standard benchmark suite to provide a step towards disentanglement in natural data.
Revised: We propose that said datasets should be included in the standard benchmark suite to provide a step towards disentanglement in natural data.

Original: (1) Laplace Transitions (LAP) is a procedure for constructing image pairs from DisLib datasets by sampling from a sparse conditional distribution.
Revised: (1) Laplace Transitions (LAP) is a procedure for constructing image pairs from DisLib datasets by sampling from a sparse conditional distribution.

Original: For each ground-truth factor, the ﬁrst value in the pair is chosen i.i.d.
Revised: For each ground-truth factor, the ﬁrst value in the pair is chosen i.i.d.

Original: from the dataset and the second is chosen by weighting nearby factor values using Laplace distributed probabilities.
Revised: from the dataset and the second is chosen by weighting nearby factor values using Laplace distributed probabilities.

Original: LAP is a step towards natural data that closely resembles previous extensions of DisLib datasets to the time domain, but in a way that matches the marginal distribution of natural transitions (see Appendix E.1 for more details).
Revised: LAP is a step towards natural data that closely resembles previous extensions of DisLib datasets to the time domain, but in a way that matches the marginal distribution of natural transitions (see Appendix D.2 for more details).

Original: (2) Natural Sprites consists of pairs of rendered sprite images with generative factors sampled from real YouTube-VOS transitions.
Revised: (2) Natural Sprites consists of pairs of rendered sprite images with generative factors sampled from real YouTube-VOS transitions.

Original: For a given image pair, the position and scale of the sprites are set using measured values from adjacent time points in YouTube-VOS.
Revised: For a given image pair, the position and scale of the sprites are set using measured values from adjacent time points in YouTube-VOS.

Original: The sprite shapes and orientations are simple, like dSprites, and are ﬁxed for a given pair.
Revised: The sprite shapes and orientations are simple, like dSprites and are ﬁxed for a given pair.

Original: While ﬁxing shape follows the natural transitions of objects, it is unclear how to accurately estimate object orientation from the masks, and thus we ﬁxed the factor to avoid introducing artiﬁcial transitions.
Revised: While ﬁxing shape follows the natural transitions of objects, it is unclear how to accurately estimate object orientation from the masks, and thus we ﬁxed the factor to avoid introducing artiﬁcial transitions.

Original: We additionally consider a version that is discretized to the same number of object states as dSprites, which i) allows us to leverage the standard DisLib evaluation metrics and ii) helps isolate the effect of including natural transitions from the effect of increasing data complexity (see Appendix E.3 for more details).
Revised: We additionally consider a version that is discretized to the same number of object states as dSprites, which i) allows us to use the standard DisLib evaluation metrics and ii) helps isolate the effect of including natural transitions from the effect of increasing data complexity (see Appendix D.4 for more details).

Original: (3) KITTI Masks is composed of pedestrian segmentation masks from an autonomous driving vision benchmark KITTI-MOTS, thus with natural shapes and continuous natural transitions in all underlying factors.
Revised: (3) KITTI Masks is composed of pedestrian segmentation masks from the autonomous driving vision benchmark KITTI-MOTS, thus with natural shapes and continuous natural transitions in all underlying factors.

Original: We consider adjacent frames which correspond to mean (∆ t )
Revised: We consider adjacent frames which correspond to mean (∆ t )

Original: = 0 .
Revised: = 0 .

Original: 05 s in physical time (we report the mean because of variable sampling rates in the original data); as well as frames with a larger temporal gap of mean (∆ t )
Revised: 05 s in physical time (we report the mean because of variable sampling rates in the original data); as well as frames with a larger temporal gap of mean (∆ t )

Original: = 0 .
Revised: = 0 .

Original: 15 s , which corresponds to samples of pairs that are at most 5 frames apart.
Revised: 15 s , which corresponds to samples of pairs that are at most 5 frames apart.

Original: We show in Appendix H.3 that SlowVAE disentanglement performance increases and then plateaus as we continue to increase mean (∆ t ) .
Revised: We show in Appendix G.3 that SlowVAE disentanglement performance increases and then plateaus as we continue to increase mean (∆ t ) .

Original: In summary, we construct datasets with (1) imposed sparse transitions, (2) augmented with natural continuous generative factors using measurements from unstructured natural videos, as well as (3) data from unstructured natural videos themselves, but provided in segmentation mask form to ensure visual complexity  manageable for current methods.
Revised: In summary, we construct datasets with (1) imposed sparse transitions, (2) augmented with natural continuous generative factors using measurements from unstructured natural videos, as well as (3) data from unstructured natural videos themselves, but provided as segmentation masks to ensure visual complexity is manageable for current methods.

Original: For the provided datasets, the object categories  never change across transitions – reﬂecting natural object permanence.
Revised: For the provided datasets, the object categories [Table] never change across transitions – reﬂecting natural object permanence.

Original: Finally, as (2) and (3) use factor transitions measured from natural videos, they exhibit any natural statistical structure present for those factors, such as natural dependencies (see Appendix G.2 for further discussion on dependencies and Appendix E for further discussion on the datasets themselves).
Revised: Finally, as (2) and (3) use factor transitions measured from natural videos, they exhibit any natural statistical structure present for those factors, such as natural dependencies (further discussion is in Appendix F.2).

Original:  E XPERIMENTS 5. E MPIRICAL S TUDIES
Revised:  E XPERIMENTS 5. E MPIRICAL S TUDIES

Original: We evaluate models using the DisLib implementation for the following supervised metrics: BetaVAE (Higgins et al., 2017); FactorVAE (Kim and Mnih, 2018); Mutual Information Gap (MIG, (Chen et al., 2018)); Disentanglement, Compactness, and Informativeness (DCI [Disentanglement], (Eastwood and Williams, 2018)); Modularity (Ridgeway and Mozer, 2018); and Separated
Revised: We evaluate models using the DisLib implementation for the following supervised metrics: BetaVAE (Higgins et al., 2017); FactorVAE (Kim and Mnih, 2018); Mutual Information Gap (MIG; Chen et al., 2018); Disentanglement, Compactness, and Informativeness (DCI / Disentanglement; Eastwood and Williams, 2018); Modularity (Ridgeway and Mozer, 2018); and Separated

Original: Attribute Predictability (SAP, (Kumar et al., 2018))
Revised: Attribute Predictability (SAP; Kumar et al., 2018)

Original: (see Appendix D for metric details).
Revised: (see Appendix C for metric details).

Original: None of the
Revised: None of the

Original: DisLib metrics support ground-truth labels with continuous variation, which is required for evaluation[Table] on the continuous Natural Sprites and KITTI Masks datasets.
Revised: DisLib metrics support ground-truth labels with continuous variation, which is required for evaluation on the continuous Natural Sprites and KITTI Masks datasets.

Original: To reconcile this, we measure the Mean
Revised: To reconcile this, we measure the Mean

Original: Correlation Coefﬁcient (MCC), a standard metric in the ICA literature that is applicable to continuous variables.
Revised: Correlation Coefﬁcient (MCC), a standard metric in the ICA literature that is applicable to continuous variables.

Original: We report mean and standard deviation across 10 random seeds.
Revised: We report mean and standard deviation across 10 random seeds.

Original: In order to select the conditional prior regularization and the prior rate in an unsupervised manner, we perform a random search over γ ∈ [1 , 16] and λ ∈ [1 , 10] and compute the recently proposed unsupervised disentanglement ranking (UDR) scores (Duan et al., 2020).
Revised: In order to select the conditional prior regularization and the prior rate in an unsupervised manner, we perform a random search over γ ∈ [1 , 16] and λ ∈ [1 , 10] and compute the recently proposed unsupervised disentanglement ranking (UDR) scores (Duan et al., 2020).

Original: We notice that the optimal values are close to γ = 10 and λ = 6 on most datasets, and thus use these values for all experiments.
Revised: We notice that the optimal values are close to γ = 10 and λ = 6 on most datasets, and thus use these values for all experiments.

Original: We leave ﬁnding optimal values for speciﬁc datasets to future work, but note that it is a strong advantage of our approach that it works well with the same model speciﬁcation across 13 datasets (counting LAP and UNI for DisLib and optional discretization for Natural Sprites), addressing a concern posed in (Locatello et al., 2018).
Revised: We leave ﬁnding optimal values for speciﬁc datasets to future work, but note that it is a strong advantage of our approach that it works well with the same model speciﬁcation across 13 datasets (counting LAP and UNI for DisLib and optional discretization for Natural Sprites), addressing a concern posed in (Locatello et al., 2018).

Original: Additional details on model selection and training can be found in Appendix F.1. Although we train on image pairs, our model does not need paired data points at test time.
Revised: Additional details on model selection and training can be found in Appendix E. Although we train on image pairs, our model does not need paired data points at test time.

Original: For all visualizations, we pick the models with the highest average score across the DisLib metrics.
Revised: For all visualizations, we pick the models with the highest average score across the DisLib metrics.

Original: To compare our model fairly against other methods that also take image pairs as inputs [Table] we took Permutation-Contrastive Learning from nonlinear ICA (PCL, Hyvärinen and Morioka, 2017) and AdaGVAE, the leading method in the study by (Locatello et al., 2020).
Revised: To compare our model fairly against other methods that also take image pairs as inputs, we also present performance for Permutation-Contrastive Learning from nonlinear ICA (PCL, Hyvärinen and Morioka, 2017) and Ada-GVAE, the leading method in the study by (Locatello et al., 2020).

Original: We scaled up the implementation of PCL for evaluation on our high-dimensional pixel inputs [Table] and note this method does not have any hyperparameters.
Revised: We scaled up the implementation of PCL for evaluation on our high-dimensional pixel inputs, and note this method does not have any hyperparameters.

Original: For Ada-GVAE, following the paper’s recommendations, we select β (per dataset) using the considered parameter set [1 , 2 , 4 , 6 , 8 , 16] [Table] and use theweakly supervised reconstruction loss as the unsupervised model selection criterion (Locatello et al., 2020) [Table]
Revised: For Ada-GVAE, following the paper’s recommendations, we select β (per dataset) using the considered parameter set [1 , 2 , 4 , 6 , 8 , 16] , and use the reconstruction loss as the unsupervised model selection criterion (Locatello et al., 2020).

Original: 5. R ESULTS ON D IS L IB AND N EW B ENCHMARKS
Revised: 5. R ESULTS ON D IS L IB AND N EW B ENCHMARKS

Original: [Table] See Table 1 for details. In Table 1 we demonstrate favorable performance compared to PCL and Ada-GVAE across all applicable metrics for discrete ground-truth variable datasets.
Revised: In Table 1 we demonstrate favorable performance compared to PCL and Ada-GVAE across all applicable metrics for discrete ground-truth variable datasets.

Original: The relative improvement on UNI is particularly surprising given the drastic mismatch between UNI and SlowVAE’s assumptions.
Revised: The relative improvement on UNI is particularly surprising given the drastic mismatch between UNI and SlowVAE’s assumptions.

Original: In Appendix H, we report results for the remaining DisLib datasets, where the observed dSprites results largely transfer.
Revised: In Appendix G, we report results for the remaining DisLib datasets, where the observed dSprites results largely transfer.

Original: We also outperform PCL with a (ﬂow-based) exact likelihood implementation of our slow transition prior in Appendix G.1.1. In
Revised: We also outperform PCL with a (ﬂow-based) exact likelihood implementation of our slow transition prior in Appendix F.1.1.

Original:  Appendix G.3, we show that a model with a transition L 2  ( α = 2 ) prior performs much worse, conﬁrming our theoretical prediction.
Revised: In Appendix F.3, we show that a model with an L 2 transition ( α = 2 ) prior performs much worse, supporting our theoretical prediction [Table]

Original:  On the KITTI Masks dataset, one source of variation in the data is the average temporal separation within pairs of images mean (∆ t ) .
Revised: datasets. See Table 1 for details. On the KITTI Masks dataset, one source of variation in the data is the average temporal separation within pairs of images mean (∆ t ) .

Original: We present two settings (mean (∆ t )
Revised: We present two settings (mean (∆ t )

Original: = 0 .
Revised: = 0 .

Original: 05 s , mean (∆ t ) = 0 .
Revised: 05 s , mean (∆ t ) = 0 .

Original: 15 s )
Revised: 15 s )

Original: [Figure]

Original: S h a p e

Original: S c a l e - R o t x - P

Original: o s y - P o s and observe a comparative increase in MCC for the latter (Table 2).
Revised: and observe a comparative increase in MCC for the latter (Table 2).

Original: Namely, the increase in performance for larger time gap is more pronounced with SlowVAE than the baselines, resulting in a statistically signiﬁcant MCC gain.
Revised: Namely, the increase in performance for larger time gap is more pronounced with SlowVAE than the baselines, resulting in a statistically signiﬁcant MCC gain.

Original: We provide details on the settings and ablate over the mean (∆ t ) parameter in Appendix H.3, where we observe a positive trend between mean (∆ t ) and MCC (reﬂecting Table 2, in Oord et al., 2018).
Revised: We provide details on the settings and ablate over the mean (∆ t ) parameter in Appendix G.3, where we observe a positive trend between mean (∆ t ) and MCC (reﬂecting Table 2, in Oord et al., 2018).

Original: Finally, we also verify that the transition distributions remain sparse despite the increase in this parameter (Appendix H.3).
Revised: Finally, we also verify that the transition distributions remain sparse despite the increase in this parameter (Appendix G.3).

Original: In Fig.
Revised: In Fig.

Original: 4, we can see that SlowVAE has learned latent dimensions which have correspondence with the estimated ground truth factors of x/y-position and scale.
Revised: 4, we can see that SlowVAE has learned latent dimensions which have correspondence with the estimated ground truth factors of x/y-position and scale.

Original: Locatello et al.
Revised: Locatello et al.

Original: (2018) showed that all i.i.d.
Revised: (2018) showed that all i.i.d.

Original: models performed similarly across the DisLib datasets and metrics when testing was carefully controlled.
Revised: models performed similarly across the DisLib datasets and metrics when testing was carefully controlled.

Original: However, in Fig.
Revised: However, in Fig.

Original: 5 we observe that the different modeling assumptions result in differences in representation quality.
Revised: 5 we observe that the different modeling assumptions result in differences in representation quality.

Original: To construct the visuals, we ﬁrst compute the sorted correlationcoefﬁcient matrix between the latents  and generative factors, which we visualize in the top row.
Revised: To construct the visuals, we ﬁrst compute the sorted correlation matrix between the latents (rows) and generative factors (columns), which we visualize as a correlation matrices.

Original: The correlation matrix is sorted via linear sum assignment such that each ground-truth factor is non-greedily associated with the latent variable with highest correlation.
Revised: The matrices are sorted via linear sum assignment such that each ground-truth factor is non-greedily associated with the latent variable with highest correlation (Hyvärinen and Morioka, 2016).

Original: Each row of the  scatter plots corresponds to an individual generative factor.
Revised: Below the matrices are scatter plots that reveal the decodability of the assigned latent factors.

Original: In each subplot, the horizontal axis indicates the ground truth value, the vertical axis indicates the corresponding latent value, and the colors indicate object shape.
Revised: In each scatter plot, the horizontal axis indicates the ground truth value, the vertical axis indicates the corresponding latent value, and the colors indicate object shape.

Original: The models displayed are those with the maximum average score across evaluated metrics.
Revised: The models displayed are those with the maximum average score across evaluated metrics.

Original: The latent space visualizations leverage the known ground-truth factors to aid in understanding how each factor is encoded in a way that is more informative than exclusively visualizing latent traversals or embeddings of pairs of latent units (Cheung et al., 2014; Chen et al., 2016; Szabó et al., 2017; Ma et al., 2018).
Revised: The latent space visualizations use the known ground-truth factors to aid in understanding how each factor is encoded in a way that is more informative than exclusively visualizing latent traversals or embeddings of pairs of latent units (Cheung et al., 2014; Chen et al., 2016; Szabó et al., 2017; Ma et al., 2018).

Original: For example, in the third row, we observe that several models have a sinusoidal variation with frequencies ∼ ω, 2 ω, and 4 ω , which correspond to the three distinct rotational symmetries of the shapes: heart, ellipse and square.
Revised: For example, in the third row, we observe that several models have a sinusoidal variation with frequencies ∼ ω, 2 ω, and 4 ω , which correspond to the three distinct rotational symmetries of the shapes: heart, ellipse and square.

Original: This directly impacts MCC performance (third row in the MCC matrix), which measures rank correlation between the matching latent factor (an angular variable) and the ground truth, which encodes the angles with monotonically increasing indices.
Revised: This directly impacts MCC performance (third row in the MCC matrix), which measures rank correlation between the matching latent factor (an angular variable) and the ground truth, which encodes the angles with monotonically increasing indices.

Original: Furthermore, the square has a four-fold rotational symmetry and repeats after 90 ◦ , but it is represented in a full 360 ◦ rotation in the DisLib ground truth encoding format, resulting in different ground truth labels for identical input images. 
Revised: Furthermore, the square has a four-fold rotational symmetry and repeats after 90 ◦ , but it is represented in a full 360 ◦ rotation in the DisLib ground truth encoding format, resulting in different ground truth labels for identical input images. [Table] [Figure]

Original: A similar observation can be made with respect to the categorical factors, which are also represented as ordinal ground truth variables.
Revised: A similar observation can be made with respect to the categorical factors, which are also represented as ordinal ground truth variables.

Original: For example, the PCL correlation score (top left element in the PCL
Revised: For example, the PCL correlation score (top left element in the PCL

Original: MCC matrix) is quite high, while the corresponding shape correlation score for SlowVAE is quite low.
Revised: MCC matrix) is quite high, while the corresponding shape correlation score for SlowVAE is quite low.

Original: However, if we consider the shape scatter plots, we clearly see that SlowVAE separates the three shapes more distinctively than PCL, only in an order that differs from the ground truth.
Revised: However, if we consider the shape scatter plots, we clearly see that SlowVAE separates the three shapes more distinctively than PCL, only in an order that differs from the ground truth.

Original: One solution is to modify MCC to report the maximum correlation over all permutations of the ground truth assignments, although brute force methods for this would scale poorly with the number of categories.
Revised: One solution is to modify MCC to report the maximum correlation over all permutations of the ground truth assignments, although brute force methods for this would scale poorly with the number of categories.

Original: We also note that datasets where we see small performance differences among models (e.g., Cars3D) have signiﬁcantly more discrete categories (e.g., 183 ) than the other datasets ( 3 − 6 ).
Revised: We also note that datasets where we see small performance differences among models (e.g., Cars3D) have signiﬁcantly more discrete categories (e.g., 183 ) than the other datasets ( 3 − 6 ).

Original: This could also explain why all models considered in Table 1 and 2 perform comparably on the Natural Sprites datasets, where unlike KITTI Masks the ground truth evaluation includes categorical and angular variables.
Revised: This could also explain why all models considered in Table 1 and 2 perform comparably on the Natural Sprites datasets, where unlike KITTI Masks the ground truth evaluation includes categorical and angular variables.

Original: We note that properly evaluating disentanglement is an ongoing area of research, with notable  results in recent work (Higgins et al., 2018; Duan et al., 2020; Anonymous, 2021a;b).
Revised: We note that properly evaluating disentanglement is an ongoing area of research (Duan et al., 2020), with notable preliminary results in recent work (Higgins et al., 2018; Bouchacourt et al., 2021; Tonnaer et al., 2020).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We provide evidence to support the hypothesis that natural scenes exhibit highly sparse marginal transition probabilities.
Revised: We provide evidence to support the hypothesis that natural scenes exhibit highly sparse marginal transition probabilities.

Original: Leveraging this ﬁnding, we contribute a novel nonlinear ICA framework that is provably identiﬁable up to permutations and sign-ﬂips — a stronger result than has been achieved previously.
Revised: Leveraging this ﬁnding, we contribute a novel nonlinear ICA framework that is provably identiﬁable up to permutations and sign-ﬂips — a stronger result than has been achieved previously.

Original: With the SlowVAE model we provide a parsimonious implementation that is inspired by a long history of learning visual representations from temporal data (Sutton, 1988; Hinton, 1990; Földiák, 1991).
Revised: With the SlowVAE model we provide a parsimonious implementation that is inspired by a long history of learning visual representations from temporal data (Sutton, 1988; Hinton, 1990; Földiák, 1991).

Original: We apply this model to current metric-based disentanglement benchmarks to demonstrate that it outperforms existing approaches (Locatello et al., 2020; Hyvärinen and Morioka, 2017) on aggregate without any tuning of hyperparameters to individual datasets.
Revised: We apply this model to current metric-based disentanglement benchmarks to demonstrate that it outperforms existing approaches (Locatello et al., 2020; Hyvärinen and Morioka, 2017) on aggregate without any tuning of hyperparameters to individual datasets.

Original: We also provide novel video dataset benchmarks to guide disentanglement research towards more natural domains.
Revised: We also provide novel video dataset benchmarks to guide disentanglement research towards more natural domains.

Original: We observe that these datasets have complex dependencies that our theory will have to be extended to account for, although we demonstrate with empirical comparisons the efﬁcacy of our approach.
Revised: We observe that these datasets have complex dependencies that our theory will have to be extended to account for, although we demonstrate with empirical comparisons the efﬁcacy of our approach.

Original: In addition to Natural Sprites and KITTI Masks, we suggest that YouTube-VOS will be valuable as a large-scale dataset that is unconstrained by object type and scenario for more advanced models.
Revised: In addition to Natural Sprites and KITTI Masks, we suggest that YouTube-VOS will be valuable as a large-scale dataset that is unconstrained by object type and scenario for more advanced models.

Original: Variance in such categorical factors is problematic for evaluation due to the cited drawbacks of existing quantitative metrics, which should be addressed in tandem with scaling to natural data.
Revised: Variance in such categorical factors is problematic for evaluation due to the cited drawbacks of existing quantitative metrics, which should be addressed in tandem with scaling to natural data.

Original: Taken together, our dataset and model proposals set the stage for utilizing knowledge of natural scene statistics to advance unsupervised disentangled representation learning.
Revised: Taken together, our dataset and model proposals set the stage for utilizing knowledge of natural scene statistics to advance unsupervised disentangled representation learning.

Original: In our experiments we see that approximate identiﬁcation as measured by the different disentanglement metrics increases despite violations of theoretical assumptions, which is in line with prior studies (Shu et al., 2019; Khemakhem et al., 2020a;
Revised: In our experiments we see that approximate identiﬁcation as measured by the different disentanglement metrics increases despite violations of theoretical assumptions, which is in line with prior studies (Shu et al., 2019; Khemakhem et al., 2020a;

Original: Locatello et al., 2020).
Revised: Locatello et al., 2020).

Original: Nevertheless, future work should address gaining a better understanding of the theoretical and empirical consequences of such model misspeciﬁcations, in order to make the theory of disentanglement more predictive about empirically found solutions.
Revised: Nevertheless, future work should address gaining a better understanding of the theoretical and empirical consequences of such model misspeciﬁcations, in order to make the theory of disentanglement more predictive about empirically found solutions.

Original: A PPENDIX A Broader Impact
Revised: A CKNOWLEDGEMENTS

Revised: The authors would like to thank Francesco Locatello for valuable discussions and providing numerical results to facilitate our experimental comparisons. Additionally, we thank Luigi Gresele, Matthias Tangemann, Roland Zimmermann, Robert Geirhos, Matthias Kümmerer, Cornelius Schröder, Charles

Revised: Frye, and Sarah Master for helpful feedback in preparing the manuscript.

Revised: Finally, the authors would like to thank Johannes Ballé, Jon Shlens and Eero Simoncelli for early discussions related to the ideas developed in this paper.

Revised: This work was supported by the Deutsche Forschungsgemeinschaft (DFG) in the priority program 1835 under grant BR2321/5-2 and by SFB 1233, Robust Vision: Inference Principles and Neural

Revised: Mechanisms (TP3), project number: 276693517.

Revised: We thank the International Max Planck Research

Revised: School for Intelligent Systems (IMPRS-IS) for supporting LS and YS.

Revised: DP was supported by the

Revised: German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039A).

Revised: IU, WB, and MB are supported by the Intelligence Advanced Research Projects

Revised: Activity (IARPA) via Department of Interior/Interior Business Center (DoI/IBC) contract number

Revised: D16PC00003.

Revised: The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.

Revised: Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies or endorsements, either expressed or implied, of IARPA, DoI/IBC, or the U.S. Government.

Revised: The authors declare no conﬂicts of interests.

Revised: B ROADER I MPACT

Revised: Representation learning is at the heart of model building for cognition.

Revised: Our speciﬁc contribution is focused on core methods for modeling natural videos and the datasets used are more simplistic than real-world examples.

Revised: However, foundational research on unsupervised representation learning has potentially large impact on AI for advancing the power of self-learning systems.

Revised: The broader ﬁeld of representation learning has a large number of focused research directions that span machine learning and computational neuroscience.

Revised: As such, the application space for this work is vast.

Revised: For example, applications in unsupervised analysis of complicated and unintuitive data, such as medical imaging and gene expression information, have great potential to solve fundamental problems in health sciences.

Revised: A future iteration of our disentangling approach could be used to encode such complicated data into a lower-dimensional and more understandable space that might reveal important factors of variation to medical researchers.

Revised: Another important and complex modeling space that could potentially be improved by this line of research is in environmental sciences and combating global climate change.

Revised: Nonetheless, we acknowledge that any machine learning method can be used for nefarious purposes, which can be mitigated via effective, scientiﬁcally informed communication, outreach, and policy direction.

Revised: We unconditionally denounce the use of derivatives of our work for weaponized or wartime applications.

Revised: Additionally, due to the lack of interpretability generally found in modern deep learning approaches, it is possible for practitioners to inadvertently introduce harmful biases or errors in machine learning applications.

Revised: Although we certainly do not solve this problem, our focus on providing identiﬁable solutions to representation learning is likely beneﬁcial for both interpretability and fairness in machine learning.

Original: B Formal Methods
Revised: A PPENDIX A Formal Methods

Original: B.1 Proof of Identiﬁability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.1 Proof of Identiﬁability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2 Kullback Leibler Divergence of Slow Variational Autoencoder . . . . . . . . . . .
Revised: A.2 Kullback Leibler Divergence of Slow Variational Autoencoder . . . . . . . . . . .

Original: C Choosing a Latent Variable Model
Revised: B Choosing a Latent Variable Model

Original: D Disentanglement Metrics
Revised: C Disentanglement Metrics

Original: D.1 Mean Correlation Coefﬁcient . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1 Mean Correlation Coefﬁcient . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 DisLib Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.2 DisLib Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E Natural Datasets
Revised: D Natural Datasets

Original: E.1 Laplace Transitions  . . . . .  . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1 Uniform Transitions (UNI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 Laplace Transitions (LAP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.2 YouTube-VOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.3 YouTube-VOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.3 Natural Sprites . .
Revised: D.4 Natural Sprites . .

Original: E.4 KITTI MOTS Pedestrian Masks  . .F Implementation Details F.
Revised: D.5 KITTI MOTS Pedestrian Masks (KITTI Masks) . . . . . . . . . . . . . . . . . . .

Original:  Model Training and Selection. .
Revised: E Model Training and Selection

Revised: F Extended Comparisons and Controls

Original: F.
Revised: F.

Original: Dataset Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G Extended Comparisons and Controls

Original: G.1 Comparison to Nonlinear ICA . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Comparison to Nonlinear ICA . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.
Revised: F.

Original: [Table]

Original: Joint Factor Dependence Evaluation . .. . . . . . . . . . . . . . . . . . . . . . .
Revised: Joint Factor Dependence Evaluation . .

Original: G
Revised: F

Original: [Table]

Original: Transition Prior Ablation . .
Revised: Ablation . .

Original: H Additional Results
Revised: G Additional Results

Original: H.

Original: [Table]

Original:  Extended Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .H.
Revised: G.1 Extended Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [Table]

Original:  All DisLib Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .H.
Revised: G.2 All DisLib Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [Table] Masks ∆ t Ablation . .
Revised: G.3 KITTI Masks ∆ t Ablation . .

Original: H.
Revised: G

Original: [Table]
Revised: [Table]

Original: Space Visualizations . . . . 
Revised: Visualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A B ROADER I MPACT
Revised: A F ORMAL M ETHODS

Original: Representation learning is at the heart of model building for cognition.

Original: Our speciﬁc contribution is focused on core methods for modeling natural videos and the datasets used are more simplistic than real-world examples.

Original: However, foundational research on unsupervised representation learning has potentially large impact on AI for advancing the power of self-learning systems.

Original: The broader ﬁeld of representation learning has a large number of focused research directions that span machine learning and computational neuroscience.

Original: As such, the application space for this work is vast.

Original: For example, applications in unsupervised analysis of complicated and unintuitive data, such as medical imaging and gene expression information, have great potential to solve fundamental problems in health sciences.

Original: A future iteration of our disentangling approach could be used to encode such complicated data into a lower-dimensional and more understandable space that might reveal important factors of variation to medical researchers.

Original: Another important and complex modeling space that could potentially be improved by this line of research is in environmental sciences and combating global climate change.

Original: Nonetheless, we acknowledge that any machine learning method can be used for nefarious purposes, which can be mitigated via effective, scientiﬁcally informed communication, outreach, and policy direction.

Original: We unconditionally denounce the use of derivatives of our work for weaponized or wartime applications.

Original: Additionally, due to the lack of interpretability generally found in modern deep learning approaches, it is possible for practitioners to inadvertently introduce harmful biases or errors in machine learning applications.

Original: Although we certainly do not solve this problem, our focus on providing identiﬁable solutions to representation learning is likely beneﬁcial for both interpretability and fairness in machine learning.

Original: B F ORMAL M ETHODS Function / variable

Original: [Table]
Revised: [Table]

Original: B. P ROOF OF I DENTIFIABILITY
Revised: A. P ROOF OF I DENTIFIABILITY

Original: To study disentanglement, we assume that the generative factors z ∈ R D are mapped to images x ∈ R N (usually D (cid:28) N , but see section C) by a nonlinear ground-truth generator g ∗ :
Revised: To study disentanglement, we assume that the generative factors z ∈ R D are mapped to images x ∈ R N (usually D (cid:28) N , but see section B) by a nonlinear ground-truth generator g ∗ :

Original: z (cid:55)→ x .
Revised: z (cid:55)→ x .

Original: Theorem 1 Let ( g ∗ , λ ∗ , α ∗ ) and ( g, λ, α ) respectively be ground-truth and learned generative models as deﬁned in Eq.
Revised: Theorem 1 Let ( g ∗ , λ ∗ , α ∗ ) and ( g, λ, α ) respectively be ground-truth and learned generative models as deﬁned in Eq.

Original: If the following conditions are satisﬁed
Revised: If the following conditions are satisﬁed:

Original: : (i)
Revised: (i)

Original: The generators g ∗ and g are deﬁned everywhere in the latent space.
Revised: The generators g ∗ and g are deﬁned everywhere in the latent space.

Original: Moreover, they are injective and differentiable almost everywhere, (ii) There is no model misspeciﬁcationi.e. α = α ∗ and λ
Revised: Moreover, they are injective and differentiable almost everywhere, (ii) There is no model misspeciﬁcation

Original:  = λ ∗ , so z ∼ p ( z ) =
Revised: i e. α = α ∗ and λ = λ ∗ , so z ∼ p ( z ) =

Original: p ∗ ( z ) , (iii) Pairs of images are generated as ( x ∗ t − 1 , x ∗ t )
Revised: p ∗ ( z ) , (iii) Pairs of images are generated as ( x ∗ t − 1 , x ∗ t )

Original: =
Revised: =

Original: ( g ∗ ( z t − 1 ) , g ∗ ( z t )) and ( x t − 1 , x t ) =
Revised: ( g ∗ ( z t − 1 ) , g ∗ ( z t )) and ( x t − 1 , x t ) =

Original: ( g ( z t − 1 ) , g ( z t )) , (iv) The distributions of ( x ∗ t − 1 , x ∗ t ) and ( x t − 1 , x t ) are the same (i
Revised: ( g ( z t − 1 ) , g ( z t )) , (iv) The distributions of ( x ∗ t − 1 , x ∗ t ) and ( x t − 1 , x t ) are the same (i

Original: e. the corresponding densities are equal almost everywhere: p ∗ ( x t − 1 , x t ) =
Revised: e. the corresponding densities are equal almost everywhere: p ∗ ( x t − 1 , x t ) =

Original: p ( x t − 1 , x t ) ,
Revised: p ( x t − 1 , x t ) ,

Original: then g = g ∗ ◦ σ , where σ is a composition of a permutation and sign ﬂips.
Revised: then g = g ∗ ◦ σ , where σ is a composition of a permutation and sign ﬂips.

Original: Proof.
Revised: Proof.

Original: Since x = g ( z ) can be written as x = ( g ∗ ◦ ( g ∗ ) − 1 ◦ g )
Revised: Since x = g ( z ) can be written as x = ( g ∗ ◦ ( g ∗ ) − 1 ◦ g )

Original: ( z ) , we can assume that g = g ∗ ◦ h for some function h on the latent space.
Revised: ( z ) , we can assume that g = g ∗ ◦ h for some function h on the latent space.

Original: We ﬁrst show that the function h is a bijection on the latent space.
Revised: We ﬁrst show that the function h is a bijection on the latent space.

Original: It is injective, since both g and g ∗ are injective.
Revised: It is injective, since both g and g ∗ are injective.

Original: If it were not surjective, there would be a point ˜ z in the ground-truth latent space that would not have a pre-image under h . Because of continuity of h ,  there would be some neighborhood U ˜
Revised: Because of continuity of h , if it were not surjective, there would be some neighborhood U ˜

Original: z of
Revised: z of

Original: ˜ z that would not have a pre-image under h .
Revised: ˜ z that would not have a pre-image under h .

Original: This would mean that images generated by g ∗ from U ˜ z would have zero density under the distribution of images generated by g (i.e. p ( g ∗ ( U ˜ z ))
Revised: This would mean that images generated by g ∗ from U ˜ z would have zero density under the distribution of images generated by g (i.e. p ( g ∗ ( U ˜ z ))

Original: = 0 ), while this density would be non-zero under the distribution of images directly generated by the ground-truth generator g ∗ (i.e. p ∗ ( g ∗ ( U ˜ z )) (cid:54) = 0 ), which contradicts the assumption that these distributions are equal.
Revised: = 0 ). This density would be non-zero under the distribution of images directly generated by the ground-truth generator g ∗ (i.e. p ∗ ( g ∗ ( U ˜ z )) (cid:54) = 0 ), which contradicts the assumption that these distributions are equal.

Original: It follows that h is bijective.
Revised: It follows that h is bijective.

Original: In the next step, we show that the distribution of latent space pairs ( h ( z t − 1 ) , h ( z t )) matches the latent space prior distribution (i.e. h preserves the prior distribution in the latent space).
Revised: In the next step, we show that the distribution of latent space pairs ( h ( z t − 1 ) , h ( z t )) matches the latent space prior distribution (i.e. h preserves the prior distribution in the latent space).

Original: Indeed, using the assumption that the distributions of ( g ∗ ( z t − 1 ) , g ∗ ( z t )) and (( g ∗ ◦ h )( z t − 1 ) , ( g ∗ ◦ h )( z t )) are the same, we can write the following equality using the change of variables formula:
Revised: Indeed, using the assumption that the distributions of ( g ∗ ( z t − 1 ) , g ∗ ( z t )) and (( g ∗ ◦ h )( z t − 1 ) , ( g ∗ ◦ h )( z t )) are the same, we can write the following equality using the change of variables formula:

Original: [Equation]
Revised: [Equation]

Original: where p and p h are densities of ( z t − 1 , z t ) and ( h ( z t − 1 ) , h ( z t )) .
Revised: where p and p h are densities of ( z t − 1 , z t ) and ( h ( z t − 1 ) , h ( z t )) .

Original: Since the determinants above cancel, these densities are equal at the pre-image of any pair of images ( x t − 1 , x t ) .
Revised: Since the determinants above cancel, these densities are equal at the pre-image of any pair of images ( x t − 1 , x t ) .

Original: Because g ∗ is deﬁned everywhere in the latent space, p and p h are equal for any pair of latent space points.
Revised: Because g ∗ is deﬁned everywhere in the latent space, p and p h are equal for any pair of latent space points.

Original: Applying the change of variables formula again [Equation] we obtain the following equation:
Revised: Applying the change of variables formula again, we obtain the following equation:

Original: [Equation]
Revised: [Equation]

Original: Note that the probability measure p is the same before and after the change of variables, since we showed that the prior distribution in the latent space must be invariant under the function h .
Revised: Note that the probability measure p is the same before and after the change of variables, since we showed that the prior distribution in the latent space must be invariant under the function h .

Original: The same condition for the marginal p ( z t − 1 ) is as follows:
Revised: The same condition for the marginal p ( z t − 1 ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: Solving for the determinant of the Jacobian in (7) and plugging it into (6), we obtain
Revised: Solving for the determinant of the Jacobian in (7) and plugging it into (6), we obtain

Original: [Equation]
Revised: [Equation]

Original: Taking logs of both sides, we arrive at the following equation:
Revised: Taking logs of both sides, we arrive at the following equation:

Original: [Equation]
Revised: [Equation]

Original: where A and B are the constants appearing in the exponentials in p ( z t − 1 ) and p ( z t | z t − 1 ) .
Revised: where A and B are the constants appearing in the exponentials in p ( z t − 1 ) and p ( z t | z t − 1 ) .

Original: The logs of normalization constants cancel out.
Revised: The logs of normalization constants cancel out.

Original: For any z t we can choose z t − 1 =
Revised: For any z t we can choose z t − 1 =

Original: z t making the left hand side in (9) equal to zero.
Revised: z t making the left hand side in (9) equal to zero.

Original: This implies that || z t || 22 =
Revised: This implies that || z t || 22 =

Original: ||
Revised: ||

Original: h − 1 ( z t ) || 22 for any z t , i.e. function h − 1 preserves the 2-norm.
Revised: h − 1 ( z t ) || 22 for any z t , i.e. function h − 1 preserves the 2-norm.

Original: Moreover, the preservation of the 2-norm implies that p ( z t − 1 ) =
Revised: Moreover, the preservation of the 2-norm implies that p ( z t − 1 ) =

Original: p ( h − 1 ( z t − 1 ))
Revised: p ( h − 1 ( z t − 1 ))

Original: and therefore it follows from (7) that for any z
Revised: and therefore it follows from (7) that for any z

Original: [Equation]
Revised: [Equation]

Original: Thus, the left hand side of (9) can be re-written as
Revised: Thus, the left hand side of (9) can be re-written as

Original: [Equation]
Revised: [Equation]

Original: This means that h − 1 preserves the α -distances between points.
Revised: This means that h − 1 preserves the α -distances between points.

Original: Moreover, because h is bijective, the
Revised: Moreover, because h is bijective, the

Original: Mazur-Ulam theorem (Mazur and Ulam, 1932) tells us that h must be an afﬁne transform.
Revised: Mazur-Ulam theorem (Mazur and Ulam, 1932) tells us that h must be an afﬁne transform.

Original: In the next step, to prove that h must be a permutation and sign ﬂip, let us choose an arbitrary point z t − 1 and z t =
Revised: In the next step, to prove that h must be a permutation and sign ﬂip, let us choose an arbitrary point z t − 1 and z t =

Original: z t − 1 + ε e k
Revised: z t − 1 + ε e k

Original: =
Revised: =

Original: z 1 , 1 , . .
Revised: z 1 , 1 , . .

Original: , z 1 ,k
Revised: , z 1 ,k

Original: + ε, . .
Revised: + ε, . .

Original: , z 1 ,D ) .
Revised: , z 1 ,D ) .

Original: Using (11) and performing a Taylor expansion around z t − 1 , we obtain the following:
Revised: Using (11) and performing a Taylor expansion around z t − 1 , we obtain the following:

Original: [Equation]
Revised: [Equation]

Original: The higher-order terms O ( ε 2 ) are zero since h is afﬁne, therefore dividing both sides of the above equation by ε
Revised: The higher-order terms O ( ε 2 ) are zero since h is afﬁne, therefore dividing both sides of the above equation by ε

Original: α
Revised: α

Original: we ﬁnd that
Revised: we ﬁnd that

Original: [Equation]
Revised: [Equation]

Original: The vectors of k -th partial derivatives of components of h − 1 are columns of the Jacobian matrix (cid:16) d h [Equation] 1 ( z [Equation]
Revised: The vectors of k -th partial derivatives of components of h − 1 are columns of the Jacobian matrix (cid:16) d h − 1 ( z ) d z (cid:17) .

Original: d z (cid:17) .

Original: Using the fact that the determinant of that matrix is equal to one and applying Hadamard’s inequality
Revised: Using the fact that the determinant of that matrix is equal to one and applying Hadamard’s inequality

Original: [Equation]
Revised: [Equation]

Original: we obtain that
Revised: we obtain that

Original: [Equation]
Revised: [Equation]

Original: Since α < [Equation] for any vector v it holds 
Revised: Since α < 2 , for any vector v it holds that

Original: that ||
Revised: ||

Original: v || 2 ≤
Revised: v || 2 ≤

Original: ||
Revised: ||

Original: v || α , with equality only if at most one component of v is non-zero.
Revised: v || α , with equality only if at most one component of v is non-zero.

Original: This inequality implies that both (13) and (14) hold at the same time if and only if
Revised: This inequality implies that both (13) and (14) hold at the same time if and only if

Original: [Equation]
Revised: [Equation]

Original: meaning that only one element of these vectors of k -th partial derivatives is non-zero, and it is equal to 1 or -1.
Revised: meaning that only one element of these vectors of k -th partial derivatives is non-zero, and it is equal to 1 or -1.

Original: Thus, the function h is a composition of a permutation and sign ﬂips at every point.
Revised: Thus, the function h is a composition of a permutation and sign ﬂips at every point.

Original: Potentially, this permutation might be input-dependent, but we argued above that h is afﬁne, therefore the permutation must be the same for all points.
Revised: Potentially, this permutation might be input-dependent, but we argued above that h is afﬁne, therefore the permutation must be the same for all points.

Original: [Equation]
Revised: [Equation]

Original: B. K ULLBACK L EIBLER D IVERGENCE OF S LOW V ARIATIONAL A UTOENCODER
Revised: A. K ULLBACK L EIBLER D IVERGENCE OF S LOW V ARIATIONAL A UTOENCODER

Original: The VAE learns a variational approximation to the true posterior by maximizing a lower bound on the log-likelihood of the empirical data distribution D
Revised: The VAE learns a variational approximation to the true posterior by maximizing a lower bound on the log-likelihood of the empirical data distribution D

Original: [Equation]
Revised: [Equation]

Original: For this, we need to compute the Kullback-Leibler divergence (KL) between the posterior q ( z t , z t − 1 | x t , x t − 1 ) and the prior p ( z t , z t − 1 ) .
Revised: For this [Equation] we need to compute the Kullback-Leibler divergence (KL) between the posterior q ( z t , z t − 1 | x t , x t − 1 ) and the prior p ( z t , z t − 1 ) .

Original: Since all of these distributions are per design factorial, we will, for simplicity, derive the KL below for scalar variables (log-probabilities will simply have to be summed to obtain the full expression).
Revised: Since all of these distributions are per design factorial, we will, for simplicity, derive the KL below for scalar variables (log-probabilities will simply have to be summed to obtain the full expression).

Original: Recall that the model prior and posterior factorize like
Revised: Recall that the model prior and posterior factorize like

Original: [Equation]
Revised: [Equation]

Revised: Then

Original: Then, given a pair of inputs ( x t − 1 , x t ), the KL can be written
Revised: [Equation] given a pair of inputs ( x t − [Equation] x t )

Revised: [Equation]

Revised: the KL can be written

Original: [Equation]
Revised: [Equation]

Original: Where we use the fact that KL divergences decompose like D KL ( X, Y )
Revised: Where we use the fact that KL divergences decompose like D KL ( X, Y )

Original: =
Revised: =

Original: H ( X, Y ) − H
Revised: H ( X, Y ) − H

Original: ( X ) into (differential) cross-entropy H ( X, Y ) and entropy H ( X ) .
Revised: ( X ) into (differential) cross-entropy H ( X, Y ) and entropy H ( X ) .

Original: The ﬁrst term of the last line in (18) is the same KL divergence as in the standard VAE, namely between a Gaussian distribution q ( z t − 1 | x t − 1 ) with some µ ( x t − 1 ) and σ ( x t − 1 ) and a standard Normal distribution p ( z t − 1 ) .
Revised: The ﬁrst term of the last line in (18) is the same KL divergence as in the standard VAE, namely between a Gaussian distribution q ( z t − 1 | x t − 1 ) with some µ ( x t − 1 ) and σ ( x t − 1 ) and a standard Normal distribution p ( z t − 1 ) .

Original: The solution of the KL is given by D 
Revised: The solution of the KL is given by D KL

Original: [Equation]
Revised: [Equation]

Original: (Bishop, 2006).
Revised: (Bishop, 2006).

Original: The second term on the RHS, i.e. the entropy of a Gaussian is simply given by 
Revised: The second term on the RHS, i.e. the entropy of a Gaussian is simply given by H

Original: [Equation]
Revised: [Equation]

Original: To compute the last term on the RHS
Revised: t |

Original: [Equation]
Revised: [Equation]

Original:  let us recall the Laplace form of the conditional prior[Equation]
Revised: To compute the last term on the RHS, let us recall the Laplace form of the conditional prior

Original: [Equation]
Revised: [Equation]

Original: Thus the cross-entropy becomes
Revised: Thus the cross-entropy becomes

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: + [Equation] t

Original: [Equation]
Revised: [Equation]

Original: if some random variable X ∼ N ( µ, σ 2 )
Revised: t |

Original: [Equation]
Revised: [Equation]

Revised: t )

Original: then Y =
Revised: [ | [Equation] [Equation] Now, if some random variable X ∼ N ( µ, σ 2 )

Original:  | X | follows a folded normal distribution , for which the mean is deﬁned as
Revised: , then Y = | X | follows a folded normal distribution , for which the mean is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where Φ is the cumulative distribution function of a standard normal distribution (mean zero and variance one)
Revised: where Φ is the cumulative distribution function of a standard normal distribution (mean zero and variance one).

Revised: Thus, denoting µ ( x t ) and σ ( x t ) the mean and variance of q ( z t | x t ) , and deﬁning µ ( x t , z t − 1 ) = µ ( x t ) − z t − 1 , we can rewrite further

Revised: H ( q ( z t | x t )

Original: [Equation]
Revised: [Equation]

Original:  z ∗ ∆ z ∗ x ∗ z ∆ z x µ σ 5.
Revised: p ( z t | z t [Equation] = [Equation]

Original: MCC loss 0 π 2 π 0 π 2 π
Revised: (cid:19)(cid:19)(cid:33)

Original: [Figure]
Revised: [Equation]

Original: z ∗ ∆ z ∗ x ∗ z ∆ z 2.

Original: [Figure]
Revised: [Figure]

Original: C C HOOSING A L ATENT V ARIABLE M ODEL
Revised: B C HOOSING A L ATENT V ARIABLE M ODEL

Original: Our proposed method for disentanglement can be implemented in conjunction with different probabilistic latent variable models.
Revised: Our proposed method for disentanglement can be implemented in conjunction with different probabilistic latent variable models.

Original: In this section, we compare VAEs and normalizing ﬂows as possible candidates.
Revised: In this section, we compare VAEs and normalizing ﬂows as possible candidates.

Original: Variational Autoencoders (VAEs) (Kingma and Welling, 2013) are a widely used probabilistic latent variable model.
Revised: Variational Autoencoders (VAEs) (Kingma and Welling, 2013) are a widely used probabilistic latent variable model.

Original: Despite their simple structure and empirical success, VAEs can converge to a pathological solution called posterior collapse (Lucas et al., 2019; Bowman et al., 2016;
Revised: Despite their simple structure and empirical success, VAEs can converge to a pathological solution called posterior collapse (Lucas et al., 2019; Bowman et al., 2016;

Original: He et al., 2019).
Revised: He et al., 2019).

Original: This solution results in the encoder’s variational posterior approximation matching the prior, which is typically chosen to be a multivariate standard normal q ( z | x ) ≈ p ( z ) =
Revised: This solution results in the encoder’s variational posterior approximation matching the prior, which is typically chosen to be a multivariate standard normal q ( z | x ) ≈ p ( z ) =

Original: N ( 0 , I ) .
Revised: N ( 0 , I ) .

Original: This disconnects the encoder from the decoder, making them approximately independent, i.e. p ( x | z )
Revised: This disconnects the encoder from the decoder, making them approximately independent, i.e. p ( x | z )

Original: ≈ p ( x ) .
Revised: ≈ p ( x ) .

Original: The failure mode is often observed when the decoder architecture is overly expressive, i.e. with autoregressive models, or when the likelihood p ( x ) is easy to estimate.
Revised: The failure mode is often observed when the decoder architecture is overly expressive, i.e. with autoregressive models, or when the likelihood p ( x ) is easy to estimate.

Original: Approaches that alleviate this problem rely on modifying the ELBO training objective (Bowman et al., 2016; Kingma et al., 2016) or restricting the decoder structure (Dieng et al., 2019; Maaløe et al., 2019).
Revised: Approaches that alleviate this problem rely on modifying the ELBO training objective (Bowman et al., 2016; Kingma et al., 2016) or restricting the decoder structure (Dieng et al., 2019; Maaløe et al., 2019).

Original: However, these approaches come with various drawbacks, including optimization issues (Lucas et al., 2019).
Revised: However, these approaches come with various drawbacks, including optimization issues (Lucas et al., 2019).

Original: Another approach to estimate latent variables are normalizing ﬂows which describe a sequence of invertible mappings by iteratively applying the change of variables rule (Dinh et al., 2017b).
Revised: Another approach to estimate latent variables are normalizing ﬂows which describe a sequence of invertible mappings by iteratively applying the change of variables rule (Dinh et al., 2017b).

Original: Unlike
Revised: Unlike

Original: VAEs, ﬂow based latent variable models allow for a direct optimization of the likelihood (Dinh et al., 2017b).
Revised: VAEs, ﬂow based latent variable models allow for a direct optimization of the likelihood (Dinh et al., 2017b).

Original: Most normalizing ﬂow models rely on a fast and reliable calculation of the determinant of the Jacobian of the outputs with respect to the inputs, which constrains the architectural design and limits the capacity of the network (Tabak et al., 2010; Tabak and Turner, 2013;
Revised: Most normalizing ﬂow models rely on a fast and reliable calculation of the determinant of the Jacobian of the outputs with respect to the inputs, which constrains the architectural design and limits the capacity of the network (Tabak et al., 2010; Tabak and Turner, 2013;

Original: Dinh et al., 2017b).
Revised: Dinh et al., 2017b).

Original: Thus, competitive ﬂows require very deep architectures in practice (Kingma and Dhariwal, 2018).
Revised: Thus, competitive ﬂows require very deep architectures in practice (Kingma and Dhariwal, 2018).

Original: Furthermore, ﬂows are not directly suited for a scenario where the observation space is higher dimensional than the generating latent factors, dim ( z ) < dim ( x ) , as the computation of the determinant requires a square Jacobian matrix.
Revised: Furthermore, ﬂows are not directly suited for a scenario where the observation space is higher dimensional than the generating latent factors, dim ( z ) < dim ( x ) , as the computation of the determinant requires a square Jacobian matrix.

Original: We tried setting dim ( z )
Revised: We tried setting dim ( z )

Original: = dim ( x ) > dim ( z ∗ ) , but observed instability while optimizing the objective deﬁned below.
Revised: = dim ( x ) > dim ( z ∗ ) , but observed instability while optimizing the objective deﬁned below.

Original: It is straightforward to derive a ﬂow-based objective based on the assumptions in Eq.
Revised: It is straightforward to derive a ﬂow-based objective based on the assumptions in Eq.

Original: We consider a normalizing ﬂow with with K blocks f ( x ) =
Revised: We consider a normalizing ﬂow with with K blocks f ( x ) =

Original: f K ◦ ... ◦
Revised: f K ◦ ... ◦

Original: f 1 : x
Revised: f 1 : x

Original: (cid:55)→ z .
Revised: (cid:55)→ z .

Original: The coupling blocks can refer to nonlinear mixing similar to Kingma and Dhariwal (2018), or in the linear case ( K = 1 ) to an invertible de-mixing matrix.
Revised: The coupling blocks can refer to nonlinear mixing similar to Kingma and Dhariwal (2018), or in the linear case ( K = 1 ) to an invertible de-mixing matrix.

Original: This leads to the following estimation of the likelihoodp
Revised: This leads to the following estimation of the likelihood

Original: [Equation]
Revised: [Equation]

Original: Note that p ( f ( x t − 1 )) is Gaussian and p ( f ( x t )
Revised: Note that p ( f ( x t − 1 )) is Gaussian and p ( f ( x t )

Original: | f
Revised: | f

Original: ( x t − 1 )) is a Laplacian, similar to Eq.
Revised: ( x t − 1 )) is a Laplacian, similar to Eq.

Original: During optimization we take the − log of both sides and minimize w.r.t.
Revised: During optimization we take the − log of both sides and minimize w.r.t.

Original: the parameters of f .
Revised: the parameters of f .

Original: We refer to this estimator as SlowFlow .
Revised: We refer to this estimator as SlowFlow .

Original: Our SlowFlow model is very similar to the ﬂow described in (Pineau et al., 2020), who use a Gaussian transition prior and therefore would have weaker identiﬁability guarantees.
Revised: Our SlowFlow model is very similar to the ﬂow described in (Pineau et al., 2020), who use a Gaussian transition prior and therefore would have weaker identiﬁability guarantees.

Original: Next, we compare SlowFlow and SlowVAE in the context of disentanglement.
Revised: Next, we compare SlowFlow and SlowVAE in the context of disentanglement.

Original: To demonstrate the posterior collapse in VAEs, we generate data points ( x t , x t − 1 ) according to
Revised: To demonstrate the posterior collapse in VAEs, we generate data points ( x t , x t − 1 ) according to

Original: Eq.
Revised: Eq.

Original: (2) with a two dimensional latent space dim ( z ∗ ) = 2 .
Revised: (2) with a two dimensional latent space dim ( z ∗ ) = 2 .

Original: We consider a trivial linear mixing of x ∗ =
Revised: We consider a trivial linear mixing of x ∗ =

Original: W ∗ z ∗ =
Revised: W ∗ z ∗ =

Original: g ∗ ( z ∗ ) with
Revised: g ∗ ( z ∗ ) with

Original: [Equation]
Revised: [Equation]

Original: and κ ∈ [0 . 1 , 1] .
Revised: and κ ∈ [0 . 1 , 1] .

Original: As can be seen by looking at the σ and µ outputs of the encoder in Fig 6a, for κ < 0 .
Revised: As can be seen by looking at the σ and µ outputs of the encoder in Fig 6a, for κ < 0 .

Original: 4 , the encoder for the minor axis collapses to the prior.
Revised: 4 , the encoder for the minor axis collapses to the prior.

Original: The decoder then tries to minimize the reconstruction loss by solely covering the ﬁrst principal component of the data, which is also described in Rolinek et al.
Revised: The decoder then tries to minimize the reconstruction loss by solely covering the ﬁrst principal component of the data, which is also described in Rolinek et al.

Original: Despite the collapse and decrease in MCC, the SlowVAE loss from Eq.
Revised: Despite the collapse and decrease in MCC, the SlowVAE loss from Eq.

Original: (4) still improves during training.
Revised: (4) still improves during training.

Original: On the other hand, a simple linear SlowFlow model f ( x ) =
Revised: On the other hand, a simple linear SlowFlow model f ( x ) =

Original: W x , which directly optimizes the likelihood, recovers the latents consistently as seen by the
Revised: W x , which directly optimizes the likelihood, recovers the latents consistently as seen by the

Original: MCC measure (Fig 6b).
Revised: MCC measure (Fig 6b).

Original: To show the strength of the VAE model we increase the complexity of the data-distribution by using a non-linear expanding decoder such that dim ( x ) (cid:29) dim ( z ∗ ) .
Revised: To show the strength of the VAE model we increase the complexity of the data-distribution by using a non-linear expanding decoder such that dim ( x ) (cid:29) dim ( z ∗ ) .

Original: In Fig.
Revised: In Fig.

Original: 7 we observe that increasing the input dimensionality is sufﬁcient for SlowVAE to ﬁnd the corresponding latents and achieve high
Revised: 7 we observe that increasing the input dimensionality is sufﬁcient for SlowVAE to ﬁnd the corresponding latents and achieve high

Original: MCC with low loss.
Revised: MCC with low loss.

Original: z ∗ ∆ z ∗ x ∗ z ∆ z x µ σ 34.

Original: MCC loss

Original: [Figure]

Original: VAEs prefer data dimensions to be greater than latent dimensions.
Revised: VAEs prefer data dimensions to be greater than latent dimensions.

Original: Individual subplots are as described in Fig.
Revised: Individual subplots are as described in Fig.

Original: For all data in this experiment we used a 20-dimensional latent space, dim ( z ∗ ) = 20 .
Revised: For all data in this experiment we used a 20-dimensional latent space, dim ( z ∗ ) = 20 .

Original: Each row corresponds to the dimensionality of the x ∗ , with values of 20 , 200 , and 2000 .
Revised: Each row corresponds to the dimensionality of the x ∗ , with values of 20 , 200 , and 2000 .

Original: The ﬁrst two dimensions of z ∗ are plotted as well as the two dimensions of z with the highest corresponding mean correlation coefﬁcient (MCC).
Revised: The ﬁrst two dimensions of z ∗ are plotted as well as the two dimensions of z with the highest corresponding mean correlation coefﬁcient (MCC).

Original: The x ∗ and x data are projected onto their ﬁrst two principal component axes before plotting.
Revised: The x ∗ and x data are projected onto their ﬁrst two principal component axes before plotting.

Original: A two-layer mixing matrix was used to transform data from Z gt to
Revised: A two-layer mixing matrix was used to transform data from Z gt to

Original: X gt .
Revised: X gt .

Original: As one increases the data dimensionality, the SlowVAE network performs increasingly better in terms of MCC, although worse in terms of total training loss.
Revised: As one increases the data dimensionality, the SlowVAE network performs increasingly better in terms of MCC, although worse in terms of total training loss.

Original: Each estimation method is practically useful in different experimental settings.
Revised: Each estimation method is practically useful in different experimental settings.

Original: In the case when the mixing operation is trivially deﬁned (Eq.
Revised: In the case when the mixing operation is trivially deﬁned (Eq.

Original: (24), or when the number of dimensions in z ∗ match those in x ∗ ), the VAE estimator tends to learn a pathological solution.
Revised: (24), or when the number of dimensions in z ∗ match those in x ∗ ), the VAE estimator tends to learn a pathological solution.

Original: On the other hand, the normalizing ﬂow estimator does not scale well to high dimensional data due to the requirement of computing the network Jacobian.
Revised: On the other hand, the normalizing ﬂow estimator does not scale well to high dimensional data due to the requirement of computing the network Jacobian.

Original: Additionally, the framework for constructing normalizing ﬂow estimators assumes the latent dimensionality is equal to the data dimensionality to allow for an invertible transform.
Revised: Additionally, the framework for constructing normalizing ﬂow estimators assumes the latent dimensionality is equal to the data dimensionality to allow for an invertible transform.

Original: Together these results lead us to choose an estimator based on the nature of the problem.
Revised: Together these results lead us to choose an estimator based on the nature of the problem.

Original: For our contributed datasets and the DisLib experiments we adopt the VAE framework.
Revised: For our contributed datasets and the DisLib experiments we adopt the VAE framework.

Original: However, if one aims to perform simpliﬁed experiments such as those typically conducted in the nonlinear ICA literature, it will often make practical sense to switch to a ﬂow-based estimator.
Revised: However, if one aims to perform simpliﬁed experiments such as those typically conducted in the nonlinear ICA literature, it will often make practical sense to switch to a ﬂow-based estimator.

Original: D D ISENTANGLEMENT M ETRICS
Revised: C D ISENTANGLEMENT M ETRICS

Original: Several recent studies have brought to light shortcomings in a number of proposed disentanglement metrics (Kim and Mnih, 2018; Eastwood and Williams, 2018;
Revised: Several recent studies have brought to light shortcomings in a number of proposed disentanglement metrics (Kim and Mnih, 2018; Eastwood and Williams, 2018;

Original: Chen et al., 2018; Higgins et al., 2018;
Revised: Chen et al., 2018; Higgins et al., 2018;

Original: Mathieu et al., 2019), many of which have been compiled in the DisLib benchmark.
Revised: Mathieu et al., 2019), many of which have been compiled in the DisLib benchmark.

Original: In addition to the concerns they raise, it is important to note
Revised: In addition to the concerns they raise, it is important to note

Original: that none of the supervised metrics implemented in
Revised: that none of the supervised metrics implemented in

Original: DisLib allow for continuous ground-truth factors, which is necessary for evaluating with the Natural
Revised: DisLib allow for continuous ground-truth factors, which is necessary for evaluating with the Natural

Original: Sprites and KITTI Masks datasets, as factors such as position and scale are effectively continuous in reality.
Revised: Sprites and KITTI Masks datasets, as factors such as position and scale are effectively continuous in reality.

Original: To rectify this issue without introducing novel metrics, we include the Mean Correlation
Revised: To rectify this issue without introducing novel metrics, we include the Mean Correlation

Original: Coefﬁcient (MCC) in our evaluations, using the implementation of Hyvärinen and Morioka (2016), which is described below.
Revised: Coefﬁcient (MCC) in our evaluations, using the implementation of Hyvärinen and Morioka (2016), which is described below.

Original: We measure all metrics presented below between 10 , 000 samples of latent factors z and the corresponding encoded means of our model µ ( g ∗ ( z )) .
Revised: We measure all metrics presented below between 10 , 000 samples of latent factors z and the corresponding encoded means of our model µ ( g ∗ ( z )) .

Original: We increase this sample size to 100 , 000 for
Revised: We increase this sample size to 100 , 000 for

Original: Modularity and MIG to stabilize the entropy estimates.
Revised: Modularity and MIG to stabilize the entropy estimates.

Original: D. M EAN C ORRELATION C OEFFICIENT
Revised: C. M EAN C ORRELATION C OEFFICIENT

Original: In addition to the DisLib metrics, we also compute the Mean Correlation Coefﬁcient (MCC) in order to perform quantitative evaluation with continuous variables.
Revised: In addition to the DisLib metrics, we also compute the Mean Correlation Coefﬁcient (MCC) in order to perform quantitative evaluation with continuous variables.

Original: Because of Theorem 1, perfect disentanglement in the noiseless case should always lead to a correlation coefﬁcient of 1 or − 1 , although note that we report 100 times the absolute value of the correlation coefﬁcient.
Revised: Because of Theorem 1, perfect disentanglement in the noiseless case should always lead to a correlation coefﬁcient of 1 or − 1 , although note that we report 100 times the absolute value of the correlation coefﬁcient.

Original: In our experiments, MCC is used without modiﬁcation from the authors’ open-sourced code (Morioka, 2018).
Revised: In our experiments, MCC is used without modiﬁcation from the authors’ open-sourced code (Morioka, 2018).

Original: The method ﬁrst measures correlation between the ground-truth factors and the encoded latent variables.
Revised: The method ﬁrst measures correlation between the ground-truth factors and the encoded latent variables.

Original: The initial correlation matrix is then used to match each latent unit with a preferred groundtruth factor.
Revised: The initial correlation matrix is then used to match each latent unit with a preferred groundtruth factor.

Original: This is an assignment problem that can be solved in polynomial time via the Munkres algorithm, as described in the code release from Morioka (2018).
Revised: This is an assignment problem that can be solved in polynomial time via the Munkres algorithm, as described in the code release from Morioka (2018).

Original: After solving the assignment problem, the correlation coefﬁcients are computed again for the vector of ground-truth factors and the resulting permuted vector of latent encodings, where the output is a matrix of correlation coefﬁcients with D columns for each ground-truth factor and D (cid:48) rows for each latent variable.
Revised: After solving the assignment problem, the correlation coefﬁcients are computed again for the vector of ground-truth factors and the resulting permuted vector of latent encodings, where the output is a matrix of correlation coefﬁcients with D columns for each ground-truth factor and D (cid:48) rows for each latent variable.

Original: We use the (absolute value of the) Spearman coefﬁcient as our correlation measure which assumes a monotonic relationship between the ground-truth factors and latent encodings but tolerates deviations from a strictly linear correspondence.
Revised: We use the (absolute value of the) Spearman coefﬁcient as our correlation measure which assumes a monotonic relationship between the ground-truth factors and latent encodings but tolerates deviations from a strictly linear correspondence.

Original: In the existing implementation for MCC, the ground truth factors, latent encodings, and mixed signal inputs are assumed to have the same dimensionality, i.e. D = D (cid:48)
Revised: In the existing implementation for MCC, the ground truth factors, latent encodings, and mixed signal inputs are assumed to have the same dimensionality, i.e. D = D (cid:48)

Original: =
Revised: =

Original: N .
Revised: N .

Original: However, in our case, the ground-truth generating factors are much lower dimensional than the signal, N (cid:28) D , and the latent encoding is higher dimensional than the ground-truth factors D (cid:48)
Revised: However, in our case, the ground-truth generating factors are much lower dimensional than the signal, N (cid:28) D , and the latent encoding is higher dimensional than the ground-truth factors D (cid:48)

Original: > D (see Appendix F.1 for details).
Revised: > D (see Appendix E for details).

Original: To resolve this discrepancy, we add D (cid:48) − D standard Gaussian noise channels to the ground-truth factors.
Revised: To resolve this discrepancy, we add D (cid:48) − D standard Gaussian noise channels to the ground-truth factors.

Original: To compute the MCC score, we take the mean of the absolute value of the upper diagonal of the correlation matrix.
Revised: To compute the MCC score, we take the mean of the absolute value of the upper diagonal of the correlation matrix.

Original: The upper diagonal is the diagonal of the square matrix of D ground-truth factors by the top D most correlated latent dimensions after sorting.
Revised: The upper diagonal is the diagonal of the square matrix of D ground-truth factors by the top D most correlated latent dimensions after sorting.

Original: In this way, we obtain an MCC estimate which averages only over the D correlation coefﬁcients of the D ground truth factors with their corresponding best matching latent factors.
Revised: In this way, we obtain an MCC estimate which averages only over the D correlation coefﬁcients of the D ground truth factors with their corresponding best matching latent factors.

Original: D. D IS L IB M ETRICS BetaVAE (Higgins et al., 2017)
Revised: C. D IS L IB M ETRICS BetaVAE (Higgins et al., 2017)

Original: The BetaVAE metric uses a biased estimator with tunable hyperparameters, although we follow the convention established in (Locatello et al., 2018) of using the scikit-learn defaults.
Revised: The BetaVAE metric uses a biased estimator with tunable hyperparameters, although we follow the convention established in (Locatello et al., 2018) of using the scikit-learn defaults.

Original: For a sample in a batch, a pair of images, ( x 1 , x 2 ) , is generated by ﬁxing the value of one of the data generative factors while uniformly sampling the rest.
Revised: For a sample in a batch, a pair of images, ( x 1 , x 2 ) , is generated by ﬁxing the value of one of the data generative factors while uniformly sampling the rest.

Original: The absolute value of the difference between the latent codes produced from the image pairs is then taken, z diff = | z 1 − z 2 | .
Revised: The absolute value of the difference between the latent codes produced from the image pairs is then taken, z diff = | z 1 − z 2 | .

Original: A logistic classiﬁer is ﬁt with batches of z diff variables and the corresponding index of the ﬁxed ground-truth factor serves as the label.
Revised: A logistic classiﬁer is ﬁt with batches of z diff variables and the corresponding index of the ﬁxed ground-truth factor serves as the label.

Original: Once the classiﬁer is trained, the metric itself is the mean classiﬁer accuracy on a batch of held-out test data.
Revised: Once the classiﬁer is trained, the metric itself is the mean classiﬁer accuracy on a batch of held-out test data.

Original: The training minimizes the following loss:
Revised: The training minimizes the following loss:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where w and c are the learnable weight matrix and bias, respectively, and y is the index of the ﬁxed ground-truth factor for the batch.
Revised: where w and c are the learnable weight matrix and bias, respectively, and y is the index of the ﬁxed ground-truth factor for the batch.

Original: The network is trained using the lbfgs optimizer (Byrd et al., 1995), which is implemented via the scikit-learn Python package (Pedregosa et al., 2011) in the Disentanglement Library (DisLib, Locatello et al., 2018).
Revised: The network is trained using the lbfgs optimizer (Byrd et al., 1995), which is implemented via the scikit-learn Python package (Pedregosa et al., 2011) in the Disentanglement Library (DisLib, Locatello et al., 2018).

Original: In the original work, the authors argue that their metric improves over a correlation metric such as the mean correlation coefﬁcient by additionally measuring interpretability.
Revised: In the original work, the authors argue that their metric improves over a correlation metric such as the mean correlation coefﬁcient by additionally measuring interpretability.

Original: However, the linear operation of z T diff ,i w + c can perform demixing, which means the measure gives no direct indication of identiﬁability and thus does not guarantee that the latent encodings are interpretable, especially in the case of dependent factors.
Revised: However, the linear operation of z T diff ,i w + c can perform demixing, which means the measure gives no direct indication of identiﬁability and thus does not guarantee that the latent encodings are interpretable, especially in the case of dependent factors.

Original: Additionally, as noted by Kim and Mnih (2018), BetaVAE can report perfect accuracy when all but one of the ground-truth factors are disentangled, since the classiﬁer can trivially attribute the remaining factor to the remaining latents.
Revised: Additionally, as noted by Kim and Mnih (2018), BetaVAE can report perfect accuracy when all but one of the ground-truth factors are disentangled, since the classiﬁer can trivially attribute the remaining factor to the remaining latents.

Original: FactorVAE (Kim and Mnih, 2018)
Revised: FactorVAE (Kim and Mnih, 2018)

Original: For the FactorVAE metric, the variance of the latent encodings is computed for a large (10,000 in
Revised: For the FactorVAE metric, the variance of the latent encodings is computed for a large (10,000 in

Original: DisLib) batch of data where all factors could possibly be changing.
Revised: DisLib) batch of data where all factors could possibly be changing.

Original: Latent dimensions with variance below some threshold ( 0 . 05 in DisLib) are rejected and not considered further.
Revised: Latent dimensions with variance below some threshold ( 0 . 05 in DisLib) are rejected and not considered further.

Original: Next, the encoding variance is computed again on a smaller batch ( 64 in DisLib) of data where one factor is ﬁxed during sampling.
Revised: Next, the encoding variance is computed again on a smaller batch ( 64 in DisLib) of data where one factor is ﬁxed during sampling.

Original: The quotient of these two quantities (with the larger batch variance as the denominator) is then taken to obtain a normalized variance estimate per latent factor.
Revised: The quotient of these two quantities (with the larger batch variance as the denominator) is then taken to obtain a normalized variance estimate per latent factor.

Original: Finally, a majority-vote classiﬁer is trained to predict the index of the ground-truth factor with the latent unit that has the lowest normalized variance.
Revised: Finally, a majority-vote classiﬁer is trained to predict the index of the ground-truth factor with the latent unit that has the lowest normalized variance.

Original: The FactorVAE score is the classiﬁcation accuracy for a batch of held-out data.
Revised: The FactorVAE score is the classiﬁcation accuracy for a batch of held-out data.

Original: Mutual Information Gap (Chen et al., 2018)
Revised: Mutual Information Gap (Chen et al., 2018)

Original: The Mutual Information Gap (MIG) metric was introduced as an alternative to the classiﬁer-based metrics.
Revised: The Mutual Information Gap (MIG) metric was introduced as an alternative to the classiﬁer-based metrics.

Original: It provides a normalized measure of the mean difference in mutual information between each ground truth factor and the two latent codes that have the highest mutual information with the given ground truth factor.
Revised: It provides a normalized measure of the mean difference in mutual information between each ground truth factor and the two latent codes that have the highest mutual information with the given ground truth factor.

Original: As it is implemented in DisLib, MIG measures entropy by discretizing the model’s latent code using a histogram with 20 bins equally spaced between the representation minimum and maximum.
Revised: As it is implemented in DisLib, MIG measures entropy by discretizing the model’s latent code using a histogram with 20 bins equally spaced between the representation minimum and maximum.

Original: It then computes the discrete mutual information between the groundtruth values and the discretized latents using the scikit-learn metrics.mutual_info_score function (Pedregosa et al., 2011).
Revised: It then computes the discrete mutual information between the groundtruth values and the discretized latents using the scikit-learn metrics .mutual_info_score function (Pedregosa et al., 2011).

Original: For the normalization it divides this difference by the entropy of the discretized ground truth factors.
Revised: For the normalization it divides this difference by the entropy of the discretized ground truth factors.

Original: Modularity (Ridgeway and Mozer, 2018)
Revised: Modularity (Ridgeway and Mozer, 2018)

Original: Ridgeway and Mozer (2018) measure disentanglement in terms of three factors: modularity, compactness, and explicitness.
Revised: Ridgeway and Mozer (2018) measure disentanglement in terms of three factors: modularity, compactness, and explicitness.

Original: For modularity, they ﬁrst measure the mutual information between the discretized latents and ground-truth factors using the same histogram procedure that was used for the MIG, resulting in a matrix, M ∈ R D (cid:48)
Revised: For modularity, they ﬁrst measure the mutual information between the discretized latents and ground-truth factors using the same histogram procedure that was used for the MIG, resulting in a matrix, M ∈ R D (cid:48)

Original: × D with entries for each mutual information pair.
Revised: × D with entries for each mutual information pair.

Original: Their measure of modularity is then
Revised: Their measure of modularity is then

Original: [Equation]
Revised: [Equation]

Original: where max ( M 2 i ) returns the maximum of the vector of squared mutual information measurements between ground truth i and each latent factor.
Revised: where max ( M 2 i ) returns the maximum of the vector of squared mutual information measurements between ground truth i and each latent factor.

Original: Additionally, Θ is a selection function that returns zero for any i where max ( M 2 i ) = 0 and otherwise acts as the identity function.
Revised: Additionally, Θ is a selection function that returns zero for any i where max ( M 2 i ) = 0 and otherwise acts as the identity function.

Original: DCI Disentanglement (Eastwood and Williams, 2018)
Revised: DCI Disentanglement (Eastwood and Williams, 2018)

Original: The DCI scores measure disentanglement, completeness, and informativeness, which have intuitive correspondence to the modularity, compactness, and explicitness of (Ridgeway and Mozer, 2018), respectively.
Revised: The DCI scores measure disentanglement, completeness, and informativeness, which have intuitive correspondence to the modularity, compactness, and explicitness of (Ridgeway and Mozer, 2018), respectively.

Original: To measure DCI Disentanglement, D regressors are trained to predict each ground truth factor state given the latent encoding.
Revised: To measure DCI Disentanglement, D regressors are trained to predict each ground truth factor state given the latent encoding.

Original: The DisLib implementation uses the ensemble.
Revised: The DisLib implementation uses the ensemble.

Original: GradientBoostingClassifier function from scikit-learn with default parameters, which trains D gradient boosted logistic regression tree classiﬁers.
Revised: GradientBoostingClassifier function from scikit-learn with default parameters, which trains D gradient boosted logistic regression tree classiﬁers.

Original: Importance is assigned to each latent factor using the built-in feature_importance_ property of the classiﬁer, which computes the normalized total reduction of the classiﬁer criterion loss contributed by each latent.
Revised: Importance is assigned to each latent factor using the built-in feature_importance_ property of the classiﬁer, which computes the normalized total reduction of the classiﬁer criterion loss contributed by each latent.

Original: Disentanglement is then measured as
Revised: Disentanglement is then measured as

Original: [Equation]
Revised: [Equation]

Original: 27) where H is the entropy computed with the stats.entropy function from scikit-learn ,I ∈
Revised: [Equation] where H is the entropy computed with the stats.entropy function from scikit-learn ,

Original:  R D × D [Equation] is a matrix of the absolute value of the feature importance between each factor and each ground truth, and ˜ I is a normalized version of the matrix
Revised: I ∈ R D × D (cid:48) is a matrix of the absolute value of the feature importance between each factor and each ground truth, and ˜ I is a normalized version of the matrix

Original: [Equation]
Revised: [Equation]

Original: SAP Score (Kumar etal., 2018)
Revised: SAP Score (Kumar et

Revised: al

Original: To compute the SAP score, Kumar et al.
Revised: To compute the SAP score, Kumar et al.

Original: ﬁrst train a linear support vector classiﬁer with squared hinge loss and L 2 penalty to predict each ground truth factor from each latent variable.
Revised: ﬁrst train a linear support vector classiﬁer with squared hinge loss and L 2 penalty to predict each ground truth factor from each latent variable.

Original: In DisLib this is implemented with the svm.
Revised: In DisLib this is implemented with the svm.

Original: LinearSVC function with default parameters from scikit-learn .
Revised: LinearSVC function with default parameters from scikit-learn .

Original: They construct a score matrix S ∈ R D (cid:48)
Revised: They construct a score matrix S ∈ R D (cid:48)

Original: × D , where each entry in the matrix is the batch-mean classiﬁer accuracy for predicting each ground truth given each individual latent encoding.
Revised: × D , where each entry in the matrix is the batch-mean classiﬁer accuracy for predicting each ground truth given each individual latent encoding.

Original: For each generative factor, they compute the difference between the top two most predictive latent dimensions, which are the two highest scores in a given column of S .
Revised: For each generative factor, they compute the difference between the top two most predictive latent dimensions, which are the two highest scores in a given column of S .

Original: The mean (across ground-truth factors) of these differences is the SAP score.
Revised: The mean (across ground-truth factors) of these differences is the SAP score.

Original: E N ATURAL D ATASETS
Revised: D N ATURAL D ATASETS

Original: We introduce several datasets to investigate disentanglement in more natural scenarios.
Revised: We introduce several datasets to investigate disentanglement in more natural scenarios.

Original: Here, we provide an overview on the motivation and design of each dataset.
Revised: Here, we provide an overview on the motivation and design of each dataset.

Revised: We have chosen to work with pairs of inputs as minimal sequences because we are interested in the ﬁrst temporal derivative, more speciﬁcally in the sparsity of the transitions between pairs of images.

Original: In Appendix F.2 we give speciﬁc implementation details.
Revised: Other methods that look at the second temporal derivative, such as work from Hénaff et al. (2019) on straightening, would require triplets as minimal sequences. Extending our approach beyond this minimal requirement would be simple in terms of the resulting ELBO (which would still factorise like in Eq. 4 because of the Markov property).

Revised: The only additional complexity would be in the data and loss handling.

Original: An issue with evaluating disentanglement on natural datasets is the fact that the existing disentanglement metrics require knowledge of the underlying generative process of the given data.
Revised: An issue with evaluating disentanglement on natural datasets is the fact that the existing disentanglement metrics require knowledge of the underlying generative process of the given data.

Original: Although we can observe that the world is composed of distinct entities that vary according to rules imposed by physics, we are unable to determine the appropriate “factors” that generate such scenes.
Revised: Although we can observe that the world is composed of distinct entities that vary according to rules imposed by physics, we are unable to determine the appropriate “factors” that generate such scenes.

Original: To mitigate this problem, we compile object measurements by calculating the x and y coordinates of the center of mass as well as the area of object masks in natural video frames.
Revised: To mitigate this problem, we compile object measurements by calculating the x and y coordinates of the center of mass as well as the area of object masks in natural video frames.

Original: We use these measurements to a) inform new disentanglement benchmarks with natural transitions that have similar complexity to existing benchmarks (Natural Sprites) and b) evaluate the ability of algorithms to decode intrinsic object properties (KITTI Masks).
Revised: We use these measurements to a) inform new disentanglement benchmarks with natural transitions that have similar complexity to existing benchmarks (Natural Sprites) and b) evaluate the ability of algorithms to decode intrinsic object properties (KITTI Masks).

Original: We additionally propose a simple extension to the existing DisLib datasets in the form of collecting images into pairs that exhibit sparse (i.e. Laplace) transition probabilities.
Revised: We additionally propose a simple extension to the existing DisLib datasets in the form of collecting images into pairs that exhibit sparse (i.e. Laplace) transition probabilities.

Original: E. L APLACE T RANSITIONS 
Revised: D. U NIFORM T RANSITIONS (UNI)

Revised: The UNI extension is based on the description given by Locatello et al. (2020), where the number of changing factors is determined using draws from a uniform distribution. The key differences between our implementation and theirs is: (i) their code 3 randomly (with 50% probability) sets k = 1 even in the k =

Revised: Rnd setting, and (ii) we ensure that exactly k factors change.

Revised: Though we consider these discrepancies minor, we nonetheless label all results reported directly from Locatello et al.

Revised: (2020) with “LOC”, as opposed to “UNI”, for clarity.

Revised: D. L APLACE T RANSITIONS (LAP)

Original: For each of the datasets in DisLib, we collect pairs of images.
Revised: For each of the datasets in DisLib, we collect pairs of images.

Original: For each ground-truth factor, the ﬁrst value in the pair is chosen from a uniform distribution across all possible values in latent space, while the second is chosen by weighting nearby values in latent space using Laplace distributed probabilities (see Eq. 2).
Revised: For each ground-truth factor, the ﬁrst value in the pair is chosen from a uniform distribution across all possible values in latent space, while the second is chosen by weighting nearby values in latent space using Laplace distributed probabilities (see Eq. 2).

Original: We reject samples that would push a factor outside of the preset range provided by the dataset.
Revised: We reject samples that would push a factor outside of the preset range provided by the dataset.

Original: We call this the LAP DisLib extension.
Revised: We call this the LAP DisLib extension.

Original: Although the sparse prior indicates that any individual factor is more likely to remain constant, the number of factors that change in a given transition is still typically greater than one.
Revised: Although the sparse prior indicates that any individual factor is more likely to remain constant, the number of factors that change in a given transition is still typically greater than one.

Original: To show this in Fig.
Revised: To show this in Fig.

Original: 8, we sampled 10, transitions from each DisLib dataset with LAP transitions and computed the number of factors that had changed within a pair.
Revised: 8, we sampled 10, transitions from each DisLib dataset with LAP transitions and computed the number of factors that had changed within a pair.

Original: This extension of the DisLib datasets provides a bridge from i.i.d.
Revised: This extension of the DisLib datasets provides a bridge from i.i.d.

Original: data to natural data by explicitly modeling the observed sparse marginal transition distributions.
Revised: data to natural data by explicitly modeling the observed sparse marginal transition distributions.

Revised: When training models on the LAP dataset it is possible to reject samples without transitions (i.e. all factors remain constant) since the pair would not result in any temporal learning signal.

Revised: However, it would arguably be more natural to leave these samples as they would more accurately reﬂect occurrences of stationary objects in real data.

Revised: We report the rejection setting in the main text, but found no signiﬁcant difference between the two settings (see Appendix G).

Revised: This dataset also introduces a hyper-parameter λ that controls the rate of the Laplace sampling distribution, while the location is set by the initial factor value.

Original: E. Y OU T UBE -VOS
Revised: Effectively, when this rate is λ =most of the factors change most of the time, whereas for a rate of λ = 10 most of the factors will not change most of the time. Note that this means λ (inversely) changes the scale, which results in larger or smaller movements, but does not affect the distribution itself. In other words, the sparsity is unchanged, as the sparsity is controlled by the shape α . We ﬁx λ = 1 , which yields multiple changes, thus making this dataset fundamentally different both in spirit and in practice, from the UNI dataset.

Revised: D.

Original:  For the YouTube dataset, we download annotations from the 2019 version of the video instance segmentation (Youtube-VIS) dataset
Revised: [Equation] For the YouTube dataset, we download annotations from the 2019 version of the video instance segmentation (Youtube-VIS) dataset

Original: (Yang et al., 2019) 3 , which is built on top of the video object segmentation (Youtube-VOS) dataset (Xu et al., 2018).
Revised: (Yang et al., 2019) 4 , which is built on top of the video object segmentation (Youtube-VOS) dataset (Xu et al., 2018).

Original: The dataset has multi-object annotations for every ﬁve frames in a 30fps video, which results in a 6fps sampling rate.
Revised: The dataset has multi-object annotations for every ﬁve frames in a 30fps video, which results in a 6fps sampling rate.

Original: The authors state that the temporal correlation between ﬁve consecutive frames is sufﬁciently strong that annotations can be omitted for intermediate frames to reduce the annotation efforts.
Revised: The authors state that the temporal correlation between ﬁve consecutive frames is sufﬁciently strong that annotations can be omitted for intermediate frames to reduce the annotation efforts.

Original: Such a skip-frame annotation strategy enables scaling up the number of videos and objects annotated under the same budget, yielding 131,000 annotations for 2,883 videos, with 4,883 unique video object instances.
Revised: Such a skip-frame annotation strategy enables scaling up the number of videos and objects annotated under the same budget, yielding 131,000 annotations for 2,883 videos, with 4,883 unique video object instances.

Original: Although we do not evaluate against YouTube-VOS in this study, we see it as the logical next step in transitioning to natural data.
Revised: Although we do not evaluate against YouTube-VOS in this study, we see it as the logical next step in transitioning to natural data.

Original: The large scale, lack of environmental constraints, and abundance of object types makes it the most challenging of the datasets considered herein.
Revised: The large scale, lack of environmental constraints, and abundance of object types makes it the most challenging of the datasets considered herein.

Revised: The original image size of the YouTube-VOS dataset is 720 × 1280 .

Revised: In order to preserve the statistics of the transitions, we choose not to directly downsample to 64 × 64 , but instead preserve the aspect ratio by downsampling to 64 × 128 .

Revised: In order to minimize the bias yielded by the extraction method, noting the center bias typically present in human videos, we extract three overlapping, equally spaced 64 × 64 pixel windows with a stride of 32 .

Revised: For each resulting 64 × 64 × T sequence, where T denotes the number of time steps in the sequence, we ﬁlter out all pairs where the given object instance is not present in adjacent frames, resulting in 234,652 pairs.

Original: E. N ATURAL S PRITES
Revised: D. N ATURAL S PRITES

Revised: The benchmark is available at https://zenodo.org/record/3948069 .

Original: Without a deﬁnition of disentanglement that can be applied to unknown data generating processes, we are limited to synthetic datasets with known ground-truth factors.
Revised: Without a metric for disentanglement that can be applied to unknown data generating processes, we are limited to synthetic datasets with known ground-truth factors.

Original: Let us take dSprites (Matthey et al., 2017) as an example.
Revised: Let us take dSprites (Matthey et al., 2017) as an example.

Original: The dataset consists of all combinations of a set of latent factor values, namely,
Revised: The dataset consists of all combinations of a set of latent factor values, namely,

Original: • Color: white  • Shape: square, ellipse, heart • Scale: 6 values linearly spaced in [0 . 5 , 1] • Orientation: 40 values in [0 , 2 π ] • Position X : 32 values in [0 , 1] • Position Y : 32 values in [0 , 1]
Revised: • Color: white [Table] • Shape: square, ellipse, heart • Scale: 6 values linearly spaced in [0 . 5 , 1] • Orientation: 40 values in [0 , 2 π ] • Position X : 32 values in [0 , 1] • Position Y : 32 values in [0 , 1]

Original: Given the limited set of discrete values each factor can take on, all possible samples can be described by a tractable dataset, compiled and released to the public.
Revised: Given the limited set of discrete values each factor can take on, all possible samples can be described by a tractable dataset, compiled and released to the public.

Original: But, in reality, all of these factors should be continuous: a spectrum of possible colors, shapes, scales, orientations, and positions exist.
Revised: But, in reality, all of these factors should be continuous: a spectrum of possible colors, shapes, scales, orientations, and positions exist.

Original: We address this by constructing a dataset that is augmented with natural and continuous ground truth factors, using the mask properties measured from the YouTube dataset described in Appendix F.2.
Revised: We address this by constructing a dataset that is augmented with natural and continuous ground truth factors, using the mask properties measured from the YouTube dataset described in Appendix D.3.

Original: We can choose the complexity of the dataset by discretizing the 234,652 transition pairs of position and scale into an arbitrary number of bins.
Revised: We can choose the complexity of the dataset by discretizing the 234,652 transition pairs of position and scale into an arbitrary number of bins.

Original: In this study, we discretize to match the number of possible object states as dSprites, which we present in Table 4.
Revised: In this study, we discretize to match the number of possible object states as dSprites, which we present in Table 4.

Original: This helps isolate the effect of including natural transitions from the effect of increasing data complexity.
Revised: This helps isolate the effect of including natural transitions from the effect of increasing data complexity.

Original: We produce a pair by ﬁxing the color, shape, and orientation, but updating the position and scale with transitions sampled from the
Revised: We produce a pair by ﬁxing the color, shape, and orientation, but updating the position and scale with transitions sampled from the

Original: YouTube measurements.
Revised: YouTube measurements.

Original: We motivate ﬁxing shape and color by noting that this is consistent with object permanence in the real world.
Revised: We motivate ﬁxing shape and color by noting that this is consistent with object permanence in the real world.

Original: We decided to ﬁx the orientation because we do not currently have a way to approximate it from object masks and we did not want to introduce artiﬁcial transition probabilities.
Revised: We decided to ﬁx the orientation because we do not currently have a way to approximate it from object masks and we did not want to introduce artiﬁcial transition probabilities.

Original: To minimize the effect of extreme outliers, we ﬁlter out 10% of the data by removing frames if the mask area falls below the 5% or above the 95% quantiles, which reduces the number of pairs to 207,794.
Revised: To minimize the effect of extreme outliers, we ﬁlter out 10% of the data by removing frames if the mask area falls below the 5% or above the 95% quantiles, which reduces the number of pairs to 207,794.

Original: Finally, we use the Spriteworld (Watters et al., 2019) renderer to generate the images.
Revised: Finally, we use the Spriteworld (Watters et al., 2019) renderer to generate the images.

Original: Spriteworld allows us to render entirely new sprite objects at the precise position and scale as was measured from YouTube.
Revised: Spriteworld allows us to render entirely new sprite objects at the precise position and scale as was measured from YouTube.

Original: For example, if one would want to apply YouTube-VOS transitions to MPI3D (Gondal et al., 2019), this option is unavailable without the associated renderer.
Revised: For example, if one would want to apply YouTube-VOS transitions to MPI3D (Gondal et al., 2019), this option is unavailable without the associated renderer.

Original: The benchmark is available at https://zenodo.org/record/3948069 .

Original: In relation to the Laplace transitions described in section E.1, this update i) produces pairs that correspond to transitions observed in real data, ii) allows for smooth transitions by deﬁning the data generation process as opposed to being limited by the given collected dataset (e.g. dSprites), and iii) includes complex dependencies among factors that are present in natural data.
Revised: In relation to the Laplace transitions described in section D.2, this update i) produces pairs that correspond to transitions observed in real data, ii) allows for smooth transitions by deﬁning the data generation process as opposed to being limited by the given collected dataset (e.g. dSprites), and iii) includes complex dependencies among factors that are present in natural data.

Original: We generate the data online, thus training the model to ﬁt the underlying distribution as opposed to a sampled ﬁnite dataset.
Revised: We generate the data online, thus training the model to ﬁt the underlying distribution as opposed to a sampled ﬁnite dataset.

Original: However, as noted previously, all supervised metrics aggregated in DisLib are inapplicable to continuous factors, which is problematic as the generating distribution is effectively continuous with respect to a subset of the factors.
Revised: However, as noted previously, all supervised metrics aggregated in DisLib are inapplicable to continuous factors, which is problematic as the generating distribution is effectively continuous with respect to a subset of the factors.

Original: Therefore, we limit our quantitative evaluation to MCC for continuous datasets.
Revised: Therefore, we limit our quantitative evaluation to MCC for continuous datasets.

Original: However, we are able to evaluate disentanglement with the standard metrics on the discretized version.
Revised: However, we are able to evaluate disentanglement with the standard metrics on the discretized version.

Original: E. KITTI MOTS P EDESTRIAN M ASKS 
Revised: D. KITTI MOTS P EDESTRIAN M ASKS (KITTI M ASKS )

Revised: The benchmark is available at https://zenodo.org/record/3931823 .

Original: While Natural Sprites enables evaluation of disentanglement with natural transitions, we note that any disentanglement framework that requires knowledge of the underlying generative factors is unrealistic [Table] [Table] [Figure]
Revised: While Natural Sprites enables evaluation of disentanglement with natural transitions, we note that any disentanglement framework that requires knowledge of the underlying generative factors is unrealistic for real-world data.

Original: corresponds to the area. for real-world data. Measurements such as scale and position correspond to object properties that are ecologically relevant to the observer and can serve as suitable alternatives to the typical generative factors.
Revised: Measurements such as scale and position correspond to object properties that are ecologically relevant to the observer and can serve as suitable alternatives to the typical generative factors.

Original: We directly test this using our KITTI Masks dataset.
Revised: We directly test this using our KITTI Masks dataset.

Original: To create the dataset, we download annotations from the Multi-Object Tracking and Segmentation (MOTS) Evaluation Benchmark (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016),  which is split into KITTI MOTS and MOTSChallenge 4 .
Revised: To create the dataset, we download annotations from the Multi-Object Tracking and Segmentation (MOTS) Evaluation Benchmark (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016), [Table] [Figure] which is split into KITTI MOTS and MOTSChallenge 5 .

Original: Both datasets contain sequences of pedestrians with their positions densely annotated in the time and pixel domains.
Revised: Both datasets contain sequences of pedestrians with their positions densely annotated in the time and pixel domains.

Original: For simplicity, we only consider the instance segmentation masks for pedestrians and do not use the raw data.
Revised: For simplicity, we only consider the instance segmentation masks for pedestrians and do not use the raw data.

Original: The resulting KITTI Masks dataset consists of 2,120 sequences of individual pedestrians with lengths between 2 and 710 frames each, resulting in a total of 84,626 individual frames.
Revised: The resulting KITTI Masks dataset consists of 2,120 sequences of individual pedestrians with lengths between 2 and 710 frames each, resulting in a total of 84,626 individual frames.

Original: As we did with
Revised: As we did with

Original: YouTube-VOS, we estimate ground truth factors by calculating the x and y coordinates of the center of mass of each pedestrian mask in each frame.
Revised: YouTube-VOS, we estimate ground truth factors by calculating the x and y coordinates of the center of mass of each pedestrian mask in each frame.

Original: We deﬁne the object size as the area of the mask, i.e. the total number of pixels.
Revised: We deﬁne the object size as the area of the mask, i.e. the total number of pixels.

Original: We consider the disentanglement performance for different mean time gaps between image pairs in table 2 and Appendix H.3.
Revised: We consider the disentanglement performance for different mean time gaps between image pairs in table 2 and Appendix G.3.

Original: In Appendix F.2.4 we discuss how the variable framerate of the KITTI datasets leads to us reporting the mean time gap between frames for a given image pair.

Original: For samples and the corresponding ground truth factors see Fig. 9.
Revised: For samples and the corresponding ground truth factors see Fig. 9.

Original: The benchmark is available at https://zenodo.org/record/3931823 .

Original: F I MPLEMENTATION D ETAILS F. M ODEL T RAINING AND S ELECTION

Original: We train all models on all datasets provided in DisLib with the UNI and LAP variants.

Original: All models are implemented in PyTorch (Paszke et al., 2019).

Original: To facilitate comparison, the training parameters, e.g. optimizer, batch size, number of training steps, as well as the VAE encoder and decoder architecture are identical to those reported in (Locatello et al., 2018; 2020).

Original: We use this architecture for all datasets, only adjusting the number of input channels (greyscale for dSprites, smallNORB, and KITTI Masks; three color channels for all other datasets).

Original: The model formulation is agnostic to the direction of time.

Original: Therefore, to increase the temporal training signal at a ﬁxed computational cost for each batch of input pairs ( x 0 , x 1 ) , we optimize the model in both directions i.e. optimizing the model objective for both t 0 = 0 , t 1 = 1 as well as t 0 = 1 , t 1 = 0 .

Original: F. D ATASET D ETAILS

Original: We have chosen to work with pairs of inputs as minimal sequences because we are interested in the ﬁrst temporal derivative, more speciﬁcally in the sparsity of the transitions between pairs of images.

Original: Other methods that look at the second temporal derivative, such as work from Hénaff et al.

Original: (2019) on straightening, would require triplets as minimal sequences.

Original: Extending our approach beyond this minimal requirement would be simple in terms of the resulting ELBO (which would still factorise like in Eq.

Original: 4 because of the Markov property).

Original: The only additional complexity would be in the data and loss handling.

Original: We note that both YouTube-VOS (Xu et al., 2018; Yang et al., 2019) and KITTI-MOTS (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) are multi-object datasets, although we consider each unique object (mask) separately.

Original: Multi-object representation learning and disentanglement are highly connected, in fact they have recently begun to be used interchangeably (Wulfmeier et al., 2020).

Original: To brieﬂy comment on possible extensions in this direction, we see no reason why our prior would not be beneﬁcial to multi-object methods such as MONet (Burgess et al., 2019) and IODINE (Greff et al., 2019), or video extensions such as ViMON (Weis et al., 2020) and OP3 (Veerapaneni et al., 2019).

Original: F.2. UNI

Original: The key differences between our implementation of UNI , which we based on the description in the paper (Locatello et al., 2020), and the what was used by Locatello et al.

Original: (2020) is: (i) their coderandomly (with 50% probability) sets k = 1 even in the k =

Original: Rnd setting, and (ii) we ensure that exactly k factors change.

Original: Though we consider this discrepancy minor, we nonetheless label all results from (Locatello et al., 2020) with “LOC”, as opposed to “UNI”, for clarity.

Original: F.2. LAP

Original: When training models on the LAP dataset it is possible to reject samples without transitions (i.e. all factors remain constant) since the pair would not result in any temporal learning signal.

Original: However, it would arguably be more natural to leave these samples as they would more accurately reﬂect occurrences of stationary objects in real data.

Original: We report the rejection setting in the main text, but found no signiﬁcant difference between the two settings (see Appendix H).

Original: This dataset also introduces a hyper-parameter λ that controls the rate of the Laplace sampling distribution, while the location is set by the initial factor value.

Original: Effectively, when this rate is λ =most of the factors change most of the time, whereas for a rate of λ = 10 most of the factors will not change most of the time.

Original: Note that this means λ (inversely) changes the scale, which results in larger or smaller movements, but does not affect the distribution itself.

Original: In other words, the sparsity is unchanged, as the sparsity is controlled by the shape α .

Original: We ﬁx λ = 1 , which yields multiple changes, thus making this dataset fundamentally different both in spirit and in practice, from the UNI dataset modeled after Locatello et al.

Original: F.2. Y OU T UBE -VOS

Original: The original image size of the YouTube-VOS dataset is 720 × 1280 .

Original: In order to preserve the statistics of the transitions, we choose not to directly downsample to 64 × 64 , but instead preserve the aspect ratio by downsampling to 64 × 128 .

Original: In order to minimize the bias yielded by the extraction method, noting the center bias typically present in human videos, we extract three overlapping, equally spaced 64 × 64 pixel windows with a stride of 32 .

Original: For each resulting 64 × 64 × T sequence, where T denotes the number of time steps in the sequence, we ﬁlter out all pairs where the given object instance is not present in adjacent frames, resulting in 234,652 pairs.

Original: F.2. KITTI M ASKS

Original: The original KITTI image sizes are 1080 × 1920 or 480 × 640 resolution for MOTSChallenge and between 370 and 374 pixels tall by 1224 and 1242 pixels wide for KITTI MOTS.
Revised: The original KITTI image sizes are 1080 × 1920 or 480 × 640 resolution for MOTSChallenge and between 370 and 374 pixels tall by 1224 and 1242 pixels wide for KITTI MOTS.

Original: The frame rates of the videos vary from 14 to 30 fps, which can be seen in Table 2 of Milan et al.
Revised: The frame rates of the videos vary from 14 to 30 fps, which can be seen in Table 2 of Milan et al.

Original: We use nearest neighbor down-sampling for each frame such that the height was 64 pixels and the width is set to conserve the aspect ratio.
Revised: We use nearest neighbor down-sampling for each frame such that the height was 64 pixels and the width is set to conserve the aspect ratio.

Original: After down-sampling, we use a horizontal sliding window approach to extract six equally spaced windows of size 64 × 64 (with overlap) for each sequence in both datasets
Revised: After down-sampling, we use a horizontal sliding window approach to extract six equally spaced windows of size 64 × 64 (with overlap) for each sequence in both datasets.

Original: This results in a 64 × 64 × T sequence, where T denotes the number of time steps in the sequence.
Revised: This results in a 64 × 64 × T sequence, where T denotes the number of time steps in the sequence.

Original: Note that here we make reasonable assumptions on horizontal translation and scale invariance of the dataset.
Revised: Note that here we make reasonable assumptions on horizontal translation and scale invariance of the dataset.

Original: We justify the assumed scale invariance by observing that the data is collected from a camera mounted onto a car which has varying distance to pedestrians.
Revised: We justify the assumed scale invariance by observing that the data is collected from a camera mounted onto a car which has varying distance to pedestrians.

Original: To conﬁrm the translation invariance, we performed an ablation study on the number of horizontal images.
Revised: To conﬁrm the translation invariance, we performed an ablation study on the number of horizontal images.

Original: Instead of six horizontal, equally spaced sliding windows, we only use two which leads to differently placed windows.
Revised: Instead of six horizontal, equally spaced sliding windows, we only use two which leads to differently placed windows.

Original: We do not observe signiﬁcant changes in the reported data statistics (e.g. the kurtosis of the ﬁt stays within ± 10% of the previous value for ∆ x transitions).
Revised: We do not observe signiﬁcant changes in the reported data statistics (e.g. the kurtosis of the ﬁt stays within ± 10% of the previous value for ∆ x transitions).

Original: The values of ∆ y and ∆ area do not change signiﬁcantly compared to Table 7.
Revised: The values of ∆ y and ∆ area do not change signiﬁcantly compared to Table 7.

Original: For each resulting 64 × 64 × T sequence, where T denotes the number of time steps in the sequence, we extract all individual pedestrian masks based on their object instance identity and create a new sequence for each pedestrian such that each resulting sequence only contains a single pedestrian.
Revised: For each resulting 64 × 64 × T sequence, where T denotes the number of time steps in the sequence, we extract all individual pedestrian masks based on their object instance identity and create a new sequence for each pedestrian such that each resulting sequence only contains a single pedestrian.

Original: We ignore images with masks that have less than 30 pixels as they are too far away or occluded and were  not recognizable by the authors.
Revised: We ignore images with masks that have less than 30 pixels as they are too far away or occluded and were [Figure] not recognizable by the authors.

Original: We keep all sequences of two or more frames, as the algorithm only requires pairs of frames for training.
Revised: We keep all sequences of two or more frames, as the algorithm only requires pairs of frames for training.

Original: We leave the maximum distance between time frames within a pair, max (∆ frames ) , as a hyperparameter.
Revised: We leave the maximum distance between time frames within a pair, max (∆ frames ) , as a hyperparameter.

Original: For a given max (∆ frames ) , we report the mean change in physical time in seconds (denoted by mean (∆ t ) ).
Revised: For a given max (∆ frames ) , we report the mean change in physical time in seconds (denoted by mean (∆ t ) ).

Original: We test adjacent frames (max (∆ frames ) = 1 ), which corresponds to a mean (∆ t = 0 . 05) and max (∆ frames )
Revised: We test adjacent frames (max (∆ frames ) = 1 ), which corresponds to a mean (∆ t = 0 . 05) and max (∆ frames )

Original: = 5 , which corresponds to a mean (∆ t = 0 . 15) .
Revised: = 5 , which corresponds to a mean (∆ t = 0 . 15) .

Original: This procedure is motivated by the fact that different sequences were recorded with different frame rates and reporting the mean (∆ t ) in seconds allows for a physical interpretation.
Revised: This procedure is motivated by the fact that different sequences were recorded with different frame rates and reporting the mean (∆ t ) in seconds allows for a physical interpretation.

Original: The relationship between max (∆ frames ) and mean (∆ t ) is in Fig.
Revised: The relationship between max (∆ frames ) and mean (∆ t ) is in Fig.

Original: We show results for testing additional values of mean (∆ t ) in Appendix H.3.
Revised: We show results for testing additional values of mean (∆ t ) in Appendix G.3.

Original: During training, we augment the data by applying horizontal and vertical translations of ± 5 pixels and rotations of ± 2 ◦ degree.
Revised: During training, we augment the data by applying horizontal and vertical translations of ± 5 pixels and rotations of ± 2 ◦ degree.

Original: We apply the exact same data augmentation to both images within a pair to not change any transition statistics.
Revised: We apply the exact same data augmentation to both images within a pair to not change any transition statistics.

Revised: We note that both YouTube-VOS (Xu et al., 2018; Yang et al., 2019) and KITTI-MOTS (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) are multi-object datasets, although we consider each unique object (mask) separately.

Revised: Multi-object representation learning and disentanglement are highly connected, in fact they have recently begun to be used interchangeably (Wulfmeier et al., 2020).

Original: G E XTENDED C OMPARISONS AND C ONTROLS G. C OMPARISON TO N ONLINEAR ICA G.1. T HEORETICAL C OMPARISON
Revised: To brieﬂy comment on possible extensions in this direction, we see no reason why our prior would not be beneﬁcial to multi-object methods such as MONet (Burgess et al., 2019) and IODINE (Greff et al., 2019), or video extensions such as ViMON (Weis et al., 2020) and OP3 (Veerapaneni et al., 2019). E M ODEL T RAINING AND S ELECTION We train all models on all datasets provided in DisLib with the UNI and LAP variants. All models are implemented in PyTorch (Paszke et al., 2019).

Revised: To facilitate comparison, the training parameters, e.g. optimizer, batch size, number of training steps, as well as the VAE encoder and decoder architecture are identical to those reported in (Locatello et al., 2018; 2020).

Revised: We use this architecture for all datasets, only adjusting the number of input channels (greyscale for dSprites, smallNORB, and KITTI Masks; three color channels for all other datasets).

Revised: The model formulation is agnostic to the direction of time.

Revised: Therefore, to increase the temporal training signal at a ﬁxed computational cost for each batch of input pairs ( x 0 , x 1 ) , we optimize the model in both directions i.e. optimizing the model objective for both t 0 = 0 , t 1 = 1 as well as t 0 = 1 , t 1 = 0 .

Revised: F E XTENDED C OMPARISONS AND C ONTROLS F. C OMPARISON TO N ONLINEAR ICA F.1. T HEORETICAL C OMPARISON

Original: Nonlinear ICA has recently been advanced signiﬁcantly by several papers from Hyvärinen and colleagues.
Revised: Nonlinear ICA has recently been advanced signiﬁcantly by several papers from Hyvärinen and colleagues.

Original: Of these studies, the two that are most comparable to our work is Hyvärinen and Morioka (2017), which uses an unsupervised contrastive loss for nonlinear demixing and Khemakhem et al.[Figure]
Revised: Of these studies, the two that are most comparable to our work is Hyvärinen and Morioka (2017), which uses an unsupervised contrastive loss for nonlinear demixing and Khemakhem et al.

Original: (2020a), which extends the nonlinear ICA framework to include variational autoencoders (VAEs).
Revised: (2020a), which extends the nonlinear ICA framework to include variational autoencoders (VAEs).

Original: However, our theory covers an important class of transitions relevant for natural data that is not covered by the identiﬁability proofs of either of the aforementioned studies.
Revised: However, our theory covers an important class of transitions relevant for natural data that is not covered by the identiﬁability proofs of either of the aforementioned studies.

Original: As a speciﬁc comparison to the ﬁrst paper, the non-Gaussian autoregressive model that their identiﬁability proof rests upon (Eq. 8 in Hyvärinen and Morioka, 2017) assumes that the second derivative of the innovation probability density function is less than zero to satisfy uniform dependence , which is only met for α > 1 for generalized Laplace transition distributions.
Revised: As a speciﬁc comparison to the ﬁrst paper, the non-Gaussian autoregressive model that their identiﬁability proof rests upon (Eq. 8 in Hyvärinen and Morioka, 2017) assumes that the second derivative of the innovation probability density function is less than zero to satisfy uniform dependence , which is only met for α > 1 for generalized Laplace transition distributions.

Original: While they denote (footnote 3) that Laplace distributions ( α = 1 ) are not covered by their theory, they offer a suggestion for a smooth approximation.
Revised: While they denote (footnote 3) that Laplace distributions ( α = 1 ) are not covered by their theory, they offer a suggestion for a smooth approximation.

Original: However, they do not demonstrate that this approximation is useful in practice, or offer a solution to a general class of sparse distributions for α ≤ 1 .
Revised: However, they do not demonstrate that this approximation is useful in practice, or offer a solution to a general class of sparse distributions for α ≤ 1 .

Original: We chose a generalized Laplacian to ﬁt our data and for our model assumption as it allows for simple parameterization of ﬁts to data (e.g. α = 0 .
Revised: We chose a generalized Laplacian to ﬁt our data and for our model assumption as it allows for simple parameterization of ﬁts to data (e.g. α = 0 .

Original: 5 for natural movie transitions), but is simultaneously quite expressive (Sinz et al., 2009).
Revised: 5 for natural movie transitions), but is simultaneously quite expressive (Sinz et al., 2009).

Original: Though we use α = 1 in practice for our estimation method, we prove identiﬁability up to permutations and sign ﬂips for any α < 2 , covering all sparse distributions under the expressive generalized
Revised: Though we use α = 1 in practice for our estimation method, we prove identiﬁability up to permutations and sign ﬂips for any α < 2 , covering all sparse distributions under the expressive generalized

Original: Laplacian model.
Revised: Laplacian model.

Original: In addition, we assume a Gaussian marginal distribution that allows us to derive a fundamentally stronger proof of identiﬁability – where we identify up to permutation and sign-ﬂips.
Revised: In addition, we assume a Gaussian marginal distribution that allows us to derive a fundamentally stronger proof of identiﬁability – where we identify up to permutation and sign-ﬂips.

Original: Hyvärinen and Morioka (2017) only identify the sources up to arbitrary non-linear element-wise transformations.
Revised: Hyvärinen and Morioka (2017) only identify the sources up to arbitrary non-linear element-wise transformations.

Original: Thus they require a subsequent step of ICA (under the typical assumption that at most one marginal source distribution is Gaussian) to recover the signal up to permutations and sign ﬂips for a class of distributions where it is unclear whether they account for temporal sparsity.
Revised: Thus they require a subsequent step of ICA (under the typical assumption that at most one marginal source distribution is Gaussian) to recover the signal up to permutations and sign ﬂips for a class of distributions where it is unclear whether they account for temporal sparsity.

Original: The work of Khemakhem et al.
Revised: The work of Khemakhem et al.

Original: (2020a) has a couple of differences from our own, most notable of which is the form of the conditional prior, p ( z t | z t − 1 ) .
Revised: (2020a) has a couple of differences from our own, most notable of which is the form of the conditional prior, p ( z t | z t − 1 ) .

Original: They assume that the conditional posterior is part of the exponential family, which does not include Laplacian conditionals.
Revised: They assume that the conditional posterior is part of the exponential family, which does not include Laplacian conditionals.

Original: Though the exponential family contains the Laplace distribution with ﬁxed mean as its member, it does not allow their approach to model sparse transitions.
Revised: Though the exponential family contains the Laplace distribution with ﬁxed mean as its member, it does not allow their approach to model sparse transitions.

Original: They assume that the natural parameters of the exponential family distribution are conditioned on z t − 1 , meaning that only the scale but not the mean of the Laplace prior for z t can be modulated by the previous time step, thus not allowing for sparse transition probabilities.
Revised: They assume that the natural parameters of the exponential family distribution are conditioned on z t − 1 , meaning that only the scale but not the mean of the Laplace prior for z t can be modulated by the previous time step, thus not allowing for sparse transition probabilities.

Original: Additionally, their implementation requires the number of classes (i.e. states of the conditioning variable) to equal the number of stationary segments, which is impractical for the datasets we consider.
Revised: Additionally, their implementation requires the number of classes (i.e. states of the conditioning variable) to equal the number of stationary segments, which is impractical for the datasets we consider.

Original: Thus, we provide a closer match to natural data transitions, with a stronger identiﬁability result.
Revised: Thus, we provide a closer match to natural data transitions, with a stronger identiﬁability result.

Original: We provide validation by performing an extensive evaluation leveraging our contributed datasets as well as the models, metrics, and datasets provided by the Disentanglement Library (DisLib, discussed in section 4).
Revised: We provide validation by performing an extensive evaluation leveraging our contributed datasets as well as the models, metrics, and datasets provided by the Disentanglement Library (DisLib, discussed in section 4).

Original: We consider methods from the disentanglement literature (Locatello et al., 2020) as well as nonlinear ICA (Hyvärinen and Morioka, 2017), that are functionally capable of processing transitions.
Revised: We consider methods from the disentanglement literature (Locatello et al., 2020) as well as nonlinear ICA (Hyvärinen and Morioka, 2017), that are functionally capable of processing transitions.

Original: G.1. E MPIRICAL C OMPARISON
Revised: F.1. E MPIRICAL C OMPARISON

Original: Hyvärinen and Morioka (2017) conducted a simulation where the sources in the nonlinear ICA model come from a linear autoregressive (AR) model with non-Gaussian innovations.
Revised: Hyvärinen and Morioka (2017) conducted a simulation where the sources in the nonlinear ICA model come from a linear autoregressive (AR) model with non-Gaussian innovations.

Original: Speciﬁcally, temporally dependent 20 -dimensional source signals were randomly generated according to log p ( s ( t ) | s ( t − 1)) =
Revised: Speciﬁcally, temporally dependent 20 -dimensional source signals were randomly generated according to log p ( s ( t ) | s ( t − 1)) =

Original: −| s ( t ) − 0 .
Revised: −| s ( t ) − 0 .

Original: 7 s ( t − 1) | .
Revised: 7 s ( t − 1) | .

Original: Though this generative process was noted to not be covered by the theory presented in (Hyvärinen and Morioka, 2017), the authors demonstrated that[Table]
Revised: Though this generative process was noted to not be covered by the theory presented in (Hyvärinen and Morioka, 2017), the authors demonstrated that

Original: PCL could reconstruct the source signals reasonably well even for the nonlinear mixture case.
Revised: PCL could reconstruct the source signals reasonably well even for the nonlinear mixture case.

Original: Given our practical use of a Laplacian conditional, we found it a valuable comparison to evaluate our theory in this artiﬁcial setting. 
Revised: Given our practical use of a Laplacian conditional, we found it a valuable comparison to evaluate our theory in this artiﬁcial setting. [Table]

Original: Given the discussion in Appendix C, we use SlowFlow for these experiments.
Revised: Given the discussion in Appendix B, we use SlowFlow for these experiments.

Original: For computational tractability in demixing highly nonlinear transformations, we consider normalizing ﬂows (Dinh et al., 2017a;b; Kingma and Dhariwal, 2018), namely volume-preserving ﬂows (Sorrenson et al., 2017), as we ﬁnd constraining the Jacobian determinant stabilizes learning.
Revised: For computational tractability in demixing highly nonlinear transformations, we consider normalizing ﬂows (Dinh et al., 2017a;b; Kingma and Dhariwal, 2018), namely volume-preserving ﬂows (Sorrenson et al., 2017), as we ﬁnd constraining the Jacobian determinant stabilizes learning.

Original: To ensure sufﬁcient expressivity, we consider 6 coupling blocks, each containing a 2 -layer MLP with 500 hidden units and ReLU nonlinearities.
Revised: To ensure sufﬁcient expressivity, we consider 6 coupling blocks, each containing a 2 -layer MLP with 500 hidden units and ReLU nonlinearities.

Original: We compare to the PCL implementation presented in (Hyvärinen and Morioka, 2017), where an MLP with the same number of hidden layers as the mixing MLP was adopted.
Revised: We compare to the PCL implementation presented in (Hyvärinen and Morioka, 2017), where an MLP with the same number of hidden layers as the mixing MLP was adopted.

Original: We use 100 hidden units as we did not ﬁnd increasing the value improved performance.
Revised: We use 100 hidden units as we did not ﬁnd increasing the value improved performance.

Original: To account for the architectural difference serving as a possible confounder, we use the same normalizing ﬂow encoder for optimizing the PCL objective, which we term “PCL (NF)”.
Revised: To account for the architectural difference serving as a possible confounder, we use the same normalizing ﬂow encoder for optimizing the PCL objective, which we term “PCL (NF)”.

Original: While (Hyvärinen and Morioka, 2017) used leaky ReLU nonlinearities to make the mixing invertible, said mixing is non-differentiable.
Revised: While (Hyvärinen and Morioka, 2017) used leaky ReLU nonlinearities to make the mixing invertible, said mixing is non-differentiable.

Original: This is problematic for SlowFlow, as it involves gradient optimization of the Jacobian term, and more importantly, unlike PCL, aims to explicitly recover the mixing process.
Revised: This is problematic for SlowFlow, as it involves gradient optimization of the Jacobian term, and more importantly, unlike PCL, aims to explicitly recover the mixing process.

Original: We thus use a a smooth version of the leaky-ReLU activation function with a hyperparameter α (Gresele et al., 2020),
Revised: We thus use a a smooth version of the leaky-ReLU activation function with a hyperparameter α (Gresele et al., 2020),

Original: [Equation]
Revised: [Equation]

Original: By ensuring the mixing process is smooth, we ﬁnd that SlowFlow performs favorably relative to
Revised: By ensuring the mixing process is smooth, we ﬁnd that SlowFlow performs favorably relative to

Original: PCL (Table 5) when evaluated in the same setting, converging to a better optimum at higher levels of mixing.
Revised: PCL (Table 5) when evaluated in the same setting, converging to a better optimum at higher levels of mixing.

Original: G. J OINT F ACTOR D EPENDENCE E VALUATION
Revised: F. J OINT F ACTOR D EPENDENCE E VALUATION

Original: In order to consider joint dependencies among natural generative factors, we leverage Natural Sprites to construct modiﬁed datasets where time-pairs of factors are shufﬂed per-factor (e.g. combining the x transition from one clip with the y transition from a different clip).
Revised: In order to consider joint dependencies among natural generative factors, we leverage Natural Sprites to construct modiﬁed datasets where time-pairs of factors are shufﬂed per-factor (e.g. combining the x transition from one clip with the y transition from a different clip).

Original: This destroys dependencies between the factors, while maintaining the sparse marginal distributions.
Revised: This destroys dependencies between the factors, while maintaining the sparse marginal distributions.

Original: In Fig.
Revised: In Fig.

Original: 11 (right), we show 2D marginals before (blue) and after (orange) this shufﬂing.
Revised: 11 (right), we show 2D marginals before (blue) and after (orange) this shufﬂing.

Original: The additional density on the diagonals in the unshufﬂed data reveals dependencies between pairs of factors on both datasets.
Revised: The additional density on the diagonals in the unshufﬂed data reveals dependencies between pairs of factors on both datasets.

Original: As mentioned in section 3.4, the observed dependency is mismatched from the theoretical assumptions of our model.
Revised: As mentioned in section 3.4, the observed dependency is mismatched from the theoretical assumptions of our model.

Original: We test how robust SlowVAE is to such a mismatch by training it on the permuted data and reevaluating disentanglement.
Revised: We test how robust SlowVAE is to such a mismatch by training it on the permuted data and reevaluating disentanglement.

Original: In Table 22, we highlight that the improvement of SlowVAE on the permuted (i.e. independent) continuous Natural Sprites is not signiﬁcant.
Revised: In Table 22, we highlight that the improvement of SlowVAE on the permuted (i.e. independent) continuous Natural Sprites is not signiﬁcant.

Original: In Table 21, we surprisingly ﬁnd an overall improved score with non-permuted transitions (i.e. with dependencies), with three out of seven metrics showing a signiﬁcant improvement.
Revised: In Table 21, we surprisingly ﬁnd an overall improved score with non-permuted transitions (i.e. with dependencies), with three out of seven metrics showing a signiﬁcant improvement.

Original: This is in line with Fig.
Revised: This is in line with Fig.

Original: 1f in Khemakhem et al.
Revised: 1f in Khemakhem et al.

Original: (2020b), where, at least for simple mixing, a model (Khemakhem et al., 2020a) that does not account for dependencies performs as well as one that does (Khemakhem et al., 2020b).
Revised: (2020b), where, at least for simple mixing, a model (Khemakhem et al., 2020a) that does not account for dependencies performs as well as one that does (Khemakhem et al., 2020b).

Original: We conclude that these preliminary results do not support the hypothesis that SlowVAE’s disentanglement is reliant upon the model assumption that the factors are independent, but do acknowledge that the empirical effect of statistical dependence in natural video warrants further exploration (Träuble et al., 2020;
Revised: We conclude that these preliminary results do not support the hypothesis that SlowVAE’s disentanglement is reliant upon the model assumption that the factors are independent, but do acknowledge that the empirical effect of statistical dependence in natural video warrants further exploration (Träuble et al., 2020;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: G. T RANSITION P RIOR A BLATION
Revised: F. T RANSITION P RIOR A BLATION

Original: We consider an ablated model which minimizes a KL-divergence term between the posteriors at time-step t and time-step t − 1 .
Revised: We consider an ablated model which minimizes a KL-divergence term between the posteriors at time-step t and time-step t − 1 .

Original: This encourages the model to match the posteriors of both time points as closely as possible, and resembles a probabilistic variant of Slow Feature Analysis (Turner and
Revised: This encourages the model to match the posteriors of both time points as closely as possible, and resembles a probabilistic variant of Slow Feature Analysis (Turner and

Original: Sahani, 2007).
Revised: Sahani, 2007).

Original: Speciﬁcally, we set p ( z t | z t − 1 ) =
Revised: Speciﬁcally, we set p ( z t | z t − 1 ) =

Original: q ( z t − 1 | x t − 1 ) , replacing the Laplace prior with the posterior of the previous time step.
Revised: q ( z t − 1 | x t − 1 ) , replacing the Laplace prior with the posterior of the previous time step.

Original: This is equivalent to a Gaussian ( α = 2 ) transition prior, where the mean and variance are speciﬁed by the previous time step.
Revised: This is equivalent to a Gaussian ( α = 2 ) transition prior, where the mean and variance are speciﬁed by the previous time step.

Original: We ablate over the regularization parameter γ and provide results in Tables 14 and 15, although we note that we still use the same hyperparameter values for SlowVAE as in all other experiments.
Revised: We ablate over the regularization parameter γ and provide results in Tables 14 and 15, although we note that we still use the same hyperparameter values for SlowVAE as in all other experiments.

Original: As predicted by our theoretical result, α = 2 leads to entangled representations in aggregate across evaluated datasets and metrics, even when considering a spectrum of γ values, resulting in a drastic reduction in scores, particularly on dSprites and Natural Sprites.
Revised: As predicted by our theoretical result, α = 2 leads to entangled representations in aggregate across evaluated datasets and metrics, even when considering a spectrum of γ values, resulting in a drastic reduction in scores, particularly on dSprites and Natural Sprites.

Original: H A DDITIONAL R ESULTSH. E XTENDED D ATA A NALYSIS
Revised: G A DDITIONAL R ESULTS

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We report the empirical estimates of Kurtosis in Table 7.
Revised: We report the empirical estimates of Kurtosis in Table 7.

Original: We report the log-likelihood scores for the ∆ area, ∆ x, ∆ y statistics in Tables 8, 9, and 10, respectively for a Normal, a Laplace and a generalized Laplace/Normal distribution.
Revised: We report the log-likelihood scores for the ∆ area, ∆ x, ∆ y statistics in Tables 8, 9, and 10, respectively for a Normal, a Laplace and a generalized Laplace/Normal distribution.

Original: For these distributions, we also report the ﬁt parameters for the ∆ area, ∆ x, ∆ y statistics in Tables 11, 12, and 13, respectively, where the shape parameter α of the generalized Laplacian is in bold face.
Revised: For these distributions, we also report the ﬁt parameters for the ∆ area, ∆ x, ∆ y statistics in Tables 11, 12, and 13, respectively, where the shape parameter α of the generalized Laplacian is in bold face.

Original: As a higher likelihood indicates a better ﬁt, we can see further evidence that natural transitions are highly leptokurtic; a Laplace distribution ( α = 1 ) is a better ﬁt than a Gaussian ( α = 2 ), while the generalized Laplacian yields the highest likelihood consistently with α ≈ 0 .
Revised: As a higher likelihood indicates a better ﬁt, we can see further evidence that natural transitions are highly leptokurtic; a Laplace distribution ( α = 1 ) is a better ﬁt than a Gaussian ( α = 2 ), while the generalized Laplacian yields the highest likelihood consistently with α ≈ 0 .

Original: 5 for all measurements, as indicated in the main paper.
Revised: 5 for all measurements, as indicated in the main paper.

Original: For the plots in Figs.
Revised: For the plots in Figs.

Original: 1 and 11, we set the standard deviation of each component to 1 and clipped the minimum ( − 5 ) and maximum ( 5 ) values.
Revised: 1 and 11, we set the standard deviation of each component to 1 and clipped the minimum ( − 5 ) and maximum ( 5 ) values.

Original: We note that while the marginal transitions appear sparse in metrics computed from the given object masks, since the object masks are projections of3D objects onto 2D frames, there’s potential room for bias.
Revised: We note that while the marginal transitions appear sparse in metrics computed from the given object masks, our analysis considers 2D projections of objects instead of the transition statistics in their 3D environment.

Original: This is a compelling question from a broader perspective of visual processing, but unfortunately,[Table] [Table] the KITTI-MOTS masks (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) lack the associated depth data required to answer it.
Revised: Understanding the relationship between 3D and 2D transition statistics is a compelling question from a broader perspective of visual processing, but unfortunately, the KITTI-MOTS masks (Voigtlaender et al., 2019; Geiger et al., 2012; Milan et al., 2016) lack the associated depth data required to answer it.

Original: Nonetheless, the natural scene statistics we compute arestill relevant, given that most computer vision models and vision-based animals see the 3D world as projected onto their 2D receptor arrays. 
Revised: Nonetheless, the natural scene statistics we compute are relevant, given that most computer vision models and vision-based animals see the 3D world as projected onto their 2D receptor arrays. [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: H. A LL D IS L IB R ESULTS
Revised: [Table] [Table] [Figure] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  We include results on all DisLib datasets, dSprites (Matthey et al., 2017), Cars3D (Reed et al., 2015),
Revised: [Figure] [Table] G. A LL D IS L IB R ESULTS We include results on all DisLib datasets, dSprites (Matthey et al., 2017), Cars3D (Reed et al., 2015),

Original: SmallNORB (LeCun et al., 2004), Shapes3D (Kim and Mnih, 2018), MPI3D (Gondal et al., 2019), in
Revised: SmallNORB (LeCun et al., 2004), Shapes3D (Kim and Mnih, 2018), MPI3D (Gondal et al., 2019), in

Original: Tables 16, 17, 18, 19, and 20, respectively.
Revised: Tables 16, 17, 18, 19, and 20, respectively.

Original: We report both median (a.d.) to compare to the previous median scores reported in (Locatello et al., 2020), as well as the the more common mean (s.d.) scores for future comparisons and straightforward statistical estimates of signiﬁcant differences between models.
Revised: We report both median (a.d.) to compare to the previous median scores reported in (Locatello et al., 2020), as well as the the more common mean (s.d.) scores for future comparisons and straightforward statistical estimates of signiﬁcant differences between models.

Original: We also consider allowing for static transitions, which we denote with “NC”, e.g. “LAP-NC”, in the tabular results.
Revised: We also consider allowing for static transitions, which we denote with “NC”, e.g. “LAP-NC”, in the tabular results.

Revised: As mentioned in Section 5, we use the same parameter settings for SlowVAE in all experiments, while model selection was performed not only per dataset, but per seed, for results from (Locatello et al., 2020).

Original: H. KITTI M ASKS ∆ t A BLATION
Revised: G. KITTI M ASKS ∆ t A BLATION

Original: As seen in the main text, considering image pairs separated further apart in time appears beneﬁcial.
Revised: As seen in the main text, considering image pairs separated further apart in time appears beneﬁcial.

Original: Here we evaluate a wider range by taking frames which are further apart in a sequence.
Revised: Here we evaluate a wider range by taking frames which are further apart in a sequence.

Original: max (∆ frames ) =
Revised: max (∆ frames ) =

Original: N indicates that all pairs differ by at most N frames.
Revised: N indicates that all pairs differ by at most N frames.

Original: We chose an upper bound of N , rather than sampling pairs with a ﬁxed separation, to account for the variable frame rates and sequence lengths in the original dataset (Milan et al., 2016), without introducing a confounding factor of varying dataset size.
Revised: We chose an upper bound of N , rather than sampling pairs with a ﬁxed separation, to account for the variable frame rates and sequence lengths in the original dataset (Milan et al., 2016) without introducing a confounding factor of varying dataset size.

Original: We report in Fig.
Revised: We report in Fig.

Original: 10 how the max (∆ frames ) criterion corresponds to the mean time gap between image pairs (mean (∆ t ) ) in seconds.
Revised: 10 how the max (∆ frames ) criterion corresponds to the mean time gap between image pairs (mean (∆ t ) ) in seconds.

Original: For further details, we refer to
Revised: For further details, we refer to

Original: Appendix E.4.
Revised: Appendix D.5.

Original: In Fig.
Revised: In Fig.

Original: 13 we visualize an ablation over mean (∆ t ) .
Revised: 13 we visualize an ablation over mean (∆ t ) .

Original: We ﬁnd that model performance increased initially with larger temporal separation between data points, then plateaued.
Revised: We ﬁnd that model performance increased initially with larger temporal separation between data points, then plateaued.

Original: We also observe in Fig. 14 that the measured factor marginals remain sparse, with α < 1 , for all tested settings of mean (∆ t ) . 
Revised: We also observe in Fig.that the measured factor marginals remain sparse, with α < 1 , for all tested settings of mean (∆ t ) . [Table]

Original: While it is intuitive that increasing mean (∆ t ) leads to increased diversity, and thus learning signal, in the transitions, it is worth noting that since SlowVAE assumes α = 1 , an increase in α from increasing the temporal gap leads to a reduction in mismatch.
Revised: [Table] Increasing mean (∆ t ) leads to increased diversity, and thus more information in the learning signal. However, it is worth noting that since SlowVAE assumes α = 1 in the transitions, an increase in α from increasing the temporal gap leads to a reduction in mismatch.

Original: Our results on increasing the temporal difference within pairs of inputs links to recent work by Oord et al.
Revised: Our results on increasing the temporal difference within pairs of inputs is in agreement with recent work by Oord et al

Original: (2018, Table 2), who show increased performance in representation learning for larger

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Figure] [Table] [Table] [Table]
Revised: (2018, Table 2)

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original:  separation between positive samples in a contrastive objective function.
Revised: who show increased performance in representation learning for larger separation between positive samples in a contrastive objective function.

Original: It also links to recent work by Tschannen et al.
Revised: Additional related work from Tschannen et al.

Original: (2019), who show that temporal separation between frame embeddings inﬂuences the representation that is learned from videos.[Figure] [Table] [Table]
Revised: (2019) shows that temporal separation between frame embeddings inﬂuences the representation that is learned from videos.

Original: H. L ATENT S PACE V ISUALIZATIONS
Revised: G. L ATENT S PACE V ISUALIZATIONS

Original: We visualize differences in learned latent representations using image embedding in Figures 15- 28.
Revised: We visualize differences in learned latent representations using image embedding in Figures 15- 28.