Original: T HE U NION OF M ANIFOLDS H YPOTHESIS 
Revised: V ERIFYING THE U NION OF M ANIFOLDS H YPOTHESIS FOR I MAGE D ATA

Original: Anonymous authors
Revised: Layer 6 AI anthony@layer Brendan Leigh Ross

Original: Paper under double-blind review
Revised: Layer 6 AI brendan@layer6.ai Layer 6 AI jesse@layer6.ai Gabriel Loaiza-Ganem Layer 6 AI gabriel@layer6.ai

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep learning has had tremendous success at learning low-dimensional representations of high-dimensional data.
Revised: Deep learning has had tremendous success at learning low-dimensional representations of high-dimensional data.

Original: This success would be impossible if there was no hidden low-dimensional structure in data of interest; this existence is posited by the manifold hypothesis , which states that the data lies on an unknown manifold of low intrinsic dimension.
Revised: This success would be impossible if there was no hidden low-dimensional structure in data of interest; this existence is posited by the manifold hypothesis , which states that the data lies on an unknown manifold of low intrinsic dimension.

Original: In this paper, we argue that this hypothesis does not properly capture the low-dimensional structure typically present in image data.
Revised: In this paper, we argue that this hypothesis does not properly capture the low-dimensional structure typically present in image data.

Original: Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation.
Revised: Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation.

Original: To address this deficiency, we put forth the union of manifolds hypothesis , which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions.
Revised: To address this deficiency, we consider the union of manifolds hypothesis , which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions.

Original: We empirically verify this hypothesis on commonly-used image datasets, finding that indeed, observed data lies on a disconnected set and that intrinsic dimension is not constant.
Revised: We empirically verify this hypothesis on commonlyused image datasets, finding that indeed, observed data lies on a disconnected set and that intrinsic dimension is not constant.

Original: We also provide insights into the implications  the union of manifolds hypothesis has for deep learning, both supervised and unsupervised, showing that designing models with an inductive bias for this structure improves performance across classification and generative modelling tasks. 
Revised: We also provide insights into the implications of the union of manifolds hypothesis in deep learning, both supervised and unsupervised, showing that designing models with an inductive bias for this structure improves performance across classification and generative modelling tasks. Our code is available at https://github.com/layer6ai-labs/UoMH .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The manifold hypothesis (Bengio et al., 2013) states that high-dimensional data of interest often lives in an unknown lower-dimensional manifold embedded in ambient space, and there is strong evidence supporting this hypothesis.
Revised: The manifold hypothesis (Bengio et al., 2013) states that high-dimensional data of interest often lives in an unknown lower-dimensional manifold embedded in ambient space, and there is strong evidence supporting this hypothesis.

Original: From a theoretical perspective, it is known that both manifold learning and density estimation scale exponentially with the (low) intrinsic dimension when such structure exists (Ozakin & Gray, 2009; Narayanan & Mitter, 2010), while scaling exponentially with the (high) ambient dimension otherwise (Cacoullos, 1966).
Revised: From a theoretical perspective, it is known that both manifold learning and density estimation scale exponentially with the (low) intrinsic dimension when such structure exists (Ozakin & Gray, 2009; Narayanan & Mitter, 2010), while scaling exponentially with the (high) ambient dimension otherwise (Cacoullos, 1966).

Original: Thus, the most plausible explanation for the success of machine learning methods on high-dimensional data is the existence of far lower intrinsic dimension, which facilitates learning on datasets of fairly reasonable size.
Revised: Thus, the most plausible explanation for the success of machine learning methods on high-dimensional data is the existence of far lower intrinsic dimension, which facilitates learning on datasets of fairly reasonable size.

Original: This is verified empirically by Pope et al.
Revised: This is verified empirically by Pope et al.

Original: (2021), in which a comprehensive study estimating the intrinsic dimension of commonly-used image datasets is performed, clearly finding low-dimensional structure.
Revised: (2021), in which a comprehensive study estimating the intrinsic dimension of commonly-used image datasets is performed, clearly finding low-dimensional structure.

Original: However, thinking of observed data as lying on a single unknown low-dimensional manifold is quite limiting, as this implies that the intrinsic dimension throughout the dataset is constant.
Revised: However, thinking of observed data as lying on a single unknown low-dimensional manifold is quite limiting, as this implies that the intrinsic dimension throughout the dataset is constant.

Original: If we consider the intrinsic dimensionality to be the number of factors of variation generating the data, we can see that this formulation prevents distinct regions of the data’s support from having differing quantities of factors of variation.
Revised: If we consider the intrinsic dimensionality to be the number of factors of variation generating the data, we can see that this formulation prevents distinct regions of the data’s support from having differing quantities of factors of variation.

Original: Yet this seems to be unrealistic: for example, we should not expect the number of factors needed to describe 8 s and 1 s in the MNIST dataset (LeCun et al., 1998) to be equal.
Revised: Yet this seems to be unrealistic: for example, we should not expect the number of factors needed to describe 8 s and 1 s in the MNIST dataset (LeCun et al., 1998) to be equal.

Original: To accommodate this intuition, in this paper we propose the union of manifolds hypothesis. We posit that high-dimensional image data often lies not on a single manifold, but on a disjoint union of manifolds of different intrinsic dimensions .
Revised: To accommodate this intuition, in this paper we consider the union of manifolds hypothesis: that high-dimensional image data often lies not on a single manifold, but on a disjoint union of manifolds of different intrinsic dimensions .

Original: 1 While this hypothesis has motivated work in the clustering literature (Vidal, 2011; Elhamifar & Vidal, 2011; 2012; 2013; Zhang et al., 2019; Abdolali & Gillis, 2021;
Revised: 1 While this hypothesis has motivated work in the clustering literature (Vidal, 2011; Elhamifar & Vidal, 2011; 2012; 2013; Zhang et al., 2019; Abdolali & Gillis, 2021;

Original: Cai et al., 2022), to the best of our knowledge it has never been empirically explored analogously to the way that Pope et al.
Revised: Cai et al., 2022), to the best of our knowledge it has never been empirically explored analogously to the way that Pope et al.

Original: (2021) probe the manifold hypothesis.
Revised: (2021) probe the manifold hypothesis.

Original: In this work we carry out this verification on commonly-used image datasets, first by confirming their supports are disconnected, and then by estimating the intrinsic dimension on each component, finding that there is indeed variation in these estimates.
Revised: In this work we carry out this verification on commonly-used image datasets, first by confirming their supports are disconnected, and then by estimating the intrinsic dimension on each component, finding that there is indeed variation in these estimates.

Original: In order to verify that the support of the data is disconnected, we leverage pushforward deep generative models (Salmona et al., 2022; Ross et al., 2022), which generate samples by transforming noise through a neural network G .
Revised: In order to verify that the support of the data is disconnected, we leverage pushforward deep generative models (Salmona et al., 2022; Ross et al., 2022), which generate samples by transforming noise through a neural network G .

Original: We first prove that these deep generative models (DGMs) are incapable of modelling disconnected supports.
Revised: We first prove that these deep generative models (DGMs) are incapable of modelling disconnected supports.

Original: We then argue that the class labels provided in our considered datasets approximately identify connected components (i.e. different classes are mostly disconnected from each other), and show that training a pushforward model on each class outperforms training a single such model on the entire dataset, even when using the same computational budget: this improvement is a firm indicator that the support is truly disconnected.
Revised: We then argue that the class labels provided in our considered datasets approximately identify connected components (i.e. different classes are mostly disconnected from each other), and show that training a pushforward model on each class outperforms training a single such model on the entire dataset, even when using the same computational budget: this improvement is a firm indicator that the support is truly disconnected.

Original: After empirically verifying the union of manifolds hypothesis, we turn our attention to some of its implications in deep learning.
Revised: After empirically verifying the union of manifolds hypothesis, we turn our attention to some of its implications in deep learning.

Original: We establish that classes with higher intrinsic dimension are harder to classify, and guided by this insight, we also show that classification accuracy can be improved by more heavily weighting the terms corresponding to classes of higher intrinsic dimension in the cross entropy loss.
Revised: We establish that classes with higher intrinsic dimension are harder to classify, and guided by this insight, we also show that classification accuracy can be improved by more heavily weighting the terms corresponding to classes of higher intrinsic dimension in the cross entropy loss.

Original: We also show that the same DGMs we used to confirm the disconnectedness of the data support, which we call disconnected DGMs, provide a performant class of models which are competitive with their non-disconnected baselines across a wide range of datasets and models, and thus provide a promising direction towards improving generative models.
Revised: We also show that the same DGMs we used to confirm the disconnectedness of the data support, which we call disconnected DGMs, provide a performant class of models which are competitive with their non-disconnected baselines across a wide range of datasets and models, and thus provide a promising direction towards improving generative models.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: Throughout our paper, we consider the setup where we have access to a dataset D
Revised: Throughout our paper, we consider the setup where we have access to a dataset D

Original: = { x i } ni =1 , generated i.i.d.
Revised: = { x i } ni =1 , generated i.i.d.

Original: from some distribution P ∗ in a high dimensional ambient space X = R D .
Revised: from some distribution P ∗ in a high dimensional ambient space X = R D .

Original: Pushforward DGMs As mentioned in the introduction, we leverage DGMs – in particular pushforward DGMs – in order to verify the disconnectedness of the support of P ∗ .
Revised: Pushforward DGMs As mentioned in the introduction, we leverage DGMs – in particular pushforward DGMs – in order to verify the disconnectedness of the support of P ∗ .

Original: We call a pushforward model any DGM whose samples X are given by:
Revised: We call a pushforward model any DGM whose samples X are given by:

Original: [Equation]
Revised: [Equation]

Original: X

Original: [Equation] G ( Z [Equation] where P Z is a (potentially trainable) base distribution in some latent space Z , and G : Z → X is a neural network.
Revised: where P Z is a (potentially trainable) base distribution in some latent space Z , and G : Z → X is a neural network.

Original: We highlight many popular DGMs fall into this category, including (Gaussian) variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), normalizing flows (NFs)
Revised: We highlight many popular DGMs fall into this category, including (Gaussian) variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), normalizing flows (NFs)

Original: (Dinh et al., 2017; Kingma & Dhariwal, 2018; Behrmann et al., 2019;
Revised: (Dinh et al., 2017; Kingma & Dhariwal, 2018; Behrmann et al., 2019;

Original: Chen et al., 2019; Durkan et al., 2019), generative adversarial networks (GANs) (Goodfellow et al., 2014), and Wasserstein autoencoders (WAEs) (Tolstikhin et al., 2018).
Revised: Chen et al., 2019; Durkan et al., 2019), generative adversarial networks (GANs) (Goodfellow et al., 2014), and Wasserstein autoencoders (WAEs) (Tolstikhin et al., 2018).

Original: Intrinsic dimension estimation If we assume that P ∗ is supported on  a d -dimensional manifold embedded in X for some unknown d < D , a natural question is how to estimate d from D .
Revised: Intrinsic dimension estimation If we assume that P ∗ is supported on the closure of a d -dimensional manifold embedded in X for some unknown d < D , a natural question is how to estimate d from D .

Original: We follow Pope et al.
Revised: We follow Pope et al.

Original: and use the Levina & Bickel (2004) estimator with the MacKay & Ghahramani (2005) extension, given by:
Revised: and use the Levina & Bickel (2004) estimator with the MacKay & Ghahramani (2005) extension, given by:

Original: [Equation]
Revised: [Equation]

Original: where T j ( x ) is the Euclidean distance from x to its j th -nearest neighbour in D \ { x } , and k is a hyperparameter specifying the maximum number of nearest neighbours to consider.
Revised: where T j ( x ) is the Euclidean distance from x to its j th -nearest neighbour in D \ { x } , and k is a hyperparameter specifying the maximum number of nearest neighbours to consider.

Original: While other estimators have been recently proposed (Block et al., 2021; Lim et al., 2021; Tempczyk et al., 2022), we stick with (2) throughout this work as it is well-established in the literature.
Revised: While other estimators have been recently proposed (Block et al., 2021; Lim et al., 2021; Tempczyk et al., 2022), we stick with (2) throughout this work as it is well-established in the literature.

Original: As we will see, popular image datasets exhibit different intrinsic dimensions in separate regions of data space.
Revised: As we will see, popular image datasets exhibit different intrinsic dimensions in separate regions of data space.

Original: On the relevance of understanding low-dimensional structure As mentioned in the introduction, the manifold hypothesis provides intuition as to why deep learning has been so successful.
Revised: On the relevance of understanding low-dimensional structure As mentioned in the introduction, the manifold hypothesis provides intuition as to why deep learning has been so successful.

Original: One might na¨ıvely believe its usefulness is merely conceptual, yet uncovering and exploiting the structure underlying data of interest is a highly relevant problem (Bronstein et al., 2021) which has recently started receiving increased attention.
Revised: One might na¨ıvely believe its usefulness is merely conceptual, yet uncovering and exploiting the structure underlying data of interest is a highly relevant problem (Bronstein et al., 2021) which has recently started receiving increased attention.

Original: Pope et al.
Revised: Pope et al.

Original: (2021) use (2) to estimate the intrinsic dimension of image datasets, not only finding strong evidence of low-dimensional structure, but also uncovering that datasets of higher intrinsic dimension are harder to classify.
Revised: (2021) use (2) to estimate the intrinsic dimension of image datasets, not only finding strong evidence of low-dimensional structure, but also uncovering that datasets of higher intrinsic dimension are harder to classify.

Original: Similarly, Ansuini et al.
Revised: Similarly, Ansuini et al.

Original: (2019) link the intrinsic dimension of the representations of deep classifiers to their classification accuracy.
Revised: (2019) link the intrinsic dimension of the representations of deep classifiers to their classification accuracy.

Original: Furthermore, the low-dimensional structure of data not only affects supervised methods: for example, training a high-dimensional density to model data lying on a low-dimensional manifold through maximum-likelihood can result in manifold overfitting (Dai & Wipf, 2019; Loaiza-Ganem et al., 2022), a phenomenon resulting in P ∗ not being learned, even if an infinite amount of data was observed.
Revised: Furthermore, the low-dimensional structure of data not only affects supervised methods: for example, training a high-dimensional density to model data lying on a low-dimensional manifold through maximum-likelihood can result in manifold overfitting (Dai & Wipf, 2019; Loaiza-Ganem et al., 2022), a phenomenon resulting in P ∗ not being learned, even if an infinite amount of data was observed.

Original: Indeed, there is a rapidly growing body of work aiming to account for the manifold hypothesis within DGMs (Gemici et al., 2016; Rezende et al., 2020; Brehmer & Cranmer, 2020;
Revised: Indeed, there is a rapidly growing body of work aiming to account for the manifold hypothesis within DGMs (Gemici et al., 2016; Rezende et al., 2020; Brehmer & Cranmer, 2020;

Original: Mathieu & Nickel, 2020; Arbel et al., 2021; Kothari et al., 2021; Caterini et al., 2021; Ross &
Revised: Mathieu & Nickel, 2020; Arbel et al., 2021; Kothari et al., 2021; Caterini et al., 2021; Ross &

Original: Cresswell, 2021; Cunningham et al., 2022; De Bortoli et al., 2022; Ross et al., 2022).
Revised: Cresswell, 2021; Cunningham et al., 2022; De Bortoli et al., 2022; Ross et al., 2022).

Original: All these works highlight the relevance of properly understanding and accounting for the low-dimensional structure of data, an understanding we improve upon in this work.
Revised: All these works highlight the relevance of properly understanding and accounting for the low-dimensional structure of data, an understanding we improve upon in this work.

Original:  T HE UNION OF MANIFOLDS HYPOTHESIS
Revised:  T HE UNION OF MANIFOLDS HYPOTHESIS

Original: We will assume that the support of P ∗ , supp ( P ∗ ) , can be written as:
Revised: We will assume that the support of P ∗ , supp ( P ∗ ) , can be written as:

Original: [Equation]
Revised: [Equation]

Original: where ⊔ denotes disjoint union,and each M ℓ is a connected manifold of dimension d ( ℓ )  .
Revised: where ⊔ denotes disjoint union, each M ℓ is a connected manifold of dimension d ( ℓ ) , and cl ( · ) denotes closure in X .

Original: We call this assumption the union of manifolds hypothesis .
Revised: We refer to this assumption the union of manifolds hypothesis .

Original: Note that we make the assumption that each M ℓ is connected merely for notational simplicity, as we could collapse each union of manifolds of matching dimension into a single disconnected manifold, but this notation allows us to reason about different  M ℓ  as connected components of interest.
Revised: Note that we make the assumption that each M ℓ is connected merely for notational simplicity, as we could collapse each union of manifolds of matching dimension into a single disconnected manifold, but this notation allows us to reason about different cl ( M ℓ ) as connected components of interest.

Original: Note also that the standard manifold hypothesis is equivalent to adding the assumption that all d ( ℓ ) s are identical.
Revised: Note also that the standard manifold hypothesis is equivalent to adding the assumption that all d ( ℓ ) s are identical.

Original: In the rest of this section we empirically verify our hypothesis on various commonly-used image datasets in two parts: first by establishing the disconnectedness of supp ( P ∗ ) , and then by estimating the intrinsic dimension on each of its (approximate) connected components.
Revised: In the rest of this section we empirically verify this hypothesis on various commonly-used image datasets in two parts: first by establishing the disconnectedness of supp ( P ∗ ) , and then by estimating the intrinsic dimension on each of its (approximate) connected components.

Original: 3. V ERIFYING THE HYPOTHESIS : D ISCONNECTEDNESS
Revised: 3. V ERIFYING THE HYPOTHESIS : D ISCONNECTEDNESS

Original: Before empirically verifying that the support of commonly-used image datasets is disconnected, let us develop intuition as to why this might be the case.
Revised: Before empirically verifying that the support of commonly-used image datasets is disconnected, let us develop intuition as to why this might be the case.

Original: For example, on MNIST we can likely continuously transform any 2 into any other 2 without leaving the manifold of 2 s, whereas attempting to similarly transform a 2 into an 8 would likely result in intermediate images that are neither 2 s nor 8 s. We conjecture that class labels, which are often available for our considered datasets, provide a sensible proxy for the connected components  M ℓ  of supp ( P ∗ ) , precisely because one can conceive of continuously transforming any one member of any class into any other member of the same class without ever leaving the class.
Revised: For example, on MNIST we can likely continuously transform any 2 into any other 2 without leaving the manifold of 2 s, whereas attempting to similarly transform a 2 into an 8 would likely result in intermediate images that are neither 2 s nor 8 s. We conjecture that class labels, which are often available for our considered datasets, provide a sensible proxy for the connected components cl ( M ℓ ) of supp ( P ∗ ) , precisely because one can conceive of continuously transforming any one member of any class into any other member of the same class without ever leaving the class.

Original: In order to verify this conjecture, we leverage a shortcoming of pushforward models, namely their inability to model disconnected supports, which we formalize in the following proposition.
Revised: In order to verify this conjecture, we leverage a shortcoming of pushforward models, namely their inability to model disconnected supports, which we formalize in the following proposition.

Original: We use measure-theoretic notation, where the model from (1) is given by the pushforward measure G # P Z .
Revised: We use measure-theoretic notation, where the model from (1) is given by the pushforward measure G # P Z .

Original: Proposition 1 : Let Z and X be topological spaces and G : Z → X be continuous.
Revised: Proposition 1 : Let Z and X be topological spaces and G : Z → X be continuous.

Original: Considering Z and X as measurable spaces with their respective Borel σ -algebras, let P Z be a probability measure on Z such that supp ( P Z ) is connected and P Z ( supp ( P Z ))
Revised: Considering Z and X as measurable spaces with their respective Borel σ -algebras, let P Z be a probability measure on Z such that supp ( P Z ) is connected and P Z ( supp ( P Z ))

Original: = 1 .
Revised: = 1 .

Original: Then supp ( G # P Z ) is connected.
Revised: Then supp ( G # P Z ) is connected.

Original: Proof sketch : We begin by pointing out that from the formal definition of support, supp ( G # P Z ) need not be equal toG ( supp ( P Z )) .
Revised: Proof sketch : From the formal definition of support, supp ( G # P Z ) need not be equal to

Original: In other words, the proof does not immediately follow from the fact that continuous functions map connected sets to connected sets.
Revised: G ( supp ( P Z )) , so the proof does not immediately follow from the fact that continuous functions map connected sets to connected sets.

Original: In our proof we first show that supp ( G # P Z ) =
Revised: We show that supp ( G # P Z ) =

Original: cl ( G ( supp ( P Z ))) ,where cl ( · ) denotes closure in X , and the result follows from the closure of connected sets being connected.
Revised: cl ( G ( supp ( P Z ))) , and the result follows from the closure of connected sets being connected.

Original: The formal proof along with the definition of support is included in Appendix A.
Revised: See Appendix A for the formal proof.

Original: The implication of Proposition 1 is simple: if the support of the data is not connected (i.e. has multiple connected components), then pushforward models are insufficient to properly model the data, since in practice supp ( P Z ) is always connected.
Revised: The implication of Proposition 1 is simple: if the support of the data is not connected (i e. has multiple connected components) then pushforward models are insufficient to properly model the data, since in practice supp ( P Z ) is always connected.

Original: Intuitively, this result rules out the possibility of G # P Z assigning 0 probability to regions of G ( supp ( P Z )) which connect different disconnected components of the data support supp ( P ∗ ) , as illustrated in Figure 1.
Revised: Intuitively, this result rules out the possibility of G # P Z assigning 0 probability to regions of G ( supp ( P Z )) which connect different disconnected components of the data support supp ( P ∗ ) , as illustrated in Figure 1.

Original: We highlight that topological limitations of pushforward models have been studied before (Cornish et al., 2020; Salmona et al., 2022; Ross et al., 2022), although to the best of our knowledge not to the generality of Proposition 1.
Revised: We highlight that topological limitations of pushforward models have been studied before (Cornish et al., 2020; Salmona et al., 2022; Ross et al., 2022), although to the best of our knowledge not to the generality of Proposition 1.

Original: Indeed, there are many methods aiming to better model disconnected supports with pushforward models (Arora et al., 2017; Banijamali et al., 2017; Locatello et al., 2018;
Revised: Indeed, there are many methods aiming to better model disconnected supports with pushforward models (Arora et al., 2017; Banijamali et al., 2017; Locatello et al., 2018;

Original: Khayatkhoei et al., 2018;
Revised: Khayatkhoei et al., 2018;

Original: Dinh et al., 2019; Tanielian et al., 2020; Luzi et al., 2020; Pires & Figueiredo, 2020; Duan, 2021; Ye & Bors, 2021).
Revised: Dinh et al., 2019; Tanielian et al., 2020; Luzi et al., 2020; Pires & Figueiredo, 2020; Duan, 2021; Ye & Bors, 2021).

Original: Figure 2, which we will use as a running example throughout this section, further visualizes this result, with the first panel showing a synthetic dataset obeying the union of manifolds hypothesis (two connected components of respective intrinsic dimensions 1 and 2 ), along with a Gaussian VAE trained on this dataset on the second panel.The
Revised: Figure 2, which we will use as a running example throughout this section, further visualizes this result, with the first panel showing a synthetic dataset obeying the union of manifolds hypothesis (two connected components of respective intrinsic dimensions 1 and 2 ), along with a Gaussian VAE trained on this dataset on the second panel.

Original:  VAE fails to recover a disconnected support, further confirming this limitation of pushforward models.
Revised: The VAE fails to recover a disconnected support, further confirming this limitation of pushforward models.

Original: We include all experimental details pertaining to this section in Appendix D.1.
Revised: We include all experimental details pertaining to this section in Appendix D.1.

Original: As a means to empirically verify that supp ( P ∗ ) is disconnected, we introduce disconnected DGMs : rather than train a single pushforward model ( P Z , G ) on D as in standard DGMs, we first partition the data as D = ⊔
Revised: As a means to empirically verify that supp ( P ∗ ) is disconnected, we introduce disconnected DGMs : rather than train a single pushforward model ( P Z , G ) on D as in standard DGMs, we first partition the data as D = ⊔

Original: Lℓ =1 D ℓ , where D ℓ contains only the datapoints belonging to the ℓ th class, and we then train a pushforward DGM ( P ( ℓ ) Z , G ℓ ) on each cluster D ℓ .
Revised: Lℓ =1 D ℓ , where D ℓ contains only the datapoints belonging to the ℓ th class, and we then train a pushforward DGM ( P ( ℓ ) Z , G ℓ ) on each cluster D ℓ .

Original: In order to obtain a sample X from a trained disconnected DGM, we simply sample ℓ with probability proportional to the number ofdatapoints in D ℓ , |D ℓ | , and then sample from the corresponding DGM: [Equation] which can be performed in a memory-efficient way (i.e. without having to load L models in memory) as described in Appendix B.2.
Revised: In order to obtain a sample X from a trained disconnected DGM, we simply sample ℓ with probability proportional to the size of D ℓ , |D ℓ | , and then sample from the corresponding DGM, which can be performed in a memory-efficient way (i.e. without having to load L models in memory) as described in Appendix B.2.

Original: We see disconnected DGMs as the most straightforward modification that can be done to pushforward models so as to enable them to model disconnected supports.
Revised: We see disconnected DGMs as the most straightforward modification that can be done to pushforward models so as to enable them to model disconnected supports.

Original: Disconnected DGMs also allow to straightforwardly compare against standard pushforward models in a FLOP-equivalent manner that uses the same architectures (see Appendix B.1 for details).
Revised: Disconnected DGMs also allow to straightforwardly compare against standard pushforward models in a FLOP-equivalent manner that uses the same architectures (see Appendix B.1 for details).

Original: If we observe any improvement of disconnected DGMs over their standard counterparts, the simplest explanation for this behaviour would be that supp ( P ∗ ) is disconnected .
Revised: If we observe any improvement of disconnected DGMs over their standard counterparts, the simplest explanation for this behaviour would be that supp ( P ∗ ) is disconnected .

Original: The third panel of Figure 2 shows a disconnected VAE (denoted D-VAE), which correctly recovers two connected components (albeit not their intrinsic dimensions, which we will address later with the fourth panel), and was given the same computational budget as the VAE on the second panel: demonstrating that disconnected DGMs show improvementsover their standard counterparts when data is disconnected, thus providing more evidence for the validity of our test of disconnectedness.
Revised: The third panel of Figure 2 shows a disconnected VAE (denoted D-VAE), which correctly recovers two connected components (albeit not their intrinsic dimensions, which we will address later with the fourth panel), and was given the same computational budget as the VAE on the second panel: demonstrating that disconnected DGMs show improvements

Original: [Table]
Revised: [Table]

Revised: over their standard counterparts when data is disconnected, thus providing more evidence for the validity of our test of disconnectedness.

Original: The top half of Table 1 shows analogous results in the more realistic setting of image datasets.
Revised: The top half of Table 1 shows analogous results in the more realistic setting of image datasets.

Original: We use the FID score (Heusel et al., 2017) (lower is better), a commonly-used sample quality metric, to measure performance on the MNIST, FMNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011),
Revised: We use the FID score (Heusel et al., 2017) (lower is better), a commonly-used sample quality metric, to measure performance on the MNIST, FMNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011),

Original: CIFAR-10, and CIFAR-100 (Krizhevsky et al., 2009) datasets.
Revised: CIFAR-10, and CIFAR-100 (Krizhevsky et al., 2009) datasets.

Original: We can see that disconnected VAEs (indicated as “D-VAE (classes)”) consistently outperform VAEs 3 , strongly supporting our hypothesis that supp ( P ∗ ) is indeed not connected in these datasets.
Revised: We can see that disconnected VAEs (indicated as “D-VAE (classes)”) consistently outperform VAEs 4 , strongly supporting our hypothesis that supp ( P ∗ ) is indeed not connected in these datasets.

Original: We highlight once again that disconnected
Revised: We highlight once again that disconnected

Original: VAEs were given the exact same computational budget as VAEs (both in terms of floating point operations and memory usage) to ensure fair comparisons.
Revised: VAEs were given the exact same computational budget as VAEs (both in terms of floating point operations and memory usage) to ensure fair comparisons.

Original: Nonetheless, disconnected VAEs do have more parameters.
Revised: Nonetheless, disconnected VAEs do have more parameters.

Original: In order to ensure that the added parameters are not the reason that disconnected
Revised: In order to ensure that the added parameters are not the reason that disconnected

Original: VAEs perform better, we perform an ablation, where we train a disconnected VAE but use random labels instead of class labels (indicated as “D-VAE (random)”).
Revised: VAEs perform better, we perform an ablation, where we train a disconnected VAE but use random labels instead of class labels (indicated as “D-VAE (random)”).

Original: We can see that disconnected VAEs using labels are still the best performing option, once again providing strong empirical evidence to the claim that one can indeed think of classes of these datasets as  different connected componentsM ℓ of supp ( P ∗ ) .
Revised: We can see that disconnected VAEs using labels are still the best performing option, once again providing strong empirical evidence to the claim that one can indeed think of classes of these datasets as approximating different connected components of supp ( P ∗ ) .

Original: Finally, we point out that these results are not exclusive to VAEs: we carried out the same comparisons using other pushforward models, and found the conclusions to be the same, further backing our claims.
Revised: Finally, we point out that these results are not exclusive to VAEs: we carried out the same comparisons using other pushforward models, and found the conclusions to be the same, further backing our claims.

Original: Due to space constraints, we present these results in Tables 3, 4, and 5 for GANs, NFs, and WAEs, respectively, in Appendix C.1.
Revised: Due to space constraints, we present these results in Tables 3, 4, and 5 for GANs, NFs, and WAEs, respectively, in Appendix C.1.

Original: 3. V ERIFYING THE HYPOTHESIS : V ARYING INTRINSIC DIMENSIONS
Revised: 3. V ERIFYING THE HYPOTHESIS : V ARYING INTRINSIC DIMENSIONS

Original: So far we have empirically verified that commonly-used image datasets have disconnected supports and that class labels provide a good proxy for the corresponding connected components.
Revised: So far we have empirically verified that commonly-used image datasets have disconnected supports and that class labels provide a good proxy for the corresponding connected components.

Original: We now confirm that these disconnected components have varying intrinsic dimensions.
Revised: We now confirm that these disconnected components have varying intrinsic dimensions.

Original: We achieve this by applying (2) to each D ℓ (the subset of the data corresponding to the ℓ th class) separately: observing similar estimates across different values of ℓ would support the manifold hypothesis, whereas observing different values would instead support the union of manifolds hypothesis.
Revised: We achieve this by applying (2) to each D ℓ (the subset of the data corresponding to the ℓ th class) separately: observing similar estimates across different values of ℓ would support the manifold hypothesis, whereas observing different values would instead support the union of manifolds hypothesis.

Original: Figure 3 shows, for different values of the hyperparameter k , boxplots of the values ˆ d ( ℓ ) k for class labels ℓ = 1 , . . .
Revised: Figure 3 shows, for different values of the hyperparameter k , boxplots of the values ˆ d ( ℓ ) k for class labels ℓ = 1 , . . .

Original: , L in our considered datasets: MNIST, FMNIST, SVHN, CIFAR-10, CIFAR-100, and ImageNet (Russakovsky et al., 2015).
Revised: , L in our considered datasets: MNIST, FMNIST, SVHN, CIFAR-10, CIFAR-100, and ImageNet (Russakovsky et al., 2015).

Original: Two relevant patterns emerge.
Revised: Two relevant patterns emerge.

Original: First, within each dataset, results are mostly consistent across different choices of k , so that any conclusions we draw are not caused by a specific choice of this hyperparameter.
Revised: First, within each dataset, results are mostly consistent across different choices of k , so that any conclusions we draw are not caused by a specific choice of this hyperparameter.

Original: Second, for all datasets except SVHN, we can observe a wide range of estimated intrinsic dimensions.
Revised: Second, for all datasets except SVHN, we can observe a wide range of estimated intrinsic dimensions.

Original: These results empirically verify that the union of manifolds hypothesis is a more appropriate way to think about images than the manifold hypothesis.
Revised: These results empirically verify that the union of manifolds hypothesis is a more appropriate way to think about images than the manifold hypothesis.

Revised: Additionally, we estimate the variance of these estimators through subsampling in Appendix C.2, ensuring that the observed variability of estimated intrinsic dimensions is not due to chance.

Original: We highlight that (2) is known to underestimate the true intrinsic dimension when the dataset is not large enough, which might in principle affect our estimates on CIFAR-100 and
Revised: We highlight that (2) is known to underestimate the true intrinsic dimension when the dataset is not large enough, which might in principle affect our estimates on CIFAR-100 and

Original: ImageNet, since these datasets have fewer datapoints per class.
Revised: ImageNet, since these datasets have fewer datapoints per class.

Original: Strong consistency across k on
Revised: Strong consistency across k on

Original: CIFAR-100 however suggests this is not a concern for this dataset; and even if the estimates we obtained on ImageNet do exhibit slightly more variation across k , the overall range of estimated intrinsic dimensions remains consistent across k .
Revised: CIFAR-100 however suggests this is not a concern for this dataset; and even if the estimates we obtained on ImageNet do exhibit slightly more variation across k , the overall range of estimated intrinsic dimensions remains consistent across k .

Original: In other words, these results still strongly support the union of manifolds hypothesis.
Revised: In other words, these results still strongly support the union of manifolds hypothesis.

Original: We take a more granular look into these estimates in Appendix
Revised: We take a more granular look into these estimates in Appendix

Original: C.2, where we show further choices of k as well as the intrinsic dimensions of individual classes.
Revised: C.2, where we show further choices of k as well as the intrinsic dimensions of individual classes.

Original:  E XPLOITING THE UNION OF MANIFOLDS HYPOTHESIS
Revised:  E XPLOITING THE UNION OF MANIFOLDS HYPOTHESIS

Original: Now that we have empirically verified the union of manifolds hypothesis on images– both the disconnectedness of the support of the data and its varying intrinsic dimensions – we turn our attention to the benefits brought to deep learning by being aware of the union of manifolds structure present in observed data.
Revised: Now that we have empirically verified the union of manifolds hypothesis on images– both the disconnectedness of the support of the data and its varying intrinsic dimensions – we turn our attention to the benefits brought to deep learning by being aware of the union of manifolds structure present in observed data.

Original: We see the results in this section as a sensible and promising first step towards realizing these benefits, and hope to encourage the community to further explore them.
Revised: We see the results in this section as a sensible and promising first step towards realizing these benefits, and hope to encourage the community to further explore them.

Original: 4. S UPERVISED LEARNING : C LASSIFICATION ACCURACY AND INTRINSIC DIMENSION
Revised: 4. S UPERVISED LEARNING : C LASSIFICATION ACCURACY AND INTRINSIC DIMENSION

Original: Pope et al.
Revised: Pope et al.

Original: (2021) showed that datasets of high intrinsic dimension are harder to classify.
Revised: (2021) showed that datasets of high intrinsic dimension are harder to classify.

Original: Here we provide a more granular look, by showing that classes of higher intrinsic dimension are harder to classify.
Revised: Here we provide a more granular look, by showing that classes of higher intrinsic dimension are harder to classify.

Original: We train 3 classifiers on CIFAR-100 (see Appendix D.2 for details): a VGG-19 (Simonyan &
Revised: We train 3 classifiers on CIFAR-100 (see Appendix D.2 for details): a VGG-19 (Simonyan &

Original: Zisserman, 2015), a ResNet-18, and a ResNet-34 (He et al., 2016).
Revised: Zisserman, 2015), a ResNet-18, and a ResNet-34 (He et al., 2016).

Original: We focused on CIFAR-100 here as the other datasets considered in this work were either too simple to classify (MNIST, FMNIST, SVHN,
Revised: We focused on CIFAR-100 here as the other datasets considered in this work were either too simple to classify (MNIST, FMNIST, SVHN,

Original: CIFAR-10), or produced less-reliable intrinsic dimension estimates (ImageNet).
Revised: CIFAR-10), or produced less-reliable intrinsic dimension estimates (ImageNet).

Original: For each class, we compute its classification accuracy and plot this against its estimated intrinsic dimension in Figure 4.
Revised: For each class, we compute its classification accuracy and plot this against its estimated intrinsic dimension in Figure 4.

Original: We can see that, consistently across classifiers, there is an inverse relationship between estimated intrinsic dimension and classification accuracy.
Revised: We can see that, consistently across classifiers, there is an inverse relationship between estimated intrinsic dimension and classification accuracy.

Original: We also quantitatively compare intrinsic dimension and classification accuracy by computing their correlation, along with a p -value for independence with a t -test: we find that the negative correlation is significant (see figure caption).
Revised: We also quantitatively compare intrinsic dimension and classification accuracy by computing their correlation, along with a p -value for independence with a t -test: we find that the negative correlation is significant (see figure caption).

Original: In other words, the higher the intrinsic dimension of a class, the harder it is to classify.
Revised: In other words, the higher the intrinsic dimension of a class, the harder it is to classify.

Original: While clearly intrinsic dimension does not fully explain accuracy, this correlation suggests the following intuition: learning useful representations for classes with higher intrinsic dimension requires learning more factors of variation and is thus more challenging.
Revised: While clearly intrinsic dimension does not fully explain accuracy, this correlation suggests the following intuition: learning useful representations for classes with higher intrinsic dimension requires learning more factors of variation and is thus more challenging.

Original: [Table]
Revised: [Table]

Original: In order to check if this insight can help improve classifiers, we train the ResNetin two different ways (see Appendix D. for experimental setup): in the first, we use the standard cross entropy loss, and in the second, we weight the terms corresponding to each class in the cross entropy loss in a manner proportional to their intrinsic dimension.
Revised: To check if this insight can help improve classifiers, we train the ResNet-18 in two different ways (see Appendix D.2 for experimental setup): in the first, we use the standard cross entropy loss, and in the second, we weight the terms corresponding to each class in the cross entropy loss proportionally to their intrinsic dimension.

Original: The idea behind this reweighting of the loss is simply to focus more on classes of higher intrinsic dimension, as we have just shown these are harder to classify.
Revised: This reweighting of the loss focuses more on classes of higher intrinsic dimension, as we have just shown these are harder to classify.

Original: Results are shown in Table 2, where we can see that this very simple change to the cross entropy loss marginally (but significantly, as error bars do not overlap) improves the accuracy of the network, providing an essentially “free” improvement, given the low computational overhead of estimating intrinsic dimension.
Revised: Results are shown in Table 2, where we can see that this very simple change to the cross entropy loss marginally (but significantly, as error bars do not overlap) improves the accuracy of the network, providing an essentially “free” improvement, given the low computational overhead of estimating intrinsic dimension.

Original: Note that we did not perform any data augmentation scheme so as to not affect the intrinsic dimension of the data.
Revised: Note that we did not perform any data augmentation scheme so as to not affect the intrinsic dimension of the data.

Original: 4. U NSUPERVISED LEARNING : D ISCONNECTED DGM S THROUGH CLUSTERING
Revised: 4. U NSUPERVISED LEARNING : D ISCONNECTED DGM S THROUGH CLUSTERING

Original: In subsection 3.1 we used disconnected DGMs to show that the support of image datasets is disconnected.
Revised: In subsection 3.1 we used disconnected DGMs to show that the support of image datasets is disconnected.

Original: Here we show that despite their simplicity, disconnected DGMs are performant models which can provide improvements over competing alternatives.
Revised: Here we show that despite their simplicity, disconnected DGMs are performant models which can provide improvements over competing alternatives.

Original: We highlight that the aim of these experiments is not to achieve state-of-the-art performance, but rather to show that disconnected DGMs outperform comparable non-disconnected models, emphasizing the relevance of properly accounting for the union of manifolds structure present in data.
Revised: We highlight that the aim of these experiments is not to achieve state-of-the-art performance, but rather to show that disconnected DGMs outperform comparable non-disconnected models, emphasizing the relevance of properly accounting for the union of manifolds structure present in data.

Original: To this end, we introduce a fully-unsupervised version of disconnected DGMs, where we run a clustering algorithm to partition the dataset instead of using class labels.
Revised: To this end, we introduce a fully-unsupervised version of disconnected DGMs, where we run a clustering algorithm to partition the dataset instead of using class labels.

Original: The idea behind this modification is simply that if class labels are unavailable, a clustering algorithm might manage to recover connected components of the data support as clusters, at least approximately.
Revised: The idea behind this modification is simply that if class labels are unavailable, a clustering algorithm might manage to recover connected components of the data support as clusters, at least approximately.

Original: Our training and clustering procedures are detailed in Appendix B.1.
Revised: Our training and clustering procedures are detailed in Appendix B.1.

Original: We highlight once again that we use the same computational budget for training disconnected
Revised: We highlight once again that we use the same computational budget for training disconnected

Original: DGMs as for their standard counterparts.
Revised: DGMs as for their standard counterparts.

Original: The only computational overhead is running the clustering algorithm itself – and more disk space is required to store the model – but the number of floating point operations and memory usage throughout training is identical (see Appendix B.1).
Revised: The only computational overhead is running the clustering algorithm itself – and more disk space is required to store the model – but the number of floating point operations and memory usage throughout training is identical (see Appendix B.1).

Original: All experimental details relevant to this section are included in Appendix D.1.
Revised: All experimental details relevant to this section are included in Appendix D.1.

Original: 4.2. B ETTER MODELLING OF DISCONNECTED SUPPORTS
Revised: 4.2. B ETTER MODELLING OF DISCONNECTED SUPPORTS

Original: As we already showed in subsection 3.1, D-VAEs outperform standard VAEs.
Revised: As we already showed in subsection 3.1, D-VAEs outperform standard VAEs.

Original: The comparisons we have discussed so far were however meant to confirm the union of manifolds hypothesis, and remain unfair as comparisons of generative performance since the D-VAE (classes) model has access to class labels.
Revised: The comparisons we have discussed so far were however meant to confirm the union of manifolds hypothesis, and remain unfair as comparisons of generative performance since the D-VAE (classes) model has access to class labels.

Original: The bottom half of Table 1 includes additional comparisons to show that D-VAEs are not just useful as a way of confirming the disconnectedness of the data support, but that they are also an effective way of accounting for this structure.
Revised: The bottom half of Table 1 includes additional comparisons to show that D-VAEs are not just useful as a way of confirming the disconnectedness of the data support, but that they are also an effective way of accounting for this structure.

Original: We include comparisons against standard conditional
Revised: We include comparisons against standard conditional

Original: VAEs (Sohn et al., 2015), which also have access to class labels (indicated as “Conditional VAE (classes)”), but as additional inputs to their neural networks, rather than having separate models for each class like D-VAEs.
Revised: VAEs (Sohn et al., 2015), which also have access to class labels (indicated as “Conditional VAE (classes)”), but as additional inputs to their neural networks, rather than having separate models for each class like D-VAEs.

Original: We can see that conditional VAEs outperform our D-VAEs only on
Revised: We can see that conditional VAEs outperform our D-VAEs only on

Original: CIFAR-10 and CIFAR-100, highlighting that D-VAEs are performant DGMs, despite their simplicity.
Revised: CIFAR-10 and CIFAR-100, highlighting that D-VAEs are performant DGMs, despite their simplicity.

Original: Importanly, while we believe that D-DGMs are better than conditional models for verifying the disconnectedness of the data support (as they allow for FLOP and memory-equivalent comparisons), conditional VAEs outperforming VAEs provides further evidence supporting the union of manifolds hypothesis.
Revised: Importanly, while we believe that D-DGMs are better than conditional models for verifying the disconnectedness of the data support (as they allow for FLOP and memory-equivalent comparisons), conditional VAEs outperforming VAEs provides further evidence supporting the union of manifolds hypothesis.

Original: Note that conditional VAEs outperform VAEs on CIFAR-100, which also supports our previous conjecture that the poor performance of D-VAEs on this dataset is due to having too few datapoints per class, rather than the union of manifolds hypothesis not holding.
Revised: Note that conditional VAEs outperform VAEs on CIFAR-100, which also supports our previous conjecture that the poor performance of D-VAEs on this dataset is due to having too few datapoints per class, rather than the union of manifolds hypothesis not holding.

Original: We also show results for the fully-unsupervised version of the D-VAE (indicated as “D-VAE (clusters)”), and we can see that it not only outperforms the standard VAE, but that it also outperforms the same conditional VAE, now conditioned on cluster membership rather than class labels for a fully-unsupervised baseline (indicated as “Conditional VAE (clusters)”), even on CIFAR-100 (as we use 10 clusters, rather that 100 ).
Revised: We also show results for the fully-unsupervised version of the D-VAE (indicated as “D-VAE (clusters)”), and we can see that it not only outperforms the standard VAE, but that it also outperforms the same conditional VAE, now conditioned on cluster membership rather than class labels for a fully-unsupervised baseline (indicated as “Conditional VAE (clusters)”), even on CIFAR-100 (as we use 10 clusters, rather that 100 ).

Original: Finally, we also include a comparison against a VAE whose prior P Z is given by a mixture of L Gaussians (indicated as “VAE (GMM)”) instead of a standard
Revised: Finally, we also include a comparison against a VAE whose prior P Z is given by a mixture of L Gaussians (indicated as “VAE (GMM)”) instead of a standard

Original: Gaussian as in the other models.
Revised: Gaussian as in the other models.

Original: Having a multimodal P Z is a popular strategy in pushforward models to better model multimodal target distributions (Nalisnick et al., 2016;
Revised: Having a multimodal P Z is a popular strategy in pushforward models to better model multimodal target distributions (Nalisnick et al., 2016;

Original: Dilokthanakul et al., 2017; Jiang et al., 2017; Ben-Yosef & Weinshall, 2018; Izmailov et al., 2020; Cao et al., 2020; ´
Revised: Dilokthanakul et al., 2017; Jiang et al., 2017; Ben-Yosef & Weinshall, 2018; Izmailov et al., 2020; Cao et al., 2020; ´

Original: Smieja et al., 2020; Potapczynski et al., 2020).
Revised: Smieja et al., 2020; Potapczynski et al., 2020).

Original: We can see that our D-VAE models outperform VAE (GMM) in most cases: this is not too surprising in light of Proposition 1, since the support of P Z remains connected when using mixtures, even if it is now multimodal.
Revised: We can see that our D-VAE models outperform VAE (GMM) in most cases: this is not too surprising in light of Proposition 1, since the support of P Z remains connected when using mixtures, even if it is now multimodal.

Original: Once again, we omit comparisons with other pushforward models here due to space constraints, but highlight that Appendix C.1 includes analogous comparisons with GANs, NFs, and WAEs, in Tables 3, 4, and 5, respectively, where we see the exact same behaviour of disconnected DGMs outperforming or remaining competitive with na¨ıve conditioning and using mixtures as base distributions.
Revised: Once again, we omit comparisons with other pushforward models here due to space constraints, but highlight that Appendix C.1 includes analogous comparisons with GANs, NFs, and WAEs, in Tables 3, 4, and 5, respectively, where we see the exact same behaviour of disconnected DGMs outperforming or remaining competitive with na¨ıve conditioning and using mixtures as base distributions.

Original: 4.2. B ETTER MODELLING OF VARYING INTRINSIC DIMENSIONS
Revised: 4.2. B ETTER MODELLING OF VARYING INTRINSIC DIMENSIONS

Original: While so far we have only focused on the ability of D-DGMs to model disconnected supports, our method also easily allows modelling varying intrinsic dimensions, thus fully accounting for the union of manifolds hypothesis.
Revised: While so far we have only focused on the ability of D-DGMs to model disconnected supports, our method also easily allows modelling varying intrinsic dimensions, thus fully accounting for the union of manifolds hypothesis.

Original: In order to do this, we simply set different dimensions for the latent spaces Z ℓ of each of the L DGMs.
Revised: In order to do this, we simply set different dimensions for the latent spaces Z ℓ of each of the L DGMs.

Original: In particular, we use (2) to obtain an estimate ˆ d ( ℓ ) k of the intrinsic dimension of M ℓ using D ℓ , and then set Z ℓ =
Revised: In particular, we use (2) to obtain an estimate ˆ d ( ℓ ) k of the intrinsic dimension of M ℓ using D ℓ , and then set Z ℓ =

Original: R ˆ d ( ℓ ) k .
Revised: R ˆ d ( ℓ ) k .

Original: Any observed improvement of disconnected
Revised: Any observed improvement of disconnected

Original: DGMs with varying ˆ d ( ℓ ) k s over disconnected DGMs with fixed ˆ d ( ℓ ) k s can thus be attributed to having accounted for varying intrinsic dimensions.
Revised: DGMs with varying ˆ d ( ℓ ) k s over disconnected DGMs with fixed ˆ d ( ℓ ) k s can thus be attributed to having accounted for varying intrinsic dimensions.

Original: For each of the L DGMs ( P ( ℓ ) Z , G ℓ ) , we use two-step models (Dai & Wipf, 2019; Loaiza-Ganem et al., 2022), which were specifically designed to properly model manifold-supported data by avoiding manifold overfitting: we first train an autoencoder-type model as a first step, whose decoder is G ℓ ; and then train a DGM P ( ℓ ) Z on the low-dimensional representations obtained by running D ℓ through the encoder.
Revised: For each of the L DGMs ( P ( ℓ ) Z , G ℓ ) , we use two-step models (Dai & Wipf, 2019; Loaiza-Ganem et al., 2022), which were specifically designed to properly model manifold-supported data by avoiding manifold overfitting: we first train an autoencoder-type model as a first step, whose decoder is G ℓ ; and then train a DGM P ( ℓ ) Z on the low-dimensional representations obtained by running D ℓ through the encoder.

Original: In other words, we use a disconnected version of two-step models as a way to account for varying intrinsic dimensions.
Revised: In other words, we use a disconnected version of two-step models as a way to account for varying intrinsic dimensions.

Original: The fourth panel of
Revised: The fourth panel of

Original: Figure 2 illustrates the benefits of this approach by training a disconnected two-step VAE (indicated as “D-VAE+VAE”).
Revised: Figure 2 illustrates the benefits of this approach by training a disconnected two-step VAE (indicated as “D-VAE+VAE”).

Original: This model is trained by clustering the data to obtain its connected components, estimating the respective intrinsic dimensions as 2 and 1 , and then training a VAE+VAE model on each of these clusters.
Revised: This model is trained by clustering the data to obtain its connected components, estimating the respective intrinsic dimensions as 2 and 1 , and then training a VAE+VAE model on each of these clusters.

Original: In the first cluster (of intrinsic dimension 2 ), the first VAE obtains 2 -dimensional representations, and the second VAE learns the distribution of these representations.
Revised: In the first cluster (of intrinsic dimension 2 ), the first VAE obtains 2 -dimensional representations, and the second VAE learns the distribution of these representations.

Original: The same is done for the second cluster, except the first VAE obtains 1 -dimensional re
Revised: The same is done for the second cluster, except the first VAE obtains 1 -dimensional representations.

Original: In Table 6 in Appendix C.1, we carry out an analogous comparison on image datasets using VAE+NF models.
Revised: In Table 6 in Appendix C.1, we carry out an analogous comparison on image datasets using VAE+NF models.

Original: Surprisingly, we found that while both versions of disconnected DGMs (i.e. with fixed or varying intrinsic dimensions) outperformed non-disconnected two-step models – once again confirming the conclusions about better modelling of disconnected supports – setting intrinsic dimensions differently on every cluster did not improve performance over keeping intrinsic dimension constant across clusters.
Revised: Surprisingly, we found that while both versions of disconnected DGMs (i.e. with fixed or varying intrinsic dimensions) outperformed non-disconnected two-step models – once again confirming the conclusions about better modelling of disconnected supports – setting intrinsic dimensions differently on every cluster did not improve performance over keeping intrinsic dimension constant across clusters.

Original: At a first glance this might appear to imply that there is no benefit to accounting for varying intrinsic dimensions, seemingly contradicting the theoretical result of LoaizaGanem et al.
Revised: At a first glance this might appear to imply that there is no benefit to accounting for varying intrinsic dimensions, seemingly contradicting the theoretical result of LoaizaGanem et al.

Original: (2022, Theorem 1), which states that manifold overfitting will occur whenever intrinsic dimension is overspecified.
Revised: (2022, Theorem 1), which states that manifold overfitting will occur whenever intrinsic dimension is overspecified.

Original: However, upon closer inspection of the proof of Theorem 1 in LoaizaGanem et al.
Revised: However, upon closer inspection of the proof of Theorem 1 in LoaizaGanem et al.

Original: (2022), we can see that the rate at which manifold overfitting happens depends on the difference between ambient and intrinsic dimension.
Revised: (2022), we can see that the rate at which manifold overfitting happens depends on the difference between ambient and intrinsic dimension.

Original: This observation explains both the good empirical results of Loaiza-Ganem et al.
Revised: This observation explains both the good empirical results of Loaiza-Ganem et al.

Original: and our aforementioned results: the former are driven by the difference between ambient and intrinsic dimension (on the order of hundreds/thousands), while the latter by the error between the true and estimated intrinsic dimension (on the order of ones/tens).
Revised: and our aforementioned results: the former are driven by the difference between ambient and intrinsic dimension (on the order of hundreds/thousands), while the latter by the error between the true and estimated intrinsic dimension (on the order of ones/tens).

Original: While a partially negative result, we provide further insight into the connection between the low-dimensional structure of the data and DGMs.
Revised: While a partially negative result, we provide further insight into the connection between the low-dimensional structure of the data and DGMs.

Original: In order to verify that properly setting varying intrinsic dimensions in DGMs is advantageous, provided the true intrinsic dimensions are different enough, we generate a synthetic dataset using a pretrained BigGAN (Brock et al., 2019).
Revised: In order to verify that properly setting varying intrinsic dimensions in DGMs is advantageous, provided the true intrinsic dimensions are different enough, we generate a synthetic dataset using a pretrained BigGAN (Brock et al., 2019).

Original: We use this model, which has Z = R 120 , to generate 64 ,samples from the golden retriever class.
Revised: We use this model, which has Z = R 120 , to generate 64 , 000 samples from the golden retriever class.

Original: These samples have true intrinsic dimension at most 120 , and their estimated intrinsic dimension through (2) is 22 (we rounded the real-valued estimate up).
Revised: These samples have true intrinsic dimension at most 120 , and their estimated intrinsic dimension through (2) is 22 (we rounded the real-valued estimate up).

Original: We also generate another 64 , 000 samples from this model, except we zero out all but m = 1 latent coordinates before passing them through the generator, resulting in samples of true intrinsic dimension of at most m , and estimated intrinsic dimension 3 .
Revised: We also generate another 64 , 000 samples from this model, except we zero out all but m = 1 latent coordinates before passing them through the generator, resulting in samples of true intrinsic dimension of at most m , and estimated intrinsic dimension 3 .

Original: We then combine these two types of samples into a single dataset of size 128 , 000 , with each type of sample corresponding to a class.
Revised: We then combine these two types of samples into a single dataset of size 128 , 000 , with each type of sample corresponding to a class.

Original: This dataset has the property that its classes have a large difference in intrinsic dimension, and estimating the intrinsic dimension of the entire dataset while ignoring classes gives a value of 5 .
Revised: This dataset has the property that its classes have a large difference in intrinsic dimension, and estimating the intrinsic dimension of the entire dataset while ignoring classes gives a value of 5 .

Original: We then train two models on this dataset: a disconnected two-step model, where the first models are 
Revised: We then train two models on this dataset: a disconnected two-step model, where the first models are WAEs with fixed intrinsic dimension set to(i.e.

Original: WAEs with fixed intrinsic dimension set to 5 (i.e. the single estimate of intrinsic dimension over the entire dataset) and the second models NFs (indicated as “D-WAE+NF (constant ˆ d )”); and a disconnected version of the same model except clusters have their latent dimensions set to 22 and 3 (i.e. class intrinsic dimension estimates), respectively (indicated as “D-WAE+NF”).
Revised: the single estimate of intrinsic dimension over the entire dataset) and the second models NFs (indicated as “D-WAE+NF (constant ˆ d )”); and a disconnected version of the same model except clusters have their latent dimensions set to 22 and 3 (i.e. class intrinsic dimension estimates), respectively (indicated as “D-WAE+NF”).

Original: We then repeat this entire process for m = 3 and m = 5 , and show the resulting FID scores in Figure 5 for 3 runs.
Revised: We then repeat this entire process for m = 3 and m = 5 , and show the resulting FID scores in Figure 5 for 3 runs.

Original: We can see that setting the latent dimension of each cluster to its estimated intrinsic dimension results in a large performance improvement versus setting it without accounting for varying intrinsic dimensions.
Revised: We can see that setting the latent dimension of each cluster to its estimated intrinsic dimension results in a large performance improvement versus setting it without accounting for varying intrinsic dimensions.

Original: We can also see that this gap in performance tightens as the difference in true intrinsic dimensions decreases.
Revised: We can also see that this gap in performance tightens as the difference in true intrinsic dimensions decreases.

Original: These results show that considering varying intrinsic dimensions is practically relevant in DGMs, particularly when the difference in true intrinsic dimensions is large.
Revised: These results show that considering varying intrinsic dimensions is practically relevant in DGMs, particularly when the difference in true intrinsic dimensions is large.

Original:  C ONCLUSIONS , LIMITATIONS , AND FUTURE WORK
Revised:  C ONCLUSIONS , LIMITATIONS , AND FUTURE WORK

Original: Conclusions In this work, we have proposed the union of manifolds hypothesis, which extends the standard manifold hypothesis to accommodate data of varying intrinsic dimensions. We first empirically verify this hypothesis by showingboth that  datasets have disconnected supports, and that the intrinsic dimension on the respective connected components can vary widely.
Revised: In this work, we empirically verified the union of manifolds hypothesis by showing that image datasets have multiple connected components whose intrinsic dimensions vary widely.

Original: We also establish uses of the union of manifolds hypothesis, showing that classes of higher intrinsic dimension are harder to classify and how this insight can be used to improve classification accuracy.
Revised: We also establish uses of the union of manifolds hypothesis, showing that classes of higher intrinsic dimension are harder to classify and how this insight can be used to improve classification accuracy.

Original: Finally, we also propose disconnected DGMs, which in addition to being a useful tool in the verification of our main hypothesis, are also a promising direction for DGM research.

Original: We anticipate that the broader deep learning community will further unlock the potential of the union of manifolds hypothesis for both understanding natural data and discovering empirical improvements.
Revised: We anticipate that the broader deep learning community will further unlock the potential of the union of manifolds hypothesis for both understanding natural data and discovering empirical improvements.

Original: Limitations and future work Although we confirmed the union of manifolds hypothesis on commonly-used datasets, we have not tested on anything besides image data; verification of the union of manifolds hypothesis and the efficacy of disconnected DGMs on other types of data remain to be proven.
Revised: Although we confirmed the union of manifolds hypothesis on commonly-used datasets, we have not tested on anything besides image data; verification of the union of manifolds hypothesis and the efficacy of disconnected DGMs on other types of data remain to be proven.

Original: We have reason to believe our improvements will generalize though, as has often been the case with other DGM innovations developed on image-like data.
Revised: We have reason to believe our improvements will generalize though, as has often been the case with other DGM innovations developed on image-like data.

Original: We also highlight that, while our experiments show that classes are a good approximation for connected components, they do not imply an exact match: our results are consistent with some classes overlapping, e.g. 1 s and 7 s on MNIST might belong to the same connected component.
Revised: We also highlight that, while our experiments show that classes are a good approximation for connected components, they do not imply an exact match: our results are consistent with some classes overlapping, e.g. 1 s and 7 s on MNIST might belong to the same connected component.

Original: Additionally, while we improve upon the manifold hypothesis, the realistic scenario of intrinsic dimension varying within a connected component is not covered by the union of manifolds hypothesis, and we believe that further probing data for this structure to be an interesting direction for future work.
Revised: Additionally, while we improve upon the manifold hypothesis, the realistic scenario of intrinsic dimension varying within a connected component is not covered by the union of manifolds hypothesis, and we believe that further probing data for this structure to be an interesting direction for future work.

Original: Another point to note is that, while we did obtain classification accuracy improvements through the union of manifolds hypothesis, these were marginal.
Revised: Another point to note is that, while we did obtain classification accuracy improvements through the union of manifolds hypothesis, these were marginal.

Original: Finally, even though our main goal with disconnected DGMs was to verify the union of manifolds hypothesis, they can likely be improved upon as DGMs in several ways, for example through parameter sharing, or by learning the number of clusters (Salvador & Chan, 2004; Kulis & Jordan, 2012; Villecroze et al., 2022).
Revised: Finally, even though our main goal with disconnected DGMs was to verify the union of manifolds hypothesis, they can likely be improved upon as DGMs in several ways, for example through parameter sharing, or by learning the number of clusters (Salvador & Chan, 2004; Kulis & Jordan, 2012; Villecroze et al., 2022).

Original: A P ROOF OF P ROPOSITION
Revised: A D EFINITION OF SUPPORT AND PROOF OF PROPOSITION

Original: Before giving the standard definition of support we motivate it.
Revised: Before giving the standard definition of support, we motivate it.

Original: Intuitively, the support of a distribution P is “the smallest set” to which P assigns probability 1 .
Revised: Intuitively, the support of a distribution P is “the smallest set” to which P assigns probability 1 .

Original: The idea of “the smallest set” satisfying a property is often formalized by taking the intersection of all sets satisfying the property.
Revised: The idea of “the smallest set” satisfying a property is often formalized by taking the intersection of all sets satisfying the property.

Original: Na¨ıve application of this principle however results in a “definition” of support with undesirable properties.
Revised: Na¨ıve application of this principle however results in a “definition” of support with undesirable properties.

Original: Consider for example the case where P is a standard Gaussian distribution on R .
Revised: Consider for example the case where P is a standard Gaussian distribution on R .

Original: Clearly any definition of support that we use should obey that the support of P is R .
Revised: Clearly any definition of support that we use should obey that the support of P is R .

Original: However, if we simply follow the “definition” of support as the intersection of all sets of probability 1 , we would not obtain this: for any x ∈ R , clearly P ( R \ { x } )
Revised: However, if we simply follow the “definition” of support as the intersection of all sets of probability 1 , we would not obtain this: for any x ∈ R , clearly P ( R \ { x } )

Original: = 1 , yet (cid:84) x ∈ R R \ { x } = ∅ , where ∅ denotes the empty set.
Revised: = 1 , yet (cid:84) x ∈ R R \ { x } = ∅ , where ∅ denotes the empty set.

Original: In other words, arbitrary intersections of sets of probability 1 need not have probability 1 , and in fact can have probability 0 .
Revised: In other words, arbitrary intersections of sets of probability 1 need not have probability 1 , and in fact can have probability 0 .

Original: More abstractly, sets of probability 1 are not closed under arbitrary intersections, which is desirable when defining “the smallest set” satisfying a given property as the intersection of all such sets.
Revised: More abstractly, sets of probability 1 are not closed under arbitrary intersections, which is desirable when defining “the smallest set” satisfying a given property as the intersection of all such sets.

Original: In order to solve this problem, the support is defined as the intersection of all closed sets having probability 1 (closed sets are indeed closed under intersections).
Revised: In order to solve this problem, the support is defined as the intersection of all closed sets having probability 1 (closed sets are indeed closed under intersections).

Original: Definition 1 (Support of probability distributions) :
Revised: Definition 1 (Support of probability distributions) :

Original: Let Z be a topological space.
Revised: Let Z be a topological space.

Original: Equipping Z with its Borel σ -algebra to make it a measurable space, let P be a probability distribution on Z .
Revised: Equipping Z with its Borel σ -algebra to make it a measurable space, let P be a probability distribution on Z .

Original: Let
Revised: Let

Original: C ( P ) be the collection of all closed sets C in Z such that P ( C )
Revised: C ( P ) be the collection of all closed sets C in Z such that P ( C )

Original: = 1 .
Revised: = 1 .

Original: The support of P , denoted as supp ( P ) , is defined as:
Revised: The support of P , denoted as supp ( P ) , is defined as:

Original: [Equation]
Revised: [Equation]

Original: Before continuing, we point out that there exist counterexamples showing that the support of a distribution need not have probability 1 (Schilling & K ¨ uhn, 2021, Examples 6.2 and 6.3).
Revised: Before continuing, we point out that there exist counterexamples showing that the support of a distribution need not have probability 1 (Schilling & K ¨ uhn, 2021, Examples 6.2 and 6.3).

Original: In other words, even though closed sets are closed under intersections, closed sets of probability 1 need not share this property.
Revised: In other words, even though closed sets are closed under intersections, closed sets of probability 1 need not share this property.

Original: These examples are however much more contrived than the one above and of no practical interest (if Z is a separable metric space, as is the case of R d , then any Borel measure P on Z satisfies P ( supp ( P ))
Revised: These examples are however much more contrived than the one above and of no practical interest (if Z is a separable metric space, as is the case of R d , then any Borel measure P on Z satisfies P ( supp ( P ))

Original: = 1 , see Bogachev (2007, Proposition 7.2.9)), and thus support is commonly defined as above.
Revised: = 1 , see Bogachev (2007, Proposition 7.2.9)), and thus support is commonly defined as above.

Original: Note that the assumption that P Z ( supp ( P Z ))
Revised: Note that the assumption that P Z ( supp ( P Z ))

Original: = 1 could be replaced by the requirement that Z be a separable metric space and P Z a Borel measure on it (which covers all settings of practical interest), although this would yield a slightly less general result.
Revised: = 1 in Propositioncould be replaced by the requirement that Z be a separable metric space and P Z a Borel measure on it (which covers all settings of practical interest), although this would yield a slightly less general result.

Original: We restate Proposition 1 below for convenience:
Revised: We restate Proposition 1 below for convenience:

Original: Proposition 1 : Let Z and X be topological spaces and G : Z → X be continuous.
Revised: Proposition 1 : Let Z and X be topological spaces and G : Z → X be continuous.

Original: Considering Z and X as measurable spaces with their respective Borel σ -algebras, let P Z be a probability measure on Z such that supp ( P Z ) is connected and P Z ( supp ( P Z ))
Revised: Considering Z and X as measurable spaces with their respective Borel σ -algebras, let P Z be a probability measure on Z such that supp ( P Z ) is connected and P Z ( supp ( P Z ))

Original: = 1 .
Revised: = 1 .

Original: Then supp ( G # P Z ) is connected.
Revised: Then supp ( G # P Z ) is connected.

Original: Proof : As mentioned in the proof sketch in the main manuscript, showing that supp ( G # P Z ) =
Revised: Proof : As mentioned in the proof sketch in the main manuscript, showing that supp ( G # P Z ) =

Original: cl ( G ( supp ( P Z ))) , where cl ( · ) denotes closure in X , is enough, since by continuity of G ,
Revised: cl ( G ( supp ( P Z ))) , where cl ( · ) denotes closure in X , is enough, since by continuity of G ,

Original: G ( supp ( P Z )) is connected, and the closure of connected sets is itself connected.
Revised: G ( supp ( P Z )) is connected, and the closure of connected sets is itself connected.

Original: We begin by verifying that cl ( G ( supp ( P Z )))
Revised: We begin by verifying that cl ( G ( supp ( P Z )))

Original: ⊆ supp ( G # P Z ) .
Revised: ⊆ supp ( G # P Z ) .

Original: Let C ∈ C
Revised: Let C ∈ C

Original: ( G # P Z ) .
Revised: ( G # P Z ) .

Original: By definition of
Revised: By definition of

Original: C ( G # P Z ) , C is closed in X and is such that G # P Z ( C )
Revised: C ( G # P Z ) , C is closed in X and is such that G # P Z ( C )

Original: = 1 .
Revised: = 1 .

Original: By definition of pushforward measure, it follows that:
Revised: By definition of pushforward measure, it follows that:

Original: [Equation]
Revised: [Equation]

Original: and by continuity of G , it also follows that G − 1 ( C ) is closed in Z .
Revised: and by continuity of G , it also follows that G − 1 ( C ) is closed in Z .

Original: Then, G − 1 ( C ) ∈ C ( P Z ) , and by definition of support, supp ( P Z ) ⊆
Revised: Then, G − 1 ( C ) ∈ C ( P Z ) , and by definition of support, supp ( P Z ) ⊆

Original: G − 1 ( C ) .
Revised: G − 1 ( C ) .

Original: It then follows that G ( supp ( P Z )) ⊆
Revised: It then follows that G ( supp ( P Z [Equation]

Original:  G ( G − 1 ( C )) ⊆ C .
Revised: ⊆ G ( G − 1 ( C )) ⊆ C .

Original: Since this holds for every C ∈ C ( G # P Z ) , we have that:
Revised: Since this holds for every C ∈ C ( G # P Z ) , we have that:

Original: [Equation]
Revised: [Equation]

Original: Applying cl ( · ) on both sides, and using the fact that supp ( G # P Z ) is closed in X (since it is an intersection of closed sets), yields cl ( G ( supp ( P Z )))
Revised: Applying cl ( · ) on both sides, and using the fact that supp ( G # P Z ) is closed in X (since it is an intersection of closed sets), yields cl ( G ( supp ( P Z )))

Original: ⊆ supp ( G # P Z ) .
Revised: ⊆ supp ( G # P Z ) .

Original: It now only remains to show that supp ( G # P Z ) ⊆ cl ( G ( supp ( P Z ))) .
Revised: It now only remains to show that supp ( G # P Z ) ⊆ cl ( G ( supp ( P Z ))) .

Original: By definition of pushforward measure, we have that:
Revised: By definition of pushforward measure, we have that:

Original: [Equation]
Revised: [Equation]

Original: where the inequality follows from supp ( P Z ) ⊆
Revised: where the inequality follows from supp ( P Z ) ⊆

Original: G − 1 ( G ( supp ( P Z ))) .
Revised: G − 1 ( G ( supp ( P Z ))) .

Original: Since G ( supp ( P Z )) ⊆ cl ( G ( supp ( P Z ))) , then G # P Z ( cl ( G ( supp ( P Z ))))
Revised: Since G ( supp ( P Z )) ⊆ cl ( G ( supp ( P Z ))) , then G # P Z ( cl ( G ( supp ( P Z ))))

Original: = 1 .
Revised: = 1 .

Original: It follows that cl ( G ( supp ( P Z ))) ∈
Revised: It follows that cl ( G ( supp ( P Z ))) ∈

Original: C ( G # P Z ) , and thus supp ( G # P Z ) ⊆ cl ( G ( supp ( P Z ))) by definition of support.
Revised: C ( G # P Z ) , and thus supp ( G # P Z ) ⊆ cl ( G ( supp ( P Z ))) by definition of support.

Original: B D ISCONNECTED DGM S : D ETAILS B. T RAINING DISCONNECTED DGM S
Revised: B D ISCONNECTED DGM S : D ETAILS B. T RAINING DISCONNECTED DGM S

Revised: Input: clustering algorithm ( · ) , D

Revised: Output: { ( P ( ℓ ) Z , G ℓ ) } Lℓ = 1 D 1 , . . .

Revised: , D L ← clustering algorithm ( D ) 2 for ℓ = 1 to L do

Revised: Potentially initialize P ( ℓ ) Z and G ℓ

Revised: Train G ℓ and potentially P ( ℓ ) Z on D ℓ 5 end

Original: Training We summarize the training procedure for disconnected DGMs that we mentioned in the main manuscript in Algorithm 1.
Revised: Training We summarize the training procedure for disconnected DGMs that we mentioned in the main manuscript in Algorithm 1.

Original: Note that we use the exact same computational budget to train a disconnected DGM { ( P ( ℓ ) Z , G ℓ ) } Lℓ =1 as we do to train an equivalent non-disconnected baseline ( P Z , G ) .
Revised: Note that we use the exact same computational budget to train a disconnected DGM { ( P ( ℓ ) Z , G ℓ ) } Lℓ =1 as we do to train an equivalent non-disconnected baseline ( P Z , G ) .

Original: We begin with the observation that due to the serial nature of Algorithm 1, all L models need not be in memory when training a disconnected DGM: once the ℓ th model has been trained, it can be moved to disk before loading the ( ℓ + 1) th model to memory.
Revised: We begin with the observation that due to the serial nature of Algorithm 1, all L models need not be in memory when training a disconnected DGM: once the ℓ th model has been trained, it can be moved to disk before loading the ( ℓ + 1) th model to memory.

Original: In other words, even though disconnected DGMs do have more parameters than their non-disconnected counterparts and thus require more disk space to be stored, it is not needed at any point during training to load more than a single model in memory.
Revised: In other words, even though disconnected DGMs do have more parameters than their non-disconnected counterparts and thus require more disk space to be stored, it is not needed at any point during training to load more than a single model in memory.

Original: We also make the key observation that, if using the same architecture throughout (i.e. G and every G ℓ have the same architecture, as do P Z and P ( ℓ ) Z if they are trainable), then training the non-disconnected DGM ( P Z , G ) for N epochs on D requires the same amount of compute as training each ( P ( ℓ ) Z , G ℓ ) for N epochs, since every ( P ( ℓ ) Z , G ℓ ) is trained on a smaller dataset D ℓ .
Revised: We also make the key observation that, if using the same architecture throughout (i.e. G and every G ℓ have the same architecture, as do P Z and P ( ℓ ) Z if they are trainable), then training the non-disconnected DGM ( P Z , G ) for N epochs on D requires the same amount of compute as training each ( P ( ℓ ) Z , G ℓ ) for N epochs, since every ( P ( ℓ ) Z , G ℓ ) is trained on a smaller dataset D ℓ .

Original: To see this in more detail, assume training ( P Z , G ) for a single epoch requires T gradient steps, so the total training cost is TN gradient steps.
Revised: To see this in more detail, assume training ( P Z , G ) for a single epoch requires T gradient steps, so the total training cost is TN gradient steps.

Original: In order to train a single ( P ( ℓ ) Z , G ℓ ) model for a single epoch, since D ℓ is |D| / |D ℓ | times smaller than D , only |D ℓ | / |D| T gradient steps are required.
Revised: In order to train a single ( P ( ℓ ) Z , G ℓ ) model for a single epoch, since D ℓ is |D| / |D ℓ | times smaller than D , only |D ℓ | / |D| T gradient steps are required.

Original: Thus, training a single ( P ( ℓ ) Z , G ℓ ) model for N epochs requires |D ℓ | / |D| TN gradient steps, and it follows that training all the ( P ( ℓ ) Z , G ℓ ) for ℓ = 1 , . . .
Revised: Thus, training a single ( P ( ℓ ) Z , G ℓ ) model for N epochs requires |D ℓ | / |D| TN gradient steps, and it follows that training all the ( P ( ℓ ) Z , G ℓ ) for ℓ = 1 , . . .

Original: , L requires
Revised: , L requires

Original: [Equation]
Revised: [Equation]

Original: gradient steps.
Revised: gradient steps.

Original: Finally, since exactly the same number of steps are required to train disconnected and non-disconnected models, and these models share architectures, it follows that the FLOP and memory costs of training them are indeed equivalent under our setup.
Revised: Finally, since exactly the same number of steps are required to train disconnected and non-disconnected models, and these models share architectures, it follows that the FLOP and memory costs of training them are indeed equivalent under our setup.

Original: Clustering algorithm
Revised: Clustering algorithm

Original: In order to generate the dataset clusters to train each component in our disconnected DGMs (with the goal of approximating the connected components of the full dataset) without labels, we perform agglomerative clustering (Rokach & Maimon, 2005) on the dataset.
Revised: In order to generate the dataset clusters to train each component in our disconnected DGMs (with the goal of approximating the connected components of the full dataset) without labels, we perform agglomerative clustering (Rokach & Maimon, 2005) on the dataset.

Original: At the start of the algorithm, each datapoint is in its own cluster.
Revised: At the start of the algorithm, each datapoint is in its own cluster.

Original: Then, at every step, the two clusters with the smallest linkage distance are merged, decreasing the number of clusters by 1.
Revised: Then, at every step, the two clusters with the smallest linkage distance are merged, decreasing the number of clusters by 1.

Original: This occurs until a pre-specified number of clusters, L , is reached.
Revised: This occurs until a pre-specified number of clusters, L , is reached.

Original: For all of our datasets, we set L = 10 which is the number of classes for the MNIST, FMNIST, SVHN, and CIFAR-10 datasets.
Revised: For all of our datasets, we set L = 10 which is the number of classes for the MNIST, FMNIST, SVHN, and CIFAR-10 datasets.

Original: We experimented with ranges of L between 7 and 15 and did not find a large variation in performance.
Revised: We experimented with ranges of L between 7 and 15 and did not find a large variation in performance.

Original: Automatically inferring L from a dataset is the subject of future work and would make our clustering models completely unsupervised.
Revised: Automatically inferring L from a dataset is the subject of future work and would make our clustering models completely unsupervised.

Original: For all of our experiments involving agglomerative clustering, we used
Revised: For all of our experiments involving agglomerative clustering, we used

Original: Ward’s linkage criterion (Ward Jr, 1963).
Revised: Ward’s linkage criterion (Ward Jr, 1963).

Original: The distance between two clusters with this criterion is the variance of the Euclidean distance between all datapoints in the clusters being merged.
Revised: The distance between two clusters with this criterion is the variance of the Euclidean distance between all datapoints in the clusters being merged.

Original: Therefore, at each step, the two clusters with the smallest variance will be combined.
Revised: Therefore, at each step, the two clusters with the smallest variance will be combined.

Original: We experimented with using single linkage, which defines the distance between two clusters as the minimum distance between a datapoint in both clusters, but found that this led to very unbalanced cluster sizes and had very poor performance when training our DGMs.
Revised: We experimented with using single linkage, which defines the distance between two clusters as the minimum distance between a datapoint in both clusters, but found that this led to very unbalanced cluster sizes and had very poor performance when training our DGMs.

Original: We also tried a custom linkage metric that merges the two clusters at each step to maximize the inter-cluster intrinsic dimension estimate variance, with the intuition that building clusters with a large variance in intrinsic dimension would better model the submanifolds of the dataset.
Revised: We also tried a custom linkage metric that merges the two clusters at each step to maximize the inter-cluster intrinsic dimension estimate variance, with the intuition that building clusters with a large variance in intrinsic dimension would better model the submanifolds of the dataset.

Original: In practice, we found this method very sensitive to initialization and saw the same unbalanced cluster failure mode as the single linkage criterion.
Revised: In practice, we found this method very sensitive to initialization and saw the same unbalanced cluster failure mode as the single linkage criterion.

Original: For all of our experiments, we also tried using the k-means++ (Arthur & Vassilvitskii, 2007) clustering algorithm.
Revised: For all of our experiments, we also tried using the k-means++ (Arthur & Vassilvitskii, 2007) clustering algorithm.

Original: This also worked well but gave slightly worse results than agglomerative clustering with Ward’s criterion across the board.
Revised: This also worked well but gave slightly worse results than agglomerative clustering with Ward’s criterion across the board.

Original: B. S 
Revised: B. S AMPLING FROM DISCONNECTED DGM S

Original: trained cluster sizes

Original: ∼
Revised: Input: m , trained disconnected DGM { ( P ( ℓ ) Z , G ℓ ) } Lℓ =1 , and corresponding cluster sizes |D 1 | , . . . , |D L | . Output: S 1 S ← ∅ 2 ( m 1 , . . .

Original:  (cid:32) m, (cid:32) |D 1 | (cid:80)
Revised: , m L ) ∼ Multinomial (cid:32) m, (cid:32) |D 1 | (cid:80)

Original: Lℓ ′
Revised: Lℓ ′

Original: =
Revised: =

Original: 1 |D ℓ ′ | , . . .
Revised: 1 |D ℓ ′ | , . . .

Original: , |D L | (cid:80)
Revised: , |D L | (cid:80)

Original: Lℓ ′
Revised: Lℓ ′

Original: =
Revised: =

Original: 1 |D ℓ ′ | (cid:33)(cid:33) 3 for  do for  do
Revised: 1 |D ℓ ′ | (cid:33)(cid:33) 3 for ℓ = 1 to L do for t = 1 to m ℓ do

Original: Z ∼ P (  Z
Revised: Z ∼ P ( ℓ ) Z

Original: X = G ℓ ( Z )
Revised: X = G ℓ ( Z )

Original: S ← S ∪ { X } end 9 end
Revised: S ← S ∪ { X } end 9 end

Original: As mentioned in the main manuscript,the sampling process described in (4) can be made memoryefficient.
Revised: As mentioned in the main manuscript, sampling from a disconnected DGM is achieved by first sampling ℓ with probability proportional to |D ℓ | , and then sampling from the corresponding DGM: ℓ ∼ p ( ℓ ) = |D ℓ | (cid:80) Lℓ ′

Original: We now describe how to achieve this.
Revised: = 1 |D ℓ ′ | , Z | ℓ ∼ P ( ℓ ) Z , and X = G ℓ ( Z ) . Sampling from (9) can be made memory-efficient, and we now describe how to do so.

Original: Assume we want to generate m samples from a trained disconnected DGM model { ( P ( ℓ ) Z , G ℓ ) } Lℓ =1 .
Revised: Assume we want to generate m samples from a trained disconnected DGM model { ( P ( ℓ ) Z , G ℓ ) } Lℓ =1 .

Original: The idea is to make sure that each of the L DGMs are only loaded into memory once and one at a time, which requires first knowing exactly how many samples will be required from each model.
Revised: The idea is to make sure that each of the L DGMs are only loaded into memory once and one at a time, which requires first knowing exactly how many samples will be required from each model.

Original: Note that, if we denote the number of samples coming from the ℓ th model as m ℓ , then clearly ( m 1 , . . .
Revised: Note that, if we denote the number of samples coming from the ℓ th model as m ℓ , then clearly ( m 1 , . . .

Original: , m L ) follows a multinomial distribution with parameters m and ( |D 1 | , . . .
Revised: , m L ) follows a multinomial distribution with parameters m and ( |D 1 | , . . .

Original: , |D L | ) / (cid:80) Lℓ ′ = 1 |D ℓ ′ | .
Revised: , |D L | ) / (cid:80) L ℓ ′ =1 |D ℓ ′ | .

Original: We can thus first sample from this multinomial, and then sample the appropriate number of times from each model, as described in Algorithm 2.
Revised: We can thus first sample from this multinomial, and then sample the appropriate number of times from each model, as described in Algorithm 2.

Original: Note that, similarly to Algorithm 1, the outer for loop in Algorithm 2 can be trivially made memoryefficient by loading the required model ( P ( ℓ ) Z , G ℓ ) into memory and then removing it from memory once it has been sampled from m ℓ times (and the inner for loop can be trivially parallelized).
Revised: Note that, similarly to Algorithm 1, the outer for loop in Algorithm 2 can be trivially made memory-efficient by loading the required model ( P ( ℓ ) Z , G ℓ ) into memory and then removing it from memory once it has been sampled from m ℓ times (and the inner for loop can be trivially parallelized).

Original: C A DDITIONAL RESULTS C. D ISCONNECTED DGM S
Revised: C A DDITIONAL RESULTS C. D ISCONNECTED DGM S

Original: Tables 3, 4, and 5 show analogous results to those of Table 1 for GANs, NFs, and WAEs, respectively.
Revised: Tables 3, 4, and 5 show analogous results to those of Table 1 for GANs, NFs, and WAEs, respectively.

Original: We can see that, modulo a few exceptions, the same results observed for VAEs in the main manuscript also hold for these pushforward models, further confirming the disconnectedness of the support of the data, and that disconnected DGMs are a performant class of DGMs.
Revised: We can see that, modulo a few exceptions, the same results observed for VAEs in the main manuscript also hold for these pushforward models, further confirming the disconnectedness of the support of the data, and that disconnected DGMs are a performant class of DGMs.

Original: Table 6 shows results of running a VAE+NF two-step model mentioned in subsubsection 4.2.2.
Revised: Table 6 shows results of running a VAE+NF two-step model mentioned in subsubsection 4.2.2.

Original: We can see that the best performing disconnected versions still mostly outperform the non-disconnected version.
Revised: We can see that the best performing disconnected versions still mostly outperform the non-disconnected version.

Original: This is consistent with our previous results and highlights once again the relevance of accounting for multiple connected components.
Revised: This is consistent with our previous results and highlights once again the relevance of accounting for multiple connected components.

Original: As mentioned in the main manuscript though, there is no significant improvement between fixing the latent dimension of every cluster to its estimated intrinsic  dimension 
Revised: As mentioned in the main manuscript though, there is no significant improvement between fixing the latent dimension of every cluster to its estimated intrinsic dimension, and simply fixing the intrinsic dimension of every cluster to the same value (the estimated intrinsic dimension of the entire dataset)

Original: [Table]
Revised: [Table]

Original: and simply fixing the intrinsic dimension of every cluster to the same value (the estimated intrinsic dimension of the entire dataset)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C. I NTRINSIC DIMENSION ESTIMATES
Revised: C. I NTRINSIC DIMENSION ESTIMATES

Revised: Variability of intrinsic dimension estimates In order to ensure that the variability of estimated intrinsic dimensions across classes observed in Figure 3 is caused by the true intrinsic dimensions being different rather than the variance of the estimator we used, we carry out the following experiment: For each dataset and each of their classes, we randomly subsample (without replacement) 1000 datapoints (except for CIFAR-100 where use use 300 instead, as there are only 500 datapoints per class), and estimate the corresponding class intrinsic dimension using only this reduced dataset.

Revised: We repeat this process an additional 10 times, and compute the variance of estimated intrinsic dimension across classes (i.e. inter-class), and across subsamples (i.e. intra-class).

Revised: Observing much lower variability across subsamples – as we do in Figure 6 for all datasets except SVHN – shows that the variability observed across classes cannot be explained by the variance of the estimator, once again supporting the union of manifolds hypothesis.

Revised: Note that we sample without replacement to avoid duplicates, as these would result in a distance of 0 to their nearest neighbour.

Original:  We show a more granular breakdown of Figure 3, with intrinsic dimension estimate values for each class in Figure6 for MNIST, Figure 7 for FMNIST, Figure 8 for SVHN
Revised: Intrinsic dimension of specific classes We show a more granular breakdown of Figure 3, with intrinsic dimension estimate values for each class in Figure 7 for MNIST, Figure 8 for FMNIST,

Original: [Table] and Figure 9 for  CIFAR-10.
Revised: Figure 9 for SVHN, and Figure 10 for CIFAR-10.

Original: Since CIFAR-100 and ImageNet have too many classes to show, we only include the top 5 highest and lowest intrinsic dimension classes in Figure 10 for CIFAR-100, and Figure 11 for ImageNet.[Table]
Revised: Since CIFAR-100 and ImageNet have too many classes to show, we only include the top 5 highest and lowest intrinsic dimension classes in Figurefor CIFAR-100, and Figure 12 for ImageNet.

Original: D E XPERIMENTAL DETAILS D. D ISCONNECTED DGM S
Revised: D E XPERIMENTAL DETAILS D. D ISCONNECTED DGM S

Original: For all models, we randomly select 10% of the training dataset to be used for validation and train on the remaining 90% .
Revised: For all models, we randomly select 10% of the training dataset to be used for validation and train on the remaining 90% .

Original: We use a separate test dataset to report all metrics for our models.
Revised: We use a separate test dataset to report all metrics for our models.

Original: A batch size of 128 is used for all datasets.
Revised: A batch size of 128 is used for all datasets.

Original: Unless otherwise noted, at the beginning of training, we scale all the data to between 0 and 1 .
Revised: Unless otherwise noted, at the beginning of training, we scale all the data to between 0 and 1 .

Original: For all experiments, we use the Adam optimizer, typically with learning rate 0 .
Revised: For all experiments, we use the ADAM optimizer (Kingma & Ba, 2015), typically with learning rate 0 .

Original: 001 and cosine annealing for a maximum of 100 epochs.
Revised: 001 and cosine annealing for a maximum of 100 epochs.

Original: We also use gradient norm clipping with a value of 10 .
Revised: We also use gradient norm clipping with a value of 10 .

Original: Simulated Data To generate the ground truth data, we create a uniform mixture of two components.
Revised: Simulated Data To generate the ground truth data, we create a uniform mixture of two components.

Original: The first is a two dimensional square of width 1 centered at (0 , 0 .
Revised: The first is a two dimensional square of width 1 centered at (0 , 0 .

Original: The distribution along the x and y axis are independent beta distributions with ( α, β ) set to (2 , 2) for the y axis and (2 , 5) for the x axis.
Revised: The distribution along the x and y axis are independent beta distributions with ( α, β ) set to (2 , 2) for the y axis and (2 , 5) for the x axis.

Original: The second component is a one dimensional sinusoid (with y as the input and x as the output) that is one unit in length centred at 0 .
Revised: The second component is a one dimensional sinusoid (with y as the input and x as the output) that is one unit in length centred at 0 .

Original: 5 with an amplitude of 0 .
Revised: 5 with an amplitude of 0 .

Original: 5 and a period of 1 .
Revised: 5 and a period of 1 .

Original: The distribution along this line is a beta distribution with ( α, β ) set to (2 , 5) .
Revised: The distribution along this line is a beta distribution with ( α, β ) set to (2 , 5) .

Original: We generate 10 , 000 samples for training, 10 , 000 for validation and 5 , 000 for testing.
Revised: We generate 10 , 000 samples for training, 10 , 000 for validation and 5 , 000 for testing.

Original: We train three models on this simulated data.
Revised: We train three models on this simulated data.

Original: For the baseline model we use a VAE with MLPs for the encoder and decoder, with ReLU activations.
Revised: For the baseline model we use a VAE with MLPs for the encoder and decoder, with ReLU activations.

Original: The encoder and decoder each have a single hidden layer with 25 units.
Revised: The encoder and decoder each have a single hidden layer with 25 units.

Original: The latent dimension of the model is set to 2 .
Revised: The latent dimension of the model is set to 2 .

Original: The learning rate is set to 0 . 0001 .
Revised: The learning rate is set to 0 . 0001 .

Original: We do not use early stopping and train forepochs.
Revised: We do not use early stopping and train forepochs.

Original: For the second model, we train a clustering VAE with two components trained using the same setup as the baseline model.
Revised: For the second model, we train a clustering VAE with two components trained using the same setup as the baseline model.

Original: For the third model we train a disconnected VAE+VAE where each component is a two step VAE+VAE model.
Revised: For the third model we train a disconnected VAE+VAE where each component is a two step VAE+VAE model.

Original: Both VAEs have the same architecture and are each trained the same way as the baseline model.
Revised: Both VAEs have the same architecture and are each trained the same way as the baseline model.

Original: The latent dimension of the GAE is obtained by running the MLE intrinsic dimension estimator 2 on each component.
Revised: The latent dimension of the GAE is obtained by running the MLE intrinsic dimension estimator 2 on each component.

Original: VAEs For MNIST and FMNIST, we use MLPs for the encoder and decoder, with ReLU activations.
Revised: VAEs For MNIST and FMNIST, we use MLPs for the encoder and decoder, with ReLU activations.

Original: The encoder and decoder have each a single hidden layer with 512 units.
Revised: The encoder and decoder have each a single hidden layer with 512 units.

Original: For SVHN, CIFAR-100 and
Revised: For SVHN, CIFAR-100 and

Original: CIFAR-10, we use convolutional networks.
Revised: CIFAR-10, we use convolutional networks.

Original: The encoder and decoder have 4 convolutional layers with (32 , 32 , 16 , 16) and (16 , 16 , 32 , 32) channels, respectively, followed by a flattening operation and a fully-connected layer.
Revised: The encoder and decoder have 4 convolutional layers with (32 , 32 , 16 , 16) and (16 , 16 , 32 , 32) channels, respectively, followed by a flattening operation and a fully-connected layer.

Original: The convolutional networks also use ReLU activations, and have kernel size 3 and stride 1 .
Revised: The convolutional networks also use ReLU activations, and have kernel size 3 and stride 1 .

Original: The latent dimension of the model is set to 20 .
Revised: The latent dimension of the model is set to 20 .

Original: We do not use early stopping and always train for 100 epochs.
Revised: We do not use early stopping and always train for 100 epochs.

Original: For the GMM, we set the prior to have 10 modes with learnable mixture weights, means and standard deviations.
Revised: For the GMM, we set the prior to have 10 modes with learnable mixture weights, means and standard deviations.

Original: The mixture components of the prior were initialized with fixed means spaced 3 units apart centred at 0 , standard deviations of 1 and uniform mixture weights.
Revised: The mixture components of the prior were initialized with fixed means spaced 3 units apart centred at 0 , standard deviations of 1 and uniform mixture weights.

Original: GANs For all datasets we use a 3-layerfully convolutional generator and discriminator.
Revised: GANs For all datasets we use a 3-layer convolutional generator and discriminator.

Original: Each layer is composed of a convolution (transpose convolution for the discriminator), instance normalization and a leaky relu activation with slope set to 0.01.
Revised: Each layer is composed of a convolution (transpose convolution for the generator), instance normalization and a leaky relu activation with slope set to 0.01.

Original: For the generator, the number of filters was set to 256, 128, 64 with kernel sizes of 4, 4, 4 and strides 2, 2, 2.
Revised: For the generator, the number of filters was set to 256, 128, 64 with kernel sizes of 4, 4, 4 and strides 2, 2, 2.

Original: The discriminator followed the same hyperparameters in reverse order.
Revised: The discriminator followed the same hyperparameters in reverse order.

Original: The generator had a final convolution to reduce the number of channels to that of the data followed by a Tanh activation.
Revised: The generator had a final convolution to reduce the number of channels to that of the data followed by a tanh activation.

Original: The discriminator had a finaltranspose convolution to reduce the number of channels to one.
Revised: The discriminator had a final convolution to reduce the number of channels to one.

Original: The latent dimension was set to 100.
Revised: The latent dimension was set to 100.

Original: We trained the generator for one step every five discriminator steps.
Revised: We trained the generator for one step every five discriminator steps.

Original: We trained using the Adam optimizer with betas set to (0.5,0.999) with a constant learning rate of 0.001.
Revised: We trained using the ADAM optimizer with betas set to (0.5,0.999) with a constant learning rate of 0.001.

Original: We train used the WGAN loss and gradient penalty with lambda set to 10.
Revised: We train using the WGAN loss and gradient penalty with lambda set to 10.

Original: For the SVHN, CIFAR-10 and CIFAR-
Revised: For the SVHN, CIFAR-10 and CIFAR

Original:  datasets, we use a whitening transform at the start of training to make the data zero-mean and marginally unit-variance; note that this affine transformation does not affect the manifold structure of the data.
Revised: -100 datasets, we use a whitening transform at the start of training to make the data zero-mean and marginally unit-variance; note that this affine transformation does not affect the manifold structure of the data.

Original: NFs We use a rational quadratic spline flow (Durkan et al with  hidden 
Revised: NFs We use a rational quadratic spline flow (Durkan et al., 2019) with 64 hidden units, 4 layers, and 3 blocks per layer.

Original: units , 4 layers, and 3 blocks per layer. For the SVHN, CIFAR-10 and CIFAR-100 datasets, we use a whitening transform at the start of training to make the data zero-mean and marginally unit-variance; note that this affine transformation does not affect the manifold structure of the data.
Revised: For the SVHN, CIFAR-10 and CIFAR-100 datasets, we use a whitening transform at the start of training to make the data zero-mean and marginally unit-variance; note that this affine transformation does not affect the manifold structure of the data.

Original: Otherwise we scale the data followed by a logit transform with epsilon set to 0 .
Revised: Otherwise we scale the data followed by a logit transform with epsilon set to 0 .

Original: We configure the mixture base distribution in the same way as the VAE prior.
Revised: We configure the mixture base distribution in the same way as the VAE prior.

Original: We set the initial learning rate to 0 . 0005 .
Revised: We set the initial learning rate to 0 . 0005 .

Original: WAEs For all datasets, we use an MLP with 2 hidden layers of size 256 each for the discriminator.

Original: Theencoder anddecoder for all datasets are the same convolutional networks used for VAEs except we increase the channels to (64 , 64 , 32 , 32) and (32 , 32 , 64 , 64) respectively.
Revised: The and datasets are the same convolutional networks used for VAEs except we increase the channels to (64 , 64 , 32 , 32) and (32 , 32 , 64 , 64) respectively.

Original: The latent dimension of the model is also set to 20 .
Revised: The latent dimension of the model is also set to 20 .

Original: We train every model for a total of 300 epochs and perform early stopping on the reconstruction error with a patience of 30 epochs only for the GMM baselines.
Revised: We train every model for a total of 300 epochs and perform early stopping on the reconstruction error with a patience of 30 epochs only for the GMM baselines.

Original: We note that when training for the full 300 epochs (without early stopping) the GMM baselines perform significantly worse and the other models perform better.
Revised: We note that when training for the full 300 epochs (without early stopping) the GMM baselines perform significantly worse and the other models perform better.

Original: We use a learning rate of 0 . 0005 for the discriminator and do not use any learning rate schedule for the model.
Revised: We use a learning rate of 0 . 0005 for the discriminator and do not use any learning rate schedule for the model.

Original: We weight the discriminator loss with a lambda value of 10 .
Revised: We weight the discriminator loss with a lambda value of 10 .

Original: For the GMM, we set the prior to have 10 modes with fixed means spaced 3 units apart centred at 0 , fixed standard deviations of 1 and uniform mixture weights.
Revised: For the GMM, we set the prior to have 10 modes with fixed means spaced 3 units apart centred at 0 , fixed standard deviations of 1 and uniform mixture weights.

Original: We attempted to learn these parameters similar to the VAE but observed worse results.
Revised: We attempted to learn these parameters similar to the VAE but observed worse results.

Original: VAE+NF For the VAE, we followed the same setup as the single VAE except we keep the learning rate fixed to 0 .
Revised: VAE+NF For the VAE, we followed the same setup as the single VAE except we keep the learning rate fixed to 0 .

Original: 001 throughout training.
Revised: 001 throughout training.

Original: The NF density estimator is also identical to the single NF with the following differences: (1) for all datasets, we standardize the data (by subtracting the mean and dividing by the standard deviation) and do not scale the data or apply a logit transform, and (2) we set the learning rate to 0 . 001 .
Revised: The NF density estimator is also identical to the single NF with the following differences: (1) for all datasets, we standardize the data (by subtracting the mean and dividing by the standard deviation) and do not scale the data or apply a logit transform, and (2) we set the learning rate to 0 . 001 .

Original: WAE+NF For the WAE, we followed the same setup as the single WAE except we use residual networks  to handle the larger and more complex golden retriever datasets.
Revised: WAE+NF For the WAE, we followed the same setup as the single WAE except we use residual networks (He et al., 2016) to handle the larger and more complex golden retriever datasets.

Original: The encoder first applies a convolution (increasing the channel dimension to 16 ), batch normalization, and ReLU activation to the input before a max pool operation with stride 2 and kernel size of 3 .
Revised: The encoder first applies a convolution (increasing the channel dimension to 16 ), batch normalization, and ReLU activation to the input before a max pool operation with stride 2 and kernel size of 3 .

Original: The main body of the encoder is a set of 4 residual layers with channel dimensions (16 , 32 , 64 , 128) .
Revised: The main body of the encoder is a set of 4 residual layers with channel dimensions (16 , 32 , 64 , 128) .

Original: Each layer consists of two basic blocks which are two consecutive convolution, BatchNorm  and ReLU layers where the input of the block is added to the activation before the final ReLU.
Revised: Each layer consists of two basic blocks which are two consecutive convolution, BatchNorm (Ioffe & Szegedy, 2015) and ReLU layers where the input of the block is added to the activation before the final ReLU.

Original: The stride of the first convolution of each layer is set to 2 to halve the spatial resolution.
Revised: The stride of the first convolution of each layer is set to 2 to halve the spatial resolution.

Original: The skip connection is downsampled by a convolution with a stride of 2 .
Revised: The skip connection is downsampled by a convolution with a stride of 2 .

Original: The activations are then average-pooled across each channel and projected to the latent space dimension by a linear layer.
Revised: The activations are then average-pooled across each channel and projected to the latent space dimension by a linear layer.

Original: The decoder is identical to the encoder main body except there are no downsampling operations, the spatial resolution of the activation is doubled at each layer by bilinear upsampling and the channel dimensions are (128 , 64 , 32 , 16) .
Revised: The decoder is identical to the encoder main body except there are no downsampling operations, the spatial resolution of the activation is doubled at each layer by bilinear upsampling and the channel dimensions are (128 , 64 , 32 , 16) .

Original: We perform early stopping on validation loss with a patience of 30 epochs, for a maximum of 300 epochs.
Revised: We perform early stopping on validation loss with a patience ofepochs, for a maximum of 300 epochs.

Original: The NF density estimator is also identical to the single NF with the following differences: (1) for all datasets, we standardize the data (by subtracting the mean and dividing by the standard deviation) and do not scale the data or apply a logit transform, (2) we set the learning rate to 0 .
Revised: The NF density estimator is also identical to the single NF with the following differences: (1) for all datasets, we standardize the data (by subtracting the mean and dividing by the standard deviation) and do not scale the data or apply a logit transform, (2) we set the learning rate to 0 .

Original: 001 , and (3) we perform early stopping on validation loss with a patience of 30 epochs, for a maximum of 100 epochs.
Revised: 001 , and (3) we perform early stopping on validation loss with a patience of 30 epochs, for a maximum of 100 epochs.

Original: D. E XPLOITING THE UNION OF MANIFOLDS HYPOTHESIS : S UPERVISED LEARNING
Revised: D. E XPLOITING THE UNION OF MANIFOLDS HYPOTHESIS : S UPERVISED LEARNING

Original: In order to investigate if there is a correlation between intrinsic dimension and the difficulty of classification for a given class, we train three different classifiers on CIFAR-100 and measure the accuracy for each class.
Revised: In order to investigate if there is a correlation between intrinsic dimension and the difficulty of classification for a given class, we train three different classifiers on CIFAR-100 and measure the accuracy for each class.

Original: We use the same hyperparameters and training strategy for each of the three models we consider (VGG-19 (Simonyan & Zisserman, 2015), ResNet-18, and ResNet- (He et al., 2016)).
Revised: We use the same hyperparameters and training strategy for each of the three models we consider (VGG-19 (Simonyan & Zisserman, 2015), ResNet-18, and ResNet- (He et al., 2016)).

Original: We do not modify the model architecture except for changing the dimension of the last linear layer to project into a 100-dimensional vector to match the number of classes in
Revised: We do not modify the model architecture except for changing the dimension of the last linear layer to project into a 100-dimensional vector to match the number of classes in

Original: CIFAR-100.
Revised: CIFAR-100.

Original: We use the CIFAR-100 dataset test split ( 10 , 000 images) then randomly select 10% of the training dataset as our validation set ( 5 , 000 images) and use the remaining 90% as our train split.
Revised: We use the CIFAR-100 dataset test split ( 10 , 000 images) then randomly select 10% of the training dataset as our validation set ( 5 , 000 images) and use the remaining 90% as our train split.

Original: Note that this validation split is constant across all runs.
Revised: Note that this validation split is constant across all runs.

Original: The classification accuracy is reported as the accuracy on the test set on the same epoch where the best validation accuracy is achieved.
Revised: The classification accuracy is reported as the accuracy on the test set on the same epoch where the best validation accuracy is achieved.

Original: We do not add any data augmentations as doing so could modify intrinsic dimension estimates and confound our results.
Revised: We do not add any data augmentations as doing so could modify intrinsic dimension estimates and confound our results.

Original: Studying the relationship between intrinsic dimension and data augmentations is an interesting direction for future work.
Revised: Studying the relationship between intrinsic dimension and data augmentations is an interesting direction for future work.

Original: The combination of (1) no data augmentations, and (2) the presence of a validation set causes our results to be lower than the state-of-the-art for the models, although the results remain self-consistent here.
Revised: The combination of (1) no data augmentations, and (2) the presence of a validation set causes our results to be lower than the state-of-the-art for the models, although the results remain self-consistent here.

Original: We train each model using the cross entropy loss for a total of 200 epochs starting with a learning rate of 0 . 1 and decrease this to 10 − 2 , 4 × 10 − 3 , and 8 × 10 − 4 after 60 , 120 , and 160 epochs respectively.
Revised: We train each model using the cross entropy loss for a total of 200 epochs starting with a learning rate of 0 . 1 and decrease this to 10 − 2 , 4 × 10 − 3 , and 8 × 10 − 4 after 60 , 120 , and 160 epochs respectively.

Original: We use the standard SGD optimizer with momentum set to 0 .
Revised: We use the standard SGD optimizer with momentum set to 0 .

Original: 9 and train with a weight decay of 5 ×
Revised: 9 and train with a weight decay of 5 ×

Original: 10 − 4 .
Revised: 10 − 4 .

Original: For all datasets, we normalize by the channel-wise mean and standard deviation of all training images.
Revised: For all datasets, we normalize by the channel-wise mean and standard deviation of all training images.

Original: We use a batch size of 128 for training, and a batch size of 100 for both validation and test.
Revised: We use a batch size of 128 for training, and a batch size of 100 for both validation and test.

Original: We now detail the exact loss we used for our experiments with the re-weighted cross entropy loss.
Revised: We now detail the exact loss we used for our experiments with the re-weighted cross entropy loss.

Original: A weighted version of the categorical cross entropy loss is given by:
Revised: A weighted version of the categorical cross entropy loss is given by:

Original: [Equation]
Revised: [Equation]

Original: where y i is a one-hot vector of length L corresponding to the label of x i , f θ ( x i ) is the L -dimensional output of the classifier containing assigned class probabilities, and ω ℓ is the scalar weight given to the ℓ th class.
Revised: where y i is a one-hot vector of length L corresponding to the label of x i , f θ ( x i ) is the L -dimensional output of the classifier containing assigned class probabilities, and ω ℓ is the scalar weight given to the ℓ th class.

Original: The standard categorical cross entropy uses the weights ω ℓ = 1 for ℓ = 1 , . . .
Revised: The standard categorical cross entropy uses the weights ω ℓ = 1 for ℓ = 1 , . . .

Original: , L .
Revised: , L .

Original: Our modified weights are given by:
Revised: Our modified weights are given by:

Original: [Equation]
Revised: [Equation]

Original: Note that when the intrinsic dimension estimates of all classes match, our proposed weights recover the standard ones, but more heavily weight classes of higher intrinsic dimension otherwise.
Revised: Note that when the intrinsic dimension estimates of all classes match, our proposed weights recover the standard ones, but more heavily weight classes of higher intrinsic dimension otherwise.