Original: P ROGRESSIVE LEARNING AND DISENTANGLEMENT OF HIERARCHICAL REPRESENTATIONS
Revised: P ROGRESSIVE LEARNING AND DISENTANGLEMENT OF HIERARCHICAL REPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE).
Revised: Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE).

Original: However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised.
Revised: However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised.

Original: Motivated by the concept of “starting small”, we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions.
Revised: Motivated by the concept of “starting small”, we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions.

Original: The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new representations at different levels of abstraction.
Revised: The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new representations at different levels of abstraction.

Original: We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark data sets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap.
Revised: We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark data sets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap.

Original: We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations.
Revised: We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations.

Original: By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the ﬁrst attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.
Revised: By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the ﬁrst attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Variational auto-encoder (VAE), a popular deep generative model (DGM), has shown great promise in learning interpretable and semantically meaningful representations of data (Higgins et al.
Revised: Variational auto-encoder (VAE), a popular deep generative model (DGM), has shown great promise in learning interpretable and semantically meaningful representations of data (Higgins et al.

Original: Chen et al.
Revised: Chen et al.

Original: ; Kim & Mnih (2018)).
Revised: ; Kim & Mnih (2018)).

Original: However, VAE has not been able to fully utilize the depth of neural networks like its supervised counterparts, for which a fundamental cause lies in the inherent conﬂict between the bottom-up inference and top-down generation process (Zhao et al.
Revised: However, VAE has not been able to fully utilize the depth of neural networks like its supervised counterparts, for which a fundamental cause lies in the inherent conﬂict between the bottom-up inference and top-down generation process (Zhao et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2016)): while the bottom-up abstraction is able to extract high-level representations helpful for discriminative tasks, the goal of generation requires the preservation of all generative factors that are likely at different abstraction levels.
Revised: (2016)): while the bottom-up abstraction is able to extract high-level representations helpful for discriminative tasks, the goal of generation requires the preservation of all generative factors that are likely at different abstraction levels.

Original: This issue was addressed in recent works by allowing VAEs to generate from details added at different depths of the network, using either memory modules between top-down generation layers (Li et al.
Revised: This issue was addressed in recent works by allowing VAEs to generate from details added at different depths of the network, using either memory modules between top-down generation layers (Li et al.

Original: (2016)), or hierarchical latent representations extracted at different depths via a variational ladder autoencoder (VLAE, Zhao et al.
Revised: (2016)), or hierarchical latent representations extracted at different depths via a variational ladder autoencoder (VLAE, Zhao et al.

Original: However, it is difﬁcult to learn to extract and disentangle all generative factors at once, especially at different abstraction levels.
Revised: However, it is difﬁcult to learn to extract and disentangle all generative factors at once, especially at different abstraction levels.

Original: Inspired by human cognition system, Elman (1993) suggested the importance of “starting small” in two aspects of the learning process of neural networks: incremental input in which a network is trained with data and tasks of increasing complexity, and incremental memory in which the network capacity undergoes developmental changes given ﬁxed external data and tasks — both pointing to an incremental learning strategy for simplifying a complex ﬁnal task.
Revised: Inspired by human cognition system, Elman (1993) suggested the importance of “starting small” in two aspects of the learning process of neural networks: incremental input in which a network is trained with data and tasks of increasing complexity, and incremental memory in which the network capacity undergoes developmental changes given ﬁxed external data and tasks — both pointing to an incremental learning strategy for simplifying a complex ﬁnal task.

Original: Indeed, the former concept of incremental input has underpinned the success of curriculum learning (Bengio et al.
Revised: Indeed, the former concept of incremental input has underpinned the success of curriculum learning (Bengio et al.

Original: In the context of DGMs, various stacked versions of generative adversarial networks (GANs) have been proposed to decompose the ﬁnal task of high-resolution image generation into progressive sub-tasks of generating small to large images (Denton et al.
Revised: In the context of DGMs, various stacked versions of generative adversarial networks (GANs) have been proposed to decompose the ﬁnal task of high-resolution image generation into progressive sub-tasks of generating small to large images (Denton et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: The latter aspect of “starting small” with incremental growth of network capacity is less explored, although recent works have demonstrated the advantage of progressively growing the depth of GANs for generating high-resolution images (Karras et al.
Revised: The latter aspect of “starting small” with incremental growth of network capacity is less explored, although recent works have demonstrated the advantage of progressively growing the depth of GANs for generating high-resolution images (Karras et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: These works, so far, have focused on progressive learning as a strategy to improve image generation.
Revised: These works, so far, have focused on progressive learning as a strategy to improve image generation.

Original: We are motivated to investigate the possibility to use progressive learning strategies to improve learning and disentangling of hierarchical representations.
Revised: We are motivated to investigate the possibility to use progressive learning strategies to improve learning and disentangling of hierarchical representations.

Original: At a high level, the idea of progressively or sequentially learning latent representations has been previously considered in VAE.
Revised: At a high level, the idea of progressively or sequentially learning latent representations has been previously considered in VAE.

Original: In Gregor et al.
Revised: In Gregor et al.

Original: (2015), the network learned to sequentially reﬁne generated images through recurrent networks.
Revised: (2015), the network learned to sequentially reﬁne generated images through recurrent networks.

Original: In
Revised: In

Original: Lezama (2019), a teacher-student training strategy was used to progressively increase the number of latent dimensions in VAE to improve the generation of images while preserving the disentangling ability of the teacher model.
Revised: Lezama (2019), a teacher-student training strategy was used to progressively increase the number of latent dimensions in VAE to improve the generation of images while preserving the disentangling ability of the teacher model.

Original: However, these works primarily focus on progressively growing the capacity of VAE to generate, rather than to extract and disentangle hierarchical representations.
Revised: However, these works primarily focus on progressively growing the capacity of VAE to generate, rather than to extract and disentangle hierarchical representations.

Original: In comparison, in this work, we focus on 1) progressively growing the capacity of the network to extract hierarchical representations, and 2) these hierarchical representations are extracted and used in generation from different abstraction levels.
Revised: In comparison, in this work, we focus on 1) progressively growing the capacity of the network to extract hierarchical representations, and 2) these hierarchical representations are extracted and used in generation from different abstraction levels.

Original: We present a simple progressive training strategy that grows the hierarchical latent representations from different depths of the inference and generation model, learning from high- to low-levels of abstractions as the capacity of the model architecture grows.
Revised: We present a simple progressive training strategy that grows the hierarchical latent representations from different depths of the inference and generation model, learning from high- to low-levels of abstractions as the capacity of the model architecture grows.

Original: Because it can be viewed as a progressive strategy to train the VLAE presented in Zhao et al.
Revised: Because it can be viewed as a progressive strategy to train the VLAE presented in Zhao et al.

Original: (2017), we term the presented model pro -VLAE.
Revised: (2017), we term the presented model pro -VLAE.

Original: We quantitatively demonstrate the ability of pro -VLAE to improve disentanglement on two benchmark data sets using three disentanglement metrics, including a new metric we proposed to complement the metric of mutual information gap (MIG) previously presented in Chen et al.
Revised: We quantitatively demonstrate the ability of pro -VLAE to improve disentanglement on two benchmark data sets using three disentanglement metrics, including a new metric we proposed to complement the metric of mutual information gap (MIG) previously presented in Chen et al.

Original: These quantitative studies include comprehensive comparisons to β -VAE (Higgins et al.
Revised: These quantitative studies include comprehensive comparisons to β -VAE (Higgins et al.

Original: , VLAE (Zhao et al.
Revised: , VLAE (Zhao et al.

Original: and the teacher-student strategy as presented in (Lezama (2019)) at different values of the hyperparameter β .
Revised: and the teacher-student strategy as presented in (Lezama (2019)) at different values of the hyperparameter β .

Original: We further present both qualitative and quantitative evidence that pro -VLAE is able to ﬁrst learn the most abstract representations and then progressively disentangle existing factors or learn new factors at lower levels of abstraction, improving disentangling of hierarhical representations in the process.
Revised: We further present both qualitative and quantitative evidence that pro -VLAE is able to ﬁrst learn the most abstract representations and then progressively disentangle existing factors or learn new factors at lower levels of abstraction, improving disentangling of hierarhical representations in the process.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: A hierarchy of feature maps can be naturally formed in stacked discriminative models (Zeiler &
Revised: A hierarchy of feature maps can be naturally formed in stacked discriminative models (Zeiler &

Original: Fergus (2014)).
Revised: Fergus (2014)).

Original: Similarly, in DGM, many works have proposed stacked-VAEs as a common way to learn a hierarchy of latent variables and thereby improve image generation (Sønderby et al.
Revised: Similarly, in DGM, many works have proposed stacked-VAEs as a common way to learn a hierarchy of latent variables and thereby improve image generation (Sønderby et al.

Original: Bachman (2016); Kingma et al. (2016)).
Revised: Bachman (2016); Kingma et al. (2016)).

Original: However, this stacked hierarchy is not only difﬁcult to train as the depths increases (Sønderby et al.
Revised: However, this stacked hierarchy is not only difﬁcult to train as the depths increases (Sønderby et al.

Original: ; Bachman (2016)), but also has an unclear beneﬁt for learning either hierarchical or disentangled representations: as shown in Zhao et al.
Revised: ; Bachman (2016)), but also has an unclear beneﬁt for learning either hierarchical or disentangled representations: as shown in Zhao et al.

Original: (2017), when fully optimized, it is equivalent to a model with a single layer of latent variables.
Revised: (2017), when fully optimized, it is equivalent to a model with a single layer of latent variables.

Original: Alternatively, instead of a hierarchy of latent variables, independent hierarchical representations at different abstraction levels can be extracted and used in generation from different depths of the network (Rezende et al.
Revised: Alternatively, instead of a hierarchy of latent variables, independent hierarchical representations at different abstraction levels can be extracted and used in generation from different depths of the network (Rezende et al.

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: A similar idea was presented in Li et al.
Revised: A similar idea was presented in Li et al.

Original: (2016) to generate lost details from memory and attention modules at different depths of the top-down generation process.
Revised: (2016) to generate lost details from memory and attention modules at different depths of the top-down generation process.

Original: The presented work aligns with existing works (Rezende et al.
Revised: The presented work aligns with existing works (Rezende et al.

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: in learning independent hierarchical representation from different levels of abstraction, and we look to facilitate this learning by progressively learning the representations from high- to low-levels.
Revised: in learning independent hierarchical representation from different levels of abstraction, and we look to facilitate this learning by progressively learning the representations from high- to low-levels.

Original: Progressive learning has been successful for high-quality image generation, mostly in the setting of
Revised: Progressive learning has been successful for high-quality image generation, mostly in the setting of

Original: GANs.
Revised: GANs.

Original: Following the seminar work of Elman (1993), these progressive strategies can be loosely grouped into two categories.
Revised: Following the seminar work of Elman (1993), these progressive strategies can be loosely grouped into two categories.

Original: Mostly, in line with incremental input, several works have proposed to divide the ﬁnal task of image generation into progressive tasks of generating low-resolution to high-resolution images with multi-scale supervision (Denton et al.
Revised: Mostly, in line with incremental input, several works have proposed to divide the ﬁnal task of image generation into progressive tasks of generating low-resolution to high-resolution images with multi-scale supervision (Denton et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: Alternatively, in line with incremental memory, a small number of works have demonstrated the ability to simply grow the architecture of GANs from a shallow network with limited capacity for generating low-resolution images, to a deep network capable of generating super-resolution images (Karras et al.
Revised: Alternatively, in line with incremental memory, a small number of works have demonstrated the ability to simply grow the architecture of GANs from a shallow network with limited capacity for generating low-resolution images, to a deep network capable of generating super-resolution images (Karras et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: This approach was also shown to be time-efﬁcient since the early-stage small networks require less time to converge comparing to training a full network from the beginning.
Revised: This approach was also shown to be time-efﬁcient since the early-stage small networks require less time to converge comparing to training a full network from the beginning.

Original: This latter group of works provided compelling evidence for the beneﬁt of progressively growing the capacity of a network to generate images, although its extension for growing the capacity of a network to learn hierarchical representations has not been explored.
Revised: This latter group of works provided compelling evidence for the beneﬁt of progressively growing the capacity of a network to generate images, although its extension for growing the capacity of a network to learn hierarchical representations has not been explored.

Original: Limited work has considered incremental learning of representations in VAE.
Revised: Limited work has considered incremental learning of representations in VAE.

Original: In Gregor et al.
Revised: In Gregor et al.

Original: , recurrent networks with attention mechanisms were used to sequentially reﬁnes the details in generated images.
Revised: , recurrent networks with attention mechanisms were used to sequentially reﬁnes the details in generated images.

Original: It however focused on the generation performance of VAE without considering the learned representations.
Revised: It however focused on the generation performance of VAE without considering the learned representations.

Original: In Lezama (2019), a teacher-student strategy was used to progressively grow the dimension of the latent representations in VAE.
Revised: In Lezama (2019), a teacher-student strategy was used to progressively grow the dimension of the latent representations in VAE.

Original: Its fundamental motivation was that, given a teacher model that has learned to effectively disentangle major factors of variations, progressively learning additional nuisance variables will improve generation without compromising the disentangling ability of the teacher – the latter accomplished via a newly-proposed Jacobian supervision.
Revised: Its fundamental motivation was that, given a teacher model that has learned to effectively disentangle major factors of variations, progressively learning additional nuisance variables will improve generation without compromising the disentangling ability of the teacher – the latter accomplished via a newly-proposed Jacobian supervision.

Original: The capacity of this model to grow, thus, is by design limited to the extraction of nuisance variables.
Revised: The capacity of this model to grow, thus, is by design limited to the extraction of nuisance variables.

Original: In comparison, we are interested in a more signiﬁcant growth of the VAE capacity to progressively learn and improve disentangling of important factors of variations which, as we will later demonstrate, is not what the model in Lezama (2019) is intended for.
Revised: In comparison, we are interested in a more signiﬁcant growth of the VAE capacity to progressively learn and improve disentangling of important factors of variations which, as we will later demonstrate, is not what the model in Lezama (2019) is intended for.

Original: In addition, neither of these works considered learning different levels of abstractions at different depths of the network, and the presented pro- VLAE provides a simpler training strategy to achieve progressive representation learning.
Revised: In addition, neither of these works considered learning different levels of abstractions at different depths of the network, and the presented pro- VLAE provides a simpler training strategy to achieve progressive representation learning.

Original: Learning disentangled representation is a primary motivation of our work, and an important topic in VAE.
Revised: Learning disentangled representation is a primary motivation of our work, and an important topic in VAE.

Original: Existing works mainly tackle this by promoting the independence among the learned latent factors in VAE (Higgins et al.
Revised: Existing works mainly tackle this by promoting the independence among the learned latent factors in VAE (Higgins et al.

Original: Kim & Mnih (2018); Chen et al.
Revised: Kim & Mnih (2018); Chen et al.

Original: The presented progressive learning strategy provides an entirely different approach to improve disentangling that is orthogonal to these existing methods and a possibility to augment them in the future.
Revised: The presented progressive learning strategy provides a novel approach to improve disentangling that is different to these existing methods and a possibility to augment them in the future.

Original:  M ETHODS 3. MODEL : VAE WITH HIERARCHICAL REPRESENTATIONS
Revised:  M ETHODS 3. MODEL : VAE WITH HIERARCHICAL REPRESENTATIONS

Original: We assume a generative model p ( x , z )
Revised: We assume a generative model p ( x , z )

Original: =
Revised: =

Original: p ( x | z ) p ( z ) for observed x and its latent variable z .
Revised: p ( x | z ) p ( z ) for observed x and its latent variable z .

Original: To learn hierarchical representations of x , we decompose z into { z 1 , z 2 , ..., z L } with z l ( l = 1 , 2 , 3 , ..., L ) from different abstraction levels.
Revised: To learn hierarchical representations of x , we decompose z into { z 1 , z 2 , ..., z L } with z l ( l = 1 , 2 , 3 , ..., L ) from different abstraction levels that are loosely guided by the depth of neural network as in (Zhao et al.

Original: We deﬁne the hierarchical generative model p θ as:
Revised: We deﬁne the hierarchical generative model p θ as:

Original: [Equation]
Revised: [Equation]

Original: Note that there is no hierarchical dependence among the latent variables as in common hierarchical latent variable models.
Revised: Note that there is no hierarchical dependence among the latent variables as in common hierarchical latent variable models.

Original: Rather, similar to that in Rezende et al.
Revised: Rather, similar to that in Rezende et al.

Original: (2014) and Zhao et al.
Revised: (2014) and Zhao et al.

Original: (2017), z l ’s are independent and each represents generative factors at an abstraction level not captured in other levels.
Revised: (2017), z l ’s are independent and each represents generative factors at an abstraction level not captured in other levels.

Original: We then deﬁne an inference model q φ to approximate the posterior as:
Revised: We then deﬁne an inference model q φ to approximate the posterior as:

Original: [Equation]
Revised: [Equation]

Original: where h l ( x ) represents a particular level of bottom-up abstraction of x .
Revised: where h l ( x ) represents a particular level of bottom-up abstraction of x .

Original: We parameterize p θ and q φ with an encoding-decoding structure and, as in Zhao et al.
Revised: We parameterize p θ and q φ with an encoding-decoding structure and, as in Zhao et al.

Original: (2017), we approximate the abstraction level with the network depth.
Revised: (2017), we approximate the abstraction level with the network depth.

Original: The full model is illustrated in Fig.
Revised: The full model is illustrated in Fig.

Original: 1(c), with a ﬁnal goal to maximize a modiﬁed evidence lower bound (ELBO) of the marginal likelihood of data x :
Revised: 1(c), with a ﬁnal goal to maximize a modiﬁed evidence lower bound (ELBO) of the marginal likelihood of data x :

Original: [Equation]
Revised: [Equation]

Original: where KL denotes the Kullback-Leibler divergence, prior p ( z ) is set to isotropic Gaussian N (0 , I ) according to standard practice, and β is a hyperparameter introduced in Higgins et al.
Revised: where KL denotes the Kullback-Leibler divergence, prior p ( z ) is set to isotropic Gaussian N (0 , I ) according to standard practice, and β is a hyperparameter introduced in Higgins et al.

Original: (2017) to promote disentangling, defaulting to the standrd ELBO objective when β
Revised: (2017) to promote disentangling, defaulting to the standrd ELBO objective when β

Original: = 1 .
Revised: = 1 .

Original: 3. PROGRESSIVE LEARNING OF HIERARCHICAL REPRESENTATION
Revised: 3. PROGRESSIVE LEARNING OF HIERARCHICAL REPRESENTATION

Original: We present a progressive learning strategy, as illustrated in Fig.
Revised: We present a progressive learning strategy, as illustrated in Fig.

Original: 1, to achieve the ﬁnal goal in equation (3) by learning the latent variables z l progressively from the highest ( l = L ) to the lowest l = 1) level of abstractions.
Revised: 1, to achieve the ﬁnal goal in equation (3) by learning the latent variables z l progressively from the highest ( l = L ) to the lowest l = 1) level of abstractions.

Original: We start by learning the most abstraction representations at layer L as show in Fig. 1(a).
Revised: We start by learning the most abstraction representations at layer L as show in Fig. 1(a).

Original: In this case, our model degenerates to a vanilla VAE with latent variables z L at the deepest layer.
Revised: In this case, our model degenerates to a vanilla VAE with latent variables z L at the deepest layer.

Original: We keep the dimension of z L small to start small in terms of the capacity to learn latent representations, where we deﬁne the inference model at progressive step s = 0 as:
Revised: We keep the dimension of z L small to start small in terms of the capacity to learn latent representations, where we deﬁne the inference model at progressive step s = 0 as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] ( h L ) , σ L ( h L )) , h l = f el ( h l − 1 ) , for l = 1 , 2 , ..., L, and h 0 ≡ x ,
Revised: 1 ) , for l = 1 , 2 , ..., L, and h 0 ≡ x ,

Original: [Equation]
Revised: [Equation]

Original: 4) and the generative model as:
Revised: and the generative model as:

Original: [Equation]
Revised: [Equation]

Original: where f el , µ L , and σ L are parts of the encoder architecture, f dl are parts of the decoder architecture, and D is the distribution of x parametrized by f d 0 ( g 0 ) , which can be either Bernoulli or Gaussian depending on the data.
Revised: where f el , µ L , and σ L are parts of the encoder architecture, f dl are parts of the decoder architecture, and D is the distribution of x parametrized by f d 0 ( g 0 ) , which can be either Bernoulli or Gaussian depending on the data.

Original: Next, as shown in Fig.
Revised: Next, as shown in Fig.

Original: 1, we progressively grow the model to learn z L − 1 , ..., z 2 , z 1 from high to low abstraction levels.
Revised: 1, we progressively grow the model to learn z L − 1 , ..., z 2 , z 1 from high to low abstraction levels.

Original: At each progressive step s = 1 , 2 , ..., L − 1 , we move down one abstraction level, and grow the inference model by introducing new latent code:
Revised: At each progressive step s = 1 , 2 , ..., L − 1 , we move down one abstraction level, and grow the inference model by introducing new latent code:

Original: [Equation]
Revised: [Equation]

Original: Simultaneously, we grow the decoder such that it can generate with the new latent code as:
Revised: Simultaneously, we grow the decoder such that it can generate with the new latent code as:

Original: [Equation]
Revised: [Equation]

Original: where m l includes transposed convolution layers outputting a feature map in the same shape as g l +1 , and [ · ; · ] denotes a concatenation operation.
Revised: where m l includes transposed convolution layers outputting a feature map in the same shape as g l +1 , and [ · ; · ] denotes a concatenation operation.

Original: The training objective at progressive step s is then:
Revised: The training objective at progressive step s is then:

Original: [Equation]
Revised: [Equation]

Original: By replacing the full objective in equation (3) with a sequence of the objectives in equation (8) as the training progresses, we incrementally learn to extract and generate with hierarchical latent representations z l ’s from high to low levels of abstractions.
Revised: By replacing the full objective in equation (3) with a sequence of the objectives in equation (8) as the training progresses, we incrementally learn to extract and generate with hierarchical latent representations z l ’s from high to low levels of abstractions.

Original: Once trained, the full model as shown in Fig.
Revised: Once trained, the full model as shown in Fig.

Original: 1(c) will be used for inference and generation, and progressive processes are no loner needed.
Revised: 1(c) will be used for inference and generation, and progressive processes are no loner needed.

Original: 3. IMPLEMENTATION STRATEGIES
Revised: 3. IMPLEMENTATION STRATEGIES

Original: Two important strategies are utilized to implement the proposed progressive representation learning.
Revised: Two important strategies are utilized to implement the proposed progressive representation learning.

Original: First, directly adding new components to a trained network often introduce a sudden shock to the gradient: in VAEs, this often leads to the explosion of the variance in the latent distributions.
Revised: First, directly adding new components to a trained network often introduce a sudden shock to the gradient: in VAEs, this often leads to the explosion of the variance in the latent distributions.

Original: To avoid this shock, we adopt the popular method of “fade-in” (Karras et al.
Revised: To avoid this shock, we adopt the popular method of “fade-in” (Karras et al.

Original: (2018)) to smoothly blend the new and existing network components.
Revised: (2018)) to smoothly blend the new and existing network components.

Original: In speciﬁc, we introduce a “fade-in” coefﬁcient α to equations (6) and (7) when growing new components in the encoder and the decoder:
Revised: In speciﬁc, we introduce a “fade-in” coefﬁcient α to equations (6) and (7) when growing new components in the encoder and the decoder:

Original: [Equation]
Revised: [Equation]

Original: where α increases from 0 to 1 within a certain number of iterations (5000 in our experiments) since the addition of the new network components µ l , σ l , and m l .
Revised: where α increases from 0 to 1 within a certain number of iterations (5000 in our experiments) since the addition of the new network components µ l , σ l , and m l .

Original: Second, we further stabilize the training by weakly constraining the distribution of z l ’s before they are added to the network.
Revised: Second, we further stabilize the training by weakly constraining the distribution of z l ’s before they are added to the network.

Original: This can be achieved by a applying a KL penalty, modulated by a small coefﬁcient γ , to all latent variables that have not been used in the generation at progressive step s :
Revised: This can be achieved by a applying a KL penalty, modulated by a small coefﬁcient γ , to all latent variables that have not been used in the generation at progressive step s :

Original: [Equation]
Revised: [Equation]

Original: where γ is set to 0.5 in our experiments.
Revised: where γ is set to 0.5 in our experiments.

Original: The ﬁnal training objective at step s then becomes:
Revised: The ﬁnal training objective at step s then becomes:

Original: [Equation]
Revised: [Equation]

Original: Note that the latent variables at the hierarchy lower than L − s are neither meaningfully inferred nor used in generation at progressive step s , and L pre − trained merely intends to regularize the distribution of these latent variables before they are added to the network.
Revised: Note that the latent variables at the hierarchy lower than L − s are neither meaningfully inferred nor used in generation at progressive step s , and L pre − trained merely intends to regularize the distribution of these latent variables before they are added to the network.

Original: In the experiments below, we use both “fade-in” and L pre − trained when implementing the progressive training strategy.
Revised: In the experiments below, we use both “fade-in” and L pre − trained when implementing the progressive training strategy.

Original: 3. DISENTANGLEMENT METRIC
Revised: 3. DISENTANGLEMENT METRIC

Original: Various quantitative metrics for measuring disentanglement have been proposed (Higgins et al.
Revised: Various quantitative metrics for measuring disentanglement have been proposed (Higgins et al.

Original: (2017); Kim & Mnih (2018); Chen et al.
Revised: (2017); Kim & Mnih (2018); Chen et al.

Original: For instance, the recently proposed MIG metrics (Chen et al.
Revised: For instance, the recently proposed MIG metrics (Chen et al.

Original: (2018)) measures the gap of mutual information between the top two latent dimensions that have the highest mutual information with a given generative factor.
Revised: (2018)) measures the gap of mutual information between the top two latent dimensions that have the highest mutual information with a given generative factor.

Original: A low MIG score, therefore, suggests an undesired outcome that the same factor is split into multiple dimensions.
Revised: A low MIG score, therefore, suggests an undesired outcome that the same factor is split into multiple dimensions.

Original: However, if different generative factors are entangled into the same latent dimension, the MIG score will not be affected.
Revised: However, if different generative factors are entangled into the same latent dimension, the MIG score will not be affected.

Original: Therefore, we propose a new disentanglement metric to supplement MIG by recognizing the entanglement of multiple generative factors into the same latent dimension.
Revised: Therefore, we propose a new disentanglement metric to supplement MIG by recognizing the entanglement of multiple generative factors into the same latent dimension.

Original: We deﬁne MIG-sup as:
Revised: We deﬁne MIG-sup as:

Original: [Equation]
Revised: [Equation]

Original:  where z is the latent variables and v is the ground truth factors, k ( j ) =
Revised: (12) where z is the latent variables and v is the ground truth factors, k ( j ) =

Original: argmax k
Revised: argmax k

Original: I norm ( z j ; v k ) , J is the number of meaningful latent dimensions, and I norm ( z j ; v k ) is normalized mutual information
Revised: I norm ( z j ; v k ) , J is the number of meaningful latent dimensions, and I norm ( z j ; v k ) is normalized mutual information

Original: I ( z j ; v k ) /H
Revised: I ( z j ; v k ) /H

Original: ( v k ) .
Revised: ( v k ) .

Original: Considering MIG and MIG-sup together will provide a more complete measure of disentanglement, accounting for both the splitting of one factor into multiple dimensions and the encoding of multiple factors into the same dimension.
Revised: Considering MIG and MIG-sup together will provide a more complete measure of disentanglement, accounting for both the splitting of one factor into multiple dimensions and the encoding of multiple factors into the same dimension.

Original: In an ideal disentanglement, both MIG and MIG-sup should be 1, recognizing a one-to-one relationship between a generative factor and a latent dimension.
Revised: In an ideal disentanglement, both MIG and MIG-sup should be 1, recognizing a one-to-one relationship between a generative factor and a latent dimension.

Original: This would have a similar effect to the metric that was proposed in (Eastwood &
Revised: This would have a similar effect to the metric that was proposed in Eastwood &

Original: Williams (2018)), although MIG-based metrics do not rely on training extra classiﬁers or regressors and are unbiased for hyperparameter settings.
Revised: Williams (2018), although MIG-based metrics do not rely on training extra classiﬁers or regressors and are unbiased for hyperparameter settings.

Original: The factor metric (Kim & Mnih (2018)) also has similar properties with MIG-sup, although MIG-sup is stricter on penalizing any amount of other minor factors in the same dimension.
Revised: The factor metric Kim & Mnih (2018) also has similar properties with MIG-sup, although MIG-sup is stricter on penalizing any amount of other minor factors in the same dimension.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We tested the presented pro -VLAE on four benchmark data sets: dSprites (Matthey et al.
Revised: We tested the presented pro -VLAE on four benchmark data sets: dSprites (Matthey et al.

Original: , 3DShapes (Kim & Mnih (2018)), MNIST (LeCun et al.
Revised: , 3DShapes (Kim & Mnih (2018)), MNIST (LeCun et al.

Original: (1998)), and CelebA (Liu et al.
Revised: (1998)), and CelebA (Liu et al.

Original: (2015)), where the ﬁrst two include ground-truth generative factors that allow us to carry out comprehensive quantitative comparisons of disentangling metrics with existing models.
Revised: (2015)), where the ﬁrst two include ground-truth generative factors that allow us to carry out comprehensive quantitative comparisons of disentangling metrics with existing models.

Original: In the following, we ﬁrst quantitatively compare the disentangling ability of pro -VLAE in comparison to three existing models using three disentanglement metrics.
Revised: In the following, we ﬁrst quantitatively compare the disentangling ability of pro -VLAE in comparison to three existing models using three disentanglement metrics.

Original: We then analyze pro -VLAE from the aspects of how it learns progressively, its ability to disentangle, and its ability to learn abstractions at different levels.
Revised: We then analyze pro -VLAE from the aspects of how it learns progressively, its ability to disentangle, and its ability to learn abstractions at different levels.

Original: Comparisons in quantitative disentanglement metrics: For quantitative comparisons, we considered the factor metric in Kim & Mnih (2018), the MIG in Chen et al.
Revised: Comparisons in quantitative disentanglement metrics: For quantitative comparisons, we considered the factor metric in Kim & Mnih (2018), the MIG in Chen et al.

Original: (2018), and the MIG-sup presented in this work.
Revised: (2018), and the MIG-sup presented in this work.

Original: We compared pro -VLAE (changing β ) with beta-VAE (Higgins et al.
Revised: We compared pro -VLAE (changing β ) with beta-VAE (Higgins et al.

Original: VLAE (Zhao et al.
Revised: VLAE (Zhao et al.

Original: as a hierarchical baseline without progressive training, and the teacherstudent model (Lezama (2019)) as the most related progressive VAE without hierarchical representations.
Revised: as a hierarchical baseline without progressive training, and the teacherstudent model (Lezama (2019)) as the most related progressive VAE without hierarchical representations.

Original: All models were considered at different values of β except the teacher-student model: the comparison of β -VAE, VLAE, and the presented pro -VLAE thus also provides an ablation study on the effect of learning hierarchical representations and doing so in a progressive manner.
Revised: All models were considered at different values of β except the teacher-student model: the comparison of β -VAE, VLAE, and the presented pro -VLAE thus also provides an ablation study on the effect of learning hierarchical representations and doing so in a progressive manner.

Original: For fair comparisons, we strictly required all models to have the same number of latent variables and the same number of training iterations.
Revised: For fair comparisons, we strictly required all models to have the same number of latent variables and the same number of training iterations.

Original: For instance, if a hierarchical model has three layers that each has three latent dimensions, a non-hierarchical model will have nine latent dimensions; if a progressive method has three progressive steps with 15 epochs of training each, a non-progressive method will be trained for 45 epochs.
Revised: For instance, if a hierarchical model has three layers that each has three latent dimensions, a non-hierarchical model will have nine latent dimensions; if a progressive method has three progressive steps with 15 epochs of training each, a non-progressive method will be trained for 45 epochs.

Original: Three to ﬁve experiments were conducted for each model at each β value, and the average of the top three is used for reporting the quantitative results in Fig.
Revised: Three to ﬁve experiments were conducted for each model at each β value, and the average of the top three is used for reporting the quantitative results in Fig.

Original: As shown, for MIG and MIG-sup, VLAE generally outperformed β -VAE at most β values, while pro -VLAE showed a clear margin of improvement over both methods.
Revised: As shown, for MIG and MIG-sup, VLAE generally outperformed β -VAE at most β values, while pro -VLAE showed a clear margin of improvement over both methods.

Original: With the factor metric, proVLAE was still among the top performers, although with a smaller margin and a larger overlap with
Revised: With the factor metric, proVLAE was still among the top performers, although with a smaller margin and a larger overlap with

Original: VLAE on 3DShapes, and with β -VAE ( β = 10 ) on dSprites.
Revised: VLAE on 3DShapes, and with β -VAE ( β = 10 ) on dSprites.

Original: The teacher-student strategy with
Revised: The teacher-student strategy with

Original: Jacobian supervision in general had a low to moderate disentangling score, especially on 3DShapes.
Revised: Jacobian supervision in general had a low to moderate disentangling score, especially on 3DShapes.

Original: This is consistent with the original motivation of the method for progressively learning nuisance variables after the teacher learns to disentangle effectively, rather than progressively disentangling hierarchical factors of variations as intended by pro -VLAE.
Revised: This is consistent with the original motivation of the method for progressively learning nuisance variables after the teacher learns to disentangle effectively, rather than progressively disentangling hierarchical factors of variations as intended by pro -VLAE.

Original: Note that pro -VLAE in general performed better with a smaller value of β ( β < 20 ), suggesting that progressive learning already had an effect of promoting disentangling and a high value of β may over-promote disentangling at the expense of reconstruction quality.
Revised: Note that pro -VLAE in general performed better with a smaller value of β ( β < 20 ), suggesting that progressive learning already had an effect of promoting disentangling and a high value of β may over-promote disentangling at the expense of reconstruction quality.

Original: Fig.
Revised: Fig.

Original: 3 shows MIG vs .
Revised: 3 shows MIG vs .

Original: MIG-sup scores among the tested models.
Revised: MIG-sup scores among the tested models.

Original: As shown, results from pro -VLAE were well separated from the other three models at the right top quadrant of the plots, obtaining simultaneously high MIG and MIG-sup scores as a clear evidence for improved disentangling ability.
Revised: As shown, results from pro -VLAE were well separated from the other three models at the right top quadrant of the plots, obtaining simultaneously high MIG and MIG-sup scores as a clear evidence for improved disentangling ability.

Original: Fig.
Revised: Fig.

Original: 4 provides images generated by traversing each latent dimension using the best pro -VLAE ( β = 8 ), the best VLAE ( β = 10 ), and the teacher-student model on 3DShapes data.
Revised: 4 provides images generated by traversing each latent dimension using the best pro -VLAE ( β = 8 ), the best VLAE ( β = 10 ), and the teacher-student model on 3DShapes data.

Original: As shown, pro -VLAE learned to disentangle the object, wall, and ﬂoor color in the deepest layer; the following hierarchy of representations then disentangled objective scale, orientation, and shape, while the lowest-level of abstractions ran out of meaningful generative factors to learn.
Revised: As shown, pro -VLAE learned to disentangle the object, wall, and ﬂoor color in the deepest layer; the following hierarchy of representations then disentangled objective scale, orientation, and shape, while the lowest-level of abstractions ran out of meaningful generative factors to learn.

Original: In comparison, the
Revised: In comparison, the

Original: VLAE distributed six generative factors over the nine latent dimensions, where color was split across the hierarchy and sometimes entangled with the object scale (in z 2 ).
Revised: VLAE distributed six generative factors over the nine latent dimensions, where color was split across the hierarchy and sometimes entangled with the object scale (in z 2 ).

Original: The teacher-student model was much less disentangled, which we will delve into further in the following section.
Revised: The teacher-student model was much less disentangled, which we will delve into further in the following section.

Original: Information ﬂow during progressive learning: To further understand what happened during progressive learning, we use mutual information I ( x , z l ) as a surrogate to track the amount of informa- tion learned in each hierarchy of latent variables z l during the progressive learning.
Revised: Information ﬂow during progressive learning: To further understand what happened during progressive learning, we use mutual information I ( x , z l ) as a surrogate to track the amount of information learned in each hierarchy of latent variables z l during the progressive learning.

Original: We adopted the approach in Chen et al.
Revised: We adopted the approach in Chen et al.

Original: (2018) to empirically estimate the mutual information by stratiﬁed sampling.
Revised: (2018) to empirically estimate the mutual information by stratiﬁed sampling.

Original: Fig.
Revised: Fig.

Original: 5 shows an example from 3DShapes.
Revised: 5 shows an example from 3DShapes.

Original: At progressive step 0, pro -VAE was only learning the deepest latent variables in z 3 , discovering most of the generative factors including color, objective shape, and orientation entangled within z 3 .
Revised: At progressive step 0, pro -VAE was only learning the deepest latent variables in z 3 , discovering most of the generative factors including color, objective shape, and orientation entangled within z 3 .

Original: At progressive step 1, interestingly, the model was able to “drag” out shape and rotation factors from z 3 and disentangle them into z 2 along with a new scale factor.
Revised: At progressive step 1, interestingly, the model was able to “drag” out shape and rotation factors from z 3 and disentangle them into z 2 along with a new scale factor.

Original: Thus I ( x ; z 3) decreased from 10.59 to 6.94 while I ( x ; z 2) increased from 0.02 to 5.98 in this progression, while the total mutual information I ( x ; z ) increased from 10.61 to 12.84, suggesting the overall learning of more detailed information.
Revised: Thus I ( x ; z 3) decreased from 10.59 to 6.94 while I ( x ; z 2) increased from 0.02 to 5.98 in this progression, while the total mutual information I ( x ; z ) increased from 10.61 to 12.84, suggesting the overall learning of more detailed information.

Original: Since 3DShapes only has 6 factors, the lowest-level representation z 1 had nothing to learn in progressive step 2, and the allocation of mutual information remained nearly unchanged.
Revised: Since 3DShapes only has 6 factors, the lowest-level representation z 1 had nothing to learn in progressive step 2, and the allocation of mutual information remained nearly unchanged.

Original: Note that the sum of I ( x , z l ) ’s does not equal to
Revised: Note that the sum of I ( x , z l ) ’s does not equal to

Original: I ( x , z ) and I over = (cid:80)
Revised: I ( x , z ) and I over = (cid:80)

Original: L 1 I ( x , z l ) − I ( x , z ) suggests the amount of information that is entangled.
Revised: L 1 I ( x , z l ) − I ( x , z ) suggests the amount of information that is entangled.

Original: In comparison, the teacher-student model was less effective in progressively dragging entangled representations to newly added latent dimensions, as suggested by the slowing changing of I ( x , z l ) ’s during progression and the larger value of I over .
Revised: In comparison, the teacher-student model was less effective in progressively dragging entangled representations to newly added latent dimensions, as suggested by the slowing changing of I ( x , z l ) ’s during progression and the larger value of I over .

Original: This suggests that, since the teacher-student model was motivated for progressively learning nuisance variables, the extent to which its capacity can grow for learning new representations is limited by two fundamental causes: 1) because it increases the dimension of the same latent vectors at the same depth, the growth of the network capacity is limited in comparison to pro -VLAE, and 2) the Jacobian supervision further restricts the student model to maintain the same disentangling ability of the teacher model.
Revised: This suggests that, since the teacher-student model was motivated for progressively learning nuisance variables, the extent to which its capacity can grow for learning new representations is limited by two fundamental causes: 1) because it increases the dimension of the same latent vectors at the same depth, the growth of the network capacity is limited in comparison to pro -VLAE and 2) the Jacobian supervision further restricts the student model to maintain the same disentangling ability of the teacher model.

Original: Disentangling hierarchical representations: We also qualitatively examined pro -VLAE on data with both relatively simple (MNIST) and complex (CelebA) factors of variations, all done in unsupervised training.
Revised: Disentangling hierarchical representations: We also qualitatively examined pro -VLAE on data with both relatively simple (MNIST) and complex (CelebA) factors of variations, all done in unsupervised training.

Original: On MNIST (Figure 6), while the deepest latent representations encoded the highest-level features in terms of digit identity, the representations learned at shallower levels encoded changes in writing styles.
Revised: On MNIST (Figure 6), while the deepest latent representations encoded the highest-level features in terms of digit identity, the representations learned at shallower levels encoded changes in writing styles.

Original: In Figure 7, we show the latent representation progressively learned in CelebA from the highest to lowest levels of abstractions, along with disentangling within each level demonstrated by traversing one selected dimension at a time.
Revised: In Figure 7, we show the latent representation progressively learned in CelebA from the highest to lowest levels of abstractions, along with disentangling within each level demonstrated by traversing one selected dimension at a time.

Original: These dimensions are selected as examples associated with clear semantic meanings.
Revised: These dimensions are selected as examples associated with clear semantic meanings.

Original: As shown, while the deepest latent representation z 4 learned to disentangle high-level features such as gender and race, the shallowest representation z 1 learned to disentangle low-level features such as eye-shadow.
Revised: As shown, while the deepest latent representation z 4 learned to disentangle high-level features such as gender and race, the shallowest representation z 1 learned to disentangle low-level features such as eye-shadow.

Original: Moreover, the number of distinct representations learned decreased from deep to shallow layers.
Revised: Moreover, the number of distinct representations learned decreased from deep to shallow layers.

Original: While demonstrating disentangling by traversing each individual latent dimension or by hierarchically-learned representations has been separately reported in previous works (Higgins et al.
Revised: While demonstrating disentangling by traversing each individual latent dimension or by hierarchically-learned representations has been separately reported in previous works (Higgins et al.

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: (2017)), to our knowledge this is the ﬁrst time the ability of a model to disentangle individual latent factors in a hierarchical manner has been demonstrated.
Revised: (2017)), to our knowledge this is the ﬁrst time the ability of a model to disentangle individual latent factors in a hierarchical manner has been demonstrated.

Original: This provides evidence that the presented progressive strategy of learning can improve the disentangling of ﬁrst the most abstract representations followed by progressively lower levels of abstractions.
Revised: This provides evidence that the presented progressive strategy of learning can improve the disentangling of ﬁrst the most abstract representations followed by progressively lower levels of abstractions.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we present a progressive strategy for learning and disentangling hierarchical representations.
Revised: In this work we present a progressive strategy for learning and disentangling hierarchical representations.

Original: Starting from a simple VAE, the model ﬁrst learn the most abstract representation.
Revised: Starting from a simple VAE, the model ﬁrst learn the most abstract representation.

Original: Next, the model learn independent representations from high- to low-levels of abstraction by progressively growing the capacity of the VAE deep to shallow.
Revised: Next, the model learn independent representations from high- to low-levels of abstraction by progressively growing the capacity of the VAE deep to shallow.

Original: Experiments on several benchmark data sets demonstrated the advantages of the presented method.
Revised: Experiments on several benchmark data sets demonstrated the advantages of the presented method.

Original: An immediate future work is to include stronger guidance for allocating information across the hierarchy of abstraction levels, either through external multi-scale image supervision or internal information-theoretic regularization strategies.
Revised: An immediate future work is to include stronger guidance for allocating information across the hierarchy of abstraction levels, either through external multi-scale image supervision or internal information-theoretic regularization strategies.

Original: A PPENDIX A E XAMPLES OF PERFORMANCE OF DIFFERENT METRICS
Revised: A PPENDIX A E XAMPLES OF PERFORMANCE OF DIFFERENT METRICS

Original: BO N MNIST DATASET : A CLOSER COMPARISON WITH VLAE  
Revised: B A CLOSER COMPARISON WITH VLAE B.

Revised: [Figure] B. I NFORMATION ALLOCATION IN EACH LAYER [Table]

Revised: C A BLATION STUDY ON IMPLEMENTATION STRATEGIES

Revised: [Table]

Revised: D C LOSER INVESTIGATION OF I NFORMATION FLOW OVER LATENT VARIABLES

Revised: In this section, we present additional quantitative results on how information ﬂow among the latent variables during progressive training.

Revised: We conducted experiments on both 3DShapes and MNIST data sets, considering different hierarchical architectures including a combination of different number of latent layers L and different number of latent dimensions z dim for each layer.

Revised: Each experiment was repeated three times with random initializations, from which the mean and the standard deviation of mutual information

Revised: I ( x ; z l ) were computed.

Revised: As shown in Tables 3-8, for all hierarchical architectures, the information amount in each layer is captured in a clear descending order, which aligns with the motivation of the presented progressive learning strategy.

Revised: Generally, the information also tends to ﬂow from previous layers to new layers, suggesting a disentanglement of latent factors as new latent layers are added.

Revised: This is especially obvious for 3DShapes data where the generative factors are better deﬁned.

Revised: In addition, models with small latent codes ( z dim = 1 ) are not able to learn the same amount of information (total I ( x, z ) ) as those with larger latent codes ( z dim = 3 ).

Revised: The variance of information in each layer in the former also appears to be high.

Revised: We reason that it may be because that the model is trying to squeeze too much information into a small code, resulting in large vibrations during progressive learning.

Revised: On the other hand, while a model has large latent codes ( L = 4 , z dim = 3 ), the information ﬂow becomes less clear after the addition of certain layers.

Revised: Overall, assuming there are K generative factors and there are D dimensions in total available in model, ideally we would like to design the model such that D = K .

Revised: However, since K is unknown in most data, L and z dim become hyperparameters that need to be tuned for different data sets.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]