Original: T RADING I NFORMATION BETWEEN L ATENTS IN H IERARCHICAL V ARIATIONAL A UTOENCODERS
Revised: T RADING I NFORMATION BETWEEN L ATENTS IN H IERARCHICAL V ARIATIONAL A UTOENCODERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Variational Autoencoders (VAEs) were originally motivated (Kingma & Welling, 2013) as probabilistic generative models in which one performs approximate
Revised: Variational Autoencoders (VAEs) were originally motivated (Kingma & Welling, 2013) as probabilistic generative models in which one performs approximate

Original: Bayesian inference.
Revised: Bayesian inference.

Original: The proposal of β -VAEs (Higgins et al., 2016) breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing an objective function that allows practitioners to trade off between the information content (“bit rate”) of the latent representation and the distortion of reconstructed data (Alemi et al., 2018).
Revised: The proposal of β -VAEs (Higgins et al., 2016) breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing an objective function that allows practitioners to trade off between the information content (“bit rate”) of the latent representation and the distortion of reconstructed data (Alemi et al., 2018).

Original: In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.e., VAEs with more than one layer of latent variables.
Revised: In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.e., VAEs with more than one layer of latent variables.

Original: Wepropose a method to control each layer’s contribution to the rate independently. We identify the most general class of inference models to which our proposed method is applicable, and we derive theoretical bounds on the performance of downstream tasks as functions of the individual layers’ rates.
Revised: We identify a general class of inference models for which one can split the rate into contributions from each layer, which can then be tuned independently. We derive theoretical bounds on the performance of downstream tasks as functions of the individual layers’ rates and verify our theoretical findings in large-scale experiments.

Original: Our experiments demonstrate that the proposed method allows us to better tune hierarchical VAEs for a diverse set of practical use cases.
Revised: Our results provide guidance for practitioners on which region in rate-space to target for a given application.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) are a class of deep generative models that are used in a diverse set of application domains including density modeling (Takahashi et al., 2018), clustering (Jiang et al., 2016), nonlinear dimensionality reduction of scientific measurements (Laloy et al., 2017), data compression (Ball ´ e et al., 2016), anomaly detection (Xu et al., 2018), and image generation (Razavi et al., 2019).
Revised: Variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) are a class of deep generative models that are used in a diverse set of application domains including density modeling (Takahashi et al., 2018), clustering (Jiang et al., 2016), nonlinear dimensionality reduction of scientific measurements (Laloy et al., 2017), data compression (Ball ´ e et al., 2016), anomaly detection (Xu et al., 2018), and image generation (Razavi et al., 2019).

Original: One of the reasons why the use cases of VAEs (more precisely, β -VAEs (Higgins et al., 2016)) span such a diverse set of application domains is that they can be tuned to a specific task without changing the network architecture, in a way that is well understood from information theory (Alemi et al., 2018).
Revised: One of the reasons why the use cases of VAEs (more precisely, β -VAEs (Higgins et al., 2016)) span such a diverse set of application domains is that they can be tuned to a specific task without changing the network architecture, in a way that is well understood from information theory (Alemi et al., 2018).

Original: The original proposal of VAEs (Kingma & Welling, 2013) motivates them from the perspective of generative probabilistic modeling and approximate Bayesian inference.
Revised: The original proposal of VAEs (Kingma & Welling, 2013) motivates them from the perspective of generative probabilistic modeling and approximate Bayesian inference.

Original: However, the generalization to β -VAEs breaks this interpretation as they are no longer trained by maximizing a lower bound on the marginal data likelihood.
Revised: However, the generalization to β -VAEs breaks this interpretation as they are no longer trained by maximizing a lower bound on the marginal data likelihood.

Original: These models are better described as neural networks that are trained to learn the identity function, i.e., to make their output resemble the input as closely as possible.
Revised: These models are better described as neural networks that are trained to learn the identity function, i.e., to make their output resemble the input as closely as possible.

Original: This task is made nontrivial by introducing a so-called (variational) information bottleneck (Alemi et al., 2016; Tishby & Zaslavsky, 2015) at one or more layers, which restricts the information content that passes through these layers.
Revised: This task is made nontrivial by introducing a so-called (variational) information bottleneck (Alemi et al., 2016; Tishby & Zaslavsky, 2015) at one or more layers, which restricts the information content that passes through these layers.

Original: The network activations at the information bottleneck are called latent representations (or simply “latents”), and they split the network into an encoder part (from input to latents) and a decoder part (from latents to output).
Revised: The network activations at the information bottleneck are called latent representations (or simply “latents”), and they split the network into an encoder part (from input to latents) and a decoder part (from latents to output).

Original: This separation of the model into an encoder and a decoder allows us to categorize  applications of VAEs into three domains:
Revised: This separation of the model into an encoder and a decoder allows us to categorize the wide variety of applications of VAEs into three domains:

Original: data reconstruction tasks , i.e., applications that involve both the encoder and the decoder ; these include various nonlinear inter- and extrapolations (e
Revised: data reconstruction tasks , i.e., applications that involve both the encoder and the decoder ; these include various nonlinear inter- and extrapolations (e

Original: g., image upscaling, denoising, or inpainting), and VAE-based methods for lossy data compression; 2.
Revised: g., image upscaling, denoising, or inpainting), and VAE-based methods for lossy data compression; 2.

Original: representation learning tasks , i.e., applications that involve only the encoder ; they serve a downstream task that operates on the (typically lower dimensional) latent representation, e.g., classification, regression, visualization, clustering, or anomaly detection; and 3.
Revised: representation learning tasks , i.e., applications that involve only the encoder ; they serve a downstream task that operates on the (typically lower dimensional) latent representation, e.g., classification, regression, visualization, clustering, or anomaly detection; and 3.

Original: generative modeling tasks , i.e., applications that involve only the decoder are less common but include generating new samples that resemble training data.
Revised: generative modeling tasks , i.e., applications that involve only the decoder are less common but include generating new samples that resemble training data.

Original: The information bottleneck incentivizes the VAE to encode information into the latents efficiently by removing any redundancies from the input.
Revised: The information bottleneck incentivizes the VAE to encode information into the latents efficiently by removing any redundancies from the input.

Original: How agressively this is done can be controlled by tuning the strength β of the information bottleneck (Alemi et al., 2018).
Revised: How agressively this is done can be controlled by tuning the strength β of the information bottleneck (Alemi et al., 2018).

Original: Unfortunately, information theory distinguishes relevant from redundant information only in a quantitative way that is agnostic to the qualitative features that each piece of information represents about some data point.
Revised: Unfortunately, information theory distinguishes relevant from redundant information only in a quantitative way that is agnostic to the qualitative features that each piece of information represents about some data point.

Original: In practice, many VAE-architectures (Deng et al., 2017; Yingzhen & Mandt, 2018;
Revised: In practice, many VAE-architectures (Deng et al., 2017; Yingzhen & Mandt, 2018;

Original: Ball ´ e et al., 2018) try to separate qualitatively different features into different parts of the latent representation by making the model architecture reflect some prior assumptions about the semantic structure of the data.
Revised: Ball ´ e et al., 2018) try to separate qualitatively different features into different parts of the latent representation by making the model architecture reflect some prior assumptions about the semantic structure of the data.

Original: This allows downstream applications from the three domains discussed above to more precisely target specific qualitative aspects of the data by using or manipulating only the corresponding part of the latent representation.
Revised: This allows downstream applications from the three domains discussed above to more precisely target specific qualitative aspects of the data by using or manipulating only the corresponding part of the latent representation.

Original: However, in this approach, the degree of detail to which each qualitative aspect is encoded in the latents can be controlled at most indirectly by tuning network layer sizes.
Revised: However, in this approach, the degree of detail to which each qualitative aspect is encoded in the latents can be controlled at most indirectly by tuning network layer sizes.

Original: In this paper we argue both theoretically and empirically that the three different application domains of VAEs identified above require different trade-offs in the amount of information that is encoded in each part of the latent representation.
Revised: In this paper we argue both theoretically and empirically that the three different application domains of VAEs identified above require different trade-offs in the amount of information that is encoded in each part of the latent representation.

Original: We propose a method to independently control the information content (or “rate”) of each layer of latent representations, generalizing the rate/distortion theory of β -VAEs (Alemi et al., 2018) for VAEs with more than one layer of latents (“hierarchical VAEs” or
Revised: We propose a method to independently control the information content (or “rate”) of each layer of latent representations, generalizing the rate/distortion theory of β -VAEs (Alemi et al., 2018) for VAEs with more than one layer of latents (“hierarchical VAEs” or

Original: HVAEs for short).
Revised: HVAEs for short).

Original: We identify the most general model architecture that is compatible with our proposal and analyze how both theoretical performance bounds and empirically measured performances in each of the above three application domains depend on how rate is distributed across layers.
Revised: We identify the most general model architecture that is compatible with our proposal and analyze how both theoretical performance bounds and empirically measured performances in each of the above three application domains depend on how rate is distributed across layers.

Original: Our approach is summarized in Figure 1 (discussed in more detail in Section 5.2), where the 3dplot shows empirically measured performance metrics  for the three application domains identified above, and each point on the colored surface corresponds toa different distribution of rates between two layers of latents in a VAE.
Revised: Our approach is summarized in Figure 1. The 3d-plot shows empirically measured performance metrics (discussed in detail in Section 5.2) for the three application domains identified above. Each point on the colored surface corresponds to different layer-wise rates in an HVAE with two layers of latents.

Original: Crucially, the rate distribution that leads to optimal performance is different for each of the three application domains (see markers △ , (cid:68) , and ⋄ in Figure 1), and none of these three optimal models coincide with a conventional β -VAE (dashed red line in right panels).
Revised: Crucially, the rates that lead to optimal performance are different for each of the three application domains (see markers △ , (cid:68) , and ⋄ in Figure 1), and none of these three optimal models coincide with a conventional β -VAE (dashed red lines in right panels).

Original: Thus, the additional control provided by the information-trading method proposed in this paper allows practitioners to more specifically target one of the many application domains of VAEs.
Revised: Thus, being able to control each layer’s individual rate allows practitioners to train VAEs that target a specific application.

Original: The paper is structured as follows.
Revised: The paper is structured as follows.

Original: Section 2 summarizes related work.
Revised: Section 2 summarizes related work.

Original: Section 3 introduces the proposed information-trading method.
Revised: Section 3 introduces the proposed information-trading method.

Original: We  analyze how controlling individual layers’ rates can be used to tune VAEs for specific tasks, i.e., how performance in each of the three application domains identified above depends on the distribution of rates across layers.
Revised: We then analyze how controlling individual layers’ rates can be used to tune HVAEs for specific tasks, i.e., how performance in each of the three application domains identified above depends on the allocation of rates across layers.

Original: This analysis is done theoretically in Section 4 and empirically in Section 5.
Revised: This analysis is done theoretically in Section 4 and empirically in Section 5.

Original: Section 6 provides concluding remarks.
Revised: Section 6 provides concluding remarks.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We group related work into work on model architectures for hierarchical VAEs, and on β -VAEs.
Revised: We group related work into work on model architectures for hierarchical VAEs, and on β -VAEs.

Original: Model Design for Hierarchical VAEs.
Revised: Model Design for Hierarchical VAEs.

Original: The original VAE design (Kingma & Welling, 2013;
Revised: The original VAE design (Kingma & Welling, 2013;

Original: Rezende et al., 2014) has a single layer of latent variables, but recent works (Vahdat & Kautz, 2020;
Revised: Rezende et al., 2014) has a single layer of latent variables, but recent works (Vahdat & Kautz, 2020;

Original: Child, 2020), found that increasing the number of stochastic layers in hierarchical VAEs (HVAEs) improves performance.
Revised: Child, 2020), found that increasing the number of stochastic layers in hierarchical VAEs (HVAEs) improves performance.

Original: HVAEs have various designs for their inference models.
Revised: HVAEs have various designs for their inference models.

Original: Sønderby et al.
Revised: Sønderby et al.

Original: (2016) introduced Ladder VAE (LVAE) with a top-down inference path rather than the naive bottomup inference (see Section 3),
Revised: (2016) introduced Ladder VAE (LVAE) with a top-down inference path rather than the naive bottomup inference (see Section 3),

Original: whereas the Bidirectional-Inference VAE (BIVA) (Maaløe et al., 2019) uses a combination of top-down and bottom-up.
Revised: whereas the Bidirectional-Inference VAE (BIVA) (Maaløe et al., 2019) uses a combination of top-down and bottom-up.

Original: Our proposed framework applies to a large class of inference models (see Section 3) that includes the popular LVAE (Sønderby et al., 2016).
Revised: Our proposed framework applies to a large class of inference models (see Section 3) that includes the popular LVAE (Sønderby et al., 2016).

Original: β -VAEs
Revised: β -VAEs

Original: And Their Information-Theoretical Interpretations.
Revised: And Their Information-Theoretical Interpretations.

Original: Higgins et al.
Revised: Higgins et al.

Original: (2016) introduced an extra hyperparameter β in the objective of VAEs that tunes the strength of the information bottleneck, and they observed that large β leads to a disentangled latent representation.
Revised: (2016) introduced an extra hyperparameter β in the objective of VAEs that tunes the strength of the information bottleneck, and they observed that large β leads to a disentangled latent representation.

Original: An informationtheoretical interpretation of β -VAEs was provided in (Alemi et al., 2018) by applying the concept of a (variational) bottleneck (Tishby & Zaslavsky, 2015;
Revised: An informationtheoretical interpretation of β -VAEs was provided in (Alemi et al., 2018) by applying the concept of a (variational) bottleneck (Tishby & Zaslavsky, 2015;

Original: Alemi et al., 2016) to autoencoders.
Revised: Alemi et al., 2016) to autoencoders.

Original: Due to this information-theoretical interetation, β -VAEs are popular models for data compression (Ball ´ e et al., 2016; Minnen et al., 2018;
Revised: Due to this information-theoretical interetation, β -VAEs are popular models for data compression (Ball ´ e et al., 2016; Minnen et al., 2018;

Original: Yang et al., 2020), where tuning β allows trading off between the bit rate of compressed data and data distortion.
Revised: Yang et al., 2020), where tuning β allows trading off between the bit rate of compressed data and data distortion.

Original: In the present work, we generalize β -VAEs when applied to HVAEs, and we introduce a framework for tuning the rate of each latent layer individually.
Revised: In the present work, we generalize β -VAEs when applied to HVAEs, and we introduce a framework for tuning the rate of each latent layer individually.

Original:  A H IERARCHICAL I NFORMATION T RADING F RAMEWORK
Revised:  A H IERARCHICAL I NFORMATION T RADING F RAMEWORK

Original: We propose a refinement of the rate/distortion theory of β -VAEs (Alemi et al., 2018) that admits controlling individual layers’ rates in VAEs with more than one layers of latents (hierarchical VAEs).
Revised: We propose a refinement of the rate/distortion theory of β -VAEs (Alemi et al., 2018) that admits controlling individual layers’ rates in VAEs with more than one layers of latents (hierarchical VAEs).

Original: 3. C ONVENTIONAL β -VAE W ITH H IERARCHICAL L ATENT R EPRESENTATIONS
Revised: 3. C ONVENTIONAL β -VAE W ITH H IERARCHICAL L ATENT R EPRESENTATIONS

Original: We consider a hierarchical VAE (HVAE) for data x with L layers of latent representations { z ℓ } Lℓ =1 .
Revised: We consider a hierarchical VAE (HVAE) for data x with L layers of latent representations { z ℓ } Lℓ =1 .

Original: Figure 2, discussed further in Section 3.2 below, illustrates various model architectures for the example of L = 2 .
Revised: Figure 2, discussed further in Section 3.2 below, illustrates various model architectures for the example of L = 2 .

Original: Solid arrows depict the generative model p θ ( { z ℓ } , x ) , where θ are model parameters (neural network weights).
Revised: Solid arrows depict the generative model p θ ( { z ℓ } , x ) , where θ are model parameters (neural network weights).

Original: We assume that the generative (or “decoder”) model is a Markov chain,
Revised: We assume that the generative (or “decoder”) model is a Markov chain,

Original: [Equation]
Revised: [Equation]

Original: where the numbering of latent layers from L down to 1 follows the common convention in the literature (Sønderby et al., 2016; Gulrajani et al., 2016; Child, 2020).
Revised: where the numbering of latent layers from L down to 1 follows the common convention in the literature (Sønderby et al., 2016; Gulrajani et al., 2016; Child, 2020).

Original: The loss function of a normal β -VAE (Higgins et al., 2016) with this generic model architecture would be
Revised: The loss function of a normal β -VAE (Higgins et al., 2016) with this generic model architecture would be

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, q ϕ ( { z ℓ } | x ) is the inference (or “encoder”) model with parameteres ϕ , X train is the training set, D KL [ · || · ] denotes Kullback-Leibler divergence, and the Lagrange parameter β > 0 trades off between a (total) rate R and a distortion D (Alemi et al., 2018).
Revised: Here, q ϕ ( { z ℓ } | x ) is the inference (or “encoder”) model with parameteres ϕ , X train is the training set, D KL [ · || · ] denotes Kullback-Leibler divergence, and the Lagrange parameter β > 0 trades off between a (total) rate R and a distortion D (Alemi et al., 2018).

Original: Setting β
Revised: Setting β

Original: = 1 turns Eq.
Revised: = 1 turns Eq.

Original: 2 into the negative ELBO objective of a regular VAE (Kingma & Welling, 2013).
Revised: 2 into the negative ELBO objective of a regular VAE (Kingma & Welling, 2013).

Original: The rate R obtains its name as it measures the (total) information content that q ϕ encodes into the latent representations { z ℓ } , which would manifest itself in the expected bit rate when one optimally encodes a random draw { z ℓ } ∼ q ϕ ( { z ℓ } | x ) using p θ ( { z ℓ } ) as an entropy model (Agustsson & Theis, 2020; Bennett et al., 2002).
Revised: The rate R obtains its name as it measures the (total) information content that q ϕ encodes into the latent representations { z ℓ } , which would manifest itself in the expected bit rate when one optimally encodes a random draw { z ℓ } ∼ q ϕ ( { z ℓ } | x ) using p θ ( { z ℓ } ) as an entropy model (Agustsson & Theis, 2020; Bennett et al., 2002).

Original: An important observation pointed out in (Alemi et al., 2016) is that, regardless how rate R is traded off against distortion D by tuning β , their sum R + D is—in expectation under any data distribution p data ( x ) —always lower bounded by the entropy H [ p data ( x )] :
Revised: An important observation pointed out in (Alemi et al., 2016) is that, regardless how rate R is traded off against distortion D by tuning β , their sum R + D is—in expectation under any data distribution p data ( x ) —always lower bounded by the entropy H [ p data ( x )] :

Original: [Equation]
Revised: [Equation]

Original: x ) [
Revised: x ) [

Original: [Equation]
Revised: [Equation]

Original: data ( x )] ,
Revised: data ( x )] ,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Limitations.
Revised: Limitations.

Original: The rate R in Eq.
Revised: The rate R in Eq.

Original: 2 is a property of the collection { z ℓ } of all latents, which can limit its interpretability for some inference models.
Revised: 2 is a property of the collection { z ℓ } of all latents, which can limit its interpretability for some inference models.

Original: For example, the common convention of enumerating layers z ℓ from ℓ =
Revised: For example, the common convention of enumerating layers z ℓ from ℓ =

Original: L down to 1 in Eq.
Revised: L down to 1 in Eq.

Original: 1 is reminiscent of a naive architecture for the inference model that factorizes in reverse order compared to Eq.
Revised: 1 is reminiscent of a naive architecture for the inference model that factorizes in reverse order compared to Eq.

Original: 1 (“bottom up”, see dashed arrows in Figure 2(a)), i.e., q ϕ ( { z ℓ } | x ) =
Revised: 1 (“bottom up”, see dashed arrows in Figure 2(a)), i.e., q ϕ ( { z ℓ } | x ) =

Original: q ϕ ( z 1 | x ) q ϕ ( z 2 | z 1 ) · · · q ϕ ( z L | z
Revised: q ϕ ( z 1 | x ) q ϕ ( z 2 | z 1 ) · · · q ϕ ( z L | z

Original: L − 1 ) .
Revised: L − 1 ) .

Original: Using a HVAE with such a “bottom-up” inference model to reconstruct some given data point x would map x to z 1 using q ϕ ( z 1 |
Revised: Using a HVAE with such a “bottom-up” inference model to reconstruct some given data point x would map x to z 1 using q ϕ ( z 1 |

Original: x
Revised: x

Original: and then map z 1 back to the data space using p θ ( x | z 1 ) , thus ignoring all latents z ℓ with ℓ > 1 .
Revised: and then map z 1 back to the data space using p θ ( x | z 1 ) , thus ignoring all latents z ℓ with ℓ > 1 .

Original: Yet, the rate term in Eq.
Revised: Yet, the rate term in Eq.

Original: 2 still depends on all latents, including the ones not needed to reconstruct any data (practical VAE-based compression methods using bits-back coding (Frey & Hinton, 1997) would, however, indeed use z ℓ with ℓ > 1 as auxiliary variables for computational efficiency).
Revised: 2 still depends on all latents, including the ones not needed to reconstruct any data (practical VAE-based compression methods using bits-back coding (Frey & Hinton, 1997) would, however, indeed use z ℓ with ℓ > 1 as auxiliary variables for computational efficiency).

Original: 3. T RADING I NFORMATION B ETWEEN L ATENTS
Revised: 3. T RADING I NFORMATION B ETWEEN L ATENTS

Original: Many HVAEs used in the literature allow us  resolve the limitations identified in Section 3.1.
Revised: Many HVAEs used in the literature allow us to resolve the limitations identified in Section 3.1.

Original: For example, the popular LVAE architecture (Sønderby et al., 2016), (Figure 2(b)), uses an inference model (dashed arrows) that traverses the latents { z ℓ } in the same order as the generative model (solid arrows).
Revised: For example, the popular LVAE architecture (Sønderby et al., 2016), (Figure 2(b)), uses an inference model (dashed arrows) that traverses the latents { z ℓ } in the same order as the generative model (solid arrows).

Original: We consider the following generalization of this architecture (see Figure 2(c)),
Revised: We consider the following generalization of this architecture (see Figure 2(c)),

Original: [Equation]
Revised: [Equation]

Original: This architecture is motivated by factorizing in the same form as the true posterior p θ ( { z ℓ } | x ) :
Revised: This architecture is motivated by factorizing in the same form as the true posterior p θ ( { z ℓ } | x ) :

Original: = p θ ( { z ℓ } , x ) /p
Revised: = p θ ( { z ℓ } , x ) /p

Original: θ ( x ) (because the generative model p θ ( { z ℓ } , x ) in Eq.
Revised: θ ( x ) (because the generative model p θ ( { z ℓ } , x ) in Eq.

Original: 4 is a Markov chain).
Revised: 1 is a Markov chain).

Original: Eq.includes
Revised: Eq.includes

Original: LVAE as a special case that ties some parameters of the inference to the generative model.
Revised: LVAE as a special case that ties some parameters between inference and generative model.

Original: Using a trained HVAE with the inference model architecture from Eq.
Revised: Using a trained HVAE with the inference model architecture from Eq.

Original: 4 to map a data point x to its reconstruction ˆ x now involves all latents (either drawing a sample or taking the mode at each step):
Revised: 4 to map a data point x to its reconstruction ˆ x now involves all latents (either drawing a sample or taking the mode at each step):

Original: [Equation]
Revised: [Equation]

Original: Layer-wise Rates.
Revised: Layer-wise Rates.

Original: We can interpret Eq.
Revised: We can interpret Eq.

Original: 5 in that it first maps x to a “crude” representation z L , which gets iteratively refined to z 1 , which is then used to find a reconstruction ˆ x .
Revised: 5 in that it first maps x to a “crude” representation z L , which gets iteratively refined to z 1 , which is then used to find a reconstruction ˆ x .

Original: Note that each factor q ϕ ( z ℓ − 1 | z ℓ , x ) of the inference model is conditioned not only on the previous layer z ℓ but also on the original data x , which allows the inference model to target each refinement such that the reconstruction ˆ x becomes close to x .
Revised: Note that each factor q ϕ ( z ℓ − 1 | z ℓ , x ) of the inference model is conditioned not only on the previous layer z ℓ but also on the original data x , which allows the inference model to target each refinement such that the reconstruction ˆ x becomes close to x .

Original: More formally, the iterative refinement towards the reconstruction ˆ x is expressed by the data processing inequality (MacKay, 2003): the part of Eq.
Revised: More formally, the iterative refinement towards the reconstruction ˆ x is expressed by the data processing inequality (MacKay, 2003): the part of Eq.

Original: 5 from z L to ˆ x forms a Markov chain that is conditioned on x , and therefore the mutual information
Revised: 5 from z L to ˆ x forms a Markov chain that is conditioned on x , and therefore the mutual information

Original: I ( z ℓ ; ˆ x | x ) between a latent z ℓ and the reconstruction ˆ x given the original data point x is nondecreasing along the chain, i.e., I ( z L ; ˆ x | x ) ≤ I ( z L − 1 ;
Revised: I ( z ℓ ; ˆ x | x ) between a latent z ℓ and the reconstruction ˆ x given the original data point x is nondecreasing along the chain, i.e., I ( z L ; ˆ x | x ) ≤ I ( z L − 1 ;

Original: ˆ x | x ) ≤ . . .
Revised: ˆ x | x ) ≤ . . .

Original: ≤ I ( z 1 ; ˆ x | x ) .
Revised: ≤ I ( z 1 ; ˆ x | x ) .

Original: Further, since the inference architecture in Eq.
Revised: Further, since the inference architecture in Eq.

Original: 4 factorizes over { z ℓ } in the same order as the generative model (Eq. 1), we can split the total rate R in an autoregressive way into a sum of layer-wise rates,
Revised: 4 factorizes over { z ℓ } in the same order as the generative model (Eq. 1), we can split the total rate R in an autoregressive way into a sum of layer-wise rates,

Original: [Equation]
Revised: [Equation]

Original: | z 2 ) .
Revised: z 2 ) .

Original: Here,

Original: R

Original: [Equation]
Revised: [Equation]

Original: p θ
Revised: Here,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: quantify the information content of the highest-order latent representation z L and the (expected) increase in information content for each  stepfrom latent z ℓ to latent z ℓ − 1 in Eq.
Revised: quantify the information content of the highest-order latent representation z L and the (expected) increase in information content in each refinement step z ℓ → z ℓ − 1 in Eq.

Original: 5, respectively.
Revised: 5, respectively.

Original: Controlling Each Layer’s Rate.
Revised: Controlling Each Layer’s Rate.

Original: Using Eqs.
Revised: Using Eqs.

Original: 6-7, wecan generalize the rate/distortion trade-off from Section 3.1 by introducing L individual Lagrange multipliers β L , β L − 1 | L , ..., β 1 | 2 , collectively denoted as boldface β .
Revised: 6-7, we generalize the rate/distortion trade-off from Section 3.1 by introducing L individual Lagrange multipliers β L , β L − 1 | L , ..., β 1 | 2 , collectively denoted as boldface β .

Original: This leads to a new loss function that generalizes Eq. 2 as follows,
Revised: This leads to a new loss function that generalizes Eq. 2 as follows,

Original: [Equation]
Revised: [Equation]

Original: [Equation] Setting all β ’s to the same value recovers the conventional β -VAE (Eq. 2), which trades off distortion against total information content in { z ℓ } .
Revised: Setting all β ’s to the same value recovers the conventional β -VAE (Eq. 2), which trades off distortion against total information content in { z ℓ } .

Original: Tuning each β -hyperparameter individually gives us more control and allows trading-off information content across latents.
Revised: Tuning each β -hyperparameter individually gives us more control and allows trading-off information content across latents.

Original: We analyze how this can be useful depending on application domain theoretically in Section 4 and empirically in Section 5.
Revised: We analyze how this can be useful depending on application domain theoretically in Section 4 and empirically in Section 5.

Original:  I NFORMATION -T HEORETICAL P ERFORMANCE B OUNDS FOR HVAE S
Revised:  I NFORMATION -T HEORETICAL P ERFORMANCE B OUNDS FOR HVAE S

Original: In this section, we analyze theoretically how various performance metrics for HVAEs are restricted by the individual layers’ rates R ( z L ) and R ( z ℓ − 1 | z ℓ ) identified in Eq.
Revised: In this section, we analyze theoretically how various performance metrics for HVAEs are restricted by the individual layers’ rates R ( z L ) and R ( z ℓ − 1 | z ℓ ) identified in Eq.

Original: 7 for a HVAE with “topdown” inference model.
Revised: 7 for a HVAE with “topdown” inference model.

Original: Our analysis motivates the use of the information-trading loss function in
Revised: Our analysis motivates the use of the information-trading loss function in

Original: Eq. 8 for training HVAEs, following the argument from the introduction that VAEs are commonly used for a vast variety of tasks.
Revised: Eq. 8 for training HVAEs, following the argument from the introduction that VAEs are commonly used for a vast variety of tasks.

Original: As we show, different tasks require different trade-offs that can be targeted by tuning the Lagrange multipliers β in Eq.
Revised: As we show, different tasks require different trade-offs that can be targeted by tuning the Lagrange multipliers β in Eq.

Original: We group tasks into the application domains  (i) data reconstruction and manipulation, (ii) representation learning, and (iii) data generation.
Revised: We group tasks into the application domains of (i) data reconstruction and manipulation, (ii) representation learning, and (iii) data generation.

Original: Data Reconstruction and Manipulation.
Revised: Data Reconstruction and Manipulation.

Original: The most obvious class of application domains of VAEs includes tasks that combine encoder and decoder to map some data point x to representations { z ℓ } and then back to the data space.
Revised: The most obvious class of application domains of VAEs includes tasks that combine encoder and decoder to map some data point x to representations { z ℓ } and then back to the data space.

Original: The simplest performance metric for such data reconstruction tasks is the expected distortion E p data ( x ) [ D ] , which we can bound by combining Eq.
Revised: The simplest performance metric for such data reconstruction tasks is the expected distortion E p data ( x ) [ D ] , which we can bound by combining Eq.

Original: 3 with Eqs.
Revised: 3 with Eqs.

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 9 would suggest that higher rates (i.e., lower β ’s) are always better for data reconstruction tasks.
Revised: 9 would suggest that higher rates (i.e., lower β ’s) are always better for data reconstruction tasks.

Original: However, in many practical tasks (e.g., image upscaling, denoising, or inpainting) the goal is not solely to reconstruct the original data but also to manipulate the latent representations { z ℓ } in a meaningful way.
Revised: However, in many practical tasks (e.g., image upscaling, denoising, or inpainting) the goal is not solely to reconstruct the original data but also to manipulate the latent representations { z ℓ } in a meaningful way.

Original: Here, lower rates can lead to more semantically meaningful representation spaces (see, e.g., Section 5.6 below).
Revised: Here, lower rates can lead to more semantically meaningful representation spaces (see, e.g., Section 5.6 below).

Original: Being able to distribute rate  across layers via Eq.
Revised: Controlling how rate is distributed across layers via Eq.

Original: 8 may allow practitioners to have a semantically meaningful high-level representation z L with low rate R ( z L ) while still retaining a high total rate R , thus allowing for low distortion D without violating Eq. 9.
Revised: 8 may allow practitioners to have a semantically meaningful high-level representation z L with low rate R ( z L ) while still retaining a high total rate R , thus allowing for low distortion D without violating Eq. 9.

Original: Representation Learning.
Revised: Representation Learning.

Original: In many practical applications,e.g., in natural sciences, VAEs are used aspowerful nonlinear dimensionality reduction methods to prepare some complicated highdimensional data x for downstream tasks such as classification, regression, visualization, clustering, or anomaly detection.
Revised: In many practical applications, VAEs are used as nonlinear dimensionality reduction methods to prepare some complicated high-dimensional data x for downstream tasks such as classification, regression, visualization, clustering, or anomaly detection.

Original: We consider a classifier p cls.
Revised: We consider a classifier p cls.

Original: ( y | z ℓ ) operating on the latents z ℓ at some level ℓ .
Revised: ( y | z ℓ ) operating on the latents z ℓ at some level ℓ .

Original: We assume that the (unknown) true data generative process p data (  x, y ) =
Revised: We assume that the (unknown) true data generative process p data ( y, x ) =

Original: p data ( y ) p data ( x | y ) generates data x conditioned on some true label y , thus defining a Markov chain y p data −−→ x q ϕ −→ z ℓ p cls.
Revised: p data ( y ) p data ( x | y ) generates data x conditioned on some true label y , thus defining a Markov chain y p data −−→ x q ϕ −→ z ℓ p cls.

Original: −−→ 
Revised: −−→ ˆ y where ˆ y := arg max y p cls.

Original: ˆ y.
Revised: ( y | z ℓ ) .

Original: Classification accuracy is bounded (Meyen, 2016) by a function of the mutual information I q ( y ; z ℓ ) 
Revised: Classification accuracy is bounded (Meyen, 2016) by a function of the mutual information I q ( y ; z ℓ ) ,

Original: I q ( y ; z ℓ ) ≤ I q ( x ; z ℓ ) ≡ E p data ( x )
Revised: I q ( y ; z ℓ ) ≤ I q ( x ; z ℓ ) ≡ E p data ( x )

Original: [Equation]
Revised: [Equation]

Original: Here, q ϕ ( z ℓ ) :=
Revised: Here, q ϕ ( z ℓ ) :=

Original: E p data ( x ) [ q ϕ ( z ℓ | x )] , the notations { z ≥ ℓ } and { z >ℓ } denote all latent representations from z L up to z ℓ (including z ℓ for the former), the first inequality comes from the data processing inequality (MacKay, 2003), and the other two inequalities result from discarding the (nonnegative)
Revised: E p data ( x ) [ q ϕ ( z ℓ | x )] , the notations { z ≥ ℓ } and { z >ℓ } denote all latent representations from z L up to z ℓ (including z ℓ for the former), the first inequality comes from the data processing inequality (MacKay, 2003), and the other two inequalities result from discarding the (nonnegative)

Original: KL-terms.
Revised: KL-terms.

Original: The classification accuracy is thus bounded by (Meyen, 2016)
Revised: The classification accuracy is thus bounded by (Meyen, 2016)

Revised: (see also Appendix A.3)

Original: [Equation]
Revised: [Equation]

Original: where f − 1 is the inverse of the monotonic function f ( α )
Revised: where f − 1 is the inverse of the monotonic function f ( α )

Original: =
Revised: =

Original: H [ p data ( y )]
Revised: H [ p data ( y )]

Original: + α log α +(1 − α ) log 1 − [Equation] M −with M being the number of classes and H [ p data ( y )] ≤ log M the marginal label entropy.
Revised: + α log α +(1 − α ) log 1 − α M −with M being the number of classes and H [ p data ( y )] ≤ log M the marginal label entropy.

Original: Eq.suggests that the accuracy of an optimal classifier on z ℓ would increase as the rate R ( z ℓ ) accumulated from z L to z ℓ grows (i.e., as β ≥ ℓ → 0 ), and that the rate added in downstream layers { z <ℓ } would be irrelevant.
Revised: Eq.suggests that the accuracy of an optimal classifier on z ℓ would increase as the rate R ( z ℓ ) accumulated from z L to z ℓ grows (i.e., as β ≥ ℓ → 0 ), and that the rate added in downstream layers { z <ℓ } would be irrelevant.

Original: Practical classifiers, however, have a limited expressiveness, which a very high rate R ( z ℓ ) might exceed by encoding too many details into z ℓ that are not necessary for classification.
Revised: Practical classifiers, however, have a limited expressiveness, which a very high rate R ( z ℓ ) might exceed by encoding too many details into z ℓ that are not necessary for classification.

Original: We observe in Section 5.6 that, in such cases, increasing the rates of downstream layers { z <ℓ } improves classification accuracy as it allows keeping z ℓ simpler by encoding details in { z <ℓ } .
Revised: We observe in Section 5.6 that, in such cases, increasing the rates of downstream layers { z <ℓ } improves classification accuracy as it allows keeping z ℓ simpler by deferring details to { z <ℓ } .

Original: Data Generation.
Revised: Data Generation.

Original: The original proposal of VAEs (Kingma & Welling, 2013) motivated them from a generative modeling perspective using that, for β = 1 , the negative of the loss function in Eq.
Revised: The original proposal of VAEs (Kingma & Welling, 2013) motivated them from a generative modeling perspective using that, for β = 1 , the negative of the loss function in Eq.

Original: 2 is a lower bound on the log marginal data likelihood.
Revised: 2 is a lower bound on the log marginal data likelihood.

Original: This suggests setting all β -hyperparameters in
Revised: This suggests setting all β -hyperparameters in

Original: Eq.
Revised: Eq.

Original: 8 to values close to 1 if a HVAE is used primarily for its generative model p θ .
Revised: 8 to values close to 1 if a HVAE is used primarily for its generative model p θ .

Original: In summary, our theoretical analysis suggests that optimally tuned layer-wise rates depend on whether a HVAE is used for data reconstruction, representation learning, or data generation.
Revised: In summary, our theoretical analysis suggests that optimally tuned layer-wise rates depend on whether a HVAE is used for data reconstruction, representation learning, or data generation.

Original: The 
Revised: The next section tests our theoretical predictions empirically for the same three application domains.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To demonstrate the features of our hierarchical information trading framework, we run large-scale grid searches over a two-dimensional rate space using two different implementations of HVAEs and three different data sets.
Revised: To demonstrate the features of our hierarchical information trading framework, we run large-scale grid searches over a two-dimensional rate space using two different implementations of HVAEs and three different data sets.

Original: Although the proposed framework is applicable for HVAEs with L ≥ 2 , we only use HVAEs with L = 2 in our experiments for simplicity and visualization purpose.
Revised: Although the proposed framework is applicable for HVAEs with L ≥ 2 , we only use HVAEs with L = 2 in our experiments for simplicity and visualization purpose.

Original: 5. E XPERIMENTAL SETUP
Revised: 5. E XPERIMENTAL SETUP

Original: Data sets.
Revised: Data sets.

Original: We used the SVHN (Netzer et al., 2011) and CIFAR-10 (Krizhevsky et al., 2009) data sets (both 32 × 32 pixel color images), and MNIST (LeCun et al., 1998) ( 28 × 28 binary pixel images).
Revised: We used the SVHN (Netzer et al., 2011) and CIFAR-10 (Krizhevsky et al., 2009) data sets (both 32 × 32 pixel color images), and MNIST (LeCun et al., 1998) ( 28 × 28 binary pixel images).

Original: SVHN consists of photographed house numbers from 0 to 9, which are geometrically simpler than the 10 classes of objects from CIFAR-10 but more complex than MNIST digits.
Revised: SVHN consists of photographed house numbers from 0 to 9, which are geometrically simpler than the 10 classes of objects from CIFAR-10 but more complex than MNIST digits.

Original: Most results shown in the main paper use SVHN; comprehensive results for CIFAR-10 and MNIST are shown in Appendix A.2, which tell a similar story except where explicitly discussed.
Revised: Most results shown in the main paper use SVHN; comprehensive results for CIFAR-10 and MNIST are shown in Appendix A.2 and tell a similar story except where explicitly discussed.

Original: Model Architectures.
Revised: Model Architectures.

Original: For the generative model (Eq. 1), we assume a (fixed) standard Gaussian prior p ( z 2 ) =
Revised: For the generative model (Eq. 1), we assume a (fixed) standard Gaussian prior p ( z 2 ) =

Original: N ( 0 , I ) , and we use diagonal Gaussian models for p θ ( z 1 | z 2 ) =
Revised: N ( 0 , I ) , and we use diagonal Gaussian models for p θ ( z 1 | z 2 ) =

Original: N ( g µ ( z 2 ) , g σ ( z 2 ) 2 ) and (for SVHN and CIFAR-10)
Revised: N ( g µ ( z 2 ) , g σ ( z 2 ) 2 ) and (for SVHN and CIFAR-10)

Original: p θ ( x | z 1 )
Revised: p θ ( x | z 1 )

Original: =
Revised: =

Original: N ( g µ ′ ( z 1 ) , σ 2 x I ) (this is similar to, e.g., (Minnen et al., 2018)).
Revised: N ( g µ ′ ( z 1 ) , σ 2 x I ) (this is similar to, e.g., (Minnen et al., 2018)).

Original: Here, g µ , g σ , and g µ ′ , denote neural networks (see details below).
Revised: Here, g µ , g σ , and g µ ′ , denote neural networks (see details below).

Original: Since MNIST has binary pixel values, we model it with a Bernoulli distribution for p θ ( x | z 1 ) =
Revised: Since MNIST has binary pixel values, we model it with a Bernoulli distribution for p θ ( x | z 1 ) =

Original: Bern( g µ ′ ( z 1 )) .
Revised: Bern( g µ ′ ( z 1 )) .

Original: For the inference model, we also use diagonal Gaussian models for q ϕ ( z 2 | x ) =
Revised: For the inference model, we also use diagonal Gaussian models for q ϕ ( z 2 | x ) =

Original: N ( f µ ( x ) , f σ ( x ) 2 ) and for q ϕ ( z 1 | x , z 2 ) =
Revised: N ( f µ ( x ) , f σ ( x ) 2 ) and for q ϕ ( z 1 | x , z 2 ) =

Original: N ( f µ ′ ( x , z 2 ) , f σ ′ ( x , z 2 ) 2 ) , where f µ , f σ , f µ ′ , and f σ ′ are again neural networks.
Revised: N ( f µ ′ ( x , z 2 ) , f σ ′ ( x , z 2 ) 2 ) , where f µ , f σ , f µ ′ , and f σ ′ are again neural networks.

Original: We examine both LVAE (Figure 2(b)) and our generalized top-down HVAEs (GHVAEs; see Figure 2(c)), using simple network architectures with only 2 to 3 convolutional and 1 fully connected layers (see Appendix A.1 for details) so that we can scan a large rate-space efficiently.
Revised: We examine both LVAE (Figure 2(b)) and our generalized top-down HVAEs (GHVAEs; see Figure 2(c)), using simple network architectures with only 2 to 3 convolutional and 1 fully connected layers (see Appendix A.1 for details) so that we can scan a large rate-space efficiently.

Original: Note that we are not trying to find the new state-of-the-art HVAEs.
Revised: Note that we are not trying to find the new state-of-the-art HVAEs.

Original: Results for LVAE are in Appendix A.2.2.
Revised: Results for LVAE are in Appendix A.2.2.

Original: We trained 441 different HVAEs for each data set/model combination, scanning the ratehyperparameters ( β 2 , β 1 | 2 ) over a 21 × 21 grid ranging from 0 .
Revised: We trained 441 different HVAEs for each data set/model combination, scanning the ratehyperparameters ( β 2 , β 1 | 2 ) over a 21 × 21 grid ranging from 0 .

Original: 1 to 10 on a log scale in both directions (see Figure 1, top right panel).
Revised: 1 to 10 on a log scale in both directions (see Figure 1 on page 2, right panels).

Original: Each model took about 2 hours to train on an RTX-2080Ti
Revised: Each model took about 2 hours to train on an RTX-2080Ti

Original: GPU ( ∼ 27 hours in total for each data set/model combination using 32 GPUs in parallel).
Revised: GPU ( ∼ 27 hours in total for each data set/model combination using 32 GPUs in parallel).

Original: Baselines.
Revised: Baselines.

Original: Our proposed framework (Eq.
Revised: Our proposed framework (Eq.

Original: 8) generalizes over both VAEs and β -VAEs (Eq. 2), which we obtain in the cases β 2 = β 1 | 2 = 1 and β 2 = β 1 | 2 , respectively.
Revised: 8) generalizes over both VAEs and β -VAEs (Eq. 2), which we obtain in the cases β 2 = β 1 | 2 = 1 and β 2 = β 1 | 2 , respectively.

Original: These baselines are indicated as blackcircles “ ” and redcircles “ ”, respectively, in Figures 3, 5, 6, and 7. [Figure] [Figure] [Figure]
Revised: These baselines are indicated as black “ ” and red “ ” circles, respectively, in Figures 3, 5, 6, and 7, discussed below.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: SVHN

Original: [Figure]
Revised: [Figure]

Original: z

Original: [Figure]

Original: [Figure]

Original: CIFAR

Original: [Figure]

Original: [Figure]

Original: Metrics.
Revised: Metrics.

Original: Performance metrics for the three application domains of VAEs mentioned in the introduction are introduced at the beginnings of the corresponding Sections 5.4-5.6.
Revised: Performance metrics for the three application domains of VAEs mentioned in the introduction are introduced at the beginnings of the corresponding Sections 5.4-5.6.

Original: In addition, we evaluate the individual rates R ( z 2 ) and R ( z 1 | z 2 )
Revised: In addition, we evaluate the individual rates R ( z 2 ) and R ( z 1 | z 2 )

Original: (Eq. 7), which we report in nats (i.e., to base e ).
Revised: (Eq. 7), which we report in nats (i.e., to base e ).

Original: 5. T HERE IS NO “O NE HVAE F ITS A LL ”
Revised: 5. T HERE IS NO “O NE HVAE F ITS A LL ”

Original: Figure 1 on page 2 summarizes our results.
Revised: Figure 1 on page 2 summarizes our results.

Original: The 21 × 21 GHVAEs trained with the grid of hyperparameters β1 | 2 and β  2 map out a surface in a 3d-space spanned by suitable metrics for the three application domains (metrics defined in Sections 5.4-5.6 below).
Revised: The 21 × 21 GHVAEs trained with the grid of hyperparameters β 2 and β 1 | 2 map out a surface in a 3d-space spanned by suitable metrics for the three application domains (metrics defined in Sections 5.4-5.6 below).

Original: The  colors of the surface encode the β -pairs that were used for training, which also identify their rates in the lower right panel of the figure.
Revised: The two upper right panels map colors on this surface to β s used for training and to the resulting layer-wise rates, respectively.

Original: The GHVAEs with best reconstruction performance  ( △ ),best generative performance ( (cid:68) ) and best representation learningperformance ( ⋄ ) are marked on the surface.
Revised: The lower right panels show performance landscapes and identify the optimal models for the three application domains of data reconstruction ( △ ), representation learning ( ⋄ ), and generative modeling ( (cid:68) ).

Original: The figure shows that moving away from the standard VAE and β -VAE space allows us to find better models for a given application domain, and that the three application domains favor models with noticeably different  β ’s. Thus, there is no single HVAE that is optimal for all tasks , and a
Revised: The figure shows that moving away from a conventional β -VAE ( β 2 = β 1 | 2 ; dashed red lines in Figure 1) allows us to find better models for a given application domain as the three application domains favor vastly different regions in β -space.

Original:  HVAE that has been optimized for one task is unlikely to be a good model for a different task.
Revised: Thus, there is no single HVAE that is optimal for all tasks , and a HVAE that has been optimized for one task can perform poorly on a different task.

Original: 5. D EFINITION OF THE O PTIMAL M ODEL FOR A G IVEN T OTAL R ATE
Revised: 5. D EFINITION OF THE O PTIMAL M ODEL FOR A G IVEN T OTAL R ATE

Original: One of the questions we study in Sections 5.4-5.6 below is: “Which distribution of rates across layers results in best model performance if we keep the total rate R fixed ”.
Revised: One of the questions we study in Sections 5.4-5.6 below is: “Which allocation of rates across layers results in best model performance if we keep the total rate R fixed ”.

Original: Unfortunately, it is difficult to keep R fixed at training time since wehave direct control  only over the Lagrange multipliers βand β 1 | 2 .
Revised: Unfortunately, it is difficult to keep R fixed at training time since we control rates only indirectly via their Lagrange multipliers βand β 1 | 2 .

Original: We instead use the following definition, illustrated in Figure 6 for a performance metric introduced in Section 5.6 below.
Revised: We instead use the following definition, illustrated in Figure 6 for a performance metric introduced in Section 5.6 below.

Original: The figure plots the performance metric over R for all 21 × 21 βsettings and highlights  all points on the upper convex hull with purple circles “ ”.
Revised: The figure plots the performance metric over R for all 21 × 21 βsettings and highlights with purple circles “ ” all points on the upper convex hull.

Original: These highlighted models are optimal for a small interval of total rates in the following sense: if we use the total rates R of all “ ” to partition the horizontal axis into intervals then, by definition of the convex hull, each “ ” represents the model with highest performance in either the interval to its left or the one to its right.
Revised: These highlighted models are optimal for a small interval of total rates in the following sense: if we use the total rates R of all “ ” to partition the horizontal axis into intervals then, by definition of the convex hull, each “ ” represents the model with highest performance in either the interval to its left or the one to its right.

Original: 5. P ERFORMANCE ON D ATA R ECONSTRUCTION
Revised: 5. P ERFORMANCE ON D ATA R ECONSTRUCTION

Original: Reconstruction is a popular task for VAEs, e.g., in the area of lossy compression (Ball´e et al., 2016).
Revised: Reconstruction is a popular task for VAEs, e.g., in the area of lossy compression (Ball´e et al., 2016).

Original: We measure reconstruction quality using the common peak signal-to-noise ratio (PSNR), which is equal to E x ∼ X test [ − log D ] up to rescaling and shifting.
Revised: We measure reconstruction quality using the common peak signal-to-noise ratio (PSNR), which is equal to E x ∼ X test [ − log D ] up to rescaling and shifting.

Original: Higher PSNR means better reconstruction.
Revised: Higher PSNR means better reconstruction.

Original: Figure 3(a) shows a 3d-plot of PSNR as a function of both R ( z 1 | z 2 ) and R ( z 2 ) for SVHN, thus generalizing the rate/distortion curve of a conventional β -VAE to a rate/rate/distortion surface.
Revised: Figure 3(a) shows a 3d-plot of PSNR as a function of both R ( z 1 | z 2 ) and R ( z 2 ) for SVHN, thus generalizing the rate/distortion curve of a conventional β -VAE to a rate/rate/distortion surface.

Original: Figure 3(b) introduces a more compact 2d-representation of the same data that we use for all remaining metrics in the rest of this section and in Appendix A.2, and it also shows results for CIFAR-10.
Revised: Figure 3(b) introduces a more compact 2d-representation of the same data that we use for all remaining metrics in the rest of this section and in Appendix A.2, and it also shows results for CIFAR-10.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: in Figure

Original: Unsurprisingly and consistently with Eq.
Revised: Unsurprisingly and consistent with Eq.

Original: 9, reconstruction performance improves as total rate grows.
Revised: 9, reconstruction performance improves as total rate grows.

Original: However, minimizing distortion without any constraints is not useful  as we can simply use the original data, which has zero distortion.
Revised: However, minimizing distortion without any constraints is not useful in practice as we can simply use the original data, which has no distortion.

Original: To simulate a practical constraint in, e g., a data-compression application, we consider models with optimal PSNR for a given total rate R (as defined in Section 5.3) which are marked as purple circles “ ” in Figure 3(b).
Revised: To simulate a practical constraint in, e.g., a datacompression application, we consider models with optimal PSNR for a given total rate R (as defined in Section 5.3) which are marked as purple circles “ ” in Figure 3(b).

Original: We see for both SVHN and CIFAR10 that conventional β -VAEs ( β 2 = β 1 | 2 , red circles) perform somewhat suboptimal for a given total rate and can be improved by trading some rate in z 2 for some rate in z 1 .
Revised: We see for both SVHN and CIFAR-10 that conventional β -VAEs ( β 2 = β 1 | 2 ; red circles) perform somewhat suboptimal for a given total rate and can be improved by trading some rate in z 2 for some rate in z 1 .

Original: Reconstruction samples for the three models marked with crosses in Figure 3(b) are shown in Figure 4 (bottom).
Revised: Reconstruction examples for the three models marked with crosses in Figure 3(b) are shown in Figure 4 (bottom).

Original: Visual reconstruction quality improves from “3” to “2” to “1”, consistent with reported PSNRs.
Revised: Visual reconstruction quality improves from “3” to “2” to “1”, consistent with reported PSNRs.

Original: 5. P ERFORMANCE ON S AMPLE G ENERATION
Revised: 5. P ERFORMANCE ON S AMPLE G ENERATION

Original: We next evaluate how tuning layer-wise rates affects the quality of samples from the generative model.
Revised: We next evaluate how tuning layer-wise rates affects the quality of samples from the generative model.

Original: We measure sample quality by the widely used Inception Score (IS) (Salimans et al., 2016),
Revised: We measure sample quality by the widely used Inception Score (IS) (Salimans et al., 2016),

Original: [Equation]
Revised: [Equation]

Original: Here, p θ is the trained generative model (Eq. 1), p cls.
Revised: Here, p θ is the trained generative model (Eq. 1), p cls.

Original: ( y | x ) is the predictive distribution of a classifier trained on the same training set, and p cls.
Revised: ( y | x ) is the predictive distribution of a classifier trained on the same training set, and p cls.

Original: ( y ) :
Revised: ( y ) :

Original: = E p θ ( x ) [ p cls.
Revised: = E p θ ( x ) [ p cls.

Original: ( y | x )] .
Revised: ( y | x )] .

Original: The second equality in Eq.follows (Barratt & Sharma, 2018) to split IS into a product of a diversity score and a sharpness score.
Revised: The second equality in Eq.follows Barratt & Sharma (2018) to split IS into a product of a diversity score and a sharpness score.

Original: Higher is better for all scores.
Revised: Higher is better for all scores.

Original: The classifier is a ResNet-18 (He et al., 2016) for SVHN (test accuracy 95 .
Revised: The classifier is a ResNet-18 (He et al., 2016) for SVHN (test accuracy 95 .

Original: 02 % ) and a DenseNet-121 (Huang et al., 2017) for CIFAR-10 (test accuracy 94 .
Revised: 02 % ) and a DenseNet-121 (Huang et al., 2017) for CIFAR-10 (test accuracy 94 .

Original: Figure 5 (left) shows IS for GHVAEs trained on SVHN.
Revised: Figure 5 (left) shows IS for GHVAEs trained on SVHN.

Original: Unlike the results for PSNR, here, higher rate does not always lead to better sample quality: for very high R ( z 2 ) and low R ( z 1 | z 2 ) , IS eventually drops.
Revised: Unlike the results for PSNR, here, higher rate does not always lead to better sample quality: for very high R ( z 2 ) and low R ( z 1 | z 2 ) , IS eventually drops.

Original: The region of high IS is in the area where β 2 < β 1 | 2 , i.e., where R ( z 2 ) is higher than in a comparable conventional β -VAE.
Revised: The region of high IS is in the area where β 2 < β 1 | 2 , i.e., where R ( z 2 ) is higher than in a comparable conventional β -VAE.

Original: The center and right panels of Figure 5 show diversity and sharpness, indicating that IS is mainly driven here by sharpness, which depends mostly on R ( z 2 ) , possibly because z 2 captures higher-level concepts than z 1 that may be more important to the classifier in Eq.
Revised: The center and right panels of Figure 5 show diversity and sharpness, indicating that IS is mainly driven here by sharpness, which depends mostly on R ( z 2 ) , possibly because z 2 captures higher-level concepts than z 1 that may be more important to the classifier in Eq.

Original: Samples from the the three models marked with crosses in Figure 5 are shown in
Revised: Samples from the the three models marked with crosses in Figure 5 are shown in

Original: Figure 4 (top).
Revised: Figure 4 (top).

Original: Visual sample quality improves from “1” to “3” to “2”, consistent with reported IS.
Revised: Visual sample quality improves from “1” to “3” to “2”, consistent with reported IS.

Original: Markers “ ”, “ ”, and “ ” same as in Figure 
Revised: Markers “ ”, “ ”, and “ ” same as in Figure 3.

Original: 5. P ERFORMANCE ON R EPRESENTATION L EARNING FOR D OWNSTREAM C LASSIFICATION
Revised: 5. P ERFORMANCE ON R EPRESENTATION L EARNING FOR D OWNSTREAM C LASSIFICATION

Original: VAEs are very popular for representation learning as they map complicated high dimensional data x to typically lower dimensional representations { z ℓ } .
Revised: VAEs are very popular for representation learning as they map complicated high dimensional data x to typically lower dimensional representations { z ℓ } .

Original: To measure the quality of learned representations, we train two sets of classifiersfor each trained HVAE on a labeled test set, each consisting of: logistic regression, a Support Vector Machine (SVM) (Boser et al., 1992) with linear kernel, an SVM with RBF kernel, and k -nearest neighbors (kNN) with k = 5 .
Revised: To measure the quality of learned representations, we train two sets of classifiers on a labeled test set for each trained HVAE, each consisting of: logistic regression, a Support Vector Machine (SVM) (Boser et al., 1992) with linear kernel, an SVM with RBF kernel, and k -nearest neighbors (kNN) with k = 5 .

Original: One set of classifiers is conditioned on the mode µ 2 of q ϕ
Revised: One set of classifiers is conditioned on the mode µ 2 of q ϕ

Original: ( z 2 | x ) and the other  on the mode µ 1 of q ϕ
Revised: ( z 2 | x ) and the other one on the mode µ 1 of q ϕ

Original: ( z 1 | z 2 , x ) , where z 2 ∼ q ϕ ( z 2 | 
Revised: ( z 1 | z 2 , x ) , where z 2 ∼ q ϕ ( z 2 | x ) .

Original: We use the implementations from scikit-learn (Pedregosa et al., 2011) for all classifiers.
Revised: We use the implementations from scikit-learn (Pedregosa et al., 2011) for all classifiers.

Original: [Table]
Revised: [Table]

Original: Figure 7 shows the classification accuracies (columns 2-5) for all classifiers trained on µ 2 .
Revised: Figure 7 shows the classification accuracies (columns 2-5) for all classifiers trained on µ 2 .

Original: The first column shows  I q (  z 2; y ) , which depends mainly on R ( z 2 ) as expected.
Revised: The first column shows the mutual information I q ( y ; z 2 ) , which depends mainly on R ( z 2 ) as expected from Eq. 10.

Original: As long as the classifier is expressive enough (e.g., RBF-SVM and kNN) and the data set is simple (SVHN; top row), higher mutual information  corresponds to higher classification accuracies, consistent with Eq.
Revised: As long as the classifier is expressive enough (e.g., RBF-SVM or kNN) and the data set is simple (SVHN; top row), higher mutual information ( ≈ higher R ( z 2 ) ) corresponds to higher classification accuracies, consistent with Eq.

Original: But for less expressive (e.g., linear) classifiers or more complex data (CIFAR-10), increasing R ( z 1 | z 2 ) improves classification accuracy (see purple circles  in corresponding panels), consistent with the discussion below Eq.
Revised: But for less expressive (e.g., linear) classifiers or more complex data (CIFAR-10; bottom row), increasing R ( z 1 | z 2 ) improves classification accuracy (see purple circles “ ” in corresponding panels), consistent with the discussion below Eq.

Original: We see a similar effect (see Table 1) for most  combinationsof classifiers and data sets when switching out µ 2 for µ 1 , which has more information about x but is also higher dimensional.
Revised: We see a similar effect (Table 1) for most classifier/data set combinations when replacing µ 2 by µ 1 , which has more information about x but is also higher dimensional.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We classified the various tasks that can be performed with Variational Autoencoders (VAEs) into three application domains and argued that each domain has different trade-offs, such that a good
Revised: We classified the various tasks that can be performed with Variational Autoencoders (VAEs) into three application domains and argued that each domain has different trade-offs, such that a good

Original: VAE for one domain is not necessarily good for another.
Revised: VAE for one domain is not necessarily good for another.

Original: This observation motivated us to propose a refinement of the rate/distortion theory of VAEs that allows trading off rates across individual layers of latents in hierarchical VAEs.
Revised: This observation motivated us to propose a refinement of the rate/distortion theory of VAEs that allows trading off rates across individual layers of latents in hierarchical VAEs.

Original: We showed both theoretically and empirically that the proposal indeed provides practitioners better control for tuning VAEs for the three application domains.
Revised: We showed both theoretically and empirically that the proposal indeed provides practitioners better control for tuning VAEs for the three application domains.

Original: In the future, it would be interesting to find an adaptive schedule for the Lagrange parameters β L and β ℓ | ℓ +1 that makes it possible to target a specific given rate for each layer in a single training run.
Revised: In the future, it would be interesting to find an adaptive schedule for the Lagrange parameters β L and β ℓ | ℓ +1 that makes it possible to target a specific given rate for each layer in a single training run.

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: We have included our code as a .zip file with the submission, which will also be open source on GitHub after the reviewing process.
Revised: We have included our code as a .zip file with the submission, which will also be open source on GitHub after the reviewing process.

Original: I MPLEMENTATION D ETAILS
Revised: I MPLEMENTATION D ETAILS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. A DDITIONAL R ESULTS
Revised: A. A DDITIONAL R ESULTS

Original: Here we attached the results for MNIST, as well as the full results for LVAE on SVHN and generalized top-down HVAEs on CIFAR-10.
Revised: Here we attached the results for MNIST, as well as the full results for LVAE on SVHN and generalized top-down HVAEs on CIFAR-10.

Original: A.2. R ESULTS FOR GENERALIZED TOP - DOWN HVAE S ON MNIST
Revised: A.2. R ESULTS FOR GENERALIZED TOP - DOWN HVAE S ON MNIST

Original: We also evaluate our proposed framework using generalized top-down HVAEs trained on binary
Revised: We also evaluate our proposed framework using generalized top-down HVAEs trained on binary

Original: MNIST data (i
Revised: MNIST data (i

Original: [Figure]
Revised: [Figure]

Original: e., black and white images rather than grayscale)
Revised: e., black and white images rather than grayscale)

Original: [Figure]
Revised: [Figure]

Original: We note that the inception score (IS) behaves different in our MNIST models compared to SVHN (see Figure 5) in that optimal IS in MNIST occurs for high R ( z 1 | z 2 ) rather than high R ( z 2 ) .
Revised: We note that the inception score (IS) behaves different in our MNIST models compared to SVHN (see Figure 5) in that optimal IS in MNIST occurs for high R ( z 1 | z 2 ) rather than high R ( z 2 ) .

Original: This indicates that semantically low-level properties (hand-writing style) of MNIST might have more variation than high level properties (the digit), whereas SVHN images show variation in additional high-level properties such as the background color
Revised: This indicates that semantically low-level properties (hand-writing style) of MNIST might have more variation than high level properties (the digit), whereas SVHN images show variation in additional high-level properties such as the background color

Original: [Figure]
Revised: [Figure]

Original: Figure
Revised: Figure

Original: A.2. R ESULTS FOR LVAE ON SVHN
Revised: A.2. R ESULTS FOR LVAE ON SVHN

Original: [Figure]
Revised: [Figure]

Original: A.2. R ESULTS FOR GENERALIZED TOP - DOWN HVAE S ON CIFAR- 
Revised: A.2. R ESULTS FOR GENERALIZED TOP - DOWN HVAE S ON CIFAR-

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: A. P ROOF OF T HE B OUND ON C LASSIFICATION A CCURACY

Revised: This section provides a proof of Eq.

Revised: 11 by reformulating the proof of Proposition 5 in the thesis by Meyen (2016) into the notation used in the present paper.

Revised: We stress that this section contains no original contribution and is provided only as a convenience to the reader, motivated by reviewer feedback.

Revised: All credits for this section belong to Meyen (2016).

Revised: We consider an (unknown) true data generative distribution p data ( y, x ) for data x with (unobserved) true labels y , and a hierarchical VAE with an inference model q ϕ ( { z ℓ } | x ) of the form of Eq. 4.

Revised: Focusing on a single layer ℓ of latents, we denote the joint probability over y , x , and z ℓ as

Original: [Figure]
Revised: [Equation]

Revised: We further consider a classifier p cls.

Revised: ( y | z ℓ ) that operates on z ℓ .

Revised: Denoting its top prediction as ˆ y := arg max y p cls.

Revised: ( y | z ℓ ) , the classification accuracy is α :=

Revised: E q [ δ y, ˆ y ] , where δ is the Kronecker delta.

Revised: Theorem 1.

Revised: The mutual information I q ( z ℓ ; y ) between the latent representation z ℓ and the true label y under the distribution q defined in Eq.

Revised: 13 is lower bounded as follows,

Revised: [Equation]

Revised: with

Revised: [Equation]

Revised: where H 2 ( α ) = − α log α − (1 − α ) log(1 − α ) is the entropy of a Bernoulli distribution with success probability α , H p data [ y ] ≤ log M is the marginal entropy of the true labels, and M denotes the number of classes.

Revised: Before we prove Theorem 1, we note that the function f is strictly monotonically increasing on the relevant interval [max y p data ( y ) , 1] .

Revised: Thus, f is invertible and we obtain the following corollary:

Revised: Corollary 1.

Revised: The classification accuracy α is upper bounded as in Eq.

Revised: 11 of the main text, i.e., α ≤ f − 1 ( I q ( z ℓ ; y )) .

Revised: Proof of Theorem 1.

Revised: We split the mutual information into two contributions,

Original: [Figure]
Revised: [Equation]

Revised: where, as clarified in the second equality,

Revised: H q [ y | z ℓ ] is the expectation over z ℓ of the conditional entropy of y given z ℓ , and q ( z ℓ ) and q ( y | z ℓ ) are marginals and conditionals of q (Eq. 13) as usual.

Revised: The marginal true label entropy H p data [ y ] in Eq.

Revised: 16 is fixed by the problem at hand.

Revised: In order to find a lower bound on I q ( z ℓ ; y ) for a given classification accuracy α , we therefore have to find an upper bound on H q

Revised: [ y | z ℓ ] subject to the constraint E q [ δ y, ˆ y ]

Revised: = α .

Revised: We do this by upper bounding the conditional entropy E y ∼ q

Revised: ( y | z ℓ ) [ − log q ( y | z ℓ )] of y given z ℓ for all z ℓ independently, and then taking the expectation over q ( z ℓ ) .

Revised: For fixed z ℓ , we first split off the contribution to E y ∼ q ( y | z ℓ ) [ − log q ( y | z ℓ )] from y

Revised: = ˆ y , where ˆ y = arg max y p cls.

Revised: ( y | z ℓ ) is the label that our classifier would predict for the fixed latent representation z ℓ ,

Revised: E y ∼

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: − (cid:88)

Revised: [Equation]

Revised: Here, the second term on the right-hand side resembles the entropy of a distribution over the remaining ( M − 1) labels ( y ̸

Revised: = ˆ y ), except that the probabilities sum to (1 − q ( y = ˆ y | z ℓ )) rather than one.

Revised: Thus, for any q ( y = ˆ y | z ℓ ) , this term is maximized if q ( y | z ℓ ) distributes the remaining probability mass (1 − q ( y = ˆ y | z ℓ )) uniformly over the remaining ( M − 1) labels, i.e.,

Original: [Figure]
Revised: E [Equation] Plugging Eq. 18 back into Eq.

Revised: 16, we obtain the bound

Revised: [Equation]

Revised: [Equation]

Revised: We arrive at the proposition (Eq. 14) by pulling the concave function H 2 out of the expectation using

Revised: Jensen’s inequality, and by then identifying E z ℓ ∼ q (

Revised: z

Revised: [Equation]

Revised: [ q ( y = ˆ y | z ℓ )]

Revised: =

Revised: q ( y = ˆ y )

Revised: =

Revised: α .