Original: T RADING I NFORMATION BETWEEN L ATENTS IN H IERARCHICAL V ARIATIONAL A UTOENCODERS
Revised: T RADING I NFORMATION BETWEEN L ATENTS IN H IERARCHICAL V ARIATIONAL A UTOENCODERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Variational Autoencoders (VAEs) were originally motivated (Kingma & Welling, 2013) as probabilistic generative models in which one performs approximate
Revised: Variational Autoencoders (VAEs) were originally motivated (Kingma & Welling, 2013) as probabilistic generative models in which one performs approximate

Original: Bayesian inference.
Revised: Bayesian inference.

Original: The proposal of β -VAEs (Higgins et al., 2016) breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing an objective function that allows practitioners to trade off between the information content (“bit rate”) of the latent representation and the distortion of reconstructed data (Alemi et al., 2018).
Revised: The proposal of β -VAEs (Higgins et al., 2016) breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing an objective function that allows practitioners to trade off between the information content (“bit rate”) of the latent representation and the distortion of reconstructed data (Alemi et al., 2018).

Original: In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.e., VAEs with more than one layer of latent variables.
Revised: In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.e., VAEs with more than one layer of latent variables.

Original: We identify a general class of inference models for which one can split the rate into contributions from each layer, which can then be tuned independently.
Revised: We identify a general class of inference models for which one can split the rate into contributions from each layer, which can then be tuned independently.

Original: We derive theoretical bounds on the performance of downstream tasks as functions of the individual layers’ rates and verify our theoretical findings in large-scale experiments.
Revised: We derive theoretical bounds on the performance of downstream tasks as functions of the individual layers’ rates and verify our theoretical findings in large-scale experiments.

Original: Our results provide guidance for practitioners on which region in rate-space to target for a given application.
Revised: Our results provide guidance for practitioners on which region in rate-space to target for a given application.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) are a class of deep generative models that are used in a diverse set of application domains including density modeling (Takahashi et al., 2018), clustering (Jiang et al., 2016), nonlinear dimensionality reduction of scientific measurements (Laloy et al., 2017), data compression (Ball ´ e et al., 2016), anomaly detection (Xu et al., 2018), and image generation (Razavi et al., 2019).
Revised: Variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) are a class of deep generative models that are used in a diverse set of application domains including density modeling (Takahashi et al., 2018), clustering (Jiang et al., 2016), nonlinear dimensionality reduction of scientific measurements (Laloy et al., 2017), data compression (Ball ´ e et al., 2016), anomaly detection (Xu et al., 2018), and image generation (Razavi et al., 2019).

Original: One of the reasons why the use cases of VAEs (more precisely, β -VAEs (Higgins et al., 2016)) span such a diverse set of application domains is that they can be tuned to a specific task without changing the network architecture, in a way that is well understood from information theory (Alemi et al., 2018).
Revised: One of the reasons why the use cases of VAEs (more precisely, β -VAEs (Higgins et al., 2016)) span such a diverse set of application domains is that they can be tuned to a specific task without changing the network architecture, in a way that is well understood from information theory (Alemi et al., 2018).

Original: The original proposal of VAEs (Kingma & Welling, 2013) motivates them from the perspective of generative probabilistic modeling and approximate Bayesian inference.
Revised: The original proposal of VAEs (Kingma & Welling, 2013) motivates them from the perspective of generative probabilistic modeling and approximate Bayesian inference.

Original: However, the generalization to β -VAEs breaks this interpretation as they are no longer trained by maximizing a lower bound on the marginal data likelihood.
Revised: However, the generalization to β -VAEs breaks this interpretation as they are no longer trained by maximizing a lower bound on the marginal data likelihood.

Original: These models are better described as neural networks that are trained to learn the identity function, i.e., to make their output resemble the input as closely as possible.
Revised: These models are better described as neural networks that are trained to learn the identity function, i.e., to make their output resemble the input as closely as possible.

Original: This task is made nontrivial by introducing a so-called (variational) information bottleneck (Alemi et al., 2016; Tishby & Zaslavsky, 2015) at one or more layers, which restricts the information content that passes through these layers.
Revised: This task is made nontrivial by introducing a so-called (variational) information bottleneck (Alemi et al., 2016; Tishby & Zaslavsky, 2015) at one or more layers, which restricts the information content that passes through these layers.

Original: The network activations at the information bottleneck are called latent representations (or simply “latents”), and they split the network into an encoder part (from input to latents) and a decoder part (from latents to output).
Revised: The network activations at the information bottleneck are called latent representations (or simply “latents”), and they split the network into an encoder part (from input to latents) and a decoder part (from latents to output).

Original: This separation of the model into an encoder and a decoder allows us to categorize the wide variety of applications of VAEs into three domains:
Revised: This separation of the model into an encoder and a decoder allows us to categorize the wide variety of applications of VAEs into three domains:

Original: data reconstruction tasks , i.e., applications that involve both the encoder and the decoder ; these include various nonlinear inter- and extrapolations (e
Revised: data reconstruction tasks , i.e., applications that involve both the encoder and the decoder ; these include various nonlinear inter- and extrapolations (e

Original: g., image upscaling, denoising, or inpainting), and VAE-based methods for lossy data compression; 2.
Revised: g., image upscaling, denoising, or inpainting), and VAE-based methods for lossy data compression; 2.

Original: representation learning tasks , i.e., applications that involve only the encoder ; they serve a downstream task that operates on the (typically lower dimensional) latent representation, e.g., classification, regression, visualization, clustering, or anomaly detection; and 3.
Revised: representation learning tasks , i.e., applications that involve only the encoder ; they serve a downstream task that operates on the (typically lower dimensional) latent representation, e.g., classification, regression, visualization, clustering, or anomaly detection; and 3.

Original: generative modeling tasks , i.e., applications that involve only the decoder are less common but include generating new samples that resemble training data.
Revised: generative modeling tasks , i.e., applications that involve only the decoder are less common but include generating new samples that resemble training data.

Original: The information bottleneck incentivizes the VAE to encode information into the latents efficiently by removing any redundancies from the input.
Revised: The information bottleneck incentivizes the VAE to encode information into the latents efficiently by removing any redundancies from the input.

Original: How agressively this is done can be controlled by tuning the strength β of the information bottleneck (Alemi et al., 2018).
Revised: How agressively this is done can be controlled by tuning the strength β of the information bottleneck (Alemi et al., 2018).

Original: Unfortunately, information theory distinguishes relevant from redundant information only in a quantitative way that is agnostic to the qualitative features that each piece of information represents about some data point.
Revised: Unfortunately, information theory distinguishes relevant from redundant information only in a quantitative way that is agnostic to the qualitative features that each piece of information represents about some data point.

Original: In practice, many VAE-architectures (Deng et al., 2017; Yingzhen & Mandt, 2018; Ball ´ e et al., 2018) try to separate qualitatively different features into different parts of the latent representation by making the model architecture reflect some prior assumptions about the semantic structure of the data.
Revised: In practice, many VAE-architectures (Deng et al., 2017; Yingzhen & Mandt, 2018; Ball´e et al., 2018) try to separate qualitatively different features into different parts of the latent representation by making the model architecture reflect some prior assumptions about the semantic structure of the data.

Original: This allows downstream applications from the three domains discussed above to more precisely target specific qualitative aspects of the data by using or manipulating only the corresponding part of the latent representation.
Revised: This allows downstream applications from the three domains discussed above to more precisely target specific qualitative aspects of the data by using or manipulating only the corresponding part of the latent representation.

Original: However, in this approach, the degree of detail to which each qualitative aspect is encoded in the latents can be controlled at most indirectly by tuning network layer sizes.
Revised: However, in this approach, the degree of detail to which each qualitative aspect is encoded in the latents can be controlled at most indirectly by tuning network layer sizes.

Original: In this paper we argue both theoretically and empirically that the three different application domains of VAEs identified above require different trade-offs in the amount of information that is encoded in each part of the latent representation.
Revised: In this paper we argue both theoretically and empirically that the three different application domains of VAEs identified above require different trade-offs in the amount of information that is encoded in each part of the latent representation.

Original: We propose a method to independently control the information content (or “rate”) of each layer of latent representations, generalizing the rate/distortion theory of β -VAEs (Alemi et al., 2018) for VAEs with more than one layer of latents (“hierarchical VAEs” or
Revised: We propose a method to independently control the information content (or “rate”) of each layer of latent representations, generalizing the rate/distortion theory of β -VAEs (Alemi et al., 2018) for VAEs with more than one layer of latents (“hierarchical VAEs” or

Original: HVAEs for short).
Revised: HVAEs for short).

Original: We identify the most general model architecture that is compatible with our proposal and analyze how both theoretical performance bounds and empirically measured performances in each of the above three application domains depend on how rate is distributed across layers.
Revised: We identify the most general model architecture that is compatible with our proposal and analyze how both theoretical performance bounds and empirically measured performances in each of the above three application domains depend on how rate is distributed across layers.

Original: Our approach is summarized in Figure 1.
Revised: Our approach is summarized in Figure 1.

Original: The 3d-plot shows empirically measured performance metrics (discussed in detail in Section 5.2) for the three application domains identified above.
Revised: The 3d-plot shows empirically measured performance metrics (discussed in detail in Section 5.2) for the three application domains identified above.

Original: Each point on the colored surface corresponds to different layer-wise rates in an HVAE with two layers of latents.
Revised: Each point on the colored surface corresponds to different layer-wise rates in an HVAE with two layers of latents.

Original: Crucially, the rates that lead to optimal performance are different for each of the three application domains (see markers △ , (cid:68) , and ⋄ in Figure 1), and none of these three optimal models coincide with a conventional β -VAE (dashed red lines in right panels).
Revised: Crucially, the rates that lead to optimal performance are different for each of the three application domains (see markers △ , (cid:68) , and ⋄ in Figure 1), and none of these three optimal models coincide with a conventional β -VAE (dashed red lines in right panels).

Original: Thus, being able to control each layer’s individual rate allows practitioners to train VAEs that target a specific application.
Revised: Thus, being able to control each layer’s individual rate allows practitioners to train VAEs that target a specific application.

Original: The paper is structured as follows.
Revised: The paper is structured as follows.

Original: Section 2 summarizes related work.
Revised: Section 2 summarizes related work.

Original: Section 3 introduces the proposed information-trading method.
Revised: Section 3 introduces the proposed information-trading method.

Original: We then analyze how controlling individual layers’ rates can be used to tune HVAEs for specific tasks, i.e., how performance in each of the three application domains identified above depends on the allocation of rates across layers.
Revised: We then analyze how controlling individual layers’ rates can be used to tune HVAEs for specific tasks, i.e., how performance in each of the three application domains identified above depends on the allocation of rates across layers.

Original: This analysis is done theoretically in Section 4 and empirically in Section 5.
Revised: This analysis is done theoretically in Section 4 and empirically in Section 5.

Original: Section 6 provides concluding remarks.
Revised: Section 6 provides concluding remarks.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We group related work into work on model architectures for hierarchical VAEs, and on β -VAEs.
Revised: We group related work into work on model architectures for hierarchical VAEs, and on β -VAEs.

Original: Model Design for Hierarchical VAEs.
Revised: Model Design for Hierarchical VAEs.

Original: The original VAE design (Kingma & Welling, 2013;
Revised: The original VAE design (Kingma & Welling, 2013;

Original: Rezende et al., 2014) has a single layer of latent variables, but recent works (Vahdat & Kautz, 2020;
Revised: Rezende et al., 2014) has a single layer of latent variables, but recent works (Vahdat & Kautz, 2020;

Original: Child, 2020), found that increasing the number of stochastic layers in hierarchical VAEs (HVAEs) improves performance.
Revised: Child, 2020), found that increasing the number of stochastic layers in hierarchical VAEs (HVAEs) improves performance.

Original: HVAEs have various designs for their inference models.
Revised: HVAEs have various designs for their inference models.

Original: Sønderby et al.
Revised: Sønderby et al.

Original: (2016) introduced Ladder VAE (LVAE) with a top-down inference path rather than the naive bottomup inference (see Section 3),
Revised: (2016) introduced Ladder VAE (LVAE) with a top-down inference path rather than the naive bottomup inference (see Section 3),

Original: whereas the Bidirectional-Inference VAE (BIVA) (Maaløe et al., 2019) uses a combination of top-down and bottom-up.
Revised: whereas the Bidirectional-Inference VAE (BIVA) (Maaløe et al., 2019) uses a combination of top-down and bottom-up.

Original: Our proposed framework applies to a large class of inference models (see Section 3) that includes the popular LVAE (Sønderby et al., 2016).
Revised: Our proposed framework applies to a large class of inference models (see Section 3) that includes the popular LVAE (Sønderby et al., 2016).

Original: β -VAEs
Revised: β -VAEs

Original: And Their Information-Theoretical Interpretations.
Revised: And Their Information-Theoretical Interpretations.

Original: Higgins et al.
Revised: Higgins et al.

Original: (2016) introduced an extra hyperparameter β in the objective of VAEs that tunes the strength of the information bottleneck, and they observed that large β leads to a disentangled latent representation.
Revised: (2016) introduced an extra hyperparameter β in the objective of VAEs that tunes the strength of the information bottleneck, and they observed that large β leads to a disentangled latent representation.

Original: An informationtheoretical interpretation of β -VAEs was provided in (Alemi et al., 2018) by applying the concept of a (variational) bottleneck (Tishby & Zaslavsky, 2015;
Revised: An informationtheoretical interpretation of β -VAEs was provided in (Alemi et al., 2018) by applying the concept of a (variational) bottleneck (Tishby & Zaslavsky, 2015;

Original: Alemi et al., 2016) to autoencoders.
Revised: Alemi et al., 2016) to autoencoders.

Original: Due to this information-theoretical interetation, β -VAEs are popular models for data compression (Ball ´ e et al., 2016; Minnen et al., 2018;
Revised: Due to this information-theoretical interetation, β -VAEs are popular models for data compression (Ball ´ e et al., 2016; Minnen et al., 2018;

Original: Yang et al., 2020), where tuning β allows trading off between the bit rate of compressed data and data distortion.
Revised: Yang et al., 2020), where tuning β allows trading off between the bit rate of compressed data and data distortion.

Original: In the present work, we generalize β -VAEs when applied to HVAEs, and we introduce a framework for tuning the rate of each latent layer individually.
Revised: In the present work, we generalize β -VAEs when applied to HVAEs, and we introduce a framework for tuning the rate of each latent layer individually.

Original:  A H IERARCHICAL I NFORMATION T RADING F RAMEWORK
Revised:  A H IERARCHICAL I NFORMATION T RADING F RAMEWORK

Original: We propose a refinement of the rate/distortion theory of β -VAEs (Alemi et al., 2018) that admits controlling individual layers’ rates in VAEs with more than one layers of latents (hierarchical VAEs).
Revised: We propose a refinement of the rate/distortion theory of β -VAEs (Alemi et al., 2018) that admits controlling individual layers’ rates in VAEs with more than one layers of latents (hierarchical VAEs).

Original: 3. C ONVENTIONAL β -VAE W ITH H IERARCHICAL L ATENT R EPRESENTATIONS
Revised: 3. C ONVENTIONAL β -VAE W ITH H IERARCHICAL L ATENT R EPRESENTATIONS

Original: We consider a hierarchical VAE (HVAE) for data x with L layers of latent representations { z ℓ } Lℓ =1 .
Revised: We consider a hierarchical VAE (HVAE) for data x with L layers of latent representations { z ℓ } Lℓ =1 .

Original: Figure 2, discussed further in Section 3.2 below, illustrates various model architectures for the example of L = 2 .
Revised: Figure 2, discussed further in Section 3.2 below, illustrates various model architectures for the example of L = 2 .

Original: Solid arrows depict the generative model p θ ( { z ℓ } , x ) , where θ are model parameters (neural network weights). We assume that the generative (or “decoder”) model is a Markov chain,
Revised: Solid arrows depict the generative model p θ ( { z ℓ } , x ) , where θ are model parameters (neural network weights). We assume that the implementation factorizes p θ ( { z ℓ } , x ) as follows,

Original: [Equation]
Revised: [Equation]

Original: where the  numbering of latent layers from L down to 1 follows the common convention in the literature (Sønderby et al., 2016; Gulrajani et al., 2016; Child, 2020).
Revised: where the notation z ≥ n for any n is short for the collection of latents { z ℓ } Lℓ = n (thus, z ≥ 1 and { z ℓ } are synonymous), and the numbering of latents from L down to 1 follows the common convention in the literature (Sønderby et al., 2016; Gulrajani et al., 2016; Child, 2020).

Original: The loss function of a normal β -VAE (Higgins et al., 2016) with this genericmodel architecture would be
Revised: The loss function of a normal β -VAE (Higgins et al., 2016) with this generic architecture would be

Original: [Equation]
Revised: [Equation]

Original: [Equation] Here, q ϕ ( { z ℓ } | x ) is the inference (or “encoder”) model with parameteres ϕ , X train is the training set, D KL [ · || · ] denotes Kullback-Leibler divergence, and the Lagrange parameter β > 0 trades off between a (total) rate R and a distortion D (Alemi et al., 2018).
Revised: Here, q ϕ ( { z ℓ } | x ) is the inference (or “encoder”) model with parameteres ϕ , X train is the training set, D KL [ · || · ] denotes Kullback-Leibler divergence, and the Lagrange parameter β > 0 trades off between a (total) rate R and a distortion D (Alemi et al., 2018).

Original: Setting β
Revised: Setting β

Original: = 1 turns Eq.
Revised: = 1 turns Eq.

Original: 2 into the negative ELBO objective of a regular VAE (Kingma & Welling, 2013).
Revised: 2 into the negative ELBO objective of a regular VAE (Kingma & Welling, 2013).

Original: The rate R obtains its name as it measures the (total) information content that q ϕ encodes into the latent representations { z ℓ } , which would manifest itself in the expected bit rate when one optimally encodes a random draw { z ℓ } ∼ q ϕ ( { z ℓ } | x ) using p θ ( { z ℓ } ) as an entropy model (Agustsson & Theis, 2020; Bennett et al., 2002).
Revised: The rate R obtains its name as it measures the (total) information content that q ϕ encodes into the latent representations { z ℓ } , which would manifest itself in the expected bit rate when one optimally encodes a random draw { z ℓ } ∼ q ϕ ( { z ℓ } | x ) using p θ ( { z ℓ } ) as an entropy model (Agustsson & Theis, 2020; Bennett et al., 2002).

Original: An important observation pointed out in (Alemi et al., 2016) is that, regardless how rate R is traded off against distortion D by tuning β , their sum R + D is—in expectation under any data distribution p data ( x ) —always lower bounded by the entropy H [ p data ( x )] :
Revised: An important observation pointed out in (Alemi et al., 2016) is that, regardless how rate R is traded off against distortion D by tuning β , their sum R + D is—in expectation under any data distribution p data ( x ) —always lower bounded by the entropy H [ p data ( x )] := E p data

Original: [Equation]
Revised: [Equation]

Original: x ) [ 
Revised: x ) [ − log p data

Original: [Equation]
Revised: [Equation]

Original: data ( x )] ,
Revised: x )] ,

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Limitations.
Revised: Limitations.

Original: The rate R in Eq.
Revised: The rate R in Eq.

Original: 2 is a property of the collection { z ℓ } of all latents, which can limit its interpretability for some inference models.
Revised: 2 is a property of the collection { z ℓ } of all latents, which can limit its interpretability for some inference models.

Original: For example, the common convention of enumerating layers z ℓ from ℓ =
Revised: For example, the common convention of enumerating layers z ℓ from ℓ =

Original: L down to 1 in Eq.
Revised: L down to 1 in Eq.

Original: 1 is reminiscent of a naive architecture for the inference model that factorizes in reverse order compared to Eq.
Revised: 1 is reminiscent of a naive architecture for the inference model that factorizes in reverse order compared to Eq.

Original: 1 (“bottom up”, see dashed arrows in Figure 2(a)), i.e., q ϕ ( { z ℓ } | x ) =
Revised: 1 (“bottom up”, see dashed arrows in Figure 2(a)), i.e., q ϕ ( { z ℓ } | x ) =

Original: q ϕ ( z 1 | x ) q ϕ ( z 2 | z 1 ) · · · q ϕ ( z L | z
Revised: q ϕ ( z 1 | x ) q ϕ ( z 2 | z 1 ) · · · q ϕ ( z L | z

Original: L − 1 ) .
Revised: L − 1 ) .

Original: Using a HVAE with such a “bottom-up” inference model to reconstruct some given data point x would map x to z 1 using q ϕ ( z 1 |
Revised: Using a HVAE with such a “bottom-up” inference model to reconstruct some given data point x would map x to z 1 using q ϕ ( z 1 |

Original: x
Revised: x

Original: and then map z 1 back to the data space using p θ ( x | z 1 ) , thus ignoring all latents z ℓ with ℓ > 1 .
Revised: and then map z 1 back to the data space using p θ ( x | z 1 ) , thus ignoring all latents z ℓ with ℓ > 1 .

Original: Yet, the rate term in Eq.
Revised: Yet, the rate term in Eq.

Original: 2 still depends on all latents, including the ones not needed to reconstruct any data (practical VAE-based compression methods using bits-back coding (Frey & Hinton, 1997) would, however, indeed use z ℓ with ℓ > 1 as auxiliary variables for computational efficiency).
Revised: 2 still depends on all latents, including the ones not needed to reconstruct any data (practical VAE-based compression methods using bits-back coding (Frey & Hinton, 1997) would, however, indeed use z ℓ with ℓ > 1 as auxiliary variables for computational efficiency).

Original: 3. T RADING I NFORMATION B ETWEEN L ATENTS
Revised: 3. T RADING I NFORMATION B ETWEEN L ATENTS

Original: Many HVAEs used in the literature allow us to resolve the limitations identified in Section 3.1.
Revised: Many HVAEs used in the literature allow us to resolve the limitations identified in Section 3.1.

Original: For example, the popular LVAE architecture (Sønderby et al., 2016), (Figure 2(b)), uses an inference model (dashed arrows) that traverses the latents { z ℓ } in the same order as the generative model (solid arrows).
Revised: For example, the popular LVAE architecture (Sønderby et al., 2016), (Figure 2(b)), uses an inference model (dashed arrows) that traverses the latents { z ℓ } in the same order as the generative model (solid arrows).

Original: We consider the following generalization of this architecture (see Figure 2(c)),
Revised: We consider the following generalization of this architecture (see Figure 2(c)),

Original: [Equation]
Revised: [Equation]

Original: This architecture is motivated by factorizing in the same form as the true posterior p θ ( { z ℓ } | x ) : = p θ ( { z ℓ } , x ) /p θ ( x ) (because the generative model p θ ( { z ℓ } , x ) in Eq. 1 is a Markov chain).
Revised: Formally, Eq. 4 is just the product rule of probability theory and therefore holds for arbitrary inference models q ϕ ( { z ℓ } | x ) .

Original: Eq.includes LVAE as a special case that ties some parameters between inference and generative model.
Revised: More practically, however, we make the assumption that the actual implementation of q ϕ ( { z ℓ } | x ) follows the structure in Eq. 4.

Original: Using a trained HVAE with the inference model architecture from Eq. 4 to map a  data point x to its reconstruction ˆ x now involves all latents  (either drawing a sample or taking the mode at each step):
Revised: This means that, using the trained model, the most efficient way to map a given data point x to its reconstruction ˆ x now involves all latents z ℓ (either drawing a sample or taking the mode at each step):

Original: [Equation]
Revised: [Equation]

Original: Layer-wise Rates.
Revised: Layer-wise Rates.

Original: We can interpret Eq.
Revised: We can interpret Eq.

Original: 5 in that it first maps x to a “crude” representation z L , which gets iteratively refined to z 1 , which is then used tofind a reconstruction ˆ x .
Revised: 5 in that it first maps x to a “crude” representation z L , which gets iteratively refined to z 1 , and finally to a reconstruction ˆ x .

Original: Note that each factor q ϕ ( z ℓ− 1 | z  ℓ  , x ) of the inference model  is conditioned not only on the previous layer z  ℓ  but also on the original data x, which allows the inference model to target each refinement such that the reconstruction ˆ x becomes close to x .
Revised: Note that each factor q ϕ ( z ℓ | z ≥ ℓ +1 , x ) of the inference model in Eq. 4 is conditioned not only on the previous layers z ≥ ℓ +1 but also on the original data x .

Original: More formally, the iterative refinement towards the reconstruction ˆ x is expressed by the data processing inequality (MacKay, 2003): the part of Eq.5 from z L to ˆ x forms a Markov chain that is conditioned on x , and therefore the mutual information I ( z ℓ ; ˆ x | x ) between a latent z ℓ and the reconstruction ˆ x given the original data point x is nondecreasing along the chain, i.e., I ( z L ; ˆ x | x ) ≤ I ( z L − 1 ;
Revised: This allows the inference model to target each refinement step in Eq.

Original:  ˆ x | x ) ≤ . . . ≤ I ( z 1 ; ˆ x | x) .
Revised: 5 such that the reconstruction ˆ x becomes close to x .

Original: Further, since the inference architecture in Eq.
Revised: More formally, we chose the inference architecture in Eq.

Original: 4  factorizes over { z ℓ } in the same order as the generative model (Eq. 1), we can split the total rate Rin an autoregressive way into a sum of layer-wise rates, [Equation] z 2 ) .
Revised: 4 such that it factorizes over { z ℓ } in the same order as the generative model (Eq. 1). This allows us to split the total rate R into a sum of layer-wise rates as follows,

Original: [Equation]
Revised: [Equation]

Original: Here,
Revised: Here,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: quantify the information content of the highest-order latent representation z L and the (expected) increase in information content in each refinement step z ℓ  → z ℓ− 1 in Eq. 5, respectively.
Revised: quantify the information content of the highest-order latent representation z L and the (expected) increase in information content in each refinement step z ℓ +1 → z ℓ in Eq. 5, respectively.

Original: Controlling Each Layer’s Rate.
Revised: Controlling Each Layer’s Rate.

Original: Using Eqs.
Revised: Using Eqs.

Original: 6-7, we generalize the rate/distortion trade-off from
Revised: 6-7, we generalize the rate/distortion trade-off from

Original: Section 3.1 by introducing L individual Lagrange multipliers β L , β L − 1| L , ..., β 1| 2 , collectively denoted as boldface β .
Revised: Section 3.1 by introducing L individual Lagrange multipliers β L , β L − 1 , ..., β 1 , collectively denoted as boldface β .

Original: This leads to a new loss function that generalizes Eq. 2 as follows,
Revised: This leads to a new loss function that generalizes Eq. 2 as follows,

Original: [Equation]
Revised: [Equation]

Original: Setting all β ’s to the same value recovers the conventional β -VAE (Eq. 2), which trades off distortion against total information content in { z ℓ } .
Revised: Setting all β s to the same value recovers the conventional β -VAE (Eq. 2), which trades off distortion against total information content in { z ℓ } .

Original: Tuning each β -hyperparameter individuallygives us more control and allows trading-off information content across latents.
Revised: Tuning each β -hyperparameter individually allows trading off information content across latents.

Revised: (In a very deep HVAE (i.e., large L ) it may be more practical to group layers into only few bins and to use the same β -value for all layers within a bin.)

Original: We analyze how this can be useful depending on application domain theoretically in Section 4 and empirically in Section 5.
Revised: We analyze how to tune β s for various applications theoretically in Section 4 and empirically in Section 5.

Original:  I NFORMATION -T HEORETICAL P ERFORMANCE B OUNDS FOR HVAE S
Revised:  I NFORMATION -T HEORETICAL P ERFORMANCE B OUNDS FOR HVAE S

Original: In this section, we analyze theoretically how various performance metrics for HVAEs are restricted by the individual layers’ rates R ( z L ) and R ( z ℓ− 1 | z  ℓ  ) identified in Eq.
Revised: In this section, we analyze theoretically how various performance metrics for HVAEs are restricted by the individual layers’ rates R ( z L ) and R ( z ℓ | z ≥ ℓ +1 ) identified in Eq.

Original: 7 for a HVAE with “topdown” inference model.
Revised: 7 for a HVAE with “topdown” inference model.

Original: Our analysis motivates the use of the information-trading loss function in
Revised: Our analysis motivates the use of the information-trading loss function in

Original: Eq. 8 for training HVAEs, following the argument from the introduction that VAEs are commonly used for a vast variety of tasks.
Revised: Eq. 8 for training HVAEs, following the argument from the introduction that VAEs are commonly used for a vast variety of tasks.

Original: As we show, different tasks require different trade-offs that can be targeted by tuning the Lagrange multipliers β in Eq.
Revised: As we show, different tasks require different trade-offs that can be targeted by tuning the Lagrange multipliers β in Eq.

Original: We group tasks into the application domains of (i) data reconstruction and manipulation, (ii) representation learning, and (iii) data generation.
Revised: We group tasks into the application domains of (i) data reconstruction and manipulation, (ii) representation learning, and (iii) data generation.

Original: Data Reconstruction and Manipulation.
Revised: Data Reconstruction and Manipulation.

Original: The most obvious class of application domains of VAEs includes tasks that combine encoder and decoder to map some data point x to representations { z ℓ } and then back to the data space.
Revised: The most obvious class of application domains of VAEs includes tasks that combine encoder and decoder to map some data point x to representations { z ℓ } and then back to the data space.

Original: The simplest performance metric for such data reconstruction tasks is the expected distortion E p data ( x ) [ D ] , which we can bound by combining Eq.
Revised: The simplest performance metric for such data reconstruction tasks is the expected distortion E p data ( x ) [ D ] , which we can bound by combining Eq.

Original: 3 with Eqs.
Revised: 3 with Eqs.

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 9 would suggest that higher rates (i.e., lower β ’s) are always better for data reconstruction tasks.
Revised: 9 would suggest that higher rates (i.e., lower β ’s) are always better for data reconstruction tasks.

Original: However, in many practical tasks (e.g., image upscaling, denoising, or inpainting) the goal is not solely to reconstruct the original data but also to manipulate the latent representations { z ℓ } in a meaningful way.
Revised: However, in many practical tasks (e.g., image upscaling, denoising, or inpainting) the goal is not solely to reconstruct the original data but also to manipulate the latent representations { z ℓ } in a meaningful way.

Original: Here, lower rates can lead to more semantically meaningful representation spaces (see, e.g., Section 5.6 below).
Revised: Here, lower rates can lead to more semantically meaningful representation spaces (see, e.g., Section 5.6 below).

Original: Controlling how rate is distributed across layers via Eq.
Revised: Controlling how rate is distributed across layers via Eq.

Original: 8 may allow practitioners to have a semantically meaningful high-level representation z L with low rate R ( z L ) while still retaining a high total rate R , thus allowing for low distortion D without violating Eq. 9.
Revised: 8 may allow practitioners to have a semantically meaningful high-level representation z L with low rate R ( z L ) while still retaining a high total rate R , thus allowing for low distortion D without violating Eq. 9.

Original: Representation Learning.
Revised: Representation Learning.

Original: In many practical applications, VAEs are used as nonlinear dimensionality reduction methods to prepare some complicated high-dimensional data x for downstream tasks such as classification, regression, visualization, clustering, or anomaly detection.
Revised: In many practical applications, VAEs are used as nonlinear dimensionality reduction methods to prepare some complicated high-dimensional data x for downstream tasks such as classification, regression, visualization, clustering, or anomaly detection.

Original: We consider a classifier p cls.
Revised: We consider a classifier p cls.

Original: ( y | z ℓ ) operating on the latents z ℓ at some level ℓ .
Revised: ( y | z ℓ ) operating on the latents z ℓ at some level ℓ .

Original: We assume that the (unknown) true data generative process p data ( y, x ) =
Revised: We assume that the (unknown) true data generative process p data ( y, x ) =

Original: p data ( y ) p data ( x | y ) generates data x conditioned on some true label y , thus defining a Markov chain y p data −−→ x q ϕ −→ z ℓ p cls.
Revised: p data ( y ) p data ( x | y ) generates data x conditioned on some true label y , thus defining a Markov chain y p data [Equation] x q ϕ [Equation]

Original: −−→ ˆ y where ˆ y := arg max y p cls.
Revised: z ℓ p cls. [Equation] ˆ y where ˆ y := arg max y p cls.

Original: ( y | z ℓ ) .
Revised: ( y | z ℓ ) .

Original: Classification accuracy is bounded (Meyen, 2016) by a function of the mutual information I q ( y ; z ℓ ) , 
Revised: Classification accuracy is bounded (Meyen, 2016) by a function of the mutual information I q ( y ; z ℓ ) , I q ( y

Original: I q ( y ; z ℓ ) ≤ I q( x ; z ℓ ) ≡ E p data ( x )
Revised: [Equation] z ℓ ) [Equation] I q

Original: [Equation]
Revised: [Equation]

Original: Here, q ϕ ( z ℓ ) :=
Revised: z ℓ [Equation] ≤ [Equation]

Revised: (cid:124) (cid:123)(cid:122) (cid:125) =: R ( z

Revised: [Equation]

Original:  E p data ( x ) [ q ϕ ( z ℓ | x )] , the notations { z ≥ ℓ } and { z >ℓ } denote all latent representations from z Lup to z ℓ (including z ℓ for the former), the first inequality comes from the data processing inequality (MacKay, 2003), and the other two inequalities result from discarding the (nonnegative)
Revised: [Equation] (cid:3) . Here, q ϕ ( z ℓ ) := E p data ( x ) [ q ϕ ( z ℓ | x )] and we identify R ( z ≥ ℓ ) as the rate accumulated in all layers from z L to z ℓ .

Original:  KL-terms.
Revised: The first inequality in Eq. 10 comes from the data processing inequality (MacKay, , and the other two inequalities result from discarding the (nonnegative) KL-terms.

Original: The classification accuracy is thus bounded by (Meyen, 2016)
Revised: The classification accuracy is thus bounded by (Meyen, 2016)

Original: (see also  Appendix A.3)
Revised: (see also proof in Appendix B)

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f − 1 is the inverse of the monotonic function f ( α )
Revised: where f − 1 is the inverse of the monotonic function f ( α )

Original: =
Revised: =

Original: H [ p data ( y )]
Revised: H [ p data ( y )]

Original: + α log α +(1 − α ) log 1 − α M −with M being the number of classes and H [ p data ( y )]
Revised: + α log α +(1 − α ) log 1 − α M −with M being the number of classes and H [ p data ( y )]

Original: ≤ log M the marginal label entropy.
Revised: ≤ log M the marginal label entropy.

Original: Eq.suggests that the accuracy of an optimal classifier on z ℓ would increase as the rate R ( z  ℓ ) accumulated from z L to z ℓ grows (i.e., as β ≥ ℓ → 0 ), and that the rate added in downstream layers{ z <ℓ} would be irrelevant.
Revised: Eq.suggests that the accuracy of an optimal classifier on z ℓ would increase as the rate R ( z ≥ ℓ ) accumulated from z L to z ℓ grows (i.e., as β ≥ ℓ → 0 ), and that the rate added in downstream layers z <ℓ would be irrelevant.

Original: Practical classifiers, however, have a limited expressiveness, which a very high rate R ( z  ℓ ) might exceed by encoding too many details into z ℓ that are not necessary for classification.
Revised: Practical classifiers, however, have a limited expressiveness, which a very high rate R ( z ≥ ℓ ) might exceed by encoding too many details into z ℓ that are not necessary for classification.

Original: We observe in Section 5.6 that, in such cases, increasing the rates of downstream layers{ z <ℓ} improves classification accuracy as it allows keeping z ℓ simpler by deferring details to{ z <ℓ} .
Revised: We observe in Section 5.6 that, in such cases, increasing the rates of downstream layers z <ℓ improves classification accuracy as it allows keeping z ℓ simpler by deferring details to z <ℓ .

Original: Data Generation.
Revised: Data Generation.

Original: The original proposal of VAEs (Kingma & Welling, 2013) motivated them from a generative modeling perspective using that, for β = 1 , the negative of the loss function in Eq.
Revised: The original proposal of VAEs (Kingma & Welling, 2013) motivated them from a generative modeling perspective using that, for β = 1 , the negative of the loss function in Eq.

Original: 2 is a lower bound on the log marginal data likelihood.
Revised: 2 is a lower bound on the log marginal data likelihood.

Original: This suggests setting all β -hyperparameters in
Revised: This suggests setting all β -hyperparameters in

Original: Eq.
Revised: Eq.

Original: 8 to values close to 1 if a HVAE is used primarily for its generative model p θ .
Revised: 8 to values close to 1 if a HVAE is used primarily for its generative model p θ .

Original: In summary, our theoretical analysis suggests that optimally tuned layer-wise rates depend on whether a HVAE is used for data reconstruction, representation learning, or data generation.
Revised: In summary, our theoretical analysis suggests that optimally tuned layer-wise rates depend on whether a HVAE is used for data reconstruction, representation learning, or data generation.

Original: The next section tests our theoretical predictions empirically for the same three application domains.
Revised: The next section tests our theoretical predictions empirically for the same three application domains.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To demonstrate the features of our hierarchical information trading framework, we run large-scale grid searches over a two-dimensional rate space using two different implementations of HVAEs and three different data sets.
Revised: To demonstrate the features of our hierarchical information trading framework, we run large-scale grid searches over a two-dimensional rate space using two different implementations of HVAEs and three different data sets.

Original: Although the proposed framework is applicable for HVAEs with L ≥ 2 , we only use HVAEs with L = 2 in our experiments for simplicity and visualization purpose.
Revised: Although the proposed framework is applicable for HVAEs with L ≥ 2 , we only use HVAEs with L = 2 in our experiments for simplicity and visualization purpose.

Original: 5. E XPERIMENTAL SETUP
Revised: 5. E XPERIMENTAL SETUP

Original: Data sets.
Revised: Data sets.

Original: We used the SVHN (Netzer et al., 2011) and CIFAR-10 (Krizhevsky et al., 2009) data sets (both 32 × 32 pixel color images), and MNIST (LeCun et al., 1998) ( 28 × 28 binary pixel images).
Revised: We used the SVHN (Netzer et al., 2011) and CIFAR-10 (Krizhevsky et al., 2009) data sets (both 32 × 32 pixel color images), and MNIST (LeCun et al., 1998) ( 28 × 28 binary pixel images).

Original: SVHN consists of photographed house numbers from 0 to 9, which are geometrically simpler than the 10 classes of objects from CIFAR-10 but more complex than MNIST digits.
Revised: SVHN consists of photographed house numbers from 0 to 9, which are geometrically simpler than the 10 classes of objects from CIFAR-10 but more complex than MNIST digits.

Original: Most results shown in the main paper use SVHN; comprehensive results for CIFAR-10 and MNIST are shown in Appendix A.2 and tell a similar story except where explicitly discussed.
Revised: Most results shown in the main paper use SVHN; comprehensive results for CIFAR-10 and MNIST are shown in Appendix A.2 and tell a similar story except where explicitly discussed.

Original: Model Architectures.
Revised: Model Architectures.

Original: For the generative model (Eq. 1), we assume a (fixed) standard Gaussian prior p ( z 2 ) =
Revised: For the generative model (Eq. 1), we assume a (fixed) standard Gaussian prior p ( z 2 ) =

Original: N ( 0 , I ) , and we use diagonal Gaussian models for p θ ( z 1 | z 2 ) =
Revised: N ( 0 , I ) , and we use diagonal Gaussian models for p θ ( z 1 | z 2 ) =

Original: N ( g µ ( z 2 ) , g σ ( z 2 ) 2 ) and (for SVHN and CIFAR-10)
Revised: N ( g µ ( z 2 ) , g σ ( z 2 ) 2 ) and (for SVHN and CIFAR-10)

Original: p θ ( x | z 1 )
Revised: p θ ( x | z 1 )

Original: =
Revised: =

Original: N ( g µ ′ ( z 1 ) , σ 2 x I ) (this is similar to, e.g., (Minnen et al., 2018)).
Revised: N ( g µ ′ ( z 1 ) , σ 2 x I ) (this is similar to, e.g., (Minnen et al., 2018)).

Original: Here, g µ , g σ , and g µ ′ , denote neural networks (see details below).
Revised: Here, g µ , g σ , and g µ ′ , denote neural networks (see details below).

Original: Since MNIST has binary pixel values, we model it with a Bernoulli distribution for p θ ( x | z 1 ) =
Revised: Since MNIST has binary pixel values, we model it with a Bernoulli distribution for p θ ( x | z 1 ) =

Original: Bern( g µ ′ ( z 1 )) .
Revised: Bern( g µ ′ ( z 1 )) .

Original: For the inference model, we also use diagonal Gaussian models for q ϕ ( z 2 | x ) =
Revised: For the inference model, we also use diagonal Gaussian models for q ϕ ( z 2 | x ) =

Original: N ( f µ ( x ) , f σ ( x ) 2 ) and for q ϕ ( z 1 | x , z 2 ) =
Revised: N ( f µ ( x ) , f σ ( x ) 2 ) and for q ϕ ( z 1 | x , z 2 ) =

Original: N ( f µ ′ ( x , z 2 ) , f σ ′ ( x , z 2 ) 2 ) , where f µ , f σ , f µ ′ , and f σ ′ are again neural networks.
Revised: N ( f µ ′ ( x , z 2 ) , f σ ′ ( x , z 2 ) 2 ) , where f µ , f σ , f µ ′ , and f σ ′ are again neural networks.

Original: We examine both LVAE (Figure 2(b)) and our generalized top-down HVAEs (GHVAEs; see Figure 2(c)), using simple network architectures with only 2 to 3 convolutional and 1 fully connected layers (see Appendix A.1 for details) so that we can scan a large rate-space efficiently.
Revised: We examine both LVAE (Figure 2(b)) and our generalized top-down HVAEs (GHVAEs; see Figure 2(c)), using simple network architectures with only 2 to 3 convolutional and 1 fully connected layers (see Appendix A.1 for details) so that we can scan a large rate-space efficiently.

Original: Note that we are not trying to find the new state-of-the-art HVAEs.
Revised: Note that we are not trying to find the new state-of-the-art HVAEs.

Original: Results for LVAE are in Appendix A.2.2.
Revised: Results for LVAE are in Appendix A.2.2.

Original: We trained 441 different HVAEs for each data set/model combination, scanning the ratehyperparameters ( β 2 , β 1| 2 ) over a 21 × 21 grid ranging from 0 .
Revised: We trained 441 different HVAEs for each data set/model combination, scanning the ratehyperparameters ( β 2 , β 1 ) over a 21 × 21 grid ranging from 0 .

Original: 1 to 10 on a log scale in both directions (see Figure 1 on page 2, right panels).
Revised: 1 to 10 on a log scale in both directions (see Figure 1 on page 2, right panels).

Original: Each model took about 2 hours to train on an RTX-2080Ti
Revised: Each model took about 2 hours to train on an RTX-2080Ti

Original: GPU ( ∼ 27 hours in total for each data set/model combination using 32 GPUs in parallel).
Revised: GPU ( ∼ 27 hours in total for each data set/model combination using 32 GPUs in parallel).

Original: Baselines.
Revised: Baselines.

Original: Our proposed framework (Eq.
Revised: Our proposed framework (Eq.

Original: 8) generalizes over both VAEs and β -VAEs (Eq. 2), which we obtain in the cases β 2 = β 1| 2 = 1 and β 2 = β 1| 2 , respectively.
Revised: 8) generalizes over both VAEs and β -VAEs (Eq. 2), which we obtain in the cases β 2 = β 1 = 1 and β 2 = β 1 , respectively.

Original: These baselines are indicated as black “ ” and red “ ” circles, respectively, in Figures 3, 5, 6, and 7, discussed below.
Revised: These baselines are indicated as black “ ” and red “ ” circles, respectively, in Figures 3, 5, 6, and 7, discussed below.

Original: [Figure]
Revised: [Figure]

Original: Metrics.
Revised: Metrics.

Original: Performance metrics for the three application domains of VAEs mentioned in the introduction are introduced at the beginnings of the corresponding Sections 5.4-5.6.
Revised: Performance metrics for the three application domains of VAEs mentioned in the introduction are introduced at the beginnings of the corresponding Sections 5.4-5.6.

Original: In addition, we evaluate the individual rates R ( z 2 ) and R ( z 1 | z 2 )
Revised: In addition, we evaluate the individual rates R ( z 2 ) and R ( z 1 | z 2 )

Original: (Eq. 7), which we report in nats (i.e., to base e ).
Revised: (Eq. 7), which we report in nats (i.e., to base e ).

Original: 5. T HERE IS NO “O NE HVAE F ITS A LL ”
Revised: 5. T HERE IS NO “O NE HVAE F ITS A LL ”

Original: Figure 1 on page 2 summarizes our results.
Revised: Figure 1 on page 2 summarizes our results.

Original: The 21 × 21 GHVAEs trained with the grid of hyperparameters β 2 and β 1| 2 map out a surface in a 3d-space spanned by suitable metrics for the three application domains (metrics defined in Sections 5.4-5.6 below).
Revised: The 21 × 21 GHVAEs trained with the grid of hyperparameters β 2 and β 1 map out a surface in a 3d-space spanned by suitable metrics for the three application domains (metrics defined in Sections 5.4-5.6 below).

Original: The two upper right panels map colors on this surface to β s used for training and to the resulting layer-wise rates, respectively.
Revised: The two upper right panels map colors on this surface to β s used for training and to the resulting layer-wise rates, respectively.

Original: The lower right panels show performance landscapes and identify the optimal models for the three application domains of data reconstruction ( △ ), representation learning ( ⋄ ), and generative modeling ( (cid:68) ).
Revised: The lower right panels show performance landscapes and identify the optimal models for the three application domains of data reconstruction ( △ ), representation learning ( ⋄ ), and generative modeling ( (cid:68) ).

Original: The figure shows that moving away from a conventional β -VAE ( β 2 = β 1| 2 ; dashed red lines in
Revised: The figure shows that moving away from a conventional β -VAE ( β 2 = β 1 ; dashed red lines in

Original: Figure 1) allows us to find better models for a given application domain as the three application domains favor vastly different regions in β -space.
Revised: Figure 1) allows us to find better models for a given application domain as the three application domains favor vastly different regions in β -space.

Original: Thus, there is no single HVAE that is optimal for all tasks , and a HVAE that has been optimized for one task can perform poorly on a different task.
Revised: Thus, there is no single HVAE that is optimal for all tasks , and a HVAE that has been optimized for one task can perform poorly on a different task.

Original: 5. D EFINITION OF THE O PTIMAL M ODEL FOR A G IVEN T OTAL R ATE
Revised: 5. D EFINITION OF THE O PTIMAL M ODEL FOR A G IVEN T OTAL R ATE

Original: One of the questions we study in Sections 5.4-5.6 below is: “Which allocation of rates across layers results in best model performance if we keep the total rate R fixed ”.
Revised: One of the questions we study in Sections 5.4-5.6 below is: “Which allocation of rates across layers results in best model performance if we keep the total rate R fixed ”.

Original: Unfortunately, it is difficult to keep R fixed at training time since we control rates only indirectly via their Lagrange multipliersβand β1 | 2  .
Revised: Unfortunately, it is difficult to keep R fixed at training time since we control rates only indirectly via their Lagrange multipliers β 2 and β 1 .

Original: We instead use the following definition, illustrated in Figure 6 for a performance metric introduced in Section 5.6 below.
Revised: We instead use the following definition, illustrated in Figure 6 for a performance metric introduced in Section 5.6 below.

Original: The figure plots the performance metric over R for all 21 × 21 βsettings and highlights with purple circles “ ” all points on the upper convex hull.
Revised: The figure plots the performance metric over R for all 21 × 21 βsettings and highlights with purple circles “ ” all points on the upper convex hull.

Original: These highlighted models are optimal for a small interval of total rates in the following sense: if we use the total rates R of all “ ” to partition the horizontal axis into intervals then, by definition of the convex hull, each “ ” represents the model with highest performance in either the interval to its left or the one to its right.
Revised: These highlighted models are optimal for a small interval of total rates in the following sense: if we use the total rates R of all “ ” to partition the horizontal axis into intervals then, by definition of the convex hull, each “ ” represents the model with highest performance in either the interval to its left or the one to its right.

Original: 5. P ERFORMANCE ON D ATA R ECONSTRUCTION
Revised: 5. P ERFORMANCE ON D ATA R ECONSTRUCTION

Original: Reconstruction is a popular task for VAEs, e.g., in the area of lossy compression (Ball´e et al., 2016).
Revised: Reconstruction is a popular task for VAEs, e.g., in the area of lossy compression (Ball´e et al., 2016).

Original: We measure reconstruction quality using the common peak signal-to-noise ratio (PSNR), which is equal to E x ∼ X test [ − log D ] up to rescaling and shifting.
Revised: We measure reconstruction quality using the common peak signal-to-noise ratio (PSNR), which is equal to E x ∼ X test [ − log D ] up to rescaling and shifting.

Original: Higher PSNR means better reconstruction.
Revised: Higher PSNR means better reconstruction.

Original: Figure 3(a) shows a 3d-plot of PSNR as a function of both R ( z 1 | z 2 ) and R ( z 2 ) for SVHN, thus generalizing the rate/distortion curve of a conventional β -VAE to a rate/rate/distortion surface.
Revised: Figure 3(a) shows a 3d-plot of PSNR as a function of both R ( z 1 | z 2 ) and R ( z 2 ) for SVHN, thus generalizing the rate/distortion curve of a conventional β -VAE to a rate/rate/distortion surface.

Original: Figure 3(b) introduces a more compact 2d-representation of the same data that we use for all remaining metrics in the rest of this section and in Appendix A.2, and it also shows results for CIFAR-10.
Revised: Figure 3(b) introduces a more compact 2d-representation of the same data that we use for all remaining metrics in the rest of this section and in Appendix A.2, and it also shows results for CIFAR-10.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: markers

Revised: Figure

Original: Unsurprisingly and consistent with Eq.
Revised: Unsurprisingly and consistent with Eq.

Original: 9, reconstruction performance improves as total rate grows.
Revised: 9, reconstruction performance improves as total rate grows.

Original: However, minimizing distortion without any constraints is not useful in practice as we can simply use the original data, which has no distortion.
Revised: However, minimizing distortion without any constraints is not useful in practice as we can simply use the original data, which has no distortion.

Original: To simulate a practical constraint in, e.g., a datacompression application, we consider models with optimal PSNR for a given total rate R (as defined in Section 5.3) which are marked as purple circles “ ” in Figure 3(b).
Revised: To simulate a practical constraint in, e.g., a datacompression application, we consider models with optimal PSNR for a given total rate R (as defined in Section 5.3) which are marked as purple circles “ ” in Figure 3(b).

Original: We see for both SVHN and
Revised: We see for both SVHN and

Original: CIFAR-10 that conventional β -VAEs ( β 2 = β 1| 2 ; red circles) perform somewhat suboptimal for a given total rate and can be improved by trading some rate in z 2 for some rate in z 1 .
Revised: CIFAR-10 that conventional β -VAEs ( β 2 = β 1 ; red circles) perform somewhat suboptimal for a given total rate and can be improved by trading some rate in z 2 for some rate in z 1 .

Original: Reconstruction examples for the three models marked with crosses in Figure 3(b) are shown in Figure 4 (bottom).
Revised: Reconstruction examples for the three models marked with crosses in Figure 3(b) are shown in Figure 4 (bottom).

Original: Visual reconstruction quality improves from “3” to “2” to “1”, consistent with reported PSNRs.
Revised: Visual reconstruction quality improves from “3” to “2” to “1”, consistent with reported PSNRs.

Original: 5. P ERFORMANCE ON S AMPLE G ENERATION
Revised: 5. P ERFORMANCE ON S AMPLE G ENERATION

Original: We next evaluate how tuning layer-wise rates affects the quality of samples from the generative model.
Revised: We next evaluate how tuning layer-wise rates affects the quality of samples from the generative model.

Original: We measure sample quality by the widely used Inception Score (IS) (Salimans et al., 2016),
Revised: We measure sample quality by the widely used Inception Score (IS) (Salimans et al., 2016),

Original: [Equation]
Revised: [Equation]

Original: Here, p θ is the trained generative model (Eq. 1), p cls.
Revised: Here, p θ is the trained generative model (Eq. 1), p cls.

Original: ( y | x ) is the predictive distribution of a classifier trained on the same training set, and p cls.
Revised: ( y | x ) is the predictive distribution of a classifier trained on the same training set, and p cls.

Original: ( y ) :
Revised: ( y ) :

Original: = E p θ ( x ) [ p cls.
Revised: = E p θ ( x ) [ p cls.

Original: ( y | x )] .
Revised: ( y | x )] .

Original: The second equality in Eq.follows Barratt & Sharma (2018) to split IS into a product of a diversity score and a sharpness score.
Revised: The second equality in Eq.follows Barratt & Sharma (2018) to split IS into a product of a diversity score and a sharpness score.

Original: Higher is better for all scores.
Revised: Higher is better for all scores.

Original: The classifier is a ResNet-18 (He et al., 2016) for SVHN (test accuracy 95 .
Revised: The classifier is a ResNet-18 (He et al., 2016) for SVHN (test accuracy 95 .

Original: 02 % ) and a DenseNet-121 (Huang et al., 2017) for CIFAR-10 (test accuracy 94 .
Revised: 02 % ) and a DenseNet-121 (Huang et al., 2017) for CIFAR-10 (test accuracy 94 .

Original: Figure 5 (left) shows IS for GHVAEs trained on SVHN.
Revised: Figure 5 (left) shows IS for GHVAEs trained on SVHN.

Original: Unlike the results for PSNR, here, higher rate does not always lead to better sample quality: for very high R ( z 2 ) and low R ( z 1 | z 2 ) , IS eventually drops.
Revised: Unlike the results for PSNR, here, higher rate does not always lead to better sample quality: for very high R ( z 2 ) and low R ( z 1 | z 2 ) , IS eventually drops.

Original: The region of high IS is in the area where β 2 < β 1| 2 , i.e., where R ( z 2 ) is higher than in a comparable conventional β -VAE.
Revised: The region of high IS is in the area where β 2 < β 1 , i.e., where R ( z 2 ) is higher than in a comparable conventional β -VAE.

Original: The center and right panels of Figure 5 show diversity and sharpness, indicating that IS is mainly driven here by sharpness, which depends mostly on R ( z 2 ) , possibly because z 2 captures higher-level concepts than z 1 that may be more important to the classifier in Eq.
Revised: The center and right panels of Figure 5 show diversity and sharpness, indicating that IS is mainly driven here by sharpness, which depends mostly on R ( z 2 ) , possibly because z 2 captures higher-level concepts than z 1 that may be more important to the classifier in Eq.

Original: Samples from the the three models marked with crosses in Figure 5 are shown in
Revised: Samples from the the three models marked with crosses in Figure 5 are shown in

Original: Figure 4 (top).
Revised: Figure 4 (top).

Original: Visual sample quality improves from “1” to “3” to “2”, consistent with reported IS.
Revised: Visual sample quality improves from “1” to “3” to “2”, consistent with reported IS.

Original: Markers “ ”, “ ”, and “ ” same as in Figure 3.
Revised: Markers “ ”, “ ”, and “ ” same as in Figure 3.

Original: 5. P ERFORMANCE ON R EPRESENTATION L EARNING FOR D OWNSTREAM C LASSIFICATION
Revised: 5. P ERFORMANCE ON R EPRESENTATION L EARNING FOR D OWNSTREAM C LASSIFICATION

Original: VAEs are very popular for representation learning as they map complicated high dimensional data x to typically lower dimensional representations { z ℓ } .
Revised: VAEs are very popular for representation learning as they map complicated high dimensional data x to typically lower dimensional representations { z ℓ } .

Original: To measure the quality of learned representations, we train two sets of classifiers on a labeled test set for each trained HVAE, each consisting of: logistic regression, a Support Vector Machine (SVM) (Boser et al., 1992) with linear kernel, an SVM with RBF kernel, and k -nearest neighbors (kNN) with k = 5 .
Revised: To measure the quality of learned representations, we train two sets of classifiers on a labeled test set for each trained HVAE, each consisting of: logistic regression, a Support Vector Machine (SVM) (Boser et al., 1992) with linear kernel, an SVM with RBF kernel, and k -nearest neighbors (kNN) with k = 5 .

Original: One set of classifiers is conditioned on the mode µ 2 of q ϕ
Revised: One set of classifiers is conditioned on the mode µ 2 of q ϕ

Original: ( z 2 | x ) and the other one on the mode µ 1 of q ϕ
Revised: ( z 2 | x ) and the other one on the mode µ 1 of q ϕ

Original: ( z 1 | z 2 , x ) , where z 2 ∼ q ϕ ( z 2 | x ) .
Revised: ( z 1 | z 2 , x ) , where z 2 ∼ q ϕ ( z 2 | x ) .

Original: We use the implementations from scikit-learn (Pedregosa et al., 2011) for all classifiers.
Revised: We use the implementations from scikit-learn (Pedregosa et al., 2011) for all classifiers.

Original: [Table]
Revised: [Table]

Original: Figure 7 shows the classification accuracies (columns 2-5) for all classifiers trained on µ 2 .
Revised: Figure 7 shows the classification accuracies (columns 2-5) for all classifiers trained on µ 2 .

Original: The first column shows the mutual information I q ( y ; z 2 ) , which depends mainly on R ( z 2 ) as expected from Eq. 10.
Revised: The first column shows the mutual information I q ( y ; z 2 ) , which depends mainly on R ( z 2 ) as expected from Eq. 10.

Original: As long as the classifier is expressive enough (e.g., RBF-SVM or kNN) and the data set is simple (SVHN; top row), higher mutual information ( ≈ higher R ( z 2 ) ) corresponds to higher classification accuracies, consistent with Eq.
Revised: As long as the classifier is expressive enough (e.g., RBF-SVM or kNN) and the data set is simple (SVHN; top row), higher mutual information ( ≈ higher R ( z 2 ) ) corresponds to higher classification accuracies, consistent with Eq.

Original: But for less expressive (e.g., linear) classifiers or more complex data (CIFAR-10; bottom row), increasing R ( z 1 | z 2 ) improves classification accuracy (see purple circles “ ” in corresponding panels), consistent with the discussion below Eq.
Revised: But for less expressive (e.g., linear) classifiers or more complex data (CIFAR-10; bottom row), increasing R ( z 1 | z 2 ) improves classification accuracy (see purple circles “ ” in corresponding panels), consistent with the discussion below Eq.

Original: We see a similar effect (Table 1) for most classifier/data set combinations when replacing µ 2 by µ 1 , which has more information about x but is also higher dimensional.
Revised: We see a similar effect (Table 1) for most classifier/data set combinations when replacing µ 2 by µ 1 , which has more information about x but is also higher dimensional.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We classified the various tasks that can be performed with Variational Autoencoders (VAEs) into three application domains and argued that each domain has different trade-offs, such that a good
Revised: We classified the various tasks that can be performed with Variational Autoencoders (VAEs) into three application domains and argued that each domain has different trade-offs, such that a good

Original: VAE for one domain is not necessarily good for another.
Revised: VAE for one domain is not necessarily good for another.

Original: This observation motivated us to propose a refinement of the rate/distortion theory of VAEs that allows trading off rates across individual layers of latents in hierarchical VAEs.
Revised: This observation motivated us to propose a refinement of the rate/distortion theory of VAEs that allows trading off rates across individual layers of latents in hierarchical VAEs.

Original: We showed both theoretically and empirically that the proposal indeed provides practitioners better control for tuning VAEs for the three application domains.
Revised: We showed both theoretically and empirically that the proposal indeed provides practitioners better control for tuning VAEs for the three application domains.

Original: In the future, it would be interesting to find an adaptive schedule for the Lagrange parameters βL and β ℓ | ℓ +1 that makes it possible to target a specific given rate for each layer in a single training run.
Revised: In the future, it would be interesting to explore adaptive schedules for the Lagrange parameters β that would make it possible to target a specific given rate for each layer in a single training run, for example by using the method proposed by Rezende & Viola (2018).

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: We have included our code as a .zip file with the submission, which will also be open source on GitHub after the reviewing process.
Revised: We have included our code as a .zip file with the submission, which will also be open source on GitHub after the reviewing process.

Original:  I MPLEMENTATION D ETAILS
Revised: A E XPERIMENT S UPPLEMENTARIES A. I MPLEMENTATION D ETAILS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. A DDITIONAL R ESULTS
Revised: A. A DDITIONAL R ESULTS

Original: Here we attached the results for MNIST, as well as the full results for LVAE on SVHN and generalized top-down HVAEs on CIFAR-10.
Revised: Here we attached the results for MNIST, as well as the full results for LVAE on SVHN and generalized top-down HVAEs on CIFAR-10.

Original: A.2. R ESULTS FOR GENERALIZED TOP - DOWN HVAE S ON MNIST
Revised: A.2. R ESULTS FOR GENERALIZED TOP - DOWN HVAE S ON MNIST

Original: We also evaluate our proposed framework using generalized top-down HVAEs trained on binary
Revised: We also evaluate our proposed framework using generalized top-down HVAEs trained on binary

Original: MNIST data (i
Revised: MNIST data (i

Original: [Figure]
Revised: [Figure]

Original: e., black and white images rather than grayscale)
Revised: e., black and white images rather than grayscale)

Original: [Figure]
Revised: [Figure]

Original: We note that the inception score (IS) behaves different in our MNIST models compared to SVHN (see Figure 5) in that optimal IS in MNIST occurs for high R ( z 1 | z 2 ) rather than high R ( z 2 ) .
Revised: We note that the inception score (IS) behaves different in our MNIST models compared to SVHN (see Figure 5) in that optimal IS in MNIST occurs for high R ( z 1 | z 2 ) rather than high R ( z 2 ) .

Original: This indicates that semantically low-level properties (hand-writing style) of MNIST might have more variation than high level properties (the digit), whereas SVHN images show variation in additional high-level properties such as the background color
Revised: This indicates that semantically low-level properties (hand-writing style) of MNIST might have more variation than high level properties (the digit)

Original: [Figure]
Revised: [Figure]

Revised: whereas SVHN images show variation in additional high-level properties such as the background color

Original: Figure
Revised: [Figure]

Original: A.2. R ESULTS FOR LVAE ON SVHN
Revised: A.2. R ESULTS FOR LVAE ON SVHN

Original: [Figure]
Revised: [Figure]

Original: A.2. R ESULTS FOR GENERALIZED TOP - DOWN HVAE S ON CIFAR- 
Revised: A.2. R ESULTS FOR GENERALIZED TOP - DOWN HVAE S ON CIFAR-

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. P ROOF OF T HE B OUND ON C LASSIFICATION A CCURACY
Revised: B P ROOF OF T HE B OUND ON C LASSIFICATION A CCURACY

Original: This section provides a proof of Eq.
Revised: This section provides a proof of Eq.

Original: 11 by reformulating the proof of Proposition 5 in the thesis by Meyen (2016) into the notation used in the present paper.
Revised: 11 by reformulating the proof of Proposition 5 in the thesis by Meyen (2016) into the notation used in the present paper.

Original: We stress that this section contains no original contribution and is provided only as a convenience to the reader, motivated by reviewer feedback.
Revised: We stress that this section contains no original contribution and is provided only as a convenience to the reader, motivated by reviewer feedback.

Original: All credits for this section belong to Meyen (2016).
Revised: All credits for this section belong to Meyen (2016).

Original: We consider an (unknown) true data generative distribution p data ( y, x ) for data x with (unobserved) true labels y , and a hierarchical VAE with an inference model q ϕ ( { z ℓ } | x ) of the form of Eq. 4.
Revised: We consider an (unknown) true data generative distribution p data ( y, x ) for data x with (unobserved) true labels y , and a hierarchical VAE with an inference model q ϕ ( { z ℓ } | x ) of the form of Eq. 4.

Original: Focusing on a single layer ℓ of latents, we denote the joint probability over y , x , and z ℓ as
Revised: Focusing on a single layer ℓ of latents, we denote the joint probability over y , x , and z ℓ as

Original: [Equation]
Revised: [Equation]

Revised: (13) where the marginal q ϕ ( z ℓ | x ) of q ϕ ( { z ℓ } | x ) is defined as usual.

Original: We further consider a classifier p cls.
Revised: We further consider a classifier p cls.

Original: ( y | z ℓ ) that operates on z ℓ .
Revised: ( y | z ℓ ) that operates on z ℓ .

Original: Denoting its top prediction as ˆ y := arg max y p cls.
Revised: Denoting its top prediction as ˆ y := arg max y p cls.

Original: ( y | z ℓ ) , the classification accuracy is α :=
Revised: ( y | z ℓ ) , the classification accuracy is α :=

Original: E q [ δ y, ˆ y ] , where δ is the Kronecker delta.
Revised: E q [ δ y, ˆ y ] , where δ is the Kronecker delta.

Original: Theorem 1.
Revised: Theorem 1.

Original: The mutual information I q (  z ℓ; y ) between the latent representation z ℓ and the true label y under the distribution q defined in Eq.
Revised: The mutual information I q ( y ; z ℓ ) between the latent representation z ℓ and the true label y under the distribution q defined in Eq.

Original: 13 is lower bounded as follows,
Revised: 13 is lower bounded as follows,

Revised: I

Original: [Equation]
Revised: [Equation]

Original:  with
Revised: ) with

Original: [Equation]
Revised: [Equation]

Original: where H 2 ( α ) = − α log α − (1 − α ) log(1 − α ) is the entropy of a Bernoulli distribution with success probability α , H p data [ y ] ≤ log M is the marginal entropy of the true labels, and M denotes the number of classes.
Revised: where H 2 ( α ) = − α log α − (1 − α ) log(1 − α ) is the entropy of a Bernoulli distribution, H p data [ y ] ≤ log M is the marginal entropy of the true labels, and M denotes the number of classes.

Original: Before we prove Theorem 1, we note that the function f is strictly monotonically increasing on the relevant interval [max y p data ( y ) , 1] .
Revised: Before we prove Theorem 1, we note that the function f is strictly monotonically increasing on the relevant interval [max y p data ( y ) , 1] .

Original: Thus, f is invertible and we obtain the following corollary:
Revised: Thus, f is invertible and we obtain the following corollary:

Original: Corollary 1.
Revised: Corollary 1.

Original: The classification accuracy α is upper bounded as in Eq.
Revised: The classification accuracy α is upper bounded as in Eq.

Original: 11 of the main text, i.e., α ≤ f − 1 ( I q (  z ℓ ; y )) .
Revised: 11 of the main text, i.e., [Equation] The second inequality in Eq. 15 results from the bound I q ( y ; z ℓ ) ≤

Revised: E p data ( x ) [ R ( z ≥ ℓ )] derived in

Revised: Eq. 10, using the fact that f − 1 is monotonically increasing (since f is).

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: We split the mutual information into two contributions,
Revised: We split the mutual information into two contributions,

Original: [Equation]
Revised: [Equation]

Original: where, as clarified in the second equality,
Revised: where, as clarified in the second equality,

Original: H q [ y | z ℓ ] is the expectation over z ℓ of the conditional entropy of y given z ℓ , and q ( z ℓ ) and q ( y | z ℓ ) are marginals and conditionals of q (Eq. 13) as usual.
Revised: H q [ y | z ℓ ] is the expectation over z ℓ of the conditional entropy of y given z ℓ , and q ( z ℓ ) and q ( y | z ℓ ) are marginals and conditionals of q (Eq. 13) as usual.

Original: The marginal true label entropy H p data [ y ]in Eq. 16 is fixed by the problem at hand. In order to find a lower bound on I q (  z ℓ; y ) for a given classification accuracy α , we therefore have to find an upper bound on  H q 
Revised: Since H p data [ y ] is fixed by the problem at hand, finding a lower bound on I q ( y ; z ℓ ) for a given classification accuracy α is equivalent to finding an upper bound on the second term on the right-hand side of Eq. 16, H q [ y | z ℓ ] =

Original:  [  y | z ℓ ] subject to the constraint E q [ δ y, ˆ y ] = α .
Revised: E z ℓ ∼ q ( z ℓ ) [ E y ∼ q ( y | z ℓ ) [ − log q ( y | z ℓ )]] , with the constraint E q [ δ y, ˆ y ] = α .

Original: We do this by upper bounding the conditional entropy E y ∼ q
Revised: We do this by upper bounding the conditional entropy E y ∼ q

Original: ( y | z ℓ ) [ − log q ( y | z ℓ )] of y given z ℓ for all z ℓ independently, and then taking the expectation over  q ( z ℓ ) .
Revised: ( y | z ℓ ) [ − log q ( y | z ℓ )] of y given z ℓ for all z ℓ independently, and then taking the expectation over z ℓ ∼ q ( z ℓ ) .

Original: For  fixed  z ℓ , we first split off the contribution to E y ∼ q ( y | z ℓ ) [ − log q ( y | z ℓ )] from y
Revised: For a fixed latent representation z ℓ , we first split off the contribution to E y ∼ q ( y | z ℓ ) [ − log q ( y | z ℓ )] from y

Original: = ˆ y , where ˆ y = arg max y p cls.
Revised: = ˆ y , where ˆ y = arg max y p cls.

Original: ( y | z ℓ ) is the label that our classifier would predict forthe fixed latent representation z ℓ ,E y ∼ [Equation] =
Revised: ( y | z ℓ ) is the label that our classifier would predict for z ℓ ,

Original: [Equation]

Original: |

Original: [Equation]
Revised: [Equation]

Original: − (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Here, the second term on the right-hand side resembles the entropy of a distribution over the remaining ( M − 1) labels ( y ̸
Revised: Here, the second term on the right-hand side resembles the entropy of a distribution over the remaining ( M − 1) labels ( y ̸

Original: = ˆ y ), except that the probabilities sum to (1 − q ( y = ˆ y | z ℓ )) rather than one.
Revised: = ˆ y ), except that the probabilities sum to (1 − q ( y = ˆ y | z ℓ )) rather than one.

Original: Thus, for any q ( y = ˆ y | z ℓ ) , this term is maximized if q ( y | z ℓ ) distributes the remaining probability mass (1 − q ( y = ˆ y | z ℓ )) uniformly over the remaining ( M − 1) labels, i.e.,
Revised: Thus, regardless of the value of q ( y = ˆ y | z ℓ ) , this term is maximized if q ( y | z ℓ ) distributes the remaining probability mass (1 − q ( y = ˆ y | z ℓ )) uniformly over the remaining ( M − 1) labels, i.e.,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Plugging Eq.
Revised: Plugging Eq.

Original: 18 back into Eq.
Revised: 18 back into Eq.

Original: 16, we obtain the bound
Revised: 16, we obtain the bound

Revised: I q ( y ; z ℓ

Original: [Equation]
Revised: [Equation]

Revised: ≥

Revised: [Equation]

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Revised: log

Original: [Equation]
Revised: [Equation]

Original: We arrive at the proposition (Eq. 14) by pulling the concave function H 2 out of the expectation using
Revised: We arrive at the proposition (Eq. 14) by pulling the concave function H 2 out of the expectation using

Original: Jensen’s inequality, and by then identifying E z ℓ ∼ q ( z 
Revised: Jensen’s inequality, and by then identifying E z ℓ ∼ q ( z ℓ )

Original: [Equation]

Original: [ q ( y = ˆ y | z ℓ )]
Revised: [ q ( y = ˆ y | z ℓ )]

Original: =
Revised: =

Original: q ( y = ˆ y )
Revised: q ( y = ˆ y )

Original: =
Revised: =

Original: α .
Revised: α .