Original: H I LL O C: L OSSLESS I MAGE C OMPRESSION WITH H IERARCHICAL L ATENT V ARIABLE M ODELS
Revised: H I LL O C: L OSSLESS I MAGE C OMPRESSION WITH H IERARCHICAL L ATENT V ARIABLE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We make the following striking observation: fully convolutional VAE models trained on 32 × 32 ImageNet can generalize well, not just to 64 × 64 but also to far larger photographs, with no changes to the model.
Revised: We make the following striking observation: fully convolutional VAE models trained on 32 × 32 ImageNet can generalize well, not just to 64 × 64 but also to far larger photographs, with no changes to the model.

Original: We use this property, applying fully convolutional models to lossless compression, demonstrating a method to scale the VAE-based ‘Bits-Back with ANS’ algorithm for lossless compression (Townsend et al., 2019) to large color photographs, and achieving state of the art for compression of full size ImageNet images.
Revised: We use this property, applying fully convolutional models to lossless compression, demonstrating a method to scale the VAE-based ‘Bits-Back with ANS’ algorithm for lossless compression (Townsend et al., 2019) to large color photographs, and achieving state of the art for compression of full size ImageNet images.

Original: We release Craystack, an open source library for convenient prototyping of lossless compression using probabilistic models, along with full implementations of all of our compression results 1 .
Revised: We release Craystack, an open source library for convenient prototyping of lossless compression using probabilistic models, along with full implementations of all of our compression results 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Bits back coding (Wallace, 1990; Hinton & van Camp, 1993) is a method for performing lossless compression using a latent variable model.
Revised: Bits back coding (Wallace, 1990; Hinton & van Camp, 1993) is a method for performing lossless compression using a latent variable model.

Original: In an ideal implementation, the method can achieve an expected message length equal to the variational free energy, often referred to as the  evidence lower bound (ELBO) of the model.
Revised: In an ideal implementation, the method can achieve an expected message length equal to the variational free energy, often referred to as the negative evidence lower bound (ELBO) of the model.

Original: Bits back was ﬁrst introduced to form a theoretical argument for using the ELBO as an objective function for machine learning (Hinton & van Camp, 1993).
Revised: Bits back was ﬁrst introduced to form a theoretical argument for using the ELBO as an objective function for machine learning (Hinton & van Camp, 1993).

Original: The ﬁrst implementation of bits back coding (Frey, 1997; Frey & Hinton, 1996) made use of ﬁrst-inﬁrst-out (FIFO) arithmetic coding (AC)
Revised: The ﬁrst implementation of bits back coding (Frey, 1997; Frey & Hinton, 1996) made use of ﬁrst-inﬁrst-out (FIFO) arithmetic coding (AC)

Original: (Witten et al., 1987).
Revised: (Witten et al., 1987).

Original: However, the implementation did not achieve optimal compression, due to an incompatibility between a FIFO coder and bits back coding, and its use was only demonstrated on a small dataset of 8 × 8 binary images.
Revised: However, the implementation did not achieve optimal compression, due to an incompatibility between a FIFO coder and bits back coding, and its use was only demonstrated on a small dataset of 8 × 8 binary images.

Original: Recently, zero-overhead bits back compression with a signiﬁcantly simpler implementation has been developed  (Townsend et al., 2019).
Revised: Recently, zero-overhead bits back compression with a signiﬁcantly simpler implementation has been developed 1 (Townsend et al., 2019).

Original: This implementation makes use of asymmetric numeral systems (ANS), a last-in-ﬁrst-out (LIFO) entropy coding scheme (Duda, 2009).
Revised: This implementation makes use of asymmetric numeral systems (ANS), a last-in-ﬁrst-out (LIFO) entropy coding scheme (Duda, 2009).

Original: The method, known as ‘Bits
Revised: The method, known as ‘Bits

Original: Back with Asymmetric Numeral Systems’ (BB-ANS) was demonstrated by compressing the MNIST test set using a variational auto-encoder (VAE) model (Kingma & Welling, 2013; Rezende et al., 2014), achieving a compression rate within 1% of the model ELBO.
Revised: Back with Asymmetric Numeral Systems’ (BB-ANS) was demonstrated by compressing the MNIST test set using a variational auto-encoder (VAE) model (Kingma & Welling, 2013; Rezende et al., 2014), achieving a compression rate within 1% of the model ELBO.

Original: More recently, Hoogeboom et al.
Revised: More recently, Hoogeboom et al.

Original: and Ho et al.
Revised: and Ho et al.

Original: (2019) have proposed ﬂow-based methods for lossless compression, and Kingma et al.
Revised: (2019) have proposed ﬂow-based methods for lossless compression, and Kingma et al.

Original: (2019) have presented ‘Bit-Swap’, extending BB-ANS to hierarchical models.
Revised: (2019) have presented ‘Bit-Swap’, extending BB-ANS to hierarchical models.

Original: In this work we present an alternative method for extending to hierarchical
Revised: In this work we present an alternative method for extending to hierarchical

Original: VAEs.
Revised: VAEs.

Original: This entails the following novel techniques:
Revised: This entails the following novel techniques:

Original: Direct coding of arbitrary sized images using a fully convolutional model.
Revised: Direct coding of arbitrary sized images using a fully convolutional model.

Original: A vectorized ANS implementation supporting dynamic shape.
Revised: A vectorized ANS implementation supporting dynamic shape.

Original: Dynamic discretization to avoid having to calibrate a static discretization.
Revised: Dynamic discretization to avoid having to calibrate a static discretization.

Original: Initializing the bits back chain using a different codec.
Revised: Initializing the bits back chain using a different codec.

Original: We discuss each of these contributions in detail in Section 3.
Revised: We discuss each of these contributions in detail in Section 3.

Original: We call the combination of BB-ANS using a hierarchical latent variable model and the above techniques: ‘Hierarchical Latent Lossless
Revised: We call the combination of BB-ANS using a hierarchical latent variable model and the above techniques: ‘Hierarchical Latent Lossless

Original: Compression’ (HiLLoC).
Revised: Compression’ (HiLLoC).

Original: In our experiments (Section 4), we demonstrate that HiLLoC can be used to compress color images from the ImageNet test set at rates close to the ELBO, outperforming all of the other codecs which we benchmark.
Revised: In our experiments (Section 4), we demonstrate that HiLLoC can be used to compress color images from the ImageNet test set at rates close to the ELBO, outperforming all of the other codecs which we benchmark.

Original: We also demonstrate the speedup, of nearly three orders
Revised: We also demonstrate the speedup, of nearly three orders

Original: [Figure]
Revised: [Figure]

Original: of magnitude, resulting from vectorization.
Revised: of magnitude, resulting from vectorization.

Original: We release an open source implementation based on ‘Craystack’, a Python package which we have written for general prototyping of lossless compression with ANS.
Revised: We release an open source implementation based on ‘Craystack’, a Python package which we have written for general prototyping of lossless compression with ANS.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section we brieﬂy describe the BB-ANS algorithm ﬁrst introduced by Townsend et al.
Revised: In this section we brieﬂy describe the BB-ANS algorithm ﬁrst introduced by Townsend et al.

Original: We begin by giving a high-level description of the ANS LIFO entropy coder (Duda, 2009), along with a new notation for describing the basic ANS operations.
Revised: We begin by giving a high-level description of the ANS LIFO entropy coder (Duda, 2009), along with a new notation for describing the basic ANS operations.

Original: Throughout the rest of the paper we use log to mean the base two logarithm, usually denoted log 2 , and we measure message lengths in bits.
Revised: Throughout the rest of the paper we use log to mean the base two logarithm, usually denoted log 2 , and we measure message lengths in bits.

Original: 2. A SYMMETRIC NUMERAL SYSTEMS
Revised: 2. A SYMMETRIC NUMERAL SYSTEMS

Original: As an entropy coder, ANS was designed for compressing sequences of discretely distributed symbols.
Revised: As an entropy coder, ANS was designed for compressing sequences of discretely distributed symbols.

Original: It achieves a compressed message length equal to the negative log-probability (information content) of the sequence plus an implementation dependent constant, which is usually less than 32 bits.
Revised: It achieves a compressed message length equal to the negative log-probability (information content) of the sequence plus an implementation dependent constant, which is usually less than 32 bits.

Original: For long sequences, the constant overhead has a negligible contribution to the overall compression rate.
Revised: For long sequences, the constant overhead has a negligible contribution to the overall compression rate.

Original: Thus, by Shannon’s source coding theorem (Shannon, 1948), ANS coding is guaranteed to be near-optimal for long sequences.
Revised: Thus, by Shannon’s source coding theorem (Shannon, 1948), ANS coding is guaranteed to be near-optimal for long sequences.

Original: There are two basic operations deﬁned by ANS, which we will refer to as ‘push’ and ‘pop’.
Revised: There are two basic operations deﬁned by ANS, which we will refer to as ‘push’ and ‘pop’.

Original: Push encodes a symbol by adding it to an existing message.
Revised: Push encodes a symbol by adding it to an existing message.

Original: It has the signature
Revised: It has the signature

Original: [Equation]
Revised: [Equation]

Original: Pop is the inverse of push, and may be used to decode a symbol and recover a message identical to that before pushing.
Revised: Pop is the inverse of push, and may be used to decode a symbol and recover a message identical to that before pushing.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: When multiple symbols are pushed in sequence, they must be popped using the precise inverse procedure, which means popping the symbols in the opposite order.
Revised: When multiple symbols are pushed in sequence, they must be popped using the precise inverse procedure, which means popping the symbols in the opposite order.

Original: Hence why ANS is referred to as a last-in-ﬁrst-out coder, or a stack .
Revised: Hence why ANS is referred to as a last-in-ﬁrst-out coder, or a stack .

Original: The push and pop operations require access to a probabilistic model of symbols, summarized by a probability mass function p over the alphabet of possible symbols.
Revised: The push and pop operations require access to a probabilistic model of symbols, summarized by a probability mass function p over the alphabet of possible symbols.

Original: The way that symbols are encoded depends on the model, and pushing a symbol s according to p results in an increase in message length of log 1 p ( s ) .
Revised: The way that symbols are encoded depends on the model, and pushing a symbol s according to p results in an increase in message length of log 1 p ( s ) .

Original: Popping s results in an equal reduction in message length.
Revised: Popping s results in an equal reduction in message length.

Original: For details on how the ANS operations are implemented, see Duda (2009).
Revised: For details on how the ANS operations are implemented, see Duda (2009).

Original: Note that any model/mass function can be used for the pop operation, i.e. there’s no hard restriction to use the distribution that was used to encode the message.
Revised: Note that any model/mass function can be used for the pop operation, i.e. there’s no hard restriction to use the distribution that was used to encode the message.

Original: In this way, rather than decoding the same data that was encoded, pop can actually be used to sample a symbol from a different distribution.
Revised: In this way, rather than decoding the same data that was encoded, pop can actually be used to sample a symbol from a different distribution.

Original: The pop method itself is deterministic, so the source of randomness for the sample comes from the data contained within the message.
Revised: The pop method itself is deterministic, so the source of randomness for the sample comes from the data contained within the message.

Original: This sampling operation, which can be inverted by pushing the sample back onto the stack, is essential for bits back coding.
Revised: This sampling operation, which can be inverted by pushing the sample back onto the stack, is essential for bits back coding.

Original: For convenience, we introduce the shorthand notation s → p ( · ) for encoding (pushing) a symbol s according to p , and s ← p ( · ) for decoding (popping).
Revised: For convenience, we introduce the shorthand notation s → p ( · ) for encoding (pushing) a symbol s according to p , and s ← p ( · ) for decoding (popping).

Original: 2. B ITS BACK WITH ANS
Revised: 2. B ITS BACK WITH ANS

Original: Suppose we have a model for data x which involves a latent variable z .
Revised: Suppose we have a model for data x which involves a latent variable z .

Original: A sender and receiver wish to communicate a sample x .
Revised: A sender and receiver wish to communicate a sample x .

Original: They have access to a prior on z , denoted p ( z ) , a likelihood p ( x | z ) and a (possibly approximate) posterior q ( z | x ) , but not the marginal distribution p ( x ) .
Revised: They have access to a prior on z , denoted p ( z ) , a likelihood p ( x | z ) and a (possibly approximate) posterior q ( z | x ) , but not the marginal distribution p ( x ) .

Original: Without access to p ( x ) , sender and receiver cannot directly code x using ANS.
Revised: Without access to p ( x ) , sender and receiver cannot directly code x using ANS.

Original: However, BB-ANS speciﬁes an indirect way to push and pop x .
Revised: However, BB-ANS speciﬁes an indirect way to push and pop x .

Original: It does not require access to the marginal p ( x ) , but rather uses the prior, conditional, and posterior from the latent variable model.
Revised: It does not require access to the marginal p ( x ) , but rather uses the prior, conditional, and posterior from the latent variable model.

Original: Table 1(a) shows, in order from the top, the three steps of the BB-ANS pushing procedure which the sender can perform to encode x .
Revised: Table 1(a) shows, in order from the top, the three steps of the BB-ANS pushing procedure which the sender can perform to encode x .

Original: The ‘Variables’ column shows the variables known to the sender before each step. 1(b) shows the inverse steps which the receiver can use to pop x , with the ‘Variables’ column showing what is known to the receiver after each step.
Revised: The ‘Variables’ column shows the variables known to the sender before each step. 1(b) shows the inverse steps which the receiver can use to pop x , with the ‘Variables’ column showing what is known to the receiver after each step.

Original: After decoding x , the third step of popping, z → q ( · | x ) , is necessary to ensure that BB-ANS pop is a precise inverse of push.
Revised: After decoding x , the third step of popping, z → q ( · | x ) , is necessary to ensure that BB-ANS pop is a precise inverse of push.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: The change in message length from BB-ANS can easily be derived by adding up the quantities in the ∆ L column of Table 1.
Revised: The change in message length from BB-ANS can easily be derived by adding up the quantities in the ∆ L column of Table 1.

Original: For encoding we get
Revised: For encoding we get

Original: [Equation]
Revised: [Equation]

Original: Taking the expectation over z gives the expected message length for a datum x
Revised: Taking the expectation over z gives the expected message length for a datum x

Original: [Equation]
Revised: [Equation]

Original: which is the negative evidence lower bound (ELBO), also known as the free energy.
Revised: which is the negative evidence lower bound (ELBO), also known as the free energy.

Original: This is a commonly used training objective for latent variable models.
Revised: This is a commonly used training objective for latent variable models.

Original: The above equation implies that latent variable models trained using the ELBO are implicitly being trained to minimize the expected message length of lossless compression using BB-ANS.
Revised: The above equation implies that latent variable models trained using the ELBO are implicitly being trained to minimize the expected message length of lossless compression using BB-ANS.

Original: Note that, as Table 1 shows, the ﬁrst step of encoding a data point, x , using BB-ANS is to, counterintuitively, decode (and thereby sample) a latent z ← q ( · | x ) .
Revised: Note that, as Table 1 shows, the ﬁrst step of encoding a data point, x , using BB-ANS is to, counterintuitively, decode (and thereby sample) a latent z ← q ( · | x ) .

Original: This requires that there is already a buffer of random data pushed to the ANS coder, which can be popped.
Revised: This requires that there is already a buffer of random data pushed to the ANS coder, which can be popped.

Original: This data used to start the encoding process is recovered after the ﬁnal stage of decoding, hence the name ‘bits back’.
Revised: This data used to start the encoding process is recovered after the ﬁnal stage of decoding, hence the name ‘bits back’.

Original: If we have multiple samples to compress, then we can use ‘chaining’, which is essentially repeated application of the procedure in Table 1 (Townsend et al., 2019).
Revised: If we have multiple samples to compress, then we can use ‘chaining’, which is essentially repeated application of the procedure in Table 1 (Townsend et al., 2019).

Original: In Section 3.4 we describe how we build up an initial buffer of compressed data by using a different codec to code the ﬁrst images in a sequence.
Revised: In Section 3.4 we describe how we build up an initial buffer of compressed data by using a different codec to code the ﬁrst images in a sequence.

Original: [Figure]
Revised: [Figure]

Original: VectorizedANScoder ...
Revised: VectorizedANScoder ...

Original:  S CALING UP BITS BACK WITH ANS
Revised:  S CALING UP BITS BACK WITH ANS

Original: We now discuss the techniques we introduce to scale up BB-ANS.
Revised: We now discuss the techniques we introduce to scale up BB-ANS.

Original: 3. F ULLY CONVOLUTIONAL MODELS
Revised: 3. F ULLY CONVOLUTIONAL MODELS

Original: When all of the layers in the generative and recognition networks of a VAE are either convolutional or elementwise functions (i.e. the VAE has no densely connected layers), then it is possible to evaluate the recognition network on images of any height and width, and similarly to pass latents of any height and width through the generative network to generate an image.
Revised: When all of the layers in the generative and recognition networks of a VAE are either convolutional or elementwise functions (i.e. the VAE has no densely connected layers), then it is possible to evaluate the recognition network on images of any height and width, and similarly to pass latents of any height and width through the generative network to generate an image.

Original: Thus, such a VAE can be used as a (probabilistic) model for images of any size.
Revised: Thus, such a VAE can be used as a (probabilistic) model for images of any size.

Original: We exploit this fact, and show empirically in Section 4 that, surprisingly, a fully convolutional VAE trained on 32 × 32 images can perform well (in the sense of having a high ELBO) as a model for× 64 images as well as far larger images.
Revised: We exploit this fact, and show empirically in Section 4 that, surprisingly, a fully convolutional VAE trained on 32 × 32 images can perform well (in the sense of having a high ELBO) as a model for× 64 images as well as far larger images.

Original: This in turn corresponds to a good compression rate, and we implement lossless compression of arbitrary sized images by using a VAE in this way.
Revised: This in turn corresponds to a good compression rate, and we implement lossless compression of arbitrary sized images by using a VAE in this way.

Original: 3. V ECTORIZED LOSSLESS COMPRESSION
Revised: 3. V ECTORIZED LOSSLESS COMPRESSION

Original: The primary computational bottlenecks in the original BB-ANS implementation (Townsend et al., 2019) were loops over data and latent variables occurring in the Python interpreter.
Revised: The primary computational bottlenecks in the original BB-ANS implementation (Townsend et al., 2019) were loops over data and latent variables occurring in the Python interpreter.

Original: We have been able to vectorize these, achieving an implementation which can scale to large ImageNet images.
Revised: We have been able to vectorize these, achieving an implementation which can scale to large ImageNet images.

Original: The effect of vectorization on runtime is shown in Figure 4.
Revised: The effect of vectorization on runtime is shown in Figure 4.

Original: A vectorized implementation of ANS was described in Giesen (2014) using SIMD instructions.
Revised: A vectorized implementation of ANS was described in Giesen (2014) using SIMD instructions.

Original: This works by expanding the size of the ANS stack head, from a scalar to a vector, and interleaving the output/input bit stream.
Revised: This works by expanding the size of the ANS stack head, from a scalar to a vector, and interleaving the output/input bit stream.

Original: We implement this in our lossless compression library, Craystack, using
Revised: We implement this in our lossless compression library, Craystack, using

Original: Numpy.
Revised: Numpy.

Original:  We ensure that the compression rate overhead to vectorization is low by using the BitKnit technique described in Giesen (2015), see Appendix D for more detail.
Revised: Please refer to the Craystack code and to Giesen (2014) for more detail. We ensure that the compression rate overhead to vectorization is low by using the BitKnit technique described in Giesen (2015), see Appendix D for more detail.

Original: Having vectorized, we found that most of the compute time for our compression was spent in neural net inference, whether running on CPU or GPU, which we know to already be reasonably well optimized.
Revised: Having vectorized, we found that most of the compute time for our compression was spent in neural net inference, whether running on CPU or GPU, which we know to already be reasonably well optimized.

Original: In Craystack, we further generalize the ANS coder using Numpy’s n-dimensional array view interface, allowing the stack head to be ‘shaped’ like an n-dimensional array, or a nested Python data-structure containing arrays.
Revised: In Craystack, we further generalize the ANS coder using Numpy’s n-dimensional array view interface, allowing the stack head to be ‘shaped’ like an n-dimensional array, or a nested Python data-structure containing arrays.

Original: We can then use a shape which ﬁts that of the data that we wish to encode or decode.
Revised: We can then use a shape which ﬁts that of the data that we wish to encode or decode.

Original: When coding data according to a VAE we use an ANS stack head shaped into a pair of arrays, matching the shapes of the observation x and the latent z .
Revised: When coding data according to a VAE we use an ANS stack head shaped into a pair of arrays, matching the shapes of the observation x and the latent z .

Original: This allows for a straightforward implementation and clariﬁes the lack of data dependence between certain operations, such as the x → p ( · | z ) and z → p ( · ) during encoding, which can theoretically be performed concurrently.
Revised: This allows for a straightforward implementation and clariﬁes the lack of data dependence between certain operations, such as the x → p ( · | z ) and z → p ( · ) during encoding, which can theoretically be performed concurrently.

Original: This vectorized encoding process is visualized in Figure 2.
Revised: This vectorized encoding process is visualized in Figure 2.

Original: 3. D ISCRETIZATION
Revised: 3. D ISCRETIZATION

Original: It is standard for state of the art latent variable models to use continuous latent variables.
Revised: It is standard for state of the art latent variable models to use continuous latent variables.

Original: Since
Revised: Since

Original: ANS operates over discrete probability distributions, if we wish to use BB-ANS with such models it is necessary to discretize the latent space so that latent samples can be communicated.
Revised: ANS operates over discrete probability distributions, if we wish to use BB-ANS with such models it is necessary to discretize the latent space so that latent samples can be communicated.

Original: Townsend et al.
Revised: Townsend et al.

Original: (2019) described a static discretization scheme for the latents in a simple VAE with a single layer of continuous latent variables, and showed that this discretization has a negligible impact on compression rate.
Revised: (2019) described a static discretization scheme for the latents in a simple VAE with a single layer of continuous latent variables, and showed that this discretization has a negligible impact on compression rate.

Original: The addition of multiple layers of stochastic variables to a VAE has been shown to improve performance (Kingma et al., 2019;
Revised: The addition of multiple layers of stochastic variables to a VAE has been shown to improve performance (Kingma et al., 2019;

Original: Kingma et al., 2016; Maaløe et al., 2019;
Revised: Kingma et al., 2016; Maaløe et al., 2019;

Original: Sønderby et al., 2016).
Revised: Sønderby et al., 2016).

Original: Motivated by this, we propose a discretization scheme for hierarchical VAEs with multiple layers of latent variables.
Revised: Motivated by this, we propose a discretization scheme for hierarchical VAEs with multiple layers of latent variables.

Original: The discretization described in Townsend et al.
Revised: The discretization described in Townsend et al.

Original: (2019) is formed by dividing the latent space into intervals of equal mass under the prior p ( z ) .
Revised: (2019) is formed by dividing the latent space into intervals of equal mass under the prior p ( z ) .

Original: For a hierarchical model, the prior on each layer depends on the previous layers:
Revised: For a hierarchical model, the prior on each layer depends on the previous layers:

Original: [Equation]
Revised: [Equation]

Original: It isn’t immediately possible to use the simple static scheme from Townsend et al.
Revised: It isn’t immediately possible to use the simple static scheme from Townsend et al.

Original: (2019), since the marginals p ( z 1 ) , . .
Revised: (2019), since the marginals p ( z 1 ) , . .

Original: . , p ( z L − 1 ) are not known.
Revised: . , p ( z L − 1 ) are not known.

Original: Kingma et al.
Revised: Kingma et al.

Original: (2019) estimate these marginals by sampling, and create static bins based on the estimates.
Revised: (2019) estimate these marginals by sampling, and create static bins based on the estimates.

Original: They demonstrate that this approach can work well.
Revised: They demonstrate that this approach can work well.

Original: We propose an alternative approach, allowing the discretization to vary with the context of the latents we are trying to code.
Revised: We propose an alternative approach, allowing the discretization to vary with the context of the latents we are trying to code.

Original: We refer to our approach as dynamic discretization .
Revised: We refer to our approach as dynamic discretization .

Original: In dynamic discretization, instead of discretizing with respect to the marginals of the prior, we discretize according to the conditionals in the prior, p ( z l | z l +1: L ) .
Revised: In dynamic discretization, instead of discretizing with respect to the marginals of the prior, we discretize according to the conditionals in the prior, p ( z l | z l +1: L ) .

Original: Speciﬁcally, for each latent layer l , we partition each dimension into intervals which have equal probability mass under the conditional p ( z l | z l +1: L ) .
Revised: Speciﬁcally, for each latent layer l , we partition each dimension into intervals which have equal probability mass under the conditional p ( z l | z l +1: L ) .

Original: This directly generalizes the scheme used in BB-ANS Townsend et al.
Revised: This directly generalizes the scheme used in BB-ANS (Townsend et al., 2019).

Original: Dynamic discretization is more straightforward to implement because it doesn’t require callibrating the discretization to samples.
Revised: Dynamic discretization is more straightforward to implement because it doesn’t require callibrating the discretization to samples.

Original: However it imposes a restriction on model structure, in particular it requires that posterior inference is done top-down .
Revised: However it imposes a restriction on model structure, in particular it requires that posterior inference is done top-down .

Original: This precludes the use of Bit-Swap.
Revised: This precludes the use of Bit-Swap.

Original: In Section 3.3.1 we contrast the model restriction from dynamic discretization with the bottom-up, Markov restriction imposed by Bit-Swap itself.
Revised: In Section 3.3.1 we contrast the model restriction from dynamic discretization with the bottom-up, Markov restriction imposed by Bit-Swap itself.

Original: We give further details about the dynamic discretization implementation we use in Appendix A.
Revised: We give further details about the dynamic discretization implementation we use in Appendix A.

Original: 3.3. M ODEL RESTRICTIONS

Original: [Figure]
Revised: [Figure]

Original: The ﬁrst stage of BB-ANS encoding is to pop from the posterior, z 1: L ← q ( · | x ) .
Revised: The ﬁrst stage of BB-ANS encoding is to pop from the posterior, z 1: L ← q ( · | x ) .

Original: When using dynamic discretization, popping the layer z l requires knowledge of the discretization used for z l and thus of the conditional distribution p ( z l | z l +1: L ) .
Revised: When using dynamic discretization, popping the layer z l requires knowledge of the discretization used for z l and thus of the conditional distribution p ( z l | z l +1: L ) .

Original: This requires the latents z l +1: L to have already been popped.
Revised: This requires the latents z l +1: L to have already been popped.

Original: Because of this, latents in general must be popped (sampled) in ‘top-down’ order, i.e. z L ﬁrst, then z L − 1 and so on down to z 1 .
Revised: Because of this, latents in general must be popped (sampled) in ‘top-down’ order, i.e. z L ﬁrst, then z L − 1 and so on down to z 1 .

Original: The most general form of posterior for which top-down sampling is possible is
Revised: The most general form of posterior for which top-down sampling is possible is

Original: [Equation]
Revised: [Equation]

Original: This is illustrated, for a hierarchy of depth 3, in Figure 3b.
Revised: This is illustrated, for a hierarchy of depth 3, in Figure 3b.

Original: The Bit-Swap technique (Kingma et al., 2019) requires that inference be done bottom up, and that generative and inference models must both be a Markov chain on z 1 , . .
Revised: The Bit-Swap technique (Kingma et al., 2019) requires that inference be done bottom up, and that generative and inference models must both be a Markov chain on z 1 , . .

Original: z L , and thus cannot use skip connections.
Revised: z L , and thus cannot use skip connections.

Original: These constraints are illustrated in Figure 3c,d. Skip connections have been shown to improve model ELBO in very deep models (Sønderby et al., 2016;
Revised: These constraints are illustrated in Figure 3c,d. Skip connections have been shown to improve model ELBO in very deep models (Sønderby et al., 2016;

Original: Maaløe et al., 2019).
Revised: Maaløe et al., 2019).

Original: HiLLoC does not have this constraint, and we do utilize skip connections in our experiments.
Revised: HiLLoC does not have this constraint, and we do utilize skip connections in our experiments.

Original: 3. S TARTING THE BITS BACK CHAIN
Revised: 3. S TARTING THE BITS BACK CHAIN

Original: As discussed in Section 3.3, our dynamic discretization method precludes the use of Bit-Swap for reducing the one-time cost of starting a BB-ANS chain.
Revised: As discussed in Section 3.3, our dynamic discretization method precludes the use of Bit-Swap for reducing the one-time cost of starting a BB-ANS chain.

Original: We propose instead to use a signiﬁcantly simpler method to address the high cost of coding a small number of samples with BB-ANS, namely we code the ﬁrst samples using a different codec.
Revised: We propose instead to use a signiﬁcantly simpler method to address the high cost of coding a small number of samples with BB-ANS, namely we code the ﬁrst samples using a different codec.

Original: The purpose of this is to build up a sufﬁciently large buffer of compressed data to permit the ﬁrst stage of the BB-ANS
Revised: The purpose of this is to build up a sufﬁciently large buffer of compressed data to permit the ﬁrst stage of the BB-ANS

Original: algorithm - to pop a latent sample from the posterior.
Revised: algorithm - to pop a latent sample from the posterior.

Original: In our experiments we use the ‘Free Lossless Image Format’ (FLIF) (Sneyers & Wuille, 2016) to build up the buffer.
Revised: In our experiments we use the ‘Free Lossless Image Format’ (FLIF) (Sneyers & Wuille, 2016) to build up the buffer.

Original: We chose this codec because it performed better than other widely used codecs, but in principal any lossless codec could be used.
Revised: We chose this codec because it performed better than other widely used codecs, but in principal any lossless codec could be used.

Original: The amount of previously compressed data required to pop a posterior sample from the ANS stack (and therefore start the BB-ANS chain) is roughly proportional to the size of the image we wish to compress, since in a fully convolutional model the size of the latent space is determined by the image size.
Revised: The amount of previously compressed data required to pop a posterior sample from the ANS stack (and therefore start the BB-ANS chain) is roughly proportional to the size of the image we wish to compress, since in a fully convolutional model the size of the latent space is determined by the image size.

Original: We can exploit this to allow us to obtain a better compression rate than FLIF as quickly as possible.
Revised: We can exploit this to allow us to obtain a better compression rate than FLIF as quickly as possible.

Original: We do so by partitioning the ﬁrst images we wish to compress with HiLLoC into smaller patches.
Revised: We do so by partitioning the ﬁrst images we wish to compress with HiLLoC into smaller patches.

Original: These patches require a smaller data buffer, and thus we can use the superior HiLLoC coding sooner than if we attempted to compress full images.
Revised: These patches require a smaller data buffer, and thus we can use the superior HiLLoC coding sooner than if we attempted to compress full images.

Original: We ﬁnd experimentally that, generally, larger patches have a better coding rate than smaller patches.
Revised: We ﬁnd experimentally that, generally, larger patches have a better coding rate than smaller patches.

Original: Therefore we increase the size of the image patches being compressed with HiLLoC as more images are compressed and the size of the data buffer grows, until we ﬁnally compress full images once the buffer is sufﬁciently large.
Revised: Therefore we increase the size of the image patches being compressed with HiLLoC as more images are compressed and the size of the data buffer grows, until we ﬁnally compress full images once the buffer is sufﬁciently large.

Original: For our experiments on compressing full ImageNet images, we compress 32 × 32 patches, then 64 × 64, then 128 × 128 before switching to coding the full size images directly.
Revised: For our experiments on compressing full ImageNet images, we compress 32 × 32 patches, then 64 × 64, then 128 × 128 before switching to coding the full size images directly.

Original: Note that since our model can compress any shape image, we can compress the edge patches which will have different shape if the patch size does not divide the image dimensions exactly.
Revised: Note that since our model can compress any shape image, we can compress the edge patches which will have different shape if the patch size does not divide the image dimensions exactly.

Original: Using this technique means that our coding rate improves gradually from the FLIF coding rate towards the coding rate achieved by HiLLoC on full images.
Revised: Using this technique means that our coding rate improves gradually from the FLIF coding rate towards the coding rate achieved by HiLLoC on full images.

Original: We compress only 5 ImageNet images using FLIF before we start compressing 32 × 32 patches using
Revised: We compress only 5 ImageNet images using FLIF before we start compressing 32 × 32 patches using

Original: HiLLoC.
Revised: HiLLoC.

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: Using Craystack, we implement HiLLoC with a Resnet VAE (RVAE) (Kingma et al., 2016).
Revised: Using Craystack, we implement HiLLoC with a ResNet VAE (RVAE) (Kingma et al., 2016).

Original: This powerful hierarchical latent variable model achieves ELBOs comparable to state of the art autoregressive models 2 .
Revised: This powerful hierarchical latent variable model achieves ELBOs comparable to state of the art autoregressive models 2 .

Original: In all experiments we used an RVAE with 24 stochastic hidden layers, diagonal
Revised: In all experiments we used an RVAE with 24 stochastic hidden layers.

Original: Gaussian distributions for the conditional prior and posterior distributions on the latents of each layer, and a discretized logistic distribution over observed variables.

Original: The RVAE utilizes skip connections, which are important to be able to effectively train models with such a deep latent hierarchy.
Revised: The RVAE utilizes skip connections, which are important to be able to effectively train models with such a deep latent hierarchy.

Revised: See Appendix E for more details.

Original: We trained the RVAE on the ImageNet 32 training set, then evaluated the RVAE ELBO and HiLLoC compression rate on the ImageNet 32 test set.
Revised: We trained the RVAE on the ImageNet 32 training set, then evaluated the RVAE ELBO and HiLLoC compression rate on the ImageNet 32 test set.

Original: To test generalization, we also evaluated the ELBO and compression rate on the tests sets of ImageNet64, CIFAR10 and full size ImageNet.
Revised: To test generalization, we also evaluated the ELBO and compression rate on the tests sets of ImageNet64, CIFAR10 and full size ImageNet.

Original: For full size
Revised: For full size

Original: ImageNet, we used the partitioning method described in 3.4.
Revised: ImageNet, we used the partitioning method described in 3.4.

Original: The results are shown in Table 2. models, for which decoding time scales with number of pixels, and is in practice extremely slow, both encoding and decoding with RVAEs are fast.
Revised: The results are shown in Table 2. For HiLLoC the compression rates are for the entire test set, except for full ImageNet, where we use 2000 random images from the test set.

Original: [Table]
Revised: [Table]

Revised: Table 2 shows that HiLLoC achieves competitive compression rates on all benchmarks, and state of the art on full size ImageNet images.

Original: The fact that HiLLoC achieves state of the art compression rates relative to the baselines even under a change of distribution is striking [Table]
Revised: The fact that HiLLoC can achieve state of the art compression on ImageNet relative to the baselines, even under a change of distribution, is striking.

Original: and provides strong evidence of its efﬁcacy as a general method for lossless compression of natural images.
Revised: This provides strong evidence of its efﬁcacy as a general method for lossless compression of natural images.

Original: Naively [Table] one might expect a degradation of performance relative to the original test set when changing the test distribution—even more so when the resolution changes.
Revised: Naively, one might expect a degradation of performance relative to the original test set when changing the test distribution—even more so when the resolution changes.

Original: However, in the settings we studied, the opposite was true, in that the average per-pixel ELBO (and thus the compressed message length) was lower on all other datasets compared to the ImageNet 32 validation set.
Revised: However, in the settings we studied, the opposite was true, in that the average per-pixel ELBO (and thus the compressed message length) was lower on all other datasets compared to the ImageNet 32 validation set.

Original: In the case of CIFAR, we conjecture that the reason for this is that its images are simpler and contain more redundancy than ImageNet.
Revised: In the case of CIFAR, we conjecture that the reason for this is that its images are simpler and contain more redundancy than ImageNet.

Original: This theory is backed up by the performance of standard compression algorithms which, as shown in Table 2, also perform better on CIFAR images than they do on ImageNet 32.
Revised: This theory is backed up by the performance of standard compression algorithms which, as shown in Table 2, also perform better on CIFAR images than they do on ImageNet 32.

Original: We ﬁnd the compression rate improvement on larger images more surprising.
Revised: We ﬁnd the compression rate improvement on larger images more surprising.

Original: We hypothesize that this is because pixels at the edge of an image are harder to model because they have less context to reduce uncertainty.
Revised: We hypothesize that this is because pixels at the edge of an image are harder to model because they have less context to reduce uncertainty.

Original: The ratio of edge pixels to interior pixels is lower for larger images, thus we might expect less uncertainty per pixel in a larger image.
Revised: The ratio of edge pixels to interior pixels is lower for larger images, thus we might expect less uncertainty per pixel in a larger image.

Original: [Figure] To demonstrate the effect of vectorization we timed compression of single images at different, ﬁxed, sizes, using a fully vectorized and a fully serial implementation.
Revised: To demonstrate the effect of vectorization we timed ANS of single images at different, ﬁxed, sizes, using a fully vectorized and a fully serial implementation.

Original: The results are shown in Figure 4, which clearly shows a speedup of nearly three orders of magnitude for all image sizes.
Revised: The results are shown in Figure 4, which clearly shows a speedup of nearly three orders of magnitude for all image sizes.

Original: Local bits back (Ho et al., 2019).
Revised: We ﬁnd that the run times for encoding and decoding are roughly linear in the number of pixels, and the time to compress an average sized ImageNet image of 500 × 374 pixels (with vectorized ANS) is around 29s on a desktop computer with 6 CPU cores and a GTX 1060 GPU.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Our experiments demonstrate HiLLoC as a bridge between large scale latent variable models and compression.
Revised: Our experiments demonstrate HiLLoC as a bridge between large scale latent variable models and compression.

Original: To do this we use simple variants of pre-existing VAE models.
Revised: To do this we use simple variants of pre-existing VAE models.

Original: Having shown that bits back coding is ﬂexible enough to compress well with large, complex models, we see plenty of work still to be done in searching model structures (i.e. architecture search), optimizing with a trade-off between compression rate, encode/decode time and memory usage.
Revised: Having shown that bits back coding is ﬂexible enough to compress well with large, complex models, we see plenty of work still to be done in searching model structures (i.e. architecture search), optimizing with a trade-off between compression rate, encode/decode time and memory usage.

Original: Particularly pertinent for HiLLoC  is latent dimensionality, since compute time and memory usage both scale with this.
Revised: Particularly pertinent for HiLLoC For Bit-Swap, size ImageNet images were cropped so that their side lengths were multiples [Figure] is latent dimensionality, since compute time and memory usage both scale with this.

Original: Since the model must be stored/transmitted to use HiLLoC, weight compression is also highly relevant.
Revised: Since the model must be stored/transmitted to use HiLLoC, weight compression is also highly relevant.

Original: This is a well-established research area in machine learning (Han et al., 2016; Ullrich et al., 2017).
Revised: This is a well-established research area in machine learning (Han et al., 2016; Ullrich et al., 2017).

Original: Our experiments also demonstrated that one can achieve good performance on a dataset of large images by training on smaller images.
Revised: Our experiments also demonstrated that one can achieve good performance on a dataset of large images by training on smaller images.

Original: This result is promising, but future work should be done to discover what the best training datasets are for coding generic images.
Revised: This result is promising, but future work should be done to discover what the best training datasets are for coding generic images.

Original: One question in particular is whether results could be improved by training on larger images and/or images of varying size.
Revised: One question in particular is whether results could be improved by training on larger images and/or images of varying size.

Original: We leave this to future work.
Revised: We leave this to future work.

Original: Another related direction for improvement is batch compression of images of different sizes using masking, analogous to how samples of different length may be processed in batches by recurrent neural nets.
Revised: Another related direction for improvement is batch compression of images of different sizes using masking, analogous to how samples of different length may be processed in batches by recurrent neural nets.

Original: Whilst this work has focused on latent variable models, there is also promise in applying state of the art fully observed auto-regressive models to lossless compression.
Revised: Whilst this work has focused on latent variable models, there is also promise in applying state of the art fully observed auto-regressive models to lossless compression.

Original: We look forward to future work investigating the performance of models such as WaveNet (van den Oord et al., 2016) for lossless audio compression as well as PixelCNN++ (Salimans et al., 2017) and the state of the art models in Menick & Kalchbrenner (2019) for images.
Revised: We look forward to future work investigating the performance of models such as WaveNet (van den Oord et al., 2016) for lossless audio compression as well as PixelCNN++ (Salimans et al., 2017) and the state of the art models in Menick & Kalchbrenner (2019) for images.

Original: Sampling speed for these models, and thus decompression, scales with autoregressive sequence length, and can be very slow.
Revised: Sampling speed for these models, and thus decompression, scales with autoregressive sequence length, and can be very slow.

Original: This could be a serious limitation, particularly in common applications where encoding is performed once but decoding is performed many times.
Revised: This could be a serious limitation, particularly in common applications where encoding is performed once but decoding is performed many times.

Original: This effect can be mitigated by using dynamic programming (Le
Revised: This effect can be mitigated by using dynamic programming (Le

Original: Paine et al., 2016;
Revised: Paine et al., 2016;

Original: Ramachandran et al., 2017), and altering model architecture (Reed et al., 2017), but on parallel architectures sampling/decompression is still signiﬁcantly slower than with VAE models.
Revised: Ramachandran et al., 2017), and altering model architecture (Reed et al., 2017), but on parallel architectures sampling/decompression is still signiﬁcantly slower than with VAE models.

Original: On the other hand, fully observed  models  do not require bits back coding, and therefore do not have to pay the one-off cost of starting a chain.
Revised: On the other hand, fully observed models, as well as the ﬂow based models of Hoogeboom et al. (2019) and Ho et al. (2019), do not require bits back coding, and therefore do not have to pay the one-off cost of starting a chain.

Original: Therefore they may be well suited to situations where one or a few i.i.d.
Revised: Therefore they may be well suited to situations where one or a few i.i.d.

Original: samples are to be communicated.
Revised: samples are to be communicated.

Original: Similar to the way that we use FLIF to code the ﬁrst images for our experiments, one could initially code images using a fully observed model then switch to a faster latent variable model once a stack of bits has been built up.
Revised: Similar to the way that we use FLIF to code the ﬁrst images for our experiments, one could initially code images using a fully observed model then switch to a faster latent variable model once a stack of bits has been built up.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented HiLLoC, an extension of BB-ANS to hierarchical latent variable models, and show that
Revised: We presented HiLLoC, an extension of BB-ANS to hierarchical latent variable models, and show that

Original: HiLLoC can perform well with large models.
Revised: HiLLoC can perform well with large models.

Original: We open-sourced our implementation, along with the
Revised: We open-sourced our implementation, along with the

Original: Craystack package for prototyping lossless compression.
Revised: Craystack package for prototyping lossless compression.

Original: We have also explored generalization of large VAE models, and established that fully convolutional
Revised: We have also explored generalization of large VAE models, and established that fully convolutional

Original: VAEs can generalize well to other datasets, including images of very different size to those they were trained on.
Revised: VAEs can generalize well to other datasets, including images of very different size to those they were trained on.

Original: We have described how to compress images of arbitrary size with HiLLoC, achieving a compression rate superior to the best available codecs on ImageNet images.
Revised: We have described how to compress images of arbitrary size with HiLLoC, achieving a compression rate superior to the best available codecs on ImageNet images.

Original: We look forward to future work reuniting machine learning and lossless compression.
Revised: We look forward to future work reuniting machine learning and lossless compression.

Original: A R EPARAMETERIZING DISCRETIZED LATENTS
Revised: A R EPARAMETERIZING DISCRETIZED LATENTS

Original: After discretizing the latent space, the latent variable at layer l can be treated as simply an index i l into one of the intervals created by the discretization.
Revised: After discretizing the latent space, the latent variable at layer l can be treated as simply an index i l into one of the intervals created by the discretization.

Original: As such, we introduce the following notation for pushing and popping according to a discretized version of the posterior.
Revised: As such, we introduce the following notation for pushing and popping according to a discretized version of the posterior.

Original: [Equation]
Revised: [Equation]

Original: Where Q l ( · | i l +1: L , x ) is the distribution over the intervals of the discretized latent space for z l , with interval masses equal to their probability under q ( z l | ˜
Revised: Where Q l ( · | i l +1: L , x ) is the distribution over the intervals of the discretized latent space for z l , with interval masses equal to their probability under q ( z l | ˜

Original: z l +1: L , x ) .
Revised: z l +1: L , x ) .

Original: The discretization is created from splitting the latent space into equal mass intervals under p ( z l | ˜
Revised: The discretization is created from splitting the latent space into equal mass intervals under p ( z l | ˜

Original: z l +1: L ) .
Revised: z l +1: L ) .

Original: The mass of a given interval under some distribution is the CDF at the upper bound of the interval minus the CDF at the lower end of the interval.
Revised: The mass of a given interval under some distribution is the CDF at the upper bound of the interval minus the CDF at the lower end of the interval.

Original: We have used ˜ z to indicate that these will be discrete z l values that are reconstructed from the indices i l .
Revised: We have used ˜ z to indicate that these will be discrete z l values that are reconstructed from the indices i l .

Original: In practise we take ˜ z l ( i l ) to be the centre of the interval indexed by i l .
Revised: In practise we take ˜ z l ( i l ) to be the centre of the interval indexed by i l .

Original: It is important to note that the Q l has an implicit dependence on the previous prior distributions p ( z k | z
Revised: It is important to note that the Q l has an implicit dependence on the previous prior distributions p ( z k | z

Original: k +1: L ) for k ≥ l , as these prior distributions are required to calculate ˜ z
Revised: k +1: L ) for k ≥ l , as these prior distributions are required to calculate ˜ z

Original: l +1:
Revised: l +1:

Original: L and the discretization of the latent space.
Revised: L and the discretization of the latent space.

Original: Since we discretize each latent layer to be intervals of equal mass under the prior, the prior distribution over the indices i l becomes a uniform distribution over the interval indices, U ( i l ) , which is not dependent on i (cid:54)
Revised: Since we discretize each latent layer to be intervals of equal mass under the prior, the prior distribution over the indices i l becomes a uniform distribution over the interval indices, U ( i l ) , which is not dependent on i (cid:54)

Original: =
Revised: =

Original: l .
Revised: l .

Original: Note that this allows us to push/pop the i l according to the prior in parallel.
Revised: Note that this allows us to push/pop the i l according to the prior in parallel.

Original: The full encoding and decoding procedures with a hierarchical latent model and the dynamic discretization we have described are shown in Table 3.
Revised: The full encoding and decoding procedures with a hierarchical latent model and the dynamic discretization we have described are shown in Table 3.

Original: Note that the operations in the two tables are ordered top to bottom.
Revised: Note that the operations in the two tables are ordered top to bottom.

Original: Variables
Revised: Variables

Original: Operation
Revised: Operation

Original: [Equation]
Revised: [Equation]

Original: L , x ) ...
Revised: L , x ) ...

Original: x, i 2: L i←
Revised: x, i 2: L i←

Original: Q 1 ( · | i 2: L , x ) x, i 1: L x → p ( · | ˜ z 1: L ( i 1: L ))
Revised: Q 1 ( · | i 2: L , x ) x, i 1: L x → p ( · | ˜ z 1: L ( i 1: L ))

Original: i 1: L i 1: L →
Revised: i 1: L i 1: L →

Original: U ( · ) (a) Encoding
Revised: U ( · ) (a) Encoding

Original: Operation
Revised: Operation

Original: Variables
Revised: Variables

Original: [Table]
Revised: [Table]

Original: ←
Revised: ←

Original: U
Revised: U

Original: [Table] [Table]
Revised: [Table] [Table]

Original: x ← p ( · |
Revised: x ← p ( · |

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: →
Revised: →

Original: [Table]
Revised: [Table]

Original: ( · |
Revised: ( · |

Original: [Table] [Table]
Revised: [Table] [Table]

Original: →
Revised: →

Original: [Table]
Revised: [Table]

Original: B C ODEC FOR VARIABLE IMAGE SIZES
Revised: B C ODEC FOR VARIABLE IMAGE SIZES

Original: Here we describe a codec to compress a set of images of arbitrary size.
Revised: Here we describe a codec to compress a set of images of arbitrary size.

Original: The encoder now adds the dimensions of the image being coded to the stream of compressed data, such that the decoder knows what shape the image will be before decoding it.
Revised: The encoder now adds the dimensions of the image being coded to the stream of compressed data, such that the decoder knows what shape the image will be before decoding it.

Original: Since we are using a vectorized ANS coder, as described in Section 3.2, we resize the top of the coder in between each coding/decoding step such that the size of the top of the coder matches the sizes of the image and latents being coded.
Revised: Since we are using a vectorized ANS coder, as described in Section 3.2, we resize the top of the coder in between each coding/decoding step such that the size of the top of the coder matches the sizes of the image and latents being coded.

Original: The codec is detailed in Table 4.
Revised: The codec is detailed in Table 4.

Original: To make the resizing procedure efﬁcient, we resize via ‘folding’ the top of the vectorized ANS coder such that we are roughly halving/doubling the number of individual ANS coders each time we fold.
Revised: To make the resizing procedure efﬁcient, we resize via ‘folding’ the top of the vectorized ANS coder such that we are roughly halving/doubling the number of individual ANS coders each time we fold.

Original: This makes the cost of the resize logarithmic with the size difference
Revised: This makes the cost of the resize logarithmic with the size difference

Original: [Table]
Revised: [Table]

Original: C C OMPRESSION WITH P IXEL VAE
Revised: C C OMPRESSION WITH P IXEL VAE

Original: To further demonstrate HiLLoC, we implement it with a PixelVAE model.
Revised: To further demonstrate HiLLoC, we implement it with a PixelVAE model.

Original: We use a model with two latent layers, although the posterior is fully factorized.
Revised: We use a model with two latent layers, although the posterior is fully factorized.

Original: The implementation requires nesting an autoregressive codec inside the BB-ANS codec, since the observations and one of the latent layers in PixelVAE have autoregressive generative distributions.
Revised: The implementation requires nesting an autoregressive codec inside the BB-ANS codec, since the observations and one of the latent layers in PixelVAE have autoregressive generative distributions.

Original: Handling this complexity showcases
Revised: Handling this complexity showcases

Original: Craystack, which was designed to support this kind of composition.
Revised: Craystack, which was designed to support this kind of composition.

Original: It would also have been prohibitively slow to run on the datasets we compress without the vectorized ANS scheme discussed in Section 3.2.
Revised: It would also have been prohibitively slow to run on the datasets we compress without the vectorized ANS scheme discussed in Section 3.2.

Original: The achieved compression rate on the entire ImageNet validation set is displayed in Table 5.
Revised: The achieved compression rate on the entire ImageNet validation set is displayed in Table 5.

Original: The autoregressive component of the PixelVAE generative model leads to an asymmetry between the times required for compression and decompression.
Revised: The autoregressive component of the PixelVAE generative model leads to an asymmetry between the times required for compression and decompression.

Original: Compression with the PixelVAE model is readily parallelizable across pixels, since we already have access to the pixel values we wish to compress and thus also the conditional distributions on each pixel.
Revised: Compression with the PixelVAE model is readily parallelizable across pixels, since we already have access to the pixel values we wish to compress and thus also the conditional distributions on each pixel.

Original: However, decompression (equivalently, sampling) is not parallelizable across pixels, since we must decompress a pixel value in order to give us access to the conditional distribution on the next pixel.
Revised: However, decompression (equivalently, sampling) is not parallelizable across pixels, since we must decompress a pixel value in order to give us access to the conditional distribution on the next pixel.

Original: This means the time complexity of decompression is linear in the number of pixels, making it prohibitively slow for most image sizes.
Revised: This means the time complexity of decompression is linear in the number of pixels, making it prohibitively slow for most image sizes.

Original: [Table]
Revised: [Table]

Original: D V ECTORIZATION WITHOUT OVERHEADS
Revised: D V ECTORIZATION WITHOUT OVERHEADS

Original: To ensure that the compression rate overhead from using vectorization is low, we use a technique from the BitKnit codec (Giesen, 2015).
Revised: To ensure that the compression rate overhead from using vectorization is low, we use a technique from the BitKnit codec (Giesen, 2015).

Original: When we reach the end of encoding, we could simply concatenate the integers in the (vector) stack head to form the ﬁnal output message.
Revised: When we reach the end of encoding, we could simply concatenate the integers in the (vector) stack head to form the ﬁnal output message.

Original: However, this is inefﬁcient because the stack head is not uniformly distributed.
Revised: However, this is inefﬁcient because the stack head is not uniformly distributed.

Original: As discussed in Giesen (2015), elements of the top of the stack have a probability mass roughly
Revised: As discussed in Giesen (2015), elements of the top of the stack have a probability mass roughly

Original: [Equation]
Revised: [Equation]

Original: Equivalently, the length of h is approximately uniformly distributed.
Revised: Equivalently, the length of h is approximately uniformly distributed.

Original: More detailed discussion and an empirical demonstration of this is given by Bloom (2014).
Revised: More detailed discussion and an empirical demonstration of this is given by Bloom (2014).

Original: An efﬁcient way to form the ﬁnal output message at the end of decoding, is to fold the stack head vector by repeatedly encoding half of it onto the other half, until only a scalar remains, using the above distribution for the encoding.
Revised: An efﬁcient way to form the ﬁnal output message at the end of decoding, is to fold the stack head vector by repeatedly encoding half of it onto the other half, until only a scalar remains, using the above distribution for the encoding.

Original: We implement this technique in Craystack and use it for our experiments.
Revised: We implement this technique in Craystack and use it for our experiments.

Original: The number of (vectorized) encode steps required is logarithmic in the size (i.e. the number of elements) of the stack head.
Revised: The number of (vectorized) encode steps required is logarithmic in the size (i.e. the number of elements) of the stack head.

Original: Some of the overhead from vectorization also comes at the start of encoding, when, in existing implementations, the elements of the stack head vector are initialized to copies of a ﬁxed constant.
Revised: Some of the overhead from vectorization also comes at the start of encoding, when, in existing implementations, the elements of the stack head vector are initialized to copies of a ﬁxed constant.

Original: Information from these copies ends up in the message and introduces redundancy which scales with the size of the head.
Revised: Information from these copies ends up in the message and introduces redundancy which scales with the size of the head.

Original: This overhead can be removed by initializing the stack head to a vector of length 1 and then growing the length of the stack head vector gradually as more random data is added to the stack, by decoding new stack head vector elements according to the distribution (9). 
Revised: This overhead can be removed by initializing the stack head to a vector of length 1 and then growing the length of the stack head vector gradually as more random data is added to the stack, by decoding new stack head vector elements according to the distribution (9). E R ES N ET VAE ARCHITECTURE

Revised: A full description of the RVAE architecture is given in Kingma et al.

Revised: (2016), and a full implementation can be found in our repository https://github.com/hilloc-submission/hilloc , but we give a short description below.

Revised: The RVAE is a hierarchical latent model, trained by maximization of the usual evidence lower bound (ELBO) on the log-likelihood:

Revised: [Equation]

Revised: Take the latent hierarchy to be depth L , such that the latents are z 1: L .

Revised: There are skip connections in both the generative model, p ( x, z 1: L ) , and the inference model, q ( z 1: L | x ) .

Revised: Due to our requirement of using dynamic discretization, we use a top-down inference model 7 .

Revised: This means that we can write

Revised: [Equation]

Revised: And the ELBO as

Revised: [Equation]

Revised: [Equation]

Revised: Where D KL is the KL divergence.

Revised: As in Kingma et al.

Revised: (2016), the KL terms are individually clamped as max( D KL , λ ) , where λ is some constant.

Revised: This is an optimization technique known as free bits , and aims to prevent latent layers in the hierarchy collapsing such that the posterior is equal to the prior.

Revised: Each layer in the hierarchy consists of a ResNet block with two sets of activations.

Revised: One set of activations are calculated bottom-up (in the direction of x to z L ), and the other are calculated top-down.

Revised: The bottom-up activations are used only within q ( z 1: L | x ) , whereas the top-down activations are used by both q ( z 1: L | x ) and p ( x, z 1: L ) .

Revised: Every conditional distribution on a latent z l is parameterized as a diagonal Gaussian distribution, with mean and covariance a function of the activations within the ResNet block, and the conditional distribution on x is parameterized by a discretized logistic distribution.

Revised: Given activations for previous ResNet blocks, the activations at the following ResNet block are a combination of stochastic and deterministic features of the previous latent layer, as well as from skip connections directly passing the previous activations.

Revised: The features are calculated by convolutions.

Revised: Note also that all latent layers are the same shape.

Revised: Since we retained the default hyperparameters from the original implementation, each latent layer has 32 feature maps and spatial dimensions half those of the input (e.g. h 2 × w 2 for input of shape h × w ).