Original: Your Dataset is a Multiset and You Should Compress it Like One
Revised: Your Dataset is a Multiset and You Should Compress it Like One

Original: Abstract
Revised: Abstract

Original: Neural Compressors (NCs) are codecs that leverage neural networks and entropycoding to achieve competitive compression performance for images, audio, andother data types.
Revised: Neural Compressors (NCs) are codecs that leverage neural networks and entropy coding to achieve competitive compression performance for images, audio, and other data types.

Original: These compressors exploit parallel hardware, and are particularlywell suited to compressing i.i.d.
Revised: These compressors exploit parallel hardware, and are particularly well suited to compressing i.i.d.

Original: batches of data.
Revised: batches of data.

Original: The average number of bits neededto represent each example is at least the well-known cross-entropy.
Revised: The average number of bits needed to represent each example is at least the well-known cross-entropy.

Original: However, thecross-entropy bound assumes the order of the compressed examples in a batch ispreserved, which in many applications is not necessary.
Revised: However, the cross-entropy bound assumes the order of the compressed examples in a batch is preserved, which in many applications is not necessary.

Original: The amount of bits usedto implicitly store the order information is the logarithm of the number of uniquepermutations of the dataset.
Revised: The number of bits used to implicitly store the order information is the logarithm of the number of unique permutations of the dataset.

Original: In this work, we present a method that allows anycodec to compress below the cross-entropy by exactly the number of bits needed tostore the order, at the expense of shufﬂing the dataset in the process.
Revised: In this work, we present a method that reduces the bitrate of any codec by exactly the number of bits needed to store the order, at the expense of shufﬂing the dataset in the process.

Original: Conceptually,our method applies bits-back coding to a latent variable model with observedsymbol counts (i.e. multiset) and a latent permutation deﬁning the ordering, anddoes not require retraining any models.
Revised: Conceptually, our method applies bits-back coding to a latent variable model with observed symbol counts (i.e. multiset) and a latent permutation deﬁning the ordering, and does not require retraining any models.

Original: Experiments with lossless NCs and lossyoff-the-shelf codecs such as WebP achieve savings of up to 7 .
Revised: We present experiments with both lossy off-the-shelf codecs (WebP) as well as lossless NCs. On Binarized MNIST, lossless NCs achieved savings of up to 7 .

Original: 6% on Binarized MNIST, while adding only 10% extra compute time.
Revised: 6% , while adding only 10% extra compute time.

Original:  Introduction
Revised:  Introduction

Original: [Figure]

Original: A data source is usually modeled as an orderedsequence of random variables with some jointdistribution.
Revised: A data source is usually modeled as an ordered sequence of random variables with some joint distribution.

Original: The objective of lossless data compression is to map each trajectory, i.e. the sequence of instances, to a compact representationsuch as a string of bits.
Revised: The objective of lossless data compression is to map each trajectory, i e. the sequence of instances, to a compact representation such as a string of bits.

Original: The average numberof bits of the smallest lossless representationfor any source is completely characterized byit’s probability distribution, and is known as theentropy .
Revised: The average number of bits of the smallest lossless representation for any source is completely characterized by it’s probability distribution, and is known as the entropy .

Original: Compression algorithms such as Arithmetic Coding (AC) and Asymmetric Numeral 
Revised: Compression algorithms such as Arithmetic Coding (AC) and Asymmetric Numeral Systems (ANS)

Original: Systems (ANS) can compress any sequence to asize very close to the entropy, if the probabilitydistribution is known.
Revised: [3] can compress any sequence to a size very close to the entropy, if the probability distribution is known.

Original: In real-world applications, the source distribution is not known and must be estimated from data.
Revised: In real-world applications, the source distribution is not known and must be estimated from data.

Original: Thebetter the estimate, the closer the size of the representation is to the entropy.
Revised: The better the estimate, the closer the size of the representation is to the entropy.

Original: Deep generative modelsare good estimators for this task, and have recently been shown to reach competitive compressionperformance when paired with AC and ANS [6, 14, 16, 7, 1, 2, 5].
Revised: Deep generative models are good estimators for this task, and have recently been shown to reach competitive compression performance when paired with AC and ANS [7, 15, 17, 8, 1, 2, 6].

Original: The pairings of these models withcompression algorithms are increasingly being referred to as Neural Compressors (NCs) [8].
Revised: The pairings of these models with compression algorithms are increasingly being referred to as Neural Compressors (NCs) [9].

Original: In the compression literature, the elements of an ordered sequence are referred to as symbols .
Revised: In the compression literature, the elements of an ordered sequence are referred to as symbols .

Original: AC and ANS preserve the ordering of symbols in the input sequence.
Revised: AC and ANS preserve the ordering of symbols in the input sequence.

Original: However, there are data types whereorder is not meaningful, such as collections of ﬁles, rows in a database, nodes in a graph, and, notably,datasets in machine learning applications.
Revised: However, there are data types where order is not meaningful, such as collections of ﬁles, rows in a database, nodes in a graph, and, notably, datasets in machine learning applications.

Original: Formally, these are known as multisets : a generalization ofa set that allows for repetition of elements.
Revised: Formally, these are known as multisets : a generalization of a set that allows for repetition of elements.

Original: A multiset can be compressed with NCs by somehowordering the elements and communicating the corresponding ordered sequence, but this is wasteful,because the order contains information, and therefore more bits are used to represent the source thanare truly necessary.
Revised: A multiset can be compressed with NCs by somehow ordering the elements and communicating the corresponding ordered sequence, but this is wasteful, because the order contains information, and therefore more bits are used to represent the source than are truly necessary.

Original: Current entropy coders tailored to multisets of i.i.d.
Revised: Current entropy coders tailored to multisets of i.i.d.

Original: symbols either have a sub-optimal compressionrate or compute time which scales linearly with data dimensionality, making them unsuitable formultisets of images and other high dimensional sources [12, 13, 11].
Revised: symbols either have a sub-optimal compression rate or compute time which scales linearly with data dimensionality, making them unsuitable for multisets of images and other high dimensional sources [13, 14, 12].

Original: To leverage the efﬁciency andcompression performance of existing NCs, while simultaneously compressing a multiset optimally,requires some mechanism that can forget the ordering between symbols during encoding.
Revised: To leverage the efﬁciency and compression performance of existing NCs, while simultaneously compressing a multiset optimally, requires some mechanism that can forget the ordering between symbols during encoding.

Original: In thiswork, we present such a method based on bits-back coding, and the key observation that any modelover sequences can be seen as a latent variable model, where the observed variable is the unorderedmultiset, and the latent variable is a permutation that deﬁnes the order between symbols [17].
Revised: In this work, we present such a method based on bits-back coding, and the key observation that any model over sequences can be seen as a latent variable model, where the observed variable is the unordered multiset, and the latent variable is a permutation that deﬁnes the order between symbols [18].

Original: A moregeneral, detailed exposition of the core multiset compression method is given in [10].
Revised: A more general, detailed exposition of the core multiset compression method is given in [11].

Original:  Background
Revised:  Background

Original: Given access to some model P over sequences x n = ( x 1 , . . .
Revised: Given access to some model P over sequences x n = ( x 1 , . . .

Original: , x n ) of discrete elements, entropycoders are functions that map x n to bit-strings of approximately − log P ( x n ) in length.
Revised: , x n ) of discrete elements, entropy coders are functions that map x n to bit-strings of approximately − log P ( x n ) in length.

Original: Under thei.i.d.
Revised: Under the i.i.d.

Original: assumption x i ∼ P d , the average bit-length − 1 n log P ( x n ) = − 1 n (cid:80) ni
Revised: assumption x i ∼ P d , the average bit-length − 1 n log P ( x n ) = − 1 n (cid:80) ni

Original: =1 log P ( x i ) approachesthe sequence-cross-entropy H ( P d ) + D KL ( P d (cid:107) P ) for large n , where H ( P d ) is the entropy of thedata source.
Revised: =1 log P ( x i ) approaches the sequence-cross-entropy H ( P d ) + D KL ( P d (cid:107) P ) for large n , where H ( P d ) is the entropy of the data source.

Original: Sequence-cross-entropy is the smallest possible average bit-length that can be achievedfor a given model P .
Revised: Sequence-cross-entropy is the smallest possible average bit-length that can be achieved for a given model P .

Original: We say an entropy coder is optimal if it can compress sequences close to thesequence-cross-entropy.
Revised: We say an entropy coder is optimal if it can compress sequences close to the sequence-cross-entropy.

Original: The entropy coder we develop in this paper depends on another well known entropy coder called“asymmetric numeral systems” (ANS) [3].
Revised: The entropy coder we develop in this paper depends on another well known entropy coder called “asymmetric numeral systems” (ANS) [3].

Original: In contrast to Arithmetic Coding, ANS is last-in-ﬁrstout (i.e. stack-like), as the encoded sequence x n is decoded in reverse order x n , . . .
Revised: In contrast to Arithmetic Coding, ANS is last-in-ﬁrstout (i.e. stack-like), as the encoded sequence x n is decoded in reverse order x n , . . .

Original: , x 1 .
Revised: , x 1 .

Original: ANSencodes sequences into a single, large, natural number s .
Revised: ANS encodes sequences into a single, large, natural number s .

Original: The outputted bit-string is just the binaryrepresentation of s , which is log s bits in size.
Revised: The outputted bit-string is just the binary representation of s , which is log s bits in size.

Original: Each encoded symbol x i increases the bit-string byapproximately − log P ( x i ) , and decreases it by the same amount during decoding.
Revised: Each encoded symbol x i increases the bit-string by approximately − log P ( x i ) , and decreases it by the same amount during decoding.

Original: A key property of ANS is that performing a decode operation using distribution P will producesamples distributed according to P .
Revised: A key property of ANS is that performing a decode operation using distribution P will produce samples distributed according to P .

Original: Decoding reduces the ANS state, which can be viewed as arandom seed that is slowly consumed as samples are drawn.
Revised: Decoding reduces the ANS state, which can be viewed as a random seed that is slowly consumed as samples are drawn.

Original: The random seed can be recoveredby performing an encode with the sampled symbol.
Revised: The random seed can be recovered by performing an encode with the sampled symbol.

Original: Therefore, ANS can be used as an invertiblesampler .
Revised: Therefore, ANS can be used as an invertible sampler .

Original: Invertible sampling is also possible with other entropy coders and is commonly known as‘bits-back coding’ [4].
Revised: Invertible sampling is also possible with other entropy coders and is commonly known as ‘bits-back coding’ [4].

Original: However, the stack-like nature of ANS allows for interleaving of sampling(decoding) and encoding with different distributions.
Revised: However, the stack-like nature of ANS allows for interleaving of sampling (decoding) and encoding with different distributions.

Original: This observation was ﬁrst made by [14], andused to compress images with latent variable models .
Revised: This observation was ﬁrst made by [15], and used to compress images with latent variable models .

Original: Follow-up works propose elaborations on theoriginal idea for speciﬁc classes of latent variable models [7, 16, 15].
Revised: Follow-up works propose elaborations on the original idea for speciﬁc classes of latent variable models [8, 17, 16].

Original:  Method
Revised:  Method

Original: In this section we present a method that converts an entropy coder for sequences x n of i.i.d.
Revised: In this section we present a method that converts an entropy coder for sequences x n of i.i.d.

Original: symbolsinto one for multisets M of i.i.d symbols.
Revised: symbols into one for multisets M of i.i.d symbols.

Original: We describe only the compression scheme, as decompression follows from all steps being invertible.
Revised: We describe only the compression scheme, as decompression follows from all steps being invertible.

Original: The multiset-cross-entropy achieved by compressing Mis at most 1 n log n !
Revised: The multiset-cross-entropy achieved by compressing M is at most 1 n log n !

Original: ≈ log n bits less than the usual sequence-cross-entropy resulting from compressingx n , with equality when there are no repeated symbols.
Revised: ≈ log n bits less than the usual sequence-cross-entropy resulting from compressing x n , with equality when there are no repeated symbols.

Original: This is exactly the information requiredto represent the ordering.
Revised: This is exactly the information required to represent the ordering.

Original: Our method is optimal in the sense that it can recover all 1 n log n !
Revised: Our method is optimal in the sense that it can recover all 1 n log n !

Original: bits,irrespective of how well the original sequence codec performs on x n .
Revised: bits, irrespective of how well the original sequence codec performs on x n .

Original: The naive implementationdiscussed next can be seen as a McBits [9] entropy coder with an exact posterior, where the jointdistribution is the entropy coder chosen for conversion.
Revised: The naive implementation discussed next can be seen as a McBits [10] entropy coder with an exact posterior, where the joint distribution is the entropy coder chosen for conversion.

Original: Key to our method is that any model over sequences can be seen as a latent variable model overobserved multisets (i.e. symbol counts) and latent permutations deﬁning the ordering [17].
Revised: Key to our method is that any model over sequences can be seen as a latent variable model over observed multisets (i.e. symbol counts) and latent permutations deﬁning the ordering [18].

Original: To see how, let [ x n ] represent the set of all possible permutations of x n .
Revised: To see how, let [ x n ] represent the set of all possible permutations of x n .

Original: All sequences in this set havethe same frequency count of symbols, and therefore [ x n ] has a one-to-one correspondence with aunique multiset M .
Revised: All sequences in this set have the same frequency count of symbols, and therefore [ x n ] has a one-to-one correspondence with a unique multiset M .

Original: The amount of permutations | [ x n ] | ≤ n ! is equal to the multinomial coefﬁcient91 calculated from symbol counts in
Revised: The amount of permutations | [ x n ] | ≤ n ! is equal to the multinomial coefﬁcient calculated from symbol counts in

Original: M , with equality when there are no repeated symbols.
Revised: M , with equality when there are no repeated symbols.

Original: If thesymbols x i are i.i.d.
Revised: If the symbols x i are i.i.d.

Original: under model P , then all sequences in [ x n ] have mass P ( x n ) , implying
Revised: under model P , then all sequences in [ x n ] have mass P ( x n ) , implying

Original: [Equation]
Revised: [Equation]

Original: A source that generates sequences x n of i.i.d.
Revised: A source that generates sequences x n of i.i.d.

Original: symbols can thus be seen as a compound source thatﬁrst selects a multiset M and then picks an order uniformly at random from the corresponding set[ x n ] .
Revised: symbols can thus be seen as a compound source that ﬁrst selects a multiset M and then picks an order uniformly at random from the corresponding set [ x n ] .

Original: If σ represents a positive integer that indexes into | [ x n ] | , then the joint and posterior distributionsunder model P are
Revised: If σ represents a positive integer that indexes into | [ x n ] | , then the joint and posterior distributions under model P are

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The distribution P ( M ) does not factorize over symbols,making it inefﬁcient for entropy coding.
Revised: The distribution P ( M ) does not factorize over symbols, making it inefﬁcient for entropy coding.

Original: For example, wecan compress M by always picking ˜ x n ∈ | [ x n ] | such that˜ x 1 ≤ ˜ x 2 ≤ · · · ≤ ˜ x n , together with P (˜ x i | ˜ x
Revised: For example, we can compress M by always picking ˜ x n ∈ | [ x n ] | such that ˜ x 1 ≤ ˜ x 2 ≤ · · · ≤ ˜ x n , together with P (˜ x i | ˜ x

Original: i − 1 ) .
Revised: i − 1 ) .

Original: This101 requires updating the mass for all symbols a ≥ ˜ x i in the102 alphabet at each step i .
Revised: This requires updating the mass for all symbols a ≥ ˜ x i in the alphabet at each step i .

Original: In contrast, P ( x n ) factorizes overx i and hence does not require updates, but uses exactlylog | [ x n ] | surplus bits.
Revised: In contrast, P ( x n ) factorizes over x i and hence does not require updates, but uses exactly log | [ x n ] | surplus bits.

Original: By using invertible sampling torandomly pick any sequence in | [ x n ] | , we can compressefﬁciently with P ( x n ) while removing exactly the surplusbits from the ANS state.
Revised: By using invertible sampling to randomly pick any sequence in | [ x n ] | , we can compress efﬁciently with P ( x n ) while removing exactly the surplus bits from the ANS state.

Original: Equivalently, we augment themultiset to a sequence by pairing it with an order [9], andthen recover the bits using bits-back (Figure 1).
Revised: Equivalently, we augment the multiset to a sequence by pairing it with an order [10], and then recover the bits using bits-back (Figure 1).

Original: First,an ANS state s is initialized and invertible sampling (decoding) is performed with P ( σ | M ) to sample an index112 σ .
Revised: First, an ANS state s is initialized and invertible sampling (decoding) is performed with P ( σ | M ) to sample an index σ .

Original: The sequence ˜ x n ∈ [ x n ] corresponding to σ is thenencoded with P (˜ x n ) =
Revised: The sequence ˜ x n ∈ [ x n ] corresponding to σ is then encoded with P (˜ x n ) =

Original: P ( M , σ ) .
Revised: P ( M , σ ) .

Original: Sampling reduces s ,while encoding increases it.
Revised: Sampling reduces s , while encoding increases it.

Original: Overall, the bit-length of sincreases by exactly the number of bits required to represent M
Revised: Overall, the bit-length of s increases by exactly the number of bits required to represent M

Original: [Equation]
Revised: [Equation]

Original: The scheme as described above would only be optimal when amortizing over a collection of multisets,due to the log s initial bits required to initialize the ANS state s .
Revised: The scheme as described above would only be optimal when amortizing over a collection of multisets, due to the log s initial bits required to initialize the ANS state s .

Original: It is possible to compress asingle multiset while still achieving − log P ( M ) via an incremental version of our method thatamortizes over symbols.
Revised: It is possible to compress a single multiset while still achieving − log P ( M ) via an incremental version of our method that amortizes over symbols.

Original: At each step, a symbol is sampled without replacement from the multisetand immediately encoded with P .
Revised: At each step, a symbol is sampled without replacement from the multiset and immediately encoded with P .

Original: Sampling is performed using ANS with distribution S M , whichassigns probabilities proportional to the frequency counts in M .
Revised: Sampling is performed using ANS with distribution S M , which assigns probabilities proportional to the frequency counts in M .

Original: This is shown in Figure 2 fora multiset of size 2.
Revised: This is shown in Figure 2 for a multiset of size 2.

Original: The sequence ˜ x n generated from the invertible sampling steps  is a randompermutation of x n .
Revised: The sequence ˜ x n generated from the invertible sampling steps (see Section 2) is a random permutation of x n .

Original: Therefore, the order information between symbols in x n has been destroyed, asonly ˜ x n is stored.
Revised: Therefore, the order information between symbols in x n has been destroyed, as only ˜ x n is stored.

Original: This implies our method is optimal, since it removes the maximum amount ofredundant order information ( log | [ x n ] | ), and achieves the multiset-cross-entropy if P achieves thesequence-cross-entropy.
Revised: This implies our method is optimal, since it removes the maximum amount of redundant order information ( log | [ x n ] | ), and achieves the multiset-cross-entropy if P achieves the sequence-cross-entropy.

Original: Sampling without replacement from the multiset can be done efﬁciently by using a binary searchtree (BST).
Revised: Sampling without replacement from the multiset can be done efﬁciently by using a binary search tree (BST), which we describe in detail in [11].

Original: Traversing the BST requires performing comparison operations between symbols.
Revised: Traversing the BST requires comparing symbols under a ﬁxed (usually lexicographic) ordering.

Original: Inthe worst-case, each comparison requires O (log | A  |) bit-wise operations, where A is the alphabet, 130 although average time may have no dependence on | A | .
Revised: In the extreme case where symbols in some alphabet A are represented by binary strings of length log | A | , a single comparison between symbols would require log | A | bit-wise operations.

Revised: However, ‘short-circuit’ evaluations, where the next bits are compared only if all previous comparisons result in equality, make this exponentially unlikely.

Original: All operations required have worst-case131 and average complexity equivalent to that of a search on a balanced BST.
Revised: All operations required have worst-case and average complexity equivalent to that of a search on a balanced BST.

Original: Overall, our method adds O ( n log m ) in average compute time to the original sequence codec, where n and m are the total andunique number of symbols in the multiset.
Revised: Overall, our method adds O ( n log m ) in average compute time to the original sequence codec, where n and m are the total and unique number of symbols in the multiset.

Original:  Experiments
Revised:  Experiments

Original: In the experiments that follow, we used the ANS implementation in the Craystack library [16].
Revised: In the experiments that follow, we used the ANS implementation in the Craystack library [17].

Original: Detailsof each experiment are discussed further in the appendix.
Revised: Details of each experiment are discussed further in the appendix.

Original: [Table] Toy multisets are compressed  to provide evidence of the computational complexity and optimal compression rate of the method.
Revised: Toy multisets We compressed synthetically generated toy multisets to provide evidence of the computational complexity and optimal compression rate of the method.

Original: The alphabet A is always a subset of N .
Revised: The alphabet A is always a subset of N .

Original: For each run, we generate amultiset with m = 512 unique and |M| = n total symbols.
Revised: For each run, we generate a multiset with m = 512 unique and |M| = n total symbols.

Original: The true data distribution is used for P ,140 which is sampled from a Dirichlet prior with coefﬁcients α k = k for k = 1 , . .
Revised: The true data distribution is used for P , which is sampled from a Dirichlet prior with coefﬁcients α k = k for k = 1 , . .

Original: , | A 
Revised: , | A | .

Original: Results are141 shown in Figure 4 averaged over 20 runs with shaded regions  the % to 1%  intervals.
Revised: Results are shown in Section 4 averaged over 20 runs, with shaded regions representing the 99% to 1% conﬁdence intervals.

Original: The new codec compresses a multiset close to it’s  ( − log  varying  in the top plot
Revised: The new codec compresses a multiset close to it’s information content ( − log P ( M )) for varying alphabet sizes, as can be seen in the top plot.

Original: The unaffectedby [Table] [Figure] [Table]
Revised: The total encode plus decode time is unaffected by the alphabet size | A | .

Original: [Figure] [Table] [Figure] [Table]
Revised: The total time scales linearly with the multiset size |M| (bottom plot), as expected.

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] Lossless Neural Compression on Binarized MNIST wasperformed using ’Bits-back with ANS’ (BB-ANS) 
Revised: Lossless Neural Compression on Binarized MNIST Here, a lossless compression algorithm called ’Bits-back with ANS’ (BB-ANS) [15] was used to compressed Binarized MNIST.

Original: [14] as theentropy coder for sequences.

Original: BB-ANS is an entropy coder for latent variable models with factorized joint P ( x z ) =
Revised: BB-ANS is an entropy coder for latent variable models with factorized joint P ( x, z ) =

Original: P ( z ) P ( x | z )  and approximate posterior Q ( z | x )  observed x and latent z .
Revised: P ( z ) P ( x | z ) , and approximate posterior Q ( z | x ) , over observed x and latent z .

Original: The observed model P ( x | z ) is composedof an independent Bernoulli distribution over each pixel, whilethe approximate posterior Q is a factorized Gaussian.
Revised: The observed model P ( x | z ) is composed of a factorized Bernoulli distribution over each pixel, while the approximate posterior Q is a factorized Gaussian.

Original: The average bit-length achieved by BB-ANS is an upper-bound onthesequence-cross-entropy called the Negative Evidence Lower
Revised: The average bit-length achieved by BB-ANS is an upper-bound on the sequence-cross-entropy.

Original:  Bound , with equality when the approximate posterior perfectlymatches the true posterior P ( z | x ) .
Revised: It is equal to the Negative Evidence Lower Bound of the discretized model, with equality when the approximate posterior perfectly matches the true posterior P ( z | x ) .

Original: We used the pre-trainedmodel and code made publicly available by the author 1 .
Revised: We used the pre-trained model and code made publicly available by the author 2 .

Original: First,invertible sampling is performed to select a binarized MNISTimage for compression.
Revised: First, invertible sampling is performed to select a binarized MNIST image for compression.

Original: Then, BB-ANS is applied as describedin the experimental section of [16].
Revised: Then, BB-ANS is applied as described in the experimental section of [17].

Original: This process is repeated,until all 10 , 000 images are compressed.
Revised: This process is repeated, until all 10 , 000 images are compressed.

Original: We compared the average bit-length with and without the invertible sampling step.
Revised: We compared the average bit-length with and without the invertible sampling step.

Original: Results are shown in Table 1.
Revised: Results are shown in Table 1.

Original: Since the images are all unique,the maximum theoretical savings is log(10 , 000!)bits
Revised: Since the images are all unique, the maximum theoretical savings is log(10 , 000!)

Original:  ≈ 14 kB. This represents a potential savings of 7 . 6% , which is achievedby our method, at the cost of 10% extra compute time on average. 
Revised: bits ≈ 14 kB. This represents a potential savings of 7 . 6% , which is achieved by our method, at the cost of 10% extra compute time on average. [Table]

Original: Compressing  MNIST with lossy WebP was performed toshowcase our method with off-the-shelf codecs.
Revised: Compressing standard MNIST with lossy WebP We showcase our method with WebP, an offthe-shelf lossy codec, on standard MNIST.

Original: WebP is a lossycompression algorithm that outputs a variable-length sequenceof bytes which we call a byte-array .
Revised: WebP is a lossy compression algorithm that outputs a variable-length sequence of bytes which we call a byte-array .

Original: To encode, we ﬁrst performinvertible sampling to select an image from the test set.
Revised: To encode, we ﬁrst perform invertible sampling to select an image from the test set.

Original: WebP isapplied to the image, and the bytes in the byte-array are encodedsequentially into the ANS state using a uniform distribution.
Revised: WebP is applied to the image, and the bytes in the byte-array are encoded sequentially into the ANS state using a uniform distribution.

Original: Finally, because the byte-array is variable in length, the lengthitself must also be encoded.
Revised: Finally, because the byte-array is variable in length, the length itself must also be encoded.

Original: As in the BB-ANS experiment, all 10 , 000 images are compressed.
Revised: As in the BB-ANS experiment, all 10 , 000 images are compressed.

Original: Results are shown in Table 1.
Revised: Results are shown in Table 1.

Original: The byte-arrays are all unique and therefore the maximum theoreticalsavings in raw bits is also log(10 , 000!) 
Revised: The byte-arrays are all unique and therefore the maximum theoretical savings in raw bits is also log(10 , 000!) bits

Original: bits ≈ 14 kB. This represents a potential savings of 1 . 5% ,which is achieved by our method, at the cost of 37% extra compute time on average.  
Revised: ≈ 14 kB. This represents a potential savings of 1 . 5% , which is achieved by our method, at the cost of 37% extra compute time on average.  Acknowledgements

Original: Appendix
Revised: In preparing this research, authors afﬁliated to the Vector Institute were funded, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We would like to thank Tim Vieira, whose blog post “Heaps for incremental computation” inspired the binary search tree data structure 3 .

Revised: All plots were made using the Science

Original:  In this section we present experiments on synthetically generated multisets with known sourcedistribution, multisets of grayscale images with lossy codecs (MNIST), and collections of JSON mapsrepresented as a multiset of multisets.
Revised: Plots package [5].  Appendix In this section we present experiments on synthetically generated multisets with known source distribution, multisets of grayscale images with lossy codecs (MNIST), and collections of JSON maps represented as a multiset of multisets.

Original: We used the ANS implementation in the Craystack library [16]for all experiments.
Revised: We used the ANS implementation in the Craystack library [17] for all experiments.

Original: 5.1 Synthetic multisets
Revised: 6.1 Synthetic multisets

Original: Here, synthetically generated multisets are compressed to provide evidence of the computationalcomplexity and optimal compression rate of the method.
Revised: Here, synthetically generated multisets are compressed to provide evidence of the computational complexity and optimal compression rate of the method.

Original: We grow the alphabet size | A | while226 sampling from the source in a way that guarantees a ﬁxed number of unique symbols M = 512 .
Revised: We grow the alphabet size | A | while sampling from the source in a way that guarantees a ﬁxed number of unique symbols m = 512 .

Original: Thealphabet A is always a subset of N .
Revised: The alphabet A is always a subset of N .

Original: For each run, we generate a multiset with M = 512 unique symbols, and use a skewed distribution,sampled from a Dirichlet prior with coefﬁcients α k = k for k = 1 , . .
Revised: For each run, we generate a multiset with m = 512 unique symbols, and use a skewed distribution, sampled from a Dirichlet prior with coefﬁcients α k = k for k = 1 , . .

Original: , | A | , as the true datadistribution D .
Revised: , | A | , as the true data distribution D .

Original: The ﬁnal compressed size of the multiset and the information content (i.e. Shannon lower bound)assuming the distribution D is used, are shown in Figure 4 for different settings of |A| , alongside the233 total encode plus decode time.
Revised: The ﬁnal compressed size of the multiset and the information content (i.e. Shannon lower bound) assuming the distribution D is used, are shown in Section 4 for different settings of |A| , alongside the total encode plus decode time.

Original: Results are averaged over 20 runs, with shaded regions representingthe 99% to 1% conﬁdence intervals.
Revised: Results are averaged over 20 runs, with shaded regions representing the 99% to 1% conﬁdence intervals.

Original: In general, the new codec compresses a multiset close to it’sinformation content for varying alphabet sizes, as can be seen in the left plot.
Revised: In general, the new codec compresses a multiset close to it’s information content for varying alphabet sizes, as can be seen in the left plot.

Original: The total encode plus decode time is unaffected by the alphabet size | A | .
Revised: The total encode plus decode time is unaffected by the alphabet size | A | .

Original: As discussed previously,237 the overall complexity depends on that of coding under the chosen codec used with distribution D .
Revised: As discussed previously, the overall complexity depends on that of coding under the chosen codec used with distribution D .

Original: Here, the codec does include a logarithmic time binary search over | A | , but this is implemented239 efﬁciently and the alphabet size can be seen to have little effect on overall time.
Revised: Here, the codec does include a logarithmic time binary search over | A | , but this is implemented efﬁciently and the alphabet size can be seen to have little effect on overall time.

Original: The total time scaleslinearly with the multiset size |M| (right plot), as expected.[Figure]
Revised: The total time scales linearly with the multiset size |M| (right plot), as expected.

Original: 5.2 MNIST with lossy WebP
Revised: 6.2 MNIST with lossy WebP

Original: We implemented compression of a multiset M of grayscale images using the lossy WebP codec.
Revised: We implemented compression of a multiset M of grayscale images using the lossy WebP codec.

Original: Wetested on the MNIST test set, which is composed of |M| = 10 , 000 distinct grayscale images of244 handwritten digits, each× 28 in size.
Revised: We tested on the MNIST test set, which is composed of |M| = 10 , 000 distinct grayscale images of handwritten digits, each× 28 in size.

Original: To encode the multiset, we perform the sampling procedure toselect an image to compress, as usual.
Revised: To encode the multiset, we perform the sampling procedure to select an image to compress, as usual.

Original: The output of WebP is a preﬁx-free, variable-length sequenceof bytes, which we encoded into the ANS state via a sequence of encode steps with a uniformdistribution.
Revised: The output of WebP is a preﬁx-free, variable-length sequence of bytes, which we encoded into the ANS state via a sequence of encode steps with a uniform distribution.

Original: It is also possible (and faster) to move the WebP output directly into the lower-order bitsof the ANS state.
Revised: It is also possible (and faster) to move the WebP output directly into the lower-order bits of the ANS state.

Original: We compared the ﬁnal compressed length with and without the sampling step.
Revised: We compared the ﬁnal compressed length with and without the sampling step.

Original: In other words, treatingthe dataset as a multiset and treating it as an ordered sequence.
Revised: In other words, treating the dataset as a multiset and treating it as an ordered sequence.

Original: The savings achieved by using our method are shown in ﬁg. 5.
Revised: The savings achieved by using our method are shown in Figure 4.

Original: The theoretical limit shown in the left plot is log |M| !
Revised: The theoretical limit shown in the left plot is log |M| !

Original: , while in the rightplot this quantity is divided by the number of bits needed to compress the data sequentially.
Revised: , while in the right plot this quantity is divided by the number of bits needed to compress the data sequentially.

Original: [Figure]

Original: Note that the maximum savings per symbol − 1 |M| log 2 |M| !
Revised: Note that the maximum savings per symbol − 1 |M| log 2 |M| !

Original: ≈ log 2 |M| depends only on the sizeof multiset.
Revised: ≈ log 2 |M| depends only on the size of multiset.

Original: Therefore, when the representation of the symbol requires a large number of bits,the percentage savings are marginal (roughly 1.5% for 10,000 images, in our case).
Revised: Therefore, when the representation of the symbol requires a large number of bits, the percentage savings are marginal (roughly 1.5% for 10,000 images, in our case).

Original: To improvepercentage savings ( ), one could use a better symbol codec or an adaptive codec whichdoesn’t treat the symbols as independent.
Revised: To improve percentage savings (right plot), one could use a better symbol codec or an adaptive codec which doesn’t treat the symbols as independent.

Original: However, as mentioned, the savings in raw bits (left plot)would remain the same, as it depends only on the multiset size |M| .
Revised: However, as mentioned, the savings in raw bits (left plot) would remain the same, as it depends only on the multiset size |M| .

Original: 5.3 Collection of JSON maps as nested multisets
Revised: 6.3 Collection of JSON maps as nested multisets

Original: The method can be nested to compress a multiset of multisets, by performing an additional samplingstep that ﬁrst chooses which inner multiset to compress.
Revised: The method can be nested to compress a multiset of multisets, by performing an additional sampling step that ﬁrst chooses which inner multiset to compress.

Original: In this section we show results for a collectionof JSON maps M =
Revised: In this section we show results for a collection of JSON maps M =

Original: {J 1 , . . .
Revised: {J 1 , . . .

Original: , J |M| } , where each map J i =
Revised: , J |M| } , where each map J i =

Original: { ( k 1 , v 1 ) , . .
Revised: { ( k 1 , v 1 ) , . .

Original: , ( k |J i | , v |J i | ) } is itself263 a multiset of key-value pairs.
Revised: , ( k |J i | , v |J i | ) } is itself a multiset of key-value pairs.

Original: To compress, a depth-ﬁrst approach is taken.
Revised: To compress, a depth-ﬁrst approach is taken.

Original: First, some
Revised: First, some

Original: J ∈ M is264 sampled without replacement.
Revised: J ∈ M is sampled without replacement.

Original: Key-value pairs are then sampled from J , also without replacement,and compressed to the ANS state until J is depleted.
Revised: Key-value pairs are then sampled from J , also without replacement, and compressed to the ANS state until J is depleted.

Original: This procedure repeats, until the outer multiset
Revised: This procedure repeats, until the outer multiset

Revised: [Figure]

Original: M is empty.
Revised: M is empty.

Original: Assuming all maps are unique, the maximum number of savable bits is
Revised: Assuming all maps are unique, the maximum number of savable bits is

Original: [Equation]
Revised: [Equation]

Original: The collection of JSON maps is composed of public GitHub user data taken from a release of the
Revised: The collection of JSON maps is composed of public GitHub user data taken from a release of the

Original: Zstandard project 2 .
Revised: Zstandard project 4 .

Original: All key-value elements were cast to strings, for simplicity, and are encoded as
Revised: All key-value elements were cast to strings, for simplicity, and are encoded as

Original: UTF-8 bytes using a uniform distribtuion.
Revised: UTF-8 bytes using a uniform distribtuion.

Original: Figure 5 shows the number and percentage of saved bits.
Revised: Figure 4 shows the number and percentage of saved bits.

Original: The theoretical limit curve shows the maximum savable bits with nesting, i.e. eq.
Revised: The theoretical limit curve shows the maximum savable bits with nesting, i.e. eq.

Original: Note that,without nesting, the theoretical limit would be that of MNIST (i.e. log |M| ! ).
Revised: Note that, without nesting, the theoretical limit would be that of MNIST (i.e. log |M| ! ).

Original: The method gets veryclose to the maximum possible savings, for various numbers of JSON maps.
Revised: The method gets very close to the maximum possible savings, for various numbers of JSON maps.

Original: The rate savings weresmall, but these could be improved by using a better technique to encode the UTF-8 strings.
Revised: The rate savings were small, but these could be improved by using a better technique to encode the UTF-8 strings.

Original: Assuming J represents the JSON map in M with the largest number of key-value pairs, and thatthe time complexity of comparing two JSON maps is O ( |J | ) , the complexity of the four BST276 operations for the outer multiset is
Revised: Assuming J represents the JSON map in M with the largest number of key-value pairs, and that the time complexity of comparing two JSON maps is O ( |J | ) , the complexity of the four BST operations for the outer multiset is

Original: O ( |M||J | log |M| ) .
Revised: [Equation]

Original: The overall expected time complexity for277 both encoding and decoding is therefore
Revised: The overall expected time complexity for both encoding and decoding is therefore

Original: O ( |M||J | (log |M| + log |J | )) .
Revised: O ( |M||J | (log |M| + log |J | )) .

Original: We believe it may be278 possible to reduce this by performing the inner sampling steps in parallel, or by speeding up the
Revised: We believe it may be possible to reduce this by performing the inner sampling steps in parallel, or by speeding up the

Original: JSON map comparisons.
Revised: JSON map comparisons.