Paper: Entropy Coding of Unordered Data Structures (ICLR-2024)

Reviewer comment (Novelty):
An alternative to the bit-swap approach mentioned by authors is correlation communication [Harsha 2010, The Communication Complexity of Correlation] [Li 2018, Strong Functional Representation Lemma and Applications to Coding Theorems] [Theis 2021, Algorithms for the Communication of Samples], which has extra overhead of approximately $\log \log N!$, but does not require any initial bits.

Author response:
While correlation communication methods could theoretically be an alternative to the future work of interleaving, these methods have computational complexity which scale exponentially with the entropy of the source, making them impractical. However, the reviewer is correct that these methods could in theory be used to avoid the initial bits problem, if we disregard computational complexity.

These methods relate to bits-back coding in the following way. Consider the source $X$ and another random variable $Y$ from which we can recover $X$ deterministically, i.e., $X = f(Y)$ for some function $f$. Note this implies that $H(X|Y) = 0$. As a concrete example from BB-ANS, $Y = (Z, X)$ where $Z$ is the latent variable of the VAE. Then, bits-back coding implements rate $$R = H(Y) - H(Y|X) = I(X; Y) = H(X) - H(X|Y) = H(X),$$ where $-H(Y|X)$ is the savings from the bits-back step. 

Correlation communication methods can be used to communicate a sample Y | X, achieving a rate of $R + log(R+1) + 5$ via the Poisson Functional Representation Lemma (in this case, the discrete version which falls back onto the exponential races) [Li & El Gammal 2017], and X can then be recovered from that sample. Unfortunately, as mentioned, the computational complexity would be much larger than our method. 

Bits-back can be viewed as an extremely efficient algorithm for “deterministic” correlation communication (i.e., when $X = f(Y)$ for some $f$).

Reviewer comment (Evaluation):
For practically large structure, what is the computational cost for finding and sampling from the isomorphism class $\hat{f}$? For practical codec, the metrics such as latency and throughput should also be discussed.

Author response:
We provided relative timings for nauty, the library we used to find the canonical labeling and automorphism group (these are the computational bottlenecks for sampling from the isomorphism class), in Appendix E. As mentioned in the paper, no polynomial-time algorithm for the graph isomorphism is known, but nauty and Traces solve this problem efficiently for various graph classes. Thorough benchmarks of nauty and traces on different graph types and sizes can be found at https://pallini.di.uniroma1.it/.

---

We have updated Appendix E to now report compression throughput. We believe that latency is more appropriate for streaming applications, such as video decoding. We would envision our codec being used in a non-streaming setting, where a whole file would be downloaded and then decompressed.

Reviewer comment (Reproducibility):
Initial bit overhead: Due to the requirement of initial bits, shuffle coding introduces a communication overhead to the transmitted message. This can be significant in the one-shot or few-shot case, making the method impractical for compressing small sequences of messages. The authors should specify the amount of extra bits used in the experiments (see Table 3) in the main text.

Author response:
**Initial bit overhead:** The average initial bit cost per TU dataset in Table 3 is 0.01 bits per edge for both ER and PU, demonstrating good amortization. We added this result to section 5.

Reviewer comment (Theory):
Contribution as a general solution seems somewhat exaggerated: While the shuffle coding approach is general from a theoretical point of view, it requires access to a "canonicalisation" function. The authors focus on graphs, for which existing libraries offering this functionality exist. For other permutable classes, the authors argue that one can "embed objects into graphs in such a way that the structure is preserved and the canonization remains valid." However, it is unclear whether this embedding might affect the compression rate, or how difficult it is to construct such an embedding. In light of this point, the statement that the authors' "implementation can easily be adapted to different data types" in the abstract may be regarded as exaggerated (if, for example, coming up with an algorithm that constructs such graph embeddings is challenging, or if the computational/memory complexity of such an algorithm is large). A more measured statement in the appendix and/or positioning in the main text might be beneficial.

Author response:
**Contribution as a general solution seems somewhat exaggerated:** We updated section 2.1.1 to clarify that our method critically depends on the availability of function to retrieve a canonical ordering (as well as a set of generators of the automorphism group) for elements of the permutable class. As mentioned in Anders and Schweitzer (2021), embedding into vertex-colored graphs, and then running nauty/traces, is the standard technique for computing the automorphism group (and canonization) of other objects. It would be great though to prove that this is always possible, and to have a general embedding method. We actually used an embedding of edge-colored graphs into vertex-colored graphs in order to canonize and compress graphs with edge attributes (which are not directly supported by nauty/traces) for our experiments. We have updated the paper to mention that we did this.

Reviewer comment (Evaluation):
Clarification on the discount factors for Table 3: The ER and PU models are specific probabilistic models for graphs and, as the authors explain, they can be swapped in with any other exchangeable model of graphs. Therefore, while it is important to report the overall communication cost in the experiments, an equally important quantity to report is the rate discount afforded by shuffle coding. What are the discounts corresponding to the results in table 3? Perhaps the authors can report these as an extra column, since the discount is a function of the graph alone and not the modelling distribution \(P\).

Author response:
**Clarification of the discount factors for Table 3:** We added a column to report the discounts for all datasets in Table 3. As in Table 2, the discounts are very significant for compression performance.

Reviewer comment (Evaluation):
Extending the framework with approximate isomorphism solutions: The authors claim that while no known polynomial-time solution for solving the exact graph isomorphism problem is known, "this limitation can be overcome by approximating an object’s canonical ordering, instead of calculating it exactly. This introduces a trade-off between speed and compression rate in the method, and lowers runtime complexity to polynomial time." Can the authors comment on the tradeoff between the speed and compression rate of this modified method, either here or in the main text?

Author response:
**Extending the framework with approximate isomorphism solutions:** The tradeoff between rate and performance of the approximate version of shuffle coding is promising: Preliminary experiments show that on SZIP graphs, we can achieve speedups of multiple orders of magnitude for a few-percent increase in compression rate compared to optimal-rate shuffle coding, i. e. a 1000x speedup for a 4% rate increase on the “geom” dataset. We implemented a version of shuffle coding that has pseudo-linear runtime and completes on graphs with more than 10 million edges. The approximate method saves the “easy-to-find” redundant bits, leaving the “hard” bits out that cause most of the computational complexity when calculating the canonical ordering in nauty and traces. On all datasets we tested, there seems to be a favorable distribution between these, i. e. many easy bits vs. few hard bits. Characterizing and predicting this tradeoff would be an interesting avenue for future work.

Reviewer comment (Theory):
Definition 2.5 not totally accurate: Presumably, what the authors mean by "inverse functions" is that \(\texttt{decode}\) is the inverse function of \(\texttt{encode}\). In that case the \(\texttt{decode}\) function isn’t really needed and/or can be defined as \(\texttt{decode} = \texttt{encode}^{-1}\). Also, the statement "with respect to \(P\)" does not seem to make sense on its own. A clearer statement would be: "An optimal codec for \(P\) is an invertible function \(\texttt{encode} : M \times X \to M\), which is optimal with respect to \(P\), in the sense that for any [...]." More importantly, in shuffle coding, the \(\texttt{encode}\) function is not invertible, because for \(f \in \mathcal{F}\), applying \(\texttt{decode}(\texttt{encode}(M, f))\) returns the message \(M\) together with the canonical representation \(\bar{f}\) rather than \(f\) itself. While this is a technicality that does not seem to affect the validity of the algorithm, the authors should modify their definition and/or exposition to account for this.

Author response:
**Invertibility of encode function.** The encode function for shuffle coding *is* invertible because it takes an unordered object of type $X=\tilde{F}$ (not $F$), so the codec signature is $M×\tilde{F}→M$. We updated section 3.1 to mention this explicitly. We merely use an ordered object with arbitrary order to represent the unordered object in our implementation, as described in section 3.1. Their order has no meaning in the sense that equality of unordered objects is implemented by comparing canonized versions of their ordered representations. In practice, we use a wrapper data type called `Unordered` to define equality in this way (i. e. `Unordered(‘abc’) == Unordered(‘bca’)`), so that the universal invertibility property `c.decode(c.encode(m, x)) == (m, x)` of codecs is upheld. This is omitted in the listing of section 3.1 for simplicity.

Reviewer comment (Writing):
Initial bit overhead statement needs clarification: The authors argue that the constant bit overhead incurred by bits-back "exists for all entropy coding methods." While all entropy coding methods have a constant overhead amortised as more data are compressed, typical methods such as Arithmetic Coding (AC) have overheads of only a couple of bits. This is far smaller than the claimed overhead of "at most 64 bits" for bits-back in shuffle coding. The authors' phrasing may misrepresent the overhead of other entropy coding methods. A more accurate statement would be: "As in other entropy coding methods, which invariably have similar (though typically smaller) constant overheads, the constant 'initial bit cost' of bits-back is amortised as more data are compressed."

Author response:
We agree with all of the reviewer’s suggestions and have implemented them in our updated paper version. This includes a simplified and better-framed definition 2.5, an updated statement regarding the initial bit overhead of rANS, and a missing clarification regarding units (all results are indeed in bits, this information was missing for Table 5).

Reviewer comment (Writing):
All units in bits?: Are all reported rates, including those in Table 1, in bits? If so, a statement to this effect should be added early in the paper to clarify.

Author response:
We agree with all of the reviewer’s suggestions and have implemented them in our updated paper version. This includes a simplified and better-framed definition 2.5, an updated statement regarding the initial bit overhead of rANS, and a missing clarification regarding units (all results are indeed in bits, this information was missing for Table 5).