Paper: LLMZip: Lossless Text Compression using Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Only Llama2 has been used as the language model in the experiment. It is unclear how the performance of LLMZip would vary when employing LLMs of different types and sizes.

Author response:
We also report the performance of using LLaMa1, which is the older model, in Table 1, and we do see an improvement in performance when LLaMa2 is used. Using LLMs out of the box for LLMzip compression can be very computationally intensive. As the experiments involve processing 1M tokens for each context size , more gpus are needed for longer durations when using larger versions. Hence we limited our focus to just the 7B version of LLaMa.

Reviewer comment (Reproducibility):
LLMs usually require GPUs for execution, whereas traditional compression algorithms can run on CPUs, which are more widely accessible and easier to democratize.

Author response:
This is a valid point. Yet, the article points to a significant gap and a new approach warrants dissemination. The approach may not be ready for widespread dissemination; however, it is a significant step in a new direction.

Reviewer comment (Experiments):
How does the performance of LLMZip vary when using LLMs with various scales, e.g., 13B or 65/70B?

Author response:
We haven’t tried LLMZip with LLMs other than LLaMa1 and LLaMa2, as we believe they suffice to show that pre-trained LLMs do indeed offer substantial compression improvements.

Reviewer comment (Evaluation):
What does the end-to-end running time of LLMZip look like when compared to Zlib?

Author response:
The run times for 100K tokens is 10 hrs for 512 context length and  35 mins for a context length of 32, while zlib does it in less than a second. While these are clearly not impressive run times, these are just out-of-the-box applications, with a lot of scope to be improved. It is also worth noting that 32 context length LLM still significantly outperforms Zlib compression. We do not have runtime comparisons for the other SOTA algorithms like paq8h or bsc.

Reviewer comment (Experiments):
What are the baseline results mentioned in Table 1? I would suggest the authors include more introductions about the baseline methods and their performances.

Author response:
In Table 1, we presented results for different encoding schemes which have been explained in sections 2.1.1, 2.1.2, and 2.1.3, and the entropy bound was presented in section 2.2. We compare our results to ZPAQ and PAQ8h in the discussion of the results (section 4.1). These are state-of-the-art algorithms as reported in various papers on compression. Given the page limitations, we weren’t able to include more background on these algorithms, but we can include more references to these.

Reviewer comment (Evaluation):
Table 3 shows the performance varies based on the summary quality. How can we determine if the side information is helpful or not before compression?

Author response:
Thank you for your insightful question about the variability of performance based on summary quality. While it is challenging to precisely quantify the effectiveness of a summary prior to compression, our observations indicate a trend. Particularly, a detailed and less compressible summary (like Summary 3 in our study) tends to offer more valuable information for compression, as seen in its high compression ratio (1918 characters to 1375 bits, or 0.72 bits per character).

Reviewer comment (Evaluation):
Why does side information available in the encoder and decoder perform better than the encoder-only?

Author response:
When side information is available only at the encoder, the compression ratio is computed as (compressed bits of summary + compressed bits of text)/(number of characters in text), denoted by $\tilde{\rho}$​. When the summary is available at both ends, the compressed summary bits need not be transmitted/stored, hence we need not account for it in the compression ratio.

Reviewer comment (Evaluation):
From the perspective of evaluation metric, does a lower bpc indicate a stronger LLM model potential in common downstream NLP tasks (summarization, translation, classification, etc.)?

Author response:
We are not entirely sure whether a lower bpc indicates a stronger LLM model potential in downstream NLP tasks. We do not make any such claims in the paper. We also don’t think this possibility can be ruled out either given the consistent performance improvement of LLaMA2 over LLaMA1 (as shown in Table 1, last two columns) across all window sizes.

Reviewer comment (Presentation):
Sections 2.1.1 and 2.1.2, which discuss text compression methods other than arithmetic coding, appear superfluous.

Author response:
We included a detailed explanation of Arithmetic coding in our paper to ensure it is self-contained and accessible to those unfamiliar with the concept. This decision aligns with the suggestion from reviewer 1ivf to provide more background on text compression. We believe our approach achieves a balance, but we are open to relocating this section to an appendix if you find it more appropriate.

Reviewer comment (Presentation):
The detailed explanation of Arithmetic coding might be redundant; perhaps it would be better placed in an appendix.

Author response:
We included a detailed explanation of Arithmetic coding in our paper to ensure it is self-contained and accessible to those unfamiliar with the concept. This decision aligns with the suggestion from reviewer 1ivf to provide more background on text compression. We believe our approach achieves a balance, but we are open to relocating this section to an appendix if you find it more appropriate.

Reviewer comment (Novelty):
In Section 2.2 regarding Entropy bounds, the equation H(S) = H(X) / E[B] does not appear to be a significant finding.

Author response:
Section 2.2 is necessary to formally examine entropy bounds in the presence of tokenization in terms of bits per character. In particular, it is important to realize that the token sequence has to satisfy the conditions necessary for the Shannon McMillan Breiman theorem to hold for us to meaningfully use these entropy bounds as a lower bound for compression in terms of bits/character.

Reviewer comment (Theory):
The use of text-summary for compression seems misjudged. Theoretically, adding bits to describe a summary would only be beneficial if the probability estimation isn't flawless.

Author response:
We are not sure why the reviewer thinks that this section is misjudged. In reality, every language model only provides a flawed estimate of the conditional probability and hence, this section is very much relevant to reality. This is an idea that, to the best of our knowledge, hasn’t been presented in any other work. We have provided empirical evidence that this is beneficial when used with a real LLM (Llama2).

Reviewer comment (Evaluation):
The claims about the entropy bounds of the English language are debatable. For instance, Table 1 lists 0.6936, while Table 2 cites 0.7741 from a different dataset. The input data chosen for testing does not seem to be a true representation of English text, with entropy rates that fluctuate depending on the input.

Author response:
There is nothing called a “true representation of English text” that can be used for testing. The statistical nature of the English language is complicated and estimates of the bounds will vary with the realization. Our estimates of the bound (0.6936 / 0.7741) show that the upper bound estimate may be significantly lower than the estimates from Thomas & Cover (1.2) and the reviewer should focus on how large this difference is and the implications of this. Of course, we can average the entropy over more and more inputs to get sharper estimates. That would be a pedantic exercise.